Top 10 Tips for Performance Test Tool Evaluation from STP Online Summit

I had the pleasure of hosting the another Online Summit, delivered by Software Test Professionals: Survey of Performance Testing Tools. The online summit format consists of 11 sessions over 3 consecutive days. The sessions for this summit were:

• A Survey of Performance Testing Tools Introduction – Scott Barber & Mark Tomlinson
• Neotys - NeoLoad – Bill Nicholson
• SOASTA - Cloud Test Performance – Dan Bartow
• BlazeMeter - The Load and Performance Testing JMeter Cloud – Alon Girmonsky
• AgileLoad – Adam Brown
• HP - LoadRunner – Shane Evans
• Web Performance Load Tester – Michael Czeiszperger
• Telerik - Test Studio – Jim Holmes
• Compuware - Gomez 360 Web Load Testing – Duane Dorch
• SmartBear - LoadComplete –Michael J. Punsky
• A Survey of Performance Testing Tools Tips and Panel Discussion – Scott Barber, Mark Tomlinson & Summit Speakers

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top tips" list. This is what I came up with:

Scott's Top 10 Tips for Performance Testing Tool Evaluation from:

CloudTest Lite - A Game Changer in the Performance Tool Market

Yesterday, SOASTA announced their new product, CloudTest Lite (Press Release). It's not common that I get excited about a tool product release, but this is different. This product has the potential to change the market for the better.
Scratch that. I'll be shocked if it doesn't change the market for the better.

Why is that, you ask? Consider the following attributes of CloutTest Lite:

• It's a fully featured, easy to learn and use, enterprise class, modern, performance testing tool for web & mobile applications
• All you need to use it is a reasonably modern machine connected to the internet and a web browser.
  • You don't need to buy, install, configure or maintain load generation machines.
  • The "license" is tied to your personal credentials, so you can design, create, execute, and analyze your tests from any machine you want without needing to figure out how to point to the license server, or how to get onto the corporate network from your favorite internet cafe.
  • You can even do much of the design, test enhancement, and analysis entirely off-line.
• You can simulate up to 100 virtual users any time you want. No more scheduling time on the controller days or weeks in advance guessing the app will be ready for your test. No more having to wait until your next scheduled time to re-run your test when you see something 'wonky' in your data.
• It's free.
  • Yes, I said free.
  • As in, you never need to pay a dime. Not today, not when the trial expires, not a year from now to continue your maintenance contract.
  • That's right, it is free from now until the sun explodes (or at least until well beyond when anything we're building or planning to build today is long gone and forgotten)

Imagine the implications:

Thoughts on Performance Testing w/o "Tools"

I was recently asked the following question via the "Ask The Expert" feature of SearchSoftwareQuality.com.

How can we conduct performance testing, stress testing, and load testing of a Web application manually without using any tools?

My commentary is reproduced below -- you'll have to click through to see my actual recommendations.

First, I want to express my sympathy to anyone who finds himself in a position of being asked to create multi-user simulations (i.e. the load part of performance/load/stress testing), requesting a load-generation tool, and being denied. In my experience, the excuse of not being able to afford a load-generation tool is almost always just that -- an excuse.

Only once in my career have I found the only tool capable of generating a production-like load against the application I was testing to be prohibitively expensive. (In that case, the tool cost more to purchase than the application was anticipated to earn in a year.) In every other case, either the risk justified the cost or an adequate, inexpensive, or free tool has been available.

The only really good reason to be in this position is if all available tools were considered and none of them supported the application under test (even with customizations and extensions) and you don't have access to the skills (internally or externally) to build a tool of your own -– and that seems unlikely to me.

The simple truth is that if a company is building an application that is realistically expected to have enough users to justify the expense of performance testing, even if that expense is just the time of an employee, that company ought to be projecting enough revenue from the application, or lose enough credibility by having a poorly performing application, to justify either the cost of a tool or the risk (in some company's eyes) of using a free or open source tool.

Now, after saying all of that, I must admit I have found that the vast majority of value that is gained by quality performance testing comes outside of the load-generation tool. Some of my favorite techniques (assuming you are testing websites):

Read the rest of the article here.
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

HP to buy Mercury Interactive

On Tuesday 7/25/2006, CNNMoney.com (along with *many* others) broke the news that the rumored HP/Mercury deal is really happening. A summary and my reaction is below. See the entire release here and draw your own conclusions.

July 26 2006: 9:22 AM EDT

NEW YORK (Reuters) -- Hewlett-Packard agreed on Tuesday to buy Mercury Interactive for about $4.5 billion in stock, or $52 per share, in a bid to expand the computer maker's business software operations.

The deal, which sent shares of the No. 2 personal computer maker down 4 percent, should help boost sales of HP's (Charts) OpenView systems management software, which makes it easier for far-flung businesses to monitor the hardware, software and networks running throughout their organizations.

The purchase of the former star Israeli technology company also puts HP in closer competition with other major systems management software providers, including IBM's Tivoli unit, CA Inc.'s UniCenter and BMC Software.

Since last year, a number of top Mercury executive have left amid a regulatory probe into its stock option granting practices. The financial scandal drove Mercury, once a top performing stock, to delist from the Nasdaq market.

Folks, you may not realize it, but this is major. Until about a year ago, over 75% (up to 90% depending on which year and which report you read) of the total revenue in the test automation and test management tools market went to Mercury, Rational and Segue since the beginning of the "Dot-Com Era". Over the last 13 months this seemingly consistent market has been turned on it's head: