Top 10 Tips for Performance Test Tool Evaluation from STP Online Summit

I had the pleasure of hosting the another Online Summit, delivered by Software Test Professionals: Survey of Performance Testing Tools. The online summit format consists of 11 sessions over 3 consecutive days. The sessions for this summit were:

• A Survey of Performance Testing Tools Introduction – Scott Barber & Mark Tomlinson
• Neotys - NeoLoad – Bill Nicholson
• SOASTA - Cloud Test Performance – Dan Bartow
• BlazeMeter - The Load and Performance Testing JMeter Cloud – Alon Girmonsky
• AgileLoad – Adam Brown
• HP - LoadRunner – Shane Evans
• Web Performance Load Tester – Michael Czeiszperger
• Telerik - Test Studio – Jim Holmes
• Compuware - Gomez 360 Web Load Testing – Duane Dorch
• SmartBear - LoadComplete –Michael J. Punsky
• A Survey of Performance Testing Tools Tips and Panel Discussion – Scott Barber, Mark Tomlinson & Summit Speakers

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top tips" list. This is what I came up with:

Scott's Top 10 Tips for Performance Testing Tool Evaluation from:

Notes:

• Not every tip comes directly from a presentation
• Several tips are derived from points made by more than one presenter
• Like every good “Top 10 List”, tips are presented in reverse order (according to me)
• I’ve paraphrased many of the tips to make them “quippy” :)

Tip #10: Start with a *Business* needs analysis

• Last thing you want is to buy a tool you don’t actually need!
• Only slightly better is spending extra $ for features that don’t provide business value

  Tip #9: Consider build, lease, open source/free and outsourced options

• Sometimes you need to buy a pick-up truck
• Sometimes it’s more cost effective to rent a moving van
• Sometimes you should just buy pizza and beer for that buddy of yours with a truck

  Tip #8: Evaluate your technical needs for today… and the future

• If the future technical needs are overly vague, revisit Tip 9!
• If a tool won’t grow with you, the CDs it comes on make some cool, sparkly coasters (except they don’t actually keep liquid off the desk)

  Tip #7: Easy buttons are great, but don’t sacrifice extensibility for easy

• There will always be *something* you want to do that there’s no easy button for
• If you can’t disable the easy button, there *will* come a day when you regret it

  Tip #6: Do your homework, network & check blogs & forums

• Be cautious… almost every tool has both haters & not-obviously-identified-employees with blogs and on forums
• You can often learn more from an hour of reading/talking to users than from an hour working with the tool or reading/talking to vendors

  Tip #5: Make a friend in your procurement department

• Learn your companies rules, regulations, procedures and “no-no’s”

  Tip #4: Choose a vendor you share values with

• Buying a tool is entering into a long term relationship
• Getting married can be lots of fun, getting a divorce is *never* fun

  Tip #3: Take seriously the preferences of the people who will use the tool day-to-day

• Performance testers who like their tool will *find* a way to make it work
• Performance testers who *don’t* like their tools will find an excuse not to bother making it work

  Tip #2: Don't consider buying it until you drive it

• You wouldn’t buy a car without taking it for a test drive… and many of these tools will cost you far more than you paid for your last car!
• Don’t settle for a Demo, or a test drive against a sample application
• IMHO, if a vendor won’t enable you to test drive their tool against your application, they don’t deserve your business

  Tip #1: Don’t factor in the price tag until you’ve narrowed the field to all viable candidates

• Use price as a differentiator among otherwise equivalent options
• Do not use price to decide which options to evaluate
• Do not start with a budget & decide how to spend it

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."