10 Take Aways from STP Summit on Metrics

I had the pleasure of hosting the third Online Summit, delivered by Software Test Professionals: Deliver Value with Testing Metrics: Move Beyond the Myth.  The online summit format consists of 3 sessions each for 3 consecutive days.  The sessions for this summit were:

• My Testing Metrics Allergy – Dawn Haynes
• The Math Behind the Lies – Doug Hoffman
• Change the Questions for More Valuable Answers  – Scott Barber
• Controversies in Quality Metrics – Yvette Francino
• Telling The Testing Story – Fiona Charles
• More "T" and Less "BS" – Jon Bach
• A New World: Metrics That Really Count – Harry Robinson
• Risk Metrics & Reporting – Julie Gardiner
• Top Take Aways  – Scott Barber  
• Speaker Panel – Scott Barber, Fiona Charles, Jon Bach, Doug Hoffman, Harry Robinson, Yvette Francino

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top take aways" list.  This is what I came up with:

Scott's Top 10 Take Aways from:

10 Must-Know Tips for Performance Test Automation

More than other automation, bad performance test automation leads to:

• Undetectably incorrect results
• Good release decisions, based on bad data
• Surprising, catastrophic failures in production
• Incorrect hardware purchases
• Extended down-time
• Significant media coverage and brand erosion

More than other automation, performance test automation demands:

• Clear objectives (not pass/fail requirements)
• Valid application usage models
• Detailed knowledge of the system and the business
• External test monitoring
• Cross-team collaboration

Unfortunately, bad performance test automation is:

• Very easy to create,
• Difficult to detect, and
• More difficult to correct.

The following 10 tips, based on my own experiences, will help you avoid creating bad performance test automation in the first place.

Tip #10: Data Design

• *Lots* of test data is essential (at least 3 sets per user to be simulated – 10 is not uncommon)
• Test Data to be unique and minimally overlapping (updating the same row in a database 1000 times has a different performance profile than 1000 different rows)
• Consider changed/consumed data (a search will provide different results, and a item to be purchased may be out of stock without careful planning)
• Don’t share your data environment (see above)