10 Take Aways from STP Summit on Metrics

I had the pleasure of hosting the third Online Summit, delivered by Software Test Professionals: Deliver Value with Testing Metrics: Move Beyond the Myth.  The online summit format consists of 3 sessions each for 3 consecutive days.  The sessions for this summit were:

• My Testing Metrics Allergy – Dawn Haynes
• The Math Behind the Lies – Doug Hoffman
• Change the Questions for More Valuable Answers  – Scott Barber
• Controversies in Quality Metrics – Yvette Francino
• Telling The Testing Story – Fiona Charles
• More "T" and Less "BS" – Jon Bach
• A New World: Metrics That Really Count – Harry Robinson
• Risk Metrics & Reporting – Julie Gardiner
• Top Take Aways  – Scott Barber  
• Speaker Panel – Scott Barber, Fiona Charles, Jon Bach, Doug Hoffman, Harry Robinson, Yvette Francino

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top take aways" list.  This is what I came up with:

Scott's Top 10 Take Aways from:

10 Must-Know Tips for Performance Test Automation

More than other automation, bad performance test automation leads to:

• Undetectably incorrect results
• Good release decisions, based on bad data
• Surprising, catastrophic failures in production
• Incorrect hardware purchases
• Extended down-time
• Significant media coverage and brand erosion

More than other automation, performance test automation demands:

• Clear objectives (not pass/fail requirements)
• Valid application usage models
• Detailed knowledge of the system and the business
• External test monitoring
• Cross-team collaboration

Unfortunately, bad performance test automation is:

• Very easy to create,
• Difficult to detect, and
• More difficult to correct.

The following 10 tips, based on my own experiences, will help you avoid creating bad performance test automation in the first place.

Tip #10: Data Design

• *Lots* of test data is essential (at least 3 sets per user to be simulated – 10 is not uncommon)
• Test Data to be unique and minimally overlapping (updating the same row in a database 1000 times has a different performance profile than 1000 different rows)
• Consider changed/consumed data (a search will provide different results, and a item to be purchased may be out of stock without careful planning)
• Don’t share your data environment (see above)

10 Things About Testing That Should Die

I've taken some heat for discussing the whole "is test dead" concept due to a feeling that I was validating the concept of testing being unnecessary. Allow me to clarify my position. I do not believe, for one heartbeat, that testing as an activity is in any way unnecessary. I do believe that there are things related to the current state of and common beliefs about testing that should die. With that said...

Scott Barber's Top 10 Things About Testing That Should Die: 

10. Egocentricity

Face reality testers, neither the product nor the business revolve around you. Think about it. No business, no product, no developers => no need for testers. In fact, if developers wrote perfect code, there'd be no need for you. You are a service provider and your primary clients are the managers, developers, and/or executives. Your secondary clients are product users and investors. So stop whining and stomping your feet when your clients don't make decisions you like with the information you provide. It's not your call. If you want it to be your call, get on track to become a project manager, product manager, or executive, otherwise, get right with the fact that you provide a service (hopefully a valuable one) and get back to providing it.