10 Take Aways from STP Summit on Metrics

I had the pleasure of hosting the third Online Summit, delivered by Software Test Professionals: Deliver Value with Testing Metrics: Move Beyond the Myth.  The online summit format consists of 3 sessions each for 3 consecutive days.  The sessions for this summit were:

• My Testing Metrics Allergy – Dawn Haynes
• The Math Behind the Lies – Doug Hoffman
• Change the Questions for More Valuable Answers  – Scott Barber
• Controversies in Quality Metrics – Yvette Francino
• Telling The Testing Story – Fiona Charles
• More "T" and Less "BS" – Jon Bach
• A New World: Metrics That Really Count – Harry Robinson
• Risk Metrics & Reporting – Julie Gardiner
• Top Take Aways  – Scott Barber  
• Speaker Panel – Scott Barber, Fiona Charles, Jon Bach, Doug Hoffman, Harry Robinson, Yvette Francino

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top take aways" list.  This is what I came up with:

Scott's Top 10 Take Aways from:

Notes:

• Not every take away is specific to metrics
• Several take aways are points made by more than one presenter
• Like every good “Top 10 List”, take aways are presented in reverse order (according to me)
• I’ve paraphrased many of the take aways to make them “quippy”:)

Take Away #10: Numbers alone rarely (if ever) tell the whole story

• If you are using numbers to help you tell the story, make sure they are helpful, not harmful

Take Away #9: Don’t do math with “ranking” numbers

• Pri 2 issue does not imply ½ as important as Pri 1 issues
• It might be fun to find the standard deviation, of “defect fairy” interactions per Priority over Severity by release cycle, but it’s not mathematically valid!

Take Away #8: Avoid allowing your numbers imply extra accuracy

• Apply “Significant Digits” rules
• Priority 1.03579 might *seem* cool, but it’s kinda like reporting 1.68 children/household when asked how many children live in *this* home

Take Away #7: Beware Biases

• Seeing what you expect to see
• Workers do what managers check
• Its not the numbers that need to be optimized, its the item/activity

Take Away #6: Metrics are inherently game-able

• Don’t try to win the game, try to avoid the game

Take Away #5: Some metrics are invitations to conversations

• T, B, S is a great example… the numbers are interesting, the conversation is insightful!

Take Away #4: Metrics are only valuable when they reliably answer a question

• Specifically the question they were designed to answer
• If you don’t know what question a metric is going to be used to answer, find out
• Those questions often relate to product risk

Take Away #3: Dashboards help to keep metrics in context

• Dashboards that are customized by project, culture, goals, etc. are best
• The only way to know how well a particular dashboard will work is to test it!

Take Away #2: The experts agree, only use metrics that add value

• They all seem to think that different ones add value
• Some recommend the same metrics as indicators of *crazy-different* things
• Unless you have one of those experts consulting on site, your team knows the value better than the expert’s book does

Take Away #1: The story about the information matters, not the medium

• The #1 story managers and executives want to hear is the risk story
• Numbers are a medium
• So are traffic lights
• And smiley/sad faces

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
Director, Computer Measurement Group
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."