Top 10 Automation Tips from STP Online Summit

I had the pleasure of hosting the second Online Summit, delivered by Software Test Professionals: Achieving Business Value With Test Automation.  The online summit format consists of 3 sessions each for 3 consecutive days.  The sessions for this summit were:

• Software Test Automation, Where's the Value? – Scott Barber
• Achieving Value with Automated Acceptance Tests – Linda Hayes  
• Developer Level Automation Value from "Velcro" to "Velocity"  – Rob Sabourin  
• The Simple and Powerful Build Verification Tests (BVT) – Alan Page  
• Functional Automation is the Gateway Drug to Continuous Delivery – Adam Goucher  
• Test Automation at the API Layer...Why Bother? – BJ Rollison 
• Experiences with Adding Value Through Test Automation – Mark Fewster  
• Test Automation Misconceptions and Money Pits – Hans Buwalda  
• Top Tips for Addressing the Challenge – Scott Barber  
• Speaker Panel – Scott Barber, Linda Hayes, Alan Page, Adam Goucher, Mark Fewster & Guest Panelist Griffin Jones 

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top tips" list.  This is what I came up with:

Scott's Top 10 Automation Tips from:

Notes:

• Not every tip is specific to test automation
• Several tips are points made by more than one presenter
• Like every good “Top 10 List”, tips are presented in reverse order (according to me)
• I’ve paraphrased many of the tips to make them “quippy” :)

Tip #10: Pure record/playback => FAIL

• Record/playback is simple, seductive, and is (in most cases) not the most valuable answer to a question worth answering.

Tip #9: Learn automation in low risk situations

• Do not start out by automating tests against production
• Do get your skills “up to snuff” before making promises your automation can’t keep

Tip #8: Forget ROI, think Cost/Value

• Provide Business Value
• The ROI of testing can’t be a positive number (unless your business product is testing services)
• Determine what value automation can *reasonably* provide
• Estimate the cost of providing that value
• Ask “is that value worth the cost?”

Tip #7: Valuable automation is trustworthy

• To be trustworthy, automation must not be excessively complex
• Diligently avoid false positives.
• Err on the side of false negatives.

Tip #6: Design Build Verification Tests (BVTs) to tell you if additional testing is valuable at this time

• Do not try to design BVTs to tell you whether or not the build is release ready

Tip #5: Test at different levels to accomplish different goals -- use the tools that work/makes sense

• Unit, Object, Integration, API, and/or GUI
• This is true for manual, automated, or other forms of testing

Tip #4: In the land of automation, test design is king

• Brainlessly automating manual tests typically leads to a whole lot of NO VALUE
• Modularity
• Executable Requirements, Keywords, Classes, & Actions
• Test Data, Test Data, Test Data

Tip #3: Collaborate ... Just do it!

• There is no wall between developers and testers, there are only politics.
• Reject politics and collaborate to accomplish the mission
• Testing is not a union that prohibits others from testing or testers from doing other things.
• Enable the right people, with the right skills to add the most testing value at the right time in the most efficient/effective manner

Tip #2: Automate things that are better suited for computers to do than humans -- only if they have value

• Application deployment is a great candidate for automation – whether or not any automated testing is included.

Tip #1: Do not automate what requires human judgement.

• Just don’t.
• ‘nuff said.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."