Peak Performance: Test Design

Showing posts with label Test Design. Show all posts

Friday, April 6, 2012

Desperately Seeking "Performance Unit Testing" Examples

I've been talking about what I term "Performance Unit Testing" in classes and training courses for a long time. I've been teaching (more inspiring with hints toward implementation) client development teams about it for almost as long. Problem is, all I've got is stories that I can't attribute (NDAs and such) and that simply doesn't cut it when trying to make a point to someone who doesn't (or doesn't want to) get it.

So I'm looking for attributable samples, examples, stories, and/or case studies related to "Performance Unit Testing" that I can use (clearly, with attribution) in talks, training classes, maybe even blogs & articles. If you have something, please email me.

If you're not sure if you've got what I'm looking for, lemme share some desired attributes of what I'm looking for:

10 Must-Know Tips for Performance Test Automation

More than other automation, bad performance test automation leads to:

Undetectably incorrect results
Good release decisions, based on bad data
Surprising, catastrophic failures in production
Incorrect hardware purchases
Extended down-time
Significant media coverage and brand erosion

More than other automation, performance test automation demands:

Clear objectives (not pass/fail requirements)
Valid application usage models
Detailed knowledge of the system and the business
External test monitoring
Cross-team collaboration

Unfortunately, bad performance test automation is:

Very easy to create,
Difficult to detect, and
More difficult to correct.

The following 10 tips, based on my own experiences, will help you avoid creating bad performance test automation in the first place.

Tip #10: Data Design

*Lots* of test data is essential (at least 3 sets per user to be simulated – 10 is not uncommon)
Test Data to be unique and minimally overlapping (updating the same row in a database 1000 times has a different performance profile than 1000 different rows)
Consider changed/consumed data (a search will provide different results, and a item to be purchased may be out of stock without careful planning)
Don’t share your data environment (see above)

Top 10 Automation Tips from STP Online Summit

I had the pleasure of hosting the second Online Summit, delivered by Software Test Professionals: Achieving Business Value With Test Automation. The online summit format consists of 3 sessions each for 3 consecutive days. The sessions for this summit were:

Software Test Automation, Where's the Value? – Scott Barber
Achieving Value with Automated Acceptance Tests – Linda Hayes
Developer Level Automation Value from "Velcro" to "Velocity" – Rob Sabourin
The Simple and Powerful Build Verification Tests (BVT) – Alan Page
Functional Automation is the Gateway Drug to Continuous Delivery – Adam Goucher
Test Automation at the API Layer...Why Bother? – BJ Rollison
Experiences with Adding Value Through Test Automation – Mark Fewster
Test Automation Misconceptions and Money Pits – Hans Buwalda
Top Tips for Addressing the Challenge – Scott Barber
Speaker Panel – Scott Barber, Linda Hayes, Alan Page, Adam Goucher, Mark Fewster & Guest Panelist Griffin Jones

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top tips" list. This is what I came up with:

Scott's Top 10 Automation Tips from:

Making Every Test Count

This is from a while back, but I wouldn't call it dated. It's a webinar, it runs for 48 min. I like it, for whatever that's worth. ;)

Abstract:

Do you ever find yourself wondering what the point is to executing this test... again!?! Have you ever felt like the purpose of a test is to ensure there is a check mark in a particular check box? Are you ever asked to get *more* information in even less time with even fewer resources than the lst test project you worked on?

In this presentation, Scott Barber will introduce you to a variety of tips and techniques you can apply to virtually any testing you do as you strive to make ever test you execute add value to the project.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Saturday, August 25, 2007

Model Workloads for Performance Testing: FIBLOTS

This is the third installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


For years, I have championed the use of production logs to create workload models for performance testing. During the same period, I've been researching and experimenting with methods to quickly create "good enough" workload models without empirical data that increase the value of the performance tests. I recently realized that these two ideas are actually complimentary, not exclusionary, and that with or without empirical usage data from production logs, I do the same thing, I:

FIBLOTS.

While the play on words makes this mnemonic particularly memorable, I'm not saying that I just make it up. Rather the acronym represents the following guideword heuristics that have served me well in deciding what to include in my workload models over the years.

Friday, August 3, 2007

Classify Performance Tests: IVECTRAS

This is the second installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


I have struggled for over 7 years now with first figuring out and then trying to explain all the different "types" of performance tests. You know the ones:


Well, I finally have an alternative.

IVECTRAS

IVECTRAS is valuable for classifying performance tests (or test cases if you like that term better) and performance test objectives. Better still, it is easy to map to Criteria, Requirements, Goals, Targets, Thresholds, Milestones, Phases, Project Goals, Risks, Business Requirements, Scripts, Suites, Test Data, etc. Yet even better, you can use it as a heuristic to assist with determining performance testing objectives and performance test design. So what is it?

To determine, design or classify a performance test objective or test, ask is this an:

INVESTIGATION or VALIDATION
of END-TO-END or COMPONENT
under ANTICIPATED or STRESSFUL conditions

For me (and my clients since I came up with this) there is a lot less confusion when one says "We need to INVESTIGATE COMPONENT level RESOURCE consumption for the application server under STRESSFUL conditions" than it is to say "We need to do a unit stress test against the application server." Even if there are still questions to be answered after applying IVECTRAS, at least the questions should be more obvious -- and if nothing else, *that* adds value for me.

Peak Performance