- This is the second installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.
- Other posts about performance testing heuristics and mnemonics are:
- Installment 1 - Performance Testing Core Principles: CCD IS EARI
- Installment 3 - Model Workloads for Performance Testing: FIBLOTS
- I have struggled for over 7 years now with first figuring out and then trying to explain all the different "types" of performance tests. You know the ones:
- Performance Test
- Load Test
- Stress Test
- Spike Test
- Endurance Test
- Reliability Test
- Component Test
- Configuration Test
- {insert your favorite word} Test
- Well, I finally have an alternative.
- IVECTRAS
- IVECTRAS is valuable for classifying performance tests (or test cases if you like that term better) and performance test objectives. Better still, it is easy to map to Criteria, Requirements, Goals, Targets, Thresholds, Milestones, Phases, Project Goals, Risks, Business Requirements, Scripts, Suites, Test Data, etc. Yet even better, you can use it as a heuristic to assist with determining performance testing objectives and performance test design. So what is it?
- To determine, design or classify a performance test objective or test, ask is this an:
- INVESTIGATION or VALIDATION
- of END-TO-END or COMPONENT response TIMES and/or RESOURCE consumption
- under ANTICIPATED or STRESSFUL conditions
- For me (and my clients since I came up with this) there is a lot less confusion when one says "We need to INVESTIGATE COMPONENT level RESOURCE consumption for the application server under STRESSFUL conditions" than it is to say "We need to do a unit stress test against the application server." Even if there are still questions to be answered after applying IVECTRAS, at least the questions should be more obvious -- and if nothing else, *that* adds value for me.
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me
Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing
"If you can see it in your mind...
you will find it in your life."
2 comments:
Scott,
Performance tests may be described from different points of view and the problem of traditional terminology is that it is not well defined and, moreover, there are conflicting definitions.
For example, when I use terms "load" and "stress" test, I mean "Anticipated" and "Stressful" conditions respectively. Not all people do. So any explisit specification is better that just using vague terms that may be interpreted differently. IVECTRAS is a great step in this direction.
I am a little confused how you classify "Investigation" and "Validation". In my understanding sometimes the same test may be used for validation and for investigation depending on the context.
Some old terms describe the test from other points of view. For example, spike or endurance test. First tells us about load pattern and the second tells us about the length. They probably get into "Investigation", but it doesn't give us information about load pattern or length. Should we add more letters to IVECTRAS?
Alex
Absolutely the same tests are sometimes used for investigation and validation. They sole distinction is in how the results/data/information/etc. are reported/communicated/used. In my experience, knowing before I "press the run button" whether the expected "result" is pass/fail or lots of data to analyze to help folks understand is useful. Knowing that, for example, may impact the degree of logging I use or what things I monitor.
As I define Spike testing, it would be an I(E/C)(T/R)S, where E/C and T/R depend entirely on context. Endurance could fall in any IVECTRAS class... which is part of what I like about the classification system.
Basically, I want the team to decide what they would like the performance testing to accomplish and trust me to design a test that will accomplish it. When a team or client tells me to do X testing, it almost never adds the value they were hoping for unless I am smart enough to schedule a meeting (that usually lasts longer than it would have taken to both design and execute the test in the first place) to figure out what they are hoping to accomplish.
Thus far, asking clients or teams to completely dispose of the terms that folks like you and I have struggled to understand, define and explain and instead use this, have caused them to think about intent and desired outcome BEFORE asking me to run a test, which means that when they do ask me for a test, I've gotten almost all of the information I've needed to create and execute a test that will assist them without needing additional meetings, without tempers flaring, without frustration, and without anyone feeling like an idiot.
I'm afraid that adding more letters will get counter productive.
Cognitively, human beings find it relatively easy to hold 7 items in a list in their head at once. It gets dramatically more challenging to remember the list starting at 8 items. An 8 item list with a mnemonic
is approximately the same difficulty to remember as a 7 without. At 9, with or without a mnemonic, it would be unreasonable to expect anyone who didn't use the list on a virtual daily basis to remember it. For that reason alone, I'd rather keep the list at 8 items knowing that it covers most of what seems to be relevant and can be remembered by most people with relative ease than to add letters and end up with a more "complete" taxonomy that the majority of folks won't use because it's too hard to remember.
Post a Comment