Peak Performance

With the Context-Driven School "closed" what's next?

2012-03-01T01:24:00.001-05:00

This is Part II in a series of entries related to the following quote from the "about page" of context-driven-testing.com hosted by Cem Kaner:

"...However, over the past 11 years, the founders have gone our separate ways. We have developed distinctly different visions. If there ever was one context-driven school, there is not one now..."

If you haven't done so already, I recommend starting with Part I: Is Testing Dead? Dunno, but the Context-Driven School Is

Much like when one completes an educational program at one institution and ponders whether or not to enroll in another program (and if so, which one), or to enter the workforce and continue their learning along the professional development or self-education path, I think it's fair for those who have come to self-identify as members of the Context-Driven School to be asking themselves similar questions.

And much like completing an educational program does not equate to losing the lessons learned (as opposed to the lesson's taught) in the program, the Context-Driven Principles and the lessons many of us have learned by studying in (or, for that matter, rebelling against) the Context-Driven School remain despite Cem's announcement that (in my words) the school is now closed.

As I've said many times before, my life is and has been for as long as I can remember, Context-Driven. My actions and reactions, decisions and non-decisions are all driven by situations and the context surrounding those situations. There are some things about the context of my life, however, that are rather static, and rather common, for example:

I need air, sleep and nourishment fairly regularly if I expect to continue living.
I need money to provide for the basic needs of my family.
I am a U.S. Citizen and therefore must comply with (or at least not get caught failing to comply with) certain rules to avoid the associated consequences (i.e. I pay taxes)

The time that I spent studying in the Context-Driven School helped me to better understand, and to better articulate to others, my actions, reactions and decision making processes more effectively.

The same is true about my testing. What I do and do not test, what methods I employ and what approaches I avoid are all driven by situations and the context surrounding those situations. There are some things about the context of my testing, however, that are rather static, and rather common, for example:

I test for organizations that create software and/or software systems to generate revenue, or to help them keep more of the revenue they generate.
I am not the sole decision-maker related to the software/systems I test.
The organizations I test for expect (but can only rarely articulate) that the testing I do will help the organization create and ship software/systems as quickly and cheaply as possible at a satisfactory level of quality for the software/system to meet or exceed the value to the organization that was anticipated when the decision to create the software/system was made.

The time that I spent studying in the Context-Driven School helped me to better understand, and to better articulate to others, what I do and do not test, and the thought processes behind my choices of methods or avoidance of approaches. Additionally, my time in the Context-Driven School helped me to become a far better coach and mentor to testers, in no small part by providing me with the Context-Driven Principles to use as a sort of a topical outline for lessons I'd have skipped entirely (to the detriment of those I was coaching or mentoring) believing they were as unnecessary as lessons in blinking.

In that light, I thank the Context-Driven School and am sad to see it's doors closed.

but there is another light under which to view the closing of the Context-Driven School that I feel compelled to share. Compelled by my inner tester, my ethics and my unwillingness to knowingly present only one side of a story -- even when it may serve me better to do so. The Context-Driven School undeniably had impact, instigated events and inspired actions that some (maybe many) people reasonably classify as negative. Whether some or any of those were deliberate or intentional side-effects is not particularly relevant at this point (and not at all relevant to the point of this post). Examples of those "reasonable to view as negative side-effects" include:

Polarization and bad blood in TesterLand (already well covered elsewhere)
A non-trivial number of vendors and service providers spending irresponsible volumes of time, money and energy attacking and/or trying to refute Context-Driven Principles and Supporters in defense of their prized and profitable (though all-too-frequently anti-valuable) snake oil, sold under the label "Best Practices." I say irresponsible because to me, responsible behavior would have been to spend that same volume of time, money and energy working toward making those products and services at least *less* anti-valuable.

{personal note} I will probably spend the rest of my life using what little influence I may have to encourage all corporate entities to embrace some degree of interest in at least *trying* to add some value to something other than, or even in addition to, their own profitability, but as I noted earlier, there are some things about context that are rather static and rather common. C'est la vie. {/personal note}
A not insignificant number of testers misused the Context-Driven Principles to justify taking an excessively idealistic, "high and mighty" attitude about testing -- basically by making them feel they had not only the right, but the duty, to say:"I am the tester. I have evaluated the context and determined X to be the appropriate course of action, therefor you, the employer, shall take this course of action, else I shall blame *you* for all bad things that happen while *I* continue down my a path toward what *I* know to be the one true answer in this context."

That last point is indicative of the one thing that always gave me pause regarding the Context-Driven School, is the inspiration behind a project that I've been working on, with several collaborators, in the shadows for some time, and is now driving me to share where I'd like to see TesterLand focus on next... Business Value.

I think that the Context-Driven Principles are brilliant in their universal applicability, but (to me) the single mention of "Product" is a far too subtle reference Business Value, which I believe to be the Context-Driver that is both most important and most common across the overwhelming majority of testers. In other words, I believe that with few exceptions:

Testers are employed by organizations who reasonably expect that the employed testers will provide value to that organization
The value testers are employed to provide relates to helping the organization create and deliver software/systems that are intended to either generate revenue or enable higher profit margins from revenue generated by other products/services, as quickly and cheaply as possible, without sacrificing quality to the point of undermining the revenue-related goals that led to the organization to initially fund the creation of said software/system.
Tester's primary value contribution is expected to result from testing.

Unsurprisingly, I'd recommend that testers employ a Context-Driven approach to delivering Business Value, but as this blog entry is already epic and "A Context-Driven Approach to Delivering Business Value" sounds like a fabulous title for my next installment, I think I'll close here for now.

Stay tuned, more to follow soon.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
Director, Computer Measurement Group
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Is Testing Dead? Dunno, but the Context-Driven School Is

2012-02-28T09:11:00.000-05:00

Well, I'm sure this is a bit of a shocker for many of you, but the following quote comes from the "about page" of context-driven-testing.com hosted by Cem Kaner:

"...However, over the past 11 years, the founders have gone our separate ways. We have developed distinctly different visions. If there ever was one context-driven school, there is not one now..."

This is Part I of a series of entries on this topic. Links to subsequent parts will be added to the bottom of this entry as they are posted.

Of course, this doesn't negate or erase the Context-Driven Principles, and Cem has committed to keeping the original content on landing page of the revised site:

"...When you land on this site, you see the context-driven-testing.com landing page (the Principles) as it was when we originally published it. I’ll keep it that way (with the same set of Principles), because several people have found it useful..."

To my way of thinking, the *most* important point made by Cem on the About Page is the following:

..."This notion of evolution comes with a built-in assumption: If my thinking will evolve to something else in the future, it must be wrong today. Progress on my path to better understanding and practice of testing (and of anything else that I’m serious about) includes discovering what needs to be changed in my thinking, and changing it.

This is an important aspect of science. We don’t run experiments to confirm what we already know. We run experiments to prove that what we think we already know is wrong. And to help us develop something better..."

This is the point I'd like folks to focus on.
Context-Driven thinking and approaches are not going away & they have served many of us well. Naming them has also caused some interesting, many unfortunate, and some downright detrimental effects on what I like to call "TesterLand". As my regular readers are aware, I've self-identified as Context-Driven almost since I first heard the term -- not just when it comes to testing, but as a description of my approach to life in general. That is true whether or not there is an official School, whether it's popular, or whether helps or hinders my reputation.

When it comes to TesterLand, the question is... what comes next? Does a new school form? Do folks collaborate or compete for the new rallying cry? I have my own thoughts that I'll share in a subsequent post, but for now, I invite you to ponder this and decide what it means to you.

Part II: With the Context-Driven School "closed" what's next?

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
Director, Computer Measurement Group
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Curse of the Performance Tester?

2011-12-25T17:40:00.000-05:00

Seriously?!? After wrapping gifts until nearly 5am (I was behind by even my standards due mostly to work travel, client commitments & preparing to close the corporate books for 2011), and getting up before 8am to celebrate Christmas with my boys, I finally stole a few minutes when I noticed they'd both fallen asleep on the couch to play with *my* new toy (i.e. install Skyrim), only to be foiled by...

... poor performance!

You tell me, is this the curse of the performance tester? or is this just another reminder that there are still plenty of orgs out there that might want to seriously consider investing in some training and/or services from a "top-shelf" performance tester -- whether that's me ('cause it would be nice to make some $ during my unexpected availability due to Skyrim's unavailability), or someone else ('cause if Steam servers get less busy, it' be nice to have some play time before getting back on the road) isn't the point.

The point is, as consumers, we deserve - and should demand - better. And as technologists we should stop pretending that what we're doing is good-enough, and proactively make the appropriate investments in performance-related services, education, processes and attitudes before consumers rebel en mass (like they did when GoDaddy announced, then backed down, from strongly supporting SOPA earlier this week).

Oh yeah, I almost forgot - I wish you all a great holiday season and hope none of your holiday plans are foiled by the Curse of the Performance Tester.

Update: T=0 hrs

Steam connected. Skyrim updates begin downloading @ 34Kbs (Tested my connection and data rate -- I'm well over 750Kbs)

Update: T+5hrs

Still connected, still downloading Skyrim updates. Down to 31Kbs. 40% complete. Dunno if it's Steam, Bethesda, or even if the two are related, but I wonder if it would still be downloading so slowly if I'd spent the last 5 hrs doing some perf testing instead of periodically looking at the number to see if I'll even get the game installed before my first con-call in the morning.

Update: T+5hrs 39min

Suddenly jumped to 1.8Mbs and am at 62%. Would love to know if someone fixed something, or if a bunch of folks just gave up around 11pm

Update: T+ 6hrs 8min

Updates downloaded... time to install... wonder if I'll even managed to stay awake long enough to get through character creation.

Update: 4 days later

Last night I installed the game on a new machine... went through the entire process and *averaged* 1.8Mbs for the download with a peak of 2.0Mbs. In theory, I should get up to 10Mbs for downloads through my service provider, so I'm curious whether there's a 2.0Mbs speed limit on the server side, on my router, due to wireless signal strength, etc, but I think I'd rather spend my time w/ my kids, doing some billable work, and/or actually *playing* the game than research that right now.

BTW, the game lives up to the hype, so in the end, I guess it's worth it -- making me just one more Netizen guilty of tolerating poor performance because "getting it done online eventually" is still better than the alternatives.

And that, my friends, is the entire point. Every one of us that quietly tolerates poor performance is giving companies tacit permission to continue delivering poor performance. We are the reason we are faced with rampant poor performance, not the service providers. Companies simply aren't going to put any more focus on improving performance than they have to in order to keep us at "quietly tolerating". If we want better, we need to be less quiet about it... myself included.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
Director, Computer Measurement Group
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

10 Take Aways from STP Summit on Metrics

2011-12-16T13:13:00.000-05:00

I had the pleasure of hosting the third Online Summit, delivered by Software Test Professionals: Deliver Value with Testing Metrics: Move Beyond the Myth. The online summit format consists of 3 sessions each for 3 consecutive days. The sessions for this summit were:

My Testing Metrics Allergy – Dawn Haynes
The Math Behind the Lies – Doug Hoffman
Change the Questions for More Valuable Answers – Scott Barber
Controversies in Quality Metrics – Yvette Francino
Telling The Testing Story – Fiona Charles
More "T" and Less "BS" – Jon Bach
A New World: Metrics That Really Count – Harry Robinson
Risk Metrics & Reporting – Julie Gardiner
Top Take Aways – Scott Barber
Speaker Panel – Scott Barber, Fiona Charles, Jon Bach, Doug Hoffman, Harry Robinson, Yvette Francino

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top take aways" list. This is what I came up with:

Scott's Top 10 Take Aways from:

Notes:

Not every take away is specific to metrics
Several take aways are points made by more than one presenter
Like every good “Top 10 List”, take aways are presented in reverse order (according to me)
I’ve paraphrased many of the take aways to make them “quippy”:)

Take Away #10: Numbers alone rarely (if ever) tell the whole story

If you are using numbers to help you tell the story, make sure they are helpful, not harmful

Take Away #9: Don’t do math with “ranking” numbers

Pri 2 issue does not imply ½ as important as Pri 1 issues
It might be fun to find the standard deviation, of “defect fairy” interactions per Priority over Severity by release cycle, but it’s not mathematically valid!

Take Away #8: Avoid allowing your numbers imply extra accuracy

Apply “Significant Digits” rules
Priority 1.03579 might *seem* cool, but it’s kinda like reporting 1.68 children/household when asked how many children live in *this* home

Take Away #7: Beware Biases

Seeing what you expect to see
Workers do what managers check
Its not the numbers that need to be optimized, its the item/activity

Take Away #6: Metrics are inherently game-able

Don’t try to win the game, try to avoid the game

Take Away #5: Some metrics are invitations to conversations

T, B, S is a great example… the numbers are interesting, the conversation is insightful!

Take Away #4: Metrics are only valuable when they reliably answer a question

Specifically the question they were designed to answer
If you don’t know what question a metric is going to be used to answer, find out
Those questions often relate to product risk

Take Away #3: Dashboards help to keep metrics in context

Dashboards that are customized by project, culture, goals, etc. are best
The only way to know how well a particular dashboard will work is to test it!

Take Away #2: The experts agree, only use metrics that add value

They all seem to think that different ones add value
Some recommend the same metrics as indicators of *crazy-different* things
Unless you have one of those experts consulting on site, your team knows the value better than the expert’s book does

Take Away #1: The story about the information matters, not the medium

The #1 story managers and executives want to hear is the risk story
Numbers are a medium
So are traffic lights
And smiley/sad faces

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
Director, Computer Measurement Group
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

10 Must-Know Tips for Performance Test Automation

2011-12-05T21:53:00.001-05:00

More than other automation, bad performance test automation leads to:

Undetectably incorrect results
Good release decisions, based on bad data
Surprising, catastrophic failures in production
Incorrect hardware purchases
Extended down-time
Significant media coverage and brand erosion

More than other automation, performance test automation demands:

Clear objectives (not pass/fail requirements)
Valid application usage models
Detailed knowledge of the system and the business
External test monitoring
Cross-team collaboration

Unfortunately, bad performance test automation is:

Very easy to create,
Difficult to detect, and
More difficult to correct.

The following 10 tips, based on my own experiences, will help you avoid creating bad performance test automation in the first place.

Tip #10: Data Design

*Lots* of test data is essential (at least 3 sets per user to be simulated – 10 is not uncommon)
Test Data to be unique and minimally overlapping (updating the same row in a database 1000 times has a different performance profile than 1000 different rows)
Consider changed/consumed data (a search will provide different results, and a item to be purchased may be out of stock without careful planning)
Don’t share your data environment (see above)

Tip #9: Variance

Static delays yield unrealistic results (a range of +/- 50% is typically adequate)
Delays between each page should be different (users do not spend the same amount of time on every page)
Script multiple paths to the same result (not every user will take a direct path to their desired result)
Don’t let every path run to completion (not every user will finish what they started)

Tip #8: Object Orientation

Separate scripts for every path is unrealistic (this can lead to a 1:1 ratio of scripts to simulated users)
Many paths have overlapping activities (without OO, a change to single webpage can lead to dozens of script edits)
Script maintenance is difficult enough (building OO scripts can make maintenance up to 10x simpler)
Makes custom functions viable (code once, reuse over and over – even on future projects)

Tip #7: Iterative/Agile

Writing performance scripts is development (if you don’t treat it as such, you’ll regret it at execution time)
Code some, test some (formal development practices are not generally necessary; applying sound principles is)
The application will change, so will scripts (it’s more efficient to keep up with changesbuild-to-build than all at once)
Use configuration management(when scripts work against a build, check them into the CM system with the build – roll-backs happen)

Tip #6: Error Detection

Tools have weak error detection (particularly if your site has custom error pages/messages)
Error pages tend to load *very* quickly (a test that has 50% undetected “page not found” errors will have fantastic performance results)
Custom functions are often needed (yes, this means writing real code – get help if you need it)
Don’t believe your performance results until you check the logs (see above)

Tip #5: Human Validation

Building scripts that *seem* to work is easy (building scripts that *do* work can be hard... Check logs & use the application manually while tests are running)
Performance test results can be misleading (reported response times aren’t always similar to what users see – get humans on the system while it’s under load)
Numbers don’t tell the whole story (4 seconds may sound good, but users may experience 8 seconds outside your firewall)
Users like consistent performance (get users on the system, then inject load – pay attention to their responses)

Tip #4: Model Production

Results are only as accurate as your models (focus on how the system *will* be used, not how someone *hopes* it will be used)
Use multiple profiles/models (usage patterns can vary dramatically over time – the same volume of traffic in a different pattern can change performance remarkably)
Don’t extrapolate results (when the environments don’t match, don’t guess what production will be)
Validate models before it’s too late (limited beta releases are often best)

Tip #3: Reverse Validate

Released does not mean done (almost everyone pushes a patch shortly after release 1 – plan on it)
Check your model against production usage (typically at the end of week 1 and month 1 are good)
Re-run in test environment with revised models (you may be surprised how much the performance results differ)
Compare results from re-run against previous runs *and* production (this is the only way to validate your predictions and/or improve future predictions)

Tip #2: Tool-Driven Design

The tool was not made to test your application (expect to have to do some things the tool doesn’t make easy)
Do not limit your tests to what is easy in the tool (it is frequently the things the tool doesn’t handle that causes performance problems)
Don’t be afraid to use multiple tools (sometimes its simply easier to launch two tests from different tools than it is to get one tool to do everything)
Tools should make your job easier (else, get new tools)

Tip #1: Value First

Sometimes the best automation is no automation (spending a week to script a difficult rare activity is not a good use of time – do that activity manually during test runs)
Don’t fall in love with your scripts (applications change, treat your scripts as disposable – it’s often more efficient to re-record than to debug)
Make custom code reusable (reusable custom functions save time later)
Before choosing to build a complicated script, ask:

“Is this the most valuable use of my time?”

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

10 Things About Testing That Should Die

2011-11-29T19:18:00.001-05:00

I've taken some heat for discussing the whole "is test dead" concept due to a feeling that I was validating the concept of testing being unnecessary. Allow me to clarify my position. I do not believe, for one heartbeat, that testing as an activity is in any way unnecessary. I do believe that there are things related to the current state of and common beliefs about testing that should die. With that said...

Scott Barber's Top 10 Things About Testing That Should Die:

10. Egocentricity

Face reality testers, neither the product nor the business revolve around you. Think about it. No business, no product, no developers => no need for testers. In fact, if developers wrote perfect code, there'd be no need for you. You are a service provider and your primary clients are the managers, developers, and/or executives. Your secondary clients are product users and investors. So stop whining and stomping your feet when your clients don't make decisions you like with the information you provide. It's not your call. If you want it to be your call, get on track to become a project manager, product manager, or executive, otherwise, get right with the fact that you provide a service (hopefully a valuable one) and get back to providing it.

9. Unreasonable Demands

Testers, none of the following people work for you: managers, executives, or product team. You provide a service to them (see point 10 above). Stop trying to tell them what they must do before you're willing to start doing what you were hired to do. It is completely unreasonable to demand "complete and accurate requirements or specifications." Seriously, it's not going to happen -- and even when someone claims that they are complete, how many times have you seen a software product not get released due to a single, minor "requirement" not being met? It is also completely unreasonable to demand stable releases before you start testing. Honestly, what the team needs is for you to help them figure out what needs to be fixed, not for you to tell them to fix it before you're even willing to try to help. It's ok to let folks know what things you can't reasonably do without X, but after that it's time to get to work doing what you can to help the project. If that's testing, great. If that's creating end-user documentation, fine. If that's taking a break and doing your quarterly expense report because you're blocked, that's fine too.

8. Process Weinie-ness

Even though many testers have the label of QA, very, very few are actually responsible and accountable for enforcing processes across the team. The reality is that processes were put in place to manage or fix something that wasn't working, not to block progress. Just do what makes sense to help make the project as successful as possible as quickly, cheaply and painlessly as possible. For example:

If your team likes you to report bugs verbally as you find them, and they get fixed directly, is there really a reason to write up some complex report in the defect tracking system? Defect tracking systems are just customized workflow engines anyway. If the work can be complete before you can get it entered into the system, there's no work to track. If you mine the defect tracker for patterns, then just enter the info needed for mining -- which I'm willing to bet doesn't include detailed replication steps.
If the "process" says that you're supposed to write detailed, step-by-step test cases that a brain damaged zombie could follow before you ever even see a prototype or a mock-up and you know that they won't get used or updated, don't waste your time. Instead, find out what problem that process was put in place to solve and help the appropriate folks find a better solution.

*Note* Sometimes there are processes in place for legal or regulatory reasons. In these cases, it's still not your job to enforce them, just to comply with them... and to report anyone who asks or encourages you to do otherwise through the appropriate channels.

7. Isolation

Unless you are doing IV & V (if you don't know what I mean, you aren't - trust me), I simply cannot understand the value proposition of the big, opaque, soundproof wall I frequently see between the developers and testers. I can't even come up with a realistic risk it might mitigate that can't be mitigated better some other way without sacrificing the value of collaboration. The bottom line is that we are all paid by the same company to help them build and deliver a viable product as quickly and cheaply as possible with a degree of quality that is acceptable to the buyer/end-user at the price the company plans to charge. Until someone can explain to me how to do this better with isolated vs. collaborative teams, I can only presume that the reason the isolation is either the result of some turf war at the management (or higher) level, a naive historical artifact, or unadulterated stupidity.

*Note* Even if you are doing IV & V, or there is some valid reason to have a group doing isolated, independent, "unbiased by collaboration or inside knowledge" testing, that's no excuse for not having another group of testers collaborating with the rest of the team.

6. Excessive Faith in "Dumb" Automation and Checking

Sometimes, it is important to check things. Automation can be a fast and useful way to check things. Automation can be an amazing time saving device for brain engaged testers. And, of course, automation is absolutely critical to some specialized types of testing, like load and stress testing. That being said, unless your product is so simple that checks are sufficient, or your automation includes some super-advanced artificial intelligence that can reliably mimic human judgement, don't kid yourself into believing that just because a bunch of automated checks turn the magic cell green, that means that the product is flawless, needing no further inspection, validation, or testing by a brain-engaged human being. Do yourself, your project teammates and your company a favor, don't put excessive faith in your automation, and do your best not to let anyone else do it either.

5. Misguided Metrics

Counting test cases as if they are standardized units of measures as an indication of coverage. Reporting percentage of tests passed vs. failed as if all tests are both binary and of equal value as an indicator of quality. Tracking number of bugs found over time as if it's an indication of the number of defects remaining in the system instead of an indication that the test team has run out of new test ideas. I could go on, but I won't (this post is excessively long already). These are all misguided metrics that have far more potential to cause harm than to help folks make good decisions.

If you don't know what questions your metrics are being used to answer, or you have doubts about whether or not the metrics that are getting reported indicate what people think they do, pause for a moment and ask yourself the following questions:

What are the questions the recipients of these metrics are trying to answer?
Would you use these metrics to answer those questions?
If not, why not?
What metrics would you use?
Can you provide those metrics?
If not, what metrics can you provide that you'd find more useful?

Take those answers seriously and do something valuable with them... even if valuable means tossing the metrics entirely in favor of brief verbal statements. (I know, heretical, right?!?)

4. Testing Phase (aka Belief That Only Testers Can Test)

I can't figure this one out at all. Breaking testing out into it's own phase leads to isolation (see point 7 above), and implies to me that testing should be reserved for this phase. How does that work, exactly? Every time analysts ask questions, that's testing. Every time developers compile or execute code to see if it's working right, that's testing. Every member of the team can, and probably does, test - and anyone who cares about product and project success wants them to. All I can guess is that folks advocating for testing phases either have some special definition of testing that makes the idea of a test phase seem, at least, to be not stupid in a particular context, or are trying to leverage some kind of power or control for some political reason. In either case, I say "ick".

What is the perceived value of a testing phase anyway? An instance of the application that isn't changing during testing? You don't need a phase for that! Just spin up a virtual environment somewhere and promote the build you want to test until you've found enough bugs to make further testing of that build less valuable than moving to a newer build. Don't tell me it's too expensive either... even 3rd party solutions are $0.08/hr for a pretty reasonable virtual server in the cloud. For 10 testers each testing 2000 hrs per year, that's a whopping $1600 total for each of them to have their very own test environment! That's a small price to pay for not having to sit around waiting for the next test phase after you find a showstopper!

And what do we lose by phasing testing (aside from collaboration)? We ensure the developer has moved on to something else by the time we identify things s/he wants/needs to fix, thus having completely lost the thread of what s/he was thinking when s/he coded that section in the first place. So s/he ends up injecting more bugs as compared to if we found them sooner or if we paired with him/her (gasp) and did at least some testing before s/he merged the code into the main branch.

Bottom line; stop being selfish about testing. You want everyone doing as much testing as is valuable and reasonable. You want to be helping them make their testing better. Think about it, the more testing other folks do, and the better they do it, the more chances you'll have to do the really cool and valuable testing that you want to do, but never seem to have time for!

*Note* I'm not advocating against legal or regulatory mandated "final validation" phases. Those are simply beyond the scope of this commentary.

3. Misleading and Unbacked Certifications

As my father, the 8th grade metal shop teacher turned middle school guidance counselor (now retired) told me the day he received a nifty plaque in the mail that said something like "Supervisory Certificate, K-12":

"Cool, according to this, I am now certified to teach any class, at any grade level, in any public school in the state. Of course, I'm far from qualified to teach most of them but with this I'm allowed to anyway"

Of course this became a running joke between he and I which got shortened over time to simply "Certified != Qualified".
As far as I am concerned, any so-called "certification" issued by an org not subject to regulatory oversight that either makes no claims about what you are qualified to do as a holder of the certification or makes claims but offers no guarantees and takes no legal or fiduciary responsibility if the "certified" individual fails to be able to live up to those claims, is not a certification. It may be a certificate of course completion, hoop jumping, or fee paying, but it's no certification. Testerland is full of these. I've written on this topic before, several times, I'll step down from this soapbox and move on to the next one.

2. Ignorance of Businesses Risk and Value

This is really a corollary to point 10 above. Testers seem to have a knack for deciding what they thing matters without any direct knowledge of what the business and/or product owners (eg the ones whose budget your salary comes from) really care about.

Maybe its time to market. In the software business, you often need to be either first to market or best in class to maximize the business value of a product. Trying to do both at the same time will simply result in doing neither effectively.
Maybe its cost minimization. Often times, there is only so much money available to get a product to market, and not getting the product to market means no more money.
Maybe its getting good or avoiding bad press. A bug that causes bad press can be far more expensive than a bug that generates support calls. Bad press keeps people from buying the product in the first place, support calls reduce the profit margin of the products that have been sold.
Maybe its something that appears entirely irrational to you.

Whatever those risks are that the company wants to avoid, mitigate or control, and whatever the company perceives to be a value proposition is what they are paying us to help with. Find out what those things are and tune/focus your testing accordingly. If you find out what those things are and they are morally or ethically distasteful to you, start looking for a new employer. Otherwise, cage up your pet peeves, customer advocacy, and/or commentary on how you'd do things differently if you were in charge and get to work.

If this concept seems strange to you, or you don't feel you understand how focusing on identifying business risks, assessing business risk controls and mitigation measures, and testing for business value would change what you are doing, I recommend getting some training in business and/or product management. I assure you, doing so will make many things that currently confuse/frustrate you about your testing job much more clear.

1. The Under-Informed Leading the Under-Trained to do the Irrelevant

This is how I see the current state of the practice of software testing, not the leaders and not the broken, but rather the "norm". Admittedly, I expect that it's mostly the leaders and the soon-to-be leaders who have read this far, so don't assume I'm talking about you. Having said that, I think it's very rare indeed when this statement isn't at least partly true. If you're not sure what I mean by it, let me explain:

Managers and executives of companies that employ testers are regularly directing testers to do these things they don't really understand rumored to be "best practices"-- and of course, then blame the testers when those "best practices" don't provide sufficient value.

There is no "basic training" for testers, and there is no "manager basic training" that covers testing and test management as it relates to value to the business.

Basically, it all comes down to education and training. For years we've been trying to figure out how to get training to testers and hoping they'd then train their managers and executives about testing but been widely unsuccessful. I think it's time to start putting some effort into educating managers and executives directly. More importantly, I think we are way overdue for testers to start learning more about business and how to apply their skills to better serve the business so we can stop doing the irrelevant as a result of us not knowing how to engage with our managers and executives in their language.

Concluding Statements

The act of testing will never be dead as long as code is being written -- testing is simply part of how software is developed. Who conducts that testing, when, and how is what is really being discussed when folks talk about whether or not "test is dead." I don't believe that testers who provide understandable value to their companies will ever have to worry about their jobs going away. I do believe; however, that testers who aren't able to distinguish their value proposition to the business from the value proposition of other people or groups that can also conduct some degree of testing, or aren't able to deliver on that value proposition, could well find themselves getting pruned from their company's payroll like dead wood.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

On the Alleged Death of Testing

2011-11-08T11:19:00.000-05:00

Out of respect for your time, I'll give you the bottom line up front as a simulated interview that I privately hoped for, but never came. After the mock interview is a supporting narrative for those of you more interested in my thinking on the matter.

Q: There's been a lot of talk recently about testing being dead, so my first question is testing dead?
A: No.

Q: Some of those talking about the alleged death of testing are saying that it's not that testing as a whole is dead, but that testing as it is commonly understood today dead. Is it?
A: No.

Q: Ok, so is testing as it is commonly understood today dying?
A: Not that I can see.

Q: Then why all the talk about testing "as we know it" being dead?
A: IMHO? Wishful thinking.

Q: How so?
A: The state of the testing practice is not evolving nearly as quickly as development, business, or products containing or depending on software. The testing practice is on long-term life support, and that life-support seems to be the ticket to immortality.

Q: Should testers be concerned about this evolution gap?
A: Only if they are afraid of change, have stagnated in their own professional development, and/or believe what they are doing today "is right and will continue to be right for at least as long as they will be testers".

Q: Should others be concerned about this evolution gap?
A: Yes - particularly consumers who care about appropriate quality for cost of products depending on software, and businesses who care about delivering those products with an appropriate balance of speed, cost, market satisfaction and profitability.

Q: What can testers do to narrow this evolution gap?
A: IME, not much. All they can do is embrace evolution as it happens vs. fighting it and thus slowing the evolution yet further and focus on adding value to their businesses rather than focusing on doing what they have decided is important.

Q: If it's not the testers, who can lead the narrowing of the evolution gap and how?
A: Consumers must stop paying for products that depend on software that doesn't achieve their cost for quality bar; Businesses must commit to improving the balance of cost, speed, market satisfaction and profitability; Executives, middle-managers and line-managers must admit that they don't understand the business value they could be, but aren't, getting from testing, get themselves educated and become part of the solution instead of part of the problem.

--

Yes, you read that correctly. I "wish" that testing as it is commonly understood today, at least as it is commonly understood by the corporations, managers, and testers I train and consult with, were dead -- or better still, had not contracted the life-threatening diseases that have required testing to spend the last 30 or so years on life-support systems so restrictive as to limit it's value adding ability.

The problem here, at least in my opinion, is simple.

Testers have become so focused on legitimizing their discipline and their role that they've forgotten that their role and their discipline exists to serve the companies that employ them.
Conversely, the managers and executives of companies that employ testers are regularly directing testers to do these things they don't really understand rumored to be "best practices"-- and of course, then blame the testers when those "best practices" don't provide sufficient value.
There is no "basic training" for testers, and there is no "manager basic training" that covers testing and test management as it relates to value to the business -- and even if you believe the training exists, you can't deny that:
a) many folks wouldn't agree that training is of value
b) even more folks who "should" have received that training haven't even heard of it

Put more simply, software testing today, in my opinion, is dominated by:

The under-informed
leading the under-trained
to do the irrelevant.

The simple truth is that companies don't want to pay for testers and they don't want to have to think about testing. Companies want to create products, as quickly and cheaply as possible, that generate revenue or reduce costs at an acceptable degree of risk to the company. Product quality is a second or third order effect derived from those corporate desires.

It seems clear to me that testing, then, should be all about helping companies create products that are viable for generating revenue and/or reducing costs, quickly and cheaply, while identifying, mitigating, and/or controlling business risk -- not protecting the end-user from that annoying bug. Protecting the end-user is called consumer-advocacy (an honorable discipline to be sure, but I'm willing to bet that it's not the task your company is hiring testers to perform).

So when I say that I wish testing as we've come to know it were dead, what I mean is that I think the propensity of the testing discipline is stuck in trying to force employers to pay them to be consumer-advocates when what those employers really need are trusted advisers focused on helping the company and its products achieve success along an often conflicting continuum of 'success criteria'.

Personally, I'd not be sad if the role of "tester" went away because I know that I have the skills and knowledge to add value to companies by helping them to deliver products (whether those products are software, contain software, use software, or are designed/developed/manufactured using software) faster, cheaper and appropriately fit for use & purpose, while helping those companies identify, mitigate and control associated business risks.

Can you say the same thing?

Would your employer agree?

Think about it.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

WOPR 17, my takeaways

2011-10-27T10:00:00.000-04:00

The WorkshopOn Performance and Reliability (WOPR) 17 was held Oct 20-22, 2011 on thetheme of “Finding Bottlenecks”. Beyondthe fact that this was a historic event in the sense that no other peerworkshop inspired by LAWST has convened thismany times. Of course, as a co-founderof WOPR, I’m (somewhat unreasonably) proud of this accomplishment, but the factthat over the last 9 years so many folks have been so inspired by the communityand value of WOPR that they have been willing to volunteer their time to planand organize these events, their companies have been willing to donate meetingspace (and often food & goodies), and participants have been frequentlywilling to pay their own way (sometime taking vacation time) to attend makes 17events, one every 6 months, since WOPR 1 is a significant achievement – whetheror not my “founder’s pride” is justified. :)

As is the tradition of WOPR 20-25 folks, selected or invitedby the “content owner” (a.k.a. the person or team who chose the theme to beexplored this time) brought their personal experiences related to “FindingBottlenecks” to share and explore with one another. Also as is the tradition, certain patternsand commonalities emerged as these experiences were described and discussed.Everyone has their own take, there are no official findings, and I’m not evengoing to pretend that I can attribute all the contributing experiences and/orconversations to my takeaways below.

Finding bottlenecks can be technicallychallenging, examples include:

Analyzing the test & the data is far fromstraight forward
The “most useful” tools to narrow down thebottleneck may not be available – forcing us to be technically “creative” towork around those roadblocks.

Finding bottlenecks can be *very*socio-politically challenging, examples include:o

Lack of Trust (e.g. “That’s not a bottleneck,that’s the tool!”)
Denial (e.g. “It’s not possible that’s relatedto my code!”)
Lack of cross-team collaboration (e.g. “No, youcan’t install that monitor on *our* system!”)

Sometimes human bottlenecks need to be resolvedbefore technical bottlenecks can be found. (e.g. Perf Team being redirected,resources being re-allocated, excessive micromanagement, etc.)

Some other topics that came up that were relevant andinteresting (such as the frequent discrepancy between tester/technical goals& business goals), but since these weren’t “on theme” we didn’t discussthese topics deeply enough for me to draw any conclusions other than “thepoints and positions that did come up were consistent with what I would haveanticipated if I’d thought about it in advance”, which, for me, is a niceconfirmation.

My point in sharing these thoughts on finding bottlenecks isso that all the folks out there who feel like theirs is the only organizationthat is thwarted by socio-political challenges even more than technical onescan realize that they really aren’t alone.

The findings of WOPR17 are the result of the collectiveeffort of the workshop participants: AJ Alhait, Scott Barber, Goranka Bjedov,Jeremy Brown, Dan Downing, Craig Fuget, Dawn Haynes, Doug Hoffman, PaulHolland, Pam Holt, Ed King, Richard Leeke, Yury Makedonov, Emily Maslyn, GregMcNelly, John Meza, Blaine Morgan, Mimi Niemiller, Eric Proegler, RaymondRivest, Bob Sklar, Roland Stens, and Nishi Uppal.

Best Ice Cream Practice

2011-10-24T10:48:00.000-04:00

A twitter conversation from Friday, Oct 21...

@TesterAB Anna Baik: What's best practice for icecream? I don't know what flavour of icecream I should eat, and I'm afraid to get it wrong.

@skillinen Sylvia Killinen: @TesterAB Test ALL the ice cream, that way you'll know which one best satisfies conformance. :)

@TesterAB Anna Baik: @skillinen I like this idea very much :)

@adampknight Adam Knight: @TesterAB it's vanilla, if you're not eating vanilla you are doing it wrong. I'd suggest getting yourself CVM certified as soon as you can

@adampknight Adam Knight: @sbarber @TesterAB we should be specific. I'll clarify in my "10 ways to check if you are truly vanilla" blog post #BestIceCreamPractice

@sbarber Scott Barber: @adampknight @TesterAB Certified Valuation Manager ™? No, no, no, that's only appropriate for *children's* icecream! #BestIceCreamPractice

@testingqa Guy Mason: @TesterAB Best to go for that which you most prefer at that point in time?

@TesterAB Anna Baik: @testingqa No no no. There must be one flavour of icecream that is best for everybody to eat at all points in time.

@sbarber Scott Barber: @TesterAB @testingqa ROTFL!! That would be the #BestIceCreamPractice, right?

@TesterAB Anna Baik: @sbarber @testingqa Yes! None of this wishy-washy nonsense, I only want to eat the BEST flavour of icecream. #BestIceCreamPractice

@sbarber Scott Barber: @TesterAB @testingqa So chocolate, pistachio, lemon sorbet, raspberry swirl, topped with caramel & orange soda, right? #BestIceCreamPractice

@TesterAB Anna Baik: @sbarber @testingqa #BestIceCreamPractice Finally, someone who'll give me an answer! ...wait. How do I know you're qualified?

@sbarber Scott Barber: @TesterAB @testingqa #BestIceCreamPractice I founded a non-profit to establish BICP qualification stds and issued myself a certification.

@TesterAB Anna Baik: @sbarber @testingqa #BestIceCreamPractice Sounds reassuring, I knew there'd be an Official Body somewhere to tell me what icecream to eat

@sbarber Scott Barber: @TesterAB @testingqa #BestIceCreamPractice The invoice for my services are in the mail. $400/hr + $25,000 for the BICP flavor report.

@can_test Paul Carvalho: @sbarber @TesterAB @testingqa do you have a template for that report? ;)

@sbarber Scott Barber: @can_test @TesterAB @testingqa Oh yes, but it's only for members. Membership is a modest $2,500/year #BestIceCreamPractice

@TesterAB Anna Baik: @sbarber @testingqa Eeek! Don't I even get something to show to people to prove I now know #BestIceCreamPractice?

@sbarber Scott Barber: @TesterAB @testingqa #BestIceCreamPractice when check clears we mail you a Certified BICP Practitioner Certificate (suitable for framing)

@testalways Eusebiu Blindu: @sbarber @adampknight @TesterAB I am thinking to create some "opensource" certificate, free to use in CV, with listings in some website

@sbarber Scott Barber: @testalways @adampknight @TesterAB #BestIceCreamPractice no worries, I have mtgs w/ governments to mandate BICP certs for icecream buyers

Questions? No? Didn't think so. :)

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Please, no new "certifications"

2011-10-18T14:12:00.000-04:00

I just saw an advertisement for this Building a Certification Testing Program - Cutting through the hype to see how it really works on LinkedIn, and I couldn't stop myself from adding the following comment:

Please make it stop. We don't need more "certification" programs -- not unless you are going to be the first organization that allows itself to be held legally and financially accountable when people you "certify" can't do what you "certified" they can.

Otherwise, conduct all the training you want. Assess student performance if you want. Only "pass" students who "pass" the assessment if you want.

Just do us all a favor and *STOP* calling it certification until you are willing to do things like:
reimburse hiring expenses to employers who hire folks you certified as being able to X who can't X
implement periodic re-assessment to enforce some bar of continued knowledge/skill/ability over time
implement some way to revoke certifications of folks who fail to demonstrate knowledge/skill/ability in the workforce
The list goes on, but I know it's pointless. The certification machine will continue no matter how loudly, or how frequently I point out the ways in which it is frequently (at least arguably) unethical and fraudulent - at least in "testerland."

Seriously, this drives me insane. Others can make stands about content, assessment methods, etc. -- I have my opinions on those things, but honestly that part of the topic bores me. People decide what university to attend, what to major in, what electives to take, etc. for their degree programs ... they can decide on whether or not the content of some professional development program (with or without "certification" program) is worth their effort. What I want to see is the "certifying bodies" being held accountable for complying with the claims they make about the individuals they "certify."

I mean, seriously, have any of you seen any data that you'd consider either statistically significant, empirical (vs. anecdotal), or free enough from obvious experimental design flaws to support the claims we see from "certifying bodies"? If you have, please share the data with me and I'll list it in line -- unless of course, it's flawed, in which case, I'd be happy to point out how and why the data doesn't support the conclusion.

Otherwise, please, please, please don't engage in creating more of these things. Please.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Having lunch with a giant...

2011-10-17T23:22:00.000-04:00

I "officially" started my career in software performance in Feb of 2000, only much later to realize I'd started down that path years prior. In the fall of 2001 (10 years ago), I felt I was stagnating in my self-guided education and went on a hunt for books, articles, training, and/or people to learn from. I found some peers (and eventually co-founded WOPR with Ross Collard to maximize peer learning) and I found 3 "giants" on whose shoulders I've stood since then (meaning, all of my work was and has remained consistent, complimentary, and/or extended from their work in the field). Those "giants" are Connie Smith, Ph.D. (Software Performance Engineering), Daniel Menasce, Ph,D. (Capacity and Scalability Planning) and Alberto Savoia (Performance Testing).

Last fall, I had the honor of being on a panel with Connie and spending some time talking to Daniel during the CMG conference in Orlando. I'd never spoken or corresponded with them before that, but it was nice to meet them and we had some great conversations.

Over the years, however, I have corresponded regularly with Alberto Savoia. As it turns out, he was moving on to what he would now call his next "it" from software performance as I was becoming known in the industry, so we didn't converse regularly, but we did follow each others careers. During that time, I drew a lot of inspiration from Alberto. Not just from the work he'd done in the software performance space, but also from his other accomplishments in technology, the kind and complimentary recommendations he gave me and by graciously agreeing to write a forward for Performance Testing Guidance for Web Applications when I asked.

So earlier this year when I had the chance, I dropped everything to review and comment on his new "it", Pretotyping. He said the review was helpful and that some of what I'd commented on would be included in the next version.

Today, I finally met Alberto face to face. We had lunch. We talked about projects & passions old and new, we recalled history and speculated about the future. He gave me a signed copy of Pretotype It, and I gave him a signed copy of Web Load Testing for Dummies, both of which had been prepared in advance. And while Alberto has accomplished far more in his technology career than I have, somehow I didn't feel like I was having lunch with the "giant" on whose shoulders most of the work I am known for stands, I felt like I was having lunch with an old friend that I hadn't seen in too long.

To some of you, I suspect this seems a silly thing for me to be making a big deal about, but for a guy who left a small town twenty-some-odd years ago, never imagining that I'd meet anyone "famous", let alone become a "celebrity" of sorts in my (admittedly very small) field, it means a lot to me that someone who I've often credited as being a luminary to me, would not only take the time to have lunch with me, but to share thoughts and ideas with me like friends do.

So, thanks Alberto. Thanks for the years of inspiration & thanks for the confirmation of friendship. It means a lot to me, and know that you've provide me with another lead I intend to follow with anyone I may inspire during my career and later have the opportunity to meet.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Top 10 Automation Tips from STP Online Summit

2011-10-13T23:51:00.001-04:00

I had the pleasure of hosting the second Online Summit, delivered by Software Test Professionals: Achieving Business Value With Test Automation. The online summit format consists of 3 sessions each for 3 consecutive days. The sessions for this summit were:

Software Test Automation, Where's the Value? – Scott Barber
Achieving Value with Automated Acceptance Tests – Linda Hayes
Developer Level Automation Value from "Velcro" to "Velocity" – Rob Sabourin
The Simple and Powerful Build Verification Tests (BVT) – Alan Page
Functional Automation is the Gateway Drug to Continuous Delivery – Adam Goucher
Test Automation at the API Layer...Why Bother? – BJ Rollison
Experiences with Adding Value Through Test Automation – Mark Fewster
Test Automation Misconceptions and Money Pits – Hans Buwalda
Top Tips for Addressing the Challenge – Scott Barber
Speaker Panel – Scott Barber, Linda Hayes, Alan Page, Adam Goucher, Mark Fewster & Guest Panelist Griffin Jones

One of my duties as host was to try to summarize the most valuable nuggets of information from across all of the presentations into a "top tips" list. This is what I came up with:

Scott's Top 10 Automation Tips from:

Notes:

Not every tip is specific to test automation
Several tips are points made by more than one presenter
Like every good “Top 10 List”, tips are presented in reverse order (according to me)
I’ve paraphrased many of the tips to make them “quippy” :)

Tip #10: Pure record/playback => FAIL

Record/playback is simple, seductive, and is (in most cases) not the most valuable answer to a question worth answering.

Tip #9: Learn automation in low risk situations

Do not start out by automating tests against production
Do get your skills “up to snuff” before making promises your automation can’t keep

Tip #8: Forget ROI, think Cost/Value

Provide Business Value
The ROI of testing can’t be a positive number (unless your business product is testing services)

Determine what value automation can *reasonably* provide
Estimate the cost of providing that value
Ask “is that value worth the cost?”

Tip #7: Valuable automation is trustworthy

To be trustworthy, automation must not be excessively complex
Diligently avoid false positives.

Err on the side of false negatives.

Tip #6: Design Build Verification Tests (BVTs) to tell you if additional testing is valuable at this time

Do not try to design BVTs to tell you whether or not the build is release ready

Tip #5: Test at different levels to accomplish different goals -- use the tools that work/makes sense

Unit, Object, Integration, API, and/or GUI
This is true for manual, automated, or other forms of testing

Tip #4: In the land of automation, test design is king

Brainlessly automating manual tests typically leads to a whole lot of NO VALUE
Modularity
Executable Requirements, Keywords, Classes, & Actions
Test Data, Test Data, Test Data

Tip #3: Collaborate ... Just do it!

There is no wall between developers and testers, there are only politics.

Reject politics and collaborate to accomplish the mission

Testing is not a union that prohibits others from testing or testers from doing other things.
Enable the right people, with the right skills to add the most testing value at the right time in the most efficient/effective manner

Tip #2: Automate things that are better suited for computers to do than humans -- only if they have value

Application deployment is a great candidate for automation – whether or not any automated testing is included.

Tip #1: Do not automate what requires human judgement.

Just don’t.
‘nuff said.

Web Load Testing for Dummies: Book Announcement

2011-10-05T18:28:00.000-04:00

"More so now than ever before, your company’s website and web applications are critical to the success of your business initiatives. Think of all the business generated or sustained via the World Wide Web today compared to any other time in history — in today’s digital culture, a business with any sort of crucial web presence needs to make sure that its website is working hard for the business and not against it. That’s what web load testing is all about.

"Key to success on the web is customer experience, which means that web application performance is a priority. Not convinced? Spend a few moments thinking about the impact to your business (in other words, think about how angry the CEO and/or investors would be) if:
✓ Your new application launch is delayed due to performance problems
✓ Your site breaks under the load of your successful marketing promotion
✓ High-traffic volume causes such poor web performance on your busiest online shopping day that abandonment skyrockets and conversions plummet
✓ Your new infrastructure is configured improperly, grinding the website to a crawl

"Managers and executives of organizations that derive significant portions of their revenue from web applications realize that they need to focus more on protecting revenue, reducing risk, and ensuring that customers have great experiences. They see how web applications that perform well on release day and throughout their production lives strengthen the company’s brand and reputation, creating customer loyalty. In other words, web load testing is a critical component to any risk management plan for web applications."

Get the eBook version free here.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Is Junosphere the world's first cloud testing environment? Not really - IT in Context

2011-10-04T12:06:00.000-04:00

Is Junosphere the world's first cloud testing environment? Not really - IT in Context

I don't get too irked by companies coining new phrases to make subtle marketing distinctions in services, but when they do it so they can make first/best claims flips my bozo bit. Seriously, if your service is so bland or weak that you need to invent a new term so you can claim that it's the "best the best thing called blah" without being called out for fraud, maybe you should just improve your service.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

An overview of Performance Testing for Agile/Lean teams

2011-10-03T14:02:00.000-04:00

I'm going to be giving a short webinar on Oct 20 titled "An overview of Performance Testing for Agile/Lean teams" as part of a really cool recurring online mini-conference/webinar series call "Bathtub Conferences" Check out the website for more information.

http://bit.ly/nWIvzk

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Stop Cheating and Start Running Realistic Tests

2011-10-03T11:35:00.000-04:00

I did a webinar with SOASTA on 9/29/2011, in case you missed it, I've copied the description and links from SOASTA's Info Center so you can have a look. If the twitter-verse is to be believed, it didn't suck. :)

--

Stop Cheating and Start Running Realistic Tests

Constrained by inflexible test hardware, poor tool scalability, exorbitant pricing models, and lack of real time performance information, performance testers have been forced to cheat for too long! Cloud Testing opens up elastic, full-scale load generation from global locations at affordable cost, rapid and accurate test building, and real time views of internal and external performance metrics.

Stop removing “think times” to work around technical or license issues
Build tests using real business workflow, not just a flood of page hits
Run tests that preserve session states and accurate timings, end-to-end
Inspect every component as tests run, not just from the outside-in

Watch the Webinar | Download the Webinar

Agile backlash series...

2011-09-30T11:17:00.001-04:00

From SearchSoftwareQuality.com:

Agile backlash series: Exploring Agile development problems and solutions

I think Jan Stafford did a great job on this series. I don't agree with every opinion from everyone interviewed, but I wouldn't expect to. I think it's fair, honest, insightful, and (best of all) focuses on experiences, challenges, and ideas about overcoming challenges instead of theory, marketing fluff, and excessive exaggeration. :)

Of course, I'm always happy when someone is willing to publish quotes of mine like the following excerpts from Why Agile should not marginalize software testers:

"SSQ: You come in frequently to integrate testing into Agile development. What kind ofproblems do you see organizations having when integrating testing?

Scott Barber: The first thing that I hear about is, ‘What do we need testers for if we’redoing Agile? Isn’t everyone in Agile a generalist?’

Well, the answer is that if you are really, really good at Agile and all your developers arereally good and don’t put any bugs in the code, and your clients are right there on site and doinguser acceptance testing pretty much in line, what’s the job of the tester? Yes, maybe you don’tneed as many testers in that case; but that’s not often the case.

SSQ: Why not? Why is the idea of development “generalists” flawed?

Barber: Testers have a certain mindset, a drive to help expose things that other peopledon’t find. That’s what they do. Developers are creators. Testers are explorers. Their job, theirwhole mindset, is to find the stuff that others don’t.
I’ve always told developers that I want them to deliver something that does what they think itought to do. Sure, I want it to be pretty stable; but I don’t want developers thinking about everypossible thing that can go wrong...

So there’s a value in having the tester’s perspective; that of a person who has dedicated anentire career to figuring out what is it that the user is going to do that no one expects them todo, and what’s it going to do to my system. There’s also value in that testing involves businessrisk mitigation....

...in Agile you might need fewer testers; maybe you’re replacing some of yourtesting with your real users, your acceptance test people. That’s not eliminating test. It ischanging personnel.

SSQ: How much emphasis can be put on acceptance testing of applications?

Barber: ...one of my hesitations ofover-trusting acceptance test-driven stuff. I’m not against acceptance testing, I’m just alittle hesitant, because until you actually put it in people’s work environment, they don’t realizethat they’re using it differently. I can’t say that’s a problem with the process. It’s just a humanthing.

SSQ: What is a key way the tester’s role changes in Agile?

Barber: Testers say, “How am I supposed to protect the user when I’m just sitting therecollaborating, and we’re doing stuff together? Aren’t I too close?’ Well, yes, but that’s notyour whole job anymore. In Agile, it’s no longer the tester’s job to say, “This is no good. Itshouldn’t ship.’...

In Agile, the tester’s job to make sure that everything that gets checked in is shippable...

Until testers are willing to accept that change in role, they’re going to resist the wholething. The more resistant testers are, the less developers want them on the team. In Agile,successful testers are those who say, ‘It’s my job now to help the developers achieve their visionand keep that vision in sync with the end user’s vision.’

...[testers]who have enough technical chops to be able to do a little simple unit testing, execute the unittest or read through some code fare well in Agile. But it’s the attitude more than the technicalskills that really help."

Seriously though, read the whole series, it's worth it.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Making Every Test Count

2011-09-29T11:52:00.000-04:00

This is from a while back, but I wouldn't call it dated. It's a webinar, it runs for 48 min. I like it, for whatever that's worth. ;)

Abstract:

Do you ever find yourself wondering what the point is to executing this test... again!?! Have you ever felt like the purpose of a test is to ensure there is a check mark in a particular check box? Are you ever asked to get *more* information in even less time with even fewer resources than the lst test project you worked on?

In this presentation, Scott Barber will introduce you to a variety of tips and techniques you can apply to virtually any testing you do as you strive to make ever test you execute add value to the project.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Candidate Statement for CMG Director

2011-09-20T22:36:00.000-04:00

I've been nominated as a director candidate for the CMG. My candidate statement is posted below because my views related to CMG mirror my views for application performance in organizations and the industry as a whole and I believe that is (or, at least, I hope it is) interesting to anyone involved or concerned with challenges related to application performance now and in the future.

If you are a CMG member, I encourage you to review all of the candidate statements and to vote your conscience here.
Remember, if you don't vote, you have no right to complain. ;)

Statement of Willingness to Serve:
I am willing and would consider it an honor to serve as a director for CMG if elected.

Professional Work Experience:
In my nearly 20 years of experience working in software and technology, I have performed the duties associated with virtually all of the commonly thought of roles; from analyst to project management, configuration management to IT support, and developer to CIO. These many experiences coalesced shortly after Y2K into a career focused on helping organizations improve software system performance to enhance user experience and enable smooth growth while avoiding speed, stability, and scalability catastrophes in a fiscally responsible manner.

Today, I am the President and Chief Technologist of PerfTestPlus, Inc. PerfTestPlus is a small company that I founded in 2005 focused on assisting organizations to implement or enhance an entire life-cycle approach to managing the performance of their systems cost efficiently through training, mentoring, and consulting at the project, management, and executive levels.

Prior to founding PerfTestPlus, I was employed in a variety IT specialist roles by several custom software development companies, a government contracting agency, and a hardware start-up, all after serving as a Captain in the U.S. Army. I hold a M.S. in Information Technology and a B.S. in Civil Engineering.

Other Professional Experience:
Over the past 10 years, I have become recognized among the world's most prominent thought-leaders in the area of software system performance testing and increasingly in how businesses can bring high quality technology products to market while maximizing the overall business value of those products. Put simply, since starting my career in technology, I've been focused on figuring out how businesses can deliver user-satisfying technology solutions in a manner that is maintainable, low risk, and profitable.

One of the ways I've tried to accomplish this is by sharing what I've learned with others through writing and speaking. I have composed over 100 publications including 4 books; Web Load Testing for Dummies, Compuware 2011 (Author), Performance Testing Guidance for Web Applications, Microsoft Press, 2007 (Co-Author), Beautiful Testing, O'Reilly, 2010 (Contributing Author) and How to Reduce the Cost of Testing, Auerbach Publications, 2011 (Contributing Author). I've also delivered seemingly countless talks to groups and corporations, small and large, in person and via web, including keynote addresses for major conferences on 4 continents.

Another way I've worked toward this goal is by building and supporting communities that focus on collaborative learning and ultimately consolidation and promotion of good ideas and practices across the industry. Aside from participating in online and local communities, I co-founded the Workshop on Performance and Reliability and served for 4 years as Vice President and Executive Director for the Association for Software Testing, a non-profit organization dedicated to advancing the understanding and practice of software testing.

Candidate statement:
My major contributions to the computer measurement and management field all center around improving communication and collaboration across teams and specializations throughout the technology lifecycle, particularly as it relates to the performance, scalability, and reliability of technology solutions. It is through the extension and enhancement of this idea, combined with my experience as a business owner, non-profit executive director, conference coordinator, and community builder/manager that I seek to serve the CMG membership.

Historically speaking, computer measurement and management has been primarily the responsibility of IT and Support Operations groups. In today's world of Agile, the Cloud and virtualized environments, having IT and Support Operations start measurement and management activities as products enter final preparations for production release is too late. To increase the likelihood of success, businesses need be doing appropriate measurement and management activities, unbroken, throughout the lifecycle. Taking the lead in helping businesses make this organizational shift is where I see CMG's largest opportunity to add value to the industry over the next several years. I believe CMG can accomplish this by providing thought leadership and expanding its influence to new audiences.

CMG has a history of supporting and promoting leading thoughts and thinkers, and this should certainly continue. I believe CMG would benefit from becoming recognized as a thought leading organization in its own right. Making this transition need not be difficult. All it takes is for CMG, as an organization, to identify current challenges, provide education about the value of resolving those challenges, and share some principles and practices that can be employed to help resolve them. Helping organizations to become thought leaders in their own right is something I have successfully accomplished before, and I would be proud to bring those lessons and experiences to help CMG do the same.

In terms of expand its influence to new audiences, I believe I am uniquely suited to help CMG do exactly that. With me comes virtually unrestricted access to the most natural expansion point for CMG, architects, developers, testers, vendors, and communities focused on the delivery of highly performant, scalable and reliable technology solutions. Today these individuals and groups cluster in underutilized online communities, small corners of larger conferences, and unconnected local groups. In other words, CMG has the opportunity to merge these groups with its already strong and active membership base, creating a well respected and organized home for mutual growth and collaboration. I believe that CMG has all of the tools it needs to make this happen, and in just a few short years, return to enjoying the benefits that come with serving a larger membership, without sacrificing any of the personality and services that have kept CMG strong for all of these years.

A vote for me is, in effect, a vote to leverage the existing strengths, collective knowledge, and brand recognition of CMG to expand CMG's reach to all of the groups and individuals that participate, or should be participating in, computer measurement and management activities throughout the technology lifecycle while enhancing it's reputation of identifying and supporting thought leaders to additionally become a thought leader in it's own right.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Thoughts on Agile & Agile Testing

2011-09-02T22:33:00.000-04:00

This past weekend, I finally made time to start reading Agile Testing: A Practical Guide For Testers And Agile Teams, Lisa Crispin & Janet Gregory, Addison-Wesley (2009). I made it through the first two chapters before life called me away. After I put the book down and starting going about accomplishing a mundane series of errands, I realized that I was feeling disappointed and that the disappointment had started growing just a few pages into the book. Not because of what the book had to say, what it said was pretty good – not exactly how I would have expressed a few things, but thus is the plight of a writer reading what someone else has written on a topic they also care and write about. What was disappointing me was the fact that the stuff in those chapters needed to be said at all.

You see, as Lisa and Janet were describing what Agile Testing and Testing on Agile Teams was all about, and explaining how it is “different” than “traditional testing”, my first thought was:

Boy was I ever fortunate to start out my career in software in the company I did. I don’t think I’d have made it 3 months in one of these ‘traditional’ environments.

My second thought was:

What earned this stuff ‘traditional’ as an identifier anyway? That’s certainly not how I learned to think about testing, but I guess there’s nothing to be done about the fact that before my time, some person or people, decided that software testing was fundamentally different from anything else I’ve ever encountered that was called testing.

My third thought was:

No wonder I could never figure out what the big deal about testing and Agile was… it’s because this thing, apparently called Agile Testing, is exactly what I thought software testing was before I ever worked on a software project.

Pondering those thoughts while running errands, I realized what was making me feel disappointed.

I found myself disappointed with the entire field of testing -- every person, organization, and software or business related field that enabled or allowed software testing to get to such a state that those two chapters should be classified as a “must read” for everyone who has anything to do with testing software, directly or indirectly, before being permitted onto their first software project. At least coming from the background I did, it seems entirely baffling that these two chapters are describing some kind of nirvana that most testers and teams either aspire to or fear instead of these two chapters representing the givens, the lowest acceptable bar to entry, ya’know, all the stuff that while you’re reading, you keep thinking “Duh. Stop wasting my time telling me all the stuff everyone already knows and get to the good stuff!”

I found myself disappointed that after all the hours and years that so many people have dedicated to making software testing a respected and reputable career, the vast majority of testers and organizations, for reasons that aren’t their fault, at least not entirely, haven’t even made it *up* to what I’d consider “ground-zero”.

Honestly, how messed up must software development have been for a group of some of the most respected and influential people in the field to feel the need to get together and collectively remind us that (to paraphrase the Agile Manifesto):

People, collaboratively, solve problems. Processes and tools will never solve a problem without people.
No matter how well we document what we’re trying to do, it’s pointless if we don’t actually do it.
No contract will ever make a customer find value in a product.
“Stuff” happens, and if we don’t deal with it, we might complete our product cheaper or sooner, but it will be the wrong product and no one is going to want it.

And how messed up have we remained that for the subsequent decade the entire movement that claims the same name as these self-evident truths, has been rejected, debated, and wildly misapplied?

Isn’t it about time that we get over our hang-ups and start simply doing the right thing? Have we become so caught up in the things we’ve seen misapplied that we’re willing to “throw out the baby with the bathwater”? Is the entire software development industry so broken that testers feel they need these “traditional” approaches to protect the business, the users, and themselves from evil? Or has the term “Agile” simply become so overloaded because of a decade of people misapplying the principles and misusing the term that we need to start over again with a new term? If that’s the case, why don’t we just say that the process of developing software should be:

Mission focused – meaning that if we’re not delivering working software that serves the purpose for which it was created, what *are* we doing?
Accomplished in manageable units – 7 year, strict waterfall, development projects simply don’t work. You figure out what a manageable unit is *this* time, and *next* time too.
Collaborative across the entire team – testers included
Value to Cost appropriate
Able to react to change
Reliable
Sustainable

Call it MACVARS, agree to avoid all the “I’m gonna rebrand my old stuff as ‘Agile’ to improve its search engine rankings… whether it’s actually ‘Agile’ or not” crap, and get on with doing the right thing.

If that’s not the case, I wish someone could enlighten me as to what the all the debate and rejection of collaborating to deliver products that serve their desired purpose without running up expenses by doing lots of not-so-value-adding stuff is all about, ‘cause I don’t get it.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

STP Online Summit: Achieving Business Value with Test Automation

2011-08-23T22:29:00.000-04:00

Due to the overwhelming success and positive reviews of the last STP Online Summit: Business Value of Performance Testing, we've decided to do it again -- only this time, we're going to explore Achieving Business Value with Test Automation.

Join me (while I continue practicing my radio host skills for my emergency back-up career as a sportscaster) and 7 other presenters that I consider to be elite practitioners, teachers, and thinkers in their test automation areas of specialization for 3 half days online to learn their tips and methods for achieving business value with test automation. If you or your organization are using, or thinking about using, automation to enhance or improve your testing, you're not going to want to miss this online summit. I honestly can't think of anywhere else you can get this concentration of relevant and thematically targeted information at a better price, but you be the judge:

When: Tuesday October 11 10:00AM - Thursday October 13 1:30PM PST

Cost: $195 USD before 9/26/11 $245 USD after 9/26/11

Theme: For more than 15 years organizations have been investing in the promise of better, cheaper, and faster testing through automation. While some companies have achieved demonstrable business value from their forays into test automation, many others have experienced questionable to negative returns on their investments. Join your host, Scott Barber, for this three day online summit, to hear how seven recognized leaders in test automation have achieved real business value by implementing a variety of automation flavors and styles for their employers and clients. Learn how to answer the ROI question by focusing on business value instead of testing tasks, and how to implement automation in ways that deliver that value to the business, not just to the development and/or test team.

Presentations:

Software Test Automation, Where's the Value? – Scott Barber
Achieving Value with Automated Acceptance Tests – Linda Hayes
Developer Level Automation Value from "Velcro" to "Velocity" – Rob Sabourin
The Simple and Powerful Build Verification Tests (BVT) – Alan Page
Functional Automation is the Gateway Drug to Continuous Delivery – Adam Goucher
Test Automation at the API Layer...Why Bother? – BJ Rollison
Experiences with Adding Value Through Test Automation – Mark Fewster
Test Automation Misconceptions and Money Pits – Hans Buwalda
Top Tips for Addressing the Challenge – Scott Barber
Ask the Testing Savants – Speaker Panel

Last time, we sold out so reserve your seat early!

Who Should Attend: The Achieving Business Value with Test Automation summit was designed for software test and quality assurance professionals responsible for planning and implementing a test automation strategy. Specifically, it is designed for those professionals who are driven to make their test automation framework more effective efficient and valuable to their organization. Novices will be introduced to new concepts and practical ideas, while seasoned professionals will learn new approaches to enhance their abilities.

Why Attend?

This is a one-of-a-kind event that was created for software testers, by software testers.
We’ve put together a content-rich program in a comfortable structured environment delivered by seasoned presenters who are forward thinkers and test automation authorities.
Participate in networking opportunities designed to encourage the exchange of peer-to-peer knowledge from other test automation engineers and the session speakers.
Travel is not required. Attend all of the sessions, interact with the presenters and network with your peers right from the comfort of your home or office!
You don't have to stop working to attend so you can start applying what you learn immediately. The audio and video from each session will be recorded. If you miss a session, you will have access to all of the session recordings once the summit has ended.

Learn How the live STP Online Summit Works
Frequently Asked Questions

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Scott Barber Interviewed by Matt Heusser; Podcast

2011-08-04T20:11:00.000-04:00

Two part podcast on the STP site. I say some interesting stuff... or at least I say some stuff that's interesting to me. :)

Twist #52 - With Scott Barber

Twist #53 - The Return of the Barber

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Performance Testing Practice Named During Online Summit

2011-08-01T20:10:00.000-04:00

Last week, I hosted STP's Online Performance Summit, a 3 half-day, 9 session, live, interactive webinar. As far as I know, this was the first multi-presenter, multi-day, live webinar by testers for testers. The feedback from attendees and presenters that I have seen has all been very positive, and personally, I think it went very well. On top of that, I had a whole lot of fun playing "radio talk show host".

The event sold out early at 100 attendees with more folks wanting to attend, but were unable. Since this was an experiment of sorts in terms of format and delivery, we made a commitment to the smallest and least expensive level of service from the webinar technology provider, and by the time we realized we had more interest than "seats", it was simply too late to make the necessary service changes to accommodate more folks. We won't be making that mistake again for our next online summit to be held October 11-13 on the topic of "Achieving Business Value with Test Automation". Keep your eyes on the STP website for more information about that and other future summits.

With all of that context, now to the point of this post. During Eric Proegler's session (Strategies for Performance Testing Integrated Sub-Systems), a conversation emerged in which it became apparent that many performance testers conduct some kind of testing that involves real users interacting with the system under test while a performance/load/stress test was running for the purposes of:

Linking the numbers generated through performance tests to the degree of satisfaction of actual human users.
Identifying items that human users classify as performance issues that do not appear to be issues based on the numbers alone.
Convincing stakeholders that the only metric we can collect that can be conclusively linked to user satisfaction with production performance is the percent of users satisfied with performance during production conditions.

The next thing that became apparent was that everyone who engaged in the conversation called this something different. So we didn't do what one would justifiably expect a bunch of testers to do (i.e. have an ugly argument about who's term came first, is more correct, that continues until no decision is made and all goodwill is lost). Instead, we held a contest to name the practice. We invited the speakers and attendees to submit their ideas, from which we'd select a name of the practice. The stakes were that the submitter of the winning submission would receive a signed copy of Jerry Weinberg's book Perfect Software, and that the speakers and attendees would use and promote the term.

The speakers and attendees submitted nearly 50 ideas. The speakers voted that list down to their top 4, and then the attendees voted for their favorite. In a very close vote, the winning submission from Philip Nguyen was User Experience Under Load (congratulations Philip!).

So, next time you, or someone in your organization proposes putting users on a system that is currently under load, you can say "Let's run a User Experience Under Load test to assess end-user satisfaction."

I strongly encourage you to do that... and follow it up by mentioning that Philip Nguyen of Citrix Systems coined that phrase. I, for one, will, from this moment forward use this name to refer to this practice (which is a practice that I fully support as a good practice in a wide variety of contexts, and believe is widely under utilized) in my writing, speaking and consulting, and will be diligent about attributing the name to Philip.

I hope all of you see the value in what happened here. "Testerland" is so filled with overloaded terms coined for marketing purposes that are not widely understood or agreed upon that using terms will often hinder communication more than help it. In this case, a group of 100 performance testers first agreed on a description of a practice, and then agreed on what to call it -- with no agenda other than naming a good practice to make it easier to talk about.

I hope this inspires other groups of diverse individuals to stop debating which existing term is "better" than each other existing term, and rather identify a practice through description (not definition) and then jointly agree on a term to refer to it that makes sense. Personally, I'm tired of watching a bunch of consultants, trainers, and/or vendors get into heated debates over whose term should be *the* term to force down the throats of everyone in "Testerland" because it supports their business agenda. If that reality bugs you as much as it bugs me, take the first step. Adopt User Experience Under Load as the name of this practice, and tell the story of how Philip Nguyen's proposed name won out in a democratic vote of every day performance testers who just happened to be in the right webinar at the right time.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Google Page Speed Service – The death of the Web Performance Optimization consultant?

2011-07-29T20:09:00.000-04:00

Fred Beringer of SOASTA posed that question on his blog yesterday.

An interesting question, so being a tester, what did I do? Right, I tested it. It took all of one test for me to come to my conclusion...

NOT WITH RESULTS LIKE THIS!!

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

CloudTest Lite - A Game Changer in the Performance Tool Market

2011-07-20T20:07:00.000-04:00

Yesterday, SOASTA announced their new product, CloudTest Lite (Press Release). It's not common that I get excited about a tool product release, but this is different. This product has the potential to change the market for the better.
Scratch that. I'll be shocked if it doesn't change the market for the better.

Why is that, you ask? Consider the following attributes of CloutTest Lite:

It's a fully featured, easy to learn and use, enterprise class, modern, performance testing tool for web & mobile applications
All you need to use it is a reasonably modern machine connected to the internet and a web browser.
- You don't need to buy, install, configure or maintain load generation machines.
- The "license" is tied to your personal credentials, so you can design, create, execute, and analyze your tests from any machine you want without needing to figure out how to point to the license server, or how to get onto the corporate network from your favorite internet cafe.
- You can even do much of the design, test enhancement, and analysis entirely off-line.
You can simulate up to 100 virtual users any time you want. No more scheduling time on the controller days or weeks in advance guessing the app will be ready for your test. No more having to wait until your next scheduled time to re-run your test when you see something 'wonky' in your data.
It's free.
- Yes, I said free.
- As in, you never need to pay a dime. Not today, not when the trial expires, not a year from now to continue your maintenance contract.
- That's right, it is free from now until the sun explodes (or at least until well beyond when anything we're building or planning to build today is long gone and forgotten)

Imagine the implications:

There is no longer a financial excuse to not be testing performance for any website, of any size, throughout the lifecycle.
No more resources wasted installing and configuring load generation labs.
No more time lost waiting for licenses or controllers.
Trainers can teach public performance testing classes, with an enterprise class tool without investing (and charging) a fortune to build and maintain a legally licensed lab.
You can launch all those late night tests from home.

Ok, true, it's "only" 100 virtual users, but be honest, how often do you need more than that? And when you do, it is simple and seamless to buy the additional load you need with the CloudTest On Demand service -- no additional installations, not changes to your tests necessary. Just pay for what you need when you need it.

SOASTA has done what I've been begging vendors to do for a decade. They've eliminated the bar to entry for life-cycle performance testing and committing to providing a positive user experience. They've build a business model where they get paid by the owners of the sites make a lot of money as a result of large volumes of traffic to their site instead of trying to wring non-existent nickles out of the owners of "mom & pop" businesses who want to add a little value to their customers with a user expectation meeting website.

Congratulations SOASTA for implementing a business model that actually demonstrates, not just pays lip service to, social responsibility and software user advocacy without skimping on features, usability, or the ability to pay your employees a fair salary for their hard work. That alone, even if I didn't like the tool (and I *do* like the tool) makes you one of *very* few companies that I am proud to support.

All you other vendors, consider this fair warning. Unless your tool does something special, like handles some unique protocol or accommodates a specialized technology or architecture, as of yesterday, I rate your market share as "legitimately under attack".

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Uruguay surpasses world with professional development program for software testers.

2011-06-07T20:06:00.000-04:00

The Centro de Ensayos de Software (CES), a non-profit software testing laboratory in Uruguay, has recently launched a program that is certain to become the new “gold standard” in professional development for software testers. The program, endorsed by the Universidad de la Republica (Uruguay), the Universidad Castilla La Mancha (Spain), and sanctioned by the Uruguayan IT Chamber (CUTI), is the most comprehensive, affordable, and publicly available training program for software testers on the market. Based on my market research and comprehensive review of the program, I have no reservation in rating it as market leading.

Software Testing, the software development activity responsible for identifying issues with software and providing a wide variety of quality-related information to stakeholders and decision-makers prior to release, is the primary job of many millions world-wide, yet the majority of software testers learn their craft entirely on the job. Yes, there are various “take a class or two, pass an information-based (not a skill-based) test, and receive a certification” programs – some more respectable than others and most far more expensive than the CES program. There is even a new certificate coming to market that involves three, one month, on-line courses where students are taught and assessed by experienced testers and university professors, but none of those rise to the level of the CES’s program.

The CES offers courses leading to three levels of credential, Software Tester, Software Testing Professional, and Software Testing Leader; each tier is comparable in quality and format to a “Professional Enrichment Certificate” from an accredited university, and when viewed collectively the three levels are only marginally short of the criteria published by North American degree accreditation organizations for consideration as an Associate’s Degree. Each tier is both information- and skill-based and requires takes 6 months to complete. The successful student is expected to dedicate a minimum of 400 hours of instruction and project work that is delivered and assessed by university approved instructors and significantly experienced software testers to complete each tier of the program. The three tiers span the career development of a software tester:

Software Tester: The ideal program for entry-level, junior, and career-switching software testers. Successfully completing this credential will give you the knowledge and experience most employers expect from testers with 1-2 years of on-the-job experience – effectively enabling you to start your career needing only to acclimate to the specific expectations of the employer and demonstrate your skills at work before being acknowledged as a mid-level tester.
Software Testing Professional: For testers with several years of testing experience on-the-job who are looking to make the jump from mid- to senior-level, this program is designed to teach high-quality individual contributors how to be effective technical leaders within their testing organization. Effectively giving the tester the new skills they need, in addition to their existing hands-on testing knowledge, to prepare them for the additional responsibilities of a technical testing manager or of a manager of small to mid-sized testing projects.
Software Testing Leader: This final step in the program has been designed for senior-level testers who desire to be successful managers or directors of corporate testing programs. Frequently the most challenging step for career software testers is to transition from being a technical leader to a management role focused on the interface between quality testing and executive-level business value. Successful completion of this stage of the program will give the student the tools they need to make the jump from technical leader to manager – opening the door to further advancement to positions like “Director of Testing Services” or “VP of Software Product Quality”.

I believe that this is exactly the program that testers and employers of testers have needed and desired for many years. Without hesitation, I would move job candidates who have completed any or all of this program to the top of my consideration list. In fact, the only thing that keeps me from promoting this program even more aggressively is that, at least for the time being, it is only available in Spanish. Even so, if you speak Spanish, or have tester employees who do, I strongly encourage you to take a very serious look at this program and draw your own conclusions. I am confident that you will come to the same conclusion I have… that there is no currently available program that is more comprehensive or higher quality at any price, let alone at surprisingly affordable cost of approximately U$1,300/tier.

Visit the website (www.ces.com.uy in Spanish), or the view the brochure (www.perftestplus.com/CES.pdf in English) for more information.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

What being a Context-Driven Tester means to me

2011-04-11T20:04:00.000-04:00

I guess it’s that time again. What time is that, you ask? It’s the time when discussion/debate flares up over Context-Driven. I’m not going to weigh in on the whole discussion of pros/cons, value/distraction, etc. I am a consultant. I am Context-Driven (and not just as a tester, it's simply the way I have operated since long before I was a tester and long before I became aware someone had coined a term and composed a set of principles around how I already operated). The license plate on my car says “CONTEXT”. It works for me. But my point isn’t to convince you that it’s right for you. My point is to address a comment that I frequently hear that *feels* very sad to me.

“Where I work, I don’t have the freedom or authority to implement all this Context-Driven stuff, so I guess I don’t get to be part of the club.”

I find this sad, because I don’t agree. It is my opinion that “Where I work, I don’t have the freedom or authority…” *is* a "driving context", making smart decisions about what you are empowered to choose, and appropriately trying to inform/educate those who are "driving your context" that there are other options qualifies as being Context-Driven... at least to me.

What follows is something I drafted for an org that had recently decided that it wanted to adopt the principles of being Context-Driven, but didn’t want to inadvertently offend members whose context was largely dictated by decisions outside of their sphere of influence. Due to a wide variety of unrelated circumstances, what I wrote never got presented to the org & got lost and forgotten on my hard drive. I recently found it and wanted to share it with everyone because I think it’s valuable.

Please note, the document below is entirely the opinion of Scott Barber. It has not been reviewed, approved, sanctioned, dismissed, or debated by the folks who composed and maintain the principles of the Context-Driven School of Software Testing. They may agree, they may not, but that doesn’t change the fact that I believe these things. Also note that I’m 100% certain that this is not perfectly composed, doesn’t cover every possible situation, and if anyone wanted to adopt or “officialize” it, it deserves a healthy copy-edit and peer review.

Ok, with all the “context” and disclaimers out of the way, here is what it means to *me* to be Context-Driven.

All Roles:

I understand that there are many different kinds of value that can be obtained from testing software.
I understand that there are many variables that influence what test tool(s), technique(s), method(s), model(s), and/or approach(es) (subsequently "method(s)") are likely to provide the most value in a given situation.
I understand that the most valuable method(s) for my current situation may not continue to be the most valuable when any of the many variables related to that situation changes.
I believe that when a situation changes, it is valuable to take pause and consider whether the method(s) I am currently using are still the best available choice, and adapt appropriately if not.
I respect the diversity of methods available to software testers and believe there is inherent value in being aware of as many of them as I can.
I do not discount or belittle a method simply because I have not yet experienced added value from using/applying it.
I am not afraid to express that I feel that a particular method is unlikely to add value in my current situation, or did not add value in a particular situation in the past.
I am skeptical of methods that claim to be valuable in all situations, claim to be a “best practice”, or promise to add value regardless of situation.
I understand that situations change during the course of a project, and from project to project and I strive to adapt to continue providing maximum possible value in the face of changing situations.
When the parameters I am given & my scope of control prohibit me from adding the value I believe I could/should be adding, I respectfully make my superior(s) aware of that fact and accept his/her guidance/decision.

Testers:

I strive to understand the value my testing provides to my stakeholders, including:

my peers
my managers
my developers
my project
my executives
my organization/company
my regulators
the users/clients

I strive to maximize the value of my testing to stakeholders within the parameters I am given & my scope of control.
I strive to learn and understand many methods so that I am better able to adapt in the face of change.

Test Manager/Directory of Testing:

I strive to understand the value my team’s testing provides to my stakeholders.
I strive to ensure that every member of my team understands the value their testing provides to our stakeholders.
I strive to maximize the value of my team’s testing to stakeholders within the parameters I am given & my scope of control.
I strive to learn and understand many testing methods so that I am better able to help my team adapt in the face of change.
I strive to train my team in many testing methods so that they are better able to adapt in the face of change.

Executives/Organizations:

I strive to ensure my test directors/managers understand the value their team’s testing provides to stakeholders.
I strive to encourage/enable test directors/managers to maximize the value of their team’s testing to stakeholders by thoughtfully and carefully setting their parameters & scope of control.
I strive to adjust parameters & the scope of my control appropriately when my test directors/managers respectfully inform me that they are prohibiting their team(s) from making their testing more valuable.
I strive encourage/enable my test directors/managers to train their teams in many testing methods so that they are better able to adapt in the face of change.

Contractors/Consultants/Trainers:

I strive to learn and understand many methods and the kinds of situations in which they are likely/unlikely to add value to stakeholders so that I am better able to adapt in the face of change, advise my clients, and/or share that knowledge with my students.
I strive to help my clients/students learn to adapt to provide maximum possible value in the face of changing situations within the parameters they are given and their scope of control.
I understand that my skills/course material is not a good fit for every possible situation, I strive to advise my clients/students of this fact and help them make informed decisions about whether my services/classes are right for them at this time. And when they are not, I strive to help my clients/students find contractors/consultants/trainers that are better suited to help them meet their current needs.
I advise my clients/students to be skeptical of methods that claim to be valuable in all situations, claim to be a “best practice”, or promise to add value regardless of situation.

Why {org} is “Context-Driven”

Acknowledging that situations change during the course of a project, from project to project, from organization to organization, and from industry to industry; and
Embracing the fact that testers need to be able to adapt to changing situations if they are to provide the most possible value to their stakeholders; and
Respecting the value of diverse tools, techniques, methods, models, and approaches available for software testing; and
Understanding that learning and understanding many of the available tools, techniques, methods, models, and approaches can help testers and managers of testers to adapt to add the most possible value to their stakeholders in the face of changing situations; and
Recognizing that what is “best” for a tester today may not be “best” for that same tester tomorrow, and that what is “best” for one tester today may not be “best” for another tester, even when assigned the same task on the same team on the same day

{org} therefore believes that the most value that it can ethically provide to the broadest community of testers and their managers is to focus on making our services consistent with the 7 principles of the Context-Driven school of Software Testing, which are:

The value of any practice depends on its context.
There are good practices in context, but there are no best practices.
People, working together, are the most important part of any project's context.
Projects unfold over time in ways that are often not predictable.
The product is a solution. If the problem isn't solved, the product doesn't work.
Good software testing is a challenging intellectual process.
Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

Testing the Limits... Interviews by uTest

2010-05-12T20:02:00.000-04:00

uTest is doing a series of interviews with interesting/well-known testers. I was flattered to be asked to be interviewed recently and found that I enjoyed answering the questions... and didn't hate my answers when I read them as posted (any of you who have ever done that sort of thing know that is a fairly significant statement!)

My interview is "Testing the Limits with Scott Barber": Part I, Part II, Part III

Other folks interviewed include James Bach, Jon Bach, Michael Bolton, Matt Heusser, and Rosie Sherry. If you've got the time, I think they are worth the read.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Testing vs. Checking ... my 2 cents.

2009-09-23T20:01:00.000-04:00

I was pleased to see Michael Bolton's series on Testing vs. Checking. If you haven't been following, what I consider to be the central thread of the topic (and the unfortunately inevitable fallout that seems to happen in "testerland" almost any time someone says something that makes sense).
From Michael:

From James Bach:

http://www.satisfice.com/blog/archives/358

From Scott Barber:

http://www.perftestplus.com/resources/014PeakPerf.pdf Original, ST&P Magazine, 2005
http://www.logigear.com/newsletter/investigation_vs_validation.asp Reprinted by Logigear, 2007

See, I hopped up on the "test vs. check" soapbox back in '05 when I got sick of being asked for performance requirements and test cases that start with "The system shall [x]" and "Verify that the system does [x]" (I provide much more detail about my distaste for Performance Test Cases in my Chapter of Beautiful Testing, due out in October). I used the terms "investigate vs. validate" in the hope of getting people to focus on what I was saying instead of arguing about whether or not validating (or checking) is "testing". It didn't really work. Instead of discussing/assessing/debating/enhancing the value and usefulness of the distinction, I found myself endlessly fielding question about whether I really *meant* to say "verifying" not "validating" because "At my company that's what we call it."

So I guess I'm jumping on the bandwagon, but it never hurts to have more supporting references.

As an aside, I cannot express how tired I am of hearing "At my company we call that...", "Well, that wouldn't work at my company", and "We don't do that at my company" used as a dismissal in response to almost any idea that requires thought, consideration, or (gasp) change. Ya'know what folks? The entire world is not like "your company." And if it were, then what, exactly, do you expect to find in training courses, magazines, blogs and conferences that will be of value to you? Seriously, if you think you know it all, stop being so greedy and start sharing it so the rest of us can catch up. Otherwise, what harm would it do you to actually take a moment to understand and consider what others have to say instead of being so rudely dismissive? Maybe what others are saying is appropriate for "your company" and maybe it's not, but how can you be sure when you're spending your energy dismissing the possibility rather than considering the possible applications, implications and derivations that *just might* be valuable or useful to "your company," or maybe even "your next company".

Honestly, to me it feels like testing, on the whole, is simply afraid to change. I don't get it. The rest of the software industry is changing constantly, but the overall state of testing seems to be bolted firmly in place. This isn't true for every individual, nor every company, so please, if this description doesn't fit you, don't take it personally. But if you don't believe this is the general state of affairs, go to testing conferences -- once every three years or so will do -- and tell me if you see any significant advancement (other than an amazing advancement in folks ability to repackage old ideas as new ones).
However, if you are contributing to testing's fear of change,to the fear of having to admit that someone else might have come up with something potentially or situationally valuable, or to the inertia that appears to be the root of why many testers are unwilling to consider ideas different from their own, then do the industry a favor and stay out of training classes, stop reading blogs, and above all please stop spending your time deliberately derailing discourse among those who actually want to collaborate thoughtfully in the hope of advancing the state of software testing beyond the '70's. When you're ready, come back with an open mind, your thinking cap on, your experiences at your side and you'll be welcomed with open arms -- at least by me.

This is just another case of the same 'ole debates resurfacing every few years. Maybe someday we'll advance beyond outdated ignorance and inertia, but in case that doesn't happen soon, I'm marking my calendar to republish my Investigate vs. Validate article in 2013 (with links to these discussions, and enhancements to encompass what I learn between now and then.) If the value, applications, and implications of thinking about the distinction doesn't catch on this time around, maybe it will then.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Thorkil Sonne: Recruit Autistics

2009-09-21T19:58:00.000-04:00

Wired.com ran their smart list today. If you aren't familiar with it or don't care, at least check out the great press fellow software tester, entrepreneur, and social innovator Thorkil Sonne is getting for Specialisterne here:

http://www.wired.com/techbiz/people/magazine/17-10/ff_smartlist_sonne

While you're at it, why not digg it!

I know there is a lot that we testers disagree about, but if there is one thing we should be able to agree upon, it's that Thorkil, Specialisterne, and the very special people they serve deserve our support and best wishes. I can only hope that this is the spark that get's this (and other such, responsible programs) moving globally. While that would certainly make me happy for Thorkil, the real winners when this takes off will be those who would finally find themselves filling jobs well suited to their skills, those who are reluctantly (and often poorly) doing those jobs now, and their employers who can reassign those reluctant folks to something *they* are better suited for and will complain about less (we all hope) .

Congratulations Thorkil & Specialisterne!!!

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

A misleading benchmark...

2009-01-03T19:52:00.000-05:00

No further commentary needed.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Thoughts on Performance Testing w/o "Tools"

2008-12-20T19:49:00.000-05:00

I was recently asked the following question via the "Ask The Expert" feature of SearchSoftwareQuality.com.

How can we conduct performance testing, stress testing, and load testing of a Web application manually without using any tools?

My commentary is reproduced below -- you'll have to click through to see my actual recommendations.

First, I want to express my sympathy to anyone who finds himself in a position of being asked to create multi-user simulations (i.e. the load part of performance/load/stress testing), requesting a load-generation tool, and being denied. In my experience, the excuse of not being able to afford a load-generation tool is almost always just that -- an excuse.

Only once in my career have I found the only tool capable of generating a production-like load against the application I was testing to be prohibitively expensive. (In that case, the tool cost more to purchase than the application was anticipated to earn in a year.) In every other case, either the risk justified the cost or an adequate, inexpensive, or free tool has been available.

The only really good reason to be in this position is if all available tools were considered and none of them supported the application under test (even with customizations and extensions) and you don't have access to the skills (internally or externally) to build a tool of your own -– and that seems unlikely to me.

The simple truth is that if a company is building an application that is realistically expected to have enough users to justify the expense of performance testing, even if that expense is just the time of an employee, that company ought to be projecting enough revenue from the application, or lose enough credibility by having a poorly performing application, to justify either the cost of a tool or the risk (in some company's eyes) of using a free or open source tool.

Now, after saying all of that, I must admit I have found that the vast majority of value that is gained by quality performance testing comes outside of the load-generation tool. Some of my favorite techniques (assuming you are testing websites):

Read the rest of the article here.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Latest Column -- The controversy surrounding the schools of software testing

2008-12-05T19:48:00.000-05:00

My latest column...

Periodically, discussions break out in various software testing communities around the Web regarding the schools of software testing.

As I write this, there are discussions going in SQAForums, on the Software-Testing Yahoo! group, and various blogs that (at least up to the time I started writing this piece) reside on or are fed to Testing Reflections. In principle, I'm always pleased when these discussions break out. The point of identifying the schools in the first place was to increase the overall awareness of the diversity in ideologies, practices, and values (i.e. schools of thought) in our field and to stimulate discussion about the situational pros and cons of each. That said, the discussions that actually take place tend to drift off in one or more directions that end up being disappointing, unnecessarily confrontational, and generally not useful.

After witnessing this pattern, participating in these recent discussions, and listening to comments from those who followed the discussions for several years, I've identified several areas in which these discussions go awry. Below, I call those out and share my thoughts about each.

Read the rest of the column.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Latest Column -- Testing training: Disturbing behaviors of students

2008-11-07T19:46:00.000-05:00

My latest column...

Drive-by training. Never heard of it? It is exactly what it sounds like. You drive to a training facility (or an instructor drives to you), for a day or three the instructor delivers the pre-packaged training class, then everyone drives back home. It's not the best training model ever invented. There is generally no student assessment, and the only instructor/course provider accountability is reputation. Even so, many good ideas can be shared and lots of students come away feeling that it was well worth "the drive."

As it turns out, I've been delivering a lot of drive-by training to software testers this fall. That in itself isn't particularly noteworthy -- end-of-the-budget year is a popular time for drive-by training -- but something that is noteworthy is that I have noticed a rise in some disturbing behaviors among the individuals and organizations that select and attend drive-by training.
At first, I thought it was just me. But after an informal poll (and some lively discussions) with my employees and trainer friends in the testing realm, I became increasingly convinced that the behaviors I'm noticing are not exclusive to me and that I'm not the only one who thinks they are on the rise.

Read the rest of the column.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Latest Column -- Software Testers are not helpless

2008-10-05T19:39:00.000-04:00

My latest column...

During a coffee break at a class the other week, I overheard the following comment from one student to another:

Tester: "This stinks! All of my automated test scripts are broken and I can't seem to get the tool to work now that the developers have enabled Secure Sockets Layer. I'm going to have to work through the weekend."

I know that it's generally considered rude to eavesdrop, and ruder still to comment on a conversation you weren't invited to, but I figured that since I was teaching the class I'd be forgiven. Besides, I simply couldn't help myself.

Read the rest of the column.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Latest Column -- Avoid "Center of the Universe Syndrome"

2008-08-19T19:34:00.000-04:00

My latest column cautioning testers not to think they are the center of the development team's universe http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1325828,00.html.
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

5 Questions with Scott Barber by a Braidy Tester

2008-08-06T19:27:00.000-04:00

I recently had the honor of being interviewed by Michael Hunter, a Braidy Tester, for Dr. Dobbs Portal. Check it out: 5 Questions with Scott Barber.
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Latest Column -- Inspired by taking AST's Bug Advocacy Class

2008-07-04T19:16:00.000-04:00

I recently completed (successfully, I might add) the second of the Association for Software Testing's all online, free to members Black Box Software Testing course. Each of these courses is four weeks in length. I've been involved with this program since years before it became a program, and I am an instructor for the first course in the series, called Foundations. For this course, called Bug Advocacy, I was a student.

Bug Advocacy focuses on the skills and concepts needed to compose high-quality, easily understood, appropriately compelling and well organized defect reports. I know, it sounds pretty boring to me too, but it was anything but boring. These classes are designed so that you watch recorded lectures (in this class the lecturer is Cem Kaner), answer some quiz questions (to make sure you watched the lectures), participate in class discussions, do both individual and group projects (in this class the project centered around evaluating and enhancing unconfirmed OpenOffice bug reports), peer reviewing one another's assignments, and taking a far-from-trivial closed-book essay exam. All in all, I spent about 40 hours participating in the class over the four week period.

This approach isn't just about writing a good bug report, it's about making sure you do the right testing after you find a bug.
There was one idea in particular from the class that I found absolutely brilliant and wanted to share with you. Below is actually a very lightly edited version of my answer to one of the exam questions asking us to describe a six-factor approach to bug reporting that Cem remembers using the mnemonic "RIMGEA." If you are a regular reader of mine, you know that I have a fondness for mnemonic devices, but that's not what I thought was so great about the approach. What I think is brilliant is that this approach isn't just about writing a good bug report, it's also about making sure you do the right testing after you find a bug to enable you to write a good bug report. Take a look -- you'll see what I mean.

Click here to read the rest of the column

Click here for more information about AST's free-for-members, online training

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Testing Lessons From Civil Engineering

2008-06-28T19:15:00.000-04:00

Below is the paper I submitted as a prologue to an experience report, discussion, and (hopefully) additional research that I'm presenting for the first time during CAST08:

Engineers don’t look at the world the same way that testers do. Engineers look at the world with an eye to solving problems. Testers look at the world with an eye toward finding problems to solve. This seems logical. What is less logical is the fact that engineers, and I’m talking about the kind of engineers that deal with physical objects, seem to be much more sophisticated in their testing than testers. In fact, most of what I know about testing, I learned as a civil engineering student. We didn’t call most of it testing. We didn’t even identify it as anything other than “You really want to get this right.”Maybe Civil Engineers test better than software testers because of the motivations to “get it right”. Consider what happens when a piece of Civil Engineering, like a bridge fails:

Huge amounts of money is lost.
Engineers lose their jobs and their licenses to practice.
Lots of TV news coverage.
Innocent, unsuspecting people die.

Consider what happens when a piece of software, like a program to assist with submitting personal taxes, fails:

Some executives probably don’t get bonuses.
Some smart people put in some overtime.
The Government extends the tax deadline.
Even more people use the software the next year, figuring the problem has been resolved.

I guess it’s no wonder Civil Engineers go the extra mile to “get it right”. Maybe it’s not even appropriate (let alone cost effective) for software teams to test with the same kind of rigor as Civil Engineers. But wouldn’t it at least make sense for us to take a look at how they approach this testing? Might there not be lessons there that we can apply without breaking the budget or extending the project duration? I believe so. Some of the principles and techniques, as I recall them from engineering school, that I’ve applied or adapted to software testing, which I believe have had a positive impact on my testing include:

Prototyping
Safety Factors
Failure Modes
Risk Assessment
Independent and Collective Testing of Materials and Designs
Experimental Design and Execution
Thought Experiments
Realism in Environmental and Usage Modeling
Validation of Models
Sub-section Isolation

I make no claim to being an expert in any of these areas as they relate to Civil Engineering. I’m 15 years removed from these topics in that context. In fact, the titles I’ve given these ideas predominantly come from my head. I’m certain that, from a Civil Engineering perspective, what I recall about these items is at best incomplete, and possibly, just plain wrong. Be that as it may, the influence that these items have had on my development and success as a tester are profound. I believe that makes this concept worth exploring.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Identity crisis or delusions of grandeur?

2008-05-05T19:30:00.000-04:00

In this month's installment of "Peak Performance" I discuss the frequently erroneous and often grandiose titles software testers have on their business cards or in their e-mail SIGs. Identity crisis or delusions of grandeur?
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Performance Testing Guidance for Web Applications book

2007-12-17T19:09:00.000-05:00

Some time back, I blogged about a book I’d been significantly contributing to being available as a free .pdf download. (see the entry here)

Well, the book quietly appeared in “dead tree format” (as Stuart Moncrieff put it in his blog post about the book) a couple of weeks ago and I’ve been getting light heartedly scolded by some of my friends and readers for not making a big announcement, so here’s my “big announcement.”

Performance Testing Guidance for Web Applications

by: J.D. Meier, Scott Barber, Carlos Farre, Prashant Bansode, and Dennis Rea is now available on Amazon.

Reviewed by: Alberto Savoia, Ben Simo, Cem Kaner, Chris Loosley, Corey Goldberg, Dawn Haynes, Derek Mead, Karen N. Johnson, Mike Bonar, Pradeep Soundararajan, Richard Leeke, Roland Stens, Ross Collard, Steven Woody, Alan Ridlehoover, Clint Huffman, Edmund Wong, Ken Perilman, Larry Brader, Mark Tomlinson, Paul Williams, Pete Coupland, and Rico Mariani.

The best part is that you can buy the book on Amazon, download the PDF, browse the HTML, or do any combination of the above.

Even though this as a Microsoft patterns&practices book, it is a tool, technology, & process agnostic book (which, along with the content of the book being available for free, and me not losing IP rights to any of the content I contributed, were the commitments MS made to me before I agreed to contribute). What that means is that the book should apply equally well to a LoadRunner/Eclipse/Agile project it applies to a VSTS/.NET/CMMI project.

What that also means is that if you’re looking for a “cookbook” or a “how-to” manual, this isn’t it. This is a book that focuses on core principles that deliberate and successful performance test projects seem to have in common and topics the authors and reviews agreed (or at least mostly agreed) were common enough to be worthy of your consideration. Not everything in this book will apply to everyone – in fact, I’d be shocked and concerned to learn that everything in this book applied to ANY one person or project.

Even though I wrote the initial draft of all but 3 of the 18 chapters in the book and was the last set of eyes on the manuscript to be focused almost entirely on the content, there are parts of the book that I love and parts that I’m already ready to re-write. That said, one of my goals for this book is to give the authors, trainers, bloggers, researchers, and doers a common point of reference to debate over and grow from (even if ALL of the growth ends up being counter to the guidance in the book!) Many of the authors and reviewers of this book have enough individual knowledge and experience to write one or more performance testing books on their own (as do many folks who aren’t authors or reviewers of this book), but none of the ones I talked to wanted to write the first tool, technology & process neutral, project life-cycle book for practitioners about testing the performance of software systems. We all knew it was an impossible undertaking as there is simply no way to cram an encyclopedia’s worth of information into one easy to digest book. And none of us wanted to be the one to fail at it.

I don’t think this book fails at it – though it comes far from succeeding. The book represents a consensus from a very wide range of individuals, schools of thought, experiences, and backgrounds. It is filled with the stuff we seem to be able to agree on. Some people call those agreements “Best Practices”. I call them “reasonable places to start”, which is what I hope this book becomes; a reasonable place for all of the performance testers with knowledge and experience to share to use as a starting point to tell their own stories and share their own experiences.

To tell the truth, that is exactly what I’ve done with my new Performance Testing Courses, that I’ll write more about next time. I built my experiences, personality, preferences, and probably some of my biases, into a collection of workshops that (IMHO) extends and compliments the book. I hope others do the same… whether they do so through blogs, articles, training, or through applying concepts from the book with a healthy dose of individuality, doesn’t much matter to me. What matters to me is that this book serves to remove the barrier of not wanting to write “the book that cannot be written” so that we can get more quality information into the world for performance testers to read, reference, use, debate, extend and explore. If that happens, I’ll count the book as a great success.

Happy reading. I hope you find the book valuable.
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

From The Web: "Noncertified IT pros earn more..."

2007-10-23T19:07:00.000-04:00

Stop the presses! Can it be true? The industry wants effective, qualified, multi-dimensional people who are capable of understanding business drivers & risk mitigation and applying that in a sapient way to their job as opposed to folks who paid someone to teach them how to pass a multiple-choice exam?!? Amazing!

Noncertified IT pros earn more than certified counterparts: survey

From the Mailbox: Software Development: Art or Science?

2007-10-18T19:05:00.000-04:00

Here’s a question that I didn’t realize I had much to say about until I read my own response.

The Question:
Software Development: Is it an art or a science? An age old question I know, but what do you think and why?
My Response:

I refer to new software development as a scientific art. I've seen some maintenance work, platform porting, etc. that has been almost entirely mechanical -- I'm not sure what that counts as, but I certainly didn't witness anything "artistic".

I came to this distinction watching my brother's career. He is a scientific-glass-blower to earn a living. He has a degree, he makes high precision, custom order, scientific instruments out of glass. It's amazing to watch. It's hands on. Most people would call it a trade, but the degree to which he has to understand composition of materials, thermodynamics, stress points, and on, and on, and on, is nothing short of scientific. During the day, there isn't much art there, but the items he creates are absolutely artistic to my eye.

In the evenings, he does artistic glass blowing in the glass lab he (with the help of my father, the retired shop teacher) built in the garage. There the art comes first, but to make the art, again, he has to use scientific knowledge, such as exploiting 10 degree differences in melting points between colors of glass to create his art (or something to that effect, I admit that I don't have all the details and that I may be messing that up, but I think the point is clear though).

I think the closed commonly used term for something like that is a "craft", but (my best guess anyway) is that software folks reject the idea of being labeled "craftsmen" because software folks want to be seen as "white collar" and most folks seem to associate "craftsmen" with "blue collar".

I'm sure there are plenty of other careers/industries that have the same trouble classifying what they do. The challenge with software development is that because it is (relatively speaking) so young and poorly understood that the titles and classifications that get stapled to it impact the practices that are expected.

Purely artistic practices are not useful for commercially driven software. Purely scientific/engineering practices are not useful for software containing virtually any degree of innovation. I believe that most (and if not most, certainly a *whole lot* of) software is both commercially driven AND innovative.

Thus, until someone comes up with something better, I'm gonna stick with "Scientific Art".

From the Mailbox: What makes software "good" or "bad"?

2007-10-16T18:59:00.000-04:00

I was asked the question below (lightly edited for anonymity, clarity, and length) today and found it intriguing, so I thought I'd post it here.

The Question:
This is an attempt to understand how (and why) users, practitioners, and professionals perceive the difference between a good software product and a bad software product, specifically released software products.
My Response:

I can think of several ways to address that question, but the two that jump out at me are:


I believe that an end-user primarily makes their assessment based on their ability to use the software product to accomplish the task they wish to accomplish with the software product. This in itself is interesting, because from an end-user perspective, the software product that does the best job at helping them accomplish their task, for instance, may be priced to highly for them to afford, or be too hard to learn, or not work on their system -- thus making it "bad" in their eyes.

On the other hand, a stakeholder may view a software product as good if it generates sufficient revenue or publicity, even if that software product is generally considered to be of "low quality".

Let me use a software product that is no longer being actively sold (it may not even be available anymore) as an example. Before IBM purchased Rational, Rational had a load generation tool that (at least at one point in time) was called Performance Studio. Performance Studio was (probably still is) my favorite load generator of all time. It's the tool I learned on. I knew how to make that tool do *almost* anything one could want from a load generator. I had (still have, actually) the most amazing peer support group around that tool I could imagine.

That said, over years and years of using that tool, I came to find out the following:


I could go on, but I'll stop there. The point is, even with all of these obvious "bad" traits, I *loved* that software product. For me, it was (and still would be) a *good* software product. For much of the rest of the industry who had need for the functionality that product provided, it was considered *bad*.

I guess, all that is to say:

"Good is in the eyes of the stakeholder."

Gentleman, Start Your Engines!!!

2007-09-17T18:58:00.000-04:00

My most recent column, inspired by a surprise trip to the Brickyard 400, has just been posted on TechTarget in which I discuss the distinction between "delivery" and "done" when it comes to testing the performance of software systems.

Countless hours of development are now in the past. Testing indicates that everything is ready for the big day. The whole team is on hand, and the world is watching. It's the moment of truth; time to find out if all of the hard work is going to pay off. Anticipation builds until the command is given…

"Gentlemen, start your engines!"

The cars come to life. They take a few pace laps and at last, the green flag drops. In fewer than 90 seconds the cars are back on the front stretch approaching speeds of 200 mph -- the pinnacle of stock car performance.

This summer I worked on a project in Indianapolis. Usually when I travel to remote client sites I fly home on the weekends, but there was one weekend that I chose to stay. I chose to stay for two reasons. First, the flights for that weekend were insanely expensive and second, I have some friends in Indianapolis whom I'm always happy to have an excuse to visit. As luck would have it, the flights were expensive because that was the weekend of the Brickyard 400, and one of the friends I wanted to spend time with had a spare ticket, which I shamelessly accepted when he offered.

During the pomp and circumstance leading up to the start of the race I realized what a fabulous example the race was of one of my most-quoted sound bites related to performance testing: "Don't confuse delivery with done."


See the column for the rest of the story.

Model Workloads for Performance Testing: FIBLOTS

2007-08-25T12:49:00.000-04:00

This is the third installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


For years, I have championed the use of production logs to create workload models for performance testing. During the same period, I've been researching and experimenting with methods to quickly create "good enough" workload models without empirical data that increase the value of the performance tests. I recently realized that these two ideas are actually complimentary, not exclusionary, and that with or without empirical usage data from production logs, I do the same thing, I:

FIBLOTS.

While the play on words makes this mnemonic particularly memorable, I'm not saying that I just make it up. Rather the acronym represents the following guideword heuristics that have served me well in deciding what to include in my workload models over the years.

Classify Performance Tests: IVECTRAS

2007-08-03T12:46:00.000-04:00

This is the second installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


I have struggled for over 7 years now with first figuring out and then trying to explain all the different "types" of performance tests. You know the ones:


Well, I finally have an alternative.

IVECTRAS

IVECTRAS is valuable for classifying performance tests (or test cases if you like that term better) and performance test objectives. Better still, it is easy to map to Criteria, Requirements, Goals, Targets, Thresholds, Milestones, Phases, Project Goals, Risks, Business Requirements, Scripts, Suites, Test Data, etc. Yet even better, you can use it as a heuristic to assist with determining performance testing objectives and performance test design. So what is it?

To determine, design or classify a performance test objective or test, ask is this an:

INVESTIGATION or VALIDATION
of END-TO-END or COMPONENT
under ANTICIPATED or STRESSFUL conditions

For me (and my clients since I came up with this) there is a lot less confusion when one says "We need to INVESTIGATE COMPONENT level RESOURCE consumption for the application server under STRESSFUL conditions" than it is to say "We need to do a unit stress test against the application server." Even if there are still questions to be answered after applying IVECTRAS, at least the questions should be more obvious -- and if nothing else, *that* adds value for me.

Hourly Rant...

2007-07-30T12:40:00.000-04:00

I just finished answering a question posted on LinkedIn by Esther Schindler in researching a article she is working on for CIO.com

She asks (summarized):

"There's just one question to answer: If you could get the (client) boss(es) to understand JUST ONE THING about computer consulting and contracting, what would it be?

Or, to put the same question another way: If you were given a single wish of something to change (about a current or past client) what would it be?"

My response (lightly edited from the original):

If you are going to hire a consultant, do not ask them to bill by the hour. As a senior person with expertise in an area that you/your team doesn't have expertise in, a consultant had better know what they are doing. As such, it is not the number of hours they spend planted in one of your chair that you are paying them for, it's their results.

For example, if you were to decide that you needed a top of the line, custom-designed, tile floor in your kitchen you would expect to get some kind of quote for the total cost of installation and you would expect to pay exactly that price after the work was done. You would not expect to get a huge discount in price if the work took 1hr instead of the 60hrs you expected... in fact, you might just be inclined to give the installation crew a bonus for not inconveniencing you by driving you out of your kitchen for a week and a half!

Why, then, is it that people who hire consultants insist on paying them by the hour?

First of all, once I get on a plane, my time is yours. Typically I find that I can grossly exceed expectations in about 2 hrs a day OR that I need to put in 20+ hours a day to help you achieve your desired outcome. Either way, to me, each day is still a day that I am away from my family, a day that I can't spend doing something else, another plane ticket, rental car & hotel on my credit card until the client pays, and another day of eating junk from a vending machine, junk from an affordable chain, or wildly overpriced food from a restaurant that still can't compete with home. Basically, in most cases, a day is pretty much a day.

Second, which hours would you like me to bill? The ones that:

a) I'm sitting in your nice little mini-cubical with no computer, no phone, no internet access, no security badge, and essentially no way to accomplish anything while I wait for you to set up some meeting to (in too many cases) brief me on all the stuff that matters least to the ultimate solution?

b) In the hotel with my laptop (that you won't let me bring onto the premises for security reasons), the internet, my notes and my entire network of niche experts on the phone or chat trying to solve your problem?

c) Tailgating at the Brickyard 400, with a key member of the team who invited me to the race, taking mad notes while interviewing him about the *real* reasons behind needing a consultant is for this particular project.

Needless to say, most executives choose a) which essentially encourages the not-so-ethical consultants out there to spend a lot of time sitting in an office looking busy to clock enough hours to be able to bill enough $ to justify them being away from home.

If executives would simply pay based on results instead of hours, I submit that both the consultants and their clients would stop feeling ripped off.

Software Testing Lessons from my Children

2007-06-18T12:33:00.000-04:00

My most recent column has just been posted on TechTarget in which I discuss some of the lessons I‘ve learned from my children about software testing.

I had planned an entirely different topic for this month, but I‘m sitting down to write this on Father‘s Day while my sons (Nicholas, age 8, and Taylor, age 4) are napping, and realizing that I‘ve never written about what I have learned about testing from my boys.

Before I share some of these lessons, let me first share a little about me and fatherhood. For all of the dedication, time, and passion I give to my career, it is not even comparable to the dedication, time. and passion I give to my boys. For example, I stopped consulting for a while so I could see my boys every day when they were young because I couldn‘t stand the thought of being on the road for their first steps, new words, and all of the other developmental wonders that occur on almost a daily basis during the first several years of life. When I went back to consulting, I started my own company—not because I wanted to run a company, but because I didn‘t want to have to answer to anyone else when I chose to not travel during baseball season so I could coach my son’s team. In the same spirit, when I work from home, I frequently do so in a room with my boys, who are naturally curious about what I‘m doing. Over the past few years of this, I’ve learned a lot of things about being a good tester from them. Some of the most significant are these:

Don‘t be afraid to ask "Why?" Exploratory play is learning Recording your testing is invaluable "Intuitive" means different things to different people Fast enough depends on the user You can never tell what a user may try to do with your software Sometimes the most valuable thing you can do is take a break

See the column for more behind the lessons

Performance Testing Core Principles: CCD IS EARI

2007-05-21T12:29:00.000-04:00

This is the first installment of a currently unknown number of posts about heuristics and mnemonics I find valuable when teaching and conducting performance testing.

Other posts about performance testing heuristics and mnemonics are:


There is not a "one-size-fits-most" approach to performance testing, but I have become rather convinced that there are nine principles that are (almost always) applied (or at least actively considered) in successful performance testing projects. I remember those principles by remembering:

CCD IS EARI


For more see Developing an approach to performance testing -- CCD IS EARI.

Five Questions with Jon Bach, by the Braidy Tester

2007-03-30T10:43:00.000-04:00

I met Jon about 3 years ago. It was a funny story, actually. I was at STAREast talking with a bunch of folks at the bar after the last presentation of the day. Some guy came over and introduced himself to the person sitting next to me.

I heard his name and I stopped, mid-word, stood up excitedly, started shaking his hand and talking a mile-a- minute...

(Scott) "OhMyGod! Jon Bach! I'mSoExcitedToMeetYou! IReadYourBookAnd... I'm sorry, my name is Scott Barber, I've done some work with your brother..."
(Jon) "Wait! JamesToldMeAboutYou! You'reThePerformanceGuy! IRunATestingLabInSeattleAnd... How about we sit at the bar, I'll buy you a beer."

Jon has been one of my best friends ever since. Oh yeah, he's also one of the best testers and teachers of testers I've ever met. If you don't know Jon -- or even if you do -- Michael Hunter posted Five Questions With Jon Bach today. Take a look, it's a good read.
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Custom Performance Testing Search Engine

2007-03-22T10:42:00.000-04:00

About 24 hours ago, Google Co-op publicly released the ability for folks to make their own CSE's (Custom Search Engines). From the site:

Harness the power of Google search
Create a highly specialized Custom Search Engine that reflects your knowledge and interests. Place it on your website and, using our AdSense for Search program, make money from the resulting traffic.

See examples of how a Custom Search Engine works.

What you can do with a Custom Search Engine

Place a search box and search results on your website.
Specify or prioritize the sites you want to include in searches.
Customize the look and feel to match your website.
Invite your community to contribute to the search engine.

Sound cool? I thought so. So cool, in fact, that it threw me into a fit of ADD obsession. It took me about 21 of the last 24 hours to do, but now there is CSE just for Performance Testers. Just think about it, no more results for tuning sports cars, training for a marathon or measuring employee productivity when searching for material related to software performance testing.

Check it out and let me know what you think!

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."

Resumes in Context

2007-01-31T01:58:00.000-05:00

On a forum related to James Bach's Rapid Software Testing On-line (Beta) class (which I highly recommend! A few more technical issues to work out and it should be ready for prime-time) another student (Anne Marie Martin, from Atlanta) posted the following (lightly edited):

Here's something I struggle with though, and would love to hear thoughts on. I have about 11 years experience in testing, and try to invest time in learning more about testing, and learning more in general that can help me with testing - such as the things we've all been discussing about philosophy and learning and Weinberg and a hundred other things that have tickled my brain during our discussions and threads that made my 'to do' list of things to read or explore or learn from.

Where I struggle is in practically applying those things to a potential employer. In some cases, I still find myself at a disadvantage because I am not a college graduate. {...snip...} because of the face of the industry at the time, experience was valued more than the degree.

Increasingly today, as I'm researching available options in my area, I see many employers who *require* a college degree, or who desire certification + a given experience level. I do not think there is any value at all in the professional certifications available today for testers, personally, and it's not that I would never finish my degree, it's just that I think I do things to educate myself, and that I am not sure how to 'sell' that.

When dealing with a prospective employer, how does one best 'sell' the things that they do to self-educate? It's not exactly standard to put things of that nature on a resume, that I've seen. Listing classes or professional organizations, assuredly, but listing out things you're teaching yourself about, or have researched and learned from that you think bring value to your testing? I'm not sure how to do that. In an interview, it's far easier, because these are things you can bring up - it's 'getting in the door', so to speak, that I think requires something up front, even, and I'm not sure how that's best achieved.

This is sadly common, employers looking for more and more checkboxes that "demonstrate" that a person will be a good fit for the job and do good testing. When are employers going to learn that none of the best testers I can think of (my apologies if I'm not thinking of someone, or spaced on your educational background) have Computer Science degrees? Heck several don't have college degrees and at least one expert tester (as demonstrated by the number of times he has been an Expert Witness related to software testing in court) doesn't even have a H.S. diploma! (Yes folks, that would be James Bach - one of the smartest and most educated people I have ever had the honor to meet). Anyway, below is my response (also lightly edited):

Yes, sometimes employers really do *require* things that make no sense to us mere mortals. (U.S. Government contractors seem to be among the worst). In those cases you have little choice but to check the box, or forget them as potential employers.

As for the Resume, the way I look at it (and I am probably an exception rather than a rule, but I am speaking from experience as a job hunter, manager who hires and an adviser to a successful technical recruiting firm) standard Resumes are for standard people.

It's been about 10 years since I gave up on a "standard" resume format and only on rare occasions when my resume was being submitted as part of a proposal with other people have I been asked to reformat it.

Remember resumes (CV's whatever) serve exactly 2 purposes.1) Get you past the intern someone hired to screen resumes.2) Make you appear interesting enough to call in for a chat.

I recommend to people that they customize their resume for each position, do their research first, substitute your preferred terminology with the job posting's buzzwords to the degree your ethics allow, and otherwise use it as a 2 page attention getter.

For instance:

I don't quantify everything. I leave some bullets open ended in a way that makes a manager want to call me *just* to see how the story ends.

I had someone else craft my intro paragraph (I'm actually considering replacing it with one of the endorsements from my website, with attribution). I've had lots of positive comments on that.

I list my skills up front, independent from experience and training. Let's face it, who cares if you learned how to program in VB from a book, from a H.S. teacher, from a college course, on the job or from your kid? As long as you can do it, it's a skill you have.

For a long time I separated my experience from my employment history (I still do to some degree - once you start consulting, the "standard" format breaks down anyway). For experiences to be relevant, they don't need to have be tied to money. OpenSource projects, building a website for your church, testing prototype exercises on the side for James Bach... these are real and relevant experiences. In fact, they are probably more educational than many of the paid jobs people do.

Under a section like "Education/Training" don't be afraid to list things like peer workshops, impromptu classes with trainers at conferences (though you should check in with them first... wouldn't it be embarrassing if the potential employer asked the person and they said "Huh? Maybe, but I don't remember doing that.")

When you get stuck and don't feel like your resume is telling the story you want it to tell, ask someone you respect to write either a short endorsement to put on your resume or a letter of endorsement to include. Jon Bach (James' brother) actually has "James Bach approved Rapid Tester, Cem Kaner recommended something or another, etc." on his resume... like 5 of them if I remember correctly.

Sure things like blogging and article writing can help, but they can also hurt. Some companies are psycho about Intellectual Property and actively avoid writers. No intern is going to check your blog to see which stack of resumes you belong in and most managers won't either. Blogs and articles helps with unsolicited jobs and contracts. I've rarely seen them help folks who are responding to job posts.

The other thing to do is to use your personal network. Ask folks you know, even casually, if they know someone looking for someone like you and then ask them to forward your resume. I get that all the time. The truth is that even when I forward a resume with a note that says "I don't know this person very well, but their resume looks decent and they were polite when I talked to them", about 90% of the time they at least get a call from the manager. It bypasses the whole intern process and the fact that it is coming through a friend (i.e. me) that interests them enough to make a call.

At the end of the day, that is what you really want. A call. A chance to demonstrate your skills. That is what earns you a job, not the resume.

The other thing to remember is that way to often, employers list requirements for what they wish they could get and a price that's half of what anyone who actually has those requirements would take. Don't worry about that. Apply for the jobs doing the things at the companies that sound interesting to you, craft a resume that would be interesting to the manager you'd like to work for and you really ought to be fine.

*Note* none of that is code for "lie", "stretch the truth" or anything of the kind. The truth is the truth. But unless your entire life story fits on 2 pages, you've got to summarize and highlight. It is not unethical or untruthful to highlight those true things that are most likely to get you an interview.

Now don't read this and immediately send me your resume and expect me to spam it around for you. But if this rings true with you, print it out and when no one is looking, pin it up on the "for managers only" bulletin board in your office. Maybe it'll help some of these fabulous testers land the jobs they deserve, but don't have the right checkboxes to be considered for.

--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
you will find it in your life."