Thoughts on Performance Testing w/o "Tools"

I was recently asked the following question via the "Ask The Expert" feature of SearchSoftwareQuality.com.

How can we conduct performance testing, stress testing, and load testing of a Web application manually without using any tools?

My commentary is reproduced below -- you'll have to click through to see my actual recommendations.

First, I want to express my sympathy to anyone who finds himself in a position of being asked to create multi-user simulations (i.e. the load part of performance/load/stress testing), requesting a load-generation tool, and being denied. In my experience, the excuse of not being able to afford a load-generation tool is almost always just that -- an excuse.

Only once in my career have I found the only tool capable of generating a production-like load against the application I was testing to be prohibitively expensive. (In that case, the tool cost more to purchase than the application was anticipated to earn in a year.) In every other case, either the risk justified the cost or an adequate, inexpensive, or free tool has been available.

The only really good reason to be in this position is if all available tools were considered and none of them supported the application under test (even with customizations and extensions) and you don't have access to the skills (internally or externally) to build a tool of your own -– and that seems unlikely to me.

The simple truth is that if a company is building an application that is realistically expected to have enough users to justify the expense of performance testing, even if that expense is just the time of an employee, that company ought to be projecting enough revenue from the application, or lose enough credibility by having a poorly performing application, to justify either the cost of a tool or the risk (in some company's eyes) of using a free or open source tool.

Now, after saying all of that, I must admit I have found that the vast majority of value that is gained by quality performance testing comes outside of the load-generation tool. Some of my favorite techniques (assuming you are testing websites):

Read the rest of the article here.
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Latest Column -- The controversy surrounding the schools of software testing

My latest column...

Periodically, discussions break out in various software testing communities around the Web regarding the schools of software testing.

As I write this, there are discussions going in SQAForums, on the Software-Testing Yahoo! group, and various blogs that (at least up to the time I started writing this piece) reside on or are fed to Testing Reflections. In principle, I'm always pleased when these discussions break out. The point of identifying the schools in the first place was to increase the overall awareness of the diversity in ideologies, practices, and values (i.e. schools of thought) in our field and to stimulate discussion about the situational pros and cons of each. That said, the discussions that actually take place tend to drift off in one or more directions that end up being disappointing, unnecessarily confrontational, and generally not useful.

After witnessing this pattern, participating in these recent discussions, and listening to comments from those who followed the discussions for several years, I've identified several areas in which these discussions go awry. Below, I call those out and share my thoughts about each.

Read the rest of the column.
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Latest Column -- Testing training: Disturbing behaviors of students

My latest column...

Drive-by training. Never heard of it? It is exactly what it sounds like. You drive to a training facility (or an instructor drives to you), for a day or three the instructor delivers the pre-packaged training class, then everyone drives back home. It's not the best training model ever invented. There is generally no student assessment, and the only instructor/course provider accountability is reputation. Even so, many good ideas can be shared and lots of students come away feeling that it was well worth "the drive."

As it turns out, I've been delivering a lot of drive-by training to software testers this fall. That in itself isn't particularly noteworthy -- end-of-the-budget year is a popular time for drive-by training -- but something that is noteworthy is that I have noticed a rise in some disturbing behaviors among the individuals and organizations that select and attend drive-by training.
At first, I thought it was just me. But after an informal poll (and some lively discussions) with my employees and trainer friends in the testing realm, I became increasingly convinced that the behaviors I'm noticing are not exclusive to me and that I'm not the only one who thinks they are on the rise.

Read the rest of the column.
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Latest Column -- Software Testers are not helpless

My latest column...

During a coffee break at a class the other week, I overheard the following comment from one student to another:

Tester: "This stinks! All of my automated test scripts are broken and I can't seem to get the tool to work now that the developers have enabled Secure Sockets Layer. I'm going to have to work through the weekend."

I know that it's generally considered rude to eavesdrop, and ruder still to comment on a conversation you weren't invited to, but I figured that since I was teaching the class I'd be forgiven. Besides, I simply couldn't help myself.

Read the rest of the column.
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Latest Column -- Avoid "Center of the Universe Syndrome"

My latest column cautioning testers not to think they are the center of the development team's universe http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1325828,00.html. 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

5 Questions with Scott Barber by a Braidy Tester

I recently had the honor of being interviewed by Michael Hunter, a Braidy Tester, for Dr. Dobbs Portal. Check it out: 5 Questions with Scott Barber. 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Latest Column -- Inspired by taking AST's Bug Advocacy Class

I recently completed (successfully, I might add) the second of the Association for Software Testing's all online, free to members Black Box Software Testing course. Each of these courses is four weeks in length. I've been involved with this program since years before it became a program, and I am an instructor for the first course in the series, called Foundations. For this course, called Bug Advocacy, I was a student.

Bug Advocacy focuses on the skills and concepts needed to compose high-quality, easily understood, appropriately compelling and well organized defect reports. I know, it sounds pretty boring to me too, but it was anything but boring. These classes are designed so that you watch recorded lectures (in this class the lecturer is Cem Kaner), answer some quiz questions (to make sure you watched the lectures), participate in class discussions, do both individual and group projects (in this class the project centered around evaluating and enhancing unconfirmed OpenOffice bug reports), peer reviewing one another's assignments, and taking a far-from-trivial closed-book essay exam. All in all, I spent about 40 hours participating in the class over the four week period.

This approach isn't just about writing a good bug report, it's about making sure you do the right testing after you find a bug.
There was one idea in particular from the class that I found absolutely brilliant and wanted to share with you. Below is actually a very lightly edited version of my answer to one of the exam questions asking us to describe a six-factor approach to bug reporting that Cem remembers using the mnemonic "RIMGEA." If you are a regular reader of mine, you know that I have a fondness for mnemonic devices, but that's not what I thought was so great about the approach. What I think is brilliant is that this approach isn't just about writing a good bug report, it's also about making sure you do the right testing after you find a bug to enable you to write a good bug report. Take a look -- you'll see what I mean.

Click here to read the rest of the column

Click here for more information about AST's free-for-members, online training
 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."

Testing Lessons From Civil Engineering

Below is the paper I submitted as a prologue to an experience report, discussion, and (hopefully) additional research that I'm presenting for the first time during CAST08:

Engineers don’t look at the world the same way that testers do.  Engineers look at the world with an eye to solving problems.  Testers look at the world with an eye toward finding problems to solve.  This seems logical.  What is less logical is the fact that engineers, and I’m talking about the kind of engineers that deal with physical objects, seem to be much more sophisticated in their testing than testers.  In fact, most of what I know about testing, I learned as a civil engineering student.  We didn’t call most of it testing.  We didn’t even identify it as anything other than “You really want to get this right.” Maybe Civil Engineers test better than software testers because of the motivations to “get it right”.  Consider what happens when a piece of Civil Engineering, like a bridge fails:

Identity crisis or delusions of grandeur?

In this month's installment of "Peak Performance" I discuss the frequently erroneous and often grandiose titles software testers have on their business cards or in their e-mail SIGs. Identity crisis or delusions of grandeur? 
--
Scott Barber
Chief Technologist, PerfTestPlus, Inc.
About.me

Co-Author, Performance Testing Guidance for Web Applications
Author, Web Load Testing for Dummies
Contributing Author, Beautiful Testing, and How To Reduce the Cost of Testing

"If you can see it in your mind...
     you will find it in your life."