Load Tests: Failures To Avoid and Opportunities to Seize

In no particular order, here are a few final things that have worked well for me and some lessons I’ve learned while load testing.  Examine each one. Think about them in the context of your organization, your personality, your duties, and the daily challenges you face.  Use what works and discard the rest.

Load Testing Is a Team Sport

teamLoad testing requires the help and cooperation of many people. This is usually extra work done at very inconvenient times. Those people can be ordered to cooperate, but they give you so much more if they understand what’s going on and get treated with a little respect. Here are a few good things to do:

  • Ask for their help.
  • Give plenty of advance notice.
  • Explain what you are doing with the test and inquire about any unforeseen difficulties the test might create, or opportunities the test might offer if a small change is made.
  • Bring food.
  • Share your results after the test.
  • Say ”Thank You” and mean it.
  • Pay it forward by always being helpful to others.

Reality vs. The Lab

It’s vastly easier to test the application in its native environment than it is to recreate and test a copy of it in a lab on rented equipment. If you have to go into the lab, expect many delays as you discover just how fussy a computer program can be about the computer it runs on, its surroundings, the placement of files, and network connectivity.

Lab-based Load Test Failures I’ve Seen More Than Oncelab

The team showed up with almost everything they needed. Pack carefully for the trip.

The team showed up with unfinished or poorly tested software. Don’t plan to fix it in the lab.

The team assumed that key equipment would be there. They assumed the right version of software and hardware would not only be available, but already be installed for them. They assumed an expert would be available at a moment’s notice. Don’t assume, check.

The team finally ran the load test successfully after several hardware and configuration changes. Sadly, no one recorded key details of the configuration that finally worked, so the results were meaningless.  Take careful notes and make load tests as self-documenting as possible.

The team did not plan how they would reset the system quickly between tests, so they spent 75% of their lab time resetting for the next run. Plan for multiple runs and do what you can to speed up test resets.

Use Checklists

checklistThe smallest forgotten detail can ruin a perfectly good load test. Have a checklist and use it. Checklists work for NASA, for surgeons, for pilots, and for load testers.

Ironically, all checklists are incomplete. When you find something missing from the list, take the time to update the master copy of the list.

Communicate During The Test

A full-power peak load test has the potential to be a disruptive event. Before the test starts, open a voice bridge so that all interested parties can follow what’s going on and report troubles quickly.  Check in with everyone periodically at key points. For example:

  • Before testing starts ask everyone: “Are you ready?”
  • When test is first running at low power: “Do you see that load in your meters?”
  • As the load is ramping up: “Any problems?”

Time is of The Essence.

timeEverything you can do to save yourself time, and everything you can do to speed up and automate your testing, is a good thing to do. This allows you to do more testing in a given amount time and to rapidly diagnose problems.  During the test many people will be impatiently waiting for you.

Spend time before the test preparing your tools to:

  • Start the metering and then the test
  • Check if the test and the metering are running as expected
  • Rapidly capture every possible clue when the test fails
  • Reset things to a known state after a run
  • Do rapid performance analysis of the metering data looking for problems and bottlenecks

Clean Up After Yourself

trashThis will not be your only load test. Make sure you have time in the schedule for cleanup.

If you are in your own internal lab, or some vendor’s lab, leave it nicely picked up when you are done testing. The people who will have to clean up your mess are also the people who will help you set up for the next test.

Someday, you’ll be back in that lab and need their help again.

Sharpen the sword

After you do a big load test, you should take time to review your notes, logs, and memories looking for things that worked well and things that did not. Improve your tools for next time.  This is also a good time to spend time publicly thanking those who helped you.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s