When Tension Is High, Start With Some Good News

Sometimes the success or failure of your company hangs on your results. When the stakes are high, you might want to think about going to the movies. Specifically an old-school, action movie, like Indiana Jones.

A big reason audiences enjoy an action movie is that they are reasonably sure from the onset that they will like the ending. They are sure that the hero will triumph, and the wrongs will be righted.

There is a temptation to use a dramatic style when presenting the results of your work because you naturally want to tell a story that builds in excitement and drama and finishes with thunderous applause. That is a fine thing to do, but it works much better if you tell them very early in the presentation that all will be well. Then the audience can relax and enjoy the ride. So start your talk with something like this:

With the current configuration, we will not be able to handle the upcoming peak. However, I’ve identified the bottlenecks and I have workarounds to propose for all of them. Let me show you what I’ve found.

I have seen presentations, without this early calming statement, go badly. When the presenter was about half way through the list of all the serious problems ahead some participant will start angrily demanding something like:

Are we screwed?!?
Please Tell Me There Is a Fix For Some Of This?!?

This is not what you want. Whenever possible, lead with the good news that there is a solution to all the problems you have discovered. When you don’t have a solution for a problem, at least tell them that you’ve found the root/cause of the problem they are currently having. People need hope.

Other helpful hints on presenting your results, as well as capacity planning and load testing, can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


Judge Not

When you present your performance results, you are not a superhero striking down evil. You are not there to make yourself look big by making others feel small. You are not there to judge.

nosuper noyellnojudge

You are a member of the team, dispassionately presenting well-checked information and potential solutions for problems. Stick to the facts, and leave the judgments to others.

I have seen presentations where the speaker delivered the bad news in a mocking and sometimes directly insulting way that hurt group cohesion and deeply offended people in front of their peers. That approach did not aid in answering the question, but it did unleash a wave of back-stabbing and other bad behavior. Every time I have seen someone be intentionally cruel or hurtful to a co-worker, it has not worked out well for them, especially in the long run.

Care Deeply

The single biggest “trick” I use in every talk I’ve every given is to care about the people in the audience. This is not a trivial closing thought. This is a core truth. You can tell when someone cares about you, and you can tell when they don’t. You naturally give more attention and show more compassion to people who care about you.


When you construct the presentation and in the time just before you start to give it, let yourself focus on deeply caring about the audience. Care about their needs and concerns. Care about the fact that they have been sitting in meetings for two hours before you started. Care about them as people and co-workers. Caring connects you to them in a powerful and positive way. If you honestly care, they can’t help but listen.

Other helpful hints on doing performance work, as well as capacity planning and load testing, can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


When Does The Warmup End?

Unless you are focused on optimizing the time to restart an application, the meters you might collect at application start are pretty useless as things are warming up. The first few transactions will find the disks nearly idle, queues empty, locks unlocked, nothing useful in cache, etc. As the work flows in, queues will build, buffers will fill, and the application will settle down after a short period.warmup

There have been many math geniuses that have spent years trying to find a mathematical way to know when the warm-up period is over and you can start believing the data. To date there is no good mathematical answer. You just have to eyeball it. Fire up the application and let it run. Your eye will clearly note if, and when, the response times and utilizations stabilize. Ignore the data during the warm-up period.

Other helpful hints can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


Deconstructing Response Time

The overall response time is what most people care about. It is the average amount of time it takes for a job (a.k.a. request, transaction, etc.) to get processed.  The two big contributors to response time (ignoring transmission time for the moment) are the service time: the time to do the work and the wait time: the time you waited for your turn to be serviced.  Here is the formula: ResponseTime = WaitTime + ServiceTime

service center 3

If you know the wait time, you can show how much faster things will flow if your company spends the money to fix the problem(s) you’ve discovered. If you know the service time then you know the max throughput as MaxThroughput ≤ 1 / AverageServiceTime 

For example: A key process in your transaction path with an average service time of 0.1 seconds has a maximum throughput of: 1 / 0.1 = 10 per second.

Sadly, response time is the only number that most meters are likely to give you. So how do you find the wait and the service time if there are no meters for them? The service time can be determined by metering the response time under a very light load when there are plenty of resources available. Specifically, when:

  • Transactions are coming in slowly with no overlap
  • There have been a few minutes of warm-up transactions
  • The machines are almost idle

Under these conditions, the response time will equal the service time, as the wait time is approximately zero.

ServiceTime + WaitTime = ResponseTime
ServiceTime + 0 = ResponseTime
ServiceTime = ResponseTime          

The wait time can be calculated under any load by simply subtracting the average service time from the average response time.

WaitTime = ResponseTime – ServiceTime

Performance work is all about time and money. When you’ve found a problem, a question like: “How much better will things be when you fix this?” it is a very reasonable thing for the managers to ask. These simple calculations can help you answer that question.

Other helpful hints can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.

Fast Frozen Fun

As some of you may know I’m retired from performance work and my only “job” is working one day a week as a tour guide at Ben & Jerry’s.  I love ice cream and I love hands-on experimenting and lately I’ve been experimenting with instant fruit ice cream and it’s great. Here is the recipe.

Put 10 -12 ounces (280 – 340grams) of hard frozen fruit in a food processor. I’ve tried strawberries, blueberries, mangos, and peaches and they all taste great. Add to that 1/2 cup (100grams) of sugar. Now run it until the fruit begins to break down into small bits and then add about 1 cup (250ml) of half-n-half. What you end up with is soft-serve ice cream that you should plan to eat immediately.  Total time: less than 2 minutes.yum

Some notes from my experiments:

  • If you want a stiffer product, pre-freeze the work bowl and blade of the food processor as well as the sugar.
  • Try dropping a chocolate bar into the food processor as it is spinning. Strawberry and chocolate go so well together.
  • Try adding a few drops of vanilla, almond, or cherry extract. Yum!
  • Heavy cream did not work for me, as the food processor acts like a churn and the ice cream had a distinctive greasy mouth feel that food scientists refer to as “buttering”.


When You Are Close To The Edge

acliffAt the Grand Canyon there are many places where you can walk right up to a cliff where, with one more step, you will fall hundreds of feet to your death. The closer you are to the edge of a cliff, the more precisely you need to know your location. In your campsite, a half-mile away, your exact location is not so critical. This is also true in performance work.

If the numbers show a resource will be 20-25% busy at peak, I would not spend more time getting a more precise version of that number. You could be off by a factor of two and the resource would most likely be fine at 40-50% busy. The closer you are to some performance limit, the more careful you have to be with your calculations and predictions.

With any prediction of future behavior there will also be some error, some uncertainly. Some of this is your fault, some of it is the fault of the person who specified the peak load to plan for, and some of it is the fault of the users who didn’t do exactly what was anticipated on that peak day.

When the boss says plan for a peak load that is two times the observed load, do what you are asked. Then, look to see if you are close to “the edge” of some performance cliff. If you are close, go back to the boss and show what you’ve found and ask: “How sure are you about your predicted peak load?

I’ve seen many cases where, when shown how close to the edge a system would be at peak, the decision makers change their minds and give a different number to plan for. Sometimes that number is:

  • Bigger because they want to buy a new stuff
  • Smaller because they don’t want to spend money
  • Bigger to protect the budget for next year
  • Smaller because they just got new growth projections
  • Different than the last number because of the crisis they are dealing with at the moment you happened to ask

It’s your job is to advise, not decide. Present your data, give your best advice, and be at peace. A business decision weighs costs, risks, politics, and the art of what is possible.

This sound advice came from: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


The Sample Length of The Meter

Any meter that gives you an averaged value has to average the results over a period of time. If you don’t precisely understand that averaging, then you can get into a lot of trouble.

The two graphs below show exactly the same data with the only difference being the sample length of the meter. In the chart below the data was averaged every minute. Notice the very impressive spike in utilization in the middle of the graph. During this spike this resource had little left to give.dailypeak1

In the chart below the same data was averaged every 10-minutes. Notice that the spike almost disappears as the samples were taken at such times that part of the spike was averaged into different samples. Adjusting the sample length can dramatically change the story.dailypeak2

Some meters just report a count, and you’ve got to know when that count gets reset to zero or rolls over because the value is too big for the variable to hold. Some values start incrementing at system boot, some at process birth.

Some meters calculate the average periodically on their own schedule, and you just sample the current results when you ask for the data. For example, a key utilization meter is calculated once every 60 seconds and, no matter what is going on, the system reports exactly the same utilization figure for the entire 60 seconds. This may sound like a picky detail to you now, but when you need to understand what’s happening in the first 30 seconds of market open, these little details matter.

Below you will see a big difference in the data you collect depending on how you collect and average it.  In the one-second average (red line) you are buried in data. In the one-minute average (sampled in the yellow area) you missed a significant and sustained peak because of when you sampled. The 10-minute average (sampled in the green area) will also look reassuringly low because it averages the peaks and the valleys.avg3

Take the time, when you have the time, to understand exactly when the meters are collected and what period they are averaged over. The best way to do that is to meter a mostly idle system and then use a little program to bring a load onto the system for a very precise amount of time and see what the meters report. The better you understand your tools, the more precisely and powerfully you can use them.

This hint and many others are in: The Every Computer Performance Book which is available at AmazonPowell’s Books, and on iTunes.