Deconstructing Response Time

The overall response time is what most people care about. It is the average amount of time it takes for a job (a.k.a. request, transaction, etc.) to get processed.  The two big contributors to response time (ignoring transmission time for the moment) are the service time: the time to do the work and the wait time: the time you waited for your turn to be serviced.  Here is the formula: ResponseTime = WaitTime + ServiceTime

service center 3

If you know the wait time, you can show how much faster things will flow if your company spends the money to fix the problem(s) you’ve discovered. If you know the service time then you know the max throughput as MaxThroughput ≤ 1 / AverageServiceTime 

For example: A key process in your transaction path with an average service time of 0.1 seconds has a maximum throughput of: 1 / 0.1 = 10 per second.

Sadly, response time is the only number that most meters are likely to give you. So how do you find the wait and the service time if there are no meters for them? The service time can be determined by metering the response time under a very light load when there are plenty of resources available. Specifically, when:

  • Transactions are coming in slowly with no overlap
  • There have been a few minutes of warm-up transactions
  • The machines are almost idle

Under these conditions, the response time will equal the service time, as the wait time is approximately zero.

ServiceTime + WaitTime = ResponseTime
ServiceTime + 0 = ResponseTime
ServiceTime = ResponseTime          

The wait time can be calculated under any load by simply subtracting the average service time from the average response time.

WaitTime = ResponseTime - ServiceTime

Performance work is all about time and money. When you’ve found a problem, a question like: “How much better will things be when you fix this?” it is a very reasonable thing for the managers to ask. These simple calculations can help you answer that question.


Other helpful hints can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


Fast Frozen Fun

As some of you may know I’m retired from performance work and my only “job” is working one day a week as a tour guide at Ben & Jerry’s.  I love ice cream and I love hands-on experimenting and lately I’ve been experimenting with instant fruit ice cream and it’s great. Here is the recipe.

Put 10 -12 ounces (280 – 340grams) of hard frozen fruit in a food processor. I’ve tried strawberries, blueberries, mangos, and peaches and they all taste great. Add to that 1/2 cup (100grams) of sugar. Now run it until the fruit begins to break down into small bits and then add about 1 cup (250ml) of half-n-half. What you end up with is soft-serve ice cream that you should plan to eat immediately.  Total time: less than 2 minutes.yum

Some notes from my experiments:

  • If you want a stiffer product, pre-freeze the work bowl and blade of the food processor as well as the sugar.
  • Try dropping a chocolate bar into the food processor as it is spinning. Strawberry and chocolate go so well together.
  • Try adding a few drops of vanilla, almond, or cherry extract. Yum!
  • Heavy cream did not work for me, as the food processor acts like a churn and the ice cream had a distinctive greasy mouth feel that food scientists refer to as “buttering”.

 

When You Are Close To The Edge

acliffAt the Grand Canyon there are many places where you can walk right up to a cliff where, with one more step, you will fall hundreds of feet to your death. The closer you are to the edge of a cliff, the more precisely you need to know your location. In your campsite, a half-mile away, your exact location is not so critical. This is also true in performance work.

If the numbers show a resource will be 20-25% busy at peak, I would not spend more time getting a more precise version of that number. You could be off by a factor of two and the resource would most likely be fine at 40-50% busy. The closer you are to some performance limit, the more careful you have to be with your calculations and predictions.

With any prediction of future behavior there will also be some error, some uncertainly. Some of this is your fault, some of it is the fault of the person who specified the peak load to plan for, and some of it is the fault of the users who didn’t do exactly what was anticipated on that peak day.

When the boss says plan for a peak load that is two times the observed load, do what you are asked. Then, look to see if you are close to “the edge” of some performance cliff. If you are close, go back to the boss and show what you’ve found and ask: “How sure are you about your predicted peak load?

I’ve seen many cases where, when shown how close to the edge a system would be at peak, the decision makers change their minds and give a different number to plan for. Sometimes that number is:

  • Bigger because they want to buy a new stuff
  • Smaller because they don’t want to spend money
  • Bigger to protect the budget for next year
  • Smaller because they just got new growth projections
  • Different than the last number because of the crisis they are dealing with at the moment you happened to ask

It your job is to advise, not decide. Present your data, give your best advice, and be at peace. A business decision weighs costs, risks, politics, and the art of what is possible.


This sound advice came from: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

The Sample Length of The Meter

Any meter that gives you an averaged value has to average the results over a period of time. If you don’t precisely understand that averaging, then you can get into a lot of trouble.

The two graphs below show exactly the same data with the only difference being the sample length of the meter. In the chart below the data was averaged every minute. Notice the very impressive spike in utilization in the middle of the graph. During this spike this resource had little left to give.dailypeak1

In the chart below the same data was averaged every 10-minutes. Notice that the spike almost disappears as the samples were taken at such times that part of the spike was averaged into different samples. Adjusting the sample length can dramatically change the story.dailypeak2

Some meters just report a count, and you’ve got to know when that count gets reset to zero or rolls over because the value is too big for the variable to hold. Some values start incrementing at system boot, some at process birth.

Some meters calculate the average periodically on their own schedule, and you just sample the current results when you ask for the data. For example, a key utilization meter is calculated once every 60 seconds and, no matter what is going on, the system reports exactly the same utilization figure for the entire 60 seconds. This may sound like a picky detail to you now, but when you need to understand what’s happening in the first 30 seconds of market open, these little details matter.

Below you will see a big difference in the data you collect depending on how you collect and average it.  In the one-second average (red line) you are buried in data. In the one-minute average (sampled in the yellow area) you missed a significant and sustained peak because of when you sampled. The 10-minute average (sampled in the green area) will also look reassuringly low because it averages the peaks and the valleys.avg3

Take the time, when you have the time, to understand exactly when the meters are collected and what period they are averaged over. The best way to do that is to meter a mostly idle system and then use a little program to bring a load onto the system for a very precise amount of time and see what the meters report. The better you understand your tools, the more precisely and powerfully you can use them.


This hint and many others are in: The Every Computer Performance Book which is available at AmazonPowell’s Books, and on iTunes.


 

The Five-Minute Trick

More times than I can count, I’ve been told I have an hour to present my findings and then, at the last minute, have found out that “Mr. Big Cheese is running late” and I’m either being bumped from the schedule altogether or cut back to a small fraction of the time I was originally allotted.time2

For a critical talk I always prepare a second, totally separate presentation that lasts no more than five minutes, and I offer that to whoever is doing the scheduling for the meeting.  99% of the time that offer gets gratefully accepted. Often, Mr. Big Cheese is intrigued with what I have to say in those five minutes, and I’m asked to go into more detail, while the scheduler goes off to tell someone else that they’ve been cut from the schedule.

This short presentation is not just going though your slides faster. It is a completely different presentation developed, edited, and optimized to deliver in that brief time. Please don’t start whining that you can’t possibly do justice to your months of detailed work and analysis in five-minutes. I’ve successfully given executives presentations this brief:

“You will easily make it through your seasonal peak.
The details are in my report.”

“The fix for your current problem is moving one
very busy file to three new disks. Your staff
knows what to do.”

As the performance person you can make a big impression when you are prepared to move swiftly, and can help keep the executives running on time.  That positive impression lasts a long time and is good for your career.


This hint and many others are in: The Every Computer Performance Book which is available at AmazonPowell’s Books, and on iTunes.


 

How To Become A Performance Guru

Performance work is a great career as everything change over time and with each change comes new performance challenges. There are always things to do and things to learn. Good performance work can save the company and put your kids though college. Yay!

Bad News… The Path Is Not Easy

This is a hard skill to learn as the knowledge required is diffused throughout many different sources. Let me explain…

First, performance books… Some are built on very difficult math that most people can’t do and most problems don’t require. They unnecessarily discourage many people. Many books focus on a specific product version, but you don’t have that version in your computing world. There is often no performance book for a key part of your transaction path.

Turning to manuals… Almost all manuals focus on a specific version of a technology and were written under tremendous time pressure at about the same time the engineering was being completed; thus the engineers had little time to talk to the writers. The manuals ship with the product. The result is that these books document, but they don’t illuminate. They explain the what, but not the why. They cover the surface, but don’t show the deep connections.

Should you accept the bad news, stop reading here and give up?  I don’t think so. There is hope. Hear me out…

First Of All, Don’t Worry About The Math

For 99% of the performance work out there you don’t need to use complex performance math equations.  The most complex formula I used in 25+ years of performance work is the one that approximately predicts how the response time will change as the utilization of a resource increases:
                    R = S / (1 – U)
If you can replace S with the number 2 and U with the number 0.5 and calculate that R is equal to (spoiler alert) 4, then you have all the math you need for a long career in performance.

Mine Low Grade Information for Gold…

rand_472

The UltraBogus 3000 features fully-puffed marketing literature, a backwards-compatible front door, and a Stooge-enabled, three-idiot architecture that processes transactions with a minimum of efficiency. Its two-bit bus runs conveniently between your office and the repair facility every Tuesday. The steering wheel was added because the marketing VP thought it needed more chrome.

RTFM (Read The ‘Fine’ Manual)

If your company just bought an UltraBogus 3000 (see picture to left) to handle your peak load then read the manuals cover-to-cover. You’ll be surprised what you find.

Sometimes what you find is a limit that is better discovered now than when you blindly hit it at the seasonal peak. Sometimes it is a question you never thought to ask. Sometimes it is a way to make your job vastly easier – even the worst product has some good features. You have to mine a ton of ore to find an ounce of gold.

You’ll (hopefully) be doing this job for years, take 15 minutes a day and chew your way though the manuals.

Lastly, reading the manuals teaches you the vocabulary you need to use when you call tech support. If you want to talk to their wizards, first you need to convince the people who initially take the call that you’re not an idiot.

readRead Performance Books

I wrote one that I think is generally useful and there are many others that will illuminate particular problems and show different ways of solving them.

They all have their strengths and weaknesses, but there is good stuff to be found there. Especially if the company is buying, try reading the ones with scary looking equations. Push yourself into unfamiliar territory. Even if you can’t understand it, having it on your bookshelf will intimidate your enemies. ;-)

If you want to be a performance guru then be all you can be. Read.

Search and Connect

Search engines are your friend. If you have a problem with X-technology, then it is highly likely that someone else has too. Ask simple questions and see what comes up. A lot of it is low-grade information, but sometimes you find just the hint you need.

LinkedIn has groups that are focused on every conceivable technology. Join a few a see if you can find a rich vein of information. There is also CMG and websites focused on performance like PracticalPerformanceAanalyst or PerfBytes to explore.

Now comes the tough part…

After you explore the sources above there are still many things of great importance you still won’t know. Performance work is in many ways a skill you teach yourself with the help of others. You have to dive in, like an explorer on a new planet, and try to make sense of the computing world you stand upon. startI’m often asked: Where do I begin?  My answer is to pick a small performance-related thing that interests you and deeply explore it. As you explore, you’ll find other mysteries. Don’t worry about them, just put them on the list. Once you master the first thing, go for the next thing on the list. Over time you’ll have more and more helpful things to contribute and your job will mutate into a performance job. Most performance people start as something else (like a programmer or a sys admin) and slowly move into the performance field. You don’t have to know it all day one. Actually, you never know it all and that is what makes the work interesting.


This blog is based on: The Every Computer Performance Book which is available at AmazonPowell’s Books, and on iTunes.