How To Fly

During my career I flew over a million air miles. Even though I’m just shy of seven feet tall, I mostly enjoyed the experience once I figured out a few key things about flying. I hope these insights help you. 

Attitude is everything in flying

People are attracted to (and want to help) grateful, kind, and pleasant people. Think about your own life. When you have served others, what kind of person did you bend rules and go the extra mile for? When a problem happens, it is very rare that the ungrateful, unkind, and unpleasant person gets to their destination any faster than the kind person.

Consider the alternative

Regardless of how many things go wrong on a trip, flying is so much better than a bus. Until transporter technology is perfected, flying (even with all its hassles) is really your best high-speed choice for transport.

Air travel is like prison, but in a good way

Flying (especially after 9/11) is just about a total surrender of your civil rights and any illusion of control you might have. Realize that when you fly you make a trade: You surrender almost all your rights and they move you across the planet at over 500mph. It is only a good trade if you accept both sides of the deal. If you don’t, then don’t fly because many people will suffer as you hold up the security line.

All airlines are great and all airlines stink

Amongst the people I know who have flown more than a million miles they all have a favorite airline they LOVE and an airline that they HATE. The interesting thing is that they all love/hate different airlines. Even though collectively we have lots of data points (flights) there is no consensus. These love/hate feelings are often rooted in just a few good/bad incidents. On any given day a given airline is either awful or glorious.

Airline employees

Airlines are huge companies so don’t expect perfection from all 120,000 employees. For all large groups there is often a bell-shaped curve of performance. A few do wondrous work, the vast bulge in the middle do what is expected, and a few at the other end take sadistic pleasure in creating a private hell just for you. It’s a crapshoot who you will meet and if they are having a bad/good day.

• Flight attendants have no power to change anything about the flight, but they do occasionally bring you an extra cookie.  Help them by staying in your seat during meal times and taking your seat quickly when asked to do so.
• Ticket/Gate agents have almost no power to give you extra perks, but they do have the awesome power of not offering you the help you haven’t specifically asked for. They are the most yelled at employees of any airline. Never yell at them because a flight is delayed, canceled or otherwise screwed up. It is not their fault. Although on general principles I believe in treating people nice because it is the right thing to do, I can assure you that nice people are offered more choices and are sometimes upgraded. It pays to be genuinely nice.

Connections

Almost all flights connect. Choose your connecting flights so you have options and time. If possible, always avoid a connecting flight that is the last one of the day to your destination. When booking a flight, your various flight options are usually sorted so the connecting flight that leaves as soon as possible after your first flight arrives is at the top of the list.   Often that is a tight (not much time to run from plane to plane) connection. Why sweat a tight connection when you can leave a little earlier? I personally like a four-hour connection and in the last 10 years of my business flying I never missed a connection. Not one.

It is always sunny at 35,000 feet

If you like to look out the window, the view is much better if choose the seat that is on the “shady” side of the plane – where the sun is behind you. Think about the flight direction and time of day to figure that out. The “A” seats (as in seat 23A) are on the left side of the plane.

Leave early

If (as I have heard so many people loudly proclaim) missing this flight will make you miss some critical event (interview, meeting, wedding, birth, …) then you are a fool for cutting it that close. All airlines have three major partners that they have no control over: the Federal Aviation Administration, Homeland Security, and Mother Nature. If your are traveling for a once-in-a-lifetime, super-important reason, leave two days early. Three days early if it is your wedding. Really.

When trouble strikes

When you fly, and things are not going well, there are a few key facts-of-life you must understand to rationally evaluate what your options are.

1. There is no spare plane. The cost of keeping a spare 100 million dollar plane sitting around is very high. Even at huge hub airports there is no spare plane and no spare crew waiting to fly it. If your plane breaks then either the passengers are spread out over other flights or the airline cancels some other flight and assigns the plane to your flight.
2. Flight status displays lie right up to the last minute. The airlines typically show the flight status of your flight as “on time” right up to the last minute. To get a better idea of your probability of flying on time look at the arrivals monitor for the flight that lands at the gate your flight is scheduled to depart from. Typically that flight arrives about an hour before your departure. If it’s delayed…the probability that your departing flight will be delayed goes way up.
3. Insignificant weather matters. When the airlines say “bad weather at the destination” is causing the delay sometimes it is violent weather. But most times it is just the local conditions that lower the airports overall capacity to move airplanes:
   • Unusual wind conditions can force the airport to use a set of runways that has less capacity for takeoffs and landings.
   • Visibility can be just bad enough so that they have to switch to a different set of flight rules that either further spaces out takeoffs and landings or it prevents planes from simultaneously landing on parallel runways.
4. Planes do not hurry. Once in the air, the captain of a delayed flight will often say something like “We will do what we can to make up time.” What they can do is basically nothing. Fuel is expensive and going just a little faster burns a lot more fuel. Also the difference between the cruise and max speeds, for several commonly used jet aircraft, is less than 10%. If you get in the air late, you will arrive late.

If you think about each one of the above facts of life you might see analogs in the computer performance work you do.

When you are stuck

When a huge storm shuts down the airport… accept your fate. The only rational thing to do is to ride the chaos with grace and style. Stay flexible, stay pleasant, and be helpful to others. Plan to convert this dreary experience into a great story. You can write the story of The Massive Airport Blizzard to read either:

1. I yelled at dozens of people to no effect and got home two days late.
2. I had some really interesting conversations, helped someone, made a new friend, and got home two days late.

It is your choice. Choose to be happy, because grumpy rarely works for anyone.

---

I also have many useful hints about doing computer performance work once you land at your destination in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.

---

 

While You Are Young

I’ve left the performance game to the young, the optimistic, and the hopeful.  As you boldly charge off into the acronym-littered future, I trust you will have as much fun solving problems, righting wrongs, and saving your company’s bacon as I did.

Here I am screwing around on the night shift in 1978. The machine I’m leaning on is my beloved Xerox Sigma 6… a million dollar, 1-MIP, 128k magnetic core memory beast. If I could go back in time and give this idiot some advice (other than well timed stock tips) this is what I’d tell him.

Have some cash in the bank. When opportunity comes knocking, sometimes it requires you to take some risk. I was lucky and foolish at the right times, but I had several pals that chickened-out because they had no cushion to fall back on. A buddy of mine avoided the financial risk of taking a job at a little start-up called Google. He would have been in the first 50 employees. Ouch. So I know that there are many shiny things to spend money on, but if you want to go on a few adventures, you’ve got to have 6 months of cash in the bank.

Take the extra day to see the city. If you travel a lot for work, once in a while take an extra day (yes, on your own nickel) to see the place you are visiting. I’ve flown over a million air miles. I’ve been everywhere and seen nothing. I know most major airports well, but I rarely stayed the extra day to explore on my own. It is the greatest regret of my career.

Once in a while, work a 40 hour week. It’s easy to work a lot of hours when you are young and just getting started in a career. What I’ve noticed over the years is that work expands to fill the time allotted to it. If you are regularly working 60 or 80 hours a week every once in a while take a “work-cation” and limit yourself to just 40 hours. When you have less time to work, you tend to focus and work in a more efficient way. This helps you when you go back to longer hours and the extra free time during that week can help you remember the sweetness of life.

When You Are Close To The Edge

At the Grand Canyon there are many places where you can walk right up to a cliff where, with one more step, you will fall hundreds of feet to your death. The closer you are to the edge of a cliff, the more precisely you need to know your location. In your campsite, a half-mile away, your exact location is not so critical. This is also true in performance work.

If the numbers show a resource will be 20-25% busy at peak, I would not spend more time getting a more precise version of that number. You could be off by a factor of two and the resource would most likely be fine at 40-50% busy. The closer you are to some performance limit, the more careful you have to be with your calculations and predictions.

With any prediction of future behavior there will also be some error, some uncertainly. Some of this is your fault, some of it is the fault of the person who specified the peak load to plan for, and some of it is the fault of the users who didn’t do exactly what was anticipated on that peak day.

When the boss says plan for a peak load that is two times the observed load, do what you are asked. Then, look to see if you are close to “the edge” of some performance cliff. If you are close, go back to the boss and show what you’ve found and ask: “How sure are you about your predicted peak load?”

I’ve seen many cases where, when shown how close to the edge a system would be at peak, the decision makers change their minds and give a different number to plan for. Sometimes that number is:

• Bigger because they want to buy a new stuff
• Smaller because they don’t want to spend money
• Bigger to protect the budget for next year
• Smaller because they just got new growth projections
• Different than the last number because of the crisis they are dealing with at the moment you happened to ask

It’s your job is to advise, not decide. Present your data, give your best advice, and be at peace. A business decision weighs costs, risks, politics, and the art of what is possible.

---

This sound advice came from: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.

---

 

What To Wear While Driving

This morning I spent some time on the phone with 911 after I came upon an accident. The car had veered off the road and flipped upside down in a ditch. By the time I arrived the driver had just crawled out of the wreck. He had no obvious injuries, but was a little woozy. The normal speed on this stretch of road is 50mph / 80KPH. This could have ended badly.

Wear your seatbelt, he was. 

When You Care Enough To Do Less

It’s the oldest joke in the book…

      Patient: When I do this it hurts.

      Doctor: Well don’t do that.

Sometimes performance work is not about adding hardware or tuning applications, its about doing less and doing it smarter. Send a kilobyte not a megabyte, don’t lock all the records when you don’t need to, etc.

For example, what you put in the files served by your website has a huge impact on performance that no amount of server-side hardware can overcome because you don’t control all the computers/networks between you and the enduser. Many times the only way to fix website response time problems is to send less stuff in a smarter way.

I recently ran across Zoompf.com which has a nice tool to analyze your website and make helpful recommendations to speed it up. They do a good job of explaining why the changes they recommend are important and further provide helpful references to more information about each recommendation.

To avoid mistakes you haven’t made yet, you might also want to read a wonderful little book called High Performance Web Sites by Steve Souders. It points out a lot of small changes that can make a big difference in website performance.

Mostly companies prefer to throw hardware at performance problems, rather than adjust applications, algorithms, or outputs, because it is seen as the low-risk path. Sometimes that works, but sometimes the right thing to recommend is: “don’t do that.”

---

After you read Steve’s book, try mine: The Every Computer Performance Book at  Amazon, Powell’s Books, and on iTunes.

---

My Worst Day At Work

A key customer was having a performance problem and our company had assembled a team to fly down and work on it. I was part of that team and this was going to turn into my worst day at work – ever.

This customer had developed their application on our proprietary operating system and it worked great. However, we had recently shipped a UNIX OS on that same hardware and the customer had ported their application to it and it ran as slow as mud.
The way they saw it: Same hardware, same application, different OS, bad performance. This must be the vendor’s fault. The way we saw it: Applications ported to new operating systems often have performance problems in the same way a human from Earth would have to work extra hard to make it on a different planet.

So we arrived onsite and the initial one hour meeting with Mr. Big Cheese and his henchmen was deeply unpleasant and accusatory. Then we spent the rest of the day in a conference room logged in to the test system looking for some solution. At no time were we ever left alone. There was always at least one of his henchmen there asking us what we were doing and often unhelpfully commenting on our efforts: “We already tried that.”, “Anyone can see that this is not the problem.”, and “You’re wasting time.”

As the day drew to a close we had a final meeting with Mr. Big Cheese and he was not interested in what little performance-enhancing crumbs we had found. He used that hour to imply we were all idiots and demand that our company send down a real “UNIX kernel hacker”. Over and over, he made it clear that only a “UNIX kernel hacker” could solve this problem. We were not that, so he told us to go.

This company had made two key mistakes with us: One: They were jerks to the people who had come to help them. So we did what we were required to do, but not what we could have done if we wanted to call in favors, bend rules, and do heroic things. Be nice to the people who come to help you, even if they are idiots. Why? Because they go back to that company and advocate for you and spread the word that you are worthy of heroic, rule-bending efforts. Two: They never left us alone. We were never free to work as a group for fear that we’d say or do something stupid. If experts fly in, give them private space to call other experts at headquarters and talk amongst themselves. They will get to any possible solution much faster.

We left at the end of the day as a group and walked to nearby restaurant. We ordered a round of beers. When they came, I picked up my glass and chugged the whole beer in a few seconds, something I’d not done since my college days.

Everyone was a little shocked. I set the glass down and slyly said: “It doesn’t do you any f#@king good in the glass.” That broke the ice and to this day we laugh, and laugh, and laugh about what a really horrible day that was and the fact that we are not, and never will be, “UNIX kernel hackers”.

Alcohol is not a solution to man’s problems, but laughter is.

A Career Built On Kindness

VERY few people go to college specifically with the goal to be a performance guru.

They start out holding some other job (like sys admin or programmer) and then, by circumstance or desire, slowly move into the performance world.

They learn some performance fundamentals, master a performance tool, notice patterns in the metered data, and slowly pickup the detailed tech-specific knowledge.  They take some chances, make some performance predictions and suggestions and… Voilà, they become a performance guru!

If you are starting your journey, welcome. Keep reading this blog, because I am writing it just for you. Learn, play, explore, and grow.

My best fundamental bit of advice is to be relentlessly kind to those around you. You need their help and can not do this on your own. Once you know something useful, share it. When you can be helpful to others, do it. Be easy to work with. Be kind.

These acts of kindness will help others, but they also help you. First of all, in kindness we find our freedom. Your boss can order you to do many things, but you decide to be kind and in that decision is a freedom that feels very good. Kindness can also be the source of new friendships. Friends will often give you more help than they are required to give because they have seen you do the same. Friends, who move on to other companies, often call you up and give you the inside track on great jobs opportunities.

In my high tech career every single job I ever had came to me through a friend. I was offered these jobs even though I was often missing a key skill-set. This is not because I’m super-smart, or beautiful, it’s because I am kind, helpful, and easy to work with.

Be kind. It will serve you well, and make the world a much nicer place.

Interactive Computer Latency Numbers Through Time

Go here: http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

Grab the slider at the top of the screen and see how latency values for common computer tasks have changed starting in 1991 and projected out to 2020. To me, the precise values aren’t as interesting as seeing how the performance battles programmers fight change over time.

Sherman, set the wayback machine to…

Modeling Projects Begin With a Question

In the beginning there is an idea, a goal, a mandate, or a proposal that leads to a performance question.

If you can answer that question though simple performance measurement or capacity planning, then do it and be done.

If this is a new application (and thus no metering is possible), or your computing world is changing radically, then you may need to build a model.

Some ideas take very little modeling to shoot down…

The first step to doing that is to really understand the question. Start with what they give you, and then ask very picky questions to clarify:

Boss: Will this plan to consolidate systems work?

You: At our seasonal peak load?
Boss: At the seasonal peak.

You: How much should I add to last year’s peak to scale for this years peak?
Boss: Plan on adding 10%.

You: How sure are you of that number?
Boss: Pretty sure, plus or minus 10%.

You: So I should plan for 11% (10% + (10% * 10%)?
Boss: No. To be safe, plan for last year’s peak plus 20%.

You: Is there any money available new hardware?
Boss: Not a penny.

Ask all the key players for clarification and additional information in both positive and a negative ways: “What do you want? What must we avoid?” Keep at it until you really understand the critical success factors such as:

• What success looks like in terms of throughput and response time
• Constraints on budget and time that will limit your options for achieving the goals
• Legal and availability concerns that will limit the configuration options
• What is politically and bureaucratically possible

To be clear, this is not a license to waste people’s time by acting like a three year old asking “Why?” over and over and over. Keep your goal in mind, which is a clearly defined question to answer.

Brainstorm, Refine and Choose

To build a model to find an answer to a question you first have to guess the answer; then you can build a model to see if it is really the answer. To guess the answer, you first brainstorm a list of possible answers and then thin that list down to the best candidates.

Create a List Of Possible Solutions

Start with that question, and your knowledge of your computing world, and come up with a couple of workable solutions. This is the intuitive, creative part of modeling, and it is very much like writing a hit song.  There is no step-by-step procedure that will always end up with a great song. However, there are guidelines that will improve your odds of creating a workable solution to your question:

• At first, don’t be judgmental. Any idea that, in any way, answers any part of the question is a good one at this point.
• As you are brainstorming, be sure to write things down.  The saddest thing is to watch someone struggle to remember, like you do with a fading dream, the key insight that made an idea work.
• Now sift the ideas you have based on key limits and demands of your question.  For example, if there is no money for new hardware, then set those ideas requiring new hardware aside, but do not discard them. You may need these ideas later.
• If you have an abundance of ideas that may answer the question, then sort them based on simplicity, risk, and total cost, and take the top three.
• If you have no ideas left that may answer the question, then you should ask the people in charge for guidance as to what in the question can be loosened, ignored, or worked around.
• If you are stuck, go back to those failed ideas you set aside earlier. See if any of them will work for you now.

Refine The Solutions

You should now have a few possible solutions that might work. Before you go through the work to build models, refine your list of solutions by using a bit of simple capacity planning.

To answer the whole question you might need a model, but to figure out that a proposed solution doesn’t work can sometimes be seen with simple capacity planning of some small part of the whole question.  That sad news might get you to abandon the idea altogether, or it might lead to a modification of the original idea to make it better.  Use simple, rapid tools to find problems in your plan before wasting more time modeling a solution that can never work. Let’s look at an example.

We are planning to add the workload generated by the new XYZ transaction to a key system. There is no test data yet, but reasonable people agree that, due to its complexity, the XYZ transaction should take 2.5 times the CPU resources of the current BBQ transaction on that key system.

The BBQ transaction consumes 20ms of CPU per transaction, so the XYZ transaction will most likely use (2.5 * 20ms)= 50ms of CPU.  At peak load you are planning for 100 BBQ TX/sec and estimating that there will be 20 XYZ TX/sec. On this key system, just these two transactions will use three seconds per second of CPU as you can see below.

If that machine is a two CPU machine, then this idea is dead in the water.

Seek Out Missing Information

There will come a point where you know most of what you need to know but are missing a key bit of information on throughput, service time, utilization, etc. Here are a few key performance laws that will help you find the missing piece.

Little’s Law: Mean#InSystem = MeanResponseTime * MeanThroughput (Example)

Utilization Law: MeanUtilization = ServiceTime * MeanThroughput (Example)

Service Demand Law: ServiceTime = MeanUtilization / MeanThroughput

These equations can always be rearranged mathematically (e.g. if A=B*C then B=A/C and C = A/B) to find the thing you are missing.

Double Check Your Work

As you are exploring these solutions, be sure to keep looking for a number that rubs you the wrong way or an assertion you can’t stop thinking is wrong.  Keep asking yourself questions like these.

• Will this work at peak load?
• Can that number really be that low?
• Am I using the right metering data?
• Did I mess up the units in this calculation?

A Note To Beginners

If this is your first time doing this, you are starting from scratch. Don’t make the mistake of doing that again next year. Save your report, your notes, and any tools you created to help you write this report.  Next time, you’ll have a good base to start with and build on. This will save you time and make sure you don’t forget things. As the time between peaks goes by, you’ll usually find new things to add to the model. Write them down, build them into your performance tools, and file them with all your other modeling data so you don’t forget them when you plan the next peak.

---

“You’ll do this again. Always take time to make things easier
for your future self.”  – Bob’s Tenth Rule of Performance Work

---

For more info on performance modeling see: The Every Computer Performance Book  on Amazon and iTunes.

---

 

Standards

The brilliant comic XKCD perfectly describes my feelings about standards in performance work with this simple cartoon:

There are many paths to the top of the performance mountain. If, whatever you believe is the right way to do performance work, gets you to the top then go with that.

But, while climbing the mountain, keep your eyes open and notice what the other climbers are doing. If you see some cool climbing trick, learn it. Whenever possible, share what you know and help those that are struggling. If you are struggling, that is often natures way of telling you to open your mind other options, opinions, pathways, and climbing techniques.

It is not about the dogma, its about getting the work done and keeping the users happy.