All Good Things…

Dear readers,

This my last post to this blog. Why?  Because I’ve said all I have to say about performance. I could make up some new buzzwords, or push a new paradigm, and thus have an excuse to keep blogging. However, I feel the best action is to value quality over bafflegab and just stop here.

coverThe Every Computer Performance Book is the sum of what I had to pass on to the next generation and this blog contains a lot of what is in that book, plus a few extra goodies. I wrote the book to give back useful things that will always be true about performance work. I wrote the blog to help you find the book and for those that prefer their information in a digital form.

The blog entries are carefully tagged and categorized. If you just found this blog, I’d start at the first entry and read it forward.

I love to hear from my readers. If you care to write you can find me on the web, but know that above all, I love getting postcards and letters.  If you are so moved, you can write to me at the address below. This is my high-fun, part-time, post-retirement job.

Bob Wescott – Tour Guide
c/o Ben & Jerry’s Factory Tour
1281 Waterbury-Stowe Road
Waterbury, VT 05676
USA

My computer performance career was a grand adventure, but all good things must come to an end.

"So, five-card stud, nothing wild... and the sky's the limit."

StarTrek: The Next Generation “All Good Things…” Series Finale, Final Scene

 

A Thousand Thanks

Late last night some nice person bought the 1000th copy of my book. Wow!1000thankyous

I’d like to thank each and every one of my readers. When I started this book I did it to give back to the field of performance analysis. I’d hoped to sell 500 copies. Why 500? Since I can’t find everyone who might need to read this book, I decided to be happy with 500 as that is the number of students I’d have in an average teaching year when I was teaching the performance course at Stratus. Apparently I aimed too low.

The best part of this experience has been hearing from my readers. Their personal messages to me, comments on this blog, and recommendations of my book have been a joy to read. Yes, even the comment that kindly pointed out my typo in chapter eight where I wrote “pubic speaking” when I meant “public speaking.”

 

How To Fly

During my career I flew over a million air miles. Even though I’m just shy of seven feet tall, I mostly enjoyed the experience once I figured out a few key things about flying. I hope these insights help you. planes

Attitude is everything in flying

People are attracted to (and want to help) grateful, kind, and pleasant people. Think about your own life. When you have served others, what kind of person did you bend rules and go the extra mile for? When a problem happens, it is very rare that the ungrateful, unkind, and unpleasant person gets to their destination any faster than the kind person.

Consider the alternative

Regardless of how many things go wrong on a trip, flying is so much better than a bus. Until transporter technology is perfected, flying (even with all its hassles) is really your best high-speed choice for transport.

Air travel is like prison, but in a good way

Flying (especially after 9/11) is just about a total surrender of your civil rights and any illusion of control you might have. Realize that when you fly you make a trade: You surrender almost all your rights and they move you across the planet at over 500mph. It is only a good trade if you accept both sides of the deal. If you don’t, then don’t fly because many people will suffer as you hold up the security line.

All airlines are great and all airlines stink

Amongst the people I know who have flown more than a million miles they all have a favorite airline they LOVE and an airline that they HATE. The interesting thing is that they all love/hate different airlines. Even though collectively we have lots of data points (flights) there is no consensus. These love/hate feelings are often rooted in just a few good/bad incidents. On any given day a given airline is either awful or glorious.

Airline employees

Airlines are huge companies so don’t expect perfection from all 120,000 employees. For all large groups there is often a bell-shaped curve of performance. A few do wondrous work, the vast bulge in the middle do what is expected, and a few at the other end take sadistic pleasure in creating a private hell just for you. It’s a crapshoot who you will meet and if they are having a bad/good day.

  • Flight attendants have no power to change anything about the flight, but they do occasionally bring you an extra cookie.  Help them by staying in your seat during meal times and taking your seat quickly when asked to do so.
  • Ticket/Gate agents have almost no power to give you extra perks, but they do have the awesome power of not offering you the help you haven’t specifically asked for. They are the most yelled at employees of any airline. Never yell at them because a flight is delayed, canceled or otherwise screwed up. It is not their fault. Although on general principles I believe in treating people nice because it is the right thing to do, I can assure you that nice people are offered more choices and are sometimes upgraded. It pays to be genuinely nice.

Connections

Almost all flights connect. Choose your connecting flights so you have options and time. If possible, always avoid a connecting flight that is the last one of the day to your destination. When booking a flight, your various flight options are usually sorted so the connecting flight that leaves as soon as possible after your first flight arrives is at the top of the list.   Often that is a tight (not much time to run from plane to plane) connection. Why sweat a tight connection when you can leave a little earlier? I personally like a four-hour connection and in the last 10 years of my business flying I never missed a connection. Not one.

It is always sunny at 35,000 feet

If you like to look out the window, the view is much better if choose the seat that is on the “shady” side of the plane – where the sun is behind you. Think about the flight direction and time of day to figure that out. The “A” seats (as in seat 23A) are on the left side of the plane.seating

Leave early

If (as I have heard so many people loudly proclaim) missing this flight will make you miss some critical event (interview, meeting, wedding, birth, …) then you are a fool for cutting it that close. All airlines have three major partners that they have no control over: the Federal Aviation Administration, Homeland Security, and Mother Nature. If your are traveling for a once-in-a-lifetime, super-important reason, leave two days early. Three days early if it is your wedding. Really.

When trouble strikes

When you fly, and things are not going well, there are a few key facts-of-life you must understand to rationally evaluate what your options are.

  1. There is no spare plane. The cost of keeping a spare 100 million dollar plane sitting around is very high. Even at huge hub airports there is no spare plane and no spare crew waiting to fly it. If your plane breaks then either the passengers are spread out over other flights or the airline cancels some other flight and assigns the plane to your flight.
  2. Flight status displays lie right up to the last minute. The airlines typically show the flight status of your flight as “on time” right up to the last minute. To get a better idea of your probability of flying on time look at the arrivals monitor for the flight that lands at the gate your flight is scheduled to depart from. Typically that flight arrives about an hour before your departure. If it’s delayed…the probability that your departing flight will be delayed goes way up.
  3. Insignificant weather matters. When the airlines say “bad weather at the destination” is causing the delay sometimes it is violent weather. But most times it is just the local conditions that lower the airports overall capacity to move airplanes:
    • Unusual wind conditions can force the airport to use a set of runways that has less capacity for takeoffs and landings.
    • Visibility can be just bad enough so that they have to switch to a different set of flight rules that either further spaces out takeoffs and landings or it prevents planes from simultaneously landing on parallel runways.
  4. Planes do not hurry. Once in the air, the captain of a delayed flight will often say something like “We will do what we can to make up time.” What they can do is basically nothing. Fuel is expensive and going just a little faster burns a lot more fuel. Also the difference between the cruise and max speeds, for several commonly used jet aircraft, is less than 10%. If you get in the air late, you will arrive late.

If you think about each one of the above facts of life you might see analogs in the computer performance work you do.

When you are stucksnow

When a huge storm shuts down the airport… accept your fate. The only rational thing to do is to ride the chaos with grace and style. Stay flexible, stay pleasant, and be helpful to others. Plan to convert this dreary experience into a great story. You can write the story of The Massive Airport Blizzard to read either:

  1. I yelled at dozens of people to no effect and got home two days late.
  2. I had some really interesting conversations, helped someone, made a new friend, and got home two days late.

It is your choice. Choose to be happy, because grumpy rarely works for anyone.


I also have many useful hints about doing computer performance work once you land at your destination in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

While You Are Young

I’ve left the performance game to the young, the optimistic, and the hopeful.  As you boldly charge off into the acronym-littered future, I trust you will have as much fun solving problems, righting wrongs, and saving your company’s bacon as I did.Bob Wescott in his first machine room with the Xerox Sigma 6 computer

Here I am screwing around on the night shift in 1978. The machine I’m leaning on is my beloved Xerox Sigma 6… a million dollar, 1-MIP, 128k magnetic core memory beast. If I could go back in time and give this idiot some advice (other than well timed stock tips) this is what I’d tell him.

Have some cash in the bank. When opportunity comes knocking, sometimes it requires you to take some risk. I was lucky and foolish at the right times, but I had several pals that chickened-out because they had no cushion to fall back on. A buddy of mine avoided the financial risk of taking a job at a little start-up called Google. He would have been in the first 50 employees. Ouch. So I know that there are many shiny things to spend money on, but if you want to go on a few adventures, you’ve got to have 6 months of cash in the bank.

Take the extra day to see the city. If you travel a lot for work, once in a while take an extra day (yes, on your own nickel) to see the place you are visiting. I’ve flown over a million air miles. I’ve been everywhere and seen nothing. I know most major airports well, but I rarely stayed the extra day to explore on my own. It is the greatest regret of my career.

Once in a while, work a 40 hour week. It’s easy to work a lot of hours when you are young and just getting started in a career. What I’ve noticed over the years is that work expands to fill the time allotted to it. If you are regularly working 60 or 80 hours a week every once in a while take a “work-cation” and limit yourself to just 40 hours. When you have less time to work, you tend to focus and work in a more efficient way. This helps you when you go back to longer hours and the extra free time during that week can help you remember the sweetness of life.

Three Tools You Should Build

Given that it is a good idea to keep an eye on performance all the time, there are lots of companies that only allow you pay periodic attention to performance. They focus on it when there is a problem, or before the annual peak, but the rest of the year they give you other tasks to work on.

toolsThis is a lot like my old job in Professional Services – A customer has a problem, I fly in, find the trouble, and then don’t see them until the next problem crops up.

To do that job I relied on three tools that I created for myself and that you might start building to help you work on periodic performance problems.

Three Tools

List All – The first tool would dig through the system and list all the things that could be known about the system: config options, OS release, IO, network, number of processes, what files were open, etc. The output was useful by itself as now I had looked in every corner of the system and knew what I was working on. Several times it saved me days of work as the customer had initially logged me into the wrong system. It always made my work easier as I had all the data I needed, in one place, conveniently organized, and in a familiar order.

Changes – If I’d been to this customer before, this tool allowed me to compare the state of the system with the previous state. It just read through the output of the List All I’d just done and compared it with the data I collected on my last visit. Boy, was this useful as I could quickly check the customer’s assurance that “Nothing had changed since my last visit.” I remember the shocked look on the customer’s face when I asked: “Why did you downgrade memory?”

Odd Things – Most performance limiting, or availability threatening, behavior is easy to spot. But for any OS, and any application, there are some things that can really hurt performance that you have to dig for in odd places with obscure meters. These are a pain to look for and are rare, so nobody looks for them. Through the years as I discovered each odd thing, I would write a little tool to help me detect the problem and then I’d add that tool to the end of my odd things tool.  I’d run this tool on every customer system I looked at and, on occasion, I would find something that surprised everyone: “You haven’t backed up this system in over a year.” or solved a performance problem by noticing a foolish less than optimal configuration choice.

With most everything happening on servers somewhere in the net/cloud these days, knowing exactly where you are and what you’ve got to work with is important. Being able to quickly gather that data in a matter of minutes allows you to focus on the problem at hand confident that you’ve done a through job.

Stone_SoupAll three of these tools were built slowly over time. Get started with a few simple things. The output of all three is just text – no fancy GUI interface or pretty plots are required. When you have time, write the code to gather the next most useful bit of information and add that to your tool.

Just like the old folk story of stone soup, your tools will get built over time with the contributions of others. Remember to thank each contributor for the gifts they give you and share what you have freely with others.


Other useful hints can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

 

The Five Minute Rule

I used to travel to companies to work on their performance problems. Once I arrived, and we had gone through the initial pleasantries, I would ask them a simple question:

How busy is your system right now?

If the person I asked had a ballpark estimate that they quickly confirmed with a meter, I’d know that whatever problem they called me to solve would be an obscure one that would take some digging to find. If they had no idea, or only knew the answer to a resolution of an entire day, then I was pretty sure that there would be plenty of performance-related ugliness to discover. This little question became the basis for a rule of thumb that I used my entire career.  five

The less a company knows about the work their system did in the last five minutes, the more deeply screwed up they are.

Your job is likely different than mine was, but there is a general truth here. If the staff can easily access data that tells them in the last few minutes how much work is flowing through the system (transactions, page views, etc.) and how the system is reacting to that work (key utilization numbers, queue depths, error counts, etc.), then they have the data to isolate and solve most common performance problems. Awareness is curative. The company will solve more of it own performance problems, see future performance problems coming sooner, and spend less money on outside performance experts.

 


Other rules of thumb can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

When Tension Is High, Start With Some Good News

Sometimes the success or failure of your company hangs on your results. When the stakes are high, you might want to think about going to the movies. Specifically an old-school, action movie, like Indiana Jones.

A big reason audiences enjoy an action movie is that they are reasonably sure from the onset that they will like the ending. They are sure that the hero will triumph, and the wrongs will be righted.

There is a temptation to use a dramatic style when presenting the results of your work because you naturally want to tell a story that builds in excitement and drama and finishes with thunderous applause. That is a fine thing to do, but it works much better if you tell them very early in the presentation that all will be well. Then the audience can relax and enjoy the ride. So start your talk with something like this:

With the current configuration, we will not be able to handle the upcoming peak. However, I’ve identified the bottlenecks and I have workarounds to propose for all of them. Let me show you what I’ve found.

I have seen presentations, without this early calming statement, go badly. When the presenter was about half way through the list of all the serious problems ahead some participant will start angrily demanding something like:

Are we screwed?!?
Please Tell Me There Is a Fix For Some Of This?!?

This is not what you want. Whenever possible, lead with the good news that there is a solution to all the problems you have discovered. When you don’t have a solution for a problem, at least tell them that you’ve found the root/cause of the problem they are currently having. People need hope.


Other helpful hints on presenting your results, as well as capacity planning and load testing, can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

Judge Not

When you present your performance results, you are not a superhero striking down evil. You are not there to make yourself look big by making others feel small. You are not there to judge.

nosuper noyellnojudge

You are a member of the team, dispassionately presenting well-checked information and potential solutions for problems. Stick to the facts, and leave the judgments to others.

I have seen presentations where the speaker delivered the bad news in a mocking and sometimes directly insulting way that hurt group cohesion and deeply offended people in front of their peers. That approach did not aid in answering the question, but it did unleash a wave of back-stabbing and other bad behavior. Every time I have seen someone be intentionally cruel or hurtful to a co-worker, it has not worked out well for them, especially in the long run.

Care Deeply

The single biggest “trick” I use in every talk I’ve every given is to care about the people in the audience. This is not a trivial closing thought. This is a core truth. You can tell when someone cares about you, and you can tell when they don’t. You naturally give more attention and show more compassion to people who care about you.

pals

When you construct the presentation and in the time just before you start to give it, let yourself focus on deeply caring about the audience. Care about their needs and concerns. Care about the fact that they have been sitting in meetings for two hours before you started. Care about them as people and co-workers. Caring connects you to them in a powerful and positive way. If you honestly care, they can’t help but listen.


Other helpful hints on doing performance work, as well as capacity planning and load testing, can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.


 

When Does The Warmup End?

Unless you are focused on optimizing the time to restart an application, the meters you might collect at application start are pretty useless as things are warming up. The first few transactions will find the disks nearly idle, queues empty, locks unlocked, nothing useful in cache, etc. As the work flows in, queues will build, buffers will fill, and the application will settle down after a short period.warmup

There have been many math geniuses that have spent years trying to find a mathematical way to know when the warm-up period is over and you can start believing the data. To date there is no good mathematical answer. You just have to eyeball it. Fire up the application and let it run. Your eye will clearly note if, and when, the response times and utilizations stabilize. Ignore the data during the warm-up period.


Other helpful hints can be found in: The Every Computer Performance Book which is available at Amazon, B&N, or Powell’s Books. The e-book is on iTunes.