When You Care Enough To Do Less

doctor_groucho
It’s the oldest joke in the book…

      Patient: When I do this it hurts.

      Doctor: Well don’t do that.

Sometimes performance work is not about adding hardware or tuning applications, its about doing less and doing it smarter. Send a kilobyte not a megabyte, don’t lock all the records when you don’t need to, etc.

For example, what you put in the files served by your website has a huge impact on performance that no amount of server-side hardware can overcome because you don’t control all the computers/networks between you and the enduser. Many times the only way to fix website response time problems is to send less stuff in a smarter way.

I recently ran across Zoompf.com which has a nice tool to analyze your website and make helpful recommendations to speed it up. They do a good job of explaining why the changes they recommend are important and further provide helpful references to more information about each recommendation.

To avoid mistakes you haven’t made yet, you might also want to read a wonderful little book called High Performance Web Sites by Steve Souders. It points out a lot of small changes that can make a big difference in website performance.

Mostly companies prefer to throw hardware at performance problems, rather than adjust applications, algorithms, or outputs, because it is seen as the low-risk path. Sometimes that works, but sometimes the right thing to recommend is: “don’t do that.”


After you read Steve’s book, try mine: The Every Computer Performance Book at  AmazonPowell’s Books, and on iTunes.


Working A Little Harder

Sometimes you have to work a little harder than you’d like to find the data you need to solve a performance problem or answer a performance question. Sometimes you have no elegant tool and just have to jury-rig some ugly collection of hacks to get what you need.data collection

 

The photographer pictured above was not comfortable, safe, or delighted to have this assignment at the this moment. However, I bet to the end of his days he told this story with great pride. If you don’t have everything you need in its most convenient place, just at the right time… do what you can, with what you got. Yeah, it’s a pain, but it is also the start of a great story.


For more specific hints on getting performance work done see: The Every Computer Performance Book at  AmazonPowell’s Books, and on iTunes.


The Hidden Agenda 

In all talks there are two agendas: surface and hidden. The surface agenda is the subject of your talk.  It is in your slides. You say it out loud. The hidden agenda (e.g., get new hardware, influence the overall design to your liking, keep your job) is why you are really there. It is never spoken out loud.  It is not on the slides, but it is important to you. hidden

When you are presenting, if things start going badly and it is not clear what to do next, remember your hidden agenda. This can keep you from committing career suicide over a point you later realize was small potatoes. To be clear… Even when serving your hidden agenda, you have to tell the truth, be clear, and follow the data to its fortunate (or unfortunate) conclusion. In a tough spot, your hidden agenda can help you with emphasis, focus, balance between competing ideas, and when to gracefully concede to your opponent.

My Worst Day At Work

A key customer was having a performance problem and our company had assembled a team to fly down and work on it. I was part of that team and this was going to turn into my worst day at work – ever.

This customer had developed their application on our proprietary operating system and it worked great. However, we had recently shipped a UNIX OS on that same hardware and the customer had ported their application to it and it ran as slow as mud.
The way they saw it: Same hardware, same application, different OS, bad performance. This must be the vendor’s fault. The way we saw it: Applications ported to new operating systems often have performance problems in the same way a human from Earth would have to work extra hard to make it on a different planet.

So we arrived onsite and the initial one hour meeting with Mr. Big Cheese and his henchmen was deeply unpleasant and accusatory. Then we spent the rest of the day in a conference room logged in to the test system looking for some solution. At no time were we ever left alone. There was always at least one of his henchmen there asking us what we were doing and often unhelpfully commenting on our efforts: “We already tried that.”, “Anyone can see that this is not the problem.”, and “You’re wasting time.”

As the day drew to a close we had a final meeting with Mr. Big Cheese and he was not interested in what little performance-enhancing crumbs we had found. He used that hour to imply we were all idiots and demand that our company send down a real “UNIX kernel hacker”. Over and over, he made it clear that only a “UNIX kernel hacker” could solve this problem. We were not that, so he told us to go.

This company had made two key mistakes with us: One: They were jerks to the people who had come to help them. So we did what we were required to do, but not what we could have done if we wanted to call in favors, bend rules, and do heroic things. Be nice to the people who come to help you, even if they are idiots. Why? Because they go back to that company and advocate for you and spread the word that you are worthy of heroic, rule-bending efforts. Two: They never left us alone. We were never free to work as a group for fear that we’d say or do something stupid. If experts fly in, give them private space to call other experts at headquarters and talk amongst themselves. They will get to any possible solution much faster.

beerWe left at the end of the day as a group and walked to nearby restaurant. We ordered a round of beers. When they came, I picked up my glass and chugged the whole beer in a few seconds, something I’d not done since my college days.

Everyone was a little shocked. I set the glass down and slyly said: “It doesn’t do you any f#@king good in the glass.” That broke the ice and to this day we laugh, and laugh, and laugh about what a really horrible day that was and the fact that we are not, and never will be, “UNIX kernel hackers”.

Alcohol is not a solution to man’s problems, but laughter is.

A Career Built On Kindness

John_BlutarskyVERY few people go to college specifically with the goal to be a performance guru.

They start out holding some other job (like sys admin or programmer) and then, by circumstance or desire, slowly move into the performance world.

They learn some performance fundamentals, master a performance tool, notice patterns in the metered data, and slowly pickup the detailed tech-specific knowledge.  They take some chances, make some performance predictions and suggestions and… Voilà, they become a performance guru!

If you are starting your journey, welcome. Keep reading this blog, because I am writing it just for you. Learn, play, explore, and grow.

My best fundamental bit of advice is to be relentlessly kind to those around you. You need their help and can not do this on your own. Once you know something useful, share it. When you can be helpful to others, do it. Be easy to work with. Be kind.

These acts of kindness will help others, but they also help you. First of all, in kindness we find our freedom. Your boss can order you to do many things, but you decide to be kind and in that decision is a freedom that feels very good. Kindness can also be the source of new friendships. Friends will often give you more help than they are required to give because they have seen you do the same. Friends, who move on to other companies, often call you up and give you the inside track on great jobs opportunities.

In my high tech career every single job I ever had came to me through a friend. I was offered these jobs even though I was often missing a key skill-set. This is not because I’m super-smart, or beautiful, it’s because I am kind, helpful, and easy to work with.

Be kind. It will serve you well, and make the world a much nicer place.

kindness

No Bad Surprises In Public

Never plan to surprise the person responsible for a problem in a public meeting. The goals of performance work are measured in response time and throughput, not in how much drama you create when you point your accusing finger at the unsuspecting culprit.    drama

When you locate a problem, the first person you should find is the person who is responsible for that part of the computing world, and discuss that problem with him or her. Why? That person may know a lot more about that part of your computing world than you do, and may have further insights as to the root cause and the reason(s) why things are done this way. Often, I find that when I privately share my concerns and ask for help in crafting a list of possible solutions, that person is quite willing to be helpful.

I have made the mistake of not involving the person I believed was responsible for the problem and have suffered these consequences, usually in this exact order:

  1. The person responsible for that part of the computing world got angry and defensive and worked relentlessly to tear down my work and credibility.
  2. That person points out my ignorance and further points out the real problem is caused by some other part of the computing world owned by a different person. Now there are two angry people in the room.
  3. Now the manager becomes angry with me for creating tension among the staff.

It always works better when I talk to the responsible person privately well before I write up my recommendations. We look at the problem and explore solutions. Then I can walk into the meeting and say something like: “The problem is in this subnet. With the help of your networking guru, we have a few ideas on how to improve the situation.


For more hints on presenting your work see: The Every Computer Performance Book at  AmazonPowell’s Books, and on iTunes.


 

The Truth That Is Hard To Swallow

There will be times when the meters and your expert analysis show that bad things are in the future. The company is looking at big changes, or spending massive amounts of money, or enduring very disruptive fixes. bad news

I have seen decision makers on multiple occasions, when presented with unrelenting bad news, simply reduce the scaling factor or lower the desired goals so that the problem simply evaporates. The essential nature of business is risk-and-reward. Sometimes a business will just have to hope that next year will be better, but sometimes what you are watching is a very human reaction to staggeringly bad news.

When people are presented with bad news, that contains no possibility of escape, a significant fraction of them will go into denial regardless of the evidence. The key to greater acceptance of your message is to present the bad news with a possible solution, or at least a way of improving the situation.

The XYZ server will run out of CPU at peak, but
we can mitigate some of that that by…

Interactive Computer Latency Numbers Through Time

Go here: http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html

Grab the slider at the top of the screen and see how latency values for common computer tasks have changed starting in 1991 and projected out to 2020. To me, the precise values aren’t as interesting as seeing how the performance battles programmers fight change over time.

Sherman, set the wayback machine to…wayback

Tuning Chocolate Chip Cookies

In most computer performance work, big changes (both good and bad) can result from small adjustments. In the blog posts below, you’ll find someone who has done some excellent experimental science as she adjusts the tuning on a chocolate chip cookie recipe. She starts with a control recipe (Nestle Tollhouse Chocolate Chip Cookie) and she varies one thing at a time and makes a batch. She shows her work, details the techniques, and keeps excellent notes.

So why is a blog on computer performance linking to a cooking blog? Because the things I write about here are the things that will always be true on any computer system. Good experimental technique will always be useful and I’m pretty sure that a huge fraction of people will always gratefully accept a well made chocolate chip cookie. Her work is a delicious example of good science, which is an essential ingredient of all computer performance work.

The Ultimate Guide to Chocolate Chip Cookies – Part 1
The Ultimate Guide to Chocolate Chip Cookies – Part 2
The Ultimate Guide to Chocolate Chip Cookies – Part 3
The Ultimate Guide to Chocolate Chip Cookies – Part 4

Cookie-Testing02TEXT

 

Designing A Performance Presentation

This is a collection of hints on presenting performance results that have worked for me throughout the years as I’ve presented my results to both friendly and skeptical audiences of managers, technical staff, and executives all the way up to the CIO/CEO level.  This is not generic advice on public speaking. You can find that elsewhere.

seerTo Reveal The Future…

When presenting your results, in many ways you are like the Crystal Seer. Perhaps the turban and the crystal ball would be a little over the top for your presentation in conference room three, but overall this is not a bad metaphor.

When doing performance work, you are uncovering a hidden truth few can see, and predicting the future.

We have all seen a poorly explained truth go down in flames and a beautifully told lie carry the day. If the inmates are running the asylum where you work, then they are most likely very good at presenting their very bad ideas.

How clearly and convincingly you present your results determines how successful you are.

Proof

As Carl Sagan once said, “Extraordinary claims require extraordinary evidence.” Look at your results and conclusions and ask yourself how your audience will react.

The more disruptive, shocking, or expensive your conclusions and recommendations are, the more backup data you need and the more effort you want to expend in making an airtight case. If you are claiming bacon is good for you, then you will have an easier time with the National Pork Producers Board than with a group of vegan cardiologists.

However, just because you have 30 backup slides for your shocking revelation, doesn’t mean you need to show them all.  Pay attention to your audience. Once you’ve convinced them, forget the remaining 24 backup slides, and move on to your next point.

The Nature of Truth

pinocchio
When preparing to present your report, there can be tremendous pressure to lie. Your work may help justify a purchase everyone wants to make or force unpleasant changes that no one wants to endure. The politics can get very serious.

First and foremost, stick tightly to the data you collected. It is the truth. Everything you do, say, and recommend flows from it. Never change that data. Never cherry pick the “good” numbers. Never ignore the bad numbers. If the powers that be order you to change that data, then start looking for another job because this is not the place anyone wants to work.

Be open to other interpretations of the data. If they do not violate the laws of physics, or performance, they may be valid. A device being 50% busy is a fact. What that fact means depends on the question at hand and the business realities that you have to live within. I’ve done performance work at companies where 30% busy on a peak day was a crisis and others where 95% busy was the norm.  Both companies were doing wildly different things with their machines, but they, and their customers, were quite content with the performance they were getting.

Simplify

You’ve done weeks of monitoring, calculation, and testing, and now you’ve got to explain your work to people who have been (for the most part) blissfully ignorant of your efforts and struggles. There is the natural tendency to show the detail and talk at length about how hard you worked. Don’t do that.

diamondIt takes an incredible amount of ingenuity, work, skill, and craftsmanship to lift a raw diamond out of the Earth and craft it into a sparkling gem. The same is true of your work on this performance project. In both cases, the end product is prized for its clarity. That clarity comes from the internal structure, the lack of flaws, and the raw material you discarded. When writing and presenting be a minimalist.

You will be presenting to people who have natural limits. Most people, as a rule, are not that good at holding several numbers in their head simultaneously. People also have a finite ability to give a damn about what you are saying.  When you exceed that limit, they stop listening, even if you are explaining how to make perfect $20 bills on a laser printer. What follows are some goals to strive for when crafting a presentation.

Eliminate anything extraneous, as every new thing takes energy to understand. For example, your system might have 15 tuning parameters, but when only three of them matter to this question, put the rest of them (if you include them at all) in an appendix. This gets them out of the main flow of your presentation, and yet it shows you did your due diligence.

Make sure that each point you make requires your audience to remember no more than two numbers at the same time. Having a nicely designed table of numbers is fine, as no memorization is required.

platesEvery new and unfamiliar term you introduce is one more plate they have to mentally keep spinning as you are building your next point.

Use consistent terms and introduce the least number of new terms possible. Call a “dog” a “dog” all the way though your presentation.

Since graphs are a key element of most performance presentations, do your audience a favor and label your graphs consistently. Put a title and a legend on each graph, and put them in the same place. Label the X and Y axis. Strive to use a common unit (bits vs. bytes) in all your graphs. Use consistent colors so they quickly learn, for example, that the metered values are always blue, the projected values are always red, and the theoretical limit line is always black.

Lastly, establish a pattern in your presentation so people know what to expect. For example, imagine you had capacity planned the performance of a key computer at a future peak. In your presentation, for each subsystem of that computer, show where you are now, then show the projected peak, then state if this will be a problem, and lastly describe any proposed changes to work around the problem. People like a repeating pattern of information in a presentation. They find this comforting and an aid to overall understanding.

The Invisible Presentation

If your audience has trouble seeing what you are presenting, then it is harder for them to understand your wisdom.

hard1

Use a big font (24-30 point), that is easy to read (no funky fonts), and has high contrast (black letters on a white background).  Save the small fonts for your written summary, and avoid colored fonts on a colored background like the plague.

In most meetings a significant part of the audience can’t see the bottom 25% of your slide due to the people in front of them.  Put the good stuff at the top of the slide. Reserve the bottom for things that are less important to the question at hand, like the page number or the snazzy corporate artwork.

hard2Depending on where you live, seven to nine percent of men and 0.4% of women are red/green colorblind, be sure not to make these the two the most critical colors on your graphs.  Also, everyone is colorblind when looking at a black and white printed copy of your presentation.


For more info on performance work see: The Every Computer Performance Book at  AmazonPowell’s Books, and on iTunes.