Saving the day when performance unexpectedly suffers takes two things: the courage to act and preparation. The courage comes from within, but being prepared is easy; it can happen daily as part of your normal work. Here are some hints:
Have a phone list
The saddest thing is to watch someone wasting time looking up a phone number in a crisis. Create a phone list, and periodically call every number on it to make sure that the number is current and the person is still responsible for the thing you think they are.
Have a checklist
Make a list of things to do and check in a crisis. In the first rush of the problem you might not use it, but when you are stumped, it is a good thing to have.
Create fast analysis tools
Most of the time your meters are monitoring your computing world at a leisurely pace. When bad things happen, waiting for these meters is agony. Create a fast analysis tool that meters and reports in less than a minute.
Understand the differences in your tools
Many companies have a mix of performance tools. In a crisis, it is easy for two people looking at different tools to confuse the situation. Work to understand the differences in these tools during the quiet times. These tools may label things differently, use different low-level meters, sample at a different rate, or average over a different interval.
Get to know the local experts before you need them
Take time in your day to find and build relationships with the other experts in your company. Notice I did not say “meet with.” Ignore the bureaucracy and talk to them as people about how you can help each other. I’ve visited many companies, and the ones that handle problems the best are the ones where the key experts all know each other well.
Master tech support
Sometimes an expert can only be reached through tech support. The first time you ever call them should not be in a middle of a crisis. Once in a while call tech support with a question. A good first question is: “I’m preparing for a possible performance problem. Tell me about the basic meters/info you need?” Learn to collect those meters.
Every tech support department has ways to prioritize calls and protect their key wizards. If you know the system, you can get what you need quickly. Learn these ways by calling before the crisis.
Lastly, every tech support person I know has this advice for you. If you act like a jerk, you’ll get their least helpful service. If you are calm, clear, and prepared you’ll get their most helpful service. If you say “Thank you” and, when appropriate, CC their boss you’ll get better service the next time you call.
More information on this, and many other useful ideas, can be found in my book:
The Every Computer Performance Book