My Worst Day At Work

A key customer was having a performance problem and our company had assembled a team to fly down and work on it. I was part of that team and this was going to turn into my worst day at work – ever.

This customer had developed their application on our proprietary operating system and it worked great. However, we had recently shipped a UNIX OS on that same hardware and the customer had ported their application to it and it ran as slow as mud.
The way they saw it: Same hardware, same application, different OS, bad performance. This must be the vendor’s fault. The way we saw it: Applications ported to new operating systems often have performance problems in the same way a human from Earth would have to work extra hard to make it on a different planet.

So we arrived onsite and the initial one hour meeting with Mr. Big Cheese and his henchmen was deeply unpleasant and accusatory. Then we spent the rest of the day in a conference room logged in to the test system looking for some solution. At no time were we ever left alone. There was always at least one of his henchmen there asking us what we were doing and often unhelpfully commenting on our efforts: “We already tried that.”, “Anyone can see that this is not the problem.”, and “You’re wasting time.”

As the day drew to a close we had a final meeting with Mr. Big Cheese and he was not interested in what little performance-enhancing crumbs we had found. He used that hour to imply we were all idiots and demand that our company send down a real “UNIX kernel hacker”. Over and over, he made it clear that only a “UNIX kernel hacker” could solve this problem. We were not that, so he told us to go.

This company had made two key mistakes with us: One: They were jerks to the people who had come to help them. So we did what we were required to do, but not what we could have done if we wanted to call in favors, bend rules, and do heroic things. Be nice to the people who come to help you, even if they are idiots. Why? Because they go back to that company and advocate for you and spread the word that you are worthy of heroic, rule-bending efforts. Two: They never left us alone. We were never free to work as a group for fear that we’d say or do something stupid. If experts fly in, give them private space to call other experts at headquarters and talk amongst themselves. They will get to any possible solution much faster.

beerWe left at the end of the day as a group and walked to nearby restaurant. We ordered a round of beers. When they came, I picked up my glass and chugged the whole beer in a few seconds, something I’d not done since my college days.

Everyone was a little shocked. I set the glass down and slyly said: “It doesn’t do you any f#@king good in the glass.” That broke the ice and to this day we laugh, and laugh, and laugh about what a really horrible day that was and the fact that we are not, and never will be, “UNIX kernel hackers”.

Alcohol is not a solution to man’s problems, but laughter is.


7 thoughts on “My Worst Day At Work

  1. Good points about how to treat consultants and visiting experts; but you ever figure out what the problem was?

    • Noah, yes we did. The application was written using a lot of vendor-specific OS-level file system subroutine calls that were not available in UNIX, so they wrote their own. As you can imagine, their first attempt at replicating (for example) indexed files was not very efficient and had significant locking issues that killed performance. Ironically, the one wizard they wanted (a UNIX kernel Hacker) could not have helped them.

  2. The four most deadly words in High Tech: “It’s just a port.” Any time I hear those words, I cringe, because it’s NEVER “just a port” — there are always nuances to where it’s coming from or where it’s going to that make the project last 10 times longer than the planners think it will take. Sounds like Mr. Big Cheese was under the impression that this was “just a port.”

  3. Yes. Beer drinking. I remember those days.

    Incidentally, I like the blog. You’re a good writer as well as the best public speaker EVAAARRRR.

    Fair Winds,


  4. Hey Bob – long time, no speak. That raised a smile over my morning cuppa today (not beer at this time of the day). Cheers, Adrian

  5. LOL! This story reminded me of the time I flew to a bank to fix an application problem. Dire straights. I figured out the config issue (combing logs for 8 hours) but was “held hostage” for another 2 days – until they got proof from offsite backups (a typo in a session cookie name). I remember calling my manager and telling him we already knew the root cause and can I please come HOME? A favor to hop on a plane turned into a resentful fiasco. The rescuer was turned into a hostage! Never again.

  6. This type of customers will cost you more than you ever will earn. The best thing that can happen is that they go to your competitor.
    But these things might be come worse when system maintenance and network are outsourced to other parties….

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s