International Hug a Systems Administrator Day

No, it's not an official holiday, but it should be.

This industry seems to think that software developers, the programmers, are more important and more valuable than other information technology professionals, such as database administrators, systems administrators, deployment engineers, software configurers. I've seen this often reflected in differential salaries, but I see it even more in the respect and attention afforded people. The programmers often get to be the rockstars.

I think we would do well to remember that software developers can't do what they do without the support of excellent database administrators, systems administrators. Installing and configuring systems is what makes them go and deliver actual real value to an actual real client. Until that happens, software is the heady stuff of platonic forms, interesting abstractions without a connection to the real world.

I've been blessed with a number of excellent systems administrators and operations professionals in my short programming career to date. In my gig at Yale's Technology and Planning group, I had the privilege of working with Drew Mazurek, who has since moved on to other interests. Besides being a programmer and filling other roles, Drew was the go to systems programmer and administered the group's internal systems and played a role in administering external systems. His competence was extraordinary. In the systems administrator role he was a value multiplier for the those playing the developer role. And he saved my bacon that time I trashed the CVS server (sorry, Drew.)

In support of production applications there was the production services group. They administered and instrumented the production servers and saw to it that they were regularly backed up and restorable. They drilled some of this, and developed prepared procedures for all of it, so that when things go wrong, the first steps of response are pre-baked. They rolled new software versions into production with grace. They provided critically valuable diagnostic and troubleshooting services.

And I've gotta say, those regular backups and ability to restore from them, those are a mighty fine safety net for the programmers doing the production software tightrope dance. Log rotation. Load monitoring. Intrusion detection. These are things you sure wish you have in a pinch.


Human beings make mistakes. It is in their nature. It is inevitable. Things will go wrong. Rare is the day when i don't do at least one stupid thing that would have bitten me but for the ability to "undo" on some scale, whether it's a menu bar command, a source control revert, or rolling back to a prior database snapshot. It is not appropriate, or fair, or realistic, to construct your expectations around people never making mistakes.

Systems administrators are the folks who mitigate peoples' mistakes. Whether it's a developer who accidentally trashed the CVS server, or a developer who introduced a subtle bug that causes the software to leak connections, or a power outage, or spilling the coffee into the web server. How bad it is when Bad Things Happen is in the hands of the systems administrators.

I'm very grateful to have worked with a number of talented, caretaking, proactive systems administrators.

Developers more important than systems administrators? Bah. I'm the guy that writes the buggy code. They're the ones who make the buggy code work in production and solve real problems. I ask you, which is more noble?

I've met more than a couple talented systems administrators, and I mean really talented diamonds in the rough, who tell me they aspire to be Programmers. And I've gotta ask myself, why? You're already doing the most incredibly valuable thing you could be doing. And I'm very grateful you're doing it. And not everyone can do it the way you do it.

Oh, sure, I can fumble around in grep. And absolutely, it is quite possible to become a capable, competent, valuable systems administrator by hard work, practice, and care. But there is nonetheless a spark, a gift that some people have for this role. I don't have it. I have no doubt I could fill the SA role if slotted into it, but I know I'd never be as excellent at it as these systems administration rockstars. I'd be too inclined to get distracted and go scribble on a whteboard or explain excitedly about some software or project, preferably while talking with my hands, to give the log file rotation regime and failover procedures the full and careful attention they deserve. So how do we (should we?) entice the exceptional systems administrators to stay fulfilled in that role?

Perhaps as an industry we need to have more respect, appreciation, and yes, compensation, for the systems administrators.

You can start by seeking yours out and applying a hug. If your next big mistake is going to be a little less big because of the hard work and care of someone else, I think it's fair to advance a down payment on the gratitude you can expect to have the opportunity to feel later.

Mistakes are inevitable. It's the consequences of the mistakes that can be mitigated.