End Users: Please, Don’t Lie.


My last post was all about how I’ve come to terms with my career path: I even like it! Of course, even with things we like, there are going to be aspects that bother us. The following series of posts will explore a few of those aspects that I feel more strongly about.

I don’t know how my screen got smashed. I just closed my laptop lid, went to a meeting, and when I got there, it was just like this!

My computer is completely down! It won’t power on! It’s totally broken!
<show up at desk to find that the space bar on the keyboard doesn’t work>

Yes, End User, I understand. We IT minions can be jerks. The worst of us can be insufferable know-it-alls who make you feel belittled for even uttering the word “computer,” as though you’re somehow not worthy. You feel like the only way you can make us take you seriously is by over-exaggerating the problem. Or maybe you feel like we’re sitting there, silently judging you for clumsily dropping your open laptop lid-first. Or perhaps the coffee spilled, the sparks flew, you panicked. The end result is always the same: “I don’t know what happened. It just stopped working.”

I can only assume that your thought process goes something like this:

Oh crap, did I seriously just do that? Please work, please work, please work… Noooooo, it’s not working. SON OF A… Ugh, I have to call IT. I don’t want to call IT. The only thing worse than breaking my computer is dealing with those people. Sigh… I wonder how expensive that laptop is. I hope I don’t get in trouble for this. Well, those insufferable know-it-alls already assume I don’t know anything. I’ll just tell them I don’t know what happened. Fits their expectations. I’m just an “ignorant plebe” to them anyway. Alright, let’s just pull this rip cord… I know, I’ll just leave the computer on my desk and go to lunch. Maybe they’ll fix it before I get back.

Do you really think that those of us who work with computers every single day have never done something dumb, too? Of course we’ve dropped our phones or laptops. Of course we’ve forgotten to back up our files. Of course we’ve spilled that dark, hot, bitter, nectar of the morning gods all over our poor keyboards (and we were probably a lot more angry about the loss of the coffee than the keyboard). We’ve done far worse than you ever have, because our capacity for screwing up is so much greater. I know a guy who accidentally deleted an entire pool of servers in VMWare by accident. Shit happens.

What makes me upset is not that you accidentally broke your computer. I’ve been there, I’ve done that. What makes me upset is when you lie about it. Do you understand how long I spend troubleshooting problems just because you left out some critical piece of obvious information? It’s so bad that I almost have to assume you’re lying to me, now.

The funny thing is: most of us don’t judge you. You have a particular skill set, we have ours. I’m always pleased to find computer-competent end users, but at this point, I don’t expect any of you to really know the difference between your desktop tower and your Windows desktop, where you store a copy of every single file on your computer. I don’t need you to know anything about computers. I just need you to be honest with me.

Here’s the other thing: you won’t get in trouble. If every employee got into trouble for breaking equipment, there wouldn’t be anymore employees! Accidents happen. Mistakes are made. The business knows this, and while it tries to do what it can to encourage end user responsibility, it also (if it’s smart) budgets precisely for these kinds of accidents. Just like you budget to get your car fixed or to do basic maintenance on your home.

Trust me, you can’t possibly screw up worse than we can. You don’t have access to the truly critical systems that can cause the most damage. And even if one of my Desktop brethren makes you feel dumb for not noticing that pen on your keyboard before you closed your laptop lid, don’t let him ruin your day. He’s almost definitely done worse.

So please, please don’t lie to us. Just tell us what you did. We’ll fix it faster, you’ll get better service, and chances are that we’ll actually respect you a lot more for your honesty than we will for your cover up.

Because here’s the last thing you need to know about being in Desktop support. We know. We always know. You’re not fooling anyone.

So you might as well just tell us the truth.

#desktop-support, #end-users, #lying

The Road from There to Here

“I will be a writer, I will be a writer, I will be a writer…” Thus the mantra in my head went, day-in day-out, throughout my high school years.

When I went to college, I wanted to be an author and a pastor. I couldn’t conceive of being anything other than one of those, and I thought by now I’d be both. I studied philosophy and classical languages, because I thought it would help me in seminary. Then, when I got to seminary and started to question whether this pastor-thing was for me, I turned back to writing, my first love.

It took much longer than it should have to realize that I had to give that up, too. I was not very good at work-life balance: I tend to throw myself into things. If being a writer was what I was going to be, then I was going to be a writer all the time – except, I also had a wife and a daughter who relied on me. I couldn’t be a writer all the time. So I did a bunch of odd jobs – working at a theater, a factory, an in-home care organization, etc. And I wrote when I was at home.

Funny enough, there is one other thing that I’m very good at, but I resisted it entirely during this time. I’m good at problem solving, at understanding computer systems, at conceptually grasping those systems, implementing them, fixing them, etc. Despite being “better than average” at these things, I tried very hard to avoid them. I dislike a lot of what we do with computers, including many aspects of social media, tracking, advertising, written-versus-verbal communication, etc. I think computers can be dehumanizing when they’re used inappropriately, and it seems to me that we use them inappropriately more often than not.

That being said, I cannot deny that I have a talent with computers. I’m pretty sure it’s the same talent that people who are good with mathematics and philosophy have. It’s the ability to break a problem down into basic components, then rebuild it again. The only difference between computer science, math, and philosophy is what you’re breaking down, what you’re trying to understand. Your “physical” tools might be different. The mental “how” is basically the same.

Over the last two or three years, I’ve come to embrace my talents when it comes to working with computer systems. Why did I resist for so long? It seems ridiculous, now. I have a skill set that can be used to make money to provide for my family, in a field that I’m good at, in an environment that provides continuous problem-solving opportunities (what I love most about my job). Obviously this is what I should have transitioned to the moment I left seminary.

I’ve spent a lot of time kicking myself over this. How much farther would I be in my career if I’d started sooner? I’ve seen a lot of people go through something similar in their own lives. I cannot speak to all of the reasons others might have for their own difficult “life transitions” – but I think I have a clearer understanding of why I found my own so difficult.

Before I even went to college, I constructed an “internal” mental identity of myself, and without that identity, I didn’t know who I was. When faced with the prospect of losing that identity (be a pastor or a writer), the uncertainty was too daunting to deal with, so I just didn’t. I held on to what I thought it meant to “be me” even when it proved disastrous for myself and those around me, especially my family. In other words, I was being a selfish asshole.

There’s a lot of times in our lives when we’re selfish and we know we’re being selfish. We do it anyway, because even though we didn’t put another person first, it doesn’t seem that anyone got hurt. Or maybe someone even did get hurt, and we felt guilty about it for a really long time. Whatever our selfish habits are, it’s not uncommon for us to recognize them and either try to manage them when they rear their ugly heads, or to feel terrible when we realize we’ve done something selfish.

But there’s a certain kind of selfishness that is so fixated on the self, it literally cannot see others or abide being told that it’s selfish. That second kind is nearly impossible for anyone to break through from the outside. I say that, because I can see now that it’s what I was doing at that point in my life. More than one person, even people I looked up to, tried to hint or even outright tell me that that’s what I was doing. I didn’t hold them in derision or scoff at them – I simply didn’t think they understood me. So I didn’t listen. Why would I? I’ve known what I was supposed to be doing since I was a kid. They didn’t. It wasn’t their fault that they didn’t understand, and I could “forgive” them for that. I would keep doing what I was doing and eventually they would see it.

It finally took my wife to tell me, “You’re hurting me.” This was a big deal, because as much as I wanted to be a writer (and before that a pastor), and I didn’t want to go work with computers in IT, the truth is, more than anything else in the world, I didn’t want to hurt her. At first, we had a lot of long conversations. I raged a bit on the inside, not toward her, but toward myself. It’s one thing to be told, “You’re not doing the right thing.” It’s another thing entirely to be told, “You’re doing something that hurts me, someone you love.” The first is easy to blow off in all kinds of creative ways. The second isn’t. Especially when it comes from a person you love, and you know to be about as selfless as anyone you’ve ever known. It’s a punch straight to the gut, an unavoidable truth: I’m being selfish.

I didn’t see then that my selfishness was tied to the mental identity I’d carefully crafted for myself. That would come later. I only saw at that point that I needed to change things up, however grating it might be.

So I went back to work with computers.

And you know what? It wasn’t working with computers, and forcing myself to look for things I enjoyed, that changed me. It was knowing that I was doing it for someone else that made the biggest difference. First, it was for my wife. Eventually, it was for my team. And after a long enough period, it even became for the end users I was supporting. Over time, I’ve come to genuinely enjoy the work I do, to the point where I actually think about my career in IT for the foreseeable future.

What changed? I started doing what I should have done the day I got married (even before). I learned to let go of my carefully constructed identity, to find myself, not in what I thought would be most fulfilling in my life, but rather in what I hope is most helpful to those (especially my family) I’m trying to help.

I’m not claiming, by the way, that I’m particularly good at being selfless. Like I said, we’re all selfish. I try to keep my head screwed on straight, and my eyes focused on serving my neighbor, not on fulfilling my own desires. I’ve even found, over time, that my desires have become more and more aligned with wanting to serve my neighbor. But I obviously fail, too. Thankfully, I have no end of friends and family who are willing to point out when that happens and keep me on track.

I also try to do regular self-evaluations, sometimes even daily. If I do things that make me happy, I try to examine what I did, why I did it, and what impact it had not just on me, but also those around me. I make it sound laborious here, but it’s really not that bad. It’s one of those mental exercises I do almost without thinking. It doesn’t mean I enjoy things that make me happy any less. It means I’m simply trying to be careful not to enjoy them at the expense of anyone else.

I haven’t given up on the notion that I might write and publish something one of these days. That may never happen. But it also might. Who knows? The point is, if it does happen, it will be because it’s the right thing to do at that moment in time – the right thing not just for me, but also for those I love.

For now, I’m just happy not to deal with the daily angst of feeling like I’m being torn apart, like I’m losing “myself” (which, I know now, was never really me). Hopefully, having gone through that once, it will prove useful in identifying any other identity crises that may come along one of these days. I assume I’m not the only person who has them. Maybe sharing my experience (since that’s all I have) will help someone else, too.

And, if nothing else, it gave me something to write about.

Learn to be a Linux Sysadmin

I stumbled across a great post on Reddit not long ago, which detailed “what to do” if you want to learn to be a Linux SysAdmin. It’s a bit dated, it looks like – around 2 years old. But it still looks to me like a pretty solid start, and I’m at a point where I feel like I need a long-term project to really kick this whole system administrator thing off. So I think I’m going to give it a shot, though with a few minor tweaks to the original.

  1. Setup a KVM hypervisor.
  2. Install a Spacewalk server inside of that hypervisor. Use CentOS 7 as the distro for all following works. Set up errata importation on the CentOS channels, so you can properly see security update advisory information.
  3. Create a VM to provide named and dhcpd service to your entire environment. Setup the dhcp daemon to use the Spacewalk server as the pxeboot machine, thus allowing you to use Cobbler to do unattended OS installs. Ensure that every forward zone has a reverse zone associated with it. Use something like “internal.virtnet” (but not “.local”) as your internal DNS zone.
  4. Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, which which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind; do not use unencrypted LDAP.
  5. Reconfigure all 3 servers to use LDAP authentication.
  6. Create two new VMs which will then be Postgresql VMs. Use pgpool-ll to setup Master/Master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.
  7. Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. Cheat and use Ansible for deployment purposes, again plugging into the Spacewalk server.
  8. Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.
  9. Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM’s image on the iscsi LUN, and every other machine on the NFS share.
  10. Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.
  11. Deploy two more VMs. These will have tomcat on them. Use Jboss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.
  12. Deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.
  13. Deploy another VM. Install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.
  14. Deploy another VM. Set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a “is the right port open” check, and a “I got the right kind of response” check, and “we still have file system space free” check.
  15. Deploy another VM. On this VM, set up a syslog daemon to listen to every other server’s input. Reconfigure each other server to send their logging output to various files on the syslog server. Setup logstash or kibana or greylog to parse those logs.
  16. Document every last step you did in getting to this point in your brand new Wiki.
  17. Go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.
  18. Go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.
  19. Destroy every secondary machine you’ve created and use the above profile to recreate them, joining them to the clusters as needed.
  20. Dreate three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.

I’ll be honest. I’m familiar with some of this, but there are some things in here that I’ve never touched, and a few that I’ve never even heard of. I look forward to getting started!

#how-to, #linux, #sysadmin

Linux and Beer

Time: 8:13 PM

I installed Windows 10 on my computer a couple of months ago so that I could learn it. I mostly did this because I’m a Desktop tech, and I work on Windows for a living. I also wanted to play video games, but my Xbox One solves that problem.

I kind of hate Windows, as anyone who knows me quickly finds out. So tonight, I’m switching back to a sane operating system.

It’s time to crack open some beers and install my distro of choice: Arch Linux.

This my plan for the first stage of my setup:

Distribution: Arch Linux
Window Manager: i3

Too simple? Nah. I’m going to keep it minimalist for now. I want it to be pretty (unixporn pretty), because aesthetics do matter a lot to me. But that will probably come tomorrow. For tonight, let’s just rid my computer of Windows forever more.

Time: 8:34 PM

While doing one last backup of my hard drive, I downloaded the Arch iso and used USBWriter to create a bootable USB stick. (No, I don’t have Cygwin installed, so no dd.)

I also printed off the Installation instructions on the Arch wiki, because it’s been a while, and I dislike using my phone to follow instructions like that.
Backup is now complete.

Goodbye, Microsoft.

Time: 8:49 PM

Hahaha. I almost forgot to backup my second hard drive, and I really want to wipe them both when I do this. Woops.

Ah well. Backup continuing while beer is consumed.

Time: 9:27 PM

Just now getting to the Arch install. Apparently it doesn’t like my Nvidia card with the default Noveau drivers. Had to set the kernel parameter to nomodeset to get the installer to boot. This prevents it from ising those drivers, which I guess affects the installer’s resolution?

I don’t really care. It’s just a bunch of text right now, anyway. Who needs resolution?

Time: 10:17 PM

Yeesj, this is taking longer than I thought it would. Stupid wpa_supplicant. I know it’s probably my wireless USB adapter, but I finally just conceded defeat and ran an Ethernet cable to my PC. I’ll figure it out after I get Arch installed.

Time: 11:23 PM

Well, I had hoped to write more, but I kept running into problems… And it’s hard to write while also reading the Wiki, especially on my phone. So suffice to say that Arch is finally installed, along with i3, and I’m calling it a night.

After one last beer…

#arch, #linux