Chapter 42: Derailing train.

When things go very wrong in a company, it is quite amusing to analyze the diversity of the reactions from people facing the consequences. That is, if you're not one of them —or, at the very least, if you are able to entertain the notion of that impending calamity releasing you from your jail of predictability.

A total shutdown for 32 hours. The site, blacked out. Most of the times showing a message error from a reverse proxy server.

Thirty-two hours. That's a quantifiable loss and it's not small for a site that size. It seemed like utter incompetence from our part —almost as if if had been done on purpose.

On top of that, there was the data loss.

How did everything start? Apparently, the classical mix-up of production server configurations with staging. And a colossal mistake that basically wiped every database that existed. Plus, the incompetence of not-really-making-backups. The last backup Hank could have access to was months old, when they migrated a big part of the site to cloud services —someone had misconfigured the backup service from the cloud provider. Or probably assumed that it would be automatic.

At one point, everyone thought we were done for. Without any recent backup in existence, there weren't many ways to recover.

The shit show brought many personality types to the table.

Carl was the fatalist. He would be on the verge of screaming, unable to react logically. He'd be the one throwing his chair through the window and probably jumping after it. RIP Carl, it was just a drill. Life, that is.

Mallory and Gary seemed to be in sync. They were both orderly executing a contingency plan. Or so they pretended. Nah, they were really improvising. But they were good at it, professionally looking. Sarcasm aside, the calm assurance of a seemingly strong direction as portrayed by their small theater piece was helping. Helping many.

Hank. Blaming people is not helpful. However, Hank was the one that was ultimately responsible for the screw-up —and he wouldn't do anything but to assign blame to others. Frame others, including me. I get it, it's a defensive mechanism, but it shows a very shitty personality trait there.

The key was that I had a very recent backup: in my server, in my room at Eve's. One that, oficially, I shouldn’t have had. But before bedbugs, construction, and hurt egos had turned that place inhospitable, I had set up a series of backup scripts running weekly.

I really hadn't much processing power in my room, at least compared to what Dangerous Crush had in their own data center. That, and the issues I mentioned, made working from home something a bit incoherent.

If that was so incoherent, why did I keep whole copies of what was most valuable for the business? I definitely wasn't intending to start a business to compete —most information was valuable only to them. It was because of... precautions? maybe my project was meant to last longer than their project, and I needed the data. They would surely shut down my project: they wanted to do it their own way. I wasn't stealing it. I was borrowing it to fulfill a vision that they were too obsessed with monetization to share with me.

Regardless of the reasons, in those thirty-two hours, I wasn't sure how telling them that their solution was in my hard drives would play out. It wasn't until the night —10 hours after the cataclysm— that it was clear they didn't have proper backups, thus I decided to come clean as a solution to their problems.

In the eleventh hour, I isolated Mallory in order to talk to her.

“You know, I can fix this.” —decisively, I said.

“Alright...”

“I have a backup in my computer.” —obviously hidding that it was my home server.

“Of the whole thing?” —Mallory was excited.

“Yeah. Mostly. It's about eighty terabytes. Databases, files —profile pictures and all. I even stored all the training data of my match-making algorithm.” —I wasn't lying, but my match-making algorithm was shit at that time. Still, in their weaker moments, I was trying —maybe subconsciously— to push my ‘algorithm’ as the long-time savior they should believe in. Because I had been wasting much of their processing allotment, yet I knew that it was an iterative process destined to succeed. Eventually.

“Alright! Let's get Hank involved, he'd be happy to hear it.”

“Yeah, sure. Or maybe, I can just do it. We don't need him to do a restore some dumps.”

“Oh... okay... do you need admin?” —she conceded.

“I suppose so.” —I didn't. Officially or not, I already had root in most of the servers. The relevant servers. Which... I’m not sure it was how it was supposed to be. Perks of working in small teams.

I knew that my unnotified data backups to an offsite location were potentially unlawful. However, small team. I wasn't even backing up using SQL dumps, but incremental backups of the whole servers' filesystems. That is, rsync through ssh as ‘root’, not pg_dump.

Granting myself root access to the servers had been easy because nobody cared. Again, I knew these things happened all the time with small teams. We couldn’t bureaucratize everything. That'd be hell.

Bringing the solution on was more deserving of praise than the ‘leak’ was deserving of legal repercussions, even if they had refused to understand that it wasn’t really a leak but an unofficial offsite backup. I was sure of that.

Once everything was restored and the whole thing back online, we cheered. I felt like the heroine everyone needed.

The elation lasted only a couple days.

During the crisis, Hank had tried to throw me under the bus. Mallory had been initially cheerful and grateful that I resolved the situation, but her afterthought wasn't as laudatory. She felt that she had had no control, that there was only chaos. That changes were necessary.

Hank kept consistently badmouthing me. The bitterness had spread. He wanted to erase the trails of his gross incompetence that had led to the crisis. Of course, he only had smiles and praise when directly faced.

Mallory set up a meeting. All of us, together.

“Well, we had a rough week, but we'd get over it. I have rearranged the different projects, reanalyzed their cost contingency... and... well, changed priorities. We —I dropped the ball by not defining a good disaster recovery strategy.”

“So... the priority now is to finish migration to cloud services and devise a good strategy for backups, et cetera. Alice, you'll be dealing with it.” —Mallory dropped, and she looked at me expecting a reply.

I disliked the inclination towards further externalization. Really, they had a decent data-center for the demand. Just hire a competent systems administrator able to scale properly, and to set-up a backup routine to an off-site location, maybe some cloud backup service.

But why rely on the fucking cloud for the site itself?

And beyond that, was I going to become a systems administrator? That was not my job.

Carl was cool. As much as he had frozen in despair, he was effective when having some direction. Why wouldn't he deal with the backup architecture and all that, while they hired the competent systems administrator they should have hired months before?

I told her so.

“Carl is set to other projects. Our top priorities now are reliability and monetization.” —Mallory answered.

I saw something in her presentation that upset me.

“The new match-making system... are you removing it?” —I asked.

“I'm sorry, we just have to cancel it for now... No, not to cancel, but to postpone it.” —Mallory concluded, maladapting her terms to my reaction.

Great. I had been working months on it. All to put it aside.

Why had they hired me? Even if I had shown some versatility, I wasn't a fucking systems administrator. If I had taken that job, it had been under the requisite that they would let me consummate my vision. That requisite had been only my assumption, apparently. The low salary wouldn't grant me the freedom that I wanted, but gave them a reason to assign me roles that weren't part of my main skill set.

No decision taken in that meeting was a good decision. Even rewriting the front-end in Angular —something I didn't care at all about, as I was as much a front-end developer as I was a systems administrator— seemed quite silly.