Jonathan's Blog
name

Jonathan's Blog

Mindful Leadership and Technology

TagSimple
Featured

Technology Simple Cloud

Sometimes Mindfulness, Sometimes Repairing the Blog Site that You Blew Up

Posted by Jonathan FriesJonathan Fries on .

Today's lesson: always be smart before you're stupid. This helps you prevent some damage from your stupidity. If you're stupid first, anything can happen.

For the first time in a good long while I missed my evening meditation because I was repairing this site. Sorry if you cam here and got the 502 Bad Gateway message.

I started an upgrade process as I was leaving work, but by the time I got home I realized I had done something bad.

Blog site down. Bad Gateway! Bad! Bad, naughty, awful, shameful gateway:


Then my kids had scouts and I didn't really get to jump in and fix it until about 8:45 PM.

The solution for me was, ultimately, not to fix it. I have backups of everything, and I am not a Linux admin or even much of a user anymore. What I needed to do was:

  1. Start a new VM from my backup disk image.
  2. Delete and reimport content in ghost.
  3. Zip images on old VM and restore them on new VM.
  4. Change the DNS entries to point to new VM.
  5. Shut off old VM and release the IP address.

All of this should have taken about 20 minutes if I did it right the first time.

It took about an hour because I had to figure several of the steps out along the way.

It's kind of amazing that you can do this. You do need to be careful with backups - make sure you have them, make sure you back things up frequently and before you do anything dumb.

Always backup before you do ANYTHING that's dumb.

But once you have your backups, and once you have reached a state of self-awareness that allows you to ensure that you make them before making mistakes, the cloud makes the world a very simple place. Kudos Cloud Makers (Google in particular) you saved me a lot of time. Also, Ghost - their backup and recovery worked pretty well. I shouldn't really have had to do steps 4 and 5, but it was still pretty painless.

I'm a little sad that I couldn't fix whatever problem I created, but sometimes you must cut your losses and provision a new VM.