Minor Disaster
Somewhere around 4:20 am, on June 25th, I made a colossal mistake, more through over-confidence and exhaustion, than incompetence, though that played a role as well. There are two fields in the database record for each locomotive, labelled ‘State’ and ‘Fullstate’. The idea for these fields was that we’d be able to have a state or province, and its proper abbreviation. ‘Minnesota’ and ‘MN’, for example.
So, what did I do? There was a locomotive in Russia that was listed in the wrong province. Rather than using the site’s administrative interface (slow and clumsy) to fix it, I decided to edit the database directly. And I correctly set up the State and Fullstate values for the locomotive in question. And applied the change to the entire database instead of to the one locomotive I’d planned on fixing. This had no impact on countries where the two fields weren’t in use, but for Russia, Canada, the United States, South Africa, the United Kingdom, Germany, Japan and Australia it was catastrophic. Essentially 10,689 locomotives suddenly became invisible on the website. If I editted the record for a locomotive to put the correct values back in those fields, it reappeared with its data intact. But the prospect of hand editting 10,689 records was somewhat daunting.
I opened an emergency support ticket with my hosting provider, and confusion ensued. They were intially of the impression that they were backing up our database nightly, but literally couldn’t find the backup files. After some hours, some discussion and a fair bit of self-loathing, I sat down and began the hand editting process. After a bit, I got a pretty good feel for how fast I could do this, and my initial estimate was that the site would be back to normal in about 3 months. So I sat down and began writing a tool to speed up the process. About midway through the writing process I got an excited phone call from the hosting provider stating that they’d found the backups, that they were about 27 hours old and did I want them to restore the database? Yes. Oh heck yes! Si. Da. Jawohl. And they did.
The website and database returned to the precise state they were in at 1:30 am on Jun 24th, 2017. If notes were posted, links or photos added, or even locomotives added to the site between the time of the backup and the time of the mistake, they were gone. Fortunately it was neither difficult nor tedious to re-enter most of it. And the prolific user of Notes, ILFORD, who has provided a vast quantity of useful information, simply reposted the information that disappeared.
So, the site is caught up, the database is clean and tidy, and I’ve developed a set of stored procedures that will protect me from myself. Thanks for your patience.
And to those of you who didn’t notice: you need to visit us more often (that’s supposed to be a joke).