Kiran Jonnalagadda’s Blog

My blogging these days is largely confined to Twitter, with the very occasional picture on the moblog.

I’ve long regarded blogging as an outlet for self-expression first, everything else a distant second. My work-related responsibilities and associated communication needs have grown tremendously over the past year, taking away much of the energy otherwise channelled into such expression. Barcamp Bangalore has similarly taken its cut.

What’s left works rather well at crafting an expression in 140 characters.

I’ve updated my mms2lj script to support creating a static thumbnail image. The result is the moblog widget you see in the sidebar (If you’re reading via the feed, please come to the site).

You can use this to link to Jacemobile from your site too. Here’s the code:

<a href="http://community.livejournal.com/jacemobile">
  <img src="http://home.seacrow.com/~jace/moblog/static.jpg"
      width="160" height="120" alt="Jacemobile" />
</a>

The server on which this site is hosted experienced a hard disk failure earlier this week, taking down everything that was hosted on it. The crash was not instantaneous. Services failed one by one, leaving a trail of incomprehensible error messages. At first we assumed a full disk. Since even SSH had failed, we saw no option but to reboot the machine. The ISP then discovered the disk was failing and wouldn’t boot anymore. They replaced it with a new drive and a freshly installed OS. I’ve spent the last few days probing the old disk to see what could be teased out of it. This is the first instance of data loss this site has experienced in over six years online. Backups have been highly irregular, so, much of what is lost is lost permanently.

The following description includes much technical jargon. Please feel free to skip if you’re not up to it. It doesn’t matter in the grand scheme of things.

This server ran three instances of Zope, with Plone versions ranging from 2.0 to 2.5, one Trac database, one instance of Lotus Notes, and the image archives of Jacemobile. All three Zope servers use ZODB via ZEO (also separate instances) with the default FileStorage backend. This means that the entire database is stored as a single file on disk. Damage to the file becomes damage to the entire database. As luck would have it, only one of three databases was rescued unscathed. This site’s database lost a few megabytes towards the end of the file.

But ZODB FileStorage is also of an incredibly resilient design. It is an append-only database. Any changes, whether additions or deletions, are added to the end of the file. Changes are undone by adding yet another record to the file to invalidate a previous record. A pack operation must be performed periodically to discard all invalidated records. What this means in the context of the crash is that losing the tail end of the file was equivalent to undoing the last few changes, which in this case turned out to be a few days worth of comment spam. Not a bad deal at all.

The other bits were not as lucky. Having lost several files in the Zope and Plone code bases, I decided I might as well upgrade to Zope 2.9 (from 2.8) and Plone 2.5.1 (from 2.5). Zope 2.9 requires Python 2.4, installing which promised to be a miniature nightmare. This server runs Red Hat Enterprise Linux 3. If I had a choice, it would have been something Debian-based, like Ubuntu, because in my experience, Red Hat-based systems inevitably turn into package maintenance hell. RHEL3 comes with Python 2.2, which Zope has never supported. There are no standard Python 2.3 or 2.4 RPM packages for RHEL. I had previously built them for 2.3 and those files were thankfully uncorrupted too, but for 2.4, I had to start over.

Python.org has Fedora RPMs for 2.4. Rebuilding the SRPM worked on RHEL, but Zope wouldn’t take it. 2.4.3 wanted. Py.org’s 2.4.3 RPMs page refers to 2.4.2 — must report that to the webmaster — and the 2.4.2 RPM won’t build on RHEL. After similar lack of luck with other SRPMs obtained from the distribution networks, I had to get down to editing Py.org’s 2.4 RPM’s spec file to build from the 2.4.3 sources. That finally worked, but after blowing out two hours of my time for what should have been a 2-minute affair at most.

One Zope site (this) is up and running now. Two more to go. Lotus Notes was totally hosed but that has regular offline backup. The worst casualty was Jacemobile, my moblog. The entire 2005 archives were lost. A phone storage card crash earlier this year wiped out my local backup too. The last backup was in November 2004, when switching servers. The WayBack Machine missed it too. It appears the 2005 archives are permanently lost.

Unless I can lay my hands on an ext2 recovery tool. Do you know of one?