Kiran Jonnalagadda’s Blog

The server on which this site is hosted experienced a hard disk failure earlier this week, taking down everything that was hosted on it. The crash was not instantaneous. Services failed one by one, leaving a trail of incomprehensible error messages. At first we assumed a full disk. Since even SSH had failed, we saw no option but to reboot the machine. The ISP then discovered the disk was failing and wouldn’t boot anymore. They replaced it with a new drive and a freshly installed OS. I’ve spent the last few days probing the old disk to see what could be teased out of it. This is the first instance of data loss this site has experienced in over six years online. Backups have been highly irregular, so, much of what is lost is lost permanently.

The following description includes much technical jargon. Please feel free to skip if you’re not up to it. It doesn’t matter in the grand scheme of things.

This server ran three instances of Zope, with Plone versions ranging from 2.0 to 2.5, one Trac database, one instance of Lotus Notes, and the image archives of Jacemobile. All three Zope servers use ZODB via ZEO (also separate instances) with the default FileStorage backend. This means that the entire database is stored as a single file on disk. Damage to the file becomes damage to the entire database. As luck would have it, only one of three databases was rescued unscathed. This site’s database lost a few megabytes towards the end of the file.

But ZODB FileStorage is also of an incredibly resilient design. It is an append-only database. Any changes, whether additions or deletions, are added to the end of the file. Changes are undone by adding yet another record to the file to invalidate a previous record. A pack operation must be performed periodically to discard all invalidated records. What this means in the context of the crash is that losing the tail end of the file was equivalent to undoing the last few changes, which in this case turned out to be a few days worth of comment spam. Not a bad deal at all.

The other bits were not as lucky. Having lost several files in the Zope and Plone code bases, I decided I might as well upgrade to Zope 2.9 (from 2.8) and Plone 2.5.1 (from 2.5). Zope 2.9 requires Python 2.4, installing which promised to be a miniature nightmare. This server runs Red Hat Enterprise Linux 3. If I had a choice, it would have been something Debian-based, like Ubuntu, because in my experience, Red Hat-based systems inevitably turn into package maintenance hell. RHEL3 comes with Python 2.2, which Zope has never supported. There are no standard Python 2.3 or 2.4 RPM packages for RHEL. I had previously built them for 2.3 and those files were thankfully uncorrupted too, but for 2.4, I had to start over.

Python.org has Fedora RPMs for 2.4. Rebuilding the SRPM worked on RHEL, but Zope wouldn’t take it. 2.4.3 wanted. Py.org’s 2.4.3 RPMs page refers to 2.4.2 — must report that to the webmaster — and the 2.4.2 RPM won’t build on RHEL. After similar lack of luck with other SRPMs obtained from the distribution networks, I had to get down to editing Py.org’s 2.4 RPM’s spec file to build from the 2.4.3 sources. That finally worked, but after blowing out two hours of my time for what should have been a 2-minute affair at most.

One Zope site (this) is up and running now. Two more to go. Lotus Notes was totally hosed but that has regular offline backup. The worst casualty was Jacemobile, my moblog. The entire 2005 archives were lost. A phone storage card crash earlier this year wiped out my local backup too. The last backup was in November 2004, when switching servers. The WayBack Machine missed it too. It appears the 2005 archives are permanently lost.

Unless I can lay my hands on an ext2 recovery tool. Do you know of one?

It appears India’s Department of Telecommunications (DoT) has issued a directive to Indian ISPs to block BlogSpot and TypePad, and several ISPs have complied. LiveJournal is spared. I’ve been unable to access BlogSpot since Friday — the connection times out.

Here are reports from Mridula Dwivedi and Neha Viswanathan, and on DesiPundit. There’s a new Bloggers Collective group for tracking updates. Shivam Vij is currently working the phone with ISPs and government departments and so far has confirmation that this blockage was not ordered by CERT-IN, the only body authorised to issue orders to ISPs under the IT Act 2003. The order came from DoT on Friday (CERT-IN is required to route orders via DoT) and the list of sites being blocked is not public. Shivam’s asked for the list but been told it is “highly confidential”.

I’ll post updates as they become available.

Update: Shivam Vij managed to get through to DoT official Dr Gulshan Rai, who it appears is also director of CERT-IN. His response: “Somebody must have asked for some sites to be blocked. What is your problem?” Please tell him what your problem is. According to the directory, his phone number is +91 (11) 2436 3081. Email. Nandan Babla’s posted a guide to filing a Right to Information (RTI) application (bypass block).

There’s a wiki page now for reporting ISPs that are participating in the block. If you can’t access BlogSpot, please report your ISP.

Update 2: GeoCities is also blocked. Dina Mehta has her take on the situation. Amit Agarwal has a collection of tips on how to bypass the block (but first you’ll have to bypass the block to read that). Shivam Vij now has a longer write-up on the information he dug up this morning.

Update 3 – 5:15 PM: Airtel (and possibly) Sify have also started blocking.

Update 4 – 10:55 PM: Sify and Tata Indicom (previously VSNL) are also confirmed blocking now. Shivam Vij has an article out at Rediff. Boing Boing’s carrying links (hello BB readers!) too. That should get the word out a bit. Others on the Bloggers Collective group have been pursuing journalists at various publications.

I had a late evening meeting with the technical head of a large, non-consumer ISP. It was work related, so I can’t reveal who until appropriate. He confirmed that DoT has a regular practice of sending a list of URLs to be blocked, and that it is illegal for an ISP to block anything other than this list. Since it comes from a government department, the list is not confidential. I hope to have my hands on it shortly.

Neha’s collecting other updates, by far the most comprehensive yet.

Update 5 – July 18, 12:15 PM: Mainstream media is picking up the story. There’s a list on the wiki. Far too much noise on Bloggers Collective group about how censorship can be routed around via proxies. Get this, folks. This isn’t about censoring bloggers. This is about curtailment of civil rights of all internet users. That is what we should be fighting against.

Update 6 – 5:15 PM: The group is now getting extremely noisy. 235 members and 570 messages, in just two days. I wrote a piece for the Times on how to circumvent mistaken censorship. If it clears the editors, it’ll hopefully be in print tomorrow. Getting around the block is easy, but we need people to be aware of how. Nishant Shah offers the thought that maybe the government ordered this block knowing fully well that it could be circumvented. Their point is made, anyway.

Update 7 – July 19, 9:50 AM: So much going on now, I’ve stopped keeping track. Neha’s not. This is taking way too much time away from other priorities. As should be clear by now, the government has not decided to block blogs. This is a case of mass ISP incompetence (or intentional goof so as to raise awareness, the conspiracy theorist in me wants to believe). My article made it to several editions of the Times of India, even making front page in the Hyderabad edition. I haven’t seen it yet. It’s not in the online edition and ePaper isn’t working for me just now. Kamla Bhatt did a podcast on the affair last night. Amit Agarwal, Neha Viswanathan, Suresh Ramasubramanian and I were interviewed.

Update 8 – 3:30 PM: My article appeared in the Times of India in Hyderabad (front page!), Bangalore (page 9), Mumbai (page 12), Delhi (page 15) and Lucknow (page 11). Because each edition was differently edited depending on space constraints, here is the full length version.

Update 9 – 8:40 PM: The Indian Consulate in NYC has offered an explanation:

From: A.R.Ghanashyam <dcg@[snip]>

A two-page write up containing extremely derogatory references to Islam and the holy prophet which had the potential to inflame religious sensitivities in India and create serious law and order problems in the country appeared in a blog facilitated by well known search engines. The matter was immediately taken note of by our CERT (Computer Emergency Response Team) and the Department of Telecommunications (DOT) was informed of it. The DOT took up the matter forthwith with the search engines and instructions were also issued to all Internet providers to block the two impertinent pages. Because of a technological error, the Internet providers went beyond what was expected of them which in turn resulted in the unfortunate blocking of all blogs. Department of Telecommunications have now clarified the issue and the error is being rectified and it is expected that normalcy in respect of blogs will soon be restored.

Update 10 – 11:05 PM: Shivam Vij calls the bluff. None of the blocked sites appear to have anything to do with threatening the national interest.

HinduUnity.org and HinduHumanRights.org are among 17 sites sought to be blocked, on the grounds that they are spreading Hindu nationalist propaganda. Accessed through an anonymizer, HinduUnity.org was found to have articles against Congress party President Sonia Gandhi and Indian Muslims. It also had a ‘hit list’ of people it considered anti-Hindu.

Another site on the list is Rahulyadav.com, set up by a US-based person who calls himself a member of the Vishwa Hindu Parishad. Dalitstan.org, on the other hand, calls itself a ‘human rights organisation working for the upliftment of Dalits.’

None of the sites seem to possess any direct security threat to India, or have any connection with the recent Mumbai blasts. Even more bizarre are the blogs sought to be blocked. ‘Princess Kimberley’ is a defunct blog with just two postings in 2004 about an American teenager’s depressing life. ‘Commonfolk Commonsense’ is a Chinese language blog, while 'Exposing the Left' is written by someone in Southern Illinois!

CNN-IBN covered the issue in a news segment at 10 PM. Peter Griffin posted a scan of a fax of the order asking for the sites to be blocked:

1. http://www.hinduunity.org
2. http://mypetjawa.mu.nu
3. http://pajamaeditors.blogspot.com
4. http://exposingtheleft.blogspot.com
5. http://thepiratescove.us
6. http://commonfolkcommonsense.blogspot.com
7. http://bamapachyderm.com
8. http://prinesskimberly.blogspot.com
9. http://merrimusings.typepad.com
10. http://mackers-world.com
11. http://dalitstan.com
12. http://hinduhumanrights.org/hindufocus.html
13. http://nndh.com (fax scan unclear, could be wrong)
14. http://bloodroyaltriped.com
15. http://imagessearchyahoo.com
16. http://imamali8.com
17. http://rahulyadav.com

Number 15 on that list, imagessearchyahoo.com, is a typo for image.search.yahoo.com, Yahoo!’s image search site. The typo domain is also owned by Yahoo!. Searching for images via a typo of the domain name is against the national interest? I’d love to see how the government justifies this one.

Update 11 – July 20, 8:15 AM: Gopal Sankaranarayan, a lawyer, is upset that if the block is being lifted as reported, it will kill the momentum to file a PIL to ensure such blocks do not happen again. He’s right. It’s not just the petition, but any form of coordination against this sort of thing happening again. (More.)

Update 12 – 9:00 PM: DoT now says ISPs are at fault for blocking more than necessary and demands an explanation. ISPAI in return tells DoT that blocking has practical constraints. Full story. DoT also claims the order to block came from the Department of Information Technology (DIT). DoT secretary D S Mathur says “DoT is the licensor and the controlling agency for ISPs, and when we get a request from DIT to block sites, we have to act accordingly.”

Update 13 – 11:30 PM: Arka Mukhopadhyay is mailing public intellectuals to register their protest. She has confirmation from Dr U R Ananthamurthy (bypass block). This is a novel method, one that hadn’t occurred to me before. For it to be effective, however, we must take their voices beyond blogs, into public consciousness. I should note here that the bypass link this time is via pkblogs.com, a resource set up by Pakistani bloggers when fighting censorship in their country and now usable in India too. They were gracious enough to share their code. Dr Awab Alvi, Sabahat Iqbal Ashraf (bypass) and Omer Alvie (bypass) have been fairly active helping out with the Indian blockade.

Update 14 – July 22, 11:30 PM: Just a notice that I’m no longer adding to this post. Updates will be in new posts henceforth.