Recovering from hitting the 2 gig data base limit

Created by . Last modified on 2003/08/05.

There is a problem with having the Data.fs as one file and that is hitting the 2 gig file limit that occurs in many OS's and python.

The solutions are numerous, use mounted databases or external relational database to store data and so on. Some OS's dont have this problem Solaris, Windows NT/2000, AIX, and FreeBSD (too name a few). However that does not mean that python supports the 2 gig limit, mostly notably as I found out on Windows.

So here's the problem I was trying to solve recently, my database just hit 2 gigs, Zope crashes. What do you do? You can't start Zope to pack it, scripts such as fsrecover.py won't run because Zope won't start, you're hooped.

My first try was move the db to an os that supports large file system. We tried 64-bit Linux, where Python 2.0 supports large files, of course this means porting Zope to python 2.0 and 64-bit Linux. An interesting problem to do someday. Then I looked at Solaris, but we didnt have the right version running, so we'd have to set up a box etc.

I rummaged around Zope.org and found nothing, which is why Im creating this how-to. I came up with the idea of truncating the file and then running Zope and praying. I later found out this works.

1. Make a backup of your Data.fs.
2. Truncate the database using python (or perl).
• To truncate use python to chop 100 bytes off:
  

  file = "C:\\Zope\\Var\\Data.fs"
  f = open(file, "ab")
  f.truncate(2000000000) 
  f.close() 

• Be sure to use binary mode, otherwise your database will be totally screwed.
3. Restart Zope.
• The restart will take a long time.
• Zope will realise the db is corrupt and start at the beginning and reconstruct each record in the database.
• Since you did chop the datbase, there will be some data loss of the last transaction or so (depending upon how much you truncate).

Other references:
Recovering Corrupted Data.fs ZODB files, thanks itamar!