InterbaseStorage.txt

""" Zope Interbase Storage """

OVERVIEW

Zope Interbase Storage is a Zope storage that can make use of Borland/Inprise Interbase 6.0 (and possibly 5.0 and 4.0) RDBMS to store ZODB (Zope Object Data Base) information. It is a full-featured storage, supporting versioning and undo (although not yet Zope 2.2-style historying). It has not yet been benchmarked against other storages, but it is almost certainly slower than FileStorage. Nonetheless, its performance seems acceptable.

Using Zope backed by a relational storage such as this one has several advantages. If you're clever, using InterbaseStorage, you can bypass the arbitrary filesize limit on some OSes imposed by FileStorage by partitioning the relational database across several files. Additionally, you can use the failover and replication capabilities of the RDB to provide a higher level of availability to the storage. Also, some folks just "feel better" when things are stuffed inside a relational database, for whatever reason (think "management") as opposed to a FileStorage. InterbaseStorage also makes use of the transactional capabilies of Interbase, which should make it fairly impervious to data corruption (if it's been written correctly! ahem... :-)

To make use of the storage, you must install Interbase 6.0 which is freely available from http://www.interbase.com for the Linux, Windows, and Solaris platforms. It is available in binary-only format for these platforms (although it is supposed to be open-sourced soon). Interbase is a fully transactional relational database that bills itself as a database for embedded applications. It is solid and full- featured.

To use InterbaseStorage, you must also install Alexander Kuznetsov's excellent Kinterbasdb DB-API 1.0-compliant Python module, available from http://thor.prohosting.com/~alexan/Kinterbasdb. This module allows the Zope Interbase Storage module to talk to the Interbase server. Instructions on its installation are included with the Kinterbasdb tarball. Binaries for this module do not seem to be available (you need to compile it manually), and I have no idea whether it will work under Windows. If someone can try to compile it and test this, it'd be nice. It might mean that InterbaseStorage would work on a Windows box. At this time, though, if you want to use InterbaseStorage, you'll only be able to run it on a UNIX variant unless you can get Kinterbasdb working on Windows.

INSTALLATION

Prerequisites

First, install Interbase. Then install Kinterbasdb. Once you've got these two working together and you can access any Interbase DB from Python using Kinterbasdb, proceed. Information about using relational databases from Python is available at http://www.python.org/topics/database. This is probably the hardest step, and unfortunately I can't give you too much help, only having done it once myself. :-) Several newsgroups for the Interbase platform sponsored by Inprise exist and can be found via the http://www.interbase.com website. Questions about Kinterbasdb are sometimes answered on the main Zope mailing list.

As of this time, InterbaseStorage has only been verified to run on Linux against an Interbase database on the same machine which hosts Zope. Testing of InterbaseStorage in other configurations (multiple- file partitioned databases, databases configured for high availability, a database accessed across a network, whatever) is encouraged.

Creating an Interbase Database for Zope Storage Use

Unpack the InterbaseStorage product into your Zope directory. It will create a few files under $ZOPEHOME/lib/python/Products/InterbaseStorage.

First edit the $ZOPEHOME/lib/python/Products/InterbaseStorage/InterbaseStorage.sql file's CREATE DATABASE path argument so it contains the file path that you'd like to use as an InterbaseStorage database. This file will be created for you by the DDL contained in the InterbaseStorage.sql file, you don't need to create it manually.

To create the Interbase database, use Interbase's isql facility to process the DDL in the $ZOPEHOME/lib/python/Products/InterbaseStorage/InterbaseStorage.sql file, e.g.:

    /usr/interbase/bin/isql -i lib/python/Products/InterbaseStorage/InterbaseStorage.sql

This command will create an Interbase database suitable for use with Zope Interbase Storage. I'm not sure exactly how to access an Interbase DB across a network or as a replicated set of files, thus these instructions and the DDL permit only a single- file Interbase DB. Accessing an Interbase DB across a network or accessing a highly-available replicated Interbase DB has not yet been tested.

Creating Or Modifying Your Zope's custom_zodb.py File

Create or modify your $ZOPEHOME/custom_zodb.py file. This file's existence tells Zope to use a storage other than the default FileStorage that ships with Zope. It should exist in the Zope home directory (the directory containing start, and z2.py). This file's contents should be something like the following:

    from Products.InterbaseStorage.InterbaseStorage import InterbaseStorage
    import kinterbasdb
    db = /home/chrism/zope_storage.gdb
    username = chrism
    password = mypassword
    conn = kinterbasdb.connect(db,username,password)
    Storage = InterbaseStorage(conn=conn)

Note that for the import kinterbasdb line to work properly, you might need to mess around a little with your PYTHONPATH, or you might just want to install kinterbasdb into your Python installation's site-packages directory, which allows it to be imported without any PYTHONPATH changes. For more information about PYTHONPATH and Python modules in general, see http://www.python.org and search the online documentation. If you run into problems doing this, make sure to consult the Interbase and Kinterbasdb documentation before posting a message to the Zope mailing list.

You'll need to change the path pointing to the db (use the path you've used in the CREATE DATABASE line of the DDL in InterbaseStorage.sql), your Interbase username, and your Interbase password.

A custom_zodb.py.sample file has been included with this release.

Starting Zope and Using InterbaseStorage

Just start Zope like usual. The existence of the custom_zodb.py file tells Zope to try to use whatever storage is defined within it. A new Zope instance will be created within the InterbaseStorage and it will operate almost identically to other storages such as FileStorage, meaning you don't really have to think of it all that much past here.

There's no easy way to copy stuff from an existing FileStorage- backed Zope instance to one running InterbaseStorage. For now, the best you can do is to make use of Zope's export facility to export all the objects hanging off the root folder of a FileStorage- backed Zope, then import them one-by-one into a Zope instance backed by InterbaseStorage. You will lose properties of the objects in the root folder that you've exported this way that will need to be re-entered manually.

InterbaseStorage (as well as any other storage) is usable as a ZEO storage.

The size of the database as reported in Zope's Control_Panel is not the size of the disk file. It's instead an educated guess at the amount of actual ZODB-related data stored inside the database. In most cases, this works out to about one-half the size of the actual disk file for a single-file Interbase DB.

SUPPORT

InterbaseStorage is not supported in any official capacity by the publishers of Zope (Digital Creations). InterbaseStorage is essentially unsupported software at this time, though its author will attempt to resolve issues that arise in its use. The author makes no guarantees that this software won't completely munge your data.

Though InterbaseStorage is not an official part of Zope, support will be provided primarily via the main Zope mailing list. Information about the main Zope mailing list ([email protected]) is available at http://www.zope.org/Resources/MailingLists. Requests for information about InterbaseStorage sent directly to the authors or designers will probably be redirected to the Zope mailing list.

A WikiWeb has also been set up on Zope.org to discuss all things related to storages backed by relational databases. It can be visited at http://www.zope.org/Members/jim/ZODB/RelationalStorage.

Please read the HISTORY and TODO sections of this document before reporting problems or errors to the author or to the Zope mailing list.

OPTIONS

You can get semi-useful operational debugging information from InterbaseStorage by using an additional argument to the line in custom_zodb.py that instantiates the InterbaseStorage, ala:

      Storage = InterbaseStorage(conn=conn, debug=1)

This will cause operational debugging messages to be printed to the console that launched Zope.

You can also rename your Interbase tables to your liking. The default names of the three tables created are zodb_data (the main data table), zodb_trans (the transaction logging table), and zodb_pack (a temporary table used during pack).

To use different names for these tables, change the DDL provided and then pass in parameters during the instantiation of InterbaseStorage:

      Storage = InterbaseStorage(conn=conn,
                                 zodb_data=mydatatable,
                                 zodb_trans=mytranstable,
                                 zodb_pack=mypacktable)

This might be useful if you want to run two InterbaseStorages against the same Interbase database.

TODO

Optimizations via stored procedures and better SQL
Figure out why VersionLockErrors are getting masked.
Implement Zope 2.2 historying
Implement committing between versions
FileStorage-to-InterbaseStorage coversion facility.
Undoable undo (probably part of ZODB4).
More accurate getSize for Control_Panel reporting (need to figure out common-case algorithm for most common setups).
Possibly share a common base class with other relational database-backed storages.

CREDITS

Chris McDonough ([email protected]): design and implementation
Jonothan Farr ([email protected]): design/reference implementation (MySQL)
Evan Simpson ([email protected]): design
Tres Seaver ([email protected]): design

HISTORY

6/10/2000 (v. .98): Initial alpha release by chrism

Known bug on startup that causes medusa to raise an error and emit a traceback:

      Traceback (innermost last):
        File "/home/chrism/Zope2.2a/z2.py", line 599, in ?
          logger_object=lg)
        File "/home/chrism/Zope2.2a/ZServer/medusa/http_server.py", line 543, in __init__
          self.bind ((ip, port))
        File "/home/chrism/Zope2.2a/ZServer/medusa/asyncore.py", line 243, in bind
           return self.socket.bind (addr)
      TypeError: argument 1: expected string without null bytes, string found
      Exception exceptions.AttributeError: "None object has no attribute 'detach_db'" in  ignored

This error seems harmless and has no consequence on the operations of Zope or the storage and generally only happens on the first startup of a Zope using InterbaseStorage. It does not usually prevent Zope from starting (well.. once it did, but an immediate subsequent restart made things go OK). Please ignore it for now or help me figure it out :-).

Committing between versions is unimplemented (I didnt even know you could do that).
Historying is unimplemented.
Version locking works, but a VersionLockError isn't reported to the user after an exception. I have to figure out why this is getting masked. This is harmless, but can be confusing for the user.
Testing under Windows or any UNIX version other than Linux has not been done. There's no easy way to make this work under Windows because of its dependency on kinterbasdb, though I imagine if some enterprising soul wanted to provide precompiled binaries of kinterbasdb for Windows, it would work as advertised.
Testing of InterbaseStorage has been performed under both Zope 2.1.6 and a CVS checkout (after 6/5/2000) of Zope. CVS checkouts of Zope between 5/25/2000 - 6/5/2000 contain a bug in Transaction.py that prevents the storage from starting properly. If you're working from a CVS checkout, it's probably best to update your checkout to prevent this from biting you.