Windows

• (3.2.10a1) As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO when multiple processes try to create an "asyncore trigger" simultaneously. ZEO/zrpc/trigger.py was replaced by ZODB 3.4's version of that file, to work around this bug when it occurs.

ZEO

• (3.2.10b1) Collector 1900. In some cases of pickle exceptions raised by low-level ZEO communication code, callers of marshal.encode() could attempt to catch an exception that didn't actually exist, leading to an erroneous AttributeError exception. Thanks to Tres Seaver for the diagnosis.

BaseStorage

• (3.2.10b2) Nothing done by tpc_abort() should raise an exception. However, if something does (an error case), BaseStorage.tpc_abort() left the commit lock in the acquired state, causing any later attempt to commit changes hang.

BTrees

• (3.2.10a1) Collector 1843. When a non-integer was passed to a method like keys() of a Bucket or Set with integer keys, an internal error code was overlooked, leading to everything from "delayed errors" to segfaults. Such cases raise TypeError now, as intended.

Tools

• (3.2.10a1) Collector #1846: If an uncommitted transaction was found, fsrecover.py fell into an infinite loop.

Subtransactions

• (3.2.9b1) Doing a subtransaction commit erroneously processed invalidations, which could lead to an inconsistent view of the database. For example, let T be the transaction of which the subtransaction commit was a part. If T read a persistent object O's state before the subtransaction commit, did not commit new state of its own for O during its subtransaction commit, and O was modified before the subtransaction commit by a different transaction, then the subtransaction commit processed an invalidation for O, and the state T read for O originally was discarded in T. If T went on to access O again, it saw the newly committed (by a different transaction) state for O:

  o_attr = O.some_attribute
  get_transaction().commit(True)
  assert o_attr == O.some_attribute
  

  could fail, and despite that T never modified O.

FileStorage.UndoSearch

• (3.2.9b1) The _readnext() method now returns the transaction size as the value of the "size" key. Thanks to Dieter Maurer for the patch, from http://mail.zope.org/pipermail/zodb-dev/2003-October/006157.html. "This is very valuable when you want to spot strange transaction sizes via Zope's 'Undo' tab".

Error reporting

• (3.2.9b3) Collector #1831: The BTree minKey() and maxKey() methods gave a misleading message if no key satisfying the constraints existed in a non-empty tree.
• (3.2.9b1) In the unlikely event that referencesf() reports an unpickling error (for example, a corrupt database can cause this), the message it produces no longer contains unprintable characters.

BTrees interface

• (3.2.9b2) Collector 1829. Clarified that the minKey() and maxKey() methods raise an exception if no key exists satsifying the constraints.

ZopeUndo

The ZopeUndo.Prefix class is actually part of ZODB, for the benefit of ZEO servers talking with Zope application servers. A bugfix to Prefix comparison in 3.2.7 had the unfortunate side effect of changing the set of attributes on a Prefix instance, so that trying to mix a pre-3.2.7 ZEO client with a 3.2.7 ZEO server, or vice versa, could ship a Prefix instance from the ZEO client that the ZEO server could not make sense of. The ZEO server could then disconnect the client due to an AttributeError when trying to reference a non-existent attribute of Prefix. The implementation of ZopeUndo.Prefix has been changed to tolerate mixing old and new in both directions. Thanks to Tres Seaver for the diagnosis.

DemoStorage

Appropriate implementations of the storage API's registerDB() and new_oid() methods were added, delegating to the base storage. This was needed to support wrapping a ZEO client storage as a DemoStorage base storage, as some new Zope tests want to do.

BaseStorage

new_oid()'s undocumented last= argument was removed. It was used only for internal recursion, and injured code sanity elsewhere because not all storages included it in their new_oid()'s signature. Straightening this out required adding last= everywhere, or removing it everywhere. Since recursion isn't actually needed, and there was no other use for last=, removing it everywhere was the obvious choice.

ZEO on Windows

Thanks to Mark Hammond for these runzeo.py enhancements on Windows:

• A pid file (containing the process id as a decimal string) is created now for a ZEO server started via runzeo.py. External programs can read the pid from this file and derive a "signal name" used in a new signal-emulation scheme for Windows. This is only necessary on Windows, but the pid file is created on all platforms that implement os.getpid(), as long as the pid-filename option is set, or environment variable INSTANCE_HOME is defined. The pid-filename option can be set in a ZEO config file, or passed as the new --pid-file argument to runzeo.py.
• If available, runzeo.py now uses Zope's new 'Signal' mechanism for Windows, to implement clean shutdown and log rotation handlers for Windows. Note that the Python in use on the ZEO server must also have the Python Win32 extensions installed for this to be useful.

BTrees

Collector #1734: BTrees conflict resolution leads to index inconsistencies.

Silent data loss could occur due to BTree conflict resolution when one transaction T1 added a new key to a BTree containing at least three buckets, and a concurrent transaction T2 deleted all keys in the bucket to which the new key was added. Conflict resolution then created a bucket containing the newly added key, but the bucket remained isolated, disconnected from the BTree. In other words, the committed BTree didn't contain the new key added by T1. Conflict resolution doesn't have enough information to repair this, so ConflictError is now raised in such cases.

Tests

The various flavors of the check2ZODBThreads and check7ZODBThreads tests are much less likely to suffer sproadic failures now.

Documentation

Persistent classes should not define __del__() methods. This has been documented in the ZODB 3.3 programming guide for over a year, but those changes didn't get copied over to the 3.2 version of the guide.

ZEO

Repaired subtle race conditions in establishing ZEO connections, both client- and server-side. These account for intermittent cases where ZEO failed to make a connection (or reconnection), accompanied by a log message showing an error caught in asyncore and having a traceback ending with:

UnpicklingError: invalid load key, 'Z'.

or:

ZRPCError: bad handshake '(K\x00K\x00U\x0fgetAuthProtocol)t.'

or:

error: (9, 'Bad file descriptor')

or an AttributeError.

These were exacerbated when running the test suite, because of an unintended busy loop in the test scaffolding, which could starve the thread trying to make a connection. The ZEO reconnection tests may run much faster now, depending on platform, and should suffer far fewer (if any) intermittent "timed out waiting for storage to connect" failures.

FileStorage

• The .store() and .restore() methods didn't update the storage's belief about the largest oid in use when passed an oid larger than the largest oid the storage already knew about. Because .restore() in particular is used by copyTransactionsFrom(), and by the first stage of ZRS recovery, a large database could be created that believed the only oid in use was oid 0 (the special oid reserved for the root object). In rare cases, it could go on from there assigning duplicate oids to new objects, starting over from oid 1 again. This has been repaired. A new set_max_oid() method was added to the BaseStorage class so that derived storages can update the largest oid in use in a threadsafe way.

• A FileStorage's index file tried to maintain the index's largest oid as a separate piece of data, incrementally updated over the storage's lifetime. This scheme was more complicated than necessary, so was also more brittle and slower than necessary. It indirectly participated in a rare but critical bug: when a FileStorage was created via copyTransactionsFrom(), the "maximum oid" saved in the index file was always 0. Use that FileStorage, and it could then create "new" oids starting over at 0 again, despite that those oids were already in use by old objects in the database. Packing a FileStorage has no reason to try to update the maximum oid in the index file either, so this kind of damage could (and did) persist even across packing.

  The index file's maximum-oid data is ignored now, but is still written out so that .index files can be read by older versions of ZODB. Finding the true maximum oid is done now by exploiting that the main index is really a kind of BTree (long ago, this wasn't true), and finding the largest key in a BTree is inexpensive.

• A FileStorage's index file could be updated on disk even if the storage was opened in read-only mode. That bug has been repaired.

• An efficient maxKey() implementation was added to class fsIndex.

Pickle (in-memory Connection) Cache

You probably never saw this exception:

ValueError: Can not re-register object under a different oid

It's been changed to say what it meant:

ValueError: A different object already has the same oid

This happens if an attempt is made to add distinct objects to the cache that have the same oid (object identifier). ZODB should never do this, but it's possible for application code to force such an attempt.

Release

Collector #1676: zodb.pdf in the release tarball was damaged (truncated), because CVS mistakenly believed it was a text file.

fsIndex

An efficient maxKey() method was implemented for the fsIndex class. This makes it possible to determine the largest oid in a FileStorage index efficiently, directly, and reliably, replacing a more delicate scheme that tried to keep track of this by saving an oid high water mark in the index file and incrementally updating it.

ZEO

Collector 1503: excessive logging. It was possible for a ZEO client to log "waiting for cache verification to finish" messages at a very high rate, producing gigabytes of such messages in short order. ClientStorage._wait_sync() was changed to log no more than one such message per 5 minutes.

FileStorage

Collector 1581: When an attempt to pack a corrupted Data.fs file was made, it was possible for the pack routine to die with a reference to an undefined global while it was trying to raise CorruptedError. It raises CorruptedError, as it always intended, in these cases now.

ExtensionClass

The deallocation function for ExtensionClass instances wasn't entirely correct when an instance's __del__ method resurrected the instance. This could lead to segfaults, but only under a debug build of Python and ExtensionClass.

ZEO

• The encoding of RPC calls between server and client was being done with protocol 0 ("text mode") pickles, which could require sending four times as many bytes as necessary. Protocol 1 pickles are used now. Thanks to Andreas Jung for the diagnosis and cure.
• Collector 1541: small changes to zrpc were backported from ZODB 3.3, in order to play nice with asyncore changes in Python 2.4.

ReadConflictError

An explicit transaction abort() wasn't clearing connections' memory of ReadConflictErrors raised during the transaction, and so new changes to objects that suffered ReadConflictError still couldn't be committed in the new transaction either. Thanks to Chris McDonough for the diagnosis and a focused test case.

ConflictError

The undocumented get_old_serial() and get_new_serial() methods were swapped (the first returned the new serial, and the second returned the old serial).

Connection

New exception ConnectionStateError is raised if an attempt is made to close a connection while modifications are pending (the connection is involved in a transaction that hasn't been abort()'ed or commit()'ed). Otherwise, because Connection objects are reused from a pool, and retain their caches across closing and (re)opening, opening a Connection could pick up data from a previous, "abandoned" transaction. In the worst case of this, if a subtransaction was abandoned, close+reopen could leave the subtransaction changes in cache, but not reflected anywhere in a storage, and POSKeyErrors could result. See Collector issue 789 for one nasty instance of this.

There are two flavors of ConnectionStateError that may be raised:

"Cannot close a connection joined to a transaction"
"Cannot close a connection with a pending subtransaction"

The first kind means that changes are pending outside of a subtransaction. The wording is a bit strange in ZODB 3.2; it's phrased this way so that the message is the same as in ZODB 3.3, where the wording makes better sense (3.3 has a different transaction model, where data managers (like Connection) "join" transactions).

Avoiding the new exception should be easy: if you start a transaction, finish it (do a commit() or an abort()) before closing any connections it involves. If you see the new exception in "working" code, that code was working by accident before.

Transactions

Transaction.begin() is supposed to imply Transaction.abort(), but if the only pending changes were in subtransactions, the subtransactions were not aborted by begin(). This has been repaired.

If ReadConflictError was raised by an attempt to load an object with a _p_independent() method that returned false, attempting to commit the transaction failed to (re)raise ReadConflictError for that object. Note that ZODB intends to prevent committing a transaction in which a ReadConflictError occurred; this was an obscure case it missed.

POSException

Collector #1488 (TemporaryStorage -- going backward in time). This confusion was really due to that the detail on a ConflictError exception didn't make sense. It called the current revision "was", and the old revision "now". The detail is much more informative now. For example, if the exception said:

ConflictError: database conflict error (oid 0xcb22,
serial was 0x03441422948b4399, now 0x034414228c3728d5)

before, it now says:

ConflictError: database conflict error (oid 0xcb22,
serial this txn started with 0x034414228c3728d5 2002-04-14 20:50:32.863000,
serial currently committed 0x03441422948b4399 2002-04-14 20:50:34.815000)

Tools

ZODB.utils.oid_repr() changed to add a leading "0x", and to strip leading zeroes. This is used, e.g., in the detail of a POSKeyError exception, to identify the missing oid. Before, the output was ambiguous. For example, oid 17 was displayed as 0000000000000011. As a Python integer, that's octal 9. Or was it meant to be decimal 11? Or was it meant to be hex? Now it displays as 0x11.

fsrefs.py:

When run with -v, produced tracebacks for objects whose creation was merely undone. This was confusing. Tracebacks are now produced only if there's "a real" problem loading an oid.

If the current revision of object O refers to an object P whose creation has been undone, this is now identified as a distinct case.

Captured and ignored most attempts to stop it via Ctrl+C. Repaired.

Now makes two passes, so that an accurate report can be given of all invalid references.

analyze.py produced spurious "len of unsized object" messages when finding a data record for an object uncreation or version abort. These no longer appear.

ZEO

The ZEO client cache implementation didn't check for temporary cache files correctly, sometimes creating named files in the filesystem by mistake (with names like 1-None-0). This could lead to filesystem permission problems, and other nasty effects. Thanks to Dieter Maurer for the diagnosis and the fix.

Storages

Collector #1327: FileStorage init confused by time travel

If the system clock "went backwards" a long time between the times a FileStorage was closed and reopened, new transaction ids could be smaller than transaction ids already in the storage, violating a key invariant. Now transaction ids are guaranteed to be increasing even when this happens. If time appears to have run backwards at all when a FileStorage is opened, a new message saying so is logged at warning level; if time appears to have run backwards at least 30 minutes, the message is logged at panic level (and you should investigate to find and repair the true cause).

Tools

fsrefs.py: Fleshed out the module docstring, and repaired a bug wherein spurious error msgs could be produced after reporting a problem with an unloadable object.

Test suite

Collector #1397: testTimeStamp fails on FreeBSD

The BSD distributions are unique in that their mktime() implementation usually ignores the input tm_isdst value. Test checkFullTimeStamp() was sensitive to this platform quirk.

Storages

Collector #1379: Undo + FileStorage.restore. In the presence of multiple-object transactions involving undo or versions, the FileStorage.restore() method could raise an inappropiate exception (bad argument to file.seek()), or silently fail to find the data it was looking for. Base Zope makes no use of restore(), but this method is crucial to ZRS recovery, so this is a critical bugfix at least for sites running ZRS. Thanks to Tres Seaver and Jim Fulton for the diagnosis and cure.

As announced with 3.2.2b1, the Berkeley-based storages are no longer supported. The code for them still exists, but the Berkeley tests have been disabled (primarily to prevent segfaults on Linux systems with newer versions of Sleepycat Berkeley DB).

ZODB

Collector #1208: Infinite loop in cPickleCache. If a persistent object had a __del__ method (probably not a good idea regardless, but we don't prevent it) that referenced an attribute of self, the code to deactivate objects in the cache could get into an infinite loop: ghostifying the object could lead to calling its __del__ method, the latter would load the object into cache again to satsify the attribute reference, the cache would again decide that the object should be ghostified, and so on. The infinite loop no longer occurs, but note that objects of this kind still aren't sensible (they're effectively immortal). Thanks to Toby Dickenson for suggesting a nice cure.

Under exceedingly unlikely conditions, it was possible for DB.open() to block with as few as two threads active. This was revealed by the intermittent failure of the checkConcurrentUpdates1Storage test on a hyper-threaded 3.2GHz P4 machine. That test has never been reported to fail elsewhere, and didn't fail on that box either if hyper-threading was disabled in the BIOS. It was a legitimate thread race, but without hyper-threading it was extremely unlikely to happen.

Storages

Doc/storages.html changed to record that the experimental BDB-based storages are unsupported and will be dropped in ZODB 3.3. This is due to lack of visible interest, and of resources (whether internal or external) for support of this code.

Tools

When trying to do recovery to a time earlier than that of the most recent full backup, repozo.py failed to find the appropriate files, erroneously claiming "No files in repository before <specified time>". This has been repaired.

Collector #1330: repozo.py -R can create corrupt .fs. When looking for the backup files needed to recreate a Data.fs file, repozo could (unintentionally) include its meta .dat files in the list, or random files of any kind created by the user in the backup directory. These would then get copied verbatim into the reconstructed file, filling parts with junk. Repaired by filtering the file list to include only files with the data extensions repozo.py creates (.fs, .fsz, .deltafs, and .deltafsz). Thanks to James Henderson for the diagnosis.

BTrees

Collector #553: sporadic Zope crashes In rare circumstances, it was possible for BTree conflict resolution to be handed three empty BTrees (code needed to fetch an empty BTree from the database, add keys to it, then remove them all again, before committing the transaction). This could cause a memory fault in C code.

FileStorage

Fixed a serious bug in the new pack implementation. If pack was called on the storage and passed a time earlier than a previous pack time, data could be lost. In other words, if there are any two pack calls, where the time argument passed to the second call was earlier than the first call, data loss could occur. The bug was fixed by causing the second call to raise a StorageError before performing any work.

ZEO

Fixed a bug in the ZEO authentication implementation that could cause a properly authenticated connection to fail unexpectedly with a ValueError -- "Received message without HMAC."

Fixed a bug in the ZEO server's getInvalidations() method. If it had the requested invalidations, it would send all the invalidations in its queue rather than just the requested ones. This didn't affect correctness of the cache, but did cause it to throw out valid data.

Fixed a bug that prevented the -m / --monitor argument from working.

Fixed and enhanced the zeopasswd.py script. It now runs as a script and supports command-line arguments for all the options that can be specified in the configuration file.

ZODB

Invalidations are now processed atomically. Each transaction will see all the changes caused by an earlier transaction or none of them. Before this patch, it was possible for a transaction to see invalid data because it saw only a subset of the invalidations. This is the most likely cause of reported BTrees corruption, where keys were stored in the wrong bucket. When a BTree bucket splits, the bucket and the bucket's parent are both modified. If a transaction sees the invalidation for the bucket but not the parent, the BTree in memory will be internally inconsistent and keys can be put in the wrong bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose the traceback that caused the failure. The transaction code was fixed to report the original error as well as any errors that occur while trying to recover from the original error.

The "other" argument to copyTransactionsFrom() only needs to have an .iterator() method. For convenience, change FileStorage's and BDBFullStorage's iterator to have this method, which just returns self.

Mount points are now visible from mounted objects.

Fixed memory leak involving database connections and caches. When a connection or database was closed, the cache and database leaked, because of a circular reference involving the cache. Fixed the cache to explicitly clear out its contents when its connection is closed.

The ZODB cache has fewer methods. It used to expose methods that could mutate the dictionary, which allowed users to violate internal invariants.

ZConfig

It is now possible to configure ZODB databases and storages and ZEO servers using ZConfig.

ZEO & zdaemon

ZEO now supports authenticated client connections. The default authentication protocol uses a hash-based challenge-response protocol to prove identity and establish a session key for message authentication. The architecture is pluggable to allow third-parties to developer better authentication protocols.

There is a new HOWTO for running a ZEO server. The draft in this release is incomplete, but provides more guidance than previous releases. See the file Doc/ZEO/howto.txt.

The ZEO storage server's transaction timeout feature was refactored and made slightly more rebust.

A new ZEO utility script, ZEO/mkzeoinst.py, was added. This creates a standard directory structure and writes a configuration file with mostly default values, and a bootstrap script that can be used to manage and monitor the server using zdctl.py (see below).

Much work was done to improve zdaemon's zdctl.py and zdrun.py scripts. (In the alpha 1 release, zdrun.py was called zdaemon.py, but installing it in <prefix>/bin caused much breakage due to the name conflict with the zdaemon package.) Together with the new mkzeoinst.py script, this makes controlling a ZEO server a breeze.

A ZEO client will not read from its cache during cache verification. This fix was necessary to prevent the client from reading inconsistent data.

The isReadOnly() method of a ZEO client was fixed to return the false when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc connection now do both input and output.

The short_repr() function used to generate log messages was fixed so that it does not blow up creating a repr of very long tuples.

Storages

FileStorage has a new pack() implementation that fixes several reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage. undoLog() did not handle its arguments correctly and pack() could accidentally delete objects created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed. It used to stop after the first object, leaving all the other objects in the version.

BTrees

Trying to store an object of a non-integer type into an IIBTree or OIBTree could leave the bucket in a variety of insane states. For example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test suite, but could have been much worse (most likely in real life is that a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one child, a reference to the second child leaked. This could cause the entire bucket chain to leak (not be collected as garbage despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Tools

New tool zeoqueue.py for parsing ZEO log files, looking for blocked transactions.

New tool repozo.py (originally by Anthony Baxter) for performing incremental backups of Data.fs files.

The fsrecover.py script now does a better job of recovering from errors the occur in the middle of a transaction record. Fixed several bugs that caused partial or total failures in earlier versions.

ZODB

An experimental new transaction API was added. The Connection class has a new method, setLocalTransaction(). ZODB applications can call this method to bind transactions to connections rather than threads. This is especially useful for GUI applications, which often have only one thread but multiple independent activities within that thread (generally one per window). Thanks to Christian Reis for championing this feature.

Applications that take advantage of this feature should not use the get_transaction() function. Until now, ZODB itself sometimes assumed get_transaction() was the only way to get the transaction. Minor corrections have been added. The ZODB test suite, on the other hand, can continue to use get_transaction(), since it is free to assume that transactions are bound to threads.

ZEO

There is a new recommended script for starting a storage server. We recommend using ZEO/runzeo.py instead of ZEO/start.py. The start.py script is still available in this release, but it will no longer be maintained and will eventually be removed.

There is a new zdaemon implementation. This version is a separate script that runs an arbitrary daemon. To run the ZEO server as a daemon, you would run "zdrun.py runzeo.py". There is also a simple shell, zdctl.py, that can be used to manage a daemon. Try "zdctl.py -p runzeo.py".

There is a new version of the ZEO protocol in this release and a first stab at protocol negotiation. (It's a first stab because the protocol checking supporting in ZODB 3.1 was too primitive to support anything better.) A ZODB 3.2 ZEO client can talk to an old server, but a ZODB 3.2 server can't talk to an old client. It's safe to upgrade all the clients first and upgrade the server last. The ZEO client cache format changed, so you'll need to delete persistent caches before restarting clients.

The ZEO cache verification protocol was revised to require many fewer messages in cases where a client or server restarts quickly.

The performance of full cache verification has improved dramatically. XXX Get measurements from Jim -- somewhere in 2x-5x recall. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely.

A bug was fixed in the ZEO protocol that allowed clients to read stale cache data while cache verification was being performed. The fixed version prevents the client from using the storage until after verification completes.

The ZEO server has an experimental monitoring interface that reports usage statistics for the storage server including number of connected clients and number of transactions active and committed. It can be enabled by passing the -m flag to runsvr.py.

The ZEO ClientStorage no longer supports the environment variables CLIENT_HOME, INSTANCE_HOME, or ZEO_CLIENT.

The ZEO1 package is still included with this release, but there is no longer an option to install it.

BTrees

The BTrees package now has a check module that inspects a BTree to check internal invariants. Bugs in older versions of the code code leave a BTree in an inconsistent state. Calling BTrees.check.check() on a BTree object should verify its consistency. (See the NEWS section for 3.1 beta 1 below to for the old BTrees bugs.)

Fixed a rare conflict resolution problem in the BTrees that could cause an segfault when the conflict resolution resulted in any empty bucket.

Installation

The distutils setup now installs several Python scripts. The runzeo.py and zdrun.py scripts mentioned above and several fsXXX.py scripts from the Tools directory.

The test.py script does not run all the ZEO tests by default, because the ZEO tests take a long time to run. Use --all to run all the tests. Otherwise a subset of the tests, mostly using MappingStorage, are run.

Storages

There are two new storages based on Sleepycat's BerkeleyDB in the BDBStorage package. Barry will have to write more here, because I don't know how different they are from the old bsddb3Storage storages. See Doc/BDBStorage.txt for more information.

It now takes less time to open an existing FileStorage. The FileStorage uses a BTree-based index that is faster to pickle and unpickle. It also saves the index periodically so that subsequent opens will go fast even if the storage was not closed cleanly.

Misc

The new ZConfig package, which will be used by Zope and ZODB, is included. ZConfig provides a configuration syntax, similar to Apache's syntax. The package can be used to configure the ZEO server and ZODB databases. See the module ZODB.config for functions to open the database from configuration. See ZConfig/doc for more info.

The zLOG package now uses the logging package by Vinay Sajip, which will be included in Python 2.3.

The Sync extension was removed from ExtensionClass, because it was not used by ZODB.

ZODB

Invalidations are now processed atomically. Each transaction will see all the changes caused by an earlier transaction or none of them. Before this patch, it was possible for a transaction to see invalid data because it saw only a subset of the invalidations. This is the most likely cause of reported BTrees corruption, where keys were stored in the wrong bucket. When a BTree bucket splits, the bucket and the bucket's parent are both modified. If a transaction sees the invalidation for the bucket but not the parent, the BTree in memory will be internally inconsistent and keys can be put in the wrong bucket. The atomic invalidation fix prevents this problem.

A number of minor reference count fixes in the object cache were fixed. That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose the traceback that caused the failure. The transaction code was fixed to report the original error as well as any errors that occur while trying to recover from the original error.

ZEO

A ZEO client will not read from its cache during cache verification. This fix was necessary to prevent the client from reading inconsistent data.

The isReadOnly() method of a ZEO client was fixed to return the false when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc connection now do both input and output.

The short_repr() function used to generate log messages was fixed so that it does not blow up creating a repr of very long tuples.

Storages

FileStorage has a new pack() implementation that fixes several reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage. undoLog() did not handle its arguments correctly and pack() could accidentally delete objects created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed. It used to stop after the first object, leaving all the other objects in the version.

BTrees

Trying to store an object of a non-integer type into an IIBTree or OIBTree could leave the bucket in a variety of insane states. For example, trying

b[obj] = "I'm a string, not an integer"

where b is an OIBTree. This manifested as a refcount leak in the test suite, but could have been much worse (most likely in real life is that a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one child, a reference to the second child leaked. This could cause the entire bucket chain to leak (not be collected as garbage despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Other

Comparing a Missing.Value object to a C type that provide its own comparison operation could lead to a segfault when the Missing.Value was on the right-hand side of the comparison operator. The Missing class was fixed so that its coercion and comparison operations are safe.

Tools

Four tools are now installed by setup.py: fsdump.py, fstest.py, repozo.py, and zeopack.py.

Tools

Updated repozo.py tool

ZEO

The ZEO version number was bumped to 2.0.2 on account of the following minor feature additions.

The performance of full cache verification has improved dramatically. XXX Get measurements from Jim -- somewhere in 2x-5x recall. The implementation was fixed to use the very-fast getSerial() method on the storage instead of the comparatively slow load().

The ZEO server has an optional timeout feature that will abort a connection that does not commit within a certain amount of time. The timeout works by closing the socket the client is using, causing both client and server to abort the transaction and continue. This is a drastic step, but can be useful to prevent a hung client or other bug from blocking a server indefinitely.

If a client was disconnected during a transaction, the tpc_abort() call did not properly reset the internal state about the transaction. The bug caused the next transaction to fail in its tpc_finish(). Also, any ClientDisconnected exceptions raised during tpc_abort() are ignored.

ZEO logging has been improved by adding more logging for important events, and changing the logging level for existing messages to a more appropriate level (usually lower).

New ZODB cache

Toby Dickenson implemented a new Connection cache for ZODB. The cache is responsible for pointer swizzling (translating between oids and Python objects) and for keeping recently used objects in memory. The new cache is a big improvement over the old cache. It strictly honors its size limit, where size is specified in number of objects, and it evicts objects in least recently used (LRU) order.

Users should take care when setting the cache size, which has a default value of 400 objects. The old version of the cache often held many more objects than its specified size. An application may not perform as well with a small cache size, because the cache no longer exceeds the limit.

Storages

The index used by FileStorage was reimplemented using a custom BTrees object. The index maps oids to file offsets, and is kept in memory at all times. The new index uses about 1/4 the memory of the old, dictionary-based index. See the module ZODB.fsIndex for details.

A security flaw was corrected in transactionalUndo(). The transaction ids returned by undoLog() and used for transactionalUndo() contained a file offset. An attacker could construct a pickle with a bogus transaction record in its binary data, deduce the position of the pickle in the file from the undo log, then submit an undo with a bogus file position that caused the pickle to get written as a regular data record. The implementation was fixed so that file offsets are not included in the transaction ids.

Several storages now have an explicit read-only mode. For example, passing the keyword argument read_only=1 to FileStorage will make it read-only. If a write operation is performed on a read-only storage, a ReadOnlyError will be raised.

The storage API was extended with new methods that support the Zope Replication Service (ZRS), a proprietary Zope Corp product. We expect these methods to be generally useful. The methods are:

• restore(oid, serialno, data, version, transaction)

  Perform a store without doing consistency checks. A client can use this method to provide a new current revision of an object. The serialno argument is the new serialno to use for the object, not the serialno of the previous revision.

• lastTransaction()

  Returns the transaction id of the last committed transaction.

• lastSerial(oid)

  Return the current serialno for oid or None.

• iterator(start=None, stop=None)

  The iterator method isn't new, but the optional start and stop arguments are. These arguments can be used to specify the range of the iterator -- an inclusive range [start, stop].

FileStorage is now more cautious about creating a new file when it believes a file does not exist. This change is a workaround for bug in Python versions upto and including 2.1.3. If the interpreter was builtin without large file support but the platform had it, os.path.exists() would return false for large files. The fix is to try to open the file first, and decide whether to create a new file based on errno.

The undoLog() and undoInfo() methods of FileStorage can run concurrently with other methods. The internal storage lock is released periodically to give other threads a chance to run. This should increase responsiveness of ZEO clients when used with ZEO 2.

New serial numbers are assigned consistently for abortVersion() and commitVersion(). When a version is committed, the non-version data gets a new serial number. When a version is aborted, the serial number for non-version data does not change. This means that the abortVersion() transaction record has the unique property that its transaction id is not the serial number of the data records.

Berkeley Storages

Berkeley storage constructors now take an optional config argument, which is an instance whose attributes can be used to configure such BerkeleyDB policies as an automatic checkpointing interval, lock table sizing, and the log directory. See bsddb3Storage/BerkeleyBase.py for details.

A getSize() method has been added to all Berkeley storages.

Berkeley storages open their environments with the DB_THREAD flag.

Some performance optimizations have been implemented in Full storage, including the addition of a helper C extension when used with Python 2.2. More performance improvements will be added for the ZODB 3.1 final release.

A new experimental Autopack storage was added which keeps only a certain amount of old revision information. The concepts in this storage will be folded into Full and Autopack will likely go away in ZODB 3.1 final. ZODB 3.1 final will also have much improved Minimal and Full storages, which eliminate Berkeley lock exhaustion problems, reduce memory use, and improve performance.

It is recommended that you use BerkeleyDB 4.0.14 and PyBSDDB 3.4.0 with the Berkeley storages. See bsddb3Storage/README.txt for details.

BTrees

BTrees no longer ignore exceptions raised when two keys are compared.

Tim Peters fixed several endcase bugs in the BTrees code. Most importantly, after a mix of inserts and deletes in a BTree or TreeSet, it was possible (but unlikely) for the internal state of the object to become inconsistent. Symptoms then varied; most often this manifested as a mysterious failure to find a key that you knew was present, or that tree.keys() would yield an object that disgreed with the tree about how many keys there were.

If you suspect such a problem, BTrees and TreeSets now support a ._check() method, which does a thorough job of examining the internal tree pointers for consistency. It raises AssertionError if it finds any problems, else returns None. If ._check() raises an exception, the object is damaged, and rebuilding the object is the best solution. All known ways for a BTree or TreeSet object to become internally inconsistent have been repaired.

Other fixes include:

• Many fixes for range search endcases, including the "range search bug:" If the smallest key S in a bucket in a BTree was deleted, doing a range search on the BTree with S on the high end could claim that the range was empty even when it wasn't.
• Zope Collector #419: repaired off-by-1 errors and IndexErrors when slicing BTree-based data structures. For example, an_IIBTree.items()[0:0] had length 1 (should be empty) if the tree wasn't empty.
• The BTree module functions weightedIntersection() and weightedUnion() now treat negative weights as documented. It's hard to explain what their effects were before this fix, as the sign bits were getting confused with an internal distinction between whether the result should be a set or a mapping.

ZEO

For news about ZEO2, see the file ZEO-NEWS.txt.

This version of ZODB ships with two different versions of ZEO. It includes ZEO 2.0 beta 1, the recommended new version. (ZEO 2 will reach final release before ZODB3.) The ZEO 2.0 protocol is not compatible with ZEO 1.0, so we have also included ZEO 1.0 to support people already using ZEO 1.0.

Other features

When a ConflictError is raised, the exception object now has a sensible structure, thanks to a patch from Greg Ward. The exception now uses the following standard attributes: oid, class_name, message, serials. See the ZODB.POSException.ConflictError doc string for details.

It is now easier to customize the registration of persistent objects with a transaction. The low-level persistence mechanism in cPersistence.c registers with the object's jar instead of with the current transaction. The jar (Connection) then registers with the transaction. This redirection would allow specialized Connections to change the default policy on how the transaction manager is selected without hacking the Transaction module.

Empty transactions can be committed without interacting with the storage. It is possible for registration to occur unintentionally and for a persistent object to compensate by making itself as unchanged. When this happens, it's possible to commit a transaction with no modified objects. The change allows such transactions to finish even on a read-only storage.

Two new tools were added to the Tools directory. The analyze.py script, based on a tool by Matt Kromer, prints a summary of space usage in a FileStorage Data.fs. The checkbtrees.py script scans a FileStorage Data.fs. When it finds a BTrees object, it loads the object and calls the _check method. It prints warning messages for any corrupt BTrees objects found.

Documentation

The user's guide included with this release is still woefully out of date.

Other bugs fixed

If an exception occurs inside an _p_deactivate() method, a traceback is printed on stderr. Previous versions of ZODB silently cleared the exception.

ExtensionClass and ZODB now work correctly with a Python debug build.

All C code has been fixed to use a consistent set of functions from the Python memory API. This allows ZODB to be used in conjunction with pymalloc, the default allocator in Python 2.3.

zdaemon, which can be used to run a ZEO server, more clearly reports the exit status of its child processes.

The ZEO server will reinitialize zLOG when it receives a SIGHUP. This allows log file rotation without restarting the server.