BrewingIdeas

Version and Data Tables Should Be Combined
------------------------------------------
A previous version of the relational storage table structures dictates that
version data should be kept in a separate table.  Because the most common
queries join this table with the data table, and because it doesn't seem
to buy us anything in the way of speed, it makes sense to just
consolidate the data and versions tables.  Here is a diagram of the proposed
table structure:



(image last updated:

This table structure should be modified to include a pickle table.
Object Creation, Versioning, and Undo State Chart
-------------------------------------------------
A record for an object needs the following properties:
serial (timestamp generated when record is written)
nv (non-version serial number)
pre (previous object serial number for undo)
dataserial (a serial number which contains the pickle if it isn't in
this record)
We need the following methods:
LoadMethod (implicit)
GetHistorical (implicit)
StoreMethod
FinishMethod
UndoMethod
TransactionExists
TransactionIsNotUndoable
MakeDummyUndoRecords
UndoPopulatedRecords
UndoUnpopulatedRecords
ClearUndoStatus
GetCurrentOids
GetCurrent
InsertObject
InsertTrans
SetStatus
CommitVersion
GetCurrentOidsInVersion
InsertCommittedVersion
ClearVersionStatus
One version of the Interbase tables left out the dataserial element.
This was a mistake.  It needs to be there.
We may have some record locking issues with the scheme outlined below.
Actually, object locking issues.  All writes to the DB other than pack
and status-setting stuff is an append.  There will be an issue if one
thread stores a copy of the same object another thread is trying to store
at the same time for whatever reason.  I'm not sure I understand it quite
fully, but Evan and Tres say that it can probably be co-opted at the
Interbase level.  If it can't, we'll need to forego the use of the
z_status flag in the zodb_data table and instead use in-mem indexes.
On to the homegrown state chart...
The Tedious Life Of An Object in the InterbaseStorage:
---------------------------------------
I.  A object gets created:
serial     nv     pre     dataserial
A          None   None    None
Assertion:  if the object is brand new, write it with no nv, no pre, and
no dataserial.
Methods used:
StoreMethod --> GetCurrent --> InsertObject --> FinishMethod --> InsertTrans
Things that happened in StoreMethod:
GetCurrent (oid) returned None
InsertObject (oid, newserial, "", "c", p, len(p), version, "", "")
Things that happened in FinishMethod:
InsertTrans (serial, u, d, e)
--------------------------------------
--------------------------------------
II.  It gets modified in a nonversion transaction and a new object takes
its place:
serial     nv     pre     dataserial
B          None   A       None
Assertion:  if a nonversion object is not new, write it with nv=None,
pre=oldserial and dataserial=None.
Methods used:
StoreMethod --> GetCurrent --> SetStatus --> InsertObject --> FinishMethod --> InsertTrans
Things that happened in StoreMethod:
GetCurrent (oid) returned A
SetStatus (oid, A's serial, "h")
InsertObject (oid, newserial, A's serial, "c", p, len(p), version, "", "")
Things that happened in FinishMethod:
InsertTrans (serial, u, d, e)
--------------------------------------
--------------------------------------
III.  Then the object is modified within a version for the first time:
serial     nv     pre     dataserial
C          B      None    None
Assertion:  if a version object is new within the version, write it with
nv=nonversion serial, pre=None, dataserial=None.
Methods used:
StoreMethod --> GetCurrent --> SetStatus --> InsertObject --> FinishMethod --> InsertTrans
Things that happened in StoreMethod:
GetCurrent (oid) returned B
SetStatus ("h", oid, B's serial)
InsertObject (oid, newserial, "", "c", p, len(p), version, B's serial, "")
Things that happened in FinishMethod:
InsertTrans (newserial, u, d, e)
--------------------------------------
--------------------------------------
IV.  The object is remodified within a version:
serial     nv     pre     dataserial
D          B      C       None
Assertion:  if a version object is not new within the version, write it
with nv=nonversion serial, pre=old current serial within version,
dataserial=None.
Methods used:
StoreMethod --> GetCurrent --> SetStatus --> InsertObject --> FinishMethod --> InsertTrans
Things that happened in StoreMethod:
GetCurrent (oid) returned C
SetStatus ("h", oid, c's serial)
InsertObject (oid, newserial, C's serial, "c", p, len(p), version, B's serial, "")
Things that happened in FinishMethod:
InsertTrans (newserial, u, d, e)
--------------------------------------
--------------------------------------
IV.  The version is committed, and a new record is created for the
object:
serial     nv     pre     dataserial
E          None   D       D
Assertion: If a version is committed, write a new object where nv=None,
pre=most current serial for this oid from version, dataserial = most
current serial for this oid from version
Methods used:
CommitVersion --> GetCurrentOidsInVersion --> InsertCommittedVersion --> ClearVersionStatus --> FinishMethod --> InsertTrans
Things that happened in CommitVersion:
GetCurrentOidsInVersion (src)
InsertCommittedVersion (newserial, src)
ClearVersionStatus (src)
Things tha happened in FinishMethod:
InsertTrans(newserial, u, d, e)
--------------------------------------
--------------------------------------
V.  The object is remodified outside a version:
serial     nv     pre     dataserial
F          None   E       None
Assertion:  if a nonversion object is not new, write it with nv=None,
pre=old current serial outside any version and dataserial=None.
Methods used:
StoreMethod --> GetCurrent --> SetStatus --> InsertObject --> FinishMethod --> InsertTrans
Things that happened in StoreMethod:
GetCurrent (oid) returned E
SetStatus (oid, E's serial, "h")
InsertObject (oid, newserial, E's serial, "c", p, len(p), version, "", "")
Things that happened in FinishMethod:
InsertTrans (serial, u, d, e)
--------------------------------------
--------------------------------------
VI.  Undo rolls back the modification, and a new record is created for
the object:
serial     nv     pre     dataserial
G          None   D       D
Assertion:  Undo creates a new object, sets nv = the previous object's
nv, pre = the previous object's pre (*), and dataserial = the previous
object's dataserial or the previous object's serial if the previous
object's dataserial is None.
(*) if pre is None, make a dummy record with a null pickle and null dataserial.
UndoMethod --> TransactionExists --> TransactionIsNotUndoable --> MakeDummyUndoRecords --> UndoPopulatedRecords -->
UndoUnpopulatedRecords --> ClearUndoStatus
The majority of undo processing happens in the database.  There is 
little Python logic in UndoMethod.
--------------------------------------
--------------------------------------
VII.  The object is modified within a version again (no one expects the
version inquisition!):
serial     nv     pre     dataserial
H          G      None    None
Assertion:  if a version object is new within the version, write it with
nv=nonversion serial, pre=None, dataserial=None.
--------------------------------------
--------------------------------------
VIII.  The object is remodified within a version:
serial     nv     pre    dataserial
I          G      H      None
Assertion:  if a version object is not new within the version, write it
with nv=nonversion serial, pre=old versioned object serial,
dataserial=None.
--------------------------------------
--------------------------------------
VIIII.  The remodification of the object is undone:
serial    nv     pre     dataserial
J         G      None    H
Assertion:  Undo creates a new object, sets nv = the previous object's
nv, pre = the previous object's pre, and dataserial = the previous
object's dataserial or the previous object's serial if the previous
object's dataserial is None.
--------------------------------------
--------------------------------------
X.  The version that contains the object is aborted:
serial     nv     pre     dataserial
K          None   J       D
Assertion: If a version is aborted, write a new object where nv=None,
pre=most current serial for this oid from version, dataserial = either
1) the most current serial for this oid from nonversion index or 2) the
most current serial's dataserial if a dataserial exists in its record.
--------------------------------------
--------------------------------------
XI.  The object is deleted outside any version:
containing object
serial    nv    pre    dataserial
L         None  K      None
Assertion:  if a nonversion object is not new, write it with nv=None,
pre=oldserial and dataserial=None. (same as II) -- this is actually
really just a modification of the containing object that happens to
make it not show up anymore.
--------------------------------------