Log in
Forgot your password?
TeamZope
Team Development With Zope
This document is a modest proposal on team development, with a focus on source management. It is tongue in cheek, if you are easily offended please close your eyes while reading. if it makes you feel better please preface every sentence with IMO.
Road Map
- a look at ideal development practices
- an examination of zope, and its problems in implementing ideal practices
- a look at existing zope tools, that try to offer handle the problem.
- a look at zope tools and development practices that help the problem.
- a modest proposal to solve the problem.
Terminology
- blob - an opaque binary large object
- instance space - objects within the zodb.
Valhalla
Standard to Ideal Development practices for most web-app production systems, are to use a multi stage development policy. comprised of
- Development sandboxes (ideally) one per developer, where new code is developed. working code is saved in a source control management system.
- Staging/Testing servers, check points from the source control management system used for developer and customer testing.
- Production, stable check points/branches from the scm that are deployed for production use.
While actual details of multi-stage development tend to vary a bit, the key to multi-stage development is a source control management (SCM) system, that allows the diffing, merging, and branching/tagging of various states of the application code. An excellent introduction to SCM for web development with cvs, can be found here
These types of setups are common, best practices in software development.
Incidentally, enlightened organizations generally run unit and functional tests in a cron facility against the staging server.
Additonal best practices which I won't be covering in this analysis include using a bug/feature tracking, unit testing, and functional testing.
Purgatory
Team Development with Zope can be problematic. Get Over it.
The problem is rooted in the fact that Zope blurs the lines between content and logic. This manifests itself in two distinct issues, application code that is created and executed from the database (ZODB) alongside stored content, and configuration that is stored in instance space. These are common problems from both from a practical and ideological point of view.
Sirens's Call
Probably the worst offendor in blurring that distinction is paradoxically one of Zope's most alluring assets, the Zope Management Interface.
After you spend a couple of hours configuring objects via, the zmi, take a few minutes and ask yourself what you've done from a SCM perspective. You've turned your work into binary blobs and timestamps.
Thus begins the road to madness.
From a team development perspective this translates into code that cannot be used with an SCM, and thus places the burden of development coordination and code management on the developer without providing adequate tools to do so. Its a slippery slope that begins with developers calling out to each other in a room, about what their changing, or via email or chat.
It can be useful for prototyping to develop within the ZMI and then use tools like FSDump to convert to a useful filesystem representation when its ready and to check that in into a SCM, using Directory Views (covered latter) to bring the code back into Zope. But be wary of configuration that is done via the ZMI.
Code and object configuration stored in the zodb, is fragile for auditing and discovery. A single lost misconfiguration, can hose a site, without the developers maintaining explicit knowledge of the changes.
My advice, resist the siren's call, and stay away from the ZMI for any real development.
Here after zmi development will be known in this paper as blob creation, and the management of instance space code within Zope as blob management.
- Levels
An upward ascent through analysis of existing Zope tools, that try to combat team madness. these techniques represent the current state of the art of blob management in Zope.
i'd like to note that as blob management techinques none of these mechanisms offers the features of an SCM system like cvs, subversion, perforce, etc (diffs, merges, branching, etc.), additionally most of these techiques, with the sole exception of zsyncer are for single Zope deployment, ie one instance of Zope. In and of themselves, I regard these two flaws as fatals in consideration of these techniques for team development, but I'd still like to go through them for edification purposes.
I'd like to note that these tools are well written and useful for certain tasks. But in the context of team development, I'd like to take this opportunity to poke a few holes.
Versions
Theoretically Versions are long running ZODB transactions, they allow you to change a group of objects within that version, while providing read only copies of objects within the version, to requests which are not particpating in the version.
Originally these were much touted as a mechanism to do concurrent development on a site. But Versions are EVIL. They are a database level feature masquerading as an application tool.
The biggest problem with versions, aside from the complete lack of SCM, is that any object modified in the course of a version, is locked by that version, and cannot be modifed outside of it. This is hugely problematic for development on the same instance of Zope, if for example you want to test some new code that creates an object which is cataloged, you end up locking the whole catalog, meaning new content created outside of the version won't be cataloged, and will likely throw an error. OUCH!, and 'nough said.
ZSyncer
Being a tool most efficent and useful for transfer of binary blobs and timestamps via xml-rpc between Zope instances. Its the only tool that does remote object transfer between Zope instances.
Zsyncer uses a simple algorithm for migrating changes between servers, it basically looks at the timestamp on the blob its transfering. The problem with this aside from the fact that it doesn't even come close to dealing with merging or such esoterica as blob confict resolution, is that the timestamps are faulty.
The timestamps on these blobs are derived from the database. The problem is that these timestamps are database specific, and thus do not reflect timestamps on the objects but instead on the times the objects were last changed within the database, which in the case of a transported object is the time of transport.
This makes it problematic to impossible to effectively use zsyncer for a team development tool.
This isn't to say that zsyncer isn't useful for one way transfers of blobs for read only copies. 'nough said.
Zope Version Control (and kissing cousins)
ZVC refines blob management to a new art. It provides for blob versioning, in an api akin to cvs. It defines methods for checking in and out blob versions, label of versions, with rudimentary support for branches. Combined with the CMFStaging tools it is the best existing solution for blob management within Zope.
while I find that ZVC to be inadequate for code versioning for the aforementioned reasons, it is an excellent tool, for object content versioning although it could use a little polishing.
ZVC is available from Zope.org's cvs
ZCVS
Stores blobs in cvs. As such, it allows for tagging and branching of multiple objects, but even though its using an underlying SCM, it cannot derive much benefit from it, in the sense of code annotation or diffing. ZCVS and ZVC are remarkably similiar in functionality, Both ZCVS and ZVC allow for reverting to older versions of code objects, although ZVC allows for extra functionality by allowing for write locks against objects which are not in a checked out state, from the most current branch.
Functionally, the biggest difference between them is that ZVC allows versioning of containers indepedent of subobjects, while ZCVS, because it uses the export/import mechanism, versions subobjects with containers. Both are useful under different use cases.
That said, some zope development teams do use ZCVS successfully, although imo they are taking gestalt snapshots of objects rather than managing code, but in the abscense of readily accepted alternatives it easy to understand why they have embraced ZCVS as a minimal sanitity retention tool.
Stemming the Tide
The following are products and techinques currently in common use, which can help in managing code and stopping the tides of insanity, mostly by allowing file system based development or object representation and thus integration with a common SCM and the associated benefits.
Python Products
Python Products allow you to take full power of Zope and python in constructing webapplications. More relevantly, they allow for traditional file system based development with source maintained by an SCM.
Directory Views
Come bundled with the CMF. They allow for common content types to be developed on the file system (ZSQL Methods, ZPT, DTML, etc.) (Note they are easy to separate out of the cmf, an example being this )
CVSFile/ExtFile
A suite of relatively new products by Ariel Partner.
The real problem with these tools is that they can't create a site for you. They are tools that help separate code from database content, and restore the code to the filesystem such that they can be managed by an SCM, but as such they don't help in configuring the instance space, and thus can't help in deploying directly from cvs.
take for example a cmf based site, a lot of configuration and wiring must be done on the tools in instance space for things to work properly, such as registering content types, creating workflows, adding actions, arranging skin paths.
Still I think there is a solution.
A Modest Proposal
Throw away your zodb. Take your database and throw it out the window. Do it on a regular basis. its very liberating.
Because today, here and now, their is no scm for Zope. Its time to tackle the problem from a traditional approach, because if you can't deploy your site from your scm, you're just asking for trouble.
The solution is simple, create something on the filesystem that will initialize your site in deployable form from an empty zodb. Check that tool into your SCM, and deploy to various stages from cvs. Development checkpoints should be checked into your SCM, and the zodb should be thrown away regularly to avoid possiblity of configuration or development not reflected in the SCM.
This is only a partial solution, that helps solve instance space configuration. It does not deal with bringing in production data into development environments, or versioning content. Still for me it has a been a huge step forward in the right direction.
I've opensourced the second-generation of a tool I use to do this, here it provides for reuse, depedency tracking, logging, and configuration specified in text files. With it, I'm able to deploy sites that were developed by a team straight from cvs, without hassles.
Event Horizon
In the future some of the discrepancy between database and code may be addressed to the benefit of both in the form of persistent modules. Code that would be written on the filesystem, executed from the database (and thus transparently replicated in zeo setup). These concepts are being actively explored by PythonLabs (a division of Zope Corporation, that houses the brain trust that does most python core development).
Another interesting tool in development, ZFS, attempts to provide transparent marshalling of objects to file system editable forms.
On a smaller and more focused scope, I've heard that Tres Seaver and Adrian Hungate are working on a CMFInstaller api, that will help with the instance registration required for new CMF Product installation, thus hopefully lessening the amount of explicit configuration that must be done.
The real gem, that would enable more functionality out of the blob management tools is an ObjectDiff Tool.
But my advice and preference for the here and now, when it comes to code management, is throwaway your zodb on a regular basis. Free yourself and your team from madness.
Author's Note. No chickens were harmed in the writing of this paper.
Credits
- kapil thangavelu Author
- Steve Alexander - Reviewer
Resources
Bug/Feature Tracking
Introduction to CVS for Web Development
Source Control Management
Zope Tools
History
- 1 First Public Release, Added Some notes regarding ArielPartner's toolset. September 2002
- 0 Initial Version, August 2002
License
Copyright Kapil Thangavelu
This document is released under the FSF's Free Documentation License. For the complete text of this license please look here