RepositoryOrganizationNotes

Repository Organization Notes

Principles behind and incidental info about the organization of the repository. This is actually the notes accumulated during design of the organization.

Repository Organization Requirements

• The organization of the repository should be designed to value clarity and ease of collaboration over ease of performing mechanical processes (such as making release distributions). Release-making scripts will be responsible for creating "release bits" that are appropriate, and "release bits" may or may not look like what is in the repository.
• The organization should support our open development process and be compatible with "product" processes. For example, it should be possible for a "shepherd" to manage a particular part of the repository and manage development in the Fishbowl. At the same time, a "product" shepherd must be able to manage a product lifecycle (which includes that part of the repository as a component part) without imposing too much overhead on the component shepherd.

  This means, for example, that the Zope (product) shepherd must be able to tag a specific revision of ZODB (component) for inclusion in a specific version of the product bits. It should be possible to do this (possibly for several products) with a relatively lightweight amount of coordination. It should also be able to happen in a way that doesn't impede the lifecycle of the component (which implies that our branching policy may need to change somewhat to avoid "freezes" outside of the product).

• Some parts of the repository can be used in multiple ways, and have software bits that are unique to a particular usage (like things only useful in a release distribution, but not in a direct-checkout sort of usage). It is up to the shepherd of a repository area to arrange things so that they are useful in the appropriate contexts. For example, the ZEO part of the repository might manage things like start and stop scripts that are only used for a packaged release in a separate subdirectory so that it is not in the way for use as a checkout. The "release builder" for ZEO would be responsible for moving them to whatever place was appropriate in the release bits.
• Software artifacts in the repository that have an identity should manage any documenation related to that identity in that place in the repository. For example, ZODB is a component part of Zope, but it is also commonly used as a component in other systems (and may have its own standalone release cycle). ZODB-specific documentation should be managed in the repository with the software. Scripts, CVS modules or other mechanisms used to make "release bits" should take appropriate action if the documentation needs to be augmented or moved around in the release bits.
• Some community users like to "run from CVS", which has not been a problem until now because the repository (basically) reflected the "release bits", making it possible to "run a checkout". It is a goal to retain this ability, though it will likely be implemented using CVS "modules" that build configurations of repository bits that are shaped like release bits and can be run directly.
• The repository organization should largely reflect the areas of responsibility of various "shepherds", and be as self-contained as possible. Shepherds are expected to know what is going on in their bits of the repository, and have given people write access with the understanding that they are only to make changes to their own bits. Keeping things self-contained will help prevent accidents.
• The repository organization should strongly encourage the use of Python packages. Things that are not currently packages will live in an area with a derogatory name, just to make us dislike putting things there and want to move things out.

Issues

• Should there be a higher-level "technology area" organization, like "templating", "persistence", "relational", etc.? The reality is that (at least in the beginning) shepherds will be scarce - too scarce to have one for each DB adapter for example. One solution would be to have the "templating" shepherd, "persistence" shepherd, etc.

  Resolution: no, we are going to keep the repository as flat as possible, as management process is too fluid to design the repository around it.

• Should "documentation" be a top-level area? This would relate basically to what Amos and Mike are currently doing on SF. Should it continue at SF? We don't have to answer that immediately to achieve the goals of this project.

  Resolution: yes, documentation will be a top-level area. Most likely the current SF documentation repository will move there.

The New Repository Organization

Here is the proposed repository organization.

Packages AccessControl App BTrees DateTime DocumentTemplate HelpSys Interface OFS SearchIndex Shared StructuredText Testing TreeDisplay ZClasses ZLogger ZODB ZPublisher Zope ZServer ZEO webdav

Products ExternalMethod MIMETools MailHost OFSP PythonScripts SiteAccess StandardCacheManagers ZCatalog ZGadflyDA ZSQLMethods ZopeTutorial PageTemplates

Documentation TheZopeBook ZopeDevGuide etc.

Releases Zope CMF etc.

Binaries xxx

JunkPile [known as Cruft in the actual implementation] Zope core stuff not located elsewhere ExtensionClass BTree cPickle? PCGI zlib

jim (May 10, 2001 3:34 pm; Comment #1)

 >     - The organization of the repository should be designed 
 >       to value clarity and ease of collaboration over ease of 
 >       performing mechanical processes (such as making release 
 >       distributions). Release-making scripts will be responsible 
 >       for creating "release bits" that are appropriate, and 
 >       "release bits" may or may not look like what is in the 
 >       repository. 
 

Overall, I agree, however, I do think that there is an important bit of machinery that should be supported. It should be easy to check out releases. That means that organization of a release should be possible via CVS modules.

jim (May 10, 2001 3:40 pm; Comment #2)

 >       This means, for example, that the Zope (product) shepherd must 
 >       be able to tag a specific revision of ZODB (component) for 
 >       inclusion in a specific version of the product bits. It should 
 >       be possible to do this (possibly for several products) with 
 >       a relatively lightweight amount of coordination. It should also 
 >       be able to happen in a way that doesn't impede the lifecycle 
 >       of the component (which implies that our branching policy may
 >       need to change somewhat to avoid "freezes" outside of the product).
 

If a particular configuration can be expressed via a CVS module, then this can be done in a very straightforward fashion.

 >     - Software artifacts in the repository that have an identity 
 >       should manage any documenation related to that identity in that 
 >       place in the repository. For example, ZODB is a component part of 
 >       Zope, but it is also commonly used as a component in other systems
 >       (and may have its own standalone release cycle). ZODB-specific 
 >       documentation should be managed in the repository with the 
 >       software. Scripts, CVS modules or other mechanisms used to make 
 >       "release bits" should take appropriate action if the documentation 
 >       needs to be augmented or moved around in the release bits.
 

I agree, but this may be at odds with plans to manage some API documentation as part of the Zope developer's guide.

 >     - Some community users like to "run from CVS", which has not been 
 >       a problem until now because the repository (basically) reflected 
 >       the "release bits", making it possible to "run a checkout". It is 
 >       a goal to retain this ability, though it will likely be implemented 
 >       using CVS "modules" that build configurations of repository bits 
 >       that are shaped like release bits and can be run directly.
 

Yup (thanks;). Another issue is that if/as we move to the distutils model of installing from a release into a separate location, we may lose the ability to "cvs up" and get changes without a reinstall. The answer, I think, is to make sure that distutils setup scripts copy CVS directories when they are available and that releases be set up to make this work correctly.

jim (May 10, 2001 4:38 pm; Comment #3)

 >   Issues
 > 
 >     - Should there be a higher-level "technology area" organization,
 >       like "templating", "persistence", "relational", etc.? The 
 >       reality is that (at least in the beginning) shepherds will 
 >       be scarce -
 

Plus, I'm not sure that all template lanuages would have the same shepard.

 >       too scarce to have one for each DB adapter for
 >       example. One solution would be to have the "templating" shepherd,
 >       "persistence" shepherd, etc.
 

Ditto. Hm, This makes me wonder whether responsibility (shepard) is any better an organizing principle than release. Responsibility may be more fluid that release.

Maybe a much simpler organization is more appropriate. Maybe the CVS should simply be organized by package.

 >     - Should "documentation" be a top-level area? This would relate 
 >       basically to what Amos and Mike are currently doing on SF. 
 >       Should it continue at SF?
 

Absolutely not, IMO. It should be possible to checkout docs and SW from one repository. But this is a separate question from whether the software and docs should have separate trees. I've already argued that releases and SW should have separate trees, so I guess that the same thing would aply to docs. My one request is that the organization within docs should mirror the software documentation.

Put another way, I want to be able to create a module, say for DocumentTemplate? that stitches together software, release material, and documentation, so that is all available in one checkout.

jeremy (May 29, 2001 6:59 pm; Comment #4) Editor Remark Requested - done

What's the difference between "Package" and "Product"? It's not clear how I would decide where to look for an existing software artifact or where to create a new one?

klm, July 19, 2001

"Product" refers to instances of a specific, python-based Zope extension mechanism, typically used to define new entities the can be interactively added to Zope folders. By "Package", we mean python packages (having an __init__.py) that are part of an application, but not unitary Products in their own right.

The Zope codebase includes a number of non-Product packages, and also a few Products. These now are decomposed in the respective Package and Product repository sections, and knitted together in the modules for various standalone applications (whose superstructures are located in the Releases repository section.