################################################################

$RCSfile: README.txt,v $

Author: Craeg Strong

$Date: 2003/04/21 01:52:57 $

################################################################

Contents

1. Description
2. Default Behavior
3. Extensibility
4. Creating Instances Programmatically
5. Known Limitations
6. Contributions
7. Schema Migration
8. Notes
9. Unit Testing on Win32

Description

This is the ExternalFile Zope Product. It is similar to ExtFile (see ExtFile ), except that it references the file in situ, rather than copying it to a file system repository within the context of the Zope installation. It also supports in-browser editing for ASCII files.

An ExternalFile is a reference to a file in the filesystem, like a symbolic link in a UNIX file system (or windows shortcut). You can create multiple instances of ExternalFile pointing to the same physical file.

The file referred to by an instance of ExternalFile must be accessible to the Zope server. This usually means that it must reside in a local filesystem, or possibly one that is mounted on the Zope server machine through NFS, Samba, Novell, or some such protocol. It also means that Zope must have permission to access it. Since Zope is usually running as an administrative user or some special user, this can be a bit tricky. For example, on UNIX systems Zope is usually running as "nobody." One possible solution might be to place the Zope user in the same group as the developers. This way they can view and edit their own files online as long as they make them group writable. Of course, if you start up your own private Zope server on your local system, you won't encounter any of these issues.

Why refer to the files in place? There are many possible reasons, one of which is to support the usage of a sandbox-oriented version control system. By creating instances of ExternalFile pointing to the content files in the developer sandbox, one can continue to use the version control system in a very natural way.

There are two ways to create ExternalFiles from the Zope Management Interface (ZMI): one at a time or in a batch. To use the batch option, simply select "External File Batch" from the standard Zope Add menu. It provides a way to automatically add an instance of ExternalFile for every file within a directory. Because ExternalFile instances are merely links or aliases, they can be deleted without affecting the underlying file. Therefore, an easy way to create ExternalFile instances is to create them a batch at a time and then delete the unwanted instances (for example, those pointing to junk files, "tilde files", etc.).

ExternalFile is intended to be subclassed. For example, CVSFile is a Zope Product that inherits from ExternalFile and adds support for working with external files within a CVS sandbox.

ExternalFile can behave like standard Zope Products. The current list is:

• DTMLDocument
• DTMLMethod
• Image
• File
• PageTemplate

When an ExternalFile is set to behave_like a DTMLDocument, for example, it should be able to be used in place of a DTMLDocument. The same goes for the others in the list above. If you find that an ExternalFile with a certain behave_like setting does not work the same way as the "real" Zope object, please report this as a bug. A small testcase demonstrating the problem would be greatly appreciated. The one known exception is that ExternalFile does not currently support webDAV locking.

ExternalFile now requires Page Templates. For Zope releases 2.5 and above, Page Templates are available by default. For earlier Zope releases they must be added by separately installing the ZopePageTemplates package.

ExternalFile has been tested in a production website setting, with four active content authors, 150 pages of content, and 100 hits per day. It may therefore be considered as being stable and production ready.

ExternalFile encapsulates a file in the filesystem for use as a Zope object. The filesystem must be locally accessible to the Zope server. The metadata associated with the ExternalFile is stored in the Zope object database, but the content itself is not. When the content of an ExternalFile is accessed by Zope, it is automatically retrieved from the filesystem. An ExternalFile always points to a file that exists locally on the server. That file may be:

• already existing on the server,
• created by uploading a local file to the server and specifying a target filepath,
• created by uploading a local file to the server and overwriting a file already existing on the server, or
• created on the server, with initially empty content.

Default Behavior

The behavior of an ExternalFile can be changed at any time, simply by modifying the behave_like property in the Properties tab in the Zope Management Interface. However, when an ExternalFile is first created, the default setting for behave_like is chosen according to the mimetype of the target file. For a precise description, see "Display Types", but the table below provides a rough idea:

    Content-Type    Default 'behave_like'

    ============    =====================

    HTML,DTML       DTMLDocument

    JPG,GIF,BMP     Image

    Other           File

Extensibility

When a Zope server that has the ExternalFile product installed is first started, it loads in two ASCII files, located in the etc subdirectory: [ZOPE]/lib/python/Products/ExternalFile/etc/mime.types and [ZOPE]/lib/python/Products/ExternalFile/etc/display.types. Feel free to customize these files as you see fit for your installation.

CAVEAT: Understand, however, that removing* entries from one of the files might produce unexpected results, for example, if you had an existing instance of ExternalFile using a mimetype listed in but then removed from mime.types. Don't do that.*

Mime Types

Each instance of ExternalFile can be assigned a mimetype from a list in the Properties tab of the Zope Management Interface.

This list is created by adding the types listed in mime.types to those available by default in Python. To view the current list of Python-default mimetypes, look here N.B. That is not actually the list, but should be roughly close. I have not been able to find the list online. It is available on page 268 of the Python Essential Reference, Second Edition.

The format of the mime.types file is the same as that used by the Apache web server (de facto standard). Each line consists of a mimetype and zero, one, or more file extensions. The file extensions are used to guess mimetypes from files. For example, if you add a new Zope object called foo.jpg, the extension jpg will be looked up in the map to retrieve the image/jpeg mimetype. Here is an example entry::

text/css css # Cascading stylesheets

Instances of ExternalFile (and therefore its subclasses, such as CVSFile) store their mimetype persistently as a property. The mimetype may be changed through the Properties tab in the ZMI.

Display Types

DisplayType is an invented concept that is used solely to decide how to edit and view ExternalFiles in the ZMI. There are three flavors of DisplayType:

       ascii   -> Editable Though The Web, viewable via HTML readonly text area

       image   -> Not editable TTW, viewable via HTML image tag

       binary  -> Not editable TTW, viewable via HTML hyperlink

Generally, the DisplayType is derived by parsing the mimetype into type/subtype. That is:

      image/jpeg       -> image

      text/plain       -> ascii

      (anything else)  -> binary

In certain circumstances, the algorithm does not yield correct results. For example, application/xml should be ascii rather than binary. display.types is used for those circumstances i.e. when the algorithm would yield incorrect results. You may add to it or modify as needed.

For example, you may find that your browser cannot display certain unusual image types correctly in <img> tags. This could be corrected by an entry such as:

      image/x-cmu-raster        binary

Instances of ExternalFile do not maintain their displaytype persistently, rather they query ProductProperties at run-time for their displaytype based on their mimetype. That means that changing the entries in display.types will change the behavior of all instances of ExternalFile (and those of its subclasses, such as CVSFile).

Creating Instances Programmatically

Normally instances of ExternalFile are created by hand in the Zope Management Interface. There are a number of screens (a mini-workflow) that are traversed in order to ensure that the instance is created properly, to deal with all of the things that can go wrong (creating new files, overwriting existing files, permission issues, etc.) and to confirm exactly what was done.

However, there are times when you don't want that. You simply want to run a Python script and create an instance of ExternalFile. Do not despair! This is easy to do. Check out the extras subdirectory. You will find the source code for an example: uploadCreate. It is commented and should be easy to adapt to your uses. Basically it calls the manage_addExternalFile() method in ExternalFile.py directly, bypassing the GUI creation workflow code (which has all been moved into a separate module: CreationDialog.py).

Known Limitations

ExternalFiles do not support webDAV the way real DTMLDocuments do. However, they do support most other things such as through the web (TTW) editing, file upload, and FTP. Frankly, we are not planning to fix this shortcoming until the whole thing is revisited in the context of Zope3. In fact, rumor has it that the Zope3 object database will feature a file system synchronization technology that may even obviate ExternalFile!

ExternalFiles do not handle security exactly the same way DTMLDocuments do. This means that there are certain common-usage scenarios where ExternalFiles can NOT be used in place of DTMLDocuments. We have not been able to nail down a simple reproducible test case for this as yet. If you have one, please email it to us so we can fix this annoying bug.

Contributions

We hope others find this code useful. If you have extended or improved this product, please feel free to submit your changes to us.

If there is enough interest, we would certainly consider setting up an open-source project on SourceForge.

Schema Migration

From time to time, you may find yourself with a new version of this product, either because we have released a "new improved" version or from some changes you may have made on your own. How do you deal with all of the existing instances in your ZODB that were created with the old definition? Here is our preferred technique:

1. Find the repair() method (we usually put it at the bottom of the respective python source file)
2. Change the repair() method so that it updates the object from the old version to the new version
3. Create a python script in the root folder that recursively calls repair() on each of the object with a given metatype
4. Execute the python script (the easy way is via the "test" tab)

Here is our python script. We call it "repairAll" and put it in the ZODB root folder:

    objects = context.ZopeFind(context, obj_metatypes=[metaType], search_sub=1)
    map(lambda x: x[1].repair(), objects)
    return map(lambda x: x[0], objects)

In general, the repair() method is coded to convert an instance from the previous version to the current version. To convert from an earlier version (that is, to skip a version) you may have to code it yourself. It is actually quite easy to do, but don't hesitate to email if you need pointers or advice.

Notes

This document is written in structured text. For a quickie intro to structured text, look here.

Unit Testing on Win32

If you don't plan to run the automated unit test suite, this section is irrelevant.

For some reason, the automated unit tests don't run properly for me on win32 using the FAT file system under Zopes earlier than 2.5.1. I found the following workaround. In the file name:

    [ZOPE]\lib\python\Testing\custom_zodb.py

Where ZOPE stands for the directory in which you installed Zope, Change line number seven:

    Storage = DemoStorage(base=FileStorage(dfi,read_only=1), quota=(1<<20))

to instead read:

    Storage = DemoStorage(base=FileStorage(dfi, read_only=0), quota=(1<<20))

My guess is that FAT does not support the read_only attribute as required.