README

Introduction

The Ultraseek Data Adapter (Ultraseek DA) provides an interface to an external Ultraseek Server search engine. It allows a Zope application to provide the user interface for a search service and retrieve search results as record-oriented data rather than formatted HTML from a remote Ultraseek Server install. The application can leverage the Zope application server framework, using DTML and other Zope features to format and perform other operations on the returned search result data. Data is transferred between Zope and Ultraseek Server using HTTP in an encrypted format for security.

Using Ultraseek DA, one can provide multiple customized and branded search interfaces to the same Ultraseek Server. Zope allows each site to be managed through the web by (possibly different) content managers. The Ultraseek DA architecture allows the existing back-end Ultraseek Server to operate as a data source without affecting its current functionality. The back-end Ultraseek Server can continue to operate as usual while also supporting an arbitrary number of Ultraseek DAs as a search data provider.

The Ultraseek DA product does not provide any functionality for management of Ultraseek Server, it simply provides a means of proxying search requests to the remote server and receiving the results of the search request as record-oriented data. It also provides a set of default Zope Documents which use the DA to mimic the HTML interface of a basic Ultraseek server. This document assumes that the reader has a working knowledge of the Zope web publishing environment as well as experience using and managing an Ultraseek Server installation.

An example usage of Ultraseek DA would be to implement a search service, targeted and branded for several specific audiences, based on a single Ultraseek Server installation. Consider the following case: Company XYZ wants to use a single Ultraseek Server to index resources pertaining to movies, television and books. Based on this corpus, XYZ wants to implement three internet sites: MovieSearch.com, TVSearch. com and BookSearch.com. These sites will be highly branded as separate entities and managed by different people, but will use the same search corpus for ease of maintenance. XYZ could use a single Zope installation to implement all three services, or it could use separate Zope installations (possibly on three different machines) to implement the services. In either case, an Ultraseek DA can be used in each service to provide connectivity to the single Ultraseek Server installation. Each service can provide completely different user interfaces and results formatting. Each could even be integrated with other Zope functionality to provide membership-based services, integration with relational data, stored queries and other advanced services.

Installation

Ultraseek Data Adapter for Zope is an add-on product for the Zope web publishing environment. To use Ultraseek DA, you must have a working Zope installation and full access to a working Ultraseek Server version 1.1.0 or greater.

Installation of Ultraseek DA consists of two steps. First you must install the Ultraseek DA product into your existing Zope installation, then you must install data server support on your Ultraseek Server. Data server support consists of only two HTML files, and will not interfere with the current operation of your Ultraseek Server.

The Ultraseek DA product should be installed into your Zope installation the same way you would install any other Zope add-on product. Simply extract the Ultraseek DA distribution file into the top-level directory of your Zope installation. This will create a new directory "lib/python/Products/UltraseekDA/" relative to the root of your Zope installation. You will need to restart your Zope installation for Zope to recognize the newly installed product.

Server-side support for Ultraseek DA must be installed into your existing Ultraseek Server installation. Server-side support for Ultraseek DA consists of two HTML files, and does not modify your existing Ultraseek Server software in any way. Addition of these two HTML files to an accessible area of your Ultraseek Server documents directory is all that is required to provide Ultraseek DA connectivity to your server. The two HTML files are:

config. html - This file relays information pertaining to the configuration of your Ultraseek Server to an Ultraseek DA. An Ultraseek DA requests this file from the Ultraseek Server and caches the configuration information. This allows you to test the values of most Ultraseek Server configuration variables from within Zope using DTML.
query. html - This file acts identically to the top-level scripted search page provided by Ultraseek Server, except that instead of rendering and returning HTML output it marshals the search results as record-oriented data and returns them in an encoded format. When the query() method of an Ultraseek DA object is called from within Zope, the DA object automatically calls the URL for this query.html page on the Ultraseek Server, passing in search query parameters and decoding the result into record-oriented data.

The two HTML files which must be installed on your Ultraseek Server can be found in the directory "lib/python/Products/UltraseekDA/server_docs/" relative to your Zope installation root directory.

It is recommended that you create a new subdirectory in your docs/ directory on your Ultraseek Server to hold these two DA interface documents. This will prevent them from getting mixed in with your existing Ultraseek Server HTML pages and make it easier to remove the DA interface at a later time. After copying config.html and query.html to a directory on your Ultraseek Server, you will need to use the Ultraseek Server's management interface to "reload" your server-parsed HTML documents.

Note that you only need to install server-side support once on your Ultraseek Server. Multiple Ultraseek DAs in multiple Zope installations can then access that Ultraseek Server if they share the same encryption key.

Using Ultraseek DA

Now that you have installed the Ultraseek DA product into your Zope installation and installed the Ultraseek DA server-side support files on your Ultraseek Server, you are ready to begin using the Ultraseek DA product.

Creating an Ultraseek DA

The Ultraseek DA product adds a new object type to the Zope "Add" list. To add an Ultraseek DA, navigate to the Folder in which you wish to place the new DA, select "Ultraseek DA" from the Add list and click the "Add" button. You will be presented with the Add Ultraseek DA Form.

The Add Ultraseek DA Form provides four fields:

Id - The id for the new Ultraseek DA object (required).
Title - A title for the new Ultraseek DA object (optional)
Ultraseek URL - The URL to the directory on your Ultraseek Server where you installed the Ultraseek DA interface HTML files (required). For instance, if during installation you created the directory "da" in the main documents directory of your Ultraseek Server to hold the DA interface HTML files, the URL would be: http://search.mydomain.com:8765/da. Do not add a trailing "/" to the Ultraseek URL field value.
Encrypt Key - A string that will be used as a seed for encryption of data transferred between the Zope Ultraseek DA and your Ultraseek Server (required). IMPORTANT: There is also an encrypt key variable defined at the top of each of the Ultraseek DA interface HTML files that you installed on your Ultraseek Server. The encrypt key defined in these HTML files MUST match the encrypt key entered on the Add Ultraseek DA form for data transfer to succeed! The default encrypt key value on the Add Ultraseek DA form matches the default value used in the interface HTML files distributed with the product. If you change the value from the default on the Add Ultraseek DA form, you must change it in the interface HTML files on your Ultraseek Server as well.

Once you have submitted the Add Ultraseek DA form, a new DA object will be created. At the time it is created, the new DA object will contact the Ultraseek Server using the URL provided in the Add form to retrieve configuration information. If the Ultraseek Server cannot be contacted, or if the client and server-side encrypt keys do not match, the new DA will not be created successfully.

Changing Ultraseek DA properties

After you have created an Ultraseek DA, you can change its properties by clicking on the Properties tab of the DA object in the Zope management interface. This will present an edit form which allows you to modify the Title, Ultraseek URL and Encrypt Key for the DA object. Upon saving changes to DA properties, the DA object will contact the Ultraseek Server using the new URL and encrypt key to retrieve configuration information. If the Ultraseek Server cannot be contacted, or if the client and server-side encrypt keys do not match, an error will be raised and the property changes will be discarded.

Generating default search interface documents

The Ultraseek DA object itself provides simple connectivity to a back-end Ultraseek Server. To use this connectivity, an HTML user interface must be built which collects a user's search criteria, uses DTML statements to pass the user request to the DA, and formats the search result data. Because this can be quite a chore, Ultraseek DA objects can automatically generate a set of Zope Documents which mimic the default user interface of an Ultraseek Server site. A developer can then use these default Documents as a starting point for changing the look and feel of the search service.

To have an Ultraseek DA object generate default Documents into the Folder containing the DA, navigate to the Ultraseek DA object in the Zope management interface and click Generate default documents. The default Documents generated are fairly straightforward translations to DTML of the various scripted HTML pages bundled with your Ultraseek Server. Note that default Document generation will fail if a Document with the same name as a generated Document already exists in the Folder containing the Ultraseek DA object.

Default values for the various two and three-letter variables used by Ultraseek Server are initialized in the default Document initvars_html. These values are used in the same way as they are in Ultraseek Server's scripted HTML pages, with the exception that some of the variables are typed in the Zope environment for greater error checking. A Python dictionary named config is inserted into the REQUEST namespace by initvars_html which contains cached configuration information about the remote Ultraseek Server. This config dictionary contains most variables that would be found by referencing the config module in Ultraseek Server scripted HTML code.

Retrieving search result data

An Ultraseek DA application uses user interface Documents to collect query information, then uses DTML to call the DA object, passing the HTTP request object, to obtain search results. The Ultraseek DA uses the passed-in REQUEST object to construct an appropriate request for the remote Ultraseek Server. Any form variables in the REQUEST object (such as qt, nh, etc.) are passed on to the remote Ultraseek Server. On the remote server, the DA interface performs the search using variables found in the HTTP request, and returns an encoded sequence of 0 or more result records to the Ultraseek DA object.

The Ultraseek DA then decodes the server response and converts the result back into a sequence of 0 or more result records which are returned to the calling DTML code. The following example demonstrates the process of obtaining search results from DTML. This snippet passes the REQUEST to the Ultraseek DA object named da, and stores the results in the REQUEST namespace as titles for later use in a DTML in tag:

      <dtml-call "REQUEST.set('titles', da(REQUEST=REQUEST))">

Each record in the result set returned by the call to the Ultraseek DA has the following schema:

      score, title, summary, url, publisher, iterms, size, modtime,
      idxtime, flags, nlinks, docdbs, docids, dterms, quality, hits,
      ndocs, terms, href

The following example demonstrates a simple result listing in DTML, noting the score, hyperlinked title and summary of each search result:

      <dl> 
      <dtml-in titles> 
      <dt> Score: <dtml-var score>% 
      <a href="<dtml-var href>"><dtml-var title></a> 
      <dd><dtml-var summary> 
      </dtml-in> 
      </dl>

The following describes the fields of each record in a result set:

score - A percentage-based relevance score for the result record.
title - The title of the result document.
summary - Summary text for the result record.
url - The url of the source document.
publisher - See Ultraseek Server documentation.
iterms - See Ultraseek Server documentation.
size - The size of the source document.
modtime - The modification time of the source document.
idxtime - The time the source document was indexed.
flags - Ultraseek flags returned from the search.
nlinks - See Ultraseek Server documentation.
docdbs - The names of the collections in which the result was found.
docids - Internal Ultraseek Server document identifier.
dterms - Search terms which matched for the current result record.
quality - See Ultraseek Server documentation.
hits - The number of matching result records found.
ndocs - See Ultraseek Server documentation.
terms - Query terms used in the search.
seekhref - A query string used internally to build search links.

The default Documents generated by an Ultraseek DA closely mimic the default Ultraseek Server search interface. These Documents provide a great starting point for customization, and are themselves the best examples for working with query terms and result data. If you make mistakes while making changes to the default Documents, you can always use the Zope Undo facility to revert your changes, or simply delete the generated Documents and generate a fresh set.

Advanced Topics

Ultraseek DA objects give developers the ability to create powerful search services backed by a popular search engine while taking full advantage of the powerful Zope environment. Because Ultraseek DA objects are fully integrated with the Zope environment, you can take advantage of Zope objects such as UserFolders to provide access-controlled search services. Simply drop a UserFolder into the Folder containing an Ultraseek DA and set permissions such that your users have roles which give them View and Use Ultraseek DAs permission. Since you can create multiple Ultraseek DAs that all point to the same back-end Ultraseek Server, you can easily set up multiple views of your Ultraseek data with separate branding and access control for different audiences.

More ambitious applications could use Z SQL Methods in conjunction with a supported relational database to store frequently used user queries, or to join search result data with data from the relational database based on search result content. While the details of such advanced applications are beyond the scope of this document, such services are fairly straightforward to implement for those with a good knowledge of both the Zope environment and Ultraseek Server software. Detailed documentation for the Zope environment is available from the Zope web site. Ultraseek Server documentation can be found on the Ultraseek Software web site.

Questions and Answers

Q: Is it safe to use Ultraseek DA with my Ultraseek Server?

A: Yes. The scripted HTML file support installed on your Ultraseek Server performs the same exact search function as the scripted HTML files included with Ultraseek. The only difference is that the Ultraseek DA support files render the search result data into a marshalled data format rather than HTML.

Q: If someone gets my encryption key, can they break into my Ultraseek Server?

A: No. The encryption key has nothing to do with Ultraseek Server. It is simply a "shared secret" between the Ultraseek DA object and the Ultraseek DA support files on the server that makes it more difficult for a third party to make any use of returned search result data should it somehow be intercepted in transit over the network.

Q: Can this be used to manage my Ultraseek Server?

A: No. Ultraseek DA is intended solely to allow a remote Ultraseek Server to be treated as a data-producing facility from within the Zope environment. Ultraseek DA does not, and is not intended to provide any functionality for managing your remote Ultraseek Server.

Q: Will this work between different kinds of machines?

A: Yes. The Zope environment runs on many platforms and machine architectures. An Ultraseek DA installed on any supported Zope platform will work with a remote Ultraseek Server on any platform that Ultraseek Server supports.

Q: What are the performance implications?

A: Ultraseek DA uses the standard HTTP protocol to communicate with an Ultraseek Server. This is true even if the Ultraseek Server is installed on the same machine as the Ultraseek DA. Because of this, network response time can affect the speed at which an Ultraseek DA can retrieve results from its backing server. In situations where the Ultraseek DA and Ultraseek Server reside on the same network, latency should be very minimal. In general, the latency experienced by an Ultraseek DA when retrieving results from a remote server is the same as the latency you would experience accessing the remote Ultraseek Server yourself using a Web browser -the same network latency issues apply.