AdvancedQuery

Introduction

AdvancedQuery is a Zope product aimed to overcome several limitations of ZCatalog's native search function.

Like ZCatalog's search, it supports elementary index searches. While ZCatalog can combine such elementary searches only by "and", AdvancedQuery allows them to be combined arbitrary with & (and), | (or) and ~ (not).

While ZCatalog supports an efficient sorting via an index on one level, AdvancedQuery supports sorting via any level of (field) indexes. Moreover, it sorts the result incrementally -- only as far as you access your result. This can drastically speed up the time required for sorting. It uses Python's generators for this (and thus requires Python 2.2 or better).

Query Objects

Queries are specified by (full blown) Python objects. They are constructed in the following way:

Expressionprinted asMeaning
Eq(index, value) index = value the documents indexed by index under value
Le(index, value) index <= value the documents indexed by index under a value less or equal value
Ge(index, value) index >= value the documents indexed by index under a value greater or equal value
MatchGlob(index, pattern) index =~ pattern the documents indexed by index under a value matching the glob pattern. A glob pattern can contain wildcards * (matches any sequence of characters) and ? (matches any single character).
This query type is only supported by most string or unicode based ManagableIndexes (exception: RangeIndex). Many TextIndexes support glob matching via the Eq query.
MatchRegexp(index, regexp) index =~~ regexp the documents indexed by index under a value matching the regular expression regexp. See the re module documentation in the Python Library Reference, for a description of regular expressions.
This query type is only supported by most string or unicode based ManagableIndexes (exception: RangeIndex).
Between(index, low, high) low <= index <= high the documents indexed by index under a value between low and high
In(index, sequence) index in sequence the documents indexed by index under a value in sequence
Generic(index, value) index ~~ value this query type is used to pass any search expression to index as understood by it. Such search expressions usually take the form of a dictionary with query as the most essential key. Generic is necessary to use the full power of specialized indexes, such as the level argument for PathIndex searches.
Indexed(index) Indexed(index) the documents that are indexed by index. This does not work for all index types.
~ query ~ query Not: the documents that do not satisfy query
query1 & query2 (query1 & query2) And: the documents satisfying both query1 and query2
And(*queries) (query1 & ... & queryn) And: the documents satisfying all queries; if queries is empty, any document satisfies this And query
query1 | query2 (query1 | query2) Or: the documents satisfying either query1 or query2 (or both)
Or(*queries) (query1 | ... | queryn) Or: the documents satisfying (at least) one of queries; if queries is empty, no document satisfies this Or query

And and Or queries are so called CompositeQuerys. They possess a method addSubquery(query) to add an additional subquery.

The constructors are imported from Products.AdvancedQuery.

AdvancedQuery uses so called Monkey Patching to give ZCatalog the new method makeAdvancedQuery(catalogSearchSpec). A catalogSearchSpec is a search specification as described in the Zope Book for ZCatalog searches (essentially a dictionary mapping index names to search specifications). makeAdvancedQuery returns the equivalent AdvancedQuery search object.

Query evaluation

AdvancedQuery uses so called Monkey Patching to give ZCatalog and (if available) the CMF CatalogTool the new method evalAdvancedQuery(query, sortSpec=()). evalAdvancedQuery evaluates query and then sorts the document result set according to sortSpec.

The CatalogTool's evalAdvancedQuery uses a new ValidityRange index (if present) in preference to the effective and expires indexes to restrict searches to valid objects. ValidityRange is expected to be a ManagableIndex RangeIndex. Searches via such a range index are considerably more efficient than those via the individual indexes.

Sorting

AdvancedQuery supports incremental multi-level lexicographic sorting via field index like indexes. If an index used for sorting is not field index like (i.e. does not index an object under at most one value), you may get funny (and partly non determistic) results.

Sorting is specified by a sequence of sort specifications, each for a single level. Such a specification is either an index name or a pair index name and direction. A direction is 'asc' (ascending) or 'desc' (descending); if the direction is not specified, 'asc' is assumed.

When the result contains documents not indexed by a sorting index, such documents are delivered after indexed documents. This happens always, independant of search direction.

Examples

from Products.AdvancedQuery import Eq, Between, Le

# search for objects below 'a/b/c' with ids between 'a' and 'z~'
query = Eq('path','a/b/c') & Between('id', 'a', 'z~')

# evaluate and sort descending by 'modified' and ascending by 'Creator'
context.Catalog.evalAdvancedQuery(query, (('modified','desc'), 'Creator',))

# search 'News' not yet archived and 'File's not yet expired.
now = context.ZopeTime()
query = Eq('portal_type', 'News') & ~ Le('ArchivalDate', now)
	| Eq('portal_type', 'File') & ~ Le('expires', now)
context.Catalog.evalAdvancedQuery(query)

Important note about caching

You must not cache the result of an AdvancedQuery unless you have ensured that sorting has finished (e.g. by accessing the last element in the result). This is because AdvancedQuery uses incremental sorting with BTrees iterators. Like any iterator, they do not like when the base object changes during iteration. Nasty types of (apparently) non-deterministic errors can happen when the index changes during sorting.

Download and installation

Download the most recent version via my Zope Page.

Install by unpacking the tar archive into your Products folder and restart your Zope.

License

This software is open source and licensed under a BSD style license. See the license file in the distribution for details.


Dieter Maurer
Last modified: Wed Aug 4 21:27:13 CEST 2004