RamCacheManagerWithAge/0040775000175100000450000000000007330036224014274 5ustar gboperatorRamCacheManagerWithAge/__init__.py0100644000175100000450000001027507327476042016421 0ustar gboperator############################################################################## # # Zope Public License (ZPL) Version 1.0 # ------------------------------------- # # Copyright (c) Digital Creations. All rights reserved. # # This license has been certified as Open Source(tm). # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are # met: # # 1. Redistributions in source code must retain the above copyright # notice, this list of conditions, and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions, and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # 3. Digital Creations requests that attribution be given to Zope # in any manner possible. Zope includes a "Powered by Zope" # button that is installed by default. While it is not a license # violation to remove this button, it is requested that the # attribution remain. A significant investment has been put # into Zope, and this effort will continue if the Zope community # continues to grow. This is one way to assure that growth. # # 4. All advertising materials and documentation mentioning # features derived from or use of this software must display # the following acknowledgement: # # "This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/)." # # In the event that the product being advertised includes an # intact Zope distribution (with copyright and license included) # then this clause is waived. # # 5. Names associated with Zope or Digital Creations must not be used to # endorse or promote products derived from this software without # prior written permission from Digital Creations. # # 6. Modified redistributions of any form whatsoever must retain # the following acknowledgment: # # "This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/)." # # Intact (re-)distributions of any official Zope release do not # require an external acknowledgement. # # 7. Modifications are encouraged but must be packaged separately as # patches to official Zope releases. Distributions that do not # clearly separate the patches from the original work must be clearly # labeled as unofficial distributions. Modifications which do not # carry the name Zope may be packaged in any form, as long as they # conform to all of the clauses above. # # # Disclaimer # # THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY # EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF # USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT # OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # # This software consists of contributions made by Digital Creations and # many individuals on behalf of Digital Creations. Specific # attributions are listed in the accompanying credits file. # ############################################################################## ''' A RAM Cache Manager with Age properties (Objects expire over time) $Id: __init__.py,v 1.1 2001/07/25 07:58:26 gb Exp $ ''' import RAMCacheManagerWithAge def initialize(context): context.registerClass( RAMCacheManagerWithAge.RAMCacheManagerWithAge, constructors = (RAMCacheManagerWithAge.manage_addRAMCacheManagerForm, RAMCacheManagerWithAge.manage_addRAMCacheManager), icon="cache.gif" ) context.registerHelp() RamCacheManagerWithAge/cache.gif0100664000175100000450000000023607327472611016036 0ustar gboperatorGIF89atttbbb\\\HHH$$$! Cache manager! ,:I8O*0tgg K6G3+h b2sZXM;RamCacheManagerWithAge/help/0040775000175100000450000000000007327476004015236 5ustar gboperatorRamCacheManagerWithAge/help/RAM.stx0100664000175100000450000000506307327473115016416 0ustar gboperator RAM Cache Managers With Age For background information, see the description of cache management. The RAM cache manager allows you to cache the result of calling DTML methods, Python scripts, and SQL methods in memory. It allows you to cache entire pages as well as parts of pages. It provides access statistics and simple configuration options. Storing the result in memory results in the fastest possible cache retrieval, but carries some risks. Unconstrained, it can consume too much RAM. And it doesn't reduce network traffic, it only helps Zope return a result more quickly. Fortunately, RAM cache managers have tunable parameters. You can configure the threshold on the number of entries that should be in the cache, which defaults to 1000. Reduce it if the cache is taking up too much memory or increase it if entries are being cleared too often. You can also configure the cleanup interval. If the RAM cache is fluctuating too much in memory usage, reduce the cleanup interval. Finally, you can configure the list of REQUEST variables that will be used in the cache key. This can be a simple and effective way to distinguish requests from authenticated versus anonymous users or those with session cookies. If you find that some of your objects need certain cache parameters while others need somewhat different parameters, create multiple RAM cache managers. The 'Statistics' tab allows you to view a summary of the contents of the cache. Click the column headers to re-sort the list, twice to sort backwards. You can use the statistics to gauge the benefit of caching each of your objects. For a given object, if the number of hits is less than or not much greater than the number of misses, you probably need to re-evaluate how that object is cached. Although Zope does not prevent you from doing so, it generally does not make sense to associate an image or a file object with a RAM cache manager. It will not cache the image or file data, since the data is already available in RAM. However, another kind of cache manager, an *accelerated HTTP cache manager*, is available and is suitable for images and file objects. This kind of RAM Cache Managers ages objects over time - after the object reaches it's maximum age, it is expired and reloaded from the system. This is helpfull for dynamic objects with changing content that should update only every other minute. RamCacheManagerWithAge/dtml/0040775000175100000450000000000007327606103015241 5ustar gboperatorRamCacheManagerWithAge/dtml/addRCM.dtml0100664000175100000450000000130607327473226017222 0ustar gboperator
Id
RamCacheManagerWithAge/dtml/propsRCM.dtml0100664000175100000450000000340107327602065017626 0ustar gboperator
Title
REQUEST Variables
Threshold Entries
Cleanup Interval (Seconds)
Maximum Age (Seconds)
RamCacheManagerWithAge/dtml/statsRCM.dtml0100664000175100000450000000442107327605027017625 0ustar gboperator

Memory usage is approximate. It is based on the pickled value of the cached data. The cache is cleaned up by removing the least frequently accessed entries since the last cleanup operation. The determination is made using the recent hits counter.

&dtml-path;
&dtml-hits;
&dtml-counter;
&dtml-misses;
&dtml-size;
&dtml-entries;

Nothing is in the cache.

RamCacheManagerWithAge/version.txt0100664000175100000450000000003507327606157016533 0ustar gboperatorRamCacheManagerWithAge-1-0-1 RamCacheManagerWithAge/RAMCacheManagerWithAge.py0100644000175100017510000004115607327606061017531 0ustar gbgb############################################################################## # # Zope Public License (ZPL) Version 1.0 # ------------------------------------- # # Copyright (c) Digital Creations. All rights reserved. # # This license has been certified as Open Source(tm). # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are # met: # # 1. Redistributions in source code must retain the above copyright # notice, this list of conditions, and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions, and the following disclaimer in # the documentation and/or other materials provided with the # distribution. # # 3. Digital Creations requests that attribution be given to Zope # in any manner possible. Zope includes a "Powered by Zope" # button that is installed by default. While it is not a license # violation to remove this button, it is requested that the # attribution remain. A significant investment has been put # into Zope, and this effort will continue if the Zope community # continues to grow. This is one way to assure that growth. # # 4. All advertising materials and documentation mentioning # features derived from or use of this software must display # the following acknowledgement: # # "This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/)." # # In the event that the product being advertised includes an # intact Zope distribution (with copyright and license included) # then this clause is waived. # # 5. Names associated with Zope or Digital Creations must not be used to # endorse or promote products derived from this software without # prior written permission from Digital Creations. # # 6. Modified redistributions of any form whatsoever must retain # the following acknowledgment: # # "This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/)." # # Intact (re-)distributions of any official Zope release do not # require an external acknowledgement. # # 7. Modifications are encouraged but must be packaged separately as # patches to official Zope releases. Distributions that do not # clearly separate the patches from the original work must be clearly # labeled as unofficial distributions. Modifications which do not # carry the name Zope may be packaged in any form, as long as they # conform to all of the clauses above. # # # Disclaimer # # THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY # EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF # USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT # OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # # This software consists of contributions made by Digital Creations and # many individuals on behalf of Digital Creations. Specific # attributions are listed in the accompanying credits file. # ############################################################################## ''' RAM cache manager -- Caches the results of method calls in RAM. $Id: RAMCacheManagerWithAge.py,v 1.2 2001/07/25 18:13:05 gb Exp $ ''' from OFS.Cache import Cache, CacheManager from OFS.SimpleItem import SimpleItem from thread import allocate_lock import time import Globals from Globals import DTMLFile from string import join, split try: from cPickle import Pickler except: from pickle import Pickler try: from cStringIO import dumps except: from pickle import dumps _marker = [] # Create a new marker object. class CacheException (Exception): ''' A cache-related exception. ''' class CacheEntry: ''' Represents a cached value. ''' def __init__(self, index, data, view_name, maximum_age): try: # This is a protective barrier that hopefully prevents # us from caching something that might result in memory # leaks. It's also convenient for determining the # approximate memory usage of the cache entry. self.size = len(dumps(index)) + len(dumps(data)) except: raise CacheException('The data for the cache is not pickleable.') self.data = data self.view_name = view_name self.access_count = 0 self.maximum_age = maximum_age class ObjectCacheEntries: ''' Represents the cache for one Zope object. ''' hits = 0 misses = 0 def __init__(self, path): self.physical_path = path self.lastmod = 0 self.entries = {} def aggregateIndex(self, view_name, req, req_names, local_keys): ''' Returns the index to be used when looking for or inserting a cache entry. view_name is a string. local_keys is a mapping or None. ''' req_index = [] # Note: req_names is already sorted. for key in req_names: if key <> '': if req is None: val = '' else: val = req.get(key, '') req_index.append((str(key), str(val))) if local_keys: local_index = [] for key, val in local_keys.items(): local_index.append((str(key), str(val))) local_index.sort() else: local_index = () return (str(view_name), tuple(req_index), tuple(local_index)) def getEntry(self, lastmod, index): if self.lastmod < lastmod: # Expired. self.entries = {} self.lastmod = lastmod return _marker entry = self.entries.get(index, _marker) if entry is _marker: return _marker now = time.time() if entry.maximum_age < now: return _marker return entry def setEntry(self, lastmod, index, data, view_name, maximum_age): self.lastmod = lastmod self.entries[index] = CacheEntry(index, data, view_name, maximum_age) class RAMCache (Cache): # Note the need to take thread safety into account. # Also note that objects of this class are not persistent, # nor do they make use of acquisition. def __init__(self): # cache maps physical paths to ObjectCacheEntries. self.cache = {} self.writelock = allocate_lock() self.next_cleanup = 0 def initSettings(self, kw): # Note that we lazily allow RAMCacheManager # to verify the correctness of the internal settings. self.__dict__.update(kw) def getObjectCacheEntries(self, ob, create=0): """ Finds or creates the associated ObjectCacheEntries object. Remember to lock writelock when calling with the 'create' flag. """ cache = self.cache path = ob.getPhysicalPath() oc = cache.get(path, None) if oc is None: if create: cache[path] = oc = ObjectCacheEntries(path) else: return None return oc def countAllEntries(self): ''' Returns the count of all cache entries. ''' count = 0 for oc in self.cache.values(): count = count + len(oc.entries) return count def countAccesses(self): ''' Returns a mapping of (n) -> number of entries accessed (n) times ''' counters = {} for oc in self.cache.values(): for entry in oc.entries.values(): access_count = entry.access_count counters[access_count] = counters.get( access_count, 0) + 1 return counters def clearAccessCounters(self): ''' Clears access_count for each cache entry. ''' for oc in self.cache.values(): for entry in oc.entries.values(): entry.access_count = 0 def deleteEntriesAtOrBelowThreshold(self, threshold_access_count): """ Deletes entries that haven't been accessed recently. """ self.writelock.acquire() try: for p, oc in self.cache.items(): for agindex, entry in oc.entries.items(): if entry.access_count <= threshold_access_count: del oc.entries[agindex] if len(oc.entries) < 1: del self.cache[p] finally: self.writelock.release() def cleanup(self): ''' Removes cache entries. ''' new_count = self.countAllEntries() if new_count > self.threshold: counters = self.countAccesses() priorities = counters.items() # Remove the least accessed entries until we've reached # our target count. if len(priorities) > 0: priorities.sort() access_count = 0 for access_count, effect in priorities: new_count = new_count - effect if new_count <= self.threshold: break self.deleteEntriesAtOrBelowThreshold(access_count) self.clearAccessCounters() def getCacheReport(self): """ Reports on the contents of the cache. """ rval = [] for oc in self.cache.values(): size = 0 ac = 0 views = [] keys = [] for key in oc.entries.keys(): entry = oc.entries.get(key) kinfo = {'count': entry.access_count, 'key': key, 'maxage': entry.maximum_age } keys.append(kinfo) size = size + entry.size ac = ac + entry.access_count view = entry.view_name or '' if view not in views: views.append(view) views.sort() info = {'path': join(oc.physical_path, '/'), 'hits': oc.hits, 'misses': oc.misses, 'size': size, 'counter': ac, 'views': views, 'entries': len(oc.entries), 'keys': keys, } rval.append(info) return rval def ZCache_invalidate(self, ob): ''' Invalidates the cache entries that apply to ob. ''' path = ob.getPhysicalPath() # Invalidates all subobjects as well. self.writelock.acquire() try: for p, oc in self.cache.items(): pp = oc.physical_path if pp[:len(path)] == path: del self.cache[p] finally: self.writelock.release() def ZCache_get(self, ob, view_name='', keywords=None, mtime_func=None, default=None): ''' Gets a cache entry or returns default. ''' oc = self.getObjectCacheEntries(ob) if oc is None: return default lastmod = ob.ZCacheable_getModTime(mtime_func) index = oc.aggregateIndex(view_name, ob.REQUEST, self.request_vars, keywords) entry = oc.getEntry(lastmod, index) if entry is _marker: return default oc.hits = oc.hits + 1 entry.access_count = entry.access_count + 1 return entry.data def ZCache_set(self, ob, data, view_name='', keywords=None, mtime_func=None): ''' Sets a cache entry. ''' now = time.time() if self.next_cleanup <= now: self.cleanup() self.next_cleanup = now + self.cleanup_interval lastmod = ob.ZCacheable_getModTime(mtime_func) self.writelock.acquire() try: oc = self.getObjectCacheEntries(ob, create=1) index = oc.aggregateIndex(view_name, ob.REQUEST, self.request_vars, keywords) oc.setEntry(lastmod, index, data, view_name, now+self.maximum_age) oc.misses = oc.misses + 1 finally: self.writelock.release() caches = {} PRODUCT_DIR = split(__name__, '.')[-2] class RAMCacheManagerWithAge (CacheManager, SimpleItem): ' ' __ac_permissions__ = ( ('View management screens', ('getSettings', 'manage_main', 'manage_stats', 'getCacheReport', 'sort_link',)), ('Change cache managers', ('manage_editProps',), ('Manager',)), ) manage_options = ( {'label':'Properties', 'action':'manage_main', 'help':(PRODUCT_DIR, 'RAM.stx'),}, {'label':'Statistics', 'action':'manage_stats', 'help':(PRODUCT_DIR, 'RAM.stx'),}, ) + CacheManager.manage_options + SimpleItem.manage_options meta_type = 'RAM Cache Manager With Age' def __init__(self, ob_id): self.id = ob_id self.title = '' self._settings = { 'threshold': 1000, 'cleanup_interval': 300, 'maximum_age': 300, 'request_vars': ('AUTHENTICATED_USER',)} self.__cacheid = '%s_%f' % (id(self), time.time()) def getId(self): ' ' return self.id ZCacheManager_getCache__roles__ = () def ZCacheManager_getCache(self): cacheid = self.__cacheid try: return caches[cacheid] except KeyError: cache = RAMCache() cache.initSettings(self._settings) caches[cacheid] = cache return cache def getSettings(self): 'Returns the current cache settings.' return self._settings.copy() manage_main = DTMLFile('dtml/propsRCM', globals()) def manage_editProps(self, title, settings=None, REQUEST=None): 'Changes the cache settings.' if settings is None: settings = REQUEST self.title = str(title) request_vars = list(settings['request_vars']) request_vars.sort() self._settings = { 'threshold': int(settings['threshold']), 'cleanup_interval': int(settings['cleanup_interval']), 'maximum_age': int(settings['maximum_age']), 'request_vars': tuple(request_vars)} cache = self.ZCacheManager_getCache() cache.initSettings(self._settings) if REQUEST is not None: return self.manage_main( self, REQUEST, manage_tabs_message='Properties changed.') manage_stats = DTMLFile('dtml/statsRCM', globals()) def _getSortInfo(self): """ Returns the value of sort_by and sort_reverse. If not found, returns default values. """ req = self.REQUEST sort_by = req.get('sort_by', 'hits') sort_reverse = int(req.get('sort_reverse', 1)) return sort_by, sort_reverse def getCacheReport(self): """ Returns the list of objects in the cache, sorted according to the user's preferences. """ sort_by, sort_reverse = self._getSortInfo() c = self.ZCacheManager_getCache() rval = c.getCacheReport() if sort_by: rval.sort(lambda e1, e2, sort_by=sort_by: cmp(e1[sort_by], e2[sort_by])) if sort_reverse: rval.reverse() return rval def sort_link(self, name, id): """ Utility for generating a sort link. """ sort_by, sort_reverse = self._getSortInfo() url = self.absolute_url() + '/manage_stats?sort_by=' + id newsr = 0 if sort_by == id: newsr = not sort_reverse url = url + '&sort_reverse=' + (newsr and '1' or '0') return '%s' % (url, name) Globals.default__class_init__(RAMCacheManagerWithAge) manage_addRAMCacheManagerForm = DTMLFile('dtml/addRCM', globals()) def manage_addRAMCacheManager(self, id, REQUEST=None): 'Adds a RAM cache manager to the folder.' self._setObject(id, RAMCacheManagerWithAge(id)) if REQUEST is not None: return self.manage_main(self, REQUEST)