Zope-2.8.7-final/0000775000076400007640000000000010435610645012544 5ustar ajungajungZope-2.8.7-final/import/0000775000076400007640000000000010435500331014044 5ustar ajungajungZope-2.8.7-final/import/README.txt0000775000076400007640000000032310435500331015543 0ustar ajungajungThis directory is used by the running Zope process to import objects into the ZODB. Please place any files that you wish to be able to import into this directy. For more information, please see The Zope Book. Zope-2.8.7-final/import/ZopeTutorialExamples.zexp0000664000076400007640000046032710435500331021130 0ustar ajungajungZEXP ((U OFS.FolderqUFolderqtqNt.}q(Ulesson3q(U q(hUFolderqttQUlesson2q(U q (hUFolderq ttQUlesson1q (U q (hUFolderq ttQUlesson7q(U q(hUFolderqttQUlesson6q(U q(hUFolderqttQUlesson5q(U q(hUFolderqttQUtitleqUTutorial ExamplesqUlesson9q(U q(hUFolderqttQUlesson8q(U q(hUFolderqttQU index_htmlq(U q (UOFS.DTMLMethodq!U DTMLMethodq"ttQU_ownerq#(]q$U acl_usersq%aUamosq&tUlesson12q'(U q((hUFolderq)ttQUlesson11q*(U q+(hUFolderq,ttQUlesson10q-(U q.(hUFolderq/ttQU_objectsq0(}q1(U meta_typeq2UFolderq3Uidq4h u}q5(U meta_typeq6UFolderq7Uidq8h*u}q9(h6h7h8hu}q:(U meta_typeq;UFolderq(h;h View this DTML Method to begin the Zope Tutorial.

Welcome to the Zope Tutorial. Click the button below to open the Zope Tutorial in a new window.

Make sure cookies are enabled in your web browser. The Zope Tutorial uses cookies to keep track of where the tutorial examples are located.

');"> ">
qUglobalsq }q Utitleq UREADMEq U__name__q U index_htmlqU__ac_local_roles__q}qUamosq]qUOwnerqasu. ((U OFS.FolderqUFolderqtqNt.}q(U form.htmlq(U 3q(U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqttQUtitleq U!Lesson 12. The Elvis Files, cont.q UinsertSightingq (U q (UProducts.ZSQLMethods.SQLq USQLqttQUsightings.htmlq(U 2q(hUZopePageTemplateqttQU getSightingsq(U q(h USQLqttQU connectionq(U q(UProducts.ZGadflyDA.DAqU ConnectionqttQU action.pyq(U 5q(U#Products.PythonScripts.PythonScriptqU PythonScriptqttQU_objectsq(}q(U meta_typeqU Z SQL Methodq Uidq!hu}q"(U meta_typeq#UZ Gadfly Database Connectionq$Uidq%hu}q&(U meta_typeq'U Z SQL Methodq(Uidq)h u}q*(U meta_typeq+U Page Templateq,Uidq-Usightings.htmlq.u}q/(h+h,h-U form.htmlq0u}q1(h+h,h-U thanks.htmlq2u}q3(h+UScript (Python)q4h-U action.pyq5utU thanks.htmlq6(U 4q7(hUZopePageTemplateq8ttQU__ac_local_roles__q9}q:Uamosq;]qUlesson12q?u. m((U OFS.FolderqUFolderqtqNt.}q(U getSightingsq(U q(UProducts.ZSQLMethods.SQLqUSQLqttQU connectionq (U q (UProducts.ZGadflyDA.DAq U Connectionq ttQUtitleq ULesson 11. The Elvis FilesqU_objectsq(}q(U meta_typeqUFolderqUidqUSetupqu}q(U meta_typeqU Z SQL MethodqUidqhu}q(U meta_typeqUZ Gadfly Database ConnectionqUidqh u}q(U meta_typeqU Page TemplateqUidq Usightings.htmlq!uth(U q"(hUFolderq#ttQUsightings.htmlq$(U 1q%(U'Products.PageTemplates.ZopePageTemplateq&UZopePageTemplateq'ttQU__ac_local_roles__q(}q)Uamosq*]q+UOwnerq,ashUlesson11q-u. ((U OFS.FolderqUFolderqtqNt.}q(U form.htmlq(U .q(U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqttQUtitleq ULesson 10. So You've Seen Elvisq U webmasterq UUmailhostq (U q (UProducts.MailHost.MailHostqUMailHostqttQU action.pyq(U 0q(U#Products.PythonScripts.PythonScriptqU PythonScriptqttQU _propertiesq(}q(UtypeqUstringqUidqh u}q(hUstringqhU webmasterqutU_objectsq(}q(U meta_typeqU Mail HostqUidq Umailhostq!u}q"(U meta_typeq#U Page Templateq$Uidq%hu}q&(h#h$h%U thanks.htmlq'u}q((h#UScript (Python)q)h%U action.pyq*uth'(U /q+(hUZopePageTemplateq,ttQU__ac_local_roles__q-}q.Uamosq/]q0UOwnerq1ashUlesson10q2u. ((U OFS.FolderqUFolderqtqNt.}q(UtitleqUSite Contact InformationqUfoodq(U q(hUFolderq ttQU home.htmlq (U q (U'Products.PageTemplates.ZopePageTemplateq UZopePageTemplateq ttQUstandard_template.ptq(U q(h UZopePageTemplateqttQU header.gifq(U q(U OFS.ImageqUImageqttQU gracelandq(U q(hUFolderqttQU _propertiesq(}q(UtypeqUstringqUidqUtitleqUmodeqUwutU_objectsq(}q (U meta_typeq!UImageq"Uidq#hu}q$(h!UFolderq%h#hu}q&(U meta_typeq'UFolderq(Uidq)U gracelandq*u}q+(U meta_typeq,U Page Templateq-hU home.htmlq.u}q/(h,h-hhutU__ac_local_roles__q0}q1Uamosq2]q3UOwnerq4ash#Ulesson4q5u. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqUc

title

This pie is fit for a King.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqULemon Meringue PieqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUpie.htmlqUexpandqKu. O((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

This sandwhich was an Elvis favorite.

There is some question as to whether the bananas were sliced or mashed on the King's sandwiches.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU(Fried Peanutbutter and Banana SandwichesqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsandwiches.htmlqUexpandqKu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqTO

Welcome to title, your source for information about Elvis and Elvis sightings.

Elvis liked to eat sandwiches, pie, and meatloaf.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU Elvis LivesqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU home.htmlqUexpandqKu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqUq

title

What could be better than a loaf of meat?

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUMeatloafqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU meatloaf.htmlqUexpandqKu. ((U OFS.ImageqUImageqtqNt.}q(UwidthqU289qU preconditionqUU content_typeqU image/gifq Utitleq UElvis Lives Headerq U__name__q U header.gifq UheightqU58qUdataqT GIF89a!:֐ՐՏԏҎҎюЍЍόΌ͌ˋȉlj ň ć Ç ~~}|yxxu"t#s$s$q'p(o*n*l-l-j0h2f4|e6{d7zd7ub:sa;r`طr*?١ .PW-gɑ]57*:Iwi;X92b)L-WcOs@C)*t)kr3@]l45姈T(1q=7:ݶ t3 Az*b6Ny"jK<Гl/[vGU2x7!/o C@lQLQJ)N/ANAGD~(AQ3AH 6F#   blb=cvnGBgk; ! BNu"JCY)k1H`Ely >p0ْt )II4R;nrq8&%^p C5j9j\tSh-iYw'`L& MN:D#'1YPNH7]mpAȥpq"^gx1.F+fQ̠ȭ"ZRLccxa\I*D'E 4TEݔpQƑ S6 7,98c+@ p'[bcSDb蘲! HRł'):=bG:CJGKtc( *! %!wGEȍ)YnUp#J3\O z)i/ v M)>PG7!f;L S)֭N(隫YtSdBpx+U_'v4Fڍ5xZ Ԕ&KNota R6yd nڵ%u:0&o@N^T58VD{r@ˎ r!ۋB쬲jTгUE!z'IBR1d`?s*](nsjj6%&n#nBa}r$T( PAjm7i!ElWE$*ȥA(6ŕMI0p[34%GdB  dD O8:HrZ9||klcQpY}"ڧZPm#XAǫs"}D"HIĝA# s܌:z~Dd&>'JޒE\gMZָεw^א;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM u. ((U OFS.FolderqUFolderqtqNt.}q(U graceland.jpgq(U q(U OFS.ImageqUImageqttQUtitleq UGraceland, Home of Elvisq U_objectsq (}q (U meta_typeq UImageqUidqhu}q(U meta_typeqU Page TemplateqUidqU index_htmlqutU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU gracelandqU index_htmlq(U q(U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqttQu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

Welcome to title, your source for information about Elvis and Elvis sightings.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU Elvis LivesqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU home.htmlqUexpandqKu. ((U OFS.ImageqUImageqtqNt.}q(UwidthqU289qU preconditionqUU content_typeqU image/gifq Utitleq UElvis Lives Headerq U__name__q U header.gifq UheightqU58qUdataqT GIF89a!:֐ՐՏԏҎҎюЍЍόΌ͌ˋȉlj ň ć Ç ~~}|yxxu"t#s$s$q'p(o*n*l-l-j0h2f4|e6{d7zd7ub:sa;r`طr*?١ .PW-gɑ]57*:Iwi;X92b)L-WcOs@C)*t)kr3@]l45姈T(1q=7:ݶ t3 Az*b6Ny"jK<Гl/[vGU2x7!/o C@lQLQJ)N/ANAGD~(AQ3AH 6F#   blb=cvnGBgk; ! BNu"JCY)k1H`Ely >p0ْt )II4R;nrq8&%^p C5j9j\tSh-iYw'`L& MN:D#'1YPNH7]mpAȥpq"^gx1.F+fQ̠ȭ"ZRLccxa\I*D'E 4TEݔpQƑ S6 7,98c+@ p'[bcSDb蘲! HRł'):=bG:CJGKtc( *! %!wGEȍ)YnUp#J3\O z)i/ v M)>PG7!f;L S)֭N(隫YtSdBpx+U_'v4Fڍ5xZ Ԕ&KNota R6yd nڵ%u:0&o@N^T58VD{r@ˎ r!ۋB쬲jTгUE!z'IBR1d`?s*](nsjj6%&n#nBa}r$T( PAjm7i!ElWE$*ȥA(6ŕMI0p[34%GdB  dD O8:HrZ9||klcQpY}"ڧZPm#XAǫs"}D"HIĝA# s܌:z~Dd&>'JޒE\gMZָεw^א;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM u. ((U OFS.ImageqUImageqtqNt.}q(UwidthqU289qU preconditionqUU content_typeqU image/gifq Utitleq UElvis Lives Headerq U__name__q U header.gifq UheightqU58qUdataqT GIF89a!:֐ՐՏԏҎҎюЍЍόΌ͌ˋȉlj ň ć Ç ~~}|yxxu"t#s$s$q'p(o*n*l-l-j0h2f4|e6{d7zd7ub:sa;r`طr*?١ .PW-gɑ]57*:Iwi;X92b)L-WcOs@C)*t)kr3@]l45姈T(1q=7:ݶ t3 Az*b6Ny"jK<Гl/[vGU2x7!/o C@lQLQJ)N/ANAGD~(AQ3AH 6F#   blb=cvnGBgk; ! BNu"JCY)k1H`Ely >p0ْt )II4R;nrq8&%^p C5j9j\tSh-iYw'`L& MN:D#'1YPNH7]mpAȥpq"^gx1.F+fQ̠ȭ"ZRLccxa\I*D'E 4TEݔpQƑ S6 7,98c+@ p'[bcSDb蘲! HRł'):=bG:CJGKtc( *! %!wGEȍ)YnUp#J3\O z)i/ v M)>PG7!f;L S)֭N(隫YtSdBpx+U_'v4Fڍ5xZ Ԕ&KNota R6yd nڵ%u:0&o@N^T58VD{r@ˎ r!ۋB쬲jTгUE!z'IBR1d`?s*](nsjj6%&n#nBa}r$T( PAjm7i!ElWE$*ȥA(6ŕMI0p[34%GdB  dD O8:HrZ9||klcQpY}"ڧZPm#XAǫs"}D"HIĝA# s܌:z~Dd&>'JޒE\gMZָεw^א;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM u. $((U OFS.FolderqUFolderqtqNt.}q(U elvis2.gifq(U q(U OFS.ImageqUImageqttQUtitleq UElvis Photo Archiveq U_objectsq (}q (U meta_typeq UImageqUidqU elvis1.gifqu}q(h hhhuth(U q(hUImageqttQU__ac_local_roles__q}qUamosq]qUOwnerqashU photoArchivequ. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqUx

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU photos.htmlqUexpandqKu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

Sighting goes here
qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqURecent SightingsqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightings.htmlqUexpandqKu. F((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUsightingsFolderqU_objectsq(}q (U meta_typeq UFileq Uidq U sighting1q u}q(h h h U sighting2qu}q(h h h U sighting3qutU__ac_local_roles__q}qUamosq]qUOwnerqash(U q(U OFS.ImageqUFileqttQh(U q(hUFileqttQh (U q(hUFileqttQu. !((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

Sighting goes here
qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqURecent SightingsqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightings.htmlqUexpandqKu. F((U OFS.FolderqUFolderqtqNt.}q(UtitleqUU_objectsq(}q(U meta_typeqUFileq Uidq U sighting1q u}q (hh h U sighting2q u}q(hh h U sighting3quth (U q(U OFS.ImageqUFileqttQU__ac_local_roles__q}qUamosq]qUOwnerqash(U q(hUFileqttQh (U q(hUFileqttQUidqUsightingsFolderqu. %((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqTG

title

New

Sighting goes here

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqURecent SightingsqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightings.htmlqUexpandqKu. &((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U Python_magicqU- qUerrorsq)Uwarningsq)U Script_magicq KU_bodyq T-""" Returns the time of the last visit as a DateTime object. Also sets records the current time in a cookie. """ request = container.REQUEST if request.cookies.has_key('lastVisited'): # retrieve the last visit time from the cookie last=request.cookies['lastVisited'] else: # there's no cookie so pick a time in the past last='1/1/1999' # convert the date string to a DateTime object last=DateTime(last) # finally set current time in the cookie request.response.setCookie('lastVisited', DateTime()) # return time of last visit return last q U_codeq TcsdZdS(Nc st}t}|td}|||dddo|||dd}n  d}t|}|||dddt|SdS(NsREQUESTscookiesshas_keys lastVisiteds1/1/1999sresponses setCookie(s _getitem_s_getitems _getattr_s_getattrs containersrequestslastsDateTime(s _getitem_s_getitems _getattr_s_getattrs containersrequestslastsDateTime((sScript (Python)s lastVisit.pys (s lastVisit.py(s lastVisit.py((sScript (Python)ssq U func_defaultsqNU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(U name_contextqUcontextqU name_subpathqUtraverse_subpathqU name_m_selfqUscriptqUname_containerqU containerqusbU func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q (U co_argcountq!KU co_varnamesq"(U _getitem_q#U_getitemq$U _getattr_q%U_getattrq&hUrequestq'Ulastq(UDateTimeq)tubU__ac_local_roles__q*}q+Uamosq,]q-UOwnerq.asUidq/U lastVisit.pyq0U_paramsq1UU_dav_writelocksq2(U 'q3(U Persistenceq4UPersistentMappingq5ttQu. F((U OFS.FolderqUFolderqtqNt.}q(UtitleqUU sighting2q(U +q(U OFS.ImageqUFileq ttQU sighting3q (U ,q (hUFileq ttQU sighting1q (U *q(hUFileqttQU_objectsq(}q(U meta_typeqUFileqUidqh u}q(hhhhu}q(hhhh utU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightingsFolderqu. ]((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT

title

Upload a picture to the Elvis Photo Archive.

File:

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU Submit PhotoqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU form.htmlqUexpandqKu. ")((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title (size bytes)

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU photos.htmlqUexpandqKu. $((U OFS.FolderqUFolderqtqNt.}q(U elvis2.gifq(U q(U OFS.ImageqUImageqttQUtitleq UElvis Photo Archiveq U_objectsq (}q (U meta_typeq UImageqUidqU elvis1.gifqu}q(h hhhuth(U q(hUImageqttQU__ac_local_roles__q}qUamosq]qUOwnerqashU photoArchivequ. ((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U Python_magicqU- qUerrorsq)Uwarningsq)U Script_magicq KU_bodyq U""" Create a new image in the photo archive folder. Then return a confirmation page. """ folder=container['photoArchive'] folder.manage_addImage(id='', file=file) page=container['thanks.html'] return page() q U_codeq TDcsdZdS(Ncs`t}t}|td}||dddd||td} |SdS(Ns photoArchivesmanage_addImagesidssfiles thanks.html(s _getitem_s_getitems _getattr_s_getattrs containersfoldersfilespage(sfiles _getitem_s_getitems _getattr_s_getattrs containersfolderspage((sScript (Python)s action.pys(s action.py(s action.py((sScript (Python)ssq U func_defaultsqNU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(U name_contextqUcontextqU name_subpathqUtraverse_subpathqU name_m_selfqUscriptqUname_containerqU containerqusbU__ac_local_roles__q}qUamosq]q UOwnerq!asUidq"U action.pyq#U_paramsq$Ufileq%U func_codeq&(cShared.DC.Scripts.Signature FuncCode q'oq(}q)(U co_argcountq*KU co_varnamesq+(Ufileq,U _getitem_q-U_getitemq.U _getattr_q/U_getattrq0hUfolderq1Upageq2tq3ubu. !((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU}

title

You can view the photos here.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU Thanks for Your Photo SubmissionqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU thanks.htmlqUexpandqKu. 3u((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT

title

You the form below to report an Elvis sighting. Your sighting information will be entered into our database.

Your name:

Sighting location:

Sighting date:

Describe the sighting:

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUReport an Elvis SightingqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU form.htmlqUexpandqKu. ((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(U_argq(cShared.DC.ZRDB.Aqueduct Args qo}q(U_keysq]q (Ulocationq Udateq Unameq U descriptionq eU_dataq}q(h }qh }qh }qh }quubU arguments_srcqU!location date name descriptionqUsrcqUinsert into elvis_sightings values( , , , )qUtemplateqcShared.DC.ZRDB.DA SQL qNRq}q(U_varsq}qU__name__qUqUglobalsq }q!Urawq"hubU_colq#)Utitleq$UInsert Elvis Sightingq%U__ac_local_roles__q&}q'Uamosq(]q)UOwnerq*asUidq+UinsertSightingq,U connection_idq-U connectionq.u. 2((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT

Report a sighting

title

date -- location

Reported by name

description

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqURecent Elvis SightingsqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightings.htmlqUexpandqKu. I((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(U_argq(cShared.DC.ZRDB.Aqueduct Args qo}q(U_keysq]q U_dataq }q ubU arguments_srcq UUsrcq Uselect * from elvis_sightingsqUtemplateqcShared.DC.ZRDB.DA SQL qNRq}q(U_varsq}qU__name__qUqUglobalsq}qUrawqhubU_colq]q(}q(UwidthqNUnullqNUtypeqUsUnameq UNAMEq!u}q"(hNhNhUsh U DESCRIPTIONq#u}q$(hNhNhUsh ULOCATIONq%u}q&(hNhNhUsh UDATEq'ueUtitleq(U!Get Elvis Sightings From Databaseq)U__ac_local_roles__q*}q+Uamosq,]q-UOwnerq.asUidq/U getSightingsq0U connection_idq1U connectionq2u. ((UProducts.ZGadflyDA.DAqU ConnectionqtqNt.}q(Uconnection_stringqUdemoqU__ac_local_roles__q}qUamosq ]q UOwnerq asUidq U connectionq UtitleqUTutorial Database Connectionqu. 5+((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U Python_magicqU- qUerrorsq)Uwarningsq)U Script_magicq KU_bodyq T""" Insert sighting information into the datebase """ # get sql method insert=container['insertSighting'] # call it, passing arguments insert(name=name, location=location, date=date, description=description) # return a confirmation page page=container['thanks.html'] return page() q U_codeq T_csdZdS(Nc s]t}|td} |d|d|d|d| |td} |SdS(NsinsertSightingsnameslocationsdates descriptions thanks.html( s _getitem_s_getitems containersinsertsnameslocationsdates descriptionspage( snameslocationsdates descriptions _getitem_s_getitems containersinsertspage((sScript (Python)s action.pys "(s action.py(s action.py((sScript (Python)ssq U func_defaultsqNU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(U name_contextqUcontextqU name_subpathqUtraverse_subpathqU name_m_selfqUscriptqUname_containerqU containerqusbU__ac_local_roles__q}qUamosq]q UOwnerq!asUidq"U action.pyq#U_paramsq$U!name, location, date, descriptionq%U func_codeq&(cShared.DC.Scripts.Signature FuncCode q'oq(}q)(U co_argcountq*KU co_varnamesq+(Unameq,Ulocationq-Udateq.U descriptionq/U _getitem_q0U_getitemq1hUinsertq2Upageq3tubu. 4((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

Thanks for submitting your sighting.

Return to the list of sightings

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUProcess an Elivis SightingqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU thanks.htmlqUexpandqKu. K((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(UsrcqUselect * from elvis_sightingsqU arguments_srcqUUtemplateqcShared.DC.ZRDB.DA SQL q NRq }q (U_varsq }q UrawqhUglobalsq}qU__name__qUqubUtitleqU!Get Elvis Sightings From DatabaseqU_argq(cShared.DC.ZRDB.Aqueduct Args qoq}q(U_keysq]qU_dataq}qubU_colq]q(}q(Uwidthq NUnullq!NUtypeq"UsUnameq#UDATEq$u}q%(h Nh!Nh"Ush#U DESCRIPTIONq&u}q'(h Nh!Nh"Ush#ULOCATIONq(u}q)(h Nh!Nh"Ush#UNAMEq*ueU__ac_local_roles__q+}q,Uamosq-]q.UOwnerq/asUidq0U getSightingsq1U connection_idq2U connectionq3u. ((UProducts.ZGadflyDA.DAqU ConnectionqtqNt.}q(Uconnection_stringqUdemoqU__ac_local_roles__q}qUamosq ]q UOwnerq asUidq U connectionq UtitleqUTutorial Database Connectionqu. ((U OFS.FolderqUFolderqtqNt.}q(Uinsertq(U q(UProducts.ZSQLMethods.SQLqUSQLqttQUtitleq UTutorial Database Setupq Usetupq (U q (UOFS.DTMLMethodq U DTMLMethodqttQUdropq(U q(hUSQLqttQUidqUSetupqU_objectsq(}q(U meta_typeqU Z SQL MethodqhUcreatequ}q(hhhUinsertqu}q(hU DTML MethodqhU sample_dataqu}q(hhhUsetupqu}q (U meta_typeq!U Z SQL Methodq"Uidq#U table_existsq$u}q%(h!h"h#Udropq&u}q'(U meta_typeq(U Z SQL Methodq)Uidq*Udeleteq+uth$(U q,(hUSQLq-ttQU__ac_local_roles__q.}q/Uamosq0]q1UOwnerq2asUcreateq3(U q4(hUSQLq5ttQU sample_dataq6(U q7(h U DTMLMethodq8ttQUdeleteq9(U q:(hUSQLq;ttQu. 1((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT]

title

date -- location

Reported by name

description

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqURecent Elvis SightingsqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsightings.htmlqUexpandqKu. .=((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT

title

You the form below to report an Elvis sighting. Your sighting information will be mailed to the webmaster.

Your name:

Sighting location:

Describe the sighting:

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUSubmit an Elvis SightingqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU form.htmlqUexpandqKu. ((UProducts.MailHost.MailHostqUMailHostqtqNt.}q(U__ac_local_roles__q}qUamosq]qUOwnerq asU smtp_portq KU smtp_hostq Umail.aracnet.comq Uidq UmailhostqUtitleqUMail Host -- Sends Emailqu. 0((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U Python_magicqU- qUerrorsq)Uwarningsq)U Script_magicq KU_bodyq T""" Sends email documenting an elvis sighting """ # locate the mailhost mailhost=container['mailhost'] # email message settings recipient='user@host' sender='user@host' subject='New Elvis Sighting' # format the email message message='Elvis was spotted by ' + name + ' in ' + location + '\n\n' + description # send the message mailhost.simple_send(recipient, sender, subject, message) # return a confirmation message page=container['thanks.html'] return page() q U_codeq TVcsdZdS(Ncst}t}|td}d}  d}  d}  d|d|d|} ||d| | | | |td} | SdS( Nsmailhosts user@hostsNew Elvis SightingsElvis was spotted by s in s s simple_sends thanks.html(s _getitem_s_getitems _getattr_s_getattrs containersmailhosts recipientssenderssubjectsnameslocations descriptionsmessagespage(snameslocations descriptions _getitem_s_getitems _getattr_s_getattrs containersmailhosts recipientssenderssubjectsmessagespage((sScript (Python)s action.pys    (s action.py(s action.py((sScript (Python)ssq U func_defaultsqNU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(U name_contextqUcontextqU name_subpathqUtraverse_subpathqU name_m_selfqUscriptqUname_containerqU containerqusbU__ac_local_roles__q}qUamosq]q UOwnerq!asUidq"U action.pyq#U_paramsq$Uname, location, descriptionq%U func_codeq&(cShared.DC.Scripts.Signature FuncCode q'oq(}q)(U co_argcountq*KU co_varnamesq+(Unameq,Ulocationq-U descriptionq.U _getitem_q/U_getitemq0U _getattr_q1U_getattrq2U containerq3Umailhostq4U recipientq5Usenderq6Usubjectq7Umessageq8Upageq9tq:ubu. /((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

Thanks for submitting your sighting. Mail was sent to the webmaster.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUProcess an Elivis SightingqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU thanks.htmlqUexpandqKu. ((U OFS.FolderqUFolderqtqNt.}q(Upie.htmlq(U q(U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqttQUsandwiches.htmlq (U q (hUZopePageTemplateq ttQUtitleq UElvis's Favorite Foodsq U_objectsq(}q(U meta_typeqU Page TemplateqUidqU meatloaf.htmlqu}q(hhhUpie.htmlqu}q(hhhUsandwiches.htmlqutU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUfoodqU meatloaf.htmlq(U q (hUZopePageTemplateq!ttQu. S((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT

Welcome to title, your source for information about Elvis and Elvis sightings.

Elvis was born in Graceland.

Elvis liked to eat sandwiches, pie, and meatloaf.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU Elvis LivesqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU home.htmlqUexpandqKu. |((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT The Title
This is where the page's body text goes.
qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUU__ac_local_roles__q}qUSystem Processesq]qUOwnerqasUidqUstandard_template.ptqUexpandqKu. ((U OFS.ImageqUImageqtqNt.}q(UwidthqU289qU preconditionqUU content_typeqU image/gifq Utitleq UElvis Lives Headerq U__name__q U header.gifq UheightqU58qUdataqT GIF89a!:֐ՐՏԏҎҎюЍЍόΌ͌ˋȉlj ň ć Ç ~~}|yxxu"t#s$s$q'p(o*n*l-l-j0h2f4|e6{d7zd7ub:sa;r`طr*?١ .PW-gɑ]57*:Iwi;X92b)L-WcOs@C)*t)kr3@]l45姈T(1q=7:ݶ t3 Az*b6Ny"jK<Гl/[vGU2x7!/o C@lQLQJ)N/ANAGD~(AQ3AH 6F#   blb=cvnGBgk; ! BNu"JCY)k1H`Ely >p0ْt )II4R;nrq8&%^p C5j9j\tSh-iYw'`L& MN:D#'1YPNH7]mpAȥpq"^gx1.F+fQ̠ȭ"ZRLccxa\I*D'E 4TEݔpQƑ S6 7,98c+@ p'[bcSDb蘲! HRł'):=bG:CJGKtc( *! %!wGEȍ)YnUp#J3\O z)i/ v M)>PG7!f;L S)֭N(隫YtSdBpx+U_'v4Fڍ5xZ Ԕ&KNota R6yd nڵ%u:0&o@N^T58VD{r@ˎ r!ۋB쬲jTгUE!z'IBR1d`?s*](nsjj6%&n#nBa}r$T( PAjm7i!ElWE$*ȥA(6ŕMI0p[34%GdB  dD O8:HrZ9||klcQpY}"ڧZPm#XAǫs"}D"HIĝA# s܌:z~Dd&>'JޒE\gMZָεw^א;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM u. ((U OFS.FolderqUFolderqtqNt.}q(U graceland.jpgq(U q(U OFS.ImageqUImageqttQUtitleq UGraceland, Home of Elvisq U_objectsq (}q (U meta_typeq UImageqUidqhu}q(U meta_typeqU Page TemplateqUidqU index_htmlqutU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU gracelandqU index_htmlq(U q(U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqttQu. e((U OFS.ImageqUImageqtqNt.}q(UwidthqU301qU preconditionqUU content_typeqU image/jpegq Utitleq UGraceland photoq U__name__q U graceland.jpgq UheightqU192qUdataqTdJFIFC     C   -" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?-o>vh>s =ɶxEd`+!xwM6OgHfw5Os3IPٸ3^ JoR}>[)?MyqGsOdǀTae=x3Ρ˫ΑB- a1:ͥJWdoJFùIYVEm5Kk/X<|spiΝWխQxH>Y  E4STc㷁4-BQGH]3s~B`sԚ ,~o:mǪPHB w8r0˧[<ї +fN"4{ho I+ gp9Sj[KO ז:Ɠ%Z( }Tl[, 1q>hÛm YCՄjer.͘<6٭$՞Ci%嵝Җ/G( (9pq_ߍď{Yqux&ݷͲ,>Փ+vkbl|QFx~ h3̓@[D&$"w0AqC| 񥏆`ap_} *p+ʬnFncpY&"gsCx߆Αc+,v_NU@dr1='̎uf Wzפ)IVT #@#`8zZfhL {Tmca߃xKSc"r'Hx }3SUe aYwUR?u#?|#qyou͕[1VTa!+:dc5Ưh)ռ}+Z o}jSh#@/XR{"Q:9WhV =kV 7v%GF ̑DQgRwVj&7~74x4xt2ZW$erUHO0.y`lQ$9 ܧjr׏4_]Ocߛ+z*LIdB+3``w> |.կhK4I. !s P>IE1r w~M=d/ReXϕ_g$^U08 8:qZbPt g6­{2,oMmq_4(%Nam~W+]kk2_;bQ]NхeD ֭h?wzS ,]$A ` (-ZO .kQ]X ֯uxvWɅFcgBґkL&yE)?媡@]F_!7`^sj?}6K $ ܍m}UljfdҭtM}jUHq pIb},+3|I,:tQjH>T#r'H)|}N]R ,-CszɩZIVf8EHˍ r>o<τφmʸԢծ\q$T`,07~x PڦiwZkia*()V]CQ֦JnPʷc]%hC`:Ƌ}txJ={MFW>|K'OCdt&L׎/^g%/88VڌaSԊII{8X/CO_cTP0sJ>.|SbġbxIX\dǯSE𺺒@wg *B3!1 |@:~% f瞏Dr$k[CQO+<g._39WZi3霅c*~!€nAVV/^kĉge .F8s\#+6 f˾Fo5HU\0JAzƳdRE9s H\)v[{-h!e?\qi^o[ÖJaDzսs]儍r>[5- )ϯk?e)+ay |??${wOUJ{fľ(ZAqg2ĭnw(KO'w } zaj@RkyM'2}QQ]xlygM7(oϻӼR.[X8KկgwQ($]zi+p/*q܏/ i|1կ-`61-i$$ IaK>`W[Vw7$p߄6"~oWGIX I]oU\]_my$Q99ZNmB7nƍM4;A̒ʃo8SÂ~85 `W$.::X|Cyǥ eM s 8#]j׷6{$6CyX ۯ-[&6kͮ<6A6G})F +1c:Jռ'FSUcQkv]/+ʸ ̪8wACL~2|SJ8@K;(`yt ^އS.>.99ߞ} <=}yXfuݢBw++b+4cm_6{,-7Q^)vzǏ+JRKDs(G5jF:@u;ۉf9s>SczװmkLXUb$)gvr:{kU}x'3gl5;(w\s?/׎L)Y9仸4*N1͟~k]14+ _*.ȞO3à4-}y4Pq6aUN?u: U:mA[y1?SSZ5צyy11dӐ: GbR@ϹIeؗKytMm5׫sy$v铊)eZ@돟t{9 Mq_zBVW!~~Wkr՞ \WLgёs,ZZhV44q=}F5XKe噂RO|X Ǝ<;] фe28Y) t<;?/RgU2JXYZfOEnղ}N֑M~FZxfٵJU{eG~}0=x|}oڛ޸'dc޼V?Fn?⵴ߌAo`7 苅>8kv;)Vc'1?1ZPI i `'C[HʟZn,l}7v=?|w&U\}n=$<8Z gIw=Lۊ޿^#`w y]&ך΁|]̋) Tg[(H:ǯz9U_;2}#9$aYJkI%h8m{➧-ay0K @8@tB:ŝy&E >s8o:mw/e{L6*ŷ zV/^$yB#g8PTf"G^?.1<^'ᥳ)JsgtmI$cqח:F2c8G/ٚ_toxo($TM ,Dz ^A5eXdu/NӚUxZNAimV6ڪ$.wr=` 5mьhy]Py-2 t7OGxa!WK2[>㱯> ܲ6gKb2bF^8ԌbI<x&mr@,R|`/&h[Z<%{P`<2+{`VZ؇_ _F١*uaQ+͛PA_M |ݳZrG3nk=$|P*bLeqVER>%axvRFp7oFDqZ)I4q>Ke;$C(~+kkP#h&hGL"|pf<-eeH;cft9V=ׁ?O_+$,.%DXd`kwՅksb/<ʻ!PB6g G7jτlQ4IFr298+ʨ~d6V} y26?u:Dai:7[㤼 U sҾax[+`w>YgN$}u|rnuO #`}rk\SݯƁ eO&ܘXK0xG]u  V[vuB!V;kZU0K.R<Ǭ[2":<WT \M{=O&8d _3|G~ۙERH-'?~fb}/XĜ6AҖvŇÛ^ >T'-&`q>I\n㏃ iE Ym'gdh$~O&l͐(_~<~=2?l0cZ|UJj?R6ztVrFG|aZAU{pz45_EYYyI#h\8<5^ӭ{kmʹt8t=0=y Fg a__^EᲭ-"H0]Ww +Oeect ^yckI%4b`Yq'_AA* v@.CpAT%U4S> y.$d< pWn`6?zd[.,c}$=O|Mk9?3WS}3-&~w 8?k23$tgHr]+J-im=$ez}wsL@'>kk`UqGKJNw>kX\#>V mZ`s#i6"9~B0yx>].;+\t$$>#Y@ Һ~ϩi<׳C$Ч+ПF s#-J ~׾ +k}'SiHP58,5^'$DHѶGK\7tU{ <_|6٬LBO q21*j<-Yo!h68 ]2Y;o.uXaw>:~t[j?iI`LGN k߉[ob}ߜ;`q*w#ZOd L| ַ:i%Uj$Cho<7P"2K^ٯpg0I2pyq9u[\,i2Bʋivwv}Ym|Gc}Ϙ(W|/]py; e&f'9#rM"̷b #Ą1Azgy৻C鯮dE ~#sW-:pOFs|\^ǨH+?+`{־Aqc[FfLpP^?5Kc2a>˰䈕{ӌ{W^J hOKTA3Epߕ]x@]ݞj! ^!s9q\OkU΀bdG?2;gd43Mvd\tWn"UlqBMc[x F%2ZI 9<{О0B:kʿg >.n[,`q瞞>0?Ȕ~5ʥyOח܋>dkZ#nM{=m<9`pOR3e {pײ- Y.?U2E(~^hFGg+DK$+v2pdt>ot.;ozK[>B\?޻Q&R~9_, tFv֯EA'Zr-nqs}EDb81 {k ԤԮZ)| `8E$Re.osid[Q$PD`y)ӎ>J+¶0<`|Evgm@T0 pw!~ZCmh偄$r$+i="Ez}k۬~;T"=-{߄c+ G^Y9!p0*c;đ#" ֌~ ]^:\A$~b&NwXDzM-hM:½~fܼk".$SO޿Bt|kw6-"%:Vs'p2xY[kk dl6F?z,0lv"Ke"k) btȯ;-y<i 0V]n`alc8~Xzs NzGr5 6ca%r06H {y2Uʜ{N_o|Bn.-|m-IDo V>мk:6vmK+pnɌc$8irͽaԌ!Wl{\B-3!VX-_Z͂S5amVb;L$'0ǫ{רh n,<;Ƴ7X]>^̿Roz_1\I? 0][#ƻ[} o`m=f#A[Qn>wfs5(EvL]_=;bpn@8 9xWy[xY,v p:t^B{9}dHSI3yOͭƏa:|!oq}3sҲTiyJgg@~D%XQz)vP.{'u+KQc0)#tZռ|`1No~ I_ߴ:?vlK$z5Yc;q=˕r^<v{ ݵΫvue@ b@zҭM6.,v?tqꇋAI*![M#@z`g1\Mi$fK)E4s4[Y"Indt+/Ěez Vώ& Uv݌ K\I +fD(Pg}9ݫOfF*Ԇ/M>h,n4YSdp;J[5̡#%}\Bnlxs#]TqXڎlib&8$0 8#$+nOGԿ|mAM]b9#9`׳GIk8nU"1yc'p_|QY Ɨ<ȱfR8l=rw5_o"ּmos,]%Òv'$*= 6UQzWՋPdVBIᕛ ՙ2V9.6Ο4o.K>qy$~>X4(UT/ J:IrbJo9K4msr.dp($y-αj(\~'</joLot1~ H;z^uyxfk]}~4@z-vg y\>5F5BnrKrp {ru8IZ:qqї-ַOb'!Uִ-xAYmfMqsr#cd(+Iᄌ}ZP̻,[z/t}BPn=מ}j^ԮLVi>wlqtLl'/1Hqs8w%k<=%MWe q(+48ȁ~{ڬ^01. U)Z1T4۩p!nIijd?k>kƊ2);2{s%@/skiψ~\ڄ4e;xX~pA긮rP٘ZߍE[hDCU?L`ߣ0ٞ}W#Σ3]"iwZ{H+m a*Ky?UhȒ5'9_=eB6Sȩ^2ؾ 5H!1&%bu$RpGLSuC\2d# xÎq#?Pl I!|9k+Mh.v$H8^Êxy'քF+F@ i+ReV7PŢWnRF= ~xy .bU!XUZgmUmRs2zSa\Og]oAE0yW!QN:gSЪhP(eTJ{e HqL֨ɩ1 d$3չY"5mVi$*ImFoΑw${~xV{{>~ <U dˑ~5h6IIJ)*yǭynM+d9 st7_ &-}lBsw-FـR67@=ڽ I??q }ߩ]>\ K4WhШv~k~mm;YΞ), )[?(sۥr,B'~iF_rl|bZks$Kv &M5Ԃ)mVG^M|)Rث,{{ioC>x3t޾Ź1Ϛ;draڒ:W4ϭ|!bi𽐗Ò_3$6>ozm[f?Ydkϕx##$9,R%΍4|Z vRgQ?C]5jb2D>P$ |oSI=U3LI}~/L]>QS*CbipnERq #7!d||-^nȡ@25oGRY C! /`3V#mjr~I !4.8짦: +_[HcFbb,r;\OK/XXdcᐝMr!?Zh˒oc$vz>G},^l@1^{W%I<p&A6d#8 u< OV y1t 0zqzqnugĐ2๋~\8bvV&V~Lw zAeiMCH4$FAF <_Зu{6N\|̸ c:oX'x\"ݼo ,(>`+ n8~+n־~6mu5BI` >;G@+O.4Fe}06vr#8;fp1PcmgMeҦkL|"z%cx<-eu-1XA8V?yv|-du[HNn9CSq 驠%s_"ƾ XG}rCD&pw|0FrV#&WCe)\|A##^Ww/*pЛp1@#aUr얽ь))ϳ ᔎ? ]L.yA(ss\O][ǺXhG̰dJ|n9]b4_h߄.$b>hrXВOMeQi6pq)a$+1q+ܜeXbi*/N#>-Ԏx-/+Bzf:Ƴyc:ǧC<`Ux ?18<?";7t5"xV8 6A#UP2dxx|&-?\D<ܕ] SRU[ .|֦ŖqjahLJ2F ^ⴌV-o3 ~.{(hPbv^Nk/"eB~R:[ypmPǓ{:W }vPy~bG=s3Z[+4;wchr>EC 3fªsH>lM/^4o-n-NDgn >eG\۱6m_{g> Xk"(>\NmFIů#> A!Go/'_qU`{DH!#$p s8 OT5M.#cy[Ng89qU:$_;ց#ҥfH88dw^F_z+\đ!BqXGS0VWJj/IXט Bݑjv44EtRSGe>&i"#@α.ONOQos6&UD@g##CFK_rsn\gɰ!U^HøNS,v=At'觟x)Z<. IW]|(g :ćIG^%l?u_/pBc#s<ƽ\] zR]Ov0VzKno5V.w0<֯.k-DՃMc''d:?8ϣ/c\M'K.!`s/qp!VŇW>!V__6|o nNp\=$2mB+_ji黉1 |#C?igɮK%4Хȍ=+˾1¶58@F`y)^}y,;-Oؐl2'xⰪuݓ8ihWz[ &uH  N{qPXsK#I&|GR\g}Hc@링ٜWM]-vO wߴ4CNUH+ncU8'IK(o!q\ U||ceŲ[ArpgMsલT<"t|4'9EZPg'?}?7KBN-?I<3ΙfqlҼ"ڟR= l0y{Ã> ldYu9i^L?,z|O'铌kf>}b8k"S eOb195g^sY57}M~Ƶew?M|n^_ O0$o<`mFrs4xguee_*+g 0knnZ9*M3T${sr`:n㧥qQl`;c{csރ ߈~1NdXSk;Fz=kdZ]Hmy 篭{:_k0//Ul4cxXČX8ݏMxC/\cMK˽$Ul99Qsmo/0Fz/O+^aj -͵B=+CC[P#M&cvczP %>d;7;pzW!JƲy.aƟ>3úFHeK7ntr.1 ic}/z,z)fAJ*y:u]88'R_3!v=\',ecؼ)}?j9$>%)V 򂗖 x\עFRO FUZx=el/|y/Ń)[Yae{tW8 Ff|0T"}BmgROɎ.dȊkTHޯ R֓G!Hz՛qS&, G_>O&>qoeP;W_TS 9 VkW9Gh.N7k|Z-u?ڕ\ nj1 m*331O$i1c!E8$\ey6 >]AHB rv&2TY(ҔO٘<D#ج<10: O '$f AXqN9>Ŀ|5hT)0pCI VLdαN9$ Sr]_+\u)m/gʼx6A{4_ h+س2]$)p*T` μW_׺V~ oB.m%9HS9Zڟ4HcJ7h'$"eL•`Jo$GzUGU> B Cfٳ(}BOԂ8n`ҧ$ࠑ92Bv#zo_b!Xr,BIyp[iu E;į_=K!ٰ'pq qIu%w񿆟|Kv\!v+9$6wmIkW *1Urp@8|VG~<tgʖ9cs d; q '9maq:ebg 2M3L. :"Rpq2HIѿhuXmo쁸p,܉8 {w⼺oF!u{G 9i?-t뫛Ur!V.G|Wƽ)-m4֫5hךXi%qݘ>@Pa3(mlt+s4yWX:E\$Uƥ0Q b1r\瞵̓yֆB1U%3>j!񳮝ocur}y46U\!^i{iwQ,MCc=*Q4&0S9OW-}fР YX1#*S{X TPrPzĿҭE7JQdDXA?&@35CKr [=式q\*)&z.w-> ?BMnmn#Wfr/C"l2kܿhi|pו̃Xn?WG1P uGs*8R|S^#U]ÆX#뚷;cMJdȊ:Ҷdh&Ö bb%ݵ :dEK>撻z^G9 Lbꚍ d,z[}4L!$~5ڄ"Cq17@7~K =@=2@}xn.d[&cxnd2gq;^GMN:;mVGp Os⋇ės,*Fryq_B~^^a;tRlD4H/ͷnCp) $G]7.ӌ~VEc-F[%YY c5/Өy_ 1ae'c;_>?(m>Ԅ,)P*݇yy9սz8Nퟜީ{\w0$Xg<:;E]rX##P<;:I}.7_gúfZ S2d8 @$8澇Gxk5BMO6b8% 9+֫Y%uV6}OoZ}3^ѵ 2=IWsXQ+bO@OLG;kU[Wc嗑1{dؒgoi6$M\U,/lU;lMs=aGx+|ǖĿφAuB46wi>Ih r 1DžYG?zx_%wc9#@/+o^ٶIt+]CKFIIcPP*|_vx'7IwGķ*JGyeEr>vbI잝(ΫwTQ{fVui0G1oֻ>e搇pp"6;''ׄjfY,\M(BsJ+u4k@9>Q,vҊ3NKfzT$ӭ'i5 l(*`p>N6о(n$\W!'ng֌ e2NA܁V8h@ty86St }K[<86sc5,pԬ%,0{?rڞst{uaqI㧕82G[lmR¿TyQ)vwWzm/H4`p~8&?ƽX\U׭Ӵx`',B8$IKP1 !bi8=vAW_ C:jhKe1~ӎۂG-Y5?|]KI/c]W c?.@r-e-bݥOs$k #d 3~G=+/ ѵNjK5ܬF|cU$kХS^Y ͅh825iFױq>v~[;%ׅM֋$X$ 2@rw@u vʌeEZy5hL\M`3+:.g=IX>\EX]̓J:':ug2(:bm{>?.{Z,4V "blfcp$c?K:6Ǩ\Hn + ˲5T'}ϰu#T:wmF@x9#+{Wקu]/:I9Ufn=>k׵>?-@$=џ$'=*Z_b< o\fM}ǚZ\ZLX/!}Gʁ \ͮ( T.sӧND9ボ洔u>]ţTp0d8a`ϨH''s8}jue{ZjxSº]fv#kOIvİ(XE.x+xKVf@Igں 16 v"?|[$o<u2y|/ |@?ZioO>MXI38\ vb&)bgʌTB1=GaԞ|F1 *GT[wkYhCWv9< ڙ^i\iQhWpiq"$7c$nP@ڠ-o( (+#H5Sp0F?8vk5g*<1n3׊ݝeY2a('NG_jLj}s.h;m"SU; Zş& !o3M wR˽ dǹ~?I+ub[X$vH".w1;lz7 Vy\iXSm?{;^]|4ZZo{D{rEG#  cA].%e14g kdVlK>`{|eZWو;,nϰx)MIy߄Y/#.'i\n9гUG p}++BN-;BGG#]4]Q)ר?FRs>j˚ "6whn?V1 DZw7q- .]FW:[q47PC0Fx]Sō)֪J|~ǡ+sJ4Ϛu?Q/AnJdF͑H'3:p*<5؈85aS_CQqU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMdu. .((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

Graceland is home to Elvis and Elvis merchandise. Come tour Graceland and get in touch with the King.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU index_htmlqUexpandqKu. +*((U OFS.ImageqUImageqtqNt.}q(UwidthqU95qU preconditionqUU content_typeqU image/gifq Utitleq U Young Elvisq U__name__q U elvis2.gifq UheightqU123qUdataqT<*GIF89a_{)FBp AҘ>& fյ<@ |mC~YfI԰࿊H <ՔLku*Qgc!e7=Ա44[`fDB'A'9ܙ?`s{z9|~D(UyhsPMVhUKQ>z7MpDSǵIBGfLnKgÔU=D<Ԃ~fRg3#m 8}=5EodfMɗ@#ӹzqVxs)+Mj̫ JBYwI2Y>VZ fɮ$ +{OeT(ns#!~|mqqHmSF6{;}sHj4D+w{fQ6ks/r>75+Dwx!U[M-ΙRRMϘ$PF V7OK՗΀=:l8r}jVFUոBwPRvvgv?K@_UA㡴!&-̜xDQwtrY=) Ks9x͛~yjaVW3.ϻPg{U|g4zV=ONx4ijEƌBRYTMLZf80(bUeDDz4c~ơTVY^YbfI0j(QrZZ1A1jQk)֌,_{GyX  !2Bⅇ@:BH0c=_O40:rʜ)0aZ!58aL.h1,'Zf) Z 2jՓ=^⑁T6ڴ!Ί)'`Ɲfd;B@+T52GiU*_ 4Z2e-cJurfzZM1ŬYٱK 2\@v) ${L5V=OzUI5o5|+1%S{@Z@r`vAP '~Aϟ#(2JH1R .(I}mUh*sF1  ,r FQYuI|H1^lD%TR? 2<, y0 5aL xp3 [lBH2őQ2+BA>n 7s J$3 E4YQJe `I%ܢI%Nh1NQM c43 Y8E +87=#81ײp _߰q:.E& .0 3,^zKR;6p 3؊ MP@J= F,& XЂ€@-d3]#`G@9L.(I+F@6pC&2E= #HR,d 5BIXS H A" &)` Yǰ6F r@{ J߂FTeU 1L"^L)<&> A !I|$|>D"z`7xJ^Yx籵h gPC I#sl@(S~TAECwTTP4@"S,jH'H!Wx$ I5`a4f G t6xo@YTxj)ZH%)h`UJUH*Ph7lL ,X@+x\~)&hKTQ~5.+@$l|Cd4J6$ ?p%TbB ':)>WfiLKaj/8pp<+a!%d iޑ Q%&HlF!&79 q @~$(K@9V?^9ja|E!P6hv;ہh6=! EaSQy@ #>W0*@[̣j1=XeBІ  QohR"Ԣ|&X Rn~&0! G\V:|@ G85 ht9bi* 0@cP)]2Y88@ e!SJw6 gȅ}R7dThDEp+c ]J[LggHhԠ o`B/ #bA+Kgh<ִcN! L G[(~ X6 q16#{L~ɍ) $a+C @ $x<9H< F Q$~%k .8P=G52 ~B *@)> P! g d B0 ^4 Wz g{[/@ /sĀti p8 "!?0$)` ^bJX4]80}e/0 r"HP#6|Dg0hr1 WsI >UU X ` K c F0F@E > `(*ԑzP/r@@ z@k7`P4DS m 2@55p B1=7eP; #zB Sg>0OBgis‘p;p@mi30fPp %<|xȣ<_;0S3Ai 3C `GNk/Kzݓ 9h Π@ M0=P@0w xP \` {@ /T5l` ~ 8}Wб(0` gG66a ~?IH $N04! {  (]]Ē>c~ @G3pvQ0L&g~~ Ă$L0 Va  6БcJD@ FE"K0nEV M'zr\b9. q#M6zd p@ C%*QM !|D6@1+D=08tJ YeK=r ;k23#8 QfO9DfD(0Ed% K)p pF3y3yr$3>leۉ2NM V p{j pp JW$1(p C`T =I x?9EH[ҾE+M,I#OpU I H  `j`kM@j `DCDp`p'yE"yx4z5 ׀Dj ڱ0 Hd `OVfJ` ,ϱOc:3e AxF!C#O#:DC4xrij+X lh('>8(D.!-JѴq:O IrAb%9ha * Š!vHa#"x03ӑDK :ʪAih :%C88jq$vт<ȨKn:#2N3%k# q e[ F Na>E A>dӎlGi92ȴT⬬3(ZDj`Dbƥ]*9x$@8PR A˺Vd6y9y)fԁY$!NgpnH_Ia@#(U)KxII"9IhF=`FB4@;# )&[ =JHEC(Ct2bȅ $`k&!@Q ,# L@P<0up}fb)]I˺{ ;PYc>p*@ ^d4^7u< c1-}a7{٥uMs !Y9SJ^>ы<@UVmd`EDFa ؂<0;΅!:U3Bq%yZ'$hEZ%`=s63 W! } |-^W|@h)j&d@)XA0D<Z0C(qmi"0$/zu&`ڲ FgP)7BEG\+d޸) @Cdf "wI׭͉ "M1xxt p$PC#(7=#JA 0%TЄ \fg:L͡gHGJC[8y"؇Tr0E VTK`s(}6HmRx2%@' `cG-r hlZhK' -&A)"%tK #\f`3ӁMdiDGH1:0 sz!?;V TIap1j;</p>2l &d3MA=AQ_:xom(kecoIx'81gt1Q WbcG:@ Fmˋa` M:A}BZp\r5ILU^[BNZ bp%r"lpFU`R P׵wR[< 7k3hďD,,`B<*tCE2#4vuh‹Kȓ3\P `>2T&ڙp#XC^14xX Z {F^*XP2FPy*fc-K}b>$1pBXC$A,[ٚ+y# 08e` Λ$1 7mRP@ g,۠6R d8\S=Z*Alkhb9^67 p?PJ"@ HBs윑Zdr @֎bQMTdjA%ɻa$Ҡ)d?85;?y#>'@:yP;W,0(yC3,-ցk3+c1U쑵 8;?8U/(8b[ cJ;JJ``53p걠?b23r"3V-#" J`$35#mЬӄ#ߣA)<JSF`Z>a# 7*rיz!5pVzI yɀ S6K۰ypH vʁ`j?\|1!$(BpBC+z H ,ȁUb@:Ӱek66HMM8cdd৽jdF&;$NPK&Mp0xK(@:ICE'|JʡI&C7C|f<; Gpa bB`ɑ[2jT@94Kf$*x0Ej@8pS19p6!w1;bL,8聚MRPSXPcH=XZ#^cXvH/qt,,P9ІVPcz ѹB49y/4Ka AMxc 0p9Sf`s0X`.aSAHRS|!, ,DIC0W@G?%@a@p Sp`% ؃nP P{N4p׺'0b4 32QP R=0R  $I.0.p>d5XR <R@p XG`RC1N'f|-)rWL- W}GX RXSP p OP Po-Pl|11`;BuBO{k!U~`~P2 RLx X 0FK Nh5OЊ6H]OZ<;xiJ?$8 RM#RЅ( ; VP z !JDpoIS`GH(hh8)r!A& ^x7P@Nnx<vVR؀7 X iŀY@ A(QL!cb,@ "zN50b  @ 0I5 7PG 8&5hG* }(Bا`~\B35(L20-o I VD8\Ȧ Gx5آSpf !P#NSX) 8(Pr śxy x! Xg Oxɵ]NPkOZ(Re}g) >ļvQA gR8^f7G`PNhS@Sxv7r^8wx  ;u@]+ݗ "HWk]KXoRRP O؆R@@8%PoEvPRx ΀܇Xi Ŕ]A ͑,5*h ݅pPE! MACGCon 1Written by GIFConverter 2.3.7 of Jan 29, 1994;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM<*u. +7((U OFS.ImageqUImageqtqNt.}q(UwidthqU86qU preconditionqUU content_typeqU image/gifq Utitleq UKing with collarq U__name__q U elvis1.gifq UheightqU133qUdataqTD*GIF87aV  !!!"""###$$$%%%&&&'''((()))***+++,,,---...///000111222333444555666777888999:::;;;<<<===>>>???@@@AAABBBCCCDDDEEEFFFGGGHHHIIIJJJKKKLLLMMMNNNOOOPPPQQQRRRSSSTTTUUUVVVWWWXXXYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffggghhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuvvvwwwxxxyyyzzz{{{|||}}}~~~,V H@菟?뷐aEK(3}rb!@j"!#(`:VV2f‰9⿋: 4Hӳ| miAJbܳ`V" ,ZI |H!RouH5 N C5ft ,Wi (A|ûhFa2`17%bXu..YpCVY*Ohlp߾Y'\[t@{D6Q06%*+K|p*lZ-[^2CG(BopC ?|YRRJ*s@б<4aJ1BL/JR< =#%E%`@x8SFR+h[5pL(Km,C+ ,9N6l1@Ax08xbcб <* s3"o# 1(3/ 5'R 0?:1XA @@)0\Ё$@TlA ц)4PwL3lX+T.֗ѳJ)4@@@@HPppo`P4KBP4 rM0$3M7L31K.Rp!`@:@0 @@,@Po@D$RN6 C 9XN5K,Q?eCO+2X0 &@8@' pܼÏ9h,5Lc6֜s1آL1>x8Tp@|/ @R 0 `xs3 8 C +^OO0| r v F ; 3LjHS`Z1oyCͮǼ1GVh80! ^B)T Ua7q@78B "VA3`` 0 D`z04J XD5AP PؒH x< \†<[XMpb `N480ΑFjNf1/T5VzyʅTbX92gCuF , OTĨ],x _8B%V>1h\`Rvnx/эX}.| `v 5#lgM(:TH$+񮝥e0 {H8q uЅ*da !@yGrJm`Ke*Јx]  yHG9zcP7a ZB?16<5!jm'lf4HQ9@J1xCԘE0! \dr`@ؘ'HEmrhiư.@"tl}+4Z<1abT$JqC -ɻuV?4+9ՙ"/Ӏ#d@ea}v;ZP@W` PRrŦ`VAIpZ! E.ꠃHc@õo 8A C 4܅^$ 5(KyjU oú$&E-zo|0C@)F1bA#d@믔^#Z6pbEt'a4Σ_taVSt ca03CS` t +SQ;xGS0`0h;gf\1 P (3 a - KAu0D` 0@0 l2[Znh w  f Ѐp1 ΰi6 Up/ C08 0 p qp@#PfNS!:;03` 2x2Na0 ͐  Yw co`q  J%ְ w`M !0D^D0#I`@~E|: T@rW6P@p 0`D ̘o1c69-QIxFN àRT!P =K 0r`H@'<#/jIRtNhv[Y` "Ő(m&  @H(p P ~ gK&` % ` #DJV[@ ` !d]|P %(@P % y[1"0<i  `X@0wdƎ` `11q@hw/P@Ip `0 @ H/P9@ p .0's WJj ݐ  bl rɚ0֐ P ! j*`2 ؀ [&`c  `̰dDe` j B~HS eS S0Y   [Pg3 j 1YN`P !T Ѡ Ā P `a@#bVO ~ c+ݨhR$ LRs+9~[0  PtY` F 0 jPtSCp  hJ P9#m*0#bAwUk c  0 0P  "D P:`VO3 +6˲EPƠ ܐ qpč2@S`qA[`]Ej L0*0-`cp &  j 7('A_{  /NvJ$]r |0E#P+0T#Y *P `0 }60>#8 9\; pp Y9#E @ ESssvH3=5L @X@ e`'"pv0 P tN0w y gVjTX|s PPpX  j`g@I06` Ð P%؁4vk` 0#pIag'0f%Y| @ WAqKz 0)\+rthP \M1PVD*Kӌ`Ӡ .keNPP аPW pu*p0E`6`X@8@07 ʀ 0ME@~uq3` m1 _uD" I #   @h  d p WgpũU,߰<1H0 @03/0 ` >AȢP@Жq$z .`jo6E  4$us ܐ @@9m ʰ ۀ P Ppqᚔ@ 'o"8pps KF]Cp {po@ 怏( c 7M`َݚɢ`7#]'0#"0Ux ZY #p p P Т݀a'z "Z j0*TuZaisyNd0ɰpJp& j r½0(dZ MVo9j`} TJ' /Y#Gq-i՚ KQ ؀b%L q 4]`@ְ 0x*B0 q& y@a $K0Gt/ P `@:@5|ؠMB0jl딪~հF&|q '@0 ̰  p@&:Z7o9À|70eN` Ow007q p (P 49ꐖP0g&pw+@ GFPPb@ pbW/ q!@  ԐYm1_Ww k*`0lNŜ l ՚ @@:ː >ZΖ{^0asCDT`!# s[@ ТP s@ rP Ġ Ѐ8P (PW<7! ^ }pPrj P ڠ {i0v`w oO€ @ pW(y}t4N0Ul*` NT` Φ]V͚0QQD)z:{mU9wBCN]sw. `" D xGN?|}[Kl<*֌fȈ5ڸjtpaݳ;lxa{/9sԱeA*fPp@ 4DuΩcfS 0]| 5kU,e@zCu'77;x rϊhsZ F'H .˵6^ȀM,ꖭVmKGsV|)}u!kFw!'s!+pqƌp@@ FG?r|c݇7\ш.@ -}Q so>B-c8a`ܡMg HNP}_ X( KAJО;vz8-3^ /L *7g0 T5'-qu /$8 Aw$: bPJAas$G(;Q ,T@ XCBdmC  p:lAYT: XL Q D,\Fd1 SuָFTqD@A&E6ΒM|+[= |! }84R;Aw &VYx.I@ a0TJ^s!!0GZz8G3H!=D%x xD 2"@>+,ƣ4R:ٶ~!u Dfu$>pBT>L `U(PȇFꖄCqX(SdRI[ ^`J088*1{0Rĭr(:WuC8L=*Ct`70I YdO@;ĩ%X)h9.9`l}P3n@@@껲Y#|@[gh,H:iYC6:-.,B3,vHg$B@A` 8(ګHlYyhЄD^PpfUE@E ;X9=x8u3\aY(؄&;,>[KRx.``]`mkP``MJE8L?T@0-tR:QL! H7 am(J/Uuk!*gaS@QRX H`I|Ѓ k5t?6: HO9IlJ/[LHgqxmc`ȅXȅa[ T"$ oH.۪.!:@*`/Z#4'PPAP؃iȆlie0fxZUx /pa !2"#@'zp! m [frPr0n$ok]Bs-X@0C,@x +2+@8:p@4U1i^hoXv t81+g`spMW?('Q?85 5? 41,Vp080vXTsm<˾l(0nS"x@Hoj()=a %4,BQXhy@+;֚$sw2hPTxu ѐA~:  aawKxpw荴ʬvh 1rJ* X58y@ppxUxteSK Ȇt:xJի7߈"rI05"0 Q9PB(Ѐ V&xj=S pQI**ks)zЈ7xHw@)'kP:ㄑe~  @88M "+mC0Ȝɭ^fhQ&̔s%E O(4(Rl؂ H8u5H:֧@Ք-Pl%{< ohH{p{bvP5B'P ZHdh"e-P} :{P- ^`Vj0jsy`i(mIڜ0*KxF&m+ԥpnӬnnX2^F`ohbqH.3ixoIJ'P .x8%6^kP܁cp@m6*yQ`8g vaxQB xi"K] Q5M~6hmfVP81]bt!`Y wFȄiQtkfӬX0WdlЫ#i |P_(tPŜDEXZӄ:͸S0͘3/21/2000. San Francisco, California.

I was in IHOP. A very large man, who can really shake ordered a large stack of apple pancakes. He seemed vaguely familiar so I as a fellow customer, asked him his name. He said, "The King." and I said, "Who?" "You know Elvis." was his reply. After conversing for about an hour I knew this was no imposter. I was no longer skeptical, just in disbelief that I could be so fortunate. Wow! I just wanted to share with the rest of the world.

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMu. ((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqU Shelbyvilleq U__name__q U sighting2q U_EtagSupport__etagq U ts69463800.18q UdataqU

2/23/2000. Shelbyville, Illinois.

He was in a gas station. He came in and paid for his gas driving a dark brown caddy with gold rims he looked right at me and said, "You didn't see nothing OK man."

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqKu. >((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqUMasonq U__name__q U sighting1q U_EtagSupport__etagq U ts69463829.96q UdataqTQ

3/4/2000. Mason, West Virginia.

My furnace broke down and I called to have it fixed and when the guy came over to fix it he looked just like Elvis but I thought to myself that this was to good to be true, but when I gave him the money then I knew it was him because he looked at me and said "Thank ya thank ya very much!"

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMQu. >((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqUMasonq U__name__q U sighting1q U_EtagSupport__etagq U ts69463829.96q UdataqTQ

3/4/2000. Mason, West Virginia.

My furnace broke down and I called to have it fixed and when the guy came over to fix it he looked just like Elvis but I thought to myself that this was to good to be true, but when I gave him the money then I knew it was him because he looked at me and said "Thank ya thank ya very much!"

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMQu. ((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqU San Franciscoq U__name__q U sighting3q U_EtagSupport__etagq U ts69463787.62q UdataqT

3/21/2000. San Francisco, California.

I was in IHOP. A very large man, who can really shake ordered a large stack of apple pancakes. He seemed vaguely familiar so I as a fellow customer, asked him his name. He said, "The King." and I said, "Who?" "You know Elvis." was his reply. After conversing for about an hour I knew this was no imposter. I was no longer skeptical, just in disbelief that I could be so fortunate. Wow! I just wanted to share with the rest of the world.

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMu. ((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqU Shelbyvilleq U__name__q U sighting2q U_EtagSupport__etagq U ts69463800.18q UdataqU

2/23/2000. Shelbyville, Illinois.

He was in a gas station. He came in and paid for his gas driving a dark brown caddy with gold rims he looked right at me and said, "You didn't see nothing OK man."

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqKu. 'B((U PersistenceqUPersistentMappingqtqNt.}qU _containerq}qs. +((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqU Shelbyvilleq U__name__q U sighting2q U_EtagSupport__etagq U ts69463800.18q UdataqU

2/23/2000. Shelbyville, Illinois.

He was in a gas station. He came in and paid for his gas driving a dark brown caddy with gold rims he looked right at me and said, "You didn't see nothing OK man."

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqKu. ,((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqU San Franciscoq U__name__q U sighting3q U_EtagSupport__etagq U ts69463787.62q UdataqT

3/21/2000. San Francisco, California.

I was in IHOP. A very large man, who can really shake ordered a large stack of apple pancakes. He seemed vaguely familiar so I as a fellow customer, asked him his name. He said, "The King." and I said, "Who?" "You know Elvis." was his reply. After conversing for about an hour I knew this was no imposter. I was no longer skeptical, just in disbelief that I could be so fortunate. Wow! I just wanted to share with the rest of the world.

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMu. *>((U OFS.ImageqUFileqtqNt.}q(U preconditionqUU content_typeqU text/htmlqUtitleqUMasonq U__name__q U sighting1q U_EtagSupport__etagq U ts69463829.96q UdataqTQ

3/4/2000. Mason, West Virginia.

My furnace broke down and I called to have it fixed and when the guy came over to fix it he looked just like Elvis but I thought to myself that this was to good to be true, but when I gave him the money then I knew it was him because he looked at me and said "Thank ya thank ya very much!"

qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMQu. +*((U OFS.ImageqUImageqtqNt.}q(UwidthqU95qU preconditionqUU content_typeqU image/gifq Utitleq U Young Elvisq U__name__q U elvis2.gifq UheightqU123qUdataqT<*GIF89a_{)FBp AҘ>& fյ<@ |mC~YfI԰࿊H <ՔLku*Qgc!e7=Ա44[`fDB'A'9ܙ?`s{z9|~D(UyhsPMVhUKQ>z7MpDSǵIBGfLnKgÔU=D<Ԃ~fRg3#m 8}=5EodfMɗ@#ӹzqVxs)+Mj̫ JBYwI2Y>VZ fɮ$ +{OeT(ns#!~|mqqHmSF6{;}sHj4D+w{fQ6ks/r>75+Dwx!U[M-ΙRRMϘ$PF V7OK՗΀=:l8r}jVFUոBwPRvvgv?K@_UA㡴!&-̜xDQwtrY=) Ks9x͛~yjaVW3.ϻPg{U|g4zV=ONx4ijEƌBRYTMLZf80(bUeDDz4c~ơTVY^YbfI0j(QrZZ1A1jQk)֌,_{GyX  !2Bⅇ@:BH0c=_O40:rʜ)0aZ!58aL.h1,'Zf) Z 2jՓ=^⑁T6ڴ!Ί)'`Ɲfd;B@+T52GiU*_ 4Z2e-cJurfzZM1ŬYٱK 2\@v) ${L5V=OzUI5o5|+1%S{@Z@r`vAP '~Aϟ#(2JH1R .(I}mUh*sF1  ,r FQYuI|H1^lD%TR? 2<, y0 5aL xp3 [lBH2őQ2+BA>n 7s J$3 E4YQJe `I%ܢI%Nh1NQM c43 Y8E +87=#81ײp _߰q:.E& .0 3,^zKR;6p 3؊ MP@J= F,& XЂ€@-d3]#`G@9L.(I+F@6pC&2E= #HR,d 5BIXS H A" &)` Yǰ6F r@{ J߂FTeU 1L"^L)<&> A !I|$|>D"z`7xJ^Yx籵h gPC I#sl@(S~TAECwTTP4@"S,jH'H!Wx$ I5`a4f G t6xo@YTxj)ZH%)h`UJUH*Ph7lL ,X@+x\~)&hKTQ~5.+@$l|Cd4J6$ ?p%TbB ':)>WfiLKaj/8pp<+a!%d iޑ Q%&HlF!&79 q @~$(K@9V?^9ja|E!P6hv;ہh6=! EaSQy@ #>W0*@[̣j1=XeBІ  QohR"Ԣ|&X Rn~&0! G\V:|@ G85 ht9bi* 0@cP)]2Y88@ e!SJw6 gȅ}R7dThDEp+c ]J[LggHhԠ o`B/ #bA+Kgh<ִcN! L G[(~ X6 q16#{L~ɍ) $a+C @ $x<9H< F Q$~%k .8P=G52 ~B *@)> P! g d B0 ^4 Wz g{[/@ /sĀti p8 "!?0$)` ^bJX4]80}e/0 r"HP#6|Dg0hr1 WsI >UU X ` K c F0F@E > `(*ԑzP/r@@ z@k7`P4DS m 2@55p B1=7eP; #zB Sg>0OBgis‘p;p@mi30fPp %<|xȣ<_;0S3Ai 3C `GNk/Kzݓ 9h Π@ M0=P@0w xP \` {@ /T5l` ~ 8}Wб(0` gG66a ~?IH $N04! {  (]]Ē>c~ @G3pvQ0L&g~~ Ă$L0 Va  6БcJD@ FE"K0nEV M'zr\b9. q#M6zd p@ C%*QM !|D6@1+D=08tJ YeK=r ;k23#8 QfO9DfD(0Ed% K)p pF3y3yr$3>leۉ2NM V p{j pp JW$1(p C`T =I x?9EH[ҾE+M,I#OpU I H  `j`kM@j `DCDp`p'yE"yx4z5 ׀Dj ڱ0 Hd `OVfJ` ,ϱOc:3e AxF!C#O#:DC4xrij+X lh('>8(D.!-JѴq:O IrAb%9ha * Š!vHa#"x03ӑDK :ʪAih :%C88jq$vт<ȨKn:#2N3%k# q e[ F Na>E A>dӎlGi92ȴT⬬3(ZDj`Dbƥ]*9x$@8PR A˺Vd6y9y)fԁY$!NgpnH_Ia@#(U)KxII"9IhF=`FB4@;# )&[ =JHEC(Ct2bȅ $`k&!@Q ,# L@P<0up}fb)]I˺{ ;PYc>p*@ ^d4^7u< c1-}a7{٥uMs !Y9SJ^>ы<@UVmd`EDFa ؂<0;΅!:U3Bq%yZ'$hEZ%`=s63 W! } |-^W|@h)j&d@)XA0D<Z0C(qmi"0$/zu&`ڲ FgP)7BEG\+d޸) @Cdf "wI׭͉ "M1xxt p$PC#(7=#JA 0%TЄ \fg:L͡gHGJC[8y"؇Tr0E VTK`s(}6HmRx2%@' `cG-r hlZhK' -&A)"%tK #\f`3ӁMdiDGH1:0 sz!?;V TIap1j;</p>2l &d3MA=AQ_:xom(kecoIx'81gt1Q WbcG:@ Fmˋa` M:A}BZp\r5ILU^[BNZ bp%r"lpFU`R P׵wR[< 7k3hďD,,`B<*tCE2#4vuh‹Kȓ3\P `>2T&ڙp#XC^14xX Z {F^*XP2FPy*fc-K}b>$1pBXC$A,[ٚ+y# 08e` Λ$1 7mRP@ g,۠6R d8\S=Z*Alkhb9^67 p?PJ"@ HBs윑Zdr @֎bQMTdjA%ɻa$Ҡ)d?85;?y#>'@:yP;W,0(yC3,-ցk3+c1U쑵 8;?8U/(8b[ cJ;JJ``53p걠?b23r"3V-#" J`$35#mЬӄ#ߣA)<JSF`Z>a# 7*rיz!5pVzI yɀ S6K۰ypH vʁ`j?\|1!$(BpBC+z H ,ȁUb@:Ӱek66HMM8cdd৽jdF&;$NPK&Mp0xK(@:ICE'|JʡI&C7C|f<; Gpa bB`ɑ[2jT@94Kf$*x0Ej@8pS19p6!w1;bL,8聚MRPSXPcH=XZ#^cXvH/qt,,P9ІVPcz ѹB49y/4Ka AMxc 0p9Sf`s0X`.aSAHRS|!, ,DIC0W@G?%@a@p Sp`% ؃nP P{N4p׺'0b4 32QP R=0R  $I.0.p>d5XR <R@p XG`RC1N'f|-)rWL- W}GX RXSP p OP Po-Pl|11`;BuBO{k!U~`~P2 RLx X 0FK Nh5OЊ6H]OZ<;xiJ?$8 RM#RЅ( ; VP z !JDpoIS`GH(hh8)r!A& ^x7P@Nnx<vVR؀7 X iŀY@ A(QL!cb,@ "zN50b  @ 0I5 7PG 8&5hG* }(Bا`~\B35(L20-o I VD8\Ȧ Gx5آSpf !P#NSX) 8(Pr śxy x! Xg Oxɵ]NPkOZ(Re}g) >ļvQA gR8^f7G`PNhS@Sxv7r^8wx  ;u@]+ݗ "HWk]KXoRRP O؆R@@8%PoEvPRx ΀܇Xi Ŕ]A ͑,5*h ݅pPE! MACGCon 1Written by GIFConverter 2.3.7 of Jan 29, 1994;qU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqM<*u. +7((U OFS.ImageqUImageqtqNt.}q(UwidthqU86qU preconditionqUU content_typeqU image/gifq Utitleq UKing with collarq U__name__q U elvis1.gifq UheightqU133qUdataqTD*GIF87aV  !!!"""###$$$%%%&&&'''((()))***+++,,,---...///000111222333444555666777888999:::;;;<<<===>>>???@@@AAABBBCCCDDDEEEFFFGGGHHHIIIJJJKKKLLLMMMNNNOOOPPPQQQRRRSSSTTTUUUVVVWWWXXXYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffggghhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuvvvwwwxxxyyyzzz{{{|||}}}~~~,V H@菟?뷐aEK(3}rb!@j"!#(`:VV2f‰9⿋: 4Hӳ| miAJbܳ`V" ,ZI |H!RouH5 N C5ft ,Wi (A|ûhFa2`17%bXu..YpCVY*Ohlp߾Y'\[t@{D6Q06%*+K|p*lZ-[^2CG(BopC ?|YRRJ*s@б<4aJ1BL/JR< =#%E%`@x8SFR+h[5pL(Km,C+ ,9N6l1@Ax08xbcб <* s3"o# 1(3/ 5'R 0?:1XA @@)0\Ё$@TlA ц)4PwL3lX+T.֗ѳJ)4@@@@HPppo`P4KBP4 rM0$3M7L31K.Rp!`@:@0 @@,@Po@D$RN6 C 9XN5K,Q?eCO+2X0 &@8@' pܼÏ9h,5Lc6֜s1آL1>x8Tp@|/ @R 0 `xs3 8 C +^OO0| r v F ; 3LjHS`Z1oyCͮǼ1GVh80! ^B)T Ua7q@78B "VA3`` 0 D`z04J XD5AP PؒH x< \†<[XMpb `N480ΑFjNf1/T5VzyʅTbX92gCuF , OTĨ],x _8B%V>1h\`Rvnx/эX}.| `v 5#lgM(:TH$+񮝥e0 {H8q uЅ*da !@yGrJm`Ke*Јx]  yHG9zcP7a ZB?16<5!jm'lf4HQ9@J1xCԘE0! \dr`@ؘ'HEmrhiư.@"tl}+4Z<1abT$JqC -ɻuV?4+9ՙ"/Ӏ#d@ea}v;ZP@W` PRrŦ`VAIpZ! E.ꠃHc@õo 8A C 4܅^$ 5(KyjU oú$&E-zo|0C@)F1bA#d@믔^#Z6pbEt'a4Σ_taVSt ca03CS` t +SQ;xGS0`0h;gf\1 P (3 a - KAu0D` 0@0 l2[Znh w  f Ѐp1 ΰi6 Up/ C08 0 p qp@#PfNS!:;03` 2x2Na0 ͐  Yw co`q  J%ְ w`M !0D^D0#I`@~E|: T@rW6P@p 0`D ̘o1c69-QIxFN àRT!P =K 0r`H@'<#/jIRtNhv[Y` "Ő(m&  @H(p P ~ gK&` % ` #DJV[@ ` !d]|P %(@P % y[1"0<i  `X@0wdƎ` `11q@hw/P@Ip `0 @ H/P9@ p .0's WJj ݐ  bl rɚ0֐ P ! j*`2 ؀ [&`c  `̰dDe` j B~HS eS S0Y   [Pg3 j 1YN`P !T Ѡ Ā P `a@#bVO ~ c+ݨhR$ LRs+9~[0  PtY` F 0 jPtSCp  hJ P9#m*0#bAwUk c  0 0P  "D P:`VO3 +6˲EPƠ ܐ qpč2@S`qA[`]Ej L0*0-`cp &  j 7('A_{  /NvJ$]r |0E#P+0T#Y *P `0 }60>#8 9\; pp Y9#E @ ESssvH3=5L @X@ e`'"pv0 P tN0w y gVjTX|s PPpX  j`g@I06` Ð P%؁4vk` 0#pIag'0f%Y| @ WAqKz 0)\+rthP \M1PVD*Kӌ`Ӡ .keNPP аPW pu*p0E`6`X@8@07 ʀ 0ME@~uq3` m1 _uD" I #   @h  d p WgpũU,߰<1H0 @03/0 ` >AȢP@Жq$z .`jo6E  4$us ܐ @@9m ʰ ۀ P Ppqᚔ@ 'o"8pps KF]Cp {po@ 怏( c 7M`َݚɢ`7#]'0#"0Ux ZY #p p P Т݀a'z "Z j0*TuZaisyNd0ɰpJp& j r½0(dZ MVo9j`} TJ' /Y#Gq-i՚ KQ ؀b%L q 4]`@ְ 0x*B0 q& y@a $K0Gt/ P `@:@5|ؠMB0jl딪~հF&|q '@0 ̰  p@&:Z7o9À|70eN` Ow007q p (P 49ꐖP0g&pw+@ GFPPb@ pbW/ q!@  ԐYm1_Ww k*`0lNŜ l ՚ @@:ː >ZΖ{^0asCDT`!# s[@ ТP s@ rP Ġ Ѐ8P (PW<7! ^ }pPrj P ڠ {i0v`w oO€ @ pW(y}t4N0Ul*` NT` Φ]V͚0QQD)z:{mU9wBCN]sw. `" D xGN?|}[Kl<*֌fȈ5ڸjtpaݳ;lxa{/9sԱeA*fPp@ 4DuΩcfS 0]| 5kU,e@zCu'77;x rϊhsZ F'H .˵6^ȀM,ꖭVmKGsV|)}u!kFw!'s!+pqƌp@@ FG?r|c݇7\ш.@ -}Q so>B-c8a`ܡMg HNP}_ X( KAJО;vz8-3^ /L *7g0 T5'-qu /$8 Aw$: bPJAas$G(;Q ,T@ XCBdmC  p:lAYT: XL Q D,\Fd1 SuָFTqD@A&E6ΒM|+[= |! }84R;Aw &VYx.I@ a0TJ^s!!0GZz8G3H!=D%x xD 2"@>+,ƣ4R:ٶ~!u Dfu$>pBT>L `U(PȇFꖄCqX(SdRI[ ^`J088*1{0Rĭr(:WuC8L=*Ct`70I YdO@;ĩ%X)h9.9`l}P3n@@@껲Y#|@[gh,H:iYC6:-.,B3,vHg$B@A` 8(ګHlYyhЄD^PpfUE@E ;X9=x8u3\aY(؄&;,>[KRx.``]`mkP``MJE8L?T@0-tR:QL! H7 am(J/Uuk!*gaS@QRX H`I|Ѓ k5t?6: HO9IlJ/[LHgqxmc`ȅXȅa[ T"$ oH.۪.!:@*`/Z#4'PPAP؃iȆlie0fxZUx /pa !2"#@'zp! m [frPr0n$ok]Bs-X@0C,@x +2+@8:p@4U1i^hoXv t81+g`spMW?('Q?85 5? 41,Vp080vXTsm<˾l(0nS"x@Hoj()=a %4,BQXhy@+;֚$sw2hPTxu ѐA~:  aawKxpw荴ʬvh 1rJ* X58y@ppxUxteSK Ȇt:xJի7߈"rI05"0 Q9PB(Ѐ V&xj=S pQI**ks)zЈ7xHw@)'kP:ㄑe~  @88M "+mC0Ȝɭ^fhQ&̔s%E O(4(Rl؂ H8u5H:֧@Ք-Pl%{< ohH{p{bvP5B'P ZHdh"e-P} :{P- ^`Vj0jsy`i(mIڜ0*KxF&m+ԥpnӬnnX2^F`ohbqH.3ixoIJ'P .x8%6^kP܁cp@m6*yQ`8g vaxQB xi"K] Q5M~6hmfVP81]bt!`Y wFȄiQtkfӬX0WdlЫ#i |P_(tPŜDEXZӄ:͸S0͘, , , )qUtemplateqcShared.DC.ZRDB.DA SQL qNRq}q(U_varsq}qUrawqhUglobalsq }q!U__name__q"Uq#ubU_colq$)Utitleq%UInsert Elvis Sightingq&U__ac_local_roles__q'}q(Uamosq)]q*UOwnerq+asUidq,Uinsertq-U connection_idq.U connectionq/u. n((UOFS.DTMLMethodqU DTMLMethodqtqNt.}q(U_varsq}qUrawqT qUglobalsq }q Utitleq UTutorial Database Setupq U__name__q UsetupqU__ac_local_roles__q}qUamosq]qUOwnerqasu. ((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(UsrcqUdrop table elvis_sightingsqU arguments_srcqUUtemplateqcShared.DC.ZRDB.DA SQL q NRq }q (U_varsq }q U__name__qUqUglobalsq}qUrawqhubU_colq)U_argq(cShared.DC.ZRDB.Aqueduct Args qo}q(U_keysq]qU_dataq}qubUtitleqUDrop Tutorial DatabaseqU__ac_local_roles__q}qUamosq]q UOwnerq!asUidq"Udropq#U connection_idq$U connectionq%u. 2((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(UsrcqUCselect * from __table_names__ where table_name = 'ELVIS_SIGHTINGS' qU arguments_srcqUUtemplateqcShared.DC.ZRDB.DA SQL q NRq }q (U_varsq }q UrawqhUglobalsq}qU__name__qUqubUtitleqUTutorial Database ExistsqU_argq(cShared.DC.ZRDB.Aqueduct Args qoq}q(U_keysq]qU_dataq}qubU_colq]q(}q(Uwidthq NUnullq!NUtypeq"UsUnameq#UIS_VIEWq$u}q%(h Nh!Nh"Ush#U TABLE_NAMEq&ueU__ac_local_roles__q'}q(Uamosq)]q*UOwnerq+asUidq,U table_existsq-U connection_idq.U connectionq/u. ((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(U_argq(cShared.DC.ZRDB.Aqueduct Args qoq}q(U_keysq ]q U_dataq }q ubU arguments_srcq UUsrcqUlcreate table elvis_sightings( location varchar, date varchar, name varchar, description varchar ) qUtemplateqcShared.DC.ZRDB.DA SQL qNRq}q(U_varsq}qUrawqhUglobalsq}qU__name__qUqubU_colq)UtitleqUCreate Tutorial DatabaseqU__ac_local_roles__q}qUamosq ]q!UOwnerq"asUidq#Ucreateq$U connection_idq%U connectionq&u. ((UOFS.DTMLMethodqU DTMLMethodqtqNt.}q(U_varsq}qUrawqTqUglobalsq }q Utitleq UTutorial Database Sample Dataq U__name__q U sample_dataqU__ac_local_roles__q}qUamosq]qUOwnerqasu. ((UProducts.ZSQLMethods.SQLqUSQLqtqNt.}q(UsrcqUdelete from elvis_sightingsqU arguments_srcqUUtemplateqcShared.DC.ZRDB.DA SQL q NRq }q (U_varsq }q UrawqhUglobalsq}qU__name__qUqubUtitleqUDelete Zope Tutorial DataqU_argq(cShared.DC.ZRDB.Aqueduct Args qoq}q(U_keysq]qU_dataq}qubU_colq)U__ac_local_roles__q}qUamosq ]q!UOwnerq"asUidq#Udeleteq$U connection_idq%U connectionq&u. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

This pie is fit for a King.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqULemon Meringue PieqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUpie.htmlqUexpandqKu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqT5

title

This sandwhich was an Elvis favorite.

There is some question as to whether the bananas were sliced or mashed on the King's sandwiches.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqU(Fried Peanutbutter and Banana SandwichesqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqUsandwiches.htmlqUexpandqKu. %((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqU

title

What could be better than a loaf of meat?

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUMeatloafqU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU meatloaf.htmlqUexpandqKu. e((U OFS.ImageqUImageqtqNt.}q(UwidthqU301qU preconditionqUU content_typeqU image/jpegq Utitleq UGraceland photoq U__name__q U graceland.jpgq UheightqU192qUdataqTdJFIFC     C   -" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?-o>vh>s =ɶxEd`+!xwM6OgHfw5Os3IPٸ3^ JoR}>[)?MyqGsOdǀTae=x3Ρ˫ΑB- a1:ͥJWdoJFùIYVEm5Kk/X<|spiΝWխQxH>Y  E4STc㷁4-BQGH]3s~B`sԚ ,~o:mǪPHB w8r0˧[<ї +fN"4{ho I+ gp9Sj[KO ז:Ɠ%Z( }Tl[, 1q>hÛm YCՄjer.͘<6٭$՞Ci%嵝Җ/G( (9pq_ߍď{Yqux&ݷͲ,>Փ+vkbl|QFx~ h3̓@[D&$"w0AqC| 񥏆`ap_} *p+ʬnFncpY&"gsCx߆Αc+,v_NU@dr1='̎uf Wzפ)IVT #@#`8zZfhL {Tmca߃xKSc"r'Hx }3SUe aYwUR?u#?|#qyou͕[1VTa!+:dc5Ưh)ռ}+Z o}jSh#@/XR{"Q:9WhV =kV 7v%GF ̑DQgRwVj&7~74x4xt2ZW$erUHO0.y`lQ$9 ܧjr׏4_]Ocߛ+z*LIdB+3``w> |.կhK4I. !s P>IE1r w~M=d/ReXϕ_g$^U08 8:qZbPt g6­{2,oMmq_4(%Nam~W+]kk2_;bQ]NхeD ֭h?wzS ,]$A ` (-ZO .kQ]X ֯uxvWɅFcgBґkL&yE)?媡@]F_!7`^sj?}6K $ ܍m}UljfdҭtM}jUHq pIb},+3|I,:tQjH>T#r'H)|}N]R ,-CszɩZIVf8EHˍ r>o<τφmʸԢծ\q$T`,07~x PڦiwZkia*()V]CQ֦JnPʷc]%hC`:Ƌ}txJ={MFW>|K'OCdt&L׎/^g%/88VڌaSԊII{8X/CO_cTP0sJ>.|SbġbxIX\dǯSE𺺒@wg *B3!1 |@:~% f瞏Dr$k[CQO+<g._39WZi3霅c*~!€nAVV/^kĉge .F8s\#+6 f˾Fo5HU\0JAzƳdRE9s H\)v[{-h!e?\qi^o[ÖJaDzսs]儍r>[5- )ϯk?e)+ay |??${wOUJ{fľ(ZAqg2ĭnw(KO'w } zaj@RkyM'2}QQ]xlygM7(oϻӼR.[X8KկgwQ($]zi+p/*q܏/ i|1կ-`61-i$$ IaK>`W[Vw7$p߄6"~oWGIX I]oU\]_my$Q99ZNmB7nƍM4;A̒ʃo8SÂ~85 `W$.::X|Cyǥ eM s 8#]j׷6{$6CyX ۯ-[&6kͮ<6A6G})F +1c:Jռ'FSUcQkv]/+ʸ ̪8wACL~2|SJ8@K;(`yt ^އS.>.99ߞ} <=}yXfuݢBw++b+4cm_6{,-7Q^)vzǏ+JRKDs(G5jF:@u;ۉf9s>SczװmkLXUb$)gvr:{kU}x'3gl5;(w\s?/׎L)Y9仸4*N1͟~k]14+ _*.ȞO3à4-}y4Pq6aUN?u: U:mA[y1?SSZ5צyy11dӐ: GbR@ϹIeؗKytMm5׫sy$v铊)eZ@돟t{9 Mq_zBVW!~~Wkr՞ \WLgёs,ZZhV44q=}F5XKe噂RO|X Ǝ<;] фe28Y) t<;?/RgU2JXYZfOEnղ}N֑M~FZxfٵJU{eG~}0=x|}oڛ޸'dc޼V?Fn?⵴ߌAo`7 苅>8kv;)Vc'1?1ZPI i `'C[HʟZn,l}7v=?|w&U\}n=$<8Z gIw=Lۊ޿^#`w y]&ך΁|]̋) Tg[(H:ǯz9U_;2}#9$aYJkI%h8m{➧-ay0K @8@tB:ŝy&E >s8o:mw/e{L6*ŷ zV/^$yB#g8PTf"G^?.1<^'ᥳ)JsgtmI$cqח:F2c8G/ٚ_toxo($TM ,Dz ^A5eXdu/NӚUxZNAimV6ڪ$.wr=` 5mьhy]Py-2 t7OGxa!WK2[>㱯> ܲ6gKb2bF^8ԌbI<x&mr@,R|`/&h[Z<%{P`<2+{`VZ؇_ _F١*uaQ+͛PA_M |ݳZrG3nk=$|P*bLeqVER>%axvRFp7oFDqZ)I4q>Ke;$C(~+kkP#h&hGL"|pf<-eeH;cft9V=ׁ?O_+$,.%DXd`kwՅksb/<ʻ!PB6g G7jτlQ4IFr298+ʨ~d6V} y26?u:Dai:7[㤼 U sҾax[+`w>YgN$}u|rnuO #`}rk\SݯƁ eO&ܘXK0xG]u  V[vuB!V;kZU0K.R<Ǭ[2":<WT \M{=O&8d _3|G~ۙERH-'?~fb}/XĜ6AҖvŇÛ^ >T'-&`q>I\n㏃ iE Ym'gdh$~O&l͐(_~<~=2?l0cZ|UJj?R6ztVrFG|aZAU{pz45_EYYyI#h\8<5^ӭ{kmʹt8t=0=y Fg a__^EᲭ-"H0]Ww +Oeect ^yckI%4b`Yq'_AA* v@.CpAT%U4S> y.$d< pWn`6?zd[.,c}$=O|Mk9?3WS}3-&~w 8?k23$tgHr]+J-im=$ez}wsL@'>kk`UqGKJNw>kX\#>V mZ`s#i6"9~B0yx>].;+\t$$>#Y@ Һ~ϩi<׳C$Ч+ПF s#-J ~׾ +k}'SiHP58,5^'$DHѶGK\7tU{ <_|6٬LBO q21*j<-Yo!h68 ]2Y;o.uXaw>:~t[j?iI`LGN k߉[ob}ߜ;`q*w#ZOd L| ַ:i%Uj$Cho<7P"2K^ٯpg0I2pyq9u[\,i2Bʋivwv}Ym|Gc}Ϙ(W|/]py; e&f'9#rM"̷b #Ą1Azgy৻C鯮dE ~#sW-:pOFs|\^ǨH+?+`{־Aqc[FfLpP^?5Kc2a>˰䈕{ӌ{W^J hOKTA3Epߕ]x@]ݞj! ^!s9q\OkU΀bdG?2;gd43Mvd\tWn"UlqBMc[x F%2ZI 9<{О0B:kʿg >.n[,`q瞞>0?Ȕ~5ʥyOח܋>dkZ#nM{=m<9`pOR3e {pײ- Y.?U2E(~^hFGg+DK$+v2pdt>ot.;ozK[>B\?޻Q&R~9_, tFv֯EA'Zr-nqs}EDb81 {k ԤԮZ)| `8E$Re.osid[Q$PD`y)ӎ>J+¶0<`|Evgm@T0 pw!~ZCmh偄$r$+i="Ez}k۬~;T"=-{߄c+ G^Y9!p0*c;đ#" ֌~ ]^:\A$~b&NwXDzM-hM:½~fܼk".$SO޿Bt|kw6-"%:Vs'p2xY[kk dl6F?z,0lv"Ke"k) btȯ;-y<i 0V]n`alc8~Xzs NzGr5 6ca%r06H {y2Uʜ{N_o|Bn.-|m-IDo V>мk:6vmK+pnɌc$8irͽaԌ!Wl{\B-3!VX-_Z͂S5amVb;L$'0ǫ{רh n,<;Ƴ7X]>^̿Roz_1\I? 0][#ƻ[} o`m=f#A[Qn>wfs5(EvL]_=;bpn@8 9xWy[xY,v p:t^B{9}dHSI3yOͭƏa:|!oq}3sҲTiyJgg@~D%XQz)vP.{'u+KQc0)#tZռ|`1No~ I_ߴ:?vlK$z5Yc;q=˕r^<v{ ݵΫvue@ b@zҭM6.,v?tqꇋAI*![M#@z`g1\Mi$fK)E4s4[Y"Indt+/Ěez Vώ& Uv݌ K\I +fD(Pg}9ݫOfF*Ԇ/M>h,n4YSdp;J[5̡#%}\Bnlxs#]TqXڎlib&8$0 8#$+nOGԿ|mAM]b9#9`׳GIk8nU"1yc'p_|QY Ɨ<ȱfR8l=rw5_o"ּmos,]%Òv'$*= 6UQzWՋPdVBIᕛ ՙ2V9.6Ο4o.K>qy$~>X4(UT/ J:IrbJo9K4msr.dp($y-αj(\~'</joLot1~ H;z^uyxfk]}~4@z-vg y\>5F5BnrKrp {ru8IZ:qqї-ַOb'!Uִ-xAYmfMqsr#cd(+Iᄌ}ZP̻,[z/t}BPn=מ}j^ԮLVi>wlqtLl'/1Hqs8w%k<=%MWe q(+48ȁ~{ڬ^01. U)Z1T4۩p!nIijd?k>kƊ2);2{s%@/skiψ~\ڄ4e;xX~pA긮rP٘ZߍE[hDCU?L`ߣ0ٞ}W#Σ3]"iwZ{H+m a*Ky?UhȒ5'9_=eB6Sȩ^2ؾ 5H!1&%bu$RpGLSuC\2d# xÎq#?Pl I!|9k+Mh.v$H8^Êxy'քF+F@ i+ReV7PŢWnRF= ~xy .bU!XUZgmUmRs2zSa\Og]oAE0yW!QN:gSЪhP(eTJ{e HqL֨ɩ1 d$3չY"5mVi$*ImFoΑw${~xV{{>~ <U dˑ~5h6IIJ)*yǭynM+d9 st7_ &-}lBsw-FـR67@=ڽ I??q }ߩ]>\ K4WhШv~k~mm;YΞ), )[?(sۥr,B'~iF_rl|bZks$Kv &M5Ԃ)mVG^M|)Rث,{{ioC>x3t޾Ź1Ϛ;draڒ:W4ϭ|!bi𽐗Ò_3$6>ozm[f?Ydkϕx##$9,R%΍4|Z vRgQ?C]5jb2D>P$ |oSI=U3LI}~/L]>QS*CbipnERq #7!d||-^nȡ@25oGRY C! /`3V#mjr~I !4.8짦: +_[HcFbb,r;\OK/XXdcᐝMr!?Zh˒oc$vz>G},^l@1^{W%I<p&A6d#8 u< OV y1t 0zqzqnugĐ2๋~\8bvV&V~Lw zAeiMCH4$FAF <_Зu{6N\|̸ c:oX'x\"ݼo ,(>`+ n8~+n־~6mu5BI` >;G@+O.4Fe}06vr#8;fp1PcmgMeҦkL|"z%cx<-eu-1XA8V?yv|-du[HNn9CSq 驠%s_"ƾ XG}rCD&pw|0FrV#&WCe)\|A##^Ww/*pЛp1@#aUr얽ь))ϳ ᔎ? ]L.yA(ss\O][ǺXhG̰dJ|n9]b4_h߄.$b>hrXВOMeQi6pq)a$+1q+ܜeXbi*/N#>-Ԏx-/+Bzf:Ƴyc:ǧC<`Ux ?18<?";7t5"xV8 6A#UP2dxx|&-?\D<ܕ] SRU[ .|֦ŖqjahLJ2F ^ⴌV-o3 ~.{(hPbv^Nk/"eB~R:[ypmPǓ{:W }vPy~bG=s3Z[+4;wchr>EC 3fªsH>lM/^4o-n-NDgn >eG\۱6m_{g> Xk"(>\NmFIů#> A!Go/'_qU`{DH!#$p s8 OT5M.#cy[Ng89qU:$_;ց#ҥfH88dw^F_z+\đ!BqXGS0VWJj/IXט Bݑjv44EtRSGe>&i"#@α.ONOQos6&UD@g##CFK_rsn\gɰ!U^HøNS,v=At'觟x)Z<. IW]|(g :ćIG^%l?u_/pBc#s<ƽ\] zR]Ov0VzKno5V.w0<֯.k-DՃMc''d:?8ϣ/c\M'K.!`s/qp!VŇW>!V__6|o nNp\=$2mB+_ji黉1 |#C?igɮK%4Хȍ=+˾1¶58@F`y)^}y,;-Oؐl2'xⰪuݓ8ihWz[ &uH  N{qPXsK#I&|GR\g}Hc@링ٜWM]-vO wߴ4CNUH+ncU8'IK(o!q\ U||ceŲ[ArpgMsલT<"t|4'9EZPg'?}?7KBN-?I<3ΙfqlҼ"ڟR= l0y{Ã> ldYu9i^L?,z|O'铌kf>}b8k"S eOb195g^sY57}M~Ƶew?M|n^_ O0$o<`mFrs4xguee_*+g 0knnZ9*M3T${sr`:n㧥qQl`;c{csރ ߈~1NdXSk;Fz=kdZ]Hmy 篭{:_k0//Ul4cxXČX8ݏMxC/\cMK˽$Ul99Qsmo/0Fz/O+^aj -͵B=+CC[P#M&cvczP %>d;7;pzW!JƲy.aƟ>3úFHeK7ntr.1 ic}/z,z)fAJ*y:u]88'R_3!v=\',ecؼ)}?j9$>%)V 򂗖 x\עFRO FUZx=el/|y/Ń)[Yae{tW8 Ff|0T"}BmgROɎ.dȊkTHޯ R֓G!Hz՛qS&, G_>O&>qoeP;W_TS 9 VkW9Gh.N7k|Z-u?ڕ\ nj1 m*331O$i1c!E8$\ey6 >]AHB rv&2TY(ҔO٘<D#ج<10: O '$f AXqN9>Ŀ|5hT)0pCI VLdαN9$ Sr]_+\u)m/gʼx6A{4_ h+س2]$)p*T` μW_׺V~ oB.m%9HS9Zڟ4HcJ7h'$"eL•`Jo$GzUGU> B Cfٳ(}BOԂ8n`ҧ$ࠑ92Bv#zo_b!Xr,BIyp[iu E;į_=K!ٰ'pq qIu%w񿆟|Kv\!v+9$6wmIkW *1Urp@8|VG~<tgʖ9cs d; q '9maq:ebg 2M3L. :"Rpq2HIѿhuXmo쁸p,܉8 {w⼺oF!u{G 9i?-t뫛Ur!V.G|Wƽ)-m4֫5hךXi%qݘ>@Pa3(mlt+s4yWX:E\$Uƥ0Q b1r\瞵̓yֆB1U%3>j!񳮝ocur}y46U\!^i{iwQ,MCc=*Q4&0S9OW-}fР YX1#*S{X TPrPzĿҭE7JQdDXA?&@35CKr [=式q\*)&z.w-> ?BMnmn#Wfr/C"l2kܿhi|pו̃Xn?WG1P uGs*8R|S^#U]ÆX#뚷;cMJdȊ:Ҷdh&Ö bb%ݵ :dEK>撻z^G9 Lbꚍ d,z[}4L!$~5ڄ"Cq17@7~K =@=2@}xn.d[&cxnd2gq;^GMN:;mVGp Os⋇ės,*Fryq_B~^^a;tRlD4H/ͷnCp) $G]7.ӌ~VEc-F[%YY c5/Өy_ 1ae'c;_>?(m>Ԅ,)P*݇yy9սz8Nퟜީ{\w0$Xg<:;E]rX##P<;:I}.7_gúfZ S2d8 @$8澇Gxk5BMO6b8% 9+֫Y%uV6}OoZ}3^ѵ 2=IWsXQ+bO@OLG;kU[Wc嗑1{dؒgoi6$M\U,/lU;lMs=aGx+|ǖĿφAuB46wi>Ih r 1DžYG?zx_%wc9#@/+o^ٶIt+]CKFIIcPP*|_vx'7IwGķ*JGyeEr>vbI잝(ΫwTQ{fVui0G1oֻ>e搇pp"6;''ׄjfY,\M(BsJ+u4k@9>Q,vҊ3NKfzT$ӭ'i5 l(*`p>N6о(n$\W!'ng֌ e2NA܁V8h@ty86St }K[<86sc5,pԬ%,0{?rڞst{uaqI㧕82G[lmR¿TyQ)vwWzm/H4`p~8&?ƽX\U׭Ӵx`',B8$IKP1 !bi8=vAW_ C:jhKe1~ӎۂG-Y5?|]KI/c]W c?.@r-e-bݥOs$k #d 3~G=+/ ѵNjK5ܬF|cU$kХS^Y ͅh825iFױq>v~[;%ׅM֋$X$ 2@rw@u vʌeEZy5hL\M`3+:.g=IX>\EX]̓J:':ug2(:bm{>?.{Z,4V "blfcp$c?K:6Ǩ\Hn + ˲5T'}ϰu#T:wmF@x9#+{Wקu]/:I9Ufn=>k׵>?-@$=џ$'=*Z_b< o\fM}ǚZ\ZLX/!}Gʁ \ͮ( T.sӧND9ボ洔u>]ţTp0d8a`ϨH''s8}jue{ZjxSº]fv#kOIvİ(XE.x+xKVf@Igں 16 v"?|[$o<u2y|/ |@?ZioO>MXI38\ vb&)bgʌTB1=GaԞ|F1 *GT[wkYhCWv9< ڙ^i\iQhWpiq"$7c$nP@ڠ-o( (+#H5Sp0F?8vk5g*<1n3׊ݝeY2a('NG_jLj}s.h;m"SU; Zş& !o3M wR˽ dǹ~?I+ub[X$vH".w1;lz7 Vy\iXSm?{;^]|4ZZo{D{rEG#  cA].%e14g kdVlK>`{|eZWو;,nϰx)MIy߄Y/#.'i\n9гUG p}++BN-;BGG#]4]Q)ר?FRs>j˚ "6whn?V1 DZw7q- .]FW:[q47PC0Fx]Sō)֪J|~ǡ+sJ4Ϛu?Q/AnJdF͑H'3:p*<5؈85aS_CQqU__ac_local_roles__q}qUamosq]qUOwnerqasUsizeqMdu. ((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(U content_typeqU text/htmlqU_textqTC

title

Graceland is home to Elvis and Elvis merchandise. Come tour Graceland and get in touch with the King.

qU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q oq }q U_asgnsq }qU name_subpathqUtraverse_subpathqssbUtitleqUU__ac_local_roles__q}qUamosq]qUOwnerqasUidqU index_htmlqUexpandqKu.Zope-2.8.7-final/import/Examples.zexp0000664000076400007640000016147510435500331016550 0ustar ajungajungZEXP Z((U OFS.FolderqUFolderqtqNt.}q(U examples.cssq(U ^q(U OFS.ImageqUFileqttQUidq UExamplesq U Navigationq (U _q (hUFolderq ttQU_objectsq(}q(U meta_typeqU Page Templateqh U index_htmlqu}q(hUFileqh hu}q(hUFolderqh U Navigationqu}q(U meta_typeqUFolderqUidqU GuestBookqu}q(U meta_typeqUFolderqUidq U FileLibraryq!u}q"(U meta_typeq#UFolderq$Uidq%U ShoppingCartq&utU__ac_local_roles__q'}q(U_ownerq)(]q*U acl_usersq+aUadminq,th(U \q-(U'Products.PageTemplates.ZopePageTemplateq.UZopePageTemplateq/ttQh!(U ]q0(hUFolderq1ttQUtitleq2UExample Applicationsq3h&(U [q4(hUFolderq5ttQh(U `q6(hUFolderq7ttQu. ^;((U OFS.ImageqUFileqtqNt.}q(UsizeqMSUdataqTSH1, H2, H3, H4, H5, H6 { font-family: Verdana, Arial, Helvetica, sans-serif; font-weight: bold; color: #6699CC; } body, .normal { background: #FFFFFF; font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt; } .small{ background: #FFFFFF; font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 9pt; } .grey{ background: #DDDDDD; } .white{ background: #FFFFFF; } .bold{ font-weight:bold; } .normal{ font-weight:normal; } .message{ background: #FFFFAA; border-width: thin; border-style: solid; padding: 10pt; font-weight:bold; } qU_EtagSupport__etagqU ts05767609.14q U__name__q U examples.cssq Utitleq UU content_typeq Utext/cssqU preconditionqUU__ac_local_roles__q}qUamosq]qUOwnerqasu. _((U OFS.FolderqUFolderqtqNt.}q(UidqU NavigationqU__ac_local_roles__q}qUamosq ]q UOwnerq asUBirdsq (U pq (hUFolderqttQU_objectsq(}q(U meta_typeqUScript (Python)qhUbreadcrumbs.pyqu}q(hUFolderqhUMammalsqu}q(hhhh u}q(hhhUReptilesqu}q(U meta_typeqUScript (Python)qUidqU siblings.pyqu}q(hU Page Templateq hU index_htmlq!uth(U mq"(hUFolderq#ttQh(U rq$(U#Products.PythonScripts.PythonScriptq%U PythonScriptq&ttQU index_htmlq'(U qq((U'Products.PageTemplates.ZopePageTemplateq)UZopePageTemplateq*ttQh(U oq+(hUFolderq,ttQU siblings.pyq-(U nq.(h%U PythonScriptq/ttQUtitleq0UAutomatic Navigation Devicesq1u. \((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU index_htmlqU__ac_local_roles__q}q U _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q o}q U_asgnsq }qU name_subpathqUtraverse_subpathqssbU_textqT The document title or folder title or the example title

The title

Welcome to Zope. We've collected some example applications that demonstrate simple Zope features. You can examine how these applications work by looking at them in the management interface. If you'd like to contribute an example application to add to this collection, write to us. Enjoy!

You can delete the Examples folder if you wish. You can also reload the examples by importing the import/Examples.zexp file.

qU content_typeqU text/htmlqUtitleqUu. ]((U OFS.FolderqUFolderqtqNt.}q(UidqU FileLibraryqU__ac_local_roles__q}qUamosq ]q UOwnerq asU_objectsq (}q (U meta_typeqU Page TemplateqhU index_htmlqu}q(hUFolderqhUFilesqu}q(hUScript (Python)qhUgetFilesqu}q(hhhUgetLinkqu}q(hhhUgetSizequ}q(hhhUaddFilequtUgetLinkq(U iq(U#Products.PythonScripts.PythonScriptqU PythonScriptq ttQUgetSizeq!(U jq"(hU PythonScriptq#ttQU index_htmlq$(U lq%(U'Products.PageTemplates.ZopePageTemplateq&UZopePageTemplateq'ttQh(U kq((hU PythonScriptq)ttQUtitleq*U File Libraryq+h(U hq,(hU PythonScriptq-ttQh(U gq.(hUFolderq/ttQu. [((U OFS.FolderqUFolderqtqNt.}q(UaddItemsq(U fq(U#Products.PythonScripts.PythonScriptqU PythonScriptqttQUidq U ShoppingCartq U__ac_local_roles__q }q Uamosq ]qUOwnerqasU_objectsq(}q(U meta_typeqUScript (Python)qh U currentItemsqu}q(hhh UgetItemqu}q(hhh UgetTotalqu}q(hU Page Templateqh U index_htmlqu}q(U meta_typeqUScript (Python)qUidqU deleteItemsq u}q!(hhhhuth(U cq"(hU PythonScriptq#ttQU index_htmlq$(U eq%(U'Products.PageTemplates.ZopePageTemplateq&UZopePageTemplateq'ttQh(U dq((hU PythonScriptq)ttQUtitleq*U Shopping Cartq+h (U bq,(hU PythonScriptq-ttQh(U aq.(hU PythonScriptq/ttQu. `((U OFS.FolderqUFolderqtqNt.}q(UidqU GuestBookqU__ac_local_roles__q}qUamosq ]q UOwnerq asU_objectsq (}q (U meta_typeqU Page TemplateqhU index_htmlqu}q(hUFolderqhUEntriesqu}q(hhhU addEntry.htmlqu}q(U meta_typeqUScript (Python)qUidqUaddEntryqu}q(hhhU formatEntryqu}q(hhhU getEntriesquth(U vq(hUFolderq ttQU index_htmlq!(U wq"(U'Products.PageTemplates.ZopePageTemplateq#UZopePageTemplateq$ttQh(U xq%(h#UZopePageTemplateq&ttQUtitleq'U Guest Bookq(h(U tq)(U#Products.PythonScripts.PythonScriptq*U PythonScriptq+ttQh(U sq,(h*U PythonScriptq-ttQh(U uq.(h*U PythonScriptq/ttQu. p((U OFS.FolderqUFolderqtqNt.}q(UidqUBirdsqU__ac_local_roles__q}qUamosq ]q UOwnerq asUPigeonsq (U q (hUFolderqtqtqQU_objectsq(}q(U meta_typeqUFolderqhUCrowsqu}q(hhhh utUtitleqUh(U ~q(hUFolderqtqtqQu. mr((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUReptilesqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu. r ((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_argcountq KU co_varnamesq (U _getitem_q U_getitemq U _getattr_q U_getattrqUProducts.PythonScripts.standardqU html_quoteqUlinksqUcontextqUparentqtubU_paramsqUUidqUbreadcrumbs.pyqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq }q!(Uname_containerq"U containerq#U name_subpathq$Utraverse_subpathq%U name_m_selfq&Uscriptq'U name_contextq(Ucontextq)usbU func_defaultsq*NU_codeq+TcsdZdS(s This script creates breadcrumb style navigation links. It walks the REQUEST.PARENTS list of parents and creates a hyperlink for each parent. It stops at the Examples folder. c st}t}ddkl} g} x|||tddddd rt}| |ddd |||d |||d f ||d d joPnq\W|d d |SdS(s This script creates breadcrumb style navigation links. It walks the REQUEST.PARENTS list of parents and creates a hyperlink for each parent. It stops at the Examples folder. (s html_quotesREQUESTsPARENTSiNisinserts%ss absolute_urlsgetIdsExampless/sjoin( s _getitem_s_getitems _getattr_s_getattrsProducts.PythonScripts.standards html_quoteslinksscontextsparent( s _getitem_s_getitems _getattr_s_getattrsProducts.PythonScripts.standards html_quoteslinksscontextsparent((sScript (Python)sbreadcrumbs.pys 95 N(sbreadcrumbs.py(sbreadcrumbs.py((sScript (Python)ssq,U_bodyq-T""" This script creates breadcrumb style navigation links. It walks the REQUEST.PARENTS list of parents and creates a hyperlink for each parent. It stops at the Examples folder. """ from Products.PythonScripts.standard import html_quote links=[] for parent in context.REQUEST.PARENTS[1:]: links.insert(0, """%s""" % \ (html_quote(parent.absolute_url()), html_quote(parent.getId()))) if parent.getId() == "Examples": break return "/".join(links) q.U Script_magicq/KUwarningsq0)Uerrorsq1)U Python_magicq2U* q3u. q q((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU index_htmlqU__ac_local_roles__q}q Uamosq ]q UOwnerq asU_textq T% The document title or container title

breadcrumbs navigation

The title

This example shows how to create simple automatic navigation devices. Take a look at the Navigation folder to see how it was built. The automatically created links on this page allow you to traverse a collection of folders and sub-folders.

Current Folder: Folder ID

Parent Links

The links you see across the top of the page are called breadcrumbs. They show you where you are in the site and let you navigate up the site tree. They are created with a Python-based script, breadcrumbs.py.

Child Links

These automatic links allow you to navigate deeper into the site. These links are created with a Python expression in a Page Template. You could also use a Python-based Script to create the links.

You could customize these links to only show certain sub-folders, rather than listing all sub-folders.

There are no sub-folders here.

Sibling Links

This navigation device allows you to go to sibling folders. A sibling folder is one that is in the same parent as the current folder. The list of siblings is generated by the siblings.py script.

There are no sibling folders here.

qU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}qU name_subpathqUtraverse_subpathqssbUtitleqUU content_typeqU text/htmlqu. o:((U OFS.FolderqUFolderqtqNt.}q(UidqUMammalsqU__ac_local_roles__q}qUamosq ]q UOwnerq asUPrimatesq (U }q (hUFolderqtqtqQU_objectsq(}q(U meta_typeqUFolderqhh u}q(hhhUBatsqu}q(hhhUWhalesquth(U |q(hUFolderqtqtqQUtitleqUh(U {q(hUFolderqtq tq!Qu. ne((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (U _getattr_q U_getattrq Ucontextq Ufoldersq tU co_argcountqKubU_paramsqUUidqU siblings.pyqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(Uname_containerqU containerqU name_subpathqUtraverse_subpathq U name_m_selfq!Uscriptq"U name_contextq#Ucontextq$usbU func_defaultsq%NU_codeq&T3csdZdS(s/ This script returns a list of sibling folders csRt}d||tddd}||dt|SdS(s/ This script returns a list of sibling folders s aq_parents objectValuessFoldersremoveN(s _getattr_s_getattrscontextsfolders(s _getattr_s_getattrscontextsfolders((sScript (Python)s siblings.pys $N(s siblings.py(s siblings.py((sScript (Python)ssq'U_bodyq(U""" This script returns a list of sibling folders """ # get my parent's children folders=context.aq_parent.objectValues('Folder') # remove me from the list of children folders.remove(context) return folders q)U Script_magicq*KUwarningsq+)Uerrorsq,)U Python_magicq-U* q.u. iT((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Unew_sortq Usortq Ureverseq U _getattr_q U_getattrqUcontextqUurlqU new_reverseqtU co_argcountqKubU_paramsqUnew_sort, sort, reverseqUidqUgetLinkqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq }q!(Uname_containerq"U containerq#U name_subpathq$Utraverse_subpathq%U name_contextq&Ucontextq'U name_m_selfq(Uscriptq)usbU func_defaultsq*NU_codeq+TcsdZdS(sA Returns a URL which gives sorting commands in the query string. c sjt}d||tdd}||jo d}n | } d|||fSdS(sA Returns a URL which gives sorting commands in the query string. sREQUESTsURL0is%s?sort=%s&reverse:int=%sN(s _getattr_s_getattrscontextsurlsnew_sortssorts new_reversesreverse(snew_sortssortsreverses _getattr_s_getattrscontextsurls new_reverse((sScript (Python)sgetLinks   N(sgetLink(sgetLink((sScript (Python)ssq,U_bodyq-U""" Returns a URL which gives sorting commands in the query string. """ url=context.REQUEST.URL0 if new_sort != sort: new_reverse=0 else: new_reverse=not reverse return "%s?sort=%s&reverse:int=%s" % (url, new_sort, new_reverse) q.U Script_magicq/KUwarningsq0)Uerrorsq1)U Python_magicq2U* q3u. j(((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Ubytesq Umegsq UktU co_argcountq KubU_paramsq UbytesqUidqUgetSizeqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(Uname_containerqU containerqU name_subpathqUtraverse_subpathqU name_contextq Ucontextq!U name_m_selfq"Uscriptq#usbU func_defaultsq$NU_codeq%TcsdZdS(s. Format a number of bytes as bytes, K, or MB. csqd|d}|djod|Sn|d}|djo d|Sn d|SdS(s. Format a number of bytes as bytes, K, or MB. f 1048576.0is%.2f MBf1024.0s%d Ks%d bytesN(sbytessmegssk(sbytessmegssk((sScript (Python)sgetSizes  N(sgetSize(sgetSize((sScript (Python)ssq&U_bodyq'U""" Format a number of bytes as bytes, K, or MB. """ megs=bytes / 1048576.0 if megs > 1: return "%.2f MB" % megs k=bytes / 1024.0 if k > 1: return "%d K" % k return "%d bytes" % bytes q(U Script_magicq)KUwarningsq*)Uerrorsq+)U Python_magicq,U* q-u. l((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU index_htmlqU__ac_local_roles__q}q Uamosq ]q UOwnerq asU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}qU name_subpathqUtraverse_subpathqssbU_textqT? The document title or folder title or the example title

The title

This example shows how to create a simple file library. It demonstrates batching and sorting. You can click on the column names to sort the files. If you click twice on a column then the files are sorted in reverse order. If you upload more than eight files, then the results will be split up over more than one page. This is called batching.

The files are located in the Files folder. The getFiles script returns a sorted list of files. The getLink and getSize scripts do small formatting tasks to build hyperlinks and format file sizes. The batching is done in the index_html template. The template also makes use of the test function to assign different CSS classes to elements depending on conditions. This is how the sorted column name is bolded and alternate table rows are made gray.

Files

message goes here

Files 1 to 10 of 17.
< previous 10 next 10 >

ID Type Size Last Modified
example.zip application/x-zip-compressed 12 K Nov 14, 2001 9:31 am

There are no files currently available.

Upload a File

You can add a file to the library by uploading it from your local computer. The addFile script accepts the file upload and creates a new file object in the Files folder. It then returns this page with a confirmation message. For demonstration purposes, only 10 files of no larger than 2Kbytes may be uploaded.

Adding files is protected by the "Add Documents, Images and Files" permission. If you want anonymous users to upload files, you could visit the "Security" tab of the addFile script, and give it the "Manager" proxy role.


qU content_typeqU text/htmlqUtitleqUu. k ((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_argcountq KU co_varnamesq (Ufileq U _getitem_q U_getitemq U _getattr_qU_getattrqUProducts.PythonScripts.standardqU url_quoteqU ExceptionqU LimitExceededqUhasattrqUsizeqUlenqU containerqUmessageqUcontextqtubU_paramsqUfileqUidqUaddFileqU__ac_local_roles__q}qUamosq ]q!UOwnerq"asU _bind_namesq#(cShared.DC.Scripts.Bindings NameAssignments q$oq%}q&U_asgnsq'}q((Uname_containerq)U containerq*U name_subpathq+Utraverse_subpathq,U name_contextq-Ucontextq.U name_m_selfq/Uscriptq0usbU func_defaultsq1NU_codeq2TcsdZdS(s Adds a file to the library. cst}t}ddkl}dtfdY} t|do t|d o ddn| |ddd  ||d} ||ddt ||t d d d d jod| djoD||||t d dddddddd|n |dn |dd||d} |||tdddd|t d|| fSdS( s Adds a file to the library. (s url_quotes LimitExceededcs dRS(s$raised when file limits are exceeded((((sScript (Python)s LimitExceededssseekstells Bad RequestsInvalid file specifiediisFiless objectValuessFilei ismanage_addProductsOFSPsmanage_addFilesidsstitlesfiles(Uploaded file must be less than 2Kbytes.s%limit on number of files (5) exceededs File '%s' uploaded successfully.sfilenamesREQUESTsRESPONSEsredirects %s?message=%ss absolute_urlN(s _getitem_s_getitems _getattr_s_getattrsProducts.PythonScripts.standards url_quotes Exceptions LimitExceededshasattrsfilessizeslens containersmessagescontext(sfiles _getitem_s_getitems _getattr_s_getattrsProducts.PythonScripts.standards url_quotes Exceptions LimitExceededshasattrssizeslens containersmessagescontext((sScript (Python)saddFiles$.8 N(saddFile(saddFile((sScript (Python)ssq3U _proxy_rolesq4)U_bodyq5T""" Adds a file to the library. """ from Products.PythonScripts.standard import url_quote class LimitExceeded(Exception): """raised when file limits are exceeded""" # create up to 10 files of 2k apiece if not (hasattr(file, 'seek') and hasattr(file, 'tell')): raise "Bad Request", "Invalid file specified" file.seek(0,2) size=file.tell() file.seek(0) # rewind if len(container.Files.objectValues('File')) < 10: if size < 2048: container.Files.manage_addProduct['OFSP'].manage_addFile(id='', title='', file=file) else: raise LimitExceeded, "Uploaded file must be less than 2Kbytes." else: raise LimitExceeded, "limit on number of files (5) exceeded" # create a success message message="File '%s' uploaded successfully." % file.filename # redirect to main page - this is necessary to make all the URLs # on the main page work correctly. return context.REQUEST.RESPONSE.redirect("%s?message=%s" % (container.absolute_url(), url_quote(message))) q6U Script_magicq7KUwarningsq8)Uerrorsq9)U Python_magicq:U* q;u. h ?((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Usortq Ureverseq U _getattr_q U_getattrq U containerqUfilesqUargsqUsequenceqtU co_argcountqKubU_paramsqUsort='id', reverse=0qUidqUgetFilesqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq }q!(Uname_containerq"U containerq#U name_subpathq$Utraverse_subpathq%U name_contextq&Ucontextq'U name_m_selfq(Uscriptq)usbU func_defaultsq*(Uidq+Ktq,U_codeq-TcsdddZdS(s; Returns a list of files sorted by id, date, type or size. sidic st}d||tddd}f}|djo7 | o dff}n dddff}n |djoC| od fdff}nd ddfdff}n|d joC| od fdff}nd ddfdff}nT|d joC| od ddfdff}nd fdff}n|td||SdS(s; Returns a list of files sorted by id, date, type or size. sFiless objectValuessFilesidsgetIdscmpsdescsdatesbobobase_modification_timestypesgetContentTypessizesgetSizessortN(s _getattr_s_getattrs containersfilessargsssortsreversessequence(ssortsreverses _getattr_s_getattrs containersfilessargsssequence((sScript (Python)sgetFiless( !     N(sgetFiles(sgetFiles((sScript (Python)ssq.U_bodyq/T""" Returns a list of files sorted by id, date, type or size. """ files=container.Files.objectValues('File') args=() # build arguments for sequence.sort if sort=='id': if not reverse: args=(('getId',),) else: args=(('getId', 'cmp', 'desc'),) elif sort=='date': if not reverse: args=(('bobobase_modification_time',), ('getId',)) else: args=(('bobobase_modification_time', 'cmp', 'desc'), ('getId',)) elif sort=='type': if not reverse: args=(('getContentType',), ('getId',)) else: args=(('getContentType', 'cmp', 'desc'), ('getId',)) elif sort=='size': if not reverse: args=(('getSize', 'cmp', 'desc'), ('getId',)) else: args=(('getSize',), ('getId',)) return sequence.sort(files, args) q0U Script_magicq1KUwarningsq2)Uerrorsq3)U Python_magicq4U* q5u. g((U OFS.FolderqUFolderqtqNt.}q(UidqUFilesqU__ac_local_roles__q}qUamosq ]q UOwnerq asU example.htmlq (U zq (U OFS.ImageqUFileqtqtqQU_objectsq(}q(U meta_typeqUFileqUidqU example.txtqu}q(hhhh utUtitleqUh(U yq(hUFileqttqQu. f .((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_argcountq KU co_varnamesq (Uordersq UREQUESTq U_write_q U_writeqU _getitem_qU_getitemqU _getattr_qU_getattrqUcontextqUsessionqUitemsqUappendqU$append0qU containerqUdU valid_itemsqUorderqUhasattrqUintqUquantityqUNoneqtubU_paramsqUorders, REQUEST=Noneq Uidq!UaddItemsq"U__ac_local_roles__q#}q$Uamosq%]q&UOwnerq'asU _bind_namesq((cShared.DC.Scripts.Bindings NameAssignments q)oq*}q+U_asgnsq,}q-(Uname_containerq.U containerq/U name_subpathq0Utraverse_subpathq1U name_contextq2Ucontextq3U name_m_selfq4Uscriptq5usbU func_defaultsq6(Ntq7U_codeq8TcstdZdS(sf Adds one or more items to the shopping cart. Each order is a record with an id and a quantity field. cst}t}t}d||tdd} || ddh}  gi } |t dd r}| ||dqq~ } x|d r}t|do&||dot||dpd}|djo#t|do||d|jo?|| d||dd||| ||dsq9U_bodyq:T@""" Adds one or more items to the shopping cart. Each order is a record with an id and a quantity field. """ session=context.REQUEST.SESSION # get the items, items is a dictionary that maps item ids to quantities items=session.get('items',{}) # get the valid item names for form validation valid_items=[d['id'] for d in container.getItem()] for order in orders: quantity=hasattr(order, 'quantity') and order.quantity and \ int(order.quantity) or 0 if quantity != 0 and hasattr(order, 'id') and order.id in valid_items: items[order.id]=items.get(order.id, 0) + quantity # save items back to session (this is necessary # since items is a mutable non-persistent object) session['items']=items # if called from the web return a response page if REQUEST is not None: return container.index_html(REQUEST) q;U Script_magicq)U Python_magicq?U* q@u. cp((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Uidq U_write_q U_writeq U _getattr_q U_getattrqUitemsqUNoneqUitemqUresultsqtU co_argcountqKubU_paramsqUid=NoneqUidqUgetItemqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}q U_asgnsq!}q"(Uname_containerq#U containerq$U name_subpathq%Utraverse_subpathq&U name_m_selfq'Uscriptq(U name_contextq)Ucontextq*usbU func_defaultsq+(Ntq,U_codeq-T~cstdZdS(s Returns an item given its id. If no id is provided, returns all available items. In a real shopping cart application, this script would probably consult a database of some sort. Each item is a dictionary with an id, title, description, and price. c sTt}t}dhhdd<dd<dd<d<hdd<d d<d d<d <hd d<d d<dd<d<}|tj oQ||d|}|tjo tSn|||d<|Sng}xN||ddr5\}} |||d<|!|d|q W"|SdS(s Returns an item given its id. If no id is provided, returns all available items. In a real shopping cart application, this script would probably consult a database of some sort. Each item is a dictionary with an id, title, description, and price. s Econo FeederstitlesThese feeders are made in Taiwan from a light-weight metal with plastic ends, and have an anti-spill edge to prevent the birds from kicking the feed out of the tray. They have a wooden rollbar to prevent the birds from sitting on them.s descriptionf7.95sprices510-115s Hopper Feeders7Need to go out of town for a few days, and no one can feed your pigeons? Don't worry, we now have the virtually spillproof hopper feeder. Made from birch plywood it holds from 30 to 35 pounds of grain. Pigeons can get at the feed through holes in the plexiglass cover, but will not be able to kick out any feed.f55.0s510-122sDust Free Bullet WaterersA favorite with many fanciers, this drinker has a dust cover over each individual drinking hole, to keep out the dirt and dust.f16.0s510-007sgetsidsitemsisappendN( s_write_s_writes _getattr_s_getattrsitemssidsNonesitemsresults( sids_write_s_writes _getattr_s_getattrsitemssNonesitemsresults((sScript (Python)sgetItemsu    'N(sNonesgetItem(sNonesgetItem((sScript (Python)ssq.U_bodyq/TZ""" Returns an item given its id. If no id is provided, returns all available items. In a real shopping cart application, this script would probably consult a database of some sort. Each item is a dictionary with an id, title, description, and price. """ items={ '510-115' : {'title' : 'Econo Feeder', 'description' : 'These feeders are made in Taiwan from a light-weight metal with plastic ends, and have an anti-spill edge to prevent the birds from kicking the feed out of the tray. They have a wooden rollbar to prevent the birds from sitting on them.', 'price' : 7.95}, '510-122' : {'title' : 'Hopper Feeder', 'description' : 'Need to go out of town for a few days, and no one can feed your pigeons? Don\'t worry, we now have the virtually spillproof hopper feeder. Made from birch plywood it holds from 30 to 35 pounds of grain. Pigeons can get at the feed through holes in the plexiglass cover, but will not be able to kick out any feed.', 'price' : 55.0}, '510-007' : {'title' : 'Dust Free Bullet Waterer', 'description' : 'A favorite with many fanciers, this drinker has a dust cover over each individual drinking hole, to keep out the dirt and dust.', 'price' : 16.0}, } # return requested item if id is not None: item=items.get(id) if item is None: # no item by that id return None # add an id field to the record item['id']=id return item # return all items results=[] for id, item in items.items(): item['id']=id results.append(item) return results q0U Script_magicq1KUwarningsq2)Uerrorsq3)U Python_magicq4U* q5u. e((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU index_htmlqU__ac_local_roles__q}q Uamosq ]q UOwnerq asU_textq T The document title or folder title or the example title

The title

This example shows how to use sessions. Notice that Zope remembers the contents of your shopping cart until you close your browser.

The contents of the cart are stored in a session. The session variable items keeps track of the item ids and quantities of items in your shopping cart. The currentItems script uses this information to retrieve records describing the items in the shopping card. The deleteItems script removes items from the shopping cart.

Your Shopping Cart

Item Price Quantity
Item $10.00 2
Total
$25.00

There are currently no items in your shopping cart.

Add Items

Item Description Price Quantity
Item Description $10.00

The addItems script adds items to your shopping cart. The above form uses records to build an order list of to pass to the script. Each item in the order list has a id and a quantity field.

qU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}qU name_subpathqUtraverse_subpathqssbUtitleqUU content_typeqU text/htmlqu. d((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_argcountq KU co_varnamesq (U_write_q U_writeq U _getattr_q U_getattrqUcontextqUsessionqU quantitiesqUitemsqUidqUquantityqU containerqUitemqUNoneqtubU_paramsqUUidqU currentItemsqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q!oq"}q#U_asgnsq$}q%(Uname_containerq&U containerq'U name_subpathq(Utraverse_subpathq)U name_contextq*Ucontextq+U name_m_selfq,Uscriptq-usbU func_defaultsq.NU_codeq/TtcsdZdS(s Returns a list of items that are in the shopping cart. Each item has a 'quantity' field as well as the normal 'id' and 'description' fields. c st}t}d||tdd}||ddh}g}xz||ddra\}}  |t d|}  | t j o- | || d<| |d| nqmW |Sd S( s Returns a list of items that are in the shopping cart. Each item has a 'quantity' field as well as the normal 'id' and 'description' fields. sREQUESTsSESSIONsgetsitemsisgetItemsquantitysappendN( s_write_s_writes _getattr_s_getattrscontextssessions quantitiessitemssidsquantitys containersitemsNone( s_write_s_writes _getattr_s_getattrscontextssessions quantitiessitemssidsquantitys containersitemsNone((sScript (Python)s currentItemss 'N(s currentItems(s currentItems((sScript (Python)ssq0U_bodyq1T""" Returns a list of items that are in the shopping cart. Each item has a 'quantity' field as well as the normal 'id' and 'description' fields. """ session=context.REQUEST.SESSION quantities=session.get('items', {}) items=[] for id, quantity in quantities.items(): item=container.getItem(id) if item is not None: item['quantity']=quantity items.append(item) return items q2U Script_magicq3KUwarningsq4)Uerrorsq5)U Python_magicq6U* q7u. b((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Uidsq UREQUESTq U_write_q U_writeq U _getattr_qU_getattrqUcontextqUsessionqUitemsqUidqUNoneqU containerqtU co_argcountqKubU_paramsqUids, REQUEST=NoneqUidqU deleteItemsqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq (cShared.DC.Scripts.Bindings NameAssignments q!oq"}q#U_asgnsq$}q%(Uname_containerq&U containerq'U name_subpathq(Utraverse_subpathq)U name_contextq*Ucontextq+U name_m_selfq,Uscriptq-usbU func_defaultsq.(Ntq/U_codeq0TcstdZdS(s2 Remove one or more items from the shopping cart. c st}t}d||tdd}||ddh} x=|d r0} | |d| o||| =nqXW|||d<|t j o|t d|SndS( s2 Remove one or more items from the shopping cart. sREQUESTsSESSIONsgetsitemsishas_keys index_htmlN( s_write_s_writes _getattr_s_getattrscontextssessionsitemssidssidsREQUESTsNones container( sidssREQUESTs_write_s_writes _getattr_s_getattrscontextssessionsitemssidsNones container((sScript (Python)s deleteItemss+N(sNones deleteItems(sNones deleteItems((sScript (Python)ssq1U_bodyq2T""" Remove one or more items from the shopping cart. """ session=context.REQUEST.SESSION # get the items items=session.get('items',{}) # remove the items for id in ids: if items.has_key(id): del items[id] # save changes back to the session session['items']=items # if this script is called from the web return a response page if REQUEST is not None: return container.index_html(REQUEST) q3U Script_magicq4KUwarningsq5)Uerrorsq6)U Python_magicq7U* q8u. a9((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (U _getitem_q U_getitemq U _getattr_q U_getattrq U containerqUitemsqUtotalqUitemqtU co_argcountqKubU_paramsqUUidqUgetTotalqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q (Uname_containerq!U containerq"U name_subpathq#Utraverse_subpathq$U name_contextq%Ucontextq&U name_m_selfq'Uscriptq(usbU func_defaultsq)NU_codeq*T:csdZdS(Ncsqt}t}|td}d}x6|dr)}|||d||d7}q6W|SdS(Ns currentItemsf0.0ispricesquantity(s _getitem_s_getitems _getattr_s_getattrs containersitemsstotalsitem(s _getitem_s_getitems _getattr_s_getattrs containersitemsstotalsitem((sScript (Python)sgetTotals! '(sgetTotal(sgetTotal((sScript (Python)ssq+U_bodyq,Uuitems=container.currentItems() total=0.0 for item in items: total+=item['price'] * item['quantity'] return total q-U Script_magicq.KUwarningsq/)Uerrorsq0)U Python_magicq1U* q2u. v((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUentry_0q(U q(U OFS.ImageqUFileq tq tq QUidq UEntriesq U__ac_local_roles__q}qUamosq]qUOwnerqasU_objectsq(}q(U meta_typeqUFileqUidqhutu. w 9((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU index_htmlqU__ac_local_roles__q}q Uamosq ]q UOwnerq asU_textq T The document title or folder title or the example title

The title

This example shows how to create a simple guest book.

The guest book stores its entries as file objects in the Entries folder. This page displays the entries by looping over a sorted list of entries and displaying each one. The list of entries is created by the getEntries script. The formatEntry script formats each guest book entry. To remove an entry, remove its file from the Entries folder. An alternate version of this application could use ZSQL Methods to store guest book entries in a relational database.

The addEntry.html template provides a form that allows you to sign the guest book. The form is processed by the addEntry script. The addEntry script stores the entry as a file. Adding files is protected by the "Add Documents, Images and Files" permission; if you want anonymous users to sign the guest book, you could visit the 'Security' tab of the addEntry script, and give it the "Manager" proxy role.

Sign the guest book

Entries

By Guest Name
2001/12/12 		Entry contents

There are no guest book entries yet.

qU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}qU name_subpathqUtraverse_subpathqssbUtitleqUU content_typeqU text/htmlqu. x((U'Products.PageTemplates.ZopePageTemplateqUZopePageTemplateqtqNt.}q(UexpandqKUidqU addEntry.htmlqU__ac_local_roles__q}q Uamosq ]q UOwnerq asU_textq TU The document title or folder title or the example title

The document title

You can sign the guest book by filling out this form. You cannot include HTML in your comments, however, Zope will preserve your linebreaks.

Your Name
Comments
qU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}qU name_subpathqUtraverse_subpathqssbUtitleqUSign the Guest BookqU content_typeqU text/htmlqu. t;((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (U _getattr_q U_getattrq U containerq Ufilesq UsequenceqtU co_argcountqKubU_paramsqUUidqU getEntriesqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(Uname_containerqU containerqU name_subpathq Utraverse_subpathq!U name_contextq"Ucontextq#U name_m_selfq$Uscriptq%usbU func_defaultsq&NU_codeq'TcsdZdS(sJ Returns a list of entries (file objects) in order from newest to oldest. csZt}d||tddd}|td|dddffSdS( sJ Returns a list of entries (file objects) in order from newest to oldest. sEntriess objectValuessFilessortsbobobase_modification_timescmpsdescN(s _getattr_s_getattrs containersfilesssequence(s _getattr_s_getattrs containersfilesssequence((sScript (Python)s getEntriess !N(s getEntries(s getEntries((sScript (Python)ssq(U_bodyq)T""" Returns a list of entries (file objects) in order from newest to oldest. """ files=container.Entries.objectValues('File') # sort the files based on modification time, with the most recent files first return sequence.sort(files, (('bobobase_modification_time', 'cmp', 'desc'),)) q*U Script_magicq+KUwarningsq,)Uerrorsq-)U Python_magicq.U* q/u. s((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_varnamesq (Uentryq UProducts.PythonScripts.standardq U html_quoteq U newline_to_brq tU co_argcountqKubU_paramsqUentryqUidqU formatEntryqU__ac_local_roles__q}qUamosq]qUOwnerqasU _bind_namesq(cShared.DC.Scripts.Bindings NameAssignments qoq}qU_asgnsq}q(Uname_containerqU containerqU name_contextq Ucontextq!U name_subpathq"Utraverse_subpathq#Uname_nsq$U_U name_m_selfq%Uscriptq&usbU func_defaultsq'NU_codeq(TcsdZdS(Ncs-dkl}l}|||SdS(N(s html_quotes newline_to_br(sProducts.PythonScripts.standards html_quotes newline_to_brsentry(sentrysProducts.PythonScripts.standards html_quotes newline_to_br((sScript (Python)s formatEntrys(s formatEntry(s formatEntry((sScript (Python)ssq)U_bodyq*Unfrom Products.PythonScripts.standard import html_quote, newline_to_br return newline_to_br(html_quote(entry)) q+U Script_magicq,KUwarningsq-)Uerrorsq.)U Python_magicq/U* q0u. u W((U#Products.PythonScripts.PythonScriptqU PythonScriptqtqNt.}q(U func_codeq(cShared.DC.Scripts.Signature FuncCode qoq}q(U co_argcountq KU co_varnamesq (Ucommentsq U guest_nameq UREQUESTq U _getitem_qU_getitemqU _getattr_qU_getattrqU ExceptionqU LimitExceededqU containerqUentries_folderqUlenqUnUidqUgetattrqUentryqUNoneqtubU_paramsqU.comments, guest_name='Anonymous', REQUEST=NoneqUidqUaddEntryqU__ac_local_roles__q}q Uamosq!]q"UOwnerq#asU _bind_namesq$(cShared.DC.Scripts.Bindings NameAssignments q%oq&}q'U_asgnsq(}q)(Uname_containerq*U containerq+U name_subpathq,Utraverse_subpathq-U name_contextq.Ucontextq/U name_m_selfq0Uscriptq1usbU func_defaultsq2(U Anonymousq3Ntq4U_codeq5TcsdtdZdS(s Create a guest book entry. s Anonymouscs:t}t}ddtfdY}|td}  t|| d}  | djo |dn d| } |||| dd d | d d d dd||ddt | | }||dd||ddd|tj o|td|SndS(s Create a guest book entry. s LimitExceededcs dRS(s*raised when too many entries are submitted((((sScript (Python)s LimitExceededssEntriess objectIdsis'limit on number of entries (5) exceededsentry_%dsmanage_addProductsOFSPsmanage_addFilestitless content_types text/plainsfileNismanage_addPropertys guest_namei sstrings index_html(s _getitem_s_getitems _getattr_s_getattrs Exceptions LimitExceededs containersentries_folderslensnsidscommentssgetattrsentrys guest_namesREQUESTsNone(scommentss guest_namesREQUESTs _getitem_s_getitems _getattr_s_getattrs Exceptions LimitExceededs containersentries_folderslensnsidsgetattrsentrysNone((sScript (Python)saddEntrys@(N(sNonesaddEntry(sNonesaddEntry((sScript (Python)ssq6Uerrorsq7)U_bodyq8T<""" Create a guest book entry. """ class LimitExceeded(Exception): """raised when too many entries are submitted""" entries_folder=container.Entries # create up-to 5 unique entry ids, a real app might not impose such a # small limit, but we do because this is just an example after all n=len(entries_folder.objectIds()) if n > 4: raise LimitExceeded, "limit on number of entries (5) exceeded" id='entry_%d' % n # create the entry, limit body to 128 bytes entries_folder.manage_addProduct['OFSP'].manage_addFile(id, title="", content_type="text/plain", file=comments[:128]) # add a guest_name string property, limited to 32 bytes entry=getattr(entries_folder, id) entry.manage_addProperty('guest_name', guest_name[:32], 'string') # return a response page if REQUEST is not None: return container.index_html(REQUEST) q9U Script_magicq:KUwarningsq;)U _proxy_rolesq<)U Python_magicq=U* q>u. q((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUPigeonsqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu. ~o((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUCrowsqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu. }((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUPrimatesqUMonkeysq(U q (hUFolderq tq tq QU__ac_local_roles__q }qUamosq]qUOwnerqasU_objectsq(}q(U meta_typeqUFolderqUidqhutu. |p((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUWhalesqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu. {n((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUBatsqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu. z((U OFS.ImageqUFileqtqNt.}q(UsizeqKUdataqU Example

Example

This is an example HTML file.

qU_EtagSupport__etagqU ts56412592.16q U__name__q U example.htmlq Utitleq UU content_typeq U text/htmlqU preconditionqUU__ac_local_roles__q}qUamosq]qUOwnerqasu. y((U OFS.ImageqUFileqtqNt.}q(UsizeqKUdataqUThis is an example text file.qU_EtagSupport__etagqU ts56412603.29q U__name__q U example.txtq Utitleq UU content_typeq U text/plainqU preconditionqUU__ac_local_roles__q}qUamosq]qUOwnerqasu. ((U OFS.ImageqUFileqtqNt.}q(UsizeqMMUdataqTMThis is an example comment. It can contain multiple lines of text. Note that you cannot include HTML in your comments, since the HTML is quoted. This guest book entry is stored in a file named entry_0 in the Entries folder. To edit the entry, simple edit that file. You can also delete an entry by deleting its file.qU_EtagSupport__etagqU ts05745452.64q U__name__q Uentry_0q U _propertiesq (}q (UidqUtitleqUtypeqUstringqu}q(hU content_typeqhhu}q(hU guest_nameqhhuthU text/plainqhUU preconditionqUU__ac_local_roles__q}qUamosq]qUOwnerqashUAmosqu. q((U OFS.FolderqUFolderqtqNt.}q(UtitleqUUidqUMonkeysqU__ac_local_roles__q}q Uamosq ]q UOwnerq asu.Zope-2.8.7-final/skel/0000775000076400007640000000000010435500331013470 5ustar ajungajungZope-2.8.7-final/skel/import/0000775000076400007640000000000010435500331015002 5ustar ajungajungZope-2.8.7-final/skel/import/README.txt0000775000076400007640000000032310435500331016501 0ustar ajungajungThis directory is used by the running Zope process to import objects into the ZODB. Please place any files that you wish to be able to import into this directy. For more information, please see The Zope Book. Zope-2.8.7-final/skel/Products/0000775000076400007640000000000010435500331015273 5ustar ajungajungZope-2.8.7-final/skel/Products/README.txt0000664000076400007640000000025210435500331016770 0ustar ajungajungAdditional products for your Zope instance should be installed in this directory. A freshly created instance should only contain this README.txt file in this directory. Zope-2.8.7-final/skel/lib/0000775000076400007640000000000010435500331014236 5ustar ajungajungZope-2.8.7-final/skel/lib/python/0000775000076400007640000000000010435500331015557 5ustar ajungajungZope-2.8.7-final/skel/lib/python/README.txt.in0000664000076400007640000000045410435500331017665 0ustar ajungajungThis directory will hold any additional Python packages installed into this Zope instance. Installing Distutils-based packages should be done using the command: <> setup.py install --home <> This will cause the newly installed Python packages to appear in this directory. Zope-2.8.7-final/skel/README.txt0000664000076400007640000000074110435500331015170 0ustar ajungajungThis directory contains an "instance home" for the Zope application server. It contains the following directories: bin/ Scripts used to control the Zope instance etc/ Configuration files Extensions/ Python sources for External Methods log/ Log files lib/python/ Installed Python packages specific to the instance Products/ Installed Zope Products specific to the instance var/ Run-time data files, including the object database Zope-2.8.7-final/skel/etc/0000775000076400007640000000000010435500331014243 5ustar ajungajungZope-2.8.7-final/skel/etc/zope.conf.in0000664000076400007640000006442610435500331016510 0ustar ajungajung############################################################################### # Welcome to Zope 2. ############################################################################### # # This is the Zope configuration file. The Zope configuration file # shows what the default configuration directives are, and show # examples for each directive. To declare a directive, make sure that # you add it to a line that does not begin with '#'. Note that comments # are only allowed at the beginning of a line: you may not add comments # after directive text on the same line. # # Note for Developers # =================== # # This file is *not* auto-generated. If you create a new directive you # very likely want to include an example of how to use the new # directive in this file. # # You shouldn't modify 'zope.conf.in' to change # configuration. Instead, you should make a copy into 'zope.conf' and # modify that to avoid checking in changes to this file by mistake. # ZConfig "defines" used for later textual substitution %define INSTANCE <> %define ZOPE <> # Directive: instancehome # # Description: # The path to the data files, local product files, import directory, # and Extensions directory used by Zope. # # Required (no default) # # Example: # # instancehome /home/chrism/projects/sessions instancehome $INSTANCE # Directive: clienthome # # Description: # The directory in which a running Zope's process identifier files are # placed. # # Default: $INSTANCE/var # # Example: # # clienthome /home/chrism/projects/sessions/var # Directive: path # # Description: # Name of a directory which should be inserted into the # the beginning of Python's module search path. This directive # may be specified as many times as needed to insert additional # directories. The set of directories specified is inserted into the # beginning of the module search path in the order which they are specified # here. Note that the processing of this directive may happen too late # under some circumstances; it is recommended that you use the PYTHONPATH # environment variable if using this directive doesn't work for you. # # Default: $INSTANCE/lib/python # # Example: # # path $INSTANCE/mypymodules # Directive: products # # Description: # Name of a directory that contains additional Product packages. This # directive may be used as many times as needed to add additional # collections of products. Each directory identified will be # added to the __path__ of the Products package. All Products are # initialized in ascending alphabetical order by product name. If # two products with the same name exist in two Products directories, # the order in which the packages appear here defines the load # order. The master Products directory exists in Zope's software home, # and cannot be removed from the products path (and should not be added # to it here). # # Default: $INSTANCE/Products # # Example: # # products /home/chrism/projects/myproducts # Directive: environment # # Description: # A section which can be used to define arbitrary key-value pairs # for use as environment variables during Zope's run cycle. It # is not recommended to set system-related environment variables such as # PYTHONPATH within this section. # # Default: unset # # Example: # # # MY_PRODUCT_ENVVAR foobar # # Directive: debug-mode # # Description: # A switch which controls several aspects of Zope operation useful for # developing under Zope. When debug mode is on: # # - The process will not detach from the controlling terminal # # - Errors in product initialization will cause startup to fail # (instead of writing error messages to the event log file). # # - Filesystem-based scripts such as skins, PageTemplateFiles, and # DTMLFiles can be edited while the server is running and the server # will detect these changes in real time. When this switch is # off, you must restart the server to see the changes. # # Setting this to 'off' when Zope is in a production environment is # encouraged, as it speeds execution (sometimes dramatically). # # Default: off # # Example: # # debug-mode on # Directive: effective-user # # Description: # If you intend to run Zope as the "root" user, you must supply this # directive with an effective username or userid number to which Zope # will 'suid' after the server ports are bound. This directive only # has effect under UNIX and if Zope is started as the root user. # # Default: unset # # Example: # # effective-user chrism # Directive: enable-product-installation # # Description: # If this directive is turned on, Zope performs 'product installation' # (the registration of Python modules in various Products directories) # at startup. Turning this off can speed Zope/ZEO startup time, # but it can also cause your Control_Panel Product list to become # desynchronized with the contents of your Products # directories. NOTE: Zope *must* be started at least once with # this directive set to "on" or you will receive an error. If using ZEO, # at least one ZEO client must be run with this directive set to "on" # once, the others can have it turned off. # NOTE: If your main storage is mounted read-only, # you must set this directive to "off". # # Default: on # # Example: # # enable-product-installation off # Directive: locale # # Description: # Enable locale (internationalization) support by supplying a locale # name to be used. See your operating system documentation for locale # information specific to your system. If your Python module does not # support the locale module, or if the requested locale is not # supported by your system, an error will be raised and Zope will not # start. # # Default: unset # # Example: # # locale fr_FR # Directive: datetime-format # # Description: # Set this variable either to "us" or "international" to force the # DateTime module to parse date strings either with # month-before-days-before-year ("us") or # days-before-month-before-year ("international"). The default # behaviour of DateTime (when this setting is left unset) is to # parse dates as US dates. # # Default: us # # Example: # # datetime-format international # Directive: zserver-threads # # Description: # Specify the number of threads that Zope's ZServer web server will use # to service requests. The default is 4. # # Default: 4 # # Example: # # zserver-threads 10 # Directive: python-check-interval # # Description: # Specify an integer representing the Python interpreter "check # interval" This interval determines how often the interpreter checks # for periodic things such as thread switches and signal handlers. The # Zope default is 500, but you may want to experiment with other values # in order to attempt to increae performance in your particular # environment. # # Default: 500 # # Example: # # python-check-interval 1000 # Directive: zserver-read-only-mode # # Description: # If this directive is set to 'on', it will cause Zope to inhibit the # creation of log files and pid files. Access and event log files will # be presented on standard output. Setting this directive 'on' causes # pcgi, fastcgi, and daemon-related directives to have no effect. # # Default: off # # Example: # # zserver-read-only-mode on # Directive: pid-filename # # Description: # The path to the file in which the Zope process id(s) will be written. # This defaults to client-home/Z2.pid. # # Default: CLIENT_HOME/Z2.pid # # Example: # # pid-filename /home/chrism/projects/sessions/var/Z2.pid # Directive: lock-filename # # Description: # The path to a "lock file" which will be locked by Zope while it's # running. This file is used by zopectl.py to determine if Zope is # currently running. This defaults to CLIENT_HOME/Z2.lock. # # Default: CLIENT_HOME/Z2.lock # # Example: # # lock-filename /home/chrism/projects/sessions/var/Z2.lock # Directive: mime-types # # Description: # Tells Zope about additional mime.types files that should be # loaded. The files have the same format as the mime.types file # distributed with Apache. The "mime-types" setting may be given # more than once in the configuration file. # # Example: # # mime-types $INSTANCE/etc/mime.types # Directive: structured-text-header-level # # Description: # Set the default starting HTML header level for structured text # documents. The default is 3, which implies that top-level headers # will be created with an

tag. # # Default: 3 # # Example: # # structured-text-header-level 1 # Directive: rest-input-encoding # # Description: # Specifies the input encoding of re-StructuredText documents # (e.g. 'utf-8', 'iso-8859-15' or any other valid encoding recognized # by Python). The default is your Python's default encoding. # # Default: unset (uses system default) # # Example: # # rest-input-encoding iso-8859-15 # Directive: rest-output-encoding # # Description: # Specifies the output encoding of re-StructuredText documents # (e.g. 'utf-8', 'iso-8859-15' or any other valid encoding recognized # by Python). The default is your Python's default encoding. # # Default: unset (uses system default) # # Example: # # rest-output-encoding iso-8859-15 # Directive: rest-header-level # # Description: # Set the default starting HTML header level for restructured text # documents. The default is 3, which implies that top-level headers # will be created with an

tag. # # Default: 3 # # Example: # # rest-header-level 2 # Directive: rest-language-code # # Description: # Language code used for some internal translations inside of the docutils # package and for DTD bibliographic elements mapping. See # lib/python/docutils/languages/ for a list of supported language codes. # # Default: en # # Example: # # rest-language-code de # Directive: cgi-environment # # Description: # A section which allows a user to define arbitrary key-value pairs for # use as the initial CGI environment variables. This is useful # when you want to proxy requests from another web server to Zserver, # and would like Zserver's CGI environment to reflect the CGI # environment of the other web server. # # Default: unset # # Example: # # # HTTPS_SERVER Foobar Server 1.0 # HTTPS_PORT 443 # # Directive: dns-server # # Description: # Specify the IP address of your DNS server in order to cause resolved # hostnames to be written to Zope's access log. By default, Zope will # not resolve hostnames unless this is set. # # Default: unset # # Example: # # dns-server 127.0.0.1 # Directive: ip-address # # Description: # The default IP address on which Zope's various server protocol # implementations will listen for requests. If this is unset, Zope # will listen on all IP addresses supported by the machine. This # directive can be overridden on a per-server basis in the servers # section. # # Default: unset # # Example: # # ip-address 127.0.0.1 # Directive: http-realm # # Description: # The HTTP "Realm" header value sent by this Zope instance. This value # often shows up in basic authentication dialogs. # # Default: Zope # # Example: # # http-realm Slipknot # Directive: cgi-maxlen # # Description: # Set this value to limit the amount of form data being processed # by Zope to prevent DoS attacks. # # Default: 0 (= no restrictions) # # Example: # # cgi-maxlen 10000 # Directive: http-header-max-length # # Description: # Maximum number of bytes allowed within a HTTP request header. The request # is discarded and considered as a DoS attack if the header size exceeds # this limit. # # Default: 8192 # # Example: # # http-header-max-length 16384 # Directive: automatically-quote-dtml-request-data # # Description: # Set this directive to 'off' in order to disable the autoquoting of # implicitly retrieved REQUEST data by DTML code which contains a '<' # when used in construction. When this directive is 'on', # all data implicitly retrieved from the REQUEST in DTML (as opposed to # addressing REQUEST.somevarname directly) that contains a '<' will be # HTML-quoted when interpolated via a or &dtml- construct. This # mitigates the possibility that DTML programmers will leave their # sites open to a "client-side trojan" attack. # # Default: on # # Example: # # automatically-quote-dtml-request-data on # Directive: trusted-proxy # # Description: # Define one or more 'trusted-proxies' directives, each of which is a # hostname or an IP address. The set of definitions comprises a list # of front-end proxies that are trusted to supply an accurate # X-Forwarded-For header to Zope. If a connection comes from # a trusted proxy, Zope will trust any X-Forwarded header to contain # the user's real IP address for the purposes of address-based # authentication restriction. # # Default: unset # # Example: # # trusted-proxy www.example.com # trusted-proxy 192.168.1.1 # Directive: publisher-profile-file # # Description: # Names a file on the filesystem which causes Zope's Python # profiling capabilities to be enabled. For more information, see # the Debug Information - > Profiling tab of Zope's Control_Panel # via the Zope Management Interface. IMPORTANT: setting this # filename will cause Zope code to be executed much more slowly # than normal. This should not be enabled in production. # # Default: unset # # Example: # # publisher-profile-file $INSTANCE/var/profile.dat # Directive: security-policy-implementation # # Description: # The default Zope security machinery is implemented in C. Change # this to "python" to use the Python version of the Zope security # machinery. This setting may impact performance but is useful # for debugging purposes. See also the "verbose-security" option # below. # # Default: C # # Example: # # security-policy-implementation python # Directive: skip-authentication-checking # # Description: # Set this directive to 'on' to cause Zope to skip checks related # to authentication, for servers which serve only anonymous content. # Only works if security-policy-implementation is 'C'. # # Default: off # # Example: # # skip-authentication-checking on # Directive: skip-ownership-checking # # Description: # Set this directive to 'on' to cause Zope to ignore ownership checking # when attempting to execute "through the web" code. By default, this # directive is on in order to prevent 'trojan horse' security problems # whereby a user with less privilege can cause a user with more # privilege to execute dangerous code. # # Default: off # # Example: # # skip-ownership-checking on # Directive: verbose-security # # Description: # By default, Zope reports authorization failures in a terse manner in # order to avoid revealing unnecessary information. This option # modifies the Zope security policy to report more information about # the reason for authorization failures. It's designed for debugging. # If you enable this option, you must also set the # 'security-policy-implementation' to 'python'. # # Default: off # # Example: # # security-policy-implementation python # verbose-security on # Directive: maximum-number-of-session-objects # # Description: # An integer value representing the number of items to use as a # "maximum number of subobjects" value of the # '/temp_folder/session_data' transient object container. # # Default: 1000 # # Example: # # maximum-number-of-session-objects 10000 # Directive: session-add-notify-script-path # # Description: # An optional fill Zope path name of a callable object to be set as the # "script to call on object addition" of the sessioN_data transient # object container created in the /temp_folder folder at startup. # # Default: unset # # Example: # # session-add-notify-script-path /scripts/add_notifier # Directive: session-delete-notify-script-path # # Description: # An optional fill Zope path name of a callable object to be set as the # "script to call on object deletion" of the sessioN_data transient # object container created in the /temp_folder folder at startup. # # Default: unset # # Example: # # session-delete-notify-script-path /scripts/del_notifier # Directive: session-timeout-minutes # # Description: # An integer value representing the number of minutes to be used as the # "data object timeout" of the '/temp_folder/session_data' transient # object container. # # Default: 20 # # Example: # # session-timeout-minutes 30 # Directive: session-resolution-seconds # # Description: # An integer value representing the number of seconds to be used as the # "timeout resolution" of the '/temp_folder/session_data' transient # object container. # # Default: 20 # # Example: # # session-resolution-seconds 60 # Directive: suppress-all-access-rules # # Description: # If this directive is set to on, no access rules in your Zope site # will be executed. This is useful if you "lock yourself out" of a # particular part of your site by setting an improper access rule. # # Default: off # # Example: # # suppress-all-access-rules on # Directive: suppress-all-site-roots # # Description: # If this directive is set to on, no site roots in your Zope site will # be effective. This is useful if you "lock yourself out" of a # particular part of your site by setting an improper site root. # # Default: off # # Example: # # suppress-all-site-roots on # Directive: database-quota-size # # Description: # Set this directive to an integer in bytes in order to place a hard # limit on the size which the default FileStorage-backed Zope database # can grow. Additions to the database will not be permitted once this # filesize is exceeded. # # Default: unset # # Example: # # database-quota-size 1000000 # Directive: read-only-database # # Description: # This causes the main Zope FileStorage-backed ZODB to be opened in # read-only mode. # # Default: off # # Example: # # read-only-database on # Directive: zeo-client-name # # Description: # If you want a persistent ZEO client cache which retains cache # contents across ClientStorage restarts, you need to define a # zeo-client-name. If you use ZEO and you don't set a # zeo-client-name, the client cache is stored in temporary files # which are removed when the ClientStorage shuts down. The value # of zeo-client-name is used to uniquely identify the local cache # files created if this Zope is a ZEO client. # # Default: unset # # Example: # # zeo-client-name zeo1 # Directives: logger # # Description: # This area should define one or more "logger" sections of the # names "access", "event", and "trace". The "access" logger logs # Zope server access. The "event" logger logs Zope event # information. The "trace" logger logs detailed server request # information (for debugging purposes only). Each logger section # may contain a "level" name/value pair which indicates the level # of logging detail to capture for this logger. The default level # is INFO. Level may be any of "CRITICAL", 'ERROR", WARN", "INFO", # "DEBUG", and "ALL". Each logger section may additionally contain # one or more "handler" sections which indicates a types of log # "handlers" (file, syslog, NT event log, etc) to be used for the # logger being defined. There are 5 types of handlers: logfile, # syslog, win32-eventlog, http-handler, email-notifier. Each # handler type has its own set of allowable subkeys which define # aspects of the handler. All handler sections also allow for the # specification of a "format" (the log message format string), a # "dateformat" (the log message format for date strings), and a # "level", which has the same semantics of the overall logger # level but overrides the logger's level for the handler it's # defined upon. XXXX much more detail necessary here # # Default: # # The access log will log to the file /log/Z2.log at # level INFO, the event log will log to the file # /log/event.log at level INFO, and the trace log # will not be written anywhere. level info path $INSTANCE/log/event.log level info level WARN path $INSTANCE/log/Z2.log format %(message)s # # level WARN # # path $INSTANCE/log/trace.log # format %(message)s # # # Directive: warnfilter # # Description: # A section that allows you to define a warning filter. # The following keys are valid within a warnfilter section: # # action: one of the following strings: # # "error" turn matching warnings into exceptions # "ignore" never print matching warnings # "always" always print matching warnings # "default" print the first occurrence of matching warnings # for each location where the warning is issued # "module" print the first occurrence of matching warnings # for each module where the warning is issued # "once" print only the first occurrence of matching # warnings, regardless of location # # message: a string containing a regular expression that the # warning message must match (the match is compiled to # always be case-insensitive) # # category: a Python dotted-path classname (must be a subclass of # Warning) of which the warning category must be a subclass in # order to match # # module: a string containing a regular expression that the # module name must match (the match is compiled to be # case-sensitive) # # lineno: an integer that the line number where the warning # occurred must match, or 0 to match all line numbers # # All keys within a warnfilter section are optional. More than # one warnfilter section may be specified. # # Default: unset # # Example: # # # action ignore # category exceptions.DeprecationWarning # # Directive: max-listen-sockets # # Description: # The maximum number of sockets that ZServer will attempt to open # in order to service incoming connections. # # Default: 1000 # # Example: # # max-listen-sockets 500 # Directives: port-base # # Description: # Offset applied to the port numbers used for ZServer # configurations. For example, if the http-server port is 8080 and # the port-base is 1000, the HTTP server will listen on port 9080. # This makes it easy to change the complete set of ports used by a # Zope server process # # Default: # # 0 # # Example: # # port-base 1000 # Directive: large-file-threshold # # Description: # Requests bigger than this size get saved into a temporary file # instead of being read completely into memory. # # Default: 512K # # Example: # # large-file-threshold 1Mb # Directive: default-zpublisher-encoding # # Description: # This controls what character set is used to encode unicode # data that reaches ZPublisher without any other specified encoding. # # Default: iso-8859-15 # # Example: # # default-zpublisher-encoding utf-8 # Directives: servers # # Description: # A set of sections which allow the specification of Zope's various # ZServer servers. 7 different server types may be defined: # http-server, ftp-server, webdav-source-server, persistent-cgi, # fast-cgi, monitor-server, and icp-server. If no servers are # defined, the default servers are used. # # Ports may be specified using the 'address' directive either in simple # form (80) or in complex form including hostname 127.0.0.1:80. If the # hostname is "left off", the default-ip-address is used as the hostname. # # Port numbers are offset by the setting of port-base. # # To run the monitor-server an emergency user must be defined # (through an 'access' file). # # Default: # # An HTTP server starts on port 8080. # valid keys are "address" and "force-connection-close" address 8080 # force-connection-close on # Examples: # # # # valid key is "address" # address 8021 # # # # # valid keys are "address" and "force-connection-close" # address 1980 # force-connection-close off # # # # # valid key is "path" # path somefile # # # # # valid key is "address"; the address may be hostname:port, port, # # or a path for a Unix-domain socket # address somefile # # # # # valid keys are "address" # address 8099 # # # # # valid key is "address" # address 888 # # Database (zodb_db) section # # Description: # A database section allows the definition of custom database and # storage types. More than one zodb_db section can be defined. # # Default: unset. # IMPORTANT: At least one database with a mount-point of "/" # must be specified for Zope to start properly. # Main FileStorage database path $INSTANCE/var/Data.fs mount-point / # Temporary storage database (for sessions) name temporary storage for sessioning mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer # Other storage examples # # ZEO client storage: # # # mount-point / # # ZODB cache, in number of objects # cache-size 5000 # # server localhost:8100 # storage 1 # name zeostorage # var $INSTANCE/var # # ZEO client cache, in bytes # cache-size 20MB # # Uncomment to have a persistent disk cache # #client zeo1 # # Zope-2.8.7-final/skel/Extensions/0000775000076400007640000000000010435500331015627 5ustar ajungajungZope-2.8.7-final/skel/Extensions/README.txt0000664000076400007640000000026410435500331017327 0ustar ajungajungFiles containing Python source code for External Method should be placed in this directory. A freshly created instance should only contain this README.txt file in this directory. Zope-2.8.7-final/skel/bin/0000775000076400007640000000000010435500331014240 5ustar ajungajungZope-2.8.7-final/skel/bin/zopectl.in0000775000076400007640000000052710435500331016257 0ustar ajungajung#! /bin/sh PYTHON="<>" ZOPE_HOME="<>" INSTANCE_HOME="<>" CONFIG_FILE="<>/etc/zope.conf" SOFTWARE_HOME="<>" PYTHONPATH="$SOFTWARE_HOME" export PYTHONPATH INSTANCE_HOME SOFTWARE_HOME ZDCTL="$SOFTWARE_HOME/Zope2/Startup/zopectl.py" exec "$PYTHON" "$ZDCTL" -C "$CONFIG_FILE" "$@" Zope-2.8.7-final/skel/bin/runzope.bat.in0000664000076400007640000000050210435500331017034 0ustar ajungajung@set PYTHON=<> @set ZOPE_HOME=<> @set INSTANCE_HOME=<> @set SOFTWARE_HOME=<> @set CONFIG_FILE=<>\etc\zope.conf @set PYTHONPATH=%SOFTWARE_HOME% @set ZOPE_RUN=%SOFTWARE_HOME%\Zope2\Startup\run.py "%PYTHON%" "%ZOPE_RUN%" -C "%CONFIG_FILE%" %1 %2 %3 %4 %5 %6 %7 Zope-2.8.7-final/skel/bin/runzope.in0000775000076400007640000000053110435500331016274 0ustar ajungajung#! /bin/sh PYTHON="<>" ZOPE_HOME="<>" INSTANCE_HOME="<>" CONFIG_FILE="<>/etc/zope.conf" SOFTWARE_HOME="<>" PYTHONPATH="$SOFTWARE_HOME" export PYTHONPATH INSTANCE_HOME SOFTWARE_HOME ZOPE_RUN="$SOFTWARE_HOME/Zope2/Startup/run.py" exec "$PYTHON" "$ZOPE_RUN" -C "$CONFIG_FILE" "$@" Zope-2.8.7-final/skel/bin/zopeservice.py.in0000664000076400007640000001102710435500331017556 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ A Zope Windows NT service frontend. Usage: Installation The Zope service should be installed by the Zope Windows installer. You can manually install, uninstall the service from the commandline. ntservice.py [options] install|update|remove|start [...] |stop|restart [...]|debug [...] Options for 'install' and 'update' commands only: --username domain\username : The Username the service is to run under --password password : The password for the username --startup [manual|auto|disabled] : How the service starts, default = manual Commands install : Installs the service update : Updates the service. Use this if you change any configuration settings and need the service to be re-registered. remove : Removes the service start : Starts the service, this can also be done from the services control panel stop : Stops the service, this can also be done from the services control panel restart : Restarts the service debug : Runs the service in debug mode You can view the usage options by running this module without any arguments. Starting Zope Start Zope by clicking the 'start' button in the services control panel. You can set Zope to automatically start at boot time by choosing 'Auto' startup by clicking the 'statup' button. Stopping Zope Stop Zope by clicking the 'stop' button in the services control panel. You can also stop Zope through the web by going to the Zope control panel and by clicking 'Shutdown'. Event logging Service related events (such as startup, shutdown, or errors executing the Zope process) are logged to the NT application event log. Use the event viewer to see these events. Zope Events are still written to the Zope event logs. """ import sys, os # these are replacements from mkzopeinstance PYTHON = r'<>' SOFTWARE_HOME=r'<>' INSTANCE_HOME = r'<>' ZOPE_HOME = r'<>' ZOPE_RUN = r'%s\Zope2\Startup\run.py' % SOFTWARE_HOME CONFIG_FILE= os.path.join(INSTANCE_HOME, 'etc', 'zope.conf') PYTHONSERVICE_EXE=r'%s\bin\PythonService.exe' % ZOPE_HOME # Setup the environment, so sub-processes see these variables for check_dir in (os.path.join(SOFTWARE_HOME, 'third_party', 'docutils', 'extras'), os.path.join(SOFTWARE_HOME, 'third_party', 'docutils'), SOFTWARE_HOME, ): parts = os.environ.get("PYTHONPATH", "").split(os.pathsep) if check_dir not in parts: parts = filter(None, [check_dir] + parts) os.environ["PYTHONPATH"] = os.pathsep.join(parts) os.environ["INSTANCE_HOME"] = INSTANCE_HOME # Ensure SOFTWARE_HOME is on our current sys.path so we can import the # nt_svcutils package. Note we don't need the docutils dirs in sys.path, as # only Zope itself (our child process) uses it, and that happens via # PYTHONPATH if SOFTWARE_HOME not in sys.path: sys.path.insert(0, SOFTWARE_HOME) from nt_svcutils.service import Service servicename = 'Zope_%s' % str(hash(INSTANCE_HOME.lower())) class InstanceService(Service): _svc_name_ = servicename _svc_display_name_ = 'Zope instance at %s' % INSTANCE_HOME # _svc_description_ can also be set (but what to say isn't clear!) # If the exe we expect is not there, let the service framework search # for it. This will be true for people running from source builds and # relying on pre-installed pythonservice.exe. # Note this is only used at install time, not runtime. if os.path.isfile(PYTHONSERVICE_EXE): _exe_name_ = PYTHONSERVICE_EXE process_runner = PYTHON process_args = '"%s" -C "%s"' % (ZOPE_RUN, CONFIG_FILE) if __name__ == '__main__': import win32serviceutil win32serviceutil.HandleCommandLine(InstanceService) Zope-2.8.7-final/skel/log/0000775000076400007640000000000010435500331014251 5ustar ajungajungZope-2.8.7-final/skel/log/README.txt0000664000076400007640000000007110435500331015745 0ustar ajungajungThis is the directory used to hold log files by default. Zope-2.8.7-final/skel/var/0000775000076400007640000000000010435500331014260 5ustar ajungajungZope-2.8.7-final/skel/var/README.txt0000664000076400007640000000045310435500331015760 0ustar ajungajungThis directory contains files created and maintained by the Zope application server and related utilities while the server is running. This can include the object database as well as supplemental files (such as "pidfiles"). Log files are normally stored in the log/ directory in the instance home. Zope-2.8.7-final/lib/0000775000076400007640000000000010435500331013300 5ustar ajungajungZope-2.8.7-final/lib/python/0000775000076400007640000000000010435610645014633 5ustar ajungajungZope-2.8.7-final/lib/python/ComputedAttribute/0000775000076400007640000000000010435610641020273 5ustar ajungajungZope-2.8.7-final/lib/python/ComputedAttribute/__init__.py0000664000076400007640000000004110435500325022374 0ustar ajungajungfrom _ComputedAttribute import * Zope-2.8.7-final/lib/python/ComputedAttribute/setup.py0000664000076400007640000000170210435500325022002 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from distutils.core import setup, Extension setup(name="_ComputedAttribute", version="2.0", ext_modules=[ Extension("_ComputedAttribute", ["_ComputedAttribute.c"], include_dirs = ['.', '../ExtensionClass'], depends = ['../ExtensionClass/ExtensionClass.h']), ]) Zope-2.8.7-final/lib/python/ComputedAttribute/tests.py0000664000076400007640000000417610435500325022014 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Computed Attributes Computed attributes work much like properties: >>> import math >>> class Point(Base): ... def __init__(self, x, y): ... self.x, self.y = x, y ... length = ComputedAttribute(lambda self: math.sqrt(self.x**2+self.y**2)) >>> p = Point(3, 4) >>> "%.1f" % p.length '5.0' Except that you can also use computed attributes with instances: >>> p.angle = ComputedAttribute(lambda self: math.atan(self.y*1.0/self.x)) >>> "%.2f" % p.angle '0.93' $Id: tests.py 29895 2005-04-07 03:49:30Z tim_one $ """ def test_wrapper_support(): """Wrapper support To support acquisition wrappers, computed attributes have a level. The computation is only done when the level is zero. Retrieving a computed attribute with a level > 0 returns a computed attribute with a decremented level. >>> class X(Base): ... pass >>> x = X() >>> x.n = 1 >>> x.n2 = ComputedAttribute(lambda self: self.n*2) >>> x.n2 2 >>> x.n2.__class__.__name__ 'int' >>> x.n2 = ComputedAttribute(lambda self: self.n*2, 2) >>> x.n2.__class__.__name__ 'ComputedAttribute' >>> x.n2 = x.n2 >>> x.n2.__class__.__name__ 'ComputedAttribute' >>> x.n2 = x.n2 >>> x.n2.__class__.__name__ 'int' """ import unittest from zope.testing.doctest import DocTestSuite from ExtensionClass import Base from ComputedAttribute import ComputedAttribute def test_suite(): return unittest.TestSuite((DocTestSuite(),)) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/ComputedAttribute/_ComputedAttribute.c0000664000076400007640000000550610435500325024245 0ustar ajungajung/***************************************************************************** Copyright (c) 1996-2003 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "ExtensionClass.h" #define UNLESS(E) if(!(E)) #define OBJECT(O) ((PyObject*)(O)) typedef struct { PyObject_HEAD PyObject *callable; int level; } CA; static PyObject * CA__init__(CA *self, PyObject *args) { PyObject *callable; int level=0; UNLESS(PyArg_ParseTuple(args,"O|i",&callable, &level)) return NULL; if (level > 0) { callable=PyObject_CallFunction(OBJECT(self->ob_type), "Oi", callable, level-1); UNLESS (callable) return NULL; self->level=level; } else { Py_INCREF(callable); self->level=0; } self->callable=callable; Py_INCREF(Py_None); return Py_None; } static void CA_dealloc(CA *self) { Py_DECREF(self->callable); Py_DECREF(self->ob_type); PyMem_DEL(self); } static PyObject * CA_of(CA *self, PyObject *args) { if (self->level > 0) { Py_INCREF(self->callable); return self->callable; } if (PyString_Check(self->callable)) { /* Special case string as simple alias. */ PyObject *o; UNLESS (PyArg_ParseTuple(args,"O", &o)) return NULL; return PyObject_GetAttr(o, self->callable); } return PyObject_CallObject(self->callable, args); } static struct PyMethodDef CA_methods[] = { {"__init__",(PyCFunction)CA__init__, METH_VARARGS, ""}, {"__of__", (PyCFunction)CA_of, METH_VARARGS, ""}, {NULL, NULL} /* sentinel */ }; static PyExtensionClass ComputedAttributeType = { PyObject_HEAD_INIT(NULL) 0, "ComputedAttribute", sizeof(CA), 0, (destructor)CA_dealloc, 0,0,0,0,0, 0,0,0, 0,0,0,0,0, 0,0, "ComputedAttribute(callable) -- Create a computed attribute", METHOD_CHAIN(CA_methods), (void*)(EXTENSIONCLASS_BINDABLE_FLAG) }; static struct PyMethodDef methods[] = { {NULL, NULL} }; void init_ComputedAttribute(void) { PyObject *m, *d; UNLESS(ExtensionClassImported) return; /* Create the module and add the functions */ m = Py_InitModule4("_ComputedAttribute", methods, "Provide Computed Attributes\n\n" "$Id: _ComputedAttribute.c 24763 2004-05-17 05:59:28Z philikon $\n", OBJECT(NULL),PYTHON_API_VERSION); d = PyModule_GetDict(m); PyExtensionClass_Export(d,"ComputedAttribute",ComputedAttributeType); } Zope-2.8.7-final/lib/python/DBTab/0000775000076400007640000000000010435610641015543 5ustar ajungajungZope-2.8.7-final/lib/python/DBTab/Exceptions.py0000664000076400007640000000165310435500330020234 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """DBTab exception classes. $Id: Exceptions.py 24763 2004-05-17 05:59:28Z philikon $ """ class DBTabConfigurationError (Exception): """Error in dbtab configuration""" args = () class DBTabOverrideError (Exception): """DBTab has taken over some piece of functionality""" args = () Zope-2.8.7-final/lib/python/DBTab/CHANGES.txt0000664000076400007640000000443210435500330017350 0ustar ajungajungHEAD - Merged into Zope 2.7+. Split into two pieces: the ZODBMountPoint Product and the DBTab package. Neither custom_zodb.py nor dbtab.conf is now required (all configuration is performed via zope.conf). Version 1.2.1 - Began unit tests. - Fixed a race condition on connection close. The symptom was spurious "Should not load state when connection closed" errors under high load. - DemoStorage configurations can now include a base_type option, taking advantage of DemoStorage's layering feature. - The mount status page (visible by selecting "Add DBTab Mount Point") sometimes indicated there were objects mounted that really were not. Fixed. Version 1.2 - Fixed activity monitoring for mounted databases. - Removed import of AdaptableStorage. Argument converters now work when you specify the full module of a storage class. - You can now specify a container_class to generate folderish objects other than standard folders when mounting a new database. See dbtab.conf.in. Version 1.1 - Changed DBTab's mounting strategy so that mounted connections stay bound to a root connection. This change is designed to: - eliminate issues with volatile attributes in application code that cross mount boundaries. Now it's safe to use cross-database volatile attributes. - eliminate the global registry of open connections, which seemed to have a rare race condition (ugh!) - go faster. :-) The mount point traversal penalty is much lower now, since the mount point can keep a long-lived reference to the mounted object. Version 1.0.2 - Updated to work with the latest BDBStorage and AdaptableStorage. Version 1.0.1 - Deferred startup until after MainConfiguration has been imported. Needed for ZRS. - Added AdaptableStorage and BerkeleyStorage to the list of easily-configured storage types. - Fixed bug reported by Robert Boulanger: If the Storage/Databasename is the same like the mountpath it is not possible to access the database management screens in the Control Panel. Instead getting Admin screens for Cache and Activity you will be redirected to the manage workspace of the folder. - Arranged for Zope to properly close open database connections on clean shutdown. Zope-2.8.7-final/lib/python/DBTab/__init__.py0000664000076400007640000000200310435500330017640 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """DBTab product. $Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $ """ # importing ThreadedAsync has the side effect of patching asyncore so # that loop callbacks get invoked. You need this to # mount a ZEO client connection if the main database is not a ZEO client. # Otherwise ZEO never receives the message telling it to start using the # main async loop. import ThreadedAsync Zope-2.8.7-final/lib/python/DBTab/ClassFactories.py0000664000076400007640000000521010435500330021011 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Available ZODB class factories. $Id: ClassFactories.py 25206 2004-06-02 22:44:36Z jim $""" import OFS.Uninstalled class_factories = {} def minimalClassFactory(jar, module, name, _silly=('__doc__',), _globals={}, ): """Minimal class factory. If any class is not found, this class factory will propagate the exception to the application, unlike the other class factories. """ m = __import__(module, _globals, _globals, _silly) return getattr(m, name) class_factories['minimal'] = minimalClassFactory def simpleClassFactory(jar, module, name, _silly=('__doc__',), _globals={}, ): """Class factory without ZClass support. """ try: m = __import__(module, _globals, _globals, _silly) return getattr(m, name) except: return OFS.Uninstalled.Broken(jar, None, (module, name)) class_factories['simple'] = simpleClassFactory def zopeClassFactory(jar, module, name, _silly=('__doc__',), _globals={}, ): """Class factory with ZClass support. """ try: if module[:1]=='*': # ZCLass! Yee ha! return jar.root()['ZGlobals'][module] else: m=__import__(module, _globals, _globals, _silly) return getattr(m, name) except: return OFS.Uninstalled.Broken(jar, None, (module, name)) class_factories['zope'] = zopeClassFactory def autoClassFactory(jar, module, name): """Class factory with ZClasses and support for central class definitions. """ # If not the root connection, use the class factory from # the root database, otherwise use the Zope class factory. root_conn = getattr(jar, '_root_connection', None) root_db = getattr(root_conn, '_db', None) if root_db is not None: return root_db.classFactory(root_conn, module, name) else: return zopeClassFactory(jar, module, name) class_factories['auto'] = autoClassFactory Zope-2.8.7-final/lib/python/DBTab/DBTab.py0000664000076400007640000001002310435500330017016 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """DBTab and DatabaseFactory classes. $Id: DBTab.py 24763 2004-05-17 05:59:28Z philikon $ """ import sys from thread import allocate_lock from ZODB.ActivityMonitor import ActivityMonitor import Globals from Exceptions import DBTabConfigurationError class DBTab: """A Zope database configuration, similar in purpose to /etc/fstab. """ def __init__(self, db_factories, mount_paths): self._started = 0 self.opened = {} # { name -> Database instance } self.lock = allocate_lock() self.db_factories = db_factories # { name -> DatabaseFactory } self.mount_paths = mount_paths # { virtual path -> name } def startup(self): """Opens the databases set to open_at_startup.""" if self._started: return self._started = 1 for name, factory in self.db_factories.items(): if factory.getOpenAtStartup(): self.getDatabase(name=name) def listMountPaths(self): """Returns a sequence of (virtual_mount_path, database_name). """ return self.mount_paths.items() def listDatabaseNames(self): """Returns a sequence of names. """ return self.db_factories.keys() def hasDatabase(self, name): """Returns true if name is the name of a configured database.""" return self.db_factories.has_key(name) def _mountPathError(self, mount_path): if mount_path == '/': raise DBTabConfigurationError( "No root database configured") else: raise DBTabConfigurationError( "No database configured for mount point at %s" % mount_path) def getDatabase(self, mount_path=None, name=None, is_root=0): """Returns an opened database. Requires either mount_path or name. """ self.startup() if name is None: if mount_path is None: raise ValueError('Either mount_path or name is required') name = self.mount_paths.get(mount_path) if name is None: self._mountPathError(mount_path) db = self.opened.get(name) if db is None: if not self.db_factories.has_key(name): raise KeyError('%s is not a configured database' % repr(name)) self.lock.acquire() try: # Check again, since the database may have been created # by another thread before the lock was acquired. db = self.opened.get(name) if db is None: db = self._createDatabase(name, is_root) finally: self.lock.release() return db def getDatabaseFactory(self, mount_path=None, name=None): if name is None: if mount_path is None: raise ValueError('Either mount_path or name is required') name = self.mount_paths.get(mount_path) if name is None: self._mountPathError(mount_path) return self.db_factories[name] def _createDatabase(self, name, is_root): factory = self.db_factories[name] db = factory.open() self.opened[name] = db if not is_root: Globals.opened.append(db) # If it is the root database, Zope will add the database to # Globals.opened. A database should not be listed twice. return db Zope-2.8.7-final/lib/python/Acquisition/0000775000076400007640000000000010435610641017117 5ustar ajungajungZope-2.8.7-final/lib/python/Acquisition/__init__.py0000664000076400007640000000003310435500317021222 0ustar ajungajungfrom _Acquisition import * Zope-2.8.7-final/lib/python/Acquisition/setup.py0000664000076400007640000000173710435500317020637 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from distutils.core import setup, Extension setup(name="_Acquisition", version="2.0", ext_modules=[ Extension("_Acquisition", ["_Acquisition.c"], include_dirs = ['.', '../ExtensionClass'], depends = ["Acquisition.h", '../ExtensionClass/ExtensionClass.h']), ]) Zope-2.8.7-final/lib/python/Acquisition/tests.py0000664000076400007640000010536510435500317020643 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Acquisition test cases (and useful examples) Acquisition [1] is a mechanism that allows objects to obtain attributes from their environment. It is similar to inheritence, except that, rather than traversing an inheritence hierarchy to obtain attributes, a containment hierarchy is traversed. The "ExtensionClass":ExtensionClass.html. release includes mix-in extension base classes that can be used to add acquisition as a feature to extension subclasses. These mix-in classes use the context-wrapping feature of ExtensionClasses to implement acquisition. Consider the following example:: >>> import ExtensionClass, Acquisition >>> class C(ExtensionClass.Base): ... color='red' >>> class A(Acquisition.Implicit): ... def report(self): ... print self.color >>> a = A() >>> c = C() >>> c.a = a >>> c.a.report() red >>> d = C() >>> d.color = 'green' >>> d.a = a >>> d.a.report() green >>> a.report() # raises an attribute error Traceback (most recent call last): ... AttributeError: color The class 'A' inherits acquisition behavior from 'Acquisition.Implicit'. The object, 'a', "has" the color of objects 'c' and 'd' when it is accessed through them, but it has no color by itself. The object 'a' obtains attributes from it's environment, where it's environment is defined by the access path used to reach 'a'. Acquisition wrappers When an object that supports acquisition is accessed through an extension class instance, a special object, called an acquisition wrapper, is returned. In the example above, the expression 'c.a' returns an acquisition wrapper that contains references to both 'c' and 'a'. It is this wrapper that performs attribute lookup in 'c' when an attribute cannot be found in 'a'. Aquisition wrappers provide access to the wrapped objects through the attributes 'aq_parent', 'aq_self', 'aq_base'. In the example above, the expressions:: >>> c.a.aq_parent is c 1 and:: >>> c.a.aq_self is a 1 both evaluate to true, but the expression:: >>> c.a is a 0 evaluates to false, because the expression 'c.a' evaluates to an acquisition wrapper around 'c' and 'a', not 'a' itself. The attribute 'aq_base' is similar to 'aq_self'. Wrappers may be nested and 'aq_self' may be a wrapped object. The 'aq_base' attribute is the underlying object with all wrappers removed. Acquisition Control Two styles of acquisition are supported in the current ExtensionClass release, implicit and explicit aquisition. Implicit acquisition Implicit acquisition is so named because it searches for attributes from the environment automatically whenever an attribute cannot be obtained directly from an object or through inheritence. An attribute may be implicitly acquired if it's name does not begin with an underscore, '_'. To support implicit acquisition, an object should inherit from the mix-in class 'Acquisition.Implicit'. Explicit Acquisition When explicit acquisition is used, attributes are not automatically obtained from the environment. Instead, the method 'aq_aquire' must be used, as in:: print c.a.aq_acquire('color') To support explicit acquisition, an object should inherit from the mix-in class 'Acquisition.Explicit'. Controlled Acquisition A class (or instance) can provide attribute by attribute control over acquisition. This is done by: - subclassing from 'Acquisition.Explicit', and - setting all attributes that should be acquired to the special value: 'Acquisition.Acquired'. Setting an attribute to this value also allows inherited attributes to be overridden with acquired ones. For example, in:: >>> class E(Acquisition.Explicit): ... id = 1 ... secret = 2 ... color = Acquisition.Acquired ... __roles__ = Acquisition.Acquired The *only* attributes that are automatically acquired from containing objects are 'color', and '__roles__'. >>> c = C() >>> c.foo = 'foo' >>> c.e = E() >>> c.e.color 'red' >>> c.e.foo Traceback (most recent call last): ... AttributeError: foo Note also that the '__roles__' attribute is acquired even though it's name begins with an underscore: >>> c.__roles__ = 'Manager', 'Member' >>> c.e.__roles__ ('Manager', 'Member') In fact, the special 'Acquisition.Acquired' value can be used in 'Acquisition.Implicit' objects to implicitly acquire selected objects that smell like private objects. >>> class I(Acquisition.Implicit): ... __roles__ = Acquisition.Acquired >>> c.x = C() >>> c.x.__roles__ Traceback (most recent call last): ... AttributeError: __roles__ >>> c.x = I() >>> c.x.__roles__ ('Manager', 'Member') Filtered Acquisition The acquisition method, 'aq_acquire', accepts two optional arguments. The first of the additional arguments is a "filtering" function that is used when considering whether to acquire an object. The second of the additional arguments is an object that is passed as extra data when calling the filtering function and which defaults to 'None'. The filter function is called with five arguments: - The object that the 'aq_acquire' method was called on, - The object where an object was found, - The name of the object, as passed to 'aq_acquire', - The object found, and - The extra data passed to 'aq_acquire'. If the filter returns a true object that the object found is returned, otherwise, the acquisition search continues. For example, in:: >>> from Acquisition import Explicit >>> class HandyForTesting: ... def __init__(self, name): self.name=name ... def __str__(self): ... return "%s(%s)" % (self.name, self.__class__.__name__) ... __repr__=__str__ >>> class E(Explicit, HandyForTesting): ... pass >>> class Nice(HandyForTesting): ... isNice=1 ... def __str__(self): ... return HandyForTesting.__str__(self)+' and I am nice!' ... __repr__=__str__ >>> a = E('a') >>> a.b = E('b') >>> a.b.c = E('c') >>> a.p = Nice('spam') >>> a.b.p = E('p') >>> def find_nice(self, ancestor, name, object, extra): ... return hasattr(object,'isNice') and object.isNice >>> print a.b.c.aq_acquire('p', find_nice) spam(Nice) and I am nice! The filtered acquisition in the last line skips over the first attribute it finds with the name 'p', because the attribute doesn't satisfy the condition given in the filter. Acquisition and methods Python methods of objects that support acquisition can use acquired attributes as in the 'report' method of the first example above. When a Python method is called on an object that is wrapped by an acquisition wrapper, the wrapper is passed to the method as the first argument. This rule also applies to user-defined method types and to C methods defined in pure mix-in classes. Unfortunately, C methods defined in extension base classes that define their own data structures, cannot use aquired attributes at this time. This is because wrapper objects do not conform to the data structures expected by these methods. Acquiring Acquiring objects Consider the following example:: >>> from Acquisition import Implicit >>> class C(Implicit): ... def __init__(self, name): self.name=name ... def __str__(self): ... return "%s(%s)" % (self.name, self.__class__.__name__) ... __repr__=__str__ >>> a = C("a") >>> a.b = C("b") >>> a.b.pref = "spam" >>> a.b.c = C("c") >>> a.b.c.color = "red" >>> a.b.c.pref = "eggs" >>> a.x = C("x") >>> o = a.b.c.x The expression 'o.color' might be expected to return '"red"'. In earlier versions of ExtensionClass, however, this expression failed. Acquired acquiring objects did not acquire from the environment they were accessed in, because objects were only wrapped when they were first found, and were not rewrapped as they were passed down the acquisition tree. In the current release of ExtensionClass, the expression "o.color" does indeed return '"red"'. >>> o.color 'red' When searching for an attribute in 'o', objects are searched in the order 'x', 'a', 'b', 'c'. So, for example, the expression, 'o.pref' returns '"spam"', not '"eggs"':: >>> o.pref 'spam' In earlier releases of ExtensionClass, the attempt to get the 'pref' attribute from 'o' would have failed. If desired, the current rules for looking up attributes in complex expressions can best be understood through repeated application of the '__of__' method: 'a.x' -- 'x.__of__(a)' 'a.b' -- 'b.__of__(a)' 'a.b.x' -- 'x.__of__(a).__of__(b.__of__(a))' 'a.b.c' -- 'c.__of__(b.__of__(a))' 'a.b.c.x' -- 'x.__of__(a).__of__(b.__of__(a)).__of__(c.__of__(b.__of__(a)))' and by keeping in mind that attribute lookup in a wrapper is done by trying to lookup the attribute in the wrapped object first and then in the parent object. In the expressions above involving the '__of__' method, lookup proceeds from left to right. Note that heuristics are used to avoid most of the repeated lookups. For example, in the expression: 'a.b.c.x.foo', the object 'a' is searched no more than once, even though it is wrapped three times. .. [1] Gil, J., Lorenz, D., "Environmental Acquisition--A New Inheritance-Like Abstraction Mechanism", http://www.bell-labs.com/people/cope/oopsla/Oopsla96TechnicalProgramAbstracts.html#GilLorenz, OOPSLA '96 Proceedings, ACM SIG-PLAN, October, 1996 $Id: tests.py 30282 2005-05-06 15:08:46Z jim $ """ import ExtensionClass import Acquisition class I(Acquisition.Implicit): def __init__(self, id): self.id = id def __repr__(self): return self.id class E(Acquisition.Explicit): def __init__(self, id): self.id = id def __repr__(self): return self.id def test_unwrapped(): """ >>> c = I('unwrapped') >>> show(c) unwrapped >>> c.aq_parent Traceback (most recent call last): ... AttributeError: aq_parent >>> Acquisition.aq_acquire(c, 'id') 'unwrapped' >>> Acquisition.aq_acquire(c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_acquire(c, 'id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'unwrapped' >>> Acquisition.aq_base(c) is c 1 >>> Acquisition.aq_chain(c) [unwrapped] >>> Acquisition.aq_chain(c, 1) [unwrapped] >>> Acquisition.aq_get(c, 'id') 'unwrapped' >>> Acquisition.aq_get(c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_get(c, 'x', 'foo') 'foo' >>> Acquisition.aq_get(c, 'x', 'foo', 1) 'foo' >>> Acquisition.aq_inner(c) is c 1 >>> Acquisition.aq_parent(c) >>> Acquisition.aq_self(c) is c 1 """ def test_simple(): """ >>> a = I('a') >>> a.y = 42 >>> a.b = I('b') >>> a.b.c = I('c') >>> show(a.b.c) c | b | a >>> show(a.b.c.aq_parent) b | a >>> show(a.b.c.aq_self) c >>> show(a.b.c.aq_base) c >>> show(a.b.c.aq_inner) c | b | a >>> a.b.c.y 42 >>> a.b.c.aq_chain [c, b, a] >>> a.b.c.aq_inContextOf(a) 1 >>> a.b.c.aq_inContextOf(a.b) 1 >>> a.b.c.aq_inContextOf(a.b.c) 1 >>> a.b.c.aq_acquire('y') 42 >>> a.b.c.aq_acquire('id') 'c' >>> a.b.c.aq_acquire('x') Traceback (most recent call last): ... AttributeError: x >>> a.b.c.aq_acquire('id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> Acquisition.aq_acquire(a.b.c, 'id') 'c' >>> Acquisition.aq_acquire(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_acquire(a.b.c, 'y') 42 >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> show(Acquisition.aq_base(a.b.c)) c >>> Acquisition.aq_chain(a.b.c) [c, b, a] >>> Acquisition.aq_chain(a.b.c, 1) [c, b, a] >>> Acquisition.aq_get(a.b.c, 'id') 'c' >>> Acquisition.aq_get(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_get(a.b.c, 'x', 'foo') 'foo' >>> Acquisition.aq_get(a.b.c, 'x', 'foo', 1) 'foo' >>> show(Acquisition.aq_inner(a.b.c)) c | b | a >>> show(Acquisition.aq_parent(a.b.c)) b | a >>> show(Acquisition.aq_self(a.b.c)) c """ def test__of__exception(): """ Wrapper_findattr did't check for an exception in a user defined __of__ method before passing the result to the filter. In this case the 'value' argument of the filter was NULL, which caused a segfault when being accessed. >>> class UserError(Exception): ... pass ... >>> class X(Acquisition.Implicit): ... def __of__(self, parent): ... if Acquisition.aq_base(parent) is not parent: ... raise UserError, 'ack' ... return X.inheritedAttribute('__of__')(self, parent) ... >>> a = I('a') >>> a.b = I('b') >>> a.b.x = X('x') >>> Acquisition.aq_acquire(a.b, 'x', ... lambda self, object, name, value, extra: repr(value)) Traceback (most recent call last): ... UserError: ack """ def test_muliple(): r""" >>> a = I('a') >>> a.color = 'red' >>> a.a1 = I('a1') >>> a.a1.color = 'green' >>> a.a1.a11 = I('a11') >>> a.a2 = I('a2') >>> a.a2.a21 = I('a21') >>> show(a.a1.a11.a2.a21) a21 | (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | a1 | | | a | a11 | a1 | a >>> a.a1.a11.a2.a21.color 'red' >>> show(a.a1.a11.a2.a21.aq_parent) (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | a1 | | | a | a11 | a1 | a >>> show(a.a1.a11.a2.a21.aq_parent.aq_parent) a11 | a1 | a >>> show(a.a1.a11.a2.a21.aq_self) a21 >>> show(a.a1.a11.a2.a21.aq_parent.aq_self) (a2) | \ | a2 | | | a | a1 | a >>> show(a.a1.a11.a2.a21.aq_base) a21 >>> show(a.a1.a11.a2.a21.aq_inner) a21 | (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | a1 | | | a | a11 | a1 | a >>> show(a.a1.a11.a2.a21.aq_inner.aq_parent.aq_inner) a2 | a >>> show(a.a1.a11.a2.a21.aq_inner.aq_parent.aq_inner.aq_parent) a >>> a.a1.a11.a2.a21.aq_chain [a21, a2, a11, a1, a] >>> a.a1.a11.a2.a21.aq_inContextOf(a) 1 >>> a.a1.a11.a2.a21.aq_inContextOf(a.a2) 1 >>> a.a1.a11.a2.a21.aq_inContextOf(a.a1) 0 >>> a.a1.a11.a2.a21.aq_acquire('color') 'red' >>> a.a1.a11.a2.a21.aq_acquire('id') 'a21' >>> a.a1.a11.a2.a21.aq_acquire('color', ... lambda ob, parent, name, v, extra: extra) Traceback (most recent call last): ... AttributeError: color >>> a.a1.a11.a2.a21.aq_acquire('color', ... lambda ob, parent, name, v, extra: extra, 1) 'red' >>> a.a1.y = 42 >>> a.a1.a11.a2.a21.aq_acquire('y') 42 >>> a.a1.a11.a2.a21.aq_acquire('y', containment=1) Traceback (most recent call last): ... AttributeError: y Much of the same, but with methods: >>> show(Acquisition.aq_parent(a.a1.a11.a2.a21)) (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | a1 | | | a | a11 | a1 | a >>> show(Acquisition.aq_parent(a.a1.a11.a2.a21.aq_parent)) a11 | a1 | a >>> show(Acquisition.aq_self(a.a1.a11.a2.a21)) a21 >>> show(Acquisition.aq_self(a.a1.a11.a2.a21.aq_parent)) (a2) | \ | a2 | | | a | a1 | a >>> show(Acquisition.aq_base(a.a1.a11.a2.a21)) a21 >>> show(Acquisition.aq_inner(a.a1.a11.a2.a21)) a21 | (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | a1 | | | a | a11 | a1 | a >>> show(Acquisition.aq_inner(a.a1.a11.a2.a21.aq_inner.aq_parent)) a2 | a >>> show(Acquisition.aq_parent( ... a.a1.a11.a2.a21.aq_inner.aq_parent.aq_inner)) a >>> Acquisition.aq_chain(a.a1.a11.a2.a21) [a21, a2, a11, a1, a] >>> Acquisition.aq_chain(a.a1.a11.a2.a21, 1) [a21, a2, a] >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'color') 'red' >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'id') 'a21' >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'color', ... lambda ob, parent, name, v, extra: extra) Traceback (most recent call last): ... AttributeError: color >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'color', ... lambda ob, parent, name, v, extra: extra, 1) 'red' >>> a.a1.y = 42 >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'y') 42 >>> Acquisition.aq_acquire(a.a1.a11.a2.a21, 'y', containment=1) Traceback (most recent call last): ... AttributeError: y """ def test_pinball(): r""" >>> a = I('a') >>> a.a1 = I('a1') >>> a.a1.a11 = I('a11') >>> a.a1.a12 = I('a12') >>> a.a2 = I('a2') >>> a.a2.a21 = I('a21') >>> a.a2.a22 = I('a22') >>> show(a.a1.a11.a1.a12.a2.a21.a2.a22) a22 | (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | (a2) | | \ | | (a2) | | | \ | | | a2 | | | | | | | a | | | | | (a1) | | | \ | | | (a1) | | | | \ | | | | a1 | | | | | | | | | a | | | | | | | a1 | | | | | | | a | | | | | a11 | | | | | a1 | | | | | a | | | a12 | | | (a1) | | \ | | (a1) | | | \ | | | a1 | | | | | | | a | | | | | a1 | | | | | a | | | a11 | | | a1 | | | a | a21 | (a2) | \ | (a2) | | \ | | a2 | | | | | a | | | (a1) | | \ | | (a1) | | | \ | | | a1 | | | | | | | a | | | | | a1 | | | | | a | | | a11 | | | a1 | | | a | a12 | (a1) | \ | (a1) | | \ | | a1 | | | | | a | | | a1 | | | a | a11 | a1 | a """ def test_explicit(): """ >>> a = E('a') >>> a.y = 42 >>> a.b = E('b') >>> a.b.c = E('c') >>> show(a.b.c) c | b | a >>> show(a.b.c.aq_parent) b | a >>> show(a.b.c.aq_self) c >>> show(a.b.c.aq_base) c >>> show(a.b.c.aq_inner) c | b | a >>> a.b.c.y Traceback (most recent call last): ... AttributeError: y >>> a.b.c.aq_chain [c, b, a] >>> a.b.c.aq_inContextOf(a) 1 >>> a.b.c.aq_inContextOf(a.b) 1 >>> a.b.c.aq_inContextOf(a.b.c) 1 >>> a.b.c.aq_acquire('y') 42 >>> a.b.c.aq_acquire('id') 'c' >>> a.b.c.aq_acquire('x') Traceback (most recent call last): ... AttributeError: x >>> a.b.c.aq_acquire('id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> Acquisition.aq_acquire(a.b.c, 'id') 'c' >>> Acquisition.aq_acquire(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_acquire(a.b.c, 'y') 42 >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> show(Acquisition.aq_base(a.b.c)) c >>> Acquisition.aq_chain(a.b.c) [c, b, a] >>> Acquisition.aq_chain(a.b.c, 1) [c, b, a] >>> Acquisition.aq_get(a.b.c, 'id') 'c' >>> Acquisition.aq_get(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_get(a.b.c, 'y') 42 >>> Acquisition.aq_get(a.b.c, 'x', 'foo') 'foo' >>> Acquisition.aq_get(a.b.c, 'x', 'foo', 1) 'foo' >>> show(Acquisition.aq_inner(a.b.c)) c | b | a >>> show(Acquisition.aq_parent(a.b.c)) b | a >>> show(Acquisition.aq_self(a.b.c)) c """ def test_mixed_explicit_and_explicit(): """ >>> a = I('a') >>> a.y = 42 >>> a.b = E('b') >>> a.b.z = 3 >>> a.b.c = I('c') >>> show(a.b.c) c | b | a >>> show(a.b.c.aq_parent) b | a >>> show(a.b.c.aq_self) c >>> show(a.b.c.aq_base) c >>> show(a.b.c.aq_inner) c | b | a >>> a.b.c.y 42 >>> a.b.c.z 3 >>> a.b.c.aq_chain [c, b, a] >>> a.b.c.aq_inContextOf(a) 1 >>> a.b.c.aq_inContextOf(a.b) 1 >>> a.b.c.aq_inContextOf(a.b.c) 1 >>> a.b.c.aq_acquire('y') 42 >>> a.b.c.aq_acquire('z') 3 >>> a.b.c.aq_acquire('z', explicit=False) 3 >>> a.b.c.aq_acquire('id') 'c' >>> a.b.c.aq_acquire('x') Traceback (most recent call last): ... AttributeError: x >>> a.b.c.aq_acquire('id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> Acquisition.aq_acquire(a.b.c, 'id') 'c' >>> Acquisition.aq_acquire(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_acquire(a.b.c, 'y') 42 >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra) Traceback (most recent call last): ... AttributeError: id >>> Acquisition.aq_acquire(a.b.c, 'id', ... lambda searched, parent, name, ob, extra: extra, ... 1) 'c' >>> show(Acquisition.aq_base(a.b.c)) c >>> Acquisition.aq_chain(a.b.c) [c, b, a] >>> Acquisition.aq_chain(a.b.c, 1) [c, b, a] >>> Acquisition.aq_get(a.b.c, 'id') 'c' >>> Acquisition.aq_get(a.b.c, 'x') Traceback (most recent call last): ... AttributeError: x >>> Acquisition.aq_get(a.b.c, 'y') 42 >>> Acquisition.aq_get(a.b.c, 'x', 'foo') 'foo' >>> Acquisition.aq_get(a.b.c, 'x', 'foo', 1) 'foo' >>> show(Acquisition.aq_inner(a.b.c)) c | b | a >>> show(Acquisition.aq_parent(a.b.c)) b | a >>> show(Acquisition.aq_self(a.b.c)) c """ def old_tests(): """ >>> from ExtensionClass import Base >>> import Acquisition >>> class B(Base): ... color='red' >>> class A(Acquisition.Implicit): ... def hi(self): ... print "%s()" % self.__class__.__name__, self.color >>> b=B() >>> b.a=A() >>> b.a.hi() A() red >>> b.a.color='green' >>> b.a.hi() A() green >>> try: ... A().hi() ... raise 'Program error', 'spam' ... except AttributeError: pass A() New test for wrapper comparisons. >>> foo = b.a >>> bar = b.a >>> foo == bar 1 >>> c = A() >>> b.c = c >>> b.c.d = c >>> b.c.d == c 1 >>> b.c.d == b.c 1 >>> b.c == c 1 >>> def checkContext(self, o): ... # Python equivalent to aq_inContextOf ... from Acquisition import aq_base, aq_parent, aq_inner ... subob = self ... o = aq_base(o) ... while 1: ... if aq_base(subob) is o: return 1 ... self = aq_inner(subob) ... if self is None: break ... subob = aq_parent(self) ... if subob is None: break >>> checkContext(b.c, b) 1 >>> not checkContext(b.c, b.a) 1 >>> b.a.aq_inContextOf(b) 1 >>> b.c.aq_inContextOf(b) 1 >>> b.c.d.aq_inContextOf(b) 1 >>> b.c.d.aq_inContextOf(c) 1 >>> b.c.d.aq_inContextOf(b.c) 1 >>> not b.c.aq_inContextOf(foo) 1 >>> not b.c.aq_inContextOf(b.a) 1 >>> not b.a.aq_inContextOf('somestring') 1 """ def test_AqAlg(): """ >>> A=I('A') >>> A.B=I('B') >>> A.B.color='red' >>> A.C=I('C') >>> A.C.D=I('D') >>> A A >>> Acquisition.aq_chain(A) [A] >>> Acquisition.aq_chain(A, 1) [A] >>> map(Acquisition.aq_base, Acquisition.aq_chain(A, 1)) [A] >>> A.C C >>> Acquisition.aq_chain(A.C) [C, A] >>> Acquisition.aq_chain(A.C, 1) [C, A] >>> map(Acquisition.aq_base, Acquisition.aq_chain(A.C, 1)) [C, A] >>> A.C.D D >>> Acquisition.aq_chain(A.C.D) [D, C, A] >>> Acquisition.aq_chain(A.C.D, 1) [D, C, A] >>> map(Acquisition.aq_base, Acquisition.aq_chain(A.C.D, 1)) [D, C, A] >>> A.B.C C >>> Acquisition.aq_chain(A.B.C) [C, B, A] >>> Acquisition.aq_chain(A.B.C, 1) [C, A] >>> map(Acquisition.aq_base, Acquisition.aq_chain(A.B.C, 1)) [C, A] >>> A.B.C.D D >>> Acquisition.aq_chain(A.B.C.D) [D, C, B, A] >>> Acquisition.aq_chain(A.B.C.D, 1) [D, C, A] >>> map(Acquisition.aq_base, Acquisition.aq_chain(A.B.C.D, 1)) [D, C, A] >>> A.B.C.D.color 'red' >>> Acquisition.aq_get(A.B.C.D, "color", None) 'red' >>> Acquisition.aq_get(A.B.C.D, "color", None, 1) """ def test_explicit_acquisition(): """ >>> from ExtensionClass import Base >>> import Acquisition >>> class B(Base): ... color='red' >>> class A(Acquisition.Explicit): ... def hi(self): ... print self.__class__.__name__, self.acquire('color') >>> b=B() >>> b.a=A() >>> b.a.hi() A red >>> b.a.color='green' >>> b.a.hi() A green >>> try: ... A().hi() ... raise 'Program error', 'spam' ... except AttributeError: pass A """ def test_creating_wrappers_directly(): """ >>> from ExtensionClass import Base >>> from Acquisition import ImplicitAcquisitionWrapper >>> class B(Base): ... pass >>> a = B() >>> a.color = 'red' >>> a.b = B() >>> w = ImplicitAcquisitionWrapper(a.b, a) >>> w.color 'red' >>> w = ImplicitAcquisitionWrapper(a.b) Traceback (most recent call last): ... TypeError: __init__() takes exactly 2 arguments (1 given) We can reassign aq_parent >>> x = B() >>> x.color = 'green' >>> w.aq_parent = x >>> w.color 'green' >>> w = ImplicitAcquisitionWrapper() Traceback (most recent call last): ... TypeError: __init__() takes exactly 2 arguments (0 given) >>> w = ImplicitAcquisitionWrapper(obj=1) Traceback (most recent call last): ... TypeError: kwyword arguments not allowed """ def test_cant_pickle_acquisition_wrappers_classic(): """ >>> import pickle >>> class X: ... def __getstate__(self): ... return 1 We shouldn't be able to pickle wrappers: >>> from Acquisition import ImplicitAcquisitionWrapper >>> w = ImplicitAcquisitionWrapper(X(), X()) >>> pickle.dumps(w) Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. But that's not enough. We need to defeat persistence as well. :) This is tricky. We want to generate the error in __getstate__, not in the attr access, as attribute errors are too-often hidden: >>> getstate = w.__getstate__ >>> getstate() Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. We shouldn't be able to pickle wrappers: >>> from Acquisition import ExplicitAcquisitionWrapper >>> w = ExplicitAcquisitionWrapper(X(), X()) >>> pickle.dumps(w) Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. But that's not enough. We need to defeat persistence as well. :) This is tricky. We want to generate the error in __getstate__, not in the attr access, as attribute errors are too-often hidden: >>> getstate = w.__getstate__ >>> getstate() Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. """ def test_cant_pickle_acquisition_wrappers_newstyle(): """ >>> import pickle >>> class X(object): ... def __getstate__(self): ... return 1 We shouldn't be able to pickle wrappers: >>> from Acquisition import ImplicitAcquisitionWrapper >>> w = ImplicitAcquisitionWrapper(X(), X()) >>> pickle.dumps(w) Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. But that's not enough. We need to defeat persistence as well. :) This is tricky. We want to generate the error in __getstate__, not in the attr access, as attribute errors are too-often hidden: >>> getstate = w.__getstate__ >>> getstate() Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. We shouldn't be able to pickle wrappers: >>> from Acquisition import ExplicitAcquisitionWrapper >>> w = ExplicitAcquisitionWrapper(X(), X()) >>> pickle.dumps(w) Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. But that's not enough. We need to defeat persistence as well. :) This is tricky. We want to generate the error in __getstate__, not in the attr access, as attribute errors are too-often hidden: >>> getstate = w.__getstate__ >>> getstate() Traceback (most recent call last): ... TypeError: Can't pickle objects in acquisition wrappers. """ def show(x): print showaq(x).strip() def showaq(m_self, indent=''): rval = '' obj = m_self base = getattr(obj, 'aq_base', obj) try: id = base.id except: id = str(base) try: id = id() except: pass if hasattr(obj, 'aq_self'): if hasattr(obj.aq_self, 'aq_self'): rval = rval + indent + "(" + id + ")\n" rval = rval + indent + "| \\\n" rval = rval + showaq(obj.aq_self, '| ' + indent) rval = rval + indent + "|\n" rval = rval + showaq(obj.aq_parent, indent) elif hasattr(obj, 'aq_parent'): rval = rval + indent + id + "\n" rval = rval + indent + "|\n" rval = rval + showaq(obj.aq_parent, indent) else: rval = rval + indent + id + "\n" return rval def test_Basic_gc(): """Test to make sure that EC instances participate in GC >>> from ExtensionClass import Base >>> import gc >>> thresholds = gc.get_threshold() >>> gc.set_threshold(0) >>> for B in I, E: ... class C1(B): ... pass ... ... class C2(Base): ... def __del__(self): ... print 'removed' ... ... a=C1('a') ... a.b = C1('a.b') ... a.b.a = a ... a.b.c = C2() ... ignore = gc.collect() ... del a ... removed = gc.collect() ... print removed > 0 removed True removed True >>> gc.set_threshold(*thresholds) """ def test_Wrapper_gc(): """Test to make sure that EC instances participate in GC >>> import gc >>> thresholds = gc.get_threshold() >>> gc.set_threshold(0) >>> for B in I, E: ... class C: ... def __del__(self): ... print 'removed' ... ... a=B('a') ... a.b = B('b') ... a.a_b = a.b # circ ref through wrapper ... a.b.c = C() ... ignored = gc.collect() ... del a ... removed = gc.collect() ... removed > 0 removed True removed True >>> gc.set_threshold(*thresholds) """ import unittest from zope.testing.doctest import DocTestSuite def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/Acquisition/Acquisition.h0000664000076400007640000000515110435500317021560 0ustar ajungajung/***************************************************************************** Copyright (c) 1996-2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #ifndef __ACQUISITION_H_ #define __ACQUISITION_H_ typedef struct { PyObject *(*AQ_Acquire) (PyObject *obj, PyObject *name, PyObject *filter, PyObject *extra, int explicit, PyObject *deflt, int containment); PyObject *(*AQ_Get) (PyObject *obj, PyObject *name, PyObject *deflt, int containment); int (*AQ_IsWrapper) (PyObject *obj); PyObject *(*AQ_Base) (PyObject *obj); PyObject *(*AQ_Parent) (PyObject *obj); PyObject *(*AQ_Self) (PyObject *obj); PyObject *(*AQ_Inner) (PyObject *obj); PyObject *(*AQ_Chain) (PyObject *obj, int containment); } ACQUISITIONCAPI; #ifndef _IN_ACQUISITION_C #define aq_Acquire(obj, name, filter, extra, explicit, deflt, containment ) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Acquire(obj, name, filter, extra, explicit, deflt, containment))) #define aq_acquire(obj, name) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Acquire(obj, name, NULL, NULL, 1, NULL, 0))) #define aq_get(obj, name, deflt, containment) (AcquistionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Get(obj, name, deflt, containment))) #define aq_isWrapper(obj) (AcquisitionCAPI == NULL ? -1 : (AcquisitionCAPI->AQ_IsWrapper(obj))) #define aq_base(obj) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Base(obj))) #define aq_parent(obj) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Parent(obj))) #define aq_self(obj) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Self(obj))) #define aq_inner(obj) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_Inner(obj))) #define aq_chain(obj, containment) (AcquisitionCAPI == NULL ? NULL : (AcquisitionCAPI->AQ_CHain(obj, containment))) static ACQUISITIONCAPI *AcquisitionCAPI = NULL; #define aq_init() { \ PyObject *module; \ PyObject *api; \ if (! (module = PyImport_ImportModule("Acquisition"))) return; \ if (! (api = PyObject_GetAttrString(module,"AcquisitionCAPI"))) return; \ Py_DECREF(module); \ AcquisitionCAPI = PyCObject_AsVoidPtr(api); \ Py_DECREF(api); \ } #endif #endif Zope-2.8.7-final/lib/python/Acquisition/_Acquisition.c0000664000076400007640000011644410435500317021722 0ustar ajungajung/***************************************************************************** Copyright (c) 1996-2003 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "ExtensionClass.h" #define _IN_ACQUISITION_C #include "Acquisition.h" static ACQUISITIONCAPI AcquisitionCAPI; static void PyVar_Assign(PyObject **v, PyObject *e) { Py_XDECREF(*v); *v=e; } #define ASSIGN(V,E) PyVar_Assign(&(V),(E)) #define UNLESS(E) if (!(E)) #define UNLESS_ASSIGN(V,E) ASSIGN(V,E); UNLESS(V) #define OBJECT(O) ((PyObject*)(O)) static PyObject *py__add__, *py__sub__, *py__mul__, *py__div__, *py__mod__, *py__pow__, *py__divmod__, *py__lshift__, *py__rshift__, *py__and__, *py__or__, *py__xor__, *py__coerce__, *py__neg__, *py__pos__, *py__abs__, *py__nonzero__, *py__invert__, *py__int__, *py__long__, *py__float__, *py__oct__, *py__hex__, *py__getitem__, *py__setitem__, *py__delitem__, *py__getslice__, *py__setslice__, *py__delslice__, *py__len__, *py__of__, *py__call__, *py__repr__, *py__str__, *py__cmp__; static PyObject *Acquired=0; static void init_py_names(void) { #define INIT_PY_NAME(N) py ## N = PyString_FromString(#N) INIT_PY_NAME(__add__); INIT_PY_NAME(__sub__); INIT_PY_NAME(__mul__); INIT_PY_NAME(__div__); INIT_PY_NAME(__mod__); INIT_PY_NAME(__pow__); INIT_PY_NAME(__divmod__); INIT_PY_NAME(__lshift__); INIT_PY_NAME(__rshift__); INIT_PY_NAME(__and__); INIT_PY_NAME(__or__); INIT_PY_NAME(__xor__); INIT_PY_NAME(__coerce__); INIT_PY_NAME(__neg__); INIT_PY_NAME(__pos__); INIT_PY_NAME(__abs__); INIT_PY_NAME(__nonzero__); INIT_PY_NAME(__invert__); INIT_PY_NAME(__int__); INIT_PY_NAME(__long__); INIT_PY_NAME(__float__); INIT_PY_NAME(__oct__); INIT_PY_NAME(__hex__); INIT_PY_NAME(__getitem__); INIT_PY_NAME(__setitem__); INIT_PY_NAME(__delitem__); INIT_PY_NAME(__getslice__); INIT_PY_NAME(__setslice__); INIT_PY_NAME(__delslice__); INIT_PY_NAME(__len__); INIT_PY_NAME(__of__); INIT_PY_NAME(__call__); INIT_PY_NAME(__repr__); INIT_PY_NAME(__str__); INIT_PY_NAME(__cmp__); #undef INIT_PY_NAME } static PyObject * CallMethodO(PyObject *self, PyObject *name, PyObject *args, PyObject *kw) { if (! args && PyErr_Occurred()) return NULL; UNLESS(name=PyObject_GetAttr(self,name)) { if (args) { Py_DECREF(args); } return NULL; } ASSIGN(name,PyEval_CallObjectWithKeywords(name,args,kw)); if (args) { Py_DECREF(args); } return name; } #define Build Py_BuildValue /* Declarations for objects of type Wrapper */ typedef struct { PyObject_HEAD PyObject *obj; PyObject *container; } Wrapper; staticforward PyExtensionClass Wrappertype, XaqWrappertype; #define isWrapper(O) ((O)->ob_type==(PyTypeObject*)&Wrappertype || \ (O)->ob_type==(PyTypeObject*)&XaqWrappertype) #define WRAPPER(O) ((Wrapper*)(O)) static int Wrapper__init__(Wrapper *self, PyObject *args, PyObject *kwargs) { PyObject *obj, *container; if (kwargs && PyDict_Size(kwargs) != 0) { PyErr_SetString(PyExc_TypeError, "kwyword arguments not allowed"); return -1; } UNLESS(PyArg_ParseTuple(args, "OO:__init__", &obj, &container)) return -1; if (self == WRAPPER(obj)) { PyErr_SetString(PyExc_ValueError, "Cannot wrap acquisition wrapper in itself (Wrapper__init__)"); return -1; } Py_INCREF(obj); self->obj=obj; if (container != Py_None) { Py_INCREF(container); self->container=container; } return 0; } /* ---------------------------------------------------------------- */ static PyObject * __of__(PyObject *inst, PyObject *parent) { PyObject *r, *t; UNLESS(r=PyObject_GetAttr(inst, py__of__)) return NULL; UNLESS(t=PyTuple_New(1)) goto err; Py_INCREF(parent); PyTuple_SET_ITEM(t,0,parent); ASSIGN(r,PyObject_CallObject(r,t)); Py_DECREF(t); if (r != NULL && isWrapper(r) && WRAPPER(r)->container && isWrapper(WRAPPER(r)->container) ) while (WRAPPER(r)->obj && isWrapper(WRAPPER(r)->obj) && (WRAPPER(WRAPPER(r)->obj)->container == WRAPPER(WRAPPER(r)->container)->obj) ) { if (r->ob_refcnt !=1 ) { t = PyObject_CallFunctionObjArgs((PyObject *)(r->ob_type), WRAPPER(r)->obj, WRAPPER(r)->container, NULL); Py_DECREF(r); if (t==NULL) return NULL; r = t; } /* Simplify wrapper */ Py_XINCREF(WRAPPER(WRAPPER(r)->obj)->obj); ASSIGN(WRAPPER(r)->obj, WRAPPER(WRAPPER(r)->obj)->obj); } return r; err: Py_DECREF(r); return NULL; } static PyObject * Wrapper_descrget(Wrapper *self, PyObject *inst, PyObject *cls) { if (inst == NULL) { Py_INCREF(self); return (PyObject *)self; } return __of__((PyObject *)self, inst); } #define newWrapper(obj, container, Wrappertype) \ PyObject_CallFunctionObjArgs(OBJECT(Wrappertype), obj, container, NULL) static int Wrapper_traverse(Wrapper *self, visitproc visit, void *arg) { int vret; if (self->obj) { vret = visit(self->obj, arg); if (vret != 0) return vret; } if (self->container) { vret = visit(self->container, arg); if (vret != 0) return vret; } return 0; } static int Wrapper_clear(Wrapper *self) { PyObject *tmp; tmp = self->obj; self->obj = NULL; Py_XDECREF(tmp); tmp = self->container; self->container = NULL; Py_XDECREF(tmp); return 0; } static void Wrapper_dealloc(Wrapper *self) { Wrapper_clear(self); self->ob_type->tp_free((PyObject*)self); } static PyObject * Wrapper_special(Wrapper *self, char *name, PyObject *oname) { PyObject *r=0; switch(*name) { case 'b': if (strcmp(name,"base")==0) { if (self->obj) { r=self->obj; while (isWrapper(r) && WRAPPER(r)->obj) r=WRAPPER(r)->obj; } else r=Py_None; Py_INCREF(r); return r; } break; case 'p': if (strcmp(name,"parent")==0) { if (self->container) r=self->container; else r=Py_None; Py_INCREF(r); return r; } break; case 's': if (strcmp(name,"self")==0) { if (self->obj) r=self->obj; else r=Py_None; Py_INCREF(r); return r; } break; case 'e': if (strcmp(name,"explicit")==0) { if (self->ob_type != (PyTypeObject *)&XaqWrappertype) return newWrapper(self->obj, self->container, (PyTypeObject *)&XaqWrappertype); Py_INCREF(self); return OBJECT(self); } break; case 'a': if (strcmp(name,"acquire")==0) { return Py_FindAttr(OBJECT(self),oname); } break; case 'c': if (strcmp(name,"chain")==0) { if ((r = PyList_New(0))) while (1) { if (PyList_Append(r,OBJECT(self)) >= 0) { if (isWrapper(self) && self->container) { self=WRAPPER(self->container); continue; } } else { Py_DECREF(r); } break; } return r; } break; case 'i': if (strcmp(name,"inContextOf")==0) { return Py_FindAttr(OBJECT(self),oname); } if (strcmp(name,"inner")==0) { if (self->obj) { r=self->obj; while (isWrapper(r) && WRAPPER(r)->obj) { self=WRAPPER(r); r=WRAPPER(r)->obj; } r=OBJECT(self); } else r=Py_None; Py_INCREF(r); return r; } break; case 'u': if (strcmp(name,"uncle")==0) { return PyString_FromString("Bob"); } break; } return NULL; } static int apply_filter(PyObject *filter, PyObject *inst, PyObject *oname, PyObject *r, PyObject *extra, PyObject *orig) { /* Calls the filter, passing arguments. Returns 1 if the filter accepts the value, 0 if not, -1 if an exception occurred. Note the special reference counting rule: This function decrements the refcount of 'r' when it returns 0 or -1. When it returns 1, it leaves the refcount unchanged. */ PyObject *fr; int ir; UNLESS(fr=PyTuple_New(5)) goto err; PyTuple_SET_ITEM(fr,0,orig); Py_INCREF(orig); PyTuple_SET_ITEM(fr,1,inst); Py_INCREF(inst); PyTuple_SET_ITEM(fr,2,oname); Py_INCREF(oname); PyTuple_SET_ITEM(fr,3,r); Py_INCREF(r); PyTuple_SET_ITEM(fr,4,extra); Py_INCREF(extra); UNLESS_ASSIGN(fr,PyObject_CallObject(filter, fr)) goto err; ir=PyObject_IsTrue(fr); Py_DECREF(fr); if (ir) return 1; Py_DECREF(r); return 0; err: Py_DECREF(r); return -1; } static PyObject * Wrapper_acquire(Wrapper *self, PyObject *oname, PyObject *filter, PyObject *extra, PyObject *orig, int explicit, int containment); static PyObject * Wrapper_findattr(Wrapper *self, PyObject *oname, PyObject *filter, PyObject *extra, PyObject *orig, int sob, int sco, int explicit, int containment) { PyObject *r, *v, *tb; char *name=""; if (PyString_Check(oname)) name=PyString_AS_STRING(oname); if (*name=='a' && name[1]=='q' && name[2]=='_') if ((r=Wrapper_special(self, name+3, oname))) { if (filter) switch(apply_filter(filter,OBJECT(self),oname,r,extra,orig)) { case -1: return NULL; case 1: return r; } else return r; } else PyErr_Clear(); else if (*name=='_' && name[1]=='_' && (strcmp(name+2,"reduce__")==0 || strcmp(name+2,"reduce_ex__")==0 || strcmp(name+2,"getstate__")==0 )) return PyObject_GenericGetAttr(((PyObject*)self), oname); /* If we are doing a containment search, then replace self with aq_inner */ if (containment) while (self->obj && isWrapper(self->obj)) self=WRAPPER(self->obj); if (sob && self->obj) { if (isWrapper(self->obj)) { if (self == WRAPPER(self->obj)) { PyErr_SetString(PyExc_RuntimeError, "Recursion detected in acquisition wrapper"); return NULL; } if ((r=Wrapper_findattr(WRAPPER(self->obj), oname, filter, extra, orig, 1, /* Search object container if explicit, or object is implicit acquirer */ explicit || self->obj->ob_type == (PyTypeObject*)&Wrappertype, explicit, containment))) { if (PyECMethod_Check(r) && PyECMethod_Self(r)==self->obj) ASSIGN(r,PyECMethod_New(r,OBJECT(self))); else if (has__of__(r)) ASSIGN(r,__of__(r,OBJECT(self))); return r; } PyErr_Fetch(&r,&v,&tb); if (r && (r != PyExc_AttributeError)) { PyErr_Restore(r,v,tb); return NULL; } Py_XDECREF(r); Py_XDECREF(v); Py_XDECREF(tb); r=NULL; } else if ((r=PyObject_GetAttr(self->obj,oname))) { if (r==Acquired) { Py_DECREF(r); return Wrapper_acquire(self, oname, filter, extra, orig, 1, containment); } if (PyECMethod_Check(r) && PyECMethod_Self(r)==self->obj) ASSIGN(r,PyECMethod_New(r,OBJECT(self))); else if (has__of__(r)) ASSIGN(r,__of__(r,OBJECT(self))); if (r && filter) switch(apply_filter(filter,OBJECT(self),oname,r,extra,orig)) { case -1: return NULL; case 1: return r; } else return r; } else { PyErr_Fetch(&r,&v,&tb); if (r != PyExc_AttributeError) { PyErr_Restore(r,v,tb); return NULL; } Py_XDECREF(r); Py_XDECREF(v); Py_XDECREF(tb); r=NULL; } PyErr_Clear(); } if (sco && (*name != '_' || explicit)) return Wrapper_acquire(self, oname, filter, extra, orig, explicit, containment); PyErr_SetObject(PyExc_AttributeError,oname); return NULL; } static PyObject * Wrapper_acquire(Wrapper *self, PyObject *oname, PyObject *filter, PyObject *extra, PyObject *orig, int explicit, int containment) { PyObject *r; int sob=1, sco=1; if (self->container) { if (isWrapper(self->container)) { if (self->obj && isWrapper(self->obj)) { /* Try to optimize search by recognizing repeated obs in path */ if (WRAPPER(self->obj)->container== WRAPPER(self->container)->container) sco=0; else if (WRAPPER(self->obj)->container== WRAPPER(self->container)->obj) sob=0; } r=Wrapper_findattr((Wrapper*)self->container, oname, filter, extra, orig, sob, sco, explicit, containment); if (r && has__of__(r)) ASSIGN(r,__of__(r,OBJECT(self))); return r; } else { if ((r=PyObject_GetAttr(self->container,oname))) { if (r == Acquired) { Py_DECREF(r); } else { if (filter) { switch(apply_filter(filter,self->container,oname,r, extra,orig)) { case -1: return NULL; case 1: if (has__of__(r)) ASSIGN(r,__of__(r,OBJECT(self))); return r; } } else { if (has__of__(r)) ASSIGN(r,__of__(r,OBJECT(self))); return r; } } } else { /* May be AttributeError or some other kind of error */ return NULL; } } } PyErr_SetObject(PyExc_AttributeError, oname); return NULL; } static PyObject * Wrapper_getattro(Wrapper *self, PyObject *oname) { if (self->obj || self->container) return Wrapper_findattr(self, oname, NULL, NULL, NULL, 1, 1, 0, 0); /* Maybe we are getting initialized? */ return Py_FindAttr(OBJECT(self),oname); } static PyObject * Xaq_getattro(Wrapper *self, PyObject *oname) { char *name=""; /* Special case backward-compatible acquire method. */ if (PyString_Check(oname)) name=PyString_AS_STRING(oname); if (*name=='a' && name[1]=='c' && strcmp(name+2,"quire")==0) return Py_FindAttr(OBJECT(self),oname); if (self->obj || self->container) return Wrapper_findattr(self, oname, NULL, NULL, NULL, 1, 0, 0, 0); /* Maybe we are getting initialized? */ return Py_FindAttr(OBJECT(self),oname); } static int Wrapper_setattro(Wrapper *self, PyObject *oname, PyObject *v) { char *name=""; /* Allow assignment to parent, to change context. */ if (PyString_Check(oname)) name=PyString_AS_STRING(oname); if (*name=='a' && name[1]=='q' && name[2]=='_' && strcmp(name+3,"parent")==0) { Py_XINCREF(v); ASSIGN(self->container, v); return 0; } if (self->obj) { /* Unwrap passed in wrappers! */ while (v && isWrapper(v)) v=WRAPPER(v)->obj; if (v) return PyObject_SetAttr(self->obj, oname, v); else return PyObject_DelAttr(self->obj, oname); } PyErr_SetString(PyExc_AttributeError, "Attempt to set attribute on empty acquisition wrapper"); return -1; } static int Wrapper_compare(Wrapper *self, PyObject *w) { PyObject *obj, *wobj; PyObject *m; int r; if (OBJECT(self) == w) return 0; UNLESS (m=PyObject_GetAttr(OBJECT(self), py__cmp__)) { /* Unwrap self completely -> obj. */ while (self->obj && isWrapper(self->obj)) self=WRAPPER(self->obj); obj = self->obj; /* Unwrap w completely -> wobj. */ if (isWrapper(w)) { while (WRAPPER(w)->obj && isWrapper(WRAPPER(w)->obj)) w=WRAPPER(w)->obj; wobj = WRAPPER(w)->obj; } else wobj = w; PyErr_Clear(); if (obj == wobj) return 0; return (obj < w) ? -1 : 1; } ASSIGN(m, PyObject_CallFunction(m, "O", w)); UNLESS (m) return -1; r=PyInt_AsLong(m); Py_DECREF(m); return r; } static PyObject * Wrapper_repr(Wrapper *self) { PyObject *r; if ((r=PyObject_GetAttr(OBJECT(self),py__repr__))) { ASSIGN(r,PyObject_CallFunction(r,NULL,NULL)); return r; } else { PyErr_Clear(); return PyObject_Repr(self->obj); } } static PyObject * Wrapper_str(Wrapper *self) { PyObject *r; if ((r=PyObject_GetAttr(OBJECT(self),py__str__))) { ASSIGN(r,PyObject_CallFunction(r,NULL,NULL)); return r; } else { PyErr_Clear(); return PyObject_Str(self->obj); } } static long Wrapper_hash(Wrapper *self) { return PyObject_Hash(self->obj); } static PyObject * Wrapper_call(Wrapper *self, PyObject *args, PyObject *kw) { Py_INCREF(args); return CallMethodO(OBJECT(self),py__call__,args,kw); } /* Code to handle accessing Wrapper objects as sequence objects */ static int Wrapper_length(Wrapper *self) { long l; PyObject *r; UNLESS(r=PyObject_GetAttr(OBJECT(self), py__len__)) return -1; UNLESS_ASSIGN(r,PyObject_CallObject(r,NULL)) return -1; l=PyInt_AsLong(r); Py_DECREF(r); return l; } static PyObject * Wrapper_add(Wrapper *self, PyObject *bb) { return CallMethodO(OBJECT(self),py__add__,Build("(O)", bb) ,NULL); } static PyObject * Wrapper_mul(Wrapper *self, int n) { return CallMethodO(OBJECT(self),py__mul__,Build("(i)", n),NULL); } static PyObject * Wrapper_item(Wrapper *self, int i) { return CallMethodO(OBJECT(self),py__getitem__, Build("(i)", i),NULL); } static PyObject * Wrapper_slice(Wrapper *self, int ilow, int ihigh) { return CallMethodO(OBJECT(self),py__getslice__, Build("(ii)", ilow, ihigh),NULL); } static int Wrapper_ass_item(Wrapper *self, int i, PyObject *v) { if (v) { UNLESS(v=CallMethodO(OBJECT(self),py__setitem__, Build("(iO)", i, v),NULL)) return -1; } else { UNLESS(v=CallMethodO(OBJECT(self),py__delitem__, Build("(i)", i),NULL)) return -1; } Py_DECREF(v); return 0; } static int Wrapper_ass_slice(Wrapper *self, int ilow, int ihigh, PyObject *v) { if (v) { UNLESS(v=CallMethodO(OBJECT(self),py__setslice__, Build("(iiO)", ilow, ihigh, v),NULL)) return -1; } else { UNLESS(v=CallMethodO(OBJECT(self),py__delslice__, Build("(ii)", ilow, ihigh),NULL)) return -1; } Py_DECREF(v); return 0; } static PySequenceMethods Wrapper_as_sequence = { (inquiry)Wrapper_length, /*sq_length*/ (binaryfunc)Wrapper_add, /*sq_concat*/ (intargfunc)Wrapper_mul, /*sq_repeat*/ (intargfunc)Wrapper_item, /*sq_item*/ (intintargfunc)Wrapper_slice, /*sq_slice*/ (intobjargproc)Wrapper_ass_item, /*sq_ass_item*/ (intintobjargproc)Wrapper_ass_slice, /*sq_ass_slice*/ }; /* -------------------------------------------------------------- */ /* Code to access Wrapper objects as mappings */ static PyObject * Wrapper_subscript(Wrapper *self, PyObject *key) { return CallMethodO(OBJECT(self),py__getitem__,Build("(O)", key),NULL); } static int Wrapper_ass_sub(Wrapper *self, PyObject *key, PyObject *v) { if (v) { UNLESS(v=CallMethodO(OBJECT(self),py__setitem__, Build("(OO)", key, v),NULL)) return -1; } else { UNLESS(v=CallMethodO(OBJECT(self),py__delitem__, Build("(O)", key),NULL)) return -1; } Py_XDECREF(v); return 0; } static PyMappingMethods Wrapper_as_mapping = { (inquiry)Wrapper_length, /*mp_length*/ (binaryfunc)Wrapper_subscript, /*mp_subscript*/ (objobjargproc)Wrapper_ass_sub, /*mp_ass_subscript*/ }; /* -------------------------------------------------------------- */ /* Code to access Wrapper objects as numbers */ static PyObject * Wrapper_sub(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__sub__,Build("(O)", o),NULL); } static PyObject * Wrapper_div(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__div__,Build("(O)", o),NULL); } static PyObject * Wrapper_mod(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__mod__,Build("(O)", o),NULL); } static PyObject * Wrapper_divmod(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__divmod__,Build("(O)", o),NULL); } static PyObject * Wrapper_pow(Wrapper *self, PyObject *o, PyObject *m) { return CallMethodO(OBJECT(self),py__pow__,Build("(OO)", o, m),NULL); } static PyObject * Wrapper_neg(Wrapper *self) { return CallMethodO(OBJECT(self), py__neg__, NULL, NULL); } static PyObject * Wrapper_pos(Wrapper *self) { return CallMethodO(OBJECT(self), py__pos__, NULL, NULL); } static PyObject * Wrapper_abs(Wrapper *self) { return CallMethodO(OBJECT(self), py__abs__, NULL, NULL); } static PyObject * Wrapper_invert(Wrapper *self) { return CallMethodO(OBJECT(self), py__invert__, NULL, NULL); } static PyObject * Wrapper_lshift(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__lshift__,Build("(O)", o),NULL); } static PyObject * Wrapper_rshift(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__rshift__,Build("(O)", o),NULL); } static PyObject * Wrapper_and(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__and__,Build("(O)", o),NULL); } static PyObject * Wrapper_xor(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__xor__,Build("(O)", o),NULL); } static PyObject * Wrapper_or(Wrapper *self, PyObject *o) { return CallMethodO(OBJECT(self),py__or__,Build("(O)", o),NULL); } static int Wrapper_coerce(Wrapper **self, PyObject **o) { PyObject *m; UNLESS (m=PyObject_GetAttr(OBJECT(*self), py__coerce__)) { PyErr_Clear(); Py_INCREF(*self); Py_INCREF(*o); return 0; } ASSIGN(m, PyObject_CallFunction(m, "O", *o)); UNLESS (m) return -1; UNLESS (PyArg_ParseTuple(m,"OO", self, o)) goto err; Py_INCREF(*self); Py_INCREF(*o); Py_DECREF(m); return 0; err: Py_DECREF(m); return -1; } static PyObject * Wrapper_int(Wrapper *self) { return CallMethodO(OBJECT(self), py__int__, NULL, NULL); } static PyObject * Wrapper_long(Wrapper *self) { return CallMethodO(OBJECT(self), py__long__, NULL, NULL); } static PyObject * Wrapper_float(Wrapper *self) { return CallMethodO(OBJECT(self), py__float__, NULL, NULL); } static PyObject * Wrapper_oct(Wrapper *self) { return CallMethodO(OBJECT(self), py__oct__, NULL, NULL); } static PyObject * Wrapper_hex(Wrapper *self) { return CallMethodO(OBJECT(self), py__hex__, NULL, NULL); } static int Wrapper_nonzero(Wrapper *self) { long l; PyObject *r; UNLESS(r=PyObject_GetAttr(OBJECT(self), py__nonzero__)) { PyErr_Clear(); /* Try len */ UNLESS(r=PyObject_GetAttr(OBJECT(self), py__len__)) { /* No len, it's true :-) */ PyErr_Clear(); return 1; } } UNLESS_ASSIGN(r,PyObject_CallObject(r,NULL)) return -1; l=PyInt_AsLong(r); Py_DECREF(r); return l; } static PyNumberMethods Wrapper_as_number = { (binaryfunc)Wrapper_add, /*nb_add*/ (binaryfunc)Wrapper_sub, /*nb_subtract*/ (binaryfunc)Wrapper_mul, /*nb_multiply*/ (binaryfunc)Wrapper_div, /*nb_divide*/ (binaryfunc)Wrapper_mod, /*nb_remainder*/ (binaryfunc)Wrapper_divmod, /*nb_divmod*/ (ternaryfunc)Wrapper_pow, /*nb_power*/ (unaryfunc)Wrapper_neg, /*nb_negative*/ (unaryfunc)Wrapper_pos, /*nb_positive*/ (unaryfunc)Wrapper_abs, /*nb_absolute*/ (inquiry)Wrapper_nonzero, /*nb_nonzero*/ (unaryfunc)Wrapper_invert, /*nb_invert*/ (binaryfunc)Wrapper_lshift, /*nb_lshift*/ (binaryfunc)Wrapper_rshift, /*nb_rshift*/ (binaryfunc)Wrapper_and, /*nb_and*/ (binaryfunc)Wrapper_xor, /*nb_xor*/ (binaryfunc)Wrapper_or, /*nb_or*/ (coercion)Wrapper_coerce, /*nb_coerce*/ (unaryfunc)Wrapper_int, /*nb_int*/ (unaryfunc)Wrapper_long, /*nb_long*/ (unaryfunc)Wrapper_float, /*nb_float*/ (unaryfunc)Wrapper_oct, /*nb_oct*/ (unaryfunc)Wrapper_hex, /*nb_hex*/ }; /* -------------------------------------------------------- */ static char *acquire_args[] = {"object", "name", "filter", "extra", "explicit", "default", "containment", NULL}; static PyObject * Wrapper_acquire_method(Wrapper *self, PyObject *args, PyObject *kw) { PyObject *name, *filter=0, *extra=Py_None; PyObject *expl=0, *defalt=0; int explicit=1; int containment=0; PyObject *result; /* DM 2005-08-25: argument "default" ignored */ UNLESS (PyArg_ParseTupleAndKeywords( args, kw, "O|OOOOi", acquire_args+1, &name, &filter, &extra, &explicit, &defalt, &containment )) return NULL; if (expl) explicit=PyObject_IsTrue(expl); if (filter==Py_None) filter=0; /* DM 2005-08-25: argument "default" ignored -- fix it! */ # if 0 return Wrapper_findattr(self,name,filter,extra,OBJECT(self),1, explicit || self->ob_type==(PyTypeObject*)&Wrappertype, explicit, containment); # else result = Wrapper_findattr(self,name,filter,extra,OBJECT(self),1, explicit || self->ob_type==(PyTypeObject*)&Wrappertype, explicit, containment); if (result == NULL && defalt != NULL) { /* as "Python/bltinmodule.c:builtin_getattr" turn only 'AttributeError' into a default value, such that e.g. "ConflictError" and errors raised by the filter are not mapped to the default value. */ if (PyErr_ExceptionMatches(PyExc_AttributeError)) { PyErr_Clear(); Py_INCREF(defalt); result = defalt; } } return result; # endif } static PyObject * Wrapper_inContextOf(Wrapper *self, PyObject *args) { PyObject *subob, *o, *c; int inner=1; UNLESS(PyArg_ParseTuple(args,"O|i",&o,&inner)) return NULL; if (inner) { /* subob = self */ subob = OBJECT(self); /* o = aq_base(o) */ while (isWrapper(o) && WRAPPER(o)->obj) o=WRAPPER(o)->obj; /* while 1: */ while (1) { /* if aq_base(subob) is o: return 1 */ c = subob; while (isWrapper(c) && WRAPPER(c)->obj) c = WRAPPER(c)->obj; if (c == o) return PyInt_FromLong(1); /* self = aq_inner(subob) */ /* if self is None: break */ if (isWrapper(subob)) { self = WRAPPER(subob); while (self->obj && isWrapper(self->obj)) self = WRAPPER(self->obj); } else break; /* subob = aq_parent(self) */ /* if subob is None: break */ if (self->container) subob = self->container; else break; } } else { /* Follow wrappers instead. */ c = OBJECT(self); while (1) { if (c==o) return PyInt_FromLong(1); if (c && isWrapper(c)) c=WRAPPER(c)->container; else break; } } return PyInt_FromLong(0); } PyObject * Wrappers_are_not_picklable(PyObject *wrapper, PyObject *args) { PyErr_SetString(PyExc_TypeError, "Can't pickle objects in acquisition wrappers."); return NULL; } static struct PyMethodDef Wrapper_methods[] = { {"acquire", (PyCFunction)Wrapper_acquire_method, METH_VARARGS|METH_KEYWORDS, "Get an attribute, acquiring it if necessary"}, {"aq_acquire", (PyCFunction)Wrapper_acquire_method, METH_VARARGS|METH_KEYWORDS, "Get an attribute, acquiring it if necessary"}, {"aq_inContextOf", (PyCFunction)Wrapper_inContextOf, METH_VARARGS, "Test whether the object is currently in the context of the argument"}, {"__getstate__", (PyCFunction)Wrappers_are_not_picklable, METH_VARARGS, "Wrappers are not picklable"}, {"__reduce__", (PyCFunction)Wrappers_are_not_picklable, METH_VARARGS, "Wrappers are not picklable"}, {"__reduce_ex__", (PyCFunction)Wrappers_are_not_picklable, METH_VARARGS, "Wrappers are not picklable"}, {NULL, NULL} /* sentinel */ }; static PyExtensionClass Wrappertype = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ "ImplicitAcquirerWrapper", /*tp_name*/ sizeof(Wrapper), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)Wrapper_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)0, /*tp_getattr*/ (setattrfunc)0, /*tp_setattr*/ (cmpfunc)Wrapper_compare, /*tp_compare*/ (reprfunc)Wrapper_repr, /*tp_repr*/ &Wrapper_as_number, /*tp_as_number*/ &Wrapper_as_sequence, /*tp_as_sequence*/ &Wrapper_as_mapping, /*tp_as_mapping*/ (hashfunc)Wrapper_hash, /*tp_hash*/ (ternaryfunc)Wrapper_call, /*tp_call*/ (reprfunc)Wrapper_str, /*tp_str*/ (getattrofunc)Wrapper_getattro, /*tp_getattr with object key*/ (setattrofunc)Wrapper_setattro, /*tp_setattr with object key*/ /* tp_as_buffer */ 0, /* tp_flags */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC , "Wrapper object for implicit acquisition", /* Documentation string */ /* tp_traverse */ (traverseproc)Wrapper_traverse, /* tp_clear */ (inquiry)Wrapper_clear, /* tp_richcompare */ (richcmpfunc)0, /* tp_weaklistoffset */ (long)0, /* tp_iter */ (getiterfunc)0, /* tp_iternext */ (iternextfunc)0, /* tp_methods */ Wrapper_methods, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ (descrgetfunc)Wrapper_descrget, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ (initproc)Wrapper__init__, }; static PyExtensionClass XaqWrappertype = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ "ExplicitAcquirerWrapper", /*tp_name*/ sizeof(Wrapper), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)Wrapper_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)0, /*tp_getattr*/ (setattrfunc)0, /*tp_setattr*/ (cmpfunc)Wrapper_compare, /*tp_compare*/ (reprfunc)Wrapper_repr, /*tp_repr*/ &Wrapper_as_number, /*tp_as_number*/ &Wrapper_as_sequence, /*tp_as_sequence*/ &Wrapper_as_mapping, /*tp_as_mapping*/ (hashfunc)Wrapper_hash, /*tp_hash*/ (ternaryfunc)Wrapper_call, /*tp_call*/ (reprfunc)Wrapper_str, /*tp_str*/ (getattrofunc)Xaq_getattro, /*tp_getattr with object key*/ (setattrofunc)Wrapper_setattro, /*tp_setattr with object key*/ /* tp_as_buffer */ 0, /* tp_flags */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC , "Wrapper object for implicit acquisition", /* Documentation string */ /* tp_traverse */ (traverseproc)Wrapper_traverse, /* tp_clear */ (inquiry)Wrapper_clear, /* tp_richcompare */ (richcmpfunc)0, /* tp_weaklistoffset */ (long)0, /* tp_iter */ (getiterfunc)0, /* tp_iternext */ (iternextfunc)0, /* tp_methods */ Wrapper_methods, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ (descrgetfunc)Wrapper_descrget, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ (initproc)Wrapper__init__, }; static PyObject * acquire_of(PyObject *self, PyObject *args) { PyObject *inst; UNLESS(PyArg_ParseTuple(args, "O", &inst)) return NULL; UNLESS(PyExtensionInstance_Check(inst)) { PyErr_SetString(PyExc_TypeError, "attempt to wrap extension method using an object that\n" "is not an extension class instance."); return NULL; } return newWrapper(self, inst, (PyTypeObject *)&Wrappertype); } static PyObject * xaq_of(PyObject *self, PyObject *args) { PyObject *inst; UNLESS(PyArg_ParseTuple(args, "O", &inst)) return NULL; UNLESS(PyExtensionInstance_Check(inst)) { PyErr_SetString(PyExc_TypeError, "attempt to wrap extension method using an object that\n" "is not an extension class instance."); return NULL; } return newWrapper(self, inst, (PyTypeObject *)&XaqWrappertype); } static struct PyMethodDef Acquirer_methods[] = { {"__of__",(PyCFunction)acquire_of, METH_VARARGS, "__of__(context) -- return the object in a context"}, {NULL, NULL} /* sentinel */ }; static struct PyMethodDef Xaq_methods[] = { {"__of__",(PyCFunction)xaq_of, METH_VARARGS,""}, {NULL, NULL} /* sentinel */ }; static PyObject * capi_aq_acquire(PyObject *self, PyObject *name, PyObject *filter, PyObject *extra, int explicit, PyObject *defalt, int containment) { PyObject *result; if (filter==Py_None) filter=0; /* We got a wrapped object, so business as usual */ if (isWrapper(self)) return Wrapper_findattr( WRAPPER(self), name, filter, extra, OBJECT(self),1, explicit || WRAPPER(self)->ob_type==(PyTypeObject*)&Wrappertype, explicit, containment); /* Not wrapped and no filter, so just getattr */ if (! filter) return PyObject_GetAttr(self, name); /* Crap, we've got to construct a wrapper so we can use Wrapper_findattr */ UNLESS (self=newWrapper(self, Py_None, (PyTypeObject*)&Wrappertype)) return NULL; result=Wrapper_findattr(WRAPPER(self), name, filter, extra, OBJECT(self), 1, 1, explicit, containment); /* get rid of temp wrapper */ Py_DECREF(self); return result; } static PyObject * module_aq_acquire(PyObject *ignored, PyObject *args, PyObject *kw) { PyObject *self; PyObject *name, *filter=0, *extra=Py_None; PyObject *expl=0, *defalt=0; int explicit=1, containment=0; UNLESS (PyArg_ParseTupleAndKeywords( args, kw, "OO|OOOOi", acquire_args, &self, &name, &filter, &extra, &expl, &defalt, &containment )) return NULL; if (expl) explicit=PyObject_IsTrue(expl); return capi_aq_acquire(self, name, filter, extra, explicit, defalt, containment); } static PyObject * capi_aq_get(PyObject *self, PyObject *name, PyObject *defalt, int containment) { PyObject *result = NULL; /* We got a wrapped object, so business as usual */ if (isWrapper(self)) result=Wrapper_findattr(WRAPPER(self), name, 0, 0, OBJECT(self), 1, 1, 1, containment); else result=PyObject_GetAttr(self, name); if (! result && defalt) { PyErr_Clear(); result=defalt; Py_INCREF(result); } return result; } static PyObject * module_aq_get(PyObject *r, PyObject *args) { PyObject *self, *name, *defalt=0; int containment=0; UNLESS (PyArg_ParseTuple(args, "OO|Oi", &self, &name, &defalt, &containment )) return NULL; return capi_aq_get(self, name, defalt, containment); } static int capi_aq_iswrapper(PyObject *self) { return isWrapper(self); } static PyObject * capi_aq_base(PyObject *self) { PyObject *result; if (! isWrapper(self)) { Py_INCREF(self); return self; } if (WRAPPER(self)->obj) { result=WRAPPER(self)->obj; while (isWrapper(result) && WRAPPER(result)->obj) result=WRAPPER(result)->obj; } else result=Py_None; Py_INCREF(result); return result; } static PyObject * module_aq_base(PyObject *ignored, PyObject *args) { PyObject *self; UNLESS (PyArg_ParseTuple(args, "O", &self)) return NULL; return capi_aq_base(self); } static PyObject * capi_aq_parent(PyObject *self) { PyObject *result=Py_None; if (isWrapper(self) && WRAPPER(self)->container) result=WRAPPER(self)->container; Py_INCREF(result); return result; } static PyObject * module_aq_parent(PyObject *ignored, PyObject *args) { PyObject *self; UNLESS (PyArg_ParseTuple(args, "O", &self)) return NULL; return capi_aq_parent(self); } static PyObject * capi_aq_self(PyObject *self) { PyObject *result; if (! isWrapper(self)) { Py_INCREF(self); return self; } if (WRAPPER(self)->obj) result=WRAPPER(self)->obj; else result=Py_None; Py_INCREF(result); return result; } static PyObject * module_aq_self(PyObject *ignored, PyObject *args) { PyObject *self; UNLESS (PyArg_ParseTuple(args, "O", &self)) return NULL; return capi_aq_self(self); } static PyObject * capi_aq_inner(PyObject *self) { PyObject *result; if (! isWrapper(self)) { Py_INCREF(self); return self; } if (WRAPPER(self)->obj) { result=WRAPPER(self)->obj; while (isWrapper(result) && WRAPPER(result)->obj) { self=result; result=WRAPPER(result)->obj; } result=self; } else result=Py_None; Py_INCREF(result); return result; } static PyObject * module_aq_inner(PyObject *ignored, PyObject *args) { PyObject *self; UNLESS (PyArg_ParseTuple(args, "O", &self)) return NULL; return capi_aq_inner(self); } static PyObject * capi_aq_chain(PyObject *self, int containment) { PyObject *result; UNLESS (result=PyList_New(0)) return NULL; while (1) { if (isWrapper(self)) { if (WRAPPER(self)->obj) { if (containment) while (WRAPPER(self)->obj && isWrapper(WRAPPER(self)->obj)) self=WRAPPER(self)->obj; if (PyList_Append(result,OBJECT(self)) < 0) goto err; } if (WRAPPER(self)->container) { self=WRAPPER(self)->container; continue; } } else if (PyList_Append(result, self) < 0) goto err; break; } return result; err: Py_DECREF(result); return result; } static PyObject * module_aq_chain(PyObject *ignored, PyObject *args) { PyObject *self; int containment=0; UNLESS (PyArg_ParseTuple(args, "O|i", &self, &containment)) return NULL; return capi_aq_chain(self, containment); } static struct PyMethodDef methods[] = { {"aq_acquire", (PyCFunction)module_aq_acquire, METH_VARARGS|METH_KEYWORDS, "aq_acquire(ob, name [, filter, extra, explicit]) -- " "Get an attribute, acquiring it if necessary" }, {"aq_get", (PyCFunction)module_aq_get, METH_VARARGS, "aq_get(ob, name [, default]) -- " "Get an attribute, acquiring it if necessary." }, {"aq_base", (PyCFunction)module_aq_base, METH_VARARGS, "aq_base(ob) -- Get the object unwrapped"}, {"aq_parent", (PyCFunction)module_aq_parent, METH_VARARGS, "aq_parent(ob) -- Get the parent of an object"}, {"aq_self", (PyCFunction)module_aq_self, METH_VARARGS, "aq_self(ob) -- Get the object with the outermost wrapper removed"}, {"aq_inner", (PyCFunction)module_aq_inner, METH_VARARGS, "aq_inner(ob) -- " "Get the object with alll but the innermost wrapper removed"}, {"aq_chain", (PyCFunction)module_aq_chain, METH_VARARGS, "aq_chain(ob [, containment]) -- " "Get a list of objects in the acquisition environment"}, {NULL, NULL} }; void init_Acquisition(void) { PyObject *m, *d; PyObject *api; PURE_MIXIN_CLASS(Acquirer, "Base class for objects that implicitly" " acquire attributes from containers\n" , Acquirer_methods); PURE_MIXIN_CLASS(ExplicitAcquirer, "Base class for objects that explicitly" " acquire attributes from containers\n" , Xaq_methods); UNLESS(ExtensionClassImported) return; UNLESS(Acquired=PyString_FromStringAndSize(NULL,42)) return; strcpy(PyString_AsString(Acquired), ""); /* Create the module and add the functions */ m = Py_InitModule4("_Acquisition", methods, "Provide base classes for acquiring objects\n\n" "$Id: _Acquisition.c 38586 2005-09-24 12:10:07Z andreasjung $\n", OBJECT(NULL),PYTHON_API_VERSION); d = PyModule_GetDict(m); init_py_names(); PyExtensionClass_Export(d,"Acquirer",AcquirerType); PyExtensionClass_Export(d,"ImplicitAcquisitionWrapper",Wrappertype); PyExtensionClass_Export(d,"ExplicitAcquirer",ExplicitAcquirerType); PyExtensionClass_Export(d,"ExplicitAcquisitionWrapper",XaqWrappertype); /* Create aliases */ PyDict_SetItemString(d,"Implicit",OBJECT(&AcquirerType)); PyDict_SetItemString(d,"Explicit",OBJECT(&ExplicitAcquirerType)); PyDict_SetItemString(d,"Acquired",Acquired); AcquisitionCAPI.AQ_Acquire = capi_aq_acquire; AcquisitionCAPI.AQ_Get = capi_aq_get; AcquisitionCAPI.AQ_IsWrapper = capi_aq_iswrapper; AcquisitionCAPI.AQ_Base = capi_aq_base; AcquisitionCAPI.AQ_Parent = capi_aq_parent; AcquisitionCAPI.AQ_Self = capi_aq_self; AcquisitionCAPI.AQ_Inner = capi_aq_inner; AcquisitionCAPI.AQ_Chain = capi_aq_chain; api = PyCObject_FromVoidPtr(&AcquisitionCAPI, NULL); PyDict_SetItemString(d, "AcquisitionCAPI", api); Py_DECREF(api); } Zope-2.8.7-final/lib/python/Interface/0000775000076400007640000000000010435610641016527 5ustar ajungajungZope-2.8.7-final/lib/python/Interface/Method.py0000664000076400007640000000551210435500320020314 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Method interfaces Revision information: $Id: Method.py 24763 2004-05-17 05:59:28Z philikon $ """ import Exceptions from Attribute import Attribute sig_traits = ['positional', 'required', 'optional', 'varargs', 'kwargs'] CO_VARARGS = 4 CO_VARKEYWORDS = 8 class Method(Attribute): """Method interfaces The idea here is that you have objects that describe methods. This provides an opportunity for rich meta-data. """ # We can't say this yet because we don't have enough # infrastructure in place. # #__implements__ = IMethod interface='' def __call__(self, *args, **kw): raise Exceptions.BrokenImplementation(self.interface, self.__name__) def getSignatureInfo(self): info = {} for t in sig_traits: info[t] = getattr(self, t) return info def getSignatureString(self): sig = "(" for v in self.positional: sig = sig + v if v in self.optional.keys(): sig = sig + "=%s" % `self.optional[v]` sig = sig + ", " if self.varargs: sig = sig + ("*%s, " % self.varargs) if self.kwargs: sig = sig + ("**%s, " % self.kwargs) # slice off the last comma and space if self.positional or self.varargs or self.kwargs: sig = sig[:-2] sig = sig + ")" return sig def fromFunction(func, interface='', imlevel=0): m=Method(func.__name__, func.__doc__) defaults=func.func_defaults or () c=func.func_code na=c.co_argcount-imlevel names=c.co_varnames[imlevel:] d={} nr=na-len(defaults) if nr < 0: defaults=defaults[-nr:] nr=0 for i in range(len(defaults)): d[names[i+nr]]=defaults[i] m.positional=names[:na] m.required=names[:nr] m.optional=d argno = na if c.co_flags & CO_VARARGS: m.varargs = names[argno] argno = argno + 1 else: m.varargs = None if c.co_flags & CO_VARKEYWORDS: m.kwargs = names[argno] else: m.kwargs = None m.interface=interface return m def fromMethod(meth, interface=''): func = meth.im_func return fromFunction(func, interface, imlevel=1) Zope-2.8.7-final/lib/python/Interface/_Element.py0000664000076400007640000000345110435500320020624 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: _Element.py 24763 2004-05-17 05:59:28Z philikon $ """ from _object import object class Element(object): # We can't say this yet because we don't have enough # infrastructure in place. # #__implements__ = IElement def __init__(self, __name__=None, __doc__=''): """Create an 'attribute' description """ if not __doc__ and __name__ and __name__.find(' ') >= 0: __doc__ = __name__ __name__ = None self.__name__=__name__ self.__doc__=__doc__ self.__tagged_values = {} def getName(self): """ Returns the name of the object. """ return self.__name__ def getDoc(self): """ Returns the documentation for the object. """ return self.__doc__ def getTaggedValue(self, tag): """ Returns the value associated with 'tag'. """ return self.__tagged_values[tag] def getTaggedValueTags(self): """ Returns a list of all tags. """ return self.__tagged_values.keys() def setTaggedValue(self, tag, value): """ Associates 'value' with 'key'. """ self.__tagged_values[tag] = value Zope-2.8.7-final/lib/python/Interface/Exceptions.py0000664000076400007640000000356310435500320021221 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## class DoesNotImplement(Exception): """ This object does not implement """ def __init__(self, interface): self.interface = interface def __str__(self): return """An object does not implement interface %(interface)s """ % self.__dict__ class BrokenImplementation(Exception): """An attribute is not completely implemented. """ def __init__(self, interface, name): self.interface=interface self.name=name def __str__(self): return """An object has failed to implement interface %(interface)s The %(name)s attribute was not provided. """ % self.__dict__ class BrokenMethodImplementation(Exception): """An method is not completely implemented. """ def __init__(self, method, mess): self.method=method self.mess=mess def __str__(self): return """The implementation of %(method)s violates its contract because %(mess)s. """ % self.__dict__ class InvalidInterface(Exception): """The interface has invalid contents """ class BadImplements(TypeError): """An implementation assertion is invalid because it doesn't contain an interface or a sequence of valid implementation assertions. """ Zope-2.8.7-final/lib/python/Interface/_Interface.py0000664000076400007640000000242710435500320021135 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Interface object implementation Revision information: $Id: _Interface.py 24763 2004-05-17 05:59:28Z philikon $ """ from _InterfaceClass import Interface as InterfaceClass Interface = InterfaceClass("Interface") # Now we can create the interesting interfaces and wire them up: def wire(): from Implements import implements from Attribute import Attribute from IAttribute import IAttribute implements(Attribute, IAttribute) from Method import Method from IMethod import IMethod implements(Method, IMethod) from IInterface import IInterface implements(InterfaceClass, IInterface) wire() del wire del InterfaceClass Zope-2.8.7-final/lib/python/Interface/IAttribute.py0000664000076400007640000000146710435500320021155 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: IAttribute.py 24763 2004-05-17 05:59:28Z philikon $ """ from IElement import IElement class IAttribute(IElement): """Attribute descriptors""" Zope-2.8.7-final/lib/python/Interface/__init__.py0000664000076400007640000000446510435500320020641 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Interfaces This package implements the Python "scarecrow" proposal. The package exports a single name, 'Interface' directly. Interface is used to create an interface with a class statement, as in: from Interface import Interface class IMyInterface(Interface): '''Interface documentation ''' def meth(arg1, arg2): '''Documentation for meth ''' # Note that there is no self argument To find out what you can do with interfaces, see the interface interface, IInterface in the IInterface module. The package has several public modules: o Attribute has the implementation for interface attributes for people who want to build interfaces by hand. (Maybe someone should cry YAGNI for this. ;) o Document has a utility for documenting an interface as structured text. o Exceptions has the interface-defined exceptions o IAttribute defines the attribute descriptor interface. o IElement defined the base interface for IAttribute, IInterface, and IMethod. o IInterface defines the interface interface o IMethod defined the method interface. o Implements has various utilities for examining interface assertions. o Method has the implementation for interface methods. See above. o Verify has utilities for verifying (sort of) interfaces. See the module doc strings for more information. There is also a script, pyself.py in the package that can be used to create interface skeletins. Run it without arguments to get documentation. Revision information: $Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $ """ from _Interface import Interface from Attribute import Attribute Base = Interface # XXX We need to stamp out Base usage Zope-2.8.7-final/lib/python/Interface/Verify.py0000664000076400007640000000624610435500320020345 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from Exceptions import BrokenImplementation, DoesNotImplement from Exceptions import BrokenMethodImplementation from types import FunctionType from Method import fromMethod, fromFunction from _object import MethodTypes def _verify(iface, candidate, tentative=0, vtype=None): """ Verify that 'candidate' might correctly implements 'iface'. This involves: o Making sure the candidate defines all the necessary methods o Making sure the methods have the correct signature o Making sure the candidate asserts that it implements the interface Note that this isn't the same as verifying that the class does implement the interface. If optional tentative is true, suppress the "is implemented by" test. """ if vtype is 'c': tester = iface.isImplementedByInstancesOf else: tester = iface.isImplementedBy if not tentative and not tester( candidate ): raise DoesNotImplement(iface) for n, d in iface.namesAndDescriptions(1): if not hasattr(candidate, n): raise BrokenImplementation(iface, n) attr = getattr(candidate, n) if type(attr) is FunctionType: # should never get here meth = fromFunction(attr, n) elif type(attr) in MethodTypes: meth = fromMethod(attr, n) else: continue # must be an attribute... d=d.getSignatureInfo() meth = meth.getSignatureInfo() mess = _incompat(d, meth) if mess: raise BrokenMethodImplementation(n, mess) return 1 def verifyClass(iface, candidate, tentative=0): return _verify(iface, candidate, tentative, vtype='c') def verifyObject(iface, candidate, tentative=0): return _verify(iface, candidate, tentative, vtype='o') def _incompat(required, implemented): #if (required['positional'] != # implemented['positional'][:len(required['positional'])] # and implemented['kwargs'] is None): # return 'imlementation has different argument names' if len(implemented['required']) > len(required['required']): return 'implementation requires too many arguments' if ((len(implemented['positional']) < len(required['positional'])) and not implemented['varargs']): return "implementation doesn't allow enough arguments" if required['kwargs'] and not implemented['kwargs']: return "implementation doesn't support keyword arguments" if required['varargs'] and not implemented['varargs']: return "implementation doesn't support variable arguments" Zope-2.8.7-final/lib/python/Interface/pyskel.py0000664000076400007640000001323210435500320020401 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Generate method skeletins for intefaces. Usage: python pyskel.py dotted_name Example: cd lib/python python Interface/pyskel.py Zope2.App.Security.IRoleService.IRoleService The dotted name is the module name and interface object name connected with a dot. Revision information: $Id: pyskel.py 28991 2005-01-31 15:45:46Z jim $ """ import sys, os, re sys.path.insert(0, os.getcwd()) from _object import isInstance from types import ModuleType from Interface.Method import Method from Interface.Attribute import Attribute class_re = re.compile(r'\s*class\s+([a-zA-Z_][a-zA-Z0-9_]*)') def_re = re.compile(r'\s*def\s+([a-zA-Z_][a-zA-Z0-9_]*)') attr_re = re.compile(r'\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*Attribute') def rskel(iface, print_iface=1): name = "%s.%s" % (iface.__module__, iface.__name__) file = resolve(iface.__module__).__file__ if file.endswith('pyc'): file = file[:-1] order = guessOrder(open(file)) namesAndDescriptions = getAttributesInOrder(iface, order) if namesAndDescriptions and print_iface: print print " ######################################" print " # from:", name for aname, ades in namesAndDescriptions: if isInstance(ades, Method): sig = ades.getSignatureString()[1:-1] if sig: sig = "self, %s" % sig else: sig = "self" print print " def %s(%s):" % (aname, sig) print " 'See %s'" % name elif isInstance(ades, Attribute): print print " # See %s" % name print " %s = None" %aname for base in iface.__bases__: if base.__name__ not in ('Interface',): rskel(base) def skel(name): iface = resolve(name) class_name = iface.__name__ if class_name.startswith('I'): class_name = class_name[1:] print "from %s import %s" % (iface.__module__, iface.__name__) print print "class %s:" %class_name print print " __implements__ = ", iface.__name__ print print " ############################################################" print " # Implementation methods for interface" print " #", name rskel(iface, 0) print print " #" print " ############################################################" def resolve(name, _silly=('__doc__',), _globals={}): # Support for file path syntax; this way I can use TAB to search for # the module. if '/' in name or name.endswith('.py'): # We got a relative path. Let's try to get the full one and then # make a package path out of it. if not name.startswith('/'): cwd = os.getcwd() for path in sys.path[1:]: # Yeah, we need to exclude the cwd itself if path != '' and cwd.startswith(path): name = os.path.join(cwd[len(path)+1:], name) name = os.path.normpath(name) break # get rid of the file ending :) if name.endswith('.py'): name = name[:-3] name = name.replace('/', '.') # Now to the regular lookup if name[:1]=='.': name='ZopeProducts'+name if name[-1:] == '.': name = name[:-1] repeat = 1 else: repeat = 0 names=name.split('.') last=names[-1] mod='.'.join(names[:-1]) while 1: m=__import__(mod, _globals, _globals, _silly) try: a=getattr(m, last) except AttributeError: pass else: if not repeat or (type(a) is not ModuleType): return a mod += '.' + last def guessOrder(source_file): order = {} # { class name -> list of methods } lines = source_file.readlines() class_name = None for line in lines: m = class_re.match(line) if m and m.groups(): class_name = m.groups()[0] else: for m in (def_re.match(line), attr_re.match(line)): if m and m.groups(): def_name = m.groups()[0] name_order = order.get(class_name) if name_order is None: name_order = [] order[class_name] = name_order name_order.append(def_name) return order def getAttributesInOrder(interface, order): # order is the dictionary returned from guessOrder(). # interface is a metaclass-based interface object. name_order = order.get(interface.__name__) if name_order is None: # Something's wrong. Oh well. return interface.__dict__.items() else: items = [] for key, value in interface.namesAndDescriptions(): if key in name_order: items.append((name_order.index(key), key, value)) else: items.append((99999, key, value)) # Go to end. items.sort() return map(lambda item: item[1:], items) if __name__ == '__main__': for a in sys.argv[1:]: skel(a) Zope-2.8.7-final/lib/python/Interface/Attribute.py0000664000076400007640000000167110435500320021041 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: Attribute.py 24763 2004-05-17 05:59:28Z philikon $ """ from _Element import Element class Attribute(Element): """Attribute descriptions """ # We can't say this yet because we don't have enough # infrastructure in place. # #__implements__ = IAttribute Zope-2.8.7-final/lib/python/Interface/bridge.py0000664000076400007640000000645210435500320020334 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Z3 -> Z2 bridge utilities. $Id: bridge.py 37774 2005-08-07 18:07:31Z yuppie $ """ from Interface._InterfaceClass import Interface as Z2_InterfaceClass from Interface import Interface as Z2_Interface from Interface import Attribute as Z2_Attribute from Interface.Method import Method as Z2_Method from zope.interface.interface import InterfaceClass as Z3_InterfaceClass from zope.interface.interface import Interface as Z3_Interface from zope.interface.interface import Attribute as Z3_Attribute from zope.interface.interface import Method as Z3_Method _bridges = {Z3_Interface: Z2_Interface} def fromZ3Interface(z3i): """ Return a Zope 2 interface corresponding to 'z3i'. o 'z3i' must be a Zope 3 interface. """ if not isinstance(z3i, Z3_InterfaceClass): raise ValueError, 'Not a Zope 3 interface!' if z3i in _bridges: return _bridges[z3i] name = z3i.getName() bases = [ fromZ3Interface(x) for x in z3i.getBases() ] attrs = {} for k, v in z3i.namesAndDescriptions(): if isinstance(v, Z3_Method): v = fromZ3Method(v) elif isinstance(v, Z3_Attribute): v = fromZ3Attribute(v) attrs[k] = v # XXX: Note that we pass the original interface's __module__; # we may live to regret that. z2i = Z2_InterfaceClass(name=name, bases=tuple(bases), attrs=attrs, __doc__=z3i.getDoc(), __module__=z3i.__module__) _bridges[z3i] = z2i return z2i def fromZ3Attribute(z3a): """ Return a Zope 2 interface attribute corresponding to 'z3a'. o 'z3a' must be a Zope 3 interface attribute. """ if not isinstance(z3a, Z3_Attribute): raise ValueError, 'Not a Zope 3 interface attribute!' return Z2_Attribute(z3a.getName(), z3a.getDoc()) def fromZ3Method(z3m): """ Return a Zope 2 interface method corresponding to 'z3m'. o 'z3m' must be a Zope 3 interface method. """ if not isinstance(z3m, Z3_Method): raise ValueError, 'Not a Zope 3 interface method!' z2m = Z2_Method(z3m.getName(), z3m.getDoc()) sig = z3m.getSignatureInfo() z2m.positional = sig['positional'] z2m.required = sig['required'] z2m.optional = sig['optional'] z2m.varargs = sig['varargs'] z2m.kwargs = sig['kwargs'] return z2m def createZope3Bridge(zope3, package, name): # Map a Zope 3 interface into a Zope2 interface, seated within 'package' # as 'name'. z2i = fromZ3Interface(zope3) if name is not None: z2i.__dict__['__name__'] = name z2i.__dict__['__module__'] = package.__name__ setattr(package, z2i.getName(), z2i) Zope-2.8.7-final/lib/python/Interface/Document.py0000664000076400007640000000735510435500320020661 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Pretty-Print an Interface object as structured text (Yum) This module provides a function, asStructuredText, for rendering an interface as structured text. Revision information: $Id: Document.py 24763 2004-05-17 05:59:28Z philikon $ """ from string import maketrans def asStructuredText(I, munge=0): """ Output structured text format. Note, this will wack any existing 'structured' format of the text. """ r = ["%s\n\n" % I.getName()] outp = r.append level = 1 if I.getDoc(): outp(_justify_and_indent(_trim_doc_string(I.getDoc()), level)+ "\n\n") if I.getBases(): outp((" " * level) + "This interface extends:\n\n") level = level + 1 for b in I.getBases(): item = "o %s" % b.getName() outp(_justify_and_indent(_trim_doc_string(item), level, munge) + "\n\n") level = level - 1 namesAndDescriptions = list(I.namesAndDescriptions()) namesAndDescriptions.sort() outp(_justify_and_indent("Attributes:", level, munge)+'\n\n') level = level + 1 for name, desc in namesAndDescriptions: if not hasattr(desc, 'getSignatureString'): # ugh... item = "%s -- %s" % (desc.getName(), desc.getDoc() or 'no documentation') outp(_justify_and_indent(_trim_doc_string(item), level, munge) + "\n\n") level = level - 1 outp(_justify_and_indent("Methods:", level, munge)+'\n\n') level = level + 1 for name, desc in namesAndDescriptions: if hasattr(desc, 'getSignatureString'): # ugh... item = "%s%s -- %s" % (desc.getName(), desc.getSignatureString(), desc.getDoc() or 'no documentation') outp(_justify_and_indent(_trim_doc_string(item), level, munge) + "\n\n") return "".join(r) def _trim_doc_string(text): """ Trims a doc string to make it format correctly with structured text. """ text = text.strip().replace('\r\n', '\n') lines = text.split('\n') nlines = [lines[0]] if len(lines) > 1: min_indent=None for line in lines[1:]: indent=len(line) - len(line.lstrip()) if indent < min_indent or min_indent is None: min_indent=indent for line in lines[1:]: nlines.append(line[min_indent:]) return '\n'.join(nlines) _trans = maketrans("\r\n", " ") def _justify_and_indent(text, level, munge=0, width=72): """ indent and justify text, rejustify (munge) if specified """ lines = [] if munge: line = " " * level text = text.translate(text, _trans).strip().split() for word in text: line = ' '.join([line, word]) if len(line) > width: lines.append(line) line = " " * level else: lines.append(line) return "\n".join(lines) else: text = text.replace("\r\n", "\n").split("\n") for line in text: lines.append( (" " * level) + line) return '\n'.join(lines) Zope-2.8.7-final/lib/python/Interface/_InterfaceClass.py0000664000076400007640000001521710435500320022124 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Interface object implementation Revision information: $Id: _InterfaceClass.py 24763 2004-05-17 05:59:28Z philikon $ """ from inspect import currentframe import sys from Method import Method, fromFunction from Attribute import Attribute from types import FunctionType import Exceptions from _Element import Element from _object import isInstance class Interface(Element): """Prototype (scarecrow) Interfaces Implementation """ # We can't say this yet because we don't have enough # infrastructure in place. # #__implements__ = IInterface def __init__(self, name, bases=(), attrs=None, __doc__=None, __module__=None): if __module__ is None: if attrs is not None and attrs.has_key('__module__'): __module__ = attrs['__module__'] del attrs['__module__'] else: try: # Figure out what module defined the interface. # This is how cPython figures out the module of # a class, but of course it does it in C. :-/ __module__ = currentframe().f_back.f_globals['__name__'] except (AttributeError, KeyError): pass self.__module__ = __module__ for b in bases: if not isInstance(b, Interface): raise TypeError, 'Expected base interfaces' self.__bases__=bases if attrs is None: attrs={} if attrs.has_key('__doc__'): if __doc__ is None: __doc__=attrs['__doc__'] del attrs['__doc__'] if __doc__ is not None: self.__doc__=__doc__ else: self.__doc__ = "" Element.__init__(self, name, __doc__) for k, v in attrs.items(): if isInstance(v, Attribute): v.interface=name if not v.__name__: v.__name__ = k elif isinstance(v, FunctionType): attrs[k]=fromFunction(v, name) else: raise Exceptions.InvalidInterface( "Concrete attribute, %s" % k) self.__attrs = attrs def getBases(self): return self.__bases__ def extends(self, other, strict=1): """Does an interface extend another? """ if not strict and self is other: return 1 for b in self.__bases__: if b == other: return 1 if b.extends(other): return 1 return 0 def isEqualOrExtendedBy(self, other): """Same interface or extends? """ if self == other: return 1 return other.extends(self) def isImplementedBy(self, object): """Does the given object implement the interface? """ i = getImplements(object) if i is not None: return visitImplements( i, object, self.isEqualOrExtendedBy, self._getInterface) return 0 def isImplementedByInstancesOf(self, klass): """Do instances of the given class implement the interface? """ i = getImplementsOfInstances(klass) if i is not None: return visitImplements( i, klass, self.isEqualOrExtendedBy, self._getInterface) return 0 def names(self, all=0): """Return the attribute names defined by the interface """ if not all: return self.__attrs.keys() r = {} for name in self.__attrs.keys(): r[name] = 1 for base in self.__bases__: for name in base.names(all): r[name] = 1 return r.keys() def namesAndDescriptions(self, all=0): """Return the attribute names and descriptions defined by the interface """ if not all: return self.__attrs.items() r = {} for name, d in self.__attrs.items(): r[name] = d for base in self.__bases__: for name, d in base.namesAndDescriptions(all): if not r.has_key(name): r[name] = d return r.items() def getDescriptionFor(self, name): """Return the attribute description for the given name """ r = self.queryDescriptionFor(name) if r is not None: return r raise KeyError, name def queryDescriptionFor(self, name, default=None): """Return the attribute description for the given name """ r = self.__attrs.get(name, self) if r is not self: return r for base in self.__bases__: r = base.queryDescriptionFor(name, self) if r is not self: return r return default def deferred(self): """Return a defered class corresponding to the interface """ if hasattr(self, "_deferred"): return self._deferred klass={} exec "class %s: pass" % self.__name__ in klass klass=klass[self.__name__] self.__d(klass.__dict__) self._deferred=klass return klass def _getInterface(self, ob, name): ''' Retrieve a named interface. ''' return None def __d(self, dict): for k, v in self.__attrs.items(): if isInstance(v, Method) and not dict.has_key(k): dict[k]=v for b in self.__bases__: b.__d(dict) def __repr__(self): name = self.__name__ m = self.__module__ if m: name = '%s.%s' % (m, name) return "<%s %s at %x>" % (self.__class__.__name__, name, id(self)) def __reduce__(self): return self.__name__ def __hash__(self): """ interface instances need to be hashable, and inheriting from extensionclass makes instances unhashable unless we declare a __hash__ method here""" return id(self) # We import this here to deal with module dependencies. from Implements import getImplementsOfInstances, visitImplements, getImplements from Implements import instancesOfObjectImplements Zope-2.8.7-final/lib/python/Interface/IMethod.py0000664000076400007640000000174710435500320020433 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: IMethod.py 24763 2004-05-17 05:59:28Z philikon $ """ from IAttribute import IAttribute class IMethod(IAttribute): """Method attributes """ # XXX What the heck should methods provide? Grrrr def getSignatureString(): """Return a signature string suitable for inclusion in documentation. """ Zope-2.8.7-final/lib/python/Interface/iclass.py0000664000076400007640000000070210435500320020346 0ustar ajungajungimport warnings warnings.warn("""\ The Interface.iclass module is no more. This is a stub module to allow ZClasses that subclass ObjectManager to continue to function - please fix your ZClasses (using the 'Subobjects' tab)""", DeprecationWarning) # Old interface object. Provided for backwards compatibility - allows ZClasses # that subclass ObjectManager to be used in 2.6. class Interface: def __init__(self, *args, **kwargs): pass Zope-2.8.7-final/lib/python/Interface/Implements.py0000664000076400007640000001266410435500320021217 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Implemantation assertion facilities. Revision information: $Id: Implements.py 24763 2004-05-17 05:59:28Z philikon $ """ import Exceptions from types import ClassType from Verify import verifyClass from _InterfaceClass import Interface as InterfaceClass from types import TupleType, ClassType, StringType # Special value indicating the object supports # what its class supports. CLASS_INTERFACES = 1 from _object import ClassTypes, isInstance _typeImplements={} def getImplements(object): t = type(object) if t in ClassTypes: if hasattr(object, '__class_implements__'): return object.__class_implements__ elif hasattr(object, '__implements__'): return object.__implements__ return _typeImplements.get(t, None) def getImplementsOfInstances(klass, tiget=_typeImplements.get): if type(klass) in ClassTypes: if hasattr(klass, '__implements__'): return klass.__implements__ else: return None else: return tiget(klass, None) def visitImplements(implements, object, visitor, getInterface=None): """ Visits the interfaces described by an __implements__ attribute, invoking the visitor for each interface object. If the visitor returns anything true, the loop stops. This does not, and should not, visit superinterfaces. """ # this allows us to work with proxy wrappers in Python 2.2, # yet remain compatible with earlier versions of python. implements_class = getattr(implements, '__class__', None) if implements_class == InterfaceClass or \ isInstance(implements, InterfaceClass): return visitor(implements) elif implements == CLASS_INTERFACES: klass = getattr(object, '__class__', None) if klass is not None: i = getImplementsOfInstances(klass) if i: return visitImplements(i, object, visitor, getInterface) elif implements_class == StringType or type(implements) is StringType: if getInterface is not None: # Look up a named interface. i = getInterface(object, implements) if i is not None: return visitImplements(i, object, visitor, getInterface) elif implements_class == TupleType or type(implements) is TupleType: for i in implements: r = visitImplements(i, object, visitor, getInterface) if r: # If the visitor returns anything true, stop. return r else: if implements_class is not None and \ type(implements) != implements_class: raise Exceptions.BadImplements( """__implements__ should be an interface or tuple, not a %s pretending to be a %s""" % (type(implements).__name__, implements_class.__name__) ) raise Exceptions.BadImplements( """__implements__ should be an interface or tuple, not a %s""" % type(implements).__name__) return 0 def assertTypeImplements(type, interfaces): """Assign a set of interfaces to a Python type such as int, str, tuple, list and dict. """ _typeImplements[type]=interfaces def objectImplements(object, getInterface=None): r = [] implements = getImplements(object) if not implements: return r visitImplements(implements, object, r.append, getInterface) return r def instancesOfObjectImplements(klass, getInterface=None): r = [] implements = getImplementsOfInstances(klass) if not implements: return r visitImplements(implements, klass, r.append, getInterface) return r def _flatten(i, append): append(i) bases = i.getBases() if bases: for b in bases: _flatten(b, append) def _detuplize(interface, append): if type(interface) is TupleType: for subinterface in interface: _detuplize(subinterface, append) else: append(interface) def flattenInterfaces(interfaces, remove_duplicates=1): detupledInterfaces = [] for interface in interfaces: _detuplize(interface, detupledInterfaces.append) res = [] for i in detupledInterfaces: _flatten(i, res.append) if remove_duplicates: # Remove duplicates in reverse. # Similar to Python 2.2's method resolution order. seen = {} index = len(res) - 1 while index >= 0: i = res[index] if seen.has_key(i): del res[index] else: seen[i] = 1 index = index - 1 return res def implements(klass, interface, check=1): if check: verifyClass(interface, klass, tentative=1) old=getattr(klass, '__implements__', None) if old is None: klass.__implements__ = interface else: klass.__implements__ = old, interface Zope-2.8.7-final/lib/python/Interface/IElement.py0000664000076400007640000000256210435500320020600 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: IElement.py 24763 2004-05-17 05:59:28Z philikon $ """ from _Interface import Interface from Attribute import Attribute class IElement(Interface): """Objects that have basic documentation and tagged values. """ __name__ = Attribute('__name__', 'The object name') __doc__ = Attribute('__doc__', 'The object doc string') def getName(): """Returns the name of the object.""" def getDoc(): """Returns the documentation for the object.""" def getTaggedValue(tag): """Returns the value associated with 'tag'.""" def getTaggedValueTags(): """Returns a list of all tags.""" def setTaggedValue(tag, value): """Associates 'value' with 'key'.""" Zope-2.8.7-final/lib/python/Interface/Mapping.py0000664000076400007640000000421110435500320020462 0ustar ajungajungimport Basic, Util class Mapping(Basic.Base): "anything supporting __getitem__" def __getitem__(key): """Get the value for the given key Raise a key error if the key if not in the collection. """ class QueryMapping(Mapping): def has_key(key): """Check whether the object has an item with the given key""" def get(key, default=None): """Get the value for the given key Return the default if the key is not in the collection. """ class Sized(Basic.Base): "anything supporting __len" def __len__(): """Return the number of items in the container""" class MutableMapping(Basic.Mutable): "Has __setitem__ and __delitem__" def __setitem__(key, value): """Set the value for the given key""" def __delitem__(key): """delete the value for the given key Raise a key error if the key if not in the collection.""" class EnumerableMapping(Mapping): def keys(): """Return an Sequence containing the keys in the collection The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. """ class MinimalDictionary(QueryMapping, Sized, MutableMapping, EnumerableMapping): """Provide minimal dictionary-like behavior """ def values(): """Return a IReadSequence containing the values in the collection The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. """ def items(): """Return a IReadSequence containing the items in the collection An item is a key-value tuple. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. """ class Sequence(Mapping): "Keys must be integers in a sequence starting at 0." class Sequential(Sequence): "Keys must be used in order" Util.assertTypeImplements(type(()), (Sequence, Sized, Basic.HashKey)) Util.assertTypeImplements(type([]), (Sequence, Sized, MutableMapping)) Util.assertTypeImplements(type({}), (Mapping, Sized, MutableMapping)) Zope-2.8.7-final/lib/python/Interface/Common/0000775000076400007640000000000010435610641017757 5ustar ajungajungZope-2.8.7-final/lib/python/Interface/Common/__init__.py0000664000076400007640000000120310435500320022054 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## Zope-2.8.7-final/lib/python/Interface/Common/Mapping.py0000664000076400007640000000310710435500320021715 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: Mapping.py 24763 2004-05-17 05:59:28Z philikon $ """ from Interface import Interface class IReadMapping(Interface): """Basic mapping interface """ def __getitem__(key): """Get a value for a key A KeyError is raised if there is no value for the key. """ def get(key, default=None): """Get a value for a key The default is returned if there is no value for the key. """ def has_key(key): """Tell if a key exists in the mapping """ class IEnumerableMapping(IReadMapping): """Mapping objects whose items can be enumerated """ def keys(): """Return the keys of the mapping object """ def values(): """Return the values of the mapping object """ def items(): """Return the items of the mapping object """ def __len__(): """Return the number of items """ Zope-2.8.7-final/lib/python/Interface/Common/tests/0000775000076400007640000000000010435610641021121 5ustar ajungajungZope-2.8.7-final/lib/python/Interface/Common/tests/BaseTestMapping.py0000664000076400007640000000742410435500320024520 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: BaseTestMapping.py 24763 2004-05-17 05:59:28Z philikon $ """ from operator import __getitem__ def testIReadMapping(self, inst, state, absent): for key in state: self.assertEqual(inst[key], state[key]) self.assertEqual(inst.get(key, None), state[key]) self.failUnless(inst.has_key(key)) for key in absent: self.assertEqual(inst.get(key, None), None) self.assertEqual(inst.get(key), None) self.assertEqual(inst.get(key, self), self) self.assertRaises(KeyError, __getitem__, inst, key) def test_keys(self, inst, state): """Return the keys of the mapping object """ inst_keys = list(inst.keys()); inst_keys.sort() state_keys = list(state.keys()) ; state_keys.sort() self.assertEqual(inst_keys, state_keys) def test_values(self, inst, state): """Return the values of the mapping object """ inst_values = list(inst.values()); inst_values.sort() state_values = list(state.values()) ; state_values.sort() self.assertEqual(inst_values, state_values) def test_items(self, inst, state): """Return the items of the mapping object """ inst_items = list(inst.items()); inst_items.sort() state_items = list(state.items()) ; state_items.sort() self.assertEqual(inst_items, state_items) def test___len__(self, inst, state): """Return the number of items """ self.assertEqual(len(inst), len(state)) def testIEnumerableMapping(self, inst, state): test_keys(self, inst, state) test_items(self, inst, state) test_values(self, inst, state) test___len__(self, inst, state) class BaseTestIReadMapping: def testIReadMapping(self): inst = self._IReadMapping__sample() state = self._IReadMapping__stateDict() absent = self._IReadMapping__absentKeys() testIReadMapping(self, inst, state, absent) class BaseTestIEnumerableMapping(BaseTestIReadMapping): """Mapping objects whose items can be enumerated """ def test_keys(self): """Return the keys of the mapping object """ inst = self._IEnumerableMapping__sample() state = self._IEnumerableMapping__stateDict() test_keys(self, inst, state) def test_values(self): """Return the values of the mapping object """ inst = self._IEnumerableMapping__sample() state = self._IEnumerableMapping__stateDict() test_values(self, inst, state) def test_items(self): """Return the items of the mapping object """ inst = self._IEnumerableMapping__sample() state = self._IEnumerableMapping__stateDict() test_items(self, inst, state) def test___len__(self): """Return the number of items """ inst = self._IEnumerableMapping__sample() state = self._IEnumerableMapping__stateDict() test___len__(self, inst, state) def _IReadMapping__stateDict(self): return self._IEnumerableMapping__stateDict() def _IReadMapping__sample(self): return self._IEnumerableMapping__sample() def _IReadMapping__absentKeys(self): return self._IEnumerableMapping__absentKeys() Zope-2.8.7-final/lib/python/Interface/Common/tests/__init__.py0000664000076400007640000000120310435500320023216 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## Zope-2.8.7-final/lib/python/Interface/pprint.py0000664000076400007640000000444710435500320020416 0ustar ajungajung""" Pretty-Print an Interface object as structured text (Yum) """ import string def trim_doc_string(text): """ Trims a doc string to make it format correctly with structured text. """ text=text.strip() text=text.replace('\r\n', '\n') lines=text.split('\n') nlines=[lines[0]] if len(lines) > 1: min_indent=None for line in lines[1:]: indent=len(line) - len(line.lstrip()) if indent < min_indent or min_indent is None: min_indent=indent for line in lines[1:]: nlines.append(line[min_indent:]) return '\n'.join(nlines) def justify_and_indent(text, level, munge=0, width=72): """ indent and justify text, rejustify (munge) if specified """ lines = [] if munge: line = " " * level text = (string.translate(text, string.maketrans("\r\n", " "))).strip.split() for word in text: line = ''.join([line, word]) if len(line) > width: lines.append(line) line = " " * level else: lines.append(line) return '\n'.join(lines) else: text = text.replace("\r\n", "\n").split( "\n") for line in text: lines.append( (" " * level) + line) return '\n'.join(lines) def interface_as_stx(I, munge=0): """ Output structured text format. Note, this will wack any existing 'structured' format of the text. """ outp = "%s\n\n" % I.getName() level = 1 if I.getDoc(): outp = outp + justify_and_indent(trim_doc_string(I.getDoc()), level) + "\n\n" if I.getBases(): outp = outp + (" " * level) + "This interface extends:\n\n" level = level + 1 for b in I.getBases(): item = "o %s" % b.getName() outp = outp + justify_and_indent(trim_doc_string(item), level, munge) + "\n\n" level = level - 1 level = level + 1 for name, desc in I.namesAndDescriptions(): if hasattr(desc, 'getSignatureRepr'): # ugh... item = "%s%s -- %s" % (desc.getName(), desc.getSignatureRepr(), desc.getDoc()) else: item = "%s -- %s" % (desc.getName(), desc.getDoc()) outp = outp + justify_and_indent(trim_doc_string(item), level, munge) + "\n\n" return outp Zope-2.8.7-final/lib/python/Interface/tests/0000775000076400007640000000000010435610641017671 5ustar ajungajungZope-2.8.7-final/lib/python/Interface/tests/testVisitImplements.py0000664000076400007640000000341210435500320024267 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest, sys from Interface.Implements import visitImplements from Interface import Interface from Interface.Exceptions import BadImplements class I1(Interface): pass class I2(Interface): pass class I3(Interface): pass class Test(unittest.TestCase): def testSimpleImplements(self): data=[] visitImplements(I1, None, data.append) self.assertEqual(data, [I1]) def testSimpleBadImplements(self): data=[] self.assertRaises(BadImplements, visitImplements, unittest, None, data.append) def testComplexImplements(self): data=[] visitImplements((I1, (I2, I3)), None, data.append) data = map(lambda i: i.__name__, data) self.assertEqual(data, ['I1', 'I2', 'I3']) def testComplexBadImplements(self): data=[] self.assertRaises(BadImplements, visitImplements, (I1, (I2, unittest)), None, data.append) def test_suite(): loader=unittest.TestLoader() return loader.loadTestsFromTestCase(Test) if __name__=='__main__': unittest.TextTestRunner().run(test_suite()) Zope-2.8.7-final/lib/python/Interface/tests/dummy.py0000664000076400007640000000133210435500320021365 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from Interface.tests.IFoo import IFoo __implements__ = IFoo def bar( baz ): pass Zope-2.8.7-final/lib/python/Interface/tests/bridge.txt0000664000076400007640000001024110435500320021654 0ustar ajungajung====== Bridge ====== The ``bridge`` module provides functionality to convert a Zope 3 interface into a Zope 2 one. First we'll import all we know about interfaces from the two generations: >>> from Interface import Interface as Z2_Interface >>> from Interface import Attribute as Z2_Attribute >>> from Interface.Method import Method as Z2_Method >>> from zope.interface import Interface as Z3_Interface >>> from zope.interface import Attribute as Z3_Attribute >>> from zope.schema import List, TextLine An empty interface ------------------ >>> class IEmpty(Z3_Interface): ... pass >>> from Interface.bridge import fromZ3Interface >>> IEmptyConverted = fromZ3Interface(IEmpty) >>> Z2_Interface.isEqualOrExtendedBy(IEmptyConverted) 1 >>> len(IEmptyConverted.names()) 0 Bases ----- >>> class IBase(Z3_Interface): ... pass >>> class IDerived(IBase): ... pass >>> IBase.getBases() == (Z3_Interface,) True >>> IDerived.getBases() == (IBase,) True >>> IDerived.extends(IBase) True >>> IDerived.extends(IEmpty) False >>> IBaseConverted = fromZ3Interface(IBase) >>> IDerivedConverted = fromZ3Interface(IDerived) >>> IBaseConverted.getBases() == (Z2_Interface,) True >>> IDerivedConverted.getBases() == (IBaseConverted,) True >>> IDerivedConverted.extends(IBaseConverted) 1 >>> IDerivedConverted.extends(IEmptyConverted) 0 Attributes ---------- >>> class IAttributes(Z3_Interface): ... one = Z3_Attribute('one', 'One attribute') ... another = Z3_Attribute('another', 'Another attribute') >>> converted = fromZ3Interface(IAttributes) >>> Z2_Interface.isEqualOrExtendedBy(converted) 1 >>> len(converted.names()) 2 >>> 'one' in converted.names() True >>> 'another' in converted.names() True >>> one = converted.getDescriptionFor('one') >>> isinstance(one, Z2_Attribute) True >>> one.getName() 'one' >>> one.getDoc() 'One attribute' >>> another = converted.getDescriptionFor('another') >>> isinstance(another, Z2_Attribute) True >>> another.getName() 'another' >>> another.getDoc() 'Another attribute' Fields ------ >>> class IFields(Z3_Interface): ... one = TextLine(title=u'one', description=u'One field') ... another = List(title=u'another', description=u'Another field', ... value_type = TextLine()) >>> converted = fromZ3Interface(IFields) >>> Z2_Interface.isEqualOrExtendedBy(converted) 1 >>> len(converted.names()) 2 >>> 'one' in converted.names() True >>> 'another' in converted.names() True >>> one = converted.getDescriptionFor('one') >>> isinstance(one, Z2_Attribute) True >>> one.getName() 'one' >>> one.getDoc() u'one\n\nOne field' >>> another = converted.getDescriptionFor('another') >>> isinstance(another, Z2_Attribute) True >>> another.getName() 'another' >>> another.getDoc() u'another\n\nAnother field' Methods ------- >>> class IMethods(Z3_Interface): ... def one(): ... """One method.""" ... def another(arg1, arg2): ... """Another method, taking arguments.""" >>> converted = fromZ3Interface(IMethods) >>> Z2_Interface.isEqualOrExtendedBy(converted) 1 >>> len(converted.names()) 2 >>> 'one' in converted.names() True >>> 'another' in converted.names() True >>> one = converted.getDescriptionFor('one') >>> isinstance(one, Z2_Method) True >>> one.getName() 'one' >>> one.getDoc() 'One method.' >>> one.getSignatureString() '()' >>> another = converted.getDescriptionFor('another') >>> isinstance(another, Z2_Method) True >>> another.getName() 'another' >>> another.getDoc() 'Another method, taking arguments.' >>> another.getSignatureString() '(arg1, arg2)' Invalid parameters ------------------ >>> fromZ3Interface(None) Traceback (most recent call last): ... ValueError: Not a Zope 3 interface! >>> fromZ3Interface(object()) Traceback (most recent call last): ... ValueError: Not a Zope 3 interface! >>> class IZ2_NotAllowed(Z2_Interface): ... pass >>> fromZ3Interface(IZ2_NotAllowed) Traceback (most recent call last): ... ValueError: Not a Zope 3 interface! Zope-2.8.7-final/lib/python/Interface/tests/IFoo.py0000664000076400007640000000147310435500320021074 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from Interface import Interface class IFoo( Interface ): """ Dummy interface for unit tests. """ def bar( baz ): """ Just a note. """ Zope-2.8.7-final/lib/python/Interface/tests/test_bridge.py0000664000076400007640000000200310435500320022521 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Unit tests for Z3 -> Z2 bridge utilities. $Id: test_bridge.py 37774 2005-08-07 18:07:31Z yuppie $ """ import os, sys if __name__ == '__main__': execfile(os.path.join(sys.path[0], 'framework.py')) def test_suite(): from Testing.ZopeTestCase import ZopeDocFileSuite return ZopeDocFileSuite('bridge.txt', package="Interface.tests") if __name__ == '__main__': framework() Zope-2.8.7-final/lib/python/Interface/tests/__init__.py0000664000076400007640000000122610435500320021773 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Packagize. """ Zope-2.8.7-final/lib/python/Interface/tests/testVerify.py0000664000076400007640000000742110435500320022403 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: testVerify.py 24763 2004-05-17 05:59:28Z philikon $ """ from __future__ import nested_scopes from Interface import Interface from Interface.Verify import verifyClass, verifyObject from Interface.Exceptions import DoesNotImplement, BrokenImplementation from Interface.Exceptions import BrokenMethodImplementation import unittest, sys class Test(unittest.TestCase): def testNotImplemented(self): class C: pass class I(Interface): pass self.assertRaises(DoesNotImplement, verifyClass, I, C) C.__implements__=I verifyClass(I, C) def testMissingAttr(self): class I(Interface): def f(): pass class C: __implements__=I self.assertRaises(BrokenImplementation, verifyClass, I, C) C.f=lambda self: None verifyClass(I, C) def testMissingAttr_with_Extended_Interface(self): class II(Interface): def f(): pass class I(II): pass class C: __implements__=I self.assertRaises(BrokenImplementation, verifyClass, I, C) C.f=lambda self: None verifyClass(I, C) def testWrongArgs(self): class I(Interface): def f(a): pass class C: def f(self, b): pass __implements__=I # We no longer require names to match. #self.assertRaises(BrokenMethodImplementation, verifyClass, I, C) C.f=lambda self, a: None verifyClass(I, C) C.f=lambda self, **kw: None self.assertRaises(BrokenMethodImplementation, verifyClass, I, C) C.f=lambda self, a, *args: None verifyClass(I, C) C.f=lambda self, a, *args, **kw: None verifyClass(I, C) C.f=lambda self, *args: None verifyClass(I, C) def testExtraArgs(self): class I(Interface): def f(a): pass class C: def f(self, a, b): pass __implements__=I self.assertRaises(BrokenMethodImplementation, verifyClass, I, C) C.f=lambda self, a: None verifyClass(I, C) C.f=lambda self, a, b=None: None verifyClass(I, C) def testNoVar(self): class I(Interface): def f(a, *args): pass class C: def f(self, a): pass __implements__=I self.assertRaises(BrokenMethodImplementation, verifyClass, I, C) C.f=lambda self, a, *foo: None verifyClass(I, C) def testNoKW(self): class I(Interface): def f(a, **args): pass class C: def f(self, a): pass __implements__=I self.assertRaises(BrokenMethodImplementation, verifyClass, I, C) C.f=lambda self, a, **foo: None verifyClass(I, C) def testModule(self): from Interface.tests.IFoo import IFoo from Interface.tests import dummy verifyObject(IFoo, dummy) def test_suite(): loader=unittest.TestLoader() return loader.loadTestsFromTestCase(Test) if __name__=='__main__': unittest.TextTestRunner().run(test_suite()) Zope-2.8.7-final/lib/python/Interface/tests/testInterface.py0000664000076400007640000001130710435500320023035 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import Interface from unitfixtures import * # hehehe from Interface.Exceptions import BrokenImplementation from Interface.Implements import instancesOfObjectImplements from Interface.Implements import objectImplements from Interface import Interface from Interface.Attribute import Attribute class InterfaceTests(unittest.TestCase): def setUp(self): pass def tearDown(self): pass def testClassImplements(self): assert IC.isImplementedByInstancesOf(C) assert I1.isImplementedByInstancesOf(A) assert I1.isImplementedByInstancesOf(B) assert not I1.isImplementedByInstancesOf(C) assert I1.isImplementedByInstancesOf(D) assert I1.isImplementedByInstancesOf(E) assert not I2.isImplementedByInstancesOf(A) assert I2.isImplementedByInstancesOf(B) assert not I2.isImplementedByInstancesOf(C) assert not I2.isImplementedByInstancesOf(D) assert not I2.isImplementedByInstancesOf(E) def testUtil(self): f = instancesOfObjectImplements assert IC in f(C) assert I1 in f(A) assert not I1 in f(C) assert I2 in f(B) assert not I2 in f(C) f = objectImplements assert IC in f(C()) assert I1 in f(A()) assert not I1 in f(C()) assert I2 in f(B()) assert not I2 in f(C()) def testObjectImplements(self): assert IC.isImplementedBy(C()) assert I1.isImplementedBy(A()) assert I1.isImplementedBy(B()) assert not I1.isImplementedBy(C()) assert I1.isImplementedBy(D()) assert I1.isImplementedBy(E()) assert not I2.isImplementedBy(A()) assert I2.isImplementedBy(B()) assert not I2.isImplementedBy(C()) assert not I2.isImplementedBy(D()) assert not I2.isImplementedBy(E()) def testDeferredClass(self): a = A() self.assertRaises(BrokenImplementation, a.ma) def testInterfaceExtendsInterface(self): assert BazInterface.extends(BobInterface) assert BazInterface.extends(BarInterface) assert BazInterface.extends(FunInterface) assert not BobInterface.extends(FunInterface) assert not BobInterface.extends(BarInterface) assert BarInterface.extends(FunInterface) assert not BarInterface.extends(BazInterface) def testVerifyImplementation(self): from Interface.Verify import verifyClass assert verifyClass(FooInterface, Foo) assert Interface.isImplementedBy(I1) def test_names(self): names = list(_I2.names()); names.sort() self.assertEqual(names, ['f21', 'f22', 'f23']) names = list(_I2.names(1)); names.sort() self.assertEqual(names, ['a1', 'f11', 'f12', 'f21', 'f22', 'f23']) def test_namesAndDescriptions(self): names = [nd[0] for nd in _I2.namesAndDescriptions()]; names.sort() self.assertEqual(names, ['f21', 'f22', 'f23']) names = [nd[0] for nd in _I2.namesAndDescriptions(1)]; names.sort() self.assertEqual(names, ['a1', 'f11', 'f12', 'f21', 'f22', 'f23']) for name, d in _I2.namesAndDescriptions(1): self.assertEqual(name, d.__name__) def test_getDescriptionFor(self): self.assertEqual(_I2.getDescriptionFor('f11').__name__, 'f11') self.assertEqual(_I2.getDescriptionFor('f22').__name__, 'f22') self.assertEqual(_I2.queryDescriptionFor('f33', self), self) self.assertRaises(KeyError, _I2.getDescriptionFor, 'f33') def testAttr(self): description = _I2.getDescriptionFor('a1') self.assertEqual(description.__name__, 'a1') self.assertEqual(description.__doc__, 'This is an attribute') class _I1(Interface): a1 = Attribute("This is an attribute") def f11(): pass def f12(): pass class __I1(_I1): pass class ___I1(__I1): pass class _I2(___I1): def f21(): pass def f22(): pass def f23(): pass def test_suite(): return unittest.makeSuite(InterfaceTests) def main(): unittest.TextTestRunner().run(test_suite()) if __name__=="__main__": main() Zope-2.8.7-final/lib/python/Interface/tests/testImplements.py0000664000076400007640000000364110435500320023254 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from __future__ import nested_scopes from Interface import Interface from Interface.Implements import implements from Interface.Exceptions import DoesNotImplement, BrokenImplementation from Interface.Exceptions import BrokenMethodImplementation import unittest, sys class Test(unittest.TestCase): def testSimple(self): class I(Interface): def f(): pass class C: pass self.assertRaises(BrokenImplementation, implements, C, I) C.f=lambda self: None implements(C, I) self.assertEqual(C.__implements__, I) def testAdd(self): class I(Interface): def f(): pass class I2(Interface): def g(): pass class C: __implements__=I2 self.assertRaises(BrokenImplementation, implements, C, I) self.assertRaises(BrokenImplementation, implements, C, I2) C.f=lambda self: None implements(C, I) self.assertEqual(C.__implements__, (I2, I)) self.assertRaises(BrokenImplementation, implements, C, I2) C.g=C.f implements(C, I) implements(C, I2) def test_suite(): loader=unittest.TestLoader() return loader.loadTestsFromTestCase(Test) if __name__=='__main__': unittest.TextTestRunner().run(test_suite()) Zope-2.8.7-final/lib/python/Interface/tests/unitfixtures.py0000664000076400007640000000505010435500320023004 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from Interface import Interface from Interface.Attribute import Attribute class mytest(Interface): pass class C: def m1(self, a, b): "return 1" return 1 def m2(self, a, b): "return 2" return 2 # testInstancesOfClassImplements # YAGNI IC=Interface.impliedInterface(C) class IC(Interface): def m1(a, b): "return 1" def m2(a, b): "return 2" C.__implements__=IC class I1(Interface): def ma(): "blah" class I2(I1): pass class I3(Interface): pass class I4(Interface): pass class A(I1.deferred()): __implements__=I1 class B: __implements__=I2, I3 class D(A, B): pass class E(A, B): __implements__ = A.__implements__, C.__implements__ class FooInterface(Interface): """ This is an Abstract Base Class """ foobar = Attribute("fuzzed over beyond all recognition") def aMethod(foo, bar, bingo): """ This is aMethod """ def anotherMethod(foo=6, bar="where you get sloshed", bingo=(1,3,)): """ This is anotherMethod """ def wammy(zip, *argues): """ yadda yadda """ def useless(**keywords): """ useless code is fun! """ class Foo: """ A concrete class """ __implements__ = FooInterface, foobar = "yeah" def aMethod(self, foo, bar, bingo): """ This is aMethod """ return "barf!" def anotherMethod(self, foo=6, bar="where you get sloshed", bingo=(1,3,)): """ This is anotherMethod """ return "barf!" def wammy(self, zip, *argues): """ yadda yadda """ return "barf!" def useless(self, **keywords): """ useless code is fun! """ return "barf!" foo_instance = Foo() class Blah: pass new = Interface.__class__ FunInterface = new('FunInterface') BarInterface = new('BarInterface', [FunInterface]) BobInterface = new('BobInterface') BazInterface = new('BazInterface', [BobInterface, BarInterface]) Zope-2.8.7-final/lib/python/Interface/tests/framework.py0000664000076400007640000000700510435500320022232 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZopeTestCase framework COPY THIS FILE TO YOUR 'tests' DIRECTORY. This version of framework.py will use the SOFTWARE_HOME environment variable to locate Zope and the Testing package. If the tests are run in an INSTANCE_HOME installation of Zope, Products.__path__ and sys.path with be adjusted to include the instance's Products and lib/python directories respectively. If you explicitly set INSTANCE_HOME prior to running the tests, auto-detection is disabled and the specified path will be used instead. If the 'tests' directory contains a custom_zodb.py file, INSTANCE_HOME will be adjusted to use it. If you set the ZEO_INSTANCE_HOME environment variable a ZEO setup is assumed, and you can attach to a running ZEO server (via the instance's custom_zodb.py). The following code should be at the top of every test module: import os, sys if __name__ == '__main__': execfile(os.path.join(sys.path[0], 'framework.py')) ...and the following at the bottom: if __name__ == '__main__': framework() $Id: framework.py 37774 2005-08-07 18:07:31Z yuppie $ """ __version__ = '0.2.4' # Save start state # __SOFTWARE_HOME = os.environ.get('SOFTWARE_HOME', '') __INSTANCE_HOME = os.environ.get('INSTANCE_HOME', '') if __SOFTWARE_HOME.endswith(os.sep): __SOFTWARE_HOME = os.path.dirname(__SOFTWARE_HOME) if __INSTANCE_HOME.endswith(os.sep): __INSTANCE_HOME = os.path.dirname(__INSTANCE_HOME) # Find and import the Testing package # if not sys.modules.has_key('Testing'): p0 = sys.path[0] if p0 and __name__ == '__main__': os.chdir(p0) p0 = '' s = __SOFTWARE_HOME p = d = s and s or os.getcwd() while d: if os.path.isdir(os.path.join(p, 'Testing')): zope_home = os.path.dirname(os.path.dirname(p)) sys.path[:1] = [p0, p, zope_home] break p, d = s and ('','') or os.path.split(p) else: print 'Unable to locate Testing package.', print 'You might need to set SOFTWARE_HOME.' sys.exit(1) import Testing, unittest execfile(os.path.join(os.path.dirname(Testing.__file__), 'common.py')) # Include ZopeTestCase support # if 1: # Create a new scope p = os.path.join(os.path.dirname(Testing.__file__), 'ZopeTestCase') if not os.path.isdir(p): print 'Unable to locate ZopeTestCase package.', print 'You might need to install ZopeTestCase.' sys.exit(1) ztc_common = 'ztc_common.py' ztc_common_global = os.path.join(p, ztc_common) f = 0 if os.path.exists(ztc_common_global): execfile(ztc_common_global) f = 1 if os.path.exists(ztc_common): execfile(ztc_common) f = 1 if not f: print 'Unable to locate %s.' % ztc_common sys.exit(1) # Debug # print 'SOFTWARE_HOME: %s' % os.environ.get('SOFTWARE_HOME', 'Not set') print 'INSTANCE_HOME: %s' % os.environ.get('INSTANCE_HOME', 'Not set') sys.stdout.flush() Zope-2.8.7-final/lib/python/Interface/tests/testElement.py0000664000076400007640000000255210435500320022530 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest from Interface._Element import Element class ElementTests(unittest.TestCase): def test_taggedValues(self): """Test that we can update tagged values of more than one element """ e1 = Element("foo") e2 = Element("bar") e1.setTaggedValue("x", 1) e2.setTaggedValue("x", 2) self.assertEqual(e1.getTaggedValue("x"), 1) self.assertEqual(e2.getTaggedValue("x"), 2) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(ElementTests)) return suite def test_suite(): return unittest.makeSuite(ElementTests) def main(): unittest.TextTestRunner().run(test_suite()) if __name__=="__main__": main() Zope-2.8.7-final/lib/python/Interface/tests/testDocument.py0000664000076400007640000000305110435500320022710 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: testDocument.py 24763 2004-05-17 05:59:28Z philikon $ """ from unittest import TestCase, TestSuite, main, makeSuite from Interface import Interface from Interface.Attribute import Attribute class Test(TestCase): def testBlech(self): from Interface.Document import asStructuredText self.assertEqual(asStructuredText(I2), '''\ I2 I2 doc This interface extends: o _I1 Attributes: a1 -- no documentation a2 -- a2 doc Methods: f21() -- f21 doc f22() -- no documentation f23() -- f23 doc ''') def test_suite(): return TestSuite(( makeSuite(Test), )) class _I1(Interface): def f11(): pass def f12(): pass class I2(_I1): "I2 doc" a1 = Attribute('a1') a2 = Attribute('a2', 'a2 doc') def f21(): "f21 doc" def f22(): pass def f23(): "f23 doc" if __name__=='__main__': main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/Interface/IInterface.py0000664000076400007640000001616510435500320021113 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ Revision information: $Id: IInterface.py 24763 2004-05-17 05:59:28Z philikon $ """ from IElement import IElement class IInterface(IElement): """Interface objects Interface objects describe the behavior of an object by containing useful information about the object. This information includes: o Prose documentation about the object. In Python terms, this is called the "doc string" of the interface. In this element, you describe how the object works in prose language and any other useful information about the object. o Descriptions of attributes. Attribute descriptions include the name of the attribute and prose documentation describing the attributes usage. o Descriptions of methods. Method descriptions can include: o Prose "doc string" documentation about the method and its usage. o A description of the methods arguments; how many arguments are expected, optional arguments and their default values, the position or arguments in the signature, whether the method accepts arbitrary arguments and whether the method accepts arbitrary keyword arguments. o Optional tagged data. Interface objects (and their attributes and methods) can have optional, application specific tagged data associated with them. Examples uses for this are examples, security assertions, pre/post conditions, and other possible information you may want to associate with an Interface or its attributes. Not all of this information is mandatory. For example, you may only want the methods of your interface to have prose documentation and not describe the arguments of the method in exact detail. Interface objects are flexible and let you give or take any of these components. Interfaces are created with the Python class statement using either Interface.Interface or another interface, as in:: from Interface import Interface class IMyInterface(Interface): '''Interface documentation ''' def meth(arg1, arg2): '''Documentation for meth ''' # Note that there is no self argument class IMySubInterface(IMyInterface): '''Interface documentation ''' def meth2(): '''Documentation for meth2 ''' You use interfaces in two ways: o You assert that your object implement the interfaces. There are several ways that you can assert that an object implements an interface:: 1. Include an '__implements__' attribute in the object's class definition. The value of the '__implements__' attribute must be an implementation specification. An implementation specification is either an interface or a tuple of implementation specifications. 2. Incluse an '__implements__' attribute in the object. Because Python classes don't have their own attributes, to assert that a class implements interfaces, you must provide a '__class_implements__' attribute in the class definition. **Important**: A class usually doesn't implement the interfaces that it's instances implement. The class and it's instances are separate objects with their own interfaces. 3. Call 'Interface.Implements.implements' to assert that instances of a class implement an interface. For example:: from Interface.Implements import implements implements(some_class, some_interface) This is approach is useful when it is not an option to modify the class source. Note that this doesn't affect what the class itself implements, but only what it's instances implement. 4. For types that can't be modified, you can assert that instances of the type implement an interface using 'Interface.Implements.assertTypeImplements'. For example:: from Interface.Implements import assertTypeImplements assertTypeImplements(some_type, some_interface) o You query interface meta-data. See the IInterface methods and attributes for details. """ def getBases(): """Return a sequence of the base interfaces """ def extends(other, strict=1): """Test whether the interface extends another interface A true value is returned in the interface extends the other interface, and false otherwise. Normally, an interface doesn't extend itself. If a false value is passed as the second argument, or via the 'strict' keyword argument, then a true value will be returned if the interface and the other interface are the same. """ def isImplementedBy(object): """Test whether the interface is implemented by the object. Return true of the object asserts that it implements the interface, including asseting that it implements an extended interface. """ def isImplementedByInstancesOf(class_): """Test whether the interface is implemented by instances of the class Return true of the class asserts that it's instances implement the interface, including asseting that they implement an extended interface. """ def names(all=0): """Get the interface attribute names. Return a sequence of the names of the attributes, including methods, included in the interface definition. Normally, only directly defined attributes are included. If a true positional or keyword argument is given, then attributes defined by nase classes will be included. """ def namesAndDescriptions(all=0): """Get the interface attribute names and descriptions. Return a sequence of the names and descriptions of the attributes, including methods, as name-value pairs, included in the interface definition. Normally, only directly defined attributes are included. If a true positional or keyword argument is given, then attributes defined by nase classes will be included. """ def getDescriptionFor(name): """Get the description for a name If the named attribute does not exist, a KeyError is raised. """ def queryDescriptionFor(name, default=None): """Get the description for a name Return the default if no description exists. """ Zope-2.8.7-final/lib/python/Interface/_object.py0000664000076400007640000000275010435500320020502 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Provide a halfway believable rendition of Python 2.2's object $Id: _object.py 24763 2004-05-17 05:59:28Z philikon $ """ class _x: def m(self): pass ClassTypes = (type(_x), ) MethodTypes = (type(_x.m), ) isInstance = isinstance try: object except NameError: # Python 2.1 try: from ExtensionClass import Base as object except ImportError: class object: pass else: # Python 2.2 ClassTypes += (type, ) object = object try: import ExtensionClass except ImportError: # ExtensionClass is not present pass else: # ExtensionClass is present def isInstance(ob, klass): if type(type(ob)) is type(klass): return isinstance(ob, klass) return 0 class _x(ExtensionClass.Base): def m(self): pass ClassTypes += (type(_x), ) MethodTypes += (type(_x.m), ) Zope-2.8.7-final/lib/python/nt_svcutils/0000775000076400007640000000000010435610641017204 5ustar ajungajungZope-2.8.7-final/lib/python/nt_svcutils/__init__.py0000664000076400007640000000004010435500330021300 0ustar ajungajung""" Placeholder module file """ Zope-2.8.7-final/lib/python/nt_svcutils/service.py0000664000076400007640000004021010435500330021204 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Windows Services installer/controller for Zope/ZEO/ZRS instance homes""" import sys, os, time, threading, signal import pywintypes import winerror, win32con import win32api, win32event, win32file, win32pipe, win32process, win32security import win32service, win32serviceutil, servicemanager # the max seconds we're allowed to spend backing off BACKOFF_MAX = 300 # if the process runs successfully for more than BACKOFF_CLEAR_TIME # seconds, we reset the backoff stats to their initial values BACKOFF_CLEAR_TIME = 30 # the initial backoff interval (the amount of time we wait to restart # a dead process) BACKOFF_INITIAL_INTERVAL = 5 # We execute a new thread that captures the tail of the output from our child # process. If the child fails, it is written to the event log. # This process is unconditional, and the output is never written to disk # (except obviously via the event log entry) # Size of the blocks we read from the child process's output. CHILDCAPTURE_BLOCK_SIZE = 80 # The number of BLOCKSIZE blocks we keep as process output. This gives # is 4k, which should be enough to see any tracebacks etc, but not so # large as to prematurely fill the event log. CHILDCAPTURE_MAX_BLOCKS = 50 class Service(win32serviceutil.ServiceFramework): """Base class for a Windows Server to manage an external process. Subclasses can be used to managed an instance home-based Zope or ZEO process. The win32 Python service module registers a specific file and class for a service. To manage an instance, a subclass should be created in the instance home. """ # The PythonService model requires that an actual on-disk class declaration # represent a single service. Thus, the definitions below for the instance # must be overridden in a subclass in a file within the instance home for # each instance. # The values below are just examples. _svc_name_ = r'Zope-Instance' _svc_display_name_ = r'Zope instance at C:\Zope-Instance' process_runner = r'C:\Program Files\Zope-2.7.0-a1\bin\python.exe' process_args = r'{path_to}\run.py -C {path_to}\zope.conf' evtlog_name = 'Zope' def __init__(self, args): win32serviceutil.ServiceFramework.__init__(self, args) # Just say "Zope", instead of "Zope_-xxxxx" try: servicemanager.SetEventSourceName(self.evtlog_name) except AttributeError: # old pywin32 - that's ok. pass # Create an event which we will use to wait on. # The "service stop" request will set this event. # We create it inheritable so we can pass it to the child process, so # it too can act on the stop event. sa = win32security.SECURITY_ATTRIBUTES() sa.bInheritHandle = True self.hWaitStop = win32event.CreateEvent(sa, 0, 0, None) self.redirect_thread = None def SvcStop(self): # Before we do anything, tell the SCM we are starting the stop process. self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) self.onStop() # Set the stop event - the main loop takes care of termination. win32event.SetEvent(self.hWaitStop) # SvcStop only gets triggered when the user explictly stops (or restarts) # the service. To shut the service down cleanly when Windows is shutting # down, we also need to hook SvcShutdown. SvcShutdown = SvcStop def onStop(self): # A hook for subclasses to override pass def createProcess(self, cmd): self.start_time = time.time() return self.createProcessCaptureIO(cmd) def logmsg(self, event): # log a service event using servicemanager.LogMsg try: servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE, event, (self._svc_name_, " (%s)" % self._svc_display_name_)) except win32api.error, details: # Failed to write a log entry - most likely problem is # that the event log is full. We don't want this to kill us try: print "FAILED to write INFO event", event, ":", details except IOError: pass def _dolog(self, func, msg): try: fullmsg = "%s (%s): %s" % \ (self._svc_name_, self._svc_display_name_, msg) func(fullmsg) except win32api.error, details: # Failed to write a log entry - most likely problem is # that the event log is full. We don't want this to kill us try: print "FAILED to write event log entry:", details print msg except IOError: # And if running as a service, its likely our sys.stdout # is invalid pass def info(self, s): self._dolog(servicemanager.LogInfoMsg, s) def warning(self, s): self._dolog(servicemanager.LogWarningMsg, s) def error(self, s): self._dolog(servicemanager.LogErrorMsg, s) def SvcDoRun(self): # indicate to Zope that the process is daemon managed (restartable) os.environ['ZMANAGED'] = '1' # XXX the restart behavior is different here than it is for # zdaemon.zdrun. we should probably do the same thing in both # places. # daemon behavior: we want to to restart the process if it # dies, but if it dies too many times, we need to give up. # we use a simple backoff algorithm to determine whether # we should try to restart a dead process: for each # time the process dies unexpectedly, we wait some number of # seconds to restart it, as determined by the backoff interval, # which doubles each time the process dies. if we exceed # BACKOFF_MAX seconds in cumulative backoff time, we give up. # at any time if we successfully run the process for more thab # BACKOFF_CLEAR_TIME seconds, the backoff stats are reset. # the initial number of seconds between process start attempts self.backoff_interval = BACKOFF_INITIAL_INTERVAL # the cumulative backoff seconds counter self.backoff_cumulative = 0 self.logmsg(servicemanager.PYS_SERVICE_STARTED) while 1: # We pass *this* file and the handle as the first 2 params, then # the 'normal' startup args. # See the bottom of this script for how that is handled. cmd = '"%s" %s' % (self.process_runner, self.process_args) info = self.createProcess(cmd) # info is (hProcess, hThread, pid, tid) self.hZope = info[0] # process handle # XXX why the test before the log message? if self.backoff_interval > BACKOFF_INITIAL_INTERVAL: self.info("created process") if not (self.run() and self.checkRestart()): break self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) # Stop the child process by opening the special named event. # We give it 90 seconds to shutdown normally. If that doesn't # stop things, we give it 30 seconds to do a "fast" shutdown. # After that, we just knock it on the head. winver = sys.getwindowsversion() for sig, timeout in ((signal.SIGINT, 30), (signal.SIGTERM, 10)): event_name = "Zope-%d-%d" % (info[2], sig) # sys.getwindowsversion() -> major, minor, build, platform_id, ver_string # for platform_id, 2==VER_PLATFORM_WIN32_NT if winver[0] >= 5 and winver[3] == 2: event_name = "Global\\" + event_name try: he = win32event.OpenEvent(win32event.EVENT_MODIFY_STATE, 0, event_name) except win32event.error, details: if details[0] == winerror.ERROR_FILE_NOT_FOUND: # process already dead! break # no other expected error - report it. self.warning("Failed to open child shutdown event %s" % (event_name,)) continue win32event.SetEvent(he) # It should be shutting down now - wait for termination, reporting # progress as we go. for i in range(timeout): self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) rc = win32event.WaitForSingleObject(self.hZope, 3000) if rc == win32event.WAIT_OBJECT_0: break # Process terminated - no need to try harder. if rc == win32event.WAIT_OBJECT_0: break self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) # If necessary, kill it if win32process.GetExitCodeProcess(self.hZope)==win32con.STILL_ACTIVE: win32api.TerminateProcess(self.hZope, 3) self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) # Wait for the redirect thread - it should have died as the remote # process terminated. # As we are shutting down, we do the join with a little more care, # reporting progress as we wait (even though we never will ) if self.redirect_thread is not None: for i in range(5): self.redirect_thread.join(1) self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING) if not self.redirect_thread.isAlive(): break else: self.warning("Redirect thread did not stop!") self.logmsg(servicemanager.PYS_SERVICE_STOPPED) def run(self): """Monitor the daemon process. Returns True if the service should continue running and False if the service process should exit. On True return, the process exited unexpectedly and the caller should restart it. """ keep_running = True rc = win32event.WaitForMultipleObjects([self.hWaitStop, self.hZope], 0, # bWaitAll win32event.INFINITE) if rc == win32event.WAIT_OBJECT_0: # user sent a stop service request keep_running = False elif rc == win32event.WAIT_OBJECT_0 + 1: # user did not send a service stop request, but # the process died; this may be an error condition status = win32process.GetExitCodeProcess(self.hZope) # exit status 0 means the user caused a clean shutdown, # presumably via the web interface. Any other status # is an error that gets written to the event log. if status != 0: # This should never block - the child process terminating # has closed the redirection pipe, so our thread dies. self.redirect_thread.join(5) if self.redirect_thread.isAlive(): self.warning("Redirect thread did not stop!") self.warning("process terminated with exit code %d.\n%s" \ % (status, "".join(self.captured_blocks))) keep_running = status != 0 else: # No other valid return codes. assert 0, rc return keep_running def checkRestart(self): # this was an abormal shutdown. if self.backoff_cumulative > BACKOFF_MAX: self.error("restarting too frequently; quit") return False self.warning("sleep %s to avoid rapid restarts" % self.backoff_interval) if time.time() - self.start_time > BACKOFF_CLEAR_TIME: self.backoff_interval = BACKOFF_INITIAL_INTERVAL self.backoff_cumulative = 0 # sleep for our backoff, but still respond to stop requests. if win32event.WAIT_OBJECT_0 == \ win32event.WaitForSingleObject(self.hWaitStop, self.backoff_interval * 1000): return False self.backoff_cumulative += self.backoff_interval self.backoff_interval *= 2 return True def createProcessCaptureIO(self, cmd): hInputRead, hInputWriteTemp = self.newPipe() hOutReadTemp, hOutWrite = self.newPipe() pid = win32api.GetCurrentProcess() # This one is duplicated as inheritable. hErrWrite = win32api.DuplicateHandle(pid, hOutWrite, pid, 0, 1, win32con.DUPLICATE_SAME_ACCESS) # These are non-inheritable duplicates. hOutRead = self.dup(hOutReadTemp) hInputWrite = self.dup(hInputWriteTemp) # dup() closed hOutReadTemp, hInputWriteTemp si = win32process.STARTUPINFO() si.hStdInput = hInputRead si.hStdOutput = hOutWrite si.hStdError = hErrWrite si.dwFlags = win32process.STARTF_USESTDHANDLES | \ win32process.STARTF_USESHOWWINDOW si.wShowWindow = win32con.SW_HIDE # pass True to allow handles to be inherited. Inheritance is # problematic in general, but should work in the controlled # circumstances of a service process. create_flags = win32process.CREATE_NEW_CONSOLE info = win32process.CreateProcess(None, cmd, None, None, True, create_flags, None, None, si) # (NOTE: these really aren't necessary for Python - they are closed # as soon as they are collected) hOutWrite.Close() hErrWrite.Close() hInputRead.Close() # We don't use stdin hInputWrite.Close() # start a thread collecting output t = threading.Thread(target=self.redirectCaptureThread, args = (hOutRead,)) t.start() self.redirect_thread = t return info def redirectCaptureThread(self, handle): # Only one of these running at a time, and handling both stdout and # stderr on a single handle. The read data is never referenced until # the thread dies - so no need for locks around self.captured_blocks. self.captured_blocks = [] #self.info("Redirect thread starting") while 1: try: ec, data = win32file.ReadFile(handle, CHILDCAPTURE_BLOCK_SIZE) except pywintypes.error, err: # ERROR_BROKEN_PIPE means the child process closed the # handle - ie, it terminated. if err[0] != winerror.ERROR_BROKEN_PIPE: self.warning("Error reading output from process: %s" % err) break self.captured_blocks.append(data) del self.captured_blocks[CHILDCAPTURE_MAX_BLOCKS:] handle.Close() #self.info("Redirect capture thread terminating") def newPipe(self): sa = win32security.SECURITY_ATTRIBUTES() sa.bInheritHandle = True return win32pipe.CreatePipe(sa, 0) def dup(self, pipe): # create a duplicate handle that is not inherited, so that # it can be closed in the parent. close the original pipe in # the process. pid = win32api.GetCurrentProcess() dup = win32api.DuplicateHandle(pid, pipe, pid, 0, 0, win32con.DUPLICATE_SAME_ACCESS) pipe.Close() return dup # Real __main__ bootstrap code is in the instance's service module. if __name__ == '__main__': print "This is a framework module - you don't run it directly." print "See your $SOFTWARE_HOME\bin directory for the service script." sys.exit(1) Zope-2.8.7-final/lib/python/Globals.py0000775000076400007640000000311710435500331016563 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Global definitions""" __version__='$Revision: 1.56 $'[11:-2] # Global constants: __replaceable__ flags: NOT_REPLACEABLE = 0 REPLACEABLE = 1 UNIQUE = 2 import Acquisition, ComputedAttribute, App.PersistentExtra, os import TreeDisplay from App.Common import package_home, attrget, Dictionary from App.config import getConfiguration as _getConfiguration from Persistence import Persistent, PersistentMapping from App.special_dtml import HTML, HTMLFile, DTMLFile from App.class_init import default__class_init__, ApplicationDefaultPermissions # Nicer alias for class initializer. InitializeClass = default__class_init__ from App.Dialogs import MessageDialog from App.ImageFile import ImageFile VersionNameName='Zope-Version' _cfg = _getConfiguration() data_dir = _cfg.clienthome # backward compatibility INSTANCE_HOME = _cfg.instancehome SOFTWARE_HOME = _cfg.softwarehome ZOPE_HOME = _cfg.zopehome opened=[] DevelopmentMode=_cfg.debug_mode del _cfg, _getConfiguration Zope-2.8.7-final/lib/python/StructuredText/0000775000076400007640000000000010435610641017640 5ustar ajungajungZope-2.8.7-final/lib/python/StructuredText/ST.py0000664000076400007640000002154510435500325020544 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import re, STDOM from types import ListType ##################################################################### # Updated functions # ##################################################################### def indention(str,front = re.compile("^\s+").match): """ Find the number of leading spaces. If none, return 0. """ result = front(str) if result is not None: start, end = result.span() return end-start else: return 0 # no leading spaces def insert(struct, top, level): """ find what will be the parant paragraph of a sentence and return that paragraph's sub-paragraphs. The new paragraph will be appended to those sub-paragraphs """ #print "struct", struct, top-1 if not top-1 in range(len(struct)): if struct: return struct[len(struct)-1].getSubparagraphs() return struct run = struct[top-1] i = 0 while i+1 < level: run = run.getSubparagraphs()[len(run.getSubparagraphs())-1] i = i + 1 #print "parent for level ", level, " was => ", run.getColorizableTexts() return run.getSubparagraphs() def display(struct): """ runs through the structure and prints out the paragraphs. If the insertion works correctly, display's results should mimic the orignal paragraphs. """ if struct.getColorizableTexts(): print join(struct.getColorizableTexts()),"\n" if struct.getSubparagraphs(): for x in struct.getSubparagraphs(): display(x) def display2(struct): """ runs through the structure and prints out the paragraphs. If the insertion works correctly, display's results should mimic the orignal paragraphs. """ if struct.getNodeValue(): print struct.getNodeValue(),"\n" if struct.getSubparagraphs(): for x in struct.getSubparagraphs(): display(x) def findlevel(levels,indent): """ remove all level information of levels with a greater level of indentation. Then return which level should insert this paragraph """ keys = levels.keys() for key in keys: if levels[key] > indent: del(levels[key]) keys = levels.keys() if not(keys): return 0 else: for key in keys: if levels[key] == indent: return key highest = 0 for key in keys: if key > highest: highest = key return highest-1 para_delim = r'(\n\s*\n|\r\n\s*\r\n)' # UNIX or DOS line endings, respectively ##################################################################### # Golly, the capitalization of this function always makes me think it's a class def StructuredText(paragraphs, delimiter=re.compile(para_delim)): """ StructuredText accepts paragraphs, which is a list of lines to be parsed. StructuredText creates a structure which mimics the structure of the paragraphs. Structure => [paragraph,[sub-paragraphs]] """ currentlevel = 0 currentindent = 0 levels = {0:0} level = 0 # which header are we under struct = [] # the structure to be returned run = struct paragraphs = paragraphs.expandtabs() paragraphs = '%s%s%s' % ('\n\n', paragraphs, '\n\n') paragraphs = delimiter.split(paragraphs) paragraphs = [ x for x in paragraphs if x.strip() ] if not paragraphs: return StructuredTextDocument() ind = [] # structure based on indention levels for paragraph in paragraphs: ind.append([indention(paragraph), paragraph]) currentindent = indention(paragraphs[0]) levels[0] = currentindent ############################################################# # updated # ############################################################# for indent,paragraph in ind : if indent == 0: level = level + 1 currentlevel = 0 currentindent = 0 levels = {0:0} struct.append(StructuredTextParagraph(paragraph, indent=indent, level=currentlevel)) elif indent > currentindent: currentlevel = currentlevel + 1 currentindent = indent levels[currentlevel] = indent run = insert(struct,level,currentlevel) run.append(StructuredTextParagraph(paragraph, indent=indent, level=currentlevel)) elif indent < currentindent: result = findlevel(levels,indent) if result > 0: currentlevel = result currentindent = indent if not level: struct.append(StructuredTextParagraph(paragraph, indent=indent, level=currentlevel)) else: run = insert(struct,level,currentlevel) run.append(StructuredTextParagraph(paragraph, indent=indent, level=currentlevel)) else: if insert(struct,level,currentlevel): run = insert(struct,level,currentlevel) else: run = struct currentindent = indent run.append(StructuredTextParagraph(paragraph, indent=indent, level=currentlevel)) return StructuredTextDocument(struct) Basic = StructuredText class StructuredTextParagraph(STDOM.Element): indent=0 def __init__(self, src, subs=None, **kw): if subs is None: subs=[] self._src=src self._subs=list(subs) self._attributes=kw.keys() for k, v in kw.items(): setattr(self, k, v) def getChildren(self): src=self._src if not isinstance(src, ListType): src=[src] return src+self._subs def getAttribute(self, name): return getattr(self, name, None) def getAttributeNode(self, name): if hasattr(self, name): return STDOM.Attr(name, getattr(self, name)) else: return None def getAttributes(self): d={} for a in self._attributes: d[a]=getattr(self, a, '') return STDOM.NamedNodeMap(d) def getSubparagraphs(self): return self._subs def setSubparagraphs(self, subs): self._subs=subs def getColorizableTexts(self): return (self._src,) def setColorizableTexts(self, src): self._src=src[0] def __repr__(self): r=[]; a=r.append a((' '*(self.indent or 0))+ ('%s(' % self.__class__.__name__) +str(self._src)+', [' ) for p in self._subs: a(`p`) a((' '*(self.indent or 0))+'])') return '\n'.join(r) """ create aliases for all above functions in the pythony way. """ def _get_Children(self): return self.getChildren() def _get_Attribute(self, name): return self.getAttribute(name) def _get_AttributeNode(self, name): return self.getAttributeNode(name) def _get_Attributes(self): return self.getAttributes() def _get_Subparagraphs(self): return self.getSubparagraphs() def _set_Subparagraphs(self, subs): return self.setSubparagraphs(subs) def _get_ColorizableTexts(self): return self.getColorizableTexts() def _set_ColorizableTexts(self, src): return self.setColorizableTexts(src) class StructuredTextDocument(StructuredTextParagraph): """ A StructuredTextDocument holds StructuredTextParagraphs as its subparagraphs. """ _attributes=() def __init__(self, subs=None, **kw): StructuredTextParagraph.__init__(self, '', subs, **kw) def getChildren(self): return self._subs def getColorizableTexts(self): return () def setColorizableTexts(self, src): pass def __repr__(self): r=[]; a=r.append a('%s([' % self.__class__.__name__) for p in self._subs: a(`p`+',') a('])') return '\n'.join(r) """ create aliases for all above functions in the pythony way. """ def _get_Children(self): return self.getChildren() def _get_ColorizableTexts(self): return self.getColorizableTexts() def _set_ColorizableTexts(self, src): return self.setColorizableTexts(src) Zope-2.8.7-final/lib/python/StructuredText/regressions/0000775000076400007640000000000010435610641022203 5ustar ajungajungZope-2.8.7-final/lib/python/StructuredText/regressions/table.ref0000664000076400007640000000345010435500325023767 0ustar ajungajung

Function

Documentation

__str__

This method converts the the object to a string.

  • Blah
  • Blaf

    Name

    Favorite Color

    Jim

    Red

    John

    Blue

This should give a row with colspan 3

Col 1

Col 2

Col 3

Col 1

Col 2

Col 1

Col 2

Zope-2.8.7-final/lib/python/StructuredText/regressions/examples1.ref0000664000076400007640000000033310435500325024574 0ustar ajungajung Test

Test

For instance: 

    <table border="0">
      <tr><td>blabla</td></tr>
    </table>

Zope-2.8.7-final/lib/python/StructuredText/regressions/ExtensionClass.ref0000664000076400007640000010037310435500325025644 0ustar ajungajung Extension Classes, Python Extension Types Become Classes

Extension Classes, Python Extension Types Become Classes

Jim Fulton, Digital Creations, Inc. jim@digicool.com

Copyright (C) 1996-1998, Digital Creations.

Abstract

A lightweight mechanism has been developed for making Python extension types more class-like. Classes can be developed in an extension language, such as C or C++, and these classes can be treated like other python classes:

  • They can be sub-classed in python,
  • They provide access to method documentation strings, and
  • They can be used to directly create new instances.

An example class shows how extension classes are implemented and how they differ from extension types.

Extension classes provide additional extensions to class and instance semantics, including:

  • A protocol for accessing subobjects "in the context of" their containers. This is used to implement custom method types and environmental acquisition.
  • A protocol for overriding method call semantics. This is used to implement "synchonized" classes and could be used to implement argument type checking.
  • A protocol for class initialization that supports execution of a special __class_init__ method after a class has been initialized.

Extension classes illustrate how the Python class mechanism can be extended and may provide a basis for improved or specialized class models.

Releases

To find out what's changed in this release, see the release notes.

Problem

Currently, Python provides two ways of defining new kinds of objects:

  • Python classes
  • Extension types

Each approach has it's strengths. Extension types provide much greater control to the programmer and, generally, better performance. Because extension types are written in C, the programmer has greater access to external resources. (Note that Python's use of the term type has little to do with the notion of type as a formal specification.)

Classes provide a higher level of abstraction and are generally much easier to develop. Classes provide full inheritance support, while support for inheritance when developing extension types is very limited. Classes provide run-time meta-data, such as method documentation strings, that are useful for documentation and discovery. Classes act as factories for creating instances, while separate functions must be provided to create instances of types.

It would be useful to combine the features of the two approaches. It would be useful to be able to have better support for inheritance for types, or to be able to subclass from types in Python. It would be useful to be able to have class-like meta-data support for types and the ability to construct instances directly from types.

Our software is developed in Python. When necessary, we convert debugged Python routines and classes to C for improved performance. In most cases, a small number of methods in a class is responsible for most of the computation. It should be possible to convert only these methods to C, while leaving the other method in Python. A natural way to approach this is to create a base class in C that contains only the performance-critical aspects of a class' implementation and mix this base class into a Python class.

We have need, in a number of projects, for semantics that are slightly different than the usual class and instance semantics, yet we don't want to do most of our development in C. For example, we have developed a persistence mechanism [1] that redefines __getattr__ and __setattr__ to take storage-related actions when object state is accessed or modified. We want to be able to take certain actions on every attribute reference, but for python class instances, __getattr__ is only called when attribute lookup fails by normal means.

As another example, we would like to have greater control over how methods are bound. Currently, when accessing a class instance attribute, the attribute value is bound together with the instance in a method object if and only if the attribute value is a python function. For some applications, we might also want to be able to bind extension functions, or other types of callable objects, such as HTML document templates [2]. Furthermore, we might want to have greater control over how objects are bound. For example, we might want to bind instances and callable objects with special method objects that assure that no more than one thread accesses the object or method at one time.

We can provide these special semantics in extension types, but we wish to provide them for classes developed in Python.

Background

At the first Python Workshop, Don Beaudry presented work [3] done at V.I. Corp to integrate Python with C++ frameworks. This system provided a number of important features, including:

  • Definition of extension types that provide class-like meta-data and that can be called to create instances.
  • Ability to subclass in python from C types.
  • Ability to define classes in python who's data are stored as C structures rather than in dictionaries to better interface to C and C++ libraries, and for better performance.
  • Less dynamic data structures. In particular, the data structure for a class is declared during class definition.
  • Support for enumeration types.

This work was not released, initially.

Shortly after the workshop, changes were made to Python to support the sub-classing features described in [3]. These changes were not documented until the fourth Python Workshop [4].

At the third Python workshop, I presented some work I had done on generating module documentation for extension types. Based on the discussion at this workshop, I developed a meta-type proposal [5]. This meta-type proposal was for an object that simply stored meta-information for a type, for the purpose of generating module documentation.

In the summer of 1996, Don Beaudry released the system described in [3] under the name MESS [6]. MESS addresses a number of needs but has a few drawbacks:

  • Only single inheritance is supported.
  • The mechanisms for defining MESS extension types is very different from and more complicated than the standard Python type creation mechanism.
  • Defining MESS types requires the use of an extensive C applications programming interface. This presents problems for configuring dynamically-loaded extension modules unless the MESS library is linked into the Python interpreter.
  • Because the system tries to do a number of different things, it is fairly large, about 15,000 lines.
  • There is very little documentation, especially for the C programming interface.
  • The system is a work in progress, with a number of outstanding bugs.

As MESS matures, we expect most of these problems to be addressed.

Extension Classes

To meet short term needs for a C-based persistence mechanism [1], an extension class module was developed using the mechanism described in [4] and building on ideas from MESS [6]. The extension class module recasts extension types as "extension classes" by seeking to eliminate, or at least reduce semantic differences between types and classes. The module was designed to meet the following goal:

  • Provide class-like behavior for extension types, including interfaces for meta information and for constructing instances.
  • Support sub-classing in Python from extension classes, with support for multiple inheritance.
  • Provide a small hardened implementation that can be used for current products.
  • Provide a mechanism that requires minimal modification to existing extension types.
  • Provide a basis for research on alternative semantics for classes and inheritance.

Note: I use non-standard terminology here. By standard python terminology, only standard python classes can be called classes. ExtensionClass "classes" are technically just "types" that happen to swim, walk and quack like python classes.

Base extension classes and extension subclasses

Base extension classes are implemented in C. Extension subclasses are implemented in Python and inherit, directly or indirectly from one or more base extension classes. An extension subclass may inherit from base extension classes, extension subclasses, and ordinary python classes. The usual inheritance order rules apply. Currently, extension subclasses must conform to the following two rules:

  • The first super class listed in the class statement defining an extension subclass must be either a base extension class or an extension subclass. This restriction will be removed in Python-1.5.
  • At most one base extension direct or indirect super class may define C data members. If an extension subclass inherits from multiple base extension classes, then all but one must be mix-in classes that provide extension methods but no data.

Meta Information

Like standard python classes, extension classes have the following attributes containing meta-data:

__doc__
a documentation string for the class,
__name__
the class name,
__bases__
a sequence of base classes,
__dict__
a class dictionary, and
__module__
the name of the module in which the class was defined.

The class dictionary provides access to unbound methods and their documentation strings, including extension methods and special methods, such as methods that implement sequence and numeric protocols. Unbound methods can be called with instance first arguments.

Subclass instance data

Extension subclass instances have instance dictionaries, just like Python class instances do. When fetching attribute values, extension class instances will first try to obtain data from the base extension class data structure, then from the instance dictionary, then from the class dictionary, and finally from base classes. When setting attributes, extension classes first attempt to use extension base class attribute setting operations, and if these fail, then data are placed in the instance dictionary.

Implementing base extension classes

A base extension class is implemented in much the same way that an extension type is implemented, except:

  • The include file, ExtensionClass.h, must be included.
  • The type structure is declared to be of type PyExtensionClass, rather than of type PyTypeObject.
  • The type structure has an additional member that must be defined after the documentation string. This extra member is a method chain (PyMethodChain) containing a linked list of method definition (PyMethodDef) lists. Method chains can be used to implement method inheritance in C. Most extensions don't use method chains, but simply define method lists, which are null-terminated arrays of method definitions. A macro, METHOD_CHAIN is defined in ExtensionClass.h that converts a method list to a method chain. (See the example below.)
  • Module functions that create new instances must be replaced by __init__ methods that initialize, but does not create storage for instances.
  • The extension class must be initialized and exported to the module with: 
              PyExtensionClass_Export(d,"name",type);

where name is the module name and type is the extension class type object.

Attribute lookup

Attribute lookup is performed by calling the base extension class getattr operation for the base extension class that includes C data, or for the first base extension class, if none of the base extension classes include C data. ExtensionClass.h defines a macro Py_FindAttrString that can be used to find an object's attributes that are stored in the object's instance dictionary or in the object's class or base classes: 

         v = Py_FindAttrString(self,name);

where name is a C string containing the attribute name.

In addition, a macro is provided that replaces Py_FindMethod calls with logic to perform the same sort of lookup that is provided by Py_FindAttrString.

If an attribute name is contained in a Python string object, rather than a C string object, then the macro Py_FindAttr should be used to look up an attribute value.

Linking

The extension class mechanism was designed to be useful with dynamically linked extension modules. Modules that implement extension classes do not have to be linked against an extension class library. The macro PyExtensionClass_Export imports the ExtensionClass module and uses objects imported from this module to initialize an extension class with necessary behavior.

Example: MultiMapping objects

An example, is provided that illustrates the changes needed to convert an existing type to an ExtensionClass.

Implementing base extension class constructors

Some care should be taken when implementing or overriding base class constructors. When a Python class overrides a base class constructor and fails to call the base class constructor, a program using the class may fail, but it will not crash the interpreter. On the other hand, an extension subclass that overrides a constructor in an extension base class must call the extension base class constructor or risk crashing the interpreter. This is because the base class constructor may set C pointers that, if not set properly, will cause the interpreter to crash when accessed. This is the case with the MultiMapping extension base class shown in the example above.

If no base class constructor is provided, extension class instance memory will be initialized to 0. It is a good idea to design extension base classes so that instance methods check for uninitialized memory and perform initialialization if necessary. This was not done above to simplify the example.

Overriding methods inherited from Python base classes

A problem occurs when trying to overide methods inherited from Python base classes. Consider the following example: 

      from ExtensionClass import Base

      class Spam:

        def __init__(self, name):
          self.name=name

      class ECSpam(Base, Spam):

        def __init__(self, name, favorite_color):
          Spam.__init__(self,name)
          self.favorite_color=favorite_color

This implementation will fail when an ECSpam object is instantiated. The problem is that ECSpam.__init__ calls Spam.__init__, and Spam.__init__ can only be called with a Python instance (an object of type "instance") as the first argument. The first argument passed to Spam.__init__ will be an ECSpam instance (an object of type ECSPam).

To overcome this problem, extension classes provide a class method inheritedAttribute that can be used to obtain an inherited attribute that is suitable for calling with an extension class instance. Using the inheritedAttribute method, the above example can be rewritten as: 

      from ExtensionClass import Base

      class Spam:

        def __init__(self, name):
          self.name=name

      class ECSpam(Base, Spam):

        def __init__(self, name, favorite_color):
          ECSpam.inheritedAttribute('__init__')(self,name)
          self.favorite_color=favorite_color

This isn't as pretty but does provide the desired result.

New class and instance semantics

Context Wrapping

It is sometimes useful to be able to wrap up an object together with a containing object. I call this "context wrapping" because an object is accessed in the context of the object it is accessed through.

We have found many applications for this, including:

  • User-defined method objects,
  • Acquisition, and
  • Computed attributes

User-defined method objects

Python classes wrap Python function attributes into methods. When a class has a function attribute that is accessed as an instance attribute, a method object is created and returned that contains references to the original function and instance. When the method is called, the original function is called with the instance as the first argument followed by any arguments passed to the method.

Extension classes provide a similar mechanism for attributes that are Python functions or inherited extension functions. In addition, if an extension class attribute is an instance of an extension class that defines an __of__ method, then when the attribute is accessed through an instance, it's __of__ method will be called to create a bound method.

Consider the following example: 

          import ExtensionClass

          class CustomMethod(ExtensionClass.Base):

            def __call__(self,ob): 
              print 'a %s was called' % ob.__class__.__name__

            class wrapper:

              def __init__(self,m,o): self.meth, self.ob=m,o

              def __call__(self): self.meth(self.ob)

            def __of__(self,o): return self.wrapper(self,o)

          class bar(ExtensionClass.Base):
            hi=CustomMethod()

          x=bar()
          hi=x.hi()

Note that ExtensionClass.Base is a base extension class that provides very basic ExtensionClass behavior.

When run, this program outputs: a bar was called.

Computed Attributes

It is not uncommon to wish to expose information via the attribute interface without affecting implementation data structures. One can use a custom __getattr__ method to implement computed attributes, however, this can be a bit cumbersome and can interfere with other uses of __getattr__, such as for persistence.

The __of__ protocol provides a convenient way to implement computed attributes. First, we define a ComputedAttribute class. a ComputedAttribute is constructed with a function to be used to compute an attribute, and calls the function when it's __of__ method is called:

import ExtensionClass

class ComputedAttribute(ExtensionClass.Base):

def __init__(self, func): self.func=func

def __of__(self, parent): return self.func(parent)

Then we can use this class to create computed attributes. In the example below, we create a computed attribute, 'radius':

from math import sqrt

class Point(ExtensionClass.Base):

def __init__(self, x, y): self.x, self.y = x, y

radius=ComputedAttribute(lambda self: sqrt(self.x2+self.y2))

which we can use just like an ordinary attribute:

p=Point(2,2) print p.radius

Overriding method calls

Normally, when a method is called, the function wrapped by the method is called directly by the method. In some cases, it is useful for user-defined logic to participate in the actual function call. Extension classes introduce a new protocol that provides extension classes greater control over how their methods are called. If an extension class defines a special method, __call_method__, then this method will be called to call the functions (or other callable object) wrapped by the method. The method. __call_method__ should provide the same interface as provided by the Python builtin apply function.

For example, consider the expression: x.meth(arg1, arg2). The expression is evaluated by first computing a method object that wraps x and the attribute of x stored under the name meth. Assuming that x has a __call_method__ method defined, then the __call_method__ method of x will be called with two arguments, the attribute of x stored under the name meth, and a tuple containing x, arg1, and arg2.

To see how this feature may be used, see the Python module, Syn.py, which is included in the ExtensionClass distribution. This module provides a mix-in class that provides Java-like "synchonized" classes that limit access to their methods to one thread at a time.

An interesting application of this mechanism would be to implement interface checking on method calls.

Method attributes

Methods of ExtensionClass instances can have user-defined attributes, which are stored in their associated instances.

For example: 

        class C(ExtensionClass.Base):

          def get_secret(self):
            "Get a secret"
            ....

        c=C()

        c.f.__roles__=['Trusted People']

        print c.f.__roles__ # outputs ['Trusted People']
        print c.f__roles__  # outputs ['Trusted People']

        print C.f.__roles__ # fails, unbound method

A bound method attribute is set by setting an attribute in it's instance with a name consisting of the concatination of the method's __name__ attribute and the attribute name. Attributes cannot be set on unbound methods.

Class initialization

Normal Python class initialization is similar to but subtley different from instance initialization. An instance __init__ function is called on an instance immediately after it is created. An instance __init__ function can use instance information, like it's class and can pass the instance to other functions. On the other hand, the code in class statements is executed immediately before the class is created. This means that the code in a class statement cannot use class attributes, like __bases__, or pass the class to functions.

Extension classes provide a mechanism for specifying code to be run after a class has been created. If a class or one of it's base classes defines a __class_init__ method, then this method will be called just after a class has been created. The one argument passed to the method will be the class, not an instance of the class.

Useful macros defined in ExtensionClass.h

A number of useful macros are defined in ExtensionClass.h. These are documented in ExtensionClass.h.

Pickleability

Classes created with ExtensionClass, including extension base classes are automatically pickleable. The usual gymnastics necessary to pickle non-standard types are not necessray for types that have been modified to be extension base classes.

Status

The current release of the extension class module is 1.1. The core implementation has less than four thousand lines of code, including comments. This release requires Python 1.4 or higher.

To find out what's changed in this release, see the release notes.

Installation instructions, are provided.

Issues

There are a number of issues that came up in the course of this work and that deserve mention.

  • In Python 1.4, the class extension mechanism described in [4] required that the first superclass in a list of super-classes must be of the extended class type. This may not be convenient if mix-in behavior is desired. If a list of base classes starts with a standard python class, but includes an extension class, then an error was raised. It would be more useful if, when a list of base classes contains one or more objects that are not python classes, the first such object was used to control the extended class definition. To get around this, the ExtensionClass module exports a base extension class, Base, that can be used as the first base class in a list of base classes to assure that an extension subclass is created.

    Python 1.5 allows the class extension even if the first non-class object in the list of base classes is not the first object in the list. This issue appears to go away in Python 1.5, however, the restriction that the first non-class object in a list of base classes must be the first in the list may reappear in later versions of Python.

  • Currently, only one base extension class can define any data in C. The data layout of subclasses-instances is the same as for the base class that defines data in C, except that the data structure is extended to hold an instance dictionary. The data structure begins with a standard python header, and extension methods expect the C instance data to occur immediately after the object header. If two or more base classes defined C data, the methods for the different base classes would expect their data to be in the same location. A solution might be to allocate base class instances and store pointers to these instances in the subclass data structure. The method binding mechanism would have to be a more complicated to make sure that methods were bound to the correct base data structure. Alternatively, the signature of C methods could be expanded to allow pointers to expected class data to be passed in addition to object pointers.
  • There is currently no support for sub-classing in C, beyond that provided by method chains.
  • Rules for mixed-type arithmetic are different for python class instances than they are for extension type instances. Python classes can define right and left versions of numeric binary operators, or they can define a coercion operator for converting binary operator operands to a common type. For extension types, only the latter, coercion-based, approach is supported. The coercion-based approach does not work well for many data types for which coercion rules depend on the operator. Because extension classes are based on extension types, they are currently limited to the coercion-based approach. It should be possible to extend the extension class implementation to allow both types of mixed-type arithmetic control.
  • I considered making extension classes immutable, meaning that class attributes could not be set after class creation. I also considered making extension subclasses cache inherited attributes. Both of these are related and attractive for some applications, however, I decided that it would be better to retain standard class instance semantics and provide these features as options at a later time.
  • The extension class module defines new method types to bind C and python methods to extension class instances. It would be useful for these method objects to provide access to function call information, such as the number and names of arguments and the number of defaults, by parsing extension function documentation strings.

Applications

Aside from test and demonstration applications, the extension class mechanism has been used to provide an extension-based implementation of the persistence mechanism described in [1]. We have developed this further to provide features such as automatic deactivation of objects not used after some period of time and to provide more efficient persistent-object cache management.

Acquisition has been heavily used in our recent products. Synchonized classes have also been used in recent products.

Summary

The extension-class mechanism described here provides a way to add class services to extension types. It allows:

  • Sub-classing extension classes in Python,
  • Construction of extension class instances by calling extension classes,
  • Extension classes to provide meta-data, such as unbound methods and their documentation string.

In addition, the extension class module provides a relatively concise example of the use of mechanisms that were added to Python to support MESS [6], and that were described at the fourth Python Workshop [4]. It is hoped that this will spur research in improved and specialized models for class implementation in Python.

References

[1] Fulton, J., Providing Persistence for World-Wide-Web Applications, Proceedings of the 5th Python Workshop.

[2] Page, R. and Cropper, S., Document Template, Proceedings of the 5th Python Workshop.

[3] Beaudry, D., Deriving Built-In Classes in Python, Proceedings of the First International Python Workshop.

[4] Van Rossum, G., Don Beaudry Hack - MESS, presented in the Developer's Future Enhancements session of the 4th Python Workshop.

[5] Fulton, J., Meta-Type Object, This is a small proposal, the text of which is contained in a sample implementation source file,

[6] Beaudry, D., and Ascher, D., The Meta-Extension Set.

Zope-2.8.7-final/lib/python/StructuredText/regressions/examples1.stx0000664000076400007640000000014210435500325024634 0ustar ajungajungTest For instance::
blabla
Zope-2.8.7-final/lib/python/StructuredText/regressions/table.stx0000664000076400007640000000253210435500325024031 0ustar ajungajung|-------------------------------------------------| | Function | Documentation | |=================================================| | '__str__' | This method converts the | | | the object to a string. | | | | | | - Blah | | | | | | - Blaf | | | | | | |--------------------------| | | | | Name | Favorite | | | | | | Color | | | | |==========================| | | | | Jim | Red | | | | |--------------------------| | | | | John | Blue | | | | |--------------------------| | |-------------------------------------------------| |---------------------------------------| | This should give a row with colspan 3 | |---------------------------------------| | Col 1 | Col 2 | Col 3 | |---------------------------------------| | Col 1 | Col 2 | |---------------------------------------| | Col 1 | Col 2 | |---------------------------------------| Zope-2.8.7-final/lib/python/StructuredText/regressions/ExtensionClass.stx0000664000076400007640000007261710435500325025717 0ustar ajungajungExtension Classes, Python Extension Types Become Classes Jim Fulton, Digital Creations, Inc. jim@digicool.com "Copyright (C) 1996-1998, Digital Creations":COPYRIGHT.html. Abstract A lightweight mechanism has been developed for making Python extension types more class-like. Classes can be developed in an extension language, such as C or C++, and these classes can be treated like other python classes: - They can be sub-classed in python, - They provide access to method documentation strings, and - They can be used to directly create new instances. An example class shows how extension classes are implemented and how they differ from extension types. Extension classes provide additional extensions to class and instance semantics, including: - A protocol for accessing subobjects "in the context of" their containers. This is used to implement custom method types and "environmental acquisition":Acquisition.html. - A protocol for overriding method call semantics. This is used to implement "synchonized" classes and could be used to implement argument type checking. - A protocol for class initialization that supports execution of a special '__class_init__' method after a class has been initialized. Extension classes illustrate how the Python class mechanism can be extended and may provide a basis for improved or specialized class models. Releases To find out what's changed in this release, see the "release notes":release.html. Problem Currently, Python provides two ways of defining new kinds of objects: - Python classes - Extension types Each approach has it's strengths. Extension types provide much greater control to the programmer and, generally, better performance. Because extension types are written in C, the programmer has greater access to external resources. (Note that Python's use of the term type has little to do with the notion of type as a formal specification.) Classes provide a higher level of abstraction and are generally much easier to develop. Classes provide full inheritance support, while support for inheritance when developing extension types is very limited. Classes provide run-time meta-data, such as method documentation strings, that are useful for documentation and discovery. Classes act as factories for creating instances, while separate functions must be provided to create instances of types. It would be useful to combine the features of the two approaches. It would be useful to be able to have better support for inheritance for types, or to be able to subclass from types in Python. It would be useful to be able to have class-like meta-data support for types and the ability to construct instances directly from types. Our software is developed in Python. When necessary, we convert debugged Python routines and classes to C for improved performance. In most cases, a small number of methods in a class is responsible for most of the computation. It should be possible to convert only these methods to C, while leaving the other method in Python. A natural way to approach this is to create a base class in C that contains only the performance-critical aspects of a class' implementation and mix this base class into a Python class. We have need, in a number of projects, for semantics that are slightly different than the usual class and instance semantics, yet we don't want to do most of our development in C. For example, we have developed a persistence mechanism [1] that redefines '__getattr__' and '__setattr__' to take storage-related actions when object state is accessed or modified. We want to be able to take certain actions on *every* attribute reference, but for python class instances, '__getattr__' is only called when attribute lookup fails by normal means. As another example, we would like to have greater control over how methods are bound. Currently, when accessing a class instance attribute, the attribute value is bound together with the instance in a method object *if and only if* the attribute value is a python function. For some applications, we might also want to be able to bind extension functions, or other types of callable objects, such as HTML document templates [2]. Furthermore, we might want to have greater control over how objects are bound. For example, we might want to bind instances and callable objects with special method objects that assure that no more than one thread accesses the object or method at one time. We can provide these special semantics in extension types, but we wish to provide them for classes developed in Python. Background At the first Python Workshop, Don Beaudry presented work [3] done at V.I. Corp to integrate Python with C++ frameworks. This system provided a number of important features, including: - Definition of extension types that provide class-like meta-data and that can be called to create instances. - Ability to subclass in python from C types. - Ability to define classes in python who's data are stored as C structures rather than in dictionaries to better interface to C and C++ libraries, and for better performance. - Less dynamic data structures. In particular, the data structure for a class is declared during class definition. - Support for enumeration types. This work was not released, initially. Shortly after the workshop, changes were made to Python to support the sub-classing features described in [3]. These changes were not documented until the fourth Python Workshop [4]. At the third Python workshop, I presented some work I had done on generating module documentation for extension types. Based on the discussion at this workshop, I developed a meta-type proposal [5]. This meta-type proposal was for an object that simply stored meta-information for a type, for the purpose of generating module documentation. In the summer of 1996, Don Beaudry released the system described in [3] under the name MESS [6]. MESS addresses a number of needs but has a few drawbacks: - Only single inheritance is supported. - The mechanisms for defining MESS extension types is very different from and more complicated than the standard Python type creation mechanism. - Defining MESS types requires the use of an extensive C applications programming interface. This presents problems for configuring dynamically-loaded extension modules unless the MESS library is linked into the Python interpreter. - Because the system tries to do a number of different things, it is fairly large, about 15,000 lines. - There is very little documentation, especially for the C programming interface. - The system is a work in progress, with a number of outstanding bugs. As MESS matures, we expect most of these problems to be addressed. Extension Classes To meet short term needs for a C-based persistence mechanism [1], an extension class module was developed using the mechanism described in [4] and building on ideas from MESS [6]. The extension class module recasts extension types as "extension classes" by seeking to eliminate, or at least reduce semantic differences between types and classes. The module was designed to meet the following goal: - Provide class-like behavior for extension types, including interfaces for meta information and for constructing instances. - Support sub-classing in Python from extension classes, with support for multiple inheritance. - Provide a small hardened implementation that can be used for current products. - Provide a mechanism that requires minimal modification to existing extension types. - Provide a basis for research on alternative semantics for classes and inheritance. **Note:** I use *non-standard* terminology here. By standard *python* terminology, only standard python classes can be called classes. ExtensionClass "classes" are technically just "types" that happen to swim, walk and quack like python classes. Base extension classes and extension subclasses Base extension classes are implemented in C. Extension subclasses are implemented in Python and inherit, directly or indirectly from one or more base extension classes. An extension subclass may inherit from base extension classes, extension subclasses, and ordinary python classes. The usual inheritance order rules apply. Currently, extension subclasses must conform to the following two rules: - The first super class listed in the class statement defining an extension subclass must be either a base extension class or an extension subclass. This restriction will be removed in Python-1.5. - At most one base extension direct or indirect super class may define C data members. If an extension subclass inherits from multiple base extension classes, then all but one must be mix-in classes that provide extension methods but no data. Meta Information Like standard python classes, extension classes have the following attributes containing meta-data: '__doc__' -- a documentation string for the class, '__name__' -- the class name, '__bases__' -- a sequence of base classes, '__dict__' -- a class dictionary, and '__module__' -- the name of the module in which the class was defined. The class dictionary provides access to unbound methods and their documentation strings, including extension methods and special methods, such as methods that implement sequence and numeric protocols. Unbound methods can be called with instance first arguments. Subclass instance data Extension subclass instances have instance dictionaries, just like Python class instances do. When fetching attribute values, extension class instances will first try to obtain data from the base extension class data structure, then from the instance dictionary, then from the class dictionary, and finally from base classes. When setting attributes, extension classes first attempt to use extension base class attribute setting operations, and if these fail, then data are placed in the instance dictionary. Implementing base extension classes A base extension class is implemented in much the same way that an extension type is implemented, except: - The include file, 'ExtensionClass.h', must be included. - The type structure is declared to be of type 'PyExtensionClass', rather than of type 'PyTypeObject'. - The type structure has an additional member that must be defined after the documentation string. This extra member is a method chain ('PyMethodChain') containing a linked list of method definition ('PyMethodDef') lists. Method chains can be used to implement method inheritance in C. Most extensions don't use method chains, but simply define method lists, which are null-terminated arrays of method definitions. A macro, 'METHOD_CHAIN' is defined in 'ExtensionClass.h' that converts a method list to a method chain. (See the example below.) - Module functions that create new instances must be replaced by '__init__' methods that initialize, but does not create storage for instances. - The extension class must be initialized and exported to the module with:: PyExtensionClass_Export(d,"name",type); where 'name' is the module name and 'type' is the extension class type object. Attribute lookup Attribute lookup is performed by calling the base extension class 'getattr' operation for the base extension class that includes C data, or for the first base extension class, if none of the base extension classes include C data. 'ExtensionClass.h' defines a macro 'Py_FindAttrString' that can be used to find an object's attributes that are stored in the object's instance dictionary or in the object's class or base classes:: v = Py_FindAttrString(self,name); where 'name' is a C string containing the attribute name. In addition, a macro is provided that replaces 'Py_FindMethod' calls with logic to perform the same sort of lookup that is provided by 'Py_FindAttrString'. If an attribute name is contained in a Python string object, rather than a C string object, then the macro 'Py_FindAttr' should be used to look up an attribute value. Linking The extension class mechanism was designed to be useful with dynamically linked extension modules. Modules that implement extension classes do not have to be linked against an extension class library. The macro 'PyExtensionClass_Export' imports the 'ExtensionClass' module and uses objects imported from this module to initialize an extension class with necessary behavior. Example: MultiMapping objects An "example":MultiMapping.html, is provided that illustrates the changes needed to convert an existing type to an ExtensionClass. Implementing base extension class constructors Some care should be taken when implementing or overriding base class constructors. When a Python class overrides a base class constructor and fails to call the base class constructor, a program using the class may fail, but it will not crash the interpreter. On the other hand, an extension subclass that overrides a constructor in an extension base class must call the extension base class constructor or risk crashing the interpreter. This is because the base class constructor may set C pointers that, if not set properly, will cause the interpreter to crash when accessed. This is the case with the 'MultiMapping' extension base class shown in the example above. If no base class constructor is provided, extension class instance memory will be initialized to 0. It is a good idea to design extension base classes so that instance methods check for uninitialized memory and perform initialialization if necessary. This was not done above to simplify the example. Overriding methods inherited from Python base classes A problem occurs when trying to overide methods inherited from Python base classes. Consider the following example:: from ExtensionClass import Base class Spam: def __init__(self, name): self.name=name class ECSpam(Base, Spam): def __init__(self, name, favorite_color): Spam.__init__(self,name) self.favorite_color=favorite_color This implementation will fail when an 'ECSpam' object is instantiated. The problem is that 'ECSpam.__init__' calls 'Spam.__init__', and 'Spam.__init__' can only be called with a Python instance (an object of type '"instance"') as the first argument. The first argument passed to 'Spam.__init__' will be an 'ECSpam' instance (an object of type 'ECSPam'). To overcome this problem, extension classes provide a class method 'inheritedAttribute' that can be used to obtain an inherited attribute that is suitable for calling with an extension class instance. Using the 'inheritedAttribute' method, the above example can be rewritten as:: from ExtensionClass import Base class Spam: def __init__(self, name): self.name=name class ECSpam(Base, Spam): def __init__(self, name, favorite_color): ECSpam.inheritedAttribute('__init__')(self,name) self.favorite_color=favorite_color This isn't as pretty but does provide the desired result. New class and instance semantics Context Wrapping It is sometimes useful to be able to wrap up an object together with a containing object. I call this "context wrapping" because an object is accessed in the context of the object it is accessed through. We have found many applications for this, including: - User-defined method objects, - "Acquisition":Acquisition.html, and - Computed attributes User-defined method objects Python classes wrap Python function attributes into methods. When a class has a function attribute that is accessed as an instance attribute, a method object is created and returned that contains references to the original function and instance. When the method is called, the original function is called with the instance as the first argument followed by any arguments passed to the method. Extension classes provide a similar mechanism for attributes that are Python functions or inherited extension functions. In addition, if an extension class attribute is an instance of an extension class that defines an '__of__' method, then when the attribute is accessed through an instance, it's '__of__' method will be called to create a bound method. Consider the following example:: import ExtensionClass class CustomMethod(ExtensionClass.Base): def __call__(self,ob): print 'a %s was called' % ob.__class__.__name__ class wrapper: def __init__(self,m,o): self.meth, self.ob=m,o def __call__(self): self.meth(self.ob) def __of__(self,o): return self.wrapper(self,o) class bar(ExtensionClass.Base): hi=CustomMethod() x=bar() hi=x.hi() Note that 'ExtensionClass.Base' is a base extension class that provides very basic ExtensionClass behavior. When run, this program outputs: 'a bar was called'. Computed Attributes It is not uncommon to wish to expose information via the attribute interface without affecting implementation data structures. One can use a custom '__getattr__' method to implement computed attributes, however, this can be a bit cumbersome and can interfere with other uses of '__getattr__', such as for persistence. The '__of__' protocol provides a convenient way to implement computed attributes. First, we define a ComputedAttribute class. a ComputedAttribute is constructed with a function to be used to compute an attribute, and calls the function when it's '__of__' method is called: import ExtensionClass class ComputedAttribute(ExtensionClass.Base): def __init__(self, func): self.func=func def __of__(self, parent): return self.func(parent) Then we can use this class to create computed attributes. In the example below, we create a computed attribute, 'radius': from math import sqrt class Point(ExtensionClass.Base): def __init__(self, x, y): self.x, self.y = x, y radius=ComputedAttribute(lambda self: sqrt(self.x**2+self.y**2)) which we can use just like an ordinary attribute: p=Point(2,2) print p.radius Overriding method calls Normally, when a method is called, the function wrapped by the method is called directly by the method. In some cases, it is useful for user-defined logic to participate in the actual function call. Extension classes introduce a new protocol that provides extension classes greater control over how their methods are called. If an extension class defines a special method, '__call_method__', then this method will be called to call the functions (or other callable object) wrapped by the method. The method. '__call_method__' should provide the same interface as provided by the Python builtin 'apply' function. For example, consider the expression: 'x.meth(arg1, arg2)'. The expression is evaluated by first computing a method object that wraps 'x' and the attribute of 'x' stored under the name 'meth'. Assuming that 'x' has a '__call_method__' method defined, then the '__call_method__' method of 'x' will be called with two arguments, the attribute of 'x' stored under the name 'meth', and a tuple containing 'x', 'arg1', and 'arg2'. To see how this feature may be used, see the Python module, 'Syn.py', which is included in the ExtensionClass distribution. This module provides a mix-in class that provides Java-like "synchonized" classes that limit access to their methods to one thread at a time. An interesting application of this mechanism would be to implement interface checking on method calls. Method attributes Methods of ExtensionClass instances can have user-defined attributes, which are stored in their associated instances. For example:: class C(ExtensionClass.Base): def get_secret(self): "Get a secret" .... c=C() c.f.__roles__=['Trusted People'] print c.f.__roles__ # outputs ['Trusted People'] print c.f__roles__ # outputs ['Trusted People'] print C.f.__roles__ # fails, unbound method A bound method attribute is set by setting an attribute in it's instance with a name consisting of the concatination of the method's '__name__' attribute and the attribute name. Attributes cannot be set on unbound methods. Class initialization Normal Python class initialization is similar to but subtley different from instance initialization. An instance '__init__' function is called on an instance immediately *after* it is created. An instance '__init__' function can use instance information, like it's class and can pass the instance to other functions. On the other hand, the code in class statements is executed immediately *before* the class is created. This means that the code in a class statement cannot use class attributes, like '__bases__', or pass the class to functions. Extension classes provide a mechanism for specifying code to be run *after* a class has been created. If a class or one of it's base classes defines a '__class_init__' method, then this method will be called just after a class has been created. The one argument passed to the method will be the class, *not* an instance of the class. Useful macros defined in ExtensionClass.h A number of useful macros are defined in ExtensionClass.h. These are documented in 'ExtensionClass.h'. Pickleability Classes created with ExtensionClass, including extension base classes are automatically pickleable. The usual gymnastics necessary to pickle 'non-standard' types are not necessray for types that have been modified to be extension base classes. Status The current release of the extension class module is "1.1", http://www.digicool.com/releases/ExtensionClass/ExtensionClass-1.1.tar.gz. The core implementation has less than four thousand lines of code, including comments. This release requires Python 1.4 or higher. To find out what's changed in this release, see the "release notes":release.html. "Installation instructions":Installation.html, are provided. Issues There are a number of issues that came up in the course of this work and that deserve mention. - In Python 1.4, the class extension mechanism described in [4] required that the first superclass in a list of super-classes must be of the extended class type. This may not be convenient if mix-in behavior is desired. If a list of base classes starts with a standard python class, but includes an extension class, then an error was raised. It would be more useful if, when a list of base classes contains one or more objects that are not python classes, the first such object was used to control the extended class definition. To get around this, the 'ExtensionClass' module exports a base extension class, 'Base', that can be used as the first base class in a list of base classes to assure that an extension subclass is created. Python 1.5 allows the class extension even if the first non-class object in the list of base classes is not the first object in the list. This issue appears to go away in Python 1.5, however, the restriction that the first non-class object in a list of base classes must be the first in the list may reappear in later versions of Python. - Currently, only one base extension class can define any data in C. The data layout of subclasses-instances is the same as for the base class that defines data in C, except that the data structure is extended to hold an instance dictionary. The data structure begins with a standard python header, and extension methods expect the C instance data to occur immediately after the object header. If two or more base classes defined C data, the methods for the different base classes would expect their data to be in the same location. A solution might be to allocate base class instances and store pointers to these instances in the subclass data structure. The method binding mechanism would have to be a more complicated to make sure that methods were bound to the correct base data structure. Alternatively, the signature of C methods could be expanded to allow pointers to expected class data to be passed in addition to object pointers. - There is currently no support for sub-classing in C, beyond that provided by method chains. - Rules for mixed-type arithmetic are different for python class instances than they are for extension type instances. Python classes can define right and left versions of numeric binary operators, or they can define a coercion operator for converting binary operator operands to a common type. For extension types, only the latter, coercion-based, approach is supported. The coercion-based approach does not work well for many data types for which coercion rules depend on the operator. Because extension classes are based on extension types, they are currently limited to the coercion-based approach. It should be possible to extend the extension class implementation to allow both types of mixed-type arithmetic control. - I considered making extension classes immutable, meaning that class attributes could not be set after class creation. I also considered making extension subclasses cache inherited attributes. Both of these are related and attractive for some applications, however, I decided that it would be better to retain standard class instance semantics and provide these features as options at a later time. - The extension class module defines new method types to bind C and python methods to extension class instances. It would be useful for these method objects to provide access to function call information, such as the number and names of arguments and the number of defaults, by parsing extension function documentation strings. Applications Aside from test and demonstration applications, the extension class mechanism has been used to provide an extension-based implementation of the persistence mechanism described in [1]. We have developed this further to provide features such as automatic deactivation of objects not used after some period of time and to provide more efficient persistent-object cache management. Acquisition has been heavily used in our recent products. Synchonized classes have also been used in recent products. Summary The extension-class mechanism described here provides a way to add class services to extension types. It allows: - Sub-classing extension classes in Python, - Construction of extension class instances by calling extension classes, - Extension classes to provide meta-data, such as unbound methods and their documentation string. In addition, the extension class module provides a relatively concise example of the use of mechanisms that were added to Python to support MESS [6], and that were described at the fourth Python Workshop [4]. It is hoped that this will spur research in improved and specialized models for class implementation in Python. References .. [1] Fulton, J., "Providing Persistence for World-Wide-Web Applications", http://www.digicool.com/papers/Persistence.html, Proceedings of the 5th Python Workshop. .. [2] Page, R. and Cropper, S., "Document Template", http://www.digicool.com/papers/DocumentTemplate.html, Proceedings of the 5th Python Workshop. .. [3] Beaudry, D., "Deriving Built-In Classes in Python", http://www.python.org/workshops/1994-11/BuiltInClasses/BuiltInClasses_1.html, Proceedings of the First International Python Workshop. .. [4] Van Rossum, G., "Don Beaudry Hack - MESS", http://www.python.org/workshops/1996-06/notes/thursday.html, presented in the Developer's Future Enhancements session of the 4th Python Workshop. .. [5] Fulton, J., "Meta-Type Object", http://www.digicool.com/jim/MetaType.c, This is a small proposal, the text of which is contained in a sample implementation source file, .. [6] Beaudry, D., and Ascher, D., "The Meta-Extension Set", http://starship.skyport.net/~da/mess/. Zope-2.8.7-final/lib/python/StructuredText/regressions/InnerLinks.ref0000664000076400007640000000052410435500325024753 0ustar ajungajung This is the InnerLinkTest

This is the InnerLinkTest

see also [1] and [2]

[1] "Zope Book" by Amos Lattmeier and Michel Pelletier

[2] "Python Book" by Guido van Rossum

Zope-2.8.7-final/lib/python/StructuredText/regressions/Links.ref0000664000076400007640000000205310435500325023756 0ustar ajungajung This is LinkTest

This is LinkTest

And now a paragraph with Link 1 and Link 2 and one more link - yeah.

Zope-2.8.7-final/lib/python/StructuredText/regressions/index.stx0000664000076400007640000000340110435500325024045 0ustar ajungajung Extension Class "Copyright (C) 1996-1998, Digital Creations":COPYRIGHT.html. A lightweight mechanism has been developed for making Python extension types more class-like. Classes can be developed in an extension language, such as C or C++, and these classes can be treated like other python classes: - They can be sub-classed in python, - They provide access to method documentation strings, and - They can be used to directly create new instances. Extension classes provide additional extensions to class and instance semantics, including: - A protocol for accessing subobjects "in the context of" their containers. This is used to implement custom method types and "environmental acquisition":Acquisition.html. - A protocol for overriding method call semantics. This is used to implement "synchonized" classes and could be used to implement argument type checking. - A protocol for class initialization that supports execution of a special '__class_init__' method after a class has been initialized. Extension classes illustrate how the Python class mechanism can be extended and may provide a basis for improved or specialized class models. Releases The current release is "1.2":ExtensionClass-1.2.tar.gz, To find out what's changed in this release, see the "release notes":release.html. Documentation is available "on-line":ExtensionClass.html. Windows Binaries A win32 binary release, "ec12.zip":ec12.zip, is available. This release includes all of the ExtensionClass modules built as Windows extension modules (.pyd) files. These were built for Python 1.5.1 using Microsoft Visual C++ 5.0 in "Release" mode. Zope-2.8.7-final/lib/python/StructuredText/regressions/examples.stx0000664000076400007640000000213710435500325024561 0ustar ajungajungSmall Trials for Structured Text Formatting This paragraph should be preceded by a level 1 header. It should not, itself, be made into a header, just a regular paragraph. Here are a few presentation styles, in a list [1]: - A word: *emphasized*. - A word: _underlined_. - A word **strong**. - An inline example: '1+2'. - Another example with a different format: ``x='spam''' or ``y='spam''' or ``''.' We can use expressions in the DTML var tag as in ``'' - A mult-line example:: blah *foo bar* .. [1] (The referring text should be a paragraph, not a header, and should contain a reference to this footnote, footnote "[1]".) Some hrefs, in a definition list: _Regular_ -- "http://www.zope.org/":http://www.zope.org _W/trailing punctuation_ -- "http://www.zope.org/":http://www.zope.org. _W protocol implicit_ -- "locallink"::locallink _W protocol implicit_, alternate -- "locallink", :locallink |||| A Simple Two-column Table || || Column A || Column B || || Apples || Oranges || Zope-2.8.7-final/lib/python/StructuredText/regressions/InnerLinks.stx0000664000076400007640000000024010435500325025010 0ustar ajungajungThis is the InnerLinkTest see also [1] and [2] .. [1] "Zope Book" by Amos Lattmeier and Michel Pelletier .. [2] "Python Book" by Guido van Rossum Zope-2.8.7-final/lib/python/StructuredText/regressions/MultiMapping.stx0000664000076400007640000003200210435500325025343 0ustar ajungajungExample: MultiMapping objects "Copyright (C) 1996-1998, Digital Creations":COPYRIGHT.html. As an example, consider an extension class that implements a "MultiMapping". A multi-mapping is an object that encapsulates 0 or more mapping objects. When an attempt is made to lookup an object, the encapsulated mapping objects are searched until an object is found. Consider an implementation of a MultiMapping extension type, without use of the extension class mechanism:: #include "Python.h" #define UNLESS(E) if(!(E)) typedef struct { PyObject_HEAD PyObject *data; } MMobject; staticforward PyTypeObject MMtype; static PyObject * MM_push(MMobject *self, PyObject *args){ PyObject *src; UNLESS(PyArg_ParseTuple(args, "O", &src)) return NULL; UNLESS(-1 != PyList_Append(self->data,src)) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * MM_pop(MMobject *self, PyObject *args){ long l; PyObject *r; static PyObject *emptyList=0; UNLESS(emptyList) UNLESS(emptyList=PyList_New(0)) return NULL; UNLESS(PyArg_ParseTuple(args, "")) return NULL; UNLESS(-1 != (l=PyList_Size(self->data))) return NULL; l--; UNLESS(r=PySequence_GetItem(self->data,l)) return NULL; UNLESS(-1 != PyList_SetSlice(self->data,l,l+1,emptyList)) goto err; return r; err: Py_DECREF(r); return NULL; } static struct PyMethodDef MM_methods[] = { {"push", (PyCFunction) MM_push, 1, "push(mapping_object) -- Add a data source"}, {"pop", (PyCFunction) MM_pop, 1, "pop() -- Remove and return the last data source added"}, {NULL, NULL} /* sentinel */ }; static PyObject * newMMobject(PyObject *ignored, PyObject *args){ MMobject *self; UNLESS(PyArg_ParseTuple(args, "")) return NULL; UNLESS(self = PyObject_NEW(MMobject, &MMtype)) return NULL; UNLESS(self->data=PyList_New(0)) goto err; return (PyObject *)self; err: Py_DECREF(self); return NULL; } static void MM_dealloc(MMobject *self){ Py_XDECREF(self->data); PyMem_DEL(self); } static PyObject * MM_getattr(MMobject *self, char *name){ return Py_FindMethod(MM_methods, (PyObject *)self, name); } static int MM_length(MMobject *self){ long l=0, el, i; PyObject *e=0; UNLESS(-1 != (i=PyList_Size(self->data))) return -1; while(--i >= 0) { e=PyList_GetItem(self->data,i); UNLESS(-1 != (el=PyObject_Length(e))) return -1; l+=el; } return l; } static PyObject * MM_subscript(MMobject *self, PyObject *key){ long i; PyObject *e; UNLESS(-1 != (i=PyList_Size(self->data))) return NULL; while(--i >= 0) { e=PyList_GetItem(self->data,i); if(e=PyObject_GetItem(e,key)) return e; PyErr_Clear(); } PyErr_SetObject(PyExc_KeyError,key); return NULL; } static PyMappingMethods MM_as_mapping = { (inquiry)MM_length, /*mp_length*/ (binaryfunc)MM_subscript, /*mp_subscript*/ (objobjargproc)NULL, /*mp_ass_subscript*/ }; /* -------------------------------------------------------- */ static char MMtype__doc__[] = "MultiMapping -- Combine multiple mapping objects for lookup" ; static PyTypeObject MMtype = { PyObject_HEAD_INIT(&PyType_Type) 0, /*ob_size*/ "MultMapping", /*tp_name*/ sizeof(MMobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)MM_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)MM_getattr, /*tp_getattr*/ (setattrfunc)0, /*tp_setattr*/ (cmpfunc)0, /*tp_compare*/ (reprfunc)0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ &MM_as_mapping, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)0, /*tp_call*/ (reprfunc)0, /*tp_str*/ /* Space for future expansion */ 0L,0L,0L,0L, MMtype__doc__ /* Documentation string */ }; static struct PyMethodDef MultiMapping_methods[] = { {"MultiMapping", (PyCFunction)newMMobject, 1, "MultiMapping() -- Create a new empty multi-mapping"}, {NULL, NULL} /* sentinel */ }; void initMultiMapping(){ PyObject *m; m = Py_InitModule4( "MultiMapping", MultiMapping_methods, "MultiMapping -- Wrap multiple mapping objects for lookup", (PyObject*)NULL,PYTHON_API_VERSION); if (PyErr_Occurred()) Py_FatalError("can't initialize module MultiMapping"); } This module defines an extension type, 'MultiMapping', and exports a module function, 'MultiMapping', that creates 'MultiMapping' Instances. The type provides two methods, 'push', and 'pop', for adding and removing mapping objects to the multi-mapping. The type provides mapping behavior, implementing mapping length and subscript operators but not mapping a subscript assignment operator. Now consider an extension class implementation of MultiMapping objects:: #include "Python.h" #include "ExtensionClass.h" #define UNLESS(E) if(!(E)) typedef struct { PyObject_HEAD PyObject *data; } MMobject; staticforward PyExtensionClass MMtype; static PyObject * MM_push(self, args) MMobject *self; PyObject *args; { PyObject *src; UNLESS(PyArg_ParseTuple(args, "O", &src)) return NULL; UNLESS(-1 != PyList_Append(self->data,src)) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * MM_pop(self, args) MMobject *self; PyObject *args; { long l; PyObject *r; static PyObject *emptyList=0; UNLESS(emptyList) UNLESS(emptyList=PyList_New(0)) return NULL; UNLESS(PyArg_ParseTuple(args, "")) return NULL; UNLESS(-1 != (l=PyList_Size(self->data))) return NULL; l--; UNLESS(r=PySequence_GetItem(self->data,l)) return NULL; UNLESS(-1 != PyList_SetSlice(self->data,l,l+1,emptyList)) goto err; return r; err: Py_DECREF(r); return NULL; } static PyObject * MM__init__(self, args) MMobject *self; PyObject *args; { UNLESS(PyArg_ParseTuple(args, "")) return NULL; UNLESS(self->data=PyList_New(0)) goto err; Py_INCREF(Py_None); return Py_None; err: Py_DECREF(self); return NULL; } static struct PyMethodDef MM_methods[] = { {"__init__", (PyCFunction)MM__init__, 1, "__init__() -- Create a new empty multi-mapping"}, {"push", (PyCFunction) MM_push, 1, "push(mapping_object) -- Add a data source"}, {"pop", (PyCFunction) MM_pop, 1, "pop() -- Remove and return the last data source added"}, {NULL, NULL} /* sentinel */ }; static void MM_dealloc(self) MMobject *self; { Py_XDECREF(self->data); PyMem_DEL(self); } static PyObject * MM_getattr(self, name) MMobject *self; char *name; { return Py_FindMethod(MM_methods, (PyObject *)self, name); } static int MM_length(self) MMobject *self; { long l=0, el, i; PyObject *e=0; UNLESS(-1 != (i=PyList_Size(self->data))) return -1; while(--i >= 0) { e=PyList_GetItem(self->data,i); UNLESS(-1 != (el=PyObject_Length(e))) return -1; l+=el; } return l; } static PyObject * MM_subscript(self, key) MMobject *self; PyObject *key; { long i; PyObject *e; UNLESS(-1 != (i=PyList_Size(self->data))) return NULL; while(--i >= 0) { e=PyList_GetItem(self->data,i); if(e=PyObject_GetItem(e,key)) return e; PyErr_Clear(); } PyErr_SetObject(PyExc_KeyError,key); return NULL; } static PyMappingMethods MM_as_mapping = { (inquiry)MM_length, /*mp_length*/ (binaryfunc)MM_subscript, /*mp_subscript*/ (objobjargproc)NULL, /*mp_ass_subscript*/ }; /* -------------------------------------------------------- */ static char MMtype__doc__[] = "MultiMapping -- Combine multiple mapping objects for lookup" ; static PyExtensionClass MMtype = { PyObject_HEAD_INIT(&PyType_Type) 0, /*ob_size*/ "MultMapping", /*tp_name*/ sizeof(MMobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)MM_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)MM_getattr, /*tp_getattr*/ (setattrfunc)0, /*tp_setattr*/ (cmpfunc)0, /*tp_compare*/ (reprfunc)0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ &MM_as_mapping, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)0, /*tp_call*/ (reprfunc)0, /*tp_str*/ /* Space for future expansion */ 0L,0L,0L,0L, MMtype__doc__, /* Documentation string */ METHOD_CHAIN(MM_methods) }; static struct PyMethodDef MultiMapping_methods[] = { {NULL, NULL} /* sentinel */ }; void initMultiMapping() { PyObject *m, *d; m = Py_InitModule4( "MultiMapping", MultiMapping_methods, "MultiMapping -- Wrap multiple mapping objects for lookup", (PyObject*)NULL,PYTHON_API_VERSION); d = PyModule_GetDict(m); PyExtensionClass_Export(d,"MultiMapping",MMtype); if (PyErr_Occurred()) Py_FatalError("can't initialize module MultiMapping"); } This version includes 'ExtensionClass.h'. The two declarations of 'MMtype' have been changed from 'PyTypeObject' to 'PyExtensionClass'. The 'METHOD_CHAIN' macro has been used to add methods to the end of the definition for 'MMtype'. The module function, newMMobject has been replaced by the 'MMtype' method, 'MM__init__'. Note that this method does not create or return a new object. Finally, the lines:: d = PyModule_GetDict(m); PyExtensionClass_Export(d,"MultiMapping",MMtype); Have been added to both initialize the extension class and to export it in the module dictionary. To use this module, compile, link, and import it as with any other extension module. The following python code illustrates the module's use:: from MultiMapping import MultiMapping m=MultiMapping() m.push({'spam':1, 'eggs':2}) m.push({'spam':3, 'ham':4}) m['spam'] # returns 3 m['ham'] # returns 4 m['foo'] # raises a key error Creating the 'MultiMapping' object took three steps, one to create an empty 'MultiMapping', and two to add mapping objects to it. We might wish to simplify the process of creating MultiMapping objects by providing a constructor that takes source mapping objects as parameters. We can do this by sub-classing MultiMapping in Python:: from MultiMapping import MultiMapping class ExtendedMultiMapping(MultiMapping): def __init__(self,*data): MultiMapping.__init__(self) for d in data: self.push(d) m=ExtendedMultiMapping({'spam':1, 'eggs':2}, {'spam':3, 'ham':4}) m['spam'] # returns 3 m['ham'] # returns 4 m['foo'] # raises a key error Note that the source file included in the ExtensionClass distribution has numerous enhancements beyond the version shown in this document. Zope-2.8.7-final/lib/python/StructuredText/regressions/Acquisition.stx0000664000076400007640000002175610435500325025243 0ustar ajungajungAcquisition "Copyright (C) 1996-1998, Digital Creations":COPYRIGHT.html. Acquisition [1] is a mechanism that allows objects to obtain attributes from their environment. It is similar to inheritence, except that, rather than traversing an inheritence hierarchy to obtain attributes, a containment hierarchy is traversed. The "ExtensionClass":ExtensionClass.html. release includes mix-in extension base classes that can be used to add acquisition as a feature to extension subclasses. These mix-in classes use the context-wrapping feature of ExtensionClasses to implement acquisition. Consider the following example:: import ExtensionClass, Acquisition class C(ExtensionClass.Base): color='red' class A(Acquisition.Implicit): def report(self): print self.color a=A() c=C() c.a=A() c.a.report() # prints 'red' d=C() d.color='green' d.a=a d.a.report() # prints 'green' a.report() # raises an attribute error The class 'A' inherits acquisition behavior from 'Acquisition.Implicit'. The object, 'a', "has" the color of objects 'c' and 'd' when it is accessed through them, but it has no color by itself. The object 'a' obtains attributes from it's environment, where it's environment is defined by the access path used to reach 'a'. Acquisition wrappers When an object that supports acquisition is accessed through an extension class instance, a special object, called an acquisition wrapper, is returned. In the example above, the expression 'c.a' returns an acquisition wrapper that contains references to both 'c' and 'a'. It is this wrapper that performs attribute lookup in 'c' when an attribute cannot be found in 'a'. Aquisition wrappers provide access to the wrapped objects through the attributes 'aq_parent', 'aq_self', 'aq_base'. In the example above, the expressions:: 'c.a.aq_parent is c' and:: 'c.a.aq_self is a' both evaluate to true, but the expression:: 'c.a is a' evaluates to false, because the expression 'c.a' evaluates to an acquisition wrapper around 'c' and 'a', not 'a' itself. The attribute 'aq_base' is similar to 'aq_self'. Wrappers may be nested and 'aq_self' may be a wrapped object. The 'aq_base' attribute is the underlying object with all wrappers removed. Acquisition Control Two styles of acquisition are supported in the current ExtensionClass release, implicit and explicit aquisition. Implicit acquisition Implicit acquisition is so named because it searches for attributes from the environment automatically whenever an attribute cannot be obtained directly from an object or through inheritence. An attribute may be implicitly acquired if it's name does not begin with an underscore, '_'. To support implicit acquisition, an object should inherit from the mix-in class 'Acquisition.Implicit'. Explicit Acquisition When explicit acquisition is used, attributes are not automatically obtained from the environment. Instead, the method 'aq_aquire' must be used, as in:: print c.a.aq_acquire('color') To support explicit acquisition, an object should inherit from the mix-in class 'Acquisition.Explicit'. Controlled Acquisition A class (or instance) can provide attribute by attribute control over acquisition. This is done by: - subclassing from 'Acquisition.Explicit', and - setting all attributes that should be acquired to the special value: 'Acquisition.Acquired'. Setting an attribute to this value also allows inherited attributes to be overridden with acquired ones. For example, in:: class C(Acquisition.Explicit): id=1 secret=2 color=Acquisition.Acquired __roles__=Acquisition.Acquired The *only* attributes that are automatically acquired from containing objects are 'color', and '__roles__'. Note also that the '__roles__' attribute is acquired even though it's name begins with an underscore. In fact, the special 'Acquisition.Acquired' value can be used in 'Acquisition.Implicit' objects to implicitly acquire selected objects that smell like private objects. Filtered Acquisition The acquisition method, 'aq_acquire', accepts two optional arguments. The first of the additional arguments is a "filtering" function that is used when considering whether to acquire an object. The second of the additional arguments is an object that is passed as extra data when calling the filtering function and which defaults to 'None'. The filter function is called with five arguments: - The object that the 'aq_acquire' method was called on, - The object where an object was found, - The name of the object, as passed to 'aq_acquire', - The object found, and - The extra data passed to 'aq_acquire'. If the filter returns a true object that the object found is returned, otherwise, the acquisition search continues. For example, in:: from Acquisition import Explicit class HandyForTesting: def __init__(self, name): self.name=name def __str__(self): return "%s(%s)" % (self.name, self.__class__.__name__) __repr__=__str__ class E(Explicit, HandyForTesting): pass class Nice(HandyForTesting): isNice=1 def __str__(self): return HandyForTesting.__str__(self)+' and I am nice!' __repr__=__str__ a=E('a') a.b=E('b') a.b.c=E('c') a.p=Nice('spam') a.b.p=E('p') def find_nice(self, ancestor, name, object, extra): return hasattr(object,'isNice') and object.isNice print a.b.c.aq_acquire('p', find_nice) The filtered acquisition in the last line skips over the first attribute it finds with the name 'p', because the attribute doesn't satisfy the condition given in the filter. The output of the last line is:: spam(Nice) and I am nice! Acquisition and methods Python methods of objects that support acquisition can use acquired attributes as in the 'report' method of the first example above. When a Python method is called on an object that is wrapped by an acquisition wrapper, the wrapper is passed to the method as the first argument. This rule also applies to user-defined method types and to C methods defined in pure mix-in classes. Unfortunately, C methods defined in extension base classes that define their own data structures, cannot use aquired attributes at this time. This is because wrapper objects do not conform to the data structures expected by these methods. Acquiring Acquiring objects Consider the following example:: from Acquisition import Implicit class C(Implicit): def __init__(self, name): self.name=name def __str__(self): return "%s(%s)" % (self.name, self.__class__.__name__) __repr__=__str__ a=C("a") a.b=C("b") a.b.pref="spam" a.b.c=C("c") a.b.c.color="red" a.b.c.pref="eggs" a.x=C("x") o=a.b.c.x The expression 'o.color' might be expected to return '"red"'. In earlier versions of ExtensionClass, however, this expression failed. Acquired acquiring objects did not acquire from the environment they were accessed in, because objects were only wrapped when they were first found, and were not rewrapped as they were passed down the acquisition tree. In the current release of ExtensionClass, the expression "o.color" does indeed return '"red"'. When searching for an attribute in 'o', objects are searched in the order 'x', 'a', 'b', 'c'. So, for example, the expression, 'o.pref' returns '"spam"', not '"eggs"'. In earlier releases of ExtensionClass, the attempt to get the 'pref' attribute from 'o' would have failed. If desired, the current rules for looking up attributes in complex expressions can best be understood through repeated application of the '__of__' method: 'a.x' -- 'x.__of__(a)' 'a.b' -- 'b.__of__(a)' 'a.b.x' -- 'x.__of__(a).__of__(b.__of__(a))' 'a.b.c' -- 'c.__of__(b.__of__(a))' 'a.b.c.x' -- 'x.__of__(a).__of__(b.__of__(a)).__of__(c.__of__(b.__of__(a)))' and by keeping in mind that attribute lookup in a wrapper is done by trying to lookup the attribute in the wrapped object first and then in the parent object. In the expressions above involving the '__of__' method, lookup proceeds from left to right. Note that heuristics are used to avoid most of the repeated lookups. For example, in the expression: 'a.b.c.x.foo', the object 'a' is searched no more than once, even though it is wrapped three times. .. [1] Gil, J., Lorenz, D., "Environmental Acquisition--A New Inheritance-Like Abstraction Mechanism", http://www.bell-labs.com/people/cope/oopsla/Oopsla96TechnicalProgramAbstracts.html#GilLorenz, OOPSLA '96 Proceedings, ACM SIG-PLAN, October, 1996 Zope-2.8.7-final/lib/python/StructuredText/regressions/index.ref0000664000076400007640000000376510435500325024020 0ustar ajungajung Extension Class

Extension Class

Copyright (C) 1996-1998, Digital Creations.

A lightweight mechanism has been developed for making Python extension types more class-like. Classes can be developed in an extension language, such as C or C++, and these classes can be treated like other python classes:

  • They can be sub-classed in python,
  • They provide access to method documentation strings, and
  • They can be used to directly create new instances.

Extension classes provide additional extensions to class and instance semantics, including:

  • A protocol for accessing subobjects "in the context of" their containers. This is used to implement custom method types and environmental acquisition.
  • A protocol for overriding method call semantics. This is used to implement "synchonized" classes and could be used to implement argument type checking.
  • A protocol for class initialization that supports execution of a special __class_init__ method after a class has been initialized.

Extension classes illustrate how the Python class mechanism can be extended and may provide a basis for improved or specialized class models.

Releases

The current release is 1.2, To find out what's changed in this release, see the release notes.

Documentation is available on-line.

Windows Binaries

A win32 binary release, ec12.zip, is available. This release includes all of the ExtensionClass modules built as Windows extension modules (.pyd) files. These were built for Python 1.5.1 using Microsoft Visual C++ 5.0 in "Release" mode.

Zope-2.8.7-final/lib/python/StructuredText/regressions/create_referencesfiles.py0000664000076400007640000000172310435500325027244 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os,sys from StructuredText.StructuredText import HTML if len(sys.argv)>1: files = sys.argv[1:] else: files = os.listdir('.') files = filter(lambda x: x.endswith('.stx'), files) for f in files: data = open(f,'r').read() html = HTML(data) outfile = f.replace('.stx','.ref') open(outfile,'w').write(html) Zope-2.8.7-final/lib/python/StructuredText/regressions/Acquisition.ref0000664000076400007640000002507410435500325025176 0ustar ajungajung Acquisition

Acquisition

Copyright (C) 1996-1998, Digital Creations.

Acquisition [1] is a mechanism that allows objects to obtain attributes from their environment. It is similar to inheritence, except that, rather than traversing an inheritence hierarchy to obtain attributes, a containment hierarchy is traversed.

The ExtensionClass. release includes mix-in extension base classes that can be used to add acquisition as a feature to extension subclasses. These mix-in classes use the context-wrapping feature of ExtensionClasses to implement acquisition. Consider the following example: 

    import ExtensionClass, Acquisition

    class C(ExtensionClass.Base):
      color='red'

    class A(Acquisition.Implicit):

      def report(self):
        print self.color

    a=A()
    c=C()
    c.a=A()

    c.a.report() # prints 'red'

    d=C()
    d.color='green'
    d.a=a

    d.a.report() # prints 'green'

    a.report() # raises an attribute error

The class A inherits acquisition behavior from Acquisition.Implicit. The object, a, "has" the color of objects c and d when it is accessed through them, but it has no color by itself. The object a obtains attributes from it's environment, where it's environment is defined by the access path used to reach a.

Acquisition wrappers

When an object that supports acquisition is accessed through an extension class instance, a special object, called an acquisition wrapper, is returned. In the example above, the expression c.a returns an acquisition wrapper that contains references to both c and a. It is this wrapper that performs attribute lookup in c when an attribute cannot be found in a.

Aquisition wrappers provide access to the wrapped objects through the attributes aq_parent, aq_self, aq_base. In the example above, the expressions: 

       'c.a.aq_parent is c'

and: 

       'c.a.aq_self is a'

both evaluate to true, but the expression: 

       'c.a is a'

evaluates to false, because the expression c.a evaluates to an acquisition wrapper around c and a, not a itself.

The attribute aq_base is similar to aq_self. Wrappers may be nested and aq_self may be a wrapped object. The aq_base attribute is the underlying object with all wrappers removed.

Acquisition Control

Two styles of acquisition are supported in the current ExtensionClass release, implicit and explicit aquisition.

Implicit acquisition

Implicit acquisition is so named because it searches for attributes from the environment automatically whenever an attribute cannot be obtained directly from an object or through inheritence.

An attribute may be implicitly acquired if it's name does not begin with an underscore, _.

To support implicit acquisition, an object should inherit from the mix-in class Acquisition.Implicit.

Explicit Acquisition

When explicit acquisition is used, attributes are not automatically obtained from the environment. Instead, the method aq_aquire must be used, as in: 

        print c.a.aq_acquire('color')

To support explicit acquisition, an object should inherit from the mix-in class Acquisition.Explicit.

Controlled Acquisition

A class (or instance) can provide attribute by attribute control over acquisition. This is done by:

  • subclassing from Acquisition.Explicit, and
  • setting all attributes that should be acquired to the special value: Acquisition.Acquired. Setting an attribute to this value also allows inherited attributes to be overridden with acquired ones.

    For example, in: 

              class C(Acquisition.Explicit):
             id=1
             secret=2
             color=Acquisition.Acquired
             __roles__=Acquisition.Acquired

    The only attributes that are automatically acquired from containing objects are color, and __roles__. Note also that the __roles__ attribute is acquired even though it's name begins with an underscore. In fact, the special Acquisition.Acquired value can be used in Acquisition.Implicit objects to implicitly acquire selected objects that smell like private objects.

Filtered Acquisition

The acquisition method, aq_acquire, accepts two optional arguments. The first of the additional arguments is a "filtering" function that is used when considering whether to acquire an object. The second of the additional arguments is an object that is passed as extra data when calling the filtering function and which defaults to None.

The filter function is called with five arguments:

  • The object that the aq_acquire method was called on,
  • The object where an object was found,
  • The name of the object, as passed to aq_acquire,
  • The object found, and
  • The extra data passed to aq_acquire.

If the filter returns a true object that the object found is returned, otherwise, the acquisition search continues.

For example, in: 

        from Acquisition import Explicit

        class HandyForTesting:
            def __init__(self, name): self.name=name
            def __str__(self):
                return "%s(%s)" % (self.name, self.__class__.__name__)
            __repr__=__str__

        class E(Explicit, HandyForTesting): pass

        class Nice(HandyForTesting):
            isNice=1
            def __str__(self):
                return HandyForTesting.__str__(self)+' and I am nice!'
            __repr__=__str__

        a=E('a')
        a.b=E('b')
        a.b.c=E('c')
        a.p=Nice('spam')
        a.b.p=E('p')

        def find_nice(self, ancestor, name, object, extra):
            return hasattr(object,'isNice') and object.isNice

        print a.b.c.aq_acquire('p', find_nice)

The filtered acquisition in the last line skips over the first attribute it finds with the name p, because the attribute doesn't satisfy the condition given in the filter. The output of the last line is: 

        spam(Nice) and I am nice!

Acquisition and methods

Python methods of objects that support acquisition can use acquired attributes as in the report method of the first example above. When a Python method is called on an object that is wrapped by an acquisition wrapper, the wrapper is passed to the method as the first argument. This rule also applies to user-defined method types and to C methods defined in pure mix-in classes.

Unfortunately, C methods defined in extension base classes that define their own data structures, cannot use aquired attributes at this time. This is because wrapper objects do not conform to the data structures expected by these methods.

Acquiring Acquiring objects

Consider the following example: 

      from Acquisition import Implicit

      class C(Implicit):
          def __init__(self, name): self.name=name
          def __str__(self):
              return "%s(%s)" % (self.name, self.__class__.__name__)
          __repr__=__str__

      a=C("a")
      a.b=C("b")
      a.b.pref="spam"
      a.b.c=C("c")
      a.b.c.color="red"
      a.b.c.pref="eggs"
      a.x=C("x")

      o=a.b.c.x

The expression o.color might be expected to return "red". In earlier versions of ExtensionClass, however, this expression failed. Acquired acquiring objects did not acquire from the environment they were accessed in, because objects were only wrapped when they were first found, and were not rewrapped as they were passed down the acquisition tree.

In the current release of ExtensionClass, the expression "o.color" does indeed return "red".

When searching for an attribute in o, objects are searched in the order x, a, b, c. So, for example, the expression, o.pref returns "spam", not "eggs". In earlier releases of ExtensionClass, the attempt to get the pref attribute from o would have failed.

If desired, the current rules for looking up attributes in complex expressions can best be understood through repeated application of the __of__ method:

a.x
x.__of__(a)
a.b
b.__of__(a)
a.b.x
x.__of__(a).__of__(b.__of__(a))
a.b.c
c.__of__(b.__of__(a))
a.b.c.x
x.__of__(a).__of__(b.__of__(a)).__of__(c.__of__(b.__of__(a)))

and by keeping in mind that attribute lookup in a wrapper is done by trying to lookup the attribute in the wrapped object first and then in the parent object. In the expressions above involving the __of__ method, lookup proceeds from left to right.

Note that heuristics are used to avoid most of the repeated lookups. For example, in the expression: a.b.c.x.foo, the object a is searched no more than once, even though it is wrapped three times.

[1] Gil, J., Lorenz, D., Environmental Acquisition--A New Inheritance-Like Abstraction Mechanism, OOPSLA '96 Proceedings, ACM SIG-PLAN, October, 1996

Zope-2.8.7-final/lib/python/StructuredText/regressions/MultiMapping.ref0000664000076400007640000003245410435500325025314 0ustar ajungajung Example: MultiMapping objects

Example: MultiMapping objects

Copyright (C) 1996-1998, Digital Creations.

As an example, consider an extension class that implements a "MultiMapping". A multi-mapping is an object that encapsulates 0 or more mapping objects. When an attempt is made to lookup an object, the encapsulated mapping objects are searched until an object is found.

Consider an implementation of a MultiMapping extension type, without use of the extension class mechanism: 

    #include "Python.h"

    #define UNLESS(E) if(!(E))

    typedef struct {
        PyObject_HEAD
        PyObject *data;
    } MMobject;

    staticforward PyTypeObject MMtype;

    static PyObject *
    MM_push(MMobject *self, PyObject *args){
        PyObject *src;
        UNLESS(PyArg_ParseTuple(args, "O", &src)) return NULL;
        UNLESS(-1 != PyList_Append(self->data,src)) return NULL;
        Py_INCREF(Py_None);
        return Py_None;
    }

    static PyObject *
    MM_pop(MMobject *self, PyObject *args){
        long l;
        PyObject *r;
        static PyObject *emptyList=0;

        UNLESS(emptyList) UNLESS(emptyList=PyList_New(0)) return NULL;
        UNLESS(PyArg_ParseTuple(args, "")) return NULL;
        UNLESS(-1 != (l=PyList_Size(self->data))) return NULL;
        l--;
        UNLESS(r=PySequence_GetItem(self->data,l)) return NULL;
        UNLESS(-1 != PyList_SetSlice(self->data,l,l+1,emptyList)) goto err;
        return r;
    err:
        Py_DECREF(r);
        return NULL;
    }

    static struct PyMethodDef MM_methods[] = {
        {"push", (PyCFunction) MM_push, 1,
         "push(mapping_object) -- Add a data source"},
        {"pop",  (PyCFunction) MM_pop,  1,
         "pop() -- Remove and return the last data source added"}, 
        {NULL,              NULL}           /* sentinel */
    };

    static PyObject *
    newMMobject(PyObject *ignored, PyObject *args){
        MMobject *self;

        UNLESS(PyArg_ParseTuple(args, "")) return NULL;
        UNLESS(self = PyObject_NEW(MMobject, &MMtype)) return NULL;
        UNLESS(self->data=PyList_New(0)) goto err;
        return (PyObject *)self;
    err:
        Py_DECREF(self);
        return NULL;
    }

    static void
    MM_dealloc(MMobject *self){
        Py_XDECREF(self->data);
        PyMem_DEL(self);
    }

    static PyObject *
    MM_getattr(MMobject *self, char *name){
        return Py_FindMethod(MM_methods, (PyObject *)self, name);
    }

    static int
    MM_length(MMobject *self){
        long l=0, el, i;
        PyObject *e=0;

        UNLESS(-1 != (i=PyList_Size(self->data))) return -1;
        while(--i >= 0)
          {
            e=PyList_GetItem(self->data,i);
            UNLESS(-1 != (el=PyObject_Length(e))) return -1;
            l+=el;
          }
        return l;
    }

    static PyObject *
    MM_subscript(MMobject *self, PyObject *key){
        long i;
        PyObject *e;

        UNLESS(-1 != (i=PyList_Size(self->data))) return NULL;
        while(--i >= 0)
          {
            e=PyList_GetItem(self->data,i);
            if(e=PyObject_GetItem(e,key)) return e;
            PyErr_Clear();
          }
        PyErr_SetObject(PyExc_KeyError,key);
        return NULL;
    }

    static PyMappingMethods MM_as_mapping = {
              (inquiry)MM_length,           /*mp_length*/
              (binaryfunc)MM_subscript,             /*mp_subscript*/
              (objobjargproc)NULL,          /*mp_ass_subscript*/
    };

    /* -------------------------------------------------------- */

    static char MMtype__doc__[] = 
    "MultiMapping -- Combine multiple mapping objects for lookup"
    ;

    static PyTypeObject MMtype = {
              PyObject_HEAD_INIT(&PyType_Type)
              0,                            /*ob_size*/
              "MultMapping",                        /*tp_name*/
              sizeof(MMobject),             /*tp_basicsize*/
              0,                            /*tp_itemsize*/
              /* methods */
              (destructor)MM_dealloc,               /*tp_dealloc*/
              (printfunc)0,                 /*tp_print*/
              (getattrfunc)MM_getattr,      /*tp_getattr*/
              (setattrfunc)0,                       /*tp_setattr*/
              (cmpfunc)0,                   /*tp_compare*/
              (reprfunc)0,                  /*tp_repr*/
              0,                            /*tp_as_number*/
              0,                            /*tp_as_sequence*/
              &MM_as_mapping,                       /*tp_as_mapping*/
              (hashfunc)0,                  /*tp_hash*/
              (ternaryfunc)0,                       /*tp_call*/
              (reprfunc)0,                  /*tp_str*/

              /* Space for future expansion */
              0L,0L,0L,0L,
              MMtype__doc__ /* Documentation string */
    };

    static struct PyMethodDef MultiMapping_methods[] = {
        {"MultiMapping", (PyCFunction)newMMobject, 1,
         "MultiMapping() -- Create a new empty multi-mapping"},
        {NULL,              NULL}           /* sentinel */
    };

    void
    initMultiMapping(){
        PyObject *m;

        m = Py_InitModule4(
            "MultiMapping", MultiMapping_methods,
              "MultiMapping -- Wrap multiple mapping objects for lookup",
              (PyObject*)NULL,PYTHON_API_VERSION);

        if (PyErr_Occurred()) 
           Py_FatalError("can't initialize module MultiMapping");
    }

This module defines an extension type, MultiMapping, and exports a module function, MultiMapping, that creates MultiMapping Instances. The type provides two methods, push, and pop, for adding and removing mapping objects to the multi-mapping. The type provides mapping behavior, implementing mapping length and subscript operators but not mapping a subscript assignment operator.

Now consider an extension class implementation of MultiMapping objects: 

    #include "Python.h"
    #include "ExtensionClass.h"

    #define UNLESS(E) if(!(E))

    typedef struct {
        PyObject_HEAD
        PyObject *data;
    } MMobject;

    staticforward PyExtensionClass MMtype;

    static PyObject *
    MM_push(self, args)
              MMobject *self;
              PyObject *args;
    {
        PyObject *src;
        UNLESS(PyArg_ParseTuple(args, "O", &src)) return NULL;
        UNLESS(-1 != PyList_Append(self->data,src)) return NULL;
        Py_INCREF(Py_None);
        return Py_None;
    }

    static PyObject *
    MM_pop(self, args)
              MMobject *self;
              PyObject *args;
    {
        long l;
        PyObject *r;
        static PyObject *emptyList=0;

        UNLESS(emptyList) UNLESS(emptyList=PyList_New(0)) return NULL;
        UNLESS(PyArg_ParseTuple(args, "")) return NULL;
        UNLESS(-1 != (l=PyList_Size(self->data))) return NULL;
        l--;
        UNLESS(r=PySequence_GetItem(self->data,l)) return NULL;
        UNLESS(-1 != PyList_SetSlice(self->data,l,l+1,emptyList)) goto err;
        return r;
    err:
        Py_DECREF(r);
        return NULL;
    }

    static PyObject *
    MM__init__(self, args)
           MMobject *self;
           PyObject *args;
    {
        UNLESS(PyArg_ParseTuple(args, "")) return NULL;
        UNLESS(self->data=PyList_New(0)) goto err;
        Py_INCREF(Py_None);
        return Py_None;
    err:
        Py_DECREF(self);
        return NULL;
    }

    static struct PyMethodDef MM_methods[] = {
        {"__init__", (PyCFunction)MM__init__, 1,
         "__init__() -- Create a new empty multi-mapping"},
        {"push", (PyCFunction) MM_push, 1,
         "push(mapping_object) -- Add a data source"},
        {"pop",  (PyCFunction) MM_pop,  1,
         "pop() -- Remove and return the last data source added"}, 
        {NULL,              NULL}           /* sentinel */
    };

    static void
    MM_dealloc(self)
           MMobject *self;
    {
        Py_XDECREF(self->data);
        PyMem_DEL(self);
    }

    static PyObject *
    MM_getattr(self, name)
              MMobject *self;
              char *name;
    {
        return Py_FindMethod(MM_methods, (PyObject *)self, name);
    }

    static int
    MM_length(self)
              MMobject *self;
    {
        long l=0, el, i;
        PyObject *e=0;

        UNLESS(-1 != (i=PyList_Size(self->data))) return -1;
        while(--i >= 0)
          {
            e=PyList_GetItem(self->data,i);
            UNLESS(-1 != (el=PyObject_Length(e))) return -1;
            l+=el;
          }
        return l;
    }

    static PyObject *
    MM_subscript(self, key)
              MMobject *self;
              PyObject *key;
    {
        long i;
        PyObject *e;

        UNLESS(-1 != (i=PyList_Size(self->data))) return NULL;
        while(--i >= 0)
          {
            e=PyList_GetItem(self->data,i);
            if(e=PyObject_GetItem(e,key)) return e;
            PyErr_Clear();
          }
        PyErr_SetObject(PyExc_KeyError,key);
        return NULL;
    }

    static PyMappingMethods MM_as_mapping = {
              (inquiry)MM_length,           /*mp_length*/
              (binaryfunc)MM_subscript,             /*mp_subscript*/
              (objobjargproc)NULL,          /*mp_ass_subscript*/
    };

    /* -------------------------------------------------------- */

    static char MMtype__doc__[] = 
    "MultiMapping -- Combine multiple mapping objects for lookup"
    ;

    static PyExtensionClass MMtype = {
              PyObject_HEAD_INIT(&PyType_Type)
              0,                            /*ob_size*/
              "MultMapping",                        /*tp_name*/
              sizeof(MMobject),             /*tp_basicsize*/
              0,                            /*tp_itemsize*/
              /* methods */
              (destructor)MM_dealloc,               /*tp_dealloc*/
              (printfunc)0,                 /*tp_print*/
              (getattrfunc)MM_getattr,      /*tp_getattr*/
              (setattrfunc)0,                       /*tp_setattr*/
              (cmpfunc)0,                   /*tp_compare*/
              (reprfunc)0,                  /*tp_repr*/
              0,                            /*tp_as_number*/
              0,                            /*tp_as_sequence*/
              &MM_as_mapping,                       /*tp_as_mapping*/
              (hashfunc)0,                  /*tp_hash*/
              (ternaryfunc)0,                       /*tp_call*/
              (reprfunc)0,                  /*tp_str*/

              /* Space for future expansion */
              0L,0L,0L,0L,
              MMtype__doc__, /* Documentation string */
              METHOD_CHAIN(MM_methods)
    };

    static struct PyMethodDef MultiMapping_methods[] = {
        {NULL,              NULL}           /* sentinel */
    };

    void
    initMultiMapping()
    {
        PyObject *m, *d;

        m = Py_InitModule4(
            "MultiMapping", MultiMapping_methods,
            "MultiMapping -- Wrap multiple mapping objects for lookup",
            (PyObject*)NULL,PYTHON_API_VERSION);
        d = PyModule_GetDict(m);
        PyExtensionClass_Export(d,"MultiMapping",MMtype);

        if (PyErr_Occurred()) 
           Py_FatalError("can't initialize module MultiMapping");
    }

This version includes ExtensionClass.h. The two declarations of MMtype have been changed from PyTypeObject to PyExtensionClass. The METHOD_CHAIN macro has been used to add methods to the end of the definition for MMtype. The module function, newMMobject has been replaced by the MMtype method, MM__init__. Note that this method does not create or return a new object. Finally, the lines: 

    d = PyModule_GetDict(m);
    PyExtensionClass_Export(d,"MultiMapping",MMtype);

Have been added to both initialize the extension class and to export it in the module dictionary.

To use this module, compile, link, and import it as with any other extension module. The following python code illustrates the module's use: 

    from MultiMapping import MultiMapping
    m=MultiMapping()
    m.push({'spam':1, 'eggs':2})
    m.push({'spam':3, 'ham':4})

    m['spam'] # returns 3
    m['ham']  # returns 4
    m['foo']  # raises a key error

Creating the MultiMapping object took three steps, one to create an empty MultiMapping, and two to add mapping objects to it. We might wish to simplify the process of creating MultiMapping objects by providing a constructor that takes source mapping objects as parameters. We can do this by sub-classing MultiMapping in Python: 

    from MultiMapping import MultiMapping
    class ExtendedMultiMapping(MultiMapping):
        def __init__(self,*data):
          MultiMapping.__init__(self)
          for d in data: self.push(d)

    m=ExtendedMultiMapping({'spam':1, 'eggs':2}, {'spam':3, 'ham':4})

    m['spam'] # returns 3
    m['ham']  # returns 4
    m['foo']  # raises a key error

Note that the source file included in the ExtensionClass distribution has numerous enhancements beyond the version shown in this document.

Zope-2.8.7-final/lib/python/StructuredText/regressions/Links.stx0000664000076400007640000000142210435500325024017 0ustar ajungajungThis is LinkTest - please click "here":/Members/Zope - please click "here":/Members/Zope?a=b&c=d%20blabla - please click "here":http://www.zope.org - please click "here":http://www.zope.org/members/ - please click "here":http://www.zope.org:2001 - please click "here":http://www.zope.org:2001/members/ - please click "here":http://www.zope.org:2001/%20/Members/zope?a=222&b=213&_xc=just%20a%20test - please click "here":http://www.zope.org:2001/%20/Members/zope?a=222&b=213&_xc=just%20a%20test - please click "here", http://www.zope.org:2001/%20/Members/zope?a=222&b=213&_xc=just%20a%20test And now a paragraph with "Link 1":http://www.zope-rocks.org and "Link 2":http://www.zope-is-kewl.com and "one more link - yeah.":http://www.freshmeat.net Zope-2.8.7-final/lib/python/StructuredText/regressions/examples.ref0000664000076400007640000000301210435500325024510 0ustar ajungajung Small Trials for Structured Text Formatting

Small Trials for Structured Text Formatting

This paragraph should be preceded by a level 1 header. It should not, itself, be made into a header, just a regular paragraph.

Here are a few presentation styles, in a list [1]:

  • A word: emphasized.
  • A word: underlined.
  • A word strong.
  • An inline example: 1+2.
  • Another example with a different format: ``x='spam''' or ``y='spam''' or ``'.

    We can use expressions in the DTML var tag as in ``''

  • A mult-line example: 
         blah

     *foo bar*

     <dtml-var yeha>

[1] (The referring text should be a paragraph, not a header, and should contain a reference to this footnote, footnote "[1]".)

Some hrefs, in a definition list:

Regular
http://www.zope.org/
W/trailing punctuation
http://www.zope.org/.
W protocol implicit
locallink
W protocol implicit, alternate
"locallink", :locallink

|||| A Simple Two-column Table || || Column A || Column B || || Apples || Oranges ||

Zope-2.8.7-final/lib/python/StructuredText/StructuredText.py0000664000076400007640000000452510435500325023226 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Alias module for StructuredTextClassic compatibility which makes use of StructuredTextNG """ import HTMLClass, DocumentClass import DocumentWithImages, HTMLWithImages from ST import Basic import re, sys from STletters import letters Document = DocumentClass.DocumentClass() HTMLNG = HTMLClass.HTMLClass() DocumentImages = DocumentWithImages.DocumentWithImages() HTMLNGImages = HTMLWithImages.HTMLWithImages() def HTML(aStructuredString, level=1, header=1): st = Basic(aStructuredString) doc = DocumentImages(st) return HTMLNGImages(doc,header=header,level=level) def StructuredText(aStructuredString, level=1): return HTML(aStructuredString,level) def html_with_references(text, level=1, header=1): text = re.sub( r'[\000\n]\.\. \[([0-9_%s-]+)\]' % letters, r'\n [\1]', text) text = re.sub( r'([\000- ,])\[(?P[0-9_%s-]+)\]([\000- ,.:])' % letters, r'\1[\2]\3', text) text = re.sub( r'([\000- ,])\[([^]]+)\.html\]([\000- ,.:])', r'\1[\2]\3', text) return HTML(text,level=level,header=header) def html_quote(v, character_entities=( (re.compile('&'), '&'), (re.compile("<"), '<' ), (re.compile(">"), '>' ), (re.compile('"'), '"') )): #" text=str(v) for re,name in character_entities: text=re.sub(name,text) return text if __name__=='__main__': import getopt opts,args = getopt.getopt(sys.argv[1:],'',[]) for k,v in opts: pass for f in args: print HTML(open(f).read()) Zope-2.8.7-final/lib/python/StructuredText/__init__.py0000664000076400007640000000272110435500325021750 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import HTMLClass, DocumentClass import ClassicDocumentClass from StructuredText import html_with_references, HTML, html_quote from ST import Basic import DocBookClass import HTMLWithImages from types import StringType, UnicodeType import DocumentWithImages ClassicHTML=HTML HTMLNG=HTMLClass.HTMLClass() def HTML(src, level=1): if isinstance(src, StringType) or isinstance(src, UnicodeType): return ClassicHTML(src, level) return HTMLNG(src, level) Classic=ClassicDocumentClass.DocumentClass() Document=DocumentClass.DocumentClass() DocumentWithImages=DocumentWithImages.DocumentWithImages() HTMLWithImages=HTMLWithImages.HTMLWithImages() DocBookBook=DocBookClass.DocBookBook() DocBookChapter=DocBookClass.DocBookChapter() DocBookChapterWithFigures=DocBookClass.DocBookChapterWithFigures() DocBookArticle=DocBookClass.DocBookArticle() Zope-2.8.7-final/lib/python/StructuredText/DocBookClass.py0000664000076400007640000002217010435500325022517 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## class DocBookClass: element_types={ '#text': '_text', 'StructuredTextDocument': 'document', 'StructuredTextParagraph': 'paragraph', 'StructuredTextExample': 'example', 'StructuredTextBullet': 'bullet', 'StructuredTextNumbered': 'numbered', 'StructuredTextDescription': 'description', 'StructuredTextDescriptionTitle': 'descriptionTitle', 'StructuredTextDescriptionBody': 'descriptionBody', 'StructuredTextSection': 'section', 'StructuredTextSectionTitle': 'sectionTitle', 'StructuredTextLiteral': 'literal', 'StructuredTextEmphasis': 'emphasis', 'StructuredTextStrong': 'strong', 'StructuredTextLink': 'link', 'StructuredTextXref': 'xref', 'StructuredTextSGML': 'sgml', } def dispatch(self, doc, level, output): getattr(self, self.element_types[doc.getNodeName()])(doc, level, output) def __call__(self, doc, level=1): r=[] self.dispatch(doc, level-1, r.append) return ''.join(r) def _text(self, doc, level, output): if doc.getNodeName() == 'StructuredTextLiteral': output(doc.getNodeValue()) else: output(doc.getNodeValue().lstrip()) def document(self, doc, level, output): output('\n') output('\n') children=doc.getChildNodes() if (children and children[0].getNodeName() == 'StructuredTextSection'): output('%s' % children[0].getChildNodes()[0].getNodeValue()) for c in children: getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('\n') def section(self, doc, level, output): output('\n
\n') children=doc.getChildNodes() for c in children: getattr(self, self.element_types[c.getNodeName()])(c, level+1, output) output('\n
\n') def sectionTitle(self, doc, level, output): output('') for c in doc.getChildNodes(): try: getattr(self, self.element_types[c.getNodeName()])(c, level, output) except: print "failed", c.getNodeName(), c output('\n') def description(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) n=doc.getNextSibling() if n is None or n.getNodeName() is not doc.getNodeName(): output('\n') def descriptionTitle(self, doc, level, output): output('\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('\n') def descriptionBody(self, doc, level, output): output('\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('\n') output('\n') def bullet(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('\n') output('\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) n=doc.getNextSibling() output('\n') if n is None or n.getNodeName() is not doc.getNodeName(): output('\n') def numbered(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('\n') output('\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) n=doc.getNextSibling() output('\n') if n is None or n.getNodeName() is not doc.getNodeName(): output('\n') def example(self, doc, level, output): i=0 for c in doc.getChildNodes(): if i==0: output('\n' in your body will break this... ## output(prestrip(c.getNodeValue())) output('\n]]>\n') else: getattr(self, self.element_types[c.getNodeName()])( c, level, output) def paragraph(self, doc, level, output): output('\n\n') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])( c, level, output) output('\n\n') def link(self, doc, level, output): output('' % doc.href) for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('') def emphasis(self, doc, level, output): output('') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) output(' ') def literal(self, doc, level, output): output('') for c in doc.getChildNodes(): output(c.getNodeValue()) output('') def strong(self, doc, level, output): output('') for c in doc.getChildNodes(): getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('') def xref(self, doc, level, output): output('' % doc.getNodeValue()) def sgml(self, doc, level, output): output(doc.getNodeValue()) def prestrip(v): v=v.replace( '\r\n', '\n') v=v.replace( '\r', '\n') v=v.replace( '\t', ' ') lines=v.split('\n') indent=len(lines[0]) for line in lines: if not len(line): continue i=len(line)-len(line.lstrip()) if i < indent: indent=i nlines=[] for line in lines: nlines.append(line[indent:]) return '\n'.join(nlines, '\n') class DocBookChapter(DocBookClass): def document(self, doc, level, output): output('\n') children=doc.getChildNodes() if (children and children[0].getNodeName() == 'StructuredTextSection'): output('%s' % children[0].getChildNodes()[0].getNodeValue()) for c in children[0].getChildNodes()[1:]: getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('\n') ets = DocBookClass.element_types ets.update({'StructuredTextImage': 'image'}) class DocBookChapterWithFigures(DocBookChapter): element_types = ets def image(self, doc, level, output): if hasattr(doc, 'key'): output('
%s\n' % (doc.key, doc.getNodeValue()) ) else: output('
%s\n' % doc.getNodeValue()) ## for c in doc.getChildNodes(): ## getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('\n
\n' % doc.href) class DocBookArticle(DocBookClass): def document(self, doc, level, output): output('\n') output('
\n') children=doc.getChildNodes() if (children and children[0].getNodeName() == 'StructuredTextSection'): output('\n%s\n\n' % children[0].getChildNodes()[0].getNodeValue()) for c in children: getattr(self, self.element_types[c.getNodeName()])(c, level, output) output('
\n') class DocBookBook: def __init__(self, title=''): self.title = title self.chapters = [] def addChapter(self, chapter): self.chapters.append(chapter) def read(self): out = '\n\n' out = out + '%s\n' % self.title for chapter in self.chapters: out = out + chapter + '\n\n' return out def __str__(self): return self.read() Zope-2.8.7-final/lib/python/StructuredText/Zwiki.py0000664000076400007640000000533210435500325021307 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from Html import HTML from ST import DOC import re """ This is the new structured text type. """ class Zwiki_Title: def __init__(self,str=''): self.expr1 = re.compile('([A-Z]+[A-Z]+[a-zA-Z]*)').search self.expr2 = re.compile('([A-Z]+[a-z]+[A-Z]+[a-zA-Z]*)').search self.str = [str] self.typ = "Zwiki_Title" def type(self): return '%s' % self.typ def string(self): return self.str def __getitem__(self,index): return self.str[index] def __call__(self,raw_string,subs): """ The raw_string is checked to see if it matches the rules for this structured text expression. If the raw_string does, it is parsed for the sub-string which matches and a doc_inner_link instance is returned whose string is the matching substring. If raw_string does not match, nothing is returned. """ if self.expr1(raw_string): start,end = self.expr1(raw_string).span() result = Zwiki_Title(raw_string[start:end]) result.start,result.end = self.expr1(raw_string).span() return result elif self.expr2(raw_string): start,end = self.expr2(raw_string).span() result = Zwiki_Title(raw_string[start:end]) result.start,result.end = self.expr2(raw_string).span() return result else: return None def span(self): return self.start,self.end class Zwiki_doc(DOC): def __init__(self): DOC.__init__(self) """ Add the new type to self.types """ self.types.append(Zwiki_Title()) class Zwiki_parser(HTML): def __init__(self): HTML.__init__(self) self.types["Zwiki_Title"] = self.zwiki_title def zwiki_title(self,object): result = "" for x in object.string(): result = result + x result = "%s" % (result,result) #result = "" % result self.string = self.string + result Zope-2.8.7-final/lib/python/StructuredText/STletters.py0000664000076400007640000000062310435500325022141 0ustar ajungajungimport string def punc_func(exclude): punc = r'' for char in string.punctuation: if char not in exclude: punc = punc + r'\%s' % char return punc digits = string.digits letters = string.letters literal_punc = punc_func("'") dbl_quoted_punc = punc_func("\"") strongem_punc = punc_func('*') under_punc = punc_func('_<>') phrase_delimiters = r'\s\.\,\?\/\!\&\(\)' Zope-2.8.7-final/lib/python/StructuredText/STNG.txt0000664000076400007640000001252110435500325021152 0ustar ajungajungUsing Structured Text The goal of StructuredText is to make it possible to express structured text using a relatively simple plain text format. Simple structures, like bullets or headings are indicated through conventions that are natural, for some definition of "natural". Hierarchical structures are indicated through indentation. The use of indentation to express hierarchical structure is inspired by the Python programming language. Use of StructuredText consists of one to three logical steps. In the first step, a text string is converted to a network of objects using the 'StructuredText.Basic' facility, as in the following example:: raw=open("mydocument.txt").read() import StructuredText st=StructuredText.Basic(raw) The output of 'StructuredText.Basic' is simply a StructuredTextDocument object containing StructuredTextParagraph objects arranged in a hierarchy. Paragraphs are delimited by strings of two or more whitespace characters beginning and ending with newline characters. Hierarchy is indicated by indentation. The indentation of a paragraph is the minimum number of leading spaces in a line containing non-white-space characters after converting tab characters to spaces (assuming a tab stop every eight characters). StructuredTextNode objects support the read-only subset of the Document Object Model (DOM) API. It should be possible to process 'StructuredTextNode' hierarchies using XML tools such as XSLT. The second step in using StructuredText is to apply additional structuring rules based on text content. A variety of differentText rules can be used. Typically, these are used to implement a structured text language for producing documents, but any sort of structured text language could be implemented in the second step. For example, it is possible to use StructuredText to implement structured text formats for representing structured data. The second step, which could consist of multiple processing steps, is performed by processing, or "coloring", the hierarchy of generic StructuredTextParagraph objects into a network of more specialized objects. Typically, the objects produced should also implement the DOM API to allow processing with XML tools. A document processor is provided to convert a StructuredTextDocument object containing only StructuredStructuredTextParagraph objects into a StructuredTextDocument object containing a richer collection of objects such as bullets, headings, emphasis, and so on using hints in the text. Hints are selected based on conventions of the sort typically seen in electronic mail or news-group postings. It should be noted, however, that these conventions are somewhat culturally dependent, fortunately, the document processor is easily customized to implement alternative rules. Here's an example of using the DOC processor to convert the output of the previous example:: doc=StructuredText.Document(st) The final step is to process the colored networks produced from the second step to produce additional outputs. The final step could be performed by Python programs, or by XML tools. A Python outputter is provided for the document processor output that produces Hypertext Markup Language (HTML) text:: html=StructuredText.HTML(doc) Customizing the document processor The document processor is driven by two tables. The first table, named 'paragraph_types', is a sequence of callable objects or method names for coloring paragraphs. If a table entry is a string, then it is the name of a method of the document processor to be used. For each input paragraph, the objects in the table are called until one returns a value (not 'None'). The value returned replaces the original input paragraph in the output. If none of the objects in the paragraph types table return a value, then a copy of the original paragraph is used. The new object returned by calling a paragraph type should implement the ReadOnlyDOM, StructuredTextColorizable, and StructuredTextSubparagraphContainer interfaces. See the 'Document.py' source file for examples. A paragraph type may return a list or tuple of replacement paragraphs, this allowing a paragraph to be split into multiple paragraphs. The second table, 'text_types', is a sequence of callable objects or method names for coloring text. The callable objects in this table are used in sequence to transform the input text into new text or objects. The callable objects are passed a string and return nothing ('None') or a three-element tuple consisting of: - a replacement object, - a starting position, and - an ending position The text from the starting position is (logically) replaced with the replacement object. The replacement object is typically an object that implements that implements the ReadOnlyDOM, and StructuredTextColorizable interfaces. The replacement object can also be a string or a list of strings or objects. Replacement is done from beginning to end and text after the replacement ending position will be passed to the character type objects for processing. Example: adding wiki links We want to add support for Wiki links. A Wiki link is a string of text containing mixed-case letters, such that at least two of the letters are upper case and such that the first letter is upper case. Zope-2.8.7-final/lib/python/StructuredText/ts_regex.py0000664000076400007640000000743210435500325022035 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Provide a thread-safe interface to regex """ import regex, regsub #, Sync from regex import * from regsub import split, sub, gsub, splitx, capwords try: import thread except: class allocate_lock: def acquire(*args): pass def release(*args): pass else: class SafeFunction: _l=thread.allocate_lock() _a=_l.acquire _r=_l.release def __init__(self, f): self._f=f def __call__(self, *args, **kw): self._a() try: return self._f(*args, **kw) finally: self._r() split=SafeFunction(split) sub=SafeFunction(sub) gsub=SafeFunction(gsub) splitx=SafeFunction(splitx) capwords=SafeFunction(capwords) allocate_lock=thread.allocate_lock class compile: _r=None groupindex=None def __init__(self, *args): self._r=r=regex(*compile, **args) self._init(r) def _init(self, r): lock=allocate_lock() self.__a=lock.acquire self.__r=lock.release self.translate=r.translate self.givenpat=r.givenpat self.realpat=r.realpat def match(self, string, pos=0): self.__a() try: return self._r.match(string, pos) finally: self.__r() def search(self, string, pos=0): self.__a() try: return self._r.search(string, pos) finally: self.__r() def search_group(self, str, group, pos=0): """Search a string for a pattern. If the pattern was not found, then None is returned, otherwise, the location where the pattern was found, as well as any specified group are returned. """ self.__a() try: r=self._r l=r.search(str, pos) if l < 0: return None return l, r.group(*group) finally: self.__r() def match_group(self, str, group, pos=0): """Match a pattern against a string If the string does not match the pattern, then None is returned, otherwise, the length of the match, as well as any specified group are returned. """ self.__a() try: r=self._r l=r.match(str, pos) if l < 0: return None return l, r.group(*group) finally: self.__r() def search_regs(self, str, pos=0): """Search a string for a pattern. If the pattern was not found, then None is returned, otherwise, the 'regs' attribute of the expression is returned. """ self.__a() try: r=self._r r.search(str, pos) return r.regs finally: self.__r() def match_regs(self, str, pos=0): """Match a pattern against a string If the string does not match the pattern, then None is returned, otherwise, the 'regs' attribute of the expression is returned. """ self.__a() try: r=self._r r.match(str, pos) return r.regs finally: self.__r() class symcomp(compile): def __init__(self, *args): self._r=r=regex.symcomp(*args) self._init(r) self.groupindex=r.groupindex Zope-2.8.7-final/lib/python/StructuredText/DocumentWithImages.py0000664000076400007640000000246610435500325023757 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import re from DocumentClass import * class StructuredTextImage(StructuredTextMarkup): "A simple embedded image" class DocumentWithImages(DocumentClass): """ Document with images """ text_types = [ 'doc_img', ] + DocumentClass.text_types def doc_img( self, s, expr1=re.compile('\"([ _a-zA-Z0-9*.:/;,\[\]\'\-\n\~]+)\":img:([a-zA-Z0-9%\_\-.:/\?=;,\n\~]+)').search, ): r=expr1(s) if r: startt, endt = r.span(1) starth, endh = r.span(2) start, end = r.span() return (StructuredTextImage(s[startt:endt], href=s[starth:endh]), start, end) return None Zope-2.8.7-final/lib/python/StructuredText/tests/0000775000076400007640000000000010435610641021002 5ustar ajungajungZope-2.8.7-final/lib/python/StructuredText/tests/__init__.py0000664000076400007640000000125410435500325023112 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # This file is needed to make this a package. Zope-2.8.7-final/lib/python/StructuredText/tests/testStructuredText.py0000664000076400007640000002254610435500325025273 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from StructuredText import ST from StructuredText import DocumentClass from StructuredText import ClassicDocumentClass from StructuredText import StructuredText from StructuredText import HTMLClass from StructuredText.StructuredText import HTML import sys, os, unittest, cStringIO from types import UnicodeType from OFS import ndiff """ This tester first ensures that all regression files can at least parse all files. Secondly, the tester will compare the output between StructuredText and ClassicDocumentClass->HTMLClass to help ensure backwards compatability. """ package_dir = os.path.split(ST.__file__)[0] regressions=os.path.join(package_dir, 'regressions') files = ['index.stx','Acquisition.stx','ExtensionClass.stx', 'MultiMapping.stx','examples.stx','Links.stx','examples1.stx', 'table.stx','InnerLinks.stx'] def readFile(dirname,fname): myfile = open(os.path.join(dirname, fname),"r") lines = myfile.readlines() myfile.close() return ''.join(lines) class StructuredTextTests(unittest.TestCase): def testStructuredText(self): """ testing StructuredText """ for f in files: raw_text = readFile(regressions,f) assert StructuredText.StructuredText(raw_text),\ 'StructuredText failed on %s' % f assert StructuredText.StructuredText(unicode(raw_text)),\ 'StructuredText failed on Unicode %s' % f def testStructuredTextNG(self): """ testing StructuredTextNG """ for f in files: raw_text = readFile(regressions,f) assert ST.StructuredText(raw_text),\ 'StructuredText failed on %s' % f assert ST.StructuredText(unicode(raw_text)),\ 'StructuredText failed on Unicode %s' % f def testDocumentClass(self): """ testing DocumentClass""" for f in files: Doc = DocumentClass.DocumentClass() raw_text = readFile(regressions,f) text = ST.StructuredText(raw_text) assert Doc(text),\ 'DocumentClass failed on %s' % f def testClassicDocumentClass(self): """ testing ClassicDocumentClass""" for f in files: Doc = ClassicDocumentClass.DocumentClass() raw_text = readFile(regressions,f) text = ST.StructuredText(raw_text) assert Doc(text),\ 'ClassicDocumentClass failed on %s' % f def testClassicHTMLDocumentClass(self): """ testing HTML ClassicDocumentClass""" for f in files: Doc = ClassicDocumentClass.DocumentClass() HTML = HTMLClass.HTMLClass() raw_text = readFile(regressions,f) text = Doc(ST.StructuredText(raw_text)) assert HTML(text),\ 'HTML ClassicDocumentClass failed on %s' % f def testRegressionsTests(self): """ HTML regression test """ for f in files: Doc = DocumentClass.DocumentClass() HTML = HTMLClass.HTMLClass() raw_text = readFile(regressions,f) text = Doc(ST.StructuredText(raw_text)) html = HTML(text) reg_fname = f.replace('.stx','.ref') reg_html = readFile(regressions , reg_fname) if reg_html.strip()!= html.strip(): IO = cStringIO.StringIO() oldStdout = sys.stdout sys.stdout = IO try: open('_tmpout','w').write(html) ndiff.fcompare(os.path.join(regressions,reg_fname), '_tmpout') os.unlink('_tmpout') finally: sys.stdout = oldStdout raise AssertionError, \ 'HTML regression test failed on %s\nDiff:\n%s\n' % (f, IO.getvalue()) class BasicTests(unittest.TestCase): def _test(self,stxtxt , expected): if not isinstance(stxtxt, UnicodeType): res = HTML(stxtxt,level=1,header=0) if not expected in res: print "Text: ",stxtxt print "Converted:",res print "Expected: ",expected raise AssertionError,"basic test failed for '%s'" % stxtxt if isinstance(stxtxt, UnicodeType): ustxtxt = stxtxt else: ustxtxt = unicode(stxtxt) res = HTML(ustxtxt,level=1,header=0) if not expected in res: print "Text: ",stxtxt.encode('latin-1') print "Converted:",res.encode('latin-1') print "Expected: ",expected.encode('latin-1') raise AssertionError, ("basic test failed for Unicode '%s'" % stxtxt) def testUnderline(self): self._test("xx _this is html_ xx", "xx this is html xx") def testUnderline1(self): self._test("xx _this is html_", "this is html") def testEmphasis(self): self._test("xx *this is html* xx", "xx this is html xx") def testStrong(self): self._test("xx **this is html** xx", "xx this is html xx") def testUnderlineThroughoutTags(self): self._test('index_html', 'index_html') def testUnderscoresInLiteral1(self): self._test("def __init__(self)", "def __init__(self)") def testUnderscoresInLiteral2(self): self._test("this is '__a_literal__' eh", "__a_literal__") def testUnderlinesWithoutWithspaces(self): self._test("Zopes structured_text is sometimes a night_mare", "Zopes structured_text is sometimes a night_mare") def testAsterisksInLiteral(self): self._test("this is a '*literal*' eh", "*literal*") def testDoubleAsterisksInLiteral(self): self._test("this is a '**literal**' eh", "**literal**") def testLinkInLiteral(self): self._test("this is a '\"literal\":http://www.zope.org/.' eh", '"literal":http://www.zope.org/.') def testLink(self): self._test('"foo":http://www.zope.org/foo/bar', '

foo

') self._test('"foo":http://www.zope.org/foo/bar/%20x', '

foo

') self._test('"foo":http://www.zope.org/foo/bar?arg1=1&arg2=2', '

foo

') self._test('"foo bar":http://www.zope.org/foo/bar', '

foo bar

') self._test('"[link goes here]":http://www.zope.org/foo/bar', '

[link goes here]

') self._test('"[Dad\'s car]":http://www.zope.org/foo/bar', '

[Dad\'s car]

') def testImgLink(self): self._test('"foo":img:http://www.zope.org/bar.gif', 'foo') self._test('"foo":img:http://www.zope.org:8080/bar.gif', 'foo') self._test('"foo":img:http://www.zope.org:8080/foo/bar?arg=1', 'foo') self._test('"foo":img:http://www.zope.org:8080/foo/b%20ar?arg=1', 'foo') self._test('"foo bar":img:http://www.zope.org:8080/foo/bar', 'foo bar') self._test('"[link goes here]":img:http://www.zope.org:8080/foo/bar', '[link goes here]') self._test('"[Dad\'s new car]":img:http://www.zope.org:8080/foo/bar', '[Dad\'s new car]') def XXXtestUnicodeContent(self): # This fails because ST uses the default locale to get "letters" # whereas it should use \w+ and re.U if the string is Unicode. #self._test(u"h\xe9 **y\xe9** xx", # u"h\xe9 y\xe9 xx") pass def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( StructuredTextTests ) ) suite.addTest( unittest.makeSuite( BasicTests ) ) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/StructuredText/DocumentClass.py0000664000076400007640000007701610435500325022766 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import re, ST, STDOM from STletters import letters, digits, literal_punc, under_punc,\ strongem_punc, phrase_delimiters,dbl_quoted_punc from types import StringType, UnicodeType, ListType StringTypes = (StringType, UnicodeType) def flatten(obj, append): if obj.getNodeType()==STDOM.TEXT_NODE: append(obj.getNodeValue()) else: for child in obj.getChildNodes(): flatten(child, append) class StructuredTextExample(ST.StructuredTextParagraph): """Represents a section of document with literal text, as for examples""" def __init__(self, subs, **kw): t=[] a=t.append for s in subs: flatten(s, a) ST.StructuredTextParagraph.__init__(self, '\n\n'.join(t), (), **kw) def getColorizableTexts(self): return () def setColorizableTexts(self, src): pass # never color examples class StructuredTextBullet(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextNumbered(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescriptionTitle(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescriptionBody(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescription(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" def __init__(self, title, src, subs, **kw): ST.StructuredTextParagraph.__init__(self, src, subs, **kw) self._title=title def getColorizableTexts(self): return self._title, self._src def setColorizableTexts(self, src): self._title, self._src = src def getChildren(self): return (StructuredTextDescriptionTitle(self._title), StructuredTextDescriptionBody(self._src, self._subs)) class StructuredTextSectionTitle(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextSection(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" def __init__(self, src, subs=None, **kw): ST.StructuredTextParagraph.__init__( self, StructuredTextSectionTitle(src), subs, **kw) def getColorizableTexts(self): return self._src.getColorizableTexts() def setColorizableTexts(self,src): self._src.setColorizableTexts(src) # a StructuredTextTable holds StructuredTextRows class StructuredTextTable(ST.StructuredTextParagraph): """ rows is a list of lists containing tuples, which represent the columns/cells in each rows. EX rows = [[('row 1:column1',1)],[('row2:column1',1)]] """ def __init__(self, rows, src, subs, **kw): ST.StructuredTextParagraph.__init__(self, subs, **kw) self._rows = [] for row in rows: if row: self._rows.append(StructuredTextRow(row,kw)) def getRows(self): return [self._rows] def _getRows(self): return self.getRows() def getColumns(self): result = [] for row in self._rows: result.append(row.getColumns()) return result def _getColumns(self): return self.getColumns() def setColumns(self,columns): for index in range(len(self._rows)): self._rows[index].setColumns(columns[index]) def _setColumns(self,columns): return self.setColumns(columns) def getColorizableTexts(self): """ return a tuple where each item is a column/cell's contents. The tuple, result, will be of this format. ("r1 col1", "r1=col2", "r2 col1", "r2 col2") """ result = [] for row in self._rows: for column in row.getColumns()[0]: result.append(column.getColorizableTexts()[0]) return result def setColorizableTexts(self,texts): """ texts is going to a tuple where each item is the result of being mapped to the colortext function. Need to insert the results appropriately into the individual columns/cells """ for row_index in range(len(self._rows)): for column_index in range(len(self._rows[row_index]._columns)): self._rows[row_index]._columns[column_index].setColorizableTexts((texts[0],)) texts = texts[1:] def _getColorizableTexts(self): return self.getColorizableTexts() def _setColorizableTexts(self, texts): return self.setColorizableTexts(texts) # StructuredTextRow holds StructuredTextColumns class StructuredTextRow(ST.StructuredTextParagraph): def __init__(self,row,kw): """ row is a list of tuples, where each tuple is the raw text for a cell/column and the span of that cell/column. EX [('this is column one',1), ('this is column two',1)] """ ST.StructuredTextParagraph.__init__(self, [], **kw) self._columns = [] for column in row: self._columns.append(StructuredTextColumn(column[0], column[1], column[2], column[3], column[4], kw)) def getColumns(self): return [self._columns] def _getColumns(self): return [self._columns] def setColumns(self,columns): self._columns = columns def _setColumns(self,columns): return self.setColumns(columns) # this holds the text of a table cell class StructuredTextColumn(ST.StructuredTextParagraph): """ StructuredTextColumn is a cell/column in a table. A cell can hold multiple paragraphs. The cell is either classified as a StructuredTextTableHeader or StructuredTextTableData. """ def __init__(self,text,span,align,valign,typ,kw): ST.StructuredTextParagraph.__init__(self, text, [], **kw) self._span = span self._align = align self._valign = valign self._type = typ def getSpan(self): return self._span def _getSpan(self): return self._span def getAlign(self): return self._align def _getAlign(self): return self.getAlign() def getValign(self): return self._valign def _getValign(self): return self.getValign() def getType(self): return self._type def _getType(self): return self.getType() class StructuredTextTableHeader(ST.StructuredTextParagraph): pass class StructuredTextTableData(ST.StructuredTextParagraph): pass class StructuredTextMarkup(STDOM.Element): def __init__(self, v, **kw): self._value=v self._attributes=kw.keys() for k, v in kw.items(): setattr(self, k, v) def getChildren(self): v=self._value if not isinstance(v, ListType): v=[v] return v def getColorizableTexts(self): return self._value, def setColorizableTexts(self, v): self._value=v[0] def __repr__(self): return '%s(%s)' % (self.__class__.__name__, `self._value`) class StructuredTextLiteral(StructuredTextMarkup): def getColorizableTexts(self): return () def setColorizableTexts(self, v): pass class StructuredTextEmphasis(StructuredTextMarkup): pass class StructuredTextStrong(StructuredTextMarkup): pass class StructuredTextInnerLink(StructuredTextMarkup): pass class StructuredTextNamedLink(StructuredTextMarkup): pass class StructuredTextUnderline(StructuredTextMarkup): pass class StructuredTextSGML(StructuredTextMarkup): pass class StructuredTextLink(StructuredTextMarkup): pass class StructuredTextXref(StructuredTextMarkup): pass class DocumentClass: """ Class instance calls [ex.=> x()] require a structured text structure. Doc will then parse each paragraph in the structure and will find the special structures within each paragraph. Each special structure will be stored as an instance. Special structures within another special structure are stored within the 'top' structure EX : '-underline this-' => would be turned into an underline instance. '-underline **this**' would be stored as an underline instance with a strong instance stored in its string """ paragraph_types = [ 'doc_bullet', 'doc_numbered', 'doc_description', 'doc_header', 'doc_table', ] #'doc_inner_link', #'doc_named_link', #'doc_underline' text_types = [ 'doc_literal', 'doc_sgml', 'doc_inner_link', 'doc_named_link', 'doc_href1', 'doc_href2', 'doc_strong', 'doc_emphasize', 'doc_underline', 'doc_sgml', 'doc_xref', ] def __call__(self, doc): if type(doc) in StringTypes: doc=ST.StructuredText(doc) doc.setSubparagraphs(self.color_paragraphs( doc.getSubparagraphs())) else: doc=ST.StructuredTextDocument(self.color_paragraphs( doc.getSubparagraphs())) return doc def parse(self, raw_string, text_type, type=type, sts=StringTypes, lt=type([])): """ Parse accepts a raw_string, an expr to test the raw_string, and the raw_string's subparagraphs. Parse will continue to search through raw_string until all instances of expr in raw_string are found. If no instances of expr are found, raw_string is returned. Otherwise a list of substrings and instances is returned """ tmp = [] # the list to be returned if raw_string is split append=tmp.append if type(text_type) in sts: text_type=getattr(self, text_type) while 1: t = text_type(raw_string) if not t: break #an instance of expr was found t, start, end = t if start: append(raw_string[0:start]) tt=type(t) if tt in sts: # if we get a string back, add it to text to be parsed raw_string = t+raw_string[end:len(raw_string)] else: if tt is lt: # is we get a list, append it's elements tmp[len(tmp):]=t else: # normal case, an object append(t) raw_string = raw_string[end:len(raw_string)] if not tmp: return raw_string # nothing found if raw_string: append(raw_string) elif len(tmp)==1: return tmp[0] return tmp def color_text(self, str, types=None): """Search the paragraph for each special structure """ if types is None: types=self.text_types for text_type in types: if type(str) in StringTypes: str = self.parse(str, text_type) elif type(str) is ListType: r=[]; a=r.append for s in str: if type(s) in StringTypes: s=self.parse(s, text_type) if type(s) is ListType: r[len(r):]=s else: a(s) else: s.setColorizableTexts( map(self.color_text, s.getColorizableTexts() )) a(s) str=r else: r=[]; a=r.append; color=self.color_text for s in str.getColorizableTexts(): color(s, (text_type,)) a(s) str.setColorizableTexts(r) return str def color_paragraphs(self, raw_paragraphs, type=type, sequence_types=(type([]), type(())), sts=StringTypes): result=[] for paragraph in raw_paragraphs: if paragraph.getNodeName() != 'StructuredTextParagraph': result.append(paragraph) continue for pt in self.paragraph_types: if type(pt) in sts: # grab the corresponding function pt=getattr(self, pt) # evaluate the paragraph r=pt(paragraph) if r: if type(r) not in sequence_types: r=r, new_paragraphs=r for paragraph in new_paragraphs: subs = self.color_paragraphs(paragraph.getSubparagraphs()) paragraph.setSubparagraphs(subs) break else: # copy, retain attributes kw = {} atts = getattr(paragraph, '_attributes', []) for att in atts: kw[att] = getattr(paragraph, att) subs = self.color_paragraphs(paragraph.getSubparagraphs()) new_paragraphs=ST.StructuredTextParagraph( paragraph. getColorizableTexts()[0], subs, **kw), # color the inline StructuredText types # for each StructuredTextParagraph for paragraph in new_paragraphs: if paragraph.getNodeName() is "StructuredTextTable": # cells = paragraph.getColumns() text = paragraph.getColorizableTexts() text = map(ST.StructuredText,text) text = map(self.__call__,text) for t in range(len(text)): text[t] = text[t].getSubparagraphs() paragraph.setColorizableTexts(text) paragraph.setColorizableTexts( map(self.color_text, paragraph.getColorizableTexts() )) result.append(paragraph) return result def doc_table(self, paragraph, expr = re.compile(r'\s*\|[-]+\|').match): text = paragraph.getColorizableTexts()[0] m = expr(text) subs = paragraph.getSubparagraphs() if not (m): return None rows = [] spans = [] ROWS = [] COLS = [] indexes = [] ignore = [] TDdivider = re.compile("[\-]+").match THdivider = re.compile("[\=]+").match col = re.compile('\|').search innertable = re.compile('\|([-]+|[=]+)\|').search text = text.strip() rows = text.split('\n') foo = "" for row in range(len(rows)): rows[row] = rows[row].strip() # have indexes store if a row is a divider # or a cell part for index in range(len(rows)): tmpstr = rows[index][1:len(rows[index])-1] if TDdivider(tmpstr): indexes.append("TDdivider") elif THdivider(tmpstr): indexes.append("THdivider") else: indexes.append("cell") for index in range(len(indexes)): if indexes[index] is "TDdivider" or indexes[index] is "THdivider": ignore = [] # reset ignore #continue # skip dividers tmp = rows[index].strip() # clean the row up tmp = tmp[1:len(tmp)-1] # remove leading + trailing | offset = 0 # find the start and end of inner # tables. ignore everything between if innertable(tmp): tmpstr = tmp.strip() while innertable(tmpstr): start,end = innertable(tmpstr).span() if not (start,end-1) in ignore: ignore.append((start,end-1)) tmpstr = " " + tmpstr[end:] # find the location of column dividers # NOTE: |'s in inner tables do not count # as column dividers if col(tmp): while col(tmp): bar = 1 # true if start is not in ignore start,end = col(tmp).span() if not start+offset in spans: for s,e in ignore: if start+offset >= s or start+offset <= e: bar = None break if bar: # start is clean spans.append(start+offset) if not bar: foo = foo + tmp[:end] tmp = tmp[end:] offset = offset + end else: COLS.append((foo + tmp[0:start],start+offset)) foo = "" tmp = " " + tmp[end:] offset = offset + start if not offset+len(tmp) in spans: spans.append(offset+len(tmp)) COLS.append((foo + tmp,offset+len(tmp))) foo = "" ROWS.append(COLS) COLS = [] spans.sort() ROWS = ROWS[1:len(ROWS)] # find each column span cols = [] tmp = [] for row in ROWS: for c in row: tmp.append(c[1]) cols.append(tmp) tmp = [] cur = 1 tmp = [] C = [] for col in cols: for span in spans: if not span in col: cur = cur + 1 else: tmp.append(cur) cur = 1 C.append(tmp) tmp = [] for index in range(len(C)): for i in range(len(C[index])): ROWS[index][i] = (ROWS[index][i][0],C[index][i]) rows = ROWS # label things as either TableData or # Table header TD = [] TH = [] all = [] for index in range(len(indexes)): if indexes[index] is "TDdivider": TD.append(index) all.append(index) if indexes[index] is "THdivider": TH.append(index) all.append(index) TD = TD[1:] dividers = all[1:] #print "TD => ", TD #print "TH => ", TH #print "all => ", all, "\n" for div in dividers: if div in TD: index = all.index(div) for rowindex in range(all[index-1],all[index]): for i in range(len(rows[rowindex])): rows[rowindex][i] = (rows[rowindex][i][0], rows[rowindex][i][1], "td") else: index = all.index(div) for rowindex in range(all[index-1],all[index]): for i in range(len(rows[rowindex])): rows[rowindex][i] = (rows[rowindex][i][0], rows[rowindex][i][1], "th") # now munge the multi-line cells together # as paragraphs ROWS = [] COLS = [] for row in rows: for index in range(len(row)): if not COLS: COLS = range(len(row)) for i in range(len(COLS)): COLS[i] = ["",1,""] if TDdivider(row[index][0]) or THdivider(row[index][0]): ROWS.append(COLS) COLS = [] else: COLS[index][0] = COLS[index][0] + (row[index][0]) + "\n" COLS[index][1] = row[index][1] COLS[index][2] = row[index][2] # now that each cell has been munged together, # determine the cell's alignment. # Default is to center. Also determine the cell's # vertical alignment, top, middle, bottom. Default is # to middle rows = [] cols = [] for row in ROWS: for index in range(len(row)): topindent = 0 bottomindent = 0 leftindent = 0 rightindent = 0 left = [] right = [] text = row[index][0] text = text.split('\n') text = text[:len(text)-1] align = "" valign = "" for t in text: t = t.strip() if not t: topindent = topindent + 1 else: break text.reverse() for t in text: t = t.strip() if not t: bottomindent = bottomindent + 1 else: break text.reverse() tmp = '\n'.join(text[topindent:len(text)-bottomindent]) pars = re.compile("\n\s*\n").split(tmp) for par in pars: if index > 0: par = par[1:] par = par.split(' ') for p in par: if not p: leftindent = leftindent+1 else: break left.append(leftindent) leftindent = 0 par.reverse() for p in par: if not p: rightindent = rightindent + 1 else: break right.append(rightindent) rightindent = 0 left.sort() right.sort() if topindent == bottomindent: valign="middle" elif topindent < 1: valign="top" elif bottomindent < 1: valign="bottom" else: valign="middle" if left[0] < 1: align = "left" elif right[0] < 1: align = "right" elif left[0] > 1 and right[0] > 1: align="center" else: align="left" cols.append((row[index][0],row[index][1],align,valign,row[index][2])) rows.append(cols) cols = [] return StructuredTextTable(rows,text,subs,indent=paragraph.indent) def doc_bullet(self, paragraph, expr = re.compile(r'\s*[-*o]\s+').match): top=paragraph.getColorizableTexts()[0] m=expr(top) if not m: return None subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextBullet(top[m.span()[1]:], subs, indent=paragraph.indent, bullet=top[:m.span()[1]] ) def doc_numbered( self, paragraph, expr = re.compile(r'(\s*[%s]\.)|(\s*[0-9]+\.)|(\s*[0-9]+\s+)' % letters).match): # This is the old expression. It had a nasty habit # of grabbing paragraphs that began with a single # letter word even if there was no following period. #expr = re.compile('\s*' # '(([a-zA-Z]|[0-9]+|[ivxlcdmIVXLCDM]+)\.)*' # '([a-zA-Z]|[0-9]+|[ivxlcdmIVXLCDM]+)\.?' # '\s+').match): top=paragraph.getColorizableTexts()[0] m=expr(top) if not m: return None subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextNumbered(top[m.span()[1]:], subs, indent=paragraph.indent, number=top[:m.span()[1]]) def doc_description( self, paragraph, delim = re.compile(r'\s+--\s+').search, nb=re.compile(r'[^\000- ]').search, ): top=paragraph.getColorizableTexts()[0] d=delim(top) if not d: return None start, end = d.span() title=top[:start] if title.find('\n') >= 0: return None if not nb(title): return None d=top[start:end] top=top[end:] subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextDescription( title, top, subs, indent=paragraph.indent, delim=d) def doc_header(self, paragraph): subs=paragraph.getSubparagraphs() if not subs: return None top=paragraph.getColorizableTexts()[0] if not top.strip(): return None if top[-2:]=='::': subs=StructuredTextExample(subs) if top.strip()=='::': return subs # copy attrs when returning a paragraph kw = {} atts = getattr(paragraph, '_attributes', []) for att in atts: kw[att] = getattr(paragraph, att) return ST.StructuredTextParagraph(top[:-1], [subs], **kw) if top.find('\n') >= 0: return None return StructuredTextSection(top, subs, indent=paragraph.indent) def doc_literal( self, s, expr = re.compile(r"(\W+|^)'([%s%s%s\s]+)'([%s]+|$)" % (letters, digits, literal_punc, phrase_delimiters)).search,): # old expr... failed to cross newlines. # expr=re.compile( # r"(?:\s|^)'" # open # r"([^ \t\n\r\f\v']|[^ \t\n\r\f\v'][^\n\r']*[^ \t\n\r\f\v'])" # contents # r"'(?:\s|[,.;:!?]|$)" # close # ).search): r=expr(s) #or expr2(s) if r: start, end = r.span(2) return (StructuredTextLiteral(s[start:end]), start-1, end+1) else: return None def doc_emphasize( self, s, expr = re.compile(r'\*([%s%s%s\s]+?)\*' % (letters, digits, strongem_punc)).search #expr = re.compile(r'\s*\*([ \n\r%s0-9.:/;,\'\"\?\-\_\/\=\-\>\<\(\)]+)\*(?!\*|-)' % letters).search # old expr, inconsistent punctuation ): r=expr(s) if r: start, end = r.span(1) return (StructuredTextEmphasis(s[start:end]), start-1, end+1) else: return None def doc_inner_link(self, s, expr1 = re.compile(r"\.\.\s*").search, expr2 = re.compile(r"\[[%s%s]+\]" % (letters, digits) ).search): # make sure we dont grab a named link if expr2(s) and expr1(s): start1,end1 = expr1(s).span() start2,end2 = expr2(s).span() if end1 == start2: # uh-oh, looks like a named link return None else: # the .. is somewhere else, ignore it return (StructuredTextInnerLink(s[start2+1:end2-1]),start2,end2) return None elif expr2(s) and not expr1(s): start,end = expr2(s).span() return (StructuredTextInnerLink(s[start+1:end-1]),start,end) return None def doc_named_link(self, s, expr=re.compile(r"(\.\.\s)(\[[%s0-9]+\])" % letters).search): result = expr(s) if result: start,end = result.span(2) str = s[start+1:end-1] st,en = result.span() return (StructuredTextNamedLink(str),st,en) return None def doc_underline(self, s, expr=re.compile(r'_([%s%s%s\s]+)_([\s%s]|$)' % (letters, digits, under_punc,phrase_delimiters)).search): result = expr(s) if result: if result.group(1)[:1] == '_': return None # no double unders start,end = result.span(1) st,e = result.span() return (StructuredTextUnderline(s[start:end]),st,e-len(result.group(2))) else: return None def doc_strong(self, s, expr = re.compile(r'\*\*([%s%s%s\s]+?)\*\*' % (letters, digits, strongem_punc)).search #expr = re.compile(r'\s*\*\*([ \n\r%s0-9.:/;,\'\"\?\-\_\/\=\-\>\<\(\)]+)\*\*(?!\*|-)' % letters).search, # old expr, inconsistent punc, failed to cross newlines. ): r=expr(s) if r: start, end = r.span(1) return (StructuredTextStrong(s[start:end]), start-2, end+2) else: return None ## Some constants to make the doc_href() regex easier to read. _DQUOTEDTEXT = r'("[ %s0-9\n\r%s]+")' % (letters,dbl_quoted_punc) ## double quoted text _ABSOLUTE_URL=r'((http|https|ftp|mailto|file|about)[:/]+?[%s0-9_\@\.\,\?\!\/\:\;\-\#\~\=\&\%%\+]+)' % letters _ABS_AND_RELATIVE_URL=r'([%s0-9_\@\.\,\?\!\/\:\;\-\#\~\=\&\%%\+]+)' % letters _SPACES = r'(\s*)' def doc_href1(self, s, expr=re.compile(_DQUOTEDTEXT + "(:)" + _ABS_AND_RELATIVE_URL + _SPACES).search ): return self.doc_href(s, expr) def doc_href2(self, s, expr=re.compile(_DQUOTEDTEXT + r'(\,\s+)' + _ABSOLUTE_URL + _SPACES).search ): return self.doc_href(s, expr) def doc_href(self, s, expr, punctuation=re.compile(r"[\,\.\?\!\;]+").match): r=expr(s) if r: # need to grab the href part and the # beginning part start,e = r.span(1) name = s[start:e] name = name.replace('"','',2) #start = start + 1 st,end = r.span(3) if punctuation(s[end-1:end]): end = end -1 link = s[st:end] #end = end - 1 # name is the href title, link is the target # of the href return (StructuredTextLink(name, href=link), start, end) #return (StructuredTextLink(s[start:end], href=s[start:end]), # start, end) else: return None def doc_sgml(self,s,expr=re.compile(r"\<[%s0-9\.\=\'\"\:\/\-\#\+\s\*]+\>" % letters).search): """ SGML text is ignored and outputed as-is """ r = expr(s) if r: start,end = r.span() text = s[start:end] return (StructuredTextSGML(text),start,end) else: return None def doc_xref(self, s, expr = re.compile('\[([%s0-9\-.:/;,\n\r\~]+)\]' % letters).search ): r = expr(s) if r: start, end = r.span(1) return (StructuredTextXref(s[start:end]), start-1, end+1) else: return None Zope-2.8.7-final/lib/python/StructuredText/HTMLWithImages.py0000664000076400007640000000217310435500325022740 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from HTMLClass import HTMLClass ets = HTMLClass.element_types ets.update({'StructuredTextImage': 'image'}) class HTMLWithImages(HTMLClass): element_types = ets def image(self, doc, level, output): if hasattr(doc, 'key'): output('\n' % doc.key) output('%s\n' % (doc.href, doc.getNodeValue())) if doc.getNodeValue() and hasattr(doc, 'key'): output('

Figure %s %s

\n' % (doc.key, doc.getNodeValue())) Zope-2.8.7-final/lib/python/StructuredText/ClassicDocumentClass.py0000664000076400007640000005063710435500325024270 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import re, ST, STDOM from STletters import letters from types import StringType, UnicodeType, ListType StringTypes = (StringType, UnicodeType) class StructuredTextExample(ST.StructuredTextParagraph): """Represents a section of document with literal text, as for examples""" def __init__(self, subs, **kw): t=[]; a=t.append for s in subs: a(s.getNodeValue()) ST.StructuredTextParagraph.__init__(self, '\n\n'.join(t), (), **kw) def getColorizableTexts(self): return () def setColorizableTexts(self, src): pass # never color examples class StructuredTextBullet(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextNumbered(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescriptionTitle(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescriptionBody(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextDescription(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" def __init__(self, title, src, subs, **kw): ST.StructuredTextParagraph.__init__(self, src, subs, **kw) self._title=title def getColorizableTexts(self): return self._title, self._src def setColorizableTexts(self, src): self._title, self._src = src def getChildren(self): return (StructuredTextDescriptionTitle(self._title), StructuredTextDescriptionBody(self._src, self._subs)) class StructuredTextSectionTitle(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" class StructuredTextSection(ST.StructuredTextParagraph): """Represents a section of a document with a title and a body""" def __init__(self, src, subs=None, **kw): ST.StructuredTextParagraph.__init__( self, StructuredTextSectionTitle(src), subs, **kw) def getColorizableTexts(self): return self._src.getColorizableTexts() def setColorizableTexts(self,src): self._src.setColorizableTexts(src) # a StructuredTextTable holds StructuredTextRows class StructuredTextTable(ST.StructuredTextDocument): """ rows is a list of lists containing tuples, which represent the columns/cells in each rows. EX rows = [[('row 1:column1',1)],[('row2:column1',1)]] """ def __init__(self, rows, src, subs, **kw): ST.StructuredTextDocument.__init__(self, subs, **kw) self._rows = [] for row in rows: if row: self._rows.append(StructuredTextRow(row,kw)) def getRows(self): return [self._rows] def _getRows(self): return self.getRows() def getColorizableTexts(self): """ return a tuple where each item is a column/cell's contents. The tuple, result, will be of this format. ("r1 col1", "r1=col2", "r2 col1", "r2 col2") """ #result = () result = [] for row in self._rows: for column in row.getColumns()[0]: #result = result[:] + (column.getColorizableTexts(),) result.append(column.getColorizableTexts()[0]) return result def setColorizableTexts(self,texts): """ texts is going to a tuple where each item is the result of being mapped to the colortext function. Need to insert the results appropriately into the individual columns/cells """ for row_index in range(len(self._rows)): for column_index in range(len(self._rows[row_index]._columns)): self._rows[row_index]._columns[column_index].setColorizableTexts((texts[0],)) texts = texts[1:] def _getColorizableTexts(self): return self.getColorizableTexts() def _setColorizableTexts(self): return self.setColorizableTexts() # StructuredTextRow holds StructuredTextColumns class StructuredTextRow(ST.StructuredTextDocument): def __init__(self,row,kw): """ row is a list of tuples, where each tuple is the raw text for a cell/column and the span of that cell/column. EX [('this is column one',1), ('this is column two',1)] """ ST.StructuredTextDocument.__init__(self, [], **kw) self._columns = [] for column in row: self._columns.append(StructuredTextColumn(column[0],column[1],kw)) def getColumns(self): return [self._columns] def _getColumns(self): return [self._columns] # this holds the raw text of a table cell class StructuredTextColumn(ST.StructuredTextParagraph): """ StructuredTextColumn is a cell/column in a table. This contains the actual text of a column and is thus a StructuredTextParagraph. A StructuredTextColumn also holds the span of its column """ def __init__(self,text,span,kw): ST.StructuredTextParagraph.__init__(self, text, [], **kw) self._span = span def getSpan(self): return self._span def _getSpan(self): return self._span class StructuredTextMarkup(STDOM.Element): def __init__(self, v, **kw): self._value=v self._attributes=kw.keys() for k, v in kw.items(): setattr(self, k, v) def getChildren(self, type=type, lt=type([])): v=self._value if type(v) is not lt: v=[v] return v def getColorizableTexts(self): return self._value, def setColorizableTexts(self, v): self._value=v[0] def __repr__(self): return '%s(%s)' % (self.__class__.__name__, `self._value`) class StructuredTextLiteral(StructuredTextMarkup): def getColorizableTexts(self): return () def setColorizableTexts(self, v): pass class StructuredTextEmphasis(StructuredTextMarkup): pass class StructuredTextStrong(StructuredTextMarkup): pass class StructuredTextInnerLink(StructuredTextMarkup): pass class StructuredTextNamedLink(StructuredTextMarkup): pass class StructuredTextUnderline(StructuredTextMarkup): pass class StructuredTextLink(StructuredTextMarkup): "A simple hyperlink" class DocumentClass: """ Class instance calls [ex.=> x()] require a structured text structure. Doc will then parse each paragraph in the structure and will find the special structures within each paragraph. Each special structure will be stored as an instance. Special structures within another special structure are stored within the 'top' structure EX : '-underline this-' => would be turned into an underline instance. '-underline **this**' would be stored as an underline instance with a strong instance stored in its string """ paragraph_types = [ 'doc_bullet', 'doc_numbered', 'doc_description', 'doc_header', 'doc_table', ] text_types = [ 'doc_href1', 'doc_href2', 'doc_strong', 'doc_emphasize', 'doc_literal', 'doc_inner_link', 'doc_named_link', 'doc_underline', ] def __call__(self, doc): if type(doc) in StringTypes: doc=ST.StructuredText(doc) doc.setSubparagraphs(self.color_paragraphs( doc.getSubparagraphs())) else: doc=ST.StructuredTextDocument(self.color_paragraphs( doc.getSubparagraphs())) return doc def parse(self, raw_string, text_type, type=type, sts=StringTypes, lt=type([])): """ Parse accepts a raw_string, an expr to test the raw_string, and the raw_string's subparagraphs. Parse will continue to search through raw_string until all instances of expr in raw_string are found. If no instances of expr are found, raw_string is returned. Otherwise a list of substrings and instances is returned """ tmp = [] # the list to be returned if raw_string is split append=tmp.append if type(text_type) in sts: text_type=getattr(self, text_type) while 1: t = text_type(raw_string) if not t: break #an instance of expr was found t, start, end = t if start: append(raw_string[0:start]) tt=type(t) if tt in sts: # if we get a string back, add it to text to be parsed raw_string = t+raw_string[end:len(raw_string)] else: if tt is lt: # is we get a list, append it's elements tmp[len(tmp):]=t else: # normal case, an object append(t) raw_string = raw_string[end:len(raw_string)] if not tmp: return raw_string # nothing found if raw_string: append(raw_string) elif len(tmp)==1: return tmp[0] return tmp def color_text(self, str, types=None): """Search the paragraph for each special structure """ if types is None: types=self.text_types for text_type in types: if type(str) in StringTypes: str = self.parse(str, text_type) elif type(str) is ListType: r=[]; a=r.append for s in str: if type(s) in StringTypes: s=self.parse(s, text_type) if type(s) is ListType: r[len(r):]=s else: a(s) else: s.setColorizableTexts( map(self.color_text, s.getColorizableTexts() )) a(s) str=r else: r=[]; a=r.append; color=self.color_text for s in str.getColorizableTexts(): color(s, (text_type,)) a(s) str.setColorizableTexts(r) return str def color_paragraphs(self, raw_paragraphs, type=type, sequence_types=(type([]), type(())), sts=StringTypes): result=[] for paragraph in raw_paragraphs: if paragraph.getNodeName() != 'StructuredTextParagraph': result.append(paragraph) continue for pt in self.paragraph_types: if type(pt) in sts: # grab the corresponding function pt=getattr(self, pt) # evaluate the paragraph r=pt(paragraph) if r: if type(r) not in sequence_types: r=r, new_paragraphs=r for paragraph in new_paragraphs: paragraph.setSubparagraphs(self.color_paragraphs(paragraph.getSubparagraphs())) break else: new_paragraphs=ST.StructuredTextParagraph(paragraph.getColorizableTexts()[0], self.color_paragraphs(paragraph.getSubparagraphs()), indent=paragraph.indent), # color the inline StructuredText types # for each StructuredTextParagraph for paragraph in new_paragraphs: paragraph.setColorizableTexts( map(self.color_text, paragraph.getColorizableTexts() )) result.append(paragraph) return result def doc_table(self,paragraph, expr = re.compile('(\s*)([||]+)').match): #print "paragraph=>", type(paragraph), paragraph, paragraph._src text = paragraph.getColorizableTexts()[0] m = expr(text) if not (m): return None rows = [] # initial split for row in text.split("\n"): rows.append(row) # clean up the rows for index in range(len(rows)): tmp = [] rows[index] = rows[index].strip() l = len(rows[index])-2 result = rows[index][:l].split("||") for text in result: if text: tmp.append(text) tmp.append('') else: tmp.append(text) rows[index] = tmp # remove trailing '''s for index in range(len(rows)): l = len(rows[index])-1 rows[index] = rows[index][:l] result = [] for row in rows: cspan = 0 tmp = [] for item in row: if item: tmp.append((item,cspan)) cspan = 0 else: cspan = cspan + 1 result.append(tmp) subs = paragraph.getSubparagraphs() # indent=paragraph.indent return StructuredTextTable(result,text,subs,indent=paragraph.indent) def doc_bullet(self, paragraph, expr = re.compile('\s*[-*o]\s+').match): top=paragraph.getColorizableTexts()[0] m=expr(top) if not m: return None subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextBullet(top[m.span()[1]:], subs, indent=paragraph.indent, bullet=top[:m.span()[1]] ) def doc_numbered( self, paragraph, expr = re.compile('(\s*[%s]+\.)|(\s*[0-9]+\.)|(\s*[0-9]+\s+)' % letters).match): # This is the old expression. It had a nasty habit # of grabbing paragraphs that began with a single # letter word even if there was no following period. #expr = re.compile('\s*' # '(([a-zA-Z]|[0-9]+|[ivxlcdmIVXLCDM]+)\.)*' # '([a-zA-Z]|[0-9]+|[ivxlcdmIVXLCDM]+)\.?' # '\s+').match): top=paragraph.getColorizableTexts()[0] m=expr(top) if not m: return None subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextNumbered(top[m.span()[1]:], subs, indent=paragraph.indent, number=top[:m.span()[1]]) def doc_description( self, paragraph, delim = re.compile('\s+--\s+').search, nb=re.compile(r'[^\000- ]').search, ): top=paragraph.getColorizableTexts()[0] d=delim(top) if not d: return None start, end = d.span() title=top[:start] if title.find('\n') >= 0: return None if not nb(title): return None d=top[start:end] top=top[end:] subs=paragraph.getSubparagraphs() if top[-2:]=='::': subs=[StructuredTextExample(subs)] top=top[:-1] return StructuredTextDescription( title, top, subs, indent=paragraph.indent, delim=d) def doc_header(self, paragraph, expr = re.compile('[ %s0-9.:/,-_*<>\?\'\"]+' % letters).match ): subs=paragraph.getSubparagraphs() if not subs: return None top=paragraph.getColorizableTexts()[0] if not top.strip(): return None if top[-2:]=='::': subs=StructuredTextExample(subs) if top.strip()=='::': return subs return ST.StructuredTextParagraph(top[:-1], [subs], indent=paragraph.indent, level=paragraph.level) if top.find('\n') >= 0: return None return StructuredTextSection(top, subs, indent=paragraph.indent, level=paragraph.level) def doc_literal( self, s, expr=re.compile( "(?:\s|^)'" # open "([^ \t\n\r\f\v']|[^ \t\n\r\f\v'][^\n']*[^ \t\n\r\f\v'])" # contents "'(?:\s|[,.;:!?]|$)" # close ).search): r=expr(s) if r: start, end = r.span(1) return (StructuredTextLiteral(s[start:end]), start-1, end+1) else: return None def doc_emphasize( self, s, expr = re.compile('\s*\*([ \n%s0-9.:/;,\'\"\?\=\-\>\<\(\)]+)\*(?!\*|-)' % letters).search ): r=expr(s) if r: start, end = r.span(1) return (StructuredTextEmphasis(s[start:end]), start-1, end+1) else: return None def doc_inner_link(self, s, expr1 = re.compile("\.\.\s*").search, expr2 = re.compile("\[[%s0-9]+\]" % letters).search): # make sure we dont grab a named link if expr2(s) and expr1(s): start1,end1 = expr1(s).span() start2,end2 = expr2(s).span() if end1 == start2: # uh-oh, looks like a named link return None else: # the .. is somewhere else, ignore it return (StructuredTextInnerLink(s[start2+1,end2-1]),start2,end2) return None elif expr2(s) and not expr1(s): start,end = expr2(s).span() return (StructuredTextInnerLink(s[start+1:end-1]),start,end) return None def doc_named_link(self, s, expr=re.compile(r"(\.\.\s)(\[[%s0-9]+\])" % letters).search): result = expr(s) if result: start,end = result.span(2) str = s[start+1:end-1] st,en = result.span() return (StructuredTextNamedLink(str),st,en) return None def doc_underline(self, s, expr=re.compile("_([%s0-9\s\.,\?\/]+)_" % letters).search): result = expr(s) if result: start,end = result.span(1) st,e = result.span() return (StructuredTextUnderline(s[start:end]),st,e) else: return None def doc_strong(self, s, expr = re.compile('\s*\*\*([ \n%s0-9.:/;\-,!\?\'\"]+)\*\*' % letters).search ): r=expr(s) if r: start, end = r.span(1) return (StructuredTextStrong(s[start:end]), start-2, end+2) else: return None def doc_href1(self, s, expr=re.compile("(\"[ %s0-9\n\-\.\,\;\(\)\/\:\/\*\']+\")(:)([a-zA-Z0-9\@\.\,\?\!\/\:\;\-\#\~]+)([,]*\s*)" % letters).search ): return self.doc_href(s, expr) def doc_href2(self, s, expr=re.compile('(\"[ %s0-9\n\-\.\:\;\(\)\/\*\']+\")([,]+\s+)([a-zA-Z0-9\@\.\,\?\!\/\:\;\-\#\~]+)(\s*)' % letters).search ): return self.doc_href(s, expr) def doc_href(self, s, expr, punctuation = re.compile("[\,\.\?\!\;]+").match): r=expr(s) if r: # need to grab the href part and the # beginning part start,e = r.span(1) name = s[start:e] name = name.replace('"','',2) #start = start + 1 st,end = r.span(3) if punctuation(s[end-1:end]): end = end -1 link = s[st:end] #end = end - 1 # name is the href title, link is the target # of the href return (StructuredTextLink(name, href=link), start, end) #return (StructuredTextLink(s[start:end], href=s[start:end]), # start, end) else: return None Zope-2.8.7-final/lib/python/StructuredText/ClassicStructuredText.py0000664000076400007640000004244310435500325024531 0ustar ajungajung#! /usr/bin/env python -- # -*- python -*- ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''Structured Text Manipulation Parse a structured text string into a form that can be used with structured formats, like html. Structured text is text that uses indentation and simple symbology to indicate the structure of a document. A structured string consists of a sequence of paragraphs separated by one or more blank lines. Each paragraph has a level which is defined as the minimum indentation of the paragraph. A paragraph is a sub-paragraph of another paragraph if the other paragraph is the last preceding paragraph that has a lower level. Special symbology is used to indicate special constructs: - A single-line paragraph whose immediately succeeding paragraphs are lower level is treated as a header. - A paragraph that begins with a '-', '*', or 'o' is treated as an unordered list (bullet) element. - A paragraph that begins with a sequence of digits followed by a white-space character is treated as an ordered list element. - A paragraph that begins with a sequence of sequences, where each sequence is a sequence of digits or a sequence of letters followed by a period, is treated as an ordered list element. - A paragraph with a first line that contains some text, followed by some white-space and '--' is treated as a descriptive list element. The leading text is treated as the element title. - Sub-paragraphs of a paragraph that ends in the word 'example' or the word 'examples', or '::' is treated as example code and is output as is. - Text enclosed single quotes (with white-space to the left of the first quote and whitespace or punctuation to the right of the second quote) is treated as example code. - Text surrounded by '*' characters (with white-space to the left of the first '*' and whitespace or punctuation to the right of the second '*') is emphasized. - Text surrounded by '**' characters (with white-space to the left of the first '**' and whitespace or punctuation to the right of the second '**') is made strong. - Text surrounded by '_' underscore characters (with whitespace to the left and whitespace or punctuation to the right) is made underlined. - Text encloded by double quotes followed by a colon, a URL, and concluded by punctuation plus white space, *or* just white space, is treated as a hyper link. For example: "Zope":http://www.zope.org/ is ... Is interpreted as 'Zope is ....' Note: This works for relative as well as absolute URLs. - Text enclosed by double quotes followed by a comma, one or more spaces, an absolute URL and concluded by punctuation plus white space, or just white space, is treated as a hyper link. For example: "mail me", mailto:amos@digicool.com. Is interpreted as 'mail me.' - Text enclosed in brackets which consists only of letters, digits, underscores and dashes is treated as hyper links within the document. For example: As demonstrated by Smith [12] this technique is quite effective. Is interpreted as '... by Smith [12] this ...'. Together with the next rule this allows easy coding of references or end notes. - Text enclosed in brackets which is preceded by the start of a line, two periods and a space is treated as a named link. For example: .. [12] "Effective Techniques" Smith, Joe ... Is interpreted as '[12] "Effective Techniques" ...'. Together with the previous rule this allows easy coding of references or end notes. - A paragraph that has blocks of text enclosed in '||' is treated as a table. The text blocks correspond to table cells and table rows are denoted by newlines. By default the cells are center aligned. A cell can span more than one column by preceding a block of text with an equivalent number of cell separators '||'. Newlines and '|' cannot be a part of the cell text. For example: |||| **Ingredients** || || *Name* || *Amount* || ||Spam||10|| ||Eggs||3|| is interpreted as::
Ingredients
Name Amount
Spam 10
Eggs 3
''' import ts_regex import string, re def untabify(aString, indent_tab=ts_regex.compile('\(\n\|^\)\( *\)\t').search_group, ): '''\ Convert indentation tabs to spaces. ''' result='' rest=aString while 1: ts_results = indent_tab(rest, (1,2)) if ts_results: start, grps = ts_results lnl=len(grps[0]) indent=len(grps[1]) result=result+rest[:start] rest="\n%s%s" % (' ' * ((indent/8+1)*8), rest[start+indent+1+lnl:]) else: return result+rest def indent(aString, indent=2): """Indent a string the given number of spaces""" r=untabify(aString).split('\n') if not r: return '' if not r[-1]: del r[-1] tab=' '*indent return "%s%s\n" % (tab,('\n'+tab).join(r)) def reindent(aString, indent=2, already_untabified=0): "reindent a block of text, so that the minimum indent is as given" if not already_untabified: aString=untabify(aString) l=indent_level(aString)[0] if indent==l: return aString r=[] append=r.append if indent > l: tab=' ' * (indent-l) for s in aString.split('\n'): append(tab+s) else: l=l-indent for s in aString.split('\n'): append(s[l:]) return '\n'.join(r) def indent_level(aString, indent_space=ts_regex.compile('\n\( *\)').search_group, ): '''\ Find the minimum indentation for a string, not counting blank lines. ''' start=0 text='\n'+aString indent=l=len(text) while 1: ts_results = indent_space(text, (1,2), start) if ts_results: start, grps = ts_results i=len(grps[0]) start=start+i+1 if start < l and text[start] != '\n': # Skip blank lines if not i: return (0,aString) if i < indent: indent = i else: return (indent,aString) def paragraphs(list,start): l=len(list) level=list[start][0] i=start+1 while i < l and list[i][0] > level: i=i+1 return i-1-start def structure(list): if not list: return [] i=0 l=len(list) r=[] while i < l: sublen=paragraphs(list,i) i=i+1 r.append((list[i-1][1],structure(list[i:i+sublen]))) i=i+sublen return r class Table: CELL=' %s\n' ROW=' \n%s \n' TABLE='\n\n%s
' def create(self,aPar, td_reg=re.compile(r'[ \t\n]*\|\|([^\0|]*)') ): '''parses a table and returns nested list representing the table''' self.table=[] text=filter(None,aPar.split('\n')) for line in text: row=[] while 1: mo = td_reg.match(line) if not mo: return 0 pos = mo.end(1) row.append(mo.group(1)) if pos==len(line):break line=line[pos:] self.table.append(row) return 1 def html(self): '''Creates an HTML representation of table''' htmltable=[] for row in self.table: htmlrow=[] colspan=1 for cell in row: if cell=='': colspan=colspan+1 continue else: htmlrow.append(self.CELL%(colspan,cell)) colspan=1 htmltable.append(self.ROW % ''.join(htmlrow)) return self.TABLE % ''.join(htmltable) table=Table() class StructuredText: """Model text as structured collection of paragraphs. Structure is implied by the indentation level. This class is intended as a base classes that do actual text output formatting. """ def __init__(self, aStructuredString, level=0, paragraph_divider=ts_regex.compile('\(\r?\n *\)+\r?\n'), ): '''Convert a structured text string into a structured text object. Aguments: aStructuredString -- The string to be parsed. level -- The level of top level headings to be created. ''' pat = ' \"([%s0-9-_,./?=@~&]*)\":' % string.letters+ \ '([-:%s0-9_,./?=@#~&]*?)' % string.letters + \ '([.:?;] )' p_reg = re.compile(pat,re.M) aStructuredString = p_reg.sub(r'\1\3 ' , aStructuredString) pat = ' \"([%s0-9-_,./?=@~&]*)\", ' % string.letters+ \ '([-:%s0-9_,./?=@#~&]*?)' % string.letters + \ '([.:?;] )' p_reg = re.compile(pat,re.M) aStructuredString = p_reg.sub(r'\1\3 ' , aStructuredString) protoless = aStructuredString.find('\2\3',s) s=under.sub( r'\1\2\3',s) s=code.sub( r'\1\2\3',s) s=em.sub( r'\1\2\3',s) return s class HTML(StructuredText): '''\ An HTML structured text formatter. '''\ def __str__(self, extra_dl=re.compile("\n
"), extra_ul=re.compile("\n
    "), extra_ol=re.compile("\n
      "), ): '''\ Return an HTML string representation of the structured text data. ''' s=self._str(self.structure,self.level) s=extra_dl.sub('\n',s) s=extra_ul.sub('\n',s) s=extra_ol.sub('\n',s) return s def ul(self, before, p, after): if p: p="

      %s

      " % ctag(p).strip() return ('%s
      • %s\n%s\n
      \n' % (before,p,after)) def ol(self, before, p, after): if p: p="

      %s

      " % ctag(p).strip() return ('%s
      1. %s\n%s\n
      \n' % (before,p,after)) def dl(self, before, t, d, after): return ('%s
      %s

      %s

      \n%s\n
      \n' % (before,ctag(t),ctag(d),after)) def head(self, before, t, level, d): if level > 0 and level < 6: return ('%s%s\n%s\n' % (before,level,ctag(t).strip(),level,d)) t="

      %s

      " % ctag(t).strip() return ('%s
      %s\n
      %s\n
      \n' % (before,t,d)) def normal(self,before,p,after): return '%s

      %s

      \n%s\n' % (before,ctag(p),after) def pre(self,structure,tagged=0): if not structure: return '' if tagged: r='' else: r='
      \n'
        for s in structure:
            r="%s%s\n\n%s" % (r,html_quote(s[0]),self.pre(s[1],1))
        if not tagged: r=r.rstrip()+'\n
      \n' return r def table(self,before,table,after): return '%s

      %s

      \n%s\n' % (before,ctag(table),after) def _str(self,structure,level, # Static bullet=ts_regex.compile('[ \t\n]*[o*-][ \t\n]+\([^\0]*\)' ).match_group, example=ts_regex.compile('[\0- ]examples?:[\0- ]*$' ).search, dl=ts_regex.compile('\([^\n]+\)[ \t]+--[ \t\n]+\([^\0]*\)' ).match_group, nl=ts_regex.compile('\n').search, ol=ts_regex.compile( '[ \t]*\(\([0-9]+\|[%s]+\)[.)]\)+[ \t\n]+\([^\0]*\|$\)' % string.letters ).match_group, olp=ts_regex.compile('[ \t]*([0-9]+)[ \t\n]+\([^\0]*\|$\)' ).match_group, ): r='' for s in structure: ts_results = bullet(s[0], (1,)) if ts_results: p = ts_results[1] if s[0][-2:]=='::' and s[1]: ps=self.pre(s[1]) else: ps=self._str(s[1],level) r=self.ul(r,p,ps) continue ts_results = ol(s[0], (3,)) if ts_results: p = ts_results[1] if s[0][-2:]=='::' and s[1]: ps=self.pre(s[1]) else: ps=self._str(s[1],level) r=self.ol(r,p,ps) continue ts_results = olp(s[0], (1,)) if ts_results: p = ts_results[1] if s[0][-2:]=='::' and s[1]: ps=self.pre(s[1]) else: ps=self._str(s[1],level) r=self.ol(r,p,ps) continue ts_results = dl(s[0], (1,2)) if ts_results: t,d = ts_results[1] r=self.dl(r,t,d,self._str(s[1],level)) continue if example(s[0]) >= 0 and s[1]: # Introduce an example, using pre tags: r=self.normal(r,s[0],self.pre(s[1])) continue if s[0][-2:]=='::' and s[1]: # Introduce an example, using pre tags: r=self.normal(r,s[0][:-1],self.pre(s[1])) continue if table.create(s[0]): ## table support. r=self.table(r,table.html(),self._str(s[1],level)) continue else: if nl(s[0]) < 0 and s[1] and s[0][-1:] != ':': # Treat as a heading t=s[0] r=self.head(r,t,level, self._str(s[1],level and level+1)) else: r=self.normal(r,s[0],self._str(s[1],level)) return r def html_quote(v, character_entities=( (re.compile('&'), '&'), (re.compile("<"), '<' ), (re.compile(">"), '>' ), (re.compile('"'), '"') )): #" text=str(v) for re,name in character_entities: text=re.sub(name,text) return text def html_with_references(text, level=1): text = re.sub( r'[\0\n]\.\. \[([0-9_%s-]+)\]' % string.letters, r'\n       [\1]', text) text = re.sub( r'([\x00- ,])\[(?P[0-9_%s-]+)\]([\x00- ,.:])' % string.letters, r'\1[\2]\3', text) text = re.sub( r'([\0- ,])\[([^]]+)\.html\]([\0- ,.:])', r'\1[\2]\3', text) return HTML(text,level=level) def main(): import sys, getopt opts,args=getopt.getopt(sys.argv[1:],'twl') if args: [infile]=args s=open(infile,'r').read() else: s=sys.stdin.read() if opts: if filter(lambda o: o[0]=='-w', opts): print 'Content-Type: text/html\n' if filter(lambda o: o[0]=='-l', opts): import locale locale.setlocale(locale.LC_ALL,"") if s[:2]=='#!': s=re.sub('^#![^\n]+','',s) mo = re.compile('([\0-\n]*\n)').match(s) if mo is not None: s = s[len(mo.group(0)) :] s=str(html_with_references(s)) if s[:4]=='

      ': t=s[4: s.find('

      ')] s='''%s %s ''' % (t,s) print s else: print html_with_references(s) if __name__=="__main__": main() Zope-2.8.7-final/lib/python/StructuredText/STDOM.py0000664000076400007640000004120210435500325021074 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ DOM implementation in StructuredText : Read-Only methods All standard Zope objects support DOM to a limited extent. """ from types import StringType, UnicodeType StringTypes = (StringType, UnicodeType) # Node type codes # --------------- ELEMENT_NODE = 1 ATTRIBUTE_NODE = 2 TEXT_NODE = 3 CDATA_SECTION_NODE = 4 ENTITY_REFERENCE_NODE = 5 ENTITY_NODE = 6 PROCESSING_INSTRUCTION_NODE = 7 COMMENT_NODE = 8 DOCUMENT_NODE = 9 DOCUMENT_TYPE_NODE = 10 DOCUMENT_FRAGMENT_NODE = 11 NOTATION_NODE = 12 # Exception codes # --------------- INDEX_SIZE_ERR = 1 DOMSTRING_SIZE_ERR = 2 HIERARCHY_REQUEST_ERR = 3 WRONG_DOCUMENT_ERR = 4 INVALID_CHARACTER_ERR = 5 NO_DATA_ALLOWED_ERR = 6 NO_MODIFICATION_ALLOWED_ERR = 7 NOT_FOUND_ERR = 8 NOT_SUPPORTED_ERR = 9 INUSE_ATTRIBUTE_ERR = 10 # Exceptions # ---------- class DOMException(Exception): pass class IndexSizeException(DOMException): code = INDEX_SIZE_ERR class DOMStringSizeException(DOMException): code = DOMSTRING_SIZE_ERR class HierarchyRequestException(DOMException): code = HIERARCHY_REQUEST_ERR class WrongDocumentException(DOMException): code = WRONG_DOCUMENT_ERR class InvalidCharacterException(DOMException): code = INVALID_CHARACTER_ERR class NoDataAllowedException(DOMException): code = NO_DATA_ALLOWED_ERR class NoModificationAllowedException(DOMException): code = NO_MODIFICATION_ALLOWED_ERR class NotFoundException(DOMException): code = NOT_FOUND_ERR class NotSupportedException(DOMException): code = NOT_SUPPORTED_ERR class InUseAttributeException(DOMException): code = INUSE_ATTRIBUTE_ERR # Node classes # ------------ class ParentNode: """ A node that can have children, or, more precisely, that implements the child access methods of the DOM. """ def getChildNodes(self, type=type, sts=StringTypes): """ Returns a NodeList that contains all children of this node. If there are no children, this is a empty NodeList """ r=[] for n in self.getChildren(): if type(n) in sts: n=TextNode(n) r.append(n.__of__(self)) return NodeList(r) def getFirstChild(self, type=type, sts=StringTypes): """ The first child of this node. If there is no such node this returns None """ children = self.getChildren() if not children: return None n=children[0] if type(n) in sts: n=TextNode(n) return n.__of__(self) def getLastChild(self, type=type, sts=StringTypes): """ The last child of this node. If there is no such node this returns None. """ children = self.getChildren() if not children: return None n=children[-1] if type(n) in sts: n=TextNode(n) return n.__of__(self) """ create aliases for all above functions in the pythony way. """ def _get_ChildNodes(self, type=type, sts=StringTypes): return self.getChildNodes(type,sts) def _get_FirstChild(self, type=type, sts=StringTypes): return self.getFirstChild(type,sts) def _get_LastChild(self, type=type, sts=StringTypes): return self.getLastChild(type,sts) class NodeWrapper(ParentNode): """ This is an acquisition-like wrapper that provides parent access for DOM sans circular references! """ def __init__(self, aq_self, aq_parent): self.aq_self=aq_self self.aq_parent=aq_parent def __getattr__(self, name): return getattr(self.aq_self, name) def getParentNode(self): """ The parent of this node. All nodes except Document DocumentFragment and Attr may have a parent """ return self.aq_parent def _getDOMIndex(self, children, getattr=getattr): i=0 self=self.aq_self for child in children: if getattr(child, 'aq_self', child) is self: self._DOMIndex=i return i i=i+1 return None def getPreviousSibling(self): """ The node immediately preceding this node. If there is no such node, this returns None. """ children = self.aq_parent.getChildren() if not children: return None index=getattr(self, '_DOMIndex', None) if index is None: index=self._getDOMIndex(children) if index is None: return None index=index-1 if index < 0: return None try: n=children[index] except IndexError: return None else: if type(n) in StringTypes: n=TextNode(n) n._DOMIndex=index return n.__of__(self) def getNextSibling(self): """ The node immediately preceding this node. If there is no such node, this returns None. """ children = self.aq_parent.getChildren() if not children: return None index=getattr(self, '_DOMIndex', None) if index is None: index=self._getDOMIndex(children) if index is None: return None index=index+1 try: n=children[index] except IndexError: return None else: if type(n) in StringTypes: n=TextNode(n) n._DOMIndex=index return n.__of__(self) def getOwnerDocument(self): """ The Document object associated with this node, if any. """ return self.aq_parent.getOwnerDocument() """ create aliases for all above functions in the pythony way. """ def _get_ParentNode(self): return self.getParentNode() def _get_DOMIndex(self, children, getattr=getattr): return self._getDOMIndex(children,getattr) def _get_PreviousSibling(self): return self.getPreviousSibling() def _get_NextSibling(self): return self.getNextSibling() def _get_OwnerDocument(self): return self.getOwnerDocument() class Node(ParentNode): """ Node Interface """ # Get a DOM wrapper with a parent link def __of__(self, parent): return NodeWrapper(self, parent) # DOM attributes # -------------- def getNodeName(self): """ The name of this node, depending on its type """ def getNodeValue(self): """ The value of this node, depending on its type """ return None def getParentNode(self): """ The parent of this node. All nodes except Document DocumentFragment and Attr may have a parent """ def getChildren(self): """ Get a Python sequence of children """ return () def getPreviousSibling(self, type=type, sts=StringTypes, getattr=getattr): """ The node immediately preceding this node. If there is no such node, this returns None. """ def getNextSibling(self, type=type, sts=StringTypes): """ The node immediately preceding this node. If there is no such node, this returns None. """ def getAttributes(self): """ Returns a NamedNodeMap containing the attributes of this node (if it is an element) or None otherwise. """ return None def getOwnerDocument(self): """ The Document object associated with this node, if any. """ # DOM Methods # ----------- def hasChildNodes(self): """ Returns true if the node has any children, false if it doesn't. """ return len(self.getChildren()) """ create aliases for all above functions in the pythony way. """ def _get_NodeName(self): return self.getNodeName() def _get_NodeValue(self): return self.getNodeValue() def _get_ParentNode(self): return self.getParentNode() def _get_Children(self): return self.getChildren() def _get_PreviousSibling(self): return self.getPreviousSibling() def _get_NextSibling(self): return self.getNextSibling() def _get_Attributes(self): return self.getAttributes() def _get_OwnerDocument(self): return self.getOwnerDocument() def _has_ChildNodes(self): return self.hasChildNodes() class TextNode(Node): def __init__(self, str): self._value=str def getNodeType(self): return TEXT_NODE def getNodeName(self): return '#text' def getNodeValue(self): return self._value """ create aliases for all above functions in the pythony way. """ def _get_NodeType(self): return self.getNodeType() def _get_NodeName(self): return self.getNodeName() def _get_NodeValue(self): return self.getNodeValue() class Element(Node): """ Element interface """ # Element Attributes # ------------------ def getTagName(self): """The name of the element""" return self.__class__.__name__ def getNodeName(self): """The name of this node, depending on its type""" return self.__class__.__name__ def getNodeType(self): """A code representing the type of the node.""" return ELEMENT_NODE def getNodeValue(self): r=[] for c in self.getChildren(): if type(c) not in StringTypes: c=c.getNodeValue() r.append(c) return ''.join(r) def getParentNode(self): """ The parent of this node. All nodes except Document DocumentFragment and Attr may have a parent """ # Element Methods # --------------- _attributes=() def getAttribute(self, name): return getattr(self, name, None) def getAttributeNode(self, name): if hasattr(self, name): return Attr(name, getattr(self, name)) def getAttributes(self): d={} for a in self._attributes: d[a]=getattr(self, a, '') return NamedNodeMap(d) def getAttribute(self, name): """Retrieves an attribute value by name.""" return None def getAttributeNode(self, name): """ Retrieves an Attr node by name or None if there is no such attribute. """ return None def getElementsByTagName(self, tagname): """ Returns a NodeList of all the Elements with a given tag name in the order in which they would be encountered in a preorder traversal of the Document tree. Parameter: tagname The name of the tag to match (* = all tags). Return Value: A new NodeList object containing all the matched Elements. """ nodeList = [] for child in self.getChildren(): if not hasattr(child, 'getNodeType'): continue if (child.getNodeType()==ELEMENT_NODE and \ child.getTagName()==tagname or tagname== '*'): nodeList.append(child) if hasattr(child, 'getElementsByTagName'): n1 = child.getElementsByTagName(tagname) nodeList = nodeList + n1._data return NodeList(nodeList) """ create aliases for all above functions in the pythony way. """ def _get_TagName(self): return self.getTagName() def _get_NodeName(self): return self.getNodeName() def _get_NodeType(self): return self.getNodeType() def _get_NodeValue(self): return self.getNodeValue() def _get_ParentNode(self): return self.getParentNode() def _get_Attribute(self, name): return self.getAttribute(name) def _get_AttributeNode(self, name): return self.getAttributeNode(name) def _get_Attributes(self): return self.getAttributes() def _get_Attribute(self, name): return self.getAttribute(name) def _get_AttributeNode(self, name): return self.getAttributeNode(name) def _get_ElementsByTagName(self, tagname): return self.getElementsByTagName(tagname) class NodeList: """ NodeList interface - Provides the abstraction of an ordered collection of nodes. Python extensions: can use sequence-style 'len', 'getitem', and 'for..in' constructs. """ def __init__(self,list=None): self._data = list or [] def __getitem__(self, index, type=type, sts=StringTypes): return self._data[index] def __getslice__(self, i, j): return self._data[i:j] def item(self, index): """ Returns the index-th item in the collection """ try: return self._data[index] except IndexError: return None def getLength(self): """ The length of the NodeList """ return len(self._data) __len__=getLength """ create aliases for all above functions in the pythony way. """ def _get_Length(self): return self.getLength() class NamedNodeMap: """ NamedNodeMap interface - Is used to represent collections of nodes that can be accessed by name. NamedNodeMaps are not maintained in any particular order. Python extensions: can use sequence-style 'len', 'getitem', and 'for..in' constructs, and mapping-style 'getitem'. """ def __init__(self, data=None): if data is None: data = {} self._data = data def item(self, index): """ Returns the index-th item in the map """ try: return self._data.values()[index] except IndexError: return None def __getitem__(self, key): if type(key)==type(1): return self._data.values()[key] else: return self._data[key] def getLength(self): """ The length of the NodeList """ return len(self._data) __len__ = getLength def getNamedItem(self, name): """ Retrieves a node specified by name. Parameters: name Name of a node to retrieve. Return Value A Node (of any type) with the specified name, or None if the specified name did not identify any node in the map. """ if self._data.has_key(name): return self._data[name] return None """ create aliases for all above functions in the pythony way. """ def _get_Length(self): return self.getLength() def _get_NamedItem(self, name): return self.getNamedItem(name) class Attr(Node): """ Attr interface - The Attr interface represents an attriubte in an Element object. Attr objects inherit the Node Interface """ def __init__(self, name, value, specified=1): self.name = name self.value = value self.specified = specified def getNodeName(self): """ The name of this node, depending on its type """ return self.name def getName(self): """ Returns the name of this attribute. """ return self.name def getNodeValue(self): """ The value of this node, depending on its type """ return self.value def getNodeType(self): """ A code representing the type of the node. """ return ATTRIBUTE_NODE def getSpecified(self): """ If this attribute was explicitly given a value in the original document, this is true; otherwise, it is false. """ return self.specified """ create aliases for all above functions in the pythony way. """ def _get_NodeName(self): return self.getNodeName() def _get_Name(self): return self.getName() def _get_NodeValue(self): return self.getNodeValue() def _get_NodeType(self): return self.getNodeType() def _get_Specified(self): return self.getSpecified() Zope-2.8.7-final/lib/python/StructuredText/HTMLClass.py0000664000076400007640000002016110435500325021741 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from cgi import escape import re, sys, ST class HTMLClass: element_types={ '#text': '_text', 'StructuredTextDocument': 'document', 'StructuredTextParagraph': 'paragraph', 'StructuredTextExample': 'example', 'StructuredTextBullet': 'bullet', 'StructuredTextNumbered': 'numbered', 'StructuredTextDescription': 'description', 'StructuredTextDescriptionTitle': 'descriptionTitle', 'StructuredTextDescriptionBody': 'descriptionBody', 'StructuredTextSection': 'section', 'StructuredTextSectionTitle': 'sectionTitle', 'StructuredTextLiteral': 'literal', 'StructuredTextEmphasis': 'emphasis', 'StructuredTextStrong': 'strong', 'StructuredTextLink': 'link', 'StructuredTextXref': 'xref', 'StructuredTextInnerLink':'innerLink', 'StructuredTextNamedLink':'namedLink', 'StructuredTextUnderline':'underline', 'StructuredTextTable':'table', 'StructuredTextSGML':'sgml', } def dispatch(self, doc, level, output): node_name = doc.getNodeName() element_type = self.element_types[node_name] element_method = getattr(self, element_type) element_method(doc, level, output) def __call__(self, doc, level=1, header=1): r=[] self.header = header self.dispatch(doc, level-1, r.append) return ''.join(r) def _text(self, doc, level, output): output(doc.getNodeValue()) def document(self, doc, level, output): children=doc.getChildNodes() if self.header: output('\n') if (children and children[0].getNodeName() == 'StructuredTextSection'): output('\n%s\n\n' % children[0].getChildNodes()[0].getNodeValue()) output('\n') for c in children: self.dispatch(c, level, output) if self.header: output('\n') output('\n') def section(self, doc, level, output): children=doc.getChildNodes() for c in children: self.dispatch(c, level+1, output) def sectionTitle(self, doc, level, output): output('' % (level)) for c in doc.getChildNodes(): self.dispatch(c, level, output) output('\n' % (level)) def description(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('
      \n') for c in doc.getChildNodes(): self.dispatch(c, level, output) n=doc.getNextSibling() if n is None or n.getNodeName() is not doc.getNodeName(): output('
      \n') def descriptionTitle(self, doc, level, output): output('
      ') for c in doc.getChildNodes(): self.dispatch(c, level, output) output('
      \n') def descriptionBody(self, doc, level, output): output('
      ') for c in doc.getChildNodes(): self.dispatch(c, level, output) output('
      \n') def bullet(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('\n
        \n') output('
      • ') for c in doc.getChildNodes(): self.dispatch(c, level, output) n=doc.getNextSibling() output('
        • \n') if n is None or n.getNodeName() is not doc.getNodeName(): output('\n
      \n') def numbered(self, doc, level, output): p=doc.getPreviousSibling() if p is None or p.getNodeName() is not doc.getNodeName(): output('\n
        \n') output('
      1. ') for c in doc.getChildNodes(): self.dispatch(c, level, output) n=doc.getNextSibling() output('
        2. \n') if n is None or n.getNodeName() is not doc.getNodeName(): output('\n
      \n') def example(self, doc, level, output): i=0 for c in doc.getChildNodes(): if i==0: output('\n
      \n')
                output(escape(c.getNodeValue()))
                output('\n
      \n') else: self.dispatch(c, level, output) def paragraph(self, doc, level, output): output('

      ') for c in doc.getChildNodes(): if c.getNodeName() in ['StructuredTextParagraph']: self.dispatch(c, level, output) else: self.dispatch(c, level, output) output('

      \n') def link(self, doc, level, output): output('' % doc.href) for c in doc.getChildNodes(): self.dispatch(c, level, output) output('') def emphasis(self, doc, level, output): output('') for c in doc.getChildNodes(): self.dispatch(c, level, output) output('') def literal(self, doc, level, output): output('') for c in doc.getChildNodes(): output(escape(c.getNodeValue())) output('') def strong(self, doc, level, output): output('') for c in doc.getChildNodes(): self.dispatch(c, level, output) output('') def underline(self, doc, level, output): output("") for c in doc.getChildNodes(): self.dispatch(c, level, output) output("") def innerLink(self, doc, level, output): output('[') for c in doc.getChildNodes(): self.dispatch(c, level, output) output(']') def namedLink(self, doc, level, output): output('[') for c in doc.getChildNodes(): self.dispatch(c, level, output) output(']') def sgml(self,doc,level,output): for c in doc.getChildNodes(): self.dispatch(c, level, output) def xref(self, doc, level, output): val = doc.getNodeValue() output('[%s]' % (val, val) ) def table(self,doc,level,output): """ A StructuredTextTable holds StructuredTextRow(s) which holds StructuredTextColumn(s). A StructuredTextColumn is a type of StructuredTextParagraph and thus holds the actual data. """ output('\n') for row in doc.getRows()[0]: output("\n") for column in row.getColumns()[0]: if hasattr(column,"getAlign"): str = '<%s colspan="%s" align="%s" valign="%s">' % (column.getType(), column.getSpan(), column.getAlign(), column.getValign()) else: str = '\n") output("\n") output("
      ' % column.getSpan() output(str) for c in column.getChildNodes(): self.dispatch(c, level, output) if hasattr(column,"getType"): output("\n") else: output("
      \n") Zope-2.8.7-final/lib/python/ThreadedAsync/0000775000076400007640000000000010435610641017345 5ustar ajungajungZope-2.8.7-final/lib/python/ThreadedAsync/__init__.py0000664000076400007640000000150010435502411021445 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Manage the asyncore mainloop in a multi-threaded app. $Id: __init__.py 25186 2004-06-02 15:07:33Z jim $ """ from LoopCallback import register_loop_callback, loop, remove_loop_callback Zope-2.8.7-final/lib/python/ThreadedAsync/LoopCallback.py0000664000076400007640000001264410435502411022247 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Manage the asyncore mainloop in a multi-threaded app. In a multi-threaded application, only a single thread runs the asyncore mainloop. This thread (the "mainloop thread") may not start the mainloop before another thread needs to perform an async action that requires it. As a result, other threads need to coordinate with the mainloop thread to find out whether the mainloop is running. This module implements a callback mechanism that allows other threads to be notified when the mainloop starts. A thread calls register_loop_callback() to register interest. When the mainloop thread calls loop(), each registered callback will be called with the socket map as its first argument. """ import asyncore import thread # Zope pokes a non-None value into exit_status when it wants the loop() # function to exit. Indeed, there appears to be no other way to tell # Zope3 to shut down. exit_status = None _loop_lock = thread.allocate_lock() _looping = None # changes to socket map when loop() starts _loop_callbacks = [] def register_loop_callback(callback, args=(), kw=None): """Register callback function to be called when mainloop starts. The callable object callback will be invokved when the mainloop starts. If the mainloop is currently running, the callback will be invoked immediately. The callback will be called with a single argument, the mainloop socket map, unless the optional args or kw arguments are used. args defines a tuple of extra arguments to pass after the socket map. kw defines a dictionary of keyword arguments. """ _loop_lock.acquire() try: if _looping is not None: callback(_looping, *args, **(kw or {})) else: _loop_callbacks.append((callback, args, kw)) finally: _loop_lock.release() def remove_loop_callback(callback): """Remove a callback function registered earlier. This is useful if loop() was never called. """ for i, value in enumerate(_loop_callbacks): if value[0] == callback: del _loop_callbacks[i] return # Because of the exit_status magic, we can't just invoke asyncore.loop(), # and that's a shame. # The signature of asyncore.loop changed between Python 2.3 and 2.4, and # this loop() has 2.4's signature, which added the optional `count` argument. # Since we physically replace asyncore.loop with this `loop`, and want # compatibility with both Pythons, we need to support the most recent # signature. Applications running under 2.3 should (of course) avoid using # the `count` argument, since 2.3 doesn't have it. def loop(timeout=30.0, use_poll=False, map=None, count=None): global _looping global exit_status exit_status = None if map is None: map = asyncore.socket_map # This section is taken from Python 2.3's asyncore.loop, and is more # elaborate than the corresponding section of 2.4's: in 2.4 poll2 and # poll3 are aliases for the same function, in 2.3 they're different # functions. if use_poll: if hasattr(select, 'poll'): poll_fun = asyncore.poll3 else: poll_fun = asyncore.poll2 else: poll_fun = asyncore.poll # The loop is about to start: invoke any registered callbacks. _loop_lock.acquire() try: _looping = map while _loop_callbacks: cb, args, kw = _loop_callbacks.pop() cb(map, *args, **(kw or {})) finally: _loop_lock.release() # Run the loop. This is 2.4's logic, with the addition that we stop # if/when this module's exit_status global is set to a non-None value. if count is None: while map and exit_status is None: poll_fun(timeout, map) else: while map and count > 0 and exit_status is None: poll_fun(timeout, map) count -= 1 _loop_lock.acquire() try: _looping = None finally: _loop_lock.release() # Evil: rebind asyncore.loop to the above loop() function. # # Code should explicitly call ThreadedAsync.loop() instead of asyncore.loop(). # Most of ZODB has been fixed, but ripping this out may break 3rd party code. # Maybe we should issue a warning and let it continue for a while (NOTE: code # to raise DeprecationWarning was written but got commented out below; don't # know why it got commented out). Or maybe we should get rid of this # mechanism entirely, and have each piece that needs one run its own asyncore # loop in its own thread. ##def deprecated_loop(*args, **kws): ## import warnings ## warnings.warn("""\ ##ThreadedAsync.loop() called through sneaky asyncore.loop() rebinding. ##You should change your code to call ThreadedAsync.loop() explicitly.""", ## DeprecationWarning) ## loop(*args, **kws) ## ##asyncore.loop = deprecated_loop asyncore.loop = loop Zope-2.8.7-final/lib/python/ZServer/0000775000076400007640000000000010435610641016227 5ustar ajungajungZope-2.8.7-final/lib/python/ZServer/AccessLogger.py0000664000076400007640000000221710435500320021134 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ A logging module which handles ZServer access log messages. This depends on Vinay Sajip's PEP 282 logging module. """ from ZServer.BaseLogger import BaseLogger class AccessLogger(BaseLogger): def __init__(self): BaseLogger.__init__(self, 'access') def log(self, message): if not self.logger.handlers: # don't log if we have no handlers return if message.endswith('\n'): message = message[:-1] self.logger.warn(message) access_logger = AccessLogger() Zope-2.8.7-final/lib/python/ZServer/FCGIServer.py0000664000076400007640000006114110435500320020473 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ ZServer/Medusa FastCGI server, by Robin Dunn. Accepts connections from a FastCGI enabled webserver, receives request info using the FastCGi protocol, and then hands the request off to ZPublisher for processing. The response is then handed back to the webserver to send down to the browser. See http://www.fastcgi.com/fcgi-devkit-2.1/doc/fcgi-spec.html for the protocol specificaition. """ __version__ = "1.0" #---------------------------------------------------------------------- import asynchat, asyncore from medusa import logger from medusa.counter import counter from medusa.http_server import compute_timezone_for_log from ZServer import requestCloseOnExec from PubCore import handle from PubCore.ZEvent import Wakeup from ZPublisher.HTTPResponse import HTTPResponse from ZPublisher.HTTPRequest import HTTPRequest from Producers import ShutdownProducer, LoggingProducer, file_part_producer, file_close_producer import DebugLogger from cStringIO import StringIO from tempfile import TemporaryFile import socket, string, os, sys, time import thread from types import StringTypes tz_for_log = compute_timezone_for_log() #---------------------------------------------------------------------- # Set various FastCGI constants # Maximum number of requests that can be handled. Apache mod_fastcgi # never asks for these values, so we actually will handle as many # connections/requests as they attempt upto the limits of ZServer. # These values are suitable defaults for any web server that does ask. FCGI_MAX_CONNS = 10 FCGI_MAX_REQS = 50 # Supported version of the FastCGI protocol FCGI_VERSION_1 = 1 # Boolean: can this application multiplex connections? FCGI_MPXS_CONNS=0 # Record types FCGI_BEGIN_REQUEST = 1 FCGI_ABORT_REQUEST = 2 FCGI_END_REQUEST = 3 FCGI_PARAMS = 4 FCGI_STDIN = 5 FCGI_STDOUT = 6 FCGI_STDERR = 7 FCGI_DATA = 8 FCGI_GET_VALUES = 9 FCGI_GET_VALUES_RESULT = 10 FCGI_UNKNOWN_TYPE = 11 FCGI_MAXTYPE = FCGI_UNKNOWN_TYPE # Types of management records FCGI_ManagementTypes = [ FCGI_GET_VALUES ] FCGI_NULL_REQUEST_ID=0 # Masks for flags component of FCGI_BEGIN_REQUEST FCGI_KEEP_CONN = 1 # Values for role component of FCGI_BEGIN_REQUEST FCGI_RESPONDER = 1 FCGI_AUTHORIZER = 2 FCGI_FILTER = 3 # Values for protocolStatus component of FCGI_END_REQUEST FCGI_REQUEST_COMPLETE = 0 # Request completed nicely FCGI_CANT_MPX_CONN = 1 # This app can't multiplex FCGI_OVERLOADED = 2 # New request rejected; too busy FCGI_UNKNOWN_ROLE = 3 # Role value not known #---------------------------------------------------------------------- class FCGIRecord: """ This class represents the various record structures used in the FastCGI protocol. It knows how to read and build itself bits at a time as they are read from the FCGIChannel. There are really several different record types but in this case subclassing for each type is probably overkill. See the FastCGI spec for structure and other details for all these record types. """ def __init__(self, header=None): if header: # extract the record header values. vals = map(ord, header) self.version = vals[0] self.recType = vals[1] self.reqId = (vals[2] << 8) + vals[3] self.contentLength = (vals[4] << 8) + vals[5] self.paddingLength = vals[6] else: self.version = FCGI_VERSION_1 self.recType = FCGI_UNKNOWN_TYPE self.reqId = FCGI_NULL_REQUEST_ID self.content = "" def needContent(self): return (self.contentLength and not self.content) def needPadding(self): return self.paddingLength != 0 def needMore(self): if self.needContent(): return self.contentLength else: return self.paddingLength def gotPadding(self): self.paddingLength = 0 def parseContent(self, data): c = self.content = data if self.recType == FCGI_BEGIN_REQUEST: self.role = (ord(c[0]) << 8) + ord(c[1]) self.flags = ord(c[2]) elif self.recType == FCGI_UNKNOWN_TYPE: self.unknownType = ord(c[0]) elif self.recType == FCGI_GET_VALUES or self.recType == FCGI_PARAMS: self.values = {} pos = 0 while pos < len(c): name, value, pos = self.readPair(c, pos) self.values[name] = value elif self.recType == FCGI_END_REQUEST: b = map(ord, c[0:4]) self.appStatus = (b[0] << 24) + (b[1] << 16) + (b[2] << 8) + b[3] self.protocolStatus = ord(c[4]) def readPair(self, st, pos): """ Read the next name-value pair from st at pos. """ nameLen = ord(st[pos]) pos = pos + 1 if nameLen & 0x80: # is the high bit set? if so, size is 4 bytes, not 1. b = map(ord, st[pos:pos+3]) pos = pos + 3 nameLen = ((nameLen & 0x7F) << 24) + (b[0] << 16) + (b[1] << 8) + b[2] valueLen = ord(st[pos]) pos = pos + 1 if valueLen & 0x80: # same thing here... b = map(ord, st[pos:pos+3]) pos = pos + 3 valueLen = ((valueLen & 0x7F) << 24) + (b[0] << 16) + (b[1] << 8) + b[2] # pull out the name and value and return with the updated position return ( st[pos : pos+nameLen], st[pos + nameLen : pos + nameLen + valueLen], pos + nameLen + valueLen ) def writePair(name, value): """ Opposite of readPair """ l = len(name) if l < 0x80: st = chr(l) else: st = chr(0x80 | (l >> 24) & 0xFF) + chr((l >> 16) & 0xFF) + \ chr((l >> 8) & 0xFF) + chr(l & 0xFF) l = len(value) if l < 0x80: st = st + chr(l) else: st = st + chr(0x80 | (l >> 24) & 0xFF) + chr((l >> 16) & 0xFF) + \ chr((l >> 8) & 0xFF) + chr(l & 0xFF) return st + name + value def getRecordAsString(self): """ Format the record to be sent back to the web server. """ content = self.content if self.recType == FCGI_BEGIN_REQUEST: content = chr(self.role>>8) + chr(self.role & 0xFF) + \ chr(self.flags) + 5*'\000' elif self.recType == FCGI_UNKNOWN_TYPE: content = chr(self.unknownType) + 7*'\000' elif self.recType == FCGI_GET_VALUES or self.recType == FCGI_PARAMS: content = "" for i in self.values.keys(): content = content + self.writePair(i, self.values[i]) elif self.recType == FCGI_END_REQUEST: v = self.appStatus content = chr((v >> 24) & 0xFF) + chr((v >> 16) & 0xFF) + \ chr((v >> 8) & 0xFF) + chr(v & 0xFF) content = content + chr(self.protocolStatus) + 3*'\000' cLen = len(content) eLen = (cLen + 7) & (0xFFFF - 7) # align to an 8-byte boundary padLen = eLen - cLen hdr = [ self.version, self.recType, self.reqId >> 8, self.reqId & 0xFF, cLen >> 8, cLen & 0xFF, padLen, 0] hdr = string.join(map(chr, hdr), '') return hdr + content + padLen * '\000' #---------------------------------------------------------------------- class FCGIChannel(asynchat.async_chat): """ Process a FastCGI connection. This class implements most of the Application Server side of the protocol defined in http://www.fastcgi.com/fcgi-devkit-2.1/doc/fcgi-spec.html (which is the FastCGI Specification 1.0 from Open Market, Inc.) in a manner that is compatible with the asyncore medusa engine of ZServer. The main ommission from the spec is support for multiplexing multiple requests on a single connection, but since none of the web servers support it (that I know of,) and since ZServer can easily multiplex multiple connections in the same process, it's no great loss. """ closed=0 using_temp_stdin=None def __init__(self, server, sock, addr): self.server = server self.addr = addr asynchat.async_chat.__init__(self, sock) requestCloseOnExec(sock) self.setInitialState() self.remainingRecs = 1 # We have to read at least one self.env = {} self.stdin = StringIO() self.filterData = StringIO() # not currently used, but maybe someday self.requestId = 0 def setInitialState(self): self.data = StringIO() self.curRec = None self.set_terminator(8) # FastCGI record header size. def readable(self): return self.remainingRecs != 0 def collect_incoming_data(self, data): self.data.write(data) def found_terminator(self): # Are we starting a new record? If so, data is the header. if not self.curRec: self.curRec = FCGIRecord(self.data.getvalue()) if self.curRec.needMore(): self.set_terminator(self.curRec.needMore()) self.data = StringIO() return rec = self.curRec # If waiting for record content, give it to the record. if rec.needContent(): rec.parseContent(self.data.getvalue()) if rec.needMore(): self.set_terminator(rec.needMore()) self.data = StringIO() return if rec.needPadding(): rec.gotPadding() # If we get this far without returning, we've got the whole # record. Figure out what to do with it. if rec.recType in FCGI_ManagementTypes: # Apache mod_fastcgi doesn't send these, but others may self.handleManagementTypes(rec) elif rec.reqId == 0: # It's a management record of unknown type. # Complain about it... r2 = FCGIRecord() r2.recType = FCGI_UNKNOWN_TYPE r2.unknownType = rec.recType self.push(r2.getRecordAsString(), 0) # Since we don't actually have to do anything to ignore the # following conditions, they have been commented out and have # been left in the code for documentation purposes. # Ignore requests that aren't active # elif rec.reqId != self.requestId and rec.recType != FCGI_BEGIN_REQUEST: # pass # # If we're already doing a request, ignore further BEGIN_REQUESTs # elif rec.recType == FCGI_BEGIN_REQUEST and self.requestId != 0: # pass # Begin a new request elif rec.recType == FCGI_BEGIN_REQUEST and self.requestId == 0: self.requestId = rec.reqId if rec.role == FCGI_AUTHORIZER: self.remainingRecs = 1 elif rec.role == FCGI_RESPONDER: self.remainingRecs = 2 elif rec.role == FCGI_FILTER: self.remainingRecs = 3 # Read some name-value pairs (the CGI environment) elif rec.recType == FCGI_PARAMS: if rec.contentLength == 0: # end of the stream if self.env.has_key('REQUEST_METHOD'): method=self.env['REQUEST_METHOD'] else: method='GET' if self.env.has_key('PATH_INFO'): path=self.env['PATH_INFO'] else: path='' DebugLogger.log('B', id(self), '%s %s' % (method, path)) self.remainingRecs = self.remainingRecs - 1 self.content_length=string.atoi(self.env.get( 'CONTENT_LENGTH','0')) else: self.env.update(rec.values) # read some stdin data elif rec.recType == FCGI_STDIN: if rec.contentLength == 0: # end of the stream self.remainingRecs = self.remainingRecs - 1 else: # see if stdin is getting too big, and # replace it with a tempfile if necessary if len(rec.content) + self.stdin.tell() > 1048576 and \ not self.using_temp_stdin: t=TemporaryFile() t.write(self.stdin.getvalue()) self.stdin=t self.using_temp_stdin=1 self.stdin.write(rec.content) # read some filter data elif rec.recType == FCGI_DATA: if rec.contentLength == 0: # end of the stream self.remainingRecs = self.remainingRecs - 1 else: self.filterData.write(rec.content) # We've processed the record. Now what do we do? if self.remainingRecs > 0: # prepare to get the next record self.setInitialState() else: # We've got them all. Let ZPublisher do its thang. DebugLogger.log('I', id(self), self.stdin.tell()) # But first, fixup the auth header if using newest mod_fastcgi. if self.env.has_key('Authorization'): self.env['HTTP_AUTHORIZATION'] = self.env['Authorization'] del self.env['Authorization'] self.stdin.seek(0) self.send_response() def send_response(self): """ Create output pipes, request, and response objects. Give them to ZPublisher for processing. """ response = FCGIResponse(stdout = FCGIPipe(self, FCGI_STDOUT), stderr = StringIO()) response.setChannel(self) request = HTTPRequest(self.stdin, self.env, response) handle(self.server.module, request, response) def log_request(self, bytes): DebugLogger.log('E', id(self)) if self.env.has_key('HTTP_USER_AGENT'): user_agent=self.env['HTTP_USER_AGENT'] else: user_agent='' if self.env.has_key('HTTP_REFERER'): referer=self.env['HTTP_REFERER'] else: referer='' if self.env.has_key('PATH_INFO'): path=self.env['PATH_INFO'] else: path='' if self.env.has_key('REQUEST_METHOD'): method=self.env['REQUEST_METHOD'] else: method="GET" if self.addr: self.server.logger.log ( self.addr[0], '%s - - [%s] "%s %s" %d %d "%s" "%s"' % ( self.addr[1], time.strftime ( '%d/%b/%Y:%H:%M:%S ', time.localtime(time.time()) ) + tz_for_log, method, path, self.reply_code, bytes, referer, user_agent ) ) else: self.server.logger.log ( '127.0.0.1 ', '- - [%s] "%s %s" %d %d "%s" "%s"' % ( time.strftime ( '%d/%b/%Y:%H:%M:%S ', time.localtime(time.time()) ) + tz_for_log, method, path, self.reply_code, bytes, referer, user_agent ) ) def handleManagementTypes(self, rec): """ The web server has asked us what features we support... """ if rec.recType == FCGI_GET_VALUES: rec.recType = FCGI_GET_VALUES_RESULT vars={'FCGI_MAX_CONNS' : FCGI_MAX_CONNS, 'FCGI_MAX_REQS' : FCGI_MAX_REQS, 'FCGI_MPXS_CONNS': FCGI_MPXS_CONNS} rec.values = vars self.push(rec.getRecordAsString(), 0) def sendDataRecord(self, data, recType): rec = FCGIRecord() rec.recType = recType rec.reqId = self.requestId # Can't send more than 64K minus header size. 8K seems about right. if type(data)==type(''): # send some string data while data: chunk = data[:8192] data = data[8192:] rec.content = chunk self.push(rec.getRecordAsString(), 0) else: # send a producer p, cLen=data eLen = (cLen + 7) & (0xFFFF - 7) # align to an 8-byte boundary padLen = eLen - cLen hdr = [ rec.version, rec.recType, rec.reqId >> 8, rec.reqId & 0xFF, cLen >> 8, cLen & 0xFF, padLen, 0] hdr = string.join(map(chr, hdr), '') self.push(hdr, 0) self.push(p, 0) self.push(padLen * '\000', 0) def sendStreamTerminator(self, recType): rec = FCGIRecord() rec.recType = recType rec.reqId = self.requestId rec.content = "" self.push(rec.getRecordAsString(), 0) def sendEndRecord(self, appStatus=0): rec = FCGIRecord() rec.recType = FCGI_END_REQUEST rec.reqId = self.requestId rec.protocolStatus = FCGI_REQUEST_COMPLETE rec.appStatus = appStatus self.push(rec.getRecordAsString(), 0) self.requestId = 0 def push(self, producer, send=1): # this is thread-safe when send is false # note, that strings are not wrapped in # producers by default if self.closed: return self.producer_fifo.push(producer) if send: self.initiate_send() push_with_producer=push def close(self): self.closed=1 while self.producer_fifo: p=self.producer_fifo.first() if p is not None and not isinstance(p, StringTypes): p.more() # free up resources held by producer self.producer_fifo.pop() asyncore.dispatcher.close(self) #---------------------------------------------------------------------- class FCGIServer(asyncore.dispatcher): """ Listens for and accepts FastCGI requests and hands them off to a FCGIChannel for handling. FCGIServer can be configured to listen on either a specific port (for inet sockets) or socket_file (for unix domain sockets.) For inet sockets, the ip argument specifies the address from which the server will accept connections, '' indicates all addresses. If you only want to accept connections from the localhost, set ip to '127.0.0.1'. """ channel_class=FCGIChannel def __init__(self, module='Main', ip='127.0.0.1', port=None, socket_file=None, resolver=None, logger_object=None): self.ip = ip self.count=counter() asyncore.dispatcher.__init__(self) if not logger_object: logger_object = logger.file_logger(sys.stdout) if resolver: self.logger = logger.resolving_logger(resolver, logger_object) else: self.logger = logger.unresolving_logger(logger_object) # get configuration self.module = module self.port = port self.socket_file = socket_file # setup sockets if self.port: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.set_reuse_addr() self.bind((self.ip, self.port)) else: try: os.unlink(self.socket_file) except os.error: pass self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) self.set_reuse_addr() self.bind(self.socket_file) try: os.chmod(self.socket_file,0777) except os.error: pass self.listen(256) self.log_info('FastCGI Server (V%s) started at %s\n' '\tIP : %s\n' '\tPort : %s\n' '\tSocket path : %s\n' % (__version__, time.ctime(time.time()), self.ip, self.port, self.socket_file)) def handle_accept(self): self.count.increment() try: conn, addr = self.accept() except socket.error: self.log_info('Server accept() threw an exception', 'warning') return self.channel_class(self, conn, addr) def readable(self): from ZServer import CONNECTION_LIMIT return len(asyncore.socket_map) < CONNECTION_LIMIT def writable (self): return 0 def create_socket(self, family, type): asyncore.dispatcher.create_socket(self, family, type) requestCloseOnExec(self.socket) def listen(self, num): # override asyncore limits for nt's listen queue size self.accepting = 1 return self.socket.listen(num) #---------------------------------------------------------------------- class FCGIResponse(HTTPResponse): _tempfile=None _templock=None _tempstart=0 def setChannel(self, channel): self.channel = channel def write(self, data): if type(data) != type(''): raise TypeError('Value must be a string') stdout=self.stdout if not self._wrote: l=self.headers.get('content-length', None) if l is not None: try: if type(l) is type(''): l=string.atoi(l) if l > 128000: self._tempfile=TemporaryFile() self._templock=thread.allocate_lock() except: pass stdout.write(str(self)) self._wrote=1 if not data: return t=self._tempfile if t is None: stdout.write(data) else: while data: # write file producers # each producer holds 32K data chunk=data[:32768] data=data[32768:] l=len(chunk) b=self._tempstart e=b+l self._templock.acquire() try: t.seek(b) t.write(chunk) finally: self._templock.release() self._tempstart=e stdout.write((file_part_producer(t,self._templock,b,e), l)) def _finish(self): self.channel.reply_code=self.status DebugLogger.log('A', id(self.channel), '%d %d' % ( self.status, self.stdout.length)) t=self._tempfile if t is not None: self.stdout.write((file_close_producer(t), 0)) self._tempfile=None self.channel.sendStreamTerminator(FCGI_STDOUT) self.channel.sendEndRecord() self.stdout.close() self.stderr.close() if not self.channel.closed: self.channel.push_with_producer(LoggingProducer(self.channel, self.stdout.length, 'log_request'), 0) if self._shutdownRequested(): self.channel.push(ShutdownProducer(), 0) Wakeup(lambda: asyncore.close_all()) else: self.channel.push(None,0) Wakeup() self.channel=None #---------------------------------------------------------------------- class FCGIPipe: """ This class acts like a file and is used to catch stdout/stderr from ZPublisher and create FCGI records out of the data stream to send back to the web server. """ def __init__(self, channel, recType): self.channel = channel self.recType = recType self.length = 0 def write(self, data): if type(data)==type(''): datalen = len(data) else: p, datalen = data if data: self.channel.sendDataRecord(data, self.recType) self.length = self.length + datalen def close(self): self.channel = None #---------------------------------------------------------------------- Zope-2.8.7-final/lib/python/ZServer/PCGIServer.py0000664000076400007640000003221710435500320020507 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Medusa PCGI server. This server functions as the PCGI publisher--it accepts the request from the PCGI wrapper CGI program, services the request, and sends back the response. It should work with both inet and unix domain sockets. Why would you want to use it? Using PCGI to connect to ZServer from another webserver is similar to using the web server as a proxy, with the difference, that the web server gets to control the environment and headers completely. Note that ZServer can operate multiple PCGI servers. """ from medusa import logger import asynchat, asyncore from medusa.counter import counter from medusa.http_server import compute_timezone_for_log from asyncore import compact_traceback import ZServer from ZServer import requestCloseOnExec from PubCore import handle from PubCore.ZEvent import Wakeup from ZPublisher.HTTPResponse import HTTPResponse from ZPublisher.HTTPRequest import HTTPRequest from Producers import ShutdownProducer, LoggingProducer, CallbackProducer import DebugLogger from cStringIO import StringIO from tempfile import TemporaryFile import socket, string, os, sys, time from types import TupleType, StringTypes tz_for_log=compute_timezone_for_log() class ParseError(Exception): pass class PCGIChannel(asynchat.async_chat): """Processes a PCGI request by collecting the env and stdin and then passing them to ZPublisher. The result is wrapped in a producer and sent back.""" closed=0 def __init__(self,server,sock,addr): self.server = server self.addr = addr asynchat.async_chat.__init__ (self, sock) requestCloseOnExec(sock) self.env={} self.data=StringIO() self.set_terminator(10) self.size=None self.done=None def found_terminator(self): if self.size is None: # read the next size header # and prepare to read env or stdin self.data.seek(0) self.size=string.atoi(self.data.read()) self.set_terminator(self.size) if self.size==0: DebugLogger.log('I', id(self), 0) self.set_terminator('\r\n') self.data=StringIO() self.send_response() elif self.size > 1048576: self.data=TemporaryFile('w+b') else: self.data=StringIO() elif not self.env: # read env self.size=None self.data.seek(0) buff=self.data.read() for line in string.split(buff,'\000'): try: k,v = string.split(line,'=',1) self.env[k] = v except: pass # Hack around broken IIS PATH_INFO # maybe, this should go in ZPublisher... if self.env.has_key('SERVER_SOFTWARE') and \ string.find(self.env['SERVER_SOFTWARE'], 'Microsoft-IIS') != -1: script = filter(None,string.split( string.strip(self.env['SCRIPT_NAME']),'/')) path = filter(None,string.split( string.strip(self.env['PATH_INFO']),'/')) self.env['PATH_INFO'] = '/' + string.join(path[len(script):],'/') self.data=StringIO() DebugLogger.log('B', id(self), '%s %s' % (self.env['REQUEST_METHOD'], self.env.get('PATH_INFO' ,'/'))) # now read the next size header self.set_terminator(10) else: DebugLogger.log('I', id(self), self.terminator) # we're done, we've got both env and stdin self.set_terminator('\r\n') self.data.seek(0) self.send_response() def send_response(self): # create an output pipe by passing request to ZPublisher, # and requesting a callback of self.log with the module # name and PATH_INFO as an argument. self.done=1 response=PCGIResponse(stdout=PCGIPipe(self), stderr=StringIO()) request=HTTPRequest(self.data, self.env, response) handle(self.server.module, request, response) def collect_incoming_data(self, data): self.data.write(data) def readable(self): if not self.done: return 1 def log_request(self, bytes): if self.env.has_key('HTTP_USER_AGENT'): user_agent=self.env['HTTP_USER_AGENT'] else: user_agent='' if self.env.has_key('HTTP_REFERER'): referer=self.env['HTTP_REFERER'] else: referer='' if self.env.has_key('PATH_INFO'): path=self.env['PATH_INFO'] else: path='%s/' % self.server.module if self.env.has_key('REQUEST_METHOD'): method=self.env['REQUEST_METHOD'] else: method="GET" addr=self.addr if addr and type(addr) is TupleType: self.server.logger.log ( addr[0], '%d - - [%s] "%s %s" %d %d "%s" "%s"' % ( addr[1], time.strftime ( '%d/%b/%Y:%H:%M:%S ', time.localtime(time.time()) ) + tz_for_log, method, path, self.reply_code, bytes, referer, user_agent ) ) else: self.server.logger.log ( '127.0.0.1', ' - - [%s] "%s %s" %d %d "%s" "%s"' % ( time.strftime ( '%d/%b/%Y:%H:%M:%S ', time.gmtime(time.time()) ) + tz_for_log, method, path, self.reply_code, bytes, referer, user_agent ) ) def push(self, producer, send=1): # this is thread-safe when send is false # note, that strings are not wrapped in # producers by default self.producer_fifo.push(producer) if send: self.initiate_send() def __repr__(self): return "" % id(self) def close(self): self.closed=1 while self.producer_fifo: p=self.producer_fifo.first() if p is not None and not isinstance(p, StringTypes): p.more() # free up resources held by producer self.producer_fifo.pop() asyncore.dispatcher.close(self) class PCGIServer(asyncore.dispatcher): """Accepts PCGI requests and hands them off to the PCGIChannel for handling. PCGIServer can be configured with either a PCGI info file or by directly specifying the module, pid_file, and either port (for inet sockets) or socket_file (for unix domain sockets.) For inet sockets, the ip argument specifies the address from which the server will accept connections, '' indicates all addresses. If you only want to accept connections from the localhost, set ip to '127.0.0.1'.""" channel_class=PCGIChannel def __init__ (self, module='Main', ip='127.0.0.1', port=None, socket_file=None, pid_file=None, pcgi_file=None, resolver=None, logger_object=None): self.ip = ip asyncore.dispatcher.__init__(self) self.count=counter() if not logger_object: logger_object = logger.file_logger (sys.stdout) if resolver: self.logger = logger.resolving_logger (resolver, logger_object) else: self.logger = logger.unresolving_logger (logger_object) # get configuration self.module=module self.port=port self.pid_file=pid_file self.socket_file=socket_file if pcgi_file is not None: self.read_info(pcgi_file) # write pid file try: f = open(self.pid_file, 'w') f.write(str(os.getpid())) f.close() except IOError: self.log_info("Cannot write PID file.", 'error') # setup sockets if self.port: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.set_reuse_addr() self.bind((self.ip, self.port)) self.log_info( 'PCGI Server started at %s\n' '\tInet socket port: %s' % (time.ctime(time.time()), self.port) ) else: try: os.unlink(self.socket_file) except os.error: pass self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) self.set_reuse_addr() self.bind(self.socket_file) try: os.chmod(self.socket_file,0777) except os.error: pass self.log_info( 'PCGI Server started at %s\n' '\tUnix socket: %s' % (time.ctime(time.time()), self.socket_file) ) self.listen(256) def create_socket(self, family, type): asyncore.dispatcher.create_socket(self, family, type) requestCloseOnExec(self.socket) def read_info(self,info_file): "read configuration information from a PCGI info file" lines=open(info_file).readlines() directives={} try: for line in lines: line=string.strip(line) if not len(line) or line[0]=='#': continue k,v=string.split(line,'=',1) directives[string.strip(k)]=string.strip(v) except: raise ParseError, 'Error parsing PCGI info file' self.pid_file=directives.get('PCGI_PID_FILE',None) self.socket_file=directives.get('PCGI_SOCKET_FILE',None) if directives.has_key('PCGI_PORT'): self.port=string.atoi(directives['PCGI_PORT']) if directives.has_key('PCGI_MODULE'): self.module=directives['PCGI_MODULE'] elif directives.has_key('PCGI_MODULE_PATH'): path=directives['PCGI_MODULE_PATH'] path,module=os.path.split(path) module,ext=os.path.splitext(module) self.module=module def handle_accept (self): self.count.increment() try: conn, addr = self.accept() except socket.error: self.log_info('Server accept() threw an exception', 'warning') return self.channel_class(self, conn, addr) def readable(self): from ZServer import CONNECTION_LIMIT return len(asyncore.socket_map) < CONNECTION_LIMIT def writable (self): return 0 def listen(self, num): # override asyncore limits for nt's listen queue size self.accepting = 1 return self.socket.listen (num) class PCGIResponse(HTTPResponse): def write(self, data): if type(data) != type(''): raise TypeError('Value must be a string') if not self._wrote: self.stdout.write(str(self)) self._wrote=1 self.stdout.write(data) def _finish(self): self.stdout.finish(self) self.stdout.close() self.stdout=None self._request=None class PCGIPipe: """ Formats a HTTP response in PCGI format 10 digits indicating len of STDOUT STDOUT 10 digits indicating len of STDERR STDERR Note that this implementation never sends STDERR """ def __init__(self, channel): self._channel=channel self._data=StringIO() self._shutdown=0 def write(self,text): self._data.write(text) def close(self): if not self._channel.closed: data=self._data.getvalue() l=len(data) DebugLogger.log('A', id(self._channel), '%s %s' % (self._channel.reply_code, l)) self._channel.push('%010d%s%010d' % (l, data, 0), 0) self._channel.push(LoggingProducer(self._channel, l, 'log_request'), 0) self._channel.push(CallbackProducer( lambda t=('E', id(self._channel)): apply(DebugLogger.log,t)), 0) if self._shutdown: try: r=self._shutdown[0] except: r=0 ZServer.exit_code=r self._channel.push(ShutdownProducer(), 0) Wakeup(lambda: asyncore.close_all()) else: self._channel.push(None, 0) Wakeup() self._data=None self._channel=None def finish(self, response): if response._shutdownRequested(): self._shutdown = 1 self._channel.reply_code=response.status Zope-2.8.7-final/lib/python/ZServer/HTTPResponse.py0000664000076400007640000002660010435500320021073 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ ZServer HTTPResponse The HTTPResponse class takes care of server headers, response munging and logging duties. """ import time, re, sys, tempfile from cStringIO import StringIO import thread from ZPublisher.HTTPResponse import HTTPResponse from ZPublisher.Iterators import IStreamIterator from medusa.http_date import build_http_date from PubCore.ZEvent import Wakeup from medusa.producers import hooked_producer from medusa import http_server import asyncore from Producers import ShutdownProducer, LoggingProducer, CallbackProducer, \ file_part_producer, file_close_producer, iterator_producer from types import LongType import DebugLogger class ZServerHTTPResponse(HTTPResponse): "Used to push data into a channel's producer fifo" # Set this value to 1 if streaming output in # HTTP/1.1 should use chunked encoding http_chunk=1 http_chunk_size=1024 # defaults _http_version='1.0' _http_connection='close' _server_version='Zope/2.0 ZServer/2.0' # using streaming response _streaming=0 # using chunking transfer-encoding _chunking=0 _bodyproducer = None def __str__(self, html_search=re.compile('',re.I).search, ): if self._wrote: if self._chunking: return '0\r\n\r\n' else: return '' headers=self.headers body=self.body # set 204 (no content) status if 200 and response is empty # and not streaming if not headers.has_key('content-type') and \ not headers.has_key('content-length') and \ not self._streaming and \ self.status == 200: self.setStatus('nocontent') # add content length if not streaming if not headers.has_key('content-length') and \ not self._streaming: self.setHeader('content-length',len(body)) content_length= headers.get('content-length', None) if content_length>0 : self.setHeader('content-length', content_length) headersl=[] append=headersl.append status=headers.get('status', '200 OK') # status header must come first. append("HTTP/%s %s" % (self._http_version or '1.0' , status)) if headers.has_key('status'): del headers['status'] # add zserver headers append('Server: %s' % self._server_version) append('Date: %s' % build_http_date(time.time())) if self._http_version=='1.0': if self._http_connection=='keep-alive' and \ self.headers.has_key('content-length'): self.setHeader('Connection','Keep-Alive') else: self.setHeader('Connection','close') # Close the connection if we have been asked to. # Use chunking if streaming output. if self._http_version=='1.1': if self._http_connection=='close': self.setHeader('Connection','close') elif not self.headers.has_key('content-length'): if self.http_chunk and self._streaming: self.setHeader('Transfer-Encoding','chunked') self._chunking=1 else: self.setHeader('Connection','close') for key, val in headers.items(): if key.lower()==key: # only change non-literal header names key="%s%s" % (key[:1].upper(), key[1:]) start=0 l=key.find('-',start) while l >= start: key="%s-%s%s" % (key[:l],key[l+1:l+2].upper(),key[l+2:]) start=l+1 l=key.find('-',start) append("%s: %s" % (key, val)) if self.cookies: headersl=headersl+self._cookie_list() headersl[len(headersl):]=[self.accumulated_headers, body] return "\r\n".join(headersl) _tempfile=None _templock=None _tempstart=0 def write(self,data): """\ Return data as a stream HTML data may be returned using a stream-oriented interface. This allows the browser to display partial results while computation of a response to proceed. The published object should first set any output headers or cookies on the response object. Note that published objects must not generate any errors after beginning stream-oriented output. """ if type(data) != type(''): raise TypeError('Value must be a string') stdout=self.stdout if not self._wrote: l=self.headers.get('content-length', None) if l is not None: try: if type(l) is type(''): l=int(l) if l > 128000: self._tempfile=tempfile.TemporaryFile() self._templock=thread.allocate_lock() except: pass self._streaming=1 stdout.write(str(self)) self._wrote=1 if not data: return if self._chunking: data = '%x\r\n%s\r\n' % (len(data),data) l=len(data) t=self._tempfile if t is None or l<200: stdout.write(data) else: b=self._tempstart e=b+l self._templock.acquire() try: t.seek(b) t.write(data) finally: self._templock.release() self._tempstart=e stdout.write(file_part_producer(t,self._templock,b,e), l) _retried_response = None def _finish(self): if self._retried_response: try: self._retried_response._finish() finally: self._retried_response = None return stdout=self.stdout t=self._tempfile if t is not None: stdout.write(file_close_producer(t), 0) self._tempfile=None stdout.finish(self) stdout.close() self.stdout=None # need to break cycle? self._request=None def retry(self): """Return a request object to be used in a retry attempt """ # This implementation is a bit lame, because it assumes that # only stdout stderr were passed to the constructor. OTOH, I # think that that's all that is ever passed. response=self.__class__(stdout=self.stdout, stderr=self.stderr) response.headers=self.headers response._http_version=self._http_version response._http_connection=self._http_connection response._server_version=self._server_version self._retried_response = response return response def outputBody(self): """Output the response body""" self.stdout.write(str(self)) if self._bodyproducer: self.stdout.write(self._bodyproducer, 0) # we assign None to self._bodyproducer below to ensure that even # if self is part of a cycle which causes a leak that we # don't leak the bodyproducer (which often holds a reference to # an open file descriptor, and leaking file descriptors can have # particularly bad ramifications for a long-running process) self._bodyproducer = None def setBody(self, body, title='', is_error=0, **kw): """ Accept either a stream iterator or a string as the body """ if IStreamIterator.isImplementedBy(body): assert(self.headers.has_key('content-length')) # wrap the iterator up in a producer that medusa can understand self._bodyproducer = iterator_producer(body) HTTPResponse.setBody(self, '', title, is_error, **kw) return self else: HTTPResponse.setBody(self, body, title, is_error, **kw) class ChannelPipe: """Experimental pipe from ZPublisher to a ZServer Channel. Should only be used by one thread at a time. Note also that the channel will be being handled by another thread, thus restrict access to channel to the push method only.""" def __init__(self, request): self._channel=request.channel self._request=request self._shutdown=0 self._close=0 self._bytes=0 def write(self, text, l=None): if self._channel.closed: return if l is None: l=len(text) self._bytes=self._bytes + l self._channel.push(text,0) Wakeup() def close(self): DebugLogger.log('A', id(self._request), '%s %s' % (self._request.reply_code, self._bytes)) if not self._channel.closed: self._channel.push(LoggingProducer(self._request, self._bytes), 0) self._channel.push(CallbackProducer(self._channel.done), 0) self._channel.push(CallbackProducer( lambda t=('E', id(self._request)): apply(DebugLogger.log, t)), 0) if self._shutdown: self._channel.push(ShutdownProducer(), 0) Wakeup() else: if self._close: self._channel.push(None, 0) Wakeup() else: # channel closed too soon self._request.log(self._bytes) DebugLogger.log('E', id(self._request)) if self._shutdown: Wakeup(lambda: asyncore.close_all()) else: Wakeup() self._channel=None #need to break cycles? self._request=None def flush(self): pass # yeah, whatever def finish(self, response): if response._shutdownRequested(): self._shutdown = 1 if response.headers.get('connection','') == 'close' or \ response.headers.get('Connection','') == 'close': self._close=1 self._request.reply_code=response.status is_proxying_match = re.compile(r'[^ ]* [^ \\]*:').match proxying_connection_re = re.compile ('Proxy-Connection: (.*)', re.IGNORECASE) def make_response(request, headers): "Simple http response factory" # should this be integrated into the HTTPResponse constructor? response=ZServerHTTPResponse(stdout=ChannelPipe(request), stderr=StringIO()) response._http_version=request.version if request.version=='1.0' and is_proxying_match(request.request): # a request that was made as if this zope was an http 1.0 proxy. # that means we have to use some slightly different http # headers to manage persistent connections. connection_re = proxying_connection_re else: # a normal http request connection_re = http_server.CONNECTION response._http_connection = http_server.get_header(connection_re, request.header).lower() response._server_version=request.channel.server.SERVER_IDENT return response Zope-2.8.7-final/lib/python/ZServer/FTPResponse.py0000664000076400007640000000501110435500320020736 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Response class for the FTP Server. """ from ZServer.HTTPResponse import ZServerHTTPResponse from PubCore.ZEvent import Wakeup from cStringIO import StringIO import marshal class FTPResponse(ZServerHTTPResponse): """ Response to an FTP command """ def __str__(self): return '' def outputBody(self): pass def setCookie(self, name, value, **kw): self.cookies[name]=value def appendCookie(self, name, value): self.cookies[name]=self.cookies[name] + value def expireCookie(self, name, **kw): if self.cookies.has_key(name): del self.cookies[name] def _cookie_list(self): return [] def _marshalledBody(self): return marshal.loads(self.body) def setMessage(self, message): self._message = message def getMessage(self): return getattr(self, '_message', '') class CallbackPipe: """ Sends response object to a callback. Doesn't write anything. The callback takes place in Medusa's thread, not the request thread. """ def __init__(self, callback, args): self._callback=callback self._args=args self._producers=[] def close(self): pass def write(self, text, l=None): if text: self._producers.append(text) def finish(self, response): self._response=response Wakeup(self.apply) # move callback to medusas thread def apply(self): result=apply(self._callback, self._args+(self._response,)) # break cycles self._callback=None self._response=None self._args=None return result def make_response(channel, callback, *args): # XXX should this be the FTPResponse constructor instead? r=FTPResponse(stdout=CallbackPipe(callback, args), stderr=StringIO()) r.setHeader('content-type','text/plain') r.cookies=channel.cookies return r Zope-2.8.7-final/lib/python/ZServer/__init__.py0000664000076400007640000000372410435500320020336 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import sys import utils ######################################################### ### declarations used by external packages # the exit code used to exit a Zope process cleanly exit_code = 0 # the ZServer version number ZSERVER_VERSION = '1.1' # the maximum number of incoming connections to ZServer CONNECTION_LIMIT = 1000 # may be reset by max_listen_sockets handler in Zope # request bigger than this size get saved into a # temporary file instead of being read completely into memory LARGE_FILE_THRESHOLD = 1 << 19 # may be reset by large_file_threshold # handler in Zope # the Zope version string ZOPE_VERSION = utils.getZopeVersion() # backwards compatibility aliases from utils import requestCloseOnExec import asyncore from medusa import resolver, logger from HTTPServer import zhttp_server, zhttp_handler from PCGIServer import PCGIServer from FCGIServer import FCGIServer from FTPServer import FTPServer from PubCore import setNumberOfThreads from medusa.monitor import secure_monitor_server ### end declarations ########################################################## # we need to patch asyncore's dispatcher class with a new # log_info method so we see medusa messages in the zLOG log utils.patchAsyncoreLogger() # we need to patch the 'service name' of the medusa syslog logger utils.patchSyslogServiceName() Zope-2.8.7-final/lib/python/ZServer/README.txt0000664000076400007640000002057010435500320017721 0ustar ajungajungZServer README -------------- What is ZServer? ZServer is an integration of the Zope application server and the Medusa information server. See the ZServer architecture document for more information:: http://www.zope.org/Documentation/Reference/ZServer ZServer gives you HTTP, FTP, WebDAV, PCGI, and remote interactive Python access. In later releases it will probably offer more protocols such as FastCGI, etc. What is Medusa? Medusa is a Python server framework with uses a single threaded asynchronous sockets approach. For more information see:: http://www.nightmare.com/medusa There's also an interesting Medusa tutorial at:: http://www.nightmare.com:8080/nm/apps/medusa/docs/programming.html ZServer HTTP support ZServer offers HTTP 1.1 publishing for Zope. It does not support publishing files from the file system. You can specify the HTTP port using the -w command line argument for the z2.py start script. You can also specify CGI environment variables on the command line using z2.py ZServer FTP support What you can do with FTP FTP access to Zope allows you to FTP to the Zope object hierarchy in order to perform managerial tasks. You can: * Navigate the object hierarchy with 'cd' * Replace the content of Documents, Images, and Files * Create Documents, Images, Files, Folders * Delete objects and Folders. So basically you can do more than is possible with HTTP PUT. Also, unlike PUT, FTP gives you access to Document content. So when you download a Document you are getting its content, not what it looks like when it is rendered. Using FTP To FTP into Zope, ZServer must be configured to serve FTP. By default ZServer serves FTP on port 9221. So to connect to Zope you would issue a command like so:: $ ftp localhost 9221 When logging in to FTP, you have some choices. You can connect anonymously by using a username of 'anonymous' and any password. Or you can login as a Zope user. Since Zope users are defined at different locations in the object hierarchy, authentication can be problematic. There are two solutions: * login and then cd to the directory where you are defined. * login with a special name that indicates where you are defined. The format of the special name is '@'. For example:: joe@Marketing/Projects FTP permissions FTP support is provided for Folders, Documents, Images, and Files. You can control access to FTP via the new 'FTP access' permission. This permission controls the ability to 'cd' to a Folder and to download objects. Uploading and deleting and creating objects are controlled by existing permissions. FTP limits You can set limits for the number of simultaneous FTP connections. You can separately configure the number of anonymous and authenticated connections. Right now this setting is set in 'ZServerFTP.py'. In the future, it may be more easy to configure. Properties and FTP: The next step The next phase of FTP support will allow you to edit properties of all Zope objects. Probably properties will be exposed via special files which will contain an XML representation of the object's properties. You could then download the file, edit the XML and upload it to change the object's properties. We do not currently have a target date for FTP property support. How does FTP work? The ZServer's FTP channel object translates FTP requests into ZPublisher requests. The FTP channel then analyses the response and formulates an appropriate FTP response. The FTP channel stores some state such as the current working directory and the username and password. On the Zope side of things, the 'lib/python/OFS/FTPInterface.py' module defines the Zope FTP interface, for listing sub-items, stating, and getting content. The interface is implemented in 'SimpleItem', and in other Zope classes. Programmers will not need to implement the entire interface if they inherit from 'SimpleItem'. All the other FTP functions are handled by existing methods like 'manage_delObjects', and 'PUT', etc. ZServer PCGI support ZServer will service PCGI requests with both inet and unix domain sockets. This means you can use ZServer instead of 'pcgi_publisher.py' as your long running PCGI server process. In the future, PCGI may be able to activate ZServer. Using PCGI instead of HTTP allows you to forward requests from another web server to ZServer. The CGI environment and HTTP headers are controlled by the web server, so you don't need to worry about managing the ZServer environment. However, this configuration will impose a larger overhead than simply using the web server as an HTTP proxy for ZServer. To use PCGI, configure your PCGI info files to communicate with ZServer by setting the PCGI_PORT, PCGI_SOCKET_FILE, and PCGI_NAME. The other PCGI settings are currently ignored by ZServer. ZServer's PCGI support will work with mod_pcgi. ZServer monitor server ZServer now includes the Medusa monitor server. This basically gives you a remote, secure Python prompt. You can interactively access Zope. This is a very powerful, but dangerous tool. Be careful. To use the monitor server specify a monitor port number using the -m option with the z2.py start script. The default port is 9999. To connect to the monitor server use the 'ZServer/medusa/monitor_client.py' or 'ZServer/medusa/monitor_client_win32.py' script. For example:: $ python2.1 ZServer/medusa/monitor_client.py localhost 9999 You will then be asked to enter a password. This is the Zope super manager password which is stored in the 'access' file. Then you will be greeted with a Python prompt. To access Zope import the Zope module:: >>> import Zope The Zope top level Zope object is available via the 'Zope.app' function:: >>> a=Zope.app() From this object you can reach all other Zope objects as subobjects. Remember if you make changes to Zope objects and want those changes to be saved you need to commmit the transaction:: >>> import transaction >>> transaction.commit() ZServer WebDAV support WebDAV is a new protocol for managing web resources. WebDAV operates over HTTP. Since WebDAV uses HTTP, ZServer doesn't really have to do anything special, except stay out of Zope's way when handling WebDAV requests. The only major WebDAV client at this time is Internet Explorer 5. It works with Zope. Differences between ZopeHTTPServer and ZServer ZopeHTTPServer is old and no longer being actively maintained. Both ZopeHTTPServer and ZServer are Python HTTP servers. ZopeHTTPServer is built on the standard Python SimpleHTTPServer framework. ZServer is built on Medusa. ZopeHTTPServer is very limited. It can only publish one module at a time. It can only publish via HTTP. It has no support for thread pools. ZServer on the other hand is more complex and supports publishing multiple modules, thread pools, and it uses a new threaded architecture for accessing ZPublisher. Running ZServer as nobody Normally ZServer will run with the userid of the user who starts it. However, if ZServer is started by root, it will attempt to become nobody or any userid you specify with the -u argument to the z2.py start script. ZServer is similar to ZopeHTTPServer in these respects. If you run Zope with different userids you must be aware of permission issues. Zope must be able to read and write to the 'var' directory. If you change the userid Zope is running under you will probably need to change the permissions on the 'var' directory and the files in it in order for Zope to run under a different userid. Support Questions and comments should go to 'support@digicool.com'. You can report bugs and check on the status of bugs using the Zope bug collector:: http://www.zope.org/Resources/Collector/ License ZServer is covered by the ZPL despite the fact that it comes with much of the Medusa source code. The portions of Medusa that come with ZServer are licensed under the ZPL. Outstanding issues The FTP interface for Zope objects may be changed. HTTP 1.1 support is ZServer is incomplete, though it should work for most HTTP 1.1 clients. Zope-2.8.7-final/lib/python/ZServer/utils.py0000664000076400007640000000410410435500320017730 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ A set of utility routines used by asyncore initialization """ def getZopeVersion(): import App.version_txt return App.version_txt.version_txt() def patchAsyncoreLogger(): # Poke zLOG default logging into asyncore to send # messages to zLOG instead of medusa logger from zLOG import LOG, register_subsystem, BLATHER, INFO, WARNING, ERROR register_subsystem('ZServer') severity={'info':INFO, 'warning':WARNING, 'error': ERROR} def log_info(self, message, type='info'): if message[:14]=='adding channel' or \ message[:15]=='closing channel' or \ message == 'Computing default hostname': LOG('ZServer', BLATHER, message) else: LOG('ZServer', severity[type], message) import asyncore asyncore.dispatcher.log_info=log_info # A routine to try to arrange for request sockets to be closed # on exec. This makes it easier for folks who spawn long running # processes from Zope code. Thanks to Dieter Maurer for this. try: import fcntl def requestCloseOnExec(sock): try: fcntl.fcntl(sock.fileno(), fcntl.F_SETFD, fcntl.FD_CLOEXEC) except: # XXX What was this supposed to catch? pass except (ImportError, AttributeError): def requestCloseOnExec(sock): pass def patchSyslogServiceName(): from medusa import logger # override the service name in logger.syslog_logger logger.syslog_logger.svc_name='ZServer' Zope-2.8.7-final/lib/python/ZServer/DebugLogger.py0000664000076400007640000000337310435500320020765 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Logs debugging information about how ZServer is handling requests and responses. This log can be used to help locate troublesome requests. The format of a log message is:

\n') self.in_document_title = len(self.body) elif isinstance(node.parent, nodes.section): tag = 'h%s' % (self.section_level + self.initial_header_level - 1) self.body.append( self.starttag(node, tag, '', CLASS='section-subtitle') + self.starttag({}, 'span', '', CLASS='section-subtitle')) self.context.append('\n' % tag) def depart_subtitle(self, node): self.body.append(self.context.pop()) if self.in_document_title: self.subtitle = self.body[self.in_document_title:-1] self.in_document_title = 0 self.body_pre_docinfo.extend(self.body) self.html_subtitle.extend(self.body) del self.body[:] def visit_superscript(self, node): self.body.append(self.starttag(node, 'sup', '')) def depart_superscript(self, node): self.body.append('') def visit_system_message(self, node): if node['level'] < self.document.reporter.report_level: # Level is too low to display: raise nodes.SkipNode self.body.append(self.starttag(node, 'div', CLASS='system-message')) self.body.append('

') attr = {} backref_text = '' if node['ids']: attr['name'] = node['ids'][0] if len(node['backrefs']): backrefs = node['backrefs'] if len(backrefs) == 1: backref_text = ('; backlink' % backrefs[0]) else: i = 1 backlinks = [] for backref in backrefs: backlinks.append('%s' % (backref, i)) i += 1 backref_text = ('; backlinks: %s' % ', '.join(backlinks)) if node.hasattr('line'): line = ', line %s' % node['line'] else: line = '' if attr: a_start = self.starttag({}, 'a', '', **attr) a_end = '' else: a_start = a_end = '' self.body.append('System Message: %s%s/%s%s ' '(%s%s)%s

\n' % (a_start, node['type'], node['level'], a_end, self.encode(node['source']), line, backref_text)) def depart_system_message(self, node): self.body.append('\n') def visit_table(self, node): self.body.append( self.starttag(node, 'table', CLASS='docutils', border="1")) def depart_table(self, node): self.body.append('\n') def visit_target(self, node): if not (node.has_key('refuri') or node.has_key('refid') or node.has_key('refname')): self.body.append(self.starttag(node, 'span', '', CLASS='target')) self.context.append('') else: self.context.append('') def depart_target(self, node): self.body.append(self.context.pop()) def visit_tbody(self, node): self.write_colspecs() self.body.append(self.context.pop()) # '\n' or '' self.body.append(self.starttag(node, 'tbody', valign='top')) def depart_tbody(self, node): self.body.append('\n') def visit_term(self, node): self.body.append(self.starttag(node, 'dt', '')) def depart_term(self, node): """ Leave the end tag to `self.visit_definition()`, in case there's a classifier. """ pass def visit_tgroup(self, node): # Mozilla needs : self.body.append(self.starttag(node, 'colgroup')) # Appended by thead or tbody: self.context.append('\n') node.stubs = [] def depart_tgroup(self, node): pass def visit_thead(self, node): self.write_colspecs() self.body.append(self.context.pop()) # '\n' # There may or may not be a ; this is for to use: self.context.append('') self.body.append(self.starttag(node, 'thead', valign='bottom')) def depart_thead(self, node): self.body.append('\n') def visit_tip(self, node): self.visit_admonition(node, 'tip') def depart_tip(self, node): self.depart_admonition() def visit_title(self, node): """Only 6 section levels are supported by HTML.""" check_id = 0 close_tag = '

\n' if isinstance(node.parent, nodes.topic): self.body.append( self.starttag(node, 'p', '', CLASS='topic-title first')) check_id = 1 elif isinstance(node.parent, nodes.sidebar): self.body.append( self.starttag(node, 'p', '', CLASS='sidebar-title')) check_id = 1 elif isinstance(node.parent, nodes.Admonition): self.body.append( self.starttag(node, 'p', '', CLASS='admonition-title')) check_id = 1 elif isinstance(node.parent, nodes.table): self.body.append( self.starttag(node, 'caption', '')) check_id = 1 close_tag = '\n' elif self.section_level == 0: assert node.parent is self.document # document title self.head.append('%s\n' % self.encode(node.astext())) self.body.append(self.starttag(node, 'h1', '', CLASS='title')) self.context.append('\n') self.in_document_title = len(self.body) else: assert isinstance(node.parent, nodes.section) h_level = self.section_level + self.initial_header_level - 1 atts = {} if (len(node.parent) >= 2 and isinstance(node.parent[1], nodes.subtitle)): atts['CLASS'] = 'with-subtitle' self.body.append( self.starttag(node, 'h%s' % h_level, '', **atts)) atts = {} if node.parent['ids']: atts['name'] = node.parent['ids'][0] if node.hasattr('refid'): atts['class'] = 'toc-backref' atts['href'] = '#' + node['refid'] self.body.append(self.starttag({}, 'a', '', **atts)) self.context.append('\n' % (h_level)) if check_id: if node.parent['ids']: self.body.append( self.starttag({}, 'a', '', name=node.parent['ids'][0])) self.context.append('' + close_tag) else: self.context.append(close_tag) def depart_title(self, node): self.body.append(self.context.pop()) if self.in_document_title: self.title = self.body[self.in_document_title:-1] self.in_document_title = 0 self.body_pre_docinfo.extend(self.body) self.html_title.extend(self.body) del self.body[:] def visit_title_reference(self, node): self.body.append(self.starttag(node, 'cite', '')) def depart_title_reference(self, node): self.body.append('') def visit_topic(self, node): self.body.append(self.starttag(node, 'div', CLASS='topic')) self.topic_classes = node['classes'] def depart_topic(self, node): self.body.append('\n') self.topic_classes = [] def visit_transition(self, node): self.body.append(self.emptytag(node, 'hr', CLASS='docutils')) def depart_transition(self, node): pass def visit_version(self, node): self.visit_docinfo_item(node, 'version', meta=None) def depart_version(self, node): self.depart_docinfo_item() def visit_warning(self, node): self.visit_admonition(node, 'warning') def depart_warning(self, node): self.depart_admonition() def unimplemented_visit(self, node): raise NotImplementedError('visiting unimplemented node type: %s' % node.__class__.__name__) class SimpleListChecker(nodes.GenericNodeVisitor): """ Raise `nodes.NodeFound` if non-simple list item is encountered. Here "simple" means a list item containing nothing other than a single paragraph, a simple list, or a paragraph followed by a simple list. """ def default_visit(self, node): raise nodes.NodeFound def visit_bullet_list(self, node): pass def visit_enumerated_list(self, node): pass def visit_list_item(self, node): children = [] for child in node.children: if not isinstance(child, nodes.Invisible): children.append(child) if (children and isinstance(children[0], nodes.paragraph) and (isinstance(children[-1], nodes.bullet_list) or isinstance(children[-1], nodes.enumerated_list))): children.pop() if len(children) <= 1: return else: raise nodes.NodeFound def visit_paragraph(self, node): raise nodes.SkipNode def invisible_visit(self, node): """Invisible nodes should be ignored.""" raise nodes.SkipNode visit_comment = invisible_visit visit_substitution_definition = invisible_visit visit_target = invisible_visit visit_pending = invisible_visit Zope-2.8.7-final/lib/python/docutils/writers/null.py0000664000076400007640000000075210435500317021502 0ustar ajungajung# Author: David Goodger # Contact: goodger@python.org # Revision: $Revision: 3246 $ # Date: $Date: 2005-04-23 21:23:21 +0200 (Sat, 23 Apr 2005) $ # Copyright: This module has been placed in the public domain. """ A do-nothing Writer. """ from docutils import writers class Writer(writers.Writer): supported = ('null',) """Formats this writer supports.""" config_section = 'null writer' config_section_dependencies = ('writers',) def translate(self): pass Zope-2.8.7-final/lib/python/docutils/HISTORY.txt0000664000076400007640000017624210435500317020371 0ustar ajungajung================== Docutils History ================== :Author: David Goodger; open to all Docutils developers :Contact: goodger@python.org :Date: $Date: 2005/01/07 15:11:43 $ :Web site: http://docutils.sourceforge.net/ :Copyright: This document has been placed in the public domain. .. contents:: Release 0.3.7 (2004-12-24) ========================== * docutils/frontend.py: - Added options: --input-encoding-error-handler, --record-dependencies, --leave-footnote-reference-space, --strict-visitor. - Added command-line and config file support for "overrides" setting parameter. * docutils/io.py: - Added support for input encoding error handler. * docutils/nodes.py: - Added dispatch_visit and dispatch_departure methods to NodeVisitor; useful as a hook for Visitors. - Changed structure of ``line_block``; added ``line``. - Added ``compound`` node class. - Added a mechanism for Visitors to transitionally ignore new node classes. * docutils/utils.py: - Moved ``escape2null`` and ``unescape`` functions from docutils/parsers/rst/states.py. * docutils/parsers/rst/roles.py: - Added "raw" role. - Changed role function API: the "text" parameter now takes null-escaped interpreted text content. * docutils/parsers/rst/states.py: - Fixed bug where a "role" directive in a nested parse would crash the parser; the state machine's "language" attribute was not being copied over. - Added support for line block syntax. - Fixed directive parsing bug: argument-less directives didn't notice that arguments were present. - Removed error checking for transitions. - Added support for multiple classifiers in definition list items. - Moved ``escape2null`` and ``unescape`` functions to docutils/utils.py. - Changed role function API: the "text" parameter now takes null-escaped interpreted text content. - Empty sections and documents are allowed now. * docutils/parsers/rst/directives/__init__.py: - Added ``encoding`` directive option conversion function. - Allow multiple class names in class_option conversion function. * docutils/parsers/rst/directives/body.py: - Converted the line-block directive to use the new structure. - Extracted the old line-block functionality to the ``block`` function (still used). - Added ``compound`` directive (thanks to Felix Wiemann). * docutils/parsers/rst/directives/misc.py: - Added "encoding" option to "include" and "raw" directives. - Added "trim", "ltrim", and "rtrim" options to "unicode" directive. - Allow multiple class names in the "class" directive. * docutils/parsers/rst/directives/parts.py: - Directive "sectnum" now accepts "prefix", "suffix", and "start" options. Thanks to Lele Gaifax. * docutils/parsers/rst/directives/tables.py: - Added "encoding" directive to "csv-table" directive. - Added workaround for lack of Unicode support in csv.py, for non-ASCII CSV input. * docutils/transforms/misc.py: - Fixed bug when multiple "class" directives are applied to a single element. - Enabled multiple format names for "raw" directive. * docutils/transforms/references.py: - Added support for trimming whitespace from beside substitution references. * docutils/transforms/universal.py: - FinalChecks now checks for illegal transitions and moves transitions between sections. * docutils/writers/html4css1.py: - HTMLTranslator.encode now converts U+00A0 to " ". - "stylesheet" and "stylesheet_path" settings are now mutually exclusive. - Added support for the new line_block/line structure. - --footnote-references now overrides --trim-footnote-reference-space, if applicable. - Added support for ``compound`` elements. - Enabled multiple format names for "raw" directive. - ``

`` tags of a paragraph which is the only visible child of the document node are no longer stripped. - Moved paragraph-compacting logic (for stripping ``

`` tags) to new method ``should_be_compact_paragraph()``. - Added class="docutils" to ``dl``, ``hr``, ``table`` and ``tt`` elements. - "raw" elements are now surrounded by ``span`` or ``div`` tags in the output if they have their ``class`` attribute set. - The whole document is now surrounded by a ``

`` element. - Body-level images are now wrapped by their own ``
`` elements, with image classes copied to the wrapper, and for images which have the ``:align:`` option set, the surrounding ``
`` now receives a class attribute (like ``class="align-left"``). * docutils/writers/latex2e.py: - no newline after depart_term. - Added translations for some Unicode quotes. - Added option "font-encoding", made package AE the default. - "stylesheet" and "stylesheet_path" settings are now mutually exclusive. - --footnote-references now overrides --trim-footnote-reference-space, if applicable. - The footnote label style now matches the footnote reference style ("brackets" or "superscript"). - Added support for ``compound`` elements. - Enabled multiple format names for "raw" directive. * docs/ref/docutils.dtd: - Changed structure of the ``line_block`` element; added ``line``. - Added ``compound`` element. - Added "ltrim" and "rtrim" attributes to ``substitution_definition`` element. - Enabled multiple format names for ``raw`` element. - Enabled multiple classifiers in ``definition_list_item`` elements. * docs/ref/rst/directives.txt - Marked "line-block" as deprecated. - "Class" directive now allows multiple class names. - Added "Rationale for Class Attribute Value Conversion". - Added warning about "raw" overuse/abuse. * docs/ref/rst/restructuredtext.txt: - Added syntax for line blocks. - Definition list items may have multiple classifiers. * docs/ref/rst/roles.txt: - Added "raw" role. * tools/stylesheets/default.css: - Added support for the new line_block structure. - Added "docutils" class to ``dl``, ``hr``, ``table`` and ``tt``. Release 0.3.5 (2004-07-29) ========================== General: * _`Documentation cleanup/reorganization`. - Created new subdirectories of docs/: * ``docs/user/``: introductory/tutorial material for end-users * ``docs/dev/``: for core-developers (development notes, plans, etc.) * ``docs/api/``: API reference material for client-developers * ``docs/ref/``: reference material for all groups * ``docs/howto/``: for component-developers and core-developers * ``docs/peps/``: Python Enhancement Proposals - Moved ``docs/*`` to ``docs/user/``. - Moved ``pysource.dtd``, ``pysource.txt``, ``semantics.txt`` from ``spec/`` to ``docs/dev``. - Moved ``doctree.txt``, ``docutils.dtd``, ``soextblx.dtd``, ``transforms.txt`` from ``spec/`` to ``docs/ref/``. - Moved ``alternatives.txt``, and ``problems.txt`` from ``spec/rst/`` to ``docs/dev/rst/``. - Moved ``reStructuredText.txt``, ``directives.txt``, ``interpreted.txt``, and ``introduction.txt`` from ``spec/rst/`` to ``docs/ref/rst/``. Renamed ``interpreted.txt`` to ``roles.txt``, ``reStructuredText.txt`` to ``restructuredtext.txt``. - Moved ``spec/howto/`` to ``docs/howto``. In order to keep the CVS history of moved files, we supplied SourceForge with a `script for modifying the Docutils CVS repository`__. __ http://cvs.sourceforge.net/viewcvs.py/*checkout*/docutils/sandbox/davidg/infrastructure/cvs-reorg.sh?content-type=text/plain&rev=1.5 After running the cleanup script: - Added ``docs/index.txt``. - Added a ``.htaccess`` file to the ``web`` module, containing redirects for all old paths to new paths. They'll preserve fragments (the "#name" part of a URL), and won't clutter up the file system, and will correct the URL in the user's browser. - Added ``BUGS.txt``, ``docs/dev/policies.txt``, ``docs/dev/website.txt``, ``docs/dev/release.txt`` from all but the "To Do" list itself in ``docs/dev/todo.txt``. - Moved "Future Plans" from ``HISTORY.txt`` to new "Priorities" section of ``docs/dev/todo.txt``. - Added ``THANKS.txt`` from "Acknowledgements" in ``HISTORY.txt``. - Added "How To Report Bugs" to ``BUGS.txt``. - Went through all the sources and docs (including under web/) and updated links. Mostly done by Felix Wiemann; thanks Felix! (Still need to update links in the sandboxes.) Specific: * BUGS.txt: Added to project. * THANKS.txt: Added to project. * docutils/__init__.py: - 0.3.4: Post-release. * docutils/core.py: - Added special error handling & advice for UnicodeEncodeError. - Refactored Publisher.publish (simplified exception handling & extracted debug dumps). - Renamed "enable_exit" parameter of convenience functions to "enable_exit_status". - Enabled traceback (exception propagation) by default in programmatic convenience functions. - Now publish_file and publish_cmdline convenience functions return the encoded string results in addition to their regular I/O. - Extracted common code from publish_file, publish_string, and publish_parts, into new publish_programmatically. Extracted settings code to ``Publisher.process_programmatic_settings``. - In Publisher.publish, disabled ``settings_overrides`` when ``settings`` is supplied; redundant. * docutils/frontend.py: - Added help text for "--output-encoding-error-handler" and "--error-encoding-error-handler". - Renamed "--exit" to "--exit-status". - Simplified default-setting code. * docutils/parsers/rst/__init__.py: - Added "--pep-base-url" and "--rfc-base-url" options. * docutils/parsers/rst/states.py: - Made URI recognition more aggressive and intelligent. * docutils/parsers/rst/directives/__init__.py: - Added several directive option conversion functions. * docutils/parsers/rst/directives/body.py: - Moved "table" directive to tables.py. * docutils/parsers/rst/directives/tables.py: Table-related directives, added to project. * docutils/writers/latex2e.py: - Added "--table-style=(standard|booktabs|nolines)" - figures get "here" option (LaTeX per default puts them at bottom), and figure content is centered. - Rowspan support for tables. - Fix: admonition titles before first section. - Replace ``--`` in literal by ``-{}-`` because fontencoding T1 has endash. - Replave ``_`` in literal by an underlined blank, because it has the correct width. - Fix: encode pdfbookmark titles, ``#`` broke pdflatex. - A few unicode replacements, if output_encoding != utf - Add "--graphicx-option". - Indent literal-blocks. - Fix: omit ``\maketitle`` when there is no document title. * docs/index.txt: "Docutils Project Documentation Overview", added to project. * docs/api/cmdline-tool.txt: "Inside A Docutils Command-Line Front-End Tool", added to project. * docs/api/publisher.txt: "The Docutils Publisher", added to project. * docs/api/runtime-settings.txt: "Docutils Runtime Settings", added to project. * docs/dev/policies.txt: Added to project (extracted from ``docs/dev/todo.txt``, formerly ``spec/notes.txt``). * docs/dev/release.txt: Added to project (extracted from ``docs/dev/todo.txt``, formerly ``spec/notes.txt``). * docs/dev/testing.txt: Added to project. * docs/dev/website.txt: Added to project (extracted from ``docs/dev/todo.txt``, formerly ``spec/notes.txt``). * docs/ref/rst/directives.txt: - Added directives: "table", "csv-table". * docs/user/rst/cheatsheet.txt: "The reStructuredText Cheat Sheet" added to project. 1 page for syntax, and a 1 page reference for directives and roles. Source text to be used as-is; not meant to be converted to HTML. * docs/user/rst/demo.txt: Added to project; moved from tools/test.txt with a change of title. * test/functional/, contents, and test/test_functional.py: Added to project. * tools/buildhtml.py: Fixed bug with config file handling. * tools/html.py: Removed from project (duplicate of rst2html.py). * tools/pep2html.py: Removed from project (duplicate of Python's nondist/peps/pep2html.py; Docutils' tools/pep.py can be used for Docutils-related PEPs in docs/peps/). * tools/rst2pseudoxml.py: Renamed from publish.py. * tools/rst2xml.py: Renamed from docutils-xml.py. * tools/test.txt: Removed from project; moved to docs/user/rst/demo.txt. * setup.py: Now also installs ``rst2latex.py``. Release 0.3.3 (2004-05-09) ========================== * docutils/__init__.py: - 0.3.1: Reorganized config file format (multiple sections); see docs/config.txt. - Added unknown_reference_resolvers attribute to TransformSpec. - 0.3.2: Interpreted text reorganization. - 0.3.3: Released. * docutils/core.py: - Catch system messages to stop tracebacks from parsing errors. - Catch exceptions during processing report & exit without tracebacks, except when "--traceback" used. - Reordered components for OptionParser; application comes last. - Added "config_section" parameter to several methods and functions, allowing front ends to easily specify their config file sections. - Added publish_parts convenience function to allow access to individual parts of a document. * docutils/examples.py: Added to project; practical examples of Docutils client code, to be used as-is or as models for variations. * docutils/frontend.py: - Added "--traceback" & "--no-traceback" options ("traceback" setting). - Implemented support for config file reorganization: ``standard_config_files`` moved from ``ConfigParser`` to ``OptionParser``; added ``OptionParser.get_config_file_settings()`` and ``.get_standard_config_settings()``; support for old "[options]" section (with deprecation warning) and mapping from old to new settings. - Reimplemented setting validation. - Enabled flexible boolean values: yes/no, true/false, on/off. - Added ``Values``, a subclass of ``optparse.Values``, with support for list setting attributes. - Added support for new ``DOCUTILSCONFIG`` environment variable; thanks to Beni Cherniavsky. - Added "--no-section-numbering" option. * docutils/io.py: - Catch IOErrors when opening source & destination files, report & exit without tracebacks. Added ``handle_io_errors`` parameter to ``FileInput`` & ``FileOutput`` to enable caller error handling. * docutils/nodes.py: - Changed ``SparseNodeVisitor`` and ``GenericNodeVisitor`` dynamic method definitions (via ``exec``) to dynamic assignments (via ``setattr``); thanks to Roman Suzi. - Encapsulated visitor dynamic assignments in a function; thanks to Ian Bicking. - Added indirect_reference_name attribute to the Targetable class. This attribute holds the whitespace_normalized_name (contains mixed case) of a target. * docutils/statemachine.py: - Renamed ``StringList.strip_indent`` to ``.trim_left``. - Added ``StringList.get_2D_block``. * docutils/utils.py: - Added "level" attribute to SystemMessage exceptions. * docutils/languages/af.py: Added to project; Afrikaans mappings by Jannie Hofmeyr. * docutils/languages/cs.py: Added to project; Czech mappings by Marek Blaha. * docutils/languages/eo.py: Added to project; Esperanto mappings by Marcelo Huerta San Martin. * docutils/languages/pt_br.py: Added to project; Brazilian Portuguese mappings by Lalo Martins. * docutils/languages/ru.py: Added to project; Russian mappings by Roman Suzi. * docutils/parsers/rst/roles.py: Added to project. Contains interpreted text role functions, a registry for interpreted text roles, and an API for adding to and retrieving from the registry. Contributed by Edward Loper. * docutils/parsers/rst/states.py: - Updated ``RSTState.nested_parse`` for "include" in table cells. - Allowed true em-dash character and "---" as block quote attribution marker. - Added support for complex option arguments (option lists). - Fixed handling of backslashes in substitution definitions. - Fixed off-by-1 error with extra whitespace after substitution definition directive. - Added inline markup parsing to field lists' field names. - Added support for quoted (and unindented) literal blocks. Driven in part by a bribe from Frank Siebenlist (thanks!). - Parser now handles escapes in URIs correctly. - Made embedded-URIs' reference text omittable. Idea from Beni Cherniavsky. - Refactored explicit target processing code. - Added name attribute to references containing the reference name only through whitespace_normalize_name (no case changes). - parse_target no longer returns the refname after going through normalize_name. This is now handled in make_target. - Fixed bug relating to role-less interpreted text in non-English contexts. - Reorganized interpreted text processing; moved code into the new roles.py module. Contributed by Edward Loper. - Refactored ``Body.parse_directive`` into ``run_directive`` and ``parse_directive_block``. * docutils/parsers/rst/tableparser.py: - Reworked for ``StringList``, to support "include" directives in table cells. * docutils/parsers/rst/directives/__init__.py: - Renamed ``unchanged()`` directive option conversion function to ``unchanged_required``, and added a new ``unchanged``. - Catch unicode value too high error; fixes bug 781766. - Beefed up directive error reporting. * docutils/parsers/rst/directives/body.py: - Added basic "table" directive. * docutils/parsers/rst/directives/images.py: - Added "target" option to "image" directive. - Added name attribute to references containing the reference name only through whitespace_normalize_name (no case changes). * docutils/parsers/rst/directives/misc.py: - Isolated the import of the ``urllib2`` module; was causing problems on SourceForge (``libssl.so.2`` unavailable?). - Added the "role" directive for declaring custom interpreted text roles. * docutils/parsers/rst/directives/parts.py: - The "contents" directive does more work up-front, creating the "topic" and "title", and leaving the "pending" node for the transform. Allows earlier reference resolution; fixes subtle bug. * docutils/parsers/rst/languages/af.py: Added to project; Afrikaans mappings by Jannie Hofmeyr. * docutils/parsers/rst/languages/cs.py: Added to project; Czech mappings by Marek Blaha. * docutils/parsers/rst/languages/eo.py: Added to project; Esperanto mappings by Marcelo Huerta San Martin. * docutils/parsers/rst/languages/pt_br.py: Added to project; Brazilian Portuguese mappings by Lalo Martins. * docutils/parsers/rst/languages/ru.py: Added to project; Russian mappings by Roman Suzi. * docutils/transforms/parts.py: - The "contents" directive does more work up-front, creating the "topic" and "title", and leaving the "pending" node for the transform. Allows earlier reference resolution; fixes subtle bug. - Added support for disabling of section numbering. * docutils/transforms/references.py: - Verifying that external targets are truly targets and not indirect references. This is because we are now adding a "name" attribute to references in addition to targets. Note sure if this is correct! - Added code to hook into the unknown_reference_resolvers list for a transformer in resolve_indirect_target. This allows the unknown_reference_resolvers to keep around indirect targets which docutils doesn't know about. - Added specific error message for duplicate targets. * docutils/transforms/universal.py: - Added FilterMessages transform (removes system messages below the verbosity threshold). - Added hook (via docutils.TransformSpec.unknown_reference_resolvers) to FinalCheckVisitor for application-specific handling of unresolvable references. - Added specific error message for duplicate targets. * docutils/writers/__init__.py: - Added assemble_parts method to the Writer class to allow for access to a documents individual parts. - Documented & set default for ``Writer.output`` attribute. * docutils/writers/html4css1.py: - Fixed unicode handling of attribute values (bug 760673). - Prevent duplication of "class" attribute values (bug report from Kirill Lapshin). - Improved table grid/border handling (prompted by report from Bob Marshall). - Added support for table titles. - Added "" for untitled docs, for XHTML conformance; thanks to Darek Suchojad. - Added functionality to keep track of individual parts of a document and store them in a dictionary as the "parts" attribute of the writer. Contributed by Reggie Dugard at the Docutils sprint at PyCon DC 2004. - Added proper support for the "scale" attribute of the "image" element. Contributed by Brent Cook. - Added ``--initial-header-level`` option. - Fixed bug: the body_pre_docinfo segment depended on there being a docinfo; if no docinfo, the document title was incorporated into the body segment. Adversely affected the publish_parts interface. * docutils/writers/latex2e.py: - Changed default stylesheet to "no stylesheet" to avoid latex complaining about a missing file. - Added options and support: ``--compound-enumerators``, ``--section-prefix-for-enumerators``, and ``--section-enumerator-separator``. By John F Meinel Jr (SF patch 934322). - Added option ``--use-verbatim-when-possible``, to avoid problematic characters (eg, '~' in italian) in literal blocks. - It's now possible to use four section levels in the `book` and `report` LaTeX classes. The default `article` class still has three levels limit. * docs/config.txt: "Docutils Configuration Files", added to project. Moved config file entry descriptions from tools.txt. * docs/tools.txt: - Moved config file entry descriptions to config.txt. * spec/notes.txt: Continual updates. Added "Setting Up For Docutils Development". * spec/howto/rst-roles.txt: "Creating reStructuredText Interpreted Text Roles", added to project. * spec/rst/reStructuredText.txt: - Added description of support for <angle-bracketed> complex option arguments to option lists. - Added subsections for indented and quoted literal blocks. * test: Continually adding & updating tests. - Added test/test_settings.py & test/data/config_*.txt support files. - Added test/test_writers/test_htmlfragment.py. * test/DocutilsTestSupport.py: - Refactored LaTeX publisher test suite/case class names to make testing other writers easier. - Added HtmlWriterPublishTestCase and HtmlFragmentTestSuite classes to test the processing of HTML fragments which use the new publish_parts convenience function. * tools/buildhtml.py: - Added support for the "--prune" option. - Removed dependency on pep2html.py; plaintext PEPs no longer supported. * tools/docutils.conf: - Updated for configuration file reorganization. * tools/rst2html.py: - copied from tools/html.py * setup.py: - added a 'scripts' section to configuration - added 'tools/rst2html.py' to the scripts section Release 0.3 (2003-06-24) ======================== General: * Renamed "attribute" to "option" for directives/extensions. * Renamed transform method "transform" to "apply". * Renamed "options" to "settings" for runtime settings (as set by command-line options). Sometimes "option" (singular) became "settings" (plural). Some variations below: - document.options -> document.settings (stored in other objects as well) - option_spec -> settings_spec (not directives though) - OptionSpec -> SettingsSpec - cmdline_options -> settings_spec - relative_path_options -> relative_path_settings - option_default_overrides -> settings_default_overrides - Publisher.set_options -> Publisher.get_settings Specific: * COPYING.txt: Added "Public Domain Dedication". * FAQ.txt: Frequently asked questions, added to project. * setup.py: - Updated with PyPI Trove classifiers. - Conditional installation of third-party modules. * docutils/__init__.py: - Bumped version to 0.2.1 to reflect changes to I/O classes. - Bumped version to 0.2.2 to reflect changes to stylesheet options. - Factored ``SettingsSpec`` out of ``Component``; separately useful. - Bumped version to 0.2.3 because of the new "--embed-stylesheet" option and its effect on the PEP template & writer. - Bumped version to 0.2.4 due to changes to the PEP template & stylesheet. - Bumped version to 0.2.5 to reflect changes to Reporter output. - Added ``TransformSpec`` class for new transform system. - Bumped version to 0.2.6 for API changes (renaming). - Bumped version to 0.2.7 for new ``docutils.core.publish_*`` convenience functions. - Added ``Component.component_type`` attribute. - Bumped version to 0.2.8 because of the internal parser switch from plain lists to the docutils.statemachine.StringList objects. - Bumped version to 0.2.9 because of the frontend.py API changes. - Bumped version to 0.2.10 due to changes to the project layout (third-party modules removed from the "docutils" package), and signature changes in ``io.Input``/``io.Output``. - Changed version to 0.3.0 for release. * docutils/core.py: - Made ``publish()`` a bit more convenient. - Generalized ``Publisher.set_io``. - Renamed ``publish()`` to ``publish_cmdline()``; rearranged its parameters; improved its docstring. - Added ``publish_file()`` and ``publish_string()``. - Factored ``Publisher.set_source()`` and ``.set_destination()`` out of ``.set_io``. - Added support for "--dump-pseudo-xml", "--dump-settings", and "--dump-transforms" hidden options. - Added ``Publisher.apply_transforms()`` method. - Added ``Publisher.set_components()`` method; support for ``publish_*()`` conveninece functions. - Moved config file processing to docutils/frontend.py. - Added support for exit status ("exit_level" setting & ``enable_exit`` parameter for Publisher.publish() and convenience functions). * docutils/frontend.py: - Check for & exit on identical source & destination paths. - Fixed bug with absolute paths & "--config". - Set non-command-line defaults in ``OptionParser.__init__()``: ``_source`` & ``_destination``. - Distributed ``relative_path_settings`` to components; updated ``OptionParser.populate_from_components()`` to combine it all. - Require list of keys in ``make_paths_absolute`` (was implicit in global ``relative_path_settings``). - Added "--expose-internal-attribute", "--dump-pseudo-xml", "--dump-settings", and "--dump-transforms" hidden options. - Removed nasty internals-fiddling ``ConfigParser.get_section`` code, replaced with correct code. - Added validation functionality for config files. - Added "--error-encoding" option/setting, "_disable_config" internal setting. - Added encoding validation; updated "--input-encoding" and "--output-encoding"; added "--error-encoding-error-handler" and "--output-encoding-error-handler". - Moved config file processing from docutils/core.py. - Updated ``OptionParser.populate_from_components`` to handle new ``SettingsSpec.settings_defaults`` dict. - Added support for "-" => stdin/stdout. - Added "exit_level" setting ("--exit" option). * docutils/io.py: - Split ``IO`` classes into subclasses of ``Input`` and ``Output``. - Added automatic closing to ``FileInput`` and ``FileOutput``. - Delayed opening of ``FileOutput`` file until ``write()`` called. - ``FileOutput.write()`` now returns the encoded output string. - Try to get path/stream name automatically in ``FileInput`` & ``FileOutput``. - Added defaults for source & destination paths. - Allow for Unicode I/O with an explicit "unicode" encoding. - Added ``Output.encode()``. - Removed dependency on runtime settings; pass encoding directly. - Recognize Unicode strings in ``Input.decode()``. - Added support for output encoding error handlers. * docutils/nodes.py: - Added "Invisible" element category class. - Changed ``Node.walk()`` & ``.walkabout()`` to permit more tree modification during a traversal. - Added element classes: ``line_block``, ``generated``, ``address``, ``sidebar``, ``rubric``, ``attribution``, ``admonition``, ``superscript``, ``subscript``, ``inline`` - Added support for lists of nodes to ``Element.insert()``. - Fixed parent linking in ``Element.replace()``. - Added new abstract superclass ``FixedTextElement``; adds "xml:space" attribute. - Added support for "line" attribute of ``system_message`` nodes. - Added support for the observer pattern from ``utils.Reporter``. Added ``parse_messages`` and ``transform_messages`` attributes to ``document``, removed ``messages``. Added ``note_parse_message`` and ``note_transform_message`` methods. - Added support for improved diagnostics: - Added "document", "source", and "line" internal attributes to ``Node``, set by ``Node.setup_child()``. - Converted variations on ``node.parent = self`` to ``self.setup_child(node)``. - Added ``document.current_source`` & ``.current_line`` attributes, and ``.note_source`` observer method. - Changed "system_message" output to GNU-Tools format. - Added a "rawsource" attribute to the ``Text`` class, for text before backslash-escape resolution. - Support for new transform system. - Reworked ``pending`` element. - Fixed XML DOM bug (SF #660611). - Removed the ``interpeted`` element class and added ``title_reference``, ``abbreviation``, ``acronym``. - Made substitutions case-sensitive-but-forgiving; moved some code from the parser. - Fixed Unicode bug on element attributes (report: William Dode). * docutils/optik.py: Removed from project; replaced with extras/optparse.py and extras/textwrap.py. These will be installed only if they're not already present in the Python installation. * docutils/roman.py: Moved to extras/roman.py; this will be installed only if it's not already present in the Python installation. * docutils/statemachine.py: - Factored out ``State.add_initial_transitions()`` so it can be extended. - Converted whitespace-specific "blank" and "indent" transitions from special-case code to ordinary transitions: removed ``StateMachineWS.check_line()`` & ``.check_whitespace()``, added ``StateWS.add_initial_transitions()`` method, ``ws_patterns`` & ``ws_initial_transitions`` attributes. - Removed ``State.match_transition()`` after merging it into ``.check_line()``. - Added ``StateCorrection`` exception. - Added support for ``StateCorrection`` in ``StateMachine.run()`` (moved ``TransitionCorrection`` support there too.) - Changed ``StateMachine.next_line()`` and ``.goto_line()`` to raise ``EOFError`` instead of ``IndexError``. - Added ``State.no_match`` method. - Added support for the Observer pattern, triggered by input line changes. - Added ``strip_top`` parameter to ``StateMachineWS.get_first_known_indented``. - Made ``context`` a parameter to ``StateMachine.run()``. - Added ``ViewList`` & ``StringList`` classes; ``extract_indented()`` becomes ``StringList.get_indented()``. - Added ``StateMachine.insert_input()``. - Fixed ViewList slice handling for Python 2.3. Patch from (and thanks to) Fred Drake. * docutils/utils.py: - Added a ``source`` attribute to Reporter instances and ``system_message`` elements. - Added an observer pattern to ``utils.Reporter`` to keep track of system messages. - Fixed bugs in ``relative_path()``. - Added support for improved diagnostics. - Moved ``normalize_name()`` to nodes.py (``fully_normalize_name``). - Added support for encoding Reporter stderr output, and encoding error handlers. - Reporter keeps track of the highest level system message yet generated. * docutils/languages: Fixed bibliographic field language lookups. * docutils/languages/es.py: Added to project; Spanish mappings by Marcelo Huerta San Martin. * docutils/languages/fr.py: Added to project; French mappings by Stefane Fermigier. * docutils/languages/it.py: Added to project; Italian mappings by Nicola Larosa. * docutils/languages/sk.py: Added to project; Slovak mappings by Miroslav Vasko. * docutils/parser/__init__.py: - Added ``Parser.finish_parse()`` method. * docutils/parser/rst/__init__.py: - Added options: "--pep-references", "--rfc-references", "--tab-width", "--trim-footnote-reference-space". * docutils/parsers/rst/states.py: - Changed "title under/overline too short" system messages from INFO to WARNING, and fixed its insertion location. - Fixed enumerated list item parsing to allow paragraphs & section titles to begin with enumerators. - Converted system messages to use the new "line" attribute. - Fixed a substitution reference edge case. - Added support for "--pep-references" and "--rfc-references" options; reworked ``Inliner`` code to make customization easier. - Removed field argument parsing. - Added support for short section title over/underlines. - Fixed "simple reference name" regexp to ignore text like "object.__method__"; not an anonymous reference. - Added support for improved diagnostics. - Reworked directive API, based on Dethe Elza's contribution. Added ``Body.parse_directive()``, ``.parse_directive_options()``, ``.parse_directive_arguments()`` methods. - Added ``ExtensionOptions`` class, to parse directive options without parsing field bodies. Factored ``Body.parse_field_body()`` out of ``Body.field()``, overridden in ``ExtensionOptions``. - Improved definition list term/classifier parsing. - Added warnings for unknown directives. - Renamed ``Stuff`` to ``Struct``. - Now flagged as errors: transitions at the beginning or end of sections, empty sections (except title), and empty documents. - Updated for ``statemachine.StringList``. - Enabled recognition of schemeless email addresses in targets. - Added support for embedded URIs in hyperlink references. - Added backslash-escapes to inline markup end-string suffix. - Added support for correct interpreted text processing. - Fixed nested title parsing (topic, sidebar directives). - Added special processing of backslash-escaped whitespace (idea from David Abrahams). - Made substitutions case-sensitive-but-forgiving; moved some code to ``docutils.nodes``. - Added support for block quote attributions. - Added a kludge to work-around a conflict between the bubble-up parser strategy and short titles (<= 3 char-long over- & underlines). Fixes SF bug #738803 "infinite loop with multiple titles" submitted by Jason Diamond. - Added explicit interpreted text roles for standard inline markup: "emphasis", "strong", "literal". - Implemented "superscript" and "subscript" interpreted text roles. - Added initial support for "abbreviation" and "acronym" roles; incomplete. - Added support for "--trim-footnote-reference-space" option. - Optional space before colons in directives & hyperlink targets. * docutils/parsers/rst/tableparser.py: - Fixed a bug that was producing unwanted empty rows in "simple" tables. - Detect bad column spans in "simple" tables. * docutils/parsers/rst/directives: Updated all directive functions to new API. * docutils/parsers/rst/directives/__init__.py: - Added ``flag()``, ``unchanged()``, ``path()``, ``nonnegative_int()``, ``choice()``, and ``class_option()`` directive option helper functions. - Added warnings for unknown directives. - Return ``None`` for missing directives. - Added ``register_directive()``, thanks to William Dode and Paul Moore. * docutils/parsers/rst/directives/admonitions.py: - Added "admonition" directive. * docutils/parsers/rst/directives/body.py: Added to project. Contains the "topic", "sidebar" (from Patrick O'Brien), "line-block", "parsed-literal", "rubric", "epigraph", "highlights" and "pull-quote" directives. * docutils/parsers/rst/directives/images.py: - Added an "align" attribute to the "image" & "figure" directives (by Adam Chodorowski). - Added "class" option to "image", and "figclass" to "figure". * docutils/parsers/rst/directives/misc.py: - Added "include", "raw", and "replace" directives, courtesy of Dethe Elza. - Added "unicode" and "class" directives. * docutils/parsers/rst/directives/parts.py: - Added the "sectnum" directive; by Dmitry Jemerov. - Added "class" option to "contents" directive. * docutils/parsers/rst/directives/references.py: Added to project. Contains the "target-notes" directive. * docutils/parsers/rst/languages/__init__.py: - Return ``None`` from get_language() for missing language modules. * docutils/parsers/rst/languages/de.py: Added to project; German mappings by Engelbert Gruber. * docutils/parsers/rst/languages/en.py: - Added interpreted text roles mapping. * docutils/parsers/rst/languages/es.py: Added to project; Spanish mappings by Marcelo Huerta San Martin. * docutils/parsers/rst/languages/fr.py: Added to project; French mappings by William Dode. * docutils/parsers/rst/languages/it.py: Added to project; Italian mappings by Nicola Larosa. * docutils/parsers/rst/languages/sk.py: Added to project; Slovak mappings by Miroslav Vasko. * docutils/readers/__init__.py: - Added support for the observer pattern from ``utils.Reporter``, in ``Reader.parse`` and ``Reader.transform``. - Removed ``Reader.transform()`` method. - Added default parameter values to ``Reader.__init__()`` to make instantiation easier. - Removed bogus aliases: "restructuredtext" is *not* a Reader. * docutils/readers/pep.py: - Added the ``peps.TargetNotes`` transform to the Reader. - Removed PEP & RFC reference detection code; moved to parsers/rst/states.py as options (enabled here by default). - Added support for pre-acceptance PEPs (no PEP number yet). - Moved ``Inliner`` & made it a class attribute of ``Reader`` for easy subclassing. * docutils/readers/python: Python Source Reader subpackage added to project, including preliminary versions of: - __init__.py - moduleparser.py: Parser for Python modules. * docutils/transforms/__init__.py: - Added ``Transformer`` class and completed transform reform. - Added unknown_reference_resolvers list for each transformer. This list holds the list of functions provided by each component of the transformer that help resolve references. * docutils/transforms/frontmatter.py: - Improved support for generic fields. - Fixed bibliographic field language lookups. * docutils/transforms/misc.py: Added to project. Miscellaneous transforms. * docutils/transforms/parts.py: - Moved the "id" attribute from TOC list items to the references (``Contents.build_contents()``). - Added the ``SectNum`` transform; by Dmitry Jemerov. - Added "class" attribute support to ``Contents``. * docutils/transforms/peps.py: - Added ``mask_email()`` function, updating to pep2html.py's functionality. - Linked "Content-Type: text/x-rst" to PEP 12. - Added the ``TargetNotes`` PEP-specific transform. - Added ``TargetNotes.cleanup_callback``. - Added title check to ``Headers``. * docutils/transforms/references.py: - Added the ``TargetNotes`` generic transform. - Split ``Hyperlinks`` into multiple transforms. - Fixed bug with multiply-indirect references (report: Bruce Smith). - Added check for circular indirect references. - Made substitutions case-sensitive-but-forgiving. * docutils/transforms/universal.py: - Added support for the "--expose-internal-attributes" option. - Removed ``Pending`` transform classes & data. * docutils/writers/__init__.py: - Removed ``Writer.transform()`` method. * docutils/writers/docutils-xml.py: - Added XML and doctype declarations. - Added "--no-doctype" and "--no-xml-declaration" options. * docutils/writers/html4css1.py: - "name" attributes only on these tags: a, applet, form, frame, iframe, img, map. - Added "name" attribute to <a> in section titles for Netscape 4 support (bug report: Pearu Peterson). - Fixed targets (names) on footnote, citation, topic title, problematic, and system_message nodes (for Netscape 4). - Changed field names from "<td>" to "<th>". - Added "@" to "@" encoding to thwart address harvesters. - Improved the vertical whitespace optimization; ignore "invisible" nodes (targets, comments, etc.). - Improved inline literals with ``<span class="pre">`` around chunks of text and `` `` for runs of spaces. - Improved modularity of output; added ``self.body_pre_docinfo`` and ``self.docinfo`` segments. - Added support for "line_block", "address" elements. - Improved backlinks (footnotes & system_messages). - Improved system_message output. - Redefined "--stylesheet" as containing an invariant URL, used verbatim. Added "--stylesheet-path", interpreted w.r.t. the working directory. - Added "--footnote-references" option (superscript or brackets). - Added "--compact-lists" and "--no-compact-lists" options. - Added "--embed-stylesheet" and "--link-stylesheet" options; factored out ``HTMLTranslator.get_stylesheet_reference()``. - Improved field list rendering. - Added Docutils version to "generator" meta tag. - Fixed a bug with images; they must be inline, so wrapped in <p>. - Improved layout of <pre> HTML source. - Fixed attribute typo on <colspec>. - Refined XML prologue. - Support for no stylesheet. - Removed "interpreted" element support. - Added support for "title_reference", "sidebar", "attribution", "rubric", and generic "admonition" elements. - Added "--attribution" option. - Added support for "inline", "subscript", "superscript" elements. - Added initial support for "abbreviation" and "acronym"; incomplete. * docutils/writers/latex2e.py: LaTeX Writer, added by Engelbert Gruber (from the sandbox). - Added french. - Double quotes in literal blocks (special treatment for de/ngerman). - Added '--hyperlink-color' option ('0' turns off coloring of links). - Added "--attribution" option. - Right align attributions. * docutils/writers/pep_html.py: - Parameterized output encoding in PEP template. - Reworked substitutions from ``locals()`` into ``subs`` dict. - Redefined "--pep-stylesheet" as containing an invariant URL, used verbatim. Added "--pep-stylesheet-path", interpreted w.r.t. the working directory. - Added an override on the "--footnote-references" option. - Factored out ``HTMLTranslator.get_stylesheet_reference()``. - Added Docutils version to "generator" meta tag. - Added a "DO NOT EDIT THIS FILE" comment to generated HTML. * docs/tools.txt: - Added a "silent" setting for ``buildhtml.py``. - Added a "Getting Help" section. - Rearranged the structure. - Kept up to date, with new settings, command-line options etc. - Added section for ``rst2latex.py`` (Engelbert Gruber). - Converted settings table into a definition list. * docs/rst/quickstart.txt: - Added a table of contents. - Added feedback information. - Added mention of minimum section title underline lengths. - Removed the 4-character minimum for section title underlines. * docs/rst/quickref.html: - Added a "Getting Help" section. - Added a style to make section title backlinks more subtle. - Added mention of minimum section title underline lengths. - Removed the 4-character minimum for section title underlines. * extras: Directory added to project; contains third-party modules that Docutils depends on (optparse, textwrap, roman). These are only installed if they're not already present. * licenses: Directory added to project; contains copies of license files for non-public-domain files. * spec/doctree.txt: - Changed the focus. It's about DTD elements: structural relationships, semantics, and external (public) attributes. Not about the element class library. - Moved some implementation-specific stuff into ``docutils.nodes`` docstrings. - Wrote descriptions of all common attributes and parameter entities. Filled in introductory material. - Working through the element descriptions: 55 down, 37 to go. - Removed "Representation of Horizontal Rules" to spec/rst/alternatives.txt. * spec/docutils.dtd: - Added "generated" inline element. - Added "line_block" body element. - Added "auto" attribute to "title". - Changed content models of "literal_block" and "doctest_block" to ``%text.model``. - Added ``%number;`` attribute type parameter entity. - Changed ``%structural.elements;`` to ``%section.elements``. - Updated attribute types; made more specific. - Added "address" bibliographic element. - Added "line" attribute to ``system_message`` element. - Removed "field_argument" element; "field_name" may contain multiple words and whitespace. - Changed public identifier to docutils.sf.net. - Removed "interpreted" element; added "title_reference", "abbreviation", "acronym". - Removed "refuri" attribute from "footnote_reference" and "citation_reference". - Added "sidebar", "rubric", "attribution", "admonition", "superscript", "subscript", and "inline" elements. * spec/pep-0256.txt: Converted to reStructuredText & updated. * spec/pep-0257.txt: Converted to reStructuredText & updated. * spec/pep-0258.txt: Converted to reStructuredText & updated. * spec/semantics.txt: Updated with text from a Doc-SIG response to Dallas Mahrt. * spec/transforms.txt: Added to project. * spec/howto: Added subdirectory, for developer how-to docs. * spec/howto/rst-directives.txt: Added to project. Original by Dethe Elza, edited & extended by David Goodger. * spec/howto/i18n.txt: Docutils Internationalization. Added to project. * spec/rst/alternatives.txt: - Added "Doctree Representation of Transitions" from spec/doctree.txt. - Updated "Inline External Targets" & closed the debate. - Added ideas for interpreted text syntax extensions. - Added "Nested Inline Markup" section. * spec/rst/directives.txt: - Added directives: "topic", "sectnum", "target-notes", "line-block", "parsed-literal", "include", "replace", "sidebar", "admonition", "rubric", "epigraph", "highlights", "unicode" and "class". - Formalized descriptions of directive details. - Added an "align" attribute to the "image" & "figure" directives (by Adam Chodorowski). - Added "class" options to "topic", "sidebar", "line-block", "parsed-literal", "contents", and "image"; and "figclass" to "figure". * spec/rst/interpreted.txt: Added to project. Descriptions of interpreted text roles. * spec/rst/introduction.txt: - Added pointers to material for new users. * spec/rst/reStructuredText.txt: - Disambiguated comments (just add a newline after the "::"). - Updated enumerated list description; added a discussion of the second-line validity checking. - Updated directive description. - Added a note redirecting newbies to the user docs. - Expanded description of inline markup start-strings in non-markup contexts. - Removed field arguments and made field lists a generic construct. - Removed the 4-character minimum for section title underlines. - Clarified term/classifier delimiter & inline markup ambiguity (definition lists). - Added "Embedded URIs". - Updated "Interpreted Text" section. - Added "Character-Level Inline Markup" section. * test: Continually adding & updating tests. - Moved test/test_rst/ to test/test_parsers/test_rst/. - Moved test/test_pep/ to test/test_readers/test_pep/. - Added test/test_readers/test_python/. - Added test/test_writers/ (Engelbert Gruber). * tools: - Made the ``locale.setlocale()`` calls in front ends fault-tolerant. * tools/buildhtml.py: - Added "--silent" option. - Fixed bug with absolute paths & "--config". - Updated for new I/O classes. - Added some exception handling. - Separated publishers' setting defaults; prevents interference. - Updated for new ``publish_file()`` convenience function. * tools/pep-html-template: - Allow for "--embed-stylesheet". - Added Docutils version to "generator" meta tag. - Added a "DO NOT EDIT THIS FILE" comment to generated HTML. - Conform to XHTML spec. * tools/pep2html.py: - Made ``argv`` a parameter to ``main()``. - Added support for "Content-Type:" header & arbitrary PEP formats. - Linked "Content-Type: text/plain" to PEP 9. - Files skipped (due to an error) are not pushed onto the server. - Updated for new I/O classes. - Added ``check_requirements()`` & ``pep_type_error()``. - Added some exception handling. - Updated for new ``publish_string()`` convenience function. - Added a "DO NOT EDIT THIS FILE" comment to generated HTML. * tools/quicktest.py: - Added "-V"/"--version" option. * tools/rst2latex.py: LaTeX front end, added by Engelbert Gruber. * tools/unicode2rstsubs.py: Added to project. Produces character entity files (reSructuredText substitutions) from the MathML master unicode.xml file. * tools/editors: Support code for editors, added to project. Contains ``emacs/restructuredtext.el``. * tools/stylesheets/default.css: Moved into the stylesheets directory. - Added style for chunks of inline literals. - Removed margin for first child of table cells. - Right-aligned field list names. - Support for auto-numbered section titles in TOCs. - Increased the size of inline literals (<tt>) in titles. - Restored the light gray background for inline literals. - Added support for "line_block" elements. - Added style for "address" elements. - Removed "a.footnote-reference" style; doing it with ``<sup>`` now. - Improved field list rendering. - Vertical whitespace improvements. - Removed "a.target" style. * tools/stylesheets/pep.css: - Fixed nested section margins. - Other changes parallel those of ``../default.css``. Release 0.2 (2002-07-31) ======================== General: - The word "component" was being used ambiguously. From now on, "component" will be used to mean "Docutils component", as in Reader, Writer, Parser, or Transform. Portions of documents (Table of Contents, sections, etc.) will be called "document parts". - Did a grand renaming: a lot of ``verylongnames`` became ``very_long_names``. - Cleaned up imports: no more relative package imports or comma-separated lists of top-level modules. - Added support for an option values object which carries default settings and overrides (from command-line options and library use). - Added internal Unicode support, and support for both input and output encodings. - Added support for the ``docutils.io.IO`` class & subclasses. Specific: * docutils/__init__.py: - Added ``ApplicationError`` and ``DataError``, for use throughout the package. - Added ``Component`` base class for Docutils components; implements the ``supports`` method. - Added ``__version__`` (thus, ``docutils.__version__``). * docutils/core.py: - Removed many keyword parameters to ``Publisher.__init__()`` and ``publish()``; bundled into an option values object. Added "argv", "usage", "description", and "option_spec" parameters for command-line support. - Added ``Publisher.process_command_line()`` and ``.set_options()`` methods. - Reworked I/O model for ``docutils.io`` wrappers. - Updated ``Publisher.set_options()``; now returns option values object. - Added support for configuration files (/etc/docutils.conf, ./docutils.conf, ~/.docutils). - Added ``Publisher.setup_option_parser()``. - Added default usage message and description. * docutils/frontend.py: Added to project; support for front-end (command-line) scripts. Option specifications may be augmented by components. Requires Optik (http://optik.sf.net/) for option processing (installed locally as docutils/optik.py). * docutils/io.py: Added to project; uniform API for a variety of input output mechanisms. * docutils/nodes.py: - Added ``TreeCopyVisitor`` class. - Added a ``copy`` method to ``Node`` and subclasses. - Added a ``SkipDeparture`` exception for visitors. - Renamed ``TreePruningException`` from ``VisitorException``. - Added docstrings to ``TreePruningException``, subclasses, and ``Nodes.walk()``. - Improved docstrings. - Added ``SparseNodeVisitor``, refined ``NodeVisitor``. - Moved ``utils.id()`` to ``nodes.make_id()`` to avoid circular imports. - Added ``decoration``, ``header``, and ``footer`` node classes, and ``PreDecorative`` mixin. - Reworked the name/id bookkeeping; to ``document``, removed ``explicit_targets`` and ``implicit_targets`` attributes, added ``nametypes`` attribute and ``set_name_id_map`` method. - Added ``NodeFound`` exception, for use with ``NodeVisitor`` traversals. - Added ``document.has_name()`` method. - Fixed DOM generation for list-attributes. - Added category class ``Labeled`` (used by footnotes & citations). - Added ``Element.set_class()`` method (sets "class" attribute). * docutils/optik.py: Added to project. Combined from the Optik package, with added option groups and other modifications. The use of this module is probably only temporary. * docutils/statemachine.py: - Added ``runtime_init`` method to ``StateMachine`` and ``State``. - Added underscores to improve many awkward names. - In ``string2lines()``, changed whitespace normalizing translation table to regexp; restores Python 2.0 compatibility with Unicode. * docutils/urischemes.py: - Filled in some descriptions. - Added "shttp" scheme. * docutils/utils.py: - Added ``clean_rcs_keywords`` function (moved from docutils/transforms/frontmatter.py ``DocInfo.filter_rcs_keywords``). - Added underscores to improve many awkward names. - Changed names of Reporter's thresholds: warning_level -> report_level; error_level -> halt_level. - Moved ``utils.id()`` to ``nodes.make_id()``. - Added ``relative_path(source, target)``. * docutils/languages/de.py: German mappings; added to project. Thanks to Gunnar Schwant for the translations. * docutils/languages/en.py: Added "Dedication" bibliographic field mappings. * docutils/languages/sv.py: Swedish mappings; added to project by Adam Chodorowski. * docutils/parsers/rst/states.py: - Added underscores to improve many awkward names. - Added RFC-2822 header support. - Extracted the inline parsing code from ``RSTState`` to a separate class, ``Inliner``, which will allow easy subclassing. - Made local bindings for ``memo`` container & often-used contents (reduces code complexity a lot). See ``RSTState.runtime_init()``. - ``RSTState.parent`` replaces ``RSTState.statemachine.node``. - Added ``MarkupMismatch`` exception; for late corrections. - Added ``-/:`` characters to inline markup's start string prefix, ``/`` to end string suffix. - Fixed a footnote bug. - Fixed a bug with literal blocks. - Applied patch from Simon Budig: simplified regexps with symbolic names, removed ``Inliner.groups`` and ``Body.explicit.groups``. - Converted regexps from ``'%s' % var`` to ``'%(var)s' % locals()``. - Fixed a bug in ``Inliner.interpreted_or_phrase_ref()``. - Allowed non-ASCII in "simple names" (directive names, field names, references, etc.). - Converted ``Inliner.patterns.initial`` to be dynamically built from parts with ``build_regexp()`` function. - Changed ``Inliner.inline_target`` to ``.inline_internal_target``. - Updated docstrings. - Changed "table" to "grid_table"; added "simple_table" support. * docutils/parsers/rst/tableparser.py: - Changed ``TableParser`` to ``GridTableParser``. - Added ``SimpleTableParser``. - Refactored naming. * docutils/parsers/rst/directives/__init__.py: Added "en" (English) as a fallback language for directive names. * docutils/parsers/rst/directives/html.py: Changed the ``meta`` directive to use a ``pending`` element, used only by HTML writers. * docutils/parsers/rst/directives/parts.py: Renamed from components.py. - Added "backlinks" attribute to "contents" directive. * docutils/parsers/rst/languages/sv.py: Swedish mappings; added to project by Adam Chodorowski. * docutils/readers/__init__.py: Gave Readers more control over choosing and instantiating Parsers. * docutils/readers/pep.py: Added to project; for PEP processing. * docutils/transforms/__init__.py: ``Transform.__init__()`` now requires a ``component`` parameter. * docutils/transforms/components.py: Added to project; transforms related to Docutils components. * docutils/transforms/frontmatter.py: - In ``DocInfo.extract_authors``, check for a single "author" in an "authors" group, and convert it to a single "author" element. - Added support for "Dedication" and generic bibliographic fields. * docutils/transforms/peps.py: Added to project; PEP-specific. * docutils/transforms/parts.py: Renamed from old components.py. - Added filter for `Contents`, to use alt-text for inline images, and to remove inline markup that doesn't make sense in the ToC. - Added "name" attribute to TOC topic depending on its title. - Added support for optional TOC backlinks. * docutils/transforms/references.py: Fixed indirect target resolution in ``Hyperlinks`` transform. * docutils/transforms/universal.py: - Changed ``Messages`` transform to properly filter out system messages below the warning threshold. - Added ``Decorations`` transform (support for ``--generator``, ``--date``, ``--time``, ``--source-link`` options). * docutils/writers/__init__.py: Added "pdf" alias in anticipation of Engelbert Gruber's PDF writer. * docutils/writers/html4css1.py: - Made XHTML-compatible (switched to lowercase element & attribute names; empty tag format). - Escape double-dashes in comment text. - Improved boilerplate & modularity of output. - Exposed modular output in Writer class. - Added a "generator" meta tag to <head>. - Added support for the ``--stylesheet`` option. - Added support for ``decoration``, ``header``, and ``footer`` elements. - In ``HTMLTranslator.attval()``, changed whitespace normalizing translation table to regexp; restores Python 2.0 compatibility with Unicode. - Added the translator class as instance variable to the Writer, to make it easily subclassable. - Improved option list spacing (thanks to Richard Jones). - Modified field list output. - Added backlinks to footnotes & citations. - Added percentage widths to "<col>" tags (from colspec). - Option lists: "<code>" changed to "<kbd>", ``option_argument`` "<span>" changed to "<var>". - Inline literals: "<code>" changed to "<tt>". - Many changes to optimize vertical space: compact simple lists etc. - Add a command-line options & directive attributes to control TOC and footnote/citation backlinks. - Added support for optional footnote/citation backlinks. - Added support for generic bibliographic fields. - Identify backrefs. - Relative URLs for stylesheet links. * docutils/writers/pep_html.py: Added to project; HTML Writer for PEPs (subclass of ``html4css1.Writer``). * docutils/writers/pseudoxml.py: Renamed from pprint.py. * docutils/writers/docutils_xml.py: Added to project; trivial writer of the Docutils internal doctree in XML. * docs/tools.txt: "Docutils Front-End Tools", added to project. * spec/doctree.txt: - Changed the title to "The Docutils Document Tree". - Added "Hyperlink Bookkeeping" section. * spec/docutils.dtd: - Added ``decoration``, ``header``, and ``footer`` elements. - Brought ``interpreted`` element in line with the parser: changed attribute "type" to "role", added "position". - Added support for generic bibliographic fields. * spec/notes.txt: Continual updates. Added "Project Policies". * spec/pep-0256.txt: Updated. Added "Roadmap to the Doctring PEPs" section. * spec/pep-0257.txt: Clarified prohibition of signature repetition. * spec/pep-0258.txt: Updated. Added text from pysource.txt and mailing list discussions. * spec/pep-0287.txt: - Renamed to "reStructuredText Docstring Format". - Minor edits. - Reworked Q&A as an enumerated list. - Converted to reStructuredText format. * spec/pysource.dtd: - Reworked structural elements, incorporating ideas from Tony Ibbs. * spec/pysource.txt: Removed from project. Moved much of its contents to pep-0258.txt. * spec/rst/alternatives.txt: - Expanded auto-enumerated list idea; thanks to Fred Bremmer. - Added "Inline External Targets" section. * spec/rst/directives.txt: - Added "backlinks" attribute to "contents" directive. * spec/rst/problems.txt: - Updated the Enumerated List Markup discussion. - Added new alternative table markup syntaxes. * spec/rst/reStructuredText.txt: - Clarified field list usage. - Updated enumerated list description. - Clarified purpose of directives. - Added ``-/:`` characters to inline markup's start string prefix, ``/`` to end string suffix. - Updated "Authors" bibliographic field behavior. - Changed "inline hyperlink targets" to "inline internal targets". - Added "simple table" syntax to supplement the existing but newly-renamed "grid tables". - Added cautions for anonymous hyperlink use. - Added "Dedication" and generic bibliographic fields. * test: Made test modules standalone (subdirectories became packages). * test/DocutilsTestSupport.py: - Added support for PEP extensions to reStructuredText. - Added support for simple tables. - Refactored naming. * test/package_unittest.py: Renamed from UnitTestFolder.py. - Now supports true packages containing test modules (``__init__.py`` files required); fixes duplicate module name bug. * test/test_pep/: Subpackage added to project; PEP testing. * test/test_rst/test_SimpleTableParser.py: Added to project. * tools: - Updated html.py and publish.py front-end tools to use the new command-line processing facilities of ``docutils.frontend`` (exposed in ``docutils.core.Publisher``), reducing each to just a few lines of code. - Added ``locale.setlocale()`` calls to front-end tools. * tools/buildhtml.py: Added to project; batch-generates .html from all the .txt files in directories and subdirectories. * tools/default.css: - Added support for ``header`` and ``footer`` elements. - Added styles for "Dedication" topics (biblio fields). * tools/docutils.conf: A configuration file; added to project. * tools/docutils-xml.py: Added to project. * tools/pep.py: Added to project; PEP to HTML front-end tool. * tools/pep-html-template: Added to project. * tools/pep2html.py: Added to project from Python (nondist/peps). Added support for Docutils (reStructuredText PEPs). * tools/quicktest.py: - Added the ``--attributes`` option, hacked a bit. - Added a second command-line argument (output file); cleaned up. * tools/stylesheets/: Subdirectory added to project. * tools/stylesheets/pep.css: Added to project; stylesheet for PEPs. Release 0.1 (2002-04-20) ======================== This is the first release of Docutils, merged from the now inactive reStructuredText__ and `Docstring Processing System`__ projects. For the pre-Docutils history, see the `reStructuredText HISTORY`__ and the `DPS HISTORY`__ files. __ http://structuredtext.sourceforge.net/ __ http://docstring.sourceforge.net/ __ http://structuredtext.sourceforge.net/HISTORY.html __ http://docstring.sourceforge.net/HISTORY.html General changes: renamed 'dps' package to 'docutils'; renamed 'restructuredtext' subpackage to 'rst'; merged the codebases; merged the test suites (reStructuredText's test/test_states renamed to test/test_rst); and all modifications required to make it all work. * docutils/parsers/rst/states.py: - Improved diagnostic system messages for missing blank lines. - Fixed substitution_reference bug. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/������������������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�017146� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/__init__.py�������������������������������������������������0000664�0000764�0000764�00000002160�10435500330�021243� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """General exceptions that wish they were standard exceptions These exceptions are so general purpose that they don't belong in Zope application-specific packages. $Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $ """ from unauthorized import Unauthorized class BadRequest(Exception): pass class InternalError(Exception): pass class NotFound(Exception): pass class Forbidden(Exception): pass class MethodNotAllowed(Exception): pass class Redirect(Exception): pass ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/ExceptionFormatter.py���������������������������������������0000664�0000764�0000764�00000016760�10435500330�023341� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """An exception formatter that shows traceback supplements and traceback info, optionally in HTML. $Id: ExceptionFormatter.py 24763 2004-05-17 05:59:28Z philikon $ """ import sys import cgi DEBUG_EXCEPTION_FORMATTER = 1 class TextExceptionFormatter: line_sep = '\n' show_revisions = 0 def __init__(self, limit=None): self.limit = limit def escape(self, s): return s def getPrefix(self): return 'Traceback (innermost last):' def getLimit(self): limit = self.limit if limit is None: limit = getattr(sys, 'tracebacklimit', None) return limit def getRevision(self, globals): if not self.show_revisions: return None revision = globals.get('__revision__', None) if revision is None: # Incorrect but commonly used spelling revision = globals.get('__version__', None) if revision is not None: try: revision = str(revision).strip() except: revision = '???' return revision def formatSupplementLine(self, line): return ' - %s' % line def formatObject(self, object): return [self.formatSupplementLine(repr(object))] def formatSourceURL(self, url): return [self.formatSupplementLine('URL: %s' % url)] def formatSupplement(self, supplement, tb): result = [] fmtLine = self.formatSupplementLine object = getattr(supplement, 'object', None) if object is not None: result.extend(self.formatObject(object)) url = getattr(supplement, 'source_url', None) if url is not None: result.extend(self.formatSourceURL(url)) line = getattr(supplement, 'line', 0) if line == -1: line = tb.tb_lineno col = getattr(supplement, 'column', -1) if line: if col is not None and col >= 0: result.append(fmtLine('Line %s, Column %s' % ( line, col))) else: result.append(fmtLine('Line %s' % line)) elif col is not None and col >= 0: result.append(fmtLine('Column %s' % col)) expr = getattr(supplement, 'expression', None) if expr: result.append(fmtLine('Expression: %s' % expr)) warnings = getattr(supplement, 'warnings', None) if warnings: for warning in warnings: result.append(fmtLine('Warning: %s' % warning)) extra = self.formatExtraInfo(supplement) if extra: result.append(extra) return result def formatExtraInfo(self, supplement): getInfo = getattr(supplement, 'getInfo', None) if getInfo is not None: extra = getInfo() if extra: return extra return None def formatTracebackInfo(self, tbi): return self.formatSupplementLine('__traceback_info__: %s' % (tbi,)) def formatLine(self, tb): f = tb.tb_frame lineno = tb.tb_lineno co = f.f_code filename = co.co_filename name = co.co_name locals = f.f_locals globals = f.f_globals modname = globals.get('__name__', filename) s = ' Module %s, line %d' % (modname, lineno) revision = self.getRevision(globals) if revision: s = s + ', rev. %s' % revision s = s + ', in %s' % name result = [] result.append(self.escape(s)) # Output a traceback supplement, if any. if locals.has_key('__traceback_supplement__'): # Use the supplement defined in the function. tbs = locals['__traceback_supplement__'] elif globals.has_key('__traceback_supplement__'): # Use the supplement defined in the module. # This is used by Scripts (Python). tbs = globals['__traceback_supplement__'] else: tbs = None if tbs is not None: factory = tbs[0] args = tbs[1:] try: supp = factory(*args) result.extend(self.formatSupplement(supp, tb)) except: if DEBUG_EXCEPTION_FORMATTER: import traceback traceback.print_exc() # else just swallow the exception. try: tbi = locals.get('__traceback_info__', None) if tbi is not None: result.append(self.formatTracebackInfo(tbi)) except: pass return self.line_sep.join(result) def formatExceptionOnly(self, etype, value): import traceback return self.line_sep.join( traceback.format_exception_only(etype, value)) def formatLastLine(self, exc_line): return self.escape(exc_line) def formatException(self, etype, value, tb, limit=None): # The next line provides a way to detect recursion. __exception_formatter__ = 1 result = [self.getPrefix() + '\n'] if limit is None: limit = self.getLimit() n = 0 while tb is not None and (limit is None or n < limit): if tb.tb_frame.f_locals.get('__exception_formatter__'): # Stop recursion. result.append('(Recursive formatException() stopped)\n') break line = self.formatLine(tb) result.append(line + '\n') tb = tb.tb_next n = n + 1 exc_line = self.formatExceptionOnly(etype, value) result.append(self.formatLastLine(exc_line)) return result class HTMLExceptionFormatter (TextExceptionFormatter): line_sep = '<br />\r\n' def escape(self, s): return cgi.escape(s) def getPrefix(self): return '<p>Traceback (innermost last):\r\n<ul>' def formatSupplementLine(self, line): return '<b>%s</b>' % self.escape(str(line)) def formatTracebackInfo(self, tbi): s = self.escape(str(tbi)) s = s.replace('\n', self.line_sep) return '__traceback_info__: %s' % s def formatLine(self, tb): line = TextExceptionFormatter.formatLine(self, tb) return '<li>%s</li>' % line def formatLastLine(self, exc_line): return '</ul>%s</p>' % self.escape(exc_line) def formatExtraInfo(self, supplement): getInfo = getattr(supplement, 'getInfo', None) if getInfo is not None: extra = getInfo(1) if extra: return extra return None limit = 200 if hasattr(sys, 'tracebacklimit'): limit = min(limit, sys.tracebacklimit) text_formatter = TextExceptionFormatter(limit) html_formatter = HTMLExceptionFormatter(limit) def format_exception(t, v, tb, limit=None, as_html=0): if as_html: fmt = html_formatter else: fmt = text_formatter return fmt.formatException(t, v, tb, limit=limit) ����������������Zope-2.8.7-final/lib/python/zExceptions/unauthorized.py���������������������������������������������0000664�0000764�0000764�00000004574�10435500330�022240� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ $Id: unauthorized.py 24763 2004-05-17 05:59:28Z philikon $ """ from types import StringType class Unauthorized(Exception): """Some user wasn't allowed to access a resource""" def __init__(self, message=None, value=None, needed=None, name=None, **kw): """Possible signatures: Unauthorized() Unauthorized(message) # Note that message includes a space Unauthorized(name) Unauthorized(name, value) Unauthorized(name, value, needed) Unauthorized(message, value, needed, name) Where needed is a mapping objects with items represnting requirements (e.g. {'permission': 'add spam'}). Any extra keyword arguments provides are added to needed. """ if name is None and ( not isinstance(message, StringType) or len(message.split()) <= 1): # First arg is a name, not a message name=message message=None self.name=name self.message=message self.value=value if kw: if needed: needed.update(kw) else: needed=kw self.needed=needed def __str__(self): if self.message is not None: return self.message if self.name is not None: return ("You are not allowed to access '%s' in this context" % self.name) elif self.value is not None: return ("You are not allowed to access '%s' in this context" % self.getValueName()) return repr(self) def getValueName(self): v=self.value vname=getattr(v, '__name__', None) if vname: return vname c = getattr(v, '__class__', type(v)) c = getattr(c, '__name__', 'object') return "a particular %s" % c ������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/TracebackSupplement.py��������������������������������������0000664�0000764�0000764�00000001266�10435500330�023446� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# Stock __traceback_supplement__ implementations class PathTracebackSupplement: """Implementation of ITracebackSupplement""" pp = None def __init__(self, object): self.object = object if hasattr(object, 'getPhysicalPath'): self.pp = '/'.join(object.getPhysicalPath()) if hasattr(object, 'absolute_url'): self.source_url = '%s/manage_main' % object.absolute_url() def getInfo(self, as_html=0): if self.pp is None: return if as_html: from cgi import escape return '<b>Physical Path:</b>%s' % (escape(self.pp)) else: return ' - Physical Path: %s' % self.pp ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/tests/������������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�020310� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/tests/__init__.py�������������������������������������������0000664�0000764�0000764�00000000044�10435500330�022404� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������"""Package for zExceptions tests""" ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/tests/testExceptionFormatter.py�����������������������������0000664�0000764�0000764�00000010560�10435500330�025373� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ ExceptionFormatter tests. Revision information: $Id: testExceptionFormatter.py 24763 2004-05-17 05:59:28Z philikon $ """ from __future__ import nested_scopes from unittest import TestCase, TestSuite, main, makeSuite try: from Testing.CleanUp import CleanUp # Base class w registry cleanup except ImportError: class CleanUp: pass import sys from zExceptions.ExceptionFormatter import format_exception def tb(as_html=0): t, v, b = sys.exc_info() try: return ''.join(format_exception(t, v, b, as_html=as_html)) finally: del b class ExceptionForTesting (Exception): pass class TestingTracebackSupplement: source_url = '/somepath' line = 634 column = 57 warnings = ['Repent, for the end is nigh'] def __init__(self, expression): self.expression = expression class Test(CleanUp, TestCase): def testBasicNamesText(self, as_html=0): try: raise ExceptionForTesting except ExceptionForTesting: s = tb(as_html) # The traceback should include the name of this function. self.assert_(s.find('testBasicNamesText') >= 0) # The traceback should include the name of the exception. self.assert_(s.find('ExceptionForTesting') >= 0) else: self.fail('no exception occurred') def testBasicNamesHTML(self): self.testBasicNamesText(1) def testSupplement(self, as_html=0): try: __traceback_supplement__ = (TestingTracebackSupplement, "You're one in a million") raise ExceptionForTesting except ExceptionForTesting: s = tb(as_html) # The source URL self.assert_(s.find('/somepath') >= 0, s) # The line number self.assert_(s.find('634') >= 0, s) # The column number self.assert_(s.find('57') >= 0, s) # The expression self.assert_(s.find("You're one in a million") >= 0, s) # The warning self.assert_(s.find("Repent, for the end is nigh") >= 0, s) else: self.fail('no exception occurred') def testSupplementHTML(self): self.testSupplement(1) def testTracebackInfo(self, as_html=0): try: __traceback_info__ = "Adam & Eve" raise ExceptionForTesting except ExceptionForTesting: s = tb(as_html) if as_html: # Be sure quoting is happening. self.assert_(s.find('Adam & Eve') >= 0, s) else: self.assert_(s.find('Adam & Eve') >= 0, s) else: self.fail('no exception occurred') def testTracebackInfoHTML(self): self.testTracebackInfo(1) def testMultipleLevels(self): # Makes sure many levels are shown in a traceback. def f(n): """Produces a (n + 1)-level traceback.""" __traceback_info__ = 'level%d' % n if n > 0: f(n - 1) else: raise ExceptionForTesting try: f(10) except ExceptionForTesting: s = tb() for n in range(11): self.assert_(s.find('level%d' % n) >= 0, s) else: self.fail('no exception occurred') def testQuoteLastLine(self): class C: pass try: raise TypeError, C() except: s = tb(1) else: self.fail('no exception occurred') self.assert_(s.find('<') >= 0, s) self.assert_(s.find('>') >= 0, s) def test_suite(): return TestSuite(( makeSuite(Test), )) if __name__=='__main__': main(defaultTest='test_suite') ������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/zExceptions/ITracebackSupplement.py�������������������������������������0000664�0000764�0000764�00000005370�10435500330�023557� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ITracebackSupplement interface definition. $Id: ITracebackSupplement.py 24763 2004-05-17 05:59:28Z philikon $ """ from Interface import Interface from Interface.Attribute import Attribute class ITracebackSupplement(Interface): """Provides valuable information to supplement an exception traceback. The interface is geared toward providing meaningful feedback when exceptions occur in user code written in mini-languages like Zope page templates and restricted Python scripts. """ source_url = Attribute( 'source_url', """Optional. Set to URL of the script where the exception occurred. Normally this generates a URL in the traceback that the user can visit to manage the object. Set to None if unknown or not available. """ ) object = Attribute( 'object', """Optional. Set to the script or template where the exception occurred. Set to None if unknown or not available. """ ) line = Attribute( 'line', """Optional. Set to the line number (>=1) where the exception occurred. Set to 0 or None if the line number is unknown. """ ) column = Attribute( 'column', """Optional. Set to the column offset (>=0) where the exception occurred. Set to None if the column number is unknown. """ ) expression = Attribute( 'expression', """Optional. Set to the expression that was being evaluated. Set to None if not available or not applicable. """ ) warnings = Attribute( 'warnings', """Optional. Set to a sequence of warning messages. Set to None if not available, not applicable, or if the exception itself provides enough information. """ ) def getInfo(as_html=0): """Optional. Returns a string containing any other useful info. If as_html is set, the implementation must HTML-quote the result (normally using cgi.escape()). Returns None to provide no extra info. """ ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/�����������������������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�016041� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/��������������������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�016327� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/__init__.py���������������������������������������������������0000664�0000764�0000764�00000001345�10435500330�020430� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __doc__='''$Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $''' __version__='$Revision: 1.9 $'[11:-2] �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/����������������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�017127� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/__init__.py�����������������������������������������������0000775�0000764�0000764�00000001240�10435500330�021225� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.3 $'[11:-2] ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/ppml.py���������������������������������������������������0000775�0000764�0000764�00000050420�10435500330�020442� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Provide conversion between Python pickles and XML """ __version__ = "1.9" # Code version from pickle import * from string import replace import struct import base64 import string import pickle import tempfile import marshal import xyap mdumps = marshal.dumps mloads = marshal.loads xyap=xyap.xyap ListType=type([]) # Create repr mappong reprs = {} for c in map(chr,range(256)): reprs[c] = repr(c)[1:-1] reprs['\n'] = "\\n\n" reprs['\t'] = "\\t" reprs['\\'] = "\\\\" reprs['\r'] = "\\r" reprs["'"] = "\\'" reprs2={} reprs2['<'] = "\\074" reprs2['>'] = "\\076" reprs2['&'] = "\\046" # Function convert takes a string and converts it to either # repr or base64 format def convert(S, find=string.find): new = '' encoding = 'repr' new = string.join(map(reprs.get, S), '') if len(new) > (1.4*len(S)): encoding = 'base64' new = base64.encodestring(S)[:-1] elif find(new,'>') >= 0 or find(new,'<') >= 0 or find(new,'&') >= 0: if find(new, ']]>') <0 : new='<![CDATA[\n\n'+new+'\n\n]]>' encoding='cdata' else: new=string.join(map(lambda s: reprs2.get(s,s), new), '') return encoding, new # Function unconvert takes a encoding and a string and # returns the original string def unconvert(encoding,S): original = '' if encoding == 'base64': original = base64.decodestring(S) else: x = string.replace(S, '\n', '') original = eval("'"+x+"'") return original t32 = 1L << 32 def p64(v, pack=struct.pack): if v < t32: h=0 else: h=v/t32 v=v%t32 return pack(">II", h, v) def u64(v, unpack=struct.unpack): h, v = unpack(">ii", v) if v < 0: v=t32+v if h: if h < 0: h=t32+h v=h*t32+v return v class Global: def __init__(self, module, name): self.module=module self.name=name def __str__(self, indent=0): if hasattr(self, 'id'): id=' id="%s"' % self.id else: id='' name=string.lower(self.__class__.__name__) return '%s<%s%s name="%s" module="%s"/>\n' % ( ' '*indent, name, id, self.name, self.module) class Scalar: def __init__(self, v): self._v=v def value(self): return self._v def __str__(self, indent=0): if hasattr(self, 'id'): id=' id="%s"' % self.id else: id='' name=string.lower(self.__class__.__name__) return '%s<%s%s>%s</%s>\n' % ( ' '*indent, name, id, self.value(), name) def xmlstr(v): v=`v` if v[:1]=='\'': v=string.replace(v,'"','\\"') v=replace(v,'%','\\045') v=replace(v,'&','\\046') return v[1:-1] class Int(Scalar): pass class Long(Scalar): def value(self): result = str(self._v) if result[-1:] == 'L': return result[:-1] return result class Float(Scalar): pass class String(Scalar): def __init__(self, v, encoding=''): encoding, v = convert(v) self.encoding=encoding self._v=v def __str__(self,indent=0): if hasattr(self,'id'):id=' id="%s"' % self.id else: id='' if hasattr(self, 'encoding'):encoding=' encoding="%s"' % self.encoding else: encoding='' name=string.lower(self.__class__.__name__) return '%s<%s%s%s>%s</%s>\n' % ( ' '*indent, name, id, encoding, self.value(), name) class Wrapper: def __init__(self, v): self._v=v def value(self): return self._v def __str__(self, indent=0): if hasattr(self, 'id'): id=' id="%s"' % self.id else: id='' name=string.lower(self.__class__.__name__) v=self._v i=' '*indent if isinstance(v,Scalar): return '%s<%s%s> %s </%s>\n' % (i, name, id, str(v)[:-1], name) else: try: v=v.__str__(indent+2) except TypeError: v=v.__str__() return '%s<%s%s>\n%s%s</%s>\n' % (i, name, id, v, i, name) class Collection: def __str__(self, indent=0): if hasattr(self, 'id'): id=' id="%s"' % self.id else: id='' name=string.lower(self.__class__.__name__) i=' '*indent if self: return '%s<%s%s>\n%s%s</%s>\n' % ( i, name, id, self.value(indent+2), i, name) else: return '%s<%s%s/>\n' % (i, name, id) class Key(Wrapper): pass class Value(Wrapper): pass class Dictionary(Collection): def __init__(self): self._d=[] def __len__(self): return len(self._d) def __setitem__(self, k, v): self._d.append((k,v)) def value(self, indent): return string.join( map(lambda i, ind=' '*indent, indent=indent+4: '%s<item>\n' '%s' '%s' '%s</item>\n' % (ind, Key(i[0]).__str__(indent), Value(i[1]).__str__(indent), ind), self._d ), '') class Sequence(Collection): def __init__(self, v=None): if not v: v=[] self._subs=v def __len__(self): return len(self._subs) def append(self, v): self._subs.append(v) def extend(self, v): self._subs.extend(v) def _stringify(self, v, indent): try: return v.__str__(indent+2) except TypeError: return v.__str__() def value(self, indent): return string.join(map( lambda v, indent=indent: self._stringify(v, indent), self._subs),'') class List(Sequence): pass class Tuple(Sequence): pass class Klass(Wrapper): pass class State(Wrapper): pass class Pickle(Wrapper): pass class Persistent(Wrapper): pass class none: def __str__(self, indent=0): return ' '*indent+'<none/>\n' none=none() class Reference(Scalar): def __init__(self, v): self._v=v def __str__(self, indent=0): v=self._v name=string.lower(self.__class__.__name__) return '%s<%s id="%s"/>\n' % (' '*indent,name,v) Get=Reference class Object(Sequence): def __init__(self, klass, args): self._subs=[Klass(klass), args] def __setstate__(self, v): self.append(State(v)) class ToXMLUnpickler(Unpickler): def load(self): return Pickle(Unpickler.load(self)) dispatch = {} dispatch.update(Unpickler.dispatch) def persistent_load(self, v): return Persistent(v) def load_persid(self): pid = self.readline()[:-1] self.append(self.persistent_load(String(pid))) dispatch[PERSID] = load_persid def load_none(self): self.append(none) dispatch[NONE] = load_none def load_int(self): self.append(Int(string.atoi(self.readline()[:-1]))) dispatch[INT] = load_int def load_binint(self): self.append(Int(mloads('i' + self.read(4)))) dispatch[BININT] = load_binint def load_binint1(self): self.append(Int(mloads('i' + self.read(1) + '\000\000\000'))) dispatch[BININT1] = load_binint1 def load_binint2(self): self.append(Int(mloads('i' + self.read(2) + '\000\000'))) dispatch[BININT2] = load_binint2 def load_long(self): self.append(Long(string.atol(self.readline()[:-1], 0))) dispatch[LONG] = load_long def load_float(self): self.append(Float(string.atof(self.readline()[:-1]))) dispatch[FLOAT] = load_float def load_binfloat(self, unpack=struct.unpack): self.append(Float(unpack('>d', self.read(8))[0])) dispatch[BINFLOAT] = load_binfloat def load_string(self): self.append(String(eval(self.readline()[:-1], {'__builtins__': {}}))) # Let's be careful dispatch[STRING] = load_string def load_binstring(self): len = mloads('i' + self.read(4)) self.append(String(self.read(len))) dispatch[BINSTRING] = load_binstring def load_short_binstring(self): len = mloads('i' + self.read(1) + '\000\000\000') self.append(String(self.read(len))) dispatch[SHORT_BINSTRING] = load_short_binstring def load_tuple(self): k = self.marker() self.stack[k:] = [Tuple(self.stack[k+1:])] dispatch[TUPLE] = load_tuple def load_empty_tuple(self): self.stack.append(Tuple()) dispatch[EMPTY_TUPLE] = load_empty_tuple def load_empty_list(self): self.stack.append(List()) dispatch[EMPTY_LIST] = load_empty_list def load_empty_dictionary(self): self.stack.append(Dictionary()) dispatch[EMPTY_DICT] = load_empty_dictionary def load_list(self): k = self.marker() self.stack[k:] = [List(self.stack[k+1:])] dispatch[LIST] = load_list def load_dict(self): k = self.marker() d = Dictionary() items = self.stack[k+1:] for i in range(0, len(items), 2): key = items[i] value = items[i+1] d[key] = value self.stack[k:] = [d] dispatch[DICT] = load_dict def load_inst(self): k = self.marker() args = Tuple(self.stack[k+1:]) del self.stack[k:] module = self.readline()[:-1] name = self.readline()[:-1] value=Object(Global(module, name), args) self.append(value) dispatch[INST] = load_inst def load_obj(self): stack = self.stack k = self.marker() klass = stack[k + 1] del stack[k + 1] args = Tuple(stack[k + 1:]) del stack[k:] value=Object(klass,args) self.append(value) dispatch[OBJ] = load_obj def load_global(self): module = self.readline()[:-1] name = self.readline()[:-1] self.append(Global(module, name)) dispatch[GLOBAL] = load_global def load_reduce(self): stack = self.stack callable = stack[-2] arg_tup = stack[-1] del stack[-2:] value=Object(callable, arg_tup) self.append(value) dispatch[REDUCE] = load_reduce idprefix='' def load_get(self): self.append(Get(self.idprefix+self.readline()[:-1])) dispatch[GET] = load_get def load_binget(self): i = mloads('i' + self.read(1) + '\000\000\000') self.append(Get(self.idprefix+`i`)) dispatch[BINGET] = load_binget def load_long_binget(self): i = mloads('i' + self.read(4)) self.append(Get(self.idprefix+`i`)) dispatch[LONG_BINGET] = load_long_binget def load_put(self): self.stack[-1].id=self.idprefix+self.readline()[:-1] dispatch[PUT] = load_put def load_binput(self): i = mloads('i' + self.read(1) + '\000\000\000') last = self.stack[-1] if getattr(last, 'id', last) is last: last.id = self.idprefix + `i` dispatch[BINPUT] = load_binput def load_long_binput(self): i = mloads('i' + self.read(4)) last = self.stack[-1] if getattr(last, 'id', last) is last: last.id = self.idprefix + `i` dispatch[LONG_BINPUT] = load_long_binput def ToXMLload(file): return ToXMLUnpickler(file).load() def ToXMLloads(str): file = StringIO(str) return ToXMLUnpickler(file).load() class NoBlanks: def handle_data(self, data): if string.strip(data): self.append(data) def name(self, tag, data, join=string.join, strip=string.strip): return strip(join(data[2:],'')) def start_pickle(self, tag, attrs): self._pickleids={} return [tag,attrs] def end_string(self, tag, data): v=data[2] a=data[1] if a['encoding'] is not '': v=unconvert(a['encoding'],v) if a.has_key('id'): self._pickleids[a['id']]=v return v def end_list(self, tag, data): v=data[2:] a=data[1] if a.has_key('id'): self._pickleids[data[1]['id']]=v return v def end_tuple(self, tag, data): v=tuple(data[2:]) a=data[1] if a.has_key('id'): self._pickleids[data[1]['id']]=v return v def end_dictionary(self, tag, data): D={} a=data[1] for k, v in data[2:]: D[k]=v if a.has_key('id'): self._pickleids[a['id']]=D return D class xmlUnpickler(NoBlanks, xyap): start_handlers={'pickle': start_pickle} end_handlers={ 'int': lambda self,tag,data,atoi=string.atoi,name=name: atoi(name(self, tag, data)), 'long': lambda self,tag,data,atoi=string.atoi,name=name: atoi(name(self, tag, data)), 'boolean': lambda self,tag,data,atoi=string.atoi,name=name: atoi(name(self, tag, data)), 'string': end_string , 'double': lambda self,tag,data,atof=string.atof,name=name: atof(name(self, tag, data)), 'float': lambda self,tag,data,atof=string.atof,name=name: atof(name(self, tag, data)), 'none': lambda self, tag, data: None, 'list': end_list, 'tuple': end_tuple, 'dictionary': end_dictionary, 'key': lambda self, tag, data: data[2], 'value': lambda self, tag, data: data[2], 'item': lambda self, tag, data: data[2:], 'reference': lambda self, tag, data: self._pickleids[data[1]['id']], 'state': lambda self, tag, data: data[2], 'klass': lambda self, tag, data: data[2], } def save_int(self, tag, data): binary=self.binary if binary: v=string.atoi(name(self, tag, data)) i=mdumps(v)[1:] if (i[-2:] == '\000\000'): if (i[-3] == '\000'): v='K'+i[:-3] return v v='M'+i[:-2] return v v='J'+i return v v='I'+name(self, tag, data)+'\012' return v def save_float(self, tag, data): binary=self.binary if binary: v='G'+struct.pack('>d',string.atof(name(self, tag, data))) else: v='F'+name(self, tag, data)+'\012' return v def save_put(self, v, attrs): id=attrs.get('id','') if id: prefix=string.rfind(id,'.') if prefix >= 0: id=id[prefix+1:] elif id[0]=='i': id=id[1:] if self.binary: id=string.atoi(id) s=mdumps(id)[1:] if (id < 256): id=s[0] put='q' else: id=s put='r' id=put+id else: id="p"+id+"\012" return v+id return v def save_string(self, tag, data): binary=self.binary v='' a=data[1] if len(data)>2: for x in data[2:]: v=v+x encoding=a['encoding'] if encoding is not '': v=unconvert(encoding,v) put='p' if binary: l=len(v) s=mdumps(l)[1:] if (l<256): v='U'+s[0]+v else: v='T'+s+v put='q' else: v="S'"+v+"'\012" return save_put(self, v, a) def save_tuple(self, tag, data): T=data[2:] if not T: return ')' return save_put(self, '('+string.join(T,'')+'t', data[1]) def save_list(self, tag, data): L=data[2:] a=data[1] if self.binary: v=save_put(self, ']', a) if L: v=v+'('+string.join(L,'')+'e' else: v=save_put(self, '(l', a) if L: v=string.join(L,'a')+'a' return v def save_dict(self, tag, data): D=data[2:] if self.binary: v=save_put(self, '}', data[1]) if D: v=v+'('+string.join(D,'')+'u' else: v=save_put(self, '(d', data[1]) if D: v=v+string.join(D,'s')+'s' return v def save_reference(self, tag, data): binary=self.binary a=data[1] id=a['id'] prefix=string.rfind(id,'.') if prefix>=0: id=id[prefix+1:] get='g' if binary: id=string.atoi(id) s=mdumps(id)[1:] if (id < 256): id=s[0] get='h' else: id=s get='j' v=get+id else: v=get+id+'\012' return v def save_object(self, tag, data): v='('+data[2] x=data[3][1:] stop=string.rfind(x,'t') # This seems if stop>=0: x=x[:stop] # wrong! v=save_put(self, v+x+'o', data[1]) v=v+data[4]+'b' # state return v def save_global(self, tag, data): a=data[1] return save_put(self, 'c'+a['module']+'\012'+a['name']+'\012', a) def save_persis(self, tag, data): v=data[2] if self.binary: v=v+'Q' else: v='P'+v return v class xmlPickler(NoBlanks, xyap): start_handlers={ 'pickle': lambda self, tag, attrs: [tag, attrs], } end_handlers={ 'pickle': lambda self, tag, data: str(data[2])+'.', 'none': lambda self, tag, data: 'N', 'int': save_int, 'long': lambda self, tag, data: 'L'+str(data[2])+'L\012', 'float': save_float, 'string': save_string, 'reference': save_reference, 'tuple': save_tuple, 'list': save_list, 'dictionary': save_dict, 'item': lambda self, tag, data, j=string.join: j(data[2:],''), 'value': lambda self, tag, data: data[2], 'key' : lambda self, tag, data: data[2], 'object': save_object, 'klass': lambda self, tag, data: data[2], 'state': lambda self, tag, data: data[2], 'global': save_global, 'persistent': save_persis, } # The rest is used for testing only class C: def __cmp__(self, other): return cmp(self.__dict__, other.__dict__) def test(): import xmllib c=C() c.foo=1 c.bar=2 x=[0,1,2,3] y=('abc','abc',c,c) x.append(y) x.append(y) t=() l=[] s='' L = long('999999999999') x.append(t) x.append(l) x.append(s) x.append(L) x.append(55555) x.append(13) r=[x] print x f=pickle.dumps(x) print f r.append(f) q=ToXMLloads(f) q=str(q) q='<?xml version="1.0"?>\n'+q print q r.append(q) file='' F=xmlPickler(file) p=xmllib.XMLParser() p.start_handlers=F.start_handlers p.end_handlers=F.end_handlers p.handle_data=F.handle_data p.unknown_starttag=F.unknown_starttag p.unknown_endtag=F.unknown_endtag p._stack=F._stack p.push=F.push p.append=F.append p.file=F.file p.tempfile=F.tempfile p.binary=1 data=string.split(q,'\n') for l in data: p.feed(l) p.close() z=p._stack z=z[0][0] print z, '\012' r.append(z) l=pickle.loads(z) print l, '\012' r.append(l) def test1(): import xmllib q=open('Data.xml').read() file=open('out','w'+'b') F=xmlPickler(file,1) p=xmllib.XMLParser() p.start_handlers=F.start_handlers p.end_handlers=F.end_handlers p.handle_data=F.handle_data p.unknown_starttag=F.unknown_starttag p.unknown_endtag=F.unknown_endtag p._stack=F._stack p.push=F.push p.append=F.append p.file=F.file p.tempfile=F.tempfile data=string.split(q,'\n') for l in data: p.feed(l) p.close() z=p._stack z=z[0][0] print z, '\012' def test2(): import xml.parsers.pyexpat c=C() c.foo=1 c.bar=2 x=[0,1,2,3] y=('abc','abc',c,c) x.append(y) x.append(y) t=() l=[] s='' L = long('999999999999') x.append(t) x.append(l) x.append(s) x.append(L) x.append(5) x.append(13) print x, '\012' f=pickle.dumps(x) print f, '\012' q=ToXMLloads(f) q=str(q) q='<?xml version="1.0"?>\n'+q print q, '\012' file='' F=xmlPickler() F.binary=0 p=xml.parsers.pyexpat.ParserCreate() p.CharacterDataHandler=F.handle_data p.StartElementHandler=F.unknown_starttag p.EndElementHandler=F.unknown_endtag r=p.Parse(q) print r, '\012' def test3(): import xml.parsers.pyexpat data=open('Data.xml').read() file=open('out','w'+'b') F=xmlPickler() F.file=file F.binary=1 p=xml.parsers.pyexpat.ParserCreate() p.CharacterDataHandler=F.handle_data p.StartElementHandler=F.unknown_starttag p.EndElementHandler=F.unknown_endtag r=p.Parse(data) print r, '\012' if __name__ == '__main__': test() ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/xyap.py���������������������������������������������������0000664�0000764�0000764�00000011103�10435500330�020443� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������"""Yet another XML parser This is meant to be very simple: - stack based - The parser has a table of start handlers and end handlers. - start tag handlers are called with the parser instance, tag names and attributes. The result is placed on the stack. The default handler places a special object on the stack (uh, a list, with the tag name and attributes as the first two elements. ;) - end tag handlers are called with the object on the parser, the tag name, and top of the stack right after it has been removed. The result is appended to the object on the top of the stack. Note that namespace attributes should recieve some special handling. Oh well. """ import string import xmllib from pickle import * from types import ListType class xyap: start_handlers={} end_handlers={} def __init__(self): top=[] self._stack=_stack=[top] self.push=_stack.append self.append=top.append def handle_data(self, data): self.append(data) def unknown_starttag(self, tag, attrs): if type(attrs) is ListType: x=0 temp={} while x<len(attrs): temp[attrs[x]]=attrs[x+1] x=x+2 attrs=temp start=self.start_handlers if start.has_key(tag): tag = start[tag](self, tag, attrs) else: tag = [tag, attrs] self.push(tag) self.append=tag.append def unknown_endtag(self, tag): _stack=self._stack top=_stack[-1] del _stack[-1] append=self.append=_stack[-1].append end=self.end_handlers if end.has_key(tag): top=end[tag](self, tag, top) append(top) class NoBlanks: def handle_data(self, data): if string.strip(data): self.append(data) def struct(self, tag, data): r={} for k, v in data[2:]: r[k]=v return r def name(self, tag, data, join=string.join, strip=string.strip): return strip(join(data[2:],'')) def tuplef(self, tag, data): return tuple(data[2:]) class XYap(xyap, xmllib.XMLParser): def __init__(self): xmllib.XMLParser.__init__(self) top=[] self._stack=_stack=[top] self.push=_stack.append self.append=top.append class xmlrpc(NoBlanks, XYap, xmllib.XMLParser): end_handlers={ 'methodCall': tuplef, 'methodName': name, 'params': tuplef, 'param': lambda self, tag, data: data[2], 'value': lambda self, tag, data: data[2], 'i4': lambda self, tag, data, atoi=string.atoi, name=name: atoi(name(self, tag, data)), 'int': lambda self, tag, data, atoi=string.atoi, name=name: atoi(name(self, tag, data)), 'boolean': lambda self, tag, data, atoi=string.atoi, name=name: atoi(name(self, tag, data)), 'string': lambda self, tag, data, join=string.join: join(data[2:], ''), 'double': lambda self, tag, data, atof=string.atof, name=name: atof(name(self, tag, data)), 'float': lambda self, tag, data, atof=string.atof, name=name: atof(name(self, tag, data)), 'struct': struct, 'member': tuplef, 'name': name, 'array': lambda self, tag, data: data[2], 'data': lambda self, tag, data: data[2:], } def test(): data="""<?xml version="1.0"?> <methodCall> <methodName>examples.getStateName </methodName> <params> <param> <value><i4>41</i4></value> </param> <param><value> <struct> <member> <name>lowerBound</name> <value><i4>18</i4></value> </member> <member> <name>upperBound</name> <value><i4>139</i4></value> </member> </struct></value> </param> <param><value> <array> <data> <value><i4>12</i4></value> <value><string>Egypt</string></value> <value><boolean>0</boolean></value> <value><i4>-31</i4></value> </data> </array></value> </param> </params> </methodCall> """ data=string.split(data,'\n') r=[] for C in XYap, xmlrpc: p=C() for l in data: p.feed(l) p.close() r.append(p._stack) return r if __name__=='__main__': print test() �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/��������������������������������������������������0000775�0000764�0000764�00000000000�10435610645�020621� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/_checkversion.py����������������������������������0000664�0000764�0000764�00000000640�10435500330�024002� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������"""This file (which is sourced, not imported) checks the version of the "versioncheck" package. It is also an example of how to format your own _checkversion.py file""" import pyversioncheck _PACKAGE="pyexpat" _VERSION="1.3" _URL="http://www.cwi.nl/~jack/pyexpatversion.txt" try: _myverbose=VERBOSE except NameError: _myverbose=1 pyversioncheck.versioncheck(_PACKAGE, _URL, _VERSION, verbose=_myverbose) ������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/pyexpat.py����������������������������������������0000664�0000764�0000764�00000001222�10435500330�022647� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from dcpyexpat import * ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/README��������������������������������������������0000664�0000764�0000764�00000005763�10435500330�021501� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Python Expat wrapper module, version 1.3, 15-Oct-98 =================================================== If you have downloaded the binary distribution for the macintosh you can skip the "building" sections and go straight to the "using" bit. If you are using a macintosh and do want to build from source you should get the pyexpat.tgz distribution (Stuffit Expander with Expander Enhancer will know how to unpack a gzipped tar file). Changes since version 1.2 ------------------------- Minimal changes, really: a memory leak in the callback routine was plugged, and exceptions in the callback are now passed through to the mainline code. The module also exports a version attribute. Building the pyexpat module under unix -------------------------------------- - Build libexpat.a in expat. This version is very slightly different from the original by James Clark (the libexpat.a target was added, and a few C++ comments were replaced by C comments). - Edit Makefile.pre.in and set your installdir - make -f Makefile.pre.in VERSION=1.5.1 Makefile - make sharedmods - put the shared module somewhere in your sys.path (if you want a static Python edit Setup.in, and replace the last line with "make"). Building the pyexpat module on the macintosh -------------------------------------------- - Unpack the various .hqx project files. - All the projects are linked, so building pyexpat.prj should build everything. If this doesn't work you will find the library subprojects to build in the expat folder. - Use EditPythonPrefs to add the current folder to sys.path. Using the pyexpat module ----------------------- The pyexpat module exports two functions: ParserCreate(encoding) Creates a new parser object. The optional encoding arg (a string) specifies the encoding. ErrorString(number) Return a string corresponding to the given error number. Parser objects have one method: Parse(data, isfinal) Parse some data. If the optional isfinal arg is 1 this is the last bit of data. Returns true if parsing succeeded (so far), otherwise the error attributes have information on the error. Parser objects have the following attributes: StartElementHandler, EndElementHandler, CharacterDataHandler, ProcessingInstructionHandler - The Python handlers called for various events. See below for the signatures. ErrorCode, ErrorLineNumber, ErrorColumnNumber, ErrorByteIndex - Readonly integers giving information on the current parse error. Testing it ---------- There's a very minimal test script in expattest.py. It should be easy to adapt it to generate ESIS (but I'm not familiar enough with ESIS to do it). This module parsed Hamlet in 2 seconds on an 180 Mhz R5000 SGI O2. Feedback -------- This module currently lives at ftp://ftp.cwi.nl/pub/jack/python/pyexpatsrc.tgz (source) and ftp://ftp.cwi.nl/pub/jack/python/pyexpat.hqx (macintosh binary-only). The author is Jack Jansen, jack@cwi.nl. The expat proper was written by James Clark and can be found at http://www.jclark.com/xml/ . Jack Jansen, CWI, Amsterdam jack@cwi.nl �������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/__init__.py���������������������������������������0000664�0000764�0000764�00000001220�10435500330�022712� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from pyexpat import * ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/Setup���������������������������������������������0000664�0000764�0000764�00000000253�10435500330�021631� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������*shared* dcpyexpat dcpyexpat.c -DXML_NS -Iexpat/xmlparse -Iexpat/xmltok expat/xmlparse/xmlparse.c expat/xmlparse/hashtable.c expat/xmltok/xmlrole.c expat/xmltok/xmltok.c �����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/dcpyexpat.c���������������������������������������0000664�0000764�0000764�00000050755�10435500330�022767� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/*********************************************************** Copyright 1991-1995 by Stichting Mathematisch Centrum, Amsterdam, The Netherlands. All Rights Reserved Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the names of Stichting Mathematisch Centrum or CWI or Corporation for National Research Initiatives or CNRI not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission. While CWI is the initial source for this software, a modified version is made available by the Corporation for National Research Initiatives (CNRI) at the Internet address ftp://ftp.python.org. STICHTING MATHEMATISCH CENTRUM AND CNRI DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM OR CNRI BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. ******************************************************************/ #include "Python.h" #include "xmlparse.h" #include <setjmp.h> static PyObject *ErrorObject; /* ----------------------------------------------------- */ /* Declarations for objects of type xmlparser */ typedef struct { PyObject_HEAD XML_Parser itself; PyObject *StartElementHandler; PyObject *EndElementHandler; PyObject *CharacterDataHandler; PyObject *ProcessingInstructionHandler; PyObject *CommentHandler; PyObject *StartCdataSectionHandler; PyObject *EndCdataSectionHandler; PyObject *DefaultHandler; PyObject *UnparsedEntityDeclHandler; PyObject *NotationDeclHandler; PyObject *StartNamespaceDeclHandler; PyObject *EndNamespaceDeclHandler; PyObject *NotStandaloneHandler; int attrdict; int jmpbuf_valid; jmp_buf jmpbuf; } xmlparseobject; staticforward PyTypeObject Xmlparsetype; /* Callback routines */ static void my_StartElementHandler(void *userdata, const char *name, const char **atts) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; PyObject *attrs_obj; int attrs_len; const char **attrs_p; const char **attrs_k = atts; if (self->StartElementHandler != Py_None) { if (self->attrdict) { if( (attrs_obj = PyDict_New()) == NULL ) goto err; for(attrs_len=0, attrs_p = atts; *attrs_p; attrs_p++, attrs_len++) { if (attrs_len%2) { rv=PyString_FromString(*attrs_p); if (! rv) { Py_DECREF(attrs_obj); goto err; } if (PyDict_SetItemString( attrs_obj, (char*)*attrs_k, rv) < 0) { Py_DECREF(attrs_obj); goto err; } Py_DECREF(rv); } else attrs_k=attrs_p; } } else { for(attrs_len=0, attrs_p = atts; *attrs_p; attrs_p++, attrs_len++); if( (attrs_obj = PyList_New(attrs_len)) == NULL ) goto err; for(attrs_len=0, attrs_p = atts; *attrs_p; attrs_p++, attrs_len++) { rv=PyString_FromString(*attrs_p); if (! rv) { Py_DECREF(attrs_obj); goto err; } PyList_SET_ITEM(attrs_obj, attrs_len, rv); } } args = Py_BuildValue("(sO)", name, attrs_obj); Py_DECREF(attrs_obj); if (!args) goto err; rv = PyEval_CallObject(self->StartElementHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in StartElementHandler()\n"); PyErr_Clear(); } static void my_EndElementHandler(void *userdata, const char *name) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->EndElementHandler != Py_None) { args = Py_BuildValue("(s)", name); if (!args) goto err; rv = PyEval_CallObject(self->EndElementHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in EndElementHandler()\n"); PyErr_Clear(); } static void my_DefaultHandler(void *userdata, const char *data, int len) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->DefaultHandler != Py_None) { args = Py_BuildValue("(s#)", data, len); if (!args) goto err; rv = PyEval_CallObject(self->DefaultHandler, args); Py_XDECREF(args); if (rv == NULL) goto err; Py_XDECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in DefaultHandler()\n"); PyErr_Clear(); } static void my_CharacterDataHandler(void *userdata, const char *data, int len) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->CharacterDataHandler != Py_None) { args = Py_BuildValue("(s#)", data, len); if (!args) goto err; rv = PyEval_CallObject(self->CharacterDataHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } else my_DefaultHandler(userdata, data, len); return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in CharacterDataHandler()\n"); PyErr_Clear(); } static void my_ProcessingInstructionHandler(void *userdata, const char *target, const char *data) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->ProcessingInstructionHandler != Py_None) { args = Py_BuildValue("(ss)", target, data); if (!args) goto err; rv = PyEval_CallObject(self->ProcessingInstructionHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in ProcessingInstructionHandler()\n"); PyErr_Clear(); } static void my_CommentHandler(void *userdata, const char *data) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->CommentHandler != Py_None) { args = Py_BuildValue("(s)", data); if (!args) goto err; rv = PyEval_CallObject(self->CommentHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in CommentHandler()\n"); PyErr_Clear(); } static void my_StartCdataSectionHandler(void *userdata) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *rv; if (self->StartCdataSectionHandler != Py_None) { rv = PyEval_CallObject(self->StartCdataSectionHandler, NULL); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in StartCdataSectionHandler()\n"); PyErr_Clear(); } static void my_EndCdataSectionHandler(void *userdata) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *rv; if (self->EndCdataSectionHandler != Py_None) { rv = PyEval_CallObject(self->EndCdataSectionHandler, NULL); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in EndCdataSectionHandler()\n"); PyErr_Clear(); } static void my_UnparsedEntityDeclHandler(void * userdata, const char *entityName, const char *base, const char *systemId, const char *publicId, const char *notationName) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args, *rv=0; if (self->UnparsedEntityDeclHandler != Py_None) { args = Py_BuildValue("sssss", entityName, base, systemId, publicId, notationName); if (args == NULL) goto err; rv = PyEval_CallObject(self->UnparsedEntityDeclHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in UnparsedEntityDeclHandler()\n"); PyErr_Clear(); } static void my_NotationDeclHandler(void * userdata, const char *notationName, const char *base, const char *systemId, const char *publicId) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args, *rv=0; if (self->NotationDeclHandler != Py_None) { args = Py_BuildValue("ssss", notationName, base, systemId, publicId); if (args == NULL) goto err; rv = PyEval_CallObject(self->NotationDeclHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in NotationDeclHandler()\n"); PyErr_Clear(); } static void my_StartNamespaceDeclHandler(void *userdata, const char *prefix, const char *uri) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->StartNamespaceDeclHandler != Py_None) { args = Py_BuildValue("(ss)", prefix, uri); if (!args) goto err; rv = PyEval_CallObject(self->StartNamespaceDeclHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in StartNamespaceDeclHandler()\n"); PyErr_Clear(); } static void my_EndNamespaceDeclHandler(void *userdata, const char *prefix) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *args; PyObject *rv; if (self->EndNamespaceDeclHandler != Py_None) { args = Py_BuildValue("(s)", prefix); if (!args) goto err; rv = PyEval_CallObject(self->EndNamespaceDeclHandler, args); Py_DECREF(args); if (rv == NULL) goto err; Py_DECREF(rv); } return; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in EndNamespaceDeclHandler()\n"); PyErr_Clear(); } static int my_NotStandaloneHandler(void *userdata) { xmlparseobject *self = (xmlparseobject *)userdata; PyObject *rv; int irv; if (self->NotStandaloneHandler != Py_None) { rv = PyEval_CallObject(self->StartCdataSectionHandler, NULL); if (rv == NULL) goto err; irv=PyObject_IsTrue(rv); Py_DECREF(rv); return irv; } return 1; err: if (self->jmpbuf_valid) longjmp(self->jmpbuf, 1); PySys_WriteStderr("Exception in NotStandaloneHandler()\n"); PyErr_Clear(); return 0; } /* ---------------------------------------------------------------- */ static char xmlparse_Parse__doc__[] = "(data [,isfinal]) - Parse XML data" ; static PyObject * xmlparse_Parse(xmlparseobject *self, PyObject *args) { char *s; int slen; int isFinal = 0; int rv; if (!PyArg_ParseTuple(args, "s#|i", &s, &slen, &isFinal)) return NULL; if (setjmp(self->jmpbuf)) { /* Error in callback routine */ return NULL; } self->jmpbuf_valid = 1; rv = XML_Parse(self->itself, s, slen, isFinal); self->jmpbuf_valid = 0; return Py_BuildValue("i", rv); } static struct PyMethodDef xmlparse_methods[] = { {"Parse", (PyCFunction)xmlparse_Parse, METH_VARARGS, xmlparse_Parse__doc__}, {NULL, NULL} /* sentinel */ }; /* ---------- */ static xmlparseobject * newxmlparseobject(char *encoding, char *namespace_separator, int attrdict) { xmlparseobject *self; self = PyObject_NEW(xmlparseobject, &Xmlparsetype); if (self == NULL) return NULL; #define INIT_HANDLER(N) self->N=Py_None; Py_INCREF(Py_None) INIT_HANDLER(StartElementHandler); INIT_HANDLER(EndElementHandler); INIT_HANDLER(CharacterDataHandler); INIT_HANDLER(ProcessingInstructionHandler); INIT_HANDLER(CommentHandler); INIT_HANDLER(StartCdataSectionHandler); INIT_HANDLER(EndCdataSectionHandler); INIT_HANDLER(DefaultHandler); INIT_HANDLER(UnparsedEntityDeclHandler); INIT_HANDLER(NotationDeclHandler); INIT_HANDLER(StartNamespaceDeclHandler); INIT_HANDLER(EndNamespaceDeclHandler); INIT_HANDLER(NotStandaloneHandler); #undef INIT_HANDLER self->attrdict=attrdict; if (namespace_separator) { if ((self->itself = XML_ParserCreateNS(encoding, *namespace_separator)) == NULL ) { PyErr_SetString(PyExc_RuntimeError, "XML_ParserCreateNS failed"); Py_DECREF(self); return NULL; } } else { if ((self->itself = XML_ParserCreate(encoding)) == NULL ) { PyErr_SetString(PyExc_RuntimeError, "XML_ParserCreate failed"); Py_DECREF(self); return NULL; } } XML_SetUserData(self->itself, (void *)self); return self; } static void xmlparse_dealloc(xmlparseobject *self) { Py_DECREF(self->StartElementHandler); Py_DECREF(self->EndElementHandler); Py_DECREF(self->CharacterDataHandler); Py_DECREF(self->ProcessingInstructionHandler); Py_DECREF(self->CommentHandler); Py_DECREF(self->StartCdataSectionHandler); Py_DECREF(self->EndCdataSectionHandler); Py_DECREF(self->DefaultHandler); Py_DECREF(self->UnparsedEntityDeclHandler); Py_DECREF(self->NotationDeclHandler); Py_DECREF(self->StartNamespaceDeclHandler); Py_DECREF(self->EndNamespaceDeclHandler); Py_DECREF(self->NotStandaloneHandler); if (self->itself) XML_ParserFree(self->itself); self->itself = NULL; PyMem_DEL(self); } static PyObject * xmlparse_getattr(xmlparseobject *self, char *name) { #define GET_HANDLER(N) \ if (strcmp(name, #N) == 0) { \ Py_INCREF(self->N); \ return self->N; \ } GET_HANDLER(StartElementHandler); GET_HANDLER(EndElementHandler); GET_HANDLER(CharacterDataHandler); GET_HANDLER(ProcessingInstructionHandler); GET_HANDLER(CommentHandler); GET_HANDLER(StartCdataSectionHandler); GET_HANDLER(EndCdataSectionHandler); GET_HANDLER(DefaultHandler); GET_HANDLER(UnparsedEntityDeclHandler); GET_HANDLER(NotationDeclHandler); GET_HANDLER(StartNamespaceDeclHandler); GET_HANDLER(EndNamespaceDeclHandler); GET_HANDLER(NotStandaloneHandler); #undef GET_HANDLER if (strcmp(name, "ErrorCode") == 0) return Py_BuildValue("l", (long)XML_GetErrorCode(self->itself)); if (strcmp(name, "ErrorLineNumber") == 0) return Py_BuildValue("l", (long)XML_GetErrorLineNumber(self->itself)); if (strcmp(name, "ErrorColumnNumber") == 0) return Py_BuildValue("l", (long)XML_GetErrorColumnNumber(self->itself)); if (strcmp(name, "ErrorByteIndex") == 0) return Py_BuildValue("l", XML_GetErrorByteIndex(self->itself)); if (strcmp(name, "__members__") == 0) return Py_BuildValue("sssssssssssssssss", "StartElementHandler", "EndElementHandler", "CharacterDataHandler", "ProcessingInstructionHandler", "CommentHandler", "StartCdataSectionHandler", "EndCdataSectionHandler", "DefaultHandler", "UnparsedEntityDeclHandler", "NotationDeclHandler", "StartNamespaceDeclHandler", "EndNamespaceDeclHandler", "NotStandaloneHandler", "ErrorCode", "ErrorLineNumber", "ErrorColumnNumber", "ErrorByteIndex"); return Py_FindMethod(xmlparse_methods, (PyObject *)self, name); } static int xmlparse_setattr(xmlparseobject *self, char *name, PyObject *v) { /* Set attribute 'name' to value 'v'. v==NULL means delete */ if (v==NULL) v=Py_None; #define XMLSETHANDLERS(N) XML_Set ## N(self->itself, \ my_Start ## N, my_End ## N) #define SET_HANDLER(N) Py_XDECREF(self->N); self->N = v; Py_INCREF(v); if (strncmp(name, "Start", 5)==0) { if (strcmp(name+5, "ElementHandler") == 0) { SET_HANDLER(StartElementHandler); XMLSETHANDLERS(ElementHandler); return 0; } if (strcmp(name+5, "CdataSectionHandler") == 0) { SET_HANDLER(StartCdataSectionHandler); XMLSETHANDLERS(CdataSectionHandler); return 0; } if (strcmp(name+5, "NamespaceDeclHandler") == 0) { SET_HANDLER(StartNamespaceDeclHandler); XMLSETHANDLERS(NamespaceDeclHandler); return 0; } } else if (strncmp(name, "End", 3)==0) { if (strcmp(name+3, "ElementHandler") == 0) { SET_HANDLER(EndElementHandler); XMLSETHANDLERS(ElementHandler); return 0; } if (strcmp(name+3, "CdataSectionHandler") == 0) { SET_HANDLER(EndCdataSectionHandler); XMLSETHANDLERS(CdataSectionHandler); return 0; } if (strcmp(name+3, "NamespaceDeclHandler") == 0) { SET_HANDLER(EndNamespaceDeclHandler); XMLSETHANDLERS(NamespaceDeclHandler); return 0; } } #undef SET_HANDLER #undef XMLSETHANDLERS else { #define SET_HANDLER(N) \ if (strcmp(name, #N) == 0) { \ Py_XDECREF(self->N); \ self->N = v; \ Py_INCREF(v); \ XML_Set ## N(self->itself, my_ ## N); \ return 0; \ } SET_HANDLER(CharacterDataHandler); SET_HANDLER(ProcessingInstructionHandler); SET_HANDLER(CommentHandler); SET_HANDLER(DefaultHandler); SET_HANDLER(UnparsedEntityDeclHandler); SET_HANDLER(NotationDeclHandler); SET_HANDLER(NotStandaloneHandler); } #undef SET_HANDLER PyErr_SetString(PyExc_AttributeError, name); return -1; } static PyTypeObject Xmlparsetype = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ "xmlparser", /*tp_name*/ sizeof(xmlparseobject), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)xmlparse_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)xmlparse_getattr, /*tp_getattr*/ (setattrfunc)xmlparse_setattr, /*tp_setattr*/ (cmpfunc)0, /*tp_compare*/ (reprfunc)0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)0, /*tp_call*/ (reprfunc)0, /*tp_str*/ /* Space for future expansion */ 0L,0L,0L,0L, "XML parser" }; /* End of code for xmlparser objects */ /* -------------------------------------------------------- */ static char pyexpat_ParserCreate__doc__[] = "([encoding, namespace_separator]) - Return a new XML parser object" ; static PyObject * pyexpat_ParserCreate(PyObject *notused, PyObject *args, PyObject *kw) { char *encoding = NULL, *namespace_separator=NULL; PyObject *attrdict=NULL; static char *kwlist[] = {"encoding", "namespace_separator", "attrdict", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kw, "|zsO", kwlist, &encoding, &namespace_separator, &attrdict)) return NULL; return (PyObject *)newxmlparseobject(encoding, namespace_separator, attrdict ? PyObject_IsTrue(attrdict) : 0); } static char pyexpat_ErrorString__doc__[] = "(errno) Returns string error for given number" ; static PyObject * pyexpat_ErrorString(PyObject *notused, PyObject *args) { long code; if (!PyArg_ParseTuple(args, "l", &code)) return NULL; return Py_BuildValue("z", XML_ErrorString((int)code)); } /* List of methods defined in the module */ static struct PyMethodDef pyexpat_methods[] = { {"ParserCreate", (PyCFunction)pyexpat_ParserCreate, METH_VARARGS | METH_KEYWORDS, pyexpat_ParserCreate__doc__}, {"ErrorString", (PyCFunction)pyexpat_ErrorString, METH_VARARGS, pyexpat_ErrorString__doc__}, {NULL, (PyCFunction)NULL, 0, NULL} /* sentinel */ }; /* Initialization function for the module (*must* be called initpyexpat) */ static char pyexpat_module_documentation[] = "$Id: dcpyexpat.c 28839 2005-01-15 11:14:22Z andreasjung $" ; void initdcpyexpat(void) { PyObject *m, *d; static const char *rev="$Revision: 1.4 $"; Xmlparsetype.ob_type = &PyType_Type; m = Py_InitModule4("dcpyexpat", pyexpat_methods, pyexpat_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); d = PyModule_GetDict(m); ErrorObject = PyString_FromString("pyexpat.error"); PyDict_SetItemString(d, "error", ErrorObject); PyDict_SetItemString(d, "API_Revision", PyString_FromString(rev)); #define MYCONST(name) \ PyDict_SetItemString(d, #name, PyInt_FromLong(name)) MYCONST(XML_ERROR_NONE); MYCONST(XML_ERROR_NO_MEMORY); MYCONST(XML_ERROR_SYNTAX); MYCONST(XML_ERROR_NO_ELEMENTS); MYCONST(XML_ERROR_INVALID_TOKEN); MYCONST(XML_ERROR_UNCLOSED_TOKEN); MYCONST(XML_ERROR_PARTIAL_CHAR); MYCONST(XML_ERROR_TAG_MISMATCH); MYCONST(XML_ERROR_DUPLICATE_ATTRIBUTE); MYCONST(XML_ERROR_JUNK_AFTER_DOC_ELEMENT); MYCONST(XML_ERROR_PARAM_ENTITY_REF); MYCONST(XML_ERROR_UNDEFINED_ENTITY); MYCONST(XML_ERROR_RECURSIVE_ENTITY_REF); MYCONST(XML_ERROR_ASYNC_ENTITY); MYCONST(XML_ERROR_BAD_CHAR_REF); MYCONST(XML_ERROR_BINARY_ENTITY_REF); MYCONST(XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF); MYCONST(XML_ERROR_MISPLACED_XML_PI); MYCONST(XML_ERROR_UNKNOWN_ENCODING); MYCONST(XML_ERROR_INCORRECT_ENCODING); /* Check for errors */ if (PyErr_Occurred()) Py_FatalError("can't initialize module pyexpat"); } �������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/pyexpattest.py������������������������������������0000664�0000764�0000764�00000003725�10435500330�023561� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������# Very simple test - Parse a file and print what happens import sys import pyexpat import sys import os class Outputter: def __init__(self, verbose=0): self.startcount = 0 self.endcount = 0 self.cdatacount = 0 self.instcount = 0 self.verbose = verbose def startelt(self, name, attrs): self.startcount = self.startcount + 1 if self.verbose: print 'start', name for i in range(0, len(attrs), 2): print 'attr', attrs[i], attrs[i+1] def endelt(self, name): self.endcount = self.endcount + 1 if self.verbose: print 'end', name def cdata(self, data): self.cdatacount = self.cdatacount + 1 if self.verbose: print 'cdata', data def inst(self, target, data): self.instcount = self.instcount + 1 if self.verbose: print 'inst', target, data if len(sys.argv) > 1 and sys.argv[1] == '-v': out = Outputter(verbose=1) del sys.argv[1] else: out = Outputter() if len(sys.argv) != 2: if os.name == 'mac': import macfs fss, ok = macfs.StandardGetFile() if not ok: sys.exit(0) sys.argv.append(fss.as_pathname()) else: print 'Usage: pyexpattest [-v] inputfile' sys.exit(1) parser = pyexpat.ParserCreate() parser.StartElementHandler = out.startelt parser.EndElementHandler = out.endelt parser.CharacterDataHandler = out.cdata parser.ProcessingInstructionHandler = out.inst data = open(sys.argv[1]).read() rv = parser.Parse(data, 1) print 'Parser returned', rv if rv == 0: print '** Error', parser.ErrorCode print '** Line', parser.ErrorLineNumber print '** Column', parser.ErrorColumnNumber print '** Byte', parser.ErrorByteIndex print 'Summary of XML parser upcalls:' print 'start elements:', out.startcount print 'end elements:', out.endcount print 'character data:', out.cdatacount print 'processing instructions:', out.instcount �������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/��������������������������������������������0000775�0000764�0000764�00000000000�10435500330�021727� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/Makefile������������������������������������0000664�0000764�0000764�00000002020�10435500330�023361� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������CC=gcc # If you know what your system's byte order is, define XML_BYTE_ORDER: # use -DXML_BYTE_ORDER=12 for little-endian byte order; # use -DXML_BYTE_ORDER=21 for big-endian (network) byte order. # -DXML_NS adds support for checking of lexical aspects of XML namespaces spec # -DXML_MIN_SIZE makes a smaller but slower parser CFLAGS=-O2 -Ixmltok -Ixmlparse -DXML_NS # Use one of the next two lines; unixfilemap is better if it works. FILEMAP_OBJ=xmlwf/unixfilemap.o #FILEMAP_OBJ=xmlwf/readfilemap.o OBJS=xmltok/xmltok.o \ xmltok/xmlrole.o \ xmlwf/xmlwf.o \ xmlwf/xmlfile.o \ xmlwf/codepage.o \ xmlparse/xmlparse.o \ xmlparse/hashtable.o \ $(FILEMAP_OBJ) EXE= all: xmlwf/xmlwf$(EXE) xmlwf/xmlwf$(EXE): $(OBJS) $(CC) $(CFLAGS) -o $@ $(OBJS) clean: rm -f $(OBJS) xmlwf/xmlwf$(EXE) xmltok/nametab.h: gennmtab/gennmtab$(EXE) rm -f $@ gennmtab/gennmtab$(EXE) >$@ gennmtab/gennmtab$(EXE): gennmtab/gennmtab.c $(CC) $(CFLAGS) -o $@ gennmtab/gennmtab.c xmltok/xmltok.o: xmltok/nametab.h .c.o: $(CC) $(CFLAGS) -c -o $@ $< ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmlparse/�����������������������������������0000775�0000764�0000764�00000000000�10435500327�023570� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmlparse/xmlparse.c�������������������������0000664�0000764�0000764�00000257717�10435500327�025612� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include "xmldef.h" #include "xmlparse.h" #ifdef XML_UNICODE #define XML_ENCODE_MAX XML_UTF16_ENCODE_MAX #define XmlConvert XmlUtf16Convert #define XmlGetInternalEncoding XmlGetUtf16InternalEncoding #define XmlGetInternalEncodingNS XmlGetUtf16InternalEncodingNS #define XmlEncode XmlUtf16Encode #define MUST_CONVERT(enc, s) (!(enc)->isUtf16 || (((unsigned long)s) & 1)) typedef unsigned short ICHAR; #else #define XML_ENCODE_MAX XML_UTF8_ENCODE_MAX #define XmlConvert XmlUtf8Convert #define XmlGetInternalEncoding XmlGetUtf8InternalEncoding #define XmlGetInternalEncodingNS XmlGetUtf8InternalEncodingNS #define XmlEncode XmlUtf8Encode #define MUST_CONVERT(enc, s) (!(enc)->isUtf8) typedef char ICHAR; #endif #ifndef XML_NS #define XmlInitEncodingNS XmlInitEncoding #define XmlInitUnknownEncodingNS XmlInitUnknownEncoding #undef XmlGetInternalEncodingNS #define XmlGetInternalEncodingNS XmlGetInternalEncoding #define XmlParseXmlDeclNS XmlParseXmlDecl #endif #ifdef XML_UNICODE_WCHAR_T #define XML_T(x) L ## x #else #define XML_T(x) x #endif /* Round up n to be a multiple of sz, where sz is a power of 2. */ #define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1)) #include "xmltok.h" #include "xmlrole.h" #include "hashtable.h" #define INIT_TAG_BUF_SIZE 32 /* must be a multiple of sizeof(XML_Char) */ #define INIT_DATA_BUF_SIZE 1024 #define INIT_ATTS_SIZE 16 #define INIT_BLOCK_SIZE 1024 #define INIT_BUFFER_SIZE 1024 #define EXPAND_SPARE 24 typedef struct binding { struct prefix *prefix; struct binding *nextTagBinding; struct binding *prevPrefixBinding; const struct attribute_id *attId; XML_Char *uri; int uriLen; int uriAlloc; } BINDING; typedef struct prefix { const XML_Char *name; BINDING *binding; } PREFIX; typedef struct { const XML_Char *str; const XML_Char *localPart; int uriLen; } TAG_NAME; typedef struct tag { struct tag *parent; const char *rawName; int rawNameLength; TAG_NAME name; char *buf; char *bufEnd; BINDING *bindings; } TAG; typedef struct { const XML_Char *name; const XML_Char *textPtr; int textLen; const XML_Char *systemId; const XML_Char *base; const XML_Char *publicId; const XML_Char *notation; char open; } ENTITY; typedef struct block { struct block *next; int size; XML_Char s[1]; } BLOCK; typedef struct { BLOCK *blocks; BLOCK *freeBlocks; const XML_Char *end; XML_Char *ptr; XML_Char *start; } STRING_POOL; /* The XML_Char before the name is used to determine whether an attribute has been specified. */ typedef struct attribute_id { XML_Char *name; PREFIX *prefix; char maybeTokenized; char xmlns; } ATTRIBUTE_ID; typedef struct { const ATTRIBUTE_ID *id; char isCdata; const XML_Char *value; } DEFAULT_ATTRIBUTE; typedef struct { const XML_Char *name; PREFIX *prefix; int nDefaultAtts; int allocDefaultAtts; DEFAULT_ATTRIBUTE *defaultAtts; } ELEMENT_TYPE; typedef struct { HASH_TABLE generalEntities; HASH_TABLE elementTypes; HASH_TABLE attributeIds; HASH_TABLE prefixes; STRING_POOL pool; int complete; int standalone; const XML_Char *base; PREFIX defaultPrefix; } DTD; typedef struct open_internal_entity { const char *internalEventPtr; const char *internalEventEndPtr; struct open_internal_entity *next; ENTITY *entity; } OPEN_INTERNAL_ENTITY; typedef enum XML_Error Processor(XML_Parser parser, const char *start, const char *end, const char **endPtr); static Processor prologProcessor; static Processor prologInitProcessor; static Processor contentProcessor; static Processor cdataSectionProcessor; static Processor epilogProcessor; static Processor errorProcessor; static Processor externalEntityInitProcessor; static Processor externalEntityInitProcessor2; static Processor externalEntityInitProcessor3; static Processor externalEntityContentProcessor; static enum XML_Error handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName); static enum XML_Error processXmlDecl(XML_Parser parser, int isGeneralTextEntity, const char *, const char *); static enum XML_Error initializeEncoding(XML_Parser parser); static enum XML_Error doContent(XML_Parser parser, int startTagLevel, const ENCODING *enc, const char *start, const char *end, const char **endPtr); static enum XML_Error doCdataSection(XML_Parser parser, const ENCODING *, const char **startPtr, const char *end, const char **nextPtr); static enum XML_Error storeAtts(XML_Parser parser, const ENCODING *, const char *s, TAG_NAME *tagNamePtr, BINDING **bindingsPtr); static int addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId, const XML_Char *uri, BINDING **bindingsPtr); static int defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *, int isCdata, const XML_Char *dfltValue); static enum XML_Error storeAttributeValue(XML_Parser parser, const ENCODING *, int isCdata, const char *, const char *, STRING_POOL *); static enum XML_Error appendAttributeValue(XML_Parser parser, const ENCODING *, int isCdata, const char *, const char *, STRING_POOL *); static ATTRIBUTE_ID * getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start, const char *end); static int setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *); static enum XML_Error storeEntityValue(XML_Parser parser, const char *start, const char *end); static int reportProcessingInstruction(XML_Parser parser, const ENCODING *enc, const char *start, const char *end); static int reportComment(XML_Parser parser, const ENCODING *enc, const char *start, const char *end); static void reportDefault(XML_Parser parser, const ENCODING *enc, const char *start, const char *end); static const XML_Char *getContext(XML_Parser parser); static int setContext(XML_Parser parser, const XML_Char *context); static void normalizePublicId(XML_Char *s); static int dtdInit(DTD *); static void dtdDestroy(DTD *); static int dtdCopy(DTD *newDtd, const DTD *oldDtd); static void poolInit(STRING_POOL *); static void poolClear(STRING_POOL *); static void poolDestroy(STRING_POOL *); static XML_Char *poolAppend(STRING_POOL *pool, const ENCODING *enc, const char *ptr, const char *end); static XML_Char *poolStoreString(STRING_POOL *pool, const ENCODING *enc, const char *ptr, const char *end); static int poolGrow(STRING_POOL *pool); static const XML_Char *poolCopyString(STRING_POOL *pool, const XML_Char *s); static const XML_Char *poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n); #define poolStart(pool) ((pool)->start) #define poolEnd(pool) ((pool)->ptr) #define poolLength(pool) ((pool)->ptr - (pool)->start) #define poolChop(pool) ((void)--(pool->ptr)) #define poolLastChar(pool) (((pool)->ptr)[-1]) #define poolDiscard(pool) ((pool)->ptr = (pool)->start) #define poolFinish(pool) ((pool)->start = (pool)->ptr) #define poolAppendChar(pool, c) \ (((pool)->ptr == (pool)->end && !poolGrow(pool)) \ ? 0 \ : ((*((pool)->ptr)++ = c), 1)) typedef struct { /* The first member must be userData so that the XML_GetUserData macro works. */ void *m_userData; void *m_handlerArg; char *m_buffer; /* first character to be parsed */ const char *m_bufferPtr; /* past last character to be parsed */ char *m_bufferEnd; /* allocated end of buffer */ const char *m_bufferLim; long m_parseEndByteIndex; const char *m_parseEndPtr; XML_Char *m_dataBuf; XML_Char *m_dataBufEnd; XML_StartElementHandler m_startElementHandler; XML_EndElementHandler m_endElementHandler; XML_CharacterDataHandler m_characterDataHandler; XML_ProcessingInstructionHandler m_processingInstructionHandler; XML_CommentHandler m_commentHandler; XML_StartCdataSectionHandler m_startCdataSectionHandler; XML_EndCdataSectionHandler m_endCdataSectionHandler; XML_DefaultHandler m_defaultHandler; XML_UnparsedEntityDeclHandler m_unparsedEntityDeclHandler; XML_NotationDeclHandler m_notationDeclHandler; XML_StartNamespaceDeclHandler m_startNamespaceDeclHandler; XML_EndNamespaceDeclHandler m_endNamespaceDeclHandler; XML_NotStandaloneHandler m_notStandaloneHandler; XML_ExternalEntityRefHandler m_externalEntityRefHandler; void *m_externalEntityRefHandlerArg; XML_UnknownEncodingHandler m_unknownEncodingHandler; const ENCODING *m_encoding; INIT_ENCODING m_initEncoding; const XML_Char *m_protocolEncodingName; int m_ns; void *m_unknownEncodingMem; void *m_unknownEncodingData; void *m_unknownEncodingHandlerData; void (*m_unknownEncodingRelease)(void *); PROLOG_STATE m_prologState; Processor *m_processor; enum XML_Error m_errorCode; const char *m_eventPtr; const char *m_eventEndPtr; const char *m_positionPtr; OPEN_INTERNAL_ENTITY *m_openInternalEntities; int m_defaultExpandInternalEntities; int m_tagLevel; ENTITY *m_declEntity; const XML_Char *m_declNotationName; const XML_Char *m_declNotationPublicId; ELEMENT_TYPE *m_declElementType; ATTRIBUTE_ID *m_declAttributeId; char m_declAttributeIsCdata; DTD m_dtd; TAG *m_tagStack; TAG *m_freeTagList; BINDING *m_inheritedBindings; BINDING *m_freeBindingList; int m_attsSize; int m_nSpecifiedAtts; ATTRIBUTE *m_atts; POSITION m_position; STRING_POOL m_tempPool; STRING_POOL m_temp2Pool; char *m_groupConnector; unsigned m_groupSize; int m_hadExternalDoctype; XML_Char m_namespaceSeparator; } Parser; #define userData (((Parser *)parser)->m_userData) #define handlerArg (((Parser *)parser)->m_handlerArg) #define startElementHandler (((Parser *)parser)->m_startElementHandler) #define endElementHandler (((Parser *)parser)->m_endElementHandler) #define characterDataHandler (((Parser *)parser)->m_characterDataHandler) #define processingInstructionHandler (((Parser *)parser)->m_processingInstructionHandler) #define commentHandler (((Parser *)parser)->m_commentHandler) #define startCdataSectionHandler (((Parser *)parser)->m_startCdataSectionHandler) #define endCdataSectionHandler (((Parser *)parser)->m_endCdataSectionHandler) #define defaultHandler (((Parser *)parser)->m_defaultHandler) #define unparsedEntityDeclHandler (((Parser *)parser)->m_unparsedEntityDeclHandler) #define notationDeclHandler (((Parser *)parser)->m_notationDeclHandler) #define startNamespaceDeclHandler (((Parser *)parser)->m_startNamespaceDeclHandler) #define endNamespaceDeclHandler (((Parser *)parser)->m_endNamespaceDeclHandler) #define notStandaloneHandler (((Parser *)parser)->m_notStandaloneHandler) #define externalEntityRefHandler (((Parser *)parser)->m_externalEntityRefHandler) #define externalEntityRefHandlerArg (((Parser *)parser)->m_externalEntityRefHandlerArg) #define unknownEncodingHandler (((Parser *)parser)->m_unknownEncodingHandler) #define encoding (((Parser *)parser)->m_encoding) #define initEncoding (((Parser *)parser)->m_initEncoding) #define unknownEncodingMem (((Parser *)parser)->m_unknownEncodingMem) #define unknownEncodingData (((Parser *)parser)->m_unknownEncodingData) #define unknownEncodingHandlerData \ (((Parser *)parser)->m_unknownEncodingHandlerData) #define unknownEncodingRelease (((Parser *)parser)->m_unknownEncodingRelease) #define protocolEncodingName (((Parser *)parser)->m_protocolEncodingName) #define ns (((Parser *)parser)->m_ns) #define prologState (((Parser *)parser)->m_prologState) #define processor (((Parser *)parser)->m_processor) #define errorCode (((Parser *)parser)->m_errorCode) #define eventPtr (((Parser *)parser)->m_eventPtr) #define eventEndPtr (((Parser *)parser)->m_eventEndPtr) #define positionPtr (((Parser *)parser)->m_positionPtr) #define position (((Parser *)parser)->m_position) #define openInternalEntities (((Parser *)parser)->m_openInternalEntities) #define defaultExpandInternalEntities (((Parser *)parser)->m_defaultExpandInternalEntities) #define tagLevel (((Parser *)parser)->m_tagLevel) #define buffer (((Parser *)parser)->m_buffer) #define bufferPtr (((Parser *)parser)->m_bufferPtr) #define bufferEnd (((Parser *)parser)->m_bufferEnd) #define parseEndByteIndex (((Parser *)parser)->m_parseEndByteIndex) #define parseEndPtr (((Parser *)parser)->m_parseEndPtr) #define bufferLim (((Parser *)parser)->m_bufferLim) #define dataBuf (((Parser *)parser)->m_dataBuf) #define dataBufEnd (((Parser *)parser)->m_dataBufEnd) #define dtd (((Parser *)parser)->m_dtd) #define declEntity (((Parser *)parser)->m_declEntity) #define declNotationName (((Parser *)parser)->m_declNotationName) #define declNotationPublicId (((Parser *)parser)->m_declNotationPublicId) #define declElementType (((Parser *)parser)->m_declElementType) #define declAttributeId (((Parser *)parser)->m_declAttributeId) #define declAttributeIsCdata (((Parser *)parser)->m_declAttributeIsCdata) #define freeTagList (((Parser *)parser)->m_freeTagList) #define freeBindingList (((Parser *)parser)->m_freeBindingList) #define inheritedBindings (((Parser *)parser)->m_inheritedBindings) #define tagStack (((Parser *)parser)->m_tagStack) #define atts (((Parser *)parser)->m_atts) #define attsSize (((Parser *)parser)->m_attsSize) #define nSpecifiedAtts (((Parser *)parser)->m_nSpecifiedAtts) #define tempPool (((Parser *)parser)->m_tempPool) #define temp2Pool (((Parser *)parser)->m_temp2Pool) #define groupConnector (((Parser *)parser)->m_groupConnector) #define groupSize (((Parser *)parser)->m_groupSize) #define hadExternalDoctype (((Parser *)parser)->m_hadExternalDoctype) #define namespaceSeparator (((Parser *)parser)->m_namespaceSeparator) #ifdef _MSC_VER #ifdef _DEBUG Parser *asParser(XML_Parser parser) { return parser; } #endif #endif XML_Parser XML_ParserCreate(const XML_Char *encodingName) { XML_Parser parser = malloc(sizeof(Parser)); if (!parser) return parser; processor = prologInitProcessor; XmlPrologStateInit(&prologState); userData = 0; handlerArg = 0; startElementHandler = 0; endElementHandler = 0; characterDataHandler = 0; processingInstructionHandler = 0; commentHandler = 0; startCdataSectionHandler = 0; endCdataSectionHandler = 0; defaultHandler = 0; unparsedEntityDeclHandler = 0; notationDeclHandler = 0; startNamespaceDeclHandler = 0; endNamespaceDeclHandler = 0; notStandaloneHandler = 0; externalEntityRefHandler = 0; externalEntityRefHandlerArg = parser; unknownEncodingHandler = 0; buffer = 0; bufferPtr = 0; bufferEnd = 0; parseEndByteIndex = 0; parseEndPtr = 0; bufferLim = 0; declElementType = 0; declAttributeId = 0; declEntity = 0; declNotationName = 0; declNotationPublicId = 0; memset(&position, 0, sizeof(POSITION)); errorCode = XML_ERROR_NONE; eventPtr = 0; eventEndPtr = 0; positionPtr = 0; openInternalEntities = 0; tagLevel = 0; tagStack = 0; freeTagList = 0; freeBindingList = 0; inheritedBindings = 0; attsSize = INIT_ATTS_SIZE; atts = malloc(attsSize * sizeof(ATTRIBUTE)); nSpecifiedAtts = 0; dataBuf = malloc(INIT_DATA_BUF_SIZE * sizeof(XML_Char)); groupSize = 0; groupConnector = 0; hadExternalDoctype = 0; unknownEncodingMem = 0; unknownEncodingRelease = 0; unknownEncodingData = 0; unknownEncodingHandlerData = 0; namespaceSeparator = '!'; ns = 0; poolInit(&tempPool); poolInit(&temp2Pool); protocolEncodingName = encodingName ? poolCopyString(&tempPool, encodingName) : 0; if (!dtdInit(&dtd) || !atts || !dataBuf || (encodingName && !protocolEncodingName)) { XML_ParserFree(parser); return 0; } dataBufEnd = dataBuf + INIT_DATA_BUF_SIZE; XmlInitEncoding(&initEncoding, &encoding, 0); return parser; } XML_Parser XML_ParserCreateNS(const XML_Char *encodingName, XML_Char nsSep) { static const XML_Char implicitContext[] = { XML_T('x'), XML_T('m'), XML_T('l'), XML_T('='), XML_T('h'), XML_T('t'), XML_T('t'), XML_T('p'), XML_T(':'), XML_T('/'), XML_T('/'), XML_T('w'), XML_T('w'), XML_T('w'), XML_T('.'), XML_T('w'), XML_T('3'), XML_T('.'), XML_T('o'), XML_T('r'), XML_T('g'), XML_T('/'), XML_T('X'), XML_T('M'), XML_T('L'), XML_T('/'), XML_T('1'), XML_T('9'), XML_T('9'), XML_T('8'), XML_T('/'), XML_T('n'), XML_T('a'), XML_T('m'), XML_T('e'), XML_T('s'), XML_T('p'), XML_T('a'), XML_T('c'), XML_T('e'), XML_T('\0') }; XML_Parser parser = XML_ParserCreate(encodingName); if (parser) { XmlInitEncodingNS(&initEncoding, &encoding, 0); ns = 1; namespaceSeparator = nsSep; } if (!setContext(parser, implicitContext)) { XML_ParserFree(parser); return 0; } return parser; } int XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName) { if (!encodingName) protocolEncodingName = 0; else { protocolEncodingName = poolCopyString(&tempPool, encodingName); if (!protocolEncodingName) return 0; } return 1; } XML_Parser XML_ExternalEntityParserCreate(XML_Parser oldParser, const XML_Char *context, const XML_Char *encodingName) { XML_Parser parser = oldParser; DTD *oldDtd = &dtd; XML_StartElementHandler oldStartElementHandler = startElementHandler; XML_EndElementHandler oldEndElementHandler = endElementHandler; XML_CharacterDataHandler oldCharacterDataHandler = characterDataHandler; XML_ProcessingInstructionHandler oldProcessingInstructionHandler = processingInstructionHandler; XML_CommentHandler oldCommentHandler = commentHandler; XML_StartCdataSectionHandler oldStartCdataSectionHandler = startCdataSectionHandler; XML_EndCdataSectionHandler oldEndCdataSectionHandler = endCdataSectionHandler; XML_DefaultHandler oldDefaultHandler = defaultHandler; XML_StartNamespaceDeclHandler oldStartNamespaceDeclHandler = startNamespaceDeclHandler; XML_EndNamespaceDeclHandler oldEndNamespaceDeclHandler = endNamespaceDeclHandler; XML_NotStandaloneHandler oldNotStandaloneHandler = notStandaloneHandler; XML_ExternalEntityRefHandler oldExternalEntityRefHandler = externalEntityRefHandler; XML_UnknownEncodingHandler oldUnknownEncodingHandler = unknownEncodingHandler; void *oldUserData = userData; void *oldHandlerArg = handlerArg; int oldDefaultExpandInternalEntities = defaultExpandInternalEntities; void *oldExternalEntityRefHandlerArg = externalEntityRefHandlerArg; parser = (ns ? XML_ParserCreateNS(encodingName, namespaceSeparator) : XML_ParserCreate(encodingName)); if (!parser) return 0; startElementHandler = oldStartElementHandler; endElementHandler = oldEndElementHandler; characterDataHandler = oldCharacterDataHandler; processingInstructionHandler = oldProcessingInstructionHandler; commentHandler = oldCommentHandler; startCdataSectionHandler = oldStartCdataSectionHandler; endCdataSectionHandler = oldEndCdataSectionHandler; defaultHandler = oldDefaultHandler; startNamespaceDeclHandler = oldStartNamespaceDeclHandler; endNamespaceDeclHandler = oldEndNamespaceDeclHandler; notStandaloneHandler = oldNotStandaloneHandler; externalEntityRefHandler = oldExternalEntityRefHandler; unknownEncodingHandler = oldUnknownEncodingHandler; userData = oldUserData; if (oldUserData == oldHandlerArg) handlerArg = userData; else handlerArg = parser; if (oldExternalEntityRefHandlerArg != oldParser) externalEntityRefHandlerArg = oldExternalEntityRefHandlerArg; defaultExpandInternalEntities = oldDefaultExpandInternalEntities; if (!dtdCopy(&dtd, oldDtd) || !setContext(parser, context)) { XML_ParserFree(parser); return 0; } processor = externalEntityInitProcessor; return parser; } static void destroyBindings(BINDING *bindings) { for (;;) { BINDING *b = bindings; if (!b) break; bindings = b->nextTagBinding; free(b->uri); free(b); } } void XML_ParserFree(XML_Parser parser) { for (;;) { TAG *p; if (tagStack == 0) { if (freeTagList == 0) break; tagStack = freeTagList; freeTagList = 0; } p = tagStack; tagStack = tagStack->parent; free(p->buf); destroyBindings(p->bindings); free(p); } destroyBindings(freeBindingList); destroyBindings(inheritedBindings); poolDestroy(&tempPool); poolDestroy(&temp2Pool); dtdDestroy(&dtd); free((void *)atts); free(groupConnector); free(buffer); free(dataBuf); free(unknownEncodingMem); if (unknownEncodingRelease) unknownEncodingRelease(unknownEncodingData); free(parser); } void XML_UseParserAsHandlerArg(XML_Parser parser) { handlerArg = parser; } void XML_SetUserData(XML_Parser parser, void *p) { if (handlerArg == userData) handlerArg = userData = p; else userData = p; } int XML_SetBase(XML_Parser parser, const XML_Char *p) { if (p) { p = poolCopyString(&dtd.pool, p); if (!p) return 0; dtd.base = p; } else dtd.base = 0; return 1; } const XML_Char *XML_GetBase(XML_Parser parser) { return dtd.base; } int XML_GetSpecifiedAttributeCount(XML_Parser parser) { return nSpecifiedAtts; } void XML_SetElementHandler(XML_Parser parser, XML_StartElementHandler start, XML_EndElementHandler end) { startElementHandler = start; endElementHandler = end; } void XML_SetCharacterDataHandler(XML_Parser parser, XML_CharacterDataHandler handler) { characterDataHandler = handler; } void XML_SetProcessingInstructionHandler(XML_Parser parser, XML_ProcessingInstructionHandler handler) { processingInstructionHandler = handler; } void XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler) { commentHandler = handler; } void XML_SetCdataSectionHandler(XML_Parser parser, XML_StartCdataSectionHandler start, XML_EndCdataSectionHandler end) { startCdataSectionHandler = start; endCdataSectionHandler = end; } void XML_SetDefaultHandler(XML_Parser parser, XML_DefaultHandler handler) { defaultHandler = handler; defaultExpandInternalEntities = 0; } void XML_SetDefaultHandlerExpand(XML_Parser parser, XML_DefaultHandler handler) { defaultHandler = handler; defaultExpandInternalEntities = 1; } void XML_SetUnparsedEntityDeclHandler(XML_Parser parser, XML_UnparsedEntityDeclHandler handler) { unparsedEntityDeclHandler = handler; } void XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler) { notationDeclHandler = handler; } void XML_SetNamespaceDeclHandler(XML_Parser parser, XML_StartNamespaceDeclHandler start, XML_EndNamespaceDeclHandler end) { startNamespaceDeclHandler = start; endNamespaceDeclHandler = end; } void XML_SetNotStandaloneHandler(XML_Parser parser, XML_NotStandaloneHandler handler) { notStandaloneHandler = handler; } void XML_SetExternalEntityRefHandler(XML_Parser parser, XML_ExternalEntityRefHandler handler) { externalEntityRefHandler = handler; } void XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg) { if (arg) externalEntityRefHandlerArg = arg; else externalEntityRefHandlerArg = parser; } void XML_SetUnknownEncodingHandler(XML_Parser parser, XML_UnknownEncodingHandler handler, void *data) { unknownEncodingHandler = handler; unknownEncodingHandlerData = data; } int XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) { if (len == 0) { if (!isFinal) return 1; positionPtr = bufferPtr; errorCode = processor(parser, bufferPtr, parseEndPtr = bufferEnd, 0); if (errorCode == XML_ERROR_NONE) return 1; eventEndPtr = eventPtr; return 0; } else if (bufferPtr == bufferEnd) { const char *end; int nLeftOver; parseEndByteIndex += len; positionPtr = s; if (isFinal) { errorCode = processor(parser, s, parseEndPtr = s + len, 0); if (errorCode == XML_ERROR_NONE) return 1; eventEndPtr = eventPtr; return 0; } errorCode = processor(parser, s, parseEndPtr = s + len, &end); if (errorCode != XML_ERROR_NONE) { eventEndPtr = eventPtr; return 0; } XmlUpdatePosition(encoding, positionPtr, end, &position); nLeftOver = s + len - end; if (nLeftOver) { if (buffer == 0 || nLeftOver > bufferLim - buffer) { /* FIXME avoid integer overflow */ buffer = buffer == 0 ? malloc(len * 2) : realloc(buffer, len * 2); if (!buffer) { errorCode = XML_ERROR_NO_MEMORY; eventPtr = eventEndPtr = 0; return 0; } bufferLim = buffer + len * 2; } memcpy(buffer, end, nLeftOver); bufferPtr = buffer; bufferEnd = buffer + nLeftOver; } return 1; } else { memcpy(XML_GetBuffer(parser, len), s, len); return XML_ParseBuffer(parser, len, isFinal); } } int XML_ParseBuffer(XML_Parser parser, int len, int isFinal) { const char *start = bufferPtr; positionPtr = start; bufferEnd += len; parseEndByteIndex += len; errorCode = processor(parser, start, parseEndPtr = bufferEnd, isFinal ? (const char **)0 : &bufferPtr); if (errorCode == XML_ERROR_NONE) { if (!isFinal) XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position); return 1; } else { eventEndPtr = eventPtr; return 0; } } void *XML_GetBuffer(XML_Parser parser, int len) { if (len > bufferLim - bufferEnd) { /* FIXME avoid integer overflow */ int neededSize = len + (bufferEnd - bufferPtr); if (neededSize <= bufferLim - buffer) { memmove(buffer, bufferPtr, bufferEnd - bufferPtr); bufferEnd = buffer + (bufferEnd - bufferPtr); bufferPtr = buffer; } else { char *newBuf; int bufferSize = bufferLim - bufferPtr; if (bufferSize == 0) bufferSize = INIT_BUFFER_SIZE; do { bufferSize *= 2; } while (bufferSize < neededSize); newBuf = malloc(bufferSize); if (newBuf == 0) { errorCode = XML_ERROR_NO_MEMORY; return 0; } bufferLim = newBuf + bufferSize; if (bufferPtr) { memcpy(newBuf, bufferPtr, bufferEnd - bufferPtr); free(buffer); } bufferEnd = newBuf + (bufferEnd - bufferPtr); bufferPtr = buffer = newBuf; } } return bufferEnd; } enum XML_Error XML_GetErrorCode(XML_Parser parser) { return errorCode; } long XML_GetCurrentByteIndex(XML_Parser parser) { if (eventPtr) return parseEndByteIndex - (parseEndPtr - eventPtr); return -1; } int XML_GetCurrentByteCount(XML_Parser parser) { if (eventEndPtr && eventPtr) return eventEndPtr - eventPtr; return 0; } int XML_GetCurrentLineNumber(XML_Parser parser) { if (eventPtr) { XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); positionPtr = eventPtr; } return position.lineNumber + 1; } int XML_GetCurrentColumnNumber(XML_Parser parser) { if (eventPtr) { XmlUpdatePosition(encoding, positionPtr, eventPtr, &position); positionPtr = eventPtr; } return position.columnNumber; } void XML_DefaultCurrent(XML_Parser parser) { if (defaultHandler) { if (openInternalEntities) reportDefault(parser, ns ? XmlGetInternalEncodingNS() : XmlGetInternalEncoding(), openInternalEntities->internalEventPtr, openInternalEntities->internalEventEndPtr); else reportDefault(parser, encoding, eventPtr, eventEndPtr); } } const XML_LChar *XML_ErrorString(int code) { static const XML_LChar *message[] = { 0, XML_T("out of memory"), XML_T("syntax error"), XML_T("no element found"), XML_T("not well-formed"), XML_T("unclosed token"), XML_T("unclosed token"), XML_T("mismatched tag"), XML_T("duplicate attribute"), XML_T("junk after document element"), XML_T("illegal parameter entity reference"), XML_T("undefined entity"), XML_T("recursive entity reference"), XML_T("asynchronous entity"), XML_T("reference to invalid character number"), XML_T("reference to binary entity"), XML_T("reference to external entity in attribute"), XML_T("xml processing instruction not at start of external entity"), XML_T("unknown encoding"), XML_T("encoding specified in XML declaration is incorrect"), XML_T("unclosed CDATA section"), XML_T("error in processing external entity reference"), XML_T("document is not standalone") }; if (code > 0 && code < sizeof(message)/sizeof(message[0])) return message[code]; return 0; } static enum XML_Error contentProcessor(XML_Parser parser, const char *start, const char *end, const char **endPtr) { return doContent(parser, 0, encoding, start, end, endPtr); } static enum XML_Error externalEntityInitProcessor(XML_Parser parser, const char *start, const char *end, const char **endPtr) { enum XML_Error result = initializeEncoding(parser); if (result != XML_ERROR_NONE) return result; processor = externalEntityInitProcessor2; return externalEntityInitProcessor2(parser, start, end, endPtr); } static enum XML_Error externalEntityInitProcessor2(XML_Parser parser, const char *start, const char *end, const char **endPtr) { const char *next; int tok = XmlContentTok(encoding, start, end, &next); switch (tok) { case XML_TOK_BOM: start = next; break; case XML_TOK_PARTIAL: if (endPtr) { *endPtr = start; return XML_ERROR_NONE; } eventPtr = start; return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: if (endPtr) { *endPtr = start; return XML_ERROR_NONE; } eventPtr = start; return XML_ERROR_PARTIAL_CHAR; } processor = externalEntityInitProcessor3; return externalEntityInitProcessor3(parser, start, end, endPtr); } static enum XML_Error externalEntityInitProcessor3(XML_Parser parser, const char *start, const char *end, const char **endPtr) { const char *next; int tok = XmlContentTok(encoding, start, end, &next); switch (tok) { case XML_TOK_XML_DECL: { enum XML_Error result = processXmlDecl(parser, 1, start, next); if (result != XML_ERROR_NONE) return result; start = next; } break; case XML_TOK_PARTIAL: if (endPtr) { *endPtr = start; return XML_ERROR_NONE; } eventPtr = start; return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: if (endPtr) { *endPtr = start; return XML_ERROR_NONE; } eventPtr = start; return XML_ERROR_PARTIAL_CHAR; } processor = externalEntityContentProcessor; tagLevel = 1; return doContent(parser, 1, encoding, start, end, endPtr); } static enum XML_Error externalEntityContentProcessor(XML_Parser parser, const char *start, const char *end, const char **endPtr) { return doContent(parser, 1, encoding, start, end, endPtr); } static enum XML_Error doContent(XML_Parser parser, int startTagLevel, const ENCODING *enc, const char *s, const char *end, const char **nextPtr) { const ENCODING *internalEnc = ns ? XmlGetInternalEncodingNS() : XmlGetInternalEncoding(); const char **eventPP; const char **eventEndPP; if (enc == encoding) { eventPP = &eventPtr; eventEndPP = &eventEndPtr; } else { eventPP = &(openInternalEntities->internalEventPtr); eventEndPP = &(openInternalEntities->internalEventEndPtr); } *eventPP = s; for (;;) { const char *next = s; /* XmlContentTok doesn't always set the last arg */ int tok = XmlContentTok(enc, s, end, &next); *eventEndPP = next; switch (tok) { case XML_TOK_TRAILING_CR: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } *eventEndPP = end; if (characterDataHandler) { XML_Char c = 0xA; characterDataHandler(handlerArg, &c, 1); } else if (defaultHandler) reportDefault(parser, enc, s, end); if (startTagLevel == 0) return XML_ERROR_NO_ELEMENTS; if (tagLevel != startTagLevel) return XML_ERROR_ASYNC_ENTITY; return XML_ERROR_NONE; case XML_TOK_NONE: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } if (startTagLevel > 0) { if (tagLevel != startTagLevel) return XML_ERROR_ASYNC_ENTITY; return XML_ERROR_NONE; } return XML_ERROR_NO_ELEMENTS; case XML_TOK_INVALID: *eventPP = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_PARTIAL_CHAR; case XML_TOK_ENTITY_REF: { const XML_Char *name; ENTITY *entity; XML_Char ch = XmlPredefinedEntityName(enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (ch) { if (characterDataHandler) characterDataHandler(handlerArg, &ch, 1); else if (defaultHandler) reportDefault(parser, enc, s, next); break; } name = poolStoreString(&dtd.pool, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!name) return XML_ERROR_NO_MEMORY; entity = (ENTITY *)lookup(&dtd.generalEntities, name, 0); poolDiscard(&dtd.pool); if (!entity) { if (dtd.complete || dtd.standalone) return XML_ERROR_UNDEFINED_ENTITY; if (defaultHandler) reportDefault(parser, enc, s, next); break; } if (entity->open) return XML_ERROR_RECURSIVE_ENTITY_REF; if (entity->notation) return XML_ERROR_BINARY_ENTITY_REF; if (entity) { if (entity->textPtr) { enum XML_Error result; OPEN_INTERNAL_ENTITY openEntity; if (defaultHandler && !defaultExpandInternalEntities) { reportDefault(parser, enc, s, next); break; } entity->open = 1; openEntity.next = openInternalEntities; openInternalEntities = &openEntity; openEntity.entity = entity; openEntity.internalEventPtr = 0; openEntity.internalEventEndPtr = 0; result = doContent(parser, tagLevel, internalEnc, (char *)entity->textPtr, (char *)(entity->textPtr + entity->textLen), 0); entity->open = 0; openInternalEntities = openEntity.next; if (result) return result; } else if (externalEntityRefHandler) { const XML_Char *context; entity->open = 1; context = getContext(parser); entity->open = 0; if (!context) return XML_ERROR_NO_MEMORY; if (!externalEntityRefHandler(externalEntityRefHandlerArg, context, dtd.base, entity->systemId, entity->publicId)) return XML_ERROR_EXTERNAL_ENTITY_HANDLING; poolDiscard(&tempPool); } else if (defaultHandler) reportDefault(parser, enc, s, next); } break; } case XML_TOK_START_TAG_WITH_ATTS: if (!startElementHandler) { enum XML_Error result = storeAtts(parser, enc, s, 0, 0); if (result) return result; } /* fall through */ case XML_TOK_START_TAG_NO_ATTS: { TAG *tag; if (freeTagList) { tag = freeTagList; freeTagList = freeTagList->parent; } else { tag = malloc(sizeof(TAG)); if (!tag) return XML_ERROR_NO_MEMORY; tag->buf = malloc(INIT_TAG_BUF_SIZE); if (!tag->buf) return XML_ERROR_NO_MEMORY; tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE; } tag->bindings = 0; tag->parent = tagStack; tagStack = tag; tag->name.localPart = 0; tag->rawName = s + enc->minBytesPerChar; tag->rawNameLength = XmlNameLength(enc, tag->rawName); if (nextPtr) { /* Need to guarantee that: tag->buf + ROUND_UP(tag->rawNameLength, sizeof(XML_Char)) <= tag->bufEnd - sizeof(XML_Char) */ if (tag->rawNameLength + (int)(sizeof(XML_Char) - 1) + (int)sizeof(XML_Char) > tag->bufEnd - tag->buf) { int bufSize = tag->rawNameLength * 4; bufSize = ROUND_UP(bufSize, sizeof(XML_Char)); tag->buf = realloc(tag->buf, bufSize); if (!tag->buf) return XML_ERROR_NO_MEMORY; tag->bufEnd = tag->buf + bufSize; } memcpy(tag->buf, tag->rawName, tag->rawNameLength); tag->rawName = tag->buf; } ++tagLevel; if (startElementHandler) { enum XML_Error result; XML_Char *toPtr; for (;;) { const char *rawNameEnd = tag->rawName + tag->rawNameLength; const char *fromPtr = tag->rawName; int bufSize; if (nextPtr) toPtr = (XML_Char *)(tag->buf + ROUND_UP(tag->rawNameLength, sizeof(XML_Char))); else toPtr = (XML_Char *)tag->buf; tag->name.str = toPtr; XmlConvert(enc, &fromPtr, rawNameEnd, (ICHAR **)&toPtr, (ICHAR *)tag->bufEnd - 1); if (fromPtr == rawNameEnd) break; bufSize = (tag->bufEnd - tag->buf) << 1; tag->buf = realloc(tag->buf, bufSize); if (!tag->buf) return XML_ERROR_NO_MEMORY; tag->bufEnd = tag->buf + bufSize; if (nextPtr) tag->rawName = tag->buf; } *toPtr = XML_T('\0'); result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings)); if (result) return result; startElementHandler(handlerArg, tag->name.str, (const XML_Char **)atts); poolClear(&tempPool); } else { tag->name.str = 0; if (defaultHandler) reportDefault(parser, enc, s, next); } break; } case XML_TOK_EMPTY_ELEMENT_WITH_ATTS: if (!startElementHandler) { enum XML_Error result = storeAtts(parser, enc, s, 0, 0); if (result) return result; } /* fall through */ case XML_TOK_EMPTY_ELEMENT_NO_ATTS: if (startElementHandler || endElementHandler) { const char *rawName = s + enc->minBytesPerChar; enum XML_Error result; BINDING *bindings = 0; TAG_NAME name; name.str = poolStoreString(&tempPool, enc, rawName, rawName + XmlNameLength(enc, rawName)); if (!name.str) return XML_ERROR_NO_MEMORY; poolFinish(&tempPool); result = storeAtts(parser, enc, s, &name, &bindings); if (result) return result; poolFinish(&tempPool); if (startElementHandler) startElementHandler(handlerArg, name.str, (const XML_Char **)atts); if (endElementHandler) { if (startElementHandler) *eventPP = *eventEndPP; endElementHandler(handlerArg, name.str); } poolClear(&tempPool); while (bindings) { BINDING *b = bindings; if (endNamespaceDeclHandler) endNamespaceDeclHandler(handlerArg, b->prefix->name); bindings = bindings->nextTagBinding; b->nextTagBinding = freeBindingList; freeBindingList = b; b->prefix->binding = b->prevPrefixBinding; } } else if (defaultHandler) reportDefault(parser, enc, s, next); if (tagLevel == 0) return epilogProcessor(parser, next, end, nextPtr); break; case XML_TOK_END_TAG: if (tagLevel == startTagLevel) return XML_ERROR_ASYNC_ENTITY; else { int len; const char *rawName; TAG *tag = tagStack; tagStack = tag->parent; tag->parent = freeTagList; freeTagList = tag; rawName = s + enc->minBytesPerChar*2; len = XmlNameLength(enc, rawName); if (len != tag->rawNameLength || memcmp(tag->rawName, rawName, len) != 0) { *eventPP = rawName; return XML_ERROR_TAG_MISMATCH; } --tagLevel; if (endElementHandler && tag->name.str) { if (tag->name.localPart) { XML_Char *to = (XML_Char *)tag->name.str + tag->name.uriLen; const XML_Char *from = tag->name.localPart; while ((*to++ = *from++) != 0) ; } endElementHandler(handlerArg, tag->name.str); } else if (defaultHandler) reportDefault(parser, enc, s, next); while (tag->bindings) { BINDING *b = tag->bindings; if (endNamespaceDeclHandler) endNamespaceDeclHandler(handlerArg, b->prefix->name); tag->bindings = tag->bindings->nextTagBinding; b->nextTagBinding = freeBindingList; freeBindingList = b; b->prefix->binding = b->prevPrefixBinding; } if (tagLevel == 0) return epilogProcessor(parser, next, end, nextPtr); } break; case XML_TOK_CHAR_REF: { int n = XmlCharRefNumber(enc, s); if (n < 0) return XML_ERROR_BAD_CHAR_REF; if (characterDataHandler) { XML_Char buf[XML_ENCODE_MAX]; characterDataHandler(handlerArg, buf, XmlEncode(n, (ICHAR *)buf)); } else if (defaultHandler) reportDefault(parser, enc, s, next); } break; case XML_TOK_XML_DECL: return XML_ERROR_MISPLACED_XML_PI; case XML_TOK_DATA_NEWLINE: if (characterDataHandler) { XML_Char c = 0xA; characterDataHandler(handlerArg, &c, 1); } else if (defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_CDATA_SECT_OPEN: { enum XML_Error result; if (startCdataSectionHandler) startCdataSectionHandler(handlerArg); #if 0 /* Suppose you doing a transformation on a document that involves changing only the character data. You set up a defaultHandler and a characterDataHandler. The defaultHandler simply copies characters through. The characterDataHandler does the transformation and writes the characters out escaping them as necessary. This case will fail to work if we leave out the following two lines (because & and < inside CDATA sections will be incorrectly escaped). However, now we have a start/endCdataSectionHandler, so it seems easier to let the user deal with this. */ else if (characterDataHandler) characterDataHandler(handlerArg, dataBuf, 0); #endif else if (defaultHandler) reportDefault(parser, enc, s, next); result = doCdataSection(parser, enc, &next, end, nextPtr); if (!next) { processor = cdataSectionProcessor; return result; } } break; case XML_TOK_TRAILING_RSQB: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } if (characterDataHandler) { if (MUST_CONVERT(enc, s)) { ICHAR *dataPtr = (ICHAR *)dataBuf; XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd); characterDataHandler(handlerArg, dataBuf, dataPtr - (ICHAR *)dataBuf); } else characterDataHandler(handlerArg, (XML_Char *)s, (XML_Char *)end - (XML_Char *)s); } else if (defaultHandler) reportDefault(parser, enc, s, end); if (startTagLevel == 0) { *eventPP = end; return XML_ERROR_NO_ELEMENTS; } if (tagLevel != startTagLevel) { *eventPP = end; return XML_ERROR_ASYNC_ENTITY; } return XML_ERROR_NONE; case XML_TOK_DATA_CHARS: if (characterDataHandler) { if (MUST_CONVERT(enc, s)) { for (;;) { ICHAR *dataPtr = (ICHAR *)dataBuf; XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd); *eventEndPP = s; characterDataHandler(handlerArg, dataBuf, dataPtr - (ICHAR *)dataBuf); if (s == next) break; *eventPP = s; } } else characterDataHandler(handlerArg, (XML_Char *)s, (XML_Char *)next - (XML_Char *)s); } else if (defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_PI: if (!reportProcessingInstruction(parser, enc, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_COMMENT: if (!reportComment(parser, enc, s, next)) return XML_ERROR_NO_MEMORY; break; default: if (defaultHandler) reportDefault(parser, enc, s, next); break; } *eventPP = s = next; } /* not reached */ } /* If tagNamePtr is non-null, build a real list of attributes, otherwise just check the attributes for well-formedness. */ static enum XML_Error storeAtts(XML_Parser parser, const ENCODING *enc, const char *s, TAG_NAME *tagNamePtr, BINDING **bindingsPtr) { ELEMENT_TYPE *elementType = 0; int nDefaultAtts = 0; const XML_Char **appAtts; int attIndex = 0; int i; int n; int nPrefixes = 0; BINDING *binding; const XML_Char *localPart; if (tagNamePtr) { elementType = (ELEMENT_TYPE *)lookup(&dtd.elementTypes, tagNamePtr->str, 0); if (!elementType) { tagNamePtr->str = poolCopyString(&dtd.pool, tagNamePtr->str); if (!tagNamePtr->str) return XML_ERROR_NO_MEMORY; elementType = (ELEMENT_TYPE *)lookup(&dtd.elementTypes, tagNamePtr->str, sizeof(ELEMENT_TYPE)); if (!elementType) return XML_ERROR_NO_MEMORY; if (ns && !setElementTypePrefix(parser, elementType)) return XML_ERROR_NO_MEMORY; } nDefaultAtts = elementType->nDefaultAtts; } n = XmlGetAttributes(enc, s, attsSize, atts); if (n + nDefaultAtts > attsSize) { int oldAttsSize = attsSize; attsSize = n + nDefaultAtts + INIT_ATTS_SIZE; atts = realloc((void *)atts, attsSize * sizeof(ATTRIBUTE)); if (!atts) return XML_ERROR_NO_MEMORY; if (n > oldAttsSize) XmlGetAttributes(enc, s, n, atts); } appAtts = (const XML_Char **)atts; for (i = 0; i < n; i++) { ATTRIBUTE_ID *attId = getAttributeId(parser, enc, atts[i].name, atts[i].name + XmlNameLength(enc, atts[i].name)); if (!attId) return XML_ERROR_NO_MEMORY; if ((attId->name)[-1]) { if (enc == encoding) eventPtr = atts[i].name; return XML_ERROR_DUPLICATE_ATTRIBUTE; } (attId->name)[-1] = 1; appAtts[attIndex++] = attId->name; if (!atts[i].normalized) { enum XML_Error result; int isCdata = 1; if (attId->maybeTokenized) { int j; for (j = 0; j < nDefaultAtts; j++) { if (attId == elementType->defaultAtts[j].id) { isCdata = elementType->defaultAtts[j].isCdata; break; } } } result = storeAttributeValue(parser, enc, isCdata, atts[i].valuePtr, atts[i].valueEnd, &tempPool); if (result) return result; if (tagNamePtr) { appAtts[attIndex] = poolStart(&tempPool); poolFinish(&tempPool); } else poolDiscard(&tempPool); } else if (tagNamePtr) { appAtts[attIndex] = poolStoreString(&tempPool, enc, atts[i].valuePtr, atts[i].valueEnd); if (appAtts[attIndex] == 0) return XML_ERROR_NO_MEMORY; poolFinish(&tempPool); } if (attId->prefix && tagNamePtr) { if (attId->xmlns) { if (!addBinding(parser, attId->prefix, attId, appAtts[attIndex], bindingsPtr)) return XML_ERROR_NO_MEMORY; --attIndex; } else { attIndex++; nPrefixes++; (attId->name)[-1] = 2; } } else attIndex++; } nSpecifiedAtts = attIndex; if (tagNamePtr) { int j; for (j = 0; j < nDefaultAtts; j++) { const DEFAULT_ATTRIBUTE *da = elementType->defaultAtts + j; if (!(da->id->name)[-1] && da->value) { if (da->id->prefix) { if (da->id->xmlns) { if (!addBinding(parser, da->id->prefix, da->id, da->value, bindingsPtr)) return XML_ERROR_NO_MEMORY; } else { (da->id->name)[-1] = 2; nPrefixes++; appAtts[attIndex++] = da->id->name; appAtts[attIndex++] = da->value; } } else { (da->id->name)[-1] = 1; appAtts[attIndex++] = da->id->name; appAtts[attIndex++] = da->value; } } } appAtts[attIndex] = 0; } i = 0; if (nPrefixes) { for (; i < attIndex; i += 2) { if (appAtts[i][-1] == 2) { ATTRIBUTE_ID *id; ((XML_Char *)(appAtts[i]))[-1] = 0; id = (ATTRIBUTE_ID *)lookup(&dtd.attributeIds, appAtts[i], 0); if (id->prefix->binding) { int j; const BINDING *b = id->prefix->binding; const XML_Char *s = appAtts[i]; for (j = 0; j < b->uriLen; j++) { if (!poolAppendChar(&tempPool, b->uri[j])) return XML_ERROR_NO_MEMORY; } while (*s++ != ':') ; do { if (!poolAppendChar(&tempPool, *s)) return XML_ERROR_NO_MEMORY; } while (*s++); appAtts[i] = poolStart(&tempPool); poolFinish(&tempPool); } if (!--nPrefixes) break; } else ((XML_Char *)(appAtts[i]))[-1] = 0; } } for (; i < attIndex; i += 2) ((XML_Char *)(appAtts[i]))[-1] = 0; if (!tagNamePtr) return XML_ERROR_NONE; for (binding = *bindingsPtr; binding; binding = binding->nextTagBinding) binding->attId->name[-1] = 0; if (elementType->prefix) { binding = elementType->prefix->binding; if (!binding) return XML_ERROR_NONE; localPart = tagNamePtr->str; while (*localPart++ != XML_T(':')) ; } else if (dtd.defaultPrefix.binding) { binding = dtd.defaultPrefix.binding; localPart = tagNamePtr->str; } else return XML_ERROR_NONE; tagNamePtr->localPart = localPart; tagNamePtr->uriLen = binding->uriLen; i = binding->uriLen; do { if (i == binding->uriAlloc) { binding->uri = realloc(binding->uri, binding->uriAlloc *= 2); if (!binding->uri) return XML_ERROR_NO_MEMORY; } binding->uri[i++] = *localPart; } while (*localPart++); tagNamePtr->str = binding->uri; return XML_ERROR_NONE; } static int addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId, const XML_Char *uri, BINDING **bindingsPtr) { BINDING *b; int len; for (len = 0; uri[len]; len++) ; if (namespaceSeparator) len++; if (freeBindingList) { b = freeBindingList; if (len > b->uriAlloc) { b->uri = realloc(b->uri, len + EXPAND_SPARE); if (!b->uri) return 0; b->uriAlloc = len + EXPAND_SPARE; } freeBindingList = b->nextTagBinding; } else { b = malloc(sizeof(BINDING)); if (!b) return 0; b->uri = malloc(sizeof(XML_Char) * len + EXPAND_SPARE); if (!b->uri) { free(b); return 0; } b->uriAlloc = len; } b->uriLen = len; memcpy(b->uri, uri, len * sizeof(XML_Char)); if (namespaceSeparator) b->uri[len - 1] = namespaceSeparator; b->prefix = prefix; b->attId = attId; b->prevPrefixBinding = prefix->binding; if (*uri == XML_T('\0') && prefix == &dtd.defaultPrefix) prefix->binding = 0; else prefix->binding = b; b->nextTagBinding = *bindingsPtr; *bindingsPtr = b; if (startNamespaceDeclHandler) startNamespaceDeclHandler(handlerArg, prefix->name, prefix->binding ? uri : 0); return 1; } /* The idea here is to avoid using stack for each CDATA section when the whole file is parsed with one call. */ static enum XML_Error cdataSectionProcessor(XML_Parser parser, const char *start, const char *end, const char **endPtr) { enum XML_Error result = doCdataSection(parser, encoding, &start, end, endPtr); if (start) { processor = contentProcessor; return contentProcessor(parser, start, end, endPtr); } return result; } /* startPtr gets set to non-null is the section is closed, and to null if the section is not yet closed. */ static enum XML_Error doCdataSection(XML_Parser parser, const ENCODING *enc, const char **startPtr, const char *end, const char **nextPtr) { const char *s = *startPtr; const char **eventPP; const char **eventEndPP; if (enc == encoding) { eventPP = &eventPtr; *eventPP = s; eventEndPP = &eventEndPtr; } else { eventPP = &(openInternalEntities->internalEventPtr); eventEndPP = &(openInternalEntities->internalEventEndPtr); } *eventPP = s; *startPtr = 0; for (;;) { const char *next; int tok = XmlCdataSectionTok(enc, s, end, &next); *eventEndPP = next; switch (tok) { case XML_TOK_CDATA_SECT_CLOSE: if (endCdataSectionHandler) endCdataSectionHandler(handlerArg); #if 0 /* see comment under XML_TOK_CDATA_SECT_OPEN */ else if (characterDataHandler) characterDataHandler(handlerArg, dataBuf, 0); #endif else if (defaultHandler) reportDefault(parser, enc, s, next); *startPtr = next; return XML_ERROR_NONE; case XML_TOK_DATA_NEWLINE: if (characterDataHandler) { XML_Char c = 0xA; characterDataHandler(handlerArg, &c, 1); } else if (defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_DATA_CHARS: if (characterDataHandler) { if (MUST_CONVERT(enc, s)) { for (;;) { ICHAR *dataPtr = (ICHAR *)dataBuf; XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd); *eventEndPP = next; characterDataHandler(handlerArg, dataBuf, dataPtr - (ICHAR *)dataBuf); if (s == next) break; *eventPP = s; } } else characterDataHandler(handlerArg, (XML_Char *)s, (XML_Char *)next - (XML_Char *)s); } else if (defaultHandler) reportDefault(parser, enc, s, next); break; case XML_TOK_INVALID: *eventPP = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL_CHAR: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_PARTIAL_CHAR; case XML_TOK_PARTIAL: case XML_TOK_NONE: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_UNCLOSED_CDATA_SECTION; default: abort(); } *eventPP = s = next; } /* not reached */ } static enum XML_Error initializeEncoding(XML_Parser parser) { const char *s; #ifdef XML_UNICODE char encodingBuf[128]; if (!protocolEncodingName) s = 0; else { int i; for (i = 0; protocolEncodingName[i]; i++) { if (i == sizeof(encodingBuf) - 1 || protocolEncodingName[i] >= 0x80 || protocolEncodingName[i] < 0) { encodingBuf[0] = '\0'; break; } encodingBuf[i] = (char)protocolEncodingName[i]; } encodingBuf[i] = '\0'; s = encodingBuf; } #else s = protocolEncodingName; #endif if ((ns ? XmlInitEncodingNS : XmlInitEncoding)(&initEncoding, &encoding, s)) return XML_ERROR_NONE; return handleUnknownEncoding(parser, protocolEncodingName); } static enum XML_Error processXmlDecl(XML_Parser parser, int isGeneralTextEntity, const char *s, const char *next) { const char *encodingName = 0; const ENCODING *newEncoding = 0; const char *version; int standalone = -1; if (!(ns ? XmlParseXmlDeclNS : XmlParseXmlDecl)(isGeneralTextEntity, encoding, s, next, &eventPtr, &version, &encodingName, &newEncoding, &standalone)) return XML_ERROR_SYNTAX; if (!isGeneralTextEntity && standalone == 1) dtd.standalone = 1; if (defaultHandler) reportDefault(parser, encoding, s, next); if (!protocolEncodingName) { if (newEncoding) { if (newEncoding->minBytesPerChar != encoding->minBytesPerChar) { eventPtr = encodingName; return XML_ERROR_INCORRECT_ENCODING; } encoding = newEncoding; } else if (encodingName) { enum XML_Error result; const XML_Char *s = poolStoreString(&tempPool, encoding, encodingName, encodingName + XmlNameLength(encoding, encodingName)); if (!s) return XML_ERROR_NO_MEMORY; result = handleUnknownEncoding(parser, s); poolDiscard(&tempPool); if (result == XML_ERROR_UNKNOWN_ENCODING) eventPtr = encodingName; return result; } } return XML_ERROR_NONE; } static enum XML_Error handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName) { if (unknownEncodingHandler) { XML_Encoding info; int i; for (i = 0; i < 256; i++) info.map[i] = -1; info.convert = 0; info.data = 0; info.release = 0; if (unknownEncodingHandler(unknownEncodingHandlerData, encodingName, &info)) { ENCODING *enc; unknownEncodingMem = malloc(XmlSizeOfUnknownEncoding()); if (!unknownEncodingMem) { if (info.release) info.release(info.data); return XML_ERROR_NO_MEMORY; } enc = (ns ? XmlInitUnknownEncodingNS : XmlInitUnknownEncoding)(unknownEncodingMem, info.map, info.convert, info.data); if (enc) { unknownEncodingData = info.data; unknownEncodingRelease = info.release; encoding = enc; return XML_ERROR_NONE; } } if (info.release) info.release(info.data); } return XML_ERROR_UNKNOWN_ENCODING; } static enum XML_Error prologInitProcessor(XML_Parser parser, const char *s, const char *end, const char **nextPtr) { enum XML_Error result = initializeEncoding(parser); if (result != XML_ERROR_NONE) return result; processor = prologProcessor; return prologProcessor(parser, s, end, nextPtr); } static enum XML_Error prologProcessor(XML_Parser parser, const char *s, const char *end, const char **nextPtr) { for (;;) { const char *next; int tok = XmlPrologTok(encoding, s, end, &next); if (tok <= 0) { if (nextPtr != 0 && tok != XML_TOK_INVALID) { *nextPtr = s; return XML_ERROR_NONE; } switch (tok) { case XML_TOK_INVALID: eventPtr = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_NONE: return XML_ERROR_NO_ELEMENTS; case XML_TOK_PARTIAL: return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: return XML_ERROR_PARTIAL_CHAR; case XML_TOK_TRAILING_CR: eventPtr = s + encoding->minBytesPerChar; return XML_ERROR_NO_ELEMENTS; default: abort(); } } switch (XmlTokenRole(&prologState, tok, s, next, encoding)) { case XML_ROLE_XML_DECL: { enum XML_Error result = processXmlDecl(parser, 0, s, next); if (result != XML_ERROR_NONE) return result; } break; case XML_ROLE_DOCTYPE_SYSTEM_ID: if (!dtd.standalone && notStandaloneHandler && !notStandaloneHandler(handlerArg)) return XML_ERROR_NOT_STANDALONE; hadExternalDoctype = 1; break; case XML_ROLE_DOCTYPE_PUBLIC_ID: case XML_ROLE_ENTITY_PUBLIC_ID: if (!XmlIsPublicId(encoding, s, next, &eventPtr)) return XML_ERROR_SYNTAX; if (declEntity) { XML_Char *tem = poolStoreString(&dtd.pool, encoding, s + encoding->minBytesPerChar, next - encoding->minBytesPerChar); if (!tem) return XML_ERROR_NO_MEMORY; normalizePublicId(tem); declEntity->publicId = tem; poolFinish(&dtd.pool); } break; case XML_ROLE_INSTANCE_START: processor = contentProcessor; if (hadExternalDoctype) dtd.complete = 0; return contentProcessor(parser, s, end, nextPtr); case XML_ROLE_ATTLIST_ELEMENT_NAME: { const XML_Char *name = poolStoreString(&dtd.pool, encoding, s, next); if (!name) return XML_ERROR_NO_MEMORY; declElementType = (ELEMENT_TYPE *)lookup(&dtd.elementTypes, name, sizeof(ELEMENT_TYPE)); if (!declElementType) return XML_ERROR_NO_MEMORY; if (declElementType->name != name) poolDiscard(&dtd.pool); else { poolFinish(&dtd.pool); if (!setElementTypePrefix(parser, declElementType)) return XML_ERROR_NO_MEMORY; } break; } case XML_ROLE_ATTRIBUTE_NAME: declAttributeId = getAttributeId(parser, encoding, s, next); if (!declAttributeId) return XML_ERROR_NO_MEMORY; declAttributeIsCdata = 0; break; case XML_ROLE_ATTRIBUTE_TYPE_CDATA: declAttributeIsCdata = 1; break; case XML_ROLE_IMPLIED_ATTRIBUTE_VALUE: case XML_ROLE_REQUIRED_ATTRIBUTE_VALUE: if (dtd.complete && !defineAttribute(declElementType, declAttributeId, declAttributeIsCdata, 0)) return XML_ERROR_NO_MEMORY; break; case XML_ROLE_DEFAULT_ATTRIBUTE_VALUE: case XML_ROLE_FIXED_ATTRIBUTE_VALUE: { const XML_Char *attVal; enum XML_Error result = storeAttributeValue(parser, encoding, declAttributeIsCdata, s + encoding->minBytesPerChar, next - encoding->minBytesPerChar, &dtd.pool); if (result) return result; attVal = poolStart(&dtd.pool); poolFinish(&dtd.pool); if (dtd.complete && !defineAttribute(declElementType, declAttributeId, declAttributeIsCdata, attVal)) return XML_ERROR_NO_MEMORY; break; } case XML_ROLE_ENTITY_VALUE: { enum XML_Error result = storeEntityValue(parser, s, next); if (result != XML_ERROR_NONE) return result; } break; case XML_ROLE_ENTITY_SYSTEM_ID: if (declEntity) { declEntity->systemId = poolStoreString(&dtd.pool, encoding, s + encoding->minBytesPerChar, next - encoding->minBytesPerChar); if (!declEntity->systemId) return XML_ERROR_NO_MEMORY; declEntity->base = dtd.base; poolFinish(&dtd.pool); } break; case XML_ROLE_ENTITY_NOTATION_NAME: if (declEntity) { declEntity->notation = poolStoreString(&dtd.pool, encoding, s, next); if (!declEntity->notation) return XML_ERROR_NO_MEMORY; poolFinish(&dtd.pool); if (unparsedEntityDeclHandler) { eventPtr = eventEndPtr = s; unparsedEntityDeclHandler(handlerArg, declEntity->name, declEntity->base, declEntity->systemId, declEntity->publicId, declEntity->notation); } } break; case XML_ROLE_GENERAL_ENTITY_NAME: { const XML_Char *name; if (XmlPredefinedEntityName(encoding, s, next)) { declEntity = 0; break; } name = poolStoreString(&dtd.pool, encoding, s, next); if (!name) return XML_ERROR_NO_MEMORY; if (dtd.complete) { declEntity = (ENTITY *)lookup(&dtd.generalEntities, name, sizeof(ENTITY)); if (!declEntity) return XML_ERROR_NO_MEMORY; if (declEntity->name != name) { poolDiscard(&dtd.pool); declEntity = 0; } else poolFinish(&dtd.pool); } else { poolDiscard(&dtd.pool); declEntity = 0; } } break; case XML_ROLE_PARAM_ENTITY_NAME: declEntity = 0; break; case XML_ROLE_NOTATION_NAME: declNotationPublicId = 0; declNotationName = 0; if (notationDeclHandler) { declNotationName = poolStoreString(&tempPool, encoding, s, next); if (!declNotationName) return XML_ERROR_NO_MEMORY; poolFinish(&tempPool); } break; case XML_ROLE_NOTATION_PUBLIC_ID: if (!XmlIsPublicId(encoding, s, next, &eventPtr)) return XML_ERROR_SYNTAX; if (declNotationName) { XML_Char *tem = poolStoreString(&tempPool, encoding, s + encoding->minBytesPerChar, next - encoding->minBytesPerChar); if (!tem) return XML_ERROR_NO_MEMORY; normalizePublicId(tem); declNotationPublicId = tem; poolFinish(&tempPool); } break; case XML_ROLE_NOTATION_SYSTEM_ID: if (declNotationName && notationDeclHandler) { const XML_Char *systemId = poolStoreString(&tempPool, encoding, s + encoding->minBytesPerChar, next - encoding->minBytesPerChar); if (!systemId) return XML_ERROR_NO_MEMORY; eventPtr = eventEndPtr = s; notationDeclHandler(handlerArg, declNotationName, dtd.base, systemId, declNotationPublicId); } poolClear(&tempPool); break; case XML_ROLE_NOTATION_NO_SYSTEM_ID: if (declNotationPublicId && notationDeclHandler) { eventPtr = eventEndPtr = s; notationDeclHandler(handlerArg, declNotationName, dtd.base, 0, declNotationPublicId); } poolClear(&tempPool); break; case XML_ROLE_ERROR: eventPtr = s; switch (tok) { case XML_TOK_PARAM_ENTITY_REF: return XML_ERROR_PARAM_ENTITY_REF; case XML_TOK_XML_DECL: return XML_ERROR_MISPLACED_XML_PI; default: return XML_ERROR_SYNTAX; } case XML_ROLE_GROUP_OPEN: if (prologState.level >= groupSize) { if (groupSize) groupConnector = realloc(groupConnector, groupSize *= 2); else groupConnector = malloc(groupSize = 32); if (!groupConnector) return XML_ERROR_NO_MEMORY; } groupConnector[prologState.level] = 0; break; case XML_ROLE_GROUP_SEQUENCE: if (groupConnector[prologState.level] == '|') { eventPtr = s; return XML_ERROR_SYNTAX; } groupConnector[prologState.level] = ','; break; case XML_ROLE_GROUP_CHOICE: if (groupConnector[prologState.level] == ',') { eventPtr = s; return XML_ERROR_SYNTAX; } groupConnector[prologState.level] = '|'; break; case XML_ROLE_PARAM_ENTITY_REF: if (!dtd.standalone && notStandaloneHandler && !notStandaloneHandler(handlerArg)) return XML_ERROR_NOT_STANDALONE; dtd.complete = 0; break; case XML_ROLE_NONE: switch (tok) { case XML_TOK_PI: eventPtr = s; eventEndPtr = next; if (!reportProcessingInstruction(parser, encoding, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_COMMENT: eventPtr = s; eventEndPtr = next; if (!reportComment(parser, encoding, s, next)) return XML_ERROR_NO_MEMORY; break; } break; } if (defaultHandler) { switch (tok) { case XML_TOK_PI: case XML_TOK_COMMENT: case XML_TOK_BOM: case XML_TOK_XML_DECL: break; default: eventPtr = s; eventEndPtr = next; reportDefault(parser, encoding, s, next); } } s = next; } /* not reached */ } static enum XML_Error epilogProcessor(XML_Parser parser, const char *s, const char *end, const char **nextPtr) { processor = epilogProcessor; eventPtr = s; for (;;) { const char *next; int tok = XmlPrologTok(encoding, s, end, &next); eventEndPtr = next; switch (tok) { case XML_TOK_TRAILING_CR: if (defaultHandler) { eventEndPtr = end; reportDefault(parser, encoding, s, end); } /* fall through */ case XML_TOK_NONE: if (nextPtr) *nextPtr = end; return XML_ERROR_NONE; case XML_TOK_PROLOG_S: if (defaultHandler) reportDefault(parser, encoding, s, next); break; case XML_TOK_PI: if (!reportProcessingInstruction(parser, encoding, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_COMMENT: if (!reportComment(parser, encoding, s, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_INVALID: eventPtr = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_UNCLOSED_TOKEN; case XML_TOK_PARTIAL_CHAR: if (nextPtr) { *nextPtr = s; return XML_ERROR_NONE; } return XML_ERROR_PARTIAL_CHAR; default: return XML_ERROR_JUNK_AFTER_DOC_ELEMENT; } eventPtr = s = next; } } static enum XML_Error errorProcessor(XML_Parser parser, const char *s, const char *end, const char **nextPtr) { return errorCode; } static enum XML_Error storeAttributeValue(XML_Parser parser, const ENCODING *enc, int isCdata, const char *ptr, const char *end, STRING_POOL *pool) { enum XML_Error result = appendAttributeValue(parser, enc, isCdata, ptr, end, pool); if (result) return result; if (!isCdata && poolLength(pool) && poolLastChar(pool) == 0x20) poolChop(pool); if (!poolAppendChar(pool, XML_T('\0'))) return XML_ERROR_NO_MEMORY; return XML_ERROR_NONE; } static enum XML_Error appendAttributeValue(XML_Parser parser, const ENCODING *enc, int isCdata, const char *ptr, const char *end, STRING_POOL *pool) { const ENCODING *internalEnc = ns ? XmlGetInternalEncodingNS() : XmlGetInternalEncoding(); for (;;) { const char *next; int tok = XmlAttributeValueTok(enc, ptr, end, &next); switch (tok) { case XML_TOK_NONE: return XML_ERROR_NONE; case XML_TOK_INVALID: if (enc == encoding) eventPtr = next; return XML_ERROR_INVALID_TOKEN; case XML_TOK_PARTIAL: if (enc == encoding) eventPtr = ptr; return XML_ERROR_INVALID_TOKEN; case XML_TOK_CHAR_REF: { XML_Char buf[XML_ENCODE_MAX]; int i; int n = XmlCharRefNumber(enc, ptr); if (n < 0) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_BAD_CHAR_REF; } if (!isCdata && n == 0x20 /* space */ && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20)) break; n = XmlEncode(n, (ICHAR *)buf); if (!n) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_BAD_CHAR_REF; } for (i = 0; i < n; i++) { if (!poolAppendChar(pool, buf[i])) return XML_ERROR_NO_MEMORY; } } break; case XML_TOK_DATA_CHARS: if (!poolAppend(pool, enc, ptr, next)) return XML_ERROR_NO_MEMORY; break; break; case XML_TOK_TRAILING_CR: next = ptr + enc->minBytesPerChar; /* fall through */ case XML_TOK_ATTRIBUTE_VALUE_S: case XML_TOK_DATA_NEWLINE: if (!isCdata && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20)) break; if (!poolAppendChar(pool, 0x20)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_ENTITY_REF: { const XML_Char *name; ENTITY *entity; XML_Char ch = XmlPredefinedEntityName(enc, ptr + enc->minBytesPerChar, next - enc->minBytesPerChar); if (ch) { if (!poolAppendChar(pool, ch)) return XML_ERROR_NO_MEMORY; break; } name = poolStoreString(&temp2Pool, enc, ptr + enc->minBytesPerChar, next - enc->minBytesPerChar); if (!name) return XML_ERROR_NO_MEMORY; entity = (ENTITY *)lookup(&dtd.generalEntities, name, 0); poolDiscard(&temp2Pool); if (!entity) { if (dtd.complete) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_UNDEFINED_ENTITY; } } else if (entity->open) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_RECURSIVE_ENTITY_REF; } else if (entity->notation) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_BINARY_ENTITY_REF; } else if (!entity->textPtr) { if (enc == encoding) eventPtr = ptr; return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF; } else { enum XML_Error result; const XML_Char *textEnd = entity->textPtr + entity->textLen; entity->open = 1; result = appendAttributeValue(parser, internalEnc, isCdata, (char *)entity->textPtr, (char *)textEnd, pool); entity->open = 0; if (result) return result; } } break; default: abort(); } ptr = next; } /* not reached */ } static enum XML_Error storeEntityValue(XML_Parser parser, const char *entityTextPtr, const char *entityTextEnd) { STRING_POOL *pool = &(dtd.pool); entityTextPtr += encoding->minBytesPerChar; entityTextEnd -= encoding->minBytesPerChar; for (;;) { const char *next; int tok = XmlEntityValueTok(encoding, entityTextPtr, entityTextEnd, &next); switch (tok) { case XML_TOK_PARAM_ENTITY_REF: eventPtr = entityTextPtr; return XML_ERROR_SYNTAX; case XML_TOK_NONE: if (declEntity) { declEntity->textPtr = pool->start; declEntity->textLen = pool->ptr - pool->start; poolFinish(pool); } else poolDiscard(pool); return XML_ERROR_NONE; case XML_TOK_ENTITY_REF: case XML_TOK_DATA_CHARS: if (!poolAppend(pool, encoding, entityTextPtr, next)) return XML_ERROR_NO_MEMORY; break; case XML_TOK_TRAILING_CR: next = entityTextPtr + encoding->minBytesPerChar; /* fall through */ case XML_TOK_DATA_NEWLINE: if (pool->end == pool->ptr && !poolGrow(pool)) return XML_ERROR_NO_MEMORY; *(pool->ptr)++ = 0xA; break; case XML_TOK_CHAR_REF: { XML_Char buf[XML_ENCODE_MAX]; int i; int n = XmlCharRefNumber(encoding, entityTextPtr); if (n < 0) { eventPtr = entityTextPtr; return XML_ERROR_BAD_CHAR_REF; } n = XmlEncode(n, (ICHAR *)buf); if (!n) { eventPtr = entityTextPtr; return XML_ERROR_BAD_CHAR_REF; } for (i = 0; i < n; i++) { if (pool->end == pool->ptr && !poolGrow(pool)) return XML_ERROR_NO_MEMORY; *(pool->ptr)++ = buf[i]; } } break; case XML_TOK_PARTIAL: eventPtr = entityTextPtr; return XML_ERROR_INVALID_TOKEN; case XML_TOK_INVALID: eventPtr = next; return XML_ERROR_INVALID_TOKEN; default: abort(); } entityTextPtr = next; } /* not reached */ } static void normalizeLines(XML_Char *s) { XML_Char *p; for (;; s++) { if (*s == XML_T('\0')) return; if (*s == 0xD) break; } p = s; do { if (*s == 0xD) { *p++ = 0xA; if (*++s == 0xA) s++; } else *p++ = *s++; } while (*s); *p = XML_T('\0'); } static int reportProcessingInstruction(XML_Parser parser, const ENCODING *enc, const char *start, const char *end) { const XML_Char *target; XML_Char *data; const char *tem; if (!processingInstructionHandler) { if (defaultHandler) reportDefault(parser, enc, start, end); return 1; } start += enc->minBytesPerChar * 2; tem = start + XmlNameLength(enc, start); target = poolStoreString(&tempPool, enc, start, tem); if (!target) return 0; poolFinish(&tempPool); data = poolStoreString(&tempPool, enc, XmlSkipS(enc, tem), end - enc->minBytesPerChar*2); if (!data) return 0; normalizeLines(data); processingInstructionHandler(handlerArg, target, data); poolClear(&tempPool); return 1; } static int reportComment(XML_Parser parser, const ENCODING *enc, const char *start, const char *end) { XML_Char *data; if (!commentHandler) { if (defaultHandler) reportDefault(parser, enc, start, end); return 1; } data = poolStoreString(&tempPool, enc, start + enc->minBytesPerChar * 4, end - enc->minBytesPerChar * 3); if (!data) return 0; normalizeLines(data); commentHandler(handlerArg, data); poolClear(&tempPool); return 1; } static void reportDefault(XML_Parser parser, const ENCODING *enc, const char *s, const char *end) { if (MUST_CONVERT(enc, s)) { const char **eventPP; const char **eventEndPP; if (enc == encoding) { eventPP = &eventPtr; eventEndPP = &eventEndPtr; } else { eventPP = &(openInternalEntities->internalEventPtr); eventEndPP = &(openInternalEntities->internalEventEndPtr); } do { ICHAR *dataPtr = (ICHAR *)dataBuf; XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd); *eventEndPP = s; defaultHandler(handlerArg, dataBuf, dataPtr - (ICHAR *)dataBuf); *eventPP = s; } while (s != end); } else defaultHandler(handlerArg, (XML_Char *)s, (XML_Char *)end - (XML_Char *)s); } static int defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, int isCdata, const XML_Char *value) { DEFAULT_ATTRIBUTE *att; if (type->nDefaultAtts == type->allocDefaultAtts) { if (type->allocDefaultAtts == 0) { type->allocDefaultAtts = 8; type->defaultAtts = malloc(type->allocDefaultAtts*sizeof(DEFAULT_ATTRIBUTE)); } else { type->allocDefaultAtts *= 2; type->defaultAtts = realloc(type->defaultAtts, type->allocDefaultAtts*sizeof(DEFAULT_ATTRIBUTE)); } if (!type->defaultAtts) return 0; } att = type->defaultAtts + type->nDefaultAtts; att->id = attId; att->value = value; att->isCdata = isCdata; if (!isCdata) attId->maybeTokenized = 1; type->nDefaultAtts += 1; return 1; } static int setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType) { const XML_Char *name; for (name = elementType->name; *name; name++) { if (*name == XML_T(':')) { PREFIX *prefix; const XML_Char *s; for (s = elementType->name; s != name; s++) { if (!poolAppendChar(&dtd.pool, *s)) return 0; } if (!poolAppendChar(&dtd.pool, XML_T('\0'))) return 0; prefix = (PREFIX *)lookup(&dtd.prefixes, poolStart(&dtd.pool), sizeof(PREFIX)); if (!prefix) return 0; if (prefix->name == poolStart(&dtd.pool)) poolFinish(&dtd.pool); else poolDiscard(&dtd.pool); elementType->prefix = prefix; } } return 1; } static ATTRIBUTE_ID * getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start, const char *end) { ATTRIBUTE_ID *id; const XML_Char *name; if (!poolAppendChar(&dtd.pool, XML_T('\0'))) return 0; name = poolStoreString(&dtd.pool, enc, start, end); if (!name) return 0; ++name; id = (ATTRIBUTE_ID *)lookup(&dtd.attributeIds, name, sizeof(ATTRIBUTE_ID)); if (!id) return 0; if (id->name != name) poolDiscard(&dtd.pool); else { poolFinish(&dtd.pool); if (!ns) ; else if (name[0] == 'x' && name[1] == 'm' && name[2] == 'l' && name[3] == 'n' && name[4] == 's' && (name[5] == XML_T('\0') || name[5] == XML_T(':'))) { if (name[5] == '\0') id->prefix = &dtd.defaultPrefix; else id->prefix = (PREFIX *)lookup(&dtd.prefixes, name + 6, sizeof(PREFIX)); id->xmlns = 1; } else { int i; for (i = 0; name[i]; i++) { if (name[i] == XML_T(':')) { int j; for (j = 0; j < i; j++) { if (!poolAppendChar(&dtd.pool, name[j])) return 0; } if (!poolAppendChar(&dtd.pool, XML_T('\0'))) return 0; id->prefix = (PREFIX *)lookup(&dtd.prefixes, poolStart(&dtd.pool), sizeof(PREFIX)); if (id->prefix->name == poolStart(&dtd.pool)) poolFinish(&dtd.pool); else poolDiscard(&dtd.pool); break; } } } } return id; } #define CONTEXT_SEP XML_T('\f') static const XML_Char *getContext(XML_Parser parser) { HASH_TABLE_ITER iter; int needSep = 0; if (dtd.defaultPrefix.binding) { int i; int len; if (!poolAppendChar(&tempPool, XML_T('='))) return 0; len = dtd.defaultPrefix.binding->uriLen; if (namespaceSeparator != XML_T('\0')) len--; for (i = 0; i < len; i++) if (!poolAppendChar(&tempPool, dtd.defaultPrefix.binding->uri[i])) return 0; needSep = 1; } hashTableIterInit(&iter, &(dtd.prefixes)); for (;;) { int i; int len; const XML_Char *s; PREFIX *prefix = (PREFIX *)hashTableIterNext(&iter); if (!prefix) break; if (!prefix->binding) continue; if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP)) return 0; for (s = prefix->name; *s; s++) if (!poolAppendChar(&tempPool, *s)) return 0; if (!poolAppendChar(&tempPool, XML_T('='))) return 0; len = prefix->binding->uriLen; if (namespaceSeparator != XML_T('\0')) len--; for (i = 0; i < len; i++) if (!poolAppendChar(&tempPool, prefix->binding->uri[i])) return 0; needSep = 1; } hashTableIterInit(&iter, &(dtd.generalEntities)); for (;;) { const XML_Char *s; ENTITY *e = (ENTITY *)hashTableIterNext(&iter); if (!e) break; if (!e->open) continue; if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP)) return 0; for (s = e->name; *s; s++) if (!poolAppendChar(&tempPool, *s)) return 0; needSep = 1; } if (!poolAppendChar(&tempPool, XML_T('\0'))) return 0; return tempPool.start; } static int setContext(XML_Parser parser, const XML_Char *context) { const XML_Char *s = context; while (*context != XML_T('\0')) { if (*s == CONTEXT_SEP || *s == XML_T('\0')) { ENTITY *e; if (!poolAppendChar(&tempPool, XML_T('\0'))) return 0; e = (ENTITY *)lookup(&dtd.generalEntities, poolStart(&tempPool), 0); if (e) e->open = 1; if (*s != XML_T('\0')) s++; context = s; poolDiscard(&tempPool); } else if (*s == '=') { PREFIX *prefix; if (poolLength(&tempPool) == 0) prefix = &dtd.defaultPrefix; else { if (!poolAppendChar(&tempPool, XML_T('\0'))) return 0; prefix = (PREFIX *)lookup(&dtd.prefixes, poolStart(&tempPool), sizeof(PREFIX)); if (!prefix) return 0; if (prefix->name == poolStart(&tempPool)) poolFinish(&tempPool); else poolDiscard(&tempPool); } for (context = s + 1; *context != CONTEXT_SEP && *context != XML_T('\0'); context++) if (!poolAppendChar(&tempPool, *context)) return 0; if (!poolAppendChar(&tempPool, XML_T('\0'))) return 0; if (!addBinding(parser, prefix, 0, poolStart(&tempPool), &inheritedBindings)) return 0; poolDiscard(&tempPool); if (*context != XML_T('\0')) ++context; s = context; } else { if (!poolAppendChar(&tempPool, *s)) return 0; s++; } } return 1; } static void normalizePublicId(XML_Char *publicId) { XML_Char *p = publicId; XML_Char *s; for (s = publicId; *s; s++) { switch (*s) { case 0x20: case 0xD: case 0xA: if (p != publicId && p[-1] != 0x20) *p++ = 0x20; break; default: *p++ = *s; } } if (p != publicId && p[-1] == 0x20) --p; *p = XML_T('\0'); } static int dtdInit(DTD *p) { poolInit(&(p->pool)); hashTableInit(&(p->generalEntities)); hashTableInit(&(p->elementTypes)); hashTableInit(&(p->attributeIds)); hashTableInit(&(p->prefixes)); p->complete = 1; p->standalone = 0; p->base = 0; p->defaultPrefix.name = 0; p->defaultPrefix.binding = 0; return 1; } static void dtdDestroy(DTD *p) { HASH_TABLE_ITER iter; hashTableIterInit(&iter, &(p->elementTypes)); for (;;) { ELEMENT_TYPE *e = (ELEMENT_TYPE *)hashTableIterNext(&iter); if (!e) break; if (e->allocDefaultAtts != 0) free(e->defaultAtts); } hashTableDestroy(&(p->generalEntities)); hashTableDestroy(&(p->elementTypes)); hashTableDestroy(&(p->attributeIds)); hashTableDestroy(&(p->prefixes)); poolDestroy(&(p->pool)); } /* Do a deep copy of the DTD. Return 0 for out of memory; non-zero otherwise. The new DTD has already been initialized. */ static int dtdCopy(DTD *newDtd, const DTD *oldDtd) { HASH_TABLE_ITER iter; if (oldDtd->base) { const XML_Char *tem = poolCopyString(&(newDtd->pool), oldDtd->base); if (!tem) return 0; newDtd->base = tem; } /* Copy the prefix table. */ hashTableIterInit(&iter, &(oldDtd->prefixes)); for (;;) { const XML_Char *name; const PREFIX *oldP = (PREFIX *)hashTableIterNext(&iter); if (!oldP) break; name = poolCopyString(&(newDtd->pool), oldP->name); if (!name) return 0; if (!lookup(&(newDtd->prefixes), name, sizeof(PREFIX))) return 0; } hashTableIterInit(&iter, &(oldDtd->attributeIds)); /* Copy the attribute id table. */ for (;;) { ATTRIBUTE_ID *newA; const XML_Char *name; const ATTRIBUTE_ID *oldA = (ATTRIBUTE_ID *)hashTableIterNext(&iter); if (!oldA) break; /* Remember to allocate the scratch byte before the name. */ if (!poolAppendChar(&(newDtd->pool), XML_T('\0'))) return 0; name = poolCopyString(&(newDtd->pool), oldA->name); if (!name) return 0; ++name; newA = (ATTRIBUTE_ID *)lookup(&(newDtd->attributeIds), name, sizeof(ATTRIBUTE_ID)); if (!newA) return 0; newA->maybeTokenized = oldA->maybeTokenized; if (oldA->prefix) { newA->xmlns = oldA->xmlns; if (oldA->prefix == &oldDtd->defaultPrefix) newA->prefix = &newDtd->defaultPrefix; else newA->prefix = (PREFIX *)lookup(&(newDtd->prefixes), oldA->prefix->name, 0); } } /* Copy the element type table. */ hashTableIterInit(&iter, &(oldDtd->elementTypes)); for (;;) { int i; ELEMENT_TYPE *newE; const XML_Char *name; const ELEMENT_TYPE *oldE = (ELEMENT_TYPE *)hashTableIterNext(&iter); if (!oldE) break; name = poolCopyString(&(newDtd->pool), oldE->name); if (!name) return 0; newE = (ELEMENT_TYPE *)lookup(&(newDtd->elementTypes), name, sizeof(ELEMENT_TYPE)); if (!newE) return 0; if (oldE->nDefaultAtts) { newE->defaultAtts = (DEFAULT_ATTRIBUTE *)malloc(oldE->nDefaultAtts * sizeof(DEFAULT_ATTRIBUTE)); if (!newE->defaultAtts) return 0; } newE->allocDefaultAtts = newE->nDefaultAtts = oldE->nDefaultAtts; if (oldE->prefix) newE->prefix = (PREFIX *)lookup(&(newDtd->prefixes), oldE->prefix->name, 0); for (i = 0; i < newE->nDefaultAtts; i++) { newE->defaultAtts[i].id = (ATTRIBUTE_ID *)lookup(&(newDtd->attributeIds), oldE->defaultAtts[i].id->name, 0); newE->defaultAtts[i].isCdata = oldE->defaultAtts[i].isCdata; if (oldE->defaultAtts[i].value) { newE->defaultAtts[i].value = poolCopyString(&(newDtd->pool), oldE->defaultAtts[i].value); if (!newE->defaultAtts[i].value) return 0; } else newE->defaultAtts[i].value = 0; } } /* Copy the entity table. */ hashTableIterInit(&iter, &(oldDtd->generalEntities)); for (;;) { ENTITY *newE; const XML_Char *name; const ENTITY *oldE = (ENTITY *)hashTableIterNext(&iter); if (!oldE) break; name = poolCopyString(&(newDtd->pool), oldE->name); if (!name) return 0; newE = (ENTITY *)lookup(&(newDtd->generalEntities), name, sizeof(ENTITY)); if (!newE) return 0; if (oldE->systemId) { const XML_Char *tem = poolCopyString(&(newDtd->pool), oldE->systemId); if (!tem) return 0; newE->systemId = tem; if (oldE->base) { if (oldE->base == oldDtd->base) newE->base = newDtd->base; tem = poolCopyString(&(newDtd->pool), oldE->base); if (!tem) return 0; newE->base = tem; } } else { const XML_Char *tem = poolCopyStringN(&(newDtd->pool), oldE->textPtr, oldE->textLen); if (!tem) return 0; newE->textPtr = tem; newE->textLen = oldE->textLen; } if (oldE->notation) { const XML_Char *tem = poolCopyString(&(newDtd->pool), oldE->notation); if (!tem) return 0; newE->notation = tem; } } newDtd->complete = oldDtd->complete; newDtd->standalone = oldDtd->standalone; return 1; } static void poolInit(STRING_POOL *pool) { pool->blocks = 0; pool->freeBlocks = 0; pool->start = 0; pool->ptr = 0; pool->end = 0; } static void poolClear(STRING_POOL *pool) { if (!pool->freeBlocks) pool->freeBlocks = pool->blocks; else { BLOCK *p = pool->blocks; while (p) { BLOCK *tem = p->next; p->next = pool->freeBlocks; pool->freeBlocks = p; p = tem; } } pool->blocks = 0; pool->start = 0; pool->ptr = 0; pool->end = 0; } static void poolDestroy(STRING_POOL *pool) { BLOCK *p = pool->blocks; while (p) { BLOCK *tem = p->next; free(p); p = tem; } pool->blocks = 0; p = pool->freeBlocks; while (p) { BLOCK *tem = p->next; free(p); p = tem; } pool->freeBlocks = 0; pool->ptr = 0; pool->start = 0; pool->end = 0; } static XML_Char *poolAppend(STRING_POOL *pool, const ENCODING *enc, const char *ptr, const char *end) { if (!pool->ptr && !poolGrow(pool)) return 0; for (;;) { XmlConvert(enc, &ptr, end, (ICHAR **)&(pool->ptr), (ICHAR *)pool->end); if (ptr == end) break; if (!poolGrow(pool)) return 0; } return pool->start; } static const XML_Char *poolCopyString(STRING_POOL *pool, const XML_Char *s) { do { if (!poolAppendChar(pool, *s)) return 0; } while (*s++); s = pool->start; poolFinish(pool); return s; } static const XML_Char *poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n) { if (!pool->ptr && !poolGrow(pool)) return 0; for (; n > 0; --n, s++) { if (!poolAppendChar(pool, *s)) return 0; } s = pool->start; poolFinish(pool); return s; } static XML_Char *poolStoreString(STRING_POOL *pool, const ENCODING *enc, const char *ptr, const char *end) { if (!poolAppend(pool, enc, ptr, end)) return 0; if (pool->ptr == pool->end && !poolGrow(pool)) return 0; *(pool->ptr)++ = 0; return pool->start; } static int poolGrow(STRING_POOL *pool) { if (pool->freeBlocks) { if (pool->start == 0) { pool->blocks = pool->freeBlocks; pool->freeBlocks = pool->freeBlocks->next; pool->blocks->next = 0; pool->start = pool->blocks->s; pool->end = pool->start + pool->blocks->size; pool->ptr = pool->start; return 1; } if (pool->end - pool->start < pool->freeBlocks->size) { BLOCK *tem = pool->freeBlocks->next; pool->freeBlocks->next = pool->blocks; pool->blocks = pool->freeBlocks; pool->freeBlocks = tem; memcpy(pool->blocks->s, pool->start, (pool->end - pool->start) * sizeof(XML_Char)); pool->ptr = pool->blocks->s + (pool->ptr - pool->start); pool->start = pool->blocks->s; pool->end = pool->start + pool->blocks->size; return 1; } } if (pool->blocks && pool->start == pool->blocks->s) { int blockSize = (pool->end - pool->start)*2; pool->blocks = realloc(pool->blocks, offsetof(BLOCK, s) + blockSize * sizeof(XML_Char)); if (!pool->blocks) return 0; pool->blocks->size = blockSize; pool->ptr = pool->blocks->s + (pool->ptr - pool->start); pool->start = pool->blocks->s; pool->end = pool->start + blockSize; } else { BLOCK *tem; int blockSize = pool->end - pool->start; if (blockSize < INIT_BLOCK_SIZE) blockSize = INIT_BLOCK_SIZE; else blockSize *= 2; tem = malloc(offsetof(BLOCK, s) + blockSize * sizeof(XML_Char)); if (!tem) return 0; tem->size = blockSize; tem->next = pool->blocks; pool->blocks = tem; memcpy(tem->s, pool->start, (pool->ptr - pool->start) * sizeof(XML_Char)); pool->ptr = tem->s + (pool->ptr - pool->start); pool->start = tem->s; pool->end = tem->s + blockSize; } return 1; } �������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmlparse/hashtable.c������������������������0000664�0000764�0000764�00000007023�10435500327�025671� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in csompliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include "xmldef.h" #ifdef XML_UNICODE_WCHAR_T #ifndef XML_UNICODE #define XML_UNICODE #endif #endif #include "hashtable.h" #define INIT_SIZE 64 static int keyeq(KEY s1, KEY s2) { for (; *s1 == *s2; s1++, s2++) if (*s1 == 0) return 1; return 0; } static unsigned long hash(KEY s) { unsigned long h = 0; while (*s) h = (h << 5) + h + (unsigned char)*s++; return h; } NAMED *lookup(HASH_TABLE *table, KEY name, size_t createSize) { size_t i; if (table->size == 0) { if (!createSize) return 0; table->v = calloc(INIT_SIZE, sizeof(NAMED *)); if (!table->v) return 0; table->size = INIT_SIZE; table->usedLim = INIT_SIZE / 2; i = hash(name) & (table->size - 1); } else { unsigned long h = hash(name); for (i = h & (table->size - 1); table->v[i]; i == 0 ? i = table->size - 1 : --i) { if (keyeq(name, table->v[i]->name)) return table->v[i]; } if (!createSize) return 0; if (table->used == table->usedLim) { /* check for overflow */ size_t newSize = table->size * 2; NAMED **newV = calloc(newSize, sizeof(NAMED *)); if (!newV) return 0; for (i = 0; i < table->size; i++) if (table->v[i]) { size_t j; for (j = hash(table->v[i]->name) & (newSize - 1); newV[j]; j == 0 ? j = newSize - 1 : --j) ; newV[j] = table->v[i]; } free(table->v); table->v = newV; table->size = newSize; table->usedLim = newSize/2; for (i = h & (table->size - 1); table->v[i]; i == 0 ? i = table->size - 1 : --i) ; } } table->v[i] = calloc(1, createSize); if (!table->v[i]) return 0; table->v[i]->name = name; (table->used)++; return table->v[i]; } void hashTableDestroy(HASH_TABLE *table) { size_t i; for (i = 0; i < table->size; i++) { NAMED *p = table->v[i]; if (p) free(p); } free(table->v); } void hashTableInit(HASH_TABLE *p) { p->size = 0; p->usedLim = 0; p->used = 0; p->v = 0; } void hashTableIterInit(HASH_TABLE_ITER *iter, const HASH_TABLE *table) { iter->p = table->v; iter->end = iter->p + table->size; } NAMED *hashTableIterNext(HASH_TABLE_ITER *iter) { while (iter->p != iter->end) { NAMED *tem = *(iter->p)++; if (tem) return tem; } return 0; } �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmlparse/xmlparse.h�������������������������0000664�0000764�0000764�00000043176�10435500327�025607� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #ifndef XmlParse_INCLUDED #define XmlParse_INCLUDED 1 #ifdef __cplusplus extern "C" { #endif #ifndef XMLPARSEAPI #define XMLPARSEAPI /* as nothing */ #endif typedef void *XML_Parser; #ifdef XML_UNICODE_WCHAR_T /* XML_UNICODE_WCHAR_T will work only if sizeof(wchar_t) == 2 and wchar_t uses Unicode. */ /* Information is UTF-16 encoded as wchar_ts */ #ifndef XML_UNICODE #define XML_UNICODE #endif #include <stddef.h> typedef wchar_t XML_Char; typedef wchar_t XML_LChar; #else /* not XML_UNICODE_WCHAR_T */ #ifdef XML_UNICODE /* Information is UTF-16 encoded as unsigned shorts */ typedef unsigned short XML_Char; typedef char XML_LChar; #else /* not XML_UNICODE */ /* Information is UTF-8 encoded. */ typedef char XML_Char; typedef char XML_LChar; #endif /* not XML_UNICODE */ #endif /* not XML_UNICODE_WCHAR_T */ /* Constructs a new parser; encoding is the encoding specified by the external protocol or null if there is none specified. */ XML_Parser XMLPARSEAPI XML_ParserCreate(const XML_Char *encoding); /* Constructs a new parser and namespace processor. Element type names and attribute names that belong to a namespace will be expanded; unprefixed attribute names are never expanded; unprefixed element type names are expanded only if there is a default namespace. The expanded name is the concatenation of the namespace URI, the namespace separator character, and the local part of the name. If the namespace separator is '\0' then the namespace URI and the local part will be concatenated without any separator. When a namespace is not declared, the name and prefix will be passed through without expansion. */ XML_Parser XMLPARSEAPI XML_ParserCreateNS(const XML_Char *encoding, XML_Char namespaceSeparator); /* atts is array of name/value pairs, terminated by 0; names and values are 0 terminated. */ typedef void (*XML_StartElementHandler)(void *userData, const XML_Char *name, const XML_Char **atts); typedef void (*XML_EndElementHandler)(void *userData, const XML_Char *name); /* s is not 0 terminated. */ typedef void (*XML_CharacterDataHandler)(void *userData, const XML_Char *s, int len); /* target and data are 0 terminated */ typedef void (*XML_ProcessingInstructionHandler)(void *userData, const XML_Char *target, const XML_Char *data); /* data is 0 terminated */ typedef void (*XML_CommentHandler)(void *userData, const XML_Char *data); typedef void (*XML_StartCdataSectionHandler)(void *userData); typedef void (*XML_EndCdataSectionHandler)(void *userData); /* This is called for any characters in the XML document for which there is no applicable handler. This includes both characters that are part of markup which is of a kind that is not reported (comments, markup declarations), or characters that are part of a construct which could be reported but for which no handler has been supplied. The characters are passed exactly as they were in the XML document except that they will be encoded in UTF-8. Line boundaries are not normalized. Note that a byte order mark character is not passed to the default handler. There are no guarantees about how characters are divided between calls to the default handler: for example, a comment might be split between multiple calls. */ typedef void (*XML_DefaultHandler)(void *userData, const XML_Char *s, int len); /* This is called for a declaration of an unparsed (NDATA) entity. The base argument is whatever was set by XML_SetBase. The entityName, systemId and notationName arguments will never be null. The other arguments may be. */ typedef void (*XML_UnparsedEntityDeclHandler)(void *userData, const XML_Char *entityName, const XML_Char *base, const XML_Char *systemId, const XML_Char *publicId, const XML_Char *notationName); /* This is called for a declaration of notation. The base argument is whatever was set by XML_SetBase. The notationName will never be null. The other arguments can be. */ typedef void (*XML_NotationDeclHandler)(void *userData, const XML_Char *notationName, const XML_Char *base, const XML_Char *systemId, const XML_Char *publicId); /* When namespace processing is enabled, these are called once for each namespace declaration. The call to the start and end element handlers occur between the calls to the start and end namespace declaration handlers. For an xmlns attribute, prefix will be null. For an xmlns="" attribute, uri will be null. */ typedef void (*XML_StartNamespaceDeclHandler)(void *userData, const XML_Char *prefix, const XML_Char *uri); typedef void (*XML_EndNamespaceDeclHandler)(void *userData, const XML_Char *prefix); /* This is called if the document is not standalone (it has an external subset or a reference to a parameter entity, but does not have standalone="yes"). If this handler returns 0, then processing will not continue, and the parser will return a XML_ERROR_NOT_STANDALONE error. */ typedef int (*XML_NotStandaloneHandler)(void *userData); /* This is called for a reference to an external parsed general entity. The referenced entity is not automatically parsed. The application can parse it immediately or later using XML_ExternalEntityParserCreate. The parser argument is the parser parsing the entity containing the reference; it can be passed as the parser argument to XML_ExternalEntityParserCreate. The systemId argument is the system identifier as specified in the entity declaration; it will not be null. The base argument is the system identifier that should be used as the base for resolving systemId if systemId was relative; this is set by XML_SetBase; it may be null. The publicId argument is the public identifier as specified in the entity declaration, or null if none was specified; the whitespace in the public identifier will have been normalized as required by the XML spec. The context argument specifies the parsing context in the format expected by the context argument to XML_ExternalEntityParserCreate; context is valid only until the handler returns, so if the referenced entity is to be parsed later, it must be copied. The handler should return 0 if processing should not continue because of a fatal error in the handling of the external entity. In this case the calling parser will return an XML_ERROR_EXTERNAL_ENTITY_HANDLING error. Note that unlike other handlers the first argument is the parser, not userData. */ typedef int (*XML_ExternalEntityRefHandler)(XML_Parser parser, const XML_Char *context, const XML_Char *base, const XML_Char *systemId, const XML_Char *publicId); /* This structure is filled in by the XML_UnknownEncodingHandler to provide information to the parser about encodings that are unknown to the parser. The map[b] member gives information about byte sequences whose first byte is b. If map[b] is c where c is >= 0, then b by itself encodes the Unicode scalar value c. If map[b] is -1, then the byte sequence is malformed. If map[b] is -n, where n >= 2, then b is the first byte of an n-byte sequence that encodes a single Unicode scalar value. The data member will be passed as the first argument to the convert function. The convert function is used to convert multibyte sequences; s will point to a n-byte sequence where map[(unsigned char)*s] == -n. The convert function must return the Unicode scalar value represented by this byte sequence or -1 if the byte sequence is malformed. The convert function may be null if the encoding is a single-byte encoding, that is if map[b] >= -1 for all bytes b. When the parser is finished with the encoding, then if release is not null, it will call release passing it the data member; once release has been called, the convert function will not be called again. Expat places certain restrictions on the encodings that are supported using this mechanism. 1. Every ASCII character that can appear in a well-formed XML document, other than the characters $@\^`{}~ must be represented by a single byte, and that byte must be the same byte that represents that character in ASCII. 2. No character may require more than 4 bytes to encode. 3. All characters encoded must have Unicode scalar values <= 0xFFFF, (ie characters that would be encoded by surrogates in UTF-16 are not allowed). Note that this restriction doesn't apply to the built-in support for UTF-8 and UTF-16. 4. No Unicode character may be encoded by more than one distinct sequence of bytes. */ typedef struct { int map[256]; void *data; int (*convert)(void *data, const char *s); void (*release)(void *data); } XML_Encoding; /* This is called for an encoding that is unknown to the parser. The encodingHandlerData argument is that which was passed as the second argument to XML_SetUnknownEncodingHandler. The name argument gives the name of the encoding as specified in the encoding declaration. If the callback can provide information about the encoding, it must fill in the XML_Encoding structure, and return 1. Otherwise it must return 0. If info does not describe a suitable encoding, then the parser will return an XML_UNKNOWN_ENCODING error. */ typedef int (*XML_UnknownEncodingHandler)(void *encodingHandlerData, const XML_Char *name, XML_Encoding *info); void XMLPARSEAPI XML_SetElementHandler(XML_Parser parser, XML_StartElementHandler start, XML_EndElementHandler end); void XMLPARSEAPI XML_SetCharacterDataHandler(XML_Parser parser, XML_CharacterDataHandler handler); void XMLPARSEAPI XML_SetProcessingInstructionHandler(XML_Parser parser, XML_ProcessingInstructionHandler handler); void XMLPARSEAPI XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler); void XMLPARSEAPI XML_SetCdataSectionHandler(XML_Parser parser, XML_StartCdataSectionHandler start, XML_EndCdataSectionHandler end); /* This sets the default handler and also inhibits expansion of internal entities. The entity reference will be passed to the default handler. */ void XMLPARSEAPI XML_SetDefaultHandler(XML_Parser parser, XML_DefaultHandler handler); /* This sets the default handler but does not inhibit expansion of internal entities. The entity reference will not be passed to the default handler. */ void XMLPARSEAPI XML_SetDefaultHandlerExpand(XML_Parser parser, XML_DefaultHandler handler); void XMLPARSEAPI XML_SetUnparsedEntityDeclHandler(XML_Parser parser, XML_UnparsedEntityDeclHandler handler); void XMLPARSEAPI XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler); void XMLPARSEAPI XML_SetNamespaceDeclHandler(XML_Parser parser, XML_StartNamespaceDeclHandler start, XML_EndNamespaceDeclHandler end); void XMLPARSEAPI XML_SetNotStandaloneHandler(XML_Parser parser, XML_NotStandaloneHandler handler); void XMLPARSEAPI XML_SetExternalEntityRefHandler(XML_Parser parser, XML_ExternalEntityRefHandler handler); /* If a non-null value for arg is specified here, then it will be passed as the first argument to the external entity ref handler instead of the parser object. */ void XMLPARSEAPI XML_SetExternalEntityRefHandlerArg(XML_Parser, void *arg); void XMLPARSEAPI XML_SetUnknownEncodingHandler(XML_Parser parser, XML_UnknownEncodingHandler handler, void *encodingHandlerData); /* This can be called within a handler for a start element, end element, processing instruction or character data. It causes the corresponding markup to be passed to the default handler. */ void XMLPARSEAPI XML_DefaultCurrent(XML_Parser parser); /* This value is passed as the userData argument to callbacks. */ void XMLPARSEAPI XML_SetUserData(XML_Parser parser, void *userData); /* Returns the last value set by XML_SetUserData or null. */ #define XML_GetUserData(parser) (*(void **)(parser)) /* This is equivalent to supplying an encoding argument to XML_CreateParser. It must not be called after XML_Parse or XML_ParseBuffer. */ int XMLPARSEAPI XML_SetEncoding(XML_Parser parser, const XML_Char *encoding); /* If this function is called, then the parser will be passed as the first argument to callbacks instead of userData. The userData will still be accessible using XML_GetUserData. */ void XMLPARSEAPI XML_UseParserAsHandlerArg(XML_Parser parser); /* Sets the base to be used for resolving relative URIs in system identifiers in declarations. Resolving relative identifiers is left to the application: this value will be passed through as the base argument to the XML_ExternalEntityRefHandler, XML_NotationDeclHandler and XML_UnparsedEntityDeclHandler. The base argument will be copied. Returns zero if out of memory, non-zero otherwise. */ int XMLPARSEAPI XML_SetBase(XML_Parser parser, const XML_Char *base); const XML_Char XMLPARSEAPI * XML_GetBase(XML_Parser parser); /* Returns the number of the attributes passed in last call to the XML_StartElementHandler that were specified in the start-tag rather than defaulted. */ int XMLPARSEAPI XML_GetSpecifiedAttributeCount(XML_Parser parser); /* Parses some input. Returns 0 if a fatal error is detected. The last call to XML_Parse must have isFinal true; len may be zero for this call (or any other). */ int XMLPARSEAPI XML_Parse(XML_Parser parser, const char *s, int len, int isFinal); void XMLPARSEAPI * XML_GetBuffer(XML_Parser parser, int len); int XMLPARSEAPI XML_ParseBuffer(XML_Parser parser, int len, int isFinal); /* Creates an XML_Parser object that can parse an external general entity; context is a '\0'-terminated string specifying the parse context; encoding is a '\0'-terminated string giving the name of the externally specified encoding, or null if there is no externally specified encoding. The context string consists of a sequence of tokens separated by formfeeds (\f); a token consisting of a name specifies that the general entity of the name is open; a token of the form prefix=uri specifies the namespace for a particular prefix; a token of the form =uri specifies the default namespace. This can be called at any point after the first call to an ExternalEntityRefHandler so longer as the parser has not yet been freed. The new parser is completely independent and may safely be used in a separate thread. The handlers and userData are initialized from the parser argument. Returns 0 if out of memory. Otherwise returns a new XML_Parser object. */ XML_Parser XMLPARSEAPI XML_ExternalEntityParserCreate(XML_Parser parser, const XML_Char *context, const XML_Char *encoding); enum XML_Error { XML_ERROR_NONE, XML_ERROR_NO_MEMORY, XML_ERROR_SYNTAX, XML_ERROR_NO_ELEMENTS, XML_ERROR_INVALID_TOKEN, XML_ERROR_UNCLOSED_TOKEN, XML_ERROR_PARTIAL_CHAR, XML_ERROR_TAG_MISMATCH, XML_ERROR_DUPLICATE_ATTRIBUTE, XML_ERROR_JUNK_AFTER_DOC_ELEMENT, XML_ERROR_PARAM_ENTITY_REF, XML_ERROR_UNDEFINED_ENTITY, XML_ERROR_RECURSIVE_ENTITY_REF, XML_ERROR_ASYNC_ENTITY, XML_ERROR_BAD_CHAR_REF, XML_ERROR_BINARY_ENTITY_REF, XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF, XML_ERROR_MISPLACED_XML_PI, XML_ERROR_UNKNOWN_ENCODING, XML_ERROR_INCORRECT_ENCODING, XML_ERROR_UNCLOSED_CDATA_SECTION, XML_ERROR_EXTERNAL_ENTITY_HANDLING, XML_ERROR_NOT_STANDALONE }; /* If XML_Parse or XML_ParseBuffer have returned 0, then XML_GetErrorCode returns information about the error. */ enum XML_Error XMLPARSEAPI XML_GetErrorCode(XML_Parser parser); /* These functions return information about the current parse location. They may be called when XML_Parse or XML_ParseBuffer return 0; in this case the location is the location of the character at which the error was detected. They may also be called from any other callback called to report some parse event; in this the location is the location of the first of the sequence of characters that generated the event. */ int XMLPARSEAPI XML_GetCurrentLineNumber(XML_Parser parser); int XMLPARSEAPI XML_GetCurrentColumnNumber(XML_Parser parser); long XMLPARSEAPI XML_GetCurrentByteIndex(XML_Parser parser); /* Return the number of bytes in the current event. Returns 0 if the event is in an internal entity. */ int XMLPARSEAPI XML_GetCurrentByteCount(XML_Parser parser); /* For backwards compatibility with previous versions. */ #define XML_GetErrorLineNumber XML_GetCurrentLineNumber #define XML_GetErrorColumnNumber XML_GetCurrentColumnNumber #define XML_GetErrorByteIndex XML_GetCurrentByteIndex /* Frees memory used by the parser. */ void XMLPARSEAPI XML_ParserFree(XML_Parser parser); /* Returns a string describing the error. */ const XML_LChar XMLPARSEAPI *XML_ErrorString(int code); #ifdef __cplusplus } #endif #endif /* not XmlParse_INCLUDED */ ��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmlparse/hashtable.h������������������������0000664�0000764�0000764�00000003724�10435500327�025702� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include <stddef.h> #ifdef XML_UNICODE #ifdef XML_UNICODE_WCHAR_T typedef const wchar_t *KEY; #else /* not XML_UNICODE_WCHAR_T */ typedef const unsigned short *KEY; #endif /* not XML_UNICODE_WCHAR_T */ #else /* not XML_UNICODE */ typedef const char *KEY; #endif /* not XML_UNICODE */ typedef struct { KEY name; } NAMED; typedef struct { NAMED **v; size_t size; size_t used; size_t usedLim; } HASH_TABLE; NAMED *lookup(HASH_TABLE *table, KEY name, size_t createSize); void hashTableInit(HASH_TABLE *); void hashTableDestroy(HASH_TABLE *); typedef struct { NAMED **p; NAMED **end; } HASH_TABLE_ITER; void hashTableIterInit(HASH_TABLE_ITER *, const HASH_TABLE *); NAMED *hashTableIterNext(HASH_TABLE_ITER *); ��������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/�������������������������������������0000775�0000764�0000764�00000000000�10435500330�023245� 5����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmltok_impl.h������������������������0000664�0000764�0000764�00000003434�10435500330�025761� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ enum { BT_NONXML, BT_MALFORM, BT_LT, BT_AMP, BT_RSQB, BT_LEAD2, BT_LEAD3, BT_LEAD4, BT_TRAIL, BT_CR, BT_LF, BT_GT, BT_QUOT, BT_APOS, BT_EQUALS, BT_QUEST, BT_EXCL, BT_SOL, BT_SEMI, BT_NUM, BT_LSQB, BT_S, BT_NMSTRT, BT_COLON, BT_HEX, BT_DIGIT, BT_NAME, BT_MINUS, BT_OTHER, /* known not to be a name or name start character */ BT_NONASCII, /* might be a name or name start character */ BT_PERCNT, BT_LPAR, BT_RPAR, BT_AST, BT_PLUS, BT_COMMA, BT_VERBAR }; #include <stddef.h> ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmltok.c�����������������������������0000664�0000764�0000764�00000110243�10435500330�024730� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include "xmldef.h" #include "xmltok.h" #include "nametab.h" #define VTABLE1 \ { PREFIX(prologTok), PREFIX(contentTok), PREFIX(cdataSectionTok) }, \ { PREFIX(attributeValueTok), PREFIX(entityValueTok) }, \ PREFIX(sameName), \ PREFIX(nameMatchesAscii), \ PREFIX(nameLength), \ PREFIX(skipS), \ PREFIX(getAtts), \ PREFIX(charRefNumber), \ PREFIX(predefinedEntityName), \ PREFIX(updatePosition), \ PREFIX(isPublicId) #define VTABLE VTABLE1, PREFIX(toUtf8), PREFIX(toUtf16) #define UCS2_GET_NAMING(pages, hi, lo) \ (namingBitmap[(pages[hi] << 3) + ((lo) >> 5)] & (1 << ((lo) & 0x1F))) /* A 2 byte UTF-8 representation splits the characters 11 bits between the bottom 5 and 6 bits of the bytes. We need 8 bits to index into pages, 3 bits to add to that index and 5 bits to generate the mask. */ #define UTF8_GET_NAMING2(pages, byte) \ (namingBitmap[((pages)[(((byte)[0]) >> 2) & 7] << 3) \ + ((((byte)[0]) & 3) << 1) \ + ((((byte)[1]) >> 5) & 1)] \ & (1 << (((byte)[1]) & 0x1F))) /* A 3 byte UTF-8 representation splits the characters 16 bits between the bottom 4, 6 and 6 bits of the bytes. We need 8 bits to index into pages, 3 bits to add to that index and 5 bits to generate the mask. */ #define UTF8_GET_NAMING3(pages, byte) \ (namingBitmap[((pages)[((((byte)[0]) & 0xF) << 4) \ + ((((byte)[1]) >> 2) & 0xF)] \ << 3) \ + ((((byte)[1]) & 3) << 1) \ + ((((byte)[2]) >> 5) & 1)] \ & (1 << (((byte)[2]) & 0x1F))) #define UTF8_GET_NAMING(pages, p, n) \ ((n) == 2 \ ? UTF8_GET_NAMING2(pages, (const unsigned char *)(p)) \ : ((n) == 3 \ ? UTF8_GET_NAMING3(pages, (const unsigned char *)(p)) \ : 0)) #define UTF8_INVALID3(p) \ ((*p) == 0xED \ ? (((p)[1] & 0x20) != 0) \ : ((*p) == 0xEF \ ? ((p)[1] == 0xBF && ((p)[2] == 0xBF || (p)[2] == 0xBE)) \ : 0)) #define UTF8_INVALID4(p) ((*p) == 0xF4 && ((p)[1] & 0x30) != 0) static int isNever(const ENCODING *enc, const char *p) { return 0; } static int utf8_isName2(const ENCODING *enc, const char *p) { return UTF8_GET_NAMING2(namePages, (const unsigned char *)p); } static int utf8_isName3(const ENCODING *enc, const char *p) { return UTF8_GET_NAMING3(namePages, (const unsigned char *)p); } #define utf8_isName4 isNever static int utf8_isNmstrt2(const ENCODING *enc, const char *p) { return UTF8_GET_NAMING2(nmstrtPages, (const unsigned char *)p); } static int utf8_isNmstrt3(const ENCODING *enc, const char *p) { return UTF8_GET_NAMING3(nmstrtPages, (const unsigned char *)p); } #define utf8_isNmstrt4 isNever #define utf8_isInvalid2 isNever static int utf8_isInvalid3(const ENCODING *enc, const char *p) { return UTF8_INVALID3((const unsigned char *)p); } static int utf8_isInvalid4(const ENCODING *enc, const char *p) { return UTF8_INVALID4((const unsigned char *)p); } struct normal_encoding { ENCODING enc; unsigned char type[256]; #ifdef XML_MIN_SIZE int (*byteType)(const ENCODING *, const char *); int (*isNameMin)(const ENCODING *, const char *); int (*isNmstrtMin)(const ENCODING *, const char *); int (*byteToAscii)(const ENCODING *, const char *); int (*charMatches)(const ENCODING *, const char *, int); #endif /* XML_MIN_SIZE */ int (*isName2)(const ENCODING *, const char *); int (*isName3)(const ENCODING *, const char *); int (*isName4)(const ENCODING *, const char *); int (*isNmstrt2)(const ENCODING *, const char *); int (*isNmstrt3)(const ENCODING *, const char *); int (*isNmstrt4)(const ENCODING *, const char *); int (*isInvalid2)(const ENCODING *, const char *); int (*isInvalid3)(const ENCODING *, const char *); int (*isInvalid4)(const ENCODING *, const char *); }; #ifdef XML_MIN_SIZE #define STANDARD_VTABLE(E) \ E ## byteType, \ E ## isNameMin, \ E ## isNmstrtMin, \ E ## byteToAscii, \ E ## charMatches, #else #define STANDARD_VTABLE(E) /* as nothing */ #endif #define NORMAL_VTABLE(E) \ E ## isName2, \ E ## isName3, \ E ## isName4, \ E ## isNmstrt2, \ E ## isNmstrt3, \ E ## isNmstrt4, \ E ## isInvalid2, \ E ## isInvalid3, \ E ## isInvalid4 static int checkCharRefNumber(int); #include "xmltok_impl.h" #ifdef XML_MIN_SIZE #define sb_isNameMin isNever #define sb_isNmstrtMin isNever #endif #ifdef XML_MIN_SIZE #define MINBPC(enc) ((enc)->minBytesPerChar) #else /* minimum bytes per character */ #define MINBPC(enc) 1 #endif #define SB_BYTE_TYPE(enc, p) \ (((struct normal_encoding *)(enc))->type[(unsigned char)*(p)]) #ifdef XML_MIN_SIZE static int sb_byteType(const ENCODING *enc, const char *p) { return SB_BYTE_TYPE(enc, p); } #define BYTE_TYPE(enc, p) \ (((const struct normal_encoding *)(enc))->byteType(enc, p)) #else #define BYTE_TYPE(enc, p) SB_BYTE_TYPE(enc, p) #endif #ifdef XML_MIN_SIZE #define BYTE_TO_ASCII(enc, p) \ (((const struct normal_encoding *)(enc))->byteToAscii(enc, p)) static int sb_byteToAscii(const ENCODING *enc, const char *p) { return *p; } #else #define BYTE_TO_ASCII(enc, p) (*p) #endif #define IS_NAME_CHAR(enc, p, n) \ (((const struct normal_encoding *)(enc))->isName ## n(enc, p)) #define IS_NMSTRT_CHAR(enc, p, n) \ (((const struct normal_encoding *)(enc))->isNmstrt ## n(enc, p)) #define IS_INVALID_CHAR(enc, p, n) \ (((const struct normal_encoding *)(enc))->isInvalid ## n(enc, p)) #ifdef XML_MIN_SIZE #define IS_NAME_CHAR_MINBPC(enc, p) \ (((const struct normal_encoding *)(enc))->isNameMin(enc, p)) #define IS_NMSTRT_CHAR_MINBPC(enc, p) \ (((const struct normal_encoding *)(enc))->isNmstrtMin(enc, p)) #else #define IS_NAME_CHAR_MINBPC(enc, p) (0) #define IS_NMSTRT_CHAR_MINBPC(enc, p) (0) #endif #ifdef XML_MIN_SIZE #define CHAR_MATCHES(enc, p, c) \ (((const struct normal_encoding *)(enc))->charMatches(enc, p, c)) static int sb_charMatches(const ENCODING *enc, const char *p, int c) { return *p == c; } #else /* c is an ASCII character */ #define CHAR_MATCHES(enc, p, c) (*(p) == c) #endif #define PREFIX(ident) normal_ ## ident #include "xmltok_impl.c" #undef MINBPC #undef BYTE_TYPE #undef BYTE_TO_ASCII #undef CHAR_MATCHES #undef IS_NAME_CHAR #undef IS_NAME_CHAR_MINBPC #undef IS_NMSTRT_CHAR #undef IS_NMSTRT_CHAR_MINBPC #undef IS_INVALID_CHAR enum { /* UTF8_cvalN is value of masked first byte of N byte sequence */ UTF8_cval1 = 0x00, UTF8_cval2 = 0xc0, UTF8_cval3 = 0xe0, UTF8_cval4 = 0xf0 }; static void utf8_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim, char **toP, const char *toLim) { char *to; const char *from; if (fromLim - *fromP > toLim - *toP) { /* Avoid copying partial characters. */ for (fromLim = *fromP + (toLim - *toP); fromLim > *fromP; fromLim--) if (((unsigned char)fromLim[-1] & 0xc0) != 0x80) break; } for (to = *toP, from = *fromP; from != fromLim; from++, to++) *to = *from; *fromP = from; *toP = to; } static void utf8_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim, unsigned short **toP, const unsigned short *toLim) { unsigned short *to = *toP; const char *from = *fromP; while (from != fromLim && to != toLim) { switch (((struct normal_encoding *)enc)->type[(unsigned char)*from]) { case BT_LEAD2: *to++ = ((from[0] & 0x1f) << 6) | (from[1] & 0x3f); from += 2; break; case BT_LEAD3: *to++ = ((from[0] & 0xf) << 12) | ((from[1] & 0x3f) << 6) | (from[2] & 0x3f); from += 3; break; case BT_LEAD4: { unsigned long n; if (to + 1 == toLim) break; n = ((from[0] & 0x7) << 18) | ((from[1] & 0x3f) << 12) | ((from[2] & 0x3f) << 6) | (from[3] & 0x3f); n -= 0x10000; to[0] = (unsigned short)((n >> 10) | 0xD800); to[1] = (unsigned short)((n & 0x3FF) | 0xDC00); to += 2; from += 4; } break; default: *to++ = *from++; break; } } *fromP = from; *toP = to; } #ifdef XML_NS static const struct normal_encoding utf8_encoding_ns = { { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 }, { #include "asciitab.h" #include "utf8tab.h" }, STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_) }; #endif static const struct normal_encoding utf8_encoding = { { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 }, { #define BT_COLON BT_NMSTRT #include "asciitab.h" #undef BT_COLON #include "utf8tab.h" }, STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_) }; #ifdef XML_NS static const struct normal_encoding internal_utf8_encoding_ns = { { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 }, { #include "iasciitab.h" #include "utf8tab.h" }, STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_) }; #endif static const struct normal_encoding internal_utf8_encoding = { { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 }, { #define BT_COLON BT_NMSTRT #include "iasciitab.h" #undef BT_COLON #include "utf8tab.h" }, STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_) }; static void latin1_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim, char **toP, const char *toLim) { for (;;) { unsigned char c; if (*fromP == fromLim) break; c = (unsigned char)**fromP; if (c & 0x80) { if (toLim - *toP < 2) break; *(*toP)++ = ((c >> 6) | UTF8_cval2); *(*toP)++ = ((c & 0x3f) | 0x80); (*fromP)++; } else { if (*toP == toLim) break; *(*toP)++ = *(*fromP)++; } } } static void latin1_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim, unsigned short **toP, const unsigned short *toLim) { while (*fromP != fromLim && *toP != toLim) *(*toP)++ = (unsigned char)*(*fromP)++; } #ifdef XML_NS static const struct normal_encoding latin1_encoding_ns = { { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 }, { #include "asciitab.h" #include "latin1tab.h" }, STANDARD_VTABLE(sb_) }; #endif static const struct normal_encoding latin1_encoding = { { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 }, { #define BT_COLON BT_NMSTRT #include "asciitab.h" #undef BT_COLON #include "latin1tab.h" }, STANDARD_VTABLE(sb_) }; static void ascii_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim, char **toP, const char *toLim) { while (*fromP != fromLim && *toP != toLim) *(*toP)++ = *(*fromP)++; } #ifdef XML_NS static const struct normal_encoding ascii_encoding_ns = { { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 }, { #include "asciitab.h" /* BT_NONXML == 0 */ }, STANDARD_VTABLE(sb_) }; #endif static const struct normal_encoding ascii_encoding = { { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 }, { #define BT_COLON BT_NMSTRT #include "asciitab.h" #undef BT_COLON /* BT_NONXML == 0 */ }, STANDARD_VTABLE(sb_) }; static int unicode_byte_type(char hi, char lo) { switch ((unsigned char)hi) { case 0xD8: case 0xD9: case 0xDA: case 0xDB: return BT_LEAD4; case 0xDC: case 0xDD: case 0xDE: case 0xDF: return BT_TRAIL; case 0xFF: switch ((unsigned char)lo) { case 0xFF: case 0xFE: return BT_NONXML; } break; } return BT_NONASCII; } #define DEFINE_UTF16_TO_UTF8(E) \ static \ void E ## toUtf8(const ENCODING *enc, \ const char **fromP, const char *fromLim, \ char **toP, const char *toLim) \ { \ const char *from; \ for (from = *fromP; from != fromLim; from += 2) { \ int plane; \ unsigned char lo2; \ unsigned char lo = GET_LO(from); \ unsigned char hi = GET_HI(from); \ switch (hi) { \ case 0: \ if (lo < 0x80) { \ if (*toP == toLim) { \ *fromP = from; \ return; \ } \ *(*toP)++ = lo; \ break; \ } \ /* fall through */ \ case 0x1: case 0x2: case 0x3: \ case 0x4: case 0x5: case 0x6: case 0x7: \ if (toLim - *toP < 2) { \ *fromP = from; \ return; \ } \ *(*toP)++ = ((lo >> 6) | (hi << 2) | UTF8_cval2); \ *(*toP)++ = ((lo & 0x3f) | 0x80); \ break; \ default: \ if (toLim - *toP < 3) { \ *fromP = from; \ return; \ } \ /* 16 bits divided 4, 6, 6 amongst 3 bytes */ \ *(*toP)++ = ((hi >> 4) | UTF8_cval3); \ *(*toP)++ = (((hi & 0xf) << 2) | (lo >> 6) | 0x80); \ *(*toP)++ = ((lo & 0x3f) | 0x80); \ break; \ case 0xD8: case 0xD9: case 0xDA: case 0xDB: \ if (toLim - *toP < 4) { \ *fromP = from; \ return; \ } \ plane = (((hi & 0x3) << 2) | ((lo >> 6) & 0x3)) + 1; \ *(*toP)++ = ((plane >> 2) | UTF8_cval4); \ *(*toP)++ = (((lo >> 2) & 0xF) | ((plane & 0x3) << 4) | 0x80); \ from += 2; \ lo2 = GET_LO(from); \ *(*toP)++ = (((lo & 0x3) << 4) \ | ((GET_HI(from) & 0x3) << 2) \ | (lo2 >> 6) \ | 0x80); \ *(*toP)++ = ((lo2 & 0x3f) | 0x80); \ break; \ } \ } \ *fromP = from; \ } #define DEFINE_UTF16_TO_UTF16(E) \ static \ void E ## toUtf16(const ENCODING *enc, \ const char **fromP, const char *fromLim, \ unsigned short **toP, const unsigned short *toLim) \ { \ /* Avoid copying first half only of surrogate */ \ if (fromLim - *fromP > ((toLim - *toP) << 1) \ && (GET_HI(fromLim - 2) & 0xF8) == 0xD8) \ fromLim -= 2; \ for (; *fromP != fromLim && *toP != toLim; *fromP += 2) \ *(*toP)++ = (GET_HI(*fromP) << 8) | GET_LO(*fromP); \ } #define SET2(ptr, ch) \ (((ptr)[0] = ((ch) & 0xff)), ((ptr)[1] = ((ch) >> 8))) #define GET_LO(ptr) ((unsigned char)(ptr)[0]) #define GET_HI(ptr) ((unsigned char)(ptr)[1]) DEFINE_UTF16_TO_UTF8(little2_) DEFINE_UTF16_TO_UTF16(little2_) #undef SET2 #undef GET_LO #undef GET_HI #define SET2(ptr, ch) \ (((ptr)[0] = ((ch) >> 8)), ((ptr)[1] = ((ch) & 0xFF))) #define GET_LO(ptr) ((unsigned char)(ptr)[1]) #define GET_HI(ptr) ((unsigned char)(ptr)[0]) DEFINE_UTF16_TO_UTF8(big2_) DEFINE_UTF16_TO_UTF16(big2_) #undef SET2 #undef GET_LO #undef GET_HI #define LITTLE2_BYTE_TYPE(enc, p) \ ((p)[1] == 0 \ ? ((struct normal_encoding *)(enc))->type[(unsigned char)*(p)] \ : unicode_byte_type((p)[1], (p)[0])) #define LITTLE2_BYTE_TO_ASCII(enc, p) ((p)[1] == 0 ? (p)[0] : -1) #define LITTLE2_CHAR_MATCHES(enc, p, c) ((p)[1] == 0 && (p)[0] == c) #define LITTLE2_IS_NAME_CHAR_MINBPC(enc, p) \ UCS2_GET_NAMING(namePages, (unsigned char)p[1], (unsigned char)p[0]) #define LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p) \ UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[1], (unsigned char)p[0]) #ifdef XML_MIN_SIZE static int little2_byteType(const ENCODING *enc, const char *p) { return LITTLE2_BYTE_TYPE(enc, p); } static int little2_byteToAscii(const ENCODING *enc, const char *p) { return LITTLE2_BYTE_TO_ASCII(enc, p); } static int little2_charMatches(const ENCODING *enc, const char *p, int c) { return LITTLE2_CHAR_MATCHES(enc, p, c); } static int little2_isNameMin(const ENCODING *enc, const char *p) { return LITTLE2_IS_NAME_CHAR_MINBPC(enc, p); } static int little2_isNmstrtMin(const ENCODING *enc, const char *p) { return LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p); } #undef VTABLE #define VTABLE VTABLE1, little2_toUtf8, little2_toUtf16 #else /* not XML_MIN_SIZE */ #undef PREFIX #define PREFIX(ident) little2_ ## ident #define MINBPC(enc) 2 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */ #define BYTE_TYPE(enc, p) LITTLE2_BYTE_TYPE(enc, p) #define BYTE_TO_ASCII(enc, p) LITTLE2_BYTE_TO_ASCII(enc, p) #define CHAR_MATCHES(enc, p, c) LITTLE2_CHAR_MATCHES(enc, p, c) #define IS_NAME_CHAR(enc, p, n) 0 #define IS_NAME_CHAR_MINBPC(enc, p) LITTLE2_IS_NAME_CHAR_MINBPC(enc, p) #define IS_NMSTRT_CHAR(enc, p, n) (0) #define IS_NMSTRT_CHAR_MINBPC(enc, p) LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p) #include "xmltok_impl.c" #undef MINBPC #undef BYTE_TYPE #undef BYTE_TO_ASCII #undef CHAR_MATCHES #undef IS_NAME_CHAR #undef IS_NAME_CHAR_MINBPC #undef IS_NMSTRT_CHAR #undef IS_NMSTRT_CHAR_MINBPC #undef IS_INVALID_CHAR #endif /* not XML_MIN_SIZE */ #ifdef XML_NS static const struct normal_encoding little2_encoding_ns = { { VTABLE, 2, 0, #if XML_BYTE_ORDER == 12 1 #else 0 #endif }, { #include "asciitab.h" #include "latin1tab.h" }, STANDARD_VTABLE(little2_) }; #endif static const struct normal_encoding little2_encoding = { { VTABLE, 2, 0, #if XML_BYTE_ORDER == 12 1 #else 0 #endif }, { #define BT_COLON BT_NMSTRT #include "asciitab.h" #undef BT_COLON #include "latin1tab.h" }, STANDARD_VTABLE(little2_) }; #if XML_BYTE_ORDER != 21 #ifdef XML_NS static const struct normal_encoding internal_little2_encoding_ns = { { VTABLE, 2, 0, 1 }, { #include "iasciitab.h" #include "latin1tab.h" }, STANDARD_VTABLE(little2_) }; #endif static const struct normal_encoding internal_little2_encoding = { { VTABLE, 2, 0, 1 }, { #define BT_COLON BT_NMSTRT #include "iasciitab.h" #undef BT_COLON #include "latin1tab.h" }, STANDARD_VTABLE(little2_) }; #endif #define BIG2_BYTE_TYPE(enc, p) \ ((p)[0] == 0 \ ? ((struct normal_encoding *)(enc))->type[(unsigned char)(p)[1]] \ : unicode_byte_type((p)[0], (p)[1])) #define BIG2_BYTE_TO_ASCII(enc, p) ((p)[0] == 0 ? (p)[1] : -1) #define BIG2_CHAR_MATCHES(enc, p, c) ((p)[0] == 0 && (p)[1] == c) #define BIG2_IS_NAME_CHAR_MINBPC(enc, p) \ UCS2_GET_NAMING(namePages, (unsigned char)p[0], (unsigned char)p[1]) #define BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p) \ UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[0], (unsigned char)p[1]) #ifdef XML_MIN_SIZE static int big2_byteType(const ENCODING *enc, const char *p) { return BIG2_BYTE_TYPE(enc, p); } static int big2_byteToAscii(const ENCODING *enc, const char *p) { return BIG2_BYTE_TO_ASCII(enc, p); } static int big2_charMatches(const ENCODING *enc, const char *p, int c) { return BIG2_CHAR_MATCHES(enc, p, c); } static int big2_isNameMin(const ENCODING *enc, const char *p) { return BIG2_IS_NAME_CHAR_MINBPC(enc, p); } static int big2_isNmstrtMin(const ENCODING *enc, const char *p) { return BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p); } #undef VTABLE #define VTABLE VTABLE1, big2_toUtf8, big2_toUtf16 #else /* not XML_MIN_SIZE */ #undef PREFIX #define PREFIX(ident) big2_ ## ident #define MINBPC(enc) 2 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */ #define BYTE_TYPE(enc, p) BIG2_BYTE_TYPE(enc, p) #define BYTE_TO_ASCII(enc, p) BIG2_BYTE_TO_ASCII(enc, p) #define CHAR_MATCHES(enc, p, c) BIG2_CHAR_MATCHES(enc, p, c) #define IS_NAME_CHAR(enc, p, n) 0 #define IS_NAME_CHAR_MINBPC(enc, p) BIG2_IS_NAME_CHAR_MINBPC(enc, p) #define IS_NMSTRT_CHAR(enc, p, n) (0) #define IS_NMSTRT_CHAR_MINBPC(enc, p) BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p) #include "xmltok_impl.c" #undef MINBPC #undef BYTE_TYPE #undef BYTE_TO_ASCII #undef CHAR_MATCHES #undef IS_NAME_CHAR #undef IS_NAME_CHAR_MINBPC #undef IS_NMSTRT_CHAR #undef IS_NMSTRT_CHAR_MINBPC #undef IS_INVALID_CHAR #endif /* not XML_MIN_SIZE */ #ifdef XML_NS static const struct normal_encoding big2_encoding_ns = { { VTABLE, 2, 0, #if XML_BYTE_ORDER == 21 1 #else 0 #endif }, { #include "asciitab.h" #include "latin1tab.h" }, STANDARD_VTABLE(big2_) }; #endif static const struct normal_encoding big2_encoding = { { VTABLE, 2, 0, #if XML_BYTE_ORDER == 21 1 #else 0 #endif }, { #define BT_COLON BT_NMSTRT #include "asciitab.h" #undef BT_COLON #include "latin1tab.h" }, STANDARD_VTABLE(big2_) }; #if XML_BYTE_ORDER != 12 #ifdef XML_NS static const struct normal_encoding internal_big2_encoding_ns = { { VTABLE, 2, 0, 1 }, { #include "iasciitab.h" #include "latin1tab.h" }, STANDARD_VTABLE(big2_) }; #endif static const struct normal_encoding internal_big2_encoding = { { VTABLE, 2, 0, 1 }, { #define BT_COLON BT_NMSTRT #include "iasciitab.h" #undef BT_COLON #include "latin1tab.h" }, STANDARD_VTABLE(big2_) }; #endif #undef PREFIX static int streqci(const char *s1, const char *s2) { for (;;) { char c1 = *s1++; char c2 = *s2++; if ('a' <= c1 && c1 <= 'z') c1 += 'A' - 'a'; if ('a' <= c2 && c2 <= 'z') c2 += 'A' - 'a'; if (c1 != c2) return 0; if (!c1) break; } return 1; } static void initUpdatePosition(const ENCODING *enc, const char *ptr, const char *end, POSITION *pos) { normal_updatePosition(&utf8_encoding.enc, ptr, end, pos); } static int toAscii(const ENCODING *enc, const char *ptr, const char *end) { char buf[1]; char *p = buf; XmlUtf8Convert(enc, &ptr, end, &p, p + 1); if (p == buf) return -1; else return buf[0]; } static int isSpace(int c) { switch (c) { case 0x20: case 0xD: case 0xA: case 0x9: return 1; } return 0; } /* Return 1 if there's just optional white space or there's an S followed by name=val. */ static int parsePseudoAttribute(const ENCODING *enc, const char *ptr, const char *end, const char **namePtr, const char **valPtr, const char **nextTokPtr) { int c; char open; if (ptr == end) { *namePtr = 0; return 1; } if (!isSpace(toAscii(enc, ptr, end))) { *nextTokPtr = ptr; return 0; } do { ptr += enc->minBytesPerChar; } while (isSpace(toAscii(enc, ptr, end))); if (ptr == end) { *namePtr = 0; return 1; } *namePtr = ptr; for (;;) { c = toAscii(enc, ptr, end); if (c == -1) { *nextTokPtr = ptr; return 0; } if (c == '=') break; if (isSpace(c)) { do { ptr += enc->minBytesPerChar; } while (isSpace(c = toAscii(enc, ptr, end))); if (c != '=') { *nextTokPtr = ptr; return 0; } break; } ptr += enc->minBytesPerChar; } if (ptr == *namePtr) { *nextTokPtr = ptr; return 0; } ptr += enc->minBytesPerChar; c = toAscii(enc, ptr, end); while (isSpace(c)) { ptr += enc->minBytesPerChar; c = toAscii(enc, ptr, end); } if (c != '"' && c != '\'') { *nextTokPtr = ptr; return 0; } open = c; ptr += enc->minBytesPerChar; *valPtr = ptr; for (;; ptr += enc->minBytesPerChar) { c = toAscii(enc, ptr, end); if (c == open) break; if (!('a' <= c && c <= 'z') && !('A' <= c && c <= 'Z') && !('0' <= c && c <= '9') && c != '.' && c != '-' && c != '_') { *nextTokPtr = ptr; return 0; } } *nextTokPtr = ptr + enc->minBytesPerChar; return 1; } static int doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *, const char *, const char *), int isGeneralTextEntity, const ENCODING *enc, const char *ptr, const char *end, const char **badPtr, const char **versionPtr, const char **encodingName, const ENCODING **encoding, int *standalone) { const char *val = 0; const char *name = 0; ptr += 5 * enc->minBytesPerChar; end -= 2 * enc->minBytesPerChar; if (!parsePseudoAttribute(enc, ptr, end, &name, &val, &ptr) || !name) { *badPtr = ptr; return 0; } if (!XmlNameMatchesAscii(enc, name, "version")) { if (!isGeneralTextEntity) { *badPtr = name; return 0; } } else { if (versionPtr) *versionPtr = val; if (!parsePseudoAttribute(enc, ptr, end, &name, &val, &ptr)) { *badPtr = ptr; return 0; } if (!name) { if (isGeneralTextEntity) { /* a TextDecl must have an EncodingDecl */ *badPtr = ptr; return 0; } return 1; } } if (XmlNameMatchesAscii(enc, name, "encoding")) { int c = toAscii(enc, val, end); if (!('a' <= c && c <= 'z') && !('A' <= c && c <= 'Z')) { *badPtr = val; return 0; } if (encodingName) *encodingName = val; if (encoding) *encoding = encodingFinder(enc, val, ptr - enc->minBytesPerChar); if (!parsePseudoAttribute(enc, ptr, end, &name, &val, &ptr)) { *badPtr = ptr; return 0; } if (!name) return 1; } if (!XmlNameMatchesAscii(enc, name, "standalone") || isGeneralTextEntity) { *badPtr = name; return 0; } if (XmlNameMatchesAscii(enc, val, "yes")) { if (standalone) *standalone = 1; } else if (XmlNameMatchesAscii(enc, val, "no")) { if (standalone) *standalone = 0; } else { *badPtr = val; return 0; } while (isSpace(toAscii(enc, ptr, end))) ptr += enc->minBytesPerChar; if (ptr != end) { *badPtr = ptr; return 0; } return 1; } static int checkCharRefNumber(int result) { switch (result >> 8) { case 0xD8: case 0xD9: case 0xDA: case 0xDB: case 0xDC: case 0xDD: case 0xDE: case 0xDF: return -1; case 0: if (latin1_encoding.type[result] == BT_NONXML) return -1; break; case 0xFF: if (result == 0xFFFE || result == 0xFFFF) return -1; break; } return result; } int XmlUtf8Encode(int c, char *buf) { enum { /* minN is minimum legal resulting value for N byte sequence */ min2 = 0x80, min3 = 0x800, min4 = 0x10000 }; if (c < 0) return 0; if (c < min2) { buf[0] = (c | UTF8_cval1); return 1; } if (c < min3) { buf[0] = ((c >> 6) | UTF8_cval2); buf[1] = ((c & 0x3f) | 0x80); return 2; } if (c < min4) { buf[0] = ((c >> 12) | UTF8_cval3); buf[1] = (((c >> 6) & 0x3f) | 0x80); buf[2] = ((c & 0x3f) | 0x80); return 3; } if (c < 0x110000) { buf[0] = ((c >> 18) | UTF8_cval4); buf[1] = (((c >> 12) & 0x3f) | 0x80); buf[2] = (((c >> 6) & 0x3f) | 0x80); buf[3] = ((c & 0x3f) | 0x80); return 4; } return 0; } int XmlUtf16Encode(int charNum, unsigned short *buf) { if (charNum < 0) return 0; if (charNum < 0x10000) { buf[0] = charNum; return 1; } if (charNum < 0x110000) { charNum -= 0x10000; buf[0] = (charNum >> 10) + 0xD800; buf[1] = (charNum & 0x3FF) + 0xDC00; return 2; } return 0; } struct unknown_encoding { struct normal_encoding normal; int (*convert)(void *userData, const char *p); void *userData; unsigned short utf16[256]; char utf8[256][4]; }; int XmlSizeOfUnknownEncoding() { return sizeof(struct unknown_encoding); } static int unknown_isName(const ENCODING *enc, const char *p) { int c = ((const struct unknown_encoding *)enc) ->convert(((const struct unknown_encoding *)enc)->userData, p); if (c & ~0xFFFF) return 0; return UCS2_GET_NAMING(namePages, c >> 8, c & 0xFF); } static int unknown_isNmstrt(const ENCODING *enc, const char *p) { int c = ((const struct unknown_encoding *)enc) ->convert(((const struct unknown_encoding *)enc)->userData, p); if (c & ~0xFFFF) return 0; return UCS2_GET_NAMING(nmstrtPages, c >> 8, c & 0xFF); } static int unknown_isInvalid(const ENCODING *enc, const char *p) { int c = ((const struct unknown_encoding *)enc) ->convert(((const struct unknown_encoding *)enc)->userData, p); return (c & ~0xFFFF) || checkCharRefNumber(c) < 0; } static void unknown_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim, char **toP, const char *toLim) { char buf[XML_UTF8_ENCODE_MAX]; for (;;) { const char *utf8; int n; if (*fromP == fromLim) break; utf8 = ((const struct unknown_encoding *)enc)->utf8[(unsigned char)**fromP]; n = *utf8++; if (n == 0) { int c = ((const struct unknown_encoding *)enc) ->convert(((const struct unknown_encoding *)enc)->userData, *fromP); n = XmlUtf8Encode(c, buf); if (n > toLim - *toP) break; utf8 = buf; *fromP += ((const struct normal_encoding *)enc)->type[(unsigned char)**fromP] - (BT_LEAD2 - 2); } else { if (n > toLim - *toP) break; (*fromP)++; } do { *(*toP)++ = *utf8++; } while (--n != 0); } } static void unknown_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim, unsigned short **toP, const unsigned short *toLim) { while (*fromP != fromLim && *toP != toLim) { unsigned short c = ((const struct unknown_encoding *)enc)->utf16[(unsigned char)**fromP]; if (c == 0) { c = (unsigned short)((const struct unknown_encoding *)enc) ->convert(((const struct unknown_encoding *)enc)->userData, *fromP); *fromP += ((const struct normal_encoding *)enc)->type[(unsigned char)**fromP] - (BT_LEAD2 - 2); } else (*fromP)++; *(*toP)++ = c; } } ENCODING * XmlInitUnknownEncoding(void *mem, int *table, int (*convert)(void *userData, const char *p), void *userData) { int i; struct unknown_encoding *e = mem; for (i = 0; i < sizeof(struct normal_encoding); i++) ((char *)mem)[i] = ((char *)&latin1_encoding)[i]; for (i = 0; i < 128; i++) if (latin1_encoding.type[i] != BT_OTHER && latin1_encoding.type[i] != BT_NONXML && table[i] != i) return 0; for (i = 0; i < 256; i++) { int c = table[i]; if (c == -1) { e->normal.type[i] = BT_MALFORM; /* This shouldn't really get used. */ e->utf16[i] = 0xFFFF; e->utf8[i][0] = 1; e->utf8[i][1] = 0; } else if (c < 0) { if (c < -4) return 0; e->normal.type[i] = BT_LEAD2 - (c + 2); e->utf8[i][0] = 0; e->utf16[i] = 0; } else if (c < 0x80) { if (latin1_encoding.type[c] != BT_OTHER && latin1_encoding.type[c] != BT_NONXML && c != i) return 0; e->normal.type[i] = latin1_encoding.type[c]; e->utf8[i][0] = 1; e->utf8[i][1] = (char)c; e->utf16[i] = c == 0 ? 0xFFFF : c; } else if (checkCharRefNumber(c) < 0) { e->normal.type[i] = BT_NONXML; /* This shouldn't really get used. */ e->utf16[i] = 0xFFFF; e->utf8[i][0] = 1; e->utf8[i][1] = 0; } else { if (c > 0xFFFF) return 0; if (UCS2_GET_NAMING(nmstrtPages, c >> 8, c & 0xff)) e->normal.type[i] = BT_NMSTRT; else if (UCS2_GET_NAMING(namePages, c >> 8, c & 0xff)) e->normal.type[i] = BT_NAME; else e->normal.type[i] = BT_OTHER; e->utf8[i][0] = (char)XmlUtf8Encode(c, e->utf8[i] + 1); e->utf16[i] = c; } } e->userData = userData; e->convert = convert; if (convert) { e->normal.isName2 = unknown_isName; e->normal.isName3 = unknown_isName; e->normal.isName4 = unknown_isName; e->normal.isNmstrt2 = unknown_isNmstrt; e->normal.isNmstrt3 = unknown_isNmstrt; e->normal.isNmstrt4 = unknown_isNmstrt; e->normal.isInvalid2 = unknown_isInvalid; e->normal.isInvalid3 = unknown_isInvalid; e->normal.isInvalid4 = unknown_isInvalid; } e->normal.enc.utf8Convert = unknown_toUtf8; e->normal.enc.utf16Convert = unknown_toUtf16; return &(e->normal.enc); } /* If this enumeration is changed, getEncodingIndex and encodings must also be changed. */ enum { UNKNOWN_ENC = -1, ISO_8859_1_ENC = 0, US_ASCII_ENC, UTF_8_ENC, UTF_16_ENC, UTF_16BE_ENC, UTF_16LE_ENC, /* must match encodingNames up to here */ NO_ENC }; static int getEncodingIndex(const char *name) { static const char *encodingNames[] = { "ISO-8859-1", "US-ASCII", "UTF-8", "UTF-16", "UTF-16BE" "UTF-16LE", }; int i; if (name == 0) return NO_ENC; for (i = 0; i < sizeof(encodingNames)/sizeof(encodingNames[0]); i++) if (streqci(name, encodingNames[i])) return i; return UNKNOWN_ENC; } /* For binary compatibility, we store the index of the encoding specified at initialization in the isUtf16 member. */ #define INIT_ENC_INDEX(enc) ((enc)->initEnc.isUtf16) /* This is what detects the encoding. encodingTable maps from encoding indices to encodings; INIT_ENC_INDEX(enc) is the index of the external (protocol) specified encoding; state is XML_CONTENT_STATE if we're parsing an external text entity, and XML_PROLOG_STATE otherwise. */ static int initScan(const ENCODING **encodingTable, const INIT_ENCODING *enc, int state, const char *ptr, const char *end, const char **nextTokPtr) { const ENCODING **encPtr; if (ptr == end) return XML_TOK_NONE; encPtr = enc->encPtr; if (ptr + 1 == end) { /* only a single byte available for auto-detection */ /* a well-formed document entity must have more than one byte */ if (state != XML_CONTENT_STATE) return XML_TOK_PARTIAL; /* so we're parsing an external text entity... */ /* if UTF-16 was externally specified, then we need at least 2 bytes */ switch (INIT_ENC_INDEX(enc)) { case UTF_16_ENC: case UTF_16LE_ENC: case UTF_16BE_ENC: return XML_TOK_PARTIAL; } switch ((unsigned char)*ptr) { case 0xFE: case 0xFF: case 0xEF: /* possibly first byte of UTF-8 BOM */ if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE) break; /* fall through */ case 0x00: case 0x3C: return XML_TOK_PARTIAL; } } else { switch (((unsigned char)ptr[0] << 8) | (unsigned char)ptr[1]) { case 0xFEFF: if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE) break; *nextTokPtr = ptr + 2; *encPtr = encodingTable[UTF_16BE_ENC]; return XML_TOK_BOM; /* 00 3C is handled in the default case */ case 0x3C00: if ((INIT_ENC_INDEX(enc) == UTF_16BE_ENC || INIT_ENC_INDEX(enc) == UTF_16_ENC) && state == XML_CONTENT_STATE) break; *encPtr = encodingTable[UTF_16LE_ENC]; return XmlTok(*encPtr, state, ptr, end, nextTokPtr); case 0xFFFE: if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE) break; *nextTokPtr = ptr + 2; *encPtr = encodingTable[UTF_16LE_ENC]; return XML_TOK_BOM; case 0xEFBB: /* Maybe a UTF-8 BOM (EF BB BF) */ /* If there's an explicitly specified (external) encoding of ISO-8859-1 or some flavour of UTF-16 and this is an external text entity, don't look for the BOM, because it might be a legal data. */ if (state == XML_CONTENT_STATE) { int e = INIT_ENC_INDEX(enc); if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC || e == UTF_16LE_ENC || e == UTF_16_ENC) break; } if (ptr + 2 == end) return XML_TOK_PARTIAL; if ((unsigned char)ptr[2] == 0xBF) { *encPtr = encodingTable[UTF_8_ENC]; return XML_TOK_BOM; } break; default: if (ptr[0] == '\0') { /* 0 isn't a legal data character. Furthermore a document entity can only start with ASCII characters. So the only way this can fail to be big-endian UTF-16 if it it's an external parsed general entity that's labelled as UTF-16LE. */ if (state == XML_CONTENT_STATE && INIT_ENC_INDEX(enc) == UTF_16LE_ENC) break; *encPtr = encodingTable[UTF_16BE_ENC]; return XmlTok(*encPtr, state, ptr, end, nextTokPtr); } else if (ptr[1] == '\0') { /* We could recover here in the case: - parsing an external entity - second byte is 0 - no externally specified encoding - no encoding declaration by assuming UTF-16LE. But we don't, because this would mean when presented just with a single byte, we couldn't reliably determine whether we needed further bytes. */ if (state == XML_CONTENT_STATE) break; *encPtr = encodingTable[UTF_16LE_ENC]; return XmlTok(*encPtr, state, ptr, end, nextTokPtr); } break; } } *encPtr = encodingTable[INIT_ENC_INDEX(enc)]; return XmlTok(*encPtr, state, ptr, end, nextTokPtr); } #define NS(x) x #define ns(x) x #include "xmltok_ns.c" #undef NS #undef ns #ifdef XML_NS #define NS(x) x ## NS #define ns(x) x ## _ns #include "xmltok_ns.c" #undef NS #undef ns ENCODING * XmlInitUnknownEncodingNS(void *mem, int *table, int (*convert)(void *userData, const char *p), void *userData) { ENCODING *enc = XmlInitUnknownEncoding(mem, table, convert, userData); if (enc) ((struct normal_encoding *)enc)->type[':'] = BT_COLON; return enc; } #endif /* XML_NS */ �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmldef.h�����������������������������0000664�0000764�0000764�00000003757�10435500330�024711� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include <string.h> #ifdef XML_WINLIB #define WIN32_LEAN_AND_MEAN #define STRICT #include <windows.h> #define malloc(x) HeapAlloc(GetProcessHeap(), 0, (x)) #define calloc(x, y) HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, (x)*(y)) #define free(x) HeapFree(GetProcessHeap(), 0, (x)) #define realloc(x, y) HeapReAlloc(GetProcessHeap(), 0, x, y) #define abort() /* as nothing */ #else /* not XML_WINLIB */ #include <stdlib.h> #endif /* not XML_WINLIB */ /* This file can be used for any definitions needed in particular environments. */ #ifdef MOZILLA #include "nspr.h" #define malloc(x) PR_Malloc(x) #define realloc(x, y) PR_Realloc((x), (y)) #define calloc(x, y) PR_Calloc((x),(y)) #define free(x) PR_Free(x) #define int int32 #endif /* MOZILLA */ �����������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmltok.h�����������������������������0000664�0000764�0000764�00000025040�10435500330�024735� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #ifndef XmlTok_INCLUDED #define XmlTok_INCLUDED 1 #ifdef __cplusplus extern "C" { #endif #ifndef XMLTOKAPI #define XMLTOKAPI /* as nothing */ #endif /* The following token may be returned by XmlContentTok */ #define XML_TOK_TRAILING_RSQB -5 /* ] or ]] at the end of the scan; might be start of illegal ]]> sequence */ /* The following tokens may be returned by both XmlPrologTok and XmlContentTok */ #define XML_TOK_NONE -4 /* The string to be scanned is empty */ #define XML_TOK_TRAILING_CR -3 /* A CR at the end of the scan; might be part of CRLF sequence */ #define XML_TOK_PARTIAL_CHAR -2 /* only part of a multibyte sequence */ #define XML_TOK_PARTIAL -1 /* only part of a token */ #define XML_TOK_INVALID 0 /* The following tokens are returned by XmlContentTok; some are also returned by XmlAttributeValueTok, XmlEntityTok, XmlCdataSectionTok */ #define XML_TOK_START_TAG_WITH_ATTS 1 #define XML_TOK_START_TAG_NO_ATTS 2 #define XML_TOK_EMPTY_ELEMENT_WITH_ATTS 3 /* empty element tag <e/> */ #define XML_TOK_EMPTY_ELEMENT_NO_ATTS 4 #define XML_TOK_END_TAG 5 #define XML_TOK_DATA_CHARS 6 #define XML_TOK_DATA_NEWLINE 7 #define XML_TOK_CDATA_SECT_OPEN 8 #define XML_TOK_ENTITY_REF 9 #define XML_TOK_CHAR_REF 10 /* numeric character reference */ /* The following tokens may be returned by both XmlPrologTok and XmlContentTok */ #define XML_TOK_PI 11 /* processing instruction */ #define XML_TOK_XML_DECL 12 /* XML decl or text decl */ #define XML_TOK_COMMENT 13 #define XML_TOK_BOM 14 /* Byte order mark */ /* The following tokens are returned only by XmlPrologTok */ #define XML_TOK_PROLOG_S 15 #define XML_TOK_DECL_OPEN 16 /* <!foo */ #define XML_TOK_DECL_CLOSE 17 /* > */ #define XML_TOK_NAME 18 #define XML_TOK_NMTOKEN 19 #define XML_TOK_POUND_NAME 20 /* #name */ #define XML_TOK_OR 21 /* | */ #define XML_TOK_PERCENT 22 #define XML_TOK_OPEN_PAREN 23 #define XML_TOK_CLOSE_PAREN 24 #define XML_TOK_OPEN_BRACKET 25 #define XML_TOK_CLOSE_BRACKET 26 #define XML_TOK_LITERAL 27 #define XML_TOK_PARAM_ENTITY_REF 28 #define XML_TOK_INSTANCE_START 29 /* The following occur only in element type declarations */ #define XML_TOK_NAME_QUESTION 30 /* name? */ #define XML_TOK_NAME_ASTERISK 31 /* name* */ #define XML_TOK_NAME_PLUS 32 /* name+ */ #define XML_TOK_COND_SECT_OPEN 33 /* <![ */ #define XML_TOK_COND_SECT_CLOSE 34 /* ]]> */ #define XML_TOK_CLOSE_PAREN_QUESTION 35 /* )? */ #define XML_TOK_CLOSE_PAREN_ASTERISK 36 /* )* */ #define XML_TOK_CLOSE_PAREN_PLUS 37 /* )+ */ #define XML_TOK_COMMA 38 /* The following token is returned only by XmlAttributeValueTok */ #define XML_TOK_ATTRIBUTE_VALUE_S 39 /* The following token is returned only by XmlCdataSectionTok */ #define XML_TOK_CDATA_SECT_CLOSE 40 /* With namespace processing this is returned by XmlPrologTok for a name with a colon. */ #define XML_TOK_PREFIXED_NAME 41 #define XML_N_STATES 3 #define XML_PROLOG_STATE 0 #define XML_CONTENT_STATE 1 #define XML_CDATA_SECTION_STATE 2 #define XML_N_LITERAL_TYPES 2 #define XML_ATTRIBUTE_VALUE_LITERAL 0 #define XML_ENTITY_VALUE_LITERAL 1 /* The size of the buffer passed to XmlUtf8Encode must be at least this. */ #define XML_UTF8_ENCODE_MAX 4 /* The size of the buffer passed to XmlUtf16Encode must be at least this. */ #define XML_UTF16_ENCODE_MAX 2 typedef struct position { /* first line and first column are 0 not 1 */ unsigned long lineNumber; unsigned long columnNumber; } POSITION; typedef struct { const char *name; const char *valuePtr; const char *valueEnd; char normalized; } ATTRIBUTE; struct encoding; typedef struct encoding ENCODING; struct encoding { int (*scanners[XML_N_STATES])(const ENCODING *, const char *, const char *, const char **); int (*literalScanners[XML_N_LITERAL_TYPES])(const ENCODING *, const char *, const char *, const char **); int (*sameName)(const ENCODING *, const char *, const char *); int (*nameMatchesAscii)(const ENCODING *, const char *, const char *); int (*nameLength)(const ENCODING *, const char *); const char *(*skipS)(const ENCODING *, const char *); int (*getAtts)(const ENCODING *enc, const char *ptr, int attsMax, ATTRIBUTE *atts); int (*charRefNumber)(const ENCODING *enc, const char *ptr); int (*predefinedEntityName)(const ENCODING *, const char *, const char *); void (*updatePosition)(const ENCODING *, const char *ptr, const char *end, POSITION *); int (*isPublicId)(const ENCODING *enc, const char *ptr, const char *end, const char **badPtr); void (*utf8Convert)(const ENCODING *enc, const char **fromP, const char *fromLim, char **toP, const char *toLim); void (*utf16Convert)(const ENCODING *enc, const char **fromP, const char *fromLim, unsigned short **toP, const unsigned short *toLim); int minBytesPerChar; char isUtf8; int isUtf16; }; /* Scan the string starting at ptr until the end of the next complete token, but do not scan past eptr. Return an integer giving the type of token. Return XML_TOK_NONE when ptr == eptr; nextTokPtr will not be set. Return XML_TOK_PARTIAL when the string does not contain a complete token; nextTokPtr will not be set. Return XML_TOK_INVALID when the string does not start a valid token; nextTokPtr will be set to point to the character which made the token invalid. Otherwise the string starts with a valid token; nextTokPtr will be set to point to the character following the end of that token. Each data character counts as a single token, but adjacent data characters may be returned together. Similarly for characters in the prolog outside literals, comments and processing instructions. */ #define XmlTok(enc, state, ptr, end, nextTokPtr) \ (((enc)->scanners[state])(enc, ptr, end, nextTokPtr)) #define XmlPrologTok(enc, ptr, end, nextTokPtr) \ XmlTok(enc, XML_PROLOG_STATE, ptr, end, nextTokPtr) #define XmlContentTok(enc, ptr, end, nextTokPtr) \ XmlTok(enc, XML_CONTENT_STATE, ptr, end, nextTokPtr) #define XmlCdataSectionTok(enc, ptr, end, nextTokPtr) \ XmlTok(enc, XML_CDATA_SECTION_STATE, ptr, end, nextTokPtr) /* This is used for performing a 2nd-level tokenization on the content of a literal that has already been returned by XmlTok. */ #define XmlLiteralTok(enc, literalType, ptr, end, nextTokPtr) \ (((enc)->literalScanners[literalType])(enc, ptr, end, nextTokPtr)) #define XmlAttributeValueTok(enc, ptr, end, nextTokPtr) \ XmlLiteralTok(enc, XML_ATTRIBUTE_VALUE_LITERAL, ptr, end, nextTokPtr) #define XmlEntityValueTok(enc, ptr, end, nextTokPtr) \ XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr) #define XmlSameName(enc, ptr1, ptr2) (((enc)->sameName)(enc, ptr1, ptr2)) #define XmlNameMatchesAscii(enc, ptr1, ptr2) \ (((enc)->nameMatchesAscii)(enc, ptr1, ptr2)) #define XmlNameLength(enc, ptr) \ (((enc)->nameLength)(enc, ptr)) #define XmlSkipS(enc, ptr) \ (((enc)->skipS)(enc, ptr)) #define XmlGetAttributes(enc, ptr, attsMax, atts) \ (((enc)->getAtts)(enc, ptr, attsMax, atts)) #define XmlCharRefNumber(enc, ptr) \ (((enc)->charRefNumber)(enc, ptr)) #define XmlPredefinedEntityName(enc, ptr, end) \ (((enc)->predefinedEntityName)(enc, ptr, end)) #define XmlUpdatePosition(enc, ptr, end, pos) \ (((enc)->updatePosition)(enc, ptr, end, pos)) #define XmlIsPublicId(enc, ptr, end, badPtr) \ (((enc)->isPublicId)(enc, ptr, end, badPtr)) #define XmlUtf8Convert(enc, fromP, fromLim, toP, toLim) \ (((enc)->utf8Convert)(enc, fromP, fromLim, toP, toLim)) #define XmlUtf16Convert(enc, fromP, fromLim, toP, toLim) \ (((enc)->utf16Convert)(enc, fromP, fromLim, toP, toLim)) typedef struct { ENCODING initEnc; const ENCODING **encPtr; } INIT_ENCODING; int XMLTOKAPI XmlParseXmlDecl(int isGeneralTextEntity, const ENCODING *enc, const char *ptr, const char *end, const char **badPtr, const char **versionPtr, const char **encodingNamePtr, const ENCODING **namedEncodingPtr, int *standalonePtr); int XMLTOKAPI XmlInitEncoding(INIT_ENCODING *, const ENCODING **, const char *name); const ENCODING XMLTOKAPI *XmlGetUtf8InternalEncoding(void); const ENCODING XMLTOKAPI *XmlGetUtf16InternalEncoding(void); int XMLTOKAPI XmlUtf8Encode(int charNumber, char *buf); int XMLTOKAPI XmlUtf16Encode(int charNumber, unsigned short *buf); int XMLTOKAPI XmlSizeOfUnknownEncoding(void); ENCODING XMLTOKAPI * XmlInitUnknownEncoding(void *mem, int *table, int (*conv)(void *userData, const char *p), void *userData); int XMLTOKAPI XmlParseXmlDeclNS(int isGeneralTextEntity, const ENCODING *enc, const char *ptr, const char *end, const char **badPtr, const char **versionPtr, const char **encodingNamePtr, const ENCODING **namedEncodingPtr, int *standalonePtr); int XMLTOKAPI XmlInitEncodingNS(INIT_ENCODING *, const ENCODING **, const char *name); const ENCODING XMLTOKAPI *XmlGetUtf8InternalEncodingNS(void); const ENCODING XMLTOKAPI *XmlGetUtf16InternalEncodingNS(void); ENCODING XMLTOKAPI * XmlInitUnknownEncodingNS(void *mem, int *table, int (*conv)(void *userData, const char *p), void *userData); #ifdef __cplusplus } #endif #endif /* not XmlTok_INCLUDED */ ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/nametab.h����������������������������0000664�0000764�0000764�00000015612�10435500330�025032� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������static const unsigned namingBitmap[] = { 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x00000000, 0x04000000, 0x87FFFFFE, 0x07FFFFFE, 0x00000000, 0x00000000, 0xFF7FFFFF, 0xFF7FFFFF, 0xFFFFFFFF, 0x7FF3FFFF, 0xFFFFFDFE, 0x7FFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFE00F, 0xFC31FFFF, 0x00FFFFFF, 0x00000000, 0xFFFF0000, 0xFFFFFFFF, 0xFFFFFFFF, 0xF80001FF, 0x00000003, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFD740, 0xFFFFFFFB, 0x547F7FFF, 0x000FFFFD, 0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF, 0xFFFF0003, 0xFFFFFFFF, 0xFFFF199F, 0x033FCFFF, 0x00000000, 0xFFFE0000, 0x027FFFFF, 0xFFFFFFFE, 0x0000007F, 0x00000000, 0xFFFF0000, 0x000707FF, 0x00000000, 0x07FFFFFE, 0x000007FE, 0xFFFE0000, 0xFFFFFFFF, 0x7CFFFFFF, 0x002F7FFF, 0x00000060, 0xFFFFFFE0, 0x23FFFFFF, 0xFF000000, 0x00000003, 0xFFF99FE0, 0x03C5FDFF, 0xB0000000, 0x00030003, 0xFFF987E0, 0x036DFDFF, 0x5E000000, 0x001C0000, 0xFFFBAFE0, 0x23EDFDFF, 0x00000000, 0x00000001, 0xFFF99FE0, 0x23CDFDFF, 0xB0000000, 0x00000003, 0xD63DC7E0, 0x03BFC718, 0x00000000, 0x00000000, 0xFFFDDFE0, 0x03EFFDFF, 0x00000000, 0x00000003, 0xFFFDDFE0, 0x03EFFDFF, 0x40000000, 0x00000003, 0xFFFDDFE0, 0x03FFFDFF, 0x00000000, 0x00000003, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFE, 0x000D7FFF, 0x0000003F, 0x00000000, 0xFEF02596, 0x200D6CAE, 0x0000001F, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFEFF, 0x000003FF, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFF003F, 0x007FFFFF, 0x0007DAED, 0x50000000, 0x82315001, 0x002C62AB, 0x40000000, 0xF580C900, 0x00000007, 0x02010800, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0FFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x03FFFFFF, 0x3F3FFFFF, 0xFFFFFFFF, 0xAAFF3F3F, 0x3FFFFFFF, 0xFFFFFFFF, 0x5FDFFFFF, 0x0FCF1FDC, 0x1FDC1FFF, 0x00000000, 0x00004C40, 0x00000000, 0x00000000, 0x00000007, 0x00000000, 0x00000000, 0x00000000, 0x00000080, 0x000003FE, 0xFFFFFFFE, 0xFFFFFFFF, 0x001FFFFF, 0xFFFFFFFE, 0xFFFFFFFF, 0x07FFFFFF, 0xFFFFFFE0, 0x00001FFF, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0000003F, 0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0000000F, 0x00000000, 0x00000000, 0x00000000, 0x07FF6000, 0x87FFFFFE, 0x07FFFFFE, 0x00000000, 0x00800000, 0xFF7FFFFF, 0xFF7FFFFF, 0x00FFFFFF, 0x00000000, 0xFFFF0000, 0xFFFFFFFF, 0xFFFFFFFF, 0xF80001FF, 0x00030003, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0x0000003F, 0x00000003, 0xFFFFD7C0, 0xFFFFFFFB, 0x547F7FFF, 0x000FFFFD, 0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF, 0xFFFF007B, 0xFFFFFFFF, 0xFFFF199F, 0x033FCFFF, 0x00000000, 0xFFFE0000, 0x027FFFFF, 0xFFFFFFFE, 0xFFFE007F, 0xBBFFFFFB, 0xFFFF0016, 0x000707FF, 0x00000000, 0x07FFFFFE, 0x0007FFFF, 0xFFFF03FF, 0xFFFFFFFF, 0x7CFFFFFF, 0xFFEF7FFF, 0x03FF3DFF, 0xFFFFFFEE, 0xF3FFFFFF, 0xFF1E3FFF, 0x0000FFCF, 0xFFF99FEE, 0xD3C5FDFF, 0xB080399F, 0x0003FFCF, 0xFFF987E4, 0xD36DFDFF, 0x5E003987, 0x001FFFC0, 0xFFFBAFEE, 0xF3EDFDFF, 0x00003BBF, 0x0000FFC1, 0xFFF99FEE, 0xF3CDFDFF, 0xB0C0398F, 0x0000FFC3, 0xD63DC7EC, 0xC3BFC718, 0x00803DC7, 0x0000FF80, 0xFFFDDFEE, 0xC3EFFDFF, 0x00603DDF, 0x0000FFC3, 0xFFFDDFEC, 0xC3EFFDFF, 0x40603DDF, 0x0000FFC3, 0xFFFDDFEC, 0xC3FFFDFF, 0x00803DCF, 0x0000FFC3, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFE, 0x07FF7FFF, 0x03FF7FFF, 0x00000000, 0xFEF02596, 0x3BFF6CAE, 0x03FF3F5F, 0x00000000, 0x03000000, 0xC2A003FF, 0xFFFFFEFF, 0xFFFE03FF, 0xFEBF0FDF, 0x02FE3FFF, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x1FFF0000, 0x00000002, 0x000000A0, 0x003EFFFE, 0xFFFFFFFE, 0xFFFFFFFF, 0x661FFFFF, 0xFFFFFFFE, 0xFFFFFFFF, 0x77FFFFFF, }; static const unsigned char nmstrtPages[] = { 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x00, 0x00, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13, 0x00, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x15, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x17, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, }; static const unsigned char namePages[] = { 0x19, 0x03, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x00, 0x00, 0x1F, 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13, 0x26, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x27, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x17, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, }; ����������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/iasciitab.h��������������������������0000664�0000764�0000764�00000005652�10435500330�025356� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ /* Like asciitab.h, except that 0xD has code BT_S rather than BT_CR */ /* 0x00 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML, /* 0x0C */ BT_NONXML, BT_S, BT_NONXML, BT_NONXML, /* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM, /* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS, /* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS, /* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL, /* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT, /* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT, /* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI, /* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST, /* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX, /* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT, /* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB, /* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT, /* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX, /* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT, /* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER, /* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER, ��������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/latin1tab.h��������������������������0000664�0000764�0000764�00000005631�10435500330�025302� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ /* 0x80 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x84 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x88 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x8C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x90 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x94 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x98 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0x9C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xA0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xA4 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xA8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER, /* 0xAC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xB0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xB4 */ BT_OTHER, BT_NMSTRT, BT_OTHER, BT_NAME, /* 0xB8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER, /* 0xBC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER, /* 0xC0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xC4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xC8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xCC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xD0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xD4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER, /* 0xD8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xDC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xE0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xE4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xE8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xEC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xF0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xF4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER, /* 0xF8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0xFC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, �������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmltok_ns.c��������������������������0000775�0000764�0000764�00000004555�10435500330�025443� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������const ENCODING *NS(XmlGetUtf8InternalEncoding)() { return &ns(internal_utf8_encoding).enc; } const ENCODING *NS(XmlGetUtf16InternalEncoding)() { #if XML_BYTE_ORDER == 12 return &ns(internal_little2_encoding).enc; #elif XML_BYTE_ORDER == 21 return &ns(internal_big2_encoding).enc; #else const short n = 1; return *(const char *)&n ? &ns(internal_little2_encoding).enc : &ns(internal_big2_encoding).enc; #endif } static const ENCODING *NS(encodings)[] = { &ns(latin1_encoding).enc, &ns(ascii_encoding).enc, &ns(utf8_encoding).enc, &ns(big2_encoding).enc, &ns(big2_encoding).enc, &ns(little2_encoding).enc, &ns(utf8_encoding).enc /* NO_ENC */ }; static int NS(initScanProlog)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_PROLOG_STATE, ptr, end, nextTokPtr); } static int NS(initScanContent)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_CONTENT_STATE, ptr, end, nextTokPtr); } int NS(XmlInitEncoding)(INIT_ENCODING *p, const ENCODING **encPtr, const char *name) { int i = getEncodingIndex(name); if (i == UNKNOWN_ENC) return 0; INIT_ENC_INDEX(p) = (char)i; p->initEnc.scanners[XML_PROLOG_STATE] = NS(initScanProlog); p->initEnc.scanners[XML_CONTENT_STATE] = NS(initScanContent); p->initEnc.updatePosition = initUpdatePosition; p->encPtr = encPtr; *encPtr = &(p->initEnc); return 1; } static const ENCODING *NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end) { #define ENCODING_MAX 128 char buf[ENCODING_MAX]; char *p = buf; int i; XmlUtf8Convert(enc, &ptr, end, &p, p + ENCODING_MAX - 1); if (ptr != end) return 0; *p = 0; if (streqci(buf, "UTF-16") && enc->minBytesPerChar == 2) return enc; i = getEncodingIndex(buf); if (i == UNKNOWN_ENC) return 0; return NS(encodings)[i]; } int NS(XmlParseXmlDecl)(int isGeneralTextEntity, const ENCODING *enc, const char *ptr, const char *end, const char **badPtr, const char **versionPtr, const char **encodingName, const ENCODING **encoding, int *standalone) { return doParseXmlDecl(NS(findEncoding), isGeneralTextEntity, enc, ptr, end, badPtr, versionPtr, encodingName, encoding, standalone); } ���������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/asciitab.h���������������������������0000664�0000764�0000764�00000005544�10435500330�025205� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ /* 0x00 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML, /* 0x0C */ BT_NONXML, BT_CR, BT_NONXML, BT_NONXML, /* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM, /* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS, /* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS, /* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL, /* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT, /* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT, /* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI, /* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST, /* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX, /* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT, /* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB, /* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT, /* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX, /* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT, /* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, /* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER, /* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER, ������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmlrole.h����������������������������0000664�0000764�0000764�00000006335�10435500330�025107� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #ifndef XmlRole_INCLUDED #define XmlRole_INCLUDED 1 #include "xmltok.h" #ifdef __cplusplus extern "C" { #endif enum { XML_ROLE_ERROR = -1, XML_ROLE_NONE = 0, XML_ROLE_XML_DECL, XML_ROLE_INSTANCE_START, XML_ROLE_DOCTYPE_NAME, XML_ROLE_DOCTYPE_SYSTEM_ID, XML_ROLE_DOCTYPE_PUBLIC_ID, XML_ROLE_DOCTYPE_CLOSE, XML_ROLE_GENERAL_ENTITY_NAME, XML_ROLE_PARAM_ENTITY_NAME, XML_ROLE_ENTITY_VALUE, XML_ROLE_ENTITY_SYSTEM_ID, XML_ROLE_ENTITY_PUBLIC_ID, XML_ROLE_ENTITY_NOTATION_NAME, XML_ROLE_NOTATION_NAME, XML_ROLE_NOTATION_SYSTEM_ID, XML_ROLE_NOTATION_NO_SYSTEM_ID, XML_ROLE_NOTATION_PUBLIC_ID, XML_ROLE_ATTRIBUTE_NAME, XML_ROLE_ATTRIBUTE_TYPE_CDATA, XML_ROLE_ATTRIBUTE_TYPE_ID, XML_ROLE_ATTRIBUTE_TYPE_IDREF, XML_ROLE_ATTRIBUTE_TYPE_IDREFS, XML_ROLE_ATTRIBUTE_TYPE_ENTITY, XML_ROLE_ATTRIBUTE_TYPE_ENTITIES, XML_ROLE_ATTRIBUTE_TYPE_NMTOKEN, XML_ROLE_ATTRIBUTE_TYPE_NMTOKENS, XML_ROLE_ATTRIBUTE_ENUM_VALUE, XML_ROLE_ATTRIBUTE_NOTATION_VALUE, XML_ROLE_ATTLIST_ELEMENT_NAME, XML_ROLE_IMPLIED_ATTRIBUTE_VALUE, XML_ROLE_REQUIRED_ATTRIBUTE_VALUE, XML_ROLE_DEFAULT_ATTRIBUTE_VALUE, XML_ROLE_FIXED_ATTRIBUTE_VALUE, XML_ROLE_ELEMENT_NAME, XML_ROLE_CONTENT_ANY, XML_ROLE_CONTENT_EMPTY, XML_ROLE_CONTENT_PCDATA, XML_ROLE_GROUP_OPEN, XML_ROLE_GROUP_CLOSE, XML_ROLE_GROUP_CLOSE_REP, XML_ROLE_GROUP_CLOSE_OPT, XML_ROLE_GROUP_CLOSE_PLUS, XML_ROLE_GROUP_CHOICE, XML_ROLE_GROUP_SEQUENCE, XML_ROLE_CONTENT_ELEMENT, XML_ROLE_CONTENT_ELEMENT_REP, XML_ROLE_CONTENT_ELEMENT_OPT, XML_ROLE_CONTENT_ELEMENT_PLUS, XML_ROLE_PARAM_ENTITY_REF }; typedef struct prolog_state { int (*handler)(struct prolog_state *state, int tok, const char *ptr, const char *end, const ENCODING *enc); unsigned level; } PROLOG_STATE; void XMLTOKAPI XmlPrologStateInit(PROLOG_STATE *); #define XmlTokenRole(state, tok, ptr, end, enc) \ (((state)->handler)(state, tok, ptr, end, enc)) #ifdef __cplusplus } #endif #endif /* not XmlRole_INCLUDED */ ���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmlrole.c����������������������������0000664�0000764�0000764�00000057112�10435500330�025101� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #include "xmldef.h" #include "xmlrole.h" /* Doesn't check: that ,| are not mixed in a model group content of literals */ #ifndef MIN_BYTES_PER_CHAR #define MIN_BYTES_PER_CHAR(enc) ((enc)->minBytesPerChar) #endif typedef int PROLOG_HANDLER(struct prolog_state *state, int tok, const char *ptr, const char *end, const ENCODING *enc); static PROLOG_HANDLER prolog0, prolog1, prolog2, doctype0, doctype1, doctype2, doctype3, doctype4, doctype5, internalSubset, entity0, entity1, entity2, entity3, entity4, entity5, entity6, entity7, entity8, entity9, notation0, notation1, notation2, notation3, notation4, attlist0, attlist1, attlist2, attlist3, attlist4, attlist5, attlist6, attlist7, attlist8, attlist9, element0, element1, element2, element3, element4, element5, element6, element7, declClose, error; static int syntaxError(PROLOG_STATE *); static int prolog0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: state->handler = prolog1; return XML_ROLE_NONE; case XML_TOK_XML_DECL: state->handler = prolog1; return XML_ROLE_XML_DECL; case XML_TOK_PI: state->handler = prolog1; return XML_ROLE_NONE; case XML_TOK_COMMENT: state->handler = prolog1; case XML_TOK_BOM: return XML_ROLE_NONE; case XML_TOK_DECL_OPEN: if (!XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "DOCTYPE")) break; state->handler = doctype0; return XML_ROLE_NONE; case XML_TOK_INSTANCE_START: state->handler = error; return XML_ROLE_INSTANCE_START; } return syntaxError(state); } static int prolog1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_PI: case XML_TOK_COMMENT: case XML_TOK_BOM: return XML_ROLE_NONE; case XML_TOK_DECL_OPEN: if (!XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "DOCTYPE")) break; state->handler = doctype0; return XML_ROLE_NONE; case XML_TOK_INSTANCE_START: state->handler = error; return XML_ROLE_INSTANCE_START; } return syntaxError(state); } static int prolog2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_PI: case XML_TOK_COMMENT: return XML_ROLE_NONE; case XML_TOK_INSTANCE_START: state->handler = error; return XML_ROLE_INSTANCE_START; } return syntaxError(state); } static int doctype0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = doctype1; return XML_ROLE_DOCTYPE_NAME; } return syntaxError(state); } static int doctype1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_OPEN_BRACKET: state->handler = internalSubset; return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = prolog2; return XML_ROLE_DOCTYPE_CLOSE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "SYSTEM")) { state->handler = doctype3; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr, "PUBLIC")) { state->handler = doctype2; return XML_ROLE_NONE; } break; } return syntaxError(state); } static int doctype2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = doctype3; return XML_ROLE_DOCTYPE_PUBLIC_ID; } return syntaxError(state); } static int doctype3(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = doctype4; return XML_ROLE_DOCTYPE_SYSTEM_ID; } return syntaxError(state); } static int doctype4(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_OPEN_BRACKET: state->handler = internalSubset; return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = prolog2; return XML_ROLE_DOCTYPE_CLOSE; } return syntaxError(state); } static int doctype5(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = prolog2; return XML_ROLE_DOCTYPE_CLOSE; } return syntaxError(state); } static int internalSubset(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_DECL_OPEN: if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "ENTITY")) { state->handler = entity0; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "ATTLIST")) { state->handler = attlist0; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "ELEMENT")) { state->handler = element0; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), "NOTATION")) { state->handler = notation0; return XML_ROLE_NONE; } break; case XML_TOK_PI: case XML_TOK_COMMENT: return XML_ROLE_NONE; case XML_TOK_PARAM_ENTITY_REF: return XML_ROLE_PARAM_ENTITY_REF; case XML_TOK_CLOSE_BRACKET: state->handler = doctype5; return XML_ROLE_NONE; } return syntaxError(state); } static int entity0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_PERCENT: state->handler = entity1; return XML_ROLE_NONE; case XML_TOK_NAME: state->handler = entity2; return XML_ROLE_GENERAL_ENTITY_NAME; } return syntaxError(state); } static int entity1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: state->handler = entity7; return XML_ROLE_PARAM_ENTITY_NAME; } return syntaxError(state); } static int entity2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "SYSTEM")) { state->handler = entity4; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr, "PUBLIC")) { state->handler = entity3; return XML_ROLE_NONE; } break; case XML_TOK_LITERAL: state->handler = declClose; return XML_ROLE_ENTITY_VALUE; } return syntaxError(state); } static int entity3(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = entity4; return XML_ROLE_ENTITY_PUBLIC_ID; } return syntaxError(state); } static int entity4(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = entity5; return XML_ROLE_ENTITY_SYSTEM_ID; } return syntaxError(state); } static int entity5(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = internalSubset; return XML_ROLE_NONE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "NDATA")) { state->handler = entity6; return XML_ROLE_NONE; } break; } return syntaxError(state); } static int entity6(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: state->handler = declClose; return XML_ROLE_ENTITY_NOTATION_NAME; } return syntaxError(state); } static int entity7(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "SYSTEM")) { state->handler = entity9; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr, "PUBLIC")) { state->handler = entity8; return XML_ROLE_NONE; } break; case XML_TOK_LITERAL: state->handler = declClose; return XML_ROLE_ENTITY_VALUE; } return syntaxError(state); } static int entity8(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = entity9; return XML_ROLE_ENTITY_PUBLIC_ID; } return syntaxError(state); } static int entity9(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = declClose; return XML_ROLE_ENTITY_SYSTEM_ID; } return syntaxError(state); } static int notation0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: state->handler = notation1; return XML_ROLE_NOTATION_NAME; } return syntaxError(state); } static int notation1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "SYSTEM")) { state->handler = notation3; return XML_ROLE_NONE; } if (XmlNameMatchesAscii(enc, ptr, "PUBLIC")) { state->handler = notation2; return XML_ROLE_NONE; } break; } return syntaxError(state); } static int notation2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = notation4; return XML_ROLE_NOTATION_PUBLIC_ID; } return syntaxError(state); } static int notation3(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = declClose; return XML_ROLE_NOTATION_SYSTEM_ID; } return syntaxError(state); } static int notation4(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = declClose; return XML_ROLE_NOTATION_SYSTEM_ID; case XML_TOK_DECL_CLOSE: state->handler = internalSubset; return XML_ROLE_NOTATION_NO_SYSTEM_ID; } return syntaxError(state); } static int attlist0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = attlist1; return XML_ROLE_ATTLIST_ELEMENT_NAME; } return syntaxError(state); } static int attlist1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = internalSubset; return XML_ROLE_NONE; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = attlist2; return XML_ROLE_ATTRIBUTE_NAME; } return syntaxError(state); } static int attlist2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: { static const char *types[] = { "CDATA", "ID", "IDREF", "IDREFS", "ENTITY", "ENTITIES", "NMTOKEN", "NMTOKENS", }; int i; for (i = 0; i < (int)(sizeof(types)/sizeof(types[0])); i++) if (XmlNameMatchesAscii(enc, ptr, types[i])) { state->handler = attlist8; return XML_ROLE_ATTRIBUTE_TYPE_CDATA + i; } } if (XmlNameMatchesAscii(enc, ptr, "NOTATION")) { state->handler = attlist5; return XML_ROLE_NONE; } break; case XML_TOK_OPEN_PAREN: state->handler = attlist3; return XML_ROLE_NONE; } return syntaxError(state); } static int attlist3(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NMTOKEN: case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = attlist4; return XML_ROLE_ATTRIBUTE_ENUM_VALUE; } return syntaxError(state); } static int attlist4(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_CLOSE_PAREN: state->handler = attlist8; return XML_ROLE_NONE; case XML_TOK_OR: state->handler = attlist3; return XML_ROLE_NONE; } return syntaxError(state); } static int attlist5(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_OPEN_PAREN: state->handler = attlist6; return XML_ROLE_NONE; } return syntaxError(state); } static int attlist6(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: state->handler = attlist7; return XML_ROLE_ATTRIBUTE_NOTATION_VALUE; } return syntaxError(state); } static int attlist7(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_CLOSE_PAREN: state->handler = attlist8; return XML_ROLE_NONE; case XML_TOK_OR: state->handler = attlist6; return XML_ROLE_NONE; } return syntaxError(state); } /* default value */ static int attlist8(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_POUND_NAME: if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), "IMPLIED")) { state->handler = attlist1; return XML_ROLE_IMPLIED_ATTRIBUTE_VALUE; } if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), "REQUIRED")) { state->handler = attlist1; return XML_ROLE_REQUIRED_ATTRIBUTE_VALUE; } if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), "FIXED")) { state->handler = attlist9; return XML_ROLE_NONE; } break; case XML_TOK_LITERAL: state->handler = attlist1; return XML_ROLE_DEFAULT_ATTRIBUTE_VALUE; } return syntaxError(state); } static int attlist9(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_LITERAL: state->handler = attlist1; return XML_ROLE_FIXED_ATTRIBUTE_VALUE; } return syntaxError(state); } static int element0(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = element1; return XML_ROLE_ELEMENT_NAME; } return syntaxError(state); } static int element1(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: if (XmlNameMatchesAscii(enc, ptr, "EMPTY")) { state->handler = declClose; return XML_ROLE_CONTENT_EMPTY; } if (XmlNameMatchesAscii(enc, ptr, "ANY")) { state->handler = declClose; return XML_ROLE_CONTENT_ANY; } break; case XML_TOK_OPEN_PAREN: state->handler = element2; state->level = 1; return XML_ROLE_GROUP_OPEN; } return syntaxError(state); } static int element2(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_POUND_NAME: if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), "PCDATA")) { state->handler = element3; return XML_ROLE_CONTENT_PCDATA; } break; case XML_TOK_OPEN_PAREN: state->level = 2; state->handler = element6; return XML_ROLE_GROUP_OPEN; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT; case XML_TOK_NAME_QUESTION: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_OPT; case XML_TOK_NAME_ASTERISK: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_REP; case XML_TOK_NAME_PLUS: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_PLUS; } return syntaxError(state); } static int element3(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_CLOSE_PAREN: case XML_TOK_CLOSE_PAREN_ASTERISK: state->handler = declClose; return XML_ROLE_GROUP_CLOSE_REP; case XML_TOK_OR: state->handler = element4; return XML_ROLE_NONE; } return syntaxError(state); } static int element4(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = element5; return XML_ROLE_CONTENT_ELEMENT; } return syntaxError(state); } static int element5(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_CLOSE_PAREN_ASTERISK: state->handler = declClose; return XML_ROLE_GROUP_CLOSE_REP; case XML_TOK_OR: state->handler = element4; return XML_ROLE_NONE; } return syntaxError(state); } static int element6(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_OPEN_PAREN: state->level += 1; return XML_ROLE_GROUP_OPEN; case XML_TOK_NAME: case XML_TOK_PREFIXED_NAME: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT; case XML_TOK_NAME_QUESTION: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_OPT; case XML_TOK_NAME_ASTERISK: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_REP; case XML_TOK_NAME_PLUS: state->handler = element7; return XML_ROLE_CONTENT_ELEMENT_PLUS; } return syntaxError(state); } static int element7(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_CLOSE_PAREN: state->level -= 1; if (state->level == 0) state->handler = declClose; return XML_ROLE_GROUP_CLOSE; case XML_TOK_CLOSE_PAREN_ASTERISK: state->level -= 1; if (state->level == 0) state->handler = declClose; return XML_ROLE_GROUP_CLOSE_REP; case XML_TOK_CLOSE_PAREN_QUESTION: state->level -= 1; if (state->level == 0) state->handler = declClose; return XML_ROLE_GROUP_CLOSE_OPT; case XML_TOK_CLOSE_PAREN_PLUS: state->level -= 1; if (state->level == 0) state->handler = declClose; return XML_ROLE_GROUP_CLOSE_PLUS; case XML_TOK_COMMA: state->handler = element6; return XML_ROLE_GROUP_SEQUENCE; case XML_TOK_OR: state->handler = element6; return XML_ROLE_GROUP_CHOICE; } return syntaxError(state); } static int declClose(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_PROLOG_S: return XML_ROLE_NONE; case XML_TOK_DECL_CLOSE: state->handler = internalSubset; return XML_ROLE_NONE; } return syntaxError(state); } #if 0 static int ignore(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { switch (tok) { case XML_TOK_DECL_CLOSE: state->handler = internalSubset; return 0; default: return XML_ROLE_NONE; } return syntaxError(state); } #endif static int error(PROLOG_STATE *state, int tok, const char *ptr, const char *end, const ENCODING *enc) { return XML_ROLE_NONE; } static int syntaxError(PROLOG_STATE *state) { state->handler = error; return XML_ROLE_ERROR; } void XmlPrologStateInit(PROLOG_STATE *state) { state->handler = prolog0; } ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/utf8tab.h����������������������������0000664�0000764�0000764�00000005547�10435500330�025006� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ /* 0x80 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x84 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x88 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x8C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x90 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x94 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x98 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0x9C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xA0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xA4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xA8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xAC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xB0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xB4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xB8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xBC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL, /* 0xC0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xC4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xC8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xCC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xD0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xD4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xD8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xDC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2, /* 0xE0 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3, /* 0xE4 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3, /* 0xE8 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3, /* 0xEC */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3, /* 0xF0 */ BT_LEAD4, BT_LEAD4, BT_LEAD4, BT_LEAD4, /* 0xF4 */ BT_LEAD4, BT_NONXML, BT_NONXML, BT_NONXML, /* 0xF8 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML, /* 0xFC */ BT_NONXML, BT_NONXML, BT_MALFORM, BT_MALFORM, ���������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/dllmain.c����������������������������0000664�0000764�0000764�00000002642�10435500330�025035� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #define STRICT 1 #define WIN32_LEAN_AND_MEAN 1 #include <windows.h> BOOL WINAPI DllMain(HANDLE hInst, ULONG ul_reason_for_call, LPVOID lpReserved) { return TRUE; } ����������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/xmltok/xmltok_impl.c������������������������0000664�0000764�0000764�00000117076�10435500330�025764� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������/* The contents of this file are subject to the Mozilla Public License Version 1.1 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/ Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. The Original Code is expat. The Initial Developer of the Original Code is James Clark. Portions created by James Clark are Copyright (C) 1998, 1999 James Clark. All Rights Reserved. Contributor(s): Alternatively, the contents of this file may be used under the terms of the GNU General Public License (the "GPL"), in which case the provisions of the GPL are applicable instead of those above. If you wish to allow use of your version of this file only under the terms of the GPL and not to allow others to use your version of this file under the MPL, indicate your decision by deleting the provisions above and replace them with the notice and other provisions required by the GPL. If you do not delete the provisions above, a recipient may use your version of this file under either the MPL or the GPL. */ #ifndef IS_INVALID_CHAR #define IS_INVALID_CHAR(enc, ptr, n) (0) #endif #define INVALID_LEAD_CASE(n, ptr, nextTokPtr) \ case BT_LEAD ## n: \ if (end - ptr < n) \ return XML_TOK_PARTIAL_CHAR; \ if (IS_INVALID_CHAR(enc, ptr, n)) { \ *(nextTokPtr) = (ptr); \ return XML_TOK_INVALID; \ } \ ptr += n; \ break; #define INVALID_CASES(ptr, nextTokPtr) \ INVALID_LEAD_CASE(2, ptr, nextTokPtr) \ INVALID_LEAD_CASE(3, ptr, nextTokPtr) \ INVALID_LEAD_CASE(4, ptr, nextTokPtr) \ case BT_NONXML: \ case BT_MALFORM: \ case BT_TRAIL: \ *(nextTokPtr) = (ptr); \ return XML_TOK_INVALID; #define CHECK_NAME_CASE(n, enc, ptr, end, nextTokPtr) \ case BT_LEAD ## n: \ if (end - ptr < n) \ return XML_TOK_PARTIAL_CHAR; \ if (!IS_NAME_CHAR(enc, ptr, n)) { \ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ ptr += n; \ break; #define CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) \ case BT_NONASCII: \ if (!IS_NAME_CHAR_MINBPC(enc, ptr)) { \ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ case BT_NMSTRT: \ case BT_HEX: \ case BT_DIGIT: \ case BT_NAME: \ case BT_MINUS: \ ptr += MINBPC(enc); \ break; \ CHECK_NAME_CASE(2, enc, ptr, end, nextTokPtr) \ CHECK_NAME_CASE(3, enc, ptr, end, nextTokPtr) \ CHECK_NAME_CASE(4, enc, ptr, end, nextTokPtr) #define CHECK_NMSTRT_CASE(n, enc, ptr, end, nextTokPtr) \ case BT_LEAD ## n: \ if (end - ptr < n) \ return XML_TOK_PARTIAL_CHAR; \ if (!IS_NMSTRT_CHAR(enc, ptr, n)) { \ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ ptr += n; \ break; #define CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) \ case BT_NONASCII: \ if (!IS_NMSTRT_CHAR_MINBPC(enc, ptr)) { \ *nextTokPtr = ptr; \ return XML_TOK_INVALID; \ } \ case BT_NMSTRT: \ case BT_HEX: \ ptr += MINBPC(enc); \ break; \ CHECK_NMSTRT_CASE(2, enc, ptr, end, nextTokPtr) \ CHECK_NMSTRT_CASE(3, enc, ptr, end, nextTokPtr) \ CHECK_NMSTRT_CASE(4, enc, ptr, end, nextTokPtr) #ifndef PREFIX #define PREFIX(ident) ident #endif /* ptr points to character following "<!-" */ static int PREFIX(scanComment)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr != end) { if (!CHAR_MATCHES(enc, ptr, '-')) { *nextTokPtr = ptr; return XML_TOK_INVALID; } ptr += MINBPC(enc); while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { INVALID_CASES(ptr, nextTokPtr) case BT_MINUS: if ((ptr += MINBPC(enc)) == end) return XML_TOK_PARTIAL; if (CHAR_MATCHES(enc, ptr, '-')) { if ((ptr += MINBPC(enc)) == end) return XML_TOK_PARTIAL; if (!CHAR_MATCHES(enc, ptr, '>')) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_COMMENT; } break; default: ptr += MINBPC(enc); break; } } } return XML_TOK_PARTIAL; } /* ptr points to character following "<!" */ static int PREFIX(scanDecl)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { case BT_MINUS: return PREFIX(scanComment)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_LSQB: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_COND_SECT_OPEN; case BT_NMSTRT: case BT_HEX: ptr += MINBPC(enc); break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { case BT_PERCNT: if (ptr + MINBPC(enc) == end) return XML_TOK_PARTIAL; /* don't allow <!ENTITY% foo "whatever"> */ switch (BYTE_TYPE(enc, ptr + MINBPC(enc))) { case BT_S: case BT_CR: case BT_LF: case BT_PERCNT: *nextTokPtr = ptr; return XML_TOK_INVALID; } /* fall through */ case BT_S: case BT_CR: case BT_LF: *nextTokPtr = ptr; return XML_TOK_DECL_OPEN; case BT_NMSTRT: case BT_HEX: ptr += MINBPC(enc); break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(checkPiTarget)(const ENCODING *enc, const char *ptr, const char *end, int *tokPtr) { int upper = 0; *tokPtr = XML_TOK_PI; if (end - ptr != MINBPC(enc)*3) return 1; switch (BYTE_TO_ASCII(enc, ptr)) { case 'x': break; case 'X': upper = 1; break; default: return 1; } ptr += MINBPC(enc); switch (BYTE_TO_ASCII(enc, ptr)) { case 'm': break; case 'M': upper = 1; break; default: return 1; } ptr += MINBPC(enc); switch (BYTE_TO_ASCII(enc, ptr)) { case 'l': break; case 'L': upper = 1; break; default: return 1; } if (upper) return 0; *tokPtr = XML_TOK_XML_DECL; return 1; } /* ptr points to character following "<?" */ static int PREFIX(scanPi)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { int tok; const char *target = ptr; if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_S: case BT_CR: case BT_LF: if (!PREFIX(checkPiTarget)(enc, target, ptr, &tok)) { *nextTokPtr = ptr; return XML_TOK_INVALID; } ptr += MINBPC(enc); while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { INVALID_CASES(ptr, nextTokPtr) case BT_QUEST: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (CHAR_MATCHES(enc, ptr, '>')) { *nextTokPtr = ptr + MINBPC(enc); return tok; } break; default: ptr += MINBPC(enc); break; } } return XML_TOK_PARTIAL; case BT_QUEST: if (!PREFIX(checkPiTarget)(enc, target, ptr, &tok)) { *nextTokPtr = ptr; return XML_TOK_INVALID; } ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (CHAR_MATCHES(enc, ptr, '>')) { *nextTokPtr = ptr + MINBPC(enc); return tok; } /* fall through */ default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(scanCdataSection)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { int i; /* CDATA[ */ if (end - ptr < 6 * MINBPC(enc)) return XML_TOK_PARTIAL; for (i = 0; i < 6; i++, ptr += MINBPC(enc)) { if (!CHAR_MATCHES(enc, ptr, "CDATA["[i])) { *nextTokPtr = ptr; return XML_TOK_INVALID; } } *nextTokPtr = ptr; return XML_TOK_CDATA_SECT_OPEN; } static int PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_NONE; if (MINBPC(enc) > 1) { size_t n = end - ptr; if (n & (MINBPC(enc) - 1)) { n &= ~(MINBPC(enc) - 1); if (n == 0) return XML_TOK_PARTIAL; end = ptr + n; } } switch (BYTE_TYPE(enc, ptr)) { case BT_RSQB: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (!CHAR_MATCHES(enc, ptr, ']')) break; ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (!CHAR_MATCHES(enc, ptr, '>')) { ptr -= MINBPC(enc); break; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CDATA_SECT_CLOSE; case BT_CR: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (BYTE_TYPE(enc, ptr) == BT_LF) ptr += MINBPC(enc); *nextTokPtr = ptr; return XML_TOK_DATA_NEWLINE; case BT_LF: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_DATA_NEWLINE; INVALID_CASES(ptr, nextTokPtr) default: ptr += MINBPC(enc); break; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: \ if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \ *nextTokPtr = ptr; \ return XML_TOK_DATA_CHARS; \ } \ ptr += n; \ break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_NONXML: case BT_MALFORM: case BT_TRAIL: case BT_CR: case BT_LF: case BT_RSQB: *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; default: ptr += MINBPC(enc); break; } } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; } /* ptr points to character following "</" */ static int PREFIX(scanEndTag)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_S: case BT_CR: case BT_LF: for (ptr += MINBPC(enc); ptr != end; ptr += MINBPC(enc)) { switch (BYTE_TYPE(enc, ptr)) { case BT_S: case BT_CR: case BT_LF: break; case BT_GT: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_END_TAG; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; #ifdef XML_NS case BT_COLON: /* no need to check qname syntax here, since end-tag must match exactly */ ptr += MINBPC(enc); break; #endif case BT_GT: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_END_TAG; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } /* ptr points to character following "&#X" */ static int PREFIX(scanHexCharRef)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { case BT_DIGIT: case BT_HEX: break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } for (ptr += MINBPC(enc); ptr != end; ptr += MINBPC(enc)) { switch (BYTE_TYPE(enc, ptr)) { case BT_DIGIT: case BT_HEX: break; case BT_SEMI: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CHAR_REF; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } } return XML_TOK_PARTIAL; } /* ptr points to character following "&#" */ static int PREFIX(scanCharRef)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr != end) { if (CHAR_MATCHES(enc, ptr, 'x')) return PREFIX(scanHexCharRef)(enc, ptr + MINBPC(enc), end, nextTokPtr); switch (BYTE_TYPE(enc, ptr)) { case BT_DIGIT: break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } for (ptr += MINBPC(enc); ptr != end; ptr += MINBPC(enc)) { switch (BYTE_TYPE(enc, ptr)) { case BT_DIGIT: break; case BT_SEMI: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CHAR_REF; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } } return XML_TOK_PARTIAL; } /* ptr points to character following "&" */ static int PREFIX(scanRef)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) case BT_NUM: return PREFIX(scanCharRef)(enc, ptr + MINBPC(enc), end, nextTokPtr); default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_SEMI: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_ENTITY_REF; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } /* ptr points to character following first character of attribute name */ static int PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { #ifdef XML_NS int hadColon = 0; #endif while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) #ifdef XML_NS case BT_COLON: if (hadColon) { *nextTokPtr = ptr; return XML_TOK_INVALID; } hadColon = 1; ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) default: *nextTokPtr = ptr; return XML_TOK_INVALID; } break; #endif case BT_S: case BT_CR: case BT_LF: for (;;) { int t; ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; t = BYTE_TYPE(enc, ptr); if (t == BT_EQUALS) break; switch (t) { case BT_S: case BT_LF: case BT_CR: break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } /* fall through */ case BT_EQUALS: { int open = 0; #ifdef XML_NS hadColon = 0; #endif for (;;) { ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; open = BYTE_TYPE(enc, ptr); if (open == BT_QUOT || open == BT_APOS) break; switch (open) { case BT_S: case BT_LF: case BT_CR: break; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } ptr += MINBPC(enc); /* in attribute value */ for (;;) { int t; if (ptr == end) return XML_TOK_PARTIAL; t = BYTE_TYPE(enc, ptr); if (t == open) break; switch (t) { INVALID_CASES(ptr, nextTokPtr) case BT_AMP: { int tok = PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, &ptr); if (tok <= 0) { if (tok == XML_TOK_INVALID) *nextTokPtr = ptr; return tok; } break; } case BT_LT: *nextTokPtr = ptr; return XML_TOK_INVALID; default: ptr += MINBPC(enc); break; } } ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { case BT_S: case BT_CR: case BT_LF: break; case BT_SOL: goto sol; case BT_GT: goto gt; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } /* ptr points to closing quote */ for (;;) { ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) case BT_S: case BT_CR: case BT_LF: continue; case BT_GT: gt: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_START_TAG_WITH_ATTS; case BT_SOL: sol: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (!CHAR_MATCHES(enc, ptr, '>')) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_EMPTY_ELEMENT_WITH_ATTS; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } break; } break; } default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } /* ptr points to character following "<" */ static int PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { #ifdef XML_NS int hadColon; #endif if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) case BT_EXCL: if ((ptr += MINBPC(enc)) == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { case BT_MINUS: return PREFIX(scanComment)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_LSQB: return PREFIX(scanCdataSection)(enc, ptr + MINBPC(enc), end, nextTokPtr); } *nextTokPtr = ptr; return XML_TOK_INVALID; case BT_QUEST: return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_SOL: return PREFIX(scanEndTag)(enc, ptr + MINBPC(enc), end, nextTokPtr); default: *nextTokPtr = ptr; return XML_TOK_INVALID; } #ifdef XML_NS hadColon = 0; #endif /* we have a start-tag */ while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) #ifdef XML_NS case BT_COLON: if (hadColon) { *nextTokPtr = ptr; return XML_TOK_INVALID; } hadColon = 1; ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) default: *nextTokPtr = ptr; return XML_TOK_INVALID; } break; #endif case BT_S: case BT_CR: case BT_LF: { ptr += MINBPC(enc); while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) case BT_GT: goto gt; case BT_SOL: goto sol; case BT_S: case BT_CR: case BT_LF: ptr += MINBPC(enc); continue; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } return PREFIX(scanAtts)(enc, ptr, end, nextTokPtr); } return XML_TOK_PARTIAL; } case BT_GT: gt: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_START_TAG_NO_ATTS; case BT_SOL: sol: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (!CHAR_MATCHES(enc, ptr, '>')) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_EMPTY_ELEMENT_NO_ATTS; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_NONE; if (MINBPC(enc) > 1) { size_t n = end - ptr; if (n & (MINBPC(enc) - 1)) { n &= ~(MINBPC(enc) - 1); if (n == 0) return XML_TOK_PARTIAL; end = ptr + n; } } switch (BYTE_TYPE(enc, ptr)) { case BT_LT: return PREFIX(scanLt)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_AMP: return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_CR: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_TRAILING_CR; if (BYTE_TYPE(enc, ptr) == BT_LF) ptr += MINBPC(enc); *nextTokPtr = ptr; return XML_TOK_DATA_NEWLINE; case BT_LF: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_DATA_NEWLINE; case BT_RSQB: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_TRAILING_RSQB; if (!CHAR_MATCHES(enc, ptr, ']')) break; ptr += MINBPC(enc); if (ptr == end) return XML_TOK_TRAILING_RSQB; if (!CHAR_MATCHES(enc, ptr, '>')) { ptr -= MINBPC(enc); break; } *nextTokPtr = ptr; return XML_TOK_INVALID; INVALID_CASES(ptr, nextTokPtr) default: ptr += MINBPC(enc); break; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: \ if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \ *nextTokPtr = ptr; \ return XML_TOK_DATA_CHARS; \ } \ ptr += n; \ break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_RSQB: if (ptr + MINBPC(enc) != end) { if (!CHAR_MATCHES(enc, ptr + MINBPC(enc), ']')) { ptr += MINBPC(enc); break; } if (ptr + 2*MINBPC(enc) != end) { if (!CHAR_MATCHES(enc, ptr + 2*MINBPC(enc), '>')) { ptr += MINBPC(enc); break; } *nextTokPtr = ptr + 2*MINBPC(enc); return XML_TOK_INVALID; } } /* fall through */ case BT_AMP: case BT_LT: case BT_NONXML: case BT_MALFORM: case BT_TRAIL: case BT_CR: case BT_LF: *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; default: ptr += MINBPC(enc); break; } } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; } /* ptr points to character following "%" */ static int PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) case BT_S: case BT_LF: case BT_CR: case BT_PERCNT: *nextTokPtr = ptr; return XML_TOK_PERCENT; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_SEMI: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_PARAM_ENTITY_REF; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(scanPoundName)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_CR: case BT_LF: case BT_S: case BT_RPAR: case BT_GT: case BT_PERCNT: case BT_VERBAR: *nextTokPtr = ptr; return XML_TOK_POUND_NAME; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(scanLit)(int open, const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { while (ptr != end) { int t = BYTE_TYPE(enc, ptr); switch (t) { INVALID_CASES(ptr, nextTokPtr) case BT_QUOT: case BT_APOS: ptr += MINBPC(enc); if (t != open) break; if (ptr == end) return XML_TOK_PARTIAL; *nextTokPtr = ptr; switch (BYTE_TYPE(enc, ptr)) { case BT_S: case BT_CR: case BT_LF: case BT_GT: case BT_PERCNT: case BT_LSQB: return XML_TOK_LITERAL; default: return XML_TOK_INVALID; } default: ptr += MINBPC(enc); break; } } return XML_TOK_PARTIAL; } static int PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { int tok; if (ptr == end) return XML_TOK_NONE; if (MINBPC(enc) > 1) { size_t n = end - ptr; if (n & (MINBPC(enc) - 1)) { n &= ~(MINBPC(enc) - 1); if (n == 0) return XML_TOK_PARTIAL; end = ptr + n; } } switch (BYTE_TYPE(enc, ptr)) { case BT_QUOT: return PREFIX(scanLit)(BT_QUOT, enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_APOS: return PREFIX(scanLit)(BT_APOS, enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_LT: { ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { case BT_EXCL: return PREFIX(scanDecl)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_QUEST: return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_NMSTRT: case BT_HEX: case BT_NONASCII: case BT_LEAD2: case BT_LEAD3: case BT_LEAD4: *nextTokPtr = ptr - MINBPC(enc); return XML_TOK_INSTANCE_START; } *nextTokPtr = ptr; return XML_TOK_INVALID; } case BT_CR: if (ptr + MINBPC(enc) == end) return XML_TOK_TRAILING_CR; /* fall through */ case BT_S: case BT_LF: for (;;) { ptr += MINBPC(enc); if (ptr == end) break; switch (BYTE_TYPE(enc, ptr)) { case BT_S: case BT_LF: break; case BT_CR: /* don't split CR/LF pair */ if (ptr + MINBPC(enc) != end) break; /* fall through */ default: *nextTokPtr = ptr; return XML_TOK_PROLOG_S; } } *nextTokPtr = ptr; return XML_TOK_PROLOG_S; case BT_PERCNT: return PREFIX(scanPercent)(enc, ptr + MINBPC(enc), end, nextTokPtr); case BT_COMMA: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_COMMA; case BT_LSQB: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_OPEN_BRACKET; case BT_RSQB: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; if (CHAR_MATCHES(enc, ptr, ']')) { if (ptr + MINBPC(enc) == end) return XML_TOK_PARTIAL; if (CHAR_MATCHES(enc, ptr + MINBPC(enc), '>')) { *nextTokPtr = ptr + 2*MINBPC(enc); return XML_TOK_COND_SECT_CLOSE; } } *nextTokPtr = ptr; return XML_TOK_CLOSE_BRACKET; case BT_LPAR: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_OPEN_PAREN; case BT_RPAR: ptr += MINBPC(enc); if (ptr == end) return XML_TOK_PARTIAL; switch (BYTE_TYPE(enc, ptr)) { case BT_AST: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CLOSE_PAREN_ASTERISK; case BT_QUEST: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CLOSE_PAREN_QUESTION; case BT_PLUS: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_CLOSE_PAREN_PLUS; case BT_CR: case BT_LF: case BT_S: case BT_GT: case BT_COMMA: case BT_VERBAR: case BT_RPAR: *nextTokPtr = ptr; return XML_TOK_CLOSE_PAREN; } *nextTokPtr = ptr; return XML_TOK_INVALID; case BT_VERBAR: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_OR; case BT_GT: *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_DECL_CLOSE; case BT_NUM: return PREFIX(scanPoundName)(enc, ptr + MINBPC(enc), end, nextTokPtr); #define LEAD_CASE(n) \ case BT_LEAD ## n: \ if (end - ptr < n) \ return XML_TOK_PARTIAL_CHAR; \ if (IS_NMSTRT_CHAR(enc, ptr, n)) { \ ptr += n; \ tok = XML_TOK_NAME; \ break; \ } \ if (IS_NAME_CHAR(enc, ptr, n)) { \ ptr += n; \ tok = XML_TOK_NMTOKEN; \ break; \ } \ *nextTokPtr = ptr; \ return XML_TOK_INVALID; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_NMSTRT: case BT_HEX: tok = XML_TOK_NAME; ptr += MINBPC(enc); break; case BT_DIGIT: case BT_NAME: case BT_MINUS: #ifdef XML_NS case BT_COLON: #endif tok = XML_TOK_NMTOKEN; ptr += MINBPC(enc); break; case BT_NONASCII: if (IS_NMSTRT_CHAR_MINBPC(enc, ptr)) { ptr += MINBPC(enc); tok = XML_TOK_NAME; break; } if (IS_NAME_CHAR_MINBPC(enc, ptr)) { ptr += MINBPC(enc); tok = XML_TOK_NMTOKEN; break; } /* fall through */ default: *nextTokPtr = ptr; return XML_TOK_INVALID; } while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) case BT_GT: case BT_RPAR: case BT_COMMA: case BT_VERBAR: case BT_LSQB: case BT_PERCNT: case BT_S: case BT_CR: case BT_LF: *nextTokPtr = ptr; return tok; #ifdef XML_NS case BT_COLON: ptr += MINBPC(enc); switch (tok) { case XML_TOK_NAME: if (ptr == end) return XML_TOK_PARTIAL; tok = XML_TOK_PREFIXED_NAME; switch (BYTE_TYPE(enc, ptr)) { CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) default: tok = XML_TOK_NMTOKEN; break; } break; case XML_TOK_PREFIXED_NAME: tok = XML_TOK_NMTOKEN; break; } break; #endif case BT_PLUS: if (tok == XML_TOK_NMTOKEN) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_NAME_PLUS; case BT_AST: if (tok == XML_TOK_NMTOKEN) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_NAME_ASTERISK; case BT_QUEST: if (tok == XML_TOK_NMTOKEN) { *nextTokPtr = ptr; return XML_TOK_INVALID; } *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_NAME_QUESTION; default: *nextTokPtr = ptr; return XML_TOK_INVALID; } } return XML_TOK_PARTIAL; } static int PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { const char *start; if (ptr == end) return XML_TOK_NONE; start = ptr; while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: ptr += n; break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_AMP: if (ptr == start) return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr); *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_LT: /* this is for inside entity references */ *nextTokPtr = ptr; return XML_TOK_INVALID; case BT_LF: if (ptr == start) { *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_DATA_NEWLINE; } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_CR: if (ptr == start) { ptr += MINBPC(enc); if (ptr == end) return XML_TOK_TRAILING_CR; if (BYTE_TYPE(enc, ptr) == BT_LF) ptr += MINBPC(enc); *nextTokPtr = ptr; return XML_TOK_DATA_NEWLINE; } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_S: if (ptr == start) { *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_ATTRIBUTE_VALUE_S; } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; default: ptr += MINBPC(enc); break; } } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; } static int PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr, const char *end, const char **nextTokPtr) { const char *start; if (ptr == end) return XML_TOK_NONE; start = ptr; while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: ptr += n; break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_AMP: if (ptr == start) return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr); *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_PERCNT: if (ptr == start) return PREFIX(scanPercent)(enc, ptr + MINBPC(enc), end, nextTokPtr); *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_LF: if (ptr == start) { *nextTokPtr = ptr + MINBPC(enc); return XML_TOK_DATA_NEWLINE; } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; case BT_CR: if (ptr == start) { ptr += MINBPC(enc); if (ptr == end) return XML_TOK_TRAILING_CR; if (BYTE_TYPE(enc, ptr) == BT_LF) ptr += MINBPC(enc); *nextTokPtr = ptr; return XML_TOK_DATA_NEWLINE; } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; default: ptr += MINBPC(enc); break; } } *nextTokPtr = ptr; return XML_TOK_DATA_CHARS; } static int PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end, const char **badPtr) { ptr += MINBPC(enc); end -= MINBPC(enc); for (; ptr != end; ptr += MINBPC(enc)) { switch (BYTE_TYPE(enc, ptr)) { case BT_DIGIT: case BT_HEX: case BT_MINUS: case BT_APOS: case BT_LPAR: case BT_RPAR: case BT_PLUS: case BT_COMMA: case BT_SOL: case BT_EQUALS: case BT_QUEST: case BT_CR: case BT_LF: case BT_SEMI: case BT_EXCL: case BT_AST: case BT_PERCNT: case BT_NUM: #ifdef XML_NS case BT_COLON: #endif break; case BT_S: if (CHAR_MATCHES(enc, ptr, '\t')) { *badPtr = ptr; return 0; } break; case BT_NAME: case BT_NMSTRT: if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f)) break; default: switch (BYTE_TO_ASCII(enc, ptr)) { case 0x24: /* $ */ case 0x40: /* @ */ break; default: *badPtr = ptr; return 0; } break; } } return 1; } /* This must only be called for a well-formed start-tag or empty element tag. Returns the number of attributes. Pointers to the first attsMax attributes are stored in atts. */ static int PREFIX(getAtts)(const ENCODING *enc, const char *ptr, int attsMax, ATTRIBUTE *atts) { enum { other, inName, inValue } state = inName; int nAtts = 0; int open = 0; for (ptr += MINBPC(enc);; ptr += MINBPC(enc)) { switch (BYTE_TYPE(enc, ptr)) { #define START_NAME \ if (state == other) { \ if (nAtts < attsMax) { \ atts[nAtts].name = ptr; \ atts[nAtts].normalized = 1; \ } \ state = inName; \ } #define LEAD_CASE(n) \ case BT_LEAD ## n: START_NAME ptr += (n - MINBPC(enc)); break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_NONASCII: case BT_NMSTRT: case BT_HEX: START_NAME break; #undef START_NAME case BT_QUOT: if (state != inValue) { if (nAtts < attsMax) atts[nAtts].valuePtr = ptr + MINBPC(enc); state = inValue; open = BT_QUOT; } else if (open == BT_QUOT) { state = other; if (nAtts < attsMax) atts[nAtts].valueEnd = ptr; nAtts++; } break; case BT_APOS: if (state != inValue) { if (nAtts < attsMax) atts[nAtts].valuePtr = ptr + MINBPC(enc); state = inValue; open = BT_APOS; } else if (open == BT_APOS) { state = other; if (nAtts < attsMax) atts[nAtts].valueEnd = ptr; nAtts++; } break; case BT_AMP: if (nAtts < attsMax) atts[nAtts].normalized = 0; break; case BT_S: if (state == inName) state = other; else if (state == inValue && nAtts < attsMax && atts[nAtts].normalized && (ptr == atts[nAtts].valuePtr || BYTE_TO_ASCII(enc, ptr) != ' ' || BYTE_TO_ASCII(enc, ptr + MINBPC(enc)) == ' ' || BYTE_TYPE(enc, ptr + MINBPC(enc)) == open)) atts[nAtts].normalized = 0; break; case BT_CR: case BT_LF: /* This case ensures that the first attribute name is counted Apart from that we could just change state on the quote. */ if (state == inName) state = other; else if (state == inValue && nAtts < attsMax) atts[nAtts].normalized = 0; break; case BT_GT: case BT_SOL: if (state != inValue) return nAtts; break; default: break; } } /* not reached */ } static int PREFIX(charRefNumber)(const ENCODING *enc, const char *ptr) { int result = 0; /* skip &# */ ptr += 2*MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'x')) { for (ptr += MINBPC(enc); !CHAR_MATCHES(enc, ptr, ';'); ptr += MINBPC(enc)) { int c = BYTE_TO_ASCII(enc, ptr); switch (c) { case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': result <<= 4; result |= (c - '0'); break; case 'A': case 'B': case 'C': case 'D': case 'E': case 'F': result <<= 4; result += 10 + (c - 'A'); break; case 'a': case 'b': case 'c': case 'd': case 'e': case 'f': result <<= 4; result += 10 + (c - 'a'); break; } if (result >= 0x110000) return -1; } } else { for (; !CHAR_MATCHES(enc, ptr, ';'); ptr += MINBPC(enc)) { int c = BYTE_TO_ASCII(enc, ptr); result *= 10; result += (c - '0'); if (result >= 0x110000) return -1; } } return checkCharRefNumber(result); } static int PREFIX(predefinedEntityName)(const ENCODING *enc, const char *ptr, const char *end) { switch ((end - ptr)/MINBPC(enc)) { case 2: if (CHAR_MATCHES(enc, ptr + MINBPC(enc), 't')) { switch (BYTE_TO_ASCII(enc, ptr)) { case 'l': return '<'; case 'g': return '>'; } } break; case 3: if (CHAR_MATCHES(enc, ptr, 'a')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'm')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'p')) return '&'; } } break; case 4: switch (BYTE_TO_ASCII(enc, ptr)) { case 'q': ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'u')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'o')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 't')) return '"'; } } break; case 'a': ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'p')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 'o')) { ptr += MINBPC(enc); if (CHAR_MATCHES(enc, ptr, 's')) return '\''; } } break; } } return 0; } static int PREFIX(sameName)(const ENCODING *enc, const char *ptr1, const char *ptr2) { for (;;) { switch (BYTE_TYPE(enc, ptr1)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: \ if (*ptr1++ != *ptr2++) \ return 0; LEAD_CASE(4) LEAD_CASE(3) LEAD_CASE(2) #undef LEAD_CASE /* fall through */ if (*ptr1++ != *ptr2++) return 0; break; case BT_NONASCII: case BT_NMSTRT: #ifdef XML_NS case BT_COLON: #endif case BT_HEX: case BT_DIGIT: case BT_NAME: case BT_MINUS: if (*ptr2++ != *ptr1++) return 0; if (MINBPC(enc) > 1) { if (*ptr2++ != *ptr1++) return 0; if (MINBPC(enc) > 2) { if (*ptr2++ != *ptr1++) return 0; if (MINBPC(enc) > 3) { if (*ptr2++ != *ptr1++) return 0; } } } break; default: if (MINBPC(enc) == 1 && *ptr1 == *ptr2) return 1; switch (BYTE_TYPE(enc, ptr2)) { case BT_LEAD2: case BT_LEAD3: case BT_LEAD4: case BT_NONASCII: case BT_NMSTRT: #ifdef XML_NS case BT_COLON: #endif case BT_HEX: case BT_DIGIT: case BT_NAME: case BT_MINUS: return 0; default: return 1; } } } /* not reached */ } static int PREFIX(nameMatchesAscii)(const ENCODING *enc, const char *ptr1, const char *ptr2) { for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) { if (!CHAR_MATCHES(enc, ptr1, *ptr2)) return 0; } switch (BYTE_TYPE(enc, ptr1)) { case BT_LEAD2: case BT_LEAD3: case BT_LEAD4: case BT_NONASCII: case BT_NMSTRT: #ifdef XML_NS case BT_COLON: #endif case BT_HEX: case BT_DIGIT: case BT_NAME: case BT_MINUS: return 0; default: return 1; } } static int PREFIX(nameLength)(const ENCODING *enc, const char *ptr) { const char *start = ptr; for (;;) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: ptr += n; break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_NONASCII: case BT_NMSTRT: #ifdef XML_NS case BT_COLON: #endif case BT_HEX: case BT_DIGIT: case BT_NAME: case BT_MINUS: ptr += MINBPC(enc); break; default: return ptr - start; } } } static const char *PREFIX(skipS)(const ENCODING *enc, const char *ptr) { for (;;) { switch (BYTE_TYPE(enc, ptr)) { case BT_LF: case BT_CR: case BT_S: ptr += MINBPC(enc); break; default: return ptr; } } } static void PREFIX(updatePosition)(const ENCODING *enc, const char *ptr, const char *end, POSITION *pos) { while (ptr != end) { switch (BYTE_TYPE(enc, ptr)) { #define LEAD_CASE(n) \ case BT_LEAD ## n: \ ptr += n; \ break; LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4) #undef LEAD_CASE case BT_LF: pos->columnNumber = (unsigned)-1; pos->lineNumber++; ptr += MINBPC(enc); break; case BT_CR: pos->lineNumber++; ptr += MINBPC(enc); if (ptr != end && BYTE_TYPE(enc, ptr) == BT_LF) ptr += MINBPC(enc); pos->columnNumber = (unsigned)-1; break; default: ptr += MINBPC(enc); break; } pos->columnNumber++; } } #undef DO_LEAD_CASE #undef MULTIBYTE_CASES #undef INVALID_CASES #undef CHECK_NAME_CASE #undef CHECK_NAME_CASES #undef CHECK_NMSTRT_CASE #undef CHECK_NMSTRT_CASES ������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/MPL-1_0.html��������������������������������0000664�0000764�0000764�00000047303�10435500330�023631� 0����������������������������������������������������������������������������������������������������ustar �ajung���������������������������ajung������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������<TITLE>Mozilla Public License version 1.0

MOZILLA PUBLIC LICENSE
Version 1.0

1. Definitions.

    1.1. ``Contributor'' means each entity that creates or contributes to the creation of Modifications.

    1.2. ``Contributor Version'' means the combination of the Original Code, prior Modifications used by a Contributor, and the Modifications made by that particular Contributor.

    1.3. ``Covered Code'' means the Original Code or Modifications or the combination of the Original Code and Modifications, in each case including portions thereof.

    1.4. ``Electronic Distribution Mechanism'' means a mechanism generally accepted in the software development community for the electronic transfer of data.

    1.5. ``Executable'' means Covered Code in any form other than Source Code.

    1.6. ``Initial Developer'' means the individual or entity identified as the Initial Developer in the Source Code notice required by Exhibit A.

    1.7. ``Larger Work'' means a work which combines Covered Code or portions thereof with code not governed by the terms of this License.

    1.8. ``License'' means this document.

    1.9. ``Modifications'' means any addition to or deletion from the substance or structure of either the Original Code or any previous Modifications. When Covered Code is released as a series of files, a Modification is:

      A. Any addition to or deletion from the contents of a file containing Original Code or previous Modifications.

      B. Any new file that contains any part of the Original Code or previous Modifications.

    1.10. ``Original Code'' means Source Code of computer software code which is described in the Source Code notice required by Exhibit A as Original Code, and which, at the time of its release under this License is not already Covered Code governed by this License.

    1.11. ``Source Code'' means the preferred form of the Covered Code for making modifications to it, including all modules it contains, plus any associated interface definition files, scripts used to control compilation and installation of an Executable, or a list of source code differential comparisons against either the Original Code or another well known, available Covered Code of the Contributor's choice. The Source Code can be in a compressed or archival form, provided the appropriate decompression or de-archiving software is widely available for no charge.

    1.12. ``You'' means an individual or a legal entity exercising rights under, and complying with all of the terms of, this License or a future version of this License issued under Section 6.1. For legal entities, ``You'' includes any entity which controls, is controlled by, or is under common control with You. For purposes of this definition, ``control'' means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of fifty percent (50%) or more of the outstanding shares or beneficial ownership of such entity.

2. Source Code License.
    2.1. The Initial Developer Grant.
    The Initial Developer hereby grants You a world-wide, royalty-free, non-exclusive license, subject to third party intellectual property claims:

      (a) to use, reproduce, modify, display, perform, sublicense and distribute the Original Code (or portions thereof) with or without Modifications, or as part of a Larger Work; and

      (b) under patents now or hereafter owned or controlled by Initial Developer, to make, have made, use and sell (``Utilize'') the Original Code (or portions thereof), but solely to the extent that any such patent is reasonably necessary to enable You to Utilize the Original Code (or portions thereof) and not to any greater extent that may be necessary to Utilize further Modifications or combinations.

    2.2. Contributor Grant.
    Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license, subject to third party intellectual property claims:

      (a) to use, reproduce, modify, display, perform, sublicense and distribute the Modifications created by such Contributor (or portions thereof) either on an unmodified basis, with other Modifications, as Covered Code or as part of a Larger Work; and

      (b) under patents now or hereafter owned or controlled by Contributor, to Utilize the Contributor Version (or portions thereof), but solely to the extent that any such patent is reasonably necessary to enable You to Utilize the Contributor Version (or portions thereof), and not to any greater extent that may be necessary to Utilize further Modifications or combinations.

3. Distribution Obligations.
    3.1. Application of License.
    The Modifications which You create or to which You contribute are governed by the terms of this License, including without limitation Section 2.2. The Source Code version of Covered Code may be distributed only under the terms of this License or a future version of this License released under Section 6.1, and You must include a copy of this License with every copy of the Source Code You distribute. You may not offer or impose any terms on any Source Code version that alters or restricts the applicable version of this License or the recipients' rights hereunder. However, You may include an additional document offering the additional rights described in Section 3.5.

    3.2. Availability of Source Code.
    Any Modification which You create or to which You contribute must be made available in Source Code form under the terms of this License either on the same media as an Executable version or via an accepted Electronic Distribution Mechanism to anyone to whom you made an Executable version available; and if made available via Electronic Distribution Mechanism, must remain available for at least twelve (12) months after the date it initially became available, or at least six (6) months after a subsequent version of that particular Modification has been made available to such recipients. You are responsible for ensuring that the Source Code version remains available even if the Electronic Distribution Mechanism is maintained by a third party.

    3.3. Description of Modifications.
    You must cause all Covered Code to which you contribute to contain a file documenting the changes You made to create that Covered Code and the date of any change. You must include a prominent statement that the Modification is derived, directly or indirectly, from Original Code provided by the Initial Developer and including the name of the Initial Developer in (a) the Source Code, and (b) in any notice in an Executable version or related documentation in which You describe the origin or ownership of the Covered Code.

    3.4. Intellectual Property Matters

      (a) Third Party Claims.
      If You have knowledge that a party claims an intellectual property right in particular functionality or code (or its utilization under this License), you must include a text file with the source code distribution titled ``LEGAL'' which describes the claim and the party making the claim in sufficient detail that a recipient will know whom to contact. If you obtain such knowledge after You make Your Modification available as described in Section 3.2, You shall promptly modify the LEGAL file in all copies You make available thereafter and shall take other steps (such as notifying appropriate mailing lists or newsgroups) reasonably calculated to inform those who received the Covered Code that new knowledge has been obtained.

      (b) Contributor APIs.
      If Your Modification is an application programming interface and You own or control patents which are reasonably necessary to implement that API, you must also include this information in the LEGAL file.

    3.5. Required Notices.
    You must duplicate the notice in Exhibit A in each file of the Source Code, and this License in any documentation for the Source Code, where You describe recipients' rights relating to Covered Code. If You created one or more Modification(s), You may add your name as a Contributor to the notice described in Exhibit A. If it is not possible to put such notice in a particular Source Code file due to its structure, then you must include such notice in a location (such as a relevant directory file) where a user would be likely to look for such a notice. You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Code. However, You may do so only on Your own behalf, and not on behalf of the Initial Developer or any Contributor. You must make it absolutely clear than any such warranty, support, indemnity or liability obligation is offered by You alone, and You hereby agree to indemnify the Initial Developer and every Contributor for any liability incurred by the Initial Developer or such Contributor as a result of warranty, support, indemnity or liability terms You offer.

    3.6. Distribution of Executable Versions.
    You may distribute Covered Code in Executable form only if the requirements of Section 3.1-3.5 have been met for that Covered Code, and if You include a notice stating that the Source Code version of the Covered Code is available under the terms of this License, including a description of how and where You have fulfilled the obligations of Section 3.2. The notice must be conspicuously included in any notice in an Executable version, related documentation or collateral in which You describe recipients' rights relating to the Covered Code. You may distribute the Executable version of Covered Code under a license of Your choice, which may contain terms different from this License, provided that You are in compliance with the terms of this License and that the license for the Executable version does not attempt to limit or alter the recipient's rights in the Source Code version from the rights set forth in this License. If You distribute the Executable version under a different license You must make it absolutely clear that any terms which differ from this License are offered by You alone, not by the Initial Developer or any Contributor. You hereby agree to indemnify the Initial Developer and every Contributor for any liability incurred by the Initial Developer or such Contributor as a result of any such terms You offer.

    3.7. Larger Works.
    You may create a Larger Work by combining Covered Code with other code not governed by the terms of this License and distribute the Larger Work as a single product. In such a case, You must make sure the requirements of this License are fulfilled for the Covered Code.

4. Inability to Comply Due to Statute or Regulation.

    If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Code due to statute or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be included in the LEGAL file described in Section 3.4 and must be included with all distributions of the Source Code. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it.

5. Application of this License.
    This License applies to code to which the Initial Developer has attached the notice in Exhibit A, and to related Covered Code.
6. Versions of the License.
    6.1. New Versions.
    Netscape Communications Corporation (``Netscape'') may publish revised and/or new versions of the License from time to time. Each version will be given a distinguishing version number.

    6.2. Effect of New Versions.
    Once Covered Code has been published under a particular version of the License, You may always continue to use it under the terms of that version. You may also choose to use such Covered Code under the terms of any subsequent version of the License published by Netscape. No one other than Netscape has the right to modify the terms applicable to Covered Code created under this License.

    6.3. Derivative Works.
    If you create or use a modified version of this License (which you may only do in order to apply it to code which is not already Covered Code governed by this License), you must (a) rename Your license so that the phrases ``Mozilla'', ``MOZILLAPL'', ``MOZPL'', ``Netscape'', ``NPL'' or any confusingly similar phrase do not appear anywhere in your license and (b) otherwise make it clear that your version of the license contains terms which differ from the Mozilla Public License and Netscape Public License. (Filling in the name of the Initial Developer, Original Code or Contributor in the notice described in Exhibit A shall not of themselves be deemed to be modifications of this License.)

7. DISCLAIMER OF WARRANTY.
    COVERED CODE IS PROVIDED UNDER THIS LICENSE ON AN ``AS IS'' BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT THE COVERED CODE IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE COVERED CODE IS WITH YOU. SHOULD ANY COVERED CODE PROVE DEFECTIVE IN ANY RESPECT, YOU (NOT THE INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF ANY COVERED CODE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS DISCLAIMER.
8. TERMINATION.
    This License and the rights granted hereunder will terminate automatically if You fail to comply with terms herein and fail to cure such breach within 30 days of becoming aware of the breach. All sublicenses to the Covered Code which are properly granted shall survive any termination of this License. Provisions which, by their nature, must remain in effect beyond the termination of this License shall survive.
9. LIMITATION OF LIABILITY.
    UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL THE INITIAL DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF COVERED CODE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE LIABLE TO YOU OR ANY OTHER PERSON FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES, EVEN IF SUCH PARTY SHALL HAVE BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL INJURY RESULTING FROM SUCH PARTY'S NEGLIGENCE TO THE EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THAT EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU.
10. U.S. GOVERNMENT END USERS.
    The Covered Code is a ``commercial item,'' as that term is defined in 48 C.F.R. 2.101 (Oct. 1995), consisting of ``commercial computer software'' and ``commercial computer software documentation,'' as such terms are used in 48 C.F.R. 12.212 (Sept. 1995). Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4 (June 1995), all U.S. Government End Users acquire Covered Code with only those rights set forth herein.
11. MISCELLANEOUS.
    This License represents the complete agreement concerning subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. This License shall be governed by California law provisions (except to the extent applicable law, if any, provides otherwise), excluding its conflict-of-law provisions. With respect to disputes in which at least one party is a citizen of, or an entity chartered or registered to do business in, the United States of America: (a) unless otherwise agreed in writing, all disputes relating to this License (excepting any dispute relating to intellectual property rights) shall be subject to final and binding arbitration, with the losing party paying all costs of arbitration; (b) any arbitration relating to this Agreement shall be held in Santa Clara County, California, under the auspices of JAMS/EndDispute; and (c) any litigation relating to this Agreement shall be subject to the jurisdiction of the Federal Courts of the Northern District of California, with venue lying in Santa Clara County, California, with the losing party responsible for costs, including without limitation, court costs and reasonable attorneys fees and expenses. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not apply to this License.
12. RESPONSIBILITY FOR CLAIMS.
    Except in cases where another Contributor has failed to comply with Section 3.4, You are responsible for damages arising, directly or indirectly, out of Your utilization of rights under this License, based on the number of copies of Covered Code you made available, the revenues you received from utilizing such rights, and other relevant factors. You agree to work with affected parties to distribute responsibility on an equitable basis.
EXHIBIT A.
    ``The contents of this file are subject to the Mozilla Public License Version 1.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/

    Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.

    The Original Code is ______________________________________.

    The Initial Developer of the Original Code is ________________________. Portions created by ______________________ are Copyright (C) ______ _______________________. All Rights Reserved.

    Contributor(s): ______________________________________.''

Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/expat.html0000664000076400007640000000534210435500330023742 0ustar ajungajung expat

expat - XML Parser Toolkit

Version 1.1

Copyright (c) 1998, 1999 James Clark. Expat is subject to the Mozilla Public License Version 1.1. Alternatively you may use expat under the GNU General Public License instead. Please contact me if you wish to negotiate an alternative license.

Expat is an XML 1.0 parser written in C. It aims to be fully conforming. It is currently not a validating XML processor. The current production version of expat can be downloaded from ftp://ftp.jclark.com/pub/xml/expat.zip.

The directory xmltok contains a low-level library for tokenizing XML. The interface is documented in xmltok/xmltok.h.

The directory xmlparse contains an XML parser library which is built on top of the xmltok library. The interface is documented in xmlparse/xmlparse.h. The directory sample contains a simple example program using this interface; sample/build.bat is a batch file to build the example using Visual C++.

The directory xmlwf contains the xmlwf application, which uses the xmlparse library. The arguments to xmlwf are one or more files which are each to be checked for well-formedness. An option -d dir can be specified; for each well-formed input file the corresponding canonical XML will be written to dir/f, where f is the filename (without any path) of the input file. A -x option will cause references to external general entities to be processed. A -s option will make documents that are not standalone cause an error (a document is considered standalone if either it is intrinsically standalone because it has no external subset and no references to parameter entities in the internal subset or it is declared as standalone in the XML declaration).

The bin directory contains Win32 executables. The lib directory contains Win32 import libraries.

Answers to some frequently asked questions about expat can be found in the expat FAQ.

James Clark
Zope-2.8.7-final/lib/python/Shared/DC/xml/pyexpat/expat/expat.mak0000664000076400007640000007505310435500330023554 0ustar ajungajung# Microsoft Developer Studio Generated NMAKE File, Format Version 4.20 # ** DO NOT EDIT ** # TARGTYPE "Win32 (x86) Console Application" 0x0103 # TARGTYPE "Win32 (x86) Dynamic-Link Library" 0x0102 !IF "$(CFG)" == "" CFG=xmltok - Win32 Debug !MESSAGE No configuration specified. Defaulting to xmltok - Win32 Debug. !ENDIF !IF "$(CFG)" != "xmlwf - Win32 Release" && "$(CFG)" != "xmlwf - Win32 Debug" &&\ "$(CFG)" != "gennmtab - Win32 Release" && "$(CFG)" != "gennmtab - Win32 Debug"\ && "$(CFG)" != "xmlparse - Win32 Release" && "$(CFG)" !=\ "xmlparse - Win32 Debug" && "$(CFG)" != "xmltok - Win32 Release" && "$(CFG)" !=\ "xmltok - Win32 Debug" !MESSAGE Invalid configuration "$(CFG)" specified. !MESSAGE You can specify a configuration when running NMAKE on this makefile !MESSAGE by defining the macro CFG on the command line. For example: !MESSAGE !MESSAGE NMAKE /f "expat.mak" CFG="xmltok - Win32 Debug" !MESSAGE !MESSAGE Possible choices for configuration are: !MESSAGE !MESSAGE "xmlwf - Win32 Release" (based on "Win32 (x86) Console Application") !MESSAGE "xmlwf - Win32 Debug" (based on "Win32 (x86) Console Application") !MESSAGE "gennmtab - Win32 Release" (based on\ "Win32 (x86) Console Application") !MESSAGE "gennmtab - Win32 Debug" (based on "Win32 (x86) Console Application") !MESSAGE "xmlparse - Win32 Release" (based on\ "Win32 (x86) Dynamic-Link Library") !MESSAGE "xmlparse - Win32 Debug" (based on "Win32 (x86) Dynamic-Link Library") !MESSAGE "xmltok - Win32 Release" (based on "Win32 (x86) Dynamic-Link Library") !MESSAGE "xmltok - Win32 Debug" (based on "Win32 (x86) Dynamic-Link Library") !MESSAGE !ERROR An invalid configuration is specified. !ENDIF !IF "$(OS)" == "Windows_NT" NULL= !ELSE NULL=nul !ENDIF ################################################################################ # Begin Project # PROP Target_Last_Scanned "xmltok - Win32 Debug" !IF "$(CFG)" == "xmlwf - Win32 Release" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 0 # PROP BASE Output_Dir "xmlwf\Release" # PROP BASE Intermediate_Dir "xmlwf\Release" # PROP BASE Target_Dir "xmlwf" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 0 # PROP Output_Dir "xmlwf\Release" # PROP Intermediate_Dir "xmlwf\Release" # PROP Target_Dir "xmlwf" OUTDIR=.\xmlwf\Release INTDIR=.\xmlwf\Release ALL : "xmlparse - Win32 Release" ".\bin\xmlwf.exe" CLEAN : -@erase "$(INTDIR)\codepage.obj" -@erase "$(INTDIR)\win32filemap.obj" -@erase "$(INTDIR)\xmlwf.obj" -@erase ".\bin\xmlwf.exe" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_CONSOLE" /YX /c # ADD CPP /nologo /W3 /GX /O2 /I "." /I "xmlparse" /D "NDEBUG" /D "WIN32" /D "_CONSOLE" /D XMLTOKAPI=__declspec(dllimport) /D XMLPARSEAPI=__declspec(dllimport) /YX /c CPP_PROJ=/nologo /ML /W3 /GX /O2 /I "." /I "xmlparse" /D "NDEBUG" /D "WIN32" /D\ "_CONSOLE" /D XMLTOKAPI=__declspec(dllimport) /D\ XMLPARSEAPI=__declspec(dllimport) /Fp"$(INTDIR)/xmlwf.pch" /YX /Fo"$(INTDIR)/"\ /c CPP_OBJS=.\xmlwf\Release/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< RSC=rc.exe # ADD BASE RSC /l 0x809 /d "NDEBUG" # ADD RSC /l 0x809 /d "NDEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmlwf.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /machine:I386 # ADD LINK32 setargv.obj kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /machine:I386 /out:"bin/xmlwf.exe" LINK32_FLAGS=setargv.obj kernel32.lib user32.lib gdi32.lib winspool.lib\ comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib\ odbc32.lib odbccp32.lib /nologo /subsystem:console /incremental:no\ /pdb:"$(OUTDIR)/xmlwf.pdb" /machine:I386 /out:"bin/xmlwf.exe" LINK32_OBJS= \ "$(INTDIR)\codepage.obj" \ "$(INTDIR)\win32filemap.obj" \ "$(INTDIR)\xmlwf.obj" \ ".\xmlparse\Release\xmlparse.lib" ".\bin\xmlwf.exe" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "xmlwf - Win32 Debug" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 1 # PROP BASE Output_Dir "xmlwf\Debug" # PROP BASE Intermediate_Dir "xmlwf\Debug" # PROP BASE Target_Dir "xmlwf" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 1 # PROP Output_Dir "xmlwf\Debug" # PROP Intermediate_Dir "xmlwf\Debug" # PROP Target_Dir "xmlwf" OUTDIR=.\xmlwf\Debug INTDIR=.\xmlwf\Debug ALL : "xmlparse - Win32 Debug" ".\dbgbin\xmlwf.exe" CLEAN : -@erase "$(INTDIR)\codepage.obj" -@erase "$(INTDIR)\vc40.idb" -@erase "$(INTDIR)\vc40.pdb" -@erase "$(INTDIR)\win32filemap.obj" -@erase "$(INTDIR)\xmlwf.obj" -@erase "$(OUTDIR)\xmlwf.pdb" -@erase ".\dbgbin\xmlwf.exe" -@erase ".\dbgbin\xmlwf.ilk" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /YX /c # ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "." /I "xmlparse" /D "_DEBUG" /D "WIN32" /D "_CONSOLE" /D XMLTOKAPI=__declspec(dllimport) /D XMLPARSEAPI=__declspec(dllimport) /YX /c CPP_PROJ=/nologo /MDd /W3 /Gm /GX /Zi /Od /I "." /I "xmlparse" /D "_DEBUG" /D\ "WIN32" /D "_CONSOLE" /D XMLTOKAPI=__declspec(dllimport) /D\ XMLPARSEAPI=__declspec(dllimport) /Fp"$(INTDIR)/xmlwf.pch" /YX /Fo"$(INTDIR)/"\ /Fd"$(INTDIR)/" /c CPP_OBJS=.\xmlwf\Debug/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< RSC=rc.exe # ADD BASE RSC /l 0x809 /d "_DEBUG" # ADD RSC /l 0x809 /d "_DEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmlwf.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /debug /machine:I386 # ADD LINK32 setargv.obj kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /debug /machine:I386 /out:"dbgbin/xmlwf.exe" LINK32_FLAGS=setargv.obj kernel32.lib user32.lib gdi32.lib winspool.lib\ comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib\ odbc32.lib odbccp32.lib /nologo /subsystem:console /incremental:yes\ /pdb:"$(OUTDIR)/xmlwf.pdb" /debug /machine:I386 /out:"dbgbin/xmlwf.exe" LINK32_OBJS= \ "$(INTDIR)\codepage.obj" \ "$(INTDIR)\win32filemap.obj" \ "$(INTDIR)\xmlwf.obj" \ ".\xmlparse\Debug\xmlparse.lib" ".\dbgbin\xmlwf.exe" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "gennmtab - Win32 Release" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 0 # PROP BASE Output_Dir "gennmtab\Release" # PROP BASE Intermediate_Dir "gennmtab\Release" # PROP BASE Target_Dir "gennmtab" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 0 # PROP Output_Dir "gennmtab\Release" # PROP Intermediate_Dir "gennmtab\Release" # PROP Target_Dir "gennmtab" OUTDIR=.\gennmtab\Release INTDIR=.\gennmtab\Release ALL : "$(OUTDIR)\gennmtab.exe" CLEAN : -@erase "$(INTDIR)\gennmtab.obj" -@erase "$(OUTDIR)\gennmtab.exe" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_CONSOLE" /YX /c # ADD CPP /nologo /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_CONSOLE" /YX /c CPP_PROJ=/nologo /ML /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_CONSOLE"\ /Fp"$(INTDIR)/gennmtab.pch" /YX /Fo"$(INTDIR)/" /c CPP_OBJS=.\gennmtab\Release/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< RSC=rc.exe # ADD BASE RSC /l 0x809 /d "NDEBUG" # ADD RSC /l 0x809 /d "NDEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/gennmtab.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /machine:I386 LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /subsystem:console /incremental:no\ /pdb:"$(OUTDIR)/gennmtab.pdb" /machine:I386 /out:"$(OUTDIR)/gennmtab.exe" LINK32_OBJS= \ "$(INTDIR)\gennmtab.obj" "$(OUTDIR)\gennmtab.exe" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "gennmtab - Win32 Debug" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 1 # PROP BASE Output_Dir "gennmtab\Debug" # PROP BASE Intermediate_Dir "gennmtab\Debug" # PROP BASE Target_Dir "gennmtab" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 1 # PROP Output_Dir "gennmtab\Debug" # PROP Intermediate_Dir "gennmtab\Debug" # PROP Target_Dir "gennmtab" OUTDIR=.\gennmtab\Debug INTDIR=.\gennmtab\Debug ALL : "$(OUTDIR)\gennmtab.exe" CLEAN : -@erase "$(INTDIR)\gennmtab.obj" -@erase "$(INTDIR)\vc40.idb" -@erase "$(INTDIR)\vc40.pdb" -@erase "$(OUTDIR)\gennmtab.exe" -@erase "$(OUTDIR)\gennmtab.ilk" -@erase "$(OUTDIR)\gennmtab.pdb" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /YX /c # ADD CPP /nologo /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /YX /c CPP_PROJ=/nologo /MLd /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE"\ /Fp"$(INTDIR)/gennmtab.pch" /YX /Fo"$(INTDIR)/" /Fd"$(INTDIR)/" /c CPP_OBJS=.\gennmtab\Debug/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< RSC=rc.exe # ADD BASE RSC /l 0x809 /d "_DEBUG" # ADD RSC /l 0x809 /d "_DEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/gennmtab.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /debug /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:console /debug /machine:I386 LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /subsystem:console /incremental:yes\ /pdb:"$(OUTDIR)/gennmtab.pdb" /debug /machine:I386\ /out:"$(OUTDIR)/gennmtab.exe" LINK32_OBJS= \ "$(INTDIR)\gennmtab.obj" "$(OUTDIR)\gennmtab.exe" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "xmlparse - Win32 Release" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 0 # PROP BASE Output_Dir "xmlparse\Release" # PROP BASE Intermediate_Dir "xmlparse\Release" # PROP BASE Target_Dir "xmlparse" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 0 # PROP Output_Dir "xmlparse\Release" # PROP Intermediate_Dir "xmlparse\Release" # PROP Target_Dir "xmlparse" OUTDIR=.\xmlparse\Release INTDIR=.\xmlparse\Release ALL : "xmltok - Win32 Release" ".\bin\xmlparse.dll" CLEAN : -@erase "$(INTDIR)\hashtable.obj" -@erase "$(INTDIR)\xmlparse.obj" -@erase "$(OUTDIR)\xmlparse.exp" -@erase "$(OUTDIR)\xmlparse.lib" -@erase ".\bin\xmlparse.dll" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /MT /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /YX /c # ADD CPP /nologo /W3 /GX /O2 /I "xmltok" /I "xmlwf" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllimport) /D XMLPARSEAPI=__declspec(dllexport) /YX /c CPP_PROJ=/nologo /ML /W3 /GX /O2 /I "xmltok" /I "xmlwf" /D "NDEBUG" /D "WIN32"\ /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllimport) /D\ XMLPARSEAPI=__declspec(dllexport) /Fp"$(INTDIR)/xmlparse.pch" /YX\ /Fo"$(INTDIR)/" /c CPP_OBJS=.\xmlparse\Release/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< MTL=mktyplib.exe # ADD BASE MTL /nologo /D "NDEBUG" /win32 # ADD MTL /nologo /D "NDEBUG" /win32 MTL_PROJ=/nologo /D "NDEBUG" /win32 RSC=rc.exe # ADD BASE RSC /l 0x809 /d "NDEBUG" # ADD RSC /l 0x809 /d "NDEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmlparse.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /base:0x20000000 /subsystem:windows /dll /machine:I386 /out:"bin/xmlparse.dll" # SUBTRACT LINK32 /profile LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /base:0x20000000 /subsystem:windows /dll /incremental:no\ /pdb:"$(OUTDIR)/xmlparse.pdb" /machine:I386 /out:"bin/xmlparse.dll"\ /implib:"$(OUTDIR)/xmlparse.lib" LINK32_OBJS= \ "$(INTDIR)\hashtable.obj" \ "$(INTDIR)\xmlparse.obj" \ ".\xmltok\Release\xmltok.lib" ".\bin\xmlparse.dll" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "xmlparse - Win32 Debug" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 1 # PROP BASE Output_Dir "xmlparse\Debug" # PROP BASE Intermediate_Dir "xmlparse\Debug" # PROP BASE Target_Dir "xmlparse" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 1 # PROP Output_Dir "xmlparse\Debug" # PROP Intermediate_Dir "xmlparse\Debug" # PROP Target_Dir "xmlparse" OUTDIR=.\xmlparse\Debug INTDIR=.\xmlparse\Debug ALL : "xmltok - Win32 Debug" ".\dbgbin\xmlparse.dll" CLEAN : -@erase "$(INTDIR)\hashtable.obj" -@erase "$(INTDIR)\vc40.idb" -@erase "$(INTDIR)\vc40.pdb" -@erase "$(INTDIR)\xmlparse.obj" -@erase "$(OUTDIR)\xmlparse.exp" -@erase "$(OUTDIR)\xmlparse.lib" -@erase "$(OUTDIR)\xmlparse.pdb" -@erase ".\dbgbin\xmlparse.dll" -@erase ".\dbgbin\xmlparse.ilk" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /YX /c # ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "xmltok" /I "xmlwf" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllimport) /D XMLPARSEAPI=__declspec(dllexport) /YX /c CPP_PROJ=/nologo /MDd /W3 /Gm /GX /Zi /Od /I "xmltok" /I "xmlwf" /D "_DEBUG" /D\ "WIN32" /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllimport) /D\ XMLPARSEAPI=__declspec(dllexport) /Fp"$(INTDIR)/xmlparse.pch" /YX\ /Fo"$(INTDIR)/" /Fd"$(INTDIR)/" /c CPP_OBJS=.\xmlparse\Debug/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< MTL=mktyplib.exe # ADD BASE MTL /nologo /D "_DEBUG" /win32 # ADD MTL /nologo /D "_DEBUG" /win32 MTL_PROJ=/nologo /D "_DEBUG" /win32 RSC=rc.exe # ADD BASE RSC /l 0x809 /d "_DEBUG" # ADD RSC /l 0x809 /d "_DEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmlparse.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /base:0x20000000 /subsystem:windows /dll /debug /machine:I386 /out:"dbgbin/xmlparse.dll" LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /base:0x20000000 /subsystem:windows /dll /incremental:yes\ /pdb:"$(OUTDIR)/xmlparse.pdb" /debug /machine:I386 /out:"dbgbin/xmlparse.dll"\ /implib:"$(OUTDIR)/xmlparse.lib" LINK32_OBJS= \ "$(INTDIR)\hashtable.obj" \ "$(INTDIR)\xmlparse.obj" \ ".\xmltok\Debug\xmltok.lib" ".\dbgbin\xmlparse.dll" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "xmltok - Win32 Release" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 0 # PROP BASE Output_Dir "xmltok\Release" # PROP BASE Intermediate_Dir "xmltok\Release" # PROP BASE Target_Dir "xmltok" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 0 # PROP Output_Dir "xmltok\Release" # PROP Intermediate_Dir "xmltok\Release" # PROP Target_Dir "xmltok" OUTDIR=.\xmltok\Release INTDIR=.\xmltok\Release ALL : ".\bin\xmltok.dll" CLEAN : -@erase "$(INTDIR)\dllmain.obj" -@erase "$(INTDIR)\xmlrole.obj" -@erase "$(INTDIR)\xmltok.obj" -@erase "$(OUTDIR)\xmltok.exp" -@erase "$(OUTDIR)\xmltok.lib" -@erase ".\bin\xmltok.dll" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /MT /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /YX /c # ADD CPP /nologo /MT /W3 /GX /O2 /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllexport) /YX /c CPP_PROJ=/nologo /MT /W3 /GX /O2 /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /D\ XMLTOKAPI=__declspec(dllexport) /Fp"$(INTDIR)/xmltok.pch" /YX /Fo"$(INTDIR)/"\ /c CPP_OBJS=.\xmltok\Release/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< MTL=mktyplib.exe # ADD BASE MTL /nologo /D "NDEBUG" /win32 # ADD MTL /nologo /D "NDEBUG" /win32 MTL_PROJ=/nologo /D "NDEBUG" /win32 RSC=rc.exe # ADD BASE RSC /l 0x809 /d "NDEBUG" # ADD RSC /l 0x809 /d "NDEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmltok.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /entry:"DllMain" /subsystem:windows /dll /machine:I386 /out:"bin/xmltok.dll" LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /entry:"DllMain" /subsystem:windows /dll /incremental:no\ /pdb:"$(OUTDIR)/xmltok.pdb" /machine:I386 /out:"bin/xmltok.dll"\ /implib:"$(OUTDIR)/xmltok.lib" LINK32_OBJS= \ "$(INTDIR)\dllmain.obj" \ "$(INTDIR)\xmlrole.obj" \ "$(INTDIR)\xmltok.obj" ".\bin\xmltok.dll" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ELSEIF "$(CFG)" == "xmltok - Win32 Debug" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 1 # PROP BASE Output_Dir "xmltok\Debug" # PROP BASE Intermediate_Dir "xmltok\Debug" # PROP BASE Target_Dir "xmltok" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 1 # PROP Output_Dir "xmltok\Debug" # PROP Intermediate_Dir "xmltok\Debug" # PROP Target_Dir "xmltok" OUTDIR=.\xmltok\Debug INTDIR=.\xmltok\Debug ALL : ".\dbgbin\xmltok.dll" CLEAN : -@erase "$(INTDIR)\dllmain.obj" -@erase "$(INTDIR)\vc40.idb" -@erase "$(INTDIR)\vc40.pdb" -@erase "$(INTDIR)\xmlrole.obj" -@erase "$(INTDIR)\xmltok.obj" -@erase "$(OUTDIR)\xmltok.exp" -@erase "$(OUTDIR)\xmltok.lib" -@erase "$(OUTDIR)\xmltok.pdb" -@erase ".\dbgbin\xmltok.dll" -@erase ".\dbgbin\xmltok.ilk" "$(OUTDIR)" : if not exist "$(OUTDIR)/$(NULL)" mkdir "$(OUTDIR)" CPP=cl.exe # ADD BASE CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /YX /c # ADD CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /D XMLTOKAPI=__declspec(dllexport) /YX /c CPP_PROJ=/nologo /MTd /W3 /Gm /GX /Zi /Od /D "_DEBUG" /D "WIN32" /D "_WINDOWS"\ /D XMLTOKAPI=__declspec(dllexport) /Fp"$(INTDIR)/xmltok.pch" /YX\ /Fo"$(INTDIR)/" /Fd"$(INTDIR)/" /c CPP_OBJS=.\xmltok\Debug/ CPP_SBRS=.\. .c{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_OBJS)}.obj: $(CPP) $(CPP_PROJ) $< .c{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cpp{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< .cxx{$(CPP_SBRS)}.sbr: $(CPP) $(CPP_PROJ) $< MTL=mktyplib.exe # ADD BASE MTL /nologo /D "_DEBUG" /win32 # ADD MTL /nologo /D "_DEBUG" /win32 MTL_PROJ=/nologo /D "_DEBUG" /win32 RSC=rc.exe # ADD BASE RSC /l 0x809 /d "_DEBUG" # ADD RSC /l 0x809 /d "_DEBUG" BSC32=bscmake.exe # ADD BASE BSC32 /nologo # ADD BSC32 /nologo BSC32_FLAGS=/nologo /o"$(OUTDIR)/xmltok.bsc" BSC32_SBRS= \ LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 # ADD LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 /out:"dbgbin/xmltok.dll" LINK32_FLAGS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib\ advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib\ odbccp32.lib /nologo /subsystem:windows /dll /incremental:yes\ /pdb:"$(OUTDIR)/xmltok.pdb" /debug /machine:I386 /out:"dbgbin/xmltok.dll"\ /implib:"$(OUTDIR)/xmltok.lib" LINK32_OBJS= \ "$(INTDIR)\dllmain.obj" \ "$(INTDIR)\xmlrole.obj" \ "$(INTDIR)\xmltok.obj" ".\dbgbin\xmltok.dll" : "$(OUTDIR)" $(DEF_FILE) $(LINK32_OBJS) $(LINK32) @<< $(LINK32_FLAGS) $(LINK32_OBJS) << !ENDIF ################################################################################ # Begin Target # Name "xmlwf - Win32 Release" # Name "xmlwf - Win32 Debug" !IF "$(CFG)" == "xmlwf - Win32 Release" !ELSEIF "$(CFG)" == "xmlwf - Win32 Debug" !ENDIF ################################################################################ # Begin Source File SOURCE=.\xmlwf\xmlwf.c DEP_CPP_XMLWF=\ ".\xmlparse\xmlparse.h"\ ".\xmlwf\codepage.h"\ ".\xmlwf\filemap.h"\ "$(INTDIR)\xmlwf.obj" : $(SOURCE) $(DEP_CPP_XMLWF) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) # End Source File ################################################################################ # Begin Source File SOURCE=.\xmlwf\win32filemap.c DEP_CPP_WIN32=\ ".\xmlwf\filemap.h"\ "$(INTDIR)\win32filemap.obj" : $(SOURCE) $(DEP_CPP_WIN32) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) # End Source File ################################################################################ # Begin Source File SOURCE=.\xmlwf\unixfilemap.c DEP_CPP_UNIXF=\ ".\xmlwf\filemap.h"\ {$(INCLUDE)}"\sys\stat.h"\ {$(INCLUDE)}"\sys\TYPES.H"\ # PROP Exclude_From_Build 1 # End Source File ################################################################################ # Begin Source File SOURCE=.\xmlwf\readfilemap.c DEP_CPP_READF=\ {$(INCLUDE)}"\sys\stat.h"\ {$(INCLUDE)}"\sys\TYPES.H"\ # PROP Exclude_From_Build 1 # End Source File ################################################################################ # Begin Project Dependency # Project_Dep_Name "xmlparse" !IF "$(CFG)" == "xmlwf - Win32 Release" "xmlparse - Win32 Release" : $(MAKE) /$(MAKEFLAGS) /F ".\expat.mak" CFG="xmlparse - Win32 Release" !ELSEIF "$(CFG)" == "xmlwf - Win32 Debug" "xmlparse - Win32 Debug" : $(MAKE) /$(MAKEFLAGS) /F ".\expat.mak" CFG="xmlparse - Win32 Debug" !ENDIF # End Project Dependency ################################################################################ # Begin Source File SOURCE=.\xmlwf\codepage.c DEP_CPP_CODEP=\ ".\xmlwf\codepage.h"\ "$(INTDIR)\codepage.obj" : $(SOURCE) $(DEP_CPP_CODEP) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) # End Source File # End Target ################################################################################ # Begin Target # Name "gennmtab - Win32 Release" # Name "gennmtab - Win32 Debug" !IF "$(CFG)" == "gennmtab - Win32 Release" !ELSEIF "$(CFG)" == "gennmtab - Win32 Debug" !ENDIF ################################################################################ # Begin Source File SOURCE=.\gennmtab\gennmtab.c "$(INTDIR)\gennmtab.obj" : $(SOURCE) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) # End Source File # End Target ################################################################################ # Begin Target # Name "xmlparse - Win32 Release" # Name "xmlparse - Win32 Debug" !IF "$(CFG)" == "xmlparse - Win32 Release" !ELSEIF "$(CFG)" == "xmlparse - Win32 Debug" !ENDIF ################################################################################ # Begin Source File SOURCE=.\xmlparse\xmlparse.c !IF "$(CFG)" == "xmlparse - Win32 Release" DEP_CPP_XMLPA=\ ".\xmlparse\hashtable.h"\ ".\xmlparse\xmlparse.h"\ ".\xmltok\xmldef.h"\ ".\xmltok\xmlrole.h"\ ".\xmltok\xmltok.h"\ NODEP_CPP_XMLPA=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmlparse.obj" : $(SOURCE) $(DEP_CPP_XMLPA) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ELSEIF "$(CFG)" == "xmlparse - Win32 Debug" DEP_CPP_XMLPA=\ ".\xmlparse\hashtable.h"\ ".\xmlparse\xmlparse.h"\ ".\xmltok\xmldef.h"\ ".\xmltok\xmlrole.h"\ ".\xmltok\xmltok.h"\ NODEP_CPP_XMLPA=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmlparse.obj" : $(SOURCE) $(DEP_CPP_XMLPA) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ENDIF # End Source File ################################################################################ # Begin Source File SOURCE=.\xmlparse\hashtable.c !IF "$(CFG)" == "xmlparse - Win32 Release" DEP_CPP_HASHT=\ ".\xmlparse\hashtable.h"\ ".\xmltok\xmldef.h"\ NODEP_CPP_HASHT=\ ".\xmltok\nspr.h"\ "$(INTDIR)\hashtable.obj" : $(SOURCE) $(DEP_CPP_HASHT) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ELSEIF "$(CFG)" == "xmlparse - Win32 Debug" DEP_CPP_HASHT=\ ".\xmlparse\hashtable.h"\ ".\xmltok\xmldef.h"\ NODEP_CPP_HASHT=\ ".\xmltok\nspr.h"\ "$(INTDIR)\hashtable.obj" : $(SOURCE) $(DEP_CPP_HASHT) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ENDIF # End Source File ################################################################################ # Begin Project Dependency # Project_Dep_Name "xmltok" !IF "$(CFG)" == "xmlparse - Win32 Release" "xmltok - Win32 Release" : $(MAKE) /$(MAKEFLAGS) /F ".\expat.mak" CFG="xmltok - Win32 Release" !ELSEIF "$(CFG)" == "xmlparse - Win32 Debug" "xmltok - Win32 Debug" : $(MAKE) /$(MAKEFLAGS) /F ".\expat.mak" CFG="xmltok - Win32 Debug" !ENDIF # End Project Dependency # End Target ################################################################################ # Begin Target # Name "xmltok - Win32 Release" # Name "xmltok - Win32 Debug" !IF "$(CFG)" == "xmltok - Win32 Release" !ELSEIF "$(CFG)" == "xmltok - Win32 Debug" !ENDIF ################################################################################ # Begin Source File SOURCE=.\xmltok\xmlrole.c !IF "$(CFG)" == "xmltok - Win32 Release" DEP_CPP_XMLRO=\ ".\xmltok\xmldef.h"\ ".\xmltok\xmlrole.h"\ ".\xmltok\xmltok.h"\ NODEP_CPP_XMLRO=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmlrole.obj" : $(SOURCE) $(DEP_CPP_XMLRO) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ELSEIF "$(CFG)" == "xmltok - Win32 Debug" DEP_CPP_XMLRO=\ ".\xmltok\xmldef.h"\ ".\xmltok\xmlrole.h"\ ".\xmltok\xmltok.h"\ NODEP_CPP_XMLRO=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmlrole.obj" : $(SOURCE) $(DEP_CPP_XMLRO) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ENDIF # End Source File ################################################################################ # Begin Source File SOURCE=.\xmltok\xmltok.c !IF "$(CFG)" == "xmltok - Win32 Release" DEP_CPP_XMLTO=\ ".\xmltok\asciitab.h"\ ".\xmltok\iasciitab.h"\ ".\xmltok\latin1tab.h"\ ".\xmltok\nametab.h"\ ".\xmltok\utf8tab.h"\ ".\xmltok\xmldef.h"\ ".\xmltok\xmltok.h"\ ".\xmltok\xmltok_impl.c"\ ".\xmltok\xmltok_impl.h"\ NODEP_CPP_XMLTO=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmltok.obj" : $(SOURCE) $(DEP_CPP_XMLTO) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ELSEIF "$(CFG)" == "xmltok - Win32 Debug" DEP_CPP_XMLTO=\ ".\xmltok\asciitab.h"\ ".\xmltok\iasciitab.h"\ ".\xmltok\latin1tab.h"\ ".\xmltok\nametab.h"\ ".\xmltok\utf8tab.h"\ ".\xmltok\xmldef.h"\ ".\xmltok\xmltok.h"\ ".\xmltok\xmltok_impl.c"\ ".\xmltok\xmltok_impl.h"\ NODEP_CPP_XMLTO=\ ".\xmltok\nspr.h"\ "$(INTDIR)\xmltok.obj" : $(SOURCE) $(DEP_CPP_XMLTO) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ENDIF # End Source File ################################################################################ # Begin Source File SOURCE=.\xmltok\dllmain.c !IF "$(CFG)" == "xmltok - Win32 Release" "$(INTDIR)\dllmain.obj" : $(SOURCE) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ELSEIF "$(CFG)" == "xmltok - Win32 Debug" "$(INTDIR)\dllmain.obj" : $(SOURCE) "$(INTDIR)" $(CPP) $(CPP_PROJ) $(SOURCE) !ENDIF # End Source File # End Target # End Project ################################################################################ Zope-2.8.7-final/lib/python/Shared/DC/xml/xmllib.py0000664000076400007640000006510710435500330020766 0ustar ajungajung# A parser for XML, using the derived class as static DTD. # Author: Sjoerd Mullender. import re import string version = '0.1' # Regular expressions used for parsing _S = '[ \t\r\n]+' _opS = '[ \t\r\n]*' _Name = '[a-zA-Z_:][-a-zA-Z0-9._:]*' illegal = re.compile('[^\t\r\n -\176\240-\377]') # illegal chars in content interesting = re.compile('[]&<]') amp = re.compile('&') ref = re.compile('&(' + _Name + '|#[0-9]+|#x[0-9a-fA-F]+)[^-a-zA-Z0-9._:]') entityref = re.compile('&(?P' + _Name + ')[^-a-zA-Z0-9._:]') charref = re.compile('&#(?P[0-9]+[^0-9]|x[0-9a-fA-F]+[^0-9a-fA-F])') space = re.compile(_S + '$') newline = re.compile('\n') starttagopen = re.compile('<' + _Name) endtagopen = re.compile('/?)>') endbracket = re.compile(_opS + '>') tagfind = re.compile(_Name) cdataopen = re.compile(r'') # this matches one of the following: # SYSTEM SystemLiteral # PUBLIC PubidLiteral SystemLiteral _SystemLiteral = '(?P<%s>\'[^\']*\'|"[^"]*")' _PublicLiteral = '(?P<%s>"[-\'()+,./:=?;!*#@$_%% \n\ra-zA-Z0-9]*"|' \ "'[-()+,./:=?;!*#@$_%% \n\ra-zA-Z0-9]*')" _ExternalId = '(?:SYSTEM|' \ 'PUBLIC'+_S+_PublicLiteral%'pubid'+ \ ')'+_S+_SystemLiteral%'syslit' doctype = re.compile(''+_Name+')' '(?:'+_S+_ExternalId+')?'+_opS) xmldecl = re.compile('<\?xml'+_S+ 'version'+_opS+'='+_opS+'(?P\'[^\']*\'|"[^"]*")'+ '(?:'+_S+'encoding'+_opS+'='+_opS+ "(?P'[A-Za-z][-A-Za-z0-9._]*'|" '"[A-Za-z][-A-Za-z0-9._]*"))?' '(?:'+_S+'standalone'+_opS+'='+_opS+ '(?P\'(?:yes|no)\'|"(?:yes|no)"))?'+ _opS+'\?>') procopen = re.compile(r'<\?(?P' + _Name + ')' + _opS) procclose = re.compile(_opS + r'\?>') commentopen = re.compile('') doubledash = re.compile('--') attrfind = re.compile( _S + '(?P' + _Name + ')' '(' + _opS + '=' + _opS + '(?P\'[^\']*\'|"[^"]*"|[-a-zA-Z0-9.:+*%?!()_#=~]+))') attrtrans = string.maketrans(' \r\n\t', ' ') # XML parser base class -- find tags and call handler functions. # Usage: p = XMLParser(); p.feed(data); ...; p.close(). # The dtd is defined by deriving a class which defines methods with # special names to handle tags: start_foo and end_foo to handle # and , respectively. The data between tags is passed to the # parser by calling self.handle_data() with some data as argument (the # data may be split up in arbutrary chunks). Entity references are # passed by calling self.handle_entityref() with the entity reference # as argument. class XMLParser: # Interface -- initialize and reset this instance def __init__(self, verbose=0): self.verbose = verbose self.reset() # Interface -- reset this instance. Loses all unprocessed data def reset(self): self.rawdata = '' self.stack = [] self.nomoretags = 0 self.literal = 0 self.lineno = 1 self.__at_start = 1 self.__seen_doctype = None self.__seen_starttag = 0 # For derived classes only -- enter literal mode (CDATA) till EOF def setnomoretags(self): self.nomoretags = self.literal = 1 # For derived classes only -- enter literal mode (CDATA) def setliteral(self, *args): self.literal = 1 # Interface -- feed some data to the parser. Call this as # often as you want, with as little or as much text as you # want (may include '\n'). (This just saves the text, all the # processing is done by goahead().) def feed(self, data): self.rawdata = self.rawdata + data self.goahead(0) # Interface -- handle the remaining data def close(self): self.goahead(1) # Interface -- translate references def translate_references(self, data, all = 1): i = 0 while 1: res = amp.search(data, i) if res is None: return data res = ref.match(data, res.start(0)) if res is None: self.syntax_error("bogus `&'") i =i+1 continue i = res.end(0) if data[i - 1] != ';': self.syntax_error("`;' missing after entity/char reference") i = i-1 str = res.group(1) pre = data[:res.start(0)] post = data[i:] if str[0] == '#': if str[1] == 'x': str = chr(string.atoi(str[2:], 16)) else: str = chr(string.atoi(str[1:])) data = pre + str + post i = res.start(0)+len(str) elif all: if self.entitydefs.has_key(str): data = pre + self.entitydefs[str] + post i = res.start(0) # rescan substituted text else: self.syntax_error('reference to unknown entity') # can't do it, so keep the entity ref in data = pre + '&' + str + ';' + post i = res.start(0) + len(str) + 2 else: # just translating character references pass # i is already postioned correctly # Internal -- handle data as far as reasonable. May leave state # and data to be processed by a subsequent call. If 'end' is # true, force handling all data as if followed by EOF marker. def goahead(self, end): rawdata = self.rawdata i = 0 n = len(rawdata) while i < n: if i > 0: self.__at_start = 0 if self.nomoretags: data = rawdata[i:n] self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') i = n break res = interesting.search(rawdata, i) if res: j = res.start(0) else: j = n if i < j: if self.__at_start: self.syntax_error('illegal data at start of file') self.__at_start = 0 data = rawdata[i:j] if not self.stack and not space.match(data): self.syntax_error('data not in content') if illegal.search(data): self.syntax_error('illegal character in content') self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') i = j if i == n: break if rawdata[i] == '<': if starttagopen.match(rawdata, i): if self.literal: data = rawdata[i] self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') i = i+1 continue k = self.parse_starttag(i) if k < 0: break self.__seen_starttag = 1 self.lineno = self.lineno + string.count(rawdata[i:k], '\n') i = k continue if endtagopen.match(rawdata, i): k = self.parse_endtag(i) if k < 0: break self.lineno = self.lineno + string.count(rawdata[i:k], '\n') i = k self.literal = 0 continue if commentopen.match(rawdata, i): if self.literal: data = rawdata[i] self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') i = i+1 continue k = self.parse_comment(i) if k < 0: break self.lineno = self.lineno + string.count(rawdata[i:k], '\n') i = k continue if cdataopen.match(rawdata, i): k = self.parse_cdata(i) if k < 0: break self.lineno = self.lineno + string.count(rawdata[i:i], '\n') i = k continue res = xmldecl.match(rawdata, i) if res: if not self.__at_start: self.syntax_error(" declaration not at start of document") version, encoding, standalone = res.group('version', 'encoding', 'standalone') if version[1:-1] != '1.0': raise RuntimeError, 'only XML version 1.0 supported' if encoding: encoding = encoding[1:-1] if standalone: standalone = standalone[1:-1] self.handle_xml(encoding, standalone) i = res.end(0) continue res = procopen.match(rawdata, i) if res: k = self.parse_proc(i) if k < 0: break self.lineno = self.lineno + string.count(rawdata[i:k], '\n') i = k continue res = doctype.match(rawdata, i) if res: if self.literal: data = rawdata[i] self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') i = i+1 continue if self.__seen_doctype: self.syntax_error('multiple DOCTYPE elements') if self.__seen_starttag: self.syntax_error('DOCTYPE not at beginning of document') k = self.parse_doctype(res) if k < 0: break self.__seen_doctype = res.group('name') self.lineno = self.lineno + string.count(rawdata[i:k], '\n') i = k continue elif rawdata[i] == '&': res = charref.match(rawdata, i) if res is not None: i = res.end(0) if rawdata[i-1] != ';': self.syntax_error("`;' missing in charref") i = i-1 if not self.stack: self.syntax_error('data not in content') self.handle_charref(res.group('char')[:-1]) self.lineno = self.lineno + string.count(res.group(0), '\n') continue res = entityref.match(rawdata, i) if res is not None: i = res.end(0) if rawdata[i-1] != ';': self.syntax_error("`;' missing in entityref") i = i-1 name = res.group('name') if self.entitydefs.has_key(name): self.rawdata = rawdata = rawdata[:res.start(0)] + self.entitydefs[name] + rawdata[i:] n = len(rawdata) i = res.start(0) else: self.syntax_error('reference to unknown entity') self.unknown_entityref(name) self.lineno = self.lineno + string.count(res.group(0), '\n') continue elif rawdata[i] == ']': if n-i < 3: break if cdataclose.match(rawdata, i): self.syntax_error("bogus `]]>'") self.handle_data(rawdata[i]) i = i+1 continue else: raise RuntimeError, 'neither < nor & ??' # We get here only if incomplete matches but # nothing else break # end while if i > 0: self.__at_start = 0 if end and i < n: data = rawdata[i] self.syntax_error("bogus `%s'" % data) if illegal.search(data): self.syntax_error('illegal character in content') self.handle_data(data) self.lineno = self.lineno + string.count(data, '\n') self.rawdata = rawdata[i+1:] return self.goahead(end) self.rawdata = rawdata[i:] if end: if not self.__seen_starttag: self.syntax_error('no elements in file') if self.stack: self.syntax_error('missing end tags') while self.stack: self.finish_endtag(self.stack[-1]) # Internal -- parse comment, return length or -1 if not terminated def parse_comment(self, i): rawdata = self.rawdata if rawdata[i:i+4] <> '\n' % (s,t,v)), tb finally: tb=None self._v_connected=DateTime() return self def sql_quote__(self, v): if find(v,"\'") >= 0: v=join(split(v,"\'"),"''") return "'%s'" % v Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/0000775000076400007640000000000010435500330020015 5ustar ajungajungZope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/customDefaultReport.dtml0000664000076400007640000000157310435500330024720 0ustar ajungajung (Previous results) %(else no_table)[ %(heading)s %(else)] %(row)s %(else no_table)[
%(else)] (Next results) There was no data matching this &dtml-title_or_id; query. Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/customDefaultZPTReport.dtml0000664000076400007640000000260010435500330025306 0ustar ajungajung

previous 20 results next 20 results

%(else no_table)[ %(heading)s %(else)] %(row)s %(else no_table)[
%(else)]

previous 20 results next 20 results

Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/connectionAdd.dtml0000664000076400007640000000276710435500330023463 0ustar ajungajung
Id
Title
Connection String
Connect Immediately?

Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/advanced.dtml0000664000076400007640000000646710435500330022461 0ustar ajungajung
Connection Hook
">
Maximum rows to retrieve
Maximum results to cache
Maximum time (sec) to cache
Allow "simple" direct traversal
checked />

You may specify a class for the data records. This class must be defined in a file that resides in the Extensions directory of this Zope installation.
Class Name
Class File

You may specify a ZClass for the data records.
ZClass
Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/connectionTestForm.dtml0000664000076400007640000000170210435500330024522 0ustar ajungajung

You can write a test query and run it with this database connection using the form below. Click submit query to run the query.

This database connection is currently closed. You must click on the status tab and open the connection before you can test it.

Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/searchAdd.dtml0000664000076400007640000000566310435500330022567 0ustar ajungajung

A Search Interface allows you to search Zope databases. The Search Interface will create a search-input form and a report for displaying the search results.

In the form below, seachable objects are the objects (usually SQL Methods) to be searched. report id and search input id are the ids of the report and search form objects that will be created. report style indicates the type of report to generate.

Select one
or more
searchable
objects
Report Id
Report Title
Report Style
Search Input Id
Search Input Title
Generate DTML Methods
Generate Page Templates
Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/connectionEdit.dtml0000664000076400007640000000247110435500330023650 0ustar ajungajung
Id
&dtml-id;
Title
Connection String
Connect Immediately

Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dtml/connectionStatus.dtml0000664000076400007640000000107210435500330024242 0ustar ajungajung

The database connection is openclosed.

Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/sqltest.py0000664000076400007640000001640310435500330021132 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''Inserting optional tests with 'sqlgroup' It is sometimes useful to make inputs to an SQL statement optinal. Doing so can be difficult, because not only must the test be inserted conditionally, but SQL boolean operators may or may not need to be inserted depending on whether other, possibly optional, comparisons have been done. The 'sqlgroup' tag automates the conditional insertion of boolean operators. The 'sqlgroup' tag is a block tag that has no attributes. It can have any number of 'and' and 'or' continuation tags. Suppose we want to find all people with a given first or nick name and optionally constrain the search by city and minimum and maximum age. Suppose we want all inputs to be optional. We can use DTML source like the following:: age >= age <= This example illustrates how groups can be nested to control boolean evaluation order. It also illustrates that the grouping facility can also be used with other DTML tags like 'if' tags. The 'sqlgroup' tag checks to see if text to be inserted contains other than whitespace characters. If it does, then it is inserted with the appropriate boolean operator, as indicated by use of an 'and' or 'or' tag, otherwise, no text is inserted. ''' __rcs_id__='$Id: sqltest.py 28517 2004-11-26 16:33:03Z efge $' ############################################################################ # Copyright # # Copyright 1996 Digital Creations, L.C., 910 Princess Anne # Street, Suite 300, Fredericksburg, Virginia 22401 U.S.A. All # rights reserved. # ############################################################################ __version__='$Revision: 1.5 $'[11:-2] import sys from DocumentTemplate.DT_Util import ParseError, parse_params, name_param str=__builtins__['str'] from string import find, split, join, atoi, atof from types import ListType, TupleType, StringType class SQLTest: name='sqltest' optional=multiple=None def __init__(self, args): args = parse_params(args, name='', expr='', type=None, column=None, multiple=1, optional=1, op=None) name,expr = name_param(args,'sqlvar',1) if expr is None: expr=name else: expr=expr.eval self.__name__, self.expr = name, expr self.args=args if not args.has_key('type'): raise ParseError, ('the type attribute is required', 'sqltest') self.type=t=args['type'] if not valid_type(t): raise ParseError, ('invalid type, %s' % t, 'sqltest') if args.has_key('optional'): self.optional=args['optional'] if args.has_key('multiple'): self.multiple=args['multiple'] if args.has_key('column'): self.column=args['column'] elif self.__name__ is None: err = ' the column attribute is required if an expression is used' raise ParseError, (err, 'sqltest') else: self.column=self.__name__ # Deal with optional operator specification op = '=' # Default if args.has_key('op'): op = args['op'] # Try to get it from the chart, otherwise use the one provided op = comparison_operators.get(op, op) self.op = op def render(self, md): name=self.__name__ t=self.type args=self.args try: expr=self.expr if type(expr) is type(''): v=md[expr] else: v=expr(md) except KeyError: if args.has_key('optional') and args['optional']: return '' raise ValueError, 'Missing input variable, %s' % name if type(v) in (ListType, TupleType): if len(v) > 1 and not self.multiple: raise ValueError, ( 'multiple values are not allowed for %s' % name) else: v=[v] vs=[] for v in v: if not v and type(v) is StringType and t != 'string': continue if t=='int': try: if type(v) is StringType: if v[-1:]=='L': v=v[:-1] atoi(v) else: v=str(int(v)) except ValueError: raise ValueError, ( 'Invalid integer value for %s' % name) elif t=='float': if not v and type(v) is StringType: continue try: if type(v) is StringType: atof(v) else: v=str(float(v)) except ValueError: raise ValueError, ( 'Invalid floating-point value for %s' % name) else: if not isinstance(v, (str, unicode)): v = str(v) v=md.getitem('sql_quote__',0)(v) #if find(v,"\'") >= 0: v=join(split(v,"\'"),"''") #v="'%s'" % v vs.append(v) if not vs and t=='nb': if args.has_key('optional') and args['optional']: return '' else: err = 'Invalid empty string value for %s' % name raise ValueError, err if not vs: if self.optional: return '' raise ValueError, ( 'No input was provided for %s' % name) if len(vs) > 1: vs=join(map(str,vs),', ') if self.op == '<>': ## Do the equivalent of 'not-equal' for a list, ## "a not in (b,c)" return "%s not in (%s)" % (self.column, vs) else: ## "a in (b,c)" return "%s in (%s)" % (self.column, vs) return "%s %s %s" % (self.column, self.op, vs[0]) __call__=render valid_type={'int':1, 'float':1, 'string':1, 'nb': 1}.has_key comparison_operators = { 'eq': '=', 'ne': '<>', 'lt': '<', 'le': '<=', 'lte': '<=', 'gt': '>', 'ge': '>=', 'gte': '>=' } Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/dbi_db.py0000664000076400007640000001151010435500330020630 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __doc__='''short description $Id: dbi_db.py 29694 2005-03-28 16:34:11Z tim_one $''' # Copyright # # Copyright 1997 Digital Creations, Inc, 910 Princess Anne # Street, Suite 300, Fredericksburg, Virginia 22401 U.S.A. All # rights reserved. # __version__='$Revision: 1.11 $'[11:-2] import string, sys from string import strip, split, find, join import transaction failures=0 calls=0 nonselect_desc=[ ('Query', 'STRING', 62, 62, 0, 0, 1), ('Status', 'STRING', 12, 12, 0, 0, 1), ('Calls', 'STRING', 12, 12, 0, 0, 1), ] class QueryError(Exception): pass class DB: _p_oid=_p_changed=_registered=None defs={'STRING':'s', 'NUMBER':'n', 'DATE':'d'} def Database_Connection(self, string): # Create a dbi-compatible database connection raise NotImplemetedError, ( 'attempt to create a database connection for an abstract dbi') Database_Error='Should be overriden by subclass' def __init__(self,connection): self.connection=connection db=self.db=self.Database_Connection(connection) self.cursor=db.cursor() def str(self,v, StringType=type('')): if v is None: return '' r=str(v) if r[-1:]=='L' and type(v) is not StringType: r=r[:-1] return r def __inform_commit__(self, *ignored): self._registered=None self.db.commit() def __inform_abort__(self, *ignored): self._registered=None self.db.rollback() def register(self): if self._registered: return transaction.get().register(self) self._registered=1 def query(self,query_string, max_rows=9999999): global failures, calls calls=calls+1 try: c=self.cursor self.register() queries=filter(None, map(strip,split(query_string, '\0'))) if not queries: raise QueryError, 'empty query' if len(queries) > 1: result=[] for qs in queries: r=c.execute(qs) if r is None: raise QueryError, ( 'select in multiple sql-statement query' ) result.append((qs, str(`r`), calls)) desc=nonselect_desc else: query_string=queries[0] r=c.execute(query_string) if r is None: result=c.fetchmany(max_rows) desc=c.description else: result=((query_string, str(`r`), calls),) desc=nonselect_desc failures=0 c.close() except self.Database_Error, mess: c.close() self.db.rollback() failures=failures+1 if ((find(mess,": invalid") < 0 and find(mess,"PARSE") < 0) or # DBI IS stupid find(mess, "Error while trying to retrieve text for error") > 0 or # If we have a large number of consecutive failures, # our connection is probably dead. failures > 100 ): # Hm. maybe the db is hosed. Let's try once to restart it. failures=0 c.close() self.db.close() db=self.db=self.Database_Connection(self.connection) self.cursor=db.cursor() c=self.cursor c.execute(query_string) result=c.fetchall() desc=c.description else: raise sys.exc_info() if result: result=join( map( lambda row, self=self: join(map(self.str,row),'\t'), result), '\n')+'\n' else: result='' return ( "%s\n%s\n%s" % ( join(map(lambda d: d[0],desc), '\t'), join( map( lambda d, defs=self.defs: "%d%s" % (d[2],defs[d[1]]), desc), '\t'), result, ) ) Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/THUNK.py0000664000076400007640000000300410435500330020315 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import TM, ThreadLock from TM import Surrogate import transaction thunk_lock=ThreadLock.allocate_lock() class THUNKED_TM(TM.TM): """A big heavy hammer for handling non-thread safe DAs """ def _register(self): if not self._registered: thunk_lock.acquire() try: transaction.get().register(Surrogate(self)) self._begin() except: thunk_lock.release() raise else: self._registered=1 def tpc_finish(self, *ignored): if self._registered: try: self._finish() finally: thunk_lock.release() self._registered=0 def abort(self, *ignored): if self._registered: try: self._abort() finally: thunk_lock.release() self._registered=0 Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/sqlgroup.py0000664000076400007640000001162410435500330021307 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Inserting optional tests with 'sqlgroup' It is sometimes useful to make inputs to an SQL statement optinal. Doing so can be difficult, because not only must the test be inserted conditionally, but SQL boolean operators may or may not need to be inserted depending on whether other, possibly optional, comparisons have been done. The 'sqlgroup' tag automates the conditional insertion of boolean operators. The 'sqlgroup' tag is a block tag. It can have any number of 'and' and 'or' continuation tags. The 'sqlgroup' tag has an optional attribure, 'required' to specify groups that must include at least one test. This is useful when you want to make sure that a query is qualified, but want to be very flexible about how it is qualified. Suppose we want to find people with a given first or nick name, city or minimum and maximum age. Suppose we want all inputs to be optional, but want to require *some* input. We can use DTML source like the following:: age >= age <= This example illustrates how groups can be nested to control boolean evaluation order. It also illustrates that the grouping facility can also be used with other DTML tags like 'if' tags. The 'sqlgroup' tag checks to see if text to be inserted contains other than whitespace characters. If it does, then it is inserted with the appropriate boolean operator, as indicated by use of an 'and' or 'or' tag, otherwise, no text is inserted. $Id: sqlgroup.py 38487 2005-09-16 14:42:07Z tseaver $ """ ############################################################################ # Copyright # # Copyright 1996 Digital Creations, L.C., 910 Princess Anne # Street, Suite 300, Fredericksburg, Virginia 22401 U.S.A. All # rights reserved. # ############################################################################ __rcs_id__='$Id: sqlgroup.py 38487 2005-09-16 14:42:07Z tseaver $' __version__='$Revision: 1.10 $'[11:-2] from DocumentTemplate.DT_Util import parse_params str = __builtins__['str'] from string import strip, join import sys _TNAME_MAPPING = {'comma': ','} class SQLGroup: blockContinuations = 'and', 'or', 'comma' name = 'sqlgroup' required = None where = None set = None noparens = None def __init__(self, blocks): self.blocks = blocks tname, args, section = blocks[0] self.__name__ = "%s %s" % (tname, args) args = parse_params(args, required=1, where=1, set=1, noparens=1) if args.has_key(''): args[args['']] = 1 if args.has_key('required'): self.required = args['required'] if args.has_key('where'): self.where = args['where'] if args.has_key('set'): self.set = args['set'] if args.has_key('noparens'): self.noparens = args['noparens'] def render(self,md): r = [] for tname, args, section in self.blocks: __traceback_info__ = tname s = strip(section(None, md)) if s: if r: r.append(_TNAME_MAPPING.get(tname, tname)) if self.noparens: r.append(s) else: r.append("%s\n" % s) if r: if len(r) > 1: if self.noparens: r = "%s\n" % join(r,' ') else: r = "(%s)\n" % join(r,' ') else: r = r[0] if self.set: r = "set\n"+r if self.where: r = "where\n"+r return r if self.required: raise ValueError, 'Not enough input was provided!

' return '' __call__ = render Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/TM.py0000664000076400007640000000467610435500330017764 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Provide support for linking an external transaction manager with Zope's """ import transaction class TM: """Mix-in class that provides transaction management support A sub class should call self._register() whenever it performs any transaction-dependent operations (e.g. sql statements). The sub class will need to override _finish, to finalize work, _abort, to roll-back work, and perhaps _begin, if any work is needed at the start of a transaction. A subclass that uses locking during transaction commit must defined a sortKey() method. """ _registered=None def _begin(self): pass def _register(self): if not self._registered: try: transaction.get().register(Surrogate(self)) self._begin() self._registered = 1 self._finalize = 0 except: pass def tpc_begin(self, *ignored): pass commit=tpc_begin def _finish(self): self.db.commit() def _abort(self): self.db.rollback() def tpc_vote(self, *ignored): self._finalize = 1 def tpc_finish(self, *ignored): if self._finalize: try: self._finish() finally: self._registered=0 def abort(self, *ignored): try: self._abort() finally: self._registered=0 tpc_abort = abort def sortKey(self, *ignored): """ The sortKey method is used for recent ZODB compatibility which needs to have a known commit order for lock acquisition. Most DA's talking to RDBMS systems do not care about commit order, so return the constant 1 """ return 1 class Surrogate: def __init__(self, db): self._p_jar=db self.__inform_commit__=db.tpc_finish self.__inform_abort__=db.tpc_abort Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/Results.py0000664000076400007640000001140210435500330021066 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import ExtensionClass from Acquisition import Implicit from Record import Record class SQLAlias(ExtensionClass.Base): def __init__(self, name): self._n=name def __of__(self, parent): return getattr(parent, self._n) class NoBrains: pass class Results: """Class for providing a nice interface to DBI result data """ _index=None # We need to allow access to not-explicitly-protected # individual record objects contained in the result. __allow_access_to_unprotected_subobjects__=1 def __init__(self,(items,data),brains=NoBrains, parent=None, zbrains=None): self._data=data self.__items__=items self._parent=parent self._names=names=[] self._schema=schema={} self._data_dictionary=dd={} aliases=[] if zbrains is None: zbrains=NoBrains i=0 for item in items: name=item['name'] name=name.strip() if not name: raise ValueError, 'Empty column name, %s' % name if schema.has_key(name): raise ValueError, 'Duplicate column name, %s' % name schema[name]=i n=name.lower() if n != name: aliases.append((n, SQLAlias(name))) n=name.upper() if n != name: aliases.append((n, SQLAlias(name))) dd[name]=item names.append(name) i=i+1 self._nv=nv=len(names) # Create a record class to hold the records. names=tuple(names) class r(Record, Implicit, brains, zbrains): 'Result record class' r.__record_schema__=schema for k in Record.__dict__.keys(): if k[:2]=='__': setattr(r,k,getattr(Record,k)) # Add SQL Aliases for k, v in aliases: if not hasattr(r, k): setattr(r, k, v) if hasattr(brains, '__init__'): binit=brains.__init__ if hasattr(binit,'im_func'): binit=binit.im_func def __init__(self, data, parent, binit=binit): Record.__init__(self,data) if parent is not None: self=self.__of__(parent) binit(self) setattr(r, '__init__', __init__) self._class=r # OK, we've read meta data, now get line indexes def _searchable_result_columns(self): return self.__items__ def names(self): return self._names def data_dictionary(self): return self._data_dictionary def __len__(self): return len(self._data) def __getitem__(self,index): if index==self._index: return self._row parent=self._parent fields=self._class(self._data[index], parent) if parent is not None: fields=fields.__of__(parent) self._index=index self._row=fields return fields def tuples(self): return map(tuple, self) def dictionaries(self): r=[] a=r.append names=self.names() for row in self: d={} for n in names: d[n]=row[n] a(d) return r def asRDB(self): # Waaaaa r=[] append=r.append strings=[] nstrings=[] items=self.__items__ indexes=range(len(items)) for i in indexes: item=items[i] t=item['type'].lower() if t=='s' or t=='t': t=='t' strings.append(i) else: nstrings.append(i) if item.has_key('width'): append('%s%s' % (item['width'], t)) else: r.append(t) r=['\t'.join(self._names), '\t'.join(r)] append=r.append row=['']*len(items) tostr=str for d in self._data: for i in strings: v=tostr(d[i]) if v: if v.find('\\') > 0: v='\\\\'.join(v.split('\\')) if v.find('\t') > 0: v='\\t'.join(v.split('\t')) if v.find('\n') > 0: v='\\n'.join(v.split('\n')) row[i]=v for i in nstrings: row[i]=tostr(d[i]) append('\t'.join(row)) append('') return '\n'.join(r) Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/tests/0000775000076400007640000000000010435610645020232 5ustar ajungajungZope-2.8.7-final/lib/python/Shared/DC/ZRDB/tests/test_sqlgroup.py0000664000076400007640000001652010435500330023510 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from UserDict import UserDict def _sql_quote(v): return '"%s"' % v class SQLGroupTests(unittest.TestCase): def _getTargetClass(self): from Shared.DC.ZRDB.sqlgroup import SQLGroup return SQLGroup def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def test_ctor_empty_args(self): group = self._makeOne([('sqlgroup', '', None)]) self.assertEqual(group.__name__, 'sqlgroup ') self.failIf(group.required) self.failIf(group.where) self.failIf(group.set) self.failIf(group.noparens) def test_ctor_required(self): group = self._makeOne([('sqlgroup', 'required', None)]) self.assertEqual(group.__name__, 'sqlgroup required') self.failUnless(group.required) self.failIf(group.where) self.failIf(group.set) self.failIf(group.noparens) def test_ctor_where(self): group = self._makeOne([('sqlgroup', 'where', None)]) self.assertEqual(group.__name__, 'sqlgroup where') self.failIf(group.required) self.failUnless(group.where) self.failIf(group.set) self.failIf(group.noparens) def test_ctor_noparens(self): group = self._makeOne([('sqlgroup', 'noparens', None)]) self.assertEqual(group.__name__, 'sqlgroup noparens') self.failIf(group.required) self.failIf(group.where) self.failIf(group.set) self.failUnless(group.noparens) def test_ctor_set(self): group = self._makeOne([('sqlgroup', 'set', None)]) self.assertEqual(group.__name__, 'sqlgroup set') self.failIf(group.required) self.failIf(group.where) self.failUnless(group.set) self.failIf(group.noparens) def test_render_empty_optional(self): group = self._makeOne([('sqlgroup', '', lambda x, y:'')]) md = {} self.assertEqual(group.render(md), '') def test_render_empty_optional_where(self): group = self._makeOne([('sqlgroup', 'where', lambda x, y:'')]) md = {} self.assertEqual(group.render(md), '') def test_render_empty_optional_set(self): group = self._makeOne([('sqlgroup', 'set', lambda x, y:'')]) md = {} self.assertEqual(group.render(md), '') def test_render_empty_required_raises_ValueError(self): group = self._makeOne([('sqlgroup', 'required', lambda x, y:'')]) md = {} self.assertRaises(ValueError, group.render, md) def test_render_one_block(self): group = self._makeOne([('sqlgroup', '', lambda x, y:'abc'), ]) md = {} rendered = group.render(md) rendered = ''.join(rendered.split('\n')) self.assertEqual(rendered, 'abc') def test_render_one_block_where(self): group = self._makeOne([('sqlgroup', 'where', lambda x, y:'abc'), ]) md = {} rendered = group.render(md) self.assertEqual(rendered, 'where\nabc\n') def test_render_one_block_set(self): group = self._makeOne([('sqlgroup', 'set', lambda x, y:'abc'), ]) md = {} rendered = group.render(md) self.assertEqual(rendered, 'set\nabc\n') def test_render_multiple_blocks_with_tname(self): group = self._makeOne([('sqlgroup', '', lambda x, y:'abc'), ('baz', '', lambda x, y: 'def'), ('qux', '', lambda x, y: 'ghi'), ]) md = {} rendered = group.render(md) rendered = ''.join(rendered.split('\n')) self.assertEqual(rendered, '(abc baz def qux ghi)') def test_render_multiple_blocks_with_tname_noparens(self): group = self._makeOne([('sqlgroup', 'noparens', lambda x, y:'abc'), ('baz', '', lambda x, y: 'def'), ('qux', '', lambda x, y: 'ghi'), ]) md = {} rendered = group.render(md) rendered = ''.join(rendered.split('\n')) self.assertEqual(rendered, 'abc baz def qux ghi') def test_render_multiple_blocks_with_tname_and_where(self): group = self._makeOne([('sqlgroup', 'where', lambda x, y:'abc'), ('baz', '', lambda x, y: 'def'), ('qux', '', lambda x, y: 'ghi'), ]) md = {} rendered = group.render(md) rendered = ''.join(rendered.split('\n')) self.assertEqual(rendered, 'where(abc baz def qux ghi)') def test_parsed_rendered_complex_where(self): # something of a functional test, as we use nvSQL to get parsed. from Shared.DC.ZRDB.DA import nvSQL template = nvSQL(WHERE_EXAMPLE) mapping = {} mapping['name'] = 'Goofy' mapping['home_town'] = 'Orlando' mapping['sql_quote__'] = _sql_quote rendered = template(None, mapping) self.assertEqual(rendered, 'select * from actors\n' 'where\n' '((nick_name = "Goofy"\n' ' or first_name = "Goofy"\n)\n' ' and home_town = "Orlando"\n)\n' ) def test_parsed_rendered_complex_set(self): # something of a functional test, as we use nvSQL to get parsed. from Shared.DC.ZRDB.DA import nvSQL template = nvSQL(UPDATE_EXAMPLE) mapping = {} mapping['nick_name'] = 'Goofy' mapping['home_town'] = 'Orlando' mapping['sql_quote__'] = _sql_quote rendered = template(None, mapping) self.assertEqual(rendered, 'update actors\n' 'set\nnick_name = "Goofy" , home_town = "Orlando"\n' ) WHERE_EXAMPLE = """\ select * from actors age >= age <= """ UPDATE_EXAMPLE = """\ update actors """ def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(SQLGroupTests)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/tests/__init__.py0000664000076400007640000000001610435500330022325 0ustar ajungajung# import this Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/tests/test_sqlvar.py0000664000076400007640000001244510435500330023146 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from UserDict import UserDict def _sql_quote(v): return '"%s"' % v class FauxMultiDict(UserDict): def getitem(self, key, call): if key == 'sql_quote__': return _sql_quote v = self[key] if v is not None: if call and callable(v): v = v() return v class SQLVarTests(unittest.TestCase): def _getTargetClass(self): from Shared.DC.ZRDB.sqlvar import SQLVar return SQLVar def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def test_constructor_no_type(self): from DocumentTemplate.DT_Util import ParseError self.assertRaises(ParseError, self._makeOne, 'foo') def test_constructor_invalid_type(self): from DocumentTemplate.DT_Util import ParseError self.assertRaises(ParseError, self._makeOne, 'foo type="nonesuch"') def test_constructor_valid_type(self): from DocumentTemplate.DT_Util import ParseError v = self._makeOne('foo type="string"') self.assertEqual(v.__name__, 'foo') self.assertEqual(v.expr, 'foo') self.assertEqual(v.args['type'], 'string') def test_render_name_returns_value(self): v = self._makeOne('foo type="string"') self.assertEqual(v.render(FauxMultiDict(foo='FOO')), '"FOO"') def test_render_name_missing_required_raises_ValueError(self): v = self._makeOne('foo type="string"') self.assertRaises(ValueError, v.render, FauxMultiDict()) def test_render_name_missing_optional_returns_null(self): v = self._makeOne('foo type="string" optional') self.assertEqual(v.render(FauxMultiDict()), 'null') def test_render_expr_returns_value(self): v = self._makeOne('expr="foo" type="string"') self.assertEqual(v.render(FauxMultiDict(foo='FOO')), '"FOO"') def test_render_expr_missing_required_raises_NameError(self): v = self._makeOne('expr="foo" type="string"') self.assertRaises(NameError, v.render, FauxMultiDict()) def test_render_expr_missing_optional_returns_null(self): v = self._makeOne('expr="foo" type="string" optional') self.assertEqual(v.render(FauxMultiDict()), 'null') def test_render_int_returns_int_without_quoting(self): v = self._makeOne('expr="foo" type="int"') self.assertEqual(v.render(FauxMultiDict(foo=42)), '42') def test_render_int_with_long_returns_value_without_L(self): v = self._makeOne('expr="foo" type="int"') self.assertEqual(v.render(FauxMultiDict(foo='42L')), '42') def test_render_int_required_invalid_raises_ValueError(self): v = self._makeOne('expr="foo" type="int"') self.assertRaises(ValueError, v.render, FauxMultiDict(foo='')) def test_render_int_optional_invalid_returns_null(self): v = self._makeOne('expr="foo" type="int" optional') self.assertEqual(v.render(FauxMultiDict(foo='')), 'null') def test_render_float_returns_float_without_quoting(self): v = self._makeOne('expr="foo" type="float"') self.assertEqual(v.render(FauxMultiDict(foo=3.1415)), '3.1415') def test_render_float_with_long_returns_value_without_L(self): v = self._makeOne('expr="foo" type="float"') self.assertEqual(v.render(FauxMultiDict(foo='42L')), '42') def test_render_float_required_invalid_raises_ValueError(self): v = self._makeOne('expr="foo" type="float"') self.assertRaises(ValueError, v.render, FauxMultiDict(foo='')) def test_render_float_optional_invalid_returns_null(self): v = self._makeOne('expr="foo" type="float" optional') self.assertEqual(v.render(FauxMultiDict(foo='')), 'null') def test_render_nb_required_with_blank_raises_ValueError(self): v = self._makeOne('expr="foo" type="nb"') self.assertRaises(ValueError, v.render, FauxMultiDict(foo='')) def test_render_nb_optional_with_blank_returns_null(self): v = self._makeOne('expr="foo" type="nb" optional') self.assertEqual(v.render(FauxMultiDict(foo='')), 'null') def test_render_name_with_none_returns_null(self): # Collector #556, patch from Dieter Maurer v = self._makeOne('foo type="string"') self.assertEqual(v.render(FauxMultiDict(foo=None)), 'null') def test_render_expr_with_none_returns_null(self): # Collector #556, patch from Dieter Maurer v = self._makeOne('expr="foo" type="string"') self.assertEqual(v.render(FauxMultiDict(foo=None)), 'null') def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(SQLVarTests)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/tests/testResult.py0000664000076400007640000000401010435500330022742 0ustar ajungajungfrom unittest import TestCase, TestSuite, makeSuite, main from cStringIO import StringIO from ExtensionClass import Base from Shared.DC.ZRDB.Results import Results from Shared.DC.ZRDB import RDB class Brain: def __init__(self, *args): pass Parent = Base() class TestResults(TestCase): def test_results(self): r = Results(([{'name':'foo', 'type':'integer'}, {'name':'bar', 'type':'integer'}], ((1, 2), (3, 4))), brains=Brain, parent=Parent) self.assertEquals(len(r), 2) row = r[0] self.assertEquals(row[0], 1) self.assertEquals(row[1], 2) self.assertEquals(row.foo, 1) self.assertEquals(row.bar, 2) self.assertEquals(row.FOO, 1) self.assertEquals(row.BAR, 2) row = r[1] self.assertEquals(row[0], 3) self.assertEquals(row[1], 4) self.assertEquals(row.foo, 3) self.assertEquals(row.bar, 4) self.assertEquals(row.FOO, 3) self.assertEquals(row.BAR, 4) self.failUnless(isinstance(row, Brain)) def test_rdb_file(self): infile = StringIO("""\ foo\tbar 2i\t2i 1\t2 3\t4\ """) r = RDB.File(infile, brains=Brain, parent=Parent) self.assertEquals(len(r), 2) row = r[0] self.assertEquals(row[0], 1) self.assertEquals(row[1], 2) self.assertEquals(row.foo, 1) self.assertEquals(row.bar, 2) self.assertEquals(row.FOO, 1) self.assertEquals(row.BAR, 2) row = r[1] self.assertEquals(row[0], 3) self.assertEquals(row[1], 4) self.assertEquals(row.foo, 3) self.assertEquals(row.bar, 4) self.assertEquals(row.FOO, 3) self.assertEquals(row.BAR, 4) self.failUnless(isinstance(row, Brain)) def test_suite(): return TestSuite((makeSuite(TestResults),)) if __name__ == '__main__': main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/sqlvar.py0000664000076400007640000001167610435500330020752 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''Inserting values with the 'sqlvar' tag The 'sqlvar' tag is used to type-safely insert values into SQL text. The 'sqlvar' tag is similar to the 'var' tag, except that it replaces text formatting parameters with SQL type information. The sqlvar tag has the following attributes: name -- The name of the variable to insert. As with other DTML tags, the 'name=' prefix may be, and usually is, ommitted. type -- The data type of the value to be inserted. This attribute is required and may be one of 'string', 'int', 'float', or 'nb'. The 'nb' data type indicates a string that must have a length that is greater than 0. optional -- A flag indicating that a value is optional. If a value is optional and is not provided (or is blank when a non-blank value is expected), then the string 'null' is inserted. For example, given the tag:: if the value of 'x' is:: Let\'s do it then the text inserted is: 'Let''s do it' however, if x is ommitted or an empty string, then the value inserted is 'null'. ''' __rcs_id__='$Id: sqlvar.py 38274 2005-09-02 23:12:46Z tseaver $' ############################################################################ # Copyright # # Copyright 1996 Digital Creations, L.C., 910 Princess Anne # Street, Suite 300, Fredericksburg, Virginia 22401 U.S.A. All # rights reserved. # ############################################################################ __version__='$Revision: 1.15 $'[11:-2] from DocumentTemplate.DT_Util import ParseError, parse_params, name_param from string import find, split, join, atoi, atof StringType=type('') str=__builtins__['str'] class SQLVar: name='sqlvar' def __init__(self, args): args = parse_params(args, name='', expr='', type=None, optional=1) name,expr=name_param(args,'sqlvar',1) if expr is None: expr=name else: expr=expr.eval self.__name__, self.expr = name, expr self.args=args if not args.has_key('type'): raise ParseError, ('the type attribute is required', 'dtvar') t=args['type'] if not valid_type(t): raise ParseError, ('invalid type, %s' % t, 'dtvar') def render(self, md): name=self.__name__ args=self.args t=args['type'] try: expr=self.expr if type(expr) is type(''): v=md[expr] else: v=expr(md) except: if args.has_key('optional') and args['optional']: return 'null' if type(expr) is not type(''): raise raise ValueError, 'Missing input variable, %s' % name if v is None: return 'null' if t=='int': try: if type(v) is StringType: if v[-1:]=='L': v=v[:-1] atoi(v) else: v=str(int(v)) except: if not v and args.has_key('optional') and args['optional']: return 'null' raise ValueError, ( 'Invalid integer value for %s' % name) elif t=='float': try: if type(v) is StringType: if v[-1:]=='L': v=v[:-1] atof(v) else: v=str(float(v)) except: if not v and args.has_key('optional') and args['optional']: return 'null' raise ValueError, ( 'Invalid floating-point value for %s' % name) else: if not isinstance(v, (str, unicode)): v=str(v) if not v and t=='nb': if args.has_key('optional') and args['optional']: return 'null' else: raise ValueError, ( 'Invalid empty string value for %s' % name) v=md.getitem('sql_quote__',0)(v) #if find(v,"\'") >= 0: v=join(split(v,"\'"),"''") #v="'%s'" % v return v __call__=render valid_type={'int':1, 'float':1, 'string':1, 'nb': 1}.has_key Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/RDB.py0000664000076400007640000001475210435500330020047 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __doc__='''Class for reading RDB files $Id: RDB.py 29520 2005-03-17 12:09:24Z sidnei $''' __version__='$Revision: 1.33 $'[11:-2] from string import split, strip, lower, upper, atof, atoi, atol, find, join,find import DateTime,re from Missing import MV from array import array from Record import Record from Acquisition import Implicit import ExtensionClass def parse_text(s): if find(s,'\\') < 0 and (find(s,'\\t') < 0 and find(s,'\\n') < 0): return s r=[] for x in split(s,'\\\\'): x=join(split(x,'\\n'),'\n') r.append(join(split(x,'\\t'),'\t')) return join(r,'\\') Parsers={'n': atof, 'i': atoi, 'l': atol, 'd': DateTime.DateTime, 't': parse_text, } class SQLAlias(ExtensionClass.Base): def __init__(self, name): self._n=name def __of__(self, parent): return getattr(parent, self._n) class NoBrains: pass class DatabaseResults: """Class for reading RDB files """ _index=None # We need to allow access to not-explicitly-protected # individual record objects contained in the result. __allow_access_to_unprotected_subobjects__=1 def __init__(self,file,brains=NoBrains, parent=None, zbrains=None): self._file=file readline=file.readline line=readline() self._parent=parent if zbrains is None: zbrains=NoBrains while line and line.find('#') != -1 : line=readline() line=line[:-1] if line and line[-1:] in '\r\n': line=line[:-1] self._names=names=split(line,'\t') if not names: raise ValueError, 'No column names' aliases=[] self._schema=schema={} i=0 for name in names: name=strip(name) if not name: raise ValueError, 'Empty column name, %s' % name if schema.has_key(name): raise ValueError, 'Duplicate column name, %s' % name schema[name]=i n=lower(name) if n != name: aliases.append((n, SQLAlias(name))) n=upper(name) if n != name: aliases.append((n, SQLAlias(name))) i=i+1 self._nv=nv=len(names) line=readline() line=line[:-1] if line[-1:] in '\r\n': line=line[:-1] self._defs=defs=split(line,'\t') if not defs: raise ValueError, 'No column definitions' if len(defs) != nv: raise ValueError, ( """The number of column names and the number of column definitions are different.""") i=0 self._parsers=parsers=[] defre=re.compile(r'([0-9]*)([a-zA-Z])?') self._data_dictionary=dd={} self.__items__=items=[] for _def in defs: _def=strip(_def) if not _def: raise ValueError, ('Empty column definition for %s' % names[i]) mo = defre.match(_def) if mo is None: raise ValueError, ( 'Invalid column definition for, %s, for %s' % _def, names[i]) type = mo.group(2).lower() width = mo.group(1) if width: width=atoi(width) else: width=8 try: parser=Parsers[type] except: parser=str name=names[i] d={'name': name, 'type': type, 'width': width, 'parser': parser} items.append(d) dd[name]=d parsers.append((i,parser)) i=i+1 # Create a record class to hold the records. names=tuple(names) class r(Record, Implicit, brains, zbrains): 'Result record class' r.__record_schema__=schema for k in filter(lambda k: k[:2]=='__', Record.__dict__.keys()): setattr(r,k,getattr(Record,k)) # Add SQL Aliases for k, v in aliases: if not hasattr(r,k): setattr(r, k, v) if hasattr(brains, '__init__'): binit=brains.__init__ if hasattr(binit,'im_func'): binit=binit.im_func def __init__(self, data, parent, binit=binit): Record.__init__(self,data) binit(self.__of__(parent)) setattr(r, '__init__', __init__) self._class=r # OK, we've read meta data, now get line indexes p=file.tell() save=self._lines=array('i') save=save.append l=readline() while l: save(p) p=p+len(l) l=readline() def _searchable_result_columns(self): return self.__items__ def names(self): return self._names def data_dictionary(self): return self._data_dictionary def __len__(self): return len(self._lines) def __getitem__(self,index): if index==self._index: return self._row file=self._file file.seek(self._lines[index]) line=file.readline() line=line[:-1] if line and line[-1:] in '\r\n': line=line[:-1] fields=split(line,'\t') l=len(fields) nv=self._nv if l != nv: if l < nv: fields=fields+['']*(nv-l) else: raise ValueError, ( """The number of items in record %s is invalid 

%s\n%s\n%s\n%s
""" % (index, ('='*40), line, ('='*40), fields)) for i, parser in self._parsers: try: v=parser(fields[i]) except: if fields[i]: raise ValueError, ( """Invalid value, %s, for %s in record %s""" % (fields[i], self._names[i], index)) else: v=MV fields[i]=v parent=self._parent fields=self._class(fields, parent) self._index=index self._row=fields if parent is None: return fields return fields.__of__(parent) File=DatabaseResults Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/www/0000775000076400007640000000000010435500330017701 5ustar ajungajungZope-2.8.7-final/lib/python/Shared/DC/ZRDB/www/DBAdapter_icon.gif0000775000076400007640000000162010435500330023170 0ustar ajungajungGIF89a/OOp!,@m @A(0!PaCظ #Nذaď$daȑ%/8P8s,2G cLy"K%T"ŋ2A,L!"Z+;Zope-2.8.7-final/lib/python/Shared/DC/ZRDB/www/DBAdapterFolder_icon.gif0000775000076400007640000000160110435500330024323 0ustar ajungajungGIF89aPPP@@@@@!,^ :X #`H1C4searchable objects were selected') if not report_id: raise ValueError, ( 'No report id were specified') if input_title and not input_id: raise ValueError, ( 'No input id were specified') qs=map(lambda q, self=self: _getquery(self, q), queries) arguments={} keys=[] checkPermission=getSecurityManager().checkPermission for q in qs: url=q.absolute_url() if input_id: for name, arg in q._searchable_arguments().items(): if len(qs) > 1: key="%s/%s" % (id,name) else: key=name arguments[key]=arg keys.append(key) if q._searchable_result_columns() is None: raise ValueError,( """The input searchable object, %s, has not been tested. Until it has been tested, it\'s output schema is unknown, and a report cannot be generated. Before creating a report from this query, you must try out the query. To try out the query, click here. """ % (escape(q.title_and_id()), escape(url, 1))) if object_type == 'dtml_methods': if not checkPermission('Add DTML Methods', self): raise Unauthorized, ( 'You are not authorized to add DTML Methods.' ) if input_id: arguments=Args(arguments, keys) self.manage_addDocument( input_id,input_title, default_input_form(arguments, report_id)) self.manage_addDocument( report_id,report_title, ('\n%s\n' '' % join(map(lambda q, report_style=report_style: custom_default_report(q.id, q, no_table=report_style), qs), '\n
\n'))) if REQUEST: return self.manage_main(self,REQUEST) elif object_type == 'page_templates': if not checkPermission('Add Page Templates', self): raise Unauthorized, ( 'You are not authorized to add Page Templates.' ) if input_id: arguments = Args(arguments, keys) self.manage_addProduct['PageTemplates'].manage_addPageTemplate( input_id, input_title, default_input_zpt_form(arguments, report_id)) self.manage_addProduct['PageTemplates'].manage_addPageTemplate( report_id,report_title, ('\n%s\n' '' % join(map(lambda q, report_style=report_style: custom_default_zpt_report(q.id, q, no_table=report_style), qs), '\n
\n'))) if REQUEST: return self.manage_main(self,REQUEST) def ZQueryIds(self): # Note that report server configurations will expend on this t=[] ids={} old=ids.has_key o=self n=0 while 1: # Look for queries try: map=o.objectMap() except AttributeError: map=() for i in map: try: id=i['id'] if (not old(id) and hasattr(getattr(o,id),'_searchable_arguments') ): t.append(i['id']) ids[id]=1 except: pass # Now extend search to parent try: o=o.aq_parent except: return t if n > 100: return t # Seat belt n=n+1 def _getquery(self,id): o=self i=0 while 1: __traceback_info__=o q=getattr(o,id) try: if hasattr(q,'_searchable_arguments'): try: q=q.__of__(self.aq_parent) except: pass return q except: pass if i > 100: raise AttributeError, id i=i+1 o=o.aq_parent def default_input_form(arguments,action='query', tabs=''): if arguments: items=arguments.items() return ( "%s\n%s%s" % ( '\n%s\n' '
\n' '

\n' 'Enter query parameters:
' '\n' % (tabs,action), join( map( lambda a: ('\n' ' ' % (nicify(a[0]), ( a[1].has_key('type') and ("%s:%s" % (a[0],a[1]['type'])) or a[0] ), a[1].has_key('default') and a[1]['default'] or '' )) , items ), '\n'), '\n\n
%s' '
\n' '\n' '
\n
\n' '\n' ) ) else: return ( '\n%s\n' '
\n' '

\n' 'This query requires no input.

\n' '\n' '

\n' '\n' % (tabs, action) ) def default_input_zpt_form(arguments,action='query', tabs=''): if arguments: items=arguments.items() return ( "%s\n%s%s" % ( '\n%s\n' '
\n' '

Title

\n' 'Enter query parameters:
' '\n' % (tabs,action), join( map( lambda a: ('\n' ' ' % (nicify(a[0]), ( a[1].has_key('type') and ("%s:%s" % (a[0],a[1]['type'])) or a[0] ), a[1].has_key('default') and a[1]['default'] or '' )) , items ), '\n'), '\n\n
%s' '
\n' '\n' '
\n
\n' '\n' ) ) else: return ( '\n%s\n' '
\n' '

Title

\n' '

This query requires no input.

\n' '\n' '
\n' '\n' % (tabs, action) ) Zope-2.8.7-final/lib/python/Shared/__init__.py0000664000076400007640000000134510435500330020142 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __doc__='''$Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $''' __version__='$Revision: 1.7 $'[11:-2] Zope-2.8.7-final/lib/python/webdav/0000775000076400007640000000000010435610645016103 5ustar ajungajungZope-2.8.7-final/lib/python/webdav/EtagSupport.py0000664000076400007640000001336310435500330020725 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Etag support. $Id: EtagSupport.py 30356 2005-05-15 15:56:18Z yuppie $ """ import time from Interface import Interface from webdav.common import PreconditionFailed class EtagBaseInterface(Interface): """\ Basic Etag support interface, meaning the object supports generating an Etag that can be used by certain HTTP and WebDAV Requests. """ def http__etag(): """\ Entity tags are used for comparing two or more entities from the same requested resource. Predominantly used for Caching, Etags can also be used to deal with the 'Lost Updates Problem'. An HTTP Client such as Amaya that supports PUT for editing can use the Etag value returned in the head of a GET response in the 'if-match' header submitted with a PUT request. If the Etag for the requested resource in the PUT request's 'if-match' header is different from the current Etag value returned by this method, the PUT will fail (it means that the state of the resource has changed since the last copy the Client recieved) because the precondition (the 'if-match') fails (the submitted Etag does not match the current Etag). """ def http__refreshEtag(): """\ While it may make sense to use the ZODB Object Id or bobobase_modification_time to generate an Etag, this could fail on certain REQUESTS because: o The object is not stored in the ZODB, or o A Request such as PUT changes the oid or bobobase_modification_time *AFTER* the Response has been written out, but the Etag needs to be updated and returned with the Response of the PUT request. Thus, Etags need to be refreshed manually when an object changes. """ class EtagSupport: """\ This class is the basis for supporting Etags in Zope. It's main function right now is to support the *Lost Updates Problem* by allowing Etags and If-Match headers to be checked on PUT calls to provide a *Seatbelt* style functionality. The Etags is based on the bobobase_modification_time, and thus is updated whenever the object is updated. If a PUT request, or other HTTP or Dav request comes in with an Etag different than the current one, that request can be rejected according to the type of header (If-Match, If-None-Match). """ __implements__ = (EtagBaseInterface,) def http__etag(self, readonly=0): try: etag = self.__etag except AttributeError: if readonly: # Don't refresh the etag on reads return self.http__refreshEtag() etag = self.__etag return etag def http__refreshEtag(self): self.__etag = 'ts%s' % str(time.time())[2:] def http__parseMatchList(self, REQUEST, header="if-match"): # Return a sequence of strings found in the header specified # (should be one of {'if-match' or 'if-none-match'}). If the # header is not in the request, returns None. Otherwise, # returns a tuple of Etags. matchlist = REQUEST.get_header(header) if matchlist is None: matchlist = REQUEST.get_header(header.title()) if matchlist is None: return None matchlist = [ x.strip() for x in matchlist.split(',')] r = [] for match in matchlist: if match == '*': r.insert(0, match) elif (match[0] + match[-1] == '""') and (len(match) > 2): r.append(match[1:-1]) return tuple(r) def http__processMatchHeaders(self, REQUEST=None): # Process if-match and if-none-match headers if REQUEST is None: REQUEST = self.aq_acquire('REQUEST') matchlist = self.http__parseMatchList(REQUEST, 'if-match') nonematch = self.http__parseMatchList(REQUEST, 'if-none-match') if matchlist is None: # There's no Matchlist, but 'if-none-match' might need processing pass elif ('*' in matchlist): return 1 # * matches everything elif self.http__etag() not in matchlist: # The resource etag is not in the list of etags required # to match, as specified in the 'if-match' header. The # condition fails and the HTTP Method may *not* execute. raise PreconditionFailed elif self.http__etag() in matchlist: return 1 if nonematch is None: # There's no 'if-none-match' header either, so there's no # problem continuing with the request return 1 elif ('*' in nonelist): # if-none-match: * means that the operation should not # be performed if the specified resource exists # (webdav.NullResource will want to do special behavior # here) raise PreconditionFailed elif self.http__etag() in nonelist: # The opposite of if-match, the condition fails # IF the resources Etag is in the if-none-match list raise PreconditionFailed elif self.http__etag() not in nonelist: return 1 Zope-2.8.7-final/lib/python/webdav/Resource.py0000664000076400007640000005772410435500330020250 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """WebDAV support - resource objects. $Id: Resource.py 65606 2006-02-28 19:55:33Z sidnei $ """ import sys import mimetypes from urllib import unquote import Globals import ExtensionClass from Acquisition import aq_base from AccessControl import getSecurityManager from ZPublisher.HTTPRangeSupport import HTTPRangeInterface from zExceptions import Unauthorized, Forbidden from zExceptions import BadRequest, MethodNotAllowed, NotFound from WriteLockInterface import WriteLockInterface import Lockable from common import absattr, urlfix, rfc1123_date, tokenFinder, urlbase from common import IfParser from common import isDavCollection from common import Locked, Conflict, PreconditionFailed import davcmds class Resource(ExtensionClass.Base, Lockable.LockableItem): """The Resource mixin class provides basic WebDAV support for non-collection objects. It provides default implementations for most supported WebDAV HTTP methods, however certain methods such as PUT should be overridden to ensure correct behavior in the context of the object type.""" __dav_resource__=1 __http_methods__=('GET', 'HEAD', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'TRACE', 'PROPFIND', 'PROPPATCH', 'MKCOL', 'COPY', 'MOVE', 'LOCK', 'UNLOCK', ) __ac_permissions__=( ('View', ('HEAD',)), ('WebDAV access', ('PROPFIND', 'manage_DAVget', 'listDAVObjects'), ('Authenticated', 'Manager')), ('Manage properties', ('PROPPATCH',)), ('Delete objects', ('DELETE',)), ('WebDAV Lock items', ('LOCK',)), ('WebDAV Unlock items', ('UNLOCK',)), ) def dav__init(self, request, response): # Init expected HTTP 1.1 / WebDAV headers which are not # currently set by the base response object automagically. # # We sniff for a ZServer response object, because we don't # want to write duplicate headers (since ZS writes Date # and Connection itself). if not hasattr(response, '_server_version'): response.setHeader('Connection', 'close') response.setHeader('Date', rfc1123_date(), 1) # HTTP Range support if HTTPRangeInterface.isImplementedBy(self): response.setHeader('Accept-Ranges', 'bytes') else: response.setHeader('Accept-Ranges', 'none') def dav__validate(self, object, methodname, REQUEST): msg='You are not authorized to access this resource.' method=None if hasattr(object, methodname): method=getattr(object, methodname) else: try: method=object.aq_acquire(methodname) except: method=None if method is not None: try: return getSecurityManager().validate(None, object, methodname, method) except: pass raise Unauthorized, msg def dav__simpleifhandler(self, request, response, method='PUT', col=0, url=None, refresh=0): ifhdr = request.get_header('If', None) if Lockable.wl_isLocked(self) and (not ifhdr): raise Locked, "Resource is locked." if not ifhdr: return None if not Lockable.wl_isLocked(self): return None # Since we're a simple if handler, and since some clients don't # pass in the port information in the resource part of an If # header, we're only going to worry about if the paths compare if url is None: url = urlfix(request['URL'], method) url = urlbase(url) # Gets just the path information # if 'col' is passed in, an operation is happening on a submember # of a collection, while the Lock may be on the parent. Lob off # the final part of the URL (ie '/a/b/foo.html' becomes '/a/b/') if col: url = url[:url.rfind('/')+1] havetag = lambda x, self=self: self.wl_hasLock(x) found = 0; resourcetagged = 0 taglist = IfParser(ifhdr) for tag in taglist: if not tag.resource: # There's no resource (url) with this tag tag_list = map(tokenFinder, tag.list) wehave = filter(havetag, tag_list) if not wehave: continue if tag.NOTTED: continue if refresh: for token in wehave: self.wl_getLock(token).refresh() resourcetagged = 1 found = 1; break elif urlbase(tag.resource) == url: resourcetagged = 1 tag_list = map(tokenFinder, tag.list) wehave = filter(havetag, tag_list) if not wehave: continue if tag.NOTTED: continue if refresh: for token in wehave: self.wl_getLock(token).refresh() found = 1; break if resourcetagged and (not found): raise PreconditionFailed, 'Condition failed.' elif resourcetagged and found: return 1 else: return 0 # WebDAV class 1 support def HEAD(self, REQUEST, RESPONSE): """Retrieve resource information without a response body.""" self.dav__init(REQUEST, RESPONSE) content_type=None if hasattr(self, 'content_type'): content_type=absattr(self.content_type) if content_type is None: url=urlfix(REQUEST['URL'], 'HEAD') name=unquote(filter(None, url.split( '/')[-1])) content_type, encoding=mimetypes.guess_type(name) if content_type is None: if hasattr(self, 'default_content_type'): content_type=absattr(self.default_content_type) if content_type is None: content_type = 'application/octet-stream' RESPONSE.setHeader('Content-Type', content_type.lower()) if hasattr(aq_base(self), 'get_size'): RESPONSE.setHeader('Content-Length', absattr(self.get_size)) if hasattr(self, '_p_mtime'): mtime=rfc1123_date(self._p_mtime) RESPONSE.setHeader('Last-Modified', mtime) if hasattr(aq_base(self), 'http__etag'): etag = self.http__etag(readonly=1) if etag: RESPONSE.setHeader('Etag', etag) RESPONSE.setStatus(200) return RESPONSE def PUT(self, REQUEST, RESPONSE): """Replace the GET response entity of an existing resource. Because this is often object-dependent, objects which handle PUT should override the default PUT implementation with an object-specific implementation. By default, PUT requests fail with a 405 (Method Not Allowed).""" self.dav__init(REQUEST, RESPONSE) raise MethodNotAllowed, 'Method not supported for this resource.' OPTIONS__roles__=None def OPTIONS(self, REQUEST, RESPONSE): """Retrieve communication options.""" self.dav__init(REQUEST, RESPONSE) RESPONSE.setHeader('Allow', ', '.join(self.__http_methods__)) RESPONSE.setHeader('Content-Length', 0) RESPONSE.setHeader('DAV', '1,2', 1) RESPONSE.setStatus(200) return RESPONSE TRACE__roles__=None def TRACE(self, REQUEST, RESPONSE): """Return the HTTP message received back to the client as the entity-body of a 200 (OK) response. This will often usually be intercepted by the web server in use. If not, the TRACE request will fail with a 405 (Method Not Allowed), since it is not often possible to reproduce the HTTP request verbatim from within the Zope environment.""" self.dav__init(REQUEST, RESPONSE) raise MethodNotAllowed, 'Method not supported for this resource.' def DELETE(self, REQUEST, RESPONSE): """Delete a resource. For non-collection resources, DELETE may return either 200 or 204 (No Content) to indicate success.""" self.dav__init(REQUEST, RESPONSE) ifhdr = REQUEST.get_header('If', '') url = urlfix(REQUEST['URL'], 'DELETE') name = unquote(filter(None, url.split( '/')[-1])) parent = self.aq_parent # Lock checking if Lockable.wl_isLocked(self): if ifhdr: self.dav__simpleifhandler(REQUEST, RESPONSE, 'DELETE') else: # We're locked, and no if header was passed in, so # the client doesn't own a lock. raise Locked, 'Resource is locked.' elif WriteLockInterface.isImplementedBy(parent) and \ parent.wl_isLocked(): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, 'DELETE', col=1) else: # Our parent is locked, and no If header was passed in. # When a parent is locked, members cannot be removed raise PreconditionFailed, 'Resource is locked, and no '\ 'condition was passed in.' # Either we're not locked, or a succesful lock token was submitted # so we can delete the lock now. # ajung: Fix for Collector # 2196 if parent.manage_delObjects([name],REQUEST=None) is None: RESPONSE.setStatus(204) else: RESPONSE.setStatus(403) return RESPONSE def PROPFIND(self, REQUEST, RESPONSE): """Retrieve properties defined on the resource.""" self.dav__init(REQUEST, RESPONSE) cmd=davcmds.PropFind(REQUEST) result=cmd.apply(self) # work around MSIE DAV bug for creation and modified date if (REQUEST.get_header('User-Agent') == 'Microsoft Data Access Internet Publishing Provider DAV 1.1'): result = result.replace('', '') result = result.replace('', '') RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) return RESPONSE def PROPPATCH(self, REQUEST, RESPONSE): """Set and/or remove properties defined on the resource.""" self.dav__init(REQUEST, RESPONSE) if not hasattr(aq_base(self), 'propertysheets'): raise MethodNotAllowed, ( 'Method not supported for this resource.') # Lock checking ifhdr = REQUEST.get_header('If', '') if Lockable.wl_isLocked(self): if ifhdr: self.dav__simpleifhandler(REQUEST, RESPONSE, 'PROPPATCH') else: raise Locked, 'Resource is locked.' cmd=davcmds.PropPatch(REQUEST) result=cmd.apply(self) RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) return RESPONSE def MKCOL(self, REQUEST, RESPONSE): """Create a new collection resource. If called on an existing resource, MKCOL must fail with 405 (Method Not Allowed).""" self.dav__init(REQUEST, RESPONSE) raise MethodNotAllowed, 'The resource already exists.' COPY__roles__=('Anonymous',) def COPY(self, REQUEST, RESPONSE): """Create a duplicate of the source resource whose state and behavior match that of the source resource as closely as possible. Though we may later try to make a copy appear seamless across namespaces (e.g. from Zope to Apache), COPY is currently only supported within the Zope namespace.""" self.dav__init(REQUEST, RESPONSE) if not hasattr(aq_base(self), 'cb_isCopyable') or \ not self.cb_isCopyable(): raise MethodNotAllowed, 'This object may not be copied.' depth=REQUEST.get_header('Depth', 'infinity') if not depth in ('0', 'infinity'): raise BadRequest, 'Invalid Depth header.' dest=REQUEST.get_header('Destination', '') while dest and dest[-1]=='/': dest=dest[:-1] if not dest: raise BadRequest, 'Invalid Destination header.' try: path = REQUEST.physicalPathFromURL(dest) except ValueError: raise BadRequest, 'Invalid Destination header' name = path.pop() oflag=REQUEST.get_header('Overwrite', 'F').upper() if not oflag in ('T', 'F'): raise BadRequest, 'Invalid Overwrite header.' try: parent=self.restrictedTraverse(path) except ValueError: raise Conflict, 'Attempt to copy to an unknown namespace.' except NotFound: raise Conflict, 'Object ancestors must already exist.' except: t, v, tb=sys.exc_info() raise t, v if hasattr(parent, '__null_resource__'): raise Conflict, 'Object ancestors must already exist.' existing=hasattr(aq_base(parent), name) if existing and oflag=='F': raise PreconditionFailed, 'Destination resource exists.' try: parent._checkId(name, allow_dup=1) except: raise Forbidden, sys.exc_info()[1] try: parent._verifyObjectPaste(self) except Unauthorized: raise except: raise Forbidden, sys.exc_info()[1] # Now check locks. The If header on a copy only cares about the # lock on the destination, so we need to check out the destinations # lock status. ifhdr = REQUEST.get_header('If', '') if existing: # The destination itself exists, so we need to check its locks destob = aq_base(parent)._getOb(name) if WriteLockInterface.isImplementedBy(destob) and \ destob.wl_isLocked(): if ifhdr: itrue = destob.dav__simpleifhandler( REQUEST, RESPONSE, 'COPY', refresh=1) if not itrue: raise PreconditonFailed else: raise Locked, 'Destination is locked.' elif WriteLockInterface.isImplementedBy(parent) and \ parent.wl_isLocked(): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, 'COPY', refresh=1) else: raise Locked, 'Destination is locked.' self._notifyOfCopyTo(parent, op=0) ob = self._getCopy(parent) ob._setId(name) if depth=='0' and isDavCollection(ob): for id in ob.objectIds(): ob._delObject(id) if existing: object=getattr(parent, name) self.dav__validate(object, 'DELETE', REQUEST) parent._delObject(name) parent._setObject(name, ob) ob = parent._getOb(name) ob._postCopy(parent, op=0) ob.manage_afterClone(ob) # We remove any locks from the copied object because webdav clients # don't track the lock status and the lock token for copied resources ob.wl_clearLocks() RESPONSE.setStatus(existing and 204 or 201) if not existing: RESPONSE.setHeader('Location', dest) RESPONSE.setBody('') return RESPONSE MOVE__roles__=('Anonymous',) def MOVE(self, REQUEST, RESPONSE): """Move a resource to a new location. Though we may later try to make a move appear seamless across namespaces (e.g. from Zope to Apache), MOVE is currently only supported within the Zope namespace.""" self.dav__init(REQUEST, RESPONSE) self.dav__validate(self, 'DELETE', REQUEST) if not hasattr(aq_base(self), 'cb_isMoveable') or \ not self.cb_isMoveable(): raise MethodNotAllowed, 'This object may not be moved.' dest=REQUEST.get_header('Destination', '') try: path = REQUEST.physicalPathFromURL(dest) except ValueError: raise BadRequest, 'No destination given' flag=REQUEST.get_header('Overwrite', 'F') flag=flag.upper() name = path.pop() parent_path = '/'.join(path) try: parent = self.restrictedTraverse(path) except ValueError: raise Conflict, 'Attempt to move to an unknown namespace.' except 'Not Found': raise Conflict, 'The resource %s must exist.' % parent_path except: t, v, tb=sys.exc_info() raise t, v if hasattr(parent, '__null_resource__'): raise Conflict, 'The resource %s must exist.' % parent_path existing=hasattr(aq_base(parent), name) if existing and flag=='F': raise PreconditionFailed, 'Resource %s exists.' % dest try: parent._checkId(name, allow_dup=1) except: raise Forbidden, sys.exc_info()[1] try: parent._verifyObjectPaste(self) except Unauthorized: raise except: raise Forbidden, sys.exc_info()[1] # Now check locks. Since we're affecting the resource that we're # moving as well as the destination, we have to check both. ifhdr = REQUEST.get_header('If', '') if existing: # The destination itself exists, so we need to check its locks destob = aq_base(parent)._getOb(name) if WriteLockInterface.isImplementedBy(destob) and \ destob.wl_isLocked(): if ifhdr: itrue = destob.dav__simpleifhandler( REQUEST, RESPONSE, 'MOVE', url=dest, refresh=1) if not itrue: raise PreconditionFailed else: raise Locked, 'Destination is locked.' elif WriteLockInterface.isImplementedBy(parent) and \ parent.wl_isLocked(): # There's no existing object in the destination folder, so # we need to check the folders locks since we're changing its # member list if ifhdr: itrue = parent.dav__simpleifhandler(REQUEST, RESPONSE, 'MOVE', col=1, url=dest, refresh=1) if not itrue: raise PreconditionFailed, 'Condition failed.' else: raise Locked, 'Destination is locked.' if Lockable.wl_isLocked(self): # Lastly, we check ourselves if ifhdr: itrue = self.dav__simpleifhandler(REQUEST, RESPONSE, 'MOVE', refresh=1) if not itrue: raise PreconditionFailed, 'Condition failed.' else: raise PreconditionFailed, 'Source is locked and no '\ 'condition was passed in.' # try to make ownership explicit so that it gets carried # along to the new location if needed. self.manage_changeOwnershipType(explicit=1) self._notifyOfCopyTo(parent, op=1) ob = aq_base(self._getCopy(parent)) self.aq_parent._delObject(absattr(self.id)) ob._setId(name) if existing: object=getattr(parent, name) self.dav__validate(object, 'DELETE', REQUEST) parent._delObject(name) parent._setObject(name, ob) ob = parent._getOb(name) ob._postCopy(parent, op=1) # try to make ownership implicit if possible ob.manage_changeOwnershipType(explicit=0) RESPONSE.setStatus(existing and 204 or 201) if not existing: RESPONSE.setHeader('Location', dest) RESPONSE.setBody('') return RESPONSE # WebDAV Class 2, Lock and Unlock def LOCK(self, REQUEST, RESPONSE): """Lock a resource""" self.dav__init(REQUEST, RESPONSE) security = getSecurityManager() creator = security.getUser() body = REQUEST.get('BODY', '') ifhdr = REQUEST.get_header('If', None) depth = REQUEST.get_header('Depth', 'infinity') alreadylocked = Lockable.wl_isLocked(self) if body and alreadylocked: # This is a full LOCK request, and the Resource is # already locked, so we need to raise the alreadylocked # exception. RESPONSE.setStatus(423) elif body: # This is a normal lock request with an XML payload cmd = davcmds.Lock(REQUEST) token, result = cmd.apply(self, creator, depth=depth) if result: # Return the multistatus result (there were multiple # errors. Note that davcmds.Lock.apply aborted the # transaction already. RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) else: # Success lock = self.wl_getLock(token) RESPONSE.setStatus(200) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setHeader('Lock-Token', 'opaquelocktoken:' + token) RESPONSE.setBody(lock.asXML()) else: # There's no body, so this likely to be a refresh request if not ifhdr: raise PreconditionFailed, 'If Header Missing' taglist = IfParser(ifhdr) found = 0 for tag in taglist: for listitem in tag.list: token = tokenFinder(listitem) if token and self.wl_hasLock(token): lock = self.wl_getLock(token) timeout = REQUEST.get_header('Timeout', 'Infinite') lock.setTimeout(timeout) # automatically refreshes found = 1 RESPONSE.setStatus(200) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(lock.asXML()) break if found: break if not found: RESPONSE.setStatus(412) # Precondition failed return RESPONSE def UNLOCK(self, REQUEST, RESPONSE): """Remove an existing lock on a resource.""" self.dav__init(REQUEST, RESPONSE) security = getSecurityManager() token = REQUEST.get_header('Lock-Token', '') url = REQUEST['URL'] token = tokenFinder(token) cmd = davcmds.Unlock() result = cmd.apply(self, token, url) if result: RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) else: RESPONSE.setStatus(204) # No Content response code return RESPONSE def manage_DAVget(self): """Gets the document source""" # The default implementation calls manage_FTPget return self.manage_FTPget() def listDAVObjects(self): return [] Globals.default__class_init__(Resource) Zope-2.8.7-final/lib/python/webdav/hookable_PUT.py0000664000076400007640000000042710435500330020761 0ustar ajungajung# Implement the "hookable PUT" hook. import re, OFS.DTMLMethod TEXT_PATTERN = re.compile( r'^text/.*$' ) def PUT_factory( self, name, typ, body ): """ """ if TEXT_PATTERN.match( typ ): return OFS.DTMLMethod.DTMLMethod( '', __name__=name ) return None Zope-2.8.7-final/lib/python/webdav/xmltools.py0000664000076400007640000001316610435500330020332 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ WebDAV XML request parsing tool using xml.minidom as xml parser. Code contributed by Simon Eisenmann, struktur AG, Stuttgart, Germany """ __version__='$Revision: 1.15.2.1 $'[11:-2] """ TODO: - Check the methods Node.addNode and find out if some code uses/requires this method. => If yes implement them, else forget them. NOTE: So far i didn't have any problems. If you have problems please report them. - We are using a hardcoded default of latin-1 for encoding unicode strings. While this is suboptimal, it does match the expected encoding from OFS.PropertySheet. We need to find a the encoding somehow, maybe use the same encoding as the ZMI is using? """ from StringIO import StringIO from xml.dom import minidom from xml.sax.saxutils import escape as _escape, unescape as _unescape escape_entities = {'"': '"', "'": ''', } unescape_entities = {'"': '"', ''': "'", } def escape(value, entities=None): _ent = escape_entities if entities is not None: _ent = _ent.copy() _ent.update(entities) return _escape(value, entities) def unescape(value, entities=None): _ent = unescape_entities if entities is not None: _ent = _ent.copy() _ent.update(entities) return _unescape(value, entities) # XXX latin-1 is hardcoded on OFS.PropertySheets as the expected # encoding properties will be stored in. Optimally, we should use the # same encoding as the 'default_encoding' property that is used for # the ZMI. zope_encoding = 'latin-1' class Node: """ Our nodes no matter what type """ node = None def __init__(self, node): self.node = node def elements(self, name=None, ns=None): nodes = [] for n in self.node.childNodes: if (n.nodeType == n.ELEMENT_NODE and ((name is None) or ((n.localName.lower())==name)) and ((ns is None) or (n.namespaceURI==ns))): nodes.append(Element(n)) return nodes def qname(self): return '%s%s' % (self.namespace(), self.name()) def addNode(self, node): # XXX: no support for adding nodes here raise NotImplementedError, 'addNode not implemented' def toxml(self): return self.node.toxml() def strval(self): return self.toxml().encode(zope_encoding) def name(self): return self.node.localName def value(self): return self.node.nodeValue def nodes(self): return self.node.childNodes def nskey(self): return self.node.namespaceURI def namespace(self): return self.nskey() def attrs(self): return [Node(n) for n in self.node.attributes.values()] def del_attr(self, name): # NOTE: zope calls this after remapping to remove namespace # zope passes attributes like xmlns:n # but the :n isnt part of the attribute name .. gash! attr = name.split(':')[0] if (self.node.hasAttributes() and self.node.attributes.has_key(attr)): # Only remove attributes if they exist return self.node.removeAttribute(attr) def remap(self, dict, n=0, top=1): # XXX: this method is used to do some strange remapping of elements # and namespaces .. someone wants to explain that code? # XXX: i also dont understand why this method returns anything # as the return value is never used # NOTE: zope calls this to change namespaces in PropPatch and Lock # we dont need any fancy remapping here and simply remove # the attributes in del_attr return {},0 def __repr__(self): if self.namespace(): return "" % (self.name(), self.namespace()) else: return "" % self.name() class Element(Node): def toxml(self): # When dealing with Elements, we only want the Element's content. writer = StringIO(u'') for n in self.node.childNodes: if n.nodeType == n.CDATA_SECTION_NODE: # CDATA sections should not be unescaped. writer.write(n.data) elif n.nodeType == n.ELEMENT_NODE: writer.write(n.toxml()) else: # TEXT_NODE and what else? value = n.toxml() # Unescape possibly escaped values. We do this # because the value is *always* escaped in it's XML # representation, and if we store it escaped it will come # out *double escaped* when doing a PROPFIND. value = unescape(value, entities=unescape_entities) writer.write(value) return writer.getvalue() class XmlParser: """ Simple wrapper around minidom to support the required interfaces for zope.webdav """ dom = None def __init__(self): pass def parse(self, data): self.dom = minidom.parseString(data) return Node(self.dom) Zope-2.8.7-final/lib/python/webdav/WriteLockInterface.py0000664000076400007640000001674710435500330022205 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Write lock interfaces. $Id: WriteLockInterface.py 30356 2005-05-15 15:56:18Z yuppie $ """ from Interface import Interface class LockItemInterface(Interface): """A LockItem contains information about a lock. This includes: o The locktoken uri (used to identify the lock by WebDAV) o The lock owner (The string passed in the 'owner' property by WebDAV) o The lock creator (the Zope user who physically owns the lock) o Depth o Timeout information o Modified time (for calculating timeouts) o LockType (only EXCLUSIVE is supported right now) """ # XXX: WAAAA! What is a ctor doing in the interface? def __init__(creator, owner, depth=0, timeout='Infinity', locktype='write', lockscope='exclusive', token=None): """\ If any of the following are untrue, a **ValueError** exception will be raised. - **creator** MUST be a Zope user object or string to find a valid user object. - **owner** MUST be a nonempty string, or type that can be converted to a nonempty string. - **depth** MUST be in the set {0,'infinity'} - **timeout** MUST either be an integer, or a string in the form of 'Seconds-nnn' where nnn is an integer. The timeout value MUST be less than (2^32)-1. *IF* timeout is the string value 'Infinite', the timeout value will be set to 1800 (30 minutes). (Timeout is the value in seconds from creation\modification time until the lock MAY time out). - **locktype** not in set {'write'} *this may expand later* - **lockscope** not in set {'exclusive'} *this may expand later* If the value passed in to 'token' is 'None', the a new locktoken will be generated during the construction process. __init__ must generate the opaquelocktoken uri used to identify the lock (if 'token' is 'None')and set all of the above attributes on the object. """ def getCreator(): """ Returns the Zope user who created the lock. This is returned in a tuple containing the Users ID and the path to the user folder they came from.""" def getCreatorPath(): """ Returns a string of the path to the user object in the user folder they were found in. """ def getOwner(): """ Returns the string value of the 'owner' property sent in by WebDAV """ def getLockToken(): """ returns the opaque lock token """ def getDepth(): """ returns the depth of the lock """ def getTimeout(): """ returns an integer value of the timeout setting """ def getTimeoutString(): """ returns the timeout value in a form acceptable by WebDAV (ie - 'Seconds-40800') """ def setTimeout(newtimeout): """ refreshes the timeout information """ def getModifiedTime(): """ returns a time.time value of the last time the Lock was modified. From RFC 2518: The timeout counter SHOULD be restarted any time an owner of the lock sends a method to any member of the lock, including unsupported methods or methods which are unsucscessful. The lock MUST be refreshed if a refresh LOCK method is successfully received. The modified time is used to calculate the refreshed value """ def refresh(): """ Tickles the locks modified time by setting it to the current time.time() value. (As stated in the RFC, the timeout counter SHOULD be restarted for any HTTP method called by the lock owner on the locked object). """ def isValid(): """ Returns true if (self.getModifiedTime() + self.getTimeout()) is greater than the current time.time() value. """ # now = time.time() # modified = self.getModifiedTime() # timeout = self.getTimeout() # # return (modified + timeout > now) # there's time remaining def getLockType(): """ returns the lock type ('write') """ def getLockScope(): """ returns the lock scope ('exclusive') """ def asLockDiscoveryProperty(ns='d'): """ Return the lock rendered as an XML representation of a WebDAV 'lockdiscovery' property. 'ns' is the namespace identifier used on the XML elements.""" def asXML(): """ Render a full XML representation of a lock for WebDAV, used when returning the value of a newly created lock. """ class WriteLockInterface(Interface): """Basic protocol needed to support the write lock machinery. It must be able to answer the questions: o Is the object locked? o Is the lock owned by the current user? o What lock tokens are associated with the current object? o What is their state (how long until they're supposed to time out?, what is their depth? what type are they? And it must be able to do the following: o Grant a write lock on the object to a specified user. - *If lock depth is infinite, this must also grant locks on **all** subobjects, or fail altogether* o Revoke a lock on the object. - *If lock depth is infinite, this must also revoke locks on all subobjects* **All methods in the WriteLock interface that deal with checking valid locks MUST check the timeout values on the lockitem (ie, by calling 'lockitem.isValid()'), and DELETE the lock if it is no longer valid** """ def wl_lockItems(killinvalids=0): """ Returns (key, value) pairs of locktoken, lock. if 'killinvalids' is true, invalid locks (locks whose timeout has been exceeded) will be deleted""" def wl_lockValues(killinvalids=0): """ Returns a sequence of locks. if 'killinvalids' is true, invalid locks will be deleted""" def wl_lockTokens(killinvalids=0): """ Returns a sequence of lock tokens. if 'killinvalids' is true, invalid locks will be deleted""" def wl_hasLock(token, killinvalids=0): """ Returns true if the lock identified by the token is attached to the object. """ def wl_isLocked(): """ Returns true if 'self' is locked at all. If invalid locks still exist, they should be deleted.""" def wl_setLock(locktoken, lock): """ Store the LockItem, 'lock'. The locktoken will be used to fetch and delete the lock. If the lock exists, this MUST overwrite it if all of the values except for the 'timeout' on the old and new lock are the same. """ def wl_getLock(locktoken): """ Returns the locktoken identified by the locktokenuri """ def wl_delLock(locktoken): """ Deletes the locktoken identified by the locktokenuri """ def wl_clearLocks(): """ Deletes ALL DAV locks on the object - should only be called by lock management machinery. """ Zope-2.8.7-final/lib/python/webdav/__init__.py0000664000076400007640000000356410435500330020211 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """The webdav package provides WebDAV capability for common Zope objects. Current WebDAV support in Zope provides for the correct handling of HTTP GET, HEAD, POST, PUT, DELETE, OPTIONS, TRACE, PROPFIND, PROPPATCH, MKCOL, COPY and MOVE methods, as appropriate for the object that is the target of the operation. Objects which do not support a given operation should respond appropriately with a "405 Method Not Allowed" response. Note that the ability of a Zope installation to support WebDAV HTTP methods depends on the willingness of the web server to defer handling of those methods to the Zope process. In most cases, servers will allow the process to handle any request, so the Zope portion of your url namespace may well be able to handle WebDAV operations even though your web server software is not WebDAV-aware itself. Zope installations which use bundled server implementations such as ZopeHTTPServer or ZServer should fully support WebDAV functions. References: [WebDAV] Y. Y. Goland, E. J. Whitehead, Jr., A. Faizi, S. R. Carter, D. Jensen, "HTTP Extensions for Distributed Authoring - WebDAV." RFC 2518. Microsoft, U.C. Irvine, Netscape, Novell. February, 1999.""" __version__='$Revision: 1.7 $'[11:-2] Zope-2.8.7-final/lib/python/webdav/LockItem.py0000664000076400007640000001370710435500330020161 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__ = "$Revision: 1.7 $"[11:-2] from Globals import Persistent from WriteLockInterface import LockItemInterface from AccessControl import ClassSecurityInfo from AccessControl.Owned import ownerInfo from common import generateLockToken import time MAXTIMEOUT = (2L**32)-1 # Maximum timeout time DEFAULTTIMEOUT = 12 * 60L # Default timeout def validateTimeout(timeout): # Timeout *should* be in the form "Seconds-XXX" or "Infinite" errors = [] try: t =str(timeout).split('-')[-1] if t.lower() == 'infinite': timeout = DEFAULTTIMEOUT # Default to 1800 secods for infinite else: # requests timeout = long(t) except ValueError: errors.append("Bad timeout value") if timeout > MAXTIMEOUT: errors.append("Timeout request is greater than %s" % MAXTIMEOUT) return timeout, errors class LockItem(Persistent): __implements__ = (LockItemInterface,) # Use the Zope 2.3 declarative security to manage access security = ClassSecurityInfo() security.declarePublic('getOwner', 'getLockToken', 'getDepth', 'getTimeout', 'getTimeoutString', 'getModifiedTime', 'isValid', 'getLockScope', 'getLockType') security.declareProtected('Change Lock Information', 'setTimeout', 'refresh') security.declareProtected('Access contents information', 'getCreator', 'getCreatorPath') def __init__(self, creator, owner='', depth=0, timeout='Infinite', locktype='write', lockscope='exclusive', token=None): errors = [] # First check the values and raise value errors if outside of contract if not getattr(creator, 'getUserName', None): errors.append("Creator not a user object") if str(depth).lower() not in ('0', 'infinity'): errors.append("Depth must be 0 or infinity") if locktype.lower() != 'write': errors.append("Lock type '%s' not supported" % locktype) if lockscope.lower() != 'exclusive': errors.append("Lock scope '%s' not supported" % lockscope) timeout, e = validateTimeout(timeout) errors = errors + e # Finally, if there were errors, report them ALL to on high if errors: raise ValueError, errors # AccessControl.Owned.ownerInfo returns the id of the creator # and the path to the UserFolder they're defined in self._creator = ownerInfo(creator) self._owner = owner self._depth = depth self._timeout = timeout self._locktype = locktype self._lockscope = lockscope self._modifiedtime = time.time() if token is None: self._token = generateLockToken() else: self._token = token def getCreator(self): return self._creator def getCreatorPath(self): db, name = self._creator path = '/'.join(db) return "/%s/%s" % (path, name) def getOwner(self): return self._owner def getLockToken(self): return self._token def getDepth(self): return self._depth def getTimeout(self): return self._timeout def getTimeoutString(self): t = str(self._timeout) if t[-1] == 'L': t = t[:-1] # lob off Long signifier return "Second-%s" % t def setTimeout(self, newtimeout): timeout, errors = validateTimeout(newtimeout) if errors: raise ValueError, errors else: self._timeout = timeout self._modifiedtime = time.time() # reset modified def getModifiedTime(self): return self._modifiedtime def refresh(self): self._modifiedtime = time.time() def isValid(self): now = time.time() modified = self._modifiedtime timeout = self._timeout return (modified + timeout) > now def getLockType(self): return self._locktype def getLockScope(self): return self._lockscope def asLockDiscoveryProperty(self, ns='d',fake=0): if fake: token = 'this-is-a-faked-no-permission-token' else: token = self._token s = (' <%(ns)s:activelock>\n' ' <%(ns)s:locktype><%(ns)s:%(locktype)s/>\n' ' <%(ns)s:lockscope><%(ns)s:%(lockscope)s/>\n' ' <%(ns)s:depth>%(depth)s\n' ' <%(ns)s:owner>%(owner)s\n' ' <%(ns)s:timeout>%(timeout)s\n' ' <%(ns)s:locktoken>\n' ' <%(ns)s:href>opaquelocktoken:%(locktoken)s\n' ' \n' ' \n' ) % { 'ns': ns, 'locktype': self._locktype, 'lockscope': self._lockscope, 'depth': self._depth, 'owner': self._owner, 'timeout': self.getTimeoutString(), 'locktoken': token, } return s def asXML(self): s = """ %s """ % self.asLockDiscoveryProperty(ns="d") return s Zope-2.8.7-final/lib/python/webdav/dtml/0000775000076400007640000000000010435500330017030 5ustar ajungajungZope-2.8.7-final/lib/python/webdav/dtml/locknullmain.dtml0000664000076400007640000000076310435500330022410 0ustar ajungajung

This item is locked by WebDAV as a Lock-Null Resource. A lock-null resource is created when a resource is locked before it is fully created, basically reserving its name. When the owner of this resource issues a command to either fill it with content, or turn it into a collection (folder), that object will replace this one.

Zope-2.8.7-final/lib/python/webdav/Lockable.py0000664000076400007640000001273210435500330020163 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """WebDAV support - lockable item. $Id: Lockable.py 40281 2005-11-21 00:08:00Z efge $ """ from AccessControl import ClassSecurityInfo from Globals import InitializeClass from Globals import PersistentMapping import Acquisition from EtagSupport import EtagSupport from WriteLockInterface import LockItemInterface from WriteLockInterface import WriteLockInterface class ResourceLockedError(Exception): pass class LockableItem(EtagSupport): """\ Implements the WriteLockInterface, and is inherited by Resource which is then inherited by the majority of Zope objects. For an object to be lockable, however, it should have the WriteLockInterface in its __implements__ list, ie: __implements__ = (WriteLockInterface,) """ # Protect methods using declarative security security = ClassSecurityInfo() security.declarePrivate('wl_lockmapping') security.declarePublic('wl_isLocked', 'wl_getLock', 'wl_isLockedByUser', 'wl_lockItems', 'wl_lockValues', 'wl_lockTokens',) security.declareProtected('WebDAV Lock items', 'wl_setLock') security.declareProtected('WebDAV Unlock items', 'wl_delLock') security.declareProtected('Manage WebDAV Locks', 'wl_clearLocks') # Setting default roles for permissions - we want owners of conent # to be able to lock. security.setPermissionDefault('WebDAV Lock items', ('Manager', 'Owner',)) security.setPermissionDefault('WebDAV Unlock items',('Manager','Owner',)) def wl_lockmapping(self, killinvalids=0, create=0): """ if 'killinvalids' is 1, locks who are no longer valid will be deleted """ try: locks = getattr(self, '_dav_writelocks', None) except: locks = None if locks is None: if create: locks = self._dav_writelocks = PersistentMapping() else: # Don't generate a side effect transaction. locks = {} return locks elif killinvalids: # Delete invalid locks for token, lock in locks.items(): if not lock.isValid(): del locks[token] if (not locks) and hasattr(Acquisition.aq_base(self), '__no_valid_write_locks__'): self.__no_valid_write_locks__() return locks else: return locks def wl_lockItems(self, killinvalids=0): return self.wl_lockmapping(killinvalids).items() def wl_lockValues(self, killinvalids=0): return self.wl_lockmapping(killinvalids).values() def wl_lockTokens(self, killinvalids=0): return self.wl_lockmapping(killinvalids).keys() def wl_hasLock(self, token, killinvalids=0): if not token: return 0 return token in self.wl_lockmapping(killinvalids).keys() def wl_isLocked(self): # returns true if 'self' is locked at all # We set 'killinvalids' to 1 to delete all locks who are no longer # valid (timeout has been exceeded) locks = self.wl_lockmapping(killinvalids=1) if locks.keys(): return 1 else: return 0 def wl_setLock(self, locktoken, lock): locks = self.wl_lockmapping(create=1) if LockItemInterface.isImplementedBy(lock): if locktoken == lock.getLockToken(): locks[locktoken] = lock else: raise ValueError, 'Lock tokens do not match' else: raise ValueError, 'Lock does not implement the LockItem Interface' def wl_getLock(self, locktoken): locks = self.wl_lockmapping(killinvalids=1) return locks.get(locktoken, None) def wl_delLock(self, locktoken): locks = self.wl_lockmapping() if locks.has_key(locktoken): del locks[locktoken] def wl_clearLocks(self): # Called by lock management machinery to quickly and effectively # destroy all locks. try: locks = self.wl_lockmapping() locks.clear() except: # The locks may be totally messed up, so we'll just delete # and replace. if hasattr(self, '_dav_writelocks'): del self._dav_writelocks if WriteLockInterface.isImplementedBy(self): self._dav_writelocks = PersistentMapping() # Call into a special hook used by LockNullResources to delete # themselves. Could be used by other objects who want to deal # with the state of empty locks. if hasattr(Acquisition.aq_base(self), '__no_valid_write_locks__'): self.__no_valid_write_locks__() InitializeClass(LockableItem) ### Utility functions def wl_isLocked(ob): """ Returns true if the object is locked, returns 0 if the object is not locked or does not implement the WriteLockInterface """ return WriteLockInterface.isImplementedBy(ob) and ob.wl_isLocked() Zope-2.8.7-final/lib/python/webdav/Collection.py0000664000076400007640000001317010435500330020537 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """WebDAV support - collection objects. $Id: Collection.py 30356 2005-05-15 15:56:18Z yuppie $ """ import Globals, davcmds, Lockable from common import urlfix, rfc1123_date from Resource import Resource from AccessControl import getSecurityManager from urllib import unquote from zExceptions import MethodNotAllowed, NotFound from webdav.common import Locked, PreconditionFailed class Collection(Resource): """The Collection class provides basic WebDAV support for collection objects. It provides default implementations for all supported WebDAV HTTP methods. The behaviors of some WebDAV HTTP methods for collections are slightly different than those for non-collection resources.""" __dav_collection__=1 def dav__init(self, request, response): # We are allowed to accept a url w/o a trailing slash # for a collection, but are supposed to provide a # hint to the client that it should be using one. # [WebDAV, 5.2] pathinfo=request.get('PATH_INFO','') if pathinfo and pathinfo[-1] != '/': location='%s/' % request['URL1'] response.setHeader('Content-Location', location) response.setHeader('Connection', 'close', 1) response.setHeader('Date', rfc1123_date(), 1) def HEAD(self, REQUEST, RESPONSE): """Retrieve resource information without a response body.""" self.dav__init(REQUEST, RESPONSE) # Note that we are willing to acquire the default document # here because what we really care about is whether doing # a GET on this collection / would yield a 200 response. if hasattr(self, 'index_html'): if hasattr(self.index_html, 'HEAD'): return self.index_html.HEAD(REQUEST, RESPONSE) raise MethodNotAllowed, ( 'Method not supported for this resource.' ) raise NotFound, 'The requested resource does not exist.' def PUT(self, REQUEST, RESPONSE): """The PUT method has no inherent meaning for collection resources, though collections are not specifically forbidden to handle PUT requests. The default response to a PUT request for collections is 405 (Method Not Allowed).""" self.dav__init(REQUEST, RESPONSE) raise MethodNotAllowed, 'Method not supported for collections.' def DELETE(self, REQUEST, RESPONSE): """Delete a collection resource. For collection resources, DELETE may return either 200 (OK) or 204 (No Content) to indicate total success, or may return 207 (Multistatus) to indicate partial success. Note that in Zope a DELETE currently never returns 207.""" self.dav__init(REQUEST, RESPONSE) ifhdr = REQUEST.get_header('If', '') url = urlfix(REQUEST['URL'], 'DELETE') name = unquote(filter(None, url.split( '/'))[-1]) parent = self.aq_parent user = getSecurityManager().getUser() token = None # if re.match("/Control_Panel",REQUEST['PATH_INFO']): # RESPONSE.setStatus(403) # RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') # return RESPONSE # Level 1 of lock checking (is the collection or its parent locked?) if Lockable.wl_isLocked(self): if ifhdr: self.dav__simpleifhandler(REQUEST, RESPONSE, 'DELETE', col=1) else: raise Locked elif Lockable.wl_isLocked(parent): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, 'DELETE', col=1) else: raise PreconditionFailed # Second level of lock\conflict checking (are any descendants locked, # or is the user not permitted to delete?). This results in a # multistatus response if ifhdr: tokens = self.wl_lockTokens() for tok in tokens: # We already know that the simple if handler succeeded, # we just want to get the right token out of the header now if ifhdr.find(tok) > -1: token = tok cmd = davcmds.DeleteCollection() result = cmd.apply(self, token, user, REQUEST['URL']) if result: # There were conflicts, so we need to report them RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) else: # There were no conflicts, so we can go ahead and delete # ajung: additional check if we really could delete the collection # (Collector #2196) if parent.manage_delObjects([name],REQUEST=None) is None: RESPONSE.setStatus(204) else: RESPONSE.setStatus(403) return RESPONSE def listDAVObjects(self): objectValues = getattr(self, 'objectValues', None) if objectValues is not None: return objectValues() return [] Globals.default__class_init__(Collection) Zope-2.8.7-final/lib/python/webdav/client.py0000664000076400007640000004657310435500330017737 0ustar ajungajung"""HTTP 1.1 / WebDAV client library.""" __version__='$Revision: 1.22 $'[11:-2] import sys, os, time, types,re import httplib, mimetools from types import FileType from mimetypes import guess_type from base64 import encodestring from common import rfc1123_date from cStringIO import StringIO from random import random from urllib import quote class NotAvailable(Exception): pass class HTTP(httplib.HTTP): # A revised version of the HTTP class that can do basic # HTTP 1.1 connections, and also compensates for a bug # that occurs on some platforms in 1.5 and 1.5.1 with # socket.makefile().read() read_bug=sys.version[:5] in ('1.5 (','1.5.1') def putrequest(self, request, selector, ver='1.1'): selector=selector or '/' str = '%s %s HTTP/%s\r\n' % (request, selector, ver) self.send(str) def getreply(self): file=self.sock.makefile('rb') data=''.join(file.readlines()) file.close() self.file=StringIO(data) line = self.file.readline() try: [ver, code, msg] = line.split( None, 2) except ValueError: try: [ver, code] = line.split( None, 1) msg = "" except ValueError: return -1, line, None if ver[:5] != 'HTTP/': return -1, line, None code=int(code) msg =msg.strip() headers =mimetools.Message(self.file, 0) return ver, code, msg, headers class Resource: """An object representing a web resource.""" def __init__(self, url, username=None, password=None): self.username=username self.password=password self.url=url mo = urlreg.match(url) if mo: host,port,uri=mo.group(1,2,3) self.host=host self.port=port and int(port[1:]) or 80 self.uri=uri or '/' else: raise ValueError, url def __getattr__(self, name): url=os.path.join(self.url, name) return self.__class__(url, username=self.username, password=self.password) def __get_headers(self, kw={}): headers={} headers=self.__set_authtoken(headers) headers['User-Agent']='WebDAV.client %s' % __version__ headers['Host']=self.host headers['Connection']='close' headers['Accept']='*/*' if kw.has_key('headers'): for name, val in kw['headers'].items(): headers[name]=val del kw['headers'] return headers def __set_authtoken(self, headers, atype='Basic'): if not (self.username and self.password): return headers if headers.has_key('Authorization'): return headers if atype=='Basic': headers['Authorization']=( "Basic %s" % (encodestring('%s:%s' % (self.username,self.password))).replace( '\012','')) return headers raise ValueError, 'Unknown authentication scheme: %s' % atype def __enc_formdata(self, args={}): formdata=[] for key, val in args.items(): n=key.rfind( '__') if n > 0: tag=key[n+2:] key=key[:n] else: tag='string' func=varfuncs.get(tag, marshal_string) formdata.append(func(key, val)) return '&'.join(formdata) def __enc_multipart(self, args={}): return MultiPart(args).render() def __snd_request(self, method, uri, headers={}, body='', eh=1): try: h=HTTP() h.connect(self.host, self.port) h.putrequest(method, uri) for n, v in headers.items(): h.putheader(n, v) if eh: h.endheaders() if body: h.send(body) ver, code, msg, hdrs=h.getreply() data=h.getfile().read() h.close() except: raise NotAvailable, sys.exc_value return http_response(ver, code, msg, hdrs, data) # HTTP methods def get(self, **kw): headers=self.__get_headers(kw) query=self.__enc_formdata(kw) uri=query and '%s?%s' % (self.uri, query) or self.uri return self.__snd_request('GET', uri, headers) def head(self, **kw): headers=self.__get_headers(kw) query=self.__enc_formdata(kw) uri=query and '%s?%s' % (self.uri, query) or self.uri return self.__snd_request('HEAD', uri, headers) def post(self, **kw): headers=self.__get_headers(kw) content_type=None for key, val in kw.items(): if (key[-6:]=='__file') or hasattr(val, 'read'): content_type='multipart/form-data' break if content_type=='multipart/form-data': body=self.__enc_multipart(kw) return self.__snd_request('POST', self.uri, headers, body, eh=0) else: body=self.__enc_formdata(kw) headers['Content-Type']='application/x-www-form-urlencoded' headers['Content-Length']=str(len(body)) return self.__snd_request('POST', self.uri, headers, body) def put(self, file='', content_type='', content_enc='', isbin=re.compile(r'[\000-\006\177-\277]').search, **kw): headers=self.__get_headers(kw) filetype=type(file) if filetype is type('') and (isbin(file) is None) and \ os.path.exists(file): ob=open(file, 'rb') body=ob.read() ob.close() c_type, c_enc=guess_type(file) elif filetype is FileType: body=file.read() c_type, c_enc=guess_type(file.name) elif filetype is type(''): body=file c_type, c_enc=guess_type(self.url) else: raise ValueError, 'File must be a filename, file or string.' content_type=content_type or c_type content_enc =content_enc or c_enc if content_type: headers['Content-Type']=content_type if content_enc: headers['Content-Encoding']=content_enc headers['Content-Length']=str(len(body)) return self.__snd_request('PUT', self.uri, headers, body) def options(self, **kw): headers=self.__get_headers(kw) return self.__snd_request('OPTIONS', self.uri, headers) def trace(self, **kw): headers=self.__get_headers(kw) return self.__snd_request('TRACE', self.uri, headers) def delete(self, **kw): headers=self.__get_headers(kw) return self.__snd_request('DELETE', self.uri, headers) def propfind(self, body='', depth=0, **kw): headers=self.__get_headers(kw) headers['Depth']=str(depth) headers['Content-Type']='text/xml; charset="utf-8"' headers['Content-Length']=str(len(body)) return self.__snd_request('PROPFIND', self.uri, headers, body) def proppatch(self, body, **kw): headers=self.__get_headers(kw) if body: headers['Content-Type']='text/xml; charset="utf-8"' headers['Content-Length']=str(len(body)) return self.__snd_request('PROPPATCH', self.uri, headers, body) def copy(self, dest, depth='infinity', overwrite=0, **kw): """Copy a resource to the specified destination.""" headers=self.__get_headers(kw) headers['Overwrite']=overwrite and 'T' or 'F' headers['Destination']=dest headers['Depth']=depth return self.__snd_request('COPY', self.uri, headers) def move(self, dest, depth='infinity', overwrite=0, **kw): """Move a resource to the specified destination.""" headers=self.__get_headers(kw) headers['Overwrite']=overwrite and 'T' or 'F' headers['Destination']=dest headers['Depth']=depth return self.__snd_request('MOVE', self.uri, headers) def mkcol(self, **kw): headers=self.__get_headers(kw) return self.__snd_request('MKCOL', self.uri, headers) # class 2 support def lock(self, scope='exclusive', type='write', owner='', depth='infinity', timeout='Infinite', **kw): """Create a lock with the specified scope, type, owner, depth and timeout on the resource. A locked resource prevents a principal without the lock from executing a PUT, POST, PROPPATCH, LOCK, UNLOCK, MOVE, DELETE, or MKCOL on the locked resource.""" if not scope in ('shared', 'exclusive'): raise ValueError, 'Invalid lock scope.' if not type in ('write',): raise ValueError, 'Invalid lock type.' if not depth in ('0', 'infinity'): raise ValueError, 'Invalid depth.' headers=self.__get_headers(kw) body='\n' \ '\n' \ ' \n' \ ' \n' \ ' %s\n' \ ' \n' \ ' %s\n' \ ' \n' \ '' % (scope, type, depth, owner) headers['Content-Type']='text/xml; charset="utf-8"' headers['Content-Length']=str(len(body)) headers['Timeout']=timeout headers['Depth']=depth return self.__snd_request('LOCK', self.uri, headers, body) def unlock(self, token, **kw): """Remove the lock identified by token from the resource and all other resources included in the lock. If all resources which have been locked under the submitted lock token can not be unlocked the unlock method will fail.""" headers=self.__get_headers(kw) token='' % str(token) headers['Lock-Token']=token return self.__snd_request('UNLOCK', self.uri, headers) def allprops(self, depth=0): return self.propfind('', depth) def propnames(self, depth=0): body='\n' \ '\n' \ ' \n' \ '' return self.propfind(body, depth) def getprops(self, *names): if not names: return self.propfind() tags='/>\n <'.join(names ) body='\n' \ '\n' \ ' \n' \ ' <%s>\n' \ ' \n' \ '' % tags return self.propfind(body, 0) def setprops(self, **props): if not props: raise ValueError, 'No properties specified.' tags=[] for key, val in props.items(): tags.append(' <%s>%s' % (key, val, key)) tags='\n'.join(tags ) body='\n' \ '\n' \ '\n' \ ' \n' \ ' %s\n' \ ' \n' \ '\n' \ '' % tags return self.proppatch(body) def delprops(self, *names): if not names: raise ValueError, 'No property names specified.' tags='/>\n <'.join(names) body='\n' \ '\n' \ '\n' \ ' \n' \ ' <%s>\n' \ ' \n' \ '\n' \ '' % tags return self.proppatch(body) def __str__(self): return '' % self.url __repr__=__str__ class http_response: def __init__(self, ver, code, msg, headers, body): self.version=ver self.code=code self.msg=msg self.headers=headers self.body=body def get_status(self): return '%s %s' % (self.code, self.msg) def get_header(self, name, val=None): return self.headers.dict.get(name.lower(), val) def get_headers(self): return self.headers.dict def get_body(self): return self.body def __str__(self): data=[] data.append('%s %s %s\r\n' % (self.version, self.code, self.msg)) map(data.append, self.headers.headers) data.append('\r\n') data.append(self.body) return ''.join(data) set_xml=""" Brian Lloyd My New Title """ funny=""" Brian Lloyd blue 72 Brian Lloyd red """ rem_xml=""" """ find_xml=""" """ ############################################################################## # Implementation details below here urlreg=re.compile(r'http://([^:/]+)(:[0-9]+)?(/.+)?', re.I) def marshal_string(name, val): return '%s=%s' % (name, quote(str(val))) def marshal_float(name, val): return '%s:float=%s' % (name, val) def marshal_int(name, val): return '%s:int=%s' % (name, val) def marshal_long(name, val): value = '%s:long=%s' % (name, val) if value[-1] == 'L': value = value[:-1] return value def marshal_list(name, seq, tname='list', lt=type([]), tt=type(())): result=[] for v in seq: tp=type(v) if tp in (lt, tt): raise TypeError, 'Invalid recursion in data to be marshaled.' result.append(marshal_var("%s:%s" % (name, tname), v)) return '&'.join(result) def marshal_tuple(name, seq): return marshal_list(name, seq, 'tuple') varfuncs={} vartypes=(('int', type(1), marshal_int), ('float', type(1.0), marshal_float), ('long', type(1L), marshal_long), ('list', type([]), marshal_list), ('tuple', type(()), marshal_tuple), ('string', type(''), marshal_string), ('file', types.FileType, None), ) for name, tp, func in vartypes: varfuncs[name]=func varfuncs[tp]=func def marshal_var(name, val): return varfuncs.get(type(val), marshal_string)(name, val) class MultiPart: def __init__(self,*args): c=len(args) if c==1: name,val=None,args[0] elif c==2: name,val=args[0],args[1] else: raise ValueError, 'Invalid arguments' h={'Content-Type': {'_v':''}, 'Content-Transfer-Encoding': {'_v':''}, 'Content-Disposition': {'_v':''}, } dt=type(val) b=t=None if dt==types.DictType: t=1 b=self.boundary() d=[] h['Content-Type']['_v']='multipart/form-data; boundary=%s' % b for n,v in val.items(): d.append(MultiPart(n,v)) elif (dt==types.ListType) or (dt==types.TupleType): raise ValueError, 'Sorry, nested multipart is not done yet!' elif dt==types.FileType or hasattr(val,'read'): if hasattr(val,'name'): ct, enc=guess_type(val.name) if not ct: ct='application/octet-stream' fn=val.name.replace('\\','/') fn=fn[(fn.rfind('/')+1):] else: ct='application/octet-stream' enc='' fn='' enc=enc or (ct[:6] in ('image/', 'applic') and 'binary' or '') h['Content-Disposition']['_v'] ='form-data' h['Content-Disposition']['name'] ='"%s"' % name h['Content-Disposition']['filename']='"%s"' % fn h['Content-Transfer-Encoding']['_v']=enc h['Content-Type']['_v'] =ct d=[] l=val.read(8192) while l: d.append(l) l=val.read(8192) else: n=name.rfind( '__') if n > 0: name='%s:%s' % (name[:n], name[n+2:]) h['Content-Disposition']['_v']='form-data' h['Content-Disposition']['name']='"%s"' % name d=[str(val)] self._headers =h self._data =d self._boundary=b self._top =t def boundary(self): return '%s_%s_%s' % (int(time.time()), os.getpid(), random()) def render(self): h=self._headers s=[] if self._top: for n,v in h.items(): if v['_v']: s.append('%s: %s' % (n,v['_v'])) for k in v.keys(): if k != '_v': s.append('; %s=%s' % (k, v[k])) s.append('\n') p=[] t=[] b=self._boundary for d in self._data: p.append(d.render()) t.append('--%s\n' % b) t.append(('\n--%s\n' % b).join(p)) t.append('\n--%s--\n' % b) t=''.join(t) s.append('Content-Length: %s\n\n' % len(t)) s.append(t) return ''.join(s) else: for n,v in h.items(): if v['_v']: s.append('%s: %s' % (n,v['_v'])) for k in v.keys(): if k != '_v': s.append('; %s=%s' % (k, v[k])) s.append('\n') s.append('\n') if self._boundary: p=[] b=self._boundary for d in self._data: p.append(d.render()) s.append('--%s\n' % b) s.append(('\n--%s\n' % b).join(p)) s.append('\n--%s--\n' % b) return ''.join(s) else: return ''.join(s+self._data) _extmap={'': 'text/plain', 'rdb': 'text/plain', 'html': 'text/html', 'dtml': 'text/html', 'htm': 'text/html', 'dtm': 'text/html', 'gif': 'image/gif', 'jpg': 'image/jpeg', 'exe': 'application/octet-stream', None : 'application/octet-stream', } _encmap={'image/gif': 'binary', 'image/jpg': 'binary', 'application/octet-stream': 'binary', } bri =Resource('http://tarzan.digicool.com/dev/brian3/', username='brian', password='123') abri=Resource('http://tarzan.digicool.com/dev/brian3/') dav =Resource('http://tarzan.digicool.com/dev/dav/', username='brian', password='123') adav=Resource('http://tarzan.digicool.com/dev/dav/') Zope-2.8.7-final/lib/python/webdav/NullResource.py0000664000076400007640000004445210435500330021075 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """WebDAV support - null resource objects.""" __version__='$Revision: 1.43 $'[11:-2] import sys, Globals, davcmds import Acquisition, OFS.content_types from common import aq_base, tokenFinder, IfParser from AccessControl import getSecurityManager from Resource import Resource from Globals import Persistent, DTMLFile from WriteLockInterface import WriteLockInterface import OFS.SimpleItem from zExceptions import Unauthorized, NotFound, Forbidden, BadRequest from zExceptions import MethodNotAllowed from common import isDavCollection from common import Locked, Conflict, PreconditionFailed, UnsupportedMediaType from OFS.CopySupport import CopyError class NullResource(Persistent, Acquisition.Implicit, Resource): """Null resources are used to handle HTTP method calls on objects which do not yet exist in the url namespace.""" __implements__ = (WriteLockInterface,) __null_resource__=1 __ac_permissions__=( ('View', ('HEAD',)), ('Add Folders', ('MKCOL',)), ('WebDAV Lock items', ('LOCK',)), ) def __init__(self, parent, name, request=None): self.__name__=name self.__parent__=parent def __bobo_traverse__(self, REQUEST, name=None): # We must handle traversal so that we can recognize situations # where a 409 Conflict must be returned instead of the normal # 404 Not Found, per [WebDAV 8.3.1]. try: return getattr(self, name) except: pass method=REQUEST.get('REQUEST_METHOD', 'GET') if method in ('PUT', 'MKCOL', 'LOCK'): raise Conflict, 'Collection ancestors must already exist.' raise NotFound, 'The requested resource was not found.' def HEAD(self, REQUEST, RESPONSE): """Retrieve resource information without a response message body.""" self.dav__init(REQUEST, RESPONSE) raise NotFound, 'The requested resource does not exist.' # Most methods return 404 (Not Found) for null resources. DELETE=TRACE=PROPFIND=PROPPATCH=COPY=MOVE=HEAD index_html = HEAD def _default_PUT_factory( self, name, typ, body ): # Return DTMLDoc/PageTemplate/Image/File, based on sniffing. if name and name.endswith('.pt'): from Products.PageTemplates.ZopePageTemplate import ZopePageTemplate ob = ZopePageTemplate(name, body, content_type=typ) elif typ in ('text/html', 'text/xml', 'text/plain'): from OFS.DTMLDocument import DTMLDocument ob = DTMLDocument( '', __name__=name ) elif typ[:6]=='image/': from OFS.Image import Image ob=Image(name, '', body, content_type=typ) else: from OFS.Image import File ob=File(name, '', body, content_type=typ) return ob PUT__roles__ = ('Anonymous',) def PUT(self, REQUEST, RESPONSE): """Create a new non-collection resource. """ from ZServer import LARGE_FILE_THRESHOLD self.dav__init(REQUEST, RESPONSE) name = self.__name__ parent = self.__parent__ ifhdr = REQUEST.get_header('If', '') if WriteLockInterface.isImplementedBy(parent) and parent.wl_isLocked(): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, col=1) else: # There was no If header at all, and our parent is locked, # so we fail here raise Locked elif ifhdr: # There was an If header, but the parent is not locked raise PreconditionFailed # SDS: Only use BODY if the file size is smaller than # LARGE_FILE_THRESHOLD, otherwise read LARGE_FILE_THRESHOLD # bytes from the file which should be enough to trigger # content_type detection, and possibly enough for CMF's # content_type_registry too. # # Note that body here is really just used for detecting the # content type and figuring out the correct factory. The correct # file content will be uploaded on ob.PUT(REQUEST, RESPONSE) after # the object has been created. # # A problem I could see is content_type_registry predicates # that do depend on the whole file being passed here as an # argument. There's none by default that does this though. If # they really do want to look at the file, they should use # REQUEST['BODYFILE'] directly and try as much as possible not # to read the whole file into memory. if int(REQUEST.get('CONTENT_LENGTH') or 0) > LARGE_FILE_THRESHOLD: file = REQUEST['BODYFILE'] body = file.read(LARGE_FILE_THRESHOLD) file.seek(0) else: body = REQUEST.get('BODY', '') typ=REQUEST.get_header('content-type', None) if typ is None: typ, enc=OFS.content_types.guess_content_type(name, body) factory = getattr(parent, 'PUT_factory', self._default_PUT_factory ) ob = factory(name, typ, body) if ob is None: ob = self._default_PUT_factory(name, typ, body) # We call _verifyObjectPaste with verify_src=0, to see if the # user can create this type of object (and we don't need to # check the clipboard. try: parent._verifyObjectPaste(ob.__of__(parent), 0) except CopyError: raise Unauthorized, sys.exc_info()[1] # Delegate actual PUT handling to the new object, # SDS: But just *after* it has been stored. self.__parent__._setObject(name, ob) ob = self.__parent__._getOb(name) ob.PUT(REQUEST, RESPONSE) RESPONSE.setStatus(201) RESPONSE.setBody('') return RESPONSE def MKCOL(self, REQUEST, RESPONSE): """Create a new collection resource.""" self.dav__init(REQUEST, RESPONSE) if REQUEST.get('BODY', ''): raise UnsupportedMediaType, 'Unknown request body.' name=self.__name__ parent = self.__parent__ if hasattr(aq_base(parent), name): raise MethodNotAllowed, 'The name %s is in use.' % name if not isDavCollection(parent): raise Forbidden, 'Cannot create collection at this location.' ifhdr = REQUEST.get_header('If', '') if WriteLockInterface.isImplementedBy(parent) and parent.wl_isLocked(): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, col=1) else: raise Locked elif ifhdr: # There was an If header, but the parent is not locked raise PreconditionFailed # Add hook for webdav/FTP MKCOL (Collector #2254) (needed for CMF) # parent.manage_addFolder(name) mkcol_handler = getattr(parent,'MKCOL_handler' ,parent.manage_addFolder) mkcol_handler(name) RESPONSE.setStatus(201) RESPONSE.setBody('') return RESPONSE def LOCK(self, REQUEST, RESPONSE): """ LOCK on a Null Resource makes a LockNullResource instance """ self.dav__init(REQUEST, RESPONSE) security = getSecurityManager() creator = security.getUser() body = REQUEST.get('BODY', '') ifhdr = REQUEST.get_header('If', '') depth = REQUEST.get_header('Depth', 'infinity') name = self.__name__ parent = self.__parent__ if WriteLockInterface.isImplementedBy(parent) and parent.wl_isLocked(): if ifhdr: parent.dav__simpleifhandler(REQUEST, RESPONSE, col=1) else: raise Locked elif ifhdr: # There was an If header, but the parent is not locked. raise PreconditionFailed # The logic involved in locking a null resource is simpler than # a regular resource, since we know we're not already locked, # and the lock isn't being refreshed. if not body: raise BadRequest, 'No body was in the request' locknull = LockNullResource(name) parent._setObject(name, locknull) locknull = parent._getOb(name) cmd = davcmds.Lock(REQUEST) token, result = cmd.apply(locknull, creator, depth=depth) if result: # Return the multistatus result (there were multiple errors) # This *shouldn't* happen for locking a NullResource, but it's # inexpensive to handle and is good coverage for any future # changes in davcmds.Lock RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) else: # The command was succesful lock = locknull.wl_getLock(token) RESPONSE.setStatus(200) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setHeader('Lock-Token', 'opaquelocktoken:' + token) RESPONSE.setBody(lock.asXML()) Globals.default__class_init__(NullResource) class LockNullResource(NullResource, OFS.SimpleItem.Item_w__name__): """ A Lock-Null Resource is created when a LOCK command is succesfully executed on a NullResource, essentially locking the Name. A PUT or MKCOL deletes the LockNull resource from its container and replaces it with the target object. An UNLOCK deletes it. """ __implements__ = (WriteLockInterface,) __locknull_resource__ = 1 meta_type = 'WebDAV LockNull Resource' __ac_permissions__ = ( ('WebDAV Unlock items', ('UNLOCK',)), ('View', ('manage_main', 'manage_workspace', 'manage')), ('Add Folders', ('MKCOL',)), ('WebDAV Lock items', ('LOCK',)), ) manage_options = ({'label': 'Info', 'action': 'manage_main'},) manage = manage_main = DTMLFile('dtml/locknullmain', globals()) manage_workspace = manage manage_main._setName('manage_main') # explicit def __no_valid_write_locks__(self): # A special hook (for better or worse) called when there are no # valid locks left. We have to delete ourselves from our container # now. parent = Acquisition.aq_parent(self) if parent: parent._delObject(self.id) def __init__(self, name): self.id = self.__name__ = name self.title = "LockNull Resource '%s'" % name title_or_id__roles__=None def title_or_id(self): return 'Foo' def PROPFIND(self, REQUEST, RESPONSE): """Retrieve properties defined on the resource.""" return Resource.PROPFIND(self, REQUEST, RESPONSE) def LOCK(self, REQUEST, RESPONSE): """ A Lock command on a LockNull resource should only be a refresh request (one without a body) """ self.dav__init(REQUEST, RESPONSE) body = REQUEST.get('BODY', '') ifhdr = REQUEST.get_header('If', '') if body: # If there's a body, then this is a full lock request # which conflicts with the fact that we're already locked RESPONSE.setStatus(423) else: # There's no body, so this is likely to be a refresh request if not ifhdr: raise PreconditionFailed taglist = IfParser(ifhdr) found = 0 for tag in taglist: for listitem in tag.list: token = tokenFinder(listitem) if token and self.wl_hasLock(token): lock = self.wl_getLock(token) timeout = REQUEST.get_header('Timeout', 'infinite') lock.setTimeout(timeout) # Automatically refreshes found = 1 RESPONSE.setStatus(200) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(lock.asXML()) if found: break if not found: RESPONSE.setStatus(412) # Precondition failed return RESPONSE def UNLOCK(self, REQUEST, RESPONSE): """ Unlocking a Null Resource removes it from its parent """ self.dav__init(REQUEST, RESPONSE) security = getSecurityManager() user = security.getUser() token = REQUEST.get_header('Lock-Token', '') url = REQUEST['URL'] if token: token = tokenFinder(token) else: raise BadRequest, 'No lock token was submitted in the request' cmd = davcmds.Unlock() result = cmd.apply(self, token, url) parent = Acquisition.aq_parent(self) parent._delObject(self.id) if result: RESPONSE.setStatus(207) RESPONSE.setHeader('Content-Type', 'text/xml; charset="utf-8"') RESPONSE.setBody(result) else: RESPONSE.setStatus(204) return RESPONSE PUT__roles__ = ('Anonymous',) def PUT(self, REQUEST, RESPONSE): """ Create a new non-collection resource, deleting the LockNull object from the container before putting the new object in. """ self.dav__init(REQUEST, RESPONSE) name = self.__name__ parent = self.aq_parent parenturl = parent.absolute_url() ifhdr = REQUEST.get_header('If', '') # Since a Lock null resource is always locked by definition, all # operations done by an owner of the lock that affect the resource # MUST have the If header in the request if not ifhdr: raise PreconditionFailed, 'No If-header' # First we need to see if the parent of the locknull is locked, and # if the user owns that lock (checked by handling the information in # the If header). if WriteLockInterface.isImplementedBy(parent) and parent.wl_isLocked(): itrue = parent.dav__simpleifhandler(REQUEST, RESPONSE, 'PUT', col=1, url=parenturl, refresh=1) if not itrue: raise PreconditionFailed, ( 'Condition failed against resources parent') # Now we need to check the If header against our own lock state itrue = self.dav__simpleifhandler(REQUEST, RESPONSE, 'PUT', refresh=1) if not itrue: raise PreconditionFailed, ( 'Condition failed against locknull resource') # All of the If header tests succeeded, now we need to remove ourselves # from our parent. We need to transfer lock state to the new object. locks = self.wl_lockItems() parent._delObject(name) # Now we need to go through the regular operations of PUT body = REQUEST.get('BODY', '') typ = REQUEST.get_header('content-type', None) if typ is None: typ, enc = OFS.content_types.guess_content_type(name, body) factory = getattr(parent, 'PUT_factory', self._default_PUT_factory) ob = (factory(name, typ, body) or self._default_PUT_factory(name, typ, body)) # Verify that the user can create this type of object try: parent._verifyObjectPaste(ob.__of__(parent), 0) except Unauthorized: raise except: raise Forbidden, sys.exc_info()[1] # Put the locks on the new object if not WriteLockInterface.isImplementedBy(ob): raise MethodNotAllowed, ( 'The target object type cannot be locked') for token, lock in locks: ob.wl_setLock(token, lock) # Delegate actual PUT handling to the new object. ob.PUT(REQUEST, RESPONSE) parent._setObject(name, ob) RESPONSE.setStatus(201) RESPONSE.setBody('') return RESPONSE def MKCOL(self, REQUEST, RESPONSE): """ Create a new Collection (folder) resource. Since this is being done on a LockNull resource, this also involves removing the LockNull object and transferring its locks to the newly created Folder """ self.dav__init(REQUEST, RESPONSE) if REQUEST.get('BODY', ''): raise UnsupportedMediaType, 'Unknown request body.' name = self.__name__ parent = self.aq_parent parenturl = parent.absolute_url() ifhdr = REQUEST.get_header('If', '') if not ifhdr: raise PreconditionFailed, 'No If-header' # If the parent object is locked, that information should be in the # if-header if the user owns a lock on the parent if WriteLockInterface.isImplementedBy(parent) and parent.wl_isLocked(): itrue = parent.dav__simpleifhandler(REQUEST, RESPONSE, 'MKCOL', col=1, url=parenturl, refresh=1) if not itrue: raise PreconditionFailed, ( 'Condition failed against resources parent') # Now we need to check the If header against our own lock state itrue = self.dav__simpleifhandler(REQUEST,RESPONSE,'MKCOL',refresh=1) if not itrue: raise PreconditionFailed, ( 'Condition failed against locknull resource') # All of the If header tests succeeded, now we need to remove ourselves # from our parent. We need to transfer lock state to the new folder. locks = self.wl_lockItems() parent._delObject(name) parent.manage_addFolder(name) folder = parent._getOb(name) for token, lock in locks: folder.wl_setLock(token, lock) RESPONSE.setStatus(201) RESPONSE.setBody('') return RESPONSE Globals.default__class_init__(LockNullResource) Zope-2.8.7-final/lib/python/webdav/common.py0000664000076400007640000000753510435500330017744 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Commonly used functions for WebDAV support modules.""" __version__='$Revision: 1.18 $'[11:-2] import time, urllib, re from App.Common import iso8601_date, rfc850_date, rfc1123_date from App.Common import aq_base import random _randGen = random.Random(time.time()) class WebDAVException(Exception): pass class Locked(WebDAVException): pass class PreconditionFailed(WebDAVException): pass class Conflict(WebDAVException): pass class UnsupportedMediaType(WebDAVException): pass def absattr(attr): if callable(attr): return attr() return attr def urlfix(url, s): n=len(s) if url[-n:]==s: url=url[:-n] if len(url) > 1 and url[-1]=='/': url=url[:-1] return url def is_acquired(ob): # Return true if this object is not a direct # subobject of its aq_parent object. if not hasattr(ob, 'aq_parent'): return 0 if hasattr(aq_base(ob.aq_parent), absattr(ob.id)): return 0 if hasattr(aq_base(ob), 'isTopLevelPrincipiaApplicationObject') and \ ob.isTopLevelPrincipiaApplicationObject: return 0 return 1 def urlbase(url, ftype=urllib.splittype, fhost=urllib.splithost): # Return a '/' based url such as '/foo/bar', removing # type, host and port information if necessary. if url[0]=='/': return url type, uri=ftype(url) host, uri=fhost(uri) return uri or '/' def generateLockToken(): # Generate a lock token return '%s-%s-00105A989226:%.03f' % \ (_randGen.random(),_randGen.random(),time.time()) def isDavCollection(object): """Return true if object is a DAV collection.""" return getattr(object, '__dav_collection__', 0) def tokenFinder(token): # takes a string like ' and returns the token # part. if not token: return None # An empty string was passed in if token[0] == '[': return None # An Etag was passed in if token[0] == '<': token = token[1:-1] return token[token.find(':')+1:] ### If: header handling support. IfParser returns a sequence of ### TagList objects in the order they were parsed which can then ### be used in WebDAV methods to decide whether an operation can ### proceed or to raise HTTP Error 412 (Precondition failed) IfHdr = re.compile( r"(?P<.+?>)?\s*\((?P[^)]+)\)" ) ListItem = re.compile( r"(?Pnot)?\s*(?P<[a-zA-Z]+:[^>]*>|\[.*?\])", re.I) class TagList: def __init__(self): self.resource = None self.list = [] self.NOTTED = 0 def IfParser(hdr): out = [] i = 0 while 1: m = IfHdr.search(hdr[i:]) if not m: break i = i + m.end() tag = TagList() tag.resource = m.group('resource') if tag.resource: # We need to delete < > tag.resource = tag.resource[1:-1] listitem = m.group('listitem') tag.NOTTED, tag.list = ListParser(listitem) out.append(tag) return out def ListParser(listitem): out = [] NOTTED = 0 i = 0 while 1: m = ListItem.search(listitem[i:]) if not m: break i = i + m.end() out.append(m.group('listitem')) if m.group('not'): NOTTED = 1 return NOTTED, out Zope-2.8.7-final/lib/python/webdav/tests/0000775000076400007640000000000010435610645017245 5ustar ajungajungZope-2.8.7-final/lib/python/webdav/tests/testNullResource.py0000664000076400007640000000153510435500330023132 0ustar ajungajungimport unittest class TestLockNullResource(unittest.TestCase): def test_z2interfaces(self): from Interface.Verify import verifyClass from webdav.NullResource import LockNullResource from webdav.WriteLockInterface import WriteLockInterface verifyClass(WriteLockInterface, LockNullResource) class TestNullResource(unittest.TestCase): def test_z2interfaces(self): from Interface.Verify import verifyClass from webdav.NullResource import NullResource from webdav.WriteLockInterface import WriteLockInterface verifyClass(WriteLockInterface, NullResource) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(TestLockNullResource), unittest.makeSuite(TestNullResource), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/webdav/tests/testEtagSupport.py0000664000076400007640000000074410435500330022766 0ustar ajungajungimport unittest class TestEtagSupport(unittest.TestCase): def test_z2interfaces(self): from Interface.Verify import verifyClass from webdav.EtagSupport import EtagBaseInterface from webdav.EtagSupport import EtagSupport verifyClass(EtagBaseInterface, EtagSupport) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(TestEtagSupport), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/webdav/tests/__init__.py0000664000076400007640000000007510435500330021345 0ustar ajungajung# # This file is necessary to make this directory a package. Zope-2.8.7-final/lib/python/webdav/tests/testLockItem.py0000664000076400007640000000073410435500330022217 0ustar ajungajungimport unittest class TestLockItem(unittest.TestCase): def test_z2interfaces(self): from Interface.Verify import verifyClass from webdav.LockItem import LockItem from webdav.WriteLockInterface import LockItemInterface verifyClass(LockItemInterface, LockItem) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(TestLockItem), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/webdav/davcmds.py0000664000076400007640000005075710435500330020101 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """WebDAV xml request objects. $Id: davcmds.py 37530 2005-07-28 10:14:13Z andreasjung $ """ import sys from common import absattr, aq_base, urlfix, urlbase from OFS.PropertySheets import DAVProperties from LockItem import LockItem from WriteLockInterface import WriteLockInterface from Acquisition import aq_parent from xmltools import XmlParser from cStringIO import StringIO from urllib import quote from AccessControl import getSecurityManager from zExceptions import BadRequest, Forbidden from common import isDavCollection from common import PreconditionFailed from ZConfig.url import urljoin import transaction def safe_quote(url, mark=r'%'): if url.find(mark) > -1: return url return quote(url) class DAVProps(DAVProperties): """Emulate required DAV properties for objects which do not themselves support properties. This is mainly so that non-PropertyManagers can appear to support DAV PROPFIND requests.""" def __init__(self, obj): self.__obj__=obj def v_self(self): return self.__obj__ p_self=v_self class PropFind: """Model a PROPFIND request.""" def __init__(self, request): self.request=request self.depth='infinity' self.allprop=0 self.propname=0 self.propnames=[] self.parse(request) def parse(self, request, dav='DAV:'): self.depth=request.get_header('Depth', 'infinity') if not (self.depth in ('0','1','infinity')): raise BadRequest, 'Invalid Depth header.' body=request.get('BODY', '') self.allprop=(not len(body)) if not body: return try: root=XmlParser().parse(body) except: raise BadRequest, sys.exc_info()[1] e=root.elements('propfind', ns=dav) if not e: raise BadRequest, 'Invalid xml request.' e=e[0] if e.elements('allprop', ns=dav): self.allprop=1 return if e.elements('propname', ns=dav): self.propname=1 return prop=e.elements('prop', ns=dav) if not prop: raise BadRequest, 'Invalid xml request.' prop=prop[0] for val in prop.elements(): self.propnames.append((val.name(), val.namespace())) if (not self.allprop) and (not self.propname) and \ (not self.propnames): raise BadRequest, 'Invalid xml request.' return def apply(self, obj, url=None, depth=0, result=None, top=1): if result is None: result=StringIO() depth=self.depth url=urlfix(self.request['URL'], 'PROPFIND') url=urlbase(url) result.write('\n' \ '\n') iscol=isDavCollection(obj) if iscol and url[-1] != '/': url=url+'/' result.write('\n%s\n' % safe_quote(url)) if hasattr(aq_base(obj), 'propertysheets'): propsets=obj.propertysheets.values() obsheets=obj.propertysheets else: davprops=DAVProps(obj) propsets=(davprops,) obsheets={'DAV:': davprops} if self.allprop: stats=[] for ps in propsets: if hasattr(aq_base(ps), 'dav__allprop'): stats.append(ps.dav__allprop()) stats=''.join(stats) or '200 OK\n' result.write(stats) elif self.propname: stats=[] for ps in propsets: if hasattr(aq_base(ps), 'dav__propnames'): stats.append(ps.dav__propnames()) stats=''.join(stats) or '200 OK\n' result.write(stats) elif self.propnames: rdict={} for name, ns in self.propnames: ps=obsheets.get(ns, None) if ps is not None and hasattr(aq_base(ps), 'dav__propstat'): stat=ps.dav__propstat(name, rdict) else: prop='' % (name, ns) code='404 Not Found' if not rdict.has_key(code): rdict[code]=[prop] else: rdict[code].append(prop) keys=rdict.keys() keys.sort() for key in keys: result.write('\n' \ ' \n' \ ) map(result.write, rdict[key]) result.write(' \n' \ ' HTTP/1.1 %s\n' \ '\n' % key ) else: raise BadRequest, 'Invalid request' result.write('\n') if depth in ('1', 'infinity') and iscol: for ob in obj.listDAVObjects(): if hasattr(ob,"meta_type"): if ob.meta_type=="Broken Because Product is Gone": continue dflag=hasattr(ob, '_p_changed') and (ob._p_changed == None) if hasattr(ob, '__locknull_resource__'): # Do nothing, a null resource shouldn't show up to DAV if dflag: ob._p_deactivate() elif hasattr(ob, '__dav_resource__'): uri=urljoin(url, absattr(ob.id)) depth=depth=='infinity' and depth or 0 self.apply(ob, uri, depth, result, top=0) if dflag: ob._p_deactivate() if not top: return result result.write('') return result.getvalue() class PropPatch: """Model a PROPPATCH request.""" def __init__(self, request): self.request=request self.values=[] self.parse(request) def parse(self, request, dav='DAV:'): body=request.get('BODY', '') try: root=XmlParser().parse(body) except: raise BadRequest, sys.exc_info()[1] vals=self.values e=root.elements('propertyupdate', ns=dav) if not e: raise BadRequest, 'Invalid xml request.' e=e[0] for ob in e.elements(): if ob.name()=='set' and ob.namespace()==dav: proptag=ob.elements('prop', ns=dav) if not proptag: raise BadRequest, 'Invalid xml request.' proptag=proptag[0] for prop in proptag.elements(): # We have to ensure that all tag attrs (including # an xmlns attr for all xml namespaces used by the # element and its children) are saved, per rfc2518. name, ns=prop.name(), prop.namespace() e, attrs=prop.elements(), prop.attrs() if (not e) and (not attrs): # simple property item=(name, ns, prop.strval(), {}) vals.append(item) else: # xml property attrs={} prop.remap({ns:'n'}) prop.del_attr('xmlns:n') for attr in prop.attrs(): attrs[attr.qname()]=attr.value() md={'__xml_attrs__':attrs} item=(name, ns, prop.strval(), md) vals.append(item) if ob.name()=='remove' and ob.namespace()==dav: proptag=ob.elements('prop', ns=dav) if not proptag: raise BadRequest, 'Invalid xml request.' proptag=proptag[0] for prop in proptag.elements(): item=(prop.name(), prop.namespace()) vals.append(item) def apply(self, obj): url=urlfix(self.request['URL'], 'PROPPATCH') if isDavCollection(obj): url=url+'/' result=StringIO() errors=[] result.write('\n' \ '\n' \ '\n' \ '%s\n' % quote(url)) propsets=obj.propertysheets for value in self.values: status='200 OK' if len(value) > 2: name, ns, val, md=value propset=propsets.get(ns, None) if propset is None: propsets.manage_addPropertySheet('', ns) propset=propsets.get(ns) if propset.hasProperty(name): try: propset._updateProperty(name, val, meta=md) except: errors.append(str(sys.exc_info()[1])) status='409 Conflict' else: try: propset._setProperty(name, val, meta=md) except: errors.append(str(sys.exc_info()[1])) status='409 Conflict' else: name, ns=value propset=propsets.get(ns, None) if propset is None or not propset.hasProperty(name): # removing a non-existing property is not an error! # according to RFC 2518 status='200 OK' else: try: propset._delProperty(name) except: errors.append('%s cannot be deleted.' % name) status='409 Conflict' result.write('\n' \ ' \n' \ ' \n' \ ' \n' \ ' HTTP/1.1 %s\n' \ '\n' % (ns, name, status)) errmsg='\n'.join(errors) or 'The operation succeded.' result.write('\n' \ '%s\n' \ '\n' \ '\n' \ '' % errmsg) result=result.getvalue() if not errors: return result # This is lame, but I cant find a way to keep ZPublisher # from sticking a traceback into my xml response :( transaction.abort() result=result.replace( '200 OK', '424 Failed Dependency') return result class Lock: """Model a LOCK request.""" def __init__(self, request): self.request = request data = request.get('BODY', '') self.scope = 'exclusive' self.type = 'write' self.owner = '' timeout = request.get_header('Timeout', 'infinite') self.timeout = timeout.split(',')[-1].strip() self.parse(data) def parse(self, data, dav='DAV:'): root = XmlParser().parse(data) info = root.elements('lockinfo', ns=dav)[0] ls = info.elements('lockscope', ns=dav)[0] self.scope = ls.elements()[0].name() lt = info.elements('locktype', ns=dav)[0] self.type = lt.elements()[0].name() lockowner = info.elements('owner', ns=dav) if lockowner: # Since the Owner element may contain children in different # namespaces (or none at all), we have to find them for potential # remapping. Note that Cadaver doesn't use namespaces in the # XML it sends. lockowner = lockowner[0] for el in lockowner.elements(): name, elns = el.name(), el.namespace() if not elns: # There's no namespace, so we have to add one lockowner.remap({dav:'ot'}) el.__nskey__ = 'ot' for subel in el.elements(): if not subel.namespace(): el.__nskey__ = 'ot' else: el.remap({dav:'o'}) self.owner = lockowner.strval() def apply(self, obj, creator=None, depth='infinity', token=None, result=None, url=None, top=1): """ Apply, built for recursion (so that we may lock subitems of a collection if requested """ if result is None: result = StringIO() url = urlfix(self.request['URL'], 'LOCK') url = urlbase(url) iscol = isDavCollection(obj) if iscol and url[-1] != '/': url = url + '/' errmsg = None lock = None try: lock = LockItem(creator, self.owner, depth, self.timeout, self.type, self.scope, token) if token is None: token = lock.getLockToken() except ValueError: errmsg = "412 Precondition Failed" except: errmsg = "403 Forbidden" try: if not WriteLockInterface.isImplementedBy(obj): if top: # This is the top level object in the apply, so we # do want an error errmsg = "405 Method Not Allowed" else: # We're in an infinity request and a subobject does # not support locking, so we'll just pass pass elif obj.wl_isLocked(): errmsg = "423 Locked" else: method = getattr(obj, 'wl_setLock') vld = getSecurityManager().validate(None, obj, 'wl_setLock', method) if vld and token and (lock is not None): obj.wl_setLock(token, lock) else: errmsg = "403 Forbidden" except: errmsg = "403 Forbidden" if errmsg: if top and ((depth in (0, '0')) or (not iscol)): # We don't need to raise multistatus errors raise errmsg[4:] elif not result.getvalue(): # We haven't had any errors yet, so our result is empty # and we need to set up the XML header result.write('\n' \ '\n') result.write('\n %s\n' % url) result.write(' HTTP/1.1 %s\n' % errmsg) result.write('\n') if depth == 'infinity' and iscol: for ob in obj.objectValues(): if hasattr(obj, '__dav_resource__'): uri = urljoin(url, absattr(ob.id)) self.apply(ob, creator, depth, token, result, uri, top=0) if not top: return token, result if result.getvalue(): # One or more subitems probably failed, so close the multistatus # element and clear out all succesful locks result.write('') transaction.abort() # This *SHOULD* clear all succesful locks return token, result.getvalue() class Unlock: """ Model an Unlock request """ def apply(self, obj, token, url=None, result=None, top=1): if result is None: result = StringIO() url = urlfix(url, 'UNLOCK') url = urlbase(url) iscol = isDavCollection(obj) if iscol and url[-1] != '/': url = url + '/' errmsg = None islockable = WriteLockInterface.isImplementedBy(obj) if islockable and obj.wl_hasLock(token): method = getattr(obj, 'wl_delLock') vld = getSecurityManager().validate(None,obj,'wl_delLock',method) if vld: obj.wl_delLock(token) else: errmsg = "403 Forbidden" elif not islockable: # Only set an error message if the command is being applied # to a top level object. Otherwise, we're descending a tree # which may contain many objects that don't implement locking, # so we just want to avoid them if top: errmsg = "405 Method Not Allowed" if errmsg: if top and (not iscol): # We don't need to raise multistatus errors if errmsg[:3] == '403': raise Forbidden else: raise PreconditionFailed elif not result.getvalue(): # We haven't had any errors yet, so our result is empty # and we need to set up the XML header result.write('\n' \ '\n') result.write('\n %s\n' % url) result.write(' HTTP/1.1 %s\n' % errmsg) result.write('\n') if iscol: for ob in obj.objectValues(): if hasattr(ob, '__dav_resource__') and \ WriteLockInterface.isImplementedBy(ob): uri = urljoin(url, absattr(ob.id)) self.apply(ob, token, uri, result, top=0) if not top: return result if result.getvalue(): # One or more subitems probably failed, so close the multistatus # element and clear out all succesful unlocks result.write('') transaction.abort() return result.getvalue() class DeleteCollection: """ With WriteLocks in the picture, deleting a collection involves checking *all* descendents (deletes on collections are always of depth infinite) for locks and if the locks match. """ def apply(self, obj, token, user, url=None, result=None, top=1): if result is None: result = StringIO() url = urlfix(url, 'DELETE') url = urlbase(url) iscol = isDavCollection(obj) errmsg = None parent = aq_parent(obj) islockable = WriteLockInterface.isImplementedBy(obj) if parent and (not user.has_permission('Delete objects', parent)): # User doesn't have permission to delete this object errmsg = "403 Forbidden" elif islockable and obj.wl_isLocked(): if token and obj.wl_hasLock(token): # Object is locked, and the token matches (no error) errmsg = "" else: errmsg = "423 Locked" if errmsg: if top and (not iscol): err = errmsg[4:] raise err elif not result.getvalue(): # We haven't had any errors yet, so our result is empty # and we need to set up the XML header result.write('\n' \ '\n') result.write('\n %s\n' % url) result.write(' HTTP/1.1 %s\n' % errmsg) result.write('\n') if iscol: for ob in obj.objectValues(): dflag = hasattr(ob,'_p_changed') and (ob._p_changed == None) if hasattr(ob, '__dav_resource__'): uri = urljoin(url, absattr(ob.id)) self.apply(ob, token, user, uri, result, top=0) if dflag: ob._p_deactivate() if not top: return result if result.getvalue(): # One or more subitems can't be delted, so close the multistatus # element result.write('\n') return result.getvalue() Zope-2.8.7-final/lib/python/webdav/www/0000775000076400007640000000000010435500330016714 5ustar ajungajungZope-2.8.7-final/lib/python/webdav/www/davlock.gif0000664000076400007640000000014510435500330021026 0ustar ajungajungGIF89a ̙fff333!, *"Qj\T"Š0x=@l*X {) #;Zope-2.8.7-final/lib/python/Missing/0000775000076400007640000000000010435610645016244 5ustar ajungajungZope-2.8.7-final/lib/python/Missing/__init__.py0000664000076400007640000000002710435500325020345 0ustar ajungajungfrom _Missing import * Zope-2.8.7-final/lib/python/Missing/setup.py0000664000076400007640000000164410435500325017754 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from distutils.core import setup, Extension setup(name="_Missing", version="2.0", ext_modules=[ Extension("_Missing", ["_Missing.c"], include_dirs = ['.', '../ExtensionClass'], depends = ['../ExtensionClass/ExtensionClass.h']), ]) Zope-2.8.7-final/lib/python/Missing/tests.py0000664000076400007640000000214610435500325017754 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test for missing values. >>> from Missing import Value >>> Value != 12 1 >>> 12 != Value 1 >>> u"abc" != Value 1 >>> Value != u"abc" 1 >>> 1 + Value == Value 1 >>> Value + 1 == Value 1 >>> Value == 1 + Value 1 >>> Value == Value + 1 1 $Id: tests.py 29895 2005-04-07 03:49:30Z tim_one $ """ import unittest from zope.testing.doctest import DocTestSuite def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/Missing/_Missing.c0000664000076400007640000001634110435500325020156 0ustar ajungajung/***************************************************************************** Copyright (c) 1996-2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ static char Missing_module_documentation[] = "" "\n$Id: _Missing.c 24763 2004-05-17 05:59:28Z philikon $" ; #include "ExtensionClass.h" /* Declarations for objects of type Missing */ typedef struct { PyObject_HEAD } Missing; static PyObject *vname=0, *Missing_dot_Value=0, *empty_string=0, *reduce=0; static PyObject *theValue, *notMissing; static void Missing_dealloc(Missing *self) { Py_DECREF(self->ob_type); PyMem_DEL(self); } static PyObject * Missing_repr(Missing *self) { Py_INCREF(Missing_dot_Value); return Missing_dot_Value; } static PyObject * Missing_str(Missing *self) { Py_INCREF(empty_string); return empty_string; } /* Code to access Missing objects as numbers. We must guarantee that notMissing is never returned to Python code, because it would violate the guarantee that all Python-accessible Missing values are equal to each other. */ static PyObject * Missing_bin(PyObject *v, PyObject *w) { if (v == notMissing) v = w; assert(v != notMissing); Py_INCREF(v); return v; } static PyObject * Missing_pow(PyObject *v, PyObject *w, PyObject *z) { if (v == notMissing) v = w; assert(v != notMissing); Py_INCREF(v); return v; } static PyObject * Missing_un(PyObject *v) { Py_INCREF(v); return v; } static int Missing_nonzero(PyObject *v) { return 0; } /* Always return the distinguished notMissing object as the result of the coercion. The notMissing object does not compare equal to other Missing objects. */ static int Missing_coerce(PyObject **pv, PyObject **pw) { Py_INCREF(*pv); Py_INCREF(notMissing); *pw = notMissing; return 0; } static PyNumberMethods Missing_as_number = { (binaryfunc)Missing_bin, /*nb_add*/ (binaryfunc)Missing_bin, /*nb_subtract*/ (binaryfunc)Missing_bin, /*nb_multiply*/ (binaryfunc)Missing_bin, /*nb_divide*/ (binaryfunc)Missing_bin, /*nb_remainder*/ (binaryfunc)Missing_bin, /*nb_divmod*/ (ternaryfunc)Missing_pow, /*nb_power*/ (unaryfunc)Missing_un, /*nb_negative*/ (unaryfunc)Missing_un, /*nb_positive*/ (unaryfunc)Missing_un, /*nb_absolute*/ (inquiry)Missing_nonzero, /*nb_nonzero*/ (unaryfunc)Missing_un, /*nb_invert*/ (binaryfunc)Missing_bin, /*nb_lshift*/ (binaryfunc)Missing_bin, /*nb_rshift*/ (binaryfunc)Missing_bin, /*nb_and*/ (binaryfunc)Missing_bin, /*nb_xor*/ (binaryfunc)Missing_bin, /*nb_or*/ (coercion)Missing_coerce, /*nb_coerce*/ 0, /*nb_int*/ 0, /*nb_long*/ 0, /*nb_float*/ 0, /*nb_oct*/ 0, /*nb_hex*/ #if PY_MAJOR_VERSION == 2 && PY_MINOR_VERSION != 1 0, /* nb_inplace_add */ 0, /* nb_inplace_subtract */ 0, /* nb_inplace_multiply */ 0, /* nb_inplace_divide */ 0, /* nb_inplace_remainder */ 0, /* nb_inplace_power */ 0, /* nb_inplace_lshift */ 0, /* nb_inplace_rshift */ 0, /* nb_inplace_and */ 0, /* nb_inplace_xor */ 0, /* nb_inplace_or */ Missing_bin, /* nb_floor_divide */ Missing_bin, /* nb_true_divide */ 0, /* nb_inplace_floor_divide */ 0, /* nb_inplace_true_divide */ #endif }; /* ------------------------------------------------------- */ static PyObject * Missing_reduce(PyObject *self, PyObject *args, PyObject *kw) { if(self==theValue) { Py_INCREF(vname); return vname; } return Py_BuildValue("O()",self->ob_type); } static struct PyMethodDef reduce_ml[] = { {"__reduce__", (PyCFunction)Missing_reduce, 1, "Return a missing value reduced to standard python objects" } }; static PyObject * Missing_getattr(PyObject *self, PyObject *name) { char *c, *legal; if(!(c=PyString_AsString(name))) return NULL; legal=c; if (strchr("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ", *legal) != NULL) { for (legal++; *legal; legal++) if (strchr("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_", *legal) == NULL) { legal=NULL; break; } } else legal=NULL; if(! legal) { if(strcmp(c,"__reduce__")==0) { if(self==theValue) { Py_INCREF(reduce); return reduce; } return PyCFunction_New(reduce_ml, self); } PyErr_SetObject(PyExc_AttributeError, name); return NULL; } Py_INCREF(self); return self; } static PyObject * Missing_call(PyObject *self, PyObject *args, PyObject *kw) { Py_INCREF(self); return self; } /* All Missing objects are equal to each other, except for the special notMissing object. It is returned by coerce to indicate that Missing is being compare to something else. */ static int Missing_cmp(PyObject *m1, PyObject *m2) { if (m1 == notMissing) return -1; return (m2 == notMissing); } static PyExtensionClass MissingType = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ "Missing", /*tp_name*/ sizeof(Missing), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)Missing_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)0, /*obsolete tp_getattr*/ (setattrfunc)0, /*obsolete tp_setattr*/ Missing_cmp, /*tp_compare*/ (reprfunc)Missing_repr, /*tp_repr*/ &Missing_as_number, /*tp_as_number*/ 0, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)Missing_call, /*tp_call*/ (reprfunc)Missing_str, /*tp_str*/ (getattrofunc)Missing_getattr, /*tp_getattro*/ (setattrofunc)0, /*tp_setattro*/ /* Space for future expansion */ 0L,0L, "Represent totally unknown quantities\n" "\n" "Missing values are used to represent numeric quantities that are\n" "unknown. They support all mathematical operations except\n" "conversions by returning themselves.\n", METHOD_CHAIN(NULL) }; /* End of code for Missing objects */ /* -------------------------------------------------------- */ /* List of methods defined in the module */ static struct PyMethodDef Module_Level__methods[] = { {NULL, (PyCFunction)NULL, 0, NULL} /* sentinel */ }; void init_Missing(void) { PyObject *m, *d; if(! ((vname=PyString_FromString("V")) && (Missing_dot_Value=PyString_FromString("Missing.Value")) && (empty_string=PyString_FromString("")) )) return; /* Create the module and add the functions */ m = Py_InitModule4("_Missing", Module_Level__methods, Missing_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); PyExtensionClass_Export(d,"Missing",MissingType); theValue = PyObject_CallObject((PyObject*)&MissingType, NULL); notMissing = PyObject_CallObject((PyObject*)&MissingType, NULL); reduce=PyCFunction_New(reduce_ml, theValue); PyDict_SetItemString(d, "Value", theValue); PyDict_SetItemString(d, "V", theValue); PyDict_SetItemString(d, "MV", theValue); } Zope-2.8.7-final/lib/python/BTrees/0000775000076400007640000000000010435610645016017 5ustar ajungajungZope-2.8.7-final/lib/python/BTrees/SETUP.cfg0000664000076400007640000000540110435502401017366 0ustar ajungajung# Extension information for zpkg. # These extensions depend on a header provided by another component; # this header is named using depends-on in the sections for each # extension that requires it, but must also be identified as a public # header in the component that needs it. source _fsBTree.c # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c source _IIBTree.c # Specialization: depends-on intkeymacros.h depends-on intvaluemacros.h # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c source _IFBTree.c # Specialization: depends-on intkeymacros.h depends-on floatvaluemacros.h # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c source _IOBTree.c # Specialization: depends-on intkeymacros.h depends-on objectvaluemacros.h # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c source _OIBTree.c # Specialization: depends-on objectkeymacros.h depends-on intvaluemacros.h # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c source _OOBTree.c # Specialization: depends-on objectkeymacros.h depends-on objectvaluemacros.h # Common btree includes: depends-on BTreeItemsTemplate.c depends-on BTreeModuleTemplate.c depends-on BTreeTemplate.c depends-on BucketTemplate.c depends-on MergeTemplate.c depends-on SetOpTemplate.c depends-on SetTemplate.c depends-on TreeSetTemplate.c depends-on sorters.c Zope-2.8.7-final/lib/python/BTrees/Length.py0000664000076400007640000000343310435502401017603 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import persistent class Length(persistent.Persistent): """BTree lengths are too expensive to compute Objects that use BTrees need to keep track of lengths themselves. This class provides an object for doing this. As a bonus, the object support application-level conflict resolution. It is tempting to to assign length objects to __len__ attributes to provide instance-specific __len__ methods. However, this no longer works as expected, because new-style classes cache class-defined slot methods (like __len__) in C type slots. Thus, instance-define slot fillers are ignores. """ def __init__(self, v=0): self.value = v def __getstate__(self): return self.value def __setstate__(self, v): self.value = v def set(self, v): self.value = v def _p_resolveConflict(self, old, s1, s2): return s1 + s2 - old def _p_independent(self): # My state doesn't depend on or materially effect the state of # other objects. return 1 def change(self, delta): self.value += delta def __call__(self, *args): return self.value Zope-2.8.7-final/lib/python/BTrees/objectvaluemacros.h0000664000076400007640000000101510435502401021663 0ustar ajungajung #define VALUEMACROS_H "$Id: objectvaluemacros.h 24682 2004-05-14 19:32:43Z tim_one $\n" #define VALUE_TYPE PyObject * #define VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(VALUE, TARGET) PyObject_Compare((VALUE),(TARGET)) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define INCREF_VALUE(k) Py_INCREF(k) #define DECREF_VALUE(k) Py_DECREF(k) #define COPY_VALUE(k,e) k=(e) #define COPY_VALUE_TO_OBJECT(O, K) O=(K); Py_INCREF(O) #define COPY_VALUE_FROM_ARG(TARGET, ARG, S) TARGET=(ARG) #define NORMALIZE_VALUE(V, MIN) Py_INCREF(V) Zope-2.8.7-final/lib/python/BTrees/BTreeTemplate.c0000664000076400007640000016462110435502401020660 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define BTREETEMPLATE_C "$Id: BTreeTemplate.c 31010 2005-07-05 17:25:52Z tim_one $\n" /* Sanity-check a BTree. This is a private helper for BTree_check. Return: * -1 Error. If it's an internal inconsistency in the BTree, * AssertionError is set. * 0 No problem found. * * nextbucket is the bucket "one beyond the end" of the BTree; the last bucket * directly reachable from following right child pointers *should* be linked * to nextbucket (and this is checked). */ static int BTree_check_inner(BTree *self, Bucket *nextbucket) { int i; Bucket *bucketafter; Sized *child; char *errormsg = "internal error"; /* someone should have overriden */ Sized *activated_child = NULL; int result = -1; /* until proved innocent */ #define CHECK(CONDITION, ERRORMSG) \ if (!(CONDITION)) { \ errormsg = (ERRORMSG); \ goto Error; \ } PER_USE_OR_RETURN(self, -1); CHECK(self->len >= 0, "BTree len < 0"); CHECK(self->len <= self->size, "BTree len > size"); if (self->len == 0) { /* Empty BTree. */ CHECK(self->firstbucket == NULL, "Empty BTree has non-NULL firstbucket"); result = 0; goto Done; } /* Non-empty BTree. */ CHECK(self->firstbucket != NULL, "Non-empty BTree has NULL firstbucket"); /* Obscure: The first bucket is pointed to at least by self->firstbucket * and data[0].child of whichever BTree node it's a child of. However, * if persistence is enabled then the latter BTree node may be a ghost * at this point, and so its pointers "don't count": we can only rely * on self's pointers being intact. */ #ifdef PERSISTENT CHECK(self->firstbucket->ob_refcnt >= 1, "Non-empty BTree firstbucket has refcount < 1"); #else CHECK(self->firstbucket->ob_refcnt >= 2, "Non-empty BTree firstbucket has refcount < 2"); #endif for (i = 0; i < self->len; ++i) { CHECK(self->data[i].child != NULL, "BTree has NULL child"); } if (SameType_Check(self, self->data[0].child)) { /* Our children are also BTrees. */ child = self->data[0].child; UNLESS (PER_USE(child)) goto Done; activated_child = child; CHECK(self->firstbucket == BTREE(child)->firstbucket, "BTree has firstbucket different than " "its first child's firstbucket"); PER_ALLOW_DEACTIVATION(child); activated_child = NULL; for (i = 0; i < self->len; ++i) { child = self->data[i].child; CHECK(SameType_Check(self, child), "BTree children have different types"); if (i == self->len - 1) bucketafter = nextbucket; else { BTree *child2 = BTREE(self->data[i+1].child); UNLESS (PER_USE(child2)) goto Done; bucketafter = child2->firstbucket; PER_ALLOW_DEACTIVATION(child2); } if (BTree_check_inner(BTREE(child), bucketafter) < 0) goto Done; } } else { /* Our children are buckets. */ CHECK(self->firstbucket == BUCKET(self->data[0].child), "Bottom-level BTree node has inconsistent firstbucket belief"); for (i = 0; i < self->len; ++i) { child = self->data[i].child; UNLESS (PER_USE(child)) goto Done; activated_child = child; CHECK(!SameType_Check(self, child), "BTree children have different types"); CHECK(child->len >= 1, "Bucket length < 1"); /* no empty buckets! */ CHECK(child->len <= child->size, "Bucket len > size"); #ifdef PERSISTENT CHECK(child->ob_refcnt >= 1, "Bucket has refcount < 1"); #else CHECK(child->ob_refcnt >= 2, "Bucket has refcount < 2"); #endif if (i == self->len - 1) bucketafter = nextbucket; else bucketafter = BUCKET(self->data[i+1].child); CHECK(BUCKET(child)->next == bucketafter, "Bucket next pointer is damaged"); PER_ALLOW_DEACTIVATION(child); activated_child = NULL; } } result = 0; goto Done; Error: PyErr_SetString(PyExc_AssertionError, errormsg); result = -1; Done: /* No point updating access time -- this isn't a "real" use. */ PER_ALLOW_DEACTIVATION(self); if (activated_child) { PER_ALLOW_DEACTIVATION(activated_child); } return result; #undef CHECK } /* Sanity-check a BTree. This is the ._check() method. Return: * NULL Error. If it's an internal inconsistency in the BTree, * AssertionError is set. * Py_None No problem found. */ static PyObject* BTree_check(BTree *self) { PyObject *result = NULL; int i = BTree_check_inner(self, NULL); if (i >= 0) { result = Py_None; Py_INCREF(result); } return result; } /* ** _BTree_get ** ** Search a BTree. ** ** Arguments ** self a pointer to a BTree ** keyarg the key to search for, as a Python object ** has_key true/false; when false, try to return the associated ** value; when true, return a boolean ** Return ** When has_key false: ** If key exists, its associated value. ** If key doesn't exist, NULL and KeyError is set. ** When has_key true: ** A Python int is returned in any case. ** If key exists, the depth of the bucket in which it was found. ** If key doesn't exist, 0. */ static PyObject * _BTree_get(BTree *self, PyObject *keyarg, int has_key) { KEY_TYPE key; PyObject *result = NULL; /* guilty until proved innocent */ int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return NULL; PER_USE_OR_RETURN(self, NULL); if (self->len == 0) { /* empty BTree */ if (has_key) result = PyInt_FromLong(0); else PyErr_SetObject(PyExc_KeyError, keyarg); } else { for (;;) { int i; Sized *child; BTREE_SEARCH(i, self, key, goto Done); child = self->data[i].child; has_key += has_key != 0; /* bump depth counter, maybe */ if (SameType_Check(self, child)) { PER_UNUSE(self); self = BTREE(child); PER_USE_OR_RETURN(self, NULL); } else { result = _bucket_get(BUCKET(child), keyarg, has_key); break; } } } Done: PER_UNUSE(self); return result; } static PyObject * BTree_get(BTree *self, PyObject *key) { return _BTree_get(self, key, 0); } /* Create a new bucket for the BTree or TreeSet using the class attribute _bucket_type, which is normally initialized to BucketType or SetType as appropriate. */ static Sized * BTree_newBucket(BTree *self) { PyObject *factory; Sized *result; /* _bucket_type_str defined in BTreeModuleTemplate.c */ factory = PyObject_GetAttr((PyObject *)self->ob_type, _bucket_type_str); if (factory == NULL) return NULL; /* TODO: Should we check that the factory actually returns something of the appropriate type? How? The C code here is going to depend on any custom bucket type having the same layout at the C level. */ result = SIZED(PyObject_CallObject(factory, NULL)); Py_DECREF(factory); return result; } /* * Move data from the current BTree, from index onward, to the newly created * BTree 'next'. self and next must both be activated. If index is OOB (< 0 * or >= self->len), use self->len / 2 as the index (i.e., split at the * midpoint). self must have at least 2 children on entry, and index must * be such that self and next each have at least one child at exit. self's * accessed time is updated. * * Return: * -1 error * 0 OK */ static int BTree_split(BTree *self, int index, BTree *next) { int next_size; Sized *child; if (index < 0 || index >= self->len) index = self->len / 2; next_size = self->len - index; ASSERT(index > 0, "split creates empty tree", -1); ASSERT(next_size > 0, "split creates empty tree", -1); next->data = BTree_Malloc(sizeof(BTreeItem) * next_size); if (!next->data) return -1; memcpy(next->data, self->data + index, sizeof(BTreeItem) * next_size); next->size = next_size; /* but don't set len until we succeed */ /* Set next's firstbucket. self->firstbucket is still correct. */ child = next->data[0].child; if (SameType_Check(self, child)) { PER_USE_OR_RETURN(child, -1); next->firstbucket = BTREE(child)->firstbucket; PER_UNUSE(child); } else next->firstbucket = BUCKET(child); Py_INCREF(next->firstbucket); next->len = next_size; self->len = index; return PER_CHANGED(self) >= 0 ? 0 : -1; } /* Fwd decl -- BTree_grow and BTree_split_root reference each other. */ static int BTree_grow(BTree *self, int index, int noval); /* Split the root. This is a little special because the root isn't a child * of anything else, and the root needs to retain its object identity. So * this routine moves the root's data into a new child, and splits the * latter. This leaves the root with two children. * * Return: * 0 OK * -1 error * * CAUTION: The caller must call PER_CHANGED on self. */ static int BTree_split_root(BTree *self, int noval) { BTree *child; BTreeItem *d; /* Create a child BTree, and a new data vector for self. */ child = BTREE(PyObject_CallObject(OBJECT(self->ob_type), NULL)); if (!child) return -1; d = BTree_Malloc(sizeof(BTreeItem) * 2); if (!d) { Py_DECREF(child); return -1; } /* Move our data to new BTree. */ child->size = self->size; child->len = self->len; child->data = self->data; child->firstbucket = self->firstbucket; Py_INCREF(child->firstbucket); /* Point self to child and split the child. */ self->data = d; self->len = 1; self->size = 2; self->data[0].child = SIZED(child); /* transfers reference ownership */ return BTree_grow(self, 0, noval); } /* ** BTree_grow ** ** Grow a BTree ** ** Arguments: self The BTree ** index self->data[index].child needs to be split. index ** must be 0 if self is empty (len == 0), and a new ** empty bucket is created then. ** noval Boolean; is this a set (true) or mapping (false)? ** ** Returns: 0 on success ** -1 on failure ** ** CAUTION: If self is empty on entry, this routine adds an empty bucket. ** That isn't a legitimate BTree; if the caller doesn't put something in ** in the bucket (say, because of a later error), the BTree must be cleared ** to get rid of the empty bucket. */ static int BTree_grow(BTree *self, int index, int noval) { int i; Sized *v, *e = 0; BTreeItem *d; if (self->len == self->size) { if (self->size) { d = BTree_Realloc(self->data, sizeof(BTreeItem) * self->size * 2); if (d == NULL) return -1; self->data = d; self->size *= 2; } else { d = BTree_Malloc(sizeof(BTreeItem) * 2); if (d == NULL) return -1; self->data = d; self->size = 2; } } if (self->len) { d = self->data + index; v = d->child; /* Create a new object of the same type as the target value */ e = (Sized *)PyObject_CallObject((PyObject *)v->ob_type, NULL); if (e == NULL) return -1; UNLESS(PER_USE(v)) { Py_DECREF(e); return -1; } /* Now split between the original (v) and the new (e) at the midpoint*/ if (SameType_Check(self, v)) i = BTree_split((BTree *)v, -1, (BTree *)e); else i = bucket_split((Bucket *)v, -1, (Bucket *)e); PER_ALLOW_DEACTIVATION(v); if (i < 0) { Py_DECREF(e); assert(PyErr_Occurred()); return -1; } index++; d++; if (self->len > index) /* Shift up the old values one array slot */ memmove(d+1, d, sizeof(BTreeItem)*(self->len-index)); if (SameType_Check(self, v)) { COPY_KEY(d->key, BTREE(e)->data->key); /* We take the unused reference from e, so there's no reason to INCREF! */ /* INCREF_KEY(self->data[1].key); */ } else { COPY_KEY(d->key, BUCKET(e)->keys[0]); INCREF_KEY(d->key); } d->child = e; self->len++; if (self->len >= MAX_BTREE_SIZE(self) * 2) /* the root is huge */ return BTree_split_root(self, noval); } else { /* The BTree is empty. Create an empty bucket. See CAUTION in * the comments preceding. */ assert(index == 0); d = self->data; d->child = BTree_newBucket(self); if (d->child == NULL) return -1; self->len = 1; Py_INCREF(d->child); self->firstbucket = (Bucket *)d->child; } return 0; } /* Return the rightmost bucket reachable from following child pointers * from self. The caller gets a new reference to this bucket. Note that * bucket 'next' pointers are not followed: if self is an interior node * of a BTree, this returns the rightmost bucket in that node's subtree. * In case of error, returns NULL. * * self must not be a ghost; this isn't checked. The result may be a ghost. * * Pragmatics: Note that the rightmost bucket's last key is the largest * key in self's subtree. */ static Bucket * BTree_lastBucket(BTree *self) { Sized *pchild; Bucket *result; UNLESS (self->data && self->len) { IndexError(-1); /* is this the best action to take? */ return NULL; } pchild = self->data[self->len - 1].child; if (SameType_Check(self, pchild)) { self = BTREE(pchild); PER_USE_OR_RETURN(self, NULL); result = BTree_lastBucket(self); PER_UNUSE(self); } else { Py_INCREF(pchild); result = BUCKET(pchild); } return result; } static int BTree_deleteNextBucket(BTree *self) { Bucket *b; UNLESS (PER_USE(self)) return -1; b = BTree_lastBucket(self); if (b == NULL) goto err; if (Bucket_deleteNextBucket(b) < 0) goto err; Py_DECREF(b); PER_UNUSE(self); return 0; err: Py_XDECREF(b); PER_ALLOW_DEACTIVATION(self); return -1; } /* ** _BTree_clear ** ** Clears out all of the values in the BTree (firstbucket, keys, and children); ** leaving self an empty BTree. ** ** Arguments: self The BTree ** ** Returns: 0 on success ** -1 on failure ** ** Internal: Deallocation order is important. The danger is that a long ** list of buckets may get freed "at once" via decref'ing the first bucket, ** in which case a chain of consequenct Py_DECREF calls may blow the stack. ** Luckily, every bucket has a refcount of at least two, one due to being a ** BTree node's child, and another either because it's not the first bucket in ** the chain (so the preceding bucket points to it), or because firstbucket ** points to it. By clearing in the natural depth-first, left-to-right ** order, the BTree->bucket child pointers prevent Py_DECREF(bucket->next) ** calls from freeing bucket->next, and the maximum stack depth is equal ** to the height of the tree. **/ static int _BTree_clear(BTree *self) { const int len = self->len; if (self->firstbucket) { /* Obscure: The first bucket is pointed to at least by * self->firstbucket and data[0].child of whichever BTree node it's * a child of. However, if persistence is enabled then the latter * BTree node may be a ghost at this point, and so its pointers "don't * count": we can only rely on self's pointers being intact. */ #ifdef PERSISTENT ASSERT(self->firstbucket->ob_refcnt > 0, "Invalid firstbucket pointer", -1); #else ASSERT(self->firstbucket->ob_refcnt > 1, "Invalid firstbucket pointer", -1); #endif Py_DECREF(self->firstbucket); self->firstbucket = NULL; } if (self->data) { int i; if (len > 0) { /* 0 is special because key 0 is trash */ Py_DECREF(self->data[0].child); } for (i = 1; i < len; i++) { #ifdef KEY_TYPE_IS_PYOBJECT DECREF_KEY(self->data[i].key); #endif Py_DECREF(self->data[i].child); } free(self->data); self->data = NULL; } self->len = self->size = 0; return 0; } /* Set (value != 0) or delete (value=0) a tree item. If unique is non-zero, then only change if the key is new. If noval is non-zero, then don't set a value (the tree is a set). Return: -1 error 0 successful, and number of entries didn't change >0 successful, and number of entries did change Internal There are two distinct return values > 0: 1 Successful, number of entries changed, but firstbucket did not go away. 2 Successful, number of entries changed, firstbucket did go away. This can only happen on a delete (value == NULL). The caller may need to change its own firstbucket pointer, and in any case *someone* needs to adjust the 'next' pointer of the bucket immediately preceding the bucket that went away (it needs to point to the bucket immediately following the bucket that went away). */ static int _BTree_set(BTree *self, PyObject *keyarg, PyObject *value, int unique, int noval) { int changed = 0; /* did I mutate? */ int min; /* index of child I searched */ BTreeItem *d; /* self->data[min] */ int childlength; /* len(self->data[min].child) */ int status; /* our return value; and return value from callee */ int self_was_empty; /* was self empty at entry? */ KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); if (!copied) return -1; PER_USE_OR_RETURN(self, -1); self_was_empty = self->len == 0; if (self_was_empty) { /* We're empty. Make room. */ if (value) { if (BTree_grow(self, 0, noval) < 0) goto Error; } else { /* Can't delete a key from an empty BTree. */ PyErr_SetObject(PyExc_KeyError, keyarg); goto Error; } } /* Find the right child to search, and hand the work off to it. */ BTREE_SEARCH(min, self, key, goto Error); d = self->data + min; if (SameType_Check(self, d->child)) status = _BTree_set(BTREE(d->child), keyarg, value, unique, noval); else { int bucket_changed = 0; status = _bucket_set(BUCKET(d->child), keyarg, value, unique, noval, &bucket_changed); #ifdef PERSISTENT /* If a BTree contains only a single bucket, BTree.__getstate__() * includes the bucket's entire state, and the bucket doesn't get * an oid of its own. So if we have a single oid-less bucket that * changed, it's *our* oid that should be marked as changed -- the * bucket doesn't have one. */ if (bucket_changed && self->len == 1 && self->data[0].child->oid == NULL) { changed = 1; } #endif } if (status == 0) goto Done; if (status < 0) goto Error; assert(status == 1 || status == 2); /* The child changed size. Get its new size. Note that since the tree * rooted at the child changed size, so did the tree rooted at self: * our status must be >= 1 too. */ UNLESS(PER_USE(d->child)) goto Error; childlength = d->child->len; PER_UNUSE(d->child); if (value) { /* A bucket got bigger -- if it's "too big", split it. */ int toobig; assert(status == 1); /* can be 2 only on deletes */ if (SameType_Check(self, d->child)) toobig = childlength > MAX_BTREE_SIZE(d->child); else toobig = childlength > MAX_BUCKET_SIZE(d->child); if (toobig) { if (BTree_grow(self, min, noval) < 0) goto Error; changed = 1; /* BTree_grow mutated self */ } goto Done; /* and status still == 1 */ } /* A bucket got smaller. This is much harder, and despite that we * don't try to rebalance the tree. */ if (status == 2) { /* this is the last reference to child status */ /* Two problems to solve: May have to adjust our own firstbucket, * and the bucket that went away needs to get unlinked. */ if (min) { /* This wasn't our firstbucket, so no need to adjust ours (note * that it can't be the firstbucket of any node above us either). * Tell "the tree to the left" to do the unlinking. */ if (BTree_deleteNextBucket(BTREE(d[-1].child)) < 0) goto Error; status = 1; /* we solved the child's firstbucket problem */ } else { /* This was our firstbucket. Update to new firstbucket value. */ Bucket *nextbucket; UNLESS(PER_USE(d->child)) goto Error; nextbucket = BTREE(d->child)->firstbucket; PER_UNUSE(d->child); Py_XINCREF(nextbucket); Py_DECREF(self->firstbucket); self->firstbucket = nextbucket; changed = 1; /* The caller has to do the unlinking -- we can't. Also, since * it was our firstbucket, it may also be theirs. */ assert(status == 2); } } /* If the child isn't empty, we're done! We did all that was possible for * us to do with the firstbucket problems the child gave us, and since the * child isn't empty don't create any new firstbucket problems of our own. */ if (childlength) goto Done; /* The child became empty: we need to remove it from self->data. * But first, if we're a bottom-level node, we've got more bucket-fiddling * to set up. */ if (!SameType_Check(self, d->child)) { /* We're about to delete a bucket. */ if (min) { /* It's not our first bucket, so we can tell the previous * bucket to adjust its reference to it. It can't be anyone * else's first bucket either, so the caller needn't do anything. */ if (Bucket_deleteNextBucket(BUCKET(d[-1].child)) < 0) goto Error; /* status should be 1, and already is: if it were 2, the * block above would have set it to 1 in its min != 0 branch. */ assert(status == 1); } else { Bucket *nextbucket; /* It's our first bucket. We can't unlink it directly. */ /* 'changed' will be set true by the deletion code following. */ UNLESS(PER_USE(d->child)) goto Error; nextbucket = BUCKET(d->child)->next; PER_UNUSE(d->child); Py_XINCREF(nextbucket); Py_DECREF(self->firstbucket); self->firstbucket = nextbucket; status = 2; /* we're giving our caller a new firstbucket problem */ } } /* Remove the child from self->data. */ Py_DECREF(d->child); #ifdef KEY_TYPE_IS_PYOBJECT if (min) { DECREF_KEY(d->key); } else if (self->len > 1) { /* We're deleting the first child of a BTree with more than one * child. The key at d+1 is about to be shifted into slot 0, * and hence never to be referenced again (the key in slot 0 is * trash). */ DECREF_KEY((d+1)->key); } /* Else min==0 and len==1: we're emptying the BTree entirely, and * there is no key in need of decrefing. */ #endif --self->len; if (min < self->len) memmove(d, d+1, (self->len - min) * sizeof(BTreeItem)); changed = 1; Done: #ifdef PERSISTENT if (changed) { if (PER_CHANGED(self) < 0) goto Error; } #endif PER_UNUSE(self); return status; Error: assert(PyErr_Occurred()); if (self_was_empty) { /* BTree_grow may have left the BTree in an invalid state. Make * sure the tree is a legitimate empty tree. */ _BTree_clear(self); } PER_UNUSE(self); return -1; } /* ** BTree_setitem ** ** wrapper for _BTree_set ** ** Arguments: self The BTree ** key The key to insert ** v The value to insert ** ** Returns -1 on failure ** 0 on success */ static int BTree_setitem(BTree *self, PyObject *key, PyObject *v) { if (_BTree_set(self, key, v, 0, 0) < 0) return -1; return 0; } #ifdef PERSISTENT static PyObject * BTree__p_deactivate(BTree *self, PyObject *args, PyObject *keywords) { int ghostify = 1; PyObject *force = NULL; if (args && PyTuple_GET_SIZE(args) > 0) { PyErr_SetString(PyExc_TypeError, "_p_deactivate takes not positional arguments"); return NULL; } if (keywords) { int size = PyDict_Size(keywords); force = PyDict_GetItemString(keywords, "force"); if (force) size--; if (size) { PyErr_SetString(PyExc_TypeError, "_p_deactivate only accepts keyword arg force"); return NULL; } } if (self->jar && self->oid) { ghostify = self->state == cPersistent_UPTODATE_STATE; if (!ghostify && force) { if (PyObject_IsTrue(force)) ghostify = 1; if (PyErr_Occurred()) return NULL; } if (ghostify) { if (_BTree_clear(self) < 0) return NULL; PER_GHOSTIFY(self); } } Py_INCREF(Py_None); return Py_None; } #endif static PyObject * BTree_clear(BTree *self) { UNLESS (PER_USE(self)) return NULL; if (self->len) { if (_BTree_clear(self) < 0) goto err; if (PER_CHANGED(self) < 0) goto err; } PER_UNUSE(self); Py_INCREF(Py_None); return Py_None; err: PER_UNUSE(self); return NULL; } /* * Return: * * For an empty BTree (self->len == 0), None. * * For a BTree with one child (self->len == 1), and that child is a bucket, * and that bucket has a NULL oid, a one-tuple containing a one-tuple * containing the bucket's state: * * ( * ( * child[0].__getstate__(), * ), * ) * * Else a two-tuple. The first element is a tuple interleaving the BTree's * keys and direct children, of size 2*self->len - 1 (key[0] is unused and * is not saved). The second element is the firstbucket: * * ( * (child[0], key[1], child[1], key[2], child[2], ..., * key[len-1], child[len-1]), * self->firstbucket * ) * * In the above, key[i] means self->data[i].key, and similarly for child[i]. */ static PyObject * BTree_getstate(BTree *self) { PyObject *r = NULL; PyObject *o; int i, l; UNLESS (PER_USE(self)) return NULL; if (self->len) { r = PyTuple_New(self->len * 2 - 1); if (r == NULL) goto err; if (self->len == 1 && self->data->child->ob_type != self->ob_type #ifdef PERSISTENT && BUCKET(self->data->child)->oid == NULL #endif ) { /* We have just one bucket. Save its data directly. */ o = bucket_getstate((Bucket *)self->data->child); if (o == NULL) goto err; PyTuple_SET_ITEM(r, 0, o); ASSIGN(r, Py_BuildValue("(O)", r)); } else { for (i=0, l=0; i < self->len; i++) { if (i) { COPY_KEY_TO_OBJECT(o, self->data[i].key); PyTuple_SET_ITEM(r, l, o); l++; } o = (PyObject *)self->data[i].child; Py_INCREF(o); PyTuple_SET_ITEM(r,l,o); l++; } ASSIGN(r, Py_BuildValue("OO", r, self->firstbucket)); } } else { r = Py_None; Py_INCREF(r); } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } static int _BTree_setstate(BTree *self, PyObject *state, int noval) { PyObject *items, *firstbucket = NULL; BTreeItem *d; int len, l, i, copied=1; if (_BTree_clear(self) < 0) return -1; /* The state of a BTree can be one of the following: None -- an empty BTree A one-tuple -- a single bucket btree A two-tuple -- a BTree with more than one bucket See comments for BTree_getstate() for the details. */ if (state == Py_None) return 0; if (!PyArg_ParseTuple(state, "O|O:__setstate__", &items, &firstbucket)) return -1; len = PyTuple_Size(items); if (len < 0) return -1; len = (len + 1) / 2; assert(len > 0); /* If the BTree is empty, it's state is None. */ assert(self->size == 0); /* We called _BTree_clear(). */ self->data = BTree_Malloc(sizeof(BTreeItem) * len); if (self->data == NULL) return -1; self->size = len; for (i = 0, d = self->data, l = 0; i < len; i++, d++) { PyObject *v; if (i) { /* skip the first key slot */ COPY_KEY_FROM_ARG(d->key, PyTuple_GET_ITEM(items, l), copied); l++; if (!copied) return -1; INCREF_KEY(d->key); } v = PyTuple_GET_ITEM(items, l); if (PyTuple_Check(v)) { /* Handle the special case in __getstate__() for a BTree with a single bucket. */ d->child = BTree_newBucket(self); if (!d->child) return -1; if (noval) { if (_set_setstate(BUCKET(d->child), v) < 0) return -1; } else { if (_bucket_setstate(BUCKET(d->child), v) < 0) return -1; } } else { d->child = (Sized *)v; Py_INCREF(v); } l++; } if (!firstbucket) firstbucket = (PyObject *)self->data->child; if (!PyObject_IsInstance(firstbucket, (PyObject *) (noval ? &SetType : &BucketType))) { PyErr_SetString(PyExc_TypeError, "No firstbucket in non-empty BTree"); return -1; } self->firstbucket = BUCKET(firstbucket); Py_INCREF(firstbucket); #ifndef PERSISTENT /* firstbucket is also the child of some BTree node, but that node may * be a ghost if persistence is enabled. */ assert(self->firstbucket->ob_refcnt > 1); #endif self->len = len; return 0; } static PyObject * BTree_setstate(BTree *self, PyObject *arg) { int r; PER_PREVENT_DEACTIVATION(self); r = _BTree_setstate(self, arg, 0); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } #ifdef PERSISTENT /* Recognize the special cases of a BTree that's empty or contains a single * bucket. In the former case, return a borrowed reference to Py_None. * In this single-bucket case, the bucket state is embedded directly in the * BTree state, like so: * * ( * ( * thebucket.__getstate__(), * ), * ) * * When this obtains, return a borrowed reference to thebucket.__getstate__(). * Else return NULL with an exception set. The exception should always be * ConflictError then, but may be TypeError if the state makes no sense at all * for a BTree (corrupted or hostile state). */ PyObject * get_bucket_state(PyObject *t) { if (t == Py_None) return Py_None; /* an empty BTree */ if (! PyTuple_Check(t)) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected tuple or None for state"); return NULL; } if (PyTuple_GET_SIZE(t) == 2) { /* A non-degenerate BTree. */ return merge_error(-1, -1, -1, 11); } /* We're in the one-bucket case. */ if (PyTuple_GET_SIZE(t) != 1) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected 1- or 2-tuple for state"); return NULL; } t = PyTuple_GET_ITEM(t, 0); if (! PyTuple_Check(t) || PyTuple_GET_SIZE(t) != 1) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected 1-tuple containing " "bucket state"); return NULL; } t = PyTuple_GET_ITEM(t, 0); if (! PyTuple_Check(t)) { PyErr_SetString(PyExc_TypeError, "_p_resolveConflict: expected tuple for bucket state"); return NULL; } return t; } /* Tricky. The only kind of BTree conflict we can actually potentially * resolve is the special case of a BTree containing a single bucket, * in which case this becomes a fancy way of calling the bucket conflict * resolution code. */ static PyObject * BTree__p_resolveConflict(BTree *self, PyObject *args) { PyObject *s[3]; PyObject *x, *y, *z; if (!PyArg_ParseTuple(args, "OOO", &x, &y, &z)) return NULL; s[0] = get_bucket_state(x); if (s[0] == NULL) return NULL; s[1] = get_bucket_state(y); if (s[1] == NULL) return NULL; s[2] = get_bucket_state(z); if (s[2] == NULL) return NULL; if (PyObject_IsInstance((PyObject *)self, (PyObject *)&BTreeType)) x = _bucket__p_resolveConflict(OBJECT(&BucketType), s); else x = _bucket__p_resolveConflict(OBJECT(&SetType), s); if (x == NULL) return NULL; return Py_BuildValue("((N))", x); } #endif /* BTree_findRangeEnd -- Find one end, expressed as a bucket and position, for a range search. If low, return bucket and index of the smallest item >= key, otherwise return bucket and index of the largest item <= key. If exclude_equal, exact matches aren't acceptable; if one is found, move right if low, or left if !low (this is for range searches exclusive of an endpoint). Return: -1 Error; offset and bucket unchanged 0 Not found; offset and bucket unchanged 1 Correct bucket and offset stored; the caller owns a new reference to the bucket. Internal: We do binary searches in BTree nodes downward, at each step following C(i) where K(i) <= key < K(i+1). As always, K(i) <= C(i) < K(i+1) too. (See Maintainer.txt for the meaning of that notation.) That eventually leads to a bucket where we do Bucket_findRangeEnd. That usually works, but there are two cases where it can fail to find the correct answer: 1. On a low search, we find a bucket with keys >= K(i), but that doesn't imply there are keys in the bucket >= key. For example, suppose a bucket has keys in 1..100, its successor's keys are in 200..300, and we're doing a low search on 150. We'll end up in the first bucket, but there are no keys >= 150 in it. K(i+1) > key, though, and all the keys in C(i+1) >= K(i+1) > key, so the first key in the next bucket (if any) is the correct result. This is easy to find by following the bucket 'next' pointer. 2. On a high search, again that the keys in the bucket are >= K(i) doesn't imply that any key in the bucket is <= key, but it's harder for this to fail (and an earlier version of this routine didn't catch it): if K(i) itself is in the bucket, it works (then K(i) <= key is *a* key in the bucket that's in the desired range). But when keys get deleted from buckets, they aren't also deleted from BTree nodes, so there's no guarantee that K(i) is in the bucket. For example, delete the smallest key S from some bucket, and S remains in the interior BTree nodes. Do a high search for S, and the BTree nodes direct the search to the bucket S used to be in, but all keys remaining in that bucket are > S. The largest key in the *preceding* bucket (if any) is < K(i), though, and K(i) <= key, so the largest key in the preceding bucket is < key and so is the proper result. This is harder to get at efficiently, as buckets are linked only in the increasing direction. While we're searching downward, deepest_smaller is set to the node deepest in the tree where we *could* have gone to the left of C(i). The rightmost bucket in deepest_smaller's subtree is the bucket preceding the bucket we find at first. This is clumsy to get at, but efficient. */ static int BTree_findRangeEnd(BTree *self, PyObject *keyarg, int low, int exclude_equal, Bucket **bucket, int *offset) { Sized *deepest_smaller = NULL; /* last possibility to move left */ int deepest_smaller_is_btree = 0; /* Boolean; if false, it's a bucket */ Bucket *pbucket; int self_got_rebound = 0; /* Boolean; when true, deactivate self */ int result = -1; /* Until proven innocent */ int i; KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return -1; /* We don't need to: PER_USE_OR_RETURN(self, -1); because the caller does. */ UNLESS (self->data && self->len) return 0; /* Search downward until hitting a bucket, stored in pbucket. */ for (;;) { Sized *pchild; int pchild_is_btree; BTREE_SEARCH(i, self, key, goto Done); pchild = self->data[i].child; pchild_is_btree = SameType_Check(self, pchild); if (i) { deepest_smaller = self->data[i-1].child; deepest_smaller_is_btree = pchild_is_btree; } if (pchild_is_btree) { if (self_got_rebound) { PER_UNUSE(self); } self = BTREE(pchild); self_got_rebound = 1; PER_USE_OR_RETURN(self, -1); } else { pbucket = BUCKET(pchild); break; } } /* Search the bucket for a suitable key. */ i = Bucket_findRangeEnd(pbucket, keyarg, low, exclude_equal, offset); if (i < 0) goto Done; if (i > 0) { Py_INCREF(pbucket); *bucket = pbucket; result = 1; goto Done; } /* This may be one of the two difficult cases detailed in the comments. */ if (low) { Bucket *next; UNLESS(PER_USE(pbucket)) goto Done; next = pbucket->next; if (next) { result = 1; Py_INCREF(next); *bucket = next; *offset = 0; } else result = 0; PER_UNUSE(pbucket); } /* High-end search: if it's possible to go left, do so. */ else if (deepest_smaller) { if (deepest_smaller_is_btree) { UNLESS(PER_USE(deepest_smaller)) goto Done; /* We own the reference this returns. */ pbucket = BTree_lastBucket(BTREE(deepest_smaller)); PER_UNUSE(deepest_smaller); if (pbucket == NULL) goto Done; /* error */ } else { pbucket = BUCKET(deepest_smaller); Py_INCREF(pbucket); } UNLESS(PER_USE(pbucket)) goto Done; result = 1; *bucket = pbucket; /* transfer ownership to caller */ *offset = pbucket->len - 1; PER_UNUSE(pbucket); } else result = 0; /* simply not found */ Done: if (self_got_rebound) { PER_UNUSE(self); } return result; } static PyObject * BTree_maxminKey(BTree *self, PyObject *args, int min) { PyObject *key=0; Bucket *bucket = NULL; int offset, rc; int empty_tree = 1; UNLESS (PyArg_ParseTuple(args, "|O", &key)) return NULL; UNLESS (PER_USE(self)) return NULL; UNLESS (self->data && self->len) goto empty; /* Find the range */ if (key) { if ((rc = BTree_findRangeEnd(self, key, min, 0, &bucket, &offset)) <= 0) { if (rc < 0) goto err; empty_tree = 0; goto empty; } PER_UNUSE(self); UNLESS (PER_USE(bucket)) { Py_DECREF(bucket); return NULL; } } else if (min) { bucket = self->firstbucket; PER_UNUSE(self); PER_USE_OR_RETURN(bucket, NULL); Py_INCREF(bucket); offset = 0; } else { bucket = BTree_lastBucket(self); PER_UNUSE(self); UNLESS (PER_USE(bucket)) { Py_DECREF(bucket); return NULL; } assert(bucket->len); offset = bucket->len - 1; } COPY_KEY_TO_OBJECT(key, bucket->keys[offset]); PER_UNUSE(bucket); Py_DECREF(bucket); return key; empty: PyErr_SetString(PyExc_ValueError, empty_tree ? "empty tree" : "no key satisfies the conditions"); err: PER_UNUSE(self); if (bucket) { PER_UNUSE(bucket); Py_DECREF(bucket); } return NULL; } static PyObject * BTree_minKey(BTree *self, PyObject *args) { return BTree_maxminKey(self, args, 1); } static PyObject * BTree_maxKey(BTree *self, PyObject *args) { return BTree_maxminKey(self, args, 0); } /* ** BTree_rangeSearch ** ** Generates a BTreeItems object based on the two indexes passed in, ** being the range between them. ** */ static PyObject * BTree_rangeSearch(BTree *self, PyObject *args, PyObject *kw, char type) { PyObject *min = Py_None; PyObject *max = Py_None; int excludemin = 0; int excludemax = 0; int rc; Bucket *lowbucket = NULL; Bucket *highbucket = NULL; int lowoffset; int highoffset; PyObject *result; if (args) { if (! PyArg_ParseTupleAndKeywords(args, kw, "|OOii", search_keywords, &min, &max, &excludemin, &excludemax)) return NULL; } UNLESS (PER_USE(self)) return NULL; UNLESS (self->data && self->len) goto empty; /* Find the low range */ if (min != Py_None) { if ((rc = BTree_findRangeEnd(self, min, 1, excludemin, &lowbucket, &lowoffset)) <= 0) { if (rc < 0) goto err; goto empty; } } else { lowbucket = self->firstbucket; lowoffset = 0; if (excludemin) { int bucketlen; UNLESS (PER_USE(lowbucket)) goto err; bucketlen = lowbucket->len; PER_UNUSE(lowbucket); if (bucketlen > 1) lowoffset = 1; else if (self->len < 2) goto empty; else { /* move to first item in next bucket */ Bucket *next; UNLESS (PER_USE(lowbucket)) goto err; next = lowbucket->next; PER_UNUSE(lowbucket); assert(next != NULL); lowbucket = next; /* and lowoffset is still 0 */ assert(lowoffset == 0); } } Py_INCREF(lowbucket); } /* Find the high range */ if (max != Py_None) { if ((rc = BTree_findRangeEnd(self, max, 0, excludemax, &highbucket, &highoffset)) <= 0) { Py_DECREF(lowbucket); if (rc < 0) goto err; goto empty; } } else { int bucketlen; highbucket = BTree_lastBucket(self); assert(highbucket != NULL); /* we know self isn't empty */ UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; bucketlen = highbucket->len; PER_UNUSE(highbucket); highoffset = bucketlen - 1; if (excludemax) { if (highoffset > 0) --highoffset; else if (self->len < 2) goto empty_and_decref_buckets; else { /* move to last item of preceding bucket */ int status; assert(highbucket != self->firstbucket); Py_DECREF(highbucket); status = PreviousBucket(&highbucket, self->firstbucket); if (status < 0) { Py_DECREF(lowbucket); goto err; } assert(status > 0); Py_INCREF(highbucket); UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; highoffset = highbucket->len - 1; PER_UNUSE(highbucket); } } assert(highoffset >= 0); } /* It's still possible that the range is empty, even if min < max. For * example, if min=3 and max=4, and 3 and 4 aren't in the BTree, but 2 and * 5 are, then the low position points to the 5 now and the high position * points to the 2 now. They're not necessarily even in the same bucket, * so there's no trick we can play with pointer compares to get out * cheap in general. */ if (lowbucket == highbucket && lowoffset > highoffset) goto empty_and_decref_buckets; /* definitely empty */ /* The buckets differ, or they're the same and the offsets show a non- * empty range. */ if (min != Py_None && max != Py_None && /* both args user-supplied */ lowbucket != highbucket) /* and different buckets */ { KEY_TYPE first; KEY_TYPE last; int cmp; /* Have to check the hard way: see how the endpoints compare. */ UNLESS (PER_USE(lowbucket)) goto err_and_decref_buckets; COPY_KEY(first, lowbucket->keys[lowoffset]); PER_UNUSE(lowbucket); UNLESS (PER_USE(highbucket)) goto err_and_decref_buckets; COPY_KEY(last, highbucket->keys[highoffset]); PER_UNUSE(highbucket); TEST_KEY_SET_OR(cmp, first, last) goto err_and_decref_buckets; if (cmp > 0) goto empty_and_decref_buckets; } PER_UNUSE(self); result = newBTreeItems(type, lowbucket, lowoffset, highbucket, highoffset); Py_DECREF(lowbucket); Py_DECREF(highbucket); return result; err_and_decref_buckets: Py_DECREF(lowbucket); Py_DECREF(highbucket); err: PER_UNUSE(self); return NULL; empty_and_decref_buckets: Py_DECREF(lowbucket); Py_DECREF(highbucket); empty: PER_UNUSE(self); return newBTreeItems(type, 0, 0, 0, 0); } /* ** BTree_keys */ static PyObject * BTree_keys(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'k'); } /* ** BTree_values */ static PyObject * BTree_values(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'v'); } /* ** BTree_items */ static PyObject * BTree_items(BTree *self, PyObject *args, PyObject *kw) { return BTree_rangeSearch(self, args, kw, 'i'); } static PyObject * BTree_byValue(BTree *self, PyObject *omin) { PyObject *r=0, *o=0, *item=0; VALUE_TYPE min; VALUE_TYPE v; int copied=1; SetIteration it = {0, 0, 1}; UNLESS (PER_USE(self)) return NULL; COPY_VALUE_FROM_ARG(min, omin, copied); UNLESS(copied) return NULL; UNLESS (r=PyList_New(0)) goto err; it.set=BTree_rangeSearch(self, NULL, NULL, 'i'); UNLESS(it.set) goto err; if (nextBTreeItems(&it) < 0) goto err; while (it.position >= 0) { if (TEST_VALUE(it.value, min) >= 0) { UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, it.key); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); COPY_VALUE(v, it.value); NORMALIZE_VALUE(v, min); COPY_VALUE_TO_OBJECT(o, v); DECREF_VALUE(v); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); if (PyList_Append(r, item) < 0) goto err; Py_DECREF(item); item = 0; } if (nextBTreeItems(&it) < 0) goto err; } item=PyObject_GetAttr(r,sort_str); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; ASSIGN(item, PyObject_GetAttr(r, reverse_str)); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; Py_DECREF(item); finiSetIteration(&it); PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); finiSetIteration(&it); Py_XDECREF(item); return NULL; } /* ** BTree_getm */ static PyObject * BTree_getm(BTree *self, PyObject *args) { PyObject *key, *d=Py_None, *r; UNLESS (PyArg_ParseTuple(args, "O|O", &key, &d)) return NULL; if ((r=_BTree_get(self, key, 0))) return r; UNLESS (PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); Py_INCREF(d); return d; } static PyObject * BTree_has_key(BTree *self, PyObject *key) { return _BTree_get(self, key, 1); } /* Search BTree self for key. This is the sq_contains slot of the * PySequenceMethods. * * Return: * -1 error * 0 not found * 1 found */ static int BTree_contains(BTree *self, PyObject *key) { PyObject *asobj = _BTree_get(self, key, 1); int result = -1; if (asobj != NULL) { result = PyInt_AsLong(asobj) ? 1 : 0; Py_DECREF(asobj); } return result; } static PyObject * BTree_addUnique(BTree *self, PyObject *args) { int grew; PyObject *key, *v; UNLESS (PyArg_ParseTuple(args, "OO", &key, &v)) return NULL; if ((grew=_BTree_set(self, key, v, 1, 0)) < 0) return NULL; return PyInt_FromLong(grew); } /**************************************************************************/ /* Iterator support. */ /* A helper to build all the iterators for BTrees and TreeSets. * If args is NULL, the iterator spans the entire structure. Else it's an * argument tuple, with optional low and high arguments. * kind is 'k', 'v' or 'i'. * Returns a BTreeIter object, or NULL if error. */ static PyObject * buildBTreeIter(BTree *self, PyObject *args, PyObject *kw, char kind) { BTreeIter *result = NULL; BTreeItems *items = (BTreeItems *)BTree_rangeSearch(self, args, kw, kind); if (items) { result = BTreeIter_new(items); Py_DECREF(items); } return (PyObject *)result; } /* The implementation of iter(BTree_or_TreeSet); the BTree tp_iter slot. */ static PyObject * BTree_getiter(BTree *self) { return buildBTreeIter(self, NULL, NULL, 'k'); } /* The implementation of BTree.iterkeys(). */ static PyObject * BTree_iterkeys(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'k'); } /* The implementation of BTree.itervalues(). */ static PyObject * BTree_itervalues(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'v'); } /* The implementation of BTree.iteritems(). */ static PyObject * BTree_iteritems(BTree *self, PyObject *args, PyObject *kw) { return buildBTreeIter(self, args, kw, 'i'); } /* End of iterator support. */ /* Caution: Even though the _firstbucket attribute is read-only, a program could do arbitrary damage to the btree internals. For example, it could call clear() on a bucket inside a BTree. We need to decide if the convenience for inspecting BTrees is worth the risk. */ static struct PyMemberDef BTree_members[] = { {"_firstbucket", T_OBJECT, offsetof(BTree, firstbucket), RO}, {NULL} }; static struct PyMethodDef BTree_methods[] = { {"__getstate__", (PyCFunction) BTree_getstate, METH_NOARGS, "__getstate__() -> state\n\n" "Return the picklable state of the BTree."}, {"__setstate__", (PyCFunction) BTree_setstate, METH_O, "__setstate__(state)\n\n" "Set the state of the BTree."}, {"has_key", (PyCFunction) BTree_has_key, METH_O, "has_key(key)\n\n" "Return true if the BTree contains the given key."}, {"keys", (PyCFunction) BTree_keys, METH_KEYWORDS, "keys([min, max]) -> list of keys\n\n" "Returns the keys of the BTree. If min and max are supplied, only\n" "keys greater than min and less than max are returned."}, {"values", (PyCFunction) BTree_values, METH_KEYWORDS, "values([min, max]) -> list of values\n\n" "Returns the values of the BTree. If min and max are supplied, only\n" "values corresponding to keys greater than min and less than max are\n" "returned."}, {"items", (PyCFunction) BTree_items, METH_KEYWORDS, "items([min, max]) -> -- list of key, value pairs\n\n" "Returns the items of the BTree. If min and max are supplied, only\n" "items with keys greater than min and less than max are returned."}, {"byValue", (PyCFunction) BTree_byValue, METH_O, "byValue(min) -> list of value, key pairs\n\n" "Returns list of value, key pairs where the value is >= min. The\n" "list is sorted by value. Note that items() returns keys in the\n" "opposite order."}, {"get", (PyCFunction) BTree_getm, METH_VARARGS, "get(key[, default=None]) -> Value for key or default\n\n" "Return the value or the default if the key is not found."}, {"maxKey", (PyCFunction) BTree_maxKey, METH_VARARGS, "maxKey([max]) -> key\n\n" "Return the largest key in the BTree. If max is specified, return\n" "the largest key <= max."}, {"minKey", (PyCFunction) BTree_minKey, METH_VARARGS, "minKey([mi]) -> key\n\n" "Return the smallest key in the BTree. If min is specified, return\n" "the smallest key >= min."}, {"clear", (PyCFunction) BTree_clear, METH_NOARGS, "clear()\n\nRemove all of the items from the BTree."}, {"insert", (PyCFunction)BTree_addUnique, METH_VARARGS, "insert(key, value) -> 0 or 1\n\n" "Add an item if the key is not already used. Return 1 if the item was\n" "added, or 0 otherwise."}, {"update", (PyCFunction) Mapping_update, METH_O, "update(collection)\n\n Add the items from the given collection."}, {"iterkeys", (PyCFunction) BTree_iterkeys, METH_KEYWORDS, "B.iterkeys([min[,max]]) -> an iterator over the keys of B"}, {"itervalues", (PyCFunction) BTree_itervalues, METH_KEYWORDS, "B.itervalues([min[,max]]) -> an iterator over the values of B"}, {"iteritems", (PyCFunction) BTree_iteritems, METH_KEYWORDS, "B.iteritems([min[,max]]) -> an iterator over the (key, value) items of B"}, {"_check", (PyCFunction) BTree_check, METH_NOARGS, "Perform sanity check on BTree, and raise exception if flawed."}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) BTree__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) BTree__p_deactivate, METH_KEYWORDS, "_p_deactivate()\n\nReinitialize from a newly created copy."}, #endif {NULL, NULL} }; static int BTree_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "BTree", &v)) return -1; if (v) return update_from_seq(self, v); else return 0; } static void BTree_dealloc(BTree *self) { if (self->state != cPersistent_GHOST_STATE) _BTree_clear(self); cPersistenceCAPI->pertype->tp_dealloc((PyObject *)self); } static int BTree_traverse(BTree *self, visitproc visit, void *arg) { int err = 0; int i, len; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ goto Done; \ } if (self->ob_type == &BTreeType) assert(self->ob_type->tp_dictoffset == 0); /* Call our base type's traverse function. Because BTrees are * subclasses of Peristent, there must be one. */ err = cPersistenceCAPI->pertype->tp_traverse((PyObject *)self, visit, arg); if (err) goto Done; /* If this is registered with the persistence system, cleaning up cycles * is the database's problem. It would be horrid to unghostify BTree * nodes here just to chase pointers every time gc runs. */ if (self->state == cPersistent_GHOST_STATE) goto Done; len = self->len; #ifdef KEY_TYPE_IS_PYOBJECT /* Keys are Python objects so need to be traversed. Note that the * key 0 slot is unused and should not be traversed. */ for (i = 1; i < len; i++) VISIT(self->data[i].key); #endif /* Children are always pointers, and child 0 is legit. */ for (i = 0; i < len; i++) VISIT(self->data[i].child); VISIT(self->firstbucket); Done: return err; #undef VISIT } static int BTree_tp_clear(BTree *self) { if (self->state != cPersistent_GHOST_STATE) _BTree_clear(self); return 0; } /* * Return the number of elements in a BTree. nonzero is a Boolean, and * when true requests just a non-empty/empty result. Testing for emptiness * is efficient (constant-time). Getting the true length takes time * proportional to the number of leaves (buckets). * * Return: * When nonzero true: * -1 error * 0 empty * 1 not empty * When nonzero false (possibly expensive!): * -1 error * >= 0 number of elements. */ static int BTree_length_or_nonzero(BTree *self, int nonzero) { int result; Bucket *b; Bucket *next; PER_USE_OR_RETURN(self, -1); b = self->firstbucket; PER_UNUSE(self); if (nonzero) return b != NULL; result = 0; while (b) { PER_USE_OR_RETURN(b, -1); result += b->len; next = b->next; PER_UNUSE(b); b = next; } return result; } static int BTree_length( BTree *self) { return BTree_length_or_nonzero(self, 0); } static PyMappingMethods BTree_as_mapping = { (inquiry)BTree_length, /*mp_length*/ (binaryfunc)BTree_get, /*mp_subscript*/ (objobjargproc)BTree_setitem, /*mp_ass_subscript*/ }; static PySequenceMethods BTree_as_sequence = { (inquiry)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (intargfunc)0, /* sq_repeat */ (intargfunc)0, /* sq_item */ (intintargfunc)0, /* sq_slice */ (intobjargproc)0, /* sq_ass_item */ (intintobjargproc)0, /* sq_ass_slice */ (objobjproc)BTree_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static int BTree_nonzero(BTree *self) { return BTree_length_or_nonzero(self, 1); } static PyNumberMethods BTree_as_number_for_nonzero = { 0,0,0,0,0,0,0,0,0,0, (inquiry)BTree_nonzero}; static PyTypeObject BTreeType = { PyObject_HEAD_INIT(NULL) /* PyPersist_Type */ 0, /* ob_size */ MODULE_NAME MOD_NAME_PREFIX "BTree",/* tp_name */ sizeof(BTree), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)BTree_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ &BTree_as_number_for_nonzero, /* tp_as_number */ &BTree_as_sequence, /* tp_as_sequence */ &BTree_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)BTree_traverse, /* tp_traverse */ (inquiry)BTree_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTree_getiter, /* tp_iter */ 0, /* tp_iternext */ BTree_methods, /* tp_methods */ BTree_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ BTree_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; Zope-2.8.7-final/lib/python/BTrees/_IOBTree.c0000664000076400007640000000211710435502401017542 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IOBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* IOBTree - int key, object value BTree Implements a collection using int type keys and object type values */ #define PERSISTENT #define MOD_NAME_PREFIX "IO" #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 500 #define INITMODULE init_IOBTree #include "intkeymacros.h" #include "objectvaluemacros.h" #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/IIBTree.py0000664000076400007640000000130710435502401017603 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # hack to overcome dynamic-linking headache. from _IIBTree import * Zope-2.8.7-final/lib/python/BTrees/_IIBTree.c0000664000076400007640000000210410435502401017530 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IIBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* IIBTree - int key, int value BTree Implements a collection using int type keys and int type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "II" #define INITMODULE init_IIBTree #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #include "intkeymacros.h" #include "intvaluemacros.h" #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/BTreeItemsTemplate.c0000664000076400007640000004770610435502401021666 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define BTREEITEMSTEMPLATE_C "$Id: BTreeItemsTemplate.c 25186 2004-06-02 15:07:33Z jim $\n" /* A BTreeItems struct is returned from calling .items(), .keys() or * .values() on a BTree-based data structure, and is also the result of * taking slices of those. It represents a contiguous slice of a BTree. * * The start of the slice is in firstbucket, at offset first. The end of * the slice is in lastbucket, at offset last. Both endpoints are inclusive. * It must possible to get from firstbucket to lastbucket via following * bucket 'next' pointers zero or more times. firstbucket, first, lastbucket, * and last are readonly after initialization. An empty slice is represented * by firstbucket == lastbucket == currentbucket == NULL. * * 'kind' determines whether this slice represents 'k'eys alone, 'v'alues * alone, or 'i'items (key+value pairs). 'kind' is also readonly after * initialization. * * The combination of currentbucket, currentoffset and pseudoindex acts as * a search finger. Offset currentoffset in bucket currentbucket is at index * pseudoindex, where pseudoindex==0 corresponds to offset first in bucket * firstbucket, and pseudoindex==-1 corresponds to offset last in bucket * lastbucket. The function BTreeItems_seek() can be used to set this combo * correctly for any in-bounds index, and uses this combo on input to avoid * needing to search from the start (or end) on each call. Calling * BTreeItems_seek() with consecutive larger positions is very efficent. * Calling it with consecutive smaller positions is more efficient than if * a search finger weren't being used at all, but is still quadratic time * in the number of buckets in the slice. */ typedef struct { PyObject_HEAD Bucket *firstbucket; /* First bucket */ Bucket *currentbucket; /* Current bucket (search finger) */ Bucket *lastbucket; /* Last bucket */ int currentoffset; /* Offset in currentbucket */ int pseudoindex; /* search finger index */ int first; /* Start offset in firstbucket */ int last; /* End offset in lastbucket */ char kind; /* 'k', 'v', 'i' */ } BTreeItems; #define ITEMS(O)((BTreeItems*)(O)) static PyObject * newBTreeItems(char kind, Bucket *lowbucket, int lowoffset, Bucket *highbucket, int highoffset); static void BTreeItems_dealloc(BTreeItems *self) { Py_XDECREF(self->firstbucket); Py_XDECREF(self->lastbucket); Py_XDECREF(self->currentbucket); PyObject_DEL(self); } static int BTreeItems_length_or_nonzero(BTreeItems *self, int nonzero) { int r; Bucket *b, *next; b = self->firstbucket; if (b == NULL) return 0; r = self->last + 1 - self->first; if (nonzero && r > 0) /* Short-circuit if all we care about is nonempty */ return 1; if (b == self->lastbucket) return r; Py_INCREF(b); PER_USE_OR_RETURN(b, -1); while ((next = b->next)) { r += b->len; if (nonzero && r > 0) /* Short-circuit if all we care about is nonempty */ break; if (next == self->lastbucket) break; /* we already counted the last bucket */ Py_INCREF(next); PER_UNUSE(b); Py_DECREF(b); b = next; PER_USE_OR_RETURN(b, -1); } PER_UNUSE(b); Py_DECREF(b); return r >= 0 ? r : 0; } static int BTreeItems_length( BTreeItems *self) { return BTreeItems_length_or_nonzero(self, 0); } /* ** BTreeItems_seek ** ** Find the ith position in the BTreeItems. ** ** Arguments: self The BTree ** i the index to seek to, in 0 .. len(self)-1, or in ** -len(self) .. -1, as for indexing a Python sequence. ** ** ** Returns 0 if successful, -1 on failure to seek (like out-of-bounds). ** Upon successful return, index i is at offset self->currentoffset in bucket ** self->currentbucket. */ static int BTreeItems_seek(BTreeItems *self, int i) { int delta, pseudoindex, currentoffset; Bucket *b, *currentbucket; int error; pseudoindex = self->pseudoindex; currentoffset = self->currentoffset; currentbucket = self->currentbucket; if (currentbucket == NULL) goto no_match; delta = i - pseudoindex; while (delta > 0) { /* move right */ int max; /* Want to move right delta positions; the most we can move right in * this bucket is currentbucket->len - currentoffset - 1 positions. */ PER_USE_OR_RETURN(currentbucket, -1); max = currentbucket->len - currentoffset - 1; b = currentbucket->next; PER_UNUSE(currentbucket); if (delta <= max) { currentoffset += delta; pseudoindex += delta; if (currentbucket == self->lastbucket && currentoffset > self->last) goto no_match; break; } /* Move to start of next bucket. */ if (currentbucket == self->lastbucket || b == NULL) goto no_match; currentbucket = b; pseudoindex += max + 1; delta -= max + 1; currentoffset = 0; } while (delta < 0) { /* move left */ int status; /* Want to move left -delta positions; the most we can move left in * this bucket is currentoffset positions. */ if ((-delta) <= currentoffset) { currentoffset += delta; pseudoindex += delta; if (currentbucket == self->firstbucket && currentoffset < self->first) goto no_match; break; } /* Move to end of previous bucket. */ if (currentbucket == self->firstbucket) goto no_match; status = PreviousBucket(¤tbucket, self->firstbucket); if (status == 0) goto no_match; else if (status < 0) return -1; pseudoindex -= currentoffset + 1; delta += currentoffset + 1; PER_USE_OR_RETURN(currentbucket, -1); currentoffset = currentbucket->len - 1; PER_UNUSE(currentbucket); } assert(pseudoindex == i); /* Alas, the user may have mutated the bucket since the last time we * were called, and if they deleted stuff, we may be pointing into * trash memory now. */ PER_USE_OR_RETURN(currentbucket, -1); error = currentoffset < 0 || currentoffset >= currentbucket->len; PER_UNUSE(currentbucket); if (error) { PyErr_SetString(PyExc_RuntimeError, "the bucket being iterated changed size"); return -1; } Py_INCREF(currentbucket); Py_DECREF(self->currentbucket); self->currentbucket = currentbucket; self->currentoffset = currentoffset; self->pseudoindex = pseudoindex; return 0; no_match: IndexError(i); return -1; } /* Return the right kind ('k','v','i') of entry from bucket b at offset i. * b must be activated. Returns NULL on error. */ static PyObject * getBucketEntry(Bucket *b, int i, char kind) { PyObject *result = NULL; assert(b); assert(0 <= i && i < b->len); switch (kind) { case 'k': COPY_KEY_TO_OBJECT(result, b->keys[i]); break; case 'v': COPY_VALUE_TO_OBJECT(result, b->values[i]); break; case 'i': { PyObject *key; PyObject *value;; COPY_KEY_TO_OBJECT(key, b->keys[i]); if (!key) break; COPY_VALUE_TO_OBJECT(value, b->values[i]); if (!value) { Py_DECREF(key); break; } result = PyTuple_New(2); if (result) { PyTuple_SET_ITEM(result, 0, key); PyTuple_SET_ITEM(result, 1, value); } else { Py_DECREF(key); Py_DECREF(value); } break; } default: PyErr_SetString(PyExc_AssertionError, "getBucketEntry: unknown kind"); break; } return result; } /* ** BTreeItems_item ** ** Arguments: self a BTreeItems structure ** i Which item to inspect ** ** Returns: the BTreeItems_item_BTree of self->kind, i ** (ie pulls the ith item out) */ static PyObject * BTreeItems_item(BTreeItems *self, int i) { PyObject *result; if (BTreeItems_seek(self, i) < 0) return NULL; PER_USE_OR_RETURN(self->currentbucket, NULL); result = getBucketEntry(self->currentbucket, self->currentoffset, self->kind); PER_UNUSE(self->currentbucket); return result; } /* ** BTreeItems_slice ** ** Creates a new BTreeItems structure representing the slice ** between the low and high range ** ** Arguments: self The old BTreeItems structure ** ilow The start index ** ihigh The end index ** ** Returns: BTreeItems item */ static PyObject * BTreeItems_slice(BTreeItems *self, int ilow, int ihigh) { Bucket *lowbucket; Bucket *highbucket; int lowoffset; int highoffset; int length = -1; /* len(self), but computed only if needed */ /* Complications: * A Python slice never raises IndexError, but BTreeItems_seek does. * Python did only part of index normalization before calling this: * ilow may be < 0 now, and ihigh may be arbitrarily large. It's * our responsibility to clip them. * A Python slice is exclusive of the high index, but a BTreeItems * struct is inclusive on both ends. */ /* First adjust ilow and ihigh to be legit endpoints in the Python * sense (ilow inclusive, ihigh exclusive). This block duplicates the * logic from Python's list_slice function (slicing for builtin lists). */ if (ilow < 0) ilow = 0; else { if (length < 0) length = BTreeItems_length(self); if (ilow > length) ilow = length; } if (ihigh < ilow) ihigh = ilow; else { if (length < 0) length = BTreeItems_length(self); if (ihigh > length) ihigh = length; } assert(0 <= ilow && ilow <= ihigh); assert(length < 0 || ihigh <= length); /* Now adjust for that our struct is inclusive on both ends. This is * easy *except* when the slice is empty: there's no good way to spell * that in an inclusive-on-both-ends scheme. For example, if the * slice is btree.items([:0]), ilow == ihigh == 0 at this point, and if * we were to subtract 1 from ihigh that would get interpreted by * BTreeItems_seek as meaning the *entire* set of items. Setting ilow==1 * and ihigh==0 doesn't work either, as BTreeItems_seek raises IndexError * if we attempt to seek to ilow==1 when the underlying sequence is empty. * It seems simplest to deal with empty slices as a special case here. */ if (ilow == ihigh) { /* empty slice */ lowbucket = highbucket = NULL; lowoffset = 1; highoffset = 0; } else { assert(ilow < ihigh); --ihigh; /* exclusive -> inclusive */ if (BTreeItems_seek(self, ilow) < 0) return NULL; lowbucket = self->currentbucket; lowoffset = self->currentoffset; if (BTreeItems_seek(self, ihigh) < 0) return NULL; highbucket = self->currentbucket; highoffset = self->currentoffset; } return newBTreeItems(self->kind, lowbucket, lowoffset, highbucket, highoffset); } static PySequenceMethods BTreeItems_as_sequence = { (inquiry) BTreeItems_length, (binaryfunc)0, (intargfunc)0, (intargfunc) BTreeItems_item, (intintargfunc) BTreeItems_slice, }; /* Number Method items (just for nb_nonzero!) */ static int BTreeItems_nonzero(BTreeItems *self) { return BTreeItems_length_or_nonzero(self, 1); } static PyNumberMethods BTreeItems_as_number_for_nonzero = { 0,0,0,0,0,0,0,0,0,0, (inquiry)BTreeItems_nonzero}; static PyTypeObject BTreeItemsType = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ MOD_NAME_PREFIX "BTreeItems", /*tp_name*/ sizeof(BTreeItems), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor) BTreeItems_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)0, /*obsolete tp_getattr*/ (setattrfunc)0, /*obsolete tp_setattr*/ (cmpfunc)0, /*tp_compare*/ (reprfunc)0, /*tp_repr*/ &BTreeItems_as_number_for_nonzero, /*tp_as_number*/ &BTreeItems_as_sequence, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)0, /*tp_call*/ (reprfunc)0, /*tp_str*/ 0, /*tp_getattro*/ 0, /*tp_setattro*/ /* Space for future expansion */ 0L,0L, "Sequence type used to iterate over BTree items." /* Documentation string */ }; /* Returns a new BTreeItems object representing the contiguous slice from * offset lowoffset in bucket lowbucket through offset highoffset in bucket * highbucket, inclusive. Pass lowbucket == NULL for an empty slice. * The currentbucket is set to lowbucket, currentoffset ot lowoffset, and * pseudoindex to 0. kind is 'k', 'v' or 'i' (see BTreeItems struct docs). */ static PyObject * newBTreeItems(char kind, Bucket *lowbucket, int lowoffset, Bucket *highbucket, int highoffset) { BTreeItems *self; UNLESS (self = PyObject_NEW(BTreeItems, &BTreeItemsType)) return NULL; self->kind=kind; self->first=lowoffset; self->last=highoffset; if (! lowbucket || ! highbucket || (lowbucket == highbucket && lowoffset > highoffset)) { self->firstbucket = 0; self->lastbucket = 0; self->currentbucket = 0; } else { Py_INCREF(lowbucket); self->firstbucket = lowbucket; Py_INCREF(highbucket); self->lastbucket = highbucket; Py_INCREF(lowbucket); self->currentbucket = lowbucket; } self->currentoffset = lowoffset; self->pseudoindex = 0; return OBJECT(self); } static int nextBTreeItems(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); DECREF_VALUE(i->value); } if (BTreeItems_seek(ITEMS(i->set), i->position) >= 0) { Bucket *currentbucket; currentbucket = BUCKET(ITEMS(i->set)->currentbucket); UNLESS(PER_USE(currentbucket)) { /* Mark iteration terminated, so that finiSetIteration doesn't * try to redundantly decref the key and value */ i->position = -1; return -1; } COPY_KEY(i->key, currentbucket->keys[ITEMS(i->set)->currentoffset]); INCREF_KEY(i->key); COPY_VALUE(i->value, currentbucket->values[ITEMS(i->set)->currentoffset]); INCREF_VALUE(i->value); i->position ++; PER_UNUSE(currentbucket); } else { i->position = -1; PyErr_Clear(); } } return 0; } static int nextTreeSetItems(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); } if (BTreeItems_seek(ITEMS(i->set), i->position) >= 0) { Bucket *currentbucket; currentbucket = BUCKET(ITEMS(i->set)->currentbucket); UNLESS(PER_USE(currentbucket)) { /* Mark iteration terminated, so that finiSetIteration doesn't * try to redundantly decref the key and value */ i->position = -1; return -1; } COPY_KEY(i->key, currentbucket->keys[ITEMS(i->set)->currentoffset]); INCREF_KEY(i->key); i->position ++; PER_UNUSE(currentbucket); } else { i->position = -1; PyErr_Clear(); } } return 0; } /* Support for the iteration protocol new in Python 2.2. */ static PyTypeObject BTreeIter_Type; /* The type of iterator objects, returned by e.g. iter(IIBTree()). */ typedef struct { PyObject_HEAD /* We use a BTreeItems object because it's convenient and flexible. * We abuse it two ways: * 1. We set currentbucket to NULL when the iteration is finished. * 2. We don't bother keeping pseudoindex in synch. */ BTreeItems *pitems; } BTreeIter; /* Return a new iterator object, to traverse the keys and/or values * represented by pitems. pitems must not be NULL. Returns NULL if error. */ static BTreeIter * BTreeIter_new(BTreeItems *pitems) { BTreeIter *result; assert(pitems != NULL); result = PyObject_New(BTreeIter, &BTreeIter_Type); if (result) { Py_INCREF(pitems); result->pitems = pitems; } return result; } /* The iterator's tp_dealloc slot. */ static void BTreeIter_dealloc(BTreeIter *bi) { Py_DECREF(bi->pitems); PyObject_Del(bi); } /* The implementation of the iterator's tp_iternext slot. Returns "the next" * item; returns NULL if error; returns NULL without setting an error if the * iteration is exhausted (that's the way to terminate the iteration protocol). */ static PyObject * BTreeIter_next(BTreeIter *bi, PyObject *args) { PyObject *result = NULL; /* until proven innocent */ BTreeItems *items = bi->pitems; int i = items->currentoffset; Bucket *bucket = items->currentbucket; if (bucket == NULL) /* iteration termination is sticky */ return NULL; PER_USE_OR_RETURN(bucket, NULL); if (i >= bucket->len) { /* We never leave this routine normally with i >= len: somebody * else mutated the current bucket. */ PyErr_SetString(PyExc_RuntimeError, "the bucket being iterated changed size"); /* Arrange for that this error is sticky too. */ items->currentoffset = INT_MAX; goto Done; } /* Build the result object, from bucket at offset i. */ result = getBucketEntry(bucket, i, items->kind); /* Advance position for next call. */ if (bucket == items->lastbucket && i >= items->last) { /* Next call should terminate the iteration. */ Py_DECREF(items->currentbucket); items->currentbucket = NULL; } else { ++i; if (i >= bucket->len) { Py_XINCREF(bucket->next); items->currentbucket = bucket->next; Py_DECREF(bucket); i = 0; } items->currentoffset = i; } Done: PER_UNUSE(bucket); return result; } static PyObject * BTreeIter_getiter(PyObject *it) { Py_INCREF(it); return it; } static PyTypeObject BTreeIter_Type = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ MOD_NAME_PREFIX "-iterator", /* tp_name */ sizeof(BTreeIter), /* tp_basicsize */ 0, /* tp_itemsize */ /* methods */ (destructor)BTreeIter_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /*PyObject_GenericGetAttr,*/ /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTreeIter_getiter, /* tp_iter */ (iternextfunc)BTreeIter_next, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ }; Zope-2.8.7-final/lib/python/BTrees/floatvaluemacros.h0000664000076400007640000000167610435502401021537 0ustar ajungajung #define VALUEMACROS_H "$Id: floatvaluemacros.h 28617 2004-12-09 23:43:03Z tim_one $\n" #define VALUE_TYPE float #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) #define VALUE_SAME(VALUE, TARGET) ( (VALUE) == (TARGET) ) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define VALUE_PARSE "f" #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (V=(E)) #define COPY_VALUE_TO_OBJECT(O, K) O=PyFloat_FromDouble(K) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if (PyFloat_Check(ARG)) TARGET = (float)PyFloat_AsDouble(ARG); \ else if (PyInt_Check(ARG)) TARGET = (float)PyInt_AsLong(ARG); \ else { \ PyErr_SetString(PyExc_TypeError, "expected float or int value"); \ (STATUS)=0; (TARGET)=0; } #define NORMALIZE_VALUE(V, MIN) ((MIN) > 0) ? ((V)/=(MIN)) : 0 #define MERGE_DEFAULT 1.0f #define MERGE(O1, w1, O2, w2) ((O1)*(w1)+(O2)*(w2)) #define MERGE_WEIGHT(O, w) ((O)*(w)) Zope-2.8.7-final/lib/python/BTrees/_IFBTree.c0000664000076400007640000000211210435502401017524 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _IFBTree.c 28613 2004-12-09 21:02:35Z tim_one $\n" /* IIBTree - int key, int value BTree Implements a collection using int type keys and int type values */ /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "IF" #define INITMODULE init_IFBTree #define DEFAULT_MAX_BUCKET_SIZE 120 #define DEFAULT_MAX_BTREE_SIZE 500 #include "intkeymacros.h" #include "floatvaluemacros.h" #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/_fsBTree.c0000664000076400007640000000533610435502401017651 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _fsBTree.c 41596 2006-02-11 21:03:06Z tseaver $\n" /* fsBTree - FileStorage index BTree This BTree implements a mapping from 2-character strings to six-character strings. This allows us to efficiently store a FileStorage index as a nested mapping of 6-character oid prefix to mapping of 2-character oid suffix to 6-character (byte) file positions. */ typedef unsigned char char2[2]; typedef unsigned char char6[6]; /* Setup template macros */ #define PERSISTENT #define MOD_NAME_PREFIX "fs" #define INITMODULE init_fsBTree #define DEFAULT_MAX_BUCKET_SIZE 500 #define DEFAULT_MAX_BTREE_SIZE 500 /*#include "intkeymacros.h"*/ #define KEYMACROS_H "$Id: _fsBTree.c 41596 2006-02-11 21:03:06Z tseaver $\n" #define KEY_TYPE char2 #undef KEY_TYPE_IS_PYOBJECT #define KEY_CHECK(K) (PyString_Check(K) && PyString_GET_SIZE(K)==2) #define TEST_KEY_SET_OR(V, K, T) if ( ( (V) = ((*(K) < *(T) || (*(K) == *(T) && (K)[1] < (T)[1])) ? -1 : ((*(K) == *(T) && (K)[1] == (T)[1]) ? 0 : 1)) ), 0 ) #define DECREF_KEY(KEY) #define INCREF_KEY(k) #define COPY_KEY(KEY, E) (*(KEY)=*(E), (KEY)[1]=(E)[1]) #define COPY_KEY_TO_OBJECT(O, K) O=PyString_FromStringAndSize((const char*)K,2) #define COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) \ if (KEY_CHECK(ARG)) memcpy(TARGET, PyString_AS_STRING(ARG), 2); else { \ PyErr_SetString(PyExc_TypeError, "expected two-character string key"); \ (STATUS)=0; } /*#include "intvaluemacros.h"*/ #define VALUEMACROS_H "$Id: _fsBTree.c 41596 2006-02-11 21:03:06Z tseaver $\n" #define VALUE_TYPE char6 #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) memcmp(K,T,6) #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (memcpy(V, E, 6)) #define COPY_VALUE_TO_OBJECT(O, K) O=PyString_FromStringAndSize((const char*)K,6) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if ((PyString_Check(ARG) && PyString_GET_SIZE(ARG)==6)) \ memcpy(TARGET, PyString_AS_STRING(ARG), 6); else { \ PyErr_SetString(PyExc_TypeError, "expected six-character string key"); \ (STATUS)=0; } #define NORMALIZE_VALUE(V, MIN) #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/_OIBTree.c0000664000076400007640000000211710435502401017542 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _OIBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* OIBTree - object key, int value BTree Implements a collection using object type keys and int type values */ #define PERSISTENT #define MOD_NAME_PREFIX "OI" #define INITMODULE init_OIBTree #define DEFAULT_MAX_BUCKET_SIZE 60 #define DEFAULT_MAX_BTREE_SIZE 250 #include "objectkeymacros.h" #include "intvaluemacros.h" #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/__init__.py0000664000076400007640000000003410435502401020113 0ustar ajungajung# This is a Python package. Zope-2.8.7-final/lib/python/BTrees/Maintainer.txt0000664000076400007640000003545010435502401020644 0ustar ajungajungThis document provides information for developers who maintain or extend BTrees. Macros ====== BTrees are defined using a "template", roughly akin to a a C++ template. To create a new family of BTrees, create a source file that defines macros used to handle differences in key and value types: Configuration Macros MASTER_ID A string to hold an RCS/CVS Id key to be included in compiled binaries. MOD_NAME_PREFIX A string (like "IO" or "OO") that provides the prefix used for the module. This gets used to generate type names and the internal module name string. DEFAULT_MAX_BUCKET_SIZE An int giving the maximum bucket size (number of key/value pairs). When a bucket gets larger than this due to an insertion *into a BTREE*, it splits. Inserting into a bucket directly doesn't split, and functions that produce a bucket output (e.g., union()) also have no bound on how large a bucket may get. Someday this will be tunable on BTree instances. DEFAULT_MAX_BTREE_SIZE An int giving the maximum size (number of children) of an internal btree node. Someday this will be tunable on BTree instances. Macros for Keys KEY_TYPE The C type declaration for keys (e.g., int or PyObject*). KEY_TYPE_IS_PYOBJECT Define if KEY_TYPE is a PyObject*, else undef. KEY_CHECK(K) Tests whether the PyObject* K can be converted to the (C) key type (KEY_TYPE). The macro should return a boolean (zero for false, non-zero for true). When it returns false, its caller should probably set a TypeError exception. TEST_KEY_SET_OR(V, K, T) Like Python's cmp(). Compares K(ey) to T(arget), where K & T are C values of type KEY_TYPE. V is assigned an int value depending on the outcome: < 0 if K < T == 0 if K == T > 0 if K > T This macro acts like an 'if', where the following statement is executed only if a Python exception has been raised because the values could not be compared. DECREF_KEY(K) K is a value of KEY_TYPE. If KEY_TYPE is a flavor of PyObject*, write this to do Py_DECREF(K). Else (e.g., KEY_TYPE is int) make it a nop. INCREF_KEY(K) K is a value of KEY_TYPE. If KEY_TYPE is a flavor of PyObject*, write this to do Py_INCREF(K). Else (e.g., KEY_TYPE is int) make it a nop. COPY_KEY(K, E) Like K=E. Copy a key from E to K, both of KEY_TYPE. Note that this doesn't decref K or incref E when KEY_TYPE is a PyObject*; the caller is responsible for keeping refcounts straight. COPY_KEY_TO_OBJECT(O, K) Roughly like O=K. O is a PyObject*, and the macro must build a Python object form of K, assign it to O, and ensure that O owns the reference to its new value. It may do this by creating a new Python object based on K (e.g., PyInt_FromLong(K) when KEY_TYPE is int), or simply by doing Py_INCREF(K) if KEY_TYPE is a PyObject*. COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) Copy an argument to the target without creating a new reference to ARG. ARG is a PyObject*, and TARGET is of type KEY_TYPE. If this can't be done (for example, KEY_CHECK(ARG) returns false), set a Python error and set status to 0. If there is no error, leave status alone. Macros for Values VALUE_TYPE The C type declaration for values (e.g., int or PyObject*). VALUE_TYPE_IS_PYOBJECT Define if VALUE_TYPE is a PyObject*, else undef. TEST_VALUE(X, Y) Like Python's cmp(). Compares X to Y, where X & Y are C values of type VALUE_TYPE. The macro returns an int, with value < 0 if X < Y == 0 if X == Y > 0 if X > Y Bug: There is no provision for determining whether the comparison attempt failed (set a Python exception). DECREF_VALUE(K) Like DECREF_KEY, except applied to values of VALUE_TYPE. INCREF_VALUE(K) Like INCREF_KEY, except applied to values of VALUE_TYPE. COPY_VALUE(K, E) Like COPY_KEY, except applied to values of VALUE_TYPE. COPY_VALUE_TO_OBJECT(O, K) Like COPY_KEY_TO_OBJECT, except applied to values of VALUE_TYPE. COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) Like COPY_KEY_FROM_ARG, except applied to values of VALUE_TYPE. NORMALIZE_VALUE(V, MIN) Normalize the value, V, using the parameter MIN. This is almost certainly a YAGNI. It is a no op for most types. For integers, V is replaced by V/MIN only if MIN > 0. Macros for Set Operations MERGE_DEFAULT A value of VALUE_TYPE specifying the value to associate with set elements when sets are merged with mappings via weighed union or weighted intersection. MERGE(O1, w1, O2, w2) Performs a weighted merge of two values, O1 and O2, using weights w1 and w2. The result must be of VALUE_TYPE. Note that weighted unions and weighted intersections are not enabled if this macro is left undefined. MERGE_WEIGHT(O, w) Computes a weighted value for O. The result must be of VALUE_TYPE. This is used for "filling out" weighted unions, i.e. to compute a weighted value for keys that appear in only one of the input mappings. If left undefined, MERGE_WEIGHT defaults to #define MERGE_WEIGHT(O, w) (O) MULTI_INT_UNION The value doesn't matter. If defined, SetOpTemplate.c compiles code for a multiunion() function (compute a union of many input sets at high speed). This currently makes sense only for structures with integer keys. BTree Clues =========== More or less random bits of helpful info. + In papers and textbooks, this flavor of BTree is usually called a B+-Tree, where "+" is a superscript. + All keys and all values live in the bucket leaf nodes. Keys in interior (BTree) nodes merely serve to guide a search efficiently toward the correct leaf. + When a key is deleted, it's physically removed from the bucket it's in, but this doesn't propagate back up the tree: since keys in interior nodes only serve to guide searches, it's OK-- and saves time --to leave "stale" keys in interior nodes. + No attempt is made to rebalance the tree after a deletion, unless a bucket thereby becomes entirely empty. "Classic BTrees" do rebalance, keeping all buckets at least half full (provided there are enough keys in the entire tree to fill half a bucket). The tradeoffs are murky. Pathological cases in the presence of deletion do exist. Pathologies include trees tending toward only one key per bucket, and buckets at differing depths (all buckets are at the same depth in a classic BTree). + DEFAULT_MAX_BUCKET_SIZE and DEFAULT_MAX_BTREE_SIZE are chosen mostly to "even out" pickle sizes in storage. That's why, e.g., an IIBTree has larger values than an OOBTree: pickles store ints more efficiently than they can store arbitrary Python objects. + In a non-empty BTree, every bucket node contains at least one key, and every BTree node contains at least one child and a non-NULL firstbucket pointer. However, a BTree node may not contain any keys. + An empty BTree consists solely of a BTree node with len==0 and firstbucket==NULL. + Although a BTree can become unbalanced under a mix of inserts and deletes (meaning both that there's nothing stronger that can be said about buckets than that they're not empty, and that buckets can appear at different depths), a BTree node always has children of the same kind: they're all buckets, or they're all BTree nodes. The BTREE_SEARCH Macro ====================== For notational ease, consider a fixed BTree node x, and let K(i) mean x->data.key[i] C(i) mean all the keys reachable from x->data.child[i] For each i in 0 to x->len-1 inclusive, K(i) <= C(i) < K(i+1) is a BTree node invariant, where we pretend that K(0) holds a key smaller than any possible key, and K(x->len) holds a key larger than any possible key. (Note that K(x->len) doesn't actually exist, and K(0) is never used although space for it exists in non-empty BTree nodes.) When searching for a key k, then, the child pointer we want to follow is the one at index i such that K(i) <= k < K(i+1). There can be at most one such i, since the K(i) are strictly increasing. And there is at least one such i provided the tree isn't empty (so that 0 < len). For the moment, assume the tree isn't empty (we'll get back to that later). The macro's chief loop invariant is K(lo) < k < K(hi) This holds trivially at the start, since lo is set to 0, and hi to x->len, and we pretend K(0) is minus infinity and K(len) is plus infinity. Inside the loop, if K(i) < k we set lo to i, and if K(i) > k we set hi to i. These obviously preserve the invariant. If K(i) == k, the loop breaks and sets the result to i, and since K(i) == k in that case i is obviously the correct result. Other cases depend on how i = floor((lo + hi)/2) works, exactly. Suppose lo + d = hi for some d >= 0. Then i = floor((lo + lo + d)/2) = floor(lo + d/2) = lo + floor(d/2). So: a. [d == 0] (lo == i == hi) if and only if (lo == hi). b. [d == 1] (lo == i < hi) if and only if (lo+1 == hi). c. [d > 1] (lo < i < hi) if and only if (lo+1 < hi). If the node is empty (x->len == 0), then lo==i==hi==0 at the start, and the loop exits immediately (the first "i > lo" test fails), without entering the body. Else lo < hi at the start, and the invariant K(lo) < k < K(hi) holds. If lo+1 < hi, we're in case #c: i is strictly between lo and hi, so the loop body is entered, and regardless of whether the body sets the new lo or the new hi to i, the new lo is strictly less than the new hi, and the difference between the new lo and new hi is strictly less than the difference between the old lo and old hi. So long as the new lo + 1 remains < the new hi, we stay in this case. We can't stay in this case forever, though: because hi-lo decreases on each trip but remains > 0, lo+1 == hi must eventually become true. (In fact, it becomes true quickly, in about log2(x->len) trips; the point is more that lo doesn't equal hi when the loop ends, it has to end with lo+1==hi and i==lo). Then we're in case #b: i==lo==hi-1 then, and the loop exits. The invariant still holds, with lo==i and hi==lo+1==i+1: K(i) < k < K(i+1) so i is again the correct answer. Optimization points: + Division by 2 is done via shift rather via "/2". These are signed ints, and almost all C compilers treat signed int division as truncating, and shifting is not the same as truncation for signed int division. The compiler has no way to know these values aren't negative, so has to generate longer-winded code for "/2". But we know these values aren't negative, and exploit it. + The order of _cmp comparisons matters. We're in an interior BTree node, and are looking at only a tiny fraction of all the keys that exist. So finding the key exactly in this node is unlikely, and checking _cmp == 0 is a waste of time to the same extent. It doesn't matter whether we check for _cmp < 0 or _cmp > 0 first, so long as we do both before worrying about equality. + At the start of a routine, it's better to run this macro even if x->len is 0 (check for that afterwards). We just called a function and so probably drained the pipeline. If the first thing we do then is read up self->len and check it against 0, we just sit there waiting for the data to get read up, and then another immediate test-and-branch, and for a very unlikely case (BTree nodes are rarely empty). It's better to get into the loop right away so the normal case makes progress ASAP. The BUCKET_SEARCH Macro ======================= This has a different job than BTREE_SEARCH: the key 0 slot is legitimate in a bucket, and we want to find the index at which the key belongs. If the key is larger than the bucket's largest key, a new slot at index len is where it belongs, else it belongs at the smallest i with keys[i] >= the key we're looking for. We also need to know whether or not the key is present (BTREE_SEARCH didn't care; it only wanted to find the next node to search). The mechanics of the search are quite similar, though. The primary loop invariant changes to (say we're searching for key k): K(lo-1) < k < K(hi) where K(i) means keys[i], and we pretend K(-1) is minus infinity and K(len) is plus infinity. If the bucket is empty, lo=hi=i=0 at the start, the loop body is never entered, and the macro sets INDEX to 0 and ABSENT to true. That's why _cmp is initialized to 1 (_cmp becomes ABSENT). Else the bucket is not empty, lok, hi is set to i, preserving that K[hi] = K[i] > k. If the loop exits after either of those, _cmp != 0, so ABSENT becomes true. If K[i]=k, the loop breaks, so that INDEX becomes i, and ABSENT becomes false (_cmp=0 in this case). The same case analysis for BTREE_SEARCH on lo and hi holds here: a. (lo == i == hi) if and only if (lo == hi). b. (lo == i < hi) if and only if (lo+1 == hi). c. (lo < i < hi) if and only if (lo+1 < hi). So long as lo+1 < hi, we're in case #c, and either break with equality (in which case the right results are obviously computed) or narrow the range. If equality doesn't obtain, the range eventually narrows to cases #a or #b. To go from #c to #a, we must have lo+2==hi at the start, and K[i]=K[lo+1] key), because when it pays it narrows the range more (we get a little boost from setting lo=i+1 in this case; the other case sets hi=i, which isn't as much of a narrowing). Zope-2.8.7-final/lib/python/BTrees/intkeymacros.h0000664000076400007640000000116110435502401020665 0ustar ajungajung #define KEYMACROS_H "$Id: intkeymacros.h 24682 2004-05-14 19:32:43Z tim_one $\n" #define KEY_TYPE int #undef KEY_TYPE_IS_PYOBJECT #define KEY_CHECK PyInt_Check #define TEST_KEY_SET_OR(V, K, T) if ( ( (V) = (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) ) , 0 ) #define DECREF_KEY(KEY) #define INCREF_KEY(k) #define COPY_KEY(KEY, E) (KEY=(E)) #define COPY_KEY_TO_OBJECT(O, K) O=PyInt_FromLong(K) #define COPY_KEY_FROM_ARG(TARGET, ARG, STATUS) \ if (PyInt_Check(ARG)) TARGET=PyInt_AS_LONG(ARG); else { \ PyErr_SetString(PyExc_TypeError, "expected integer key"); \ (STATUS)=0; (TARGET)=0; } #define MULTI_INT_UNION 1 Zope-2.8.7-final/lib/python/BTrees/convert.py0000664000076400007640000000217110435502401020040 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # TODO: does this script still serve a purpose? Writing this in 2005, # "old btree" doesn't mean much to me. import transaction def convert(old, new, threshold=200, f=None): "Utility for converting old btree to new" n=0 for k, v in old.items(): if f is not None: v=f(v) new[k]=v n=n+1 if n > threshold: transaction.savepoint() old._p_jar.cacheMinimize() n=0 transaction.savepoint() old._p_jar.cacheMinimize() Zope-2.8.7-final/lib/python/BTrees/objectkeymacros.h0000664000076400007640000000072010435502401021341 0ustar ajungajung#define KEYMACROS_H "$Id: objectkeymacros.h 24682 2004-05-14 19:32:43Z tim_one $\n" #define KEY_TYPE PyObject * #define KEY_TYPE_IS_PYOBJECT #define TEST_KEY_SET_OR(V, KEY, TARGET) if ( ( (V) = PyObject_Compare((KEY),(TARGET)) ), PyErr_Occurred() ) #define INCREF_KEY(k) Py_INCREF(k) #define DECREF_KEY(KEY) Py_DECREF(KEY) #define COPY_KEY(KEY, E) KEY=(E) #define COPY_KEY_TO_OBJECT(O, K) O=(K); Py_INCREF(O) #define COPY_KEY_FROM_ARG(TARGET, ARG, S) TARGET=(ARG) Zope-2.8.7-final/lib/python/BTrees/BucketTemplate.c0000664000076400007640000013132610435502401021070 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define BUCKETTEMPLATE_C "$Id: BucketTemplate.c 38115 2005-08-26 20:40:45Z tim_one $\n" /* Use BUCKET_SEARCH to find the index at which a key belongs. * INDEX An int lvalue to hold the index i such that KEY belongs at * SELF->keys[i]. Note that this will equal SELF->len if KEY * is larger than the bucket's largest key. Else it's the * smallest i such that SELF->keys[i] >= KEY. * ABSENT An int lvalue to hold a Boolean result, true (!= 0) if the * key is absent, false (== 0) if the key is at INDEX. * SELF A pointer to a Bucket node. * KEY The key you're looking for, of type KEY_TYPE. * ONERROR What to do if key comparison raises an exception; for example, * perhaps 'return NULL'. * * See Maintainer.txt for discussion: this is optimized in subtle ways. * It's recommended that you call this at the start of a routine, waiting * to check for self->len == 0 after (if an empty bucket is special in * context; INDEX becomes 0 and ABSENT becomes true if this macro is run * with an empty SELF, and that may be all the invoker needs to know). */ #define BUCKET_SEARCH(INDEX, ABSENT, SELF, KEY, ONERROR) { \ int _lo = 0; \ int _hi = (SELF)->len; \ int _i; \ int _cmp = 1; \ for (_i = _hi >> 1; _lo < _hi; _i = (_lo + _hi) >> 1) { \ TEST_KEY_SET_OR(_cmp, (SELF)->keys[_i], (KEY)) \ ONERROR; \ if (_cmp < 0) _lo = _i + 1; \ else if (_cmp == 0) break; \ else _hi = _i; \ } \ (INDEX) = _i; \ (ABSENT) = _cmp; \ } /* ** _bucket_get ** ** Search a bucket for a given key. ** ** Arguments ** self The bucket ** keyarg The key to look for ** has_key Boolean; if true, return a true/false result; else return ** the value associated with the key. ** ** Return ** If has_key: ** Returns the Python int 0 if the key is absent, else returns ** has_key itself as a Python int. A BTree caller generally passes ** the depth of the bucket for has_key, so a true result returns ** the bucket depth then. ** Note that has_key should be true when searching set buckets. ** If not has_key: ** If the key is present, returns the associated value, and the ** caller owns the reference. Else returns NULL and sets KeyError. ** Whether or not has_key: ** If a comparison sets an exception, returns NULL. */ static PyObject * _bucket_get(Bucket *self, PyObject *keyarg, int has_key) { int i, cmp; KEY_TYPE key; PyObject *r = NULL; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return NULL; UNLESS (PER_USE(self)) return NULL; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (has_key) r = PyInt_FromLong(cmp ? 0 : has_key); else { if (cmp == 0) { COPY_VALUE_TO_OBJECT(r, self->values[i]); } else PyErr_SetObject(PyExc_KeyError, keyarg); } Done: PER_UNUSE(self); return r; } static PyObject * bucket_getitem(Bucket *self, PyObject *key) { return _bucket_get(self, key, 0); } /* ** Bucket_grow ** ** Resize a bucket. ** ** Arguments: self The bucket. ** newsize The new maximum capacity. If < 0, double the ** current size unless the bucket is currently empty, ** in which case use MIN_BUCKET_ALLOC. ** noval Boolean; if true, allocate only key space and not ** value space ** ** Returns: -1 on error, and MemoryError exception is set ** 0 on success */ static int Bucket_grow(Bucket *self, int newsize, int noval) { KEY_TYPE *keys; VALUE_TYPE *values; if (self->size) { if (newsize < 0) newsize = self->size * 2; if (newsize < 0) /* int overflow */ goto Overflow; UNLESS (keys = BTree_Realloc(self->keys, sizeof(KEY_TYPE) * newsize)) return -1; UNLESS (noval) { values = BTree_Realloc(self->values, sizeof(VALUE_TYPE) * newsize); if (values == NULL) { free(keys); return -1; } self->values = values; } self->keys = keys; } else { if (newsize < 0) newsize = MIN_BUCKET_ALLOC; UNLESS (self->keys = BTree_Malloc(sizeof(KEY_TYPE) * newsize)) return -1; UNLESS (noval) { self->values = BTree_Malloc(sizeof(VALUE_TYPE) * newsize); if (self->values == NULL) { free(self->keys); self->keys = NULL; return -1; } } } self->size = newsize; return 0; Overflow: PyErr_NoMemory(); return -1; } /* So far, bucket_append is called only by multiunion_m(), so is called * only when MULTI_INT_UNION is defined. Flavors of BTree/Bucket that * don't support MULTI_INT_UNION don't call bucket_append (yet), and * gcc complains if bucket_append is compiled in those cases. So only * compile bucket_append if it's going to be used. */ #ifdef MULTI_INT_UNION /* * Append a slice of the "from" bucket to self. * * self Append (at least keys) to this bucket. self must be activated * upon entry, and remains activated at exit. If copyValues * is true, self must be empty or already have a non-NULL values * pointer. self's access and modification times aren't updated. * from The bucket from which to take keys, and possibly values. from * must be activated upon entry, and remains activated at exit. * If copyValues is true, from must have a non-NULL values * pointer. self and from must not be the same. from's access * time isn't updated. * i, n The slice from[i : i+n] is appended to self. Must have * i >= 0, n > 0 and i+n <= from->len. * copyValues Boolean. If true, copy values from the slice as well as keys. * In this case, from must have a non-NULL values pointer, and * self must too (unless self is empty, in which case a values * vector will be allocated for it). * overallocate Boolean. If self doesn't have enough room upon entry to hold * all the appended stuff, then if overallocate is false exactly * enough room will be allocated to hold the new stuff, else if * overallocate is true an excess will be allocated. overallocate * may be a good idea if you expect to append more stuff to self * later; else overallocate should be false. * * CAUTION: If self is empty upon entry (self->size == 0), and copyValues is * false, then no space for values will get allocated. This can be a trap if * the caller intends to copy values itself. * * Return * -1 Error. * 0 OK. */ static int bucket_append(Bucket *self, Bucket *from, int i, int n, int copyValues, int overallocate) { int newlen; assert(self && from && self != from); assert(i >= 0); assert(n > 0); assert(i+n <= from->len); /* Make room. */ newlen = self->len + n; if (newlen > self->size) { int newsize = newlen; if (overallocate) /* boost by 25% -- pretty arbitrary */ newsize += newsize >> 2; if (Bucket_grow(self, newsize, ! copyValues) < 0) return -1; } assert(newlen <= self->size); /* Copy stuff. */ memcpy(self->keys + self->len, from->keys + i, n * sizeof(KEY_TYPE)); if (copyValues) { assert(self->values); assert(from->values); memcpy(self->values + self->len, from->values + i, n * sizeof(VALUE_TYPE)); } self->len = newlen; /* Bump refcounts. */ #ifdef KEY_TYPE_IS_PYOBJECT { int j; PyObject **p = from->keys + i; for (j = 0; j < n; ++j, ++p) { Py_INCREF(*p); } } #endif #ifdef VALUE_TYPE_IS_PYOBJECT if (copyValues) { int j; PyObject **p = from->values + i; for (j = 0; j < n; ++j, ++p) { Py_INCREF(*p); } } #endif return 0; } #endif /* MULTI_INT_UNION */ /* ** _bucket_set: Assign a value to a key in a bucket, delete a key+value ** pair, or just insert a key. ** ** Arguments ** self The bucket ** keyarg The key to look for ** v The value to associate with key; NULL means delete the key. ** If NULL, it's an error (KeyError) if the key isn't present. ** Note that if this is a set bucket, and you want to insert ** a new set element, v must be non-NULL although its exact ** value will be ignored. Passing Py_None is good for this. ** unique Boolean; when true, don't replace the value if the key is ** already present. ** noval Boolean; when true, operate on keys only (ignore values) ** changed ignored on input ** ** Return ** -1 on error ** 0 on success and the # of bucket entries didn't change ** 1 on success and the # of bucket entries did change ** *changed If non-NULL, set to 1 on any mutation of the bucket. */ static int _bucket_set(Bucket *self, PyObject *keyarg, PyObject *v, int unique, int noval, int *changed) { int i, cmp; KEY_TYPE key; /* Subtle: there may or may not be a value. If there is, we need to * check its type early, so that in case of error we can get out before * mutating the bucket. But because value isn't used on all paths, if * we don't initialize value then gcc gives a nuisance complaint that * value may be used initialized (it can't be, but gcc doesn't know * that). So we initialize it. However, VALUE_TYPE can be various types, * including int, PyObject*, and char[6], so it's a puzzle to spell * initialization. It so happens that {0} is a valid initializer for all * these types. */ VALUE_TYPE value = {0}; /* squash nuisance warning */ int result = -1; /* until proven innocent */ int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS(copied) return -1; /* Copy the value early (if needed), so that in case of error a * pile of bucket mutations don't need to be undone. */ if (v && !noval) { COPY_VALUE_FROM_ARG(value, v, copied); UNLESS(copied) return -1; } UNLESS (PER_USE(self)) return -1; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (cmp == 0) { /* The key exists, at index i. */ if (v) { /* The key exists at index i, and there's a new value. * If unique, we're not supposed to replace it. If noval, or this * is a set bucket (self->values is NULL), there's nothing to do. */ if (unique || noval || self->values == NULL) { result = 0; goto Done; } /* The key exists at index i, and we need to replace the value. */ #ifdef VALUE_SAME /* short-circuit if no change */ if (VALUE_SAME(self->values[i], value)) { result = 0; goto Done; } #endif if (changed) *changed = 1; DECREF_VALUE(self->values[i]); COPY_VALUE(self->values[i], value); INCREF_VALUE(self->values[i]); if (PER_CHANGED(self) >= 0) result = 0; goto Done; } /* The key exists at index i, and should be deleted. */ DECREF_KEY(self->keys[i]); self->len--; if (i < self->len) memmove(self->keys + i, self->keys + i+1, sizeof(KEY_TYPE)*(self->len - i)); if (self->values) { DECREF_VALUE(self->values[i]); if (i < self->len) memmove(self->values + i, self->values + i+1, sizeof(VALUE_TYPE)*(self->len - i)); } if (! self->len) { self->size = 0; free(self->keys); self->keys = NULL; if (self->values) { free(self->values); self->values = NULL; } } if (changed) *changed = 1; if (PER_CHANGED(self) >= 0) result = 1; goto Done; } /* The key doesn't exist, and belongs at index i. */ if (!v) { /* Can't delete a non-existent key. */ PyErr_SetObject(PyExc_KeyError, keyarg); goto Done; } /* The key doesn't exist and should be inserted at index i. */ if (self->len == self->size && Bucket_grow(self, -1, noval) < 0) goto Done; if (self->len > i) { memmove(self->keys + i + 1, self->keys + i, sizeof(KEY_TYPE) * (self->len - i)); if (self->values) { memmove(self->values + i + 1, self->values + i, sizeof(VALUE_TYPE) * (self->len - i)); } } COPY_KEY(self->keys[i], key); INCREF_KEY(self->keys[i]); if (! noval) { COPY_VALUE(self->values[i], value); INCREF_VALUE(self->values[i]); } self->len++; if (changed) *changed = 1; if (PER_CHANGED(self) >= 0) result = 1; Done: PER_UNUSE(self); return result; } /* ** bucket_setitem ** ** wrapper for _bucket_setitem (eliminates +1 return code) ** ** Arguments: self The bucket ** key The key to insert under ** v The value to insert ** ** Returns 0 on success ** -1 on failure */ static int bucket_setitem(Bucket *self, PyObject *key, PyObject *v) { if (_bucket_set(self, key, v, 0, 0, 0) < 0) return -1; return 0; } /** ** Accepts a sequence of 2-tuples, or any object with an items() ** method that returns an iterable object producing 2-tuples. */ static int update_from_seq(PyObject *map, PyObject *seq) { PyObject *iter, *o, *k, *v; int err = -1; /* One path creates a new seq object. The other path has an INCREF of the seq argument. So seq must always be DECREFed on the way out. */ /* Use items() if it's not a sequence. Alas, PySequence_Check() * returns true for a PeristentMapping or PersistentDict, and we * want to use items() in those cases too. */ if (!PySequence_Check(seq) || /* or it "looks like a dict" */ PyObject_HasAttrString(seq, "iteritems")) { PyObject *items; items = PyObject_GetAttrString(seq, "items"); if (items == NULL) return -1; seq = PyObject_CallObject(items, NULL); Py_DECREF(items); if (seq == NULL) return -1; } else Py_INCREF(seq); iter = PyObject_GetIter(seq); if (iter == NULL) goto err; while (1) { o = PyIter_Next(iter); if (o == NULL) { if (PyErr_Occurred()) goto err; else break; } if (!PyTuple_Check(o) || PyTuple_GET_SIZE(o) != 2) { Py_DECREF(o); PyErr_SetString(PyExc_TypeError, "Sequence must contain 2-item tuples"); goto err; } k = PyTuple_GET_ITEM(o, 0); v = PyTuple_GET_ITEM(o, 1); if (PyObject_SetItem(map, k, v) < 0) { Py_DECREF(o); goto err; } Py_DECREF(o); } err = 0; err: Py_DECREF(iter); Py_DECREF(seq); return err; } static PyObject * Mapping_update(PyObject *self, PyObject *seq) { if (update_from_seq(self, seq) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } /* ** bucket_split ** ** Splits one bucket into two ** ** Arguments: self The bucket ** index the index of the key to split at (O.O.B use midpoint) ** next the new bucket to split into ** ** Returns: 0 on success ** -1 on failure */ static int bucket_split(Bucket *self, int index, Bucket *next) { int next_size; ASSERT(self->len > 1, "split of empty bucket", -1); if (index < 0 || index >= self->len) index = self->len / 2; next_size = self->len - index; next->keys = BTree_Malloc(sizeof(KEY_TYPE) * next_size); if (!next->keys) return -1; memcpy(next->keys, self->keys + index, sizeof(KEY_TYPE) * next_size); if (self->values) { next->values = BTree_Malloc(sizeof(VALUE_TYPE) * next_size); if (!next->values) { free(next->keys); next->keys = NULL; return -1; } memcpy(next->values, self->values + index, sizeof(VALUE_TYPE) * next_size); } next->size = next_size; next->len = next_size; self->len = index; next->next = self->next; Py_INCREF(next); self->next = next; if (PER_CHANGED(self) < 0) return -1; return 0; } /* Set self->next to self->next->next, i.e. unlink self's successor from * the chain. * * Return: * -1 error * 0 OK */ static int Bucket_deleteNextBucket(Bucket *self) { int result = -1; /* until proven innocent */ Bucket *successor; PER_USE_OR_RETURN(self, -1); successor = self->next; if (successor) { Bucket *next; /* Before: self -> successor -> next * After: self --------------> next */ UNLESS (PER_USE(successor)) goto Done; next = successor->next; PER_UNUSE(successor); Py_XINCREF(next); /* it may be NULL, of course */ self->next = next; Py_DECREF(successor); if (PER_CHANGED(self) < 0) goto Done; } result = 0; Done: PER_UNUSE(self); return result; } /* Bucket_findRangeEnd -- Find the index of a range endpoint (possibly) contained in a bucket. Arguments: self The bucket keyarg The key to match against low Boolean; true for low end of range, false for high exclude_equal Boolean; if true, don't accept an exact match, and if there is one then move right if low and left if !low. offset The output offset If low true, *offset <- index of the smallest item >= key, if low false the index of the largest item <= key. In either case, if there is no such index, *offset is left alone and 0 is returned. Return: 0 No suitable index exists; *offset has not been changed 1 The correct index was stored into *offset -1 Error Example: Suppose the keys are [2, 4], and exclude_equal is false. Searching for 2 sets *offset to 0 and returns 1 regardless of low. Searching for 4 sets *offset to 1 and returns 1 regardless of low. Searching for 1: If low true, sets *offset to 0, returns 1. If low false, returns 0. Searching for 3: If low true, sets *offset to 1, returns 1. If low false, sets *offset to 0, returns 1. Searching for 5: If low true, returns 0. If low false, sets *offset to 1, returns 1. The 1, 3 and 5 examples are the same when exclude_equal is true. */ static int Bucket_findRangeEnd(Bucket *self, PyObject *keyarg, int low, int exclude_equal, int *offset) { int i, cmp; int result = -1; /* until proven innocent */ KEY_TYPE key; int copied = 1; COPY_KEY_FROM_ARG(key, keyarg, copied); UNLESS (copied) return -1; UNLESS (PER_USE(self)) return -1; BUCKET_SEARCH(i, cmp, self, key, goto Done); if (cmp == 0) { /* exact match at index i */ if (exclude_equal) { /* but we don't want an exact match */ if (low) ++i; else --i; } } /* Else keys[i-1] < key < keys[i], picturing infinities at OOB indices, * and i has the smallest item > key, which is correct for low. */ else if (! low) /* i-1 has the largest item < key (unless i-1 is 0OB) */ --i; result = 0 <= i && i < self->len; if (result) *offset = i; Done: PER_UNUSE(self); return result; } static PyObject * Bucket_maxminKey(Bucket *self, PyObject *args, int min) { PyObject *key=0; int rc, offset; int empty_bucket = 1; if (args && ! PyArg_ParseTuple(args, "|O", &key)) return NULL; PER_USE_OR_RETURN(self, NULL); UNLESS (self->len) goto empty; /* Find the low range */ if (key) { if ((rc = Bucket_findRangeEnd(self, key, min, 0, &offset)) <= 0) { if (rc < 0) return NULL; empty_bucket = 0; goto empty; } } else if (min) offset = 0; else offset = self->len -1; COPY_KEY_TO_OBJECT(key, self->keys[offset]); PER_UNUSE(self); return key; empty: PyErr_SetString(PyExc_ValueError, empty_bucket ? "empty bucket" : "no key satisfies the conditions"); PER_UNUSE(self); return NULL; } static PyObject * Bucket_minKey(Bucket *self, PyObject *args) { return Bucket_maxminKey(self, args, 1); } static PyObject * Bucket_maxKey(Bucket *self, PyObject *args) { return Bucket_maxminKey(self, args, 0); } static int Bucket_rangeSearch(Bucket *self, PyObject *args, PyObject *kw, int *low, int *high) { PyObject *min = Py_None; PyObject *max = Py_None; int excludemin = 0; int excludemax = 0; int rc; if (args) { if (! PyArg_ParseTupleAndKeywords(args, kw, "|OOii", search_keywords, &min, &max, &excludemin, &excludemax)) return -1; } UNLESS (self->len) goto empty; /* Find the low range */ if (min != Py_None) { rc = Bucket_findRangeEnd(self, min, 1, excludemin, low); if (rc < 0) return -1; if (rc == 0) goto empty; } else { *low = 0; if (excludemin) { if (self->len < 2) goto empty; ++*low; } } /* Find the high range */ if (max != Py_None) { rc = Bucket_findRangeEnd(self, max, 0, excludemax, high); if (rc < 0) return -1; if (rc == 0) goto empty; } else { *high = self->len - 1; if (excludemax) { if (self->len < 2) goto empty; --*high; } } /* If min < max to begin with, it's quite possible that low > high now. */ if (*low <= *high) return 0; empty: *low = 0; *high = -1; return 0; } /* ** bucket_keys ** ** Generate a list of all keys in the bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns: list of bucket keys */ static PyObject * bucket_keys(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r = NULL, *key; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; r = PyList_New(high-low+1); if (r == NULL) goto err; for (i=low; i <= high; i++) { COPY_KEY_TO_OBJECT(key, self->keys[i]); if (PyList_SetItem(r, i-low , key) < 0) goto err; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } /* ** bucket_values ** ** Generate a list of all values in the bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns list of values */ static PyObject * bucket_values(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r=0, *v; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; UNLESS (r=PyList_New(high-low+1)) goto err; for (i=low; i <= high; i++) { COPY_VALUE_TO_OBJECT(v, self->values[i]); UNLESS (v) goto err; if (PyList_SetItem(r, i-low, v) < 0) goto err; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); return NULL; } /* ** bucket_items ** ** Returns a list of all items in a bucket ** ** Arguments: self The Bucket ** args (unused) ** ** Returns: list of all items in the bucket */ static PyObject * bucket_items(Bucket *self, PyObject *args, PyObject *kw) { PyObject *r=0, *o=0, *item=0; int i, low, high; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &low, &high) < 0) goto err; UNLESS (r=PyList_New(high-low+1)) goto err; for (i=low; i <= high; i++) { UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, self->keys[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); COPY_VALUE_TO_OBJECT(o, self->values[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); if (PyList_SetItem(r, i-low, item) < 0) goto err; item = 0; } PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); Py_XDECREF(item); return NULL; } static PyObject * bucket_byValue(Bucket *self, PyObject *omin) { PyObject *r=0, *o=0, *item=0; VALUE_TYPE min; VALUE_TYPE v; int i, l, copied=1; PER_USE_OR_RETURN(self, NULL); COPY_VALUE_FROM_ARG(min, omin, copied); UNLESS(copied) return NULL; for (i=0, l=0; i < self->len; i++) if (TEST_VALUE(self->values[i], min) >= 0) l++; UNLESS (r=PyList_New(l)) goto err; for (i=0, l=0; i < self->len; i++) { if (TEST_VALUE(self->values[i], min) < 0) continue; UNLESS (item = PyTuple_New(2)) goto err; COPY_KEY_TO_OBJECT(o, self->keys[i]); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 1, o); COPY_VALUE(v, self->values[i]); NORMALIZE_VALUE(v, min); COPY_VALUE_TO_OBJECT(o, v); DECREF_VALUE(v); UNLESS (o) goto err; PyTuple_SET_ITEM(item, 0, o); if (PyList_SetItem(r, l, item) < 0) goto err; l++; item = 0; } item=PyObject_GetAttr(r,sort_str); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; ASSIGN(item, PyObject_GetAttr(r, reverse_str)); UNLESS (item) goto err; ASSIGN(item, PyObject_CallObject(item, NULL)); UNLESS (item) goto err; Py_DECREF(item); PER_UNUSE(self); return r; err: PER_UNUSE(self); Py_XDECREF(r); Py_XDECREF(item); return NULL; } static int _bucket_clear(Bucket *self) { const int len = self->len; /* Don't declare i at this level. If neither keys nor values are * PyObject*, i won't be referenced, and you'll get a nuisance compiler * wng for declaring it here. */ self->len = self->size = 0; if (self->next) { Py_DECREF(self->next); self->next = NULL; } /* Silence compiler warning about unused variable len for the case when neither key nor value is an object, i.e. II. */ (void)len; if (self->keys) { #ifdef KEY_TYPE_IS_PYOBJECT int i; for (i = 0; i < len; ++i) DECREF_KEY(self->keys[i]); #endif free(self->keys); self->keys = NULL; } if (self->values) { #ifdef VALUE_TYPE_IS_PYOBJECT int i; for (i = 0; i < len; ++i) DECREF_VALUE(self->values[i]); #endif free(self->values); self->values = NULL; } return 0; } #ifdef PERSISTENT static PyObject * bucket__p_deactivate(Bucket *self, PyObject *args, PyObject *keywords) { int ghostify = 1; PyObject *force = NULL; if (args && PyTuple_GET_SIZE(args) > 0) { PyErr_SetString(PyExc_TypeError, "_p_deactivate takes not positional arguments"); return NULL; } if (keywords) { int size = PyDict_Size(keywords); force = PyDict_GetItemString(keywords, "force"); if (force) size--; if (size) { PyErr_SetString(PyExc_TypeError, "_p_deactivate only accepts keyword arg force"); return NULL; } } if (self->jar && self->oid) { ghostify = self->state == cPersistent_UPTODATE_STATE; if (!ghostify && force) { if (PyObject_IsTrue(force)) ghostify = 1; if (PyErr_Occurred()) return NULL; } if (ghostify) { if (_bucket_clear(self) < 0) return NULL; PER_GHOSTIFY(self); } } Py_INCREF(Py_None); return Py_None; } #endif static PyObject * bucket_clear(Bucket *self, PyObject *args) { PER_USE_OR_RETURN(self, NULL); if (self->len) { if (_bucket_clear(self) < 0) return NULL; if (PER_CHANGED(self) < 0) goto err; } PER_UNUSE(self); Py_INCREF(Py_None); return Py_None; err: PER_UNUSE(self); return NULL; } /* * Return: * * For a set bucket (self->values is NULL), a one-tuple or two-tuple. The * first element is a tuple of keys, of length self->len. The second element * is the next bucket, present if and only if next is non-NULL: * * ( * (keys[0], keys[1], ..., keys[len-1]), * next iff non-NULL> * ) * * For a mapping bucket (self->values is not NULL), a one-tuple or two-tuple. * The first element is a tuple interleaving keys and values, of length * 2 * self->len. The second element is the next bucket, present iff next is * non-NULL: * * ( * (keys[0], values[0], keys[1], values[1], ..., * keys[len-1], values[len-1]), * next iff non-NULL> * ) */ static PyObject * bucket_getstate(Bucket *self) { PyObject *o = NULL, *items = NULL, *state; int i, len, l; PER_USE_OR_RETURN(self, NULL); len = self->len; if (self->values) { /* Bucket */ items = PyTuple_New(len * 2); if (items == NULL) goto err; for (i = 0, l = 0; i < len; i++) { COPY_KEY_TO_OBJECT(o, self->keys[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, l, o); l++; COPY_VALUE_TO_OBJECT(o, self->values[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, l, o); l++; } } else { /* Set */ items = PyTuple_New(len); if (items == NULL) goto err; for (i = 0; i < len; i++) { COPY_KEY_TO_OBJECT(o, self->keys[i]); if (o == NULL) goto err; PyTuple_SET_ITEM(items, i, o); } } if (self->next) state = Py_BuildValue("OO", items, self->next); else state = Py_BuildValue("(O)", items); Py_DECREF(items); PER_UNUSE(self); return state; err: PER_UNUSE(self); Py_XDECREF(items); return NULL; } static int _bucket_setstate(Bucket *self, PyObject *state) { PyObject *k, *v, *items; Bucket *next = NULL; int i, l, len, copied=1; KEY_TYPE *keys; VALUE_TYPE *values; if (!PyArg_ParseTuple(state, "O|O:__setstate__", &items, &next)) return -1; len = PyTuple_Size(items); if (len < 0) return -1; len /= 2; for (i = self->len; --i >= 0; ) { DECREF_KEY(self->keys[i]); DECREF_VALUE(self->values[i]); } self->len = 0; if (self->next) { Py_DECREF(self->next); self->next = NULL; } if (len > self->size) { keys = BTree_Realloc(self->keys, sizeof(KEY_TYPE)*len); if (keys == NULL) return -1; values = BTree_Realloc(self->values, sizeof(VALUE_TYPE)*len); if (values == NULL) return -1; self->keys = keys; self->values = values; self->size = len; } for (i=0, l=0; i < len; i++) { k = PyTuple_GET_ITEM(items, l); l++; v = PyTuple_GET_ITEM(items, l); l++; COPY_KEY_FROM_ARG(self->keys[i], k, copied); if (!copied) return -1; COPY_VALUE_FROM_ARG(self->values[i], v, copied); if (!copied) return -1; INCREF_KEY(self->keys[i]); INCREF_VALUE(self->values[i]); } self->len = len; if (next) { self->next = next; Py_INCREF(next); } return 0; } static PyObject * bucket_setstate(Bucket *self, PyObject *state) { int r; PER_PREVENT_DEACTIVATION(self); r = _bucket_setstate(self, state); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * bucket_has_key(Bucket *self, PyObject *key) { return _bucket_get(self, key, 1); } /* Search bucket self for key. This is the sq_contains slot of the * PySequenceMethods. * * Return: * -1 error * 0 not found * 1 found */ static int bucket_contains(Bucket *self, PyObject *key) { PyObject *asobj = _bucket_get(self, key, 1); int result = -1; if (asobj != NULL) { result = PyInt_AsLong(asobj) ? 1 : 0; Py_DECREF(asobj); } return result; } /* ** bucket_getm ** */ static PyObject * bucket_getm(Bucket *self, PyObject *args) { PyObject *key, *d=Py_None, *r; if (!PyArg_ParseTuple(args, "O|O:get", &key, &d)) return NULL; r = _bucket_get(self, key, 0); if (r) return r; if (!PyErr_ExceptionMatches(PyExc_KeyError)) return NULL; PyErr_Clear(); Py_INCREF(d); return d; } /**************************************************************************/ /* Iterator support. */ /* A helper to build all the iterators for Buckets and Sets. * If args is NULL, the iterator spans the entire structure. Else it's an * argument tuple, with optional low and high arguments. * kind is 'k', 'v' or 'i'. * Returns a BTreeIter object, or NULL if error. */ static PyObject * buildBucketIter(Bucket *self, PyObject *args, PyObject *kw, char kind) { BTreeItems *items; int lowoffset, highoffset; BTreeIter *result = NULL; PER_USE_OR_RETURN(self, NULL); if (Bucket_rangeSearch(self, args, kw, &lowoffset, &highoffset) < 0) goto Done; items = (BTreeItems *)newBTreeItems(kind, self, lowoffset, self, highoffset); if (items == NULL) goto Done; result = BTreeIter_new(items); /* win or lose, we're done */ Py_DECREF(items); Done: PER_UNUSE(self); return (PyObject *)result; } /* The implementation of iter(Bucket_or_Set); the Bucket tp_iter slot. */ static PyObject * Bucket_getiter(Bucket *self) { return buildBucketIter(self, NULL, NULL, 'k'); } /* The implementation of Bucket.iterkeys(). */ static PyObject * Bucket_iterkeys(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'k'); } /* The implementation of Bucket.itervalues(). */ static PyObject * Bucket_itervalues(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'v'); } /* The implementation of Bucket.iteritems(). */ static PyObject * Bucket_iteritems(Bucket *self, PyObject *args, PyObject *kw) { return buildBucketIter(self, args, kw, 'i'); } /* End of iterator support. */ #ifdef PERSISTENT static PyObject *merge_error(int p1, int p2, int p3, int reason); static PyObject *bucket_merge(Bucket *s1, Bucket *s2, Bucket *s3); static PyObject * _bucket__p_resolveConflict(PyObject *ob_type, PyObject *s[3]) { PyObject *result = NULL; /* guilty until proved innocent */ Bucket *b[3] = {NULL, NULL, NULL}; PyObject *meth = NULL; PyObject *a = NULL; int i; for (i = 0; i < 3; i++) { PyObject *r; b[i] = (Bucket*)PyObject_CallObject((PyObject *)ob_type, NULL); if (b[i] == NULL) goto Done; if (s[i] == Py_None) /* None is equivalent to empty, for BTrees */ continue; meth = PyObject_GetAttr((PyObject *)b[i], __setstate___str); if (meth == NULL) goto Done; a = PyTuple_New(1); if (a == NULL) goto Done; PyTuple_SET_ITEM(a, 0, s[i]); Py_INCREF(s[i]); r = PyObject_CallObject(meth, a); /* b[i].__setstate__(s[i]) */ if (r == NULL) goto Done; Py_DECREF(r); Py_DECREF(a); Py_DECREF(meth); a = meth = NULL; } if (b[0]->next != b[1]->next || b[0]->next != b[2]->next) merge_error(-1, -1, -1, 0); else result = bucket_merge(b[0], b[1], b[2]); Done: Py_XDECREF(meth); Py_XDECREF(a); Py_XDECREF(b[0]); Py_XDECREF(b[1]); Py_XDECREF(b[2]); return result; } static PyObject * bucket__p_resolveConflict(Bucket *self, PyObject *args) { PyObject *s[3]; if (!PyArg_ParseTuple(args, "OOO", &s[0], &s[1], &s[2])) return NULL; return _bucket__p_resolveConflict((PyObject *)self->ob_type, s); } #endif /* Caution: Even though the _next attribute is read-only, a program could do arbitrary damage to the btree internals. For example, it could call clear() on a bucket inside a BTree. We need to decide if the convenience for inspecting BTrees is worth the risk. */ static struct PyMemberDef Bucket_members[] = { {"_next", T_OBJECT, offsetof(Bucket, next)}, {NULL} }; static struct PyMethodDef Bucket_methods[] = { {"__getstate__", (PyCFunction) bucket_getstate, METH_NOARGS, "__getstate__() -- Return the picklable state of the object"}, {"__setstate__", (PyCFunction) bucket_setstate, METH_O, "__setstate__() -- Set the state of the object"}, {"keys", (PyCFunction) bucket_keys, METH_KEYWORDS, "keys([min, max]) -- Return the keys"}, {"has_key", (PyCFunction) bucket_has_key, METH_O, "has_key(key) -- Test whether the bucket contains the given key"}, {"clear", (PyCFunction) bucket_clear, METH_VARARGS, "clear() -- Remove all of the items from the bucket"}, {"update", (PyCFunction) Mapping_update, METH_O, "update(collection) -- Add the items from the given collection"}, {"maxKey", (PyCFunction) Bucket_maxKey, METH_VARARGS, "maxKey([key]) -- Find the maximum key\n\n" "If an argument is given, find the maximum <= the argument"}, {"minKey", (PyCFunction) Bucket_minKey, METH_VARARGS, "minKey([key]) -- Find the minimum key\n\n" "If an argument is given, find the minimum >= the argument"}, {"values", (PyCFunction) bucket_values, METH_KEYWORDS, "values([min, max]) -- Return the values"}, {"items", (PyCFunction) bucket_items, METH_KEYWORDS, "items([min, max])) -- Return the items"}, {"byValue", (PyCFunction) bucket_byValue, METH_O, "byValue(min) -- " "Return value-keys with values >= min and reverse sorted by values"}, {"get", (PyCFunction) bucket_getm, METH_VARARGS, "get(key[,default]) -- Look up a value\n\n" "Return the default (or None) if the key is not found."}, {"iterkeys", (PyCFunction) Bucket_iterkeys, METH_KEYWORDS, "B.iterkeys([min[,max]]) -> an iterator over the keys of B"}, {"itervalues", (PyCFunction) Bucket_itervalues, METH_KEYWORDS, "B.itervalues([min[,max]]) -> an iterator over the values of B"}, {"iteritems", (PyCFunction) Bucket_iteritems, METH_KEYWORDS, "B.iteritems([min[,max]]) -> an iterator over the (key, value) items of B"}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) bucket__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) bucket__p_deactivate, METH_KEYWORDS, "_p_deactivate() -- Reinitialize from a newly created copy"}, #endif {NULL, NULL} }; static int Bucket_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "Bucket", &v)) return -1; if (v) return update_from_seq(self, v); else return 0; } static void bucket_dealloc(Bucket *self) { if (self->state != cPersistent_GHOST_STATE) _bucket_clear(self); cPersistenceCAPI->pertype->tp_dealloc((PyObject *)self); } static int bucket_traverse(Bucket *self, visitproc visit, void *arg) { int err = 0; int i, len; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ goto Done; \ } /* Call our base type's traverse function. Because buckets are * subclasses of Peristent, there must be one. */ err = cPersistenceCAPI->pertype->tp_traverse((PyObject *)self, visit, arg); if (err) goto Done; /* If this is registered with the persistence system, cleaning up cycles * is the database's problem. It would be horrid to unghostify buckets * here just to chase pointers every time gc runs. */ if (self->state == cPersistent_GHOST_STATE) goto Done; len = self->len; (void)i; /* if neither keys nor values are PyObject*, "i" is otherwise unreferenced and we get a nuisance compiler wng */ #ifdef KEY_TYPE_IS_PYOBJECT /* Keys are Python objects so need to be traversed. */ for (i = 0; i < len; i++) VISIT(self->keys[i]); #endif #ifdef VALUE_TYPE_IS_PYOBJECT if (self->values != NULL) { /* self->values exists (this is a mapping bucket, not a set bucket), * and are Python objects, so need to be traversed. */ for (i = 0; i < len; i++) VISIT(self->values[i]); } #endif VISIT(self->next); Done: return err; #undef VISIT } static int bucket_tp_clear(Bucket *self) { if (self->state != cPersistent_GHOST_STATE) _bucket_clear(self); return 0; } /* Code to access Bucket objects as mappings */ static int Bucket_length( Bucket *self) { int r; UNLESS (PER_USE(self)) return -1; r = self->len; PER_UNUSE(self); return r; } static PyMappingMethods Bucket_as_mapping = { (inquiry)Bucket_length, /*mp_length*/ (binaryfunc)bucket_getitem, /*mp_subscript*/ (objobjargproc)bucket_setitem, /*mp_ass_subscript*/ }; static PySequenceMethods Bucket_as_sequence = { (inquiry)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (intargfunc)0, /* sq_repeat */ (intargfunc)0, /* sq_item */ (intintargfunc)0, /* sq_slice */ (intobjargproc)0, /* sq_ass_item */ (intintobjargproc)0, /* sq_ass_slice */ (objobjproc)bucket_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static PyObject * bucket_repr(Bucket *self) { PyObject *i, *r; char repr[10000]; int rv; i = bucket_items(self, NULL, NULL); if (!i) return NULL; r = PyObject_Repr(i); Py_DECREF(i); if (!r) { return NULL; } rv = PyOS_snprintf(repr, sizeof(repr), "%s(%s)", self->ob_type->tp_name, PyString_AS_STRING(r)); if (rv > 0 && rv < sizeof(repr)) { Py_DECREF(r); return PyString_FromStringAndSize(repr, strlen(repr)); } else { /* The static buffer wasn't big enough */ int size; PyObject *s; /* 3 for the parens and the null byte */ size = strlen(self->ob_type->tp_name) + PyString_GET_SIZE(r) + 3; s = PyString_FromStringAndSize(NULL, size); if (!s) { Py_DECREF(r); return r; } PyOS_snprintf(PyString_AS_STRING(s), size, "%s(%s)", self->ob_type->tp_name, PyString_AS_STRING(r)); Py_DECREF(r); return s; } } static PyTypeObject BucketType = { PyObject_HEAD_INIT(NULL) /* PyPersist_Type */ 0, /* ob_size */ MODULE_NAME MOD_NAME_PREFIX "Bucket",/* tp_name */ sizeof(Bucket), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)bucket_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)bucket_repr, /* tp_repr */ 0, /* tp_as_number */ &Bucket_as_sequence, /* tp_as_sequence */ &Bucket_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)bucket_traverse, /* tp_traverse */ (inquiry)bucket_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)Bucket_getiter, /* tp_iter */ 0, /* tp_iternext */ Bucket_methods, /* tp_methods */ Bucket_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ Bucket_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; static int nextBucket(SetIteration *i) { if (i->position >= 0) { UNLESS(PER_USE(BUCKET(i->set))) return -1; if (i->position) { DECREF_KEY(i->key); DECREF_VALUE(i->value); } if (i->position < BUCKET(i->set)->len) { COPY_KEY(i->key, BUCKET(i->set)->keys[i->position]); INCREF_KEY(i->key); COPY_VALUE(i->value, BUCKET(i->set)->values[i->position]); INCREF_VALUE(i->value); i->position ++; } else { i->position = -1; PER_ACCESSED(BUCKET(i->set)); } PER_ALLOW_DEACTIVATION(BUCKET(i->set)); } return 0; } Zope-2.8.7-final/lib/python/BTrees/MergeTemplate.c0000664000076400007640000002623710435502401020716 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define MERGETEMPLATE_C "$Id: MergeTemplate.c 29676 2005-03-24 23:04:52Z tim_one $\n" /**************************************************************************** Set operations ****************************************************************************/ static int merge_output(Bucket *r, SetIteration *i, int mapping) { if (r->len >= r->size && Bucket_grow(r, -1, !mapping) < 0) return -1; COPY_KEY(r->keys[r->len], i->key); INCREF_KEY(r->keys[r->len]); if (mapping) { COPY_VALUE(r->values[r->len], i->value); INCREF_VALUE(r->values[r->len]); } r->len++; return 0; } /* The "reason" argument is a little integer giving "a reason" for the * error. In the Zope3 codebase, these are mapped to explanatory strings * via zodb/btrees/interfaces.py. */ static PyObject * merge_error(int p1, int p2, int p3, int reason) { PyObject *r; UNLESS (r=Py_BuildValue("iiii", p1, p2, p3, reason)) r=Py_None; if (ConflictError == NULL) { ConflictError = PyExc_ValueError; Py_INCREF(ConflictError); } PyErr_SetObject(ConflictError, r); if (r != Py_None) { Py_DECREF(r); } return NULL; } /* It's hard to explain "the rules" for bucket_merge, in large part because * any automatic conflict-resolution scheme is going to be incorrect for * some endcases of *some* app. The scheme here is pretty conservative, * and should be OK for most apps. It's easier to explain what the code * allows than what it forbids: * * Leaving things alone: it's OK if both s2 and s3 leave a piece of s1 * alone (don't delete the key, and don't change the value). * * Key deletion: a transaction (s2 or s3) can delete a key (from s1), but * only if the other transaction (of s2 and s3) doesn't delete the same key. * However, it's not OK for s2 and s3 to, between them, end up deleting all * the keys. This is a higher-level constraint, due to that the caller of * bucket_merge() doesn't have enough info to unlink the resulting empty * bucket from its BTree correctly. It's also not OK if s2 or s3 are empty, * because the transaction that emptied the bucket unlinked the bucket from * the tree, and nothing we do here can get it linked back in again. * * Key insertion: s2 or s3 can add a new key, provided the other transaction * doesn't insert the same key. It's not OK even if they insert the same * pair. * * Mapping value modification: s2 or s3 can modify the value associated * with a key in s1, provided the other transaction doesn't make a * modification of the same key to a different value. It's OK if s2 and s3 * both give the same new value to the key while it's hard to be precise about * why, this doesn't seem consistent with that it's *not* OK for both to add * a new key mapping to the same value). */ static PyObject * bucket_merge(Bucket *s1, Bucket *s2, Bucket *s3) { Bucket *r=0; PyObject *s; SetIteration i1 = {0,0,0}, i2 = {0,0,0}, i3 = {0,0,0}; int cmp12, cmp13, cmp23, mapping, set; /* If either "after" bucket is empty, punt. */ if (s2->len == 0 || s3->len == 0) { merge_error(-1, -1, -1, 12); goto err; } if (initSetIteration(&i1, OBJECT(s1), 1) < 0) goto err; if (initSetIteration(&i2, OBJECT(s2), 1) < 0) goto err; if (initSetIteration(&i3, OBJECT(s3), 1) < 0) goto err; mapping = i1.usesValue | i2.usesValue | i3.usesValue; set = !mapping; if (mapping) r = (Bucket *)PyObject_CallObject((PyObject *)&BucketType, NULL); else r = (Bucket *)PyObject_CallObject((PyObject *)&SetType, NULL); if (r == NULL) goto err; if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; if (i3.next(&i3) < 0) goto err; /* Consult zodb/btrees/interfaces.py for the meaning of the last * argument passed to merge_error(). */ /* TODO: This isn't passing on errors raised by value comparisons. */ while (i1.position >= 0 && i2.position >= 0 && i3.position >= 0) { TEST_KEY_SET_OR(cmp12, i1.key, i2.key) goto err; TEST_KEY_SET_OR(cmp13, i1.key, i3.key) goto err; if (cmp12==0) { if (cmp13==0) { if (set || (TEST_VALUE(i1.value, i2.value) == 0)) { /* change in i3 value or all same */ if (merge_output(r, &i3, mapping) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i3.value) == 0)) { /* change in i2 value */ if (merge_output(r, &i2, mapping) < 0) goto err; } else { /* conflicting value changes in i2 and i3 */ merge_error(i1.position, i2.position, i3.position, 1); goto err; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (cmp13 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i2.value) == 0)) { /* deleted in i3 */ if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { /* conflicting del in i3 and change in i2 */ merge_error(i1.position, i2.position, i3.position, 2); goto err; } } else if (cmp13 == 0) { if (cmp12 > 0) { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else if (set || (TEST_VALUE(i1.value, i3.value) == 0)) { /* deleted in i2 */ if (i1.next(&i1) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* conflicting del in i2 and change in i3 */ merge_error(i1.position, i2.position, i3.position, 3); goto err; } } else { /* Both keys changed */ TEST_KEY_SET_OR(cmp23, i2.key, i3.key) goto err; if (cmp23==0) { /* dueling inserts or deletes */ merge_error(i1.position, i2.position, i3.position, 4); goto err; } if (cmp12 > 0) { /* insert i2 */ if (cmp23 > 0) { /* insert i3 first */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* insert i2 first */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } } else if (cmp13 > 0) { /* Insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* 1<2 and 1<3: both deleted 1.key */ merge_error(i1.position, i2.position, i3.position, 5); goto err; } } } while (i2.position >= 0 && i3.position >= 0) { /* New inserts */ TEST_KEY_SET_OR(cmp23, i2.key, i3.key) goto err; if (cmp23==0) { /* dueling inserts */ merge_error(i1.position, i2.position, i3.position, 6); goto err; } if (cmp23 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } } while (i1.position >= 0 && i2.position >= 0) { /* remainder of i1 deleted in i3 */ TEST_KEY_SET_OR(cmp12, i1.key, i2.key) goto err; if (cmp12 > 0) { /* insert i2 */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else if (cmp12==0 && (set || (TEST_VALUE(i1.value, i2.value) == 0))) { /* delete i3 */ if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { /* Dueling deletes or delete and change */ merge_error(i1.position, i2.position, i3.position, 7); goto err; } } while (i1.position >= 0 && i3.position >= 0) { /* remainder of i1 deleted in i2 */ TEST_KEY_SET_OR(cmp13, i1.key, i3.key) goto err; if (cmp13 > 0) { /* insert i3 */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else if (cmp13==0 && (set || (TEST_VALUE(i1.value, i3.value) == 0))) { /* delete i2 */ if (i1.next(&i1) < 0) goto err; if (i3.next(&i3) < 0) goto err; } else { /* Dueling deletes or delete and change */ merge_error(i1.position, i2.position, i3.position, 8); goto err; } } if (i1.position >= 0) { /* Dueling deletes */ merge_error(i1.position, i2.position, i3.position, 9); goto err; } while (i2.position >= 0) { /* Inserting i2 at end */ if (merge_output(r, &i2, mapping) < 0) goto err; if (i2.next(&i2) < 0) goto err; } while (i3.position >= 0) { /* Inserting i3 at end */ if (merge_output(r, &i3, mapping) < 0) goto err; if (i3.next(&i3) < 0) goto err; } /* If the output bucket is empty, conflict resolution doesn't have * enough info to unlink it from its containing BTree correctly. */ if (r->len == 0) { merge_error(-1, -1, -1, 10); goto err; } finiSetIteration(&i1); finiSetIteration(&i2); finiSetIteration(&i3); if (s1->next) { Py_INCREF(s1->next); r->next = s1->next; } s = bucket_getstate(r); Py_DECREF(r); return s; err: finiSetIteration(&i1); finiSetIteration(&i2); finiSetIteration(&i3); Py_XDECREF(r); return NULL; } Zope-2.8.7-final/lib/python/BTrees/_OOBTree.c0000664000076400007640000000213010435502401017543 0ustar ajungajung/*############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################*/ #define MASTER_ID "$Id: _OOBTree.c 25186 2004-06-02 15:07:33Z jim $\n" /* OOBTree - object key, object value BTree Implements a collection using object type keys and object type values */ #define PERSISTENT #define MOD_NAME_PREFIX "OO" #define INITMODULE init_OOBTree #define DEFAULT_MAX_BUCKET_SIZE 30 #define DEFAULT_MAX_BTREE_SIZE 250 #include "objectkeymacros.h" #include "objectvaluemacros.h" #include "BTreeModuleTemplate.c" Zope-2.8.7-final/lib/python/BTrees/BTreeModuleTemplate.c0000664000076400007640000003627310435502401022027 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "Python.h" /* include structmember.h for offsetof */ #include "structmember.h" #ifdef PERSISTENT #include "cPersistence.h" #else #define PER_USE_OR_RETURN(self, NULL) #define PER_ALLOW_DEACTIVATION(self) #define PER_PREVENT_DEACTIVATION(self) #define PER_DEL(self) #define PER_USE(O) 1 #define PER_ACCESSED(O) 1 #endif /* So sue me. This pair gets used all over the place, so much so that it * interferes with understanding non-persistence parts of algorithms. * PER_UNUSE can be used after a successul PER_USE or PER_USE_OR_RETURN. * It allows the object to become ghostified, and tells the persistence * machinery that the object's fields were used recently. */ #define PER_UNUSE(OBJ) do { \ PER_ALLOW_DEACTIVATION(OBJ); \ PER_ACCESSED(OBJ); \ } while (0) /* The tp_name slots of the various BTree types contain the fully qualified names of the types, e.g. zodb.btrees.OOBTree.OOBTree. The full name is usd to support pickling and because it is not possible to modify the __module__ slot of a type dynamically. (This may be a bug in Python 2.2). */ #define MODULE_NAME "BTrees._" MOD_NAME_PREFIX "BTree." static PyObject *sort_str, *reverse_str, *__setstate___str, *_bucket_type_str; static PyObject *ConflictError = NULL; static void PyVar_Assign(PyObject **v, PyObject *e) { Py_XDECREF(*v); *v=e;} #define ASSIGN(V,E) PyVar_Assign(&(V),(E)) #define UNLESS(E) if (!(E)) #define OBJECT(O) ((PyObject*)(O)) #define MIN_BUCKET_ALLOC 16 #define MAX_BTREE_SIZE(B) DEFAULT_MAX_BTREE_SIZE #define MAX_BUCKET_SIZE(B) DEFAULT_MAX_BUCKET_SIZE #define SameType_Check(O1, O2) ((O1)->ob_type==(O2)->ob_type) #define ASSERT(C, S, R) if (! (C)) { \ PyErr_SetString(PyExc_AssertionError, (S)); return (R); } /* Various kinds of BTree and Bucket structs are instances of * "sized containers", and have a common initial layout: * The stuff needed for all Python objects, or all Persistent objects. * int size: The maximum number of things that could be contained * without growing the container. * int len: The number of things currently contained. * * Invariant: 0 <= len <= size. * * A sized container typically goes on to declare one or more pointers * to contiguous arrays with 'size' elements each, the initial 'len' of * which are currently in use. */ #ifdef PERSISTENT #define sizedcontainer_HEAD \ cPersistent_HEAD \ int size; \ int len; #else #define sizedcontainer_HEAD \ PyObject_HEAD \ int size; \ int len; #endif /* Nothing is actually of type Sized, but (pointers to) BTree nodes and * Buckets can be cast to Sized* in contexts that only need to examine * the members common to all sized containers. */ typedef struct Sized_s { sizedcontainer_HEAD } Sized; #define SIZED(O) ((Sized*)(O)) /* A Bucket wraps contiguous vectors of keys and values. Keys are unique, * and stored in sorted order. The 'values' pointer may be NULL if the * Bucket is used to implement a set. Buckets serving as leafs of BTrees * are chained together via 'next', so that the entire BTree contents * can be traversed in sorted order quickly and easily. */ typedef struct Bucket_s { sizedcontainer_HEAD struct Bucket_s *next; /* the bucket with the next-larger keys */ KEY_TYPE *keys; /* 'len' keys, in increasing order */ VALUE_TYPE *values; /* 'len' corresponding values; NULL if a set */ } Bucket; #define BUCKET(O) ((Bucket*)(O)) /* A BTree is complicated. See Maintainer.txt. */ typedef struct BTreeItem_s { KEY_TYPE key; Sized *child; /* points to another BTree, or to a Bucket of some sort */ } BTreeItem; typedef struct BTree_s { sizedcontainer_HEAD /* firstbucket points to the bucket containing the smallest key in * the BTree. This is found by traversing leftmost child pointers * (data[0].child) until reaching a Bucket. */ Bucket *firstbucket; /* The BTree points to 'len' children, via the "child" fields of the data * array. There are len-1 keys in the 'key' fields, stored in increasing * order. data[0].key is unused. For i in 0 .. len-1, all keys reachable * from data[i].child are >= data[i].key and < data[i+1].key, at the * endpoints pretending that data[0].key is minus infinity and * data[len].key is positive infinity. */ BTreeItem *data; } BTree; static PyTypeObject BTreeType; static PyTypeObject BucketType; #define BTREE(O) ((BTree*)(O)) /* Use BTREE_SEARCH to find which child pointer to follow. * RESULT An int lvalue to hold the index i such that SELF->data[i].child * is the correct node to search next. * SELF A pointer to a BTree node. * KEY The key you're looking for, of type KEY_TYPE. * ONERROR What to do if key comparison raises an exception; for example, * perhaps 'return NULL'. * * See Maintainer.txt for discussion: this is optimized in subtle ways. * It's recommended that you call this at the start of a routine, waiting * to check for self->len == 0 after. */ #define BTREE_SEARCH(RESULT, SELF, KEY, ONERROR) { \ int _lo = 0; \ int _hi = (SELF)->len; \ int _i, _cmp; \ for (_i = _hi >> 1; _i > _lo; _i = (_lo + _hi) >> 1) { \ TEST_KEY_SET_OR(_cmp, (SELF)->data[_i].key, (KEY)) \ ONERROR; \ if (_cmp < 0) _lo = _i; \ else if (_cmp > 0) _hi = _i; \ else /* equal */ break; \ } \ (RESULT) = _i; \ } /* SetIteration structs are used in the internal set iteration protocol. * When you want to iterate over a set or bucket or BTree (even an * individual key!), * 1. Declare a new iterator: * SetIteration si = {0,0,0}; * Using "{0,0,0}" or "{0,0}" appear most common. Only one {0} is * necssary. At least one must be given so that finiSetIteration() works * correctly even if you don't get around to calling initSetIteration(). * 2. Initialize it via * initSetIteration(&si, PyObject *s, useValues) * It's an error if that returns an int < 0. In case of error on the * init call, calling finiSetIteration(&si) is optional. But if the * init call succeeds, you must eventually call finiSetIteration(), * and whether or not subsequent calls to si.next() fail. * 3. Get the first element: * if (si.next(&si) < 0) { there was an error } * If the set isn't empty, this sets si.position to an int >= 0, * si.key to the element's key (of type KEY_TYPE), and maybe si.value to * the element's value (of type VALUE_TYPE). si.value is defined * iff si.usesValue is true. * 4. Process all the elements: * while (si.position >= 0) { * do something with si.key and/or si.value; * if (si.next(&si) < 0) { there was an error; } * } * 5. Finalize the SetIterator: * finiSetIteration(&si); * This is mandatory! si may contain references to iterator objects, * keys and values, and they must be cleaned up else they'll leak. If * this were C++ we'd hide that in the destructor, but in C you have to * do it by hand. */ typedef struct SetIteration_s { PyObject *set; /* the set, bucket, BTree, ..., being iterated */ int position; /* initialized to 0; set to -1 by next() when done */ int usesValue; /* true iff 'set' has values & we iterate them */ KEY_TYPE key; /* next() sets to next key */ VALUE_TYPE value; /* next() may set to next value */ int (*next)(struct SetIteration_s*); /* function to get next key+value */ } SetIteration; /* Finish the set iteration protocol. This MUST be called by everyone * who starts a set iteration, unless the initial call to initSetIteration * failed; in that case, and only that case, calling finiSetIteration is * optional. */ static void finiSetIteration(SetIteration *i) { assert(i != NULL); if (i->set == NULL) return; Py_DECREF(i->set); i->set = NULL; /* so it doesn't hurt to call this again */ if (i->position > 0) { /* next() was called at least once, but didn't finish iterating * (else position would be negative). So the cached key and * value need to be cleaned up. */ DECREF_KEY(i->key); if (i->usesValue) { DECREF_VALUE(i->value); } } i->position = -1; /* stop any stray next calls from doing harm */ } static PyObject * IndexError(int i) { PyObject *v; v = PyInt_FromLong(i); if (!v) { v = Py_None; Py_INCREF(v); } PyErr_SetObject(PyExc_IndexError, v); Py_DECREF(v); return NULL; } /* Search for the bucket immediately preceding *current, in the bucket chain * starting at first. current, *current and first must not be NULL. * * Return: * 1 *current holds the correct bucket; this is a borrowed reference * 0 no such bucket exists; *current unaltered * -1 error; *current unaltered */ static int PreviousBucket(Bucket **current, Bucket *first) { Bucket *trailing = NULL; /* first travels; trailing follows it */ int result = 0; assert(current && *current && first); if (first == *current) return 0; do { trailing = first; PER_USE_OR_RETURN(first, -1); first = first->next; PER_UNUSE(trailing); if (first == *current) { *current = trailing; result = 1; break; } } while (first); return result; } static void * BTree_Malloc(size_t sz) { void *r; ASSERT(sz > 0, "non-positive size malloc", NULL); r = malloc(sz); if (r) return r; PyErr_NoMemory(); return NULL; } static void * BTree_Realloc(void *p, size_t sz) { void *r; ASSERT(sz > 0, "non-positive size realloc", NULL); if (p) r = realloc(p, sz); else r = malloc(sz); UNLESS (r) PyErr_NoMemory(); return r; } /* Shared keyword-argument list for BTree/Bucket * (iter)?(keys|values|items) */ static char *search_keywords[] = {"min", "max", "excludemin", "excludemax", 0}; #include "BTreeItemsTemplate.c" #include "BucketTemplate.c" #include "SetTemplate.c" #include "BTreeTemplate.c" #include "TreeSetTemplate.c" #include "SetOpTemplate.c" #include "MergeTemplate.c" static struct PyMethodDef module_methods[] = { {"difference", (PyCFunction) difference_m, METH_VARARGS, "difference(o1, o2) -- " "compute the difference between o1 and o2" }, {"union", (PyCFunction) union_m, METH_VARARGS, "union(o1, o2) -- compute the union of o1 and o2\n" }, {"intersection", (PyCFunction) intersection_m, METH_VARARGS, "intersection(o1, o2) -- " "compute the intersection of o1 and o2" }, #ifdef MERGE {"weightedUnion", (PyCFunction) wunion_m, METH_VARARGS, "weightedUnion(o1, o2 [, w1, w2]) -- compute the union of o1 and o2\n" "\nw1 and w2 are weights." }, {"weightedIntersection", (PyCFunction) wintersection_m, METH_VARARGS, "weightedIntersection(o1, o2 [, w1, w2]) -- " "compute the intersection of o1 and o2\n" "\nw1 and w2 are weights." }, #endif #ifdef MULTI_INT_UNION {"multiunion", (PyCFunction) multiunion_m, METH_VARARGS, "multiunion(seq) -- compute union of a sequence of integer sets.\n" "\n" "Each element of seq must be an integer set, or convertible to one\n" "via the set iteration protocol. The union returned is an IISet." }, #endif {NULL, NULL} /* sentinel */ }; static char BTree_module_documentation[] = "\n" MASTER_ID BTREEITEMSTEMPLATE_C "$Id: BTreeModuleTemplate.c 29815 2005-04-01 22:41:47Z fdrake $\n" BTREETEMPLATE_C BUCKETTEMPLATE_C KEYMACROS_H MERGETEMPLATE_C SETOPTEMPLATE_C SETTEMPLATE_C TREESETTEMPLATE_C VALUEMACROS_H BTREEITEMSTEMPLATE_C ; int init_persist_type(PyTypeObject *type) { type->ob_type = &PyType_Type; type->tp_base = cPersistenceCAPI->pertype; if (PyType_Ready(type) < 0) return 0; return 1; } void INITMODULE (void) { PyObject *m, *d, *c; sort_str = PyString_InternFromString("sort"); if (!sort_str) return; reverse_str = PyString_InternFromString("reverse"); if (!reverse_str) return; __setstate___str = PyString_InternFromString("__setstate__"); if (!__setstate___str) return; _bucket_type_str = PyString_InternFromString("_bucket_type"); if (!_bucket_type_str) return; /* Grab the ConflictError class */ m = PyImport_ImportModule("ZODB.POSException"); if (m != NULL) { c = PyObject_GetAttrString(m, "BTreesConflictError"); if (c != NULL) ConflictError = c; Py_DECREF(m); } if (ConflictError == NULL) { Py_INCREF(PyExc_ValueError); ConflictError=PyExc_ValueError; } /* Initialize the PyPersist_C_API and the type objects. */ cPersistenceCAPI = PyCObject_Import("persistent.cPersistence", "CAPI"); if (cPersistenceCAPI == NULL) return; BTreeItemsType.ob_type = &PyType_Type; BTreeIter_Type.ob_type = &PyType_Type; BTreeIter_Type.tp_getattro = PyObject_GenericGetAttr; BucketType.tp_new = PyType_GenericNew; SetType.tp_new = PyType_GenericNew; BTreeType.tp_new = PyType_GenericNew; TreeSetType.tp_new = PyType_GenericNew; if (!init_persist_type(&BucketType)) return; if (!init_persist_type(&BTreeType)) return; if (!init_persist_type(&SetType)) return; if (!init_persist_type(&TreeSetType)) return; if (PyDict_SetItem(BTreeType.tp_dict, _bucket_type_str, (PyObject *)&BucketType) < 0) { fprintf(stderr, "btree failed\n"); return; } if (PyDict_SetItem(TreeSetType.tp_dict, _bucket_type_str, (PyObject *)&SetType) < 0) { fprintf(stderr, "bucket failed\n"); return; } /* Create the module and add the functions */ m = Py_InitModule4("_" MOD_NAME_PREFIX "BTree", module_methods, BTree_module_documentation, (PyObject *)NULL, PYTHON_API_VERSION); /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); if (PyDict_SetItemString(d, MOD_NAME_PREFIX "Bucket", (PyObject *)&BucketType) < 0) return; if (PyDict_SetItemString(d, MOD_NAME_PREFIX "BTree", (PyObject *)&BTreeType) < 0) return; if (PyDict_SetItemString(d, MOD_NAME_PREFIX "Set", (PyObject *)&SetType) < 0) return; if (PyDict_SetItemString(d, MOD_NAME_PREFIX "TreeSet", (PyObject *)&TreeSetType) < 0) return; if (PyDict_SetItemString(d, MOD_NAME_PREFIX "TreeIterator", (PyObject *)&BTreeIter_Type) < 0) return; } Zope-2.8.7-final/lib/python/BTrees/TreeSetTemplate.c0000664000076400007640000001553010435502401021224 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define TREESETTEMPLATE_C "$Id: TreeSetTemplate.c 25186 2004-06-02 15:07:33Z jim $\n" static PyObject * TreeSet_insert(BTree *self, PyObject *args) { PyObject *key; int i; if (!PyArg_ParseTuple(args, "O:insert", &key)) return NULL; i = _BTree_set(self, key, Py_None, 1, 1); if (i < 0) return NULL; return PyInt_FromLong(i); } /* _Set_update and _TreeSet_update are identical except for the function they call to add the element to the set. */ static int _TreeSet_update(BTree *self, PyObject *seq) { int n = -1; PyObject *iter, *v; int ind; iter = PyObject_GetIter(seq); if (iter == NULL) return -1; while (1) { v = PyIter_Next(iter); if (v == NULL) { if (PyErr_Occurred()) goto err; else break; } ind = _BTree_set(self, v, Py_None, 1, 1); Py_DECREF(v); if (ind < 0) goto err; else n += ind; } /* n starts out at -1, which is the error return value. If this point is reached, then there is no error. n must be incremented to account for the initial value of -1 instead of 0. */ n++; err: Py_DECREF(iter); return n; } static PyObject * TreeSet_update(BTree *self, PyObject *args) { PyObject *seq = NULL; int n = 0; if (!PyArg_ParseTuple(args, "|O:update", &seq)) return NULL; if (seq) { n = _TreeSet_update(self, seq); if (n < 0) return NULL; } return PyInt_FromLong(n); } static PyObject * TreeSet_remove(BTree *self, PyObject *args) { PyObject *key; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if (_BTree_set(self, key, NULL, 0, 1) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static PyObject * TreeSet_setstate(BTree *self, PyObject *args) { int r; if (!PyArg_ParseTuple(args,"O",&args)) return NULL; PER_PREVENT_DEACTIVATION(self); r=_BTree_setstate(self, args, 1); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static struct PyMethodDef TreeSet_methods[] = { {"__getstate__", (PyCFunction) BTree_getstate, METH_NOARGS, "__getstate__() -> state\n\n" "Return the picklable state of the TreeSet."}, {"__setstate__", (PyCFunction) TreeSet_setstate, METH_VARARGS, "__setstate__(state)\n\n" "Set the state of the TreeSet."}, {"has_key", (PyCFunction) BTree_has_key, METH_O, "has_key(key)\n\n" "Return true if the TreeSet contains the given key."}, {"keys", (PyCFunction) BTree_keys, METH_KEYWORDS, "keys([min, max]) -> list of keys\n\n" "Returns the keys of the TreeSet. If min and max are supplied, only\n" "keys greater than min and less than max are returned."}, {"maxKey", (PyCFunction) BTree_maxKey, METH_VARARGS, "maxKey([max]) -> key\n\n" "Return the largest key in the BTree. If max is specified, return\n" "the largest key <= max."}, {"minKey", (PyCFunction) BTree_minKey, METH_VARARGS, "minKey([mi]) -> key\n\n" "Return the smallest key in the BTree. If min is specified, return\n" "the smallest key >= min."}, {"clear", (PyCFunction) BTree_clear, METH_NOARGS, "clear()\n\nRemove all of the items from the BTree."}, {"insert", (PyCFunction)TreeSet_insert, METH_VARARGS, "insert(id,[ignored]) -- Add an id to the set"}, {"update", (PyCFunction)TreeSet_update, METH_VARARGS, "update(collection)\n\n Add the items from the given collection."}, {"remove", (PyCFunction)TreeSet_remove, METH_VARARGS, "remove(id) -- Remove a key from the set"}, {"_check", (PyCFunction) BTree_check, METH_NOARGS, "Perform sanity check on TreeSet, and raise exception if flawed."}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) BTree__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) BTree__p_deactivate, METH_KEYWORDS, "_p_deactivate()\n\nReinitialize from a newly created copy."}, #endif {NULL, NULL} /* sentinel */ }; static PyMappingMethods TreeSet_as_mapping = { (inquiry)BTree_length, /*mp_length*/ }; static PySequenceMethods TreeSet_as_sequence = { (inquiry)0, /* sq_length */ (binaryfunc)0, /* sq_concat */ (intargfunc)0, /* sq_repeat */ (intargfunc)0, /* sq_item */ (intintargfunc)0, /* sq_slice */ (intobjargproc)0, /* sq_ass_item */ (intintobjargproc)0, /* sq_ass_slice */ (objobjproc)BTree_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static int TreeSet_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "TreeSet", &v)) return -1; if (v) return _TreeSet_update((BTree *)self, v); else return 0; } static PyTypeObject TreeSetType = { PyObject_HEAD_INIT(NULL) /* PyPersist_Type */ 0, /* ob_size */ MODULE_NAME MOD_NAME_PREFIX "TreeSet",/* tp_name */ sizeof(BTree), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)BTree_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ &BTree_as_number_for_nonzero, /* tp_as_number */ &TreeSet_as_sequence, /* tp_as_sequence */ &TreeSet_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)BTree_traverse, /* tp_traverse */ (inquiry)BTree_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)BTree_getiter, /* tp_iter */ 0, /* tp_iternext */ TreeSet_methods, /* tp_methods */ BTree_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ TreeSet_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; Zope-2.8.7-final/lib/python/BTrees/check.py0000664000076400007640000003472010435502401017442 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Utilities for working with BTrees (TreeSets, Buckets, and Sets) at a low level. The primary function is check(btree), which performs value-based consistency checks of a kind btree._check() does not perform. See the function docstring for details. display(btree) displays the internal structure of a BTree (TreeSet, etc) to stdout. CAUTION: When a BTree node has only a single bucket child, it can be impossible to get at the bucket from Python code (__getstate__() may squash the bucket object out of existence, as a pickling storage optimization). In such a case, the code here synthesizes a temporary bucket with the same keys (and values, if the bucket is of a mapping type). This has no first-order consequences, but can mislead if you pay close attention to reported object addresses and/or object identity (the synthesized bucket has an address that doesn't exist in the actual BTree). """ from types import TupleType from BTrees.OOBTree import OOBTree, OOBucket, OOSet, OOTreeSet from BTrees.OIBTree import OIBTree, OIBucket, OISet, OITreeSet from BTrees.IOBTree import IOBTree, IOBucket, IOSet, IOTreeSet from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet from BTrees.IFBTree import IFBTree, IFBucket, IFSet, IFTreeSet from ZODB.utils import positive_id, oid_repr TYPE_UNKNOWN, TYPE_BTREE, TYPE_BUCKET = range(3) _type2kind = {IOBTree: (TYPE_BTREE, True), IIBTree: (TYPE_BTREE, True), IFBTree: (TYPE_BTREE, True), OIBTree: (TYPE_BTREE, True), OOBTree: (TYPE_BTREE, True), IOBucket: (TYPE_BUCKET, True), IIBucket: (TYPE_BUCKET, True), IFBucket: (TYPE_BUCKET, True), OIBucket: (TYPE_BUCKET, True), OOBucket: (TYPE_BUCKET, True), IOTreeSet: (TYPE_BTREE, False), IITreeSet: (TYPE_BTREE, False), IFTreeSet: (TYPE_BTREE, False), OITreeSet: (TYPE_BTREE, False), OOTreeSet: (TYPE_BTREE, False), IOSet: (TYPE_BUCKET, False), IISet: (TYPE_BUCKET, False), IFSet: (TYPE_BUCKET, False), OISet: (TYPE_BUCKET, False), OOSet: (TYPE_BUCKET, False), } # Return pair # # TYPE_BTREE or TYPE_BUCKET, is_mapping def classify(obj): return _type2kind[type(obj)] BTREE_EMPTY, BTREE_ONE, BTREE_NORMAL = range(3) # If the BTree is empty, returns # # BTREE_EMPTY, [], [] # # If the BTree has only one bucket, sometimes returns # # BTREE_ONE, bucket_state, None # # Else returns # # BTREE_NORMAL, list of keys, list of kids # # and the list of kids has one more entry than the list of keys. # # BTree.__getstate__() docs: # # For an empty BTree (self->len == 0), None. # # For a BTree with one child (self->len == 1), and that child is a bucket, # and that bucket has a NULL oid, a one-tuple containing a one-tuple # containing the bucket's state: # # ( # ( # child[0].__getstate__(), # ), # ) # # Else a two-tuple. The first element is a tuple interleaving the BTree's # keys and direct children, of size 2*self->len - 1 (key[0] is unused and # is not saved). The second element is the firstbucket: # # ( # (child[0], key[1], child[1], key[2], child[2], ..., # key[len-1], child[len-1]), # self->firstbucket # ) _btree2bucket = {IOBTree: IOBucket, IOTreeSet: IOSet, IIBTree: IIBucket, IITreeSet: IISet, IFBTree: IFBucket, IFTreeSet: IFSet, OIBTree: OIBucket, OITreeSet: OISet, OOBTree: OOBucket, OOTreeSet: OOSet} def crack_btree(t, is_mapping): state = t.__getstate__() if state is None: return BTREE_EMPTY, [], [] assert isinstance(state, TupleType) if len(state) == 1: state = state[0] assert isinstance(state, TupleType) and len(state) == 1 state = state[0] return BTREE_ONE, state, None assert len(state) == 2 data, firstbucket = state n = len(data) assert n & 1 kids = [] keys = [] i = 0 for x in data: if i & 1: keys.append(x) else: kids.append(x) i += 1 return BTREE_NORMAL, keys, kids # Returns # # keys, values # for a mapping; len(keys) == len(values) in this case # or # keys, [] # for a set # # bucket.__getstate__() docs: # # For a set bucket (self->values is NULL), a one-tuple or two-tuple. The # first element is a tuple of keys, of length self->len. The second element # is the next bucket, present if and only if next is non-NULL: # # ( # (keys[0], keys[1], ..., keys[len-1]), # next iff non-NULL> # ) # # For a mapping bucket (self->values is not NULL), a one-tuple or two-tuple. # The first element is a tuple interleaving keys and values, of length # 2 * self->len. The second element is the next bucket, present iff next is # non-NULL: # # ( # (keys[0], values[0], keys[1], values[1], ..., # keys[len-1], values[len-1]), # next iff non-NULL> # ) def crack_bucket(b, is_mapping): state = b.__getstate__() assert isinstance(state, TupleType) assert 1 <= len(state) <= 2 data = state[0] if not is_mapping: return data, [] keys = [] values = [] n = len(data) assert n & 1 == 0 i = 0 for x in data: if i & 1: values.append(x) else: keys.append(x) i += 1 return keys, values def type_and_adr(obj): if hasattr(obj, '_p_oid'): oid = oid_repr(obj._p_oid) else: oid = 'None' return "%s (0x%x oid=%s)" % (type(obj).__name__, positive_id(obj), oid) # Walker implements a depth-first search of a BTree (or TreeSet or Set or # Bucket). Subclasses must implement the visit_btree() and visit_bucket() # methods, and arrange to call the walk() method. walk() calls the # visit_XYZ() methods once for each node in the tree, in depth-first # left-to-right order. class Walker: def __init__(self, obj): self.obj = obj # obj is the BTree (BTree or TreeSet). # path is a list of indices, from the root. For example, if a BTree node # is child[5] of child[3] of the root BTree, [3, 5]. # parent is the parent BTree object, or None if this is the root BTree. # is_mapping is True for a BTree and False for a TreeSet. # keys is a list of the BTree's internal keys. # kids is a list of the BTree's children. # If the BTree is an empty root node, keys == kids == []. # Else len(kids) == len(keys) + 1. # lo and hi are slice bounds on the values the elements of keys *should* # lie in (lo inclusive, hi exclusive). lo is None if there is no lower # bound known, and hi is None if no upper bound is known. def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): raise NotImplementedError # obj is the bucket (Bucket or Set). # path is a list of indices, from the root. For example, if a bucket # node is child[5] of child[3] of the root BTree, [3, 5]. # parent is the parent BTree object. # is_mapping is True for a Bucket and False for a Set. # keys is a list of the bucket's keys. # values is a list of the bucket's values. # If is_mapping is false, values == []. Else len(keys) == len(values). # lo and hi are slice bounds on the values the elements of keys *should* # lie in (lo inclusive, hi exclusive). lo is None if there is no lower # bound known, and hi is None if no upper bound is known. def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): raise NotImplementedError def walk(self): obj = self.obj path = [] stack = [(obj, path, None, None, None)] while stack: obj, path, parent, lo, hi = stack.pop() kind, is_mapping = classify(obj) if kind is TYPE_BTREE: bkind, keys, kids = crack_btree(obj, is_mapping) if bkind is BTREE_NORMAL: # push the kids, in reverse order (so they're popped off # the stack in forward order) n = len(kids) for i in range(len(kids)-1, -1, -1): newlo, newhi = lo, hi if i < n-1: newhi = keys[i] if i > 0: newlo = keys[i-1] stack.append((kids[i], path + [i], obj, newlo, newhi)) elif bkind is BTREE_EMPTY: pass else: assert bkind is BTREE_ONE # Yuck. There isn't a bucket object to pass on, as # the bucket state is embedded directly in the BTree # state. Synthesize a bucket. assert kids is None # and "keys" is really the bucket # state bucket = _btree2bucket[type(obj)]() bucket.__setstate__(keys) stack.append((bucket, path + [0], obj, lo, hi)) keys = [] kids = [bucket] self.visit_btree(obj, path, parent, is_mapping, keys, kids, lo, hi) else: assert kind is TYPE_BUCKET keys, values = crack_bucket(obj, is_mapping) self.visit_bucket(obj, path, parent, is_mapping, keys, values, lo, hi) class Checker(Walker): def __init__(self, obj): Walker.__init__(self, obj) self.errors = [] def check(self): self.walk() if self.errors: s = "Errors found in %s:" % type_and_adr(self.obj) self.errors.insert(0, s) s = "\n".join(self.errors) raise AssertionError(s) def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): self.check_sorted(obj, path, keys, lo, hi) def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): self.check_sorted(obj, path, keys, lo, hi) def check_sorted(self, obj, path, keys, lo, hi): i, n = 0, len(keys) for x in keys: if lo is not None and not lo <= x: s = "key %r < lower bound %r at index %d" % (x, lo, i) self.complain(s, obj, path) if hi is not None and not x < hi: s = "key %r >= upper bound %r at index %d" % (x, hi, i) self.complain(s, obj, path) if i < n-1 and not x < keys[i+1]: s = "key %r at index %d >= key %r at index %d" % ( x, i, keys[i+1], i+1) self.complain(s, obj, path) i += 1 def complain(self, msg, obj, path): s = "%s, in %s, path from root %s" % ( msg, type_and_adr(obj), ".".join(map(str, path))) self.errors.append(s) class Printer(Walker): def __init__(self, obj): Walker.__init__(self, obj) def display(self): self.walk() def visit_btree(self, obj, path, parent, is_mapping, keys, kids, lo, hi): indent = " " * len(path) print "%s%s %s with %d children" % ( indent, ".".join(map(str, path)), type_and_adr(obj), len(kids)) indent += " " n = len(keys) for i in range(n): print "%skey %d: %r" % (indent, i, keys[i]) def visit_bucket(self, obj, path, parent, is_mapping, keys, values, lo, hi): indent = " " * len(path) print "%s%s %s with %d keys" % ( indent, ".".join(map(str, path)), type_and_adr(obj), len(keys)) indent += " " n = len(keys) for i in range(n): print "%skey %d: %r" % (indent, i, keys[i]), if is_mapping: print "value %r" % (values[i],) def check(btree): """Check internal value-based invariants in a BTree or TreeSet. The btree._check() method checks internal C-level pointer consistency. The check() function here checks value-based invariants: whether the keys in leaf bucket and internal nodes are in strictly increasing order, and whether they all lie in their expected range. The latter is a subtle invariant that can't be checked locally -- it requires propagating range info down from the root of the tree, and modifying it at each level for each child. Raises AssertionError if anything is wrong, with a string detail explaining the problems. The entire tree is checked before AssertionError is raised, and the string detail may be large (depending on how much went wrong). """ Checker(btree).check() def display(btree): "Display the internal structure of a BTree, Bucket, TreeSet or Set." Printer(btree).display() Zope-2.8.7-final/lib/python/BTrees/intvaluemacros.h0000664000076400007640000000153610435502401021217 0ustar ajungajung #define VALUEMACROS_H "$Id: intvaluemacros.h 24682 2004-05-14 19:32:43Z tim_one $\n" #define VALUE_TYPE int #undef VALUE_TYPE_IS_PYOBJECT #define TEST_VALUE(K, T) (((K) < (T)) ? -1 : (((K) > (T)) ? 1: 0)) #define VALUE_SAME(VALUE, TARGET) ( (VALUE) == (TARGET) ) #define DECLARE_VALUE(NAME) VALUE_TYPE NAME #define VALUE_PARSE "i" #define DECREF_VALUE(k) #define INCREF_VALUE(k) #define COPY_VALUE(V, E) (V=(E)) #define COPY_VALUE_TO_OBJECT(O, K) O=PyInt_FromLong(K) #define COPY_VALUE_FROM_ARG(TARGET, ARG, STATUS) \ if (PyInt_Check(ARG)) TARGET=PyInt_AsLong(ARG); else { \ PyErr_SetString(PyExc_TypeError, "expected integer value"); \ (STATUS)=0; (TARGET)=0; } #define NORMALIZE_VALUE(V, MIN) ((MIN) > 0) ? ((V)/=(MIN)) : 0 #define MERGE_DEFAULT 1 #define MERGE(O1, w1, O2, w2) ((O1)*(w1)+(O2)*(w2)) #define MERGE_WEIGHT(O, w) ((O)*(w)) Zope-2.8.7-final/lib/python/BTrees/tests/0000775000076400007640000000000010435610645017161 5ustar ajungajungZope-2.8.7-final/lib/python/BTrees/tests/testSetOps.py0000664000076400007640000004241410435502400021642 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from unittest import TestCase, TestSuite, TextTestRunner, makeSuite from BTrees.OOBTree import OOBTree, OOBucket, OOSet, OOTreeSet from BTrees.IOBTree import IOBTree, IOBucket, IOSet, IOTreeSet from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet from BTrees.OIBTree import OIBTree, OIBucket, OISet, OITreeSet # Subclasses have to set up: # builders - functions to build inputs, taking an optional keys arg # intersection, union, difference - set to the type-correct versions class SetResult(TestCase): def setUp(self): self.Akeys = [1, 3, 5, 6 ] self.Bkeys = [ 2, 3, 4, 6, 7] self.As = [makeset(self.Akeys) for makeset in self.builders] self.Bs = [makeset(self.Bkeys) for makeset in self.builders] self.emptys = [makeset() for makeset in self.builders] # Slow but obviously correct Python implementations of basic ops. def _union(self, x, y): result = list(x) for e in y: if e not in result: result.append(e) result.sort() return result def _intersection(self, x, y): result = [] for e in x: if e in y: result.append(e) return result def _difference(self, x, y): result = list(x) for e in y: if e in result: result.remove(e) # Difference preserves LHS values. if hasattr(x, "values"): result = [(k, x[k]) for k in result] return result def testNone(self): for op in self.union, self.intersection, self.difference: C = op(None, None) self.assert_(C is None) for op in self.union, self.intersection, self.difference: for A in self.As: C = op(A, None) self.assert_(C is A) C = op(None, A) if op is self.difference: self.assert_(C is None) else: self.assert_(C is A) def testEmptyUnion(self): for A in self.As: for E in self.emptys: C = self.union(A, E) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), self.Akeys) C = self.union(E, A) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), self.Akeys) def testEmptyIntersection(self): for A in self.As: for E in self.emptys: C = self.intersection(A, E) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), []) C = self.intersection(E, A) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), []) def testEmptyDifference(self): for A in self.As: for E in self.emptys: C = self.difference(A, E) # Difference preserves LHS values. self.assertEqual(hasattr(C, "values"), hasattr(A, "values")) if hasattr(A, "values"): self.assertEqual(list(C.items()), list(A.items())) else: self.assertEqual(list(C), self.Akeys) C = self.difference(E, A) self.assertEqual(hasattr(C, "values"), hasattr(E, "values")) self.assertEqual(list(C), []) def testUnion(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.union(A, B) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), self._union(A, B)) def testIntersection(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.intersection(A, B) self.assert_(not hasattr(C, "values")) self.assertEqual(list(C), self._intersection(A, B)) def testDifference(self): inputs = self.As + self.Bs for A in inputs: for B in inputs: C = self.difference(A, B) # Difference preserves LHS values. self.assertEqual(hasattr(C, "values"), hasattr(A, "values")) want = self._difference(A, B) if hasattr(A, "values"): self.assertEqual(list(C.items()), want) else: self.assertEqual(list(C), want) def testLargerInputs(self): from random import randint MAXSIZE = 200 MAXVAL = 400 for i in range(3): n = randint(0, MAXSIZE) Akeys = [randint(1, MAXVAL) for j in range(n)] As = [makeset(Akeys) for makeset in self.builders] Akeys = IISet(Akeys) n = randint(0, MAXSIZE) Bkeys = [randint(1, MAXVAL) for j in range(n)] Bs = [makeset(Bkeys) for makeset in self.builders] Bkeys = IISet(Bkeys) for op, simulator in ((self.union, self._union), (self.intersection, self._intersection), (self.difference, self._difference)): for A in As: for B in Bs: got = op(A, B) want = simulator(Akeys, Bkeys) self.assertEqual(list(got), want, (A, B, Akeys, Bkeys, list(got), want)) # Given a mapping builder (IIBTree, OOBucket, etc), return a function # that builds an object of that type given only a list of keys. def makeBuilder(mapbuilder): def result(keys=[], mapbuilder=mapbuilder): return mapbuilder(zip(keys, keys)) return result class PureII(SetResult): from BTrees.IIBTree import union, intersection, difference builders = IISet, IITreeSet, makeBuilder(IIBTree), makeBuilder(IIBucket) class PureIO(SetResult): from BTrees.IOBTree import union, intersection, difference builders = IOSet, IOTreeSet, makeBuilder(IOBTree), makeBuilder(IOBucket) class PureOO(SetResult): from BTrees.OOBTree import union, intersection, difference builders = OOSet, OOTreeSet, makeBuilder(OOBTree), makeBuilder(OOBucket) class PureOI(SetResult): from BTrees.OIBTree import union, intersection, difference builders = OISet, OITreeSet, makeBuilder(OIBTree), makeBuilder(OIBucket) # Subclasses must set up (as class variables): # multiunion, union # mkset, mktreeset # mkbucket, mkbtree class MultiUnion(TestCase): def testEmpty(self): self.assertEqual(len(self.multiunion([])), 0) def testOne(self): for sequence in [3], range(20), range(-10, 0, 2) + range(1, 10, 2): seq1 = sequence[:] seq2 = sequence[:] seq2.reverse() seqsorted = sequence[:] seqsorted.sort() for seq in seq1, seq2, seqsorted: for builder in self.mkset, self.mktreeset: input = builder(seq) output = self.multiunion([input]) self.assertEqual(len(seq), len(output)) self.assertEqual(seqsorted, list(output)) def testValuesIgnored(self): for builder in self.mkbucket, self.mkbtree: input = builder([(1, 2), (3, 4), (5, 6)]) output = self.multiunion([input]) self.assertEqual([1, 3, 5], list(output)) def testBigInput(self): N = 100000 input = self.mkset(range(N)) output = self.multiunion([input] * 10) self.assertEqual(len(output), N) self.assertEqual(output.minKey(), 0) self.assertEqual(output.maxKey(), N-1) self.assertEqual(list(output), range(N)) def testLotsOfLittleOnes(self): from random import shuffle N = 5000 inputs = [] mkset, mktreeset = self.mkset, self.mktreeset for i in range(N): base = i * 4 - N inputs.append(mkset([base, base+1])) inputs.append(mktreeset([base+2, base+3])) shuffle(inputs) output = self.multiunion(inputs) self.assertEqual(len(output), N*4) self.assertEqual(list(output), range(-N, 3*N)) def testFunkyKeyIteration(self): # The internal set iteration protocol allows "iterating over" a # a single key as if it were a set. N = 100 union, mkset = self.union, self.mkset slow = mkset() for i in range(N): slow = union(slow, mkset([i])) fast = self.multiunion(range(N)) # acts like N distinct singleton sets self.assertEqual(len(slow), N) self.assertEqual(len(fast), N) self.assertEqual(list(slow), list(fast)) self.assertEqual(list(fast), range(N)) class TestIIMultiUnion(MultiUnion): from BTrees.IIBTree import multiunion, union from BTrees.IIBTree import IISet as mkset, IITreeSet as mktreeset from BTrees.IIBTree import IIBucket as mkbucket, IIBTree as mkbtree class TestIOMultiUnion(MultiUnion): from BTrees.IOBTree import multiunion, union from BTrees.IOBTree import IOSet as mkset, IOTreeSet as mktreeset from BTrees.IOBTree import IOBucket as mkbucket, IOBTree as mkbtree # Check that various special module functions are and aren't imported from # the expected BTree modules. class TestImports(TestCase): def testWeightedUnion(self): from BTrees.IIBTree import weightedUnion from BTrees.OIBTree import weightedUnion try: from BTrees.IOBTree import weightedUnion except ImportError: pass else: self.fail("IOBTree shouldn't have weightedUnion") try: from BTrees.OOBTree import weightedUnion except ImportError: pass else: self.fail("OOBTree shouldn't have weightedUnion") def testWeightedIntersection(self): from BTrees.IIBTree import weightedIntersection from BTrees.OIBTree import weightedIntersection try: from BTrees.IOBTree import weightedIntersection except ImportError: pass else: self.fail("IOBTree shouldn't have weightedIntersection") try: from BTrees.OOBTree import weightedIntersection except ImportError: pass else: self.fail("OOBTree shouldn't have weightedIntersection") def testMultiunion(self): from BTrees.IIBTree import multiunion from BTrees.IOBTree import multiunion try: from BTrees.OIBTree import multiunion except ImportError: pass else: self.fail("OIBTree shouldn't have multiunion") try: from BTrees.OOBTree import multiunion except ImportError: pass else: self.fail("OOBTree shouldn't have multiunion") # Subclasses must set up (as class variables): # weightedUnion, weightedIntersection # builders -- sequence of constructors, taking items # union, intersection -- the module routines of those names # mkbucket -- the module bucket builder class Weighted(TestCase): def setUp(self): self.Aitems = [(1, 10), (3, 30), (5, 50), (6, 60)] self.Bitems = [(2, 21), (3, 31), (4, 41), (6, 61), (7, 71)] self.As = [make(self.Aitems) for make in self.builders] self.Bs = [make(self.Bitems) for make in self.builders] self.emptys = [make([]) for make in self.builders] weights = [] for w1 in -3, -1, 0, 1, 7: for w2 in -3, -1, 0, 1, 7: weights.append((w1, w2)) self.weights = weights def testBothNone(self): for op in self.weightedUnion, self.weightedIntersection: w, C = op(None, None) self.assert_(C is None) self.assertEqual(w, 0) w, C = op(None, None, 42, 666) self.assert_(C is None) self.assertEqual(w, 0) def testLeftNone(self): for op in self.weightedUnion, self.weightedIntersection: for A in self.As + self.emptys: w, C = op(None, A) self.assert_(C is A) self.assertEqual(w, 1) w, C = op(None, A, 42, 666) self.assert_(C is A) self.assertEqual(w, 666) def testRightNone(self): for op in self.weightedUnion, self.weightedIntersection: for A in self.As + self.emptys: w, C = op(A, None) self.assert_(C is A) self.assertEqual(w, 1) w, C = op(A, None, 42, 666) self.assert_(C is A) self.assertEqual(w, 42) # If obj is a set, return a bucket with values all 1; else return obj. def _normalize(self, obj): if isaset(obj): obj = self.mkbucket(zip(obj, [1] * len(obj))) return obj # Python simulation of weightedUnion. def _wunion(self, A, B, w1=1, w2=1): if isaset(A) and isaset(B): return 1, self.union(A, B).keys() A = self._normalize(A) B = self._normalize(B) result = [] for key in self.union(A, B): v1 = A.get(key, 0) v2 = B.get(key, 0) result.append((key, v1*w1 + v2*w2)) return 1, result def testUnion(self): inputs = self.As + self.Bs + self.emptys for A in inputs: for B in inputs: want_w, want_s = self._wunion(A, B) got_w, got_s = self.weightedUnion(A, B) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) for w1, w2 in self.weights: want_w, want_s = self._wunion(A, B, w1, w2) got_w, got_s = self.weightedUnion(A, B, w1, w2) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) # Python simulation weightedIntersection. def _wintersection(self, A, B, w1=1, w2=1): if isaset(A) and isaset(B): return w1 + w2, self.intersection(A, B).keys() A = self._normalize(A) B = self._normalize(B) result = [] for key in self.intersection(A, B): result.append((key, A[key]*w1 + B[key]*w2)) return 1, result def testIntersection(self): inputs = self.As + self.Bs + self.emptys for A in inputs: for B in inputs: want_w, want_s = self._wintersection(A, B) got_w, got_s = self.weightedIntersection(A, B) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) for w1, w2 in self.weights: want_w, want_s = self._wintersection(A, B, w1, w2) got_w, got_s = self.weightedIntersection(A, B, w1, w2) self.assertEqual(got_w, want_w) if isaset(got_s): self.assertEqual(got_s.keys(), want_s) else: self.assertEqual(got_s.items(), want_s) # Given a set builder (like OITreeSet or OISet), return a function that # takes a list of (key, value) pairs and builds a set out of the keys. def itemsToSet(setbuilder): def result(items, setbuilder=setbuilder): return setbuilder([key for key, value in items]) return result class TestWeightedII(Weighted): from BTrees.IIBTree import weightedUnion, weightedIntersection from BTrees.IIBTree import union, intersection from BTrees.IIBTree import IIBucket as mkbucket builders = IIBucket, IIBTree, itemsToSet(IISet), itemsToSet(IITreeSet) class TestWeightedOI(Weighted): from BTrees.OIBTree import weightedUnion, weightedIntersection from BTrees.OIBTree import union, intersection from BTrees.OIBTree import OIBucket as mkbucket builders = OIBucket, OIBTree, itemsToSet(OISet), itemsToSet(OITreeSet) # 'thing' is a bucket, btree, set or treeset. Return true iff it's one of the # latter two. def isaset(thing): return not hasattr(thing, 'values') def test_suite(): s = TestSuite() for klass in (TestIIMultiUnion, TestIOMultiUnion, TestImports, PureII, PureIO, PureOI, PureOO, TestWeightedII, TestWeightedOI): s.addTest(makeSuite(klass)) return s def main(): TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/BTrees/tests/testConflict.py0000664000076400007640000006632610435502400022176 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os from unittest import TestCase, TestSuite, makeSuite from BTrees.OOBTree import OOBTree, OOBucket, OOSet, OOTreeSet from BTrees.IOBTree import IOBTree, IOBucket, IOSet, IOTreeSet from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet from BTrees.IFBTree import IFBTree, IFBucket, IFSet, IFTreeSet from BTrees.OIBTree import OIBTree, OIBucket, OISet, OITreeSet import transaction from ZODB.POSException import ConflictError class Base: """ Tests common to all types: sets, buckets, and BTrees """ storage = None def tearDown(self): transaction.abort() del self.t if self.storage is not None: self.storage.close() self.storage.cleanup() def openDB(self): from ZODB.FileStorage import FileStorage from ZODB.DB import DB n = 'fs_tmp__%s' % os.getpid() self.storage = FileStorage(n) self.db = DB(self.storage) class MappingBase(Base): """ Tests common to mappings (buckets, btrees) """ def _deletefail(self): del self.t[1] def _setupConflict(self): l=[ -5124, -7377, 2274, 8801, -9901, 7327, 1565, 17, -679, 3686, -3607, 14, 6419, -5637, 6040, -4556, -8622, 3847, 7191, -4067] e1=[(-1704, 0), (5420, 1), (-239, 2), (4024, 3), (-6984, 4)] e2=[(7745, 0), (4868, 1), (-2548, 2), (-2711, 3), (-3154, 4)] base=self.t base.update([(i, i*i) for i in l[:20]]) b1=base.__class__(base) b2=base.__class__(base) bm=base.__class__(base) items=base.items() return base, b1, b2, bm, e1, e2, items def testSimpleConflict(self): # Unlike all the other tests, invoke conflict resolution # by committing a transaction and catching a conflict # in the storage. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open().root() copy = r2["t"] list(copy) # unghostify self.assertEqual(self.t._p_serial, copy._p_serial) self.t.update({1:2, 2:3}) transaction.commit() copy.update({3:4}) transaction.commit() def testMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] del b2[items[5][0]] del b1[items[-1][0]] del b2[items[-2][0]] del bm[items[0][0]] del bm[items[5][0]] del bm[items[-1][0]] del bm[items[-2][0]] test_merge(base, b1, b2, bm, 'merge delete') def testMergeDeleteAndUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] b2[items[5][0]]=1 del b1[items[-1][0]] b2[items[-2][0]]=2 del bm[items[0][0]] bm[items[5][0]]=1 del bm[items[-1][0]] bm[items[-2][0]]=2 test_merge(base, b1, b2, bm, 'merge update and delete') def testMergeUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[items[0][0]]=1 b2[items[5][0]]=2 b1[items[-1][0]]=3 b2[items[-2][0]]=4 bm[items[0][0]]=1 bm[items[5][0]]=2 bm[items[-1][0]]=3 bm[items[-2][0]]=4 test_merge(base, b1, b2, bm, 'merge update') def testFailMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] del b2[items[0][0]] test_merge(base, b1, b2, bm, 'merge conflicting delete', should_fail=1) def testFailMergeUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[items[0][0]]=1 b2[items[0][0]]=2 test_merge(base, b1, b2, bm, 'merge conflicting update', should_fail=1) def testFailMergeDeleteAndUpdate(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() del b1[items[0][0]] b2[items[0][0]]=-9 test_merge(base, b1, b2, bm, 'merge conflicting update and delete', should_fail=1) def testMergeInserts(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[-99999]=-99999 b1[e1[0][0]]=e1[0][1] b2[99999]=99999 b2[e1[2][0]]=e1[2][1] bm[-99999]=-99999 bm[e1[0][0]]=e1[0][1] bm[99999]=99999 bm[e1[2][0]]=e1[2][1] test_merge(base, b1, b2, bm, 'merge insert') def testMergeInsertsFromEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() base.clear() b1.clear() b2.clear() bm.clear() b1.update(e1) bm.update(e1) b2.update(e2) bm.update(e2) test_merge(base, b1, b2, bm, 'merge insert from empty') def testFailMergeEmptyAndFill(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() b2.update(e2) bm.update(e2) test_merge(base, b1, b2, bm, 'merge insert from empty', should_fail=1) def testMergeEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() test_merge(base, b1, b2, bm, 'empty one and not other', should_fail=1) def testFailMergeInsert(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1[-99999]=-99999 b1[e1[0][0]]=e1[0][1] b2[99999]=99999 b2[e1[0][0]]=e1[0][1] test_merge(base, b1, b2, bm, 'merge conflicting inserts', should_fail=1) class SetTests(Base): "Set (as opposed to TreeSet) specific tests." def _setupConflict(self): l=[ -5124, -7377, 2274, 8801, -9901, 7327, 1565, 17, -679, 3686, -3607, 14, 6419, -5637, 6040, -4556, -8622, 3847, 7191, -4067] e1=[-1704, 5420, -239, 4024, -6984] e2=[7745, 4868, -2548, -2711, -3154] base=self.t base.update(l) b1=base.__class__(base) b2=base.__class__(base) bm=base.__class__(base) items=base.keys() return base, b1, b2, bm, e1, e2, items def testMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.remove(items[0]) b2.remove(items[5]) b1.remove(items[-1]) b2.remove(items[-2]) bm.remove(items[0]) bm.remove(items[5]) bm.remove(items[-1]) bm.remove(items[-2]) test_merge(base, b1, b2, bm, 'merge delete') def testFailMergeDelete(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.remove(items[0]) b2.remove(items[0]) test_merge(base, b1, b2, bm, 'merge conflicting delete', should_fail=1) def testMergeInserts(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.insert(-99999) b1.insert(e1[0]) b2.insert(99999) b2.insert(e1[2]) bm.insert(-99999) bm.insert(e1[0]) bm.insert(99999) bm.insert(e1[2]) test_merge(base, b1, b2, bm, 'merge insert') def testMergeInsertsFromEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() base.clear() b1.clear() b2.clear() bm.clear() b1.update(e1) bm.update(e1) b2.update(e2) bm.update(e2) test_merge(base, b1, b2, bm, 'merge insert from empty') def testFailMergeEmptyAndFill(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() b2.update(e2) bm.update(e2) test_merge(base, b1, b2, bm, 'merge insert from empty', should_fail=1) def testMergeEmpty(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.clear() bm.clear() test_merge(base, b1, b2, bm, 'empty one and not other', should_fail=1) def testFailMergeInsert(self): base, b1, b2, bm, e1, e2, items = self._setupConflict() b1.insert(-99999) b1.insert(e1[0]) b2.insert(99999) b2.insert(e1[0]) test_merge(base, b1, b2, bm, 'merge conflicting inserts', should_fail=1) def test_merge(o1, o2, o3, expect, message='failed to merge', should_fail=0): s1 = o1.__getstate__() s2 = o2.__getstate__() s3 = o3.__getstate__() expected = expect.__getstate__() if expected is None: expected = ((((),),),) if should_fail: try: merged = o1._p_resolveConflict(s1, s2, s3) except ConflictError, err: pass else: assert 0, message else: merged = o1._p_resolveConflict(s1, s2, s3) assert merged == expected, message class BucketTests(MappingBase): """ Tests common to all buckets """ class BTreeTests(MappingBase): """ Tests common to all BTrees """ ## BTree tests class TestIOBTrees(BTreeTests, TestCase): def setUp(self): self.t = IOBTree() class TestOOBTrees(BTreeTests, TestCase): def setUp(self): self.t = OOBTree() class TestOIBTrees(BTreeTests, TestCase): def setUp(self): self.t = OIBTree() class TestIIBTrees(BTreeTests, TestCase): def setUp(self): self.t = IIBTree() class TestIFBTrees(BTreeTests, TestCase): def setUp(self): self.t = IFBTree() ## Set tests class TestIOSets(SetTests, TestCase): def setUp(self): self.t = IOSet() class TestOOSets(SetTests, TestCase): def setUp(self): self.t = OOSet() class TestIISets(SetTests, TestCase): def setUp(self): self.t = IISet() class TestIFSets(SetTests, TestCase): def setUp(self): self.t = IFSet() class TestOISets(SetTests, TestCase): def setUp(self): self.t = OISet() class TestIOTreeSets(SetTests, TestCase): def setUp(self): self.t = IOTreeSet() class TestOOTreeSets(SetTests, TestCase): def setUp(self): self.t = OOTreeSet() class TestIITreeSets(SetTests, TestCase): def setUp(self): self.t = IITreeSet() class TestIFTreeSets(SetTests, TestCase): def setUp(self): self.t = IFTreeSet() class TestOITreeSets(SetTests, TestCase): def setUp(self): self.t = OITreeSet() ## Bucket tests class TestIOBuckets(BucketTests, TestCase): def setUp(self): self.t = IOBucket() class TestOOBuckets(BucketTests, TestCase): def setUp(self): self.t = OOBucket() class TestIIBuckets(BucketTests, TestCase): def setUp(self): self.t = IIBucket() class TestIFBuckets(BucketTests, TestCase): def setUp(self): self.t = IFBucket() class TestOIBuckets(BucketTests, TestCase): def setUp(self): self.t = OIBucket() class NastyConfict(Base, TestCase): def setUp(self): self.t = OOBTree() # This tests a problem that cropped up while trying to write # testBucketSplitConflict (below): conflict resolution wasn't # working at all in non-trivial cases. Symptoms varied from # strange complaints about pickling (despite that the test isn't # doing any *directly*), thru SystemErrors from Python and # AssertionErrors inside the BTree code. def testResolutionBlowsUp(self): b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open().root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) self.t.update({1:2, 2:3}) transaction.commit() copy.update({3:4}) transaction.commit() # if this doesn't blow up list(copy.values()) # and this doesn't either, then fine def testBucketSplitConflict(self): # Tests that a bucket split is viewed as a conflict. # It's (almost necessarily) a white-box test, and sensitive to # implementation details. b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) # In one transaction, add 16 new keys to bucket1, to force a bucket # split. b = self.t numtoadd = 16 candidate = 60 while numtoadd: if not b.has_key(candidate): b[candidate] = candidate numtoadd -= 1 candidate += 1 # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 61 .. 74 # bucket 2 has 16 values: [75, 76 .. 81] + [84, 88 ..116] # bucket 3 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((b0, 60, b1, 75, b2, 120, b3), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state) , 2) self.assertEqual(len(state[0]), 7) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 75) self.assertEqual(state[0][5], 120) transaction.commit() # In the other transaction, add 3 values near the tail end of bucket1. # This doesn't cause a split. b = copy for i in range(112, 116): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 18 values: 60, 64 .. 112, 113, 114, 115, 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) self.assertRaises(ConflictError, transaction.commit) transaction.abort() # horrible things happen w/o this def testEmptyBucketConflict(self): # Tests that an emptied bucket *created by* conflict resolution is # viewed as a conflict: conflict resolution doesn't have enough # info to unlink the empty bucket from the BTree correctly. b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) # In one transaction, delete half of bucket 1. b = self.t for k in 60, 64, 68, 72, 76, 80, 84, 88: del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 7 values: 92, 96, 100, 104, 108, 112, 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state) , 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) transaction.commit() # In the other transaction, delete the other half of bucket 1. b = copy for k in 92, 96, 100, 104, 108, 112, 116: del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 8 values: 60, 64, 68, 72, 76, 80, 84, 88 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Conflict resolution empties bucket1 entirely. This used to # create an "insane" BTree (a legit BTree cannot contain an empty # bucket -- it contains NULL pointers the BTree code doesn't # expect, and segfaults result). self.assertRaises(ConflictError, transaction.commit) transaction.abort() # horrible things happen w/o this def testEmptyBucketNoConflict(self): # Tests that a plain empty bucket (on input) is not viewed as a # conflict. b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Invoke conflict resolution by committing a transaction. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open().root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) # In one transaction, just add a key. b = self.t b[1] = 1 # bucket 0 has 16 values: [0, 1] + [4, 8 .. 56] # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) transaction.commit() # In the other transaction, delete bucket 2. b = copy for k in range(120, 200, 4): del b[k] # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1), firstbucket) # The next block is still verifying preconditions. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 3) self.assertEqual(state[0][1], 60) # This shouldn't create a ConflictError. transaction.commit() # And the resulting BTree shouldn't have internal damage. b._check() # The snaky control flow in _bucket__p_resolveConflict ended up trying # to decref a NULL pointer if conflict resolution was fed 3 empty # buckets. http://collector.zope.org/Zope/553 def testThreeEmptyBucketsNoSegfault(self): self.openDB() r1 = self.db.open().root() self.assertEqual(len(self.t), 0) r1["t"] = b = self.t # an empty tree transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) # In one transaction, add and delete a key. b[2] = 2 del b[2] transaction.commit() # In the other transaction, also add and delete a key. b = copy b[1] = 1 del b[1] # If the commit() segfaults, the C code is still wrong for this case. self.assertRaises(ConflictError, transaction.commit) transaction.abort() def testCantResolveBTreeConflict(self): # Test that a conflict involving two different changes to # an internal BTree node is unresolvable. An internal node # only changes when there are enough additions or deletions # to a child bucket that the bucket is split or removed. # It's (almost necessarily) a white-box test, and sensitive to # implementation details. b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Set up database connections to provoke conflict. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) # Now one transaction should add enough keys to cause a split, # and another should remove all the keys in one bucket. for k in range(200, 300, 4): self.t[k] = k transaction.commit() for k in range(0, 60, 4): del copy[k] try: transaction.commit() except ConflictError, detail: self.assert_(str(detail).startswith('database conflict error')) transaction.abort() else: self.fail("expected ConflictError") def testConflictWithOneEmptyBucket(self): # If one transaction empties a bucket, while another adds an item # to the bucket, all the changes "look resolvable": bucket conflict # resolution returns a bucket containing (only) the item added by # the latter transaction, but changes from the former transaction # removing the bucket are uncontested: the bucket is removed from # the BTree despite that resolution thinks it's non-empty! This # was first reported by Dieter Maurer, to zodb-dev on 22 Mar 2005. b = self.t for i in range(0, 200, 4): b[i] = i # bucket 0 has 15 values: 0, 4 .. 56 # bucket 1 has 15 values: 60, 64 .. 116 # bucket 2 has 20 values: 120, 124 .. 196 state = b.__getstate__() # Looks like: ((bucket0, 60, bucket1, 120, bucket2), firstbucket) # If these fail, the *preconditions* for running the test aren't # satisfied -- the test itself hasn't been run yet. self.assertEqual(len(state), 2) self.assertEqual(len(state[0]), 5) self.assertEqual(state[0][1], 60) self.assertEqual(state[0][3], 120) # Set up database connections to provoke conflict. self.openDB() r1 = self.db.open().root() r1["t"] = self.t transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(self.t._p_serial, copy._p_serial) # Now one transaction empties the first bucket, and another adds a # key to the first bucket. for k in range(0, 60, 4): del self.t[k] transaction.commit() copy[1] = 1 try: transaction.commit() except ConflictError, detail: self.assert_(str(detail).startswith('database conflict error')) transaction.abort() else: self.fail("expected ConflictError") # Same thing, except commit the transactions in the opposite order. b = OOBTree() for i in range(0, 200, 4): b[i] = i r1 = self.db.open().root() r1["t"] = b transaction.commit() r2 = self.db.open(synch=False).root() copy = r2["t"] # Make sure all of copy is loaded. list(copy.values()) self.assertEqual(b._p_serial, copy._p_serial) # Now one transaction empties the first bucket, and another adds a # key to the first bucket. b[1] = 1 transaction.commit() for k in range(0, 60, 4): del copy[k] try: transaction.commit() except ConflictError, detail: self.assert_(str(detail).startswith('database conflict error')) transaction.abort() else: self.fail("expected ConflictError") def test_suite(): suite = TestSuite() for k in ( TestIIBTrees, TestIISets, TestIITreeSets, TestIIBuckets, TestIFBTrees, TestIFSets, TestIFTreeSets, TestIFBuckets, TestIOBTrees, TestIOSets, TestIOTreeSets, TestIOBuckets, TestOOBTrees, TestOOSets, TestOOTreeSets, TestOOBuckets, TestOIBTrees, TestOISets, TestOITreeSets, TestOIBuckets, NastyConfict): suite.addTest(makeSuite(k)) return suite Zope-2.8.7-final/lib/python/BTrees/tests/test_check.py0000664000076400007640000000665110435502400021644 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test the BTree check.check() function.""" import unittest from BTrees.OOBTree import OOBTree from BTrees.check import check class CheckTest(unittest.TestCase): def setUp(self): self.t = t = OOBTree() for i in range(31): t[i] = 2*i self.state = t.__getstate__() def testNormal(self): s = self.state # Looks like (state, first_bucket) # where state looks like (bucket0, 15, bucket1). self.assertEqual(len(s), 2) self.assertEqual(len(s[0]), 3) self.assertEqual(s[0][1], 15) self.t._check() # shouldn't blow up check(self.t) # shouldn't blow up def testKeyTooLarge(self): # Damage an invariant by dropping the BTree key to 14. s = self.state news = (s[0][0], 14, s[0][2]), s[1] self.t.__setstate__(news) self.t._check() # not caught try: # Expecting "... key %r >= upper bound %r at index %d" check(self.t) except AssertionError, detail: self.failUnless(str(detail).find(">= upper bound") > 0) else: self.fail("expected self.t_check() to catch the problem") def testKeyTooSmall(self): # Damage an invariant by bumping the BTree key to 16. s = self.state news = (s[0][0], 16, s[0][2]), s[1] self.t.__setstate__(news) self.t._check() # not caught try: # Expecting "... key %r < lower bound %r at index %d" check(self.t) except AssertionError, detail: self.failUnless(str(detail).find("< lower bound") > 0) else: self.fail("expected self.t_check() to catch the problem") def testKeysSwapped(self): # Damage an invariant by swapping two key/value pairs. s = self.state # Looks like (state, first_bucket) # where state looks like (bucket0, 15, bucket1). (b0, num, b1), firstbucket = s self.assertEqual(b0[4], 8) self.assertEqual(b0[5], 10) b0state = b0.__getstate__() self.assertEqual(len(b0state), 2) # b0state looks like # ((k0, v0, k1, v1, ...), nextbucket) pairs, nextbucket = b0state self.assertEqual(pairs[8], 4) self.assertEqual(pairs[9], 8) self.assertEqual(pairs[10], 5) self.assertEqual(pairs[11], 10) newpairs = pairs[:8] + (5, 10, 4, 8) + pairs[12:] b0.__setstate__((newpairs, nextbucket)) self.t._check() # not caught try: check(self.t) except AssertionError, detail: self.failUnless(str(detail).find( "key 5 at index 4 >= key 4 at index 5") > 0) else: self.fail("expected self.t_check() to catch the problem") def test_suite(): return unittest.makeSuite(CheckTest) Zope-2.8.7-final/lib/python/BTrees/tests/__init__.py0000664000076400007640000000006410435502400021257 0ustar ajungajung# If tests is a package, debugging is a bit easier. Zope-2.8.7-final/lib/python/BTrees/tests/test_compare.py0000664000076400007640000000454310435502400022213 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test errors during comparison of BTree keys.""" import unittest from BTrees.OOBTree import OOBucket as Bucket, OOSet as Set import transaction from ZODB.MappingStorage import MappingStorage from ZODB.DB import DB class CompareTest(unittest.TestCase): s = "A string with hi-bit-set characters: \700\701" u = u"A unicode string" def setUp(self): # These defaults only make sense if the default encoding # prevents s from being promoted to Unicode. self.assertRaises(UnicodeError, unicode, self.s) # An object needs to be added to the database to self.db = DB(MappingStorage()) root = self.db.open().root() self.bucket = root["bucket"] = Bucket() self.set = root["set"] = Set() transaction.commit() def tearDown(self): self.assert_(self.bucket._p_changed != 2) self.assert_(self.set._p_changed != 2) transaction.abort() def assertUE(self, callable, *args): self.assertRaises(UnicodeError, callable, *args) def testBucketGet(self): self.bucket[self.s] = 1 self.assertUE(self.bucket.get, self.u) def testSetGet(self): self.set.insert(self.s) self.assertUE(self.set.remove, self.u) def testBucketSet(self): self.bucket[self.s] = 1 self.assertUE(self.bucket.__setitem__, self.u, 1) def testSetSet(self): self.set.insert(self.s) self.assertUE(self.set.insert, self.u) def testBucketMinKey(self): self.bucket[self.s] = 1 self.assertUE(self.bucket.minKey, self.u) def testSetMinKey(self): self.set.insert(self.s) self.assertUE(self.set.minKey, self.u) def test_suite(): return unittest.makeSuite(CompareTest) Zope-2.8.7-final/lib/python/BTrees/tests/testBTreesUnicode.py0000664000076400007640000000465310435502400023123 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest from BTrees.OOBTree import OOBTree # When an OOBtree contains unicode strings as keys, # it is neccessary accessing non-unicode strings are # either ascii strings or encoded as unicoded using the # corresponding encoding encoding = 'ISO-8859-1' class TestBTreesUnicode(unittest.TestCase): """ test unicode""" def setUp(self): """setup an OOBTree with some unicode strings""" self.s = unicode('dreit\xe4gigen', 'latin1') self.data = [('alien', 1), ('k\xf6nnten', 2), ('fox', 3), ('future', 4), ('quick', 5), ('zerst\xf6rt', 6), (unicode('dreit\xe4gigen','latin1'), 7), ] self.tree = OOBTree() for k, v in self.data: if isinstance(k, str): k = unicode(k, 'latin1') self.tree[k] = v def testAllKeys(self): # check every item of the tree for k, v in self.data: if isinstance(k, str): k = unicode(k, encoding) self.assert_(self.tree.has_key(k)) self.assertEqual(self.tree[k], v) def testUnicodeKeys(self): # try to access unicode keys in tree k, v = self.data[-1] self.assertEqual(k, self.s) self.assertEqual(self.tree[k], v) self.assertEqual(self.tree[self.s], v) def testAsciiKeys(self): # try to access some "plain ASCII" keys in the tree for k, v in self.data[0], self.data[2]: self.assert_(isinstance(k, str)) self.assertEqual(self.tree[k], v) def test_suite(): return unittest.makeSuite(TestBTreesUnicode) def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/BTrees/tests/test_btreesubclass.py0000664000076400007640000000266110435502400023425 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from BTrees.OOBTree import OOBTree, OOBucket class B(OOBucket): pass class T(OOBTree): _bucket_type = B import unittest class SubclassTest(unittest.TestCase): def testSubclass(self): # test that a subclass that defines _bucket_type gets buckets # of that type t = T() # There's no good way to get a bucket at the moment. # __getstate__() is as good as it gets, but the default # getstate explicitly includes the pickle of the bucket # for small trees, so we have to be clever :-( # make sure there is more than one bucket in the tree for i in range(1000): t[i] = i state = t.__getstate__() self.assert_(state[0][0].__class__ is B) def test_suite(): return unittest.makeSuite(SubclassTest) Zope-2.8.7-final/lib/python/BTrees/tests/testBTrees.py0000664000076400007640000014233010435502400021607 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import random from unittest import TestCase, TestSuite, TextTestRunner, makeSuite from BTrees.OOBTree import OOBTree, OOBucket, OOSet, OOTreeSet from BTrees.IOBTree import IOBTree, IOBucket, IOSet, IOTreeSet from BTrees.IIBTree import IIBTree, IIBucket, IISet, IITreeSet from BTrees.IFBTree import IFBTree, IFBucket, IFSet, IFTreeSet from BTrees.OIBTree import OIBTree, OIBucket, OISet, OITreeSet from BTrees.check import check import transaction from ZODB import DB from ZODB.MappingStorage import MappingStorage class Base(TestCase): """ Tests common to all types: sets, buckets, and BTrees """ db = None def tearDown(self): if self.db is not None: self.db.close() self.t = None del self.t def _getRoot(self): if self.db is None: # Unclear: On the next line, the ZODB4 flavor of this routine # [asses a cache_size argument: # self.db = DB(MappingStorage(), cache_size=1) # If that's done here, though, testLoadAndStore() and # testGhostUnghost() both nail the CPU and seemingly # never finish. self.db = DB(MappingStorage()) return self.db.open().root() def _closeRoot(self, root): root._p_jar.close() def testLoadAndStore(self): for i in 0, 10, 1000: t = self.t.__class__() self._populate(t, i) root = None root = self._getRoot() root[i] = t transaction.commit() root2 = self._getRoot() if hasattr(t, 'items'): self.assertEqual(list(root2[i].items()) , list(t.items())) else: self.assertEqual(list(root2[i].keys()) , list(t.keys())) self._closeRoot(root) self._closeRoot(root2) def testGhostUnghost(self): for i in 0, 10, 1000: t = self.t.__class__() self._populate(t, i) root = self._getRoot() root[i] = t transaction.commit() root2 = self._getRoot() root2[i]._p_deactivate() transaction.commit() if hasattr(t, 'items'): self.assertEqual(list(root2[i].items()) , list(t.items())) else: self.assertEqual(list(root2[i].keys()) , list(t.keys())) self._closeRoot(root) self._closeRoot(root2) def testSimpleExclusiveKeyRange(self): t = self.t.__class__() self.assertEqual(list(t.keys()), []) self.assertEqual(list(t.keys(excludemin=True)), []) self.assertEqual(list(t.keys(excludemax=True)), []) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) self._populate(t, 1) self.assertEqual(list(t.keys()), [0]) self.assertEqual(list(t.keys(excludemin=True)), []) self.assertEqual(list(t.keys(excludemax=True)), []) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 2) self.assertEqual(list(t.keys()), [0, 1]) self.assertEqual(list(t.keys(excludemin=True)), [1]) self.assertEqual(list(t.keys(excludemax=True)), [0]) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 3) self.assertEqual(list(t.keys()), [0, 1, 2]) self.assertEqual(list(t.keys(excludemin=True)), [1, 2]) self.assertEqual(list(t.keys(excludemax=True)), [0, 1]) self.assertEqual(list(t.keys(excludemin=True, excludemax=True)), [1]) self.assertEqual(list(t.keys(-1, 3, excludemin=True, excludemax=True)), [0, 1, 2]) self.assertEqual(list(t.keys(0, 3, excludemin=True, excludemax=True)), [1, 2]) self.assertEqual(list(t.keys(-1, 2, excludemin=True, excludemax=True)), [0, 1]) self.assertEqual(list(t.keys(0, 2, excludemin=True, excludemax=True)), [1]) class MappingBase(Base): """ Tests common to mappings (buckets, btrees) """ def _populate(self, t, l): # Make some data for i in range(l): t[i]=i def testRepr(self): # test the repr because buckets have a complex repr implementation # internally the cutoff from a stack allocated buffer to a heap # allocated buffer is 10000. for i in range(1000): self.t[i] = i r = repr(self.t) # Make sure the repr is 10000 bytes long for a bucket. # But since the test is also run for btrees, skip the length # check if the repr starts with '<' if not r.startswith('<'): self.assert_(len(r) > 10000) def testGetItemFails(self): self.assertRaises(KeyError, self._getitemfail) def _getitemfail(self): return self.t[1] def testGetReturnsDefault(self): self.assertEqual(self.t.get(1) , None) self.assertEqual(self.t.get(1, 'foo') , 'foo') def testSetItemGetItemWorks(self): self.t[1] = 1 a = self.t[1] self.assertEqual(a , 1, `a`) def testReplaceWorks(self): self.t[1] = 1 self.assertEqual(self.t[1] , 1, self.t[1]) self.t[1] = 2 self.assertEqual(self.t[1] , 2, self.t[1]) def testLen(self): added = {} r = range(1000) for x in r: k = random.choice(r) self.t[k] = x added[k] = x addl = added.keys() self.assertEqual(len(self.t) , len(addl), len(self.t)) def testHasKeyWorks(self): self.t[1] = 1 self.assert_(self.t.has_key(1)) self.assert_(1 in self.t) self.assert_(0 not in self.t) self.assert_(2 not in self.t) def testValuesWorks(self): for x in range(100): self.t[x] = x*x v = self.t.values() for i in range(100): self.assertEqual(v[i], i*i) self.assertRaises(IndexError, lambda: v[i+1]) i = 0 for value in self.t.itervalues(): self.assertEqual(value, i*i) i += 1 def testValuesWorks1(self): for x in range(100): self.t[99-x] = x for x in range(40): lst = list(self.t.values(0+x,99-x)) lst.sort() self.assertEqual(lst,range(0+x,99-x+1)) lst = list(self.t.values(max=99-x, min=0+x)) lst.sort() self.assertEqual(lst,range(0+x,99-x+1)) def testValuesNegativeIndex(self): L = [-3, 6, -11, 4] for i in L: self.t[i] = i L.sort() vals = self.t.values() for i in range(-1, -5, -1): self.assertEqual(vals[i], L[i]) self.assertRaises(IndexError, lambda: vals[-5]) def testKeysWorks(self): for x in range(100): self.t[x] = x v = self.t.keys() i = 0 for x in v: self.assertEqual(x,i) i = i + 1 self.assertRaises(IndexError, lambda: v[i]) for x in range(40): lst = self.t.keys(0+x,99-x) self.assertEqual(list(lst), range(0+x, 99-x+1)) lst = self.t.keys(max=99-x, min=0+x) self.assertEqual(list(lst), range(0+x, 99-x+1)) self.assertEqual(len(v), 100) def testKeysNegativeIndex(self): L = [-3, 6, -11, 4] for i in L: self.t[i] = i L.sort() keys = self.t.keys() for i in range(-1, -5, -1): self.assertEqual(keys[i], L[i]) self.assertRaises(IndexError, lambda: keys[-5]) def testItemsWorks(self): for x in range(100): self.t[x] = 2*x v = self.t.items() i = 0 for x in v: self.assertEqual(x[0], i) self.assertEqual(x[1], 2*i) i += 1 self.assertRaises(IndexError, lambda: v[i+1]) i = 0 for x in self.t.iteritems(): self.assertEqual(x, (i, 2*i)) i += 1 items = list(self.t.items(min=12, max=20)) self.assertEqual(items, zip(range(12, 21), range(24, 43, 2))) items = list(self.t.iteritems(min=12, max=20)) self.assertEqual(items, zip(range(12, 21), range(24, 43, 2))) def testItemsNegativeIndex(self): L = [-3, 6, -11, 4] for i in L: self.t[i] = i L.sort() items = self.t.items() for i in range(-1, -5, -1): self.assertEqual(items[i], (L[i], L[i])) self.assertRaises(IndexError, lambda: items[-5]) def testDeleteInvalidKeyRaisesKeyError(self): self.assertRaises(KeyError, self._deletefail) def _deletefail(self): del self.t[1] def testMaxKeyMinKey(self): self.t[7] = 6 self.t[3] = 10 self.t[8] = 12 self.t[1] = 100 self.t[5] = 200 self.t[10] = 500 self.t[6] = 99 self.t[4] = 150 del self.t[7] t = self.t self.assertEqual(t.maxKey(), 10) self.assertEqual(t.maxKey(6), 6) self.assertEqual(t.maxKey(9), 8) self.assertEqual(t.minKey(), 1) self.assertEqual(t.minKey(3), 3) self.assertEqual(t.minKey(9), 10) try: t.maxKey(t.minKey() - 1) except ValueError, err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") try: t.minKey(t.maxKey() + 1) except ValueError, err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") def testClear(self): r = range(100) for x in r: rnd = random.choice(r) self.t[rnd] = 0 self.t.clear() diff = lsubtract(list(self.t.keys()), []) self.assertEqual(diff, []) def testUpdate(self): d={} l=[] for i in range(10000): k=random.randrange(-2000, 2001) d[k]=i l.append((k, i)) items=d.items() items.sort() self.t.update(d) self.assertEqual(list(self.t.items()), items) self.t.clear() self.assertEqual(list(self.t.items()), []) self.t.update(l) self.assertEqual(list(self.t.items()), items) # Before ZODB 3.4.2, update/construction from PersistentMapping failed. def testUpdateFromPersistentMapping(self): from persistent.mapping import PersistentMapping pm = PersistentMapping({1: 2}) self.t.update(pm) self.assertEqual(list(self.t.items()), [(1, 2)]) # Construction goes thru the same internals as .update(). t = self.t.__class__(pm) self.assertEqual(list(t.items()), [(1, 2)]) def testEmptyRangeSearches(self): t = self.t t.update([(1,1), (5,5), (9,9)]) self.assertEqual(list(t.keys(-6,-4)), [], list(t.keys(-6,-4))) self.assertEqual(list(t.keys(2,4)), [], list(t.keys(2,4))) self.assertEqual(list(t.keys(6,8)), [], list(t.keys(6,8))) self.assertEqual(list(t.keys(10,12)), [], list(t.keys(10,12))) self.assertEqual(list(t.keys(9, 1)), [], list(t.keys(9, 1))) # For IITreeSets, this one was returning 31 for len(keys), and # list(keys) produced a list with 100 elements. t.clear() t.update(zip(range(300), range(300))) keys = t.keys(200, 50) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) self.assertEqual(list(t.iterkeys(200, 50)), []) keys = t.keys(max=50, min=200) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) self.assertEqual(list(t.iterkeys(max=50, min=200)), []) def testSlicing(self): # Test that slicing of .keys()/.values()/.items() works exactly the # same way as slicing a Python list with the same contents. # This tests fixes to several bugs in this area, starting with # http://collector.zope.org/Zope/419, # "BTreeItems slice contains 1 too many elements". t = self.t for n in range(10): t.clear() self.assertEqual(len(t), 0) keys = [] values = [] items = [] for key in range(n): value = -2 * key t[key] = value keys.append(key) values.append(value) items.append((key, value)) self.assertEqual(len(t), n) kslice = t.keys() vslice = t.values() islice = t.items() self.assertEqual(len(kslice), n) self.assertEqual(len(vslice), n) self.assertEqual(len(islice), n) # Test whole-structure slices. x = kslice[:] self.assertEqual(list(x), keys[:]) x = vslice[:] self.assertEqual(list(x), values[:]) x = islice[:] self.assertEqual(list(x), items[:]) for lo in range(-2*n, 2*n+1): # Test one-sided slices. x = kslice[:lo] self.assertEqual(list(x), keys[:lo]) x = kslice[lo:] self.assertEqual(list(x), keys[lo:]) x = vslice[:lo] self.assertEqual(list(x), values[:lo]) x = vslice[lo:] self.assertEqual(list(x), values[lo:]) x = islice[:lo] self.assertEqual(list(x), items[:lo]) x = islice[lo:] self.assertEqual(list(x), items[lo:]) for hi in range(-2*n, 2*n+1): # Test two-sided slices. x = kslice[lo:hi] self.assertEqual(list(x), keys[lo:hi]) x = vslice[lo:hi] self.assertEqual(list(x), values[lo:hi]) x = islice[lo:hi] self.assertEqual(list(x), items[lo:hi]) # The specific test case from Zope collector 419. t.clear() for i in xrange(100): t[i] = 1 tslice = t.items()[20:80] self.assertEqual(len(tslice), 60) self.assertEqual(list(tslice), zip(range(20, 80), [1]*60)) def testIterators(self): t = self.t for keys in [], [-2], [1, 4], range(-170, 2000, 6): t.clear() for k in keys: t[k] = -3 * k self.assertEqual(list(t), keys) x = [] for k in t: x.append(k) self.assertEqual(x, keys) it = iter(t) self.assert_(it is iter(it)) x = [] try: while 1: x.append(it.next()) except StopIteration: pass self.assertEqual(x, keys) self.assertEqual(list(t.iterkeys()), keys) self.assertEqual(list(t.itervalues()), list(t.values())) self.assertEqual(list(t.iteritems()), list(t.items())) def testRangedIterators(self): t = self.t for keys in [], [-2], [1, 4], range(-170, 2000, 13): t.clear() values = [] for k in keys: value = -3 * k t[k] = value values.append(value) items = zip(keys, values) self.assertEqual(list(t.iterkeys()), keys) self.assertEqual(list(t.itervalues()), values) self.assertEqual(list(t.iteritems()), items) if not keys: continue min_mid_max = (keys[0], keys[len(keys) >> 1], keys[-1]) for key1 in min_mid_max: for lo in range(key1 - 1, key1 + 2): # Test one-sided range iterators. goodkeys = [k for k in keys if lo <= k] got = t.iterkeys(lo) self.assertEqual(goodkeys, list(got)) goodvalues = [t[k] for k in goodkeys] got = t.itervalues(lo) self.assertEqual(goodvalues, list(got)) gooditems = zip(goodkeys, goodvalues) got = t.iteritems(lo) self.assertEqual(gooditems, list(got)) for key2 in min_mid_max: for hi in range(key2 - 1, key2 + 2): goodkeys = [k for k in keys if lo <= k <= hi] got = t.iterkeys(min=lo, max=hi) self.assertEqual(goodkeys, list(got)) goodvalues = [t[k] for k in goodkeys] got = t.itervalues(lo, max=hi) self.assertEqual(goodvalues, list(got)) gooditems = zip(goodkeys, goodvalues) got = t.iteritems(max=hi, min=lo) self.assertEqual(gooditems, list(got)) def testBadUpdateTupleSize(self): # This one silently ignored the excess in Zope3. try: self.t.update([(1, 2, 3)]) except TypeError: pass else: self.fail("update() with 3-tuple didn't complain") # This one dumped core in Zope3. try: self.t.update([(1,)]) except TypeError: pass else: self.fail("update() with 1-tuple didn't complain") # This one should simply succeed. self.t.update([(1, 2)]) self.assertEqual(list(self.t.items()), [(1, 2)]) def testSimpleExclusivRanges(self): def identity(x): return x def dup(x): return [(y, y) for y in x] for methodname, f in (("keys", identity), ("values", identity), ("items", dup), ("iterkeys", identity), ("itervalues", identity), ("iteritems", dup)): t = self.t.__class__() meth = getattr(t, methodname, None) if meth is None: continue self.assertEqual(list(meth()), []) self.assertEqual(list(meth(excludemin=True)), []) self.assertEqual(list(meth(excludemax=True)), []) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) self._populate(t, 1) self.assertEqual(list(meth()), f([0])) self.assertEqual(list(meth(excludemin=True)), []) self.assertEqual(list(meth(excludemax=True)), []) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 2) self.assertEqual(list(meth()), f([0, 1])) self.assertEqual(list(meth(excludemin=True)), f([1])) self.assertEqual(list(meth(excludemax=True)), f([0])) self.assertEqual(list(meth(excludemin=True, excludemax=True)), []) t.clear() self._populate(t, 3) self.assertEqual(list(meth()), f([0, 1, 2])) self.assertEqual(list(meth(excludemin=True)), f([1, 2])) self.assertEqual(list(meth(excludemax=True)), f([0, 1])) self.assertEqual(list(meth(excludemin=True, excludemax=True)), f([1])) self.assertEqual(list(meth(-1, 3, excludemin=True, excludemax=True)), f([0, 1, 2])) self.assertEqual(list(meth(0, 3, excludemin=True, excludemax=True)), f([1, 2])) self.assertEqual(list(meth(-1, 2, excludemin=True, excludemax=True)), f([0, 1])) self.assertEqual(list(meth(0, 2, excludemin=True, excludemax=True)), f([1])) class NormalSetTests(Base): """ Test common to all set types """ def _populate(self, t, l): # Make some data t.update(range(l)) def testInsertReturnsValue(self): t = self.t self.assertEqual(t.insert(5) , 1) def testDuplicateInsert(self): t = self.t t.insert(5) self.assertEqual(t.insert(5) , 0) def testInsert(self): t = self.t t.insert(1) self.assert_(t.has_key(1)) self.assert_(1 in t) self.assert_(2 not in t) def testBigInsert(self): t = self.t r = xrange(10000) for x in r: t.insert(x) for x in r: self.assert_(t.has_key(x)) self.assert_(x in t) def testRemoveSucceeds(self): t = self.t r = xrange(10000) for x in r: t.insert(x) for x in r: t.remove(x) def testRemoveFails(self): self.assertRaises(KeyError, self._removenonexistent) def _removenonexistent(self): self.t.remove(1) def testHasKeyFails(self): t = self.t self.assert_(not t.has_key(1)) self.assert_(1 not in t) def testKeys(self): t = self.t r = xrange(1000) for x in r: t.insert(x) diff = lsubtract(t.keys(), r) self.assertEqual(diff, []) def testClear(self): t = self.t r = xrange(1000) for x in r: t.insert(x) t.clear() diff = lsubtract(t.keys(), []) self.assertEqual(diff , [], diff) def testMaxKeyMinKey(self): t = self.t t.insert(1) t.insert(2) t.insert(3) t.insert(8) t.insert(5) t.insert(10) t.insert(6) t.insert(4) self.assertEqual(t.maxKey() , 10) self.assertEqual(t.maxKey(6) , 6) self.assertEqual(t.maxKey(9) , 8) self.assertEqual(t.minKey() , 1) self.assertEqual(t.minKey(3) , 3) self.assertEqual(t.minKey(9) , 10) self.assert_(t.minKey() in t) self.assert_(t.minKey()-1 not in t) self.assert_(t.maxKey() in t) self.assert_(t.maxKey()+1 not in t) try: t.maxKey(t.minKey() - 1) except ValueError, err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") try: t.minKey(t.maxKey() + 1) except ValueError, err: self.assertEqual(str(err), "no key satisfies the conditions") else: self.fail("expected ValueError") def testUpdate(self): d={} l=[] for i in range(10000): k=random.randrange(-2000, 2001) d[k]=i l.append(k) items = d.keys() items.sort() self.t.update(l) self.assertEqual(list(self.t.keys()), items) def testEmptyRangeSearches(self): t = self.t t.update([1, 5, 9]) self.assertEqual(list(t.keys(-6,-4)), [], list(t.keys(-6,-4))) self.assertEqual(list(t.keys(2,4)), [], list(t.keys(2,4))) self.assertEqual(list(t.keys(6,8)), [], list(t.keys(6,8))) self.assertEqual(list(t.keys(10,12)), [], list(t.keys(10,12))) self.assertEqual(list(t.keys(9,1)), [], list(t.keys(9,1))) # For IITreeSets, this one was returning 31 for len(keys), and # list(keys) produced a list with 100 elements. t.clear() t.update(range(300)) keys = t.keys(200, 50) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) keys = t.keys(max=50, min=200) self.assertEqual(len(keys), 0) self.assertEqual(list(keys), []) def testSlicing(self): # Test that slicing of .keys() works exactly the same way as slicing # a Python list with the same contents. t = self.t for n in range(10): t.clear() self.assertEqual(len(t), 0) keys = range(10*n, 11*n) t.update(keys) self.assertEqual(len(t), n) kslice = t.keys() self.assertEqual(len(kslice), n) # Test whole-structure slices. x = kslice[:] self.assertEqual(list(x), keys[:]) for lo in range(-2*n, 2*n+1): # Test one-sided slices. x = kslice[:lo] self.assertEqual(list(x), keys[:lo]) x = kslice[lo:] self.assertEqual(list(x), keys[lo:]) for hi in range(-2*n, 2*n+1): # Test two-sided slices. x = kslice[lo:hi] self.assertEqual(list(x), keys[lo:hi]) def testIterator(self): t = self.t for keys in [], [-2], [1, 4], range(-170, 2000, 6): t.clear() t.update(keys) self.assertEqual(list(t), keys) x = [] for k in t: x.append(k) self.assertEqual(x, keys) it = iter(t) self.assert_(it is iter(it)) x = [] try: while 1: x.append(it.next()) except StopIteration: pass self.assertEqual(x, keys) class ExtendedSetTests(NormalSetTests): def testLen(self): t = self.t r = xrange(10000) for x in r: t.insert(x) self.assertEqual(len(t) , 10000, len(t)) def testGetItem(self): t = self.t r = xrange(10000) for x in r: t.insert(x) for x in r: self.assertEqual(t[x] , x) class BTreeTests(MappingBase): """ Tests common to all BTrees """ def tearDown(self): self.t._check() check(self.t) MappingBase.tearDown(self) def testDeleteNoChildrenWorks(self): self.t[5] = 6 self.t[2] = 10 self.t[6] = 12 self.t[1] = 100 self.t[3] = 200 self.t[10] = 500 self.t[4] = 99 del self.t[4] diff = lsubtract(self.t.keys(), [1,2,3,5,6,10]) self.assertEqual(diff , [], diff) def testDeleteOneChildWorks(self): self.t[5] = 6 self.t[2] = 10 self.t[6] = 12 self.t[1] = 100 self.t[3] = 200 self.t[10] = 500 self.t[4] = 99 del self.t[3] diff = lsubtract(self.t.keys(), [1,2,4,5,6,10]) self.assertEqual(diff , [], diff) def testDeleteTwoChildrenNoInorderSuccessorWorks(self): self.t[5] = 6 self.t[2] = 10 self.t[6] = 12 self.t[1] = 100 self.t[3] = 200 self.t[10] = 500 self.t[4] = 99 del self.t[2] diff = lsubtract(self.t.keys(), [1,3,4,5,6,10]) self.assertEqual(diff , [], diff) def testDeleteTwoChildrenInorderSuccessorWorks(self): # 7, 3, 8, 1, 5, 10, 6, 4 -- del 3 self.t[7] = 6 self.t[3] = 10 self.t[8] = 12 self.t[1] = 100 self.t[5] = 200 self.t[10] = 500 self.t[6] = 99 self.t[4] = 150 del self.t[3] diff = lsubtract(self.t.keys(), [1,4,5,6,7,8,10]) self.assertEqual(diff , [], diff) def testDeleteRootWorks(self): # 7, 3, 8, 1, 5, 10, 6, 4 -- del 7 self.t[7] = 6 self.t[3] = 10 self.t[8] = 12 self.t[1] = 100 self.t[5] = 200 self.t[10] = 500 self.t[6] = 99 self.t[4] = 150 del self.t[7] diff = lsubtract(self.t.keys(), [1,3,4,5,6,8,10]) self.assertEqual(diff , [], diff) def testRandomNonOverlappingInserts(self): added = {} r = range(100) for x in r: k = random.choice(r) if not added.has_key(k): self.t[k] = x added[k] = 1 addl = added.keys() addl.sort() diff = lsubtract(list(self.t.keys()), addl) self.assertEqual(diff , [], (diff, addl, list(self.t.keys()))) def testRandomOverlappingInserts(self): added = {} r = range(100) for x in r: k = random.choice(r) self.t[k] = x added[k] = 1 addl = added.keys() addl.sort() diff = lsubtract(self.t.keys(), addl) self.assertEqual(diff , [], diff) def testRandomDeletes(self): r = range(1000) added = [] for x in r: k = random.choice(r) self.t[k] = x added.append(k) deleted = [] for x in r: k = random.choice(r) if self.t.has_key(k): self.assert_(k in self.t) del self.t[k] deleted.append(k) if self.t.has_key(k): self.fail( "had problems deleting %s" % k ) badones = [] for x in deleted: if self.t.has_key(x): badones.append(x) self.assertEqual(badones , [], (badones, added, deleted)) def testTargetedDeletes(self): r = range(1000) for x in r: k = random.choice(r) self.t[k] = x for x in r: try: del self.t[x] except KeyError: pass self.assertEqual(realseq(self.t.keys()) , [], realseq(self.t.keys())) def testPathologicalRightBranching(self): r = range(1000) for x in r: self.t[x] = 1 self.assertEqual(realseq(self.t.keys()) , r, realseq(self.t.keys())) for x in r: del self.t[x] self.assertEqual(realseq(self.t.keys()) , [], realseq(self.t.keys())) def testPathologicalLeftBranching(self): r = range(1000) revr = r[:] revr.reverse() for x in revr: self.t[x] = 1 self.assertEqual(realseq(self.t.keys()) , r, realseq(self.t.keys())) for x in revr: del self.t[x] self.assertEqual(realseq(self.t.keys()) , [], realseq(self.t.keys())) def testSuccessorChildParentRewriteExerciseCase(self): add_order = [ 85, 73, 165, 273, 215, 142, 233, 67, 86, 166, 235, 225, 255, 73, 175, 171, 285, 162, 108, 28, 283, 258, 232, 199, 260, 298, 275, 44, 261, 291, 4, 181, 285, 289, 216, 212, 129, 243, 97, 48, 48, 159, 22, 285, 92, 110, 27, 55, 202, 294, 113, 251, 193, 290, 55, 58, 239, 71, 4, 75, 129, 91, 111, 271, 101, 289, 194, 218, 77, 142, 94, 100, 115, 101, 226, 17, 94, 56, 18, 163, 93, 199, 286, 213, 126, 240, 245, 190, 195, 204, 100, 199, 161, 292, 202, 48, 165, 6, 173, 40, 218, 271, 228, 7, 166, 173, 138, 93, 22, 140, 41, 234, 17, 249, 215, 12, 292, 246, 272, 260, 140, 58, 2, 91, 246, 189, 116, 72, 259, 34, 120, 263, 168, 298, 118, 18, 28, 299, 192, 252, 112, 60, 277, 273, 286, 15, 263, 141, 241, 172, 255, 52, 89, 127, 119, 255, 184, 213, 44, 116, 231, 173, 298, 178, 196, 89, 184, 289, 98, 216, 115, 35, 132, 278, 238, 20, 241, 128, 179, 159, 107, 206, 194, 31, 260, 122, 56, 144, 118, 283, 183, 215, 214, 87, 33, 205, 183, 212, 221, 216, 296, 40, 108, 45, 188, 139, 38, 256, 276, 114, 270, 112, 214, 191, 147, 111, 299, 107, 101, 43, 84, 127, 67, 205, 251, 38, 91, 297, 26, 165, 187, 19, 6, 73, 4, 176, 195, 90, 71, 30, 82, 139, 210, 8, 41, 253, 127, 190, 102, 280, 26, 233, 32, 257, 194, 263, 203, 190, 111, 218, 199, 29, 81, 207, 18, 180, 157, 172, 192, 135, 163, 275, 74, 296, 298, 265, 105, 191, 282, 277, 83, 188, 144, 259, 6, 173, 81, 107, 292, 231, 129, 65, 161, 113, 103, 136, 255, 285, 289, 1 ] delete_order = [ 276, 273, 12, 275, 2, 286, 127, 83, 92, 33, 101, 195, 299, 191, 22, 232, 291, 226, 110, 94, 257, 233, 215, 184, 35, 178, 18, 74, 296, 210, 298, 81, 265, 175, 116, 261, 212, 277, 260, 234, 6, 129, 31, 4, 235, 249, 34, 289, 105, 259, 91, 93, 119, 7, 183, 240, 41, 253, 290, 136, 75, 292, 67, 112, 111, 256, 163, 38, 126, 139, 98, 56, 282, 60, 26, 55, 245, 225, 32, 52, 40, 271, 29, 252, 239, 89, 87, 205, 213, 180, 97, 108, 120, 218, 44, 187, 196, 251, 202, 203, 172, 28, 188, 77, 90, 199, 297, 282, 141, 100, 161, 216, 73, 19, 17, 189, 30, 258 ] for x in add_order: self.t[x] = 1 for x in delete_order: try: del self.t[x] except KeyError: if self.t.has_key(x): self.assertEqual(1,2,"failed to delete %s" % x) def testRangeSearchAfterSequentialInsert(self): r = range(100) for x in r: self.t[x] = 0 diff = lsubtract(list(self.t.keys(0, 100)), r) self.assertEqual(diff , [], diff) def testRangeSearchAfterRandomInsert(self): r = range(100) a = {} for x in r: rnd = random.choice(r) self.t[rnd] = 0 a[rnd] = 0 diff = lsubtract(list(self.t.keys(0, 100)), a.keys()) self.assertEqual(diff , [], diff) def testPathologicalRangeSearch(self): t = self.t # Build a 2-level tree with at least two buckets. for i in range(200): t[i] = i items, dummy = t.__getstate__() self.assert_(len(items) > 2) # at least two buckets and a key # All values in the first bucket are < firstkey. All in the # second bucket are >= firstkey, and firstkey is the first key in # the second bucket. firstkey = items[1] therange = t.keys(-1, firstkey) self.assertEqual(len(therange), firstkey + 1) self.assertEqual(list(therange), range(firstkey + 1)) # Now for the tricky part. If we delete firstkey, the second bucket # loses its smallest key, but firstkey remains in the BTree node. # If we then do a high-end range search on firstkey, the BTree node # directs us to look in the second bucket, but there's no longer any # key <= firstkey in that bucket. The correct answer points to the # end of the *first* bucket. The algorithm has to be smart enough # to "go backwards" in the BTree then; if it doesn't, it will # erroneously claim that the range is empty. del t[firstkey] therange = t.keys(min=-1, max=firstkey) self.assertEqual(len(therange), firstkey) self.assertEqual(list(therange), range(firstkey)) def testInsertMethod(self): t = self.t t[0] = 1 self.assertEqual(t.insert(0, 1) , 0) self.assertEqual(t.insert(1, 1) , 1) self.assertEqual(lsubtract(list(t.keys()), [0,1]) , []) def testDamagedIterator(self): # A cute one from Steve Alexander. This caused the BTreeItems # object to go insane, accessing memory beyond the allocated part # of the bucket. If it fails, the symptom is either a C-level # assertion error (if the BTree code was compiled without NDEBUG), # or most likely a segfault (if the BTree code was compiled with # NDEBUG). t = self.t.__class__() self._populate(t, 10) # In order for this to fail, it's important that k be a "lazy" # iterator, referring to the BTree by indirect position (index) # instead of a fully materialized list. Then the position can # end up pointing into trash memory, if the bucket pointed to # shrinks. k = t.keys() for dummy in range(20): try: del t[k[0]] except RuntimeError, detail: self.assertEqual(str(detail), "the bucket being iterated " "changed size") break # tests of various type errors class TypeTest(TestCase): def testBadTypeRaises(self): self.assertRaises(TypeError, self._stringraises) self.assertRaises(TypeError, self._floatraises) self.assertRaises(TypeError, self._noneraises) class TestIOBTrees(TypeTest): def setUp(self): self.t = IOBTree() def _stringraises(self): self.t['c'] = 1 def _floatraises(self): self.t[2.5] = 1 def _noneraises(self): self.t[None] = 1 class TestOIBTrees(TypeTest): def setUp(self): self.t = OIBTree() def _stringraises(self): self.t[1] = 'c' def _floatraises(self): self.t[1] = 1.4 def _noneraises(self): self.t[1] = None def testEmptyFirstBucketReportedByGuido(self): b = self.t for i in xrange(29972): # reduce to 29971 and it works b[i] = i for i in xrange(30): # reduce to 29 and it works del b[i] b[i+40000] = i self.assertEqual(b.keys()[0], 30) class TestIIBTrees(TestCase): def setUp(self): self.t = IIBTree() def testNonIntegerKeyRaises(self): self.assertRaises(TypeError, self._stringraiseskey) self.assertRaises(TypeError, self._floatraiseskey) self.assertRaises(TypeError, self._noneraiseskey) def testNonIntegerValueRaises(self): self.assertRaises(TypeError, self._stringraisesvalue) self.assertRaises(TypeError, self._floatraisesvalue) self.assertRaises(TypeError, self._noneraisesvalue) def _stringraiseskey(self): self.t['c'] = 1 def _floatraiseskey(self): self.t[2.5] = 1 def _noneraiseskey(self): self.t[None] = 1 def _stringraisesvalue(self): self.t[1] = 'c' def _floatraisesvalue(self): self.t[1] = 1.4 def _noneraisesvalue(self): self.t[1] = None class TestIFBTrees(TestCase): def setUp(self): self.t = IFBTree() def testNonIntegerKeyRaises(self): self.assertRaises(TypeError, self._stringraiseskey) self.assertRaises(TypeError, self._floatraiseskey) self.assertRaises(TypeError, self._noneraiseskey) def testNonNumericValueRaises(self): self.assertRaises(TypeError, self._stringraisesvalue) self.assertRaises(TypeError, self._noneraisesvalue) self.t[1] = 1 self.t[1] = 1.0 def _stringraiseskey(self): self.t['c'] = 1 def _floatraiseskey(self): self.t[2.5] = 1 def _noneraiseskey(self): self.t[None] = 1 def _stringraisesvalue(self): self.t[1] = 'c' def _floatraisesvalue(self): self.t[1] = 1.4 def _noneraisesvalue(self): self.t[1] = None class TestIOSets(TestCase): def setUp(self): self.t = IOSet() def testNonIntegerInsertRaises(self): self.assertRaises(TypeError,self._insertstringraises) self.assertRaises(TypeError,self._insertfloatraises) self.assertRaises(TypeError,self._insertnoneraises) def _insertstringraises(self): self.t.insert('a') def _insertfloatraises(self): self.t.insert(1.4) def _insertnoneraises(self): self.t.insert(None) class DegenerateBTree(TestCase): # Build a degenerate tree (set). Boxes are BTree nodes. There are # 5 leaf buckets, each containing a single int. Keys in the BTree # nodes don't appear in the buckets. Seven BTree nodes are purely # indirection nodes (no keys). Buckets aren't all at the same depth: # # +------------------------+ # | 4 | # +------------------------+ # | | # | v # | +-+ # | | | # | +-+ # | | # v v # +-------+ +-------------+ # | 2 | | 6 10 | # +-------+ +-------------+ # | | | | | # v v v v v # +-+ +-+ +-+ +-+ +-+ # | | | | | | | | | | # +-+ +-+ +-+ +-+ +-+ # | | | | | # v v v v v # 1 3 +-+ 7 11 # | | # +-+ # | # v # 5 # # This is nasty for many algorithms. Consider a high-end range search # for 4. The BTree nodes direct it to the 5 bucket, but the correct # answer is the 3 bucket, which requires going in a different direction # at the very top node already. Consider a low-end range search for # 9. The BTree nodes direct it to the 7 bucket, but the correct answer # is the 11 bucket. This is also a nasty-case tree for deletions. def _build_degenerate_tree(self): # Build the buckets and chain them together. bucket11 = IISet([11]) bucket7 = IISet() bucket7.__setstate__(((7,), bucket11)) bucket5 = IISet() bucket5.__setstate__(((5,), bucket7)) bucket3 = IISet() bucket3.__setstate__(((3,), bucket5)) bucket1 = IISet() bucket1.__setstate__(((1,), bucket3)) # Build the deepest layers of indirection nodes. ts = IITreeSet tree1 = ts() tree1.__setstate__(((bucket1,), bucket1)) tree3 = ts() tree3.__setstate__(((bucket3,), bucket3)) tree5lower = ts() tree5lower.__setstate__(((bucket5,), bucket5)) tree5 = ts() tree5.__setstate__(((tree5lower,), bucket5)) tree7 = ts() tree7.__setstate__(((bucket7,), bucket7)) tree11 = ts() tree11.__setstate__(((bucket11,), bucket11)) # Paste together the middle layers. tree13 = ts() tree13.__setstate__(((tree1, 2, tree3), bucket1)) tree5711lower = ts() tree5711lower.__setstate__(((tree5, 6, tree7, 10, tree11), bucket5)) tree5711 = ts() tree5711.__setstate__(((tree5711lower,), bucket5)) # One more. t = ts() t.__setstate__(((tree13, 4, tree5711), bucket1)) t._check() check(t) return t, [1, 3, 5, 7, 11] def testBasicOps(self): t, keys = self._build_degenerate_tree() self.assertEqual(len(t), len(keys)) self.assertEqual(list(t.keys()), keys) # has_key actually returns the depth of a bucket. self.assertEqual(t.has_key(1), 4) self.assertEqual(t.has_key(3), 4) self.assertEqual(t.has_key(5), 6) self.assertEqual(t.has_key(7), 5) self.assertEqual(t.has_key(11), 5) for i in 0, 2, 4, 6, 8, 9, 10, 12: self.assert_(i not in t) def _checkRanges(self, tree, keys): self.assertEqual(len(tree), len(keys)) sorted_keys = keys[:] sorted_keys.sort() self.assertEqual(list(tree.keys()), sorted_keys) for k in keys: self.assert_(k in tree) if keys: lokey = sorted_keys[0] hikey = sorted_keys[-1] self.assertEqual(lokey, tree.minKey()) self.assertEqual(hikey, tree.maxKey()) else: lokey = hikey = 42 # Try all range searches. for lo in range(lokey - 1, hikey + 2): for hi in range(lo - 1, hikey + 2): for skipmin in False, True: for skipmax in False, True: wantlo, wanthi = lo, hi if skipmin: wantlo += 1 if skipmax: wanthi -= 1 want = [k for k in keys if wantlo <= k <= wanthi] got = list(tree.keys(lo, hi, skipmin, skipmax)) self.assertEqual(want, got) def testRanges(self): t, keys = self._build_degenerate_tree() self._checkRanges(t, keys) def testDeletes(self): # Delete keys in all possible orders, checking each tree along # the way. # This is a tough test. Previous failure modes included: # 1. A variety of assertion failures in _checkRanges. # 2. Assorted "Invalid firstbucket pointer" failures at # seemingly random times, coming out of the BTree destructor. # 3. Under Python 2.3 CVS, some baffling # RuntimeWarning: tp_compare didn't return -1 or -2 for exception # warnings, possibly due to memory corruption after a BTree # goes insane. t, keys = self._build_degenerate_tree() for oneperm in permutations(keys): t, keys = self._build_degenerate_tree() for key in oneperm: t.remove(key) keys.remove(key) t._check() check(t) self._checkRanges(t, keys) # We removed all the keys, so the tree should be empty now. self.assertEqual(t.__getstate__(), None) # A damaged tree may trigger an "invalid firstbucket pointer" # failure at the time its destructor is invoked. Try to force # that to happen now, so it doesn't look like a baffling failure # at some unrelated line. del t # trigger destructor class IIBucketTest(MappingBase): def setUp(self): self.t = IIBucket() class IFBucketTest(MappingBase): def setUp(self): self.t = IFBucket() class IOBucketTest(MappingBase): def setUp(self): self.t = IOBucket() class OIBucketTest(MappingBase): def setUp(self): self.t = OIBucket() class OOBucketTest(MappingBase): def setUp(self): self.t = OOBucket() class IITreeSetTest(NormalSetTests): def setUp(self): self.t = IITreeSet() class IFTreeSetTest(NormalSetTests): def setUp(self): self.t = IFTreeSet() class IOTreeSetTest(NormalSetTests): def setUp(self): self.t = IOTreeSet() class OITreeSetTest(NormalSetTests): def setUp(self): self.t = OITreeSet() class OOTreeSetTest(NormalSetTests): def setUp(self): self.t = OOTreeSet() class IISetTest(ExtendedSetTests): def setUp(self): self.t = IISet() # Collector 1843. Error returns were effectively ignored in # Bucket_rangeSearch(), leading to "delayed" errors, or worse. def testNonIntKeyRaises(self): self.t.insert(1) # This one used to fail to raise the TypeError when it occurred. self.assertRaises(TypeError, self.t.keys, "") # This one used to segfault. self.assertRaises(TypeError, self.t.keys, 0, "") class IFSetTest(ExtendedSetTests): def setUp(self): self.t = IFSet() class IOSetTest(ExtendedSetTests): def setUp(self): self.t = IOSet() class OISetTest(ExtendedSetTests): def setUp(self): self.t = OISet() class OOSetTest(ExtendedSetTests): def setUp(self): self.t = OOSet() class IIBTreeTest(BTreeTests): def setUp(self): self.t = IIBTree() class IFBTreeTest(BTreeTests): def setUp(self): self.t = IFBTree() class IOBTreeTest(BTreeTests): def setUp(self): self.t = IOBTree() class OIBTreeTest(BTreeTests): def setUp(self): self.t = OIBTree() class OOBTreeTest(BTreeTests): def setUp(self): self.t = OOBTree() # cmp error propagation tests class DoesntLikeBeingCompared: def __cmp__(self,other): raise ValueError('incomparable') class TestCmpError(TestCase): def testFoo(self): t = OOBTree() t['hello world'] = None try: t[DoesntLikeBeingCompared()] = None except ValueError,e: self.assertEqual(str(e), 'incomparable') else: self.fail('incomarable objects should not be allowed into ' 'the tree') def test_suite(): s = TestSuite() for klass in ( IIBucketTest, IIBTreeTest, IISetTest, IITreeSetTest, IFBucketTest, IFBTreeTest, IFSetTest, IFTreeSetTest, IOBucketTest, IOBTreeTest, IOSetTest, IOTreeSetTest, OOBucketTest, OOBTreeTest, OOSetTest, OOTreeSetTest, OIBucketTest, OIBTreeTest, OISetTest, OITreeSetTest, # Note: there is no TestOOBTrees. The next three are # checking for assorted TypeErrors, and when both keys # and values oare objects (OO), there's nothing to test. TestIIBTrees, TestIFBTrees, TestIOBTrees, TestOIBTrees, TestIOSets, DegenerateBTree, TestCmpError): s.addTest(makeSuite(klass)) return s ## utility functions def lsubtract(l1, l2): l1 = list(l1) l2 = list(l2) l = filter(lambda x, l1=l1: x not in l1, l2) l = l + filter(lambda x, l2=l2: x not in l2, l1) return l def realseq(itemsob): return [x for x in itemsob] def permutations(x): # Return a list of all permutations of list x. n = len(x) if n <= 1: return [x] result = [] x0 = x[0] for i in range(n): # Build the (n-1)! permutations with x[i] in the first position. xcopy = x[:] first, xcopy[i] = xcopy[i], x0 result.extend([[first] + p for p in permutations(xcopy[1:])]) return result def main(): TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/BTrees/sorters.c0000664000076400007640000003472510435502401017665 0ustar ajungajung/***************************************************************************** Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /* Revision information: $Id: sorters.c 25186 2004-06-02 15:07:33Z jim $ */ /* The only routine here intended to be used outside the file is size_t sort_int4_nodups(int *p, size_t n) Sort the array of n ints pointed at by p, in place, and also remove duplicates. Return the number of unique elements remaining, which occupy a contiguous and monotonically increasing slice of the array starting at p. Example: If the input array is [3, 1, 2, 3, 1, 5, 2], sort_int4_nodups returns 4, and the first 4 elements of the array are changed to [1, 2, 3, 5]. The content of the remaining array positions is not defined. Notes: + This is specific to 4-byte signed ints, with endianness natural to the platform. + 4*n bytes of available heap memory are required for best speed. */ #include #include #include #include #include /* The type of array elements to be sorted. Most of the routines don't care about the type, and will work fine for any scalar C type (provided they're recompiled with element_type appropriately redefined). However, the radix sort has to know everything about the type's internal representation. */ typedef int element_type; /* The radixsort is faster than the quicksort for large arrays, but radixsort has high fixed overhead, making it a poor choice for small arrays. The crossover point isn't critical, and is sensitive to things like compiler and machine cache structure, so don't worry much about this. */ #define QUICKSORT_BEATS_RADIXSORT 800U /* In turn, the quicksort backs off to an insertion sort for very small slices. MAX_INSERTION is the largest slice quicksort leaves entirely to insertion. Because this version of quicksort uses a median-of-3 rule for selecting a pivot, MAX_INSERTION must be at least 2 (so that quicksort has at least 3 values to look at in a slice). Again, the exact value here isn't critical. */ #define MAX_INSERTION 25U #if MAX_INSERTION < 2U # error "MAX_INSERTION must be >= 2" #endif /* LSB-first radix sort of the n elements in 'in'. 'work' is work storage at least as large as 'in'. Depending on how many swaps are done internally, the final result may come back in 'in' or 'work'; and that pointer is returned. radixsort_int4 is specific to signed 4-byte ints, with natural machine endianness. */ static element_type* radixsort_int4(element_type *in, element_type *work, size_t n) { /* count[i][j] is the number of input elements that have byte value j in byte position i, where byte position 0 is the LSB. Note that holding i fixed, the sum of count[i][j] over all j in range(256) is n. */ size_t count[4][256]; size_t i; int offset, offsetinc; /* Which byte position are we working on now? 0=LSB, 1, 2, ... */ int bytenum; assert(sizeof(element_type) == 4); assert(in); assert(work); /* Compute all of count in one pass. */ memset(count, 0, sizeof(count)); for (i = 0; i < n; ++i) { element_type const x = in[i]; ++count[0][(x ) & 0xff]; ++count[1][(x >> 8) & 0xff]; ++count[2][(x >> 16) & 0xff]; ++count[3][(x >> 24) & 0xff]; } /* For p an element_type* cast to char*, offset is how much farther we have to go to get to the LSB of the element; this is 0 for little- endian boxes and sizeof(element_type)-1 for big-endian. offsetinc is 1 or -1, respectively, telling us which direction to go from p+offset to get to the element's more-significant bytes. */ { int one = 1; if (*(char*)&one) { /* Little endian. */ offset = 0; offsetinc = 1; } else { /* Big endian. */ offset = sizeof(element_type) - 1; offsetinc = -1; } } /* The radix sort. */ for (bytenum = 0; bytenum < sizeof(element_type); ++bytenum, offset += offsetinc) { /* Do a stable distribution sort on byte position bytenum, from in to work. index[i] tells us the work index at which to store the next in element with byte value i. pinbyte points to the correct byte in the input array. */ size_t index[256]; unsigned char* pinbyte; size_t total = 0; size_t *pcount = count[bytenum]; /* Compute the correct output starting index for each possible byte value. */ if (bytenum < sizeof(element_type) - 1) { for (i = 0; i < 256; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 256) { /* All bytes in the current position have value i, so there's nothing to do on this pass. */ continue; } } else { /* The MSB of signed ints needs to be distributed differently than the other bytes, in order 0x80, 0x81, ... 0xff, 0x00, 0x01, ... 0x7f */ for (i = 128; i < 256; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 256) continue; for (i = 0; i < 128; ++i) { const size_t icount = pcount[i]; index[i] = total; total += icount; if (icount == n) break; } if (i < 128) continue; } assert(total == n); /* Distribute the elements according to byte value. Note that this is where most of the time is spent. Note: The loop is unrolled 4x by hand, for speed. This may be a pessimization someday, but was a significant win on my MSVC 6.0 timing tests. */ pinbyte = (unsigned char *)in + offset; i = 0; /* Reduce number of elements to copy to a multiple of 4. */ while ((n - i) & 0x3) { unsigned char byte = *pinbyte; work[index[byte]++] = in[i]; ++i; pinbyte += sizeof(element_type); } for (; i < n; i += 4, pinbyte += 4 * sizeof(element_type)) { unsigned char byte1 = *(pinbyte ); unsigned char byte2 = *(pinbyte + sizeof(element_type)); unsigned char byte3 = *(pinbyte + 2 * sizeof(element_type)); unsigned char byte4 = *(pinbyte + 3 * sizeof(element_type)); element_type in1 = in[i ]; element_type in2 = in[i+1]; element_type in3 = in[i+2]; element_type in4 = in[i+3]; work[index[byte1]++] = in1; work[index[byte2]++] = in2; work[index[byte3]++] = in3; work[index[byte4]++] = in4; } /* Swap in and work (just a pointer swap). */ { element_type *temp = in; in = work; work = temp; } } return in; } /* Remove duplicates from sorted array in, storing exactly one of each distinct element value into sorted array out. It's OK (and expected!) for in == out, but otherwise the n elements beginning at in must not overlap with the n beginning at out. Return the number of elements in out. */ static size_t uniq(element_type *out, element_type *in, size_t n) { size_t i; element_type lastelt; element_type *pout; assert(out); assert(in); if (n == 0) return 0; /* i <- first index in 'in' that contains a duplicate. in[0], in[1], ... in[i-1] are unique, but in[i-1] == in[i]. Set i to n if everything is unique. */ for (i = 1; i < n; ++i) { if (in[i-1] == in[i]) break; } /* in[:i] is unique; copy to out[:i] if needed. */ assert(i > 0); if (in != out) memcpy(out, in, i * sizeof(element_type)); pout = out + i; lastelt = in[i-1]; /* safe even when i == n */ for (++i; i < n; ++i) { element_type elt = in[i]; if (elt != lastelt) *pout++ = lastelt = elt; } return pout - out; } #if 0 /* insertionsort is no longer referenced directly, but I'd like to keep * the code here just in case. */ /* Straight insertion sort of the n elements starting at 'in'. */ static void insertionsort(element_type *in, size_t n) { element_type *p, *q; element_type minimum; /* smallest seen so far */ element_type *plimit = in + n; assert(in); if (n < 2) return; minimum = *in; for (p = in+1; p < plimit; ++p) { /* *in <= *(in+1) <= ... <= *(p-1). Slide *p into place. */ element_type thiselt = *p; if (thiselt < minimum) { /* This is a new minimum. This saves p-in compares when it happens, but should happen so rarely that it's not worth checking for its own sake: the point is that the far more popular 'else' branch can exploit that thiselt is *not* the smallest so far. */ memmove(in+1, in, (p - in) * sizeof(*in)); *in = minimum = thiselt; } else { /* thiselt >= minimum, so the loop will find a q with *q <= thiselt. This saves testing q >= in on each trip. It's such a simple loop that saving a per-trip test is a major speed win. */ for (q = p-1; *q > thiselt; --q) *(q+1) = *q; *(q+1) = thiselt; } } } #endif /* The maximum number of elements in the pending-work stack quicksort maintains. The maximum stack depth is approximately log2(n), so arrays of size up to approximately MAX_INSERTION * 2**STACKSIZE can be sorted. The memory burden for the stack is small, so better safe than sorry. */ #define STACKSIZE 60 /* A _stacknode remembers a contiguous slice of an array that needs to sorted. lo must be <= hi, and, unlike Python array slices, this includes both ends. */ struct _stacknode { element_type *lo; element_type *hi; }; static void quicksort(element_type *plo, size_t n) { element_type *phi; /* Swap two array elements. */ element_type _temp; #define SWAP(P, Q) (_temp = *(P), *(P) = *(Q), *(Q) = _temp) /* Stack of pending array slices to be sorted. */ struct _stacknode stack[STACKSIZE]; struct _stacknode *stackfree = stack; /* available stack slot */ /* Push an array slice on the pending-work stack. */ #define PUSH(PLO, PHI) \ do { \ assert(stackfree - stack < STACKSIZE); \ assert((PLO) <= (PHI)); \ stackfree->lo = (PLO); \ stackfree->hi = (PHI); \ ++stackfree; \ } while(0) assert(plo); phi = plo + n - 1; for (;;) { element_type pivot; element_type *pi, *pj; assert(plo <= phi); n = phi - plo + 1; if (n <= MAX_INSERTION) { /* Do a small insertion sort. Contra Knuth, we do this now instead of waiting until the end, because this little slice is likely still in cache now. */ element_type *p, *q; element_type minimum = *plo; for (p = plo+1; p <= phi; ++p) { /* *plo <= *(plo+1) <= ... <= *(p-1). Slide *p into place. */ element_type thiselt = *p; if (thiselt < minimum) { /* New minimum. */ memmove(plo+1, plo, (p - plo) * sizeof(*p)); *plo = minimum = thiselt; } else { /* thiselt >= minimum, so the loop will find a q with *q <= thiselt. */ for (q = p-1; *q > thiselt; --q) *(q+1) = *q; *(q+1) = thiselt; } } /* Pop another slice off the stack. */ if (stack == stackfree) break; /* no more slices -- we're done */ --stackfree; plo = stackfree->lo; phi = stackfree->hi; continue; } /* Parition the slice. For pivot, take the median of the leftmost, rightmost, and middle elements. First sort those three; then the median is the middle one. For technical reasons, the middle element is swapped to plo+1 first (see Knuth Vol 3 Ed 2 section 5.2.2 exercise 55 -- reverse-sorted arrays can take quadratic time otherwise!). */ { element_type *plop1 = plo + 1; element_type *pmid = plo + (n >> 1); assert(plo < pmid && pmid < phi); SWAP(plop1, pmid); /* Sort plo, plop1, phi. */ /* Smaller of rightmost two -> middle. */ if (*plop1 > *phi) SWAP(plop1, phi); /* Smallest of all -> left; if plo is already the smallest, the sort is complete. */ if (*plo > *plop1) { SWAP(plo, plop1); /* Largest of all -> right. */ if (*plop1 > *phi) SWAP(plop1, phi); } pivot = *plop1; pi = plop1; } assert(*plo <= pivot); assert(*pi == pivot); assert(*phi >= pivot); pj = phi; /* Partition wrt pivot. This is the time-critical part, and nearly every decision in the routine aims at making this loop as fast as possible -- even small points like arranging that all loop tests can be done correctly at the bottoms of loops instead of the tops, and that pointers can be derefenced directly as-is (without fiddly +1 or -1). The aim is to make the C here so simple that a compiler has a good shot at doing as well as hand-crafted assembler. */ for (;;) { /* Invariants: 1. pi < pj. 2. All elements at plo, plo+1 .. pi are <= pivot. 3. All elements at pj, pj+1 .. phi are >= pivot. 4. There is an element >= pivot to the right of pi. 5. There is an element <= pivot to the left of pj. Note that #4 and #5 save us from needing to check that the pointers stay in bounds. */ assert(pi < pj); do { ++pi; } while (*pi < pivot); assert(pi <= pj); do { --pj; } while (*pj > pivot); assert(pj >= pi - 1); if (pi < pj) SWAP(pi, pj); else break; } assert(plo+1 < pi && pi <= phi); assert(plo < pj && pj < phi); assert(*pi >= pivot); assert( (pi == pj && *pj == pivot) || (pj + 1 == pi && *pj <= pivot) ); /* Swap pivot into its final position, pj. */ assert(plo[1] == pivot); plo[1] = *pj; *pj = pivot; /* Subfiles are from plo to pj-1 inclusive, and pj+1 to phi inclusive. Push the larger one, and loop back to do the smaller one directly. */ if (pj - plo >= phi - pj) { PUSH(plo, pj-1); plo = pj+1; } else { PUSH(pj+1, phi); phi = pj-1; } } #undef PUSH #undef SWAP } /* Sort p and remove duplicates, as fast as we can. */ static size_t sort_int4_nodups(int *p, size_t n) { size_t nunique; element_type *work; assert(sizeof(int) == sizeof(element_type)); assert(p); /* Use quicksort if the array is small, OR if malloc can't find enough temp memory for radixsort. */ work = NULL; if (n > QUICKSORT_BEATS_RADIXSORT) work = (element_type *)malloc(n * sizeof(element_type)); if (work) { element_type *out = radixsort_int4(p, work, n); nunique = uniq(p, out, n); free(work); } else { quicksort(p, n); nunique = uniq(p, p, n); } return nunique; } Zope-2.8.7-final/lib/python/BTrees/DEPENDENCIES.cfg0000664000076400007640000000004610435502401020314 0ustar ajungajungpersistent transaction zope.interface Zope-2.8.7-final/lib/python/BTrees/IOBTree.py0000664000076400007640000000130710435502401017611 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # hack to overcome dynamic-linking headache. from _IOBTree import * Zope-2.8.7-final/lib/python/BTrees/IFBTree.py0000664000076400007640000000130710435502401017600 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # hack to overcome dynamic-linking headache. from _IFBTree import * Zope-2.8.7-final/lib/python/BTrees/SetOpTemplate.c0000664000076400007640000003635510435502401020713 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /**************************************************************************** Set operations ****************************************************************************/ #define SETOPTEMPLATE_C "$Id: SetOpTemplate.c 28613 2004-12-09 21:02:35Z tim_one $\n" #ifdef KEY_CHECK static int nextKeyAsSet(SetIteration *i) { if (i->position >= 0) { if (i->position) { DECREF_KEY(i->key); i->position = -1; } else i->position = 1; } return 0; } #endif /* initSetIteration * * Start the set iteration protocol. See the comments at struct SetIteration. * * Arguments * i The address of a SetIteration control struct. * s The address of the set, bucket, BTree, ..., to be iterated. * useValues Boolean; if true, and s has values (is a mapping), copy * them into i->value each time i->next() is called; else * ignore s's values even if s is a mapping. * * Return * 0 on success; -1 and an exception set if error. * i.usesValue is set to 1 (true) if s has values and useValues was * true; else usesValue is set to 0 (false). * i.set gets a new reference to s, or to some other object used to * iterate over s. * i.position is set to 0. * i.next is set to an appropriate iteration function. * i.key and i.value are left alone. * * Internal * i.position < 0 means iteration terminated. * i.position = 0 means iteration hasn't yet begun (next() hasn't * been called yet). * In all other cases, i.key, and possibly i.value, own references. * These must be cleaned up, either by next() routines, or by * finiSetIteration. * next() routines must ensure the above. They should return without * doing anything when i.position < 0. * It's the responsibility of {init, fini}setIteration to clean up * the reference in i.set, and to ensure that no stale references * live in i.key or i.value if iteration terminates abnormally. * A SetIteration struct has been cleaned up iff i.set is NULL. */ static int initSetIteration(SetIteration *i, PyObject *s, int useValues) { i->set = NULL; i->position = -1; /* set to 0 only on normal return */ i->usesValue = 0; /* assume it's a set or that values aren't iterated */ if (PyObject_IsInstance(s, (PyObject *)&BucketType)) { i->set = s; Py_INCREF(s); if (useValues) { i->usesValue = 1; i->next = nextBucket; } else i->next = nextSet; } else if (PyObject_IsInstance(s, (PyObject *)&SetType)) { i->set = s; Py_INCREF(s); i->next = nextSet; } else if (PyObject_IsInstance(s, (PyObject *)&BTreeType)) { i->set = BTree_rangeSearch(BTREE(s), NULL, NULL, 'i'); UNLESS(i->set) return -1; if (useValues) { i->usesValue = 1; i->next = nextBTreeItems; } else i->next = nextTreeSetItems; } else if (PyObject_IsInstance(s, (PyObject *)&TreeSetType)) { i->set = BTree_rangeSearch(BTREE(s), NULL, NULL, 'k'); UNLESS(i->set) return -1; i->next = nextTreeSetItems; } #ifdef KEY_CHECK else if (KEY_CHECK(s)) { int copied = 1; COPY_KEY_FROM_ARG(i->key, s, copied); UNLESS (copied) return -1; INCREF_KEY(i->key); i->set = s; Py_INCREF(s); i->next = nextKeyAsSet; } #endif else { PyErr_SetString(PyExc_TypeError, "invalid argument"); return -1; } i->position = 0; return 0; } #ifndef MERGE_WEIGHT #define MERGE_WEIGHT(O, w) (O) #endif static int copyRemaining(Bucket *r, SetIteration *i, int merge, /* See comment # 42 */ #ifdef MERGE VALUE_TYPE w) #else int w) #endif { while (i->position >= 0) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) return -1; COPY_KEY(r->keys[r->len], i->key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i->value, w)); INCREF_VALUE(r->values[r->len]); } r->len++; if (i->next(i) < 0) return -1; } return 0; } /* This is the workhorse for all set merge operations: the weighted and * unweighted flavors of union and intersection, and set difference. The * algorithm is conceptually simple but the code is complicated due to all * the options. * * s1, s2 * The input collections to be merged. * * usevalues1, usevalues2 * Booleans. In the output, should values from s1 (or s2) be used? This * only makes sense when an operation intends to support mapping outputs; * these should both be false for operations that want pure set outputs. * * w1, w2 * If usevalues1(2) are true, these are the weights to apply to the * input values. * * c1 * Boolean. Should keys that appear in c1 but not c2 appear in the output? * c12 * Boolean. Should keys that appear in both inputs appear in the output? * c2 * Boolean. Should keys that appear in c2 but not c1 appear in the output? * * Returns NULL if error, else a Set or Bucket, depending on whether a set or * mapping was requested. */ static PyObject * set_operation(PyObject *s1, PyObject *s2, int usevalues1, int usevalues2, /* Comment # 42 The following ifdef works around a template/type problem Weights are passed as integers. In particular, the weight passed by difference is one. This works find in the int value and float value cases but makes no sense in the object value case. In the object value case, we don't do merging, so we don't use the weights, so it doesn't matter what they are. */ #ifdef MERGE VALUE_TYPE w1, VALUE_TYPE w2, #else int w1, int w2, #endif int c1, int c12, int c2) { Bucket *r=0; SetIteration i1 = {0,0,0}, i2 = {0,0,0}; int cmp, merge; if (initSetIteration(&i1, s1, usevalues1) < 0) goto err; if (initSetIteration(&i2, s2, usevalues2) < 0) goto err; merge = i1.usesValue | i2.usesValue; if (merge) { #ifndef MERGE if (c12 && i1.usesValue && i2.usesValue) goto invalid_set_operation; #endif if (! i1.usesValue&& i2.usesValue) { SetIteration t; int i; /* See comment # 42 above */ #ifdef MERGE VALUE_TYPE v; #else int v; #endif t=i1; i1=i2; i2=t; i=c1; c1=c2; c2=i; v=w1; w1=w2; w2=v; } #ifdef MERGE_DEFAULT i1.value=MERGE_DEFAULT; i2.value=MERGE_DEFAULT; #else if (i1.usesValue) { if (! i2.usesValue && c2) goto invalid_set_operation; } else { if (c1 || c12) goto invalid_set_operation; } #endif UNLESS(r=BUCKET(PyObject_CallObject(OBJECT(&BucketType), NULL))) goto err; } else { UNLESS(r=BUCKET(PyObject_CallObject(OBJECT(&SetType), NULL))) goto err; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; while (i1.position >= 0 && i2.position >= 0) { TEST_KEY_SET_OR(cmp, i1.key, i2.key) goto err; if(cmp < 0) { if(c1) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i1.key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i1.value, w1)); INCREF_VALUE(r->values[r->len]); } r->len++; } if (i1.next(&i1) < 0) goto err; } else if(cmp==0) { if(c12) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i1.key); INCREF_KEY(r->keys[r->len]); if (merge) { #ifdef MERGE r->values[r->len] = MERGE(i1.value, w1, i2.value, w2); #else COPY_VALUE(r->values[r->len], i1.value); INCREF_VALUE(r->values[r->len]); #endif } r->len++; } if (i1.next(&i1) < 0) goto err; if (i2.next(&i2) < 0) goto err; } else { if(c2) { if(r->len >= r->size && Bucket_grow(r, -1, ! merge) < 0) goto err; COPY_KEY(r->keys[r->len], i2.key); INCREF_KEY(r->keys[r->len]); if (merge) { COPY_VALUE(r->values[r->len], MERGE_WEIGHT(i2.value, w2)); INCREF_VALUE(r->values[r->len]); } r->len++; } if (i2.next(&i2) < 0) goto err; } } if(c1 && copyRemaining(r, &i1, merge, w1) < 0) goto err; if(c2 && copyRemaining(r, &i2, merge, w2) < 0) goto err; finiSetIteration(&i1); finiSetIteration(&i2); return OBJECT(r); #ifndef MERGE_DEFAULT invalid_set_operation: PyErr_SetString(PyExc_TypeError, "invalid set operation"); #endif err: finiSetIteration(&i1); finiSetIteration(&i2); Py_XDECREF(r); return NULL; } static PyObject * difference_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None || o2 == Py_None) { /* difference(None, X) -> None; difference(X, None) -> X */ Py_INCREF(o1); return o1; } return set_operation(o1, o2, 1, 0, /* preserve values from o1, ignore o2's */ 1, 0, /* o1's values multiplied by 1 */ 1, 0, 0); /* take only keys unique to o1 */ } static PyObject * union_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None) { Py_INCREF(o2); return o2; } else if (o2 == Py_None) { Py_INCREF(o1); return o1; } return set_operation(o1, o2, 0, 0, /* ignore values in both */ 1, 1, /* the weights are irrelevant */ 1, 1, 1); /* take all keys */ } static PyObject * intersection_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; UNLESS(PyArg_ParseTuple(args, "OO", &o1, &o2)) return NULL; if (o1 == Py_None) { Py_INCREF(o2); return o2; } else if (o2 == Py_None) { Py_INCREF(o1); return o1; } return set_operation(o1, o2, 0, 0, /* ignore values in both */ 1, 1, /* the weights are irrelevant */ 0, 1, 0); /* take only keys common to both */ } #ifdef MERGE static PyObject * wunion_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; VALUE_TYPE w1 = 1, w2 = 1; UNLESS(PyArg_ParseTuple(args, "OO|" VALUE_PARSE VALUE_PARSE, &o1, &o2, &w1, &w2) ) return NULL; if (o1 == Py_None) return Py_BuildValue(VALUE_PARSE "O", (o2 == Py_None ? 0 : w2), o2); else if (o2 == Py_None) return Py_BuildValue(VALUE_PARSE "O", w1, o1); o1 = set_operation(o1, o2, 1, 1, w1, w2, 1, 1, 1); if (o1) ASSIGN(o1, Py_BuildValue(VALUE_PARSE "O", (VALUE_TYPE)1, o1)); return o1; } static PyObject * wintersection_m(PyObject *ignored, PyObject *args) { PyObject *o1, *o2; VALUE_TYPE w1 = 1, w2 = 1; UNLESS(PyArg_ParseTuple(args, "OO|" VALUE_PARSE VALUE_PARSE, &o1, &o2, &w1, &w2) ) return NULL; if (o1 == Py_None) return Py_BuildValue(VALUE_PARSE "O", (o2 == Py_None ? 0 : w2), o2); else if (o2 == Py_None) return Py_BuildValue(VALUE_PARSE "O", w1, o1); o1 = set_operation(o1, o2, 1, 1, w1, w2, 0, 1, 0); if (o1) ASSIGN(o1, Py_BuildValue(VALUE_PARSE "O", ((o1->ob_type == (PyTypeObject*)(&SetType)) ? w2+w1 : 1), o1)); return o1; } #endif #ifdef MULTI_INT_UNION #include "sorters.c" /* Input is a sequence of integer sets (or convertible to sets by the set iteration protocol). Output is the union of the sets. The point is to run much faster than doing pairs of unions. */ static PyObject * multiunion_m(PyObject *ignored, PyObject *args) { PyObject *seq; /* input sequence */ int n; /* length of input sequence */ PyObject *set = NULL; /* an element of the input sequence */ Bucket *result; /* result set */ SetIteration setiter = {0}; int i; UNLESS(PyArg_ParseTuple(args, "O", &seq)) return NULL; n = PyObject_Length(seq); if (n < 0) return NULL; /* Construct an empty result set. */ result = BUCKET(PyObject_CallObject(OBJECT(&SetType), NULL)); if (result == NULL) return NULL; /* For each set in the input sequence, append its elements to the result set. At this point, we ignore the possibility of duplicates. */ for (i = 0; i < n; ++i) { set = PySequence_GetItem(seq, i); if (set == NULL) goto Error; /* If set is a bucket, do a straight resize + memcpy. */ if (set->ob_type == (PyTypeObject*)&SetType || set->ob_type == (PyTypeObject*)&BucketType) { Bucket *b = BUCKET(set); int status = 0; UNLESS (PER_USE(b)) goto Error; if (b->len) status = bucket_append(result, b, 0, b->len, 0, i < n-1); PER_UNUSE(b); if (status < 0) goto Error; } else { /* No cheap way: iterate over set's elements one at a time. */ if (initSetIteration(&setiter, set, 0) < 0) goto Error; if (setiter.next(&setiter) < 0) goto Error; while (setiter.position >= 0) { if (result->len >= result->size && Bucket_grow(result, -1, 1) < 0) goto Error; COPY_KEY(result->keys[result->len], setiter.key); ++result->len; /* We know the key is an int, so no need to incref it. */ if (setiter.next(&setiter) < 0) goto Error; } finiSetIteration(&setiter); } Py_DECREF(set); set = NULL; } /* Combine, sort, remove duplicates, and reset the result's len. If the set shrinks (which happens if and only if there are duplicates), no point to realloc'ing the set smaller, as we expect the result set to be short-lived. */ if (result->len > 0) { size_t newlen; /* number of elements in final result set */ newlen = sort_int4_nodups(result->keys, (size_t)result->len); result->len = (int)newlen; } return (PyObject *)result; Error: Py_DECREF(result); Py_XDECREF(set); finiSetIteration(&setiter); return NULL; } #endif Zope-2.8.7-final/lib/python/BTrees/SetTemplate.c0000664000076400007640000002071710435502401020407 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define SETTEMPLATE_C "$Id: SetTemplate.c 25186 2004-06-02 15:07:33Z jim $\n" static PyObject * Set_insert(Bucket *self, PyObject *args) { PyObject *key; int i; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if ( (i=_bucket_set(self, key, Py_None, 1, 1, 0)) < 0) return NULL; return PyInt_FromLong(i); } /* _Set_update and _TreeSet_update are identical except for the function they call to add the element to the set. */ static int _Set_update(Bucket *self, PyObject *seq) { int n = -1; PyObject *iter, *v; int ind; iter = PyObject_GetIter(seq); if (iter == NULL) return -1; while (1) { v = PyIter_Next(iter); if (v == NULL) { if (PyErr_Occurred()) goto err; else break; } ind = _bucket_set(self, v, Py_None, 1, 1, 0); Py_DECREF(v); if (ind < 0) goto err; else n += ind; } /* n starts out at -1, which is the error return value. If this point is reached, then there is no error. n must be incremented to account for the initial value of -1 instead of 0. */ n++; err: Py_DECREF(iter); return n; } static PyObject * Set_update(Bucket *self, PyObject *args) { PyObject *seq = NULL; int n = 0; if (!PyArg_ParseTuple(args, "|O:update", &seq)) return NULL; if (seq) { n = _Set_update(self, seq); if (n < 0) return NULL; } return PyInt_FromLong(n); } static PyObject * Set_remove(Bucket *self, PyObject *args) { PyObject *key; UNLESS (PyArg_ParseTuple(args, "O", &key)) return NULL; if (_bucket_set(self, key, NULL, 0, 1, 0) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static int _set_setstate(Bucket *self, PyObject *args) { PyObject *k, *items; Bucket *next=0; int i, l, copied=1; KEY_TYPE *keys; UNLESS (PyArg_ParseTuple(args, "O|O", &items, &next)) return -1; if ((l=PyTuple_Size(items)) < 0) return -1; for (i=self->len; --i >= 0; ) { DECREF_KEY(self->keys[i]); } self->len=0; if (self->next) { Py_DECREF(self->next); self->next=0; } if (l > self->size) { UNLESS (keys=BTree_Realloc(self->keys, sizeof(KEY_TYPE)*l)) return -1; self->keys=keys; self->size=l; } for (i=0; ikeys[i], k, copied); UNLESS (copied) return -1; INCREF_KEY(self->keys[i]); } self->len=l; if (next) { self->next=next; Py_INCREF(next); } return 0; } static PyObject * set_setstate(Bucket *self, PyObject *args) { int r; UNLESS (PyArg_ParseTuple(args, "O", &args)) return NULL; PER_PREVENT_DEACTIVATION(self); r=_set_setstate(self, args); PER_UNUSE(self); if (r < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static struct PyMethodDef Set_methods[] = { {"__getstate__", (PyCFunction) bucket_getstate, METH_VARARGS, "__getstate__() -- Return the picklable state of the object"}, {"__setstate__", (PyCFunction) set_setstate, METH_VARARGS, "__setstate__() -- Set the state of the object"}, {"keys", (PyCFunction) bucket_keys, METH_KEYWORDS, "keys() -- Return the keys"}, {"has_key", (PyCFunction) bucket_has_key, METH_O, "has_key(key) -- Test whether the bucket contains the given key"}, {"clear", (PyCFunction) bucket_clear, METH_VARARGS, "clear() -- Remove all of the items from the bucket"}, {"maxKey", (PyCFunction) Bucket_maxKey, METH_VARARGS, "maxKey([key]) -- Find the maximum key\n\n" "If an argument is given, find the maximum <= the argument"}, {"minKey", (PyCFunction) Bucket_minKey, METH_VARARGS, "minKey([key]) -- Find the minimum key\n\n" "If an argument is given, find the minimum >= the argument"}, #ifdef PERSISTENT {"_p_resolveConflict", (PyCFunction) bucket__p_resolveConflict, METH_VARARGS, "_p_resolveConflict() -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction) bucket__p_deactivate, METH_KEYWORDS, "_p_deactivate() -- Reinitialize from a newly created copy"}, #endif {"insert", (PyCFunction)Set_insert, METH_VARARGS, "insert(id,[ignored]) -- Add a key to the set"}, {"update", (PyCFunction)Set_update, METH_VARARGS, "update(seq) -- Add the items from the given sequence to the set"}, {"remove", (PyCFunction)Set_remove, METH_VARARGS, "remove(id) -- Remove an id from the set"}, {NULL, NULL} /* sentinel */ }; static int Set_init(PyObject *self, PyObject *args, PyObject *kwds) { PyObject *v = NULL; if (!PyArg_ParseTuple(args, "|O:" MOD_NAME_PREFIX "Set", &v)) return -1; if (v) return _Set_update((Bucket *)self, v); else return 0; } static PyObject * set_repr(Bucket *self) { static PyObject *format; PyObject *r, *t; if (!format) format = PyString_FromString(MOD_NAME_PREFIX "Set(%s)"); UNLESS (t = PyTuple_New(1)) return NULL; UNLESS (r = bucket_keys(self, NULL, NULL)) goto err; PyTuple_SET_ITEM(t, 0, r); r = t; ASSIGN(r, PyString_Format(format, r)); return r; err: Py_DECREF(t); return NULL; } static int set_length(Bucket *self) { int r; PER_USE_OR_RETURN(self, -1); r = self->len; PER_UNUSE(self); return r; } static PyObject * set_item(Bucket *self, int index) { PyObject *r=0; PER_USE_OR_RETURN(self, NULL); if (index >= 0 && index < self->len) { COPY_KEY_TO_OBJECT(r, self->keys[index]); } else IndexError(index); PER_UNUSE(self); return r; } static PySequenceMethods set_as_sequence = { (inquiry)set_length, /* sq_length */ (binaryfunc)0, /* sq_concat */ (intargfunc)0, /* sq_repeat */ (intargfunc)set_item, /* sq_item */ (intintargfunc)0, /* sq_slice */ (intobjargproc)0, /* sq_ass_item */ (intintobjargproc)0, /* sq_ass_slice */ (objobjproc)bucket_contains, /* sq_contains */ 0, /* sq_inplace_concat */ 0, /* sq_inplace_repeat */ }; static PyTypeObject SetType = { PyObject_HEAD_INIT(NULL) /* PyPersist_Type */ 0, /* ob_size */ MODULE_NAME MOD_NAME_PREFIX "Set", /* tp_name */ sizeof(Bucket), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)bucket_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ (reprfunc)set_repr, /* tp_repr */ 0, /* tp_as_number */ &set_as_sequence, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ (traverseproc)bucket_traverse, /* tp_traverse */ (inquiry)bucket_tp_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ (getiterfunc)Bucket_getiter, /* tp_iter */ 0, /* tp_iternext */ Set_methods, /* tp_methods */ Bucket_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ Set_init, /* tp_init */ 0, /* tp_alloc */ 0, /*PyType_GenericNew,*/ /* tp_new */ }; static int nextSet(SetIteration *i) { if (i->position >= 0) { UNLESS(PER_USE(BUCKET(i->set))) return -1; if (i->position) { DECREF_KEY(i->key); } if (i->position < BUCKET(i->set)->len) { COPY_KEY(i->key, BUCKET(i->set)->keys[i->position]); INCREF_KEY(i->key); i->position ++; } else { i->position = -1; PER_ACCESSED(BUCKET(i->set)); } PER_ALLOW_DEACTIVATION(BUCKET(i->set)); } return 0; } Zope-2.8.7-final/lib/python/BTrees/OOBTree.py0000664000076400007640000000130710435502401017617 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # hack to overcome dynamic-linking headache. from _OOBTree import * Zope-2.8.7-final/lib/python/BTrees/Interfaces.py0000664000076400007640000003301110435502401020440 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from zope.interface import Interface class ICollection(Interface): def clear(): """Remove all of the items from the collection.""" def __nonzero__(): """Check if the collection is non-empty. Return a true value if the collection is non-empty and a false value otherwise. """ class IReadSequence(Interface): def __getitem__(index): """Return the value at the given index. An IndexError is raised if the index cannot be found. """ def __getslice__(index1, index2): """Return a subsequence from the original sequence. The subsequence includes the items from index1 up to, but not including, index2. """ class IKeyed(ICollection): def has_key(key): """Check whether the object has an item with the given key. Return a true value if the key is present, else a false value. """ def keys(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the keys in the collection. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to keys greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to keys strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the smallest key is excluded. If a max is specified, then output is constrained to keys less than or equal to the given max, and, if excludemax is specified and true, is further constrained to keys strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the largest key is excluded. """ def maxKey(key=None): """Return the maximum key. If a key argument if provided and not None, return the largest key that is less than or equal to the argument. Raise an exception if no such key exists. """ def minKey(key=None): """Return the minimum key. If a key argument if provided and not None, return the smallest key that is greater than or equal to the argument. Raise an exception if no such key exists. """ class ISetMutable(IKeyed): def insert(key): """Add the key (value) to the set. If the key was already in the set, return 0, otherwise return 1. """ def remove(key): """Remove the key from the set.""" def update(seq): """Add the items from the given sequence to the set.""" class ISized(Interface): """An object that supports __len__.""" def __len__(): """Return the number of items in the container.""" class IKeySequence(IKeyed, ISized): def __getitem__(index): """Return the key in the given index position. This allows iteration with for loops and use in functions, like map and list, that read sequences. """ class ISet(IKeySequence, ISetMutable): pass class ITreeSet(IKeyed, ISetMutable): pass class IMinimalDictionary(ISized, IKeyed): def get(key, default): """Get the value associated with the given key. Return the default if has_key(key) is false. """ def __setitem__(key, value): """Set the value associated with the given key.""" def __delitem__(key): """Delete the value associated with the given key. Raise KeyError if the key if has_key(key) is false. """ def values(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the values in the collection. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to values whose keys are greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to values whose keys are strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the value corresponding to the smallest key is excluded. If a max is specified, then output is constrained to values whose keys are less than or equal to the given max, and, if excludemax is specified and true, is further constrained to values whose keys are strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the value corresponding to the largest key is excluded. """ def items(min=None, max=None, excludemin=False, excludemax=False): """Return an IReadSequence containing the items in the collection. An item is a 2-tuple, a (key, value) pair. The type of the IReadSequence is not specified. It could be a list or a tuple or some other type. All arguments are optional, and may be specified as keyword arguments, or by position. If a min is specified, then output is constrained to items whose keys are greater than or equal to the given min, and, if excludemin is specified and true, is further constrained to items whose keys are strictly greater than min. A min value of None is ignored. If min is None or not specified, and excludemin is true, the item with the smallest key is excluded. If a max is specified, then output is constrained to items whose keys are less than or equal to the given max, and, if excludemax is specified and true, is further constrained to items whose keys are strictly less than max. A max value of None is ignored. If max is None or not specified, and excludemax is true, the item with the largest key is excluded. """ class IDictionaryIsh(IMinimalDictionary): def update(collection): """Add the items from the given collection object to the collection. The input collection must be a sequence of (key, value) 2-tuples, or an object with an 'items' method that returns a sequence of (key, value) pairs. """ def byValue(minValue): """Return a sequence of (value, key) pairs, sorted by value. Values < minValue are omitted and other values are "normalized" by the minimum value. This normalization may be a noop, but, for integer values, the normalization is division. """ class IBTree(IDictionaryIsh): def insert(key, value): """Insert a key and value into the collection. If the key was already in the collection, then there is no change and 0 is returned. If the key was not already in the collection, then the item is added and 1 is returned. This method is here to allow one to generate random keys and to insert and test whether the key was there in one operation. A standard idiom for generating new keys will be:: key = generate_key() while not t.insert(key, value): key=generate_key() """ class IMerge(Interface): """Object with methods for merging sets, buckets, and trees. These methods are supplied in modules that define collection classes with particular key and value types. The operations apply only to collections from the same module. For example, the IIBTree.union can only be used with IIBTree.IIBTree, IIBTree.IIBucket, IIBTree.IISet, and IIBTree.IITreeSet. The implementing module has a value type. The IOBTree and OOBTree modules have object value type. The IIBTree and OIBTree modules have integer value types. Other modules may be defined in the future that have other value types. The individual types are classified into set (Set and TreeSet) and mapping (Bucket and BTree) types. """ def difference(c1, c2): """Return the keys or items in c1 for which there is no key in c2. If c1 is None, then None is returned. If c2 is None, then c1 is returned. If neither c1 nor c2 is None, the output is a Set if c1 is a Set or TreeSet, and is a Bucket if c1 is a Bucket or BTree. """ def union(c1, c2): """Compute the Union of c1 and c2. If c1 is None, then c2 is returned, otherwise, if c2 is None, then c1 is returned. The output is a Set containing keys from the input collections. """ def intersection(c1, c2): """Compute the intersection of c1 and c2. If c1 is None, then c2 is returned, otherwise, if c2 is None, then c1 is returned. The output is a Set containing matching keys from the input collections. """ class IIMerge(IMerge): """Merge collections with integer value type. A primary intent is to support operations with no or integer values, which are used as "scores" to rate indiviual keys. That is, in this context, a BTree or Bucket is viewed as a set with scored keys, using integer scores. """ def weightedUnion(c1, c2, weight1=1, weight2=1): """Compute the weighted union of c1 and c2. If c1 and c2 are None, the output is (0, None). If c1 is None and c2 is not None, the output is (weight2, c2). If c1 is not None and c2 is None, the output is (weight1, c1). Else, and hereafter, c1 is not None and c2 is not None. If c1 and c2 are both sets, the output is 1 and the (unweighted) union of the sets. Else the output is 1 and a Bucket whose keys are the union of c1 and c2's keys, and whose values are:: v1*weight1 + v2*weight2 where: v1 is 0 if the key is not in c1 1 if the key is in c1 and c1 is a set c1[key] if the key is in c1 and c1 is a mapping v2 is 0 if the key is not in c2 1 if the key is in c2 and c2 is a set c2[key] if the key is in c2 and c2 is a mapping Note that c1 and c2 must be collections. """ def weightedIntersection(c1, c2, weight1=1, weight2=1): """Compute the weighted intersection of c1 and c2. If c1 and c2 are None, the output is (0, None). If c1 is None and c2 is not None, the output is (weight2, c2). If c1 is not None and c2 is None, the output is (weight1, c1). Else, and hereafter, c1 is not None and c2 is not None. If c1 and c2 are both sets, the output is the sum of the weights and the (unweighted) intersection of the sets. Else the output is 1 and a Bucket whose keys are the intersection of c1 and c2's keys, and whose values are:: v1*weight1 + v2*weight2 where: v1 is 1 if c1 is a set c1[key] if c1 is a mapping v2 is 1 if c2 is a set c2[key] if c2 is a mapping Note that c1 and c2 must be collections. """ class IMergeIntegerKey(IMerge): """IMerge-able objects with integer keys. Concretely, this means the types in IOBTree and IIBTree. """ def multiunion(seq): """Return union of (zero or more) integer sets, as an integer set. seq is a sequence of objects each convertible to an integer set. These objects are convertible to an integer set: + An integer, which is added to the union. + A Set or TreeSet from the same module (for example, an IIBTree.TreeSet for IIBTree.multiunion()). The elements of the set are added to the union. + A Bucket or BTree from the same module (for example, an IOBTree.IOBTree for IOBTree.multiunion()). The keys of the mapping are added to the union. The union is returned as a Set from the same module (for example, IIBTree.multiunion() returns an IIBTree.IISet). The point to this method is that it can run much faster than doing a sequence of two-input union() calls. Under the covers, all the integers in all the inputs are sorted via a single linear-time radix sort, then duplicates are removed in a second linear-time pass. """ ############################################################### # IMPORTANT NOTE # # Getting the length of a BTree, TreeSet, or output of keys, # values, or items of same is expensive. If you need to get the # length, you need to maintain this separately. # # Eventually, I need to express this through the interfaces. # ################################################################ Zope-2.8.7-final/lib/python/BTrees/OIBTree.py0000664000076400007640000000130710435502401017611 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # hack to overcome dynamic-linking headache. from _OIBTree import * Zope-2.8.7-final/lib/python/ZopeUndo/0000775000076400007640000000000010435610645016376 5ustar ajungajungZope-2.8.7-final/lib/python/ZopeUndo/__init__.py0000664000076400007640000000120210435502425020476 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## Zope-2.8.7-final/lib/python/ZopeUndo/Prefix.py0000664000076400007640000000345610435502425020211 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ZODB undo support for Zope2. This package is used to support the Prefix object that Zope uses for undo. It is a separate package only to aid configuration management. This package is included in Zope and ZODB3, so that ZODB3 is suitable for running a ZEO server that handles Zope undo. """ class Prefix: """A Prefix() is equal to any path it is a prefix of. This class can be compared to a string. The comparison will return True if all path elements of the Prefix are found at the beginning of the string being compared. Two Prefixes can not be compared. """ __no_side_effects__ = 1 def __init__(self, path): path_list = path.split('/') self.length = len(path_list) self.path = path_list def __cmp__(self, o): other_path = o.split('/') if other_path and ' ' in other_path[-1]: # don't include logged username in comparison pos = other_path[-1].rfind(' ') other_path[-1] = other_path[-1][:pos] return cmp(other_path[:self.length], self.path) def __repr__(self): # makes failing tests easier to read return "Prefix('%s')" % '/'.join(self.path) Zope-2.8.7-final/lib/python/ZopeUndo/tests/0000775000076400007640000000000010435610645017540 5ustar ajungajungZope-2.8.7-final/lib/python/ZopeUndo/tests/__init__.py0000664000076400007640000000120210435502425021640 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## Zope-2.8.7-final/lib/python/ZopeUndo/tests/testPrefix.py0000664000076400007640000000326510435502425022251 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from ZopeUndo.Prefix import Prefix import unittest class PrefixTest(unittest.TestCase): def test(self): p1 = Prefix("/a/b") for equal in ("/a/b", "/a/b/c", "/a/b/c/d"): self.assertEqual(p1, equal) for notEqual in ("", "/a/c", "/a/bbb", "///"): self.assertNotEqual(p1, notEqual) p2 = Prefix("") for equal in ("", "/", "/def", "/a/b", "/a/b/c", "/a/b/c/d"): self.assertEqual(p2, equal) def test_username_info(self): # Zope Collector 1810; user paths have username appended p1 = Prefix('/a/b') for equal in ('/a/b spam', '/a/b/c spam', '/a/b/c/b spam'): self.assertEqual(p1, equal) for notEqual in (" spam", "/a/c spam", "/a/bbb spam", "/// spam"): self.assertNotEqual(p1, notEqual) p2 = Prefix("") for equal in (" eggs", "/ eggs", "/def eggs", "/a/b eggs", "/a/b/c eggs", "/a/b/c/d eggs"): self.assertEqual(p2, equal) def test_suite(): return unittest.makeSuite(PrefixTest) Zope-2.8.7-final/lib/python/ZConfig/0000775000076400007640000000000010435610645016172 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/SETUP.cfg0000664000076400007640000000016210435500361017543 0ustar ajungajung# Metadata used by zpkg. #documentation doc/zconfig.pdf #documentation doc/schema.dtd script scripts/zconfig* Zope-2.8.7-final/lib/python/ZConfig/schema.py0000664000076400007640000005101310435500361017775 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Parser for ZConfig schemas.""" import os import xml.sax import ZConfig from ZConfig import info from ZConfig import url def parseResource(resource, loader): parser = SchemaParser(loader, resource.url) xml.sax.parse(resource.file, parser) return parser._schema def parseComponent(resource, loader, schema): parser = ComponentParser(loader, resource.url, schema) xml.sax.parse(resource.file, parser) def _srepr(ob): if isinstance(ob, type(u'')): # drop the leading "u" from a unicode repr return `ob`[1:] else: return `ob` class BaseParser(xml.sax.ContentHandler): _cdata_tags = "description", "metadefault", "example", "default" _handled_tags = ("import", "abstracttype", "sectiontype", "key", "multikey", "section", "multisection") _allowed_parents = { "description": ["key", "section", "multikey", "multisection", "sectiontype", "abstracttype", "schema", "component"], "example": ["key", "section", "multikey", "multisection"], "metadefault": ["key", "section", "multikey", "multisection"], "default": ["key", "multikey"], "import": ["schema", "component"], "abstracttype": ["schema", "component"], "sectiontype": ["schema", "component"], "key": ["schema", "sectiontype"], "multikey": ["schema", "sectiontype"], "section": ["schema", "sectiontype"], "multisection": ["schema", "sectiontype"], } def __init__(self, loader, url): self._registry = loader.registry self._loader = loader self._basic_key = self._registry.get("basic-key") self._identifier = self._registry.get("identifier") self._cdata = None self._locator = None self._prefixes = [] self._schema = None self._stack = [] self._url = url self._elem_stack = [] # SAX 2 ContentHandler methods def setDocumentLocator(self, locator): self._locator = locator def startElement(self, name, attrs): attrs = dict(attrs) if self._elem_stack: parent = self._elem_stack[-1] if not self._allowed_parents.has_key(name): self.error("Unknown tag " + name) if parent not in self._allowed_parents[name]: self.error("%s elements may not be nested in %s elements" % (_srepr(name), _srepr(parent))) elif name != self._top_level: self.error("Unknown document type " + name) self._elem_stack.append(name) if name == self._top_level: if self._schema is not None: self.error("schema element improperly nested") getattr(self, "start_" + name)(attrs) elif name in self._handled_tags: if self._schema is None: self.error(name + " element outside of schema") getattr(self, "start_" + name)(attrs) elif name in self._cdata_tags: if self._schema is None: self.error(name + " element outside of schema") if self._cdata is not None: self.error(name + " element improperly nested") self._cdata = [] self._position = None self._attrs = attrs def characters(self, data): if self._cdata is not None: if self._position is None: self._position = self.get_position() self._cdata.append(data) elif data.strip(): self.error("unexpected non-blank character data: " + `data.strip()`) def endElement(self, name): del self._elem_stack[-1] if name in self._handled_tags: getattr(self, "end_" + name)() else: data = ''.join(self._cdata).strip() self._cdata = None getattr(self, "characters_" + name)(data) def endDocument(self): if self._schema is None: self.error("no %s found" % self._top_level) # helper methods def get_position(self): if self._locator: return (self._locator.getLineNumber(), self._locator.getColumnNumber(), (self._locator.getSystemId() or self._url)) else: return None, None, self._url def get_handler(self, attrs): v = attrs.get("handler") if v is None: return v else: return self.basic_key(v) def push_prefix(self, attrs): name = attrs.get("prefix") if name: if self._prefixes: convert = self._registry.get("dotted-suffix") else: convert = self._registry.get("dotted-name") try: name = convert(name) except ValueError, err: self.error("not a valid prefix: %s (%s)" % (_srepr(name), str(err))) if name[0] == ".": prefix = self._prefixes[-1] + name else: prefix = name elif self._prefixes: prefix = self._prefixes[-1] else: prefix = '' self._prefixes.append(prefix) def pop_prefix(self): del self._prefixes[-1] def get_classname(self, name): name = str(name) if name.startswith("."): return self._prefixes[-1] + name else: return name def get_datatype(self, attrs, attrkey, default, base=None): if attrs.has_key(attrkey): dtname = self.get_classname(attrs[attrkey]) else: convert = getattr(base, attrkey, None) if convert is not None: return convert dtname = default try: return self._registry.get(dtname) except ValueError, e: self.error(e[0]) def get_sect_typeinfo(self, attrs, base=None): keytype = self.get_datatype(attrs, "keytype", "basic-key", base) valuetype = self.get_datatype(attrs, "valuetype", "string") datatype = self.get_datatype(attrs, "datatype", "null", base) return keytype, valuetype, datatype def get_required(self, attrs): if attrs.has_key("required"): v = attrs["required"] if v == "yes": return True elif v == "no": return False self.error("value for 'required' must be 'yes' or 'no'") else: return False def get_ordinality(self, attrs): # used by start_multi*() min, max = 0, info.Unbounded if self.get_required(attrs): min = 1 return min, max def get_sectiontype(self, attrs): type = attrs.get("type") if not type: self.error("section must specify type") return self._schema.gettype(type) def get_key_info(self, attrs, element): any, name, attribute = self.get_name_info(attrs, element) if any == '*': self.error(element + " may not specify '*' for name") if not name and any != '+': self.error(element + " name may not be omitted or empty") datatype = self.get_datatype(attrs, "datatype", "string") handler = self.get_handler(attrs) return name or any, datatype, handler, attribute def get_name_info(self, attrs, element, default=None): name = attrs.get("name", default) if not name: self.error(element + " name must be specified and non-empty") aname = attrs.get("attribute") if aname: aname = self.identifier(aname) if aname.startswith("getSection"): # reserved; used for SectionValue methods to get meta-info self.error("attribute names may not start with 'getSection'") if name in ("*", "+"): if not aname: self.error( "container attribute must be specified and non-empty" " when using '*' or '+' for a section name") return name, None, aname else: # run the keytype converter to make sure this is a valid key try: name = self._stack[-1].keytype(name) except ValueError, e: self.error("could not convert key name to keytype: " + str(e)) if not aname: aname = self.basic_key(name) aname = self.identifier(aname.replace('-', '_')) return None, name, aname # schema loading logic def characters_default(self, data): key = self._attrs.get("key") self._stack[-1].adddefault(data, self._position, key) def characters_description(self, data): if self._stack[-1].description is not None: self.error( "at most one may be used for each element") self._stack[-1].description = data def characters_example(self, data): self._stack[-1].example = data def characters_metadefault(self, data): self._stack[-1].metadefault = data def start_import(self, attrs): src = attrs.get("src", "").strip() pkg = attrs.get("package", "").strip() file = attrs.get("file", "").strip() if not (src or pkg): self.error("import must specify either src or package") if src and pkg: self.error("import may only specify one of src or package") if src: if file: self.error("import may not specify file and src") src = url.urljoin(self._url, src) src, fragment = url.urldefrag(src) if fragment: self.error("import src many not include" " a fragment identifier") schema = self._loader.loadURL(src) for n in schema.gettypenames(): self._schema.addtype(schema.gettype(n)) else: if os.path.dirname(file): self.error("file may not include a directory part") pkg = self.get_classname(pkg) src = self._loader.schemaComponentSource(pkg, file) if not self._schema.hasComponent(src): self._schema.addComponent(src) self.loadComponent(src) def loadComponent(self, src): r = self._loader.openResource(src) parser = ComponentParser(self._loader, src, self._schema) try: xml.sax.parse(r.file, parser) finally: r.close() def end_import(self): pass def start_sectiontype(self, attrs): name = attrs.get("name") if not name: self.error("sectiontype name must not be omitted or empty") name = self.basic_key(name) self.push_prefix(attrs) if attrs.has_key("extends"): basename = self.basic_key(attrs["extends"]) base = self._schema.gettype(basename) if base.isabstract(): self.error("sectiontype cannot extend an abstract type") keytype, valuetype, datatype = self.get_sect_typeinfo(attrs, base) sectinfo = self._schema.deriveSectionType( base, name, keytype, valuetype, datatype) else: keytype, valuetype, datatype = self.get_sect_typeinfo(attrs) sectinfo = self._schema.createSectionType( name, keytype, valuetype, datatype) if attrs.has_key("implements"): ifname = self.basic_key(attrs["implements"]) interface = self._schema.gettype(ifname) if not interface.isabstract(): self.error( "type specified by implements is not an abstracttype") interface.addsubtype(sectinfo) self._stack.append(sectinfo) def end_sectiontype(self): self.pop_prefix() self._stack.pop() def start_section(self, attrs): sectiontype = self.get_sectiontype(attrs) handler = self.get_handler(attrs) min = self.get_required(attrs) and 1 or 0 any, name, attribute = self.get_name_info(attrs, "section", "*") if any and not attribute: self.error( "attribute must be specified if section name is '*' or '+'") section = info.SectionInfo(any or name, sectiontype, min, 1, handler, attribute) self._stack[-1].addsection(name, section) self._stack.append(section) def end_section(self): self._stack.pop() def start_multisection(self, attrs): sectiontype = self.get_sectiontype(attrs) min, max = self.get_ordinality(attrs) any, name, attribute = self.get_name_info(attrs, "multisection", "*") if any not in ("*", "+"): self.error("multisection must specify '*' or '+' for the name") handler = self.get_handler(attrs) section = info.SectionInfo(any or name, sectiontype, min, max, handler, attribute) self._stack[-1].addsection(name, section) self._stack.append(section) def end_multisection(self): self._stack.pop() def start_abstracttype(self, attrs): name = attrs.get("name") if not name: self.error("abstracttype name must not be omitted or empty") name = self.basic_key(name) abstype = info.AbstractType(name) self._schema.addtype(abstype) self._stack.append(abstype) def end_abstracttype(self): self._stack.pop() def start_key(self, attrs): name, datatype, handler, attribute = self.get_key_info(attrs, "key") min = self.get_required(attrs) and 1 or 0 key = info.KeyInfo(name, datatype, min, handler, attribute) if attrs.has_key("default"): if min: self.error("required key cannot have a default value") key.adddefault(str(attrs["default"]).strip(), self.get_position()) if name != "+": key.finish() self._stack[-1].addkey(key) self._stack.append(key) def end_key(self): key = self._stack.pop() if key.name == "+": key.computedefault(self._stack[-1].keytype) key.finish() def start_multikey(self, attrs): if attrs.has_key("default"): self.error("default values for multikey must be given using" " 'default' elements") name, datatype, handler, attribute = self.get_key_info(attrs, "multikey") min, max = self.get_ordinality(attrs) key = info.MultiKeyInfo(name, datatype, min, max, handler, attribute) self._stack[-1].addkey(key) self._stack.append(key) def end_multikey(self): multikey = self._stack.pop() if multikey.name == "+": multikey.computedefault(self._stack[-1].keytype) multikey.finish() # datatype conversion wrappers def basic_key(self, s): try: return self._basic_key(s) except ValueError, e: self.error(e[0]) def identifier(self, s): try: return self._identifier(s) except ValueError, e: self.error(e[0]) # exception setup helpers def initerror(self, e): if self._locator is not None: e.colno = self._locator.getColumnNumber() e.lineno = self._locator.getLineNumber() e.url = self._locator.getSystemId() return e def error(self, message): raise self.initerror(ZConfig.SchemaError(message)) class SchemaParser(BaseParser): # needed by startElement() and endElement() _handled_tags = BaseParser._handled_tags + ("schema",) _top_level = "schema" def __init__(self, loader, url, extending_parser=None): BaseParser.__init__(self, loader, url) self._extending_parser = extending_parser self._base_keytypes = [] self._base_datatypes = [] def start_schema(self, attrs): self.push_prefix(attrs) handler = self.get_handler(attrs) keytype, valuetype, datatype = self.get_sect_typeinfo(attrs) if self._extending_parser is None: # We're not being inherited, so we need to create the schema self._schema = info.SchemaType(keytype, valuetype, datatype, handler, self._url, self._registry) else: # Parse into the extending ("subclass") parser's schema self._schema = self._extending_parser._schema self._stack = [self._schema] if attrs.has_key("extends"): sources = attrs["extends"].split() sources.reverse() for src in sources: src = url.urljoin(self._url, src) src, fragment = url.urldefrag(src) if fragment: self.error("schema extends many not include" " a fragment identifier") self.extendSchema(src) # Inherit keytype from bases, if unspecified and not conflicting if self._base_keytypes and not attrs.has_key("keytype"): keytype = self._base_keytypes[0] for kt in self._base_keytypes[1:]: if kt is not keytype: self.error("base schemas have conflicting keytypes," " but no keytype was specified in the" " extending schema") # Inherit datatype from bases, if unspecified and not conflicting if self._base_datatypes and not attrs.has_key("datatype"): datatype = self._base_datatypes[0] for dt in self._base_datatypes[1:]: if dt is not datatype: self.error("base schemas have conflicting datatypes," " but no datatype was specified in the" " extending schema") # Reset the schema types to our own, while we parse the schema body self._schema.keytype = keytype self._schema.valuetype = valuetype self._schema.datatype = datatype # Update base key/datatypes for the "extending" parser if self._extending_parser is not None: self._extending_parser._base_keytypes.append(keytype) self._extending_parser._base_datatypes.append(datatype) def extendSchema(self,src): parser = SchemaParser(self._loader, src, self) r = self._loader.openResource(src) try: xml.sax.parse(r.file, parser) finally: r.close() def end_schema(self): del self._stack[-1] assert not self._stack self.pop_prefix() assert not self._prefixes class ComponentParser(BaseParser): _handled_tags = BaseParser._handled_tags + ("component",) _top_level = "component" def __init__(self, loader, url, schema): BaseParser.__init__(self, loader, url) self._parent = schema def characters_description(self, data): if self._stack: self._stack[-1].description = data def start_key(self, attrs): self._check_not_toplevel("key") BaseParser.start_key(self, attrs) def start_multikey(self, attrs): self._check_not_toplevel("multikey") BaseParser.start_multikey(self, attrs) def start_section(self, attrs): self._check_not_toplevel("section") BaseParser.start_section(self, attrs) def start_multisection(self, attrs): self._check_not_toplevel("multisection") BaseParser.start_multisection(self, attrs) def start_component(self, attrs): self._schema = self._parent self.push_prefix(attrs) def end_component(self): self.pop_prefix() def _check_not_toplevel(self, what): if not self._stack: self.error("cannot define top-level %s in a schema %s" % (what, self._top_level)) Zope-2.8.7-final/lib/python/ZConfig/__init__.py0000664000076400007640000001014510435500361020275 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Configuration data structures and loader for the ZRS. $Id: __init__.py,v 1.18 2004/04/15 20:33:32 fdrake Exp $ """ version_info = (2, 3) __version__ = ".".join([str(n) for n in version_info]) from ZConfig.loader import loadConfig, loadConfigFile from ZConfig.loader import loadSchema, loadSchemaFile class ConfigurationError(Exception): """Base class for ZConfig exceptions.""" def __init__(self, msg, url=None): self.message = msg self.url = url Exception.__init__(self, msg) def __str__(self): return self.message class _ParseError(ConfigurationError): def __init__(self, msg, url, lineno, colno=None): self.lineno = lineno self.colno = colno ConfigurationError.__init__(self, msg, url) def __str__(self): s = self.message if self.url: s += "\n(" elif (self.lineno, self.colno) != (None, None): s += " (" if self.lineno: s += "line %d" % self.lineno if self.colno is not None: s += ", column %d" % self.colno if self.url: s += " in %s)" % self.url else: s += ")" elif self.url: s += self.url + ")" return s class SchemaError(_ParseError): """Raised when there's an error in the schema itself.""" def __init__(self, msg, url=None, lineno=None, colno=None): _ParseError.__init__(self, msg, url, lineno, colno) class SchemaResourceError(SchemaError): """Raised when there's an error locating a resource required by the schema. """ def __init__(self, msg, url=None, lineno=None, colno=None, path=None, package=None, filename=None): self.filename = filename self.package = package if path is not None: path = path[:] self.path = path SchemaError.__init__(self, msg, url, lineno, colno) def __str__(self): s = SchemaError.__str__(self) if self.package is not None: s += "\n Package name: " + repr(self.package) if self.filename is not None: s += "\n File name: " + repr(self.filename) if self.package is not None: s += "\n Package path: " + repr(self.path) return s class ConfigurationSyntaxError(_ParseError): """Raised when there's a syntax error in a configuration file.""" class DataConversionError(ConfigurationError, ValueError): """Raised when a data type conversion function raises ValueError.""" def __init__(self, exception, value, position): ConfigurationError.__init__(self, str(exception)) self.exception = exception self.value = value self.lineno, self.colno, self.url = position def __str__(self): s = "%s (line %s" % (self.message, self.lineno) if self.colno is not None: s += ", %s" % self.colno if self.url: s += ", in %s)" % self.url else: s += ")" return s class SubstitutionSyntaxError(ConfigurationError): """Raised when interpolation source text contains syntactical errors.""" class SubstitutionReplacementError(ConfigurationSyntaxError, LookupError): """Raised when no replacement is available for a reference.""" def __init__(self, source, name, url=None, lineno=None): self.source = source self.name = name ConfigurationSyntaxError.__init__( self, "no replacement for " + `name`, url, lineno) Zope-2.8.7-final/lib/python/ZConfig/substitution.py0000664000076400007640000000534010435500361021313 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Substitution support for ZConfig values.""" import ZConfig def substitute(s, mapping): """Interpolate variables from `mapping` into `s`.""" if "$" in s: result = '' rest = s while rest: p, name, namecase, rest = _split(rest) result += p if name: v = mapping.get(name) if v is None: raise ZConfig.SubstitutionReplacementError(s, namecase) result += v return result else: return s def isname(s): """Return True iff s is a valid substitution name.""" m = _name_match(s) if m: return m.group() == s else: return False def _split(s): # Return a four tuple: prefix, name, namecase, suffix # - prefix is text that can be used literally in the result (may be '') # - name is a referenced name, or None # - namecase is the name with case preserved # - suffix is trailling text that may contain additional references # (may be '' or None) if "$" in s: i = s.find("$") c = s[i+1:i+2] if c == "": raise ZConfig.SubstitutionSyntaxError( "illegal lone '$' at end of source") if c == "$": return s[:i+1], None, None, s[i+2:] prefix = s[:i] if c == "{": m = _name_match(s, i + 2) if not m: raise ZConfig.SubstitutionSyntaxError( "'${' not followed by name") name = m.group(0) i = m.end() + 1 if not s.startswith("}", i - 1): raise ZConfig.SubstitutionSyntaxError( "'${%s' not followed by '}'" % name) else: m = _name_match(s, i+1) if not m: raise ZConfig.SubstitutionSyntaxError( "'$' not followed by '$' or name") name = m.group(0) i = m.end() return prefix, name.lower(), name, s[i:] else: return s, None, None, None import re _name_match = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*").match del re Zope-2.8.7-final/lib/python/ZConfig/README.txt0000664000076400007640000000367210435500361017671 0ustar ajungajungThis is ZConfig. ZConfig is a configuration library intended for general use. It supports a hierarchical schema-driven configuration model that allows a schema to specify data conversion routines written in Python. ZConfig's model is very different from the model supported by the ConfigParser module found in Python's standard library, and is more suitable to configuration-intensive applications. ZConfig schema are written in an XML-based language and are able to "import" schema components provided by Python packages. Since components are able to bind to conversion functions provided by Python code in the package (or elsewhere), configuration objects can be arbitrarily complex, with values that have been verified against arbitrary constraints. This makes it easy for applications to separate configuration support from configuration loading even with configuration data being defined and consumed by a wide range of separate packages. ZConfig is licensed under the Zope Public License, version 2.1. See the file LICENSE.txt in the distribution for the full license text. Reference documentation is available in the doc/ directory. Information on the latest released version of the ZConfig package is available at http://www.zope.org/Members/fdrake/zconfig/ You may either create an RPM and install this, or install directly from the source distribution. Creating RPMS: python setup.py bdist_rpm If you need to force the Python interpreter to, for example, python2: python2 setup.py bdist_rpm --python=python2 Installation from the source distribution: python setup.py install To install to a user's home-dir: python setup.py install --home= To install to another prefix (eg. /usr/local) python setup.py install --prefix=/usr/local If you need to force the python interpreter to e.g. python2: python2 setup.py install For more information please refer to http://www.python.org/doc/current/inst/inst.html Zope-2.8.7-final/lib/python/ZConfig/PUBLICATION.cfg0000664000076400007640000000312610435500361020417 0ustar ajungajungMetadata-Version: 1.1 Name: ZConfig Summary: Structured Configuration Library Home-page: http://www.zope.org/Members/fdrake/zconfig/ Author: Fred L. Drake, Jr. Author-email: fred@zope.com License: ZPL 2 Description: ZConfig is a configuration library intended for general use. It supports a hierarchical schema-driven configuration model that allows a schema to specify data conversion routines written in Python. ZConfig's model is very different from the model supported by the ConfigParser module found in Python's standard library, and is more suitable to configuration-intensive applications. ZConfig schema are written in an XML-based language and are able to "import" schema components provided by Python packages. Since components are able to bind to conversion functions provided by Python code in the package (or elsewhere), configuration objects can be arbitrarily complex, with values that have been verified against arbitrary constraints. This makes it easy for applications to separate configuration support from configuration loading even with configuration data being defined and consumed by a wide range of separate packages. Platform: POSIX Platform: Windows Classifier: Intended Audience :: Developers Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: Zope Public License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: Topic :: Software Development :: Libraries :: Python Modules Zope-2.8.7-final/lib/python/ZConfig/components/0000775000076400007640000000000010435610645020357 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/components/basic/0000775000076400007640000000000010435610645021440 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/components/basic/mapping.py0000664000076400007640000000137510435500360023443 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python datatype for the ZConfig.components.basic.mapping section type.""" def mapping(section): return section.mapping Zope-2.8.7-final/lib/python/ZConfig/components/basic/__init__.py0000664000076400007640000000003410435500360023536 0ustar ajungajung# This is a Python package. Zope-2.8.7-final/lib/python/ZConfig/components/basic/tests/0000775000076400007640000000000010435610645022602 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/components/basic/tests/__init__.py0000664000076400007640000000003410435500360024700 0ustar ajungajung# This is a Python package. Zope-2.8.7-final/lib/python/ZConfig/components/basic/tests/test_mapping.py0000664000076400007640000000477310435500360025651 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the 'basic' section types provided as part of ZConfig.components.basic.""" import unittest from ZConfig.tests import support SIMPLE_SCHEMA = '''\
''' class BasicSectionTypeTestCase(support.TestBase): schema = None def setUp(self): if self.schema is None: self.__class__.schema = self.load_schema_text(SIMPLE_SCHEMA) def test_simple_empty_dict(self): conf = self.load_config_text(self.schema, "") self.assertEqual(conf.simple_dict, {}) conf = self.load_config_text(self.schema, """\ # comment """) self.assertEqual(conf.simple_dict, {}) def test_simple_dict(self): conf = self.load_config_text(self.schema, """\ key-one value-one key-two value-two """) L = conf.simple_dict.items() L.sort() self.assertEqual(L, [("key-one", "value-one"), ("key-two", "value-two")]) def test_derived_dict(self): conf = self.load_config_text(self.schema, """\ 1 foo 2 bar 42 question? """) L = conf.int_dict.items() L.sort() self.assertEqual(L, [(1, "foo"), (2, "bar"), (42, "question?")]) def test_suite(): return unittest.makeSuite(BasicSectionTypeTestCase) Zope-2.8.7-final/lib/python/ZConfig/components/basic/mapping.xml0000664000076400007640000000166110435500360023611 0ustar ajungajung Section that provides a simple mapping implementation. An application should derive a more specific section type for use in configuration files: <import package="ZConfig.components.basic" file="mapping.xml" /> <sectiontype name="mapping" extends="ZConfig.basic.mapping" /> If a non-standard keytype is needed, it can be overridden as well: <sectiontype name="system-map" extends="ZConfig.basic.mapping" keytype="mypkg.datatypes.system_name" /> Zope-2.8.7-final/lib/python/ZConfig/components/basic/component.xml0000664000076400007640000000031610435500360024154 0ustar ajungajung Convenient loader which causes all the "basic" components to be loaded. Zope-2.8.7-final/lib/python/ZConfig/components/__init__.py0000664000076400007640000000003410435500360022455 0ustar ajungajung# This is a Python package. Zope-2.8.7-final/lib/python/ZConfig/components/logger/0000775000076400007640000000000010435610645021636 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/components/logger/logger.xml0000664000076400007640000000270210435500360023630 0ustar ajungajung Indicates whether events that reach this logger should be propogated toward the root of the logger hierarchy. If true (the default), events will be passed to the logger's parent after being handled. If false, events will be handled and the parent will not be informed. There is not a way to control propogation by the severity of the event. The dotted name of the logger. This give it a location in the logging hierarchy. Most applications provide a specific set of subsystem names for which logging is meaning; consult the application documentation for the set of names that are actually interesting for the application. Zope-2.8.7-final/lib/python/ZConfig/components/logger/eventlog.xml0000664000076400007640000000073610435500360024201 0ustar ajungajung Configuration for the root logger. Zope-2.8.7-final/lib/python/ZConfig/components/logger/loghandler.py0000664000076400007640000000551010435500360024320 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Handlers which can plug into a PEP 282 logger.""" import os import sys from logging import Handler, StreamHandler from logging.handlers import SysLogHandler, BufferingHandler from logging.handlers import HTTPHandler, SMTPHandler from logging.handlers import NTEventLogHandler as Win32EventLogHandler class FileHandler(StreamHandler): """File handler which supports reopening of logs. Re-opening should be used instead of the 'rollover' feature of the FileHandler from the standard library's logging package. """ def __init__(self, filename, mode="a"): filename = os.path.abspath(filename) StreamHandler.__init__(self, open(filename, mode)) self.baseFilename = filename self.mode = mode def close(self): self.stream.close() def reopen(self): self.close() self.stream = open(self.baseFilename, self.mode) class Win32FileHandler(FileHandler): """File-based log handler for Windows that supports an additional 'rotate' method. reopen() is generally useless since Windows cannot do a move on an open file. """ def rotate(self, rotateFilename=None): if not rotateFilename: rotateFilename = self.baseFilename + ".last" error = None self.close() try: os.rename(self.baseFilename, rotateFilename) except OSError: pass self.stream = open(self.baseFilename, self.mode) if os.name == "nt": # Make it the default for Windows - we install a 'reopen' handler that # tries to rotate the logfile. FileHandler = Win32FileHandler class NullHandler(Handler): """Handler that does nothing.""" def emit(self, record): pass def handle(self, record): pass class StartupHandler(BufferingHandler): """Handler which stores messages in a buffer until later. This is useful at startup before we can know that we can safely write to a configuration-specified handler. """ def __init__(self): BufferingHandler.__init__(self, sys.maxint) def shouldFlush(self, record): return False def flushBufferTo(self, target): while self.buffer: target.handle(self.buffer.pop(0)) Zope-2.8.7-final/lib/python/ZConfig/components/logger/__init__.py0000664000076400007640000000161710435500360023744 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZConfig schema component package for logging configuration.""" # Make sure we can't import this if "logging" isn't available; we # don't want partial imports to appear to succeed. try: import logging except ImportError: import sys del sys.modules[__name__] Zope-2.8.7-final/lib/python/ZConfig/components/logger/base-logger.xml0000664000076400007640000000301710435500360024540 0ustar ajungajung Base definition for the logger types defined by ZConfig.components.logger. This exists entirely to provide shared key definitions and documentation. Verbosity setting for the logger. Values must be a name of a level, or an integer in the range [0..50]. The names of the levels, in order of increasing verbosity (names on the same line are equivalent): critical, fatal error warn, warning info blather debug trace all The special name "notset", or the numeric value 0, indicates that the setting for the parent logger should be used. It is strongly recommended that names be used rather than numeric values to ensure that configuration files can be deciphered more easily. Handlers to install on this logger. Each handler describes how logging events should be presented. Zope-2.8.7-final/lib/python/ZConfig/components/logger/logger.py0000664000076400007640000000666710435500360023476 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZConfig factory datatypes for loggers.""" from ZConfig.components.logger.factory import Factory class LoggerFactoryBase(Factory): """Base class for logger factories. Factory used to create loggers while delaying actual logger instance construction. We need to do this because we may want to reference a logger before actually instantiating it (for example, to allow the app time to set an effective user). An instance of this wrapper is a callable which, when called, returns a logger object. """ def __init__(self, section): Factory.__init__(self) self.level = section.level self.handler_factories = section.handlers def create(self): # set the logger up import logging logger = logging.getLogger(self.name) logger.setLevel(self.level) if self.handler_factories: for handler_factory in self.handler_factories: handler = handler_factory() logger.addHandler(handler) else: from ZConfig.components.logger import loghandler logger.addHandler(loghandler.NullHandler()) return logger def startup(self): # make sure we've instantiated the logger self() def getLowestHandlerLevel(self): """Return the lowest log level provided by any configured handler. If all handlers and the logger itself have level==NOTSET, this returns NOTSET. """ import logging lowest = self.level for factory in self.handler_factories: level = factory.getLevel() if level != logging.NOTSET: if lowest == logging.NOTSET: lowest = level else: lowest = min(lowest, level) return lowest def reopen(self): """Re-open any handlers for which this is a meaningful operation. This only works on handlers on the logger provided by this factory directly; handlers for child loggers are not affected. (This can be considered a bug, but is sufficient at the moment.) """ logger = self() for handler in logger.handlers: reopen = getattr(handler, "reopen", None) if reopen is not None and callable(reopen): reopen() class EventLogFactory(LoggerFactoryBase): """Logger factory that returns the root logger.""" name = None class LoggerFactory(LoggerFactoryBase): """Logger factory that returns the named logger.""" def __init__(self, section): LoggerFactoryBase.__init__(self, section) self.name = section.name self.propagate = section.propagate def create(self): logger = LoggerFactoryBase.create(self) logger.propagate = self.propagate return logger Zope-2.8.7-final/lib/python/ZConfig/components/logger/handlers.py0000664000076400007640000001251010435500360023777 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZConfig factory datatypes for log handlers.""" import sys from ZConfig.components.logger.factory import Factory _log_format_variables = { 'name': '', 'levelno': '3', 'levelname': 'DEBUG', 'pathname': 'apath', 'filename': 'afile', 'module': 'amodule', 'lineno': 1, 'created': 1.1, 'asctime': 'atime', 'msecs': 1, 'relativeCreated': 1, 'thread': 1, 'message': 'amessage', 'process': 1, } def log_format(value): value = ctrl_char_insert(value) try: # Make sure the format string uses only names that will be # provided, and has reasonable type flags for each, and does # not expect positional args. value % _log_format_variables except (ValueError, KeyError): raise ValueError, 'Invalid log format string %s' % value return value _control_char_rewrites = {r'\n': '\n', r'\t': '\t', r'\b': '\b', r'\f': '\f', r'\r': '\r'}.items() def ctrl_char_insert(value): for pattern, replacement in _control_char_rewrites: value = value.replace(pattern, replacement) return value class HandlerFactory(Factory): def __init__(self, section): Factory.__init__(self) self.section = section def create_loghandler(self): raise NotImplementedError( "subclasses must override create_loghandler()") def create(self): import logging logger = self.create_loghandler() logger.setFormatter(logging.Formatter(self.section.format, self.section.dateformat)) logger.setLevel(self.section.level) return logger def getLevel(self): return self.section.level class FileHandlerFactory(HandlerFactory): def create_loghandler(self): from ZConfig.components.logger import loghandler path = self.section.path if path == "STDERR": handler = loghandler.StreamHandler(sys.stderr) elif path == "STDOUT": handler = loghandler.StreamHandler(sys.stdout) else: handler = loghandler.FileHandler(path) return handler _syslog_facilities = { "auth": 1, "authpriv": 1, "cron": 1, "daemon": 1, "kern": 1, "lpr": 1, "mail": 1, "news": 1, "security": 1, "syslog": 1, "user": 1, "uucp": 1, "local0": 1, "local1": 1, "local2": 1, "local3": 1, "local4": 1, "local5": 1, "local6": 1, "local7": 1, } def syslog_facility(value): value = value.lower() if not _syslog_facilities.has_key(value): L = _syslog_facilities.keys() L.sort() raise ValueError("Syslog facility must be one of " + ", ".join(L)) return value class SyslogHandlerFactory(HandlerFactory): def create_loghandler(self): from ZConfig.components.logger import loghandler return loghandler.SysLogHandler(self.section.address.address, self.section.facility) class Win32EventLogFactory(HandlerFactory): def create_loghandler(self): from ZConfig.components.logger import loghandler return loghandler.Win32EventLogHandler(self.section.appname) def http_handler_url(value): import urlparse scheme, netloc, path, param, query, fragment = urlparse.urlparse(value) if scheme != 'http': raise ValueError, 'url must be an http url' if not netloc: raise ValueError, 'url must specify a location' if not path: raise ValueError, 'url must specify a path' q = [] if param: q.append(';') q.append(param) if query: q.append('?') q.append(query) if fragment: q.append('#') q.append(fragment) return (netloc, path + ''.join(q)) def get_or_post(value): value = value.upper() if value not in ('GET', 'POST'): raise ValueError('method must be "GET" or "POST", instead received: ' + repr(value)) return value class HTTPHandlerFactory(HandlerFactory): def create_loghandler(self): from ZConfig.components.logger import loghandler host, selector = self.section.url return loghandler.HTTPHandler(host, selector, self.section.method) class SMTPHandlerFactory(HandlerFactory): def create_loghandler(self): from ZConfig.components.logger import loghandler host, port = self.section.smtp_server if not port: mailhost = host else: mailhost = host, port return loghandler.SMTPHandler(mailhost, self.section.fromaddr, self.section.toaddrs, self.section.subject) Zope-2.8.7-final/lib/python/ZConfig/components/logger/factory.py0000664000076400007640000000244110435500360023650 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## _marker = object() class Factory: """Generic wrapper for instance construction. Calling the factory causes the instance to be created if it hasn't already been created, and returns the object. Calling the factory multiple times returns the same object. The instance is created using the factory's create() method, which must be overriden by subclasses. """ def __init__(self): self.instance = _marker def __call__(self): if self.instance is _marker: self.instance = self.create() return self.instance def create(self): raise NotImplementedError("subclasses need to override create()") Zope-2.8.7-final/lib/python/ZConfig/components/logger/datatypes.py0000664000076400007640000000220510435500360024175 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZConfig datatypes for logging support.""" _logging_levels = { "critical": 50, "fatal": 50, "error": 40, "warn": 30, "warning": 30, "info": 20, "blather": 15, "debug": 10, "trace": 5, "all": 1, "notset": 0, } def logging_level(value): s = str(value).lower() if _logging_levels.has_key(s): return _logging_levels[s] else: v = int(s) if v < 0 or v > 50: raise ValueError("log level not in range: " + `v`) return v Zope-2.8.7-final/lib/python/ZConfig/components/logger/abstract.xml0000664000076400007640000000022710435500360024154 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/components/logger/handlers.xml0000664000076400007640000000546510435500360024162 0ustar ajungajung Base type for most log handlers. This is cannot be used as a loghandler directly since it doesn't implement the loghandler abstract section type. Zope-2.8.7-final/lib/python/ZConfig/components/logger/tests/0000775000076400007640000000000010435610645023000 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/components/logger/tests/__init__.py0000664000076400007640000000003410435500360025076 0ustar ajungajung# This is a Python package. Zope-2.8.7-final/lib/python/ZConfig/components/logger/tests/test_logger.py0000664000076400007640000002237210435500360025666 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for logging configuration via ZConfig.""" import cStringIO as StringIO import logging import sys import tempfile import unittest import ZConfig from ZConfig.components.logger import datatypes from ZConfig.components.logger import handlers from ZConfig.components.logger import loghandler class LoggingTestBase(unittest.TestCase): # XXX This tries to save and restore the state of logging around # the test. Somewhat surgical; there may be a better way. name = None def setUp(self): self._old_logger = logging.getLogger(self.name) self._old_level = self._old_logger.level self._old_handlers = self._old_logger.handlers[:] self._old_logger.handlers[:] = [] self._old_logger.setLevel(logging.WARN) def tearDown(self): for h in self._old_logger.handlers: self._old_logger.removeHandler(h) for h in self._old_handlers: self._old_logger.addHandler(h) self._old_logger.setLevel(self._old_level) _schema = None def get_schema(self): if self._schema is None: sio = StringIO.StringIO(self._schematext) self.__class__._schema = ZConfig.loadSchemaFile(sio) return self._schema def get_config(self, text): conf, handler = ZConfig.loadConfigFile(self.get_schema(), StringIO.StringIO(text)) self.assert_(not handler) return conf class TestConfig(LoggingTestBase): _schematext = """
""" def test_logging_level(self): # Make sure the expected names are supported; it's not clear # how to check the values in a meaningful way. # Just make sure they're case-insensitive. convert = datatypes.logging_level for name in ["notset", "all", "trace", "debug", "blather", "info", "warn", "warning", "error", "fatal", "critical"]: self.assertEqual(convert(name), convert(name.upper())) self.assertRaises(ValueError, convert, "hopefully-not-a-valid-value") def test_http_method(self): convert = handlers.get_or_post self.assertEqual(convert("get"), "GET") self.assertEqual(convert("GET"), "GET") self.assertEqual(convert("post"), "POST") self.assertEqual(convert("POST"), "POST") self.assertRaises(ValueError, convert, "") self.assertRaises(ValueError, convert, "foo") def test_syslog_facility(self): convert = handlers.syslog_facility for name in ["auth", "authpriv", "cron", "daemon", "kern", "lpr", "mail", "news", "security", "syslog", "user", "uucp", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7"]: self.assertEqual(convert(name), name) self.assertEqual(convert(name.upper()), name) self.assertRaises(ValueError, convert, "hopefully-never-a-valid-value") def test_config_without_logger(self): conf = self.get_config("") self.assert_(conf.eventlog is None) def test_config_without_handlers(self): logger = self.check_simple_logger("") # Make sure there's a NullHandler, since a warning gets # printed if there are no handlers: self.assertEqual(len(logger.handlers), 1) self.assert_(isinstance(logger.handlers[0], loghandler.NullHandler)) def test_with_logfile(self): import os fn = tempfile.mktemp() logger = self.check_simple_logger("\n" " \n" " path %s\n" " level debug\n" " \n" "" % fn) logfile = logger.handlers[0] self.assertEqual(logfile.level, logging.DEBUG) self.assert_(isinstance(logfile, loghandler.FileHandler)) logfile.close() os.remove(fn) def test_with_stderr(self): self.check_standard_stream("stderr") def test_with_stdout(self): self.check_standard_stream("stdout") def check_standard_stream(self, name): old_stream = getattr(sys, name) conf = self.get_config(""" level info path %s """ % name.upper()) self.assert_(conf.eventlog is not None) # The factory has already been created; make sure it picks up # the stderr we set here when we create the logger and # handlers: sio = StringIO.StringIO() setattr(sys, name, sio) try: logger = conf.eventlog() finally: setattr(sys, name, old_stream) logger.warn("woohoo!") self.assert_(sio.getvalue().find("woohoo!") >= 0) def test_with_syslog(self): logger = self.check_simple_logger("\n" " \n" " level error\n" " facility local3\n" " \n" "") syslog = logger.handlers[0] self.assertEqual(syslog.level, logging.ERROR) self.assert_(isinstance(syslog, loghandler.SysLogHandler)) def test_with_http_logger_localhost(self): logger = self.check_simple_logger("\n" " \n" " level error\n" " method post\n" " \n" "") handler = logger.handlers[0] self.assertEqual(handler.host, "localhost") # XXX The "url" attribute of the handler is misnamed; it # really means just the selector portion of the URL. self.assertEqual(handler.url, "/") self.assertEqual(handler.level, logging.ERROR) self.assertEqual(handler.method, "POST") self.assert_(isinstance(handler, loghandler.HTTPHandler)) def test_with_http_logger_remote_host(self): logger = self.check_simple_logger("\n" " \n" " method get\n" " url http://example.com/log/\n" " \n" "") handler = logger.handlers[0] self.assertEqual(handler.host, "example.com") # XXX The "url" attribute of the handler is misnamed; it # really means just the selector portion of the URL. self.assertEqual(handler.url, "/log/") self.assertEqual(handler.level, logging.NOTSET) self.assertEqual(handler.method, "GET") self.assert_(isinstance(handler, loghandler.HTTPHandler)) def test_with_email_notifier(self): logger = self.check_simple_logger("\n" " \n" " to sysadmin@example.com\n" " to sa-pager@example.com\n" " from zlog-user@example.com\n" " level fatal\n" " \n" "") handler = logger.handlers[0] self.assertEqual(handler.toaddrs, ["sysadmin@example.com", "sa-pager@example.com"]) self.assertEqual(handler.fromaddr, "zlog-user@example.com") self.assertEqual(handler.level, logging.FATAL) def check_simple_logger(self, text, level=logging.INFO): conf = self.get_config(text) self.assert_(conf.eventlog is not None) self.assertEqual(conf.eventlog.level, level) logger = conf.eventlog() self.assert_(isinstance(logger, logging.Logger)) self.assertEqual(len(logger.handlers), 1) return logger def test_suite(): return unittest.makeSuite(TestConfig) if __name__ == '__main__': unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZConfig/components/logger/component.xml0000664000076400007640000000056710435500360024362 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/datatypes.py0000664000076400007640000003203310435500361020534 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Selection of standard datatypes for ZConfig.""" import os import re import sys import datetime try: unicode except NameError: have_unicode = False else: have_unicode = True class MemoizedConversion: """Conversion helper that caches the results of expensive conversions.""" def __init__(self, conversion): self._memo = {} self._conversion = conversion def __call__(self, value): try: return self._memo[value] except KeyError: v = self._conversion(value) self._memo[value] = v return v class RangeCheckedConversion: """Conversion helper that range checks another conversion.""" def __init__(self, conversion, min=None, max=None): self._min = min self._max = max self._conversion = conversion def __call__(self, value): v = self._conversion(value) if self._min is not None and v < self._min: raise ValueError("%s is below lower bound (%s)" % (`v`, `self._min`)) if self._max is not None and v > self._max: raise ValueError("%s is above upper bound (%s)" % (`v`, `self._max`)) return v class RegularExpressionConversion: reason = "value did not match regular expression" def __init__(self, regex): self._rx = re.compile(regex) def __call__(self, value): m = self._rx.match(value) if m and m.group() == value: return value else: raise ValueError("%s: %s" % (self.reason, repr(value))) def check_locale(value): import locale prev = locale.setlocale(locale.LC_ALL) try: try: locale.setlocale(locale.LC_ALL, value) finally: locale.setlocale(locale.LC_ALL, prev) except locale.Error: raise ValueError( 'The specified locale "%s" is not supported by your system.\n' 'See your operating system documentation for more\n' 'information on locale support.' % value) else: return value class BasicKeyConversion(RegularExpressionConversion): def __init__(self): RegularExpressionConversion.__init__(self, "[a-zA-Z][-._a-zA-Z0-9]*") def __call__(self, value): value = str(value) return RegularExpressionConversion.__call__(self, value).lower() class ASCIIConversion(RegularExpressionConversion): def __call__(self, value): value = RegularExpressionConversion.__call__(self, value) if have_unicode and isinstance(value, unicode): value = value.encode("ascii") return value _ident_re = "[_a-zA-Z][_a-zA-Z0-9]*" class IdentifierConversion(ASCIIConversion): reason = "not a valid Python identifier" def __init__(self): ASCIIConversion.__init__(self, _ident_re) class DottedNameConversion(ASCIIConversion): reason = "not a valid dotted name" def __init__(self): ASCIIConversion.__init__(self, r"%s(?:\.%s)*" % (_ident_re, _ident_re)) class DottedNameSuffixConversion(ASCIIConversion): reason = "not a valid dotted name or suffix" def __init__(self): ASCIIConversion.__init__(self, r"(?:%s)(?:\.%s)*|(?:\.%s)+" % (_ident_re, _ident_re, _ident_re)) def integer(value): try: return int(value) except ValueError: return long(value) except OverflowError: return long(value) def null_conversion(value): return value def asBoolean(s): """Convert a string value to a boolean value.""" ss = str(s).lower() if ss in ('yes', 'true', 'on'): return True elif ss in ('no', 'false', 'off'): return False else: raise ValueError("not a valid boolean value: " + repr(s)) def string_list(s): """Convert a string to a list of strings using .split().""" return s.split() port_number = RangeCheckedConversion(integer, min=1, max=0xffff).__call__ class InetAddress: def __init__(self, default_host): self.DEFAULT_HOST = default_host def __call__(self, s): # returns (host, port) tuple host = '' port = None if ":" in s: host, s = s.split(":", 1) if s: port = port_number(s) host = host.lower() else: try: port = port_number(s) except ValueError: if len(s.split()) != 1: raise ValueError("not a valid host name: " + repr(s)) host = s.lower() if not host: host = self.DEFAULT_HOST return host, port if sys.platform[:3] == "win": DEFAULT_HOST = "localhost" else: DEFAULT_HOST = "" inet_address = InetAddress(DEFAULT_HOST) inet_connection_address = InetAddress("127.0.0.1") inet_binding_address = InetAddress("") class SocketAddress: def __init__(self, s): import socket if "/" in s or s.find(os.sep) >= 0: self.family = getattr(socket, "AF_UNIX", None) self.address = s else: self.family = socket.AF_INET self.address = self._parse_address(s) def _parse_address(self, s): return inet_address(s) class SocketBindingAddress(SocketAddress): def _parse_address(self, s): return inet_binding_address(s) class SocketConnectionAddress(SocketAddress): def _parse_address(self, s): return inet_connection_address(s) def float_conversion(v): if isinstance(v, basestring): if v.lower() in ["inf", "-inf", "nan"]: raise ValueError(`v` + " is not a portable float representation") return float(v) class IpaddrOrHostname(RegularExpressionConversion): def __init__(self): # IP address regex from the Perl Cookbook, Recipe 6.23 (revised ed.) # We allow underscores in hostnames although this is considered # illegal according to RFC1034. expr = (r"(^(\d|[01]?\d\d|2[0-4]\d|25[0-5])\." #ipaddr r"(\d|[01]?\d\d|2[0-4]\d|25[0-5])\." #ipaddr cont'd r"(\d|[01]?\d\d|2[0-4]\d|25[0-5])\." #ipaddr cont'd r"(\d|[01]?\d\d|2[0-4]\d|25[0-5])$)" #ipaddr cont'd r"|([A-Za-z_][-A-Za-z0-9_.]*[-A-Za-z0-9_])") # or hostname RegularExpressionConversion.__init__(self, expr) def __call__(self, value): return RegularExpressionConversion.__call__(self, value).lower() def existing_directory(v): nv = os.path.expanduser(v) if os.path.isdir(nv): return nv raise ValueError, '%s is not an existing directory' % v def existing_path(v): nv = os.path.expanduser(v) if os.path.exists(nv): return nv raise ValueError, '%s is not an existing path' % v def existing_file(v): nv = os.path.expanduser(v) if os.path.exists(nv): return nv raise ValueError, '%s is not an existing file' % v def existing_dirpath(v): nv = os.path.expanduser(v) dir = os.path.dirname(nv) if not dir: # relative pathname with no directory component return nv if os.path.isdir(dir): return nv raise ValueError, ('The directory named as part of the path %s ' 'does not exist.' % v) class SuffixMultiplier: # d is a dictionary of suffixes to integer multipliers. If no suffixes # match, default is the multiplier. Matches are case insensitive. Return # values are in the fundamental unit. def __init__(self, d, default=1): self._d = d self._default = default # all keys must be the same size self._keysz = None for k in d.keys(): if self._keysz is None: self._keysz = len(k) else: assert self._keysz == len(k) def __call__(self, v): v = v.lower() for s, m in self._d.items(): if v[-self._keysz:] == s: return int(v[:-self._keysz]) * m return int(v) * self._default def timedelta(s): # Unlike the standard time-interval data type, which returns a float # number of seconds, this datatype takes a wider range of syntax and # returns a datetime.timedelta # # Accepts suffixes: # w - weeks # d - days # h - hours # m - minutes # s - seconds # # and all arguments may be integers or floats, positive or negative. # More than one time interval suffix value may appear on the line, but # they should all be separated by spaces, e.g.: # # sleep_time 4w 2d 7h 12m 0.00001s weeks = days = hours = minutes = seconds = 0 for part in s.split(): val = float(part[:-1]) suffix = part[-1] if suffix == 'w': weeks = val elif suffix == 'd': days = val elif suffix == 'h': hours = val elif suffix == 'm': minutes = val elif suffix == 's': seconds = val else: raise TypeError('bad part %s in %s' % (part, s)) return datetime.timedelta(weeks=weeks, days=days, hours=hours, minutes=minutes, seconds=seconds) stock_datatypes = { "boolean": asBoolean, "dotted-name": DottedNameConversion(), "dotted-suffix": DottedNameSuffixConversion(), "identifier": IdentifierConversion(), "integer": integer, "float": float_conversion, "string": str, "string-list": string_list, "null": null_conversion, "locale": MemoizedConversion(check_locale), "port-number": port_number, "basic-key": BasicKeyConversion(), "inet-address": inet_address, "inet-binding-address": inet_binding_address, "inet-connection-address": inet_connection_address, "socket-address": SocketAddress, "socket-binding-address": SocketBindingAddress, "socket-connection-address": SocketConnectionAddress, "ipaddr-or-hostname":IpaddrOrHostname(), "existing-directory":existing_directory, "existing-path": existing_path, "existing-file": existing_file, "existing-dirpath": existing_dirpath, "byte-size": SuffixMultiplier({'kb': 1024, 'mb': 1024*1024, 'gb': 1024*1024*1024L, }), "time-interval": SuffixMultiplier({'s': 1, 'm': 60, 'h': 60*60, 'd': 60*60*24, }), "timedelta": timedelta, } class Registry: def __init__(self, stock=None): if stock is None: stock = stock_datatypes.copy() self._stock = stock self._other = {} self._basic_key = None def get(self, name): if '.' not in name: if self._basic_key is None: self._basic_key = self._other.get("basic-key") if self._basic_key is None: self._basic_key = self._stock.get("basic-key") if self._basic_key is None: self._basic_key = stock_datatypes["basic-key"] name = self._basic_key(name) t = self._stock.get(name) if t is None: t = self._other.get(name) if t is None: t = self.search(name) return t def register(self, name, conversion): if self._stock.has_key(name): raise ValueError("datatype name conflicts with built-in type: " + `name`) if self._other.has_key(name): raise ValueError("datatype name already registered: " + `name`) self._other[name] = conversion def search(self, name): if not "." in name: raise ValueError("unloadable datatype name: " + `name`) components = name.split('.') start = components[0] g = {} package = __import__(start, g, g) modulenames = [start] for component in components[1:]: modulenames.append(component) try: package = getattr(package, component) except AttributeError: n = '.'.join(modulenames) package = __import__(n, g, g, component) self._other[name] = package return package Zope-2.8.7-final/lib/python/ZConfig/PACKAGE.cfg0000664000076400007640000000103210435500361017673 0ustar ajungajung# Load the license from an external source, so we don't have to keep a # copy of it sitting around: LICENSE.txt http://cvs.zope.org/Zope3/ZopePublicLicense.txt?rev=HEAD # Add a few things to the distribution root. doc LICENSE.txt NEWS.txt README.txt # Specify what is included in the component. # Python modules from the package: *.py # Child packages: components tests # Other files and directories needed when distutils runs: scripts Zope-2.8.7-final/lib/python/ZConfig/info.py0000664000076400007640000004146410435500361017501 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Objects that can describe a ZConfig schema.""" import copy import ZConfig class UnboundedThing: __metaclass__ = type __slots__ = () def __lt__(self, other): return False def __le__(self, other): return isinstance(other, self.__class__) def __gt__(self, other): return True def __ge__(self, other): return True def __eq__(self, other): return isinstance(other, self.__class__) def __ne__(self, other): return not isinstance(other, self.__class__) def __repr__(self): return "" Unbounded = UnboundedThing() class ValueInfo: __metaclass__ = type __slots__ = 'value', 'position' def __init__(self, value, position): self.value = value # position is (lineno, colno, url) self.position = position def convert(self, datatype): try: return datatype(self.value) except ValueError, e: raise ZConfig.DataConversionError(e, self.value, self.position) class BaseInfo: """Information about a single configuration key.""" description = None example = None metadefault = None def __init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute): if maxOccurs is not None and maxOccurs < 1: if maxOccurs < 1: raise ZConfig.SchemaError( "maxOccurs must be at least 1") if minOccurs is not None and minOccurs < maxOccurs: raise ZConfig.SchemaError( "minOccurs must be at least maxOccurs") self.name = name self.datatype = datatype self.minOccurs = minOccurs self.maxOccurs = maxOccurs self.handler = handler self.attribute = attribute def __repr__(self): clsname = self.__class__.__name__ return "<%s for %s>" % (clsname, `self.name`) def isabstract(self): return False def ismulti(self): return self.maxOccurs > 1 def issection(self): return False class BaseKeyInfo(BaseInfo): _rawdefaults = None def __init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute): assert minOccurs is not None BaseInfo.__init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute) self._finished = False def finish(self): if self._finished: raise ZConfig.SchemaError( "cannot finish KeyInfo more than once") self._finished = True def adddefault(self, value, position, key=None): if self._finished: raise ZConfig.SchemaError( "cannot add default values to finished KeyInfo") # Check that the name/keyed relationship is right: if self.name == "+" and key is None: raise ZConfig.SchemaError( "default values must be keyed for name='+'") elif self.name != "+" and key is not None: raise ZConfig.SchemaError( "unexpected key for default value") self.add_valueinfo(ValueInfo(value, position), key) def add_valueinfo(self, vi, key): """Actually add a ValueInfo to this key-info object. The appropriate value of None-ness of key has already been checked with regard to the name of the key, and has been found permissible to add. This method is a requirement for subclasses, and should not be called by client code. """ raise NotImplementedError( "add_valueinfo() must be implemented by subclasses of BaseKeyInfo") def prepare_raw_defaults(self): assert self.name == "+" if self._rawdefaults is None: self._rawdefaults = self._default self._default = {} class KeyInfo(BaseKeyInfo): _default = None def __init__(self, name, datatype, minOccurs, handler, attribute): BaseKeyInfo.__init__(self, name, datatype, minOccurs, 1, handler, attribute) if self.name == "+": self._default = {} def add_valueinfo(self, vi, key): if self.name == "+": if self._default.has_key(key): # not ideal: we're presenting the unconverted # version of the key raise ZConfig.SchemaError( "duplicate default value for key %s" % `key`) self._default[key] = vi elif self._default is not None: raise ZConfig.SchemaError( "cannot set more than one default to key with maxOccurs == 1") else: self._default = vi def computedefault(self, keytype): self.prepare_raw_defaults() for k, vi in self._rawdefaults.iteritems(): key = ValueInfo(k, vi.position).convert(keytype) self.add_valueinfo(vi, key) def getdefault(self): # Use copy.copy() to make sure we don't allow polution of # our internal data without having to worry about both the # list and dictionary cases: return copy.copy(self._default) class MultiKeyInfo(BaseKeyInfo): def __init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute): BaseKeyInfo.__init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute) if self.name == "+": self._default = {} else: self._default = [] def add_valueinfo(self, vi, key): if self.name == "+": # This is a keyed value, not a simple value: if key in self._default: self._default[key].append(vi) else: self._default[key] = [vi] else: self._default.append(vi) def computedefault(self, keytype): self.prepare_raw_defaults() for k, vlist in self._rawdefaults.iteritems(): key = ValueInfo(k, vlist[0].position).convert(keytype) for vi in vlist: self.add_valueinfo(vi, key) def getdefault(self): return copy.copy(self._default) class SectionInfo(BaseInfo): def __init__(self, name, sectiontype, minOccurs, maxOccurs, handler, attribute): # name - name of the section; one of '*', '+', or name1 # sectiontype - SectionType instance # minOccurs - minimum number of occurances of the section # maxOccurs - maximum number of occurances; if > 1, name # must be '*' or '+' # handler - handler name called when value(s) must take effect, # or None # attribute - name of the attribute on the SectionValue object if maxOccurs > 1: if name not in ('*', '+'): raise ZConfig.SchemaError( "sections which can occur more than once must" " use a name of '*' or '+'") if not attribute: raise ZConfig.SchemaError( "sections which can occur more than once must" " specify a target attribute name") if sectiontype.isabstract(): datatype = None else: datatype = sectiontype.datatype BaseInfo.__init__(self, name, datatype, minOccurs, maxOccurs, handler, attribute) self.sectiontype = sectiontype def __repr__(self): clsname = self.__class__.__name__ return "<%s for %s (%s)>" % ( clsname, self.sectiontype.name, `self.name`) def issection(self): return True def allowUnnamed(self): return self.name == "*" def isAllowedName(self, name): if name == "*" or name == "+": return False elif self.name == "+": return name and True or False elif self.name == "*": return True else: return name == self.name def getdefault(self): # sections cannot have defaults if self.maxOccurs > 1: return [] else: return None class AbstractType: __metaclass__ = type __slots__ = '_subtypes', 'name', 'description' def __init__(self, name): self._subtypes = {} self.name = name self.description = None def addsubtype(self, type): self._subtypes[type.name] = type def getsubtype(self, name): try: return self._subtypes[name] except KeyError: raise ZConfig.SchemaError("no sectiontype %s in abstracttype %s" % (`name`, `self.name`)) def hassubtype(self, name): """Return true iff this type has 'name' as a concrete manifestation.""" return name in self._subtypes.keys() def getsubtypenames(self): """Return the names of all concrete types as a sorted list.""" L = self._subtypes.keys() L.sort() return L def isabstract(self): return True class SectionType: def __init__(self, name, keytype, valuetype, datatype, registry, types): # name - name of the section, or '*' or '+' # datatype - type for the section itself # keytype - type for the keys themselves # valuetype - default type for key values self.name = name self.datatype = datatype self.keytype = keytype self.valuetype = valuetype self.handler = None self.description = None self.registry = registry self._children = [] # [(key, info), ...] self._attrmap = {} # {attribute: info, ...} self._keymap = {} # {key: info, ...} self._types = types def gettype(self, name): n = name.lower() try: return self._types[n] except KeyError: raise ZConfig.SchemaError("unknown type name: " + `name`) def gettypenames(self): return self._types.keys() def __len__(self): return len(self._children) def __getitem__(self, index): return self._children[index] def _add_child(self, key, info): # check naming constraints assert key or info.attribute if key and self._keymap.has_key(key): raise ZConfig.SchemaError( "child name %s already used" % key) if info.attribute and self._attrmap.has_key(info.attribute): raise ZConfig.SchemaError( "child attribute name %s already used" % info.attribute) # a-ok, add the item to the appropriate maps if info.attribute: self._attrmap[info.attribute] = info if key: self._keymap[key] = info self._children.append((key, info)) def addkey(self, keyinfo): self._add_child(keyinfo.name, keyinfo) def addsection(self, name, sectinfo): assert name not in ("*", "+") self._add_child(name, sectinfo) def getinfo(self, key): if not key: raise ZConfig.ConfigurationError( "cannot match a key without a name") try: return self._keymap[key] except KeyError: raise ZConfig.ConfigurationError("no key matching " + `key`) def getrequiredtypes(self): d = {} if self.name: d[self.name] = 1 stack = [self] while stack: info = stack.pop() for key, ci in info._children: if ci.issection(): t = ci.sectiontype if not d.has_key(t.name): d[t.name] = 1 stack.append(t) return d.keys() def getsectioninfo(self, type, name): for key, info in self._children: if key: if key == name: if not info.issection(): raise ZConfig.ConfigurationError( "section name %s already in use for key" % key) st = info.sectiontype if st.isabstract(): try: st = st.getsubtype(type) except ZConfig.ConfigurationError: raise ZConfig.ConfigurationError( "section type %s not allowed for name %s" % (`type`, `key`)) if not st.name == type: raise ZConfig.ConfigurationError( "name %s must be used for a %s section" % (`name`, `st.name`)) return info # else must be a sectiontype or an abstracttype: elif info.sectiontype.name == type: if not (name or info.allowUnnamed()): raise ZConfig.ConfigurationError( `type` + " sections must be named") return info elif info.sectiontype.isabstract(): st = info.sectiontype if st.name == type: raise ZConfig.ConfigurationError( "cannot define section with an abstract type") try: st = st.getsubtype(type) except ZConfig.ConfigurationError: # not this one; maybe a different one pass else: return info raise ZConfig.ConfigurationError( "no matching section defined for type='%s', name='%s'" % ( type, name)) def isabstract(self): return False class SchemaType(SectionType): def __init__(self, keytype, valuetype, datatype, handler, url, registry): SectionType.__init__(self, None, keytype, valuetype, datatype, registry, {}) self._components = {} self.handler = handler self.url = url def addtype(self, typeinfo): n = typeinfo.name if self._types.has_key(n): raise ZConfig.SchemaError("type name cannot be redefined: " + `typeinfo.name`) self._types[n] = typeinfo def allowUnnamed(self): return True def isAllowedName(self, name): return False def issection(self): return True def getunusedtypes(self): alltypes = self.gettypenames() reqtypes = self.getrequiredtypes() for n in reqtypes: alltypes.remove(n) if self.name and self.name in alltypes: alltypes.remove(self.name) return alltypes def createSectionType(self, name, keytype, valuetype, datatype): t = SectionType(name, keytype, valuetype, datatype, self.registry, self._types) self.addtype(t) return t def deriveSectionType(self, base, name, keytype, valuetype, datatype): if isinstance(base, SchemaType): raise ZConfig.SchemaError( "cannot derive sectiontype from top-level schema") t = self.createSectionType(name, keytype, valuetype, datatype) t._attrmap.update(base._attrmap) t._keymap.update(base._keymap) t._children.extend(base._children) for i in range(len(t._children)): key, info = t._children[i] if isinstance(info, BaseKeyInfo) and info.name == "+": # need to create a new info object and recompute the # default mapping based on the new keytype info = copy.copy(info) info.computedefault(t.keytype) t._children[i] = (key, info) return t def addComponent(self, name): if self._components.has_key(name): raise ZConfig.SchemaError("already have component %s" % name) self._components[name] = name def hasComponent(self, name): return self._components.has_key(name) def createDerivedSchema(base): new = SchemaType(base.keytype, base.valuetype, base.datatype, base.handler, base.url, base.registry) new._components.update(base._components) new.description = base.description new._children[:] = base._children new._attrmap.update(base._attrmap) new._keymap.update(base._keymap) new._types.update(base._types) return new Zope-2.8.7-final/lib/python/ZConfig/loader.py0000664000076400007640000002362410435500361020012 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Schema loader utility.""" import os.path import sys import urllib import urllib2 import ZConfig import ZConfig.cfgparser import ZConfig.datatypes import ZConfig.info import ZConfig.matcher import ZConfig.schema import ZConfig.url def loadSchema(url): return SchemaLoader().loadURL(url) def loadSchemaFile(file, url=None): return SchemaLoader().loadFile(file, url) def loadConfig(schema, url, overrides=()): return _get_config_loader(schema, overrides).loadURL(url) def loadConfigFile(schema, file, url=None, overrides=()): return _get_config_loader(schema, overrides).loadFile(file, url) def _get_config_loader(schema, overrides): if overrides: from ZConfig import cmdline loader = cmdline.ExtendedConfigLoader(schema) for opt in overrides: loader.addOption(opt) else: loader = ConfigLoader(schema) return loader class BaseLoader: def __init__(self): pass def createResource(self, file, url): return Resource(file, url) def loadURL(self, url): url = self.normalizeURL(url) r = self.openResource(url) try: return self.loadResource(r) finally: r.close() def loadFile(self, file, url=None): if not url: url = _url_from_file(file) r = self.createResource(file, url) try: return self.loadResource(r) finally: r.close() # utilities def loadResource(self, resource): raise NotImplementedError( "BaseLoader.loadResource() must be overridden by a subclass") def openResource(self, url): # ConfigurationError exceptions raised here should be # str()able to generate a message for an end user. # # XXX This should be replaced to use a local cache for remote # resources. The policy needs to support both re-retrieve on # change and provide the cached resource when the remote # resource is not accessible. url = str(url) try: file = urllib2.urlopen(url) except urllib2.URLError, e: # urllib2.URLError has a particularly hostile str(), so we # generally don't want to pass it along to the user. self._raise_open_error(url, e.reason) except (IOError, OSError), e: # Python 2.1 raises a different error from Python 2.2+, # so we catch both to make sure we detect the situation. self._raise_open_error(url, str(e)) return self.createResource(file, url) def _raise_open_error(self, url, message): if url[:7].lower() == "file://": what = "file" ident = urllib.url2pathname(url[7:]) else: what = "URL" ident = url raise ZConfig.ConfigurationError( "error opening %s %s: %s" % (what, ident, message), url) def normalizeURL(self, url): if self.isPath(url): url = "file://" + urllib.pathname2url(os.path.abspath(url)) newurl, fragment = ZConfig.url.urldefrag(url) if fragment: raise ZConfig.ConfigurationError( "fragment identifiers are not supported", url) return newurl def isPath(self, s): """Return True iff 's' should be handled as a filesystem path.""" if ":" in s: # XXX This assumes that one-character scheme identifiers # are always Windows drive letters; I don't know of any # one-character scheme identifiers. scheme, rest = urllib.splittype(s) return len(scheme) == 1 else: return True def _url_from_file(file): name = getattr(file, "name", None) if name and name[0] != "<" and name[-1] != ">": return "file://" + urllib.pathname2url(os.path.abspath(name)) else: return None class SchemaLoader(BaseLoader): def __init__(self, registry=None): if registry is None: registry = ZConfig.datatypes.Registry() BaseLoader.__init__(self) self.registry = registry self._cache = {} def loadResource(self, resource): if resource.url and self._cache.has_key(resource.url): schema = self._cache[resource.url] else: schema = ZConfig.schema.parseResource(resource, self) self._cache[resource.url] = schema return schema # schema parser support API def schemaComponentSource(self, package, file): parts = package.split(".") if not parts: raise ZConfig.SchemaError( "illegal schema component name: " + `package`) if "" in parts: # '' somewhere in the package spec; still illegal raise ZConfig.SchemaError( "illegal schema component name: " + `package`) file = file or "component.xml" try: __import__(package) except ImportError, e: raise ZConfig.SchemaResourceError( "could not load package %s: %s" % (package, str(e)), filename=file, package=package) pkg = sys.modules[package] if not hasattr(pkg, "__path__"): raise ZConfig.SchemaResourceError( "import name does not refer to a package", filename=file, package=package) for dir in pkg.__path__: dirname = os.path.abspath(dir) fn = os.path.join(dirname, file) if os.path.exists(fn): return "file://" + urllib.pathname2url(fn) else: raise ZConfig.SchemaResourceError("schema component not found", filename=file, package=package, path=pkg.__path__) class ConfigLoader(BaseLoader): def __init__(self, schema): if schema.isabstract(): raise ZConfig.SchemaError( "cannot check a configuration an abstract type") BaseLoader.__init__(self) self.schema = schema self._private_schema = False def loadResource(self, resource): sm = self.createSchemaMatcher() self._parse_resource(sm, resource) result = sm.finish(), CompositeHandler(sm.handlers, self.schema) return result def createSchemaMatcher(self): return ZConfig.matcher.SchemaMatcher(self.schema) # config parser support API def startSection(self, parent, type, name): t = self.schema.gettype(type) if t.isabstract(): raise ZConfig.ConfigurationError( "concrete sections cannot match abstract section types;" " found abstract type " + `type`) return parent.createChildMatcher(t, name) def endSection(self, parent, type, name, matcher): sectvalue = matcher.finish() parent.addSection(type, name, sectvalue) def importSchemaComponent(self, pkgname): schema = self.schema if not self._private_schema: # replace the schema with an extended schema on the first %import self._loader = SchemaLoader(self.schema.registry) schema = ZConfig.info.createDerivedSchema(self.schema) self._private_schema = True self.schema = schema url = self._loader.schemaComponentSource(pkgname, '') if schema.hasComponent(url): return resource = self.openResource(url) schema.addComponent(url) try: ZConfig.schema.parseComponent(resource, self._loader, schema) finally: resource.close() def includeConfiguration(self, section, url, defines): url = self.normalizeURL(url) r = self.openResource(url) try: self._parse_resource(section, r, defines) finally: r.close() # internal helper def _parse_resource(self, matcher, resource, defines=None): parser = ZConfig.cfgparser.ZConfigParser(resource, self, defines) parser.parse(matcher) class CompositeHandler: def __init__(self, handlers, schema): self._handlers = handlers self._convert = schema.registry.get("basic-key") def __call__(self, handlermap): d = {} for name, callback in handlermap.items(): n = self._convert(name) if d.has_key(n): raise ZConfig.ConfigurationError( "handler name not unique when converted to a basic-key: " + `name`) d[n] = callback L = [] for handler, value in self._handlers: if not d.has_key(handler): L.append(handler) if L: raise ZConfig.ConfigurationError( "undefined handlers: " + ", ".join(L)) for handler, value in self._handlers: f = d[handler] if f is not None: f(value) def __len__(self): return len(self._handlers) class Resource: def __init__(self, file, url): self.file = file self.url = url def close(self): if self.file is not None: self.file.close() self.file = None self.closed = True def __getattr__(self, name): return getattr(self.file, name) Zope-2.8.7-final/lib/python/ZConfig/doc/0000775000076400007640000000000010435500361016730 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/doc/Makefile0000664000076400007640000000273210435500361020374 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # Rules to convert the documentation to a single PDF file. # # PostScript, HTML, and plain text output are also supported, though # PDF is the default. # # See the README.txt file for information on the mkhowto program used # to generate the formatted versions of the documentation. .PHONY: default all html pdf ps text default: pdf all: html pdf ps text html: zconfig/zconfig.html pdf: zconfig.pdf ps: zconfig.ps text: zconfig.txt zconfig/zconfig.html: zconfig.tex schema.dtd xmlmarkup.perl mkhowto --html $< zconfig.pdf: zconfig.tex schema.dtd xmlmarkup.sty mkhowto --pdf $< zconfig.ps: zconfig.tex schema.dtd xmlmarkup.sty mkhowto --postscript $< zconfig.txt: zconfig.tex schema.dtd xmlmarkup.sty mkhowto --text $< clean: rm -f zconfig.l2h zconfig.l2h~ clobber: clean rm -f zconfig.pdf zconfig.ps zconfig.txt rm -rf zconfig Zope-2.8.7-final/lib/python/ZConfig/doc/zconfig.pdf0000664000076400007640000027650510435500361021101 0ustar ajungajung%PDF-1.4 3 0 obj << /Length 1972 /Filter /FlateDecode >> stream xZ]}У D~Jd6,`nw&m$YQֶ\KCeѲ2"ɣs=䕆D#QF (QHLs/8ZFJѳq%킧},-$`>FeTELfcDFR(M"W?w?cL˄f2~8΋%|]<\r[+0v@(`XM7E^tGKD( F,z﮹V'NmU4BstdH0 pCbP2JR$eQB2RnGcl8[O77K"b}攔xDhFLh0>РR!)3K7M{ȋφE'`b~T59NU]p#/ʦ8TlJ;޼w+)P#+標#Ne 0pOvMoχ T (s%A(՘ҁ1#Ql {0wU:|6ю ȱvk HS<5E;R.P 5,?aLEZ?< PeKS(>]o7 }p A2\^y]^ سy 6/ {<Лާo?¦Tǡ\]wc9,Jas=֧ eLD2##ʿH axH?iePA4:)`w%#,hFP)(A$UP.X ^G{n\F+Zjq^EVM mD2S2"D,j 2$kWp󹃡gsAl@GQ)Q dCͲ Ak\ɚPJK%F)|^ZRnZMض<t>`{X3ˡ`ܡۼ}6j"aG١f>!$4ht1h.47m7/는:6*Au9)@2uNys:AL4qP m?zț ZP`@,(ȤE X!)G6ӣƉA>9,RDfz`5ˌR3{ݿΔ֡gAtϗM^GcP r6$5KZPՈ'ml2Eg7܇iS M>-#&P{ցg hf\I‚] dӳtpn>,3Am6Z渇C;=B]NN*5cOQ" I3䦃ݸ~t`z)AR8y݄^PE8M/`ⴳjil`H2oi}S;Fw.j ^Nedl@s:x-hzpi2:_ld$ʦ/@TGsYOzRrNA g߀=bB;D?? LɅ HCR1UVBoC \x s͚}_Yvv0oz26߭sK<UU>N(BBw9z4qW۴Ƿendstream endobj 2 0 obj << /Type /Page /Contents 3 0 R /Resources 1 0 R /MediaBox [0 0 612 792] /Parent 36 0 R /Annots [ 10 0 R 13 0 R 14 0 R 15 0 R 16 0 R 17 0 R 18 0 R 19 0 R 20 0 R 23 0 R 24 0 R 27 0 R 28 0 R 29 0 R 30 0 R 31 0 R 32 0 R 33 0 R 34 0 R 35 0 R ] >> endobj 10 0 obj << /Type /Annot /Border [0 0 0] /Rect [455.098 503.939 540 512.417] /S /URI /URI (http://www.zope.com/) >> endobj 13 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 394.106 143.501 401.11] /Subtype /Link /A << /S /GoTo /D (page002) >> >> endobj 14 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 370.301 181.05 379.192] /Subtype /Link /A << /S /GoTo /D (page002) >> >> endobj 15 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 358.216 257.354 367.167] /Subtype /Link /A << /S /GoTo /D (page003) >> >> endobj 16 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 348.278 229.398 355.212] /Subtype /Link /A << /S /GoTo /D (page004) >> >> endobj 17 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 324.472 221.169 333.364] /Subtype /Link /A << /S /GoTo /D (page005) >> >> endobj 18 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 314.405 183.461 321.339] /Subtype /Link /A << /S /GoTo /D (page005) >> >> endobj 19 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 300.433 196.751 309.384] /Subtype /Link /A << /S /GoTo /D (page010) >> >> endobj 20 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 278.644 216.238 287.536] /Subtype /Link /A << /S /GoTo /D (page011) >> >> endobj 23 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 256.726 265.503 265.618] /Subtype /Link /A << /S /GoTo /D (page012) >> >> endobj 24 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 245.264 253.32 253.593] /Subtype /Link /A << /S /GoTo /D (page013) >> >> endobj 27 0 obj << /Type /Annot /Border [0 0 0] /Rect [109.858 232.687 220.851 241.638] /Subtype /Link /A << /S /GoTo /D (page013) >> >> endobj 28 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 221.354 259.298 229.683] /Subtype /Link /A << /S /GoTo /D (page014) >> >> endobj 29 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 198.943 246.903 207.835] /Subtype /Link /A << /S /GoTo /D (page016) >> >> endobj 30 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 177.025 261.638 185.917] /Subtype /Link /A << /S /GoTo /D (page018) >> >> endobj 31 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 164.941 159.382 173.892] /Subtype /Link /A << /S /GoTo /D (page020) >> >> endobj 32 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 143.152 312.617 152.044] /Subtype /Link /A << /S /GoTo /D (page020) >> >> endobj 33 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 121.235 294.435 130.126] /Subtype /Link /A << /S /GoTo /D (page021) >> >> endobj 34 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 109.15 170.998 118.101] /Subtype /Link /A << /S /GoTo /D (page022) >> >> endobj 35 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 87.362 329.164 96.253] /Subtype /Link /A << /S /GoTo /D (page023) >> >> endobj 1 0 obj << /Font << /F29 5 0 R /F30 7 0 R /F32 9 0 R /F34 12 0 R /F37 22 0 R /F38 26 0 R >> /ProcSet [ /PDF /Text ] >> endobj 54 0 obj << /Length 2526 /Filter /FlateDecode >> stream xk E 0췫 _qȕD"U.ٳ;3˧|v?( X~. v/< < /\@sgxy~wa˽< w9W=(G@ #?޶ͱ:yz8ꇾj2'|tp]՜hv(eBB>f^0l.e#cgȂW5ʨ  GWHwcOr^G 23ҫq%-!Dk3BD%iI L{{ܻi9H pQMO}9⳽BMu/0_05*6WJƿ~MJxO^y/ʳ!J2^~Zil$b rNaX r8K}][,SbE e(ϜAC*w$roM/?\6%'ȱRp:i7pD>sJS/٩kۀ{FImwm]eDZ8 y>W]^e#$^$˹*δ4!ZkÌA^ֵ.F^YAUG,mM*3aZ ƷRjPy3s"ha:-h V8$p'ҍ6jtEd;Q!Juy~WB 1oƾS8[րaD߶+UGDP8у}H4bٓEeW 7B=p⎇ L!4Y:@C㍷R/ (=3A3ƺjϊ{ 4)'KZW CveV_UQ܅1NHb}!~jA8 L @o4nЧ("|:W'܋j Ge9.LNEohOJ'{&#lNuyl™n" |AW{mMzԴ'-krG_cmuČjP2܎~uoZ+DP3r÷0`6l)1iȇa"۔YRΔUYІjT#5\v`*\m胈uW t1?g}̟ ǔ$>QQJ5 7;—4S'34/*y5Ɯ*mᛗ*_s4Em{[s,?Zs}R ,&"EVr̞llRjj)BJB!35F-ִ醦!i`C.[6tIjz&X㈸ϲA! 1 rgQq絻lL[>}"qF]ikfU76G;CJ`j&sۺn1aߛ|.LcSY{U1`tu[pM1ByM_IE|󺇙/?\xy"I<~q}D~$ <x@8Bо_Toendstream endobj 53 0 obj << /Type /Page /Contents 54 0 R /Resources 52 0 R /MediaBox [0 0 612 792] /Parent 36 0 R /Annots [ 55 0 R 56 0 R 57 0 R 62 0 R ] >> endobj 55 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 708.01 300.881 716.902] /Subtype /Link /A << /S /GoTo /D (page023) >> >> endobj 56 0 obj << /Type /Annot /Border [0 0 0] /Rect [86.944 695.925 149.15 704.877] /Subtype /Link /A << /S /GoTo /D (page024) >> >> endobj 57 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 674.137 236.751 683.029] /Subtype /Link /A << /S /GoTo /D (page024) >> >> endobj 62 0 obj << /Type /Annot /Border [0 0 0] /Rect [114.485 367.125 186.216 376.076] /S /URI /URI (http://docs.python.org/lib/module-ConfigParser.html) >> endobj 37 0 obj << /D [53 0 R /XYZ 72 744.907 null] >> endobj 58 0 obj << /D [53 0 R /XYZ 173.268 607.292 null] >> endobj 61 0 obj << /D [53 0 R /XYZ 231.327 397.217 null] >> endobj 52 0 obj << /Font << /F34 12 0 R /F37 22 0 R /F32 9 0 R /F29 5 0 R /F38 26 0 R /F41 60 0 R >> /ProcSet [ /PDF /Text ] >> endobj 66 0 obj << /Length 1742 /Filter /FlateDecode >> stream xXK6W(@%w(!b$@i8zڴ9fr_?EEz{3FqE,wRo$܁&-i2|}bL|9r==h9m:tӷH !쀅pXZ?6<|YIA^h%{:@PqɼSG+IҀ#@_ Ii$$Y b_#yl8[I1$[e$B(Rmi,̠*.08ඍlh +yq#a |,y1M',]\ׂ|U5;B Pm)]]t7n: 鈒A$K90uڹcZpQH݂ 8T1C0Ίh"NfQ$blH!6Cl,?]o+-Y*N1ߪiI#\>\YXà AWYKgS]+ݸzr@I`' >&1$&X@e5jōh*-".l[qwe0Gk(˱csRm0O\nkO':i[Zۂ%Чq9 `V+ oA^]nuhj;mk.6tgYMĥ%0RZWnoچđp LDؔ zE")ڈCűIڠ]o -Nၘ uP㾍Uy`T?K tg%ms,ΠA1D 겧}T ,Ó;,L -Ӳ&Gp[v20*={2Eo~56U*$Ϡ MAxo|GF.Xb(AtRęk+RQɨRz3G9x] ιFFqfl X3`q&ƆL4׭=~h:h(W4Ej`E\0ZcxxtVm!{Oؽ@W(Q|kF =aS~қ6}Z-f4L8=K+녂0.~ > endobj 38 0 obj << /D [65 0 R /XYZ 72 744.907 null] >> endobj 64 0 obj << /Font << /F38 26 0 R /F32 9 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 69 0 obj << /Length 1934 /Filter /FlateDecode >> stream xXIDW‘ǻۀ@3б;̐OUWdx$TUWUz=| Gxy:O''}/S'sτT~!}Ǿ-/8x:{`#d?°؋D`؟j>HܿFӼGg 0܍*qFT㯯8lN/yT}Y:~綡Nf"oPĬɯ_I?hF5. $1$]ߢ=/Th ux#X;bB^Z91ˆ)\*xcӢ.#ji晆X_z9jJXĕJybeӨ!LX?ɠ cH 6ak6rfOi-48> 8mf3/ I^YUF+5JS;݈UuZ2fPcgtS8&``$a4ރYgʞXwB<cŝB/-!`B/VMӎ4>;=X5e6pBP=C>X_Qwx $Z ([8$% LWdv,bحTѽ'`Dt488:AY&JA]WBZS@RUHH6q:^('%e5"!0Na=j0mEX\Q 3,M2f)mC1"; xV%aS`|5p_8H&SJK pr7$X+d#GeX8aF&3LgE8UqjL\YBܒG 4m.Q͒o| ו6I|SV=ӝr.AUZDTC ~0 9}[vnXiV@NpGA/.eAF 9~(r)(yts\$uL˼ٔw}"(^ VAԗF2}P([^=/?2NRo $F;xxP?W@qxJoM ea%.ʎzsW(5_u°ϝX$N8miq0d xbS,U)YnCTuUe!@!Ƕd$F;@tM& D}PN+:x)΀iYj^6FIS%6ֆHs1>f3JOLM1?pnK}1Áݾg\j@ DOG]c['۔8<8!ICFt35'W (TطQYz.t7Q"Hɱ%9=I4!jy@ˤNj} ~U =ֿꢆN$}.i5bOJך0ڢPp!D™$/XPeZ @5wG0{^@f`bpNɣQd,ti,MO _p]Ű/{eNgԩ"^2jT'ݨE[WZNP ۗ:QSҏM9mM"Sw> endobj 39 0 obj << /D [68 0 R /XYZ 72 744.907 null] >> endobj 67 0 obj << /Font << /F38 26 0 R /F32 9 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 72 0 obj << /Length 2827 /Filter /FlateDecode >> stream xڕYY~_hQHX1^ n4jcgȏOlKbꫯ8_?}6yp^@cܿ}>Py g8 Z(ym<`E+4q d*H/i9@ќUyP>ɋR)y"C~_Hy‹.O(7J"}['Ѷ>.%hzaID%Hmo3l~Kmg>َ:ٶ$d_7fJmŕU]tݳN4Z8_@[E[[O^;t,Z<е(N5t_Qok.EStgl)jh5qSóZN#G>ohVَ)ǏHM~(h P6_ J=N_,%$B/]T o҈"i}Eݨ,Y2c~eOF @`A){ K7MhM-^zSPKaV8B^o|*.Zb`~ èxR +/OC^1e?s؂dN_93WngW ВJqK{" upu>//E©]VЈj Tm@T)ևa!:=|(j'd4Dځ-,+\SݗjCۜzsHK?*BYo`c9`qzưw%;m=ް(@׃w=IP:ȍBuHr?߅]87_kvi1nM҆h]ٮw"T ιZW=ׇԢr,lR ;{e z`l !Yv!-KbdEy$[qws/RE a(+:L7;""Q83:M6G:d6D;4)C# ^_b+ӭk=c( Rs%HNB@% +1 fdqvB#N՜+]P/OoRj5#Tlxg+>ROGER @1reޏ֪% |)DMjV/Ira g7@8N|6ҭ(?(A 3#S ', 7J 8b9K;6P*ypOKǖdQj `qӟ_~]gSu>oS\|8ԒHcQ4zΓ5q @)/sݕdSlgj5΍,P}o,f,p2Ȁ&͕O}:uu'w+VC}muJ!ݽ#L~ !0QyVcdXn4:q"ztп+ZįEO4tDM d,gےߖ@3/ӎAiH<[:[^Azԏ30)i=./ uY҈;8[Џc7 BqaQg&ZDEijWi[nHo,&e1]g;;W01kkatLj茏<&9 qN!!ۮU—ӪtIWĩufU-ζGBh3p37p;bȱWTO@R<bNlt>|ý=bߤE||דNj<`w,XdOeYzHt?@ D6?jD=_c[`B/ (pHyr  o'`4 bb~s/Z%`1~%$75I{ܫG䪄=hmP7|Y,H; ఒcb);^ :a$?}SPd[/"9LdVHBєz_዇$-W?~%4"4=Ea_D~熯c|.wflHmY[+wLZyI\7ye`eAO@Wz)X$?6}aO`+ 2 hY:껔S9 Ǚ]ʑm5q%f4jЪr5!-!S/TpHD<6klqW"IWB̺ؓXhqn֜wA0LsgRZ…j /w!yU#?؃"x?߷Y,Ȼ+nbU0I7o.puBmmV<ߜ4bkw>Ղ`jO *_52endstream endobj 71 0 obj << /Type /Page /Contents 72 0 R /Resources 70 0 R /MediaBox [0 0 612 792] /Parent 36 0 R /Annots [ 73 0 R ] >> endobj 73 0 obj << /Type /Annot /Border [0 0 0] /Rect [420.448 559.854 540 568.805] /Subtype /Link /A << /S /GoTo /D (label-module-ZConfig.substitution) >> >> endobj 40 0 obj << /D [71 0 R /XYZ 72 744.907 null] >> endobj 74 0 obj << /D [71 0 R /XYZ 288.926 397.773 null] >> endobj 75 0 obj << /D [71 0 R /XYZ 200.899 282.43 null] >> endobj 70 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F41 60 0 R /F29 5 0 R /F34 12 0 R /F11 78 0 R /F37 22 0 R >> /ProcSet [ /PDF /Text ] >> endobj 82 0 obj << /Length 2877 /Filter /FlateDecode >> stream xZY~_HGl$Xm|;H~HԈYI*ǧdf Oj6/oo^}"X/nw )(UfH ,n?/jWkeCiO<J.z=w\)炖׻կ@mL :5Ů|ϫFl42E@0l[j[xh.ZȖL^0?ovcs0CvŖeE(DLt[a Zjک@DxkʻsWaj-v+) ݾli*G/7nqӮ?USf,mX(/Dkh,|פU<8"8h 46T̨. U!nO;!C2nD;7 jƉѩ9y߲gF@s44#e%f/mд]V5M~aSthclfF^(-k:&z6cU/cc 'g!T]y?mʺ"?賷f RiC'Ͳ#@-p?E blLU #oavx:UTZ%(VY˵ ;EU(5ޱrXhDVv:`ݔᑦy7kDX#hk7'zV`pa8 (%QFy=@Ƙ*Tx~VLX_bB82(3e&J=]xE#7 SH^[73 Rw`jct'م I-páQŖC#]:MC<$G]w'?;LX5$]mͮ>Wْn{6V%o伔m@E#.(|r7:-M9ft|Ճ!皗0$ZzcHss(S)s8p`n@@8$#[z_%›xٌ]A*2=S6+8ý5"Cin+Cv4Orr„8Jhff#c_d]d/mX+rZp!VWx4&9ĚEB1b%$, "̄(V4Xc03BY`̃ (6c|Pw{ &_uαkvK08*PBKu0_Z'jٙGR O.需WéV bfp21,E*F1On4䡒УCp qkoL1?ɝ. R Vxx7x"DhA?5NQ XqyÒ0Uxite~-X_)4]k\޹V cP\0)rriY([Ϝ,8CfBbNB]11y`ɓPCoGJb@vwE<+m= BSI \'Ua}eգ@lR$ &TD$ކ4zgb}$&TOoz}z7e&Ŕ䘷WSŬ Gz][$cO͔|"Rܜjga9CNdf 9u_k%٭12紤%.6D'2sLlƈ4y!Sw-eCC M7.Xcl¸o_̗U^өș@ cYޏ솛7D8yo1hR(+IRDj'7>AkÖr鷄އa^51o;\'NSt75xfUU'P C 9/g_`Xl"E|3uu}p5@÷k$+,'MF T[:H]eR0,H,B!-l`T-6Ǜn~U,7bݍt.W.Gۛ֞zDK9gĞlgRq{u o̢8Y. n0XոKnzDPq.J93HHMJeJ5 S`-9b [ 0W=>|N;;QZo C i$ q{UIRA5d>j_^~<1]stޮ cUpO$@TX b!oޤ$穜uMFV`}~<)0>1 XyDr"jCfiwiJwLx~6tFX0E0A=m͑A> endobj 83 0 obj << /D [81 0 R /XYZ 72 744.907 null] >> endobj 80 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F34 12 0 R /F37 22 0 R /F11 78 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 86 0 obj << /Length 3120 /Filter /FlateDecode >> stream xZY~_Rj$@`0GvDvg}ɦHjs?/vWUW}u.RHҤoV?7&V 駿tv"OS>Un󱢅I򷼑 Z);QJ}Vk䲠MsشUǃ|&uZbJ}PEW TT8`73LI^b%F8~(Nf~zfVڦO ݲ麪\aF$i6p 0er=#&מMjyf} raܲjᏏ摛nG8wuUV S>[3wmO+5߇'yvam,p9Շ?# 9VmݔVk#@j2&5AV'\"TL㱭Ӕ362 Euz8> "ǦLo4`[| |ĖÑLC)Lha+y?!ZM fȇ!66/򷀋I%E4-40?ɓJU)((& ) (- yb *A`r[0?/O_AB /'*މ`t_<Ә*#f-# 5i"p| 9nog0[ %oG؟x},ʲ]kFl,lqЂ]I!ϳYetD@#KqKps{VH 9嵸wb1hy-\F_!^ImSO6R82&)+0; EYmw ęygKP)OO`"D47tf#'syȴAQڤ@dž/?dp%"Un[9pc9Hϴd;I^WlQ误Y2uv3L2"cor> P1ށA)]eҢbGAěS=# zR :jfl8(m X5{q<;8BEZ8|ׇU&yWK|ȄN S8Z6)BqF(r+ )}}xu &#`ϼNMy-MamAj<э7 㛶ne wpCDzQ`y7J[E^[$oiO * RzsUi_ 0kŽ}S^Z?8S/J|bv2o IF蹭w8t7AuDP߲Höix'5pDd${h[ۓPTöqu 4YLPkZC4: _"Waq SBy |%5+~Ln9#"Ӟ5"dg>}wɔmP :5(*;f&ip:"EKЊ #A** )9&3]̘@ _̮ol-W3WAţDT>\fr@?y|ߎ/v ,[Vۊ~9r)5A݋ IJ$!)<0J2%M a q+AyA[/,jҹT4"Afۊђ*RyO:y3F5D)LPxWGA4JϢ9'-0suyG`Qaz`8:\ %3(t NUhNPaZ-ݡi3&Z}}rуȍy!_N>"I$pT v]/UjoY3G*8NOyԪhRZBG> endobj 87 0 obj << /D [85 0 R /XYZ 72 744.907 null] >> endobj 84 0 obj << /Font << /F11 78 0 R /F38 26 0 R /F37 22 0 R /F32 9 0 R /F34 12 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 91 0 obj << /Length 2603 /Filter /FlateDecode >> stream xZߏ~߿o/Q"HhHڳ[r.wCʤtD#Ùf_}w/,F-0E.8iY}W?jw -F܈wr?{}QLK۪9U7L,:@źPl,sjX6GzaKkj+ɗz7I}֯+Jd uvyC5Z̖ᷕJ,&uE$|Smm~(qɁ 'לTDSP*[Χ[{M\Qn邝PmT~>T Jz=Ѧ7mG=ͮ:lΌY-wDӱڐl?clsdV5ZfO/s ۬4{᫮Կ[#wUsگK_f iļ[KP"d-2fx[;׉Gd9 x{(OfMLgӒI-lM m'v*N7ɋI JoOo^85EN$|l '3j)}lSV(ay&[*<2|]X@[86c3L JI%0xp[|[g0a@cmEmk 8E%2& Ǯzhz;D{DHS9Yx~XA V@x 'H L4+qMdY˺#o!0Yla 0os;Qynct;鍙C2ff=*&ؑmc;+a(@Lo ps`*GkM '/.܅IA n)#NI28ĩrpj4bZޝw5έ53*:Mg9*dTLr#N8e?W}`=k`b6 fEb8MGJsDjI>|̓y>,Ƿ5z/K} 'րՆ*ڎVԅL}:crf$ &"a@y3P>d,I8&Sᤩ`SXURf:ޓUW>P`%1m^@<9BcH>F2JjE!T!Zo/2^Fekؒ ӕ@[i ʞ R򄞥< UwEu ½`1?H Ia9<2AnPYF}UnvOTBp.,n ,yă🦝:(ʹMp$4Ь!6&r4CKO(%~hiU<ί6=B3^'Np5yg\2.trXO ]x;dOB먺A ty >c U@ "ۖN &U|n^ C`07h9 +Y!B {1ie>`:34RuQa GNky S;.].ɭS<vKU:A>F]Kk|pt~4J E\bW~ 0A;TR+J[Yg̤ vtHQ3# }b4ԭ dÐk1$oYjl7sGod{n>T~g+>Sa#tY)^KRi}u _1PąG 1Sx5?7Og(g +?fMFgӍ)?+xo?CFd^W1{?hfF"Q? R#]C/Xi"i陲RLP(Nu*auu4LcM~RQ3tXLziв^]|ގˆh ^F<6<",KnŬ?"-iSԅXf0W7 rQx bby -;m~~Ŕ~#,N t8BQO<$5lQRiҏ&5\dmWg{=~yuC)FȄ݈GғǩM.෸J%^3JmrWڛP9ei+!հ`_qӆCY^7[/4:\$[n.!d 3>}}|Q>6jBP=B|Ԡ@1Myn/" O/lxm΄Ý#qݛ Sau߶rW$G-Z|# ІSp&1/ҼX!bk_vFendstream endobj 90 0 obj << /Type /Page /Contents 91 0 R /Resources 89 0 R /MediaBox [0 0 612 792] /Parent 88 0 R >> endobj 92 0 obj << /D [90 0 R /XYZ 72 744.907 null] >> endobj 89 0 obj << /Font << /F37 22 0 R /F32 9 0 R /F34 12 0 R /F38 26 0 R /F11 78 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 95 0 obj << /Length 3125 /Filter /FlateDecode >> stream xڥZY~_ј( 'E6M7g[پVRzTvQ,:8߾}#ĢʪB-/J(E\w?ܾQ6Z~qdUY]O]ȬV<&"ʳYe]ι*SVi)؟fۻCR.[7?tuȗt#ͲޝoO0ayl]ϹJ{HYfViͪݶa&5֝\ݻ }GT,D9]tjk:&|i𜟚͹ѬG%:ϭnY"fkxk"$y?*0*ϔfe fyl17<ӺxAY*@gKZU- SZ}9'DwIkWޠpEuOuݹ/Zu߷ݍ̗P>m(黶%ӑ.{IؒDD]i?"/dV xuG[e{Y.r *Tqf9 B y:]5GF%KM,¨D.LؐxӪ9tG>xǬHaRye\":ޒ\ ?I*ڧ>)&+󐣼ywG$ :4nDlZK e*_~W3W6.RhHT\@˰ -0u1ぽP(-:'&E0d\pZfWZ@d!XywzqyXnPrP\ _V 1ni~ºmNHOsi+~_1hR?'r ^% QT!G$ze xM#Ghw\-ռj *+J}uyOS1_ƒbϥ[("M Og6,q *KsqE+B=co^TxQ, / *r݆w z cJ/z[Lli_dVdTX؟{* z·QV`g߻ խy%z+901}>NpVK=/lis#Eun |b+*AUf7jelqOhϜ|S0p9';d%Ǹ|Ȯ[[׽J^ \[qEDvB9sXA8g3ʻL.63O~CGJ,"-cbp$y~$}4U^s\0U[*^/aL̀ј,zʔlB- XZĬ{;(c@̿5?xlJ@a2=/:d|&UB<_jB %cGiD@o%UfULcJ ? ?{"XkPFJpIv7%'}Gl1]PR TL%Rd+S=),0heQ1X6tQ`NǑq .[]t,w$|Mڝ7\$S)Okܓ(NJe8%V$hR\N,/Y $SDfb#@MK %t D|CV⇆,HP1Vef%rЕ!s-;nTΝMendstream endobj 94 0 obj << /Type /Page /Contents 95 0 R /Resources 93 0 R /MediaBox [0 0 612 792] /Parent 88 0 R /Annots [ 97 0 R ] >> endobj 97 0 obj << /Type /Annot /Border [0 0 0] /Rect [354.82 563.99 396.663 572.941] /Subtype /Link /A << /S /GoTo /D (label-module-ZConfig) >> >> endobj 96 0 obj << /D [94 0 R /XYZ 72 744.907 null] >> endobj 93 0 obj << /Font << /F11 78 0 R /F38 26 0 R /F37 22 0 R /F32 9 0 R /F34 12 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 101 0 obj << /Length 2894 /Filter /FlateDecode >> stream xZێ} =6MR k$;H DRO K[R|XEz H*V:,w}]*(=vJ%"f+)doYù;)ӵˆml!kۢl5g,K2^%pUW_n.<粪h豸k2R" C7mxЁH8Ex<LJ@R(Rcg ˆ Id(y@<1ݽ!-q&i a($l2ܣ5<Άlf{`q>Ru `h dur2@]vhjo>> v|; @GiO(k"`U/űD1/<3S_2XI, T(FkPڕBZ. j9mG%j슡Z-&PYE q{j_E)>_K,bj(FVc%{!|ɋؕǍ-j#R|-gk k!;ŸIEb^^ϗ$O-(Bh_{D#vbb'l@oF?s _g7kkǯ47zhcT.e:S&mv̏tͶcom/b@tQGEo2S)*{H$.},φs6= Dt'tf d\'!_l6A4rxY%!F}Ow]|wi6_jrLͯ ~F7>OpE_N@&D"2/~ێ}!+jOeږp/wjݱ.`wЖB鮾3QFq|4C#'UJ[m>+?@Ճޥ<:C$ 0m`5=tdendstream endobj 100 0 obj << /Type /Page /Contents 101 0 R /Resources 99 0 R /MediaBox [0 0 612 792] /Parent 88 0 R >> endobj 41 0 obj << /D [100 0 R /XYZ 72 744.907 null] >> endobj 102 0 obj << /D [100 0 R /XYZ 218.844 299.571 null] >> endobj 99 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F37 22 0 R /F34 12 0 R /F11 78 0 R /F29 5 0 R /F41 60 0 R >> /ProcSet [ /PDF /Text ] >> endobj 105 0 obj << /Length 2708 /Filter /FlateDecode >> stream xڵZm۸"+H( $iS\E 4d^%WpHYd3ϼS~ssUTwL2HfS|eq]. +.oñmT3ж9dYɒ0,*Ib:I 1Z$< TMӪV<<ڎ۽:8Գ%0!SsޱS;u[\_$[$w(T0*lʃZҬe+l&ϝ%W#WEkxTM* ﺖA[neMU lG=iTQu:;ĠPKYa{)ӛ`1_}^pQ,[5 tov?v{4\.o37dgDj^yةR#:]Un~s4NFu4À `)S|"BF1Νi>e2g2q9Su QxdO\0{iRuȣ( >eS]5rry/Y")|6;!Lns9v p訶YҎ & _yI#:]RHXC{4/{)Lf (d3{ZʲS!O8x}]<+#²x=K@束2 Y-[(NLj9GHA_:ڶw-DeɲIpM}r|>bĕӞ7"}9 QmMH N X#DҕﰆzLj$O;=y,=(gK1ñv0Oo+meE̍i xB );t DLaC!d l7ث' tlz)!W],>Kg~HkO cC3>S;h+סԒ3.!NI#y.D8I9f듽/낍n4lP&kK >Ȳ_ sQ$4<ΐ.r[ZitRx>-[a +)D|طypl`ۘ͵/^%v[.C`O۟A~3-F>Wr,=Bo&s% ɝJ`1̈.[{M786C/L wwi|5&&mLL򙉁ًH3z`^z|/pl[魞}-z  `x[z)nkW!X8fVi =eGJ)}K"tD|F8Icݨ!,bq߿'D T7>!`M%R-vbryoaҏW܂Tfc_C9biߑm%(sgFB1xX.VC? F=U^p=8>a#MV it|f#to1Hxg/%U:=(z48Gp4G[08%4l2CycKswWE)uZ [W_lkƸm&1/gE fK#O 7Cǿ]$![m79ZU7W0ڶ(V2F!~'y쀢^N$q.>;> fLendstream endobj 104 0 obj << /Type /Page /Contents 105 0 R /Resources 103 0 R /MediaBox [0 0 612 792] /Parent 88 0 R >> endobj 42 0 obj << /D [104 0 R /XYZ 72 744.907 null] >> endobj 106 0 obj << /D [104 0 R /XYZ 285.355 608.322 null] >> endobj 103 0 obj << /Font << /F11 78 0 R /F38 26 0 R /F37 22 0 R /F32 9 0 R /F34 12 0 R /F29 5 0 R /F12 109 0 R /F41 60 0 R >> /ProcSet [ /PDF /Text ] >> endobj 112 0 obj << /Length 2986 /Filter /FlateDecode >> stream xZKsWfBcsک67VBŌD*$^קCF\rh4unQGb}|D"SsPV[˪(U^ͽL۶> հPE" ە"3-~WݯʂOս :/" #d/^l<6鰍ԩ0?t;DrcD{{,|D}"]xjr/v3 E])DNsڃ39/3nw LOnGOkK[A̕I~'U" y7W8:~GsoÌlAA 9$U^EN2\\*)txg_lsU4 6Sg&2Yqa 1KR M{o^p~0a,. )Y`M^#$WE%m򍂀&LDu| A ײ:˄J$d4ro!bO_,&MES9,`Uy[K !~Wygh#eAO?耠t@P)ܹdPXMDctۢ|)Uu28-~<ϙ;<ݩ>V%o!"#u$2Hjωe: ; vipmGZ?g4ߔfwf䛼[ [fwڂD2<-4|pf&d ߫Y;‘e>O'D*ASr8 ^m,m9W32@0b` sORM#{d j~##Neu$(~r?ڒ =j?XL 7qmE4σħk8 I=_*@Ȼ;: 'B ^tUuܯLqGBЅCǮ 0_;xHQ/]= }Wh%VPݛEs[!ors,Ʊ4߈ %J">NQ_ IH^^yELg"zi#%6`ue,_cLG"5z-GFA1O1@DǝY0Ӿ.}To *~#H*Clv87sSsu⎈:c6m}l6% ^SUnŽDph%-E9ҍAQ%Fϸw9KK0B2jIj{{!|<;3Jb%L:IVmٕ{=$=B\\#4ݝ =YmP?jB xwyoy'sސ1gK;hbw(.y6PBm?XB˛q&=@R]gIץ2UET`uq D/'b8%<E^;2R4h';Hҍkp޸/KUrܷ16OEx-ZRb_/DFVjo:\a󌡰J{XesW;'R8W-6_@Tj'YemDFÅ"E*| zJ+՞'r|RazhbS!B"@VY& ci$o^a]cPW G@X{&I|Indq .̄i{r Vҕ@:D~v$fэ̒?R\I|iCS-q$\Hu!<%s bb\)ǐi5R~;s'i,*GjqПYO7hB";o}mqp B o"W=X L9 "Aʄ}92Uchxw֗endstream endobj 111 0 obj << /Type /Page /Contents 112 0 R /Resources 110 0 R /MediaBox [0 0 612 792] /Parent 88 0 R >> endobj 43 0 obj << /D [111 0 R /XYZ 72 744.907 null] >> endobj 113 0 obj << /D [111 0 R /XYZ 362.694 124.792 null] >> endobj 110 0 obj << /Font << /F34 12 0 R /F32 9 0 R /F38 26 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 116 0 obj << /Length 1558 /Filter /FlateDecode >> stream xڵXK6W{YJ6=$HyV+E{g8$-N Г7p8gE8덓ND> ާQ>c sd(/nS߳ow}'idwbK-:+|-1BI^`ÀI8g1XCr;Q~"wwCRWrDJᎭh:M=4 GHjw`eoۈ\I_,xX(^lY*& S+X0/W%SOz^ፁ$fA9esSTy XTƋiWquzժATB'a-c0cnE,`8HZ㑱mendstream endobj 115 0 obj << /Type /Page /Contents 116 0 R /Resources 114 0 R /MediaBox [0 0 612 792] /Parent 118 0 R >> endobj 44 0 obj << /D [115 0 R /XYZ 72 744.907 null] >> endobj 117 0 obj << /D [115 0 R /XYZ 192.626 543.377 null] >> endobj 114 0 obj << /Font << /F29 5 0 R /F38 26 0 R /F32 9 0 R /F34 12 0 R >> /ProcSet [ /PDF /Text ] >> endobj 121 0 obj << /Length 1782 /Filter /FlateDecode >> stream xڝX[4~?"ZTEč;jPBll8,N3'ˡ*O7}sHy(Ȼy20ʼ/ðJ}M7nL5uCSz6O "E4I{۾{a-_lP>ȓ.*|fägն*?"j˧=DcLf.$;_N2NcuܮT㈩{A&" /RIb~܁$*Z -(;39wǮIIQ(NAf&R[ aW?v އIb+XiyJDT.N~M,0v+|UbDoHmYglorr,G,@* ޴O{=w;d)uwC SGŰ/PK'8S! C0(dvGŭ.Rq-!zk'>l _uۺ=={~^~{] (avޮnf|IVȜd UA؂g_l1_t?DdyE*xy“ՍsQ'uqV&IdHԹaҔGXgjWfԛ(-j@Jf P PS̄I-0"MP99&TB$2U|sqY 9@;%ង@Ɍ?l1;Г^ T.Ed^ށ{q$$ih\6_2D Λ.NEyn煆3 hm!.PnuWhϖӸ/I3H 04hsZAuO 08̢,La%ʕe['1j|=Vb]eO |ZKF?f.yh?a ]-CKhc’ 4,b4oxcUIϱGhbQC9z@L fp;2n\iv(A7M.MǩFNLHbsi]g 3][.q&Qeם.q|> Neub9.ϝIb?cl"dt5iX$ٍkp}84"8YSgC 2+Ei{eV>-7acgfuy7ΠDek:0έ*{"VeuLPzG;eH01!o !]M^@z ,O檇;9X !$v垾ȹ'9~82eKM05A s mQ>bSM h[;p~$ (dl- }1.t=PJGާaZכv#Ϣ4yN71*`0pCJ SA*QdYsJ}B 惹@LR C2purKk ҦݲEOS'~zd[`%r=nFr@nGR??7;e2XNgF2̭vG3xv'"p:(.Ns^7w~UL OTRT&l} tz/H|h~rLoŐ.R^>J@A|Pgv7/o*endstream endobj 120 0 obj << /Type /Page /Contents 121 0 R /Resources 119 0 R /MediaBox [0 0 612 792] /Parent 118 0 R /Annots [ 122 0 R ] >> endobj 122 0 obj << /Type /Annot /Border [0 0 0] /Rect [448.323 408.9 527.526 417.852] /S /URI /URI (http://docs.python.org/lib/module-logging.html) >> endobj 45 0 obj << /D [120 0 R /XYZ 72 744.907 null] >> endobj 119 0 obj << /Font << /F32 9 0 R /F34 12 0 R /F38 26 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 125 0 obj << /Length 2122 /Filter /FlateDecode >> stream xڭَ]_!(7k$ #<ƆZIj'!rv7fT쮮~z'SVR 8˄Lzn n?پzSAu*PFee}>fCȧLv`]Z͛?l~n-;%E6\h yl 3NN Eib;{ٖIJ]#~AQ1ŶmeSyv;J%%ȡcaSj)"-tl$+-Q>os#*!idqIg"CWA QA\&(!c=}_h h%/KʄǼEB5:!e^)c!,839sn8C: !"5 P'hP8%kyY:3歩]G+VX"wD&o ZT7s(yu,BD/ P+esr1SHij:Qj>ڰM՟.Z@?ӉuYm>MYouXddw&!"*M#5ND 6:搏 }?exwv1NZ,Ch_6^q_"-)/QNI$/ڽ:E]13KRAAK!EK-XdK^,`g2%Tߠ>cxKQYU}LPBR) b;!]G`}!N=N{ڴb?5I Fc,C|cGn$0*5#x& n鐌&`nyZg(QRThb*15::{I0x򫔅H 뎅y2S/Шtυs*g^W{ ?YWuJ?'x!'vm+?$ؠI@ETQ‹n_Cv–Sv1y|_;x!ǝ텚Ҋ!,ָ\:mgIǠ'qh@hEytn5B{ y CB%$T2<h <.ҁPys }ش֜NTDR.Ɩ\;:| S7JOhuϛHhzɀB_3fT8o&wp90:uf2~.&EK7DrV< Xl4!aB<ɑ!o/P/+8LY&>,87+~H8v!; }P nzZIrr&xC"~q#׎E)KOnb~x'&0~8YDA6P#]YE=8mʿr@_b&q(e<-3g5նcn(BhǧΗr0Eq>\W~Y+n%3fl}].W?ޓUu u6zPye^H$#R5V/Vn,c]:"|endstream endobj 124 0 obj << /Type /Page /Contents 125 0 R /Resources 123 0 R /MediaBox [0 0 612 792] /Parent 118 0 R >> endobj 126 0 obj << /D [124 0 R /XYZ 72 744.907 null] >> endobj 123 0 obj << /Font << /F38 26 0 R /F32 9 0 R /F34 12 0 R /F29 5 0 R /F41 60 0 R >> /ProcSet [ /PDF /Text ] >> endobj 129 0 obj << /Length 2194 /Filter /FlateDecode >> stream xڝ]}P")5(pW\znч$\+K(3>,%/=&TmEFjy, 5۝LlEOhH/<8JuULOJm_xUl7}`2ddL,Tq7ɔc{4}aɓ!ԀPowBFQP5֡E@CȈ?T8 tw'@dTpvR8(-d`)@'i$xkd`'c>2xz) 'BBfgϦUARE&ԙsg,̲xAk3(^1`gɊ.K2D/4+E6+WRw|U3C w` T} _ap&"T Sc{;a aXdjUJ<ʘc>T[d(T WK2/ɘ󖡚h񬛲6Eˆqߊ *9i;*yHbK IRf7 37Vxj!G U4œ{SR~l"`q̈́y%8E9,i/M2^\moN{"pe]Yr!sמ[k*iY[tz_9+qd`txg(US;٪! g}tWRͮNwmwm6*W Ħ8ARΆe@nD{ﺋl]kJxޛc*;ZEU]8Ayͭ RtƁ9U |`kF yG8;)3X fJ!z/d*~;x? ̐)L" U.f/c}U cӒwLN^b(̐7ɮ沇7,:} K5Yx&7,\UdZDv0CBsAK)fkO綁0~7MI@cV$:A2KKKrݓL2],|vƺiE~Sٮ_UzTKϹEsذ1ѮgRΛE%9( !*s.9nzxu䜨)㚩ś?xsc&Йn垇 弎k]q02$a u托|Bs!ψv/P|!튀JZ ,,OȌ:Fj &Ph>^Ѣx)J*)rw 4ߺ%l|\1KHNtXBu_w(qI,1ĕc+Nk/fP`FJ'd+@\knnsw8J0+qA혙f:vZi{A9GO.);c.In|1%uYzbR#Oq床wK9?^5y1L3w}!C*i8yT0!)Ss @7,g>% $rA;l]H̴F La*x6eso6@odDt9`i0mq G8xh}Г{)5 F+8PEfo_ыd w~=h)'Y2 L8WpWgycMJ Bb&fȇ *9E.& "`)Uc=9Fe|΢9wUtvhrTlݨV =-ܽᮚO- /-^њ^Pđ/ZҀms4qtaqV_M 0<|1'y9i_m$h[,C!$9dݻ*>~O珢Uendstream endobj 128 0 obj << /Type /Page /Contents 129 0 R /Resources 127 0 R /MediaBox [0 0 612 792] /Parent 118 0 R /Annots [ 130 0 R 131 0 R 132 0 R ] >> endobj 130 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 391.793 237.997 400.745] /S /URI /URI (http://docs.python.org/lib/module-logging.html) >> endobj 131 0 obj << /Type /Annot /Border [0 0 0] /Rect [314.459 379.838 417.422 388.814] /S /URI /URI (http://docs.python.org/lib/lib.html) >> endobj 132 0 obj << /Type /Annot /Border [0 0 0] /Rect [72 362.214 215.391 371.165] /S /URI /URI (http://www.red-dove.com/python\unhbox \voidb@x \penalty \@M \hskip \z@skip \global \let \OT1\textunderscore \unhbox \voidb@x \kern .06em\vbox {\hrule width.3em}\OT1\textunderscore \discretionary {-}{}{}\penalty \@M \hskip \z@skip logging.html) >> endobj 46 0 obj << /D [128 0 R /XYZ 72 744.907 null] >> endobj 127 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F34 12 0 R /F41 60 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 135 0 obj << /Length 1682 /Filter /FlateDecode >> stream xڭXY6~p]++[>$HȢ9m +KEc mH}0c8ǹa0ν< qS{~ʏ멈VM Zoj^4kZƱѴ~-a^VJ9ȵlt5>ZO4lnWļ1Û?vE%Ad}@9z+P-m@0MlN_%-bmU4Jm[v-'A5,04eM1,P `фXp7uw^Z:ڨۢt :HH.f?tjD1 U`DTsƼ-\ren=O0ˀc{Y0y<`+S0 ZZR5XˎŎM{}'k9VZq1?B- MU?.oGJ mД%M 6qjѕҘ|сu&qw[:~c,0]۫i{ E4r ~!{%6j`ZI*lҨIlZabmTe2 3ۤrVm_s~32u>2MF+ ګ&UwwFj%En2uIɽn^091`/Ӕ&K,ͽ3(hc7Gy"3 L 4'1:>9 L)HJޙ(a^f\nHL7aOIt^5LKOu/y. [ ۰3# ˟# (7Jem4YԾ{B2ꖨ8^%&yD"8c"GL%Wcg3}MD߁q\q?u pz N|L? 89-N^`Mb_Bzw7Ql \13uŃXZR"R}=7zֵY@h_ W "χFfK?2p J*˰h(Oon~P~^g- h|֣(x\ޏܳp{`lAP,@V6{endstream endobj 134 0 obj << /Type /Page /Contents 135 0 R /Resources 133 0 R /MediaBox [0 0 612 792] /Parent 118 0 R >> endobj 136 0 obj << /D [134 0 R /XYZ 72 744.907 null] >> endobj 133 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F29 5 0 R /F34 12 0 R >> /ProcSet [ /PDF /Text ] >> endobj 139 0 obj << /Length 2449 /Filter /FlateDecode >> stream xڝk~"p}o\ H" (i}Uᐻ\힛y?WX:׉Sj6[?^VwzM%}ڮɫ=Mv{67~ɢW4w zwguOkQwԪ߼HNG(dI䯹R7,laE,Mbc_u%ci= OX&V$,C&AY<'Xh HX`G)lFH1n;P$!}׋9ZtgٍDiо.aåTS]T@󖾒H\U7WPXS ;iHc_;Կ!B E [miH4{V4d_ȶ%X}Y!, &Yxs"<jPڪ%#Q5(`>p* [}"ulQcͧ1 ԗ&0<0P>GoՊQسdXc&5rOgsf`z)\?exKӱşU ŌK@$=], U*Hԉ9N;l66:> 4.}ЌHHxFM%zÄlԏ@uԑp}Wy8t6t0E#~Guk8ŐM_0jp…,(uw]:4"M,me/t*fKo:*(4 ){ {+[HoL_)D`fEP7뢠]##(.+3ЙׁCf 9Qhƾrj Y\zxdOs$ \NlyK7x5誖((M}ߚ/c~Z@mgz4 TyXh Ѧ!%YCHV$ y/! ̽5.h4V6h@]oTPr)k7vܢzC8Xqph%Cdž²$!iU)M70#:vhPDT. X[a_P!@aq2~Ծ.l@" v#uV/Jf(v׹lg<&wlm(0|AĠ7;Jy5Ԩ νں;5r5};XYO% NTO?rttFL#`~@%l #$.Oy03ݹ<# ҝܬ~JzkxGO` {㖾>]O]. '.Dž`3v%~Ao\8NmgX/ 'd?t=ȢP 9+KY 7`5t)o ¬Si/7KᘉDxwb!LC؈җ,{eO/k x(pP&$2*öu k5~% #DGЁZ-Q/Yէ|}X+*ϟ`̙Ȳu Ԍ՛_[?tp 0(R|c42xAL#v>F2(_*endstream endobj 138 0 obj << /Type /Page /Contents 139 0 R /Resources 137 0 R /MediaBox [0 0 612 792] /Parent 118 0 R >> endobj 47 0 obj << /D [138 0 R /XYZ 72 744.907 null] >> endobj 98 0 obj << /D [138 0 R /XYZ 72 220.397 null] >> endobj 137 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F34 12 0 R /F29 5 0 R /F37 22 0 R /F41 60 0 R >> /ProcSet [ /PDF /Text ] >> endobj 142 0 obj << /Length 3607 /Filter /FlateDecode >> stream xڭZK6e70fco X 6 -w+%G3~X$%Yt?&sh*U_=ȷw7_SrƛbKv |Q4bIK 6-SktpIbk~XT6#MD^& W0FT 5Xb}g=qe$Q[p>n9f pa;K. *mA6;SWCI_{H{USD>Zpm w()-z3O܎ { Fa4{nv㺁% g]paoq1}7{*>h|u~6z1*iKݕ .9'tW3)v͎V $`}\tpKVL ]rT 5Ld=('g)F}[Tk5B@R3= G.gx}ơX|"v펳bdP}#"02wQ]Yf0'3xyAI]WܓOqkOUt%5MϧSہgG_9@w\[hArF[" Rq;G+{BnP:'νH0g!n6 mغg] "TDzOhr_]1/_gߊ3ȦM O ]f&2f&swt؁;E im/mq$LkLMJGCvHvQO ,:s\p14x!h\BoC$ĸK#=׎OR lEvp1N|t0QnAQ2S~Fi:42R#b27FL v"W%)L2.8˴4clAy0%G@c|.A.3*\kDlgsJbMOvḿV_W#jR>s? 9 9үDk2]|KdVo EK U^f1Qu[GvJ[Lt;7kɧbGq_H23&J+Ԓe76ie3j6IoY *̋RtLOFP;oU\0^A!;v𕯿tC8u-2":5 Ga{ H 9.C9L;x*W&|MӶ-`oT&'OAD |[VlWB9uOf|fBGL GnwT?U LLa.B[Fgsҿo4?}jWη+R l? |D8e};-`C<2|p/f[1eIw}5~pJd%|׭ŀօJdni̓uW)`ˢxrnT1x`+BC)6R߈oLJj@`P?VwI)PjvLt>nKW )Kstg5oh kaL /C6ryNNUC@:h@HdF ',*j|3_^uVRJ# ݵ3dus8$cK\"O]}Ƣ HQZzw=nS0a6fJ]S 0©.b|δ/ h 򮺯"Rn)RId/V!k+X228YTA~ ``U7c(O5R:X6+;END(xNЦ /m%ǝL\1̄up eH(0bi~jʼ8ɆHB!r2 :yQ&af Yt'c5c='4; d x&]ĩ82 7l:$-̣1N &uXlw&4r&,@C+FeSV9K|KB͋n,2 bә QO&dN1@ C)p(Il)c"tOƼ5慀Sr>HᢥC"vE_T{az:{{PsNow^ʯОzc M[ guNp\u<]vgunR>[0TL|,ޫi]1ߍegiT`u81s>> endobj 143 0 obj << /D [141 0 R /XYZ 72 744.907 null] >> endobj 140 0 obj << /Font << /F32 9 0 R /F41 60 0 R /F37 22 0 R /F38 26 0 R /F34 12 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 147 0 obj << /Length 1538 /Filter /FlateDecode >> stream xڭXK6W23$R mS9hIZD!i˖9|׋$Ib(`Bmt6Ow )ID,(lz]}6,Tv®k;w::3A,'#9^ZqjJ]+Z᣶oEhY6}Ju{m>Yݺ,v);ڴp>R5ޘҹqL$Zn6es?3NHzl$\x]y7h`= HEFDhk $er/aDIl֐P.DnvФ'7caT[)YBsFuZy>vp:X쏫 Ԅ83N8;lq}MrG ɩtM#5q }|khkD(rFoBVvCzi^UHYfZe" ԡ'99I^۷m,lWav1ap,`$iPJr(0r{@[0Y6~;$ىnPkV!C&ӌZ4HA39x$&nI a鷤=kUn;/VJAZB.23/g,a, e0)GfZ\mTQ (YyHܢk *MĥtPGQc-;U{GEWnC3 N))\M!g Si;mz4$5܆9G9Aaq̖hQ{<:Ұilڸ~deb'פS 3<$@?aGg21 PŇ{#p\ޠA >ecA {jjc]M+wb X\7n˥5YA$Egݧ48c(G>&ʱe<1j{m @HOxxE/&LW^u-3jly5ݫ~.1`QF¥R7Chu@_Vj=}/a9%4.%^P`  ~[8~rcگ+s {aݎ(ߊx9bGo>CO0?xc̾0@$^T'Li=7(VNb-$<Vhm&bae63 > endobj 148 0 obj << /Type /Annot /Border [0 0 0] /Rect [155.448 583.899 197.291 592.85] /Subtype /Link /A << /S /GoTo /D (label-module-ZConfig) >> >> endobj 48 0 obj << /D [146 0 R /XYZ 72 744.907 null] >> endobj 149 0 obj << /D [146 0 R /XYZ 72 87.553 null] >> endobj 145 0 obj << /Font << /F34 12 0 R /F37 22 0 R /F32 9 0 R /F41 60 0 R /F38 26 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 152 0 obj << /Length 3424 /Filter /FlateDecode >> stream xڽZKϯQ18u'>CB#f%Rcg_nt$$j5r"F_7oUIzڭRJEQz*Z?旧`L6idwKP^x O86p0m6"^Jg{=9VMM}bPivpleFO^Z˼T]ߞiKnD>čB )<hHkڰmvԴ)|-ZRrA"l믈h=MKm>  qL]p]-CPk&{/0$EgFұd['\4=1鲬p8tg$ aJ$R)_8 -Ht $p(]<(,c\u4M詯kn=n]R #z@6C|ûOUݛt0xULO ev|(bw .BۍiR@ah&eRRg4z|s6 N(@`#@_D ,e[Fk/U?q#c<»(m;SWd%Ĵh.0Q2T龌uç"Ӱi9FCAwwr,, T$#mk\HQfg6@C˪j62YxEy#õۛ;Z+* ^j,jpZ!<z͝Pևԡ_6 AN@jڶhZ::N pQ|L* 2pPAy񐟮Gi[{fz^_Aϡs wgUS5Qv?PȒ@]6\}@%^n[Ry|a3qDIGG>bIAU칱k3ՠBa}*0_Y}ӁZ38?o2 76Nf1A8sJ;Om3UvNu=.-o ")9 d##`T?vvPJN>Pw'ST Ld'8QF7" ]0RѶ̞8!liz/b/@(T>!x K]W(@ ^3ZBSAt4 )=X ~1!/mdzYVa0)9"dhֳ࢜E }"ssZ# cN/!hMSҙMώfm]ۚ U)΅ XyΦ5tD7Љ2VoyIfљN t@:= eq6G m#dгxxy}: H'?}-PNP%7 `gڜo4jTH}G%l!hze7MP<.; $|c|}*!G_8 _g mtYŐ$=رjd^Ɔd7KHS?|Û 9<@ԯYbE8;(#RD7M=:(a< HǫYJ Y˳y{˔3@eO +#lh='r@7P6l#&RȊ9 5 K&O .J%Yj˙j>]qʲ&!\Ѡou ^#3XhӬ+bm݂7Q=g<D  IbNh,9n( u0.]T9҈bRhfK$[O 0A i*Cro*fd* EA[g`M`2sW1ڡQAbۖ׆wθ ]ht 4Jn7AQU58 s6fH ubٛ kXkd kTPNem//E;wJI_cKhACGo-6uTr`ǙpFr KY`UߥX=0z1 hYX+(C<(.sPl% D0 \si}'[贲EJcݿ#*l#1SY ,um5(H~*d F%˱_o<~l~6MSSնF R&v4tXcJ -ďx!U?7]*{x)B}NW̕v~gj0 3n [Vφoe~}ĭt/QdZ֣<-4"H1w[9`+WR>"FMpN9qEwAXAfرi1g sqeCf Kw -^JM:ppnм1\A9QE؂* yCx=&#>[hGI7N:]7hkC`OFw9@t3 z/H^=LW]#45%R'2PZCj%a>x?f [.]ୋWtl@5G6}hOYH3N"/mUGm3X'c+Q$8#f8Om@yjg|IRӬ> 7pNB'קӡºC@c!(~s?eߢ15 "D^!g _^I*7o718̋C{XY0m{5~,`t7C_ݟ8Q\qO peA3T7N'&?S1>FwLD3O8p&N_{@+˒wUtk `檱|ywHĺ^ ZŤPHN+A(+95lh'ۡEsP54˝A2}kn_'\ 4VA𯇟~ WC!^:_TqF[GU8oJ0 !.Kendstream endobj 151 0 obj << /Type /Page /Contents 152 0 R /Resources 150 0 R /MediaBox [0 0 612 792] /Parent 144 0 R >> endobj 49 0 obj << /D [151 0 R /XYZ 72 744.907 null] >> endobj 153 0 obj << /D [151 0 R /XYZ 72 140.582 null] >> endobj 150 0 obj << /Font << /F32 9 0 R /F38 26 0 R /F41 60 0 R /F34 12 0 R /F37 22 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 156 0 obj << /Length 2857 /Filter /FlateDecode >> stream xڭZ[w~#c oٞ9i]M(D$EH<"`f7Cy2Ĩ|u[E +(^o6~?v{xOoIr&V=TS]ȞM\榄ZJ /=뵍 4Da¯}8a:Lt OU)JM(3]5 槔fr RD msI,bo褒yN$IKykA5\֌# -'& PYJM.Ze:ĉ`ҋXm# Z1kp/l(vͥJPgsnA{e?0wNwߺiM;^nC u9T~ʶ@VI 9..a)Sr: gW8ֹaܛCٙNio_uv{Pm8Qfþ޸7?} .,7\6RhFhP4=$1: QAKLNvc08\o&ID/+z e.>-FWA!@Vsܻ u:}l!(G,&)bL=۾OcYw+ej7w/ u,}+1N#X{IHQ"k"gE.S@1̺#nw7}hڱG47}|M\ [7~xq-$,lhZ"IzZ۠#fm9Rlx9V8* hy+ 5*Ͼ=ĆrB{{9I,\Q`›6Q7bKvA 1?6%tVN3,h?Oj맶`SftaDf- L+&ĀQ| f.8)VuCR:8E˵ 08(³Ǫr'f-THp]mKFIN g4G SS_8 o|K}fRnS8.~=6f>ڞ!9UqE տjyXINWS@1aź (*/] }^kZ@K$FTXQȴ;']zɜ)z='NI1a].Յ5 X;ΞL~x$,hUDco]5j3*PcuU`G)-OC P5UOD)4Oe#ǘIqԯ\FT F6+b ^ ",}H:`<,9i]+|@lQT󞈃v{w7cehZD@G4F̐u*ǖY8QK xJ"&ݭypIq~σKyn4]*RC/Z@{ 2G ?: )g:d[3mQ0 "D">=7eVYoţoo%$B]0>3稹3`ّ0CT1e֏@r=n6=S]5o CmbuDy 8E fn}*Ytp{XCrL>&)/ (ˤ/rS؂4.dX)]H^Պ_  C>f{T4H]Nc4Up͗ƀ=kxyȕ/MjfEM=rKy<)o;uK RRα!vDZKP7TĖ-rmͿZv7I_6P x|m+/}GO%=* gK17 d$um:AP*T8wx}昷Pq=^8+ <<y/e PXk_^~$J()jkcTmؚ֪R6BC8A:%뱔~+Bŀ;7ٙ-U7H__3H>e+t&e.7,?,:q,:q᫷mK/ڳuW輟o}\B#\l.JGliBqr=)_U+l {r-C]G*mmםܛO7 tV ,9旯to(db cJ1Íxo5XOB>_MqʠsnәIĮ/oV X0br]3sq=`Y`LY3endstream endobj 155 0 obj << /Type /Page /Contents 156 0 R /Resources 154 0 R /MediaBox [0 0 612 792] /Parent 144 0 R >> endobj 50 0 obj << /D [155 0 R /XYZ 72 744.907 null] >> endobj 154 0 obj << /Font << /F38 26 0 R /F32 9 0 R /F41 60 0 R /F34 12 0 R /F37 22 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 159 0 obj << /Length 3056 /Filter /FlateDecode >> stream xڽَ}BD޾H6'{c^,3  CR#byhyx<lԌEl6j=7L!aMDc"y>o}By0!sS{VYYԹdBFyHmn*^g v~=G˳IJA-7 U@b%gA/`DŽ3F Рh h.`VEz|* Ni1H7+{;iRwPm"1ҀQD0㴩q& )ܐu7@vC"xv:;EFT?Ec[{:bEt|dc\Ӟ+ܥHdQ倬QŗD y& Ήdw{m+e}u]M0q]g=Ucq5bk.'$ Ù|®oRZb0ܝ2IYf7 "[8ȯ+!Cyo2^Gz~:'`_TRz3誟s (>V+1z?'Fhؙ;CזiN0A f9mq[fj g~^j!Q*z2;ؽ;R6{ӻ]uBavFX,EI? u*ھRHMQ4޴ÁQ,dTڣb/g$݊F@ب.]MgO8$8,QӀ?)$1sHf !.uo_vJ:~l\> ȳDDxopW6R@yN`iOf8hh*Eu}ӯh>hղGj0rô8$a") A`mQD4 k{LP3Q~al.'3cdk-`V z̖ܭkʡX_: D($F}_Y9M LH%:9x.]aAnKL)޵m0_%1A70i1(;RE9H3 ,m;Z\ \}J7_ ɴNWvdZ^\cebC !$x"@k>M|X(.^q> 8H4Y4r+!y?a{I e*5}}}A(G_F%IN>M%DM<'m%NyI+iHu9"&uR]0&,/ #1 J76!<˦} boFA 'A>qQQhi6#A " Yل벤C(_∓8dT@o^m\5yMۉW6+mV*8.m*q(NuñHfZ[B1?Q8ijz+J\ԡ:4u ;u{@2n˕ 5WV!7ń/}va .P"#ܖ ̷QK$58{:k?bO쇹E=}} vFbxJ.i$B`^& 1U$ȕPnχ4W+Q1 "jj(htyMϫ&+T[ Աy+1Bu6]bF۹-zOVta}R@-vN6{E ey 7x[Db޽\:?8ʫ14 9a9mWket%[Io2PO/J 殯tC* M8ioDԔ=CODuE::ejGAju}Cg! i{yv5G e7XNpˎ (z zV9jُ2 ~% -E-u>Yt%9f"Jyw|M^6ѧIˈ(98k:T H5ݺX26a\} [e|EnzOx u]po!_I $g6nV_ <X]82pfG R"1ޣ- [wI to$*k2pt endstream endobj 158 0 obj << /Type /Page /Contents 159 0 R /Resources 157 0 R /MediaBox [0 0 612 792] /Parent 144 0 R /Annots [ 161 0 R 162 0 R ] >> endobj 161 0 obj << /Type /Annot /Border [0 0 0] /Rect [399.528 679.334 483.214 688.286] /Subtype /Link /A << /S /GoTo /D (label-module-ZConfig.loader) >> >> endobj 162 0 obj << /Type /Annot /Border [0 0 0] /Rect [112.158 554.73 154.001 563.681] /Subtype /Link /A << /S /GoTo /D (label-module-ZConfig) >> >> endobj 51 0 obj << /D [158 0 R /XYZ 72 744.907 null] >> endobj 160 0 obj << /D [158 0 R /XYZ 72 696.124 null] >> endobj 79 0 obj << /D [158 0 R /XYZ 72 468.225 null] >> endobj 157 0 obj << /Font << /F29 5 0 R /F38 26 0 R /F32 9 0 R /F34 12 0 R /F37 22 0 R /F41 60 0 R /F14 165 0 R >> /ProcSet [ /PDF /Text ] >> endobj 168 0 obj << /Length 1331 /Filter /FlateDecode >> stream xڵWYF~ׯ Vr!$Jb%d#Ҭ#~`aK¡VFZ׮D3 l$MSF)4ԥ*1H ZOTU:~iD<d)ɨlZ\ŋ=y\fW~jP,S6:ߓ՝d.i)HOݖ&P4uK##Ced rr̈́L,nvoJ,Exϻ'IӺ{@zVm}E膪@ 1>#W=d?PL@,CZX5)=b.3g5~ L=_|M*`tfΜfcRqˋՒ_o781 4+iU~_^U[xMh'zB0O^q:Ǯh5MG<9М%~t(HX@XGeG-H*㧍AKCFeY\d<&e ȱ]C(ү6T;/-,J,n[6~^b'CwqY-Qqg|)hXpؓ,0ByW0, EPM6mOzW{O4ml3 naNjT u=rMѨޖ4 Uٌ,~qxfnJ~5TaL1 !v/}!Y_V:]lRuyT_/"Q@~#u=X?&a??E(7Q &6̎C:Q%} }E IŽ*騊1UE;I?;ί{2J#t>ܧ4c2-`5endstream endobj 167 0 obj << /Type /Page /Contents 168 0 R /Resources 166 0 R /MediaBox [0 0 612 792] /Parent 144 0 R >> endobj 63 0 obj << /D [167 0 R /XYZ 72 744.907 null] >> endobj 169 0 obj << /D [167 0 R /XYZ 319.828 533.207 null] >> endobj 166 0 obj << /Font << /F29 5 0 R /F38 26 0 R /F32 9 0 R >> /ProcSet [ /PDF /Text ] >> endobj 172 0 obj << /Length 682 /Filter /FlateDecode >> stream xWn0)& $ UҕI钬? ΆGM>lIHHٴu7 |cnK;2 &"A \mFCgVw'!j] E޽jV ruЙDumi6ʆ==rmA>M$Y0({JrKBQu<POWkoU)#>D`Y%:'aipþM[eEG$xۼGo*;6N8Oɰj D;4bi6Ath{5ns򐲄x-6^+_َ`ggr:aSFPie^e7,V<ʗ6UfXk(c=%%Ìܺ ~;z?IOhrH4  4}L:u\^b9^5( BDW+]Q+.7Ƌq\:`;V&@ίji hbO%^<)w  tj*Fʯ'ʭrP9"9l{tq11?a !Vl:~_Sendstream endobj 171 0 obj << /Type /Page /Contents 172 0 R /Resources 170 0 R /MediaBox [0 0 612 792] /Parent 174 0 R >> endobj 173 0 obj << /D [171 0 R /XYZ 72 744.907 null] >> endobj 170 0 obj << /Font << /F38 26 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 175 0 obj << /S /GoTo /D (page002) >> endobj 177 0 obj (1 Introduction ) endobj 178 0 obj << /S /GoTo /D (page002) >> endobj 180 0 obj (2 Configuration Syntax ) endobj 181 0 obj << /S /GoTo /D (page003) >> endobj 183 0 obj (2.1 Extending the Configuration Schema) endobj 184 0 obj << /S /GoTo /D (page004) >> endobj 186 0 obj (2.2 Textual Substitution in Values) endobj 187 0 obj << /S /GoTo /D (page005) >> endobj 189 0 obj (3 Writing Configuration Schema ) endobj 190 0 obj << /S /GoTo /D (page005) >> endobj 192 0 obj (3.1 Schema Elements ) endobj 193 0 obj << /S /GoTo /D (page010) >> endobj 195 0 obj (3.2 Schema Components ) endobj 196 0 obj << /S /GoTo /D (page011) >> endobj 198 0 obj (4 Standard ZConfig Datatypes) endobj 199 0 obj << /S /GoTo /D (page012) >> endobj 201 0 obj (5 Standard ZConfig Schema Components ) endobj 202 0 obj << /S /GoTo /D (page013) >> endobj 204 0 obj (5.1 ZConfig.components.basic) endobj 205 0 obj << /S /GoTo /D (page013) >> endobj 207 0 obj (The Mapping Section Type ) endobj 208 0 obj << /S /GoTo /D (page014) >> endobj 210 0 obj (5.2 ZConfig.components.logger) endobj 211 0 obj << /S /GoTo /D (page016) >> endobj 213 0 obj (6 Using Components to Extend Schema) endobj 214 0 obj << /S /GoTo /D (page018) >> endobj 216 0 obj (7 ZConfig --- Basic configuration support) endobj 217 0 obj << /S /GoTo /D (page020) >> endobj 219 0 obj (7.1 Basic Usage) endobj 220 0 obj << /S /GoTo /D (page020) >> endobj 222 0 obj (8 ZConfig.datatypes --- Default data type registry) endobj 223 0 obj << /S /GoTo /D (page021) >> endobj 225 0 obj (9 ZConfig.loader --- Resource loading support) endobj 226 0 obj << /S /GoTo /D (page022) >> endobj 228 0 obj (9.1 Loader Objects) endobj 229 0 obj << /S /GoTo /D (page023) >> endobj 231 0 obj (10 ZConfig.cmdline --- Command-line override support) endobj 232 0 obj << /S /GoTo /D (page023) >> endobj 234 0 obj (11 ZConfig.substitution --- String substitution) endobj 235 0 obj << /S /GoTo /D (page024) >> endobj 237 0 obj (11.1 Examples) endobj 238 0 obj << /S /GoTo /D (page024) >> endobj 240 0 obj (A Schema Document Type Definition ) endobj 243 0 obj << /Length 263 /Filter /FlateDecode >> stream xEPAN0r1k'q UB =TQ#5 P%ϦfvCjJg 7Py&UCPbZW|>""_c4LӴBBRSӇn,]1s!%CQ gg<( jP3C: Q*eWgKgGo2jڦC[3-җPE[)Re BvKDaNmǟ ?+ֈ:6ǦU4RAendstream endobj 242 0 obj << /Type /Page /Contents 243 0 R /Resources 241 0 R /MediaBox [0 0 612 792] /Parent 174 0 R >> endobj 244 0 obj << /D [242 0 R /XYZ 72 744.907 null] >> endobj 241 0 obj << /Font << /F38 26 0 R /F29 5 0 R >> /ProcSet [ /PDF /Text ] >> endobj 245 0 obj << /Type /Encoding /Differences [ 0 /minus/periodcentered/multiply/asteriskmath/divide/diamondmath/plusminus/minusplus/circleplus/circleminus/circlemultiply/circledivide/circledot/circlecopyrt/openbullet/bullet/equivasymptotic/equivalence/reflexsubset/reflexsuperset/lessequal/greaterequal/precedesequal/followsequal/similar/approxequal/propersubset/propersuperset/lessmuch/greatermuch/precedes/follows/arrowleft/arrowright/arrowup/arrowdown/arrowboth/arrownortheast/arrowsoutheast/similarequal/arrowdblleft/arrowdblright/arrowdblup/arrowdbldown/arrowdblboth/arrownorthwest/arrowsouthwest/proportional/prime/infinity/element/owner/triangle/triangleinv/negationslash/mapsto/universal/existential/logicalnot/emptyset/Rfractur/Ifractur/latticetop/perpendicular/aleph/A/B/C/D/E/F/G/H/I/J/K/L/M/N/O/P/Q/R/S/T/U/V/W/X/Y/Z/union/intersection/unionmulti/logicaland/logicalor/turnstileleft/turnstileright/floorleft/floorright/ceilingleft/ceilingright/braceleft/braceright/angbracketleft/angbracketright/bar/bardbl/arrowbothv/arrowdblbothv/backslash/wreathproduct/radical/coproduct/nabla/integral/unionsq/intersectionsq/subsetsqequal/supersetsqequal/section/dagger/daggerdbl/paragraph/club/diamond/heart/spade/arrowleft 129/.notdef 161/minus/periodcentered/multiply/asteriskmath/divide/diamondmath/plusminus/minusplus/circleplus/circleminus 171/.notdef 173/circlemultiply/circledivide/circledot/circlecopyrt/openbullet/bullet/equivasymptotic/equivalence/reflexsubset/reflexsuperset/lessequal/greaterequal/precedesequal/followsequal/similar/approxequal/propersubset/propersuperset/lessmuch/greatermuch/precedes/follows/arrowleft/spade 197/.notdef] >> endobj 164 0 obj << /Length1 776 /Length2 601 /Length3 532 /Length 1157 /Filter /FlateDecode >> stream xSU uLOJu+53Rp 44P03RUu.JM,sI,IR04Tp,MW04U002225RUp/,L(Qp)2WpM-LNSM,HZRQZZTeh\ǥrg^Z9D8&UZT tБ @'T*qJB7ܭ4'/1d<80s3s**s JKR|SRЕB盚Y.Y옗khg`l ,vˬHM ,IPHK)N楠;|`EzkC,WRY`P "P*ʬP6300*B+2׼̼t#S3ĢJ.` L 2RR+R+./jQMBZ~(Z OJL.I,qaz)ZRA hi-мTOS #KRJP0~Z&0pSS+Rn^Onɚme X9X&u3SjM*,y־% mHx֮"&4׻,^5+Åa3>_xV/'x楼pZkBZAo`(&^yeO;M~wgx[4;/ߞpŦ]%R쏮x#ۡԮH𲊯]uO_xݻk?'~Yw/n9H}̉NhMI7g̴YOB77l]jiݏesOuySU|PZ=m3"/}nQCa+7;e[۴}ş59s yL}žn 1q40aw>xO1OzV.񗱦EwZziiůvhܨyUD4:޴Iswgu]|zB0DgI˾̰.JR Xl,JoVJbL,_Y[>uCV>B2ǭEܸ8Ugن[˘d_ӻn}kԀaa@rNjbQI~nbQ6deQendstream endobj 165 0 obj << /Type /Font /Subtype /Type1 /Encoding 245 0 R /FirstChar 110 /LastChar 110 /Widths 246 0 R /BaseFont /MPVYIG+CMSY10 /FontDescriptor 163 0 R >> endobj 163 0 obj << /Ascent 750 /CapHeight 683 /Descent -194 /FontName /MPVYIG+CMSY10 /ItalicAngle -14 /StemV 85 /XHeight 431 /FontBBox [-29 -960 1116 775] /Flags 4 /CharSet (/backslash) /FontFile 164 0 R >> endobj 246 0 obj [500 ] endobj 108 0 obj << /Length1 797 /Length2 691 /Length3 532 /Length 1263 /Filter /FlateDecode >> stream xRkPWVEydB_:$ ݰ`RETZ+eXT*"2G} |*ՂbJR+vC8şW{s=|gz:PI0e@Lxr(AX& a( (:* B`DF+G<@iE y=4N5KPʶTtH $E ( 2Zml|C$~I X$YjBV8 N SCc47m!M` F39ϒcRBxHRfP4iP2:@L*Fge,LxB8m}ȐC ,D%$*SpF{] q@( DahY A`snEPA*H&1@1$h K [LвbB$Ze9l()[ Z;%#"X *RI!@2[Pifd(`3A|[B~j-%dHsKN)k^Đϰ>VL|(:ia{TGS4* l4'KQW> endobj 107 0 obj << /Ascent 750 /CapHeight 683 /Descent -194 /FontName /OZDPTA+CMSY5 /ItalicAngle -14 /StemV 101 /XHeight 431 /FontBBox [21 -944 1448 791] /Flags 4 /CharSet (/floorright/ceilingleft) /FontFile 108 0 R >> endobj 247 0 obj [667 667 ] endobj 248 0 obj << /Type /Encoding /Differences [ 0 /Gamma/Delta/Theta/Lambda/Xi/Pi/Sigma/Upsilon/Phi/Psi/Omega/alpha/beta/gamma/delta/epsilon1/zeta/eta/theta/iota/kappa/lambda/mu/nu/xi/pi/rho/sigma/tau/upsilon/phi/chi/psi/omega/epsilon/theta1/pi1/rho1/sigma1/phi1/arrowlefttophalf/arrowleftbothalf/arrowrighttophalf/arrowrightbothalf/arrowhookleft/arrowhookright/triangleright/triangleleft/zerooldstyle/oneoldstyle/twooldstyle/threeoldstyle/fouroldstyle/fiveoldstyle/sixoldstyle/sevenoldstyle/eightoldstyle/nineoldstyle/period/comma/less/slash/greater/star/partialdiff/A/B/C/D/E/F/G/H/I/J/K/L/M/N/O/P/Q/R/S/T/U/V/W/X/Y/Z/flat/natural/sharp/slurbelow/slurabove/lscript/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z/dotlessi/dotlessj/weierstrass/vector/tie/psi 129/.notdef 160/space/Gamma/Delta/Theta/Lambda/Xi/Pi/Sigma/Upsilon/Phi/Psi 171/.notdef 173/Omega/alpha/beta/gamma/delta/epsilon1/zeta/eta/theta/iota/kappa/lambda/mu/nu/xi/pi/rho/sigma/tau/upsilon/phi/chi/psi/tie 197/.notdef] >> endobj 77 0 obj << /Length1 793 /Length2 1151 /Length3 532 /Length 1727 /Filter /FlateDecode >> stream xRiTS2j=,<2a  s(Ib ^@"Tʲ.EEXWgUp*zog|g3¥"ar @"s؀:4#e$c2@ VQl!i k $UIIT0eH%je(r&L BQnFXgD& 1i9 6FcMjc :]*&2(QeJ+DB8j+h58 VjST$L 6'!D2TL8h`(!Jct%S:Xk~کd Hm: b1eh@fs("ߝ5 8`2iiQ8 X` p(gr'h,ȘD\J=ʽIO5:w.pV^c!հ8^|/*WSCE.V 04M\}UwtX?SzopuH_m~) )xӢlNkkvp/P>q4I~=*tҿ#) G.8Vi']ᆶV/vo)7eKuΙO^{Ps2s<1#_۰kny3SL=Rg]o1 sy3g#v_8W/I͆=A3A%なG?_j-:MHK9yr]@I%ͳAN ڈM@xGS@y.ێ71`KJ M#G<ZmY rau).-EfGK;^V]},Ǣ"SQwrԻmzܵJcҦ8EU ;چĸ.{vzwb#voz֊E;ӄVzzЧol;ؽc;#-٪}bZJ🱕\[EƹL0*Ur0] oO1CKz`qhVw)O'rX.F͸3sIP~c.vexh uΟxH7 ҽ)YZM׼6wg>~qiN_F|pjڔbN`OѣYv^dw5Ln^%mOWÉc" ,fjܧߤ.&]|4b'1T.#W7< ?_M`Er$ibendstream endobj 78 0 obj << /Type /Font /Subtype /Type1 /Encoding 248 0 R /FirstChar 60 /LastChar 62 /Widths 249 0 R /BaseFont /QZODZZ+CMMI10 /FontDescriptor 76 0 R >> endobj 76 0 obj << /Ascent 694 /CapHeight 683 /Descent -194 /FontName /QZODZZ+CMMI10 /ItalicAngle -14 /StemV 72 /XHeight 431 /FontBBox [-32 -250 1048 750] /Flags 4 /CharSet (/less/greater) /FontFile 77 0 R >> endobj 249 0 obj [778 0 778 ] endobj 250 0 obj << /Type /Encoding /Differences [ 0 /.notdef 1/dotaccent/fi/fl/fraction/hungarumlaut/Lslash/lslash/ogonek/ring 10/.notdef 11/breve/minus 13/.notdef 14/Zcaron/zcaron/caron/dotlessi/dotlessj/ff/ffi/ffl 22/.notdef 30/grave/quotesingle/space/exclam/quotedbl/numbersign/dollar/percent/ampersand/quoteright/parenleft/parenright/asterisk/plus/comma/hyphen/period/slash/zero/one/two/three/four/five/six/seven/eight/nine/colon/semicolon/less/equal/greater/question/at/A/B/C/D/E/F/G/H/I/J/K/L/M/N/O/P/Q/R/S/T/U/V/W/X/Y/Z/bracketleft/backslash/bracketright/asciicircum/underscore/quoteleft/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z/braceleft/bar/braceright/asciitilde 127/.notdef 128/Euro 129/.notdef 130/quotesinglbase/florin/quotedblbase/ellipsis/dagger/daggerdbl/circumflex/perthousand/Scaron/guilsinglleft/OE 141/.notdef 147/quotedblleft/quotedblright/bullet/endash/emdash/tilde/trademark/scaron/guilsinglright/oe 157/.notdef 159/Ydieresis 160/.notdef 161/exclamdown/cent/sterling/currency/yen/brokenbar/section/dieresis/copyright/ordfeminine/guillemotleft/logicalnot/hyphen/registered/macron/degree/plusminus/twosuperior/threesuperior/acute/mu/paragraph/periodcentered/cedilla/onesuperior/ordmasculine/guillemotright/onequarter/onehalf/threequarters/questiondown/Agrave/Aacute/Acircumflex/Atilde/Adieresis/Aring/AE/Ccedilla/Egrave/Eacute/Ecircumflex/Edieresis/Igrave/Iacute/Icircumflex/Idieresis/Eth/Ntilde/Ograve/Oacute/Ocircumflex/Otilde/Odieresis/multiply/Oslash/Ugrave/Uacute/Ucircumflex/Udieresis/Yacute/Thorn/germandbls/agrave/aacute/acircumflex/atilde/adieresis/aring/ae/ccedilla/egrave/eacute/ecircumflex/edieresis/igrave/iacute/icircumflex/idieresis/eth/ntilde/ograve/oacute/ocircumflex/otilde/odieresis/divide/oslash/ugrave/uacute/ucircumflex/udieresis/yacute/thorn/ydieresis] >> endobj 60 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Times-Italic >> endobj 26 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Courier >> endobj 22 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Courier-Bold >> endobj 12 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Times-Bold >> endobj 9 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Times-Roman >> endobj 7 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Helvetica-Oblique >> endobj 5 0 obj << /Type /Font /Subtype /Type1 /Encoding 250 0 R /BaseFont /Helvetica >> endobj 36 0 obj << /Type /Pages /Count 6 /Parent 251 0 R /Kids [2 0 R 53 0 R 65 0 R 68 0 R 71 0 R 81 0 R] >> endobj 88 0 obj << /Type /Pages /Count 6 /Parent 251 0 R /Kids [85 0 R 90 0 R 94 0 R 100 0 R 104 0 R 111 0 R] >> endobj 118 0 obj << /Type /Pages /Count 6 /Parent 251 0 R /Kids [115 0 R 120 0 R 124 0 R 128 0 R 134 0 R 138 0 R] >> endobj 144 0 obj << /Type /Pages /Count 6 /Parent 251 0 R /Kids [141 0 R 146 0 R 151 0 R 155 0 R 158 0 R 167 0 R] >> endobj 174 0 obj << /Type /Pages /Count 2 /Parent 251 0 R /Kids [171 0 R 242 0 R] >> endobj 251 0 obj << /Type /Pages /Count 26 /Kids [36 0 R 88 0 R 118 0 R 144 0 R 174 0 R] >> endobj 252 0 obj << /Type /Outlines /First 176 0 R /Last 239 0 R /Count 12 >> endobj 239 0 obj << /Title 240 0 R /A 238 0 R /Parent 252 0 R /Prev 233 0 R >> endobj 236 0 obj << /Title 237 0 R /A 235 0 R /Parent 233 0 R >> endobj 233 0 obj << /Title 234 0 R /A 232 0 R /Parent 252 0 R /Prev 230 0 R /Next 239 0 R /First 236 0 R /Last 236 0 R /Count -1 >> endobj 230 0 obj << /Title 231 0 R /A 229 0 R /Parent 252 0 R /Prev 224 0 R /Next 233 0 R >> endobj 227 0 obj << /Title 228 0 R /A 226 0 R /Parent 224 0 R >> endobj 224 0 obj << /Title 225 0 R /A 223 0 R /Parent 252 0 R /Prev 221 0 R /Next 230 0 R /First 227 0 R /Last 227 0 R /Count -1 >> endobj 221 0 obj << /Title 222 0 R /A 220 0 R /Parent 252 0 R /Prev 215 0 R /Next 224 0 R >> endobj 218 0 obj << /Title 219 0 R /A 217 0 R /Parent 215 0 R >> endobj 215 0 obj << /Title 216 0 R /A 214 0 R /Parent 252 0 R /Prev 212 0 R /Next 221 0 R /First 218 0 R /Last 218 0 R /Count -1 >> endobj 212 0 obj << /Title 213 0 R /A 211 0 R /Parent 252 0 R /Prev 200 0 R /Next 215 0 R >> endobj 209 0 obj << /Title 210 0 R /A 208 0 R /Parent 200 0 R /Prev 203 0 R >> endobj 206 0 obj << /Title 207 0 R /A 205 0 R /Parent 203 0 R >> endobj 203 0 obj << /Title 204 0 R /A 202 0 R /Parent 200 0 R /Next 209 0 R /First 206 0 R /Last 206 0 R /Count -1 >> endobj 200 0 obj << /Title 201 0 R /A 199 0 R /Parent 252 0 R /Prev 197 0 R /Next 212 0 R /First 203 0 R /Last 209 0 R /Count -2 >> endobj 197 0 obj << /Title 198 0 R /A 196 0 R /Parent 252 0 R /Prev 188 0 R /Next 200 0 R >> endobj 194 0 obj << /Title 195 0 R /A 193 0 R /Parent 188 0 R /Prev 191 0 R >> endobj 191 0 obj << /Title 192 0 R /A 190 0 R /Parent 188 0 R /Next 194 0 R >> endobj 188 0 obj << /Title 189 0 R /A 187 0 R /Parent 252 0 R /Prev 179 0 R /Next 197 0 R /First 191 0 R /Last 194 0 R /Count -2 >> endobj 185 0 obj << /Title 186 0 R /A 184 0 R /Parent 179 0 R /Prev 182 0 R >> endobj 182 0 obj << /Title 183 0 R /A 181 0 R /Parent 179 0 R /Next 185 0 R >> endobj 179 0 obj << /Title 180 0 R /A 178 0 R /Parent 252 0 R /Prev 176 0 R /Next 188 0 R /First 182 0 R /Last 185 0 R /Count -2 >> endobj 176 0 obj << /Title 177 0 R /A 175 0 R /Parent 252 0 R /Next 179 0 R >> endobj 253 0 obj << /Names [(label-basic-mapping) 117 0 R (label-elements) 75 0 R (label-intro) 58 0 R (label-module-ZConfig) 98 0 R (label-module-ZConfig.cmdline) 160 0 R (label-module-ZConfig.datatypes) 149 0 R (label-module-ZConfig.loader) 153 0 R (label-module-ZConfig.substitution) 79 0 R (label-schema-components) 102 0 R (label-schema-dtd) 169 0 R (label-standard-components) 113 0 R (label-standard-datatypes) 106 0 R (label-syntax) 61 0 R (label-writing-schema) 74 0 R (page002) 37 0 R (page003) 38 0 R (page004) 39 0 R (page005) 40 0 R (page006) 83 0 R (page007) 87 0 R (page008) 92 0 R (page009) 96 0 R (page010) 41 0 R (page011) 42 0 R (page012) 43 0 R (page013) 44 0 R (page014) 45 0 R (page015) 126 0 R (page016) 46 0 R (page017) 136 0 R (page018) 47 0 R (page019) 143 0 R (page020) 48 0 R (page021) 49 0 R (page022) 50 0 R (page023) 51 0 R (page024) 63 0 R (page025) 173 0 R (page026) 244 0 R] /Limits [(label-basic-mapping) (page026)] >> endobj 254 0 obj << /Kids [253 0 R] >> endobj 255 0 obj << /Dests 254 0 R >> endobj 256 0 obj << /Type /Catalog /Pages 251 0 R /Outlines 252 0 R /Names 255 0 R /PageMode /UseOutlines /PTEX.Fullbanner (This is pdfTeX, Version 3.14159-1.10b) >> endobj 257 0 obj << /Producer (pdfTeX-1.10b) /Author (Zope Corporation) /Title (ZConfig Package Reference) /Creator (TeX) /CreationDate (D:20050518184800) >> endobj xref 0 258 0000000004 65535 f 0000004978 00000 n 0000002059 00000 n 0000000009 00000 n 0000000006 00000 f 0000088109 00000 n 0000000008 00000 f 0000088013 00000 n 0000000011 00000 f 0000087923 00000 n 0000002316 00000 n 0000000021 00000 f 0000087833 00000 n 0000002439 00000 n 0000002570 00000 n 0000002701 00000 n 0000002837 00000 n 0000002973 00000 n 0000003105 00000 n 0000003241 00000 n 0000003377 00000 n 0000000025 00000 f 0000087741 00000 n 0000003509 00000 n 0000003641 00000 n 0000000059 00000 f 0000087654 00000 n 0000003776 00000 n 0000003913 00000 n 0000004049 00000 n 0000004181 00000 n 0000004313 00000 n 0000004449 00000 n 0000004581 00000 n 0000004713 00000 n 0000004848 00000 n 0000088197 00000 n 0000008413 00000 n 0000010645 00000 n 0000012925 00000 n 0000016276 00000 n 0000033070 00000 n 0000036227 00000 n 0000039676 00000 n 0000041661 00000 n 0000044030 00000 n 0000049738 00000 n 0000054588 00000 n 0000060714 00000 n 0000064571 00000 n 0000067862 00000 n 0000071631 00000 n 0000008588 00000 n 0000007709 00000 n 0000005104 00000 n 0000007857 00000 n 0000007988 00000 n 0000008123 00000 n 0000008468 00000 n 0000000000 00000 f 0000087562 00000 n 0000008528 00000 n 0000008255 00000 n 0000073465 00000 n 0000010700 00000 n 0000010537 00000 n 0000008716 00000 n 0000012980 00000 n 0000012817 00000 n 0000010804 00000 n 0000016450 00000 n 0000015990 00000 n 0000013084 00000 n 0000016117 00000 n 0000016331 00000 n 0000016391 00000 n 0000085512 00000 n 0000083507 00000 n 0000085353 00000 n 0000071744 00000 n 0000019709 00000 n 0000019546 00000 n 0000016590 00000 n 0000019654 00000 n 0000023199 00000 n 0000023036 00000 n 0000019837 00000 n 0000023144 00000 n 0000088306 00000 n 0000026184 00000 n 0000026021 00000 n 0000023339 00000 n 0000026129 00000 n 0000029846 00000 n 0000029516 00000 n 0000026312 00000 n 0000029791 00000 n 0000029643 00000 n 0000054644 00000 n 0000033188 00000 n 0000032960 00000 n 0000029986 00000 n 0000033126 00000 n 0000036345 00000 n 0000036116 00000 n 0000033328 00000 n 0000036283 00000 n 0000082264 00000 n 0000080721 00000 n 0000082103 00000 n 0000039794 00000 n 0000039565 00000 n 0000036499 00000 n 0000039732 00000 n 0000041779 00000 n 0000041549 00000 n 0000039911 00000 n 0000041717 00000 n 0000088419 00000 n 0000044086 00000 n 0000043746 00000 n 0000041884 00000 n 0000043878 00000 n 0000046562 00000 n 0000046393 00000 n 0000044191 00000 n 0000046505 00000 n 0000049794 00000 n 0000048953 00000 n 0000046679 00000 n 0000049101 00000 n 0000049250 00000 n 0000049393 00000 n 0000051842 00000 n 0000051673 00000 n 0000049911 00000 n 0000051785 00000 n 0000054700 00000 n 0000054476 00000 n 0000051947 00000 n 0000058685 00000 n 0000058516 00000 n 0000054829 00000 n 0000058628 00000 n 0000088536 00000 n 0000060826 00000 n 0000060432 00000 n 0000058814 00000 n 0000060564 00000 n 0000060770 00000 n 0000064684 00000 n 0000064459 00000 n 0000060955 00000 n 0000064627 00000 n 0000067918 00000 n 0000067750 00000 n 0000064813 00000 n 0000071800 00000 n 0000071183 00000 n 0000068047 00000 n 0000071687 00000 n 0000071323 00000 n 0000071481 00000 n 0000080489 00000 n 0000079050 00000 n 0000080326 00000 n 0000073583 00000 n 0000073353 00000 n 0000071942 00000 n 0000073521 00000 n 0000074607 00000 n 0000074438 00000 n 0000073676 00000 n 0000074550 00000 n 0000088653 00000 n 0000074689 00000 n 0000090924 00000 n 0000074734 00000 n 0000074769 00000 n 0000090792 00000 n 0000074814 00000 n 0000074857 00000 n 0000090713 00000 n 0000074902 00000 n 0000074960 00000 n 0000090634 00000 n 0000075005 00000 n 0000075059 00000 n 0000090502 00000 n 0000075104 00000 n 0000075155 00000 n 0000090423 00000 n 0000075200 00000 n 0000075240 00000 n 0000090344 00000 n 0000075285 00000 n 0000075327 00000 n 0000090251 00000 n 0000075372 00000 n 0000075420 00000 n 0000090119 00000 n 0000075465 00000 n 0000075522 00000 n 0000090001 00000 n 0000075567 00000 n 0000075615 00000 n 0000089936 00000 n 0000075660 00000 n 0000075705 00000 n 0000089857 00000 n 0000075750 00000 n 0000075799 00000 n 0000089764 00000 n 0000075844 00000 n 0000075899 00000 n 0000089632 00000 n 0000075944 00000 n 0000076005 00000 n 0000089567 00000 n 0000076050 00000 n 0000076085 00000 n 0000089474 00000 n 0000076130 00000 n 0000076200 00000 n 0000089342 00000 n 0000076245 00000 n 0000076310 00000 n 0000089277 00000 n 0000076355 00000 n 0000076393 00000 n 0000089184 00000 n 0000076438 00000 n 0000076510 00000 n 0000089052 00000 n 0000076555 00000 n 0000076622 00000 n 0000088987 00000 n 0000076667 00000 n 0000076700 00000 n 0000088908 00000 n 0000076745 00000 n 0000077311 00000 n 0000077142 00000 n 0000076799 00000 n 0000077254 00000 n 0000077393 00000 n 0000080697 00000 n 0000082484 00000 n 0000082512 00000 n 0000085721 00000 n 0000085751 00000 n 0000088738 00000 n 0000088830 00000 n 0000091003 00000 n 0000091957 00000 n 0000091996 00000 n 0000092034 00000 n 0000092200 00000 n trailer << /Size 258 /Root 256 0 R /Info 257 0 R >> startxref 92360 %%EOF Zope-2.8.7-final/lib/python/ZConfig/doc/zconfig.tex0000664000076400007640000021712010435500361021114 0ustar ajungajung%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Copyright (c) 2002, 2003 Zope Corporation and Contributors. % All Rights Reserved. % % This software is subject to the provisions of the Zope Public License, % Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. % THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED % WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED % WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS % FOR A PARTICULAR PURPOSE. % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass{howto} \usepackage{xmlmarkup} \newcommand{\datatype}[1]{\strong{#1}} \title{ZConfig Package Reference} \date{18 May 2005} \release{2.3} \setshortversion{2.3} \author{Zope Corporation} \authoraddress{ Lafayette Technology Center\\ 513 Prince Edward Street\\ Fredericksburg, VA 22401\\ \url{http://www.zope.com/} } \begin{document} \maketitle \begin{abstract} \noindent This document describes the syntax and API used in configuration files for components of a Zope installation written by Zope Corporation. This configuration mechanism is itself configured using a schema specification written in XML. \end{abstract} \tableofcontents \section{Introduction \label{intro}} Zope uses a common syntax and API for configuration files designed for software components written by Zope Corporation. Third-party software which is also part of a Zope installation may use a different syntax, though any software is welcome to use the syntax used by Zope Corporation. Any software written in Python is free to use the \module{ZConfig} software to load such configuration files in order to ensure compatibility. This software is covered by the Zope Public License, version 2.0. The \module{ZConfig} package has been tested with Python 2.3. Older versions of Python are not supported. \module{ZConfig} only relies on the Python standard library. Configurations which use \module{ZConfig} are described using \dfn{schema}. A schema is a specification for the allowed structure and content of the configuration. \module{ZConfig} schema are written using a small XML-based language. The schema language allows the schema author to specify the names of the keys allowed at the top level and within sections, to define the types of sections which may be used (and where), the types of each values, whether a key or section must be specified or is optional, default values for keys, and whether a value can be given only once or repeatedly. \section{Configuration Syntax \label{syntax}} Like the \ulink{\module{ConfigParser}} {http://docs.python.org/lib/module-ConfigParser.html} format, this format supports key-value pairs arranged in sections. Unlike the \module{ConfigParser} format, sections are typed and can be organized hierarchically. Additional files may be included if needed. Schema components not specified in the application schema can be imported from the configuration file. Though both formats are substantially line-oriented, this format is more flexible. The intent of supporting nested section is to allow setting up the configurations for loosely-associated components in a container. For example, each process running on a host might get its configuration section from that host's section of a shared configuration file. The top level of a configuration file consists of a series of inclusions, key-value pairs, and sections. Comments can be added on lines by themselves. A comment has a \character{\#} as the first non-space character and extends to the end of the line: \begin{verbatim} # This is a comment \end{verbatim} An inclusion is expressed like this: \begin{verbatim} %include defaults.conf \end{verbatim} The resource to be included can be specified by a relative or absolute URL, resolved relative to the URL of the resource the \keyword{\%include} directive is located in. A key-value pair is expressed like this: \begin{verbatim} key value \end{verbatim} The key may include any non-white characters except for parentheses. The value contains all the characters between the key and the end of the line, with surrounding whitespace removed. Since comments must be on lines by themselves, the \character{\#} character can be part of a value: \begin{verbatim} key value # still part of the value \end{verbatim} Sections may be either empty or non-empty. An empty section may be used to provide an alias for another section. A non-empty section starts with a header, contains configuration data on subsequent lines, and ends with a terminator. The header for a non-empty section has this form (square brackets denote optional parts): \begin{alltt} <\var{section-type} \optional{\var{name}} > \end{alltt} \var{section-type} and \var{name} all have the same syntactic constraints as key names. The terminator looks like this: \begin{alltt} \end{alltt} The configuration data in a non-empty section consists of a sequence of one or more key-value pairs and sections. For example: \begin{verbatim} key-1 value-1 key-2 value-2 key-3 value-3 \end{verbatim} (The indentation is used here for clarity, but is not required for syntactic correctness.) The header for empty sections is similar to that of non-empty sections, but there is no terminator: \begin{alltt} <\var{section-type} \optional{\var{name}} /> \end{alltt} \subsection{Extending the Configuration Schema} As we'll see in section~\ref{writing-schema}, ``Writing Configuration Schema,'' what can be written in a configuration is controlled by schemas which can be built from \emph{components}. These components can also be used to extend the set of implementations of objects the application can handle. What this means when writing a configuration is that third-party implementations of application object types can be used wherever those application types are used in the configuration, if there's a \module{ZConfig} component available for that implementation. The configuration file can use an \keyword{\%import} directive to load a named component: \begin{verbatim} %import Products.Ape \end{verbatim} The text to the right of the \keyword{\%import} keyword must be the name of a Python package; the \module{ZConfig} component provided by that package will be loaded and incorporated into the schema being used to load the configuration file. After the import, section types defined in the component may be used in the configuration. More detail is needed for this to really make sense. A schema may define section types which are \emph{abstract}; these cannot be used directly in a configuration, but multiple concrete section types can be defined which \emph{implement} the abstract types. Wherever the application allows an abstract type to be used, any concrete type which implements that abstract type can be used in an actual configuration. The \keyword{\%import} directive allows loading schema components which provide alternate concrete section types which implement the abstract types defined by the application. This allows third-party implementations of abstract types to be used in place of or in addition to implementations provided with the application. Consider an example application application which supports logging in the same way Zope 2 does. There are some parameters which configure the general behavior of the logging mechanism, and an arbitrary number of \emph{log handlers} may be specified to control how the log messages are handled. Several log handlers are provided by the application. Here is an example logging configuration: \begin{verbatim} level verbose path /var/log/myapp/events.log \end{verbatim} A third-party component may provide a log handler to send high-priority alerts the system administrator's text pager or SMS-capable phone. All that's needed is to install the implementation so it can be imported by Python, and modify the configuration: \begin{verbatim} %import my.pager.loghandler level verbose path /var/log/myapp/events.log number 1-800-555-1234 message Something broke! \end{verbatim} \subsection{Textual Substitution in Values} \module{ZConfig} provides a limited way to re-use portions of a value using simple string substitution. To use this facility, define named bits of replacement text using the \keyword{\%define} directive, and reference these texts from values. The syntax for \keyword{\%define} is: \begin{alltt} %define \var{name} \optional{\var{value}} \end{alltt} The value of \var{name} must be a sequence of letters, digits, and underscores, and may not start with a digit; the namespace for these names is separate from the other namespaces used with \module{ZConfig}, and is case-insensitive. If \var{value} is omitted, it will be the empty string. If given, there must be whitespace between \var{name} and \var{value}; \var{value} will not include any whitespace on either side, just like values from key-value pairs. Names must be defined before they are used, and may not be re-defined. All resources being parsed as part of a configuration share a single namespace for defined names. This means that resources which may be included more than once should not define any names. References to defined names from configuration values use the syntax described for the \refmodule{ZConfig.substitution} module. Configuration values which include a \character{\$} as part of the actual value will need to use \code{\$\$} to get a single \character{\$} in the result. The values of defined names are processed in the same way as configuration values, and may contain references to named definitions. For example, the value for \code{key} will evaluate to \code{value}: \begin{verbatim} %define name value key $name \end{verbatim} %$ <-- bow to font-lock \section{Writing Configuration Schema \label{writing-schema}} \module{ZConfig} schema are written as XML documents. Data types are searched in a special namespace defined by the data type registry. The default registry has slightly magical semantics: If the value can be matched to a standard data type when interpreted as a \datatype{basic-key}, the standard data type will be used. If that fails, the value must be a \datatype{dotted-name} containing at least one dot, and a conversion function will be sought using the \method{search()} method of the data type registry used to load the schema. \subsection{Schema Elements \label{elements}} For each element, the content model is shown, followed by a description of how the element is used, and then a list of the available attributes. For each attribute, the type of the value is given as either the name of a \module{ZConfig} datatype or an XML attribute value type. Familiarity with XML's Document Type Definition language is helpful. The following elements are used to describe a schema: \begin{elementdesc}{schema}{description?, metadefault?, example?, import*, (sectiontype | abstracttype)*, (section | key | multisection | multikey)*} Document element for a \module{ZConfig} schema. \begin{attributedesc}{extends}{\datatype{space-separated-url-references}} A list of URLs of base schemas from which this section type will inherit key, section, and section type declarations. If omitted, this schema is defined using only the keys, sections, and section types contained within the \element{schema} element. \end{attributedesc} \begin{attributedesc}{datatype}{\datatype{basic-key} or \datatype{dotted-name}} The data type converter which will be applied to the value of this section. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. If any base schemas are listed in the \attribute{extends} attribute, the default value for this attribute comes from the base schemas. If the base schemas all use the same \attribute{datatype}, then that data type will be the default value for the extending schema. If there are no base schemas, the default value is \datatype{null}, which means that the \module{ZConfig} section object will be used unconverted. If the base schemas have different \attribute{datatype} definitions, you must explicitly define the \attribute{datatype} in the extending schema. \end{attributedesc} \begin{attributedesc}{handler}{\datatype{basic-key}} \end{attributedesc} \begin{attributedesc}{keytype}{\datatype{basic-key} or \datatype{dotted-name}} The data type converter which will be applied to keys found in this section. This can be used to constrain key values in different ways; two data types which may be especially useful are the \datatype{identifier} and \datatype{ipaddr-or-hostname} types. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. If any base schemas are listed in the \attribute{extends} attribute, the default value for this attribute comes from the base schemas. If the base schemas all use the same \attribute{keytype}, then that key type will be the default value for the extending schema. If there are no base schemas, the default value is \datatype{basic-key}. If the base schemas have different \attribute{keytype} definitions, you must explicitly define the \attribute{keytype} in the extending schema. \end{attributedesc} \begin{attributedesc}{prefix}{\datatype{dotted-name}} Prefix to be pre-pended in front of partial dotted-names that start with a period. The value of this attribute is used in all contexts with the \element{schema} element if it hasn't been overridden by an inner element with a \attribute{prefix} attribute. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{description}{PCDATA} Descriptive text explaining the purpose the container of the \element{description} element. Most other elements can contain a \element{description} element as their first child. At most one \element{description} element may appear in a given context. \begin{attributedesc}{format}{NMTOKEN} Optional attribute that can be added to indicate what conventions are used to mark up the contained text. This is intended to serve as a hint for documentation extraction tools. Suggested values are: \begin{tableii}{l|l}{code}{Value}{Content Format} \lineii{plain}{\mimetype{text/plain}; blank lines separate paragraphs} \lineii{rest}{reStructuredText} \lineii{stx}{Classic Structured Text} \end{tableii} \end{attributedesc} \end{elementdesc} \begin{elementdesc}{example}{PCDATA} An example value. This serves only as documentation. \end{elementdesc} \begin{elementdesc}{metadefault}{PCDATA} A description of the default value, for human readers. This may include information about how a computed value is determined when the schema does not specify a default value. \end{elementdesc} \begin{elementdesc}{abstracttype}{description?} Define an abstract section type. \begin{attributedesc}{name}{\datatype{basic-key}} The name of the abstract section type; required. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{sectiontype}{description?, (section | key | multisection | multikey)*} Define a concrete section type. \begin{attributedesc}{datatype}{\datatype{basic-key} or \datatype{dotted-name}} The data type converter which will be applied to the value of this section. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. If \attribute{datatype} is omitted and \attribute{extends} is used, the \attribute{datatype} from the section type identified by the \attribute{extends} attribute is used. \end{attributedesc} \begin{attributedesc}{extends}{\datatype{basic-key}} The name of a concrete section type from which this section type acquires all key and section declarations. This type does \emph{not} automatically implement any abstract section type implemented by the named section type. If omitted, this section is defined with only the keys and sections contained within the \element{sectiontype} element. The new section type is called a \emph{derived} section type, and the type named by this attribute is called the \emph{base} type. Values for the \attribute{datatype} and \attribute{keytype} attributes are acquired from the base type if not specified. \end{attributedesc} \begin{attributedesc}{implements}{\datatype{basic-key}} The name of an abstract section type which this concrete section type implements. If omitted, this section type does not implement any abstract type, and can only be used if it is specified directly in a schema or other section type. \end{attributedesc} \begin{attributedesc}{keytype}{\datatype{basic-key}} The data type converter which will be applied to keys found in this section. This can be used to constrain key values in different ways; two data types which may be especially useful are the \datatype{identifier} and \datatype{ipaddr-or-hostname} types. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. The default value is \datatype{basic-key}. If \attribute{keytype} is omitted and \attribute{extends} is used, the \attribute{keytype} from the section type identified by the \attribute{extends} attribute is used. \end{attributedesc} \begin{attributedesc}{name}{\datatype{basic-key}} The name of the section type; required. \end{attributedesc} \begin{attributedesc}{prefix}{\datatype{dotted-name}} Prefix to be pre-pended in front of partial dotted-names that start with a period. The value of this attribute is used in all contexts in the \element{sectiontype} element. If omitted, the prefix specified by a containing context is used if specified. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{import}{EMPTY} Import a schema component. Exactly one of the attributes \attribute{package} and \attribute{src} must be specified. \begin{attributedesc}{file}{file name without directory information} Name of the component file within a package; if not specified, \file{component.xml} is used. This may only be given when \attribute{package} is used. (The \file{component.xml} file is always used when importing via \keyword{\%import} from a configuration file.) \end{attributedesc} \begin{attributedesc}{package}{\datatype{dotted-suffix}} Name of a Python package that contains the schema component being imported. The component will be loaded from the file identified by the \attribute{file} attribute, or \file{component.xml} if \attribute{file} is not specified. If the package name given starts with a dot (\character{.}), the name used will be the current prefix and the value of this attribute concatenated. \end{attributedesc} \begin{attributedesc}{src}{\datatype{url-reference}} URL to a separate schema which can provide useful types. The referenced resource must contain a schema, not a schema component. Section types defined or imported by the referenced schema are added to the schema containing the \element{import}; top-level keys and sections are ignored. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{key}{description?, example?, metadefault?, default*} A \element{key} element is used to describe a key-value pair which may occur at most once in the section type or top-level schema in which it is listed. \begin{attributedesc}{attribute}{\datatype{identifier}} The name of the Python attribute which this key should be the value of on a \class{SectionValue} instance. This must be unique within the immediate contents of a section type or schema. If this attribute is not specified, an attribute name will be computed by converting hyphens in the key name to underscores. \end{attributedesc} \begin{attributedesc}{datatype}{\datatype{basic-key} or \datatype{dotted-name}} The data type converter which will be applied to the value of this key. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. \end{attributedesc} \begin{attributedesc}{default}{\datatype{string}} If the key-value pair is optional and this attribute is specified, the value of this attribute will be converted using the appropriate data type converter and returned to the application as the configured value. This attribute may not be specified if the \attribute{required} attribute is \code{yes}. \end{attributedesc} \begin{attributedesc}{handler}{\datatype{basic-key}} \end{attributedesc} \begin{attributedesc}{name}{\datatype{basic-key}} The name of the key, as it must be given in a configuration instance, or `\code{*}'. If the value is `\code{*}', any name not already specified as a key may be used, and the configuration value for the key will be a dictionary mapping from the key name to the value. In this case, the \attribute{attribute} attribute must be specified, and the data type for the key will be applied to each key which is found. \end{attributedesc} \begin{attributedesc}{required}{\code{yes|no}} Specifies whether the configuration instance is required to provide the key. If the value is \code{yes}, the \attribute{default} attribute may not be specified and an error will be reported if the configuration instance does not specify a value for the key. If the value is \code{no} (the default) and the configuration instance does not specify a value, the value reported to the application will be that specified by the \attribute{default} attribute, if given, or \code{None}. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{multikey}{description?, example?, metadefault?, default*} A \element{multikey} element is used to describe a key-value pair which may occur any number of times in the section type or top-level schema in which it is listed. \begin{attributedesc}{attribute}{\datatype{identifier}} The name of the Python attribute which this key should be the value of on a \class{SectionValue} instance. This must be unique within the immediate contents of a section type or schema. If this attribute is not specified, an attribute name will be computed by converting hyphens in the key name to underscores. \end{attributedesc} \begin{attributedesc}{datatype}{\datatype{basic-key} or \datatype{dotted-name}} The data type converter which will be applied to the value of this key. If the value is a \datatype{dotted-name} that begins with a period, the value of \attribute{prefix} will be pre-pended, if set. \end{attributedesc} \begin{attributedesc}{handler}{\datatype{basic-key}} \end{attributedesc} \begin{attributedesc}{name}{\datatype{basic-key}} The name of the key, as it must be given in a configuration instance, or `\code{+}'. If the value is `\code{+}', any name not already specified as a key may be used, and the configuration value for the key will be a dictionary mapping from the key name to the value. In this case, the \attribute{attribute} attribute must be specified, and the data type for the key will be applied to each key which is found. \end{attributedesc} \begin{attributedesc}{required}{\code{yes|no}} Specifies whether the configuration instance is required to provide the key. If the value is \code{yes}, no \element{default} elements may be specified and an error will be reported if the configuration instance does not specify at least one value for the key. If the value is \code{no} (the default) and the configuration instance does not specify a value, the value reported to the application will be a list containing one element for each \element{default} element specified as a child of the \element{multikey}. Each value will be individually converted according to the \attribute{datatype} attribute. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{default}{PCDATA} Each \element{default} element specifies a single default value for a \element{multikey}. This element can be repeated to produce a list of individual default values. The text contained in the element will be passed to the datatype conversion for the \element{multikey}. \begin{attributedesc}{key}{key type of the containing sectiontype} Key to associate with the default value. This is only used for defaults of a \element{key} or \element{multikey} with a \attribute{name} of \code{+}; in that case this attribute is required. It is an error to use the \attribute{key} attribute with a \element{default} element for a \element{multikey} with a name other than \code{+}. \begin{notice}[warning] The datatype of this attribute is that of the section type \emph{containing} the actual keys, not necessarily that of the section type which defines the key. If a derived section overrides the key type of the base section type, the actual key type used is that of the derived section. This can lead to confusing errors in schemas, though the \refmodule{ZConfig} package checks for this when the schema is loaded. This situation is particularly likely when a derived section type uses a key type which collapses multiple default keys which were not collapsed by the base section type. Consider this example schema: \begin{verbatim} some value some value
\end{verbatim} When this schema is loaded, a set of defaults for the \datatype{derived} section type is computed. Since \datatype{basic-key} is case-insensitive (everything is converted to lower case), \samp{foo} and \samp{Foo} are both converted to \samp{foo}, which clashes since \element{key} only allows one value for each key. \end{notice} \end{attributedesc} \end{elementdesc} \begin{elementdesc}{section}{description?} A \element{section} element is used to describe a section which may occur at most once in the section type or top-level schema in which it is listed. \begin{attributedesc}{attribute}{\datatype{identifier}} The name of the Python attribute which this section should be the value of on a \class{SectionValue} instance. This must be unique within the immediate contents of a section type or schema. If this attribute is not specified, an attribute name will be computed by converting hyphens in the section name to underscores, in which case the \attribute{name} attribute may not be \code{*} or \code{+}. \end{attributedesc} \begin{attributedesc}{handler}{\datatype{basic-key}} \end{attributedesc} \begin{attributedesc}{name}{\datatype{basic-key}} The name of the section, as it must be given in a configuration instance, \code{*}, or \code{+}. If the value is \code{*} or this attribute is omitted, any name not already specified as a key may be used. If the value is \code{*} or \code{+}, the \attribute{attribute} attribute must be specified. If the value is \code{*}, any name is allowed, or the name may be omitted. If the value is \code{+}, any name is allowed, but some name must be provided. \end{attributedesc} \begin{attributedesc}{required}{\code{yes|no}} Specifies whether the configuration instance is required to provide the section. If the value is \code{yes}, an error will be reported if the configuration instance does not include the section. If the value is \code{no} (the default) and the configuration instance does not include the section, the value reported to the application will be \code{None}. \end{attributedesc} \begin{attributedesc}{type}{\datatype{basic-key}} The section type which matching sections must implement. If the value names an abstract section type, matching sections in the configuration file must be of a type which specifies that it implements the named abstract type. If the name identifies a concrete type, the section type must match exactly. \end{attributedesc} \end{elementdesc} \begin{elementdesc}{multisection}{description?} A \element{multisection} element is used to describe a section which may occur any number of times in the section type or top-level schema in which it is listed. \begin{attributedesc}{attribute}{\datatype{identifier}} The name of the Python attribute which matching sections should be the value of on a \class{SectionValue} instance. This is required and must be unique within the immediate contents of a section type or schema. The \class{SectionValue} instance will contain a list of matching sections. \end{attributedesc} \begin{attributedesc}{handler}{\datatype{basic-key}} \end{attributedesc} \begin{attributedesc}{name}{\datatype{basic-key}} For a \element{multisection}, any name not already specified as a key may be used. If the value is \code{*} or \code{+}, the \attribute{attribute} attribute must be specified. If the value is \code{*} or this attribute is omitted, any name is allowed, or the name may be omitted. If the value is \code{+}, any name is allowed, but some name must be provided. No other value for the \attribute{name} attribute is allowed for a \element{multisection}. \end{attributedesc} \begin{attributedesc}{required}{\code{yes|no}} Specifies whether the configuration instance is required to provide at least one matching section. If the value is \code{yes}, an error will be reported if the configuration instance does not include the section. If the value is \code{no} (the default) and the configuration instance does not include the section, the value reported to the application will be \code{None}. \end{attributedesc} \begin{attributedesc}{type}{\datatype{basic-key}} The section type which matching sections must implement. If the value names an abstract section type, matching sections in the configuration file must be of types which specify that they implement the named abstract type. If the name identifies a concrete type, the section type must match exactly. \end{attributedesc} \end{elementdesc} \subsection{Schema Components \label{schema-components}} XXX need more explanation \module{ZConfig} supports schema components that can be provided by disparate components, and allows them to be knit together into concrete schema for applications. Components cannot add additional keys or sections in the application schema. A schema \dfn{component} is allowed to define new abstract and section types. Components are identified using a dotted-name, similar to a Python module name. For example, one component may be \code{zodb.storage}. Schema components are stored alongside application code since they directly reference datatype code. Schema components are provided by Python packages. The component definition is normally stored in the file \file{component.xml}; an alternate filename may be specified using the \attribute{file} attribute of the \element{import} element. Components imported using the \keyword{\%import} keyword from a configuration file must be named \file{component.xml}. The component defines the types provided by that component; it must have a \element{component} element as the document element. The following element is used as the document element for schema components. Note that schema components do not allow keys and sections to be added to the top-level of a schema; they serve only to provide type definitions. \begin{elementdesc}{component}{description?, (abstracttype | sectiontype)*} The top-level element for schema components. \begin{attributedesc}{prefix}{\datatype{dotted-name}} Prefix to be pre-pended in front of partial dotted-names that start with a period. The value of this attribute is used in all contexts within the \element{component} element if it hasn't been overridden by an inner element with a \attribute{prefix} attribute. \end{attributedesc} \end{elementdesc} \section{Standard \module{ZConfig} Datatypes\label{standard-datatypes}} There are a number of data types which can be identified using the \attribute{datatype} attribute on \element{key}, \element{sectiontype}, and \element{schema} elements. Applications may extend the set of datatypes by calling the \method{register()} method of the data type registry being used or by using Python dotted-names to refer to conversion routines defined in code. The following data types are provided by the default type registry. \begin{definitions} \term{\datatype{basic-key}} The default data type for a key in a ZConfig configuration file. The result of conversion is always lower-case, and matches the regular expression \regexp{[a-z][-._a-z0-9]*}. \term{\datatype{boolean}} Convert a human-friendly string to a boolean value. The names \code{yes}, \code{on}, and \code{true} convert to \constant{True}, while \code{no}, \code{off}, and \code{false} convert to \constant{False}. Comparisons are case-insensitive. All other input strings are disallowed. \term{\datatype{byte-size}} A specification of a size, with byte multiplier suffixes (for example, \samp{128MB}). Suffixes are case insensitive and may be \samp{KB}, \samp{MB}, or \samp{GB} \term{\datatype{dotted-name}} A string consisting of one or more \datatype{identifier} values separated by periods (\character{.}). \term{\datatype{dotted-suffix}} A string consisting of one or more \datatype{identifier} values separated by periods (\character{.}), possibly prefixed by a period. This can be used to indicate a dotted name that may be specified relative to some base dotted name. \term{\datatype{existing-dirpath}} Validates that the directory portion of a pathname exists. For example, if the value provided is \file{/foo/bar}, \file{/foo} must be an existing directory. No conversion is performed. \term{\datatype{existing-directory}} Validates that a directory by the given name exists on the local filesystem. No conversion is performed. \term{\datatype{existing-file}} Validates that a file by the given name exists. No conversion is performed. \term{\datatype{existing-path}} Validates that a path (file, directory, or symlink) by the given name exists on the local filesystem. No conversion is performed. \term{\datatype{float}} A Python float. \code{Inf}, \code{-Inf}, and \code{NaN} are not allowed. \term{\datatype{identifier}} Any valid Python identifier. \term{\datatype{inet-address}} An Internet address expressed as a \code{(\var{hostname}, \var{port})} pair. If only the port is specified, the default host will be returned for \var{hostname}. The default host is \code{localhost} on Windows and the empty string on all other platforms. If the port is omitted, \code{None} will be returned for \var{port}. \term{\datatype{inet-binding-address}} An Internet address expressed as a \code{(\var{hostname}, \var{port})} pair. The address is suitable for binding a socket. If only the port is specified, the default host will be returned for \var{hostname}. The default host is the empty string on all platforms. If the port is omitted, \code{None} will be returned for \var{port}. \term{\datatype{inet-connection-address}} An Internet address expressed as a \code{(\var{hostname}, \var{port})} pair. The address is suitable for connecting a socket to a server. If only the port is specified, \code{'127.0.0.1'} will be returned for \var{hostname}. If the port is omitted, \code{None} will be returned for \var{port}. \term{\datatype{integer}} Convert a value to an integer. This will be a Python \class{int} if the value is in the range allowed by \class{int}, otherwise a Python \class{long} is returned. \term{\datatype{ipaddr-or-hostname}} Validates a valid IP address or hostname. If the first character is a digit, the value is assumed to be an IP address. If the first character is not a digit, the value is assumed to be a hostname. Hostnames are converted to lower case. \term{\datatype{locale}} Any valid locale specifier accepted by the available \function{locale.setlocale()} function. Be aware that only the \code{'C'} locale is supported on some platforms. \term{\datatype{null}} No conversion is performed; the value passed in is the value returned. This is the default data type for section values. \term{\datatype{port-number}} Returns a valid port number as an integer. Validity does not imply that any particular use may be made of the port, however. For example, port number lower than 1024 generally cannot be bound by non-root users. \term{\datatype{socket-address}} An address for a socket. The converted value is an object providing two attributes. \member{family} specifies the address family (\constant{AF_INET} or \constant{AF_UNIX}), with \code{None} instead of \constant{AF_UNIX} on platforms that don't support it. The \member{address} attribute will be the address that should be passed to the socket's \method{bind()} method. If the family is \constant{AF_UNIX}, the specific address will be a pathname; if the family is \constant{AF_INET}, the second part will be the result of the \datatype{inet-address} conversion. \term{\datatype{string}} Returns the input value as a string. If the source is a Unicode string, this implies that it will be checked to be simple 7-bit \ASCII. This is the default data type for values in configuration files. \term{\datatype{time-interval}} A specification of a time interval in seconds, with multiplier suffixes (for example, \code{12h}). Suffixes are case insensitive and may be \samp{s} (seconds), \samp{m} (minutes), \samp{h} (hours), or \samp{d} (days). \term{\datatype{timedelta}} Similar to the \datatype{time-interval}, this data type returns a Python datetime.timedelta object instead of a float. The set of suffixes recognized by \datatype{timedelta} are: \samp{w} (weeks), \samp{d} (days), \samp{h} (hours), \samp{m} (minutes), \samp{s} (seconds). Values may be floats, for example: \code{4w 2.5d 7h 12m 0.001s}. \end{definitions} \section{Standard \module{ZConfig} Schema Components \label{standard-components}} \module{ZConfig} provides a few convenient schema components as part of the package. These may be used directly or can server as examples for creating new components. \subsection{\module{ZConfig.components.basic}} The \module{ZConfig.components.basic} package provides small components that can be helpful in composing application-specific components and schema. There is no large functionality represented by this package. The default component provided by this package simply imports all of the smaller components. This can be imported using \begin{verbatim} \end{verbatim} Each of the smaller components is documented directly; importing these selectively can reduce the time it takes to load a schema slightly, and allows replacing the other basic components with alternate components (by using different imports that define the same type names) if desired. \subsubsection{The Mapping Section Type \label{basic-mapping}} There is a basic section type that behaves like a simple Python mapping; this can be imported directly using \begin{verbatim} \end{verbatim} This defines a single section type, \datatype{ZConfig.basic.mapping}. When this is used, the section value is a Python dictionary mapping keys to string values. This type is intended to be used by extending it in simple ways. The simplest is to create a new section type name that makes more sense for the application: \begin{verbatim}
\end{verbatim} This allows a configuration to contain a mapping from \datatype{basic-key} names to string values like this: \begin{verbatim} This that and the other \end{verbatim} The value of the configuration object's \member{map} attribute would then be the dictionary \begin{verbatim} {'this': 'that', 'and': 'the other', } \end{verbatim} (Recall that the \datatype{basic-key} data type converts everything to lower case.) Perhaps a more interesting application of \datatype{ZConfig.basic.mapping} is using the derived type to override the \attribute{keytype}. If we have the conversion function: \begin{verbatim} def email_address(value): userid, hostname = value.split("@", 1) hostname = hostname.lower() # normalize what we know we can return "%s@%s" % (userid, hostname) \end{verbatim} then we can use this as the key type for a derived mapping type: \begin{verbatim}
\end{verbatim} \subsection{\module{ZConfig.components.logger}} The \module{ZConfig.components.logger} package provides configuration support for the \ulink{\module{logging} package} {http://docs.python.org/lib/module-logging.html} in Python's standard library. This component can be imported using \begin{verbatim} \end{verbatim} This component defines two abstract types and several concrete section types. These can be imported as a unit, as above, or as four smaller components usable in creating alternate logging packages. The first of the four smaller components contains the abstract types, and can be imported using \begin{verbatim} \end{verbatim} The two abstract types imported by this are: \begin{definitions} \term{\datatype{ZConfig.logger.log}} Logger objects are represented by this abstract type. \term{\datatype{ZConfig.logger.handler}} Each logger object can have one or more ``handlers'' associated with them. These handlers are responsible for writing logging events to some form of output stream using appropriate formatting. The output stream may be a file on a disk, a socket communicating with a server on another system, or a series of \code{syslog} messages. Section types which implement this type represent these handlers. \end{definitions} The second and third of the smaller components provides section types that act as factories for \class{logging.Logger} objects. These can be imported using \begin{verbatim} \end{verbatim} The types defined in these components implement the \datatype{ZConfig.logger.log} abstract type. The \file{eventlog.xml} component defines an \datatype{eventlog} type which represents the root logger from the the \module{logging} package (the return value of \function{logging.getLogger()}), while the \file{logger.xml} component defines a \datatype{logger} section type which represents a named logger (as returned by \function{logging.getLogger(\var{name})}). The third of the smaller components provides section types that are factories for \class{logging.Handler} objects. This can be imported using \begin{verbatim} \end{verbatim} The types defined in this component implement the \datatype{ZConfig.logger.handler} abstract type. The configuration objects provided by both the logger and handler types are factories for the finished loggers and handlers. These factories should be called with no arguments to retrieve the logger or log handler objects. Calling the factories repeatedly will cause the same objects to be returned each time, so it's safe to simply call them to retrieve the objects. The factories for the logger objects, whether the \datatype{eventlog} or \datatype{logger} section type is used, provide a \method{reopen()} method which may be called to close any log files and re-open them. This is useful when using a \UNIX{} signal to effect log file rotation: the signal handler can call this method, and not have to worry about what handlers have been registered for the logger. Building an application that uses the logging components is fairly straightforward. The schema needs to import the relevant components and declare their use: \begin{verbatim}
\end{verbatim} In the application, the schema and configuration file should be loaded normally. Once the configuration object is available, the logger factory should be called to configure Python's \module{logging} package: \begin{verbatim} import os import ZConfig def run(configfile): schemafile = os.path.join(os.path.dirname(__file__), "schema.xml") schema = ZConfig.loadSchema(schemafile) config, handlers = ZConfig.loadConfig(schema, configfile) # configure the logging package: config.eventlog() # now do interesting things \end{verbatim} An example configuration file for this application may look like this: \begin{verbatim} level info path /var/log/myapp format %(asctime)s %(levelname)s %(name)s %(message)s # locale-specific date/time representation dateformat %c level error address syslog.example.net:514 format %(levelname)s %(name)s %(message)s \end{verbatim} Refer to the \module{logging} package documentation for the names available in the message format strings (the \code{format} key in the log handlers). The date format strings (the \code{dateformat} key in the log handlers) are the same as those accepted by the \function{time.strftime()} function. \begin{seealso} \seepep{282}{A Logging System} {The proposal which described the logging feature for inclusion in the Python standard library.} \seelink{http://docs.python.org/lib/module-logging.html} {\module{logging} --- Logging facility for Python} {Python's \module{logging} package documentation, from the \citetitle[http://docs.python.org/lib/lib.html] {Python Library Reference}.} \seelink{http://www.red-dove.com/python_logging.html} {Original Python \module{logging} package} {This is the original source for the \module{logging} package. This is mostly of historical interest.} \end{seealso} \section{Using Components to Extend Schema} % XXX This section needs a lot of work, but should get people started % who really want to add new pieces to ZConfig-configured applications. It is possible to use schema components and the \keyword{\%import} construct to extend the set of section types available for a specific configuration file, and allow the new components to be used in place of standard components. The key to making this work is the use of abstract section types. Wherever the original schema accepts an abstract type, it is possible to load new implementations of the abstract type and use those instead of, or in addition to, the implementations loaded by the original schema. Abstract types are generally used to represent interfaces. Sometimes these are interfaces for factory objects, and sometimes not, but there's an interface that the new component needs to implement. What interface is required should be documented in the \element{description} element in the \element{abstracttype} element; this may be by reference to an interface specified in a Python module or described in some other bit of documentation. The following things need to be created to make the new component usable from the configuration file: \begin{enumerate} \item An implementation of the required interface. \item A schema component that defines a section type that contains the information needed to construct the component. \item A ``datatype'' function that converts configuration data to an instance of the component. \end{enumerate} For simplicity, let's assume that the implementation is defined by a Python class. The example component we build here will be in the \module{noise} package, but any package will do. Components loadable using \keyword{\%import} must be contained in the \file{component.xml} file; alternate filenames may not be selected by the \keyword{\%import} construct. Create a ZConfig component that provides a section type to support your component. The new section type must declare that it implements the appropriate abstract type; it should probably look something like this: \begin{verbatim} Port number to listen on. Silly way to specify a noise generation algorithm. \end{verbatim} This example uses one of the standard ZConfig datatypes, \datatype{port-number}, and requires two additional types to be provided by the \module{noise.server} module: \class{NoiseServerFactory} and \function{noise_color()}. The \function{noise_color()} function is a datatype conversion for a key, so it accepts a string and returns the value that should be used: \begin{verbatim} _noise_colors = { # color -> r,g,b 'white': (255, 255, 255), 'pink': (255, 182, 193), } def noise_color(string): if string in _noise_colors: return _noise_colors[string] else: raise ValueError('unknown noise color: %r' % string) \end{verbatim} \class{NoiseServerFactory} is a little different, as it's the datatype function for a section rather than a key. The parameter isn't a string, but a section value object with two attributes, \member{port} and \member{color}. Since the \datatype{ZServer.server} abstract type requires that the component returned is a factory object, the datatype function can be implemented at the constructor for the class of the factory object. (If the datatype function could select different implementation classes based on the configuration values, it makes more sense to use a simple function that returns the appropriate implementation.) A class that implements this datatype might look like this: \begin{verbatim} from ZServer.datatypes import ServerFactory from noise.generator import WhiteNoiseGenerator, PinkNoiseGenerator class NoiseServerFactory(ServerFactory): def __init__(self, section): # host and ip will be initialized by ServerFactory.prepare() self.host = None self.ip = None self.port = section.port self.color = section.color def create(self): if self.color == 'white': generator = WhiteNoiseGenerator() else: generator = PinkNoiseGenerator() return NoiseServer(self.ip, self.port, generator) \end{verbatim} You'll need to arrange for the package containing this component is available on Python's \code{sys.path} before the configuration file is loaded; this is mostly easily done by manipulating the \envvar{PYTHONPATH} environment variable. Your configuration file can now include the following to load and use your new component: \begin{verbatim} %import noise port 1234 color white \end{verbatim} \section{\module{ZConfig} --- Basic configuration support} \declaremodule{}{ZConfig} \modulesynopsis{Configuration package.} The main \module{ZConfig} package exports these convenience functions: \begin{funcdesc}{loadConfig}{schema, url\optional{, overrides}} Load and return a configuration from a URL or pathname given by \var{url}. \var{url} may be a URL, absolute pathname, or relative pathname. Fragment identifiers are not supported. \var{schema} is a reference to a schema loaded by \function{loadSchema()} or \function{loadSchemaFile()}. The return value is a tuple containing the configuration object and a composite handler that, when called with a name-to-handler mapping, calls all the handlers for the configuration. The optional \var{overrides} argument represents information derived from command-line arguments. If given, it must be either a sequence of value specifiers, or \code{None}. A \dfn{value specifier} is a string of the form \code{\var{optionpath}=\var{value}}. The \var{optionpath} specifies the ``full path'' to the configuration setting: it can contain a sequence of names, separated by \character{/} characters. Each name before the last names a section from the configuration file, and the last name corresponds to a key within the section identified by the leading section names. If \var{optionpath} contains only one name, it identifies a key in the top-level schema. \var{value} is a string that will be treated just like a value in the configuration file. \end{funcdesc} \begin{funcdesc}{loadConfigFile}{schema, file\optional{, url\optional{, overrides}}} Load and return a configuration from an opened file object. If \var{url} is omitted, one will be computed based on the \member{name} attribute of \var{file}, if it exists. If no URL can be determined, all \keyword{\%include} statements in the configuration must use absolute URLs. \var{schema} is a reference to a schema loaded by \function{loadSchema()} or \function{loadSchemaFile()}. The return value is a tuple containing the configuration object and a composite handler that, when called with a name-to-handler mapping, calls all the handlers for the configuration. The \var{overrides} argument is the same as for the \function{loadConfig()} function. \end{funcdesc} \begin{funcdesc}{loadSchema}{url} Load a schema definition from the URL \var{url}. \var{url} may be a URL, absolute pathname, or relative pathname. Fragment identifiers are not supported. The resulting schema object can be passed to \function{loadConfig()} or \function{loadConfigFile()}. The schema object may be used as many times as needed. \end{funcdesc} \begin{funcdesc}{loadSchemaFile}{file\optional{, url}} Load a schema definition from the open file object \var{file}. If \var{url} is given and not \code{None}, it should be the URL of resource represented by \var{file}. If \var{url} is omitted or \code{None}, a URL may be computed from the \member{name} attribute of \var{file}, if present. The resulting schema object can be passed to \function{loadConfig()} or \function{loadConfigFile()}. The schema object may be used as many times as needed. \end{funcdesc} The following exceptions are defined by this package: \begin{excdesc}{ConfigurationError} Base class for exceptions specific to the \module{ZConfig} package. All instances provide a \member{message} attribute that describes the specific error, and a \member{url} attribute that gives the URL of the resource the error was located in, or \constant{None}. \end{excdesc} \begin{excdesc}{ConfigurationSyntaxError} Exception raised when a configuration source does not conform to the allowed syntax. In addition to the \member{message} and \member{url} attributes, exceptions of this type offer the \member{lineno} attribute, which provides the line number at which the error was detected. \end{excdesc} \begin{excdesc}{DataConversionError} Raised when a data type conversion fails with \exception{ValueError}. This exception is a subclass of both \exception{ConfigurationError} and \exception{ValueError}. The \function{str()} of the exception provides the explanation from the original \exception{ValueError}, and the line number and URL of the value which provoked the error. The following additional attributes are provided: \begin{tableii}{l|l}{member}{Attribute}{Value} \lineii{colno} {column number at which the value starts, or \code{None}} \lineii{exception} {the original \exception{ValueError} instance} \lineii{lineno} {line number on which the value starts} \lineii{message} {\function{str()} returned by the original \exception{ValueError}} \lineii{value} {original value passed to the conversion function} \lineii{url} {URL of the resource providing the value text} \end{tableii} \end{excdesc} \begin{excdesc}{SchemaError} Raised when a schema contains an error. This exception type provides the attributes \member{url}, \member{lineno}, and \member{colno}, which provide the source URL, the line number, and the column number at which the error was detected. These attributes may be \code{None} in some cases. \end{excdesc} \begin{excdesc}{SchemaResourceError} Raised when there's an error locating a resource required by the schema. This is derived from \exception{SchemaError}. Instances of this exception class add the attributes \member{filename}, \member{package}, and \member{path}, which hold the filename searched for within the package being loaded, the name of the package, and the \code{__path__} attribute of the package itself (or \constant{None} if it isn't a package or could not be imported). \end{excdesc} \begin{excdesc}{SubstitutionReplacementError} Raised when the source text contains references to names which are not defined in \var{mapping}. The attributes \member{source} and \member{name} provide the complete source text and the name (converted to lower case) for which no replacement is defined. \end{excdesc} \begin{excdesc}{SubstitutionSyntaxError} Raised when the source text contains syntactical errors. \end{excdesc} \subsection{Basic Usage} The simplest use of \refmodule{ZConfig} is to load a configuration based on a schema stored in a file. This example loads a configuration file specified on the command line using a schema in the same directory as the script: \begin{verbatim} import os import sys import ZConfig try: myfile = __file__ except NameError: myfile = os.path.realpath(sys.argv[0]) mydir = os.path.dirname(myfile) schema = ZConfig.loadSchema(os.path.join(mydir, 'schema.xml')) conf, handler = ZConfig.loadConfig(schema, sys.argv[1]) \end{verbatim} If the schema file contained this schema: \begin{verbatim} \end{verbatim} and the file specified on the command line contained this text: \begin{verbatim} # sample configuration server www.example.com \end{verbatim} then the configuration object \code{conf} loaded above would have two attributes: \begin{tableii}{l|l}{member}{Attribute}{Value} \lineii{server}{\code{'www.example.com'}} \lineii{attempts}{\code{5}} \end{tableii} \section{\module{ZConfig.datatypes} --- Default data type registry} \declaremodule{}{ZConfig.datatypes} \modulesynopsis{Default implementation of a data type registry} The \module{ZConfig.datatypes} module provides the implementation of the default data type registry and all the standard data types supported by \module{ZConfig}. A number of convenience classes are also provided to assist in the creation of additional data types. A \dfn{datatype registry} is an object that provides conversion functions for data types. The interface for a registry is fairly simple. A \dfn{conversion function} is any callable object that accepts a single argument and returns a suitable value, or raises an exception if the input value is not acceptable. \exception{ValueError} is the preferred exception for disallowed inputs, but any other exception will be properly propagated. \begin{classdesc}{Registry}{\optional{stock}} Implementation of a simple type registry. If given, \var{stock} should be a mapping which defines the ``built-in'' data types for the registry; if omitted or \code{None}, the standard set of data types is used (see section~\ref{standard-datatypes}, ``Standard \module{ZConfig} Datatypes''). \end{classdesc} \class{Registry} objects have the following methods: \begin{methoddesc}{get}{name} Return the type conversion routine for \var{name}. If the conversion function cannot be found, an (unspecified) exception is raised. If the name is not provided in the stock set of data types by this registry and has not otherwise been registered, this method uses the \method{search()} method to load the conversion function. This is the only method the rest of \module{ZConfig} requires. \end{methoddesc} \begin{methoddesc}{register}{name, conversion} Register the data type name \var{name} to use the conversion function \var{conversion}. If \var{name} is already registered or provided as a stock data type, \exception{ValueError} is raised (this includes the case when \var{name} was found using the \method{search()} method). \end{methoddesc} \begin{methoddesc}{search}{name} This is a helper method for the default implementation of the \method{get()} method. If \var{name} is a Python dotted-name, this method loads the value for the name by dynamically importing the containing module and extracting the value of the name. The name must refer to a usable conversion function. \end{methoddesc} The following classes are provided to define conversion functions: \begin{classdesc}{MemoizedConversion}{conversion} Simple memoization for potentially expensive conversions. This conversion helper caches each successful conversion for re-use at a later time; failed conversions are not cached in any way, since it is difficult to raise a meaningful exception providing information about the specific failure. \end{classdesc} \begin{classdesc}{RangeCheckedConversion}{conversion\optional{, min\optional{, max}}} Helper that performs range checks on the result of another conversion. Values passed to instances of this conversion are converted using \var{conversion} and then range checked. \var{min} and \var{max}, if given and not \code{None}, are the inclusive endpoints of the allowed range. Values returned by \var{conversion} which lay outside the range described by \var{min} and \var{max} cause \exception{ValueError} to be raised. \end{classdesc} \begin{classdesc}{RegularExpressionConversion}{regex} Conversion that checks that the input matches the regular expression \var{regex}. If it matches, returns the input, otherwise raises \exception{ValueError}. \end{classdesc} \section{\module{ZConfig.loader} --- Resource loading support} \declaremodule{}{ZConfig.loader} \modulesynopsis{Support classes for resource loading} This module provides some helper classes used by the primary APIs exported by the \module{ZConfig} package. These classes may be useful for some applications, especially applications that want to use a non-default data type registry. \begin{classdesc}{Resource}{file, url\optional{, fragment}} Object that allows an open file object and a URL to be bound together to ease handling. Instances have the attributes \member{file}, \member{url}, and \member{fragment} which store the constructor arguments. These objects also have a \method{close()} method which will call \method{close()} on \var{file}, then set the \member{file} attribute to \code{None} and the \member{closed} to \constant{True}. \end{classdesc} \begin{classdesc}{BaseLoader}{} Base class for loader objects. This should not be instantiated directly, as the \method{loadResource()} method must be overridden for the instance to be used via the public API. \end{classdesc} \begin{classdesc}{ConfigLoader}{schema} Loader for configuration files. Each configuration file must conform to the schema \var{schema}. The \method{load*()} methods return a tuple consisting of the configuration object and a composite handler. \end{classdesc} \begin{classdesc}{SchemaLoader}{\optional{registry}} Loader that loads schema instances. All schema loaded by a \class{SchemaLoader} will use the same data type registry. If \var{registry} is provided and not \code{None}, it will be used, otherwise an instance of \class{ZConfig.datatypes.Registry} will be used. \end{classdesc} \subsection{Loader Objects} Loader objects provide a general public interface, an interface which subclasses must implement, and some utility methods. The following methods provide the public interface: \begin{methoddesc}[loader]{loadURL}{url} Open and load a resource specified by the URL \var{url}. This method uses the \method{loadResource()} method to perform the actual load, and returns whatever that method returns. \end{methoddesc} \begin{methoddesc}[loader]{loadFile}{file\optional{, url}} Load from an open file object, \var{file}. If given and not \code{None}, \var{url} should be the URL of the resource represented by \var{file}. If omitted or \code{None}, the \member{name} attribute of \var{file} is used to compute a \code{file:} URL, if present. This method uses the \method{loadResource()} method to perform the actual load, and returns whatever that method returns. \end{methoddesc} The following method must be overridden by subclasses: \begin{methoddesc}[loader]{loadResource}{resource} Subclasses of \class{BaseLoader} must implement this method to actually load the resource and return the appropriate application-level object. \end{methoddesc} The following methods can be used as utilities: \begin{methoddesc}[loader]{isPath}{s} Return true if \var{s} should be considered a filesystem path rather than a URL. \end{methoddesc} \begin{methoddesc}[loader]{normalizeURL}{url-or-path} Return a URL for \var{url-or-path}. If \var{url-or-path} refers to an existing file, the corresponding \code{file:} URL is returned. Otherwise \var{url-or-path} is checked for sanity: if it does not have a schema, \exception{ValueError} is raised, and if it does have a fragment identifier, \exception{ConfigurationError} is raised. This uses \method{isPath()} to determine whether \var{url-or-path} is a URL of a filesystem path. \end{methoddesc} \begin{methoddesc}[loader]{openResource}{url} Returns a resource object that represents the URL \var{url}. The URL is opened using the \function{urllib2.urlopen()} function, and the returned resource object is created using \method{createResource()}. If the URL cannot be opened, \exception{ConfigurationError} is raised. \end{methoddesc} \begin{methoddesc}[loader]{createResource}{file, url} Returns a resource object for an open file and URL, given as \var{file} and \var{url}, respectively. This may be overridden by a subclass if an alternate resource implementation is desired. \end{methoddesc} \section{\module{ZConfig.cmdline} --- Command-line override support} \declaremodule{}{ZConfig.cmdline} \modulesynopsis{Support for command-line overrides for configuration settings.} This module exports an extended version of the \class{ConfigLoader} class from the \refmodule{ZConfig.loader} module. This provides support for overriding specific settings from the configuration file from the command line, without requiring the application to provide specific options for everything the configuration file can include. \begin{classdesc}{ExtendedConfigLoader}{schema} Construct a \class{ConfigLoader} subclass that adds support for command-line overrides. \end{classdesc} The following additional method is provided, and is the only way to provide position information to associate with command-line parameters: \begin{methoddesc}{addOption}{spec\optional{, pos}} Add a single value to the list of overridden values. The \var{spec} argument is a value specified, as described for the \function{\refmodule{ZConfig}.loadConfig()} function. A source position for the specifier may be given as \var{pos}. If \var{pos} is specified and not \code{None}, it must be a sequence of three values. The first is the URL of the source (or some other identifying string). The second and third are the line number and column of the setting. These position information is only used to construct a \exception{DataConversionError} when data conversion fails. \end{methoddesc} \section{\module{ZConfig.substitution} --- String substitution} \declaremodule{}{ZConfig.substitution} \modulesynopsis{Shell-style string substitution helper.} This module provides a basic substitution facility similar to that found in the Bourne shell (\program{sh} on most \UNIX{} platforms). The replacements supported by this module include: \begin{tableiii}{l|l|c}{code}{Source}{Replacement}{Notes} \lineiii{\$\$}{\code{\$}}{(1)} \lineiii{\$\var{name}}{The result of looking up \var{name}}{(2)} \lineiii{\$\{\var{name}\}}{The result of looking up \var{name}}{} \end{tableiii} \noindent Notes: \begin{description} \item[(1)] This is different from the Bourne shell, which uses \code{\textbackslash\$} to generate a \character{\$} in the result text. This difference avoids having as many special characters in the syntax. \item[(2)] Any character which immediately follows \var{name} may not be a valid character in a name. \end{description} In each case, \var{name} is a non-empty sequence of alphanumeric and underscore characters not starting with a digit. If there is not a replacement for \var{name}, the exception \exception{SubstitutionReplacementError} is raised. Note that the lookup is expected to be case-insensitive; this module will always use a lower-case version of the name to perform the query. This module provides these functions: \begin{funcdesc}{substitute}{s, mapping} Substitute values from \var{mapping} into \var{s}. \var{mapping} can be a \class{dict} or any type that supports the \method{get()} method of the mapping protocol. Replacement values are copied into the result without further interpretation. Raises \exception{SubstitutionSyntaxError} if there are malformed constructs in \var{s}. \end{funcdesc} \begin{funcdesc}{isname}{s} Returns \constant{True} if \var{s} is a valid name for a substitution text, otherwise returns \constant{False}. \end{funcdesc} \subsection{Examples} \begin{verbatim} >>> from ZConfig.substitution import substitute >>> d = {'name': 'value', ... 'top': '$middle', ... 'middle' : 'bottom'} >>> >>> substitute('$name', d) 'value' >>> substitute('$top', d) '$middle' \end{verbatim} \appendix \section{Schema Document Type Definition \label{schema-dtd}} The following is the XML Document Type Definition for \module{ZConfig} schema: \verbatiminput{schema.dtd} \end{document} Zope-2.8.7-final/lib/python/ZConfig/doc/README.txt0000664000076400007640000000121610435500361020426 0ustar ajungajungThe zconfig.tex document in this directory contains the reference documentation for the ZConfig package. This documentation is written using the Python LaTeX styles. To format the documentation, get a copy of the Python documentation tools (the Doc/ directory from the Python sources), and create a symlink to the tools/mkhowto script from some convenient bin/ directory. You will need to have a fairly complete set of documentation tools installed on your platform; see http://www.python.org/doc/current/doc/doc.html for more information on the tools. This documentation requires the latest version of the Python documentation tools from CVS. Zope-2.8.7-final/lib/python/ZConfig/doc/xmlmarkup.perl0000664000076400007640000000350510435500361021637 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # LaTeX2HTML support for the xmlmarkup package. Doesn't do indexing. package main; sub do_cmd_element{ local($_) = @_; my $name = next_argument(); return "$name" . $_; } sub do_cmd_attribute{ local($_) = @_; my $name = next_argument(); return "$name" . $_; } sub do_env_attributedesc{ local($_) = @_; my $name = next_argument(); my $valuetype = next_argument(); return ("\n
" . "\n
$name" . "   ($valuetype)" . "\n
" . $_ . "
"); } sub do_env_elementdesc{ local($_) = @_; my $name = next_argument(); my $contentmodel = next_argument(); return ("\n
" . "\n
<" . "$name>" . "\n
$contentmodel" . "\n
</" . "$name>" . "\n
" . $_ . "
"); } 1; # Must end with this, because Perl is bogus. Zope-2.8.7-final/lib/python/ZConfig/doc/xmlmarkup.sty0000664000076400007640000000256510435500361021521 0ustar ajungajung%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Copyright (c) 2003 Zope Corporation and Contributors. % All Rights Reserved. % % This software is subject to the provisions of the Zope Public License, % Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. % THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED % WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED % WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS % FOR A PARTICULAR PURPOSE. % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Define some simple markup for the LaTeX command documentation: \ProvidesPackage{xmlmarkup} \RequirePackage{python} % fulllineitems environment \newcommand{\element}[1]{\code{#1}} \newcommand{\attribute}[1]{\code{#1}} % \begin{elementdesc}{type}{content-model} \newenvironment{elementdesc}[2]{ \begin{fulllineitems} \item[\code{\textless{\bfseries #1}\textgreater}] \code{#2} \item[\code{\textless/{\bfseries #1}\textgreater}] \index{#1 element@\py@idxcode{#1} element} \index{elements!#1@\py@idxcode{#1}} }{\end{fulllineitems}} % \begin{attributedesc}{name}{content-type} \newenvironment{attributedesc}[2]{ \begin{fulllineitems} \item[\code{\bfseries#1}{\quad(#2)}] \index{#1@\py@idxcode{#1}} }{\end{fulllineitems}} Zope-2.8.7-final/lib/python/ZConfig/doc/schema.dtd0000664000076400007640000000655510435500361020700 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/0000775000076400007640000000000010435610645017334 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/test_cmdline.py0000664000076400007640000001477310435500360022364 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the command-line integration.""" import unittest import ZConfig from ZConfig.cmdline import ExtendedConfigLoader from ZConfig.tests.support import TestBase class CommandLineTest(TestBase): def create_config_loader(self, schema): loader = ExtendedConfigLoader(schema) for item in self.clopts: loader.addOption(*item) return loader def test_loading(self): schema = self.load_schema_text("""\
""") self.clopts = [("mykey=splat!", None), ("section/innerkey=spoogey", None)] bag = self.create_config_loader(schema).cook() # Test a variety of queries on the OptionBag: self.assert_(bag.has_key("mykey")) self.assert_(not bag.has_key("another")) self.assertEqual(bag.get_section_info("st", None), None) self.assertEqual(bag.get_section_info("st", "missing-sect"), None) # Consume everything in the OptionBag: L = bag.get_key("mykey") s, pos = L[0] self.assertEqual(len(L), 1) self.assertEqual(s, "splat!") bag2 = bag.get_section_info("st", "section") self.assert_(bag2.has_key("innerkey")) self.assert_(not bag2.has_key("another")) L = bag2.get_key("innerkey") s, pos = L[0] self.assertEqual(len(L), 1) self.assertEqual(s, "spoogey") # "Finish" to make sure everything has been consumed: bag2.finish() bag.finish() def test_named_sections(self): schema = self.load_schema_text("""\
""") self.clopts = [("foo/k1=v1", None), ("bar/k2=v2", ("someurl", 2, 3))] bag = self.create_config_loader(schema).cook() foo = bag.get_section_info("st2", "foo") bar = bag.get_section_info("st2", "bar") bag.finish() self.assertEqual(bar.get_key("k2"), [("v2", ("someurl", 2, 3))]) bar.finish() # Ignore foo for now; it's not really important *when* it fails. simple_schema = None def get_simple_schema(self): if self.simple_schema is None: self.__class__.simple_schema = self.load_schema_text("""\ """) return self.simple_schema def test_reading_config(self): self.clopts = [("k1=stringvalue", None), ("k2=12", None)] schema = self.get_simple_schema() conf = self.load_config_text(schema, """\ k0 stuff k1 replaced-stuff k2 42 """) self.assertEqual(conf.k0, "stuff") self.assertEqual(conf.k1, "stringvalue") self.assertEqual(conf.k2, 12) self.assertEqual(conf.k3, 19) def test_unknown_key(self): self.clopts = [("foo=bar", None)] schema = self.get_simple_schema() self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_too_many_keys(self): self.clopts = [("k1=v1", None), ("k1=v2", None)] schema = self.get_simple_schema() self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_bad_datatype(self): self.clopts = [("k2=42.0", None)] schema = self.get_simple_schema() self.assertRaises(ZConfig.DataConversionError, self.load_config_text, schema, "") def test_without_clopts(self): self.clopts = [] schema = self.get_simple_schema() conf = self.load_config_text(schema, "k3 42") self.assertEqual(conf.k0, None) self.assertEqual(conf.k1, None) self.assertEqual(conf.k2, None) self.assertEqual(conf.k3, 42) def test_section_contents(self): schema = self.load_schema_text("""\ k3-v1 k3-v2 k3-v3
""") self.clopts = [("s1/k1=foo", None), ("s2/k3=value1", None), ("s2/k3=value2", None), ("s1/k2=99", None), ("s2/k3=value3", None), ("s2/k3=value4", None), ] conf = self.load_config_text(schema, "\n") self.assertEqual(conf.s1.k1, "foo") self.assertEqual(conf.s1.k2, 99) self.assertEqual(conf.s1.k3, ["k3-v1", "k3-v2", "k3-v3"]) self.assertEqual(conf.s2.k1, None) self.assertEqual(conf.s2.k2, 3) self.assertEqual(conf.s2.k3, ["value1", "value2", "value3", "value4"]) def test_suite(): return unittest.makeSuite(CommandLineTest) if __name__ == "__main__": unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZConfig/tests/test_cookbook.py0000664000076400007640000000451410435500360022547 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of examples from the online cookbook, so we don't break them down the road. Unless we really mean to. The ZConfig Cookbook is available online at: http://dev.zope.org/Zope3/ZConfig """ import unittest from ZConfig.tests.support import TestBase def basic_key_mapping_password_to_passwd(key): # Lower-case the key since that's what basic-key does: key = key.lower() # Now map password to passwd: if key == "password": key = "passwd" return key def user_info_conversion(section): return section class CookbookTestCase(TestBase): def test_rewriting_key_names(self): schema = self.load_schema_text("""
""" % __name__) config = self.load_config_text(schema, """\ USERID 42 USERNAME foouser PASSWORD yeah-right """) self.assertEqual(config.userinfo.userid, 42) self.assertEqual(config.userinfo.username, "foouser") self.assertEqual(config.userinfo.passwd, "yeah-right") self.assert_(not hasattr(config.userinfo, "password")) def test_suite(): return unittest.makeSuite(CookbookTestCase) if __name__ == "__main__": unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZConfig/tests/runtests.py0000775000076400007640000000400310435500360021565 0ustar ajungajung#! /usr/bin/env python ############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Script to run all the regression tests for the ZConfig package.""" import os import sys import unittest if __name__ == "__main__": __file__ = sys.argv[0] TESTDIR = os.path.dirname(os.path.abspath(__file__)) PKGDIR = os.path.dirname(TESTDIR) # the ZConfig package directory TOPDIR = os.path.dirname(PKGDIR) COMPONENTS = os.path.join(PKGDIR, "components") TESTDIRS = { "ZConfig.tests": TESTDIR, "ZConfig.components.basic.tests": os.path.join(COMPONENTS, "basic", "tests"), "ZConfig.components.logger.tests": os.path.join(COMPONENTS, "logger", "tests"), } def load_tests(pkgname, name): name = "%s.%s" % (pkgname, name) __import__(name) mod = sys.modules[name] return mod.test_suite() def test_suite(): L = [] for pkgname, path in TESTDIRS.items(): for fn in os.listdir(path): name, ext = os.path.splitext(fn) if name[:4] == "test" and ext == ".py": L.append(load_tests(pkgname, name)) if len(L) == 1: return L[0] else: suite = unittest.TestSuite() for t in L: suite.addTest(t) return suite if __name__ == "__main__": if TOPDIR not in sys.path: sys.path.insert(0, TOPDIR) unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZConfig/tests/__init__.py0000664000076400007640000000137310435500360021441 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for the configuration data structures and loader. $Id: __init__.py,v 1.2 2003/01/03 21:05:56 fdrake Exp $ """ Zope-2.8.7-final/lib/python/ZConfig/tests/input/0000775000076400007640000000000010435500360020463 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/input/logger.xml0000664000076400007640000000062410435500360022466 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/inner.conf0000664000076400007640000000005210435500360022442 0ustar ajungajungrefouter $outervar %define innervar inner Zope-2.8.7-final/lib/python/ZConfig/tests/input/simplesections.conf0000664000076400007640000000063110435500360024373 0ustar ajungajungvar foo var-0 foo-0
var bar var-one splat
var-1 foo-1
var spam var-two stuff
var-2 foo-2
var quack! var-three yet
var-3 foo-3 # An anonymous empty section:
var-4 foo-4 # A fairly trivial section: var triv var-5 foo-5 # A minimal section: var-6 foo-6 Zope-2.8.7-final/lib/python/ZConfig/tests/input/simple.xml0000664000076400007640000000166710435500360022510 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/library.xml0000664000076400007640000000023510435500360022651 0ustar ajungajung Sample library of reusable data types. Zope-2.8.7-final/lib/python/ZConfig/tests/input/base-datatype1.xml0000664000076400007640000000014210435500360024006 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/simplesections.xml0000664000076400007640000000120110435500360024240 0ustar ajungajung
Zope-2.8.7-final/lib/python/ZConfig/tests/input/base.xml0000664000076400007640000000006210435500360022115 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/include.conf0000664000076400007640000000007010435500360022752 0ustar ajungajungvar2 value2 %include simple.conf var3 value3 var4 $name Zope-2.8.7-final/lib/python/ZConfig/tests/input/base-keytype2.xml0000664000076400007640000000014010435500360023664 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/simple.conf0000664000076400007640000000067010435500360022626 0ustar ajungajungempty var1 abc int-var 12 float-var 12.02 neg-int -2 true-var-1 true true-var-2 on true-var-3 yes false-var-1 false false-var-2 off false-var-3 no list-1 list-2 abc list-3 abc def ghi list-4 [ what now? ] # These test the %define mechanism: %define dollars $$$$ %define empty %define name value %define twowords two words getname $name getnametwice $name${name} getdollars $dollars getempty x${empty}y getwords abc $twowords def Zope-2.8.7-final/lib/python/ZConfig/tests/input/base-datatype2.xml0000664000076400007640000000014210435500360024007 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/input/outer.conf0000664000076400007640000000007610435500360022473 0ustar ajungajung%define outervar outer %include inner.conf refinner $innervar Zope-2.8.7-final/lib/python/ZConfig/tests/input/base-keytype1.xml0000664000076400007640000000010610435500360023665 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/test_datatypes.py0000664000076400007640000003470510435500360022744 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of standard ZConfig datatypes.""" import os import sys import shutil import socket import datetime import tempfile import unittest import ZConfig.datatypes try: here = __file__ except NameError: here = sys.argv[0] here = os.path.abspath(here) try: unicode except NameError: have_unicode = False else: have_unicode = True class DatatypeTestCase(unittest.TestCase): types = ZConfig.datatypes.Registry() def test_datatype_basickey(self): convert = self.types.get("basic-key") eq = self.assertEqual raises = self.assertRaises eq(convert("abc"), "abc") eq(convert("ABC_DEF.123"), "abc_def.123") eq(convert("Abc-Def-456"), "abc-def-456") eq(convert("Abc.Def"), "abc.def") raises(ValueError, convert, "_abc") raises(ValueError, convert, "-abc") raises(ValueError, convert, "123") raises(ValueError, convert, "") def test_datatype_boolean(self): convert = self.types.get("boolean") check = self.assert_ raises = self.assertRaises check(convert("on")) check(convert("true")) check(convert("yes")) check(not convert("off")) check(not convert("false")) check(not convert("no")) raises(ValueError, convert, '0') raises(ValueError, convert, '1') raises(ValueError, convert, '') raises(ValueError, convert, 'junk') def test_datatype_float(self): convert = self.types.get("float") eq = self.assertEqual raises = self.assertRaises eq(convert("1"), 1.0) self.assert_(type(convert(1)) is type(1.0)) eq(convert("1.1"), 1.1) eq(convert("50.50"), 50.50) eq(convert("-50.50"), -50.50) eq(convert(0), 0.0) eq(convert("0"), 0.0) eq(convert("-0"), 0.0) eq(convert("0.0"), 0.0) raises(ValueError, convert, "junk") raises(ValueError, convert, "0x234.1.9") raises(ValueError, convert, "0.9-") # These are not portable representations; make sure they are # disallowed everywhere for consistency. raises(ValueError, convert, "inf") raises(ValueError, convert, "-inf") raises(ValueError, convert, "nan") if have_unicode: raises(ValueError, convert, unicode("inf")) raises(ValueError, convert, unicode("-inf")) raises(ValueError, convert, unicode("nan")) def test_datatype_identifier(self): convert = self.types.get("identifier") raises = self.assertRaises self.check_names(convert) self.check_never_namelike(convert) raises(ValueError, convert, ".abc") def check_names(self, convert): eq = self.assert_ascii_equal eq(convert, "AbcDef") eq(convert, "a________") eq(convert, "abc_def") eq(convert, "int123") eq(convert, "_abc") eq(convert, "_123") eq(convert, "__dict__") def assert_ascii_equal(self, convert, value): v = convert(value) self.assertEqual(v, value) self.assert_(isinstance(v, str)) if have_unicode: unicode_value = unicode(value) v = convert(unicode_value) self.assertEqual(v, value) self.assert_(isinstance(v, str)) def check_never_namelike(self, convert): raises = self.assertRaises raises(ValueError, convert, "2345") raises(ValueError, convert, "23.45") raises(ValueError, convert, ".45") raises(ValueError, convert, "23.") raises(ValueError, convert, "abc.") raises(ValueError, convert, "-abc") raises(ValueError, convert, "-123") raises(ValueError, convert, "abc-") raises(ValueError, convert, "123-") raises(ValueError, convert, "-") raises(ValueError, convert, ".") raises(ValueError, convert, "&%$*()") raises(ValueError, convert, "") def test_datatype_dotted_name(self): convert = self.types.get("dotted-name") raises = self.assertRaises self.check_names(convert) self.check_dotted_names(convert) self.check_never_namelike(convert) raises(ValueError, convert, "abc.") raises(ValueError, convert, ".abc.") raises(ValueError, convert, "abc.def.") raises(ValueError, convert, ".abc.def.") raises(ValueError, convert, ".abc.def") def test_datatype_dotted_suffix(self): convert = self.types.get("dotted-suffix") eq = self.assert_ascii_equal raises = self.assertRaises self.check_names(convert) self.check_dotted_names(convert) self.check_never_namelike(convert) eq(convert, ".a") eq(convert, ".a.b") eq(convert, ".a.b.c.d.e.f.g.h.i.j.k.l.m.n.o") raises(ValueError, convert, "abc.") raises(ValueError, convert, ".abc.") raises(ValueError, convert, "abc.def.") raises(ValueError, convert, ".abc.def.") def check_dotted_names(self, convert): eq = self.assert_ascii_equal eq(convert, "abc.def") eq(convert, "abc.def.ghi") eq(convert, "a.d.g.g.g.g.g.g.g") def test_datatype_inet_address(self): convert = self.types.get("inet-address") eq = self.assertEqual defhost = ZConfig.datatypes.DEFAULT_HOST eq(convert("Host.Example.Com:80"), ("host.example.com", 80)) eq(convert(":80"), (defhost, 80)) eq(convert("80"), (defhost, 80)) eq(convert("host.EXAMPLE.com"), ("host.example.com", None)) self.assertRaises(ValueError, convert, "40 # foo") def test_datatype_inet_binding_address(self): convert = self.types.get("inet-binding-address") eq = self.assertEqual defhost = "" eq(convert("Host.Example.Com:80"), ("host.example.com", 80)) eq(convert(":80"), (defhost, 80)) eq(convert("80"), (defhost, 80)) eq(convert("host.EXAMPLE.com"), ("host.example.com", None)) self.assertRaises(ValueError, convert, "40 # foo") def test_datatype_inet_connection_address(self): convert = self.types.get("inet-connection-address") eq = self.assertEqual defhost = "127.0.0.1" eq(convert("Host.Example.Com:80"), ("host.example.com", 80)) eq(convert(":80"), (defhost, 80)) eq(convert("80"), (defhost, 80)) eq(convert("host.EXAMPLE.com"), ("host.example.com", None)) self.assertRaises(ValueError, convert, "40 # foo") def test_datatype_integer(self): convert = self.types.get("integer") eq = self.assertEqual raises = self.assertRaises eq(convert('-100'), -100) eq(convert('-1'), -1) eq(convert('-0'), 0) eq(convert('0'), 0) eq(convert('1'), 1) eq(convert('100'), 100) eq(convert('65535'), 65535) eq(convert('65536'), 65536) big = sys.maxint + 1L # Python 2.1 needs the L suffix here s = str(big) # s won't have the suffix eq(convert(s), big) eq(convert("-" + s), -big) raises(ValueError, convert, 'abc') raises(ValueError, convert, '-0xabc') raises(ValueError, convert, '') raises(ValueError, convert, '123 456') raises(ValueError, convert, '123-') def test_datatype_locale(self): convert = self.types.get("locale") # Python supports "C" even when the _locale module is not available self.assertEqual(convert("C"), "C") self.assertRaises(ValueError, convert, "locale-does-not-exist") def test_datatype_port(self): convert = self.types.get("port-number") eq = self.assertEqual raises = self.assertRaises raises(ValueError, convert, '-1') raises(ValueError, convert, '0') eq(convert('1'), 1) eq(convert('80'), 80) eq(convert('1023'), 1023) eq(convert('1024'), 1024) eq(convert('60000'), 60000) eq(convert('65535'), 0xffff) raises(ValueError, convert, '65536') def test_datatype_socket_address(self): convert = self.types.get("socket-address") eq = self.assertEqual AF_INET = socket.AF_INET defhost = ZConfig.datatypes.DEFAULT_HOST def check(value, family, address, self=self, convert=convert): a = convert(value) self.assertEqual(a.family, family) self.assertEqual(a.address, address) check("Host.Example.Com:80", AF_INET, ("host.example.com", 80)) check(":80", AF_INET, (defhost, 80)) check("80", AF_INET, (defhost, 80)) check("host.EXAMPLE.com", AF_INET, ("host.example.com",None)) a1 = convert("/tmp/var/@345.4") a2 = convert("/tmp/var/@345.4:80") self.assertEqual(a1.address, "/tmp/var/@345.4") self.assertEqual(a2.address, "/tmp/var/@345.4:80") if hasattr(socket, "AF_UNIX"): self.assertEqual(a1.family, socket.AF_UNIX) self.assertEqual(a2.family, socket.AF_UNIX) else: self.assert_(a1.family is None) self.assert_(a2.family is None) def test_ipaddr_or_hostname(self): convert = self.types.get('ipaddr-or-hostname') eq = self.assertEqual raises = self.assertRaises eq(convert('hostname'), 'hostname') eq(convert('hostname.com'), 'hostname.com') eq(convert('www.hostname.com'), 'www.hostname.com') eq(convert('HOSTNAME'), 'hostname') eq(convert('HOSTNAME.COM'), 'hostname.com') eq(convert('WWW.HOSTNAME.COM'), 'www.hostname.com') eq(convert('127.0.0.1'), '127.0.0.1') raises(ValueError, convert, '1hostnamewithleadingnumeric') raises(ValueError, convert, '255.255') raises(ValueError, convert, '12345678') raises(ValueError, convert, '999.999.999.999') raises(ValueError, convert, 'a!badhostname') def test_existing_directory(self): convert = self.types.get('existing-directory') eq = self.assertEqual raises = self.assertRaises eq(convert('.'), '.') eq(convert(os.path.dirname(here)), os.path.dirname(here)) raises(ValueError, convert, tempfile.mktemp()) def test_existing_file(self): convert = self.types.get('existing-file') eq = self.assertEqual raises = self.assertRaises eq(convert('.'), '.') eq(convert(here), here) raises(ValueError, convert, tempfile.mktemp()) def test_existing_path(self): convert = self.types.get('existing-path') eq = self.assertEqual raises = self.assertRaises eq(convert('.'), '.') eq(convert(here), here) eq(convert(os.path.dirname(here)), os.path.dirname(here)) raises(ValueError, convert, tempfile.mktemp()) def test_existing_dirpath(self): convert = self.types.get('existing-dirpath') eq = self.assertEqual raises = self.assertRaises eq(convert('.'), '.') eq(convert(here), here) raises(ValueError, convert, '/a/hopefully/nonexistent/path') raises(ValueError, convert, here + '/bogus') def test_byte_size(self): eq = self.assertEqual raises = self.assertRaises convert = self.types.get('byte-size') eq(convert('128'), 128) eq(convert('128KB'), 128*1024) eq(convert('128MB'), 128*1024*1024) eq(convert('128GB'), 128*1024*1024*1024L) raises(ValueError, convert, '128TB') eq(convert('128'), 128) eq(convert('128kb'), 128*1024) eq(convert('128mb'), 128*1024*1024) eq(convert('128gb'), 128*1024*1024*1024L) raises(ValueError, convert, '128tb') def test_time_interval(self): eq = self.assertEqual raises = self.assertRaises convert = self.types.get('time-interval') eq(convert('120'), 120) eq(convert('120S'), 120) eq(convert('120M'), 120*60) eq(convert('120H'), 120*60*60) eq(convert('120D'), 120*60*60*24) raises(ValueError, convert, '120W') eq(convert('120'), 120) eq(convert('120s'), 120) eq(convert('120m'), 120*60) eq(convert('120h'), 120*60*60) eq(convert('120d'), 120*60*60*24) raises(ValueError, convert, '120w') def test_timedelta(self): eq = self.assertEqual raises = self.assertRaises convert = self.types.get('timedelta') eq(convert('4w'), datetime.timedelta(weeks=4)) eq(convert('2d'), datetime.timedelta(days=2)) eq(convert('7h'), datetime.timedelta(hours=7)) eq(convert('12m'), datetime.timedelta(minutes=12)) eq(convert('14s'), datetime.timedelta(seconds=14)) eq(convert('4w 2d 7h 12m 14s'), datetime.timedelta(2, 14, minutes=12, hours=7, weeks=4)) class RegistryTestCase(unittest.TestCase): def test_registry_does_not_mask_toplevel_imports(self): old_sys_path = sys.path[:] tmpdir = tempfile.mkdtemp(prefix="test_datatypes_") fn = os.path.join(tmpdir, "datatypes.py") f = open(fn, "w") f.write(TEST_DATATYPE_SOURCE) f.close() registry = ZConfig.datatypes.Registry() # we really want the temp area to override everything else: sys.path.insert(0, tmpdir) try: datatype = registry.get("datatypes.my_sample_datatype") finally: shutil.rmtree(tmpdir) sys.path[:] = old_sys_path self.assertEqual(datatype, 42) TEST_DATATYPE_SOURCE = """ # sample datatypes file my_sample_datatype = 42 """ def test_suite(): suite = unittest.makeSuite(DatatypeTestCase) suite.addTest(unittest.makeSuite(RegistryTestCase)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZConfig/tests/test_cfgimports.py0000664000076400007640000000415410435500360023116 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the %import mechanism. $Id: test_cfgimports.py,v 1.1 2003/10/03 20:01:57 fdrake Exp $ """ import unittest from StringIO import StringIO import ZConfig import ZConfig.tests.support class TestImportFromConfiguration(ZConfig.tests.support.TestBase): def test_simple_import(self): schema = self.load_schema_text("") loader = self.create_config_loader(schema) config, _ = loader.loadFile( StringIO("%import ZConfig.tests.library.widget\n")) # make sure we now have a "private" schema object; the only # way to get it is from the loader itself self.assert_(schema is not loader.schema) # make sure component types are only found on the private schema: loader.schema.gettype("widget-b") self.assertRaises(ZConfig.SchemaError, schema.gettype, "widget-b") def test_repeated_import(self): schema = self.load_schema_text("") loader = self.create_config_loader(schema) config, _ = loader.loadFile( StringIO("%import ZConfig.tests.library.widget\n" "%import ZConfig.tests.library.widget\n")) def test_missing_import(self): schema = self.load_schema_text("") loader = self.create_config_loader(schema) self.assertRaises(ZConfig.SchemaError, loader.loadFile, StringIO("%import ZConfig.tests.missing\n")) def test_suite(): return unittest.makeSuite(TestImportFromConfiguration) Zope-2.8.7-final/lib/python/ZConfig/tests/test_schema.py0000664000076400007640000012115710435500360022204 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of ZConfig schemas.""" import unittest import ZConfig from ZConfig.tests.support import TestBase, CONFIG_BASE def uppercase(value): return str(value).upper() def appsection(value): return MySection(value) def get_foo(section): return section.foo class MySection: def __init__(self, value): self.conf = value def get_section_attributes(section): L = list(section.getSectionAttributes()) L.sort() return L class SchemaTestCase(TestBase): """Tests of the basic schema support itself.""" def test_minimal_schema(self): schema = self.load_schema_text("") self.assertEqual(len(schema), 0) self.assertRaises(IndexError, lambda schema=schema: schema[0]) self.assertRaises(ZConfig.ConfigurationError, schema.getinfo, "foo") def test_simple(self): schema, conf = self.load_both("simple.xml", "simple.conf") self._verifySimpleConf(conf) def _verifySimpleConf(self,conf): eq = self.assertEqual eq(conf.var1, 'abc') eq(conf.int_var, 12) eq(conf.float_var, 12.02) eq(conf.neg_int, -2) check = self.assert_ check(conf.true_var_1) check(conf.true_var_2) check(conf.true_var_3) check(not conf.false_var_1) check(not conf.false_var_2) check(not conf.false_var_3) def test_app_datatype(self): dtname = __name__ + ".uppercase" schema = self.load_schema_text("""\ abc abc not lower case """ % (dtname, dtname, dtname, dtname)) conf = self.load_config_text(schema, """\ a qwerty c upp c er c case """) eq = self.assertEqual eq(conf.a, 'QWERTY') eq(conf.b, 'ABC') eq(conf.c, ['UPP', 'ER', 'CASE']) eq(conf.d, ['NOT', 'LOWER', 'CASE']) eq(get_section_attributes(conf), ["a", "b", "c", "d"]) def test_app_sectiontype(self): schema = self.load_schema_text("""\
""" % __name__) conf = self.load_config_text(schema, """\ sample 42 """) self.assert_(isinstance(conf, MySection)) o1 = conf.conf.sect self.assert_(isinstance(o1, MySection)) self.assertEqual(o1.conf.sample, 42) def test_empty_sections(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\
""") self.assert_(conf.s1 is not None) self.assert_(conf.s2 is not None) self.assertEqual(get_section_attributes(conf), ["s1", "s2"]) def test_deeply_nested_sections(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ key sect3-value key sect2-value """) eq = self.assertEqual eq(conf.sect.sect.sect.key, "type1-value") eq(conf.sect.sect.key, "sect2-value") eq(conf.sect.key, "sect3-value") eq(get_section_attributes(conf), ["sect"]) eq(get_section_attributes(conf.sect), ["key", "sect"]) eq(get_section_attributes(conf.sect.sect), ["key", "sect"]) eq(get_section_attributes(conf.sect.sect.sect), ["key"]) def test_multivalued_keys(self): schema = self.load_schema_text("""\ 1 2 3 4 5 """) conf = self.load_config_text(schema, """\ a foo a bar c 41 c 42 c 43 """, num_handlers=2) L = [] self.handlers({'abc': L.append, 'DEF': L.append}) self.assertEqual(L, [['foo', 'bar'], conf]) L = [] self.handlers({'abc': None, 'DEF': L.append}) self.assertEqual(L, [conf]) self.assertEqual(conf.a, ['foo', 'bar']) self.assertEqual(conf.b, [1, 2]) self.assertEqual(conf.c, [41, 42, 43]) self.assertEqual(conf.d, []) self.assertEqual(get_section_attributes(conf), ["a", "b", "c", "d"]) def test_multikey_required(self): schema = self.load_schema_text("""\ """) self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_multisection_required(self): schema = self.load_schema_text("""\ """) self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_key_required_but_missing(self): schema = self.load_schema_text("""\ """) self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_section_required_but_missing(self): schema = self.load_schema_text("""\
""") self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_key_default_element(self): self.assertRaises( ZConfig.SchemaError, self.load_schema_text, """\ text """) def test_bad_handler_maps(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, """\ a foo b bar """, num_handlers=2) self.assertEqual(get_section_attributes(conf), ["a", "b"]) self.assertRaises(ZConfig.ConfigurationError, self.handlers, {'abc': id, 'ABC': id, 'def': id}) self.assertRaises(ZConfig.ConfigurationError, self.handlers, {}) def test_handler_ordering(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ """, num_handlers=3) L = [] self.handlers({'a': L.append, 'b': L.append, 'c': L.append}) outer = conf.sect_outer inner = outer.sect_inner self.assertEqual(L, [inner, outer, conf]) def test_duplicate_section_names(self): schema = self.load_schema_text("""\
""") self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, """\ """) conf = self.load_config_text(schema, """\ """) def test_disallowed_duplicate_attribute(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) def test_unknown_datatype_name(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, "") def test_load_abstracttype(self): schema = self.load_schema_text("""\ This is an abstract section type. """) # check the types that get defined t = schema.gettype("group") self.assert_(t.isabstract()) t1 = schema.gettype("t1") self.assert_(not t1.isabstract()) self.assert_(t.getsubtype("t1") is t1) t2 = schema.gettype("t2") self.assert_(not t2.isabstract()) self.assert_(t.getsubtype("t2") is t2) self.assertRaises(ZConfig.ConfigurationError, t.getsubtype, "group") self.assert_(t1 is not t2) # try loading a config that relies on this schema conf = self.load_config_text(schema, """\ k1 value1 k2 value2 """) eq = self.assertEqual eq(get_section_attributes(conf), ["g"]) eq(len(conf.g), 4) eq(conf.g[0].k1, "default1") eq(conf.g[1].k1, "value1") eq(conf.g[2].k2, "default2") eq(conf.g[3].k2, "value2") # white box: self.assert_(conf.g[0].getSectionDefinition() is t1) self.assert_(conf.g[1].getSectionDefinition() is t1) self.assert_(conf.g[2].getSectionDefinition() is t2) self.assert_(conf.g[3].getSectionDefinition() is t2) def test_abstracttype_extension(self): schema = self.load_schema_text("""\
""") abstype = schema.gettype("group") self.assert_(schema.gettype("extra") is abstype.getsubtype("extra")) # make sure we can use the extension in a config: conf = self.load_config_text(schema, "") self.assertEqual(conf.thing.getSectionType(), "extra") self.assertEqual(get_section_attributes(conf), ["thing"]) self.assertEqual(get_section_attributes(conf.thing), []) def test_abstracttype_extension_errors(self): # specifying a non-existant abstracttype self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) # specifying something that isn't an abstracttype self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) def test_arbitrary_key(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, "some-key 42") self.assertEqual(conf.keymap, {'some-key': 42}) self.assertEqual(get_section_attributes(conf), ["keymap"]) def test_arbitrary_multikey_required(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, """\ some-key 42 some-key 43 """) self.assertEqual(conf.keymap, {'some-key': [42, 43]}) def test_arbitrary_multikey_optional(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ some-key 42 some-key 43 """) self.assertEqual(conf.stuff.keymap, {'some-key': ['42', '43']}) self.assertEqual(get_section_attributes(conf), ["stuff"]) def test_arbitrary_multikey_optional_empty(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, "") self.assertEqual(conf.stuff.keymap, {}) def test_arbitrary_multikey_with_defaults(self): schema = self.load_schema_text("""\ value-a1 value-a2 value-b """) conf = self.load_config_text(schema, "") self.assertEqual(conf.keymap, {'a': ['value-a1', 'value-a2'], 'b': ['value-b']}) def test_arbitrary_multikey_with_unkeyed_default(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ value-a1 """) def test_arbitrary_key_with_defaults(self): schema = self.load_schema_text("""\ value-a value-b """) conf = self.load_config_text(schema, "") self.assertEqual(conf.keymap, {'a': 'value-a', 'b': 'value-b'}) def test_arbitrary_key_with_unkeyed_default(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ value-a1 """) def test_arbitrary_keys_with_others(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, """\ some-key 42 k2 3 """) self.assertEqual(conf.k1, 'v1') self.assertEqual(conf.k2, 3) self.assertEqual(conf.keymap, {'some-key': 42}) self.assertEqual(get_section_attributes(conf), ["k1", "k2", "keymap"]) def test_arbitrary_key_missing(self): schema = self.load_schema_text("""\ """) self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "# empty config file") def test_arbitrary_key_bad_schema(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) def test_getrequiredtypes(self): schema = self.load_schema("library.xml") self.assertEqual(schema.getrequiredtypes(), []) schema = self.load_schema_text("""\
""") L = schema.getrequiredtypes() L.sort() self.assertEqual(L, ["used"]) def test_getunusedtypes(self): schema = self.load_schema("library.xml") L = schema.getunusedtypes() L.sort() self.assertEqual(L, ["type-a", "type-b"]) schema = self.load_schema_text("""\
""") self.assertEqual(schema.getunusedtypes(), ["unused"]) def test_section_value_mutation(self): schema, conf = self.load_both("simple.xml", "simple.conf") orig = conf.empty new = [] conf.empty = new self.assert_(conf.empty is new) def test_simple_anonymous_section(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, "") self.assertEqual(conf.attr.key, "value") def test_simple_anonymous_section_without_name(self): # make sure we get the same behavior without name='*' schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, "") self.assertEqual(conf.attr.key, "value") def test_simple_anynamed_section(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, "") self.assertEqual(conf.attr.key, "value") self.assertEqual(conf.attr.getSectionName(), "name") # if we omit the name, it's an error self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "") def test_nested_abstract_sectiontype(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ """) def test_nested_abstract_sectiontype_without_name(self): # make sure we get the same behavior without name='*' schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ """) def test_reserved_attribute_prefix(self): template = """\ %s """ def check(thing, self=self, template=template): text = template % thing self.assertRaises(ZConfig.SchemaError, self.load_schema_text, text) check("") check("") check("") check("") check("
") check("
") check("") check("") def test_sectiontype_as_schema(self): schema = self.load_schema_text("""\
""") t = schema.gettype("t") conf = self.load_config_text(t, "") self.assertEqual(conf.tkey, "tkey-default") self.assertEqual(conf.section.skey, "skey-default") self.assertEqual(get_section_attributes(conf), ["section", "tkey"]) self.assertEqual(get_section_attributes(conf.section), ["skey"]) def test_datatype_conversion_error(self): schema_url = "file:///tmp/fake-url-1.xml" config_url = "file:///tmp/fake-url-2.xml" schema = self.load_schema_text("""\ """, url=schema_url) e = self.get_data_conversion_error( schema, "", config_url) self.assertEqual(e.url, schema_url) self.assertEqual(e.lineno, 2) e = self.get_data_conversion_error(schema, """\ # comment key splat """, config_url) self.assertEqual(e.url, config_url) self.assertEqual(e.lineno, 3) def get_data_conversion_error(self, schema, src, url): try: self.load_config_text(schema, src, url=url) except ZConfig.DataConversionError, e: return e else: self.fail("expected ZConfig.DataConversionError") def test_numeric_section_name(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, "") self.assertEqual(len(conf.things), 1) def test_sectiontype_extension(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ k1 k1-value k2 k2-value """) eq = self.assertEqual eq(conf.s.k1, "k1-value") eq(conf.s.k2, "k2-value") eq(get_section_attributes(conf), ["s"]) eq(get_section_attributes(conf.s), ["k1", "k2"]) def test_sectiontype_extension_errors(self): # cannot override key from base self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) # cannot extend non-existing section self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) # cannot extend abstract type self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) def test_sectiontype_derived_keytype(self): # make sure that a derived section type inherits the keytype # of its base schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ foo bar Foo BAR """) self.assertEqual(conf.foo.foo, "bar") self.assertEqual(conf.foo.Foo, "BAR") self.assertEqual(get_section_attributes(conf.foo), ["Foo", "foo"]) def test_sectiontype_override_keytype(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ ident1 foo Ident2 bar EXAMPLE.COM foo """) L = conf.base.map.items() L.sort() self.assertEqual(L, [("Ident2", "bar"), ("ident1", "foo")]) L = conf.derived.map.items() L.sort() self.assertEqual(L, [("example.com", "foo")]) self.assertEqual(get_section_attributes(conf), ["base", "derived"]) def test_keytype_applies_to_default_key(self): schema = self.load_schema_text("""\ 42 24
""") conf = self.load_config_text(schema, "") items = conf.sect.mapping.items() items.sort() self.assertEqual(items, [("bar", "24"), ("foo", "42")]) def test_duplicate_default_key_checked_in_schema(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ 42 24
""") def test_default_keys_rechecked_clash_in_derived_sectiontype(self): # If the default values associated with a can't # be supported by a new keytype for a derived sectiontype, an # error should be indicated. self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ 42 42
""") def test_default_keys_rechecked_dont_clash_in_derived_sectiontype(self): # If the default values associated with a can't # be supported by a new keytype for a derived sectiontype, an # error should be indicated. schema = self.load_schema_text("""\ 42 42
""") conf = self.load_config_text(schema, """\ """) base = conf.base.mapping.items() base.sort() self.assertEqual(base, [("Foo", ["42"]), ("foo", ["42"])]) sect = conf.sect.mapping.items() sect.sort() self.assertEqual(sect, [("foo", ["42", "42"])]) def test_sectiontype_inherited_datatype(self): schema = self.load_schema_text("""\
""") conf = self.load_config_text(schema, """\ foo bar """) self.assertEqual(conf.splat, "bar") def test_schema_keytype(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, "host.example.com 127.0.0.1\n" "www.example.org 127.0.0.2\n") table = conf.table self.assertEqual(len(table), 2) L = table.items() L.sort() self.assertEqual(L, [("host.example.com", "127.0.0.1"), ("www.example.org", "127.0.0.2")]) self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "abc. 127.0.0.1") def test_keytype_identifier(self): schema = self.load_schema_text("""\ """) conf = self.load_config_text(schema, "Foo Foo-value\n" "foo foo-value\n") self.assertEqual(conf.foo, "foo-value") self.assertEqual(conf.Foo, "Foo-value") self.assertEqual(get_section_attributes(conf), ["Foo", "foo"]) # key mis-match based on case: self.assertRaises(ZConfig.ConfigurationError, self.load_config_text, schema, "FOO frob\n") # attribute names conflict, since the keytype isn't used to # generate attribute names self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """) def test_datatype_casesensitivity(self): self.load_schema_text("") def test_simple_extends(self): schema = self.load_schema_text("""\
""" % (CONFIG_BASE, CONFIG_BASE)) self._verifySimpleConf(self.load_config(schema, "simple.conf")) def test_extends_fragment_failure(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, "" % CONFIG_BASE) def test_multi_extends_implicit_OK(self): self.load_schema_text("""\
""" % (CONFIG_BASE, CONFIG_BASE)) def test_multi_extends_explicit_datatype_OK(self): self.load_schema_text("""\
""" % (CONFIG_BASE, CONFIG_BASE)) def test_multi_extends_explicit_keytype_OK(self): self.load_schema_text("""\
""" % (CONFIG_BASE, CONFIG_BASE, __name__)) def test_multi_extends_datatype_conflict(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """ % (CONFIG_BASE, CONFIG_BASE)) def test_multi_extends_keytype_conflict(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ """ % (CONFIG_BASE, CONFIG_BASE)) def test_multiple_descriptions_is_error(self): self.assertRaises(ZConfig.SchemaError, self.load_schema_text, """\ foo bar """) def test_suite(): return unittest.makeSuite(SchemaTestCase) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZConfig/tests/test_loader.py0000664000076400007640000002720210435500360022206 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of ZConfig.loader classes and helper functions.""" import os.path import sys import tempfile import unittest import urllib2 from StringIO import StringIO import ZConfig import ZConfig.loader import ZConfig.url from ZConfig.tests.support import CONFIG_BASE, TestBase try: myfile = __file__ except NameError: myfile = sys.argv[0] myfile = os.path.abspath(myfile) LIBRARY_DIR = os.path.join(os.path.dirname(myfile), "library") class LoaderTestCase(TestBase): def test_schema_caching(self): loader = ZConfig.loader.SchemaLoader() url = ZConfig.url.urljoin(CONFIG_BASE, "simple.xml") schema1 = loader.loadURL(url) schema2 = loader.loadURL(url) self.assert_(schema1 is schema2) def test_simple_import_with_cache(self): loader = ZConfig.loader.SchemaLoader() url1 = ZConfig.url.urljoin(CONFIG_BASE, "library.xml") schema1 = loader.loadURL(url1) sio = StringIO("" " " "
" "") url2 = ZConfig.url.urljoin(CONFIG_BASE, "stringio") schema2 = loader.loadFile(sio, url2) self.assert_(schema1.gettype("type-a") is schema2.gettype("type-a")) def test_simple_import_using_prefix(self): self.load_schema_text("""\ """) def test_import_errors(self): # must specify exactly one of package or src self.assertRaises(ZConfig.SchemaError, ZConfig.loadSchemaFile, StringIO("")) self.assertRaises(ZConfig.SchemaError, ZConfig.loadSchemaFile, StringIO("" " " "")) # cannot specify src and file self.assertRaises(ZConfig.SchemaError, ZConfig.loadSchemaFile, StringIO("" " " "")) # cannot specify module as package sio = StringIO("" " " "") try: ZConfig.loadSchemaFile(sio) except ZConfig.SchemaResourceError, e: self.assertEqual(e.filename, "component.xml") self.assertEqual(e.package, "ZConfig.tests.test_loader") self.assert_(e.path is None) # make sure the str() doesn't raise an unexpected exception str(e) else: self.fail("expected SchemaResourceError") def test_import_from_package(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") schema = loader.loadFile(sio) self.assert_(schema.gettype("widget-a") is not None) def test_import_from_package_with_file(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") schema = loader.loadFile(sio) self.assert_(schema.gettype("extra-type") is not None) def test_import_from_package_extra_directory(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") schema = loader.loadFile(sio) self.assert_(schema.gettype("extra-thing") is not None) def test_import_from_package_with_missing_file(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") try: loader.loadFile(sio) except ZConfig.SchemaResourceError, e: self.assertEqual(e.filename, "notthere.xml") self.assertEqual(e.package, "ZConfig.tests.library.widget") self.assert_(e.path) # make sure the str() doesn't raise an unexpected exception str(e) else: self.fail("expected SchemaResourceError") def test_import_from_package_with_directory_file(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") self.assertRaises(ZConfig.SchemaError, loader.loadFile, sio) def test_import_two_components_one_package(self): loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " " " "") schema = loader.loadFile(sio) schema.gettype("widget-a") schema.gettype("extra-type") def test_import_component_twice_1(self): # Make sure we can import a component twice from a schema. # This is most likely to occur when the component is imported # from each of two other components, or from the top-level # schema and a component. loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " " " "") schema = loader.loadFile(sio) schema.gettype("widget-a") def test_import_component_twice_2(self): # Make sure we can import a component from a config file even # if it has already been imported from the schema. loader = ZConfig.loader.SchemaLoader() sio = StringIO("" " " "") schema = loader.loadFile(sio) loader = ZConfig.loader.ConfigLoader(schema) sio = StringIO("%import ZConfig.tests.library.widget") loader.loadFile(sio) def test_urlsplit_urlunsplit(self): # Extracted from Python's test.test_urlparse module: for url, parsed, split in [ ('http://www.python.org', ('http', 'www.python.org', '', '', '', ''), ('http', 'www.python.org', '', '', '')), ('http://www.python.org#abc', ('http', 'www.python.org', '', '', '', 'abc'), ('http', 'www.python.org', '', '', 'abc')), ('http://www.python.org/#abc', ('http', 'www.python.org', '/', '', '', 'abc'), ('http', 'www.python.org', '/', '', 'abc')), ("http://a/b/c/d;p?q#f", ('http', 'a', '/b/c/d', 'p', 'q', 'f'), ('http', 'a', '/b/c/d;p', 'q', 'f')), ('file:///tmp/junk.txt', ('file', '', '/tmp/junk.txt', '', '', ''), ('file', '', '/tmp/junk.txt', '', '')), ]: result = ZConfig.url.urlsplit(url) self.assertEqual(result, split) result2 = ZConfig.url.urlunsplit(result) self.assertEqual(result2, url) def test_file_url_normalization(self): self.assertEqual( ZConfig.url.urlnormalize("file:/abc/def"), "file:///abc/def") self.assertEqual( ZConfig.url.urlunsplit(("file", "", "/abc/def", "", "")), "file:///abc/def") self.assertEqual( ZConfig.url.urljoin("file:/abc/", "def"), "file:///abc/def") self.assertEqual( ZConfig.url.urldefrag("file:/abc/def#frag"), ("file:///abc/def", "frag")) def test_isPath(self): assert_ = self.assert_ isPath = ZConfig.loader.BaseLoader().isPath assert_(isPath("abc")) assert_(isPath("abc/def")) assert_(isPath("/abc")) assert_(isPath("/abc/def")) assert_(isPath(r"\abc")) assert_(isPath(r"\abc\def")) assert_(isPath(r"c:\abc\def")) assert_(not isPath("http://www.example.com/")) assert_(not isPath("http://www.example.com/sample.conf")) assert_(not isPath("file:///etc/zope/zope.conf")) assert_(not isPath("file:///c|/foo/bar.conf")) class TestNonExistentResources(unittest.TestCase): # XXX Not sure if this is the best approach for these. These # tests make sure that the error reported by ZConfig for missing # resources is handled in a consistent way. Since ZConfig uses # urllib2.urlopen() for opening all resources, what we do is # replace that function with one that always raises an exception. # Since urllib2.urlopen() can raise either IOError or OSError # (depending on the version of Python), we run test for each # exception. urllib2.urlopen() is restored after running the # test. def setUp(self): self.urllib2_urlopen = urllib2.urlopen urllib2.urlopen = self.fake_urlopen def tearDown(self): urllib2.urlopen = self.urllib2_urlopen def fake_urlopen(self, url): raise self.error() def test_nonexistent_file_ioerror(self): self.error = IOError self.check_nonexistent_file() def test_nonexistent_file_oserror(self): self.error = OSError self.check_nonexistent_file() def check_nonexistent_file(self): fn = tempfile.mktemp() schema = ZConfig.loadSchemaFile(StringIO("")) self.assertRaises(ZConfig.ConfigurationError, ZConfig.loadSchema, fn) self.assertRaises(ZConfig.ConfigurationError, ZConfig.loadConfig, schema, fn) self.assertRaises(ZConfig.ConfigurationError, ZConfig.loadConfigFile, schema, StringIO("%include " + fn)) self.assertRaises(ZConfig.ConfigurationError, ZConfig.loadSchema, "http://www.zope.org/no-such-document/") self.assertRaises(ZConfig.ConfigurationError, ZConfig.loadConfig, schema, "http://www.zope.org/no-such-document/") def test_suite(): suite = unittest.makeSuite(LoaderTestCase) suite.addTest(unittest.makeSuite(TestNonExistentResources)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZConfig/tests/library/0000775000076400007640000000000010435610645021000 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/library/__init__.py0000664000076400007640000000002710435500360023100 0ustar ajungajung# Make this a package. Zope-2.8.7-final/lib/python/ZConfig/tests/library/README.txt0000664000076400007640000000013010435500360022460 0ustar ajungajungThis is a sample library of configuration schema components. This is used for testing. Zope-2.8.7-final/lib/python/ZConfig/tests/library/widget/0000775000076400007640000000000010435610645022263 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/library/widget/__init__.py0000664000076400007640000000002710435500360024363 0ustar ajungajung# Make this a package. Zope-2.8.7-final/lib/python/ZConfig/tests/library/widget/extra.xml0000664000076400007640000000014710435500360024122 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/library/widget/component.xml0000664000076400007640000000040010435500360024771 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/library/thing/0000775000076400007640000000000010435610645022111 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/library/thing/extras/0000775000076400007640000000000010435500360023407 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/tests/library/thing/extras/extras.xml0000664000076400007640000000015010435500360025433 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/library/thing/__init__.py0000664000076400007640000000151010435500360024207 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Example of a package that extends its __path__. $Id: __init__.py,v 1.2 2003/10/03 17:11:33 fdrake Exp $ """ import os here = os.path.dirname(__file__) __path__.append(os.path.join(here, "extras")) Zope-2.8.7-final/lib/python/ZConfig/tests/library/thing/component.xml0000664000076400007640000000053010435500360024623 0ustar ajungajung Zope-2.8.7-final/lib/python/ZConfig/tests/test_subst.py0000664000076400007640000000647210435500360022106 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the string interpolation module.""" # This is needed to support Python 2.1. from __future__ import nested_scopes import unittest from ZConfig import SubstitutionReplacementError, SubstitutionSyntaxError from ZConfig.substitution import isname, substitute class SubstitutionTestCase(unittest.TestCase): def test_simple_names(self): d = {"name": "value", "name1": "abc", "name_": "def", "_123": "ghi"} def check(s, v): self.assertEqual(substitute(s, d), v) check("$name", "value") check(" $name ", " value ") check("${name}", "value") check(" ${name} ", " value ") check("$name$name", "valuevalue") check("$name1$name", "abcvalue") check("$name_$name", "defvalue") check("$_123$name", "ghivalue") check("$name $name", "value value") check("$name1 $name", "abc value") check("$name_ $name", "def value") check("$_123 $name", "ghi value") check("splat", "splat") check("$$", "$") check("$$$name$$", "$value$") def test_undefined_names(self): d = {"name": "value"} self.assertRaises(SubstitutionReplacementError, substitute, "$splat", d) self.assertRaises(SubstitutionReplacementError, substitute, "$splat1", d) self.assertRaises(SubstitutionReplacementError, substitute, "$splat_", d) def test_syntax_errors(self): d = {"name": "${next"} def check(s): self.assertRaises(SubstitutionSyntaxError, substitute, s, d) check("${") check("${name") check("${1name}") check("${ name}") def test_edge_cases(self): # It's debatable what should happen for these cases, so we'll # follow the lead of the Bourne shell here. def check(s): self.assertRaises(SubstitutionSyntaxError, substitute, s, {}) check("$1") check("$") check("$ stuff") def test_non_nesting(self): d = {"name": "$value"} self.assertEqual(substitute("$name", d), "$value") def test_isname(self): self.assert_(isname("abc")) self.assert_(isname("abc_def")) self.assert_(isname("_abc")) self.assert_(isname("abc_")) self.assert_(not isname("abc-def")) self.assert_(not isname("-def")) self.assert_(not isname("abc-")) self.assert_(not isname("")) def test_suite(): return unittest.makeSuite(SubstitutionTestCase) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZConfig/tests/runtests.bat0000775000076400007640000000052510435500360021710 0ustar ajungajung@echo off rem Simple script to run the tests on Windows. rem The paths to different versions of Python need to be rem edited for the system this is being run on; comment rem out lines that aren't needed or wanted. \Python213\python runtests.py \Python221\python runtests.py \Python222\python runtests.py \Python230\python runtests.py Zope-2.8.7-final/lib/python/ZConfig/tests/test_config.py0000664000076400007640000001454710435500360022215 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the configuration data structures and loader.""" import os import StringIO import tempfile import unittest import ZConfig from ZConfig.tests.support import CONFIG_BASE class ConfigurationTestCase(unittest.TestCase): schema = None def get_schema(self): if self.schema is None: ConfigurationTestCase.schema = ZConfig.loadSchema( CONFIG_BASE + "simple.xml") return self.schema def load(self, relurl, context=None): url = CONFIG_BASE + relurl self.conf, self.handlers = ZConfig.loadConfig(self.get_schema(), url) conf = self.conf #self.assertEqual(conf.url, url) self.assert_(conf.getSectionName() is None) self.assert_(conf.getSectionType() is None) #self.assert_(conf.delegate is None) return conf def loadtext(self, text): sio = StringIO.StringIO(text) return self.loadfile(sio) def loadfile(self, file): schema = self.get_schema() self.conf, self.handlers = ZConfig.loadConfigFile(schema, file) return self.conf def check_simple_gets(self, conf): self.assertEqual(conf.empty, '') self.assertEqual(conf.int_var, 12) self.assertEqual(conf.neg_int, -2) self.assertEqual(conf.float_var, 12.02) self.assertEqual(conf.var1, 'abc') self.assert_(conf.true_var_1) self.assert_(conf.true_var_2) self.assert_(conf.true_var_3) self.assert_(not conf.false_var_1) self.assert_(not conf.false_var_2) self.assert_(not conf.false_var_3) self.assertEqual(conf.list_1, []) self.assertEqual(conf.list_2, ['abc']) self.assertEqual(conf.list_3, ['abc', 'def', 'ghi']) self.assertEqual(conf.list_4, ['[', 'what', 'now?', ']']) def test_simple_gets(self): conf = self.load("simple.conf") self.check_simple_gets(conf) def test_type_errors(self): Error = ZConfig.DataConversionError raises = self.assertRaises raises(Error, self.loadtext, "int-var true") raises(Error, self.loadtext, "float-var true") raises(Error, self.loadtext, "neg-int false") raises(Error, self.loadtext, "true-var-1 0") raises(Error, self.loadtext, "true-var-1 1") raises(Error, self.loadtext, "true-var-1 -1") def test_simple_sections(self): self.schema = ZConfig.loadSchema(CONFIG_BASE + "simplesections.xml") conf = self.load("simplesections.conf") self.assertEqual(conf.var, "foo") # check each interleaved position between sections for c in "0123456": self.assertEqual(getattr(conf, "var_" +c), "foo-" + c) sect = [sect for sect in conf.sections if sect.getSectionName() == "name"][0] self.assertEqual(sect.var, "bar") self.assertEqual(sect.var_one, "splat") self.assert_(sect.var_three is None) sect = [sect for sect in conf.sections if sect.getSectionName() == "delegate"][0] self.assertEqual(sect.var, "spam") self.assertEqual(sect.var_two, "stuff") self.assert_(sect.var_three is None) def test_include(self): conf = self.load("include.conf") self.assertEqual(conf.var1, "abc") self.assertEqual(conf.var2, "value2") self.assertEqual(conf.var3, "value3") self.assertEqual(conf.var4, "value") def test_includes_with_defines(self): self.schema = ZConfig.loadSchemaFile(StringIO.StringIO("""\ """)) conf = self.load("outer.conf") self.assertEqual(conf.refinner, "inner") self.assertEqual(conf.refouter, "outer") def test_define(self): conf = self.load("simple.conf") self.assertEqual(conf.getname, "value") self.assertEqual(conf.getnametwice, "valuevalue") self.assertEqual(conf.getdollars, "$$") self.assertEqual(conf.getempty, "xy") self.assertEqual(conf.getwords, "abc two words def") def test_define_errors(self): self.assertRaises(ZConfig.ConfigurationSyntaxError, self.loadtext, "%define\n") self.assertRaises(ZConfig.ConfigurationSyntaxError, self.loadtext, "%define abc-def\n") self.assertRaises(ZConfig.ConfigurationSyntaxError, self.loadtext, "%define a value\n%define a value\n") def test_fragment_ident_disallowed(self): self.assertRaises(ZConfig.ConfigurationError, self.load, "simplesections.conf#another") def test_load_from_fileobj(self): sio = StringIO.StringIO("%define name value\n" "getname x $name y \n") cf = self.loadfile(sio) self.assertEqual(cf.getname, "x value y") def test_load_from_abspath(self): fn = self.write_tempfile() try: self.check_load_from_path(fn) finally: os.unlink(fn) def test_load_from_relpath(self): fn = self.write_tempfile() dir, name = os.path.split(fn) pwd = os.getcwd() try: os.chdir(dir) self.check_load_from_path(name) finally: os.chdir(pwd) os.unlink(fn) def write_tempfile(self): fn = tempfile.mktemp() fp = open(fn, "w") fp.write("var1 value\n") fp.close() return fn def check_load_from_path(self, path): schema = self.get_schema() ZConfig.loadConfig(schema, path) def test_suite(): return unittest.makeSuite(ConfigurationTestCase) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZConfig/tests/support.py0000664000076400007640000000451410435500360021416 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Support code shared among the tests.""" import os import StringIO import unittest import urllib import ZConfig from ZConfig.loader import ConfigLoader from ZConfig.url import urljoin try: __file__ except NameError: import sys __file__ = sys.argv[0] d = os.path.abspath(os.path.join(os.path.dirname(__file__), "input")) CONFIG_BASE = "file://%s/" % urllib.pathname2url(d) class TestBase(unittest.TestCase): """Utility methods which can be used with the schema support.""" def load_both(self, schema_url, conf_url): schema = self.load_schema(schema_url) conf = self.load_config(schema, conf_url) return schema, conf def load_schema(self, relurl): self.url = urljoin(CONFIG_BASE, relurl) self.schema = ZConfig.loadSchema(self.url) self.assert_(self.schema.issection()) return self.schema def load_schema_text(self, text, url=None): sio = StringIO.StringIO(text) self.schema = ZConfig.loadSchemaFile(sio, url) return self.schema def load_config(self, schema, conf_url, num_handlers=0): conf_url = urljoin(CONFIG_BASE, conf_url) loader = self.create_config_loader(schema) self.conf, self.handlers = loader.loadURL(conf_url) self.assertEqual(len(self.handlers), num_handlers) return self.conf def load_config_text(self, schema, text, num_handlers=0, url=None): sio = StringIO.StringIO(text) loader = self.create_config_loader(schema) self.conf, self.handlers = loader.loadFile(sio, url) self.assertEqual(len(self.handlers), num_handlers) return self.conf def create_config_loader(self, schema): return ConfigLoader(schema) Zope-2.8.7-final/lib/python/ZConfig/url.py0000664000076400007640000000407010435500361017340 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """urlparse-like helpers that normalize file: URLs. ZConfig and urllib2 expect file: URLs to consistently use the '//' hostpart seperator; the functions here enforce this constraint. """ import urlparse as _urlparse try: from urlparse import urlsplit except ImportError: def urlsplit(url): # Check for the fragment here, since Python 2.1.3 didn't get # it right for things like "http://www.python.org#frag". if '#' in url: url, fragment = url.split('#', 1) else: fragment = '' parts = list(_urlparse.urlparse(url)) parts[-1] = fragment param = parts.pop(3) if param: parts[2] += ";" + param return tuple(parts) def urlnormalize(url): lc = url.lower() if lc.startswith("file:/") and not lc.startswith("file:///"): url = "file://" + url[5:] return url def urlunsplit(parts): parts = list(parts) parts.insert(3, '') url = _urlparse.urlunparse(tuple(parts)) if (parts[0] == "file" and url.startswith("file:/") and not url.startswith("file:///")): url = "file://" + url[5:] return url def urldefrag(url): url, fragment = _urlparse.urldefrag(url) return urlnormalize(url), fragment def urljoin(base, relurl): url = _urlparse.urljoin(base, relurl) if url.startswith("file:/") and not url.startswith("file:///"): url = "file://" + url[5:] return url Zope-2.8.7-final/lib/python/ZConfig/matcher.py0000664000076400007640000002475510435500361020175 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Utility that manages the binding of configuration data to a section.""" import ZConfig from ZConfig.info import ValueInfo class BaseMatcher: def __init__(self, info, type, handlers): self.info = info self.type = type self._values = {} for key, info in type: if info.name == "+" and not info.issection(): v = {} elif info.ismulti(): v = [] else: v = None assert info.attribute is not None self._values[info.attribute] = v self._sectionnames = {} if handlers is None: handlers = [] self.handlers = handlers def __repr__(self): clsname = self.__class__.__name__ extra = "type " + `self.type.name` return "<%s for %s>" % (clsname, extra) def addSection(self, type, name, sectvalue): if name: if self._sectionnames.has_key(name): raise ZConfig.ConfigurationError( "section names must not be re-used within the" " same container:" + `name`) self._sectionnames[name] = name ci = self.type.getsectioninfo(type, name) attr = ci.attribute v = self._values[attr] if ci.ismulti(): v.append(sectvalue) elif v is None: self._values[attr] = sectvalue else: raise ZConfig.ConfigurationError( "too many instances of %s section" % `ci.sectiontype.name`) def addValue(self, key, value, position): try: realkey = self.type.keytype(key) except ValueError, e: raise ZConfig.DataConversionError(e, key, position) arbkey_info = None for i in range(len(self.type)): k, ci = self.type[i] if k == realkey: break if ci.name == "+" and not ci.issection(): arbkey_info = k, ci else: if arbkey_info is None: raise ZConfig.ConfigurationError( `key` + " is not a known key name") k, ci = arbkey_info if ci.issection(): if ci.name: extra = " in %s sections" % `self.type.name` else: extra = "" raise ZConfig.ConfigurationError( "%s is not a valid key name%s" % (`key`, extra)) ismulti = ci.ismulti() attr = ci.attribute assert attr is not None v = self._values[attr] if v is None: if k == '+': v = {} elif ismulti: v = [] self._values[attr] = v elif not ismulti: if k != '+': raise ZConfig.ConfigurationError( `key` + " does not support multiple values") elif len(v) == ci.maxOccurs: raise ZConfig.ConfigurationError( "too many values for " + `name`) value = ValueInfo(value, position) if k == '+': if ismulti: if v.has_key(realkey): v[realkey].append(value) else: v[realkey] = [value] else: if v.has_key(realkey): raise ZConfig.ConfigurationError( "too many values for " + `key`) v[realkey] = value elif ismulti: v.append(value) else: self._values[attr] = value def createChildMatcher(self, type, name): ci = self.type.getsectioninfo(type.name, name) assert not ci.isabstract() if not ci.isAllowedName(name): raise ZConfig.ConfigurationError( "%s is not an allowed name for %s sections" % (`name`, `ci.sectiontype.name`)) return SectionMatcher(ci, type, name, self.handlers) def finish(self): """Check the constraints of the section and convert to an application object.""" values = self._values for key, ci in self.type: if key: key = repr(key) else: key = "section type " + `ci.sectiontype.name` assert ci.attribute is not None attr = ci.attribute v = values[attr] if ci.name == '+' and not ci.issection(): # v is a dict if ci.minOccurs > len(v): raise ZConfig.ConfigurationError( "no keys defined for the %s key/value map; at least %d" " must be specified" % (attr, ci.minOccurs)) if v is None and ci.minOccurs: default = ci.getdefault() if default is None: raise ZConfig.ConfigurationError( "no values for %s; %s required" % (key, ci.minOccurs)) else: v = values[attr] = default[:] if ci.ismulti(): if not v: default = ci.getdefault() if isinstance(default, dict): v.update(default) else: v[:] = default if len(v) < ci.minOccurs: raise ZConfig.ConfigurationError( "not enough values for %s; %d found, %d required" % (key, len(v), ci.minOccurs)) if v is None and not ci.issection(): if ci.ismulti(): v = ci.getdefault()[:] else: v = ci.getdefault() values[attr] = v return self.constuct() def constuct(self): values = self._values for name, ci in self.type: assert ci.attribute is not None attr = ci.attribute if ci.ismulti(): if ci.issection(): v = [] for s in values[attr]: if s is not None: st = s.getSectionDefinition() try: s = st.datatype(s) except ValueError, e: raise ZConfig.DataConversionError( e, s, (-1, -1, None)) v.append(s) elif ci.name == '+': v = values[attr] for key, val in v.items(): v[key] = [vi.convert(ci.datatype) for vi in val] else: v = [vi.convert(ci.datatype) for vi in values[attr]] elif ci.issection(): if values[attr] is not None: st = values[attr].getSectionDefinition() try: v = st.datatype(values[attr]) except ValueError, e: raise ZConfig.DataConversionError( e, values[attr], (-1, -1, None)) else: v = None elif name == '+': v = values[attr] if not v: for key, val in ci.getdefault().items(): v[key] = val.convert(ci.datatype) else: for key, val in v.items(): v[key] = val.convert(ci.datatype) else: v = values[attr] if v is not None: v = v.convert(ci.datatype) values[attr] = v if ci.handler is not None: self.handlers.append((ci.handler, v)) return self.createValue() def createValue(self): return SectionValue(self._values, None, self) class SectionMatcher(BaseMatcher): def __init__(self, info, type, name, handlers): if name or info.allowUnnamed(): self.name = name else: raise ZConfig.ConfigurationError( `type.name` + " sections may not be unnamed") BaseMatcher.__init__(self, info, type, handlers) def createValue(self): return SectionValue(self._values, self.name, self) class SchemaMatcher(BaseMatcher): def __init__(self, schema): BaseMatcher.__init__(self, schema, schema, []) def finish(self): # Since there's no outer container to call datatype() # for the schema, we convert on the way out. v = BaseMatcher.finish(self) v = self.type.datatype(v) if self.type.handler is not None: self.handlers.append((self.type.handler, v)) return v class SectionValue: """Generic 'bag-of-values' object for a section. Derived classes should always call the SectionValue constructor before attempting to modify self. """ def __init__(self, values, name, matcher): self.__dict__.update(values) self._name = name self._matcher = matcher self._attributes = tuple(values.keys()) def __repr__(self): if self._name: # probably unique for a given config file; more readable than id() name = `self._name` else: # identify uniquely name = "at %#x" % id(self) clsname = self.__class__.__name__ return "<%s for %s %s>" % (clsname, self._matcher.type.name, name) def __str__(self): l = [] attrnames = [s for s in self.__dict__.keys() if s[0] != "_"] attrnames.sort() for k in attrnames: v = getattr(self, k) l.append('%-40s: %s' % (k, v)) return '\n'.join(l) def getSectionName(self): return self._name def getSectionType(self): return self._matcher.type.name def getSectionDefinition(self): return self._matcher.type def getSectionMatcher(self): return self._matcher def getSectionAttributes(self): return self._attributes Zope-2.8.7-final/lib/python/ZConfig/cfgparser.py0000664000076400007640000001447710435500361020526 0ustar ajungajung############################################################################## # # Copyright (c) 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Configuration parser.""" import ZConfig import ZConfig.url from ZConfig.substitution import isname, substitute class ZConfigParser: __metaclass__ = type __slots__ = ('resource', 'context', 'lineno', 'stack', 'defines', 'file', 'url') def __init__(self, resource, context, defines=None): self.resource = resource self.context = context self.file = resource.file self.url = resource.url self.lineno = 0 self.stack = [] # [(type, name, prevmatcher), ...] if defines is None: defines = {} self.defines = defines def nextline(self): line = self.file.readline() if line: self.lineno += 1 return False, line.strip() else: return True, None def parse(self, section): done, line = self.nextline() while not done: if line[:1] in ("", "#"): # blank line or comment pass elif line[:2] == "": self.error("malformed section end") section = self.end_section(section, line[2:-1]) elif line[0] == "<": # section start if line[-1] != ">": self.error("malformed section start") section = self.start_section(section, line[1:-1]) elif line[0] == "%": self.handle_directive(section, line[1:]) else: self.handle_key_value(section, line) done, line = self.nextline() if self.stack: self.error("unclosed sections not allowed") def start_section(self, section, rest): isempty = rest[-1:] == "/" if isempty: rest = rest[:-1] text = rest.rstrip() # parse section start stuff here m = _section_start_rx.match(text) if not m: self.error("malformed section header") type, name = m.group('type', 'name') type = type.lower() if name: name = name.lower() try: newsect = self.context.startSection(section, type, name) except ZConfig.ConfigurationError, e: self.error(e[0]) if isempty: self.context.endSection(section, type, name, newsect) return section else: self.stack.append((type, name, section)) return newsect def end_section(self, section, rest): if not self.stack: self.error("unexpected section end") type = rest.rstrip().lower() opentype, name, prevsection = self.stack.pop() if type != opentype: self.error("unbalanced section end") try: self.context.endSection( prevsection, type, name, section) except ZConfig.ConfigurationError, e: self.error(e[0]) return prevsection def handle_key_value(self, section, rest): m = _keyvalue_rx.match(rest) if not m: self.error("malformed configuration data") key, value = m.group('key', 'value') if not value: value = '' else: value = self.replace(value) try: section.addValue(key, value, (self.lineno, None, self.url)) except ZConfig.ConfigurationError, e: self.error(e[0]) def handle_directive(self, section, rest): m = _keyvalue_rx.match(rest) if not m: self.error("missing or unrecognized directive") name, arg = m.group('key', 'value') if name not in ("define", "import", "include"): self.error("unknown directive: " + `name`) if not arg: self.error("missing argument to %%%s directive" % name) if name == "include": self.handle_include(section, arg) elif name == "define": self.handle_define(section, arg) elif name == "import": self.handle_import(section, arg) else: assert 0, "unexpected directive for " + `"%" + rest` def handle_import(self, section, rest): pkgname = self.replace(rest.strip()) self.context.importSchemaComponent(pkgname) def handle_include(self, section, rest): rest = self.replace(rest.strip()) newurl = ZConfig.url.urljoin(self.url, rest) self.context.includeConfiguration(section, newurl, self.defines) def handle_define(self, section, rest): parts = rest.split(None, 1) defname = parts[0].lower() defvalue = '' if len(parts) == 2: defvalue = parts[1] if self.defines.has_key(defname): self.error("cannot redefine " + `defname`) if not isname(defname): self.error("not a substitution legal name: " + `defname`) self.defines[defname] = self.replace(defvalue) def replace(self, text): try: return substitute(text, self.defines) except ZConfig.SubstitutionReplacementError, e: e.lineno = self.lineno e.url = self.url raise def error(self, message): raise ZConfig.ConfigurationSyntaxError(message, self.url, self.lineno) import re # _name_re does not allow "(" or ")" for historical reasons. Though # the restriction could be lifted, there seems no need to do so. _name_re = r"[^\s()]+" _keyvalue_rx = re.compile(r"(?P%s)\s*(?P[^\s].*)?$" % _name_re) _section_start_rx = re.compile(r"(?P%s)" r"(?:\s+(?P%s))?" r"$" % (_name_re, _name_re)) del re Zope-2.8.7-final/lib/python/ZConfig/NEWS.txt0000664000076400007640000000624510435500361017507 0ustar ajungajungChanges since ZConfig 2.2: - Added "inet-binding-address" and "inet-connection-address" to the set of standard datatypes. These are similar to the "inet-address" type, but the default hostname is more sensible. The datatype used should reflect how the value will be used. - Alternate rotating logfile handler for Windows, to avoid platform limitations on renaming open files. Contributed by Sidnei da Silva. - For
and , if the name attribute is omitted, assume name="*", since this is what is used most often. Changes since ZConfig 2.1: - More documentation has been written. - Added a timedelta datatype function; the input is the same as for the time-interval datatype, but the resulting value is a datetime.timedelta object. - Make sure keys specified as attributes of the element are converted by the appropriate key type, and are re-checked for derived sections. - Refactored the ZConfig.components.logger schema components so that a schema can import just one of the "eventlog" or "logger" sections if desired. This can be helpful to avoid naming conflicts. - Added a reopen() method to the logger factories. - Always use an absolute pathname when opening a FileHandler. - A fix to the logger 'format' key to allow the %(process)d expansion variable that the logging package supports. - A new timedelta built-in datatype was added. Similar to time-interval except that it returns a datetime.timedelta object instead. Changes since ZConfig 2.0: - Removed compatibility with Python 2.1 and 2.2. - Schema components must really be in Python packages; the directory search has been modified to perform an import to locate the package rather than incorrectly implementing the search algorithm. - The default objects use for section values now provide a method getSectionAttributes(); this returns a list of all the attributes of the section object which store configuration-defined data (including information derived from the schema). - Default information can now be included in a schema for and by using . - More documentation has been added to discuss schema extension. - Support for a Unicode-free Python has been fixed. - Derived section types now inherit the datatype of the base type if no datatype is identified explicitly. - Derived section types can now override the keytype instead of always inheriting from their base type. - makes use of the current prefix if the package name begins witha dot. - Added two standard datatypes: dotted-name and dotted-suffix. - Added two standard schema components: ZConfig.components.basic and ZConfig.components.logger. Changes since ZConfig 1.0: - Configurations can import additional schema components using a new "%import" directive; this can be used to integrate 3rd-party components into an application. - Schemas may be extended using a new "extends" attribute on the element. - Better error messages when elements in a schema definition are improperly nested. - The "zconfig" script can now simply verify that a schema definition is valid, if that's all that's needed. Zope-2.8.7-final/lib/python/ZConfig/scripts/0000775000076400007640000000000010435500360017651 5ustar ajungajungZope-2.8.7-final/lib/python/ZConfig/scripts/zconfig0000775000076400007640000000460310435500360021241 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """zconfig: Script to check validity of a configuration file. Usage: zconfig [options] [file...] Options: -h --help Print this help text. -s file --schema file Use the schema in 'file' to validate the configuration; this must be specified. Each file named on the command line is checked for syntactical errors and schema conformance. The schema must be specified. If no files are specified and standard input is not a TTY, standard in is treated as a configuration file. Specifying a schema and no configuration files causes the schema to be checked. """ import optparse import sys import ZConfig def main(): optparser = optparse.OptionParser( usage="usage: %prog [-s FILE] [file ...]") optparser.add_option( "-s", "--schema", dest="schema", help="use the schema in FILE (can be a URL)", metavar="FILE") options, args = optparser.parse_args() if not options.schema: print >>sys.stderr, "No schema specified, but is required." usage(sys.stderr) return 2 schema = ZConfig.loadSchema(options.schema) if not args: if sys.stdin.isatty(): # just checking the schema return 0 else: # stdin is a pipe args = ["-"] errors = 0 for fn in args: try: if fn == "-": ZConfig.loadConfigFile(schema, sys.stdin) else: ZConfig.loadConfig(schema, fn) except ZConfig.ConfigurationError, e: print >>sys.stderr, str(e) errors += 1 if errors: return 1 else: return 0 def usage(fp): print >>fp, __doc__ if __name__ == "__main__": sys.exit(main()) Zope-2.8.7-final/lib/python/ZConfig/scripts/zconfig_schema2html0000775000076400007640000000614410435500360023532 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## __version__ = '$Revision: 1.3 $'[11:-2] import ZConfig.loader from ZConfig.info import * import sys, cgi def esc(x): return cgi.escape(str(x)) def dt(x): tn = type(x).__name__ if tn == 'instance': return '%s %s'%(tn, x.__class__.__module__ + '.' + x.__class__.__name__) elif tn == 'class': return '%s %s'%(tn, x.__module__ + '.' + x.__name__) else: return '%s %s'%(tn, x.__name__) class explain: done = [] def __call__(self, st): if st.name in self.done: return self.done.append(st.name) if st.description: print st.description for sub in st.getsubtypenames(): print '
' printContents(None, st.getsubtype(sub)) print '
' explain = explain() def printSchema(schema): print '
' for child in schema: printContents(*child) print '
' def printContents(name, info): if isinstance(info, SectionType): print '
', info.name, ' (%s)
'%dt(info.datatype) print '
' if info.description: print info.description print '
' for sub in info: printContents(*sub) print '
' elif isinstance(info, SectionInfo): st = info.sectiontype if st.isabstract(): print '
', st.name, '', info.name, '
' print '
' if info.description: print info.description explain(st) print '
' else: print '
', info.attribute, info.name, '' print '(%s)
'%dt(info.datatype) print '
' for sub in info.sectiontype: printContents(*sub) print '
' else: print '
',info.name, '', '(%s)'%dt(info.datatype) default = info.getdefault() if isinstance(default, ValueInfo): print '(default: %r)'%esc(default.value) elif default is not None: print '(default: %r)'%esc(default) if info.metadefault: print '(metadefault: %s)' % info.metadefault print '
' if info.description: print '
',info.description,'
' schema = ZConfig.loader.loadSchemaFile(sys.argv[1]) print ''' ''' printSchema(schema) print '' # vim: set filetype=python ts=4 sw=4 et si Zope-2.8.7-final/lib/python/ZConfig/cmdline.py0000664000076400007640000001353610435500361020160 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Support for command-line provision of settings. This module provides an extension of the ConfigLoader class which adds a way to add configuration settings from an alternate source. Each setting is described by a string of the form:: some/path/to/key=value """ import ZConfig import ZConfig.loader import ZConfig.matcher class ExtendedConfigLoader(ZConfig.loader.ConfigLoader): def __init__(self, schema): ZConfig.loader.ConfigLoader.__init__(self, schema) self.clopts = [] # [(optpath, value, source-position), ...] def addOption(self, spec, pos=None): if pos is None: pos = "", -1, -1 if "=" not in spec: e = ZConfig.ConfigurationSyntaxError( "invalid configuration specifier", *pos) e.specifier = spec raise e # For now, just add it to the list; not clear that checking # against the schema at this point buys anything. opt, val = spec.split("=", 1) optpath = opt.split("/") if "" in optpath: # // is not allowed in option path e = ZConfig.ConfigurationSyntaxError( "'//' is not allowed in an option path", *pos) e.specifier = spec raise e self.clopts.append((optpath, val, pos)) def createSchemaMatcher(self): if self.clopts: sm = ExtendedSchemaMatcher(self.schema) sm.set_optionbag(self.cook()) else: sm = ZConfig.loader.ConfigLoader.createSchemaMatcher(self) return sm def cook(self): if self.clopts: return OptionBag(self.schema, self.schema, self.clopts) else: return None class OptionBag: def __init__(self, schema, sectiontype, options): self.sectiontype = sectiontype self.schema = schema self.keypairs = {} self.sectitems = [] self._basic_key = schema.registry.get("basic-key") for item in options: optpath, val, pos = item name = sectiontype.keytype(optpath[0]) if len(optpath) == 1: self.add_value(name, val, pos) else: self.sectitems.append(item) def basic_key(self, s, pos): try: return self._basic_key(s) except ValueError: raise ZConfig.ConfigurationSyntaxError( "could not convert basic-key value", *pos) def add_value(self, name, val, pos): if self.keypairs.has_key(name): L = self.keypairs[name] else: L = [] self.keypairs[name] = L L.append((val, pos)) def has_key(self, name): return self.keypairs.has_key(name) def get_key(self, name): """Return a list of (value, pos) items for the key 'name'. The returned list may be empty. """ L = self.keypairs.get(name) if L: del self.keypairs[name] return L else: return [] def keys(self): return self.keypairs.keys() def get_section_info(self, type, name): L = [] # what pertains to the child section R = [] # what we keep for item in self.sectitems: optpath, val, pos = item s = optpath[0] bk = self.basic_key(s, pos) if name and s.lower() == name: L.append((optpath[1:], val, pos)) elif bk == type: L.append((optpath[1:], val, pos)) else: R.append(item) if L: self.sectitems[:] = R return OptionBag(self.schema, self.schema.gettype(type), L) else: return None def finish(self): if self.sectitems or self.keypairs: raise ZConfig.ConfigurationError( "not all command line options were consumed") class MatcherMixin: def set_optionbag(self, bag): self.optionbag = bag def addValue(self, key, value, position): try: realkey = self.type.keytype(key) except ValueError, e: raise ZConfig.DataConversionError(e, key, position) if self.optionbag.has_key(realkey): return ZConfig.matcher.BaseMatcher.addValue(self, key, value, position) def createChildMatcher(self, type, name): sm = ZConfig.matcher.BaseMatcher.createChildMatcher(self, type, name) bag = self.optionbag.get_section_info(type.name, name) if bag is not None: sm = ExtendedSectionMatcher( sm.info, sm.type, sm.name, sm.handlers) sm.set_optionbag(bag) return sm def finish_optionbag(self): for key in self.optionbag.keys(): for val, pos in self.optionbag.get_key(key): ZConfig.matcher.BaseMatcher.addValue(self, key, val, pos) self.optionbag.finish() class ExtendedSectionMatcher(MatcherMixin, ZConfig.matcher.SectionMatcher): def finish(self): self.finish_optionbag() return ZConfig.matcher.SectionMatcher.finish(self) class ExtendedSchemaMatcher(MatcherMixin, ZConfig.matcher.SchemaMatcher): def finish(self): self.finish_optionbag() return ZConfig.matcher.SchemaMatcher.finish(self) Zope-2.8.7-final/lib/python/Zope.py0000664000076400007640000000203510435500331016110 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Legacy Zope "package" that redirects to the new Zope 2 package $Id: Zope.py 37482 2005-07-27 13:58:22Z efge $ """ import sys, Zope2 sys.modules['Zope'] = Zope2 import warnings warnings.warn("The Zope package has been renamed to Zope2. " "Import of a package named 'Zope' is deprecated " "and will be disabled starting in Zope 2.11.", DeprecationWarning, stacklevel=2) Zope-2.8.7-final/lib/python/RestrictedPython/0000775000076400007640000000000010435610645020145 5ustar ajungajungZope-2.8.7-final/lib/python/RestrictedPython/RestrictionMutator.py0000664000076400007640000003565510435500325024407 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Modify AST to include security checks. RestrictionMutator modifies a tree produced by compiler.transformer.Transformer, restricting and enhancing the code in various ways before sending it to pycodegen. $Revision: 1.13 $ """ from SelectCompiler import ast, parse, OP_ASSIGN, OP_DELETE, OP_APPLY # These utility functions allow us to generate AST subtrees without # line number attributes. These trees can then be inserted into other # trees without affecting line numbers shown in tracebacks, etc. def rmLineno(node): """Strip lineno attributes from a code tree.""" if node.__dict__.has_key('lineno'): del node.lineno for child in node.getChildren(): if isinstance(child, ast.Node): rmLineno(child) def stmtNode(txt): """Make a "clean" statement node.""" node = parse(txt).node.nodes[0] rmLineno(node) return node # The security checks are performed by a set of six functions that # must be provided by the restricted environment. _apply_name = ast.Name("_apply_") _getattr_name = ast.Name("_getattr_") _getitem_name = ast.Name("_getitem_") _getiter_name = ast.Name("_getiter_") _print_target_name = ast.Name("_print") _write_name = ast.Name("_write_") _inplacevar_name = ast.Name("_inplacevar_") # Constants. _None_const = ast.Const(None) _write_const = ast.Const("write") _printed_expr = stmtNode("_print()").expr _print_target_node = stmtNode("_print = _print_()") class FuncInfo: print_used = False printed_used = False class RestrictionMutator: def __init__(self): self.warnings = [] self.errors = [] self.used_names = {} self.funcinfo = FuncInfo() def error(self, node, info): """Records a security error discovered during compilation.""" lineno = getattr(node, 'lineno', None) if lineno is not None and lineno > 0: self.errors.append('Line %d: %s' % (lineno, info)) else: self.errors.append(info) def checkName(self, node, name): """Verifies that a name being assigned is safe. This is to prevent people from doing things like: __metatype__ = mytype (opens up metaclasses, a big unknown in terms of security) __path__ = foo (could this confuse the import machinery?) _getattr = somefunc (not very useful, but could open a hole) Note that assigning a variable is not the only way to assign a name. def _badname, class _badname, import foo as _badname, and perhaps other statements assign names. Special case: '_' is allowed. """ if name.startswith("_") and name != "_": # Note: "_" *is* allowed. self.error(node, '"%s" is an invalid variable name because' ' it starts with "_"' % name) if name.endswith('__roles__'): self.error(node, '"%s" is an invalid variable name because ' 'it ends with "__roles__".' % name) if name == "printed": self.error(node, '"printed" is a reserved name.') def checkAttrName(self, node): """Verifies that an attribute name does not start with _. As long as guards (security proxies) have underscored names, this underscore protection is important regardless of the security policy. Special case: '_' is allowed. """ name = node.attrname if name.startswith("_") and name != "_": # Note: "_" *is* allowed. self.error(node, '"%s" is an invalid attribute name ' 'because it starts with "_".' % name) if name.endswith('__roles__'): self.error(node, '"%s" is an invalid attribute name ' 'because it ends with "__roles__".' % name) def prepBody(self, body): """Insert code for print at the beginning of the code suite.""" if self.funcinfo.print_used or self.funcinfo.printed_used: # Add code at top for creating _print_target body.insert(0, _print_target_node) if not self.funcinfo.printed_used: self.warnings.append( "Prints, but never reads 'printed' variable.") elif not self.funcinfo.print_used: self.warnings.append( "Doesn't print, but reads 'printed' variable.") def visitFunction(self, node, walker): """Checks and mutates a function definition. Checks the name of the function and the argument names using checkName(). It also calls prepBody() to prepend code to the beginning of the code suite. """ self.checkName(node, node.name) for argname in node.argnames: if isinstance(argname, str): self.checkName(node, argname) else: for name in argname: self.checkName(node, name) walker.visitSequence(node.defaults) former_funcinfo = self.funcinfo self.funcinfo = FuncInfo() node = walker.defaultVisitNode(node, exclude=('defaults',)) self.prepBody(node.code.nodes) self.funcinfo = former_funcinfo return node def visitLambda(self, node, walker): """Checks and mutates an anonymous function definition. Checks the argument names using checkName(). It also calls prepBody() to prepend code to the beginning of the code suite. """ for argname in node.argnames: self.checkName(node, argname) return walker.defaultVisitNode(node) def visitPrint(self, node, walker): """Checks and mutates a print statement. Adds a target to all print statements. 'print foo' becomes 'print >> _print, foo', where _print is the default print target defined for this scope. Alternatively, if the untrusted code provides its own target, we have to check the 'write' method of the target. 'print >> ob, foo' becomes 'print >> (_getattr(ob, 'write') and ob), foo'. Otherwise, it would be possible to call the write method of templates and scripts; 'write' happens to be the name of the method that changes them. """ node = walker.defaultVisitNode(node) self.funcinfo.print_used = True if node.dest is None: node.dest = _print_target_name else: # Pre-validate access to the "write" attribute. # "print >> ob, x" becomes # "print >> (_getattr(ob, 'write') and ob), x" node.dest = ast.And([ ast.CallFunc(_getattr_name, [node.dest, _write_const]), node.dest]) return node visitPrintnl = visitPrint def visitName(self, node, walker): """Prevents access to protected names as defined by checkName(). Also converts use of the name 'printed' to an expression. """ if node.name == 'printed': # Replace name lookup with an expression. self.funcinfo.printed_used = True return _printed_expr self.checkName(node, node.name) self.used_names[node.name] = True return node def visitCallFunc(self, node, walker): """Checks calls with *-args and **-args. That's a way of spelling apply(), and needs to use our safe _apply_ instead. """ walked = walker.defaultVisitNode(node) if node.star_args is None and node.dstar_args is None: # This is not an extended function call return walked # Otherwise transform foo(a, b, c, d=e, f=g, *args, **kws) into a call # of _apply_(foo, a, b, c, d=e, f=g, *args, **kws). The interesting # thing here is that _apply_() is defined with just *args and **kws, # so it gets Python to collapse all the myriad ways to call functions # into one manageable form. # # From there, _apply_() digs out the first argument of *args (it's the # function to call), wraps args and kws in guarded accessors, then # calls the function, returning the value. # Transform foo(...) to _apply(foo, ...) walked.args.insert(0, walked.node) walked.node = _apply_name return walked def visitAssName(self, node, walker): """Checks a name assignment using checkName().""" self.checkName(node, node.name) return node def visitFor(self, node, walker): # convert # for x in expr: # to # for x in _getiter(expr): # # Note that visitListCompFor is the same thing. # # Also for list comprehensions: # [... for x in expr ...] # to # [... for x in _getiter(expr) ...] node = walker.defaultVisitNode(node) node.list = ast.CallFunc(_getiter_name, [node.list]) return node visitListCompFor = visitFor def visitGenExprFor(self, node, walker): # convert # (... for x in expr ...) # to # (... for x in _getiter(expr) ...) node = walker.defaultVisitNode(node) node.iter = ast.CallFunc(_getiter_name, [node.iter]) return node def visitGetattr(self, node, walker): """Converts attribute access to a function call. 'foo.bar' becomes '_getattr(foo, "bar")'. Also prevents augmented assignment of attributes, which would be difficult to support correctly. """ self.checkAttrName(node) node = walker.defaultVisitNode(node) if getattr(node, 'in_aug_assign', False): # We're in an augmented assignment # We might support this later... self.error(node, 'Augmented assignment of ' 'attributes is not allowed.') return ast.CallFunc(_getattr_name, [node.expr, ast.Const(node.attrname)]) def visitSubscript(self, node, walker): """Checks all kinds of subscripts. 'foo[bar] += baz' is disallowed. 'a = foo[bar, baz]' becomes 'a = _getitem(foo, (bar, baz))'. 'a = foo[bar]' becomes 'a = _getitem(foo, bar)'. 'a = foo[bar:baz]' becomes 'a = _getitem(foo, slice(bar, baz))'. 'a = foo[:baz]' becomes 'a = _getitem(foo, slice(None, baz))'. 'a = foo[bar:]' becomes 'a = _getitem(foo, slice(bar, None))'. 'del foo[bar]' becomes 'del _write(foo)[bar]'. 'foo[bar] = a' becomes '_write(foo)[bar] = a'. The _write function returns a security proxy. """ node = walker.defaultVisitNode(node) if node.flags == OP_APPLY: # Set 'subs' to the node that represents the subscript or slice. if getattr(node, 'in_aug_assign', False): # We're in an augmented assignment # We might support this later... self.error(node, 'Augmented assignment of ' 'object items and slices is not allowed.') if hasattr(node, 'subs'): # Subscript. subs = node.subs if len(subs) > 1: # example: ob[1,2] subs = ast.Tuple(subs) else: # example: ob[1] subs = subs[0] else: # Slice. # example: obj[0:2] lower = node.lower if lower is None: lower = _None_const upper = node.upper if upper is None: upper = _None_const subs = ast.Sliceobj([lower, upper]) return ast.CallFunc(_getitem_name, [node.expr, subs]) elif node.flags in (OP_DELETE, OP_ASSIGN): # set or remove subscript or slice node.expr = ast.CallFunc(_write_name, [node.expr]) return node visitSlice = visitSubscript def visitAssAttr(self, node, walker): """Checks and mutates attribute assignment. 'a.b = c' becomes '_write(a).b = c'. The _write function returns a security proxy. """ self.checkAttrName(node) node = walker.defaultVisitNode(node) node.expr = ast.CallFunc(_write_name, [node.expr]) return node def visitExec(self, node, walker): self.error(node, 'Exec statements are not allowed.') def visitYield(self, node, walker): self.error(node, 'Yield statements are not allowed.') def visitClass(self, node, walker): """Checks the name of a class using checkName(). Should classes be allowed at all? They don't cause security issues, but they aren't very useful either since untrusted code can't assign instance attributes. """ self.checkName(node, node.name) return walker.defaultVisitNode(node) def visitModule(self, node, walker): """Adds prep code at module scope. Zope doesn't make use of this. The body of Python scripts is always at function scope. """ node = walker.defaultVisitNode(node) self.prepBody(node.node.nodes) return node def visitAugAssign(self, node, walker): """Makes a note that augmented assignment is in use. Note that although augmented assignment of attributes and subscripts is disallowed, augmented assignment of names (such as 'n += 1') is allowed. This could be a problem if untrusted code got access to a mutable database object that supports augmented assignment. """ if node.node.__class__.__name__ == 'Name': node = walker.defaultVisitNode(node) newnode = ast.Assign( [ast.AssName(node.node.name, OP_ASSIGN)], ast.CallFunc( _inplacevar_name, [ast.Const(node.op), ast.Name(node.node.name), node.expr, ] ), ) newnode.lineno = node.lineno return newnode else: node.node.in_aug_assign = True return walker.defaultVisitNode(node) def visitImport(self, node, walker): """Checks names imported using checkName().""" for name, asname in node.names: self.checkName(node, name) if asname: self.checkName(node, asname) return node Zope-2.8.7-final/lib/python/RestrictedPython/SafeMapping.py0000664000076400007640000000315210435500325022703 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __doc__='''Read-only Mapping class based on MultiMapping $Id: SafeMapping.py 24763 2004-05-17 05:59:28Z philikon $''' __version__='$Revision: 1.3 $'[11:-2] from MultiMapping import MultiMapping _marker = [] class SafeMapping(MultiMapping): '''Mapping with security declarations and limited method exposure. Since it subclasses MultiMapping, this class can be used to wrap one or more mapping objects. Restricted Python code will not be able to mutate the SafeMapping or the wrapped mappings, but will be able to read any value. ''' __allow_access_to_unprotected_subobjects__ = 1 push = pop = None def _push(self, ob): MultiMapping.push(self, ob) def _pop(self, *args): if args: return apply(MultiMapping.pop, (self,) + args) else: return MultiMapping.pop(self) def has_get(self, key): v = self.get(key, _marker) if v is _marker: return 0, None else: return 1, v Zope-2.8.7-final/lib/python/RestrictedPython/Eval.py0000664000076400007640000000745710435500325021414 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Restricted Python Expressions $Id: Eval.py 24763 2004-05-17 05:59:28Z philikon $ """ __version__='$Revision: 1.6 $'[11:-2] from RestrictedPython import compile_restricted_eval from string import translate, strip import string nltosp = string.maketrans('\r\n',' ') default_guarded_getattr = getattr # No restrictions. def default_guarded_getitem(ob, index): # No restrictions. return ob[index] PROFILE = 0 class RestrictionCapableEval: """A base class for restricted code.""" globals = {'__builtins__': None} rcode = None # restricted ucode = None # unrestricted used = None def __init__(self, expr): """Create a restricted expression where: expr -- a string containing the expression to be evaluated. """ expr = strip(expr) self.__name__ = expr expr = translate(expr, nltosp) self.expr = expr self.prepUnrestrictedCode() # Catch syntax errors. def prepRestrictedCode(self): if self.rcode is None: if PROFILE: from time import clock start = clock() co, err, warn, used = compile_restricted_eval( self.expr, '') if PROFILE: end = clock() print 'prepRestrictedCode: %d ms for %s' % ( (end - start) * 1000, `self.expr`) if err: raise SyntaxError, err[0] self.used = tuple(used.keys()) self.rcode = co def prepUnrestrictedCode(self): if self.ucode is None: # Use the standard compiler. co = compile(self.expr, '', 'eval') if self.used is None: # Examine the code object, discovering which names # the expression needs. names=list(co.co_names) used={} i=0 code=co.co_code l=len(code) LOAD_NAME=101 HAVE_ARGUMENT=90 while(i < l): c=ord(code[i]) if c==LOAD_NAME: name=names[ord(code[i+1])+256*ord(code[i+2])] used[name]=1 i=i+3 elif c >= HAVE_ARGUMENT: i=i+3 else: i=i+1 self.used=tuple(used.keys()) self.ucode=co def eval(self, mapping): # This default implementation is probably not very useful. :-( # This is meant to be overridden. self.prepRestrictedCode() code = self.rcode d = {'_getattr_': default_guarded_getattr, '_getitem_': default_guarded_getitem} d.update(self.globals) has_key = d.has_key for name in self.used: try: if not has_key(name): d[name] = mapping[name] except KeyError: # Swallow KeyErrors since the expression # might not actually need the name. If it # does need the name, a NameError will occur. pass return eval(code, d) def __call__(self, **kw): return self.eval(kw) Zope-2.8.7-final/lib/python/RestrictedPython/__init__.py0000664000076400007640000000143310435500325022250 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## ''' RestrictedPython package. $Id: __init__.py 24763 2004-05-17 05:59:28Z philikon $ ''' from SelectCompiler import * from PrintCollector import PrintCollector Zope-2.8.7-final/lib/python/RestrictedPython/MutatingWalker.py0000664000076400007640000000463210435500325023453 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.6 $'[11:-2] from SelectCompiler import ast ListType = type([]) TupleType = type(()) SequenceTypes = (ListType, TupleType) class MutatingWalker: def __init__(self, visitor): self.visitor = visitor self._cache = {} def defaultVisitNode(self, node, walker=None, exclude=None): for name, child in node.__dict__.items(): if exclude is not None and name in exclude: continue v = self.dispatchObject(child) if v is not child: # Replace the node. node.__dict__[name] = v return node def visitSequence(self, seq): res = seq for idx in range(len(seq)): child = seq[idx] v = self.dispatchObject(child) if v is not child: # Change the sequence. if type(res) is ListType: res[idx : idx + 1] = [v] else: res = res[:idx] + (v,) + res[idx + 1:] return res def dispatchObject(self, ob): ''' Expected to return either ob or something that will take its place. ''' if isinstance(ob, ast.Node): return self.dispatchNode(ob) elif type(ob) in SequenceTypes: return self.visitSequence(ob) else: return ob def dispatchNode(self, node): klass = node.__class__ meth = self._cache.get(klass, None) if meth is None: className = klass.__name__ meth = getattr(self.visitor, 'visit' + className, self.defaultVisitNode) self._cache[klass] = meth return meth(node, self) def walk(tree, visitor): return MutatingWalker(visitor).dispatchNode(tree) Zope-2.8.7-final/lib/python/RestrictedPython/RCompile.py0000664000076400007640000002102410435500325022221 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Compiles restricted code using the compiler module from the Python standard library. """ __version__='$Revision: 1.6 $'[11:-2] from compiler import ast, parse, misc, syntax, pycodegen from compiler.pycodegen import AbstractCompileMode, Expression, \ Interactive, Module, ModuleCodeGenerator, FunctionCodeGenerator, findOp import MutatingWalker from RestrictionMutator import RestrictionMutator def niceParse(source, filename, mode): try: return parse(source, mode) except: # Try to make a clean error message using # the builtin Python compiler. try: compile(source, filename, mode) except SyntaxError: raise # Some other error occurred. raise class RestrictedCompileMode(AbstractCompileMode): """Abstract base class for hooking up custom CodeGenerator.""" # See concrete subclasses below. def __init__(self, source, filename): self.rm = RestrictionMutator() AbstractCompileMode.__init__(self, source, filename) def parse(self): return niceParse(self.source, self.filename, self.mode) def _get_tree(self): tree = self.parse() MutatingWalker.walk(tree, self.rm) if self.rm.errors: raise SyntaxError, self.rm.errors[0] misc.set_filename(self.filename, tree) syntax.check(tree) return tree def compile(self): tree = self._get_tree() gen = self.CodeGeneratorClass(tree) self.code = gen.getCode() def compileAndTuplize(gen): try: gen.compile() except SyntaxError, v: return None, (str(v),), gen.rm.warnings, gen.rm.used_names return gen.getCode(), (), gen.rm.warnings, gen.rm.used_names def compile_restricted_function(p, body, name, filename, globalize=None): """Compiles a restricted code object for a function. The function can be reconstituted using the 'new' module: new.function(, ) The globalize argument, if specified, is a list of variable names to be treated as globals (code is generated as if each name in the list appeared in a global statement at the top of the function). """ gen = RFunction(p, body, name, filename, globalize) return compileAndTuplize(gen) def compile_restricted_exec(s, filename=''): """Compiles a restricted code suite.""" gen = RModule(s, filename) return compileAndTuplize(gen) def compile_restricted_eval(s, filename=''): """Compiles a restricted expression.""" gen = RExpression(s, filename) return compileAndTuplize(gen) def compile_restricted(source, filename, mode): """Replacement for the builtin compile() function.""" if mode == "single": gen = RInteractive(source, filename) elif mode == "exec": gen = RModule(source, filename) elif mode == "eval": gen = RExpression(source, filename) else: raise ValueError("compile_restricted() 3rd arg must be 'exec' or " "'eval' or 'single'") gen.compile() return gen.getCode() class RestrictedCodeGenerator: """Mixin for CodeGenerator to replace UNPACK_SEQUENCE bytecodes. The UNPACK_SEQUENCE opcode is not safe because it extracts elements from a sequence without using a safe iterator or making __getitem__ checks. This code generator replaces use of UNPACK_SEQUENCE with calls to a function that unpacks the sequence, performes the appropriate security checks, and returns a simple list. """ # Replace the standard code generator for assignments to tuples # and lists. def _gen_safe_unpack_sequence(self, num): # We're at a place where UNPACK_SEQUENCE should be generated, to # unpack num items. That's a security hole, since it exposes # individual items from an arbitrary iterable. We don't remove # the UNPACK_SEQUENCE, but instead insert a call to our _getiter_() # wrapper first. That applies security checks to each item as # it's delivered. codegen is (just) a bit messy because the # iterable is already on the stack, so we have to do a stack swap # to get things in the right order. self.emit('LOAD_GLOBAL', '_getiter_') self.emit('ROT_TWO') self.emit('CALL_FUNCTION', 1) self.emit('UNPACK_SEQUENCE', num) def _visitAssSequence(self, node): if findOp(node) != 'OP_DELETE': self._gen_safe_unpack_sequence(len(node.nodes)) for child in node.nodes: self.visit(child) visitAssTuple = _visitAssSequence visitAssList = _visitAssSequence # Call to generate code for unpacking nested tuple arguments # in function calls. def unpackSequence(self, tup): self._gen_safe_unpack_sequence(len(tup)) for elt in tup: if isinstance(elt, tuple): self.unpackSequence(elt) else: self._nameOp('STORE', elt) # A collection of code generators that adds the restricted mixin to # handle unpacking for all the different compilation modes. They # are defined here (at the end) so that can refer to RestrictedCodeGenerator. class RestrictedFunctionCodeGenerator(RestrictedCodeGenerator, pycodegen.FunctionCodeGenerator): pass class RestrictedExpressionCodeGenerator(RestrictedCodeGenerator, pycodegen.ExpressionCodeGenerator): pass class RestrictedInteractiveCodeGenerator(RestrictedCodeGenerator, pycodegen.InteractiveCodeGenerator): pass class RestrictedModuleCodeGenerator(RestrictedCodeGenerator, pycodegen.ModuleCodeGenerator): def initClass(self): ModuleCodeGenerator.initClass(self) self.__class__.FunctionGen = RestrictedFunctionCodeGenerator # These subclasses work around the definition of stub compile and mode # attributes in the common base class AbstractCompileMode. If it # didn't define new attributes, then the stub code inherited via # RestrictedCompileMode would override the real definitions in # Expression. class RExpression(RestrictedCompileMode, Expression): mode = "eval" CodeGeneratorClass = RestrictedExpressionCodeGenerator class RInteractive(RestrictedCompileMode, Interactive): mode = "single" CodeGeneratorClass = RestrictedInteractiveCodeGenerator class RModule(RestrictedCompileMode, Module): mode = "exec" CodeGeneratorClass = RestrictedModuleCodeGenerator class RFunction(RModule): """A restricted Python function built from parts.""" CodeGeneratorClass = RestrictedModuleCodeGenerator def __init__(self, p, body, name, filename, globals): self.params = p self.body = body self.name = name self.globals = globals or [] RModule.__init__(self, None, filename) def parse(self): # Parse the parameters and body, then combine them. firstline = 'def f(%s): pass' % self.params tree = niceParse(firstline, '', 'exec') f = tree.node.nodes[0] body_code = niceParse(self.body, self.filename, 'exec') # Stitch the body code into the function. f.code.nodes = body_code.node.nodes f.name = self.name # Look for a docstring, if there are any nodes at all if len(f.code.nodes) > 0: stmt1 = f.code.nodes[0] if (isinstance(stmt1, ast.Discard) and isinstance(stmt1.expr, ast.Const) and isinstance(stmt1.expr.value, str)): f.doc = stmt1.expr.value # The caller may specify that certain variables are globals # so that they can be referenced before a local assignment. # The only known example is the variables context, container, # script, traverse_subpath in PythonScripts. if self.globals: f.code.nodes.insert(0, ast.Global(self.globals)) return tree Zope-2.8.7-final/lib/python/RestrictedPython/Utilities.py0000664000076400007640000000447610435500325022476 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.7 $'[11:-2] import string, math, random import DocumentTemplate.sequence utility_builtins = {} utility_builtins['string'] = string utility_builtins['math'] = math utility_builtins['random'] = random utility_builtins['whrandom'] = random # BBB: will be removed in Zope 2.10 utility_builtins['sequence'] = DocumentTemplate.sequence try: import DateTime utility_builtins['DateTime']= DateTime.DateTime except: pass def same_type(arg1, *args): '''Compares the class or type of two or more objects.''' t = getattr(arg1, '__class__', type(arg1)) for arg in args: if getattr(arg, '__class__', type(arg)) is not t: return 0 return 1 utility_builtins['same_type'] = same_type def test(*args): l=len(args) for i in range(1, l, 2): if args[i-1]: return args[i] if l%2: return args[-1] utility_builtins['test'] = test def reorder(s, with=None, without=()): # s, with, and without are sequences treated as sets. # The result is subtract(intersect(s, with), without), # unless with is None, in which case it is subtract(s, without). if with is None: with=s d={} tt=type(()) for i in s: if type(i) is tt and len(i)==2: k, v = i else: k= v = i d[k]=v r=[] a=r.append h=d.has_key for i in without: if type(i) is tt and len(i)==2: k, v = i else: k= v = i if h(k): del d[k] for i in with: if type(i) is tt and len(i)==2: k, v = i else: k= v = i if h(k): a((k,d[k])) del d[k] return r utility_builtins['reorder'] = reorder Zope-2.8.7-final/lib/python/RestrictedPython/PrintCollector.py0000664000076400007640000000162310435500325023455 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.4 $'[11:-2] class PrintCollector: '''Collect written text, and return it when called.''' def __init__(self): self.txt = [] def write(self, text): self.txt.append(text) def __call__(self): return ''.join(self.txt) Zope-2.8.7-final/lib/python/RestrictedPython/tests/0000775000076400007640000000000010435610645021307 5ustar ajungajungZope-2.8.7-final/lib/python/RestrictedPython/tests/class.py0000664000076400007640000000034710435500325022763 0ustar ajungajungclass MyClass: def set(self, val): self.state = val def get(self): return self.state x = MyClass() x.set(12) x.set(x.get() + 1) if x.get() != 13: raise AssertionError, "expected 13, got %d" % x.get() Zope-2.8.7-final/lib/python/RestrictedPython/tests/verify.py0000664000076400007640000001437710435500325023172 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Verify simple properties of bytecode. Some of the transformations performed by the RestrictionMutator are tricky. This module checks the generated bytecode as a way to verify the correctness of the transformations. Violations of some restrictions are obvious from inspection of the bytecode. For example, the bytecode should never contain a LOAD_ATTR call, because all attribute access is performed via the _getattr_() checker function. """ import dis import types def verify(code): """Verify all code objects reachable from code. In particular, traverse into contained code objects in the co_consts table. """ verifycode(code) for ob in code.co_consts: if isinstance(ob, types.CodeType): verify(ob) def verifycode(code): try: _verifycode(code) except: dis.dis(code) raise def _verifycode(code): line = code.co_firstlineno # keep a window of the last three opcodes, with the most recent first window = (None, None, None) for op in disassemble(code): if op.line is not None: line = op.line if op.opname.endswith("LOAD_ATTR"): # All the user code that generates LOAD_ATTR should be # rewritten, but the code generated for a list comp # includes a LOAD_ATTR to extract the append method. if not (op.arg == "append" and window[0].opname == "DUP_TOP" and window[1].opname == "BUILD_LIST"): raise ValueError("direct attribute access %s: %s, %s:%d" % (op.opname, op.arg, co.co_filename, line)) if op.opname in ("STORE_ATTR", "DEL_ATTR"): if not (window[0].opname == "CALL_FUNCTION" and window[2].opname == "LOAD_GLOBAL" and window[2].arg == "_write_"): # check that arg is appropriately wrapped for i, op in enumerate(window): print i, op.opname, op.arg raise ValueError("unguard attribute set/del at %s:%d" % (code.co_filename, line)) if op.opname.startswith("UNPACK"): # An UNPACK opcode extracts items from iterables, and that's # unsafe. The restricted compiler doesn't remove UNPACK opcodes, # but rather *inserts* a call to _getiter_() before each, and # that's the pattern we need to see. if not (window[0].opname == "CALL_FUNCTION" and window[1].opname == "ROT_TWO" and window[2].opname == "LOAD_GLOBAL" and window[2].arg == "_getiter_"): raise ValueError("unguarded unpack sequence at %s:%d" % (code.co_filename, line)) # should check CALL_FUNCTION_{VAR,KW,VAR_KW} but that would # require a potentially unlimited history. need to refactor # the "window" before I can do that. if op.opname == "LOAD_SUBSCR": raise ValueError("unguarded index of sequence at %s:%d" % (code.co_filename, line)) window = (op,) + window[:2] class Op(object): __slots__ = ( "opname", # string, name of the opcode "argcode", # int, the number of the argument "arg", # any, the object, name, or value of argcode "line", # int, line number or None "target", # boolean, is this op the target of a jump "pos", # int, offset in the bytecode ) def __init__(self, opcode, pos): self.opname = dis.opname[opcode] self.arg = None self.line = None self.target = False self.pos = pos def disassemble(co, lasti=-1): code = co.co_code labels = dis.findlabels(code) linestarts = dict(findlinestarts(co)) n = len(code) i = 0 extended_arg = 0 free = co.co_cellvars + co.co_freevars while i < n: op = ord(code[i]) o = Op(op, i) i += 1 if i in linestarts and i > 0: o.line = linestarts[i] if i in labels: o.target = True if op > dis.HAVE_ARGUMENT: arg = ord(code[i]) + ord(code[i+1]) * 256 + extended_arg extended_arg = 0 i += 2 if op == dis.EXTENDED_ARG: extended_arg = arg << 16 o.argcode = arg if op in dis.hasconst: o.arg = co.co_consts[arg] elif op in dis.hasname: o.arg = co.co_names[arg] elif op in dis.hasjrel: o.arg = i + arg elif op in dis.haslocal: o.arg = co.co_varnames[arg] elif op in dis.hascompare: o.arg = dis.cmp_op[arg] elif op in dis.hasfree: o.arg = free[arg] yield o # findlinestarts is copied from Python 2.4's dis module. The code # didn't exist in 2.3, but it would be painful to code disassemble() # without it. def findlinestarts(code): """Find the offsets in a byte code which are start of lines in the source. Generate pairs (offset, lineno) as described in Python/compile.c. """ byte_increments = [ord(c) for c in code.co_lnotab[0::2]] line_increments = [ord(c) for c in code.co_lnotab[1::2]] lastlineno = None lineno = code.co_firstlineno addr = 0 for byte_incr, line_incr in zip(byte_increments, line_increments): if byte_incr: if lineno != lastlineno: yield (addr, lineno) lastlineno = lineno addr += byte_incr lineno += line_incr if lineno != lastlineno: yield (addr, lineno) Zope-2.8.7-final/lib/python/RestrictedPython/tests/testRestrictions.py0000664000076400007640000004073610435500325025254 0ustar ajungajungimport os import re import sys import unittest # Note that nothing should be imported from AccessControl, and in particular # nothing from ZopeGuards.py. Transformed code may need several wrappers # in order to run at all, and most of the production wrappers are defined # in ZopeGuards. But RestrictedPython isn't supposed to depend on # AccessControl, so we need to define throwaway wrapper implementations # here instead. from RestrictedPython import compile_restricted, PrintCollector from RestrictedPython.Eval import RestrictionCapableEval from RestrictedPython.tests import restricted_module, verify from RestrictedPython.RCompile import RModule, RFunction try: __file__ except NameError: __file__ = os.path.abspath(sys.argv[1]) _FILEPATH = os.path.abspath( __file__ ) _HERE = os.path.dirname( _FILEPATH ) def _getindent(line): """Returns the indentation level of the given line.""" indent = 0 for c in line: if c == ' ': indent = indent + 1 elif c == '\t': indent = indent + 8 else: break return indent def find_source(fn, func): """Given a func_code object, this function tries to find and return the python source code of the function. Originally written by Harm van der Heijden (H.v.d.Heijden@phys.tue.nl)""" f = open(fn,"r") for i in range(func.co_firstlineno): line = f.readline() ind = _getindent(line) msg = "" while line: msg = msg + line line = f.readline() # the following should be <= ind, but then we get # confused by multiline docstrings. Using == works most of # the time... but not always! if _getindent(line) == ind: break f.close() return fn, msg def get_source(func): """Less silly interface to find_source""" file = func.func_globals['__file__'] if file.endswith('.pyc'): file = file[:-1] source = find_source(file, func.func_code)[1] assert source.strip(), "Source should not be empty!" return source def create_rmodule(): global rmodule fn = os.path.join(_HERE, 'restricted_module.py') f = open(fn, 'r') source = f.read() f.close() # Sanity check compile(source, fn, 'exec') # Now compile it for real code = compile_restricted(source, fn, 'exec') rmodule = {'__builtins__':{'__import__':__import__, 'None':None, '__name__': 'restricted_module'}} builtins = getattr(__builtins__, '__dict__', __builtins__) for name in ('map', 'reduce', 'int', 'pow', 'range', 'filter', 'len', 'chr', 'ord', ): rmodule[name] = builtins[name] exec code in rmodule class AccessDenied (Exception): pass DisallowedObject = [] class RestrictedObject: disallowed = DisallowedObject allowed = 1 _ = 2 __ = 3 _some_attr = 4 __some_other_attr__ = 5 s = 'Another day, another test...' __writeable_attrs__ = ('writeable',) def __getitem__(self, idx): if idx == 'protected': raise AccessDenied elif idx == 0 or idx == 'safe': return 1 elif idx == 1: return DisallowedObject else: return self.s[idx] def __getslice__(self, lo, hi): return self.s[lo:hi] def __len__(self): return len(self.s) def __setitem__(self, idx, v): if idx == 'safe': self.safe = v else: raise AccessDenied def __setslice__(self, lo, hi, value): raise AccessDenied write = DisallowedObject def guarded_getattr(ob, name): v = getattr(ob, name) if v is DisallowedObject: raise AccessDenied return v SliceType = type(slice(0)) def guarded_getitem(ob, index): if type(index) is SliceType and index.step is None: start = index.start stop = index.stop if start is None: start = 0 if stop is None: v = ob[start:] else: v = ob[start:stop] else: v = ob[index] if v is DisallowedObject: raise AccessDenied return v def minimal_import(name, _globals, _locals, names): if name != "__future__": raise ValueError, "Only future imports are allowed" import __future__ return __future__ class TestGuard: '''A guard class''' def __init__(self, _ob, write=None): self.__dict__['_ob'] = _ob # Write guard methods def __setattr__(self, name, value): _ob = self.__dict__['_ob'] writeable = getattr(_ob, '__writeable_attrs__', ()) if name not in writeable: raise AccessDenied if name[:5] == 'func_': raise AccessDenied setattr(_ob, name, value) def __setitem__(self, index, value): _ob = self.__dict__['_ob'] _ob[index] = value def __setslice__(self, lo, hi, value): _ob = self.__dict__['_ob'] _ob[lo:hi] = value # A wrapper for _apply_. apply_wrapper_called = [] def apply_wrapper(func, *args, **kws): apply_wrapper_called.append('yes') return func(*args, **kws) inplacevar_wrapper_called = {} def inplacevar_wrapper(op, x, y): inplacevar_wrapper_called[op] = x, y # This is really lame. But it's just a test. :) globs = {'x': x, 'y': y} exec 'x'+op+'y' in globs return globs['x'] class RestrictionTests(unittest.TestCase): def execFunc(self, name, *args, **kw): func = rmodule[name] verify.verify(func.func_code) func.func_globals.update({'_getattr_': guarded_getattr, '_getitem_': guarded_getitem, '_write_': TestGuard, '_print_': PrintCollector, # I don't want to write something as involved as ZopeGuard's # SafeIter just for these tests. Using the builtin list() function # worked OK for everything the tests did at the time this was added, # but may fail in the future. If Python 2.1 is no longer an # interesting platform then, using 2.2's builtin iter() here should # work for everything. '_getiter_': list, '_apply_': apply_wrapper, '_inplacevar_': inplacevar_wrapper, }) return func(*args, **kw) def checkPrint(self): for i in range(2): res = self.execFunc('print%s' % i) self.assertEqual(res, 'Hello, world!') def checkPrintToNone(self): try: res = self.execFunc('printToNone') except AttributeError: # Passed. "None" has no "write" attribute. pass else: self.fail(0, res) def checkPrintStuff(self): res = self.execFunc('printStuff') self.assertEqual(res, 'a b c') def checkPrintLines(self): res = self.execFunc('printLines') self.assertEqual(res, '0 1 2\n3 4 5\n6 7 8\n') def checkPrimes(self): res = self.execFunc('primes') self.assertEqual(res, '[2, 3, 5, 7, 11, 13, 17, 19]') def checkAllowedSimple(self): res = self.execFunc('allowed_simple') self.assertEqual(res, 'abcabcabc') def checkAllowedRead(self): self.execFunc('allowed_read', RestrictedObject()) def checkAllowedWrite(self): self.execFunc('allowed_write', RestrictedObject()) def checkAllowedArgs(self): self.execFunc('allowed_default_args', RestrictedObject()) def checkTryMap(self): res = self.execFunc('try_map') self.assertEqual(res, "[2, 3, 4]") def checkApply(self): del apply_wrapper_called[:] res = self.execFunc('try_apply') self.assertEqual(apply_wrapper_called, ["yes"]) self.assertEqual(res, "321") def checkInplace(self): inplacevar_wrapper_called.clear() res = self.execFunc('try_inplace') self.assertEqual(inplacevar_wrapper_called['+='], (1, 3)) def checkDenied(self): for k in rmodule.keys(): if k[:6] == 'denied': try: self.execFunc(k, RestrictedObject()) except AccessDenied: # Passed the test pass else: self.fail('%s() did not trip security' % k) def checkSyntaxSecurity(self): # Ensures that each of the functions in security_in_syntax.py # throws a SyntaxError when using compile_restricted. fn = os.path.join(_HERE, 'security_in_syntax.py') f = open(fn, 'r') source = f.read() f.close() # Unrestricted compile. code = compile(source, fn, 'exec') m = {'__builtins__': {'__import__':minimal_import}} exec code in m for k, v in m.items(): if hasattr(v, 'func_code'): filename, source = find_source(fn, v.func_code) source = "from __future__ import generators\n\n" + source # Now compile it with restrictions try: code = compile_restricted(source, filename, 'exec') except SyntaxError: # Passed the test. pass else: self.fail('%s should not have compiled' % k) def checkOrderOfOperations(self): res = self.execFunc('order_of_operations') self.assertEqual(res, 0) def checkRot13(self): res = self.execFunc('rot13', 'Zope is k00l') self.assertEqual(res, 'Mbcr vf x00y') def checkNestedScopes1(self): res = self.execFunc('nested_scopes_1') self.assertEqual(res, 2) def checkUnrestrictedEval(self): expr = RestrictionCapableEval("{'a':[m.pop()]}['a'] + [m[0]]") v = [12, 34] expect = v[:] expect.reverse() res = expr.eval({'m':v}) self.assertEqual(res, expect) v = [12, 34] res = expr(m=v) self.assertEqual(res, expect) def checkStackSize(self): for k, rfunc in rmodule.items(): if not k.startswith('_') and hasattr(rfunc, 'func_code'): rss = rfunc.func_code.co_stacksize ss = getattr(restricted_module, k).func_code.co_stacksize self.failUnless( rss >= ss, 'The stack size estimate for %s() ' 'should have been at least %d, but was only %d' % (k, ss, rss)) def checkBeforeAndAfter(self): from RestrictedPython.RCompile import RModule from RestrictedPython.tests import before_and_after from compiler import parse defre = re.compile(r'def ([_A-Za-z0-9]+)_(after|before)\(') beforel = [name for name in before_and_after.__dict__ if name.endswith("_before")] for name in beforel: before = getattr(before_and_after, name) before_src = get_source(before) before_src = re.sub(defre, r'def \1(', before_src) rm = RModule(before_src, '') tree_before = rm._get_tree() after = getattr(before_and_after, name[:-6]+'after') after_src = get_source(after) after_src = re.sub(defre, r'def \1(', after_src) tree_after = parse(after_src) self.assertEqual(str(tree_before), str(tree_after)) rm.compile() verify.verify(rm.getCode()) if sys.version_info[:2] >= (2, 4): def checkBeforeAndAfter24(self): from RestrictedPython.RCompile import RModule from RestrictedPython.tests import before_and_after24 from compiler import parse defre = re.compile(r'def ([_A-Za-z0-9]+)_(after|before)\(') beforel = [name for name in before_and_after24.__dict__ if name.endswith("_before")] for name in beforel: before = getattr(before_and_after24, name) before_src = get_source(before) before_src = re.sub(defre, r'def \1(', before_src) rm = RModule(before_src, '') tree_before = rm._get_tree() after = getattr(before_and_after24, name[:-6]+'after') after_src = get_source(after) after_src = re.sub(defre, r'def \1(', after_src) tree_after = parse(after_src) self.assertEqual(str(tree_before), str(tree_after)) rm.compile() verify.verify(rm.getCode()) def _compile_file(self, name): path = os.path.join(_HERE, name) f = open(path, "r") source = f.read() f.close() co = compile_restricted(source, path, "exec") verify.verify(co) return co def checkUnpackSequence(self): co = self._compile_file("unpack.py") calls = [] def getiter(seq): calls.append(seq) return list(seq) globals = {"_getiter_": getiter, '_inplacevar_': inplacevar_wrapper} exec co in globals, {} # The comparison here depends on the exact code that is # contained in unpack.py. # The test doing implicit unpacking in an "except:" clause is # a pain, because there are two levels of unpacking, and the top # level is unpacking the specific TypeError instance constructed # by the test. We have to worm around that one. ineffable = "a TypeError instance" expected = [[1, 2], (1, 2), "12", [1], [1, [2, 3], 4], [2, 3], (1, (2, 3), 4), (2, 3), [1, 2, 3], 2, ('a', 'b'), ((1, 2), (3, 4)), (1, 2), ((1, 2), (3, 4)), (3, 4), ineffable, [42, 666], [[0, 1], [2, 3], [4, 5]], [0, 1], [2, 3], [4, 5], ([[[1, 2]]], [[[3, 4]]]), [[[1, 2]]], [[1, 2]], [1, 2], [[[3, 4]]], [[3, 4]], [3, 4], ] i = expected.index(ineffable) self.assert_(isinstance(calls[i], TypeError)) expected[i] = calls[i] self.assertEqual(calls, expected) def checkUnpackSequenceExpression(self): co = compile_restricted("[x for x, y in [(1, 2)]]", "", "eval") verify.verify(co) calls = [] def getiter(s): calls.append(s) return list(s) globals = {"_getiter_": getiter} exec co in globals, {} self.assertEqual(calls, [[(1,2)], (1, 2)]) def checkUnpackSequenceSingle(self): co = compile_restricted("x, y = 1, 2", "", "single") verify.verify(co) calls = [] def getiter(s): calls.append(s) return list(s) globals = {"_getiter_": getiter} exec co in globals, {} self.assertEqual(calls, [(1, 2)]) def checkClass(self): getattr_calls = [] setattr_calls = [] def test_getattr(obj, attr): getattr_calls.append(attr) return getattr(obj, attr) def test_setattr(obj): setattr_calls.append(obj.__class__.__name__) return obj co = self._compile_file("class.py") globals = {"_getattr_": test_getattr, "_write_": test_setattr, } exec co in globals, {} # Note that the getattr calls don't correspond to the method call # order, because the x.set method is fetched before its arguments # are evaluated. self.assertEqual(getattr_calls, ["set", "set", "get", "state", "get", "state"]) self.assertEqual(setattr_calls, ["MyClass", "MyClass"]) def checkLambda(self): co = self._compile_file("lambda.py") exec co in {}, {} def checkEmpty(self): rf = RFunction("", "", "issue945", "empty.py", {}) rf.parse() rf2 = RFunction("", "# still empty\n\n# by", "issue945", "empty.py", {}) rf2.parse() def checkSyntaxError(self): err = ("def f(x, y):\n" " if x, y < 2 + 1:\n" " return x + y\n" " else:\n" " return x - y\n") self.assertRaises(SyntaxError, compile_restricted, err, "", "exec") create_rmodule() def test_suite(): return unittest.makeSuite(RestrictionTests, 'check') if __name__=='__main__': unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/RestrictedPython/tests/__init__.py0000664000076400007640000000002610435500325023407 0ustar ajungajung'''Python package.''' Zope-2.8.7-final/lib/python/RestrictedPython/tests/before_and_after.py0000664000076400007640000001415110435500325025121 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Restricted Python transformation examples This module contains pairs of functions. Each pair has a before and an after function. The after function shows the source code equivalent of the before function after it has been modified by the restricted compiler. These examples are actually used in the testRestrictions.py checkBeforeAndAfter() unit tests, which verifies that the restricted compiler actually produces the same output as would be output by the normal compiler for the after function. $Id: before_and_after.py 39645 2005-10-26 17:12:35Z jim $ """ # getattr def simple_getattr_before(x): return x.y def simple_getattr_after(x): return _getattr_(x, 'y') # set attr def simple_setattr_before(): x.y = "bar" def simple_setattr_after(): _write_(x).y = "bar" # for loop and list comprehensions def simple_forloop_before(x): for x in [1, 2, 3]: pass def simple_forloop_after(x): for x in _getiter_([1, 2, 3]): pass def nested_forloop_before(x): for x in [1, 2, 3]: for y in "abc": pass def nested_forloop_after(x): for x in _getiter_([1, 2, 3]): for y in _getiter_("abc"): pass def simple_list_comprehension_before(): x = [y**2 for y in whatever if y > 3] def simple_list_comprehension_after(): x = [y**2 for y in _getiter_(whatever) if y > 3] def nested_list_comprehension_before(): x = [x**2 + y**2 for x in whatever1 if x >= 0 for y in whatever2 if y >= x] def nested_list_comprehension_after(): x = [x**2 + y**2 for x in _getiter_(whatever1) if x >= 0 for y in _getiter_(whatever2) if y >= x] # print def simple_print_before(): print "foo" def simple_print_after(): _print = _print_() print >> _print, "foo" # getitem def simple_getitem_before(): return x[0] def simple_getitem_after(): return _getitem_(x, 0) def simple_get_tuple_key_before(): x = y[1,2] def simple_get_tuple_key_after(): x = _getitem_(y, (1,2)) # set item def simple_setitem_before(): x[0] = "bar" def simple_setitem_after(): _write_(x)[0] = "bar" # delitem def simple_delitem_before(): del x[0] def simple_delitem_after(): del _write_(x)[0] # a collection of function parallels to many of the above def function_with_print_before(): def foo(): print "foo" return printed def function_with_print_after(): def foo(): _print = _print_() print >> _print, "foo" return _print() def function_with_getattr_before(): def foo(): return x.y def function_with_getattr_after(): def foo(): return _getattr_(x, 'y') def function_with_setattr_before(): def foo(x): x.y = "bar" def function_with_setattr_after(): def foo(x): _write_(x).y = "bar" def function_with_getitem_before(): def foo(x): return x[0] def function_with_getitem_after(): def foo(x): return _getitem_(x, 0) def function_with_forloop_before(): def foo(): for x in [1, 2, 3]: pass def function_with_forloop_after(): def foo(): for x in _getiter_([1, 2, 3]): pass # this, and all slices, won't work in these tests because the before code # parses the slice as a slice object, while the after code can't generate a # slice object in this way. The after code as written below # is parsed as a call to the 'slice' name, not as a slice object. # XXX solutions? #def simple_slice_before(): # x = y[:4] #def simple_slice_after(): # _getitem = _getitem_ # x = _getitem(y, slice(None, 4)) # Assignment stmts in Python can be very complicated. The "no_unpack" # test makes sure we're not doing unnecessary rewriting. def no_unpack_before(): x = y x = [y] x = y, x = (y, (y, y), [y, (y,)], x, (x, y)) x = y = z = (x, y, z) no_unpack_after = no_unpack_before # that is, should be untouched # apply() variations. Native apply() is unsafe because, e.g., # # def f(a, b, c): # whatever # # apply(f, two_element_sequence, dict_with_key_c) # # or (different spelling of the same thing) # # f(*two_element_sequence, **dict_with_key_c) # # makes the elements of two_element_sequence visible to f via its 'a' and # 'b' arguments, and the dict_with_key_c['c'] value visible via its 'c' # argument. That is, it's a devious way to extract values without going # thru security checks. def star_call_before(): foo(*a) def star_call_after(): _apply_(foo, *a) def star_call_2_before(): foo(0, *a) def star_call_2_after(): _apply_(foo, 0, *a) def starstar_call_before(): foo(**d) def starstar_call_after(): _apply_(foo, **d) def star_and_starstar_call_before(): foo(*a, **d) def star_and_starstar_call_after(): _apply_(foo, *a, **d) def positional_and_star_and_starstar_call_before(): foo(b, *a, **d) def positional_and_star_and_starstar_call_after(): _apply_(foo, b, *a, **d) def positional_and_defaults_and_star_and_starstar_call_before(): foo(b, x=y, w=z, *a, **d) def positional_and_defaults_and_star_and_starstar_call_after(): _apply_(foo, b, x=y, w=z, *a, **d) def lambda_with_getattr_in_defaults_before(): f = lambda x=y.z: x def lambda_with_getattr_in_defaults_after(): f = lambda x=_getattr_(y, "z"): x # augmented operators # Note that we don't have to worry about item, attr, or slice assignment, # as they are disallowed. Yay! ## def inplace_id_add_before(): ## x += y+z ## def inplace_id_add_after(): ## x = _inplacevar_('+=', x, y+z) Zope-2.8.7-final/lib/python/RestrictedPython/tests/before_and_after24.py0000664000076400007640000000323610435500325025271 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Restricted Python transformation examples This module contains pairs of functions. Each pair has a before and an after function. The after function shows the source code equivalent of the before function after it has been modified by the restricted compiler. These examples are actually used in the testRestrictions.py checkBeforeAndAfter() unit tests, which verifies that the restricted compiler actually produces the same output as would be output by the normal compiler for the after function. $Id: before_and_after24.py 39646 2005-10-26 17:12:37Z jim $ """ def simple_generator_expression_before(): x = (y**2 for y in whatever if y > 3) def simple_generator_expression_after(): x = (y**2 for y in _getiter_(whatever) if y > 3) def nested_generator_expression_before(): x = (x**2 + y**2 for x in whatever1 if x >= 0 for y in whatever2 if y >= x) def nested_generator_expression_after(): x = (x**2 + y**2 for x in _getiter_(whatever1) if x >= 0 for y in _getiter_(whatever2) if y >= x) Zope-2.8.7-final/lib/python/RestrictedPython/tests/unpack.py0000664000076400007640000000263510435500325023141 0ustar ajungajung# A series of short tests for unpacking sequences. def u1(L): x, y = L assert x == 1 assert y == 2 u1([1,2]) u1((1, 2)) def u1a(L): x, y = L assert x == '1' assert y == '2' u1a("12") try: u1([1]) except ValueError: pass else: raise AssertionError, "expected 'unpack list of wrong size'" def u2(L): x, (a, b), y = L assert x == 1 assert a == 2 assert b == 3 assert y == 4 u2([1, [2, 3], 4]) u2((1, (2, 3), 4)) try: u2([1, 2, 3]) except TypeError: pass else: raise AssertionError, "expected 'iteration over non-sequence'" def u3((x, y)): assert x == 'a' assert y == 'b' return x, y u3(('a', 'b')) def u4(x): (a, b), c = d, (e, f) = x assert a == 1 and b == 2 and c == (3, 4) assert d == (1, 2) and e == 3 and f == 4 u4( ((1, 2), (3, 4)) ) def u5(x): try: raise TypeError(x) # This one is tricky to test, because the first level of unpacking # has a TypeError instance. That's a headache for the test driver. except TypeError, [(a, b)]: assert a == 42 assert b == 666 u5([42, 666]) def u6(x): expected = 0 for i, j in x: assert i == expected expected += 1 assert j == expected expected += 1 u6([[0, 1], [2, 3], [4, 5]]) def u7(x): stuff = [i + j for toplevel, in x for i, j in toplevel] assert stuff == [3, 7] u7( ([[[1, 2]]], [[[3, 4]]]) ) Zope-2.8.7-final/lib/python/RestrictedPython/tests/security_in_syntax.py0000664000076400007640000000265010435500325025620 0ustar ajungajung# These are all supposed to raise a SyntaxError when using # compile_restricted() but not when using compile(). # Each function in this module is compiled using compile_restricted(). from __future__ import generators def overrideGuardWithFunction(): def _getattr(o): return o def overrideGuardWithLambda(): lambda o, _getattr=None: o def overrideGuardWithClass(): class _getattr: pass def overrideGuardWithName(): _getattr = None def overrideGuardWithArgument(): def f(_getattr=None): pass def reserved_names(): printed = '' def bad_name(): __ = 12 def bad_attr(): some_ob._some_attr = 15 def no_exec(): exec 'q = 1' def no_yield(): yield 42 def check_getattr_in_lambda(arg=lambda _getattr=(lambda ob, name: name): _getattr): 42 def import_as_bad_name(): import os as _leading_underscore def except_using_bad_name(): try: foo except NameError, _leading_underscore: # The name of choice (say, _write) is now assigned to an exception # object. Hard to exploit, but conceivable. pass def keyword_arg_with_bad_name(): def f(okname=1, __badname=2): pass def no_augmeneted_assignment_to_sub(): a[b] += c def no_augmeneted_assignment_to_attr(): a.b += c def no_augmeneted_assignment_to_slice(): a[x:y] += c def no_augmeneted_assignment_to_slice2(): a[x:y:z] += c Zope-2.8.7-final/lib/python/RestrictedPython/tests/restricted_module.py0000664000076400007640000000657210435500325025401 0ustar ajungajungfrom __future__ import nested_scopes import sys def print0(): print 'Hello, world!', return printed def print1(): print 'Hello,', print 'world!', return printed def printStuff(): print 'a', 'b', 'c', return printed def printToNone(): x = None print >>x, 'Hello, world!', return printed def printLines(): # This failed before Zope 2.4.0a2 r = range(3) for n in r: for m in r: print m + n * len(r), print return printed def try_map(): inc = lambda i: i+1 x = [1, 2, 3] print map(inc, x), return printed def try_apply(): def f(x, y, z): return x + y + z print f(*(300, 20), **{'z': 1}), return printed def try_inplace(): x = 1 x += 3 def primes(): # Somewhat obfuscated code on purpose print filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0, map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,20))), return printed def allowed_read(ob): print ob.allowed print ob.s print ob[0] print ob[2] print ob[3:-1] print len(ob) return printed def allowed_default_args(ob): def f(a=ob.allowed, s=ob.s): return a, s def allowed_simple(): q = {'x':'a'} q['y'] = 'b' q.update({'z': 'c'}) r = ['a'] r.append('b') r[2:2] = ['c'] s = 'a' s = s[:100] + 'b' s += 'c' if sys.version_info >= (2, 3): t = ['l', 'm', 'n', 'o', 'p', 'q'] t[1:5:2] = ['n', 'p'] _ = q return q['x'] + q['y'] + q['z'] + r[0] + r[1] + r[2] + s def allowed_write(ob): ob.writeable = 1 #ob.writeable += 1 [1 for ob.writeable in 1,2] ob['safe'] = 2 #ob['safe'] += 2 [1 for ob['safe'] in 1,2] def denied_print(ob): print >> ob, 'Hello, world!', def denied_getattr(ob): #ob.disallowed += 1 ob.disallowed = 1 return ob.disallowed def denied_default_args(ob): def f(d=ob.disallowed): return d def denied_setattr(ob): ob.allowed = -1 def denied_setattr2(ob): #ob.allowed += -1 ob.allowed = -1 def denied_setattr3(ob): [1 for ob.allowed in 1,2] def denied_getitem(ob): ob[1] def denied_getitem2(ob): #ob[1] += 1 ob[1] def denied_setitem(ob): ob['x'] = 2 def denied_setitem2(ob): #ob[0] += 2 ob['x'] = 2 def denied_setitem3(ob): [1 for ob['x'] in 1,2] def denied_setslice(ob): ob[0:1] = 'a' def denied_setslice2(ob): #ob[0:1] += 'a' ob[0:1] = 'a' def denied_setslice3(ob): [1 for ob[0:1] in 1,2] ##def strange_attribute(): ## # If a guard has attributes with names that don't start with an ## # underscore, those attributes appear to be an attribute of ## # anything. ## return [].attribute_of_anything def order_of_operations(): return 3 * 4 * -2 + 2 * 12 def rot13(ss): mapping = {} orda = ord('a') ordA = ord('A') for n in range(13): c1 = chr(orda + n) c2 = chr(orda + n + 13) c3 = chr(ordA + n) c4 = chr(ordA + n + 13) mapping[c1] = c2 mapping[c2] = c1 mapping[c3] = c4 mapping[c4] = c3 del c1, c2, c3, c4, orda, ordA res = '' for c in ss: res = res + mapping.get(c, c) return res def nested_scopes_1(): # Fails if 'a' is consumed by the first function. a = 1 def f1(): return a def f2(): return a return f1() + f2() class Classic: pass Zope-2.8.7-final/lib/python/RestrictedPython/tests/lambda.py0000664000076400007640000000014210435500325023067 0ustar ajungajungf = lambda x, y=1: x + y if f(2) != 3: raise ValueError if f(2, 2) != 4: raise ValueError Zope-2.8.7-final/lib/python/RestrictedPython/notes.txt0000664000076400007640000000664510435500325022042 0ustar ajungajungHow it works ============ Every time I see this code, I have to relearn it. These notes will hopefully make this a little easier. :) - The important module is RCompile. The entry points are the compile_restricted_* functions. + compile_restricted_function is used by Python scripts. + compile_restricted_eval is used by ZPT and by DTML indirectly through Eval.RestrictionCapableEval. - OK, so lets see how this works by following the logic of compile_restricted_eval. - First, we create an RExpression, passing the source and a "file name", to be used in tracebacks. Now, an RExpression is just: + a subclass of RestrictedCompileMode and Expression. Expression is a subclass of AbstractCompileMode that sets it's mode to 'eval' and everided compile. Sigh. + RestrictedCompileMode is a subclass of AbstractCompileMode that changes a bunch of things. :) These include compile, so we can ignore the compile we got from Expression. It would have been simpler to just set the dang mode in RExpression. Sigh. RestrictedCompileMode seem to be the interestng base class. I assume it implements the interesting functionality. We'll see below... - Next, we call compileAndTuplize. + This calls compile on the RExpression. It has an error handler that does something that I hope I don't care about. :) + It then calls the genCode method on the RExpression. This is boring, so we'll not worry about it. - The compile method provided by RestrictedCompileMode is interesting. + First it calls _get_tree. * It uses compiler.parse to parse the source * it uses MutatingWalker.walk to mutaate the tree using the RestrictedCompileMode's 'rm' attr, which is a RestrictionMutator. The RestrictionMutator has the recipies for mutating the parse tree. (Note, for comparison, that Zope3's zope.security.untrustedpython.rcompile module an alternative RestrictionMutator that provides a much smaller set of changes.) A mutator has visit method for different kinds of AST nodes. These visit methods may mutate nodes or return new nodes that replace the originally visited nodes. There is a default visitor that visits a node's children and replaces the children who's visitors returned new nodes. The walk function just calls the visitor for the root node of the given tree. Note _get_tree ignores the walk return value, thus assuming that the visitor for the root node doesn't return a new node. This is a theoretical bug that we can ignore. + Second, it generates the code. This too is boring. - So this seems simple enough. ;) When we want to add a check, we need to update or add a visit function in RestrictionMutator. How does a visit function work. - First, we usually call walker.defaultVisitNode(node). This transforms the node's child nodes. - Then we hack the node, or possibly return the node. To do this, we have to know how the node works. - The hack often involved changing the code to call some checker function. These have names like _name_. These are names that would be illegal in the input source. If this is a new function, we have to provide it in AccessControl.ZopeGuards._safe_globals. - Don't forget to add a test case to tests.before_and_after. Zope-2.8.7-final/lib/python/RestrictedPython/SelectCompiler.py0000664000076400007640000000204010435500325023416 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Compiler selector. $Id: SelectCompiler.py 24763 2004-05-17 05:59:28Z philikon $ """ # Use the compiler from the standard library. import compiler from compiler import ast from compiler.transformer import parse from compiler.consts import OP_ASSIGN, OP_DELETE, OP_APPLY from RCompile import \ compile_restricted, \ compile_restricted_function, \ compile_restricted_exec, \ compile_restricted_eval Zope-2.8.7-final/lib/python/RestrictedPython/Guards.py0000664000076400007640000001046010435500325021736 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__ = '$Revision: 1.14 $'[11:-2] import exceptions # This tiny set of safe builtins is extended by users of the module. # AccessControl.ZopeGuards contains a large set of wrappers for builtins. # DocumentTemplate.DT_UTil contains a few. safe_builtins = {} for name in ['False', 'None', 'True', 'abs', 'basestring', 'bool', 'callable', 'chr', 'cmp', 'complex', 'divmod', 'float', 'hash', 'hex', 'id', 'int', 'isinstance', 'issubclass', 'len', 'long', 'oct', 'ord', 'pow', 'range', 'repr', 'round', 'str', 'tuple', 'unichr', 'unicode', 'xrange', 'zip']: safe_builtins[name] = __builtins__[name] # Wrappers provided by this module: # delattr # setattr # Wrappers provided by ZopeGuards: # __import__ # apply # dict # enumerate # filter # getattr # hasattr # iter # list # map # max # min # sum # Builtins that are intentionally disabled # compile - don't let them produce new code # dir - a general purpose introspector, probably hard to wrap # execfile - no direct I/O # file - no direct I/O # globals - uncontrolled namespace access # input - no direct I/O # locals - uncontrolled namespace access # open - no direct I/O # raw_input - no direct I/O # vars - uncontrolled namespace access # There are several strings that describe Python. I think there's no # point to including these, although they are obviously safe: # copyright, credits, exit, help, license, quit # Not provided anywhere. Do something about these? Several are # related to new-style classes, which we are too scared of to support # <0.3 wink>. coerce, buffer, and reload are esoteric enough that no # one should care. # buffer # classmethod # coerce # eval # intern # object # property # reload # slice # staticmethod # super # type for name in dir(exceptions): if name[0] != "_": safe_builtins[name] = getattr(exceptions, name) def _write_wrapper(): # Construct the write wrapper class def _handler(secattr, error_msg): # Make a class method. def handler(self, *args): try: f = getattr(self.ob, secattr) except AttributeError: raise TypeError, error_msg f(*args) return handler class Wrapper: def __len__(self): # Required for slices with negative bounds. return len(self.ob) def __init__(self, ob): self.__dict__['ob'] = ob __setitem__ = _handler('__guarded_setitem__', 'object does not support item or slice assignment') __delitem__ = _handler('__guarded_delitem__', 'object does not support item or slice assignment') __setattr__ = _handler('__guarded_setattr__', 'attribute-less object (assign or del)') __delattr__ = _handler('__guarded_delattr__', 'attribute-less object (assign or del)') return Wrapper def _full_write_guard(): # Nested scope abuse! # safetype and Wrapper variables are used by guard() safetype = {dict: True, list: True}.has_key Wrapper = _write_wrapper() def guard(ob): # Don't bother wrapping simple types, or objects that claim to # handle their own write security. if safetype(type(ob)) or hasattr(ob, '_guarded_writes'): return ob # Hand the object to the Wrapper instance, then return the instance. return Wrapper(ob) return guard full_write_guard = _full_write_guard() def guarded_setattr(object, name, value): setattr(full_write_guard(object), name, value) safe_builtins['setattr'] = guarded_setattr def guarded_delattr(object, name): delattr(full_write_guard(object), name) safe_builtins['delattr'] = guarded_delattr Zope-2.8.7-final/lib/python/RestrictedPython/Limits.py0000664000076400007640000000326610435500325021760 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.5 $'[11:-2] limited_builtins = {} def limited_range(iFirst, *args): # limited range function from Martijn Pieters RANGELIMIT = 1000 if not len(args): iStart, iEnd, iStep = 0, iFirst, 1 elif len(args) == 1: iStart, iEnd, iStep = iFirst, args[0], 1 elif len(args) == 2: iStart, iEnd, iStep = iFirst, args[0], args[1] else: raise AttributeError, 'range() requires 1-3 int arguments' if iStep == 0: raise ValueError, 'zero step for range()' iLen = int((iEnd - iStart) / iStep) if iLen < 0: iLen = 0 if iLen >= RANGELIMIT: raise ValueError, 'range() too large' return range(iStart, iEnd, iStep) limited_builtins['range'] = limited_range def limited_list(seq): if isinstance(seq, str): raise TypeError, 'cannot convert string to list' return list(seq) limited_builtins['list'] = limited_list def limited_tuple(seq): if isinstance(seq, str): raise TypeError, 'cannot convert string to tuple' return tuple(seq) limited_builtins['tuple'] = limited_tuple Zope-2.8.7-final/lib/python/Persistence/0000775000076400007640000000000010435610645017117 5ustar ajungajungZope-2.8.7-final/lib/python/Persistence/mapping.py0000664000076400007640000000260210435502403021114 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Python implementation of persistent base types $Id: mapping.py 25186 2004-06-02 15:07:33Z jim $""" import Persistence import persistent from persistent.mapping import PersistentMapping if Persistence.Persistent is not persistent.Persistent: class PersistentMapping(Persistence.Persistent, PersistentMapping): """Legacy persistent mapping class This class mixes in ExtensionClass Base if it is present. Unless you actually want ExtensionClass semantics, use persistent.mapping.PersistentMapping instead. """ def __setstate__(self, state): if 'data' not in state: state['data'] = state['_container'] del state['_container'] self.__dict__.update(state) Zope-2.8.7-final/lib/python/Persistence/__init__.py0000664000076400007640000000364110435502403021224 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Persistence and ExtensionClass combined $Id: __init__.py 25186 2004-06-02 15:07:33Z jim $ """ from persistent import PickleCache try: from _Persistence import Persistent except: from warnings import warn warn("""Couldn't import the ExtensionClass-based base class There are two possibilities: 1. You don't care about ExtensionClass. You are importing Persistence because that's what you imported in the past. In this case, you should really use the persistent package instead: >>> from persistent import Persistent >>> from persistent.list import PersistentList >>> from persistent.mapping import PersistentMapping 2. You want your classes to be ExtensionClasses. In this case, you need to install the ExtensionClass package separately. ExtensionClass is no-longer included with ZODB3. """) from persistent import Persistent Overridable = Persistent from Persistence.mapping import PersistentMapping # This is a travesty. Whimper. The Data.fs.in used in Zope 2 have # ancient pickles refering to BoboPOS. Waaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa! import sys sys.modules['BoboPOS'] = sys.modules['Persistence'] sys.modules['BoboPOS.PersistentMapping'] = sys.modules['Persistence.mapping'] del sys Zope-2.8.7-final/lib/python/Persistence/_Persistence.c0000664000076400007640000000763410435502403021710 0ustar ajungajung/* Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE. */ static char _Persistence_module_documentation[] = "Persistent ExtensionClass\n" "\n" "$Id: _Persistence.c 30615 2005-06-02 22:03:32Z tim_one $\n" ; #include "ExtensionClass.h" #include "cPersistence.h" /* convert_name() returns a new reference to a string name or sets an exception and returns NULL. */ static PyObject * convert_name(PyObject *name) { #ifdef Py_USING_UNICODE /* The Unicode to string conversion is done here because the existing tp_setattro slots expect a string object as name and we wouldn't want to break those. */ if (PyUnicode_Check(name)) { name = PyUnicode_AsEncodedString(name, NULL, NULL); } else #endif if (!PyString_Check(name)) { PyErr_SetString(PyExc_TypeError, "attribute name must be a string"); return NULL; } else Py_INCREF(name); return name; } /* Returns true if the object requires unghostification. There are several special attributes that we allow access to without requiring that the object be unghostified: __class__ __del__ __dict__ __of__ __setstate__ */ static int unghost_getattr(const char *s) { if (*s++ != '_') return 1; if (*s == 'p') { s++; if (*s == '_') return 0; /* _p_ */ else return 1; } else if (*s == '_') { s++; switch (*s) { case 'c': return strcmp(s, "class__"); case 'd': s++; if (!strcmp(s, "el__")) return 0; /* __del__ */ if (!strcmp(s, "ict__")) return 0; /* __dict__ */ return 1; case 'o': return strcmp(s, "of__"); case 's': return strcmp(s, "setstate__"); default: return 1; } } return 1; } static PyObject * P_getattr(cPersistentObject *self, PyObject *name) { PyObject *v=NULL; char *s; name = convert_name(name); if (!name) return NULL; s = PyString_AS_STRING(name); if (*s != '_' || unghost_getattr(s)) { if (PER_USE(self)) { v = Py_FindAttr((PyObject*)self, name); PER_ALLOW_DEACTIVATION(self); PER_ACCESSED(self); } } else v = Py_FindAttr((PyObject*)self, name); Py_DECREF(name); return v; } static PyTypeObject Ptype = { PyObject_HEAD_INIT(NULL) /* ob_size */ 0, /* tp_name */ "Persistence.Persistent", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* tp_getattro */ (getattrofunc)P_getattr, 0, 0, /* tp_flags */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE , /* tp_doc */ "Persistent ExtensionClass", }; static struct PyMethodDef _Persistence_methods[] = { {NULL, (PyCFunction)NULL, 0, NULL} /* sentinel */ }; #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ #define PyMODINIT_FUNC void #endif PyMODINIT_FUNC init_Persistence(void) { PyObject *m; if (! ExtensionClassImported) return; cPersistenceCAPI = PyCObject_Import("persistent.cPersistence", "CAPI"); if (cPersistenceCAPI == NULL) return; Ptype.tp_bases = Py_BuildValue("OO", cPersistenceCAPI->pertype, ECBaseType); if (Ptype.tp_bases == NULL) return; Ptype.tp_base = cPersistenceCAPI->pertype; Ptype.ob_type = ECExtensionClassType; if (PyType_Ready(&Ptype) < 0) return; /* Create the module and add the functions */ m = Py_InitModule3("_Persistence", _Persistence_methods, _Persistence_module_documentation); if (m == NULL) return; /* Add types: */ if (PyModule_AddObject(m, "Persistent", (PyObject *)&Ptype) < 0) return; } Zope-2.8.7-final/lib/python/Persistence/tests/0000775000076400007640000000000010435610645020261 5ustar ajungajungZope-2.8.7-final/lib/python/Persistence/tests/test_ExtensionClass.py0000664000076400007640000003031010435502403024621 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test ExtensionClass support in Persistence.Persistent $Id: test_ExtensionClass.py 29891 2005-04-06 22:35:05Z tim_one $ """ from zope.testing.doctest import DocTestSuite import pickle from Persistence import Persistent try: # The _Persistence module is only compiled in a Zope checkout, # where ExtensionClass is available. import Persistence._Persistence except ImportError: pass else: def test_basic(): """ >>> from ExtensionClass import Base - Support for a class initialiser: >>> class C(Persistent): ... def __class_init__(self): ... print 'class init called' ... print self.__name__ ... def bar(self): ... return 'bar called' class init called C >>> c = C() >>> int(c.__class__ is C) 1 >>> int(c.__class__ is type(c)) 1 - Provide an inheritedAttribute method for looking up attributes in base classes: >>> class C2(C): ... def bar(*a): ... return C2.inheritedAttribute('bar')(*a), 42 class init called C2 >>> o = C2() >>> o.bar() ('bar called', 42) This is for compatability with old code. New code should use super instead. The base class, Base, exists mainly to support the __of__ protocol. The __of__ protocol is similar to __get__ except that __of__ is called when an implementor is retrieved from an instance as well as from a class: >>> class O(Base): ... def __of__(*a): ... return a >>> o1 = O() >>> o2 = O() >>> C.o1 = o1 >>> c.o2 = o2 >>> c.o1 == (o1, c) 1 >>> C.o1 == o1 1 >>> int(c.o2 == (o2, c)) 1 We accomplish this by making a class that implements __of__ a descriptor and treating all descriptor ExtensionClasses this way. That is, if an extension class is a descriptor, it's __get__ method will be called even when it is retrieved from an instance. >>> class O(Base): ... def __get__(*a): ... return a ... >>> o1 = O() >>> o2 = O() >>> C.o1 = o1 >>> c.o2 = o2 >>> int(c.o1 == (o1, c, type(c))) 1 >>> int(C.o1 == (o1, None, type(c))) 1 >>> int(c.o2 == (o2, c, type(c))) 1 """ def test_mixing(): """Test working with a classic class >>> class Classic: ... def x(self): ... return 42 >>> class O(Persistent): ... def __of__(*a): ... return a >>> class O2(Classic, O): ... def __of__(*a): ... return (O2.inheritedAttribute('__of__')(*a), ... O2.inheritedAttribute('x')(a[0])) >>> class C(Persistent): ... def __class_init__(self): ... print 'class init called' ... print self.__name__ ... def bar(self): ... return 'bar called' class init called C >>> c = C() >>> o2 = O2() >>> c.o2 = o2 >>> int(c.o2 == ((o2, c), 42)) 1 Test working with a new style >>> class Modern(object): ... def x(self): ... return 42 >>> class O2(Modern, O): ... def __of__(*a): ... return (O2.inheritedAttribute('__of__')(*a), ... O2.inheritedAttribute('x')(a[0])) >>> o2 = O2() >>> c.o2 = o2 >>> int(c.o2 == ((o2, c), 42)) 1 """ def proper_error_on_deleattr(): """ Florent Guillaume wrote: ... Excellent. Will it also fix this particularity of ExtensionClass: >>> class A(Persistent): ... def foo(self): ... self.gee ... def bar(self): ... del self.gee >>> a=A() >>> a.foo() Traceback (most recent call last): ... AttributeError: gee >>> a.bar() Traceback (most recent call last): ... AttributeError: 'A' object has no attribute 'gee' I.e., the fact that KeyError is raised whereas a normal class would raise AttributeError. """ def test__basicnew__(): """ >>> x = Simple.__basicnew__() >>> x.__dict__ {} """ def test_setattr_on_extension_type(): """ >>> for name in 'x', '_x', 'x_', '__x_y__', '___x__', '__x___', '_x_': ... setattr(Persistent, name, 1) ... print getattr(Persistent, name) ... delattr(Persistent, name) ... print getattr(Persistent, name, 0) 1 0 1 0 1 0 1 0 1 0 1 0 1 0 >>> Persistent.__foo__ = 1 Traceback (most recent call last): ... TypeError: can't set attributes of built-in/extension type """ \ """'Persistence.Persistent' if the attribute name begins """ \ """and ends with __ and contains only 4 _ characters >>> Persistent.__foo__ Traceback (most recent call last): ... AttributeError: type object 'Persistence.Persistent' """ \ """has no attribute '__foo__' >>> del Persistent.__foo__ Traceback (most recent call last): ... TypeError: can't set attributes of built-in/extension type """ \ """'Persistence.Persistent' if the attribute name begins """ \ """and ends with __ and contains only 4 _ characters """ def test_class_creation_under_stress(): """ >>> for i in range(100): ... class B(Persistent): ... print i, ... if i and i%20 == 0: ... print 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 >>> import gc >>> x = gc.collect() """ def print_dict(d): d = d.items() d.sort() print '{%s}' % (', '.join( [('%r: %r' % (k, v)) for (k, v) in d] )) def cmpattrs(self, other, *attrs): for attr in attrs: if attr[:3] in ('_v_', '_p_'): continue c = cmp(getattr(self, attr, None), getattr(other, attr, None)) if c: return c return 0 class Simple(Persistent): def __init__(self, name, **kw): self.__name__ = name self.__dict__.update(kw) self._v_favorite_color = 'blue' self._p_foo = 'bar' def __cmp__(self, other): return cmpattrs(self, other, '__class__', *(self.__dict__.keys())) def test_basic_pickling(): """ >>> x = Simple('x', aaa=1, bbb='foo') >>> print_dict(x.__getstate__()) {'__name__': 'x', 'aaa': 1, 'bbb': 'foo'} >>> f, (c,), state = x.__reduce__() >>> f.__name__ '__newobj__' >>> f.__module__ 'copy_reg' >>> c.__name__ 'Simple' >>> print_dict(state) {'__name__': 'x', 'aaa': 1, 'bbb': 'foo'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.__setstate__({'z': 1}) >>> x.__dict__ {'z': 1} """ class Custom(Simple): def __new__(cls, x, y): r = Persistent.__new__(cls) r.x, r.y = x, y return r def __init__(self, x, y): self.a = 42 def __getnewargs__(self): return self.x, self.y def __getstate__(self): return self.a def __setstate__(self, a): self.a = a def test_pickling_w_overrides(): """ >>> x = Custom('x', 'y') >>> x.a = 99 >>> (f, (c, ax, ay), a) = x.__reduce__() >>> f.__name__ '__newobj__' >>> f.__module__ 'copy_reg' >>> c.__name__ 'Custom' >>> ax, ay, a ('x', 'y', 99) >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ class Slotted(Persistent): __slots__ = 's1', 's2', '_p_splat', '_v_eek' def __init__(self, s1, s2): self.s1, self.s2 = s1, s2 self._v_eek = 1 self._p_splat = 2 class SubSlotted(Slotted): __slots__ = 's3', 's4' def __init__(self, s1, s2, s3): Slotted.__init__(self, s1, s2) self.s3 = s3 def __cmp__(self, other): return cmpattrs(self, other, '__class__', 's1', 's2', 's3', 's4') def test_pickling_w_slots_only(): """ >>> x = SubSlotted('x', 'y', 'z') >>> d, s = x.__getstate__() >>> d >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> d >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ class SubSubSlotted(SubSlotted): def __init__(self, s1, s2, s3, **kw): SubSlotted.__init__(self, s1, s2, s3) self.__dict__.update(kw) self._v_favorite_color = 'blue' self._p_foo = 'bar' def __cmp__(self, other): return cmpattrs(self, other, '__class__', 's1', 's2', 's3', 's4', *(self.__dict__.keys())) def test_pickling_w_slots(): """ >>> x = SubSubSlotted('x', 'y', 'z', aaa=1, bbb='foo') >>> d, s = x.__getstate__() >>> print_dict(d) {'aaa': 1, 'bbb': 'foo'} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> print_dict(d) {'aaa': 1, 'bbb': 'foo'} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ def test_pickling_w_slots_w_empty_dict(): """ >>> x = SubSubSlotted('x', 'y', 'z') >>> d, s = x.__getstate__() >>> print_dict(d) {} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> print_dict(d) {} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ def test_suite(): return DocTestSuite() Zope-2.8.7-final/lib/python/Persistence/tests/__init__.py0000664000076400007640000000001210435502403022353 0ustar ajungajung# package Zope-2.8.7-final/lib/python/Persistence/tests/test_mapping.py0000664000076400007640000000414710435502403023323 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ $Id: test_mapping.py 29891 2005-04-06 22:35:05Z tim_one $ """ import unittest from zope.testing.doctest import DocTestSuite from Persistence import PersistentMapping def test_basic_functionality(): """ >>> m = PersistentMapping({'x': 1}, a=2, b=3) >>> m['name'] = 'bob' >>> m['fred'] Traceback (most recent call last): ... KeyError: 'fred' >>> m.get('fred') >>> m.get('fred', 42) 42 >>> m.get('name', 42) 'bob' >>> m.get('name') 'bob' >>> m['name'] 'bob' >>> keys = m.keys() >>> keys.sort() >>> keys ['a', 'b', 'name', 'x'] >>> values = m.values() >>> values.sort() >>> values [1, 2, 3, 'bob'] >>> items = m.items() >>> items.sort() >>> items [('a', 2), ('b', 3), ('name', 'bob'), ('x', 1)] >>> keys = list(m.iterkeys()) >>> keys.sort() >>> keys ['a', 'b', 'name', 'x'] >>> values = list(m.itervalues()) >>> values.sort() >>> values [1, 2, 3, 'bob'] >>> items = list(m.iteritems()) >>> items.sort() >>> items [('a', 2), ('b', 3), ('name', 'bob'), ('x', 1)] >>> 'name' in m True """ def test_old_pickles(): """ >>> m = PersistentMapping() >>> m.__setstate__({'_container': {'x': 1, 'y': 2}}) >>> items = m.items() >>> items.sort() >>> items [('x', 1), ('y', 2)] """ def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/Persistence/tests/testPersistent.py0000664000076400007640000001434710435502403023674 0ustar ajungajung############################################################################# # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import pickle import time import unittest from Persistence import Persistent from persistent.cPickleCache import PickleCache from persistent.TimeStamp import TimeStamp from ZODB.utils import p64 class Jar(object): """Testing stub for _p_jar attribute.""" def __init__(self): self.cache = PickleCache(self) self.oid = 1 self.registered = {} def add(self, obj): obj._p_oid = p64(self.oid) self.oid += 1 obj._p_jar = self self.cache[obj._p_oid] = obj def close(self): pass # the following methods must be implemented to be a jar def setklassstate(self): # I don't know what this method does, but the pickle cache # constructor calls it. pass def register(self, obj): self.registered[obj] = 1 def setstate(self, obj): # Trivial setstate() implementation that just re-initializes # the object. This isn't what setstate() is supposed to do, # but it suffices for the tests. obj.__class__.__init__(obj) class P(Persistent): pass class H1(Persistent): def __init__(self): self.n = 0 def __getattr__(self, attr): self.n += 1 return self.n class H2(Persistent): def __init__(self): self.n = 0 def __getattribute__(self, attr): supergetattr = super(H2, self).__getattribute__ try: return supergetattr(attr) except AttributeError: n = supergetattr("n") self.n = n + 1 return n + 1 class PersistenceTest(unittest.TestCase): def setUp(self): self.jar = Jar() def tearDown(self): self.jar.close() def testOidAndJarAttrs(self): obj = P() self.assertEqual(obj._p_oid, None) obj._p_oid = 12 self.assertEqual(obj._p_oid, 12) del obj._p_oid self.jar.add(obj) # Can't change oid of cache object. def deloid(): del obj._p_oid self.assertRaises(ValueError, deloid) def setoid(): obj._p_oid = 12 self.assertRaises(ValueError, setoid) def deloid(): del obj._p_jar self.assertRaises(ValueError, deloid) def setoid(): obj._p_jar = 12 self.assertRaises(ValueError, setoid) def testChanged(self): obj = P() self.jar.add(obj) # The value returned for _p_changed can be one of: # 0 -- it is not changed # 1 -- it is changed # None -- it is a ghost obj.x = 1 self.assertEqual(obj._p_changed, 1) self.assert_(obj in self.jar.registered) obj._p_changed = 0 self.assertEqual(obj._p_changed, 0) self.jar.registered.clear() obj._p_changed = 1 self.assertEqual(obj._p_changed, 1) self.assert_(obj in self.jar.registered) # setting obj._p_changed to None ghostifies if the # object is in the up-to-date state, but not otherwise. obj._p_changed = None self.assertEqual(obj._p_changed, 1) obj._p_changed = 0 # Now it's a ghost. obj._p_changed = None self.assertEqual(obj._p_changed, None) obj = P() self.jar.add(obj) obj._p_changed = 1 # You can transition directly from modified to ghost if # you delete the _p_changed attribute. del obj._p_changed self.assertEqual(obj._p_changed, None) def testSerial(self): noserial = "\000" * 8 obj = P() self.assertEqual(obj._p_serial, noserial) def set(val): obj._p_serial = val self.assertRaises(ValueError, set, 1) self.assertRaises(ValueError, set, "0123") self.assertRaises(ValueError, set, "012345678") self.assertRaises(ValueError, set, u"01234567") obj._p_serial = "01234567" del obj._p_serial self.assertEqual(obj._p_serial, noserial) def testMTime(self): obj = P() self.assertEqual(obj._p_mtime, None) t = int(time.time()) ts = TimeStamp(*time.gmtime(t)[:6]) obj._p_serial = repr(ts) self.assertEqual(obj._p_mtime, t) self.assert_(isinstance(obj._p_mtime, float)) def testPicklable(self): obj = P() obj.attr = "test" s = pickle.dumps(obj) obj2 = pickle.loads(s) self.assertEqual(obj.attr, obj2.attr) def testGetattr(self): obj = H1() self.assertEqual(obj.larry, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) self.jar.add(obj) obj._p_deactivate() # The simple Jar used for testing re-initializes the object. self.assertEqual(obj.larry, 1) # The getattr hook modified the object, so it should now be # in the changed state. self.assertEqual(obj._p_changed, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) def testGetattribute(self): obj = H2() self.assertEqual(obj.larry, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) self.jar.add(obj) obj._p_deactivate() # The simple Jar used for testing re-initializes the object. self.assertEqual(obj.larry, 1) # The getattr hook modified the object, so it should now be # in the changed state. self.assertEqual(obj._p_changed, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) # TODO: Need to decide how __setattr__ and __delattr__ should work, # then write tests. def test_suite(): return unittest.makeSuite(PersistenceTest) Zope-2.8.7-final/lib/python/Persistence/DEPENDENCIES.cfg0000664000076400007640000000002010435502403021406 0ustar ajungajungZODB persistent Zope-2.8.7-final/lib/python/ZODB/0000775000076400007640000000000010435610645015371 5ustar ajungajungZope-2.8.7-final/lib/python/ZODB/SETUP.cfg0000664000076400007640000000012710435502423016744 0ustar ajungajung# Extension information for zpkg. source winlock.c Zope-2.8.7-final/lib/python/ZODB/DB.py0000664000076400007640000007012410435502423016226 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Database objects $Id: DB.py 30534 2005-05-27 18:07:18Z tim_one $""" import cPickle, cStringIO, sys import threading from time import time, ctime import logging from ZODB.broken import find_global from ZODB.utils import z64 from ZODB.Connection import Connection from ZODB.serialize import referencesf from ZODB.utils import WeakSet from ZODB.utils import DEPRECATED_ARGUMENT, deprecated36 from zope.interface import implements from ZODB.interfaces import IDatabase import transaction logger = logging.getLogger('ZODB.DB') class _ConnectionPool(object): """Manage a pool of connections. CAUTION: Methods should be called under the protection of a lock. This class does no locking of its own. There's no limit on the number of connections this can keep track of, but a warning is logged if there are more than pool_size active connections, and a critical problem if more than twice pool_size. New connections are registered via push(). This will log a message if "too many" connections are active. When a connection is explicitly closed, tell the pool via repush(). That adds the connection to a stack of connections available for reuse, and throws away the oldest stack entries if the stack is too large. pop() pops this stack. When a connection is obtained via pop(), the pool holds only a weak reference to it thereafter. It's not necessary to inform the pool if the connection goes away. A connection handed out by pop() counts against pool_size only so long as it exists, and provided it isn't repush()'ed. A weak reference is retained so that DB methods like connectionDebugInfo() can still gather statistics. """ def __init__(self, pool_size): # The largest # of connections we expect to see alive simultaneously. self.pool_size = pool_size # A weak set of all connections we've seen. A connection vanishes # from this set if pop() hands it out, it's not reregistered via # repush(), and it becomes unreachable. self.all = WeakSet() # A stack of connections available to hand out. This is a subset # of self.all. push() and repush() add to this, and may remove # the oldest available connections if the pool is too large. # pop() pops this stack. There are never more than pool_size entries # in this stack. # In Python 2.4, a collections.deque would make more sense than # a list (we push only "on the right", but may pop from both ends). self.available = [] # Change our belief about the expected maximum # of live connections. # If the pool_size is smaller than the current value, this may discard # the oldest available connections. def set_pool_size(self, pool_size): self.pool_size = pool_size self._reduce_size() # Register a new available connection. We must not know about c already. # c will be pushed onto the available stack even if we're over the # pool size limit. def push(self, c): assert c not in self.all assert c not in self.available self._reduce_size(strictly_less=True) self.all.add(c) self.available.append(c) n, limit = len(self.all), self.pool_size if n > limit: reporter = logger.warn if n > 2 * limit: reporter = logger.critical reporter("DB.open() has %s open connections with a pool_size " "of %s", n, limit) # Reregister an available connection formerly obtained via pop(). This # pushes it on the stack of available connections, and may discard # older available connections. def repush(self, c): assert c in self.all assert c not in self.available self._reduce_size(strictly_less=True) self.available.append(c) # Throw away the oldest available connections until we're under our # target size (strictly_less=False) or no more than that (strictly_less= # True, the default). def _reduce_size(self, strictly_less=False): target = self.pool_size - bool(strictly_less) while len(self.available) > target: c = self.available.pop(0) self.all.remove(c) # Pop an available connection and return it, or return None if none are # available. In the latter case, the caller should create a new # connection, register it via push(), and call pop() again. The # caller is responsible for serializing this sequence. def pop(self): result = None if self.available: result = self.available.pop() # Leave it in self.all, so we can still get at it for statistics # while it's alive. assert result in self.all return result # For every live connection c, invoke f(c). def map(self, f): self.all.map(f) class DB(object): """The Object Database ------------------- The DB class coordinates the activities of multiple database Connection instances. Most of the work is done by the Connections created via the open method. The DB instance manages a pool of connections. If a connection is closed, it is returned to the pool and its object cache is preserved. A subsequent call to open() will reuse the connection. There is no hard limit on the pool size. If more than `pool_size` connections are opened, a warning is logged, and if more than twice that many, a critical problem is logged. The class variable 'klass' is used by open() to create database connections. It is set to Connection, but a subclass could override it to provide a different connection implementation. The database provides a few methods intended for application code -- open, close, undo, and pack -- and a large collection of methods for inspecting the database and its connections' caches. :Cvariables: - `klass`: Class used by L{open} to create database connections :Groups: - `User Methods`: __init__, open, close, undo, pack, classFactory - `Inspection Methods`: getName, getSize, objectCount, getActivityMonitor, setActivityMonitor - `Connection Pool Methods`: getPoolSize, getVersionPoolSize, removeVersionPool, setPoolSize, setVersionPoolSize - `Transaction Methods`: invalidate - `Other Methods`: lastTransaction, connectionDebugInfo - `Version Methods`: modifiedInVersion, abortVersion, commitVersion, versionEmpty - `Cache Inspection Methods`: cacheDetail, cacheExtremeDetail, cacheFullSweep, cacheLastGCTime, cacheMinimize, cacheSize, cacheDetailSize, getCacheSize, getVersionCacheSize, setCacheSize, setVersionCacheSize - `Deprecated Methods`: getCacheDeactivateAfter, setCacheDeactivateAfter, getVersionCacheDeactivateAfter, setVersionCacheDeactivateAfter """ implements(IDatabase) klass = Connection # Class to use for connections _activity_monitor = None def __init__(self, storage, pool_size=7, cache_size=400, cache_deactivate_after=DEPRECATED_ARGUMENT, version_pool_size=3, version_cache_size=100, database_name='unnamed', databases=None, version_cache_deactivate_after=DEPRECATED_ARGUMENT, ): """Create an object database. :Parameters: - `storage`: the storage used by the database, e.g. FileStorage - `pool_size`: expected maximum number of open connections - `cache_size`: target size of Connection object cache - `version_pool_size`: expected maximum number of connections (per version) - `version_cache_size`: target size of Connection object cache for version connections - `cache_deactivate_after`: ignored - `version_cache_deactivate_after`: ignored """ # Allocate lock. x = threading.RLock() self._a = x.acquire self._r = x.release # Setup connection pools and cache info # _pools maps a version string to a _ConnectionPool object. self._pools = {} self._pool_size = pool_size self._cache_size = cache_size self._version_pool_size = version_pool_size self._version_cache_size = version_cache_size # warn about use of deprecated arguments if cache_deactivate_after is not DEPRECATED_ARGUMENT: deprecated36("cache_deactivate_after has no effect") if version_cache_deactivate_after is not DEPRECATED_ARGUMENT: deprecated36("version_cache_deactivate_after has no effect") self._miv_cache = {} # Setup storage self._storage=storage storage.registerDB(self, None) if not hasattr(storage,'tpc_vote'): storage.tpc_vote = lambda *args: None try: storage.load(z64,'') except KeyError: # Create the database's root in the storage if it doesn't exist from persistent.mapping import PersistentMapping root = PersistentMapping() # Manually create a pickle for the root to put in the storage. # The pickle must be in the special ZODB format. file = cStringIO.StringIO() p = cPickle.Pickler(file, 1) p.dump((root.__class__, None)) p.dump(root.__getstate__()) t = transaction.Transaction() t.description = 'initial database creation' storage.tpc_begin(t) storage.store(z64, None, file.getvalue(), '', t) storage.tpc_vote(t) storage.tpc_finish(t) # Multi-database setup. if databases is None: databases = {} self.databases = databases self.database_name = database_name if database_name in databases: raise ValueError("database_name %r already in databases" % database_name) databases[database_name] = self # Pass through methods: for m in ['history', 'supportsUndo', 'supportsVersions', 'undoLog', 'versionEmpty', 'versions']: setattr(self, m, getattr(storage, m)) if hasattr(storage, 'undoInfo'): self.undoInfo = storage.undoInfo # This is called by Connection.close(). def _closeConnection(self, connection): """Return a connection to the pool. connection._db must be self on entry. """ self._a() try: assert connection._db is self connection._db = None am = self._activity_monitor if am is not None: am.closedConnection(connection) version = connection._version try: pool = self._pools[version] except KeyError: # No such version. We must have deleted the pool. # Just let the connection go. # We need to break circular refs to make it really go. # TODO: Figure out exactly which objects are involved in the # cycle. connection.__dict__.clear() return pool.repush(connection) finally: self._r() # Call f(c) for all connections c in all pools in all versions. def _connectionMap(self, f): self._a() try: for pool in self._pools.values(): pool.map(f) finally: self._r() def abortVersion(self, version, txn=None): if txn is None: txn = transaction.get() txn.register(AbortVersion(self, version)) def cacheDetail(self): """Return information on objects in the various caches Organized by class. """ detail = {} def f(con, detail=detail): for oid, ob in con._cache.items(): module = getattr(ob.__class__, '__module__', '') module = module and '%s.' % module or '' c = "%s%s" % (module, ob.__class__.__name__) if c in detail: detail[c] += 1 else: detail[c] = 1 self._connectionMap(f) detail = detail.items() detail.sort() return detail def cacheExtremeDetail(self): detail = [] conn_no = [0] # A mutable reference to a counter def f(con, detail=detail, rc=sys.getrefcount, conn_no=conn_no): conn_no[0] += 1 cn = conn_no[0] for oid, ob in con._cache_items(): id = '' if hasattr(ob, '__dict__'): d = ob.__dict__ if d.has_key('id'): id = d['id'] elif d.has_key('__name__'): id = d['__name__'] module = getattr(ob.__class__, '__module__', '') module = module and ('%s.' % module) or '' # What refcount ('rc') should we return? The intent is # that we return the true Python refcount, but as if the # cache didn't exist. This routine adds 3 to the true # refcount: 1 for binding to name 'ob', another because # ob lives in the con._cache_items() list we're iterating # over, and calling sys.getrefcount(ob) boosts ob's # count by 1 too. So the true refcount is 3 less than # sys.getrefcount(ob) returns. But, in addition to that, # the cache holds an extra reference on non-ghost objects, # and we also want to pretend that doesn't exist. detail.append({ 'conn_no': cn, 'oid': oid, 'id': id, 'klass': "%s%s" % (module, ob.__class__.__name__), 'rc': rc(ob) - 3 - (ob._p_changed is not None), 'state': ob._p_changed, #'references': con.references(oid), }) self._connectionMap(f) return detail def cacheFullSweep(self): self._connectionMap(lambda c: c._cache.full_sweep()) def cacheLastGCTime(self): m = [0] def f(con, m=m): t = con._cache.cache_last_gc_time if t > m[0]: m[0] = t self._connectionMap(f) return m[0] def cacheMinimize(self): self._connectionMap(lambda c: c._cache.minimize()) def cacheSize(self): m = [0] def f(con, m=m): m[0] += con._cache.cache_non_ghost_count self._connectionMap(f) return m[0] def cacheDetailSize(self): m = [] def f(con, m=m): m.append({'connection': repr(con), 'ngsize': con._cache.cache_non_ghost_count, 'size': len(con._cache)}) self._connectionMap(f) m.sort() return m def close(self): """Close the database and its underlying storage. It is important to close the database, because the storage may flush in-memory data structures to disk when it is closed. Leaving the storage open with the process exits can cause the next open to be slow. What effect does closing the database have on existing connections? Technically, they remain open, but their storage is closed, so they stop behaving usefully. Perhaps close() should also close all the Connections. """ self._storage.close() def commitVersion(self, source, destination='', txn=None): if txn is None: txn = transaction.get() txn.register(CommitVersion(self, source, destination)) def getCacheSize(self): return self._cache_size def lastTransaction(self): return self._storage.lastTransaction() def getName(self): return self._storage.getName() def getPoolSize(self): return self._pool_size def getSize(self): return self._storage.getSize() def getVersionCacheSize(self): return self._version_cache_size def getVersionPoolSize(self): return self._version_pool_size def invalidate(self, tid, oids, connection=None, version=''): """Invalidate references to a given oid. This is used to indicate that one of the connections has committed a change to the object. The connection commiting the change should be passed in to prevent useless (but harmless) messages to the connection. """ if connection is not None: version = connection._version # Update modified in version cache for oid in oids.keys(): h = hash(oid) % 131 o = self._miv_cache.get(h, None) if o is not None and o[0]==oid: del self._miv_cache[h] # Notify connections. def inval(c): if (c is not connection and (not version or c._version == version)): c.invalidate(tid, oids) self._connectionMap(inval) def modifiedInVersion(self, oid): h = hash(oid) % 131 cache = self._miv_cache o = cache.get(h, None) if o and o[0] == oid: return o[1] v = self._storage.modifiedInVersion(oid) cache[h] = oid, v return v def objectCount(self): return len(self._storage) def open(self, version='', transaction=DEPRECATED_ARGUMENT, temporary=DEPRECATED_ARGUMENT, force=DEPRECATED_ARGUMENT, waitflag=DEPRECATED_ARGUMENT, mvcc=True, txn_mgr=DEPRECATED_ARGUMENT, transaction_manager=None, synch=True): """Return a database Connection for use by application code. The optional `version` argument can be used to specify that a version connection is desired. Note that the connection pool is managed as a stack, to increase the likelihood that the connection's stack will include useful objects. :Parameters: - `version`: the "version" that all changes will be made in, defaults to no version. - `mvcc`: boolean indicating whether MVCC is enabled - `transaction_manager`: transaction manager to use. None means use the default transaction manager. - `synch`: boolean indicating whether Connection should register for afterCompletion() calls. """ if temporary is not DEPRECATED_ARGUMENT: deprecated36("DB.open() temporary= ignored. " "open() no longer blocks.") if force is not DEPRECATED_ARGUMENT: deprecated36("DB.open() force= ignored. " "open() no longer blocks.") if waitflag is not DEPRECATED_ARGUMENT: deprecated36("DB.open() waitflag= ignored. " "open() no longer blocks.") if transaction is not DEPRECATED_ARGUMENT: deprecated36("DB.open() transaction= ignored.") if txn_mgr is not DEPRECATED_ARGUMENT: deprecated36("use transaction_manager= instead of txn_mgr=") if transaction_manager is None: transaction_manager = txn_mgr else: raise ValueError("cannot specify both transaction_manager= " "and txn_mgr=") self._a() try: # pool <- the _ConnectionPool for this version pool = self._pools.get(version) if pool is None: if version: size = self._version_pool_size else: size = self._pool_size self._pools[version] = pool = _ConnectionPool(size) assert pool is not None # result <- a connection result = pool.pop() if result is None: if version: size = self._version_cache_size else: size = self._cache_size c = self.klass(version=version, cache_size=size, mvcc=mvcc, transaction_manager=transaction_manager) pool.push(c) result = pool.pop() assert result is not None # Tell the connection it belongs to self. result._setDB(self, mvcc=mvcc, synch=synch, transaction_manager=transaction_manager) # A good time to do some cache cleanup. self._connectionMap(lambda c: c.cacheGC()) return result finally: self._r() def removeVersionPool(self, version): try: del self._pools[version] except KeyError: pass def connectionDebugInfo(self): result = [] t = time() def get_info(c): # `result`, `time` and `version` are lexically inherited. o = c._opened d = c.getDebugInfo() if d: if len(d) == 1: d = d[0] else: d = '' d = "%s (%s)" % (d, len(c._cache)) result.append({ 'opened': o and ("%s (%.2fs)" % (ctime(o), t-o)), 'info': d, 'version': version, }) for version, pool in self._pools.items(): pool.map(get_info) return result def getActivityMonitor(self): return self._activity_monitor def pack(self, t=None, days=0): """Pack the storage, deleting unused object revisions. A pack is always performed relative to a particular time, by default the current time. All object revisions that are not reachable as of the pack time are deleted from the storage. The cost of this operation varies by storage, but it is usually an expensive operation. There are two optional arguments that can be used to set the pack time: t, pack time in seconds since the epcoh, and days, the number of days to subtract from t or from the current time if t is not specified. """ if t is None: t = time() t -= days * 86400 try: self._storage.pack(t, referencesf) except: logger.error("packing", exc_info=True) raise def setActivityMonitor(self, am): self._activity_monitor = am def classFactory(self, connection, modulename, globalname): # Zope will rebind this method to arbitrary user code at runtime. return find_global(modulename, globalname) def setCacheSize(self, size): self._a() try: self._cache_size = size pool = self._pools.get('') if pool is not None: def setsize(c): c._cache.cache_size = size pool.map(setsize) finally: self._r() def setVersionCacheSize(self, size): self._a() try: self._version_cache_size = size def setsize(c): c._cache.cache_size = size for version, pool in self._pools.items(): if version: pool.map(setsize) finally: self._r() def setPoolSize(self, size): self._pool_size = size self._reset_pool_sizes(size, for_versions=False) def setVersionPoolSize(self, size): self._version_pool_size = size self._reset_pool_sizes(size, for_versions=True) def _reset_pool_sizes(self, size, for_versions=False): self._a() try: for version, pool in self._pools.items(): if (version != '') == for_versions: pool.set_pool_size(size) finally: self._r() def undo(self, id, txn=None): """Undo a transaction identified by id. A transaction can be undone if all of the objects involved in the transaction were not modified subsequently, if any modifications can be resolved by conflict resolution, or if subsequent changes resulted in the same object state. The value of id should be generated by calling undoLog() or undoInfo(). The value of id is not the same as a transaction id used by other methods; it is unique to undo(). :Parameters: - `id`: a storage-specific transaction identifier - `txn`: transaction context to use for undo(). By default, uses the current transaction. """ if txn is None: txn = transaction.get() txn.register(TransactionalUndo(self, id)) def versionEmpty(self, version): return self._storage.versionEmpty(version) # The following methods are deprecated and have no effect def getCacheDeactivateAfter(self): """Deprecated""" deprecated36("getCacheDeactivateAfter has no effect") def getVersionCacheDeactivateAfter(self): """Deprecated""" deprecated36("getVersionCacheDeactivateAfter has no effect") def setCacheDeactivateAfter(self, v): """Deprecated""" deprecated36("setCacheDeactivateAfter has no effect") def setVersionCacheDeactivateAfter(self, v): """Deprecated""" deprecated36("setVersionCacheDeactivateAfter has no effect") class ResourceManager(object): """Transaction participation for a version or undo resource.""" def __init__(self, db): self._db = db # Delegate the actual 2PC methods to the storage self.tpc_vote = self._db._storage.tpc_vote self.tpc_finish = self._db._storage.tpc_finish self.tpc_abort = self._db._storage.tpc_abort def sortKey(self): return "%s:%s" % (self._db._storage.sortKey(), id(self)) def tpc_begin(self, txn, sub=False): if sub: raise ValueError("doesn't support sub-transactions") self._db._storage.tpc_begin(txn) # The object registers itself with the txn manager, so the ob # argument to the methods below is self. def abort(self, obj, txn): pass def commit(self, obj, txn): pass class CommitVersion(ResourceManager): def __init__(self, db, version, dest=''): super(CommitVersion, self).__init__(db) self._version = version self._dest = dest def commit(self, ob, t): dest = self._dest tid, oids = self._db._storage.commitVersion(self._version, self._dest, t) oids = dict.fromkeys(oids, 1) self._db.invalidate(tid, oids, version=self._dest) if self._dest: # the code above just invalidated the dest version. # now we need to invalidate the source! self._db.invalidate(tid, oids, version=self._version) class AbortVersion(ResourceManager): def __init__(self, db, version): super(AbortVersion, self).__init__(db) self._version = version def commit(self, ob, t): tid, oids = self._db._storage.abortVersion(self._version, t) self._db.invalidate(tid, dict.fromkeys(oids, 1), version=self._version) class TransactionalUndo(ResourceManager): def __init__(self, db, tid): super(TransactionalUndo, self).__init__(db) self._tid = tid def commit(self, ob, t): tid, oids = self._db._storage.undo(self._tid, t) self._db.invalidate(tid, dict.fromkeys(oids, 1)) Zope-2.8.7-final/lib/python/ZODB/broken.py0000664000076400007640000002264210435502423017223 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Broken object support $Id: broken.py 29446 2005-03-11 22:07:51Z tim_one $ """ import sys import persistent broken_cache = {} class Broken(object): """Broken object base class Broken objects are placeholders for objects that can no longer be created because their class has gone away. Broken objects don't really do much of anything, except hold their state. The Broken class is used as a base class for creating classes in leu of missing classes:: >>> Atall = type('Atall', (Broken, ), {'__module__': 'not.there'}) The only thing the class can be used for is to create new objects:: >>> Atall() >>> Atall().__Broken_newargs__ () >>> Atall().__Broken_initargs__ () >>> Atall(1, 2).__Broken_newargs__ (1, 2) >>> Atall(1, 2).__Broken_initargs__ (1, 2) >>> a = Atall.__new__(Atall, 1, 2) >>> a >>> a.__Broken_newargs__ (1, 2) >>> a.__Broken_initargs__ You can't modify broken objects:: >>> a.x = 1 Traceback (most recent call last): ... BrokenModified: Can't change broken objects But you can set their state:: >>> a.__setstate__({'x': 1, }) You can pickle broken objects:: >>> r = a.__reduce__() >>> len(r) 3 >>> r[0] is rebuild True >>> r[1] ('not.there', 'Atall', 1, 2) >>> r[2] {'x': 1} >>> import cPickle >>> a2 = cPickle.loads(cPickle.dumps(a, 1)) >>> a2 >>> a2.__Broken_newargs__ (1, 2) >>> a2.__Broken_initargs__ >>> a2.__Broken_state__ {'x': 1} Cleanup:: >>> broken_cache.clear() """ __Broken_state__ = __Broken_initargs__ = None __name__ = 'broken object' def __new__(class_, *args): result = object.__new__(class_) result.__dict__['__Broken_newargs__'] = args return result def __init__(self, *args): self.__dict__['__Broken_initargs__'] = args def __reduce__(self): """We pickle broken objects in hope of being able to fix them later """ return (rebuild, ((self.__class__.__module__, self.__class__.__name__) + self.__Broken_newargs__), self.__Broken_state__, ) def __setstate__(self, state): self.__dict__['__Broken_state__'] = state def __repr__(self): return "" % ( self.__class__.__module__, self.__class__.__name__) def __setattr__(self, name, value): raise BrokenModified("Can't change broken objects") def find_global(modulename, globalname, # These are *not* optimizations. Callers can override these. Broken=Broken, type=type, ): """Find a global object, returning a broken class if it can't be found. This function looks up global variable in modules:: >>> import sys >>> find_global('sys', 'path') is sys.path True If an object can't be found, a broken class is returned:: >>> broken = find_global('ZODB.not.there', 'atall') >>> issubclass(broken, Broken) True >>> broken.__module__ 'ZODB.not.there' >>> broken.__name__ 'atall' Broken classes are cached:: >>> find_global('ZODB.not.there', 'atall') is broken True If we "repair" a missing global:: >>> class ZODBnotthere: ... atall = [] >>> sys.modules['ZODB.not'] = ZODBnotthere >>> sys.modules['ZODB.not.there'] = ZODBnotthere we can then get the repaired value:: >>> find_global('ZODB.not.there', 'atall') is ZODBnotthere.atall True Of course, if we beak it again:: >>> del sys.modules['ZODB.not'] >>> del sys.modules['ZODB.not.there'] we get the broken value:: >>> find_global('ZODB.not.there', 'atall') is broken True Cleanup:: >>> broken_cache.clear() """ try: __import__(modulename) except ImportError: pass else: module = sys.modules[modulename] try: return getattr(module, globalname) except AttributeError: pass try: return broken_cache[(modulename, globalname)] except KeyError: pass class_ = type(globalname, (Broken, ), {'__module__': modulename}) broken_cache[(modulename, globalname)] = class_ return class_ def rebuild(modulename, globalname, *args): """Recreate a broken object, possibly recreating the missing class This functions unpickles broken objects:: >>> broken = rebuild('ZODB.notthere', 'atall', 1, 2) >>> broken >>> broken.__Broken_newargs__ (1, 2) If we "repair" the brokenness:: >>> class notthere: # fake notthere module ... class atall(object): ... def __new__(self, *args): ... ob = object.__new__(self) ... ob.args = args ... return ob ... def __repr__(self): ... return 'atall %s %s' % self.args >>> sys.modules['ZODB.notthere'] = notthere >>> rebuild('ZODB.notthere', 'atall', 1, 2) atall 1 2 >>> del sys.modules['ZODB.notthere'] Cleanup:: >>> broken_cache.clear() """ class_ = find_global(modulename, globalname) return class_.__new__(class_, *args) class BrokenModified(TypeError): """Attempt to modify a broken object """ class PersistentBroken(Broken, persistent.Persistent): r"""Persistent broken objects Persistent broken objects are used for broken objects that are also persistent. In addition to having to track the original object data, they need to handle persistent meta data. Persistent broken classes are created from existing broken classes using the persistentBroken, function:: >>> Atall = type('Atall', (Broken, ), {'__module__': 'not.there'}) >>> PAtall = persistentBroken(Atall) (Note that we always get the *same* persistent broken class for a given broken class:: >>> persistentBroken(Atall) is PAtall True ) Persistent broken classes work a lot like broken classes:: >>> a = PAtall.__new__(PAtall, 1, 2) >>> a >>> a.__Broken_newargs__ (1, 2) >>> a.__Broken_initargs__ >>> a.x = 1 Traceback (most recent call last): ... BrokenModified: Can't change broken objects Unlike regular broken objects, persistent broken objects keep track of persistence meta data: >>> a._p_oid = '\0\0\0\0****' >>> a and persistent broken objects aren't directly picklable: >>> a.__reduce__() Traceback (most recent call last): ... BrokenModified: """ \ r""" but you can get their state: >>> a.__setstate__({'y': 2}) >>> a.__getstate__() {'y': 2} Cleanup:: >>> broken_cache.clear() """ def __new__(class_, *args): result = persistent.Persistent.__new__(class_) result.__dict__['__Broken_newargs__'] = args return result def __reduce__(self, *args): raise BrokenModified(self) def __getstate__(self): return self.__Broken_state__ def __setattr__(self, name, value): if name.startswith('_p_'): persistent.Persistent.__setattr__(self, name, value) else: raise BrokenModified("Can't change broken objects") def __repr__(self): return "" % ( self.__class__.__module__, self.__class__.__name__, self._p_oid) def __getnewargs__(self): return self.__Broken_newargs__ def persistentBroken(class_): try: return class_.__dict__['__Broken_Persistent__'] except KeyError: class_.__Broken_Persistent__ = ( type(class_.__name__, (PersistentBroken, class_), {'__module__': class_.__module__}, ) ) return class_.__dict__['__Broken_Persistent__'] Zope-2.8.7-final/lib/python/ZODB/transact.py0000664000076400007640000000371010435502423017555 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tools to simplify transactions within applications.""" from ZODB.POSException import ReadConflictError, ConflictError import transaction def _commit(note): t = transaction.get() if note: t.note(note) t.commit() def transact(f, note=None, retries=5): """Returns transactional version of function argument f. Higher-order function that converts a regular function into a transactional function. The transactional function will retry up to retries time before giving up. If note, it will be added to the transaction metadata when it commits. The retries occur on ConflictErrors. If some other TransactionError occurs, the transaction will not be retried. """ # TODO: deal with ZEO disconnected errors? def g(*args, **kwargs): n = retries while n: n -= 1 try: r = f(*args, **kwargs) except ReadConflictError, msg: transaction.abort() if not n: raise continue try: _commit(note) except ConflictError, msg: transaction.abort() if not n: raise continue return r raise RuntimeError, "couldn't commit transaction" return g Zope-2.8.7-final/lib/python/ZODB/Mount.py0000664000076400007640000002457510435502423017054 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Mounted database support $Id: Mount.py 28198 2004-10-15 12:52:23Z hdima $""" import time import thread import logging import persistent import Acquisition from Acquisition import aq_base from POSException import MountedStorageError logger = logging.getLogger('ZODB.Mount') # dbs is a holder for all DB objects, needed to overcome # threading issues. It maps connection params to a DB object # and a mapping of mount points. dbs = {} # dblock is locked every time dbs is accessed. dblock=thread.allocate_lock() def parentClassFactory(jar, module, name): # Use the class factory from the parent database. parent_conn = getattr(jar, '_mount_parent_jar', None) parent_db = getattr(parent_conn, '_db', None) if parent_db is None: _globals = {} _silly = ('__doc__',) return getattr(__import__( module, _globals, _globals, _silly), name) else: return parent_db.classFactory(parent_conn, module, name) class MountPoint(persistent.Persistent, Acquisition.Implicit): '''The base class for a Zope object which, when traversed, accesses a different database. ''' # Default values for non-persistent variables. _v_db = None _v_data = None _v_connect_error = None def __init__(self, path, params=None, classDefsFromRoot=1): ''' @arg path The path within the mounted database from which to derive the root. @arg params The parameters used to connect to the database. No particular format required. If there is more than one mount point referring to a database, MountPoint will detect the matching params and use the existing database. Include the class name of the storage. For example, ZEO params might be "ZODB.ZEOClient localhost 1081". @arg classDefsFromRoot If true (the default), MountPoint will try to get ZClass definitions from the root database rather than the mounted database. ''' # The only reason we need a __mountpoint_id is to # be sure we don't close a database prematurely when # it is mounted more than once and one of the points # is unmounted. self.__mountpoint_id = '%s_%f' % (id(self), time.time()) if params is None: # We still need something to use as a hash in # the "dbs" dictionary. params = self.__mountpoint_id self._params = repr(params) self._path = path self._classDefsFromRoot = classDefsFromRoot def _createDB(self): '''Gets the database object, usually by creating a Storage object and returning ZODB.DB(storage). ''' raise NotImplementedError def _getDB(self): '''Creates or opens a DB object. ''' newMount = 0 dblock.acquire() try: params = self._params dbInfo = dbs.get(params, None) if dbInfo is None: logger.info('Opening database for mounting: %s', params) db = self._createDB() newMount = 1 dbs[params] = (db, {self.__mountpoint_id:1}) if getattr(self, '_classDefsFromRoot', 1): db.classFactory = parentClassFactory else: db, mounts = dbInfo # Be sure this object is in the list of mount points. if not mounts.has_key(self.__mountpoint_id): newMount = 1 mounts[self.__mountpoint_id] = 1 self._v_db = db finally: dblock.release() return db, newMount def _getMountpointId(self): return self.__mountpoint_id def _getMountParams(self): return self._params def __repr__(self): return "%s(%s, %s)" % (self.__class__.__name__, repr(self._path), self._params) def _openMountableConnection(self, parent): # Opens a new connection to the database. db = self._v_db if db is None: self._v_close_db = 0 db, newMount = self._getDB() else: newMount = 0 jar = getattr(self, '_p_jar', None) if jar is None: # Get _p_jar from parent. self._p_jar = jar = parent._p_jar conn = db.open(version=jar.getVersion()) # Add an attribute to the connection which # makes it possible for us to find the primary # database connection. See ClassFactoryForMount(). conn._mount_parent_jar = jar mcc = MountedConnectionCloser(self, conn) jar.onCloseCallback(mcc) return conn, newMount, mcc def _getObjectFromConnection(self, conn): obj = self._getMountRoot(conn.root()) data = aq_base(obj) # Store the data object in a tuple to hide from acquisition. self._v_data = (data,) return data def _getOrOpenObject(self, parent): t = self._v_data if t is None: self._v_connect_error = None conn = None newMount = 0 mcc = None try: conn, newMount, mcc = self._openMountableConnection(parent) data = self._getObjectFromConnection(conn) except: # Possibly broken database. if mcc is not None: # Note that the next line may be a little rash-- # if, for example, a working database throws an # exception rather than wait for a new connection, # this will likely cause the database to be closed # prematurely. Perhaps DB.py needs a # countActiveConnections() method. mcc.setCloseDb() self._logConnectException() raise if newMount: try: id = data.getId() except: id = '???' # data has no getId() method. Bad. p = '/'.join(parent.getPhysicalPath() + (id,)) logger.info('Mounted database %s at %s', self._getMountParams(), p) else: data = t[0] return data.__of__(parent) def __of__(self, parent): # Accesses the database, returning an acquisition # wrapper around the connected object rather than around self. try: return self._getOrOpenObject(parent) except: return Acquisition.ImplicitAcquisitionWrapper( self, parent) def _test(self, parent): '''Tests the database connection. ''' self._getOrOpenObject(parent) return 1 def _getMountRoot(self, root): '''Gets the object to be mounted. Can be overridden to provide different behavior. ''' try: app = root['Application'] except: raise MountedStorageError, ( "No 'Application' object exists in the mountable database.") try: return app.unrestrictedTraverse(self._path) except: raise MountedStorageError, ( "The path '%s' was not found in the mountable database." % self._path) def _logConnectException(self): '''Records info about the exception that just occurred. ''' try: from cStringIO import StringIO except: from StringIO import StringIO import traceback logger.warning('Failed to mount database. %s (%s)', exc[:2], exc_info=True) f=StringIO() traceback.print_tb(exc[2], 100, f) self._v_connect_error = (exc[0], exc[1], f.getvalue()) exc = None class MountedConnectionCloser: '''Closes the connection used by the mounted database while performing other cleanup. ''' close_db = 0 def __init__(self, mountpoint, conn): # conn is the child connection. self.mp = mountpoint self.conn = conn def setCloseDb(self): self.close_db = 1 def __call__(self): # The onCloseCallback handler. # Closes a single connection to the database # and possibly the database itself. conn = self.conn close_db = 0 if conn is not None: mp = self.mp # Remove potential circular references. self.conn = None self.mp = None # Detect whether we should close the database. close_db = self.close_db t = mp.__dict__.get('_v_data', None) if t is not None: del mp.__dict__['_v_data'] data = t[0] if not close_db and data.__dict__.get( '_v__object_deleted__', 0): # This mount point has been deleted. del data.__dict__['_v__object_deleted__'] close_db = 1 # Close the child connection. try: del conn._mount_parent_jar except: pass conn.close() if close_db: # Stop using this database. Close it if no other # MountPoint is using it. dblock.acquire() try: params = mp._getMountParams() mp._v_db = None if dbs.has_key(params): dbInfo = dbs[params] db, mounts = dbInfo try: del mounts[mp._getMountpointId()] except: pass if len(mounts) < 1: # No more mount points are using this database. del dbs[params] db.close() logger.info('Closed database: %s', params) finally: dblock.release() Zope-2.8.7-final/lib/python/ZODB/config.py0000664000076400007640000001240510435502423017204 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Open database and storage from a configuration. $Id: config.py 30437 2005-05-20 00:57:18Z tim_one $""" import os from cStringIO import StringIO import ZConfig import ZODB db_schema_path = os.path.join(ZODB.__path__[0], "config.xml") _db_schema = None s_schema_path = os.path.join(ZODB.__path__[0], "storage.xml") _s_schema = None def getDbSchema(): global _db_schema if _db_schema is None: _db_schema = ZConfig.loadSchema(db_schema_path) return _db_schema def getStorageSchema(): global _s_schema if _s_schema is None: _s_schema = ZConfig.loadSchema(s_schema_path) return _s_schema def databaseFromString(s): return databaseFromFile(StringIO(s)) def databaseFromFile(f): config, handle = ZConfig.loadConfigFile(getDbSchema(), f) return databaseFromConfig(config.database) def databaseFromURL(url): config, handler = ZConfig.loadConfig(getDbSchema(), url) return databaseFromConfig(config.database) def databaseFromConfig(section): return section.open() def storageFromString(s): return storageFromFile(StringIO(s)) def storageFromFile(f): config, handle = ZConfig.loadConfigFile(getStorageSchema(), f) return storageFromConfig(config.storage) def storageFromURL(url): config, handler = ZConfig.loadConfig(getStorageSchema(), url) return storageFromConfig(config.storage) def storageFromConfig(section): return section.open() class BaseConfig: """Object representing a configured storage or database. Methods: open() -- open and return the configured object Attributes: name -- name of the storage """ def __init__(self, config): self.config = config self.name = config.getSectionName() def open(self): """Open and return the storage object.""" raise NotImplementedError class ZODBDatabase(BaseConfig): def open(self): section = self.config storage = section.storage.open() try: return ZODB.DB(storage, pool_size=section.pool_size, cache_size=section.cache_size, version_pool_size=section.version_pool_size, version_cache_size=section.version_cache_size) except: storage.close() raise class MappingStorage(BaseConfig): def open(self): from ZODB.MappingStorage import MappingStorage return MappingStorage(self.config.name) class DemoStorage(BaseConfig): def open(self): from ZODB.DemoStorage import DemoStorage if self.config.base: base = self.config.base.open() else: base = None return DemoStorage(self.config.name, base=base, quota=self.config.quota) class FileStorage(BaseConfig): def open(self): from ZODB.FileStorage import FileStorage return FileStorage(self.config.path, create=self.config.create, read_only=self.config.read_only, quota=self.config.quota) class ZEOClient(BaseConfig): def open(self): from ZEO.ClientStorage import ClientStorage # config.server is a multikey of socket-connection-address values # where the value is a socket family, address tuple. L = [server.address for server in self.config.server] return ClientStorage( L, storage=self.config.storage, cache_size=self.config.cache_size, name=self.config.name, client=self.config.client, var=self.config.var, min_disconnect_poll=self.config.min_disconnect_poll, max_disconnect_poll=self.config.max_disconnect_poll, wait=self.config.wait, read_only=self.config.read_only, read_only_fallback=self.config.read_only_fallback) class BDBStorage(BaseConfig): def open(self): from BDBStorage.BerkeleyBase import BerkeleyConfig storageclass = self.get_storageclass() bconf = BerkeleyConfig() for name in dir(BerkeleyConfig): if name.startswith('_'): continue setattr(bconf, name, getattr(self.config, name)) return storageclass(self.config.envdir, config=bconf) class BDBMinimalStorage(BDBStorage): def get_storageclass(self): import BDBStorage.BDBMinimalStorage return BDBStorage.BDBMinimalStorage.BDBMinimalStorage class BDBFullStorage(BDBStorage): def get_storageclass(self): import BDBStorage.BDBFullStorage return BDBStorage.BDBFullStorage.BDBFullStorage Zope-2.8.7-final/lib/python/ZODB/FileStorage/0000775000076400007640000000000010435610645017575 5ustar ajungajungZope-2.8.7-final/lib/python/ZODB/FileStorage/FileStorage.py0000664000076400007640000021711010435502423022347 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Storage implementation using a log written to a single file. $Revision: 1.16 $ """ import base64 from cPickle import Pickler, Unpickler, loads import errno import os import sys import time import logging from types import StringType from struct import pack, unpack # Not all platforms have fsync fsync = getattr(os, "fsync", None) from ZODB import BaseStorage, ConflictResolution, POSException from ZODB.POSException \ import UndoError, POSKeyError, MultipleUndoErrors, VersionLockError from persistent.TimeStamp import TimeStamp from ZODB.lock_file import LockFile from ZODB.utils import p64, u64, cp, z64 from ZODB.FileStorage.fspack import FileStoragePacker from ZODB.FileStorage.format \ import FileStorageFormatter, DataHeader, TxnHeader, DATA_HDR, \ DATA_HDR_LEN, TRANS_HDR, TRANS_HDR_LEN, CorruptedDataError from ZODB.loglevels import BLATHER from ZODB.fsIndex import fsIndex packed_version = "FS21" logger = logging.getLogger('ZODB.FileStorage') def panic(message, *data): logger.critical(message, *data) raise CorruptedTransactionError(message) class FileStorageError(POSException.StorageError): pass class PackError(FileStorageError): pass class FileStorageFormatError(FileStorageError): """Invalid file format The format of the given file is not valid. """ class CorruptedFileStorageError(FileStorageError, POSException.StorageSystemError): """Corrupted file storage.""" class CorruptedTransactionError(CorruptedFileStorageError): pass class FileStorageQuotaError(FileStorageError, POSException.StorageSystemError): """File storage quota exceeded.""" # Intended to be raised only in fspack.py, and ignored here. class RedundantPackWarning(FileStorageError): pass class TempFormatter(FileStorageFormatter): """Helper class used to read formatted FileStorage data.""" def __init__(self, afile): self._file = afile class FileStorage(BaseStorage.BaseStorage, ConflictResolution.ConflictResolvingStorage, FileStorageFormatter): # Set True while a pack is in progress; undo is blocked for the duration. _pack_is_in_progress = False _records_before_save = 10000 def __init__(self, file_name, create=False, read_only=False, stop=None, quota=None): if read_only: self._is_read_only = True if create: raise ValueError("can't create a read-only file") elif stop is not None: raise ValueError("time-travel only supported in read-only mode") if stop is None: stop='\377'*8 # Lock the database and set up the temp file. if not read_only: # Create the lock file self._lock_file = LockFile(file_name + '.lock') self._tfile = open(file_name + '.tmp', 'w+b') self._tfmt = TempFormatter(self._tfile) else: self._tfile = None self._file_name = file_name BaseStorage.BaseStorage.__init__(self, file_name) (index, vindex, tindex, tvindex, oid2tid, toid2tid, toid2tid_delete) = self._newIndexes() self._initIndex(index, vindex, tindex, tvindex, oid2tid, toid2tid, toid2tid_delete) # Now open the file self._file = None if not create: try: self._file = open(file_name, read_only and 'rb' or 'r+b') except IOError, exc: if exc.errno == errno.EFBIG: # The file is too big to open. Fail visibly. raise if exc.errno == errno.ENOENT: # The file doesn't exist. Create it. create = 1 # If something else went wrong, it's hard to guess # what the problem was. If the file does not exist, # create it. Otherwise, fail. if os.path.exists(file_name): raise else: create = 1 if self._file is None and create: if os.path.exists(file_name): os.remove(file_name) self._file = open(file_name, 'w+b') self._file.write(packed_version) r = self._restore_index() if r is not None: self._used_index = 1 # Marker for testing index, vindex, start, ltid = r self._initIndex(index, vindex, tindex, tvindex, oid2tid, toid2tid, toid2tid_delete) self._pos, self._oid, tid = read_index( self._file, file_name, index, vindex, tindex, stop, ltid=ltid, start=start, read_only=read_only, ) else: self._used_index = 0 # Marker for testing self._pos, self._oid, tid = read_index( self._file, file_name, index, vindex, tindex, stop, read_only=read_only, ) self._save_index() self._records_before_save = max(self._records_before_save, len(self._index)) self._ltid = tid # self._pos should always point just past the last # transaction. During 2PC, data is written after _pos. # invariant is restored at tpc_abort() or tpc_finish(). self._ts = tid = TimeStamp(tid) t = time.time() t = TimeStamp(*time.gmtime(t)[:5] + (t % 60,)) if tid > t: seconds = tid.timeTime() - t.timeTime() complainer = logger.warning if seconds > 30 * 60: # 30 minutes -- way screwed up complainer = logger.critical complainer("%s Database records %d seconds in the future", file_name, seconds) self._quota = quota # tid cache statistics. self._oid2tid_nlookups = self._oid2tid_nhits = 0 def _initIndex(self, index, vindex, tindex, tvindex, oid2tid, toid2tid, toid2tid_delete): self._index=index self._vindex=vindex self._tindex=tindex self._tvindex=tvindex self._index_get=index.get self._vindex_get=vindex.get # .store() needs to compare the passed-in serial to the # current tid in the database. _oid2tid caches the oid -> # current tid mapping for non-version data (if the current # record for oid is version data, the oid is not a key in # _oid2tid). The point is that otherwise seeking into the # storage is needed to extract the current tid, and that's # an expensive operation. For example, if a transaction # stores 4000 objects, and each random seek + read takes 7ms # (that was approximately true on Linux and Windows tests in # mid-2003), that's 28 seconds just to find the old tids. # TODO: Probably better to junk this and redefine _index as mapping # oid to (offset, tid) pair, via a new memory-efficient BTree type. self._oid2tid = oid2tid # oid->tid map to transactionally add to _oid2tid. self._toid2tid = toid2tid # Set of oids to transactionally delete from _oid2tid (e.g., # oids reverted by undo, or for which the most recent record # becomes version data). self._toid2tid_delete = toid2tid_delete def __len__(self): return len(self._index) def _newIndexes(self): # hook to use something other than builtin dict return fsIndex(), {}, {}, {}, {}, {}, {} _saved = 0 def _save_index(self): """Write the database index to a file to support quick startup.""" if self._is_read_only: return index_name = self.__name__ + '.index' tmp_name = index_name + '.index_tmp' f=open(tmp_name,'wb') p=Pickler(f,1) # Note: starting with ZODB 3.2.6, the 'oid' value stored is ignored # by the code that reads the index. We still write it, so that # .index files can still be read by older ZODBs. info={'index': self._index, 'pos': self._pos, 'oid': self._oid, 'vindex': self._vindex} p.dump(info) f.flush() f.close() try: try: os.remove(index_name) except OSError: pass os.rename(tmp_name, index_name) except: pass self._saved += 1 def _clear_index(self): index_name = self.__name__ + '.index' if os.path.exists(index_name): try: os.remove(index_name) except OSError: pass def _sane(self, index, pos): """Sanity check saved index data by reading the last undone trans Basically, we read the last not undone transaction and check to see that the included records are consistent with the index. Any invalid record records or inconsistent object positions cause zero to be returned. """ r = self._check_sanity(index, pos) if not r: logger.warning("Ignoring index for %s", self._file_name) return r def _check_sanity(self, index, pos): if pos < 100: return 0 # insane self._file.seek(0, 2) if self._file.tell() < pos: return 0 # insane ltid = None max_checked = 5 checked = 0 while checked < max_checked: self._file.seek(pos - 8) rstl = self._file.read(8) tl = u64(rstl) pos = pos - tl - 8 if pos < 4: return 0 # insane h = self._read_txn_header(pos) if not ltid: ltid = h.tid if h.tlen != tl: return 0 # inconsistent lengths if h.status == 'u': continue # undone trans, search back if h.status not in ' p': return 0 # insane if tl < h.headerlen(): return 0 # insane tend = pos + tl opos = pos + h.headerlen() if opos == tend: continue # empty trans while opos < tend and checked < max_checked: # Read the data records for this transaction h = self._read_data_header(opos) if opos + h.recordlen() > tend or h.tloc != pos: return 0 if index.get(h.oid, 0) != opos: return 0 # insane checked += 1 opos = opos + h.recordlen() return ltid def _restore_index(self): """Load database index to support quick startup.""" # Returns (index, vindex, pos, tid), or None in case of # error. # Starting with ZODB 3.2.6, the 'oid' value stored in the index # is ignored. # The index returned is always an instance of fsIndex. If the # index cached in the file is a Python dict, it's converted to # fsIndex here, and, if we're not in read-only mode, the .index # file is rewritten with the converted fsIndex so we don't need to # convert it again the next time. file_name=self.__name__ index_name=file_name+'.index' try: f = open(index_name, 'rb') except: return None p=Unpickler(f) try: info=p.load() except: exc, err = sys.exc_info()[:2] logger.warning("Failed to load database index: %s: %s", exc, err) return None index = info.get('index') pos = info.get('pos') vindex = info.get('vindex') if index is None or pos is None or vindex is None: return None pos = long(pos) if (isinstance(index, dict) or (isinstance(index, fsIndex) and isinstance(index._data, dict))): # Convert dictionary indexes to fsIndexes *or* convert fsIndexes # which have a dict `_data` attribute to a new fsIndex (newer # fsIndexes have an OOBTree as `_data`). newindex = fsIndex() newindex.update(index) index = newindex if not self._is_read_only: # Save the converted index. f = open(index_name, 'wb') p = Pickler(f, 1) info['index'] = index p.dump(info) f.close() # Now call this method again to get the new data. return self._restore_index() tid = self._sane(index, pos) if not tid: return None return index, vindex, pos, tid def close(self): self._file.close() if hasattr(self,'_lock_file'): self._lock_file.close() if self._tfile: self._tfile.close() try: self._save_index() except: # Log the error and continue logger.error("Error saving index on close()", exc_info=True) # Return tid of most recent record for oid if that's in the # _oid2tid cache. Else return None. It's important to use this # instead of indexing _oid2tid directly so that cache statistics # can be logged. def _get_cached_tid(self, oid): self._oid2tid_nlookups += 1 result = self._oid2tid.get(oid) if result is not None: self._oid2tid_nhits += 1 # Log a msg every ~8000 tries. if self._oid2tid_nlookups & 0x1fff == 0: logger.log(BLATHER, "_oid2tid size %s lookups %s hits %s rate %.1f%%", len(self._oid2tid), self._oid2tid_nlookups, self._oid2tid_nhits, 100.0 * self._oid2tid_nhits / self._oid2tid_nlookups) return result def abortVersion(self, src, transaction): return self.commitVersion(src, '', transaction, abort=True) def commitVersion(self, src, dest, transaction, abort=False): # We are going to commit by simply storing back pointers. if self._is_read_only: raise POSException.ReadOnlyError() if not (src and isinstance(src, StringType) and isinstance(dest, StringType)): raise POSException.VersionCommitError('Invalid source version') if src == dest: raise POSException.VersionCommitError( "Can't commit to same version: %s" % repr(src)) if dest and abort: raise POSException.VersionCommitError( "Internal error, can't abort to a version") if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: return self._commitVersion(src, dest, transaction, abort) finally: self._lock_release() def _commitVersion(self, src, dest, transaction, abort=False): # call after checking arguments and acquiring lock srcpos = self._vindex_get(src, 0) spos = p64(srcpos) # middle holds bytes 16:34 of a data record: # pos of transaction, len of version name, data length # commit version never writes data, so data length is always 0 middle = pack(">8sH8s", p64(self._pos), len(dest), z64) if dest: sd = p64(self._vindex_get(dest, 0)) heredelta = 66 + len(dest) else: sd = '' heredelta = 50 here = self._pos + (self._tfile.tell() + self._thl) oids = [] current_oids = {} while srcpos: h = self._read_data_header(srcpos) if self._index.get(h.oid) == srcpos: # This is a current record! self._tindex[h.oid] = here oids.append(h.oid) self._tfile.write(h.oid + self._tid + spos + middle) if dest: self._tvindex[dest] = here self._tfile.write(p64(h.pnv) + sd + dest) sd = p64(here) self._tfile.write(abort and p64(h.pnv) or spos) # data backpointer to src data here += heredelta current_oids[h.oid] = 1 else: # Hm. This is a non-current record. Is there a # current record for this oid? if not current_oids.has_key(h.oid): break srcpos = h.vprev spos = p64(srcpos) self._toid2tid_delete.update(current_oids) return self._tid, oids def getSize(self): return self._pos def _lookup_pos(self, oid): try: return self._index[oid] except KeyError: raise POSKeyError(oid) except TypeError: raise TypeError("invalid oid %r" % (oid,)) def loadEx(self, oid, version): # A variant of load() that also returns the version string. # ZEO wants this for managing its cache. self._lock_acquire() try: pos = self._lookup_pos(oid) h = self._read_data_header(pos, oid) if h.version and h.version != version: # Return data and tid from pnv (non-version data). # If we return the old record's transaction id, then # it will look to the cache like old data is current. # The tid for the current data must always be greater # than any non-current data. data = self._loadBack_impl(oid, h.pnv)[0] return data, h.tid, "" if h.plen: data = self._file.read(h.plen) return data, h.tid, h.version else: # Get the data from the backpointer, but tid from # currnt txn. data = self._loadBack_impl(oid, h.back)[0] return data, h.tid, h.version finally: self._lock_release() def load(self, oid, version): """Return pickle data and serial number.""" self._lock_acquire() try: pos = self._lookup_pos(oid) h = self._read_data_header(pos, oid) if h.version and h.version != version: data = self._loadBack_impl(oid, h.pnv)[0] return data, h.tid if h.plen: data = self._file.read(h.plen) return data, h.tid else: # Get the data from the backpointer, but tid from # currnt txn. data = self._loadBack_impl(oid, h.back)[0] return data, h.tid finally: self._lock_release() def loadSerial(self, oid, serial): # loadSerial must always return non-version data, because it # is used by conflict resolution. self._lock_acquire() try: pos = self._lookup_pos(oid) while 1: h = self._read_data_header(pos, oid) if h.tid == serial: break pos = h.prev if not pos: raise POSKeyError(oid) if h.version: return self._loadBack_impl(oid, h.pnv)[0] if h.plen: return self._file.read(h.plen) else: return self._loadBack_impl(oid, h.back)[0] finally: self._lock_release() def loadBefore(self, oid, tid): self._lock_acquire() try: pos = self._lookup_pos(oid) end_tid = None while True: h = self._read_data_header(pos, oid) if h.version: # Just follow the pnv pointer to the previous # non-version data. if not h.pnv: # Object was created in version. There is no # before data to find. return None pos = h.pnv # The end_tid for the non-version data is not affected # by versioned data records. continue if h.tid < tid: break pos = h.prev end_tid = h.tid if not pos: return None if h.back: data, _, _, _ = self._loadBack_impl(oid, h.back) return data, h.tid, end_tid else: return self._file.read(h.plen), h.tid, end_tid finally: self._lock_release() def modifiedInVersion(self, oid): self._lock_acquire() try: pos = self._lookup_pos(oid) h = self._read_data_header(pos, oid) return h.version finally: self._lock_release() def store(self, oid, serial, data, version, transaction): if self._is_read_only: raise POSException.ReadOnlyError() if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: if oid > self._oid: self.set_max_oid(oid) old = self._index_get(oid, 0) cached_tid = None pnv = None if old: cached_tid = self._get_cached_tid(oid) if cached_tid is None: h = self._read_data_header(old, oid) if h.version: if h.version != version: raise VersionLockError(oid, h.version) pnv = h.pnv cached_tid = h.tid if serial != cached_tid: rdata = self.tryToResolveConflict(oid, cached_tid, serial, data) if rdata is None: raise POSException.ConflictError( oid=oid, serials=(cached_tid, serial), data=data) else: data = rdata pos = self._pos here = pos + self._tfile.tell() + self._thl self._tindex[oid] = here new = DataHeader(oid, self._tid, old, pos, len(version), len(data)) if version: # Link to last record for this version: pv = (self._tvindex.get(version, 0) or self._vindex.get(version, 0)) if pnv is None: pnv = old new.setVersion(version, pnv, pv) self._tvindex[version] = here self._toid2tid_delete[oid] = 1 else: self._toid2tid[oid] = self._tid self._tfile.write(new.asString()) self._tfile.write(data) # Check quota if self._quota is not None and here > self._quota: raise FileStorageQuotaError( "The storage quota has been exceeded.") if old and serial != cached_tid: return ConflictResolution.ResolvedSerial else: return self._tid finally: self._lock_release() def _data_find(self, tpos, oid, data): # Return backpointer for oid. Must call with the lock held. # This is a file offset to oid's data record if found, else 0. # The data records in the transaction at tpos are searched for oid. # If a data record for oid isn't found, returns 0. # Else if oid's data record contains a backpointer, that # backpointer is returned. # Else oid's data record contains the data, and the file offset of # oid's data record is returned. This data record should contain # a pickle identical to the 'data' argument. # Unclear: If the length of the stored data doesn't match len(data), # an exception is raised. If the lengths match but the data isn't # the same, 0 is returned. Why the discrepancy? self._file.seek(tpos) h = self._file.read(TRANS_HDR_LEN) tid, tl, status, ul, dl, el = unpack(TRANS_HDR, h) self._file.read(ul + dl + el) tend = tpos + tl + 8 pos = self._file.tell() while pos < tend: h = self._read_data_header(pos) if h.oid == oid: # Make sure this looks like the right data record if h.plen == 0: # This is also a backpointer. Gotta trust it. return pos if h.plen != len(data): # The expected data doesn't match what's in the # backpointer. Something is wrong. logger.error("Mismatch between data and" " backpointer at %d", pos) return 0 _data = self._file.read(h.plen) if data != _data: return 0 return pos pos += h.recordlen() self._file.seek(pos) return 0 def restore(self, oid, serial, data, version, prev_txn, transaction): # A lot like store() but without all the consistency checks. This # should only be used when we /know/ the data is good, hence the # method name. While the signature looks like store() there are some # differences: # # - serial is the serial number of /this/ revision, not of the # previous revision. It is used instead of self._tid, which is # ignored. # # - Nothing is returned # # - data can be None, which indicates a George Bailey object # (i.e. one who's creation has been transactionally undone). # # prev_txn is a backpointer. In the original database, it's possible # that the data was actually living in a previous transaction. This # can happen for transactional undo and other operations, and is used # as a space saving optimization. Under some circumstances the # prev_txn may not actually exist in the target database (i.e. self) # for example, if it's been packed away. In that case, the prev_txn # should be considered just a hint, and is ignored if the transaction # doesn't exist. if self._is_read_only: raise POSException.ReadOnlyError() if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: if oid > self._oid: self.set_max_oid(oid) prev_pos = 0 if prev_txn is not None: prev_txn_pos = self._txn_find(prev_txn, 0) if prev_txn_pos: prev_pos = self._data_find(prev_txn_pos, oid, data) old = self._index_get(oid, 0) # Calculate the file position in the temporary file here = self._pos + self._tfile.tell() + self._thl # And update the temp file index self._tindex[oid] = here if prev_pos: # If there is a valid prev_pos, don't write data. data = None if data is None: dlen = 0 else: dlen = len(data) # Write the recovery data record new = DataHeader(oid, serial, old, self._pos, len(version), dlen) if version: pnv = self._restore_pnv(oid, old, version, prev_pos) or old vprev = self._tvindex.get(version, 0) if not vprev: vprev = self._vindex.get(version, 0) new.setVersion(version, pnv, vprev) self._tvindex[version] = here self._toid2tid_delete[oid] = 1 else: self._toid2tid[oid] = serial self._tfile.write(new.asString()) # Finally, write the data or a backpointer. if data is None: if prev_pos: self._tfile.write(p64(prev_pos)) else: # Write a zero backpointer, which indicates an # un-creation transaction. self._tfile.write(z64) else: self._tfile.write(data) finally: self._lock_release() def _restore_pnv(self, oid, prev, version, bp): # Find a valid pnv (previous non-version) pointer for this version. # If there is no previous record, there can't be a pnv. if not prev: return None # Load the record pointed to be prev h = self._read_data_header(prev, oid) if h.version: return h.pnv if h.back: # TODO: Not sure the following is always true: # The previous record is not for this version, yet we # have a backpointer to it. The current record must # be an undo of an abort or commit, so the backpointer # must be to a version record with a pnv. h2 = self._read_data_header(h.back, oid) if h2.version: return h2.pnv return None def supportsUndo(self): return 1 def supportsVersions(self): return 1 def _clear_temp(self): self._tindex.clear() self._tvindex.clear() self._toid2tid.clear() self._toid2tid_delete.clear() if self._tfile is not None: self._tfile.seek(0) def _begin(self, tid, u, d, e): self._nextpos = 0 self._thl = TRANS_HDR_LEN + len(u) + len(d) + len(e) if self._thl > 65535: # one of u, d, or e may be > 65535 # We have to check lengths here because struct.pack # doesn't raise an exception on overflow! if len(u) > 65535: raise FileStorageError('user name too long') if len(d) > 65535: raise FileStorageError('description too long') if len(e) > 65535: raise FileStorageError('too much extension data') def tpc_vote(self, transaction): self._lock_acquire() try: if transaction is not self._transaction: return dlen = self._tfile.tell() if not dlen: return # No data in this trans self._tfile.seek(0) user, descr, ext = self._ude self._file.seek(self._pos) tl = self._thl + dlen try: h = TxnHeader(self._tid, tl, "c", len(user), len(descr), len(ext)) h.user = user h.descr = descr h.ext = ext self._file.write(h.asString()) cp(self._tfile, self._file, dlen) self._file.write(p64(tl)) self._file.flush() except: # Hm, an error occured writing out the data. Maybe the # disk is full. We don't want any turd at the end. self._file.truncate(self._pos) raise self._nextpos = self._pos + (tl + 8) finally: self._lock_release() # Keep track of the number of records that we've written _records_written = 0 def _finish(self, tid, u, d, e): nextpos=self._nextpos if nextpos: file=self._file # Clear the checkpoint flag file.seek(self._pos+16) file.write(self._tstatus) file.flush() if fsync is not None: fsync(file.fileno()) self._pos = nextpos self._index.update(self._tindex) self._vindex.update(self._tvindex) self._oid2tid.update(self._toid2tid) for oid in self._toid2tid_delete.keys(): try: del self._oid2tid[oid] except KeyError: pass # Update the number of records that we've written # +1 for the transaction record self._records_written += len(self._tindex) + 1 if self._records_written >= self._records_before_save: self._save_index() self._records_written = 0 self._records_before_save = max(self._records_before_save, len(self._index)) self._ltid = tid def _abort(self): if self._nextpos: self._file.truncate(self._pos) self._nextpos=0 def supportsTransactionalUndo(self): return 1 def _undoDataInfo(self, oid, pos, tpos): """Return the tid, data pointer, data, and version for the oid record at pos""" if tpos: pos = tpos - self._pos - self._thl tpos = self._tfile.tell() h = self._tfmt._read_data_header(pos, oid) afile = self._tfile else: h = self._read_data_header(pos, oid) afile = self._file if h.oid != oid: raise UndoError("Invalid undo transaction id", oid) if h.plen: data = afile.read(h.plen) else: data = '' pos = h.back if tpos: self._tfile.seek(tpos) # Restore temp file to end return h.tid, pos, data, h.version def getTid(self, oid): self._lock_acquire() try: result = self._get_cached_tid(oid) if result is None: pos = self._lookup_pos(oid) result = self._getTid(oid, pos) return result finally: self._lock_release() def _getTid(self, oid, pos): self._file.seek(pos) h = self._file.read(16) assert oid == h[:8] return h[8:] def _getVersion(self, oid, pos): h = self._read_data_header(pos, oid) if h.version: return h.version, h.pnv else: return "", None def _transactionalUndoRecord(self, oid, pos, tid, pre, version): """Get the indo information for a data record Return a 5-tuple consisting of a pickle, data pointer, version, packed non-version data pointer, and current position. If the pickle is true, then the data pointer must be 0, but the pickle can be empty *and* the pointer 0. """ copy = 1 # Can we just copy a data pointer # First check if it is possible to undo this record. tpos = self._tindex.get(oid, 0) ipos = self._index.get(oid, 0) tipos = tpos or ipos if tipos != pos: # Eek, a later transaction modified the data, but, # maybe it is pointing at the same data we are. ctid, cdataptr, cdata, cver = self._undoDataInfo(oid, ipos, tpos) # Versions of undone record and current record *must* match! if cver != version: raise UndoError('Current and undone versions differ', oid) if cdataptr != pos: # We aren't sure if we are talking about the same data try: if ( # The current record wrote a new pickle cdataptr == tipos or # Backpointers are different self._loadBackPOS(oid, pos) != self._loadBackPOS(oid, cdataptr) ): if pre and not tpos: copy = 0 # we'll try to do conflict resolution else: # We bail if: # - We don't have a previous record, which should # be impossible. raise UndoError("no previous record", oid) except KeyError: # LoadBack gave us a key error. Bail. raise UndoError("_loadBack() failed", oid) # Return the data that should be written in the undo record. if not pre: # There is no previous revision, because the object creation # is being undone. return "", 0, "", "", ipos version, snv = self._getVersion(oid, pre) if copy: # we can just copy our previous-record pointer forward return "", pre, version, snv, ipos try: bdata = self._loadBack_impl(oid, pre)[0] except KeyError: # couldn't find oid; what's the real explanation for this? raise UndoError("_loadBack() failed for %s", oid) data = self.tryToResolveConflict(oid, ctid, tid, bdata, cdata) if data: return data, 0, version, snv, ipos raise UndoError("Some data were modified by a later transaction", oid) # undoLog() returns a description dict that includes an id entry. # The id is opaque to the client, but contains the transaction id. # The transactionalUndo() implementation does a simple linear # search through the file (from the end) to find the transaction. def undoLog(self, first=0, last=-20, filter=None): if last < 0: # -last is supposed to be the max # of transactions. Convert to # a positive index. Should have x - first = -last, which # means x = first - last. This is spelled out here because # the normalization code was incorrect for years (used +1 # instead -- off by 1), until ZODB 3.4. last = first - last self._lock_acquire() try: if self._pack_is_in_progress: raise UndoError( 'Undo is currently disabled for database maintenance.

') us = UndoSearch(self._file, self._pos, first, last, filter) while not us.finished(): # Hold lock for batches of 20 searches, so default search # parameters will finish without letting another thread run. for i in range(20): if us.finished(): break us.search() # Give another thread a chance, so that a long undoLog() # operation doesn't block all other activity. self._lock_release() self._lock_acquire() return us.results finally: self._lock_release() def undo(self, transaction_id, transaction): """Undo a transaction, given by transaction_id. Do so by writing new data that reverses the action taken by the transaction. Usually, we can get by with just copying a data pointer, by writing a file position rather than a pickle. Sometimes, we may do conflict resolution, in which case we actually copy new data that results from resolution. """ if self._is_read_only: raise POSException.ReadOnlyError() if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: return self._txn_undo(transaction_id) finally: self._lock_release() def _txn_undo(self, transaction_id): # Find the right transaction to undo and call _txn_undo_write(). tid = base64.decodestring(transaction_id + '\n') assert len(tid) == 8 tpos = self._txn_find(tid, 1) tindex = self._txn_undo_write(tpos) self._tindex.update(tindex) # Arrange to clear the affected oids from the oid2tid cache. # It's too painful to try to update them to correct current # values instead. self._toid2tid_delete.update(tindex) return self._tid, tindex.keys() def _txn_find(self, tid, stop_at_pack): pos = self._pos while pos > 39: self._file.seek(pos - 8) pos = pos - u64(self._file.read(8)) - 8 self._file.seek(pos) h = self._file.read(TRANS_HDR_LEN) _tid = h[:8] if _tid == tid: return pos if stop_at_pack: # check the status field of the transaction header if h[16] == 'p': break raise UndoError("Invalid transaction id") def _txn_undo_write(self, tpos): # a helper function to write the data records for transactional undo otloc = self._pos here = self._pos + self._tfile.tell() + self._thl base = here - self._tfile.tell() # Let's move the file pointer back to the start of the txn record. th = self._read_txn_header(tpos) if th.status != " ": raise UndoError('non-undoable transaction') tend = tpos + th.tlen pos = tpos + th.headerlen() tindex = {} # keep track of failures, cause we may succeed later failures = {} # Read the data records for this transaction while pos < tend: h = self._read_data_header(pos) if h.oid in failures: del failures[h.oid] # second chance! assert base + self._tfile.tell() == here, (here, base, self._tfile.tell()) try: p, prev, v, snv, ipos = self._transactionalUndoRecord( h.oid, pos, h.tid, h.prev, h.version) except UndoError, v: # Don't fail right away. We may be redeemed later! failures[h.oid] = v else: new = DataHeader(h.oid, self._tid, ipos, otloc, len(v), len(p)) if v: vprev = self._tvindex.get(v, 0) or self._vindex.get(v, 0) new.setVersion(v, snv, vprev) self._tvindex[v] = here # TODO: This seek shouldn't be necessary, but some other # bit of code is messing with the file pointer. assert self._tfile.tell() == here - base, (here, base, self._tfile.tell()) self._tfile.write(new.asString()) if p: self._tfile.write(p) else: self._tfile.write(p64(prev)) tindex[h.oid] = here here += new.recordlen() pos += h.recordlen() if pos > tend: raise UndoError("non-undoable transaction") if failures: raise MultipleUndoErrors(failures.items()) return tindex def versionEmpty(self, version): if not version: # The interface is silent on this case. I think that this should # be an error, but Barry thinks this should return 1 if we have # any non-version data. This would be excruciatingly painful to # test, so I must be right. ;) raise POSException.VersionError( 'The version must be an non-empty string') self._lock_acquire() try: index=self._index file=self._file seek=file.seek read=file.read srcpos=self._vindex_get(version, 0) t=tstatus=None while srcpos: seek(srcpos) oid=read(8) if index[oid]==srcpos: return 0 h=read(50) # serial, prev(oid), tloc, vlen, plen, pnv, pv tloc=h[16:24] if t != tloc: # We haven't checked this transaction before, # get its status. t=tloc seek(u64(t)+16) tstatus=read(1) if tstatus != 'u': return 1 spos=h[-8:] srcpos=u64(spos) return 1 finally: self._lock_release() def versions(self, max=None): r=[] a=r.append keys=self._vindex.keys() if max is not None: keys=keys[:max] for version in keys: if self.versionEmpty(version): continue a(version) if max and len(r) >= max: return r return r def history(self, oid, version=None, size=1, filter=None): self._lock_acquire() try: r = [] pos = self._lookup_pos(oid) wantver = version while 1: if len(r) >= size: return r h = self._read_data_header(pos) if h.version: if wantver is not None and h.version != wantver: if h.prev: pos = h.prev continue else: return r else: version = "" wantver = None th = self._read_txn_header(h.tloc) if th.ext: d = loads(th.ext) else: d = {} d.update({"time": TimeStamp(h.tid).timeTime(), "user_name": th.user, "description": th.descr, "tid": h.tid, "version": h.version, "size": h.plen, }) if filter is None or filter(d): r.append(d) if h.prev: pos = h.prev else: return r finally: self._lock_release() def _redundant_pack(self, file, pos): assert pos > 8, pos file.seek(pos - 8) p = u64(file.read(8)) file.seek(pos - p + 8) return file.read(1) not in ' u' def pack(self, t, referencesf): """Copy data from the current database file to a packed file Non-current records from transactions with time-stamp strings less than packtss are ommitted. As are all undone records. Also, data back pointers that point before packtss are resolved and the associated data are copied, since the old records are not copied. """ if self._is_read_only: raise POSException.ReadOnlyError() stop=`TimeStamp(*time.gmtime(t)[:5]+(t%60,))` if stop==z64: raise FileStorageError, 'Invalid pack time' # If the storage is empty, there's nothing to do. if not self._index: return self._lock_acquire() try: if self._pack_is_in_progress: raise FileStorageError, 'Already packing' self._pack_is_in_progress = True current_size = self.getSize() finally: self._lock_release() p = FileStoragePacker(self._file_name, stop, self._lock_acquire, self._lock_release, self._commit_lock_acquire, self._commit_lock_release, current_size) try: opos = None try: opos = p.pack() except RedundantPackWarning, detail: logger.info(str(detail)) if opos is None: return oldpath = self._file_name + ".old" self._lock_acquire() try: self._file.close() try: if os.path.exists(oldpath): os.remove(oldpath) os.rename(self._file_name, oldpath) except Exception: self._file = open(self._file_name, 'r+b') raise # OK, we're beyond the point of no return os.rename(self._file_name + '.pack', self._file_name) self._file = open(self._file_name, 'r+b') self._initIndex(p.index, p.vindex, p.tindex, p.tvindex, p.oid2tid, p.toid2tid, p.toid2tid_delete) self._pos = opos self._save_index() finally: self._lock_release() finally: if p.locked: self._commit_lock_release() self._lock_acquire() self._pack_is_in_progress = False self._lock_release() def iterator(self, start=None, stop=None): return FileIterator(self._file_name, start, stop) def lastTransaction(self): """Return transaction id for last committed transaction""" return self._ltid def lastTid(self, oid): """Return last serialno committed for object oid. If there is no serialno for this oid -- which can only occur if it is a new object -- return None. """ try: return self.getTid(oid) except KeyError: return None def cleanup(self): """Remove all files created by this storage.""" for ext in '', '.old', '.tmp', '.lock', '.index', '.pack': try: os.remove(self._file_name + ext) except OSError, e: if e.errno != errno.ENOENT: raise def record_iternext(self, next=None): index = self._index oid = index.minKey(next) oid_as_long, = unpack(">Q", oid) next_oid = pack(">Q", oid_as_long + 1) try: next_oid = index.minKey(next_oid) except ValueError: # "empty tree" error next_oid = None data, tid = self.load(oid, "") # ignore versions return oid, tid, data, next_oid def shift_transactions_forward(index, vindex, tindex, file, pos, opos): """Copy transactions forward in the data file This might be done as part of a recovery effort """ # Cache a bunch of methods seek=file.seek read=file.read write=file.write index_get=index.get vindex_get=vindex.get # Initialize, pv=z64 p1=opos p2=pos offset=p2-p1 # Copy the data in two stages. In the packing stage, # we skip records that are non-current or that are for # unreferenced objects. We also skip undone transactions. # # After the packing stage, we copy everything but undone # transactions, however, we have to update various back pointers. # We have to have the storage lock in the second phase to keep # data from being changed while we're copying. pnv=None while 1: # Read the transaction record seek(pos) h=read(TRANS_HDR_LEN) if len(h) < TRANS_HDR_LEN: break tid, stl, status, ul, dl, el = unpack(TRANS_HDR,h) if status=='c': break # Oops. we found a checkpoint flag. tl=u64(stl) tpos=pos tend=tpos+tl otpos=opos # start pos of output trans thl=ul+dl+el h2=read(thl) if len(h2) != thl: raise PackError(opos) # write out the transaction record seek(opos) write(h) write(h2) thl=TRANS_HDR_LEN+thl pos=tpos+thl opos=otpos+thl while pos < tend: # Read the data records for this transaction seek(pos) h=read(DATA_HDR_LEN) oid,serial,sprev,stloc,vlen,splen = unpack(DATA_HDR, h) plen=u64(splen) dlen=DATA_HDR_LEN+(plen or 8) if vlen: dlen=dlen+(16+vlen) pnv=u64(read(8)) # skip position of previous version record seek(8,1) version=read(vlen) pv=p64(vindex_get(version, 0)) if status != 'u': vindex[version]=opos tindex[oid]=opos if plen: p=read(plen) else: p=read(8) p=u64(p) if p >= p2: p=p-offset elif p >= p1: # Ick, we're in trouble. Let's bail # to the index and hope for the best p=index_get(oid, 0) p=p64(p) # WRITE seek(opos) sprev=p64(index_get(oid, 0)) write(pack(DATA_HDR, oid,serial,sprev,p64(otpos),vlen,splen)) if vlen: if not pnv: write(z64) else: if pnv >= p2: pnv=pnv-offset elif pnv >= p1: pnv=index_get(oid, 0) write(p64(pnv)) write(pv) write(version) write(p) opos=opos+dlen pos=pos+dlen # skip the (intentionally redundant) transaction length pos=pos+8 if status != 'u': index.update(tindex) # Record the position tindex.clear() write(stl) opos=opos+8 return opos def search_back(file, pos): seek=file.seek read=file.read seek(0,2) s=p=file.tell() while p > pos: seek(p-8) l=u64(read(8)) if l <= 0: break p=p-l-8 return p, s def recover(file_name): file=open(file_name, 'r+b') index={} vindex={} tindex={} pos, oid, tid = read_index( file, file_name, index, vindex, tindex, recover=1) if oid is not None: print "Nothing to recover" return opos=pos pos, sz = search_back(file, pos) if pos < sz: npos = shift_transactions_forward( index, vindex, tindex, file, pos, opos, ) file.truncate(npos) print "Recovered file, lost %s, ended up with %s bytes" % ( pos-opos, npos) def read_index(file, name, index, vindex, tindex, stop='\377'*8, ltid=z64, start=4L, maxoid=z64, recover=0, read_only=0): """Scan the file storage and update the index. Returns file position, max oid, and last transaction id. It also stores index information in the three dictionary arguments. Arguments: file -- a file object (the Data.fs) name -- the name of the file (presumably file.name) index -- fsIndex, oid -> data record file offset vindex -- dictionary, oid -> data record offset for version data tindex -- dictionary, oid -> data record offset tindex is cleared before return There are several default arguments that affect the scan or the return values. TODO: document them. start -- the file position at which to start scanning for oids added beyond the ones the passed-in indices know about. The .index file caches the highest ._pos FileStorage knew about when the the .index file was last saved, and that's the intended value to pass in for start; accept the default (and pass empty indices) to recreate the index from scratch maxoid -- ignored (it meant something prior to ZODB 3.2.6; the argument still exists just so the signature of read_index() stayed the same) The file position returned is the position just after the last valid transaction record. The oid returned is the maximum object id in `index`, or z64 if the index is empty. The transaction id is the tid of the last transaction, or ltid if the index is empty. """ read = file.read seek = file.seek seek(0, 2) file_size = file.tell() fmt = TempFormatter(file) if file_size: if file_size < start: raise FileStorageFormatError, file.name seek(0) if read(4) != packed_version: raise FileStorageFormatError, name else: if not read_only: file.write(packed_version) return 4L, z64, ltid index_get = index.get pos = start seek(start) tid = '\0' * 7 + '\1' while 1: # Read the transaction record h = read(TRANS_HDR_LEN) if not h: break if len(h) != TRANS_HDR_LEN: if not read_only: logger.warning('%s truncated at %s', name, pos) seek(pos) file.truncate() break tid, tl, status, ul, dl, el = unpack(TRANS_HDR, h) if tid <= ltid: logger.warning("%s time-stamp reduction at %s", name, pos) ltid = tid if pos+(tl+8) > file_size or status=='c': # Hm, the data were truncated or the checkpoint flag wasn't # cleared. They may also be corrupted, # in which case, we don't want to totally lose the data. if not read_only: logger.warning("%s truncated, possibly due to damaged" " records at %s", name, pos) _truncate(file, name, pos) break if status not in ' up': logger.warning('%s has invalid status, %s, at %s', name, status, pos) if tl < TRANS_HDR_LEN + ul + dl + el: # We're in trouble. Find out if this is bad data in the # middle of the file, or just a turd that Win 9x dropped # at the end when the system crashed. # Skip to the end and read what should be the transaction length # of the last transaction. seek(-8, 2) rtl = u64(read(8)) # Now check to see if the redundant transaction length is # reasonable: if file_size - rtl < pos or rtl < TRANS_HDR_LEN: logger.critical('%s has invalid transaction header at %s', name, pos) if not read_only: logger.warning( "It appears that there is invalid data at the end of " "the file, possibly due to a system crash. %s " "truncated to recover from bad data at end." % name) _truncate(file, name, pos) break else: if recover: return pos, None, None panic('%s has invalid transaction header at %s', name, pos) if tid >= stop: break tpos = pos tend = tpos + tl if status == 'u': # Undone transaction, skip it seek(tend) h = u64(read(8)) if h != tl: if recover: return tpos, None, None panic('%s has inconsistent transaction length at %s', name, pos) pos = tend + 8 continue pos = tpos + TRANS_HDR_LEN + ul + dl + el while pos < tend: # Read the data records for this transaction h = fmt._read_data_header(pos) dlen = h.recordlen() tindex[h.oid] = pos if h.version: vindex[h.version] = pos if pos + dlen > tend or h.tloc != tpos: if recover: return tpos, None, None panic("%s data record exceeds transaction record at %s", name, pos) if index_get(h.oid, 0) != h.prev: if h.prev: if recover: return tpos, None, None logger.error("%s incorrect previous pointer at %s", name, pos) else: logger.warning("%s incorrect previous pointer at %s", name, pos) pos += dlen if pos != tend: if recover: return tpos, None, None panic("%s data records don't add up at %s",name,tpos) # Read the (intentionally redundant) transaction length seek(pos) h = u64(read(8)) if h != tl: if recover: return tpos, None, None panic("%s redundant transaction length check failed at %s", name, pos) pos += 8 index.update(tindex) tindex.clear() # Caution: fsIndex doesn't have an efficient __nonzero__ or __len__. # That's why we do try/except instead. fsIndex.maxKey() is fast. try: maxoid = index.maxKey() except ValueError: # The index is empty. maxoid == z64 return pos, maxoid, ltid def _truncate(file, name, pos): file.seek(0, 2) file_size = file.tell() try: i = 0 while 1: oname='%s.tr%s' % (name, i) if os.path.exists(oname): i += 1 else: logger.warning("Writing truncated data from %s to %s", name, oname) o = open(oname,'wb') file.seek(pos) cp(file, o, file_size-pos) o.close() break except: logger.error("couldn\'t write truncated data for %s", name, exc_info=True) raise POSException.StorageSystemError, ( "Couldn't save truncated data") file.seek(pos) file.truncate() class Iterator: """A General simple iterator that uses the Python for-loop index protocol """ __index=-1 __current=None def __getitem__(self, i): __index=self.__index while i > __index: __index=__index+1 self.__current=self.next(__index) self.__index=__index return self.__current class FileIterator(Iterator, FileStorageFormatter): """Iterate over the transactions in a FileStorage file. """ _ltid = z64 _file = None def __init__(self, file, start=None, stop=None): if isinstance(file, str): file = open(file, 'rb') self._file = file if file.read(4) != packed_version: raise FileStorageFormatError, file.name file.seek(0,2) self._file_size = file.tell() self._pos = 4L assert start is None or isinstance(start, str) assert stop is None or isinstance(stop, str) if start: self._skip_to_start(start) self._stop = stop def __len__(self): # Define a bogus __len__() to make the iterator work # with code like builtin list() and tuple() in Python 2.1. # There's a lot of C code that expects a sequence to have # an __len__() but can cope with any sort of mistake in its # implementation. So just return 0. return 0 # This allows us to pass an iterator as the `other' argument to # copyTransactionsFrom() in BaseStorage. The advantage here is that we # can create the iterator manually, e.g. setting start and stop, and then # just let copyTransactionsFrom() do its thing. def iterator(self): return self def close(self): file = self._file if file is not None: self._file = None file.close() def _skip_to_start(self, start): # Scan through the transaction records doing almost no sanity # checks. file = self._file read = file.read seek = file.seek while 1: seek(self._pos) h = read(16) if len(h) < 16: return tid, stl = unpack(">8s8s", h) if tid >= start: return tl = u64(stl) try: self._pos += tl + 8 except OverflowError: self._pos = long(self._pos) + tl + 8 if __debug__: # Sanity check seek(self._pos - 8, 0) rtl = read(8) if rtl != stl: pos = file.tell() - 8 panic("%s has inconsistent transaction length at %s " "(%s != %s)", file.name, pos, u64(rtl), u64(stl)) def next(self, index=0): if self._file is None: # A closed iterator. Is IOError the best we can do? For # now, mimic a read on a closed file. raise IOError, 'iterator is closed' pos = self._pos while 1: # Read the transaction record try: h = self._read_txn_header(pos) except CorruptedDataError, err: # If buf is empty, we've reached EOF. if not err.buf: break raise if h.tid <= self._ltid: logger.warning("%s time-stamp reduction at %s", self._file.name, pos) self._ltid = h.tid if self._stop is not None and h.tid > self._stop: raise IndexError, index if h.status == "c": # Assume we've hit the last, in-progress transaction raise IndexError, index if pos + h.tlen + 8 > self._file_size: # Hm, the data were truncated or the checkpoint flag wasn't # cleared. They may also be corrupted, # in which case, we don't want to totally lose the data. logger.warning("%s truncated, possibly due to" " damaged records at %s", self._file.name, pos) break if h.status not in " up": logger.warning('%s has invalid status,' ' %s, at %s', self._file.name, h.status, pos) if h.tlen < h.headerlen(): # We're in trouble. Find out if this is bad data in # the middle of the file, or just a turd that Win 9x # dropped at the end when the system crashed. Skip to # the end and read what should be the transaction # length of the last transaction. self._file.seek(-8, 2) rtl = u64(self._file.read(8)) # Now check to see if the redundant transaction length is # reasonable: if self._file_size - rtl < pos or rtl < TRANS_HDR_LEN: logger.critical("%s has invalid transaction header at %s", self._file.name, pos) logger.warning( "It appears that there is invalid data at the end of " "the file, possibly due to a system crash. %s " "truncated to recover from bad data at end." % self._file.name) break else: logger.warning("%s has invalid transaction header at %s", self._file.name, pos) break tpos = pos tend = tpos + h.tlen if h.status != "u": pos = tpos + h.headerlen() e = {} if h.elen: try: e = loads(h.ext) except: pass result = RecordIterator(h.tid, h.status, h.user, h.descr, e, pos, tend, self._file, tpos) # Read the (intentionally redundant) transaction length self._file.seek(tend) rtl = u64(self._file.read(8)) if rtl != h.tlen: logger.warning("%s redundant transaction length check" " failed at %s", self._file.name, tend) break self._pos = tend + 8 return result raise IndexError, index class RecordIterator(Iterator, BaseStorage.TransactionRecord, FileStorageFormatter): """Iterate over the transactions in a FileStorage file.""" def __init__(self, tid, status, user, desc, ext, pos, tend, file, tpos): self.tid = tid self.status = status self.user = user self.description = desc self._extension = ext self._pos = pos self._tend = tend self._file = file self._tpos = tpos def next(self, index=0): pos = self._pos while pos < self._tend: # Read the data records for this transaction h = self._read_data_header(pos) dlen = h.recordlen() if pos + dlen > self._tend or h.tloc != self._tpos: logger.warning("%s data record exceeds transaction" " record at %s", file.name, pos) break self._pos = pos + dlen prev_txn = None if h.plen: data = self._file.read(h.plen) else: if h.back == 0: # If the backpointer is 0, then this transaction # undoes the object creation. It either aborts # the version that created the object or undid the # transaction that created it. Return None # instead of a pickle to indicate this. data = None else: data, tid = self._loadBackTxn(h.oid, h.back, False) # Caution: :ooks like this only goes one link back. # Should it go to the original data like BDBFullStorage? prev_txn = self.getTxnFromData(h.oid, h.back) r = Record(h.oid, h.tid, h.version, data, prev_txn, pos) return r raise IndexError, index class Record(BaseStorage.DataRecord): """An abstract database record.""" def __init__(self, oid, tid, version, data, prev, pos): self.oid = oid self.tid = tid self.version = version self.data = data self.data_txn = prev self.pos = pos class UndoSearch: def __init__(self, file, pos, first, last, filter=None): self.file = file self.pos = pos self.first = first self.last = last self.filter = filter # self.i is the index of the transaction we're _going_ to find # next. When it reaches self.first, we should start appending # to self.results. When it reaches self.last, we're done # (although we may finish earlier). self.i = 0 self.results = [] self.stop = False def finished(self): """Return True if UndoSearch has found enough records.""" # BAW: Why 39 please? This makes no sense (see also below). return self.i >= self.last or self.pos < 39 or self.stop def search(self): """Search for another record.""" dict = self._readnext() if dict is not None and (self.filter is None or self.filter(dict)): if self.i >= self.first: self.results.append(dict) self.i += 1 def _readnext(self): """Read the next record from the storage.""" self.file.seek(self.pos - 8) self.pos -= u64(self.file.read(8)) + 8 self.file.seek(self.pos) h = self.file.read(TRANS_HDR_LEN) tid, tl, status, ul, dl, el = unpack(TRANS_HDR, h) if status == 'p': self.stop = 1 return None if status != ' ': return None d = u = '' if ul: u = self.file.read(ul) if dl: d = self.file.read(dl) e = {} if el: try: e = loads(self.file.read(el)) except: pass d = {'id': base64.encodestring(tid).rstrip(), 'time': TimeStamp(tid).timeTime(), 'user_name': u, 'size': tl, 'description': d} d.update(e) return d Zope-2.8.7-final/lib/python/ZODB/FileStorage/fsdump.py0000664000076400007640000001172710435502423021447 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import struct from ZODB.FileStorage import FileIterator from ZODB.FileStorage.format \ import TRANS_HDR, TRANS_HDR_LEN, DATA_HDR, DATA_HDR_LEN from ZODB.TimeStamp import TimeStamp from ZODB.utils import u64, get_pickle_metadata from ZODB.tests.StorageTestBase import zodb_unpickle def fsdump(path, file=None, with_offset=1): iter = FileIterator(path) for i, trans in enumerate(iter): if with_offset: print >> file, "Trans #%05d tid=%016x time=%s offset=%d" % \ (i, u64(trans.tid), TimeStamp(trans.tid), trans._pos) else: print >> file, "Trans #%05d tid=%016x time=%s" % \ (i, u64(trans.tid), TimeStamp(trans.tid)) print >> file, " status=%r user=%r description=%r" % \ (trans.status, trans.user, trans.description) for j, rec in enumerate(trans): if rec.data is None: fullclass = "undo or abort of object creation" size = "" else: modname, classname = get_pickle_metadata(rec.data) size = " size=%d" % len(rec.data) fullclass = "%s.%s" % (modname, classname) if rec.version: version = " version=%r" % rec.version else: version = "" if rec.data_txn: # It would be nice to print the transaction number # (i) but it would be expensive to keep track of. bp = " bp=%016x" % u64(rec.data_txn) else: bp = "" print >> file, " data #%05d oid=%016x%s%s class=%s%s" % \ (j, u64(rec.oid), version, size, fullclass, bp) iter.close() def fmt(p64): # Return a nicely formatted string for a packaged 64-bit value return "%016x" % u64(p64) class Dumper: """A very verbose dumper for debuggin FileStorage problems.""" # TODO: Should revise this class to use FileStorageFormatter. def __init__(self, path, dest=None): self.file = open(path, "rb") self.dest = dest def dump(self): fid = self.file.read(4) print >> self.dest, "*" * 60 print >> self.dest, "file identifier: %r" % fid while self.dump_txn(): pass def dump_txn(self): pos = self.file.tell() h = self.file.read(TRANS_HDR_LEN) if not h: return False tid, tlen, status, ul, dl, el = struct.unpack(TRANS_HDR, h) end = pos + tlen print >> self.dest, "=" * 60 print >> self.dest, "offset: %d" % pos print >> self.dest, "end pos: %d" % end print >> self.dest, "transaction id: %s" % fmt(tid) print >> self.dest, "trec len: %d" % tlen print >> self.dest, "status: %r" % status user = descr = extra = "" if ul: user = self.file.read(ul) if dl: descr = self.file.read(dl) if el: extra = self.file.read(el) print >> self.dest, "user: %r" % user print >> self.dest, "description: %r" % descr print >> self.dest, "len(extra): %d" % el while self.file.tell() < end: self.dump_data(pos) stlen = self.file.read(8) print >> self.dest, "redundant trec len: %d" % u64(stlen) return 1 def dump_data(self, tloc): pos = self.file.tell() h = self.file.read(DATA_HDR_LEN) assert len(h) == DATA_HDR_LEN oid, revid, prev, tloc, vlen, dlen = struct.unpack(DATA_HDR, h) print >> self.dest, "-" * 60 print >> self.dest, "offset: %d" % pos print >> self.dest, "oid: %s" % fmt(oid) print >> self.dest, "revid: %s" % fmt(revid) print >> self.dest, "previous record offset: %d" % prev print >> self.dest, "transaction offset: %d" % tloc if vlen: pnv = self.file.read(8) sprevdata = self.file.read(8) version = self.file.read(vlen) print >> self.dest, "version: %r" % version print >> self.dest, "non-version data offset: %d" % u64(pnv) print >> self.dest, \ "previous version data offset: %d" % u64(sprevdata) print >> self.dest, "len(data): %d" % dlen self.file.read(dlen) if not dlen: sbp = self.file.read(8) print >> self.dest, "backpointer: %d" % u64(sbp) Zope-2.8.7-final/lib/python/ZODB/FileStorage/format.py0000664000076400007640000002725110435502423021440 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # # File-based ZODB storage # # Files are arranged as follows. # # - The first 4 bytes are a file identifier. # # - The rest of the file consists of a sequence of transaction # "records". # # A transaction record consists of: # # - 8-byte transaction id, which is also a time stamp. # # - 8-byte transaction record length - 8. # # - 1-byte status code # ' ' (a blank) completed transaction that hasn't been packed # 'p' completed transaction that has been packed # 'c' checkpoint -- a transaction in progress, at the end of the file; # it's been thru vote() but not finish(); if finish() completes # normally, it will be overwritten with a blank; if finish() dies # (e.g., out of disk space), cleanup code will try to truncate # the file to chop off this incomplete transaction # 'u' uncertain; no longer used; was previously used to record something # about non-transactional undo # # - 2-byte length of user name # # - 2-byte length of description # # - 2-byte length of extension attributes # # - user name # # - description # # - extension attributes # # * A sequence of data records # # - 8-byte redundant transaction length -8 # # A data record consists of # # - 8-byte oid. # # - 8-byte tid, which matches the transaction id in the transaction record. # # - 8-byte previous-record file-position. # # - 8-byte beginning of transaction record file position. # # - 2-byte version length # # - 8-byte data length # # ? 8-byte position of non-version data record # (if version length > 0) # # ? 8-byte position of previous record in this version # (if version length > 0) # # ? version string # (if version length > 0) # # ? data # (data length > 0) # # ? 8-byte position of data record containing data # (data length == 0) # # Note that the lengths and positions are all big-endian. # Also, the object ids time stamps are big-endian, so comparisons # are meaningful. # # Version handling # # There isn't a separate store for versions. Each record has a # version field, indicating what version it is in. The records in a # version form a linked list. Each record that has a non-empty # version string has a pointer to the previous record in the version. # Version back pointers are retained *even* when versions are # committed or aborted or when transactions are undone. # # There is a notion of "current" version records, which are the # records in a version that are the current records for their # respective objects. When a version is comitted, the current records # are committed to the destination version. When a version is # aborted, the current records are aborted. # # When committing or aborting, we search backward through the linked # list until we find a record for an object that does not have a # current record in the version. If we find a record for which the # non-version pointer is the same as the previous pointer, then we # forget that the corresponding object had a current record in the # version. This strategy allows us to avoid searching backward through # previously committed or aborted version records. # # Of course, we ignore records in undone transactions when committing # or aborting. # # Backpointers # # When we commit or abort a version, we don't copy (or delete) # and data. Instead, we write records with back pointers. # # A version record *never* has a back pointer to a non-version # record, because we never abort to a version. A non-version record # may have a back pointer to a version record or to a non-version # record. import struct import logging from ZODB.POSException import POSKeyError from ZODB.utils import u64, oid_repr class CorruptedError(Exception): pass class CorruptedDataError(CorruptedError): def __init__(self, oid=None, buf=None, pos=None): self.oid = oid self.buf = buf self.pos = pos def __str__(self): if self.oid: msg = "Error reading oid %s. Found %r" % (oid_repr(self.oid), self.buf) else: msg = "Error reading unknown oid. Found %r" % self.buf if self.pos: msg += " at %d" % self.pos return msg # the struct formats for the headers TRANS_HDR = ">8sQcHHH" DATA_HDR = ">8s8sQQHQ" # constants to support various header sizes TRANS_HDR_LEN = 23 DATA_HDR_LEN = 42 DATA_VERSION_HDR_LEN = 58 assert struct.calcsize(TRANS_HDR) == TRANS_HDR_LEN assert struct.calcsize(DATA_HDR) == DATA_HDR_LEN logger = logging.getLogger('ZODB.FileStorage.format') class FileStorageFormatter(object): """Mixin class that can read and write the low-level format.""" # subclasses must provide _file _metadata_size = 4L _format_version = "21" def _read_num(self, pos): """Read an 8-byte number.""" self._file.seek(pos) return u64(self._file.read(8)) def _read_data_header(self, pos, oid=None): """Return a DataHeader object for data record at pos. If ois is not None, raise CorruptedDataError if oid passed does not match oid in file. If there is version data, reads the version part of the header. If there is no pickle data, reads the back pointer. """ self._file.seek(pos) s = self._file.read(DATA_HDR_LEN) if len(s) != DATA_HDR_LEN: raise CorruptedDataError(oid, s, pos) h = DataHeaderFromString(s) if oid is not None and oid != h.oid: raise CorruptedDataError(oid, s, pos) if h.vlen: s = self._file.read(16 + h.vlen) h.parseVersion(s) if not h.plen: h.back = u64(self._file.read(8)) return h def _write_version_header(self, file, pnv, vprev, version): s = struct.pack(">8s8s", pnv, vprev) file.write(s + version) def _read_txn_header(self, pos, tid=None): self._file.seek(pos) s = self._file.read(TRANS_HDR_LEN) if len(s) != TRANS_HDR_LEN: raise CorruptedDataError(tid, s, pos) h = TxnHeaderFromString(s) if tid is not None and tid != h.tid: raise CorruptedDataError(tid, s, pos) h.user = self._file.read(h.ulen) h.descr = self._file.read(h.dlen) h.ext = self._file.read(h.elen) return h def _loadBack_impl(self, oid, back, fail=True): # shared implementation used by various _loadBack methods # # If the backpointer ultimately resolves to 0: # If fail is True, raise KeyError for zero backpointer. # If fail is False, return the empty data from the record # with no backpointer. while 1: if not back: # If backpointer is 0, object does not currently exist. raise POSKeyError(oid) h = self._read_data_header(back) if h.plen: return self._file.read(h.plen), h.tid, back, h.tloc if h.back == 0 and not fail: return None, h.tid, back, h.tloc back = h.back def _loadBackTxn(self, oid, back, fail=True): """Return data and txn id for backpointer.""" return self._loadBack_impl(oid, back, fail)[:2] def _loadBackPOS(self, oid, back): return self._loadBack_impl(oid, back)[2] def getTxnFromData(self, oid, back): """Return transaction id for data at back.""" h = self._read_data_header(back, oid) return h.tid def fail(self, pos, msg, *args): s = ("%s:%s:" + msg) % ((self._name, pos) + args) logger.error(s) raise CorruptedError(s) def checkTxn(self, th, pos): if th.tid <= self.ltid: self.fail(pos, "time-stamp reduction: %s <= %s", oid_repr(th.tid), oid_repr(self.ltid)) self.ltid = th.tid if th.status == "c": self.fail(pos, "transaction with checkpoint flag set") if not th.status in " pu": # recognize " ", "p", and "u" as valid self.fail(pos, "invalid transaction status: %r", th.status) if th.tlen < th.headerlen(): self.fail(pos, "invalid transaction header: " "txnlen (%d) < headerlen(%d)", th.tlen, th.headerlen()) def checkData(self, th, tpos, dh, pos): if dh.tloc != tpos: self.fail(pos, "data record does not point to transaction header" ": %d != %d", dh.tloc, tpos) if pos + dh.recordlen() > tpos + th.tlen: self.fail(pos, "data record size exceeds transaction size: " "%d > %d", pos + dh.recordlen(), tpos + th.tlen) if dh.prev >= pos: self.fail(pos, "invalid previous pointer: %d", dh.prev) if dh.back: if dh.back >= pos: self.fail(pos, "invalid back pointer: %d", dh.prev) if dh.plen: self.fail(pos, "data record has back pointer and data") def DataHeaderFromString(s): return DataHeader(*struct.unpack(DATA_HDR, s)) class DataHeader(object): """Header for a data record.""" __slots__ = ( "oid", "tid", "prev", "tloc", "vlen", "plen", "back", # These three attributes are only defined when vlen > 0 "pnv", "vprev", "version") def __init__(self, oid, tid, prev, tloc, vlen, plen): self.back = 0 # default self.version = "" # default self.oid = oid self.tid = tid self.prev = prev self.tloc = tloc self.vlen = vlen self.plen = plen def asString(self): s = struct.pack(DATA_HDR, self.oid, self.tid, self.prev, self.tloc, self.vlen, self.plen) if self.version: v = struct.pack(">QQ", self.pnv, self.vprev) return s + v + self.version else: return s def setVersion(self, version, pnv, vprev): self.version = version self.vlen = len(version) self.pnv = pnv self.vprev = vprev def parseVersion(self, buf): pnv, vprev = struct.unpack(">QQ", buf[:16]) self.pnv = pnv self.vprev = vprev self.version = buf[16:] def recordlen(self): rlen = DATA_HDR_LEN + (self.plen or 8) if self.version: rlen += 16 + self.vlen return rlen def TxnHeaderFromString(s): return TxnHeader(*struct.unpack(TRANS_HDR, s)) class TxnHeader(object): """Header for a transaction record.""" __slots__ = ("tid", "tlen", "status", "user", "descr", "ext", "ulen", "dlen", "elen") def __init__(self, tid, tlen, status, ulen, dlen, elen): self.tid = tid self.tlen = tlen self.status = status self.ulen = ulen self.dlen = dlen self.elen = elen assert elen >= 0 def asString(self): s = struct.pack(TRANS_HDR, self.tid, self.tlen, self.status, self.ulen, self.dlen, self.elen) return "".join(map(str, [s, self.user, self.descr, self.ext])) def headerlen(self): return TRANS_HDR_LEN + self.ulen + self.dlen + self.elen Zope-2.8.7-final/lib/python/ZODB/FileStorage/__init__.py0000664000076400007640000000020710435502423021677 0ustar ajungajung# this is a package from ZODB.FileStorage.FileStorage \ import FileStorage, RecordIterator, FileIterator, Record, packed_version Zope-2.8.7-final/lib/python/ZODB/FileStorage/fspack.py0000664000076400007640000006211410435502423021414 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """FileStorage helper to perform pack. A storage contains an ordered set of object revisions. When a storage is packed, object revisions that are not reachable as of the pack time are deleted. The notion of reachability is complicated by backpointers -- object revisions that point to earlier revisions of the same object. An object revisions is reachable at a certain time if it is reachable from the revision of the root at that time or if it is reachable from a backpointer after that time. """ import os from ZODB.serialize import referencesf from ZODB.utils import p64, u64, z64 from ZODB.fsIndex import fsIndex from ZODB.FileStorage.format \ import FileStorageFormatter, CorruptedDataError, DataHeader, \ TRANS_HDR_LEN class DataCopier(FileStorageFormatter): """Mixin class for copying transactions into a storage. The restore() and pack() methods share a need to copy data records and update pointers to data in earlier transaction records. This class provides the shared logic. The mixin extends the FileStorageFormatter with a copy() method. It also requires that the concrete class provides the following attributes: _file -- file with earlier destination data _tfile -- destination file for copied data _pos -- file pos of destination transaction _tindex -- maps oid to data record file pos _tvindex -- maps version name to data record file pos _tindex and _tvindex are updated by copy(). The copy() method does not do any locking. """ def _txn_find(self, tid, stop_at_pack): # _pos always points just past the last transaction pos = self._pos while pos > 4: self._file.seek(pos - 8) pos = pos - u64(self._file.read(8)) - 8 self._file.seek(pos) h = self._file.read(TRANS_HDR_LEN) _tid = h[:8] if _tid == tid: return pos if stop_at_pack: if h[16] == 'p': break raise UndoError(None, "Invalid transaction id") def _data_find(self, tpos, oid, data): # Return backpointer for oid. Must call with the lock held. # This is a file offset to oid's data record if found, else 0. # The data records in the transaction at tpos are searched for oid. # If a data record for oid isn't found, returns 0. # Else if oid's data record contains a backpointer, that # backpointer is returned. # Else oid's data record contains the data, and the file offset of # oid's data record is returned. This data record should contain # a pickle identical to the 'data' argument. # Unclear: If the length of the stored data doesn't match len(data), # an exception is raised. If the lengths match but the data isn't # the same, 0 is returned. Why the discrepancy? h = self._read_txn_header(tpos) tend = tpos + h.tlen pos = self._file.tell() while pos < tend: h = self._read_data_header(pos) if h.oid == oid: # Make sure this looks like the right data record if h.plen == 0: # This is also a backpointer. Gotta trust it. return pos if h.plen != len(data): # The expected data doesn't match what's in the # backpointer. Something is wrong. error("Mismatch between data and backpointer at %d", pos) return 0 _data = self._file.read(h.plen) if data != _data: return 0 return pos pos += h.recordlen() return 0 def _restore_pnv(self, oid, prev, version, bp): # Find a valid pnv (previous non-version) pointer for this version. # If there is no previous record, there can't be a pnv. if not prev: return None pnv = None h = self._read_data_header(prev, oid) # If the previous record is for a version, it must have # a valid pnv. if h.version: return h.pnv elif bp: # Unclear: Not sure the following is always true: # The previous record is not for this version, yet we # have a backpointer to it. The current record must # be an undo of an abort or commit, so the backpointer # must be to a version record with a pnv. h2 = self._read_data_header(bp, oid) if h2.version: return h2.pnv else: warn("restore could not find previous non-version data " "at %d or %d", prev, bp) return None def _resolve_backpointer(self, prev_txn, oid, data): prev_pos = 0 if prev_txn is not None: prev_txn_pos = self._txn_find(prev_txn, 0) if prev_txn_pos: prev_pos = self._data_find(prev_txn_pos, oid, data) return prev_pos def copy(self, oid, serial, data, version, prev_txn, txnpos, datapos): prev_pos = self._resolve_backpointer(prev_txn, oid, data) old = self._index.get(oid, 0) # Calculate the pos the record will have in the storage. here = datapos # And update the temp file index self._tindex[oid] = here if prev_pos: # If there is a valid prev_pos, don't write data. data = None if data is None: dlen = 0 else: dlen = len(data) # Write the recovery data record h = DataHeader(oid, serial, old, txnpos, len(version), dlen) if version: h.version = version pnv = self._restore_pnv(oid, old, version, prev_pos) if pnv is not None: h.pnv = pnv else: h.pnv = old # Link to the last record for this version h.vprev = self._tvindex.get(version, 0) if not h.vprev: h.vprev = self._vindex.get(version, 0) self._tvindex[version] = here self._tfile.write(h.asString()) # Write the data or a backpointer if data is None: if prev_pos: self._tfile.write(p64(prev_pos)) else: # Write a zero backpointer, which indicates an # un-creation transaction. self._tfile.write(z64) else: self._tfile.write(data) class GC(FileStorageFormatter): def __init__(self, file, eof, packtime): self._file = file self._name = file.name self.eof = eof self.packtime = packtime # packpos: position of first txn header after pack time self.packpos = None self.oid2curpos = fsIndex() # maps oid to current data record position self.oid2verpos = fsIndex() # maps oid to current version data # The set of reachable revisions of each object. # # This set as managed using two data structures. The first is # an fsIndex mapping oids to one data record pos. Since only # a few objects will have more than one revision, we use this # efficient data structure to handle the common case. The # second is a dictionary mapping objects to lists of # positions; it is used to handle the same number of objects # for which we must keep multiple revisions. self.reachable = fsIndex() self.reach_ex = {} # keep ltid for consistency checks during initial scan self.ltid = z64 def isReachable(self, oid, pos): """Return 1 if revision of `oid` at `pos` is reachable.""" rpos = self.reachable.get(oid) if rpos is None: return 0 if rpos == pos: return 1 return pos in self.reach_ex.get(oid, []) def findReachable(self): self.buildPackIndex() self.findReachableAtPacktime([z64]) self.findReachableFromFuture() # These mappings are no longer needed and may consume a lot # of space. del self.oid2verpos del self.oid2curpos def buildPackIndex(self): pos = 4L # We make the initial assumption that the database has been # packed before and set unpacked to True only after seeing the # first record with a status == " ". If we get to the packtime # and unpacked is still False, we need to watch for a redundant # pack. unpacked = False while pos < self.eof: th = self._read_txn_header(pos) if th.tid > self.packtime: break self.checkTxn(th, pos) if th.status != "p": unpacked = True tpos = pos end = pos + th.tlen pos += th.headerlen() while pos < end: dh = self._read_data_header(pos) self.checkData(th, tpos, dh, pos) if dh.version: self.oid2verpos[dh.oid] = pos else: self.oid2curpos[dh.oid] = pos pos += dh.recordlen() tlen = self._read_num(pos) if tlen != th.tlen: self.fail(pos, "redundant transaction length does not " "match initial transaction length: %d != %d", tlen, th.tlen) pos += 8 self.packpos = pos if unpacked: return # check for a redundant pack. If the first record following # the newly computed packpos has status 'p', then it was # packed earlier and the current pack is redudant. try: th = self._read_txn_header(pos) except CorruptedDataError, err: if err.buf != "": raise if th.status == 'p': # Delayed import to cope with circular imports. # TODO: put exceptions in a separate module. from ZODB.FileStorage.FileStorage import RedundantPackWarning raise RedundantPackWarning( "The database has already been packed to a later time" " or no changes have been made since the last pack") def findReachableAtPacktime(self, roots): """Mark all objects reachable from the oids in roots as reachable.""" todo = list(roots) while todo: oid = todo.pop() if self.reachable.has_key(oid): continue L = [] pos = self.oid2curpos.get(oid) if pos is not None: L.append(pos) todo.extend(self.findrefs(pos)) pos = self.oid2verpos.get(oid) if pos is not None: L.append(pos) todo.extend(self.findrefs(pos)) if not L: continue pos = L.pop() self.reachable[oid] = pos if L: self.reach_ex[oid] = L def findReachableFromFuture(self): # In this pass, the roots are positions of object revisions. # We add a pos to extra_roots when there is a backpointer to a # revision that was not current at the packtime. The # non-current revision could refer to objects that were # otherwise unreachable at the packtime. extra_roots = [] pos = self.packpos while pos < self.eof: th = self._read_txn_header(pos) self.checkTxn(th, pos) tpos = pos end = pos + th.tlen pos += th.headerlen() while pos < end: dh = self._read_data_header(pos) self.checkData(th, tpos, dh, pos) if dh.back and dh.back < self.packpos: if self.reachable.has_key(dh.oid): L = self.reach_ex.setdefault(dh.oid, []) if dh.back not in L: L.append(dh.back) extra_roots.append(dh.back) else: self.reachable[dh.oid] = dh.back if dh.version and dh.pnv: if self.reachable.has_key(dh.oid): L = self.reach_ex.setdefault(dh.oid, []) if dh.pnv not in L: L.append(dh.pnv) extra_roots.append(dh.pnv) else: self.reachable[dh.oid] = dh.back pos += dh.recordlen() tlen = self._read_num(pos) if tlen != th.tlen: self.fail(pos, "redundant transaction length does not " "match initial transaction length: %d != %d", tlen, th.tlen) pos += 8 for pos in extra_roots: refs = self.findrefs(pos) self.findReachableAtPacktime(refs) def findrefs(self, pos): """Return a list of oids referenced as of packtime.""" dh = self._read_data_header(pos) # Chase backpointers until we get to the record with the refs while dh.back: dh = self._read_data_header(dh.back) if dh.plen: return referencesf(self._file.read(dh.plen)) else: return [] class PackCopier(DataCopier): # PackCopier has to cope with _file and _tfile being the # same file. The copy() implementation is written assuming # that they are different, so that using one object doesn't # mess up the file pointer for the other object. # PackCopier overrides _resolve_backpointer() and _restore_pnv() # to guarantee that they keep the file pointer for _tfile in # the right place. def __init__(self, f, index, vindex, tindex, tvindex): self._file = f self._tfile = f self._index = index self._vindex = vindex self._tindex = tindex self._tvindex = tvindex self._pos = None def setTxnPos(self, pos): self._pos = pos def _resolve_backpointer(self, prev_txn, oid, data): pos = self._tfile.tell() try: return DataCopier._resolve_backpointer(self, prev_txn, oid, data) finally: self._tfile.seek(pos) def _restore_pnv(self, oid, prev, version, bp): pos = self._tfile.tell() try: return DataCopier._restore_pnv(self, oid, prev, version, bp) finally: self._tfile.seek(pos) class FileStoragePacker(FileStorageFormatter): # path is the storage file path. # stop is the pack time, as a TimeStamp. # la and lr are the acquire() and release() methods of the storage's lock. # cla and clr similarly, for the storage's commit lock. # current_size is the storage's _pos. All valid data at the start # lives before that offset (there may be a checkpoint transaction in # progress after it). def __init__(self, path, stop, la, lr, cla, clr, current_size): self._name = path # We open our own handle on the storage so that much of pack can # proceed in parallel. It's important to close this file at every # return point, else on Windows the caller won't be able to rename # or remove the storage file. self._file = open(path, "rb") self._path = path self._stop = stop self.locked = 0 self.file_end = current_size self.gc = GC(self._file, self.file_end, self._stop) # The packer needs to acquire the parent's commit lock # during the copying stage, so the two sets of lock acquire # and release methods are passed to the constructor. self._lock_acquire = la self._lock_release = lr self._commit_lock_acquire = cla self._commit_lock_release = clr # The packer will use several indexes. # index: oid -> pos # vindex: version -> pos # tindex: oid -> pos, for current txn # tvindex: version -> pos, for current txn # oid2tid: not used by the packer self.index = fsIndex() self.vindex = {} self.tindex = {} self.tvindex = {} self.oid2tid = {} self.toid2tid = {} self.toid2tid_delete = {} # Index for non-version data. This is a temporary structure # to reduce I/O during packing self.nvindex = fsIndex() def pack(self): # Pack copies all data reachable at the pack time or later. # # Copying occurs in two phases. In the first phase, txns # before the pack time are copied if the contain any reachable # data. In the second phase, all txns after the pack time # are copied. # # Txn and data records contain pointers to previous records. # Because these pointers are stored as file offsets, they # must be updated when we copy data. # TODO: Should add sanity checking to pack. self.gc.findReachable() # Setup the destination file and copy the metadata. # TODO: rename from _tfile to something clearer. self._tfile = open(self._name + ".pack", "w+b") self._file.seek(0) self._tfile.write(self._file.read(self._metadata_size)) self._copier = PackCopier(self._tfile, self.index, self.vindex, self.tindex, self.tvindex) ipos, opos = self.copyToPacktime() assert ipos == self.gc.packpos if ipos == opos: # pack didn't free any data. there's no point in continuing. self._tfile.close() self._file.close() os.remove(self._name + ".pack") return None self._commit_lock_acquire() self.locked = 1 self._lock_acquire() try: # Re-open the file in unbuffered mode. # The main thread may write new transactions to the file, # which creates the possibility that we will read a status # 'c' transaction into the pack thread's stdio buffer even # though we're acquiring the commit lock. Transactions # can still be in progress throughout much of packing, and # are written to the same physical file but via a distinct # Python file object. The code used to leave off the # trailing 0 argument, and then on every platform except # native Windows it was observed that we could read stale # data from the tail end of the file. self._file.close() # else self.gc keeps the original alive & open self._file = open(self._path, "rb", 0) self._file.seek(0, 2) self.file_end = self._file.tell() finally: self._lock_release() if ipos < self.file_end: self.copyRest(ipos) # OK, we've copied everything. Now we need to wrap things up. pos = self._tfile.tell() self._tfile.flush() self._tfile.close() self._file.close() return pos def copyToPacktime(self): offset = 0L # the amount of space freed by packing pos = self._metadata_size new_pos = pos while pos < self.gc.packpos: th = self._read_txn_header(pos) new_tpos, pos = self.copyDataRecords(pos, th) if new_tpos: new_pos = self._tfile.tell() + 8 tlen = new_pos - new_tpos - 8 # Update the transaction length self._tfile.seek(new_tpos + 8) self._tfile.write(p64(tlen)) self._tfile.seek(new_pos - 8) self._tfile.write(p64(tlen)) tlen = self._read_num(pos) if tlen != th.tlen: self.fail(pos, "redundant transaction length does not " "match initial transaction length: %d != %d", tlen, th.tlen) pos += 8 return pos, new_pos def fetchBackpointer(self, oid, back): """Return data and refs backpointer `back` to object `oid. If `back` is 0 or ultimately resolves to 0, return None and None. In this case, the transaction undoes the object creation. """ if back == 0: return None data, tid = self._loadBackTxn(oid, back, 0) return data def copyDataRecords(self, pos, th): """Copy any current data records between pos and tend. Returns position of txn header in output file and position of next record in the input file. If any data records are copied, also write txn header (th). """ copy = 0 new_tpos = 0L tend = pos + th.tlen pos += th.headerlen() while pos < tend: h = self._read_data_header(pos) if not self.gc.isReachable(h.oid, pos): pos += h.recordlen() continue pos += h.recordlen() # If we are going to copy any data, we need to copy # the transaction header. Note that we will need to # patch up the transaction length when we are done. if not copy: th.status = "p" s = th.asString() new_tpos = self._tfile.tell() self._tfile.write(s) new_pos = new_tpos + len(s) copy = 1 if h.plen: data = self._file.read(h.plen) else: # If a current record has a backpointer, fetch # refs and data from the backpointer. We need # to write the data in the new record. data = self.fetchBackpointer(h.oid, h.back) self.writePackedDataRecord(h, data, new_tpos) new_pos = self._tfile.tell() return new_tpos, pos def writePackedDataRecord(self, h, data, new_tpos): # Update the header to reflect current information, then write # it to the output file. if data is None: data = "" h.prev = 0 h.back = 0 h.plen = len(data) h.tloc = new_tpos pos = self._tfile.tell() if h.version: h.pnv = self.index.get(h.oid, 0) h.vprev = self.vindex.get(h.version, 0) self.vindex[h.version] = pos self.index[h.oid] = pos if h.version: self.vindex[h.version] = pos self._tfile.write(h.asString()) self._tfile.write(data) if not data: # Packed records never have backpointers (?). # If there is no data, write a z64 backpointer. # This is a George Bailey event. self._tfile.write(z64) def copyRest(self, ipos): # After the pack time, all data records are copied. # Copy one txn at a time, using copy() for data. # Release the commit lock every 20 copies self._lock_counter = 0 try: while 1: ipos = self.copyOne(ipos) except CorruptedDataError, err: # The last call to copyOne() will raise # CorruptedDataError, because it will attempt to read past # the end of the file. Double-check that the exception # occurred for this reason. self._file.seek(0, 2) endpos = self._file.tell() if endpos != err.pos: raise def copyOne(self, ipos): # The call below will raise CorruptedDataError at EOF. th = self._read_txn_header(ipos) self._lock_counter += 1 if self._lock_counter % 20 == 0: self._commit_lock_release() pos = self._tfile.tell() self._copier.setTxnPos(pos) self._tfile.write(th.asString()) tend = ipos + th.tlen ipos += th.headerlen() while ipos < tend: h = self._read_data_header(ipos) ipos += h.recordlen() prev_txn = None if h.plen: data = self._file.read(h.plen) else: data = self.fetchBackpointer(h.oid, h.back) if h.back: prev_txn = self.getTxnFromData(h.oid, h.back) self._copier.copy(h.oid, h.tid, data, h.version, prev_txn, pos, self._tfile.tell()) tlen = self._tfile.tell() - pos assert tlen == th.tlen self._tfile.write(p64(tlen)) ipos += 8 self.index.update(self.tindex) self.tindex.clear() self.vindex.update(self.tvindex) self.tvindex.clear() if self._lock_counter % 20 == 0: self._commit_lock_acquire() return ipos Zope-2.8.7-final/lib/python/ZODB/FileStorage/fsoids.py0000664000076400007640000001755710435502423021447 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import ZODB.FileStorage from ZODB.utils import get_pickle_metadata from ZODB.utils import p64, oid_repr, tid_repr, get_refs from ZODB.TimeStamp import TimeStamp # Extract module.class string from pickle. def get_class(pickle): return "%s.%s" % get_pickle_metadata(pickle) # Shorten a string for display. def shorten(s, size=50): if len(s) <= size: return s # Stick ... in the middle. navail = size - 5 nleading = navail // 2 ntrailing = size - nleading return s[:nleading] + " ... " + s[-ntrailing:] class Tracer(object): """Trace all occurrences of a set of oids in a FileStorage. Create passing a path to an existing FileStorage. Call register_oids(oid, ...) one or more times to specify which oids to investigate. Call run() to do the analysis. This isn't swift -- it has to read every byte in the database, in order to find all references. Call report() to display the results. """ def __init__(self, path): import os if not os.path.isfile(path): raise ValueError("must specify an existing FileStorage") self.path = path # Map an interesting tid to (status, user, description, pos). self.tid2info = {} # List of messages. Each is a tuple of the form # (oid, tid, string) # The order in the tuple is important, because it defines the # sort order for grouping. self.msgs = [] # The set of interesting oids, specified by register_oid() calls. # Maps oid to # of revisions. self.oids = {} # Maps interesting oid to its module.class name. If a creation # record for an interesting oid is never seen, it won't appear # in this mapping. self.oid2name = {} def register_oids(self, *oids): """ Declare that oids (0 or more) are "interesting". An oid can be given as a native 8-byte string, or as an integer. Info will be gathered about all appearances of this oid in the entire database, including references. """ for oid in oids: if isinstance(oid, str): assert len(oid) == 8 else: oid = p64(oid) self.oids[oid] = 0 # 0 revisions seen so far def _msg(self, oid, tid, *args): args = map(str, args) self.msgs.append( (oid, tid, ' '.join(args)) ) self._produced_msg = True def report(self): """Show all msgs, grouped by oid and sub-grouped by tid.""" msgs = self.msgs oids = self.oids oid2name = self.oid2name # First determine which oids weren't seen at all, and synthesize msgs # for them. NOT_SEEN = "this oid was not defined (no data record for it found)" for oid in oids: if oid not in oid2name: msgs.append( (oid, None, NOT_SEEN) ) msgs.sort() # oids are primary key, tids secondary current_oid = current_tid = None for oid, tid, msg in msgs: if oid != current_oid: nrev = oids[oid] revision = "revision" + (nrev != 1 and 's' or '') name = oid2name.get(oid, "") print "oid", oid_repr(oid), name, nrev, revision current_oid = oid current_tid = None if msg is NOT_SEEN: assert tid is None print " ", msg continue if tid != current_tid: current_tid = tid status, user, description, pos = self.tid2info[tid] print " tid %s offset=%d %s" % (tid_repr(tid), pos, TimeStamp(tid)) print " tid user=%r" % shorten(user) print " tid description=%r" % shorten(description) print " ", msg # Do the analysis. def run(self): """Find all occurrences of the registered oids in the database.""" # Maps oid of a reference to its module.class name. self._ref2name = {} for txn in ZODB.FileStorage.FileIterator(self.path): self._check_trec(txn) # Process next transaction record. def _check_trec(self, txn): # txn has members tid, status, user, description, # _extension, _pos, _tend, _file, _tpos self._produced_msg = False # Map and list for save data records for current transaction. self._records_map = {} self._records = [] for drec in txn: self._save_references(drec) for drec in self._records: self._check_drec(drec) if self._produced_msg: # Copy txn info for later output. self.tid2info[txn.tid] = (txn.status, txn.user, txn.description, txn._tpos) def _save_references(self, drec): # drec has members oid, tid, version, data, data_txn tid, oid, pick, pos = drec.tid, drec.oid, drec.data, drec.pos if pick: if oid in self.oids: klass = get_class(pick) self._msg(oid, tid, "new revision", klass, "at", pos) self.oids[oid] += 1 self.oid2name[oid] = self._ref2name[oid] = klass self._records_map[oid] = drec self._records.append(drec) elif oid in self.oids: # Or maybe it's a version abort. self._msg(oid, tid, "creation undo at", pos) # Process next data record. If a message is produced, self._produced_msg # will be set True. def _check_drec(self, drec): # drec has members oid, tid, version, data, data_txn tid, oid, pick, pos = drec.tid, drec.oid, drec.data, drec.pos ref2name = self._ref2name ref2name_get = ref2name.get records_map_get = self._records_map.get if pick: oid_in_oids = oid in self.oids for ref, klass in get_refs(pick): if ref in self.oids: oidclass = ref2name_get(oid, None) if oidclass is None: ref2name[oid] = oidclass = get_class(pick) self._msg(ref, tid, "referenced by", oid_repr(oid), oidclass, "at", pos) if oid_in_oids: if klass is None: klass = ref2name_get(ref, None) if klass is None: r = records_map_get(ref, None) # For save memory we only save references # seen in one transaction with interesting # objects changes. So in some circumstances # we may still got "" class name. if r is None: klass = "" else: ref2name[ref] = klass = get_class(r.data) elif isinstance(klass, tuple): ref2name[ref] = klass = "%s.%s" % klass self._msg(oid, tid, "references", oid_repr(ref), klass, "at", pos) Zope-2.8.7-final/lib/python/ZODB/persistentclass.py0000664000076400007640000001507610435502423021174 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Persistent Class Support $Id: persistentclass.py 29891 2005-04-06 22:35:05Z tim_one $ """ # Notes: # # Persistent classes are non-ghostable. This has some interesting # ramifications: # # - When an object is invalidated, it must reload it's state # # - When an object is loaded from the database, it's state must be # loaded. Unfortunately, there isn't a clear signal when an object is # loaded from the database. This should probably be fixed. # # In the mean time, we need to infer. This should be viewed as a # short term hack. # # Here's the strategy we'll use: # # - We'll have a need to be loaded flag that we'll set in # __new__, through an extra argument. # # - When setting _p_oid and _p_jar, if both are set and we need to be # loaded, then we'll load out state. # # - We'll use _p_changed is None to indicate that we're in this state. # class _p_DataDescr(object): # Descr used as base for _p_ data. Data are stored in # _p_class_dict. def __init__(self, name): self.__name__ = name def __get__(self, inst, cls): if inst is None: return self if '__global_persistent_class_not_stored_in_DB__' in inst.__dict__: raise AttributeError, self.__name__ return inst._p_class_dict.get(self.__name__) def __set__(self, inst, v): inst._p_class_dict[self.__name__] = v def __delete__(self, inst): raise AttributeError, self.__name__ class _p_oid_or_jar_Descr(_p_DataDescr): # Special descr for _p_oid and _p_jar that loads # state when set if both are set and and _p_changed is None # # See notes above def __set__(self, inst, v): get = inst._p_class_dict.get if v == get(self.__name__): return inst._p_class_dict[self.__name__] = v jar = get('_p_jar') if (jar is not None and get('_p_oid') is not None and get('_p_changed') is None ): jar.setstate(inst) class _p_ChangedDescr(object): # descriptor to handle special weird emantics of _p_changed def __get__(self, inst, cls): if inst is None: return self return inst._p_class_dict['_p_changed'] def __set__(self, inst, v): if v is None: return inst._p_class_dict['_p_changed'] = bool(v) def __delete__(self, inst): inst._p_invalidate() class _p_MethodDescr(object): """Provide unassignable class attributes """ def __init__(self, func): self.func = func def __get__(self, inst, cls): if inst is None: return cls return self.func.__get__(inst, cls) def __set__(self, inst, v): raise AttributeError, self.__name__ def __delete__(self, inst): raise AttributeError, self.__name__ special_class_descrs = '__dict__', '__weakref__' class PersistentMetaClass(type): _p_jar = _p_oid_or_jar_Descr('_p_jar') _p_oid = _p_oid_or_jar_Descr('_p_oid') _p_changed = _p_ChangedDescr() _p_serial = _p_DataDescr('_p_serial') def __new__(self, name, bases, cdict, _p_changed=False): cdict = dict([(k, v) for (k, v) in cdict.items() if not k.startswith('_p_')]) cdict['_p_class_dict'] = {'_p_changed': _p_changed} return super(PersistentMetaClass, self).__new__( self, name, bases, cdict) def __getnewargs__(self): return self.__name__, self.__bases__, {}, None __getnewargs__ = _p_MethodDescr(__getnewargs__) def _p_maybeupdate(self, name): get = self._p_class_dict.get data_manager = get('_p_jar') if ( (data_manager is not None) and (get('_p_oid') is not None) and (get('_p_changed') == False) ): self._p_changed = True data_manager.register(self) def __setattr__(self, name, v): if not ((name.startswith('_p_') or name.startswith('_v'))): self._p_maybeupdate(name) super(PersistentMetaClass, self).__setattr__(name, v) def __delattr__(self, name): if not ((name.startswith('_p_') or name.startswith('_v'))): self._p_maybeupdate(name) super(PersistentMetaClass, self).__delattr__(name) def _p_deactivate(self): # persistent classes can't be ghosts pass _p_deactivate = _p_MethodDescr(_p_deactivate) def _p_invalidate(self): # reset state self._p_class_dict['_p_changed'] = None self._p_jar.setstate(self) _p_invalidate = _p_MethodDescr(_p_invalidate) def __getstate__(self): return (self.__bases__, dict([(k, v) for (k, v) in self.__dict__.items() if not (k.startswith('_p_') or k.startswith('_v_') or k in special_class_descrs ) ]), ) __getstate__ = _p_MethodDescr(__getstate__) def __setstate__(self, state): self.__bases__, cdict = state cdict = dict([(k, v) for (k, v) in cdict.items() if not k.startswith('_p_')]) _p_class_dict = self._p_class_dict self._p_class_dict = {} to_remove = [k for k in self.__dict__ if ((k not in cdict) and (k not in special_class_descrs) and (k != '_p_class_dict') )] for k in to_remove: delattr(self, k) for k, v in cdict.items(): setattr(self, k, v) self._p_class_dict = _p_class_dict self._p_changed = False __setstate__ = _p_MethodDescr(__setstate__) def _p_activate(self): self._p_jar.setstate(self) _p_activate = _p_MethodDescr(_p_activate) Zope-2.8.7-final/lib/python/ZODB/DemoStorage.py0000664000076400007640000004414310435502423020154 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Demo ZODB storage The Demo storage serves two purposes: - Provide an example implementation of a full storage without distracting storage details, - Provide a volatile storage that is useful for giving demonstrations. The demo storage can have a "base" storage that is used in a read-only fashion. The base storage must not not to contain version data. There are three main data structures: _data -- Transaction logging information necessary for undo This is a mapping from transaction id to transaction, where a transaction is simply a 5-tuple: packed, user, description, extension_data, records where extension_data is a dictionary or None and records are the actual records in chronological order. Packed is a flag indicating whethe the transaction has been packed or not _index -- A mapping from oid to record _vindex -- A mapping from version name to version data where version data is a mapping from oid to record A record is a tuple: oid, pre, vdata, p, tid where: oid -- object id pre -- The previous record for this object (or None) vdata -- version data None if not a version, ortherwise: version, non-version-record p -- the pickle data or None tid -- the transaction id that wrote the record The pickle data will be None for a record for an object created in an aborted version. It is instructive to watch what happens to the internal data structures as changes are made. For example, in Zope, you can create an external method:: import Zope2 def info(RESPONSE): RESPONSE['Content-type']= 'text/plain' return Zope2.DB._storage._splat() and call it to monitor the storage. """ import base64, time from ZODB import POSException from ZODB.utils import z64, oid_repr from ZODB.BaseStorage import BaseStorage from persistent.TimeStamp import TimeStamp from cPickle import loads from BTrees import OOBTree class DemoStorage(BaseStorage): def __init__(self, name='Demo Storage', base=None, quota=None): BaseStorage.__init__(self, name, base) # We use a BTree because the items are sorted! self._data = OOBTree.OOBTree() self._index = {} self._vindex = {} self._base = base self._size = 0 self._quota = quota self._ltid = None self._clear_temp() if base is not None and base.versions(): raise POSException.StorageError, ( "Demo base storage has version data") # While we officially don't support wrapping a non-read-only base # storage, it has proved useful for test suites to wrap a ClientStorage # in DemoStorage. The least we can do to help support that case is # to arrange for invalidations to get delivered to the base storage. def registerDB(self, db, limit): if self._base is not None: # delegate self._base.registerDB(db, limit) # When DemoStorage needs to create a new oid, and there is a base # storage, it must use that storage's new_oid() method. Else # DemoStorage may end up assigning "new" oids that are already in use # by the base storage, leading to a variety of "impossible" problems. def new_oid(self): if self._base is None: return BaseStorage.new_oid(self) else: return self._base.new_oid() def __len__(self): base=self._base return (base and len(base) or 0) + len(self._index) def getSize(self): s=100 for tid, (p, u, d, e, t) in self._data.items(): s=s+16+24+12+4+16+len(u)+16+len(d)+16+len(e)+16 for oid, pre, vdata, p, tid in t: s=s+16+24+24+4+4+(p and (16+len(p)) or 4) if vdata: s=s+12+16+len(vdata[0])+4 s=s+16*len(self._index) for v in self._vindex.values(): s=s+32+16*len(v) self._size=s return s def abortVersion(self, src, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) if not src: raise POSException.VersionCommitError("Invalid version") self._lock_acquire() try: v = self._vindex.get(src, None) if not v: return oids = [] for r in v.values(): oid, pre, (version, nv), p, tid = r oids.append(oid) if nv: oid, pre, vdata, p, tid = nv self._tindex.append([oid, r, None, p, self._tid]) else: # effectively, delete the thing self._tindex.append([oid, r, None, None, self._tid]) return self._tid, oids finally: self._lock_release() def commitVersion(self, src, dest, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) if not src: raise POSException.VersionCommitError("Invalid source version") if src == dest: raise POSException.VersionCommitError( "Can't commit to same version: %s" % repr(src)) self._lock_acquire() try: v = self._vindex.get(src) if v is None: return newserial = self._tid tindex = self._tindex oids = [] for r in v.values(): oid, pre, vdata, p, tid = r assert vdata is not None oids.append(oid) if dest: new_vdata = dest, vdata[1] else: new_vdata = None tindex.append([oid, r, new_vdata, p, self._tid]) return self._tid, oids finally: self._lock_release() def loadEx(self, oid, version): self._lock_acquire() try: try: oid, pre, vdata, p, tid = self._index[oid] except KeyError: if self._base: return self._base.load(oid, '') raise KeyError, oid ver = "" if vdata: oversion, nv = vdata if oversion != version: if nv: # Return the current txn's tid with the non-version # data. oid, pre, vdata, p, skiptid = nv else: raise KeyError, oid ver = oversion if p is None: raise KeyError, oid return p, tid, ver finally: self._lock_release() def load(self, oid, version): return self.loadEx(oid, version)[:2] def modifiedInVersion(self, oid): self._lock_acquire() try: try: oid, pre, vdata, p, tid = self._index[oid] if vdata: return vdata[0] return '' except: return '' finally: self._lock_release() def store(self, oid, serial, data, version, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: old = self._index.get(oid, None) if old is None: # Hm, nothing here, check the base version: if self._base: try: p, tid = self._base.load(oid, '') except KeyError: pass else: old = oid, None, None, p, tid nv=None if old: oid, pre, vdata, p, tid = old if vdata: if vdata[0] != version: raise POSException.VersionLockError, oid nv=vdata[1] else: nv=old if serial != tid: raise POSException.ConflictError( oid=oid, serials=(tid, serial), data=data) r = [oid, old, version and (version, nv) or None, data, self._tid] self._tindex.append(r) s=self._tsize s=s+72+(data and (16+len(data)) or 4) if version: s=s+32+len(version) if self._quota is not None and s > self._quota: raise POSException.StorageError, ( '''Quota Exceeded
The maximum quota for this demonstration storage has been exceeded.
Have a nice day.''') finally: self._lock_release() return self._tid def supportsVersions(self): return 1 def _clear_temp(self): self._tindex = [] self._tsize = self._size + 160 def lastTransaction(self): return self._ltid def _begin(self, tid, u, d, e): self._tsize = self._size + 120 + len(u) + len(d) + len(e) def _finish(self, tid, user, desc, ext): self._size = self._tsize self._data[tid] = None, user, desc, ext, tuple(self._tindex) for r in self._tindex: oid, pre, vdata, p, tid = r old = self._index.get(oid) # If the object had version data, remove the version data. if old is not None: oldvdata = old[2] if oldvdata: v = self._vindex[oldvdata[0]] del v[oid] if not v: # If the version info is now empty, remove it. del self._vindex[oldvdata[0]] self._index[oid] = r # If there is version data, then udpate self._vindex, too. if vdata: version = vdata[0] v = self._vindex.get(version) if v is None: v = self._vindex[version] = {} v[oid] = r self._ltid = self._tid def undoLog(self, first, last, filter=None): if last < 0: # abs(last) is an upper bound on the # to return last = first - last self._lock_acquire() try: transactions = self._data.items() pos = len(transactions) r = [] i = 0 while i < last and pos: pos -= 1 tid, (p, u, d, e, t) = transactions[pos] # Bug alert: why do we skip this if the transaction # has been packed? if p: continue d = {'id': base64.encodestring(tid)[:-1], 'time': TimeStamp(tid).timeTime(), 'user_name': u, 'description': d} if e: d.update(loads(e)) if filter is None or filter(d): if i >= first: r.append(d) i += 1 return r finally: self._lock_release() def versionEmpty(self, version): return not self._vindex.get(version, None) def versions(self, max=None): r = [] for v in self._vindex.keys(): if self.versionEmpty(v): continue r.append(v) if max is not None and len(r) >= max: break return r def _build_indexes(self, stop='\377\377\377\377\377\377\377\377'): # Rebuild index structures from transaction data index = {} vindex = {} for tid, (p, u, d, e, t) in self._data.items(): if tid >= stop: break for r in t: oid, pre, vdata, p, tid = r old=index.get(oid, None) if old is not None: oldvdata=old[2] if oldvdata: v=vindex[oldvdata[0]] del v[oid] if not v: del vindex[oldvdata[0]] index[oid]=r if vdata: version=vdata[0] v=vindex.get(version, None) if v is None: v=vindex[version]={} vindex[vdata[0]][oid]=r return index, vindex def pack(self, t, referencesf): # Packing is hard, at least when undo is supported. # Even for a simple storage like this one, packing # is pretty complex. self._lock_acquire() try: stop=`TimeStamp(*time.gmtime(t)[:5]+(t%60,))` # Build indexes up to the pack time: index, vindex = self._build_indexes(stop) # TODO: This packing algorithm is flawed. It ignores # references from non-current records after the pack # time. # Now build an index of *only* those objects reachable # from the root. rootl = [z64] pindex = {} while rootl: oid = rootl.pop() if oid in pindex: continue # Scan non-version pickle for references r = index.get(oid, None) if r is None: if self._base: p, s = self._base.load(oid, '') referencesf(p, rootl) else: pindex[oid] = r oid, pre, vdata, p, tid = r referencesf(p, rootl) if vdata: nv = vdata[1] if nv: oid, pre, vdata, p, tid = nv referencesf(p, rootl) # Now we're ready to do the actual packing. # We'll simply edit the transaction data in place. # We'll defer deleting transactions till the end # to avoid messing up the BTree items. deleted = [] for tid, (p, u, d, e, records) in self._data.items(): if tid >= stop: break o = [] for r in records: c = pindex.get(r[0]) if c is None: # GC this record, no longer referenced continue if c == r: # This is the most recent revision. o.append(r) else: # This record is not the indexed record, # so it may not be current. Let's see. vdata = r[3] if vdata: # Version record are current *only* if they # are indexed continue else: # OK, this isn't a version record, so it may be the # non-version record for the indexed record. vdata = c[3] if vdata: if vdata[1] != r: # This record is not the non-version # record for the indexed record continue else: # The indexed record is not a version record, # so this record can not be the non-version # record for it. continue o.append(r) if o: if len(o) != len(records): self._data[tid] = 1, u, d, e, tuple(o) # Reset data else: deleted.append(tid) # Now delete empty transactions for tid in deleted: del self._data[tid] # Now reset previous pointers for "current" records: for r in pindex.values(): r[1] = None # Previous record if r[2] and r[2][1]: # vdata # If this record contains version data and # non-version data, then clear it out. r[2][1][2] = None # Finally, rebuild indexes from transaction data: self._index, self._vindex = self._build_indexes() finally: self._lock_release() self.getSize() def _splat(self): """Spit out a string showing state. """ o=[] o.append('Transactions:') for tid, (p, u, d, e, t) in self._data.items(): o.append(" %s %s" % (TimeStamp(tid), p)) for r in t: oid, pre, vdata, p, tid = r oid = oid_repr(oid) tid = oid_repr(tid) ## if serial is not None: serial=str(TimeStamp(serial)) pre=id(pre) if vdata and vdata[1]: vdata=vdata[0], id(vdata[1]) if p: p='' o.append(' %s: %s' % (id(r), `(oid, pre, vdata, p, tid)`)) o.append('\nIndex:') items=self._index.items() items.sort() for oid, r in items: if r: r=id(r) o.append(' %s: %s' % (oid_repr(oid), r)) o.append('\nVersion Index:') items=self._vindex.items() items.sort() for version, v in items: o.append(' '+version) vitems=v.items() vitems.sort() for oid, r in vitems: if r: r=id(r) o.append(' %s: %s' % (oid_repr(oid), r)) return '\n'.join(o) Zope-2.8.7-final/lib/python/ZODB/__init__.py0000664000076400007640000000244010435502423017474 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # The next line must use double quotes, so release.py recognizes it. __version__ = "3.4.3" import sys import __builtin__ from persistent import TimeStamp from persistent import list from persistent import mapping # Backward compat for old imports. sys.modules['ZODB.TimeStamp'] = sys.modules['persistent.TimeStamp'] sys.modules['ZODB.PersistentMapping'] = sys.modules['persistent.mapping'] sys.modules['ZODB.PersistentList'] = sys.modules['persistent.list'] del mapping, list, sys from DB import DB # TODO: get_transaction() scheduled to go away in ZODB 3.6. from transaction import get_transaction __builtin__.get_transaction = get_transaction del __builtin__ Zope-2.8.7-final/lib/python/ZODB/fsIndex.py0000664000076400007640000001351410435502423017341 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Implement an OID to File-position (long integer) mapping.""" # To save space, we do two things: # # 1. We split the keys (OIDS) into 6-byte prefixes and 2-byte suffixes. # We use the prefixes as keys in a mapping from prefix to mappings # of suffix to data: # # data is {prefix -> {suffix -> data}} # # 2. We limit the data size to 48 bits. This should allow databases # as large as 256 terabytes. # # Most of the space is consumed by items in the mappings from 2-byte # suffix to 6-byte data. This should reduce the overall memory usage to # 8-16 bytes per OID. # # Since the mapping from suffix to data contains at most 256 entries, # we use a BTree bucket instead of a full BTree to store the results. # # We use p64 to convert integers to 8-byte strings and lop off the two # high-order bytes when saving. On loading data, we add the leading # bytes back before using u64 to convert the data back to (long) # integers. import struct from BTrees._fsBTree import fsBucket from BTrees.OOBTree import OOBTree # convert between numbers and six-byte strings def num2str(n): return struct.pack(">Q", n)[2:] def str2num(s): return struct.unpack(">Q", "\000\000" + s)[0] def prefix_plus_one(s): num = str2num(s) return num2str(num + 1) def prefix_minus_one(s): num = str2num(s) return num2str(num - 1) class fsIndex(object): def __init__(self): self._data = OOBTree() def __getitem__(self, key): return str2num(self._data[key[:6]][key[6:]]) def get(self, key, default=None): tree = self._data.get(key[:6], default) if tree is default: return default v = tree.get(key[6:], default) if v is default: return default return str2num(v) def __setitem__(self, key, value): value = num2str(value) treekey = key[:6] tree = self._data.get(treekey) if tree is None: tree = fsBucket() self._data[treekey] = tree tree[key[6:]] = value def __len__(self): r = 0 for tree in self._data.itervalues(): r += len(tree) return r def update(self, mapping): for k, v in mapping.items(): self[k] = v def has_key(self, key): v = self.get(key, self) return v is not self def __contains__(self, key): tree = self._data.get(key[:6]) if tree is None: return False v = tree.get(key[6:], None) if v is None: return False return True def clear(self): self._data.clear() def __iter__(self): for prefix, tree in self._data.iteritems(): for suffix in tree: yield prefix + suffix iterkeys = __iter__ def keys(self): return list(self.iterkeys()) def iteritems(self): for prefix, tree in self._data.iteritems(): for suffix, value in tree.iteritems(): yield (prefix + suffix, str2num(value)) def items(self): return list(self.iteritems()) def itervalues(self): for tree in self._data.itervalues(): for value in tree.itervalues(): yield str2num(value) def values(self): return list(self.itervalues()) # Comment below applies for the following minKey and maxKey methods # # Obscure: what if `tree` is actually empty? We're relying here on # that this class doesn't implement __delitem__: once a key gets # into an fsIndex, the only way it can go away is by invoking # clear(). Therefore nothing in _data.values() is ever empty. # # Note that because `tree` is an fsBTree, its minKey()/maxKey() methods are # very efficient. def minKey(self, key=None): if key is None: smallest_prefix = self._data.minKey() else: smallest_prefix = self._data.minKey(key[:6]) tree = self._data[smallest_prefix] assert tree if key is None: smallest_suffix = tree.minKey() else: try: smallest_suffix = tree.minKey(key[6:]) except ValueError: # 'empty tree' (no suffix >= arg) next_prefix = prefix_plus_one(smallest_prefix) smallest_prefix = self._data.minKey(next_prefix) tree = self._data[smallest_prefix] assert tree smallest_suffix = tree.minKey() return smallest_prefix + smallest_suffix def maxKey(self, key=None): if key is None: biggest_prefix = self._data.maxKey() else: biggest_prefix = self._data.maxKey(key[:6]) tree = self._data[biggest_prefix] assert tree if key is None: biggest_suffix = tree.maxKey() else: try: biggest_suffix = tree.maxKey(key[6:]) except ValueError: # 'empty tree' (no suffix <= arg) next_prefix = prefix_minus_one(biggest_prefix) biggest_prefix = self._data.maxKey(next_prefix) tree = self._data[biggest_prefix] assert tree biggest_suffix = tree.maxKey() return biggest_prefix + biggest_suffix Zope-2.8.7-final/lib/python/ZODB/dbmStorage.py0000664000076400007640000000657410435502423020040 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Very Simple dbm-based ZODB storage This storage provides for use of dbm files as storages that don't support versions or Undo. This may be useful when implementing objects like hit counters that don't need or want to participate in undo or versions. """ from ZODB.utils import z64 from MappingStorage import MappingStorage from BaseStorage import BaseStorage import anydbm, os class anydbmStorage(MappingStorage): def __init__(self, filename, flag='r', mode=0666): BaseStorage.__init__(self, filename) self._index=anydbm.open(filename, flag, mode) self._tindex=[] keys=self._index.keys() if keys: self._oid=max(keys) def getSize(self): # This is a little iffy, since we aren't entirely sure what the file is self._lock_acquire() try: try: return (os.stat(self.__name__+'.data')[6] + os.stat(self.__name__+'.dir')[6] ) except: try: return os.stat(self.__name__)[6] except: return 0 finally: self._lock_release() class gdbmStorage(anydbmStorage): def __init__(self, filename, flag='r', mode=0666): BaseStorage.__init__(self, filename) import gdbm self._index=index=gdbm.open(filename, flag[:1]+'f', mode) self._tindex=[] m=z64 oid=index.firstkey() while oid != None: m=max(m, oid) oid=index.nextkey(oid) self._oid=m def getSize(self): self._lock_acquire() try: return os.stat(self.__name__)[6] finally: self._lock_release() def pack(self, t, referencesf): self._lock_acquire() try: # Build an index of *only* those objects reachable # from the root. index=self._index rootl=[z64] pop=rootl.pop pindex={} referenced=pindex.has_key while rootl: oid=pop() if referenced(oid): continue # Scan non-version pickle for references r=index[oid] pindex[oid]=r p=r[8:] referencesf(p, rootl) # Now delete any unreferenced entries: deleted=[] oid=index.firstkey() while oid != None: if not referenced(oid): deleted.append(oid) oid=index.nextkey(oid) pindex=referenced=None for oid in deleted: del index[oid] index.sync() index.reorganize() finally: self._lock_release() def _finish(self, tid, user, desc, ext): index=self._index for oid, p in self._tindex: index[oid]=p index.sync() Zope-2.8.7-final/lib/python/ZODB/Connection.py0000664000076400007640000013212510435502423020040 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Database connection support $Id: Connection.py 40783 2005-12-14 21:43:26Z tim_one $""" import logging import sys import tempfile import threading import warnings from time import time from persistent import PickleCache # interfaces from persistent.interfaces import IPersistentDataManager from ZODB.interfaces import IConnection from transaction.interfaces import ISavepointDataManager from transaction.interfaces import IDataManagerSavepoint from transaction.interfaces import ISynchronizer from zope.interface import implements import transaction from ZODB.ConflictResolution import ResolvedSerial from ZODB.ExportImport import ExportImport from ZODB import POSException from ZODB.POSException import InvalidObjectReference, ConnectionStateError from ZODB.POSException import ConflictError, ReadConflictError from ZODB.serialize import ObjectWriter, ObjectReader, myhasattr from ZODB.utils import DEPRECATED_ARGUMENT, deprecated36 from ZODB.utils import p64, u64, z64, oid_repr, positive_id global_reset_counter = 0 def resetCaches(): """Causes all connection caches to be reset as connections are reopened. Zope's refresh feature uses this. When you reload Python modules, instances of classes continue to use the old class definitions. To use the new code immediately, the refresh feature asks ZODB to clear caches by calling resetCaches(). When the instances are loaded by subsequent connections, they will use the new class definitions. """ global global_reset_counter global_reset_counter += 1 class Connection(ExportImport, object): """Connection to ZODB for loading and storing objects.""" implements(IConnection, ISavepointDataManager, IPersistentDataManager, ISynchronizer) _storage = _normal_storage = _savepoint_storage = None _code_timestamp = 0 ########################################################################## # Connection methods, ZODB.IConnection def __init__(self, version='', cache_size=400, cache_deactivate_after=None, mvcc=True, txn_mgr=DEPRECATED_ARGUMENT, transaction_manager=None, synch=True): """Create a new Connection.""" if txn_mgr is not DEPRECATED_ARGUMENT: deprecated36("use transaction_manager= instead of txn_mgr=") if transaction_manager is None: transaction_manager = txn_mgr else: raise ValueError("cannot specify both transaction_manager= " "and txn_mgr=") self._log = logging.getLogger("ZODB.Connection") self._debug_info = () self._opened = None # time.time() when DB.open() opened us self._version = version self._cache = cache = PickleCache(self, cache_size) if version: # Caches for versions end up empty if the version # is not used for a while. Non-version caches # keep their content indefinitely. # Unclear: Why do we want version caches to behave this way? self._cache.cache_drain_resistance = 100 self._committed = [] self._added = {} self._added_during_commit = None self._reset_counter = global_reset_counter self._load_count = 0 # Number of objects unghosted self._store_count = 0 # Number of objects stored # List of oids of modified objects (to be invalidated on an abort). self._modified = [] # List of all objects (not oids) registered as modified by the # persistence machinery. self._registered_objects = [] # Do we need to join a txn manager? self._needs_to_join = True # If a transaction manager is passed to the constructor, use # it instead of the global transaction manager. The instance # variable will hold a TM instance. self.transaction_manager = transaction_manager or transaction.manager # _synch is a boolean; if True, the Connection will register # with the TM to receive afterCompletion() calls. self._synch = synch # _invalidated queues invalidate messages delivered from the DB # _inv_lock prevents one thread from modifying the set while # another is processing invalidations. All the invalidations # from a single transaction should be applied atomically, so # the lock must be held when reading _invalidated. # It sucks that we have to hold the lock to read _invalidated. # Normally, _invalidated is written by calling dict.update, which # will execute atomically by virtue of the GIL. But some storage # might generate oids where hash or compare invokes Python code. In # that case, the GIL can't save us. self._inv_lock = threading.Lock() self._invalidated = d = {} # We intend to prevent committing a transaction in which # ReadConflictError occurs. _conflicts is the set of oids that # experienced ReadConflictError. Any time we raise ReadConflictError, # the oid should be added to this set, and we should be sure that the # object is registered. Because it's registered, Connection.commit() # will raise ReadConflictError again (because the oid is in # _conflicts). self._conflicts = {} # If MVCC is enabled, then _mvcc is True and _txn_time stores # the upper bound on transactions visible to this connection. # That is, all object revisions must be written before _txn_time. # If it is None, then the current revisions are acceptable. # If the connection is in a version, mvcc will be disabled, because # loadBefore() only returns non-version data. self._mvcc = mvcc and not version self._txn_time = None # To support importFile(), implemented in the ExportImport base # class, we need to run _importDuringCommit() from our commit() # method. If _import is not None, it is a two-tuple of arguments # to pass to _importDuringCommit(). self._import = None self.connections = None def add(self, obj): """Add a new object 'obj' to the database and assign it an oid.""" if self._storage is None: raise ConnectionStateError("The database connection is closed") marker = object() oid = getattr(obj, "_p_oid", marker) if oid is marker: raise TypeError("Only first-class persistent objects may be" " added to a Connection.", obj) elif obj._p_jar is None: assert obj._p_oid is None oid = obj._p_oid = self._storage.new_oid() obj._p_jar = self if self._added_during_commit is not None: self._added_during_commit.append(obj) self._register(obj) # Add to _added after calling register(), so that _added # can be used as a test for whether the object has been # registered with the transaction. self._added[oid] = obj elif obj._p_jar is not self: raise InvalidObjectReference(obj, obj._p_jar) def get(self, oid): """Return the persistent object with oid 'oid'.""" if self._storage is None: raise ConnectionStateError("The database connection is closed") obj = self._cache.get(oid, None) if obj is not None: return obj obj = self._added.get(oid, None) if obj is not None: return obj p, serial = self._storage.load(oid, self._version) obj = self._reader.getGhost(p) obj._p_oid = oid obj._p_jar = self obj._p_changed = None obj._p_serial = serial self._cache[oid] = obj return obj def cacheMinimize(self, dt=DEPRECATED_ARGUMENT): """Deactivate all unmodified objects in the cache.""" if dt is not DEPRECATED_ARGUMENT: deprecated36("cacheMinimize() dt= is ignored.") self._cache.minimize() # TODO: we should test what happens when cacheGC is called mid-transaction. def cacheGC(self): """Reduce cache size to target size.""" self._cache.incrgc() __onCloseCallbacks = None def onCloseCallback(self, f): """Register a callable, f, to be called by close().""" if self.__onCloseCallbacks is None: self.__onCloseCallbacks = [] self.__onCloseCallbacks.append(f) def close(self): """Close the Connection.""" if not self._needs_to_join: # We're currently joined to a transaction. raise ConnectionStateError("Cannot close a connection joined to " "a transaction") if self._cache is not None: self._cache.incrgc() # This is a good time to do some GC # Call the close callbacks. if self.__onCloseCallbacks is not None: for f in self.__onCloseCallbacks: try: f() except: # except what? f = getattr(f, 'im_self', f) self._log.error("Close callback failed for %s", f, exc_info=sys.exc_info()) self.__onCloseCallbacks = None self._storage = self._savepoint_storage = self._normal_storage = None self.new_oid = None self._debug_info = () self._opened = None # Return the connection to the pool. if self._db is not None: if self._synch: self.transaction_manager.unregisterSynch(self) self._db._closeConnection(self) # _closeConnection() set self._db to None. However, we can't # assert that here, because self may have been reused (by # another thread) by the time we get back here. def db(self): """Returns a handle to the database this connection belongs to.""" return self._db def isReadOnly(self): """Returns True if the storage for this connection is read only.""" if self._storage is None: raise ConnectionStateError("The database connection is closed") return self._storage.isReadOnly() def invalidate(self, tid, oids): """Notify the Connection that transaction 'tid' invalidated oids.""" self._inv_lock.acquire() try: if self._txn_time is None: self._txn_time = tid self._invalidated.update(oids) finally: self._inv_lock.release() def root(self): """Return the database root object.""" return self.get(z64) def getVersion(self): """Returns the version this connection is attached to.""" if self._storage is None: raise ConnectionStateError("The database connection is closed") return self._version def get_connection(self, database_name): """Return a Connection for the named database.""" connection = self.connections.get(database_name) if connection is None: new_con = self._db.databases[database_name].open() self.connections.update(new_con.connections) new_con.connections = self.connections connection = new_con return connection def sync(self): """Manually update the view on the database.""" self.transaction_manager.abort() self._storage_sync() def getDebugInfo(self): """Returns a tuple with different items for debugging the connection. """ return self._debug_info def setDebugInfo(self, *args): """Add the given items to the debug information of this connection.""" self._debug_info = self._debug_info + args def getTransferCounts(self, clear=False): """Returns the number of objects loaded and stored.""" res = self._load_count, self._store_count if clear: self._load_count = 0 self._store_count = 0 return res # Connection methods ########################################################################## ########################################################################## # Data manager (ISavepointDataManager) methods def abort(self, transaction): """Abort a transaction and forget all changes.""" # The order is important here. We want to abort registered # objects before we process the cache. Otherwise, we may un-add # objects added in savepoints. If they've been modified since # the savepoint, then they won't have _p_oid or _p_jar after # they've been unadded. This will make the code in _abort # confused. self._abort() if self._savepoint_storage is not None: self._abort_savepoint() self._tpc_cleanup() def _abort(self): """Abort a transaction and forget all changes.""" for obj in self._registered_objects: oid = obj._p_oid assert oid is not None if oid in self._added: del self._added[oid] del obj._p_jar del obj._p_oid else: # Note: If we invalidate a non-ghostifiable object # (i.e. a persistent class), the object will # immediately reread its state. That means that the # following call could result in a call to # self.setstate, which, of course, must succeed. # In general, it would be better if the read could be # delayed until the start of the next transaction. If # we read at the end of a transaction and if the # object was invalidated during this transaction, then # we'll read non-current data, which we'll discard # later in transaction finalization. Unfortnately, we # can only delay the read if this abort corresponds to # a top-level-transaction abort. We can't tell if # this is a top-level-transaction abort, so we have to # go ahead and invalidate now. Fortunately, it's # pretty unlikely that the object we are invalidating # was invalidated by another thread, so the risk of a # reread is pretty low. self._cache.invalidate(oid) def _tpc_cleanup(self): """Performs cleanup operations to support tpc_finish and tpc_abort.""" self._conflicts.clear() if not self._synch: self._flush_invalidations() self._needs_to_join = True self._registered_objects = [] # Process pending invalidations. def _flush_invalidations(self): self._inv_lock.acquire() try: # Non-ghostifiable objects may need to read when they are # invalidated, so we'll quickly just replace the # invalidating dict with a new one. We'll then process # the invalidations after freeing the lock *and* after # resetting the time. This means that invalidations will # happen after the start of the transactions. They are # subject to conflict errors and to reading old data. # TODO: There is a potential problem lurking for persistent # classes. Suppose we have an invalidation of a persistent # class and of an instance. If the instance is # invalidated first and if the invalidation logic uses # data read from the class, then the invalidation could # be performed with stale data. Or, suppose that there # are instances of the class that are freed as a result of # invalidating some object. Perhaps code in their __del__ # uses class data. Really, the only way to properly fix # this is to, in fact, make classes ghostifiable. Then # we'd have to reimplement attribute lookup to check the # class state and, if necessary, activate the class. It's # much worse than that though, because we'd also need to # deal with slots. When a class is ghostified, we'd need # to replace all of the slot operations with versions that # reloaded the object when called. It's hard to say which # is better or worse. For now, it seems the risk of # using a class while objects are being invalidated seems # small enough to be acceptable. invalidated = self._invalidated self._invalidated = {} self._txn_time = None finally: self._inv_lock.release() self._cache.invalidate(invalidated) # Now is a good time to collect some garbage. self._cache.incrgc() def tpc_begin(self, transaction): """Begin commit of a transaction, starting the two-phase commit.""" self._modified = [] # _creating is a list of oids of new objects, which is used to # remove them from the cache if a transaction aborts. self._creating = [] self._normal_storage.tpc_begin(transaction) def commit(self, transaction): """Commit changes to an object""" if self._savepoint_storage is not None: # We first checkpoint the current changes to the savepoint self.savepoint() # then commit all of the savepoint changes at once self._commit_savepoint(transaction) # No need to call _commit since savepoint did. else: self._commit(transaction) def _commit(self, transaction): """Commit changes to an object""" if self._import: # TODO: This code seems important for Zope, but needs docs # to explain why. self._importDuringCommit(transaction, *self._import) self._import = None # Just in case an object is added as a side-effect of storing # a modified object. If, for example, a __getstate__() method # calls add(), the newly added objects will show up in # _added_during_commit. This sounds insane, but has actually # happened. self._added_during_commit = [] for obj in self._registered_objects: oid = obj._p_oid assert oid if oid in self._conflicts: raise ReadConflictError(object=obj) if obj._p_jar is not self: raise InvalidObjectReference(obj, obj._p_jar) elif oid in self._added: assert obj._p_serial == z64 elif obj._p_changed: if oid in self._invalidated: resolve = getattr(obj, "_p_resolveConflict", None) if resolve is None: raise ConflictError(object=obj) self._modified.append(oid) else: # Nothing to do. It's been said that it's legal, e.g., for # an object to set _p_changed to false after it's been # changed and registered. continue self._store_objects(ObjectWriter(obj), transaction) for obj in self._added_during_commit: self._store_objects(ObjectWriter(obj), transaction) self._added_during_commit = None def _store_objects(self, writer, transaction): for obj in writer: oid = obj._p_oid serial = getattr(obj, "_p_serial", z64) if serial == z64: # obj is a new object self._creating.append(oid) # Because obj was added, it is now in _creating, so it can # be removed from _added. self._added.pop(oid, None) else: if (oid in self._invalidated and not hasattr(obj, '_p_resolveConflict')): raise ConflictError(object=obj) self._modified.append(oid) p = writer.serialize(obj) # This calls __getstate__ of obj s = self._storage.store(oid, serial, p, self._version, transaction) self._store_count += 1 # Put the object in the cache before handling the # response, just in case the response contains the # serial number for a newly created object try: self._cache[oid] = obj except: # Dang, I bet it's wrapped: # TODO: Deprecate, then remove, this. if hasattr(obj, 'aq_base'): self._cache[oid] = obj.aq_base else: raise self._handle_serial(s, oid) def _handle_serial(self, store_return, oid=None, change=1): """Handle the returns from store() and tpc_vote() calls.""" # These calls can return different types depending on whether # ZEO is used. ZEO uses asynchronous returns that may be # returned in batches by the ClientStorage. ZEO1 can also # return an exception object and expect that the Connection # will raise the exception. # When commit_sub() exceutes a store, there is no need to # update the _p_changed flag, because the subtransaction # tpc_vote() calls already did this. The change=1 argument # exists to allow commit_sub() to avoid setting the flag # again. # When conflict resolution occurs, the object state held by # the connection does not match what is written to the # database. Invalidate the object here to guarantee that # the new state is read the next time the object is used. if not store_return: return if isinstance(store_return, str): assert oid is not None self._handle_one_serial(oid, store_return, change) else: for oid, serial in store_return: self._handle_one_serial(oid, serial, change) def _handle_one_serial(self, oid, serial, change): if not isinstance(serial, str): raise serial obj = self._cache.get(oid, None) if obj is None: return if serial == ResolvedSerial: del obj._p_changed # transition from changed to ghost else: if change: obj._p_changed = 0 # transition from changed to up-to-date obj._p_serial = serial def tpc_abort(self, transaction): if self._import: self._import = None if self._savepoint_storage is not None: self._abort_savepoint() self._storage.tpc_abort(transaction) # Note: If we invalidate a non-ghostifiable object (i.e. a # persistent class), the object will immediately reread its # state. That means that the following call could result in a # call to self.setstate, which, of course, must succeed. In # general, it would be better if the read could be delayed # until the start of the next transaction. If we read at the # end of a transaction and if the object was invalidated # during this transaction, then we'll read non-current data, # which we'll discard later in transaction finalization. We # could, theoretically queue this invalidation by calling # self.invalidate. Unfortunately, attempts to make that # change resulted in mysterious test failures. It's pretty # unlikely that the object we are invalidating was invalidated # by another thread, so the risk of a reread is pretty low. # It's really not worth the effort to pursue this. self._cache.invalidate(self._modified) self._invalidate_creating() while self._added: oid, obj = self._added.popitem() del obj._p_oid del obj._p_jar self._tpc_cleanup() def _invalidate_creating(self, creating=None): """Disown any objects newly saved in an uncommitted transaction.""" if creating is None: creating = self._creating self._creating = [] for oid in creating: o = self._cache.get(oid) if o is not None: del self._cache[oid] del o._p_jar del o._p_oid def tpc_vote(self, transaction): """Verify that a data manager can commit the transaction.""" try: vote = self._storage.tpc_vote except AttributeError: return s = vote(transaction) self._handle_serial(s) def tpc_finish(self, transaction): """Indicate confirmation that the transaction is done.""" def callback(tid): d = dict.fromkeys(self._modified) self._db.invalidate(tid, d, self) # It's important that the storage calls the passed function # while it still has its lock. We don't want another thread # to be able to read any updated data until we've had a chance # to send an invalidation message to all of the other # connections! self._storage.tpc_finish(transaction, callback) self._tpc_cleanup() def sortKey(self): """Return a consistent sort key for this connection.""" return "%s:%s" % (self._storage.sortKey(), id(self)) # Data manager (ISavepointDataManager) methods ########################################################################## ########################################################################## # Transaction-manager synchronization -- ISynchronizer def beforeCompletion(self, txn): # We don't do anything before a commit starts. pass # Call the underlying storage's sync() method (if any), and process # pending invalidations regardless. Of course this should only be # called at transaction boundaries. def _storage_sync(self, *ignored): sync = getattr(self._storage, 'sync', 0) if sync: sync() self._flush_invalidations() afterCompletion = _storage_sync newTransaction = _storage_sync # Transaction-manager synchronization -- ISynchronizer ########################################################################## ########################################################################## # persistent.interfaces.IPersistentDatamanager def oldstate(self, obj, tid): """Return copy of 'obj' that was written by transaction 'tid'.""" assert obj._p_jar is self p = self._storage.loadSerial(obj._p_oid, tid) return self._reader.getState(p) def setstate(self, obj): """Turns the ghost 'obj' into a real object by loading it's from the database.""" oid = obj._p_oid if self._storage is None: msg = ("Shouldn't load state for %s " "when the connection is closed" % oid_repr(oid)) self._log.error(msg) raise ConnectionStateError(msg) try: self._setstate(obj) except ConflictError: raise except: self._log.error("Couldn't load state for %s", oid_repr(oid), exc_info=sys.exc_info()) raise def _setstate(self, obj): # Helper for setstate(), which provides logging of failures. # The control flow is complicated here to avoid loading an # object revision that we are sure we aren't going to use. As # a result, invalidation tests occur before and after the # load. We can only be sure about invalidations after the # load. # If an object has been invalidated, there are several cases # to consider: # 1. Check _p_independent() # 2. Try MVCC # 3. Raise ConflictError. # Does anything actually use _p_independent()? It would simplify # the code if we could drop support for it. # There is a harmless data race with self._invalidated. A # dict update could go on in another thread, but we don't care # because we have to check again after the load anyway. if (obj._p_oid in self._invalidated and not myhasattr(obj, "_p_independent")): # If the object has _p_independent(), we will handle it below. self._load_before_or_conflict(obj) return p, serial = self._storage.load(obj._p_oid, self._version) self._load_count += 1 self._inv_lock.acquire() try: invalid = obj._p_oid in self._invalidated finally: self._inv_lock.release() if invalid: if myhasattr(obj, "_p_independent"): # This call will raise a ReadConflictError if something # goes wrong self._handle_independent(obj) else: self._load_before_or_conflict(obj) return self._reader.setGhostState(obj, p) obj._p_serial = serial def _load_before_or_conflict(self, obj): """Load non-current state for obj or raise ReadConflictError.""" if not (self._mvcc and self._setstate_noncurrent(obj)): self._register(obj) self._conflicts[obj._p_oid] = True raise ReadConflictError(object=obj) def _setstate_noncurrent(self, obj): """Set state using non-current data. Return True if state was available, False if not. """ try: # Load data that was current before the commit at txn_time. t = self._storage.loadBefore(obj._p_oid, self._txn_time) except KeyError: return False if t is None: return False data, start, end = t # The non-current transaction must have been written before # txn_time. It must be current at txn_time, but could have # been modified at txn_time. assert start < self._txn_time, (u64(start), u64(self._txn_time)) assert end is not None assert self._txn_time <= end, (u64(self._txn_time), u64(end)) self._reader.setGhostState(obj, data) obj._p_serial = start return True def _handle_independent(self, obj): # Helper method for setstate() handles possibly independent objects # Call _p_independent(), if it returns True, setstate() wins. # Otherwise, raise a ConflictError. if obj._p_independent(): self._inv_lock.acquire() try: try: del self._invalidated[obj._p_oid] except KeyError: pass finally: self._inv_lock.release() else: self._conflicts[obj._p_oid] = 1 self._register(obj) raise ReadConflictError(object=obj) def register(self, obj): """Register obj with the current transaction manager. A subclass could override this method to customize the default policy of one transaction manager for each thread. obj must be an object loaded from this Connection. """ assert obj._p_jar is self if obj._p_oid is None: # There is some old Zope code that assigns _p_jar # directly. That is no longer allowed, but we need to # provide support for old code that still does it. # The actual complaint here is that an object without # an oid is being registered. I can't think of any way to # achieve that without assignment to _p_jar. If there is # a way, this will be a very confusing warning. deprecated36("Assigning to _p_jar is deprecated, and will be " "changed to raise an exception.") elif obj._p_oid in self._added: # It was registered before it was added to _added. return self._register(obj) def _register(self, obj=None): # The order here is important. We need to join before # registering the object, because joining may take a # savepoint, and the savepoint should not reflect the change # to the object. if self._needs_to_join: self.transaction_manager.get().join(self) self._needs_to_join = False if obj is not None: self._registered_objects.append(obj) # persistent.interfaces.IPersistentDatamanager ########################################################################## ########################################################################## # PROTECTED stuff (used by e.g. ZODB.DB.DB) def _cache_items(self): # find all items on the lru list items = self._cache.lru_items() # fine everything. some on the lru list, some not everything = self._cache.cache_data # remove those items that are on the lru list for k,v in items: del everything[k] # return a list of [ghosts....not recently used.....recently used] return everything.items() + items def _setDB(self, odb, mvcc=None, txn_mgr=DEPRECATED_ARGUMENT, transaction_manager=None, synch=None): """Register odb, the DB that this Connection uses. This method is called by the DB every time a Connection is opened. Any invalidations received while the Connection was closed will be processed. If the global module function resetCaches() was called, the cache will be cleared. Parameters: odb: database that owns the Connection mvcc: boolean indicating whether MVCC is enabled transaction_manager: transaction manager to use. None means use the default transaction manager. synch: boolean indicating whether Connection should register for afterCompletion() calls. """ if txn_mgr is not DEPRECATED_ARGUMENT: deprecated36("use transaction_manager= instead of txn_mgr=") if transaction_manager is None: transaction_manager = txn_mgr else: raise ValueError("cannot specify both transaction_manager= " "and txn_mgr=") # TODO: Why do we go to all the trouble of setting _db and # other attributes on open and clearing them on close? # A Connection is only ever associated with a single DB # and Storage. self._db = odb self._normal_storage = self._storage = odb._storage self.new_oid = odb._storage.new_oid self._opened = time() if synch is not None: self._synch = synch if mvcc is not None: self._mvcc = mvcc self.transaction_manager = transaction_manager or transaction.manager if self._reset_counter != global_reset_counter: # New code is in place. Start a new cache. self._resetCache() else: self._flush_invalidations() if self._synch: self.transaction_manager.registerSynch(self) self._reader = ObjectReader(self, self._cache, self._db.classFactory) # Multi-database support self.connections = {self._db.database_name: self} def _resetCache(self): """Creates a new cache, discarding the old one. See the docstring for the resetCaches() function. """ self._reset_counter = global_reset_counter self._invalidated.clear() cache_size = self._cache.cache_size self._cache = cache = PickleCache(self, cache_size) ########################################################################## # Python protocol def __repr__(self): if self._version: ver = ' (in version %s)' % `self._version` else: ver = '' return '' % (positive_id(self), ver) # Python protocol ########################################################################## ########################################################################## # DEPRECATION candidates __getitem__ = get def modifiedInVersion(self, oid): """Returns the version the object with the given oid was modified in. If it wasn't modified in a version, the current version of this connection is returned. """ try: return self._db.modifiedInVersion(oid) except KeyError: return self.getVersion() def exchange(self, old, new): # called by a ZClasses method that isn't executed by the test suite oid = old._p_oid new._p_oid = oid new._p_jar = self new._p_changed = 1 self._register(new) self._cache[oid] = new # DEPRECATION candidates ########################################################################## ########################################################################## # DEPRECATED methods def cacheFullSweep(self, dt=None): deprecated36("cacheFullSweep is deprecated. " "Use cacheMinimize instead.") if dt is None: self._cache.full_sweep() else: self._cache.full_sweep(dt) def getTransaction(self): """Get the current transaction for this connection. :deprecated: The transaction manager's get method works the same as this method. You can pass a transaction manager (TM) to DB.open() to control which TM the Connection uses. """ deprecated36("getTransaction() is deprecated. " "Use the transaction_manager argument " "to DB.open() instead, or access " ".transaction_manager directly on the Connection.") return self.transaction_manager.get() def setLocalTransaction(self): """Use a transaction bound to the connection rather than the thread. :deprecated: Returns the transaction manager used by the connection. You can pass a transaction manager (TM) to DB.open() to control which TM the Connection uses. """ deprecated36("setLocalTransaction() is deprecated. " "Use the transaction_manager argument " "to DB.open() instead.") if self.transaction_manager is transaction.manager: if self._synch: self.transaction_manager.unregisterSynch(self) self.transaction_manager = transaction.TransactionManager() if self._synch: self.transaction_manager.registerSynch(self) return self.transaction_manager # DEPRECATED methods ########################################################################## ##################################################################### # Savepoint support def savepoint(self): if self._savepoint_storage is None: self._savepoint_storage = TmpStore(self._version, self._normal_storage) self._storage = self._savepoint_storage self._creating = [] self._commit(None) self._storage.creating.extend(self._creating) del self._creating[:] self._registered_objects = [] state = self._storage.position, self._storage.index.copy() result = Savepoint(self, state) # While the interface doesn't guarantee this, savepoints are # sometimes used just to "break up" very long transactions, and as # a pragmatic matter this is a good time to reduce the cache # memory burden. self.cacheGC() return result def _rollback(self, state): self._abort() self._registered_objects = [] src = self._storage self._cache.invalidate(src.index) src.reset(*state) def _commit_savepoint(self, transaction): """Commit all changes made in subtransactions and begin 2-phase commit """ src = self._savepoint_storage self._storage = self._normal_storage self._savepoint_storage = None self._log.debug("Commiting savepoints of size %s", src.getSize()) oids = src.index.keys() # Copy invalidating and creating info from temporary storage: self._modified.extend(oids) self._creating.extend(src.creating) for oid in oids: data, serial = src.load(oid, src) s = self._storage.store(oid, serial, data, self._version, transaction) self._handle_serial(s, oid, change=False) src.close() def _abort_savepoint(self): """Discard all subtransaction data.""" src = self._savepoint_storage self._storage = self._normal_storage self._savepoint_storage = None # Note: If we invalidate a non-ghostifiable object (i.e. a # persistent class), the object will immediately reread it's # state. That means that the following call could result in a # call to self.setstate, which, of course, must succeed. In # general, it would be better if the read could be delayed # until the start of the next transaction. If we read at the # end of a transaction and if the object was invalidated # during this transaction, then we'll read non-current data, # which we'll discard later in transaction finalization. We # could, theoretically queue this invalidation by calling # self.invalidate. Unfortunately, attempts to make that # change resulted in mysterious test failures. It's pretty # unlikely that the object we are invalidating was invalidated # by another thread, so the risk of a reread is pretty low. # It's really not worth the effort to pursue this. self._cache.invalidate(src.index) self._invalidate_creating(src.creating) src.close() # Savepoint support ##################################################################### class Savepoint: implements(IDataManagerSavepoint) def __init__(self, datamanager, state): self.datamanager = datamanager self.state = state def rollback(self): self.datamanager._rollback(self.state) class TmpStore: """A storage-like thing to support savepoints.""" def __init__(self, base_version, storage): self._storage = storage for method in ( 'getName', 'new_oid', 'modifiedInVersion', 'getSize', 'undoLog', 'versionEmpty', 'sortKey', 'loadBefore', ): setattr(self, method, getattr(storage, method)) self._base_version = base_version self._file = tempfile.TemporaryFile() # position: current file position # _tpos: file position at last commit point self.position = 0L # index: map oid to pos of last committed version self.index = {} self.creating = [] def __len__(self): return len(self.index) def close(self): self._file.close() def load(self, oid, version): pos = self.index.get(oid) if pos is None: return self._storage.load(oid, self._base_version) self._file.seek(pos) h = self._file.read(8) oidlen = u64(h) read_oid = self._file.read(oidlen) if read_oid != oid: raise POSException.StorageSystemError('Bad temporary storage') h = self._file.read(16) size = u64(h[8:]) serial = h[:8] return self._file.read(size), serial def store(self, oid, serial, data, version, transaction): # we have this funny signature so we can reuse the normal non-commit # commit logic assert version == self._base_version self._file.seek(self.position) l = len(data) if serial is None: serial = z64 header = p64(len(oid)) + oid + serial + p64(l) self._file.write(header) self._file.write(data) self.index[oid] = self.position self.position += l + len(header) return serial def reset(self, position, index): self._file.truncate(position) self.position = position # Caution: We're typically called as part of a savepoint rollback. # Other machinery remembers the index to restore, and passes it to # us. If we simply bind self.index to `index`, then if the caller # didn't pass a copy of the index, the caller's index will mutate # when self.index mutates. This can be a disaster if the caller is a # savepoint to which the user rolls back again later (the savepoint # loses the original index it passed). Therefore, to be safe, we make # a copy of the index here. An alternative would be to ensure that # all callers pass copies. As is, our callers do not make copies. self.index = index.copy() Zope-2.8.7-final/lib/python/ZODB/utils.py0000664000076400007640000002435110435502423017102 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import sys import time import struct from struct import pack, unpack from binascii import hexlify import cPickle as pickle from cStringIO import StringIO import weakref import warnings from persistent.TimeStamp import TimeStamp __all__ = ['z64', 'p64', 'u64', 'U64', 'cp', 'newTimeStamp', 'oid_repr', 'serial_repr', 'tid_repr', 'positive_id', 'get_refs', 'readable_tid_repr', 'WeakSet', 'DEPRECATED_ARGUMENT', 'deprecated36', 'get_pickle_metadata', ] # A unique marker to give as the default value for a deprecated argument. # The method should then do a # # if that_arg is not DEPRECATED_ARGUMENT: # complain # # dance. DEPRECATED_ARGUMENT = object() # Raise DeprecationWarning, noting that the deprecated thing will go # away in ZODB 3.6. Point to the caller of our caller (i.e., at the # code using the deprecated thing). def deprecated36(msg): warnings.warn("This will be removed in ZODB 3.6:\n%s" % msg, DeprecationWarning, stacklevel=3) z64 = '\0'*8 assert sys.hexversion >= 0x02030000 # The distinction between ints and longs is blurred in Python 2.2, # so u64() are U64() really the same. def p64(v): """Pack an integer or long into a 8-byte string""" return pack(">Q", v) def u64(v): """Unpack an 8-byte string into a 64-bit long integer.""" return unpack(">Q", v)[0] U64 = u64 def cp(f1, f2, l): read = f1.read write = f2.write n = 8192 while l > 0: if n > l: n = l d = read(n) if not d: break write(d) l = l - len(d) def newTimeStamp(old=None, TimeStamp=TimeStamp, time=time.time, gmtime=time.gmtime): t = time() ts = TimeStamp(gmtime(t)[:5]+(t%60,)) if old is not None: return ts.laterThan(old) return ts def oid_repr(oid): if isinstance(oid, str) and len(oid) == 8: # Convert to hex and strip leading zeroes. as_hex = hexlify(oid).lstrip('0') # Ensure two characters per input byte. if len(as_hex) & 1: as_hex = '0' + as_hex elif as_hex == '': as_hex = '00' return '0x' + as_hex else: return repr(oid) serial_repr = oid_repr tid_repr = serial_repr # For example, produce # '0x03441422948b4399 2002-04-14 20:50:34.815000' # for 8-byte string tid '\x03D\x14"\x94\x8bC\x99'. def readable_tid_repr(tid): result = tid_repr(tid) if isinstance(tid, str) and len(tid) == 8: result = "%s %s" % (result, TimeStamp(tid)) return result # Addresses can "look negative" on some boxes, some of the time. If you # feed a "negative address" to an %x format, Python 2.3 displays it as # unsigned, but produces a FutureWarning, because Python 2.4 will display # it as signed. So when you want to prodce an address, use positive_id() to # obtain it. # _ADDRESS_MASK is 2**(number_of_bits_in_a_native_pointer). Adding this to # a negative address gives a positive int with the same hex representation as # the significant bits in the original. _ADDRESS_MASK = 256 ** struct.calcsize('P') def positive_id(obj): """Return id(obj) as a non-negative integer.""" result = id(obj) if result < 0: result += _ADDRESS_MASK assert result > 0 return result # So full of undocumented magic it's hard to fathom. # The existence of cPickle.noload() isn't documented, and what it # does isn't documented either. In general it unpickles, but doesn't # actually build any objects of user-defined classes. Despite that # persistent_load is documented to be a callable, there's an # undocumented gimmick where if it's actually a list, for a PERSID or # BINPERSID opcode cPickle just appends "the persistent id" to that list. # Also despite that "a persistent id" is documented to be a string, # ZODB persistent ids are actually (often? always?) tuples, most often # of the form # (oid, (module_name, class_name)) # So the effect of the following is to dig into the object pickle, and # return a list of the persistent ids found (which are usually nested # tuples), without actually loading any modules or classes. # Note that pickle.py doesn't support any of this, it's undocumented code # only in cPickle.c. def get_refs(a_pickle): # The pickle is in two parts. First there's the class of the object, # needed to build a ghost, See get_pickle_metadata for how complicated # this can get. The second part is the state of the object. We want # to find all the persistent references within both parts (although I # expect they can only appear in the second part). f = StringIO(a_pickle) u = pickle.Unpickler(f) u.persistent_load = refs = [] u.noload() # class info u.noload() # instance state info return refs # Given a ZODB pickle, return pair of strings (module_name, class_name). # Do this without importing the module or class object. # See ZODB/serialize.py's module docstring for the only docs that exist about # ZODB pickle format. If the code here gets smarter, please update those # docs to be at least as smart. The code here doesn't appear to make sense # for what serialize.py calls formats 5 and 6. def get_pickle_metadata(data): # ZODB's data records contain two pickles. The first is the class # of the object, the second is the object. We're only trying to # pick apart the first here, to extract the module and class names. if data.startswith('(c'): # pickle MARK GLOBAL opcode sequence global_prefix = 2 elif data.startswith('c'): # pickle GLOBAL opcode global_prefix = 1 else: global_prefix = 0 if global_prefix: # Formats 1 and 2. # Don't actually unpickle a class, because it will attempt to # load the class. Just break open the pickle and get the # module and class from it. The module and class names are given by # newline-terminated strings following the GLOBAL opcode. modname, classname, rest = data.split('\n', 2) modname = modname[global_prefix:] # strip GLOBAL opcode return modname, classname # Else there are a bunch of other possible formats. f = StringIO(data) u = pickle.Unpickler(f) try: class_info = u.load() except Exception, err: print "Error", err return '', '' if isinstance(class_info, tuple): if isinstance(class_info[0], tuple): # Formats 3 and 4. modname, classname = class_info[0] else: # Formats 5 and 6 (probably) end up here. modname, classname = class_info else: # This isn't a known format. modname = repr(class_info) classname = '' return modname, classname # A simple implementation of weak sets, supplying just enough of Python's # sets.Set interface for our needs. class WeakSet(object): """A set of objects that doesn't keep its elements alive. The objects in the set must be weakly referencable. The objects need not be hashable, and need not support comparison. Two objects are considered to be the same iff their id()s are equal. When the only references to an object are weak references (including those from WeakSets), the object can be garbage-collected, and will vanish from any WeakSets it may be a member of at that time. """ def __init__(self): # Map id(obj) to obj. By using ids as keys, we avoid requiring # that the elements be hashable or comparable. self.data = weakref.WeakValueDictionary() def __len__(self): return len(self.data) def __contains__(self, obj): return id(obj) in self.data # Same as a Set, add obj to the collection. def add(self, obj): self.data[id(obj)] = obj # Same as a Set, remove obj from the collection, and raise # KeyError if obj not in the collection. def remove(self, obj): del self.data[id(obj)] # f is a one-argument function. Execute f(elt) for each elt in the # set. f's return value is ignored. def map(self, f): for wr in self.as_weakref_list(): elt = wr() if elt is not None: f(elt) # Return a list of weakrefs to all the objects in the collection. # Because a weak dict is used internally, iteration is dicey (the # underlying dict may change size during iteration, due to gc or # activity from other threads). as_weakef_list() is safe. # # Something like this should really be a method of Python's weak dicts. # If we invoke self.data.values() instead, we get back a list of live # objects instead of weakrefs. If gc occurs while this list is alive, # all the objects move to an older generation (because they're strongly # referenced by the list!). They can't get collected then, until a # less frequent collection of the older generation. Before then, if we # invoke self.data.values() again, they're still alive, and if gc occurs # while that list is alive they're all moved to yet an older generation. # And so on. Stress tests showed that it was easy to get into a state # where a WeakSet grows without bounds, despite that almost all its # elements are actually trash. By returning a list of weakrefs instead, # we avoid that, although the decision to use weakrefs is now# very # visible to our clients. def as_weakref_list(self): # We're cheating by breaking into the internals of Python's # WeakValueDictionary here (accessing its .data attribute). return self.data.data.values() Zope-2.8.7-final/lib/python/ZODB/ActivityMonitor.py0000664000076400007640000000710010435502423021077 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ZODB transfer activity monitoring $Id: ActivityMonitor.py 25186 2004-06-02 15:07:33Z jim $""" import time class ActivityMonitor: """ZODB load/store activity monitor This simple implementation just keeps a small log in memory and iterates over the log when getActivityAnalysis() is called. It assumes that log entries are added in chronological sequence, which is only guaranteed because DB.py holds a lock when calling the closedConnection() method. """ def __init__(self, history_length=3600): self.history_length = history_length # Number of seconds self.log = [] # [(time, loads, stores)] def closedConnection(self, conn): log = self.log now = time.time() loads, stores = conn.getTransferCounts(1) log.append((now, loads, stores)) self.trim(now) def trim(self, now): log = self.log cutoff = now - self.history_length n = 0 loglen = len(log) while n < loglen and log[n][0] < cutoff: n = n + 1 if n: del log[:n] def setHistoryLength(self, history_length): self.history_length = history_length self.trim(time.time()) def getHistoryLength(self): return self.history_length def getActivityAnalysis(self, start=0, end=0, divisions=10): res = [] now = time.time() if start == 0: start = now - self.history_length if end == 0: end = now for n in range(divisions): res.append({ 'start': start + (end - start) * n / divisions, 'end': start + (end - start) * (n + 1) / divisions, 'loads': 0, 'stores': 0, 'connections': 0, }) div = res[0] div_end = div['end'] div_index = 0 connections = 0 total_loads = 0 total_stores = 0 for t, loads, stores in self.log: if t < start: # We could use a binary search to find the start. continue elif t > end: # We could use a binary search to find the end also. break while t > div_end: div['loads'] = total_loads div['stores'] = total_stores div['connections'] = connections total_loads = 0 total_stores = 0 connections = 0 div_index = div_index + 1 if div_index < divisions: div = res[div_index] div_end = div['end'] connections = connections + 1 total_loads = total_loads + loads total_stores = total_stores + stores div['stores'] = div['stores'] + total_stores div['loads'] = div['loads'] + total_loads div['connections'] = div['connections'] + connections return res Zope-2.8.7-final/lib/python/ZODB/persistentclass.txt0000664000076400007640000001653410435502423021363 0ustar ajungajungPersistent Classes ================== NOTE: persistent classes are EXPERIMENTAL and, in some sense, incomplete. This module exists largely to test changes made to support Zope 2 ZClasses, with their historical flaws. The persistentclass module provides a meta class that can be used to implement persistent classes. Persistent classes have the following properties: - They cannot be turned into ghosts - They can only contain picklable subobjects - They don't live in regular file-system modules Let's look at an example: >>> def __init__(self, name): ... self.name = name >>> def foo(self): ... return self.name, self.kind >>> import ZODB.persistentclass >>> class C: ... __metaclass__ = ZODB.persistentclass.PersistentMetaClass ... __init__ = __init__ ... __module__ = '__zodb__' ... foo = foo ... kind = 'sample' This example is obviously a bit contrived. In particular, we defined the methods outside of the class. Why? Because all of the items in a persistent class must be picklable. We defined the methods as global functions to make them picklable. Also note that we explictly set the module. Persistent classes don't live in normal Python modules. Rather, they live in the database. We use information in __module__ to record where in the database. When we want to use a database, we will need to supply a custom class factory to load instances of the class. The class we created works a lot like other persistent objects. It has standard standard persistent attributes: >>> C._p_oid >>> C._p_jar >>> C._p_serial >>> C._p_changed False Because we haven't saved the object, the jar, oid, and serial are all None and it's not changed. We can create and use instances of the class: >>> c = C('first') >>> c.foo() ('first', 'sample') We can modify the class and none of the persistent attributes will change because the object hasn't been saved. >>> def bar(self): ... print 'bar', self.name >>> C.bar = bar >>> c.bar() bar first >>> C._p_oid >>> C._p_jar >>> C._p_serial >>> C._p_changed False Now, we can store the class in a database. We're going to use an explicit transaction manager so that we can show parallel transactions without having to use threads. >>> import transaction >>> tm = transaction.TransactionManager() >>> connection = some_database.open(transaction_manager=tm) >>> connection.root()['C'] = C >>> tm.commit() Now, if we look at the persistence variables, we'll see that they have values: >>> C._p_oid '\x00\x00\x00\x00\x00\x00\x00\x01' >>> C._p_jar is not None True >>> C._p_serial is not None True >>> C._p_changed False Now, if we modify the class: >>> def baz(self): ... print 'baz', self.name >>> C.baz = baz >>> c.baz() baz first We'll see that the class has changed: >>> C._p_changed True If we abort the transaction: >>> tm.abort() Then the class will return to it's prior state: >>> c.baz() Traceback (most recent call last): ... AttributeError: 'C' object has no attribute 'baz' >>> c.bar() bar first We can open another connection and access the class there. >>> tm2 = transaction.TransactionManager() >>> connection2 = some_database.open(transaction_manager=tm2) >>> C2 = connection2.root()['C'] >>> c2 = C2('other') >>> c2.bar() bar other If we make changes without commiting them: >>> C.bar = baz >>> c.bar() baz first >>> C is C2 False Other connections are unaffected: >>> connection2.sync() >>> c2.bar() bar other Until we commit: >>> tm.commit() >>> connection2.sync() >>> c2.bar() baz other Similarly, we don't see changes made in other connections: >>> C2.color = 'red' >>> tm2.commit() >>> c.color Traceback (most recent call last): ... AttributeError: 'C' object has no attribute 'color' until we sync: >>> connection.sync() >>> c.color 'red' Instances of Persistent Classes ------------------------------- We can, of course, store instances of perstent classes in the database: >>> c.color = 'blue' >>> connection.root()['c'] = c >>> tm.commit() >>> connection2.sync() >>> connection2.root()['c'].color 'blue' NOTE: If a non-persistent instance of a persistent class is copied, the class may be copied as well. This is usually not the desired result. Persistent instances of persistent classes ------------------------------------------ Persistent instances of persistent classes are handled differently than normal instances. When we copy a persistent instances of a persistent class, we want to avoid copying the class. Lets create a persistent class that subclasses Persistent: >>> import persistent >>> class P(persistent.Persistent, C): ... __module__ = '__zodb__' ... color = 'green' >>> connection.root()['P'] = P >>> import persistent.mapping >>> connection.root()['obs'] = persistent.mapping.PersistentMapping() >>> p = P('p') >>> connection.root()['obs']['p'] = p >>> tm.commit() You might be wondering why we didn't just stick 'p' into the root object. We created an intermediate persistent object instead. We are storing persistent classes in the root object. To create a ghost for a persistent instance of a persistent class, we need to be able to be able to access the root object and it must be loaded first. If the instance was in the root object, we'd be unable to create it while loading the root object. Now, if we try to load it, we get a broken oject: >>> connection2.sync() >>> connection2.root()['obs']['p'] because the module, "__zodb__" can't be loaded. We need to provide a class factory that knows about this special module. Here we'll supply a sample class factory that looks up a class name in the database root if the module is "__zodb__". It falls back to the normal class lookup for other modules: >>> from ZODB.broken import find_global >>> def classFactory(connection, modulename, globalname): ... if modulename == '__zodb__': ... return connection.root()[globalname] ... return find_global(modulename, globalname) >>> some_database.classFactory = classFactory Normally, the classFactory should be set before a database is opened. We'll reopen the connections we're using. We'll assign the old connections to a variable first to prevent getting them from the connection pool: >>> old = connection, connection2 >>> connection = some_database.open(transaction_manager=tm) >>> connection2 = some_database.open(transaction_manager=tm2) Now, we can read the object: >>> connection2.root()['obs']['p'].color 'green' >>> connection2.root()['obs']['p'].color = 'blue' >>> tm2.commit() >>> connection.sync() >>> p = connection.root()['obs']['p'] >>> p.color 'blue' Copying ------- If we copy an instance via export/import, the copy and the original share the same class: >>> file = connection.exportFile(p._p_oid) >>> file.seek(0) >>> cp = connection.importFile(file) >>> cp.color 'blue' >>> cp is not p True >>> cp.__class__ is p.__class__ True XXX test abort of import Zope-2.8.7-final/lib/python/ZODB/interfaces.py0000664000076400007640000004717310435502423020074 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Interfaces for ZODB. $Id: interfaces.py 38093 2005-08-25 23:02:14Z tim_one $ """ from zope.interface import Interface, Attribute class IConnection(Interface): """Connection to ZODB for loading and storing objects. The Connection object serves as a data manager. The root() method on a Connection returns the root object for the database. This object and all objects reachable from it are associated with the Connection that loaded them. When a transaction commits, it uses the Connection to store modified objects. Typical use of ZODB is for each thread to have its own Connection and that no thread should have more than one Connection to the same database. A thread is associated with a Connection by loading objects from that Connection. Objects loaded by one thread should not be used by another thread. A Connection can be associated with a single version when it is created. By default, a Connection is not associated with a version; it uses non-version data. Each Connection provides an isolated, consistent view of the database, by managing independent copies of objects in the database. At transaction boundaries, these copies are updated to reflect the current state of the database. You should not instantiate this class directly; instead call the open() method of a DB instance. In many applications, root() is the only method of the Connection that you will need to use. Synchronization --------------- A Connection instance is not thread-safe. It is designed to support a thread model where each thread has its own transaction. If an application has more than one thread that uses the connection or the transaction the connection is registered with, the application should provide locking. The Connection manages movement of objects in and out of object storage. TODO: We should document an intended API for using a Connection via multiple threads. TODO: We should explain that the Connection has a cache and that multiple calls to get() will return a reference to the same object, provided that one of the earlier objects is still referenced. Object identity is preserved within a connection, but not across connections. TODO: Mention the database pool. A database connection always presents a consistent view of the objects in the database, although it may not always present the most current revision of any particular object. Modifications made by concurrent transactions are not visible until the next transaction boundary (abort or commit). Two options affect consistency. By default, the mvcc and synch options are enabled by default. If you pass mvcc=False to db.open(), the Connection will never read non-current revisions of an object. Instead it will raise a ReadConflictError to indicate that the current revision is unavailable because it was written after the current transaction began. The logic for handling modifications assumes that the thread that opened a Connection (called db.open()) is the thread that will use the Connection. If this is not true, you should pass synch=False to db.open(). When the synch option is disabled, some transaction boundaries will be missed by the Connection; in particular, if a transaction does not involve any modifications to objects loaded from the Connection and synch is disabled, the Connection will miss the transaction boundary. Two examples of this behavior are db.undo() and read-only transactions. Groups of methods: User Methods: root, get, add, close, db, sync, isReadOnly, cacheGC, cacheFullSweep, cacheMinimize, getVersion, modifiedInVersion Experimental Methods: onCloseCallbacks Database Invalidation Methods: invalidate Other Methods: exchange, getDebugInfo, setDebugInfo, getTransferCounts """ def __init__(version='', cache_size=400, cache_deactivate_after=None, mvcc=True, transaction_manager=None, synch=True): """Create a new Connection. A Connection instance should by instantiated by the DB instance that it is connected to. Parameters: version: the "version" that all changes will be made in, defaults to no version. cache_size: the target size of the in-memory object cache, measured in objects. mvcc: boolean indicating whether MVCC is enabled transaction_manager: transaction manager to use. None means use the default transaction manager. synch: boolean indicating whether Connection should register for afterCompletion() calls. """ def add(ob): """Add a new object 'obj' to the database and assign it an oid. A persistent object is normally added to the database and assigned an oid when it becomes reachable to an object already in the database. In some cases, it is useful to create a new object and use its oid (_p_oid) in a single transaction. This method assigns a new oid regardless of whether the object is reachable. The object is added when the transaction commits. The object must implement the IPersistent interface and must not already be associated with a Connection. Parameters: obj: a Persistent object Raises TypeError if obj is not a persistent object. Raises InvalidObjectReference if obj is already associated with another connection. Raises ConnectionStateError if the connection is closed. """ def get(oid): """Return the persistent object with oid 'oid'. If the object was not in the cache and the object's class is ghostable, then a ghost will be returned. If the object is already in the cache, a reference to the cached object will be returned. Applications seldom need to call this method, because objects are loaded transparently during attribute lookup. Parameters: oid: an object id Raises KeyError if oid does not exist. It is possible that an object does not exist as of the current transaction, but existed in the past. It may even exist again in the future, if the transaction that removed it is undone. Raises ConnectionStateError if the connection is closed. """ def cacheMinimize(): """Deactivate all unmodified objects in the cache. Call _p_deactivate() on each cached object, attempting to turn it into a ghost. It is possible for individual objects to remain active. """ def cacheGC(): """Reduce cache size to target size. Call _p_deactivate() on cached objects until the cache size falls under the target size. """ def onCloseCallback(f): """Register a callable, f, to be called by close(). f will be called with no arguments before the Connection is closed. Parameters: f: method that will be called on `close` """ def close(): """Close the Connection. When the Connection is closed, all callbacks registered by onCloseCallback() are invoked and the cache is garbage collected. A closed Connection should not be used by client code. It can't load or store objects. Objects in the cache are not freed, because Connections are re-used and the cache is expected to be useful to the next client. """ def db(): """Returns a handle to the database this connection belongs to.""" def isReadOnly(): """Returns True if the storage for this connection is read only.""" def invalidate(tid, oids): """Notify the Connection that transaction 'tid' invalidated oids. When the next transaction boundary is reached, objects will be invalidated. If any of the invalidated objects are accessed by the current transaction, the revision written before Connection.tid will be used. The DB calls this method, even when the Connection is closed. Parameters: tid: the storage-level id of the transaction that committed oids: oids is a set of oids, represented as a dict with oids as keys. """ def root(): """Return the database root object. The root is a persistent.mapping.PersistentMapping. """ def getVersion(): """Returns the version this connection is attached to.""" # Multi-database support. connections = Attribute("""\ A mapping from database name to a Connection to that database. In multi-database use, the Connections of all members of a database collection share the same .connections object. In single-database use, of course this mapping contains a single entry. """) # TODO: should this accept all the arguments one may pass to DB.open()? def get_connection(database_name): """Return a Connection for the named database. This is intended to be called from an open Connection associated with a multi-database. In that case, database_name must be the name of a database within the database collection (probably the name of a different database than is associated with the calling Connection instance, but it's fine to use the name of the calling Connection object's database). A Connection for the named database is returned. If no connection to that database is already open, a new Connection is opened. So long as the multi-database remains open, passing the same name to get_connection() multiple times returns the same Connection object each time. """ def sync(): """Manually update the view on the database. This includes aborting the current transaction, getting a fresh and consistent view of the data (synchronizing with the storage if possible) and calling cacheGC() for this connection. This method was especially useful in ZODB 3.2 to better support read-only connections that were affected by a couple of problems. """ # Debug information def getDebugInfo(): """Returns a tuple with different items for debugging the connection. Debug information can be added to a connection by using setDebugInfo. """ def setDebugInfo(*items): """Add the given items to the debug information of this connection.""" def getTransferCounts(clear=False): """Returns the number of objects loaded and stored. If clear is True, reset the counters. """ class IDatabase(Interface): """ZODB DB. TODO: This interface is incomplete. """ def __init__(storage, pool_size=7, cache_size=400, version_pool_size=3, version_cache_size=100, database_name='unnamed', databases=None, ): """Create an object database. storage: the storage used by the database, e.g. FileStorage pool_size: expected maximum number of open connections cache_size: target size of Connection object cache, in number of objects version_pool_size: expected maximum number of connections (per version) version_cache_size: target size of Connection object cache for version connections, in number of objects database_name: when using a multi-database, the name of this DB within the database group. It's a (detected) error if databases is specified too and database_name is already a key in it. This becomes the value of the DB's database_name attribute. databases: when using a multi-database, a mapping to use as the binding of this DB's .databases attribute. It's intended that the second and following DB's added to a multi-database pass the .databases attribute set on the first DB added to the collection. """ databases = Attribute("""\ A mapping from database name to DB (database) object. In multi-database use, all DB members of a database collection share the same .databases object. In single-database use, of course this mapping contains a single entry. """) class IStorage(Interface): """A storage is responsible for storing and retrieving data of objects. """ ## What follows is the union of methods found across various storage ## implementations. Exactly what "the storage API" is and means has ## become fuzzy over time. Methods should be uncommented here, or ## even deleted, as the storage API regains a crisp definition. ## def load(oid, version): ## """TODO""" ## ## def close(): ## """TODO""" ## ## def cleanup(): ## """TODO""" ## ## def lastSerial(): ## """TODO""" ## ## def lastTransaction(): ## """TODO""" ## ## def lastTid(oid): ## """Return last serialno committed for object oid.""" ## ## def loadSerial(oid, serial): ## """TODO""" ## ## def loadBefore(oid, tid): ## """TODO""" ## ## def iterator(start=None, stop=None): ## """TODO""" ## ## def sortKey(): ## """TODO""" ## ## def getName(): ## """TODO""" ## ## def getSize(): ## """TODO""" ## ## def history(oid, version, length=1, filter=None): ## """TODO""" ## ## def new_oid(): ## """TODO""" ## ## def set_max_oid(possible_new_max_oid): ## """TODO""" ## ## def registerDB(db, limit): ## """TODO""" ## ## def isReadOnly(): ## """TODO""" ## ## def supportsUndo(): ## """TODO""" ## ## def supportsVersions(): ## """TODO""" ## ## def tpc_abort(transaction): ## """TODO""" ## ## def tpc_begin(transaction): ## """TODO""" ## ## def tpc_vote(transaction): ## """TODO""" ## ## def tpc_finish(transaction, f=None): ## """TODO""" ## ## def getSerial(oid): ## """TODO""" ## ## def loadSerial(oid, serial): ## """TODO""" ## ## def loadBefore(oid, tid): ## """TODO""" ## ## def getExtensionMethods(): ## """TODO""" ## ## def copyTransactionsFrom(): ## """TODO""" ## ## def store(oid, oldserial, data, version, transaction): ## """ ## ## may return the new serial or not ## """ class IStorageUndoable(IStorage): """A storage supporting transactional undo. """ def undo(transaction_id, txn): """TODO""" def undoLog(first, last, filter=(lambda desc: True)): """Return a sequence of descriptions for undoable transactions. Application code should call undoLog() on a DB instance instead of on the storage directly. A transaction description is a mapping with at least these keys: "time": The time, as float seconds since the epoch, when the transaction committed. "user_name": The value of the `.user` attribute on that transaction. "description": The value of the `.description` attribute on that transaction. "id`" A string uniquely identifying the transaction to the storage. If it's desired to undo this transaction, this is the `transaction_id` to pass to `undo()`. In addition, if any name+value pairs were added to the transaction by `setExtendedInfo()`, those may be added to the transaction description mapping too (for example, FileStorage's `undoLog()` does this). `filter` is a callable, taking one argument. A transaction description mapping is passed to `filter` for each potentially undoable transaction. The sequence returned by `undoLog()` excludes descriptions for which `filter` returns a false value. By default, `filter` always returns a true value. ZEO note: Arbitrary callables cannot be passed from a ZEO client to a ZEO server, and a ZEO client's implementation of `undoLog()` ignores any `filter` argument that may be passed. ZEO clients should use the related `undoInfo()` method instead (if they want to do filtering). Now picture a list containing descriptions of all undoable transactions that pass the filter, most recent transaction first (at index 0). The `first` and `last` arguments specify the slice of this (conceptual) list to be returned: `first`: This is the index of the first transaction description in the slice. It must be >= 0. `last`: If >= 0, first:last acts like a Python slice, selecting the descriptions at indices `first`, first+1, ..., up to but not including index `last`. At most last-first descriptions are in the slice, and `last` should be at least as large as `first` in this case. If `last` is less than 0, then abs(last) is taken to be the maximum number of descriptions in the slice (which still begins at index `first`). When `last` < 0, the same effect could be gotten by passing the positive first-last for `last` instead. """ def undoInfo(first, last, specification=None): """Return a sequence of descriptions for undoable transactions. This is like `undoLog()`, except for the `specification` argument. If given, `specification` is a dictionary, and `undoInfo()` synthesizes a `filter` function `f` for `undoLog()` such that `f(desc)` returns true for a transaction description mapping `desc` if and only if `desc` maps each key in `specification` to the same value `specification` maps that key to. In other words, only extensions (or supersets) of `specification` match. ZEO note: `undoInfo()` passes the `specification` argument from a ZEO client to its ZEO server (while a ZEO client ignores any `filter` argument passed to `undoLog()`). """ def pack(t, referencesf): """TODO""" class IStorageVersioning(IStorage): """A storage supporting versions. """ ## What follows is the union of methods found across various version storage ## implementations. Exactly what "the storage API" is and means has ## become fuzzy over time. Methods should be uncommented here, or ## even deleted, as the storage API regains a crisp definition. ## def abortVersion(src, transaction): ## """TODO""" ## ## def commitVersion(src, dest, transaction): ## """TODO""" ## ## def modifiedInVersion(oid): ## """TODO""" ## ## def versionEmpty(version): ## """TODO""" ## ## def versions(max=None): ## """TODO""" Zope-2.8.7-final/lib/python/ZODB/ExportImport.py0000664000076400007640000001117310435502423020414 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Support for database export and import.""" from cStringIO import StringIO from cPickle import Pickler, Unpickler from tempfile import TemporaryFile import logging from ZODB.POSException import ExportError from ZODB.utils import p64, u64 from ZODB.serialize import referencesf logger = logging.getLogger('ZODB.ExportImport') class ExportImport: def exportFile(self, oid, f=None): if f is None: f = TemporaryFile() elif isinstance(f, str): f = open(f,'w+b') f.write('ZEXP') oids = [oid] done_oids = {} done=done_oids.has_key load=self._storage.load while oids: oid = oids.pop(0) if oid in done_oids: continue done_oids[oid] = True try: p, serial = load(oid, self._version) except: logger.debug("broken reference for oid %s", repr(oid), exc_info=True) else: referencesf(p, oids) f.writelines([oid, p64(len(p)), p]) f.write(export_end_marker) return f def importFile(self, f, clue='', customImporters=None): # This is tricky, because we need to work in a transaction! if isinstance(f, str): f = open(f, 'rb') magic = f.read(4) if magic != 'ZEXP': if customImporters and customImporters.has_key(magic): f.seek(0) return customImporters[magic](self, f, clue) raise ExportError("Invalid export header") t = self.transaction_manager.get() if clue: t.note(clue) return_oid_list = [] self._import = f, return_oid_list self._register() t.savepoint(optimistic=True) # Return the root imported object. if return_oid_list: return self.get(return_oid_list[0]) else: return None def _importDuringCommit(self, transaction, f, return_oid_list): """Import data during two-phase commit. Invoked by the transaction manager mid commit. Appends one item, the OID of the first object created, to return_oid_list. """ oids = {} def persistent_load(ooid): """Remap a persistent id to a new ID and create a ghost for it.""" klass = None if isinstance(ooid, tuple): ooid, klass = ooid if ooid in oids: oid = oids[ooid] else: if klass is None: oid = self._storage.new_oid() else: oid = self._storage.new_oid(), klass oids[ooid] = oid return Ghost(oid) version = self._version while 1: h = f.read(16) if h == export_end_marker: break if len(h) != 16: raise ExportError("Truncated export file") l = u64(h[8:16]) p = f.read(l) if len(p) != l: raise ExportError("Truncated export file") ooid = h[:8] if oids: oid = oids[ooid] if isinstance(oid, tuple): oid = oid[0] else: oids[ooid] = oid = self._storage.new_oid() return_oid_list.append(oid) pfile = StringIO(p) unpickler = Unpickler(pfile) unpickler.persistent_load = persistent_load newp = StringIO() pickler = Pickler(newp, 1) pickler.persistent_id = persistent_id pickler.dump(unpickler.load()) pickler.dump(unpickler.load()) p = newp.getvalue() self._storage.store(oid, None, p, version, transaction) export_end_marker = '\377'*16 class Ghost(object): __slots__ = ("oid",) def __init__(self, oid): self.oid = oid def persistent_id(obj): if isinstance(obj, Ghost): return obj.oid Zope-2.8.7-final/lib/python/ZODB/serialize.py0000664000076400007640000004457610435502423017744 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Support for ZODB object serialization. ZODB serializes objects using a custom format based on Python pickles. When an object is unserialized, it can be loaded as either a ghost or a real object. A ghost is a persistent object of the appropriate type but without any state. The first time a ghost is accessed, the persistence machinery traps access and loads the actual state. A ghost allows many persistent objects to be loaded while minimizing the memory consumption of referenced but otherwise unused objects. Pickle format ------------- ZODB stores serialized objects using a custom format based on pickle. Each serialized object has two parts: the class description and the object state. The class description must provide enough information to call the class's ``__new__`` and create an empty object. Once the object exists as a ghost, its state is passed to ``__setstate__``. The class description can be in a variety of formats, in part to provide backwards compatibility with earlier versions of Zope. The four current formats for class description are: 1. type(obj) 2. type(obj), obj.__getnewargs__() 3. (module name, class name), None 7. (module name, class name), obj.__getnewargs__() The second of these options is used if the object has a __getnewargs__() method. It is intended to support objects like persistent classes that have custom C layouts that are determined by arguments to __new__(). The third and fourth (#3 & #7) apply to instances of a persistent class (which means the class itself is persistent, not that it's a subclass of Persistent). The type object is usually stored using the standard pickle mechanism, which involves the pickle GLOBAL opcode (giving the type's module and name as strings). The type may itself be a persistent object, in which case a persistent reference (see below) is used. It's unclear what "usually" means in the last paragraph. There are two useful places to concentrate confusion about exactly which formats exist: - ObjectReader.getClassName() below returns a dotted "module.class" string, via actually loading a pickle. This requires that the implementation of application objects be available. - ZODB/utils.py's get_pickle_metadata() tries to return the module and class names (as strings) without importing any application modules or classes, via analyzing the pickle. Earlier versions of Zope supported several other kinds of class descriptions. The current serialization code reads these descriptions, but does not write them. The three earlier formats are: 4. (module name, class name), __getinitargs__() 5. class, None 6. class, __getinitargs__() Formats 4 and 6 are used only if the class defines a __getinitargs__() method, but we really can't tell them apart from formats 7 and 2 (respectively). Formats 5 and 6 are used if the class does not have a __module__ attribute (I'm not sure when this applies, but I think it occurs for some but not all ZClasses). Persistent references --------------------- A persistent reference is a pair containing an oid and class metadata. When one persistent object pickle refers to another persistent object, the database uses a persistent reference. The format allows a significant optimization, because ghosts can be created directly from persistent references. If the reference was just an oid, a database access would be required to determine the class of the ghost. Because the persistent reference includes the class, it is not possible to change the class of a persistent object. If a transaction changed the class of an object, a new record with new class metadata would be written but all the old references would still include the old class. """ import cPickle import cStringIO import logging from persistent import Persistent from persistent.wref import WeakRefMarker, WeakRef from ZODB import broken from ZODB.broken import Broken from ZODB.POSException import InvalidObjectReference _oidtypes = str, type(None) # Might to update or redo coptimizations to reflect weakrefs: # from ZODB.coptimizations import new_persistent_id def myhasattr(obj, name, _marker=object()): """Make sure we don't mask exceptions like hasattr(). We don't want exceptions other than AttributeError to be masked, since that too often masks other programming errors. Three-argument getattr() doesn't mask those, so we use that to implement our own hasattr() replacement. """ return getattr(obj, name, _marker) is not _marker class ObjectWriter: """Serializes objects for storage in the database. The ObjectWriter creates object pickles in the ZODB format. It also detects new persistent objects reachable from the current object. """ _jar = None def __init__(self, obj=None): self._file = cStringIO.StringIO() self._p = cPickle.Pickler(self._file, 1) self._p.persistent_id = self.persistent_id self._stack = [] if obj is not None: self._stack.append(obj) jar = obj._p_jar assert myhasattr(jar, "new_oid") self._jar = jar def persistent_id(self, obj): """Return the persistent id for obj. >>> from ZODB.tests.util import P >>> class DummyJar: ... def new_oid(self): ... return 42 >>> jar = DummyJar() >>> class O: ... _p_jar = jar >>> writer = ObjectWriter(O) Normally, object references include the oid and a cached named reference to the class. Having the class information available allows fast creation of the ghost, avoiding requiring an additional database lookup. >>> bob = P('bob') >>> oid, cls = writer.persistent_id(bob) >>> oid 42 >>> cls is P True If a persistent object does not already have an oid and jar, these will be assigned by persistent_id(): >>> bob._p_oid 42 >>> bob._p_jar is jar True If the object already has a persistent id, the id is not changed: >>> bob._p_oid = 24 >>> oid, cls = writer.persistent_id(bob) >>> oid 24 >>> cls is P True If the jar doesn't match that of the writer, an error is raised: >>> bob._p_jar = DummyJar() >>> writer.persistent_id(bob) Traceback (most recent call last): ... InvalidObjectReference: Attempt to store an object from a """ \ """foreign database connection Constructor arguments used by __new__(), as returned by __getnewargs__(), can affect memory allocation, but may also change over the life of the object. This makes it useless to cache even the object's class. >>> class PNewArgs(P): ... def __getnewargs__(self): ... return () >>> sam = PNewArgs('sam') >>> writer.persistent_id(sam) 42 >>> sam._p_oid 42 >>> sam._p_jar is jar True Check that simple objects don't get accused of persistence: >>> writer.persistent_id(42) >>> writer.persistent_id(object()) Check that a classic class doesn't get identified improperly: >>> class ClassicClara: ... pass >>> clara = ClassicClara() >>> writer.persistent_id(clara) """ # Most objects are not persistent. The following cheap test # identifies most of them. For these, we return None, # signalling that the object should be pickled normally. if not isinstance(obj, (Persistent, type, WeakRef)): # Not persistent, pickle normally return None # Any persistent object must have an oid: try: oid = obj._p_oid except AttributeError: # Not persistent, pickle normally return None if not (oid is None or isinstance(oid, str)): # Deserves a closer look: # Make sure it's not a descriptor if hasattr(oid, '__get__'): # The oid is a descriptor. That means obj is a non-persistent # class whose instances are persistent, so ... # Not persistent, pickle normally return None if oid is WeakRefMarker: # we have a weakref, see weakref.py oid = obj.oid if oid is None: obj = obj() # get the referenced object oid = obj._p_oid if oid is None: # Here we are causing the object to be saved in # the database. One could argue that we shouldn't # do this, because a weakref should not cause an object # to be added. We'll be optimistic, though, and # assume that the object will be added eventually. oid = self._jar.new_oid() obj._p_jar = self._jar obj._p_oid = oid self._stack.append(obj) return [oid] # Since we have an oid, we have either a persistent instance # (an instance of Persistent), or a persistent class. # NOTE! Persistent classes don't (and can't) subclass persistent. if oid is None: oid = obj._p_oid = self._jar.new_oid() obj._p_jar = self._jar self._stack.append(obj) elif obj._p_jar is not self._jar: raise InvalidObjectReference( "Attempt to store an object from a foreign " "database connection" ) klass = type(obj) if hasattr(klass, '__getnewargs__'): # We don't want to save newargs in object refs. # It's possible that __getnewargs__ is degenerate and # returns (), but we don't want to have to deghostify # the object to find out. # Note that this has the odd effect that, if the class has # __getnewargs__ of its own, we'll lose the optimization # of caching the class info. return oid # Note that we never get here for persistent classes. # We'll use driect refs for normal classes. return oid, klass def serialize(self, obj): # We don't use __class__ here, because obj could be a persistent proxy. # We don't want to be fooled by proxies. klass = type(obj) # We want to serialize persistent classes by name if they have # a non-None non-empty module so as not to have a direct # ref. This is important when copying. We probably want to # revisit this in the future. newargs = getattr(obj, "__getnewargs__", None) if (isinstance(getattr(klass, '_p_oid', 0), _oidtypes) and klass.__module__): # This is a persistent class with a non-empty module. This # uses pickle format #3 or #7. klass = klass.__module__, klass.__name__ if newargs is None: meta = klass, None else: meta = klass, newargs() elif newargs is None: # Pickle format #1. meta = klass else: # Pickle format #2. meta = klass, newargs() return self._dump(meta, obj.__getstate__()) def _dump(self, classmeta, state): # To reuse the existing cStringIO object, we must reset # the file position to 0 and truncate the file after the # new pickle is written. self._file.seek(0) self._p.clear_memo() self._p.dump(classmeta) self._p.dump(state) self._file.truncate() return self._file.getvalue() def __iter__(self): return NewObjectIterator(self._stack) class NewObjectIterator: # The pickler is used as a forward iterator when the connection # is looking for new objects to pickle. def __init__(self, stack): self._stack = stack def __iter__(self): return self def next(self): if self._stack: elt = self._stack.pop() return elt else: raise StopIteration class ObjectReader: def __init__(self, conn=None, cache=None, factory=None): self._conn = conn self._cache = cache self._factory = factory def _get_class(self, module, name): return self._factory(self._conn, module, name) def _get_unpickler(self, pickle): file = cStringIO.StringIO(pickle) unpickler = cPickle.Unpickler(file) unpickler.persistent_load = self._persistent_load factory = self._factory conn = self._conn def find_global(modulename, name): return factory(conn, modulename, name) unpickler.find_global = find_global return unpickler def _persistent_load(self, oid): if isinstance(oid, tuple): # Quick instance reference. We know all we need to know # to create the instance w/o hitting the db, so go for it! oid, klass = oid obj = self._cache.get(oid, None) if obj is not None: return obj if isinstance(klass, tuple): klass = self._get_class(*klass) if issubclass(klass, Broken): # We got a broken class. We might need to make it # PersistentBroken if not issubclass(klass, broken.PersistentBroken): klass = broken.persistentBroken(klass) try: obj = klass.__new__(klass) except TypeError: # Couldn't create the instance. Maybe there's more # current data in the object's actual record! return self._conn.get(oid) # TODO: should be done by connection obj._p_oid = oid obj._p_jar = self._conn # When an object is created, it is put in the UPTODATE # state. We must explicitly deactivate it to turn it into # a ghost. obj._p_changed = None self._cache[oid] = obj return obj elif isinstance(oid, list): # see weakref.py [oid] = oid obj = WeakRef.__new__(WeakRef) obj.oid = oid obj.dm = self._conn return obj obj = self._cache.get(oid, None) if obj is not None: return obj return self._conn.get(oid) def _new_object(self, klass, args): if not args and not myhasattr(klass, "__getnewargs__"): obj = klass.__new__(klass) else: obj = klass(*args) if not isinstance(klass, type): obj.__dict__.clear() return obj def getClassName(self, pickle): unpickler = self._get_unpickler(pickle) klass = unpickler.load() if isinstance(klass, tuple): klass, args = klass if isinstance(klass, tuple): # old style reference return "%s.%s" % klass return "%s.%s" % (klass.__module__, klass.__name__) def getGhost(self, pickle): unpickler = self._get_unpickler(pickle) klass = unpickler.load() if isinstance(klass, tuple): # Here we have a separate class and args. # This could be an old record, so the class module ne a named # refernce klass, args = klass if isinstance(klass, tuple): # Old module_name, class_name tuple klass = self._get_class(*klass) if args is None: args = () else: # Definitely new style direct class reference args = () if issubclass(klass, Broken): # We got a broken class. We might need to make it # PersistentBroken if not issubclass(klass, broken.PersistentBroken): klass = broken.persistentBroken(klass) return klass.__new__(klass, *args) def getState(self, pickle): unpickler = self._get_unpickler(pickle) try: unpickler.load() # skip the class metadata return unpickler.load() except EOFError, msg: log = logging.getLogger("ZODB.serialize") log.exception("Unpickling error: %r", pickle) raise def setGhostState(self, obj, pickle): state = self.getState(pickle) obj.__setstate__(state) def referencesf(p, rootl=None): if rootl is None: rootl = [] u = cPickle.Unpickler(cStringIO.StringIO(p)) l = len(rootl) u.persistent_load = rootl u.noload() try: u.noload() except: # Hm. We failed to do second load. Maybe there wasn't a # second pickle. Let's check: f = cStringIO.StringIO(p) u = cPickle.Unpickler(f) u.persistent_load = [] u.noload() if len(p) > f.tell(): raise ValueError, 'Error unpickling %r' % p # References may be: # # - A tuple, in which case they are an oid and class. # In this case, just extract the first element, which is # the oid # # - A list, which is a weak reference. We skip those. # # - Anything else must be an oid. This means that an oid # may not be a list or a tuple. This is a bit lame. # We could avoid this lamosity by allowing single-element # tuples, so that we wrap oids that are lists or tuples in # tuples. # # - oids may *not* be False. I'm not sure why. out = [] for v in rootl: assert v # Let's see if we ever get empty ones if type(v) is list: # skip wekrefs continue if type(v) is tuple: v = v[0] out.append(v) rootl[:] = out return rootl Zope-2.8.7-final/lib/python/ZODB/subtransactions.txt0000664000076400007640000000264210435502423021352 0ustar ajungajung========================= Subtransactions in ZODB 3 ========================= ZODB 3 provides limited support for subtransactions. Subtransactions are nested to *one* level. There are top-level transactions and subtransactions. When a transaction is committed, a flag is passed indicating whether it is a subtransaction or a top-level transaction. Consider the following exampler commit calls: - commit() A regular top-level transaction is committed. - commit(1) A subtransaction is committed. There is now one subtransaction of the current top-level transaction. - commit(1) A subtransaction is committed. There are now two subtransactions of the current top-level transaction. - abort(1) A subtransaction is aborted. There are still two subtransactions of the current top-level transaction; work done since the last commit(1) call is discarded. - commit() We now commit a top-level transaction. The work done in the previous two subtransactions *plus* work done since the last abort(1) call is saved. - commit(1) A subtransaction is committed. There is now one subtransaction of the current top-level transaction. - commit(1) A subtransaction is committed. There are now two subtransactions of the current top-level transaction. - abort() We now abort a top-level transaction. We discard the work done in the previous two subtransactions *plus* work done since the last commit(1) call. Zope-2.8.7-final/lib/python/ZODB/POSException.py0000664000076400007640000002505610435502423020265 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ZODB-defined exceptions $Id: POSException.py 29676 2005-03-24 23:04:52Z tim_one $""" from ZODB.utils import oid_repr, readable_tid_repr def _fmt_undo(oid, reason): s = reason and (": %s" % reason) or "" return "Undo error %s%s" % (oid_repr(oid), s) class POSError(StandardError): """Persistent object system error.""" class POSKeyError(KeyError, POSError): """Key not found in database.""" def __str__(self): return oid_repr(self.args[0]) class TransactionError(POSError): """An error occured due to normal transaction processing.""" class TransactionFailedError(POSError): """Cannot perform an operation on a transaction that previously failed. An attempt was made to commit a transaction, or to join a transaction, but this transaction previously raised an exception during an attempt to commit it. The transaction must be explicitly aborted, either by invoking abort() on the transaction, or begin() on its transaction manager. """ class ConflictError(TransactionError): """Two transactions tried to modify the same object at once. This transaction should be resubmitted. Instance attributes: oid : string the OID (8-byte packed string) of the object in conflict class_name : string the fully-qualified name of that object's class message : string a human-readable explanation of the error serials : (string, string) a pair of 8-byte packed strings; these are the serial numbers related to conflict. The first is the revision of object that is in conflict, the currently committed serial. The second is the revision the current transaction read when it started. data : string The database record that failed to commit, used to put the class name in the error message. The caller should pass either object or oid as a keyword argument, but not both of them. If object is passed, it should be a persistent object with an _p_oid attribute. """ def __init__(self, message=None, object=None, oid=None, serials=None, data=None): if message is None: self.message = "database conflict error" else: self.message = message if object is None: self.oid = None self.class_name = None else: self.oid = object._p_oid klass = object.__class__ self.class_name = klass.__module__ + "." + klass.__name__ if oid is not None: assert self.oid is None self.oid = oid if data is not None: # avoid circular import chain from ZODB.utils import get_pickle_metadata self.class_name = "%s.%s" % get_pickle_metadata(data) ## else: ## if message != "data read conflict error": ## raise RuntimeError self.serials = serials def __str__(self): extras = [] if self.oid: extras.append("oid %s" % oid_repr(self.oid)) if self.class_name: extras.append("class %s" % self.class_name) if self.serials: current, old = self.serials extras.append("serial this txn started with %s" % readable_tid_repr(old)) extras.append("serial currently committed %s" % readable_tid_repr(current)) if extras: return "%s (%s)" % (self.message, ", ".join(extras)) else: return self.message def get_oid(self): return self.oid def get_class_name(self): return self.class_name def get_old_serial(self): return self.serials[1] def get_new_serial(self): return self.serials[0] def get_serials(self): return self.serials class ReadConflictError(ConflictError): """Conflict detected when object was loaded. An attempt was made to read an object that has changed in another transaction (eg. another thread or process). """ def __init__(self, message=None, object=None, serials=None): if message is None: message = "database read conflict error" ConflictError.__init__(self, message=message, object=object, serials=serials) class BTreesConflictError(ConflictError): """A special subclass for BTrees conflict errors.""" msgs = [# 0; i2 or i3 bucket split; positions are all -1 'Conflicting bucket split', # 1; keys the same, but i2 and i3 values differ, and both values # differ from i1's value 'Conflicting changes', # 2; i1's value changed in i2, but key+value deleted in i3 'Conflicting delete and change', # 3; i1's value changed in i3, but key+value deleted in i2 'Conflicting delete and change', # 4; i1 and i2 both added the same key, or both deleted the # same key 'Conflicting inserts or deletes', # 5; i2 and i3 both deleted the same key 'Conflicting deletes', # 6; i2 and i3 both added the same key 'Conflicting inserts', # 7; i2 and i3 both deleted the same key, or i2 changed the value # associated with a key and i3 deleted that key 'Conflicting deletes, or delete and change', # 8; i2 and i3 both deleted the same key, or i3 changed the value # associated with a key and i2 deleted that key 'Conflicting deletes, or delete and change', # 9; i2 and i3 both deleted the same key 'Conflicting deletes', # 10; i2 and i3 deleted all the keys, and didn't insert any, # leaving an empty bucket; conflict resolution doesn't have # enough info to unlink an empty bucket from its containing # BTree correctly 'Empty bucket from deleting all keys', # 11; conflicting changes in an internal BTree node 'Conflicting changes in an internal BTree node', # 12; i2 or i3 was empty 'Empty bucket in a transaction', ] def __init__(self, p1, p2, p3, reason): self.p1 = p1 self.p2 = p2 self.p3 = p3 self.reason = reason def __repr__(self): return "BTreesConflictError(%d, %d, %d, %d)" % (self.p1, self.p2, self.p3, self.reason) def __str__(self): return "BTrees conflict error at %d/%d/%d: %s" % ( self.p1, self.p2, self.p3, self.msgs[self.reason]) class DanglingReferenceError(TransactionError): """An object has a persistent reference to a missing object. If an object is stored and it has a reference to another object that does not exist (for example, it was deleted by pack), this exception may be raised. Whether a storage supports this feature, it a quality of implementation issue. Instance attributes: referer: oid of the object being written missing: referenced oid that does not have a corresponding object """ def __init__(self, Aoid, Boid): self.referer = Aoid self.missing = Boid def __str__(self): return "from %s to %s" % (oid_repr(self.referer), oid_repr(self.missing)) class VersionError(POSError): """An error in handling versions occurred.""" class VersionCommitError(VersionError): """An invalid combination of versions was used in a version commit.""" class VersionLockError(VersionError, TransactionError): """Modification to an object modified in an unsaved version. An attempt was made to modify an object that has been modified in an unsaved version. """ class UndoError(POSError): """An attempt was made to undo a non-undoable transaction.""" def __init__(self, reason, oid=None): self._reason = reason self._oid = oid def __str__(self): return _fmt_undo(self._oid, self._reason) class MultipleUndoErrors(UndoError): """Several undo errors occured during a single transaction.""" def __init__(self, errs): # provide a reason and oid for clients that only look at that UndoError.__init__(self, *errs[0]) self._errs = errs def __str__(self): return "\n".join([_fmt_undo(*pair) for pair in self._errs]) class StorageError(POSError): """Base class for storage based exceptions.""" class StorageTransactionError(StorageError): """An operation was invoked for an invalid transaction or state.""" class StorageSystemError(StorageError): """Panic! Internal storage error!""" class MountedStorageError(StorageError): """Unable to access mounted storage.""" class ReadOnlyError(StorageError): """Unable to modify objects in a read-only storage.""" class TransactionTooLargeError(StorageTransactionError): """The transaction exhausted some finite storage resource.""" class ExportError(POSError): """An export file doesn't have the right format.""" class Unsupported(POSError): """A feature was used that is not supported by the storage.""" class InvalidObjectReference(POSError): """An object contains an invalid reference to another object. An invalid reference may be one of: o A reference to a wrapped persistent object. o A reference to an object in a different database connection. TODO: The exception ought to have a member that is the invalid object. """ class ConnectionStateError(POSError): """A Connection isn't in the required state for an operation. o An operation such as a load is attempted on a closed connection. o An attempt to close a connection is made while the connection is still joined to a transaction (for example, a transaction is in progress, with uncommitted modifications in the connection). """ Zope-2.8.7-final/lib/python/ZODB/fsrecover.py0000664000076400007640000002414510435502423017741 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Simple script for repairing damaged FileStorage files. Usage: %s [-f] [-v level] [-p] [-P seconds] input output Recover data from a FileStorage data file, skipping over damaged data. Any damaged data will be lost. This could lead to useless output if critical data is lost. Options: -f Overwrite output file even if it exists. -v level Set the verbosity level: 0 -- show progress indicator (default) 1 -- show transaction times and sizes 2 -- show transaction times and sizes, and show object (record) ids, versions, and sizes -p Copy partial transactions. If a data record in the middle of a transaction is bad, the data up to the bad data are packed. The output record is marked as packed. If this option is not used, transactions with any bad data are skipped. -P t Pack data to t seconds in the past. Note that if the "-p" option is used, then t should be 0. Important: The ZODB package must be importable. You may need to adjust PYTHONPATH accordingly. """ # Algorithm: # # position to start of input # while 1: # if end of file: # break # try: # copy_transaction # except: # scan for transaction # continue import sys import os import getopt import time from struct import unpack from cPickle import loads try: import ZODB except ImportError: if os.path.exists('ZODB'): sys.path.append('.') elif os.path.exists('FileStorage.py'): sys.path.append('..') import ZODB import ZODB.FileStorage from ZODB.utils import u64 from ZODB.FileStorage import RecordIterator from persistent.TimeStamp import TimeStamp def die(mess='', show_docstring=False): if mess: print >> sys.stderr, mess + '\n' if show_docstring: print >> sys.stderr, __doc__ % sys.argv[0] sys.exit(1) class ErrorFound(Exception): pass def error(mess, *args): raise ErrorFound(mess % args) def read_txn_header(f, pos, file_size, outp, ltid): # Read the transaction record f.seek(pos) h = f.read(23) if len(h) < 23: raise EOFError tid, stl, status, ul, dl, el = unpack(">8s8scHHH",h) tl = u64(stl) if pos + (tl + 8) > file_size: error("bad transaction length at %s", pos) if tl < (23 + ul + dl + el): error("invalid transaction length, %s, at %s", tl, pos) if ltid and tid < ltid: error("time-stamp reducation %s < %s, at %s", u64(tid), u64(ltid), pos) if status == "c": truncate(f, pos, file_size, outp) raise EOFError if status not in " up": error("invalid status, %r, at %s", status, pos) tpos = pos tend = tpos + tl if status == "u": # Undone transaction, skip it f.seek(tend) h = f.read(8) if h != stl: error("inconsistent transaction length at %s", pos) pos = tend + 8 return pos, None, tid pos = tpos+(23+ul+dl+el) user = f.read(ul) description = f.read(dl) if el: try: e=loads(f.read(el)) except: e={} else: e={} result = RecordIterator(tid, status, user, description, e, pos, tend, f, tpos) pos = tend # Read the (intentionally redundant) transaction length f.seek(pos) h = f.read(8) if h != stl: error("redundant transaction length check failed at %s", pos) pos += 8 return pos, result, tid def truncate(f, pos, file_size, outp): """Copy data from pos to end of f to a .trNNN file.""" # _trname is global so that the test suite can know the path too (in # order to delete the file when the test ends). global _trname i = 0 while 1: _trname = outp + ".tr%d" % i if os.path.exists(_trname): i += 1 else: break tr = open(_trname, "wb") copy(f, tr, file_size - pos) f.seek(pos) tr.close() def copy(src, dst, n): while n: buf = src.read(8096) if not buf: break if len(buf) > n: buf = buf[:n] dst.write(buf) n -= len(buf) def scan(f, pos): """Return a potential transaction location following pos in f. This routine scans forward from pos looking for the last data record in a transaction. A period '.' always occurs at the end of a pickle, and an 8-byte transaction length follows the last pickle. If a period is followed by a plausible 8-byte transaction length, assume that we have found the end of a transaction. The caller should try to verify that the returned location is actually a transaction header. """ while 1: f.seek(pos) data = f.read(8096) if not data: return 0 s = 0 while 1: l = data.find(".", s) if l < 0: pos += len(data) break # If we are less than 8 bytes from the end of the # string, we need to read more data. s = l + 1 if s > len(data) - 8: pos += l break tl = u64(data[s:s+8]) if tl < pos: return pos + s + 8 def iprogress(i): if i % 2: print ".", else: print (i/2) % 10, sys.stdout.flush() def progress(p): for i in range(p): iprogress(i) def main(): try: opts, args = getopt.getopt(sys.argv[1:], "fv:pP:") except getopt.error, msg: die(str(msg), show_docstring=True) if len(args) != 2: die("two positional arguments required", show_docstring=True) inp, outp = args force = partial = False verbose = 0 pack = None for opt, v in opts: if opt == "-v": verbose = int(v) elif opt == "-p": partial = True elif opt == "-f": force = True elif opt == "-P": pack = time.time() - float(v) recover(inp, outp, verbose, partial, force, pack) def recover(inp, outp, verbose=0, partial=False, force=False, pack=None): print "Recovering", inp, "into", outp if os.path.exists(outp) and not force: die("%s exists" % outp) f = open(inp, "rb") if f.read(4) != ZODB.FileStorage.packed_version: die("input is not a file storage") f.seek(0,2) file_size = f.tell() ofs = ZODB.FileStorage.FileStorage(outp, create=1) _ts = None ok = 1 prog1 = 0 undone = 0 pos = 4L ltid = None while pos: try: npos, txn, tid = read_txn_header(f, pos, file_size, outp, ltid) except EOFError: break except (KeyboardInterrupt, SystemExit): raise except Exception, err: print "error reading txn header:", err if not verbose: progress(prog1) pos = scan(f, pos) if verbose > 1: print "looking for valid txn header at", pos continue ltid = tid if txn is None: undone = undone + npos - pos pos = npos continue else: pos = npos tid = txn.tid if _ts is None: _ts = TimeStamp(tid) else: t = TimeStamp(tid) if t <= _ts: if ok: print ("Time stamps out of order %s, %s" % (_ts, t)) ok = 0 _ts = t.laterThan(_ts) tid = `_ts` else: _ts = t if not ok: print ("Time stamps back in order %s" % (t)) ok = 1 ofs.tpc_begin(txn, tid, txn.status) if verbose: print "begin", pos, _ts, if verbose > 1: print sys.stdout.flush() nrec = 0 try: for r in txn: if verbose > 1: if r.data is None: l = "bp" else: l = len(r.data) print "%7d %s %s" % (u64(r.oid), l, r.version) ofs.restore(r.oid, r.tid, r.data, r.version, r.data_txn, txn) nrec += 1 except (KeyboardInterrupt, SystemExit): raise except Exception, err: if partial and nrec: ofs._status = "p" ofs.tpc_vote(txn) ofs.tpc_finish(txn) if verbose: print "partial" else: ofs.tpc_abort(txn) print "error copying transaction:", err if not verbose: progress(prog1) pos = scan(f, pos) if verbose > 1: print "looking for valid txn header at", pos else: ofs.tpc_vote(txn) ofs.tpc_finish(txn) if verbose: print "finish" sys.stdout.flush() if not verbose: prog = pos * 20l / file_size while prog > prog1: prog1 = prog1 + 1 iprogress(prog1) bad = file_size - undone - ofs._pos print "\n%s bytes removed during recovery" % bad if undone: print "%s bytes of undone transaction data were skipped" % undone if pack is not None: print "Packing ..." from ZODB.serialize import referencesf ofs.pack(pack, referencesf) ofs.close() if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZODB/lock_file.py0000664000076400007640000000436510435502423017674 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os import errno import logging logger = logging.getLogger("ZODB.lock_file") try: import fcntl except ImportError: try: from winlock import LockFile as _LockFile from winlock import UnlockFile as _UnlockFile except ImportError: def lock_file(file): logger.info('No file-locking support on this platform') # Windows def lock_file(file): # Lock just the first byte _LockFile(file.fileno(), 0, 0, 1, 0) def unlock_file(file): _UnlockFile(file.fileno(), 0, 0, 1, 0) else: # Unix _flags = fcntl.LOCK_EX | fcntl.LOCK_NB def lock_file(file): fcntl.flock(file.fileno(), _flags) def unlock_file(file): # File is automatically unlocked on close pass # This is a better interface to use than the lockfile.lock_file() interface. # Creating the instance acquires the lock. The file remains open. Calling # close both closes and unlocks the lock file. class LockFile: def __init__(self, path): self._path = path try: self._fp = open(path, 'r+') except IOError, e: if e.errno <> errno.ENOENT: raise self._fp = open(path, 'w+') # Acquire the lock and piss on the hydrant try: lock_file(self._fp) except: logger.exception("Error locking file %s", path) raise print >> self._fp, os.getpid() self._fp.flush() def close(self): if self._fp is not None: unlock_file(self._fp) self._fp.close() os.unlink(self._path) self._fp = None Zope-2.8.7-final/lib/python/ZODB/conversionhack.py0000664000076400007640000000201410435502423020746 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import persistent.mapping class fixer: def __of__(self, parent): def __setstate__(state, self=parent): self._container=state del self.__setstate__ return __setstate__ fixer=fixer() class hack: pass hack=hack() def __basicnew__(): r=persistent.mapping.PersistentMapping() r.__setstate__=fixer return r hack.__basicnew__=__basicnew__ Zope-2.8.7-final/lib/python/ZODB/winlock.c0000775000076400007640000000530510435502423017203 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ static char winlock_doc_string[] = "Lock files on Windows." "\n" "$Id: winlock.c 25186 2004-06-02 15:07:33Z jim $\n"; #include "Python.h" static PyObject *Error; #ifdef MS_WIN32 #include #include /* LOCK_FUNC is the shared type of Win32 LockFile and UnlockFile. */ typedef WINBASEAPI BOOL WINAPI LOCK_FUNC(HANDLE, DWORD, DWORD, DWORD, DWORD); static PyObject * common(LOCK_FUNC func, PyObject *args) { int fileno; long h, ofslo, ofshi, lenlo, lenhi; if (! PyArg_ParseTuple(args, "illll", &fileno, &ofslo, &ofshi, &lenlo, &lenhi)) return NULL; h = _get_osfhandle(fileno); if (h == -1) { PyErr_SetString(Error, "_get_osfhandle failed"); return NULL; } if (func((HANDLE)h, ofslo, ofshi, lenlo, lenhi)) { Py_INCREF(Py_None); return Py_None; } PyErr_SetObject(Error, PyInt_FromLong(GetLastError())); return NULL; } static PyObject * winlock(PyObject *ignored, PyObject *args) { return common(LockFile, args); } static PyObject * winunlock(PyObject *ignored, PyObject *args) { return common(UnlockFile, args); } static struct PyMethodDef methods[] = { {"LockFile", (PyCFunction)winlock, METH_VARARGS, "LockFile(fileno, offsetLow, offsetHigh, lengthLow, lengthHigh) -- " "Lock the file associated with fileno"}, {"UnlockFile", (PyCFunction)winunlock, METH_VARARGS, "UnlockFile(fileno, offsetLow, offsetHigh, lengthLow, lengthHigh) -- " "Unlock the file associated with fileno"}, {NULL, NULL} /* sentinel */ }; #else static struct PyMethodDef methods[] = { {NULL, NULL} /* sentinel */ }; #endif /* Initialization function for the module (*must* be called initcStringIO) */ #ifndef DL_EXPORT /* declarations for DLL import/export */ #define DL_EXPORT(RTYPE) RTYPE #endif DL_EXPORT(void) initwinlock(void) { PyObject *m, *d; if (!(Error=PyString_FromString("winlock.error"))) return; /* Create the module and add the functions */ m = Py_InitModule4("winlock", methods, winlock_doc_string, (PyObject*)NULL, PYTHON_API_VERSION); d = PyModule_GetDict(m); PyDict_SetItemString(d, "error", Error); } Zope-2.8.7-final/lib/python/ZODB/MappingStorage.py0000664000076400007640000001120710435502423020656 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Very Simple Mapping ZODB storage The Mapping storage provides an extremely simple storage implementation that doesn't provide undo or version support. It is meant to illustrate the simplest possible storage. The Mapping storage uses a single data structure to map object ids to data. """ from ZODB.utils import u64, z64 from ZODB.BaseStorage import BaseStorage from ZODB import POSException from persistent.TimeStamp import TimeStamp class MappingStorage(BaseStorage): def __init__(self, name='Mapping Storage'): BaseStorage.__init__(self, name) # ._index maps an oid to a string s. s[:8] is the tid of the # transaction that created oid's current state, and s[8:] is oid's # current state. self._index = {} self._clear_temp() self._ltid = None # Note: If you subclass this and use a persistent mapping facility # (e.g. a dbm file), you will need to get the maximum key and save it # as self._oid. See dbmStorage. def __len__(self): return len(self._index) def getSize(self): self._lock_acquire() try: # These constants are for Python object memory overheads. Heh. s = 32 for p in self._index.itervalues(): s += 56 + len(p) return s finally: self._lock_release() def load(self, oid, version): self._lock_acquire() try: p = self._index[oid] return p[8:], p[:8] # pickle, serial finally: self._lock_release() def loadEx(self, oid, version): # Since we don't support versions, just tack the empty version # string onto load's result. return self.load(oid, version) + ("",) def getTid(self, oid): self._lock_acquire() try: # The tid is the first 8 bytes of the buffer. return self._index[oid][:8] finally: self._lock_release() def store(self, oid, serial, data, version, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) if version: raise POSException.Unsupported("Versions aren't supported") self._lock_acquire() try: if oid in self._index: oserial = self._index[oid][:8] if serial != oserial: raise POSException.ConflictError(oid=oid, serials=(oserial, serial), data=data) self._tindex[oid] = self._tid + data finally: self._lock_release() return self._tid def _clear_temp(self): # store() saves data in _tindex; if the transaction completes # successfully, _finish() merges _tindex into _index. self._tindex = {} def _finish(self, tid, user, desc, ext): self._index.update(self._tindex) self._ltid = self._tid def lastTransaction(self): return self._ltid def pack(self, t, referencesf): self._lock_acquire() try: if not self._index: return # Build an index of *only* those objects reachable from the root. rootl = [z64] pindex = {} while rootl: oid = rootl.pop() if oid not in pindex: # Scan non-version pickle for references. r = self._index[oid] pindex[oid] = r referencesf(r[8:], rootl) self._index = pindex finally: self._lock_release() def _splat(self): """Spit out a string showing state.""" o = ['Index:'] keys = self._index.keys() keys.sort() for oid in keys: r = self._index[oid] o.append(' %s: %s, %s' % (u64(oid), TimeStamp(r[:8]), repr(r[8:]))) return '\n'.join(o) Zope-2.8.7-final/lib/python/ZODB/tests/0000775000076400007640000000000010435610645016533 5ustar ajungajungZope-2.8.7-final/lib/python/ZODB/tests/HistoryStorage.py0000664000076400007640000002130410435502423022065 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Run the history() related tests for a storage. Any storage that supports the history() method should be able to pass all these tests. """ from ZODB.tests.MinPO import MinPO from transaction import Transaction class HistoryStorage: def checkSimpleHistory(self): eq = self.assertEqual # Store a couple of non-version revisions of the object oid = self._storage.new_oid() self.assertRaises(KeyError,self._storage.history,oid) revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) # Now get various snapshots of the object's history h = self._storage.history(oid, size=1) eq(len(h), 1) d = h[0] eq(d['tid'], revid3) eq(d['version'], '') # Try to get 2 historical revisions h = self._storage.history(oid, size=2) eq(len(h), 2) d = h[0] eq(d['tid'], revid3) eq(d['version'], '') d = h[1] eq(d['tid'], revid2) eq(d['version'], '') # Try to get all 3 historical revisions h = self._storage.history(oid, size=3) eq(len(h), 3) d = h[0] eq(d['tid'], revid3) eq(d['version'], '') d = h[1] eq(d['tid'], revid2) eq(d['version'], '') d = h[2] eq(d['tid'], revid1) eq(d['version'], '') # There should be no more than 3 revisions h = self._storage.history(oid, size=4) eq(len(h), 3) d = h[0] eq(d['tid'], revid3) eq(d['version'], '') d = h[1] eq(d['tid'], revid2) eq(d['version'], '') d = h[2] eq(d['tid'], revid1) eq(d['version'], '') def checkVersionHistory(self): if not self._storage.supportsVersions(): return eq = self.assertEqual # Store a couple of non-version revisions oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) # Now store some new revisions in a version version = 'test-version' revid4 = self._dostore(oid, revid=revid3, data=MinPO(14), version=version) revid5 = self._dostore(oid, revid=revid4, data=MinPO(15), version=version) revid6 = self._dostore(oid, revid=revid5, data=MinPO(16), version=version) # Now, try to get the six historical revisions (first three are in # 'test-version', followed by the non-version revisions). h = self._storage.history(oid, version, 100) eq(len(h), 6) d = h[0] eq(d['tid'], revid6) eq(d['version'], version) d = h[1] eq(d['tid'], revid5) eq(d['version'], version) d = h[2] eq(d['tid'], revid4) eq(d['version'], version) d = h[3] eq(d['tid'], revid3) eq(d['version'], '') d = h[4] eq(d['tid'], revid2) eq(d['version'], '') d = h[5] eq(d['tid'], revid1) eq(d['version'], '') def checkHistoryAfterVersionCommit(self): if not self._storage.supportsVersions(): return eq = self.assertEqual # Store a couple of non-version revisions oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) # Now store some new revisions in a version version = 'test-version' revid4 = self._dostore(oid, revid=revid3, data=MinPO(14), version=version) revid5 = self._dostore(oid, revid=revid4, data=MinPO(15), version=version) revid6 = self._dostore(oid, revid=revid5, data=MinPO(16), version=version) # Now commit the version t = Transaction() self._storage.tpc_begin(t) self._storage.commitVersion(version, '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) # After consultation with Jim, we agreed that the semantics of # revision id's after a version commit is that the committed object # gets a new serial number (a.k.a. revision id). Note that # FileStorage is broken here; the serial number in the post-commit # non-version revision will be the same as the serial number of the # previous in-version revision. # # BAW: Using load() is the only way to get the serial number of the # current revision of the object. But at least this works for both # broken and working storages. ign, revid7 = self._storage.load(oid, '') # Now, try to get the six historical revisions (first three are in # 'test-version', followed by the non-version revisions). h = self._storage.history(oid, version, 100) eq(len(h), 7) d = h[0] eq(d['tid'], revid7) eq(d['version'], '') d = h[1] eq(d['tid'], revid6) eq(d['version'], version) d = h[2] eq(d['tid'], revid5) eq(d['version'], version) d = h[3] eq(d['tid'], revid4) eq(d['version'], version) d = h[4] eq(d['tid'], revid3) eq(d['version'], '') d = h[5] eq(d['tid'], revid2) eq(d['version'], '') d = h[6] eq(d['tid'], revid1) eq(d['version'], '') def checkHistoryAfterVersionAbort(self): if not self._storage.supportsVersions(): return eq = self.assertEqual # Store a couple of non-version revisions oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) # Now store some new revisions in a version version = 'test-version' revid4 = self._dostore(oid, revid=revid3, data=MinPO(14), version=version) revid5 = self._dostore(oid, revid=revid4, data=MinPO(15), version=version) revid6 = self._dostore(oid, revid=revid5, data=MinPO(16), version=version) # Now commit the version t = Transaction() self._storage.tpc_begin(t) self._storage.abortVersion(version, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) # After consultation with Jim, we agreed that the semantics of # revision id's after a version commit is that the committed object # gets a new serial number (a.k.a. revision id). Note that # FileStorage is broken here; the serial number in the post-commit # non-version revision will be the same as the serial number of the # previous in-version revision. # # BAW: Using load() is the only way to get the serial number of the # current revision of the object. But at least this works for both # broken and working storages. ign, revid7 = self._storage.load(oid, '') # Now, try to get the six historical revisions (first three are in # 'test-version', followed by the non-version revisions). h = self._storage.history(oid, version, 100) eq(len(h), 7) d = h[0] eq(d['tid'], revid7) eq(d['version'], '') d = h[1] eq(d['tid'], revid6) eq(d['version'], version) d = h[2] eq(d['tid'], revid5) eq(d['version'], version) d = h[3] eq(d['tid'], revid4) eq(d['version'], version) d = h[4] eq(d['tid'], revid3) eq(d['version'], '') d = h[5] eq(d['tid'], revid2) eq(d['version'], '') d = h[6] eq(d['tid'], revid1) eq(d['version'], '') Zope-2.8.7-final/lib/python/ZODB/tests/test_fsdump.py0000664000076400007640000000452010435502423021435 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## r""" fsdump test =========== Let's get a temp file path to work with first. >>> import tempfile >>> path = tempfile.mktemp('.fs', 'Data') >>> print 'path:', path #doctest: +ELLIPSIS path: ...Data...fs More imports. >>> import ZODB >>> from ZODB.FileStorage import FileStorage >>> import transaction as txn >>> from BTrees.OOBTree import OOBTree >>> from ZODB.FileStorage.fsdump import fsdump # we're testing this Create an empty FileStorage. >>> st = FileStorage(path) For empty DB fsdump() output definitely empty: >>> fsdump(path) Create a root object and try again: >>> db = ZODB.DB(st) # yes, that creates a root object! >>> fsdump(path) #doctest: +ELLIPSIS Trans #00000 tid=... time=... offset=52 status=' ' user='' description='initial database creation' data #00000 oid=0000000000000000 size=66 class=persistent.mapping.PersistentMapping Now we see first transaction with root object. Let's add a BTree: >>> root = db.open().root() >>> root['tree'] = OOBTree() >>> txn.get().note('added an OOBTree') >>> txn.get().commit() >>> fsdump(path) #doctest: +ELLIPSIS Trans #00000 tid=... time=... offset=52 status=' ' user='' description='initial database creation' data #00000 oid=0000000000000000 size=66 class=persistent.mapping.PersistentMapping Trans #00001 tid=... time=... offset=207 status=' ' user='' description='added an OOBTree' data #00000 oid=0000000000000000 size=114 class=persistent.mapping.PersistentMapping data #00001 oid=0000000000000001 size=30 class=BTrees._OOBTree.OOBTree Now we see two transactions and two changed objects. Clean up. >>> st.close() >>> st.cleanup() # remove .fs, .index, etc """ from zope.testing import doctest def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/ZODB/tests/test_storage.py0000664000076400007640000001203210435502423021600 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A storage used for unittests. The primary purpose of this module is to have a minimal multi-version storage to use for unit tests. MappingStorage isn't sufficient. Since even a minimal storage has some complexity, we run standard storage tests against the test storage. """ import bisect import threading import unittest from ZODB.BaseStorage import BaseStorage from ZODB import POSException from ZODB.utils import z64 from ZODB.tests import StorageTestBase from ZODB.tests \ import BasicStorage, MTStorage, Synchronization, PackableStorage, \ RevisionStorage class Transaction(object): """Hold data for current transaction for MinimalMemoryStorage.""" def __init__(self, tid): self.index = {} self.tid = tid def store(self, oid, data): self.index[(oid, self.tid)] = data def cur(self): return dict.fromkeys([oid for oid, tid in self.index.keys()], self.tid) class MinimalMemoryStorage(BaseStorage, object): """Simple in-memory storage that supports revisions. This storage is needed to test multi-version concurrency control. It is similar to MappingStorage, but keeps multiple revisions. It does not support versions. It doesn't implement operations like pack(), because they aren't necessary for testing. """ def __init__(self): super(MinimalMemoryStorage, self).__init__("name") # _index maps oid, tid pairs to data records self._index = {} # _cur maps oid to current tid self._cur = {} def isCurrent(self, oid, serial): return serial == self._cur[oid] def hook(self, oid, tid, version): # A hook for testing pass def __len__(self): return len(self._index) def _clear_temp(self): pass def loadEx(self, oid, version): self._lock_acquire() try: assert not version tid = self._cur[oid] self.hook(oid, tid, version) return self._index[(oid, tid)], tid, "" finally: self._lock_release() def load(self, oid, version): return self.loadEx(oid, version)[:2] def _begin(self, tid, u, d, e): self._txn = Transaction(tid) def store(self, oid, serial, data, v, txn): if txn is not self._transaction: raise POSException.StorageTransactionError(self, txn) assert not v if self._cur.get(oid) != serial: if not (serial is None or self._cur.get(oid) in [None, z64]): raise POSException.ConflictError( oid=oid, serials=(self._cur.get(oid), serial), data=data) self._txn.store(oid, data) return self._tid def _abort(self): del self._txn def _finish(self, tid, u, d, e): self._lock_acquire() try: self._index.update(self._txn.index) self._cur.update(self._txn.cur()) self._ltid = self._tid finally: self._lock_release() def lastTransaction(self): return self._ltid def loadBefore(self, the_oid, the_tid): # It's okay if loadBefore() is really expensive, because this # storage is just used for testing. self._lock_acquire() try: tids = [tid for oid, tid in self._index if oid == the_oid] if not tids: raise KeyError, the_oid tids.sort() i = bisect.bisect_left(tids, the_tid) - 1 if i == -1: return None tid = tids[i] j = i + 1 if j == len(tids): end_tid = None else: end_tid = tids[j] return self._index[(the_oid, tid)], tid, end_tid finally: self._lock_release() def loadSerial(self, oid, serial): self._lock_acquire() try: return self._index[(oid, serial)] finally: self._lock_release() class MinimalTestSuite(StorageTestBase.StorageTestBase, BasicStorage.BasicStorage, MTStorage.MTStorage, Synchronization.SynchronizedStorage, RevisionStorage.RevisionStorage, ): def setUp(self): self._storage = MinimalMemoryStorage() # we don't implement undo def checkLoadBeforeUndo(self): pass def test_suite(): return unittest.makeSuite(MinimalTestSuite, "check") Zope-2.8.7-final/lib/python/ZODB/tests/testConfig.py0000664000076400007640000000646110435502423021213 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import tempfile import unittest import transaction import ZODB.config from ZODB.POSException import ReadOnlyError class ConfigTestBase(unittest.TestCase): def _opendb(self, s): return ZODB.config.databaseFromString(s) def tearDown(self): if getattr(self, "storage", None) is not None: self.storage.cleanup() def _test(self, s): db = self._opendb(s) self.storage = db._storage # Do something with the database to make sure it works cn = db.open() rt = cn.root() rt["test"] = 1 transaction.commit() db.close() class ZODBConfigTest(ConfigTestBase): def test_map_config1(self): self._test( """ """) def test_map_config2(self): self._test( """ cache-size 1000 """) def test_file_config1(self): path = tempfile.mktemp() self._test( """ path %s """ % path) def test_file_config2(self): path = tempfile.mktemp() cfg = """ path %s create false read-only true """ % path self.assertRaises(ReadOnlyError, self._test, cfg) def test_demo_config(self): cfg = """ name foo """ self._test(cfg) class ZEOConfigTest(ConfigTestBase): def test_zeo_config(self): # We're looking for a port that doesn't exist so a # connection attempt will fail. Instead of elaborate # logic to loop over a port calculation, we'll just pick a # simple "random", likely to not-exist port number and add # an elaborate comment explaining this instead. Go ahead, # grep for 9. from ZEO.ClientStorage import ClientDisconnected cfg = """ server localhost:56897 wait false """ self.assertRaises(ClientDisconnected, self._test, cfg) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(ZODBConfigTest)) suite.addTest(unittest.makeSuite(ZEOConfigTest)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZODB/tests/speed.py0000664000076400007640000000667410435502423020214 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## usage="""Test speed of a ZODB storage Options: -d file The data file to use as input. The default is this script. -n n The number of repititions -s module A module that defines a 'Storage' attribute, which is an open storage. If not specified, a FileStorage will ne used. -z Test compressing data -D Run in debug mode -L Test loads as well as stores by minimizing the cache after eachrun -M Output means only """ import sys, os, getopt, string, time sys.path.insert(0, os.getcwd()) import ZODB, ZODB.FileStorage import persistent import transaction class P(persistent.Persistent): pass def main(args): opts, args = getopt.getopt(args, 'zd:n:Ds:LM') z=s=None data=sys.argv[0] nrep=5 minimize=0 detailed=1 for o, v in opts: if o=='-n': nrep=string.atoi(v) elif o=='-d': data=v elif o=='-s': s=v elif o=='-z': global zlib import zlib z=compress elif o=='-L': minimize=1 elif o=='-M': detailed=0 elif o=='-D': global debug os.environ['STUPID_LOG_FILE']='' os.environ['STUPID_LOG_SEVERITY']='-999' if s: s=__import__(s, globals(), globals(), ('__doc__',)) s=s.Storage else: s=ZODB.FileStorage.FileStorage('zeo_speed.fs', create=1) data=open(data).read() db=ZODB.DB(s, # disable cache deactivation cache_size=4000, cache_deactivate_after=6000,) results={1:0, 10:0, 100:0, 1000:0} for j in range(nrep): for r in 1, 10, 100, 1000: t=time.time() jar=db.open() transaction.begin() rt=jar.root() key='s%s' % r if rt.has_key(key): p=rt[key] else: rt[key]=p=P() for i in range(r): if z is not None: d=z(data) else: d=data v=getattr(p, str(i), P()) v.d=d setattr(p,str(i),v) transaction.commit() jar.close() t=time.time()-t if detailed: sys.stderr.write("%s\t%s\t%.4f\n" % (j, r, t)) sys.stdout.flush() results[r]=results[r]+t rt=d=p=v=None # release all references if minimize: time.sleep(3) jar.cacheMinimize(3) if detailed: print '-'*24 for r in 1, 10, 100, 1000: t=results[r]/nrep sys.stderr.write("mean:\t%s\t%.4f\t%.4f (s/o)\n" % (r, t, t/r)) db.close() def compress(s): c=zlib.compressobj() o=c.compress(s) return o+c.flush() if __name__=='__main__': main(sys.argv[1:]) Zope-2.8.7-final/lib/python/ZODB/tests/MinPO.py0000664000076400007640000000170010435502423020057 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """A minimal persistent object to use for tests""" from persistent import Persistent class MinPO(Persistent): def __init__(self, value=None): self.value = value def __cmp__(self, aMinPO): return cmp(self.value, aMinPO.value) def __repr__(self): return "MinPO(%s)" % self.value Zope-2.8.7-final/lib/python/ZODB/tests/Corruption.py0000664000076400007640000000457110435502423021252 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Do some minimal tests of data corruption""" import os import random import stat import tempfile import ZODB, ZODB.FileStorage from StorageTestBase import StorageTestBase class FileStorageCorruptTests(StorageTestBase): def setUp(self): self.path = tempfile.mktemp() self._storage = ZODB.FileStorage.FileStorage(self.path, create=1) def tearDown(self): self._storage.close() self._storage.cleanup() def _do_stores(self): oids = [] for i in range(5): oid = self._storage.new_oid() revid = self._dostore(oid) oids.append((oid, revid)) return oids def _check_stores(self, oids): for oid, revid in oids: data, s_revid = self._storage.load(oid, '') self.assertEqual(s_revid, revid) def checkTruncatedIndex(self): oids = self._do_stores() self._close() # truncation the index file path = self.path + '.index' self.failUnless(os.path.exists(path)) f = open(path, 'r+') f.seek(0, 2) size = f.tell() f.seek(size / 2) f.truncate() f.close() self._storage = ZODB.FileStorage.FileStorage(self.path) self._check_stores(oids) def checkCorruptedIndex(self): oids = self._do_stores() self._close() # truncation the index file path = self.path + '.index' self.failUnless(os.path.exists(path)) size = os.stat(path)[stat.ST_SIZE] f = open(path, 'r+') while f.tell() < size: f.seek(random.randrange(1, size / 10), 1) f.write('\000') f.close() self._storage = ZODB.FileStorage.FileStorage(self.path) self._check_stores(oids) Zope-2.8.7-final/lib/python/ZODB/tests/testPersistentList.py0000664000076400007640000001351010435502423022773 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test the list interface to PersistentList """ import unittest from persistent.list import PersistentList l0 = [] l1 = [0] l2 = [0, 1] class TestPList(unittest.TestCase): def checkTheWorld(self): # Test constructors u = PersistentList() u0 = PersistentList(l0) u1 = PersistentList(l1) u2 = PersistentList(l2) uu = PersistentList(u) uu0 = PersistentList(u0) uu1 = PersistentList(u1) uu2 = PersistentList(u2) v = PersistentList(tuple(u)) class OtherList: def __init__(self, initlist): self.__data = initlist def __len__(self): return len(self.__data) def __getitem__(self, i): return self.__data[i] v0 = PersistentList(OtherList(u0)) vv = PersistentList("this is also a sequence") # Test __repr__ eq = self.assertEqual eq(str(u0), str(l0), "str(u0) == str(l0)") eq(repr(u1), repr(l1), "repr(u1) == repr(l1)") eq(`u2`, `l2`, "`u2` == `l2`") # Test __cmp__ and __len__ def mycmp(a, b): r = cmp(a, b) if r < 0: return -1 if r > 0: return 1 return r all = [l0, l1, l2, u, u0, u1, u2, uu, uu0, uu1, uu2] for a in all: for b in all: eq(mycmp(a, b), mycmp(len(a), len(b)), "mycmp(a, b) == mycmp(len(a), len(b))") # Test __getitem__ for i in range(len(u2)): eq(u2[i], i, "u2[i] == i") # Test __setitem__ uu2[0] = 0 uu2[1] = 100 try: uu2[2] = 200 except IndexError: pass else: raise TestFailed("uu2[2] shouldn't be assignable") # Test __delitem__ del uu2[1] del uu2[0] try: del uu2[0] except IndexError: pass else: raise TestFailed("uu2[0] shouldn't be deletable") # Test __getslice__ for i in range(-3, 4): eq(u2[:i], l2[:i], "u2[:i] == l2[:i]") eq(u2[i:], l2[i:], "u2[i:] == l2[i:]") for j in range(-3, 4): eq(u2[i:j], l2[i:j], "u2[i:j] == l2[i:j]") # Test __setslice__ for i in range(-3, 4): u2[:i] = l2[:i] eq(u2, l2, "u2 == l2") u2[i:] = l2[i:] eq(u2, l2, "u2 == l2") for j in range(-3, 4): u2[i:j] = l2[i:j] eq(u2, l2, "u2 == l2") uu2 = u2[:] uu2[:0] = [-2, -1] eq(uu2, [-2, -1, 0, 1], "uu2 == [-2, -1, 0, 1]") uu2[0:] = [] eq(uu2, [], "uu2 == []") # Test __contains__ for i in u2: self.failUnless(i in u2, "i in u2") for i in min(u2)-1, max(u2)+1: self.failUnless(i not in u2, "i not in u2") # Test __delslice__ uu2 = u2[:] del uu2[1:2] del uu2[0:1] eq(uu2, [], "uu2 == []") uu2 = u2[:] del uu2[1:] del uu2[:1] eq(uu2, [], "uu2 == []") # Test __add__, __radd__, __mul__ and __rmul__ #self.failUnless(u1 + [] == [] + u1 == u1, "u1 + [] == [] + u1 == u1") self.failUnless(u1 + [1] == u2, "u1 + [1] == u2") #self.failUnless([-1] + u1 == [-1, 0], "[-1] + u1 == [-1, 0]") self.failUnless(u2 == u2*1 == 1*u2, "u2 == u2*1 == 1*u2") self.failUnless(u2+u2 == u2*2 == 2*u2, "u2+u2 == u2*2 == 2*u2") self.failUnless(u2+u2+u2 == u2*3 == 3*u2, "u2+u2+u2 == u2*3 == 3*u2") # Test append u = u1[:] u.append(1) eq(u, u2, "u == u2") # Test insert u = u2[:] u.insert(0, -1) eq(u, [-1, 0, 1], "u == [-1, 0, 1]") # Test pop u = PersistentList([0, -1, 1]) u.pop() eq(u, [0, -1], "u == [0, -1]") u.pop(0) eq(u, [-1], "u == [-1]") # Test remove u = u2[:] u.remove(1) eq(u, u1, "u == u1") # Test count u = u2*3 eq(u.count(0), 3, "u.count(0) == 3") eq(u.count(1), 3, "u.count(1) == 3") eq(u.count(2), 0, "u.count(2) == 0") # Test index eq(u2.index(0), 0, "u2.index(0) == 0") eq(u2.index(1), 1, "u2.index(1) == 1") try: u2.index(2) except ValueError: pass else: raise TestFailed("expected ValueError") # Test reverse u = u2[:] u.reverse() eq(u, [1, 0], "u == [1, 0]") u.reverse() eq(u, u2, "u == u2") # Test sort u = PersistentList([1, 0]) u.sort() eq(u, u2, "u == u2") # Test extend u = u1[:] u.extend(u2) eq(u, u1 + u2, "u == u1 + u2") def checkBackwardCompat(self): # Verify that the sanest of the ZODB 3.2 dotted paths still works. from ZODB.PersistentList import PersistentList as oldPath self.assert_(oldPath is PersistentList) def test_suite(): return unittest.makeSuite(TestPList, 'check') if __name__ == "__main__": loader = unittest.TestLoader() loader.testMethodPrefix = "check" unittest.main(testLoader=loader) Zope-2.8.7-final/lib/python/ZODB/tests/MTStorage.py0000664000076400007640000001566310435502423020757 0ustar ajungajungimport random import sys import threading import time from persistent.mapping import PersistentMapping import transaction import ZODB from ZODB.tests.StorageTestBase \ import zodb_pickle, zodb_unpickle, handle_serials from ZODB.tests.MinPO import MinPO from ZODB.POSException import ConflictError SHORT_DELAY = 0.01 def sort(l): "Sort a list in place and return it." l.sort() return l class TestThread(threading.Thread): """Base class for defining threads that run from unittest. If the thread exits with an uncaught exception, catch it and re-raise it when the thread is joined. The re-raise will cause the test to fail. The subclass should define a runtest() method instead of a run() method. """ def __init__(self): threading.Thread.__init__(self) self._exc_info = None def run(self): try: self.runtest() except: self._exc_info = sys.exc_info() def join(self, timeout=None): threading.Thread.join(self, timeout) if self._exc_info: raise self._exc_info[0], self._exc_info[1], self._exc_info[2] class ZODBClientThread(TestThread): __super_init = TestThread.__init__ def __init__(self, db, test, commits=10, delay=SHORT_DELAY): self.__super_init() self.setDaemon(1) self.db = db self.test = test self.commits = commits self.delay = delay def runtest(self): conn = self.db.open() conn.sync() root = conn.root() d = self.get_thread_dict(root) if d is None: self.test.fail() else: for i in range(self.commits): self.commit(d, i) self.test.assertEqual(sort(d.keys()), range(self.commits)) def commit(self, d, num): d[num] = time.time() time.sleep(self.delay) transaction.commit() time.sleep(self.delay) # Return a new PersistentMapping, and store it on the root object under # the name (.getName()) of the current thread. def get_thread_dict(self, root): # This is vicious: multiple threads are slamming changes into the # root object, then trying to read the root object, simultaneously # and without any coordination. Conflict errors are rampant. It # used to go around at most 10 times, but that fairly often failed # to make progress in the 7-thread tests on some test boxes. Going # around (at most) 1000 times was enough so that a 100-thread test # reliably passed on Tim's hyperthreaded WinXP box (but at the # original 10 retries, the same test reliably failed with 15 threads). name = self.getName() MAXRETRIES = 1000 for i in range(MAXRETRIES): try: root[name] = PersistentMapping() transaction.commit() break except ConflictError: root._p_jar.sync() for i in range(MAXRETRIES): try: return root.get(name) except ConflictError: root._p_jar.sync() class StorageClientThread(TestThread): __super_init = TestThread.__init__ def __init__(self, storage, test, commits=10, delay=SHORT_DELAY): self.__super_init() self.storage = storage self.test = test self.commits = commits self.delay = delay self.oids = {} def runtest(self): for i in range(self.commits): self.dostore(i) self.check() def check(self): for oid, revid in self.oids.items(): data, serial = self.storage.load(oid, '') self.test.assertEqual(serial, revid) obj = zodb_unpickle(data) self.test.assertEqual(obj.value[0], self.getName()) def pause(self): time.sleep(self.delay) def oid(self): oid = self.storage.new_oid() self.oids[oid] = None return oid def dostore(self, i): data = zodb_pickle(MinPO((self.getName(), i))) t = transaction.Transaction() oid = self.oid() self.pause() self.storage.tpc_begin(t) self.pause() # Always create a new object, signified by None for revid r1 = self.storage.store(oid, None, data, '', t) self.pause() r2 = self.storage.tpc_vote(t) self.pause() self.storage.tpc_finish(t) self.pause() revid = handle_serials(oid, r1, r2) self.oids[oid] = revid class ExtStorageClientThread(StorageClientThread): def runtest(self): # pick some other storage ops to execute, depending in part # on the features provided by the storage. names = ["do_load", "do_modifiedInVersion"] if self.storage.supportsUndo(): names += ["do_loadSerial", "do_undoLog", "do_iterator"] ops = [getattr(self, meth) for meth in names] assert ops, "Didn't find an storage ops in %s" % self.storage # do a store to guarantee there's at least one oid in self.oids self.dostore(0) for i in range(self.commits - 1): meth = random.choice(ops) meth() self.dostore(i) self.check() def pick_oid(self): return random.choice(self.oids.keys()) def do_load(self): oid = self.pick_oid() self.storage.load(oid, '') def do_loadSerial(self): oid = self.pick_oid() self.storage.loadSerial(oid, self.oids[oid]) def do_modifiedInVersion(self): oid = self.pick_oid() self.storage.modifiedInVersion(oid) def do_undoLog(self): self.storage.undoLog(0, -20) def do_iterator(self): try: iter = self.storage.iterator() except AttributeError: # It's hard to detect that a ZEO ClientStorage # doesn't have this method, but does have all the others. return for obj in iter: pass class MTStorage: "Test a storage with multiple client threads executing concurrently." def _checkNThreads(self, n, constructor, *args): threads = [constructor(*args) for i in range(n)] for t in threads: t.start() for t in threads: t.join(60) for t in threads: self.failIf(t.isAlive(), "thread failed to finish in 60 seconds") def check2ZODBThreads(self): db = ZODB.DB(self._storage) self._checkNThreads(2, ZODBClientThread, db, self) db.close() def check7ZODBThreads(self): db = ZODB.DB(self._storage) self._checkNThreads(7, ZODBClientThread, db, self) db.close() def check2StorageThreads(self): self._checkNThreads(2, StorageClientThread, self._storage, self) def check7StorageThreads(self): self._checkNThreads(7, StorageClientThread, self._storage, self) def check4ExtStorageThread(self): self._checkNThreads(4, ExtStorageClientThread, self._storage, self) Zope-2.8.7-final/lib/python/ZODB/tests/__init__.py0000664000076400007640000000004610435502423020636 0ustar ajungajung# Having this makes debugging better. Zope-2.8.7-final/lib/python/ZODB/tests/StorageTestBase.py0000664000076400007640000001743410435502423022147 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Provide a mixin base class for storage tests. The StorageTestBase class provides basic setUp() and tearDown() semantics (which you can override), and it also provides a helper method _dostore() which performs a complete store transaction for a single object revision. """ import sys import time import types import unittest from cPickle import Pickler, Unpickler from cStringIO import StringIO import transaction from ZODB.utils import u64 from ZODB.tests.MinPO import MinPO ZERO = '\0'*8 def snooze(): # In Windows, it's possible that two successive time.time() calls return # the same value. Tim guarantees that time never runs backwards. You # usually want to call this before you pack a storage, or must make other # guarantees about increasing timestamps. now = time.time() while now == time.time(): time.sleep(0.1) def _persistent_id(obj): oid = getattr(obj, "_p_oid", None) if getattr(oid, "__get__", None) is not None: return None else: return oid def zodb_pickle(obj): """Create a pickle in the format expected by ZODB.""" f = StringIO() p = Pickler(f, 1) p.persistent_id = _persistent_id klass = obj.__class__ assert not hasattr(obj, '__getinitargs__'), "not ready for constructors" args = None mod = getattr(klass, '__module__', None) if mod is not None: klass = mod, klass.__name__ state = obj.__getstate__() p.dump((klass, args)) p.dump(state) return f.getvalue(1) def persistent_load(pid): # helper for zodb_unpickle return "ref to %s.%s oid=%s" % (pid[1][0], pid[1][1], u64(pid[0])) def zodb_unpickle(data): """Unpickle an object stored using the format expected by ZODB.""" f = StringIO(data) u = Unpickler(f) u.persistent_load = persistent_load klass_info = u.load() if isinstance(klass_info, types.TupleType): if isinstance(klass_info[0], type): # Unclear: what is the second part of klass_info? klass, xxx = klass_info assert not xxx else: if isinstance(klass_info[0], tuple): modname, klassname = klass_info[0] else: modname, klassname = klass_info if modname == "__main__": ns = globals() else: mod = import_helper(modname) ns = mod.__dict__ try: klass = ns[klassname] except KeyError: print >> sys.stderr, "can't find %s in %r" % (klassname, ns) inst = klass() else: raise ValueError, "expected class info: %s" % repr(klass_info) state = u.load() inst.__setstate__(state) return inst def handle_all_serials(oid, *args): """Return dict of oid to serialno from store() and tpc_vote(). Raises an exception if one of the calls raised an exception. The storage interface got complicated when ZEO was introduced. Any individual store() call can return None or a sequence of 2-tuples where the 2-tuple is either oid, serialno or an exception to be raised by the client. The original interface just returned the serialno for the object. """ d = {} for arg in args: if isinstance(arg, types.StringType): d[oid] = arg elif arg is None: pass else: for oid, serial in arg: if not isinstance(serial, types.StringType): raise serial # error from ZEO server d[oid] = serial return d def handle_serials(oid, *args): """Return the serialno for oid based on multiple return values. A helper for function _handle_all_serials(). """ return handle_all_serials(oid, *args)[oid] def import_helper(name): __import__(name) return sys.modules[name] class StorageTestBase(unittest.TestCase): # It would be simpler if concrete tests didn't need to extend # setUp() and tearDown(). def setUp(self): # You need to override this with a setUp that creates self._storage self._storage = None def _close(self): # You should override this if closing your storage requires additional # shutdown operations. if self._storage is not None: self._storage.close() def tearDown(self): self._close() def _dostore(self, oid=None, revid=None, data=None, version=None, already_pickled=0, user=None, description=None): """Do a complete storage transaction. The defaults are: - oid=None, ask the storage for a new oid - revid=None, use a revid of ZERO - data=None, pickle up some arbitrary data (the integer 7) - version=None, use the empty string version Returns the object's new revision id. """ if oid is None: oid = self._storage.new_oid() if revid is None: revid = ZERO if data is None: data = MinPO(7) if type(data) == types.IntType: data = MinPO(data) if not already_pickled: data = zodb_pickle(data) if version is None: version = '' # Begin the transaction t = transaction.Transaction() if user is not None: t.user = user if description is not None: t.description = description try: self._storage.tpc_begin(t) # Store an object r1 = self._storage.store(oid, revid, data, version, t) # Finish the transaction r2 = self._storage.tpc_vote(t) revid = handle_serials(oid, r1, r2) self._storage.tpc_finish(t) except: self._storage.tpc_abort(t) raise return revid def _dostoreNP(self, oid=None, revid=None, data=None, version=None, user=None, description=None): return self._dostore(oid, revid, data, version, 1, user, description) # The following methods depend on optional storage features. def _undo(self, tid, expected_oids=None, note=None): # Undo a tid that affects a single object (oid). # This is very specialized. t = transaction.Transaction() t.note(note or "undo") self._storage.tpc_begin(t) tid, oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) if expected_oids is not None: self.assertEqual(len(oids), len(expected_oids), repr(oids)) for oid in expected_oids: self.assert_(oid in oids) return self._storage.lastTransaction() def _commitVersion(self, src, dst): t = transaction.Transaction() t.note("commit %r to %r" % (src, dst)) self._storage.tpc_begin(t) tid, oids = self._storage.commitVersion(src, dst, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) return oids def _abortVersion(self, ver): t = transaction.Transaction() t.note("abort %r" % ver) self._storage.tpc_begin(t) tid, oids = self._storage.abortVersion(ver, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) return oids Zope-2.8.7-final/lib/python/ZODB/tests/testfsIndex.py0000664000076400007640000001204310435502423021377 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import random from ZODB.fsIndex import fsIndex from ZODB.utils import p64, z64 class Test(unittest.TestCase): def setUp(self): self.index = fsIndex() for i in range(200): self.index[p64(i * 1000)] = (i * 1000L + 1) def testInserts(self): index = self.index for i in range(0,200): self.assertEqual((i,index[p64(i*1000)]), (i,(i*1000L+1))) self.assertEqual(len(index), 200) key=p64(2000) self.assertEqual(index.get(key), 2001) key=p64(2001) self.assertEqual(index.get(key), None) self.assertEqual(index.get(key, ''), '') # self.failUnless(len(index._data) > 1) def testUpdate(self): index = self.index d={} for i in range(200): d[p64(i*1000)]=(i*1000L+1) index.update(d) for i in range(400,600): d[p64(i*1000)]=(i*1000L+1) index.update(d) for i in range(100, 500): d[p64(i*1000)]=(i*1000L+2) index.update(d) self.assertEqual(index.get(p64(2000)), 2001) self.assertEqual(index.get(p64(599000)), 599001) self.assertEqual(index.get(p64(399000)), 399002) self.assertEqual(len(index), 600) def testKeys(self): keys = list(iter(self.index)) keys.sort() for i, k in enumerate(keys): self.assertEqual(k, p64(i * 1000)) keys = list(self.index.iterkeys()) keys.sort() for i, k in enumerate(keys): self.assertEqual(k, p64(i * 1000)) keys = self.index.keys() keys.sort() for i, k in enumerate(keys): self.assertEqual(k, p64(i * 1000)) def testValues(self): values = list(self.index.itervalues()) values.sort() for i, v in enumerate(values): self.assertEqual(v, (i * 1000L + 1)) values = self.index.values() values.sort() for i, v in enumerate(values): self.assertEqual(v, (i * 1000L + 1)) def testItems(self): items = list(self.index.iteritems()) items.sort() for i, item in enumerate(items): self.assertEqual(item, (p64(i * 1000), (i * 1000L + 1))) items = self.index.items() items.sort() for i, item in enumerate(items): self.assertEqual(item, (p64(i * 1000), (i * 1000L + 1))) def testMaxKey(self): index = self.index index.clear() # An empty index should complain. self.assertRaises(ValueError, index.maxKey) # Now build up a tree with random values, and check maxKey at each # step. correct_max = "" # smaller than anything we'll add for i in range(1000): key = p64(random.randrange(100000000)) index[key] = i correct_max = max(correct_max, key) index_max = index.maxKey() self.assertEqual(index_max, correct_max) index.clear() a = '\000\000\000\000\000\001\000\000' b = '\000\000\000\000\000\002\000\000' c = '\000\000\000\000\000\003\000\000' d = '\000\000\000\000\000\004\000\000' index[a] = 1 index[c] = 2 self.assertEqual(index.maxKey(b), a) self.assertEqual(index.maxKey(d), c) self.assertRaises(ValueError, index.maxKey, z64) def testMinKey(self): index = self.index index.clear() # An empty index should complain. self.assertRaises(ValueError, index.minKey) # Now build up a tree with random values, and check minKey at each # step. correct_min = "\xff" * 8 # bigger than anything we'll add for i in range(1000): key = p64(random.randrange(100000000)) index[key] = i correct_min = min(correct_min, key) index_min = index.minKey() self.assertEqual(index_min, correct_min) index.clear() a = '\000\000\000\000\000\001\000\000' b = '\000\000\000\000\000\002\000\000' c = '\000\000\000\000\000\003\000\000' d = '\000\000\000\000\000\004\000\000' index[a] = 1 index[c] = 2 self.assertEqual(index.minKey(b), c) self.assertRaises(ValueError, index.minKey, d) def test_suite(): loader=unittest.TestLoader() return loader.loadTestsFromTestCase(Test) if __name__=='__main__': unittest.TextTestRunner().run(test_suite()) Zope-2.8.7-final/lib/python/ZODB/tests/ReadOnlyStorage.py0000664000076400007640000000475310435502423022152 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from ZODB.POSException import ReadOnlyError, Unsupported import transaction class ReadOnlyStorage: def _create_data(self): # test a read-only storage that already has some data self.oids = {} for i in range(10): oid = self._storage.new_oid() revid = self._dostore(oid) self.oids[oid] = revid def _make_readonly(self): self._storage.close() self.open(read_only=True) self.assert_(self._storage.isReadOnly()) def checkReadMethods(self): self._create_data() self._make_readonly() # Note that this doesn't check _all_ read methods. for oid in self.oids.keys(): data, revid = self._storage.load(oid, '') self.assertEqual(revid, self.oids[oid]) self.assert_(not self._storage.modifiedInVersion(oid)) # Storages without revisions may not have loadSerial(). try: _data = self._storage.loadSerial(oid, revid) self.assertEqual(data, _data) except Unsupported: pass def checkWriteMethods(self): self._make_readonly() self.assertRaises(ReadOnlyError, self._storage.new_oid) t = transaction.Transaction() self.assertRaises(ReadOnlyError, self._storage.tpc_begin, t) if self._storage.supportsVersions(): self.assertRaises(ReadOnlyError, self._storage.abortVersion, '', t) self.assertRaises(ReadOnlyError, self._storage.commitVersion, '', '', t) self.assertRaises(ReadOnlyError, self._storage.store, '\000' * 8, None, '', '', t) if self._storage.supportsTransactionalUndo(): self.assertRaises(ReadOnlyError, self._storage.undo, '\000' * 8, t) Zope-2.8.7-final/lib/python/ZODB/tests/testpersistentclass.py0000664000076400007640000000303510435502423023226 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZClass tests $Id: testpersistentclass.py 29891 2005-04-06 22:35:05Z tim_one $ """ import os, sys import unittest import ZODB.tests.util import transaction from zope.testing import doctest # XXX need to update files to get newer testing package class FakeModule: def __init__(self, name, dict): self.__dict__ = dict self.__name__ = name def setUp(test): test.globs['some_database'] = ZODB.tests.util.DB() module = FakeModule('ZODB.persistentclass_txt', test.globs) sys.modules[module.__name__] = module def tearDown(test): transaction.abort() test.globs['some_database'].close() del sys.modules['ZODB.persistentclass_txt'] def test_suite(): return unittest.TestSuite(( doctest.DocFileSuite("../persistentclass.txt", setUp=setUp, tearDown=tearDown), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZODB/tests/TransactionalUndoStorage.py0000664000076400007640000006600510435502423024063 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Check undo(). Any storage that supports undo() must pass these tests. """ import time import types from persistent import Persistent import transaction from transaction import Transaction from ZODB import POSException from ZODB.serialize import referencesf from ZODB.utils import p64 from ZODB import DB from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_pickle, zodb_unpickle ZERO = '\0'*8 class C(Persistent): pass def snooze(): # In Windows, it's possible that two successive time.time() calls return # the same value. Tim guarantees that time never runs backwards. You # usually want to call this before you pack a storage, or must make other # guarantees about increasing timestamps. now = time.time() while now == time.time(): time.sleep(0.1) def listeq(L1, L2): """Return True if L1.sort() == L2.sort()""" c1 = L1[:] c2 = L2[:] c1.sort() c2.sort() return c1 == c2 class TransactionalUndoStorage: def _transaction_begin(self): self.__serials = {} def _transaction_store(self, oid, rev, data, vers, trans): r = self._storage.store(oid, rev, data, vers, trans) if r: if type(r) == types.StringType: self.__serials[oid] = r else: for oid, serial in r: self.__serials[oid] = serial def _transaction_vote(self, trans): r = self._storage.tpc_vote(trans) if r: for oid, serial in r: self.__serials[oid] = serial def _transaction_newserial(self, oid): return self.__serials[oid] def _multi_obj_transaction(self, objs): newrevs = {} t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() for oid, rev, data in objs: self._transaction_store(oid, rev, data, '', t) newrevs[oid] = None self._transaction_vote(t) self._storage.tpc_finish(t) for oid in newrevs.keys(): newrevs[oid] = self._transaction_newserial(oid) return newrevs def _iterate(self): """Iterate over the storage in its final state.""" # This is testing that the iterator() code works correctly. # The hasattr() guards against ZEO, which doesn't support iterator. if not hasattr(self._storage, "iterator"): return iter = self._storage.iterator() for txn in iter: for rec in txn: pass def undo(self, tid, note): t = Transaction() t.note(note) self._storage.tpc_begin(t) oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) return oids def checkSimpleTransactionalUndo(self): eq = self.assertEqual oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(23)) revid = self._dostore(oid, revid=revid, data=MinPO(24)) revid = self._dostore(oid, revid=revid, data=MinPO(25)) info = self._storage.undoInfo() # Now start an undo transaction self._undo(info[0]["id"], [oid], note="undo1") data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(24)) # Do another one info = self._storage.undoInfo() self._undo(info[2]["id"], [oid], note="undo2") data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(23)) # Try to undo the first record info = self._storage.undoInfo() self._undo(info[4]["id"], [oid], note="undo3") # This should fail since we've undone the object's creation self.assertRaises(KeyError, self._storage.load, oid, '') # And now let's try to redo the object's creation info = self._storage.undoInfo() self._undo(info[0]["id"], [oid]) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(23)) self._iterate() def checkCreationUndoneGetSerial(self): # create an object oid = self._storage.new_oid() self._dostore(oid, data=MinPO(23)) # undo its creation info = self._storage.undoInfo() tid = info[0]['id'] t = Transaction() t.note('undo1') self._storage.tpc_begin(t) self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) # Check that calling getSerial on an uncreated object raises a KeyError # The current version of FileStorage fails this test self.assertRaises(KeyError, self._storage.getSerial, oid) def checkUndoCreationBranch1(self): eq = self.assertEqual oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(11)) revid = self._dostore(oid, revid=revid, data=MinPO(12)) # Undo the last transaction info = self._storage.undoInfo() self._undo(info[0]['id'], [oid]) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(11)) # Now from here, we can either redo the last undo, or undo the object # creation. Let's undo the object creation. info = self._storage.undoInfo() self._undo(info[2]['id'], [oid]) self.assertRaises(KeyError, self._storage.load, oid, '') self._iterate() def checkUndoCreationBranch2(self): eq = self.assertEqual oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(11)) revid = self._dostore(oid, revid=revid, data=MinPO(12)) # Undo the last transaction info = self._storage.undoInfo() self._undo(info[0]['id'], [oid]) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(11)) # Now from here, we can either redo the last undo, or undo the object # creation. Let's redo the last undo info = self._storage.undoInfo() self._undo(info[0]['id'], [oid]) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(12)) self._iterate() def checkTwoObjectUndo(self): eq = self.assertEqual # Convenience p31, p32, p51, p52 = map(zodb_pickle, map(MinPO, (31, 32, 51, 52))) oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() revid1 = revid2 = ZERO # Store two objects in the same transaction t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() self._transaction_store(oid1, revid1, p31, '', t) self._transaction_store(oid2, revid2, p51, '', t) # Finish the transaction self._transaction_vote(t) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) self._storage.tpc_finish(t) eq(revid1, revid2) # Update those same two objects t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() self._transaction_store(oid1, revid1, p32, '', t) self._transaction_store(oid2, revid2, p52, '', t) # Finish the transaction self._transaction_vote(t) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) self._storage.tpc_finish(t) eq(revid1, revid2) # Make sure the objects have the current value data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(32)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(52)) # Now attempt to undo the transaction containing two objects info = self._storage.undoInfo() self._undo(info[0]['id'], [oid1, oid2]) data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(31)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(51)) self._iterate() def checkTwoObjectUndoAtOnce(self): # Convenience eq = self.assertEqual unless = self.failUnless p30, p31, p32, p50, p51, p52 = map(zodb_pickle, map(MinPO, (30, 31, 32, 50, 51, 52))) oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() revid1 = revid2 = ZERO # Store two objects in the same transaction d = self._multi_obj_transaction([(oid1, revid1, p30), (oid2, revid2, p50), ]) eq(d[oid1], d[oid2]) # Update those same two objects d = self._multi_obj_transaction([(oid1, d[oid1], p31), (oid2, d[oid2], p51), ]) eq(d[oid1], d[oid2]) # Update those same two objects d = self._multi_obj_transaction([(oid1, d[oid1], p32), (oid2, d[oid2], p52), ]) eq(d[oid1], d[oid2]) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) eq(revid1, revid2) # Make sure the objects have the current value data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(32)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(52)) # Now attempt to undo the transaction containing two objects info = self._storage.undoInfo() tid = info[0]['id'] tid1 = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) tid, oids = self._storage.undo(tid, t) tid, oids1 = self._storage.undo(tid1, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) # We get the finalization stuff called an extra time: eq(len(oids), 2) eq(len(oids1), 2) unless(oid1 in oids) unless(oid2 in oids) data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(30)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(50)) # Now try to undo the one we just did to undo, whew info = self._storage.undoInfo() self._undo(info[0]['id'], [oid1, oid2]) data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(32)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(52)) self._iterate() def checkTwoObjectUndoAgain(self): eq = self.assertEqual p31, p32, p33, p51, p52, p53 = map( zodb_pickle, map(MinPO, (31, 32, 33, 51, 52, 53))) # Like the above, but the first revision of the objects are stored in # different transactions. oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() revid1 = self._dostore(oid1, data=p31, already_pickled=1) revid2 = self._dostore(oid2, data=p51, already_pickled=1) # Update those same two objects t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() self._transaction_store(oid1, revid1, p32, '', t) self._transaction_store(oid2, revid2, p52, '', t) # Finish the transaction self._transaction_vote(t) self._storage.tpc_finish(t) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) eq(revid1, revid2) # Now attempt to undo the transaction containing two objects info = self._storage.undoInfo() self._undo(info[0]["id"], [oid1, oid2]) data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(31)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(51)) # Like the above, but this time, the second transaction contains only # one object. t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() self._transaction_store(oid1, revid1, p33, '', t) self._transaction_store(oid2, revid2, p53, '', t) # Finish the transaction self._transaction_vote(t) self._storage.tpc_finish(t) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) eq(revid1, revid2) # Update in different transactions revid1 = self._dostore(oid1, revid=revid1, data=MinPO(34)) revid2 = self._dostore(oid2, revid=revid2, data=MinPO(54)) # Now attempt to undo the transaction containing two objects info = self._storage.undoInfo() tid = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) tid, oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) eq(len(oids), 1) self.failUnless(oid1 in oids) self.failUnless(not oid2 in oids) data, revid1 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(33)) data, revid2 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(54)) self._iterate() def checkNotUndoable(self): eq = self.assertEqual # Set things up so we've got a transaction that can't be undone oid = self._storage.new_oid() revid_a = self._dostore(oid, data=MinPO(51)) revid_b = self._dostore(oid, revid=revid_a, data=MinPO(52)) revid_c = self._dostore(oid, revid=revid_b, data=MinPO(53)) # Start the undo info = self._storage.undoInfo() tid = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) self.assertRaises(POSException.UndoError, self._storage.undo, tid, t) self._storage.tpc_abort(t) # Now have more fun: object1 and object2 are in the same transaction, # which we'll try to undo to, but one of them has since modified in # different transaction, so the undo should fail. oid1 = oid revid1 = revid_c oid2 = self._storage.new_oid() revid2 = ZERO p81, p82, p91, p92 = map(zodb_pickle, map(MinPO, (81, 82, 91, 92))) t = Transaction() self._storage.tpc_begin(t) self._transaction_begin() self._transaction_store(oid1, revid1, p81, '', t) self._transaction_store(oid2, revid2, p91, '', t) self._transaction_vote(t) self._storage.tpc_finish(t) revid1 = self._transaction_newserial(oid1) revid2 = self._transaction_newserial(oid2) eq(revid1, revid2) # Make sure the objects have the expected values data, revid_11 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(81)) data, revid_22 = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(91)) eq(revid_11, revid1) eq(revid_22, revid2) # Now modify oid2 revid2 = self._dostore(oid2, revid=revid2, data=MinPO(92)) self.assertNotEqual(revid1, revid2) self.assertNotEqual(revid2, revid_22) info = self._storage.undoInfo() tid = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) self.assertRaises(POSException.UndoError, self._storage.undo, tid, t) self._storage.tpc_abort(t) self._iterate() def checkTransactionalUndoAfterPack(self): eq = self.assertEqual # Add a few object revisions oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(51)) packtime = time.time() snooze() # time.time() now distinct from packtime revid2 = self._dostore(oid, revid=revid1, data=MinPO(52)) self._dostore(oid, revid=revid2, data=MinPO(53)) # Now get the undo log info = self._storage.undoInfo() eq(len(info), 3) tid = info[0]['id'] # Now pack just the initial revision of the object. We need the # second revision otherwise we won't be able to undo the third # revision! self._storage.pack(packtime, referencesf) # Make some basic assertions about the undo information now info2 = self._storage.undoInfo() eq(len(info2), 2) # And now attempt to undo the last transaction t = Transaction() self._storage.tpc_begin(t) tid, oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) eq(len(oids), 1) eq(oids[0], oid) data, revid = self._storage.load(oid, '') # The object must now be at the second state eq(zodb_unpickle(data), MinPO(52)) self._iterate() def checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot(self): eq = self.assertEqual db = DB(self._storage) conn = db.open() root = conn.root() o1 = C() o2 = C() root['obj'] = o1 o1.obj = o2 txn = transaction.get() txn.note('o1 -> o2') txn.commit() now = packtime = time.time() while packtime <= now: packtime = time.time() o3 = C() o2.obj = o3 txn = transaction.get() txn.note('o1 -> o2 -> o3') txn.commit() o1.obj = o3 txn = transaction.get() txn.note('o1 -> o3') txn.commit() log = self._storage.undoLog() eq(len(log), 4) for entry in zip(log, ('o1 -> o3', 'o1 -> o2 -> o3', 'o1 -> o2', 'initial database creation')): eq(entry[0]['description'], entry[1]) self._storage.pack(packtime, referencesf) log = self._storage.undoLog() for entry in zip(log, ('o1 -> o3', 'o1 -> o2 -> o3')): eq(entry[0]['description'], entry[1]) tid = log[0]['id'] db.undo(tid) txn = transaction.get() txn.note('undo') txn.commit() # undo does a txn-undo, but doesn't invalidate conn.sync() log = self._storage.undoLog() for entry in zip(log, ('undo', 'o1 -> o3', 'o1 -> o2 -> o3')): eq(entry[0]['description'], entry[1]) eq(o1.obj, o2) eq(o1.obj.obj, o3) self._iterate() def checkPackAfterUndoDeletion(self): db = DB(self._storage) cn = db.open() root = cn.root() pack_times = [] def set_pack_time(): pack_times.append(time.time()) snooze() root["key0"] = MinPO(0) root["key1"] = MinPO(1) root["key2"] = MinPO(2) txn = transaction.get() txn.note("create 3 keys") txn.commit() set_pack_time() del root["key1"] txn = transaction.get() txn.note("delete 1 key") txn.commit() set_pack_time() root._p_deactivate() cn.sync() self.assert_(listeq(root.keys(), ["key0", "key2"])) L = db.undoInfo() db.undo(L[0]["id"]) txn = transaction.get() txn.note("undo deletion") txn.commit() set_pack_time() root._p_deactivate() cn.sync() self.assert_(listeq(root.keys(), ["key0", "key1", "key2"])) for t in pack_times: self._storage.pack(t, referencesf) root._p_deactivate() cn.sync() self.assert_(listeq(root.keys(), ["key0", "key1", "key2"])) for i in range(3): obj = root["key%d" % i] self.assertEqual(obj.value, i) root.items() self._inter_pack_pause() def checkPackAfterUndoManyTimes(self): db = DB(self._storage) cn = db.open() rt = cn.root() rt["test"] = MinPO(1) transaction.commit() rt["test2"] = MinPO(2) transaction.commit() rt["test"] = MinPO(3) txn = transaction.get() txn.note("root of undo") txn.commit() packtimes = [] for i in range(10): L = db.undoInfo() db.undo(L[0]["id"]) txn = transaction.get() txn.note("undo %d" % i) txn.commit() rt._p_deactivate() cn.sync() self.assertEqual(rt["test"].value, i % 2 and 3 or 1) self.assertEqual(rt["test2"].value, 2) packtimes.append(time.time()) snooze() for t in packtimes: self._storage.pack(t, referencesf) cn.sync() # TODO: Is _cache supposed to have a clear() method, or not? # cn._cache.clear() # The last undo set the value to 3 and pack should # never change that. self.assertEqual(rt["test"].value, 3) self.assertEqual(rt["test2"].value, 2) self._inter_pack_pause() def _inter_pack_pause(self): # DirectoryStorage needs a pause between packs, # most other storages dont. pass def checkTransactionalUndoIterator(self): # check that data_txn set in iterator makes sense if not hasattr(self._storage, "iterator"): return s = self._storage BATCHES = 4 OBJECTS = 4 orig = [] for i in range(BATCHES): t = Transaction() tid = p64(i + 1) s.tpc_begin(t, tid) for j in range(OBJECTS): oid = s.new_oid() obj = MinPO(i * OBJECTS + j) revid = s.store(oid, None, zodb_pickle(obj), '', t) orig.append((tid, oid, revid)) s.tpc_vote(t) s.tpc_finish(t) i = 0 for tid, oid, revid in orig: self._dostore(oid, revid=revid, data=MinPO(revid), description="update %s" % i) # Undo the OBJECTS transactions that modified objects created # in the ith original transaction. def undo(i): info = s.undoInfo() t = Transaction() s.tpc_begin(t) base = i * OBJECTS + i for j in range(OBJECTS): tid = info[base + j]['id'] s.undo(tid, t) s.tpc_vote(t) s.tpc_finish(t) for i in range(BATCHES): undo(i) # There are now (2 + OBJECTS) * BATCHES transactions: # BATCHES original transactions, followed by # OBJECTS * BATCHES modifications, followed by # BATCHES undos iter = s.iterator() offset = 0 eq = self.assertEqual for i in range(BATCHES): txn = iter[offset] offset += 1 tid = p64(i + 1) eq(txn.tid, tid) L1 = [(rec.oid, rec.tid, rec.data_txn) for rec in txn] L2 = [(oid, revid, None) for _tid, oid, revid in orig if _tid == tid] eq(L1, L2) for i in range(BATCHES * OBJECTS): txn = iter[offset] offset += 1 eq(len([rec for rec in txn if rec.data_txn is None]), 1) for i in range(BATCHES): txn = iter[offset] offset += 1 # The undos are performed in reverse order. otid = p64(BATCHES - i) L1 = [(rec.oid, rec.data_txn) for rec in txn] L2 = [(oid, otid) for _tid, oid, revid in orig if _tid == otid] L1.sort() L2.sort() eq(L1, L2) self.assertRaises(IndexError, iter.__getitem__, offset) def checkUndoLogMetadata(self): # test that the metadata is correct in the undo log t = transaction.get() t.note('t1') t.setExtendedInfo('k2','this is transaction metadata') t.setUser('u3',path='p3') db = DB(self._storage) conn = db.open() root = conn.root() o1 = C() root['obj'] = o1 txn = transaction.get() txn.commit() l = self._storage.undoLog() self.assertEqual(len(l),2) d = l[0] self.assertEqual(d['description'],'t1') self.assertEqual(d['k2'],'this is transaction metadata') self.assertEqual(d['user_name'],'p3 u3') # A common test body for index tests on undoInfo and undoLog. Before # ZODB 3.4, they always returned a wrong number of results (one too # few _or_ too many, depending on how they were called). def _exercise_info_indices(self, method_name): db = DB(self._storage) info_func = getattr(db, method_name) cn = db.open() rt = cn.root() # Do some transactions. for key in "abcdefghijklmnopqrstuvwxyz": rt[key] = ord(key) transaction.commit() # 26 letters = 26 transactions, + the hidden transaction to make # the root object, == 27 expected. allofem = info_func(0, 100000) self.assertEqual(len(allofem), 27) # Asking for no more than 100000 should do the same. redundant = info_func(last=-1000000) self.assertEqual(allofem, redundant) # By default, we should get only 20 back. default = info_func() self.assertEqual(len(default), 20) # And they should be the most recent 20. self.assertEqual(default, allofem[:20]) # If we ask for only one, we should get only the most recent. fresh = info_func(last=1) self.assertEqual(len(fresh), 1) self.assertEqual(fresh[0], allofem[0]) # Another way of asking for only the most recent. redundant = info_func(last=-1) self.assertEqual(fresh, redundant) # Try a slice that doesn't start at 0. oddball = info_func(first=11, last=17) self.assertEqual(len(oddball), 17-11) self.assertEqual(oddball, allofem[11 : 11+len(oddball)]) # And another way to spell the same thing. redundant = info_func(first=11, last=-6) self.assertEqual(oddball, redundant) cn.close() # Caution: don't close db; the framework does that. If you close # it here, the ZODB tests still work, but the ZRS RecoveryStorageTests # fail (closing the DB here in those tests closes the ZRS primary # before a ZRS secondary even starts, and then the latter can't # find a server to recover from). def checkIndicesInUndoInfo(self): self._exercise_info_indices("undoInfo") def checkIndicesInUndoLog(self): self._exercise_info_indices("undoLog") Zope-2.8.7-final/lib/python/ZODB/tests/testPersistentMapping.py0000664000076400007640000001410610435502423023455 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Verify that PersistentMapping works with old versions of Zope. The comments in PersistentMapping.py address the issue in some detail. The pickled form of a PersistentMapping must use _container to store the actual mapping, because old versions of Zope used this attribute. If the new code doesn't generate pickles that are consistent with the old code, developers will have a hard time testing the new code. """ import unittest import transaction from transaction import Transaction import ZODB from ZODB.MappingStorage import MappingStorage import cPickle import cStringIO import sys # This pickle contains a persistent mapping pickle created from the # old code. pickle = ('((U\x0bPersistenceq\x01U\x11PersistentMappingtq\x02Nt.}q\x03U\n' '_containerq\x04}q\x05U\x07versionq\x06U\x03oldq\x07ss.\n') class PMTests(unittest.TestCase): def checkOldStyleRoot(self): # The Persistence module doesn't exist in Zope3's idea of what ZODB # is, but the global `pickle` references it explicitly. So just # bail if Persistence isn't available. try: import Persistence except ImportError: return # insert the pickle in place of the root s = MappingStorage() t = Transaction() s.tpc_begin(t) s.store('\000' * 8, None, pickle, '', t) s.tpc_vote(t) s.tpc_finish(t) db = ZODB.DB(s) # If the root can be loaded successfully, we should be okay. r = db.open().root() # But make sure it looks like a new mapping self.assert_(hasattr(r, 'data')) self.assert_(not hasattr(r, '_container')) # TODO: This test fails in ZODB 3.3a1. It's making some assumption(s) # about pickles that aren't true. Hard to say when it stopped working, # because this entire test suite hasn't been run for a long time, due to # a mysterious "return None" at the start of the test_suite() function # below. I noticed that when the new checkBackwardCompat() test wasn't # getting run. def TODO_checkNewPicklesAreSafe(self): s = MappingStorage() db = ZODB.DB(s) r = db.open().root() r[1] = 1 r[2] = 2 r[3] = r transaction.commit() # MappingStorage stores serialno + pickle in its _index. root_pickle = s._index['\000' * 8][8:] f = cStringIO.StringIO(root_pickle) u = cPickle.Unpickler(f) klass_info = u.load() klass = find_global(*klass_info[0]) inst = klass.__new__(klass) state = u.load() inst.__setstate__(state) self.assert_(hasattr(inst, '_container')) self.assert_(not hasattr(inst, 'data')) def checkBackwardCompat(self): # Verify that the sanest of the ZODB 3.2 dotted paths still works. from persistent.mapping import PersistentMapping as newPath from ZODB.PersistentMapping import PersistentMapping as oldPath self.assert_(oldPath is newPath) def checkBasicOps(self): from persistent.mapping import PersistentMapping m = PersistentMapping({'x': 1}, a=2, b=3) m['name'] = 'bob' self.assertEqual(m['name'], "bob") self.assertEqual(m.get('name', 42), "bob") self.assert_('name' in m) try: m['fred'] except KeyError: pass else: self.fail("expected KeyError") self.assert_('fred' not in m) self.assertEqual(m.get('fred'), None) self.assertEqual(m.get('fred', 42), 42) keys = m.keys() keys.sort() self.assertEqual(keys, ['a', 'b', 'name', 'x']) values = m.values() values.sort() self.assertEqual(values, [1, 2, 3, 'bob']) items = m.items() items.sort() self.assertEqual(items, [('a', 2), ('b', 3), ('name', 'bob'), ('x', 1)]) keys = list(m.iterkeys()) keys.sort() self.assertEqual(keys, ['a', 'b', 'name', 'x']) values = list(m.itervalues()) values.sort() self.assertEqual(values, [1, 2, 3, 'bob']) items = list(m.iteritems()) items.sort() self.assertEqual(items, [('a', 2), ('b', 3), ('name', 'bob'), ('x', 1)]) # PersistentMapping didn't have an __iter__ method before ZODB 3.4.2. # Check that it plays well now with the Python iteration protocol. def checkIteration(self): from persistent.mapping import PersistentMapping m = PersistentMapping({'x': 1}, a=2, b=3) m['name'] = 'bob' def check(keylist): keylist.sort() self.assertEqual(keylist, ['a', 'b', 'name', 'x']) check(list(m)) check([key for key in m]) i = iter(m) keylist = [] while 1: try: key = i.next() except StopIteration: break keylist.append(key) check(keylist) def find_global(modulename, classname): """Helper for this test suite to get special PersistentMapping""" if classname == "PersistentMapping": class PersistentMapping(object): def __setstate__(self, state): self.__dict__.update(state) return PersistentMapping else: __import__(modulename) mod = sys.modules[modulename] return getattr(mod, classname) def test_suite(): return unittest.makeSuite(PMTests, 'check') if __name__ == "__main__": unittest.main() Zope-2.8.7-final/lib/python/ZODB/tests/testBroken.py0000664000076400007640000000457310435502423021230 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test broken-object suppport $Id: testBroken.py 29891 2005-04-06 22:35:05Z tim_one $ """ import sys import unittest import persistent import transaction from zope.testing.doctest import DocTestSuite from ZODB.tests.util import DB def test_integration(): """Test the integration of broken object support with the databse: >>> db = DB() We'll create a fake module with a class: >>> class NotThere: ... Atall = type('Atall', (persistent.Persistent, ), ... {'__module__': 'ZODB.not.there'}) And stuff this into sys.modules to simulate a regular module: >>> sys.modules['ZODB.not.there'] = NotThere >>> sys.modules['ZODB.not'] = NotThere Now, we'll create and save an instance, and make sure we can load it in another connection: >>> a = NotThere.Atall() >>> a.x = 1 >>> conn1 = db.open() >>> conn1.root()['a'] = a >>> transaction.commit() >>> conn2 = db.open() >>> a2 = conn2.root()['a'] >>> a2.__class__ is a.__class__ True >>> a2.x 1 Now, we'll uninstall the module, simulating having the module go away: >>> del sys.modules['ZODB.not.there'] and we'll try to load the object in another connection: >>> conn3 = db.open() >>> a3 = conn3.root()['a'] >>> a3 >>> a3.__Broken_state__ {'x': 1} Let's clean up: >>> db.close() >>> del sys.modules['ZODB.not'] Cleanup: >>> import ZODB.broken >>> ZODB.broken.broken_cache.clear() """ def test_suite(): return unittest.TestSuite(( DocTestSuite('ZODB.broken'), DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/ZODB/tests/dangle.py0000775000076400007640000000334310435502423020337 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Functional test to produce a dangling reference.""" import time import transaction from ZODB.FileStorage import FileStorage from ZODB import DB from persistent import Persistent class P(Persistent): pass def create_dangling_ref(db): rt = db.open().root() rt[1] = o1 = P() transaction.get().note("create o1") transaction.commit() rt[2] = o2 = P() transaction.get().note("create o2") transaction.commit() c = o1.child = P() transaction.get().note("set child on o1") transaction.commit() o1.child = P() transaction.get().note("replace child on o1") transaction.commit() time.sleep(2) # The pack should remove the reference to c, because it is no # longer referenced from o1. But the object still exists and has # an oid, so a new commit of it won't create a new object. db.pack() print repr(c._p_oid) o2.child = c transaction.get().note("set child on o2") transaction.commit() def main(): fs = FileStorage("dangle.fs") db = DB(fs) create_dangling_ref(db) db.close() if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZODB/tests/testMappingStorage.py0000664000076400007640000000332410435502423022721 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import ZODB.MappingStorage import unittest from ZODB.tests import StorageTestBase from ZODB.tests \ import BasicStorage, MTStorage, Synchronization, PackableStorage class MappingStorageTests(StorageTestBase.StorageTestBase, BasicStorage.BasicStorage, MTStorage.MTStorage, PackableStorage.PackableStorage, Synchronization.SynchronizedStorage, ): def setUp(self): self._storage = ZODB.MappingStorage.MappingStorage() def tearDown(self): self._storage.close() def checkOversizeNote(self): # This base class test checks for the common case where a storage # doesnt support huge transaction metadata. This storage doesnt # have this limit, so we inhibit this test here. pass def test_suite(): suite = unittest.makeSuite(MappingStorageTests, 'check') return suite if __name__ == "__main__": loader = unittest.TestLoader() loader.testMethodPrefix = "check" unittest.main(testLoader=loader) Zope-2.8.7-final/lib/python/ZODB/tests/testConnectionSavepoint.txt0000664000076400007640000001167210435502423024165 0ustar ajungajungSavepoints ========== Savepoints provide a way to save to disk intermediate work done during a transaction allowing: - partial transaction (subtransaction) rollback (abort) - state of saved objects to be freed, freeing on-line memory for other uses Savepoints make it possible to write atomic subroutines that don't make top-level transaction commitments. Applications ------------ To demonstrate how savepoints work with transactions, we'll show an example. >>> import ZODB.tests.util >>> db = ZODB.tests.util.DB() >>> connection = db.open() >>> root = connection.root() >>> root['name'] = 'bob' As with other data managers, we can commit changes: >>> import transaction >>> transaction.commit() >>> root['name'] 'bob' and abort changes: >>> root['name'] = 'sally' >>> root['name'] 'sally' >>> transaction.abort() >>> root['name'] 'bob' Now, let's look at an application that manages funds for people. It allows deposits and debits to be entered for multiple people. It accepts a sequence of entries and generates a sequence of status messages. For each entry, it applies the change and then validates the user's account. If the user's account is invalid, we roll back the change for that entry. The success or failure of an entry is indicated in the output status. First we'll initialize some accounts: >>> root['bob-balance'] = 0.0 >>> root['bob-credit'] = 0.0 >>> root['sally-balance'] = 0.0 >>> root['sally-credit'] = 100.0 >>> transaction.commit() Now, we'll define a validation function to validate an account: >>> def validate_account(name): ... if root[name+'-balance'] + root[name+'-credit'] < 0: ... raise ValueError('Overdrawn', name) And a function to apply entries. If the function fails in some unexpected way, it rolls back all of its changes and prints the error: >>> def apply_entries(entries): ... savepoint = transaction.savepoint() ... try: ... for name, amount in entries: ... entry_savepoint = transaction.savepoint() ... try: ... root[name+'-balance'] += amount ... validate_account(name) ... except ValueError, error: ... entry_savepoint.rollback() ... print 'Error', str(error) ... else: ... print 'Updated', name ... except Exception, error: ... savepoint.rollback() ... print 'Unexpected exception', error Now let's try applying some entries: >>> apply_entries([ ... ('bob', 10.0), ... ('sally', 10.0), ... ('bob', 20.0), ... ('sally', 10.0), ... ('bob', -100.0), ... ('sally', -100.0), ... ]) Updated bob Updated sally Updated bob Updated sally Error ('Overdrawn', 'bob') Updated sally >>> root['bob-balance'] 30.0 >>> root['sally-balance'] -80.0 If we provide entries that cause an unexpected error: >>> apply_entries([ ... ('bob', 10.0), ... ('sally', 10.0), ... ('bob', '20.0'), ... ('sally', 10.0), ... ]) Updated bob Updated sally Unexpected exception unsupported operand type(s) for +=: 'float' and 'str' Because the apply_entries used a savepoint for the entire function, it was able to rollback the partial changes without rolling back changes made in the previous call to apply_entries: >>> root['bob-balance'] 30.0 >>> root['sally-balance'] -80.0 If we now abort the outer transactions, the earlier changes will go away: >>> transaction.abort() >>> root['bob-balance'] 0.0 >>> root['sally-balance'] 0.0 Savepoint invalidation ---------------------- A savepoint can be used any number of times: >>> root['bob-balance'] = 100.0 >>> root['bob-balance'] 100.0 >>> savepoint = transaction.savepoint() >>> root['bob-balance'] = 200.0 >>> root['bob-balance'] 200.0 >>> savepoint.rollback() >>> root['bob-balance'] 100.0 >>> savepoint.rollback() # redundant, but should be harmless >>> root['bob-balance'] 100.0 >>> root['bob-balance'] = 300.0 >>> root['bob-balance'] 300.0 >>> savepoint.rollback() >>> root['bob-balance'] 100.0 However, using a savepoint invalidates any savepoints that come after it: >>> root['bob-balance'] = 200.0 >>> root['bob-balance'] 200.0 >>> savepoint1 = transaction.savepoint() >>> root['bob-balance'] = 300.0 >>> root['bob-balance'] 300.0 >>> savepoint2 = transaction.savepoint() >>> savepoint.rollback() >>> root['bob-balance'] 100.0 >>> savepoint2.rollback() Traceback (most recent call last): ... InvalidSavepointRollbackError >>> savepoint1.rollback() Traceback (most recent call last): ... InvalidSavepointRollbackError >>> transaction.abort() Zope-2.8.7-final/lib/python/ZODB/tests/RevisionStorage.py0000664000076400007640000001476110435502423022233 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Check loadSerial() on storages that support historical revisions.""" from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import \ zodb_unpickle, zodb_pickle, snooze, handle_serials from ZODB.utils import p64, u64 import transaction ZERO = '\0'*8 class RevisionStorage: def checkLoadSerial(self): oid = self._storage.new_oid() revid = ZERO revisions = {} for i in range(31, 38): revid = self._dostore(oid, revid=revid, data=MinPO(i)) revisions[revid] = MinPO(i) # Now make sure all the revisions have the correct value for revid, value in revisions.items(): data = self._storage.loadSerial(oid, revid) self.assertEqual(zodb_unpickle(data), value) def checkLoadBefore(self): # Store 10 revisions of one object and then make sure that we # can get all the non-current revisions back. oid = self._storage.new_oid() revs = [] revid = None for i in range(10): # We need to ensure that successive timestamps are at least # two apart, so that a timestamp exists that's unambiguously # between successive timestamps. Each call to snooze() # guarantees that the next timestamp will be at least one # larger (and probably much more than that) than the previous # one. snooze() snooze() revid = self._dostore(oid, revid, data=MinPO(i)) revs.append(self._storage.loadEx(oid, "")) prev = u64(revs[0][1]) for i in range(1, 10): tid = revs[i][1] cur = u64(tid) middle = prev + (cur - prev) // 2 assert prev < middle < cur # else the snooze() trick failed prev = cur t = self._storage.loadBefore(oid, p64(middle)) self.assert_(t is not None) data, start, end = t self.assertEqual(revs[i-1][0], data) self.assertEqual(tid, end) def checkLoadBeforeEdges(self): # Check the edges cases for a non-current load. oid = self._storage.new_oid() self.assertRaises(KeyError, self._storage.loadBefore, oid, p64(0)) revid1 = self._dostore(oid, data=MinPO(1)) self.assertEqual(self._storage.loadBefore(oid, p64(0)), None) self.assertEqual(self._storage.loadBefore(oid, revid1), None) cur = p64(u64(revid1) + 1) data, start, end = self._storage.loadBefore(oid, cur) self.assertEqual(zodb_unpickle(data), MinPO(1)) self.assertEqual(start, revid1) self.assertEqual(end, None) revid2 = self._dostore(oid, revid=revid1, data=MinPO(2)) data, start, end = self._storage.loadBefore(oid, cur) self.assertEqual(zodb_unpickle(data), MinPO(1)) self.assertEqual(start, revid1) self.assertEqual(end, revid2) def checkLoadBeforeOld(self): # Look for a very old revision. With the BaseStorage implementation # this should require multple history() calls. oid = self._storage.new_oid() revs = [] revid = None for i in range(50): revid = self._dostore(oid, revid, data=MinPO(i)) revs.append(revid) data, start, end = self._storage.loadBefore(oid, revs[12]) self.assertEqual(zodb_unpickle(data), MinPO(11)) self.assertEqual(start, revs[11]) self.assertEqual(end, revs[12]) # Unsure: Is it okay to assume everyone testing against RevisionStorage # implements undo? def checkLoadBeforeUndo(self): # Do several transactions then undo them. oid = self._storage.new_oid() revid = None for i in range(5): revid = self._dostore(oid, revid, data=MinPO(i)) revs = [] for i in range(4): info = self._storage.undoInfo() tid = info[0]["id"] # Always undo the most recent txn, so the value will # alternate between 3 and 4. self._undo(tid, [oid], note="undo %d" % i) revs.append(self._storage.loadEx(oid, "")) prev_tid = None for i, (data, tid, ver) in enumerate(revs): t = self._storage.loadBefore(oid, p64(u64(tid) + 1)) self.assertEqual(data, t[0]) self.assertEqual(tid, t[1]) if prev_tid: self.assert_(prev_tid < t[1]) prev_tid = t[1] if i < 3: self.assertEqual(revs[i+1][1], t[2]) else: self.assertEqual(None, t[2]) def checkLoadBeforeConsecutiveTids(self): eq = self.assertEqual oid = self._storage.new_oid() def helper(tid, revid, x): data = zodb_pickle(MinPO(x)) t = transaction.Transaction() try: self._storage.tpc_begin(t, p64(tid)) r1 = self._storage.store(oid, revid, data, '', t) # Finish the transaction r2 = self._storage.tpc_vote(t) newrevid = handle_serials(oid, r1, r2) self._storage.tpc_finish(t) except: self._storage.tpc_abort(t) raise return newrevid revid1 = helper(1, None, 1) revid2 = helper(2, revid1, 2) revid3 = helper(3, revid2, 3) data, start_tid, end_tid = self._storage.loadBefore(oid, p64(2)) eq(zodb_unpickle(data), MinPO(1)) eq(u64(start_tid), 1) eq(u64(end_tid), 2) def checkLoadBeforeCreation(self): eq = self.assertEqual oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() revid1 = self._dostore(oid1) revid2 = self._dostore(oid2) results = self._storage.loadBefore(oid2, revid2) eq(results, None) # TODO: There are other edge cases to handle, including pack. Zope-2.8.7-final/lib/python/ZODB/tests/Synchronization.py0000664000076400007640000001214110435502423022277 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test the storage's implemenetation of the storage synchronization spec. The Synchronization spec http://www.zope.org/Documentation/Developer/Models/ZODB/ ZODB_Architecture_Storage_Interface_State_Synchronization_Diag.html It specifies two states committing and non-committing. A storage starts in the non-committing state. tpc_begin() transfers to the committting state; tpc_abort() and tpc_finish() transfer back to non-committing. Several other methods are only allowed in one state or another. Many methods allowed only in the committing state require that they apply to the currently committing transaction. The spec is silent on a variety of methods that don't appear to modify the state, e.g. load(), undoLog(), pack(). It's unclear whether there is a separate set of synchronization rules that apply to these methods or if the synchronization is implementation dependent, i.e. only what is need to guarantee a corrected implementation. The synchronization spec is also silent on whether there is any contract implied with the caller. If the storage can assume that a single client is single-threaded and that it will not call, e.g., store() until after it calls tpc_begin(), the implementation can be substantially simplified. New and/or unspecified methods: tpc_vote(): handled like tpc_abort undo(): how's that handled? Methods that have nothing to do with committing/non-committing: load(), loadSerial(), getName(), getSize(), __len__(), history(), undoLog(), modifiedInVersion(), versionEmpty(), versions(), pack(). Specific questions: The spec & docs say that undo() takes three arguments, the second being a transaction. If the specified arg isn't the current transaction, the undo() should raise StorageTransactionError. This isn't implemented anywhere. It looks like undo can be called at anytime. FileStorage does not allow undo() during a pack. How should this be tested? Is it a general restriction? """ from transaction import Transaction from ZODB.POSException import StorageTransactionError VERSION = "testversion" OID = "\000" * 8 SERIALNO = "\000" * 8 TID = "\000" * 8 class SynchronizedStorage: ## def verifyCommitting(self, callable, *args): ## self.assertRaises(StorageTransactionError, callable *args) def verifyNotCommitting(self, callable, *args): self.assertRaises(StorageTransactionError, callable, *args) def verifyWrongTrans(self, callable, *args): t = Transaction() self._storage.tpc_begin(t) self.assertRaises(StorageTransactionError, callable, *args) self._storage.tpc_abort(t) def checkAbortVersionNotCommitting(self): self.verifyNotCommitting(self._storage.abortVersion, VERSION, Transaction()) def checkAbortVersionWrongTrans(self): self.verifyWrongTrans(self._storage.abortVersion, VERSION, Transaction()) def checkCommitVersionNotCommitting(self): self.verifyNotCommitting(self._storage.commitVersion, VERSION, "", Transaction()) def checkCommitVersionWrongTrans(self): self.verifyWrongTrans(self._storage.commitVersion, VERSION, "", Transaction()) def checkStoreNotCommitting(self): self.verifyNotCommitting(self._storage.store, OID, SERIALNO, "", "", Transaction()) def checkStoreWrongTrans(self): self.verifyWrongTrans(self._storage.store, OID, SERIALNO, "", "", Transaction()) ## def checkNewOidNotCommitting(self): ## self.verifyNotCommitting(self._storage.new_oid) ## def checkNewOidWrongTrans(self): ## self.verifyWrongTrans(self._storage.new_oid) def checkAbortNotCommitting(self): self._storage.tpc_abort(Transaction()) def checkAbortWrongTrans(self): t = Transaction() self._storage.tpc_begin(t) self._storage.tpc_abort(Transaction()) self._storage.tpc_abort(t) def checkFinishNotCommitting(self): t = Transaction() self._storage.tpc_finish(t) self._storage.tpc_abort(t) def checkFinishWrongTrans(self): t = Transaction() self._storage.tpc_begin(t) self._storage.tpc_finish(Transaction()) self._storage.tpc_abort(t) def checkBeginCommitting(self): t = Transaction() self._storage.tpc_begin(t) self._storage.tpc_begin(t) self._storage.tpc_abort(t) # TODO: how to check undo? Zope-2.8.7-final/lib/python/ZODB/tests/multidb.txt0000664000076400007640000001071710435502423020734 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## Multi-database tests ==================== Multi-database support adds the ability to tie multiple databases into a collection. The original proposal is in the fishbowl: http://www.zope.org/Wikis/ZODB/MultiDatabases/ It was implemented during the PyCon 2005 sprints, but in a simpler form, by Jim Fulton, Christian Theune, and Tim Peters. Overview: No private attributes were added, and one new method was introduced. DB: - a new .database_name attribute holds the name of this database - a new .databases attribute maps from database name to DB object; all DBs in a multi-database collection share the same .databases object - the DB constructor has new optional arguments with the same names (database_name= and databases=). Connection: - a new .connections attribute maps from database name to a Connection for the database with that name; the .connections mapping object is also shared among databases in a collection - a new .get_connection(database_name) method returns a Connection for a database in the collection; if a connection is already open, it's returned (this is the value .connections[database_name]), else a new connection is opened (and stored as .connections[database_name]) Creating a multi-database starts with creating a named DB: >>> from ZODB.tests.test_storage import MinimalMemoryStorage >>> from ZODB import DB >>> dbmap = {} >>> db = DB(MinimalMemoryStorage(), database_name='root', databases=dbmap) The database name is accessible afterwards and in a newly created collection: >>> db.database_name 'root' >>> db.databases # doctest: +ELLIPSIS {'root': } >>> db.databases is dbmap True Adding another database to the collection works like this: >>> db2 = DB(MinimalMemoryStorage(), ... database_name='notroot', ... databases=dbmap) The new db2 now shares the 'databases' dictionary with db and has two entries: >>> db2.databases is db.databases is dbmap True >>> len(db2.databases) 2 >>> names = dbmap.keys(); names.sort(); print names ['notroot', 'root'] It's an error to try to insert a database with a name already in use: >>> db3 = DB(MinimalMemoryStorage(), ... database_name='root', ... databases=dbmap) Traceback (most recent call last): ... ValueError: database_name 'root' already in databases Because that failed, db.databases wasn't changed: >>> len(db.databases) # still 2 2 You can (still) get a connection to a database this way: >>> cn = db.open() >>> cn # doctest: +ELLIPSIS This is the only connection in this collection right now: >>> cn.connections # doctest: +ELLIPSIS {'root': } Getting a connection to a different database from an existing connection in the same database collection (this enables 'connection binding' within a given thread/transaction/context ...): >>> cn2 = cn.get_connection('notroot') >>> cn2 # doctest: +ELLIPSIS Now there are two connections in that collection: >>> cn2.connections is cn.connections True >>> len(cn2.connections) 2 >>> names = cn.connections.keys(); names.sort(); print names ['notroot', 'root'] So long as this database group remains open, the same Connection objects are returned: >>> cn.get_connection('root') is cn True >>> cn.get_connection('notroot') is cn2 True >>> cn2.get_connection('root') is cn True >>> cn2.get_connection('notroot') is cn2 True Of course trying to get a connection for a database not in the group raises an exception: >>> cn.get_connection('no way') Traceback (most recent call last): ... KeyError: 'no way' Clean up: >>> for a_db in dbmap.values(): ... a_db.close() Zope-2.8.7-final/lib/python/ZODB/tests/test_doctest_files.py0000664000076400007640000000152710435502423022772 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from zope.testing.doctestunit import DocFileSuite def test_suite(): return DocFileSuite("dbopen.txt", "multidb.txt", "synchronizers.txt", ) Zope-2.8.7-final/lib/python/ZODB/tests/testZODB.py0000664000076400007640000006064410435502423020547 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest import ZODB import ZODB.FileStorage from ZODB.POSException import ReadConflictError, ConflictError from ZODB.POSException import TransactionFailedError from ZODB.tests.warnhook import WarningsHook from persistent import Persistent from persistent.mapping import PersistentMapping import transaction class P(Persistent): pass class Independent(Persistent): def _p_independent(self): return 1 class DecoyIndependent(Persistent): def _p_independent(self): return 0 class ZODBTests(unittest.TestCase): def setUp(self): self._storage = ZODB.FileStorage.FileStorage( 'ZODBTests.fs', create=1) self._db = ZODB.DB(self._storage) def populate(self): transaction.begin() conn = self._db.open() root = conn.root() root['test'] = pm = PersistentMapping() for n in range(100): pm[n] = PersistentMapping({0: 100 - n}) transaction.get().note('created test data') transaction.commit() conn.close() def tearDown(self): self._db.close() self._storage.cleanup() def checkExportImport(self, abort_it=False): self.populate() conn = self._db.open() try: self.duplicate(conn, abort_it) finally: conn.close() conn = self._db.open() try: self.verify(conn, abort_it) finally: conn.close() def duplicate(self, conn, abort_it): transaction.begin() transaction.get().note('duplication') root = conn.root() ob = root['test'] assert len(ob) > 10, 'Insufficient test data' try: import tempfile f = tempfile.TemporaryFile() ob._p_jar.exportFile(ob._p_oid, f) assert f.tell() > 0, 'Did not export correctly' f.seek(0) new_ob = ob._p_jar.importFile(f) self.assertEqual(new_ob, ob) root['dup'] = new_ob f.close() if abort_it: transaction.abort() else: transaction.commit() except: transaction.abort() raise def verify(self, conn, abort_it): transaction.begin() root = conn.root() ob = root['test'] try: ob2 = root['dup'] except KeyError: if abort_it: # Passed the test. return else: raise else: self.failUnless(not abort_it, 'Did not abort duplication') l1 = list(ob.items()) l1.sort() l2 = list(ob2.items()) l2.sort() l1 = map(lambda (k, v): (k, v[0]), l1) l2 = map(lambda (k, v): (k, v[0]), l2) self.assertEqual(l1, l2) self.assert_(ob._p_oid != ob2._p_oid) self.assertEqual(ob._p_jar, ob2._p_jar) oids = {} for v in ob.values(): oids[v._p_oid] = 1 for v in ob2.values(): assert not oids.has_key(v._p_oid), ( 'Did not fully separate duplicate from original') transaction.commit() def checkExportImportAborted(self): self.checkExportImport(abort_it=True) def checkVersionOnly(self): # Make sure the changes to make empty transactions a no-op # still allow things like abortVersion(). This should work # because abortVersion() calls tpc_begin() itself. conn = self._db.open("version") try: r = conn.root() r[1] = 1 transaction.commit() finally: conn.close() self._db.abortVersion("version") transaction.commit() def checkResetCache(self): # The cache size after a reset should be 0. Note that # _resetCache is not a public API, but the resetCaches() # function is, and resetCaches() causes _resetCache() to be # called. self.populate() conn = self._db.open() conn.root() self.assert_(len(conn._cache) > 0) # Precondition conn._resetCache() self.assertEqual(len(conn._cache), 0) def checkResetCachesAPI(self): # Checks the resetCaches() API. # (resetCaches used to be called updateCodeTimestamp.) self.populate() conn = self._db.open() conn.root() self.assert_(len(conn._cache) > 0) # Precondition ZODB.Connection.resetCaches() conn.close() self.assert_(len(conn._cache) > 0) # Still not flushed conn._setDB(self._db) # simulate the connection being reopened self.assertEqual(len(conn._cache), 0) def checkExplicitTransactionManager(self): # Test of transactions that apply to only the connection, # not the thread. tm1 = transaction.TransactionManager() conn1 = self._db.open(transaction_manager=tm1) tm2 = transaction.TransactionManager() conn2 = self._db.open(transaction_manager=tm2) try: r1 = conn1.root() r2 = conn2.root() if r1.has_key('item'): del r1['item'] tm1.get().commit() r1.get('item') r2.get('item') r1['item'] = 1 tm1.get().commit() self.assertEqual(r1['item'], 1) # r2 has not seen a transaction boundary, # so it should be unchanged. self.assertEqual(r2.get('item'), None) conn2.sync() # Now r2 is updated. self.assertEqual(r2['item'], 1) # Now, for good measure, send an update in the other direction. r2['item'] = 2 tm2.get().commit() self.assertEqual(r1['item'], 1) self.assertEqual(r2['item'], 2) conn1.sync() conn2.sync() self.assertEqual(r1['item'], 2) self.assertEqual(r2['item'], 2) finally: conn1.close() conn2.close() def checkLocalTransactions(self): # Test of transactions that apply to only the connection, # not the thread. conn1 = self._db.open() conn2 = self._db.open() hook = WarningsHook() hook.install() try: conn1.setLocalTransaction() conn2.setLocalTransaction() r1 = conn1.root() r2 = conn2.root() if r1.has_key('item'): del r1['item'] conn1.getTransaction().commit() r1.get('item') r2.get('item') r1['item'] = 1 conn1.getTransaction().commit() self.assertEqual(r1['item'], 1) # r2 has not seen a transaction boundary, # so it should be unchanged. self.assertEqual(r2.get('item'), None) conn2.sync() # Now r2 is updated. self.assertEqual(r2['item'], 1) # Now, for good measure, send an update in the other direction. r2['item'] = 2 conn2.getTransaction().commit() self.assertEqual(r1['item'], 1) self.assertEqual(r2['item'], 2) conn1.sync() conn2.sync() self.assertEqual(r1['item'], 2) self.assertEqual(r2['item'], 2) for msg, obj, filename, lineno in hook.warnings: self.assert_(msg in [ "This will be removed in ZODB 3.6:\n" "setLocalTransaction() is deprecated. " "Use the transaction_manager argument " "to DB.open() instead.", "This will be removed in ZODB 3.6:\n" "getTransaction() is deprecated. " "Use the transaction_manager argument " "to DB.open() instead, or access " ".transaction_manager directly on the Connection."]) finally: conn1.close() conn2.close() hook.uninstall() def checkReadConflict(self): self.obj = P() self.readConflict() def readConflict(self, shouldFail=True): # Two transactions run concurrently. Each reads some object, # then one commits and the other tries to read an object # modified by the first. This read should fail with a conflict # error because the object state read is not necessarily # consistent with the objects read earlier in the transaction. tm1 = transaction.TransactionManager() conn = self._db.open(mvcc=False, transaction_manager=tm1) r1 = conn.root() r1["p"] = self.obj self.obj.child1 = P() tm1.get().commit() # start a new transaction with a new connection tm2 = transaction.TransactionManager() cn2 = self._db.open(mvcc=False, transaction_manager=tm2) # start a new transaction with the other connection r2 = cn2.root() self.assertEqual(r1._p_serial, r2._p_serial) self.obj.child2 = P() tm1.get().commit() # resume the transaction using cn2 obj = r2["p"] # An attempt to access obj should fail, because r2 was read # earlier in the transaction and obj was modified by the othe # transaction. if shouldFail: self.assertRaises(ReadConflictError, lambda: obj.child1) # And since ReadConflictError was raised, attempting to commit # the transaction should re-raise it. checkNotIndependent() # failed this part of the test for a long time. self.assertRaises(ReadConflictError, tm2.get().commit) # And since that commit failed, trying to commit again should # fail again. self.assertRaises(TransactionFailedError, tm2.get().commit) # And again. self.assertRaises(TransactionFailedError, tm2.get().commit) # Etc. self.assertRaises(TransactionFailedError, tm2.get().commit) else: # make sure that accessing the object succeeds obj.child1 tm2.get().abort() def checkReadConflictIgnored(self): # Test that an application that catches a read conflict and # continues can not commit the transaction later. root = self._db.open(mvcc=False).root() root["real_data"] = real_data = PersistentMapping() root["index"] = index = PersistentMapping() real_data["a"] = PersistentMapping({"indexed_value": 0}) real_data["b"] = PersistentMapping({"indexed_value": 1}) index[1] = PersistentMapping({"b": 1}) index[0] = PersistentMapping({"a": 1}) transaction.commit() # load some objects from one connection tm = transaction.TransactionManager() cn2 = self._db.open(mvcc=False, transaction_manager=tm) r2 = cn2.root() real_data2 = r2["real_data"] index2 = r2["index"] real_data["b"]["indexed_value"] = 0 del index[1]["b"] index[0]["b"] = 1 transaction.commit() del real_data2["a"] try: del index2[0]["a"] except ReadConflictError: # This is the crux of the text. Ignore the error. pass else: self.fail("No conflict occurred") # real_data2 still ready to commit self.assert_(real_data2._p_changed) # index2 values not ready to commit self.assert_(not index2._p_changed) self.assert_(not index2[0]._p_changed) self.assert_(not index2[1]._p_changed) self.assertRaises(ReadConflictError, tm.get().commit) self.assertRaises(TransactionFailedError, tm.get().commit) tm.get().abort() def checkIndependent(self): self.obj = Independent() self.readConflict(shouldFail=False) def checkNotIndependent(self): self.obj = DecoyIndependent() self.readConflict() def checkSubtxnCommitDoesntGetInvalidations(self): # Prior to ZODB 3.2.9 and 3.4, Connection.tpc_finish() processed # invalidations even for a subtxn commit. This could make # inconsistent state visible after a subtxn commit. There was a # suspicion that POSKeyError was possible as a result, but I wasn't # able to construct a case where that happened. # Set up the database, to hold # root --> "p" -> value = 1 # --> "q" -> value = 2 tm1 = transaction.TransactionManager() conn = self._db.open(transaction_manager=tm1) r1 = conn.root() p = P() p.value = 1 r1["p"] = p q = P() q.value = 2 r1["q"] = q tm1.commit() # Now txn T1 changes p.value to 3 locally (subtxn commit). p.value = 3 tm1.commit(True) # Start new txn T2 with a new connection. tm2 = transaction.TransactionManager() cn2 = self._db.open(transaction_manager=tm2) r2 = cn2.root() p2 = r2["p"] self.assertEqual(p._p_oid, p2._p_oid) # T2 shouldn't see T1's change of p.value to 3, because T1 didn't # commit yet. self.assertEqual(p2.value, 1) # Change p.value to 4, and q.value to 5. Neither should be visible # to T1, because T1 is still in progress. p2.value = 4 q2 = r2["q"] self.assertEqual(q._p_oid, q2._p_oid) self.assertEqual(q2.value, 2) q2.value = 5 tm2.commit() # Back to T1. p and q still have the expected values. rt = conn.root() self.assertEqual(rt["p"].value, 3) self.assertEqual(rt["q"].value, 2) # Now do another subtxn commit in T1. This shouldn't change what # T1 sees for p and q. rt["r"] = P() tm1.commit(True) # Doing that subtxn commit in T1 should not process invalidations # from T2's commit. p.value should still be 3 here (because that's # what T1 subtxn-committed earlier), and q.value should still be 2. # Prior to ZODB 3.2.9 and 3.4, q.value was 5 here. rt = conn.root() try: self.assertEqual(rt["p"].value, 3) self.assertEqual(rt["q"].value, 2) finally: tm1.abort() def checkReadConflictErrorClearedDuringAbort(self): # When a transaction is aborted, the "memory" of which # objects were the cause of a ReadConflictError during # that transaction should be cleared. root = self._db.open(mvcc=False).root() data = PersistentMapping({'d': 1}) root["data"] = data transaction.commit() # Provoke a ReadConflictError. tm2 = transaction.TransactionManager() cn2 = self._db.open(mvcc=False, transaction_manager=tm2) r2 = cn2.root() data2 = r2["data"] data['d'] = 2 transaction.commit() try: data2['d'] = 3 except ReadConflictError: pass else: self.fail("No conflict occurred") # Explicitly abort cn2's transaction. tm2.get().abort() # cn2 should retain no memory of the read conflict after an abort(), # but 3.2.3 had a bug wherein it did. data_conflicts = data._p_jar._conflicts data2_conflicts = data2._p_jar._conflicts self.failIf(data_conflicts) self.failIf(data2_conflicts) # this used to fail # And because of that, we still couldn't commit a change to data2['d'] # in the new transaction. cn2.sync() # process the invalidation for data2['d'] data2['d'] = 3 tm2.get().commit() # 3.2.3 used to raise ReadConflictError cn2.close() def checkTxnBeginImpliesAbort(self): # begin() should do an abort() first, if needed. cn = self._db.open() rt = cn.root() rt['a'] = 1 transaction.begin() # should abort adding 'a' to the root rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') # A longstanding bug: this didn't work if changes were only in # subtransactions. transaction.begin() rt = cn.root() rt['a'] = 2 transaction.commit(1) transaction.begin() rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') # One more time, mixing "top level" and subtransaction changes. transaction.begin() rt = cn.root() rt['a'] = 3 transaction.commit(1) rt['b'] = 4 transaction.begin() rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') self.assertRaises(KeyError, rt.__getitem__, 'b') # That used methods of the default transaction *manager*. Alas, # that's not necessarily the same as using methods of the current # transaction, and, in fact, when this test was written, # Transaction.begin() didn't do anything (everything from here # down failed). # Oh, bleech. Since Transaction.begin is also deprecated, we have # to goof around suppressing the deprecation warning. import warnings # First verify that Transaction.begin *is* deprecated, by turning # the warning into an error. warnings.filterwarnings("error", category=DeprecationWarning) self.assertRaises(DeprecationWarning, transaction.get().begin) del warnings.filters[0] # Now ignore DeprecationWarnings for the duration. Use a # try/finally block to ensure we reenable DeprecationWarnings # no matter what. warnings.filterwarnings("ignore", category=DeprecationWarning) try: cn = self._db.open() rt = cn.root() rt['a'] = 1 transaction.get().begin() # should abort adding 'a' to the root rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') # A longstanding bug: this didn't work if changes were only in # subtransactions. transaction.get().begin() rt = cn.root() rt['a'] = 2 transaction.get().commit(1) transaction.get().begin() rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') # One more time, mixing "top level" and subtransaction changes. transaction.get().begin() rt = cn.root() rt['a'] = 3 transaction.get().commit(1) rt['b'] = 4 transaction.get().begin() rt = cn.root() self.assertRaises(KeyError, rt.__getitem__, 'a') self.assertRaises(KeyError, rt.__getitem__, 'b') cn.close() finally: del warnings.filters[0] def checkFailingCommitSticks(self): # See also checkFailingSubtransactionCommitSticks. cn = self._db.open() rt = cn.root() rt['a'] = 1 # Arrange for commit to fail during tpc_vote. poisoned = PoisonedObject(PoisonedJar(break_tpc_vote=True)) transaction.get().register(poisoned) self.assertRaises(PoisonedError, transaction.get().commit) # Trying to commit again fails too. self.assertRaises(TransactionFailedError, transaction.get().commit) self.assertRaises(TransactionFailedError, transaction.get().commit) self.assertRaises(TransactionFailedError, transaction.get().commit) # The change to rt['a'] is lost. self.assertRaises(KeyError, rt.__getitem__, 'a') # Trying to modify an object also fails, because Transaction.join() # also raises TransactionFailedError. self.assertRaises(TransactionFailedError, rt.__setitem__, 'b', 2) # Clean up via abort(), and try again. transaction.get().abort() rt['a'] = 1 transaction.get().commit() self.assertEqual(rt['a'], 1) # Cleaning up via begin() should also work. rt['a'] = 2 transaction.get().register(poisoned) self.assertRaises(PoisonedError, transaction.get().commit) self.assertRaises(TransactionFailedError, transaction.get().commit) # The change to rt['a'] is lost. self.assertEqual(rt['a'], 1) # Trying to modify an object also fails. self.assertRaises(TransactionFailedError, rt.__setitem__, 'b', 2) # Clean up via begin(), and try again. transaction.begin() rt['a'] = 2 transaction.get().commit() self.assertEqual(rt['a'], 2) cn.close() def checkFailingSubtransactionCommitSticks(self): cn = self._db.open() rt = cn.root() rt['a'] = 1 transaction.get().commit(True) self.assertEqual(rt['a'], 1) rt['b'] = 2 # Subtransactions don't do tpc_vote, so we poison tpc_begin. poisoned = PoisonedJar() transaction.get().join(poisoned) poisoned.break_savepoint = True self.assertRaises(PoisonedError, transaction.get().commit, True) # Trying to subtxn-commit again fails too. self.assertRaises(TransactionFailedError, transaction.get().commit, True) self.assertRaises(TransactionFailedError, transaction.get().commit, True) # Top-level commit also fails. self.assertRaises(TransactionFailedError, transaction.get().commit) # The changes to rt['a'] and rt['b'] are lost. self.assertRaises(KeyError, rt.__getitem__, 'a') self.assertRaises(KeyError, rt.__getitem__, 'b') # Trying to modify an object also fails, because Transaction.join() # also raises TransactionFailedError. self.assertRaises(TransactionFailedError, rt.__setitem__, 'b', 2) # Clean up via abort(), and try again. transaction.get().abort() rt['a'] = 1 transaction.get().commit() self.assertEqual(rt['a'], 1) # Cleaning up via begin() should also work. rt['a'] = 2 poisoned = PoisonedJar() transaction.get().join(poisoned) poisoned.break_savepoint = True self.assertRaises(PoisonedError, transaction.get().commit, True) self.assertRaises(TransactionFailedError, transaction.get().commit, True) # The change to rt['a'] is lost. self.assertEqual(rt['a'], 1) # Trying to modify an object also fails. self.assertRaises(TransactionFailedError, rt.__setitem__, 'b', 2) # Clean up via begin(), and try again. transaction.begin() rt['a'] = 2 transaction.get().commit(True) self.assertEqual(rt['a'], 2) transaction.get().commit() cn2 = self._db.open() rt = cn.root() self.assertEqual(rt['a'], 2) cn.close() cn2.close() class PoisonedError(Exception): pass # PoisonedJar arranges to raise exceptions from interesting places. # For whatever reason, subtransaction commits don't call tpc_vote. class PoisonedJar: def __init__(self, break_tpc_begin=False, break_tpc_vote=False, break_savepoint=False): self.break_tpc_begin = break_tpc_begin self.break_tpc_vote = break_tpc_vote self.break_savepoint = break_savepoint def sortKey(self): return str(id(self)) # A way to poison a subtransaction commit. def tpc_begin(self, *args): if self.break_tpc_begin: raise PoisonedError("tpc_begin fails") # A way to poison a top-level commit. def tpc_vote(self, *args): if self.break_tpc_vote: raise PoisonedError("tpc_vote fails") def savepoint(self): if self.break_savepoint: raise PoisonedError("savepoint fails") def commit(*args): pass def abort(*self): pass class PoisonedObject: def __init__(self, poisonedjar): self._p_jar = poisonedjar def test_suite(): return unittest.makeSuite(ZODBTests, 'check') if __name__ == "__main__": unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZODB/tests/testTimeStamp.py0000664000076400007640000001235510435502423021710 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test the TimeStamp utility type""" import time import unittest from persistent.TimeStamp import TimeStamp EPSILON = 0.000001 class TimeStampTests(unittest.TestCase): def checkYMDTimeStamp(self): self._check_ymd(2001, 6, 3) def _check_ymd(self, yr, mo, dy): ts = TimeStamp(yr, mo, dy) self.assertEqual(ts.year(), yr) self.assertEqual(ts.month(), mo) self.assertEqual(ts.day(), dy) self.assertEquals(ts.hour(), 0) self.assertEquals(ts.minute(), 0) self.assertEquals(ts.second(), 0) t = time.gmtime(ts.timeTime()) self.assertEquals(yr, t[0]) self.assertEquals(mo, t[1]) self.assertEquals(dy, t[2]) def checkFullTimeStamp(self): native_ts = int(time.time()) # fractional seconds get in the way t = time.gmtime(native_ts) # the corresponding GMT struct tm ts = TimeStamp(*t[:6]) # Seconds are stored internally via (conceptually) multiplying by # 2**32 then dividing by 60, ending up with a 32-bit integer. # While this gives a lot of room for cramming many distinct # TimeStamps into a second, it's not good at roundtrip accuracy. # For example, 1 second is stored as int(2**32/60) == 71582788. # Converting back gives 71582788*60.0/2**32 == 0.9999999962747097. # In general, we can lose up to 0.999... to truncation during # storing, creating an absolute error up to about 1*60.0/2**32 == # 0.000000014 on the seconds value we get back. This is so even # when we have an exact integral second value going in (as we # do in this test), so we can't expect equality in any comparison # involving seconds. Minutes (etc) are stored exactly, so we # can expect equality for those. self.assert_(abs(ts.timeTime() - native_ts) < EPSILON) self.assertEqual(ts.year(), t[0]) self.assertEqual(ts.month(), t[1]) self.assertEqual(ts.day(), t[2]) self.assertEquals(ts.hour(), t[3]) self.assertEquals(ts.minute(), t[4]) self.assert_(abs(ts.second() - t[5]) < EPSILON) def checkRawTimestamp(self): t = time.gmtime() ts1 = TimeStamp(*t[:6]) ts2 = TimeStamp(`ts1`) self.assertEquals(ts1, ts2) self.assertEquals(ts1.timeTime(), ts2.timeTime()) self.assertEqual(ts1.year(), ts2.year()) self.assertEqual(ts1.month(), ts2.month()) self.assertEqual(ts1.day(), ts2.day()) self.assertEquals(ts1.hour(), ts2.hour()) self.assertEquals(ts1.minute(), ts2.minute()) self.assert_(abs(ts1.second() - ts2.second()) < EPSILON) def checkDictKey(self): t = time.gmtime() ts1 = TimeStamp(*t[:6]) ts2 = TimeStamp(2000, *t[1:6]) d = {} d[ts1] = 1 d[ts2] = 2 self.assertEquals(len(d), 2) def checkCompare(self): ts1 = TimeStamp(1972, 6, 27) ts2 = TimeStamp(1971, 12, 12) self.assert_(ts1 > ts2) self.assert_(ts2 <= ts1) def checkLaterThan(self): t = time.gmtime() ts = TimeStamp(*t[:6]) ts2 = ts.laterThan(ts) self.assert_(ts2 > ts) # TODO: should test for bogus inputs to TimeStamp constructor def checkTimeStamp(self): # Alternate test suite t = TimeStamp(2002, 1, 23, 10, 48, 5) # GMT self.assertEquals(str(t), '2002-01-23 10:48:05.000000') self.assertEquals(repr(t), '\x03B9H\x15UUU') self.assertEquals(TimeStamp('\x03B9H\x15UUU'), t) self.assertEquals(t.year(), 2002) self.assertEquals(t.month(), 1) self.assertEquals(t.day(), 23) self.assertEquals(t.hour(), 10) self.assertEquals(t.minute(), 48) self.assertEquals(round(t.second()), 5) self.assertEquals(t.timeTime(), 1011782885) t1 = TimeStamp(2002, 1, 23, 10, 48, 10) self.assertEquals(str(t1), '2002-01-23 10:48:10.000000') self.assert_(t == t) self.assert_(t != t1) self.assert_(t < t1) self.assert_(t <= t1) self.assert_(t1 >= t) self.assert_(t1 > t) self.failIf(t == t1) self.failIf(t != t) self.failIf(t > t1) self.failIf(t >= t1) self.failIf(t1 < t) self.failIf(t1 <= t) self.assertEquals(cmp(t, t), 0) self.assertEquals(cmp(t, t1), -1) self.assertEquals(cmp(t1, t), 1) self.assertEquals(t1.laterThan(t), t1) self.assert_(t.laterThan(t1) > t1) self.assertEquals(TimeStamp(2002,1,23), TimeStamp(2002,1,23,0,0,0)) def test_suite(): return unittest.makeSuite(TimeStampTests, 'check') Zope-2.8.7-final/lib/python/ZODB/tests/warnhook.py0000664000076400007640000000410310435502423020725 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import warnings class WarningsHook: """Hook to capture warnings generated by Python. The function warnings.showwarning() is designed to be hooked by application code, allowing the application to customize the way it handles warnings. This hook captures the unformatted warning information and stores it in a list. A test can inspect this list after the test is over. Issues: The warnings module has lots of delicate internal state. If a warning has been reported once, it won't be reported again. It may be necessary to extend this class with a mechanism for modifying the internal state so that we can be guaranteed a warning will be reported. If Python is run with a warnings filter, e.g. python -Werror, then a test that is trying to inspect a particular warning will fail. Perhaps this class can be extended to install more-specific filters the test to work anyway. """ def __init__(self): self.original = None self.warnings = [] def install(self): self.original = warnings.showwarning warnings.showwarning = self.showwarning def uninstall(self): assert self.original is not None warnings.showwarning = self.original self.original = None def showwarning(self, message, category, filename, lineno): self.warnings.append((str(message), category, filename, lineno)) def clear(self): self.warnings = [] Zope-2.8.7-final/lib/python/ZODB/tests/util.py0000664000076400007640000000221210435502423020051 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Conventience function for creating test databases $Id: util.py 29640 2005-03-22 20:51:18Z tim_one $ """ import time import persistent import transaction from ZODB.MappingStorage import MappingStorage from ZODB.DB import DB as _DB def DB(name='Test'): return _DB(MappingStorage(name)) def commit(): transaction.commit() def pack(db): db.pack(time.time()+1) class P(persistent.Persistent): def __init__(self, name): self.name = name def __repr__(self): return 'P(%s)' % self.name Zope-2.8.7-final/lib/python/ZODB/tests/testCache.py0000664000076400007640000003410310435502423021003 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """A few simple tests of the public cache API. Each DB Connection has a separate PickleCache. The Cache serves two purposes. It acts like a memo for unpickling. It also keeps recent objects in memory under the assumption that they may be used again. """ import gc import time import unittest import threading from persistent.cPickleCache import PickleCache from persistent.mapping import PersistentMapping import transaction import ZODB import ZODB.MappingStorage from ZODB.tests.MinPO import MinPO from ZODB.utils import p64 from persistent import Persistent class CacheTestBase(unittest.TestCase): def setUp(self): store = ZODB.MappingStorage.MappingStorage() self.db = ZODB.DB(store, cache_size = self.CACHE_SIZE) self.conns = [] def tearDown(self): for conn in self.conns: conn.close() self.db.close() CACHE_SIZE = 20 def noodle_new_connection(self): """Do some reads and writes on a new connection.""" c = self.db.open() self.conns.append(c) self.noodle_connection(c) def noodle_connection(self, c): r = c.root() i = len(self.conns) d = r.get(i) if d is None: d = r[i] = PersistentMapping() transaction.commit() for i in range(15): o = d.get(i) if o is None: o = d[i] = MinPO(i) o.value += 1 transaction.commit() # CantGetRidOfMe is used by checkMinimizeTerminates. make_trouble = True class CantGetRidOfMe(MinPO): def __init__(self, value): MinPO.__init__(self, value) self.an_attribute = 42 def __del__(self): # Referencing an attribute of self causes self to be # loaded into the cache again, which also resurrects # self. if make_trouble: self.an_attribute class DBMethods(CacheTestBase): __super_setUp = CacheTestBase.setUp def setUp(self): self.__super_setUp() for i in range(4): self.noodle_new_connection() def checkCacheDetail(self): for name, count in self.db.cacheDetail(): self.assert_(isinstance(name, str)) self.assert_(isinstance(count, int)) def checkCacheExtremeDetail(self): expected = ['conn_no', 'id', 'oid', 'rc', 'klass', 'state'] for dict in self.db.cacheExtremeDetail(): for k, v in dict.items(): self.assert_(k in expected) # TODO: not really sure how to do a black box test of the cache. # Should the full sweep and minimize calls always remove things? def checkFullSweep(self): old_size = self.db.cacheSize() self.db.cacheFullSweep() new_size = self.db.cacheSize() self.assert_(new_size < old_size, "%s < %s" % (old_size, new_size)) def checkMinimize(self): old_size = self.db.cacheSize() self.db.cacheMinimize() new_size = self.db.cacheSize() self.assert_(new_size < old_size, "%s < %s" % (old_size, new_size)) def checkMinimizeTerminates(self): # This is tricky. cPickleCache had a case where it could get into # an infinite loop, but we don't want the test suite to hang # if this bug reappears. So this test spawns a thread to run the # dangerous operation, and the main thread complains if the worker # thread hasn't finished in 30 seconds (arbitrary, but way more # than enough). In that case, the worker thread will continue # running forever (until killed externally), but at least the # test suite will move on. # # The bug was triggered by having a persistent object whose __del__ # method references an attribute of the object. An attempt to # ghostify such an object will clear the attribute, and if the # cache also releases the last Python reference to the object then # (due to ghostifying it), the __del__ method gets invoked. # Referencing the attribute loads the object again, and also # puts it back into the cPickleCache. If the cache implementation # isn't looking out for this, it can get into an infinite loop # then, endlessly trying to ghostify an object that in turn keeps # unghostifying itself again. class Worker(threading.Thread): def __init__(self, testcase): threading.Thread.__init__(self) self.testcase = testcase def run(self): global make_trouble # Make CantGetRidOfMe.__del__ dangerous. make_trouble = True conn = self.testcase.conns[0] r = conn.root() d = r[1] for i in range(len(d)): d[i] = CantGetRidOfMe(i) transaction.commit() self.testcase.db.cacheMinimize() # Defang the nasty objects. Else, because they're # immortal now, they hang around and create trouble # for subsequent tests. make_trouble = False self.testcase.db.cacheMinimize() w = Worker(self) w.start() w.join(30) if w.isAlive(): self.fail("cacheMinimize still running after 30 seconds -- " "almost certainly in an infinite loop") # TODO: don't have an explicit test for incrgc, because the # connection and database call it internally. # Same for the get and invalidate methods. def checkLRUitems(self): # get a cache c = self.conns[0]._cache c.lru_items() def checkClassItems(self): c = self.conns[0]._cache c.klass_items() class LRUCacheTests(CacheTestBase): def checkLRU(self): # verify the LRU behavior of the cache dataset_size = 5 CACHE_SIZE = dataset_size*2+1 # a cache big enough to hold the objects added in two # transactions, plus the root object self.db.setCacheSize(CACHE_SIZE) c = self.db.open() r = c.root() l = {} # the root is the only thing in the cache, because all the # other objects are new self.assertEqual(len(c._cache), 1) # run several transactions for t in range(5): for i in range(dataset_size): l[(t,i)] = r[i] = MinPO(i) transaction.commit() # commit() will register the objects, placing them in the # cache. at the end of commit, the cache will be reduced # down to CACHE_SIZE items if len(l)>CACHE_SIZE: self.assertEqual(c._cache.ringlen(), CACHE_SIZE) for i in range(dataset_size): # Check objects added in the first two transactions. # They must all be ghostified. self.assertEqual(l[(0,i)]._p_changed, None) self.assertEqual(l[(1,i)]._p_changed, None) # Check objects added in the last two transactions. # They must all still exist in memory, but have # had their changes flushed self.assertEqual(l[(3,i)]._p_changed, 0) self.assertEqual(l[(4,i)]._p_changed, 0) # Of the objects added in the middle transaction, most # will have been ghostified. There is one cache slot # that may be occupied by either one of those objects or # the root, depending on precise order of access. We do # not bother to check this def checkSize(self): self.assertEqual(self.db.cacheSize(), 0) self.assertEqual(self.db.cacheDetailSize(), []) CACHE_SIZE = 10 self.db.setCacheSize(CACHE_SIZE) CONNS = 3 for i in range(CONNS): self.noodle_new_connection() self.assertEquals(self.db.cacheSize(), CACHE_SIZE * CONNS) details = self.db.cacheDetailSize() self.assertEquals(len(details), CONNS) for d in details: self.assertEquals(d['ngsize'], CACHE_SIZE) # The assertion below is non-sensical # The (poorly named) cache size is a target for non-ghosts. # The cache *usually* contains non-ghosts, so that the # size normally exceeds the target size. #self.assertEquals(d['size'], CACHE_SIZE) def checkDetail(self): CACHE_SIZE = 10 self.db.setCacheSize(CACHE_SIZE) CONNS = 3 for i in range(CONNS): self.noodle_new_connection() gc.collect() # Obscure: The above gc.collect call is necessary to make this test # pass. # # This test then only works because the order of computations # and object accesses in the "noodle" calls is such that the # persistent mapping containing the MinPO objects is # deactivated before the MinPO objects. # # - Without the gc call, the cache will contain ghost MinPOs # and the check of the MinPO count below will fail. That's # because the counts returned by cacheDetail include ghosts. # # - If the mapping object containing the MinPOs isn't # deactivated, there will be one fewer non-ghost MinPO and # the test will fail anyway. # # This test really needs to be thought through and documented # better. for klass, count in self.db.cacheDetail(): if klass.endswith('MinPO'): self.assertEqual(count, CONNS * CACHE_SIZE) if klass.endswith('PersistentMapping'): # one root per connection self.assertEqual(count, CONNS) for details in self.db.cacheExtremeDetail(): # one 'details' dict per object if details['klass'].endswith('PersistentMapping'): self.assertEqual(details['state'], None) else: self.assert_(details['klass'].endswith('MinPO')) self.assertEqual(details['state'], 0) # The cache should never hold an unreferenced ghost. if details['state'] is None: # i.e., it's a ghost self.assert_(details['rc'] > 0) class StubDataManager: def setklassstate(self, object): pass class StubObject(Persistent): pass class CacheErrors(unittest.TestCase): def setUp(self): self.jar = StubDataManager() self.cache = PickleCache(self.jar) def checkGetBogusKey(self): self.assertEqual(self.cache.get(p64(0)), None) try: self.cache[12] except KeyError: pass else: self.fail("expected KeyError") try: self.cache[12] = 12 except TypeError: pass else: self.fail("expected TyepError") try: del self.cache[12] except TypeError: pass else: self.fail("expected TypeError") def checkBogusObject(self): def add(key, obj): self.cache[key] = obj key = p64(2) # value isn't persistent self.assertRaises(TypeError, add, key, 12) o = StubObject() # o._p_oid == None self.assertRaises(TypeError, add, key, o) o._p_oid = p64(3) self.assertRaises(ValueError, add, key, o) o._p_oid = key # o._p_jar == None self.assertRaises(Exception, add, key, o) o._p_jar = self.jar self.cache[key] = o # make sure it can be added multiple times self.cache[key] = o # same object, different keys self.assertRaises(ValueError, add, p64(0), o) def checkTwoCaches(self): jar2 = StubDataManager() cache2 = PickleCache(jar2) o = StubObject() key = o._p_oid = p64(1) o._p_jar = jar2 cache2[key] = o try: self.cache[key] = o except ValueError: pass else: self.fail("expected ValueError because object already in cache") def checkReadOnlyAttrsWhenCached(self): o = StubObject() key = o._p_oid = p64(1) o._p_jar = self.jar self.cache[key] = o try: o._p_oid = p64(2) except ValueError: pass else: self.fail("expect that you can't change oid of cached object") try: del o._p_jar except ValueError: pass else: self.fail("expect that you can't delete jar of cached object") def checkTwoObjsSameOid(self): # Try to add two distinct objects with the same oid to the cache. # This has always been an error, but the error message prior to # ZODB 3.2.6 didn't make sense. This test verifies that (a) an # exception is raised; and, (b) the error message is the intended # one. obj1 = StubObject() key = obj1._p_oid = p64(1) obj1._p_jar = self.jar self.cache[key] = obj1 obj2 = StubObject() obj2._p_oid = key obj2._p_jar = self.jar try: self.cache[key] = obj2 except ValueError, detail: self.assertEqual(str(detail), "A different object already has the same oid") else: self.fail("two objects with the same oid should have failed") def test_suite(): s = unittest.makeSuite(DBMethods, 'check') s.addTest(unittest.makeSuite(LRUCacheTests, 'check')) s.addTest(unittest.makeSuite(CacheErrors, 'check')) return s Zope-2.8.7-final/lib/python/ZODB/tests/testConnectionSavepoint.py0000664000076400007640000001065510435502423023776 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of savepoint feature $Id: testConnectionSavepoint.py 38051 2005-08-23 21:39:15Z tim_one $ """ import unittest from zope.testing import doctest import persistent.dict, transaction def testAddingThenModifyThenAbort(): """\ We ran into a problem in which abort failed after adding an object in a savepoint and then modifying the object. The problem was that, on commit, the savepoint was aborted before the modifications were aborted. Because the object was added in the savepoint, its _p_oid and _p_jar were cleared when the savepoint was aborted. The object was in the registered-object list. There's an invariant for this list that states that all objects in the list should have an oid and (correct) jar. The fix was to abort work done after the savepoint before aborting the savepoint. >>> import ZODB.tests.util >>> db = ZODB.tests.util.DB() >>> connection = db.open() >>> root = connection.root() >>> ob = persistent.dict.PersistentDict() >>> root['ob'] = ob >>> sp = transaction.savepoint() >>> ob.x = 1 >>> transaction.abort() """ def testModifyThenSavePointThenModifySomeMoreThenCommit(): """\ We got conflict errors when we committed after we modified an object in a savepoint, and then modified it some more after the last savepoint. The problem was that we were effectively commiting the object twice -- when commiting the current data and when committing the savepoint. The fix was to first make a new savepoint to move new changes to the savepoint storage and *then* to commit the savepoint storage. (This is similar to the strategy that was used for subtransactions prior to savepoints.) >>> import ZODB.tests.util >>> db = ZODB.tests.util.DB() >>> connection = db.open() >>> root = connection.root() >>> sp = transaction.savepoint() >>> root['a'] = 1 >>> sp = transaction.savepoint() >>> root['a'] = 2 >>> transaction.commit() """ def testSavepointDoesCacheGC(): """\ Although the interface doesn't guarantee this internal detail, making a savepoint should do incremental gc on connection memory caches. Indeed, one traditional use for savepoints (started by the older, related "subtransaction commit" idea) is simply to free memory space midstream during a long transaction. Before ZODB 3.4.2, making a savepoint failed to trigger cache gc, and this test verifies that it now does. >>> import ZODB >>> from ZODB.tests.MinPO import MinPO >>> from ZODB.MappingStorage import MappingStorage >>> import transaction >>> CACHESIZE = 5 # something tiny >>> LOOPCOUNT = CACHESIZE * 10 >>> st = MappingStorage("Test") >>> db = ZODB.DB(st, cache_size=CACHESIZE) >>> cn = db.open() >>> rt = cn.root() Now attach substantially more than CACHESIZE persistent objects to the root: >>> for i in range(LOOPCOUNT): ... rt[i] = MinPO(i) >>> transaction.commit() Now modify all of them; the cache should contain LOOPCOUNT MinPO objects then, + 1 for the root object: >>> for i in range(LOOPCOUNT): ... obj = rt[i] ... obj.value = -i >>> len(cn._cache) == LOOPCOUNT + 1 True Making a savepoint at this time used to leave the cache holding the same number of objects. Make sure the cache shrinks now instead. >>> dummy = transaction.savepoint() >>> len(cn._cache) <= CACHESIZE + 1 True Verify all the values are as expected: >>> failures = [] >>> for i in range(LOOPCOUNT): ... obj = rt[i] ... if obj.value != -i: ... failures.append(obj) >>> failures [] >>> transaction.abort() >>> db.close() """ def test_suite(): return unittest.TestSuite(( doctest.DocFileSuite('testConnectionSavepoint.txt'), doctest.DocTestSuite(), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZODB/tests/sampledm.py0000664000076400007640000002507110435502423020706 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample objects for use in tests $Id: sampledm.py 29891 2005-04-06 22:35:05Z tim_one $ """ class DataManager(object): """Sample data manager This class provides a trivial data-manager implementation and doc strings to illustrate the the protocol and to provide a tool for writing tests. Our sample data manager has state that is updated through an inc method and through transaction operations. When we create a sample data manager: >>> dm = DataManager() It has two bits of state, state: >>> dm.state 0 and delta: >>> dm.delta 0 Both of which are initialized to 0. state is meant to model committed state, while delta represents tentative changes within a transaction. We change the state by calling inc: >>> dm.inc() which updates delta: >>> dm.delta 1 but state isn't changed until we commit the transaction: >>> dm.state 0 To commit the changes, we use 2-phase commit. We execute the first stage by calling prepare. We need to pass a transation. Our sample data managers don't really use the transactions for much, so we'll be lazy and use strings for transactions: >>> t1 = '1' >>> dm.prepare(t1) The sample data manager updates the state when we call prepare: >>> dm.state 1 >>> dm.delta 1 This is mainly so we can detect some affect of calling the methods. Now if we call commit: >>> dm.commit(t1) Our changes are"permanent". The state reflects the changes and the delta has been reset to 0. >>> dm.state 1 >>> dm.delta 0 """ def __init__(self): self.state = 0 self.sp = 0 self.transaction = None self.delta = 0 self.prepared = False def inc(self, n=1): self.delta += n def prepare(self, transaction): """Prepare to commit data >>> dm = DataManager() >>> dm.inc() >>> t1 = '1' >>> dm.prepare(t1) >>> dm.commit(t1) >>> dm.state 1 >>> dm.inc() >>> t2 = '2' >>> dm.prepare(t2) >>> dm.abort(t2) >>> dm.state 1 It is en error to call prepare more than once without an intervening commit or abort: >>> dm.prepare(t1) >>> dm.prepare(t1) Traceback (most recent call last): ... TypeError: Already prepared >>> dm.prepare(t2) Traceback (most recent call last): ... TypeError: Already prepared >>> dm.abort(t1) If there was a preceeding savepoint, the transaction must match: >>> rollback = dm.savepoint(t1) >>> dm.prepare(t2) Traceback (most recent call last): ,,, TypeError: ('Transaction missmatch', '2', '1') >>> dm.prepare(t1) """ if self.prepared: raise TypeError('Already prepared') self._checkTransaction(transaction) self.prepared = True self.transaction = transaction self.state += self.delta def _checkTransaction(self, transaction): if (transaction is not self.transaction and self.transaction is not None): raise TypeError("Transaction missmatch", transaction, self.transaction) def abort(self, transaction): """Abort a transaction The abort method can be called before two-phase commit to throw away work done in the transaction: >>> dm = DataManager() >>> dm.inc() >>> dm.state, dm.delta (0, 1) >>> t1 = '1' >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) The abort method also throws away work done in savepoints: >>> dm.inc() >>> r = dm.savepoint(t1) >>> dm.inc() >>> r = dm.savepoint(t1) >>> dm.state, dm.delta (0, 2) >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) If savepoints are used, abort must be passed the same transaction: >>> dm.inc() >>> r = dm.savepoint(t1) >>> t2 = '2' >>> dm.abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> dm.abort(t1) The abort method is also used to abort a two-phase commit: >>> dm.inc() >>> dm.state, dm.delta (0, 1) >>> dm.prepare(t1) >>> dm.state, dm.delta (1, 1) >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) Of course, the transactions passed to prepare and abort must match: >>> dm.prepare(t1) >>> dm.abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> dm.abort(t1) """ self._checkTransaction(transaction) if self.transaction is not None: self.transaction = None if self.prepared: self.state -= self.delta self.prepared = False self.delta = 0 def commit(self, transaction): """Complete two-phase commit >>> dm = DataManager() >>> dm.state 0 >>> dm.inc() We start two-phase commit by calling prepare: >>> t1 = '1' >>> dm.prepare(t1) We complete it by calling commit: >>> dm.commit(t1) >>> dm.state 1 It is an error ro call commit without calling prepare first: >>> dm.inc() >>> t2 = '2' >>> dm.commit(t2) Traceback (most recent call last): ... TypeError: Not prepared to commit >>> dm.prepare(t2) >>> dm.commit(t2) If course, the transactions given to prepare and commit must be the same: >>> dm.inc() >>> t3 = '3' >>> dm.prepare(t3) >>> dm.commit(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '3') """ if not self.prepared: raise TypeError('Not prepared to commit') self._checkTransaction(transaction) self.delta = 0 self.transaction = None self.prepared = False def savepoint(self, transaction): """Provide the ability to rollback transaction state Savepoints provide a way to: - Save partial transaction work. For some data managers, this could allow resources to be used more efficiently. - Provide the ability to revert state to a point in a transaction without aborting the entire transaction. In other words, savepoints support partial aborts. Savepoints don't use two-phase commit. If there are errors in setting or rolling back to savepoints, the application should abort the containing transaction. This is *not* the responsibility of the data manager. Savepoints are always associated with a transaction. Any work done in a savepoint's transaction is tentative until the transaction is committed using two-phase commit. >>> dm = DataManager() >>> dm.inc() >>> t1 = '1' >>> r = dm.savepoint(t1) >>> dm.state, dm.delta (0, 1) >>> dm.inc() >>> dm.state, dm.delta (0, 2) >>> r.rollback() >>> dm.state, dm.delta (0, 1) >>> dm.prepare(t1) >>> dm.commit(t1) >>> dm.state, dm.delta (1, 0) Savepoints must have the same transaction: >>> r1 = dm.savepoint(t1) >>> dm.state, dm.delta (1, 0) >>> dm.inc() >>> dm.state, dm.delta (1, 1) >>> t2 = '2' >>> r2 = dm.savepoint(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> r2 = dm.savepoint(t1) >>> dm.inc() >>> dm.state, dm.delta (1, 2) If we rollback to an earlier savepoint, we discard all work done later: >>> r1.rollback() >>> dm.state, dm.delta (1, 0) and we can no longer rollback to the later savepoint: >>> r2.rollback() Traceback (most recent call last): ... TypeError: ('Attempt to roll back to invalid save point', 3, 2) We can roll back to a savepoint as often as we like: >>> r1.rollback() >>> r1.rollback() >>> r1.rollback() >>> dm.state, dm.delta (1, 0) >>> dm.inc() >>> dm.inc() >>> dm.inc() >>> dm.state, dm.delta (1, 3) >>> r1.rollback() >>> dm.state, dm.delta (1, 0) But we can't rollback to a savepoint after it has been committed: >>> dm.prepare(t1) >>> dm.commit(t1) >>> r1.rollback() Traceback (most recent call last): ... TypeError: Attempt to rollback stale rollback """ if self.prepared: raise TypeError("Can't get savepoint during two-phase commit") self._checkTransaction(transaction) self.transaction = transaction self.sp += 1 return Rollback(self) class Rollback(object): def __init__(self, dm): self.dm = dm self.sp = dm.sp self.delta = dm.delta self.transaction = dm.transaction def rollback(self): if self.transaction is not self.dm.transaction: raise TypeError("Attempt to rollback stale rollback") if self.dm.sp < self.sp: raise TypeError("Attempt to roll back to invalid save point", self.sp, self.dm.sp) self.dm.sp = self.sp self.dm.delta = self.delta def test_suite(): from zope.testing.doctest import DocTestSuite return DocTestSuite() if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/ZODB/tests/test_datamanageradapter.py0000664000076400007640000001232410435502423023745 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ $Id: test_datamanageradapter.py 30129 2005-04-24 01:29:02Z jim $ """ import unittest from zope.testing.doctest import DocTestSuite from transaction._transaction import DataManagerAdapter from ZODB.tests.sampledm import DataManager def test_normal_commit(): """ So, we have a data manager: >>> dm = DataManager() and we do some work that modifies uncommited state: >>> dm.inc() >>> dm.state, dm.delta (0, 1) Now we'll commit the changes. When the data manager joins a transaction, the transaction will create an adapter. >>> dma = DataManagerAdapter(dm) and register it as a modified object. At commit time, the transaction will get the "jar" like this: >>> jar = getattr(dma, '_p_jar', dma) and, of course, the jar and the adapter will be the same: >>> jar is dma True The transaction will call tpc_begin: >>> t1 = '1' >>> jar.tpc_begin(t1) Then the transaction will call commit on the jar: >>> jar.commit(t1) This doesn't actually do anything. :) >>> dm.state, dm.delta (0, 1) The transaction will then call tpc_vote: >>> jar.tpc_vote(t1) This prepares the data manager: >>> dm.state, dm.delta (1, 1) >>> dm.prepared True Finally, tpc_finish is called: >>> jar.tpc_finish(t1) and the data manager finishes the two-phase commit: >>> dm.state, dm.delta (1, 0) >>> dm.prepared False """ def test_abort(): """ So, we have a data manager: >>> dm = DataManager() and we do some work that modifies uncommited state: >>> dm.inc() >>> dm.state, dm.delta (0, 1) When the data manager joins a transaction, the transaction will create an adapter. >>> dma = DataManagerAdapter(dm) and register it as a modified object. Now we'll abort the transaction. The transaction will get the "jar" like this: >>> jar = getattr(dma, '_p_jar', dma) and, of course, the jar and the adapter will be the same: >>> jar is dma True Then the transaction will call abort on the jar: >>> t1 = '1' >>> jar.abort(t1) Which aborts the changes in the data manager: >>> dm.state, dm.delta (0, 0) """ def test_tpc_abort_phase1(): """ So, we have a data manager: >>> dm = DataManager() and we do some work that modifies uncommited state: >>> dm.inc() >>> dm.state, dm.delta (0, 1) Now we'll commit the changes. When the data manager joins a transaction, the transaction will create an adapter. >>> dma = DataManagerAdapter(dm) and register it as a modified object. At commit time, the transaction will get the "jar" like this: >>> jar = getattr(dma, '_p_jar', dma) and, of course, the jar and the adapter will be the same: >>> jar is dma True The transaction will call tpc_begin: >>> t1 = '1' >>> jar.tpc_begin(t1) Then the transaction will call commit on the jar: >>> jar.commit(t1) This doesn't actually do anything. :) >>> dm.state, dm.delta (0, 1) At this point, the transaction decides to abort. It calls tpc_abort: >>> jar.tpc_abort(t1) Which causes the state of the data manager to be restored: >>> dm.state, dm.delta (0, 0) """ def test_tpc_abort_phase2(): """ So, we have a data manager: >>> dm = DataManager() and we do some work that modifies uncommited state: >>> dm.inc() >>> dm.state, dm.delta (0, 1) Now we'll commit the changes. When the data manager joins a transaction, the transaction will create an adapter. >>> dma = DataManagerAdapter(dm) and register it as a modified object. At commit time, the transaction will get the "jar" like this: >>> jar = getattr(dma, '_p_jar', dma) and, of course, the jar and the adapter will be the same: >>> jar is dma True The transaction will call tpc_begin: >>> t1 = '1' >>> jar.tpc_begin(t1) Then the transaction will call commit on the jar: >>> jar.commit(t1) This doesn't actually do anything. :) >>> dm.state, dm.delta (0, 1) The transaction calls vote: >>> jar.tpc_vote(t1) This prepares the data manager: >>> dm.state, dm.delta (1, 1) >>> dm.prepared True At this point, the transaction decides to abort. It calls tpc_abort: >>> jar.tpc_abort(t1) Which causes the state of the data manager to be restored: >>> dm.state, dm.delta (0, 0) >>> dm.prepared False """ def test_suite(): return DocTestSuite() if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/ZODB/tests/RecoveryStorage.py0000664000076400007640000002700710435502423022230 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """More recovery and iterator tests.""" import transaction from transaction import Transaction from ZODB.tests.IteratorStorage import IteratorDeepCompare from ZODB.tests.StorageTestBase import MinPO, zodb_unpickle, snooze from ZODB import DB from ZODB.serialize import referencesf import time class RecoveryStorage(IteratorDeepCompare): # Requires a setUp() that creates a self._dst destination storage def checkSimpleRecovery(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=11) revid = self._dostore(oid, revid=revid, data=12) revid = self._dostore(oid, revid=revid, data=13) self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) def checkRecoveryAcrossVersions(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=21) revid = self._dostore(oid, revid=revid, data=22) revid = self._dostore(oid, revid=revid, data=23, version='one') revid = self._dostore(oid, revid=revid, data=34, version='one') # Now commit the version t = Transaction() self._storage.tpc_begin(t) self._storage.commitVersion('one', '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) def checkRecoverAbortVersion(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=21, version="one") revid = self._dostore(oid, revid=revid, data=23, version='one') revid = self._dostore(oid, revid=revid, data=34, version='one') # Now abort the version and the creation t = Transaction() self._storage.tpc_begin(t) tid, oids = self._storage.abortVersion('one', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) self.assertEqual(oids, [oid]) self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) # Also make sure the the last transaction has a data record # with None for its data attribute, because we've undone the # object. for s in self._storage, self._dst: iter = s.iterator() for trans in iter: pass # iterate until we get the last one data = trans[0] self.assertRaises(IndexError, lambda i, t=trans: t[i], 1) self.assertEqual(data.oid, oid) self.assertEqual(data.data, None) def checkRecoverUndoInVersion(self): oid = self._storage.new_oid() version = "aVersion" revid_a = self._dostore(oid, data=MinPO(91)) revid_b = self._dostore(oid, revid=revid_a, version=version, data=MinPO(92)) revid_c = self._dostore(oid, revid=revid_b, version=version, data=MinPO(93)) self._undo(self._storage.undoInfo()[0]['id'], [oid]) self._commitVersion(version, '') self._undo(self._storage.undoInfo()[0]['id'], [oid]) # now copy the records to a new storage self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) # The last two transactions were applied directly rather than # copied. So we can't use compare() to verify that they new # transactions are applied correctly. (The new transactions # will have different timestamps for each storage.) self._abortVersion(version) self.assert_(self._storage.versionEmpty(version)) self._undo(self._storage.undoInfo()[0]['id'], [oid]) self.assert_(not self._storage.versionEmpty(version)) # check the data is what we expect it to be data, revid = self._storage.load(oid, version) self.assertEqual(zodb_unpickle(data), MinPO(92)) data, revid = self._storage.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO(91)) # and swap the storages tmp = self._storage self._storage = self._dst self._abortVersion(version) self.assert_(self._storage.versionEmpty(version)) self._undo(self._storage.undoInfo()[0]['id'], [oid]) self.assert_(not self._storage.versionEmpty(version)) # check the data is what we expect it to be data, revid = self._storage.load(oid, version) self.assertEqual(zodb_unpickle(data), MinPO(92)) data, revid = self._storage.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO(91)) # swap them back self._storage = tmp # Now remove _dst and copy all the transactions a second time. # This time we will be able to confirm via compare(). self._dst.close() self._dst.cleanup() self._dst = self.new_dest() self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) def checkRestoreAcrossPack(self): db = DB(self._storage) c = db.open() r = c.root() obj = r["obj1"] = MinPO(1) transaction.commit() obj = r["obj2"] = MinPO(1) transaction.commit() self._dst.copyTransactionsFrom(self._storage) self._dst.pack(time.time(), referencesf) self._undo(self._storage.undoInfo()[0]['id']) # copy the final transaction manually. even though there # was a pack, the restore() ought to succeed. it = self._storage.iterator() final = list(it)[-1] self._dst.tpc_begin(final, final.tid, final.status) for r in final: self._dst.restore(r.oid, r.tid, r.data, r.version, r.data_txn, final) it.close() self._dst.tpc_vote(final) self._dst.tpc_finish(final) def checkPackWithGCOnDestinationAfterRestore(self): raises = self.assertRaises db = DB(self._storage) conn = db.open() root = conn.root() root.obj = obj1 = MinPO(1) txn = transaction.get() txn.note('root -> obj') txn.commit() root.obj.obj = obj2 = MinPO(2) txn = transaction.get() txn.note('root -> obj -> obj') txn.commit() del root.obj txn = transaction.get() txn.note('root -X->') txn.commit() # Now copy the transactions to the destination self._dst.copyTransactionsFrom(self._storage) # Now pack the destination. snooze() self._dst.pack(time.time(), referencesf) # And check to see that the root object exists, but not the other # objects. data, serial = self._dst.load(root._p_oid, '') raises(KeyError, self._dst.load, obj1._p_oid, '') raises(KeyError, self._dst.load, obj2._p_oid, '') def checkRestoreWithMultipleObjectsInUndoRedo(self): from ZODB.FileStorage import FileStorage # Undo creates backpointers in (at least) FileStorage. ZODB 3.2.1 # FileStorage._data_find() had an off-by-8 error, neglecting to # account for the size of the backpointer when searching a # transaction with multiple data records. The results were # unpredictable. For example, it could raise a Python exception # due to passing a negative offset to file.seek(), or could # claim that a transaction didn't have data for an oid despite # that it actually did. # # The former failure mode was seen in real life, in a ZRS secondary # doing recovery. On my box today, the second failure mode is # what happens in this test (with an unpatched _data_find, of # course). Note that the error can only "bite" if more than one # data record is in a transaction, and the oid we're looking for # follows at least one data record with a backpointer. # # Unfortunately, _data_find() is a low-level implementation detail, # and this test does some horrid white-box abuse to test it. is_filestorage = isinstance(self._storage, FileStorage) db = DB(self._storage) c = db.open() r = c.root() # Create some objects. r["obj1"] = MinPO(1) r["obj2"] = MinPO(1) transaction.commit() # Add x attributes to them. r["obj1"].x = 'x1' r["obj2"].x = 'x2' transaction.commit() r = db.open().root() self.assertEquals(r["obj1"].x, 'x1') self.assertEquals(r["obj2"].x, 'x2') # Dirty tricks. if is_filestorage: obj1_oid = r["obj1"]._p_oid obj2_oid = r["obj2"]._p_oid # This will be the offset of the next transaction, which # will contain two backpointers. pos = self._storage.getSize() # Undo the attribute creation. info = self._storage.undoInfo() tid = info[0]['id'] t = Transaction() self._storage.tpc_begin(t) oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) r = db.open().root() self.assertRaises(AttributeError, getattr, r["obj1"], 'x') self.assertRaises(AttributeError, getattr, r["obj2"], 'x') if is_filestorage: # _data_find should find data records for both objects in that # transaction. Without the patch, the second assert failed # (it claimed it couldn't find a data record for obj2) on my # box, but other failure modes were possible. self.assert_(self._storage._data_find(pos, obj1_oid, '') > 0) self.assert_(self._storage._data_find(pos, obj2_oid, '') > 0) # The offset of the next ("redo") transaction. pos = self._storage.getSize() # Undo the undo (restore the attributes). info = self._storage.undoInfo() tid = info[0]['id'] t = Transaction() self._storage.tpc_begin(t) oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) r = db.open().root() self.assertEquals(r["obj1"].x, 'x1') self.assertEquals(r["obj2"].x, 'x2') if is_filestorage: # Again _data_find should find both objects in this txn, and # again the second assert failed on my box. self.assert_(self._storage._data_find(pos, obj1_oid, '') > 0) self.assert_(self._storage._data_find(pos, obj2_oid, '') > 0) # Indirectly provoke .restore(). .restore in turn indirectly # provokes _data_find too, but not usefully for the purposes of # the specific bug this test aims at: copyTransactionsFrom() uses # storage iterators that chase backpointers themselves, and # return the data they point at instead. The result is that # _data_find didn't actually see anything dangerous in this # part of the test. self._dst.copyTransactionsFrom(self._storage) self.compare(self._storage, self._dst) Zope-2.8.7-final/lib/python/ZODB/tests/synchronizers.txt0000664000076400007640000000574310435502423022217 0ustar ajungajungHere are some tests that storage sync() methods get called at appropriate times in the life of a transaction. The tested behavior is new in ZODB 3.4. First define a lightweight storage with a sync() method: >>> import ZODB >>> from ZODB.MappingStorage import MappingStorage >>> import transaction >>> class SimpleStorage(MappingStorage): ... sync_called = False ... ... def sync(self, *args): ... self.sync_called = True Make a change locally: >>> st = SimpleStorage() >>> db = ZODB.DB(st) >>> cn = db.open() >>> rt = cn.root() >>> rt['a'] = 1 Sync should not have been called yet. >>> st.sync_called # False before 3.4 False sync is called by the Connection's afterCompletion() hook after the commit completes. >>> transaction.commit() >>> st.sync_called # False before 3.4 True sync is also called by the afterCompletion() hook after an abort. >>> st.sync_called = False >>> rt['b'] = 2 >>> transaction.abort() >>> st.sync_called # False before 3.4 True And sync is called whenever we explicitly start a new txn, via the newTransaction() hook. >>> st.sync_called = False >>> dummy = transaction.begin() >>> st.sync_called # False before 3.4 True Clean up. Closing db isn't enough -- closing a DB doesn't close its Connections. Leaving our Connection open here can cause the SimpleStorage.sync() method to get called later, during another test, and our doctest-synthesized module globals no longer exist then. You get a weird traceback then ;-) >>> cn.close() One more, very obscure. It was the case that if the first action a new threaded transaction manager saw was a begin() call, then synchronizers registered after that in the same transaction weren't communicated to the Transaction object, and so the synchronizers' afterCompletion() hooks weren't called when the transaction commited. None of the test suites (ZODB's, Zope 2.8's, or Zope3's) caught that, but apparently Zope3 takes this path at some point when serving pages. >>> tm = transaction.ThreadTransactionManager() >>> st.sync_called = False >>> dummy = tm.begin() # we're doing this _before_ opening a connection >>> cn = db.open(transaction_manager=tm) >>> rt = cn.root() # make a change >>> rt['c'] = 3 >>> st.sync_called False Now ensure that cn.afterCompletion() -> st.sync() gets called by commit despite that the Connection registered after the transaction began: >>> tm.commit() >>> st.sync_called True And try the same thing with a non-threaded TM: >>> cn.close() >>> tm = transaction.TransactionManager() >>> st.sync_called = False >>> dummy = tm.begin() # we're doing this _before_ opening a connection >>> cn = db.open(transaction_manager=tm) >>> rt = cn.root() # make a change >>> rt['d'] = 4 >>> st.sync_called False >>> tm.commit() >>> st.sync_called True >>> cn.close() >>> db.close() Zope-2.8.7-final/lib/python/ZODB/tests/testUtils.py0000664000076400007640000000736610435502423021113 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test the routines to convert between long and 64-bit strings""" import random import unittest from persistent import Persistent NUM = 100 from ZODB.utils import U64, p64, u64 class TestUtils(unittest.TestCase): small = [random.randrange(1, 1L<<32, int=long) for i in range(NUM)] large = [random.randrange(1L<<32, 1L<<64, int=long) for i in range(NUM)] all = small + large def checkLongToStringToLong(self): for num in self.all: s = p64(num) n = U64(s) self.assertEquals(num, n, "U64() failed") n2 = u64(s) self.assertEquals(num, n2, "u64() failed") def checkKnownConstants(self): self.assertEquals("\000\000\000\000\000\000\000\001", p64(1)) self.assertEquals("\000\000\000\001\000\000\000\000", p64(1L<<32)) self.assertEquals(u64("\000\000\000\000\000\000\000\001"), 1) self.assertEquals(U64("\000\000\000\000\000\000\000\001"), 1) self.assertEquals(u64("\000\000\000\001\000\000\000\000"), 1L<<32) self.assertEquals(U64("\000\000\000\001\000\000\000\000"), 1L<<32) def checkPersistentIdHandlesDescriptor(self): from ZODB.serialize import ObjectWriter class P(Persistent): pass writer = ObjectWriter(None) self.assertEqual(writer.persistent_id(P), None) # It's hard to know where to put this test. We're checking that the # ConflictError constructor uses utils.py's get_pickle_metadata() to # deduce the class path from a pickle, instead of actually loading # the pickle (and so also trying to import application module and # class objects, which isn't a good idea on a ZEO server when avoidable). def checkConflictErrorDoesntImport(self): from ZODB.serialize import ObjectWriter from ZODB.POSException import ConflictError from ZODB.tests.MinPO import MinPO import cPickle as pickle obj = MinPO() data = ObjectWriter().serialize(obj) # The pickle contains a GLOBAL ('c') opcode resolving to MinPO's # module and class. self.assert_('cZODB.tests.MinPO\nMinPO\n' in data) # Fiddle the pickle so it points to something "impossible" instead. data = data.replace('cZODB.tests.MinPO\nMinPO\n', 'cpath.that.does.not.exist\nlikewise.the.class\n') # Pickle can't resolve that GLOBAL opcode -- gets ImportError. self.assertRaises(ImportError, pickle.loads, data) # Verify that building ConflictError doesn't get ImportError. try: raise ConflictError(object=obj, data=data) except ConflictError, detail: # And verify that the msg names the impossible path. self.assert_('path.that.does.not.exist.likewise.the.class' in str(detail)) else: self.fail("expected ConflictError, but no exception raised") def test_suite(): return unittest.makeSuite(TestUtils, 'check') if __name__ == "__main__": loader = unittest.TestLoader() loader.testMethodPrefix = "check" unittest.main(testLoader=loader) Zope-2.8.7-final/lib/python/ZODB/tests/testDB.py0000664000076400007640000001043310435502423020265 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import os import time import unittest import warnings import transaction import ZODB import ZODB.FileStorage from ZODB.tests.MinPO import MinPO # Return total number of connections across all pools in a db._pools. def nconn(pools): return sum([len(pool.all) for pool in pools.values()]) class DBTests(unittest.TestCase): def setUp(self): self.__path = os.path.abspath('test.fs') store = ZODB.FileStorage.FileStorage(self.__path) self.db = ZODB.DB(store) def tearDown(self): self.db.close() for s in ('', '.index', '.lock', '.tmp'): if os.path.exists(self.__path+s): os.remove(self.__path+s) def dowork(self, version=''): c = self.db.open(version) r = c.root() o = r[time.time()] = MinPO(0) transaction.commit() for i in range(25): o.value = MinPO(i) transaction.commit() o = o.value c.close() # make sure the basic methods are callable def testSets(self): # test set methods that have non-trivial implementations warnings.filterwarnings("error", category=DeprecationWarning) self.assertRaises(DeprecationWarning, self.db.setCacheDeactivateAfter, 12) self.assertRaises(DeprecationWarning, self.db.setVersionCacheDeactivateAfter, 12) # Obscure: There is no API call for removing the warning we just # added, but filters appears to be a public variable. del warnings.filters[0] self.db.setCacheSize(15) self.db.setVersionCacheSize(15) def test_removeVersionPool(self): # Test that we can remove a version pool # This is white box because we check some internal data structures self.dowork() self.dowork('v2') c1 = self.db.open('v1') c1.close() # return to pool c12 = self.db.open('v1') c12.close() # return to pool self.assert_(c1 is c12) # should be same pools = self.db._pools self.assertEqual(len(pools), 3) self.assertEqual(nconn(pools), 3) self.db.removeVersionPool('v1') self.assertEqual(len(pools), 2) self.assertEqual(nconn(pools), 2) c12 = self.db.open('v1') c12.close() # return to pool self.assert_(c1 is not c12) # should be different self.assertEqual(len(pools), 3) self.assertEqual(nconn(pools), 3) def _test_for_leak(self): self.dowork() self.dowork('v2') while 1: c1 = self.db.open('v1') self.db.removeVersionPool('v1') c1.close() # return to pool def test_removeVersionPool_while_connection_open(self): # Test that we can remove a version pool # This is white box because we check some internal data structures self.dowork() self.dowork('v2') c1 = self.db.open('v1') c1.close() # return to pool c12 = self.db.open('v1') self.assert_(c1 is c12) # should be same pools = self.db._pools self.assertEqual(len(pools), 3) self.assertEqual(nconn(pools), 3) self.db.removeVersionPool('v1') self.assertEqual(len(pools), 2) self.assertEqual(nconn(pools), 2) c12.close() # should leave pools alone self.assertEqual(len(pools), 2) self.assertEqual(nconn(pools), 2) c12 = self.db.open('v1') c12.close() # return to pool self.assert_(c1 is not c12) # should be different self.assertEqual(len(pools), 3) self.assertEqual(nconn(pools), 3) def test_suite(): return unittest.makeSuite(DBTests) Zope-2.8.7-final/lib/python/ZODB/tests/dbopen.txt0000664000076400007640000001656310435502423020550 0ustar ajungajung===================== Connection Management ===================== Here we exercise the connection management done by the DB class. >>> from ZODB import DB >>> from ZODB.MappingStorage import MappingStorage as Storage Capturing log messages from DB is important for some of the examples: >>> from zope.testing.loggingsupport import InstalledHandler >>> handler = InstalledHandler('ZODB.DB') Create a storage, and wrap it in a DB wrapper: >>> st = Storage() >>> db = DB(st) By default, we can open 7 connections without any log messages: >>> conns = [db.open() for dummy in range(7)] >>> handler.records [] Open one more, and we get a warning: >>> conns.append(db.open()) >>> len(handler.records) 1 >>> msg = handler.records[0] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB WARNING DB.open() has 8 open connections with a pool_size of 7 Open 6 more, and we get 6 more warnings: >>> conns.extend([db.open() for dummy in range(6)]) >>> len(conns) 14 >>> len(handler.records) 7 >>> msg = handler.records[-1] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB WARNING DB.open() has 14 open connections with a pool_size of 7 Add another, so that it's more than twice the default, and the level rises to critical: >>> conns.append(db.open()) >>> len(conns) 15 >>> len(handler.records) 8 >>> msg = handler.records[-1] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB CRITICAL DB.open() has 15 open connections with a pool_size of 7 While it's boring, it's important to verify that the same relationships hold if the default pool size is overridden. >>> handler.clear() >>> st.close() >>> st = Storage() >>> PS = 2 # smaller pool size >>> db = DB(st, pool_size=PS) >>> conns = [db.open() for dummy in range(PS)] >>> handler.records [] A warning for opening one more: >>> conns.append(db.open()) >>> len(handler.records) 1 >>> msg = handler.records[0] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB WARNING DB.open() has 3 open connections with a pool_size of 2 More warnings through 4 connections: >>> conns.extend([db.open() for dummy in range(PS-1)]) >>> len(conns) 4 >>> len(handler.records) 2 >>> msg = handler.records[-1] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB WARNING DB.open() has 4 open connections with a pool_size of 2 And critical for going beyond that: >>> conns.append(db.open()) >>> len(conns) 5 >>> len(handler.records) 3 >>> msg = handler.records[-1] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB CRITICAL DB.open() has 5 open connections with a pool_size of 2 We can change the pool size on the fly: >>> handler.clear() >>> db.setPoolSize(6) >>> conns.append(db.open()) >>> handler.records # no log msg -- the pool is bigger now [] >>> conns.append(db.open()) # but one more and there's a warning again >>> len(handler.records) 1 >>> msg = handler.records[0] >>> print msg.name, msg.levelname, msg.getMessage() ZODB.DB WARNING DB.open() has 7 open connections with a pool_size of 6 Enough of that. >>> handler.clear() >>> st.close() More interesting is the stack-like nature of connection reuse. So long as we keep opening new connections, and keep them alive, all connections returned are distinct: >>> st = Storage() >>> db = DB(st) >>> c1 = db.open() >>> c2 = db.open() >>> c3 = db.open() >>> c1 is c2 or c1 is c3 or c2 is c3 False Let's put some markers on the connections, so we can identify these specific objects later: >>> c1.MARKER = 'c1' >>> c2.MARKER = 'c2' >>> c3.MARKER = 'c3' Now explicitly close c1 and c2: >>> c1.close() >>> c2.close() Reaching into the internals, we can see that db's connection pool now has two connections available for reuse, and knows about three connections in all: >>> pool = db._pools[''] >>> len(pool.available) 2 >>> len(pool.all) 3 Since we closed c2 last, it's at the top of the available stack, so will be reused by the next open(): >>> c1 = db.open() >>> c1.MARKER 'c2' >>> len(pool.available), len(pool.all) (1, 3) >>> c3.close() # now the stack has c3 on top, then c1 >>> c2 = db.open() >>> c2.MARKER 'c3' >>> len(pool.available), len(pool.all) (1, 3) >>> c3 = db.open() >>> c3.MARKER 'c1' >>> len(pool.available), len(pool.all) (0, 3) What about the 3 in pool.all? We've seen that closing connections doesn't reduce pool.all, and it would be bad if DB kept connections alive forever. In fact pool.all is a "weak set" of connections -- it holds weak references to connections. That alone doesn't keep connection objects alive. The weak set allows DB's statistics methods to return info about connections that are still alive. >>> len(db.cacheDetailSize()) # one result for each connection's cache 3 If a connection object is abandoned (it becomes unreachable), then it will vanish from pool.all automatically. However, connections are involved in cycles, so exactly when a connection vanishes from pool.all isn't predictable. It can be forced by running gc.collect(): >>> import gc >>> dummy = gc.collect() >>> len(pool.all) 3 >>> c3 = None >>> dummy = gc.collect() # removes c3 from pool.all >>> len(pool.all) 2 Note that c3 is really gone; in particular it didn't get added back to the stack of available connections by magic: >>> len(pool.available) 0 Nothing in that last block should have logged any msgs: >>> handler.records [] If "too many" connections are open, then closing one may kick an older closed one out of the available connection stack. >>> st.close() >>> st = Storage() >>> db = DB(st, pool_size=3) >>> conns = [db.open() for dummy in range(6)] >>> len(handler.records) # 3 warnings for the "excess" connections 3 >>> pool = db._pools[''] >>> len(pool.available), len(pool.all) (0, 6) Let's mark them: >>> for i, c in enumerate(conns): ... c.MARKER = i Closing connections adds them to the stack: >>> for i in range(3): ... conns[i].close() >>> len(pool.available), len(pool.all) (3, 6) >>> del conns[:3] # leave the ones with MARKERs 3, 4 and 5 Closing another one will purge the one with MARKER 0 from the stack (since it was the first added to the stack): >>> [c.MARKER for c in pool.available] [0, 1, 2] >>> conns[0].close() # MARKER 3 >>> len(pool.available), len(pool.all) (3, 5) >>> [c.MARKER for c in pool.available] [1, 2, 3] Similarly for the other two: >>> conns[1].close(); conns[2].close() >>> len(pool.available), len(pool.all) (3, 3) >>> [c.MARKER for c in pool.available] [3, 4, 5] Reducing the pool size may also purge the oldest closed connections: >>> db.setPoolSize(2) # gets rid of MARKER 3 >>> len(pool.available), len(pool.all) (2, 2) >>> [c.MARKER for c in pool.available] [4, 5] Since MARKER 5 is still the last one added to the stack, it will be the first popped: >>> c1 = db.open(); c2 = db.open() >>> c1.MARKER, c2.MARKER (5, 4) >>> len(pool.available), len(pool.all) (0, 2) Clean up. >>> st.close() >>> handler.uninstall() Zope-2.8.7-final/lib/python/ZODB/tests/PackableStorage.py0000664000076400007640000006271410435502423022140 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Run some tests relevant for storages that support pack().""" try: import cPickle pickle = cPickle #import cPickle as pickle except ImportError: import pickle try: from cStringIO import StringIO except ImportError: from StringIO import StringIO import time from persistent import Persistent from persistent.mapping import PersistentMapping import transaction from ZODB import DB from ZODB.serialize import referencesf from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import snooze from ZODB.POSException import ConflictError, StorageError from ZODB.tests.MTStorage import TestThread ZERO = '\0'*8 # This class is for the root object. It must not contain a getoid() method # (really, attribute). The persistent pickling machinery -- in the dumps() # function below -- will pickle Root objects as normal, but any attributes # which reference persistent Object instances will get pickled as persistent # ids, not as the object's state. This makes the referencesf stuff work, # because it pickle sniffs for persistent ids (so we have to get those # persistent ids into the root object's pickle). class Root: pass # This is the persistent Object class. Because it has a getoid() method, the # persistent pickling machinery -- in the dumps() function below -- will # pickle the oid string instead of the object's actual state. Yee haw, this # stuff is deep. ;) class Object: def __init__(self, oid): self._oid = oid def getoid(self): return self._oid class C(Persistent): pass # Here's where all the magic occurs. Sadly, the pickle module is a bit # underdocumented, but here's what happens: by setting the persistent_id # attribute to getpersid() on the pickler, that function gets called for every # object being pickled. By returning None when the object has no getoid # attribute, it signals pickle to serialize the object as normal. That's how # the Root instance gets pickled correctly. But, if the object has a getoid # attribute, then by returning that method's value, we tell pickle to # serialize the persistent id of the object instead of the object's state. # That sets the pickle up for proper sniffing by the referencesf machinery. # Fun, huh? def dumps(obj): def getpersid(obj): if hasattr(obj, 'getoid'): return obj.getoid() return None s = StringIO() p = pickle.Pickler(s) p.persistent_id = getpersid p.dump(obj) return s.getvalue() class PackableStorageBase: # We keep a cache of object ids to instances so that the unpickler can # easily return any persistent object. _cache = {} def _newobj(self): # This is a convenience method to create a new persistent Object # instance. It asks the storage for a new object id, creates the # instance with the given oid, populates the cache and returns the # object. oid = self._storage.new_oid() obj = Object(oid) self._cache[obj.getoid()] = obj return obj def _makeloader(self): # This is the other side of the persistent pickling magic. We need a # custom unpickler to mirror our custom pickler above. By setting the # persistent_load function of the unpickler to self._cache.get(), # whenever a persistent id is unpickled, it will actually return the # Object instance out of the cache. As far as returning a function # with an argument bound to an instance attribute method, we do it # this way because it makes the code in the tests more succinct. # # BUT! Be careful in your use of loads() vs. pickle.loads(). loads() # should only be used on the Root object's pickle since it's the only # special one. All the Object instances should use pickle.loads(). def loads(str, persfunc=self._cache.get): fp = StringIO(str) u = pickle.Unpickler(fp) u.persistent_load = persfunc return u.load() return loads def _initroot(self): try: self._storage.load(ZERO, '') except KeyError: from transaction import Transaction file = StringIO() p = cPickle.Pickler(file, 1) p.dump((PersistentMapping, None)) p.dump({'_container': {}}) t=Transaction() t.description='initial database creation' self._storage.tpc_begin(t) self._storage.store(ZERO, None, file.getvalue(), '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) class PackableStorage(PackableStorageBase): def checkPackEmptyStorage(self): self._storage.pack(time.time(), referencesf) def checkPackTomorrow(self): self._initroot() self._storage.pack(time.time() + 10000, referencesf) def checkPackYesterday(self): self._initroot() self._storage.pack(time.time() - 10000, referencesf) def _PackWhileWriting(self, pack_now): # A storage should allow some reading and writing during # a pack. This test attempts to exercise locking code # in the storage to test that it is safe. It generates # a lot of revisions, so that pack takes a long time. db = DB(self._storage) conn = db.open() root = conn.root() for i in range(10): root[i] = MinPO(i) transaction.commit() snooze() packt = time.time() choices = range(10) for dummy in choices: for i in choices: root[i].value = MinPO(i) transaction.commit() # How many client threads should we run, and how long should we # wait for them to finish? Hard to say. Running 4 threads and # waiting 30 seconds too often left a thread still alive on Tim's # Win98SE box, during ZEO flavors of this test. Those tend to # run one thread at a time to completion, and take about 10 seconds # per thread. There doesn't appear to be a compelling reason to # run that many threads. Running 3 threads and waiting up to a # minute seems to work well in practice. The ZEO tests normally # finish faster than that, and the non-ZEO tests very much faster # than that. NUM_LOOP_TRIP = 50 timer = ElapsedTimer(time.time()) threads = [ClientThread(db, choices, NUM_LOOP_TRIP, timer, i) for i in range(3)] for t in threads: t.start() if pack_now: db.pack(time.time()) else: db.pack(packt) for t in threads: t.join(60) liveness = [t.isAlive() for t in threads] if True in liveness: # They should have finished by now. print 'Liveness:', liveness # Combine the outcomes, and sort by start time. outcomes = [] for t in threads: outcomes.extend(t.outcomes) # each outcome list has as many of these as a loop trip got thru: # thread_id # elapsed millis at loop top # elapsed millis at attempt to assign to self.root[index] # index into self.root getting replaced # elapsed millis when outcome known # 'OK' or 'Conflict' # True if we got beyond this line, False if it raised an # exception (one possible Conflict cause): # self.root[index].value = MinPO(j) def cmp_by_time(a, b): return cmp((a[1], a[0]), (b[1], b[0])) outcomes.sort(cmp_by_time) counts = [0] * 4 for outcome in outcomes: n = len(outcome) assert n >= 2 tid = outcome[0] print 'tid:%d top:%5d' % (tid, outcome[1]), if n > 2: print 'commit:%5d' % outcome[2], if n > 3: print 'index:%2d' % outcome[3], if n > 4: print 'known:%5d' % outcome[4], if n > 5: print '%8s' % outcome[5], if n > 6: print 'assigned:%5s' % outcome[6], counts[tid] += 1 if counts[tid] == NUM_LOOP_TRIP: print 'thread %d done' % tid, print self.fail('a thread is still alive') # Iterate over the storage to make sure it's sane, but not every # storage supports iterators. if not hasattr(self._storage, "iterator"): return it = self._storage.iterator() for txn in it: for data in txn: pass it.close() def checkPackWhileWriting(self): self._PackWhileWriting(pack_now=False) def checkPackNowWhileWriting(self): self._PackWhileWriting(pack_now=True) def checkPackLotsWhileWriting(self): # This is like the other pack-while-writing tests, except it packs # repeatedly until the client thread is done. At the time it was # introduced, it reliably provoked # CorruptedError: ... transaction with checkpoint flag set # in the ZEO flavor of the FileStorage tests. db = DB(self._storage) conn = db.open() root = conn.root() choices = range(10) for i in choices: root[i] = MinPO(i) transaction.commit() snooze() packt = time.time() for dummy in choices: for i in choices: root[i].value = MinPO(i) transaction.commit() NUM_LOOP_TRIP = 100 timer = ElapsedTimer(time.time()) thread = ClientThread(db, choices, NUM_LOOP_TRIP, timer, 0) thread.start() while thread.isAlive(): db.pack(packt) snooze() packt = time.time() thread.join() # Iterate over the storage to make sure it's sane. if not hasattr(self._storage, "iterator"): return it = self._storage.iterator() for txn in it: for data in txn: pass it.close() class PackableUndoStorage(PackableStorageBase): def checkPackAllRevisions(self): self._initroot() eq = self.assertEqual raises = self.assertRaises # Create a `persistent' object obj = self._newobj() oid = obj.getoid() obj.value = 1 # Commit three different revisions revid1 = self._dostoreNP(oid, data=pickle.dumps(obj)) obj.value = 2 revid2 = self._dostoreNP(oid, revid=revid1, data=pickle.dumps(obj)) obj.value = 3 revid3 = self._dostoreNP(oid, revid=revid2, data=pickle.dumps(obj)) # Now make sure all three revisions can be extracted data = self._storage.loadSerial(oid, revid1) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 1) data = self._storage.loadSerial(oid, revid2) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 2) data = self._storage.loadSerial(oid, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 3) # Now pack all transactions; need to sleep a second to make # sure that the pack time is greater than the last commit time. now = packtime = time.time() while packtime <= now: packtime = time.time() self._storage.pack(packtime, referencesf) # All revisions of the object should be gone, since there is no # reference from the root object to this object. raises(KeyError, self._storage.loadSerial, oid, revid1) raises(KeyError, self._storage.loadSerial, oid, revid2) raises(KeyError, self._storage.loadSerial, oid, revid3) def checkPackJustOldRevisions(self): eq = self.assertEqual raises = self.assertRaises loads = self._makeloader() # Create a root object. This can't be an instance of Object, # otherwise the pickling machinery will serialize it as a persistent # id and not as an object that contains references (persistent ids) to # other objects. root = Root() # Create a persistent object, with some initial state obj = self._newobj() oid = obj.getoid() # Link the root object to the persistent object, in order to keep the # persistent object alive. Store the root object. root.obj = obj root.value = 0 revid0 = self._dostoreNP(ZERO, data=dumps(root)) # Make sure the root can be retrieved data, revid = self._storage.load(ZERO, '') eq(revid, revid0) eq(loads(data).value, 0) # Commit three different revisions of the other object obj.value = 1 revid1 = self._dostoreNP(oid, data=pickle.dumps(obj)) obj.value = 2 revid2 = self._dostoreNP(oid, revid=revid1, data=pickle.dumps(obj)) obj.value = 3 revid3 = self._dostoreNP(oid, revid=revid2, data=pickle.dumps(obj)) # Now make sure all three revisions can be extracted data = self._storage.loadSerial(oid, revid1) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 1) data = self._storage.loadSerial(oid, revid2) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 2) data = self._storage.loadSerial(oid, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 3) # Now pack just revisions 1 and 2. The object's current revision # should stay alive because it's pointed to by the root. now = packtime = time.time() while packtime <= now: packtime = time.time() self._storage.pack(packtime, referencesf) # Make sure the revisions are gone, but that object zero and revision # 3 are still there and correct data, revid = self._storage.load(ZERO, '') eq(revid, revid0) eq(loads(data).value, 0) raises(KeyError, self._storage.loadSerial, oid, revid1) raises(KeyError, self._storage.loadSerial, oid, revid2) data = self._storage.loadSerial(oid, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 3) data, revid = self._storage.load(oid, '') eq(revid, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid) eq(pobj.value, 3) def checkPackOnlyOneObject(self): eq = self.assertEqual raises = self.assertRaises loads = self._makeloader() # Create a root object. This can't be an instance of Object, # otherwise the pickling machinery will serialize it as a persistent # id and not as an object that contains references (persistent ids) to # other objects. root = Root() # Create a persistent object, with some initial state obj1 = self._newobj() oid1 = obj1.getoid() # Create another persistent object, with some initial state. Make # sure it's oid is greater than the first object's oid. obj2 = self._newobj() oid2 = obj2.getoid() self.failUnless(oid2 > oid1) # Link the root object to the persistent objects, in order to keep # them alive. Store the root object. root.obj1 = obj1 root.obj2 = obj2 root.value = 0 revid0 = self._dostoreNP(ZERO, data=dumps(root)) # Make sure the root can be retrieved data, revid = self._storage.load(ZERO, '') eq(revid, revid0) eq(loads(data).value, 0) # Commit three different revisions of the first object obj1.value = 1 revid1 = self._dostoreNP(oid1, data=pickle.dumps(obj1)) obj1.value = 2 revid2 = self._dostoreNP(oid1, revid=revid1, data=pickle.dumps(obj1)) obj1.value = 3 revid3 = self._dostoreNP(oid1, revid=revid2, data=pickle.dumps(obj1)) # Now make sure all three revisions can be extracted data = self._storage.loadSerial(oid1, revid1) pobj = pickle.loads(data) eq(pobj.getoid(), oid1) eq(pobj.value, 1) data = self._storage.loadSerial(oid1, revid2) pobj = pickle.loads(data) eq(pobj.getoid(), oid1) eq(pobj.value, 2) data = self._storage.loadSerial(oid1, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid1) eq(pobj.value, 3) # Now commit a revision of the second object obj2.value = 11 revid4 = self._dostoreNP(oid2, data=pickle.dumps(obj2)) # And make sure the revision can be extracted data = self._storage.loadSerial(oid2, revid4) pobj = pickle.loads(data) eq(pobj.getoid(), oid2) eq(pobj.value, 11) # Now pack just revisions 1 and 2 of object1. Object1's current # revision should stay alive because it's pointed to by the root, as # should Object2's current revision. now = packtime = time.time() while packtime <= now: packtime = time.time() self._storage.pack(packtime, referencesf) # Make sure the revisions are gone, but that object zero, object2, and # revision 3 of object1 are still there and correct. data, revid = self._storage.load(ZERO, '') eq(revid, revid0) eq(loads(data).value, 0) raises(KeyError, self._storage.loadSerial, oid1, revid1) raises(KeyError, self._storage.loadSerial, oid1, revid2) data = self._storage.loadSerial(oid1, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid1) eq(pobj.value, 3) data, revid = self._storage.load(oid1, '') eq(revid, revid3) pobj = pickle.loads(data) eq(pobj.getoid(), oid1) eq(pobj.value, 3) data, revid = self._storage.load(oid2, '') eq(revid, revid4) eq(loads(data).value, 11) data = self._storage.loadSerial(oid2, revid4) pobj = pickle.loads(data) eq(pobj.getoid(), oid2) eq(pobj.value, 11) def checkPackUnlinkedFromRoot(self): eq = self.assertEqual db = DB(self._storage) conn = db.open() root = conn.root() txn = transaction.get() txn.note('root') txn.commit() now = packtime = time.time() while packtime <= now: packtime = time.time() obj = C() obj.value = 7 root['obj'] = obj txn = transaction.get() txn.note('root -> o1') txn.commit() del root['obj'] txn = transaction.get() txn.note('root -x-> o1') txn.commit() self._storage.pack(packtime, referencesf) log = self._storage.undoLog() tid = log[0]['id'] db.undo(tid) txn = transaction.get() txn.note('undo root -x-> o1') txn.commit() conn.sync() eq(root['obj'].value, 7) def checkRedundantPack(self): # It is an error to perform a pack with a packtime earlier # than a previous packtime. The storage can't do a full # traversal as of the packtime, because the previous pack may # have removed revisions necessary for a full traversal. # It should be simple to test that a storage error is raised, # but this test case goes to the trouble of constructing a # scenario that would lose data if the earlier packtime was # honored. self._initroot() db = DB(self._storage) conn = db.open() root = conn.root() root["d"] = d = PersistentMapping() transaction.commit() snooze() obj = d["obj"] = C() obj.value = 1 transaction.commit() snooze() packt1 = time.time() lost_oid = obj._p_oid obj = d["anotherobj"] = C() obj.value = 2 transaction.commit() snooze() packt2 = time.time() db.pack(packt2) # BDBStorage allows the second pack, but doesn't lose data. try: db.pack(packt1) except StorageError: pass # This object would be removed by the second pack, even though # it is reachable. self._storage.load(lost_oid, "") def checkPackUndoLog(self): self._initroot() # Create a `persistent' object obj = self._newobj() oid = obj.getoid() obj.value = 1 # Commit two different revisions revid1 = self._dostoreNP(oid, data=pickle.dumps(obj)) obj.value = 2 snooze() packtime = time.time() snooze() self._dostoreNP(oid, revid=revid1, data=pickle.dumps(obj)) # Now pack the first transaction self.assertEqual(3, len(self._storage.undoLog())) self._storage.pack(packtime, referencesf) # The undo log contains only the most resent transaction self.assertEqual(1,len(self._storage.undoLog())) def dont_checkPackUndoLogUndoable(self): # A disabled test. I wanted to test that the content of the # undo log was consistent, but every storage appears to # include something slightly different. If the result of this # method is only used to fill a GUI then this difference # doesnt matter. Perhaps re-enable this test once we agree # what should be asserted. self._initroot() # Create two `persistent' object obj1 = self._newobj() oid1 = obj1.getoid() obj1.value = 1 obj2 = self._newobj() oid2 = obj2.getoid() obj2.value = 2 # Commit the first revision of each of them revid11 = self._dostoreNP(oid1, data=pickle.dumps(obj1), description="1-1") revid22 = self._dostoreNP(oid2, data=pickle.dumps(obj2), description="2-2") # remember the time. everything above here will be packed away snooze() packtime = time.time() snooze() # Commit two revisions of the first object obj1.value = 3 revid13 = self._dostoreNP(oid1, revid=revid11, data=pickle.dumps(obj1), description="1-3") obj1.value = 4 self._dostoreNP(oid1, revid=revid13, data=pickle.dumps(obj1), description="1-4") # Commit one revision of the second object obj2.value = 5 self._dostoreNP(oid2, revid=revid22, data=pickle.dumps(obj2), description="2-5") # Now pack self.assertEqual(6,len(self._storage.undoLog())) print '\ninitial undoLog was' for r in self._storage.undoLog(): print r self._storage.pack(packtime, referencesf) # The undo log contains only two undoable transaction. print '\nafter packing undoLog was' for r in self._storage.undoLog(): print r # what can we assert about that? # A number of these threads are kicked off by _PackWhileWriting(). Their # purpose is to abuse the database passed to the constructor with lots of # random write activity while the main thread is packing it. class ClientThread(TestThread): def __init__(self, db, choices, loop_trip, timer, thread_id): TestThread.__init__(self) self.root = db.open().root() self.choices = choices self.loop_trip = loop_trip self.millis = timer.elapsed_millis self.thread_id = thread_id # list of lists; each list has as many of these as a loop trip # got thru: # thread_id # elapsed millis at loop top # elapsed millis at attempt # index into self.root getting replaced # elapsed millis when outcome known # 'OK' or 'Conflict' # True if we got beyond this line, False if it raised an exception: # self.root[index].value = MinPO(j) self.outcomes = [] def runtest(self): from random import choice for j in range(self.loop_trip): assign_worked = False alist = [self.thread_id, self.millis()] self.outcomes.append(alist) try: index = choice(self.choices) alist.extend([self.millis(), index]) self.root[index].value = MinPO(j) assign_worked = True transaction.commit() alist.append(self.millis()) alist.append('OK') except ConflictError: alist.append(self.millis()) alist.append('Conflict') transaction.abort() alist.append(assign_worked) class ElapsedTimer: def __init__(self, start_time): self.start_time = start_time def elapsed_millis(self): return int((time.time() - self.start_time) * 1000) Zope-2.8.7-final/lib/python/ZODB/tests/test_cache.py0000664000076400007640000001352610435502423021210 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test behavior of Connection plus cPickleCache.""" from zope.testing import doctest from persistent import Persistent import transaction from ZODB.config import databaseFromString class RecalcitrantObject(Persistent): """A Persistent object that will not become a ghost.""" deactivations = 0 def _p_deactivate(self): self.__class__.deactivations += 1 def init(cls): cls.deactivations = 0 init = classmethod(init) class RegularObject(Persistent): deactivations = 0 invalidations = 0 def _p_deactivate(self): self.__class__.deactivations += 1 super(RegularObject, self)._p_deactivate() def _p_invalidate(self): self.__class__.invalidations += 1 super(RegularObject, self)._p_invalidate() def init(cls): cls.deactivations = 0 cls.invalidations = 0 init = classmethod(init) class CacheTests: def test_cache(self): r"""Test basic cache methods. Let's start with a clean transaction >>> transaction.abort() >>> RegularObject.init() >>> db = databaseFromString("\n" ... "cache-size 4\n" ... "\n" ... "") >>> cn = db.open() >>> r = cn.root() >>> L = [] >>> for i in range(5): ... o = RegularObject() ... L.append(o) ... r[i] = o >>> transaction.commit() After committing a transaction and calling cacheGC(), there should be cache-size (4) objects in the cache. One of the RegularObjects was deactivated. >>> cn._cache.ringlen() 4 >>> RegularObject.deactivations 1 If we explicitly activate the objects again, the ringlen should go back up to 5. >>> for o in L: ... o._p_activate() >>> cn._cache.ringlen() 5 >>> cn.cacheGC() >>> cn._cache.ringlen() 4 >>> RegularObject.deactivations 2 >>> cn.cacheMinimize() >>> cn._cache.ringlen() 0 >>> RegularObject.deactivations 6 If we activate all the objects again and mark one as modified, then the one object should not be deactivated even by a minimize. >>> for o in L: ... o._p_activate() >>> o.attr = 1 >>> cn._cache.ringlen() 5 >>> cn.cacheMinimize() >>> cn._cache.ringlen() 1 >>> RegularObject.deactivations 10 Clean up >>> transaction.abort() """ def test_cache_gc_recalcitrant(self): r"""Test that a cacheGC() call will return. It's possible for a particular object to ignore the _p_deactivate() call. We want to check several things in this case. The cache should called the real _p_deactivate() method not the one provided by Persistent. The cacheGC() call should also return when it's looked at each item, regardless of whether it became a ghost. >>> RecalcitrantObject.init() >>> db = databaseFromString("\n" ... "cache-size 4\n" ... "\n" ... "") >>> cn = db.open() >>> r = cn.root() >>> L = [] >>> for i in range(5): ... o = RecalcitrantObject() ... L.append(o) ... r[i] = o >>> transaction.commit() >>> [o._p_state for o in L] [0, 0, 0, 0, 0] The Connection calls cacheGC() after it commits a transaction. Since the cache will now have more objects that it's target size, it will call _p_deactivate() on each RecalcitrantObject. >>> RecalcitrantObject.deactivations 5 >>> [o._p_state for o in L] [0, 0, 0, 0, 0] An explicit call to cacheGC() has the same effect. >>> cn.cacheGC() >>> RecalcitrantObject.deactivations 10 >>> [o._p_state for o in L] [0, 0, 0, 0, 0] """ def test_cache_on_abort(self): r"""Test that the cache handles transaction abort correctly. >>> RegularObject.init() >>> db = databaseFromString("\n" ... "cache-size 4\n" ... "\n" ... "") >>> cn = db.open() >>> r = cn.root() >>> L = [] >>> for i in range(5): ... o = RegularObject() ... L.append(o) ... r[i] = o >>> transaction.commit() >>> RegularObject.deactivations 1 Modify three of the objects and verify that they are deactivated when the transaction aborts. >>> for i in range(0, 5, 2): ... L[i].attr = i >>> [L[i]._p_state for i in range(0, 5, 2)] [1, 1, 1] >>> cn._cache.ringlen() 5 >>> transaction.abort() >>> cn._cache.ringlen() 2 >>> RegularObject.deactivations 4 """ def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/ZODB/tests/testConnection.py0000664000076400007640000005127710435502423022112 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Unit tests for the Connection class.""" from zope.testing import doctest import unittest import warnings from persistent import Persistent import transaction from ZODB.config import databaseFromString from ZODB.utils import p64, u64 from ZODB.tests.warnhook import WarningsHook from zope.interface.verify import verifyObject class ConnectionDotAdd(unittest.TestCase): def setUp(self): from ZODB.Connection import Connection self.datamgr = Connection() self.db = StubDatabase() self.datamgr._setDB(self.db) self.transaction = StubTransaction() def tearDown(self): transaction.abort() def check_add(self): from ZODB.POSException import InvalidObjectReference obj = StubObject() self.assert_(obj._p_oid is None) self.assert_(obj._p_jar is None) self.datamgr.add(obj) self.assert_(obj._p_oid is not None) self.assert_(obj._p_jar is self.datamgr) self.assert_(self.datamgr.get(obj._p_oid) is obj) # Only first-class persistent objects may be added. self.assertRaises(TypeError, self.datamgr.add, object()) # Adding to the same connection does not fail. Object keeps the # same oid. oid = obj._p_oid self.datamgr.add(obj) self.assertEqual(obj._p_oid, oid) # Cannot add an object from a different connection. obj2 = StubObject() obj2._p_jar = object() self.assertRaises(InvalidObjectReference, self.datamgr.add, obj2) def checkResetOnAbort(self): # Check that _p_oid and _p_jar are reset when a transaction is # aborted. obj = StubObject() self.datamgr.add(obj) oid = obj._p_oid self.datamgr.abort(self.transaction) self.assert_(obj._p_oid is None) self.assert_(obj._p_jar is None) self.assertRaises(KeyError, self.datamgr.get, oid) def checkResetOnTpcAbort(self): obj = StubObject() self.datamgr.add(obj) oid = obj._p_oid # Simulate an error while committing some other object. self.datamgr.tpc_begin(self.transaction) # Let's pretend something bad happens here. # Call tpc_abort, clearing everything. self.datamgr.tpc_abort(self.transaction) self.assert_(obj._p_oid is None) self.assert_(obj._p_jar is None) self.assertRaises(KeyError, self.datamgr.get, oid) def checkTpcAbortAfterCommit(self): obj = StubObject() self.datamgr.add(obj) oid = obj._p_oid self.datamgr.tpc_begin(self.transaction) self.datamgr.commit(self.transaction) # Let's pretend something bad happened here. self.datamgr.tpc_abort(self.transaction) self.assert_(obj._p_oid is None) self.assert_(obj._p_jar is None) self.assertRaises(KeyError, self.datamgr.get, oid) self.assertEquals(self.db._storage._stored, [oid]) def checkCommit(self): obj = StubObject() self.datamgr.add(obj) oid = obj._p_oid self.datamgr.tpc_begin(self.transaction) self.datamgr.commit(self.transaction) self.datamgr.tpc_finish(self.transaction) self.assert_(obj._p_oid is oid) self.assert_(obj._p_jar is self.datamgr) # This next assert_ is covered by an assert in tpc_finish. ##self.assert_(not self.datamgr._added) self.assertEquals(self.db._storage._stored, [oid]) self.assertEquals(self.db._storage._finished, [oid]) def checkModifyOnGetstate(self): member = StubObject() subobj = StubObject() subobj.member = member obj = ModifyOnGetStateObject(subobj) self.datamgr.add(obj) self.datamgr.tpc_begin(self.transaction) self.datamgr.commit(self.transaction) self.datamgr.tpc_finish(self.transaction) storage = self.db._storage self.assert_(obj._p_oid in storage._stored, "object was not stored") self.assert_(subobj._p_oid in storage._stored, "subobject was not stored") self.assert_(member._p_oid in storage._stored, "member was not stored") self.assert_(self.datamgr._added_during_commit is None) def checkUnusedAddWorks(self): # When an object is added, but not committed, it shouldn't be stored, # but also it should be an error. obj = StubObject() self.datamgr.add(obj) self.datamgr.tpc_begin(self.transaction) self.datamgr.tpc_finish(self.transaction) self.assert_(obj._p_oid not in self.datamgr._storage._stored) class UserMethodTests(unittest.TestCase): # add isn't tested here, because there are a bunch of traditional # unit tests for it. # The version tests would require a storage that supports versions # which is a bit more work. def test_root(self): r"""doctest of root() method The root() method is simple, and the tests are pretty minimal. Ensure that a new database has a root and that it is a PersistentMapping. >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> root = cn.root() >>> type(root).__name__ 'PersistentMapping' >>> root._p_oid '\x00\x00\x00\x00\x00\x00\x00\x00' >>> root._p_jar is cn True >>> db.close() """ def test_get(self): r"""doctest of get() method The get() method return the persistent object corresponding to an oid. >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> obj = cn.get(p64(0)) >>> obj._p_oid '\x00\x00\x00\x00\x00\x00\x00\x00' The object is a ghost. >>> obj._p_state -1 And multiple calls with the same oid, return the same object. >>> obj2 = cn.get(p64(0)) >>> obj is obj2 True If all references to the object are released, then a new object will be returned. The cache doesn't keep unreferenced ghosts alive. (The next object returned my still have the same id, because Python may re-use the same memory.) >>> del obj, obj2 >>> cn._cache.get(p64(0), None) If the object is unghosted, then it will stay in the cache after the last reference is released. (This is true only if there is room in the cache and the object is recently used.) >>> obj = cn.get(p64(0)) >>> obj._p_activate() >>> y = id(obj) >>> del obj >>> obj = cn.get(p64(0)) >>> id(obj) == y True >>> obj._p_state 0 A request for an object that doesn't exist will raise a KeyError. >>> cn.get(p64(1)) Traceback (most recent call last): ... KeyError: '\x00\x00\x00\x00\x00\x00\x00\x01' """ def test_close(self): r"""doctest of close() method This is a minimal test, because most of the interesting effects on closing a connection involve its interaction with the database and the transaction. >>> db = databaseFromString("\n\n") >>> cn = db.open() It's safe to close a connection multiple times. >>> cn.close() >>> cn.close() >>> cn.close() It's not possible to load or store objects once the storage is closed. >>> cn.get(p64(0)) Traceback (most recent call last): ... ConnectionStateError: The database connection is closed >>> p = Persistent() >>> cn.add(p) Traceback (most recent call last): ... ConnectionStateError: The database connection is closed """ def test_close_with_pending_changes(self): r"""doctest to ensure close() w/ pending changes complains >>> import transaction Just opening and closing is fine. >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> cn.close() Opening, making a change, committing, and closing is fine. >>> cn = db.open() >>> cn.root()['a'] = 1 >>> transaction.commit() >>> cn.close() Opening, making a change, and aborting is fine. >>> cn = db.open() >>> cn.root()['a'] = 1 >>> transaction.abort() >>> cn.close() But trying to close with a change pending complains. >>> cn = db.open() >>> cn.root()['a'] = 10 >>> cn.close() Traceback (most recent call last): ... ConnectionStateError: Cannot close a connection joined to a transaction This leaves the connection as it was, so we can still commit the change. >>> transaction.commit() >>> cn2 = db.open() >>> cn2.root()['a'] 10 >>> cn.close(); cn2.close() Bug: We weren't catching the case where the only changes pending were in a subtransaction. >>> cn = db.open() >>> cn.root()['a'] = 100 >>> transaction.commit(True) >>> cn.close() # this was succeeding Traceback (most recent call last): ... ConnectionStateError: Cannot close a connection joined to a transaction Again this leaves the connection as it was. >>> transaction.commit() >>> cn2 = db.open() >>> cn2.root()['a'] 100 >>> cn.close(); cn2.close() Make sure we can still close a connection after aborting a pending subtransaction. >>> cn = db.open() >>> cn.root()['a'] = 1000 >>> transaction.commit(True) >>> cn.root()['a'] 1000 >>> transaction.abort() >>> cn.root()['a'] 100 >>> cn.close() >>> db.close() """ def test_onCloseCallbacks(self): r"""doctest of onCloseCallback() method >>> db = databaseFromString("\n\n") >>> cn = db.open() Every function registered is called, even if it raises an exception. They are only called once. >>> L = [] >>> def f(): ... L.append("f") >>> def g(): ... L.append("g") ... return 1 / 0 >>> cn.onCloseCallback(g) >>> cn.onCloseCallback(f) >>> cn.close() >>> L ['g', 'f'] >>> del L[:] >>> cn.close() >>> L [] The implementation keeps a list of callbacks that is reset to a class variable (which is bound to None) after the connection is closed. >>> cn._Connection__onCloseCallbacks """ def test_db(self): r"""doctest of db() method >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> cn.db() is db True >>> cn.close() >>> cn.db() """ def test_isReadOnly(self): r"""doctest of isReadOnly() method >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> cn.isReadOnly() False >>> cn.close() >>> cn.isReadOnly() Traceback (most recent call last): ... ConnectionStateError: The database connection is closed An expedient way to create a read-only storage: >>> db._storage._is_read_only = True >>> cn = db.open() >>> cn.isReadOnly() True """ def test_cache(self): r"""doctest of cacheMinimize() and cacheFullSweep() methods. These tests are fairly minimal, just verifying that the methods can be called and have some effect. We need other tests that verify the cache works as intended. >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> r = cn.root() >>> cn.cacheMinimize() >>> r._p_state -1 The next couple of tests are involved because they have to cater to backwards compatibility issues. The cacheMinimize() method used to take an argument, but now ignores it. cacheFullSweep() used to do something different than cacheMinimize(), but it doesn't anymore. We want to verify that these methods do something, but all cause deprecation warnings. To do that, we need a warnings hook. >>> hook = WarningsHook() >>> hook.install() More problems in case this test is run more than once: fool the warnings module into delivering the warnings despite that they've been seen before. >>> import warnings >>> warnings.filterwarnings("always", category=DeprecationWarning) >>> r._p_activate() >>> cn.cacheMinimize(12) >>> r._p_state -1 >>> len(hook.warnings) 1 >>> message, category, filename, lineno = hook.warnings[0] >>> print message This will be removed in ZODB 3.6: cacheMinimize() dt= is ignored. >>> category.__name__ 'DeprecationWarning' >>> hook.clear() cacheFullSweep() is a doozy. It generates two deprecation warnings, one from the Connection and one from the cPickleCache. Maybe we should drop the cPickleCache warning, but it's there for now. When passed an argument, it acts like cacheGC(). When it isn't passed an argument it acts like cacheMinimize(). >>> r._p_activate() >>> cn.cacheFullSweep(12) >>> r._p_state 0 >>> len(hook.warnings) 2 >>> message, category, filename, lineno = hook.warnings[0] >>> print message This will be removed in ZODB 3.6: cacheFullSweep is deprecated. Use cacheMinimize instead. >>> category.__name__ 'DeprecationWarning' >>> message, category, filename, lineno = hook.warnings[1] >>> message 'No argument expected' >>> category.__name__ 'DeprecationWarning' We have to uninstall the hook so that other warnings don't get lost. >>> hook.uninstall() Obscure: There is no API call for removing the filter we added, but filters appears to be a public variable. >>> del warnings.filters[0] """ class InvalidationTests(unittest.TestCase): # It's harder to write serious tests, because some of the critical # correctness issues relate to concurrency. We'll have to depend # on the various concurrent updates and NZODBThreads tests to # handle these. def test_invalidate(self): r""" This test initializes the database with several persistent objects, then manually delivers invalidations and verifies that they have the expected effect. >>> db = databaseFromString("\n\n") >>> cn = db.open() >>> p1 = Persistent() >>> p2 = Persistent() >>> p3 = Persistent() >>> r = cn.root() >>> r.update(dict(p1=p1, p2=p2, p3=p3)) >>> transaction.commit() Transaction ids are 8-byte strings, just like oids; p64() will create one from an int. >>> cn.invalidate(p64(1), {p1._p_oid: 1}) >>> cn._txn_time '\x00\x00\x00\x00\x00\x00\x00\x01' >>> p1._p_oid in cn._invalidated True >>> p2._p_oid in cn._invalidated False >>> cn.invalidate(p64(10), {p2._p_oid: 1, p64(76): 1}) >>> cn._txn_time '\x00\x00\x00\x00\x00\x00\x00\x01' >>> p1._p_oid in cn._invalidated True >>> p2._p_oid in cn._invalidated True Calling invalidate() doesn't affect the object state until a transaction boundary. >>> p1._p_state 0 >>> p2._p_state 0 >>> p3._p_state 0 The sync() method will abort the current transaction and process any pending invalidations. >>> cn.sync() >>> p1._p_state -1 >>> p2._p_state -1 >>> p3._p_state 0 >>> cn._invalidated {} """ # ---- stubs class StubObject(Persistent): pass class StubTransaction: pass class ErrorOnGetstateException(Exception): pass class ErrorOnGetstateObject(Persistent): def __getstate__(self): raise ErrorOnGetstateException class ModifyOnGetStateObject(Persistent): def __init__(self, p): self._v_p = p def __getstate__(self): self._p_jar.add(self._v_p) self.p = self._v_p return Persistent.__getstate__(self) class StubStorage: """Very simple in-memory storage that does *just* enough to support tests. Only one concurrent transaction is supported. Voting is not supported. Versions are not supported. Inspect self._stored and self._finished to see how the storage has been used during a unit test. Whenever an object is stored in the store() method, its oid is appended to self._stored. When a transaction is finished, the oids that have been stored during the transaction are appended to self._finished. """ # internal _oid = 1 _transaction = None def __init__(self): # internal self._stored = [] self._finished = [] self._data = {} self._transdata = {} self._transstored = [] def new_oid(self): oid = str(self._oid) self._oid += 1 return oid def sortKey(self): return 'StubStorage sortKey' def tpc_begin(self, transaction): if transaction is None: raise TypeError('transaction may not be None') elif self._transaction is None: self._transaction = transaction elif self._transaction != transaction: raise RuntimeError( 'StubStorage uses only one transaction at a time') def tpc_abort(self, transaction): if transaction is None: raise TypeError('transaction may not be None') elif self._transaction != transaction: raise RuntimeError( 'StubStorage uses only one transaction at a time') del self._transaction self._transdata.clear() def tpc_finish(self, transaction, callback): if transaction is None: raise TypeError('transaction may not be None') elif self._transaction != transaction: raise RuntimeError( 'StubStorage uses only one transaction at a time') self._finished.extend(self._transstored) self._data.update(self._transdata) callback(transaction) del self._transaction self._transdata.clear() self._transstored = [] def load(self, oid, version): if version != '': raise TypeError('StubStorage does not support versions.') return self._data[oid] def store(self, oid, serial, p, version, transaction): if version != '': raise TypeError('StubStorage does not support versions.') if transaction is None: raise TypeError('transaction may not be None') elif self._transaction != transaction: raise RuntimeError( 'StubStorage uses only one transaction at a time') self._stored.append(oid) self._transstored.append(oid) self._transdata[oid] = (p, serial) # Explicitly returing None, as we're not pretending to be a ZEO # storage return None class TestConnectionInterface(unittest.TestCase): def test_connection_interface(self): from ZODB.interfaces import IConnection db = databaseFromString("\n\n") cn = db.open() verifyObject(IConnection, cn) class StubDatabase: def __init__(self): self._storage = StubStorage() classFactory = None database_name = 'stubdatabase' databases = {'stubdatabase': database_name} def invalidate(self, transaction, dict_with_oid_keys, connection): pass def test_suite(): s = unittest.makeSuite(ConnectionDotAdd, 'check') s.addTest(doctest.DocTestSuite()) s.addTest(unittest.makeSuite(TestConnectionInterface)) return s Zope-2.8.7-final/lib/python/ZODB/tests/VersionStorage.py0000664000076400007640000005021310435502423022052 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Run the version related tests for a storage. Any storage that supports versions should be able to pass all these tests. """ import time import transaction from transaction import Transaction from ZODB import POSException from ZODB.serialize import referencesf from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_unpickle, snooze from ZODB import DB class VersionStorage: def checkCommitVersionSerialno(self): oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(12)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(13), version="version") oids = self._commitVersion("version", "") self.assertEqual([oid], oids) data, revid3 = self._storage.load(oid, "") # use repr() to avoid getting binary data in a traceback on error self.assertNotEqual(`revid1`, `revid3`) self.assertNotEqual(`revid2`, `revid3`) def checkAbortVersionSerialno(self): oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(12)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(13), version="version") data, tid, ver = self._storage.loadEx(oid, "version") self.assertEqual(revid2, tid) self.assertEqual(zodb_unpickle(data), MinPO(13)) oids = self._abortVersion("version") self.assertEqual([oid], oids) data, revid3 = self._storage.load(oid, "") # use repr() to avoid getting binary data in a traceback on error self.assertNotEqual(revid1, revid3) self.assertNotEqual(revid2, revid3) data, tid, ver = self._storage.loadEx(oid, "") self.assertEqual(revid3, tid) self.assertEqual(zodb_unpickle(data), MinPO(12)) self.assertEqual(tid, self._storage.lastTransaction()) def checkVersionedStoreAndLoad(self): eq = self.assertEqual # Store a couple of non-version revisions of the object oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(11)) revid1 = self._dostore(oid, revid=revid, data=MinPO(12)) # And now store some new revisions in a version version = 'test-version' revid = self._dostore(oid, revid=revid1, data=MinPO(13), version=version) revid = self._dostore(oid, revid=revid, data=MinPO(14), version=version) revid2 = self._dostore(oid, revid=revid, data=MinPO(15), version=version) # Now read back the object in both the non-version and version and # make sure the values jive. data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(12)) data, vrevid = self._storage.load(oid, version) eq(zodb_unpickle(data), MinPO(15)) if hasattr(self._storage, 'getSerial'): s = self._storage.getSerial(oid) eq(s, max(revid, vrevid)) data, tid, ver = self._storage.loadEx(oid, version) eq(zodb_unpickle(data), MinPO(15)) eq(tid, revid2) data, tid, ver = self._storage.loadEx(oid, "other version") eq(zodb_unpickle(data), MinPO(12)) eq(tid, revid2) # loadSerial returns non-version data try: data = self._storage.loadSerial(oid, revid) eq(zodb_unpickle(data), MinPO(12)) data = self._storage.loadSerial(oid, revid2) eq(zodb_unpickle(data), MinPO(12)) except POSException.Unsupported: pass def checkVersionedLoadErrors(self): oid = self._storage.new_oid() version = 'test-version' revid = self._dostore(oid, data=MinPO(11)) revid = self._dostore(oid, revid=revid, data=MinPO(12), version=version) # Try to load a bogus oid self.assertRaises(KeyError, self._storage.load, self._storage.new_oid(), '') data, revid = self._storage.load(oid, 'bogus') self.assertEqual(zodb_unpickle(data), MinPO(11)) def checkVersionLock(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(11)) version = 'test-version' revid = self._dostore(oid, revid=revid, data=MinPO(12), version=version) self.assertRaises(POSException.VersionLockError, self._dostore, oid, revid=revid, data=MinPO(14), version='another-version') def checkVersionEmpty(self): # Before we store anything, these versions ought to be empty version = 'test-version' self.failUnless(self._storage.versionEmpty(version)) # Now store some objects oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(11)) revid = self._dostore(oid, revid=revid, data=MinPO(12)) revid = self._dostore(oid, revid=revid, data=MinPO(13), version=version) revid = self._dostore(oid, revid=revid, data=MinPO(14), version=version) # The blank version should not be empty # Neither should 'test-version' self.failUnless(not self._storage.versionEmpty(version)) # But this non-existant version should be empty self.failUnless(self._storage.versionEmpty('bogus')) def checkVersions(self): unless = self.failUnless # Store some objects in the non-version oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() oid3 = self._storage.new_oid() revid1 = self._dostore(oid1, data=MinPO(11)) revid2 = self._dostore(oid2, data=MinPO(12)) revid3 = self._dostore(oid3, data=MinPO(13)) # Now create some new versions revid1 = self._dostore(oid1, revid=revid1, data=MinPO(14), version='one') revid2 = self._dostore(oid2, revid=revid2, data=MinPO(15), version='two') revid3 = self._dostore(oid3, revid=revid3, data=MinPO(16), version='three') # Ask for the versions versions = self._storage.versions() unless('one' in versions) unless('two' in versions) unless('three' in versions) # Now flex the `max' argument versions = self._storage.versions(1) self.assertEqual(len(versions), 1) unless('one' in versions or 'two' in versions or 'three' in versions) def _setup_version(self, version='test-version'): # Store some revisions in the non-version oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(49)) revid = self._dostore(oid, revid=revid, data=MinPO(50)) revid = self._dostore(oid, revid=revid, data=MinPO(51)) # Now do some stores in a version revid = self._dostore(oid, revid=revid, data=MinPO(52), version=version) revid = self._dostore(oid, revid=revid, data=MinPO(53), version=version) revid = self._dostore(oid, revid=revid, data=MinPO(54), version=version) return oid, version def checkAbortVersion(self): eq = self.assertEqual oid, version = self._setup_version() # Not sure I can write a test for getSerial() in the # presence of aborted versions, because FileStorage and # Berkeley storage give a different answer. I think Berkeley # is right and FS is wrong. oids = self._abortVersion(version) eq(len(oids), 1) eq(oids[0], oid) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(51)) def checkAbortVersionNonCurrent(self): # Make sure the non-current serial number is correctly # after a version is aborted. oid, version = self._setup_version() self._abortVersion(version) data, tid, ver = self._storage.loadEx(oid, "") # write a new revision of oid so that the aborted-version txn # is not current self._dostore(oid, revid=tid, data=MinPO(17)) ltid = self._storage.lastTransaction() ncdata, ncstart, end = self._storage.loadBefore(oid, ltid) self.assertEqual(data, ncdata) self.assertEqual(tid, ncstart) def checkAbortVersionErrors(self): eq = self.assertEqual oid, version = self._setup_version() # Now abort a bogus version t = Transaction() self._storage.tpc_begin(t) # And try to abort the empty version if (hasattr(self._storage, 'supportsTransactionalUndo') and self._storage.supportsTransactionalUndo()): self.assertRaises(POSException.VersionError, self._storage.abortVersion, '', t) # But now we really try to abort the version tid, oids = self._storage.abortVersion(version, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) eq(len(oids), 1) eq(oids[0], oid) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(51)) def checkCommitVersionErrors(self): if not (hasattr(self._storage, 'supportsTransactionalUndo') and self._storage.supportsTransactionalUndo()): return eq = self.assertEqual oid1, version1 = self._setup_version('one') data, revid1 = self._storage.load(oid1, version1) eq(zodb_unpickle(data), MinPO(54)) t = Transaction() self._storage.tpc_begin(t) try: self.assertRaises(POSException.VersionCommitError, self._storage.commitVersion, 'one', 'one', t) finally: self._storage.tpc_abort(t) def checkNewSerialOnCommitVersionToVersion(self): oid, version = self._setup_version() data, vtid = self._storage.load(oid, version) data, ntid = self._storage.load(oid, '') version2 = 'test version 2' self._commitVersion(version, version2) data, tid = self._storage.load(oid, version2) self.failUnless(tid != vtid and tid != ntid, "New tid, %r, should be different from the old " "version, %r, and non-version, %r, tids." % (tid, vtid, ntid)) def checkModifyAfterAbortVersion(self): eq = self.assertEqual oid, version = self._setup_version() self._abortVersion(version) data, revid = self._storage.load(oid, '') # And modify it a few times revid = self._dostore(oid, revid=revid, data=MinPO(52)) revid = self._dostore(oid, revid=revid, data=MinPO(53)) revid = self._dostore(oid, revid=revid, data=MinPO(54)) data, newrevid = self._storage.load(oid, '') eq(newrevid, revid) eq(zodb_unpickle(data), MinPO(54)) def checkCommitToNonVersion(self): eq = self.assertEqual oid, version = self._setup_version() data, revid = self._storage.load(oid, version) eq(zodb_unpickle(data), MinPO(54)) data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(51)) self._commitVersion(version, '') data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(54)) def checkCommitToOtherVersion(self): eq = self.assertEqual oid1, version1 = self._setup_version('one') data, revid1 = self._storage.load(oid1, version1) eq(zodb_unpickle(data), MinPO(54)) oid2, version2 = self._setup_version('two') data, revid2 = self._storage.load(oid2, version2) eq(zodb_unpickle(data), MinPO(54)) # make sure we see the non-version data when appropriate data, revid2 = self._storage.load(oid1, version2) eq(zodb_unpickle(data), MinPO(51)) data, revid2 = self._storage.load(oid2, version1) eq(zodb_unpickle(data), MinPO(51)) data, revid2 = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(51)) # Okay, now let's commit object1 to version2 oids = self._commitVersion(version1, version2) eq(len(oids), 1) eq(oids[0], oid1) data, revid = self._storage.load(oid1, version2) eq(zodb_unpickle(data), MinPO(54)) data, revid = self._storage.load(oid2, version2) eq(zodb_unpickle(data), MinPO(54)) # an object can only exist in one version, so a load from # version1 should now give the non-version data data, revid2 = self._storage.load(oid1, version1) eq(zodb_unpickle(data), MinPO(51)) # as should a version that has never been used data, revid2 = self._storage.load(oid1, 'bela lugosi') eq(zodb_unpickle(data), MinPO(51)) def checkAbortOneVersionCommitTheOther(self): eq = self.assertEqual oid1, version1 = self._setup_version('one') data, revid1 = self._storage.load(oid1, version1) eq(zodb_unpickle(data), MinPO(54)) oid2, version2 = self._setup_version('two') data, revid2 = self._storage.load(oid2, version2) eq(zodb_unpickle(data), MinPO(54)) # Let's make sure we can't get object1 in version2 data, revid2 = self._storage.load(oid1, version2) eq(zodb_unpickle(data), MinPO(51)) oids = self._abortVersion(version1) eq(len(oids), 1) eq(oids[0], oid1) data, revid = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(51)) data, revid = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(51)) data, revid = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(51)) data, revid = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(51)) data, revid = self._storage.load(oid2, version2) eq(zodb_unpickle(data), MinPO(54)) # Okay, now let's commit version2 back to the trunk oids = self._commitVersion(version2, '') eq(len(oids), 1) eq(oids[0], oid2) data, revid = self._storage.load(oid1, '') eq(zodb_unpickle(data), MinPO(51)) # But the trunk should be up to date now data, revid = self._storage.load(oid2, '') eq(zodb_unpickle(data), MinPO(54)) data, revid = self._storage.load(oid2, version2) eq(zodb_unpickle(data), MinPO(54)) oid = self._storage.new_oid() revid = self._dostore(oid, revid=revid, data=MinPO(54), version='one') self.assertRaises(KeyError, self._storage.load, oid, '') self.assertRaises(KeyError, self._storage.load, oid, 'two') def checkCreateObjectInVersionWithAbort(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=21, version="one") revid = self._dostore(oid, revid=revid, data=23, version='one') revid = self._dostore(oid, revid=revid, data=34, version='one') # Now abort the version and the creation t = Transaction() self._storage.tpc_begin(t) tid, oids = self._storage.abortVersion('one', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) self.assertEqual(oids, [oid]) def checkPackVersions(self): db = DB(self._storage) cn = db.open(version="testversion") root = cn.root() obj = root["obj"] = MinPO("obj") root["obj2"] = MinPO("obj2") txn = transaction.get() txn.note("create 2 objs in version") txn.commit() obj.value = "77" txn = transaction.get() txn.note("modify obj in version") txn.commit() # undo the modification to generate a mix of backpointers # and versions for pack to chase info = db.undoInfo() db.undo(info[0]["id"]) txn = transaction.get() txn.note("undo modification") txn.commit() snooze() self._storage.pack(time.time(), referencesf) db.commitVersion("testversion") txn = transaction.get() txn.note("commit version") txn.commit() cn = db.open() root = cn.root() root["obj"] = "no version" txn = transaction.get() txn.note("modify obj") txn.commit() self._storage.pack(time.time(), referencesf) def checkPackVersionsInPast(self): db = DB(self._storage) cn = db.open(version="testversion") root = cn.root() obj = root["obj"] = MinPO("obj") root["obj2"] = MinPO("obj2") txn = transaction.get() txn.note("create 2 objs in version") txn.commit() obj.value = "77" txn = transaction.get() txn.note("modify obj in version") txn.commit() t0 = time.time() snooze() # undo the modification to generate a mix of backpointers # and versions for pack to chase info = db.undoInfo() db.undo(info[0]["id"]) txn = transaction.get() txn.note("undo modification") txn.commit() self._storage.pack(t0, referencesf) db.commitVersion("testversion") txn = transaction.get() txn.note("commit version") txn.commit() cn = db.open() root = cn.root() root["obj"] = "no version" txn = transaction.get() txn.note("modify obj") txn.commit() self._storage.pack(time.time(), referencesf) def checkPackVersionReachable(self): db = DB(self._storage) cn = db.open() root = cn.root() names = "a", "b", "c" for name in names: root[name] = MinPO(name) transaction.commit() for name in names: cn2 = db.open(version=name) rt2 = cn2.root() obj = rt2[name] obj.value = MinPO("version") transaction.commit() cn2.close() root["d"] = MinPO("d") transaction.commit() snooze() self._storage.pack(time.time(), referencesf) cn.sync() # make sure all the non-version data is there for name, obj in root.items(): self.assertEqual(name, obj.value) # make sure all the version-data is there, # and create a new revision in the version for name in names: cn2 = db.open(version=name) rt2 = cn2.root() obj = rt2[name].value self.assertEqual(obj.value, "version") obj.value = "still version" transaction.commit() cn2.close() db.abortVersion("b") txn = transaction.get() txn.note("abort version b") txn.commit() t = time.time() snooze() L = db.undoInfo() db.undo(L[0]["id"]) txn = transaction.get() txn.note("undo abort") txn.commit() self._storage.pack(t, referencesf) cn2 = db.open(version="b") rt2 = cn2.root() self.assertEqual(rt2["b"].value.value, "still version") def checkLoadBeforeVersion(self): eq = self.assertEqual oid = self._storage.new_oid() revid1 = self._dostore(oid, data=1) revid2 = self._dostore(oid, data=2, revid=revid1, version="kobe") revid3 = self._dostore(oid, data=3, revid=revid2, version="kobe") data, start_tid, end_tid = self._storage.loadBefore(oid, revid3) eq(zodb_unpickle(data), MinPO(1)) eq(start_tid, revid1) eq(end_tid, None) Zope-2.8.7-final/lib/python/ZODB/tests/testSubTransaction.py0000664000076400007640000000662310435502423022745 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## r""" ZODB subtransaction tests ========================= Subtransactions are provided by a generic transaction interface, but only supported by ZODB. These tests verify that some of the important cases work as expected. >>> import transaction >>> from ZODB import DB >>> from ZODB.tests.test_storage import MinimalMemoryStorage >>> from ZODB.tests.MinPO import MinPO First create a few objects in the database root with a normal commit. We're going to make a series of modifications to these objects. >>> db = DB(MinimalMemoryStorage()) >>> cn = db.open() >>> rt = cn.root() >>> def init(): ... global a, b, c ... a = rt["a"] = MinPO("a0") ... b = rt["b"] = MinPO("b0") ... c = rt["c"] = MinPO("c0") ... transaction.commit() >>> init() We'll also open a second database connection and use it to verify that the intermediate results of subtransactions are not visible to other connections. >>> cn2 = db.open(synch=False) >>> rt2 = cn2.root() >>> shadow_a = rt2["a"] >>> shadow_b = rt2["b"] >>> shadow_c = rt2["c"] Subtransaction commit --------------------- We'll make a series of modifications in subtransactions. >>> a.value = "a1" >>> b.value = "b1" >>> transaction.commit(1) >>> a.value, b.value ('a1', 'b1') >>> shadow_a.value, shadow_b.value ('a0', 'b0') >>> a.value = "a2" >>> c.value = "c1" >>> transaction.commit(1) >>> a.value, c.value ('a2', 'c1') >>> shadow_a.value, shadow_c.value ('a0', 'c0') >>> a.value = "a3" >>> transaction.commit(1) >>> a.value 'a3' >>> shadow_a.value 'a0' >>> transaction.commit() >>> a.value, b.value, c.value ('a3', 'b1', 'c1') Subtransaction with nested abort -------------------------------- >>> init() >>> a.value = "a1" >>> transaction.commit(1) >>> b.value = "b1" >>> transaction.commit(1) A sub-transaction abort will undo current changes, reverting to the database state as of the last sub-transaction commit. There is (apparently) no way to abort an already-committed subtransaction. >>> c.value = "c1" >>> transaction.abort(1) >>> a.value, b.value, c.value ('a1', 'b1', 'c0') Multiple aborts have no extra effect. >>> transaction.abort(1) >>> a.value, b.value, c.value ('a1', 'b1', 'c0') >>> transaction.commit() >>> a.value, b.value, c.value ('a1', 'b1', 'c0') Subtransaction with top-level abort ----------------------------------- >>> init() >>> a.value = "a1" >>> transaction.commit(1) >>> b.value = "b1" >>> transaction.commit(1) A sub-transaction abort will undo current changes, reverting to the database state as of the last sub-transaction commit. There is (apparently) no way to abort an already-committed subtransaction. >>> c.value = "c1" >>> transaction.abort(1) >>> transaction.abort() >>> a.value, b.value, c.value ('a0', 'b0', 'c0') """ from zope.testing import doctest def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/ZODB/tests/testSerialize.py0000664000076400007640000001037210435502423021731 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the serializer.""" import cPickle import cStringIO as StringIO import sys import unittest from ZODB import serialize class ClassWithNewargs(int): def __new__(cls, value): return int.__new__(cls, value) def __getnewargs__(self): return int(self), class ClassWithoutNewargs(object): def __init__(self, value): self.value = value def make_pickle(ob): sio = StringIO.StringIO() p = cPickle.Pickler(sio, 1) p.dump(ob) return sio.getvalue() def test_factory(conn, module_name, name): return globals()[name] class SerializerTestCase(unittest.TestCase): # old format: (module, name), None old_style_without_newargs = make_pickle( ((__name__, "ClassWithoutNewargs"), None)) # old format: (module, name), argtuple old_style_with_newargs = make_pickle( ((__name__, "ClassWithNewargs"), (1,))) # new format: klass new_style_without_newargs = make_pickle( ClassWithoutNewargs) # new format: klass, argtuple new_style_with_newargs = make_pickle( (ClassWithNewargs, (1,))) def test_getClassName(self): r = serialize.ObjectReader(factory=test_factory) eq = self.assertEqual eq(r.getClassName(self.old_style_with_newargs), __name__ + ".ClassWithNewargs") eq(r.getClassName(self.new_style_with_newargs), __name__ + ".ClassWithNewargs") eq(r.getClassName(self.old_style_without_newargs), __name__ + ".ClassWithoutNewargs") eq(r.getClassName(self.new_style_without_newargs), __name__ + ".ClassWithoutNewargs") def test_getGhost(self): # Use a TestObjectReader since we need _get_class() to be # implemented; otherwise this is just a BaseObjectReader. class TestObjectReader(serialize.ObjectReader): # A production object reader would optimize this, but we # don't need to in a test def _get_class(self, module, name): __import__(module) return getattr(sys.modules[module], name) r = TestObjectReader(factory=test_factory) g = r.getGhost(self.old_style_with_newargs) self.assert_(isinstance(g, ClassWithNewargs)) self.assertEqual(g, 1) g = r.getGhost(self.old_style_without_newargs) self.assert_(isinstance(g, ClassWithoutNewargs)) g = r.getGhost(self.new_style_with_newargs) self.assert_(isinstance(g, ClassWithNewargs)) g = r.getGhost(self.new_style_without_newargs) self.assert_(isinstance(g, ClassWithoutNewargs)) def test_myhasattr(self): class OldStyle: bar = "bar" def __getattr__(self, name): if name == "error": raise ValueError("whee!") else: raise AttributeError, name class NewStyle(object): bar = "bar" def _raise(self): raise ValueError("whee!") error = property(_raise) self.assertRaises(ValueError, serialize.myhasattr, OldStyle(), "error") self.assertRaises(ValueError, serialize.myhasattr, NewStyle(), "error") self.assert_(serialize.myhasattr(OldStyle(), "bar")) self.assert_(serialize.myhasattr(NewStyle(), "bar")) self.assert_(not serialize.myhasattr(OldStyle(), "rat")) self.assert_(not serialize.myhasattr(NewStyle(), "rat")) def test_suite(): from zope.testing import doctest suite = unittest.makeSuite(SerializerTestCase) suite.addTest(doctest.DocTestSuite("ZODB.serialize")) return suite Zope-2.8.7-final/lib/python/ZODB/tests/testfsoids.py0000664000076400007640000001417510435502423021276 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## r""" fsoids test, of the workhorse fsoids.Trace class ================================================ Let's get a temp file path to work with first. >>> import tempfile >>> path = tempfile.mktemp('.fs', 'Data') >>> print 'path:', path #doctest: +ELLIPSIS path: ...Data...fs More imports. >>> import ZODB >>> from ZODB.FileStorage import FileStorage >>> import transaction as txn >>> from BTrees.OOBTree import OOBTree >>> from ZODB.FileStorage.fsoids import Tracer # we're testing this Create an empty FileStorage. >>> st = FileStorage(path) There's not a lot interesting in an empty DB! >>> t = Tracer(path) >>> t.register_oids(0x123456) >>> t.register_oids(1) >>> t.register_oids(0) >>> t.run() >>> t.report() oid 0x00 0 revisions this oid was not defined (no data record for it found) oid 0x01 0 revisions this oid was not defined (no data record for it found) oid 0x123456 0 revisions this oid was not defined (no data record for it found) That didn't tell us much, but does show that the specified oids are sorted into increasing order. Create a root object and try again: >>> db = ZODB.DB(st) # yes, that creates a root object! >>> t = Tracer(path) >>> t.register_oids(0, 1) >>> t.run(); t.report() #doctest: +ELLIPSIS oid 0x00 persistent.mapping.PersistentMapping 1 revision tid 0x... offset=4 ... tid user='' tid description='initial database creation' new revision persistent.mapping.PersistentMapping at 52 oid 0x01 0 revisions this oid was not defined (no data record for it found) So we see oid 0 has been used in our one transaction, and that it was created there, and is a PersistentMapping. 4 is the file offset to the start of the transaction record, and 52 is the file offset to the start of the data record for oid 0 within this transaction. Because tids are timestamps too, the "..." parts vary across runs. The initial line for a tid actually looks like this: tid 0x035748597843b877 offset=4 2004-08-20 20:41:28.187000 Let's add a BTree and try again: >>> root = db.open().root() >>> root['tree'] = OOBTree() >>> txn.get().note('added an OOBTree') >>> txn.get().commit() >>> t = Tracer(path) >>> t.register_oids(0, 1) >>> t.run(); t.report() #doctest: +ELLIPSIS oid 0x00 persistent.mapping.PersistentMapping 2 revisions tid 0x... offset=4 ... tid user='' tid description='initial database creation' new revision persistent.mapping.PersistentMapping at 52 tid 0x... offset=168 ... tid user='' tid description='added an OOBTree' new revision persistent.mapping.PersistentMapping at 207 references 0x01 BTrees._OOBTree.OOBTree at 207 oid 0x01 BTrees._OOBTree.OOBTree 1 revision tid 0x... offset=168 ... tid user='' tid description='added an OOBTree' new revision BTrees._OOBTree.OOBTree at 363 referenced by 0x00 persistent.mapping.PersistentMapping at 207 So there are two revisions of oid 0 now, and the second references oid 1. One more, storing a reference in the BTree back to the root object: >>> tree = root['tree'] >>> tree['root'] = root >>> txn.get().note('circling back to the root') >>> txn.get().commit() >>> t = Tracer(path) >>> t.register_oids(0, 1, 2) >>> t.run(); t.report() #doctest: +ELLIPSIS oid 0x00 persistent.mapping.PersistentMapping 2 revisions tid 0x... offset=4 ... tid user='' tid description='initial database creation' new revision persistent.mapping.PersistentMapping at 52 tid 0x... offset=168 ... tid user='' tid description='added an OOBTree' new revision persistent.mapping.PersistentMapping at 207 references 0x01 BTrees._OOBTree.OOBTree at 207 tid 0x... offset=443 ... tid user='' tid description='circling back to the root' referenced by 0x01 BTrees._OOBTree.OOBTree at 491 oid 0x01 BTrees._OOBTree.OOBTree 2 revisions tid 0x... offset=168 ... tid user='' tid description='added an OOBTree' new revision BTrees._OOBTree.OOBTree at 363 referenced by 0x00 persistent.mapping.PersistentMapping at 207 tid 0x... offset=443 ... tid user='' tid description='circling back to the root' new revision BTrees._OOBTree.OOBTree at 491 references 0x00 persistent.mapping.PersistentMapping at 491 oid 0x02 0 revisions this oid was not defined (no data record for it found) Note that we didn't create any new object there (oid 2 is still unused), we just made oid 1 refer to oid 0. Therefore there's a new "new revision" line in the output for oid 1. Note that there's also new output for oid 0, even though the root object didn't change: we got new output for oid 0 because it's a traced oid and the new transaction made a new reference *to* it. Since the Trace constructor takes only one argument, the only sane thing you can do to make it fail is to give it a path to a file that doesn't exist: >>> Tracer('/eiruowieuu/lsijflfjlsijflsdf/eurowiurowioeuri/908479287.fs') Traceback (most recent call last): ... ValueError: must specify an existing FileStorage You get the same kind of exception if you pass it a path to an existing directory (the path must be to a file, not a directory): >>> import os >>> Tracer(os.path.dirname(__file__)) Traceback (most recent call last): ... ValueError: must specify an existing FileStorage Clean up. >>> st.close() >>> st.cleanup() # remove .fs, .index, etc """ from zope.testing import doctest def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/ZODB/tests/testDemoStorage.py0000664000076400007640000000405310435502423022212 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import ZODB.DemoStorage import unittest from ZODB.tests import StorageTestBase, BasicStorage, \ VersionStorage, Synchronization class DemoStorageTests(StorageTestBase.StorageTestBase, BasicStorage.BasicStorage, VersionStorage.VersionStorage, Synchronization.SynchronizedStorage, ): def setUp(self): self._storage = ZODB.DemoStorage.DemoStorage() def tearDown(self): self._storage.close() def checkOversizeNote(self): # This base class test checks for the common case where a storage # doesnt support huge transaction metadata. This storage doesnt # have this limit, so we inhibit this test here. pass def checkAbortVersionNonCurrent(self): # TODO: Need to implement a real loadBefore for DemoStorage? pass def checkLoadBeforeVersion(self): # TODO: Need to implement a real loadBefore for DemoStorage? pass # the next three pack tests depend on undo def checkPackVersionReachable(self): pass def checkPackVersions(self): pass def checkPackVersionsInPast(self): pass def test_suite(): suite = unittest.makeSuite(DemoStorageTests, 'check') return suite if __name__ == "__main__": loader = unittest.TestLoader() loader.testMethodPrefix = "check" unittest.main(testLoader=loader) Zope-2.8.7-final/lib/python/ZODB/tests/testmvcc.py0000664000076400007640000002723710435502423020742 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## r""" Multi-version concurrency control tests ======================================= Multi-version concurrency control (MVCC) exploits storages that store multiple revisions of an object to avoid read conflicts. Normally when an object is read from the storage, its most recent revision is read. Under MVCC, an older revision may be read so that the transaction sees a consistent view of the database. ZODB guarantees execution-time consistency: A single transaction will always see a consistent view of the database while it is executing. If transaction A is running, has already read an object O1, and a different transaction B modifies object O2, then transaction A can no longer read the current revision of O2. It must either read the version of O2 that is consistent with O1 or raise a ReadConflictError. When MVCC is in use, A will do the former. This note includes doctests that explain how MVCC is implemented (and test that the implementation is correct). The tests use a MinimalMemoryStorage that implements MVCC support, but not much else. >>> from ZODB.tests.test_storage import MinimalMemoryStorage >>> from ZODB import DB >>> db = DB(MinimalMemoryStorage()) We will use two different connections with the experimental setLocalTransaction() method to make sure that the connections act independently, even though they'll be run from a single thread. >>> import transaction >>> tm1 = transaction.TransactionManager() >>> cn1 = db.open(transaction_manager=tm1) The test will just use some MinPO objects. The next few lines just setup an initial database state. >>> from ZODB.tests.MinPO import MinPO >>> r = cn1.root() >>> r["a"] = MinPO(1) >>> r["b"] = MinPO(1) >>> tm1.get().commit() Now open a second connection. >>> tm2 = transaction.TransactionManager() >>> cn2 = db.open(transaction_manager=tm2) Connection high-water mark -------------------------- The ZODB Connection tracks a transaction high-water mark, which bounds the latest transaction id that can be read by the current transaction and still present a consistent view of the database. Transactions with ids up to but not including the high-water mark are OK to read. When a transaction commits, the database sends invalidations to all the other connections; the invalidation contains the transaction id and the oids of modified objects. The Connection stores the high-water mark in _txn_time, which is set to None until an invalidation arrives. >>> cn = db.open() >>> print cn._txn_time None >>> cn.invalidate(100, dict.fromkeys([1, 2])) >>> cn._txn_time 100 >>> cn.invalidate(200, dict.fromkeys([1, 2])) >>> cn._txn_time 100 A connection's high-water mark is set to the transaction id taken from the first invalidation processed by the connection. Transaction ids are monotonically increasing, so the first one seen during the current transaction remains the high-water mark for the duration of the transaction. We'd like simple abort and commit calls to make txn boundaries, but that doesn't work unless an object is modified. sync() will abort a transaction and process invalidations. >>> cn.sync() >>> print cn._txn_time # the high-water mark got reset to None None Basic functionality ------------------- The next bit of code includes a simple MVCC test. One transaction will modify "a." The other transaction will then modify "b" and commit. >>> r1 = cn1.root() >>> r1["a"].value = 2 >>> tm1.get().commit() >>> txn = db.lastTransaction() The second connection has its high-water mark set now. >>> cn2._txn_time == txn True It is safe to read "b," because it was not modified by the concurrent transaction. >>> r2 = cn2.root() >>> r2["b"]._p_serial < cn2._txn_time True >>> r2["b"].value 1 >>> r2["b"].value = 2 It is not safe, however, to read the current revision of "a" because it was modified at the high-water mark. If we read it, we'll get a non-current version. >>> r2["a"].value 1 >>> r2["a"]._p_serial < cn2._txn_time True We can confirm that we have a non-current revision by asking the storage. >>> db._storage.isCurrent(r2["a"]._p_oid, r2["a"]._p_serial) False It's possible to modify "a", but we get a conflict error when we commit the transaction. >>> r2["a"].value = 3 >>> tm2.get().commit() Traceback (most recent call last): ... ConflictError: database conflict error (oid 0x01, class ZODB.tests.MinPO.MinPO) >>> tm2.get().abort() This example will demonstrate that we can commit a transaction if we only modify current revisions. >>> print cn2._txn_time None >>> r1 = cn1.root() >>> r1["a"].value = 3 >>> tm1.get().commit() >>> txn = db.lastTransaction() >>> cn2._txn_time == txn True >>> r2["b"].value = r2["a"].value + 1 >>> r2["b"].value 3 >>> tm2.get().commit() >>> print cn2._txn_time None Object cache ------------ A Connection keeps objects in its cache so that multiple database references will always point to the same Python object. At transaction boundaries, objects modified by other transactions are ghostified so that the next transaction doesn't see stale state. We need to be sure the non-current objects loaded by MVCC are always ghosted. It should be trivial, because MVCC is only used when an invalidation has been received for an object. First get the database back in an initial state. >>> cn1.sync() >>> r1["a"].value = 0 >>> r1["b"].value = 0 >>> tm1.get().commit() >>> cn2.sync() >>> r2["a"].value 0 >>> r2["b"].value = 1 >>> tm2.get().commit() >>> r1["b"].value 0 >>> cn1.sync() # cn2 modified 'b', so cn1 should get a ghost for b >>> r1["b"]._p_state # -1 means GHOST -1 Closing the connection, committing a transaction, and aborting a transaction, should all have the same effect on non-current objects in cache. >>> def testit(): ... cn1.sync() ... r1["a"].value = 0 ... r1["b"].value = 0 ... tm1.get().commit() ... cn2.sync() ... r2["b"].value = 1 ... tm2.get().commit() >>> testit() >>> r1["b"]._p_state # 0 means UPTODATE, although note it's an older revision 0 >>> r1["b"].value 0 >>> r1["a"].value = 1 >>> tm1.get().commit() >>> r1["b"]._p_state -1 When a connection is closed, it is saved by the database. It will be reused by the next open() call (along with its object cache). >>> testit() >>> r1["a"].value = 1 >>> tm1.get().abort() >>> cn1.close() >>> cn3 = db.open() >>> cn1 is cn3 True >>> r1 = cn1.root() Although "b" is a ghost in cn1 at this point (because closing a connection has the same effect on non-current objects in the connection's cache as committing a transaction), not every object is a ghost. The root was in the cache and was current, so our first reference to it doesn't return a ghost. >>> r1._p_state # UPTODATE 0 >>> r1["b"]._p_state # GHOST -1 Interaction with Savepoints --------------------------- Basically, making a savepoint shouldn't have any effect on what a thread sees. Before ZODB 3.4.1, the internal TmpStore used when savepoints are pending didn't delegate all the methods necessary to make this work, so we'll do a quick test of that here. First get a clean slate: >>> cn1.close(); cn2.close() >>> cn1 = db.open(transaction_manager=tm1) >>> r1 = cn1.root() >>> r1["a"].value = 0 >>> r1["b"].value = 1 >>> tm1.commit() Now modify "a", but not "b", and make a savepoint. >>> r1["a"].value = 42 >>> sp = cn1.savepoint() Over in the other connection, modify "b" and commit it. This makes the first connection's state for b "old". >>> cn2 = db.open(transaction_manager=tm2) >>> r2 = cn2.root() >>> r2["a"].value, r2["b"].value # shouldn't see the change to "a" (0, 1) >>> r2["b"].value = 43 >>> tm2.commit() >>> r2["a"].value, r2["b"].value (0, 43) Now deactivate "b" in the first connection, and (re)fetch it. The first connection should still see 1, due to MVCC, but to get this old state TmpStore needs to handle the loadBefore() method. >>> r1["b"]._p_deactivate() Before 3.4.1, the next line died with AttributeError: TmpStore instance has no attribute 'loadBefore' >>> r1["b"]._p_state # ghost -1 >>> r1["b"].value 1 Just for fun, finish the commit and make sure both connections see the same things now. >>> tm1.commit() >>> cn1.sync(); cn2.sync() >>> r1["a"].value, r1["b"].value (42, 43) >>> r2["a"].value, r2["b"].value (42, 43) Late invalidation ----------------- The combination of ZEO and MVCC adds more complexity. Since invalidations are delivered asynchronously by ZEO, it is possible for an invalidation to arrive just after a request to load the invalidated object is sent. The connection can't use the just-loaded data, because the invalidation arrived first. The complexity for MVCC is that it must check for invalidated objects after it has loaded them, just in case. Rather than add all the complexity of ZEO to these tests, the MinimalMemoryStorage has a hook. We'll write a subclass that will deliver an invalidation when it loads an object. The hook allows us to test the Connection code. >>> class TestStorage(MinimalMemoryStorage): ... def __init__(self): ... self.hooked = {} ... self.count = 0 ... super(TestStorage, self).__init__() ... def registerDB(self, db, limit): ... self.db = db ... def hook(self, oid, tid, version): ... if oid in self.hooked: ... self.db.invalidate(tid, {oid:1}) ... self.count += 1 We can execute this test with a single connection, because we're synthesizing the invalidation that is normally generated by the second connection. We need to create two revisions so that there is a non-current revision to load. >>> ts = TestStorage() >>> db = DB(ts) >>> cn1 = db.open(transaction_manager=tm1) >>> r1 = cn1.root() >>> r1["a"] = MinPO(0) >>> r1["b"] = MinPO(0) >>> tm1.get().commit() >>> r1["b"].value = 1 >>> tm1.get().commit() >>> cn1.cacheMinimize() # makes everything in cache a ghost >>> oid = r1["b"]._p_oid >>> ts.hooked[oid] = 1 Once the oid is hooked, an invalidation will be delivered the next time it is activated. The code below activates the object, then confirms that the hook worked and that the old state was retrieved. >>> oid in cn1._invalidated False >>> r1["b"]._p_state -1 >>> r1["b"]._p_activate() >>> oid in cn1._invalidated True >>> ts.count 1 >>> r1["b"].value 0 No earlier revision available ----------------------------- We'll reuse the code from the example above, except that there will only be a single revision of "b." As a result, the attempt to activate "b" will result in a ReadConflictError. >>> ts = TestStorage() >>> db = DB(ts) >>> cn1 = db.open(transaction_manager=tm1) >>> r1 = cn1.root() >>> r1["a"] = MinPO(0) >>> r1["b"] = MinPO(0) >>> tm1.get().commit() >>> cn1.cacheMinimize() # makes everything in cache a ghost >>> oid = r1["b"]._p_oid >>> ts.hooked[oid] = 1 Again, once the oid is hooked, an invalidation will be delivered the next time it is activated. The code below activates the object, but unlike the section above, this is no older state to retrieve. >>> oid in cn1._invalidated False >>> r1["b"]._p_state -1 >>> r1["b"]._p_activate() Traceback (most recent call last): ... ReadConflictError: database read conflict error (oid 0x02, class ZODB.tests.MinPO.MinPO) >>> oid in cn1._invalidated True >>> ts.count 1 """ from zope.testing import doctest def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/ZODB/tests/TransactionalUndoVersionStorage.py0000664000076400007640000001557010435502423025432 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## # Check interactions between undo() and versions. Any storage that # supports both undo() and versions must pass these tests. import time import transaction from ZODB.serialize import referencesf from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_unpickle class TransactionalUndoVersionStorage: # `hook` is a callable used by the ZRS tests. def checkUndoInVersion(self, hook=None): eq = self.assertEqual unless = self.failUnless def check_objects(nonversiondata, versiondata): data, revid = self._storage.load(oid, version) self.assertEqual(zodb_unpickle(data), MinPO(versiondata)) data, revid = self._storage.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO(nonversiondata)) oid = self._storage.new_oid() version = 'one' revid_a = self._dostore(oid, data=MinPO(91)) revid_b = self._dostore(oid, revid=revid_a, data=MinPO(92), version=version) revid_c = self._dostore(oid, revid=revid_b, data=MinPO(93), version=version) info = self._storage.undoInfo() self._undo(info[0]['id'], [oid]) data, revid = self._storage.load(oid, '') # load() always returns the tid of the most recent reversion in 3.4, # so this old test of revid can't work anymore. ##eq(revid, revid_a) # But the data should be correct for the non-version revision. eq(zodb_unpickle(data), MinPO(91)) data, revid = self._storage.load(oid, version) unless(revid > revid_b and revid > revid_c) eq(zodb_unpickle(data), MinPO(92)) # Now commit the version... oids = self._commitVersion(version, "") eq(len(oids), 1) eq(oids[0], oid) check_objects(92, 92) # ...and undo the commit info = self._storage.undoInfo() self._undo(info[0]['id'], [oid]) check_objects(91, 92) if hook: # ZRS passes a hook that arranges to start a secondary at this # point in the test. hook() # Now abort the version. oids = self._abortVersion(version) assert len(oids) == 1 assert oids[0] == oid check_objects(91, 91) # Now undo the abort. info=self._storage.undoInfo() self._undo(info[0]['id'], [oid]) # And the object should be back in versions 'one' and ''. check_objects(91, 92) def checkUndoCommitVersion(self): def load_value(oid, version=''): data, revid = self._storage.load(oid, version) return zodb_unpickle(data).value # create a bunch of packable transactions oid = self._storage.new_oid() revid = '\000' * 8 for i in range(4): revid = self._dostore(oid, revid, description='packable%d' % i) pt = time.time() time.sleep(1) oid1 = self._storage.new_oid() version = 'version' revid1 = self._dostore(oid1, data=MinPO(0), description='create1') revid2 = self._dostore(oid1, data=MinPO(1), revid=revid1, version=version, description='version1') self._dostore(oid1, data=MinPO(2), revid=revid2, version=version, description='version2') self._dostore(description='create2') t = transaction.Transaction() t.description = 'commit version' self._storage.tpc_begin(t) self._storage.commitVersion(version, '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) info = self._storage.undoInfo() t_id = info[0]['id'] self.assertEqual(load_value(oid1), 2) self.assertEqual(load_value(oid1, version), 2) self._storage.pack(pt, referencesf) self._undo(t_id, note="undo commit version") self.assertEqual(load_value(oid1), 0) self.assertEqual(load_value(oid1, version), 2) data, tid, ver = self._storage.loadEx(oid1, "") # After undoing the version commit, the non-version data # once again becomes the non-version data from 'create1'. self.assertEqual(tid, self._storage.lastTransaction()) self.assertEqual(ver, "") # The current version data comes from an undo record, which # means that it gets data via the backpointer but tid from the # current txn. data, tid, ver = self._storage.loadEx(oid1, version) self.assertEqual(ver, version) self.assertEqual(tid, self._storage.lastTransaction()) def checkUndoAbortVersion(self): def load_value(oid, version=''): data, revid = self._storage.load(oid, version) return zodb_unpickle(data).value # create a bunch of packable transactions oid = self._storage.new_oid() revid = '\000' * 8 for i in range(3): revid = self._dostore(oid, revid, description='packable%d' % i) pt = time.time() time.sleep(1) oid1 = self._storage.new_oid() version = 'version' revid1 = self._dostore(oid1, data=MinPO(0), description='create1') revid2 = self._dostore(oid1, data=MinPO(1), revid=revid1, version=version, description='version1') self._dostore(oid1, data=MinPO(2), revid=revid2, version=version, description='version2') self._dostore(description='create2') self._abortVersion(version) info = self._storage.undoInfo() t_id = info[0]['id'] self.assertEqual(load_value(oid1), 0) # after abort, we should see non-version data self.assertEqual(load_value(oid1, version), 0) self._undo(t_id, note="undo abort version") self.assertEqual(load_value(oid1), 0) # t undo will re-create the version self.assertEqual(load_value(oid1, version), 2) info = self._storage.undoInfo() t_id = info[0]['id'] self._storage.pack(pt, referencesf) self._undo(t_id, note="undo undo") # undo of undo will put as back where we started self.assertEqual(load_value(oid1), 0) # after abort, we should see non-version data self.assertEqual(load_value(oid1, version), 0) Zope-2.8.7-final/lib/python/ZODB/tests/ConflictResolution.py0000664000076400007640000001432510435502423022731 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for application-level conflict resolution.""" from ZODB.POSException import ConflictError, UndoError from persistent import Persistent from transaction import Transaction from ZODB.tests.StorageTestBase import zodb_unpickle, zodb_pickle class PCounter(Persistent): _value = 0 def __repr__(self): return "" % self._value def inc(self): self._value = self._value + 1 def _p_resolveConflict(self, oldState, savedState, newState): savedDiff = savedState['_value'] - oldState['_value'] newDiff = newState['_value'] - oldState['_value'] oldState['_value'] = oldState['_value'] + savedDiff + newDiff return oldState # Insecurity: What if _p_resolveConflict _thinks_ it resolved the # conflict, but did something wrong? class PCounter2(PCounter): def _p_resolveConflict(self, oldState, savedState, newState): raise ConflictError class PCounter3(PCounter): def _p_resolveConflict(self, oldState, savedState, newState): raise AttributeError, "no attribute (testing conflict resolution)" class PCounter4(PCounter): def _p_resolveConflict(self, oldState, savedState): raise RuntimeError, "Can't get here; not enough args" class ConflictResolvingStorage: def checkResolve(self): obj = PCounter() obj.inc() oid = self._storage.new_oid() revid1 = self._dostoreNP(oid, data=zodb_pickle(obj)) obj.inc() obj.inc() # The effect of committing two transactions with the same # pickle is to commit two different transactions relative to # revid1 that add two to _value. revid2 = self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) revid3 = self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) data, serialno = self._storage.load(oid, '') inst = zodb_unpickle(data) self.assertEqual(inst._value, 5) def checkUnresolvable(self): obj = PCounter2() obj.inc() oid = self._storage.new_oid() revid1 = self._dostoreNP(oid, data=zodb_pickle(obj)) obj.inc() obj.inc() # The effect of committing two transactions with the same # pickle is to commit two different transactions relative to # revid1 that add two to _value. revid2 = self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) try: self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) except ConflictError, err: self.assert_("PCounter2" in str(err)) else: self.fail("Expected ConflictError") def checkZClassesArentResolved(self): from ZODB.ConflictResolution import find_global, BadClassName dummy_class_tuple = ('*foobar', ()) self.assertRaises(BadClassName, find_global, '*foobar', ()) def checkBuggyResolve1(self): obj = PCounter3() obj.inc() oid = self._storage.new_oid() revid1 = self._dostoreNP(oid, data=zodb_pickle(obj)) obj.inc() obj.inc() # The effect of committing two transactions with the same # pickle is to commit two different transactions relative to # revid1 that add two to _value. revid2 = self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) self.assertRaises(ConflictError, self._dostoreNP, oid, revid=revid1, data=zodb_pickle(obj)) def checkBuggyResolve2(self): obj = PCounter4() obj.inc() oid = self._storage.new_oid() revid1 = self._dostoreNP(oid, data=zodb_pickle(obj)) obj.inc() obj.inc() # The effect of committing two transactions with the same # pickle is to commit two different transactions relative to # revid1 that add two to _value. revid2 = self._dostoreNP(oid, revid=revid1, data=zodb_pickle(obj)) self.assertRaises(ConflictError, self._dostoreNP, oid, revid=revid1, data=zodb_pickle(obj)) class ConflictResolvingTransUndoStorage: def checkUndoConflictResolution(self): # This test is based on checkNotUndoable in the # TransactionalUndoStorage test suite. Except here, conflict # resolution should allow us to undo the transaction anyway. obj = PCounter() obj.inc() oid = self._storage.new_oid() revid_a = self._dostore(oid, data=obj) obj.inc() revid_b = self._dostore(oid, revid=revid_a, data=obj) obj.inc() revid_c = self._dostore(oid, revid=revid_b, data=obj) # Start the undo info = self._storage.undoInfo() tid = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) self._storage.undo(tid, t) self._storage.tpc_finish(t) def checkUndoUnresolvable(self): # This test is based on checkNotUndoable in the # TransactionalUndoStorage test suite. Except here, conflict # resolution should allow us to undo the transaction anyway. obj = PCounter2() obj.inc() oid = self._storage.new_oid() revid_a = self._dostore(oid, data=obj) obj.inc() revid_b = self._dostore(oid, revid=revid_a, data=obj) obj.inc() revid_c = self._dostore(oid, revid=revid_b, data=obj) # Start the undo info = self._storage.undoInfo() tid = info[1]['id'] t = Transaction() self._storage.tpc_begin(t) self.assertRaises(UndoError, self._storage.undo, tid, t) self._storage.tpc_abort(t) Zope-2.8.7-final/lib/python/ZODB/tests/PersistentStorage.py0000664000076400007640000000366210435502423022573 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test that a storage's values persist across open and close.""" class PersistentStorage: def checkUpdatesPersist(self): oids = [] def new_oid_wrapper(l=oids, new_oid=self._storage.new_oid): oid = new_oid() l.append(oid) return oid self._storage.new_oid = new_oid_wrapper self._dostore() oid = self._storage.new_oid() revid = self._dostore(oid) if self._storage.supportsVersions(): self._dostore(oid, revid, data=8, version='b') oid = self._storage.new_oid() revid = self._dostore(oid, data=1) revid = self._dostore(oid, revid, data=2) self._dostore(oid, revid, data=3) # keep copies of all the objects objects = [] for oid in oids: p, s = self._storage.load(oid, '') objects.append((oid, '', p, s)) ver = self._storage.modifiedInVersion(oid) if ver: p, s = self._storage.load(oid, ver) objects.append((oid, ver, p, s)) self._storage.close() self.open() # keep copies of all the objects for oid, ver, p, s in objects: _p, _s = self._storage.load(oid, ver) self.assertEquals(p, _p) self.assertEquals(s, _s) Zope-2.8.7-final/lib/python/ZODB/tests/IteratorStorage.py0000664000076400007640000002206510435502423022222 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Run tests against the iterator() interface for storages. Any storage that supports the iterator() method should be able to pass all these tests. """ from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_pickle, zodb_unpickle from ZODB.utils import U64, p64 from transaction import Transaction class IteratorCompare: def iter_verify(self, txniter, revids, val0): eq = self.assertEqual oid = self._oid val = val0 for reciter, revid in zip(txniter, revids + [None]): eq(reciter.tid, revid) for rec in reciter: eq(rec.oid, oid) eq(rec.tid, revid) eq(rec.version, '') eq(zodb_unpickle(rec.data), MinPO(val)) val = val + 1 eq(val, val0 + len(revids)) txniter.close() class IteratorStorage(IteratorCompare): def checkSimpleIteration(self): # Store a bunch of revisions of a single object self._oid = oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) # Now iterate over all the transactions and compare carefully txniter = self._storage.iterator() self.iter_verify(txniter, [revid1, revid2, revid3], 11) def checkClose(self): self._oid = oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) txniter = self._storage.iterator() txniter.close() self.assertRaises(IOError, txniter.__getitem__, 0) def checkVersionIterator(self): if not self._storage.supportsVersions(): return self._dostore() self._dostore(version='abort') self._dostore() self._dostore(version='abort') t = Transaction() self._storage.tpc_begin(t) self._storage.abortVersion('abort', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) self._dostore(version='commit') self._dostore() self._dostore(version='commit') t = Transaction() self._storage.tpc_begin(t) self._storage.commitVersion('commit', '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) txniter = self._storage.iterator() for trans in txniter: for data in trans: pass def checkUndoZombieNonVersion(self): if not hasattr(self._storage, 'supportsTransactionalUndo'): return if not self._storage.supportsTransactionalUndo(): return oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(94)) # Get the undo information info = self._storage.undoInfo() tid = info[0]['id'] # Undo the creation of the object, rendering it a zombie t = Transaction() self._storage.tpc_begin(t) oids = self._storage.undo(tid, t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) # Now attempt to iterator over the storage iter = self._storage.iterator() for txn in iter: for rec in txn: pass # The last transaction performed an undo of the transaction that # created object oid. (As Barry points out, the object is now in the # George Bailey state.) Assert that the final data record contains # None in the data attribute. self.assertEqual(rec.oid, oid) self.assertEqual(rec.data, None) def checkTransactionExtensionFromIterator(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(1)) iter = self._storage.iterator() count = 0 for txn in iter: self.assertEqual(txn._extension, {}) count +=1 self.assertEqual(count, 1) def checkIterationIntraTransaction(self): # TODO: Try this test with logging enabled. If you see something # like # # ZODB FS FS21 warn: FileStorageTests.fs truncated, possibly due to # damaged records at 4 # # Then the code in FileIterator.next() hasn't yet been fixed. # Should automate that check. oid = self._storage.new_oid() t = Transaction() data = zodb_pickle(MinPO(0)) try: self._storage.tpc_begin(t) self._storage.store(oid, '\0'*8, data, '', t) self._storage.tpc_vote(t) # Don't do tpc_finish yet it = self._storage.iterator() for x in it: pass finally: self._storage.tpc_finish(t) def checkLoadEx(self): oid = self._storage.new_oid() self._dostore(oid, data=42) data, tid, ver = self._storage.loadEx(oid, "") self.assertEqual(zodb_unpickle(data), MinPO(42)) match = False for txn in self._storage.iterator(): for rec in txn: if rec.oid == oid and rec.tid == tid: self.assertEqual(txn.tid, tid) match = True if not match: self.fail("Could not find transaction with matching id") class ExtendedIteratorStorage(IteratorCompare): def checkExtendedIteration(self): # Store a bunch of revisions of a single object self._oid = oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) revid2 = self._dostore(oid, revid=revid1, data=MinPO(12)) revid3 = self._dostore(oid, revid=revid2, data=MinPO(13)) revid4 = self._dostore(oid, revid=revid3, data=MinPO(14)) # Note that the end points are included # Iterate over all of the transactions with explicit start/stop txniter = self._storage.iterator(revid1, revid4) self.iter_verify(txniter, [revid1, revid2, revid3, revid4], 11) # Iterate over some of the transactions with explicit start txniter = self._storage.iterator(revid3) self.iter_verify(txniter, [revid3, revid4], 13) # Iterate over some of the transactions with explicit stop txniter = self._storage.iterator(None, revid2) self.iter_verify(txniter, [revid1, revid2], 11) # Iterate over some of the transactions with explicit start+stop txniter = self._storage.iterator(revid2, revid3) self.iter_verify(txniter, [revid2, revid3], 12) # Specify an upper bound somewhere in between values revid3a = p64((U64(revid3) + U64(revid4)) / 2) txniter = self._storage.iterator(revid2, revid3a) self.iter_verify(txniter, [revid2, revid3], 12) # Specify a lower bound somewhere in between values. # revid2 == revid1+1 is very likely on Windows. Adding 1 before # dividing ensures that "the midpoint" we compute is strictly larger # than revid1. revid1a = p64((U64(revid1) + 1 + U64(revid2)) / 2) assert revid1 < revid1a txniter = self._storage.iterator(revid1a, revid3a) self.iter_verify(txniter, [revid2, revid3], 12) # Specify an empty range txniter = self._storage.iterator(revid3, revid2) self.iter_verify(txniter, [], 13) # Specify a singleton range txniter = self._storage.iterator(revid3, revid3) self.iter_verify(txniter, [revid3], 13) class IteratorDeepCompare: def compare(self, storage1, storage2): eq = self.assertEqual iter1 = storage1.iterator() iter2 = storage2.iterator() for txn1, txn2 in zip(iter1, iter2): eq(txn1.tid, txn2.tid) eq(txn1.status, txn2.status) eq(txn1.user, txn2.user) eq(txn1.description, txn2.description) eq(txn1._extension, txn2._extension) for rec1, rec2 in zip(txn1, txn2): eq(rec1.oid, rec2.oid) eq(rec1.tid, rec2.tid) eq(rec1.version, rec2.version) eq(rec1.data, rec2.data) # Make sure there are no more records left in rec1 and rec2, # meaning they were the same length. self.assertRaises(IndexError, txn1.next) self.assertRaises(IndexError, txn2.next) # Make sure ther are no more records left in txn1 and txn2, meaning # they were the same length self.assertRaises(IndexError, iter1.next) self.assertRaises(IndexError, iter2.next) iter1.close() iter2.close() Zope-2.8.7-final/lib/python/ZODB/tests/LocalStorage.py0000664000076400007640000000212410435502423021455 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## class LocalStorage: """A single test that only make sense for local storages. A local storage is one that doens't use ZEO. The __len__() implementation for ZEO is inexact. """ def checkLen(self): eq = self.assertEqual # The length of the database ought to grow by one each time eq(len(self._storage), 0) self._dostore() eq(len(self._storage), 1) self._dostore() eq(len(self._storage), 2) Zope-2.8.7-final/lib/python/ZODB/tests/testActivityMonitor.py0000664000076400007640000000631010435502423023143 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the default activity monitor. See ZODB/ActivityMonitor.py $Id: testActivityMonitor.py 25186 2004-06-02 15:07:33Z jim $ """ import unittest import time from ZODB.ActivityMonitor import ActivityMonitor class FakeConnection: loads = 0 stores = 0 def _transferred(self, loads, stores): self.loads = self.loads + loads self.stores = self.stores + stores def getTransferCounts(self, clear=0): res = self.loads, self.stores if clear: self.loads = self.stores = 0 return res class Tests(unittest.TestCase): def testAddLogEntries(self): am = ActivityMonitor(history_length=3600) self.assertEqual(len(am.log), 0) c = FakeConnection() c._transferred(1, 2) am.closedConnection(c) c._transferred(3, 7) am.closedConnection(c) self.assertEqual(len(am.log), 2) def testTrim(self): am = ActivityMonitor(history_length=0.1) c = FakeConnection() c._transferred(1, 2) am.closedConnection(c) time.sleep(0.2) c._transferred(3, 7) am.closedConnection(c) self.assert_(len(am.log) <= 1) def testSetHistoryLength(self): am = ActivityMonitor(history_length=3600) c = FakeConnection() c._transferred(1, 2) am.closedConnection(c) time.sleep(0.2) c._transferred(3, 7) am.closedConnection(c) self.assertEqual(len(am.log), 2) am.setHistoryLength(0.1) self.assertEqual(am.getHistoryLength(), 0.1) self.assert_(len(am.log) <= 1) def testActivityAnalysis(self): am = ActivityMonitor(history_length=3600) c = FakeConnection() c._transferred(1, 2) am.closedConnection(c) c._transferred(3, 7) am.closedConnection(c) res = am.getActivityAnalysis(start=0, end=0, divisions=10) lastend = 0 for n in range(9): div = res[n] self.assertEqual(div['stores'], 0) self.assertEqual(div['loads'], 0) self.assert_(div['start'] > 0) self.assert_(div['start'] >= lastend) self.assert_(div['start'] < div['end']) lastend = div['end'] div = res[9] self.assertEqual(div['stores'], 9) self.assertEqual(div['loads'], 4) self.assert_(div['start'] > 0) self.assert_(div['start'] >= lastend) self.assert_(div['start'] < div['end']) def test_suite(): return unittest.makeSuite(Tests) if __name__=='__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZODB/tests/testRecover.py0000664000076400007640000001557110435502423021415 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the file storage recovery script.""" import base64 import os import random import sys import tempfile import unittest import StringIO import ZODB from ZODB.FileStorage import FileStorage import ZODB.fsrecover from persistent.mapping import PersistentMapping import transaction class RecoverTest(unittest.TestCase): level = 2 path = None def setUp(self): self.path = tempfile.mktemp(suffix=".fs") self.storage = FileStorage(self.path) self.populate() self.dest = tempfile.mktemp(suffix=".fs") self.recovered = None def tearDown(self): self.storage.close() if self.recovered is not None: self.recovered.close() self.storage.cleanup() temp = FileStorage(self.dest) temp.close() temp.cleanup() def populate(self): db = ZODB.DB(self.storage) cn = db.open() rt = cn.root() # Create a bunch of objects; the Data.fs is about 100KB. for i in range(50): d = rt[i] = PersistentMapping() transaction.commit() for j in range(50): d[j] = "a" * j transaction.commit() def damage(self, num, size): self.storage.close() # Drop size null bytes into num random spots. for i in range(num): offset = random.randint(0, self.storage._pos - size) f = open(self.path, "a+b") f.seek(offset) f.write("\0" * size) f.close() ITERATIONS = 5 # Run recovery, from self.path to self.dest. Return whatever # recovery printed to stdout, as a string. def recover(self): orig_stdout = sys.stdout faux_stdout = StringIO.StringIO() try: sys.stdout = faux_stdout try: ZODB.fsrecover.recover(self.path, self.dest, verbose=0, partial=True, force=False, pack=1) except SystemExit: raise RuntimeError, "recover tried to exit" finally: sys.stdout = orig_stdout return faux_stdout.getvalue() # Caution: because recovery is robust against many kinds of damage, # it's almost impossible for a call to self.recover() to raise an # exception. As a result, these tests may pass even if fsrecover.py # is broken badly. testNoDamage() tries to ensure that at least # recovery doesn't produce any error msgs if the input .fs is in # fact not damaged. def testNoDamage(self): output = self.recover() self.assert_('error' not in output, output) self.assert_('\n0 bytes removed during recovery' in output, output) # Verify that the recovered database is identical to the original. before = file(self.path, 'rb') before_guts = before.read() before.close() after = file(self.dest, 'rb') after_guts = after.read() after.close() self.assertEqual(before_guts, after_guts, "recovery changed a non-damaged .fs file") def testOneBlock(self): for i in range(self.ITERATIONS): self.damage(1, 1024) output = self.recover() self.assert_('error' in output, output) self.recovered = FileStorage(self.dest) self.recovered.close() os.remove(self.path) os.rename(self.dest, self.path) def testFourBlocks(self): for i in range(self.ITERATIONS): self.damage(4, 512) output = self.recover() self.assert_('error' in output, output) self.recovered = FileStorage(self.dest) self.recovered.close() os.remove(self.path) os.rename(self.dest, self.path) def testBigBlock(self): for i in range(self.ITERATIONS): self.damage(1, 32 * 1024) output = self.recover() self.assert_('error' in output, output) self.recovered = FileStorage(self.dest) self.recovered.close() os.remove(self.path) os.rename(self.dest, self.path) def testBadTransaction(self): # Find transaction headers and blast them. L = self.storage.undoLog() r = L[3] tid = base64.decodestring(r["id"] + "\n") pos1 = self.storage._txn_find(tid, 0) r = L[8] tid = base64.decodestring(r["id"] + "\n") pos2 = self.storage._txn_find(tid, 0) self.storage.close() # Overwrite the entire header. f = open(self.path, "a+b") f.seek(pos1 - 50) f.write("\0" * 100) f.close() output = self.recover() self.assert_('error' in output, output) self.recovered = FileStorage(self.dest) self.recovered.close() os.remove(self.path) os.rename(self.dest, self.path) # Overwrite part of the header. f = open(self.path, "a+b") f.seek(pos2 + 10) f.write("\0" * 100) f.close() output = self.recover() self.assert_('error' in output, output) self.recovered = FileStorage(self.dest) self.recovered.close() # Issue 1846: When a transaction had 'c' status (not yet committed), # the attempt to open a temp file to write the trailing bytes fell # into an infinite loop. def testUncommittedAtEnd(self): # Find a transaction near the end. L = self.storage.undoLog() r = L[1] tid = base64.decodestring(r["id"] + "\n") pos = self.storage._txn_find(tid, 0) # Overwrite its status with 'c'. f = open(self.path, "r+b") f.seek(pos + 16) current_status = f.read(1) self.assertEqual(current_status, ' ') f.seek(pos + 16) f.write('c') f.close() # Try to recover. The original bug was that this never completed -- # infinite loop in fsrecover.py. Also, in the ZODB 3.2 line, # reference to an undefined global masked the infinite loop. self.recover() # Verify the destination got truncated. self.assertEqual(os.path.getsize(self.dest), pos) # Get rid of the temp file holding the truncated bytes. os.remove(ZODB.fsrecover._trname) def test_suite(): return unittest.makeSuite(RecoverTest) Zope-2.8.7-final/lib/python/ZODB/tests/testFileStorage.py0000664000076400007640000004152410435502423022211 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import os, unittest import transaction import ZODB.FileStorage from ZODB import POSException from ZODB.tests import StorageTestBase, BasicStorage, \ TransactionalUndoStorage, VersionStorage, \ TransactionalUndoVersionStorage, PackableStorage, \ Synchronization, ConflictResolution, HistoryStorage, \ IteratorStorage, Corruption, RevisionStorage, PersistentStorage, \ MTStorage, ReadOnlyStorage, RecoveryStorage from ZODB.tests.StorageTestBase import MinPO, zodb_pickle class BaseFileStorageTests(StorageTestBase.StorageTestBase): def open(self, **kwargs): self._storage = ZODB.FileStorage.FileStorage('FileStorageTests.fs', **kwargs) def setUp(self): self.open(create=1) def tearDown(self): self._storage.close() self._storage.cleanup() class FileStorageTests( BaseFileStorageTests, BasicStorage.BasicStorage, TransactionalUndoStorage.TransactionalUndoStorage, RevisionStorage.RevisionStorage, VersionStorage.VersionStorage, TransactionalUndoVersionStorage.TransactionalUndoVersionStorage, PackableStorage.PackableStorage, PackableStorage.PackableUndoStorage, Synchronization.SynchronizedStorage, ConflictResolution.ConflictResolvingStorage, ConflictResolution.ConflictResolvingTransUndoStorage, HistoryStorage.HistoryStorage, IteratorStorage.IteratorStorage, IteratorStorage.ExtendedIteratorStorage, PersistentStorage.PersistentStorage, MTStorage.MTStorage, ReadOnlyStorage.ReadOnlyStorage ): def checkLongMetadata(self): s = "X" * 75000 try: self._dostore(user=s) except POSException.StorageError: pass else: self.fail("expect long user field to raise error") try: self._dostore(description=s) except POSException.StorageError: pass else: self.fail("expect long user field to raise error") def check_use_fsIndex(self): from ZODB.fsIndex import fsIndex self.assertEqual(self._storage._index.__class__, fsIndex) # A helper for checking that when an .index contains a dict for the # index, it's converted to an fsIndex when the file is opened. def convert_index_to_dict(self): # Convert the index in the current .index file to a Python dict. # Return the index originally found. import cPickle as pickle f = open('FileStorageTests.fs.index', 'r+b') p = pickle.Unpickler(f) data = p.load() index = data['index'] newindex = dict(index) data['index'] = newindex f.seek(0) f.truncate() p = pickle.Pickler(f, 1) p.dump(data) f.close() return index def check_conversion_to_fsIndex(self, read_only=False): from ZODB.fsIndex import fsIndex # Create some data, and remember the index. for i in range(10): self._dostore() oldindex_as_dict = dict(self._storage._index) # Save the index. self._storage.close() # Convert it to a dict. old_index = self.convert_index_to_dict() self.assert_(isinstance(old_index, fsIndex)) new_index = self.convert_index_to_dict() self.assert_(isinstance(new_index, dict)) # Verify it's converted to fsIndex in memory upon open. self.open(read_only=read_only) self.assert_(isinstance(self._storage._index, fsIndex)) # Verify it has the right content. newindex_as_dict = dict(self._storage._index) self.assertEqual(oldindex_as_dict, newindex_as_dict) # Check that the type on disk has changed iff read_only is False. self._storage.close() current_index = self.convert_index_to_dict() if read_only: self.assert_(isinstance(current_index, dict)) else: self.assert_(isinstance(current_index, fsIndex)) def check_conversion_to_fsIndex_readonly(self): # Same thing, but the disk .index should continue to hold a # Python dict. self.check_conversion_to_fsIndex(read_only=True) def check_conversion_from_dict_to_btree_data_in_fsIndex(self): # To support efficient range searches on its keys as part of # implementing a record iteration protocol in FileStorage, we # converted the fsIndex class from using a dictionary as its # self._data attribute to using an OOBTree in its stead. from ZODB.fsIndex import fsIndex from BTrees.OOBTree import OOBTree # Create some data, and remember the index. for i in range(10): self._dostore() data_dict = dict(self._storage._index._data) # Replace the OOBTree with a dictionary and commit it. self._storage._index._data = data_dict transaction.commit() # Save the index. self._storage.close() # Verify it's converted to fsIndex in memory upon open. self.open() self.assert_(isinstance(self._storage._index, fsIndex)) self.assert_(isinstance(self._storage._index._data, OOBTree)) # Verify it has the right content. new_data_dict = dict(self._storage._index._data) self.assertEqual(len(data_dict), len(new_data_dict)) for k in data_dict: old_tree = data_dict[k] new_tree = new_data_dict[k] self.assertEqual(list(old_tree.items()), list(new_tree.items())) def check_save_after_load_with_no_index(self): for i in range(10): self._dostore() self._storage.close() os.remove('FileStorageTests.fs.index') self.open() self.assertEqual(self._storage._saved, 1) def check_index_oid_ignored(self): # Prior to ZODB 3.2.6, the 'oid' value stored in the .index file # was believed. But there were cases where adding larger oids # didn't update the FileStorage ._oid attribute -- the restore() # method in particular didn't update it, and that's about the only # method copyTransactionsFrom() uses. A database copy created that # way then stored an 'oid' of z64 in the .index file. This created # torturous problems, as when that file was opened, "new" oids got # generated starting over from 0 again. # Now the cached 'oid' value is ignored: verify that this is so. import cPickle as pickle from ZODB.utils import z64 from ZODB.DB import DB # Create some data. db = DB(self._storage) conn = db.open() conn.root()['xyz'] = 1 transaction.commit() true_max_oid = self._storage._oid # Save away the index, and poke in a bad 'oid' value by hand. db.close() f = open('FileStorageTests.fs.index', 'r+b') p = pickle.Unpickler(f) data = p.load() saved_oid = data['oid'] self.assertEqual(true_max_oid, saved_oid) data['oid'] = z64 f.seek(0) f.truncate() p = pickle.Pickler(f, 1) p.dump(data) f.close() # Verify that we get the correct oid again when we reopen, despite # that we stored nonsense in the .index file's 'oid'. self.open() self.assertEqual(self._storage._oid, true_max_oid) # This would make the unit tests too slow # check_save_after_load_that_worked_hard(self) def check_periodic_save_index(self): # Check the basic algorithm oldsaved = self._storage._saved self._storage._records_before_save = 10 for i in range(4): self._dostore() self.assertEqual(self._storage._saved, oldsaved) self._dostore() self.assertEqual(self._storage._saved, oldsaved+1) # Now make sure the parameter changes as we get bigger for i in range(20): self._dostore() self.failUnless(self._storage._records_before_save > 20) def checkStoreBumpsOid(self): # If .store() is handed an oid bigger than the storage knows # about already, it's crucial that the storage bump its notion # of the largest oid in use. t = transaction.Transaction() self._storage.tpc_begin(t) giant_oid = '\xee' * 8 # Store an object. # oid, serial, data, version, transaction r1 = self._storage.store(giant_oid, '\0'*8, 'data', '', t) # Finish the transaction. r2 = self._storage.tpc_vote(t) self._storage.tpc_finish(t) # Before ZODB 3.2.6, this failed, with ._oid == z64. self.assertEqual(self._storage._oid, giant_oid) def checkRestoreBumpsOid(self): # As above, if .restore() is handed an oid bigger than the storage # knows about already, it's crucial that the storage bump its notion # of the largest oid in use. Because copyTransactionsFrom(), and # ZRS recovery, use the .restore() method, this is plain critical. t = transaction.Transaction() self._storage.tpc_begin(t) giant_oid = '\xee' * 8 # Store an object. # oid, serial, data, version, prev_txn, transaction r1 = self._storage.restore(giant_oid, '\0'*8, 'data', '', None, t) # Finish the transaction. r2 = self._storage.tpc_vote(t) self._storage.tpc_finish(t) # Before ZODB 3.2.6, this failed, with ._oid == z64. self.assertEqual(self._storage._oid, giant_oid) def checkCorruptionInPack(self): # This sets up a corrupt .fs file, with a redundant transaction # length mismatch. The implementation of pack in many releases of # ZODB blew up if the .fs file had such damage: it detected the # damage, but the code to raise CorruptedError referenced an undefined # global. import time from ZODB.DB import DB from ZODB.utils import U64, p64 from ZODB.FileStorage.format import CorruptedError db = DB(self._storage) conn = db.open() conn.root()['xyz'] = 1 transaction.commit() # Ensure it's all on disk. db.close() self._storage.close() # Reopen before damaging. self.open() # Open .fs directly, and damage content. f = open('FileStorageTests.fs', 'r+b') f.seek(0, 2) pos2 = f.tell() - 8 f.seek(pos2) tlen2 = U64(f.read(8)) # length-8 of the last transaction pos1 = pos2 - tlen2 + 8 # skip over the tid at the start f.seek(pos1) tlen1 = U64(f.read(8)) # should be redundant length-8 self.assertEqual(tlen1, tlen2) # verify that it is redundant # Now damage the second copy. f.seek(pos2) f.write(p64(tlen2 - 1)) f.close() # Try to pack. This used to yield # NameError: global name 's' is not defined try: self._storage.pack(time.time(), None) except CorruptedError, detail: self.assert_("redundant transaction length does not match " "initial transaction length" in str(detail)) else: self.fail("expected CorruptedError") def check_record_iternext(self): from ZODB.DB import DB db = DB(self._storage) conn = db.open() conn.root()['abc'] = MinPO('abc') conn.root()['xyz'] = MinPO('xyz') transaction.commit() # Ensure it's all on disk. db.close() self._storage.close() self.open() key = None for x in ('\000', '\001', '\002'): oid, tid, data, next_oid = self._storage.record_iternext(key) self.assertEqual(oid, ('\000' * 7) + x) key = next_oid expected_data, expected_tid = self._storage.load(oid, '') self.assertEqual(expected_data, data) self.assertEqual(expected_tid, tid) if x == '\002': self.assertEqual(next_oid, None) else: self.assertNotEqual(next_oid, None) class FileStorageRecoveryTest( StorageTestBase.StorageTestBase, RecoveryStorage.RecoveryStorage, ): def setUp(self): self._storage = ZODB.FileStorage.FileStorage("Source.fs", create=True) self._dst = ZODB.FileStorage.FileStorage("Dest.fs", create=True) def tearDown(self): self._storage.close() self._dst.close() self._storage.cleanup() self._dst.cleanup() def new_dest(self): return ZODB.FileStorage.FileStorage('Dest.fs') class SlowFileStorageTest(BaseFileStorageTests): level = 2 def check10Kstores(self): # The _get_cached_serial() method has a special case # every 8000 calls. Make sure it gets minimal coverage. oids = [[self._storage.new_oid(), None] for i in range(100)] for i in range(100): t = transaction.Transaction() self._storage.tpc_begin(t) for j in range(100): o = MinPO(j) oid, revid = oids[j] serial = self._storage.store(oid, revid, zodb_pickle(o), "", t) oids[j][1] = serial self._storage.tpc_vote(t) self._storage.tpc_finish(t) # Raise an exception if the tids in FileStorage fs aren't # strictly increasing. def checkIncreasingTids(fs): lasttid = '\0' * 8 for txn in fs.iterator(): if lasttid >= txn.tid: raise ValueError("tids out of order %r >= %r" % (lasttid, tid)) lasttid = txn.tid # Return a TimeStamp object 'minutes' minutes in the future. def timestamp(minutes): import time from persistent.TimeStamp import TimeStamp t = time.time() + 60 * minutes return TimeStamp(*time.gmtime(t)[:5] + (t % 60,)) def testTimeTravelOnOpen(): """ >>> from ZODB.FileStorage import FileStorage >>> from ZODB.DB import DB >>> import transaction >>> from zope.testing.loggingsupport import InstalledHandler Arrange to capture log messages -- they're an important part of this test! >>> handler = InstalledHandler('ZODB.FileStorage') Create a new file storage. >>> st = FileStorage('temp.fs', create=True) >>> db = DB(st) >>> db.close() First check the normal case: transactions are recorded with increasing tids, and time doesn't run backwards. >>> st = FileStorage('temp.fs') >>> db = DB(st) >>> conn = db.open() >>> conn.root()['xyz'] = 1 >>> transaction.get().commit() >>> checkIncreasingTids(st) >>> db.close() >>> st.cleanup() # remove .fs, .index, etc files >>> handler.records # i.e., no log messages [] Now force the database to have transaction records with tids from the future. >>> st = FileStorage('temp.fs', create=True) >>> st._ts = timestamp(15) # 15 minutes in the future >>> db = DB(st) >>> db.close() >>> st = FileStorage('temp.fs') # this should log a warning >>> db = DB(st) >>> conn = db.open() >>> conn.root()['xyz'] = 1 >>> transaction.get().commit() >>> checkIncreasingTids(st) >>> db.close() >>> st.cleanup() >>> [record.levelname for record in handler.records] ['WARNING'] >>> handler.clear() And one more time, with transaction records far in the future. We expect to log a critical error then, as a time so far in the future probably indicates a real problem with the system. Shorter spans may be due to clock drift. >>> st = FileStorage('temp.fs', create=True) >>> st._ts = timestamp(60) # an hour in the future >>> db = DB(st) >>> db.close() >>> st = FileStorage('temp.fs') # this should log a critical error >>> db = DB(st) >>> conn = db.open() >>> conn.root()['xyz'] = 1 >>> transaction.get().commit() >>> checkIncreasingTids(st) >>> db.close() >>> st.cleanup() >>> [record.levelname for record in handler.records] ['CRITICAL'] >>> handler.clear() >>> handler.uninstall() """ def test_suite(): from zope.testing import doctest suite = unittest.TestSuite() for klass in [FileStorageTests, Corruption.FileStorageCorruptTests, FileStorageRecoveryTest, SlowFileStorageTest]: suite.addTest(unittest.makeSuite(klass, "check")) suite.addTest(doctest.DocTestSuite()) return suite if __name__=='__main__': unittest.main() Zope-2.8.7-final/lib/python/ZODB/tests/BasicStorage.py0000664000076400007640000002030210435502423021442 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Run the basic tests for a storage as described in the official storage API The most complete and most out-of-date description of the interface is: http://www.zope.org/Documentation/Developer/Models/ZODB/ZODB_Architecture_Storage_Interface_Info.html All storages should be able to pass these tests. """ from ZODB import POSException from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase \ import zodb_unpickle, zodb_pickle, handle_serials import transaction ZERO = '\0'*8 class BasicStorage: def checkBasics(self): t = transaction.Transaction() self._storage.tpc_begin(t) # This should simply return self._storage.tpc_begin(t) # Aborting is easy self._storage.tpc_abort(t) # Test a few expected exceptions when we're doing operations giving a # different Transaction object than the one we've begun on. self._storage.tpc_begin(t) self.assertRaises( POSException.StorageTransactionError, self._storage.store, 0, 0, 0, 0, transaction.Transaction()) try: self._storage.abortVersion('dummy', transaction.Transaction()) except (POSException.StorageTransactionError, POSException.VersionCommitError): pass # test passed ;) else: assert 0, "Should have failed, invalid transaction." try: self._storage.commitVersion('dummy', 'dummer', transaction.Transaction()) except (POSException.StorageTransactionError, POSException.VersionCommitError): pass # test passed ;) else: assert 0, "Should have failed, invalid transaction." self.assertRaises( POSException.StorageTransactionError, self._storage.store, 0, 1, 2, 3, transaction.Transaction()) self._storage.tpc_abort(t) def checkSerialIsNoneForInitialRevision(self): eq = self.assertEqual oid = self._storage.new_oid() txn = transaction.Transaction() self._storage.tpc_begin(txn) # Use None for serial. Don't use _dostore() here because that coerces # serial=None to serial=ZERO. r1 = self._storage.store(oid, None, zodb_pickle(MinPO(11)), '', txn) r2 = self._storage.tpc_vote(txn) self._storage.tpc_finish(txn) newrevid = handle_serials(oid, r1, r2) data, revid = self._storage.load(oid, '') value = zodb_unpickle(data) eq(value, MinPO(11)) eq(revid, newrevid) def checkNonVersionStore(self): revid = ZERO newrevid = self._dostore(revid=None) # Finish the transaction. self.assertNotEqual(newrevid, revid) def checkNonVersionStoreAndLoad(self): eq = self.assertEqual oid = self._storage.new_oid() self._dostore(oid=oid, data=MinPO(7)) data, revid = self._storage.load(oid, '') value = zodb_unpickle(data) eq(value, MinPO(7)) # Now do a bunch of updates to an object for i in range(13, 22): revid = self._dostore(oid, revid=revid, data=MinPO(i)) # Now get the latest revision of the object data, revid = self._storage.load(oid, '') eq(zodb_unpickle(data), MinPO(21)) def checkNonVersionModifiedInVersion(self): oid = self._storage.new_oid() self._dostore(oid=oid) self.assertEqual(self._storage.modifiedInVersion(oid), '') def checkConflicts(self): oid = self._storage.new_oid() revid1 = self._dostore(oid, data=MinPO(11)) self._dostore(oid, revid=revid1, data=MinPO(12)) self.assertRaises(POSException.ConflictError, self._dostore, oid, revid=revid1, data=MinPO(13)) def checkWriteAfterAbort(self): oid = self._storage.new_oid() t = transaction.Transaction() self._storage.tpc_begin(t) self._storage.store(oid, ZERO, zodb_pickle(MinPO(5)), '', t) # Now abort this transaction self._storage.tpc_abort(t) # Now start all over again oid = self._storage.new_oid() self._dostore(oid=oid, data=MinPO(6)) def checkAbortAfterVote(self): oid1 = self._storage.new_oid() revid1 = self._dostore(oid=oid1, data=MinPO(-2)) oid = self._storage.new_oid() t = transaction.Transaction() self._storage.tpc_begin(t) self._storage.store(oid, ZERO, zodb_pickle(MinPO(5)), '', t) # Now abort this transaction self._storage.tpc_vote(t) self._storage.tpc_abort(t) # Now start all over again oid = self._storage.new_oid() revid = self._dostore(oid=oid, data=MinPO(6)) for oid, revid in [(oid1, revid1), (oid, revid)]: data, _revid = self._storage.load(oid, '') self.assertEqual(revid, _revid) def checkStoreTwoObjects(self): noteq = self.assertNotEqual p31, p32, p51, p52 = map(MinPO, (31, 32, 51, 52)) oid1 = self._storage.new_oid() oid2 = self._storage.new_oid() noteq(oid1, oid2) revid1 = self._dostore(oid1, data=p31) revid2 = self._dostore(oid2, data=p51) noteq(revid1, revid2) revid3 = self._dostore(oid1, revid=revid1, data=p32) revid4 = self._dostore(oid2, revid=revid2, data=p52) noteq(revid3, revid4) def checkGetSerial(self): if not hasattr(self._storage, 'getSerial'): return eq = self.assertEqual p41, p42 = map(MinPO, (41, 42)) oid = self._storage.new_oid() self.assertRaises(KeyError, self._storage.getSerial, oid) # Now store a revision revid1 = self._dostore(oid, data=p41) eq(revid1, self._storage.getSerial(oid)) # And another one revid2 = self._dostore(oid, revid=revid1, data=p42) eq(revid2, self._storage.getSerial(oid)) def checkTwoArgBegin(self): # Unsure: how standard is three-argument tpc_begin()? t = transaction.Transaction() tid = '\0\0\0\0\0psu' self._storage.tpc_begin(t, tid) oid = self._storage.new_oid() data = zodb_pickle(MinPO(8)) self._storage.store(oid, None, data, '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) def checkLen(self): # len(storage) reports the number of objects. # check it is zero when empty self.assertEqual(len(self._storage),0) # check it is correct when the storage contains two object. # len may also be zero, for storages that do not keep track # of this number self._dostore(data=MinPO(22)) self._dostore(data=MinPO(23)) self.assert_(len(self._storage) in [0,2]) def checkGetSize(self): self._dostore(data=MinPO(25)) size = self._storage.getSize() # The storage API doesn't make any claims about what size # means except that it ought to be printable. str(size) def checkNote(self): oid = self._storage.new_oid() t = transaction.Transaction() self._storage.tpc_begin(t) t.note('this is a test') self._storage.store(oid, ZERO, zodb_pickle(MinPO(5)), '', t) self._storage.tpc_vote(t) self._storage.tpc_finish(t) def checkGetExtensionMethods(self): m = self._storage.getExtensionMethods() self.assertEqual(type(m),type({})) for k,v in m.items(): self.assertEqual(v,None) self.assert_(callable(getattr(self._storage,k))) Zope-2.8.7-final/lib/python/ZODB/DEPENDENCIES.cfg0000664000076400007640000000016410435502423017673 0ustar ajungajungBTrees ZConfig persistent transaction # referenced by ZODB.config and related tests ZEO zope.interface zope.testing Zope-2.8.7-final/lib/python/ZODB/config.xml0000664000076400007640000000020610435502423017350 0ustar ajungajung

Zope-2.8.7-final/lib/python/ZODB/storage.xml0000664000076400007640000000015010435502423017545 0ustar ajungajung
Zope-2.8.7-final/lib/python/ZODB/collaborations.txt0000664000076400007640000001264210435502423021144 0ustar ajungajungParticipants DB: ZODB.DB.DB C: ZODB.Connection.Connection S: ZODB.FileStorage.FileStorage T: transaction.interfaces.ITransaction TM: transaction.interfaces.ITransactionManager o1, o2, ...: pre-existing persistent objects Scenario """Simple fetch, modify, commit.""" DB.open() create C TM.registerSynch(C) TM.begin() create T C.get(1) # fetches o1 C.get(2) # fetches o2 C.get(3) # fetches o3 o1.modify() # anything that modifies o1 C.register(o1) T.join(C) o2.modify() C.register(o2) # T.join(C) does not happen again o1.modify() # C.register(o1) doesn't happen again, because o1 was already # in the changed state. T.commit() C.beforeCompletion(T) C.tpc_begin(T) S.tpc_begin(T) C.commit(T) S.store(1, ..., T) S.store(2, ..., T) # o3 is not stored, because it wasn't modified C.tpc_vote(T) S.tpc_vote(T) C.tpc_finish(T) S.tpc_finish(T, f) # f is a callback function, which arranges # to call DB.invalidate (next) DB.invalidate(tid, {1: 1, 2: 1}, C) C2.invalidate(tid, {1: 1, 2: 1}) # for all connections # C2 to DB, where C2 # is not C TM.free(T) C.afterCompletion(T) C._flush_invalidations() # Processes invalidations that may have come in from other # transactions. Participants DB: ZODB.DB.DB C: ZODB.Connection.Connection S: ZODB.FileStorage.FileStorage T: transaction.interfaces.ITransaction TM: transaction.interfaces.ITransactionManager o1, o2, ...: pre-existing persistent objects Scenario """Simple fetch, modify, abort.""" DB.open() create C TM.registerSynch(C) TM.begin() create T C.get(1) # fetches o1 C.get(2) # fetches o2 C.get(3) # fetches o3 o1.modify() # anything that modifies o1 C.register(o1) T.join(C) o2.modify() C.register(o2) # T.join(C) does not happen again o1.modify() # C.register(o1) doesn't happen again, because o1 was already # in the changed state. T.abort() C.beforeCompletion(T) C.abort(T) C._cache.invalidate(1) # toss changes to o1 C._cache.invalidate(2) # toss changes to o2 # o3 wasn't modified, and its cache entry isn't invalidated. TM.free(T) C.afterCompletion(T) C._flush_invalidations() # Processes invalidations that may have come in from other # transactions. Participants: T: ITransaction o1, o2, o3: some persistent objects C1, C2, C3: resource managers S1, S2: Transaction savepoint objects s11, s21, s22: resource-manager savepoints Scenario """Rollback of a savepoint""" create T o1.modify() C1.regisiter(o1) T.join(C1) T.savepoint() C1.savepoint() return s11 return S1 = Savepoint(T, [r11]) o1.modify() C1.regisiter(o1) o2.modify() C2.regisiter(o2) T.join(C2) T.savepoint() C1.savepoint() return s21 C2.savepoint() return s22 return S2 = Savepoint(T, [r21, r22]) o3.modify() C3.regisiter(o3) T.join(C3) S1.rollback() S2.rollback() T.discard() C1.discard() C2.discard() C3.discard() o3.invalidate() S2.discard() s21.discard() # roll back changes since previous, which is r11 C1.discard(s21) o1.invalidate() # truncates temporary storage to s21's position s22.discard() # roll back changes since previous, which is r11 C1.discard(s22) o2.invalidate() # truncates temporary storage to beginning, because # s22 was the first savepoint. (Perhaps conection # savepoints record the log position before the # data were written, which is 0 in this case. T.commit() C1.beforeCompletion(T) C2.beforeCompletion(T) C3.beforeCompletion(T) C1.tpc_begin(T) S1.tpc_begin(T) C2.tpc_begin(T) C3.tpc_begin(T) C1.commit(T) S1.store(1, ..., T) C2.commit(T) C3.commit(T) C1.tpc_vote(T) S1.tpc_vote(T) C2.tpc_vote(T) C3.tpc_vote(T) C1.tpc_finish(T) S1.tpc_finish(T, f) # f is a callback function, which arranges c# to call DB.invalidate (next) DB.invalidate(tid, {1: 1}, C) TM.free(T) C1.afterCompletion(T) C1._flush_invalidations() C2.afterCompletion(T) C2._flush_invalidations() C3.afterCompletion(T) C3._flush_invalidations() Zope-2.8.7-final/lib/python/ZODB/UndoLogCompatible.py0000664000076400007640000000300510435502423021302 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Provide backward compatibility with storages that only have undoLog().""" class UndoLogCompatible: def undoInfo(self, first=0, last=-20, specification=None): if specification: # filter(desc) returns true iff `desc` is a "superdict" # of `specification`, meaning that `desc` contains the same # (key, value) pairs as `specification`, and possibly additional # (key, value) pairs. Another way to do this might be # d = desc.copy() # d.update(specification) # return d == desc def filter(desc, spec=specification.items()): get = desc.get for k, v in spec: if get(k, None) != v: return 0 return 1 else: filter = None return self.undoLog(first, last, filter) Zope-2.8.7-final/lib/python/ZODB/ConflictResolution.py0000664000076400007640000001070110435502423021561 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import logging from cStringIO import StringIO from cPickle import Unpickler, Pickler from pickle import PicklingError from ZODB.POSException import ConflictError from ZODB.loglevels import BLATHER logger = logging.getLogger('ZODB.ConflictResolution') ResolvedSerial = 'rs' class BadClassName(Exception): pass _class_cache = {} _class_cache_get = _class_cache.get def find_global(*args): cls = _class_cache_get(args, 0) if cls == 0: # Not cached. Try to import try: module = __import__(args[0], {}, {}, ['cluck']) except ImportError: cls = 1 else: cls = getattr(module, args[1], 1) _class_cache[args] = cls if cls == 1: logger.log(BLATHER, "Unable to load class", exc_info=True) if cls == 1: # Not importable raise BadClassName(*args) return cls def state(self, oid, serial, prfactory, p=''): p = p or self.loadSerial(oid, serial) file = StringIO(p) unpickler = Unpickler(file) unpickler.find_global = find_global unpickler.persistent_load = prfactory.persistent_load unpickler.load() # skip the class tuple return unpickler.load() class PersistentReference: def __repr__(self): return "PR(%s %s)" % (id(self), self.data) def __getstate__(self): raise PicklingError, "Can't pickle PersistentReference" class PersistentReferenceFactory: data = None def persistent_load(self, oid): if self.data is None: self.data = {} r = self.data.get(oid, None) if r is None: r = PersistentReference() r.data = oid self.data[oid] = r return r def persistent_id(object): if getattr(object, '__class__', 0) is not PersistentReference: return None return object.data _unresolvable = {} def tryToResolveConflict(self, oid, committedSerial, oldSerial, newpickle, committedData=''): # class_tuple, old, committed, newstate = ('',''), 0, 0, 0 try: prfactory = PersistentReferenceFactory() file = StringIO(newpickle) unpickler = Unpickler(file) unpickler.find_global = find_global unpickler.persistent_load = prfactory.persistent_load meta = unpickler.load() if isinstance(meta, tuple): klass = meta[0] newargs = meta[1] or () if isinstance(klass, tuple): klass = find_global(*klass) else: klass = meta newargs = () if klass in _unresolvable: return None newstate = unpickler.load() inst = klass.__new__(klass, *newargs) try: resolve = inst._p_resolveConflict except AttributeError: _unresolvable[klass] = 1 return None old = state(self, oid, oldSerial, prfactory) committed = state(self, oid, committedSerial, prfactory, committedData) resolved = resolve(old, committed, newstate) file = StringIO() pickler = Pickler(file,1) pickler.persistent_id = persistent_id pickler.dump(meta) pickler.dump(resolved) return file.getvalue(1) except (ConflictError, BadClassName): return None except: # If anything else went wrong, catch it here and avoid passing an # arbitrary exception back to the client. The error here will mask # the original ConflictError. A client can recover from a # ConflictError, but not necessarily from other errors. But log # the error so that any problems can be fixed. logger.error("Unexpected error", exc_info=True) return None class ConflictResolvingStorage: "Mix-in class that provides conflict resolution handling for storages" tryToResolveConflict = tryToResolveConflict Zope-2.8.7-final/lib/python/ZODB/BaseStorage.py0000664000076400007640000003431710435502423020144 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Handy standard storage machinery $Id: BaseStorage.py 38773 2005-10-05 19:44:07Z tim_one $ """ import cPickle import threading import time import logging from struct import pack as _structpack, unpack as _structunpack from persistent.TimeStamp import TimeStamp from ZODB import POSException from ZODB.utils import z64, oid_repr from ZODB.UndoLogCompatible import UndoLogCompatible log = logging.getLogger("ZODB.BaseStorage") class BaseStorage(UndoLogCompatible): """Abstract base class that supports storage implementations. A subclass must define the following methods: load() close() cleanup() lastSerial() lastTransaction() It must override these hooks: _begin() _vote() _abort() _finish() _clear_temp() If it stores multiple revisions, it should implement loadSerial() loadBefore() iterator() If the subclass wants to implement undo, it should implement the multiple revision methods and: loadSerial() undo() undoInfo() undoLog() If the subclass wants to implement versions, it must implement: abortVersion() commitVersion() modifiedInVersion() versionEmpty() versions() Each storage will have two locks that are accessed via lock acquire and release methods bound to the instance. (Yuck.) _lock_acquire / _lock_release (reentrant) _commit_lock_acquire / _commit_lock_release The commit lock is acquired in tpc_begin() and released in tpc_abort() and tpc_finish(). It is never acquired with the other lock held. The other lock appears to protect _oid and _transaction and perhaps other things. It is always held when load() is called, so presumably the load() implementation should also acquire the lock. """ _transaction=None # Transaction that is being committed _tstatus=' ' # Transaction status, used for copying data _is_read_only = False def __init__(self, name, base=None): self.__name__= name log.debug("create storage %s", self.__name__) # Allocate locks: l = threading.RLock() self._lock_acquire = l.acquire self._lock_release = l.release l = threading.Lock() self._commit_lock_acquire = l.acquire self._commit_lock_release = l.release t=time.time() t=self._ts=apply(TimeStamp,(time.gmtime(t)[:5]+(t%60,))) self._tid = `t` # ._oid is the highest oid in use (0 is always in use -- it's # a reserved oid for the root object). Our new_oid() method # increments it by 1, and returns the result. It's really a # 64-bit integer stored as an 8-byte big-endian string. if base is None: self._oid = z64 else: self._oid = base._oid def abortVersion(self, src, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) return self._tid, [] def commitVersion(self, src, dest, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) return self._tid, [] def close(self): pass def cleanup(self): pass def sortKey(self): """Return a string that can be used to sort storage instances. The key must uniquely identify a storage and must be the same across multiple instantiations of the same storage. """ # name may not be sufficient, e.g. ZEO has a user-definable name. return self.__name__ def getName(self): return self.__name__ def getSize(self): return len(self)*300 # WAG! def history(self, oid, version, length=1, filter=None): pass def modifiedInVersion(self, oid): return '' def new_oid(self): if self._is_read_only: raise POSException.ReadOnlyError() self._lock_acquire() try: last = self._oid d = ord(last[-1]) if d < 255: # fast path for the usual case last = last[:-1] + chr(d+1) else: # there's a carry out of the last byte last_as_long, = _structunpack(">Q", last) last = _structpack(">Q", last_as_long + 1) self._oid = last return last finally: self._lock_release() # Update the maximum oid in use, under protection of a lock. The # maximum-in-use attribute is changed only if possible_new_max_oid is # larger than its current value. def set_max_oid(self, possible_new_max_oid): self._lock_acquire() try: if possible_new_max_oid > self._oid: self._oid = possible_new_max_oid finally: self._lock_release() def registerDB(self, db, limit): pass # we don't care def isReadOnly(self): return self._is_read_only def supportsUndo(self): return 0 def supportsVersions(self): return 0 def tpc_abort(self, transaction): self._lock_acquire() try: if transaction is not self._transaction: return try: self._abort() self._clear_temp() self._transaction = None finally: self._commit_lock_release() finally: self._lock_release() def _abort(self): """Subclasses should redefine this to supply abort actions""" pass def tpc_begin(self, transaction, tid=None, status=' '): if self._is_read_only: raise POSException.ReadOnlyError() self._lock_acquire() try: if self._transaction is transaction: return self._lock_release() self._commit_lock_acquire() self._lock_acquire() self._transaction = transaction self._clear_temp() user = transaction.user desc = transaction.description ext = transaction._extension if ext: ext = cPickle.dumps(ext, 1) else: ext = "" self._ude = user, desc, ext if tid is None: now = time.time() t = TimeStamp(*(time.gmtime(now)[:5] + (now % 60,))) self._ts = t = t.laterThan(self._ts) self._tid = `t` else: self._ts = TimeStamp(tid) self._tid = tid self._tstatus = status self._begin(self._tid, user, desc, ext) finally: self._lock_release() def _begin(self, tid, u, d, e): """Subclasses should redefine this to supply transaction start actions. """ pass def tpc_vote(self, transaction): self._lock_acquire() try: if transaction is not self._transaction: return self._vote() finally: self._lock_release() def _vote(self): """Subclasses should redefine this to supply transaction vote actions. """ pass def tpc_finish(self, transaction, f=None): # It's important that the storage calls the function we pass # while it still has its lock. We don't want another thread # to be able to read any updated data until we've had a chance # to send an invalidation message to all of the other # connections! self._lock_acquire() try: if transaction is not self._transaction: return try: if f is not None: f(self._tid) u, d, e = self._ude self._finish(self._tid, u, d, e) self._clear_temp() return self._tid finally: self._ude = None self._transaction = None self._commit_lock_release() finally: self._lock_release() def _finish(self, tid, u, d, e): """Subclasses should redefine this to supply transaction finish actions """ pass def undo(self, transaction_id, txn): if self._is_read_only: raise POSException.ReadOnlyError() raise POSException.UndoError, 'non-undoable transaction' def undoLog(self, first, last, filter=None): return () def versionEmpty(self, version): return 1 def versions(self, max=None): return () def pack(self, t, referencesf): if self._is_read_only: raise POSException.ReadOnlyError() def getSerial(self, oid): self._lock_acquire() try: v = self.modifiedInVersion(oid) pickledata, serial = self.load(oid, v) return serial finally: self._lock_release() def loadSerial(self, oid, serial): raise POSException.Unsupported, ( "Retrieval of historical revisions is not supported") def loadBefore(self, oid, tid): """Return most recent revision of oid before tid committed.""" # Unsure: Is it okay for loadBefore() to return current data? # There doesn't seem to be a good reason to forbid it, even # though the typical use of this method will never find # current data. But maybe we should call it loadByTid()? n = 2 start_time = None end_time = None while start_time is None: # The history() approach is a hack, because the dict # returned by history() doesn't contain a tid. It # contains a serialno, which is often the same, but isn't # required to be. We'll pretend it is for now. # A second problem is that history() doesn't say anything # about whether the transaction status. If it falls before # the pack time, we can't honor the MVCC request. # Note: history() returns the most recent record first. # TODO: The filter argument to history() only appears to be # supported by FileStorage. Perhaps it shouldn't be used. L = self.history(oid, "", n, lambda d: not d["version"]) if not L: return for d in L: if d["serial"] < tid: start_time = d["serial"] break else: end_time = d["serial"] if len(L) < n: break n *= 2 if start_time is None: return None data = self.loadSerial(oid, start_time) return data, start_time, end_time def getExtensionMethods(self): """getExtensionMethods This returns a dictionary whose keys are names of extra methods provided by this storage. Storage proxies (such as ZEO) should call this method to determine the extra methods that they need to proxy in addition to the standard storage methods. Dictionary values should be None; this will be a handy place for extra marshalling information, should we need it """ return {} def copyTransactionsFrom(self, other, verbose=0): """Copy transactions from another storage. This is typically used for converting data from one storage to another. `other` must have an .iterator() method. """ _ts=None ok=1 preindex={}; preget=preindex.get # waaaa # restore() is a new storage API method which has an identical # signature to store() except that it does not return anything. # Semantically, restore() is also identical to store() except that it # doesn't do the ConflictError or VersionLockError consistency # checks. The reason to use restore() over store() in this method is # that store() cannot be used to copy transactions spanning a version # commit or abort, or over transactional undos. # # We'll use restore() if it's available, otherwise we'll fall back to # using store(). However, if we use store, then # copyTransactionsFrom() may fail with VersionLockError or # ConflictError. restoring = hasattr(self, 'restore') fiter = other.iterator() for transaction in fiter: tid=transaction.tid if _ts is None: _ts=TimeStamp(tid) else: t=TimeStamp(tid) if t <= _ts: if ok: print ('Time stamps out of order %s, %s' % (_ts, t)) ok=0 _ts=t.laterThan(_ts) tid=`_ts` else: _ts = t if not ok: print ('Time stamps back in order %s' % (t)) ok=1 if verbose: print _ts self.tpc_begin(transaction, tid, transaction.status) for r in transaction: oid=r.oid if verbose: print oid_repr(oid), r.version, len(r.data) if restoring: self.restore(oid, r.tid, r.data, r.version, r.data_txn, transaction) else: pre=preget(oid, None) s=self.store(oid, pre, r.data, r.version, transaction) preindex[oid]=s self.tpc_vote(transaction) self.tpc_finish(transaction) fiter.close() class TransactionRecord: """Abstract base class for iterator protocol""" class DataRecord: """Abstract base class for iterator protocol""" Zope-2.8.7-final/lib/python/ZODB/fstools.py0000664000076400007640000001223210435502423017426 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tools for using FileStorage data files. TODO: This module needs tests. Caution: This file needs to be kept in sync with FileStorage.py. """ import cPickle import struct from ZODB.FileStorage.format import TRANS_HDR, DATA_HDR, TRANS_HDR_LEN, \ DATA_HDR_LEN, DATA_VERSION_HDR_LEN from ZODB.utils import u64 from persistent.TimeStamp import TimeStamp class TxnHeader: """Object representing a transaction record header. Attribute Position Value --------- -------- ----- tid 0- 8 transaction id length 8-16 length of entire transaction record - 8 status 16-17 status of transaction (' ', 'u', 'p'?) user_len 17-19 length of user field (pack code H) descr_len 19-21 length of description field (pack code H) ext_len 21-23 length of extensions (pack code H) """ def __init__(self, file, pos): self._file = file self._pos = pos self._read_header() def _read_header(self): self._file.seek(self._pos) self._hdr = self._file.read(TRANS_HDR_LEN) (self.tid, self.length, self.status, self.user_len, self.descr_len, self.ext_len) = struct.unpack(TRANS_HDR, self._hdr) def read_meta(self): """Load user, descr, and ext attributes.""" self.user = "" self.descr = "" self.ext = {} if not (self.user_len or self.descr_len or self.ext_len): return self._file.seek(self._pos + TRANS_HDR_LEN) if self.user_len: self.user = self._file.read(self.user_len) if self.descr_len: self.descr = self._file.read(self.descr_len) if self.ext_len: self._ext = self._file.read(self.ext_len) self.ext = cPickle.loads(self._ext) def get_data_offset(self): return (self._pos + TRANS_HDR_LEN + self.user_len + self.descr_len + self.ext_len) def get_timestamp(self): return TimeStamp(self.tid) def get_raw_data(self): data_off = self.get_data_offset() data_len = self.length - (data_off - self._pos) self._file.seek(data_off) return self._file.read(data_len) def next_txn(self): off = self._pos + self.length + 8 self._file.seek(off) s = self._file.read(8) if not s: return None return TxnHeader(self._file, off) def prev_txn(self): if self._pos == 4: return None self._file.seek(self._pos - 8) tlen = u64(self._file.read(8)) return TxnHeader(self._file, self._pos - (tlen + 8)) class DataHeader: """Object representing a data record header. Attribute Position Value --------- -------- ----- oid 0- 8 object id serial 8-16 object serial numver prev_rec_pos 16-24 position of previous data record for object txn_pos 24-32 position of txn header version_len 32-34 length of version data_len 34-42 length of data nonversion_pos 42-50* position of nonversion data record prev_version_pos 50-58* pos of previous version data record * these attributes are only present if version_len != 0. """ def __init__(self, file, pos): self._file = file self._pos = pos self._read_header() def _read_header(self): self._file.seek(self._pos) self._hdr = self._file.read(DATA_VERSION_HDR_LEN) # always read the longer header, just in case (self.oid, self.serial, prev_rec_pos, txn_pos, self.version_len, data_len) = struct.unpack(DATA_HDR, self._hdr[:DATA_HDR_LEN]) self.prev_rec_pos = u64(prev_rec_pos) self.txn_pos = u64(txn_pos) self.data_len = u64(data_len) if self.version_len: s = self._hdr[DATA_HDR_LEN:] self.nonversion_pos = u64(s[:8]) self.prev_version_pos = u64(s[8:]) else: self.nonversion_pos = None self.prev_version_pos = None def next_offset(self): """Return offset of next record.""" off = self._pos + self.data_len if self.version_len: off += self.version_len + DATA_VERSION_HDR_LEN else: off += DATA_HDR_LEN if self.data_len == 0: off += 8 # backpointer return off def prev_txn(f): """Return transaction located before current file position.""" f.seek(-8, 1) tlen = u64(f.read(8)) + 8 return TxnHeader(f, f.tell() - tlen) Zope-2.8.7-final/lib/python/ZODB/component.xml0000664000076400007640000001575510435502423020124 0ustar ajungajung Path name to the main storage file. The names for supplemental files, including index and lock files, will be computed from this. Flag that indicates whether the storage should be truncated if it already exists. If true, only reads may be executed against the storage. Note that the "pack" operation is not considered a write operation and is still allowed on a read-only filestorage. Maximum allowed size of the storage file. Operations which would cause the size of the storage to exceed the quota will result in a ZODB.FileStorage.FileStorageQuotaError being raised. The name of the storage that the client wants to use. If the ZEO server serves more than one storage, the client selects the storage it wants to use by name. The default name is '1', which is also the default name for the ZEO server. The maximum size of the client cache, in bytes, KB or MB. The storage name. If unspecified, the address of the server will be used as the name. Enables persistent cache files. The string passed here is used to construct the cache filenames. If it is not specified, the client creates a temporary cache that will only be used by the current object. The directory where persistent cache files are stored. By default cache files, if they are persistent, are stored in the current directory. The minimum delay in seconds between attempts to connect to the server, in seconds. Defaults to 5 seconds. The maximum delay in seconds between attempts to connect to the server, in seconds. Defaults to 300 seconds. A boolean indicating whether the constructor should wait for the client to connect to the server and verify the cache before returning. The default is true. A flag indicating whether this should be a read-only storage, defaulting to false (i.e. writing is allowed by default). A flag indicating whether a read-only remote storage should be acceptable as a fallback when no writable storages are available. Defaults to false. At most one of read_only and read_only_fallback should be true. The authentication realm of the server. Some authentication schemes use a realm to identify the logic set of usernames that are accepted by this server.
Target size, in number of objects, of each connection's object cache. The expected maximum number of simultaneously open connections. There is no hard limit (as many connections as are requested will be opened, until system resources are exhausted). Exceeding pool-size connections causes a warning message to be logged, and exceeding twice pool-size connections causes a critical message to be logged. The expected maximum number of connections simultaneously open per version. Target size, in number of objects, of each version connection's object cache. Zope-2.8.7-final/lib/python/ZODB/loglevels.py0000664000076400007640000000336110435502423017734 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Supplies custom logging levels BLATHER and TRACE. $Revision: 1.1 $ """ import logging __all__ = ["BLATHER", "TRACE"] # In the days of zLOG, there were 7 standard log levels, and ZODB/ZEO used # all of them. Here's how they map to the logging package's 5 standard # levels: # # zLOG logging # ------------- --------------- # PANIC (300) FATAL, CRITICAL (50) # ERROR (200) ERROR (40) # WARNING, PROBLEM (100) WARN (30) # INFO (0) INFO (20) # BLATHER (-100) none -- defined here as BLATHER (15) # DEBUG (-200) DEBUG (10) # TRACE (-300) none -- defined here as TRACE (5) # # TRACE is used by ZEO for extremely verbose trace output, enabled only # when chasing bottom-level communications bugs. It really should be at # a lower level than DEBUG. # # BLATHER is a harder call, and various instances could probably be folded # into INFO or DEBUG without real harm. BLATHER = 15 TRACE = 5 logging.addLevelName("BLATHER", BLATHER) logging.addLevelName("TRACE", TRACE) Zope-2.8.7-final/lib/python/OFS/0000775000076400007640000000000010435610645015262 5ustar ajungajungZope-2.8.7-final/lib/python/OFS/SimpleItem.py0000664000076400007640000003205310435500327017702 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """This module implements a simple item mix-in for objects that have a very simple (e.g. one-screen) management interface, like documents, Aqueduct database adapters, etc. This module can also be used as a simple template for implementing new item types. $Id: SimpleItem.py 40476 2005-12-02 15:44:31Z chrisw $ """ import marshal, re, sys, time import Globals, App.Management, Acquisition, App.Undo import AccessControl.Role, AccessControl.Owned, App.Common from webdav.Resource import Resource from ExtensionClass import Base from ComputedAttribute import ComputedAttribute from AccessControl import getSecurityManager, Unauthorized from AccessControl.ZopeSecurityPolicy import getRoles from Acquisition import aq_base, aq_parent, aq_inner, aq_acquire from DocumentTemplate.html_quote import html_quote from DocumentTemplate.ustr import ustr from zExceptions.ExceptionFormatter import format_exception from zExceptions import Redirect from CopySupport import CopySource from Traversable import Traversable import ZDOM HTML=Globals.HTML import logging logger = logging.getLogger() class Item(Base, Resource, CopySource, App.Management.Tabs, Traversable, ZDOM.Element, AccessControl.Owned.Owned, App.Undo.UndoSupport, ): """A common base class for simple, non-container objects.""" isPrincipiaFolderish=0 isTopLevelPrincipiaApplicationObject=0 def manage_afterAdd(self, item, container): pass def manage_beforeDelete(self, item, container): pass def manage_afterClone(self, item): pass # Direct use of the 'id' attribute is deprecated - use getId() id='' getId__roles__=None def getId(self): """Return the id of the object as a string. This method should be used in preference to accessing an id attribute of an object directly. The getId method is public. """ name=getattr(self, 'id', None) if callable(name): return name() if name is not None: return name if hasattr(self, '__name__'): return self.__name__ raise AttributeError, 'This object has no id' # Alias id to __name__, which will make tracebacks a good bit nicer: __name__=ComputedAttribute(lambda self: self.getId()) # Name, relative to SOFTWARE_URL of icon used to display item # in folder listings. icon='' # Meta type used for selecting all objects of a given type. meta_type='simple item' # Default title. title='' # Default propertysheet info: __propsets__=() manage_options=( App.Undo.UndoSupport.manage_options +AccessControl.Owned.Owned.manage_options ) # Attributes that must be acquired REQUEST=Acquisition.Acquired # Allow (reluctantly) access to unprotected attributes __allow_access_to_unprotected_subobjects__=1 def title_or_id(self): """Return the title if it is not blank and the id otherwise. """ title=self.title if callable(title): title=title() if title: return title return self.getId() def title_and_id(self): """Return the title if it is not blank and the id otherwise. If the title is not blank, then the id is included in parens. """ title=self.title if callable(title): title=title() id = self.getId() return title and ("%s (%s)" % (title,id)) or id def this(self): # Handy way to talk to ourselves in document templates. return self def tpURL(self): # My URL as used by tree tag return self.getId() def tpValues(self): # My sub-objects as used by the tree tag return () _manage_editedDialog=Globals.DTMLFile('dtml/editedDialog', globals()) def manage_editedDialog(self, REQUEST, **args): return apply(self._manage_editedDialog,(self, REQUEST), args) def raise_standardErrorMessage( self, client=None, REQUEST={}, error_type=None, error_value=None, tb=None, error_tb=None, error_message='', tagSearch=re.compile(r'[a-zA-Z]>').search, error_log_url=''): try: if error_type is None: error_type =sys.exc_info()[0] if error_value is None: error_value=sys.exc_info()[1] # allow for a few different traceback options if tb is None and error_tb is None: tb=sys.exc_info()[2] if type(tb) is not type('') and (error_tb is None): error_tb = pretty_tb(error_type, error_value, tb) elif type(tb) is type('') and not error_tb: error_tb = tb # turn error_type into a string if hasattr(error_type, '__name__'): error_type=error_type.__name__ if hasattr(self, '_v_eek'): # Stop if there is recursion. raise error_type, error_value, tb self._v_eek=1 if str(error_type).lower() in ('redirect',): raise error_type, error_value, tb if not error_message: try: s = ustr(error_value) except: s = error_value try: match = tagSearch(s) except TypeError: match = None if match is not None: error_message=error_value if client is None: client=self if not REQUEST: REQUEST=self.aq_acquire('REQUEST') try: if hasattr(client, 'standard_error_message'): s=getattr(client, 'standard_error_message') else: client = client.aq_parent s=getattr(client, 'standard_error_message') kwargs = {'error_type': error_type, 'error_value': error_value, 'error_tb': error_tb, 'error_traceback': error_tb, 'error_message': error_message, 'error_log_url': error_log_url} if getattr(aq_base(s),'isDocTemp',0): v = s(client, REQUEST, **kwargs) elif callable(s): v = s(**kwargs) else: v = HTML.__call__(s, client, REQUEST, **kwargs) except: logger.error( 'Exception while rendering an error message', exc_info=True ) try: strv = str(error_value) except: strv = ('' % str(type(error_value).__name__)) v = strv + ( (" (Also, the following error occurred while attempting " "to render the standard error message, please see the " "event log for full details: %s)")%( html_quote(sys.exc_info()[1]), )) raise error_type, v, tb finally: if hasattr(self, '_v_eek'): del self._v_eek tb=None def manage(self, URL1): """ """ raise Redirect, "%s/manage_main" % URL1 # This keeps simple items from acquiring their parents # objectValues, etc., when used in simple tree tags. def objectValues(self, spec=None): return () objectIds=objectItems=objectValues # FTP support methods def manage_FTPstat(self,REQUEST): """Psuedo stat, used by FTP for directory listings. """ from AccessControl.User import nobody mode=0100000 if (hasattr(aq_base(self),'manage_FTPget')): try: if getSecurityManager().validate( None, self, 'manage_FTPget', self.manage_FTPget): mode=mode | 0440 except Unauthorized: pass if nobody.allowed( self.manage_FTPget, getRoles(self, 'manage_FTPget', self.manage_FTPget, ()), ): mode=mode | 0004 # check write permissions if hasattr(aq_base(self),'PUT'): try: if getSecurityManager().validate(None, self, 'PUT', self.PUT): mode=mode | 0220 except Unauthorized: pass if nobody.allowed( self.PUT, getRoles(self, 'PUT', self.PUT, ()), ): mode=mode | 0002 # get size if hasattr(aq_base(self), 'get_size'): size=self.get_size() elif hasattr(aq_base(self),'manage_FTPget'): size=len(self.manage_FTPget()) else: size=0 # get modification time if hasattr(aq_base(self), 'bobobase_modification_time'): mtime=self.bobobase_modification_time().timeTime() else: mtime=time.time() # get owner and group owner=group='Zope' if hasattr(aq_base(self), 'get_local_roles'): for user, roles in self.get_local_roles(): if 'Owner' in roles: owner=user break return marshal.dumps((mode,0,0,1,owner,group,size,mtime,mtime,mtime)) def manage_FTPlist(self,REQUEST): """Directory listing for FTP. In the case of non-Foldoid objects, the listing should contain one object, the object itself. """ # check to see if we are being acquiring or not ob=self while 1: if App.Common.is_acquired(ob): raise ValueError('FTP List not supported on acquired objects') if not hasattr(ob,'aq_parent'): break ob=ob.aq_parent stat=marshal.loads(self.manage_FTPstat(REQUEST)) id = self.getId() return marshal.dumps((id,stat)) def __len__(self): return 1 def __repr__(self): """Show the physical path of the object and its context if available. """ try: path = '/'.join(self.getPhysicalPath()) except: return Base.__repr__(self) context_path = None context = aq_parent(self) container = aq_parent(aq_inner(self)) if aq_base(context) is not aq_base(container): try: context_path = '/'.join(context.getPhysicalPath()) except: context_path = None res = '<%s' % self.__class__.__name__ res += ' at %s' % path if context_path: res += ' used for %s' % context_path res += '>' return res Globals.default__class_init__(Item) class Item_w__name__(Item): """Mixin class to support common name/id functions""" def getId(self): """Return the id of the object as a string. """ return self.__name__ def title_or_id(self): """Return the title if it is not blank and the id otherwise. """ return self.title or self.__name__ def title_and_id(self): """Return the title if it is not blank and the id otherwise. If the title is not blank, then the id is included in parens. """ t=self.title return t and ("%s (%s)" % (t,self.__name__)) or self.__name__ def _setId(self, id): self.__name__=id def getPhysicalPath(self): """Get the physical path of the object. Returns a path (an immutable sequence of strings) that can be used to access this object again later, for example in a copy/paste operation. getPhysicalRoot() and getPhysicalPath() are designed to operate together. """ path = (self.__name__,) p = aq_parent(aq_inner(self)) if p is not None: path = p.getPhysicalPath() + path return path def pretty_tb(t, v, tb, as_html=1): tb = format_exception(t, v, tb, as_html=as_html) tb = '\n'.join(tb) return tb class SimpleItem(Item, Globals.Persistent, Acquisition.Implicit, AccessControl.Role.RoleManager, ): # Blue-plate special, Zope Masala """Mix-in class combining the most common set of basic mix-ins """ manage_options=Item.manage_options+( {'label':'Security', 'action':'manage_access', 'help':('OFSP', 'Security.stx')}, ) __ac_permissions__=(('View', ()),) Zope-2.8.7-final/lib/python/OFS/Uninstalled.py0000664000076400007640000000555010435500327020116 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Objects for packages that have been uninstalled. """ import SimpleItem, Globals, Acquisition from Acquisition import Acquired import Persistence from thread import allocate_lock from zLOG import LOG, WARNING from cgi import escape broken_klasses={} broken_klasses_lock = allocate_lock() class BrokenClass(Acquisition.Explicit, SimpleItem.Item, Persistence.Overridable): _p_changed=0 meta_type='Broken Because Product is Gone' icon='p_/broken' product_name='unknown' id='broken' manage_page_header = Acquired manage_page_footer = Acquired def __getstate__(self): raise SystemError, ( """This object was originally created by a product that is no longer installed. It cannot be updated. """) def __getattr__(self, name): if name[:3]=='_p_': return BrokenClass.inheritedAttribute('__getattr__')(self, name) raise AttributeError, escape(name) manage=Globals.DTMLFile('dtml/brokenEdit',globals()) manage_main=Globals.DTMLFile('dtml/brokenEdit',globals()) manage_workspace=Globals.DTMLFile('dtml/brokenEdit',globals()) def Broken(self, oid, pair): broken_klasses_lock.acquire() try: if broken_klasses.has_key(pair): klass = broken_klasses[pair] else: module, klassname = pair d={'BrokenClass': BrokenClass} exec ("class %s(BrokenClass): ' '; __module__=%s" % (klassname, `module`)) in d klass = broken_klasses[pair] = d[klassname] module=module.split('.') if len(module) > 2 and module[0]=='Products': klass.product_name= module[1] klass.title=( 'This object from the %s product ' 'is broken!' % klass.product_name) klass.info=( 'This object\'s class was %s in module %s.' % (klass.__name__, klass.__module__)) LOG('ZODB', WARNING, 'Could not import class %s ' 'from module %s' % (`klass.__name__`, `klass.__module__`)) finally: broken_klasses_lock.release() if oid is None: return klass i=klass() i._p_oid=oid i._p_jar=self return i Zope-2.8.7-final/lib/python/OFS/FindSupport.py0000664000076400007640000002413410435500327020110 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Find support $Id: FindSupport.py 25263 2004-06-04 18:58:49Z slinkp $ """ import sys, os, time, Globals, ExtensionClass from DocumentTemplate.DT_Util import Eval from AccessControl.Permission import name_trans from Globals import DTMLFile from DocumentTemplate.DT_Util import InstanceDict, TemplateDict from DateTime import DateTime from string import translate from AccessControl.DTML import RestrictedDTML from AccessControl import ClassSecurityInfo class FindSupport(ExtensionClass.Base): """Find support for Zope Folders""" #findframe is deprecated manage_findFrame=DTMLFile('dtml/findFrame', globals()) manage_findForm=DTMLFile('dtml/findForm', globals(), management_view='Find') manage_findAdv=DTMLFile('dtml/findAdv', globals(), management_view='Find', help_topic='Find_Advanced.stx', help_product='OFSP') manage_findResult=DTMLFile('dtml/findResult', globals(), management_view='Find') __ac_permissions__=( ('View management screens', ('manage_findFrame', 'manage_findForm', 'manage_findAdv', 'manage_findResult')), ) manage_options=( {'label':'Find', 'action':'manage_findForm', 'help':('OFSP','Find.stx')}, ) security = ClassSecurityInfo() security.declareProtected('View management screens', 'ZopeFind') def ZopeFind(self, obj, obj_ids=None, obj_metatypes=None, obj_searchterm=None, obj_expr=None, obj_mtime=None, obj_mspec=None, obj_permission=None, obj_roles=None, search_sub=0, REQUEST=None, result=None, pre=''): """Zope Find interface""" if result is None: result=[] if obj_metatypes and 'all' in obj_metatypes: obj_metatypes=None if obj_mtime and type(obj_mtime)==type('s'): obj_mtime=DateTime(obj_mtime).timeTime() if obj_permission: obj_permission=p_name(obj_permission) if obj_roles and type(obj_roles) is type('s'): obj_roles=[obj_roles] if obj_expr: # Setup expr machinations md=td() obj_expr=(Eval(obj_expr), md, md._push, md._pop) base=obj if hasattr(obj, 'aq_base'): base=obj.aq_base if hasattr(base, 'objectItems'): try: items=obj.objectItems() except: return result else: if getattr(base, 'meta_type', None) == 'Z Class': try: items=obj.propertysheets.methods.objectItems() except: return result else: return result try: add_result=result.append except: raise AttributeError, `result` for id, ob in items: if pre: p="%s/%s" % (pre, id) else: p=id dflag=0 if hasattr(ob, '_p_changed') and (ob._p_changed == None): dflag=1 if hasattr(ob, 'aq_base'): bs=ob.aq_base else: bs=ob if ( (not obj_ids or absattr(bs.getId()) in obj_ids) and (not obj_metatypes or (hasattr(bs, 'meta_type') and bs.meta_type in obj_metatypes)) and (not obj_searchterm or (hasattr(ob, 'PrincipiaSearchSource') and ob.PrincipiaSearchSource().find(str(obj_searchterm)) >= 0 ) or (hasattr(ob, 'SearchableText') and ob.SearchableText().find(str(obj_searchterm)) >= 0) ) and (not obj_expr or expr_match(ob, obj_expr)) and (not obj_mtime or mtime_match(ob, obj_mtime, obj_mspec)) and ( (not obj_permission or not obj_roles) or \ role_match(ob, obj_permission, obj_roles) ) ): add_result((p, ob)) dflag=0 is_zclass = getattr(bs, 'meta_type', None) == 'Z Class' if search_sub and (hasattr(bs, 'objectItems') or is_zclass): if is_zclass: subob = ob.propertysheets.methods sub_p = '%s/propertysheets/methods' % p else: subob = ob sub_p = p self.ZopeFind(subob, obj_ids, obj_metatypes, obj_searchterm, obj_expr, obj_mtime, obj_mspec, obj_permission, obj_roles, search_sub, REQUEST, result, sub_p) if dflag: ob._p_deactivate() return result security.declareProtected('View management screens', 'PrincipiaFind') PrincipiaFind=ZopeFind security.declareProtected('View management screens', 'ZopeFindAndApply') def ZopeFindAndApply(self, obj, obj_ids=None, obj_metatypes=None, obj_searchterm=None, obj_expr=None, obj_mtime=None, obj_mspec=None, obj_permission=None, obj_roles=None, search_sub=0, REQUEST=None, result=None, pre='', apply_func=None, apply_path=''): """Zope Find interface and apply""" if result is None: result=[] if obj_metatypes and 'all' in obj_metatypes: obj_metatypes=None if obj_mtime and type(obj_mtime)==type('s'): obj_mtime=DateTime(obj_mtime).timeTime() if obj_permission: obj_permission=p_name(obj_permission) if obj_roles and type(obj_roles) is type('s'): obj_roles=[obj_roles] if obj_expr: # Setup expr machinations md=td() obj_expr=(Eval(obj_expr), md, md._push, md._pop) base=obj if hasattr(obj, 'aq_base'): base=obj.aq_base if not hasattr(base, 'objectItems'): return result try: items=obj.objectItems() except: return result try: add_result=result.append except: raise AttributeError, `result` for id, ob in items: if pre: p="%s/%s" % (pre, id) else: p=id dflag=0 if hasattr(ob, '_p_changed') and (ob._p_changed == None): dflag=1 if hasattr(ob, 'aq_base'): bs=ob.aq_base else: bs=ob if ( (not obj_ids or absattr(bs.getId()) in obj_ids) and (not obj_metatypes or (hasattr(bs, 'meta_type') and bs.meta_type in obj_metatypes)) and (not obj_searchterm or (hasattr(ob, 'PrincipiaSearchSource') and ob.PrincipiaSearchSource().find(obj_searchterm) >= 0 )) and (not obj_expr or expr_match(ob, obj_expr)) and (not obj_mtime or mtime_match(ob, obj_mtime, obj_mspec)) and ( (not obj_permission or not obj_roles) or \ role_match(ob, obj_permission, obj_roles) ) ): if apply_func: apply_func(ob, (apply_path+'/'+p)) else: add_result((p, ob)) dflag=0 if search_sub and hasattr(bs, 'objectItems'): self.ZopeFindAndApply(ob, obj_ids, obj_metatypes, obj_searchterm, obj_expr, obj_mtime, obj_mspec, obj_permission, obj_roles, search_sub, REQUEST, result, p, apply_func, apply_path) if dflag: ob._p_deactivate() return result class td(RestrictedDTML, TemplateDict): pass def expr_match(ob, ed, c=InstanceDict, r=0): e, md, push, pop=ed push(c(ob, md)) try: r=e.eval(md) finally: pop() return r def mtime_match(ob, t, q, fn=hasattr): if not fn(ob, '_p_mtime'): return 0 return q=='<' and (ob._p_mtime < t) or (ob._p_mtime > t) def role_match(ob, permission, roles, lt=type([]), tt=type(())): pr=[] fn=pr.append while 1: if hasattr(ob, permission): p=getattr(ob, permission) if type(p) is lt: map(fn, p) if hasattr(ob, 'aq_parent'): ob=ob.aq_parent continue break if type(p) is tt: map(fn, p) break if p is None: map(fn, ('Manager', 'Anonymous')) break if hasattr(ob, 'aq_parent'): ob=ob.aq_parent continue break for role in roles: if not (role in pr): return 0 return 1 Globals.InitializeClass(FindSupport) # Helper functions def absattr(attr): if callable(attr): return attr() return attr def p_name(name): return '_' + translate(name, name_trans) + '_Permission' Zope-2.8.7-final/lib/python/OFS/ZDOM.py0000664000076400007640000004045310435500327016406 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ DOM implementation in ZOPE : Read-Only methods All standard Zope objects support DOM to a limited extent. """ import Acquisition # Node type codes # --------------- ELEMENT_NODE = 1 ATTRIBUTE_NODE = 2 TEXT_NODE = 3 CDATA_SECTION_NODE = 4 ENTITY_REFERENCE_NODE = 5 ENTITY_NODE = 6 PROCESSING_INSTRUCTION_NODE = 7 COMMENT_NODE = 8 DOCUMENT_NODE = 9 DOCUMENT_TYPE_NODE = 10 DOCUMENT_FRAGMENT_NODE = 11 NOTATION_NODE = 12 # Exception codes # --------------- INDEX_SIZE_ERR = 1 DOMSTRING_SIZE_ERR = 2 HIERARCHY_REQUEST_ERR = 3 WRONG_DOCUMENT_ERR = 4 INVALID_CHARACTER_ERR = 5 NO_DATA_ALLOWED_ERR = 6 NO_MODIFICATION_ALLOWED_ERR = 7 NOT_FOUND_ERR = 8 NOT_SUPPORTED_ERR = 9 INUSE_ATTRIBUTE_ERR = 10 # Exceptions # ---------- class DOMException(Exception): pass class IndexSizeException(DOMException): code = INDEX_SIZE_ERR class DOMStringSizeException(DOMException): code = DOMSTRING_SIZE_ERR class HierarchyRequestException(DOMException): code = HIERARCHY_REQUEST_ERR class WrongDocumentException(DOMException): code = WRONG_DOCUMENT_ERR class InvalidCharacterException(DOMException): code = INVALID_CHARACTER_ERR class NoDataAllowedException(DOMException): code = NO_DATA_ALLOWED_ERR class NoModificationAllowedException(DOMException): code = NO_MODIFICATION_ALLOWED_ERR class NotFoundException(DOMException): code = NOT_FOUND_ERR class NotSupportedException(DOMException): code = NOT_SUPPORTED_ERR class InUseAttributeException(DOMException): code = INUSE_ATTRIBUTE_ERR # Node classes # ------------ class Node: """ Node Interface """ __ac_permissions__=( ('Access contents information', ('getNodeName', 'getNodeValue', 'getParentNode', 'getChildNodes', 'getFirstChild', 'getLastChild', 'getPreviousSibling', 'getNextSibling', 'getOwnerDocument', 'getAttributes', 'hasChildNodes'), ), ) # DOM attributes # -------------- def getNodeName(self): """The name of this node, depending on its type""" return None def getNodeValue(self): """The value of this node, depending on its type""" return None def getParentNode(self): """The parent of this node. All nodes except Document DocumentFragment and Attr may have a parent""" return None def getChildNodes(self): """Returns a NodeList that contains all children of this node. If there are no children, this is a empty NodeList""" return NodeList() def getFirstChild(self): """The first child of this node. If there is no such node this returns None.""" return None def getLastChild(self): """The last child of this node. If there is no such node this returns None.""" return None def getPreviousSibling(self): """The node immediately preceding this node. If there is no such node, this returns None.""" return None def getNextSibling(self): """The node immediately preceding this node. If there is no such node, this returns None.""" return None def getAttributes(self): """Returns a NamedNodeMap containing the attributes of this node (if it is an element) or None otherwise.""" return None def getOwnerDocument(self): """The Document object associated with this node. When this is a document this is None""" node = self if hasattr(node, 'aq_parent'): node = self.aq_parent return node.getOwnerDocument() return node # DOM Methods # ----------- def hasChildNodes(self): """Returns true if the node has any children, false if it doesn't. """ return len(self.objectIds()) class Document(Acquisition.Explicit, Node): """ Document Interface """ __ac_permissions__=( ('Access contents information', ('getImplementation', 'getDoctype', 'getDocumentElement'), ), ) # Document Methods # ---------------- def getImplementation(self): """ The DOMImplementation object that handles this document. """ return DOMImplementation() def getDoctype(self): """ The Document Type Declaration associated with this document. For HTML documents as well as XML documents without a document type declaration this returns null. """ return None def getDocumentElement(self): """ This is a convenience attribute that allows direct access to the child node that is the root element of the document. """ return self.aq_parent # Node Methods # ------------ def getNodeName(self): """The name of this node, depending on its type""" return '#document' def getNodeType(self): """A code representing the type of the node.""" return DOCUMENT_NODE def getOwnerDocument(self): """The Document object associated with this node. When this is a document this is None""" return self def getChildNodes(self): """Returns a NodeList that contains all children of this node. If there are no children, this is a empty NodeList""" return NodeList([self.aq_parent]) def getFirstChild(self): """The first child of this node. If there is no such node this returns None.""" return self.aq_parent def getLastChild(self): """The last child of this node. If there is no such node this returns None.""" return self.aq_parent def hasChildNodes(self): """Returns true if the node has any children, false if it doesn't. """ return 1 class DOMImplementation: """ DOMImplementation Interface """ __ac_permissions__=( ('Access contents information', ('hasFeature',), ), ) def hasFeature(self, feature, version = None): """ hasFeature - Test if the DOM implementation implements a specific feature. Parameters: feature The package name of the feature to test. In Level 1, the legal values are "HTML" and "XML" (case-insensitive). version This is the version number of the package name to test. In Level 1, this is the string "1.0". If the version is not specified, supporting any version of the feature will cause the method to return true. Return Value true if the feature is implemented in the specified version, false otherwise. """ feature=feature.lower() if feature == 'html': return 0 if feature == 'xml': if version is None: return 1 if version == '1.0': return 1 return 0 class Element(Node): """ Element interface """ __ac_permissions__=( ('Access contents information', ('getTagName', 'getAttribute', 'getAttributeNode', 'getElementsByTagName'), ), ) # Element Attributes # ------------------ def getTagName(self): """The name of the element""" return self.__class__.__name__ # Node Attributes # --------------- def getNodeName(self): """The name of this node, depending on its type""" return self.getTagName() def getNodeType(self): """A code representing the type of the node.""" return ELEMENT_NODE def getParentNode(self): """The parent of this node. All nodes except Document DocumentFragment and Attr may have a parent""" return getattr(self, 'aq_parent', None) def getChildNodes(self): """Returns a NodeList that contains all children of this node. If there are no children, this is a empty NodeList""" return NodeList(self.objectValues()) def getFirstChild(self): """The first child of this node. If there is no such node this returns None""" children = self.getChildNodes() if children: return children._data[0] return None def getLastChild(self): """The last child of this node. If there is no such node this returns None.""" children = self.getChildNodes() if children: return children._data[-1] return None def getPreviousSibling(self): """The node immediately preceding this node. If there is no such node, this returns None.""" if hasattr(self, 'aq_parent'): parent = self.aq_parent ids=list(parent.objectIds()) id=self.id if type(id) is not type(''): id=id() try: index=ids.index(id) except: return None if index < 1: return None return parent.objectValues()[index-1] return None def getNextSibling(self): """The node immediately preceding this node. If there is no such node, this returns None.""" if hasattr(self, 'aq_parent'): parent = self.aq_parent ids=list(parent.objectIds()) id=self.id if type(id) is not type(''): id=id() try: index=ids.index(id) except: return None if index >= len(ids)-1: return None return parent.objectValues()[index+1] return None # Element Methods # --------------- def getAttribute(self, name): """Retrieves an attribute value by name.""" return None def getAttributeNode(self, name): """ Retrieves an Attr node by name or None if there is no such attribute. """ return None def getElementsByTagName(self, tagname): """ Returns a NodeList of all the Elements with a given tag name in the order in which they would be encountered in a preorder traversal of the Document tree. Parameter: tagname The name of the tag to match (* = all tags). Return Value: A new NodeList object containing all the matched Elements. """ nodeList = [] for child in self.objectValues(): if (child.getNodeType()==ELEMENT_NODE and \ child.getTagName()==tagname or tagname== '*'): nodeList.append(child) if hasattr(child, 'getElementsByTagName'): n1 = child.getElementsByTagName(tagname) nodeList = nodeList + n1._data return NodeList(nodeList) class ElementWithAttributes(Element): """ Elements that allow DOM access to Zope properties of type 'string'. Note: This sub-class should only be used by PropertyManagers """ def getAttributes(self): """Returns a NamedNodeMap containing the attributes of this node (if it is an element) or None otherwise.""" attribs={} for p in self._properties: if p['type'] == 'string': name=p['id'] attrib=Attr(name, self.getProperty(name,'')).__of__(self) attribs[name]=attrib return NamedNodeMap(attribs) def getAttribute(self, name): """Retrieves an attribute value by name.""" if self.getPropertyType(name) == 'string': return self.getProperty(name,'') def getAttributeNode(self, name): """Retrieves an Attr node by name or None if there is no such attribute. """ if self.getPropertyType(name) == 'string': return Attr(name, self.getProperty(name,'')).__of__(self) return None class ElementWithTitle(Element): """ Elements that allow DOM access to Zope 'title' property. Note: Don't use this sub-class for PropertyManagers """ def getAttributes(self): """Returns a NamedNodeMap containing the attributes of this node (if it is an element) or None otherwise.""" title = self.getAttributeNode('title') if title is not None: return NamedNodeMap({'title':title}) return NamedNodeMap() def getAttribute(self, name): """Retrieves an attribute value by name.""" if name=='title' and hasattr(self.aq_base, 'title'): return self.title return '' def getAttributeNode(self, name): """Retrieves an Attr node by name or None if there is no such attribute. """ value=self.getAttribute(name) if value: return Attr(name, value).__of__(self) return None class Root(ElementWithAttributes): """ The top-level Zope object. """ def getOwnerDocument(self): """ """ return Document().__of__(self) class NodeList: """NodeList interface - Provides the abstraction of an ordered collection of nodes. Python extensions: can use sequence-style 'len', 'getitem', and 'for..in' constructs. """ # The security machinery is not willing to treat this like a # list just because we act like one. We need to assert that # its ok to allow access to items in the nodelist. __allow_access_to_unprotected_subobjects__=1 def __init__(self,list=None): self._data = list or [] def __getitem__(self, index): return self._data[index] def item(self, index): """Returns the index-th item in the collection""" try: return self._data[index] except IndexError: return None def getLength(self): """The length of the NodeList""" return len(self._data) __len__=getLength class NamedNodeMap: """ NamedNodeMap interface - Is used to represent collections of nodes that can be accessed by name. NamedNodeMaps are not maintained in any particular order. Python extensions: can use sequence-style 'len', 'getitem', and 'for..in' constructs, and mapping-style 'getitem'. """ # Tell the security machinery to allow access to items. __allow_access_to_unprotected_subobjects__=1 def __init__(self, data=None): if data is None : data = {} self._data = data def item(self, index): """Returns the index-th item in the map""" try: return self._data.values()[index] except IndexError: return None def __getitem__(self, key): if type(key)==type(1): return self._data.values()[key] else: return self._data[key] def getLength(self): """The length of the NodeList""" return len(self._data) __len__ = getLength def getNamedItem(self, name): """Retrieves a node specified by name. Parameters: name Name of a node to retrieve. Return Value A Node (of any type) with the specified name, or None if the specified name did not identify any node in the map. """ if self._data.has_key(name): return self._data[name] return None class Attr(Acquisition.Implicit, Node): """ Attr interface - The Attr interface represents an attriubte in an Element object. Attr objects inherit the Node Interface """ def __init__(self, name, value): self.name = name self.value = value self.specified = 1 def getNodeName(self): """The name of this node, depending on its type""" return self.name def getName(self): """Returns the name of this attribute.""" return self.name def getNodeValue(self): """The value of this node, depending on its type""" return self.value def getNodeType(self): """A code representing the type of the node.""" return ATTRIBUTE_NODE def getSpecified(self): """If this attribute was explicitly given a value in the original document, this is true; otherwise, it is false.""" return self.specified Zope-2.8.7-final/lib/python/OFS/Folder.py0000664000076400007640000000665310435500327017054 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Folder object Folders are the basic container objects and are analogous to directories. $Id: Folder.py 24764 2004-05-17 06:13:48Z philikon $ """ import Globals, SimpleItem, ObjectManager, PropertyManager import AccessControl.Role, webdav.Collection, FindSupport from webdav.WriteLockInterface import WriteLockInterface from AccessControl import getSecurityManager from AccessControl import Unauthorized from AccessControl.Permissions import add_page_templates from AccessControl.Permissions import add_user_folders from Globals import DTMLFile manage_addFolderForm=DTMLFile('dtml/folderAdd', globals()) def manage_addFolder(self, id, title='', createPublic=0, createUserF=0, REQUEST=None): """Add a new Folder object with id *id*. If the 'createPublic' and 'createUserF' parameters are set to any true value, an 'index_html' and a 'UserFolder' objects are created respectively in the new folder. """ ob = Folder(id) ob.title = title self._setObject(id, ob) ob = self._getOb(id) checkPermission=getSecurityManager().checkPermission if createUserF: if not checkPermission(add_user_folders, ob): raise Unauthorized, ( 'You are not authorized to add User Folders.' ) ob.manage_addUserFolder() if createPublic: if not checkPermission(add_page_templates, ob): raise Unauthorized, ( 'You are not authorized to add Page Templates.' ) ob.manage_addProduct['PageTemplates'].manage_addPageTemplate( id='index_html', title='') if REQUEST is not None: return self.manage_main(self, REQUEST, update_menu=1) class Folder( ObjectManager.ObjectManager, PropertyManager.PropertyManager, AccessControl.Role.RoleManager, webdav.Collection.Collection, SimpleItem.Item, FindSupport.FindSupport, ): """ Folders are basic container objects that provide a standard interface for object management. Folder objects also implement a management interface and can have arbitrary properties. """ __implements__ = (WriteLockInterface,) meta_type='Folder' _properties=({'id':'title', 'type': 'string','mode':'wd'},) manage_options=( ObjectManager.ObjectManager.manage_options+ ( {'label':'View', 'action':'', 'help':('OFSP','Folder_View.stx')}, )+ PropertyManager.PropertyManager.manage_options+ AccessControl.Role.RoleManager.manage_options+ SimpleItem.Item.manage_options+ FindSupport.FindSupport.manage_options ) __ac_permissions__=() def __init__(self, id=None): if id is not None: self.id = str(id) Globals.default__class_init__(Folder) Zope-2.8.7-final/lib/python/OFS/History.py0000664000076400007640000001724310435500327017277 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Object Histories $Id: History.py 27720 2004-09-30 20:18:25Z andreasjung $ """ import Globals, ExtensionClass, difflib from DateTime import DateTime from Acquisition import Implicit, aq_base from struct import pack, unpack from cgi import escape from zExceptions import Redirect class TemporalParadox(Exception): pass class HistorySelectionError(Exception): pass class HystoryJar: """A ZODB Connection-like object that provides access to data but prevents history from being changed.""" def __init__(self, base): self.__base__=base def __getattr__(self, name): return getattr(self.__base__, name) def commit(self, object, transaction): if object._p_changed: raise TemporalParadox, "You can't change history!" def abort(*args, **kw): pass tpc_begin = tpc_finish = abort def historicalRevision(self, serial): state=self._p_jar.oldstate(self, serial) rev=self.__class__.__basicnew__() rev._p_jar=HystoryJar(self._p_jar) rev._p_oid=self._p_oid rev._p_serial=serial rev.__setstate__(state) rev._p_changed=0 return rev class Historian(Implicit): """An Historian's job is to find hysterical revisions of objects, given a time.""" def __getitem__(self, key): self=self.aq_parent serial=apply(pack, ('>HHHH',)+tuple(map(int, key.split('.')))) if serial == self._p_serial: return self rev=historicalRevision(self, serial) return rev.__of__(self.aq_parent) def manage_workspace(self, REQUEST): "We aren't real, so we delegate to that that spawned us!" raise Redirect, REQUEST['URL2']+'/manage_change_history_page' class Historical(ExtensionClass.Base): """Mix-in class to provide a veiw that shows hystorical changes The display is similar to that used for undo, except that the transactions are limited to those that effect the displayed object and that the interface doesn't provide an undo capability. This interface is generally *only* interesting for objects, such as methods, and documents, that are self-contained, meaning that they don't have persistent sub-objects. """ HistoricalRevisions=Historian() __ac_permissions__=( ('View History', ('manage_change_history_page','manage_change_history', 'manage_historyCompare', 'manage_historicalComparison', ) ), ) manage_options=({'label':'History', 'action':'manage_change_history_page', 'help':('OFSP','History.stx') }, ) manage_change_history_page=Globals.DTMLFile( 'dtml/history', globals(), HistoryBatchSize=20, first_transaction=0, last_transaction=20) def manage_change_history(self): first=0 last=20 request=getattr(self, 'REQUEST', None) if request is not None: first=request.get('first_transaction', first) last=request.get('last_transaction',last) r=self._p_jar.db().history(self._p_oid, None, last) if r is None: # storage doesn't support history return () r=r[first:] for d in r: d['time']=DateTime(d['time']) d['key']='.'.join(map(str, unpack(">HHHH", d['tid']))) return r def manage_beforeHistoryCopy(self): pass # ? (Hook) def manage_historyCopy(self, keys=[], RESPONSE=None, URL1=None): "Copy a selected revision to the present" if not keys: raise HistorySelectionError, ( "No historical revision was selected.

") if len(keys) > 1: raise HistorySelectionError, ( "Only one historical revision can be " "copied to the present.

") key=keys[0] serial=apply(pack, ('>HHHH',)+tuple(map(int, key.split('.')))) if serial != self._p_serial: self.manage_beforeHistoryCopy() state=self._p_jar.oldstate(self, serial) # Scrub the object before restoring the old state base = aq_base(self) base._p_changed=0 base._p_deactivate() base.__setstate__(state) base._p_changed=1 self.manage_afterHistoryCopy() if RESPONSE is not None and URL1 is not None: RESPONSE.redirect(URL1+'/manage_workspace') def manage_afterHistoryCopy(self): pass # ? (Hook) _manage_historyComparePage=Globals.DTMLFile( 'dtml/historyCompare', globals(), management_view='History') def manage_historyCompare(self, rev1, rev2, REQUEST, historyComparisonResults=''): dt1=DateTime(rev1._p_mtime) dt2=DateTime(rev2._p_mtime) return self._manage_historyComparePage( self, REQUEST, dt1=dt1, dt2=dt2, historyComparisonResults=historyComparisonResults) def manage_historicalComparison(self, REQUEST, keys=[]): "Compare two selected revisions" if not keys: raise HistorySelectionError, ( "No historical revision was selected.

") if len(keys) > 2: raise HistorySelectionError, ( "Only two historical revision can be compared

") serial=apply(pack, ('>HHHH',)+tuple(map(int, keys[-1].split('.')))) rev1=historicalRevision(self, serial) if len(keys)==2: serial=apply(pack, ('>HHHH',)+tuple(map(int, keys[0].split('.')))) rev2=historicalRevision(self, serial) else: rev2=self return self.manage_historyCompare(rev1, rev2, REQUEST) Globals.default__class_init__(Historical) def dump(tag, x, lo, hi, r): r1=[] r2=[] for i in xrange(lo, hi): r1.append(tag) r2.append(x[i]) r.append("\n" "

\n%s\n
\n" "
\n%s\n
\n" "\n" % ('\n'.join(r1), escape('\n'.join(r2)))) def replace(x, xlo, xhi, y, ylo, yhi, r): rx1=[] rx2=[] for i in xrange(xlo, xhi): rx1.append('-') rx2.append(x[i]) ry1=[] ry2=[] for i in xrange(ylo, yhi): ry1.append('+') ry2.append(y[i]) r.append("\n" "
\n%s\n%s\n
\n" "
\n%s\n%s\n
\n" "\n" % ('\n'.join(rx1), '\n'.join(ry1), escape('\n'.join(rx2)), escape('\n'.join(ry2)))) def html_diff(s1, s2): a=s1.split('\n') b=s2.split('\n') cruncher=difflib.SequenceMatcher() cruncher.set_seqs(a,b) r=[''] for tag, alo, ahi, blo, bhi in cruncher.get_opcodes(): if tag == 'replace': replace(a, alo, ahi, b, blo, bhi, r) elif tag == 'delete': dump('-', a, alo, ahi, r) elif tag == 'insert': dump('+', b, blo, bhi, r) elif tag == 'equal': dump(' ', a, alo, ahi, r) else: raise ValueError, 'unknown tag ' + `tag` r.append('
') return '\n'.join(r) Zope-2.8.7-final/lib/python/OFS/DTMLMethod.py0000664000076400007640000003631010435500327017533 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """DTML Method objects. $Id: DTMLMethod.py 28774 2005-01-10 14:16:41Z jens $ """ import History from Globals import HTML, DTMLFile, MessageDialog from SimpleItem import Item_w__name__, pretty_tb from OFS.content_types import guess_content_type from PropertyManager import PropertyManager from AccessControl.Role import RoleManager from webdav.common import rfc1123_date from webdav.Lockable import ResourceLockedError from webdav.WriteLockInterface import WriteLockInterface from ZDOM import ElementWithTitle from DateTime.DateTime import DateTime from urllib import quote import Globals, sys, Acquisition from AccessControl import getSecurityManager from AccessControl.DTML import RestrictedDTML from Cache import Cacheable from zExceptions import Forbidden from zExceptions.TracebackSupplement import PathTracebackSupplement from ZPublisher.Iterators import IStreamIterator _marker = [] # Create a new marker object. class DTMLMethod(RestrictedDTML, HTML, Acquisition.Implicit, RoleManager, ElementWithTitle, Item_w__name__, History.Historical, Cacheable, ): """DTML Method objects are DocumentTemplate.HTML objects that act as methods of their containers.""" meta_type='DTML Method' _proxy_roles=() index_html=None # Prevent accidental acquisition _cache_namespace_keys=() __implements__ = (WriteLockInterface,) # Documents masquerade as functions: class func_code: pass func_code=func_code() func_code.co_varnames='self','REQUEST','RESPONSE' func_code.co_argcount=3 manage_options=( ( {'label':'Edit', 'action':'manage_main', 'help':('OFSP','DTML-DocumentOrMethod_Edit.stx')}, #upload is deprecated #{'label':'Upload', 'action':'manage_uploadForm', # 'help':('OFSP','DTML-DocumentOrMethod_Upload.stx')}, {'label':'View', 'action':'', 'help':('OFSP','DTML-DocumentOrMethod_View.stx')}, {'label':'Proxy', 'action':'manage_proxyForm', 'help':('OFSP','DTML-DocumentOrMethod_Proxy.stx')}, ) +History.Historical.manage_options +RoleManager.manage_options +Item_w__name__.manage_options +Cacheable.manage_options ) # Careful in changes--used by DTMLDocument! __ac_permissions__=( ('View management screens', ('document_src', 'PrincipiaSearchSource')), ('Change DTML Methods', ('manage_editForm', 'manage', 'manage_main', 'manage_edit', 'manage_upload', 'PUT', 'manage_historyCopy', 'manage_beforeHistoryCopy', 'manage_afterHistoryCopy', 'ZCacheable_configHTML', 'getCacheNamespaceKeys', 'setCacheNamespaceKeys', ) ), ('Change proxy roles', ('manage_proxyForm', 'manage_proxy')), ('View', ('__call__', 'get_size', '')), ('FTP access', ('manage_FTPstat','manage_FTPget','manage_FTPlist')), ) # support a more reasonable default for content-type # for http HEAD requests. default_content_type='text/html' def __call__(self, client=None, REQUEST={}, RESPONSE=None, **kw): """Render the document given a client object, REQUEST mapping, Response, and key word arguments.""" if not self._cache_namespace_keys: data = self.ZCacheable_get(default=_marker) if data is not _marker: if ( IStreamIterator.isImplementedBy(data) and RESPONSE is not None ): # This is a stream iterator and we need to set some # headers now before giving it to medusa if RESPONSE.headers.get('content-length', None) is None: RESPONSE.setHeader('content-length', len(data)) if ( RESPONSE.headers.get('content-type', None) is None and RESPONSE.headers.get('Content-type', None) is None ): ct = ( self.__dict__.get('content_type') or self.default_content_type ) RESPONSE.setHeader('content-type', ct) # Return cached results. return data __traceback_supplement__ = (PathTracebackSupplement, self) kw['document_id'] =self.getId() kw['document_title']=self.title security=getSecurityManager() security.addContext(self) if self.__dict__.has_key('validate'): first_time_through = 0 else: self.__dict__['validate'] = security.DTMLValidate first_time_through = 1 try: if client is None: # Called as subtemplate, so don't need error propagation! r=apply(HTML.__call__, (self, client, REQUEST), kw) if RESPONSE is None: result = r else: result = decapitate(r, RESPONSE) if not self._cache_namespace_keys: self.ZCacheable_set(result) return result r=apply(HTML.__call__, (self, client, REQUEST), kw) if type(r) is not type('') or RESPONSE is None: if not self._cache_namespace_keys: self.ZCacheable_set(r) return r finally: security.removeContext(self) if first_time_through: del self.__dict__['validate'] have_key=RESPONSE.headers.has_key if not (have_key('content-type') or have_key('Content-Type')): if self.__dict__.has_key('content_type'): c=self.content_type else: c, e=guess_content_type(self.getId(), r) RESPONSE.setHeader('Content-Type', c) result = decapitate(r, RESPONSE) if not self._cache_namespace_keys: self.ZCacheable_set(result) return result def validate(self, inst, parent, name, value, md=None): return getSecurityManager().validate(inst, parent, name, value) def ZDocumentTemplate_beforeRender(self, md, default): # Tries to get a cached value. if self._cache_namespace_keys: # Use the specified keys from the namespace to identify a # cache entry. kw = {} for key in self._cache_namespace_keys: try: val = md[key] except: val = None kw[key] = val return self.ZCacheable_get(keywords=kw, default=default) return default def ZDocumentTemplate_afterRender(self, md, result): # Tries to set a cache value. if self._cache_namespace_keys: kw = {} for key in self._cache_namespace_keys: try: val = md[key] except: val = None kw[key] = val self.ZCacheable_set(result, keywords=kw) ZCacheable_configHTML = DTMLFile('dtml/cacheNamespaceKeys', globals()) def getCacheNamespaceKeys(self): ''' Returns the cacheNamespaceKeys. ''' return self._cache_namespace_keys def setCacheNamespaceKeys(self, keys, REQUEST=None): ''' Sets the list of names that should be looked up in the namespace to provide a cache key. ''' ks = [] for key in keys: key = str(key).strip() if key: ks.append(key) self._cache_namespace_keys = tuple(ks) if REQUEST is not None: return self.ZCacheable_manage(self, REQUEST) def get_size(self): return len(self.raw) # deprecated; use get_size! getSize=get_size manage_editForm=DTMLFile('dtml/documentEdit', globals()) manage_editForm._setName('manage_editForm') # deprecated! manage_uploadForm=manage_editForm manage=manage_main=manage_editDocument=manage_editForm manage_proxyForm=DTMLFile('dtml/documentProxy', globals()) _size_changes={ 'Bigger': (5,5), 'Smaller': (-5,-5), 'Narrower': (0,-5), 'Wider': (0,5), 'Taller': (5,0), 'Shorter': (-5,0), } def _er(self,data,title,SUBMIT,dtpref_cols,dtpref_rows,REQUEST): dr,dc = self._size_changes[SUBMIT] rows = str(max(1, int(dtpref_rows) + dr)) cols = str(dtpref_cols) if cols.endswith('%'): cols = str(min(100, max(25, int(cols[:-1]) + dc))) + '%' else: cols = str(max(35, int(cols) + dc)) e = (DateTime("GMT") + 365).rfc822() setCookie = REQUEST["RESPONSE"].setCookie setCookie("dtpref_rows", rows, path='/', expires=e) setCookie("dtpref_cols", cols, path='/', expires=e) REQUEST.other.update({"dtpref_cols":cols, "dtpref_rows":rows}) return self.manage_main(self, REQUEST, title=title, __str__=self.quotedHTML(data)) def manage_edit(self,data,title,SUBMIT='Change',dtpref_cols='100%', dtpref_rows='20',REQUEST=None): """ Replaces a Documents contents with Data, Title with Title. The SUBMIT parameter is also used to change the size of the editing area on the default Document edit screen. If the value is "Smaller", the rows and columns decrease by 5. If the value is "Bigger", the rows and columns increase by 5. If any other or no value is supplied, the data gets checked for DTML errors and is saved. """ self._validateProxy(REQUEST) if self._size_changes.has_key(SUBMIT): return self._er(data,title,SUBMIT,dtpref_cols,dtpref_rows,REQUEST) if self.wl_isLocked(): raise ResourceLockedError, 'This DTML Method is locked via WebDAV' self.title=str(title) if type(data) is not type(''): data=data.read() self.munge(data) self.ZCacheable_invalidate() if REQUEST: message="Saved changes." return self.manage_main(self,REQUEST,manage_tabs_message=message) def manage_upload(self,file='', REQUEST=None): """Replace the contents of the document with the text in file.""" self._validateProxy(REQUEST) if self.wl_isLocked(): raise ResourceLockedError, 'This DTML Method is locked via WebDAV' if type(file) is not type(''): if REQUEST and not file: raise ValueError, 'No file specified' file=file.read() self.munge(file) self.ZCacheable_invalidate() if REQUEST: message="Saved changes." return self.manage_main(self,REQUEST,manage_tabs_message=message) def manage_haveProxy(self,r): return r in self._proxy_roles def _validateProxy(self, request, roles=None): if roles is None: roles=self._proxy_roles if not roles: return user=u=getSecurityManager().getUser() user=user.allowed for r in roles: if r and not user(self, (r,)): user=None break if user is not None: return raise Forbidden, ( 'You are not authorized to change %s because you ' 'do not have proxy roles.\n' % (self.__name__, u, roles)) def manage_proxy(self, roles=(), REQUEST=None): "Change Proxy Roles" self._validateProxy(REQUEST, roles) self._validateProxy(REQUEST) self._proxy_roles=tuple(roles) self.ZCacheable_invalidate() if REQUEST: message="Saved changes." return self.manage_proxyForm(self,REQUEST,manage_tabs_message=message) def PrincipiaSearchSource(self): "Support for searching - the document's contents are searched." return self.read() def document_src(self, REQUEST=None, RESPONSE=None): """Return unprocessed document source.""" if RESPONSE is not None: RESPONSE.setHeader('Content-Type', 'text/plain') return self.read() ## Protocol handlers def PUT(self, REQUEST, RESPONSE): """Handle HTTP PUT requests.""" self.dav__init(REQUEST, RESPONSE) self.dav__simpleifhandler(REQUEST, RESPONSE, refresh=1) body=REQUEST.get('BODY', '') self._validateProxy(REQUEST) self.munge(body) self.ZCacheable_invalidate() RESPONSE.setStatus(204) return RESPONSE def manage_FTPget(self): "Get source for FTP download" return self.read() def manage_historyCompare(self, rev1, rev2, REQUEST, historyComparisonResults=''): return DTMLMethod.inheritedAttribute('manage_historyCompare')( self, rev1, rev2, REQUEST, historyComparisonResults=History.html_diff( rev1.read(), rev2.read() )) import re token = "[a-zA-Z0-9!#$%&'*+\-.\\\\^_`|~]+" hdr_start = re.compile(r'(%s):(.*)' % token).match def decapitate(html, RESPONSE=None): headers = [] spos = 0 eolen = 1 while 1: m = hdr_start(html, spos) if not m: if html[spos:spos+2] == '\r\n': eolen = 2 break if html[spos:spos+1] == '\n': eolen = 1 break return html header = list(m.groups()) headers.append(header) spos = m.end() + 1 while spos < len(html) and html[spos] in ' \t': eol = html.find('\r\n', spos) if eol <> -1: eolen = 2 else: eol = html.find( '\n', spos) if eol < 0: return html eolen = 1 header.append(html[spos:eol].strip()) spos = eol + eolen if RESPONSE is not None: for header in headers: hkey = header.pop(0) RESPONSE.setHeader(hkey, ' '.join(header).strip()) return html[spos + eolen:] default_dm_html="""

This is the Document in the Folder.

""" addForm=DTMLFile('dtml/methodAdd', globals()) def addDTMLMethod(self, id, title='', file='', REQUEST=None, submit=None): """Add a DTML Method object with the contents of file. If 'file' is empty, default document text is used. """ if type(file) is not type(''): file=file.read() if not file: file=default_dm_html id=str(id) title=str(title) ob=DTMLMethod(file, __name__=id) ob.title=title id=self._setObject(id, ob) if REQUEST is not None: try: u=self.DestinationURL() except: u=REQUEST['URL1'] if submit==" Add and Edit ": u="%s/%s" % (u,quote(id)) REQUEST.RESPONSE.redirect(u+'/manage_main') return '' Zope-2.8.7-final/lib/python/OFS/Cache.py0000664000076400007640000004552510435500327016645 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Cacheable object and cache management base classes. $Id: Cache.py 65847 2006-03-07 10:38:14Z shh $ """ import time, sys import Globals from Globals import DTMLFile from Acquisition import aq_get, aq_acquire, aq_inner, aq_parent, aq_base from zLOG import LOG, WARNING from AccessControl import getSecurityManager from AccessControl.Role import _isBeingUsedAsAMethod from AccessControl import Unauthorized ZCM_MANAGERS = '__ZCacheManager_ids__' ViewManagementScreensPermission = 'View management screens' ChangeCacheSettingsPermission = 'Change cache settings' def isCacheable(ob): return getattr(aq_base(ob), '_isCacheable', 0) def managersExist(ob): # Returns 1 if any CacheManagers exist in the context of ob. if aq_get(ob, ZCM_MANAGERS, None, 1): return 1 return 0 def filterCacheTab(ob): if _isBeingUsedAsAMethod(ob): # Show tab when in a ZClass def that uses Cacheable as a base. parent = aq_parent(aq_inner(ob)) return isCacheable(parent) else: return managersExist(ob) def filterCacheManagers(orig, container, name, value, extra): ''' This is a filter method for aq_acquire. It causes objects to be found only if they are in the list of cache managers. ''' if (hasattr(aq_base(container), ZCM_MANAGERS) and name in getattr(container, ZCM_MANAGERS)): return 1 return 0 def getVerifiedManagerIds(container): ''' Gets the list of cache managers in a container, verifying each one. ''' ids = getattr(container, ZCM_MANAGERS, ()) rval = [] for id in ids: if getattr(getattr(container, id, None), '_isCacheManager', 0): rval.append(id) return tuple(rval) # Anytime a CacheManager is added or removed, all _v_ZCacheable_cache # attributes must be invalidated. manager_timestamp is a way to do # that. manager_timestamp = 0 class Cacheable: '''Mix-in for cacheable objects. ''' manage_options = ({ 'label':'Cache', 'action':'ZCacheable_manage', 'filter':filterCacheTab, 'help':('OFSP','Cacheable-properties.stx'), },) __ac_permissions__ = ( (ViewManagementScreensPermission, ('ZCacheable_manage', 'ZCacheable_invalidate', 'ZCacheable_enabled', 'ZCacheable_getManagerId', 'ZCacheable_getManagerIds', 'ZCacheable_configHTML', 'ZCacheable_isAMethod', 'ZCacheable_getManagerURL', )), (ChangeCacheSettingsPermission, ('ZCacheable_setManagerId', 'ZCacheable_setEnabled', ), ('Manager',)), ) ZCacheable_manage = DTMLFile('dtml/cacheable', globals()) _v_ZCacheable_cache = None _v_ZCacheable_manager_timestamp = 0 __manager_id = None __enabled = 1 _isCacheable = 1 ZCacheable_getManager__roles__ = () def ZCacheable_getManager(self): '''Returns the currently associated cache manager.''' manager_id = self.__manager_id if manager_id is None: return None try: return aq_acquire( self, manager_id, containment=1, filter=filterCacheManagers, extra=None, default=None) except AttributeError: return None ZCacheable_getCache__roles__ = () def ZCacheable_getCache(self): '''Gets the cache associated with this object. ''' if self.__manager_id is None: return None c = self._v_ZCacheable_cache if c is not None: # We have a volatile reference to the cache. if self._v_ZCacheable_manager_timestamp == manager_timestamp: return aq_base(c) manager = self.ZCacheable_getManager() if manager is not None: c = aq_base(manager.ZCacheManager_getCache()) else: return None # Set a volatile reference to the cache then return it. self._v_ZCacheable_cache = c self._v_ZCacheable_manager_timestamp = manager_timestamp return c ZCacheable_isCachingEnabled__roles__ = () def ZCacheable_isCachingEnabled(self): ''' Returns true only if associated with a cache manager and caching of this method is enabled. ''' return self.__enabled and self.ZCacheable_getCache() def ZCacheable_isAMethod(self): ''' Returns 1 when this object is a ZClass method. ''' m = _isBeingUsedAsAMethod(self) return m ZCacheable_getObAndView__roles__ = () def ZCacheable_getObAndView(self, view_name): """ If this object is a method of a ZClass and we're working with the primary view, uses the ZClass instance as ob and our own ID as the view_name. Otherwise returns self and view_name unchanged. """ ob = self if not view_name and self.ZCacheable_isAMethod(): # This is a ZClass method. ob = aq_parent(aq_inner(self)) if isCacheable(ob): view_name = self.getId() else: # Both the parent and the child have to be # cacheable. ob = self return ob, view_name ZCacheable_get__roles__ = () def ZCacheable_get(self, view_name='', keywords=None, mtime_func=None, default=None): '''Retrieves the cached view for the object under the conditions specified by keywords. If the value is not yet cached, returns the default. ''' c = self.ZCacheable_getCache() if c is not None and self.__enabled: ob, view_name = self.ZCacheable_getObAndView(view_name) try: val = c.ZCache_get(ob, view_name, keywords, mtime_func, default) return val except: LOG('Cache', WARNING, 'ZCache_get() exception', error=sys.exc_info()) return default return default ZCacheable_set__roles__ = () def ZCacheable_set(self, data, view_name='', keywords=None, mtime_func=None): '''Cacheable views should call this method after generating cacheable results. The data argument can be of any Python type. ''' c = self.ZCacheable_getCache() if c is not None and self.__enabled: ob, view_name = self.ZCacheable_getObAndView(view_name) try: c.ZCache_set(ob, data, view_name, keywords, mtime_func) except: LOG('Cache', WARNING, 'ZCache_set() exception', error=sys.exc_info()) def ZCacheable_invalidate(self, view_name='', REQUEST=None): '''Called after a cacheable object is edited. Causes all cache entries that apply to the view_name to be removed. Returns a status message. ''' c = self.ZCacheable_getCache() if c is not None: ob, view_name = self.ZCacheable_getObAndView(view_name) try: message = c.ZCache_invalidate(ob) if not message: message = 'Invalidated.' except: exc = sys.exc_info() try: LOG('Cache', WARNING, 'ZCache_invalidate() exception', error=exc) message = 'An exception occurred: %s: %s' % exc[:2] finally: exc = None else: message = 'This object is not associated with a cache manager.' if REQUEST is not None: return self.ZCacheable_manage( self, REQUEST, management_view='Cache', manage_tabs_message=message) else: return message ZCacheable_getModTime__roles__=() def ZCacheable_getModTime(self, mtime_func=None): '''Returns the highest of the last mod times.''' # Based on: # mtime_func # self.mtime # self.__class__.mtime # (if in a ZClass) zclass_instance.mtime # zclass_instance.__class__.mtime mtime = 0 if mtime_func: # Allow mtime_func to influence the mod time. mtime = mtime_func() base = aq_base(self) mtime = max(getattr(base, '_p_mtime', mtime), mtime) klass = getattr(base, '__class__', None) if klass: mtime = max(getattr(klass, '_p_mtime', mtime), mtime) if self.ZCacheable_isAMethod(): # This is a ZClass method. instance = aq_parent(aq_inner(self)) base = aq_base(instance) mtime = max(getattr(base, '_p_mtime', mtime), mtime) klass = getattr(base, '__class__', None) if klass: mtime = max(getattr(klass, '_p_mtime', mtime), mtime) return mtime def ZCacheable_getManagerId(self): '''Returns the id of the current ZCacheManager.''' return self.__manager_id def ZCacheable_getManagerURL(self): '''Returns the URL of the current ZCacheManager.''' manager = self.ZCacheable_getManager() if manager is not None: return manager.absolute_url() return None def ZCacheable_getManagerIds(self): '''Returns a list of mappings containing the id and title of the available ZCacheManagers.''' rval = [] ob = self used_ids = {} while ob is not None: if hasattr(aq_base(ob), ZCM_MANAGERS): ids = getattr(ob, ZCM_MANAGERS) for id in ids: manager = getattr(ob, id, None) if manager is not None: id = manager.getId() if not used_ids.has_key(id): title = getattr(aq_base(manager), 'title', '') rval.append({'id':id, 'title':title}) used_ids[id] = 1 ob = aq_parent(aq_inner(ob)) return tuple(rval) def ZCacheable_setManagerId(self, manager_id, REQUEST=None): '''Changes the manager_id for this object.''' self.ZCacheable_invalidate() if not manager_id: # User requested disassociation # from the cache manager. manager_id = None else: manager_id = str(manager_id) self.__manager_id = manager_id self._v_ZCacheable_cache = None if REQUEST is not None: return self.ZCacheable_manage( self, REQUEST, management_view='Cache', manage_tabs_message='Cache settings changed.') def ZCacheable_enabled(self): '''Returns true if caching is enabled for this object or method.''' return self.__enabled def ZCacheable_setEnabled(self, enabled=0, REQUEST=None): '''Changes the enabled flag. Normally used only when setting up cacheable ZClass methods.''' self.__enabled = enabled and 1 or 0 if REQUEST is not None: return self.ZCacheable_manage( self, REQUEST, management_view='Cache', manage_tabs_message='Cache settings changed.') def ZCacheable_configHTML(self): '''Override to provide configuration of caching behavior that can only be specific to the cacheable object. ''' return '' Globals.default__class_init__(Cacheable) def findCacheables(ob, manager_id, require_assoc, subfolders, meta_types, rval, path): ''' Used by the CacheManager UI. Recursive. Similar to the Zope "Find" function. Finds all Cacheable objects in a hierarchy. ''' try: if meta_types: subobs = ob.objectValues(meta_types) else: subobs = ob.objectValues() sm = getSecurityManager() # Add to the list of cacheable objects. for subob in subobs: if not isCacheable(subob): continue associated = (subob.ZCacheable_getManagerId() == manager_id) if require_assoc and not associated: continue if not sm.checkPermission('Change cache settings', subob): continue subpath = path + (subob.getId(),) icon = getattr(aq_base(subob), 'icon', '') info = { 'sortkey': subpath, 'path': '/'.join(subpath), 'title': getattr(aq_base(subob), 'title', ''), 'icon': icon, 'associated': associated,} rval.append(info) # Visit subfolders. if subfolders: if meta_types: subobs = ob.objectValues() for subob in subobs: subpath = path + (subob.getId(),) if hasattr(aq_base(subob), 'objectValues'): if sm.checkPermission( 'Access contents information', subob): findCacheables( subob, manager_id, require_assoc, subfolders, meta_types, rval, subpath) except: # Ignore exceptions. import traceback traceback.print_exc() class Cache: ''' A base class (and interface description) for caches. Note that Cache objects are not intended to be visible by restricted code. ''' def ZCache_invalidate(self, ob): raise NotImplementedError def ZCache_get(self, ob, view_name, keywords, mtime_func, default): # view_name: If an object provides different views that would # benefit from caching, it will set view_name. # Otherwise view_name will be an empty string. # # keywords: Either None or a mapping containing keys that # distinguish this cache entry from others even though # ob and view_name are the same. DTMLMethods use keywords # derived from the DTML namespace. # # mtime_func: When the Cache calls ZCacheable_getModTime(), # it should pass this as an argument. It is provided to # allow cacheable objects to provide their own computation # of the object's modification time. # # default: If no entry is found, ZCache_get() should return # default. raise NotImplementedError def ZCache_set(self, ob, data, view_name, keywords, mtime_func): # See ZCache_get() for parameter descriptions. raise NotImplementedError class CacheManager: ''' A base class for cache managers. Implement ZCacheManager_getCache(). ''' ZCacheManager_getCache__roles__ = () def ZCacheManager_getCache(self): raise NotImplementedError _isCacheManager = 1 __ac_permissions__ = ( ('Change cache settings', ('ZCacheManager_locate', 'ZCacheManager_setAssociations', 'ZCacheManager_associate'), ('Manager',)), ) manage_options = ( {'label':'Associate', 'action':'ZCacheManager_associate', 'help':('OFSP','CacheManager-associate.stx'), }, ) def manage_afterAdd(self, item, container): # Adds self to the list of cache managers in the container. if aq_base(self) is aq_base(item): ids = getVerifiedManagerIds(container) id = self.getId() if id not in ids: setattr(container, ZCM_MANAGERS, ids + (id,)) global manager_timestamp manager_timestamp = time.time() def manage_beforeDelete(self, item, container): # Removes self from the list of cache managers. if aq_base(self) is aq_base(item): ids = getVerifiedManagerIds(container) id = self.getId() if id in ids: setattr(container, ZCM_MANAGERS, filter( lambda s, id=id: s != id, ids)) global manager_timestamp manager_timestamp = time.time() ZCacheManager_associate = DTMLFile('dtml/cmassoc', globals()) def ZCacheManager_locate(self, require_assoc, subfolders, meta_types=[], REQUEST=None): '''Locates cacheable objects. ''' ob = aq_parent(aq_inner(self)) rval = [] manager_id = self.getId() if '' in meta_types: # User selected "All". meta_types = [] findCacheables(ob, manager_id, require_assoc, subfolders, meta_types, rval, ()) if REQUEST is not None: return self.ZCacheManager_associate( self, REQUEST, show_results=1, results=rval, management_view="Associate") else: return rval def ZCacheManager_setAssociations(self, props=None, REQUEST=None): '''Associates and un-associates cacheable objects with this cache manager. ''' addcount = 0 remcount = 0 parent = aq_parent(aq_inner(self)) sm = getSecurityManager() my_id = str(self.getId()) if props is None: props = REQUEST.form for key, do_associate in props.items(): if key[:10] == 'associate_': path = key[10:] ob = parent.restrictedTraverse(path) if not sm.checkPermission('Change cache settings', ob): raise Unauthorized if not isCacheable(ob): # Not a cacheable object. continue manager_id = str(ob.ZCacheable_getManagerId()) if do_associate: if manager_id != my_id: ob.ZCacheable_setManagerId(my_id) addcount = addcount + 1 else: if manager_id == my_id: ob.ZCacheable_setManagerId(None) remcount = remcount + 1 if REQUEST is not None: return self.ZCacheManager_associate( self, REQUEST, management_view="Associate", manage_tabs_message='%d association(s) made, %d removed.'% (addcount, remcount) ) Globals.default__class_init__(CacheManager) Zope-2.8.7-final/lib/python/OFS/OrderSupport.py0000664000076400007640000002547410435500327020313 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Order support for 'Object Manager'. $Id: OrderSupport.py 28958 2005-01-26 15:40:07Z yuppie $ """ from types import StringType from AccessControl import ClassSecurityInfo from AccessControl.Permissions import access_contents_information from AccessControl.Permissions import manage_properties from Acquisition import aq_base from DocumentTemplate.sequence import sort from Globals import InitializeClass from IOrderSupport import IOrderedContainer from ObjectManager import ObjectManager class OrderSupport: """ Ordered container mixin class. This is an extension to the regular ObjectManager. It saves the objects in order and lets you change the order of the contained objects. This is particular helpful, if the order does not depend on object attributes, but is totally user-specific. """ __implements__ = IOrderedContainer security = ClassSecurityInfo() has_order_support = 1 _default_sort_key = 'position' _default_sort_reverse = 0 manage_options = ( { 'label':'Contents', 'action':'manage_main', 'help':('OFSP','OrderSupport_Contents.stx') } , ) # # ZMI Methods # security.declareProtected(manage_properties, 'manage_move_objects_up') def manage_move_objects_up(self, REQUEST, ids=None, delta=1): """ Move specified sub-objects up by delta in container. """ if ids: try: attempt = self.moveObjectsUp(ids, delta) message = '%d item%s moved up.' % ( attempt, ( (attempt!=1) and 's' or '' ) ) except ValueError, errmsg: message = 'Error: %s' % (errmsg) else: message = 'Error: No items were specified!' return self.manage_main(self, REQUEST, skey='position', manage_tabs_message=message, update_menu=1) security.declareProtected(manage_properties, 'manage_move_objects_down') def manage_move_objects_down(self, REQUEST, ids=None, delta=1): """ Move specified sub-objects down by delta in container. """ if ids: try: attempt = self.moveObjectsDown(ids, delta) message = '%d item%s moved down.' % ( attempt, ( (attempt!=1) and 's' or '' ) ) except ValueError, errmsg: message = 'Error: %s' % (errmsg) else: message = 'Error: No items were specified!' return self.manage_main(self, REQUEST, skey='position', manage_tabs_message=message, update_menu=1) security.declareProtected(manage_properties, 'manage_move_objects_to_top') def manage_move_objects_to_top(self, REQUEST, ids=None): """ Move specified sub-objects to top of container. """ if ids: try: attempt = self.moveObjectsToTop(ids) message = '%d item%s moved to top.' % ( attempt, ( (attempt!=1) and 's' or '' ) ) except ValueError, errmsg: message = 'Error: %s' % (errmsg) else: message = 'Error: No items were specified!' return self.manage_main(self, REQUEST, skey='position', manage_tabs_message=message, update_menu=1) security.declareProtected(manage_properties, 'manage_move_objects_to_bottom') def manage_move_objects_to_bottom(self, REQUEST, ids=None): """ Move specified sub-objects to bottom of container. """ if ids: try: attempt = self.moveObjectsToBottom(ids) message = '%d item%s moved to bottom.' % ( attempt, ( (attempt!=1) and 's' or '' ) ) except ValueError, errmsg: message = 'Error: %s' % (errmsg) else: message = 'Error: No items were specified!' return self.manage_main(self, REQUEST, skey='position', manage_tabs_message=message, update_menu=1) security.declareProtected(manage_properties, 'manage_set_default_sorting') def manage_set_default_sorting(self, REQUEST, key, reverse): """ Set default sorting key and direction. """ self.setDefaultSorting(key, reverse) return self.manage_main(self, REQUEST, update_menu=1) # # IOrderedContainer Interface Methods # security.declareProtected(manage_properties, 'moveObjectsByDelta') def moveObjectsByDelta(self, ids, delta, subset_ids=None): """ Move specified sub-objects by delta. """ if type(ids) is StringType: ids = (ids,) min_position = 0 objects = list(self._objects) if subset_ids == None: subset_ids = [ obj['id'] for obj in objects ] else: subset_ids = list(subset_ids) # unify moving direction if delta > 0: ids = list(ids) ids.reverse() subset_ids.reverse() counter = 0 for id in ids: old_position = subset_ids.index(id) new_position = max( old_position - abs(delta), min_position ) if new_position == min_position: min_position += 1 if not old_position == new_position: subset_ids.remove(id) subset_ids.insert(new_position, id) counter += 1 if counter > 0: if delta > 0: subset_ids.reverse() obj_dict = {} for obj in objects: obj_dict[ obj['id'] ] = obj pos = 0 for i in range( len(objects) ): if objects[i]['id'] in subset_ids: try: objects[i] = obj_dict[ subset_ids[pos] ] pos += 1 except KeyError: raise ValueError('The object with the id "%s" does ' 'not exist.' % subset_ids[pos]) self._objects = tuple(objects) return counter security.declareProtected(manage_properties, 'moveObjectsUp') def moveObjectsUp(self, ids, delta=1, subset_ids=None): """ Move specified sub-objects up by delta in container. """ return self.moveObjectsByDelta(ids, -delta, subset_ids) security.declareProtected(manage_properties, 'moveObjectsDown') def moveObjectsDown(self, ids, delta=1, subset_ids=None): """ Move specified sub-objects down by delta in container. """ return self.moveObjectsByDelta(ids, delta, subset_ids) security.declareProtected(manage_properties, 'moveObjectsToTop') def moveObjectsToTop(self, ids, subset_ids=None): """ Move specified sub-objects to top of container. """ return self.moveObjectsByDelta( ids, -len(self._objects), subset_ids ) security.declareProtected(manage_properties, 'moveObjectsToBottom') def moveObjectsToBottom(self, ids, subset_ids=None): """ Move specified sub-objects to bottom of container. """ return self.moveObjectsByDelta( ids, len(self._objects), subset_ids ) security.declareProtected(manage_properties, 'orderObjects') def orderObjects(self, key, reverse=None): """ Order sub-objects by key and direction. """ ids = [ id for id, obj in sort( self.objectItems(), ( (key, 'cmp', 'asc'), ) ) ] if reverse: ids.reverse() return self.moveObjectsByDelta( ids, -len(self._objects) ) security.declareProtected(access_contents_information, 'getObjectPosition') def getObjectPosition(self, id): """ Get the position of an object by its id. """ ids = self.objectIds() if id in ids: return ids.index(id) raise ValueError('The object with the id "%s" does not exist.' % id) security.declareProtected(manage_properties, 'moveObjectToPosition') def moveObjectToPosition(self, id, position): """ Move specified object to absolute position. """ delta = position - self.getObjectPosition(id) return self.moveObjectsByDelta(id, delta) security.declareProtected(access_contents_information, 'getDefaultSorting') def getDefaultSorting(self): """ Get default sorting key and direction. """ return self._default_sort_key, self._default_sort_reverse security.declareProtected(manage_properties, 'setDefaultSorting') def setDefaultSorting(self, key, reverse): """ Set default sorting key and direction. """ self._default_sort_key = key self._default_sort_reverse = reverse and 1 or 0 # # Override Inherited Method of ObjectManager Subclass # _old_manage_renameObject = ObjectManager.inheritedAttribute( 'manage_renameObject') def manage_renameObject(self, id, new_id, REQUEST=None): """ Rename a particular sub-object without changing its position. """ old_position = self.getObjectPosition(id) result = self._old_manage_renameObject(id, new_id, REQUEST) self.moveObjectToPosition(new_id, old_position) return result def tpValues(self): # Return a list of subobjects, used by tree tag. r=[] if hasattr(aq_base(self), 'tree_ids'): tree_ids=self.tree_ids try: tree_ids=list(tree_ids) except TypeError: pass if hasattr(tree_ids, 'sort'): tree_ids.sort() for id in tree_ids: if hasattr(self, id): r.append(self._getOb(id)) else: # this part is different from the ObjectManager code r = [ obj for obj in self.objectValues() if getattr(obj, 'isPrincipiaFolderish', False) ] r = sort( r, ( (self._default_sort_key, 'cmp', 'asc'), ) ) if self._default_sort_reverse: r.reverse() return r InitializeClass(OrderSupport) Zope-2.8.7-final/lib/python/OFS/__init__.py0000664000076400007640000000117210435500327017367 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## Zope-2.8.7-final/lib/python/OFS/Application.py0000664000076400007640000011045610435500327020101 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Application support $Id: Application.py 41579 2006-02-08 15:45:05Z anguenot $ """ import os, sys, traceback from cgi import escape from StringIO import StringIO from warnings import warn import Globals, Products, App.Product, App.ProductRegistry import transaction from AccessControl.User import UserFolder from Acquisition import aq_base from App.ApplicationManager import ApplicationManager from App.config import getConfiguration from App.Product import doInstall from App.ProductContext import ProductContext from DateTime import DateTime from HelpSys.HelpSys import HelpSys from webdav.NullResource import NullResource from zExceptions import Redirect as RedirectException, Forbidden from zLOG import LOG, ERROR, WARNING, INFO import Folder import misc_ import ZDOM from FindSupport import FindSupport from misc_ import Misc_ class Application(Globals.ApplicationDefaultPermissions, ZDOM.Root, Folder.Folder, App.ProductRegistry.ProductRegistry, FindSupport): """Top-level system object""" title ='Zope' #__roles__=['Manager', 'Anonymous'] __defined_roles__=('Manager','Anonymous','Owner') web__form__method='GET' isTopLevelPrincipiaApplicationObject=1 _isBeingUsedAsAMethod_=0 # Create the help system object HelpSys=HelpSys('HelpSys') p_=misc_.p_ misc_=misc_.misc_ _reserved_names=('Control_Panel', 'browser_id_manager', 'temp_folder') # This class-default __allow_groups__ ensures that the # emergency user can still access the system if the top-level # UserFolder is deleted. This is necessary to allow people # to replace the top-level UserFolder object. __allow_groups__=UserFolder() # Set the universal default method to index_html _object_manager_browser_default_id = 'index_html' _initializer_registry = None def __init__(self): # Initialize users uf=UserFolder() self.__allow_groups__=uf self._setObject('acl_users', uf) # Initialize control panel cpl=ApplicationManager() cpl._init() self._setObject('Control_Panel', cpl) transaction.get().note("Created Zope Application") def id(self): try: return self.REQUEST['SCRIPT_NAME'][1:] except: return self.title def title_and_id(self): return self.title def title_or_id(self): return self.title def __class_init__(self): Globals.default__class_init__(self) def PrincipiaRedirect(self, destination, URL1): """Utility function to allow user-controlled redirects""" if destination.find('//') >= 0: raise RedirectException, destination raise RedirectException, ("%s/%s" % (URL1, destination)) Redirect = ZopeRedirect = PrincipiaRedirect def __bobo_traverse__(self, REQUEST, name=None): try: return getattr(self, name) except AttributeError: pass try: return self[name] except KeyError: pass method=REQUEST.get('REQUEST_METHOD', 'GET') if not method in ('GET', 'POST'): return NullResource(self, name, REQUEST).__of__(self) # Waaa. unrestrictedTraverse calls us with a fake REQUEST. # There is proabably a better fix for this. try: REQUEST.RESPONSE.notFoundError("%s\n%s" % (name, method)) except AttributeError: raise KeyError, name def PrincipiaTime(self, *args): """Utility function to return current date/time""" return apply(DateTime, args) ZopeTime = PrincipiaTime ZopeAttributionButton__roles__=None def ZopeAttributionButton(self): """Returns an HTML fragment that displays the 'powered by zope' button along with a link to the Zope site.""" return '' % escape(self.REQUEST.BASE1, 1) def DELETE(self, REQUEST, RESPONSE): """Delete a resource object.""" self.dav__init(REQUEST, RESPONSE) raise Forbidden, 'This resource cannot be deleted.' def MOVE(self, REQUEST, RESPONSE): """Move a resource to a new location.""" self.dav__init(REQUEST, RESPONSE) raise Forbidden, 'This resource cannot be moved.' test_url___allow_groups__=None test_url = ZopeAttributionButton def absolute_url(self, relative=0): """The absolute URL of the root object is BASE1 or "/". """ if relative: return '' try: # Take advantage of computed URL cache return self.REQUEST['BASE1'] except (AttributeError, KeyError): return '/' def absolute_url_path(self): """The absolute URL path of the root object is BASEPATH1 or "/". """ try: return self.REQUEST['BASEPATH1'] or '/' except (AttributeError, KeyError): return '/' def virtual_url_path(self): """The virtual URL path of the root object is empty. """ return '' def getPhysicalRoot(self): return self def getPhysicalPath(self): """Get the physical path of the object. Returns a path (an immutable sequence of strings) that can be used to access this object again later, for example in a copy/paste operation. getPhysicalRoot() and getPhysicalPath() are designed to operate together. """ # We're at the base of the path. return ('',) fixupZClassDependencies__roles__=() def fixupZClassDependencies(self, rebuild=0): # Note that callers should not catch exceptions from this method # to ensure that the transaction gets aborted if the registry # cannot be rebuilt for some reason. Returns true if any ZClasses # were registered as a result of the call or the registry was # rebuilt. jar=self._p_jar result=0 if rebuild: from BTrees.OOBTree import OOBTree jar.root()['ZGlobals'] = OOBTree() result = 1 zglobals =jar.root()['ZGlobals'] reg_has_key=zglobals.has_key products=self.Control_Panel.Products for product in products.objectValues(): items=list(product.objectItems()) finished_dict={} finished = finished_dict.has_key while items: name, ob = items.pop() base=aq_base(ob) if finished(id(base)): continue finished_dict[id(base)] = None try: # Try to re-register ZClasses if they need it. if hasattr(base,'_register') and hasattr(base,'_zclass_'): class_id=getattr(base._zclass_, '__module__', None) if class_id and not reg_has_key(class_id): ob._register() result=1 if not rebuild: LOG('Zope', INFO, 'Registered ZClass: %s' % ob.id ) # Include subobjects. if hasattr(base, 'objectItems'): m = list(ob.objectItems()) items.extend(m) # Try to find ZClasses-in-ZClasses. if hasattr(base, 'propertysheets'): ps = ob.propertysheets if (hasattr(ps, 'methods') and hasattr(ps.methods, 'objectItems')): m = list(ps.methods.objectItems()) items.extend(m) except: LOG('Zope', WARNING, 'Broken objects exist in product %s.' % product.id, error=sys.exc_info()) return result checkGlobalRegistry__roles__=() def checkGlobalRegistry(self): """Check the global (zclass) registry for problems, which can be caused by things like disk-based products being deleted. Return true if a problem is found""" try: keys=list(self._p_jar.root()['ZGlobals'].keys()) except: LOG('Zope', ERROR, 'A problem was found when checking the global product '\ 'registry. This is probably due to a Product being '\ 'uninstalled or renamed. The traceback follows.', error=sys.exc_info()) return 1 return 0 _setInitializerRegistry__roles__ = () def _setInitializerFlag(self, flag): if self._initializer_registry is None: self._initializer_registry = {} self._initializer_registry[flag] = 1 _getInitializerRegistry__roles__ = () def _getInitializerFlag(self, flag): reg = self._initializer_registry if reg is None: reg = {} return reg.get(flag) class Expired(Globals.Persistent): icon='p_/broken' def __setstate__(self, s={}): dict=self.__dict__ if s.has_key('id'): dict['id']=s['id'] elif s.has_key('__name__'): dict['id']=s['__name__'] else: dict['id']='Unknown' dict['title']='** Expired **' def __save__(self): pass __inform_commit__=__save__ def initialize(app): initializer = AppInitializer(app) initializer.initialize() class AppInitializer: """ Initialze an Application object (called at startup) """ def __init__(self, app): self.app = (app,) def getApp(self): # this is probably necessary, but avoid acquisition anyway return self.app[0] def commit(self, note): transaction.get().note(note) transaction.commit() def initialize(self): app = self.getApp() # make sure to preserve relative ordering of calls below. self.install_cp_and_products() self.install_tempfolder_and_sdc() self.install_session_data_manager() self.install_browser_id_manager() self.install_required_roles() self.install_zglobals() self.install_inituser() self.install_errorlog() self.install_products() self.install_standards() self.install_virtual_hosting() self.check_zglobals() def install_cp_and_products(self): app = self.getApp() # Ensure that Control Panel exists. if not hasattr(app, 'Control_Panel'): cpl=ApplicationManager() cpl._init() app._setObject('Control_Panel', cpl) self.commit('Added Control_Panel') # b/c: Ensure that a ProductFolder exists. if not hasattr(aq_base(app.Control_Panel), 'Products'): app.Control_Panel.Products=App.Product.ProductFolder() self.commit('Added Control_Panel.Products') def install_tempfolder_and_sdc(self): app = self.getApp() from Products.ZODBMountPoint.MountedObject import manage_addMounts,\ MountedObject from Products.ZODBMountPoint.MountedObject import getConfiguration as \ getDBTabConfiguration dbtab_config = getDBTabConfiguration() tf = getattr(app, 'temp_folder', None) if getattr(tf, 'meta_type', None) == MountedObject.meta_type: # tf is a MountPoint object. This means that the temp_folder # couldn't be mounted properly (the meta_type would have been # the meta type of the container class otherwise). The # MountPoint object writes a message to zLOG so we don't # need to. return if tf is None: # do nothing if we've already installed one if not app._getInitializerFlag('temp_folder'): if dbtab_config is None: # DefaultConfiguration, do nothing return mount_paths = [ x[0] for x in dbtab_config.listMountPaths() ] if not '/temp_folder' in mount_paths: # we won't be able to create the mount point properly LOG('Zope Default Object Creation', ERROR, ('Could not initialze a Temporary Folder because ' 'a database was not configured to be mounted at ' 'the /temp_folder mount point')) return try: manage_addMounts(app, ('/temp_folder',)) app._setInitializerFlag('temp_folder') self.commit('Added temp_folder') tf = app.temp_folder except: LOG('Zope Default Object Creation', ERROR, ('Could not add a /temp_folder mount point due to an ' 'error.'), error=sys.exc_info()) return # Ensure that there is a transient object container in the temp folder config = getConfiguration() if not hasattr(aq_base(tf), 'session_data'): from Products.Transience.Transience import TransientObjectContainer addnotify = getattr(config, 'session_add_notify_script_path', None) delnotify = getattr(config, 'session_delete_notify_script_path', None) default_limit = 1000 default_period_secs = 20 default_timeout_mins = 20 limit = getattr(config, 'maximum_number_of_session_objects', default_limit) timeout_spec = getattr(config, 'session_timeout_minutes', default_timeout_mins) period_spec = getattr(config, 'session_resolution_seconds', default_period_secs) if addnotify and app.unrestrictedTraverse(addnotify, None) is None: LOG('Zope Default Object Creation', WARNING, ('failed to use nonexistent "%s" script as ' 'session-add-notify-script-path' % addnotify)) addnotify=None if delnotify and app.unrestrictedTraverse(delnotify, None) is None: LOG('Zope Default Object Creation', WARNING, ('failed to use nonexistent "%s" script as ' 'session-delete-notify-script-path' % delnotify)) delnotify=None toc = TransientObjectContainer( 'session_data', 'Session Data Container', addNotification = addnotify, delNotification = delnotify, limit=limit, period_secs=period_spec) if timeout_spec is not None: toc = TransientObjectContainer('session_data', 'Session Data Container', timeout_mins = timeout_spec, addNotification = addnotify, delNotification = delnotify, limit=limit, period_secs = period_spec) tf._setObject('session_data', toc) tf_reserved = getattr(tf, '_reserved_names', ()) if 'session_data' not in tf_reserved: tf._reserved_names = tf_reserved + ('session_data',) self.commit('Added session_data to temp_folder') return tf # return the tempfolder object for test purposes def install_browser_id_manager(self): app = self.getApp() if app._getInitializerFlag('browser_id_manager'): # do nothing if we've already installed one return # Ensure that a browser ID manager exists if not hasattr(app, 'browser_id_manager'): from Products.Sessions.BrowserIdManager import BrowserIdManager bid = BrowserIdManager('browser_id_manager', 'Browser Id Manager') app._setObject('browser_id_manager', bid) app._setInitializerFlag('browser_id_manager') self.commit('Added browser_id_manager') def install_session_data_manager(self): app = self.getApp() if app._getInitializerFlag('session_data_manager'): # do nothing if we've already installed one return # Ensure that a session data manager exists if not hasattr(app, 'session_data_manager'): from Products.Sessions.SessionDataManager import SessionDataManager sdm = SessionDataManager('session_data_manager', title='Session Data Manager', path='/temp_folder/session_data', requestName='SESSION') app._setObject('session_data_manager', sdm) app._setInitializerFlag('session_data_manager') self.commit('Added session_data_manager') def install_required_roles(self): app = self.getApp() # Ensure that Owner role exists. if hasattr(app, '__ac_roles__') and not ('Owner' in app.__ac_roles__): app.__ac_roles__=app.__ac_roles__ + ('Owner',) self.commit('Added Owner role') # ensure the Authenticated role exists. if hasattr(app, '__ac_roles__'): if not 'Authenticated' in app.__ac_roles__: app.__ac_roles__=app.__ac_roles__ + ('Authenticated',) self.commit('Added Authenticated role') def install_zglobals(self): app = self.getApp() # Make sure we have ZGlobals root=app._p_jar.root() if not root.has_key('ZGlobals'): from BTrees.OOBTree import OOBTree root['ZGlobals'] = OOBTree() self.commit('Added ZGlobals') def install_inituser(self): app = self.getApp() # Install the initial user. if hasattr(app, 'acl_users'): users = app.acl_users if hasattr(users, '_createInitialUser'): app.acl_users._createInitialUser() self.commit('Created initial user') def install_errorlog(self): app = self.getApp() if app._getInitializerFlag('error_log'): # do nothing if we've already installed one return # Install an error_log if not hasattr(app, 'error_log'): from Products.SiteErrorLog.SiteErrorLog import SiteErrorLog error_log = SiteErrorLog() app._setObject('error_log', error_log) app._setInitializerFlag('error_log') self.commit('Added site error_log at /error_log') def install_virtual_hosting(self): app = self.getApp() if app._getInitializerFlag('virtual_hosting'): return if not app.objectIds('Virtual Host Monster') and not hasattr(app, 'virtual_hosting'): from Products.SiteAccess.VirtualHostMonster import VirtualHostMonster vhm=VirtualHostMonster() vhm.id='virtual_hosting' vhm.addToContainer(app) app._setInitializerFlag('virtual_hosting') self.commit('Added virtual_hosting') def check_zglobals(self): if not doInstall(): return app = self.getApp() # Check for dangling pointers (broken zclass dependencies) in the # global class registry. If found, rebuild the registry. Note that # if the check finds problems but fails to successfully rebuild the # registry we abort the transaction so that we don't leave it in an # indeterminate state. did_fixups=0 bad_things=0 try: if app.checkGlobalRegistry(): LOG('Zope', INFO, 'Beginning attempt to rebuild the global ZClass registry.') app.fixupZClassDependencies(rebuild=1) did_fixups=1 LOG('Zope', INFO, 'The global ZClass registry has successfully been rebuilt.') transaction.get().note('Rebuilt global product registry') transaction.commit() except: bad_things=1 LOG('Zope', ERROR, 'The attempt to rebuild the registry failed.', error=sys.exc_info()) transaction.abort() # Now we need to see if any (disk-based) products were installed # during intialization. If so (and the registry has no errors), # there may still be zclasses dependent on a base class in the # newly installed product that were previously broken and need to # be fixed up. If any really Bad Things happened (dangling pointers # were found in the registry but it couldn't be rebuilt), we don't # try to do anything to avoid making the problem worse. if (not did_fixups) and (not bad_things): # App.Product.initializeProduct will set this if a disk-based # product was added or updated and we are not a ZEO client. if getattr(Globals, '__disk_product_installed__', None): try: LOG('Zope', INFO, ('New disk product detected, determining if we need ' 'to fix up any ZClasses.')) if app.fixupZClassDependencies(): LOG('Zope',INFO, 'Repaired broken ZClass dependencies.') self.commit('Repaired broked ZClass dependencies') except: LOG('Zope', ERROR, ('Attempt to fixup ZClass dependencies after ' 'detecting an updated disk-based product failed.'), error=sys.exc_info()) transaction.abort() def install_products(self): app = self.getApp() # this defers to a function for b/c reasons return install_products(app) def install_standards(self): app = self.getApp() # this defers to a function for b/c reasons return install_standards(app) def install_products(app): # Install a list of products into the basic folder class, so # that all folders know about top-level objects, aka products folder_permissions = get_folder_permissions() meta_types=[] done={} debug_mode = App.config.getConfiguration().debug_mode transaction.get().note('Prior to product installs') transaction.commit() products = get_products() for priority, product_name, index, product_dir in products: # For each product, we will import it and try to call the # intialize() method in the product __init__ module. If # the method doesnt exist, we put the old-style information # together and do a default initialization. if done.has_key(product_name): continue done[product_name]=1 install_product(app, product_dir, product_name, meta_types, folder_permissions, raise_exc=debug_mode) Products.meta_types=Products.meta_types+tuple(meta_types) Globals.default__class_init__(Folder.Folder) def get_products(): """ Return a list of tuples in the form: [(priority, dir_name, index, base_dir), ...] for each Product directory found, sort before returning """ products = [] i = 0 for product_dir in Products.__path__: product_names=os.listdir(product_dir) for name in product_names: fullpath = os.path.join(product_dir, name) # Products must be directories if os.path.isdir(fullpath): # Products must be directories with an __init__.py[co] if ( os.path.exists(os.path.join(fullpath, '__init__.py')) or os.path.exists(os.path.join(fullpath, '__init__.pyo')) or os.path.exists(os.path.join(fullpath, '__init__.pyc')) ): # import PluginIndexes 1st (why?) priority = (name != 'PluginIndexes') # i is used as sort ordering in case a conflict exists # between Product names. Products will be found as # per the ordering of Products.__path__ products.append((priority, name, i, product_dir)) i = i + 1 products.sort() return products def import_products(): # Try to import each product, checking for and catching errors. done={} products = get_products() debug_mode = App.config.getConfiguration().debug_mode for priority, product_name, index, product_dir in products: if done.has_key(product_name): LOG('OFS.Application', WARNING, 'Duplicate Product name', 'After loading Product %s from %s,\n' 'I skipped the one in %s.\n' % ( `product_name`, `done[product_name]`, `product_dir`) ) continue done[product_name]=product_dir import_product(product_dir, product_name, raise_exc=debug_mode) return done.keys() def import_product(product_dir, product_name, raise_exc=0, log_exc=1): path_join=os.path.join isdir=os.path.isdir exists=os.path.exists _st=type('') global_dict=globals() silly=('__doc__',) modules=sys.modules have_module=modules.has_key try: package_dir=path_join(product_dir, product_name) if not isdir(package_dir): return if not exists(path_join(package_dir, '__init__.py')): if not exists(path_join(package_dir, '__init__.pyc')): if not exists(path_join(package_dir, '__init__.pyo')): return pname="Products.%s" % product_name try: product=__import__(pname, global_dict, global_dict, silly) if hasattr(product, '__module_aliases__'): for k, v in product.__module_aliases__: if not have_module(k): if type(v) is _st and have_module(v): v=modules[v] modules[k]=v except: exc = sys.exc_info() if log_exc: LOG('Zope', ERROR, 'Could not import %s' % pname, error=exc) f=StringIO() traceback.print_exc(100,f) f=f.getvalue() try: modules[pname].__import_error__=f except: pass if raise_exc: raise exc[0], exc[1], exc[2] finally: exc = None def get_folder_permissions(): folder_permissions={} for p in Folder.Folder.__ac_permissions__: permission, names = p[:2] folder_permissions[permission]=names return folder_permissions def install_product(app, product_dir, product_name, meta_types, folder_permissions, raise_exc=0, log_exc=1): path_join=os.path.join isdir=os.path.isdir exists=os.path.exists global_dict=globals() silly=('__doc__',) if 1: # Preserve indentation for diff :-) package_dir=path_join(product_dir, product_name) __traceback_info__=product_name if not isdir(package_dir): return if not exists(path_join(package_dir, '__init__.py')): if not exists(path_join(package_dir, '__init__.pyc')): if not exists(path_join(package_dir, '__init__.pyo')): return try: product=__import__("Products.%s" % product_name, global_dict, global_dict, silly) # Install items into the misc_ namespace, used by products # and the framework itself to store common static resources # like icon images. misc_=pgetattr(product, 'misc_', {}) if misc_: if isinstance(misc_, dict): misc_=Misc_(product_name, misc_) Application.misc_.__dict__[product_name]=misc_ # Here we create a ProductContext object which contains # information about the product and provides an interface # for registering things like classes and help topics that # should be associated with that product. Products are # expected to implement a method named 'initialize' in # their __init__.py that takes the ProductContext as an # argument. productObject=App.Product.initializeProduct( product, product_name, package_dir, app) context=ProductContext(productObject, app, product) # Look for an 'initialize' method in the product. If it does # not exist, then this is an old product that has never been # updated. In that case, we will analyze the product and # build up enough information to do initialization manually. initmethod=pgetattr(product, 'initialize', None) if initmethod is not None: initmethod(context) # Support old-style product metadata. Older products may # define attributes to name their permissions, meta_types, # constructors, etc. permissions={} new_permissions={} if pgetattr(product, '__ac_permissions__', None) is not None: warn('__init__.py of %s has a long deprecated ' '\'__ac_permissions__\' attribute. ' '\'__ac_permissions__\' will be ignored by ' 'install_product in Zope 2.10. Please use registerClass ' 'instead.' % product.__name__, DeprecationWarning) for p in pgetattr(product, '__ac_permissions__', ()): permission, names, default = ( tuple(p)+('Manager',))[:3] if names: for name in names: permissions[name]=permission elif not folder_permissions.has_key(permission): new_permissions[permission]=() if pgetattr(product, 'meta_types', None) is not None: warn('__init__.py of %s has a long deprecated \'meta_types\' ' 'attribute. \'meta_types\' will be ignored by ' 'install_product in Zope 2.10. Please use registerClass ' 'instead.' % product.__name__, DeprecationWarning) for meta_type in pgetattr(product, 'meta_types', ()): # Modern product initialization via a ProductContext # adds 'product' and 'permission' keys to the meta_type # mapping. We have to add these here for old products. pname=permissions.get(meta_type['action'], None) if pname is not None: meta_type['permission']=pname meta_type['product']=productObject.id meta_type['visibility'] = 'Global' meta_types.append(meta_type) if pgetattr(product, 'methods', None) is not None: warn('__init__.py of %s has a long deprecated \'methods\' ' 'attribute. \'methods\' will be ignored by ' 'install_product in Zope 2.10. Please use registerClass ' 'instead.' % product.__name__, DeprecationWarning) for name,method in pgetattr( product, 'methods', {}).items(): if not hasattr(Folder.Folder, name): setattr(Folder.Folder, name, method) if name[-9:]!='__roles__': # not Just setting roles if (permissions.has_key(name) and not folder_permissions.has_key( permissions[name])): permission=permissions[name] if new_permissions.has_key(permission): new_permissions[permission].append(name) else: new_permissions[permission]=[name] if new_permissions: new_permissions=new_permissions.items() for permission, names in new_permissions: folder_permissions[permission]=names new_permissions.sort() Folder.Folder.__ac_permissions__=tuple( list(Folder.Folder.__ac_permissions__)+new_permissions) if not doInstall(): transaction.abort() else: transaction.get().note('Installed product '+product_name) transaction.commit() except: if log_exc: LOG('Zope',ERROR,'Couldn\'t install %s' % product_name, error=sys.exc_info()) transaction.abort() if raise_exc: raise def install_standards(app): # Check to see if we've already done this before # Don't do it twice (Casey) if getattr(app, '_standard_objects_have_been_added', 0): return # Install the replaceable standard objects from Products.PageTemplates.PageTemplateFile import PageTemplateFile std_dir = os.path.join(Globals.package_home(globals()), 'standard') wrote = 0 for fn in os.listdir(std_dir): base, ext = os.path.splitext(fn) if ext == '.dtml': ob = Globals.DTMLFile(base, std_dir) fn = base if hasattr(app, fn): continue app.manage_addProduct['OFSP'].manage_addDTMLMethod( id=fn, file=open(ob.raw)) elif ext in ('.pt', '.zpt'): ob = PageTemplateFile(fn, std_dir, __name__=fn) if hasattr(app, fn): continue app.manage_addProduct['PageTemplates'].manage_addPageTemplate( id=fn, title='', text=open(ob.filename)) else: continue wrote = 1 # Below is icky and sneaky since it makes these impossible to delete #ob.__replaceable__ = Globals.REPLACEABLE #setattr(Application, fn, ob) if wrote: app._standard_objects_have_been_added = 1 transaction.get().note('Installed standard objects') transaction.commit() def reinstall_product(app, product_name): folder_permissions = get_folder_permissions() meta_types=[] transaction.get().note('Prior to product reinstall') transaction.commit() for product_dir in Products.__path__: product_names=os.listdir(product_dir) product_names.sort() if product_name in product_names: removeProductMetaTypes(product_name) install_product(app, product_dir, product_name, meta_types, folder_permissions, raise_exc=1, log_exc=0) break Products.meta_types=Products.meta_types+tuple(meta_types) Globals.default__class_init__(Folder.Folder) def reimport_product(product_name): for product_dir in Products.__path__: product_names=os.listdir(product_dir) product_names.sort() if product_name in product_names: import_product(product_dir, product_name, raise_exc=1, log_exc=0) break def removeProductMetaTypes(pid): """Unregisters the meta types registered by a product. """ meta_types = Products.meta_types new_mts = [] changed = 0 for meta_type in meta_types: if meta_type.get('product', None) == pid: # Remove this meta type. changed = 1 else: new_mts.append(meta_type) if changed: Products.meta_types = tuple(new_mts) def pgetattr(product, name, default=install_products, __init__=0): if not __init__ and hasattr(product, name): return getattr(product, name) if hasattr(product, '__init__'): product=product.__init__ if hasattr(product, name): return getattr(product, name) if default is not install_products: return default raise AttributeError, name Zope-2.8.7-final/lib/python/OFS/XMLExportImport.py0000664000076400007640000000647010435500327020673 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import Shared.DC.xml.ppml ppml=Shared.DC.xml.ppml from base64 import encodestring from cStringIO import StringIO from ZODB.serialize import referencesf from ZODB.ExportImport import TemporaryFile, export_end_marker StringType=type('') magic='\n%s \n' % (id, aka, p) return String def exportXML(jar, oid, file=None): if file is None: file=TemporaryFile() elif type(file) is StringType: file=open(file,'w+b') write=file.write write('\012\012') version=jar._version ref=referencesf oids=[oid] done_oids={} done=done_oids.has_key load=jar._storage.load while oids: oid=oids[0] del oids[0] if done(oid): continue done_oids[oid]=1 try: p, serial = load(oid, version) except: pass # Ick, a broken reference else: ref(p, oids) write(XMLrecord(oid,len(p),p)) write('\n') return file class zopedata: def __init__(self, parser, tag, attrs): self.file=parser.file write=self.file.write write('ZEXP') def append(self, data): file=self.file write=file.write pos=file.tell() file.seek(pos) write(data) def start_zopedata(parser, tag, data): return zopedata(parser, tag, data) def save_zopedata(parser, tag, data): file=parser.file write=file.write pos=file.tell() file.seek(pos) write(export_end_marker) def save_record(parser, tag, data): file=parser.file write=file.write pos=file.tell() file.seek(pos) a=data[1] if a.has_key('id'): oid=a['id'] oid=ppml.p64(int(oid)) v='' for x in data[2:]: v=v+x l=ppml.p64(len(v)) v=oid+l+v return v def importXML(jar, file, clue=''): import Shared.DC.xml.pyexpat.pyexpat pyexpat=Shared.DC.xml.pyexpat.pyexpat if type(file) is StringType: file=open(file) outfile=TemporaryFile() data=file.read() F=ppml.xmlPickler() F.end_handlers['record'] = save_record F.end_handlers['ZopeData'] = save_zopedata F.start_handlers['ZopeData'] = start_zopedata F.binary=1 F.file=outfile p=pyexpat.ParserCreate() p.CharacterDataHandler=F.handle_data p.StartElementHandler=F.unknown_starttag p.EndElementHandler=F.unknown_endtag r=p.Parse(data) outfile.seek(0) return jar.importFile(outfile,clue) Zope-2.8.7-final/lib/python/OFS/dtml/0000775000076400007640000000000010435500327016215 5ustar ajungajungZope-2.8.7-final/lib/python/OFS/dtml/findResult.dtml0000664000076400007640000001616510435500327021227 0ustar ajungajung

Displaying items &dtml-sequence-number;-&dtml-sequence-number; of items matching your query. You can revise your search terms below.

No items were found matching your query. You can revise your search terms below.

< Previous  
Next >  
[&dtml-meta_type;]
&dtml-sequence-key; (&dtml-title;)
Find objects of type:
with ids:
">
containing:
">
expr:
modified:
where the roles:
have permission:
Sort results by:
Reverse?
CHECKED> Search only in this folder
CHECKED> Search all subfolders
Simple... Advanced...
Zope-2.8.7-final/lib/python/OFS/dtml/cmassoc.dtml0000664000076400007640000000711610435500327020534 0ustar ajungajung

Select which objects should be cached using this cache manager. Only those objects for which you have the "Change cache settings" permission are shown.

checked="checked">
  &dtml-path;(&dtml-title;)

No objects matched your query.

Locate cacheable objects:
All Associated with this cache manager
Of the type(s):
Search subfolders

Zope-2.8.7-final/lib/python/OFS/dtml/findAdv.dtml0000664000076400007640000000624110435500327020455 0ustar ajungajung
Find objects of type:
with ids:
containing:
expr:
modified:
where the roles:
have permission:
Sort results by:
Reverse?
Search only in this folder
Search all subfolders
Simple...
Zope-2.8.7-final/lib/python/OFS/dtml/propertysheets.dtml0000664000076400007640000000266410435500327022207 0ustar ajungajung
&dtml-id; () This item has been modified in this session This item has been modified in another session
Name: Namespace:
Zope-2.8.7-final/lib/python/OFS/dtml/importExport.dtml0000664000076400007640000000610210435500327021612 0ustar ajungajung

You can export Zope objects to a file in order to transfer them to a different Zope installation. You can either choose to download the export file to your local machine, or save it in the "var" directory of your Zope installation on the server.

Export object id
" class="form-element"/>
Export to
Download to local machine
Save to file on server
 
XML format?

You may import Zope objects which have been previously exported to a file, by placing the file in the "import" directory of your Zope installation on the server. You should create the "import" directory in the root of your Zope installation if it does not yet exist.

Note that by default, you will become the owner of the objects that you are importing. If you wish the imported objects to retain their existing ownership information, select "retain existing ownership information".

Import file name
Ownership
Take ownership of imported objects
Retain existing ownership information
Zope-2.8.7-final/lib/python/OFS/dtml/history.dtml0000664000076400007640000000470610435500327020607 0ustar ajungajung
< Later Revisions  
Earlier Revisions >  
(&dtml-user_name;)
&dtml-description;
revision: &dtml-revision;

No change history is available for this object.

Zope-2.8.7-final/lib/python/OFS/dtml/main.dtml0000664000076400007640000002407710435500327020035 0ustar ajungajung
 
1">
&rkey=meta_type" onMouseOver="window.status='Sort objects by type'; return true" onMouseOut="window.status=''; return true">TypeType
&rkey=id" onMouseOver="window.status='Sort objects by name'; return true" onMouseOut="window.status=''; return true">NameName
&rkey=get_size" onMouseOver="window.status='Sort objects by size'; return true" onMouseOut="window.status=''; return true">SizeSize
&rkey=bobobase_modification_time" onMouseOver="window.status='Sort objects by modification time'; return true" onMouseOut="window.status=''; return true">Last ModifiedLast Modified
PositionPosition
&dtml.missing-alt; &dtml-meta_type;  
&dtml-sequence-key; (&dtml-title;) This item has been modified in this version This item has been modified in another version (&dtml-locked_in_version;)
1 Kb 1048576"> Mb Kb    
...
/ by
There are currently no items in &dtml-title_or_id;

Zope-2.8.7-final/lib/python/OFS/dtml/fileEdit.dtml0000664000076400007640000000641110435500327020626 0ustar ajungajung

You can update the data for this file object using the form below. Select a data file from your local computer by clicking the browse button and click upload to update the contents of the file. You may also edit the file content directly if the content is a text type and small enough to be edited in a text area.

Title
Content Type
Precondition
Last Modified
File Size
bytes
Locked by WebDAV

File Data

Locked by WebDAV
Zope-2.8.7-final/lib/python/OFS/dtml/cacheable.dtml0000664000076400007640000000246010435500327020770 0ustar ajungajung
checked="checked">Cache this view of the object
Cache this object using:

Cache Settings

Zope-2.8.7-final/lib/python/OFS/dtml/folderAdd.dtml0000664000076400007640000000411210435500327020761 0ustar ajungajung

A Folder contains other objects. Use Folders to organize your web objects in to logical groups. The create public interface option creates an index document inside the Folder to give the Folder a default HTML representation. The create user folder option creates a User Folder inside the Folder to hold authorization information for the Folder.

Id
Title
Zope-2.8.7-final/lib/python/OFS/dtml/methodAdd.dtml0000664000076400007640000000371210435500327020773 0ustar ajungajung

A DTML Method is used to hold scripting tags and text content. It can contain HTML, XML, plain text, structured-text, etcetera. A DTML Method can contain DTML scripting tags which operate in the context of the DTML Method's containing object. A DTML Method differs from a DTML Document in that it does not have properties and calls methods on its container object.

You may create a new DTML Method object using the form below. You may also choose to upload an existing html file from your local computer by clicking the Browse button.

Id
Title
File
Zope-2.8.7-final/lib/python/OFS/dtml/documentProxy.dtml0000664000076400007640000000232310435500327021757 0ustar ajungajung

Proxy roles allow you to control the access that a DTML document or method has. Proxy roles replace the roles of the user who is viewing the document or method. This can be used to both expand and limit access to resources. Select the proxy roles for this object from the list below.

Proxy Roles
Zope-2.8.7-final/lib/python/OFS/dtml/imageEdit.dtml0000664000076400007640000000536210435500327020775 0ustar ajungajung

You can update the data for this &dtml-kind; using the form below. Select a data file from your local computer by clicking the browse button and click upload to update the contents of the &dtml-kind;.

Title
Content Type
Preview
250"> 250">
Last Modified
File Size
bytes
Locked by WebDAV

File Data

Locked by WebDAV
Zope-2.8.7-final/lib/python/OFS/dtml/historyCompare.dtml0000664000076400007640000000134110435500327022106 0ustar ajungajung
Changes to &dtml-id; as of
to get to &dtml-id; as of

This object does not provide comparison support.

Zope-2.8.7-final/lib/python/OFS/dtml/properties.dtml0000664000076400007640000002243310435500327021277 0ustar ajungajung A site-global encoding specification in a property. Note that this feature only works if there are no unicode objects around. This means that this feature is not likely to be supported in all future versions of zope. Thankfully no site-global encoding specification in a property. We can set UTF-8, and unicode properties will work.

You are currently working in version &dtml-Principia-Version;

Properties allow you to assign simple values to Zope objects. To change property values, edit the values and click "Save Changes".

 
Name
Value
Type
string:list" value="&dtml-id;" id="cb-&dtml-id;">
"> "> &dtml-type;" size="35" value=""> &dtml-type;" size="35" value=""> CHECKED> &dtml-type;" size="35" value="&dtml-sequence-item; ">
No value for &dtml-select_variable;.
No value for &dtml-select_variable;.
Unknown property type
&dtml-type;
Warning: be aware that removing 'title' without re-adding it might be dangerous.
 
This needs some community review before exposing it officially.  

Properties allow you to assign simple values to Zope objects. There are currently no properties defined for this item. To add a property, enter a name, type and value and click the "Add" button.

To add a new property, enter a name, type and value for the new property and click the "Add" button.

Name
string" size="30" value=""/> Type
Value
Zope-2.8.7-final/lib/python/OFS/dtml/imageView.dtml0000664000076400007640000000040010435500327021006 0ustar ajungajung

width="&dtml-width;" height="&dtml-height;" alt="&dtml-title_or_id;" />

Zope-2.8.7-final/lib/python/OFS/dtml/findFrame.dtml0000664000076400007640000000117110435500327020772 0ustar ajungajung Find MARGINWIDTH="2" MARGINHEIGHT="2" SCROLLING="auto"> Management interfaces require the use of a <B>frames-capable</B> web browser. Zope-2.8.7-final/lib/python/OFS/dtml/propertyType.dtml0000664000076400007640000001053010435500327021624 0ustar ajungajung Change Property Types

You are currently working in version &dtml-Principia-Version;

To change property names and values, edit them and click "Save Changes". To edit properties using their new type, select the new types and click "Edit with new Types"

Property Name Value New Type
value="&dtml-sequence-item; " value="" value="&dtml-new_value;" > CHECKED> No value for &dtml-select_variable;. No value for &dtml-select_variable;. Unknown property type

No properties were selected for this item.

Zope-2.8.7-final/lib/python/OFS/dtml/documentEdit.dtml0000664000076400007640000000445310435500327021531 0ustar ajungajung

You may edit the source for this document using the form below. You may also upload the source for this document from a local file. Click the browse button to select a local file to upload.

Title
Locked by WebDAV   
File  
Locked by WebDAV
Zope-2.8.7-final/lib/python/OFS/dtml/editedDialog.dtml0000664000076400007640000000113110435500327021451 0ustar ajungajung

!


&dtml-title_and_id; has been successfully edited.
Zope-2.8.7-final/lib/python/OFS/dtml/addOrderedFolder.dtml0000664000076400007640000000414410435500327022273 0ustar ajungajung

An ordered Folder contains other objects. Use Folders to organize your web objects in to logical groups. The create public interface option creates an index document inside the Folder to give the Folder a default HTML representation. The create user folder option creates a User Folder inside the Folder to hold authorization information for the Folder.

Id
Title
Zope-2.8.7-final/lib/python/OFS/dtml/documentAdd.dtml0000664000076400007640000000355710435500327021340 0ustar ajungajung

A DTML Document is used to hold text content. It can contain HTML, XML, plain text, structured-text, etcetera. A DTML document can contain DTML scripting tags. A DTML Document differs from a DTML Method in that it has properties and does not call methods on its container object.

You may create a new DTML Document using the form below. You may also choose to upload an existing html file from your local computer by clicking the Browse button.

Id
Title
File
Zope-2.8.7-final/lib/python/OFS/dtml/renameForm.dtml0000664000076400007640000000313710435500327021176 0ustar ajungajung
&dtml-id;
to: " value="&dtml-id;" /> may not be renamed.



You must select an item to rename.

Zope-2.8.7-final/lib/python/OFS/dtml/brokenEdit.dtml0000664000076400007640000000072210435500327021166 0ustar ajungajung

This object is broken because the &dtml-product_name; product that created it is no longer installed or is installed incorrectly. Please contact the product maintainer for assistance.

Note that the data associated with this product has not been lost, and will be accessible again if the product is reinstalled.

Zope-2.8.7-final/lib/python/OFS/dtml/cacheNamespaceKeys.dtml0000664000076400007640000000052710435500327022617 0ustar ajungajung

Names from the DTML namespace to use as cache keys:


Zope-2.8.7-final/lib/python/OFS/dtml/findForm.dtml0000664000076400007640000000437110435500327020650 0ustar ajungajung

Find allows you to locate Zope objects based on different criteria. For more find choices choose the Advanced find option.

Find objects of type:
with ids:
containing:
modified:
Search only in this folder
Search all subfolders
Advanced...
Zope-2.8.7-final/lib/python/OFS/dtml/imageAdd.dtml0000664000076400007640000000247710435500327020604 0ustar ajungajung

Select a file to upload from your local computer by clicking the Browse button.

Id
Title
File
Zope-2.8.7-final/lib/python/OFS/mime.types0000664000076400007640000000136410435500327017276 0ustar ajungajungapplication-x-cdf cdf application/msword doc dot wiz application/pdf pdf application/pkcs7-mime p7c application/vnd.mozilla.xul+xml xul application/vnd.ms-excel xlb xls application/vnd.ms-powerpoint pot ppa pps ppt pwz application/x-javascript js application/x-pkcs12 p12 pfx application/x-pn-realaudio ram audio/mpeg mp3 audio/x-pn-realaudio ra image/pict pct pic pict message/rfc822 eml nws mht mhtml text/css css text/plain bat c h pl ksh text/x-vcard vcf text/xml xml xsl video/mpeg m1v mpa Zope-2.8.7-final/lib/python/OFS/Image.py0000664000076400007640000007651010435500327016662 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Image object $Id: Image.py 65863 2006-03-07 21:54:19Z shh $ """ import Globals, struct from OFS.content_types import guess_content_type from Globals import DTMLFile from PropertyManager import PropertyManager from AccessControl.Role import RoleManager from webdav.common import rfc1123_date from webdav.Lockable import ResourceLockedError from webdav.WriteLockInterface import WriteLockInterface from SimpleItem import Item_w__name__ from cStringIO import StringIO from Globals import Persistent from Acquisition import Implicit from DateTime import DateTime from Cache import Cacheable from mimetools import choose_boundary from ZPublisher import HTTPRangeSupport from ZPublisher.HTTPRequest import FileUpload from ZPublisher.Iterators import filestream_iterator from zExceptions import Redirect from cgi import escape import transaction StringType=type('') manage_addFileForm=DTMLFile('dtml/imageAdd', globals(),Kind='File',kind='file') def manage_addFile(self,id,file='',title='',precondition='', content_type='', REQUEST=None): """Add a new File object. Creates a new File object 'id' with the contents of 'file'""" id=str(id) title=str(title) content_type=str(content_type) precondition=str(precondition) id, title = cookId(id, title, file) self=self.this() # First, we create the file without data: self._setObject(id, File(id,title,'',content_type, precondition)) # Now we "upload" the data. By doing this in two steps, we # can use a database trick to make the upload more efficient. if file: self._getOb(id).manage_upload(file) if content_type: self._getOb(id).content_type=content_type if REQUEST is not None: REQUEST['RESPONSE'].redirect(self.absolute_url()+'/manage_main') class File(Persistent, Implicit, PropertyManager, RoleManager, Item_w__name__, Cacheable): """A File object is a content object for arbitrary files.""" __implements__ = (WriteLockInterface, HTTPRangeSupport.HTTPRangeInterface) meta_type='File' precondition='' size=None alt='' manage_editForm =DTMLFile('dtml/fileEdit',globals(), Kind='File',kind='file') manage_editForm._setName('manage_editForm') manage=manage_main=manage_editForm manage_uploadForm=manage_editForm manage_options=( ( {'label':'Edit', 'action':'manage_main', 'help':('OFSP','File_Edit.stx')}, {'label':'View', 'action':'', 'help':('OFSP','File_View.stx')}, ) + PropertyManager.manage_options + RoleManager.manage_options + Item_w__name__.manage_options + Cacheable.manage_options ) __ac_permissions__=( ('View management screens', ('manage', 'manage_main',)), ('Change Images and Files', ('manage_edit','manage_upload','PUT')), ('View', ('index_html', 'view_image_or_file', 'get_size', 'getContentType', 'PrincipiaSearchSource', '')), ('FTP access', ('manage_FTPstat','manage_FTPget','manage_FTPlist')), ('Delete objects', ('DELETE',)), ) _properties=({'id':'title', 'type': 'string'}, {'id':'alt', 'type':'string'}, {'id':'content_type', 'type':'string'}, ) def __init__(self, id, title, file, content_type='', precondition=''): self.__name__=id self.title=title self.precondition=precondition data, size = self._read_data(file) content_type=self._get_content_type(file, data, id, content_type) self.update_data(data, content_type, size) def id(self): return self.__name__ def _if_modified_since_request_handler(self, REQUEST, RESPONSE): # HTTP If-Modified-Since header handling: return True if # we can handle this request by returning a 304 response header=REQUEST.get_header('If-Modified-Since', None) if header is not None: header=header.split( ';')[0] # Some proxies seem to send invalid date strings for this # header. If the date string is not valid, we ignore it # rather than raise an error to be generally consistent # with common servers such as Apache (which can usually # understand the screwy date string as a lucky side effect # of the way they parse it). # This happens to be what RFC2616 tells us to do in the face of an # invalid date. try: mod_since=long(DateTime(header).timeTime()) except: mod_since=None if mod_since is not None: if self._p_mtime: last_mod = long(self._p_mtime) else: last_mod = long(0) if last_mod > 0 and last_mod <= mod_since: RESPONSE.setHeader('Last-Modified', rfc1123_date(self._p_mtime)) RESPONSE.setHeader('Content-Type', self.content_type) RESPONSE.setHeader('Accept-Ranges', 'bytes') RESPONSE.setStatus(304) return True def _range_request_handler(self, REQUEST, RESPONSE): # HTTP Range header handling: return True if we've served a range # chunk out of our data. range = REQUEST.get_header('Range', None) request_range = REQUEST.get_header('Request-Range', None) if request_range is not None: # Netscape 2 through 4 and MSIE 3 implement a draft version # Later on, we need to serve a different mime-type as well. range = request_range if_range = REQUEST.get_header('If-Range', None) if range is not None: ranges = HTTPRangeSupport.parseRange(range) if if_range is not None: # Only send ranges if the data isn't modified, otherwise send # the whole object. Support both ETags and Last-Modified dates! if len(if_range) > 1 and if_range[:2] == 'ts': # ETag: if if_range != self.http__etag(): # Modified, so send a normal response. We delete # the ranges, which causes us to skip to the 200 # response. ranges = None else: # Date date = if_range.split( ';')[0] try: mod_since=long(DateTime(date).timeTime()) except: mod_since=None if mod_since is not None: if self._p_mtime: last_mod = long(self._p_mtime) else: last_mod = long(0) if last_mod > mod_since: # Modified, so send a normal response. We delete # the ranges, which causes us to skip to the 200 # response. ranges = None if ranges: # Search for satisfiable ranges. satisfiable = 0 for start, end in ranges: if start < self.size: satisfiable = 1 break if not satisfiable: RESPONSE.setHeader('Content-Range', 'bytes */%d' % self.size) RESPONSE.setHeader('Accept-Ranges', 'bytes') RESPONSE.setHeader('Last-Modified', rfc1123_date(self._p_mtime)) RESPONSE.setHeader('Content-Type', self.content_type) RESPONSE.setHeader('Content-Length', self.size) RESPONSE.setStatus(416) return True ranges = HTTPRangeSupport.expandRanges(ranges, self.size) if len(ranges) == 1: # Easy case, set extra header and return partial set. start, end = ranges[0] size = end - start RESPONSE.setHeader('Last-Modified', rfc1123_date(self._p_mtime)) RESPONSE.setHeader('Content-Type', self.content_type) RESPONSE.setHeader('Content-Length', size) RESPONSE.setHeader('Accept-Ranges', 'bytes') RESPONSE.setHeader('Content-Range', 'bytes %d-%d/%d' % (start, end - 1, self.size)) RESPONSE.setStatus(206) # Partial content data = self.data if type(data) is StringType: RESPONSE.write(data[start:end]) return True # Linked Pdata objects. Urgh. pos = 0 while data is not None: l = len(data.data) pos = pos + l if pos > start: # We are within the range lstart = l - (pos - start) if lstart < 0: lstart = 0 # find the endpoint if end <= pos: lend = l - (pos - end) # Send and end transmission RESPONSE.write(data[lstart:lend]) break # Not yet at the end, transmit what we have. RESPONSE.write(data[lstart:]) data = data.next return True else: boundary = choose_boundary() # Calculate the content length size = (8 + len(boundary) + # End marker length len(ranges) * ( # Constant lenght per set 49 + len(boundary) + len(self.content_type) + len('%d' % self.size))) for start, end in ranges: # Variable length per set size = (size + len('%d%d' % (start, end - 1)) + end - start) # Some clients implement an earlier draft of the spec, they # will only accept x-byteranges. draftprefix = (request_range is not None) and 'x-' or '' RESPONSE.setHeader('Content-Length', size) RESPONSE.setHeader('Accept-Ranges', 'bytes') RESPONSE.setHeader('Last-Modified', rfc1123_date(self._p_mtime)) RESPONSE.setHeader('Content-Type', 'multipart/%sbyteranges; boundary=%s' % ( draftprefix, boundary)) RESPONSE.setStatus(206) # Partial content data = self.data # The Pdata map allows us to jump into the Pdata chain # arbitrarily during out-of-order range searching. pdata_map = {} pdata_map[0] = data for start, end in ranges: RESPONSE.write('\r\n--%s\r\n' % boundary) RESPONSE.write('Content-Type: %s\r\n' % self.content_type) RESPONSE.write( 'Content-Range: bytes %d-%d/%d\r\n\r\n' % ( start, end - 1, self.size)) if type(data) is StringType: RESPONSE.write(data[start:end]) else: # Yippee. Linked Pdata objects. The following # calculations allow us to fast-forward through the # Pdata chain without a lot of dereferencing if we # did the work already. first_size = len(pdata_map[0].data) if start < first_size: closest_pos = 0 else: closest_pos = ( ((start - first_size) >> 16 << 16) + first_size) pos = min(closest_pos, max(pdata_map.keys())) data = pdata_map[pos] while data is not None: l = len(data.data) pos = pos + l if pos > start: # We are within the range lstart = l - (pos - start) if lstart < 0: lstart = 0 # find the endpoint if end <= pos: lend = l - (pos - end) # Send and loop to next range RESPONSE.write(data[lstart:lend]) break # Not yet at the end, transmit what we have. RESPONSE.write(data[lstart:]) data = data.next # Store a reference to a Pdata chain link so we # don't have to deref during this request again. pdata_map[pos] = data # Do not keep the link references around. del pdata_map RESPONSE.write('\r\n--%s--\r\n' % boundary) return True def index_html(self, REQUEST, RESPONSE): """ The default view of the contents of a File or Image. Returns the contents of the file or image. Also, sets the Content-Type HTTP header to the objects content type. """ if self._if_modified_since_request_handler(REQUEST, RESPONSE): # we were able to handle this by returning a 304 # unfortunately, because the HTTP cache manager uses the cache # API, and because 304 responses are required to carry the Expires # header for HTTP/1.1, we need to call ZCacheable_set here. # This is nonsensical for caches other than the HTTP cache manager # unfortunately. self.ZCacheable_set(None) return '' if self.precondition and hasattr(self, str(self.precondition)): # Grab whatever precondition was defined and then # execute it. The precondition will raise an exception # if something violates its terms. c=getattr(self, str(self.precondition)) if hasattr(c,'isDocTemp') and c.isDocTemp: c(REQUEST['PARENTS'][1],REQUEST) else: c() if self._range_request_handler(REQUEST, RESPONSE): # we served a chunk of content in response to a range request. return '' RESPONSE.setHeader('Last-Modified', rfc1123_date(self._p_mtime)) RESPONSE.setHeader('Content-Type', self.content_type) RESPONSE.setHeader('Content-Length', self.size) RESPONSE.setHeader('Accept-Ranges', 'bytes') if self.ZCacheable_isCachingEnabled(): result = self.ZCacheable_get(default=None) if result is not None: # We will always get None from RAMCacheManager and HTTP # Accelerated Cache Manager but we will get # something implementing the IStreamIterator interface # from a "FileCacheManager" return result self.ZCacheable_set(None) data=self.data if type(data) is type(''): RESPONSE.setBase(None) return data while data is not None: RESPONSE.write(data.data) data=data.next return '' def view_image_or_file(self, URL1): """ The default view of the contents of the File or Image. """ raise Redirect, URL1 def PrincipiaSearchSource(self): """ Allow file objects to be searched. """ if self.content_type.startswith('text/'): return str(self.data) return '' # private update_data__roles__=() def update_data(self, data, content_type=None, size=None): if content_type is not None: self.content_type=content_type if size is None: size=len(data) self.size=size self.data=data self.ZCacheable_invalidate() self.ZCacheable_set(None) self.http__refreshEtag() def manage_edit(self, title, content_type, precondition='', filedata=None, REQUEST=None): """ Changes the title and content type attributes of the File or Image. """ if self.wl_isLocked(): raise ResourceLockedError, "File is locked via WebDAV" self.title=str(title) self.content_type=str(content_type) if precondition: self.precondition=str(precondition) elif self.precondition: del self.precondition if filedata is not None: self.update_data(filedata, content_type, len(filedata)) else: self.ZCacheable_invalidate() if REQUEST: message="Saved changes." return self.manage_main(self,REQUEST,manage_tabs_message=message) def manage_upload(self,file='',REQUEST=None): """ Replaces the current contents of the File or Image object with file. The file or images contents are replaced with the contents of 'file'. """ if self.wl_isLocked(): raise ResourceLockedError, "File is locked via WebDAV" data, size = self._read_data(file) content_type=self._get_content_type(file, data, self.__name__, 'application/octet-stream') self.update_data(data, content_type, size) if REQUEST: message="Saved changes." return self.manage_main(self,REQUEST,manage_tabs_message=message) def _get_content_type(self, file, body, id, content_type=None): headers=getattr(file, 'headers', None) if headers and headers.has_key('content-type'): content_type=headers['content-type'] else: if type(body) is not type(''): body=body.data content_type, enc=guess_content_type( getattr(file, 'filename',id), body, content_type) return content_type def _read_data(self, file): n=1 << 16 if type(file) is StringType: size=len(file) if size < n: return file, size # Big string: cut it into smaller chunks file = StringIO(file) if isinstance(file, FileUpload) and not file: raise ValueError, 'File not specified' if hasattr(file, '__class__') and file.__class__ is Pdata: size=len(file) return file, size seek=file.seek read=file.read seek(0,2) size=end=file.tell() if size <= 2*n: seek(0) if size < n: return read(size), size return Pdata(read(size)), size # Make sure we have an _p_jar, even if we are a new object, by # doing a sub-transaction commit. transaction.savepoint(optimistic=True) if self._p_jar is None: # Ugh seek(0) return Pdata(read(size)), size # Now we're going to build a linked list from back # to front to minimize the number of database updates # and to allow us to get things out of memory as soon as # possible. next = None while end > 0: pos = end-n if pos < n: pos = 0 # we always want at least n bytes seek(pos) # Create the object and assign it a next pointer # in the same transaction, so that there is only # a single database update for it. data = Pdata(read(end-pos)) self._p_jar.add(data) data.next = next # Save the object so that we can release its memory. transaction.savepoint(optimistic=True) data._p_deactivate() # The object should be assigned an oid and be a ghost. assert data._p_oid is not None assert data._p_state == -1 next = data end = pos return next, size def PUT(self, REQUEST, RESPONSE): """Handle HTTP PUT requests""" self.dav__init(REQUEST, RESPONSE) self.dav__simpleifhandler(REQUEST, RESPONSE, refresh=1) type=REQUEST.get_header('content-type', None) file=REQUEST['BODYFILE'] data, size = self._read_data(file) content_type=self._get_content_type(file, data, self.__name__, type or self.content_type) self.update_data(data, content_type, size) RESPONSE.setStatus(204) return RESPONSE def get_size(self): """Get the size of a file or image. Returns the size of the file or image. """ size=self.size if size is None: size=len(self.data) return size # deprecated; use get_size! getSize=get_size def getContentType(self): """Get the content type of a file or image. Returns the content type (MIME type) of a file or image. """ return self.content_type def __str__(self): return str(self.data) def __len__(self): return 1 def manage_FTPget(self): """Return body for ftp.""" RESPONSE = self.REQUEST.RESPONSE if self.ZCacheable_isCachingEnabled(): result = self.ZCacheable_get(default=None) if result is not None: # We will always get None from RAMCacheManager but we will get # something implementing the IStreamIterator interface # from FileCacheManager. # the content-length is required here by HTTPResponse, even # though FTP doesn't use it. RESPONSE.setHeader('Content-Length', self.size) return result data = self.data if type(data) is type(''): RESPONSE.setBase(None) return data while data is not None: RESPONSE.write(data.data) data = data.next return '' manage_addImageForm=DTMLFile('dtml/imageAdd',globals(), Kind='Image',kind='image') def manage_addImage(self, id, file, title='', precondition='', content_type='', REQUEST=None): """ Add a new Image object. Creates a new Image object 'id' with the contents of 'file'. """ id=str(id) title=str(title) content_type=str(content_type) precondition=str(precondition) id, title = cookId(id, title, file) self=self.this() # First, we create the image without data: self._setObject(id, Image(id,title,'',content_type, precondition)) # Now we "upload" the data. By doing this in two steps, we # can use a database trick to make the upload more efficient. if file: self._getOb(id).manage_upload(file) if content_type: self._getOb(id).content_type=content_type if REQUEST is not None: try: url=self.DestinationURL() except: url=REQUEST['URL1'] REQUEST.RESPONSE.redirect('%s/manage_main' % url) return id def getImageInfo(data): data = str(data) size = len(data) height = -1 width = -1 content_type = '' # handle GIFs if (size >= 10) and data[:6] in ('GIF87a', 'GIF89a'): # Check to see if content_type is correct content_type = 'image/gif' w, h = struct.unpack("= 24) and (data[:8] == '\211PNG\r\n\032\n') and (data[12:16] == 'IHDR')): content_type = 'image/png' w, h = struct.unpack(">LL", data[16:24]) width = int(w) height = int(h) # Maybe this is for an older PNG version. elif (size >= 16) and (data[:8] == '\211PNG\r\n\032\n'): # Check to see if we have the right content type content_type = 'image/png' w, h = struct.unpack(">LL", data[8:16]) width = int(w) height = int(h) # handle JPEGs elif (size >= 2) and (data[:2] == '\377\330'): content_type = 'image/jpeg' jpeg = StringIO(data) jpeg.read(2) b = jpeg.read(1) try: while (b and ord(b) != 0xDA): while (ord(b) != 0xFF): b = jpeg.read(1) while (ord(b) == 0xFF): b = jpeg.read(1) if (ord(b) >= 0xC0 and ord(b) <= 0xC3): jpeg.read(3) h, w = struct.unpack(">HH", jpeg.read(4)) break else: jpeg.read(int(struct.unpack(">H", jpeg.read(2))[0])-2) b = jpeg.read(1) width = int(w) height = int(h) except: pass return content_type, width, height class Image(File): """Image objects can be GIF, PNG or JPEG and have the same methods as File objects. Images also have a string representation that renders an HTML 'IMG' tag. """ __implements__ = (WriteLockInterface,) meta_type='Image' height='' width='' __ac_permissions__=( ('View management screens', ('manage', 'manage_main',)), ('Change Images and Files', ('manage_edit','manage_upload','PUT')), ('View', ('index_html', 'tag', 'view_image_or_file', 'get_size', 'getContentType', '')), ('FTP access', ('manage_FTPstat','manage_FTPget','manage_FTPlist')), ('Delete objects', ('DELETE',)), ) _properties=({'id':'title', 'type': 'string'}, {'id':'content_type', 'type':'string','mode':'w'}, {'id':'height', 'type':'string'}, {'id':'width', 'type':'string'}, ) manage_options=( ({'label':'Edit', 'action':'manage_main', 'help':('OFSP','Image_Edit.stx')}, {'label':'View', 'action':'view_image_or_file', 'help':('OFSP','Image_View.stx')},) + PropertyManager.manage_options + RoleManager.manage_options + Item_w__name__.manage_options + Cacheable.manage_options ) manage_editForm =DTMLFile('dtml/imageEdit',globals(), Kind='Image',kind='image') view_image_or_file =DTMLFile('dtml/imageView',globals()) manage_editForm._setName('manage_editForm') manage=manage_main=manage_editForm manage_uploadForm=manage_editForm # private update_data__roles__=() def update_data(self, data, content_type=None, size=None): if size is None: size=len(data) self.size=size self.data=data ct, width, height = getImageInfo(data) if ct: content_type = ct if width >= 0 and height >= 0: self.width = width self.height = height # Now we should have the correct content type, or still None if content_type is not None: self.content_type = content_type self.ZCacheable_invalidate() self.ZCacheable_set(None) self.http__refreshEtag() def __str__(self): return self.tag() def tag(self, height=None, width=None, alt=None, scale=0, xscale=0, yscale=0, css_class=None, title=None, **args): """ Generate an HTML IMG tag for this image, with customization. Arguments to self.tag() can be any valid attributes of an IMG tag. 'src' will always be an absolute pathname, to prevent redundant downloading of images. Defaults are applied intelligently for 'height', 'width', and 'alt'. If specified, the 'scale', 'xscale', and 'yscale' keyword arguments will be used to automatically adjust the output height and width values of the image tag. Since 'class' is a Python reserved word, it cannot be passed in directly in keyword arguments which is a problem if you are trying to use 'tag()' to include a CSS class. The tag() method will accept a 'css_class' argument that will be converted to 'class' in the output tag to work around this. """ if height is None: height=self.height if width is None: width=self.width # Auto-scaling support xdelta = xscale or scale ydelta = yscale or scale if xdelta and width: width = str(int(round(int(width) * xdelta))) if ydelta and height: height = str(int(round(int(height) * ydelta))) result='' % result def cookId(id, title, file): if not id and hasattr(file,'filename'): filename=file.filename title=title or filename id=filename[max(filename.rfind('/'), filename.rfind('\\'), filename.rfind(':'), )+1:] return id, title class Pdata(Persistent, Implicit): # Wrapper for possibly large data next=None def __init__(self, data): self.data=data def __getslice__(self, i, j): return self.data[i:j] def __len__(self): data = str(self) return len(data) def __str__(self): next=self.next if next is None: return self.data r=[self.data] while next is not None: self=next r.append(self.data) next=self.next return ''.join(r) Zope-2.8.7-final/lib/python/OFS/Document.py0000664000076400007640000000146310435500327017411 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Deprecated - use DTMLMethod $Id: Document.py 24764 2004-05-17 06:13:48Z philikon $ """ import DTMLMethod Document=DTMLMethod.DTMLMethod manage_addDocument=DTMLMethod.addDTMLMethod Zope-2.8.7-final/lib/python/OFS/DefaultObservable.py0000664000076400007640000001377310435500327021233 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Implement Observable interface (see http://www.zope.org/Members/michel/Projects/Interfaces/ObserverAndNotification) This class is intended to be used as a mixin (note that it doesn't derive from any Zope persistence classes, for instance). $Id: DefaultObservable.py 24764 2004-05-17 06:13:48Z philikon $ """ from types import StringType class DefaultObservable: """ See the Interfaces wiki for design notes: http://www.zope.org/Members/michel/Projects/Interfaces/ObserverAndNotification DefaultObservable is intended to be used as a mix-in, like so:: from OFS.SimpleItem import SimpleItem from OFS.DefaultObservable import DefaultObservable class Foo( SimpleItem, DefaultObservable ): ''' Some foo or other ''' ... def bar( self, ... ): ''' ''' ... self.notify( "bar" ) Clients register with a Foo instance using the methods of the Observable interface, e.g.:: foo.registerObserver( self.getPhysicalPath() + ( 'watchFoo',) ) When the Foo instance has its 'bar()' method called, it will notify all registered observers, passing 'bar' as the event; in this case, the client's 'watchFoo()' method will be called, with the foo object and 'bar' passed as parameters. """ def __init__( self, debug=0 ): self._observers = [] self._debug = debug def _normalize( self, observer ): # Assert that observer is a string or a sequence of strings. if type( observer ) != StringType: observer = '/'.join( observer) return observer # # Observable interface methods. # def registerObserver( self, observer ): normal = self._normalize( observer ) if self.restrictedTraverse( normal, None ): self._observers.append( normal ) else: raise NameError, observer def unregisterObserver( self, observer ): self._observers.remove( self._normalize( observer ) ) # # Convenience method for derivatives. # def notify( self, event=None ): bozos = [] for observer in self._observers: obj = self.restrictedTraverse( observer, None ) if obj is not None: try: obj( self, event ) except: bozos.append( observer ) # Veto not allowed! if self._debug: import traceback traceback.print_exc() else: bozos.append( observer ) for bozo in bozos: try: # avoid race condition if unregister() called before now self._observers.remove( bozo ) except: pass # # Unit tests # if __name__ == '__main__': class DontGoHere( Exception ): pass class Idawanna( Exception ): pass class TestSubject( DefaultObservable ): def __init__( self, paths ): DefaultObservable.__init__( self, 0 ) self.paths = paths def restrictedTraverse( self, path, default ): return self.paths.get( path, default ) callbacks = {} def recordCallback( name, subject, event ): cbrec = callbacks.get( name, None ) if cbrec is None: cbrec = callbacks[ name ] = [] cbrec.append( ( subject, event ) ) class TestObserver: def __call__( self, subject, event ): recordCallback( 'direct', subject, event ) def namedCallback( self, subject, event ): recordCallback( 'named', subject, event ) def named2Callback( self, subject, event ): recordCallback( 'named2', subject, event ) def boundCallback( self, subject, event ): recordCallback( 'bound', subject, event ) def freefuncObserver( subject, event ): recordCallback( 'freefunc', subject, event ) def tryVeto( subject, event ): """ Simulate attempted veto. """ raise Idawanna observer = TestObserver() # Simulate Zope's path traversal mechanism. paths = {} paths[ 'direct' ] = observer paths[ 'direct/named' ] = observer.namedCallback paths[ 'direct/named2' ] = observer.named2Callback paths[ 'bound' ] = observer.boundCallback paths[ 'freefunc' ] = freefuncObserver paths[ 'tryVeto' ] = tryVeto subject = TestSubject( paths ) subject.registerObserver( 'direct' ) subject.registerObserver( 'bound' ) subject.registerObserver( 'direct/named' ) subject.registerObserver( ( 'direct', 'named2' ) ) subject.registerObserver( 'freefunc' ) subject.registerObserver( 'tryVeto' ) try: subject.registerObserver( 'nonesuch' ) raise DontGoHere( 'path not found' ) except NameError: pass except: import traceback traceback.print_exc() try: subject.registerObserver( 3.1415926 ) raise DontGoHere( 'non-path' ) except TypeError: pass except: import traceback traceback.print_exc() subject.notify( 'First event' ) subject.unregisterObserver( 'bound' ) subject.notify( { 'name' : 'Second event', 'value' : 42 } ) for key in callbacks.keys(): print '[%s]' % key for cb in callbacks[ key ]: print ' %s' % `cb` Zope-2.8.7-final/lib/python/OFS/rPickle.py0000664000076400007640000000243010435500327017217 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Restricted unpickler $Id: rPickle.py 24764 2004-05-17 06:13:48Z philikon $ """ import pickle reg={} class Unpickler(pickle.Unpickler): def find_class(self, module, name): try: return reg[(module,name)] except: raise SystemError, 'Class not registered' def load_string(self): raise pickle.UnpicklingError, 'Bad pickle: Non binstring' # Public interface from cStringIO import StringIO def loads(s): """Unpickle a string""" return Unpickler(StringIO(s)).load() def register(mod, cls, obj): """Register a class""" reg[(mod,cls)]=obj def unregister(mod, cls): """Unregister a class""" del reg[(mod,cls)] Zope-2.8.7-final/lib/python/OFS/Moniker.py0000664000076400007640000000356210435500327017241 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Object monikers An object moniker is an intelligent reference to a persistent object. A moniker can be turned back into a real object that retains its correct version context and aquisition relationships via a simple interface. $Id: Moniker.py 24764 2004-05-17 06:13:48Z philikon $ """ import Globals class Moniker: """An object moniker is an intelligent reference to a persistent object. A moniker can be turned back into a real object that retains its correct version context and acquisition relationships via a simple interface.""" def __init__(self, ob=None): if ob is None: return self.idpath = ob.getPhysicalPath() def bind(self, app): "Returns the real object named by this moniker" ob = app.unrestrictedTraverse(self.idpath) return ob def dump(self): '''Returns an object which can be reconstituted by loadMoniker(). Result must be compatible with marshal.dump(). ''' return self.idpath def loadMoniker(data): '''Re-creates a Moniker object from the given data which had been generated by Moniker.dump().''' m = Moniker() m.idpath = data return m def absattr(attr): if callable(attr): return attr() return attr Zope-2.8.7-final/lib/python/OFS/DTMLDocument.py0000664000076400007640000001466210435500327020077 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """DTML Document objects. $Id: DTMLDocument.py 24764 2004-05-17 06:13:48Z philikon $ """ from ZPublisher.Converters import type_converters from Globals import HTML, DTMLFile, MessageDialog from OFS.content_types import guess_content_type from DTMLMethod import DTMLMethod, decapitate from PropertyManager import PropertyManager from webdav.common import rfc1123_date from webdav.Lockable import ResourceLockedError from webdav.WriteLockInterface import WriteLockInterface from sgmllib import SGMLParser from urllib import quote import Globals from AccessControl import getSecurityManager from zExceptions.TracebackSupplement import PathTracebackSupplement done='done' _marker = [] # Create a new marker object. class DTMLDocument(PropertyManager, DTMLMethod): """DTML Document objects are DocumentTemplate.HTML objects that act as methods whose 'self' is the DTML Document itself.""" __implements__ = (WriteLockInterface,) meta_type='DTML Document' icon ='p_/dtmldoc' manage_options=( DTMLMethod.manage_options[:2] + PropertyManager.manage_options + DTMLMethod.manage_options[2:] ) ps = DTMLMethod.__ac_permissions__ __ac_permissions__=( ps[0], ('Change DTML Documents', ps[1][1]), ps[2], ps[3], ps[4]) del ps def manage_edit(self,data,title,SUBMIT='Change',dtpref_cols='100%', dtpref_rows='20',REQUEST=None): """ Replaces a Documents contents with Data, Title with Title. The SUBMIT parameter is also used to change the size of the editing area on the default Document edit screen. If the value is "Smaller", the rows and columns decrease by 5. If the value is "Bigger", the rows and columns increase by 5. If any other or no value is supplied, the data gets checked for DTML errors and is saved. """ self._validateProxy(REQUEST) if self._size_changes.has_key(SUBMIT): return self._er(data,title,SUBMIT,dtpref_cols,dtpref_rows,REQUEST) if self.wl_isLocked(): raise ResourceLockedError, ( 'This document has been locked via WebDAV.') self.title=str(title) if type(data) is not type(''): data=data.read() self.munge(data) self.ZCacheable_invalidate() if REQUEST: message="Content changed." return self.manage_main(self,REQUEST,manage_tabs_message=message) def manage_upload(self,file='', REQUEST=None): """Replace the contents of the document with the text in file.""" self._validateProxy(REQUEST) if self.wl_isLocked(): raise ResourceLockedError, ( 'This document has been locked via WebDAV.') if type(file) is not type(''): if REQUEST and not file: raise ValueError, 'No file specified' file=file.read() self.munge(file) self.ZCacheable_invalidate() if REQUEST: message="Content uploaded." return self.manage_main(self,REQUEST,manage_tabs_message=message) def __call__(self, client=None, REQUEST={}, RESPONSE=None, **kw): """Render the document given a client object, REQUEST mapping, Response, and key word arguments.""" if not self._cache_namespace_keys: data = self.ZCacheable_get(default=_marker) if data is not _marker: # Return cached results. return data __traceback_supplement__ = (PathTracebackSupplement, self) kw['document_id'] =self.getId() kw['document_title']=self.title if hasattr(self, 'aq_explicit'): bself=self.aq_explicit else: bself=self security=getSecurityManager() security.addContext(self) try: if client is None: # Called as subtemplate, so don't need error propigation! r=apply(HTML.__call__, (self, bself, REQUEST), kw) if RESPONSE is None: result = r else: result = decapitate(r, RESPONSE) if not self._cache_namespace_keys: self.ZCacheable_set(result) return result r=apply(HTML.__call__, (self, (client, bself), REQUEST), kw) if type(r) is not type('') or RESPONSE is None: if not self._cache_namespace_keys: self.ZCacheable_set(r) return r finally: security.removeContext(self) have_key=RESPONSE.headers.has_key if not (have_key('content-type') or have_key('Content-Type')): if self.__dict__.has_key('content_type'): c=self.content_type else: c, e=guess_content_type(self.__name__, r) RESPONSE.setHeader('Content-Type', c) result = decapitate(r, RESPONSE) if not self._cache_namespace_keys: self.ZCacheable_set(result) return result Globals.default__class_init__(DTMLDocument) default_dd_html="""

This is the Document.

""" addForm=DTMLFile('dtml/documentAdd', globals()) def addDTMLDocument(self, id, title='', file='', REQUEST=None, submit=None): """Add a DTML Document object with the contents of file. If 'file' is empty, default document text is used. """ if type(file) is not type(''): file=file.read() if not file: file=default_dd_html id=str(id) title=str(title) ob=DTMLDocument(file, __name__=id) ob.title=title id=self._setObject(id, ob) if REQUEST is not None: try: u=self.DestinationURL() except: u=REQUEST['URL1'] if submit==" Add and Edit ": u="%s/%s" % (u,quote(id)) REQUEST.RESPONSE.redirect(u+'/manage_main') return '' Zope-2.8.7-final/lib/python/OFS/ObjectManager.py0000664000076400007640000006401410435500327020335 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Object Manager $Id: ObjectManager.py 66022 2006-03-15 08:16:56Z yuppie $ """ import sys, fnmatch, copy, os, re import marshal from cgi import escape from cStringIO import StringIO from types import StringType, UnicodeType import App.Management, Acquisition, Globals, Products import App.FactoryDispatcher, Products from Globals import DTMLFile, Persistent from Globals import MessageDialog, default__class_init__ from Globals import REPLACEABLE, NOT_REPLACEABLE, UNIQUE from webdav.NullResource import NullResource from webdav.Collection import Collection from Acquisition import aq_base from webdav.Lockable import ResourceLockedError from ZODB.POSException import ConflictError import App.Common from App.config import getConfiguration from AccessControl import getSecurityManager from AccessControl.ZopeSecurityPolicy import getRoles from zLOG import LOG, ERROR from zExceptions import BadRequest from OFS.Traversable import Traversable import CopySupport # the name BadRequestException is relied upon by 3rd-party code BadRequestException = BadRequest import XMLExportImport customImporters={ XMLExportImport.magic: XMLExportImport.importXML, } bad_id=re.compile(r'[^a-zA-Z0-9-_~,.$\(\)# @]').search def checkValidId(self, id, allow_dup=0): # If allow_dup is false, an error will be raised if an object # with the given id already exists. If allow_dup is true, # only check that the id string contains no illegal chars; # check_valid_id() will be called again later with allow_dup # set to false before the object is added. if not id or not isinstance(id, StringType): if isinstance(id, UnicodeType): id = escape(id) raise BadRequest, ('Empty or invalid id specified', id) if bad_id(id) is not None: raise BadRequest, ( 'The id "%s" contains characters illegal in URLs.' % escape(id)) if id in ('.', '..'): raise BadRequest, ( 'The id "%s" is invalid because it is not traversable.' % id) if id.startswith('_'): raise BadRequest, ( 'The id "%s" is invalid because it begins with an underscore.' % id) if id.startswith('aq_'): raise BadRequest, ( 'The id "%s" is invalid because it begins with "aq_".' % id) if id.endswith('__'): raise BadRequest, ( 'The id "%s" is invalid because it ends with two underscores.' % id) if not allow_dup: obj = getattr(self, id, None) if obj is not None: # An object by the given id exists either in this # ObjectManager or in the acquisition path. flags = getattr(obj, '__replaceable__', NOT_REPLACEABLE) if hasattr(aq_base(self), id): # The object is located in this ObjectManager. if not flags & REPLACEABLE: raise BadRequest, ( 'The id "%s" is invalid - it is already in use.' % id) # else the object is replaceable even if the UNIQUE # flag is set. elif flags & UNIQUE: raise BadRequest, ('The id "%s" is reserved.' % id) if id == 'REQUEST': raise BadRequest, 'REQUEST is a reserved name.' if '/' in id: raise BadRequest, ( 'The id "%s" contains characters illegal in URLs.' % id) class BeforeDeleteException(Exception): pass # raise to veto deletion class BreakoutException(Exception): pass # raised to break out of loops _marker=[] class ObjectManager( CopySupport.CopyContainer, App.Management.Navigation, App.Management.Tabs, Acquisition.Implicit, Persistent, Collection, Traversable, ): """Generic object manager This class provides core behavior for collections of heterogeneous objects. """ __ac_permissions__=( ('View management screens', ('manage_main',)), ('Access contents information', ('objectIds', 'objectValues', 'objectItems',''), ('Anonymous', 'Manager'), ), ('Delete objects', ('manage_delObjects',)), ('FTP access', ('manage_FTPstat','manage_FTPlist')), ('Import/Export objects', ('manage_importObject','manage_importExportForm', 'manage_exportObject') ), ) meta_type = 'Object Manager' meta_types=() # Sub-object types that are specific to this object _objects = () manage_main=DTMLFile('dtml/main', globals()) manage_index_main=DTMLFile('dtml/index_main', globals()) manage_options=( {'label':'Contents', 'action':'manage_main', 'help':('OFSP','ObjectManager_Contents.stx')}, ) isAnObjectManager=1 isPrincipiaFolderish=1 def __class_init__(self): try: mt=list(self.meta_types) except: mt=[] for b in self.__bases__: try: for t in b.meta_types: if t not in mt: mt.append(t) except: pass mt.sort() self.meta_types=tuple(mt) default__class_init__(self) def all_meta_types(self, interfaces=None): # A list of products registered elsewhere external_candidates = [] # Look at _product_meta_types, if there is one _pmt=() if hasattr(self, '_product_meta_types'): _pmt=self._product_meta_types elif hasattr(self, 'aq_acquire'): try: _pmt=self.aq_acquire('_product_meta_types') except: pass external_candidates.extend(list(_pmt)) # Look at all globally visible meta types. for entry in Products.meta_types: if ( (interfaces is not None) or (entry.get("visibility", None)=="Global") ): external_candidates.append(entry) # Filter the list of external candidates based on the # specified interface constraint if interfaces is None: interface_constrained_meta_types = external_candidates else: interface_constrained_meta_types = [] for entry in external_candidates: try: eil = entry.get('interfaces',None) if eil is not None: for ei in eil: for i in interfaces: if ei is i or ei.extends(i): interface_constrained_meta_types.append(entry) raise BreakoutException # only append 1ce except BreakoutException: pass # Meta types specified by this instance are not checked against the # interface constraint. This is as it always has been, but Im not # sure it is correct. interface_constrained_meta_types.extend(list(self.meta_types)) # Filter the list based on each meta-types's container_filter meta_types = [] for entry in interface_constrained_meta_types: container_filter = entry.get('container_filter',None) if container_filter is None: meta_types.append(entry) else: if container_filter(self): meta_types.append(entry) return meta_types def _subobject_permissions(self): return (Products.__ac_permissions__+ self.aq_acquire('_getProductRegistryData')('ac_permissions') ) def filtered_meta_types(self, user=None): # Return a list of the types for which the user has # adequate permission to add that type of object. user=getSecurityManager().getUser() meta_types=[] if callable(self.all_meta_types): all=self.all_meta_types() else: all=self.all_meta_types for meta_type in all: if meta_type.has_key('permission'): if user.has_permission(meta_type['permission'],self): meta_types.append(meta_type) else: meta_types.append(meta_type) return meta_types _checkId = checkValidId def _setOb(self, id, object): setattr(self, id, object) def _delOb(self, id): delattr(self, id) def _getOb(self, id, default=_marker): # FIXME: what we really need to do here is ensure that only # sub-items are returned. That could have a measurable hit # on performance as things are currently implemented, so for # the moment we just make sure not to expose private attrs. if id[:1] != '_' and hasattr(aq_base(self), id): return getattr(self, id) if default is _marker: raise AttributeError, id return default def hasObject(self, id): """Indicate whether the folder has an item by ID. This doesn't try to be more intelligent than _getOb, and doesn't consult _objects (for performance reasons). The common use case is to check that an object does *not* exist. """ if (id in ('.', '..') or id.startswith('_') or id.startswith('aq_') or id.endswith('__')): return False return getattr(aq_base(self), id, None) is not None def _setObject(self, id, object, roles=None, user=None, set_owner=1): v=self._checkId(id) if v is not None: id=v try: t=object.meta_type except: t=None # If an object by the given id already exists, remove it. for object_info in self._objects: if object_info['id'] == id: self._delObject(id) break self._objects=self._objects+({'id':id,'meta_type':t},) self._setOb(id,object) object=self._getOb(id) if set_owner: object.manage_fixupOwnershipAfterAdd() # Try to give user the local role "Owner", but only if # no local roles have been set on the object yet. if hasattr(object, '__ac_local_roles__'): if object.__ac_local_roles__ is None: user=getSecurityManager().getUser() if user is not None: userid=user.getId() if userid is not None: object.manage_setLocalRoles(userid, ['Owner']) object.manage_afterAdd(object, self) return id def manage_afterAdd(self, item, container): for object in self.objectValues(): try: s=object._p_changed except: s=0 if hasattr(aq_base(object), 'manage_afterAdd'): object.manage_afterAdd(item, container) if s is None: object._p_deactivate() def manage_afterClone(self, item): for object in self.objectValues(): try: s=object._p_changed except: s=0 if hasattr(aq_base(object), 'manage_afterClone'): object.manage_afterClone(item) if s is None: object._p_deactivate() def manage_beforeDelete(self, item, container): for object in self.objectValues(): try: s=object._p_changed except: s=0 try: if hasattr(aq_base(object), 'manage_beforeDelete'): object.manage_beforeDelete(item, container) except BeforeDeleteException, ob: raise except ConflictError: raise except: LOG('Zope',ERROR,'manage_beforeDelete() threw', error=sys.exc_info()) # In debug mode when non-Manager, let exceptions propagate. if getConfiguration().debug_mode: if not getSecurityManager().getUser().has_role('Manager'): raise if s is None: object._p_deactivate() def _delObject(self, id, dp=1): object=self._getOb(id) try: object.manage_beforeDelete(object, self) except BeforeDeleteException, ob: raise except ConflictError: raise except: LOG('Zope', ERROR, '_delObject() threw', error=sys.exc_info()) # In debug mode when non-Manager, let exceptions propagate. if getConfiguration().debug_mode: if not getSecurityManager().getUser().has_role('Manager'): raise self._objects=tuple(filter(lambda i,n=id: i['id']!=n, self._objects)) self._delOb(id) # Indicate to the object that it has been deleted. This is # necessary for object DB mount points. Note that we have to # tolerate failure here because the object being deleted could # be a Broken object, and it is not possible to set attributes # on Broken objects. try: object._v__object_deleted__ = 1 except: pass def objectIds(self, spec=None): # Returns a list of subobject ids of the current object. # If 'spec' is specified, returns objects whose meta_type # matches 'spec'. if spec is not None: if type(spec)==type('s'): spec=[spec] set=[] for ob in self._objects: if ob['meta_type'] in spec: set.append(ob['id']) return set return [ o['id'] for o in self._objects ] def objectValues(self, spec=None): # Returns a list of actual subobjects of the current object. # If 'spec' is specified, returns only objects whose meta_type # match 'spec'. return [ self._getOb(id) for id in self.objectIds(spec) ] def objectItems(self, spec=None): # Returns a list of (id, subobject) tuples of the current object. # If 'spec' is specified, returns only objects whose meta_type match # 'spec' return [ (id, self._getOb(id)) for id in self.objectIds(spec) ] def objectMap(self): # Return a tuple of mappings containing subobject meta-data return tuple(map(lambda dict: dict.copy(), self._objects)) def objectIds_d(self, t=None): if hasattr(self, '_reserved_names'): n=self._reserved_names else: n=() if not n: return self.objectIds(t) r=[] a=r.append for id in self.objectIds(t): if id not in n: a(id) return r def objectValues_d(self, t=None): return map(self._getOb, self.objectIds_d(t)) def objectItems_d(self, t=None): r=[] a=r.append g=self._getOb for id in self.objectIds_d(t): a((id, g(id))) return r def objectMap_d(self, t=None): if hasattr(self, '_reserved_names'): n=self._reserved_names else: n=() if not n: return self._objects r=[] a=r.append for d in self._objects: if d['id'] not in n: a(d.copy()) return r def superValues(self, t): # Return all of the objects of a given type located in # this object and containing objects. if type(t)==type('s'): t=(t,) obj=self seen={} vals=[] relativePhysicalPath = () have=seen.has_key x=0 while x < 100: if not hasattr(obj,'_getOb'): break get=obj._getOb if hasattr(obj,'_objects'): for i in obj._objects: try: id=i['id'] physicalPath = relativePhysicalPath + (id,) if (not have(physicalPath)) and (i['meta_type'] in t): vals.append(get(id)) seen[physicalPath]=1 except: pass if hasattr(obj,'aq_parent'): obj=obj.aq_parent relativePhysicalPath = ('..',) + relativePhysicalPath else: return vals x=x+1 return vals manage_addProduct=App.FactoryDispatcher.ProductDispatcher() def manage_delObjects(self, ids=[], REQUEST=None): """Delete a subordinate object The objects specified in 'ids' get deleted. """ if type(ids) is type(''): ids=[ids] if not ids: return MessageDialog(title='No items specified', message='No items were specified!', action ='./manage_main',) try: p=self._reserved_names except: p=() for n in ids: if n in p: return MessageDialog(title='Not Deletable', message='%s cannot be deleted.' % escape(n), action ='./manage_main',) while ids: id=ids[-1] v=self._getOb(id, self) if v.wl_isLocked(): raise ResourceLockedError, ( 'Object "%s" is locked via WebDAV' % v.getId()) if v is self: raise BadRequest, '%s does not exist' % escape(ids[-1]) self._delObject(id) del ids[-1] if REQUEST is not None: return self.manage_main(self, REQUEST, update_menu=1) def tpValues(self): # Return a list of subobjects, used by tree tag. r=[] if hasattr(aq_base(self), 'tree_ids'): tree_ids=self.tree_ids try: tree_ids=list(tree_ids) except TypeError: pass if hasattr(tree_ids, 'sort'): tree_ids.sort() for id in tree_ids: if hasattr(self, id): r.append(self._getOb(id)) else: obj_ids=self.objectIds() obj_ids.sort() for id in obj_ids: o=self._getOb(id) if hasattr(aq_base(o), 'isPrincipiaFolderish') and \ o.isPrincipiaFolderish: r.append(o) return r def manage_exportObject(self, id='', download=None, toxml=None, RESPONSE=None,REQUEST=None): """Exports an object to a file and returns that file.""" if not id: # can't use getId() here (breaks on "old" exported objects) id=self.id if hasattr(id, 'im_func'): id=id() ob=self else: ob=self._getOb(id) suffix=toxml and 'xml' or 'zexp' if download: f=StringIO() if toxml: XMLExportImport.exportXML(ob._p_jar, ob._p_oid, f) else: ob._p_jar.exportFile(ob._p_oid, f) if RESPONSE is not None: RESPONSE.setHeader('Content-type','application/data') RESPONSE.setHeader('Content-Disposition', 'inline;filename=%s.%s' % (id, suffix)) return f.getvalue() cfg = getConfiguration() f = os.path.join(cfg.clienthome, '%s.%s' % (id, suffix)) if toxml: XMLExportImport.exportXML(ob._p_jar, ob._p_oid, f) else: ob._p_jar.exportFile(ob._p_oid, f) if REQUEST is not None: return self.manage_main(self, REQUEST, manage_tabs_message= '%s successfully exported to %s' % (id,f), title = 'Object exported') manage_importExportForm=DTMLFile('dtml/importExport',globals()) def manage_importObject(self, file, REQUEST=None, set_owner=1): """Import an object from a file""" dirname, file=os.path.split(file) if dirname: raise BadRequest, 'Invalid file name %s' % escape(file) cfg = getConfiguration() for impath in (cfg.instancehome, cfg.zopehome): filepath = os.path.join(impath, 'import', file) if os.path.exists(filepath): break else: raise BadRequest, 'File does not exist: %s' % escape(file) self._importObjectFromFile(filepath, verify=not not REQUEST, set_owner=set_owner) if REQUEST is not None: return self.manage_main( self, REQUEST, manage_tabs_message='%s successfully imported' % id, title='Object imported', update_menu=1) def _importObjectFromFile(self, filepath, verify=1, set_owner=1): # locate a valid connection connection=self._p_jar obj=self while connection is None: obj=obj.aq_parent connection=obj._p_jar ob=connection.importFile( filepath, customImporters=customImporters) if verify: self._verifyObjectPaste(ob, validate_src=0) id=ob.id if hasattr(id, 'im_func'): id=id() self._setObject(id, ob, set_owner=set_owner) # try to make ownership implicit if possible in the context # that the object was imported into. ob=self._getOb(id) ob.manage_changeOwnershipType(explicit=0) def list_imports(self): listing = [] cfg = getConfiguration() paths = [cfg.zopehome] if not cfg.instancehome in paths: paths.append(cfg.instancehome) for impath in paths: directory = os.path.join(impath, 'import') listing += [f for f in os.listdir(directory) if f.endswith('.zexp') or f.endswith('.xml')] return listing # FTP support methods def manage_FTPlist(self, REQUEST): """Directory listing for FTP. """ out=() # check to see if we are being acquiring or not ob=self while 1: if App.Common.is_acquired(ob): raise ValueError('FTP List not supported on acquired objects') if not hasattr(ob,'aq_parent'): break ob=ob.aq_parent files = list(self.objectItems()) # recursive ride through all subfolders (ls -R) (ajung) if REQUEST.environ.get('FTP_RECURSIVE',0) == 1: all_files = copy.copy(files) for f in files: if hasattr(aq_base(f[1]), 'isPrincipiaFolderish') and f[1].isPrincipiaFolderish: all_files.extend(findChildren(f[1])) files = all_files # Perform globbing on list of files (ajung) globbing = REQUEST.environ.get('GLOBBING','') if globbing : files = filter(lambda x,g=globbing: fnmatch.fnmatch(x[0],g), files) files.sort() if not (hasattr(self,'isTopLevelPrincipiaApplicationObject') and self.isTopLevelPrincipiaApplicationObject): files.insert(0,('..',self.aq_parent)) files.insert(0, ('.', self)) for k,v in files: # Note that we have to tolerate failure here, because # Broken objects won't stat correctly. If an object fails # to be able to stat itself, we will ignore it, but log # the error. try: stat=marshal.loads(v.manage_FTPstat(REQUEST)) except: LOG("FTP", ERROR, "Failed to stat file '%s'" % k, error=sys.exc_info()) stat=None if stat is not None: out=out+((k,stat),) return marshal.dumps(out) def manage_hasId(self, REQUEST): """ check if the folder has an object with REQUEST['id'] """ if not REQUEST['id'] in self.objectIds(): raise KeyError(REQUEST['id']) def manage_FTPstat(self,REQUEST): """Psuedo stat, used by FTP for directory listings. """ mode=0040000 from AccessControl.User import nobody # check to see if we are acquiring our objectValues or not if not (len(REQUEST.PARENTS) > 1 and self.objectValues() == REQUEST.PARENTS[1].objectValues()): try: if getSecurityManager().validate( None, self, 'manage_FTPlist', self.manage_FTPlist ): mode=mode | 0770 except: pass if nobody.allowed( self, getRoles(self, 'manage_FTPlist', self.manage_FTPlist, ())): mode=mode | 0007 mtime=self.bobobase_modification_time().timeTime() # get owner and group owner=group='Zope' for user, roles in self.get_local_roles(): if 'Owner' in roles: owner=user break return marshal.dumps((mode,0,0,1,owner,group,0,mtime,mtime,mtime)) def __getitem__(self, key): v=self._getOb(key, None) if v is not None: return v if hasattr(self, 'REQUEST'): request=self.REQUEST method=request.get('REQUEST_METHOD', 'GET') if request.maybe_webdav_client and not method in ('GET', 'POST'): return NullResource(self, key, request).__of__(self) raise KeyError, key def findChildren(obj,dirname=''): """ recursive walk through the object hierarchy to find all children of an object (ajung) """ lst = [] for name, child in obj.objectItems(): if hasattr(aq_base(child), 'isPrincipiaFolderish') and child.isPrincipiaFolderish: lst.extend(findChildren(child, dirname + obj.id + '/')) else: lst.append((dirname + obj.id + "/" + name, child)) return lst class IFAwareObjectManager: def all_meta_types(self, interfaces=None): if interfaces is None: if hasattr(self, '_product_interfaces'): interfaces=self._product_interfaces elif hasattr(self, 'aq_acquire'): try: interfaces=self.aq_acquire('_product_interfaces') except: pass # Bleah generic pass is bad return ObjectManager.all_meta_types(self, interfaces) Globals.default__class_init__(ObjectManager) Zope-2.8.7-final/lib/python/OFS/misc_.py0000664000076400007640000000552510435500327016730 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from App.ImageFile import ImageFile class misc_: "Miscellaneous product information" __roles__=None class p_: "Shared system information" __roles__=None broken=ImageFile('www/broken.gif', globals()) User_icon =ImageFile('AccessControl/www/User_icon.gif') locked=ImageFile('www/modified.gif', globals()) lockedo=ImageFile('www/locked.gif', globals()) davlocked=ImageFile('webdav/www/davlock.gif') pl=ImageFile('TreeDisplay/www/Plus_icon.gif') mi=ImageFile('TreeDisplay/www/Minus_icon.gif') rtab=ImageFile('App/www/rtab.gif') ltab=ImageFile('App/www/ltab.gif') sp =ImageFile('App/www/sp.gif') r_arrow_gif=ImageFile('www/r_arrow.gif', globals()) l_arrow_gif=ImageFile('www/l_arrow.gif', globals()) ControlPanel_icon=ImageFile('OFS/www/ControlPanel_icon.gif') ApplicationManagement_icon=ImageFile('App/www/cpSystem.gif') DatabaseManagement_icon=ImageFile('App/www/dbManage.gif') VersionManagement_icon=ImageFile('App/www/vManage.gif') DebugManager_icon=ImageFile('App/www/DebugManager_icon.gif') InstalledProduct_icon=ImageFile('App/www/installedProduct.gif') BrokenProduct_icon=ImageFile('App/www/brokenProduct.gif') Product_icon=ImageFile('App/www/product.gif') Factory_icon=ImageFile('App/www/factory.gif') Permission_icon=ImageFile('App/www/permission.gif') ProductFolder_icon=ImageFile('App/www/productFolder.gif') PyPoweredSmall_Gif=ImageFile('App/www/PythonPoweredSmall.gif') ZopeButton=ImageFile('App/www/zope_button.jpg') ZButton=ImageFile('App/www/z_button.jpg') zopelogo_jpg=ImageFile('App/www/zopelogo.jpg') Properties_icon=ImageFile('OFS/www/Properties_icon.gif') Methods_icon=ImageFile('ZClasses/methods.gif') Propertysheets_icon=ImageFile('ZClasses/propertysheets.gif') ProductHelp_icon=ImageFile('HelpSys/images/productHelp.gif') HelpTopic_icon=ImageFile('HelpSys/images/helpTopic.gif') class Misc_: "Miscellaneous product information" __roles__=None def __init__(self, name, dict): self._d=dict self.__name__=name def __str__(self): return self.__name__ def __getitem__(self, name): return self._d[name] def __setitem__(self, name, v): self._d[name]=v Zope-2.8.7-final/lib/python/OFS/ndiff.py0000664000076400007640000005750110435500327016725 0ustar ajungajung#! /usr/bin/env python # Module ndiff version 1.4.0 # Released to the public domain 27-Mar-1999, # by Tim Peters (tim_one@email.msn.com). # Provided as-is; use at your own risk; no warranty; no promises; enjoy! """ndiff [-q] file1 file2 or ndiff (-r1 | -r2) < ndiff_output > file1_or_file2 Print a human-friendly file difference report to stdout. Both inter- and intra-line differences are noted. In the second form, recreate file1 (-r1) or file2 (-r2) on stdout, from an ndiff report on stdin. In the first form, if -q ("quiet") is not specified, the first two lines of output are -: file1 +: file2 Each remaining line begins with a two-letter code: "- " line unique to file1 "+ " line unique to file2 " " line common to both files "? " line not present in either input file Lines beginning with "? " attempt to guide the eye to intraline differences, and were not present in either input file. These lines can be confusing if the source files contain tab characters. The first file can be recovered by retaining only lines that begin with " " or "- ", and deleting those 2-character prefixes; use ndiff with -r1. The second file can be recovered similarly, but by retaining only " " and "+ " lines; use ndiff with -r2; or, on Unix, the second file can be recovered by piping the output through sed -n '/^[+ ] /s/^..//p' See module comments for details and programmatic interface. """ __version__ = 1, 4, 0 # SequenceMatcher tries to compute a "human-friendly diff" between # two sequences (chiefly picturing a file as a sequence of lines, # and a line as a sequence of characters, here). Unlike e.g. UNIX(tm) # diff, the fundamental notion is the longest *contiguous* & junk-free # matching subsequence. That's what catches peoples' eyes. The # Windows(tm) windiff has another interesting notion, pairing up elements # that appear uniquely in each sequence. That, and the method here, # appear to yield more intuitive difference reports than does diff. This # method appears to be the least vulnerable to synching up on blocks # of "junk lines", though (like blank lines in ordinary text files, # or maybe "

" lines in HTML files). That may be because this is # the only method of the 3 that has a *concept* of "junk" . # # Note that ndiff makes no claim to produce a *minimal* diff. To the # contrary, minimal diffs are often counter-intuitive, because they # synch up anywhere possible, sometimes accidental matches 100 pages # apart. Restricting synch points to contiguous matches preserves some # notion of locality, at the occasional cost of producing a longer diff. # # With respect to junk, an earlier version of ndiff simply refused to # *start* a match with a junk element. The result was cases like this: # before: private Thread currentThread; # after: private volatile Thread currentThread; # If you consider whitespace to be junk, the longest contiguous match # not starting with junk is "e Thread currentThread". So ndiff reported # that "e volatil" was inserted between the 't' and the 'e' in "private". # While an accurate view, to people that's absurd. The current version # looks for matching blocks that are entirely junk-free, then extends the # longest one of those as far as possible but only with matching junk. # So now "currentThread" is matched, then extended to suck up the # preceding blank; then "private" is matched, and extended to suck up the # following blank; then "Thread" is matched; and finally ndiff reports # that "volatile " was inserted before "Thread". The only quibble # remaining is that perhaps it was really the case that " volatile" # was inserted after "private". I can live with that . # # NOTE on junk: the module-level names # IS_LINE_JUNK # IS_CHARACTER_JUNK # can be set to any functions you like. The first one should accept # a single string argument, and return true iff the string is junk. # The default is whether the regexp r"\s*#?\s*$" matches (i.e., a # line without visible characters, except for at most one splat). # The second should accept a string of length 1 etc. The default is # whether the character is a blank or tab (note: bad idea to include # newline in this!). # # After setting those, you can call fcompare(f1name, f2name) with the # names of the files you want to compare. The difference report # is sent to stdout. Or you can call main(args), passing what would # have been in sys.argv[1:] had the cmd-line form been used. TRACE = 0 # define what "junk" means import re def IS_LINE_JUNK(line, pat=re.compile(r"\s*#?\s*$").match): return pat(line) is not None def IS_CHARACTER_JUNK(ch, ws=" \t"): return ch in ws del re class SequenceMatcher: def __init__(self, isjunk=None, a='', b=''): # Members: # a # first sequence # b # second sequence; differences are computed as "what do # we need to do to 'a' to change it into 'b'?" # b2j # for x in b, b2j[x] is a list of the indices (into b) # at which x appears; junk elements do not appear # b2jhas # b2j.has_key # fullbcount # for x in b, fullbcount[x] == the number of times x # appears in b; only materialized if really needed (used # only for computing quick_ratio()) # matching_blocks # a list of (i, j, k) triples, where a[i:i+k] == b[j:j+k]; # ascending & non-overlapping in i and in j; terminated by # a dummy (len(a), len(b), 0) sentinel # opcodes # a list of (tag, i1, i2, j1, j2) tuples, where tag is # one of # 'replace' a[i1:i2] should be replaced by b[j1:j2] # 'delete' a[i1:i2] should be deleted # 'insert' b[j1:j2] should be inserted # 'equal' a[i1:i2] == b[j1:j2] # isjunk # a user-supplied function taking a sequence element and # returning true iff the element is "junk" -- this has # subtle but helpful effects on the algorithm, which I'll # get around to writing up someday <0.9 wink>. # DON'T USE! Only __chain_b uses this. Use isbjunk. # isbjunk # for x in b, isbjunk(x) == isjunk(x) but much faster; # it's really the has_key method of a hidden dict. # DOES NOT WORK for x in a! self.isjunk = isjunk self.a = self.b = None self.set_seqs(a, b) def set_seqs(self, a, b): self.set_seq1(a) self.set_seq2(b) def set_seq1(self, a): if a is self.a: return self.a = a self.matching_blocks = self.opcodes = None def set_seq2(self, b): if b is self.b: return self.b = b self.matching_blocks = self.opcodes = None self.fullbcount = None self.__chain_b() # For each element x in b, set b2j[x] to a list of the indices in # b where x appears; the indices are in increasing order; note that # the number of times x appears in b is len(b2j[x]) ... # when self.isjunk is defined, junk elements don't show up in this # map at all, which stops the central find_longest_match method # from starting any matching block at a junk element ... # also creates the fast isbjunk function ... # note that this is only called when b changes; so for cross-product # kinds of matches, it's best to call set_seq2 once, then set_seq1 # repeatedly def __chain_b(self): # Because isjunk is a user-defined (not C) function, and we test # for junk a LOT, it's important to minimize the number of calls. # Before the tricks described here, __chain_b was by far the most # time-consuming routine in the whole module! If anyone sees # Jim Roskind, thank him again for profile.py -- I never would # have guessed that. # The first trick is to build b2j ignoring the possibility # of junk. I.e., we don't call isjunk at all yet. Throwing # out the junk later is much cheaper than building b2j "right" # from the start. b = self.b self.b2j = b2j = {} self.b2jhas = b2jhas = b2j.has_key for i in xrange(len(b)): elt = b[i] if b2jhas(elt): b2j[elt].append(i) else: b2j[elt] = [i] # Now b2j.keys() contains elements uniquely, and especially when # the sequence is a string, that's usually a good deal smaller # than len(string). The difference is the number of isjunk calls # saved. isjunk, junkdict = self.isjunk, {} if isjunk: for elt in b2j.keys(): if isjunk(elt): junkdict[elt] = 1 # value irrelevant; it's a set del b2j[elt] # Now for x in b, isjunk(x) == junkdict.has_key(x), but the # latter is much faster. Note too that while there may be a # lot of junk in the sequence, the number of *unique* junk # elements is probably small. So the memory burden of keeping # this dict alive is likely trivial compared to the size of b2j. self.isbjunk = junkdict.has_key def find_longest_match(self, alo, ahi, blo, bhi): """Find longest matching block in a[alo:ahi] and b[blo:bhi]. If isjunk is not defined: Return (i,j,k) such that a[i:i+k] is equal to b[j:j+k], where alo <= i <= i+k <= ahi blo <= j <= j+k <= bhi and for all (i',j',k') meeting those conditions, k >= k' i <= i' and if i == i', j <= j' In other words, of all maximal matching blocks, return one that starts earliest in a, and of all those maximal matching blocks that start earliest in a, return the one that starts earliest in b. If isjunk is defined, first the longest matching block is determined as above, but with the additional restriction that no junk element appears in the block. Then that block is extended as far as possible by matching (only) junk elements on both sides. So the resulting block never matches on junk except as identical junk happens to be adjacent to an "interesting" match. If no blocks match, return (alo, blo, 0). """ # CAUTION: stripping common prefix or suffix would be incorrect. # E.g., # ab # acab # Longest matching block is "ab", but if common prefix is # stripped, it's "a" (tied with "b"). UNIX(tm) diff does so # strip, so ends up claiming that ab is changed to acab by # inserting "ca" in the middle. That's minimal but unintuitive: # "it's obvious" that someone inserted "ac" at the front. # Windiff ends up at the same place as diff, but by pairing up # the unique 'b's and then matching the first two 'a's. a, b, b2j, isbjunk = self.a, self.b, self.b2j, self.isbjunk besti, bestj, bestsize = alo, blo, 0 # find longest junk-free match # during an iteration of the loop, j2len[j] = length of longest # junk-free match ending with a[i-1] and b[j] j2len = {} nothing = [] for i in xrange(alo, ahi): # look at all instances of a[i] in b; note that because # b2j has no junk keys, the loop is skipped if a[i] is junk j2lenget = j2len.get newj2len = {} for j in b2j.get(a[i], nothing): # a[i] matches b[j] if j < blo: continue if j >= bhi: break k = newj2len[j] = j2lenget(j-1, 0) + 1 if k > bestsize: besti, bestj, bestsize = i-k+1, j-k+1, k j2len = newj2len # Now that we have a wholly interesting match (albeit possibly # empty!), we may as well suck up the matching junk on each # side of it too. Can't think of a good reason not to, and it # saves post-processing the (possibly considerable) expense of # figuring out what to do with it. In the case of an empty # interesting match, this is clearly the right thing to do, # because no other kind of match is possible in the regions. while besti > alo and bestj > blo and \ isbjunk(b[bestj-1]) and \ a[besti-1] == b[bestj-1]: besti, bestj, bestsize = besti-1, bestj-1, bestsize+1 while besti+bestsize < ahi and bestj+bestsize < bhi and \ isbjunk(b[bestj+bestsize]) and \ a[besti+bestsize] == b[bestj+bestsize]: bestsize = bestsize + 1 if TRACE: print "get_matching_blocks", alo, ahi, blo, bhi print " returns", besti, bestj, bestsize return besti, bestj, bestsize def get_matching_blocks(self): if self.matching_blocks is not None: return self.matching_blocks self.matching_blocks = [] la, lb = len(self.a), len(self.b) self.__helper(0, la, 0, lb, self.matching_blocks) self.matching_blocks.append( (la, lb, 0) ) if TRACE: print '*** matching blocks', self.matching_blocks return self.matching_blocks # builds list of matching blocks covering a[alo:ahi] and # b[blo:bhi], appending them in increasing order to answer def __helper(self, alo, ahi, blo, bhi, answer): i, j, k = x = self.find_longest_match(alo, ahi, blo, bhi) # a[alo:i] vs b[blo:j] unknown # a[i:i+k] same as b[j:j+k] # a[i+k:ahi] vs b[j+k:bhi] unknown if k: if alo < i and blo < j: self.__helper(alo, i, blo, j, answer) answer.append(x) if i+k < ahi and j+k < bhi: self.__helper(i+k, ahi, j+k, bhi, answer) def ratio(self): """Return a measure of the sequences' similarity (float in [0,1]). Where T is the total number of elements in both sequences, and M is the number of matches, this is 2*M / T. Note that this is 1 if the sequences are identical, and 0 if they have nothing in common. """ matches = reduce(lambda sum, triple: sum + triple[-1], self.get_matching_blocks(), 0) return 2.0 * matches / (len(self.a) + len(self.b)) def quick_ratio(self): """Return an upper bound on ratio() relatively quickly.""" # viewing a and b as multisets, set matches to the cardinality # of their intersection; this counts the number of matches # without regard to order, so is clearly an upper bound if self.fullbcount is None: self.fullbcount = fullbcount = {} for elt in self.b: fullbcount[elt] = fullbcount.get(elt, 0) + 1 fullbcount = self.fullbcount # avail[x] is the number of times x appears in 'b' less the # number of times we've seen it in 'a' so far ... kinda avail = {} availhas, matches = avail.has_key, 0 for elt in self.a: if availhas(elt): numb = avail[elt] else: numb = fullbcount.get(elt, 0) avail[elt] = numb - 1 if numb > 0: matches = matches + 1 return 2.0 * matches / (len(self.a) + len(self.b)) def real_quick_ratio(self): """Return an upper bound on ratio() very quickly""" la, lb = len(self.a), len(self.b) # can't have more matches than the number of elements in the # shorter sequence return 2.0 * min(la, lb) / (la + lb) def get_opcodes(self): if self.opcodes is not None: return self.opcodes i = j = 0 self.opcodes = answer = [] for ai, bj, size in self.get_matching_blocks(): # invariant: we've pumped out correct diffs to change # a[:i] into b[:j], and the next matching block is # a[ai:ai+size] == b[bj:bj+size]. So we need to pump # out a diff to change a[i:ai] into b[j:bj], pump out # the matching block, and move (i,j) beyond the match tag = '' if i < ai and j < bj: tag = 'replace' elif i < ai: tag = 'delete' elif j < bj: tag = 'insert' if tag: answer.append( (tag, i, ai, j, bj) ) i, j = ai+size, bj+size # the list of matching blocks is terminated by a # sentinel with size 0 if size: answer.append( ('equal', ai, i, bj, j) ) return answer # meant for dumping lines def dump(tag, x, lo, hi): for i in xrange(lo, hi): print tag, x[i], # figure out which mark to stick under characters in lines that # have changed (blank = same, - = deleted, + = inserted, ^ = replaced) _combine = { ' ': ' ', '. ': '-', ' .': '+', '..': '^' } def plain_replace(a, alo, ahi, b, blo, bhi): assert alo < ahi and blo < bhi # dump the shorter block first -- reduces the burden on short-term # memory if the blocks are of very different sizes if bhi - blo < ahi - alo: dump('+', b, blo, bhi) dump('-', a, alo, ahi) else: dump('-', a, alo, ahi) dump('+', b, blo, bhi) # When replacing one block of lines with another, this guy searches # the blocks for *similar* lines; the best-matching pair (if any) is # used as a synch point, and intraline difference marking is done on # the similar pair. Lots of work, but often worth it. def fancy_replace(a, alo, ahi, b, blo, bhi): if TRACE: print '*** fancy_replace', alo, ahi, blo, bhi dump('>', a, alo, ahi) dump('<', b, blo, bhi) # don't synch up unless the lines have a similarity score of at # least cutoff; best_ratio tracks the best score seen so far best_ratio, cutoff = 0.74, 0.75 cruncher = SequenceMatcher(IS_CHARACTER_JUNK) eqi, eqj = None, None # 1st indices of equal lines (if any) # search for the pair that matches best without being identical # (identical lines must be junk lines, & we don't want to synch up # on junk -- unless we have to) for j in xrange(blo, bhi): bj = b[j] cruncher.set_seq2(bj) for i in xrange(alo, ahi): ai = a[i] if ai == bj: if eqi is None: eqi, eqj = i, j continue cruncher.set_seq1(ai) # computing similarity is expensive, so use the quick # upper bounds first -- have seen this speed up messy # compares by a factor of 3. # note that ratio() is only expensive to compute the first # time it's called on a sequence pair; the expensive part # of the computation is cached by cruncher if cruncher.real_quick_ratio() > best_ratio and \ cruncher.quick_ratio() > best_ratio and \ cruncher.ratio() > best_ratio: best_ratio, best_i, best_j = cruncher.ratio(), i, j if best_ratio < cutoff: # no non-identical "pretty close" pair if eqi is None: # no identical pair either -- treat it as a straight replace plain_replace(a, alo, ahi, b, blo, bhi) return # no close pair, but an identical pair -- synch up on that best_i, best_j, best_ratio = eqi, eqj, 1.0 else: # there's a close pair, so forget the identical pair (if any) eqi = None # a[best_i] very similar to b[best_j]; eqi is None iff they're not # identical if TRACE: print '*** best_ratio', best_ratio, best_i, best_j dump('>', a, best_i, best_i+1) dump('<', b, best_j, best_j+1) # pump out diffs from before the synch point fancy_helper(a, alo, best_i, b, blo, best_j) # do intraline marking on the synch pair aelt, belt = a[best_i], b[best_j] if eqi is None: # pump out a '-', '+', '?' triple for the synched lines; atags = btags = "" cruncher.set_seqs(aelt, belt) for tag, ai1, ai2, bj1, bj2 in cruncher.get_opcodes(): la, lb = ai2 - ai1, bj2 - bj1 if tag == 'replace': atags = atags + '.' * la btags = btags + '.' * lb elif tag == 'delete': atags = atags + '.' * la elif tag == 'insert': btags = btags + '.' * lb elif tag == 'equal': atags = atags + ' ' * la btags = btags + ' ' * lb else: raise ValueError, 'unknown tag ' + `tag` la, lb = len(atags), len(btags) if la < lb: atags = atags + ' ' * (lb - la) elif lb < la: btags = btags + ' ' * (la - lb) combined = map(lambda x,y: _combine[x+y], atags, btags) print '-', aelt, '+', belt, '?', \ ''.join(combined).rstrip() else: # the synch pair is identical print ' ', aelt, # pump out diffs from after the synch point fancy_helper(a, best_i+1, ahi, b, best_j+1, bhi) def fancy_helper(a, alo, ahi, b, blo, bhi): if alo < ahi: if blo < bhi: fancy_replace(a, alo, ahi, b, blo, bhi) else: dump('-', a, alo, ahi) elif blo < bhi: dump('+', b, blo, bhi) def fail(msg): import sys out = sys.stderr.write out(msg + "\n\n") out(__doc__) return 0 # open a file & return the file object; gripe and return 0 if it # couldn't be opened def fopen(fname): try: return open(fname, 'r') except IOError, detail: return fail("couldn't open " + fname + ": " + str(detail)) # open two files & spray the diff to stdout; return false iff a problem def fcompare(f1name, f2name): f1 = fopen(f1name) f2 = fopen(f2name) if not f1 or not f2: return 0 a = f1.readlines(); f1.close() b = f2.readlines(); f2.close() cruncher = SequenceMatcher(IS_LINE_JUNK, a, b) for tag, alo, ahi, blo, bhi in cruncher.get_opcodes(): if tag == 'replace': fancy_replace(a, alo, ahi, b, blo, bhi) elif tag == 'delete': dump('-', a, alo, ahi) elif tag == 'insert': dump('+', b, blo, bhi) elif tag == 'equal': dump(' ', a, alo, ahi) else: raise ValueError, 'unknown tag ' + `tag` return 1 # crack args (sys.argv[1:] is normal) & compare; # return false iff a problem def main(args): import getopt try: opts, args = getopt.getopt(args, "qr:") except getopt.error, detail: return fail(str(detail)) noisy = 1 qseen = rseen = 0 for opt, val in opts: if opt == "-q": qseen = 1 noisy = 0 elif opt == "-r": rseen = 1 whichfile = val if qseen and rseen: return fail("can't specify both -q and -r") if rseen: if args: return fail("no args allowed with -r option") if whichfile in "12": restore(whichfile) return 1 return fail("-r value must be 1 or 2") if len(args) != 2: return fail("need 2 filename args") f1name, f2name = args if noisy: print '-:', f1name print '+:', f2name return fcompare(f1name, f2name) def restore(which): import sys tag = {"1": "- ", "2": "+ "}[which] prefixes = (" ", tag) for line in sys.stdin.readlines(): if line[:2] in prefixes: print line[2:], if __name__ == '__main__': import sys args = sys.argv[1:] if "-profile" in args: import profile, pstats args.remove("-profile") statf = "ndiff.pro" profile.run("main(args)", statf) stats = pstats.Stats(statf) stats.strip_dirs().sort_stats('time').print_stats() else: main(args) Zope-2.8.7-final/lib/python/OFS/FTPInterface.py0000664000076400007640000000354710435500327020112 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """FTP Support for Zope classes. Preliminary FTP support interface. Note, most FTP functions are provided by existing methods such as PUT and manage_delObjects. All FTP methods should be governed by a single permission: 'FTP access'. """ class FTPInterface: "Interface for FTP objects" # XXX The stat and list marshal format should probably # be XML, not marshal, maybe Andrew K's xml-marshal. # This will probably be changed later. def manage_FTPstat(self,REQUEST): """Returns a stat-like tuple. (marshalled to a string) Used by FTP for directory listings, and MDTM and SIZE""" def manage_FTPlist(self,REQUEST): """Returns a directory listing consisting of a tuple of (id,stat) tuples, marshaled to a string. Note, the listing it should include '..' if there is a Folder above the current one. In the case of non-foldoid objects it should return a single tuple (id,stat) representing itself.""" # Optional method to support FTP download. # Should not be implemented by Foldoid objects. def manage_FTPget(self): """Returns the source content of an object. For example, the source text of a Document, or the data of a file.""" Zope-2.8.7-final/lib/python/OFS/IOrderSupport.py0000664000076400007640000000674510435500327020424 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Order support interfaces. $Id: IOrderSupport.py 24763 2004-05-17 05:59:28Z philikon $ """ from Interface import Interface class IOrderedContainer(Interface): """ Ordered Container interface. This interface provides a common mechanism for maintaining ordered collections. """ def moveObjectsByDelta(ids, delta, subset_ids=None): """ Move specified sub-objects by delta. If delta is higher than the possible maximum, objects will be moved to the bottom. If delta is lower than the possible minimum, objects will be moved to the top. If subset_ids is not None, delta will be interpreted relative to the subset specified by a sequence of ids. The position of objects that are not part of this subset will not be changed. The order of the objects specified by ids will always be preserved. So if you don't want to change their original order, make sure the order of ids corresponds to their original order. If an object with id doesn't exist an error will be raised. Permission -- Manage properties Returns -- Number of moved sub-objects """ def moveObjectsUp(ids, delta=1, subset_ids=None): """ Move specified sub-objects up by delta in container. If no delta is specified, delta is 1. See moveObjectsByDelta for more details. Permission -- Manage properties Returns -- Number of moved sub-objects """ def moveObjectsDown(ids, delta=1, subset_ids=None): """ Move specified sub-objects down by delta in container. If no delta is specified, delta is 1. See moveObjectsByDelta for more details. Permission -- Manage properties Returns -- Number of moved sub-objects """ def moveObjectsToTop(ids, subset_ids=None): """ Move specified sub-objects to top of container. See moveObjectsByDelta for more details. Permission -- Manage properties Returns -- Number of moved sub-objects """ def moveObjectsToBottom(ids, subset_ids=None): """ Move specified sub-objects to bottom of container. See moveObjectsByDelta for more details. Permission -- Manage properties Returns -- Number of moved sub-objects """ def orderObjects(key, reverse=None): """ Order sub-objects by key and direction. Permission -- Manage properties Returns -- Number of moved sub-objects """ def getObjectPosition(id): """ Get the position of an object by its id. Permission -- Access contents information Returns -- Position """ def moveObjectToPosition(id, position): """ Move specified object to absolute position. Permission -- Manage properties Returns -- Number of moved sub-objects """ Zope-2.8.7-final/lib/python/OFS/tests/0000775000076400007640000000000010435610645016424 5ustar ajungajungZope-2.8.7-final/lib/python/OFS/tests/testOrderSupport.py0000664000076400007640000001561210435500327022346 0ustar ajungajungimport unittest import Testing import Zope2 Zope2.startup() from OFS.CopySupport import CopySource from OFS.ObjectManager import ObjectManager class DummyObject(CopySource): def __init__(self, id, meta_type): self.id = id self.meta_type = meta_type def cb_isMoveable(self): return 1 def manage_afterAdd(self, item, container): return def manage_beforeDelete(self, item, container): return def wl_isLocked(self): return 0 class TestOrderSupport(unittest.TestCase): def _makeOne(self): from OFS.OrderSupport import OrderSupport class OrderedObjectManager(OrderSupport, ObjectManager): # disable permission verification def _verifyObjectPaste(self, object, validate_src=1): return f = OrderedObjectManager() f._objects = ( {'id':'o1', 'meta_type':'mt1'} , {'id':'o2', 'meta_type':'mt2'} , {'id':'o3', 'meta_type':'mt1'} , {'id':'o4', 'meta_type':'mt2'} ) f.o1 = DummyObject('o1', 'mt1') f.o2 = DummyObject('o2', 'mt2') f.o3 = DummyObject('o3', 'mt1') f.o4 = DummyObject('o4', 'mt2') return f def _doCanonTest(self, methodname, table): for args, order, rval in table: f = self._makeOne() method = getattr(f, methodname) if rval == 'ValueError': self.failUnlessRaises( ValueError, method, *args ) else: self.failUnlessEqual( method(*args), rval ) self.failUnlessEqual( f.objectIds(), order ) def test_moveObjectsUp(self): self._doCanonTest( 'moveObjectsUp', ( ( ( 'o4', 1 ), ['o1', 'o2', 'o4', 'o3'], 1 ) , ( ( 'o4', 2 ), ['o1', 'o4', 'o2', 'o3'], 1 ) , ( ( ('o1', 'o3'), 1 ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o1', 'o3'), 9 ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o2', 'o3'), 1 ), ['o2', 'o3', 'o1', 'o4'], 2 ) , ( ( ('o2', 'o3'), 1, ('o1', 'o2', 'o3', 'o4') ), ['o2', 'o3', 'o1', 'o4'], 2 ) , ( ( ('o2', 'o3'), 1, ('o2', 'o3', 'o4') ), ['o1', 'o2', 'o3', 'o4'], 0 ) , ( ( ('n2', 'o3'), 1 ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') , ( ( ('o3', 'o1'), 1 ), ['o1', 'o3', 'o2', 'o4'], 1 ) ) ) def test_moveObjectsDown(self): self._doCanonTest( 'moveObjectsDown', ( ( ( 'o1', 1 ), ['o2', 'o1', 'o3', 'o4'], 1 ) , ( ( 'o1', 2 ), ['o2', 'o3', 'o1', 'o4'], 1 ) , ( ( ('o2', 'o4'), 1 ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o2', 'o4'), 9 ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o2', 'o3'), 1 ), ['o1', 'o4', 'o2', 'o3'], 2 ) , ( ( ('o2', 'o3'), 1, ('o1', 'o2', 'o3', 'o4') ), ['o1', 'o4', 'o2', 'o3'], 2 ) , ( ( ('o2', 'o3'), 1, ('o1', 'o2', 'o3') ), ['o1', 'o2', 'o3', 'o4'], 0 ) , ( ( ('n2', 'o3'), 1 ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') , ( ( ('o4', 'o2'), 1 ), ['o1', 'o3', 'o2', 'o4'], 1 ) ) ) def test_moveObjectsToTop(self): self._doCanonTest( 'moveObjectsToTop', ( ( ( 'o4', ), ['o4', 'o1', 'o2', 'o3'], 1 ) , ( ( ('o1', 'o3'), ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o2', 'o3'), ), ['o2', 'o3', 'o1', 'o4'], 2 ) , ( ( ('o2', 'o3'), ('o1', 'o2', 'o3', 'o4') ), ['o2', 'o3', 'o1', 'o4'], 2 ) , ( ( ('o2', 'o3'), ('o2', 'o3', 'o4') ), ['o1', 'o2', 'o3', 'o4'], 0 ) , ( ( ('n2', 'o3'), ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') , ( ( ('o3', 'o1'), ), ['o3', 'o1', 'o2', 'o4'], 1 ) ) ) def test_moveObjectsToBottom(self): self._doCanonTest( 'moveObjectsToBottom', ( ( ( 'o1', ), ['o2', 'o3', 'o4', 'o1'], 1 ) , ( ( ('o2', 'o4'), ), ['o1', 'o3', 'o2', 'o4'], 1 ) , ( ( ('o2', 'o3'), ), ['o1', 'o4', 'o2', 'o3'], 2 ) , ( ( ('o2', 'o3'), ('o1', 'o2', 'o3', 'o4') ), ['o1', 'o4', 'o2', 'o3'], 2 ) , ( ( ('o2', 'o3'), ('o1', 'o2', 'o3') ), ['o1', 'o2', 'o3', 'o4'], 0 ) , ( ( ('n2', 'o3'), ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') , ( ( ('o4', 'o2'), ), ['o1', 'o3', 'o4', 'o2'], 1 ) ) ) def test_orderObjects(self): self._doCanonTest( 'orderObjects', ( ( ( 'id', 'id' ), ['o4', 'o3', 'o2', 'o1'], 3) , ( ( 'meta_type', '' ), ['o1', 'o3', 'o2', 'o4'], 1) , ( ( 'meta_type', 'n' ), ['o4', 'o2', 'o3', 'o1'], 3) , ( ( 'position', 0 ), ['o1', 'o2', 'o3', 'o4'], 0) , ( ( 'position', 1 ), ['o4', 'o3', 'o2', 'o1'], 3) ) ) def test_getObjectPosition(self): self._doCanonTest( 'getObjectPosition', ( ( ( 'o2', ), ['o1', 'o2', 'o3', 'o4'], 1) , ( ( 'o4', ), ['o1', 'o2', 'o3', 'o4'], 3) , ( ( 'n2', ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') ) ) def test_moveObjectToPosition(self): self._doCanonTest( 'moveObjectToPosition', ( ( ( 'o2', 2 ), ['o1', 'o3', 'o2', 'o4'], 1) , ( ( 'o4', 2 ), ['o1', 'o2', 'o4', 'o3'], 1) , ( ( 'n2', 2 ), ['o1', 'o2', 'o3', 'o4'], 'ValueError') ) ) def test_manage_renameObject(self): self._doCanonTest( 'manage_renameObject', ( ( ( 'o2', 'n2' ), ['o1', 'n2', 'o3', 'o4'], None ) , ( ( 'o3', 'n3' ), ['o1', 'o2', 'n3', 'o4'], None ) ) ) def test_tpValues(self): f = self._makeOne() f.o2.isPrincipiaFolderish = True f.o3.isPrincipiaFolderish = True f.o4.isPrincipiaFolderish = True self.failUnlessEqual( f.tpValues(), [f.o2, f.o3, f.o4] ) f.setDefaultSorting('meta_type', False) self.failUnlessEqual( f.tpValues(), [f.o3, f.o2, f.o4] ) f.setDefaultSorting('position', True) self.failUnlessEqual( f.tpValues(), [f.o4, f.o3, f.o2] ) def test_z2interfaces(self): from Interface.Verify import verifyClass from OFS.IOrderSupport import IOrderedContainer from OFS.OrderSupport import OrderSupport verifyClass(IOrderedContainer, OrderSupport) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(TestOrderSupport), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/OFS/tests/mime.types-20000664000076400007640000000031310435500327020570 0ustar ajungajung# This is a sample mime.types file. # It contains a single bogus MIME type and extention for testing # purposes. It is not loaded during normal Zope operation. text/x-vnd.zope.test-mime-type-2 ztmt-2 Zope-2.8.7-final/lib/python/OFS/tests/test_XMLExportImport.py0000664000076400007640000000730410435500327023071 0ustar ajungajung############################################################################## # # Copyright (c) 2006 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import unittest import tempfile import transaction from StringIO import StringIO _LONG_DTML = '\n'.join([('> products <> products <> mount-point / name mappingstorage """ # CAUTION: A raw string must be used in the open() call. This is crucial # so that if you happen to be on Windows, and are passed a path containing # backslashes, the backslashes get treated *as* backslashes instead of as # string escape codes. dummy_product_init = """ misc_ = {'a':1} def amethod(self): pass def initialize(context): f=open(r'%s', 'w') f.write('didit') f.close() context.registerClass( meta_type='grabass', permission='aPermission', constructors=(amethod,), legacy=(amethod,)) """ def getSchema(): startup = os.path.dirname(os.path.realpath(Zope2.Startup.__file__)) schemafile = os.path.join(startup, 'zopeschema.xml') return ZConfig.loadSchema(schemafile) def getApp(): from App.ZApplication import ZApplicationWrapper DB = getConfiguration().dbtab.getDatabase('/') return ZApplicationWrapper(DB, 'Application', Application, (), 'foo')() original_config = None class TestProductInit( unittest.TestCase ): """ Test the application initializer object """ def setUp(self): global original_config if original_config is None: original_config = getConfiguration() self.schema = getSchema() os.makedirs(TEMPNAME) os.makedirs(TEMPPRODUCTS) os.makedirs(TEMPPRODUCTS2) def tearDown(self): import App.config del self.schema App.config.setConfiguration(original_config) shutil.rmtree(TEMPNAME) Products.__path__ = [d for d in Products.__path__ if os.path.exists(d)] def configure(self, text): # We have to create a directory of our own since the existence # of the directory is checked. This handles this in a # platform-independent way. schema = self.schema text = text.replace("<>", TEMPNAME) text = text.replace("<>", TEMPPRODUCTS) text = text.replace("<>", TEMPPRODUCTS2) sio = cStringIO.StringIO(text) conf, handler = ZConfig.loadConfigFile(schema, sio) from Zope2.Startup.handlers import handleConfig handleConfig(conf, handler) self.assertEqual(conf.instancehome, TEMPNAME) setConfiguration(conf) def makeProduct(self, proddir): os.makedirs(proddir) f = open(os.path.join(proddir, '__init__.py'), 'w') f.write('#foo') f.close() def makeFakeProducts(self): for name in FAKEPRODUCTS: proddir = os.path.join(TEMPPRODUCTS, name) self.makeProduct(proddir) def test_get_products(self): self.makeFakeProducts() self.configure(cfg) from OFS.Application import get_products names = [x[1] for x in get_products()] for name in FAKEPRODUCTS: self.assert_(name in names) def test_empty_dir_on_products_path_is_not_product(self): self.makeFakeProducts() os.makedirs(os.path.join(TEMPPRODUCTS, 'gleeb')) self.configure(cfg) from OFS.Application import get_products names = [x[1] for x in get_products()] for name in FAKEPRODUCTS: self.assert_(name in names) self.assert_('gleeb' not in names) def test_file_on_products_path_is_not_product(self): self.makeFakeProducts() f = open(os.path.join(TEMPPRODUCTS, 'README.txt'), 'w') f.write('#foo') f.close() self.configure(cfg) from OFS.Application import get_products names = [x[1] for x in get_products()] for name in FAKEPRODUCTS: self.assert_(name in names) self.assert_('README.txt' not in names) def test_multiple_product_paths(self): self.makeFakeProducts() self.makeProduct(os.path.join(TEMPPRODUCTS2, 'another')) self.configure(cfg) from OFS.Application import get_products names = [x[1] for x in get_products()] for name in FAKEPRODUCTS: self.assert_(name in names) self.assert_('another' in names) def test_import_products(self): self.makeFakeProducts() self.configure(cfg) from OFS.Application import import_products names = import_products() for name in FAKEPRODUCTS: assert name in names def test_import_product_throws(self): self.makeProduct(os.path.join(TEMPPRODUCTS, 'abar')) f = open(os.path.join(TEMPPRODUCTS, 'abar', '__init__.py'), 'w') f.write('Syntax Error!') f.close() self.configure(cfg) self.assertRaises(SyntaxError, self.import_bad_product) def import_bad_product(self): from OFS.Application import import_product import_product(TEMPPRODUCTS, 'abar', raise_exc=1) def test_install_product(self): self.makeProduct(os.path.join(TEMPPRODUCTS, 'abaz')) f = open(os.path.join(TEMPPRODUCTS, 'abaz', '__init__.py'), 'w') doneflag = os.path.join(TEMPPRODUCTS, 'abaz', 'doneflag') f.write(dummy_product_init % doneflag) f.close() self.configure(cfg) from OFS.Application import install_product, get_folder_permissions,\ Application import Products from OFS.Folder import Folder app = getApp() meta_types = [] install_product(app, TEMPPRODUCTS, 'abaz', meta_types, get_folder_permissions(), raise_exc=1) # misc_ dictionary is updated self.assert_(Application.misc_.__dict__.has_key('abaz')) # initialize is called self.assert_(os.path.exists(doneflag)) # Methods installed into folder self.assert_(hasattr(Folder, 'amethod')) # permission roles put into folder self.assert_(hasattr(Folder, 'amethod__roles__')) # Products.meta_types updated self.assert_( {'name': 'grabass', 'action': 'manage_addProduct/abaz/amethod', 'product': 'abaz', 'permission': 'aPermission', 'visibility': 'Global', 'interfaces': (), 'instance': None, 'container_filter': None} in Products.meta_types) def test_install_products(self): self.makeFakeProducts() self.configure(cfg) app = getApp() from OFS.Application import install_products install_products(app) obids = app.Control_Panel.Products.objectIds() for name in FAKEPRODUCTS: assert name in obids def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( TestProductInit ) ) return suite def main(): unittest.main(defaultTest='test_suite') if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/OFS/tests/test.gif0000775000076400007640000000157610435500327020101 0ustar ajungajungGIF89aPPP@@@@@!,[Hp *\PaB".|bċ3XĎAɑ!Ń6"`J UDycɑ)+f\sNYR|fÖ>OZi;Zope-2.8.7-final/lib/python/OFS/tests/testProperties.py0000664000076400007640000000475610435500327022041 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import os, sys, unittest from OFS.PropertyManager import PropertyManager from OFS.PropertySheets import PropertySheet class TestObject(PropertyManager): pass class TestProperties( unittest.TestCase ): """Property management tests.""" def testLinesPropertyIsTuple( self ): inst = TestObject() inst._setProperty('prop', ['xxx', 'yyy'], 'lines') self.failUnless(type(inst.getProperty('prop')) == type(())) self.failUnless(type(inst.prop) == type(())) inst._setPropValue('prop', ['xxx', 'yyy']) self.failUnless(type(inst.getProperty('prop')) == type(())) self.failUnless(type(inst.prop) == type(())) inst._updateProperty('prop', ['xxx', 'yyy']) self.failUnless(type(inst.getProperty('prop')) == type(())) self.failUnless(type(inst.prop) == type(())) inst.manage_addProperty('prop2', ['xxx', 'yyy'], 'lines') self.failUnless(type(inst.getProperty('prop2')) == type(())) self.failUnless(type(inst.prop2) == type(())) def testPropertySheetLinesPropertyIsTuple(self): inst = PropertySheet('foo') inst._setProperty('prop', ['xxx', 'yyy'], 'lines') self.failUnless(type(inst.getProperty('prop')) == type(())) self.failUnless(type(inst.prop) == type(())) inst._updateProperty('prop', ['xxx', 'yyy']) self.failUnless(type(inst.getProperty('prop')) == type(())) self.failUnless(type(inst.prop) == type(())) inst.manage_addProperty('prop2', ['xxx', 'yyy'], 'lines') self.failUnless(type(inst.getProperty('prop2')) == type(())) self.failUnless(type(inst.prop2) == type(())) def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( TestProperties ) ) return suite def main(): unittest.main(defaultTest='test_suite') if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/OFS/tests/testAppInitializer.py0000664000076400007640000001760310435500327022624 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import os, sys, unittest, tempfile, cStringIO import ZODB from OFS.Application import Application, AppInitializer, get_products import Zope2.Startup import ZConfig from App.config import getConfiguration, setConfiguration TEMPNAME = tempfile.mktemp() TEMPPRODUCTS = os.path.join(TEMPNAME, "Products") bad_cfg = """ instancehome <> mount-point / name mappingstorage """ good_cfg = bad_cfg + """ # Temporary storage database (for sessions) name temporary storage for sessioning mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer """ def getSchema(): startup = os.path.dirname(os.path.realpath(Zope2.Startup.__file__)) schemafile = os.path.join(startup, 'zopeschema.xml') return ZConfig.loadSchema(schemafile) def getApp(): from App.ZApplication import ZApplicationWrapper DB = getConfiguration().dbtab.getDatabase('/') return ZApplicationWrapper(DB, 'Application', Application, (), 'foo')() original_config = None class TestInitialization( unittest.TestCase ): """ Test the application initializer object """ def setUp(self): global original_config if original_config is None: original_config = getConfiguration() self.schema = getSchema() os.makedirs(TEMPNAME) os.makedirs(TEMPPRODUCTS) def tearDown(self): import App.config del self.schema App.config.setConfiguration(original_config) os.rmdir(TEMPPRODUCTS) os.rmdir(TEMPNAME) import Products Products.__path__ = [d for d in Products.__path__ if os.path.exists(d)] def configure(self, text): # We have to create a directory of our own since the existence # of the directory is checked. This handles this in a # platform-independent way. schema = self.schema sio = cStringIO.StringIO(text.replace("<>", TEMPNAME)) conf, handler = ZConfig.loadConfigFile(schema, sio) self.assertEqual(conf.instancehome, TEMPNAME) setConfiguration(conf) def getOne(self): app = getApp() return AppInitializer(app) def test_install_cp_and_products(self): self.configure(good_cfg) i = self.getOne() app = i.getApp() i.install_cp_and_products() self.failUnless(hasattr(app, 'Control_Panel')) self.assertEqual(app.Control_Panel.meta_type, 'Control Panel') self.failUnless(hasattr(app.Control_Panel, 'Products')) self.assertEqual(app.Control_Panel.Products.meta_type, 'Product Management') def test_install_tempfolder_and_sdc(self): self.configure(good_cfg) i = self.getOne() i.install_tempfolder_and_sdc() app = i.getApp() self.assertEqual(app.temp_folder.meta_type, 'Temporary Folder') self.assertEqual(app.temp_folder.session_data.meta_type, 'Transient Object Container') self.failUnless(app._getInitializerFlag('temp_folder')) def test_install_tempfolder_and_sdc_status(self): self.configure(good_cfg) i = self.getOne() status = i.install_tempfolder_and_sdc() self.failUnless(status) i = self.getOne() self.configure(bad_cfg) status = i.install_tempfolder_and_sdc() self.failIf(status) def test_install_tempfolder_and_sdc_unlimited_sessions(self): unlimited_cfg = good_cfg + """ maximum-number-of-session-objects 0 """ self.configure(unlimited_cfg) i = self.getOne() status = i.install_tempfolder_and_sdc() self.failUnless(status) sdc = i.getApp().temp_folder.session_data self.assertEqual(sdc.getSubobjectLimit(), 0) def test_install_browser_id_manager(self): self.configure(good_cfg) i = self.getOne() app = i.getApp() i.install_browser_id_manager() self.assertEqual(app.browser_id_manager.meta_type,'Browser Id Manager') self.failUnless(app._getInitializerFlag('browser_id_manager')) def test_install_virtual_hosting(self): self.configure(good_cfg) i = self.getOne() app = i.getApp() i.install_virtual_hosting() self.assertEqual(app.virtual_hosting.meta_type,'Virtual Host Monster') self.failUnless(app._getInitializerFlag('virtual_hosting')) def test_install_session_data_manager(self): self.configure(good_cfg) i = self.getOne() i.install_session_data_manager() app = i.getApp() self.assertEqual(app.session_data_manager.meta_type, 'Session Data Manager') self.failUnless(app._getInitializerFlag('session_data_manager')) def test_install_required_roles(self): self.configure(good_cfg) i = self.getOne() i.install_required_roles() app = i.getApp() self.failUnless('Owner' in app.__ac_roles__) self.failUnless('Authenticated' in app.__ac_roles__) def test_install_zglobals(self): from BTrees.OOBTree import OOBTree self.configure(good_cfg) i = self.getOne() i.install_zglobals() root = i.getApp()._p_jar.root() self.failUnless(root.has_key('ZGlobals')) self.failUnless(isinstance(root['ZGlobals'], OOBTree)) def test_install_inituser(self): fname = os.path.join(TEMPNAME, 'inituser') f = open(fname, 'w') f.write('theuser:password') f.close() try: self.configure(good_cfg) i = self.getOne() i.install_inituser() app = i.getApp() self.failUnless(app.acl_users.getUser('theuser')) finally: if os.path.exists(fname): os.unlink(fname) def test_install_errorlog(self): self.configure(good_cfg) i = self.getOne() i.install_errorlog() app = i.getApp() self.assertEqual(app.error_log.meta_type, 'Site Error Log') self.failUnless(app._getInitializerFlag('error_log')) def test_install_products(self): self.configure(good_cfg) i = self.getOne() i.install_products() self.failUnless(Application.misc_.__dict__.has_key('OFSP')) def test_install_standards(self): self.configure(good_cfg) i = self.getOne() i.install_products() # required i .install_standards() app = i.getApp() self.assertEqual(app.index_html.meta_type, 'DTML Method') self.assertEqual(app.standard_error_message.meta_type, 'DTML Method') self.assertEqual(app.standard_html_header.meta_type, 'DTML Method') self.assertEqual(app.standard_html_footer.meta_type, 'DTML Method') self.assertEqual(getattr(app, 'standard_template.pt').meta_type, 'Page Template') self.failUnless(hasattr(app, '_standard_objects_have_been_added')) def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( TestInitialization ) ) return suite def main(): unittest.main(defaultTest='test_suite') if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/OFS/tests/testRanges.py0000664000076400007640000003155610435500327021122 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os, sys, unittest import string, random, cStringIO, time, re import ZODB import transaction from OFS.Application import Application from OFS.Folder import manage_addFolder from OFS.Image import manage_addFile from Testing.makerequest import makerequest from webdav.common import rfc1123_date from mimetools import Message from multifile import MultiFile def makeConnection(): import ZODB from ZODB.DemoStorage import DemoStorage s = DemoStorage(quota=(1<<20)) return ZODB.DB( s ).open() def createBigFile(): # Create a file that is several 1<<16 blocks of data big, to force the # use of chained Pdata objects. # Make sure we create a file that isn't of x * 1<<16 length! Coll #671 size = (1<<16) * 5 + 12345 file = cStringIO.StringIO() def addLetter(x, add=file.write, l=string.letters, c=random.choice): add(c(l)) filter(addLetter, range(size)) return file TESTFOLDER_NAME = 'RangesTestSuite_testFolder' BIGFILE = createBigFile() class TestRequestRange(unittest.TestCase): # Test case setup and teardown def setUp(self): self.responseOut = cStringIO.StringIO() self.connection = makeConnection() try: r = self.connection.root() a = Application() r['Application'] = a self.root = a self.app = makerequest(self.root, stdout=self.responseOut) try: self.app._delObject(TESTFOLDER_NAME) except AttributeError: pass manage_addFolder(self.app, TESTFOLDER_NAME) folder = getattr( self.app, TESTFOLDER_NAME ) data = string.letters manage_addFile( folder, 'file' , file=data, content_type='text/plain') self.file = folder.file self.data = data # Hack, we need a _p_mtime for the file, so we make sure that it # has one. We use a subtransaction, which means we can rollback # later and pretend we didn't touch the ZODB. transaction.commit() except: self.connection.close() raise def tearDown(self): try: self.app._delObject(TESTFOLDER_NAME) except AttributeError: pass transaction.abort() self.app._p_jar.sync() self.connection.close() self.app = None del self.app # Utility methods def uploadBigFile(self): self.file.manage_upload(BIGFILE) self.data = BIGFILE.getvalue() def doGET(self, request, response): rv = self.file.index_html(request, response) # Large files are written to resposeOut directly, small ones are # returned from the index_html method. body = self.responseOut.getvalue() # Chop off any printed headers (only when response.write was used) if body: body = string.split(body, '\n\n', 1)[1] return body + rv def createLastModifiedDate(self, offset=0): return rfc1123_date(self.file._p_mtime + offset) def expectUnsatisfiable(self, range): req = self.app.REQUEST rsp = req.RESPONSE # Add the Range header req.environ['HTTP_RANGE'] = 'bytes=%s' % range body = self.doGET(req, rsp) self.failUnless(rsp.getStatus() == 416, 'Expected a 416 status, got %s' % rsp.getStatus()) expect_content_range = 'bytes */%d' % len(self.data) content_range = rsp.getHeader('content-range') self.failIf(content_range is None, 'No Content-Range header was set!') self.failUnless(content_range == expect_content_range, 'Received incorrect Content-Range header. Expected %s, got %s' % ( `expect_content_range`, `content_range`)) self.failUnless(body == '', 'index_html returned %s' % `body`) def expectOK(self, rangeHeader, if_range=None): req = self.app.REQUEST rsp = req.RESPONSE # Add headers req.environ['HTTP_RANGE'] = rangeHeader if if_range is not None: req.environ['HTTP_IF_RANGE'] = if_range body = self.doGET(req, rsp) self.failUnless(rsp.getStatus() == 200, 'Expected a 200 status, got %s' % rsp.getStatus()) def expectSingleRange(self, range, start, end, if_range=None): req = self.app.REQUEST rsp = req.RESPONSE # Add headers req.environ['HTTP_RANGE'] = 'bytes=%s' % range if if_range is not None: req.environ['HTTP_IF_RANGE'] = if_range body = self.doGET(req, rsp) self.failUnless(rsp.getStatus() == 206, 'Expected a 206 status, got %s' % rsp.getStatus()) expect_content_range = 'bytes %d-%d/%d' % ( start, end - 1, len(self.data)) content_range = rsp.getHeader('content-range') self.failIf(content_range is None, 'No Content-Range header was set!') self.failUnless(content_range == expect_content_range, 'Received incorrect Content-Range header. Expected %s, got %s' % ( `expect_content_range`, `content_range`)) self.failIf(rsp.getHeader('content-length') != str(len(body)), 'Incorrect Content-Length is set! Expected %s, got %s.' % ( str(len(body)), rsp.getHeader('content-length'))) self.failUnless(body == self.data[start:end], 'Incorrect range returned, expected %s, got %s' % ( `self.data[start:end]`, `body`)) def expectMultipleRanges(self, range, sets, draft=0, rangeParse=re.compile('bytes\s*(\d+)-(\d+)/(\d+)')): req = self.app.REQUEST rsp = req.RESPONSE # Add headers req.environ['HTTP_RANGE'] = 'bytes=%s' % range if draft: req.environ['HTTP_REQUEST_RANGE'] = 'bytes=%s' % range body = self.doGET(req, rsp) self.failUnless(rsp.getStatus() == 206, 'Expected a 206 status, got %s' % rsp.getStatus()) self.failIf(rsp.getHeader('content-range'), 'The Content-Range header should not be set!') ct = string.split(rsp.getHeader('content-type'), ';')[0] draftprefix = draft and 'x-' or '' self.failIf(ct != 'multipart/%sbyteranges' % draftprefix, "Incorrect Content-Type set. Expected 'multipart/%sbyteranges', " "got %s" % (draftprefix, ct)) if rsp.getHeader('content-length'): self.failIf(rsp.getHeader('content-length') != str(len(body)), 'Incorrect Content-Length is set! Expected %s, got %s.' % ( str(len(body)), rsp.getHeader('content-length'))) # Decode the multipart message bodyfile = cStringIO.StringIO('Content-Type: %s\n\n%s' % ( rsp.getHeader('content-type'), body)) bodymessage = Message(bodyfile) partfiles = MultiFile(bodyfile) partfiles.push(bodymessage.getparam('boundary')) partmessages = [] add = partmessages.append while partfiles.next(): add(Message(cStringIO.StringIO(partfiles.read()))) # Check the different parts returnedRanges = [] add = returnedRanges.append for part in partmessages: range = part['content-range'] start, end, size = rangeParse.search(range).groups() start, end, size = int(start), int(end), int(size) end = end + 1 self.failIf(size != len(self.data), 'Part Content-Range header reported incorrect length. ' 'Expected %d, got %d.' % (len(self.data), size)) part.rewindbody() body = part.fp.read() # Whotcha! Bug in MultiFile; the CRLF that is part of the boundary # is returned as part of the body. Note that this bug is resolved # in Python 2.2. if body[-2:] == '\r\n': body = body[:-2] self.failIf(len(body) != end - start, 'Part (%d, %d) is of wrong length, expected %d, got %d.' % ( start, end, end - start, len(body))) self.failIf(body != self.data[start:end], 'Part (%d, %d) has incorrect data. Expected %s, got %s.' % ( start, end, `self.data[start:end]`, `body`)) add((start, end)) # Copmare the ranges used with the expected range sets. self.failIf(returnedRanges != sets, 'Got unexpected sets, expected %s, got %s' % ( sets, returnedRanges)) # Unsatisfiable requests def testNegativeZero(self): self.expectUnsatisfiable('-0') def testStartBeyondLength(self): self.expectUnsatisfiable('1000-') def testMultipleUnsatisfiable(self): self.expectUnsatisfiable('1000-1001,2000-,-0') # Malformed Range header def testGarbage(self): self.expectOK('kjhdkjhd = ew;jkj h eewh ew') def testIllegalSpec(self): self.expectOK('notbytes=0-1000') # Single ranges def testSimpleRange(self): self.expectSingleRange('3-7', 3, 8) def testOpenEndedRange(self): self.expectSingleRange('3-', 3, len(self.data)) def testSuffixRange(self): l = len(self.data) self.expectSingleRange('-3', l - 3, l) def testWithNegativeZero(self): # A satisfiable and an unsatisfiable range self.expectSingleRange('-0,3-23', 3, 24) def testEndOverflow(self): l = len(self.data) start, end = l - 10, l + 10 range = '%d-%d' % (start, end) self.expectSingleRange(range, start, len(self.data)) def testBigFile(self): # Files of size 1<<16 are stored in linked Pdata objects. They are # treated seperately in the range code. self.uploadBigFile() join = 3 * (1<<16) # A join between two linked objects start = join - 1000 end = join + 1000 range = '%d-%d' % (start, end - 1) self.expectSingleRange(range, start, end) def testBigFileEndOverflow(self): self.uploadBigFile() l = len(self.data) start, end = l - 100, l + 100 range = '%d-%d' % (start, end) self.expectSingleRange(range, start, len(self.data)) # Multiple ranges def testAdjacentRanges(self): self.expectMultipleRanges('21-25,10-20', [(21, 26), (10, 21)]) def testMultipleRanges(self): self.expectMultipleRanges('3-7,10-15', [(3, 8), (10, 16)]) def testMultipleRangesDraft(self): self.expectMultipleRanges('3-7,10-15', [(3, 8), (10, 16)], draft=1) def testMultipleRangesBigFile(self): self.uploadBigFile() self.expectMultipleRanges('3-700,10-15,-10000', [(3, 701), (10, 16), (len(self.data) - 10000, len(self.data))]) def testMultipleRangesBigFileOutOfOrder(self): self.uploadBigFile() self.expectMultipleRanges('10-15,-10000,70000-80000', [(10, 16), (len(self.data) - 10000, len(self.data)), (70000, 80001)]) def testMultipleRangesBigFileEndOverflow(self): self.uploadBigFile() l = len(self.data) start, end = l - 100, l + 100 self.expectMultipleRanges('3-700,%s-%s' % (start, end), [(3, 701), (len(self.data) - 100, len(self.data))]) # If-Range headers def testIllegalIfRange(self): # We assume that an illegal if-range is to be ignored, just like an # illegal if-modified since. self.expectSingleRange('10-25', 10, 26, if_range='garbage') def testEqualIfRangeDate(self): self.expectSingleRange('10-25', 10, 26, if_range=self.createLastModifiedDate()) def testIsModifiedIfRangeDate(self): self.expectOK('21-25,10-20', if_range=self.createLastModifiedDate(offset=-100)) def testIsNotModifiedIfRangeDate(self): self.expectSingleRange('10-25', 10, 26, if_range=self.createLastModifiedDate(offset=100)) def testEqualIfRangeEtag(self): self.expectSingleRange('10-25', 10, 26, if_range=self.file.http__etag()) def testNotEqualIfRangeEtag(self): self.expectOK('10-25', if_range=self.file.http__etag() + 'bar') def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( TestRequestRange ) ) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/OFS/tests/testContentTypes.py0000664000076400007640000000505610435500327022336 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of the content_types extension mechanism. $Id: testContentTypes.py 24763 2004-05-17 05:59:28Z philikon $ """ import mimetypes import os.path import sys import unittest from OFS import content_types try: __file__ except NameError: __file__ = os.path.realpath(sys.argv[0]) here = os.path.dirname(os.path.abspath(__file__)) MIME_TYPES_1 = os.path.join(here, "mime.types-1") MIME_TYPES_2 = MIME_TYPES_1[:-1] + "2" class ContentTypesTestCase(unittest.TestCase): def setUp(self): self._old_state = mimetypes.__dict__.copy() def tearDown(self): mimetypes.__dict__.update(self._old_state) def check_types_count(self, delta): self.assertEqual(len(mimetypes.types_map), len(self._old_state["types_map"]) + delta) def test_add_one_file(self): ntypes = len(mimetypes.types_map) content_types.add_files([MIME_TYPES_1]) ctype, encoding = content_types.guess_content_type("foo.ztmt-1") self.assert_(encoding is None) self.assertEqual(ctype, "text/x-vnd.zope.test-mime-type-1") ctype, encoding = content_types.guess_content_type("foo.ztmt-1.gz") self.assertEqual(encoding, "gzip") self.assertEqual(ctype, "text/x-vnd.zope.test-mime-type-1") self.check_types_count(1) def test_add_two_files(self): ntypes = len(mimetypes.types_map) content_types.add_files([MIME_TYPES_1, MIME_TYPES_2]) ctype, encoding = content_types.guess_content_type("foo.ztmt-1") self.assert_(encoding is None) self.assertEqual(ctype, "text/x-vnd.zope.test-mime-type-1") ctype, encoding = content_types.guess_content_type("foo.ztmt-2") self.assert_(encoding is None) self.assertEqual(ctype, "text/x-vnd.zope.test-mime-type-2") self.check_types_count(2) def test_suite(): return unittest.makeSuite(ContentTypesTestCase) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/OFS/tests/testFileAndImage.py0000664000076400007640000002302010435500327022133 0ustar ajungajungimport unittest import Testing import Zope2 Zope2.startup() import os, sys import time from cStringIO import StringIO from OFS.Application import Application from OFS.SimpleItem import SimpleItem from OFS.Cache import ZCM_MANAGERS from OFS.Image import Pdata from ZPublisher.HTTPRequest import HTTPRequest from ZPublisher.HTTPResponse import HTTPResponse from App.Common import rfc1123_date from Testing.makerequest import makerequest from zExceptions import Redirect import transaction try: here = os.path.dirname(os.path.abspath(__file__)) except: here = os.path.dirname(os.path.abspath(sys.argv[0])) imagedata = os.path.join(here, 'test.gif') filedata = os.path.join(here, 'test.gif') def makeConnection(): import ZODB from ZODB.DemoStorage import DemoStorage s = DemoStorage(quota=(1<<20)) return ZODB.DB( s ).open() def aputrequest(file, content_type): resp = HTTPResponse(stdout=sys.stdout) environ = {} environ['SERVER_NAME']='foo' environ['SERVER_PORT']='80' environ['REQUEST_METHOD'] = 'PUT' environ['CONTENT_TYPE'] = content_type req = HTTPRequest(stdin=file, environ=environ, response=resp) return req class DummyCache: def __init__(self): self.clear() def ZCache_set(self, ob, data, view_name='', keywords=None, mtime_func=None): self.set = (ob, data) def ZCache_get(self, ob, data, view_name='', keywords=None, mtime_func=None): self.get = ob if self.si: return si def ZCache_invalidate(self, ob): self.invalidated = ob def clear(self): self.set=None self.get=None self.invalidated = None self.si = None def setStreamIterator(self, si): self.si = si ADummyCache=DummyCache() class DummyCacheManager(SimpleItem): def ZCacheManager_getCache(self): return ADummyCache class FileTests(unittest.TestCase): data = open(filedata, 'rb').read() content_type = 'application/octet-stream' factory = 'manage_addFile' def setUp( self ): self.connection = makeConnection() try: r = self.connection.root() a = Application() r['Application'] = a self.root = a responseOut = self.responseOut = StringIO() self.app = makerequest( self.root, stdout=responseOut ) self.app.dcm = DummyCacheManager() factory = getattr(self.app, self.factory) factory('file', file=self.data, content_type=self.content_type) self.app.file.ZCacheable_setManagerId('dcm') self.app.file.ZCacheable_setEnabled(enabled=1) setattr(self.app, ZCM_MANAGERS, ('dcm',)) # Hack, we need a _p_mtime for the file, so we make sure that it # has one. transaction.commit() except: self.connection.close() raise transaction.begin() self.file = getattr( self.app, 'file' ) def tearDown( self ): del self.file transaction.abort() self.connection.close() del self.app del self.responseOut del self.root del self.connection ADummyCache.clear() def testViewImageOrFile(self): self.assertRaises(Redirect, self.file.view_image_or_file, 'foo') def testUpdateData(self): self.file.update_data('foo') self.assertEqual(self.file.size, 3) self.assertEqual(self.file.data, 'foo') self.failUnless(ADummyCache.invalidated) self.failUnless(ADummyCache.set) def testReadData(self): s = "a" * (2 << 16) f = StringIO(s) data, size = self.file._read_data(f) self.failUnless(isinstance(data, Pdata)) self.assertEqual(str(data), s) self.assertEqual(len(s), len(str(data))) self.assertEqual(len(s), size) def testBigPdata(self): # Test that a big enough string is split into several Pdata # From a file s = "a" * (1 << 16) * 3 data, size = self.file._read_data(StringIO(s)) self.failIfEqual(data.next, None) # From a string data, size = self.file._read_data(s) self.failIfEqual(data.next, None) def testManageEditWithFileData(self): self.file.manage_edit('foobar', 'text/plain', filedata='ASD') self.assertEqual(self.file.title, 'foobar') self.assertEqual(self.file.content_type, 'text/plain') self.failUnless(ADummyCache.invalidated) self.failUnless(ADummyCache.set) def testManageEditWithoutFileData(self): self.file.manage_edit('foobar', 'text/plain') self.assertEqual(self.file.title, 'foobar') self.assertEqual(self.file.content_type, 'text/plain') self.failUnless(ADummyCache.invalidated) def testManageUpload(self): f = StringIO('jammyjohnson') self.file.manage_upload(f) self.assertEqual(self.file.data, 'jammyjohnson') self.assertEqual(self.file.content_type, 'application/octet-stream') def testIfModSince(self): now = time.time() e = {'SERVER_NAME':'foo', 'SERVER_PORT':'80', 'REQUEST_METHOD':'GET'} # not modified since t_notmod = rfc1123_date(now) e['HTTP_IF_MODIFIED_SINCE'] = t_notmod out = StringIO() resp = HTTPResponse(stdout=out) req = HTTPRequest(sys.stdin, e, resp) data = self.file.index_html(req,resp) self.assertEqual(resp.getStatus(), 304) self.assertEqual(data, '') # modified since t_mod = rfc1123_date(now - 100) e['HTTP_IF_MODIFIED_SINCE'] = t_mod out = StringIO() resp = HTTPResponse(stdout=out) req = HTTPRequest(sys.stdin, e, resp) data = self.file.index_html(req,resp) self.assertEqual(resp.getStatus(), 200) self.assertEqual(data, str(self.file.data)) def testPUT(self): s = '# some python\n' # with content type data = StringIO(s) req = aputrequest(data, 'text/x-python') req.processInputs() self.file.PUT(req, req.RESPONSE) self.assertEqual(self.file.content_type, 'text/x-python') self.assertEqual(str(self.file.data), s) # without content type data.seek(0) req = aputrequest(data, '') req.processInputs() self.file.PUT(req, req.RESPONSE) self.assertEqual(self.file.content_type, 'text/x-python') self.assertEqual(str(self.file.data), s) def testIndexHtmlWithPdata(self): self.file.manage_upload('a' * (2 << 16)) # 128K self.file.index_html(self.app.REQUEST, self.app.REQUEST.RESPONSE) self.assert_(self.app.REQUEST.RESPONSE._wrote) def testIndexHtmlWithString(self): self.file.manage_upload('a' * 100) # 100 bytes self.file.index_html(self.app.REQUEST, self.app.REQUEST.RESPONSE) self.assert_(not self.app.REQUEST.RESPONSE._wrote) def testStr(self): self.assertEqual(str(self.file), self.data) def testFindSupport_not_text(self): self.file.manage_edit('foobar', 'application/octet-stream', filedata=''.join([chr(x) for x in range(256)])) self.assertEqual(self.file.PrincipiaSearchSource(), '') def testFindSupport_text(self): self.file.manage_edit('foobar', 'text/plain', filedata='Now is the time for all good men to ' 'come to the aid of the Party.') self.failUnless('Party' in self.file.PrincipiaSearchSource()) def testFindFile(self): self.file.manage_edit('foobar', 'text/plain', filedata='Now is the time for all good men to ' 'come to the aid of the Party.') results = self.app.ZopeFind(self.app, obj_searchterm='Party') self.assertEqual(len(results), 1) self.assertEqual(results[0][1], self.file) def test_z2interfaces(self): from Interface.Verify import verifyClass from OFS.Image import File from webdav.WriteLockInterface import WriteLockInterface from ZPublisher.HTTPRangeSupport import HTTPRangeInterface verifyClass(HTTPRangeInterface, File) verifyClass(WriteLockInterface, File) class ImageTests(FileTests): data = open(filedata, 'rb').read() content_type = 'image/gif' factory = 'manage_addImage' def testUpdateData(self): self.file.update_data(self.data) self.assertEqual(self.file.size, len(self.data)) self.assertEqual(self.file.data, self.data) self.assertEqual(self.file.width, 16) self.assertEqual(self.file.height, 16) self.failUnless(ADummyCache.invalidated) self.failUnless(ADummyCache.set) def testStr(self): self.assertEqual(str(self.file), ('')) def testTag(self): self.assertEqual(self.file.tag(), ('')) def testViewImageOrFile(self): pass # dtml method,screw it def test_z2interfaces(self): from Interface.Verify import verifyClass from OFS.Image import Image from webdav.WriteLockInterface import WriteLockInterface verifyClass(WriteLockInterface, Image) def test_suite(): return unittest.TestSuite(( unittest.makeSuite(FileTests), unittest.makeSuite(ImageTests), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/OFS/tests/testObjectManager.py0000664000076400007640000002513710435500327022402 0ustar ajungajungimport unittest from App.config import getConfiguration from Acquisition import Implicit, aq_base, aq_parent from AccessControl.SecurityManagement import newSecurityManager from AccessControl.SecurityManagement import noSecurityManager from AccessControl.User import User from AccessControl.SpecialUsers import emergency_user, nobody, system from AccessControl.Owned import EmergencyUserCannotOwn, Owned from OFS.ObjectManager import ObjectManager from OFS.SimpleItem import SimpleItem class FauxRoot( Implicit ): id = '/' def getPhysicalRoot( self ): return self def getPhysicalPath( self ): return () class FauxUser( Implicit ): def __init__( self, id, login ): self._id = id self._login = login def getId( self ): return self._id class DeleteFailed(Exception): pass class ItemForDeletion(SimpleItem): def __init__(self, fail_on_delete=False): self.id = 'stuff' self.before_delete_called = False self.fail_on_delete = fail_on_delete def manage_beforeDelete(self, item, container): self.before_delete_called = True if self.fail_on_delete: raise DeleteFailed return SimpleItem.manage_beforeDelete(self, item, container) class ObjectManagerTests( unittest.TestCase ): def setUp(self): self.saved_cfg_debug_mode = getConfiguration().debug_mode def tearDown( self ): noSecurityManager() getConfiguration().debug_mode = self.saved_cfg_debug_mode def setDebugMode(self, mode): getConfiguration().debug_mode = mode def _getTargetClass( self ): from OFS.ObjectManager import ObjectManager return ObjectManager def _makeOne( self, *args, **kw ): return self._getTargetClass()( *args, **kw ).__of__( FauxRoot() ) def test_setObject_set_owner_with_no_user( self ): om = self._makeOne() newSecurityManager( None, None ) si = SimpleItem( 'no_user' ) om._setObject( 'no_user', si ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_set_owner_with_emergency_user( self ): om = self._makeOne() newSecurityManager( None, emergency_user ) si = SimpleItem( 'should_fail' ) self.assertEqual( si.__ac_local_roles__, None ) self.assertRaises( EmergencyUserCannotOwn , om._setObject, 'should_fail', si ) def test_setObject_set_owner_with_system_user( self ): om = self._makeOne() newSecurityManager( None, system ) si = SimpleItem( 'system' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'system', si ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_set_owner_with_anonymous_user( self ): om = self._makeOne() newSecurityManager( None, nobody ) si = SimpleItem( 'anon' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'anon', si ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_set_owner_with_user( self ): om = self._makeOne() user = User( 'user', '123', (), () ).__of__( FauxRoot() ) newSecurityManager( None, user ) si = SimpleItem( 'user_creation' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'user_creation', si ) self.assertEqual( si.__ac_local_roles__, { 'user': ['Owner'] } ) def test_setObject_set_owner_with_faux_user( self ): om = self._makeOne() user = FauxUser( 'user_id', 'user_login' ).__of__( FauxRoot() ) newSecurityManager( None, user ) si = SimpleItem( 'faux_creation' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'faux_creation', si ) self.assertEqual( si.__ac_local_roles__, { 'user_id': ['Owner'] } ) def test_setObject_no_set_owner_with_no_user( self ): om = self._makeOne() newSecurityManager( None, None ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_no_set_owner_with_emergency_user( self ): om = self._makeOne() newSecurityManager( None, emergency_user ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_no_set_owner_with_system_user( self ): om = self._makeOne() newSecurityManager( None, system ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_no_set_owner_with_anonymous_user( self ): om = self._makeOne() newSecurityManager( None, nobody ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_no_set_owner_with_user( self ): om = self._makeOne() user = User( 'user', '123', (), () ).__of__( FauxRoot() ) newSecurityManager( None, user ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_setObject_no_set_owner_with_faux_user( self ): om = self._makeOne() user = FauxUser( 'user_id', 'user_login' ).__of__( FauxRoot() ) newSecurityManager( None, user ) si = SimpleItem( 'should_be_okay' ) self.assertEqual( si.__ac_local_roles__, None ) om._setObject( 'should_be_okay', si, set_owner=0 ) self.assertEqual( si.__ac_local_roles__, None ) def test_delObject_before_delete(self): # Test that manage_beforeDelete is called om = self._makeOne() ob = ItemForDeletion() om._setObject(ob.getId(), ob) self.assertEqual(ob.before_delete_called, False) om._delObject(ob.getId()) self.assertEqual(ob.before_delete_called, True) def test_delObject_exception_manager(self): # Test exception behavior in manage_beforeDelete # Manager user self.setDebugMode(False) newSecurityManager(None, system) # Manager om = self._makeOne() ob = ItemForDeletion(fail_on_delete=True) om._setObject(ob.getId(), ob) om._delObject(ob.getId()) def test_delObject_exception(self): # Test exception behavior in manage_beforeDelete # non-Manager user self.setDebugMode(False) om = self._makeOne() ob = ItemForDeletion(fail_on_delete=True) om._setObject(ob.getId(), ob) om._delObject(ob.getId()) def test_delObject_exception_debug_manager(self): # Test exception behavior in manage_beforeDelete in debug mode # Manager user self.setDebugMode(True) newSecurityManager(None, system) # Manager om = self._makeOne() ob = ItemForDeletion(fail_on_delete=True) om._setObject(ob.getId(), ob) om._delObject(ob.getId()) def test_delObject_exception_debug(self): # Test exception behavior in manage_beforeDelete in debug mode # non-Manager user # It's the only special case: we let exceptions propagate. self.setDebugMode(True) om = self._makeOne() ob = ItemForDeletion(fail_on_delete=True) om._setObject(ob.getId(), ob) self.assertRaises(DeleteFailed, om._delObject, ob.getId()) def test_delObject_exception_debug_deep(self): # Test exception behavior in manage_beforeDelete in debug mode # non-Manager user # Test for deep subobjects. self.setDebugMode(True) om1 = self._makeOne() om2 = self._makeOne() ob = ItemForDeletion(fail_on_delete=True) om1._setObject('om2', om2, set_owner=False) om2._setObject(ob.getId(), ob) self.assertRaises(DeleteFailed, om1._delObject, 'om2') def test_hasObject(self): om = self._makeOne() self.failIf(om.hasObject('_properties')) self.failIf(om.hasObject('_getOb')) self.failIf(om.hasObject('__of__')) self.failIf(om.hasObject('.')) self.failIf(om.hasObject('..')) self.failIf(om.hasObject('aq_base')) om.zap__ = True self.failIf(om.hasObject('zap__')) self.failIf(om.hasObject('foo')) si = SimpleItem('foo') om._setObject('foo', si) self.assert_(om.hasObject('foo')) om._delObject('foo') self.failIf(om.hasObject('foo')) def test_setObject_checkId_ok(self): om = self._makeOne() si = SimpleItem('1') om._setObject('AB-dash_under0123', si) si = SimpleItem('2') om._setObject('ho.bak~', si) si = SimpleItem('3') om._setObject('dot.comma,dollar$(hi)hash# space', si) si = SimpleItem('4') om._setObject('b@r', si) si = SimpleItem('5') om._setObject('..haha', si) si = SimpleItem('6') om._setObject('.bashrc', si) def test_setObject_checkId_bad(self): from zExceptions import BadRequest om = self._makeOne() si = SimpleItem('111') om._setObject('111', si) si = SimpleItem('2') self.assertRaises(BadRequest, om._setObject, 123, si) self.assertRaises(BadRequest, om._setObject, 'a\x01b', si) self.assertRaises(BadRequest, om._setObject, 'a\\b', si) self.assertRaises(BadRequest, om._setObject, 'a:b', si) self.assertRaises(BadRequest, om._setObject, 'a;b', si) self.assertRaises(BadRequest, om._setObject, '.', si) self.assertRaises(BadRequest, om._setObject, '..', si) self.assertRaises(BadRequest, om._setObject, '_foo', si) self.assertRaises(BadRequest, om._setObject, 'aq_me', si) self.assertRaises(BadRequest, om._setObject, 'bah__', si) self.assertRaises(BadRequest, om._setObject, '111', si) self.assertRaises(BadRequest, om._setObject, 'REQUEST', si) self.assertRaises(BadRequest, om._setObject, '/', si) def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( ObjectManagerTests ) ) return suite if __name__ == "__main__": unittest.main() Zope-2.8.7-final/lib/python/OFS/www/0000775000076400007640000000000010435500327016101 5ustar ajungajungZope-2.8.7-final/lib/python/OFS/www/properties.jpg0000775000076400007640000000536010435500327021006 0ustar ajungajungJFIFHH^Photoshop 3.08BIMxHH@[ (hht @h0 @CN' dVe9CV+SR8BIMHH8BIM8BIM 8BIM' 8BIMH/fflff/ff2Z5-8BIMp8BIM 8BIM8BIM@@8BIM 2JFIFHH'File written by Adobe Photoshop 4.0Adobed            "?   3!1AQa"q2B#$Rb34rC%Scs5&DTdE£t6UeuF'Vfv7GWgw5!1AQaq"2B#R3$brCScs4%&5DTdEU6teuFVfv'7GWgw ?I]^dǏ]ۭjc'%^L]Җ ZN)$b-$-$WWru+N=.pˆI.hdZ]Umc EkeUݺ{ptKIyJs $8Ag*jڰTUe%)$JNʫχKu~u{񾖒D_wu~i/4I/wW7BqZ Q_R[K5XIOU®xx).=jv IX8VHqֵO#,uuPq^ȼm%L|uSc=/Ou_dIxIwHȒTzSﮢ,DE.M:.1.]D,*xOEWjTI$$I)I$JR[+%>eI^\JlZ2bz=^.s_lx+w_ uaL|$IJI$RI$V2J} keѤd\R4xurpqwt $^1L٪W^S)/FY))]Ұ=D^$ʢL+JuVZQi-d$CVS%5T֊2JsQ8BIM'File written by Adobe Photoshop 4.0Adobed           s!1AQa"q2B#R3b$r%C4Scs5D'6Tdt& EFVU(eufv7GWgw8HXhx)9IYiy*:JZjzm!1AQa"q2#BRbr3$4CS%cs5DT &6E'dtU7()󄔤euFVfvGWgw8HXhx9IYiy*:JZjz ?Pk_.ko6MtIiWi$QK $+^,Cc1I~'TOTa>Nqg O$,QOI3{[>៉)Z֏_C5H.}&݀k ̷&5y >df zK g{y1[Y;$gb'Jݪ\f:_Zope-2.8.7-final/lib/python/OFS/www/l_arrow.gif0000775000076400007640000000152510435500327020243 0ustar ajungajungGIF89a111p!,2 H*\C2(Q!Ŋ /Z ǎ5.92C&Ov ;Zope-2.8.7-final/lib/python/OFS/www/ControlPanel_icon.gif0000664000076400007640000000164510435500327022206 0ustar ajungajungGIF89aPPP@@@@@!,H A,( hb‹,xG&>XdE@(Oi8,2&7q$zA?Fe)P;WU(Gu*ShӞ )۷n;Zope-2.8.7-final/lib/python/OFS/www/modified.gif0000775000076400007640000000152010435500327020351 0ustar ajungajungGIF89a PPP@@@@@!, -H (0 Q"E):$p CI@@;Zope-2.8.7-final/lib/python/OFS/www/Properties_icon.gif0000664000076400007640000000162110435500327021734 0ustar ajungajungGIF89aPPP@@@@@!,@nH@Æ @"D"D|1"6V(cIGBXcCȜ9SI/]D0@#`葠ϠBbL1Ӣ>4KxZ;Zope-2.8.7-final/lib/python/OFS/www/locked.gif0000775000076400007640000000156510435500327020043 0ustar ajungajungGIF89aPPP@@@@@!,R @PX!B :bÉ#^  R`ɉDJȜ93̑5)M>&̀;Zope-2.8.7-final/lib/python/OFS/www/UpFolder_icon.gif0000664000076400007640000000162310435500327021322 0ustar ajungajungGIF89aPPP@@@@@!,@pH@, @Åŋ-JD#D 08($qb… W~(`͛8m)`$Ƀ{ y2DxeʇK*D1QNXu'֯N;Zope-2.8.7-final/lib/python/OFS/www/check.gif0000775000076400007640000000154610435500327017656 0ustar ajungajungGIF89aPPP@@@@@!,CH`*L(p 4tbĉ-"l@Ǎ=ZHERLqeH ͛8m;Zope-2.8.7-final/lib/python/OFS/www/r_arrow.gif0000775000076400007640000000152510435500327020251 0ustar ajungajungGIF89a111p!,2 H*\8Æ(qbD)bx ǎ9.bŇ@ ;Zope-2.8.7-final/lib/python/OFS/www/Help_icon.gif0000664000076400007640000000153210435500327020471 0ustar ajungajungGIF89aPPP@@@@@!,@7HA 40!C#:QÊ bpć ;r)QcI'Qn(r;Zope-2.8.7-final/lib/python/OFS/www/broken.gif0000775000076400007640000000157610435500327020064 0ustar ajungajungGIF89aPPP@@@@@!,[Hp *\PaB".|bċ3XĎAɑ!Ń6"`J UDycɑ)+f\sNYR|fÖ>OZi;Zope-2.8.7-final/lib/python/OFS/www/new.gif0000775000076400007640000000153610435500327017371 0ustar ajungajungGIF89a PPP@@@@@!, ;`A DpH| D+jlbCR9$I—0c ;Zope-2.8.7-final/lib/python/OFS/PropertyManager.py0000664000076400007640000003327710435500327020762 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Property management $Id: PropertyManager.py 68072 2006-05-10 08:24:46Z shh $ """ from cgi import escape from types import ListType import ExtensionClass, Globals from ZPublisher.Converters import type_converters from Globals import DTMLFile, MessageDialog from Acquisition import aq_base from Globals import Persistent from zExceptions import BadRequest import ZDOM from PropertySheets import DefaultPropertySheets, vps class PropertyManager(ExtensionClass.Base, ZDOM.ElementWithAttributes): """ The PropertyManager mixin class provides an object with transparent property management. An object which wants to have properties should inherit from PropertyManager. An object may specify that it has one or more predefined properties, by specifying an _properties structure in its class:: _properties=({'id':'title', 'type': 'string', 'mode': 'w'}, {'id':'color', 'type': 'string', 'mode': 'w'}, ) The _properties structure is a sequence of dictionaries, where each dictionary represents a predefined property. Note that if a predefined property is defined in the _properties structure, you must provide an attribute with that name in your class or instance that contains the default value of the predefined property. Each entry in the _properties structure must have at least an 'id' and a 'type' key. The 'id' key contains the name of the property, and the 'type' key contains a string representing the object's type. The 'type' string must be one of the values: 'float', 'int', 'long', 'string', 'lines', 'text', 'date', 'tokens', 'selection', or 'multiple section'. For 'selection' and 'multiple selection' properties, there is an addition item in the property dictionay, 'select_variable' which provides the name of a property or method which returns a list of strings from which the selection(s) can be chosen. Each entry in the _properties structure may *optionally* provide a 'mode' key, which specifies the mutability of the property. The 'mode' string, if present, must contain 0 or more characters from the set 'w','d'. A 'w' present in the mode string indicates that the value of the property may be changed by the user. A 'd' indicates that the user can delete the property. An empty mode string indicates that the property and its value may be shown in property listings, but that it is read-only and may not be deleted. Entries in the _properties structure which do not have a 'mode' key are assumed to have the mode 'wd' (writeable and deleteable). To fully support property management, including the system-provided tabs and user interfaces for working with properties, an object which inherits from PropertyManager should include the following entry in its manage_options structure:: {'label':'Properties', 'action':'manage_propertiesForm',} to ensure that a 'Properties' tab is displayed in its management interface. Objects that inherit from PropertyManager should also include the following entry in its __ac_permissions__ structure:: ('Manage properties', ('manage_addProperty', 'manage_editProperties', 'manage_delProperties', 'manage_changeProperties',)), """ manage_options=( {'label':'Properties', 'action':'manage_propertiesForm', 'help':('OFSP','Properties.stx')}, ) manage_propertiesForm=DTMLFile('dtml/properties', globals(), property_extensible_schema__=1) manage_propertyTypeForm=DTMLFile('dtml/propertyType', globals()) title='' _properties=({'id':'title', 'type': 'string', 'mode':'wd'},) _reserved_names=() __ac_permissions__=( ('Manage properties', ('manage_addProperty', 'manage_editProperties', 'manage_delProperties', 'manage_changeProperties', 'manage_propertiesForm', 'manage_propertyTypeForm', 'manage_changePropertyTypes', )), ('Access contents information', ('hasProperty', 'propertyIds', 'propertyValues','propertyItems', 'getProperty', 'getPropertyType', 'propertyMap', 'propertyLabel', 'propdict', 'valid_property_id', ''), ('Anonymous', 'Manager'), ), ) __propsets__=() propertysheets=vps(DefaultPropertySheets) def valid_property_id(self, id): if not id or id[:1]=='_' or (id[:3]=='aq_') \ or (' ' in id) or hasattr(aq_base(self), id) or escape(id) != id: return 0 return 1 def hasProperty(self, id): """Return true if object has a property 'id'. """ for p in self._properties: if id==p['id']: return 1 return 0 def getProperty(self, id, d=None): """Get the property 'id'. Returns the optional second argument or None if no such property is found. """ if self.hasProperty(id): return getattr(self, id) return d def getPropertyType(self, id): """Get the type of property 'id'. Returns None if no such property exists. """ for md in self._properties: if md['id']==id: return md.get('type', 'string') return None def _wrapperCheck(self, object): # Raise an error if an object is wrapped. if hasattr(object, 'aq_base'): raise ValueError, 'Invalid property value: wrapped object' return def _setPropValue(self, id, value): self._wrapperCheck(value) if type(value) == ListType: value = tuple(value) setattr(self,id,value) def _delPropValue(self, id): delattr(self,id) def _setProperty(self, id, value, type='string'): # for selection and multiple selection properties # the value argument indicates the select variable # of the property self._wrapperCheck(value) if not self.valid_property_id(id): raise BadRequest, 'Invalid or duplicate property id' if type in ('selection', 'multiple selection'): if not hasattr(self, value): raise BadRequest, 'No select variable %s' % value self._properties=self._properties + ( {'id':id, 'type':type, 'select_variable':value},) if type=='selection': self._setPropValue(id, '') else: self._setPropValue(id, []) else: self._properties=self._properties+({'id':id,'type':type},) self._setPropValue(id, value) def _updateProperty(self, id, value): # Update the value of an existing property. If value # is a string, an attempt will be made to convert # the value to the type of the existing property. self._wrapperCheck(value) if not self.hasProperty(id): raise BadRequest, 'The property %s does not exist' % escape(id) if type(value)==type(''): proptype=self.getPropertyType(id) or 'string' if type_converters.has_key(proptype): value=type_converters[proptype](value) self._setPropValue(id, value) def _delProperty(self, id): if not self.hasProperty(id): raise ValueError, 'The property %s does not exist' % escape(id) self._delPropValue(id) self._properties=tuple(filter(lambda i, n=id: i['id'] != n, self._properties)) def propertyIds(self): """Return a list of property ids. """ return map(lambda i: i['id'], self._properties) def propertyValues(self): """Return a list of actual property objects. """ return map(lambda i,s=self: getattr(s,i['id']), self._properties) def propertyItems(self): """Return a list of (id,property) tuples. """ return map(lambda i,s=self: (i['id'],getattr(s,i['id'])), self._properties) def _propertyMap(self): """Return a tuple of mappings, giving meta-data for properties. """ return self._properties def propertyMap(self): """Return a tuple of mappings, giving meta-data for properties. Return copies of the real definitions for security. """ return tuple(map(lambda dict: dict.copy(), self._propertyMap())) def propertyLabel(self, id): """Return a label for the given property id """ for p in self._properties: if p['id'] == id: return p.get('label', id) return id def propdict(self): dict={} for p in self._properties: dict[p['id']]=p return dict # Web interface def manage_addProperty(self, id, value, type, REQUEST=None): """Add a new property via the web. Sets a new property with the given id, type, and value. """ if type_converters.has_key(type): value=type_converters[type](value) self._setProperty(id.strip(), value, type) if REQUEST is not None: return self.manage_propertiesForm(self, REQUEST) def manage_editProperties(self, REQUEST): """Edit object properties via the web. The purpose of this method is to change all property values, even those not listed in REQUEST; otherwise checkboxes that get turned off will be ignored. Use manage_changeProperties() instead for most situations. """ for prop in self._propertyMap(): name=prop['id'] if 'w' in prop.get('mode', 'wd'): if prop['type'] == 'multiple selection': value=REQUEST.get(name, []) else: value=REQUEST.get(name, '') self._updateProperty(name, value) if REQUEST: message="Saved changes." return self.manage_propertiesForm(self,REQUEST, manage_tabs_message=message) def manage_changeProperties(self, REQUEST=None, **kw): """Change existing object properties. Change object properties by passing either a mapping object of name:value pairs {'foo':6} or passing name=value parameters """ if REQUEST is None: props={} else: props=REQUEST if kw: for name, value in kw.items(): props[name]=value propdict=self.propdict() for name, value in props.items(): if self.hasProperty(name): if not 'w' in propdict[name].get('mode', 'wd'): raise BadRequest, '%s cannot be changed' % escape(name) self._updateProperty(name, value) if REQUEST: message="Saved changes." return self.manage_propertiesForm(self,REQUEST, manage_tabs_message=message) # Note - this is experimental, pending some community input. def manage_changePropertyTypes(self, old_ids, props, REQUEST=None): """Replace one set of properties with another Delete all properties that have ids in old_ids, then add a property for each item in props. Each item has a new_id, new_value, and new_type. The type of new_value should match new_type. """ err = self.manage_delProperties(old_ids) if err: if REQUEST is not None: return err return for prop in props: self._setProperty(prop.new_id, prop.new_value, prop.new_type) if REQUEST is not None: return self.manage_propertiesForm(self, REQUEST) def manage_delProperties(self, ids=None, REQUEST=None): """Delete one or more properties specified by 'ids'.""" if REQUEST: # Bugfix for property named "ids" (Casey) if ids == self.getProperty('ids', None): ids = None ids = REQUEST.get('_ids', ids) if ids is None: return MessageDialog( title='No property specified', message='No properties were specified!', action ='./manage_propertiesForm',) propdict=self.propdict() nd=self._reserved_names for id in ids: if not hasattr(aq_base(self), id): raise BadRequest, ( 'The property %s does not exist' % escape(id)) if (not 'd' in propdict[id].get('mode', 'wd')) or (id in nd): return MessageDialog( title ='Cannot delete %s' % id, message='The property %s cannot be deleted.' % escape(id), action ='manage_propertiesForm') self._delProperty(id) if REQUEST is not None: return self.manage_propertiesForm(self, REQUEST) Globals.default__class_init__(PropertyManager) Zope-2.8.7-final/lib/python/OFS/OrderedFolder.py0000664000076400007640000000501710435500327020352 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ 'Folder' with order support. $Id: OrderedFolder.py 24763 2004-05-17 05:59:28Z philikon $ """ from AccessControl import getSecurityManager from AccessControl import Unauthorized from AccessControl.Permissions import add_page_templates from AccessControl.Permissions import add_user_folders from Globals import DTMLFile from Folder import Folder from OrderSupport import OrderSupport manage_addOrderedFolderForm = DTMLFile('dtml/addOrderedFolder', globals()) def manage_addOrderedFolder(self, id, title='', createPublic=0, createUserF=0, REQUEST=None): """Add a new ordered Folder object with id *id*. If the 'createPublic' and 'createUserF' parameters are set to any true value, an 'index_html' and a 'UserFolder' objects are created respectively in the new folder. """ ob = OrderedFolder(id) ob.title = title self._setObject(id, ob) ob = self._getOb(id) checkPermission = getSecurityManager().checkPermission if createUserF: if not checkPermission(add_user_folders, ob): raise Unauthorized, ( 'You are not authorized to add User Folders.' ) ob.manage_addUserFolder() if createPublic: if not checkPermission(add_page_templates, ob): raise Unauthorized, ( 'You are not authorized to add Page Templates.' ) ob.manage_addProduct['PageTemplates'].manage_addPageTemplate( id='index_html', title='') if REQUEST: return self.manage_main(self, REQUEST, update_menu=1) class OrderedFolder(OrderSupport, Folder): """ Extends the default Folder by order support. """ __implements__ = (OrderSupport.__implements__, Folder.__implements__) meta_type='Folder (Ordered)' manage_options = ( OrderSupport.manage_options + Folder.manage_options[1:] ) Zope-2.8.7-final/lib/python/OFS/content_types.py0000664000076400007640000000466010435500327020533 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A utility module for content-type handling. $Id: content_types.py 24764 2004-05-17 06:13:48Z philikon $ """ import mimetypes import os.path import re find_binary = re.compile('[\0-\7]').search def text_type(s): # Yuk. See if we can figure out the type by content. if (s.strip().lower()[:6] == '' or s.find(' 0): return 'text/html' elif s.strip().startswith(' &dtml-title_or_id; Zope-2.8.7-final/lib/python/OFS/standard/standard_html_footer.dtml0000664000076400007640000000002210435500327024133 0ustar ajungajung Zope-2.8.7-final/lib/python/OFS/standard/standard_template.pt0000664000076400007640000000043210435500327023114 0ustar ajungajung The Title

This is where the page's body text goes.
Zope-2.8.7-final/lib/python/OFS/standard/index_html.dtml0000664000076400007640000000003410435500327022067 0ustar ajungajung Zope-2.8.7-final/lib/python/OFS/standard/standard_error_message.dtml0000664000076400007640000000231310435500327024453 0ustar ajungajung

Site Error

An error was encountered while publishing this resource.

Error Type: &dtml-error_type;
Error Value: &dtml-error_value;

Troubleshooting Suggestions

  • This resource may be trying to reference a nonexistent object or variable &dtml-error_value;.
    •
  • The URL may be incorrect.
  • The parameters passed to this resource may be incorrect.
  • A resource that this resource relies on may be encountering an error.

For more detailed information about the error, please refer to the error log.

If the error persists please contact the site maintainer. Thank you for your patience.

Here, events like logging and other actions may also be performed, such as sending mail automatically to the administrator. Zope-2.8.7-final/lib/python/OFS/PropertySheets.py0000664000076400007640000006654610435500327020650 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Property sheets $Id: PropertySheets.py 28714 2005-01-03 20:11:59Z andreasjung $ """ import time, App.Management, Globals, sys from webdav.WriteLockInterface import WriteLockInterface from ZPublisher.Converters import type_converters from Globals import DTMLFile, MessageDialog from Acquisition import Implicit, Explicit from App.Common import rfc1123_date, iso8601_date from webdav.common import urlbase from ExtensionClass import Base from Globals import Persistent from Traversable import Traversable from Acquisition import aq_base from AccessControl import getSecurityManager from webdav.common import isDavCollection from zExceptions import BadRequest, Redirect from cgi import escape from types import ListType # DM: we would like to import this from somewhere BadRequestException= 'Bad Request' class View(App.Management.Tabs, Base): """A view of an object, typically used for management purposes This class provides bits of management framework needed by propertysheets to be used as a view on an object. """ def manage_workspace(self, URL1, RESPONSE): '''Implement a "management" interface ''' RESPONSE.redirect(URL1+'/manage') def tpURL(self): return self.getId() def manage_options(self): """Return a manage option data structure for me instance """ try: r=self.REQUEST except: r=None if r is None: pre='../../' else: pre=r['URL'] for i in (1,2,3): l=pre.rfind('/') if l >= 0: pre=pre[:l] pre=pre+'/' r=[] for d in self.aq_parent.aq_parent.manage_options: path=d['action'] option={'label': d['label'], 'action': pre+path, 'path': '../../'+path} help=d.get('help') if help is not None: option['help']=help r.append(option) return r def tabs_path_info(self, script, path): l=path.rfind('/') if l >= 0: path=path[:l] l=path.rfind('/') if l >= 0: path=path[:l] return View.inheritedAttribute('tabs_path_info')( self, script, path) def meta_type(self): try: return self.aq_parent.aq_parent.meta_type except: return '' class PropertySheet(Traversable, Persistent, Implicit): """A PropertySheet is a container for a set of related properties and metadata describing those properties. PropertySheets may or may not provide a web interface for managing its properties.""" _properties=() _extensible=1 icon='p_/Properties_icon' __ac_permissions__=( ('Manage properties', ('manage_addProperty', 'manage_editProperties', 'manage_delProperties', 'manage_changeProperties', 'manage_propertiesForm', )), ('Access contents information', ('xml_namespace', 'hasProperty', 'getProperty', 'getPropertyType', 'propertyIds', 'propertyValues','propertyItems', 'propertyInfo', 'propertyMap', ''), ('Anonymous', 'Manager'), ), ) __reserved_ids= ('values','items') def property_extensible_schema__(self): """Return a flag indicating whether new properties may be added or removed.""" return self._extensible def __init__(self, id, md=None): # Create a new property set, using the given id and namespace # string. The namespace string should be usable as an xml name- # space identifier. if id in self.__reserved_ids: raise ValueError( "'%s' is a reserved Id (forbidden Ids are: %s) " % ( id, self.__reserved_ids )) self.id=id self._md=md or {} def getId(self): return self.id def xml_namespace(self): # Return a namespace string usable as an xml namespace # for this property set. return self._md.get('xmlns', '') def v_self(self): return self def p_self(self): return self.v_self() def valid_property_id(self, id): if not id or id[:1]=='_' or (id[:3]=='aq_') \ or (' ' in id) or escape(id) != id: return 0 return 1 def hasProperty(self, id): # Return a true value if a property exists with the given id. for prop in self._propertyMap(): if id==prop['id']: return 1 return 0 def getProperty(self, id, default=None): # Return the property with the given id, returning the optional # second argument or None if no such property is found. if self.hasProperty(id): return getattr(self.v_self(), id) return default def getPropertyType(self, id): """Get the type of property 'id', returning None if no such property exists""" pself=self.p_self() for md in pself._properties: if md['id']==id: return md.get('type', 'string') return None def _wrapperCheck(self, object): # Raise an error if an object is wrapped. if hasattr(object, 'aq_base'): raise ValueError, 'Invalid property value: wrapped object' return def _setProperty(self, id, value, type='string', meta=None): # Set a new property with the given id, value and optional type. # Note that different property sets may support different typing # systems. self._wrapperCheck(value) if not self.valid_property_id(id): raise BadRequest, 'Invalid property id, %s.' % escape(id) if not self.property_extensible_schema__(): raise BadRequest, ( 'Properties cannot be added to this property sheet') pself=self.p_self() self=self.v_self() if hasattr(aq_base(self),id): if not (id=='title' and not self.__dict__.has_key(id)): raise BadRequest, ( 'Invalid property id, %s. It is in use.' % escape(id)) if meta is None: meta={} prop={'id':id, 'type':type, 'meta':meta} pself._properties=pself._properties+(prop,) if type in ('selection', 'multiple selection'): if not value: raise BadRequest, ( 'The value given for a new selection property ' 'must be a variable name

') prop['select_variable']=value if type=='selection': value=None else: value=[] # bleah - can't change kw name in api, so use ugly workaround. if sys.modules['__builtin__'].type(value) == ListType: value = tuple(value) setattr(self, id, value) def _updateProperty(self, id, value, meta=None): # Update the value of an existing property. If value is a string, # an attempt will be made to convert the value to the type of the # existing property. If a mapping containing meta-data is passed, # it will used to _replace_ the properties meta data. self._wrapperCheck(value) if not self.hasProperty(id): raise BadRequest, 'The property %s does not exist.' % escape(id) propinfo=self.propertyInfo(id) if not 'w' in propinfo.get('mode', 'wd'): raise BadRequest, '%s cannot be changed.' % escape(id) if type(value)==type(''): proptype=propinfo.get('type', 'string') if type_converters.has_key(proptype): value=type_converters[proptype](value) if meta is not None: props=[] pself=self.p_self() for prop in pself._properties: if prop['id']==id: prop['meta']=meta props.append(prop) pself._properties=tuple(props) if type(value) == ListType: value = tuple(value) setattr(self.v_self(), id, value) def _delProperty(self, id): # Delete the property with the given id. If a property with the # given id does not exist, a ValueError is raised. if not self.hasProperty(id): raise BadRequest, 'The property %s does not exist.' % escape(id) vself=self.v_self() if hasattr(vself, '_reserved_names'): nd=vself._reserved_names else: nd=() if (not 'd' in self.propertyInfo(id).get('mode', 'wd')) or (id in nd): raise BadRequest, '%s cannot be deleted.' % escape(id) delattr(vself, id) pself=self.p_self() pself._properties=tuple(filter(lambda i, n=id: i['id'] != n, pself._properties)) def propertyIds(self): # Return a list of property ids. return map(lambda i: i['id'], self._propertyMap()) def propertyValues(self): # Return a list of property values. return map(lambda i, s=self: s.getProperty(i['id']), self._propertyMap()) def propertyItems(self): # Return a list of (id, property) tuples. return map(lambda i, s=self: (i['id'], s.getProperty(i['id'])), self._propertyMap()) def propertyInfo(self, id): # Return a mapping containing property meta-data for p in self._propertyMap(): if p['id']==id: return p raise ValueError, 'The property %s does not exist.' % escape(id) def _propertyMap(self): # Return a tuple of mappings, giving meta-data for properties. # Some ZClass instances dont seem to have an _properties, so # we have to fake it... return self.p_self()._properties def propertyMap(self): # Returns a secure copy of the property definitions. return tuple(map(lambda dict: dict.copy(), self._propertyMap())) def _propdict(self): dict={} for p in self._propertyMap(): dict[p['id']]=p return dict propstat='\n' \ ' \n' \ '%s\n' \ ' \n' \ ' HTTP/1.1 %s\n%s' \ '\n' propdesc=' \n' \ ' %s\n' \ ' \n' def dav__allprop(self, propstat=propstat ): # DAV helper method - return one or more propstat elements # indicating property names and values for all properties. result=[] for item in self._propertyMap(): name, type=item['id'], item.get('type','string') value=self.getProperty(name) if type=='tokens': value=' '.join(map(str, value)) elif type=='lines': value='\n'.join(map(str, value)) # check for xml property attrs=item.get('meta', {}).get('__xml_attrs__', None) if attrs is not None: # It's a xml property. Don't escape value. attrs=map(lambda n: ' %s="%s"' % n, attrs.items()) attrs=''.join(attrs) else: # It's a non-xml property. Escape value. attrs='' if not hasattr(self,"dav__"+name): value = xml_escape(value) prop=' %s' % (name, attrs, value, name) result.append(prop) if not result: return '' result='\n'.join(result) return propstat % (self.xml_namespace(), result, '200 OK', '') def dav__propnames(self, propstat=propstat): # DAV helper method - return a propstat element indicating # property names for all properties in this PropertySheet. result=[] for name in self.propertyIds(): result.append(' ' % name) if not result: return '' result='\n'.join(result) return propstat % (self.xml_namespace(), result, '200 OK', '') def dav__propstat(self, name, result, propstat=propstat, propdesc=propdesc): # DAV helper method - return a propstat element indicating # property name and value for the requested property. xml_id=self.xml_namespace() propdict=self._propdict() if not propdict.has_key(name): prop='\n' % (name, xml_id) code='404 Not Found' if not result.has_key(code): result[code]=[prop] else: result[code].append(prop) return else: item=propdict[name] name, type=item['id'], item.get('type','string') value=self.getProperty(name) if type=='tokens': value=' '.join(map(str, value)) elif type=='lines': value='\n'.join(map(str, value)) # allow for xml properties attrs=item.get('meta', {}).get('__xml_attrs__', None) if attrs is not None: # It's a xml property. Don't escape value. attrs=map(lambda n: ' %s="%s"' % n, attrs.items()) attrs=''.join(attrs) else: # It's a non-xml property. Escape value. attrs='' if not hasattr(self, 'dav__%s' % name): value = xml_escape(value) prop='%s\n' % ( name, attrs, xml_id, value, name) code='200 OK' if not result.has_key(code): result[code]=[prop] else: result[code].append(prop) return del propstat del propdesc # Web interface manage=DTMLFile('dtml/properties', globals()) def manage_propertiesForm(self, URL1): " " raise Redirect, URL1+'/manage' def manage_addProperty(self, id, value, type, REQUEST=None): """Add a new property via the web. Sets a new property with the given id, type, and value.""" if type_converters.has_key(type): value=type_converters[type](value) self._setProperty(id, value, type) if REQUEST is not None: return self.manage(self, REQUEST) def manage_editProperties(self, REQUEST): """Edit object properties via the web.""" for prop in self._propertyMap(): name=prop['id'] if 'w' in prop.get('mode', 'wd'): value=REQUEST.get(name, '') self._updateProperty(name, value) return MessageDialog( title ='Success!', message='Your changes have been saved', action ='manage') def manage_changeProperties(self, REQUEST=None, **kw): """Change existing object properties by passing either a mapping object of name:value pairs {'foo':6} or passing name=value parameters.""" if REQUEST is None: props={} else: props=REQUEST if kw: for name, value in kw.items(): props[name]=value propdict=self._propdict() for name, value in props.items(): if self.hasProperty(name): if not 'w' in propdict[name].get('mode', 'wd'): raise BadRequest, '%s cannot be changed' % escape(name) self._updateProperty(name, value) if REQUEST is not None: return MessageDialog( title ='Success!', message='Your changes have been saved.', action ='manage') def manage_delProperties(self, ids=None, REQUEST=None): """Delete one or more properties specified by 'ids'.""" if REQUEST: # Bugfix for properties named "ids" (casey) if ids == self.getProperty('ids', None): ids = None ids = REQUEST.get('_ids', ids) if ids is None: return MessageDialog( title='No property specified', message='No properties were specified!', action ='./manage',) for id in ids: self._delProperty(id) if REQUEST is not None: return self.manage(self, REQUEST) Globals.default__class_init__(PropertySheet) class Virtual: """A virtual propertysheet stores it's properties in it's instance.""" def __init__(self): pass def v_self(self): return self.aq_parent.aq_parent class DefaultProperties(Virtual, PropertySheet, View): """The default property set mimics the behavior of old-style Zope properties -- it stores its property values in the instance of its owner.""" id='default' _md={'xmlns': 'http://www.zope.org/propsets/default'} Globals.default__class_init__(DefaultProperties) class DAVProperties(Virtual, PropertySheet, View): """WebDAV properties""" id='webdav' _md={'xmlns': 'DAV:'} pm=({'id':'creationdate', 'mode':'r'}, {'id':'displayname', 'mode':'r'}, {'id':'resourcetype', 'mode':'r'}, {'id':'getcontenttype', 'mode':'r'}, {'id':'getcontentlength', 'mode':'r'}, {'id':'source', 'mode':'r'}, {'id':'supportedlock', 'mode':'r'}, {'id':'lockdiscovery', 'mode':'r'}, ) def getProperty(self, id, default=None): method='dav__%s' % id if not hasattr(self, method): return default return getattr(self, method)() def _setProperty(self, id, value, type='string', meta=None): raise ValueError, '%s cannot be set.' % escape(id) def _updateProperty(self, id, value): raise ValueError, '%s cannot be updated.' % escape(id) def _delProperty(self, id): raise ValueError, '%s cannot be deleted.' % escape(id) def _propertyMap(self): # Only use getlastmodified if returns a value if hasattr(self.v_self(), '_p_mtime'): return self.pm + ({'id':'getlastmodified', 'mode':'r'},) return self.pm def propertyMap(self): return map(lambda dict: dict.copy(), self._propertyMap()) def dav__creationdate(self): return iso8601_date(43200.0) def dav__displayname(self): return absattr(xml_escape(self.v_self().title_or_id())) def dav__resourcetype(self): vself=self.v_self() if (isDavCollection(vself) or getattr(aq_base(vself), 'isAnObjectManager', None)): return '' return '' def dav__getlastmodified(self): return rfc1123_date(self.v_self()._p_mtime) def dav__getcontenttype(self): vself=self.v_self() if hasattr(vself, 'content_type'): return absattr(vself.content_type) if hasattr(vself, 'default_content_type'): return absattr(vself.default_content_type) return '' def dav__getcontentlength(self): vself=self.v_self() if hasattr(vself, 'get_size'): return vself.get_size() return '' def dav__source(self): vself=self.v_self() if hasattr(vself, 'document_src'): url=urlbase(vself.absolute_url()) return '\n \n' \ ' %s\n' \ ' %s/document_src\n' \ ' \n ' % (url, url) return '' def dav__supportedlock(self): vself = self.v_self() out = '\n' if WriteLockInterface.isImplementedBy(vself): out += (' \n' ' \n' ' \n' ' \n ') return out def dav__lockdiscovery(self): security = getSecurityManager() user = security.getUser().getId() vself = self.v_self() out = '\n' if WriteLockInterface.isImplementedBy(vself): locks = vself.wl_lockValues(killinvalids=1) for lock in locks: creator = lock.getCreator()[-1] if creator == user: fake=0 else: fake=1 out = '%s\n%s' % (out, lock.asLockDiscoveryProperty('n',fake=fake)) out = '%s\n' % out return out Globals.default__class_init__(DAVProperties) class PropertySheets(Traversable, Implicit, App.Management.Tabs): """A tricky container to keep property sets from polluting an object's direct attribute namespace.""" id='propertysheets' __ac_permissions__=( ('Manage properties', ('manage_addPropertySheet', 'addPropertySheet', 'delPropertySheet' )), ('Access contents information', ('items', 'values', 'get', ''), ('Anonymous', 'Manager'), ), ('View management screens', ('manage',)), ) # optionally to be overridden by derived classes PropertySheetClass= PropertySheet webdav =DAVProperties() def _get_defaults(self): return (self.webdav,) def __propsets__(self): propsets=self.aq_parent.__propsets__ __traceback_info__= propsets, type(propsets) return self._get_defaults() + propsets def __bobo_traverse__(self, REQUEST, name=None): for propset in self.__propsets__(): if propset.getId()==name: return propset.__of__(self) return getattr(self, name) def __getitem__(self, n): return self.__propsets__()[n].__of__(self) def values(self): propsets=self.__propsets__() return map(lambda n, s=self: n.__of__(s), propsets) def items(self): propsets=self.__propsets__() r=[] for n in propsets: if hasattr(n,'id'): id=n.id else: id='' r.append((id, n.__of__(self))) return r def get(self, name, default=None): for propset in self.__propsets__(): if propset.id==name or (hasattr(propset, 'xml_namespace') and \ propset.xml_namespace()==name): return propset.__of__(self) return default def manage_addPropertySheet(self, id, ns, REQUEST=None): """ """ md={'xmlns':ns} ps= self.PropertySheetClass(id, md) self.addPropertySheet(ps) if REQUEST is None: return ps ps= self.get(id) REQUEST.RESPONSE.redirect('%s/manage' % ps.absolute_url()) def addPropertySheet(self, propset): propsets=self.aq_parent.__propsets__ propsets=propsets+(propset,) self.aq_parent.__propsets__=propsets def delPropertySheet(self, name): result=[] for propset in self.aq_parent.__propsets__: if propset.getId() != name and propset.xml_namespace() != name: result.append(propset) self.aq_parent.__propsets__=tuple(result) ## DM: deletion support def isDeletable(self,name): '''currently, we say that *name* is deletable when it is not a default sheet. Later, we may further restrict deletability based on an instance attribute.''' ps= self.get(name) if ps is None: return 0 if ps in self._get_defaults(): return 0 return 1 def manage_delPropertySheets(self, ids=(), REQUEST=None): '''delete all sheets identified by *ids*.''' for id in ids: if not self.isDeletable(id): raise BadRequestException, 'attempt to delete undeletable property sheet: ' + id self.delPropertySheet(id) if REQUEST is not None: REQUEST.RESPONSE.redirect('%s/manage' % self.absolute_url()) def __len__(self): return len(self.__propsets__()) def getId(self): return self.id # Management interface: manage=Globals.DTMLFile('dtml/propertysheets', globals()) def manage_options(self): """Return a manage option data structure for me instance """ try: r=self.REQUEST except: r=None if r is None: pre='../' else: pre=r['URLPATH0'] for i in (1,2): l=pre.rfind('/') if l >= 0: pre=pre[:l] pre=pre+'/' r=[] for d in self.aq_parent.manage_options: r.append({'label': d['label'], 'action': pre+d['action']}) return r def tabs_path_info(self, script, path): l=path.rfind('/') if l >= 0: path=path[:l] return PropertySheets.inheritedAttribute('tabs_path_info')( self, script, path) Globals.default__class_init__(PropertySheets) class DefaultPropertySheets(PropertySheets): """A PropertySheets container that contains a default property sheet for compatibility with the arbitrary property mgmt design of Zope PropertyManagers.""" default=DefaultProperties() webdav =DAVProperties() def _get_defaults(self): return (self.default, self.webdav) Globals.default__class_init__(DefaultPropertySheets) class FixedSchema(PropertySheet): """A Fixed-schema property sheet has no control over it's schema It gets its schema from another proprtysheet but has control over its value storage. This mix-in is used for ZClass instance proprtysheets, which store their data in instances, but get their schema from the proprtysheet managed in the ZClass. """ def __init__(self, id, base, md=None): FixedSchema.inheritedAttribute('__init__')(self, id, md) self._base=base def _propertyMap(self): # Return a tuple of mappings, giving meta-data for properties. r = [] for d in self._base._propertyMap(): d = d.copy() mode = d.get('mode', 'wd') if 'd' in mode: d['mode']=filter(lambda c: c != 'd', mode) r.append(d) return tuple(r) def propertyMap(self): return self._propertyMap() def property_extensible_schema__(self): return 0 return self._base._extensible Globals.default__class_init__(FixedSchema) class vps(Base): """Virtual Propertysheets The vps object implements a "computed attribute" - it ensures that a PropertySheets instance is returned when the propertysheets attribute of a PropertyManager is accessed. """ def __init__(self, c=PropertySheets): self.c=c def __of__(self, parent): return self.c().__of__(parent) def absattr(attr): if callable(attr): return attr() return attr def xml_escape(value): from webdav.xmltools import escape if not isinstance(value, basestring): value = unicode(value) if not isinstance(value, unicode): # XXX It really shouldn't be hardcoded to latin-1 here. value = value.decode('latin-1') value = escape(value) return value.encode('utf-8') Zope-2.8.7-final/lib/python/OFS/Traversable.py0000664000076400007640000002255310435500327020110 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """This module implements a mix-in for traversable objects. $Id: Traversable.py 67195 2006-04-21 00:31:16Z alecm $ """ from urllib import quote from Acquisition import Acquired, aq_inner, aq_parent, aq_base from AccessControl import getSecurityManager from AccessControl import Unauthorized from AccessControl.ZopeGuards import guarded_getattr from ZODB.POSException import ConflictError from zExceptions import NotFound _marker = object() class Traversable: absolute_url__roles__=None # Public def absolute_url(self, relative=0): """Return the absolute URL of the object. This a canonical URL based on the object's physical containment path. It is affected by the virtual host configuration, if any, and can be used by external agents, such as a browser, to address the object. If the relative argument is provided, with a true value, then the value of virtual_url_path() is returned. Some Products incorrectly use '/'+absolute_url(1) as an absolute-path reference. This breaks in certain virtual hosting situations, and should be changed to use absolute_url_path() instead. """ if relative: return self.virtual_url_path() spp = self.getPhysicalPath() try: toUrl = self.REQUEST.physicalPathToURL except AttributeError: return path2url(spp[1:]) return toUrl(spp) absolute_url_path__roles__=None # Public def absolute_url_path(self): """Return the path portion of the absolute URL of the object. This includes the leading slash, and can be used as an 'absolute-path reference' as defined in RFC 2396. """ spp = self.getPhysicalPath() try: toUrl = self.REQUEST.physicalPathToURL except AttributeError: return path2url(spp) or '/' return toUrl(spp, relative=1) or '/' virtual_url_path__roles__=None # Public def virtual_url_path(self): """Return a URL for the object, relative to the site root. If a virtual host is configured, the URL is a path relative to the virtual host's root object. Otherwise, it is the physical path. In either case, the URL does not begin with a slash. """ spp = self.getPhysicalPath() try: toVirt = self.REQUEST.physicalPathToVirtualPath except AttributeError: return path2url(spp[1:]) return path2url(toVirt(spp)) getPhysicalRoot__roles__=() # Private getPhysicalRoot=Acquired getPhysicalPath__roles__=None # Public def getPhysicalPath(self): """Get the physical path of the object. Returns a path (an immutable sequence of strings) that can be used to access this object again later, for example in a copy/paste operation. getPhysicalRoot() and getPhysicalPath() are designed to operate together. """ path = (self.getId(),) p = aq_parent(aq_inner(self)) if p is not None: path = p.getPhysicalPath() + path return path unrestrictedTraverse__roles__=() # Private def unrestrictedTraverse(self, path, default=_marker, restricted=0): """Lookup an object by path. path -- The path to the object. May be a sequence of strings or a slash separated string. If the path begins with an empty path element (i.e., an empty string or a slash) then the lookup is performed from the application root. Otherwise, the lookup is relative to self. Two dots (..) as a path element indicates an upward traversal to the acquisition parent. default -- If provided, this is the value returned if the path cannot be traversed for any reason (i.e., no object exists at that path or the object is inaccessible). restricted -- If false (default) then no security checking is performed. If true, then all of the objects along the path are validated with the security machinery. Usually invoked using restrictedTraverse(). """ if not path: return self _getattr = getattr _none = None marker = _marker if isinstance(path, str): # Unicode paths are not allowed path = path.split('/') else: path = list(path) REQUEST = {'TraversalRequestNameStack': path} path.reverse() path_pop=path.pop if len(path) > 1 and not path[0]: # Remove trailing slash path.pop(0) if restricted: securityManager = getSecurityManager() else: securityManager = _none if not path[-1]: # If the path starts with an empty string, go to the root first. path_pop() self = self.getPhysicalRoot() if (restricted and not securityManager.validate(None, None, None, self)): raise Unauthorized, name try: obj = self while path: name = path_pop() __traceback_info__ = path, name if name[0] == '_': # Never allowed in a URL. raise NotFound, name if name == '..': next = aq_parent(obj) if next is not _none: if restricted and not securityManager.validate( obj, obj,name, next): raise Unauthorized, name obj = next continue bobo_traverse = _getattr(obj, '__bobo_traverse__', _none) if bobo_traverse is not _none: next = bobo_traverse(REQUEST, name) if restricted: if aq_base(next) is not next: # The object is wrapped, so the acquisition # context is the container. container = aq_parent(aq_inner(next)) elif _getattr(next, 'im_self', _none) is not _none: # Bound method, the bound instance # is the container container = next.im_self elif _getattr(aq_base(obj), name, marker) == next: # Unwrapped direct attribute of the object so # object is the container container = obj else: # Can't determine container container = _none try: validated = securityManager.validate( obj, container, name, next) except Unauthorized: # If next is a simple unwrapped property, it's # parentage is indeterminate, but it may have been # acquired safely. In this case validate will # raise an error, and we can explicitly check that # our value was acquired safely. validated = 0 if container is _none and \ guarded_getattr(obj, name, marker) is next: validated = 1 if not validated: raise Unauthorized, name else: if restricted: next = guarded_getattr(obj, name, marker) else: next = _getattr(obj, name, marker) if next is marker: try: next=obj[name] except AttributeError: # Raise NotFound for easier debugging # instead of AttributeError: __getitem__ raise NotFound, name if restricted and not securityManager.validate( obj, obj, _none, next): raise Unauthorized, name obj = next return obj except ConflictError: raise except: if default is not marker: return default else: raise restrictedTraverse__roles__=None # Public def restrictedTraverse(self, path, default=_marker): # Trusted code traversal code, always enforces security return self.unrestrictedTraverse(path, default, restricted=1) def path2url(path): return '/'.join(map(quote, path)) Zope-2.8.7-final/lib/python/OFS/CopySupport.py0000664000076400007640000005361210435500327020145 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Copy interface $Id: CopySupport.py 40550 2005-12-05 18:26:15Z yuppie $ """ import re, sys, tempfile from cgi import escape from marshal import loads, dumps from urllib import quote, unquote from warnings import warn from zlib import compress, decompress import Globals, Moniker, ExtensionClass import transaction from App.Dialogs import MessageDialog from AccessControl import getSecurityManager from AccessControl.Permissions import delete_objects as DeleteObjects from Acquisition import aq_base, aq_inner, aq_parent from zExceptions import Unauthorized, BadRequest from webdav.Lockable import ResourceLockedError CopyError='Copy Error' copy_re = re.compile('^copy([0-9]*)_of_(.*)') _marker=[] class CopyContainer(ExtensionClass.Base): """Interface for containerish objects which allow cut/copy/paste""" __ac_permissions__=( ('View management screens', ('manage_copyObjects', 'manage_pasteObjects', 'manage_renameForm', 'manage_renameObject', 'manage_renameObjects',)), ('Delete objects', ('manage_cutObjects',)), ) # The following three methods should be overridden to store sub-objects # as non-attributes. def _setOb(self, id, object): setattr(self, id, object) def _delOb(self, id): delattr(self, id) def _getOb(self, id, default=_marker): if hasattr(aq_base(self), id): return getattr(self, id) if default is _marker: raise AttributeError, id return default def manage_CopyContainerFirstItem(self, REQUEST): return self._getOb(REQUEST['ids'][0]) def manage_CopyContainerAllItems(self, REQUEST): return map(lambda i, s=self: s._getOb(i), tuple(REQUEST['ids'])) def manage_cutObjects(self, ids=None, REQUEST=None): """Put a reference to the objects named in ids in the clip board""" if ids is None and REQUEST is not None: return eNoItemsSpecified elif ids is None: raise ValueError, 'ids must be specified' if type(ids) is type(''): ids=[ids] oblist=[] for id in ids: ob=self._getOb(id) if ob.wl_isLocked(): raise ResourceLockedError, 'Object "%s" is locked via WebDAV' % ob.getId() if not ob.cb_isMoveable(): raise CopyError, eNotSupported % escape(id) m=Moniker.Moniker(ob) oblist.append(m.dump()) cp=(1, oblist) cp=_cb_encode(cp) if REQUEST is not None: resp=REQUEST['RESPONSE'] resp.setCookie('__cp', cp, path='%s' % cookie_path(REQUEST)) REQUEST['__cp'] = cp return self.manage_main(self, REQUEST) return cp def manage_copyObjects(self, ids=None, REQUEST=None, RESPONSE=None): """Put a reference to the objects named in ids in the clip board""" if ids is None and REQUEST is not None: return eNoItemsSpecified elif ids is None: raise ValueError, 'ids must be specified' if type(ids) is type(''): ids=[ids] oblist=[] for id in ids: ob=self._getOb(id) if not ob.cb_isCopyable(): raise CopyError, eNotSupported % escape(id) m=Moniker.Moniker(ob) oblist.append(m.dump()) cp=(0, oblist) cp=_cb_encode(cp) if REQUEST is not None: resp=REQUEST['RESPONSE'] resp.setCookie('__cp', cp, path='%s' % cookie_path(REQUEST)) REQUEST['__cp'] = cp return self.manage_main(self, REQUEST) return cp def _get_id(self, id): # Allow containers to override the generation of # object copy id by attempting to call its _get_id # method, if it exists. match = copy_re.match(id) if match: n = int(match.group(1) or '1') orig_id = match.group(2) else: n = 0 orig_id = id while 1: if self._getOb(id, None) is None: return id id='copy%s_of_%s' % (n and n+1 or '', orig_id) n=n+1 def manage_pasteObjects(self, cb_copy_data=None, REQUEST=None): """Paste previously copied objects into the current object. If calling manage_pasteObjects from python code, pass the result of a previous call to manage_cutObjects or manage_copyObjects as the first argument. """ cp=None if cb_copy_data is not None: cp=cb_copy_data else: if REQUEST and REQUEST.has_key('__cp'): cp=REQUEST['__cp'] if cp is None: raise CopyError, eNoData try: cp=_cb_decode(cp) except: raise CopyError, eInvalid oblist=[] op=cp[0] app = self.getPhysicalRoot() result = [] for mdata in cp[1]: m = Moniker.loadMoniker(mdata) try: ob = m.bind(app) except: raise CopyError, eNotFound self._verifyObjectPaste(ob, validate_src=op+1) oblist.append(ob) if op==0: # Copy operation for ob in oblist: if not ob.cb_isCopyable(): raise CopyError, eNotSupported % escape(ob.getId()) try: ob._notifyOfCopyTo(self, op=0) except: raise CopyError, MessageDialog( title='Copy Error', message=sys.exc_info()[1], action ='manage_main') ob=ob._getCopy(self) orig_id=ob.getId() id=self._get_id(ob.getId()) result.append({'id':orig_id, 'new_id':id}) ob._setId(id) self._setObject(id, ob) ob = self._getOb(id) ob._postCopy(self, op=0) ob.manage_afterClone(ob) ob.wl_clearLocks() if REQUEST is not None: return self.manage_main(self, REQUEST, update_menu=1, cb_dataValid=1) if op==1: # Move operation for ob in oblist: id=ob.getId() if not ob.cb_isMoveable(): raise CopyError, eNotSupported % escape(id) try: ob._notifyOfCopyTo(self, op=1) except: raise CopyError, MessageDialog( title='Move Error', message=sys.exc_info()[1], action ='manage_main') if not sanity_check(self, ob): raise CopyError, 'This object cannot be pasted into itself' # try to make ownership explicit so that it gets carried # along to the new location if needed. ob.manage_changeOwnershipType(explicit=1) aq_parent(aq_inner(ob))._delObject(id) ob = aq_base(ob) orig_id=id id=self._get_id(id) result.append({'id':orig_id, 'new_id':id }) ob._setId(id) self._setObject(id, ob, set_owner=0) ob=self._getOb(id) ob._postCopy(self, op=1) # try to make ownership implicit if possible ob.manage_changeOwnershipType(explicit=0) if REQUEST is not None: REQUEST['RESPONSE'].setCookie('__cp', 'deleted', path='%s' % cookie_path(REQUEST), expires='Wed, 31-Dec-97 23:59:59 GMT') REQUEST['__cp'] = None return self.manage_main(self, REQUEST, update_menu=1, cb_dataValid=0) return result manage_renameForm=Globals.DTMLFile('dtml/renameForm', globals()) def manage_renameObjects(self, ids=[], new_ids=[], REQUEST=None): """Rename several sub-objects""" if len(ids) != len(new_ids): raise BadRequest,'Please rename each listed object.' for i in range(len(ids)): if ids[i] != new_ids[i]: self.manage_renameObject(ids[i], new_ids[i], REQUEST) if REQUEST is not None: return self.manage_main(self, REQUEST, update_menu=1) return None def manage_renameObject(self, id, new_id, REQUEST=None): """Rename a particular sub-object""" try: self._checkId(new_id) except: raise CopyError, MessageDialog( title='Invalid Id', message=sys.exc_info()[1], action ='manage_main') ob=self._getOb(id) if ob.wl_isLocked(): raise ResourceLockedError, 'Object "%s" is locked via WebDAV' % ob.getId() if not ob.cb_isMoveable(): raise CopyError, eNotSupported % escape(id) self._verifyObjectPaste(ob) try: ob._notifyOfCopyTo(self, op=1) except: raise CopyError, MessageDialog( title='Rename Error', message=sys.exc_info()[1], action ='manage_main') self._delObject(id) ob = aq_base(ob) ob._setId(new_id) # Note - because a rename always keeps the same context, we # can just leave the ownership info unchanged. self._setObject(new_id, ob, set_owner=0) ob = self._getOb(new_id) ob._postCopy(self, op=1) if REQUEST is not None: return self.manage_main(self, REQUEST, update_menu=1) return None # Why did we give this a manage_ prefix if its really # supposed to be public since it does its own auth ? # # Because it's still a "management" function. manage_clone__roles__=None def manage_clone(self, ob, id, REQUEST=None): # Clone an object, creating a new object with the given id. if not ob.cb_isCopyable(): raise CopyError, eNotSupported % escape(ob.getId()) try: self._checkId(id) except: raise CopyError, MessageDialog( title='Invalid Id', message=sys.exc_info()[1], action ='manage_main') self._verifyObjectPaste(ob) try: ob._notifyOfCopyTo(self, op=0) except: raise CopyError, MessageDialog( title='Clone Error', message=sys.exc_info()[1], action ='manage_main') ob=ob._getCopy(self) ob._setId(id) self._setObject(id, ob) ob=self._getOb(id) ob._postCopy(self, op=0) ob.manage_afterClone(ob) return ob def cb_dataValid(self): # Return true if clipboard data seems valid. try: cp=_cb_decode(self.REQUEST['__cp']) except: return 0 return 1 def cb_dataItems(self): # List of objects in the clip board try: cp=_cb_decode(self.REQUEST['__cp']) except: return [] oblist=[] app = self.getPhysicalRoot() for mdata in cp[1]: m = Moniker.loadMoniker(mdata) oblist.append(m.bind(app)) return oblist validClipData=cb_dataValid def _verifyObjectPaste(self, object, validate_src=1): # Verify whether the current user is allowed to paste the # passed object into self. This is determined by checking # to see if the user could create a new object of the same # meta_type of the object passed in and checking that the # user actually is allowed to access the passed in object # in its existing context. # # Passing a false value for the validate_src argument will skip # checking the passed in object in its existing context. This is # mainly useful for situations where the passed in object has no # existing context, such as checking an object during an import # (the object will not yet have been connected to the acquisition # heirarchy). if not hasattr(object, 'meta_type'): raise CopyError, MessageDialog( title = 'Not Supported', message = ('The object %s does not support this' \ ' operation' % escape(absattr(object.id))), action = 'manage_main') if not hasattr(self, 'all_meta_types'): raise CopyError, MessageDialog( title = 'Not Supported', message = 'Cannot paste into this object.', action = 'manage_main') method_name = None mt_permission = None meta_types = absattr(self.all_meta_types) for d in meta_types: if d['name'] == object.meta_type: method_name = d['action'] mt_permission = d.get('permission') break if mt_permission is not None: sm = getSecurityManager() if sm.checkPermission(mt_permission, self): if validate_src: # Ensure the user is allowed to access the object on the # clipboard. try: parent = aq_parent(aq_inner(object)) except: parent = None if not sm.validate(None, parent, None, object): raise Unauthorized(absattr(object.id)) if validate_src == 2: # moving if not sm.checkPermission(DeleteObjects, parent): raise Unauthorized('Delete not allowed.') else: raise CopyError, MessageDialog( title = 'Insufficient Privileges', message = ('You do not possess the %s permission in the ' 'context of the container into which you are ' 'pasting, thus you are not able to perform ' 'this operation.' % mt_permission), action = 'manage_main') elif method_name: # BBB: fallback for missing or None permission warn("The required 'permission' key is not set or None for meta " "type '%s'. This fallback will be removed in Zope 2.9." % object.meta_type, DeprecationWarning) try: method = self.restrictedTraverse(method_name) except Unauthorized: raise CopyError, MessageDialog( title = 'Insufficient Privileges', message = ('You do not possess the permission required ' 'to call %s in the context of the container ' 'into which you are pasting, thus you are not ' 'able to perform this operation.' % method_name), action = 'manage_main') if validate_src: sm = getSecurityManager() # Ensure the user is allowed to access the object on the # clipboard. try: parent = aq_parent(aq_inner(object)) except: parent = None if not sm.validate(None,parent,None,object): raise Unauthorized, absattr(object.id) if validate_src == 2: # moving if not sm.checkPermission(DeleteObjects, parent): raise Unauthorized, 'Delete not allowed.' else: raise CopyError, MessageDialog( title = 'Not Supported', message = ('The object %s does not support this ' 'operation.' % escape(absattr(object.id))), action = 'manage_main') Globals.default__class_init__(CopyContainer) class CopySource(ExtensionClass.Base): """Interface for objects which allow themselves to be copied.""" # declare a dummy permission for Copy or Move here that we check # in cb_isCopyable. __ac_permissions__=( ('Copy or Move', (), ('Anonymous', 'Manager',)), ) def _canCopy(self, op=0): """Called to make sure this object is copyable. The op var is 0 for a copy, 1 for a move. """ return 1 def _notifyOfCopyTo(self, container, op=0): """Overide this to be pickly about where you go! If you dont want to go there, raise an exception. The op variable is 0 for a copy, 1 for a move. """ pass def _getCopy(self, container): # Commit a subtransaction to: # 1) Make sure the data about to be exported is current # 2) Ensure self._p_jar and container._p_jar are set even if # either one is a new object transaction.savepoint(optimistic=True) if self._p_jar is None: raise CopyError, ( 'Object "%s" needs to be in the database to be copied' % `self`) if container._p_jar is None: raise CopyError, ( 'Container "%s" needs to be in the database' % `container`) # Ask an object for a new copy of itself. f=tempfile.TemporaryFile() self._p_jar.exportFile(self._p_oid,f) f.seek(0) ob=container._p_jar.importFile(f) f.close() return ob def _postCopy(self, container, op=0): # Called after the copy is finished to accomodate special cases. # The op var is 0 for a copy, 1 for a move. pass def _setId(self, id): # Called to set the new id of a copied object. self.id=id def cb_isCopyable(self): # Is object copyable? Returns 0 or 1 if not (hasattr(self, '_canCopy') and self._canCopy(0)): return 0 if not self.cb_userHasCopyOrMovePermission(): return 0 return 1 def cb_isMoveable(self): # Is object moveable? Returns 0 or 1 if not (hasattr(self, '_canCopy') and self._canCopy(1)): return 0 if hasattr(self, '_p_jar') and self._p_jar is None: return 0 try: n=aq_parent(aq_inner(self))._reserved_names except: n=() if absattr(self.id) in n: return 0 if not self.cb_userHasCopyOrMovePermission(): return 0 return 1 def cb_userHasCopyOrMovePermission(self): if getSecurityManager().checkPermission('Copy or Move', self): return 1 Globals.default__class_init__(CopySource) def sanity_check(c, ob): # This is called on cut/paste operations to make sure that # an object is not cut and pasted into itself or one of its # subobjects, which is an undefined situation. ob = aq_base(ob) while 1: if aq_base(c) is ob: return 0 inner = aq_inner(c) if aq_parent(inner) is None: return 1 c = aq_parent(inner) def absattr(attr): if callable(attr): return attr() return attr def _cb_encode(d): return quote(compress(dumps(d), 9)) def _cb_decode(s): return loads(decompress(unquote(s))) def cookie_path(request): # Return a "path" value for use in a cookie that refers # to the root of the Zope object space. return request['BASEPATH1'] or "/" fMessageDialog=Globals.HTML(""" &dtml-title;

TARGET="&dtml-target;">

!

""", target='', action='manage_main', title='Changed') eNoData=MessageDialog( title='No Data', message='No clipboard data found.', action ='manage_main',) eInvalid=MessageDialog( title='Clipboard Error', message='The data in the clipboard could not be read, possibly due ' \ 'to cookie data being truncated by your web browser. Try copying ' \ 'fewer objects.', action ='manage_main',) eNotFound=MessageDialog( title='Item Not Found', message='One or more items referred to in the clipboard data was ' \ 'not found. The item may have been moved or deleted after you ' \ 'copied it.', action ='manage_main',) eNotSupported=fMessageDialog( title='Not Supported', message=( 'The action against the %s object could not be carried ' 'out. ' 'One of the following constraints caused the problem:

' 'The object does not support this operation.' '

-- OR --

' 'The currently logged-in user does not have the Copy or ' 'Move permission respective to the object.' ), action ='manage_main',) eNoItemsSpecified=MessageDialog( title='No items specified', message='You must select one or more items to perform ' \ 'this operation.', action ='manage_main' ) Zope-2.8.7-final/lib/python/AccessControl/0000775000076400007640000000000010435610645017375 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/SecurityManagement.py0000664000076400007640000000465310435500321023550 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Security management $Id: SecurityManagement.py 24764 2004-05-17 06:13:48Z philikon $ """ def getSecurityManager(): """Get a security manager, for the current thread. """ thread_id=get_ident() manager=_managers.get(thread_id, None) if manager is None: nobody = getattr(SpecialUsers, 'nobody', None) if nobody is None: # Initialize SpecialUsers by importing User.py. import User nobody = SpecialUsers.nobody manager = SecurityManager(thread_id, SecurityContext(nobody)) _managers[thread_id]=manager return manager def setSecurityManager(manager): """install *manager* as current security manager for this thread.""" thread_id=get_ident() _managers[thread_id]=manager import SpecialUsers # AccessControl.Implementation inserts SecurityManager. try: from thread import get_ident except ImportError: def get_ident(): return 0 _managers={} def newSecurityManager(request, user): """Set up a new security context for a request for a user """ thread_id=get_ident() _managers[thread_id]=SecurityManager( thread_id, SecurityContext(user), ) def noSecurityManager(): try: del _managers[get_ident()] except: pass def setSecurityPolicy(aSecurityPolicy): """Set the system default security policy. This method should only be caused by system startup code. It should never, for example, be called during a web request. """ SecurityManager.setSecurityPolicy(aSecurityPolicy) class SecurityContext: """The security context is an object used internally to the security machinery. It captures data about the current security context. """ def __init__(self, user): self.stack=[] self.user=user self.objectCache = {} Zope-2.8.7-final/lib/python/AccessControl/SecurityManager.py0000664000076400007640000000232610435500321023041 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''API module to set the security policy $Id: SecurityManager.py 24763 2004-05-17 05:59:28Z philikon $''' from AccessControl import ImplPython as _ImplPython from AccessControl.SimpleObjectPolicies import _noroles def setSecurityPolicy(aSecurityPolicy): """Set the system default security policy. This method should only be caused by system startup code. It should never, for example, be called during a web request. """ last = _ImplPython._defaultPolicy _ImplPython._defaultPolicy = aSecurityPolicy return last # AccessControl.Implementation inserts SecurityManager. Zope-2.8.7-final/lib/python/AccessControl/PermissionMapping.py0000664000076400007640000001170010435500321023377 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Permission Mapping Sometimes, we need an object's permissions to be remapped to other permissions when the object is used in specual ways. This is rather hard, since we need the object's ordinary permissions intact so we can manage it. $Id: PermissionMapping.py 30356 2005-05-15 15:56:18Z yuppie $ """ from cgi import escape import ExtensionClass, Acquisition from Globals import InitializeClass from Owned import UnownableOwner from Permission import pname class RoleManager: def manage_getPermissionMapping(self): """Return the permission mapping for the object This is a list of dictionaries with: permission_name -- The name of the native object permission class_permission -- The class permission the permission is mapped to. """ wrapper=getattr(self, '_permissionMapper', None) if wrapper is None: wrapper=PM() perms={} for p in self.possible_permissions(): perms[pname(p)]=p r=[] a=r.append for ac_perms in self.ac_inherited_permissions(1): p=perms.get(getPermissionMapping(ac_perms[0], wrapper), '') a({'permission_name': ac_perms[0], 'class_permission': p}) return r def manage_setPermissionMapping(self, permission_names=[], class_permissions=[], REQUEST=None): """Change the permission mapping """ wrapper=getattr(self, '_permissionMapper', None) if wrapper is None: wrapper=PM() perms=self.possible_permissions() for i in range(len(permission_names)): name=permission_names[i] p=class_permissions[i] if p and (p not in perms): __traceback_info__=perms, p, i raise ValueError, ( """Attempted to map a permission to a permission, %s, that is not valid. This should never happen. (Waaa). """ % escape(p)) setPermissionMapping(name, wrapper, p) self._permissionMapper=wrapper if REQUEST is not None: return self.manage_access( REQUEST, manage_tabs_message='The permission mapping has been updated') def _isBeingUsedAsAMethod(self, REQUEST =None, wannaBe=0): try: if hasattr(self, 'aq_self'): r=self.aq_acquire('_isBeingUsedAsAMethod_') else: r=self._isBeingUsedAsAMethod_ except: r=0 if REQUEST is not None: if not r != (not wannaBe): REQUEST.response.notFoundError() return r def _isBeingAccessedAsZClassDefinedInstanceMethod(self): p=getattr(self,'aq_parent',None) if p is None: return 0 # Not wrapped base=getattr(p, 'aq_base', None) return type(base) is PermissionMapper InitializeClass(RoleManager) def getPermissionMapping(name, obj, st=type('')): obj=getattr(obj, 'aq_base', obj) name=pname(name) r=getattr(obj, name, '') if type(r) is not st: r='' return r def setPermissionMapping(name, obj, v): name=pname(name) if v: setattr(obj, name, pname(v)) elif obj.__dict__.has_key(name): delattr(obj, name) class PM(ExtensionClass.Base): _owner=UnownableOwner _View_Permission='_View_Permission' _is_wrapperish = 1 def __getattr__(self, name): # We want to make sure that any non-explicitly set methods are # private! if name.startswith('_') and name.endswith("_Permission"): return '' raise AttributeError, escape(name) PermissionMapper=PM def aqwrap(object, wrapper, parent): r=Rewrapper() r._ugh=wrapper, object, parent return r class Rewrapper(ExtensionClass.Base): def __of__(self, parent): w, m, p = self._ugh return m.__of__( Acquisition.ImplicitAcquisitionWrapper( w, parent)) def __getattr__(self, name): w, m, parent = self._ugh self=m.__of__( Acquisition.ImplicitAcquisitionWrapper( w, parent)) return getattr(self, name) def __call__(self, *args, **kw): w, m, parent = self._ugh self=m.__of__( Acquisition.ImplicitAcquisitionWrapper( w, parent)) return apply(self, args, kw) Zope-2.8.7-final/lib/python/AccessControl/SpecialUsers.py0000664000076400007640000000174110435500321022341 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Place to find special users This is needed to avoid a circular import problem. The 'real' values are stored here by the AccessControl.User module as part of it's initialization. $Id: SpecialUsers.py 24764 2004-05-17 06:13:48Z philikon $ """ nobody = None system = None emergency_user = None # Note: use of the 'super' name is deprecated. super = None Zope-2.8.7-final/lib/python/AccessControl/PermissionRole.py0000664000076400007640000000161710435500321022713 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''Objects that implement Permission-based roles. $Id: PermissionRole.py 24763 2004-05-17 05:59:28Z philikon $''' # The following names are inserted by AccessControl.Implementation: # # rolesForPermissionOn, PermissionRole, imPermissionRole, # _what_not_even_god_should_do Zope-2.8.7-final/lib/python/AccessControl/Role.py0000664000076400007640000004741710435500321020652 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Access control support $Id: Role.py 30356 2005-05-15 15:56:18Z yuppie $ """ from cgi import escape from Globals import DTMLFile, MessageDialog, Dictionary from Acquisition import Implicit, Acquired, aq_get import Globals, ExtensionClass, PermissionMapping, Products from App.Common import aq_base from Permission import Permission DEFAULTMAXLISTUSERS=250 def _isBeingUsedAsAMethod(self): return aq_get(self, '_isBeingUsedAsAMethod_', 0) def _isNotBeingUsedAsAMethod(self): return not aq_get(self, '_isBeingUsedAsAMethod_', 0) class RoleManager(ExtensionClass.Base, PermissionMapping.RoleManager): """An object that has configurable permissions""" __ac_permissions__=( ('Change permissions', ('manage_access', 'permission_settings', 'ac_inherited_permissions', 'manage_roleForm', 'manage_role', 'manage_acquiredForm', 'manage_acquiredPermissions', 'manage_permissionForm', 'manage_permission', 'manage_changePermissions', 'permissionsOfRole', 'rolesOfPermission', 'acquiredRolesAreUsedBy', 'manage_defined_roles', 'userdefined_roles', 'manage_listLocalRoles', 'manage_editLocalRoles', 'manage_setLocalRoles', 'manage_addLocalRoles', 'manage_delLocalRoles' )), ) manage_options=( {'label':'Security', 'action':'manage_access', 'help':('OFSP','Security.stx'), 'filter': _isNotBeingUsedAsAMethod, }, {'label':'Define Permissions', 'action':'manage_access', 'help':('OFSP','Security_Define-Permissions.stx'), 'filter': _isBeingUsedAsAMethod, }, ) __ac_roles__=('Manager', 'Owner', 'Anonymous', 'Authenticated') permissionMappingPossibleValues=Acquired #------------------------------------------------------------ def ac_inherited_permissions(self, all=0): # Get all permissions not defined in ourself that are inherited # This will be a sequence of tuples with a name as the first item and # an empty tuple as the second. d={} perms=self.__ac_permissions__ for p in perms: d[p[0]]=None r=gather_permissions(self.__class__, [], d) if all: if hasattr(self, '_subobject_permissions'): for p in self._subobject_permissions(): pname=p[0] if not d.has_key(pname): d[pname]=1 r.append(p) r=list(perms)+r r.sort() return tuple(r) def permission_settings(self, permission=None): """Return user-role permission settings. If 'permission' is passed to the method then only the settings for 'permission' is returned. """ result=[] valid=self.valid_roles() indexes=range(len(valid)) ip=0 permissions = self.ac_inherited_permissions(1) # Filter permissions if permission: permissions = [p for p in permissions if p[0] == permission] for p in permissions: name, value = p[:2] p=Permission(name,value,self) roles=p.getRoles(default=[]) d={'name': name, 'acquire': isinstance(roles, list) and 'CHECKED' or '', 'roles': map( lambda ir, roles=roles, valid=valid, ip=ip: { 'name': "p%dr%d" % (ip,ir), 'checked': (valid[ir] in roles) and 'CHECKED' or '', }, indexes) } ip=ip+1 result.append(d) return result manage_roleForm=DTMLFile('dtml/roleEdit', globals(), management_view='Security', help_topic='Security_Manage-Role.stx', help_product='OFSP') def manage_role(self, role_to_manage, permissions=[], REQUEST=None): """Change the permissions given to the given role. """ self._isBeingUsedAsAMethod(REQUEST, 0) for p in self.ac_inherited_permissions(1): name, value = p[:2] p=Permission(name,value,self) p.setRole(role_to_manage, name in permissions) if REQUEST is not None: return self.manage_access(REQUEST) manage_acquiredForm=DTMLFile('dtml/acquiredEdit', globals(), management_view='Security', help_topic='Security_Manage-Acquisition.stx', help_product='OFSP') def manage_acquiredPermissions(self, permissions=[], REQUEST=None): """Change the permissions that acquire. """ self._isBeingUsedAsAMethod(REQUEST, 0) for p in self.ac_inherited_permissions(1): name, value = p[:2] p=Permission(name,value,self) roles=p.getRoles() if roles is None: continue if name in permissions: p.setRoles(list(roles)) else: p.setRoles(tuple(roles)) if REQUEST is not None: return self.manage_access(REQUEST) manage_permissionForm=DTMLFile('dtml/permissionEdit', globals(), management_view='Security', help_topic='Security_Manage-Permission.stx', help_product='OFSP') def manage_permission(self, permission_to_manage, roles=[], acquire=0, REQUEST=None): """Change the settings for the given permission. If optional arg acquire is true, then the roles for the permission are acquired, in addition to the ones specified, otherwise the permissions are restricted to only the designated roles. """ self._isBeingUsedAsAMethod(REQUEST, 0) for p in self.ac_inherited_permissions(1): name, value = p[:2] if name==permission_to_manage: p=Permission(name,value,self) if acquire: roles=list(roles) else: roles=tuple(roles) p.setRoles(roles) if REQUEST is not None: return self.manage_access(REQUEST) return raise ValueError, ( "The permission %s is invalid." % escape(permission_to_manage)) _normal_manage_access=DTMLFile('dtml/access', globals()) _method_manage_access=DTMLFile('dtml/methodAccess', globals()) def manage_access(self, REQUEST, **kw): """Return an interface for making permissions settings. """ if hasattr(self, '_isBeingUsedAsAMethod') and \ self._isBeingUsedAsAMethod(): return apply(self._method_manage_access,(), kw) else: return apply(self._normal_manage_access,(), kw) def manage_changePermissions(self, REQUEST): """Change all permissions settings, called by management screen. """ self._isBeingUsedAsAMethod(REQUEST, 0) valid_roles=self.valid_roles() indexes=range(len(valid_roles)) have=REQUEST.has_key permissions=self.ac_inherited_permissions(1) fails = [] for ip in range(len(permissions)): roles=[] for ir in indexes: if have("p%dr%d" % (ip,ir)): roles.append(valid_roles[ir]) name, value = permissions[ip][:2] try: p=Permission(name,value,self) if not have('a%d' % ip): roles=tuple(roles) p.setRoles(roles) except: fails.append(name) if fails: return MessageDialog(title="Warning!", message="Some permissions had errors: " + escape(', '.join(fails)), action='manage_access') return MessageDialog( title ='Success!', message='Your changes have been saved', action ='manage_access') def permissionsOfRole(self, role): """Used by management screen. """ r=[] for p in self.ac_inherited_permissions(1): name, value = p[:2] p=Permission(name,value,self) roles=p.getRoles() r.append({'name': name, 'selected': role in roles and 'SELECTED' or '', }) return r def rolesOfPermission(self, permission): """Used by management screen. """ valid_roles=self.valid_roles() for p in self.ac_inherited_permissions(1): name, value = p[:2] if name==permission: p=Permission(name,value,self) roles=p.getRoles() return map( lambda role, roles=roles: {'name': role, 'selected': role in roles and 'SELECTED' or '', }, valid_roles) raise ValueError, ( "The permission %s is invalid." % escape(permission)) def acquiredRolesAreUsedBy(self, permission): """Used by management screen. """ for p in self.ac_inherited_permissions(1): name, value = p[:2] if name==permission: p=Permission(name,value,self) roles=p.getRoles() return isinstance(roles, list) and 'CHECKED' or '' raise ValueError, ( "The permission %s is invalid." % escape(permission)) # Local roles support # ------------------- # # Local roles allow a user to be given extra roles in the context # of a particular object (and its children). When a user is given # extra roles in a particular object, an entry for that user is made # in the __ac_local_roles__ dict containing the extra roles. __ac_local_roles__=None manage_listLocalRoles=DTMLFile('dtml/listLocalRoles', globals(), management_view='Security', help_topic='Security_Local-Roles.stx', help_product='OFSP') manage_editLocalRoles=DTMLFile('dtml/editLocalRoles', globals(), management_view='Security', help_topic='Security_User-Local-Roles.stx', help_product='OFSP') def has_local_roles(self): dict=self.__ac_local_roles__ or {} return len(dict) def get_local_roles(self): dict=self.__ac_local_roles__ or {} keys=dict.keys() keys.sort() info=[] for key in keys: value=tuple(dict[key]) info.append((key, value)) return tuple(info) def users_with_local_role(self, role): got = {} for user, roles in self.get_local_roles(): if role in roles: got[user] = 1 return got.keys() def get_valid_userids(self): item=self dict={} _notfound = [] while 1: aclu = getattr(aq_base(item), '__allow_groups__', _notfound) if aclu is not _notfound: mlu = getattr(aclu, 'maxlistusers', _notfound) if not isinstance(mlu, int): mlu = DEFAULTMAXLISTUSERS if mlu < 0: raise OverflowError un = getattr(aclu, 'user_names', _notfound) if un is not _notfound: unl = un() # maxlistusers of 0 is list all if len(unl) > mlu and mlu != 0: raise OverflowError for name in un(): dict[name]=1 item = getattr(item, 'aq_parent', _notfound) if item is _notfound: break keys=dict.keys() keys.sort() return tuple(keys) def get_local_roles_for_userid(self, userid): dict=self.__ac_local_roles__ or {} return tuple(dict.get(userid, [])) def manage_addLocalRoles(self, userid, roles, REQUEST=None): """Set local roles for a user.""" if not roles: raise ValueError, 'One or more roles must be given!' dict=self.__ac_local_roles__ if dict is None: self.__ac_local_roles__ = dict = {} local_roles = list(dict.get(userid, [])) for r in roles: if r not in local_roles: local_roles.append(r) dict[userid] = local_roles self._p_changed=True if REQUEST is not None: stat='Your changes have been saved.' return self.manage_listLocalRoles(self, REQUEST, stat=stat) def manage_setLocalRoles(self, userid, roles, REQUEST=None): """Set local roles for a user.""" if not roles: raise ValueError, 'One or more roles must be given!' dict=self.__ac_local_roles__ if dict is None: self.__ac_local_roles__ = dict = {} dict[userid]=roles self._p_changed=True if REQUEST is not None: stat='Your changes have been saved.' return self.manage_listLocalRoles(self, REQUEST, stat=stat) def manage_delLocalRoles(self, userids, REQUEST=None): """Remove all local roles for a user.""" dict=self.__ac_local_roles__ if dict is None: self.__ac_local_roles__ = dict = {} for userid in userids: if dict.has_key(userid): del dict[userid] self._p_changed=True if REQUEST is not None: stat='Your changes have been saved.' return self.manage_listLocalRoles(self, REQUEST, stat=stat) #------------------------------------------------------------ access_debug_info__roles__=() def access_debug_info(self): """Return debug info. """ clas=class_attrs(self) inst=instance_attrs(self) data=[] _add=data.append for key, value in inst.items(): if key.find('__roles__') >= 0: _add({'name': key, 'value': value, 'class': 0}) if hasattr(value, '__roles__'): _add({'name': '%s.__roles__' % key, 'value': value.__roles__, 'class': 0}) for key, value in clas.items(): if key.find('__roles__') >= 0: _add({'name': key, 'value': value, 'class' : 1}) if hasattr(value, '__roles__'): _add({'name': '%s.__roles__' % key, 'value': value.__roles__, 'class': 1}) return data def valid_roles(self): """Return list of valid roles. """ obj=self dict={} dup =dict.has_key x=0 while x < 100: if hasattr(obj, '__ac_roles__'): roles=obj.__ac_roles__ for role in roles: if not dup(role): dict[role]=1 if not hasattr(obj, 'aq_parent'): break obj=obj.aq_parent x=x+1 roles=dict.keys() roles.sort() return tuple(roles) def validate_roles(self, roles): """Return true if all given roles are valid. """ valid=self.valid_roles() for role in roles: if role not in valid: return 0 return 1 def userdefined_roles(self): """Return list of user-defined roles. """ roles=list(self.__ac_roles__) for role in classattr(self.__class__,'__ac_roles__'): try: roles.remove(role) except: pass return tuple(roles) def manage_defined_roles(self, submit=None, REQUEST=None): """Called by management screen. """ if submit=='Add Role': role=reqattr(REQUEST, 'role') return self._addRole(role, REQUEST) if submit=='Delete Role': roles=reqattr(REQUEST, 'roles') return self._delRoles(roles, REQUEST) return self.manage_access(REQUEST) def _addRole(self, role, REQUEST=None): if not role: return MessageDialog( title ='Incomplete', message='You must specify a role name', action ='manage_access') if role in self.__ac_roles__: return MessageDialog( title ='Role Exists', message='The given role is already defined', action ='manage_access') data=list(self.__ac_roles__) data.append(role) self.__ac_roles__=tuple(data) if REQUEST is not None: return self.manage_access(REQUEST) def _delRoles(self, roles, REQUEST=None): if not roles: return MessageDialog( title ='Incomplete', message='You must specify a role name', action ='manage_access') data=list(self.__ac_roles__) for role in roles: try: data.remove(role) except: pass self.__ac_roles__=tuple(data) if REQUEST is not None: return self.manage_access(REQUEST) def _has_user_defined_role(self, role): return role in self.__ac_roles__ # Compatibility names only!! smallRolesWidget=selectedRoles=aclAChecked=aclPChecked=aclEChecked='' validRoles=valid_roles #manage_rolesForm=manage_access def manage_editRoles(self, REQUEST, acl_type='A', acl_roles=[]): pass def _setRoles(self, acl_type, acl_roles): pass def possible_permissions(self): d={} for p in Products.__ac_permissions__: d[p[0]]=1 for p in self.aq_acquire('_getProductRegistryData')('ac_permissions'): d[p[0]]=1 for p in self.ac_inherited_permissions(1): d[p[0]]=1 d=d.keys() d.sort() return d Globals.default__class_init__(RoleManager) def reqattr(request, attr): try: return request[attr] except: return None def classattr(cls, attr): if hasattr(cls, attr): return getattr(cls, attr) try: bases=cls.__bases__ except: bases=() for base in bases: if classattr(base, attr): return attr return None def instance_dict(inst): try: return inst.__dict__ except: return {} def class_dict(_class): try: return _class.__dict__ except: return {} def instance_attrs(inst): return instance_dict(inst) def class_attrs(inst, _class=None, data=None): if _class is None: _class=inst.__class__ data={} clas_dict=class_dict(_class) inst_dict=instance_dict(inst) inst_attr=inst_dict.has_key for key, value in clas_dict.items(): if not inst_attr(key): data[key]=value for base in _class.__bases__: data=class_attrs(inst, base, data) return data def gather_permissions(klass, result, seen): for base in klass.__bases__: if base.__dict__.has_key('__ac_permissions__'): for p in base.__ac_permissions__: name=p[0] if seen.has_key(name): continue result.append((name, ())) seen[name]=None gather_permissions(base, result, seen) return result Zope-2.8.7-final/lib/python/AccessControl/__init__.py0000664000076400007640000000241110435500321021471 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from unauthorized import Unauthorized # This has to happen early so things get initialized properly from Implementation import setImplementation from SecurityManagement import getSecurityManager, setSecurityPolicy from SecurityInfo import ClassSecurityInfo, ModuleSecurityInfo from SecurityInfo import ACCESS_PRIVATE from SecurityInfo import ACCESS_PUBLIC from SecurityInfo import ACCESS_NONE from SecurityInfo import secureModule, allow_module, allow_class from SimpleObjectPolicies import allow_type from ZopeGuards import full_write_guard, safe_builtins ModuleSecurityInfo('AccessControl').declarePublic('getSecurityManager') import DTML del DTML Zope-2.8.7-final/lib/python/AccessControl/DTML.py0000664000076400007640000000670610435500321020505 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Add security system support to Document Templates $Id: DTML.py 40318 2005-11-22 15:46:15Z tseaver $ """ from DocumentTemplate import DT_Util import SecurityManagement, string, math, random import DocumentTemplate.sequence from ZopeGuards import safe_builtins whrandom = random # BBB: will be removed in Zope 2.10 # RestrictedDTML is inserted by AccessControl.Implementation. # Allow access to unprotected attributes DT_Util.TemplateDict.__allow_access_to_unprotected_subobjects__=1 string.__allow_access_to_unprotected_subobjects__=1 math.__allow_access_to_unprotected_subobjects__=1 random.__allow_access_to_unprotected_subobjects__=1 DocumentTemplate.sequence.__allow_access_to_unprotected_subobjects__=1 # Add security testing capabilities class DTMLSecurityAPI: """API for performing security checks in DTML using '_' methods. """ def SecurityValidate(md, inst, parent, name, value): """Validate access. Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. The arguments may be provided as keyword arguments. Some of these arguments may be ommitted, however, the policy may reject access in some cases when arguments are ommitted. It is best to provide all the values possible. """ return (SecurityManagement .getSecurityManager() .validate(inst, parent, name, value) ) def SecurityCheckPermission(md, permission, object): """Check whether the security context allows the given permission on the given object. Arguments: permission -- A permission name object -- The object being accessed according to the permission """ return (SecurityManagement .getSecurityManager() .checkPermission(permission, object) ) def SecurityGetUser(md): """Gen the current authenticated user""" return (SecurityManagement .getSecurityManager() .getUser() ) def SecurityCalledByExecutable(md): """Return a boolean value indicating if this context was called by an executable""" r = (SecurityManagement .getSecurityManager() .calledByExecutable() ) if r > 0: return r-1 return r for name, v in DTMLSecurityAPI.__dict__.items(): if name[0] != '_': setattr(DT_Util.TemplateDict, name, v) from types import FunctionType for name, v in safe_builtins.items(): if type(v) is FunctionType: v = DT_Util.NotBindable(v) if name.startswith('__'): continue setattr(DT_Util.TemplateDict, name, v) Zope-2.8.7-final/lib/python/AccessControl/cAccessControl.c0000664000076400007640000016725210435500321022450 0ustar ajungajung/* ** cAccessControl.c ** ** Access control acceleration routines Copyright (c) 2001, Digital Creations, Fredericksburg, VA, USA. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: o Redistributions of source code must retain the above copyright notice, this list of conditions, and the disclaimer that follows. o Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. o Neither the name of Digital Creations nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS AND CONTRIBUTORS *AS IS* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. $Id: cAccessControl.c 40459 2005-12-01 22:44:07Z tseaver $ If you have questions regarding this software, contact: Digital Creations L.C. info@digicool.com (540) 371-6909 */ #include "ExtensionClass.h" #include "Acquisition.h" #include #include static void PyVar_Assign(PyObject **v, PyObject *e) { Py_XDECREF(*v); *v=e; } #define ASSIGN(V,E) PyVar_Assign(&(V),(E)) #define UNLESS(E) if (!(E)) #define OBJECT(o) ((PyObject *) (o)) static PyObject * callfunction1(PyObject *function, PyObject *arg) { PyObject *t, *r; t = PyTuple_New(1); if (t == NULL) return NULL; Py_INCREF(arg); PyTuple_SET_ITEM(t, 0, arg); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static PyObject * callmethod1(PyObject *self, PyObject *name, PyObject *arg) { UNLESS(self = PyObject_GetAttr(self,name)) return NULL; ASSIGN(self, callfunction1(self, arg)); return self; } static PyObject * callfunction2(PyObject *function, PyObject *arg0, PyObject *arg1) { PyObject *t, *r; t = PyTuple_New(2); if (t == NULL) return NULL; Py_INCREF(arg0); Py_INCREF(arg1); PyTuple_SET_ITEM(t, 0, arg0); PyTuple_SET_ITEM(t, 1, arg1); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static PyObject * callfunction3(PyObject *function, PyObject *arg0, PyObject *arg1, PyObject *arg2 ) { PyObject *t, *r; t = PyTuple_New(3); if (t == NULL) return NULL; Py_INCREF(arg0); Py_INCREF(arg1); Py_INCREF(arg2); PyTuple_SET_ITEM(t, 0, arg0); PyTuple_SET_ITEM(t, 1, arg1); PyTuple_SET_ITEM(t, 2, arg2); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static PyObject * callfunction4(PyObject *function, PyObject *arg0, PyObject *arg1, PyObject *arg2, PyObject *arg3 ) { PyObject *t, *r; t = PyTuple_New(4); if (t == NULL) return NULL; Py_INCREF(arg0); Py_INCREF(arg1); Py_INCREF(arg2); Py_INCREF(arg3); PyTuple_SET_ITEM(t, 0, arg0); PyTuple_SET_ITEM(t, 1, arg1); PyTuple_SET_ITEM(t, 2, arg2); PyTuple_SET_ITEM(t, 3, arg3); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static PyObject * callfunction5(PyObject *function, PyObject *arg0, PyObject *arg1, PyObject *arg2, PyObject *arg3, PyObject *arg4 ) { PyObject *t, *r; t = PyTuple_New(5); if (t == NULL) return NULL; Py_INCREF(arg0); Py_INCREF(arg1); Py_INCREF(arg2); Py_INCREF(arg3); Py_INCREF(arg4); PyTuple_SET_ITEM(t, 0, arg0); PyTuple_SET_ITEM(t, 1, arg1); PyTuple_SET_ITEM(t, 2, arg2); PyTuple_SET_ITEM(t, 3, arg3); PyTuple_SET_ITEM(t, 4, arg4); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static PyObject * callfunction6(PyObject *function, PyObject *arg0, PyObject *arg1, PyObject *arg2, PyObject *arg3, PyObject *arg4, PyObject *arg5 ) { PyObject *t, *r; t = PyTuple_New(6); if (t == NULL) return NULL; Py_INCREF(arg0); Py_INCREF(arg1); Py_INCREF(arg2); Py_INCREF(arg3); Py_INCREF(arg4); Py_INCREF(arg5); PyTuple_SET_ITEM(t, 0, arg0); PyTuple_SET_ITEM(t, 1, arg1); PyTuple_SET_ITEM(t, 2, arg2); PyTuple_SET_ITEM(t, 3, arg3); PyTuple_SET_ITEM(t, 4, arg4); PyTuple_SET_ITEM(t, 5, arg5); r = PyObject_CallObject(function, t); Py_DECREF(t); return r; } static int unpacktuple2(PyObject *args, char *name, int min, PyObject **a0, PyObject **a1) { int l; l=PyTuple_Size(args); if (l < 0) return -1; if (l < min) { PyErr_Format(PyExc_TypeError, "expected %d arguments, got %d", min, l); return -1; } if (l > 0) *a0=PyTuple_GET_ITEM(args, 0); if (l > 1) *a1=PyTuple_GET_ITEM(args, 1); return 0; } static int unpacktuple3(PyObject *args, char *name, int min, PyObject **a0, PyObject **a1, PyObject **a2) { int l; l=PyTuple_Size(args); if (l < 0) return -1; if (l < min) { PyErr_Format(PyExc_TypeError, "expected %d arguments, got %d", min, l); return -1; } if (l > 0) *a0=PyTuple_GET_ITEM(args, 0); if (l > 1) *a1=PyTuple_GET_ITEM(args, 1); if (l > 2) *a2=PyTuple_GET_ITEM(args, 2); return 0; } static int unpacktuple4(PyObject *args, char *name, int min, PyObject **a0, PyObject **a1, PyObject **a2, PyObject **a3) { int l; l=PyTuple_Size(args); if (l < 0) return -1; if (l < min) { PyErr_Format(PyExc_TypeError, "expected %d arguments, got %d", min, l); return -1; } if (l > 0) *a0=PyTuple_GET_ITEM(args, 0); if (l > 1) *a1=PyTuple_GET_ITEM(args, 1); if (l > 2) *a2=PyTuple_GET_ITEM(args, 2); if (l > 3) *a3=PyTuple_GET_ITEM(args, 3); return 0; } static int unpacktuple5(PyObject *args, char *name, int min, PyObject **a0, PyObject **a1, PyObject **a2, PyObject **a3, PyObject **a4) { int l; l=PyTuple_Size(args); if (l < 0) return -1; if (l < min) { PyErr_Format(PyExc_TypeError, "expected %d arguments, got %d", min, l); return -1; } if (l > 0) *a0=PyTuple_GET_ITEM(args, 0); if (l > 1) *a1=PyTuple_GET_ITEM(args, 1); if (l > 2) *a2=PyTuple_GET_ITEM(args, 2); if (l > 3) *a3=PyTuple_GET_ITEM(args, 3); if (l > 4) *a4=PyTuple_GET_ITEM(args, 4); return 0; } static int unpacktuple6(PyObject *args, char *name, int min, PyObject **a0, PyObject **a1, PyObject **a2, PyObject **a3, PyObject **a4, PyObject **a5) { int l; l=PyTuple_Size(args); if (l < 0) return -1; if (l < min) { PyErr_Format(PyExc_TypeError, "expected %d arguments, got %d", min, l); return -1; } if (l > 0) *a0=PyTuple_GET_ITEM(args, 0); if (l > 1) *a1=PyTuple_GET_ITEM(args, 1); if (l > 2) *a2=PyTuple_GET_ITEM(args, 2); if (l > 3) *a3=PyTuple_GET_ITEM(args, 3); if (l > 4) *a4=PyTuple_GET_ITEM(args, 4); if (l > 5) *a5=PyTuple_GET_ITEM(args, 5); return 0; } /* ** Structures */ typedef struct { PyObject_HEAD } ZopeSecurityPolicy; typedef struct { PyObject_HEAD PyObject *thread_id; PyObject *context; PyObject *policy; PyObject *validate; PyObject *checkPermission; } SecurityManager; typedef struct { PyObject_HEAD PyObject *__name__; PyObject *_p; PyObject *__roles__; } PermissionRole; typedef struct { PyObject_HEAD PyObject *_p; PyObject *_pa; PyObject *__roles__; PyObject *_v; } imPermissionRole; /* ** Prototypes */ static PyObject *ZopeSecurityPolicy_validate(PyObject *self, PyObject *args); static void ZopeSecurityPolicy_dealloc(ZopeSecurityPolicy *self); static PyObject *PermissionRole_init(PermissionRole *self, PyObject *args); static PyObject *PermissionRole_of(PermissionRole *self, PyObject *parent); static PyObject * PermissionRole_rolesForPermissionOn(PermissionRole *self, PyObject *value); static void PermissionRole_dealloc(PermissionRole *self); static PyObject *PermissionRole_getattro(PermissionRole *self, PyObject *name); static PyObject *imPermissionRole_of(imPermissionRole *self, PyObject *parent); static int imPermissionRole_length(imPermissionRole *self); static PyObject *imPermissionRole_get(imPermissionRole *self, int item); static void imPermissionRole_dealloc(imPermissionRole *self); static PyObject *rolesForPermissionOn(PyObject *self, PyObject *args); static PyObject *module_guarded_getattr(PyObject *self, PyObject *args); static PyObject *module_aq_validate(PyObject *self, PyObject *args); static PyObject *module_setDefaultBehaviors(PyObject *self, PyObject *args); static PyObject *c_rolesForPermissionOn(PyObject *self, PyObject *perm, PyObject *object, PyObject *deflt); static PyObject *permissionName(PyObject *name); static PyObject *SecurityManager_validate(SecurityManager *self, PyObject *args); static PyObject *SecurityManager_validateValue(SecurityManager *self, PyObject *args); static PyObject *SecurityManager_DTMLValidate(SecurityManager *self, PyObject *args); static PyObject *SecurityManager_checkPermission(SecurityManager *self, PyObject *args); static void SecurityManager_dealloc(SecurityManager *self); static PyObject *SecurityManager_getattro(SecurityManager *self, PyObject *name); static int SecurityManager_setattro(SecurityManager *self, PyObject *name, PyObject *value); static getattrofunc ExtensionClassGetattro; /* ** Constants */ static PyMethodDef cAccessControl_methods[] = { {"rolesForPermissionOn", (PyCFunction)rolesForPermissionOn, METH_VARARGS, "" }, {"guarded_getattr", (PyCFunction)module_guarded_getattr, METH_VARARGS, "" }, {"aq_validate", (PyCFunction)module_aq_validate, METH_VARARGS, "" }, {"setDefaultBehaviors", (PyCFunction)module_setDefaultBehaviors, METH_VARARGS, "" }, { NULL, NULL } }; static char ZopeSecurityPolicy__doc__[] = "ZopeSecurityPolicy C implementation"; static PyMethodDef ZopeSecurityPolicy_methods[] = { {"validate", (PyCFunction)ZopeSecurityPolicy_validate, METH_VARARGS, "" }, { NULL, NULL } }; static PyExtensionClass ZopeSecurityPolicyType = { PyObject_HEAD_INIT(NULL) 0, "ZopeSecurityPolicy", /* tp_name */ sizeof(ZopeSecurityPolicy), /* tp_basicsize */ 0, /* tp_itemsize */ /* Standard methods */ (destructor) ZopeSecurityPolicy_dealloc,/* tp_dealloc */ NULL, /* tp_print */ NULL, /* tp_getattr */ NULL, /* tp_setattr */ NULL, /* tp_compare */ NULL, /* tp_repr */ /* Method suites */ NULL, /* tp_as_number */ NULL, /* tp_as_sequence*/ NULL, /* tp_as_mapping */ /* More standard ops */ NULL, /* tp_hash */ NULL, /* tp_call */ NULL, /* tp_str */ NULL, /* tp_getattro */ NULL, /* tp_setattro */ /* Reserved fields */ 0, /* tp_xxx3 */ 0, /* tp_xxx4 */ /* Docstring */ ZopeSecurityPolicy__doc__, /* tp_doc */ #ifdef COUNT_ALLOCS 0, /* tp_alloc */ 0, /* tp_free */ 0, /* tp_maxalloc */ NULL, /* tp_next */ #endif METHOD_CHAIN(ZopeSecurityPolicy_methods),/* methods */ (void *)(EXTENSIONCLASS_BINDABLE_FLAG), /* flags */ }; static char SecurityManager__doc__[] = "ZopeSecurityPolicy C implementation"; static PyMethodDef SecurityManager_methods[] = { {"validate", (PyCFunction)SecurityManager_validate, METH_VARARGS, "" }, {"DTMLValidate", (PyCFunction)SecurityManager_DTMLValidate, METH_VARARGS, "" }, {"validateValue", (PyCFunction)SecurityManager_validateValue, METH_VARARGS, "" }, {"checkPermission", (PyCFunction)SecurityManager_checkPermission, METH_VARARGS, "" }, { NULL, NULL } }; static PyExtensionClass SecurityManagerType = { PyObject_HEAD_INIT(NULL) 0, "SecurityManager", /* tp_name */ sizeof(SecurityManager), /* tp_basicsize */ 0, /* tp_itemsize */ /* Standard methods */ (destructor) SecurityManager_dealloc,/* tp_dealloc */ NULL, /* tp_print */ NULL, /* tp_getattr */ NULL, /* tp_setattr */ NULL, /* tp_compare */ NULL, /* tp_repr */ /* Method suites */ NULL, /* tp_as_number */ NULL, /* tp_as_sequence*/ NULL, /* tp_as_mapping */ /* More standard ops */ NULL, /* tp_hash */ NULL, /* tp_call */ NULL, /* tp_str */ (getattrofunc)SecurityManager_getattro, /* tp_getattro */ (setattrofunc)SecurityManager_setattro, /* tp_setattro */ /* Reserved fields */ 0, /* tp_xxx3 */ 0, /* tp_xxx4 */ /* Docstring */ SecurityManager__doc__, /* tp_doc */ #ifdef COUNT_ALLOCS 0, /* tp_alloc */ 0, /* tp_free */ 0, /* tp_maxalloc */ NULL, /* tp_next */ #endif METHOD_CHAIN(SecurityManager_methods), /* methods */ 0, /* flags */ }; static char PermissionRole__doc__[] = "PermissionRole C implementation"; static PyMethodDef PermissionRole_methods[] = { {"__init__", (PyCFunction)PermissionRole_init, METH_VARARGS, "" }, {"__of__", (PyCFunction)PermissionRole_of, METH_O, "" }, {"rolesForPermissionOn", (PyCFunction)PermissionRole_rolesForPermissionOn, METH_O, "" }, { NULL, NULL } }; static PyExtensionClass PermissionRoleType = { PyObject_HEAD_INIT(NULL) 0, "PermissionRole", /* tp_name */ sizeof(PermissionRole), /* tp_basicsize */ 0, /* tp_itemsize */ /* Standard methods */ (destructor) PermissionRole_dealloc, /* tp_dealloc */ NULL, /* tp_print */ NULL, /* tp_getattr */ NULL, /* tp_setattr */ NULL, /* tp_compare */ NULL, /* tp_repr */ /* Method suites */ NULL, /* tp_as_number */ NULL, /* tp_as_sequence*/ NULL, /* tp_as_mapping */ /* More standard ops */ NULL, /* tp_hash */ NULL, /* tp_call */ NULL, /* tp_str */ (getattrofunc) PermissionRole_getattro, /* tp_getattro */ NULL, /* tp_setattro */ /* Reserved fields */ 0, /* tp_xxx3 */ 0, /* tp_xxx4 */ /* Docstring */ PermissionRole__doc__, /* tp_doc */ #ifdef COUNT_ALLOCS 0, /* tp_alloc */ 0, /* tp_free */ 0, /* tp_maxalloc */ NULL, /* tp_next */ #endif METHOD_CHAIN(PermissionRole_methods), /* methods */ (void *)(EXTENSIONCLASS_BINDABLE_FLAG)/*| EXTENSIONCLASS_INSTDICT_FLAG*/, /* flags */ NULL, /* Class dict */ 0, /* bases */ NULL, /* reserved */ }; static char imPermissionRole__doc__[] = "imPermissionRole C implementation"; static PyMethodDef imPermissionRole_methods[] = { {"__of__", (PyCFunction)imPermissionRole_of, METH_O, "" }, {"rolesForPermissionOn", (PyCFunction)imPermissionRole_of, METH_O, "" }, { NULL, NULL } }; static PySequenceMethods imSequenceMethods = { (inquiry) imPermissionRole_length, /* sq_length */ (binaryfunc) NULL, /* sq_concat */ (intargfunc) NULL, /* sq_repeat */ (intargfunc) imPermissionRole_get, /* sq_item */ (intintargfunc) NULL, /* sq_slice */ (intobjargproc) NULL, /* sq_ass_item */ (intintobjargproc) NULL, /* sq_ass_slice */ (objobjproc) NULL, /* sq_contains */ (binaryfunc) NULL, /* sq_inplace_concat */ (intargfunc) NULL /* sq_inplace_repeat */ }; static PyExtensionClass imPermissionRoleType = { PyObject_HEAD_INIT(NULL) 0, "imPermissionRole", /* tp_name */ sizeof(imPermissionRole), /* tp_basicsize */ 0, /* tp_itemsize */ /* Standard methods */ (destructor) imPermissionRole_dealloc, /* tp_dealloc */ NULL, /* tp_print */ NULL, /* tp_getattr */ NULL, /* tp_setattr */ NULL, /* tp_compare */ NULL, /* tp_repr */ /* Method suites */ NULL, /* tp_as_number */ &imSequenceMethods, /* tp_as_sequence*/ NULL, /* tp_as_mapping */ /* More standard ops */ NULL, /* tp_hash */ NULL, /* tp_call */ NULL, /* tp_str */ NULL, /* tp_getattro */ NULL, /* tp_setattro */ /* Reserved fields */ 0, /* tp_xxx3 */ 0, /* tp_xxx4 */ /* Docstring */ imPermissionRole__doc__, /* tp_doc */ #ifdef COUNT_ALLOCS 0, /* tp_alloc */ 0, /* tp_free */ 0, /* tp_maxalloc */ NULL, /* tp_next */ #endif METHOD_CHAIN(imPermissionRole_methods), /* methods */ (void *)(EXTENSIONCLASS_BINDABLE_FLAG), /* flags */ }; /* -------------------------------------------------------------- ** STATIC OBJECTS ** -------------------------------------------------------------- */ static PyObject *Containers = NULL; static PyObject *ContainerAssertions = NULL; static PyObject *Unauthorized = NULL; static PyObject *LOG = NULL; static PyObject *PROBLEM = NULL; static PyObject *NoSequenceFormat = NULL; static PyObject *_what_not_even_god_should_do = NULL; static PyObject *Anonymous = NULL; static PyObject *AnonymousTuple = NULL; static PyObject *ManagerTuple = NULL; static PyObject *imPermissionRoleObj = NULL; static PyObject *defaultPermission = NULL; static PyObject *__roles__ = NULL; static PyObject *__of__ = NULL; static PyObject *__allow_access_to_unprotected_subobjects__ = NULL; static PyObject *stack_str = NULL; static PyObject *user_str = NULL; static PyObject *validate_str = NULL; static PyObject *_proxy_roles_str = NULL; static PyObject *allowed_str = NULL; static PyObject *getOwner_str = NULL; static PyObject *getPhysicalRoot_str = NULL; static PyObject *checkPermission_str = NULL; static PyObject *getSecurityManager = NULL; static PyObject *unrestrictedTraverse_str = NULL; static PyObject *aq_validate = NULL; static PyObject *aq_parent_str = NULL; static PyObject *_check_context_str = NULL; static PyObject *getRoles = NULL; static PyObject *getWrappedOwner_str = NULL; static int ownerous = 1; static int authenticated = 1; /* -------------------------------------------------------------- ** ZopeSecurityPolicy Methods ** -------------------------------------------------------------- */ /* ZopeSecurityPolicy_setup ** ** Setup for ZopeSecurityPolicy -- load all necessary objects from ** elsewhere... (e.g. imports) */ static int ZopeSecurityPolicy_setup(void) { UNLESS (NoSequenceFormat = PyString_FromString( "'%s' passed as roles" " during validation of '%s' is not a sequence." )) return -1; UNLESS (defaultPermission = Py_BuildValue("(s)", "Manager")) return -1; UNLESS (_what_not_even_god_should_do = Py_BuildValue("[]")) return -1; UNLESS (__roles__ = PyString_FromString("__roles__")) return -1; UNLESS (__of__ = PyString_FromString("__of__")) return -1; UNLESS (Anonymous = PyString_FromString("Anonymous")) return -1; UNLESS (AnonymousTuple = Py_BuildValue("(O)", Anonymous)) return -1; UNLESS (ManagerTuple = Py_BuildValue("(s)", "Manager")) return -1; UNLESS (stack_str = PyString_FromString("stack")) return -1; UNLESS (user_str = PyString_FromString("user")) return -1; UNLESS (validate_str = PyString_FromString("validate")) return -1; UNLESS (_proxy_roles_str = PyString_FromString("_proxy_roles")) return -1; UNLESS (allowed_str = PyString_FromString("allowed")) return -1; UNLESS (getOwner_str = PyString_FromString("getOwner")) return -1; UNLESS (getWrappedOwner_str = PyString_FromString("getWrappedOwner")) return -1; UNLESS (getPhysicalRoot_str = PyString_FromString("getPhysicalRoot")) return -1; UNLESS (aq_parent_str = PyString_FromString("aq_parent")) return -1; UNLESS (_check_context_str = PyString_FromString("_check_context")) return -1; UNLESS (unrestrictedTraverse_str = PyString_FromString( "unrestrictedTraverse")) return -1; UNLESS (checkPermission_str = PyString_FromString("checkPermission")) return -1; UNLESS (__allow_access_to_unprotected_subobjects__ = PyString_FromString( "__allow_access_to_unprotected_subobjects__")) return -1; return 0; } /* ** unauthErr ** ** Generate the unauthorized error */ static void unauthErr(PyObject *name, PyObject *value) { PyObject *v; if ((v=Py_BuildValue("OO", name, value))) { PyErr_SetObject(Unauthorized, v); Py_DECREF(v); } } /* ** ZopeSecurityPolicy_validate */ static PyObject *ZopeSecurityPolicy_validate(PyObject *self, PyObject *args) { PyObject *accessed = NULL; /* Note: accessed is not used. */ PyObject *container = NULL; PyObject *name = NULL; PyObject *value = NULL; PyObject *context = NULL; PyObject *roles = NULL; /* Import from SimpleObject Policy._noroles */ /* Note that _noroles means missing roles, spelled with a NULL in C. Jim. */ PyObject *containerbase = NULL; PyObject *accessedbase = NULL; PyObject *p = NULL; PyObject *rval = NULL; PyObject *stack = NULL; PyObject *user = NULL; PyObject *method = NULL; PyObject *tmp = NULL; int i, l, contains; PyObject *r; /*| def validate(self, accessed, container, name, value, context **| roles=_noroles ... */ if (unpacktuple6(args, "validate", 5, &accessed, &container, &name, &value, &context, &roles) < 0) return NULL; /*| # Provide special rules for acquisition attributes **| if type(name) in (StringType, UnicodeType): **| if name[:3] == 'aq_' and name not in valid_aq_: **| raise Unauthorized(name, value) */ if (PyString_Check(name) || PyUnicode_Check(name)) { char *sname = PyString_AsString(name); /* Conversion to string may have failed, e.g. if name is Unicode * and can't be bashed into the default encoding. Unclear what * to do then. It's arguably conservative to raise Unauthorized * in this case. */ if (sname == NULL || /* or starts with "aq_" */ (sname[0] == 'a' && sname[1] == 'q' && sname[2] == '_' && /* and isn't aq_{parent, inner, explicit} */ strcmp(sname + 3, "parent") && strcmp(sname + 3, "inner") && strcmp(sname + 3, "explicit") ) ) { /* Access control violation */ unauthErr(name, value); return NULL; /* roles is not owned yet */ } } Py_XINCREF(roles); /* Convert the borrowed ref to a real one */ /*| containerbase = aq_base(container) **| accessedbase = aq_base(accessed) **| if accessedbase is accessed: **| # accessed is not a wrapper, so assume that the **| # value could not have been acquired. **| accessedbase = container */ containerbase = aq_base(container); if (containerbase == NULL) goto err; if (aq_isWrapper(accessed)) accessedbase = aq_base(accessed); else { Py_INCREF(container); accessedbase = container; } /*| # If roles weren't passed in, we'll try to get them from **| # the object **| **| if roles is _noroles: **| roles = getRoles(container, name, value, _noroles) */ if (roles == NULL) { roles = callfunction4(getRoles, container, name, value, getRoles); if (roles == getRoles) { Py_DECREF(roles); roles = NULL; } if (roles == NULL) PyErr_Clear(); } /*| # We still might not have any roles **| **| if roles is _noroles: */ if (roles == NULL) { /*| # We have an object without roles and we didn't get **| # a list of roles passed in. Presumably, the value **| # is some simple object like a string or a list. **| # We'll try to get roles from it's container **| **| if container is None: raise Unauthorized(name, value) */ if (container == Py_None) { unauthErr(name, value); goto err; } /*| roles = getattr(container, "__roles__", _noroles) **| if roles is _noroles: **| if containerbase is container: **| # Container is not wrapped. **| if containerbase is not accessedbase: **| raise Unauthorized(name, value) **| else: **| # Try to acquire roles **| try: roles = container.aq_aquire('__roles__') **| except AttributeError: **| if containerbase is not accessedbase: **| raise Unauthorized(name, value) */ roles = PyObject_GetAttr(container, __roles__); if (roles == NULL) { PyErr_Clear(); if (!aq_isWrapper(container)) { if (containerbase != accessedbase) { unauthErr(name, value); goto err; } } else { roles = aq_acquire(container, __roles__); if (roles == NULL) { if (PyErr_ExceptionMatches( PyExc_AttributeError)) { PyErr_Clear(); if (containerbase != accessedbase) { unauthErr(name, value); goto err; } } else goto err; } } } /*| # We need to make sure that we are allowed to get **| # unprotected attributes from the container. We are **| # allowed for certain simple containers and if the **| # container says we can. Simple containers may also **| # impose name restrictions. **| **| p = Containers(type(container), None) **| if p is None: **| p = getattr(container, **| "__allow_access_to_unprotected_subobjects__", None) */ p = callfunction2(Containers, OBJECT(container->ob_type), Py_None); if (p == NULL) goto err; if (p == Py_None) { ASSIGN(p, PyObject_GetAttr(container, __allow_access_to_unprotected_subobjects__)); if (p == NULL) PyErr_Clear(); } /*| if p is not None: **| tp = type(p) **| if tp is not IntType: **| if tp is DictType: **| if (isinstance(name, StringType) or **| isinstance(name, UnicodeType)): **| p=p.get(name, None) **| else: **| p = 1 **| else: **| p = p(name, value) */ if (p) { if (! PyInt_Check(p)) { if (PyDict_Check(p)) { if (PyString_Check(name) || PyUnicode_Check(name)) { ASSIGN(p, PyObject_GetItem(p, name)); if (p == NULL) PyErr_Clear(); } else { ASSIGN(p, PyInt_FromLong(1)); if (p == NULL) goto err; } } else { ASSIGN(p, callfunction2(p, name, value)); if (p == NULL) goto err; } } } /*| if not p: **| raise Unauthorized, cleanupName(name, value) */ if (p == NULL || ! PyObject_IsTrue(p)) { Py_XDECREF(p); unauthErr(name, value); goto err; } else Py_DECREF(p); /*| if roles is _noroles: return 1 */ if (roles == NULL) { rval = PyInt_FromLong(1); goto err; } /*| # We are going to need a security-aware object to pass **| # to allowed(). We'll use the container **| **| value = container */ value = container; /* Both are borrowed references */ } /* if (roles == NULL) */ /*| # Short-circuit tests if we can **| try: **| if roles is None or 'Anonymous' in roles: return 1 **| except TypeError: **| LOG("Zope Security Policy", PROBLEM, '"%s' passed as roles" **| " during validation of '%s' is not a sequence." % **| ('roles', name)) **| raise */ if (roles == Py_None) { rval = PyInt_FromLong(1); goto err; } else { int i; i = PySequence_Contains(roles, Anonymous); if (i > 0) { rval = PyInt_FromLong(1); goto err; } else if (i < 0) { /* Error */ PyObject *m, *t, *v, *tb; if (!PyErr_ExceptionMatches(PyExc_TypeError)) goto err; PyErr_Fetch(&t, &v, &tb); m=PyObject_Repr(roles); if (m) ASSIGN(m, Py_BuildValue("OO", m, name)); if (m) ASSIGN(m, PyString_Format(NoSequenceFormat, m)); if (m) ASSIGN(m, PyObject_CallFunction(LOG, "sOO", "Zope Security Policy", PROBLEM, m)); Py_XDECREF(m); PyErr_Restore(t, v, tb); goto err; } } /*| # Check executable security **| stack = context.stack **| if stack: */ stack = PyObject_GetAttr(context, stack_str); if (stack == NULL) goto err; if (PyObject_IsTrue(stack)) { PyObject *eo; PyObject *owner; PyObject *proxy_roles; /*| eo = stack[-1] **| # If the executable had an owner, can it execute? **| owner = eo.getOwner() **| if (owner is not None) and not owner.allowed(value, roles) **| # We don't want someone to acquire if they can't **| # get an unacquired! **| raise Unauthorized, ('You are not authorized to' **| 'access %s.' % cleanupName(name, value)) */ eo = PySequence_GetItem(stack, -1); if (eo == NULL) goto err; if (ownerous) { /* Tabbing not adjusted for diff reasons*/ owner = PyObject_GetAttr(eo, getOwner_str); if (owner) ASSIGN(owner, PyObject_CallObject(owner, NULL)); if (owner ==NULL) { Py_DECREF(eo); goto err; } if (owner != Py_None) { ASSIGN(owner,PyObject_GetAttr(owner, allowed_str)); if (owner) ASSIGN(owner, callfunction2(owner, value, roles)); if (owner == NULL) { Py_DECREF(eo); goto err; } if (! PyObject_IsTrue(owner)) { Py_DECREF(owner); Py_DECREF(eo); unauthErr(name, value); goto err; } } Py_DECREF(owner); } /* End of if ownerous */ /*| # Proxy roles, which are a lot safer now **| proxy_roles = getattr(eo, "_proxy_roles", None) **| if proxy_roles: **| # Verify that the owner actually can state the proxy role **| # in the context of the accessed item; users in subfolders **| # should not be able to use proxy roles to access items **| # above their subfolder! **| owner = eo.getWrappedOwner() **| **| if owner is not None: **| if container is not containerbase: **| if not owner._check_context(container): **| # container is higher up than the owner, **| # deny access **| raise Unauthorized(name, value) **| **| for r in proxy_roles: **| if r in roles: **| return 1 **| **| raise Unauthorized(name, value) */ proxy_roles = PyObject_GetAttr(eo, _proxy_roles_str); if (proxy_roles == NULL) { Py_DECREF(eo); PyErr_Clear(); } else if (PyObject_IsTrue(proxy_roles)) { method = PyObject_GetAttr(eo, getWrappedOwner_str); if (method == NULL) { Py_DECREF(eo); Py_DECREF(proxy_roles); goto err; } owner = PyObject_CallObject(method, NULL); Py_DECREF(method); if (owner == NULL) { Py_DECREF(eo); Py_DECREF(proxy_roles); goto err; } Py_DECREF(eo); if (owner != Py_None) { if (containerbase != container) { tmp = callmethod1(owner,_check_context_str, container ); if (tmp == NULL) { Py_DECREF(proxy_roles); Py_DECREF(owner); goto err; } if (!PyObject_IsTrue(tmp)) { Py_DECREF(proxy_roles); Py_DECREF(owner); Py_DECREF(tmp); unauthErr(name, value); goto err; } Py_DECREF(tmp); } Py_DECREF(owner); } contains = 0; if (PyTuple_Check(proxy_roles)) { l=PyTuple_GET_SIZE(proxy_roles); for (i=0; i < l; i++) { r=PyTuple_GET_ITEM(proxy_roles, i); if ((contains = PySequence_Contains(roles, r))) break; } } else { l=PySequence_Size(proxy_roles); if (l < 0) contains = -1; for (i=0; i < l; i++) { if ((r=PySequence_GetItem(proxy_roles, i))) { contains = PySequence_Contains(roles, r); Py_DECREF(r); } else contains = -1; if (contains < 0) break; } } Py_DECREF(proxy_roles); if (contains > 0) rval = PyInt_FromLong(contains); else if (contains == 0) { unauthErr(name, value); } goto err; } else { Py_DECREF(eo); Py_DECREF(proxy_roles); } } /* End of stack check */ /*| try: **| if context.user.allowed(value, roles): return 1 **| except AttributeError: pass */ if (authenticated) { /* Authentication skip for public only access */ user = PyObject_GetAttr(context, user_str); if (user) ASSIGN(user, PyObject_GetAttr(user, allowed_str)); if (user == NULL) { if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear(); else goto err; } else { ASSIGN(user, callfunction2(user, value, roles)); if (user == NULL) goto err; if (PyObject_IsTrue(user)) { rval = PyInt_FromLong(1); Py_DECREF(user); goto err; } Py_DECREF(user); } } /* End of authentiction skip for public only access */ /*| raise Unauthorized(name, value) */ unauthErr(name, value); err: Py_XDECREF(containerbase); Py_XDECREF(accessedbase); Py_XDECREF(stack); Py_XDECREF(roles); return rval; } /* ** ZopeSecurityPolicy_dealloc ** */ static void ZopeSecurityPolicy_dealloc(ZopeSecurityPolicy *self) { Py_DECREF(self->ob_type); /* Extensionclass init incref'd */ PyMem_DEL(self); } /* SecurityManager */ #define CHECK_SECURITY_MANAGER_STATE(self, R) \ UNLESS (self->policy) { \ PyErr_SetString(PyExc_AttributeError, "_policy"); return R; } \ UNLESS (self->context) { \ PyErr_SetString(PyExc_AttributeError, "_policy"); return R; } #define GET_SECURITY_MANAGER_VALIDATE(self, R) \ if (self->validate == NULL && \ ((self->validate = PyObject_GetAttr(self->policy, validate_str)) \ == NULL)) return R; static PyObject * SecurityManager_validate(SecurityManager *self, PyObject *args) { PyObject *accessed=Py_None, *container=Py_None, *name=Py_None, *value=Py_None, *roles=NULL; if (unpacktuple5(args, "validate", 0, &accessed, &container, &name, &value, &roles) < 0) return NULL; CHECK_SECURITY_MANAGER_STATE(self, NULL); GET_SECURITY_MANAGER_VALIDATE(self, NULL); if (roles== NULL) return callfunction5(self->validate, accessed, container, name, value, self->context); return callfunction6(self->validate, accessed, container, name, value, self->context, roles); } static PyObject * SecurityManager_validateValue(SecurityManager *self, PyObject *args) { PyObject *value=Py_None, *roles=NULL; if (unpacktuple2(args, "validateValue", 1, &value, &roles) < 0) return NULL; CHECK_SECURITY_MANAGER_STATE(self, NULL); GET_SECURITY_MANAGER_VALIDATE(self, NULL); if (roles==NULL) return callfunction5(self->validate, Py_None, Py_None, Py_None, value, self->context); return callfunction6(self->validate, Py_None, Py_None, Py_None, value, self->context, roles); } static PyObject * SecurityManager_DTMLValidate(SecurityManager *self, PyObject *args) { PyObject *accessed=Py_None, *container=Py_None, *name=Py_None, *value=Py_None, *md=NULL; if (unpacktuple5(args, "DTMLValidate", 0, &accessed, &container, &name, &value, &md) < 0) return NULL; CHECK_SECURITY_MANAGER_STATE(self, NULL); GET_SECURITY_MANAGER_VALIDATE(self, NULL); return callfunction5(self->validate, accessed, container, name, value, self->context); } static PyObject * SecurityManager_checkPermission(SecurityManager *self, PyObject *args) { PyObject *permission, *object; if (unpacktuple2(args, "checkPermission", 2, &permission, &object) < 0) return NULL; CHECK_SECURITY_MANAGER_STATE(self, NULL); if (self->checkPermission == NULL && ((self->checkPermission = PyObject_GetAttr(self->policy, checkPermission_str)) == NULL)) return NULL; return callfunction3(self->checkPermission, permission, object, self->context); } static void SecurityManager_dealloc(SecurityManager *self) { Py_XDECREF(self->thread_id); Py_XDECREF(self->context); Py_XDECREF(self->policy); Py_XDECREF(self->validate); Py_XDECREF(self->checkPermission); Py_DECREF(self->ob_type); /* Extensionclass init incref'd */ PyMem_DEL(self); } static PyObject * SecurityManager_getattro(SecurityManager *self, PyObject *name) { if (PyString_Check(name) || PyUnicode_Check(name)) { char *name_s = PyString_AsString(name); if (name_s == NULL) return NULL; if (name_s[0] == '_') { if (! strcmp(name_s, "_thread_id") && self->thread_id) { Py_INCREF(self->thread_id); return self->thread_id; } else if (! strcmp(name_s, "_context") && self->context) { Py_INCREF(self->context); return self->context; } else if (! strcmp(name_s, "_policy") && self->policy) { Py_INCREF(self->policy); return self->policy; } } } return Py_FindAttr(OBJECT(self), name); } static int SecurityManager_setattro(SecurityManager *self, PyObject *name, PyObject *v) { if (PyString_Check(name) || PyUnicode_Check(name)) { char *name_s = PyString_AsString(name); if (name_s == NULL) return -1; if (name_s[0] == '_') { if (! strcmp(name_s, "_thread_id")) { Py_INCREF(v); ASSIGN(self->thread_id, v); return 0; } else if (! strcmp(name_s, "_context")) { Py_INCREF(v); ASSIGN(self->context, v); return 0; } else if (! strcmp(name_s, "_policy")) { Py_INCREF(v); ASSIGN(self->policy, v); if (self->validate) { Py_DECREF(self->validate); self->validate=0; } if (self->checkPermission) { Py_DECREF(self->checkPermission); self->checkPermission=0; } return 0; } } } PyErr_SetObject(PyExc_AttributeError, name); return -1; } /* ** PermissionRole_init ** */ static PyObject *PermissionRole_init(PermissionRole *self, PyObject *args) { PyObject *name = NULL; PyObject *deflt = NULL; /*|def __init__(self, name, default=('Manager',)): **| self.__name__ = name **| self._p = "_" + string.translate(name, name_trans) + "_Permission" **| self._d = default */ if (unpacktuple2(args, "__init__", 1, &name, &deflt) < 0) return NULL; if (deflt == NULL) deflt = defaultPermission; UNLESS(self->_p = permissionName(name)) return NULL; self->__name__ = name; Py_INCREF(name); self->__roles__ = deflt; Py_INCREF(deflt); Py_INCREF(Py_None); return Py_None; } /* def __of__(self, parent): */ static PyObject * PermissionRole_of(PermissionRole *self, PyObject *parent) { imPermissionRole *r = NULL; PyObject *_p = NULL; PyObject *result = NULL; /*| r = imPermissionRole() */ r = (imPermissionRole*)PyObject_CallObject(imPermissionRoleObj, NULL); if (r == NULL) return NULL; /*| r._p = self._p */ r->_p = self->_p; Py_INCREF(r->_p); /*| r._pa = parent */ r->_pa = parent; Py_INCREF(parent); /*| r._d = self._d */ r->__roles__ = self->__roles__; Py_INCREF(r->__roles__); /*| p = getattr(parent, 'aq_inner', None) **| if p is not None: **| return r.__of__(p) **| else: **| return r */ if (aq_isWrapper(parent)) { _p = aq_inner(parent); result = callmethod1(OBJECT(r), __of__, _p); Py_DECREF(_p); /* Dont need goto */ } else { result = OBJECT(r); Py_INCREF(r); } Py_DECREF(r); return result; } /* def rolesForPermissionOn(self, value): return rolesForPermissionOn(None, value, self._d, self._p) */ static PyObject * PermissionRole_rolesForPermissionOn(PermissionRole *self, PyObject *value) { return c_rolesForPermissionOn(NULL, value, self->__roles__, self->_p); } /* ** PermissionRole_dealloc ** */ static void PermissionRole_dealloc(PermissionRole *self) { Py_XDECREF(self->__name__); Py_XDECREF(self->_p); Py_XDECREF(self->__roles__); Py_XDECREF(self->ob_type); /* Extensionclass init incref'd */ PyMem_DEL(self); } /* for DocFinder */ /* ** PermissionRole_getattro ** */ static PyObject * PermissionRole_getattro(PermissionRole *self, PyObject *name) { PyObject *result = NULL; char *name_s = PyString_AsString(name); /* see whether we know the attribute */ /* we support both the old "_d" (from the Python implementation) and the new "__roles__" */ if (name_s == NULL) PyErr_Clear(); /* defer to ExtensionClassGetattro */ else if (name_s[0] == '_') { if (! strcmp(name_s, "__name__")) result = self->__name__; else if (! strcmp(name_s, "__roles__")) result = self->__roles__; else if (! strcmp(name_s, "_p")) result = self->_p; else if (! strcmp(name_s, "_d")) result = self->__roles__; } if (result) { Py_INCREF(result); return result; } else return ExtensionClassGetattro((PyObject *)self, name); } /* def __of__(self, value): return rolesForPermissionOn(None, value, self._d, self._p) */ static PyObject * imPermissionRole_of(imPermissionRole *self, PyObject *value) { return c_rolesForPermissionOn(NULL, value, self->__roles__, self->_p); } /* ** imPermissionRole_length */ static int imPermissionRole_length(imPermissionRole *self) { int l; PyObject *v; PyObject *pa; /*| **| try: **| v=self._v **| except: **| v = self._v = self.__of__(self._pa) **| del self._pa **| **| return len(v) */ v = self->_v; if (v == NULL) { pa = self->_pa; if (pa == NULL) { PyErr_SetString(PyExc_AttributeError, "_pa"); return -1; } v = callmethod1(OBJECT(self), __of__, pa); if (v == NULL) return -1; self->_v = v; Py_DECREF(pa); self->_pa = NULL; } l = PyObject_Length(v); return l; } /* ** imPermissionRole_get */ static PyObject *imPermissionRole_get(imPermissionRole *self, int item) { PyObject *v; PyObject *pa; PyObject *result; /*| try: **| v = self._v **| except: **| v = self._v = self.__of__(self._pa) **| del self._pa **| return v[i] */ v = self->_v; if (v == NULL) { pa = self->_pa; if (pa == NULL) { PyErr_SetString(PyExc_AttributeError, "_pa"); return NULL; } v = callmethod1(OBJECT(self), __of__, pa); if (v == NULL) return NULL; self->_v = v; Py_DECREF(pa); self->_pa = NULL; } result = PySequence_GetItem(v, item); return result; } /* ** imPermissionRole_dealloc ** */ static void imPermissionRole_dealloc(imPermissionRole *self) { Py_XDECREF(self->_p); Py_XDECREF(self->_pa); Py_XDECREF(self->__roles__); Py_XDECREF(self->_v); Py_DECREF(self->ob_type); /* Extensionclass init incref'd */ PyMem_DEL(self); } /* ** rolesForPermissionOn */ static PyObject *rolesForPermissionOn(PyObject *self, PyObject *args) { PyObject *perm = NULL; PyObject *object = NULL; PyObject *deflt = NULL; PyObject *n = NULL; /*|def rolesForPermissionOn(perm, object, default=('Manager',)): **| **| """Return the roles that have the permisson on the given object""" */ if (unpacktuple4(args, "rolesForPermissionOn", 2, &perm, &object, &deflt, &n) < 0) return NULL; return c_rolesForPermissionOn(perm, object, deflt, n); } /* def rolesForPermissionOn(perm, object, default=_default_roles, n=None): """Return the roles that have the given permission on the given object """ */ static PyObject * c_rolesForPermissionOn(PyObject *perm, PyObject *object, PyObject *_default_roles, PyObject *n) { PyObject *r, *result = NULL; /* n = n or '_' + string.translate(perm, name_trans) + "_Permission" */ if (n) Py_INCREF(n); else { n = permissionName(perm); if (n == NULL) return NULL; } Py_INCREF(object); /* r = None */ r = Py_None; Py_INCREF(r); /* while 1: */ while (1) { /* if hasattr(object, n): roles = getattr(object, n) */ PyObject *roles = PyObject_GetAttr(object, n); if (roles != NULL) { /* if roles is None: return 'Anonymous', */ if (roles == Py_None) { Py_DECREF(roles); result = AnonymousTuple; Py_INCREF(result); goto end; } /* t = type(roles) */ /* if t is tuple: # If we get a tuple, then we don't acquire if r is None: return roles return r+list(roles) */ if (PyTuple_Check(roles)) { if (r == Py_None) result = roles; else { PyObject *list_roles = PySequence_List(roles); Py_DECREF(roles); if (list_roles == NULL) goto end; result = PySequence_Concat(r, list_roles); Py_DECREF(list_roles); } goto end; } /* if t is str: # We found roles set to a name. Start over # with the new permission name. If the permission # name is '', then treat as private! if roles: n = roles else: return _what_not_even_god_should_do */ if (PyString_Check(roles)) { if (PyString_GET_SIZE(roles)) { Py_DECREF(n); n = roles; } else { Py_DECREF(roles); result = _what_not_even_god_should_do; Py_INCREF(result); goto end; } } /* elif roles: if r is None: r = list(roles) else: r = r + list(roles) */ else { int bool = PyObject_IsTrue(roles); if (bool < 0) goto end; if (bool) { PyObject *list_roles = PySequence_List(roles); Py_DECREF(roles); if (list_roles == NULL) goto end; if (r == Py_None) { Py_DECREF(r); r = list_roles; } else { PyObject *tmp = PySequence_Concat(r, list_roles); Py_DECREF(list_roles); if (tmp == NULL) goto end; Py_DECREF(r); r = tmp; } } } } else /* roles == NULL */ PyErr_Clear(); /* object = getattr(object, 'aq_inner', None) if object is None: break object = object.aq_parent */ if (! aq_isWrapper(object)) break; { PyObject *tobj = aq_inner(object); if (tobj == NULL) goto end; Py_DECREF(object); object = tobj; if (object == Py_None) break; if (! aq_isWrapper(object)) break; tobj = aq_parent(object); if (tobj == NULL) goto end; Py_DECREF(object); object = tobj; } } /* if r is None: return default return r */ if (r == Py_None) { result = _default_roles; if (result == NULL) result = ManagerTuple; Py_INCREF(result); goto end; } Py_INCREF(r); result = r; end: Py_DECREF(n); Py_DECREF(object); Py_DECREF(r); return result; } /* ** permissionName ** ** Can silently truncate permission names if they are really long */ static PyObject *permissionName(PyObject *name) { char namebuff[512]; register int len = sizeof(namebuff) - 1; char *c = namebuff; char *in; char r; *c = '_'; c++; len--; in = PyString_AsString(name); if (in == NULL) return NULL; while (len && *in) { r = *(in++); if (!isalnum(r)) r='_'; *(c++) = r; len--; } if (len) { in = "_Permission"; while (len && *in) { *(c++) = *(in++); len--; } } *c = '\0'; /* Saved room in len */ return PyString_FromString(namebuff); } /* def guarded_getattr(inst, name, default=_marker): */ static PyObject * guarded_getattr(PyObject *inst, PyObject *name, PyObject *default_, PyObject *validate) { PyObject *v=0, *t=0; int i; /* if name[:1] != '_': */ if (PyString_Check(name) || PyUnicode_Check(name)) { char *name_s = PyString_AsString(name); if (name_s == NULL) return NULL; if (name_s[0] != '_') { /* # Try to get the attribute normally so that unusual # exceptions are caught early. try: v = getattr(inst, name) except AttributeError: if default is not _marker: return default raise */ v=PyObject_GetAttr(inst, name); if (v==NULL) { if (default_ && PyErr_Occurred() == PyExc_AttributeError) { PyErr_Clear(); Py_INCREF(default_); return default_; } return NULL; } /* assertion = Containers(type(inst)) */ t = PyDict_GetItem(ContainerAssertions, OBJECT(inst->ob_type)); if (t != NULL) { /* if isinstance(assertion, dict): # We got a table that lets us reason about individual # attrs assertion = assertion.get(name) if assertion: # There's an entry, but it may be a function. if callable(assertion): return assertion(inst, name) # Nope, it's boolean return v raise Unauthorized, name */ if (PyDict_Check(t)) { PyObject *attrv; attrv = PyDict_GetItem(t, name); if (attrv != NULL) { i=PyObject_IsTrue(attrv); if (i < 0) goto err; if (i) { if (attrv->ob_type->tp_call) { Py_DECREF(v); v = callfunction2(attrv, inst, name); return v; } return v; } } Py_DECREF(v); goto unauth; } /* if assertion: if callable(assertion): factory = assertion(name, v) if callable(factory): return factory(inst, name) assert factory == 1 assert callable == 1 return v */ if (PyCallable_Check(t)) { PyObject *factory; factory = callfunction2(t, name, v); if (factory == NULL) goto err; if (PyCallable_Check(factory)) { Py_DECREF(v); v = callfunction2(factory, inst, name); } Py_DECREF(factory); } return v; } /* # See if we can get the value doing a filtered acquire. # aq_acquire will either return the same value as held by # v or it will return an Unauthorized raised by validate. validate = SecurityManagement.getSecurityManager().validate aq_acquire(inst, name, aq_validate, validate) return v */ t = aq_Acquire(inst, name, aq_validate, validate, 1, NULL, 0); if (t == NULL) { Py_DECREF(v); return NULL; } Py_DECREF(t); return v; unauthErr(name, v); err: Py_DECREF(v); return NULL; } } unauth: /* raise Unauthorized, name */ PyErr_SetObject(Unauthorized, name); return NULL; } static PyObject * module_guarded_getattr(PyObject *ignored, PyObject *args) { PyObject *inst, *name, *default_=0, *validate; if (unpacktuple3(args, "guarded_getattr", 2, &inst, &name, &default_) < 0) return NULL; /* validate = getSecurityManager().validate */ validate=PyObject_CallObject(getSecurityManager, NULL); if (! validate) return NULL; ASSIGN(validate, PyObject_GetAttr(validate, validate_str)); if (! validate) return NULL; ASSIGN(validate, guarded_getattr(inst, name, default_, validate)); return validate; } /* def aq_validate(inst, obj, name, v, validate): return validate(inst, obj, name, v) */ static PyObject * module_aq_validate(PyObject *ignored, PyObject *args) { PyObject *inst, *obj, *name, *v, *validate; if (unpacktuple5(args, "validate", 0, &inst, &obj, &name, &v, &validate) < 0) return NULL; return callfunction4(validate, inst, obj, name, v); } static PyObject * module_setDefaultBehaviors(PyObject *ignored, PyObject *args) { PyObject *result = NULL; int own, auth, verbose; if (PyArg_ParseTuple(args, "iii:setDefaultBehaviors", &own, &auth, &verbose)) { if (verbose) { PyErr_SetString(PyExc_NotImplementedError, "This security policy implementation does not implement " "the verbose option. To enable verbose security " "exceptions, add 'security-policy-implementation " "python' to etc/zope.conf."); return NULL; } ownerous = own; authenticated = authenticated; result = Py_None; Py_INCREF(result); } return result; } static PyObject * dtml_guarded_getattr(PyObject *self, PyObject *args) { PyObject *ob, *name, *default_=0, *validate; if (unpacktuple3(args, "guarded_getattr", 2, &ob, &name, &default_) < 0) return NULL; UNLESS (validate = PyObject_GetAttr(self, validate_str)) { /* This section is pure paranoia at this point. It was necessary while debugging. */ PyErr_Clear(); validate=PyObject_CallObject(getSecurityManager, NULL); if (! validate) return NULL; ASSIGN(validate, PyObject_GetAttr(validate, validate_str)); if (! validate) return NULL; } ASSIGN(validate, guarded_getattr(ob, name, default_, validate)); return validate; } static struct PyMethodDef dtml_methods[] = { {"guarded_getattr", (PyCFunction)dtml_guarded_getattr, METH_VARARGS|METH_KEYWORDS, "" }, {NULL, NULL} }; /* ---------------------------------------------------------------- ** Module initialization ** ---------------------------------------------------------------- */ #define IMPORT(module, name) if ((module = PyImport_ImportModule(name)) == NULL) return; #define GETATTR(module, name) if ((name = PyObject_GetAttrString(module, #name)) == NULL) return; void initcAccessControl(void) { PyObject *module; PyObject *dict; PURE_MIXIN_CLASS(RestrictedDTMLMixin, "A mix-in for derivatives of DT_String.String " "that adds Zope security." , dtml_methods); if (!ExtensionClassImported) return; if (ZopeSecurityPolicy_setup() < 0) return; ExtensionClassGetattro= Py_FindAttr; module = Py_InitModule3("cAccessControl", cAccessControl_methods, "$Id: cAccessControl.c 40459 2005-12-01 22:44:07Z tseaver $\n"); aq_init(); /* For Python <= 2.1.1, aq_init() should be after Py_InitModule(). */ dict = PyModule_GetDict(module); PyDict_SetItemString(dict, "_what_not_even_god_should_do", _what_not_even_god_should_do); PyExtensionClass_Export(dict, "RestrictedDTMLMixin", RestrictedDTMLMixinType); PyExtensionClass_Export(dict, "ZopeSecurityPolicy", ZopeSecurityPolicyType); PyExtensionClass_Export(dict,"SecurityManager", SecurityManagerType); PyExtensionClass_Export(dict, "PermissionRole", PermissionRoleType); PyExtensionClass_Export(dict, "imPermissionRole", imPermissionRoleType); imPermissionRoleObj = PyMapping_GetItemString(dict, "imPermissionRole"); aq_validate = PyMapping_GetItemString(dict, "aq_validate"); /*| from SimpleObjectPolicies import Containers */ IMPORT(module, "AccessControl.SimpleObjectPolicies"); GETATTR(module, Containers); GETATTR(module, ContainerAssertions); Py_DECREF(module); module = NULL; /*| from ZopeSecurityPolicy import getRoles */ IMPORT(module, "AccessControl.ZopeSecurityPolicy"); GETATTR(module, getRoles); Py_DECREF(module); module = NULL; /*| from unauthorized import Unauthorized */ IMPORT(module, "AccessControl.unauthorized"); GETATTR(module, Unauthorized); Py_DECREF(module); module = NULL; /*| from AccessControl.SecurityManagement import getSecurityManager */ IMPORT(module, "AccessControl.SecurityManagement"); GETATTR(module, getSecurityManager); Py_DECREF(module); module = NULL; /*| from zLOG import LOG, PROBLEM */ IMPORT(module, "zLOG"); GETATTR(module, LOG); GETATTR(module, PROBLEM); Py_DECREF(module); module = NULL; } Zope-2.8.7-final/lib/python/AccessControl/dtml/0000775000076400007640000000000010435500320020321 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/dtml/acquiredEdit.dtml0000664000076400007640000000116710435500320023613 0ustar ajungajung

Select permissions that acquire permission settings:

Zope-2.8.7-final/lib/python/AccessControl/dtml/permissionEdit.dtml0000664000076400007640000000202510435500320024200 0ustar ajungajung

Roles assigned to the permission &dtml-permission_to_manage;

> Also use roles acquired from folders containing this object

Zope-2.8.7-final/lib/python/AccessControl/dtml/owner.dtml0000664000076400007640000000211010435500320022327 0ustar ajungajung

Almost all Zope objects can be owned. When you create an object you become its owner. Ownership matters for method objects since it determines what roles they have when they are executed. See the Proxy Roles view of method objects for more information.

This object is owned by &dtml-id; (&dtml-path;).unowned.


If you have the Take Ownership permission you can take ownership of an object. Usually when taking ownership you should also take ownership of sub-objects as well.

Also take ownership of all sub-objects

Zope-2.8.7-final/lib/python/AccessControl/dtml/methodAccess.dtml0000664000076400007640000000422410435500320023607 0ustar ajungajung

This interface is used to define how the operations of this object correspond to the operations defined by your product or ZClass.

The first column below lists the permissions for this object. The second specifies the permissions that should have this permission in this product or ZClass. For ZClass methods, only permissions that are defined for the ZClass are permitted.

In general, any permissions that include operations that change (mutate) an object should be disabled. The listing below shows the current permission mappings for this item.

Permission for this object
Permissions that correspond
to (i.e. have) this permission
&dtml-permission_name;
Zope-2.8.7-final/lib/python/AccessControl/dtml/userFolderProps.dtml0000664000076400007640000000204610435500320024343 0ustar ajungajung
Encrypt user passwords    
Role assignment presents search dialog when more users than N (-1 is always, 0 is never).

Zope-2.8.7-final/lib/python/AccessControl/dtml/mainUser.dtml0000664000076400007640000000260610435500320022772 0ustar ajungajung

The following users have been defined. Click on the name of a user to edit that user.

&dtml-sequence-item;
 

There are no users defined.

Zope-2.8.7-final/lib/python/AccessControl/dtml/roleEdit.dtml0000664000076400007640000000127510435500320022757 0ustar ajungajung

Permissions assigned to the role &dtml-role_to_manage;

Zope-2.8.7-final/lib/python/AccessControl/dtml/editLocalRoles.dtml0000775000076400007640000000310310435500320024110 0ustar ajungajung

Local roles allow you to give particular users extra roles in the context of this object, in addition to the roles they already have.

To change the local roles for this user, select the extra roles this user should have in the context of this object and click the Save Changes button.

User
Roles
&dtml-userid;
Zope-2.8.7-final/lib/python/AccessControl/dtml/editUser.dtml0000664000076400007640000000405710435500320022775 0ustar ajungajung
Name
 
New Password
(Confirm)
Domains
&dtml-sequence-item; " />
Roles
" />

Zope-2.8.7-final/lib/python/AccessControl/dtml/access.dtml0000664000076400007640000001033410435500320022445 0ustar ajungajung

The listing below shows the current security settings for this item. Permissions are rows and roles are columns. Checkboxes are used to indicate where roles are assigned permissions. You can also assign local roles to users, which give users extra roles in the context of this object and its subobjects.

When a role is assigned to a permission, users with the given role will be able to perform tasks associated with the permission on this item. When the Acquire permission settings checkbox is selected then the containing objects's permission settings are used. Note: the acquired permission settings may be augmented by selecting Roles for a permission in addition to selecting to acquire permissions.

 
Permission
">
Roles
Acquire
permission
settings?
&dtml-sequence-item;
&dtml-name;
" align="left"> " align="left">

You can define new roles by entering a role name and clicking the "Add Role" button.

User defined roles
 
Zope-2.8.7-final/lib/python/AccessControl/dtml/addUser.dtml0000664000076400007640000000410410435500320022571 0ustar ajungajung

To add a new user, enter the name ,password, confirmation and roles for the new user and click "Add". Domains is an optional list of domains from which the user is allowed to login.

Name
Password
(Confirm)
Domains
Roles

Zope-2.8.7-final/lib/python/AccessControl/dtml/listLocalRoles.dtml0000775000076400007640000000547510435500320024154 0ustar ajungajung
&dtml-stat;

Local roles allow you to give particular users extra roles in the context of this object, in addition to the roles they already have.

The following users have been given local roles. To modify the local roles given to a particular user, click on the name of the user. To remove all local roles from a user, select the checkbox next to the name of the user and click the Remove button.

&dtml-sequence-key; (&dtml-sequence-item;, )

To give a user extra roles when accessing this object (and its children), select a user from the User list below, select the extra roles that should be given to that user from the Roles list.

User
Roles
Zope-2.8.7-final/lib/python/AccessControl/Owned.py0000664000076400007640000002305710435500321021017 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Support for owned objects $Id: Owned.py 37636 2005-08-01 20:15:00Z efge $ """ import Globals, urlparse, SpecialUsers, ExtensionClass from AccessControl import getSecurityManager, Unauthorized from Acquisition import aq_get, aq_parent, aq_base UnownableOwner=[] def ownableFilter(self): _owner = aq_get(self, '_owner', None, 1) return _owner is not UnownableOwner # Marker to use as a getattr default. _mark=ownableFilter class Owned(ExtensionClass.Base): __ac_permissions__=( ('View management screens', ('manage_owner', 'owner_info')), ('Take ownership', ('manage_takeOwnership','manage_changeOwnershipType'), ("Owner",)), ) manage_options=({'label': 'Ownership', 'action': 'manage_owner', 'help': ('OFSP','Ownership.stx'), 'filter': ownableFilter }, ) manage_owner=Globals.DTMLFile('dtml/owner', globals()) def owner_info(self): """Get ownership info for display """ owner=self.getOwnerTuple() if owner is None or owner is UnownableOwner: return owner d={'path': '/'.join(owner[0]), 'id': owner[1], 'explicit': hasattr(self, '_owner'), 'userCanChangeOwnershipType': getSecurityManager().checkPermission('Take ownership', self) } return d getOwner__roles__=() def getOwner(self, info=0, aq_get=aq_get, UnownableOwner=UnownableOwner, getSecurityManager=getSecurityManager, ): """Get the owner If a true argument is provided, then only the owner path and id are returned. Otherwise, the owner object is returned. """ if info: import warnings warnings.warn('Owned.getOwner(1) is deprecated; ' 'please use getOwnerTuple() instead.', DeprecationWarning, stacklevel=2) owner=aq_get(self, '_owner', None, 1) if info or (owner is None): return owner if owner is UnownableOwner: return None udb, oid = owner root=self.getPhysicalRoot() udb=root.unrestrictedTraverse(udb, None) if udb is None: user = SpecialUsers.nobody else: user = udb.getUserById(oid, None) if user is None: user = SpecialUsers.nobody return user getOwnerTuple__roles__=() def getOwnerTuple(self): """Return a tuple, (userdb_path, user_id) for the owner. o Ownership can be acquired, but only from the containment path. o If unowned, return None. """ return aq_get(self, '_owner', None, 1) getWrappedOwner__roles__=() def getWrappedOwner(self): """Get the owner, modestly wrapped in the user folder. o If the object is not owned, return None. o If the owner's user database doesn't exist, return Nobody. o If the owner ID does not exist in the user database, return Nobody. """ owner = self.getOwnerTuple() if owner is None or owner is UnownableOwner: return None udb_path, oid = owner root = self.getPhysicalRoot() udb = root.unrestrictedTraverse(udb_path, None) if udb is None: return SpecialUsers.nobody user = udb.getUserById(oid, None) if user is None: return SpecialUsers.nobody return user.__of__(udb) changeOwnership__roles__=() def changeOwnership(self, user, recursive=0): """Change the ownership to the given user. If 'recursive' is true then also take ownership of all sub-objects, otherwise sub-objects retain their ownership information. """ new=ownerInfo(user) if new is None: return # Special user! old = self.getOwnerTuple() if old==new: return if old is UnownableOwner: return for child in self.objectValues(): if recursive: child.changeOwnership(user, 1) else: # make ownership explicit child._owner=new if old is not UnownableOwner: self._owner=new def userCanTakeOwnership(self): security=getSecurityManager() user=security.getUser() info=ownerInfo(user) if info is None: return 0 owner=self.getOwnerTuple() if owner == info: return 0 return security.checkPermission('Take ownership', self) def manage_takeOwnership(self, REQUEST, RESPONSE, recursive=0): """Take ownership (responsibility) for an object. If 'recursive' is true, then also take ownership of all sub-objects. """ security=getSecurityManager() want_referer=REQUEST['URL1']+'/manage_owner' got_referer=("%s://%s%s" % urlparse.urlparse(REQUEST['HTTP_REFERER'])[:3]) __traceback_info__=want_referer, got_referer if (want_referer != got_referer or security.calledByExecutable()): raise Unauthorized, ( 'manage_takeOwnership was called from an invalid context' ) self.changeOwnership(security.getUser(), recursive) RESPONSE.redirect(REQUEST['HTTP_REFERER']) def manage_changeOwnershipType(self, explicit=1, RESPONSE=None, REQUEST=None): """Change the type (implicit or explicit) of ownership. """ old=getattr(self, '_owner', None) if explicit: if old is not None: return owner = self.getOwnerTuple() if owner is not None and owner is not UnownableOwner: self._owner=owner else: if old is None: return new=aq_get(aq_parent(self), '_owner', None, 1) if old is new and ( self.__dict__.get('_owner', _mark) is not _mark ): del self._owner if RESPONSE is not None: RESPONSE.redirect(REQUEST['HTTP_REFERER']) def _deleteOwnershipAfterAdd(self): # Only delete _owner if it is an instance attribute. if self.__dict__.get('_owner', _mark) is not _mark: del self._owner for object in self.objectValues(): try: s=object._p_changed except: s=0 try: object._deleteOwnershipAfterAdd() except: pass if s is None: object._p_deactivate() def manage_fixupOwnershipAfterAdd(self): # Sigh, get the parent's _owner parent=getattr(self, 'aq_parent', None) if parent is not None: _owner=aq_get(parent, '_owner', None, 1) else: _owner=None if (_owner is None and ((not hasattr(self, 'aq_parent')) or (not hasattr(self, 'getPhysicalRoot')) ) ): # This is a special case. An object is # being added to an object that hasn't # been added to the object hierarchy yet. # We can delay fixing up the ownership until the # object is actually added. return None if _owner is UnownableOwner: # We want to acquire Unownable ownership! return self._deleteOwnershipAfterAdd() else: # Otherwise change the ownership user=getSecurityManager().getUser() if (SpecialUsers.emergency_user and aq_base(user) is SpecialUsers.emergency_user): __creatable_by_emergency_user__=getattr( self,'__creatable_by_emergency_user__', None) if (__creatable_by_emergency_user__ is None or (not __creatable_by_emergency_user__())): raise EmergencyUserCannotOwn, ( "Objects cannot be owned by the emergency user") self.changeOwnership(user) # Force all subs to acquire ownership! for object in self.objectValues(): try: s=object._p_changed except: s=0 try: object._deleteOwnershipAfterAdd() except: pass if s is None: object._p_deactivate() Globals.default__class_init__(Owned) class EmergencyUserCannotOwn(Exception): "The emergency user cannot own anything" class EditUnowned(Exception): "Can't edit unowned executables" def absattr(attr): if callable(attr): return attr() return attr def ownerInfo(user, getattr=getattr): if user is None: return None uid=user.getId() if uid is None: return uid db=user.aq_inner.aq_parent path=[absattr(db.id)] root=db.getPhysicalRoot() while 1: db=getattr(db,'aq_inner', None) if db is None: break db=db.aq_parent if db is root: break id=db.id if not isinstance(id, str): try: id=id() except: id=str(id) path.append(id) path.reverse() return path, uid Zope-2.8.7-final/lib/python/AccessControl/ZopeGuards.py0000664000076400007640000003701610435500321022026 0ustar ajungajung############################################################################# # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.18 $'[11:-2] import sys import RestrictedPython from RestrictedPython.Guards import safe_builtins, full_write_guard from RestrictedPython.Utilities import utility_builtins from RestrictedPython.Eval import RestrictionCapableEval from SecurityManagement import getSecurityManager from SecurityInfo import secureModule from SimpleObjectPolicies import Containers, ContainerAssertions from zExceptions import Unauthorized _marker = [] # Create a new marker object. safe_builtins = safe_builtins.copy() safe_builtins.update(utility_builtins) # AccessControl.Implementation inserts these names into this module as # module globals: aq_validate, guarded_getattr def initialize(impl): # Called by AccessControl.Implementation.setImplementation() # whenever the selected implementation changes. global guarded_getattr guarded_getattr = impl.guarded_getattr safe_builtins['getattr'] = guarded_getattr def guarded_hasattr(object, name): try: guarded_getattr(object, name) except (AttributeError, Unauthorized): return 0 return 1 safe_builtins['hasattr'] = guarded_hasattr SliceType = type(slice(0)) def guarded_getitem(object, index): if type(index) is SliceType: if index.step is not None: v = object[index] else: start = index.start stop = index.stop if start is None: start = 0 if stop is None: v = object[start:] else: v = object[start:stop] # We don't guard slices. return v v = object[index] if Containers(type(object)) and Containers(type(v)): # Simple type. Short circuit. return v if getSecurityManager().validate(object, object, index, v): return v raise Unauthorized, 'unauthorized access to element %s' % `i` # Create functions using nested scope to store state # This is less expensive then instantiating and calling instances def get_dict_get(d, name): def guarded_get(key, default=None): try: return guarded_getitem(d, key) except KeyError: return default return guarded_get def get_dict_pop(d, name): def guarded_pop(key, default=_marker): try: v = guarded_getitem(d, key) except KeyError: if default is not _marker: return default raise else: del d[key] return v return guarded_pop def get_iter(c, name): iter = getattr(c, name) def guarded_iter(): return SafeIter(iter(), c) return guarded_iter def get_list_pop(lst, name): def guarded_pop(index=-1): # XXX This is not thread safe, but we don't expect # XXX thread interactions between python scripts v = guarded_getitem(lst, index) del lst[index] return v return guarded_pop # See comment in SimpleObjectPolicies for an explanation of what the # dicts below actually mean. _dict_white_list = { 'clear':1, 'copy':1, 'fromkeys':1, 'get':get_dict_get, 'has_key':1, 'items':1, 'iteritems':1, 'keys':1, 'iterkeys': get_iter, 'itervalues':get_iter, 'pop':get_dict_pop, 'popitem':1, 'setdefault':1, 'update':1, 'values':1} def _check_dict_access(name, value): # Check whether value is a dict method self = getattr(value, '__self__', None) if self is None: # item return 1 # Disallow spoofing if type(self) is not dict: return 0 if getattr(value, '__name__', None) != name: return 0 return _dict_white_list.get(name, 0) ContainerAssertions[type({})] = _check_dict_access _list_white_list = { 'append':1, 'count':1, 'extend':1, 'index':1, 'insert':1, 'pop':get_list_pop, 'remove':1, 'reverse':1, 'sort':1} def _check_list_access(name, value): # Check whether value is a dict method self = getattr(value, '__self__', None) if self is None: # item return 1 # Disallow spoofing if type(self) is not list: return 0 if getattr(value, '__name__', None) != name: return 0 return _list_white_list.get(name, 0) ContainerAssertions[type([])] = _check_list_access # This implementation of a "safe" iterator uses a global guard() # function to implement the actual guard. This check is defined as a # global so that it can delay imports of some module to avoid circular # dependencies while also making it possible to use a faster # implementation once the imports are done (by avoiding the import # machinery on subsequent calls). Use of a method on the SafeIter # class is avoided to ensure the best performance of the resulting # function. # The NullIter class skips the guard, and can be used to wrap an # iterator that is known to be safe (as in guarded_enumerate). class SafeIter(object): #__slots__ = '_next', 'container' __allow_access_to_unprotected_subobjects__ = 1 def __init__(self, ob, container=None): self._next = iter(ob).next if container is None: container = ob self.container = container def __iter__(self): return self def next(self): ob = self._next() guard(self.container, ob) return ob class NullIter(SafeIter): def __init__(self, ob): self._next = ob.next def next(self): return self._next() def _error(index): raise Unauthorized, 'unauthorized access to element' def guarded_iter(*args): if len(args) == 1: i = args[0] # Don't double-wrap if isinstance(i, SafeIter): return i if not isinstance(i, xrange): return SafeIter(i) # Other call styles / targets don't need to be guarded return NullIter(iter(*args)) safe_builtins['iter'] = guarded_iter def guard(container, value, index=None): if Containers(type(container)) and Containers(type(value)): # Simple type. Short circuit. return if getSecurityManager().validate(container, container, index, value): return _error(index) # More replacement built-ins. def guarded_filter(f, seq, skip_unauthorized=0): if type(seq) is type(''): return filter(f, seq) if f is None: def f(x): return x v = getSecurityManager().validate result = [] a = result.append for el in seq: if v(seq, seq, None, el): if f(el): a(el) elif not skip_unauthorized: raise Unauthorized, 'unauthorized access to element' return result safe_builtins['filter'] = guarded_filter def guarded_reduce(f, seq, initial=_marker): if initial is _marker: return reduce(f, guarded_iter(seq)) else: return reduce(f, guarded_iter(seq), initial) safe_builtins['reduce'] = guarded_reduce def guarded_max(item, *items): if items: item = [item] item.extend(items) return max(guarded_iter(item)) safe_builtins['max'] = guarded_max def guarded_min(item, *items): if items: item = [item] item.extend(items) return min(guarded_iter(item)) safe_builtins['min'] = guarded_min def guarded_map(f, *seqs): safe_seqs = [] for seqno in range(len(seqs)): seq = guarded_getitem(seqs, seqno) safe_seqs.append(seq) return map(f, *safe_seqs) safe_builtins['map'] = guarded_map def guarded_import(mname, globals={}, locals={}, fromlist=None): mnameparts = mname.split('.') firstmname = mnameparts[0] validate = getSecurityManager().validate module = load_module(None, None, mnameparts, validate, globals, locals) if module is not None: if fromlist is None: fromlist = () try: for name in fromlist: if name == '*': raise ImportError, ('"from %s import *" is not allowed' % mname) v = getattr(module, name, None) if v is None: v = load_module(module, mname, [name], validate, globals, locals) if not validate(module, module, name, v): raise Unauthorized else: return __import__(mname, globals, locals, fromlist) except Unauthorized, why: raise ImportError, ('import of "%s" from "%s" is unauthorized. %s' % (name, mname, why)) raise ImportError, 'import of "%s" is unauthorized' % mname safe_builtins['__import__'] = guarded_import class GuardedListType: def __call__(self, *args, **kwargs): return list(*args, **kwargs) if sys.version_info >= (2, 4): def sorted(self, iterable, cmp=None, key=None, reverse=False): return list.sorted(iterable, cmp=None, key=None, reverse=False) safe_builtins['list'] = GuardedListType() class GuardedDictType: def __call__(self, *args, **kwargs): return dict(*args, **kwargs) def fromkeys(self, S, v=None): return dict.fromkeys(S,v) safe_builtins['dict'] = GuardedDictType() def guarded_enumerate(seq): return NullIter(enumerate(guarded_iter(seq))) safe_builtins['enumerate'] = guarded_enumerate def guarded_sum(sequence, start=0): return sum(guarded_iter(sequence), start) safe_builtins['sum'] = guarded_sum def load_module(module, mname, mnameparts, validate, globals, locals): modules = sys.modules while mnameparts: nextname = mnameparts.pop(0) if mname is None: mname = nextname else: mname = '%s.%s' % (mname, nextname) nextmodule = modules.get(mname, None) if nextmodule is None: nextmodule = secureModule(mname, globals, locals) if nextmodule is None: return else: secureModule(mname) if module and not validate(module, module, nextname, nextmodule): return module = nextmodule return module # This version of apply is used by restricted Python, which transforms # extended call syntax into a call of _apply_(), after tucking the callable # into the first element of args. For example, # f(3, eggs=1, spam=False) # is changed to # _apply_(f, 3, eggs=1, spam=False) def guarded_apply(func, *args, **kws): return builtin_guarded_apply(func, args, kws) # This version is the safe_builtins apply() replacement, so needs to match the # signature of __builtin__.apply. def builtin_guarded_apply(func, args=(), kws={}): # Check the args elements. args may be an arbitrary iterable, and # iterating over it may consume it, so we also need to save away # the arguments in a new list to pass on to the real apply(). i, arglist = 0, [] for elt in args: guard(args, elt, i) arglist.append(elt) i += 1 # Check kws similarly. Checking the keys may not be strictly necessary, # but better safe than sorry. A new argument dict is created just in # case kws is a hostile user-defined instance that may do horrid things # as a side-effect of calling items(). argdict = {} for k, v in kws.items(): guard(kws, k) guard(kws, v, k) argdict[k] = v return func(*arglist, **argdict) safe_builtins['apply'] = builtin_guarded_apply # This metaclass supplies the security declarations that allow all # attributes of a class and its instances to be read and written. def _metaclass(name, bases, dict): for k, v in dict.items(): if k.endswith('__roles__') and k[:len('__roles__')] not in dict: raise Unauthorized, "Can't override security: %s" % k ob = type(name, bases, dict) ob.__allow_access_to_unprotected_subobjects__ = 1 ob._guarded_writes = 1 return ob try: valid_inplace_types = list, set except NameError: # Python 2.3 valid_inplace_types = list inplace_slots = { '+=': '__iadd__', '-=': '__isub__', '*=': '__imul__', '/=': (1/2 == 0) and '__idiv__' or '__itruediv__', '//=': '__ifloordiv__', '%=': '__imod__', '**=': '__ipow__', '<<=': '__ilshift__', '>>=': '__irshift__', '&=': '__iand__', '^=': '__ixor__', '|=': '__ior_', } def __iadd__(x, y): x += y return x def __isub__(x, y): x -= y return x def __imul__(x, y): x *= y return x def __idiv__(x, y): x /= y return x def __ifloordiv__(x, y): x //= y return x def __imod__(x, y): x %= y return x def __ipow__(x, y): x **= y return x def __ilshift__(x, y): x <<= y return x def __irshift__(x, y): x >>= y return x def __iand__(x, y): x &= y return x def __ixor__(x, y): x ^= y return x def __ior__(x, y): x |= y return x inplace_ops = { '+=': __iadd__, '-=': __isub__, '*=': __imul__, '/=': __idiv__, '//=': __ifloordiv__, '%=': __imod__, '**=': __ipow__, '<<=': __ilshift__, '>>=': __irshift__, '&=': __iand__, '^=': __ixor__, '|=': __ior__, } def protected_inplacevar(op, var, expr): """Do an inplace operation If the var has an inplace slot, then disallow the operation unless the var is a list. """ if (hasattr(var, inplace_slots[op]) and not isinstance(var, valid_inplace_types) ): try: cls = var.__class__ except AttributeError: cls = type(var) raise TypeError( "Augmented assignment to %s objects is not allowed" " in untrusted code" % cls.__name__ ) return inplace_ops[op](var, expr) # AccessControl clients generally need to set up a safe globals dict for # use by restricted code. The get_safe_globals() function returns such # a dict, containing '__builtins__' mapped to our safe bulitins, and # bindings for all the special functions inserted into Python code by # RestrictionMutator transformations. A client may wish to add more # bindings to this dict. It's generally safe to do so, as # get_safe_globals returns a (shallow) copy of a canonical safe globals # dict. # Exception: For obscure technical reasons, clients have to import # guarded_getattr from this module (ZopeGuards) and plug it into the # dict themselves, with key '_getattr_'. _safe_globals = {'__builtins__': safe_builtins, '__metaclass__': _metaclass, '_apply_': guarded_apply, '_getitem_': guarded_getitem, '_getiter_': guarded_iter, '_print_': RestrictedPython.PrintCollector, '_write_': full_write_guard, '_inplacevar_': protected_inplacevar, # The correct implementation of _getattr_, aka # guarded_getattr, isn't known until # AccessControl.Implementation figures that out, then # stuffs it into *this* module's globals bound to # 'guarded_getattr'. We can't know what that is at ## '_getattr_': guarded_getattr, } get_safe_globals = _safe_globals.copy RestrictionCapableEval.globals.update(_safe_globals) Zope-2.8.7-final/lib/python/AccessControl/AccessControl.txt0000664000076400007640000001407310435500321022672 0ustar ajungajungSecurity Architecture --------------------- Users ----- Objects representing users may be created in Principia User Folder objects. User objects maintain the information used to authenticate users, and allow roles to be associated with a user. Permissions ----------- A "permission" is the smallest unit of access to an object, roughly equivalent to the atomic permissions seen in NT: R (Read), W(Write), X(Execute), etc. In Principia, a permission usually describes a fine-grained logical operation on an object, such as "View Management Screens", "Add Properties", etc. Different types of objects will define different permissions as appropriate for the object. Types of access --------------- A "type of access" is a named grouping of 0 or more of the permissions defined by an object. All objects have one predefined type of access called Full Access (all permissions defined by that object). A user who has the special role "Manager" always has Full Access to all objects at or below the level in the object hierarchy at which the user is defined. New types of access may be defined as combinations of the various permissions defined by a given object. These new types of access may be defined by the programmer, or by users at runtime. Roles ----- A role is a name that ties users (authentication of identity) to permissions (authorization for that identity) in the system. Roles may be defined in any Folder (or Folderish) object in the system. Sub folders can make use of roles defined higher in the hierarchy. These roles can be assigned to users. All users, including non-authenticated users have the built-in role of "Anonymous". Principia objects allow the association of defined roles with a single "type of access" each, in the context of that object. A single role is associated with one and only one type of access in the context of a given object. Examples -------- User Object1 o has the role "RoleA" o has given "RoleA" Full Access Result: the user has Full Access to Object1. User Object2 o has the role "RoleA" o has given "RoleB" Full Access o has given the role "RoleA" View Access, a custom type of access that allows only viewing of the object. Result: the user has only View Access. Notes ----- All objects define a permission called "Default permission". If this permission is given to a role, users with that role will be able to access subobjects which do not explicitly restrict access. Technical --------- Objects define their permissions as logical operations. Programmers have to determine the appropriate operations for their object type, and provide a mapping of permission name to attribute names. It is important to note that permissions cannot overlap - none of the attributes named in a permission can occur in any of the other permissions. The following are proposed permissions for some current principia objects: Folder o View management screens o Change permissions o Undo changes o Add objects o Delete objects o Add properties o Change properties o Delete properties o Default permission Confera Topic o View management screens o Change permissions o Undo changes o Add objects o Delete objects o Add properties o Change properties o Delete properties o Default permission o Change Configuration o Add Messages o Change Messages o Delete Messages Tabula Collection o View management screens o Change permissions o Undo changes o Add objects o Delete objects o Add properties o Change properties o Delete properties o Default permission o Change schema o Upload data o Add computed fields o Change computed fields o Delete computed fields Document/Image/File o View management screens o Change permissions o Change/upload data o View Session o View management screens o Change permissions o Change session config o Join/leave session o Save/discard session Mail Host o View management screens o Change permissions o Change configuration To support the architecture, developers must derive an object from the AccessControl.RoleManager mixin class, and define in their class an __ac_permissions__ attribute. This should be a tuple of tuples, where each tuple represents a permission and contains a string permission name as its first element and a list of attribute names as its second element. Example: __ac_permissions__=( ('View management screens', ['manage','manage_menu','manage_main','manage_copyright', 'manage_tabs','manage_propertiesForm','manage_UndoForm']), ('Undo changes', ['manage_undo_transactions']), ('Change permissions', ['manage_access']), ('Add objects', ['manage_addObject']), ('Delete objects', ['manage_delObjects']), ('Add properties', ['manage_addProperty']), ('Change properties', ['manage_editProperties']), ('Delete properties', ['manage_delProperties']), ('Default permission', ['']), ) The developer may also predefine useful types of access, by specifying an __ac_types__ attribute. This should be a tuple of tuples, where each tuple represents a type of access and contains a string name as its first element and a list of permission names as its second element. By default, only "Full Access" is defined (by the RoleManager mixin). If you wish to override __ac_types__ to provide convenient types of access, you must always be sure to define "Full Access" as containing the names of all possible permissions for your object. Example: __ac_types__=( ('Full Access', map(lambda x: x[0], __ac_permissions__)), ('Change', ['Add Objects', 'Add Properties', 'Change Properties']), ) Developers may also provide pre-defined role names that are not deletable via the interface by specifying an __ac_roles__ attribute. This is probably not something we'll ever use under the new architecture, but it's there if you need it. Example: __ac_roles__=('Manager', 'Anonymous') Zope-2.8.7-final/lib/python/AccessControl/Permissions.py0000664000076400007640000000601210435500321022246 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Constant definitions for built-in Zope permissions $Id: Permissions.py 29877 2005-04-05 00:35:58Z sidnei $""" access_contents_information='Access contents information' add_database_methods='Add Database Methods' add_documents_images_and_files='Add Documents, Images, and Files' add_external_methods='Add External Methods' add_folders='Add Folders' add_mailhost_objects='Add MailHost objects' add_page_templates='Add Page Templates' add_python_scripts='Add Python Scripts' add_user_folders='Add User Folders' add_versions='Add Versions' add_vocabularies='Add Vocabularies' add_z_gadfly_database_connections='Add Z Gadfly Database Connections' add_zcatalogs='Add ZCatalogs' add_zope_tutorials='Add Zope Tutorials' change_dtml_documents='Change DTML Documents' change_dtml_methods='Change DTML Methods' change_database_connections='Change Database Connections' change_database_methods='Change Database Methods' change_external_methods='Change External Methods' change_images_and_files='Change Images and Files' change_python_scripts='Change Python Scripts' change_versions='Change Versions' change_configuration='Change configuration' change_permissions='Change permissions' change_proxy_roles='Change proxy roles' create_class_instances='Create class instances' define_permissions='Define permissions' delete_objects='Delete objects' edit_factories='Edit Factories' ftp_access='FTP access' import_export_objects='Import/Export objects' join_leave_versions='Join/leave Versions' manage_vocabulary='Manage Vocabulary' manage_zclasses='Manage Z Classes' manage_zcatalog_entries='Manage ZCatalog Entries' manage_zcatalog_indexes='Manage ZCatalogIndex Entries' manage_properties='Manage properties' manage_users='Manage users' open_close_database_connection='Open/Close Database Connection' open_close_database_connections='Open/Close Database Connections' query_vocabulary='Query Vocabulary' save_discard_version_changes='Save/discard Version changes' search_zcatalog='Search ZCatalog' take_ownership='Take ownership' test_database_connections='Test Database Connections' undo_changes='Undo changes' use_database_methods='Use Database Methods' use_factories='Use Factories' use_mailhost_services='Use mailhost services' view='View' view_history='View History' view_management_screens='View management screens' copy_or_move='Copy or Move' webdav_access='WebDAV access' webdav_lock_items='WebDAV Lock items' webdav_unlock_items='WebDAV Unlock items' Zope-2.8.7-final/lib/python/AccessControl/ImplPython.py0000664000076400007640000007561610435500321022056 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python implementation of the access control machinery.""" import os import string from Acquisition import aq_base from Acquisition import aq_parent from Acquisition import aq_inner from Acquisition import aq_acquire from ExtensionClass import Base from zLOG import LOG, BLATHER, PROBLEM from zope.interface import implements # This is used when a permission maps explicitly to no permission. We # try and get this from cAccessControl first to make sure that if both # security implementations exist, we can switch between them later. try: from cAccessControl import _what_not_even_god_should_do except ImportError: _what_not_even_god_should_do = [] from AccessControl import SecurityManagement from AccessControl import Unauthorized from AccessControl.interfaces import ISecurityPolicy from AccessControl.interfaces import ISecurityManager from AccessControl.SimpleObjectPolicies import Containers, _noroles from AccessControl.ZopeGuards import guarded_getitem # AccessControl.PermissionRole # ---------------------------- _ident_chars = string.ascii_letters + string.digits + "_" name_trans = filter(lambda c, an=_ident_chars: c not in an, map(chr, range(256))) name_trans = string.maketrans(''.join(name_trans), '_' * len(name_trans)) _default_roles = ('Manager',) # If _embed_permission_in_roles is enabled, computed __roles__ # attributes will often include a special role that encodes the name # of the permission from which the roles were derived. This is useful # for verbose security exceptions. _embed_permission_in_roles = 0 def rolesForPermissionOn(perm, object, default=_default_roles, n=None): """Return the roles that have the given permission on the given object """ n = n or '_' + string.translate(perm, name_trans) + "_Permission" r = None while 1: if hasattr(object, n): roles = getattr(object, n) if roles is None: if _embed_permission_in_roles: return ('Anonymous', n) return 'Anonymous', t = type(roles) if t is tuple: # If we get a tuple, then we don't acquire if r is None: if _embed_permission_in_roles: return roles + (n,) return roles if _embed_permission_in_roles: return r + list(roles) + [n] return r + list(roles) if t is str: # We found roles set to a name. Start over # with the new permission name. If the permission # name is '', then treat as private! if roles: n = roles else: return _what_not_even_god_should_do elif roles: if r is None: r = list(roles) else: r = r + list(roles) object = getattr(object, 'aq_inner', None) if object is None: break object = object.aq_parent if r is None: if _embed_permission_in_roles: if default: if isinstance(default, tuple): return default + (n,) else: return default + [n] else: return [n] return default if _embed_permission_in_roles: return r + [n] return r class PermissionRole(Base): """Implement permission-based roles. Under normal circumstances, our __of__ method will be called with an unwrapped object. The result will then be called with a wrapped object, if the original object was wrapped. To deal with this, we have to create an intermediate object. """ def __init__(self, name, default=('Manager',)): self.__name__ = name self._p = '_' + string.translate(name, name_trans) + "_Permission" self._d = self.__roles__ = default def __of__(self, parent): r = imPermissionRole() r._p = self._p r._pa = parent r._d = self._d p = getattr(parent, 'aq_inner', None) if p is not None: return r.__of__(p) else: return r def rolesForPermissionOn(self, value): return rolesForPermissionOn(None, value, self._d, self._p) class imPermissionRole(Base): """Implement permission-based roles""" def __of__(self, value): return rolesForPermissionOn(None, value, self._d, self._p) rolesForPermissionOn = __of__ # The following methods are needed in the unlikely case that an unwrapped # object is accessed: def __getitem__(self, i): try: v = self._v except: v = self._v = self.__of__(self._pa) del self._pa return v[i] def __len__(self): try: v = self._v except: v = self._v = self.__of__(self._pa) del self._pa return len(v) # AccessControl.DTML # ------------------ class RestrictedDTML: """A mix-in for derivatives of DT_String.String that adds Zope security.""" def guarded_getattr(self, *args): # ob, name [, default] return guarded_getattr(*args) def guarded_getitem(self, ob, index): return guarded_getitem(ob, index) # AccessControl.ZopeSecurityPolicy # -------------------------------- # # TODO: implement this in cAccessControl, and have Implementation # do the indirection. # from AccessControl.ZopeSecurityPolicy import getRoles # XXX class ZopeSecurityPolicy: implements(ISecurityPolicy) def __init__(self, ownerous=1, authenticated=1, verbose=0): """Create a Zope security policy. Optional arguments may be provided: ownerous -- Untrusted users can create code (e.g. Python scripts or templates), so check that code owners can access resources. The argument must have a truth value. The default is true. authenticated -- Allow access to resources based on the privaledges of the authenticated user. The argument must have a truth value. The default is true. This (somewhat experimental) option can be set to false on sites that allow only public (unauthenticated) access. An anticipated scenario is a ZEO configuration in which some clients allow only public access and other clients allow full management. verbose -- Include debugging information in Unauthorized exceptions. Not suitable for public sites. """ self._ownerous = ownerous self._authenticated = authenticated self._verbose = verbose def validate(self, accessed, container, name, value, context, roles=_noroles, getattr=getattr, _noroles=_noroles, valid_aq_=('aq_parent','aq_inner', 'aq_explicit')): ############################################################ # Provide special rules for the acquisition attributes if isinstance(name, str): if name.startswith('aq_') and name not in valid_aq_: if self._verbose: raiseVerbose( 'aq_* names (other than %s) are not allowed' % ', '.join(valid_aq_), accessed, container, name, value, context ) raise Unauthorized(name, value) containerbase = aq_base(container) accessedbase = aq_base(accessed) if accessedbase is accessed: # accessed is not a wrapper, so assume that the # value could not have been acquired. accessedbase = container ############################################################ # If roles weren't passed in, we'll try to get them from the object if roles is _noroles: roles = getRoles(container, name, value, _noroles) ############################################################ # We still might not have any roles if roles is _noroles: ############################################################ # We have an object without roles and we didn't get a list # of roles passed in. Presumably, the value is some simple # object like a string or a list. We'll try to get roles # from its container. if container is None: # Either container or a list of roles is required # for ZopeSecurityPolicy to know whether access is # allowable. if self._verbose: raiseVerbose( 'No container provided', accessed, container, name, value, context) raise Unauthorized(name, value) roles = getattr(container, '__roles__', roles) if roles is _noroles: if containerbase is container: # Container is not wrapped. if containerbase is not accessedbase: if self._verbose: raiseVerbose( 'Unable to find __roles__ in the container ' 'and the container is not wrapped', accessed, container, name, value, context) raise Unauthorized(name, value) else: # Try to acquire roles try: roles = container.aq_acquire('__roles__') except AttributeError: if containerbase is not accessedbase: if self._verbose: raiseVerbose( 'Unable to find or acquire __roles__ ' 'from the container', accessed, container, name, value, context) raise Unauthorized(name, value) # We need to make sure that we are allowed to # get unprotected attributes from the container. We are # allowed for certain simple containers and if the # container says we can. Simple containers # may also impose name restrictions. p = Containers(type(container), None) if p is None: p = getattr(container, '__allow_access_to_unprotected_subobjects__', None) if p is not None: tp = p.__class__ if tp is not int: if tp is dict: if isinstance(name, basestring): p = p.get(name) else: p = 1 else: p = p(name, value) if not p: if self._verbose: raiseVerbose( 'The container has no security assertions', accessed, container, name, value, context) raise Unauthorized(name, value) if roles is _noroles: return 1 # We are going to need a security-aware object to pass # to allowed(). We'll use the container. value = container # Short-circuit tests if we can: try: if roles is None or 'Anonymous' in roles: return 1 except TypeError: # 'roles' isn't a sequence LOG('Zope Security Policy', PROBLEM, "'%s' passed as roles" " during validation of '%s' is not a sequence." % ( `roles`, name)) raise # Check executable security stack = context.stack if stack: eo = stack[-1] # If the executable had an owner, can it execute? if self._ownerous: owner = eo.getOwner() if (owner is not None) and not owner.allowed(value, roles): # We don't want someone to acquire if they can't # get an unacquired! if self._verbose: if len(roles) < 1: raiseVerbose( "The object is marked as private", accessed, container, name, value, context) elif userHasRolesButNotInContext(owner, value, roles): raiseVerbose( "The owner of the executing script is defined " "outside the context of the object being " "accessed", accessed, container, name, value, context, required_roles=roles, eo_owner=owner, eo=eo) else: raiseVerbose( "The owner of the executing script does not " "have the required permission", accessed, container, name, value, context, required_roles=roles, eo_owner=owner, eo=eo, eo_owner_roles=getUserRolesInContext( owner, value)) raise Unauthorized(name, value) # Proxy roles, which are a lot safer now. proxy_roles = getattr(eo, '_proxy_roles', None) if proxy_roles: # Verify that the owner actually can state the proxy role # in the context of the accessed item; users in subfolders # should not be able to use proxy roles to access items # above their subfolder! owner = eo.getWrappedOwner() if owner is not None: if container is not containerbase: # Unwrapped objects don't need checking if not owner._check_context(container): # container is higher up than the owner, # deny access if self._verbose: raiseVerbose( "The owner of the executing script is " "defined outside the context of the " "object being accessed. The script has " "proxy roles, but they do not apply in " "this context.", accessed, container, name, value, context, required_roles=roles, eo_owner=owner, eo=eo) raise Unauthorized(name, value) for r in proxy_roles: if r in roles: return 1 # Proxy roles actually limit access! if self._verbose: if len(roles) < 1: raiseVerbose( "The object is marked as private", accessed, container, name, value, context) else: raiseVerbose( "The proxy roles set on the executing script " "do not allow access", accessed, container, name, value, context, eo=eo, eo_proxy_roles=proxy_roles, required_roles=roles) raise Unauthorized(name, value) try: if self._authenticated and context.user.allowed(value, roles): return 1 except AttributeError: pass if self._verbose: if len(roles) < 1: raiseVerbose( "The object is marked as private", accessed, container, name, value, context) elif not self._authenticated: raiseVerbose( "Authenticated access is not allowed by this " "security policy", accessed, container, name, value, context) elif userHasRolesButNotInContext(context.user, value, roles): raiseVerbose( "Your user account is defined outside " "the context of the object being accessed", accessed, container, name, value, context, required_roles=roles, user=context.user) else: raiseVerbose( "Your user account does not " "have the required permission", accessed, container, name, value, context, required_roles=roles, user=context.user, user_roles=getUserRolesInContext(context.user, value)) raise Unauthorized(name, value) def checkPermission(self, permission, object, context): roles = rolesForPermissionOn(permission, object) if isinstance(roles, basestring): roles = [roles] # check executable owner and proxy roles stack = context.stack if stack: eo = stack[-1] # If the executable had an owner, can it execute? if self._ownerous: owner = eo.getOwner() if (owner is not None) and not owner.allowed(object, roles): # We don't want someone to acquire if they can't # get an unacquired! return 0 proxy_roles = getattr(eo, '_proxy_roles', None) if proxy_roles: # Verify that the owner actually can state the proxy role # in the context of the accessed item; users in subfolders # should not be able to use proxy roles to access items # above their subfolder! owner = eo.getWrappedOwner() if owner is not None: if object is not aq_base(object): if not owner._check_context(object): # object is higher up than the owner, # deny access return 0 for r in proxy_roles: if r in roles: return 1 return 0 return context.user.allowed(object, roles) # AccessControl.SecurityManager # ----------------------------- # There is no corresponding control in the C implementation of the # access control machinery (cAccessControl.c); this should probably go # away in a future version. If you're concerned about the size of # security stack, you probably have bigger problems. # try: max_stack_size = int(os.environ.get('Z_MAX_STACK_SIZE','100')) except: max_stack_size = 100 def setDefaultBehaviors(ownerous, authenticated, verbose): global _defaultPolicy global _embed_permission_in_roles _defaultPolicy = ZopeSecurityPolicy( ownerous=ownerous, authenticated=authenticated, verbose=verbose) _embed_permission_in_roles = verbose setDefaultBehaviors(True, True, False) class SecurityManager: """A security manager provides methods for checking access and managing executable context and policies """ implements(ISecurityManager) __allow_access_to_unprotected_subobjects__ = { 'validate': 1, 'checkPermission': 1, 'getUser': 1, 'calledByExecutable': 1 } def __init__(self, thread_id, context): self._thread_id = thread_id self._context = context self._policy = _defaultPolicy def validate(self, accessed=None, container=None, name=None, value=None, roles=_noroles): """Validate access. Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. roles -- The roles of the object if already known. The arguments may be provided as keyword arguments. Some of these arguments may be ommitted, however, the policy may reject access in some cases when arguments are ommitted. It is best to provide all the values possible. """ policy = self._policy if roles is _noroles: return policy.validate(accessed, container, name, value, self._context) else: return policy.validate(accessed, container, name, value, self._context, roles) def DTMLValidate(self, accessed=None, container=None, name=None, value=None, md=None): """Validate access. * THIS EXISTS FOR DTML COMPATIBILITY * Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. md -- multidict for DTML (ignored) The arguments may be provided as keyword arguments. Some of these arguments may be ommitted, however, the policy may reject access in some cases when arguments are ommitted. It is best to provide all the values possible. """ policy = self._policy return policy.validate(accessed, container, name, value, self._context) def checkPermission(self, permission, object): """Check whether the security context allows the given permission on the given object. Arguments: permission -- A permission name object -- The object being accessed according to the permission """ policy = self._policy return policy.checkPermission(permission, object, self._context) def addContext(self, anExecutableObject, getattr=getattr): """Add an ExecutableObject to the current security context. Optionally, add a new SecurityPolicy as well. """ stack = self._context.stack if len(stack) > max_stack_size: raise SystemError, 'Excessive recursion' stack.append(anExecutableObject) p = getattr(anExecutableObject, '_customSecurityPolicy', None) if p is not None: p = p() else: p = _defaultPolicy self._policy = p def removeContext(self, anExecutableObject): """Remove an ExecutableObject, and optionally, a SecurityPolicy, from the current security context. """ stack = self._context.stack if not stack: return top = stack[-1] if top is anExecutableObject: del stack[-1] else: indexes = range(len(stack)) indexes.reverse() for i in indexes: top = stack[i] if top is anExecutableObject: del stack[i:] break else: return if stack: top = stack[-1] p = getattr(top, '_customSecurityPolicy', None) if p is not None: p = p() else: p = _defaultPolicy self._policy = p else: self._policy = _defaultPolicy def getUser(self): """Get the current authenticated user""" return self._context.user def calledByExecutable(self): """Return a boolean value indicating if this context was called by an executable""" return len(self._context.stack) # AccessControl.ZopeGuards # ------------------------ def aq_validate(inst, object, name, v, validate): return validate(inst, object, name, v) _marker = object() def guarded_getattr(inst, name, default=_marker): """Retrieves an attribute, checking security in the process. Raises Unauthorized if the attribute is found but the user is not allowed to access the attribute. """ if name[:1] == '_': raise Unauthorized, name # Try to get the attribute normally so that unusual # exceptions are caught early. try: v = getattr(inst, name) except AttributeError: if default is not _marker: return default raise try: container = v.im_self except AttributeError: container = aq_parent(aq_inner(v)) or inst assertion = Containers(type(container)) if isinstance(assertion, dict): # We got a table that lets us reason about individual # attrs assertion = assertion.get(name) if assertion: # There's an entry, but it may be a function. if callable(assertion): return assertion(inst, name) # Nope, it's boolean return v raise Unauthorized, name if assertion: if callable(assertion): factory = assertion(name, v) if callable(factory): return factory(inst, name) assert factory == 1 else: assert assertion == 1 return v # See if we can get the value doing a filtered acquire. # aq_acquire will either return the same value as held by # v or it will return an Unauthorized raised by validate. validate = SecurityManagement.getSecurityManager().validate aq_acquire(inst, name, aq_validate, validate) return v # Helpers for verbose authorization exceptions # -------------------------------------------- def item_repr(ob): """Generates a repr without angle brackets (to avoid HTML quoting)""" return repr(ob).replace('<', '(').replace('>', ')') def simplifyRoles(roles): """Sorts and removes duplicates from a role list.""" d = {} for r in roles: d[r] = 1 lst = d.keys() lst.sort() return lst def raiseVerbose(msg, accessed, container, name, value, context, required_roles=None, user_roles=None, user=None, eo=None, eo_owner=None, eo_owner_roles=None, eo_proxy_roles=None, ): """Raises an Unauthorized error with a verbose explanation.""" s = '%s. Access to %s of %s' % ( msg, repr(name), item_repr(container)) if aq_base(container) is not aq_base(accessed): s += ', acquired through %s,' % item_repr(accessed) info = [s + ' denied.'] if user is not None: try: ufolder = '/'.join(aq_parent(aq_inner(user)).getPhysicalPath()) except: ufolder = '(unknown)' info.append('Your user account, %s, exists at %s.' % ( str(user), ufolder)) if required_roles is not None: p = None required_roles = list(required_roles) for r in required_roles: if r.startswith('_') and r.endswith('_Permission'): p = r[1:] required_roles.remove(r) break sr = simplifyRoles(required_roles) if p: # got a permission name info.append('Access requires %s, ' 'granted to the following roles: %s.' % (p, sr)) else: # permission name unknown info.append('Access requires one of the following roles: %s.' % sr) if user_roles is not None: info.append( 'Your roles in this context are %s.' % simplifyRoles(user_roles)) if eo is not None: s = 'The executing script is %s' % item_repr(eo) if eo_proxy_roles is not None: s += ', with proxy roles: %s' % simplifyRoles(eo_proxy_roles) if eo_owner is not None: s += ', owned by %s' % repr(eo_owner) if eo_owner_roles is not None: s += ', who has the roles %s' % simplifyRoles(eo_owner_roles) info.append(s + '.') text = ' '.join(info) LOG('Zope Security Policy', BLATHER, 'Unauthorized: %s' % text) raise Unauthorized(text) def getUserRolesInContext(user, context): """Returns user roles for a context.""" if hasattr(aq_base(user), 'getRolesInContext'): return user.getRolesInContext(context) else: return () def userHasRolesButNotInContext(user, object, object_roles): '''Returns 1 if the user has any of the listed roles but is not defined in a context which is not an ancestor of object. ''' if object_roles is None or 'Anonymous' in object_roles: return 0 usr_roles = getUserRolesInContext(user, object) for role in object_roles: if role in usr_roles: # User has the roles. return (not verifyAcquisitionContext( user, object, object_roles)) return 0 def verifyAcquisitionContext(user, object, object_roles=None): """Mimics the relevant section of User.allowed(). Returns true if the object is in the context of the user's user folder. """ ufolder = aq_parent(user) ucontext = aq_parent(ufolder) if ucontext is not None: if object is None: # This is a strange rule, though # it doesn't cause any security holes. SDH return 1 if not hasattr(object, 'aq_inContextOf'): if hasattr(object, 'im_self'): # This is a method. Grab its self. object=object.im_self if not hasattr(object, 'aq_inContextOf'): # object is not wrapped, therefore we # can't determine context. # Fail the access attempt. Otherwise # this would be a security hole. return None if not object.aq_inContextOf(ucontext, 1): if 'Shared' in object_roles: # Old role setting. Waaa object_roles=user._shared_roles(object) if 'Anonymous' in object_roles: return 1 return None # Note that if the user were not wrapped, it would # not be possible to determine the user's context # and this method would return 1. # However, as long as user folders always return # wrapped user objects, this is safe. return 1 Zope-2.8.7-final/lib/python/AccessControl/interfaces.py0000664000076400007640000001135010435500321022057 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Access control interfaces """ from AccessControl.SimpleObjectPolicies import _noroles from zope.interface import Interface from zope.interface import Attribute class ISecurityPolicy(Interface): """Plug-in policy for checking access to objects within untrusted code. """ def validate(accessed, container, name, value, context, roles=_noroles): """Check that the current user (from context) has access. o Raise Unauthorized if access is not allowed; otherwise, return a true value. Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. context -- the security context (normally supplied by the security manager). roles -- The roles of the object if already known. """ def checkPermission(permission, object, context): """Check whether the current user has a permission w.r.t. an object. """ class ISecurityManager(Interface): """Check access and manages executable context and policies. """ _policy = Attribute(u'Current Security Policy') def validate(accessed=None, container=None, name=None, value=None, roles=_noroles, ): """Validate access. Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. roles -- The roles of the object if already known. The arguments may be provided as keyword arguments. Some of these arguments may be ommitted, however, the policy may reject access in some cases when arguments are ommitted. It is best to provide all the values possible. """ def DTMLValidate(accessed=None, container=None, name=None, value=None, md=None, ): """Validate access. * THIS EXISTS FOR DTML COMPATIBILITY * Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. md -- multidict for DTML (ignored) The arguments may be provided as keyword arguments. Some of these arguments may be ommitted, however, the policy may reject access in some cases when arguments are ommitted. It is best to provide all the values possible. """ def checkPermission(permission, object): """Check whether the security context allows the given permission on the given object. Arguments: permission -- A permission name object -- The object being accessed according to the permission """ def addContext(anExecutableObject): """Add an ExecutableObject to the current security context. o If it declares a custom security policy, make that policy "current"; otherwise, make the "default" security policy current. """ def removeContext(anExecutableObject): """Remove an ExecutableObject from the current security context. o Remove all objects from the top of the stack "down" to the supplied object. o If the top object on the stack declares a custom security policy, make that policy "current". o If the stack is empty, or if the top declares no custom security policy, restore the 'default" security policy as current. """ def getUser(): """Get the currently authenticated user """ def calledByExecutable(): """Return a boolean value indicating whether this context was called in the context of an by an executable (i.e., one added via 'addContext'). """ Zope-2.8.7-final/lib/python/AccessControl/SecurityInfo.py0000664000076400007640000002704410435500321022366 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """SecurityInfo objects and access control constants. SecurityInfo objects are used in class definitions to allow a declarative style of associating access control information with class attributes. More information on using SecurityInfo and guide to Zope security for developers can be found in the dev.zope.org "Declarative Security" project: http://dev.zope.org/Wikis/DevSite/Projects/DeclarativeSecurity While SecurityInfo objects largely remove the need for Python programmers to care about the underlying implementation, there are several constants defined that should be used by code that must set __roles__ attributes directly. (the constants are also accessible from the AccessControl namespace). The defined access control constants and their meanings are: ACCESS_PUBLIC: accessible from restricted code and possibly through the web (if object has a docstring) ACCESS_PRIVATE: accessible only from python code ACCESS_NONE: no access """ import sys import Acquisition from AccessControl.ImplPython import _what_not_even_god_should_do from zLOG import LOG, WARNING # Security constants - these are imported into the AccessControl # namespace and can be referenced as AccessControl.PUBLIC etc. ACCESS_NONE = _what_not_even_god_should_do ACCESS_PRIVATE = () ACCESS_PUBLIC = None _marker = [] class SecurityInfo(Acquisition.Implicit): """Encapsulate security information.""" __security_info__ = 1 __roles__ = ACCESS_PRIVATE def __init__(self): self.names = {} self.roles = {} def _setaccess(self, names, access): for name in names: if self.names.get(name, access) != access: LOG('SecurityInfo', WARNING, 'Conflicting security ' 'declarations for "%s"' % name) self._warnings = 1 self.names[name] = access declarePublic__roles__=ACCESS_PRIVATE def declarePublic(self, name, *names): """Declare names to be publicly accessible.""" self._setaccess((name,) + names, ACCESS_PUBLIC) declarePrivate__roles__=ACCESS_PRIVATE def declarePrivate(self, name, *names): """Declare names to be inaccessible to restricted code.""" self._setaccess((name,) + names, ACCESS_PRIVATE) declareProtected__roles__=ACCESS_PRIVATE def declareProtected(self, permission_name, name, *names): """Declare names to be associated with a permission.""" self._setaccess((name,) + names, permission_name) declareObjectPublic__roles__=ACCESS_PRIVATE def declareObjectPublic(self): """Declare the object to be publicly accessible.""" self._setaccess(('',), ACCESS_PUBLIC) declareObjectPrivate__roles__=ACCESS_PRIVATE def declareObjectPrivate(self): """Declare the object to be inaccessible to restricted code.""" self._setaccess(('',), ACCESS_PRIVATE) declareObjectProtected__roles__=ACCESS_PRIVATE def declareObjectProtected(self, permission_name): """Declare the object to be associated with a permission.""" self._setaccess(('',), permission_name) setPermissionDefault__roles__=ACCESS_PRIVATE def setPermissionDefault(self, permission_name, roles): """Declare default roles for a permission""" rdict = {} for role in roles: rdict[role] = 1 if self.roles.get(permission_name, rdict) != rdict: LOG('SecurityInfo', WARNING, 'Conflicting default role' 'declarations for permission "%s"' % permission_name) self._warnings = 1 self.roles[permission_name] = rdict setDefaultAccess__roles__=ACCESS_PRIVATE def setDefaultAccess(self, access): """Declare default attribute access policy. This should be a boolean value, a map of attribute names to booleans, or a callable (name, value) -> boolean. """ if isinstance(access, str): access = access.lower() if access == 'allow': access = 1 elif access == 'deny': access = 0 else: raise ValueError, "'allow' or 'deny' expected" self.access = access class ClassSecurityInfo(SecurityInfo): """Encapsulate security information for class objects.""" __roles__ = ACCESS_PRIVATE apply__roles__ = ACCESS_PRIVATE def apply(self, classobj): """Apply security information to the given class object.""" dict = classobj.__dict__ # Check the class for an existing __ac_permissions__ and # incorporate that if present to support older classes or # classes that haven't fully switched to using SecurityInfo. if dict.has_key('__ac_permissions__'): for item in dict['__ac_permissions__']: permission_name = item[0] self._setaccess(item[1], permission_name) if len(item) > 2: self.setPermissionDefault(permission_name, item[2]) # Set __roles__ for attributes declared public or private. # Collect protected attribute names in ac_permissions. ac_permissions = {} for name, access in self.names.items(): if access in (ACCESS_PRIVATE, ACCESS_PUBLIC, ACCESS_NONE): setattr(classobj, '%s__roles__' % name, access) else: if not ac_permissions.has_key(access): ac_permissions[access] = [] ac_permissions[access].append(name) # Now transform our nested dict structure into the nested tuple # structure expected of __ac_permissions__ attributes and set # it on the class object. getRoles = self.roles.get __ac_permissions__ = [] permissions = ac_permissions.items() permissions.sort() for permission_name, names in permissions: roles = getRoles(permission_name, ()) if len(roles): entry = (permission_name, tuple(names), tuple(roles.keys())) else: entry = (permission_name, tuple(names)) __ac_permissions__.append(entry) for permission_name, roles in self.roles.items(): if permission_name not in ac_permissions: entry = (permission_name, (), tuple(roles.keys())) __ac_permissions__.append(entry) setattr(classobj, '__ac_permissions__', tuple(__ac_permissions__)) # Take care of default attribute access policy access = getattr(self, 'access', _marker) if access is not _marker: setattr(classobj, '__allow_access_to_unprotected_subobjects__', access) if getattr(self, '_warnings', None): LOG('SecurityInfo', WARNING, 'Class "%s" had conflicting ' 'security declarations' % classobj.__name__) class ClassSecurityInformation(ClassSecurityInfo): # Default policy is disallow access = 0 _moduleSecurity = {} _appliedModuleSecurity = {} def secureModule(mname, *imp): modsec = _moduleSecurity.get(mname, None) if modsec is None: return del _moduleSecurity[mname] if imp: __import__(mname, *imp) module = sys.modules[mname] modsec.apply(module.__dict__) _appliedModuleSecurity[mname] = modsec return module def ModuleSecurityInfo(module_name=None): if module_name is not None: modsec = _moduleSecurity.get(module_name, None) if modsec is not None: return modsec modsec = _appliedModuleSecurity.get(module_name, None) if modsec is not None: # Move security info back to to-apply dict (needed for product # refresh). Also invoke this check for parent packages already # applied del _appliedModuleSecurity[module_name] _moduleSecurity[module_name] = modsec dot = module_name.rfind('.') if dot > 0: ModuleSecurityInfo(module_name[:dot]) return modsec dot = module_name.rfind('.') if dot > 0: # If the module is in a package, recursively make sure # there are security declarations for the package steps # leading to the module modname = module_name[dot + 1:] pmodsec = ModuleSecurityInfo(module_name[:dot]) if not pmodsec.names.has_key(modname): pmodsec.declarePublic(modname) return _ModuleSecurityInfo(module_name) class _ModuleSecurityInfo(SecurityInfo): """Encapsulate security information for modules.""" __roles__ = ACCESS_PRIVATE def __init__(self, module_name=None): self.names = {} if module_name is not None: global _moduleSecurity _moduleSecurity[module_name] = self __call____roles__ = ACCESS_PRIVATE def __call__(self, name, value): """Callback for __allow_access_to_unprotected_subobjects__ hook.""" access = self.names.get(name, _marker) if access is not _marker: return access == ACCESS_PUBLIC return getattr(self, 'access', 0) apply__roles__ = ACCESS_PRIVATE def apply(self, dict): """Apply security information to the given module dict.""" # Start with default attribute access policy access = getattr(self, 'access', _marker) if access is not _marker or len(self.names): dict['__allow_access_to_unprotected_subobjects__'] = self if getattr(self, '_warnings', None): LOG('SecurityInfo', WARNING, 'Module "%s" had conflicting ' 'security declarations' % dict['__name__']) declareProtected__roles__=ACCESS_PRIVATE def declareProtected(self, permission_name, *names): """Cannot declare module names protected.""" pass declareObjectProtected__roles__=ACCESS_PRIVATE def declareObjectProtected(self, permission_name): """Cannot declare module protected.""" pass setDefaultRoles__roles__=ACCESS_PRIVATE def setDefaultRoles(self, permission_name, roles): """Cannot set default roles for permissions in a module.""" pass # Handy little utility functions def allow_module(module_name): """Allow a module and all its contents to be used from a restricted Script. The argument module_name may be a simple or dotted module or package name. Note that if a package path is given, all modules in the path will be available.""" ModuleSecurityInfo(module_name).setDefaultAccess(1) dot = module_name.find('.') while dot > 0: ModuleSecurityInfo(module_name[:dot]).setDefaultAccess(1) dot = module_name.find('.', dot + 1) def allow_class(Class): """Allow a class and all of its methods to be used from a restricted Script. The argument Class must be a class.""" Class._security = sec = ClassSecurityInfo() sec.declareObjectPublic() sec.setDefaultAccess(1) sec.apply(Class) from Globals import InitializeClass InitializeClass(Class) Zope-2.8.7-final/lib/python/AccessControl/unauthorized.py0000664000076400007640000000222710435500321022460 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Access control exceptions """ import zExceptions class Unauthorized(zExceptions.Unauthorized): def getValueName(self): v=self.value n=getattr(v, 'getId', v) if n is v: n=getattr(v, 'id', v) if n is v: n=getattr(v, '__name__', v) if n is not v: if callable(n): try: n = n() except TypeError: pass return n c = getattr(v, '__class__', type(v)) c = getattr(c, '__name__', 'object') return "a particular %s" % c Zope-2.8.7-final/lib/python/AccessControl/IUserFolder.py0000664000076400007640000000074610435500321022126 0ustar ajungajungclass IStandardUserFolder: def getUser(self, name): """ Returns the user object specified by name. If there is no user named 'name' in the user folder, return None. """ def getUsers(self): """ Returns a sequence of all user objects which reside in the user folder. """ def getUserNames(self): """ Returns a sequence of names of the users which reside in the user folder. """ Zope-2.8.7-final/lib/python/AccessControl/AuthEncoding.py0000664000076400007640000001171110435500321022305 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## __version__='$Revision: 1.9 $'[11:-2] import sha, binascii from binascii import b2a_base64, a2b_base64 from random import choice, randrange class PasswordEncryptionScheme: # An Interface def encrypt(pw): """ Encrypt the provided plain text password. """ def validate(reference, attempt): """ Validate the provided password string. Reference is the correct password, which may be encrypted; attempt is clear text password attempt. """ _schemes = [] def registerScheme(id, s): ''' Registers an LDAP password encoding scheme. ''' _schemes.append((id, '{%s}' % id, s)) def listSchemes(): r = [] for id, prefix, scheme in _schemes: r.append(id) return r class SSHADigestScheme: ''' SSHA is a modification of the SHA digest scheme with a salt starting at byte 20 of the base64-encoded string. ''' # Source: http://developer.netscape.com/docs/technote/ldap/pass_sha.html def generate_salt(self): # Salt can be any length, but not more than about 37 characters # because of limitations of the binascii module. # 7 is what Netscape's example used and should be enough. # All 256 characters are available. salt = '' for n in range(7): salt += chr(randrange(256)) return salt def encrypt(self, pw): pw = str(pw) salt = self.generate_salt() return b2a_base64(sha.new(pw + salt).digest() + salt)[:-1] def validate(self, reference, attempt): try: ref = a2b_base64(reference) except binascii.Error: # Not valid base64. return 0 salt = ref[20:] compare = b2a_base64(sha.new(attempt + salt).digest() + salt)[:-1] return (compare == reference) registerScheme('SSHA', SSHADigestScheme()) class SHADigestScheme: def encrypt(self, pw): return b2a_base64(sha.new(pw).digest())[:-1] def validate(self, reference, attempt): compare = b2a_base64(sha.new(attempt).digest())[:-1] return (compare == reference) registerScheme('SHA', SHADigestScheme()) # Bogosity on various platforms due to ITAR restrictions try: from crypt import crypt except ImportError: crypt = None if crypt is not None: class CryptDigestScheme: def generate_salt(self): choices = ("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789./") return choice(choices) + choice(choices) def encrypt(self, pw): return crypt(pw, self.generate_salt()) def validate(self, reference, attempt): a = crypt(attempt, reference[:2]) return (a == reference) registerScheme('CRYPT', CryptDigestScheme()) class MySQLDigestScheme: def encrypt(self, pw): nr = 1345345333L add = 7 nr2 = 0x12345671L for i in pw: if i == ' ' or i == '\t': continue nr ^= (((nr & 63) + add) * ord(i)) + (nr << 8) nr2 += (nr2 << 8) ^ nr add += ord(i) r0 = nr & ((1L << 31) - 1L) r1 = nr2 & ((1L << 31) - 1L) return "%08lx%08lx" % (r0, r1) def validate(self, reference, attempt): a = self.encrypt(attempt) return (a == reference) registerScheme('MYSQL', MySQLDigestScheme()) def pw_validate(reference, attempt): """Validate the provided password string, which uses LDAP-style encoding notation. Reference is the correct password, attempt is clear text password attempt.""" for id, prefix, scheme in _schemes: lp = len(prefix) if reference[:lp] == prefix: return scheme.validate(reference[lp:], attempt) # Assume cleartext. return (reference == attempt) def is_encrypted(pw): for id, prefix, scheme in _schemes: lp = len(prefix) if pw[:lp] == prefix: return 1 return 0 def pw_encrypt(pw, encoding='SSHA'): """Encrypt the provided plain text password using the encoding if provided and return it in an LDAP-style representation.""" for id, prefix, scheme in _schemes: if encoding == id: return prefix + scheme.encrypt(pw) raise ValueError, 'Not supported: %s' % encoding pw_encode = pw_encrypt # backward compatibility Zope-2.8.7-final/lib/python/AccessControl/ImplC.py0000664000076400007640000000335310435500321020744 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """C implementation of the access control machinery.""" try: from cAccessControl import rolesForPermissionOn, \ PermissionRole, imPermissionRole, _what_not_even_god_should_do, \ RestrictedDTMLMixin, aq_validate, guarded_getattr, \ setDefaultBehaviors from cAccessControl import ZopeSecurityPolicy as cZopeSecurityPolicy from cAccessControl import SecurityManager as cSecurityManager except ImportError: import sys # make sure a partial import doesn't pollute sys.modules del sys.modules[__name__] from ImplPython import RestrictedDTML, SecurityManager, ZopeSecurityPolicy class RestrictedDTML(RestrictedDTMLMixin, RestrictedDTML): """A mix-in for derivatives of DT_String.String that adds Zope security.""" class ZopeSecurityPolicy(cZopeSecurityPolicy, ZopeSecurityPolicy): """A security manager provides methods for checking access and managing executable context and policies """ class SecurityManager(cSecurityManager, SecurityManager): """A security manager provides methods for checking access and managing executable context and policies """ Zope-2.8.7-final/lib/python/AccessControl/Permission.py0000664000076400007640000001137510435500321022073 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Permissions $Id: Permission.py 27470 2004-09-08 07:16:06Z andreasjung $ """ import string, Products, Globals name_trans=filter(lambda c, an=string.letters+string.digits+'_': c not in an, map(chr,range(256))) name_trans=string.maketrans(''.join(name_trans), '_'*len(name_trans)) def pname(name, translate=string.translate, name_trans=name_trans): return '_'+translate(name,name_trans)+"_Permission" _marker=[] class Permission: # A Permission maps a named logical permission to a set # of attribute names. Attribute names which appear in a # permission may not appear in any other permission defined # by the object. def __init__(self,name,data,obj,default=None): self.name=name self._p='_'+string.translate(name,name_trans)+"_Permission" self.data=data if hasattr(obj, 'aq_base'): obj=obj.aq_base self.obj=obj self.default=default def getRoles(self, default=_marker): # Return the list of role names which have been given # this permission for the object in question. To do # this, we try to get __roles__ from all of the object # attributes that this permission represents. obj=self.obj name=self._p if hasattr(obj, name): return getattr(obj, name) roles=default for name in self.data: if name: if hasattr(obj, name): attr=getattr(obj, name) if hasattr(attr,'im_self'): attr=attr.im_self if hasattr(attr, '__dict__'): attr=attr.__dict__ name=name+'__roles__' if attr.has_key(name): roles=attr[name] break elif hasattr(obj, '__dict__'): attr=obj.__dict__ if attr.has_key('__roles__'): roles=attr['__roles__'] break if roles: try: if 'Shared' not in roles: return tuple(roles) roles=list(roles) roles.remove('Shared') return roles except: return [] if roles is None: return ['Manager','Anonymous'] if roles is _marker: return ['Manager'] return roles def setRoles(self, roles): obj=self.obj if isinstance(roles, list) and not roles: if hasattr(obj, self._p): delattr(obj, self._p) else: setattr(obj, self._p, roles) for name in self.data: if name=='': attr=obj else: attr=getattr(obj, name) try: del attr.__roles__ except: pass try: delattr(obj,name+'__roles__') except: pass def setRole(self, role, present): roles=self.getRoles() if role in roles: if present: return if isinstance(roles, list): roles.remove(role) else: roles=list(roles) roles.remove(role) roles=tuple(roles) elif not present: return else: if isinstance(roles, list): roles.append(role) else: roles=roles+(role,) self.setRoles(roles) def __len__(self): return 1 def __str__(self): return self.name _registeredPermissions={} _registerdPermission=_registeredPermissions.has_key def registerPermissions(permissions, defaultDefault=('Manager',)): """Register an __ac_permissions__ sequence. """ for setting in permissions: if _registerdPermission(setting[0]): continue if len(setting)==2: perm, methods = setting default = defaultDefault else: perm, methods, default = setting _registeredPermissions[perm]=1 Products.__ac_permissions__=( Products.__ac_permissions__+((perm,(),default),)) mangled=pname(perm) # get mangled permission name if not hasattr(Globals.ApplicationDefaultPermissions, mangled): setattr(Globals.ApplicationDefaultPermissions, mangled, default) Zope-2.8.7-final/lib/python/AccessControl/Setup0000664000076400007640000000011710435500321020404 0ustar ajungajung*shared* cAccessControl cAccessControl.c -I../../Components/ExtensionClass/src Zope-2.8.7-final/lib/python/AccessControl/SimpleObjectPolicies.py0000664000076400007640000001207310435500321024007 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## '''Collect rules for access to objects that don\'t have roles. The rules are expressed as a mapping from type -> assertion An assertion can be: - A dict - A callable - Something with a truth value If the assertion is a callable, then it will be called with a name being accessed and the name used. Its return value is ignored, but in may veto an access by raising an exception. If the assertion is a dictionary, then the keys are attribute names. The values may be callables or objects with boolean values. If a value is callable, it will be called with the object we are accessing an attribute of and the attribute name. It should return an attribute value. Callables are often used to returned guarded versions of methods. Otherwise, accesses are allowed if values in this dictionary are true and disallowed if the values are false or if an item for an attribute name is not present. If the assertion is not a dict and is not callable, then access to unprotected attributes is allowed if the assertion is true, and disallowed otherwise. XXX This descrition doesn't actually match what's done in ZopeGuards or in ZopeSecurityPolicy. :( $Id: SimpleObjectPolicies.py 24763 2004-05-17 05:59:28Z philikon $''' _noroles = [] # this is imported in various places import Record # Allow access to unprotected attributes Record.Record.__allow_access_to_unprotected_subobjects__=1 # ContainerAssertions are used by cAccessControl to check access to # attributes of container types, like dict, list, or string. # ContainerAssertions maps types to a either a dict, a function, or a # simple boolean value. When guarded_getattr checks the type of its # first argument against ContainerAssertions, and invokes checking # logic depending on what value it finds. # If the value for a type is: # - a boolean value: # - the value determines whether access is allowed # - a function (or callable): # - The function is called with the name of the attribute and # the actual attribute value, then the value is returned. # The function can raise an exception. # - a dict: # - The dict maps attribute names to boolean values or functions. # The boolean values behave as above, but the functions do not. # The value returned for attribute access is the result of # calling the function with the object and the attribute name. ContainerAssertions={ type(()): 1, type(''): 1, type(u''): 1, } class _dummy_class: pass from DocumentTemplate.DT_Util import TemplateDict # Temporarily create a DictInstance so that we can mark its type as # being a key in the ContainerAssertions. templateDict = TemplateDict() try: dictInstance = templateDict(dummy=1)[0] if type(dictInstance) is not type(_dummy_class()): ContainerAssertions[type(dictInstance)]=1 except: # Hmm, this may cause _() and _.namespace() to fail. # What to do? pass Containers = ContainerAssertions.get def allow_type(Type, allowed=1): """Allow a type and all of its methods and attributes to be used from restricted code. The argument Type must be a type.""" if type(Type) is not type: raise ValueError, "%s is not a type" % `Type` if hasattr(Type, '__roles__'): raise ValueError, "%s handles its own security" % `Type` if not (isinstance(allowed, int) or isinstance(allowed, dict)): raise ValueError, "The 'allowed' argument must be an int or dict." ContainerAssertions[Type] = allowed # # WAAAA! # from BTrees.OOBTree import OOBTree, OOBucket, OOSet from BTrees.OIBTree import OIBTree, OIBucket, OISet from BTrees.IOBTree import IOBTree, IOBucket, IOSet from BTrees.IIBTree import IIBTree, IIBucket, IISet for tree_type, has_values in [(OOBTree, 1), (OOBucket, 1), (OOSet, 0), (OIBTree, 1), (OIBucket, 1), (OISet, 0), (IOBTree, 1), (IOBucket, 1), (IOSet, 0), (IIBTree, 1), (IIBucket, 1), (IISet, 0), ]: tree = tree_type() key_type = type(tree.keys()) if key_type is not list: # lists have their own declarations allow_type(key_type) if has_values: assert key_type is type(tree.values()) assert key_type is type(tree.items()) Zope-2.8.7-final/lib/python/AccessControl/tests/0000775000076400007640000000000010435610645020537 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/testSecurityManager.py0000664000076400007640000002315310435500320025103 0ustar ajungajung############################################################################## # # Copyright (c) 2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for the SecurityManager implementations $Id: testSecurityManager.py 40418 2005-11-30 00:53:34Z tseaver $ """ import unittest _THREAD_ID = 123 class DummyContext: def __init__(self): self.user = object() self.stack = [] class DummyPolicy: CHECK_PERMISSION_ARGS = None CHECK_PERMISSION_RESULT = object() VALIDATE_ARGS = None def checkPermission(self, *args): self.CHECK_PERMISSION_ARGS = args return self.CHECK_PERMISSION_RESULT def validate(self, *args): self.VALIDATE_ARGS = args return True class ExecutableObject: def __init__(self, new_policy): self._new_policy = new_policy def _customSecurityPolicy(self): return self._new_policy class ISecurityManagerConformance: def test_conforms_to_ISecurityManager(self): from AccessControl.interfaces import ISecurityManager from zope.interface.verify import verifyClass verifyClass(ISecurityManager, self._getTargetClass()) class SecurityManagerTestBase(unittest.TestCase): def _makeOne(self, thread_id, context): return self._getTargetClass()(thread_id, context) def test_getUser(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) self.failUnless(mgr.getUser() is context.user) def test_calledByExecutable_no_stack(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) self.failIf(mgr.calledByExecutable()) def test_calledByExecutable_with_stack(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) executableObject = object() mgr.addContext(executableObject) self.failUnless(mgr.calledByExecutable()) def test_addContext_no_custom_policy(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) original_policy = mgr._policy executableObject = object() mgr.addContext(executableObject) self.failUnless(mgr._policy is original_policy) def test_addContext_with_custom_policy(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) new_policy = DummyPolicy() executableObject = ExecutableObject(new_policy) mgr.addContext(executableObject) self.failUnless(mgr._policy is new_policy) def test_addContext_with_custom_policy_then_none(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) original_policy = mgr._policy new_policy = DummyPolicy() executableObject = ExecutableObject(new_policy) mgr.addContext(executableObject) mgr.addContext(object()) self.failUnless(mgr._policy is original_policy) def test_removeContext_pops_items_above_EO(self): context = DummyContext() ALPHA, BETA, GAMMA, DELTA = object(), object(), object(), object() context.stack.append(ALPHA) context.stack.append(BETA) context.stack.append(GAMMA) context.stack.append(DELTA) mgr = self._makeOne(_THREAD_ID, context) mgr.removeContext(GAMMA) self.assertEqual(len(context.stack), 2) self.failUnless(context.stack[0] is ALPHA) self.failUnless(context.stack[1] is BETA) def test_removeContext_last_EO_restores_default_policy(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) original_policy = mgr._policy new_policy = mgr._policy = DummyPolicy() top = object() context.stack.append(top) mgr.removeContext(top) self.failUnless(mgr._policy is original_policy) def test_removeContext_with_top_having_custom_policy(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) new_policy = DummyPolicy() context.stack.append(ExecutableObject(new_policy)) top = object() context.stack.append(top) mgr.removeContext(top) self.failUnless(mgr._policy is new_policy) def test_removeContext_with_top_having_no_custom_policy(self): context = DummyContext() mgr = self._makeOne(_THREAD_ID, context) original_policy = mgr._policy new_policy = DummyPolicy() executableObject = ExecutableObject(new_policy) context.stack.append(executableObject) top = object() context.stack.append(top) mgr.removeContext(executableObject) self.failUnless(mgr._policy is original_policy) def test_checkPermission_delegates_to_policy(self): context = DummyContext() PERMISSION = 'PERMISSION' TARGET = object() mgr = self._makeOne(_THREAD_ID, context) new_policy = mgr._policy = DummyPolicy() result = mgr.checkPermission(PERMISSION, TARGET) self.failUnless(result is DummyPolicy.CHECK_PERMISSION_RESULT) self.failUnless(new_policy.CHECK_PERMISSION_ARGS[0] is PERMISSION) self.failUnless(new_policy.CHECK_PERMISSION_ARGS[1] is TARGET) self.failUnless(new_policy.CHECK_PERMISSION_ARGS[2] is context) def test_validate_without_roles_delegates_to_policy(self): from AccessControl.SimpleObjectPolicies import _noroles context = DummyContext() ACCESSED = object() CONTAINER = object() NAME = 'NAME' VALUE = object() mgr = self._makeOne(_THREAD_ID, context) new_policy = mgr._policy = DummyPolicy() result = mgr.validate(ACCESSED, CONTAINER, NAME, VALUE, ) self.failUnless(result) self.assertEqual(len(new_policy.VALIDATE_ARGS), 5) self.failUnless(new_policy.VALIDATE_ARGS[0] is ACCESSED) self.failUnless(new_policy.VALIDATE_ARGS[1] is CONTAINER) self.assertEqual(new_policy.VALIDATE_ARGS[2], NAME) self.failUnless(new_policy.VALIDATE_ARGS[3] is VALUE) self.failUnless(new_policy.VALIDATE_ARGS[4] is context) def test_validate_with_roles_delegates_to_policy(self): from AccessControl.SimpleObjectPolicies import _noroles context = DummyContext() ACCESSED = object() CONTAINER = object() NAME = 'NAME' VALUE = object() ROLES = ('Hamlet', 'Othello') mgr = self._makeOne(_THREAD_ID, context) new_policy = mgr._policy = DummyPolicy() result = mgr.validate(ACCESSED, CONTAINER, NAME, VALUE, ROLES, ) self.failUnless(result) self.assertEqual(len(new_policy.VALIDATE_ARGS), 6) self.failUnless(new_policy.VALIDATE_ARGS[0] is ACCESSED) self.failUnless(new_policy.VALIDATE_ARGS[1] is CONTAINER) self.assertEqual(new_policy.VALIDATE_ARGS[2], NAME) self.failUnless(new_policy.VALIDATE_ARGS[3] is VALUE) self.failUnless(new_policy.VALIDATE_ARGS[4] is context) self.assertEqual(new_policy.VALIDATE_ARGS[5], ROLES) def test_DTMLValidate_delegates_to_policy_validate(self): from AccessControl.SimpleObjectPolicies import _noroles context = DummyContext() ACCESSED = object() CONTAINER = object() NAME = 'NAME' VALUE = object() MD = {} mgr = self._makeOne(_THREAD_ID, context) new_policy = mgr._policy = DummyPolicy() result = mgr.DTMLValidate(ACCESSED, CONTAINER, NAME, VALUE, MD, ) self.failUnless(result) self.assertEqual(len(new_policy.VALIDATE_ARGS), 5) self.failUnless(new_policy.VALIDATE_ARGS[0] is ACCESSED) self.failUnless(new_policy.VALIDATE_ARGS[1] is CONTAINER) self.assertEqual(new_policy.VALIDATE_ARGS[2], NAME) self.failUnless(new_policy.VALIDATE_ARGS[3] is VALUE) self.failUnless(new_policy.VALIDATE_ARGS[4] is context) class PythonSecurityManagerTests(SecurityManagerTestBase, ISecurityManagerConformance, ): def _getTargetClass(self): from AccessControl.ImplPython import SecurityManager return SecurityManager # N.B.: The C version mixes in the Python version, which is why we # can test for conformance to ISecurityManager. class C_SecurityManagerTests(SecurityManagerTestBase, ISecurityManagerConformance, ): def _getTargetClass(self): from AccessControl.ImplC import SecurityManager return SecurityManager def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( PythonSecurityManagerTests ) ) suite.addTest( unittest.makeSuite( C_SecurityManagerTests ) ) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/AccessControl/tests/testZopeSecurityPolicy.py0000664000076400007640000004451210435500320025630 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests of ZopeSecurityPolicy """ __rcs_id__='$Id: testZopeSecurityPolicy.py 40459 2005-12-01 22:44:07Z tseaver $' __version__='$Revision: 1.10 $'[11:-2] import os, sys, unittest import ZODB try: from zExceptions import Unauthorized except ImportError: Unauthorized = 'Unauthorized' from AccessControl.User import UserFolder from AccessControl.SecurityManagement import SecurityContext from Acquisition import Implicit, Explicit, aq_base from MethodObject import Method from ComputedAttribute import ComputedAttribute user_roles = ('RoleOfUser',) eo_roles = ('RoleOfExecutableOwner',) sysadmin_roles = ('RoleOfSysAdmin',) class App(Explicit): def unrestrictedTraverse(self, path): ob = self for el in path: ob = getattr(ob, el) return ob class PublicMethod (Method): def getOwner(self): return None def __call__(*args, **kw): return args, kw def getWrappedOwner(self): return None __roles__ = None class ProtectedMethod (PublicMethod): __roles__ = user_roles class OwnedMethod (PublicMethod): __roles__ = eo_roles def getOwner(self): return self.aq_parent.aq_parent.acl_users.getUserById('theowner') def getWrappedOwner(self): acl_users = self.aq_parent.aq_parent.acl_users user = acl_users.getUserById('theowner') return user.__of__(acl_users) class setuidMethod (PublicMethod): _proxy_roles = sysadmin_roles class OwnedSetuidMethod(Implicit): __roles__ = eo_roles _proxy_roles = sysadmin_roles def getOwner(self, info=0): if info: return (('subobject', 'acl_users'), 'theowner') else: return self.aq_parent.aq_parent.acl_users.getUserById('theowner') def getWrappedOwner(self): acl_users = self.aq_parent.aq_parent.acl_users user = acl_users.getUserById('theowner') return user.__of__(acl_users) class DangerousMethod (PublicMethod): # Only accessible to sysadmin or people who use proxy roles __roles__ = sysadmin_roles class SimpleItemish (Implicit): public_m = PublicMethod() protected_m = ProtectedMethod() owned_m = OwnedMethod() setuid_m = setuidMethod() dangerous_m = DangerousMethod() public_prop = 'Public Value' private_prop = 'Private Value' class ImplictAcqObject(Implicit): pass class UnprotectedSimpleItem (SimpleItemish): __allow_access_to_unprotected_subobjects__ = 1 class OwnedSimpleItem(UnprotectedSimpleItem): def getOwner(self, info=0): if info: return (('subobject', 'acl_users'), 'theowner') else: return self.aq_parent.acl_users.getuserById('theowner') class RestrictedSimpleItem (SimpleItemish): __allow_access_to_unprotected_subobjects__ = 0 _Foo_Permission = user_roles + eo_roles _Kill_Permission = sysadmin_roles _View_Permission = eo_roles class PartlyProtectedSimpleItem1 (SimpleItemish): __allow_access_to_unprotected_subobjects__ = {'public_prop': 1,} class PartlyProtectedSimpleItem2 (SimpleItemish): def __allow_access_to_unprotected_subobjects__(self, name, value): if name == 'public_prop': return 1 return 0 class PartlyProtectedSimpleItem3 (PartlyProtectedSimpleItem1): # Set the roles of objects that are accessible because of # __allow_access_to_unprotected_subobjects__ . __roles__ = sysadmin_roles class SimpleClass: attr = 1 class ZopeSecurityPolicyTestBase(unittest.TestCase): def setUp(self): a = App() self.a = a a.item = UnprotectedSimpleItem() self.item = a.item a.r_item = RestrictedSimpleItem() a.item1 = PartlyProtectedSimpleItem1() a.item2 = PartlyProtectedSimpleItem2() a.item3 = PartlyProtectedSimpleItem3() uf = UserFolder() a.acl_users = uf self.uf = a.acl_users uf._addUser('joe', 'password', 'password', user_roles, ()) uf._addUser('theowner', 'password', 'password', eo_roles, ()) user = uf.getUserById('joe') self.user = user context = SecurityContext(user) self.context = context self.policy = self._makeOne() def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def assertPolicyAllows(self, ob, attrname): res = self.policy.validate(ob, ob, attrname, getattr(ob, attrname), self.context) if not res: self.fail('Policy quietly denied %s' % attrname) def assertPolicyDenies(self, ob, attrname): try: res = self.policy.validate(ob, ob, attrname, getattr(ob, attrname), self.context) except Unauthorized: # Passed the test. pass else: if res: self.fail('Policy quietly allowed %s' % attrname) else: self.fail('Policy denied %s, but did not ' 'throw an exception.' % attrname) def testUserAccess(self): item = self.item self.assertPolicyAllows(item, 'public_m') self.assertPolicyAllows(item, 'protected_m') self.assertPolicyDenies(item, 'owned_m') self.assertPolicyAllows(item, 'setuid_m') self.assertPolicyDenies(item, 'dangerous_m') def testOwnerAccess(self): self.context = SecurityContext(self.uf.getUserById('theowner')) item = self.item self.assertPolicyAllows(item, 'public_m') self.assertPolicyDenies(item, 'protected_m') self.assertPolicyAllows(item, 'owned_m') self.assertPolicyAllows(item, 'setuid_m') self.assertPolicyDenies(item, 'dangerous_m') def testProxyAccess(self): item = self.item self.context.stack.append(item.setuid_m) self.assertPolicyAllows(item, 'public_m') self.assertPolicyDenies(item, 'protected_m') self.assertPolicyDenies(item, 'owned_m') self.assertPolicyAllows(item, 'setuid_m') self.assertPolicyAllows(item, 'dangerous_m') def testIdentityProxy(self): eo = ImplictAcqObject() eo.getOwner = lambda: None self.context.stack.append(eo) rc = sys.getrefcount(eo) self.testUserAccess() self.assertEqual(rc, sys.getrefcount(eo)) eo._proxy_roles = () self.testUserAccess() self.assertEqual(rc, sys.getrefcount(eo)) def testAccessToUnprotectedSubobjects(self): item = self.item r_item = self.a.r_item item1 = self.a.item1 item2 = self.a.item2 item3 = self.a.item3 self.assertPolicyAllows(item, 'public_prop') self.assertPolicyDenies(r_item,'public_prop') self.assertPolicyAllows(item1, 'public_prop') self.assertPolicyAllows(item2, 'public_prop') self.assertPolicyDenies(item3,'public_prop') self.assertPolicyAllows(item, 'private_prop') self.assertPolicyDenies(r_item,'private_prop') self.assertPolicyDenies(item1, 'private_prop') self.assertPolicyDenies(item2, 'private_prop') self.assertPolicyDenies(item3, 'private_prop') def testAccessToSimpleContainer(self): self.assertPolicyAllows({}, 'keys') self.assertPolicyAllows([], 'append') self.assertPolicyDenies(SimpleClass, 'attr') self.assertPolicyDenies(SimpleClass(), 'attr') c = SimpleClass() c.attr = PublicMethod() self.assertPolicyAllows(c, 'attr') def testUnicodeAttributeLookups(self): item = self.item r_item = self.a.r_item self.assertPolicyAllows(item, u'public_prop') self.assertPolicyDenies(r_item, u'private_prop') self.assertPolicyAllows(item, u'public_m') self.assertPolicyDenies(item, u'dangerous_m') def testRolesForPermission(self): # Test of policy.checkPermission(). r_item = self.a.r_item context = self.context v = self.policy.checkPermission('View', r_item, context) self.assert_(not v, '_View_Permission should deny access to user') o_context = SecurityContext(self.uf.getUserById('theowner')) v = self.policy.checkPermission('View', r_item, o_context) self.assert_(v, '_View_Permission should grant access to theowner') def test_checkPermission_respects_proxy_roles(self): r_item = self.a.r_item context = self.context self.failIf(self.policy.checkPermission('View', r_item, context)) o_context = SecurityContext(self.uf.getUserById('joe')) # Push an executable with proxy roles on the stack eo = OwnedSetuidMethod().__of__(r_item) eo._proxy_roles = eo_roles context.stack.append(eo) self.failUnless(self.policy.checkPermission('View', r_item, context)) def test_checkPermission_proxy_roles_limit_access(self): r_item = self.a.r_item context = self.context self.failUnless(self.policy.checkPermission('Foo', r_item, context)) o_context = SecurityContext(self.uf.getUserById('joe')) # Push an executable with proxy roles on the stack eo = OwnedSetuidMethod().__of__(r_item) eo._proxy_roles = sysadmin_roles context.stack.append(eo) self.failIf(self.policy.checkPermission('Foo', r_item, context)) def test_checkPermission_proxy_role_scope(self): self.a.subobject = ImplictAcqObject() subobject = self.a.subobject subobject.acl_users = UserFolder() subobject.acl_users._addUser('theowner', 'password', 'password', eo_roles + sysadmin_roles, ()) subobject.r_item = RestrictedSimpleItem() r_subitem = subobject.r_item r_subitem.owned_setuid_m = OwnedSetuidMethod() r_subitem.getPhysicalRoot = lambda root=self.a: root r_item = self.a.r_item r_item.getPhysicalRoot = lambda root=self.a: root context = self.context context.stack.append(r_subitem.owned_setuid_m.__of__(r_subitem)) # Out of owner context self.failIf(self.policy.checkPermission('View', r_item, context)) self.failIf(self.policy.checkPermission('Kill', r_item, context)) # Inside owner context self.failIf(self.policy.checkPermission('View', r_subitem, context)) self.failUnless(self.policy.checkPermission('Kill', r_subitem, context)) def testUnicodeRolesForPermission(self): r_item = self.a.r_item context = self.context v = self.policy.checkPermission(u'View', r_item, context) self.assert_(not v, '_View_Permission should deny access to user') o_context = SecurityContext(self.uf.getUserById('theowner')) v = self.policy.checkPermission(u'View', r_item, o_context) self.assert_(v, '_View_Permission should grant access to theowner') def testAqNames(self): policy = self.policy names = { 'aq_self': 0, 'aq_base': 0, 'aq_parent': 1, 'aq_explicit': 1, 'aq_inner': 1 } for name, allowed in names.items(): if not allowed: self.assertRaises(Unauthorized, policy.validate, '', '', name, '', None) else: policy.validate('', '', name, '', None) def testProxyRoleScope(self): self.a.subobject = ImplictAcqObject() subobject = self.a.subobject subobject.acl_users = UserFolder() subobject.acl_users._addUser('theowner', 'password', 'password', eo_roles + sysadmin_roles, ()) subobject.item = UnprotectedSimpleItem() subitem = subobject.item subitem.owned_setuid_m = OwnedSetuidMethod() subitem.getPhysicalRoot = lambda root=self.a: root item = self.a.item item.getPhysicalRoot = lambda root=self.a: root self.context.stack.append(subitem.owned_setuid_m.__of__(subitem)) # Out of owner context self.assertPolicyAllows(item, 'public_m') self.assertPolicyDenies(item, 'protected_m') self.assertPolicyDenies(item, 'owned_m') self.assertPolicyAllows(item, 'setuid_m') self.assertPolicyDenies(item, 'dangerous_m') # Inside owner context self.assertPolicyAllows(subitem, 'public_m') self.assertPolicyDenies(subitem, 'protected_m') self.assertPolicyDenies(subitem, 'owned_m') self.assertPolicyAllows(subitem, 'setuid_m') self.assertPolicyAllows(subitem, 'dangerous_m') def testUnicodeName(self): policy = self.policy assert policy.validate('', '', u'foo', '', None) if 0: # This test purposely generates a log entry. # Enable it if you don't mind it adding to the log. def testInsaneRoles(self): # Makes sure the policy doesn't blow up on bad input. c = SimpleClass() m = PublicMethod() c.m = m # Test good roles self.assertPolicyAllows(c, 'm') # Test bad roles m.__roles__ = 1950 try: self.assertPolicyAllows(c, 'm') except TypeError: pass else: self.fail('Policy accepted bad __roles__') class ISecurityPolicyConformance: def test_conforms_to_ISecurityPolicy(self): from AccessControl.interfaces import ISecurityPolicy from zope.interface.verify import verifyClass verifyClass(ISecurityPolicy, self._getTargetClass()) class Python_ZSPTests(ZopeSecurityPolicyTestBase, ISecurityPolicyConformance, ): def _getTargetClass(self): from AccessControl.ImplPython import ZopeSecurityPolicy return ZopeSecurityPolicy class C_ZSPTests(ZopeSecurityPolicyTestBase, ISecurityPolicyConformance, ): def _getTargetClass(self): from AccessControl.ImplC import ZopeSecurityPolicy return ZopeSecurityPolicy def test_getRoles(): """ >>> from AccessControl.ZopeSecurityPolicy import getRoles >>> class C: ... x = 'CRole' >>> class V: ... x = 'VRole' >>> c = C() >>> c.v = V() >>> getRoles(c, None, c.v, 42) 42 >>> getRoles(c, 'inabox', c.v, 42) 42 >>> c.v.__roles__ = ['spam', 'eggs'] >>> getRoles(c, None, c.v, 42) ['spam', 'eggs'] >>> getRoles(c, 'withafox', c.v, 42) ['spam', 'eggs'] >>> del c.v.__roles__ >>> V.__roles__ = ('Manager', ) >>> getRoles(c, None, c.v, 42) ('Manager',) >>> getRoles(c, 'withafox', c.v, 42) ('Manager',) >>> del V.__roles__ >>> c.foo__roles__ = ('Foo', ) >>> getRoles(c, None, c.v, 42) 42 >>> getRoles(c, 'foo', c.v, 42) 42 >>> C.foo__roles__ = ('Editor', ) >>> getRoles(c, None, c.v, 42) 42 >>> getRoles(c, 'foo', c.v, 42) ('Editor',) >>> del C.foo__roles__ >>> class ComputedRoles: ... def __init__(self, roles): ... self.roles = roles ... def rolesForPermissionOn(self, ob): ... return [ob.x] + self.roles >>> c.v.__roles__ = ComputedRoles(['Member']) >>> getRoles(c, None, c.v, 42) ['VRole', 'Member'] >>> getRoles(c, 'foo', c.v, 42) ['VRole', 'Member'] >>> c.foo__roles__ = ComputedRoles(['Admin']) >>> getRoles(c, None, c.v, 42) ['VRole', 'Member'] >>> getRoles(c, 'foo', c.v, 42) ['VRole', 'Member'] >>> del c.v.__roles__ >>> getRoles(c, None, c.v, 42) 42 >>> getRoles(c, 'foo', c.v, 42) 42 >>> C.foo__roles__ = ComputedRoles(['Guest']) >>> getRoles(c, None, c.v, 42) 42 >>> getRoles(c, 'foo', c.v, 42) ['CRole', 'Guest'] >>> V.__roles__ = ComputedRoles(['Member']) >>> getRoles(c, None, c.v, 42) ['VRole', 'Member'] >>> getRoles(c, 'foo', c.v, 42) ['VRole', 'Member'] """ def test_zsp_gets_right_roles_for_methods(): """ >>> from AccessControl.ZopeSecurityPolicy import ZopeSecurityPolicy >>> zsp = ZopeSecurityPolicy() >>> from ExtensionClass import Base >>> class C(Base): ... def foo(self): ... pass ... foo__roles__ = ['greeneggs', 'ham'] ... def bar(self): ... pass >>> class User: ... def __init__(self, roles): ... self.roles = roles ... def allowed(self, value, roles): ... for role in roles: ... if role in self.roles: ... return True ... return False >>> class Context: ... stack = () ... def __init__(self, user): ... self.user = user >>> c = C() >>> bool(zsp.validate(c, c, 'foo', c.foo, Context(User(['greeneggs'])))) True >>> zsp.validate(c, c, 'foo', c.foo, Context(User(['spam']))) Traceback (most recent call last): ... Unauthorized: You are not allowed to access 'foo' in this context >>> c.__roles__ = ['spam'] >>> zsp.validate(c, c, 'foo', c.foo, Context(User(['spam']))) Traceback (most recent call last): ... Unauthorized: You are not allowed to access 'foo' in this context >>> zsp.validate(c, c, 'bar', c.bar, Context(User(['spam']))) Traceback (most recent call last): ... Unauthorized: You are not allowed to access 'bar' in this context >>> c.__allow_access_to_unprotected_subobjects__ = 1 >>> bool(zsp.validate(c, c, 'bar', c.bar, Context(User(['spam'])))) True """ from zope.testing.doctest import DocTestSuite def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(Python_ZSPTests, 'test')) suite.addTest(unittest.makeSuite(C_ZSPTests, 'test')) suite.addTest(DocTestSuite()) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/tests/public_module/0000775000076400007640000000000010435610645023362 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/public_module/__init__.py0000664000076400007640000000015710435500320025462 0ustar ajungajung# test module, public from Acquisition import aq_base # Foil the old ZopeSecurityPolicy def pub(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/public_module/submodule/0000775000076400007640000000000010435610645025361 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/public_module/submodule/__init__.py0000664000076400007640000000005310435500320027454 0ustar ajungajung# test module, public def pub(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/__init__.py0000664000076400007640000000000010435500320022622 0ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/private_module/0000775000076400007640000000000010435610645023556 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/private_module/__init__.py0000664000076400007640000000006110435500320025650 0ustar ajungajung# test module, all private def priv(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/private_module/submodule/0000775000076400007640000000000010435610645025555 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/private_module/submodule/__init__.py0000664000076400007640000000006110435500320027647 0ustar ajungajung# test module, all private def priv(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/testOwned.py0000664000076400007640000001046010435500320023052 0ustar ajungajung"""Unit tests for AccessControl.Owned $Id: testOwned.py 24763 2004-05-17 05:59:28Z philikon $ """ import unittest import Testing import ZODB from Acquisition import Implicit, aq_inner class FauxUser(Implicit): def __init__(self, id): self._id = id def getId(self): return self._id class FauxUserFolder(Implicit): def getUserById(self, id, default): return FauxUser(id) class FauxRoot(Implicit): def getPhysicalRoot(self): return aq_inner(self) def unrestrictedTraverse(self, path, default=None): if type(path) is type(''): path = path.split('/') if not path[0]: path = path[1:] obj = self try: while path: next, path = path[0], path[1:] obj = getattr(obj, next) except AttributeError: obj = default return obj class OwnedTests(unittest.TestCase): def _getTargetClass(self): from AccessControl.Owned import Owned return Owned def _makeOne(self, *args, **kw): return self._getTargetClass()(*args, **kw) def _makeDummy(self, *args, **kw): from AccessControl.Owned import Owned class Dummy(Implicit, Owned): pass return Dummy(*args, **kw) def test_getOwnerTuple_unowned(self): owned = self._makeOne() owner_tuple = owned.getOwnerTuple() self.assertEqual(owner_tuple, None) def test_getOwnerTuple_simple(self): owned = self._makeOne() owned._owner = ('/foobar', 'baz') owner_tuple = owned.getOwnerTuple() self.assertEqual(len(owner_tuple), 2) self.assertEqual(owner_tuple[0], '/foobar') self.assertEqual(owner_tuple[1], 'baz') def test_getOwnerTuple_acquired(self): root = FauxRoot() root._owner = ('/foobar', 'baz') owned = self._makeDummy().__of__(root) owner_tuple = owned.getOwnerTuple() self.assertEqual(len(owner_tuple), 2) self.assertEqual(owner_tuple[0], '/foobar') self.assertEqual(owner_tuple[1], 'baz') def test_getOwnerTuple_acquired_no_tricks(self): # Ensure that we acquire ownership only through containment. root = FauxRoot() root._owner = ('/foobar', 'baz') owned = self._makeDummy().__of__(root) owner_tuple = owned.getOwnerTuple() tricky = self._makeDummy() tricky._owner = ('/bambam', 'qux') tricky = tricky.__of__(root) not_tricked = owned.__of__(tricky) owner_tuple = not_tricked.getOwnerTuple() self.assertEqual(len(owner_tuple), 2) self.assertEqual(owner_tuple[0], '/foobar') self.assertEqual(owner_tuple[1], 'baz') def test_getWrappedOwner_unowned(self): owned = self._makeOne() wrapped_owner = owned.getWrappedOwner() self.assertEqual(wrapped_owner, None) def test_getWrappedOwner_unownable(self): from AccessControl.Owned import UnownableOwner owned = self._makeOne() owned._owner = UnownableOwner wrapped_owner = owned.getWrappedOwner() self.assertEqual(wrapped_owner, None) def test_getWrappedOwner_simple(self): root = FauxRoot() root.acl_users = FauxUserFolder() owned = self._makeDummy().__of__(root) owned._owner = ('/acl_users', 'user') wrapped_owner = owned.getWrappedOwner() self.assertEqual(wrapped_owner.getId(), 'user') def test_getWrappedOwner_acquired(self): root = FauxRoot() root._owner = ('/acl_users', 'user') root.acl_users = FauxUserFolder() owned = self._makeDummy().__of__(root) wrapped_owner = owned.getWrappedOwner() self.assertEqual(wrapped_owner.getId(), 'user') def test_getWrappedOwner_acquired_no_tricks(self): root = FauxRoot() root._owner = ('/acl_users', 'user') root.acl_users = FauxUserFolder() owned = self._makeDummy().__of__(root) tricky = self._makeDummy() tricky._owner = ('/acl_users', 'black_hat') tricky = tricky.__of__(root) not_tricked = owned.__of__(tricky) wrapped_owner = not_tricked.getWrappedOwner() self.assertEqual(wrapped_owner.getId(), 'user') def test_suite(): return unittest.makeSuite(OwnedTests) Zope-2.8.7-final/lib/python/AccessControl/tests/testImplementation.py0000664000076400007640000000470010435500320024763 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test of the implementation selection support.""" import unittest from AccessControl.Implementation import getImplementationName from AccessControl.Implementation import setImplementation class AccessControlImplementationTest(unittest.TestCase): have_cAccessControl = None def setUp(self): if self.have_cAccessControl is None: try: import AccessControl.cAccessControl except ImportError: v = False else: v = True self.__class__.have_cAccessControl = v self.original = getImplementationName() def tearDown(self): setImplementation(self.original) ## Note - this test is disabled because the intent for 2.7 was *not* ## to support the ability to arbitrarily switch the security policy ## at any time (which would currently be nearly impossible to do in ## a way that would be sane for 3rd party apps that may already have ## imported parts of the security machinery), but just to make sure ## that the config file could be used to initially set the implementation ## to be used. The policy setting is 'initialize once' - setImplementation ## should not be called either by user code or unit tests, as the ## effects are officially undefined. ## def test_setImplemenationC(self): ## setImplementation("C") ## name = getImplementationName() ## if self.have_cAccessControl: ## self.assertEqual(name, "C") ## else: ## self.assertEqual(name, "PYTHON") ## def test_setImplemenationPython(self): ## setImplementation("Python") ## self.assertEqual(getImplementationName(), "PYTHON") def test_suite(): return unittest.makeSuite(AccessControlImplementationTest) if __name__ == "__main__": unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/AccessControl/tests/test_safeiter.py0000664000076400007640000000442310435500320023741 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for the guarded iterartor. $Id: test_safeiter.py 24763 2004-05-17 05:59:28Z philikon $ """ import unittest # Persistence system must be initialized. import ZODB from AccessControl import ZopeGuards class SafeIterTestCase(unittest.TestCase): # XXX these tests replace the global guard() function in # AccessControl.ZopeGuards; this is not the nicest way to check # that things work, but avoids making the SafeIter unit tests from # testing things other than the guarded iterator itself. In # particular, it avoids testing the actual guard checks, which # should be tested separately. def setUp(self): self.original_guard = ZopeGuards.guard ZopeGuards.guard = self.guard self.checks = [] def tearDown(self): ZopeGuards.guard = self.original_guard def guard(self, container, value, index=None): self.checks.append((id(container), value)) def test_iteration(self): seq = [1, 2, 3] seqid = id(seq) it = ZopeGuards.SafeIter(seq) self.assertEqual(list(it), seq) self.assertEqual(self.checks, [(seqid, 1), (seqid, 2), (seqid, 3)]) def test_iteration_with_container(self): seq = [1, 2, 3] container = object() contid = id(container) it = ZopeGuards.SafeIter(seq, container) self.assertEqual(list(it), seq) self.assertEqual(self.checks, [(contid, 1), (contid, 2), (contid, 3)]) def test_suite(): return unittest.makeSuite(SafeIterTestCase) Zope-2.8.7-final/lib/python/AccessControl/tests/testBindings.py0000664000076400007640000002507310435500320023541 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test Bindings $Id: testBindings.py 29694 2005-03-28 16:34:11Z tim_one $ """ import unittest import ZODB import transaction from Acquisition import Implicit from AccessControl import ClassSecurityInfo from Globals import InitializeClass from OFS.ObjectManager import ObjectManager from OFS.Folder import Folder class SecurityManager: def __init__(self, reject=0): self.calls = [] self.reject = reject def validate(self, *args): from AccessControl import Unauthorized self.calls.append(('validate', args)) if self.reject: raise Unauthorized return 1 def validateValue(self, *args): from AccessControl import Unauthorized self.calls.append(('validateValue', args)) if self.reject: raise Unauthorized return 1 def checkPermission(self, *args): self.calls.append(('checkPermission', args)) return not self.reject def addContext(self, *args): self.calls.append(('addContext', args)) return 1 def removeContext(self, *args): self.calls.append(('removeContext', args)) return 1 class UnderprivilegedUser: def getId(self): return 'underprivileged' def allowed(self, object, object_roles=None): return 0 class RivilegedUser: def getId(self): return 'privileged' def allowed(self, object, object_roles=None): return 1 class FauxRoot(ObjectManager): def getPhysicalPath(self): return ('',) def __repr__(self): return '' class FauxFolder(Folder): security = ClassSecurityInfo() security.declareObjectPrivate() security.declarePrivate('__repr__') def __repr__(self): return '' % self.getId() security.declarePublic('methodWithRoles') def methodWithRoles(self): return 'method called' InitializeClass(FauxFolder) class TestBindings(unittest.TestCase): def setUp(self): from Testing.ZODButil import makeDB transaction.begin() self.db = makeDB() self.connection = self.db.open() def tearDown(self): from Testing.ZODButil import cleanDB from AccessControl.SecurityManagement import noSecurityManager noSecurityManager() transaction.abort() self.connection.close() self.db.close() cleanDB() def _getRoot(self): from Testing.makerequest import makerequest #true_root = self.connection.root()[ 'Application' ] #true_root = self.connection.root() #return makerequest(true_root) return makerequest(FauxRoot()) def _makeTree(self): root = self._getRoot() guarded = FauxFolder() guarded._setId('guarded') guarded.__roles__ = ( 'Manager', ) root._setOb('guarded', guarded) guarded = root._getOb('guarded') open = FauxFolder() open._setId('open') open.__roles__ = ( 'Anonymous', ) guarded._setOb('open', open) bound_unused_container_ps = self._newPS('return 1') guarded._setOb('bound_unused_container_ps', bound_unused_container_ps) bound_used_container_ps = self._newPS('return container.id') guarded._setOb('bound_used_container_ps', bound_used_container_ps) bound_used_container_ok_ps = self._newPS('return container.id') open._setOb('bound_used_container_ok_ps', bound_used_container_ok_ps) bound_unused_context_ps = self._newPS('return 1') guarded._setOb('bound_unused_context_ps', bound_unused_context_ps) bound_used_context_ps = self._newPS('return context.id') guarded._setOb('bound_used_context_ps', bound_used_context_ps) bound_used_context_methodWithRoles_ps = self._newPS( 'return context.methodWithRoles()') guarded._setOb('bound_used_context_methodWithRoles_ps', bound_used_context_methodWithRoles_ps) container_ps = self._newPS('return container') guarded._setOb('container_ps', container_ps) container_str_ps = self._newPS('return str(container)') guarded._setOb('container_str_ps', container_str_ps) context_ps = self._newPS('return context') guarded._setOb('context_ps', context_ps) context_str_ps = self._newPS('return str(context)') guarded._setOb('context_str_ps', context_str_ps) return root def _newPS(self, txt, bind=None): from Products.PythonScripts.PythonScript import PythonScript ps = PythonScript('ps') #ps.ZBindings_edit(bind or {}) ps.write(txt) ps._makeFunction() return ps # These test that the mere binding of context or container, when the # user doesn't have access to them, doesn't raise an unauthorized. An # exception *will* be raised if the script attempts to use them. This # is a b/w compatibility hack: see Bindings.py for details. def test_bound_unused_container(self): from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') ps = guarded._getOb('bound_unused_container_ps') self.assertEqual(ps(), 1) def test_bound_used_container(self): from AccessControl.SecurityManagement import newSecurityManager from AccessControl import Unauthorized newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') ps = guarded._getOb('bound_used_container_ps') self.assertRaises(Unauthorized, ps) ps = guarded._getOb('container_str_ps') self.assertRaises(Unauthorized, ps) ps = guarded._getOb('container_ps') container = ps() self.assertRaises(Unauthorized, container) self.assertRaises(Unauthorized, container.index_html) try: str(container) except Unauthorized: pass else: self.fail("str(container) didn't raise Unauthorized!") ps = guarded._getOb('bound_used_container_ps') ps._proxy_roles = ( 'Manager', ) ps() ps = guarded._getOb('container_str_ps') ps._proxy_roles = ( 'Manager', ) ps() def test_bound_used_container_allowed(self): from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') open = guarded._getOb('open') ps = open.unrestrictedTraverse('bound_used_container_ok_ps') self.assertEqual(ps(), 'open') def test_bound_unused_context(self): from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') ps = guarded._getOb('bound_unused_context_ps') self.assertEqual(ps(), 1) def test_bound_used_context(self): from AccessControl.SecurityManagement import newSecurityManager from AccessControl import Unauthorized newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') ps = guarded._getOb('bound_used_context_ps') self.assertRaises(Unauthorized, ps) ps = guarded._getOb('context_str_ps') self.assertRaises(Unauthorized, ps) ps = guarded._getOb('context_ps') context = ps() self.assertRaises(Unauthorized, context) self.assertRaises(Unauthorized, context.index_html) try: str(context) except Unauthorized: pass else: self.fail("str(context) didn't raise Unauthorized!") ps = guarded._getOb('bound_used_context_ps') ps._proxy_roles = ( 'Manager', ) ps() ps = guarded._getOb('context_str_ps') ps._proxy_roles = ( 'Manager', ) ps() def test_bound_used_context_allowed(self): from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') open = guarded._getOb('open') ps = open.unrestrictedTraverse('bound_used_context_ps') self.assertEqual(ps(), 'open') def test_ok_no_bindings(self): from AccessControl.SecurityManagement import newSecurityManager newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') boundless_ps = self._newPS('return 42') guarded._setOb('boundless_ps', boundless_ps) boundless_ps = guarded._getOb('boundless_ps') # # Clear the bindings, so that the script may execute. # boundless_ps.ZBindings_edit( {'name_context': '', 'name_container': '', 'name_m_self': '', 'name_ns': '', 'name_subpath': ''}) self.assertEqual(boundless_ps(), 42) def test_bound_used_context_method_w_roles(self): from AccessControl.SecurityManagement import newSecurityManager from AccessControl import Unauthorized newSecurityManager(None, UnderprivilegedUser()) root = self._makeTree() guarded = root._getOb('guarded') # Assert that we can call a protected method, even though we have # no access to the context directly. ps = guarded._getOb('bound_used_context_ps') self.assertRaises(Unauthorized, ps) ps = guarded._getOb('bound_used_context_methodWithRoles_ps') self.assertEqual(ps(), 'method called') def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestBindings)) return suite if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/AccessControl/tests/testSecurity.py0000664000076400007640000000771210435500320023613 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Document Template Tests """ __rcs_id__='$Id: testSecurity.py 24763 2004-05-17 05:59:28Z philikon $' __version__='$Revision: 1.13 $'[11:-2] import os, sys, unittest import ZODB from DocumentTemplate import HTML from DocumentTemplate.tests.testDTML import DTMLTests from Products.PythonScripts.standard import DTML from AccessControl import User, Unauthorized from ExtensionClass import Base class UnownedDTML(DTML): def getOwner(self): return None class SecurityTests (DTMLTests): doc_class = UnownedDTML unrestricted_doc_class = HTML def testNoImplicitAccess(self): class person: name='Jim' doc = self.doc_class( 'Hi, my name is ' '') try: doc(person=person()) except Unauthorized: # Passed the test. pass else: assert 0, 'Did not protect class instance' def testExprExplicitDeny(self): class myclass (Base): __roles__ = None # Public somemethod__roles__ = () # Private def somemethod(self): return "This is a protected operation of public object" html = self.doc_class('') self.failUnlessRaises(Unauthorized, html, myinst=myclass()) def testSecurityInSyntax(self): # Ensures syntax errors are thrown for an expr with restricted # syntax. expr = '' try: # This would be a security hole. html = self.doc_class(expr) # It might compile here... html() # or it might compile here. except SyntaxError: # Passed the test. pass else: assert 0, 'Did not catch bad expr' # Now be sure the syntax error occurred for security purposes. html = self.unrestricted_doc_class(expr) class c: y = 10 res = html(c=c) assert res == '10', res def testNewDTMLBuiltins(self): NEW_BUILTINS_TEMPLATE = """ """ EXPECTED = ['1', '3', '10', 'No', 'None', 'testing', '[1, 2, 3]'] # # XXX: these expressions seem like they should work, with # the following ExPECTED, but they raise Unauthorized # on the 'next' name. # # # # #EXPECTED = ['1', '3', '10', '1', '(0, 1)'] template = self.doc_class(NEW_BUILTINS_TEMPLATE) res = template() lines = filter(None, [x.strip() for x in res.split('\n')]) self.assertEqual(lines, EXPECTED) # Note: we need more tests! def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( SecurityTests ) ) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/tests/testModuleSecurity.py0000664000076400007640000000553410435500320024761 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Module Import Tests """ __rcs_id__='$Id: testModuleSecurity.py 24763 2004-05-17 05:59:28Z philikon $' __version__='$Revision: 1.4 $'[11:-2] import os, sys, unittest import Testing import ZODB from AccessControl import User from AccessControl import Unauthorized, ModuleSecurityInfo from AccessControl.ZopeGuards import guarded_import ModuleSecurityInfo('AccessControl.tests.mixed_module').declarePublic('pub') ModuleSecurityInfo('AccessControl.tests.public_module').declarePublic('pub') ModuleSecurityInfo('AccessControl.tests.public_module.submodule' ).declarePublic('pub') class SecurityTests(unittest.TestCase): def assertUnauth(self, module, fromlist): try: guarded_import(module, fromlist=fromlist) except (Unauthorized, ImportError): # Passed the test. pass else: assert 0, ('Did not protect module instance %s, %s' % (`module`, `fromlist`)) def assertAuth(self, module, fromlist): try: guarded_import(module, fromlist=fromlist) except (Unauthorized, ImportError): assert 0, ('Did not expose module instance %s, %s' % (`module`, `fromlist`)) def testPrivateModule(self): for name in '', '.submodule': for fromlist in (), ('priv',): self.assertUnauth( 'AccessControl.tests.private_module%s' % name, fromlist) def testMixedModule(self): self.assertAuth('AccessControl.tests.mixed_module', ()) self.assertAuth('AccessControl.tests.mixed_module', ('pub',)) self.assertUnauth('AccessControl.tests.mixed_module', ('priv',)) self.assertUnauth('AccessControl.tests.mixed_module.submodule', ()) def testPublicModule(self): for name in '', '.submodule': for fromlist in (), ('pub',): self.assertAuth( 'AccessControl.tests.public_module%s' % name, fromlist) def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( SecurityTests ) ) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/tests/testAcquisition.py0000664000076400007640000002052410435500320024270 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests demonstrating consequences of guarded_getattr fix from 2004/08/07 http://mail.zope.org/pipermail/zope-checkins/2004-August/028152.html http://zope.org/Collectors/CMF/259 """ import unittest from Testing.makerequest import makerequest import Zope2 Zope2.startup() from OFS.SimpleItem import SimpleItem from Globals import InitializeClass from AccessControl import ClassSecurityInfo from AccessControl.SecurityManagement import newSecurityManager from AccessControl.SecurityManagement import noSecurityManager from AccessControl.Permissions import view, view_management_screens from AccessControl.ImplPython import guarded_getattr as guarded_getattr_py from AccessControl.ImplC import guarded_getattr as guarded_getattr_c from Products.SiteErrorLog.SiteErrorLog import SiteErrorLog class AllowedItem(SimpleItem): id = 'allowed' security = ClassSecurityInfo() security.setDefaultAccess('allow') InitializeClass(AllowedItem) class DeniedItem(SimpleItem): id = 'denied' security = ClassSecurityInfo() security.setDefaultAccess('deny') InitializeClass(DeniedItem) class ProtectedItem(SimpleItem): id = 'protected' security = ClassSecurityInfo() security.declareObjectProtected(view_management_screens) InitializeClass(ProtectedItem) class ProtectedSiteErrorLog(SiteErrorLog): '''This differs from the base by declaring security for the object itself. ''' id = 'error_log2' security = ClassSecurityInfo() security.declareObjectProtected(view) InitializeClass(ProtectedSiteErrorLog) class TestGetAttr(unittest.TestCase): def setUp(self): import transaction self.guarded_getattr = guarded_getattr_py transaction.manager.begin() self.app = makerequest(Zope2.app()) try: # Set up a manager user self.uf = self.app.acl_users self.uf._doAddUser('manager', 'secret', ['Manager'], []) self.login('manager') # Set up objects in the root that we want to aquire self.app.manage_addFolder('plain_folder') self.app._setObject('error_log2', ProtectedSiteErrorLog()) # We also want to be able to acquire simple attributes self.app.manage_addProperty(id='simple_type', type='string', value='a string') # Set up a subfolder and the objects we want to acquire from self.app.manage_addFolder('subfolder') self.folder = self.app.subfolder self.folder._setObject('allowed', AllowedItem()) self.folder._setObject('denied', DeniedItem()) self.folder._setObject('protected', ProtectedItem()) except: self.tearDown() raise def tearDown(self): import transaction noSecurityManager() transaction.manager.get().abort() self.app._p_jar.close() def login(self, name): user = self.uf.getUserById(name) user = user.__of__(self.uf) newSecurityManager(None, user) # Acquire plain folder def testFolderAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'plain_folder') self.assertEqual(o, self.app.plain_folder) def testFolderDenied(self): o = self.guarded_getattr(self.folder.denied, 'plain_folder') self.assertEqual(o, self.app.plain_folder) def testFolderProtected(self): o = self.guarded_getattr(self.folder.protected, 'plain_folder') self.assertEqual(o, self.app.plain_folder) # Acquire user folder def testAclUsersAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'acl_users') self.assertEqual(o, self.app.acl_users) def testAclUsersDenied(self): # XXX: Fails in 2.7.3 o = self.guarded_getattr(self.folder.denied, 'acl_users') self.assertEqual(o, self.app.acl_users) def testAclUsersProtected(self): # XXX: Fails in 2.7.3 for Anonymous o = self.guarded_getattr(self.folder.protected, 'acl_users') self.assertEqual(o, self.app.acl_users) # Acquire browser id manager def testBrowserIdManagerAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'browser_id_manager') self.assertEqual(o, self.app.browser_id_manager) def testBrowserIdManagerDenied(self): o = self.guarded_getattr(self.folder.denied, 'browser_id_manager') self.assertEqual(o, self.app.browser_id_manager) def testBrowserIdManagerProtected(self): o = self.guarded_getattr(self.folder.protected, 'browser_id_manager') self.assertEqual(o, self.app.browser_id_manager) # Acquire error log def testErrorLogAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'error_log') self.assertEqual(o, self.app.error_log) def testErrorLogDenied(self): # XXX: Fails in 2.7.3 o = self.guarded_getattr(self.folder.denied, 'error_log') self.assertEqual(o, self.app.error_log) def testErrorLogProtected(self): # XXX: Fails in 2.7.3 for Anonymous o = self.guarded_getattr(self.folder.protected, 'error_log') self.assertEqual(o, self.app.error_log) # Now watch this: error log with object security declaration works fine! def testProtectedErrorLogAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'error_log2') self.assertEqual(o, self.app.error_log2) def testProtectedErrorLogDenied(self): o = self.guarded_getattr(self.folder.denied, 'error_log2') self.assertEqual(o, self.app.error_log2) def testProtectedErrorLogProtected(self): o = self.guarded_getattr(self.folder.protected, 'error_log2') self.assertEqual(o, self.app.error_log2) # This appears to mean that any potential acquiree must make sure # to declareObjectProtected(SomePermission). # From the ZDG: # We've seen how to make assertions on methods - but in the case of # someObject we are not trying to access any particular method, but # rather the object itself (to pass it to some_method). Because the # security machinery will try to validate access to someObject, we # need a way to let the security machinery know how to handle access # to the object itself in addition to protecting its methods. # IOW, acquiring an object in restricted Python now amounts to # "passing it to some_method". # Also test Richard Jones' use-case of acquiring a string: def testSimpleTypeAllowed(self): o = self.guarded_getattr(self.folder.allowed, 'simple_type') self.assertEqual(o, 'a string') def testSimpleTypeDenied(self): # XXX: Fails in 2.7.3 o = self.guarded_getattr(self.folder.denied, 'simple_type') self.assertEqual(o, 'a string') def testSimpleTypeProtected(self): # XXX: Fails in 2.7.3 for Anonymous o = self.guarded_getattr(self.folder.protected, 'simple_type') self.assertEqual(o, 'a string') class TestGetAttrAnonymous(TestGetAttr): # Run all tests again as Anonymous User def setUp(self): TestGetAttr.setUp(self) # Log out noSecurityManager() class TestGetAttr_c(TestGetAttr): def setUp(self): TestGetAttr.setUp(self) self.guarded_getattr = guarded_getattr_c class TestGetAttrAnonymous_c(TestGetAttrAnonymous): def setUp(self): TestGetAttrAnonymous.setUp(self) self.guarded_getattr = guarded_getattr_c def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(TestGetAttr)) suite.addTest(unittest.makeSuite(TestGetAttrAnonymous)) suite.addTest(unittest.makeSuite(TestGetAttr_c)) suite.addTest(unittest.makeSuite(TestGetAttrAnonymous_c)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/AccessControl/tests/testPermissionRole.py0000664000076400007640000001144010435500320024747 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests of PermissionRole """ __rcs_id__='$Id: testPermissionRole.py 30800 2005-06-15 00:48:38Z shane $' __version__='$Revision: 1.4 $'[11:-2] import os, sys, unittest from AccessControl.PermissionRole import PermissionRole from Acquisition import Implicit, Explicit, aq_base ViewPermission = 'View' EditThingsPermission = 'Edit Things!' DeletePermission = 'Delete' class AppRoot(Explicit): _View_Permission = None _Edit_Things__Permission = ('Manager', 'Owner') # No default for delete permission. class ImplicitContainer(Implicit): pass class ExplicitContainer(Explicit): pass class RestrictiveObject(Implicit): _View_Permission = ('Manager',) _Delete_Permission = () # Nobody class PermissiveObject(Explicit): _Edit_Things__Permission = ['Anonymous'] class ZClassMethodish(Implicit): # Think of this as a method that should only be visible to users # who have the edit permission. _View_Permission = '_Edit_Things__Permission' _Edit_Things__Permission = '' _Delete_Permission = '' def assertPRoles(ob, permission, expect): """ Asserts that in the context of ob, the given permission maps to the given roles. """ pr = PermissionRole(permission) roles = pr.__of__(ob) roles2 = aq_base(pr).__of__(ob) assert roles == roles2 or tuple(roles) == tuple(roles2), ( 'Different methods of checking roles computed unequal results') same = 0 if roles: # When verbose security is enabled, permission names are # embedded in the computed roles. Remove the permission # names. roles = [r for r in roles if not r.endswith('_Permission')] if roles is None or expect is None: if (roles is None or tuple(roles) == ('Anonymous',)) and ( expect is None or tuple(expect) == ('Anonymous',)): same = 1 else: got = {} for r in roles: got[r] = 1 expected = {} for r in expect: expected[r] = 1 if got == expected: # Dict compare does the Right Thing. same = 1 assert same, 'Expected roles: %s, got: %s' % (`expect`, `roles`) class PermissionRoleTests (unittest.TestCase): def testRestrictive(self, explicit=0): app = AppRoot() if explicit: app.c = ExplicitContainer() else: app.c = ImplicitContainer() app.c.o = RestrictiveObject() o = app.c.o assertPRoles(o, ViewPermission, ('Manager',)) assertPRoles(o, EditThingsPermission, ('Manager','Owner',)) assertPRoles(o, DeletePermission, ()) def testPermissive(self, explicit=0): app = AppRoot() if explicit: app.c = ExplicitContainer() else: app.c = ImplicitContainer() app.c.o = PermissiveObject() o = app.c.o assertPRoles(o, ViewPermission, ('Anonymous',)) assertPRoles(o, EditThingsPermission, ('Anonymous','Manager','Owner',)) assertPRoles(o, DeletePermission, ('Manager',)) def testExplicit(self): self.testRestrictive(1) self.testPermissive(1) def testAppDefaults(self): o = AppRoot() assertPRoles(o, ViewPermission, ('Anonymous',)) assertPRoles(o, EditThingsPermission, ('Manager','Owner',)) assertPRoles(o, DeletePermission, ('Manager',)) def testPermissionMapping(self): app = AppRoot() app.c = ImplicitContainer() app.c.o = ZClassMethodish() o = app.c.o assertPRoles(o, ViewPermission, ('Manager','Owner',)) assertPRoles(o, EditThingsPermission, ()) assertPRoles(o, DeletePermission, ()) def testPermissionRoleSupportsGetattr(self): a = PermissionRole('a') self.failUnless(getattr(a, '__roles__') == ('Manager',)) self.failUnless(getattr(a, '_d') == ('Manager',)) self.failUnless(getattr(a, '__name__') == 'a') self.failUnless(getattr(a, '_p') == '_a_Permission') def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(PermissionRoleTests)) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/tests/actual_python.py0000664000076400007640000001067210435500320023755 0ustar ajungajung# The code in this file is executed after being compiled as restricted code, # and given a globals() dict with our idea of safe builtins, and the # Zope production implementations of the special restricted-Python functions # (like _getitem_ and _getiter_, etc). # # This isn't trying to provoke security problems, it's just trying to verify # that Python code continues to work as intended after all the transformations, # and with all the special wrappers we supply. def f1(): next = iter(xrange(3)).next assert next() == 0 assert next() == 1 assert next() == 2 try: next() except StopIteration: pass else: assert 0, "expected StopIteration" f1() def f2(): assert map(lambda x: x+1, range(3)) == range(1, 4) f2() def f3(): assert filter(None, range(10)) == range(1, 10) f3() def f4(): assert [i+1 for i in range(3)] == range(*(1, 4)) f4() def f5(): x = range(5) def add(a, b): return a+b assert sum(x) == reduce(add, x, 0) f5() def f6(): class C: def display(self): return str(self.value) c1 = C() c2 = C() c1.value = 12 assert getattr(c1, 'value') == 12 assert c1.display() == '12' assert not hasattr(c2, 'value') setattr(c2, 'value', 34) assert c2.value == 34 assert hasattr(c2, 'value') del c2.value assert not hasattr(c2, 'value') # OK, if we can't set new attributes, at least verify that we can't. #try: # c1.value = 12 #except TypeError: # pass #else: # assert 0, "expected direct attribute creation to fail" #try: # setattr(c1, 'value', 12) #except TypeError: # pass #else: # assert 0, "expected indirect attribute creation to fail" assert getattr(C, "display", None) == getattr(C, "display") delattr(C, "display") #try: # setattr(C, "display", lambda self: "replaced") #except TypeError: # pass #else: # assert 0, "expected setattr() attribute replacement to fail" #try: # delattr(C, "display") #except TypeError: # pass #else: # assert 0, "expected delattr() attribute deletion to fail" f6() def f7(): d = apply(dict, [((1, 2), (3, 4))]) # {1: 2, 3: 4} expected = {'k': [1, 3], 'v': [2, 4], 'i': [(1, 2), (3, 4)]} for meth, kind in [('iterkeys', 'k'), ('iteritems', 'i'), ('itervalues', 'v'), ('keys', 'k'), ('items', 'i'), ('values', 'v')]: access = getattr(d, meth) result = list(access()) result.sort() assert result == expected[kind], (meth, kind, result, expected[kind]) f7() def f8(): import math ceil = getattr(math, 'ceil') smallest = 1e100 smallest_index = None largest = -1e100 largest_index = None all = [] for i, x in enumerate((2.2, 1.1, 3.3, 5.5, 4.4)): all.append(x) effective = ceil(x) if effective < smallest: assert min(effective, smallest) == effective smallest = effective smallest_index = i if effective > largest: assert max(effective, largest) == effective largest = effective largest_index = i assert smallest == 2 assert smallest_index == 1 assert largest == 6 assert largest_index == 3 assert min([ceil(x) for x in all]) == smallest assert max(map(ceil, all)) == largest f8() # After all the above, these wrappers were still untouched: # ['DateTime', '_print_', 'reorder', 'same_type', 'test'] # So do something to touch them. def f9(): d = DateTime() print d # this one provoked _print_ # Funky. This probably isn't an intended use of reorder, but I'm # not sure why it exists. assert reorder('edcbaxyz', 'abcdef', 'c') == zip('abde', 'abde') assert test(0, 'a', 0, 'b', 1, 'c', 0, 'd') == 'c' assert test(0, 'a', 0, 'b', 0, 'c', 0, 'd', 'e') == 'e' # Unclear that the next one is *intended* to return None (it falls off # the end of test's implementation without explicitly returning anything). assert test(0, 'a', 0, 'b', 0, 'c', 0, 'd') == None assert same_type(3, 2, 1), 'expected same type' assert not same_type(3, 2, 'a'), 'expected not same type' f9() def f10(): assert iter(enumerate(iter(iter(range(9))))).next() == (0, 0) f10() def f11(): x = 1 x += 1 f11() Zope-2.8.7-final/lib/python/AccessControl/tests/testPasswordDigest.py0000664000076400007640000000675710435500320024756 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test of AuthEncoding """ __rcs_id__='$Id: testPasswordDigest.py 24763 2004-05-17 05:59:28Z philikon $' __version__='$Revision: 1.5 $'[11:-2] import os, sys, unittest from AccessControl import AuthEncoding import unittest class PasswordDigestTests (unittest.TestCase): def testGoodPassword(self): pw = 'good_password' assert len(AuthEncoding.listSchemes()) > 0 # At least one must exist! for id in AuthEncoding.listSchemes(): enc = AuthEncoding.pw_encrypt(pw, id) assert enc != pw assert AuthEncoding.pw_validate(enc, pw) assert AuthEncoding.is_encrypted(enc) assert not AuthEncoding.is_encrypted(pw) def testBadPasword(self): pw = 'OK_pa55w0rd \n' for id in AuthEncoding.listSchemes(): enc = AuthEncoding.pw_encrypt(pw, id) assert enc != pw assert not AuthEncoding.pw_validate(enc, 'xxx') assert not AuthEncoding.pw_validate(enc, enc) if id != 'CRYPT': # crypt truncates passwords and would fail this test. assert not AuthEncoding.pw_validate(enc, pw[:-1]) assert not AuthEncoding.pw_validate(enc, pw[1:]) assert AuthEncoding.pw_validate(enc, pw) def testShortPassword(self): pw = '1' for id in AuthEncoding.listSchemes(): enc = AuthEncoding.pw_encrypt(pw, id) assert enc != pw assert AuthEncoding.pw_validate(enc, pw) assert not AuthEncoding.pw_validate(enc, enc) assert not AuthEncoding.pw_validate(enc, 'xxx') def testLongPassword(self): pw = 'Pw' * 2000 for id in AuthEncoding.listSchemes(): enc = AuthEncoding.pw_encrypt(pw, id) assert enc != pw assert AuthEncoding.pw_validate(enc, pw) assert not AuthEncoding.pw_validate(enc, enc) assert not AuthEncoding.pw_validate(enc, 'xxx') if id != 'CRYPT': # crypt truncates passwords and would fail these tests. assert not AuthEncoding.pw_validate(enc, pw[:-2]) assert not AuthEncoding.pw_validate(enc, pw[2:]) def testBlankPassword(self): pw = '' for id in AuthEncoding.listSchemes(): enc = AuthEncoding.pw_encrypt(pw, id) assert enc != pw assert AuthEncoding.pw_validate(enc, pw) assert not AuthEncoding.pw_validate(enc, enc) assert not AuthEncoding.pw_validate(enc, 'xxx') def testUnencryptedPassword(self): # Sanity check pw = 'my-password' assert AuthEncoding.pw_validate(pw, pw) def test_suite(): suite = unittest.TestSuite() suite.addTest( unittest.makeSuite( PasswordDigestTests ) ) return suite def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/tests/testUserFolder.py0000664000076400007640000002326310435500320024055 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """User folder tests """ __rcs_id__='$Id: testUserFolder.py 39545 2005-10-21 06:50:48Z tseaver $' __version__='$Revision: 1.10 $'[11:-2] import os, sys, base64, unittest from Testing.makerequest import makerequest import transaction import Zope2 Zope2.startup() from AccessControl import Unauthorized from AccessControl.SecurityManagement import newSecurityManager from AccessControl.SecurityManagement import noSecurityManager from AccessControl.User import BasicUserFolder, UserFolder from AccessControl.User import User class UserFolderTests(unittest.TestCase): def setUp(self): transaction.begin() self.app = makerequest(Zope2.app()) try: # Set up a user and role self.uf = UserFolder().__of__(self.app) self.uf._doAddUser('user1', 'secret', ['role1'], []) self.app._addRole('role1') self.app.manage_role('role1', ['View']) # Set up a published object accessible to user self.app.addDTMLMethod('doc', file='') self.app.doc.manage_permission('View', ['role1'], acquire=0) # Rig the REQUEST so it looks like we traversed to doc self.app.REQUEST.set('PUBLISHED', self.app.doc) self.app.REQUEST.set('PARENTS', [self.app]) self.app.REQUEST.steps = ['doc'] self.basic = 'Basic %s' % base64.encodestring('user1:secret') except: self.tearDown() raise def tearDown(self): noSecurityManager() transaction.abort() self.app._p_jar.close() def login(self, name): user = self.uf.getUserById(name) user = user.__of__(self.uf) newSecurityManager(None, user) def testGetUser(self): self.failIfEqual(self.uf.getUser('user1'), None) def testGetBadUser(self): self.assertEqual(self.uf.getUser('user2'), None) def testGetUserById(self): self.failIfEqual(self.uf.getUserById('user1'), None) def testGetBadUserById(self): self.assertEqual(self.uf.getUserById('user2'), None) def testGetUsers(self): users = self.uf.getUsers() self.failUnless(users) self.assertEqual(users[0].getUserName(), 'user1') def testGetUserNames(self): names = self.uf.getUserNames() self.failUnless(names) self.assertEqual(names[0], 'user1') def testIdentify(self): name, password = self.uf.identify(self.basic) self.assertEqual(name, 'user1') self.assertEqual(password, 'secret') def testGetRoles(self): user = self.uf.getUser('user1') self.failUnless('role1' in user.getRoles()) def testGetRolesInContext(self): user = self.uf.getUser('user1') self.app.manage_addLocalRoles('user1', ['Owner']) roles = user.getRolesInContext(self.app) self.failUnless('role1' in roles) self.failUnless('Owner' in roles) def testHasRole(self): user = self.uf.getUser('user1') self.failUnless(user.has_role('role1', self.app)) def testHasLocalRole(self): user = self.uf.getUser('user1') self.app.manage_addLocalRoles('user1', ['Owner']) self.failUnless(user.has_role('Owner', self.app)) def testHasPermission(self): user = self.uf.getUser('user1') self.failUnless(user.has_permission('View', self.app)) self.app.manage_role('role1', ['Add Folders']) self.failUnless(user.has_permission('Add Folders', self.app)) def testHasLocalRolePermission(self): user = self.uf.getUser('user1') self.app.manage_role('Owner', ['Add Folders']) self.app.manage_addLocalRoles('user1', ['Owner']) self.failUnless(user.has_permission('Add Folders', self.app)) def testAuthenticate(self): user = self.uf.getUser('user1') self.failUnless(user.authenticate('secret', self.app.REQUEST)) def testValidate(self): user = self.uf.validate(self.app.REQUEST, self.basic, ['role1']) self.failIfEqual(user, None) self.assertEqual(user.getUserName(), 'user1') def testNotValidateWithoutAuth(self): user = self.uf.validate(self.app.REQUEST, '', ['role1']) self.assertEqual(user, None) def testValidateWithoutRoles(self): # Note - calling uf.validate without specifying roles will cause # the security machinery to determine the needed roles by looking # at the object itself (or its container). I'm putting this note # in to clarify because the original test expected failure but it # really should have expected success, since the user and the # object being checked both have the role 'role1', even though no # roles are passed explicitly to the userfolder validate method. user = self.uf.validate(self.app.REQUEST, self.basic) self.assertEqual(user.getUserName(), 'user1') def testNotValidateWithEmptyRoles(self): user = self.uf.validate(self.app.REQUEST, self.basic, []) self.assertEqual(user, None) def testNotValidateWithWrongRoles(self): user = self.uf.validate(self.app.REQUEST, self.basic, ['Manager']) self.assertEqual(user, None) def testAllowAccessToUser(self): self.login('user1') try: self.app.restrictedTraverse('doc') except Unauthorized: self.fail('Unauthorized') def testDenyAccessToAnonymous(self): self.assertRaises(Unauthorized, self.app.restrictedTraverse, 'doc') def testMaxListUsers(self): # create a folder-ish thing which contains a roleManager, # then put an acl_users object into the folde-ish thing class Folderish(BasicUserFolder): def __init__(self, size, count): self.maxlistusers = size self.users = [] self.acl_users = self self.__allow_groups__ = self for i in xrange(count): self.users.append("Nobody") def getUsers(self): return self.users def user_names(self): return self.getUsers() tinyFolderOver = Folderish(15, 20) tinyFolderUnder = Folderish(15, 10) assert tinyFolderOver.maxlistusers == 15 assert tinyFolderUnder.maxlistusers == 15 assert len(tinyFolderOver.user_names()) == 20 assert len(tinyFolderUnder.user_names()) == 10 try: list = tinyFolderOver.get_valid_userids() assert 0, "Did not raise overflow error" except OverflowError: pass try: list = tinyFolderUnder.get_valid_userids() pass except OverflowError: assert 0, "Raised overflow error erroneously" def test__doAddUser_with_not_yet_encrypted_passwords(self): # See collector #1869 && #1926 from AccessControl.AuthEncoding import pw_validate USER_ID = 'not_yet_encrypted' PASSWORD = 'password' uf = UserFolder().__of__(self.app) uf.encrypt_passwords = True self.failIf(uf._isPasswordEncrypted(PASSWORD)) uf._doAddUser(USER_ID, PASSWORD, [], []) user = uf.getUserById(USER_ID) self.failUnless(uf._isPasswordEncrypted(user.__)) self.failUnless(pw_validate(user.__, PASSWORD)) def test__doAddUser_with_preencrypted_passwords(self): # See collector #1869 && #1926 from AccessControl.AuthEncoding import pw_validate USER_ID = 'already_encrypted' PASSWORD = 'password' uf = UserFolder().__of__(self.app) uf.encrypt_passwords = True ENCRYPTED = uf._encryptPassword(PASSWORD) uf._doAddUser(USER_ID, ENCRYPTED, [], []) user = uf.getUserById(USER_ID) self.assertEqual(user.__, ENCRYPTED) self.failUnless(uf._isPasswordEncrypted(user.__)) self.failUnless(pw_validate(user.__, PASSWORD)) class UserTests(unittest.TestCase): def testGetUserName(self): f = User('chris', '123', ['Manager'], []) self.assertEqual(f.getUserName(), 'chris') def testGetUserId(self): f = User('chris', '123', ['Manager'], []) self.assertEqual(f.getId(), 'chris') def testBaseUserGetIdEqualGetName(self): # this is true for the default user type, but will not # always be true for extended user types going forward (post-2.6) f = User('chris', '123', ['Manager'], []) self.assertEqual(f.getId(), f.getUserName()) def testGetPassword(self): f = User('chris', '123', ['Manager'], []) self.assertEqual(f._getPassword(), '123') def testGetRoles(self): f = User('chris', '123', ['Manager'], []) self.assertEqual(f.getRoles(), ('Manager', 'Authenticated')) def testGetDomains(self): f = User('chris', '123', ['Manager'], []) self.assertEqual(f.getDomains(), ()) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(UserFolderTests)) suite.addTest(unittest.makeSuite(UserTests)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/AccessControl/tests/testZopeGuards.py0000664000076400007640000006002010435500320024056 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test Zope Guards Well, at least begin testing some of the functionality $Id: testZopeGuards.py 40438 2005-12-01 01:55:42Z tseaver $ """ import os, sys import unittest from zope.testing import doctest import ZODB import AccessControl.SecurityManagement from AccessControl.SimpleObjectPolicies import ContainerAssertions from AccessControl import Unauthorized from AccessControl.ZopeGuards \ import guarded_getattr, get_dict_get, get_dict_pop, get_list_pop, \ get_iter, guarded_min, guarded_max, safe_builtins, guarded_enumerate, \ guarded_sum, guarded_apply try: __file__ except NameError: __file__ = os.path.abspath(sys.argv[1]) _FILEPATH = os.path.abspath( __file__ ) _HERE = os.path.dirname( _FILEPATH ) class SecurityManager: def __init__(self, reject=0): self.calls = [] self.reject = reject def validate(self, *args): self.calls.append(('validate', args)) if self.reject: raise Unauthorized return 1 def validateValue(self, *args): self.calls.append(('validateValue', args)) if self.reject: raise Unauthorized return 1 def checkPermission(self, *args): self.calls.append(('checkPermission', args)) return not self.reject class GuardTestCase(unittest.TestCase): def setSecurityManager(self, manager): key = AccessControl.SecurityManagement.get_ident() old = AccessControl.SecurityManagement._managers.get(key) if manager is None: del AccessControl.SecurityManagement._managers[key] else: AccessControl.SecurityManagement._managers[key] = manager return old class Method: def __init__(self, *args): self.args = args class TestGuardedGetattr(GuardTestCase): def setUp(self): self.__sm = SecurityManager() self.__old = self.setSecurityManager(self.__sm) def tearDown(self): self.setSecurityManager(self.__old) def test_unauthorized(self): obj, name = Method(), 'args' value = getattr(obj, name) rc = sys.getrefcount(value) self.__sm.reject = True self.assertRaises(Unauthorized, guarded_getattr, obj, name) self.assert_(self.__sm.calls) del self.__sm.calls[:] self.assertEqual(rc, sys.getrefcount(value)) def test_calls_validate_for_unknown_type(self): guarded_getattr(self, 'test_calls_validate_for_unknown_type') self.assert_(self.__sm.calls) def test_attr_handler_table(self): d = {} _dict = type(d) old = ContainerAssertions.get(_dict) mytable = {'keys': 1, 'values': Method, } ContainerAssertions[_dict] = mytable try: guarded_getattr(d, 'keys') self.assertEqual(len(self.__sm.calls), 0) values = guarded_getattr(d, 'values') self.assertEqual(values.__class__, Method) self.assertEqual(values.args, (d, 'values')) self.assertRaises(Unauthorized, guarded_getattr, d, 'items') finally: ContainerAssertions[_dict] = old class TestGuardedGetitem(GuardTestCase): def setUp(self): self.sm = SecurityManager() self.old = self.setSecurityManager(self.sm) def tearDown(self): self.setSecurityManager(self.old) def test_guarded_getitem_passes_index_to_validate(self): # BBB: collector #1182 specifies that guarded_getitem should be # passing the 'index' to validate, rather than 'None', # until Zope 2.9. from UserDict import UserDict from AccessControl.ZopeGuards import guarded_getitem foo = [] protected = UserDict(foo=foo) value = guarded_getitem(protected, 'foo') self.failUnless(value is foo) self.assertEqual(len(self.sm.calls), 1) self.assertEqual(self.sm.calls[0], ('validate', (protected, protected, 'foo', foo))) class TestDictGuards(GuardTestCase): def test_get_simple(self): get = get_dict_get({'foo': 'bar'}, 'get') self.assertEqual(get('foo'), 'bar') def test_get_default(self): get = get_dict_get({'foo': 'bar'}, 'get') self.failUnless(get('baz') is None) self.assertEqual(get('baz', 'splat'), 'splat') def test_get_validates(self): sm = SecurityManager() old = self.setSecurityManager(sm) get = get_dict_get({'foo':GuardTestCase}, 'get') try: get('foo') finally: self.setSecurityManager(old) self.assert_(sm.calls) def test_pop_simple(self): pop = get_dict_pop({'foo': 'bar'}, 'pop') self.assertEqual(pop('foo'), 'bar') def test_pop_raises(self): pop = get_dict_pop({'foo': 'bar'}, 'pop') self.assertRaises(KeyError, pop, 'baz') def test_pop_default(self): pop = get_dict_pop({'foo': 'bar'}, 'pop') self.assertEqual(pop('baz', 'splat'), 'splat') def test_pop_validates(self): sm = SecurityManager() old = self.setSecurityManager(sm) pop = get_dict_get({'foo':GuardTestCase}, 'pop') try: pop('foo') finally: self.setSecurityManager(old) self.assert_(sm.calls) if sys.version_info >= (2, 2): def test_iterkeys_simple(self): d = {'foo':1, 'bar':2, 'baz':3} iterkeys = get_iter(d, 'iterkeys') keys = d.keys() keys.sort() ikeys = list(iterkeys()) ikeys.sort() self.assertEqual(keys, ikeys) def test_iterkeys_empty(self): iterkeys = get_iter({}, 'iterkeys') self.assertEqual(list(iterkeys()), []) def test_iterkeys_validates(self): sm = SecurityManager() old = self.setSecurityManager(sm) iterkeys = get_iter({GuardTestCase: 1}, 'iterkeys') try: iterkeys().next() finally: self.setSecurityManager(old) self.assert_(sm.calls) def test_itervalues_simple(self): d = {'foo':1, 'bar':2, 'baz':3} itervalues = get_iter(d, 'itervalues') values = d.values() values.sort() ivalues = list(itervalues()) ivalues.sort() self.assertEqual(values, ivalues) def test_itervalues_empty(self): itervalues = get_iter({}, 'itervalues') self.assertEqual(list(itervalues()), []) def test_itervalues_validates(self): sm = SecurityManager() old = self.setSecurityManager(sm) itervalues = get_iter({GuardTestCase: 1}, 'itervalues') try: itervalues().next() finally: self.setSecurityManager(old) self.assert_(sm.calls) class TestListGuards(GuardTestCase): def test_pop_simple(self): pop = get_list_pop(['foo', 'bar', 'baz'], 'pop') self.assertEqual(pop(), 'baz') self.assertEqual(pop(0), 'foo') def test_pop_raises(self): pop = get_list_pop([], 'pop') self.assertRaises(IndexError, pop) def test_pop_validates(self): sm = SecurityManager() old = self.setSecurityManager(sm) pop = get_list_pop([GuardTestCase], 'pop') try: pop() finally: self.setSecurityManager(old) self.assert_(sm.calls) class TestBuiltinFunctionGuards(GuardTestCase): def test_min_fails(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) self.assertRaises(Unauthorized, guarded_min, [1,2,3]) self.assertRaises(Unauthorized, guarded_min, 1,2,3) self.setSecurityManager(old) def test_max_fails(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) self.assertRaises(Unauthorized, guarded_max, [1,2,3]) self.assertRaises(Unauthorized, guarded_max, 1,2,3) self.setSecurityManager(old) def test_enumerate_fails(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) enum = guarded_enumerate([1,2,3]) self.assertRaises(Unauthorized, enum.next) self.setSecurityManager(old) def test_sum_fails(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) self.assertRaises(Unauthorized, guarded_sum, [1,2,3]) self.setSecurityManager(old) def test_min_succeeds(self): sm = SecurityManager() # accepts old = self.setSecurityManager(sm) self.assertEqual(guarded_min([1,2,3]), 1) self.assertEqual(guarded_min(1,2,3), 1) self.setSecurityManager(old) def test_max_succeeds(self): sm = SecurityManager() # accepts old = self.setSecurityManager(sm) self.assertEqual(guarded_max([1,2,3]), 3) self.assertEqual(guarded_max(1,2,3), 3) self.setSecurityManager(old) def test_enumerate_succeeds(self): sm = SecurityManager() # accepts old = self.setSecurityManager(sm) enum = guarded_enumerate([1,2,3]) self.assertEqual(enum.next(), (0,1)) self.assertEqual(enum.next(), (1,2)) self.assertEqual(enum.next(), (2,3)) self.assertRaises(StopIteration, enum.next) self.setSecurityManager(old) def test_sum_succeeds(self): sm = SecurityManager() # accepts old = self.setSecurityManager(sm) self.assertEqual(guarded_sum([1,2,3]), 6) self.assertEqual(guarded_sum([1,2,3], start=36), 42) self.setSecurityManager(old) def test_apply(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) gapply = safe_builtins['apply'] def f(a=1, b=2): return a+b # This one actually succeeds, because apply isn't given anything # to unpack. self.assertEqual(gapply(f), 3) # Likewise, because the things passed are empty. self.assertEqual(gapply(f, (), {}), 3) self.assertRaises(Unauthorized, gapply, f, [1]) self.assertRaises(Unauthorized, gapply, f, (), {'a': 2}) self.assertRaises(Unauthorized, gapply, f, [1], {'a': 2}) sm = SecurityManager(0) # accepts self.setSecurityManager(sm) self.assertEqual(gapply(f), 3) self.assertEqual(gapply(f, (), {}), 3) self.assertEqual(gapply(f, [0]), 2) self.assertEqual(gapply(f, [], {'b': 18}), 19) self.assertEqual(gapply(f, [10], {'b': 1}), 11) self.setSecurityManager(old) class TestGuardedDictListTypes(unittest.TestCase): def testDictCreation(self): d = safe_builtins['dict'] self.assertEquals(d(), {}) self.assertEquals(d({1:2}), {1:2}) self.assertEquals(d(((1,2),)), {1:2}) self.assertEquals(d(foo=1), {"foo":1}) self.assertEquals(d.fromkeys((1,2,3)), {1:None, 2:None, 3:None}) self.assertEquals(d.fromkeys((1,2,3), 'f'), {1:'f', 2:'f', 3:'f'}) def testListCreation(self): l = safe_builtins['list'] self.assertEquals(l(), []) self.assertEquals(l([1,2,3]), [1,2,3]) x = [3,2,1] self.assertEquals(l(x), [3,2,1]) if sys.version_info >= (2, 4): self.assertEquals(sorted(x), [1,2,3]) class TestRestrictedPythonApply(GuardTestCase): def test_apply(self): sm = SecurityManager(1) # rejects old = self.setSecurityManager(sm) gapply = guarded_apply def f(a=1, b=2): return a+b # This one actually succeeds, because apply isn't given anything # to unpack. self.assertEqual(gapply(*(f,)), 3) # Likewise, because the things passed are empty. self.assertEqual(gapply(*(f,), **{}), 3) self.assertRaises(Unauthorized, gapply, *(f, 1)) self.assertRaises(Unauthorized, gapply, *(f,), **{'a': 2}) self.assertRaises(Unauthorized, gapply, *(f, 1), **{'a': 2}) sm = SecurityManager(0) # accepts self.setSecurityManager(sm) self.assertEqual(gapply(*(f,)), 3) self.assertEqual(gapply(*(f,), **{}), 3) self.assertEqual(gapply(*(f, 0)), 2) self.assertEqual(gapply(*(f,), **{'b': 18}), 19) self.assertEqual(gapply(*(f, 10), **{'b': 1}), 11) self.setSecurityManager(old) # Map function name to the # of times it's been called. wrapper_count = {} class FuncWrapper: def __init__(self, funcname, func): self.funcname = funcname wrapper_count[funcname] = 0 self.func = func def __call__(self, *args, **kws): wrapper_count[self.funcname] += 1 return self.func(*args, **kws) def __repr__(self): return "" % self.func # Given the high wall between AccessControl and RestrictedPython, I suppose # the next one could be called an integration test. But we're simply # trying to run restricted Python with the *intended* implementations of # the special wrappers here, so no apologies. _ProtectedBase = None class TestActualPython(GuardTestCase): _old_mgr = _old_policy = _marker = [] def setUp(self): self._wrapped_dicts = [] def tearDown( self ): self._restorePolicyAndManager() for munged, orig in self._wrapped_dicts: munged.update(orig) del self._wrapped_dicts def _initPolicyAndManager(self, manager=None): from AccessControl.SecurityManagement import get_ident from AccessControl.SecurityManagement import _managers from AccessControl.SecurityManagement import newSecurityManager from AccessControl.SecurityManager import setSecurityPolicy from AccessControl.ZopeSecurityPolicy import ZopeSecurityPolicy class UnderprivilegedUser: """ Anonymous USer for unit testing purposes. """ def getId(self): return 'Underprivileged User' getUserName = getId def allowed(self, object, object_roles=None): return 0 def getRoles(self): return () self._policy = ZopeSecurityPolicy() self._old_policy = setSecurityPolicy(self._policy) if manager is None: thread_id = get_ident() self._old_mgr = manager=_managers.get(thread_id, self._marker) newSecurityManager(None, UnderprivilegedUser()) else: self._old_mgr = self.setSecurityManager(manager) def _restorePolicyAndManager(self): from AccessControl.SecurityManagement import noSecurityManager from AccessControl.SecurityManager import setSecurityPolicy if self._old_mgr is not self._marker: self.setSecurityManager(self._old_mgr) else: noSecurityManager() if self._old_policy is not self._marker: setSecurityPolicy(self._old_policy) def _getProtectedBaseClass(self): from AccessControl.SecurityInfo import ClassSecurityInfo from ExtensionClass import Base from Globals import InitializeClass global _ProtectedBase if _ProtectedBase is None: class ProtectedBase(Base): security = ClassSecurityInfo() security.declarePrivate('private_method') def private_method(self): return 'private_method called' InitializeClass(ProtectedBase) _ProtectedBase = ProtectedBase return _ProtectedBase def testPython(self): from RestrictedPython.tests import verify code, its_globals = self._compile("actual_python.py") verify.verify(code) # Fiddle the global and safe-builtins dicts to count how many times # the special functions are called. self._wrap_replaced_dict_callables(its_globals) self._wrap_replaced_dict_callables(its_globals['__builtins__']) sm = SecurityManager() old = self.setSecurityManager(sm) try: exec code in its_globals finally: self.setSecurityManager(old) # Use wrapper_count to determine coverage. ## print wrapper_count # uncomment to see wrapper names & counts untouched = [k for k, v in wrapper_count.items() if v == 0] if untouched: untouched.sort() self.fail("Unexercised wrappers: %r" % untouched) def testPythonRealAC(self): code, its_globals = self._compile("actual_python.py") exec code in its_globals def test_derived_class_normal(self): from RestrictedPython.tests import verify NORMAL_SCRIPT = """ class Normal(ProtectedBase): pass normal = Normal() print normal.private_method() """ code, its_globals = self._compile_str(NORMAL_SCRIPT, 'normal_script') its_globals['ProtectedBase'] = self._getProtectedBaseClass() verify.verify(code) self._initPolicyAndManager() try: exec code in its_globals except Unauthorized: pass else: self.fail("Didn't raise Unauthorized: \n%s" % its_globals['_print']()) def test_derived_class_sneaky_en_suite(self): # Disallow declaration of security-affecting names in classes # defined in restricted code (compile-time check). from RestrictedPython.tests import verify SNEAKY_SCRIPT = """ class Sneaky(ProtectedBase): private_method__roles__ = None sneaky = Sneaky() print sneaky.private_method() """ try: code, its_globals = self._compile_str(SNEAKY_SCRIPT, 'sneaky_script') except SyntaxError: pass else: self.fail("Didn't raise SyntaxError!") def test_derived_sneaky_post_facto(self): # Assignment to a class outside its suite fails at # compile time with a SyntaxError. from RestrictedPython.tests import verify SNEAKY_SCRIPT = """ class Sneaky(ProtectedBase): pass Sneaky.private_method__roles__ = None sneaky = Sneaky() print sneaky.private_method() """ try: code, its_globals = self._compile_str(SNEAKY_SCRIPT, 'sneaky_script') except SyntaxError: pass else: self.fail("Didn't raise SyntaxError!") def test_derived_sneaky_instance(self): # Assignment of security-sensitive names to an instance # fails at compile time with a SyntaxError. from RestrictedPython.tests import verify SNEAKY_SCRIPT = """ class Sneaky(ProtectedBase): pass sneaky = Sneaky() sneaky.private_method__roles__ = None print sneaky.private_method() """ try: code, its_globals = self._compile_str(SNEAKY_SCRIPT, 'sneaky_script') except SyntaxError: pass else: self.fail("Didn't raise SyntaxError!") def test_dict_access(self): from RestrictedPython.tests import verify SIMPLE_DICT_ACCESS_SCRIPT = """ def foo(text): return text kw = {'text':'baz'} print foo(**kw) kw = {'text':True} print foo(**kw) """ code, its_globals = self._compile_str(SIMPLE_DICT_ACCESS_SCRIPT, 'x') verify.verify(code) sm = SecurityManager() old = self.setSecurityManager(sm) try: exec code in its_globals finally: self.setSecurityManager(old) self.assertEqual(its_globals['_print'](), 'baz\nTrue\n') def _compile_str(self, text, name): from RestrictedPython import compile_restricted from AccessControl.ZopeGuards import get_safe_globals, guarded_getattr code = compile_restricted(text, name, 'exec') g = get_safe_globals() g['_getattr_'] = guarded_getattr g['__debug__'] = 1 # so assert statements are active g['__name__'] = __name__ # so classes can be defined in the script return code, g def testPythonRealAC(self): code, its_globals = self._compile("actual_python.py") exec code in its_globals # Compile code in fname, as restricted Python. Return the # compiled code, and a safe globals dict for running it in. # fname is the string name of a Python file; it must be found # in the same directory as this file. def _compile(self, fname): from RestrictedPython import compile_restricted from AccessControl.ZopeGuards import get_safe_globals, guarded_getattr fn = os.path.join( _HERE, fname) text = open(fn).read() return self._compile_str(text, fn) # d is a dict, the globals for execution or our safe builtins. # The callable values which aren't the same as the corresponding # entries in __builtin__ are wrapped in a FuncWrapper, so we can # tell whether they're executed. def _wrap_replaced_dict_callables(self, d): import __builtin__ orig = d.copy() self._wrapped_dicts.append((d, orig)) for k, v in d.items(): if callable(v) and v is not getattr(__builtin__, k, None): d[k] = FuncWrapper(k, v) def test_inplacevar(): """ Verify the correct behavior of protected_inplacevar. >>> from AccessControl.ZopeGuards import protected_inplacevar Basic operations on objects without inplace slots work as expected: >>> protected_inplacevar('+=', 1, 2) 3 >>> protected_inplacevar('-=', 5, 2) 3 >>> protected_inplacevar('*=', 5, 2) 10 >>> protected_inplacevar('/=', 6, 2) 3 >>> protected_inplacevar('%=', 5, 2) 1 >>> protected_inplacevar('**=', 5, 2) 25 >>> protected_inplacevar('<<=', 5, 2) 20 >>> protected_inplacevar('>>=', 5, 2) 1 >>> protected_inplacevar('&=', 5, 2) 0 >>> protected_inplacevar('^=', 7, 2) 5 >>> protected_inplacevar('|=', 5, 2) 7 Inplace operations are allowed on lists: >>> protected_inplacevar('+=', [1], [2]) [1, 2] >>> protected_inplacevar('*=', [1], 2) [1, 1] But not on custom objects: >>> class C: ... def __iadd__(self, other): ... return 42 >>> protected_inplacevar('+=', C(), 2) # doctest: +NORMALIZE_WHITESPACE Traceback (most recent call last): ... TypeError: Augmented assignment to C objects is not allowed in untrusted code """ if sys.version_info[:2] >= (2, 4): def test_inplacevar_for_py24(): """ protected_inplacevar allows inplce ops on sets: >>> from AccessControl.ZopeGuards import protected_inplacevar >>> s = set((1,2,3,4)) >>> sorted(protected_inplacevar('-=', s, set((1, 3)))) [2, 4] >>> sorted(s) [2, 4] >>> sorted(protected_inplacevar('|=', s, set((1, 3, 9)))) [1, 2, 3, 4, 9] >>> sorted(s) [1, 2, 3, 4, 9] >>> sorted(protected_inplacevar('&=', s, set((1, 2, 3, 9)))) [1, 2, 3, 9] >>> sorted(s) [1, 2, 3, 9] >>> sorted(protected_inplacevar('^=', s, set((1, 3, 7, 8)))) [2, 7, 8, 9] >>> sorted(s) [2, 7, 8, 9] """ def test_suite(): suite = unittest.TestSuite([ doctest.DocTestSuite(), ]) for cls in (TestGuardedGetattr, TestGuardedGetitem, TestDictGuards, TestBuiltinFunctionGuards, TestListGuards, TestGuardedDictListTypes, TestRestrictedPythonApply, TestActualPython, ): suite.addTest(unittest.makeSuite(cls)) return suite if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/AccessControl/tests/testDeprecatedAPI.py0000664000076400007640000000476710435500320024405 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests for the DeprecationWarning thrown by accessing hasRole To be removed together with the API in due time. """ __rcs_id__='$Id: testDeprecatedAPI.py 24763 2004-05-17 05:59:28Z philikon $' __version__='$Revision: 1.6 $'[11:-2] import ZODB # Sigh. Persistent needs to be set, so we import ZODB. from AccessControl import User import unittest, warnings class DeprecatedAPI(unittest.TestCase): def setUp(self): # We test for warnings by turning them into exceptions warnings.filterwarnings('error', category=DeprecationWarning, module='AccessControl') def testDeprecatedHasRole(self): # hasRole has been deprecated, we expect a warning. try: self.userObject.hasRole(None) except DeprecationWarning: pass else: self.fail('Expected DeprecationWarning, none given') def testAllowed(self): # hasRole is an alias for allowed, which should be unaffected. try: self.userObject.allowed(None) except DeprecationWarning: self.fail('Unexpected DeprecationWarning, ' 'no warnings expected here') else: pass def tearDown(self): warnings.resetwarnings() warnings.simplefilter("ignore", category=PendingDeprecationWarning) warnings.simplefilter("ignore", category=OverflowWarning) class BasicUser(DeprecatedAPI): userObject = User.SimpleUser('JoeBloke', '123', [], []) class UnrestrictedUser(DeprecatedAPI): userObject = User.UnrestrictedUser('Special', '123', [], []) class NullUnrestrictedUser(DeprecatedAPI): userObject = User.NullUnrestrictedUser() def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(BasicUser)) suite.addTest(unittest.makeSuite(UnrestrictedUser)) suite.addTest(unittest.makeSuite(NullUnrestrictedUser)) return suite Zope-2.8.7-final/lib/python/AccessControl/tests/mixed_module/0000775000076400007640000000000010435610645023212 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/mixed_module/__init__.py0000664000076400007640000000011410435500320025303 0ustar ajungajung# test module, partially private def priv(): pass def pub(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/mixed_module/submodule/0000775000076400007640000000000010435610645025211 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/tests/mixed_module/submodule/__init__.py0000664000076400007640000000005510435500320027306 0ustar ajungajung# test module, private def priv(): pass Zope-2.8.7-final/lib/python/AccessControl/tests/testClassSecurityInfo.py0000664000076400007640000000546610435500320025421 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Unit tests for ClassSecurityInfo. """ import unittest class ClassSecurityInfoTests(unittest.TestCase): def _getTargetClass(self): from AccessControl.SecurityInfo import ClassSecurityInfo return ClassSecurityInfo def test_SetPermissionDefault(self): # Test setting default roles for permissions. import Globals # XXX: avoiding import cycle from App.class_init import default__class_init__ from ExtensionClass import Base ClassSecurityInfo = self._getTargetClass() # Setup a test class with default role -> permission decls. class Test(Base): """Test class """ __ac_roles__ = ('Role A', 'Role B', 'Role C') meta_type = "Test" security = ClassSecurityInfo() security.setPermissionDefault('Make food', ('Chef',)) security.setPermissionDefault( 'Test permission', ('Manager', 'Role A', 'Role B', 'Role C') ) security.declareProtected('Test permission', 'foo') def foo(self, REQUEST=None): """ """ pass # Do class initialization. default__class_init__(Test) # Now check the resulting class to see if the mapping was made # correctly. Note that this uses carnal knowledge of the internal # structures used to store this information! object = Test() imPermissionRole = [r for r in object.foo__roles__ if not r.endswith('_Permission')] self.failUnless(len(imPermissionRole) == 4) for item in ('Manager', 'Role A', 'Role B', 'Role C'): self.failUnless(item in imPermissionRole) # Make sure that a permission defined without accompanying method # is still reflected in __ac_permissions__ self.assertEquals([t for t in Test.__ac_permissions__ if not t[1]], [('Make food', (), ('Chef',))]) def test_suite(): suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(ClassSecurityInfoTests)) return suite if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/AccessControl/Implementation.py0000664000076400007640000000717610435500321022734 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Controller that can switch between security machinery implementations. This module allows configuration of the security implementation after the initial import of the modules. It is intended to allow runtime selection of the machinery based on Zope's configuration file. The helper function defined here switches between the 'C' and 'PYTHON' security implementations by loading the appropriate implementation module and wiring the implementation into the other modules of the AccessControl package that defined the various components before this module was introduced. """ def getImplementationName(): """Return the name of the implementation currently being used.""" return _implementation_name def setImplementation(name): """Select the policy implementation to use. The 'name' must be either 'PYTHON' or 'C'. NOTE: this function is intended to be called exactly once, so that the Zope config file can dictate the policy implementation to be used. Subsequent calls to this function will have no effect!! """ import sys global _implementation_name global _implementation_set if _implementation_set: return name = name.upper() if name == _implementation_name: return if name == "C": from AccessControl import ImplC as impl elif name == "PYTHON": from AccessControl import ImplPython as impl else: raise ValueError("unknown policy implementation: %r" % name) _implementation_name = name for modname, names in _policy_names.items(): __import__(modname) mod = sys.modules[modname] for n in names: setattr(mod, n, getattr(impl, n)) if hasattr(mod, "initialize"): mod.initialize(impl) from AccessControl.SecurityManager import setSecurityPolicy policy = impl.ZopeSecurityPolicy(True, True) setSecurityPolicy(policy) _implementation_set = 1 _implementation_name = None _implementation_set = 0 _policy_names = { "AccessControl": ("setDefaultBehaviors", ), "AccessControl.DTML": ("RestrictedDTML", ), "AccessControl.PermissionRole": ("_what_not_even_god_should_do", "rolesForPermissionOn", "PermissionRole", "imPermissionRole", ), "AccessControl.SecurityManagement": ("SecurityManager", ), "AccessControl.SecurityManager": ("SecurityManager", ), "AccessControl.ZopeGuards": ("aq_validate", "guarded_getattr", ), "AccessControl.ZopeSecurityPolicy": ("ZopeSecurityPolicy", ), } # start with the default, mostly because we need something for the tests setImplementation("C") # allow the implementation to change from the default _implementation_set = 0 Zope-2.8.7-final/lib/python/AccessControl/www/0000775000076400007640000000000010435500320020205 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/www/User_icon.gif0000775000076400007640000000154010435500320022625 0ustar ajungajungGIF89aPPP@@@@@!,=HA8XPBBpE3>Q#CAN1ȇ !VD`K 1 ;Zope-2.8.7-final/lib/python/AccessControl/www/AccessControl_icon.gif0000664000076400007640000000163210435500320024450 0ustar ajungajungGIF89aPPP@@@@@!,@w @0 @ 8a$HF t<!0aRd L8!C3)2Ĉ2$ :#ϖ2řtȗ;#։-?R O;Zope-2.8.7-final/lib/python/AccessControl/securitySuite/0000775000076400007640000000000010435610645022256 5ustar ajungajungZope-2.8.7-final/lib/python/AccessControl/securitySuite/README0000664000076400007640000000040610435500320023122 0ustar ajungajungArchitecture of the Zope SecuritySuite The base class of the suite is "SecurityBase" and contains the basic function of perform tests for permissions, roles and perform ZPublisher request: - _checkPermission() - _checkRoles() - _request() Zope-2.8.7-final/lib/python/AccessControl/securitySuite/regressionSecurity.py0000775000076400007640000001745510435500320026543 0ustar ajungajung#!/usr/bin/env python2.1 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # $Id: regressionSecurity.py 24763 2004-05-17 05:59:28Z philikon $ import os, sys, unittest import ZODB import SecurityBase from OFS.Folder import Folder from OFS.SimpleItem import SimpleItem from AccessControl import ClassSecurityInfo,getSecurityManager from AccessControl.User import nobody import Globals # let's define some permissions first MAGIC_PERMISSION1 = 'Magic Permission 1' MAGIC_PERMISSION2 = 'Magic Permission 2' ############################################################################## # TestObject class ############################################################################## class TestObject(SimpleItem): """ test object """ security = ClassSecurityInfo() __allow_access_to_unprotected_subobjects__ = 0 public_attr = 1 _protected_attr = 2 def __init__(self,id): self.id = id security.declarePrivate("private_func") def private_func(self): """ private func """ return "i am private" def manage_func(self): """ should be protected by manager role """ return "i am your manager function" security.declareProtected(MAGIC_PERMISSION2,"manage_func2") def manage_func2(self): """ should be protected by manager role """ return "i am your manager function2" security.declareProtected(MAGIC_PERMISSION1,"protected_func") def protected_func(self): """ proteced func """ return "i am protected " security.declarePublic("public_func") def public_func(self): """ public func """ return "i am public" security.setPermissionDefault(MAGIC_PERMISSION1, ("Manager","Owner")) security.setPermissionDefault(MAGIC_PERMISSION2, ("TestRole",)) Globals.InitializeClass(TestObject) ############################################################################## # Testfolder class ############################################################################## class TestFolder(Folder): """ test class """ def __init__(self,id): self.id = id def getId(self): return self.id meta_type = 'TestFolder' security = ClassSecurityInfo() Globals.InitializeClass(TestFolder) ############################################################################## # User Class ############################################################################## class User: def __init__(self,username,password,roles): self.username = username self.password = password self.roles = roles def auth(self): return "%s:%s" % (self.username,self.password) def __str__(self): return "User(%s:%s:%s)" % (self.username,self.password,self.roles) __repr__ = __str__ USERS = ( User('user1','123',[]), User('user2','123',[]), User('owner','123',('Owner',)), User('manager','123',('Manager',)) ) def getAuth(username): for user in USERS: if user.username==username: return "%s:%s" % (user.username,user.password) raise KeyError,"no such username: %" % username class AVeryBasicSecurityTest(SecurityBase.SecurityBase): ################################################################ # set up the test hierachy of objects ################################################################ def setUp(self): """ my setup """ self.root = SecurityBase.app acl = self.root.acl_users for user in USERS: try: acl._delUsers( user.username ) except: pass for user in USERS: acl._addUser(user.username,user.password,user.password, user.roles, []) # try to remove old crap if 'test' in self.root.objectIds(): self.root._delObject('test') # setup Folder hierarchy test = TestFolder('test') f1 = TestFolder('f1') f2 = TestFolder('f2') f3 = TestFolder('f3') obj = TestObject('obj3') anonobj = TestObject('anonobj') anonobj.__roles__ = ('Anonymous',) self.root._setObject('test',test) self.root.test._setObject('f1',f1) self.root.test._setObject('f2',f2) self.root.test.f1._setObject('anonobj',anonobj) self.root.test.f2._setObject('f3',f3) self.root.test.f2.f3._setObject('obj3',obj) def testNobody(self): """ check permissions for nobody user """ self._checkPermission(nobody,'test.f1', 'View',1) self._checkPermission(nobody,'test.f2', 'View',1) self._checkPermission(nobody,'test.f2.f3','View',1) self._checkPermission(nobody,'test.f1', MAGIC_PERMISSION1, None) self._checkPermission(nobody,'test.f2', MAGIC_PERMISSION1, None) self._checkPermission(nobody,'test.f2.f3',MAGIC_PERMISSION1, None) def testPermissionAccess(self): """ check permission based access """ self._checkRoles('test.f2.f3.obj3.public_func', None) self._checkRoles('test.f2.f3.obj3.protected_func', ('Manager','Owner')) self._checkRoles('test.f2.f3.obj3.manage_func', ('Manager',)) self._checkRoles('test.f2.f3.obj3.private_func', ()) def testZPublisherAccess(self): """ test access through ZPublisher """ _r = [ ('/test/f1/', None, 200), ('/test/f2', None, 200), ('/test/f2/f3', None, 200), ('/test/f2/f3/obj3/public_func', None, 200), ('/test/f2/f3/obj3/protected_func', None, 401), ('/test/f2/f3/obj3/manage_func', None, 401), ('/test/f2/f3/obj3/private_func', None, 401), ('/test/f1/', getAuth('manager'), 200), ('/test/f2', getAuth('manager'), 200), ('/test/f2/f3', getAuth('manager'), 200), ('/test/f2/f3/obj3/public_func', getAuth('manager'), 200), ('/test/f2/f3/obj3/protected_func', getAuth('manager'), 200), ('/test/f2/f3/obj3/manage_func', getAuth('manager'), 200), ('/test/f2/f3/obj3/private_func', getAuth('manager'), 401), ('/test/f1/', getAuth('owner'), 200), ('/test/f2', getAuth('owner'), 200), ('/test/f2/f3', getAuth('owner'), 200), ('/test/f2/f3/obj3/public_func', getAuth('owner'), 200), ('/test/f2/f3/obj3/protected_func', getAuth('owner'), 200), ('/test/f2/f3/obj3/manage_func', getAuth('owner'), 401), ('/test/f2/f3/obj3/private_func', getAuth('owner'), 401), ] for path,auth,expected in _r: if auth: res = self._checkRequest(path,u=auth,expected=expected) else: res = self._checkRequest(path,expected=expected) def test_suite(): return unittest.makeSuite(AVeryBasicSecurityTest) def main(): unittest.TextTestRunner().run(test_suite()) if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/AccessControl/securitySuite/ResultObject.py0000664000076400007640000000241110435500320025217 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## class ResultObject: """ result object used for keeping results from the ZPublisher.Zope() calls $Id: ResultObject.py 24763 2004-05-17 05:59:28Z philikon $ """ def __str__(self,expected=-1,with_output=1): s = '\n' s+= '-'*78 s+= "\nRequest: %s" % self.request s+= "\nUser: %s" % self.user s+= "\nExpected: %s" % expected + " got: %s %s" % (self.code,self.return_text) if with_output: s+= "\nOutput:" s+= self.output s+= "\n" return s __repr__ = __str__ def __call__(self,expected=-1): return self.__str__(expected) Zope-2.8.7-final/lib/python/AccessControl/securitySuite/framework.py0000664000076400007640000000270510435500320024615 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## ###################################################################### # Set up unit testing framework # # The following code should be at the top of every test module: # # import os, sys # execfile(os.path.join(sys.path[0], 'framework.py')) # # ...and the following at the bottom: # # framework() # Find the Testing package if not sys.modules.has_key('Testing'): p0 = sys.path[0] if p0 and __name__ == '__main__': os.chdir(p0) p0 = '' p = d = os.path.abspath(os.curdir) while d: if os.path.isdir(os.path.join(p, 'Testing')): sys.path[:1] = [p0, os.pardir, p] break p, d = os.path.split(p) else: print 'Unable to locate Testing package.' sys.exit(1) import Testing, unittest execfile(os.path.join(os.path.split(Testing.__file__)[0], 'common.py')) Zope-2.8.7-final/lib/python/AccessControl/securitySuite/SecurityBase.py0000664000076400007640000001412610435500320025222 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import sys, re, unittest, cStringIO import ZPublisher, ResultObject import OFS.Application import AccessControl.SecurityManagement # Set up a publishable, non-ZODB Zope application. app = OFS.Application.Application() def index_html(): " " return "This is index_html." app.index_html = index_html # Will index_html ever go away? ;-) class BoboApplication: # OFS.Application has a __bobo_traverse__ that ZPublisher thinks # it should use to find the "real" root of the application. # This class gets around that. def __bobo_traverse__(self, request, name=None): return app # ZPublisher will look for these vars. bobo_application = BoboApplication() zpublisher_validated_hook=AccessControl.SecurityManagement.newSecurityManager __bobo_before__=AccessControl.SecurityManagement.noSecurityManager class SecurityBase(unittest.TestCase) : """ Base class for all security tests $Id: SecurityBase.py 24763 2004-05-17 05:59:28Z philikon $ """ status_regex = re.compile("Status: ([0-9]{1,4}) (.*)",re.I)\ ################################################################ # print the object hierachy ################################################################ def _testHierarchy(self): """ print all test objects, permissions and roles """ self._PrintTestEnvironment(root=self.root.test) def _PrintTestEnvironment(self,root): """ print recursive all objects """ print '....'*len(root.getPhysicalPath()),root.getId() folderObjs = [] for id,obj in root.objectItems(): if obj.meta_type in ['Folder','TestFolder']: folderObjs.append(obj) else: print ' '*(1+len(root.getPhysicalPath())),obj.getId(), print getattr(obj,"__roles__",(None,)) for folder in folderObjs: self._PrintTestEnvironment(folder) ################################################################ # Check functions for permissions, roles and friends ################################################################ def _checkPermission(self, user, hier, perm, expected): """ permission check on an objects for a given user. -- 'user' is a user object as returned from a user folder -- 'hier' is the path to the object in the notation 'f1.f2.f3.obj' -- 'perm' is a permission name -- 'expected' is either 0 or 1 """ s = "self.root.%s" % hier obj = eval(s) res = user.has_permission(perm,obj) if res != expected: raise AssertionError, \ self._perm_debug (s,perm,res,expected) def _checkRoles(self,hier,expected_roles=()): """ check roles for a given object. -- 'hier' is the path to the object in the notation 'f1.f2.f3.obj' -- 'expected_roles' is a sequence of expected roles """ s = "self.root.%s.__roles__" % hier roles = eval(s) same = 0 if roles is None or expected_roles is None: if (roles is None or tuple(roles) == ('Anonymous',)) and ( expected_roles is None or tuple(expected_roles) == ('Anonymous',)): same = 1 else: got = {} for r in roles: got[r] = 1 expected = {} for r in expected_roles: expected[r] = 1 if got == expected: # Dict compare does the Right Thing. same = 1 if not same: raise AssertionError, self._roles_debug(hier,roles,expected_roles) def _checkRequest(self,*args,**kw): """ perform a ZPublisher request """ expected_code = kw.get('expected',200) del kw['expected'] res = apply(self._request,args,kw) if expected_code != res.code: raise AssertionError, \ self._request_debug(res,expected_code,args,kw) ################################################################ # Debugging helpers when raising AssertionError ################################################################ def _perm_debug(self, obj , perm, res, expected): s+= 'Object: %s' % obj s+= ', Permission: %s' % perm s+= ', has permission: %s' % res s+= ', expected: %s' % expected return s def _roles_debug(self,hier,got_roles,expected_roles): s = 'Object: %s' % hier s+= ', has roles: %s' % `got_roles` s+= ', expected roles: %s' % `expected_roles` return s def _request_debug(self,res,expected,args,kw): s = 'Args: %s' % str(args) s+= ', KW: %s' % str(kw) s+= '\n%s\n' % res.__str__(with_output=0,expected=expected) return s def _request(self,*args,**kw): """ perform a Zope request """ io =cStringIO.StringIO() kw['fp']=io # Publish this module. testargs = (__name__,) + args real_stdout = sys.stdout garbage_out = cStringIO.StringIO() sys.stdout = garbage_out # Silence, ZPublisher! try: ZPublisher.test(*testargs,**kw) finally: sys.stdout = real_stdout outp = io.getvalue() mo = self.status_regex.search(outp) code,txt = mo.groups() res = ResultObject.ResultObject() res.request = args res.user = kw.get('u','') res.code = int(code) res.return_text = txt res.output = outp return res Zope-2.8.7-final/lib/python/AccessControl/ZopeSecurityPolicy.py0000664000076400007640000000365510435500321023572 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Define Zope's default security policy $Id: ZopeSecurityPolicy.py 29712 2005-03-28 22:38:24Z tseaver $""" from types import MethodType # AccessControl.Implementation inserts: # ZopeSecurityPolicy, getRoles, rolesForPermissionOn from AccessControl.SimpleObjectPolicies import _noroles rolesForPermissionOn = None # XXX: avoid import loop tuple_or_list = tuple, list def getRoles(container, name, value, default): global rolesForPermissionOn # XXX: avoid import loop if rolesForPermissionOn is None: from PermissionRole import rolesForPermissionOn roles = getattr(value, '__roles__', _noroles) if roles is _noroles: if not name or not isinstance(name, basestring): return default if type(value) is MethodType: container = value.im_self cls = getattr(container, '__class__', None) if cls is None: return default roles = getattr(cls, name+'__roles__', _noroles) if roles is _noroles: return default value = container if roles is None or isinstance(roles, tuple_or_list): return roles rolesForPermissionOn = getattr(roles, 'rolesForPermissionOn', None) if rolesForPermissionOn is not None: roles = rolesForPermissionOn(value) return roles Zope-2.8.7-final/lib/python/AccessControl/User.py0000664000076400007640000012626210435500321020663 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Access control package""" __version__='$Revision: 1.182 $'[11:-2] import os import Globals, socket, SpecialUsers,re from Globals import DTMLFile, MessageDialog, Persistent, PersistentMapping from App.Management import Navigation, Tabs from Acquisition import Implicit from OFS.SimpleItem import Item from base64 import decodestring from App.ImageFile import ImageFile from Role import RoleManager, DEFAULTMAXLISTUSERS from PermissionRole import _what_not_even_god_should_do, rolesForPermissionOn import AuthEncoding from AccessControl import getSecurityManager from zExceptions import Unauthorized, BadRequest from AccessControl.SecurityManagement import newSecurityManager from AccessControl.SecurityManagement import noSecurityManager from AccessControl.ZopeSecurityPolicy import _noroles _marker=[] class BasicUser(Implicit): """Base class for all User objects""" # ---------------------------- # Public User object interface # ---------------------------- # Maybe allow access to unprotected attributes. Note that this is # temporary to avoid exposing information but without breaking # everyone's current code. In the future the security will be # clamped down and permission-protected here. Because there are a # fair number of user object types out there, this method denies # access to names that are private parts of the standard User # interface or implementation only. The other approach (only # allowing access to public names in the User interface) would # probably break a lot of other User implementations with extended # functionality that we cant anticipate from the base scaffolding. def __allow_access_to_unprotected_subobjects__(self, name, value=None): deny_names=('name', '__', 'roles', 'domains', '_getPassword', 'authenticate', '_shared_roles') if name in deny_names: return 0 return 1 def __init__(self,name,password,roles,domains): raise NotImplementedError def getUserName(self): """Return the username of a user""" raise NotImplementedError def getId(self): """Get the ID of the user. The ID can be used, at least from Python, to get the user from the user's UserDatabase""" return self.getUserName() def _getPassword(self): """Return the password of the user.""" raise NotImplementedError def getRoles(self): """Return the list of roles assigned to a user.""" raise NotImplementedError def getRolesInContext(self, object): """Return the list of roles assigned to the user, including local roles assigned in context of the passed in object.""" userid=self.getId() roles=self.getRoles() local={} object=getattr(object, 'aq_inner', object) while 1: local_roles = getattr(object, '__ac_local_roles__', None) if local_roles: if callable(local_roles): local_roles=local_roles() dict=local_roles or {} for r in dict.get(userid, []): local[r]=1 inner = getattr(object, 'aq_inner', object) parent = getattr(inner, 'aq_parent', None) if parent is not None: object = parent continue if hasattr(object, 'im_self'): object=object.im_self object=getattr(object, 'aq_inner', object) continue break roles=list(roles) + local.keys() return roles def getDomains(self): """Return the list of domain restrictions for a user""" raise NotImplementedError # ------------------------------ # Internal User object interface # ------------------------------ def authenticate(self, password, request): passwrd=self._getPassword() result = AuthEncoding.pw_validate(passwrd, password) domains=self.getDomains() if domains: return result and domainSpecMatch(domains, request) return result def _shared_roles(self, parent): r=[] while 1: if hasattr(parent,'__roles__'): roles=parent.__roles__ if roles is None: return 'Anonymous', if 'Shared' in roles: roles=list(roles) roles.remove('Shared') r=r+roles else: try: return r+list(roles) except: return r if hasattr(parent, 'aq_parent'): while hasattr(parent.aq_self,'aq_self'): parent=parent.aq_self parent=parent.aq_parent else: return r def _check_context(self, object): # Check that 'object' exists in the acquisition context of # the parent of the acl_users object containing this user, # to prevent "stealing" access through acquisition tricks. # Return true if in context, false if not or if context # cannot be determined (object is not wrapped). parent = getattr(self, 'aq_parent', None) context = getattr(parent, 'aq_parent', None) if context is not None: if object is None: return 1 if not hasattr(object, 'aq_inContextOf'): if hasattr(object, 'im_self'): # This is a method. Grab its self. object=object.im_self if not hasattr(object, 'aq_inContextOf'): # Object is not wrapped, so return false. return 0 return object.aq_inContextOf(context, 1) # This is lame, but required to keep existing behavior. return 1 def allowed(self, object, object_roles=None): """Check whether the user has access to object. The user must have one of the roles in object_roles to allow access.""" if object_roles is _what_not_even_god_should_do: return 0 # Short-circuit the common case of anonymous access. if object_roles is None or 'Anonymous' in object_roles: return 1 # Provide short-cut access if object is protected by 'Authenticated' # role and user is not nobody if 'Authenticated' in object_roles and ( self.getUserName() != 'Anonymous User'): if self._check_context(object): return 1 # Check for ancient role data up front, convert if found. # This should almost never happen, and should probably be # deprecated at some point. if 'Shared' in object_roles: object_roles = self._shared_roles(object) if object_roles is None or 'Anonymous' in object_roles: return 1 # Check for a role match with the normal roles given to # the user, then with local roles only if necessary. We # want to avoid as much overhead as possible. user_roles = self.getRoles() for role in object_roles: if role in user_roles: if self._check_context(object): return 1 return None # Still have not found a match, so check local roles. We do # this manually rather than call getRolesInContext so that # we can incur only the overhead required to find a match. inner_obj = getattr(object, 'aq_inner', object) userid = self.getId() while 1: local_roles = getattr(inner_obj, '__ac_local_roles__', None) if local_roles: if callable(local_roles): local_roles = local_roles() dict = local_roles or {} local_roles = dict.get(userid, []) for role in object_roles: if role in local_roles: if self._check_context(object): return 1 return 0 inner = getattr(inner_obj, 'aq_inner', inner_obj) parent = getattr(inner, 'aq_parent', None) if parent is not None: inner_obj = parent continue if hasattr(inner_obj, 'im_self'): inner_obj=inner_obj.im_self inner_obj=getattr(inner_obj, 'aq_inner', inner_obj) continue break return None def hasRole(self, *args, **kw): """hasRole is an alias for 'allowed' and has been deprecated. Code still using this method should convert to either 'has_role' or 'allowed', depending on the intended behaviour. """ import warnings warnings.warn('BasicUser.hasRole is deprecated, please use ' 'BasicUser.allowed instead; hasRole was an alias for allowed, but ' 'you may have ment to use has_role.', DeprecationWarning) return self.allowed(*args, **kw) domains=[] def has_role(self, roles, object=None): """Check to see if a user has a given role or roles.""" if isinstance(roles, str): roles=[roles] if object is not None: user_roles = self.getRolesInContext(object) else: # Global roles only... user_roles=self.getRoles() for role in roles: if role in user_roles: return 1 return 0 def has_permission(self, permission, object): """Check to see if a user has a given permission on an object.""" roles=rolesForPermissionOn(permission, object) if isinstance(roles, str): roles=[roles] return self.allowed(object, roles) def __len__(self): return 1 def __str__(self): return self.getUserName() __repr__=__str__ class SimpleUser(BasicUser): """A very simple user implementation that doesn't make a database commitment""" def __init__(self,name,password,roles,domains): self.name =name self.__ =password self.roles =roles self.domains=domains def getUserName(self): """Return the username of a user""" return self.name def _getPassword(self): """Return the password of the user.""" return self.__ def getRoles(self): """Return the list of roles assigned to a user.""" if self.name == 'Anonymous User': return tuple(self.roles) else: return tuple(self.roles) + ('Authenticated',) def getDomains(self): """Return the list of domain restrictions for a user""" return tuple(self.domains) class SpecialUser(SimpleUser): """Class for special users, like emergency user and nobody""" def getId(self): pass class User(SimpleUser, Persistent): """Standard User object""" class UnrestrictedUser(SpecialUser): """User that passes all security checks. Note, however, that modules like Owner.py can still impose restrictions. """ def allowed(self,parent,roles=None): return roles is not _what_not_even_god_should_do def hasRole(self, *args, **kw): """hasRole is an alias for 'allowed' and has been deprecated. Code still using this method should convert to either 'has_role' or 'allowed', depending on the intended behaviour. """ import warnings warnings.warn('UnrestrictedUser.hasRole is deprecated, please use ' 'UnrestrictedUser.allowed instead; hasRole was an alias for ' 'allowed, but you may have ment to use has_role.', DeprecationWarning) self.allowed(*args, **kw) def has_role(self, roles, object=None): return 1 def has_permission(self, permission, object): return 1 class NullUnrestrictedUser(SpecialUser): """User created if no emergency user exists. It is only around to satisfy third party userfolder implementations that may expect the emergency user to exist and to be able to call certain methods on it (in other words, backward compatibility). Note that when no emergency user is installed, this object that exists in its place is more of an anti-superuser since you cannot login as this user and it has no priveleges at all.""" __null_user__=1 def __init__(self): pass def getUserName(self): # return an unspellable username return (None, None) _getPassword=getUserName def getRoles(self): return () getDomains=getRoles def getRolesInContext(self, object): return () def authenticate(self, password, request): return 0 def allowed(self, parent, roles=None): return 0 def hasRole(self, *args, **kw): """hasRole is an alias for 'allowed' and has been deprecated. Code still using this method should convert to either 'has_role' or 'allowed', depending on the intended behaviour. """ import warnings warnings.warn('NullUnrestrictedUser.hasRole is deprecated, please use ' 'NullUnrestrictedUser.allowed instead; hasRole was an alias for ' 'allowed, but you may have ment to use has_role.', DeprecationWarning) self.allowed(*args, **kw) def has_role(self, roles, object=None): return 0 def has_permission(self, permission, object): return 0 def readUserAccessFile(filename): '''Reads an access file from the instance home. Returns name, password, domains, remote_user_mode. ''' import App.config cfg = App.config.getConfiguration() try: f = open(os.path.join(cfg.instancehome, filename), 'r') line = f.readline() f.close() except IOError: return None if line: data = line.strip().split(':') remote_user_mode = not data[1] try: ds = data[2].split(' ') except: ds = [] return data[0], data[1], ds, remote_user_mode else: return None # Create emergency user. _remote_user_mode=0 info = readUserAccessFile('access') if info: _remote_user_mode = info[3] emergency_user = UnrestrictedUser( info[0], info[1], ('manage',), info[2]) else: emergency_user = NullUnrestrictedUser() super = emergency_user # Note: use of the 'super' name is deprecated. del info nobody=SpecialUser('Anonymous User','',('Anonymous',), []) system=UnrestrictedUser('System Processes','',('manage',), []) # stuff these in a handier place for importing SpecialUsers.nobody=nobody SpecialUsers.system=system SpecialUsers.emergency_user=emergency_user # Note: use of the 'super' name is deprecated. SpecialUsers.super=emergency_user class BasicUserFolder(Implicit, Persistent, Navigation, Tabs, RoleManager, Item): """Base class for UserFolder-like objects""" meta_type='User Folder' id ='acl_users' title ='User Folder' isPrincipiaFolderish=1 isAUserFolder=1 maxlistusers = DEFAULTMAXLISTUSERS encrypt_passwords = 0 manage_options=( ( {'label':'Contents', 'action':'manage_main', 'help':('OFSP','User-Folder_Contents.stx')}, {'label':'Properties', 'action':'manage_userFolderProperties', 'help':('OFSP','User-Folder_Properties.stx')}, ) +RoleManager.manage_options +Item.manage_options ) __ac_permissions__=( ('Manage users', ('manage_users','getUserNames', 'getUser', 'getUsers', 'getUserById', 'user_names', 'setDomainAuthenticationMode', 'userFolderAddUser', 'userFolderEditUser', 'userFolderDelUsers', ) ), ) # ---------------------------------- # Public UserFolder object interface # ---------------------------------- def getUserNames(self): """Return a list of usernames""" raise NotImplementedError def getUsers(self): """Return a list of user objects""" raise NotImplementedError def getUser(self, name): """Return the named user object or None""" raise NotImplementedError def getUserById(self, id, default=None): """Return the user corresponding to the given id. """ # The connection between getting by ID and by name is not a strong # one user = self.getUser(id) if user is None: return default return user def _doAddUser(self, name, password, roles, domains, **kw): """Create a new user. This should be implemented by subclasses to do the actual adding of a user. The 'password' will be the original input password, unencrypted. The implementation of this method is responsible for performing any needed encryption.""" raise NotImplementedError def _doChangeUser(self, name, password, roles, domains, **kw): """Modify an existing user. This should be implemented by subclasses to make the actual changes to a user. The 'password' will be the original input password, unencrypted. The implementation of this method is responsible for performing any needed encryption.""" raise NotImplementedError def _doDelUsers(self, names): """Delete one or more users. This should be implemented by subclasses to do the actual deleting of users.""" raise NotImplementedError # As of Zope 2.5, userFolderAddUser, userFolderEditUser and # userFolderDelUsers offer aliases for the the _doAddUser, _doChangeUser # and _doDelUsers methods (with the difference that they can be called # from XML-RPC or untrusted scripting code, given the necessary # permissions). # # Authors of custom user folders don't need to do anything special to # support these - they will just call the appropriate '_' methods that # user folder subclasses already implement. def userFolderAddUser(self, name, password, roles, domains, **kw): """API method for creating a new user object. Note that not all user folder implementations support dynamic creation of user objects.""" if hasattr(self, '_doAddUser'): return self._doAddUser(name, password, roles, domains, **kw) raise NotImplementedError def userFolderEditUser(self, name, password, roles, domains, **kw): """API method for changing user object attributes. Note that not all user folder implementations support changing of user object attributes.""" if hasattr(self, '_doChangeUser'): return self._doChangeUser(name, password, roles, domains, **kw) raise NotImplementedError def userFolderDelUsers(self, names): """API method for deleting one or more user objects. Note that not all user folder implementations support deletion of user objects.""" if hasattr(self, '_doDelUsers'): return self._doDelUsers(names) raise NotImplementedError # ----------------------------------- # Private UserFolder object interface # ----------------------------------- _remote_user_mode=_remote_user_mode _domain_auth_mode=0 _emergency_user=emergency_user # Note: use of the '_super' name is deprecated. _super=emergency_user _nobody=nobody def identify(self, auth): if auth and auth.lower().startswith('basic '): try: name, password=tuple(decodestring( auth.split(' ')[-1]).split(':', 1)) except: raise BadRequest, 'Invalid authentication token' return name, password else: return None, None def authenticate(self, name, password, request): emergency = self._emergency_user if name is None: return None if emergency and name==emergency.getUserName(): user = emergency else: user = self.getUser(name) if user is not None and user.authenticate(password, request): return user else: return None def authorize(self, user, accessed, container, name, value, roles): user = getattr(user, 'aq_base', user).__of__(self) newSecurityManager(None, user) security = getSecurityManager() try: try: # This is evil: we cannot pass _noroles directly because # it is a special marker, and that special marker is not # the same between the C and Python policy implementations. # We __really__ need to stop using this marker pattern! if roles is _noroles: if security.validate(accessed, container, name, value): return 1 else: if security.validate(accessed, container, name, value, roles): return 1 except: noSecurityManager() raise except Unauthorized: pass return 0 def validate(self, request, auth='', roles=_noroles): """ this method performs identification, authentication, and authorization v is the object (value) we're validating access to n is the name used to access the object a is the object the object was accessed through c is the physical container of the object We allow the publishing machinery to defer to higher-level user folders or to raise an unauthorized by returning None from this method. """ v = request['PUBLISHED'] # the published object a, c, n, v = self._getobcontext(v, request) # we need to continue to support this silly mode # where if there is no auth info, but if a user in our # database has no password and he has domain restrictions, # return him as the authorized user. if not auth: if self._domain_auth_mode: for user in self.getUsers(): if user.getDomains(): if self.authenticate(user.getUserName(), '', request): if self.authorize(user, a, c, n, v, roles): return user.__of__(self) name, password = self.identify(auth) user = self.authenticate(name, password, request) # user will be None if we can't authenticate him or if we can't find # his username in this user database. emergency = self._emergency_user if emergency and user is emergency: if self._isTop(): # we do not need to authorize the emergency user against the # published object. return emergency.__of__(self) else: # we're not the top-level user folder return None elif user is None: # either we didn't find the username, or the user's password # was incorrect. try to authorize and return the anonymous user. if self._isTop() and self.authorize(self._nobody, a,c,n,v,roles): return self._nobody.__of__(self) else: # anonymous can't authorize or we're not top-level user folder return None else: # We found a user, his password was correct, and the user # wasn't the emergency user. We need to authorize the user # against the published object. if self.authorize(user, a, c, n, v, roles): return user.__of__(self) # That didn't work. Try to authorize the anonymous user. elif self._isTop() and self.authorize(self._nobody,a,c,n,v,roles): return self._nobody.__of__(self) else: # we can't authorize the user, and we either can't authorize # nobody against the published object or we're not top-level return None if _remote_user_mode: def validate(self, request, auth='', roles=_noroles): v = request['PUBLISHED'] a, c, n, v = self._getobcontext(v, request) name = request.environ.get('REMOTE_USER', None) if name is None: if self._domain_auth_mode: for user in self.getUsers(): if user.getDomains(): if self.authenticate( user.getUserName(), '', request ): if self.authorize(user, a, c, n, v, roles): return user.__of__(self) user = self.getUser(name) # user will be None if we can't find his username in this user # database. emergency = self._emergency_user if emergency and name==emergency.getUserName(): if self._isTop(): # we do not need to authorize the emergency user against #the published object. return emergency.__of__(self) else: # we're not the top-level user folder return None elif user is None: # we didn't find the username in this database # try to authorize and return the anonymous user. if self._isTop() and self.authorize(self._nobody, a, c, n, v, roles): return self._nobody.__of__(self) else: # anonymous can't authorize or we're not top-level user # folder return None else: # We found a user and the user wasn't the emergency user. # We need to authorize the user against the published object. if self.authorize(user, a, c, n, v, roles): return user.__of__(self) # That didn't work. Try to authorize the anonymous user. elif self._isTop() and self.authorize( self._nobody, a, c, n, v, roles): return self._nobody.__of__(self) else: # we can't authorize the user, and we either can't # authorize nobody against the published object or # we're not top-level return None def _getobcontext(self, v, request): """ v is the object (value) we're validating access to n is the name used to access the object a is the object the object was accessed through c is the physical container of the object """ if len(request.steps) == 0: # someone deleted root index_html request.RESPONSE.notFoundError('no default view (root default view' ' was probably deleted)') n = request.steps[-1] # default to accessed and container as v.aq_parent a = c = request['PARENTS'][0] # try to find actual container inner = getattr(v, 'aq_inner', v) innerparent = getattr(inner, 'aq_parent', None) if innerparent is not None: # this is not a method, we needn't treat it specially c = innerparent elif hasattr(v, 'im_self'): # this is a method, we need to treat it specially c = v.im_self c = getattr(v, 'aq_inner', v) request_container = getattr(request['PARENTS'][-1], 'aq_parent', []) # if pub's aq_parent or container is the request container, it # means pub was accessed from the root if a is request_container: a = request['PARENTS'][-1] if c is request_container: c = request['PARENTS'][-1] return a, c, n, v def _isTop(self): try: return self.aq_parent.aq_base.isTopLevelPrincipiaApplicationObject except: return 0 def __len__(self): return 1 _mainUser=DTMLFile('dtml/mainUser', globals()) _add_User=DTMLFile('dtml/addUser', globals(), remote_user_mode__=_remote_user_mode) _editUser=DTMLFile('dtml/editUser', globals(), remote_user_mode__=_remote_user_mode) manage=manage_main=_mainUser manage_main._setName('manage_main') _userFolderProperties = DTMLFile('dtml/userFolderProps', globals()) def manage_userFolderProperties(self, REQUEST=None, manage_tabs_message=None): """ """ return self._userFolderProperties( self, REQUEST, manage_tabs_message=manage_tabs_message, management_view='Properties') def manage_setUserFolderProperties(self, encrypt_passwords=0, update_passwords=0, maxlistusers=DEFAULTMAXLISTUSERS, REQUEST=None): """ Sets the properties of the user folder. """ self.encrypt_passwords = not not encrypt_passwords try: self.maxlistusers = int(maxlistusers) except ValueError: self.maxlistusers = DEFAULTMAXLISTUSERS if encrypt_passwords and update_passwords: changed = 0 for u in self.getUsers(): pw = u._getPassword() if not self._isPasswordEncrypted(pw): pw = self._encryptPassword(pw) self._doChangeUser(u.getUserName(), pw, u.getRoles(), u.getDomains()) changed = changed + 1 if REQUEST is not None: if not changed: msg = 'All passwords already encrypted.' else: msg = 'Encrypted %d password(s).' % changed return self.manage_userFolderProperties( REQUEST, manage_tabs_message=msg) else: return changed else: if REQUEST is not None: return self.manage_userFolderProperties( REQUEST, manage_tabs_message='Saved changes.') def _isPasswordEncrypted(self, pw): return AuthEncoding.is_encrypted(pw) def _encryptPassword(self, pw): return AuthEncoding.pw_encrypt(pw, 'SSHA') def domainSpecValidate(self,spec): for ob in spec: am = addr_match(ob) hm = host_match(ob) if am is None and hm is None: return 0 return 1 def _addUser(self,name,password,confirm,roles,domains,REQUEST=None): if not name: return MessageDialog( title ='Illegal value', message='A username must be specified', action ='manage_main') if not password or not confirm: if not domains: return MessageDialog( title ='Illegal value', message='Password and confirmation must be specified', action ='manage_main') if self.getUser(name) or (self._emergency_user and name == self._emergency_user.getUserName()): return MessageDialog( title ='Illegal value', message='A user with the specified name already exists', action ='manage_main') if (password or confirm) and (password != confirm): return MessageDialog( title ='Illegal value', message='Password and confirmation do not match', action ='manage_main') if not roles: roles=[] if not domains: domains=[] if domains and not self.domainSpecValidate(domains): return MessageDialog( title ='Illegal value', message='Illegal domain specification', action ='manage_main') self._doAddUser(name, password, roles, domains) if REQUEST: return self._mainUser(self, REQUEST) def _changeUser(self,name,password,confirm,roles,domains,REQUEST=None): if password == 'password' and confirm == 'pconfirm': # Protocol for editUser.dtml to indicate unchanged password password = confirm = None if not name: return MessageDialog( title ='Illegal value', message='A username must be specified', action ='manage_main') if password == confirm == '': if not domains: return MessageDialog( title ='Illegal value', message='Password and confirmation must be specified', action ='manage_main') if not self.getUser(name): return MessageDialog( title ='Illegal value', message='Unknown user', action ='manage_main') if (password or confirm) and (password != confirm): return MessageDialog( title ='Illegal value', message='Password and confirmation do not match', action ='manage_main') if not roles: roles=[] if not domains: domains=[] if domains and not self.domainSpecValidate(domains): return MessageDialog( title ='Illegal value', message='Illegal domain specification', action ='manage_main') self._doChangeUser(name, password, roles, domains) if REQUEST: return self._mainUser(self, REQUEST) def _delUsers(self,names,REQUEST=None): if not names: return MessageDialog( title ='Illegal value', message='No users specified', action ='manage_main') self._doDelUsers(names) if REQUEST: return self._mainUser(self, REQUEST) def manage_users(self,submit=None,REQUEST=None,RESPONSE=None): """This method handles operations on users for the web based forms of the ZMI. Application code (code that is outside of the forms that implement the UI of a user folder) are encouraged to use manage_std_addUser""" if submit=='Add...': return self._add_User(self, REQUEST) if submit=='Edit': try: user=self.getUser(reqattr(REQUEST, 'name')) except: return MessageDialog( title ='Illegal value', message='The specified user does not exist', action ='manage_main') return self._editUser(self,REQUEST,user=user,password=user.__) if submit=='Add': name =reqattr(REQUEST, 'name') password=reqattr(REQUEST, 'password') confirm =reqattr(REQUEST, 'confirm') roles =reqattr(REQUEST, 'roles') domains =reqattr(REQUEST, 'domains') return self._addUser(name,password,confirm,roles,domains,REQUEST) if submit=='Change': name =reqattr(REQUEST, 'name') password=reqattr(REQUEST, 'password') confirm =reqattr(REQUEST, 'confirm') roles =reqattr(REQUEST, 'roles') domains =reqattr(REQUEST, 'domains') return self._changeUser(name,password,confirm,roles, domains,REQUEST) if submit=='Delete': names=reqattr(REQUEST, 'names') return self._delUsers(names,REQUEST) return self._mainUser(self, REQUEST) def user_names(self): return self.getUserNames() def manage_beforeDelete(self, item, container): if item is self: try: del container.__allow_groups__ except: pass def manage_afterAdd(self, item, container): if item is self: if hasattr(self, 'aq_base'): self=self.aq_base container.__allow_groups__=self def __creatable_by_emergency_user__(self): return 1 def _setId(self, id): if id != self.id: raise Globals.MessageDialog( title='Invalid Id', message='Cannot change the id of a UserFolder', action ='./manage_main',) # Domain authentication support. This is a good candidate to # become deprecated in future Zope versions. def setDomainAuthenticationMode(self, domain_auth_mode): """Set the domain-based authentication mode. By default, this mode is off due to the high overhead of the operation that is incurred for all anonymous accesses. If you have the 'Manage Users' permission, you can call this method via the web, passing a boolean value for domain_auth_mode to turn this behavior on or off.""" v = self._domain_auth_mode = domain_auth_mode and 1 or 0 return 'Domain authentication mode set to %d' % v def domainAuthModeEnabled(self): """ returns true if domain auth mode is set to true""" return getattr(self, '_domain_auth_mode', None) class UserFolder(BasicUserFolder): """Standard UserFolder object A UserFolder holds User objects which contain information about users including name, password domain, and roles. UserFolders function chiefly to control access by authenticating users and binding them to a collection of roles.""" meta_type='User Folder' id ='acl_users' title ='User Folder' icon ='p_/UserFolder' def __init__(self): self.data=PersistentMapping() def getUserNames(self): """Return a list of usernames""" names=self.data.keys() names.sort() return names def getUsers(self): """Return a list of user objects""" data=self.data names=data.keys() names.sort() return [data[n] for n in names] def getUser(self, name): """Return the named user object or None""" return self.data.get(name, None) def hasUsers(self): """ This is not a formal API method: it is used only to provide a way for the quickstart page to determine if the default user folder contains any users to provide instructions on how to add a user for newbies. Using getUserNames or getUsers would have posed a denial of service risk.""" return not not len(self.data) def _doAddUser(self, name, password, roles, domains, **kw): """Create a new user""" if password is not None and self.encrypt_passwords \ and not self._isPasswordEncrypted(password): password = self._encryptPassword(password) self.data[name]=User(name,password,roles,domains) def _doChangeUser(self, name, password, roles, domains, **kw): user=self.data[name] if password is not None: if ( self.encrypt_passwords and not self._isPasswordEncrypted(password)): password = self._encryptPassword(password) user.__=password user.roles=roles user.domains=domains def _doDelUsers(self, names): for name in names: del self.data[name] def _createInitialUser(self): """ If there are no users or only one user in this user folder, populates from the 'inituser' file in the instance home. We have to do this even when there is already a user just in case the initial user ignored the setup messages. We don't do it for more than one user to avoid abuse of this mechanism. Called only by OFS.Application.initialize(). """ if len(self.data) <= 1: info = readUserAccessFile('inituser') if info: import App.config name, password, domains, remote_user_mode = info self._doDelUsers(self.getUserNames()) self._doAddUser(name, password, ('Manager',), domains) cfg = App.config.getConfiguration() try: os.remove(os.path.join(cfg.instancehome, 'inituser')) except: pass Globals.default__class_init__(UserFolder) def manage_addUserFolder(self,dtself=None,REQUEST=None,**ignored): """ """ f=UserFolder() self=self.this() try: self._setObject('acl_users', f) except: return MessageDialog( title ='Item Exists', message='This object already contains a User Folder', action ='%s/manage_main' % REQUEST['URL1']) self.__allow_groups__=f if REQUEST is not None: REQUEST['RESPONSE'].redirect(self.absolute_url()+'/manage_main') def rolejoin(roles, other): dict={} for role in roles: dict[role]=1 for role in other: dict[role]=1 roles=dict.keys() roles.sort() return roles addr_match=re.compile(r'((\d{1,3}\.){1,3}\*)|((\d{1,3}\.){3}\d{1,3})').match host_match=re.compile(r'(([\_0-9a-zA-Z\-]*\.)*[0-9a-zA-Z\-]*)').match def domainSpecMatch(spec, request): host='' addr='' # Fast exit for the match-all case if len(spec) == 1 and spec[0] == '*': return 1 if request.has_key('REMOTE_HOST'): host=request['REMOTE_HOST'] addr=request.getClientAddr() if not host and not addr: return 0 if not host: try: host=socket.gethostbyaddr(addr)[0] except: pass if not addr: try: addr=socket.gethostbyname(host) except: pass _host=host.split('.') _addr=addr.split('.') _hlen=len(_host) _alen=len(_addr) for ob in spec: sz=len(ob) _ob=ob.split('.') _sz=len(_ob) mo = addr_match(ob) if mo is not None: if mo.end(0)==sz: fail=0 for i in range(_sz): a=_addr[i] o=_ob[i] if (o != a) and (o != '*'): fail=1 break if fail: continue return 1 mo = host_match(ob) if mo is not None: if mo.end(0)==sz: if _hlen < _sz: continue elif _hlen > _sz: _item=_host[-_sz:] else: _item=_host fail=0 for i in range(_sz): h=_item[i] o=_ob[i] if (o != h) and (o != '*'): fail=1 break if fail: continue return 1 return 0 def absattr(attr): if callable(attr): return attr() return attr def reqattr(request, attr): try: return request[attr] except: return None Super = UnrestrictedUser # Note: use of the Super alias is deprecated. Zope-2.8.7-final/lib/python/ImageFile.py0000775000076400007640000000163310435500331017023 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Image object that is stored in a file""" __version__='$Revision: 1.13 $'[11:-2] ############################################################ # # Eventually this module will go away! # ############################################################ from App.ImageFile import ImageFile Zope-2.8.7-final/lib/python/MethodObject/0000775000076400007640000000000010435610645017202 5ustar ajungajungZope-2.8.7-final/lib/python/MethodObject/__init__.py0000664000076400007640000000003410435500330021275 0ustar ajungajungfrom _MethodObject import * Zope-2.8.7-final/lib/python/MethodObject/setup.py0000664000076400007640000000166310435500330020707 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from distutils.core import setup, Extension setup(name="_MethodObject", version="2.0", ext_modules=[ Extension("_MethodObject", ["_MethodObject.c"], include_dirs = ['.', '../ExtensionClass'], depends = ['../ExtensionClass/ExtensionClass.h']), ]) Zope-2.8.7-final/lib/python/MethodObject/tests.py0000664000076400007640000000254410435500330020710 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """XXX short summary goes here. $Id: tests.py 29895 2005-04-07 03:49:30Z tim_one $ """ import unittest from zope.testing.doctest import DocTestSuite def test_xxx(): """ >>> from ExtensionClass import Base >>> from MethodObject import Method >>> class foo(Method): ... def __call__(self, ob, *args, **kw): ... print 'called', ob, args, kw >>> class bar(Base): ... def __repr__(self): ... return "bar()" ... hi = foo() >>> x = bar() >>> hi = x.hi >>> hi(1,2,3,name='spam') called bar() (1, 2, 3) {'name': 'spam'} """ def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/MethodObject/_MethodObject.c0000664000076400007640000000361610435500330022047 0ustar ajungajung/***************************************************************************** Copyright (c) 1996-2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "ExtensionClass.h" static PyObject * of(PyObject *self, PyObject *args) { PyObject *inst; if(PyArg_Parse(args,"O",&inst)) return PyECMethod_New(self,inst); else return NULL; } struct PyMethodDef Method_methods[] = { {"__of__",(PyCFunction)of,0,""}, {NULL, NULL} /* sentinel */ }; static struct PyMethodDef methods[] = {{NULL, NULL}}; void init_MethodObject(void) { PyObject *m, *d; PURE_MIXIN_CLASS(Method, "Base class for objects that want to be treated as methods\n" "\n" "The method class provides a method, __of__, that\n" "binds an object to an instance. If a method is a subobject\n" "of an extension-class instance, the the method will be bound\n" "to the instance and when the resulting object is called, it\n" "will call the method and pass the instance in addition to\n" "other arguments. It is the responsibility of Method objects\n" "to implement (or inherit) a __call__ method.\n", Method_methods); /* Create the module and add the functions */ m = Py_InitModule4("_MethodObject", methods, "Method-object mix-in class module\n\n" "$Id: _MethodObject.c 24763 2004-05-17 05:59:28Z philikon $\n", (PyObject*)NULL,PYTHON_API_VERSION); d = PyModule_GetDict(m); PyExtensionClass_Export(d,"Method",MethodType); } Zope-2.8.7-final/lib/python/ZEO/0000775000076400007640000000000010435610645015270 5ustar ajungajungZope-2.8.7-final/lib/python/ZEO/SETUP.cfg0000664000076400007640000000045710435502417016654 0ustar ajungajung# This file is empty even though this package contains scripts. The # reason for that is that we don't want the scripts to be installed # for Zope releases (which ZEO is part of). For standalone ZODB # releases, these scripts are installed through # ZODB/releases/ZODB3/PACKAGE.cfg and .../SETUP.cfg. Zope-2.8.7-final/lib/python/ZEO/Exceptions.py0000664000076400007640000000214110435502417017756 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Exceptions for ZEO.""" from ZODB.POSException import StorageError class ClientStorageError(StorageError): """An error occured in the ZEO Client Storage.""" class UnrecognizedResult(ClientStorageError): """A server call returned an unrecognized result.""" class ClientDisconnected(ClientStorageError): """The database storage is disconnected from the storage.""" class AuthError(StorageError): """The client provided invalid authentication credentials.""" Zope-2.8.7-final/lib/python/ZEO/zrpc/0000775000076400007640000000000010435610645016246 5ustar ajungajungZope-2.8.7-final/lib/python/ZEO/zrpc/trigger.py0000664000076400007640000002030510435502417020260 0ustar ajungajung############################################################################## # # Copyright (c) 2001-2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import asyncore import os import socket import thread import errno from ZODB.utils import positive_id # Original comments follow; they're hard to follow in the context of # ZEO's use of triggers. TODO: rewrite from a ZEO perspective. # Wake up a call to select() running in the main thread. # # This is useful in a context where you are using Medusa's I/O # subsystem to deliver data, but the data is generated by another # thread. Normally, if Medusa is in the middle of a call to # select(), new output data generated by another thread will have # to sit until the call to select() either times out or returns. # If the trigger is 'pulled' by another thread, it should immediately # generate a READ event on the trigger object, which will force the # select() invocation to return. # # A common use for this facility: letting Medusa manage I/O for a # large number of connections; but routing each request through a # thread chosen from a fixed-size thread pool. When a thread is # acquired, a transaction is performed, but output data is # accumulated into buffers that will be emptied more efficiently # by Medusa. [picture a server that can process database queries # rapidly, but doesn't want to tie up threads waiting to send data # to low-bandwidth connections] # # The other major feature provided by this class is the ability to # move work back into the main thread: if you call pull_trigger() # with a thunk argument, when select() wakes up and receives the # event it will call your thunk from within that thread. The main # purpose of this is to remove the need to wrap thread locks around # Medusa's data structures, which normally do not need them. [To see # why this is true, imagine this scenario: A thread tries to push some # new data onto a channel's outgoing data queue at the same time that # the main thread is trying to remove some] class _triggerbase(object): """OS-independent base class for OS-dependent trigger class.""" kind = None # subclass must set to "pipe" or "loopback"; used by repr def __init__(self): self._closed = False # `lock` protects the `thunks` list from being traversed and # appended to simultaneously. self.lock = thread.allocate_lock() # List of no-argument callbacks to invoke when the trigger is # pulled. These run in the thread running the asyncore mainloop, # regardless of which thread pulls the trigger. self.thunks = [] def readable(self): return 1 def writable(self): return 0 def handle_connect(self): pass def handle_close(self): self.close() # Override the asyncore close() method, because it doesn't know about # (so can't close) all the gimmicks we have open. Subclass must # supply a _close() method to do platform-specific closing work. _close() # will be called iff we're not already closed. def close(self): if not self._closed: self._closed = True self.del_channel() self._close() # subclass does OS-specific stuff def _close(self): # see close() above; subclass must supply raise NotImplementedError def pull_trigger(self, thunk=None): if thunk: self.lock.acquire() try: self.thunks.append(thunk) finally: self.lock.release() self._physical_pull() # Subclass must supply _physical_pull, which does whatever the OS # needs to do to provoke the "write" end of the trigger. def _physical_pull(self): raise NotImplementedError def handle_read(self): try: self.recv(8192) except socket.error: return self.lock.acquire() try: for thunk in self.thunks: try: thunk() except: nil, t, v, tbinfo = asyncore.compact_traceback() print ('exception in trigger thunk:' ' (%s:%s %s)' % (t, v, tbinfo)) self.thunks = [] finally: self.lock.release() def __repr__(self): return '' % (self.kind, positive_id(self)) if os.name == 'posix': class trigger(_triggerbase, asyncore.file_dispatcher): kind = "pipe" def __init__(self): _triggerbase.__init__(self) r, self.trigger = self._fds = os.pipe() asyncore.file_dispatcher.__init__(self, r) def _close(self): for fd in self._fds: os.close(fd) self._fds = [] def _physical_pull(self): os.write(self.trigger, 'x') else: # Windows version; uses just sockets, because a pipe isn't select'able # on Windows. class trigger(_triggerbase, asyncore.dispatcher): kind = "loopback" def __init__(self): _triggerbase.__init__(self) # Get a pair of connected sockets. The trigger is the 'w' # end of the pair, which is connected to 'r'. 'r' is put # in the asyncore socket map. "pulling the trigger" then # means writing something on w, which will wake up r. w = socket.socket() # Disable buffering -- pulling the trigger sends 1 byte, # and we want that sent immediately, to wake up asyncore's # select() ASAP. w.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) count = 0 while 1: count += 1 # Bind to a local port; for efficiency, let the OS pick # a free port for us. # Unfortunately, stress tests showed that we may not # be able to connect to that port ("Address already in # use") despite that the OS picked it. This appears # to be a race bug in the Windows socket implementation. # So we loop until a connect() succeeds (almost always # on the first try). See the long thread at # http://mail.zope.org/pipermail/zope/2005-July/160433.html # for hideous details. a = socket.socket() a.bind(("127.0.0.1", 0)) connect_address = a.getsockname() # assigned (host, port) pair a.listen(1) try: w.connect(connect_address) break # success except socket.error, detail: if detail[0] != errno.WSAEADDRINUSE: # "Address already in use" is the only error # I've seen on two WinXP Pro SP2 boxes, under # Pythons 2.3.5 and 2.4.1. raise # (10048, 'Address already in use') # assert count <= 2 # never triggered in Tim's tests if count >= 10: # I've never seen it go above 2 a.close() w.close() raise BindError("Cannot bind trigger!") # Close `a` and try again. Note: I originally put a short # sleep() here, but it didn't appear to help or hurt. a.close() r, addr = a.accept() # r becomes asyncore's (self.)socket a.close() self.trigger = w asyncore.dispatcher.__init__(self, r) def _close(self): # self.socket is r, and self.trigger is w, from __init__ self.socket.close() self.trigger.close() def _physical_pull(self): self.trigger.send('x') Zope-2.8.7-final/lib/python/ZEO/zrpc/__init__.py0000664000076400007640000000211710435502417020355 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # zrpc is a package with the following modules # client -- manages connection creation to remote server # connection -- object dispatcher # log -- logging helper # error -- exceptions raised by zrpc # marshal -- internal, handles basic protocol issues # server -- manages incoming connections from remote clients # smac -- sized message async connections # trigger -- medusa's trigger # zrpc is not an advertised subpackage of ZEO; its interfaces are internal Zope-2.8.7-final/lib/python/ZEO/zrpc/smac.py0000664000076400007640000002705610435502417017552 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Sized Message Async Connections. This class extends the basic asyncore layer with a record-marking layer. The message_output() method accepts an arbitrary sized string as its argument. It sends over the wire the length of the string encoded using struct.pack('>I') and the string itself. The receiver passes the original string to message_input(). This layer also supports an optional message authentication code (MAC). If a session key is present, it uses HMAC-SHA-1 to generate a 20-byte MAC. If a MAC is present, the high-order bit of the length is set to 1 and the MAC immediately follows the length. """ import asyncore import errno try: import hmac except ImportError: import _hmac as hmac import sha import socket import struct import threading import logging from types import StringType from ZODB.loglevels import TRACE from ZEO.zrpc.log import log, short_repr from ZEO.zrpc.error import DisconnectedError # Use the dictionary to make sure we get the minimum number of errno # entries. We expect that EWOULDBLOCK == EAGAIN on most systems -- # or that only one is actually used. tmp_dict = {errno.EWOULDBLOCK: 0, errno.EAGAIN: 0, errno.EINTR: 0, } expected_socket_read_errors = tuple(tmp_dict.keys()) tmp_dict = {errno.EAGAIN: 0, errno.EWOULDBLOCK: 0, errno.ENOBUFS: 0, errno.EINTR: 0, } expected_socket_write_errors = tuple(tmp_dict.keys()) del tmp_dict # We chose 60000 as the socket limit by looking at the largest strings # that we could pass to send() without blocking. SEND_SIZE = 60000 MAC_BIT = 0x80000000L class SizedMessageAsyncConnection(asyncore.dispatcher): __super_init = asyncore.dispatcher.__init__ __super_close = asyncore.dispatcher.close __closed = True # Marker indicating that we're closed socket = None # to outwit Sam's getattr def __init__(self, sock, addr, map=None, debug=None): self.addr = addr if debug is not None: self._debug = debug elif not hasattr(self, '_debug'): self._debug = __debug__ # __input_lock protects __inp, __input_len, __state, __msg_size self.__input_lock = threading.Lock() self.__inp = None # None, a single String, or a list self.__input_len = 0 # Instance variables __state, __msg_size and __has_mac work together: # when __state == 0: # __msg_size == 4, and the next thing read is a message size; # __has_mac is set according to the MAC_BIT in the header # when __state == 1: # __msg_size is variable, and the next thing read is a message. # __has_mac indicates if we're in MAC mode or not (and # therefore, if we need to check the mac header) # The next thing read is always of length __msg_size. # The state alternates between 0 and 1. self.__state = 0 self.__has_mac = 0 self.__msg_size = 4 self.__output_lock = threading.Lock() # Protects __output self.__output = [] self.__closed = False # Each side of the connection sends and receives messages. A # MAC is generated for each message and depends on each # previous MAC; the state of the MAC generator depends on the # history of operations it has performed. So the MACs must be # generated in the same order they are verified. # Each side is guaranteed to receive messages in the order # they are sent, but there is no ordering constraint between # message sends and receives. If the two sides are A and B # and message An indicates the nth message sent by A, then # A1 A2 B1 B2 and A1 B1 B2 A2 are both legitimate total # orderings of the messages. # As a result, there must be seperate MAC generators for each # side of the connection. If not, the generator state would # be different after A1 A2 B1 B2 than it would be after # A1 B1 B2 A2; if the generator state was different, the MAC # could not be verified. self.__hmac_send = None self.__hmac_recv = None self.__super_init(sock, map) def setSessionKey(self, sesskey): log("set session key %r" % sesskey) self.__hmac_send = hmac.HMAC(sesskey, digestmod=sha) self.__hmac_recv = hmac.HMAC(sesskey, digestmod=sha) def get_addr(self): return self.addr # TODO: avoid expensive getattr calls? Can't remember exactly what # this comment was supposed to mean, but it has something to do # with the way asyncore uses getattr and uses if sock: def __nonzero__(self): return 1 def handle_read(self): self.__input_lock.acquire() try: # Use a single __inp buffer and integer indexes to make this fast. try: d = self.recv(8192) except socket.error, err: if err[0] in expected_socket_read_errors: return raise if not d: return input_len = self.__input_len + len(d) msg_size = self.__msg_size state = self.__state has_mac = self.__has_mac inp = self.__inp if msg_size > input_len: if inp is None: self.__inp = d elif type(self.__inp) is StringType: self.__inp = [self.__inp, d] else: self.__inp.append(d) self.__input_len = input_len return # keep waiting for more input # load all previous input and d into single string inp if isinstance(inp, StringType): inp = inp + d elif inp is None: inp = d else: inp.append(d) inp = "".join(inp) offset = 0 while (offset + msg_size) <= input_len: msg = inp[offset:offset + msg_size] offset = offset + msg_size if not state: msg_size = struct.unpack(">I", msg)[0] has_mac = msg_size & MAC_BIT if has_mac: msg_size ^= MAC_BIT msg_size += 20 elif self.__hmac_send: raise ValueError("Received message without MAC") state = 1 else: msg_size = 4 state = 0 # Obscure: We call message_input() with __input_lock # held!!! And message_input() may end up calling # message_output(), which has its own lock. But # message_output() cannot call message_input(), so # the locking order is always consistent, which # prevents deadlock. Also, message_input() may # take a long time, because it can cause an # incoming call to be handled. During all this # time, the __input_lock is held. That's a good # thing, because it serializes incoming calls. if has_mac: mac = msg[:20] msg = msg[20:] if self.__hmac_recv: self.__hmac_recv.update(msg) _mac = self.__hmac_recv.digest() if mac != _mac: raise ValueError("MAC failed: %r != %r" % (_mac, mac)) else: log("Received MAC but no session key set") elif self.__hmac_send: raise ValueError("Received message without MAC") self.message_input(msg) self.__state = state self.__has_mac = has_mac self.__msg_size = msg_size self.__inp = inp[offset:] self.__input_len = input_len - offset finally: self.__input_lock.release() def readable(self): return True def writable(self): if len(self.__output) == 0: return False else: return True def handle_write(self): self.__output_lock.acquire() try: output = self.__output while output: # Accumulate output into a single string so that we avoid # multiple send() calls, but avoid accumulating too much # data. If we send a very small string and have more data # to send, we will likely incur delays caused by the # unfortunate interaction between the Nagle algorithm and # delayed acks. If we send a very large string, only a # portion of it will actually be delivered at a time. l = 0 for i in range(len(output)): l += len(output[i]) if l > SEND_SIZE: break i += 1 # It is very unlikely that i will be 1. v = "".join(output[:i]) del output[:i] try: n = self.send(v) except socket.error, err: if err[0] in expected_socket_write_errors: break # we couldn't write anything raise if n < len(v): output.insert(0, v[n:]) break # we can't write any more finally: self.__output_lock.release() def handle_close(self): self.close() def message_output(self, message): if __debug__: if self._debug: log("message_output %d bytes: %s hmac=%d" % (len(message), short_repr(message), self.__hmac_send and 1 or 0), level=TRACE) if self.__closed: raise DisconnectedError( "This action is temporarily unavailable.

") self.__output_lock.acquire() try: # do two separate appends to avoid copying the message string if self.__hmac_send: self.__output.append(struct.pack(">I", len(message) | MAC_BIT)) self.__hmac_send.update(message) self.__output.append(self.__hmac_send.digest()) else: self.__output.append(struct.pack(">I", len(message))) if len(message) <= SEND_SIZE: self.__output.append(message) else: for i in range(0, len(message), SEND_SIZE): self.__output.append(message[i:i+SEND_SIZE]) finally: self.__output_lock.release() def close(self): if not self.__closed: self.__closed = True self.__super_close() Zope-2.8.7-final/lib/python/ZEO/zrpc/marshal.py0000664000076400007640000000561710435502417020255 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import cPickle from cStringIO import StringIO import types import logging from ZEO.zrpc.error import ZRPCError from ZEO.zrpc.log import log, short_repr class Marshaller: """Marshal requests and replies to second across network""" def encode(self, msgid, flags, name, args): """Returns an encoded message""" # (We used to have a global pickler, but that's not thread-safe. :-( ) # Note that args may contain very large binary pickles already; for # this reason, it's important to use proto 1 (or higher) pickles here # too. For a long time, this used proto 0 pickles, and that can # bloat our pickle to 4x the size (due to high-bit and control bytes # being represented by \xij escapes in proto 0). # Undocumented: cPickle.Pickler accepts a lone protocol argument; # pickle.py does not. pickler = cPickle.Pickler(1) pickler.fast = 1 # Undocumented: pickler.dump(), for a cPickle.Pickler, takes # an optional boolean argument. When true, it returns the pickle; # when false or unspecified, it returns the pickler object itself. # pickle.py does none of this. return pickler.dump((msgid, flags, name, args), 1) def decode(self, msg): """Decodes msg and returns its parts""" unpickler = cPickle.Unpickler(StringIO(msg)) unpickler.find_global = find_global try: return unpickler.load() # msgid, flags, name, args except: log("can't decode message: %s" % short_repr(msg), level=logging.ERROR) raise _globals = globals() _silly = ('__doc__',) def find_global(module, name): """Helper for message unpickler""" try: m = __import__(module, _globals, _globals, _silly) except ImportError, msg: raise ZRPCError("import error %s: %s" % (module, msg)) try: r = getattr(m, name) except AttributeError: raise ZRPCError("module %s has no global %s" % (module, name)) safe = getattr(r, '__no_side_effects__', 0) if safe: return r # TODO: is there a better way to do this? if type(r) == types.ClassType and issubclass(r, Exception): return r raise ZRPCError("Unsafe global: %s.%s" % (module, name)) Zope-2.8.7-final/lib/python/ZEO/zrpc/connection.py0000664000076400007640000010035610435502417020761 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import asyncore import errno import select import sys import threading import types import logging import ThreadedAsync from ZEO.zrpc import smac from ZEO.zrpc.error import ZRPCError, DisconnectedError from ZEO.zrpc.marshal import Marshaller from ZEO.zrpc.trigger import trigger from ZEO.zrpc.log import short_repr, log from ZODB.loglevels import BLATHER, TRACE REPLY = ".reply" # message name used for replies ASYNC = 1 class Delay: """Used to delay response to client for synchronous calls. When a synchronous call is made and the original handler returns without handling the call, it returns a Delay object that prevents the mainloop from sending a response. """ def set_sender(self, msgid, send_reply, return_error): self.msgid = msgid self.send_reply = send_reply self.return_error = return_error def reply(self, obj): self.send_reply(self.msgid, obj) def error(self, exc_info): log("Error raised in delayed method", logging.ERROR, exc_info=True) self.return_error(self.msgid, 0, *exc_info[:2]) class MTDelay(Delay): def __init__(self): self.ready = threading.Event() def set_sender(self, msgid, send_reply, return_error): Delay.set_sender(self, msgid, send_reply, return_error) self.ready.set() def reply(self, obj): self.ready.wait() Delay.reply(self, obj) def error(self, exc_info): self.ready.wait() Delay.error(self, exc_info) # PROTOCOL NEGOTIATION # # The code implementing protocol version 2.0.0 (which is deployed # in the field and cannot be changed) *only* talks to peers that # send a handshake indicating protocol version 2.0.0. In that # version, both the client and the server immediately send out # their protocol handshake when a connection is established, # without waiting for their peer, and disconnect when a different # handshake is receive. # # The new protocol uses this to enable new clients to talk to # 2.0.0 servers. In the new protocol: # # The server sends its protocol handshake to the client at once. # # The client waits until it receives the server's protocol handshake # before sending its own handshake. The client sends the lower of its # own protocol version and the server protocol version, allowing it to # talk to servers using later protocol versions (2.0.2 and higher) as # well: the effective protocol used will be the lower of the client # and server protocol. However, this changed in ZODB 3.3.1 (and # should have changed in ZODB 3.3) because an older server doesn't # support MVCC methods required by 3.3 clients. # # [Ugly details: In order to treat the first received message (protocol # handshake) differently than all later messages, both client and server # start by patching their message_input() method to refer to their # recv_handshake() method instead. In addition, the client has to arrange # to queue (delay) outgoing messages until it receives the server's # handshake, so that the first message the client sends to the server is # the client's handshake. This multiply-special treatment of the first # message is delicate, and several asyncore and thread subtleties were # handled unsafely before ZODB 3.2.6. # ] # # The ZEO modules ClientStorage and ServerStub have backwards # compatibility code for dealing with the previous version of the # protocol. The client accepts the old version of some messages, # and will not send new messages when talking to an old server. # # As long as the client hasn't sent its handshake, it can't send # anything else; output messages are queued during this time. # (Output can happen because the connection testing machinery can # start sending requests before the handshake is received.) # # UPGRADING FROM ZEO 2.0.0 TO NEWER VERSIONS: # # Because a new client can talk to an old server, but not vice # versa, all clients should be upgraded before upgrading any # servers. Protocol upgrades beyond 2.0.1 will not have this # restriction, because clients using protocol 2.0.1 or later can # talk to both older and newer servers. # # No compatibility with protocol version 1 is provided. # Connection is abstract (it must be derived from). ManagedServerConnection # and ManagedClientConnection are the concrete subclasses. They need to # supply a handshake() method appropriate for their role in protocol # negotiation. class Connection(smac.SizedMessageAsyncConnection, object): """Dispatcher for RPC on object on both sides of socket. The connection supports synchronous calls, which expect a return, and asynchronous calls, which do not. It uses the Marshaller class to handle encoding and decoding of method calls and arguments. Marshaller uses pickle to encode arbitrary Python objects. The code here doesn't ever see the wire format. A Connection is designed for use in a multithreaded application, where a synchronous call must block until a response is ready. A socket connection between a client and a server allows either side to invoke methods on the other side. The processes on each end of the socket use a Connection object to manage communication. The Connection deals with decoded RPC messages. They are represented as four-tuples containing: msgid, flags, method name, and a tuple of method arguments. The msgid starts at zero and is incremented by one each time a method call message is sent. Each side of the connection has a separate msgid state. When one side of the connection (the client) calls a method, it sends a message with a new msgid. The other side (the server), replies with a message that has the same msgid, the string ".reply" (the global variable REPLY) as the method name, and the actual return value in the args position. Note that each side of the Connection can initiate a call, in which case it will be the client for that particular call. The protocol also supports asynchronous calls. The client does not wait for a return value for an asynchronous call. The only defined flag is ASYNC. If a method call message has the ASYNC flag set, the server will raise an exception. If a method call raises an Exception, the exception is propagated back to the client via the REPLY message. The client side will raise any exception it receives instead of returning the value to the caller. """ __super_init = smac.SizedMessageAsyncConnection.__init__ __super_close = smac.SizedMessageAsyncConnection.close __super_setSessionKey = smac.SizedMessageAsyncConnection.setSessionKey # Protocol history: # # Z200 -- Original ZEO 2.0 protocol # # Z201 -- Added invalidateTransaction() to client. # Renamed several client methods. # Added several sever methods: # lastTransaction() # getAuthProtocol() and scheme-specific authentication methods # getExtensionMethods(). # getInvalidations(). # # Z303 -- named after the ZODB release 3.3 # Added methods for MVCC: # loadBefore() # loadEx() # A Z303 client cannot talk to a Z201 server, because the latter # doesn't support MVCC. A Z201 client can talk to a Z303 server, # but because (at least) the type of the root object changed # from ZODB.PersistentMapping to persistent.mapping, the older # client can't actually make progress if a Z303 client created, # or ever modified, the root. # Protocol variables: # Our preferred protocol. current_protocol = "Z303" # If we're a client, an exhaustive list of the server protocols we # can accept. servers_we_can_talk_to = [current_protocol] # If we're a server, an exhaustive list of the client protocols we # can accept. clients_we_can_talk_to = ["Z200", "Z201", current_protocol] # This is pretty excruciating. Details: # # 3.3 server 3.2 client # server sends Z303 to client # client computes min(Z303, Z201) == Z201 as the protocol to use # client sends Z201 to server # OK, because Z201 is in the server's clients_we_can_talk_to # # 3.2 server 3.3 client # server sends Z201 to client # client computes min(Z303, Z201) == Z201 as the protocol to use # Z201 isn't in the client's servers_we_can_talk_to, so client # raises exception # # 3.3 server 3.3 client # server sends Z303 to client # client computes min(Z303, Z303) == Z303 as the protocol to use # Z303 is in the client's servers_we_can_talk_to, so client # sends Z303 to server # OK, because Z303 is in the server's clients_we_can_talk_to # Client constructor passes 'C' for tag, server constructor 'S'. This # is used in log messages, and to determine whether we can speak with # our peer. def __init__(self, sock, addr, obj, tag): self.obj = None self.marshal = Marshaller() self.closed = False self.peer_protocol_version = None # set in recv_handshake() assert tag in "CS" self.tag = tag self.logger = logging.getLogger('ZEO.zrpc.Connection(%c)' % tag) if isinstance(addr, types.TupleType): self.log_label = "(%s:%d) " % addr else: self.log_label = "(%s) " % addr # Supply our own socket map, so that we don't get registered with # the asyncore socket map just yet. The initial protocol messages # are treated very specially, and we dare not get invoked by asyncore # before that special-case setup is complete. Some of that setup # occurs near the end of this constructor, and the rest is done by # a concrete subclass's handshake() method. Unfortunately, because # we ultimately derive from asyncore.dispatcher, it's not possible # to invoke the superclass constructor without asyncore stuffing # us into _some_ socket map. ourmap = {} self.__super_init(sock, addr, map=ourmap) # A Connection either uses asyncore directly or relies on an # asyncore mainloop running in a separate thread. If # thr_async is true, then the mainloop is running in a # separate thread. If thr_async is true, then the asyncore # trigger (self.trigger) is used to notify that thread of # activity on the current thread. self.thr_async = False self.trigger = None self._prepare_async() # The singleton dict is used in synchronous mode when a method # needs to call into asyncore to try to force some I/O to occur. # The singleton dict is a socket map containing only this object. self._singleton = {self._fileno: self} # msgid_lock guards access to msgid self.msgid = 0 self.msgid_lock = threading.Lock() # replies_cond is used to block when a synchronous call is # waiting for a response self.replies_cond = threading.Condition() self.replies = {} # waiting_for_reply is used internally to indicate whether # a call is in progress. setting a session key is deferred # until after the call returns. self.waiting_for_reply = False self.delay_sesskey = None self.register_object(obj) # The first message we see is a protocol handshake. message_input() # is temporarily replaced by recv_handshake() to treat that message # specially. revc_handshake() does "del self.message_input", which # uncovers the normal message_input() method thereafter. self.message_input = self.recv_handshake # Server and client need to do different things for protocol # negotiation, and handshake() is implemented differently in each. self.handshake() # Now it's safe to register with asyncore's socket map; it was not # safe before message_input was replaced, or before handshake() was # invoked. # Obscure: in Python 2.4, the base asyncore.dispatcher class grew # a ._map attribute, which is used instead of asyncore's global # socket map when ._map isn't None. Because we passed `ourmap` to # the base class constructor above, in 2.4 asyncore believes we want # to use `ourmap` instead of the global socket map -- but we don't. # So we have to replace our ._map with the global socket map, and # update the global socket map with `ourmap`. Replacing our ._map # isn't necessary before Python 2.4, but doesn't hurt then (it just # gives us an unused attribute in 2.3); updating the global socket # map is necessary regardless of Python version. self._map = asyncore.socket_map asyncore.socket_map.update(ourmap) def __repr__(self): return "<%s %s>" % (self.__class__.__name__, self.addr) __str__ = __repr__ # Defeat asyncore's dreaded __getattr__ def log(self, message, level=BLATHER, exc_info=False): self.logger.log(level, self.log_label + message, exc_info=exc_info) def close(self): if self.closed: return self._singleton.clear() self.closed = True self.close_trigger() self.__super_close() def close_trigger(self): # Overridden by ManagedClientConnection. if self.trigger is not None: self.trigger.close() def register_object(self, obj): """Register obj as the true object to invoke methods on.""" self.obj = obj # Subclass must implement. handshake() is called by the constructor, # near its end, but before self is added to asyncore's socket map. # When a connection is created the first message sent is a 4-byte # protocol version. This allows the protocol to evolve over time, and # lets servers handle clients using multiple versions of the protocol. # In general, the server's handshake() just needs to send the server's # preferred protocol; the client's also needs to queue (delay) outgoing # messages until it sees the handshake from the server. def handshake(self): raise NotImplementedError # Replaces message_input() for the first message received. Records the # protocol sent by the peer in `peer_protocol_version`, restores the # normal message_input() method, and raises an exception if the peer's # protocol is unacceptable. That's all the server needs to do. The # client needs to do additional work in response to the server's # handshake, and extends this method. def recv_handshake(self, proto): # Extended by ManagedClientConnection. del self.message_input # uncover normal-case message_input() self.peer_protocol_version = proto if self.tag == 'C': good_protos = self.servers_we_can_talk_to else: assert self.tag == 'S' good_protos = self.clients_we_can_talk_to if proto in good_protos: self.log("received handshake %r" % proto, level=logging.INFO) else: self.log("bad handshake %s" % short_repr(proto), level=logging.ERROR) raise ZRPCError("bad handshake %r" % proto) def message_input(self, message): """Decode an incoming message and dispatch it""" # If something goes wrong during decoding, the marshaller # will raise an exception. The exception will ultimately # result in asycnore calling handle_error(), which will # close the connection. msgid, flags, name, args = self.marshal.decode(message) if __debug__: self.log("recv msg: %s, %s, %s, %s" % (msgid, flags, name, short_repr(args)), level=TRACE) if name == REPLY: self.handle_reply(msgid, flags, args) else: self.handle_request(msgid, flags, name, args) def handle_reply(self, msgid, flags, args): if __debug__: self.log("recv reply: %s, %s, %s" % (msgid, flags, short_repr(args)), level=TRACE) self.replies_cond.acquire() try: self.replies[msgid] = flags, args self.replies_cond.notifyAll() finally: self.replies_cond.release() def handle_request(self, msgid, flags, name, args): if not self.check_method(name): msg = "Invalid method name: %s on %s" % (name, repr(self.obj)) raise ZRPCError(msg) if __debug__: self.log("calling %s%s" % (name, short_repr(args)), level=logging.DEBUG) meth = getattr(self.obj, name) try: self.waiting_for_reply = True try: ret = meth(*args) finally: self.waiting_for_reply = False except (SystemExit, KeyboardInterrupt): raise except Exception, msg: self.log("%s() raised exception: %s" % (name, msg), logging.INFO, exc_info=True) error = sys.exc_info()[:2] return self.return_error(msgid, flags, *error) if flags & ASYNC: if ret is not None: raise ZRPCError("async method %s returned value %s" % (name, short_repr(ret))) else: if __debug__: self.log("%s returns %s" % (name, short_repr(ret)), logging.DEBUG) if isinstance(ret, Delay): ret.set_sender(msgid, self.send_reply, self.return_error) else: self.send_reply(msgid, ret) if self.delay_sesskey: self.__super_setSessionKey(self.delay_sesskey) self.delay_sesskey = None def handle_error(self): if sys.exc_info()[0] == SystemExit: raise sys.exc_info() self.log("Error caught in asyncore", level=logging.ERROR, exc_info=True) self.close() def check_method(self, name): # TODO: This is hardly "secure". if name.startswith('_'): return None return hasattr(self.obj, name) def send_reply(self, msgid, ret): # encode() can pass on a wide variety of exceptions from cPickle. # While a bare `except` is generally poor practice, in this case # it's acceptable -- we really do want to catch every exception # cPickle may raise. try: msg = self.marshal.encode(msgid, 0, REPLY, ret) except: # see above try: r = short_repr(ret) except: r = "" err = ZRPCError("Couldn't pickle return %.100s" % r) msg = self.marshal.encode(msgid, 0, REPLY, (ZRPCError, err)) self.message_output(msg) self.poll() def return_error(self, msgid, flags, err_type, err_value): if flags & ASYNC: self.log("Asynchronous call raised exception: %s" % self, level=logging.ERROR, exc_info=True) return if type(err_value) is not types.InstanceType: err_value = err_type, err_value # encode() can pass on a wide variety of exceptions from cPickle. # While a bare `except` is generally poor practice, in this case # it's acceptable -- we really do want to catch every exception # cPickle may raise. try: msg = self.marshal.encode(msgid, 0, REPLY, (err_type, err_value)) except: # see above try: r = short_repr(err_value) except: r = "" err = ZRPCError("Couldn't pickle error %.100s" % r) msg = self.marshal.encode(msgid, 0, REPLY, (ZRPCError, err)) self.message_output(msg) self.poll() def setSessionKey(self, key): if self.waiting_for_reply: self.delay_sesskey = key else: self.__super_setSessionKey(key) # The next two public methods (call and callAsync) are used by # clients to invoke methods on remote objects def send_call(self, method, args, flags): # send a message and return its msgid self.msgid_lock.acquire() try: msgid = self.msgid self.msgid = self.msgid + 1 finally: self.msgid_lock.release() if __debug__: self.log("send msg: %d, %d, %s, ..." % (msgid, flags, method), level=TRACE) buf = self.marshal.encode(msgid, flags, method, args) self.message_output(buf) return msgid def call(self, method, *args): if self.closed: raise DisconnectedError() msgid = self.send_call(method, args, 0) r_flags, r_args = self.wait(msgid) if (isinstance(r_args, types.TupleType) and len(r_args) > 1 and type(r_args[0]) == types.ClassType and issubclass(r_args[0], Exception)): inst = r_args[1] raise inst # error raised by server else: return r_args # For testing purposes, it is useful to begin a synchronous call # but not block waiting for its response. Since these methods are # used for testing they can assume they are not in async mode and # call asyncore.poll() directly to get the message out without # also waiting for the reply. def _deferred_call(self, method, *args): if self.closed: raise DisconnectedError() msgid = self.send_call(method, args, 0) asyncore.poll(0.01, self._singleton) return msgid def _deferred_wait(self, msgid): r_flags, r_args = self.wait(msgid) if (isinstance(r_args, types.TupleType) and type(r_args[0]) == types.ClassType and issubclass(r_args[0], Exception)): inst = r_args[1] raise inst # error raised by server else: return r_args def callAsync(self, method, *args): if self.closed: raise DisconnectedError() self.send_call(method, args, ASYNC) self.poll() def callAsyncNoPoll(self, method, *args): # Like CallAsync but doesn't poll. This exists so that we can # send invalidations atomically to all clients without # allowing any client to sneak in a load request. if self.closed: raise DisconnectedError() self.send_call(method, args, ASYNC) # handle IO, possibly in async mode def _prepare_async(self): self.thr_async = False ThreadedAsync.register_loop_callback(self.set_async) # TODO: If we are not in async mode, this will cause dead # Connections to be leaked. def set_async(self, map): self.trigger = trigger() self.thr_async = True def is_async(self): # Overridden by ManagedConnection if self.thr_async: return 1 else: return 0 def _pull_trigger(self, tryagain=10): try: self.trigger.pull_trigger() except OSError: self.trigger.close() self.trigger = trigger() if tryagain > 0: self._pull_trigger(tryagain=tryagain-1) def wait(self, msgid): """Invoke asyncore mainloop and wait for reply.""" if __debug__: self.log("wait(%d), async=%d" % (msgid, self.is_async()), level=TRACE) if self.is_async(): self._pull_trigger() # Delay used when we call asyncore.poll() directly. # Start with a 1 msec delay, double until 1 sec. delay = 0.001 self.replies_cond.acquire() try: while 1: if self.closed: raise DisconnectedError() reply = self.replies.get(msgid) if reply is not None: del self.replies[msgid] if __debug__: self.log("wait(%d): reply=%s" % (msgid, short_repr(reply)), level=TRACE) return reply if self.is_async(): self.replies_cond.wait(10.0) else: self.replies_cond.release() try: try: if __debug__: self.log("wait(%d): asyncore.poll(%s)" % (msgid, delay), level=TRACE) asyncore.poll(delay, self._singleton) if delay < 1.0: delay += delay except select.error, err: self.log("Closing. asyncore.poll() raised %s." % err, level=BLATHER) self.close() finally: self.replies_cond.acquire() finally: self.replies_cond.release() def flush(self): """Invoke poll() until the output buffer is empty.""" if __debug__: self.log("flush") while self.writable(): self.poll() def poll(self): """Invoke asyncore mainloop to get pending message out.""" if __debug__: self.log("poll(), async=%d" % self.is_async(), level=TRACE) if self.is_async(): self._pull_trigger() else: asyncore.poll(0.0, self._singleton) def pending(self, timeout=0): """Invoke mainloop until any pending messages are handled.""" if __debug__: self.log("pending(), async=%d" % self.is_async(), level=TRACE) if self.is_async(): return # Inline the asyncore poll() function to know whether any input # was actually read. Repeat until no input is ready. # Pending does reads and writes. In the case of server # startup, we may need to write out zeoVerify() messages. # Always check for read status, but don't check for write status # only there is output to do. Only continue in this loop as # long as there is data to read. r = r_in = [self._fileno] x_in = [] while r and not self.closed: if self.writable(): w_in = [self._fileno] else: w_in = [] try: r, w, x = select.select(r_in, w_in, x_in, timeout) except select.error, err: if err[0] == errno.EINTR: timeout = 0 continue else: raise else: # Make sure any subsequent select does not block. The # loop is only intended to make sure all incoming data is # returned. # Insecurity: What if the server sends a lot of # invalidations, such that pending never finishes? Seems # unlikely, but possible. timeout = 0 if r: try: self.handle_read_event() except asyncore.ExitNow: raise except: self.handle_error() if w: try: self.handle_write_event() except asyncore.ExitNow: raise except: self.handle_error() class ManagedServerConnection(Connection): """Server-side Connection subclass.""" __super_init = Connection.__init__ __super_close = Connection.close def __init__(self, sock, addr, obj, mgr): self.mgr = mgr self.__super_init(sock, addr, obj, 'S') self.obj.notifyConnected(self) def handshake(self): # Send the server's preferred protocol to the client. self.message_output(self.current_protocol) def close(self): self.obj.notifyDisconnected() self.mgr.close_conn(self) self.__super_close() class ManagedClientConnection(Connection): """Client-side Connection subclass.""" __super_init = Connection.__init__ __super_close = Connection.close base_message_output = Connection.message_output def __init__(self, sock, addr, obj, mgr): self.mgr = mgr # We can't use the base smac's message_output directly because the # client needs to queue outgoing messages until it's seen the # initial protocol handshake from the server. So we have our own # message_ouput() method, and support for initial queueing. This is # a delicate design, requiring an output mutex to be wholly # thread-safe. # Caution: we must set this up before calling the base class # constructor, because the latter registers us with asyncore; # we need to guarantee that we'll queue outgoing messages before # asyncore learns about us. self.output_lock = threading.Lock() self.queue_output = True self.queued_messages = [] self.__super_init(sock, addr, obj, tag='C') self.check_mgr_async() # Our message_ouput() queues messages until recv_handshake() gets the # protocol handshake from the server. def message_output(self, message): self.output_lock.acquire() try: if self.queue_output: self.queued_messages.append(message) else: assert not self.queued_messages self.base_message_output(message) finally: self.output_lock.release() def handshake(self): # The client waits to see the server's handshake. Outgoing messages # are queued for the duration. The client will send its own # handshake after the server's handshake is seen, in recv_handshake() # below. It will then send any messages queued while waiting. assert self.queue_output # the constructor already set this def recv_handshake(self, proto): # The protocol to use is the older of our and the server's preferred # protocols. proto = min(proto, self.current_protocol) # Restore the normal message_input method, and raise an exception # if the protocol version is too old. Connection.recv_handshake(self, proto) # Tell the server the protocol in use, then send any messages that # were queued while waiting to hear the server's protocol, and stop # queueing messages. self.output_lock.acquire() try: self.base_message_output(proto) for message in self.queued_messages: self.base_message_output(message) self.queued_messages = [] self.queue_output = False finally: self.output_lock.release() # Defer the ThreadedAsync work to the manager. def close_trigger(self): # the manager should actually close the trigger # TODO: what is that comment trying to say? What 'manager'? del self.trigger def set_async(self, map): pass def _prepare_async(self): # Don't do the register_loop_callback that the superclass does pass def check_mgr_async(self): if not self.thr_async and self.mgr.thr_async: assert self.mgr.trigger is not None, \ "manager (%s) has no trigger" % self.mgr self.thr_async = True self.trigger = self.mgr.trigger return 1 return 0 def is_async(self): # TODO: could the check_mgr_async() be avoided on each test? if self.thr_async: return 1 return self.check_mgr_async() def close(self): self.mgr.close_conn(self) self.__super_close() Zope-2.8.7-final/lib/python/ZEO/zrpc/server.py0000664000076400007640000000356510435502417020134 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import asyncore import socket import types from ZEO.zrpc.connection import Connection from ZEO.zrpc.log import log import logging import ThreadedAsync.LoopCallback # Export the main asyncore loop loop = ThreadedAsync.LoopCallback.loop class Dispatcher(asyncore.dispatcher): """A server that accepts incoming RPC connections""" __super_init = asyncore.dispatcher.__init__ def __init__(self, addr, factory=Connection): self.__super_init() self.addr = addr self.factory = factory self._open_socket() def _open_socket(self): if type(self.addr) == types.TupleType: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) else: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) self.set_reuse_addr() log("listening on %s" % str(self.addr), logging.INFO) self.bind(self.addr) self.listen(5) def writable(self): return 0 def readable(self): return 1 def handle_accept(self): try: sock, addr = self.accept() except socket.error, msg: log("accepted failed: %s" % msg) return c = self.factory(sock, addr) log("connect from %s: %s" % (repr(addr), c)) Zope-2.8.7-final/lib/python/ZEO/zrpc/client.py0000664000076400007640000004656510435502417020113 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import errno import select import socket import sys import threading import time import types import logging import ThreadedAsync from ZODB.POSException import ReadOnlyError from ZODB.loglevels import BLATHER from ZEO.zrpc.log import log from ZEO.zrpc.trigger import trigger from ZEO.zrpc.connection import ManagedClientConnection class ConnectionManager(object): """Keeps a connection up over time""" def __init__(self, addrs, client, tmin=1, tmax=180): self.addrlist = self._parse_addrs(addrs) self.client = client self.tmin = tmin self.tmax = tmax self.cond = threading.Condition(threading.Lock()) self.connection = None # Protected by self.cond self.closed = 0 # If thread is not None, then there is a helper thread # attempting to connect. self.thread = None # Protected by self.cond self.trigger = None self.thr_async = 0 ThreadedAsync.register_loop_callback(self.set_async) def __repr__(self): return "<%s for %s>" % (self.__class__.__name__, self.addrlist) def _parse_addrs(self, addrs): # Return a list of (addr_type, addr) pairs. # For backwards compatibility (and simplicity?) the # constructor accepts a single address in the addrs argument -- # a string for a Unix domain socket or a 2-tuple with a # hostname and port. It can also accept a list of such addresses. addr_type = self._guess_type(addrs) if addr_type is not None: return [(addr_type, addrs)] else: addrlist = [] for addr in addrs: addr_type = self._guess_type(addr) if addr_type is None: raise ValueError, ( "unknown address in list: %s" % repr(addr)) addrlist.append((addr_type, addr)) return addrlist def _guess_type(self, addr): if isinstance(addr, types.StringType): return socket.AF_UNIX if (len(addr) == 2 and isinstance(addr[0], types.StringType) and isinstance(addr[1], types.IntType)): return socket.AF_INET # not anything I know about return None def close(self): """Prevent ConnectionManager from opening new connections""" self.closed = 1 ThreadedAsync.remove_loop_callback(self.set_async) self.cond.acquire() try: t = self.thread self.thread = None conn = self.connection finally: self.cond.release() if t is not None: log("CM.close(): stopping and joining thread") t.stop() t.join(30) if t.isAlive(): log("CM.close(): self.thread.join() timed out", level=logging.WARNING) if conn is not None: # This will call close_conn() below which clears self.connection conn.close() if self.trigger is not None: self.trigger.close() self.trigger = None ThreadedAsync.remove_loop_callback(self.set_async) def set_async(self, map): # This is the callback registered with ThreadedAsync. The # callback might be called multiple times, so it shouldn't # create a trigger every time and should never do anything # after it's closed. # It may be that the only case where it is called multiple # times is in the test suite, where ThreadedAsync's loop can # be started in a child process after a fork. Regardless, # it's good to be defensive. # We need each connection started with async==0 to have a # callback. log("CM.set_async(%s)" % repr(map), level=logging.DEBUG) if not self.closed and self.trigger is None: log("CM.set_async(): first call") self.trigger = trigger() self.thr_async = 1 # needs to be set on the Connection def attempt_connect(self): """Attempt a connection to the server without blocking too long. There isn't a crisp definition for too long. When a ClientStorage is created, it attempts to connect to the server. If the server isn't immediately available, it can operate from the cache. This method will start the background connection thread and wait a little while to see if it finishes quickly. """ # Will a single attempt take too long? # Answer: it depends -- normally, you'll connect or get a # connection refused error very quickly. Packet-eating # firewalls and other mishaps may cause the connect to take a # long time to time out though. It's also possible that you # connect quickly to a slow server, and the attempt includes # at least one roundtrip to the server (the register() call). # But that's as fast as you can expect it to be. self.connect() self.cond.acquire() try: t = self.thread conn = self.connection finally: self.cond.release() if t is not None and conn is None: event = t.one_attempt event.wait() self.cond.acquire() try: conn = self.connection finally: self.cond.release() return conn is not None def connect(self, sync=0): self.cond.acquire() try: if self.connection is not None: return t = self.thread if t is None: log("CM.connect(): starting ConnectThread") self.thread = t = ConnectThread(self, self.client, self.addrlist, self.tmin, self.tmax) t.setDaemon(1) t.start() if sync: while self.connection is None: self.cond.wait(30) if self.connection is None: log("CM.connect(sync=1): still waiting...") finally: self.cond.release() if sync: assert self.connection is not None def connect_done(self, conn, preferred): # Called by ConnectWrapper.notify_client() after notifying the client log("CM.connect_done(preferred=%s)" % preferred) self.cond.acquire() try: self.connection = conn if preferred: self.thread = None self.cond.notifyAll() # Wake up connect(sync=1) finally: self.cond.release() def close_conn(self, conn): # Called by the connection when it is closed self.cond.acquire() try: if conn is not self.connection: # Closing a non-current connection log("CM.close_conn() non-current", level=BLATHER) return log("CM.close_conn()") self.connection = None finally: self.cond.release() self.client.notifyDisconnected() if not self.closed: self.connect() def is_connected(self): self.cond.acquire() try: return self.connection is not None finally: self.cond.release() # When trying to do a connect on a non-blocking socket, some outcomes # are expected. Set _CONNECT_IN_PROGRESS to the errno value(s) expected # when an initial connect can't complete immediately. Set _CONNECT_OK # to the errno value(s) expected if the connect succeeds *or* if it's # already connected (our code can attempt redundant connects). if hasattr(errno, "WSAEWOULDBLOCK"): # Windows # Caution: The official Winsock docs claim that WSAEALREADY should be # treated as yet another "in progress" indicator, but we've never # seen this. _CONNECT_IN_PROGRESS = (errno.WSAEWOULDBLOCK,) # Win98: WSAEISCONN; Win2K: WSAEINVAL _CONNECT_OK = (0, errno.WSAEISCONN, errno.WSAEINVAL) else: # Unix _CONNECT_IN_PROGRESS = (errno.EINPROGRESS,) _CONNECT_OK = (0, errno.EISCONN) class ConnectThread(threading.Thread): """Thread that tries to connect to server given one or more addresses. The thread is passed a ConnectionManager and the manager's client as arguments. It calls testConnection() on the client when a socket connects; that should return 1 or 0 indicating whether this is a preferred or a fallback connection. It may also raise an exception, in which case the connection is abandoned. The thread will continue to run, attempting connections, until a preferred connection is seen and successfully handed over to the manager and client. As soon as testConnection() finds a preferred connection, or after all sockets have been tried and at least one fallback connection has been seen, notifyConnected(connection) is called on the client and connect_done() on the manager. If this was a preferred connection, the thread then exits; otherwise, it keeps trying until it gets a preferred connection, and then reconnects the client using that connection. """ __super_init = threading.Thread.__init__ # We don't expect clients to call any methods of this Thread other # than close() and those defined by the Thread API. def __init__(self, mgr, client, addrlist, tmin, tmax): self.__super_init(name="Connect(%s)" % addrlist) self.mgr = mgr self.client = client self.addrlist = addrlist self.tmin = tmin self.tmax = tmax self.stopped = 0 self.one_attempt = threading.Event() # A ConnectThread keeps track of whether it has finished a # call to try_connecting(). This allows the ConnectionManager # to make an attempt to connect right away, but not block for # too long if the server isn't immediately available. def stop(self): self.stopped = 1 def run(self): delay = self.tmin success = 0 # Don't wait too long the first time. # TODO: make timeout configurable? attempt_timeout = 5 while not self.stopped: success = self.try_connecting(attempt_timeout) if not self.one_attempt.isSet(): self.one_attempt.set() attempt_timeout = 75 if success > 0: break time.sleep(delay) if self.mgr.is_connected(): log("CT: still trying to replace fallback connection", level=logging.INFO) delay = min(delay*2, self.tmax) log("CT: exiting thread: %s" % self.getName()) def try_connecting(self, timeout): """Try connecting to all self.addrlist addresses. Return 1 if a preferred connection was found; 0 if no connection was found; and -1 if a fallback connection was found. If no connection is found within timeout seconds, return 0. """ log("CT: attempting to connect on %d sockets" % len(self.addrlist)) deadline = time.time() + timeout wrappers = self._create_wrappers() for wrap in wrappers.keys(): if wrap.state == "notified": return 1 try: if time.time() > deadline: return 0 r = self._connect_wrappers(wrappers, deadline) if r is not None: return r if time.time() > deadline: return 0 r = self._fallback_wrappers(wrappers, deadline) if r is not None: return r # Alas, no luck. assert not wrappers finally: for wrap in wrappers.keys(): wrap.close() del wrappers return 0 def _create_wrappers(self): # Create socket wrappers wrappers = {} # keys are active wrappers for domain, addr in self.addrlist: wrap = ConnectWrapper(domain, addr, self.mgr, self.client) wrap.connect_procedure() if wrap.state == "notified": for w in wrappers.keys(): w.close() return {wrap: wrap} if wrap.state != "closed": wrappers[wrap] = wrap return wrappers def _connect_wrappers(self, wrappers, deadline): # Next wait until they all actually connect (or fail) # The deadline is necessary, because we'd wait forever if a # sockets never connects or fails. while wrappers: if self.stopped: for wrap in wrappers.keys(): wrap.close() return 0 # Select connecting wrappers connecting = [wrap for wrap in wrappers.keys() if wrap.state == "connecting"] if not connecting: break if time.time() > deadline: break try: r, w, x = select.select([], connecting, connecting, 1.0) log("CT: select() %d, %d, %d" % tuple(map(len, (r,w,x)))) except select.error, msg: log("CT: select failed; msg=%s" % str(msg), level=logging.WARNING) continue # Exceptable wrappers are in trouble; close these suckers for wrap in x: log("CT: closing troubled socket %s" % str(wrap.addr)) del wrappers[wrap] wrap.close() # Writable sockets are connected for wrap in w: wrap.connect_procedure() if wrap.state == "notified": del wrappers[wrap] # Don't close this one for wrap in wrappers.keys(): wrap.close() return 1 if wrap.state == "closed": del wrappers[wrap] def _fallback_wrappers(self, wrappers, deadline): # If we've got wrappers left at this point, they're fallback # connections. Try notifying them until one succeeds. for wrap in wrappers.keys(): assert wrap.state == "tested" and wrap.preferred == 0 if self.mgr.is_connected(): wrap.close() else: wrap.notify_client() if wrap.state == "notified": del wrappers[wrap] # Don't close this one for wrap in wrappers.keys(): wrap.close() return -1 assert wrap.state == "closed" del wrappers[wrap] # TODO: should check deadline class ConnectWrapper: """An object that handles the connection procedure for one socket. This is a little state machine with states: closed opened connecting connected tested notified """ def __init__(self, domain, addr, mgr, client): """Store arguments and create non-blocking socket.""" self.domain = domain self.addr = addr self.mgr = mgr self.client = client # These attributes are part of the interface self.state = "closed" self.sock = None self.conn = None self.preferred = 0 log("CW: attempt to connect to %s" % repr(addr)) try: self.sock = socket.socket(domain, socket.SOCK_STREAM) except socket.error, err: log("CW: can't create socket, domain=%s: %s" % (domain, err), level=logging.ERROR) self.close() return self.sock.setblocking(0) self.state = "opened" def connect_procedure(self): """Call sock.connect_ex(addr) and interpret result.""" if self.state in ("opened", "connecting"): try: err = self.sock.connect_ex(self.addr) except socket.error, msg: log("CW: connect_ex(%r) failed: %s" % (self.addr, msg), level=logging.ERROR) self.close() return log("CW: connect_ex(%s) returned %s" % (self.addr, errno.errorcode.get(err) or str(err))) if err in _CONNECT_IN_PROGRESS: self.state = "connecting" return if err not in _CONNECT_OK: log("CW: error connecting to %s: %s" % (self.addr, errno.errorcode.get(err) or str(err)), level=logging.WARNING) self.close() return self.state = "connected" if self.state == "connected": self.test_connection() def test_connection(self): """Establish and test a connection at the zrpc level. Call the client's testConnection(), giving the client a chance to do app-level check of the connection. """ self.conn = ManagedClientConnection(self.sock, self.addr, self.client, self.mgr) self.sock = None # The socket is now owned by the connection try: self.preferred = self.client.testConnection(self.conn) self.state = "tested" except ReadOnlyError: log("CW: ReadOnlyError in testConnection (%s)" % repr(self.addr)) self.close() return except: log("CW: error in testConnection (%s)" % repr(self.addr), level=logging.ERROR, exc_info=True) self.close() return if self.preferred: self.notify_client() def notify_client(self): """Call the client's notifyConnected(). If this succeeds, call the manager's connect_done(). If the client is already connected, we assume it's a fallback connection, and the new connection must be a preferred connection. The client will close the old connection. """ try: self.client.notifyConnected(self.conn) except: log("CW: error in notifyConnected (%s)" % repr(self.addr), level=logging.ERROR, exc_info=True) self.close() return self.state = "notified" self.mgr.connect_done(self.conn, self.preferred) def close(self): """Close the socket and reset everything.""" self.state = "closed" self.mgr = self.client = None self.preferred = 0 if self.conn is not None: # Closing the ZRPC connection will eventually close the # socket, somewhere in asyncore. Guido asks: Why do we care? self.conn.close() self.conn = None if self.sock is not None: self.sock.close() self.sock = None def fileno(self): return self.sock.fileno() Zope-2.8.7-final/lib/python/ZEO/zrpc/error.py0000664000076400007640000000207310435502417017750 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## from ZODB import POSException from ZEO.Exceptions import ClientDisconnected class ZRPCError(POSException.StorageError): pass class DisconnectedError(ZRPCError, ClientDisconnected): """The database storage is disconnected from the storage server. The error occurred because a problem in the low-level RPC connection, or because the connection was closed. """ # This subclass is raised when zrpc catches the error. Zope-2.8.7-final/lib/python/ZEO/zrpc/_hmac.py0000664000076400007640000000640410435502417017670 0ustar ajungajung# This file is a slightly modified copy of Python 2.3's Lib/hmac.py. # This file is under the Python Software Foundation (PSF) license. """HMAC (Keyed-Hashing for Message Authentication) Python module. Implements the HMAC algorithm as described by RFC 2104. """ def _strxor(s1, s2): """Utility method. XOR the two strings s1 and s2 (must have same length). """ return "".join(map(lambda x, y: chr(ord(x) ^ ord(y)), s1, s2)) # The size of the digests returned by HMAC depends on the underlying # hashing module used. digest_size = None class HMAC: """RFC2104 HMAC class. This supports the API for Cryptographic Hash Functions (PEP 247). """ def __init__(self, key, msg = None, digestmod = None): """Create a new HMAC object. key: key for the keyed hash object. msg: Initial input for the hash, if provided. digestmod: A module supporting PEP 247. Defaults to the md5 module. """ if digestmod is None: import md5 digestmod = md5 self.digestmod = digestmod self.outer = digestmod.new() self.inner = digestmod.new() # Python 2.1 and 2.2 differ about the correct spelling try: self.digest_size = digestmod.digestsize except AttributeError: self.digest_size = digestmod.digest_size blocksize = 64 ipad = "\x36" * blocksize opad = "\x5C" * blocksize if len(key) > blocksize: key = digestmod.new(key).digest() key = key + chr(0) * (blocksize - len(key)) self.outer.update(_strxor(key, opad)) self.inner.update(_strxor(key, ipad)) if msg is not None: self.update(msg) ## def clear(self): ## raise NotImplementedError, "clear() method not available in HMAC." def update(self, msg): """Update this hashing object with the string msg. """ self.inner.update(msg) def copy(self): """Return a separate copy of this hashing object. An update to this copy won't affect the original object. """ other = HMAC("") other.digestmod = self.digestmod other.inner = self.inner.copy() other.outer = self.outer.copy() return other def digest(self): """Return the hash value of this hashing object. This returns a string containing 8-bit data. The object is not altered in any way by this function; you can continue updating the object after calling this function. """ h = self.outer.copy() h.update(self.inner.digest()) return h.digest() def hexdigest(self): """Like digest(), but returns a string of hexadecimal digits instead. """ return "".join([hex(ord(x))[2:].zfill(2) for x in tuple(self.digest())]) def new(key, msg = None, digestmod = None): """Create a new hashing object and return it. key: The starting key for the hash. msg: if available, will immediately be hashed into the object's starting state. You can now feed arbitrary strings into the object using its update() method, and can ask for the hash value at any time by calling its digest() method. """ return HMAC(key, msg, digestmod) Zope-2.8.7-final/lib/python/ZEO/zrpc/log.py0000664000076400007640000000470010435502417017377 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os import threading import logging from ZODB.loglevels import BLATHER LOG_THREAD_ID = 0 # Set this to 1 during heavy debugging logger = logging.getLogger('ZEO.zrpc') _label = "%s" % os.getpid() def new_label(): global _label _label = str(os.getpid()) def log(message, level=BLATHER, label=None, exc_info=False): label = label or _label if LOG_THREAD_ID: label = label + ':' + threading.currentThread().getName() logger.log(level, '(%s) %s' % (label, message), exc_info=exc_info) REPR_LIMIT = 60 def short_repr(obj): "Return an object repr limited to REPR_LIMIT bytes." # Some of the objects being repr'd are large strings. A lot of memory # would be wasted to repr them and then truncate, so they are treated # specially in this function. # Also handle short repr of a tuple containing a long string. # This strategy works well for arguments to StorageServer methods. # The oid is usually first and will get included in its entirety. # The pickle is near the beginning, too, and you can often fit the # module name in the pickle. if isinstance(obj, str): if len(obj) > REPR_LIMIT: r = repr(obj[:REPR_LIMIT]) else: r = repr(obj) if len(r) > REPR_LIMIT: r = r[:REPR_LIMIT-4] + '...' + r[-1] return r elif isinstance(obj, (list, tuple)): elts = [] size = 0 for elt in obj: r = short_repr(elt) elts.append(r) size += len(r) if size > REPR_LIMIT: break if isinstance(obj, tuple): r = "(%s)" % (", ".join(elts)) else: r = "[%s]" % (", ".join(elts)) else: r = repr(obj) if len(r) > REPR_LIMIT: return r[:REPR_LIMIT] + '...' else: return r Zope-2.8.7-final/lib/python/ZEO/ClientStub.py0000664000076400007640000000434310435502417017717 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """RPC stubs for interface exported by ClientStorage.""" class ClientStorage: """An RPC stub class for the interface exported by ClientStorage. This is the interface presented by ClientStorage to the StorageServer; i.e. the StorageServer calls these methods and they are executed in the ClientStorage. See the ClientStorage class for documentation on these methods. It is currently important that all methods here are asynchronous (meaning they don't have a return value and the caller doesn't wait for them to complete), *and* that none of them cause any calls from the client to the storage. This is due to limitations in the zrpc subpackage. The on-the-wire names of some of the methods don't match the Python method names. That's because the on-the-wire protocol was fixed for ZEO 2 and we don't want to change it. There are some aliases in ClientStorage.py to make up for this. """ def __init__(self, rpc): """Constructor. The argument is a connection: an instance of the zrpc.connection.Connection class. """ self.rpc = rpc def beginVerify(self): self.rpc.callAsync('beginVerify') def invalidateVerify(self, args): self.rpc.callAsync('invalidateVerify', args) def endVerify(self): self.rpc.callAsync('endVerify') def invalidateTransaction(self, tid, args): self.rpc.callAsyncNoPoll('invalidateTransaction', tid, args) def serialnos(self, arg): self.rpc.callAsync('serialnos', arg) def info(self, arg): self.rpc.callAsync('info', arg) Zope-2.8.7-final/lib/python/ZEO/cache.py0000664000076400007640000012427110435502417016711 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Disk-based client cache for ZEO. ClientCache exposes an API used by the ZEO client storage. FileCache stores objects on disk using a 2-tuple of oid and tid as key. ClientCache's API is similar to a storage API, with methods like load(), store(), and invalidate(). It manages in-memory data structures that allow it to map this richer API onto the simple key-based API of the lower-level FileCache. """ import bisect import logging import os import struct import tempfile import time from ZODB.utils import z64, u64 logger = logging.getLogger("ZEO.cache") ## # A disk-based cache for ZEO clients. #

# This class provides an interface to a persistent, disk-based cache # used by ZEO clients to store copies of database records from the # server. #

# The details of the constructor as unspecified at this point. #

# Each entry in the cache is valid for a particular range of transaction # ids. The lower bound is the transaction that wrote the data. The # upper bound is the next transaction that wrote a revision of the # object. If the data is current, the upper bound is stored as None; # the data is considered current until an invalidate() call is made. #

# It is an error to call store() twice with the same object without an # intervening invalidate() to set the upper bound on the first cache # entry. Perhaps it will be necessary to have a call the removes # something from the cache outright, without keeping a non-current # entry. #

Cache verification

#

# When the client is connected to the server, it receives # invalidations every time an object is modified. When the client is # disconnected then reconnects, it must perform cache verification to make # sure its cached data is synchronized with the storage's current state. #

# quick verification # full verification #

class ClientCache(object): """A simple in-memory cache.""" ## # Do we put the constructor here? # @param path path of persistent snapshot of cache state (a file path) # @param size size of cache file, in bytes # The default size of 200MB makes a lot more sense than the traditional # default of 20MB. The default here is misleading, though, since # ClientStorage is the only user of ClientCache, and it always passes an # explicit size of its own choosing. def __init__(self, path=None, size=200*1024**2): self.path = path self.size = size # The cache stores objects in a dict mapping (oid, tid) pairs # to Object() records (see below). The tid is the transaction # id that wrote the object. An object record includes data, # serialno, and end tid. It has auxillary data structures to # compute the appropriate tid, given the oid and a transaction id # representing an arbitrary point in history. # # The serialized form of the cache just stores the Object() # records. The in-memory form can be reconstructed from these # records. # Maps oid to current tid. Used to compute key for objects. self.current = {} # Maps oid to list of (start_tid, end_tid) pairs in sorted order. # Used to find matching key for load of non-current data. self.noncurrent = {} # Map oid to (version, tid) pair. If there is no entry, the object # is not modified in a version. self.version = {} # A FileCache instance does all the low-level work of storing # and retrieving objects to/from the cache file. self.fc = FileCache(size, self.path, self) self._setup_trace(self.path) def open(self): self.fc.scan(self.install) ## # Callback for FileCache.scan(), when a pre-existing file cache is # used. For each object in the file, `install()` is invoked. `f` # is the file object, positioned at the start of the serialized Object. # `ent` is an Entry giving the object's key ((oid, start_tid) pair). def install(self, f, ent): # Called by cache storage layer to insert object. o = Object.fromFile(f, ent.key, skip_data=True) if o is None: return oid = o.key[0] if o.version: self.version[oid] = o.version, o.start_tid elif o.end_tid is None: self.current[oid] = o.start_tid else: assert o.start_tid < o.end_tid this_span = o.start_tid, o.end_tid span_list = self.noncurrent.get(oid) if span_list: bisect.insort_left(span_list, this_span) else: self.noncurrent[oid] = [this_span] def close(self): self.fc.close() if self._tracefile: sync(self._tracefile) self._tracefile.close() self._tracefile = None ## # Set the last transaction seen by the cache. # @param tid a transaction id # @exception ValueError attempt to set a new tid less than the current tid def setLastTid(self, tid): self.fc.settid(tid) ## # Return the last transaction seen by the cache. # @return a transaction id # @defreturn string, or None if no transaction is yet known def getLastTid(self): if self.fc.tid == z64: return None else: return self.fc.tid ## # Return the current data record for oid and version. # @param oid object id # @param version a version string # @return (data record, serial number, tid), or None if the object is not # in the cache # @defreturn 3-tuple: (string, string, string) def load(self, oid, version=""): tid = None if version: p = self.version.get(oid) if p is None: self._trace(0x20, oid, version) return None elif p[0] == version: tid = p[1] # Otherwise, we know the cache has version data but not # for the requested version. Thus, we know it is safe # to return the non-version data from the cache. if tid is None: tid = self.current.get(oid) if tid is None: self._trace(0x20, oid, version) return None o = self.fc.access((oid, tid)) if o is None: self._trace(0x20, oid, version) return None self._trace(0x22, oid, version, o.start_tid, o.end_tid, len(o.data)) return o.data, tid, o.version ## # Return a non-current revision of oid that was current before tid. # @param oid object id # @param tid id of transaction that wrote next revision of oid # @return data record, serial number, start tid, and end tid # @defreturn 4-tuple: (string, string, string, string) def loadBefore(self, oid, tid): L = self.noncurrent.get(oid) if L is None: self._trace(0x24, oid, "", tid) return None # A pair with None as the second element is less than any pair with # the same first tid. Dubious: this relies on that None is less # than any comparable non-None object in recent Pythons. i = bisect.bisect_left(L, (tid, None)) # Now L[i-1] < (tid, None) < L[i], and the start_tid for everything in # L[:i] is < tid, and the start_tid for everything in L[i:] is >= tid. # Therefore the largest start_tid < tid must be at L[i-1]. If i is 0, # there is no start_tid < tid: we don't have any data old enougn. if i == 0: self._trace(0x24, oid, "", tid) return lo, hi = L[i-1] assert lo < tid if tid > hi: # we don't have any data in the right range self._trace(0x24, oid, "", tid) return None o = self.fc.access((oid, lo)) self._trace(0x26, oid, "", tid) return o.data, o.start_tid, o.end_tid ## # Return the version an object is modified in, or None for an # object that is not modified in a version. # @param oid object id # @return name of version in which the object is modified # @defreturn string or None def modifiedInVersion(self, oid): p = self.version.get(oid) if p is None: return None version, tid = p return version ## # Store a new data record in the cache. # @param oid object id # @param version name of version that oid was modified in. The cache # only stores current version data, so end_tid should # be None if version is not the empty string. # @param start_tid the id of the transaction that wrote this revision # @param end_tid the id of the transaction that created the next # revision of oid. If end_tid is None, the data is # current. # @param data the actual data # @exception ValueError tried to store non-current version data def store(self, oid, version, start_tid, end_tid, data): # It's hard for the client to avoid storing the same object # more than once. One case is when the client requests # version data that doesn't exist. It checks the cache for # the requested version, doesn't find it, then asks the server # for that data. The server returns the non-version data, # which may already be in the cache. if (oid, start_tid) in self.fc: return o = Object((oid, start_tid), version, data, start_tid, end_tid) if version: if end_tid is not None: raise ValueError("cache only stores current version data") if oid in self.version: if self.version[oid] != (version, start_tid): raise ValueError("data already exists for version %r" % self.version[oid][0]) self.version[oid] = version, start_tid self._trace(0x50, oid, version, start_tid, dlen=len(data)) else: if end_tid is None: _cur_start = self.current.get(oid) if _cur_start: if _cur_start != start_tid: raise ValueError( "already have current data for oid") else: return self.current[oid] = start_tid self._trace(0x52, oid, version, start_tid, dlen=len(data)) else: L = self.noncurrent.setdefault(oid, []) p = start_tid, end_tid if p in L: return # duplicate store bisect.insort_left(L, p) self._trace(0x54, oid, version, start_tid, end_tid, dlen=len(data)) self.fc.add(o) ## # Remove all knowledge of noncurrent revisions of oid, both in # self.noncurrent and in our FileCache. `version` and `tid` are used # only for trace records. def _remove_noncurrent_revisions(self, oid, version, tid): noncurrent_list = self.noncurrent.get(oid) if noncurrent_list: # Note: must iterate over a copy of noncurrent_list. The # FileCache remove() calls our _evicted() method, and that # mutates the list. for old_tid, dummy in noncurrent_list[:]: # 0x1E = invalidate (hit, discarding current or non-current) self._trace(0x1E, oid, version, tid) self.fc.remove((oid, old_tid)) # fc.remove() calling back to _evicted() should have removed # the list from noncurrent when the last non-current revision # was removed. assert oid not in self.noncurrent ## # If `tid` is None, or we have data for `oid` in a (non-empty) version, # forget all knowledge of `oid`. (`tid` can be None only for # invalidations generated by startup cache verification.) If `tid` # isn't None, we don't have version data for `oid`, and we had current # data for `oid`, stop believing we have current data, and mark the # data we had as being valid only up to `tid`. In all other cases, do # nothing. # @param oid object id # @param version name of version to invalidate. # @param tid the id of the transaction that wrote a new revision of oid, # or None to forget all cached info about oid (version, current # revision, and non-current revisions) def invalidate(self, oid, version, tid): if tid > self.fc.tid and tid is not None: self.fc.settid(tid) remove_all_knowledge_of_oid = tid is None if oid in self.version: # Forget we know about the version data. # 0x1A = invalidate (hit, version) self._trace(0x1A, oid, version, tid) dllversion, dlltid = self.version[oid] assert not version or version == dllversion, (version, dllversion) self.fc.remove((oid, dlltid)) assert oid not in self.version # .remove() got rid of it # And continue: we must also remove any non-version data from # the cache. Or, at least, I have such a poor understanding of # versions that anything less drastic would probably be wrong. remove_all_knowledge_of_oid = True if remove_all_knowledge_of_oid: self._remove_noncurrent_revisions(oid, version, tid) # Only current, non-version data remains to be handled. cur_tid = self.current.get(oid) if not cur_tid: # 0x10 == invalidate (miss) self._trace(0x10, oid, version, tid) return # We had current data for oid, but no longer. if remove_all_knowledge_of_oid: # 0x1E = invalidate (hit, discarding current or non-current) self._trace(0x1E, oid, version, tid) self.fc.remove((oid, cur_tid)) assert cur_tid not in self.current # .remove() got rid of it return # Add the data we have to the list of non-current data for oid. assert tid is not None and cur_tid < tid # 0x1C = invalidate (hit, saving non-current) self._trace(0x1C, oid, version, tid) del self.current[oid] # because we no longer have current data # Update the end_tid half of oid's validity range on disk. # TODO: Want to fetch object without marking it as accessed. o = self.fc.access((oid, cur_tid)) assert o is not None assert o.end_tid is None # i.e., o was current if o is None: # TODO: Since we asserted o is not None above, this block # should be removed; waiting on time to prove it can't happen. return o.end_tid = tid self.fc.update(o) # record the new end_tid on disk # Add to oid's list of non-current data. L = self.noncurrent.setdefault(oid, []) bisect.insort_left(L, (cur_tid, tid)) ## # Return the number of object revisions in the cache. # # Or maybe better to just return len(self.cache)? Needs clearer use case. def __len__(self): n = len(self.current) + len(self.version) if self.noncurrent: n += sum(map(len, self.noncurrent)) return n ## # Generates (oid, serial, version) triples for all objects in the # cache. This generator is used by cache verification. def contents(self): # May need to materialize list instead of iterating; # depends on whether the caller may change the cache. for o in self.fc: oid, tid = o.key if oid in self.version: obj = self.fc.access(o.key) yield oid, tid, obj.version else: yield oid, tid, "" def dump(self): from ZODB.utils import oid_repr print "cache size", len(self) L = list(self.contents()) L.sort() for oid, tid, version in L: print oid_repr(oid), oid_repr(tid), repr(version) print "dll contents" L = list(self.fc) L.sort(lambda x, y: cmp(x.key, y.key)) for x in L: end_tid = x.end_tid or z64 print oid_repr(x.key[0]), oid_repr(x.key[1]), oid_repr(end_tid) print def _evicted(self, o): # Called by the FileCache to signal that Object o has been evicted. oid, tid = o.key if o.end_tid is None: if o.version: del self.version[oid] else: del self.current[oid] else: # Although we use bisect to keep the list sorted, # we never expect the list to be very long. So the # brute force approach should normally be fine. L = self.noncurrent[oid] element = (o.start_tid, o.end_tid) if len(L) == 1: # We don't want to leave an empty list in the dict: if # the oid is never referenced again, it would consume RAM # forever more for no purpose. assert L[0] == element del self.noncurrent[oid] else: L.remove(element) # If `path` isn't None (== we're using a persistent cache file), and # envar ZEO_CACHE_TRACE is set to a non-empty value, try to open # path+'.trace' as a trace file, and store the file object in # self._tracefile. If not, or we can't write to the trace file, disable # tracing by setting self._trace to a dummy function, and set # self._tracefile to None. def _setup_trace(self, path): self._tracefile = None if path and os.environ.get("ZEO_CACHE_TRACE"): tfn = path + ".trace" try: self._tracefile = open(tfn, "ab") self._trace(0x00) except IOError, msg: self._tracefile = None logger.warning("cannot write tracefile %r (%s)", tfn, msg) else: logger.info("opened tracefile %r", tfn) if self._tracefile is None: def notrace(*args, **kws): pass self._trace = notrace def _trace(self, code, oid="", version="", tid=z64, end_tid=z64, dlen=0, # The next two are just speed hacks. time_time=time.time, struct_pack=struct.pack): # The code argument is two hex digits; bits 0 and 7 must be zero. # The first hex digit shows the operation, the second the outcome. # This method has been carefully tuned to be as fast as possible. # Note: when tracing is disabled, this method is hidden by a dummy. if version: code |= 0x80 encoded = (dlen + 255) & 0x7fffff00 | code if tid is None: tid = z64 if end_tid is None: end_tid = z64 try: self._tracefile.write( struct_pack(">iiH8s8s", time_time(), encoded, len(oid), tid, end_tid) + oid) except: print `tid`, `end_tid` raise ## # An Object stores the cached data for a single object. #

# The cached data includes the actual object data, the key, and three # data fields that describe the validity period and version of the # object. The key contains the oid and a redundant start_tid. The # actual size of an object is variable, depending on the size of the # data and whether it is in a version. #

# The serialized format does not include the key, because it is stored # in the header used by the cache file's storage format. #

# Instances of Object are generally short-lived -- they're really a way to # package data on the way to or from the disk file. class Object(object): __slots__ = (# pair (object id, txn id) -- something usable as a dict key; # the second part of the pair is equal to start_tid "key", # string, tid of txn that wrote the data "start_tid", # string, tid of txn that wrote next revision, or None # if the data is current; if not None, end_tid is strictly # greater than start_tid "end_tid", # string, name of version "version", # string, the actual data record for the object "data", # total size of serialized object; this includes the # data, version, and all overhead (header) bytes. "size", ) # A serialized Object on disk looks like: # # offset # bytes value # ------ ------- ----- # 0 8 end_tid; string # 8 2 len(version); 2-byte signed int # 10 4 len(data); 4-byte signed int # 14 len(version) version; string # 14+len(version) len(data) the object pickle; string # 14+len(version)+ # len(data) 8 oid; string # The serialization format uses an end tid of "\0"*8 (z64), the least # 8-byte string, to represent None. It isn't possible for an end_tid # to be 0, because it must always be strictly greater than the start_tid. fmt = ">8shi" # end_tid, len(self.version), len(self.data) FIXED_HEADER_SIZE = struct.calcsize(fmt) assert FIXED_HEADER_SIZE == 14 TOTAL_FIXED_SIZE = FIXED_HEADER_SIZE + 8 # +8 for the oid at the end def __init__(self, key, version, data, start_tid, end_tid): self.key = key self.version = version self.data = data self.start_tid = start_tid self.end_tid = end_tid # The size of the serialized object on disk, including the # 14-byte header, the lengths of data and version, and a # copy of the 8-byte oid. if data is not None: self.size = self.TOTAL_FIXED_SIZE + len(data) + len(version) ## # Return the fixed-sized serialization header as a string: pack end_tid, # and the lengths of the .version and .data members. def get_header(self): return struct.pack(self.fmt, self.end_tid or z64, len(self.version), len(self.data)) ## # Write the serialized representation of self to file f, at its current # position. def serialize(self, f): f.writelines([self.get_header(), self.version, self.data, self.key[0]]) ## # Write the fixed-size header for self, to file f at its current position. # The only real use for this is when the current revision of an object # in cache is invalidated. Then the end_tid field gets set to the tid # of the transaction that caused the invalidation. def serialize_header(self, f): f.write(self.get_header()) ## # fromFile is a class constructor, unserializing an Object from the # current position in file f. Exclusive access to f for the duration # is assumed. The key is a (oid, start_tid) pair, and the oid must # match the serialized oid. If `skip_data` is true, .data is left # None in the Object returned, but all the other fields are populated. # Else (`skip_data` is false, the default), all fields including .data # are populated. .data can be big, so it's prudent to skip it when it # isn't needed. def fromFile(cls, f, key, skip_data=False): s = f.read(cls.FIXED_HEADER_SIZE) if not s: return None oid, start_tid = key end_tid, vlen, dlen = struct.unpack(cls.fmt, s) if end_tid == z64: end_tid = None version = f.read(vlen) if vlen != len(version): raise ValueError("corrupted record, version") if skip_data: data = None f.seek(dlen, 1) else: data = f.read(dlen) if dlen != len(data): raise ValueError("corrupted record, data") s = f.read(8) if s != oid: raise ValueError("corrupted record, oid") return cls((oid, start_tid), version, data, start_tid, end_tid) fromFile = classmethod(fromFile) # Entry just associates a key with a file offset. It's used by FileCache. class Entry(object): __slots__ = (# object key -- something usable as a dict key. 'key', # Offset from start of file to the object's data # record; this includes all overhead bytes (status # byte, size bytes, etc). The size of the data # record is stored in the file near the start of the # record, but for efficiency we also keep size in a # dict (filemap; see later). 'offset', ) def __init__(self, key=None, offset=None): self.key = key self.offset = offset ## # FileCache stores a cache in a single on-disk file. # # On-disk cache structure. # # The file begins with a 12-byte header. The first four bytes are the # file's magic number - ZEC3 - indicating zeo cache version 3. The # next eight bytes are the last transaction id. magic = "ZEC3" ZEC3_HEADER_SIZE = 12 # After the header, the file contains a contiguous sequence of blocks. All # blocks begin with a one-byte status indicator: # # 'a' # Allocated. The block holds an object; the next 4 bytes are >I # format total block size. # # 'f' # Free. The block is free; the next 4 bytes are >I format total # block size. # # '1', '2', '3', '4' # The block is free, and consists of 1, 2, 3 or 4 bytes total. # # "Total" includes the status byte, and size bytes. There are no # empty (size 0) blocks. # Allocated blocks have more structure: # # 1 byte allocation status ('a'). # 4 bytes block size, >I format. # 16 bytes oid + tid, string. # size-OBJECT_HEADER_SIZE bytes, the serialization of an Object (see # class Object for details). OBJECT_HEADER_SIZE = 1 + 4 + 16 # The cache's currentofs goes around the file, circularly, forever. # It's always the starting offset of some block. # # When a new object is added to the cache, it's stored beginning at # currentofs, and currentofs moves just beyond it. As many contiguous # blocks needed to make enough room for the new object are evicted, # starting at currentofs. Exception: if currentofs is close enough # to the end of the file that the new object can't fit in one # contiguous chunk, currentofs is reset to ZEC3_HEADER_SIZE first. # Do all possible to ensure that the bytes we wrote to file f are really on # disk. def sync(f): f.flush() if hasattr(os, 'fsync'): os.fsync(f.fileno()) class FileCache(object): def __init__(self, maxsize, fpath, parent): # - `maxsize`: total size of the cache file, in bytes; this is # ignored path names an existing file; perhaps we should attempt # to change the cache size in that case # - `fpath`: filepath for the cache file, or None (in which case # a temp file will be created) # - `parent`: the ClientCache instance; its `_evicted()` method # is called whenever we need to evict an object to make room in # the file self.maxsize = maxsize self.parent = parent # tid for the most recent transaction we know about. This is also # stored near the start of the file. self.tid = None # There's one Entry instance, kept in memory, for each currently # allocated block in the file, and there's one allocated block in the # file per serialized Object. filemap retrieves the Entry given the # starting offset of a block, and key2entry retrieves the Entry given # an object revision's key (an (oid, start_tid) pair). From an # Entry, we can get the Object's key and file offset. # Map offset in file to pair (data record size, Entry). # Entry is None iff the block starting at offset is free. # filemap always contains a complete account of what's in the # file -- study method _verify_filemap for executable checking # of the relevant invariants. An offset is at the start of a # block iff it's a key in filemap. The data record size is # stored in the file too, so we could just seek to the offset # and read it up; keeping it in memory is an optimization. self.filemap = {} # Map key to Entry. After # obj = key2entry[key] # then # obj.key == key # is true. An object is currently stored on disk iff its key is in # key2entry. self.key2entry = {} # Always the offset into the file of the start of a block. # New and relocated objects are always written starting at # currentofs. self.currentofs = ZEC3_HEADER_SIZE # self.f is the open file object. # When we're not reusing an existing file, self.f is left None # here -- the scan() method must be called then to open the file # (and it sets self.f). self.fpath = fpath if fpath and os.path.exists(fpath): # Reuse an existing file. scan() will open & read it. self.f = None logger.info("reusing persistent cache file %r", fpath) else: if fpath: self.f = open(fpath, 'wb+') logger.info("created persistent cache file %r", fpath) else: self.f = tempfile.TemporaryFile() logger.info("created temporary cache file %r", self.f.name) # Make sure the OS really saves enough bytes for the file. self.f.seek(self.maxsize - 1) self.f.write('x') self.f.truncate() # Start with one magic header block self.f.seek(0) self.f.write(magic) self.f.write(z64) # and one free block. self.f.write('f' + struct.pack(">I", self.maxsize - ZEC3_HEADER_SIZE)) self.sync() self.filemap[ZEC3_HEADER_SIZE] = (self.maxsize - ZEC3_HEADER_SIZE, None) # Statistics: _n_adds, _n_added_bytes, # _n_evicts, _n_evicted_bytes, # _n_accesses self.clearStats() ## # Scan the current contents of the cache file, calling `install` # for each object found in the cache. This method should only # be called once to initialize the cache from disk. def scan(self, install): if self.f is not None: # we're not (re)using a pre-existing file return fsize = os.path.getsize(self.fpath) if fsize != self.maxsize: logger.warning("existing cache file %r has size %d; " "requested size %d ignored", self.fpath, fsize, self.maxsize) self.maxsize = fsize self.f = open(self.fpath, 'rb+') _magic = self.f.read(4) if _magic != magic: raise ValueError("unexpected magic number: %r" % _magic) self.tid = self.f.read(8) if len(self.tid) != 8: raise ValueError("cache file too small -- no tid at start") # Populate .filemap and .key2entry to reflect what's currently in the # file, and tell our parent about it too (via the `install` callback). # Remember the location of the largest free block. That seems a # decent place to start currentofs. max_free_size = max_free_offset = 0 ofs = ZEC3_HEADER_SIZE while ofs < fsize: self.f.seek(ofs) ent = None status = self.f.read(1) if status == 'a': size, rawkey = struct.unpack(">I16s", self.f.read(20)) key = rawkey[:8], rawkey[8:] assert key not in self.key2entry self.key2entry[key] = ent = Entry(key, ofs) install(self.f, ent) elif status == 'f': size, = struct.unpack(">I", self.f.read(4)) elif status in '1234': size = int(status) else: raise ValueError("unknown status byte value %s in client " "cache file" % 0, hex(ord(status))) self.filemap[ofs] = size, ent if ent is None and size > max_free_size: max_free_size, max_free_offset = size, ofs ofs += size if ofs != fsize: raise ValueError("final offset %s != file size %s in client " "cache file" % (ofs, fsize)) if __debug__: self._verify_filemap() self.currentofs = max_free_offset def clearStats(self): self._n_adds = self._n_added_bytes = 0 self._n_evicts = self._n_evicted_bytes = 0 self._n_accesses = 0 def getStats(self): return (self._n_adds, self._n_added_bytes, self._n_evicts, self._n_evicted_bytes, self._n_accesses ) ## # The number of objects currently in the cache. def __len__(self): return len(self.key2entry) ## # Iterate over the objects in the cache, producing an Entry for each. def __iter__(self): return self.key2entry.itervalues() ## # Test whether an (oid, tid) pair is in the cache. def __contains__(self, key): return key in self.key2entry ## # Do all possible to ensure all bytes written to the file so far are # actually on disk. def sync(self): sync(self.f) ## # Close the underlying file. No methods accessing the cache should be # used after this. def close(self): if self.f: self.sync() self.f.close() self.f = None ## # Evict objects as necessary to free up at least nbytes bytes, # starting at currentofs. If currentofs is closer than nbytes to # the end of the file, currentofs is reset to ZEC3_HEADER_SIZE first. # The number of bytes actually freed may be (and probably will be) # greater than nbytes, and is _makeroom's return value. The file is not # altered by _makeroom. filemap and key2entry are updated to reflect the # evictions, and it's the caller's responsibility both to fiddle # the file, and to update filemap, to account for all the space # freed (starting at currentofs when _makeroom returns, and # spanning the number of bytes retured by _makeroom). def _makeroom(self, nbytes): assert 0 < nbytes <= self.maxsize - ZEC3_HEADER_SIZE if self.currentofs + nbytes > self.maxsize: self.currentofs = ZEC3_HEADER_SIZE ofs = self.currentofs while nbytes > 0: size, e = self.filemap.pop(ofs) if e is not None: del self.key2entry[e.key] self._evictobj(e, size) ofs += size nbytes -= size return ofs - self.currentofs ## # Write Object obj, with data, to file starting at currentofs. # nfreebytes are already available for overwriting, and it's # guranteed that's enough. obj.offset is changed to reflect the # new data record position, and filemap and key2entry are updated to # match. def _writeobj(self, obj, nfreebytes): size = OBJECT_HEADER_SIZE + obj.size assert size <= nfreebytes excess = nfreebytes - size # If there's any excess (which is likely), we need to record a # free block following the end of the data record. That isn't # expensive -- it's all a contiguous write. if excess == 0: extra = '' elif excess < 5: extra = "01234"[excess] else: extra = 'f' + struct.pack(">I", excess) self.f.seek(self.currentofs) self.f.writelines(('a', struct.pack(">I8s8s", size, obj.key[0], obj.key[1]))) obj.serialize(self.f) self.f.write(extra) e = Entry(obj.key, self.currentofs) self.key2entry[obj.key] = e self.filemap[self.currentofs] = size, e self.currentofs += size if excess: # We need to record the free block in filemap, but there's # no need to advance currentofs beyond it. Instead it # gives some breathing room for the next object to get # written. self.filemap[self.currentofs] = excess, None ## # Add Object object to the cache. This may evict existing objects, to # make room (and almost certainly will, in steady state once the cache # is first full). The object must not already be in the cache. def add(self, object): size = OBJECT_HEADER_SIZE + object.size # A number of cache simulation experiments all concluded that the # 2nd-level ZEO cache got a much higher hit rate if "very large" # objects simply weren't cached. For now, we ignore the request # only if the entire cache file is too small to hold the object. if size > self.maxsize - ZEC3_HEADER_SIZE: return assert object.key not in self.key2entry assert len(object.key[0]) == 8 assert len(object.key[1]) == 8 self._n_adds += 1 self._n_added_bytes += size available = self._makeroom(size) self._writeobj(object, available) ## # Evict the object represented by Entry `e` from the cache, freeing # `size` bytes in the file for reuse. `size` is used only for summary # statistics. This does not alter the file, or self.filemap or # self.key2entry (those are the caller's responsibilities). It does # invoke _evicted(Object) on our parent. def _evictobj(self, e, size): self._n_evicts += 1 self._n_evicted_bytes += size # Load the object header into memory so we know how to # update the parent's in-memory data structures. self.f.seek(e.offset + OBJECT_HEADER_SIZE) o = Object.fromFile(self.f, e.key, skip_data=True) self.parent._evicted(o) ## # Return Object for key, or None if not in cache. def access(self, key): self._n_accesses += 1 e = self.key2entry.get(key) if e is None: return None offset = e.offset size, e2 = self.filemap[offset] assert e is e2 self.f.seek(offset + OBJECT_HEADER_SIZE) return Object.fromFile(self.f, key) ## # Remove Object for key from cache, if present. def remove(self, key): # If an object is being explicitly removed, we need to load # its header into memory and write a free block marker to the # disk where the object was stored. We need to load the # header to update the in-memory data structures held by # ClientCache. # We could instead just keep the header in memory at all times. e = self.key2entry.pop(key, None) if e is None: return offset = e.offset size, e2 = self.filemap[offset] assert e is e2 self.filemap[offset] = size, None self.f.seek(offset + OBJECT_HEADER_SIZE) o = Object.fromFile(self.f, key, skip_data=True) assert size >= 5 # only free blocks are tiny # Because `size` >= 5, we can change an allocated block to a free # block just by overwriting the 'a' status byte with 'f' -- the # size field stays the same. self.f.seek(offset) self.f.write('f') self.f.flush() self.parent._evicted(o) ## # Update on-disk representation of Object obj. # # This method should be called when the object header is modified. # obj must be in the cache. The only real use for this is during # invalidation, to set the end_tid field on a revision that was current # (and so had an end_tid of None, but no longer does). def update(self, obj): e = self.key2entry[obj.key] self.f.seek(e.offset + OBJECT_HEADER_SIZE) obj.serialize_header(self.f) ## # Update our idea of the most recent tid. This is stored in the # instance, and also written out near the start of the cache file. The # new tid must be strictly greater than our current idea of the most # recent tid. def settid(self, tid): if self.tid is not None and tid <= self.tid: raise ValueError("new last tid (%s) must be greater than " "previous one (%s)" % (u64(tid), u64(self.tid))) assert isinstance(tid, str) and len(tid) == 8 self.tid = tid self.f.seek(len(magic)) self.f.write(tid) self.f.flush() ## # This debug method marches over the entire cache file, verifying that # the current contents match the info in self.filemap and self.key2entry. def _verify_filemap(self, display=False): a = ZEC3_HEADER_SIZE f = self.f while a < self.maxsize: f.seek(a) status = f.read(1) if status in 'af': size, = struct.unpack(">I", f.read(4)) else: size = int(status) if display: if a == self.currentofs: print '*****', print "%c%d" % (status, size), size2, obj = self.filemap[a] assert size == size2 assert (obj is not None) == (status == 'a') if obj is not None: assert obj.offset == a assert self.key2entry[obj.key] is obj a += size if display: print assert a == self.maxsize Zope-2.8.7-final/lib/python/ZEO/auth/0000775000076400007640000000000010435610645016231 5ustar ajungajungZope-2.8.7-final/lib/python/ZEO/auth/base.py0000664000076400007640000001023610435502417017514 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Base classes for defining an authentication protocol. Database -- abstract base class for password database Client -- abstract base class for authentication client """ import os import sha class Client: # Subclass should override to list the names of methods that # will be called on the server. extensions = [] def __init__(self, stub): self.stub = stub for m in self.extensions: setattr(self.stub, m, self.stub.extensionMethod(m)) def sort(L): """Sort a list in-place and return it.""" L.sort() return L class Database: """Abstracts a password database. This class is used both in the authentication process (via get_password()) and by client scripts that manage the password database file. The password file is a simple, colon-separated text file mapping usernames to password hashes. The hashes are SHA hex digests produced from the password string. """ realm = None def __init__(self, filename, realm=None): """Creates a new Database filename: a string containing the full pathname of the password database file. Must be readable by the user running ZEO. Must be writeable by any client script that accesses the database. realm: the realm name (a string) """ self._users = {} self.filename = filename self.load() if realm: if self.realm and self.realm != realm: raise ValueError, ("Specified realm %r differs from database " "realm %r" % (realm or '', self.realm)) else: self.realm = realm def save(self, fd=None): filename = self.filename if not fd: fd = open(filename, 'w') if self.realm: print >> fd, "realm", self.realm for username in sort(self._users.keys()): print >> fd, "%s: %s" % (username, self._users[username]) def load(self): filename = self.filename if not filename: return if not os.path.exists(filename): return fd = open(filename) L = fd.readlines() if not L: return if L[0].startswith("realm "): line = L.pop(0).strip() self.realm = line[len("realm "):] for line in L: username, hash = line.strip().split(":", 1) self._users[username] = hash.strip() def _store_password(self, username, password): self._users[username] = self.hash(password) def get_password(self, username): """Returns password hash for specified username. Callers must check for LookupError, which is raised in the case of a non-existent user specified.""" if not self._users.has_key(username): raise LookupError, "No such user: %s" % username return self._users[username] def hash(self, s): return sha.new(s).hexdigest() def add_user(self, username, password): if self._users.has_key(username): raise LookupError, "User %s already exists" % username self._store_password(username, password) def del_user(self, username): if not self._users.has_key(username): raise LookupError, "No such user: %s" % username del self._users[username] def change_password(self, username, password): if not self._users.has_key(username): raise LookupError, "No such user: %s" % username self._store_password(username, password) Zope-2.8.7-final/lib/python/ZEO/auth/__init__.py0000664000076400007640000000230710435502417020341 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## _auth_modules = {} def get_module(name): if name == 'sha': from auth_sha import StorageClass, SHAClient, Database return StorageClass, SHAClient, Database elif name == 'digest': from auth_digest import StorageClass, DigestClient, DigestDatabase return StorageClass, DigestClient, DigestDatabase else: return _auth_modules.get(name) def register_module(name, storage_class, client, db): if _auth_modules.has_key(name): raise TypeError, "%s is already registred" % name _auth_modules[name] = storage_class, client, db Zope-2.8.7-final/lib/python/ZEO/auth/auth_digest.py0000664000076400007640000001234510435502417021105 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Digest authentication for ZEO This authentication mechanism follows the design of HTTP digest authentication (RFC 2069). It is a simple challenge-response protocol that does not send passwords in the clear, but does not offer strong security. The RFC discusses many of the limitations of this kind of protocol. Guard the password database as if it contained plaintext passwords. It stores the hash of a username and password. This does not expose the plaintext password, but it is sensitive nonetheless. An attacker with the hash can impersonate the real user. This is a limitation of the simple digest scheme. HTTP is a stateless protocol, and ZEO is a stateful protocol. The security requirements are quite different as a result. The HTTP protocol uses a nonce as a challenge. The ZEO protocol requires a separate session key that is used for message authentication. We generate a second nonce for this purpose; the hash of nonce and user/realm/password is used as the session key. TODO: I'm not sure if this is a sound approach; SRP would be preferred. """ import os import random import sha import struct import time from ZEO.auth.base import Database, Client from ZEO.StorageServer import ZEOStorage from ZEO.Exceptions import AuthError def get_random_bytes(n=8): if os.path.exists("/dev/urandom"): f = open("/dev/urandom") s = f.read(n) f.close() else: L = [chr(random.randint(0, 255)) for i in range(n)] s = "".join(L) return s def hexdigest(s): return sha.new(s).hexdigest() class DigestDatabase(Database): def __init__(self, filename, realm=None): Database.__init__(self, filename, realm) # Initialize a key used to build the nonce for a challenge. # We need one key for the lifetime of the server, so it # is convenient to store in on the database. self.noncekey = get_random_bytes(8) def _store_password(self, username, password): dig = hexdigest("%s:%s:%s" % (username, self.realm, password)) self._users[username] = dig def session_key(h_up, nonce): # The hash itself is a bit too short to be a session key. # HMAC wants a 64-byte key. We don't want to use h_up # directly because it would never change over time. Instead # use the hash plus part of h_up. return sha.new("%s:%s" % (h_up, nonce)).digest() + h_up[:44] class StorageClass(ZEOStorage): def set_database(self, database): assert isinstance(database, DigestDatabase) self.database = database self.noncekey = database.noncekey def _get_time(self): # Return a string representing the current time. t = int(time.time()) return struct.pack("i", t) def _get_nonce(self): # RFC 2069 recommends a nonce of the form # H(client-IP ":" time-stamp ":" private-key) dig = sha.sha() dig.update(str(self.connection.addr)) dig.update(self._get_time()) dig.update(self.noncekey) return dig.hexdigest() def auth_get_challenge(self): """Return realm, challenge, and nonce.""" self._challenge = self._get_nonce() self._key_nonce = self._get_nonce() return self.auth_realm, self._challenge, self._key_nonce def auth_response(self, resp): # verify client response user, challenge, response = resp # Since zrpc is a stateful protocol, we just store the nonce # we sent to the client. It will need to generate a new # nonce for a new connection anyway. if self._challenge != challenge: raise ValueError, "invalid challenge" # lookup user in database h_up = self.database.get_password(user) # regeneration resp from user, password, and nonce check = hexdigest("%s:%s" % (h_up, challenge)) if check == response: self.connection.setSessionKey(session_key(h_up, self._key_nonce)) return self.finish_auth(check == response) extensions = [auth_get_challenge, auth_response] class DigestClient(Client): extensions = ["auth_get_challenge", "auth_response"] def start(self, username, realm, password): _realm, challenge, nonce = self.stub.auth_get_challenge() if _realm != realm: raise AuthError("expected realm %r, got realm %r" % (_realm, realm)) h_up = hexdigest("%s:%s:%s" % (username, realm, password)) resp_dig = hexdigest("%s:%s" % (h_up, challenge)) result = self.stub.auth_response((username, challenge, resp_dig)) if result: return session_key(h_up, nonce) else: return None Zope-2.8.7-final/lib/python/ZEO/auth/hmac.py0000664000076400007640000000573010435502417017515 0ustar ajungajung"""HMAC (Keyed-Hashing for Message Authentication) Python module. Implements the HMAC algorithm as described by RFC 2104. """ def _strxor(s1, s2): """Utility method. XOR the two strings s1 and s2 (must have same length). """ return "".join(map(lambda x, y: chr(ord(x) ^ ord(y)), s1, s2)) # The size of the digests returned by HMAC depends on the underlying # hashing module used. digest_size = None class HMAC: """RFC2104 HMAC class. This supports the API for Cryptographic Hash Functions (PEP 247). """ def __init__(self, key, msg = None, digestmod = None): """Create a new HMAC object. key: key for the keyed hash object. msg: Initial input for the hash, if provided. digestmod: A module supporting PEP 247. Defaults to the md5 module. """ if digestmod is None: import md5 digestmod = md5 self.digestmod = digestmod self.outer = digestmod.new() self.inner = digestmod.new() self.digest_size = digestmod.digest_size blocksize = 64 ipad = "\x36" * blocksize opad = "\x5C" * blocksize if len(key) > blocksize: key = digestmod.new(key).digest() key = key + chr(0) * (blocksize - len(key)) self.outer.update(_strxor(key, opad)) self.inner.update(_strxor(key, ipad)) if msg is not None: self.update(msg) ## def clear(self): ## raise NotImplementedError, "clear() method not available in HMAC." def update(self, msg): """Update this hashing object with the string msg. """ self.inner.update(msg) def copy(self): """Return a separate copy of this hashing object. An update to this copy won't affect the original object. """ other = HMAC("") other.digestmod = self.digestmod other.inner = self.inner.copy() other.outer = self.outer.copy() return other def digest(self): """Return the hash value of this hashing object. This returns a string containing 8-bit data. The object is not altered in any way by this function; you can continue updating the object after calling this function. """ h = self.outer.copy() h.update(self.inner.digest()) return h.digest() def hexdigest(self): """Like digest(), but returns a string of hexadecimal digits instead. """ return "".join([hex(ord(x))[2:].zfill(2) for x in tuple(self.digest())]) def new(key, msg = None, digestmod = None): """Create a new hashing object and return it. key: The starting key for the hash. msg: if available, will immediately be hashed into the object's starting state. You can now feed arbitrary strings into the object using its update() method, and can ask for the hash value at any time by calling its digest() method. """ return HMAC(key, msg, digestmod) Zope-2.8.7-final/lib/python/ZEO/version.txt0000664000076400007640000000000610435502417017507 0ustar ajungajung3.4.3 Zope-2.8.7-final/lib/python/ZEO/__init__.py0000664000076400007640000000162610435502417017403 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ZEO -- Zope Enterprise Objects. See the file README.txt in this directory for an overview. ZEO is now part of ZODB; ZODB's home on the web is http://www.zope.org/Wikis/ZODB """ # The next line must use double quotes, so release.py recognizes it. version = "3.4.3" Zope-2.8.7-final/lib/python/ZEO/zeoctl.xml0000664000076400007640000000150510435502417017310 0ustar ajungajung This schema describes the configuration of the ZEO storage server controller. It differs from the schema for the storage server only in that the "runner" section is required.

Zope-2.8.7-final/lib/python/ZEO/README.txt0000664000076400007640000000401010435502417016756 0ustar ajungajung======= ZEO 2.0 ======= What's ZEO? ----------- ZEO stands for Zope Enterprise Objects. ZEO is an add-on for Zope that allows multiple processes to connect to a single ZODB storage. Those processes can live on different machines, but don't need to. ZEO 2 has many improvements over ZEO 1, and is incompatible with ZEO 1; if you upgrade an existing ZEO 1 installation, you must upgrade the server and all clients simultaneous. If you received ZEO 2 as part of the ZODB 3 distribution, the ZEO 1 sources are provided in a separate directory (ZEO1). Some documentation for ZEO is available in the ZODB 3 package in the Doc subdirectory. ZEO depends on the ZODB software; it can be used with the version of ZODB distributed with Zope 2.5.1 or later. More information about ZEO can be found in the ZODB Wiki: http://www.zope.org/Wikis/ZODB What's here? ------------ This list of filenames is mostly for ZEO developers:: ClientCache.py client-side cache implementation ClientStorage.py client-side storage implementation ClientStub.py RPC stubs for callbacks from server to client CommitLog.py buffer used during two-phase commit on the server Exceptions.py definitions of exceptions ICache.py interface definition for the client-side cache ServerStub.py RPC stubs for the server StorageServer.py server-side storage implementation TransactionBuffer.py buffer used for transaction data in the client __init__.py near-empty file to make this directory a package simul.py command-line tool to simulate cache behavior start.py command-line tool to start the storage server stats.py command-line tool to process client cache traces tests/ unit tests and other test utilities util.py utilities used by the server startup tool version.txt text file indicating the ZEO version zrpc/ subpackage implementing Remote Procedure Call (RPC) Zope-2.8.7-final/lib/python/ZEO/DebugServer.py0000664000076400007640000000565210435502417020064 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A debugging version of the server that records network activity.""" import struct import time from ZEO.StorageServer import StorageServer, log from ZEO.zrpc.server import ManagedServerConnection # a bunch of codes NEW_CONN = 1 CLOSE_CONN = 2 DATA = 3 ERROR = 4 class DebugManagedServerConnection(ManagedServerConnection): def __init__(self, sock, addr, obj, mgr): # mgr is the DebugServer instance self.mgr = mgr self.__super_init(sock, addr, obj) record_id = mgr._record_connection(addr) self._record = lambda code, data: mgr._record(record_id, code, data) self.obj.notifyConnected(self) def close(self): self._record(CLOSE_CONN, "") ManagedServerConnection.close(self) # override the lowest-level of asyncore's connection def recv(self, buffer_size): try: data = self.socket.recv(buffer_size) if not data: # a closed connection is indicated by signaling # a read condition, and having recv() return 0. self.handle_close() return '' else: self._record(DATA, data) return data except socket.error, why: # winsock sometimes throws ENOTCONN self._record(ERROR, why) if why[0] in [ECONNRESET, ENOTCONN, ESHUTDOWN]: self.handle_close() return '' else: raise socket.error, why class DebugServer(StorageServer): ZEOStorageClass = DebugZEOStorage ManagedServerConnectionClass = DebugManagerConnection def __init__(self, *args, **kwargs): StorageServer.__init__(*args, **kwargs) self._setup_record(kwargs["record"]) self._conn_counter = 1 def _setup_record(self, path): try: self._recordfile = open(path, "ab") except IOError, msg: self._recordfile = None log("failed to open recordfile %s: %s" % (path, msg)) def _record_connection(self, addr): cid = self._conn_counter self._conn_counter += 1 self._record(cid, NEW_CONN, str(addr)) return cid def _record(self, conn, code, data): s = struct.pack(">iii", code, time.time(), len(data)) + data self._recordfile.write(s) Zope-2.8.7-final/lib/python/ZEO/ClientStorage.py0000664000076400007640000012457510435502417020420 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """The ClientStorage class and the exceptions that it may raise. Public contents of this module: ClientStorage -- the main class, implementing the Storage API """ import cPickle import os import socket import tempfile import threading import time import types import logging from ZEO import ServerStub from ZEO.cache import ClientCache from ZEO.TransactionBuffer import TransactionBuffer from ZEO.Exceptions import ClientStorageError, ClientDisconnected, AuthError from ZEO.auth import get_module from ZEO.zrpc.client import ConnectionManager from ZODB import POSException from ZODB.loglevels import BLATHER from persistent.TimeStamp import TimeStamp logger = logging.getLogger('ZEO.ClientStorage') _pid = str(os.getpid()) def log2(msg, level=logging.INFO, subsys=_pid, exc_info=False): message = "(%s) %s" % (subsys, msg) logger.log(level, message, exc_info=exc_info) try: from ZODB.ConflictResolution import ResolvedSerial except ImportError: ResolvedSerial = 'rs' def tid2time(tid): return str(TimeStamp(tid)) def get_timestamp(prev_ts=None): """Internal helper to return a unique TimeStamp instance. If the optional argument is not None, it must be a TimeStamp; the return value is then guaranteed to be at least 1 microsecond later the argument. """ t = time.time() t = TimeStamp(*time.gmtime(t)[:5] + (t % 60,)) if prev_ts is not None: t = t.laterThan(prev_ts) return t class DisconnectedServerStub: """Internal helper class used as a faux RPC stub when disconnected. This raises ClientDisconnected on all attribute accesses. This is a singleton class -- there should be only one instance, the global disconnected_stub, os it can be tested by identity. """ def __getattr__(self, attr): raise ClientDisconnected() # Singleton instance of DisconnectedServerStub disconnected_stub = DisconnectedServerStub() MB = 1024**2 class ClientStorage(object): """A Storage class that is a network client to a remote storage. This is a faithful implementation of the Storage API. This class is thread-safe; transactions are serialized in tpc_begin(). """ # Classes we instantiate. A subclass might override. TransactionBufferClass = TransactionBuffer ClientCacheClass = ClientCache ConnectionManagerClass = ConnectionManager StorageServerStubClass = ServerStub.StorageServer def __init__(self, addr, storage='1', cache_size=20 * MB, name='', client=None, debug=0, var=None, min_disconnect_poll=5, max_disconnect_poll=300, wait_for_server_on_startup=None, # deprecated alias for wait wait=None, wait_timeout=None, read_only=0, read_only_fallback=0, username='', password='', realm=None): """ClientStorage constructor. This is typically invoked from a custom_zodb.py file. All arguments except addr should be keyword arguments. Arguments: addr -- The server address(es). This is either a list of addresses or a single address. Each address can be a (hostname, port) tuple to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection. A hostname may be a DNS name or a dotted IP address. Required. storage -- The storage name, defaulting to '1'. The name must match one of the storage names supported by the server(s) specified by the addr argument. The storage name is displayed in the Zope control panel. cache_size -- The disk cache size, defaulting to 20 megabytes. This is passed to the ClientCache constructor. name -- The storage name, defaulting to ''. If this is false, str(addr) is used as the storage name. client -- A name used to construct persistent cache filenames. Defaults to None, in which case the cache is not persistent. See ClientCache for more info. debug -- Ignored. This is present only for backwards compatibility with ZEO 1. var -- When client is not None, this specifies the directory where the persistent cache files are created. It defaults to None, in whichcase the current directory is used. min_disconnect_poll -- The minimum delay in seconds between attempts to connect to the server, in seconds. Defaults to 5 seconds. max_disconnect_poll -- The maximum delay in seconds between attempts to connect to the server, in seconds. Defaults to 300 seconds. wait_for_server_on_startup -- A backwards compatible alias for the wait argument. wait -- A flag indicating whether to wait until a connection with a server is made, defaulting to true. wait_timeout -- Maximum time to wait for a connection before giving up. Only meaningful if wait is True. read_only -- A flag indicating whether this should be a read-only storage, defaulting to false (i.e. writing is allowed by default). read_only_fallback -- A flag indicating whether a read-only remote storage should be acceptable as a fallback when no writable storages are available. Defaults to false. At most one of read_only and read_only_fallback should be true. username -- string with username to be used when authenticating. These only need to be provided if you are connecting to an authenticated server storage. password -- string with plaintext password to be used when authenticated. Note that the authentication protocol is defined by the server and is detected by the ClientStorage upon connecting (see testConnection() and doAuth() for details). """ log2("%s (pid=%d) created %s/%s for storage: %r" % (self.__class__.__name__, os.getpid(), read_only and "RO" or "RW", read_only_fallback and "fallback" or "normal", storage)) if debug: log2("ClientStorage(): debug argument is no longer used") # wait defaults to True, but wait_for_server_on_startup overrides # if not None if wait_for_server_on_startup is not None: if wait is not None and wait != wait_for_server_on_startup: log2("ClientStorage(): conflicting values for wait and " "wait_for_server_on_startup; wait prevails", level=logging.WARNING) else: log2("ClientStorage(): wait_for_server_on_startup " "is deprecated; please use wait instead") wait = wait_for_server_on_startup elif wait is None: wait = 1 self._addr = addr # For tests # A ZEO client can run in disconnected mode, using data from # its cache, or in connected mode. Several instance variables # are related to whether the client is connected. # _server: All method calls are invoked through the server # stub. When not connect, set to disconnected_stub an # object that raises ClientDisconnected errors. # _ready: A threading Event that is set only if _server # is set to a real stub. # _connection: The current zrpc connection or None. # _connection is set as soon as a connection is established, # but _server is set only after cache verification has finished # and clients can safely use the server. _pending_server holds # a server stub while it is being verified. self._server = disconnected_stub self._connection = None self._pending_server = None self._ready = threading.Event() # _is_read_only stores the constructor argument self._is_read_only = read_only # _conn_is_read_only stores the status of the current connection self._conn_is_read_only = 0 self._storage = storage self._read_only_fallback = read_only_fallback self._username = username self._password = password self._realm = realm # Flag tracking disconnections in the middle of a transaction. This # is reset in tpc_begin() and set in notifyDisconnected(). self._midtxn_disconnect = 0 # _server_addr is used by sortKey() self._server_addr = None self._tfile = None self._pickler = None self._info = {'length': 0, 'size': 0, 'name': 'ZEO Client', 'supportsUndo':0, 'supportsVersions': 0, 'supportsTransactionalUndo': 0} self._tbuf = self.TransactionBufferClass() self._db = None self._ltid = None # the last committed transaction # _serials: stores (oid, serialno) as returned by server # _seriald: _check_serials() moves from _serials to _seriald, # which maps oid to serialno # TODO: If serial number matches transaction id, then there is # no need to have all this extra infrastructure for handling # serial numbers. The vote call can just return the tid. # If there is a conflict error, we can't have a special method # called just to propagate the error. self._serials = [] self._seriald = {} self.__name__ = name or str(addr) # Standard convention for storages # A ClientStorage only allows one thread to commit at a time. # Mutual exclusion is achieved using _tpc_cond, which # protects _transaction. A thread that wants to assign to # self._transaction must acquire _tpc_cond first. A thread # that decides it's done with a transaction (whether via success # or failure) must set _transaction to None and do # _tpc_cond.notify() before releasing _tpc_cond. self._tpc_cond = threading.Condition() self._transaction = None # Prevent multiple new_oid calls from going out. The _oids # variable should only be modified while holding the # _oid_lock. self._oid_lock = threading.Lock() self._oids = [] # Object ids retrieved from new_oids() # load() and tpc_finish() must be serialized to guarantee # that cache modifications from each occur atomically. # It also prevents multiple load calls occuring simultaneously, # which simplifies the cache logic. self._load_lock = threading.Lock() # _load_oid and _load_status are protected by _lock self._load_oid = None self._load_status = None # Can't read data in one thread while writing data # (tpc_finish) in another thread. In general, the lock # must prevent access to the cache while _update_cache # is executing. self._lock = threading.Lock() # Decide whether to use non-temporary files if client is not None: dir = var or os.getcwd() cache_path = os.path.join(dir, "%s-%s.zec" % (client, storage)) else: cache_path = None self._cache = self.ClientCacheClass(cache_path, size=cache_size) # TODO: maybe there's a better time to open the cache? Unclear. self._cache.open() self._rpc_mgr = self.ConnectionManagerClass(addr, self, tmin=min_disconnect_poll, tmax=max_disconnect_poll) if wait: self._wait(wait_timeout) else: # attempt_connect() will make an attempt that doesn't block # "too long," for a very vague notion of too long. If that # doesn't succeed, call connect() to start a thread. if not self._rpc_mgr.attempt_connect(): self._rpc_mgr.connect() def _wait(self, timeout=None): if timeout is not None: deadline = time.time() + timeout log2("Setting deadline to %f" % deadline, level=BLATHER) else: deadline = None # Wait for a connection to be established. self._rpc_mgr.connect(sync=1) # When a synchronous connect() call returns, there is # a valid _connection object but cache validation may # still be going on. This code must wait until validation # finishes, but if the connection isn't a zrpc async # connection it also needs to poll for input. if self._connection.is_async(): while 1: self._ready.wait(30) if self._ready.isSet(): break if timeout and time.time() > deadline: log2("Timed out waiting for connection", level=logging.WARNING) break log2("Waiting for cache verification to finish") else: self._wait_sync(deadline) def _wait_sync(self, deadline=None): # Log no more than one "waiting" message per LOG_THROTTLE seconds. LOG_THROTTLE = 300 # 5 minutes next_log_time = time.time() while not self._ready.isSet(): now = time.time() if deadline and now > deadline: log2("Timed out waiting for connection", level=logging.WARNING) break if now >= next_log_time: log2("Waiting for cache verification to finish") next_log_time = now + LOG_THROTTLE if self._connection is None: # If the connection was closed while we were # waiting for it to become ready, start over. if deadline is None: timeout = None else: timeout = deadline - now return self._wait(timeout) # No mainloop ia running, so we need to call something fancy to # handle asyncore events. self._connection.pending(30) def close(self): """Storage API: finalize the storage, releasing external resources.""" self._tbuf.close() if self._cache is not None: self._cache.close() self._cache = None if self._rpc_mgr is not None: self._rpc_mgr.close() self._rpc_mgr = None def registerDB(self, db, limit): """Storage API: register a database for invalidation messages. This is called by ZODB.DB (and by some tests). The storage isn't really ready to use until after this call. """ self._db = db def is_connected(self): """Return whether the storage is currently connected to a server.""" # This function is used by clients, so we only report that a # connection exists when the connection is ready to use. return self._ready.isSet() def sync(self): """Handle any pending invalidation messages. This is called by the sync method in ZODB.Connection. """ # If there is no connection, return immediately. Technically, # there are no pending invalidations so they are all handled. # There doesn't seem to be much benefit to raising an exception. cn = self._connection if cn is not None: cn.pending() def doAuth(self, protocol, stub): if not (self._username and self._password): raise AuthError, "empty username or password" module = get_module(protocol) if not module: log2("%s: no such an auth protocol: %s" % (self.__class__.__name__, protocol), level=logging.WARNING) return storage_class, client, db_class = module if not client: log2("%s: %s isn't a valid protocol, must have a Client class" % (self.__class__.__name__, protocol), level=logging.WARNING) raise AuthError, "invalid protocol" c = client(stub) # Initiate authentication, returns boolean specifying whether OK return c.start(self._username, self._realm, self._password) def testConnection(self, conn): """Internal: test the given connection. This returns: 1 if the connection is an optimal match, 0 if it is a suboptimal but acceptable match. It can also raise DisconnectedError or ReadOnlyError. This is called by ZEO.zrpc.ConnectionManager to decide which connection to use in case there are multiple, and some are read-only and others are read-write. This works by calling register() on the server. In read-only mode, register() is called with the read_only flag set. In writable mode and in read-only fallback mode, register() is called with the read_only flag cleared. In read-only fallback mode only, if the register() call raises ReadOnlyError, it is retried with the read-only flag set, and if this succeeds, this is deemed a suboptimal match. In all other cases, a succeeding register() call is deemed an optimal match, and any exception raised by register() is passed through. """ log2("Testing connection %r" % conn) # TODO: Should we check the protocol version here? self._conn_is_read_only = 0 stub = self.StorageServerStubClass(conn) auth = stub.getAuthProtocol() log2("Server authentication protocol %r" % auth) if auth: skey = self.doAuth(auth, stub) if skey: log2("Client authentication successful") conn.setSessionKey(skey) else: log2("Authentication failed") raise AuthError, "Authentication failed" try: stub.register(str(self._storage), self._is_read_only) return 1 except POSException.ReadOnlyError: if not self._read_only_fallback: raise log2("Got ReadOnlyError; trying again with read_only=1") stub.register(str(self._storage), read_only=1) self._conn_is_read_only = 1 return 0 def notifyConnected(self, conn): """Internal: start using the given connection. This is called by ConnectionManager after it has decided which connection should be used. """ if self._cache is None: # the storage was closed, but the connect thread called # this method before it was stopped. return # TODO: report whether we get a read-only connection. if self._connection is not None: reconnect = 1 else: reconnect = 0 self.set_server_addr(conn.get_addr()) # If we are upgrading from a read-only fallback connection, # we must close the old connection to prevent it from being # used while the cache is verified against the new connection. if self._connection is not None: self._connection.close() self._connection = conn if reconnect: log2("Reconnected to storage: %s" % self._server_addr) else: log2("Connected to storage: %s" % self._server_addr) stub = self.StorageServerStubClass(conn) self._oids = [] self._info.update(stub.get_info()) self.verify_cache(stub) if not conn.is_async(): log2("Waiting for cache verification to finish") self._wait_sync() self._handle_extensions() def _handle_extensions(self): for name in self.getExtensionMethods().keys(): if not hasattr(self, name): setattr(self, name, self._server.extensionMethod(name)) def set_server_addr(self, addr): # Normalize server address and convert to string if isinstance(addr, types.StringType): self._server_addr = addr else: assert isinstance(addr, types.TupleType) # If the server is on a remote host, we need to guarantee # that all clients used the same name for the server. If # they don't, the sortKey() may be different for each client. # The best solution seems to be the official name reported # by gethostbyaddr(). host = addr[0] try: canonical, aliases, addrs = socket.gethostbyaddr(host) except socket.error, err: log2("Error resolving host: %s (%s)" % (host, err), level=BLATHER) canonical = host self._server_addr = str((canonical, addr[1])) def sortKey(self): # If the client isn't connected to anything, it can't have a # valid sortKey(). Raise an error to stop the transaction early. if self._server_addr is None: raise ClientDisconnected else: return '%s:%s' % (self._storage, self._server_addr) def verify_cache(self, server): """Internal routine called to verify the cache. The return value (indicating which path we took) is used by the test suite. """ # If verify_cache() finishes the cache verification process, # it should set self._server. If it goes through full cache # verification, then endVerify() should self._server. last_inval_tid = self._cache.getLastTid() if last_inval_tid is not None: ltid = server.lastTransaction() if ltid == last_inval_tid: log2("No verification necessary (last_inval_tid up-to-date)") self._server = server self._ready.set() return "no verification" # log some hints about last transaction log2("last inval tid: %r %s\n" % (last_inval_tid, tid2time(last_inval_tid))) log2("last transaction: %r %s" % (ltid, ltid and tid2time(ltid))) pair = server.getInvalidations(last_inval_tid) if pair is not None: log2("Recovering %d invalidations" % len(pair[1])) self.invalidateTransaction(*pair) self._server = server self._ready.set() return "quick verification" log2("Verifying cache") # setup tempfile to hold zeoVerify results self._tfile = tempfile.TemporaryFile(suffix=".inv") self._pickler = cPickle.Pickler(self._tfile, 1) self._pickler.fast = 1 # Don't use the memo # TODO: should batch these operations for efficiency; would need # to acquire lock ... for oid, tid, version in self._cache.contents(): server.verify(oid, version, tid) self._pending_server = server server.endZeoVerify() return "full verification" ### Is there a race condition between notifyConnected and ### notifyDisconnected? In Particular, what if we get ### notifyDisconnected in the middle of notifyConnected? ### The danger is that we'll proceed as if we were connected ### without worrying if we were, but this would happen any way if ### notifyDisconnected had to get the instance lock. There's ### nothing to gain by getting the instance lock. def notifyDisconnected(self): """Internal: notify that the server connection was terminated. This is called by ConnectionManager when the connection is closed or when certain problems with the connection occur. """ log2("Disconnected from storage: %s" % repr(self._server_addr)) self._connection = None self._ready.clear() self._server = disconnected_stub self._midtxn_disconnect = 1 def __len__(self): """Return the size of the storage.""" # TODO: Is this method used? return self._info['length'] def getName(self): """Storage API: return the storage name as a string. The return value consists of two parts: the name as determined by the name and addr argments to the ClientStorage constructor, and the string 'connected' or 'disconnected' in parentheses indicating whether the storage is (currently) connected. """ return "%s (%s)" % ( self.__name__, self.is_connected() and "connected" or "disconnected") def getSize(self): """Storage API: an approximate size of the database, in bytes.""" return self._info['size'] def getExtensionMethods(self): """getExtensionMethods This returns a dictionary whose keys are names of extra methods provided by this storage. Storage proxies (such as ZEO) should call this method to determine the extra methods that they need to proxy in addition to the standard storage methods. Dictionary values should be None; this will be a handy place for extra marshalling information, should we need it """ return self._info.get('extensionMethods', {}) def supportsUndo(self): """Storage API: return whether we support undo.""" return self._info['supportsUndo'] def supportsVersions(self): """Storage API: return whether we support versions.""" return self._info['supportsVersions'] def supportsTransactionalUndo(self): """Storage API: return whether we support transactional undo.""" return self._info['supportsTransactionalUndo'] def isReadOnly(self): """Storage API: return whether we are in read-only mode.""" if self._is_read_only: return 1 else: # If the client is configured for a read-write connection # but has a read-only fallback connection, _conn_is_read_only # will be True. return self._conn_is_read_only def _check_trans(self, trans): """Internal helper to check a transaction argument for sanity.""" if self._is_read_only: raise POSException.ReadOnlyError() if self._transaction is not trans: raise POSException.StorageTransactionError(self._transaction, trans) def abortVersion(self, version, txn): """Storage API: clear any changes made by the given version.""" self._check_trans(txn) tid, oids = self._server.abortVersion(version, id(txn)) # When a version aborts, invalidate the version and # non-version data. The non-version data should still be # valid, but older versions of ZODB will change the # non-version serialno on an abort version. With those # versions of ZODB, you'd get a conflict error if you tried to # commit a transaction with the cached data. # If we could guarantee that ZODB gave the right answer, # we could just invalidate the version data. for oid in oids: self._tbuf.invalidate(oid, '') return tid, oids def commitVersion(self, source, destination, txn): """Storage API: commit the source version in the destination.""" self._check_trans(txn) tid, oids = self._server.commitVersion(source, destination, id(txn)) if destination: # just invalidate our version data for oid in oids: self._tbuf.invalidate(oid, source) else: # destination is "", so invalidate version and non-version for oid in oids: self._tbuf.invalidate(oid, "") return tid, oids def history(self, oid, version, length=1): """Storage API: return a sequence of HistoryEntry objects. This does not support the optional filter argument defined by the Storage API. """ return self._server.history(oid, version, length) def getSerial(self, oid): """Storage API: return current serial number for oid.""" return self._server.getSerial(oid) def loadSerial(self, oid, serial): """Storage API: load a historical revision of an object.""" return self._server.loadSerial(oid, serial) def load(self, oid, version): """Storage API: return the data for a given object. This returns the pickle data and serial number for the object specified by the given object id and version, if they exist; otherwise a KeyError is raised. """ return self.loadEx(oid, version)[:2] def loadEx(self, oid, version): self._lock.acquire() # for atomic processing of invalidations try: t = self._cache.load(oid, version) if t: return t finally: self._lock.release() if self._server is None: raise ClientDisconnected() self._load_lock.acquire() try: self._lock.acquire() try: self._load_oid = oid self._load_status = 1 finally: self._lock.release() data, tid, ver = self._server.loadEx(oid, version) self._lock.acquire() # for atomic processing of invalidations try: if self._load_status: self._cache.store(oid, ver, tid, None, data) self._load_oid = None finally: self._lock.release() finally: self._load_lock.release() return data, tid, ver def loadBefore(self, oid, tid): self._lock.acquire() try: t = self._cache.loadBefore(oid, tid) if t is not None: return t finally: self._lock.release() t = self._server.loadBefore(oid, tid) if t is None: return None data, start, end = t if end is None: # This method should not be used to get current data. It # doesn't use the _load_lock, so it is possble to overlap # this load with an invalidation for the same object. # If we call again, we're guaranteed to get the # post-invalidation data. But if the data is still # current, we'll still get end == None. # Maybe the best thing to do is to re-run the test with # the load lock in the case. That's slow performance, but # I don't think real application code will ever care about # it. return data, start, end self._lock.acquire() try: self._cache.store(oid, "", start, end, data) finally: self._lock.release() return data, start, end def modifiedInVersion(self, oid): """Storage API: return the version, if any, that modfied an object. If no version modified the object, return an empty string. """ self._lock.acquire() try: v = self._cache.modifiedInVersion(oid) if v is not None: return v finally: self._lock.release() return self._server.modifiedInVersion(oid) def new_oid(self): """Storage API: return a new object identifier.""" if self._is_read_only: raise POSException.ReadOnlyError() # avoid multiple oid requests to server at the same time self._oid_lock.acquire() try: if not self._oids: self._oids = self._server.new_oids() self._oids.reverse() return self._oids.pop() finally: self._oid_lock.release() def pack(self, t=None, referencesf=None, wait=1, days=0): """Storage API: pack the storage. Deviations from the Storage API: the referencesf argument is ignored; two additional optional arguments wait and days are provided: wait -- a flag indicating whether to wait for the pack to complete; defaults to true. days -- a number of days to subtract from the pack time; defaults to zero. """ # TODO: Is it okay that read-only connections allow pack()? # rf argument ignored; server will provide its own implementation if t is None: t = time.time() t = t - (days * 86400) return self._server.pack(t, wait) def _check_serials(self): """Internal helper to move data from _serials to _seriald.""" # serials are always going to be the same, the only # question is whether an exception has been raised. if self._serials: l = len(self._serials) r = self._serials[:l] del self._serials[:l] for oid, s in r: if isinstance(s, Exception): raise s self._seriald[oid] = s return r def store(self, oid, serial, data, version, txn): """Storage API: store data for an object.""" self._check_trans(txn) self._server.storea(oid, serial, data, version, id(txn)) self._tbuf.store(oid, version, data) return self._check_serials() def tpc_vote(self, txn): """Storage API: vote on a transaction.""" if txn is not self._transaction: return self._server.vote(id(txn)) return self._check_serials() def tpc_begin(self, txn, tid=None, status=' '): """Storage API: begin a transaction.""" if self._is_read_only: raise POSException.ReadOnlyError() self._tpc_cond.acquire() self._midtxn_disconnect = 0 while self._transaction is not None: # It is allowable for a client to call two tpc_begins in a # row with the same transaction, and the second of these # must be ignored. if self._transaction == txn: self._tpc_cond.release() return self._tpc_cond.wait(30) self._transaction = txn self._tpc_cond.release() try: self._server.tpc_begin(id(txn), txn.user, txn.description, txn._extension, tid, status) except: # Client may have disconnected during the tpc_begin(). if self._server is not disconnected_stub: self.end_transaction() raise self._tbuf.clear() self._seriald.clear() del self._serials[:] def end_transaction(self): """Internal helper to end a transaction.""" # the right way to set self._transaction to None # calls notify() on _tpc_cond in case there are waiting threads self._tpc_cond.acquire() self._transaction = None self._tpc_cond.notify() self._tpc_cond.release() def lastTransaction(self): return self._cache.getLastTid() def tpc_abort(self, txn): """Storage API: abort a transaction.""" if txn is not self._transaction: return try: # Caution: Are there any exceptions that should prevent an # abort from occurring? It seems wrong to swallow them # all, yet you want to be sure that other abort logic is # executed regardless. try: self._server.tpc_abort(id(txn)) except ClientDisconnected: log2("ClientDisconnected in tpc_abort() ignored", level=BLATHER) finally: self._tbuf.clear() self._seriald.clear() del self._serials[:] self.end_transaction() def tpc_finish(self, txn, f=None): """Storage API: finish a transaction.""" if txn is not self._transaction: return self._load_lock.acquire() try: if self._midtxn_disconnect: raise ClientDisconnected( 'Calling tpc_finish() on a disconnected transaction') # The calls to tpc_finish() and _update_cache() should # never run currently with another thread, because the # tpc_cond condition variable prevents more than one # thread from calling tpc_finish() at a time. tid = self._server.tpc_finish(id(txn)) self._lock.acquire() # for atomic processing of invalidations try: self._update_cache(tid) if f is not None: f(tid) finally: self._lock.release() r = self._check_serials() assert r is None or len(r) == 0, "unhandled serialnos: %s" % r finally: self._load_lock.release() self.end_transaction() def _update_cache(self, tid): """Internal helper to handle objects modified by a transaction. This iterates over the objects in the transaction buffer and update or invalidate the cache. """ # Must be called with _lock already acquired. # Not sure why _update_cache() would be called on a closed storage. if self._cache is None: return for oid, version, data in self._tbuf: self._cache.invalidate(oid, version, tid) # If data is None, we just invalidate. if data is not None: s = self._seriald[oid] if s != ResolvedSerial: assert s == tid, (s, tid) self._cache.store(oid, version, s, None, data) self._tbuf.clear() def undo(self, trans_id, txn): """Storage API: undo a transaction. This is executed in a transactional context. It has no effect until the transaction is committed. It can be undone itself. Zope uses this to implement undo unless it is not supported by a storage. """ self._check_trans(txn) tid, oids = self._server.undo(trans_id, id(txn)) for oid in oids: self._tbuf.invalidate(oid, '') return tid, oids def undoInfo(self, first=0, last=-20, specification=None): """Storage API: return undo information.""" return self._server.undoInfo(first, last, specification) def undoLog(self, first=0, last=-20, filter=None): """Storage API: return a sequence of TransactionDescription objects. The filter argument should be None or left unspecified, since it is impossible to pass the filter function to the server to be executed there. If filter is not None, an empty sequence is returned. """ if filter is not None: return [] return self._server.undoLog(first, last) def versionEmpty(self, version): """Storage API: return whether the version has no transactions.""" return self._server.versionEmpty(version) def versions(self, max=None): """Storage API: return a sequence of versions in the storage.""" return self._server.versions(max) # Below are methods invoked by the StorageServer def serialnos(self, args): """Server callback to pass a list of changed (oid, serial) pairs.""" self._serials.extend(args) def info(self, dict): """Server callback to update the info dictionary.""" self._info.update(dict) def invalidateVerify(self, args): """Server callback to invalidate an (oid, version) pair. This is called as part of cache validation. """ # Invalidation as result of verify_cache(). # Queue an invalidate for the end the verification procedure. if self._pickler is None: # This should never happen. TODO: assert it doesn't, or log # if it does. return self._pickler.dump(args) def _process_invalidations(self, invs): # Invalidations are sent by the ZEO server as a sequence of # oid, version pairs. The DB's invalidate() method expects a # dictionary of oids. self._lock.acquire() try: # versions maps version names to dictionary of invalidations versions = {} for oid, version, tid in invs: if oid == self._load_oid: self._load_status = 0 self._cache.invalidate(oid, version, tid) versions.setdefault((version, tid), {})[oid] = tid if self._db is not None: for (version, tid), d in versions.items(): self._db.invalidate(tid, d, version=version) finally: self._lock.release() def endVerify(self): """Server callback to signal end of cache validation.""" if self._pickler is None: return # write end-of-data marker self._pickler.dump((None, None)) self._pickler = None self._tfile.seek(0) f = self._tfile self._tfile = None self._process_invalidations(InvalidationLogIterator(f)) f.close() log2("endVerify finishing") self._server = self._pending_server self._ready.set() self._pending_conn = None log2("endVerify finished") def invalidateTransaction(self, tid, args): """Invalidate objects modified by tid.""" self._lock.acquire() try: self._cache.setLastTid(tid) finally: self._lock.release() if self._pickler is not None: log2("Transactional invalidation during cache verification", level=BLATHER) for t in args: self._pickler.dump(t) return self._process_invalidations([(oid, version, tid) for oid, version in args]) # The following are for compatibility with protocol version 2.0.0 def invalidateTrans(self, args): return self.invalidateTransaction(None, args) invalidate = invalidateVerify end = endVerify Invalidate = invalidateTrans def InvalidationLogIterator(fileobj): unpickler = cPickle.Unpickler(fileobj) while 1: oid, version = unpickler.load() if oid is None: break yield oid, version, None Zope-2.8.7-final/lib/python/ZEO/monitor.py0000664000076400007640000001160110435502417017325 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Monitor behavior of ZEO server and record statistics. $Id: monitor.py 29448 2005-03-11 23:02:55Z tim_one $ """ import asyncore import socket import time import types import logging import ZEO class StorageStats: """Per-storage usage statistics.""" def __init__(self): self.loads = 0 self.stores = 0 self.commits = 0 self.aborts = 0 self.active_txns = 0 self.clients = 0 self.verifying_clients = 0 self.lock_time = None self.conflicts = 0 self.conflicts_resolved = 0 self.start = time.ctime() def parse(self, s): # parse the dump format lines = s.split("\n") for line in lines: field, value = line.split(":", 1) if field == "Server started": self.start = value elif field == "Clients": self.clients = int(value) elif field == "Clients verifying": self.verifying_clients = int(value) elif field == "Active transactions": self.active_txns = int(value) elif field == "Commit lock held for": # This assumes self.lock_time = time.time() - int(value) elif field == "Commits": self.commits = int(value) elif field == "Aborts": self.aborts = int(value) elif field == "Loads": self.loads = int(value) elif field == "Stores": self.stores = int(value) elif field == "Conflicts": self.conflicts = int(value) elif field == "Conflicts resolved": self.conflicts_resolved = int(value) def dump(self, f): print >> f, "Server started:", self.start print >> f, "Clients:", self.clients print >> f, "Clients verifying:", self.verifying_clients print >> f, "Active transactions:", self.active_txns if self.lock_time: howlong = time.time() - self.lock_time print >> f, "Commit lock held for:", int(howlong) print >> f, "Commits:", self.commits print >> f, "Aborts:", self.aborts print >> f, "Loads:", self.loads print >> f, "Stores:", self.stores print >> f, "Conflicts:", self.conflicts print >> f, "Conflicts resolved:", self.conflicts_resolved class StatsClient(asyncore.dispatcher): def __init__(self, sock, addr): asyncore.dispatcher.__init__(self, sock) self.buf = [] self.closed = 0 def close(self): self.closed = 1 # The socket is closed after all the data is written. # See handle_write(). def write(self, s): self.buf.append(s) def writable(self): return len(self.buf) def readable(self): return 0 def handle_write(self): s = "".join(self.buf) self.buf = [] n = self.socket.send(s) if n < len(s): self.buf.append(s[:n]) if self.closed and not self.buf: asyncore.dispatcher.close(self) class StatsServer(asyncore.dispatcher): StatsConnectionClass = StatsClient def __init__(self, addr, stats): asyncore.dispatcher.__init__(self) self.addr = addr self.stats = stats if type(self.addr) == types.TupleType: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) else: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) self.set_reuse_addr() logger = logging.getLogger('ZEO.monitor') logger.info("listening on %s", repr(self.addr)) self.bind(self.addr) self.listen(5) def writable(self): return 0 def readable(self): return 1 def handle_accept(self): try: sock, addr = self.accept() except socket.error: return f = self.StatsConnectionClass(sock, addr) self.dump(f) f.close() def dump(self, f): print >> f, "ZEO monitor server version %s" % ZEO.version print >> f, time.ctime() print >> f L = self.stats.keys() L.sort() for k in L: stats = self.stats[k] print >> f, "Storage:", k stats.dump(f) print >> f Zope-2.8.7-final/lib/python/ZEO/stats.py0000775000076400007640000003036510435502417017007 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Trace file statistics analyzer. Usage: stats.py [-h] [-i interval] [-q] [-s] [-S] [-v] [-X] tracefile -h: print histogram of object load frequencies -i: summarizing interval in minutes (default 15; max 60) -q: quiet; don't print summaries -s: print histogram of object sizes -S: don't print statistics -v: verbose; print each record -X: enable heuristic checking for misaligned records: oids > 2**32 will be rejected; this requires the tracefile to be seekable """ """File format: Each record is 26 bytes, plus a variable number of bytes to store an oid, with the following layout. Numbers are big-endian integers. Offset Size Contents 0 4 timestamp (seconds since 1/1/1970) 4 3 data size, in 256-byte increments, rounded up 7 1 code (see below) 8 2 object id length 10 8 start tid 18 8 end tid 26 variable object id The code at offset 7 packs three fields: Mask bits Contents 0x80 1 set if there was a non-empty version string 0x7e 6 function and outcome code 0x01 1 current cache file (0 or 1) The "current cache file" bit is no longer used; it refers to a 2-file cache scheme used before ZODB 3.3. The function and outcome codes are documented in detail at the end of this file in the 'explain' dictionary. Note that the keys there (and also the arguments to _trace() in ClientStorage.py) are 'code & 0x7e', i.e. the low bit is always zero. """ import sys import time import getopt import struct from types import StringType def usage(msg): print >> sys.stderr, msg print >> sys.stderr, __doc__ def main(): # Parse options verbose = False quiet = False dostats = True print_size_histogram = False print_histogram = False interval = 15*60 # Every 15 minutes heuristic = False try: opts, args = getopt.getopt(sys.argv[1:], "hi:qsSvX") except getopt.error, msg: usage(msg) return 2 for o, a in opts: if o == '-h': print_histogram = True elif o == "-i": interval = int(60 * float(a)) if interval <= 0: interval = 60 elif interval > 3600: interval = 3600 elif o == "-q": quiet = True verbose = False elif o == "-s": print_size_histogram = True elif o == "-S": dostats = False elif o == "-v": verbose = True elif o == '-X': heuristic = True else: assert False, (o, opt) if len(args) != 1: usage("exactly one file argument required") return 2 filename = args[0] # Open file if filename.endswith(".gz"): # Open gzipped file try: import gzip except ImportError: print >> sys.stderr, "can't read gzipped files (no module gzip)" return 1 try: f = gzip.open(filename, "rb") except IOError, msg: print >> sys.stderr, "can't open %s: %s" % (filename, msg) return 1 elif filename == '-': # Read from stdin f = sys.stdin else: # Open regular file try: f = open(filename, "rb") except IOError, msg: print >> sys.stderr, "can't open %s: %s" % (filename, msg) return 1 rt0 = time.time() bycode = {} # map code to count of occurrences byinterval = {} # map code to count in current interval records = 0 # number of trace records read versions = 0 # number of trace records with versions datarecords = 0 # number of records with dlen set datasize = 0L # sum of dlen across records with dlen set oids = {} # map oid to number of times it was loaded bysize = {} # map data size to number of loads bysizew = {} # map data size to number of writes total_loads = 0 t0 = None # first timestamp seen te = None # most recent timestamp seen h0 = None # timestamp at start of current interval he = None # timestamp at end of current interval thisinterval = None # generally te//interval f_read = f.read unpack = struct.unpack FMT = ">iiH8s8s" FMT_SIZE = struct.calcsize(FMT) assert FMT_SIZE == 26 # Read file, gathering statistics, and printing each record if verbose. try: while 1: r = f_read(FMT_SIZE) if len(r) < FMT_SIZE: break ts, code, oidlen, start_tid, end_tid = unpack(FMT, r) if ts == 0: # Must be a misaligned record caused by a crash. if not quiet: print "Skipping 8 bytes at offset", f.tell() - FMT_SIZE f.seek(f.tell() - FMT_SIZE + 8) continue oid = f_read(oidlen) if len(oid) < oidlen: break records += 1 if t0 is None: t0 = ts thisinterval = t0 // interval h0 = he = ts te = ts if ts // interval != thisinterval: if not quiet: dumpbyinterval(byinterval, h0, he) byinterval = {} thisinterval = ts // interval h0 = ts he = ts dlen, code = code & 0x7fffff00, code & 0xff if dlen: datarecords += 1 datasize += dlen if code & 0x80: version = 'V' versions += 1 else: version = '-' code &= 0x7e bycode[code] = bycode.get(code, 0) + 1 byinterval[code] = byinterval.get(code, 0) + 1 if dlen: if code & 0x70 == 0x20: # All loads bysize[dlen] = d = bysize.get(dlen) or {} d[oid] = d.get(oid, 0) + 1 elif code & 0x70 == 0x50: # All stores bysizew[dlen] = d = bysizew.get(dlen) or {} d[oid] = d.get(oid, 0) + 1 if verbose: print "%s %02x %s %016x %016x %c %s" % ( time.ctime(ts)[4:-5], code, oid_repr(oid), U64(start_tid), U64(end_tid), version, dlen and str(dlen) or "") if code & 0x70 == 0x20: oids[oid] = oids.get(oid, 0) + 1 total_loads += 1 elif code == 0x00: # restart if not quiet: dumpbyinterval(byinterval, h0, he) byinterval = {} thisinterval = ts // interval h0 = he = ts if not quiet: print time.ctime(ts)[4:-5], print '='*20, "Restart", '='*20 except KeyboardInterrupt: print "\nInterrupted. Stats so far:\n" end_pos = f.tell() f.close() rte = time.time() if not quiet: dumpbyinterval(byinterval, h0, he) # Error if nothing was read if not records: print >> sys.stderr, "No records processed" return 1 # Print statistics if dostats: print print "Read %s trace records (%s bytes) in %.1f seconds" % ( addcommas(records), addcommas(end_pos), rte-rt0) print "Versions: %s records used a version" % addcommas(versions) print "First time: %s" % time.ctime(t0) print "Last time: %s" % time.ctime(te) print "Duration: %s seconds" % addcommas(te-t0) print "Data recs: %s (%.1f%%), average size %.1f KB" % ( addcommas(datarecords), 100.0 * datarecords / records, datasize / 1024.0 / datarecords) print "Hit rate: %.1f%% (load hits / loads)" % hitrate(bycode) print codes = bycode.keys() codes.sort() print "%13s %4s %s" % ("Count", "Code", "Function (action)") for code in codes: print "%13s %02x %s" % ( addcommas(bycode.get(code, 0)), code, explain.get(code) or "*** unknown code ***") # Print histogram. if print_histogram: print print "Histogram of object load frequency" total = len(oids) print "Unique oids: %s" % addcommas(total) print "Total loads: %s" % addcommas(total_loads) s = addcommas(total) width = max(len(s), len("objects")) fmt = "%5d %" + str(width) + "s %5.1f%% %5.1f%% %5.1f%%" hdr = "%5s %" + str(width) + "s %6s %6s %6s" print hdr % ("loads", "objects", "%obj", "%load", "%cum") cum = 0.0 for binsize, count in histogram(oids): obj_percent = 100.0 * count / total load_percent = 100.0 * count * binsize / total_loads cum += load_percent print fmt % (binsize, addcommas(count), obj_percent, load_percent, cum) # Print size histogram. if print_size_histogram: print print "Histograms of object sizes" print dumpbysize(bysizew, "written", "writes") dumpbysize(bysize, "loaded", "loads") def dumpbysize(bysize, how, how2): print print "Unique sizes %s: %s" % (how, addcommas(len(bysize))) print "%10s %6s %6s" % ("size", "objs", how2) sizes = bysize.keys() sizes.sort() for size in sizes: loads = 0 for n in bysize[size].itervalues(): loads += n print "%10s %6d %6d" % (addcommas(size), len(bysize.get(size, "")), loads) def dumpbyinterval(byinterval, h0, he): loads = hits = 0 for code in byinterval: if code & 0x70 == 0x20: n = byinterval[code] loads += n if code in (0x22, 0x26): hits += n if not loads: return if loads: hr = 100.0 * hits / loads else: hr = 0.0 print "%s-%s %10s loads, %10s hits,%5.1f%% hit rate" % ( time.ctime(h0)[4:-8], time.ctime(he)[14:-8], addcommas(loads), addcommas(hits), hr) def hitrate(bycode): loads = hits = 0 for code in bycode: if code & 0x70 == 0x20: n = bycode[code] loads += n if code in (0x22, 0x26): hits += n if loads: return 100.0 * hits / loads else: return 0.0 def histogram(d): bins = {} for v in d.itervalues(): bins[v] = bins.get(v, 0) + 1 L = bins.items() L.sort() return L def U64(s): return struct.unpack(">Q", s)[0] def oid_repr(oid): if isinstance(oid, StringType) and len(oid) == 8: return '%16x' % U64(oid) else: return repr(oid) def addcommas(n): sign, s = '', str(n) if s[0] == '-': sign, s = '-', s[1:] i = len(s) - 3 while i > 0: s = s[:i] + ',' + s[i:] i -= 3 return sign + s explain = { # The first hex digit shows the operation, the second the outcome. # If the second digit is in "02468" then it is a 'miss'. # If it is in "ACE" then it is a 'hit'. 0x00: "_setup_trace (initialization)", 0x10: "invalidate (miss)", 0x1A: "invalidate (hit, version)", 0x1C: "invalidate (hit, saving non-current)", # 0x1E can occur during startup verification. 0x1E: "invalidate (hit, discarding current or non-current)", 0x20: "load (miss)", 0x22: "load (hit)", 0x24: "load (non-current, miss)", 0x26: "load (non-current, hit)", 0x50: "store (version)", 0x52: "store (current, non-version)", 0x54: "store (non-current)", } if __name__ == "__main__": sys.exit(main()) Zope-2.8.7-final/lib/python/ZEO/CommitLog.py0000664000076400007640000000320510435502417017531 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Log a transaction's commit info during two-phase commit. A storage server allows multiple clients to commit transactions, but must serialize them as the actually execute at the server. The concurrent commits are achieved by logging actions up until the tpc_vote(). At that point, the entire transaction is committed on the real storage. """ import cPickle import tempfile class CommitLog: def __init__(self): self.file = tempfile.TemporaryFile(suffix=".log") self.pickler = cPickle.Pickler(self.file, 1) self.pickler.fast = 1 self.stores = 0 self.read = 0 def size(self): return self.file.tell() def store(self, oid, serial, data, version): self.pickler.dump((oid, serial, data, version)) self.stores += 1 def get_loader(self): self.read = 1 self.file.seek(0) return self.stores, cPickle.Unpickler(self.file) def close(self): if self.file: self.file.close() self.file = None Zope-2.8.7-final/lib/python/ZEO/util.py0000664000076400007640000000350710435502417016621 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Utilities for setting up the server environment.""" import os def parentdir(p, n=1): """Return the ancestor of p from n levels up.""" d = p while n: d = os.path.dirname(d) if not d or d == '.': d = os.getcwd() n -= 1 return d class Environment: """Determine location of the Data.fs & ZEO_SERVER.pid files. Pass the argv[0] used to start ZEO to the constructor. Use the zeo_pid and fs attributes to get the filenames. """ def __init__(self, argv0): v = os.environ.get("INSTANCE_HOME") if v is None: # looking for a Zope/var directory assuming that this code # is installed in Zope/lib/python/ZEO p = parentdir(argv0, 4) if os.path.isdir(os.path.join(p, "var")): v = p else: v = os.getcwd() self.home = v self.var = os.path.join(v, "var") if not os.path.isdir(self.var): self.var = self.home pid = os.environ.get("ZEO_SERVER_PID") if pid is None: pid = os.path.join(self.var, "ZEO_SERVER.pid") self.zeo_pid = pid self.fs = os.path.join(self.var, "Data.fs") Zope-2.8.7-final/lib/python/ZEO/runzeo.py0000664000076400007640000003054310435502417017166 0ustar ajungajung#!python ############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Start the ZEO storage server. Usage: %s [-C URL] [-a ADDRESS] [-f FILENAME] [-h] Options: -C/--configuration URL -- configuration file or URL -a/--address ADDRESS -- server address of the form PORT, HOST:PORT, or PATH (a PATH must contain at least one "/") -f/--filename FILENAME -- filename for FileStorage -t/--timeout TIMEOUT -- transaction timeout in seconds (default no timeout) -h/--help -- print this usage message and exit -m/--monitor ADDRESS -- address of monitor server ([HOST:]PORT or PATH) --pid-file PATH -- relative path to output file containing this process's pid; default $(INSTANCE_HOME)/var/ZEO.pid but only if envar INSTANCE_HOME is defined Unless -C is specified, -a and -f are required. """ # The code here is designed to be reused by other, similar servers. # For the forseeable future, it must work under Python 2.1 as well as # 2.2 and above. import os import sys import signal import socket import logging import ZConfig, ZConfig.datatypes import ZEO from zdaemon.zdoptions import ZDOptions logger = logging.getLogger('ZEO.runzeo') _pid = str(os.getpid()) def log(msg, level=logging.INFO, exc_info=False): """Internal: generic logging function.""" message = "(%s) %s" % (_pid, msg) logger.log(level, message, exc_info=exc_info) def parse_binding_address(arg): # Caution: Not part of the official ZConfig API. obj = ZConfig.datatypes.SocketBindingAddress(arg) return obj.family, obj.address def windows_shutdown_handler(): # Called by the signal mechanism on Windows to perform shutdown. import asyncore asyncore.close_all() class ZEOOptionsMixin: storages = None def handle_address(self, arg): self.family, self.address = parse_binding_address(arg) def handle_monitor_address(self, arg): self.monitor_family, self.monitor_address = parse_binding_address(arg) def handle_filename(self, arg): from ZODB.config import FileStorage # That's a FileStorage *opener*! class FSConfig: def __init__(self, name, path): self._name = name self.path = path self.create = 0 self.read_only = 0 self.stop = None self.quota = None def getSectionName(self): return self._name if not self.storages: self.storages = [] name = str(1 + len(self.storages)) conf = FileStorage(FSConfig(name, arg)) self.storages.append(conf) def add_zeo_options(self): self.add(None, None, "a:", "address=", self.handle_address) self.add(None, None, "f:", "filename=", self.handle_filename) self.add("family", "zeo.address.family") self.add("address", "zeo.address.address", required="no server address specified; use -a or -C") self.add("read_only", "zeo.read_only", default=0) self.add("invalidation_queue_size", "zeo.invalidation_queue_size", default=100) self.add("transaction_timeout", "zeo.transaction_timeout", "t:", "timeout=", float) self.add("monitor_address", "zeo.monitor_address.address", "m:", "monitor=", self.handle_monitor_address) self.add('auth_protocol', 'zeo.authentication_protocol', None, 'auth-protocol=', default=None) self.add('auth_database', 'zeo.authentication_database', None, 'auth-database=') self.add('auth_realm', 'zeo.authentication_realm', None, 'auth-realm=') self.add('pid_file', 'zeo.pid_filename', None, 'pid-file=') class ZEOOptions(ZDOptions, ZEOOptionsMixin): logsectionname = "eventlog" schemadir = os.path.dirname(ZEO.__file__) def __init__(self): ZDOptions.__init__(self) self.add_zeo_options() self.add("storages", "storages", required="no storages specified; use -f or -C") class ZEOServer: def __init__(self, options): self.options = options def main(self): self.setup_default_logging() self.check_socket() self.clear_socket() self.make_pidfile() try: self.open_storages() self.setup_signals() self.create_server() self.loop_forever() finally: self.close_storages() self.clear_socket() self.remove_pidfile() def setup_default_logging(self): if self.options.config_logger is not None: return # No log file is configured; default to stderr. root = logging.getLogger() root.setLevel(logging.INFO) fmt = logging.Formatter( "------\n%(asctime)s %(levelname)s %(name)s %(message)s", "%Y-%m-%dT%H:%M:%S") handler = logging.StreamHandler() handler.setFormatter(fmt) root.addHandler(handler) def check_socket(self): if self.can_connect(self.options.family, self.options.address): self.options.usage("address %s already in use" % repr(self.options.address)) def can_connect(self, family, address): s = socket.socket(family, socket.SOCK_STREAM) try: s.connect(address) except socket.error: return 0 else: s.close() return 1 def clear_socket(self): if isinstance(self.options.address, type("")): try: os.unlink(self.options.address) except os.error: pass def open_storages(self): self.storages = {} for opener in self.options.storages: log("opening storage %r using %s" % (opener.name, opener.__class__.__name__)) self.storages[opener.name] = opener.open() def setup_signals(self): """Set up signal handlers. The signal handler for SIGFOO is a method handle_sigfoo(). If no handler method is defined for a signal, the signal action is not changed from its initial value. The handler method is called without additional arguments. """ if os.name != "posix": if os.name == "nt": self.setup_win32_signals() return if hasattr(signal, 'SIGXFSZ'): signal.signal(signal.SIGXFSZ, signal.SIG_IGN) # Special case init_signames() for sig, name in signames.items(): method = getattr(self, "handle_" + name.lower(), None) if method is not None: def wrapper(sig_dummy, frame_dummy, method=method): method() signal.signal(sig, wrapper) def setup_win32_signals(self): # Borrow the Zope Signals package win32 support, if available. # Signals does a check/log for the availability of pywin32. try: import Signals.Signals except ImportError: logger.debug("Signals package not found. " "Windows-specific signal handler " "will *not* be installed.") return SignalHandler = Signals.Signals.SignalHandler if SignalHandler is not None: # may be None if no pywin32. SignalHandler.registerHandler(signal.SIGTERM, windows_shutdown_handler) SignalHandler.registerHandler(signal.SIGINT, windows_shutdown_handler) SIGUSR2 = 12 # not in signal module on Windows. SignalHandler.registerHandler(SIGUSR2, self.handle_sigusr2) def create_server(self): from ZEO.StorageServer import StorageServer self.server = StorageServer( self.options.address, self.storages, read_only=self.options.read_only, invalidation_queue_size=self.options.invalidation_queue_size, transaction_timeout=self.options.transaction_timeout, monitor_address=self.options.monitor_address, auth_protocol=self.options.auth_protocol, auth_database=self.options.auth_database, auth_realm=self.options.auth_realm) def loop_forever(self): import ThreadedAsync.LoopCallback ThreadedAsync.LoopCallback.loop() def handle_sigterm(self): log("terminated by SIGTERM") sys.exit(0) def handle_sigint(self): log("terminated by SIGINT") sys.exit(0) def handle_sighup(self): log("restarted by SIGHUP") sys.exit(1) def handle_sigusr2(self): # TODO: this used to reinitialize zLOG. How do I achieve # the same effect with Python's logging package? # Should we restart as with SIGHUP? log("received SIGUSR2, but it was not handled!", level=logging.WARNING) def close_storages(self): for name, storage in self.storages.items(): log("closing storage %r" % name) try: storage.close() except: # Keep going log("failed to close storage %r" % name, level=logging.ERROR, exc_info=True) def _get_pidfile(self): pidfile = self.options.pid_file # 'pidfile' is marked as not required. if not pidfile: # Try to find a reasonable location if the pidfile is not # set. If we are running in a Zope environment, we can # safely assume INSTANCE_HOME. instance_home = os.environ.get("INSTANCE_HOME") if not instance_home: # If all our attempts failed, just log a message and # proceed. logger.debug("'pidfile' option not set, and 'INSTANCE_HOME' " "environment variable could not be found. " "Cannot guess pidfile location.") return self.options.pid_file = os.path.join(instance_home, "var", "ZEO.pid") def make_pidfile(self): if not self.options.read_only: self._get_pidfile() pidfile = self.options.pid_file if pidfile is None: return pid = os.getpid() try: if os.path.exists(pidfile): os.unlink(pidfile) f = open(pidfile, 'w') print >> f, pid f.close() log("created PID file '%s'" % pidfile) except IOError: logger.error("PID file '%s' cannot be opened" % pidfile) def remove_pidfile(self): if not self.options.read_only: pidfile = self.options.pid_file if pidfile is None: return try: if os.path.exists(pidfile): os.unlink(pidfile) log("removed PID file '%s'" % pidfile) except IOError: logger.error("PID file '%s' could not be removed" % pidfile) # Signal names signames = None def signame(sig): """Return a symbolic name for a signal. Return "signal NNN" if there is no corresponding SIG name in the signal module. """ if signames is None: init_signames() return signames.get(sig) or "signal %d" % sig def init_signames(): global signames signames = {} for name, sig in signal.__dict__.items(): k_startswith = getattr(name, "startswith", None) if k_startswith is None: continue if k_startswith("SIG") and not k_startswith("SIG_"): signames[sig] = name # Main program def main(args=None): options = ZEOOptions() options.realize(args) s = ZEOServer(options) s.main() if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/TransactionBuffer.py0000664000076400007640000001146610435502417021266 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A TransactionBuffer store transaction updates until commit or abort. A transaction may generate enough data that it is not practical to always hold pending updates in memory. Instead, a TransactionBuffer is used to store the data until a commit or abort. """ # A faster implementation might store trans data in memory until it # reaches a certain size. import cPickle import tempfile from threading import Lock class TransactionBuffer: # Valid call sequences: # # ((store | invalidate)* begin_iterate next* clear)* close # # get_size can be called any time # The TransactionBuffer is used by client storage to hold update # data until the tpc_finish(). It is normally used by a single # thread, because only one thread can be in the two-phase commit # at one time. # It is possible, however, for one thread to close the storage # while another thread is in the two-phase commit. We must use # a lock to guard against this race, because unpredictable things # can happen in Python if one thread closes a file that another # thread is reading. In a debug build, an assert() can fail. # Caution: If an operation is performed on a closed TransactionBuffer, # it has no effect and does not raise an exception. The only time # this should occur is when a ClientStorage is closed in one # thread while another thread is in its tpc_finish(). It's not # clear what should happen in this case. If the tpc_finish() # completes without error, the Connection using it could have # inconsistent data. This should have minimal effect, though, # because the Connection is connected to a closed storage. def __init__(self): self.file = tempfile.TemporaryFile(suffix=".tbuf") self.lock = Lock() self.closed = 0 self.count = 0 self.size = 0 # It's safe to use a fast pickler because the only objects # stored are builtin types -- strings or None. self.pickler = cPickle.Pickler(self.file, 1) self.pickler.fast = 1 def close(self): self.lock.acquire() try: self.closed = 1 try: self.file.close() except OSError: pass finally: self.lock.release() def store(self, oid, version, data): self.lock.acquire() try: self._store(oid, version, data) finally: self.lock.release() def _store(self, oid, version, data): """Store oid, version, data for later retrieval""" if self.closed: return self.pickler.dump((oid, version, data)) self.count += 1 # Estimate per-record cache size self.size = self.size + len(data) + 31 if version: # Assume version data has same size as non-version data self.size = self.size + len(version) + len(data) + 12 def invalidate(self, oid, version): self.lock.acquire() try: if self.closed: return self.pickler.dump((oid, version, None)) self.count += 1 finally: self.lock.release() def clear(self): """Mark the buffer as empty""" self.lock.acquire() try: if self.closed: return self.file.seek(0) self.count = 0 self.size = 0 finally: self.lock.release() def __iter__(self): self.lock.acquire() try: if self.closed: return self.file.flush() self.file.seek(0) return TBIterator(self.file, self.count) finally: self.lock.release() class TBIterator(object): def __init__(self, f, count): self.file = f self.count = count self.unpickler = cPickle.Unpickler(f) def __iter__(self): return self def next(self): """Return next tuple of data or None if EOF""" if self.count == 0: self.file.seek(0) self.size = 0 raise StopIteration oid_ver_data = self.unpickler.load() self.count -= 1 return oid_ver_data Zope-2.8.7-final/lib/python/ZEO/simul.py0000664000076400007640000016321510435502417017000 0ustar ajungajung#! /usr/bin/env python ############################################################################## # # Copyright (c) 2001-2005 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Cache simulation. Usage: simul.py [-s size] tracefile Options: -s size: cache size in MB (default 20 MB) """ import sys import time import getopt import struct import math import bisect from sets import Set from ZODB.utils import z64 def usage(msg): print >> sys.stderr, msg print >> sys.stderr, __doc__ def main(): # Parse options. MB = 1024**2 cachelimit = 20*MB simclass = CircularCacheSimulation try: opts, args = getopt.getopt(sys.argv[1:], "bflyz2cOaTUs:") except getopt.error, msg: usage(msg) return 2 for o, a in opts: if o == '-b': simclass = BuddyCacheSimulation elif o == '-f': simclass = SimpleCacheSimulation elif o == '-l': simclass = LRUCacheSimulation elif o == '-y': simclass = AltZEOCacheSimulation elif o == '-z': simclass = ZEOCacheSimulation elif o == '-s': cachelimit = int(float(a)*MB) elif o == '-2': simclass = TwoQSimluation elif o == '-c': simclass = CircularCacheSimulation elif o == '-O': simclass = OracleSimulation elif o == '-a': simclass = ARCCacheSimulation elif o == '-T': simclass = ThorSimulation elif o == '-U': simclass = UnboundedSimulation else: assert False, (o, a) if len(args) != 1: usage("exactly one file argument required") return 2 filename = args[0] # Open file. if filename.endswith(".gz"): # Open gzipped file. try: import gzip except ImportError: print >> sys.stderr, "can't read gzipped files (no module gzip)" return 1 try: f = gzip.open(filename, "rb") except IOError, msg: print >> sys.stderr, "can't open %s: %s" % (filename, msg) return 1 elif filename == "-": # Read from stdin. f = sys.stdin else: # Open regular file. try: f = open(filename, "rb") except IOError, msg: print >> sys.stderr, "can't open %s: %s" % (filename, msg) return 1 # Create simulation object. if simclass is OracleSimulation: sim = simclass(cachelimit, filename) else: sim = simclass(cachelimit) # Print output header. sim.printheader() # Read trace file, simulating cache behavior. f_read = f.read unpack = struct.unpack FMT = ">iiH8s8s" FMT_SIZE = struct.calcsize(FMT) assert FMT_SIZE == 26 while 1: # Read a record and decode it. r = f_read(FMT_SIZE) if len(r) < FMT_SIZE: break ts, code, oidlen, start_tid, end_tid = unpack(FMT, r) if ts == 0: # Must be a misaligned record caused by a crash; skip 8 bytes # and try again. Why 8? Lost in the mist of history. f.seek(f.tell() - FMT_SIZE + 8) continue oid = f_read(oidlen) if len(oid) < oidlen: break # Decode the code. dlen, version, code = (code & 0x7fffff00, code & 0x80, code & 0x7e) # And pass it to the simulation. sim.event(ts, dlen, version, code, oid, start_tid, end_tid) f.close() # Finish simulation. sim.finish() # Exit code from main(). return 0 class Simulation(object): """Base class for simulations. The driver program calls: event(), printheader(), finish(). The standard event() method calls these additional methods: write(), load(), inval(), report(), restart(); the standard finish() method also calls report(). """ def __init__(self, cachelimit): self.cachelimit = cachelimit # Initialize global statistics. self.epoch = None self.total_loads = 0 self.total_hits = 0 # subclass must increment self.total_invals = 0 # subclass must increment self.total_writes = 0 if not hasattr(self, "extras"): self.extras = (self.extraname,) self.format = self.format + " %7s" * len(self.extras) # Reset per-run statistics and set up simulation data. self.restart() def restart(self): # Reset per-run statistics. self.loads = 0 self.hits = 0 # subclass must increment self.invals = 0 # subclass must increment self.writes = 0 self.ts0 = None def event(self, ts, dlen, _version, code, oid, start_tid, end_tid): # Record first and last timestamp seen. if self.ts0 is None: self.ts0 = ts if self.epoch is None: self.epoch = ts self.ts1 = ts # Simulate cache behavior. Caution: the codes in the trace file # record whether the actual cache missed or hit on each load, but # that bears no necessary relationship to whether the simulated cache # will hit or miss. Relatedly, if the actual cache needed to store # an object, the simulated cache may not need to (it may already # have the data). action = code & 0x70 if action == 0x20: # Load. self.loads += 1 self.total_loads += 1 # Asserting that dlen is 0 iff it's a load miss. # assert (dlen == 0) == (code in (0x20, 0x24)) self.load(oid, dlen, start_tid) elif action == 0x50: # Store. assert dlen self.write(oid, dlen, start_tid, end_tid) elif action == 0x10: # Invalidate. self.inval(oid, start_tid) elif action == 0x00: # Restart. self.report() self.restart() else: raise ValueError("unknown trace code 0x%x" % code) def write(self, oid, size, start_tid, end_tid): pass def load(self, oid, size, start_tid): # Must increment .hits and .total_hits as appropriate. pass def inval(self, oid, start_tid): # Must increment .invals and .total_invals as appropriate. pass format = "%12s %9s %8s %8s %6s %6s %7s" # Subclass should override extraname to name known instance variables; # if extraname is 'foo', both self.foo and self.total_foo must exist: extraname = "*** please override ***" def printheader(self): print "%s, cache size %s bytes" % (self.__class__.__name__, addcommas(self.cachelimit)) self.extraheader() extranames = tuple([s.upper() for s in self.extras]) args = ("START TIME", "DURATION", "LOADS", "HITS", "INVALS", "WRITES", "HITRATE") + extranames print self.format % args def extraheader(self): pass nreports = 0 def report(self, extratext=''): if self.loads: self.nreports += 1 args = (time.ctime(self.ts0)[4:-8], duration(self.ts1 - self.ts0), self.loads, self.hits, self.invals, self.writes, hitrate(self.loads, self.hits)) args += tuple([getattr(self, name) for name in self.extras]) print self.format % args, extratext def finish(self): # Make sure that the last line of output ends with "OVERALL". This # makes it much easier for another program parsing the output to # find summary statistics. if self.nreports < 2: self.report('OVERALL') else: self.report() args = ( time.ctime(self.epoch)[4:-8], duration(self.ts1 - self.epoch), self.total_loads, self.total_hits, self.total_invals, self.total_writes, hitrate(self.total_loads, self.total_hits)) args += tuple([getattr(self, "total_" + name) for name in self.extras]) print (self.format + " OVERALL") % args # For use in CircularCacheSimulation. class CircularCacheEntry(object): __slots__ = (# object key: an (oid, start_tid) pair, where # start_tid is the tid of the transaction that created # this revision of oid 'key', # tid of transaction that created the next revision; # z64 iff this is the current revision 'end_tid', # Offset from start of file to the object's data # record; this includes all overhead bytes (status # byte, size bytes, etc). 'offset', ) def __init__(self, key, end_tid, offset): self.key = key self.end_tid = end_tid self.offset = offset from ZEO.cache import ZEC3_HEADER_SIZE class CircularCacheSimulation(Simulation): """Simulate the ZEO 3.0 cache.""" # The cache is managed as a single file with a pointer that # goes around the file, circularly, forever. New objects # are written at the current pointer, evicting whatever was # there previously. extras = "evicts", "inuse" def __init__(self, cachelimit): from ZEO import cache Simulation.__init__(self, cachelimit) self.total_evicts = 0 # number of cache evictions # Current offset in file. self.offset = ZEC3_HEADER_SIZE # Map offset in file to (size, CircularCacheEntry) pair, or to # (size, None) if the offset starts a free block. self.filemap = {ZEC3_HEADER_SIZE: (self.cachelimit - ZEC3_HEADER_SIZE, None)} # Map key to CircularCacheEntry. A key is an (oid, tid) pair. self.key2entry = {} # Map oid to tid of current revision. self.current = {} # Map oid to list of (start_tid, end_tid) pairs in sorted order. # Used to find matching key for load of non-current data. self.noncurrent = {} # The number of overhead bytes needed to store an object pickle # on disk (all bytes beyond those needed for the object pickle). self.overhead = (cache.Object.TOTAL_FIXED_SIZE + cache.OBJECT_HEADER_SIZE) def restart(self): Simulation.restart(self) self.evicts = 0 def load(self, oid, size, tid): if tid == z64: # Trying to load current revision. if oid in self.current: # else it's a cache miss self.hits += 1 self.total_hits += 1 return # May or may not be trying to load current revision. cur_tid = self.current.get(oid) if cur_tid == tid: self.hits += 1 self.total_hits += 1 return # It's a load for non-current data. Do we know about this oid? L = self.noncurrent.get(oid) if L is None: return # cache miss i = bisect.bisect_left(L, (tid, None)) if i == 0: # This tid is smaller than any we know about -- miss. return lo, hi = L[i-1] assert lo < tid if tid > hi: # No data in the right tid range -- miss. return # Cache hit. self.hits += 1 self.total_hits += 1 # (oid, tid) is in the cache. Remove it: take it out of key2entry, # and in `filemap` mark the space it occupied as being free. The # caller is responsible for removing it from `current` or `noncurrent`. def _remove(self, oid, tid): key = oid, tid e = self.key2entry.pop(key) pos = e.offset size, _e = self.filemap[pos] assert e is _e self.filemap[pos] = size, None def _remove_noncurrent_revisions(self, oid): noncurrent_list = self.noncurrent.get(oid) if noncurrent_list: self.invals += len(noncurrent_list) self.total_invals += len(noncurrent_list) for start_tid, end_tid in noncurrent_list: self._remove(oid, start_tid) del self.noncurrent[oid] def inval(self, oid, tid): if tid == z64: # This is part of startup cache verification: forget everything # about this oid. self._remove_noncurrent_revisions(oid) cur_tid = self.current.get(oid) if cur_tid is None: # We don't have current data, so nothing more to do. return # We had current data for oid, but no longer. self.invals += 1 self.total_invals += 1 del self.current[oid] if tid == z64: # Startup cache verification: forget this oid entirely. self._remove(oid, cur_tid) return # Our current data becomes non-current data. # Add the validity range to the list of non-current data for oid. assert cur_tid < tid L = self.noncurrent.setdefault(oid, []) bisect.insort_left(L, (cur_tid, tid)) # Update the end of oid's validity range in its CircularCacheEntry. e = self.key2entry[oid, cur_tid] assert e.end_tid == z64 e.end_tid = tid def write(self, oid, size, start_tid, end_tid): if end_tid == z64: # Storing current revision. if oid in self.current: # we already have it in cache return self.current[oid] = start_tid self.writes += 1 self.total_writes += 1 self.add(oid, size, start_tid) return # Storing non-current revision. L = self.noncurrent.setdefault(oid, []) p = start_tid, end_tid if p in L: return # we already have it in cache bisect.insort_left(L, p) self.writes += 1 self.total_writes += 1 self.add(oid, size, start_tid, end_tid) # Add `oid` to the cache, evicting objects as needed to make room. # This updates `filemap` and `key2entry`; it's the caller's # responsibilty to update `current` or `noncurrent` appropriately. def add(self, oid, size, start_tid, end_tid=z64): key = oid, start_tid assert key not in self.key2entry size += self.overhead avail = self.makeroom(size) e = CircularCacheEntry(key, end_tid, self.offset) self.filemap[self.offset] = size, e self.key2entry[key] = e self.offset += size # All the space made available must be accounted for in filemap. excess = avail - size if excess: self.filemap[self.offset] = excess, None # Evict enough objects to make at least `need` contiguous bytes, starting # at `self.offset`, available. Evicted objects are removed from # `filemap`, `key2entry`, `current` and `noncurrent`. The caller is # responsible for adding new entries to `filemap` to account for all # the freed bytes, and for advancing `self.offset`. The number of bytes # freed is the return value, and will be >= need. def makeroom(self, need): if self.offset + need > self.cachelimit: self.offset = ZEC3_HEADER_SIZE pos = self.offset while need > 0: assert pos < self.cachelimit size, e = self.filemap.pop(pos) if e: # there is an object here (else it's already free space) self.evicts += 1 self.total_evicts += 1 assert pos == e.offset _e = self.key2entry.pop(e.key) assert e is _e oid, start_tid = e.key if e.end_tid == z64: del self.current[oid] else: L = self.noncurrent[oid] L.remove((start_tid, e.end_tid)) need -= size pos += size return pos - self.offset # total number of bytes freed def report(self): self.check() free = used = total = 0 for size, e in self.filemap.itervalues(): total += size if e: used += size else: free += size self.inuse = round(100.0 * used / total, 1) self.total_inuse = self.inuse Simulation.report(self) def check(self): oidcount = 0 pos = ZEC3_HEADER_SIZE while pos < self.cachelimit: size, e = self.filemap[pos] if e: oidcount += 1 assert self.key2entry[e.key].offset == pos pos += size assert oidcount == len(self.key2entry) assert pos == self.cachelimit def dump(self): print len(self.filemap) L = list(self.filemap) L.sort() for k in L: v = self.filemap[k] print k, v[0], repr(v[1]) ############################################################################# # CAUTION: It's most likely that none of the simulators below this # point work anymore. A great many changes were needed to teach # CircularCacheSimulation (above) about MVCC, including method signature # changes and changes in cache file format, and none of the other simulator # classes were changed. ############################################################################# class ZEOCacheSimulation(Simulation): """Simulate the ZEO 1.0 and 2.0 cache behavior. This assumes the cache is not persistent (we don't know how to simulate cache validation.) """ extraname = "flips" def __init__(self, cachelimit): # Initialize base class Simulation.__init__(self, cachelimit) # Initialize additional global statistics self.total_flips = 0 def restart(self): # Reset base class Simulation.restart(self) # Reset additional per-run statistics self.flips = 0 # Set up simulation self.filesize = [4, 4] # account for magic number self.fileoids = [{}, {}] self.current = 0 # index into filesize, fileoids def load(self, oid, size): if (self.fileoids[self.current].get(oid) or self.fileoids[1 - self.current].get(oid)): self.hits += 1 self.total_hits += 1 else: self.write(oid, size) def write(self, oid, size): # Fudge because size is rounded up to multiples of 256. (31 # is header overhead per cache record; 127 is to compensate # for rounding up to multiples of 256.) size = size + 31 - 127 if self.filesize[self.current] + size > self.cachelimit / 2: # Cache flip self.flips += 1 self.total_flips += 1 self.current = 1 - self.current self.filesize[self.current] = 4 self.fileoids[self.current] = {} self.filesize[self.current] += size self.fileoids[self.current][oid] = 1 def inval(self, oid): if self.fileoids[self.current].get(oid): self.invals += 1 self.total_invals += 1 del self.fileoids[self.current][oid] elif self.fileoids[1 - self.current].get(oid): self.invals += 1 self.total_invals += 1 del self.fileoids[1 - self.current][oid] class AltZEOCacheSimulation(ZEOCacheSimulation): """A variation of the ZEO cache that copies to the current file. When a hit is found in the non-current cache file, it is copied to the current cache file. Exception: when the copy would cause a cache flip, we don't copy (this is part laziness, part concern over causing extraneous flips). """ def load(self, oid, size): if self.fileoids[self.current].get(oid): self.hits += 1 self.total_hits += 1 elif self.fileoids[1 - self.current].get(oid): self.hits += 1 self.total_hits += 1 # Simulate a write, unless it would cause a flip size = size + 31 - 127 if self.filesize[self.current] + size <= self.cachelimit / 2: self.filesize[self.current] += size self.fileoids[self.current][oid] = 1 del self.fileoids[1 - self.current][oid] else: self.write(oid, size) class LRUCacheSimulation(Simulation): extraname = "evicts" def __init__(self, cachelimit): # Initialize base class Simulation.__init__(self, cachelimit) # Initialize additional global statistics self.total_evicts = 0 def restart(self): # Reset base class Simulation.restart(self) # Reset additional per-run statistics self.evicts = 0 # Set up simulation self.cache = {} self.size = 0 self.head = Node(None, None) self.head.linkbefore(self.head) def load(self, oid, size): node = self.cache.get(oid) if node is not None: self.hits += 1 self.total_hits += 1 node.linkbefore(self.head) else: self.write(oid, size) def write(self, oid, size): node = self.cache.get(oid) if node is not None: node.unlink() assert self.head.next is not None self.size -= node.size node = Node(oid, size) self.cache[oid] = node node.linkbefore(self.head) self.size += size # Evict LRU nodes while self.size > self.cachelimit: self.evicts += 1 self.total_evicts += 1 node = self.head.next assert node is not self.head node.unlink() assert self.head.next is not None del self.cache[node.oid] self.size -= node.size def inval(self, oid): node = self.cache.get(oid) if node is not None: assert node.oid == oid self.invals += 1 self.total_invals += 1 node.unlink() assert self.head.next is not None del self.cache[oid] self.size -= node.size assert self.size >= 0 class Node(object): """Node in a doubly-linked list, storing oid and size as payload. A node can be linked or unlinked; in the latter case, next and prev are None. Initially a node is unlinked. """ __slots__ = ['prev', 'next', 'oid', 'size'] def __init__(self, oid, size): self.oid = oid self.size = size self.prev = self.next = None def unlink(self): prev = self.prev next = self.next if prev is not None: assert next is not None assert prev.next is self assert next.prev is self prev.next = next next.prev = prev self.prev = self.next = None else: assert next is None def linkbefore(self, next): self.unlink() prev = next.prev if prev is None: assert next.next is None prev = next self.prev = prev self.next = next prev.next = next.prev = self am = object() a1in = object() a1out = object() class Node2Q(Node): __slots__ = ["kind", "hits"] def __init__(self, oid, size, kind=None): Node.__init__(self, oid, size) self.kind = kind self.hits = 0 def linkbefore(self, next): if next.kind != self.kind: self.kind = next.kind Node.linkbefore(self, next) class TwoQSimluation(Simulation): # The original 2Q algorithm is page based and the authors offer # tuning guidlines based on a page-based cache. Our cache is # object based, so, for example, it's hard to compute the number # of oids to store in a1out based on the size of a1in. extras = "evicts", "hothit", "am_add" NodeClass = Node2Q def __init__(self, cachelimit, outlen=10000, threshold=0): Simulation.__init__(self, cachelimit) # The promotion threshold: If a hit occurs in a1out, it is # promoted to am if the number of hits on the object while it # was in a1in is at least threshold. The standard 2Q scheme # uses a threshold of 0. self.threshold = threshold self.am_limit = 3 * self.cachelimit / 4 self.a1in_limit = self.cachelimit / 4 self.cache = {} self.am_size = 0 self.a1in_size = 0 self.a1out_size = 0 self.total_evicts = 0 self.total_hothit = 0 self.total_am_add = 0 self.a1out_limit = outlen # An LRU queue of hot objects self.am = self.NodeClass(None, None, am) self.am.linkbefore(self.am) # A FIFO queue of recently referenced objects. It's purpose # is to absorb references to objects that are accessed a few # times in short order, then forgotten about. self.a1in = self.NodeClass(None, None, a1in) self.a1in.linkbefore(self.a1in) # A FIFO queue of recently reference oids. # This queue only stores the oids, not any data. If we get a # hit in this queue, promote the object to am. self.a1out = self.NodeClass(None, None, a1out) self.a1out.linkbefore(self.a1out) def makespace(self, size): for space in 0, size: if self.enoughspace(size): return self.evict_a1in(space) if self.enoughspace(size): return self.evict_am(space) def enoughspace(self, size): totalsize = self.a1in_size + self.am_size return totalsize + size < self.cachelimit def evict_a1in(self, extra): while self.a1in_size + extra > self.a1in_limit: if self.a1in.next is self.a1in: return assert self.a1in.next is not None node = self.a1in.next self.evicts += 1 self.total_evicts += 1 node.linkbefore(self.a1out) self.a1out_size += 1 self.a1in_size -= node.size if self.a1out_size > self.a1out_limit: assert self.a1out.next is not None node = self.a1out.next node.unlink() del self.cache[node.oid] self.a1out_size -= 1 def evict_am(self, extra): while self.am_size + extra > self.am_limit: if self.am.next is self.am: return assert self.am.next is not None node = self.am.next self.evicts += 1 self.total_evicts += 1 # This node hasn't been accessed in a while, so just # forget about it. node.unlink() del self.cache[node.oid] self.am_size -= node.size def write(self, oid, size): # A write always follows a read (ZODB doesn't allow blind writes). # So this write must have followed a recent read of the object. # Don't change it's position, but do update the size. # XXX For now, don't evict pages if the new version of the object # is big enough to require eviction. node = self.cache.get(oid) if node is None or node.kind is a1out: return if node.kind is am: self.am_size = self.am_size - node.size + size node.size = size else: self.a1in_size = self.a1in_size - node.size + size node.size = size def load(self, oid, size): node = self.cache.get(oid) if node is not None: if node.kind is am: self.hits += 1 self.total_hits += 1 self.hothit += 1 self.total_hothit += 1 node.hits += 1 node.linkbefore(self.am) elif node.kind is a1in: self.hits += 1 self.total_hits += 1 node.hits += 1 elif node.kind is a1out: self.a1out_size -= 1 if node.hits >= self.threshold: self.makespace(node.size) self.am_size += node.size node.linkbefore(self.am) self.cache[oid] = node self.am_add += 1 self.total_am_add += 1 else: node.unlink() self.insert(oid, size) else: self.insert(oid, size) def insert(self, oid, size): # New objects enter the cache via a1in. If they # are frequently used over a long enough time, they # will be promoted to am -- but only via a1out. self.makespace(size) node = self.NodeClass(oid, size, a1in) node.linkbefore(self.a1in) self.cache[oid] = node self.a1in_size += node.size def inval(self, oid): # The original 2Q algorithm didn't have to deal with # invalidations. My own solution: Move it to the head of # a1out. node = self.cache.get(oid) if node is None: return self.invals += 1 self.total_invals += 1 # XXX Should an invalidation to a1out count? if node.kind is a1out: return node.linkbefore(self.a1out) if node.kind is am: self.am_size -= node.size else: self.a1in_size -= node.size def restart(self): Simulation.restart(self) self.evicts = 0 self.hothit = 0 self.am_add = 0 lruT = object() lruB = object() fifoT = object() fifoB = object() class ARCCacheSimulation(Simulation): # Based on the paper ARC: A Self-Tuning, Low Overhead Replacement # Cache by Nimrod Megiddo and Dharmendra S. Modha, USENIX FAST # 2003. The paper describes a block-based cache. A lot of the # details need to be fiddled to work with an object-based cache. # For size issues, the key insight ended up being conditions # A.1-A.5 rather than the details of the algorithm in Fig. 4. extras = "lruThits", "evicts", "p" def __init__(self, cachelimit): Simulation.__init__(self, cachelimit) # There are two pairs of linked lists. Each pair has a top and # bottom half. The bottom half contains metadata, but not actual # objects. # LRU list of frequently used objects self.lruT = Node2Q(None, None, lruT) self.lruT.linkbefore(self.lruT) self.lruT_len = 0 self.lruT_size = 0 self.lruB = Node2Q(None, None, lruB) self.lruB.linkbefore(self.lruB) self.lruB_len = 0 self.lruB_size = 0 # FIFO list of objects seen once self.fifoT = Node2Q(None, None, fifoT) self.fifoT.linkbefore(self.fifoT) self.fifoT_len = 0 self.fifoT_size = 0 self.fifoB = Node2Q(None, None, fifoB) self.fifoB.linkbefore(self.fifoB) self.fifoB_len = 0 self.fifoB_size = 0 # maps oid to node self.cache = {} # The paper says that p should be adjust be 1 as the minimum: # "The compound effect of such small increments and decrements # to p s quite profound as we will demonstrated in the next # section." Not really, as far as I can tell. In my traces # with a very small cache, it was taking far too long to adjust # towards favoring some FIFO component. I would guess that the # chief difference is that our caches are much bigger than the # ones they experimented with. Their biggest cache had 512K # entries, while our smallest cache will have 40 times that many # entries. self.p = 0 # XXX multiply computed adjustments to p by walk_factor self.walk_factor = 500 # statistics self.total_hits = 0 self.total_lruThits = 0 self.total_fifoThits = 0 self.total_evicts = 0 def restart(self): Simulation.restart(self) self.hits = 0 self.lruThits = 0 self.fifoThits = 0 self.evicts = 0 def write(self, oid, size): pass def replace(self, lruB=False): self.evicts += 1 self.total_evicts += 1 if self.fifoT_size > self.p or (lruB and self.fifoT_size == self.p): node = self.fifoT.next if node is self.fifoT: return 0 assert node is not self.fifoT, self.stats() node.linkbefore(self.fifoB) self.fifoT_len -= 1 self.fifoT_size -= node.size self.fifoB_len += 1 self.fifoB_size += node.size else: node = self.lruT.next if node is self.lruT: return 0 assert node is not self.lruT, self.stats() node.linkbefore(self.lruB) self.lruT_len -= 1 self.lruT_size -= node.size self.lruB_len += 1 self.lruB_size += node.size return node.size def stats(self): self.totalsize = self.lruT_size + self.fifoT_size self.allsize = self.totalsize + self.lruB_size + self.fifoB_size print "cachelimit = %s totalsize=%s allsize=%s" % ( addcommas(self.cachelimit), addcommas(self.totalsize), addcommas(self.allsize)) fmt = ( "p=%(p)d\n" "lruT = %(lruT_len)5d / %(lruT_size)8d / %(lruThits)d\n" "fifoT = %(fifoT_len)5d / %(fifoT_size)8d / %(fifoThits)d\n" "lruB = %(lruB_len)5d / %(lruB_size)8d\n" "fifoB = %(fifoB_len)5d / %(fifoB_size)8d\n" "loads=%(loads)d hits=%(hits)d evicts=%(evicts)d\n" ) print fmt % self.__dict__ def report(self): self.total_p = self.p Simulation.report(self) ## self.stats() def load(self, oid, size): ## maybe(self.stats, p=0.002) node = self.cache.get(oid) if node is None: # cache miss: We're going to insert a new object in fifoT. # If fifo is full, we'll need to evict something to make # room for it. prev = need = size while need > 0: if size + self.fifoT_size + self.fifoB_size >= self.cachelimit: if need + self.fifoT_size >= self.cachelimit: node = self.fifoB.next assert node is not self.fifoB, self.stats() node.unlink() del self.cache[node.oid] self.fifoB_size -= node.size self.fifoB_len -= 1 self.evicts += 1 self.total_evicts += 1 else: node = self.fifoB.next assert node is not self.fifoB, self.stats() node.unlink() del self.cache[node.oid] self.fifoB_size -= node.size self.fifoB_len -= 1 if self.fifoT_size + self.lruT_size > self.cachelimit: need -= self.replace() else: incache_size = self.fifoT_size + self.lruT_size + need total_size = (incache_size + self.fifoB_size + self.lruB_size) if total_size >= self.cachelimit * 2: node = self.lruB.next if node is self.lruB: break assert node is not self.lruB node.unlink() del self.cache[node.oid] self.lruB_size -= node.size self.lruB_len -= 1 elif incache_size > self.cachelimit: need -= self.replace() else: break if need == prev: # XXX hack, apparently we can't get rid of anything else break prev = need node = Node2Q(oid, size) node.linkbefore(self.fifoT) self.fifoT_len += 1 self.fifoT_size += size self.cache[oid] = node else: # a cache hit, but possibly in a bottom list that doesn't # actually hold the object if node.kind is lruT: node.linkbefore(self.lruT) self.hits += 1 self.total_hits += 1 self.lruThits += 1 self.total_lruThits += 1 elif node.kind is fifoT: node.linkbefore(self.lruT) self.fifoT_len -= 1 self.lruT_len += 1 self.fifoT_size -= node.size self.lruT_size += node.size self.hits += 1 self.total_hits += 1 self.fifoThits += 1 self.total_fifoThits += 1 elif node.kind is fifoB: node.linkbefore(self.lruT) self.fifoB_len -= 1 self.lruT_len += 1 self.fifoB_size -= node.size self.lruT_size += node.size # XXX need a better min than 1? ## print "adapt+", max(1, self.lruB_size // self.fifoB_size) delta = max(1, self.lruB_size / max(1, self.fifoB_size)) self.p += delta * self.walk_factor if self.p > self.cachelimit: self.p = self.cachelimit need = node.size if self.lruT_size + self.fifoT_size + need > self.cachelimit: while need > 0: r = self.replace() if not r: break need -= r elif node.kind is lruB: node.linkbefore(self.lruT) self.lruB_len -= 1 self.lruT_len += 1 self.lruB_size -= node.size self.lruT_size += node.size # XXX need a better min than 1? ## print "adapt-", max(1, self.fifoB_size // self.lruB_size) delta = max(1, self.fifoB_size / max(1, self.lruB_size)) self.p -= delta * self.walk_factor if self.p < 0: self.p = 0 need = node.size if self.lruT_size + self.fifoT_size + need > self.cachelimit: while need > 0: r = self.replace(lruB=True) if not r: break need -= r def inval(self, oid): pass def extraheader(self): pass class OracleSimulation(LRUCacheSimulation): # Not sure how to implement this yet. This is a cache where I # cheat to see how good we could actually do. The cache # replacement problem for multi-size caches is NP-hard, so we're # not going to have an optimal solution. # At the moment, the oracle is mostly blind. It knows which # objects will be referenced more than once, so that it can # ignore objects referenced only once. In most traces, these # objects account for about 20% of references. def __init__(self, cachelimit, filename): LRUCacheSimulation.__init__(self, cachelimit) self.count = {} self.scan(filename) def load(self, oid, size): node = self.cache.get(oid) if node is not None: self.hits += 1 self.total_hits += 1 node.linkbefore(self.head) else: if oid in self.count: self.write(oid, size) def scan(self, filename): # scan the file in advance to figure out which objects will # be referenced more than once. f = open(filename, "rb") struct_unpack = struct.unpack f_read = f.read offset = 0 while 1: # Read a record and decode it r = f_read(8) if len(r) < 8: break offset += 8 ts, code = struct_unpack(">ii", r) if ts == 0: # Must be a misaligned record caused by a crash ##print "Skipping 8 bytes at offset", offset-8 continue r = f_read(16) if len(r) < 16: break offset += 16 oid, serial = struct_unpack(">8s8s", r) if code & 0x70 == 0x20: # only look at loads self.count[oid] = self.count.get(oid, 0) + 1 all = len(self.count) # Now remove everything with count == 1 once = [oid for oid, count in self.count.iteritems() if count == 1] for oid in once: del self.count[oid] print "Scanned file, %d unique oids, %d repeats" % ( all, len(self.count)) class BuddyCacheSimulation(LRUCacheSimulation): def __init__(self, cachelimit): LRUCacheSimulation.__init__(self, roundup(cachelimit)) def restart(self): LRUCacheSimulation.restart(self) self.allocator = self.allocatorFactory(self.cachelimit) def allocatorFactory(self, size): return BuddyAllocator(size) # LRUCacheSimulation.load() is just fine def write(self, oid, size): node = self.cache.get(oid) if node is not None: node.unlink() assert self.head.next is not None self.size -= node.size self.allocator.free(node) while 1: node = self.allocator.alloc(size) if node is not None: break # Failure to allocate. Evict something and try again. node = self.head.next assert node is not self.head self.evicts += 1 self.total_evicts += 1 node.unlink() assert self.head.next is not None del self.cache[node.oid] self.size -= node.size self.allocator.free(node) node.oid = oid self.cache[oid] = node node.linkbefore(self.head) self.size += node.size def inval(self, oid): node = self.cache.get(oid) if node is not None: assert node.oid == oid self.invals += 1 self.total_invals += 1 node.unlink() assert self.head.next is not None del self.cache[oid] self.size -= node.size assert self.size >= 0 self.allocator.free(node) class SimpleCacheSimulation(BuddyCacheSimulation): def allocatorFactory(self, size): return SimpleAllocator(size) def finish(self): BuddyCacheSimulation.finish(self) self.allocator.report() MINSIZE = 256 class BuddyAllocator: def __init__(self, cachelimit): cachelimit = roundup(cachelimit) self.cachelimit = cachelimit self.avail = {} # Map rounded-up sizes to free list node heads self.nodes = {} # Map address to node k = MINSIZE while k <= cachelimit: self.avail[k] = n = Node(None, None) # Not BlockNode; has no addr n.linkbefore(n) k += k node = BlockNode(None, cachelimit, 0) self.nodes[0] = node node.linkbefore(self.avail[cachelimit]) def alloc(self, size): size = roundup(size) k = size while k <= self.cachelimit: head = self.avail[k] node = head.next if node is not head: break k += k else: return None # Store is full, or block is too large node.unlink() size2 = node.size while size2 > size: size2 = size2 / 2 assert size2 >= size node.size = size2 buddy = BlockNode(None, size2, node.addr + size2) self.nodes[buddy.addr] = buddy buddy.linkbefore(self.avail[size2]) node.oid = 1 # Flag as in-use return node def free(self, node): assert node is self.nodes[node.addr] assert node.prev is node.next is None node.oid = None # Flag as free while node.size < self.cachelimit: buddy_addr = node.addr ^ node.size buddy = self.nodes[buddy_addr] assert buddy.addr == buddy_addr if buddy.oid is not None or buddy.size != node.size: break # Merge node with buddy buddy.unlink() if buddy.addr < node.addr: # buddy prevails del self.nodes[node.addr] node = buddy else: # node prevails del self.nodes[buddy.addr] node.size *= 2 assert node is self.nodes[node.addr] node.linkbefore(self.avail[node.size]) def dump(self, msg=""): if msg: print msg, size = MINSIZE blocks = bytes = 0 while size <= self.cachelimit: head = self.avail[size] node = head.next count = 0 while node is not head: count += 1 node = node.next if count: print "%d:%d" % (size, count), blocks += count bytes += count*size size += size print "-- %d, %d" % (bytes, blocks) def roundup(size): k = MINSIZE while k < size: k += k return k class SimpleAllocator: def __init__(self, arenasize): self.arenasize = arenasize self.avail = BlockNode(None, 0, 0) # Weird: empty block as list head self.rover = self.avail node = BlockNode(None, arenasize, 0) node.linkbefore(self.avail) self.taglo = {0: node} self.taghi = {arenasize: node} # Allocator statistics self.nallocs = 0 self.nfrees = 0 self.allocloops = 0 self.freebytes = arenasize self.freeblocks = 1 self.allocbytes = 0 self.allocblocks = 0 def report(self): print ("NA=%d AL=%d NF=%d ABy=%d ABl=%d FBy=%d FBl=%d" % (self.nallocs, self.allocloops, self.nfrees, self.allocbytes, self.allocblocks, self.freebytes, self.freeblocks)) def alloc(self, size): self.nallocs += 1 # First fit algorithm rover = stop = self.rover while 1: self.allocloops += 1 if rover.size >= size: break rover = rover.next if rover is stop: return None # We went round the list without finding space if rover.size == size: self.rover = rover.next rover.unlink() del self.taglo[rover.addr] del self.taghi[rover.addr + size] self.freeblocks -= 1 self.allocblocks += 1 self.freebytes -= size self.allocbytes += size return rover # Take space from the beginning of the roving pointer assert rover.size > size node = BlockNode(None, size, rover.addr) del self.taglo[rover.addr] rover.size -= size rover.addr += size self.taglo[rover.addr] = rover #self.freeblocks += 0 # No change here self.allocblocks += 1 self.freebytes -= size self.allocbytes += size return node def free(self, node): self.nfrees += 1 self.freeblocks += 1 self.allocblocks -= 1 self.freebytes += node.size self.allocbytes -= node.size node.linkbefore(self.avail) self.taglo[node.addr] = node self.taghi[node.addr + node.size] = node x = self.taghi.get(node.addr) if x is not None: # Merge x into node x.unlink() self.freeblocks -= 1 del self.taglo[x.addr] del self.taghi[x.addr + x.size] del self.taglo[node.addr] node.addr = x.addr node.size += x.size self.taglo[node.addr] = node x = self.taglo.get(node.addr + node.size) if x is not None: # Merge x into node x.unlink() self.freeblocks -= 1 del self.taglo[x.addr] del self.taghi[x.addr + x.size] del self.taghi[node.addr + node.size] node.size += x.size self.taghi[node.addr + node.size] = node # It's possible that either one of the merges above invalidated # the rover. # It's simplest to simply reset the rover to the newly freed block. self.rover = node def dump(self, msg=""): if msg: print msg, count = 0 bytes = 0 node = self.avail.next while node is not self.avail: bytes += node.size count += 1 node = node.next print count, "free blocks,", bytes, "free bytes" self.report() class BlockNode(Node): __slots__ = ['addr'] def __init__(self, oid, size, addr): Node.__init__(self, oid, size) self.addr = addr def testallocator(factory=BuddyAllocator): # Run one of Knuth's experiments as a test import random import heapq # This only runs with Python 2.3, folks :-) reportfreq = 100 cachelimit = 2**17 cache = factory(cachelimit) queue = [] T = 0 blocks = 0 while T < 5000: while queue and queue[0][0] <= T: time, node = heapq.heappop(queue) assert time == T ##print "free addr=%d, size=%d" % (node.addr, node.size) cache.free(node) blocks -= 1 size = random.randint(100, 2000) lifetime = random.randint(1, 100) node = cache.alloc(size) if node is None: print "out of mem" cache.dump("T=%4d: %d blocks;" % (T, blocks)) break else: ##print "alloc addr=%d, size=%d" % (node.addr, node.size) blocks += 1 heapq.heappush(queue, (T + lifetime, node)) T = T+1 if T % reportfreq == 0: cache.dump("T=%4d: %d blocks;" % (T, blocks)) def hitrate(loads, hits): return "%5.1f%%" % (100.0 * hits / max(1, loads)) def duration(secs): mm, ss = divmod(secs, 60) hh, mm = divmod(mm, 60) if hh: return "%d:%02d:%02d" % (hh, mm, ss) if mm: return "%d:%02d" % (mm, ss) return "%d" % ss def addcommas(n): sign, s = '', str(n) if s[0] == '-': sign, s = '-', s[1:] i = len(s) - 3 while i > 0: s = s[:i] + ',' + s[i:] i -= 3 return sign + s import random def maybe(f, p=0.5): if random.random() < p: f() ############################################################################# # Thor-like eviction scheme. # # The cache keeps a list of all objects, and uses a travelling pointer # to decay the worth of objects over time. class ThorNode(Node): __slots__ = ['worth'] def __init__(self, oid, size, worth=None): Node.__init__(self, oid, size) self.worth = worth class ThorListHead(Node): def __init__(self): Node.__init__(self, 0, 0) self.next = self.prev = self class ThorSimulation(Simulation): extras = "evicts", "trips" def __init__(self, cachelimit): Simulation.__init__(self, cachelimit) # Maximum total of object sizes we keep in cache. self.maxsize = cachelimit # Current total of object sizes in cache. self.currentsize = 0 # A worth byte maps to a set of all objects with that worth. # This is cheap to keep updated, and makes finding low-worth # objects for eviction trivial (just march over the worthsets # list, in order). self.worthsets = [Set() for dummy in range(256)] # We keep a circular list of all objects in cache. currentobj # walks around it forever. Each time _tick() is called, the # worth of currentobj is decreased, basically by shifting # right 1, and currentobj moves on to the next object. When # an object is first inserted, it enters the list right before # currentobj. When an object is accessed, its worth is # increased by or'ing in 0x80. This scheme comes from the # Thor system, and is an inexpensive way to account for both # recency and frequency of access: recency is reflected in # the leftmost bit set, and frequency by how many bits are # set. # # Note: because evictions are interleaved with ticks, # unlinking an object is tricky, lest we evict currentobj. The # class _unlink method takes care of this properly. self.listhead = ThorListHead() self.currentobj = self.listhead # Map an object.oid to its ThorNode. self.oid2object = {} self.total_evicts = self.total_trips = 0 # Unlink object from the circular list, taking care not to lose # track of the current object. Always call this instead of # invoking obj.unlink() directly. def _unlink(self, obj): assert obj is not self.listhead if obj is self.currentobj: self.currentobj = obj.next obj.unlink() # Change obj.worth to newworth, maintaining invariants. def _change_worth(self, obj, newworth): if obj.worth != newworth: self.worthsets[obj.worth].remove(obj) obj.worth = newworth self.worthsets[newworth].add(obj) def add(self, object): assert object.oid not in self.oid2object self.oid2object[object.oid] = object newsize = self.currentsize + object.size if newsize > self.maxsize: self._evictbytes(newsize - self.maxsize) self.currentsize += object.size object.linkbefore(self.currentobj) if object.worth is None: # Give smaller objects higher initial worth. This favors kicking # out unreferenced large objects before kicking out unreferenced # small objects. On real life traces, this is a significant # win for the hit rate. object.worth = 32 - int(round(math.log(object.size, 2))) self.worthsets[object.worth].add(object) # Decrease the worth of the current object, and advance the # current object. def _tick(self): c = self.currentobj if c is self.listhead: c = c.next if c is self.listhead: # list is empty return self.total_trips += 1 self.trips += 1 self._change_worth(c, (c.worth + 1) >> 1) self.currentobj = c.next def access(self, oid): self._tick() obj = self.oid2object.get(oid) if obj is None: return None self._change_worth(obj, obj.worth | 0x80) return obj # Evict objects of least worth first, until at least nbytes bytes # have been freed. def _evictbytes(self, nbytes): for s in self.worthsets: while s: if nbytes <= 0: return obj = s.pop() nbytes -= obj.size self._evictobj(obj) def _evictobj(self, obj): self.currentsize -= obj.size self.worthsets[obj.worth].discard(obj) del self.oid2object[obj.oid] self._unlink(obj) self.evicts += 1 self.total_evicts += 1 def _evict_without_bumping_evict_stats(self, obj): self._evictobj(obj) self.evicts -= 1 self.total_evicts -= 1 # Simulator overrides from here on. def restart(self): # Reset base class Simulation.restart(self) # Reset additional per-run statistics self.evicts = self.trips = 0 def write(self, oid, size): obj = self.oid2object.get(oid) worth = None if obj is not None: worth = obj.worth self._evict_without_bumping_evict_stats(obj) self.add(ThorNode(oid, size, worth)) def load(self, oid, size): if self.access(oid) is not None: self.hits += 1 self.total_hits += 1 else: self.write(oid, size) def inval(self, oid): obj = self.oid2object.get(oid) if obj is not None: self.invals += 1 self.total_invals += 1 self._evict_without_bumping_evict_stats(obj) # Take the "x" off to see additional stats after each restart period. def xreport(self): Simulation.report(self) print 'non-empty worth sets', sum(map(bool, self.worthsets)), print 'objects', len(self.oid2object), print 'size', self.currentsize ############################################################################# # Perfection: What if the cache were unbounded, and never forgot anything? # This simulator answers that question directly; the cache size parameter # isn't used. class UnboundedSimulation(Simulation): extraname = 'evicts' # for some reason we *have* to define >= 1 extra def __init__(self, cachelimit): Simulation.__init__(self, cachelimit) self.oids = Set() self.evicts = self.total_evicts = 0 def write(self, oid, size): self.oids.add(oid) def load(self, oid, size): if oid in self.oids: self.hits += 1 self.total_hits += 1 else: self.oids.add(oid) def inval(self, oid): if oid in self.oids: self.invals += 1 self.total_invals += 1 self.oids.remove(oid) if __name__ == "__main__": sys.exit(main()) Zope-2.8.7-final/lib/python/ZEO/ServerStub.py0000664000076400007640000002460610435502417017753 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """RPC stubs for interface exported by StorageServer.""" ## # ZEO storage server. #

# Remote method calls can be synchronous or asynchronous. If the call # is synchronous, the client thread blocks until the call returns. A # single client can only have one synchronous request outstanding. If # several threads share a single client, threads other than the caller # will block only if the attempt to make another synchronous call. # An asynchronous call does not cause the client thread to block. An # exception raised by an asynchronous method is logged on the server, # but is not returned to the client. class StorageServer: """An RPC stub class for the interface exported by ClientStorage. This is the interface presented by the StorageServer to the ClientStorage; i.e. the ClientStorage calls these methods and they are executed in the StorageServer. See the StorageServer module for documentation on these methods, with the exception of _update(), which is documented here. """ def __init__(self, rpc): """Constructor. The argument is a connection: an instance of the zrpc.connection.Connection class. """ self.rpc = rpc # Wait until we know what version the other side is using. while rpc.peer_protocol_version is None: rpc.pending() if rpc.peer_protocol_version == 'Z200': self.lastTransaction = lambda: None self.getInvalidations = lambda tid: None self.getAuthProtocol = lambda: None def extensionMethod(self, name): return ExtensionMethodWrapper(self.rpc, name).call ## # Register current connection with a storage and a mode. # In effect, it is like an open call. # @param storage_name a string naming the storage. This argument # is primarily for backwards compatibility with servers # that supported multiple storages. # @param read_only boolean # @exception ValueError unknown storage_name or already registered # @exception ReadOnlyError storage is read-only and a read-write # connectio was requested def register(self, storage_name, read_only): self.rpc.call('register', storage_name, read_only) ## # Return dictionary of meta-data about the storage. # @defreturn dict def get_info(self): return self.rpc.call('get_info') ## # Check whether the server requires authentication. Returns # the name of the protocol. # @defreturn string def getAuthProtocol(self): return self.rpc.call('getAuthProtocol') ## # Return id of the last committed transaction # @defreturn string def lastTransaction(self): # Not in protocol version 2.0.0; see __init__() return self.rpc.call('lastTransaction') ## # Return invalidations for all transactions after tid. # @param tid transaction id # @defreturn 2-tuple, (tid, list) # @return tuple containing the last committed transaction # and a list of oids that were invalidated. Returns # None and an empty list if the server does not have # the list of oids available. def getInvalidations(self, tid): # Not in protocol version 2.0.0; see __init__() return self.rpc.call('getInvalidations', tid) ## # Check whether serial numbers s and sv are current for oid. # If one or both of the serial numbers are not current, the # server will make an asynchronous invalidateVerify() call. # @param oid object id # @param s serial number on non-version data # @param sv serial number of version data or None # @defreturn async def zeoVerify(self, oid, s, sv): self.rpc.callAsync('zeoVerify', oid, s, sv) ## # Check whether current serial number is valid for oid and version. # If the serial number is not current, the server will make an # asynchronous invalidateVerify() call. # @param oid object id # @param version name of version for oid # @param serial client's current serial number # @defreturn async def verify(self, oid, version, serial): self.rpc.callAsync('verify', oid, version, serial) ## # Signal to the server that cache verification is done. # @defreturn async def endZeoVerify(self): self.rpc.callAsync('endZeoVerify') ## # Generate a new set of oids. # @param n number of new oids to return # @defreturn list # @return list of oids def new_oids(self, n=None): if n is None: return self.rpc.call('new_oids') else: return self.rpc.call('new_oids', n) ## # Pack the storage. # @param t pack time # @param wait optional, boolean. If true, the call will not # return until the pack is complete. def pack(self, t, wait=None): if wait is None: self.rpc.call('pack', t) else: self.rpc.call('pack', t, wait) ## # Return current data for oid. Version data is returned if # present. # @param oid object id # @defreturn 5-tuple # @return 5-tuple, current non-version data, serial number, # version name, version data, version data serial number # @exception KeyError if oid is not found def zeoLoad(self, oid): return self.rpc.call('zeoLoad', oid) ## # Return current data for oid in version, the tid of the transaction that # wrote the most recent revision, and the name of the version for the # data returned. Versions make this hard to understand; in particular, # the version string returned may not equal the version string passed # in, and that's "a feature" I don't understand. Similarly, the tid # returned is the tid of the most recent revision of oid, and that may # not equal the tid of the transaction that wrote the data returned. # @param oid object id # @param version string, name of version # @defreturn 3-tuple # @return data, transaction id, version # where version is the name of the version the data came # from or "" for non-version data # @exception KeyError if oid is not found def loadEx(self, oid, version): return self.rpc.call("loadEx", oid, version) ## # Return non-current data along with transaction ids that identify # the lifetime of the specific revision. # @param oid object id # @param tid a transaction id that provides an upper bound on # the lifetime of the revision. That is, loadBefore # returns the revision that was current before tid committed. # @defreturn 4-tuple # @return data, serial numbr, start transaction id, end transaction id def loadBefore(self, oid, tid): return self.rpc.call("loadBefore", oid, tid) ## # Storage new revision of oid. # @param oid object id # @param serial serial number that this transaction read # @param data new data record for oid # @param version name of version or "" # @param id id of current transaction # @defreturn async def storea(self, oid, serial, data, version, id): self.rpc.callAsync('storea', oid, serial, data, version, id) ## # Start two-phase commit for a transaction # @param id id used by client to identify current transaction. The # only purpose of this argument is to distinguish among multiple # threads using a single ClientStorage. # @param user name of user committing transaction (can be "") # @param description string containing transaction metadata (can be "") # @param ext dictionary of extended metadata (?) # @param tid optional explicit tid to pass to underlying storage # @param status optional status character, e.g "p" for pack # @defreturn async def tpc_begin(self, id, user, descr, ext, tid, status): return self.rpc.call('tpc_begin', id, user, descr, ext, tid, status) def vote(self, trans_id): return self.rpc.call('vote', trans_id) def tpc_finish(self, id): return self.rpc.call('tpc_finish', id) def tpc_abort(self, id): self.rpc.callAsync('tpc_abort', id) def abortVersion(self, src, id): return self.rpc.call('abortVersion', src, id) def commitVersion(self, src, dest, id): return self.rpc.call('commitVersion', src, dest, id) def history(self, oid, version, length=None): if length is None: return self.rpc.call('history', oid, version) else: return self.rpc.call('history', oid, version, length) def load(self, oid, version): return self.rpc.call('load', oid, version) def getSerial(self, oid): return self.rpc.call('getSerial', oid) def loadSerial(self, oid, serial): return self.rpc.call('loadSerial', oid, serial) def modifiedInVersion(self, oid): return self.rpc.call('modifiedInVersion', oid) def new_oid(self): return self.rpc.call('new_oid') def store(self, oid, serial, data, version, trans): return self.rpc.call('store', oid, serial, data, version, trans) def undo(self, trans_id, trans): return self.rpc.call('undo', trans_id, trans) def undoLog(self, first, last): return self.rpc.call('undoLog', first, last) def undoInfo(self, first, last, spec): return self.rpc.call('undoInfo', first, last, spec) def versionEmpty(self, vers): return self.rpc.call('versionEmpty', vers) def versions(self, max=None): if max is None: return self.rpc.call('versions') else: return self.rpc.call('versions', max) class ExtensionMethodWrapper: def __init__(self, rpc, name): self.rpc = rpc self.name = name def call(self, *a, **kwa): return self.rpc.call(self.name, *a, **kwa) Zope-2.8.7-final/lib/python/ZEO/tests/0000775000076400007640000000000010435610645016432 5ustar ajungajungZope-2.8.7-final/lib/python/ZEO/tests/speed.py0000664000076400007640000001377110435502417020112 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## usage="""Test speed of a ZODB storage Options: -d file The data file to use as input. The default is this script. -n n The number of repititions -s module A module that defines a 'Storage' attribute, which is an open storage. If not specified, a FileStorage will ne used. -z Test compressing data -D Run in debug mode -L Test loads as well as stores by minimizing the cache after eachrun -M Output means only -C Run with a persistent client cache -U Run ZEO using a Unix domain socket -t n Number of concurrent threads to run. """ import asyncore import sys, os, getopt, time ##sys.path.insert(0, os.getcwd()) import persistent import transaction import ZODB from ZODB.POSException import ConflictError from ZEO.tests import forker class P(persistent.Persistent): pass fs_name = "zeo-speed.fs" class ZEOExit(asyncore.file_dispatcher): """Used to exit ZEO.StorageServer when run is done""" def writable(self): return 0 def readable(self): return 1 def handle_read(self): buf = self.recv(4) assert buf == "done" self.delete_fs() os._exit(0) def handle_close(self): print "Parent process exited unexpectedly" self.delete_fs() os._exit(0) def delete_fs(self): os.unlink(fs_name) os.unlink(fs_name + ".lock") os.unlink(fs_name + ".tmp") def work(db, results, nrep, compress, data, detailed, minimize, threadno=None): for j in range(nrep): for r in 1, 10, 100, 1000: t = time.time() conflicts = 0 jar = db.open() while 1: try: transaction.begin() rt = jar.root() key = 's%s' % r if rt.has_key(key): p = rt[key] else: rt[key] = p =P() for i in range(r): v = getattr(p, str(i), P()) if compress is not None: v.d = compress(data) else: v.d = data setattr(p, str(i), v) transaction.commit() except ConflictError: conflicts = conflicts + 1 else: break jar.close() t = time.time() - t if detailed: if threadno is None: print "%s\t%s\t%.4f\t%d" % (j, r, t, conflicts) else: print "%s\t%s\t%.4f\t%d\t%d" % (j, r, t, conflicts, threadno) results[r].append((t, conflicts)) rt=d=p=v=None # release all references if minimize: time.sleep(3) jar.cacheMinimize() def main(args): opts, args = getopt.getopt(args, 'zd:n:Ds:LMt:U') s = None compress = None data=sys.argv[0] nrep=5 minimize=0 detailed=1 cache = None domain = 'AF_INET' threads = 1 for o, v in opts: if o=='-n': nrep = int(v) elif o=='-d': data = v elif o=='-s': s = v elif o=='-z': import zlib compress = zlib.compress elif o=='-L': minimize=1 elif o=='-M': detailed=0 elif o=='-D': global debug os.environ['STUPID_LOG_FILE']='' os.environ['STUPID_LOG_SEVERITY']='-999' debug = 1 elif o == '-C': cache = 'speed' elif o == '-U': domain = 'AF_UNIX' elif o == '-t': threads = int(v) zeo_pipe = None if s: s = __import__(s, globals(), globals(), ('__doc__',)) s = s.Storage server = None else: s, server, pid = forker.start_zeo("FileStorage", (fs_name, 1), domain=domain) data=open(data).read() db=ZODB.DB(s, # disable cache deactivation cache_size=4000, cache_deactivate_after=6000,) print "Beginning work..." results={1:[], 10:[], 100:[], 1000:[]} if threads > 1: import threading l = [] for i in range(threads): t = threading.Thread(target=work, args=(db, results, nrep, compress, data, detailed, minimize, i)) l.append(t) for t in l: t.start() for t in l: t.join() else: work(db, results, nrep, compress, data, detailed, minimize) if server is not None: server.close() os.waitpid(pid, 0) if detailed: print '-'*24 print "num\tmean\tmin\tmax" for r in 1, 10, 100, 1000: times = [] for time, conf in results[r]: times.append(time) t = mean(times) print "%d\t%.4f\t%.4f\t%.4f" % (r, t, min(times), max(times)) def mean(l): tot = 0 for v in l: tot = tot + v return tot / len(l) ##def compress(s): ## c = zlib.compressobj() ## o = c.compress(s) ## return o + c.flush() if __name__=='__main__': main(sys.argv[1:]) Zope-2.8.7-final/lib/python/ZEO/tests/Cache.py0000664000076400007640000000773510435502417020020 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests of the ZEO cache""" from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_unpickle from transaction import Transaction class TransUndoStorageWithCache: def checkUndoInvalidation(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(23)) revid = self._dostore(oid, revid=revid, data=MinPO(24)) revid = self._dostore(oid, revid=revid, data=MinPO(25)) info = self._storage.undoInfo() if not info: # Preserved this comment, but don't understand it: # "Perhaps we have an old storage implementation that # does do the negative nonsense." info = self._storage.undoInfo(0, 20) tid = info[0]['id'] # We may need to bail at this point if the storage doesn't # support transactional undo if not self._storage.supportsTransactionalUndo(): return # Now start an undo transaction t = Transaction() t.note('undo1') self._storage.tpc_begin(t) tid, oids = self._storage.undo(tid, t) # Make sure this doesn't load invalid data into the cache self._storage.load(oid, '') self._storage.tpc_vote(t) self._storage.tpc_finish(t) assert len(oids) == 1 assert oids[0] == oid data, revid = self._storage.load(oid, '') obj = zodb_unpickle(data) assert obj == MinPO(24) class StorageWithCache: def checkAbortVersionInvalidation(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(1)) revid = self._dostore(oid, revid=revid, data=MinPO(2)) revid = self._dostore(oid, revid=revid, data=MinPO(3), version="foo") revid = self._dostore(oid, revid=revid, data=MinPO(4), version="foo") t = Transaction() self._storage.tpc_begin(t) self._storage.abortVersion("foo", t) self._storage.load(oid, "foo") self._storage.tpc_vote(t) self._storage.tpc_finish(t) data, revid = self._storage.load(oid, "foo") obj = zodb_unpickle(data) assert obj == MinPO(2), obj def checkCommitEmptyVersionInvalidation(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(1)) revid = self._dostore(oid, revid=revid, data=MinPO(2)) revid = self._dostore(oid, revid=revid, data=MinPO(3), version="foo") t = Transaction() self._storage.tpc_begin(t) self._storage.commitVersion("foo", "", t) self._storage.load(oid, "") self._storage.tpc_vote(t) self._storage.tpc_finish(t) data, revid = self._storage.load(oid, "") obj = zodb_unpickle(data) assert obj == MinPO(3), obj def checkCommitVersionInvalidation(self): oid = self._storage.new_oid() revid = self._dostore(oid, data=MinPO(1)) revid = self._dostore(oid, revid=revid, data=MinPO(2)) revid = self._dostore(oid, revid=revid, data=MinPO(3), version="foo") t = Transaction() self._storage.tpc_begin(t) self._storage.commitVersion("foo", "bar", t) self._storage.load(oid, "") self._storage.tpc_vote(t) self._storage.tpc_finish(t) data, revid = self._storage.load(oid, "bar") obj = zodb_unpickle(data) assert obj == MinPO(3), obj Zope-2.8.7-final/lib/python/ZEO/tests/testZEO.py0000664000076400007640000001612410435502417020342 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test suite for ZEO based on ZODB.tests.""" # System imports import os import random import socket import asyncore import tempfile import unittest import logging # ZODB test support import ZODB from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_unpickle # ZODB test mixin classes from ZODB.tests import StorageTestBase, BasicStorage, VersionStorage, \ TransactionalUndoStorage, TransactionalUndoVersionStorage, \ PackableStorage, Synchronization, ConflictResolution, RevisionStorage, \ MTStorage, ReadOnlyStorage from ZEO.ClientStorage import ClientStorage from ZEO.tests import forker, Cache, CommitLockTests, ThreadTests logger = logging.getLogger('ZEO.tests.testZEO') class DummyDB: def invalidate(self, *args): pass class OneTimeTests(unittest.TestCase): def checkZEOVersionNumber(self): import ZEO # Starting with ZODB 3.4, the ZODB and ZEO version numbers should # be identical. self.assertEqual(ZODB.__version__, ZEO.version) class MiscZEOTests: """ZEO tests that don't fit in elsewhere.""" def checkLargeUpdate(self): obj = MinPO("X" * (10 * 128 * 1024)) self._dostore(data=obj) def checkZEOInvalidation(self): addr = self._storage._addr storage2 = ClientStorage(addr, wait=1, min_disconnect_poll=0.1) try: oid = self._storage.new_oid() ob = MinPO('first') revid1 = self._dostore(oid, data=ob) data, serial = storage2.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO('first')) self.assertEqual(serial, revid1) revid2 = self._dostore(oid, data=MinPO('second'), revid=revid1) for n in range(3): # Let the server and client talk for a moment. # Is there a better way to do this? asyncore.poll(0.1) data, serial = storage2.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO('second'), 'Invalidation message was not sent!') self.assertEqual(serial, revid2) finally: storage2.close() def get_port(): """Return a port that is not in use. Checks if a port is in use by trying to connect to it. Assumes it is not in use if connect raises an exception. Raises RuntimeError after 10 tries. """ for i in range(10): port = random.randrange(20000, 30000) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: try: s.connect(('localhost', port)) except socket.error: # Perhaps we should check value of error too. return port finally: s.close() raise RuntimeError, "Can't find port" class GenericTests( # Base class for all ZODB tests StorageTestBase.StorageTestBase, # ZODB test mixin classes (in the same order as imported) BasicStorage.BasicStorage, PackableStorage.PackableStorage, Synchronization.SynchronizedStorage, MTStorage.MTStorage, ReadOnlyStorage.ReadOnlyStorage, # ZEO test mixin classes (in the same order as imported) CommitLockTests.CommitLockVoteTests, ThreadTests.ThreadTests, # Locally defined (see above) MiscZEOTests ): """Combine tests from various origins in one class.""" def setUp(self): logger.info("setUp() %s", self.id()) port = get_port() zconf = forker.ZEOConfig(('', port)) zport, adminaddr, pid, path = forker.start_zeo_server(self.getConfig(), zconf, port) self._pids = [pid] self._servers = [adminaddr] self._conf_path = path self._storage = ClientStorage(zport, '1', cache_size=20000000, min_disconnect_poll=0.5, wait=1, wait_timeout=60) self._storage.registerDB(DummyDB(), None) def tearDown(self): self._storage.close() os.remove(self._conf_path) for server in self._servers: forker.shutdown_zeo_server(server) if hasattr(os, 'waitpid'): # Not in Windows Python until 2.3 for pid in self._pids: os.waitpid(pid, 0) def open(self, read_only=0): # Needed to support ReadOnlyStorage tests. Ought to be a # cleaner way. addr = self._storage._addr self._storage.close() self._storage = ClientStorage(addr, read_only=read_only, wait=1) def checkWriteMethods(self): # ReadOnlyStorage defines checkWriteMethods. The decision # about where to raise the read-only error was changed after # Zope 2.5 was released. So this test needs to detect Zope # of the 2.5 vintage and skip the test. # The __version__ attribute was not present in Zope 2.5. if hasattr(ZODB, "__version__"): ReadOnlyStorage.ReadOnlyStorage.checkWriteMethods(self) def checkSortKey(self): key = '%s:%s' % (self._storage._storage, self._storage._server_addr) self.assertEqual(self._storage.sortKey(), key) class FullGenericTests( GenericTests, Cache.StorageWithCache, Cache.TransUndoStorageWithCache, CommitLockTests.CommitLockUndoTests, ConflictResolution.ConflictResolvingStorage, ConflictResolution.ConflictResolvingTransUndoStorage, PackableStorage.PackableUndoStorage, RevisionStorage.RevisionStorage, TransactionalUndoStorage.TransactionalUndoStorage, TransactionalUndoVersionStorage.TransactionalUndoVersionStorage, VersionStorage.VersionStorage, ): """Extend GenericTests with tests that MappingStorage can't pass.""" class FileStorageTests(FullGenericTests): """Test ZEO backed by a FileStorage.""" level = 2 def getConfig(self): filename = self.__fs_base = tempfile.mktemp() return """\ path %s """ % filename class MappingStorageTests(GenericTests): """ZEO backed by a Mapping storage.""" def getConfig(self): return """""" test_classes = [OneTimeTests, FileStorageTests, MappingStorageTests] def test_suite(): suite = unittest.TestSuite() for klass in test_classes: sub = unittest.makeSuite(klass, "check") suite.addTest(sub) return suite if __name__ == "__main__": unittest.main(defaultTest="test_suite") Zope-2.8.7-final/lib/python/ZEO/tests/__init__.py0000664000076400007640000000120210435502417020533 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## Zope-2.8.7-final/lib/python/ZEO/tests/testMonitor.py0000664000076400007640000000540410435502417021333 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test that the monitor produce sensible results. $Id: testMonitor.py 29448 2005-03-11 23:02:55Z tim_one $ """ import socket import unittest from ZEO.tests.ConnectionTests import CommonSetupTearDown from ZEO.monitor import StorageStats class MonitorTests(CommonSetupTearDown): monitor = 1 def get_monitor_output(self): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(('localhost', 42000)) L = [] while 1: buf = s.recv(8192) if buf: L.append(buf) else: break s.close() return "".join(L) def parse(self, s): # Return a list of StorageStats, one for each storage. lines = s.split("\n") self.assert_(lines[0].startswith("ZEO monitor server")) # lines[1] is a date # Break up rest of lines into sections starting with Storage: # and ending with a blank line. sections = [] cur = None for line in lines[2:]: if line.startswith("Storage:"): cur = [line] elif line: cur.append(line) else: if cur is not None: sections.append(cur) cur = None assert cur is None # bug in the test code if this fails d = {} for sect in sections: hdr = sect[0] key, value = hdr.split(":") storage = int(value) s = d[storage] = StorageStats() s.parse("\n".join(sect[1:])) return d def getConfig(self, path, create, read_only): return """""" def testMonitor(self): # Just open a client to know that the server is up and running # TODO: should put this in setUp. self.storage = self.openClientStorage() s = self.get_monitor_output() self.storage.close() self.assert_(s.find("monitor") != -1) d = self.parse(s) stats = d[1] self.assertEqual(stats.clients, 1) self.assertEqual(stats.commits, 0) def test_suite(): return unittest.makeSuite(MonitorTests) Zope-2.8.7-final/lib/python/ZEO/tests/CommitLockTests.py0000664000076400007640000001652310435502417022074 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tests of the distributed commit lock.""" import threading import time from persistent.TimeStamp import TimeStamp import transaction from ZODB.tests.StorageTestBase import zodb_pickle, MinPO import ZEO.ClientStorage from ZEO.Exceptions import ClientDisconnected from ZEO.tests.TestThread import TestThread ZERO = '\0'*8 class DummyDB: def invalidate(self, *args, **kwargs): pass class WorkerThread(TestThread): # run the entire test in a thread so that the blocking call for # tpc_vote() doesn't hang the test suite. def __init__(self, storage, trans, method="tpc_finish"): self.storage = storage self.trans = trans self.method = method self.ready = threading.Event() TestThread.__init__(self) def testrun(self): try: self.storage.tpc_begin(self.trans) oid = self.storage.new_oid() p = zodb_pickle(MinPO("c")) self.storage.store(oid, ZERO, p, '', self.trans) oid = self.storage.new_oid() p = zodb_pickle(MinPO("c")) self.storage.store(oid, ZERO, p, '', self.trans) self.myvote() if self.method == "tpc_finish": self.storage.tpc_finish(self.trans) else: self.storage.tpc_abort(self.trans) except ClientDisconnected: pass def myvote(self): # The vote() call is synchronous, which makes it difficult to # coordinate the action of multiple threads that all call # vote(). This method sends the vote call, then sets the # event saying vote was called, then waits for the vote # response. It digs deep into the implementation of the client. # This method is a replacement for: # self.ready.set() # self.storage.tpc_vote(self.trans) rpc = self.storage._server.rpc msgid = rpc._deferred_call('vote', id(self.trans)) self.ready.set() rpc._deferred_wait(msgid) self.storage._check_serials() class CommitLockTests: NUM_CLIENTS = 5 # The commit lock tests verify that the storage successfully # blocks and restarts transactions when there is contention for a # single storage. There are a lot of cases to cover. # The general flow of these tests is to start a transaction by # getting far enough into 2PC to acquire the commit lock. Then # begin one or more other connections that also want to commit. # This causes the commit lock code to be exercised. Once the # other connections are started, the first transaction completes. def _cleanup(self): for store, trans in self._storages: store.tpc_abort(trans) store.close() self._storages = [] def _start_txn(self): txn = transaction.Transaction() self._storage.tpc_begin(txn) oid = self._storage.new_oid() self._storage.store(oid, ZERO, zodb_pickle(MinPO(1)), '', txn) return oid, txn def _begin_threads(self): # Start a second transaction on a different connection without # blocking the test thread. Returns only after each thread has # set it's ready event. self._storages = [] self._threads = [] for i in range(self.NUM_CLIENTS): storage = self._duplicate_client() txn = transaction.Transaction() tid = self._get_timestamp() t = WorkerThread(storage, txn) self._threads.append(t) t.start() t.ready.wait() # Close on the connections abnormally to test server response if i == 0: storage.close() else: self._storages.append((storage, txn)) def _finish_threads(self): for t in self._threads: t.cleanup() def _duplicate_client(self): "Open another ClientStorage to the same server." # It's hard to find the actual address. # The rpc mgr addr attribute is a list. Each element in the # list is a socket domain (AF_INET, AF_UNIX, etc.) and an # address. addr = self._storage._addr new = ZEO.ClientStorage.ClientStorage(addr, wait=1) new.registerDB(DummyDB(), None) return new def _get_timestamp(self): t = time.time() t = TimeStamp(*time.gmtime(t)[:5]+(t%60,)) return `t` class CommitLockVoteTests(CommitLockTests): def checkCommitLockVoteFinish(self): oid, txn = self._start_txn() self._storage.tpc_vote(txn) self._begin_threads() self._storage.tpc_finish(txn) self._storage.load(oid, '') self._finish_threads() self._dostore() self._cleanup() def checkCommitLockVoteAbort(self): oid, txn = self._start_txn() self._storage.tpc_vote(txn) self._begin_threads() self._storage.tpc_abort(txn) self._finish_threads() self._dostore() self._cleanup() def checkCommitLockVoteClose(self): oid, txn = self._start_txn() self._storage.tpc_vote(txn) self._begin_threads() self._storage.close() self._finish_threads() self._cleanup() class CommitLockUndoTests(CommitLockTests): def _get_trans_id(self): self._dostore() L = self._storage.undoInfo() return L[0]['id'] def _begin_undo(self, trans_id, txn): rpc = self._storage._server.rpc return rpc._deferred_call('undo', trans_id, id(txn)) def _finish_undo(self, msgid): return self._storage._server.rpc._deferred_wait(msgid) def checkCommitLockUndoFinish(self): trans_id = self._get_trans_id() oid, txn = self._start_txn() msgid = self._begin_undo(trans_id, txn) self._begin_threads() self._finish_undo(msgid) self._storage.tpc_vote(txn) self._storage.tpc_finish(txn) self._storage.load(oid, '') self._finish_threads() self._dostore() self._cleanup() def checkCommitLockUndoAbort(self): trans_id = self._get_trans_id() oid, txn = self._start_txn() msgid = self._begin_undo(trans_id, txn) self._begin_threads() self._finish_undo(msgid) self._storage.tpc_vote(txn) self._storage.tpc_abort(txn) self._finish_threads() self._dostore() self._cleanup() def checkCommitLockUndoClose(self): trans_id = self._get_trans_id() oid, txn = self._start_txn() msgid = self._begin_undo(trans_id, txn) self._begin_threads() self._finish_undo(msgid) self._storage.tpc_vote(txn) self._storage.close() self._finish_threads() self._cleanup() Zope-2.8.7-final/lib/python/ZEO/tests/testTransactionBuffer.py0000664000076400007640000000427310435502417023326 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import random import unittest from ZEO.TransactionBuffer import TransactionBuffer def random_string(size): """Return a random string of size size.""" l = [chr(random.randrange(256)) for i in range(size)] return "".join(l) def new_store_data(): """Return arbitrary data to use as argument to store() method.""" return random_string(8), '', random_string(random.randrange(1000)) def new_invalidate_data(): """Return arbitrary data to use as argument to invalidate() method.""" return random_string(8), '' class TransBufTests(unittest.TestCase): def checkTypicalUsage(self): tbuf = TransactionBuffer() tbuf.store(*new_store_data()) tbuf.invalidate(*new_invalidate_data()) for o in tbuf: pass def doUpdates(self, tbuf): data = [] for i in range(10): d = new_store_data() tbuf.store(*d) data.append(d) d = new_invalidate_data() tbuf.invalidate(*d) data.append(d) for i, x in enumerate(tbuf): if x[2] is None: # the tbuf add a dummy None to invalidates x = x[:2] self.assertEqual(x, data[i]) def checkOrderPreserved(self): tbuf = TransactionBuffer() self.doUpdates(tbuf) def checkReusable(self): tbuf = TransactionBuffer() self.doUpdates(tbuf) tbuf.clear() self.doUpdates(tbuf) tbuf.clear() self.doUpdates(tbuf) def test_suite(): return unittest.makeSuite(TransBufTests, 'check') Zope-2.8.7-final/lib/python/ZEO/tests/TestThread.py0000664000076400007640000000421510435502417021052 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A Thread base class for use with unittest.""" import threading import sys class TestThread(threading.Thread): """Base class for defining threads that run from unittest. The subclass should define a testrun() method instead of a run() method. Call cleanup() when the test is done with the thread, instead of join(). If the thread exits with an uncaught exception, it's captured and re-raised when cleanup() is called. cleanup() should be called by the main thread! Trying to tell unittest that a test failed from another thread creates a nightmare of timing-depending cascading failures and missed errors (tracebacks that show up on the screen, but don't cause unittest to believe the test failed). cleanup() also joins the thread. If the thread ended without raising an uncaught exception, and the join doesn't succeed in the timeout period, then the test is made to fail with a "Thread still alive" message. """ def __init__(self): threading.Thread.__init__(self) # In case this thread hangs, don't stop Python from exiting. self.setDaemon(1) self._exc_info = None def run(self): try: self.testrun() except: self._exc_info = sys.exc_info() def cleanup(self, timeout=15): self.join(timeout) if self._exc_info: raise self._exc_info[0], self._exc_info[1], self._exc_info[2] if self.isAlive(): self._testcase.fail("Thread did not finish: %s" % self) Zope-2.8.7-final/lib/python/ZEO/tests/stress.py0000664000076400007640000000730410435502417020330 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A ZEO client-server stress test to look for leaks. The stress test should run in an infinite loop and should involve multiple connections. """ # TODO: This code is currently broken. import transaction import ZODB from ZODB.MappingStorage import MappingStorage from ZODB.tests import MinPO from ZEO.ClientStorage import ClientStorage from ZEO.tests import forker import os import random import types NUM_TRANSACTIONS_PER_CONN = 10 NUM_CONNECTIONS = 10 NUM_ROOTS = 20 MAX_DEPTH = 20 MIN_OBJSIZE = 128 MAX_OBJSIZE = 2048 def an_object(): """Return an object suitable for a PersistentMapping key""" size = random.randrange(MIN_OBJSIZE, MAX_OBJSIZE) if os.path.exists("/dev/urandom"): f = open("/dev/urandom") buf = f.read(size) f.close() return buf else: f = open(MinPO.__file__) l = list(f.read(size)) f.close() random.shuffle(l) return "".join(l) def setup(cn): """Initialize the database with some objects""" root = cn.root() for i in range(NUM_ROOTS): prev = an_object() for j in range(random.randrange(1, MAX_DEPTH)): o = MinPO.MinPO(prev) prev = o root[an_object()] = o transaction.commit() cn.close() def work(cn): """Do some work with a transaction""" cn.sync() root = cn.root() obj = random.choice(root.values()) # walk down to the bottom while not isinstance(obj.value, types.StringType): obj = obj.value obj.value = an_object() transaction.commit() def main(): # Yuck! Need to cleanup forker so that the API is consistent # across Unix and Windows, at least if that's possible. if os.name == "nt": zaddr, tport, pid = forker.start_zeo_server('MappingStorage', ()) def exitserver(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(tport) s.close() else: zaddr = '', random.randrange(20000, 30000) pid, exitobj = forker.start_zeo_server(MappingStorage(), zaddr) def exitserver(): exitobj.close() while 1: pid = start_child(zaddr) print "started", pid os.waitpid(pid, 0) exitserver() def start_child(zaddr): pid = os.fork() if pid != 0: return pid try: _start_child(zaddr) finally: os._exit(0) def _start_child(zaddr): storage = ClientStorage(zaddr, debug=1, min_disconnect_poll=0.5, wait=1) db = ZODB.DB(storage, pool_size=NUM_CONNECTIONS) setup(db.open()) conns = [] conn_count = 0 for i in range(NUM_CONNECTIONS): c = db.open() c.__count = 0 conns.append(c) conn_count += 1 while conn_count < 25: c = random.choice(conns) if c.__count > NUM_TRANSACTIONS_PER_CONN: conns.remove(c) c.close() conn_count += 1 c = db.open() c.__count = 0 conns.append(c) else: c.__count += 1 work(c) if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/tests/InvalidationTests.py0000664000076400007640000004570310435502417022456 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import threading import time from random import Random import transaction from BTrees.check import check, display from BTrees.OOBTree import OOBTree from ZEO.tests.TestThread import TestThread from ZODB.DB import DB from ZODB.POSException \ import ReadConflictError, ConflictError, VersionLockError # The tests here let several threads have a go at one or more database # instances simultaneously. Each thread appends a disjoint (from the # other threads) sequence of increasing integers to an OOBTree, one at # at time (per thread). This provokes lots of conflicts, and BTrees # work hard at conflict resolution too. An OOBTree is used because # that flavor has the smallest maximum bucket size, and so splits buckets # more often than other BTree flavors. # # When these tests were first written, they provoked an amazing number # of obscure timing-related bugs in cache consistency logic, revealed # by failure of the BTree to pass internal consistency checks at the end, # and/or by failure of the BTree to contain all the keys the threads # thought they added (i.e., the keys for which transaction.commit() # did not raise any exception). class FailableThread(TestThread): # mixin class # subclass must provide # - self.stop attribute (an event) # - self._testrun() method # TestThread.run() invokes testrun(). def testrun(self): try: self._testrun() except: # Report the failure here to all the other threads, so # that they stop quickly. self.stop.set() raise class StressTask: # Append integers startnum, startnum + step, startnum + 2*step, ... # to 'tree'. If sleep is given, sleep # that long after each append. At the end, instance var .added_keys # is a list of the ints the thread believes it added successfully. def __init__(self, db, threadnum, startnum, step=2, sleep=None): self.db = db self.threadnum = threadnum self.startnum = startnum self.step = step self.sleep = sleep self.added_keys = [] self.tm = transaction.TransactionManager() self.cn = self.db.open(transaction_manager=self.tm) self.cn.sync() def doStep(self): tree = self.cn.root()["tree"] key = self.startnum tree[key] = self.threadnum def commit(self): cn = self.cn key = self.startnum self.tm.get().note("add key %s" % key) try: self.tm.get().commit() except ConflictError, msg: self.tm.abort() else: if self.sleep: time.sleep(self.sleep) self.added_keys.append(key) self.startnum += self.step def cleanup(self): self.tm.get().abort() self.cn.close() def _runTasks(rounds, *tasks): '''run *task* interleaved for *rounds* rounds.''' def commit(run, actions): actions.append(':') for t in run: t.commit() del run[:] r = Random() r.seed(1064589285) # make it deterministic run = [] actions = [] try: for i in range(rounds): t = r.choice(tasks) if t in run: commit(run, actions) run.append(t) t.doStep() actions.append(`t.startnum`) commit(run,actions) # stderr.write(' '.join(actions)+'\n') finally: for t in tasks: t.cleanup() class StressThread(FailableThread): # Append integers startnum, startnum + step, startnum + 2*step, ... # to 'tree' until Event stop is set. If sleep is given, sleep # that long after each append. At the end, instance var .added_keys # is a list of the ints the thread believes it added successfully. def __init__(self, db, stop, threadnum, commitdict, startnum, step=2, sleep=None): TestThread.__init__(self) self.db = db self.stop = stop self.threadnum = threadnum self.startnum = startnum self.step = step self.sleep = sleep self.added_keys = [] self.commitdict = commitdict def _testrun(self): cn = self.db.open() while not self.stop.isSet(): try: tree = cn.root()["tree"] break except (ConflictError, KeyError): transaction.abort() key = self.startnum while not self.stop.isSet(): try: tree[key] = self.threadnum transaction.get().note("add key %s" % key) transaction.commit() self.commitdict[self] = 1 if self.sleep: time.sleep(self.sleep) except (ReadConflictError, ConflictError), msg: transaction.abort() else: self.added_keys.append(key) key += self.step cn.close() class LargeUpdatesThread(FailableThread): # A thread that performs a lot of updates. It attempts to modify # more than 25 objects so that it can test code that runs vote # in a separate thread when it modifies more than 25 objects. def __init__(self, db, stop, threadnum, commitdict, startnum, step=2, sleep=None): TestThread.__init__(self) self.db = db self.stop = stop self.threadnum = threadnum self.startnum = startnum self.step = step self.sleep = sleep self.added_keys = [] self.commitdict = commitdict def _testrun(self): cn = self.db.open() while not self.stop.isSet(): try: tree = cn.root()["tree"] break except (ConflictError, KeyError): # print "%d getting tree abort" % self.threadnum transaction.abort() keys_added = {} # set of keys we commit tkeys = [] while not self.stop.isSet(): # The test picks 50 keys spread across many buckets. # self.startnum and self.step ensure that all threads use # disjoint key sets, to minimize conflict errors. nkeys = len(tkeys) if nkeys < 50: tkeys = range(self.startnum, 3000, self.step) nkeys = len(tkeys) step = max(int(nkeys / 50), 1) keys = [tkeys[i] for i in range(0, nkeys, step)] for key in keys: try: tree[key] = self.threadnum except (ReadConflictError, ConflictError), msg: # print "%d setting key %s" % (self.threadnum, msg) transaction.abort() break else: # print "%d set #%d" % (self.threadnum, len(keys)) transaction.get().note("keys %s" % ", ".join(map(str, keys))) try: transaction.commit() self.commitdict[self] = 1 if self.sleep: time.sleep(self.sleep) except ConflictError, msg: # print "%d commit %s" % (self.threadnum, msg) transaction.abort() continue for k in keys: tkeys.remove(k) keys_added[k] = 1 self.added_keys = keys_added.keys() cn.close() class VersionStressThread(FailableThread): def __init__(self, db, stop, threadnum, commitdict, startnum, step=2, sleep=None): TestThread.__init__(self) self.db = db self.stop = stop self.threadnum = threadnum self.startnum = startnum self.step = step self.sleep = sleep self.added_keys = [] self.commitdict = commitdict def _testrun(self): commit = 0 key = self.startnum while not self.stop.isSet(): version = "%s:%s" % (self.threadnum, key) commit = not commit if self.oneupdate(version, key, commit): self.added_keys.append(key) self.commitdict[self] = 1 key += self.step def oneupdate(self, version, key, commit=1): # The mess of sleeps below were added to reduce the number # of VersionLockErrors, based on empirical observation. # It looks like the threads don't switch enough without # the sleeps. cn = self.db.open(version) while not self.stop.isSet(): try: tree = cn.root()["tree"] break except (ConflictError, KeyError): transaction.abort() while not self.stop.isSet(): try: tree[key] = self.threadnum transaction.commit() if self.sleep: time.sleep(self.sleep) break except (VersionLockError, ReadConflictError, ConflictError), msg: transaction.abort() if self.sleep: time.sleep(self.sleep) try: while not self.stop.isSet(): try: if commit: self.db.commitVersion(version) transaction.get().note("commit version %s" % version) else: self.db.abortVersion(version) transaction.get().note("abort version %s" % version) transaction.commit() if self.sleep: time.sleep(self.sleep) return commit except ConflictError, msg: transaction.abort() finally: cn.close() return 0 class InvalidationTests: level = 2 # Minimum # of seconds the main thread lets the workers run. The # test stops as soon as this much time has elapsed, and all threads # have managed to commit a change. MINTIME = 10 # Maximum # of seconds the main thread lets the workers run. We # stop after this long has elapsed regardless of whether all threads # have managed to commit a change. MAXTIME = 300 StressThread = StressThread def _check_tree(self, cn, tree): # Make sure the BTree is sane at the C level. retries = 3 while retries: retries -= 1 try: check(tree) tree._check() except ReadConflictError: if retries: transaction.abort() else: raise except: display(tree) raise def _check_threads(self, tree, *threads): # Make sure the thread's view of the world is consistent with # the actual database state. expected_keys = [] errormsgs = [] err = errormsgs.append for t in threads: if not t.added_keys: err("thread %d didn't add any keys" % t.threadnum) expected_keys.extend(t.added_keys) expected_keys.sort() actual_keys = list(tree.keys()) if expected_keys != actual_keys: err("expected keys != actual keys") for k in expected_keys: if k not in actual_keys: err("key %s expected but not in tree" % k) for k in actual_keys: if k not in expected_keys: err("key %s in tree but not expected" % k) if errormsgs: display(tree) self.fail('\n'.join(errormsgs)) def go(self, stop, commitdict, *threads): # Run the threads for t in threads: t.start() delay = self.MINTIME start = time.time() while time.time() - start <= self.MAXTIME: stop.wait(delay) if stop.isSet(): # Some thread failed. Stop right now. break delay = 2.0 if len(commitdict) >= len(threads): break # Some thread still hasn't managed to commit anything. stop.set() # Give all the threads some time to stop before trying to clean up. # cleanup() will cause the test to fail if some thread ended with # an uncaught exception, and unittest will call the base class # tearDown then immediately, but if other threads are still # running that can lead to a cascade of spurious exceptions. for t in threads: t.join(10) for t in threads: t.cleanup() def checkConcurrentUpdates2Storages_emulated(self): self._storage = storage1 = self.openClientStorage() storage2 = self.openClientStorage() db1 = DB(storage1) db2 = DB(storage2) cn = db1.open() tree = cn.root()["tree"] = OOBTree() transaction.commit() # DM: allow time for invalidations to come in and process them time.sleep(0.1) # Run two threads that update the BTree t1 = StressTask(db1, 1, 1,) t2 = StressTask(db2, 2, 2,) _runTasks(100, t1, t2) cn.sync() self._check_tree(cn, tree) self._check_threads(tree, t1, t2) cn.close() db1.close() db2.close() def checkConcurrentUpdates2Storages(self): self._storage = storage1 = self.openClientStorage() storage2 = self.openClientStorage() db1 = DB(storage1) db2 = DB(storage2) stop = threading.Event() cn = db1.open() tree = cn.root()["tree"] = OOBTree() transaction.commit() cn.close() # Run two threads that update the BTree cd = {} t1 = self.StressThread(db1, stop, 1, cd, 1) t2 = self.StressThread(db2, stop, 2, cd, 2) self.go(stop, cd, t1, t2) while db1.lastTransaction() != db2.lastTransaction(): db1._storage.sync() db2._storage.sync() cn = db1.open() tree = cn.root()["tree"] self._check_tree(cn, tree) self._check_threads(tree, t1, t2) cn.close() db1.close() db2.close() def checkConcurrentUpdates1Storage(self): self._storage = storage1 = self.openClientStorage() db1 = DB(storage1) stop = threading.Event() cn = db1.open() tree = cn.root()["tree"] = OOBTree() transaction.commit() cn.close() # Run two threads that update the BTree cd = {} t1 = self.StressThread(db1, stop, 1, cd, 1, sleep=0.01) t2 = self.StressThread(db1, stop, 2, cd, 2, sleep=0.01) self.go(stop, cd, t1, t2) cn = db1.open() tree = cn.root()["tree"] self._check_tree(cn, tree) self._check_threads(tree, t1, t2) cn.close() db1.close() def checkConcurrentUpdates2StoragesMT(self): self._storage = storage1 = self.openClientStorage() db1 = DB(storage1) db2 = DB(self.openClientStorage()) stop = threading.Event() cn = db1.open() tree = cn.root()["tree"] = OOBTree() transaction.commit() cn.close() # Run three threads that update the BTree. # Two of the threads share a single storage so that it # is possible for both threads to read the same object # at the same time. cd = {} t1 = self.StressThread(db1, stop, 1, cd, 1, 3) t2 = self.StressThread(db2, stop, 2, cd, 2, 3, 0.01) t3 = self.StressThread(db2, stop, 3, cd, 3, 3, 0.01) self.go(stop, cd, t1, t2, t3) while db1.lastTransaction() != db2.lastTransaction(): db1._storage.sync() db2._storage.sync() cn = db1.open() tree = cn.root()["tree"] self._check_tree(cn, tree) self._check_threads(tree, t1, t2, t3) cn.close() db1.close() db2.close() # TODO: Temporarily disabled. I know it fails, and there's no point # getting an endless number of reports about that. def xxxcheckConcurrentUpdatesInVersions(self): self._storage = storage1 = self.openClientStorage() db1 = DB(storage1) db2 = DB(self.openClientStorage()) stop = threading.Event() cn = db1.open() tree = cn.root()["tree"] = OOBTree() transaction.commit() cn.close() # Run three threads that update the BTree. # Two of the threads share a single storage so that it # is possible for both threads to read the same object # at the same time. cd = {} t1 = VersionStressThread(db1, stop, 1, cd, 1, 3) t2 = VersionStressThread(db2, stop, 2, cd, 2, 3, 0.01) t3 = VersionStressThread(db2, stop, 3, cd, 3, 3, 0.01) self.go(stop, cd, t1, t2, t3) while db1.lastTransaction() != db2.lastTransaction(): db1._storage.sync() db2._storage.sync() cn = db1.open() tree = cn.root()["tree"] self._check_tree(cn, tree) self._check_threads(tree, t1, t2, t3) cn.close() db1.close() db2.close() def checkConcurrentLargeUpdates(self): # Use 3 threads like the 2StorageMT test above. self._storage = storage1 = self.openClientStorage() db1 = DB(storage1) db2 = DB(self.openClientStorage()) stop = threading.Event() cn = db1.open() tree = cn.root()["tree"] = OOBTree() for i in range(0, 3000, 2): tree[i] = 0 transaction.commit() cn.close() # Run three threads that update the BTree. # Two of the threads share a single storage so that it # is possible for both threads to read the same object # at the same time. cd = {} t1 = LargeUpdatesThread(db1, stop, 1, cd, 1, 3, 0.02) t2 = LargeUpdatesThread(db2, stop, 2, cd, 2, 3, 0.01) t3 = LargeUpdatesThread(db2, stop, 3, cd, 3, 3, 0.01) self.go(stop, cd, t1, t2, t3) while db1.lastTransaction() != db2.lastTransaction(): db1._storage.sync() db2._storage.sync() cn = db1.open() tree = cn.root()["tree"] self._check_tree(cn, tree) # Purge the tree of the dummy entries mapping to 0. losers = [k for k, v in tree.items() if v == 0] for k in losers: del tree[k] transaction.commit() self._check_threads(tree, t1, t2, t3) cn.close() db1.close() db2.close() Zope-2.8.7-final/lib/python/ZEO/tests/testZEOOptions.py0000664000076400007640000000730010435502417021712 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test suite for ZEO.runzeo.ZEOOptions.""" import os import sys import tempfile import unittest import ZODB.config from ZEO.runzeo import ZEOOptions from zdaemon.tests.testzdoptions import TestZDOptions # When a hostname isn't specified in a socket binding address, ZConfig # supplies the empty string. DEFAULT_BINDING_HOST = "" class TestZEOOptions(TestZDOptions): OptionsClass = ZEOOptions input_args = ["-f", "Data.fs", "-a", "5555"] output_opts = [("-f", "Data.fs"), ("-a", "5555")] output_args = [] configdata = """ address 5555 path Data.fs """ def setUp(self): self.tempfilename = tempfile.mktemp() f = open(self.tempfilename, "w") f.write(self.configdata) f.close() def tearDown(self): try: os.remove(self.tempfilename) except os.error: pass def test_configure(self): # Hide the base class test_configure pass def test_defaults_with_schema(self): options = self.OptionsClass() options.realize(["-C", self.tempfilename]) self.assertEqual(options.address, (DEFAULT_BINDING_HOST, 5555)) self.assertEqual(len(options.storages), 1) opener = options.storages[0] self.assertEqual(opener.name, "fs") self.assertEqual(opener.__class__, ZODB.config.FileStorage) self.assertEqual(options.read_only, 0) self.assertEqual(options.transaction_timeout, None) self.assertEqual(options.invalidation_queue_size, 100) def test_defaults_without_schema(self): options = self.OptionsClass() options.realize(["-a", "5555", "-f", "Data.fs"]) self.assertEqual(options.address, (DEFAULT_BINDING_HOST, 5555)) self.assertEqual(len(options.storages), 1) opener = options.storages[0] self.assertEqual(opener.name, "1") self.assertEqual(opener.__class__, ZODB.config.FileStorage) self.assertEqual(opener.config.path, "Data.fs") self.assertEqual(options.read_only, 0) self.assertEqual(options.transaction_timeout, None) self.assertEqual(options.invalidation_queue_size, 100) def test_commandline_overrides(self): options = self.OptionsClass() options.realize(["-C", self.tempfilename, "-a", "6666", "-f", "Wisdom.fs"]) self.assertEqual(options.address, (DEFAULT_BINDING_HOST, 6666)) self.assertEqual(len(options.storages), 1) opener = options.storages[0] self.assertEqual(opener.__class__, ZODB.config.FileStorage) self.assertEqual(opener.config.path, "Wisdom.fs") self.assertEqual(options.read_only, 0) self.assertEqual(options.transaction_timeout, None) self.assertEqual(options.invalidation_queue_size, 100) def test_suite(): suite = unittest.TestSuite() for cls in [TestZEOOptions]: suite.addTest(unittest.makeSuite(cls)) return suite if __name__ == "__main__": unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZEO/tests/ConnectionTests.py0000664000076400007640000012422210435502417022126 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os import sys import time import random import asyncore import tempfile import threading import logging import ZEO.ServerStub from ZEO.ClientStorage import ClientStorage from ZEO.Exceptions import ClientDisconnected from ZEO.zrpc.marshal import Marshaller from ZEO.tests import forker from ZODB.DB import DB from ZODB.POSException import ReadOnlyError, ConflictError from ZODB.tests.StorageTestBase import StorageTestBase from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase \ import zodb_pickle, zodb_unpickle, handle_all_serials, handle_serials import transaction from transaction import Transaction logger = logging.getLogger('ZEO.tests.ConnectionTests') ZERO = '\0'*8 class TestServerStub(ZEO.ServerStub.StorageServer): __super_getInvalidations = ZEO.ServerStub.StorageServer.getInvalidations def getInvalidations(self, tid): # squirrel the results away for inspection by test case self._last_invals = self.__super_getInvalidations(tid) return self._last_invals class TestClientStorage(ClientStorage): test_connection = False StorageServerStubClass = TestServerStub def verify_cache(self, stub): self.end_verify = threading.Event() self.verify_result = ClientStorage.verify_cache(self, stub) def endVerify(self): ClientStorage.endVerify(self) self.end_verify.set() def testConnection(self, conn): try: return ClientStorage.testConnection(self, conn) finally: self.test_connection = True class DummyDB: def invalidate(self, *args, **kwargs): pass class CommonSetupTearDown(StorageTestBase): """Common boilerplate""" __super_setUp = StorageTestBase.setUp __super_tearDown = StorageTestBase.tearDown keep = 0 invq = None timeout = None monitor = 0 db_class = DummyDB def setUp(self): """Test setup for connection tests. This starts only one server; a test may start more servers by calling self._newAddr() and then self.startServer(index=i) for i in 1, 2, ... """ self.__super_setUp() logging.info("setUp() %s", self.id()) self.file = tempfile.mktemp() self.addr = [] self._pids = [] self._servers = [] self.conf_paths = [] self.caches = [] self._newAddr() self.startServer() def tearDown(self): """Try to cause the tests to halt""" logging.info("tearDown() %s" % self.id()) for p in self.conf_paths: os.remove(p) if getattr(self, '_storage', None) is not None: self._storage.close() if hasattr(self._storage, 'cleanup'): logging.debug("cleanup storage %s" % self._storage.__name__) self._storage.cleanup() for adminaddr in self._servers: if adminaddr is not None: forker.shutdown_zeo_server(adminaddr) if hasattr(os, 'waitpid'): # Not in Windows Python until 2.3 for pid in self._pids: os.waitpid(pid, 0) for c in self.caches: for i in 0, 1: for ext in "", ".trace": base = "%s-%s.zec%s" % (c, "1", ext) path = os.path.join(tempfile.tempdir, base) # On Windows before 2.3, we don't have a way to wait for # the spawned server(s) to close, and they inherited # file descriptors for our open files. So long as those # processes are alive, we can't delete the files. Try # a few times then give up. need_to_delete = False if os.path.exists(path): need_to_delete = True for dummy in range(5): try: os.unlink(path) except: time.sleep(0.5) else: need_to_delete = False break if need_to_delete: os.unlink(path) # sometimes this is just gonna fail self.__super_tearDown() def _newAddr(self): self.addr.append(self._getAddr()) def _getAddr(self): # port+1 is also used, so only draw even port numbers return 'localhost', random.randrange(25000, 30000, 2) def getConfig(self, path, create, read_only): raise NotImplementedError cache_id = 1 def openClientStorage(self, cache=None, cache_size=200000, wait=1, read_only=0, read_only_fallback=0, username=None, password=None, realm=None): if cache is None: cache = str(self.__class__.cache_id) self.__class__.cache_id += 1 self.caches.append(cache) storage = TestClientStorage(self.addr, client=cache, var=tempfile.tempdir, cache_size=cache_size, wait=wait, min_disconnect_poll=0.1, read_only=read_only, read_only_fallback=read_only_fallback, username=username, password=password, realm=realm) storage.registerDB(DummyDB(), None) return storage def getServerConfig(self, addr, ro_svr): zconf = forker.ZEOConfig(addr) if ro_svr: zconf.read_only = 1 if self.monitor: zconf.monitor_address = ("", 42000) if self.invq: zconf.invalidation_queue_size = self.invq if self.timeout: zconf.transaction_timeout = self.timeout return zconf def startServer(self, create=1, index=0, read_only=0, ro_svr=0, keep=None): addr = self.addr[index] logging.info("startServer(create=%d, index=%d, read_only=%d) @ %s" % (create, index, read_only, addr)) path = "%s.%d" % (self.file, index) sconf = self.getConfig(path, create, read_only) zconf = self.getServerConfig(addr, ro_svr) if keep is None: keep = self.keep zeoport, adminaddr, pid, path = forker.start_zeo_server( sconf, zconf, addr[1], keep) self.conf_paths.append(path) self._pids.append(pid) self._servers.append(adminaddr) def shutdownServer(self, index=0): logging.info("shutdownServer(index=%d) @ %s" % (index, self._servers[index])) adminaddr = self._servers[index] if adminaddr is not None: forker.shutdown_zeo_server(adminaddr) self._servers[index] = None def pollUp(self, timeout=30.0, storage=None): if storage is None: storage = self._storage # Poll until we're connected. now = time.time() giveup = now + timeout while not storage.is_connected(): asyncore.poll(0.1) now = time.time() if now > giveup: self.fail("timed out waiting for storage to connect") # When the socket map is empty, poll() returns immediately, # and this is a pure busy-loop then. At least on some Linux # flavors, that can starve the thread trying to connect, # leading to grossly increased runtime (typical) or bogus # "timed out" failures. A little sleep here cures both. time.sleep(0.1) def pollDown(self, timeout=30.0): # Poll until we're disconnected. now = time.time() giveup = now + timeout while self._storage.is_connected(): asyncore.poll(0.1) now = time.time() if now > giveup: self.fail("timed out waiting for storage to disconnect") # See pollUp() for why we sleep a little here. time.sleep(0.1) class ConnectionTests(CommonSetupTearDown): """Tests that explicitly manage the server process. To test the cache or re-connection, these test cases explicit start and stop a ZEO storage server. """ def checkMultipleAddresses(self): for i in range(4): self._newAddr() self._storage = self.openClientStorage('test', 100000) oid = self._storage.new_oid() obj = MinPO(12) self._dostore(oid, data=obj) self._storage.close() def checkMultipleServers(self): # Crude test-- just start two servers and do a commit at each one. self._newAddr() self._storage = self.openClientStorage('test', 100000) self._dostore() self.shutdownServer(index=0) self.startServer(index=1) # If we can still store after shutting down one of the # servers, we must be reconnecting to the other server. did_a_store = 0 for i in range(10): try: self._dostore() did_a_store = 1 break except ClientDisconnected: time.sleep(0.5) self.assert_(did_a_store) self._storage.close() def checkReadOnlyClient(self): # Open a read-only client to a read-write server; stores fail # Start a read-only client for a read-write server self._storage = self.openClientStorage(read_only=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) self._storage.close() def checkReadOnlyServer(self): # Open a read-only client to a read-only *server*; stores fail # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Start a read-only server self.startServer(create=0, index=0, ro_svr=1) # Start a read-only client self._storage = self.openClientStorage(read_only=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) self._storage.close() # Get rid of the 'test left new threads behind' warning time.sleep(0.1) def checkReadOnlyFallbackWritable(self): # Open a fallback client to a read-write server; stores succeed # Start a read-only-fallback client for a read-write server self._storage = self.openClientStorage(read_only_fallback=1) # Stores should succeed here self._dostore() self._storage.close() def checkReadOnlyFallbackReadOnlyServer(self): # Open a fallback client to a read-only *server*; stores fail # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Start a read-only server self.startServer(create=0, index=0, ro_svr=1) # Start a read-only-fallback client self._storage = self.openClientStorage(read_only_fallback=1) self.assert_(self._storage.isReadOnly()) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) self._storage.close() # TODO: Compare checkReconnectXXX() here to checkReconnection() # further down. Is the code here hopelessly naive, or is # checkReconnection() overwrought? def checkReconnectWritable(self): # A read-write client reconnects to a read-write server # Start a client self._storage = self.openClientStorage() # Stores should succeed here self._dostore() # Shut down the server self.shutdownServer() self._servers = [] # Poll until the client disconnects self.pollDown() # Stores should fail now self.assertRaises(ClientDisconnected, self._dostore) # Restart the server self.startServer(create=0) # Poll until the client connects self.pollUp() # Stores should succeed here self._dostore() self._storage.close() def checkDisconnectionError(self): # Make sure we get a ClientDisconnected when we try to read an # object when we're not connected to a storage server and the # object is not in the cache. self.shutdownServer() self._storage = self.openClientStorage('test', 1000, wait=0) self.assertRaises(ClientDisconnected, self._storage.load, 'fredwash', '') self._storage.close() def checkDisconnectedAbort(self): self._storage = self.openClientStorage() self._dostore() oids = [self._storage.new_oid() for i in range(5)] txn = Transaction() self._storage.tpc_begin(txn) for oid in oids: data = zodb_pickle(MinPO(oid)) self._storage.store(oid, None, data, '', txn) self.shutdownServer() self.assertRaises(ClientDisconnected, self._storage.tpc_vote, txn) self._storage.tpc_abort(txn) self.startServer(create=0) self._storage._wait() self._dostore() # This test is supposed to cover the following error, although # I don't have much confidence that it does. The likely # explanation for the error is that the _tbuf contained # objects that weren't in the _seriald, because the client was # interrupted waiting for tpc_vote() to return. When the next # transaction committed, it tried to do something with the # bogus _tbuf entries. The explanation is wrong/incomplete, # because tpc_begin() should clear the _tbuf. # 2003-01-15T15:44:19 ERROR(200) ZODB A storage error occurred # in the last phase of a two-phase commit. This shouldn't happen. # Traceback (innermost last): # Module ZODB.Transaction, line 359, in _finish_one # Module ZODB.Connection, line 691, in tpc_finish # Module ZEO.ClientStorage, line 679, in tpc_finish # Module ZEO.ClientStorage, line 709, in _update_cache # KeyError: ... def checkBasicPersistence(self): # Verify cached data persists across client storage instances. # To verify that the cache is being used, the test closes the # server and then starts a new client with the server down. # When the server is down, a load() gets the data from its cache. self._storage = self.openClientStorage('test', 100000) oid = self._storage.new_oid() obj = MinPO(12) revid1 = self._dostore(oid, data=obj) self._storage.close() self.shutdownServer() self._storage = self.openClientStorage('test', 100000, wait=0) data, revid2 = self._storage.load(oid, '') self.assertEqual(zodb_unpickle(data), MinPO(12)) self.assertEqual(revid1, revid2) self._storage.close() def checkDisconnectedCacheWorks(self): # Check that the cache works when the client is disconnected. self._storage = self.openClientStorage('test') oid1 = self._storage.new_oid() obj1 = MinPO("1" * 500) self._dostore(oid1, data=obj1) oid2 = self._storage.new_oid() obj2 = MinPO("2" * 500) self._dostore(oid2, data=obj2) expected1 = self._storage.load(oid1, '') expected2 = self._storage.load(oid2, '') # Shut it all down, and try loading from the persistent cache file # without a server present. self._storage.close() self.shutdownServer() self._storage = self.openClientStorage('test', wait=False) self.assertEqual(expected1, self._storage.load(oid1, '')) self.assertEqual(expected2, self._storage.load(oid2, '')) self._storage.close() def checkDisconnectedCacheFails(self): # Like checkDisconnectedCacheWorks above, except the cache # file is so small that only one object can be remembered. self._storage = self.openClientStorage('test', cache_size=900) oid1 = self._storage.new_oid() obj1 = MinPO("1" * 500) self._dostore(oid1, data=obj1) oid2 = self._storage.new_oid() obj2 = MinPO("2" * 500) # The cache file is so small that adding oid2 will evict oid1. self._dostore(oid2, data=obj2) expected2 = self._storage.load(oid2, '') # Shut it all down, and try loading from the persistent cache file # without a server present. self._storage.close() self.shutdownServer() self._storage = self.openClientStorage('test', cache_size=900, wait=False) # oid2 should still be in cache. self.assertEqual(expected2, self._storage.load(oid2, '')) # But oid1 should have been purged, so that trying to load it will # try to fetch it from the (non-existent) ZEO server. self.assertRaises(ClientDisconnected, self._storage.load, oid1, '') self._storage.close() def checkVerificationInvalidationPersists(self): # This tests a subtle invalidation bug from ZODB 3.3: # invalidations processed as part of ZEO cache verification acted # kinda OK wrt the in-memory cache structures, but had no effect # on the cache file. So opening the file cache again could # incorrectly believe that a previously invalidated object was # still current. This takes some effort to set up. # First, using a persistent cache ('test'), create an object # MinPO(13). We used to see this again at the end of this test, # despite that we modify it, and despite that it gets invalidated # in 'test', before the end. self._storage = self.openClientStorage('test') oid = self._storage.new_oid() obj = MinPO(13) self._dostore(oid, data=obj) self._storage.close() # Now modify obj via a temp connection. `test` won't learn about # this until we open a connection using `test` again. self._storage = self.openClientStorage() pickle, rev = self._storage.load(oid, '') newobj = zodb_unpickle(pickle) self.assertEqual(newobj, obj) newobj.value = 42 # .value *should* be 42 forever after now, not 13 self._dostore(oid, data=newobj, revid=rev) self._storage.close() # Open 'test' again. `oid` in this cache should be (and is) # invalidated during cache verification. The bug was that it # got invalidated (kinda) in memory, but not in the cache file. self._storage = self.openClientStorage('test') # The invalidation happened already. Now create and store a new # object before closing this storage: this is so `test` believes # it's seen transactions beyond the one that invalidated `oid`, so # that the *next* time we open `test` it doesn't process another # invalidation for `oid`. It's also important that we not try to # load `oid` now: because it's been (kinda) invalidated in the # cache's memory structures, loading it now would fetch the # current revision from the server, thus hiding the bug. obj2 = MinPO(666) oid2 = self._storage.new_oid() self._dostore(oid2, data=obj2) self._storage.close() # Finally, open `test` again and load `oid`. `test` believes # it's beyond the transaction that modified `oid`, so its view # of whether it has an up-to-date `oid` comes solely from the disk # file, unaffected by cache verification. self._storage = self.openClientStorage('test') pickle, rev = self._storage.load(oid, '') newobj_copy = zodb_unpickle(pickle) # This used to fail, with # AssertionError: MinPO(13) != MinPO(42) # That is, `test` retained a stale revision of the object on disk. self.assertEqual(newobj_copy, newobj) self._storage.close() def checkReconnection(self): # Check that the client reconnects when a server restarts. self._storage = self.openClientStorage() oid = self._storage.new_oid() obj = MinPO(12) self._dostore(oid, data=obj) logging.info("checkReconnection(): About to shutdown server") self.shutdownServer() logging.info("checkReconnection(): About to restart server") self.startServer(create=0) oid = self._storage.new_oid() obj = MinPO(12) while 1: try: self._dostore(oid, data=obj) break except ClientDisconnected: # Maybe the exception mess is better now logging.info("checkReconnection(): Error after" " server restart; retrying.", exc_info=True) transaction.abort() # Give the other thread a chance to run. time.sleep(0.1) logging.info("checkReconnection(): finished") self._storage.close() def checkBadMessage1(self): # not even close to a real message self._bad_message("salty") def checkBadMessage2(self): # just like a real message, but with an unpicklable argument global Hack class Hack: pass msg = Marshaller().encode(1, 0, "foo", (Hack(),)) self._bad_message(msg) del Hack def _bad_message(self, msg): # Establish a connection, then send the server an ill-formatted # request. Verify that the connection is closed and that it is # possible to establish a new connection. self._storage = self.openClientStorage() self._dostore() # break into the internals to send a bogus message zrpc_conn = self._storage._server.rpc zrpc_conn.message_output(msg) try: self._dostore() except ClientDisconnected: pass else: self._storage.close() self.fail("Server did not disconnect after bogus message") self._storage.close() self._storage = self.openClientStorage() self._dostore() self._storage.close() # Test case for multiple storages participating in a single # transaction. This is not really a connection test, but it needs # about the same infrastructure (several storage servers). # TODO: with the current ZEO code, this occasionally fails. # That's the point of this test. :-) def NOcheckMultiStorageTransaction(self): # Configuration parameters (larger values mean more likely deadlocks) N = 2 # These don't *have* to be all the same, but it's convenient this way self.nservers = N self.nthreads = N self.ntrans = N self.nobj = N # Start extra servers for i in range(1, self.nservers): self._newAddr() self.startServer(index=i) # Spawn threads that each do some transactions on all storages threads = [] try: for i in range(self.nthreads): t = MSTThread(self, "T%d" % i) threads.append(t) t.start() # Wait for all threads to finish for t in threads: t.join(60) self.failIf(t.isAlive(), "%s didn't die" % t.getName()) finally: for t in threads: t.closeclients() def checkCrossDBInvalidations(self): db1 = DB(self.openClientStorage()) c1 = db1.open() r1 = c1.root() r1["a"] = MinPO("a") transaction.commit() self.assertEqual(r1._p_state, 0) # up-to-date db2 = DB(self.openClientStorage()) r2 = db2.open().root() self.assertEqual(r2["a"].value, "a") r2["b"] = MinPO("b") transaction.commit() # Make sure the invalidation is received in the other client. # We've had problems with this timing out on "slow" and/or "very # busy" machines, so we increase the sleep time on each trip, and # are willing to wait quite a long time. for i in range(20): c1.sync() if r1._p_state == -1: break time.sleep(i / 10.0) self.assertEqual(r1._p_state, -1) # ghost r1.keys() # unghostify self.assertEqual(r1._p_serial, r2._p_serial) self.assertEqual(r1["b"].value, "b") db2.close() db1.close() class InvqTests(CommonSetupTearDown): invq = 3 def checkQuickVerificationWith2Clients(self): perstorage = self.openClientStorage(cache="test") self.assertEqual(perstorage.verify_result, "full verification") self._storage = self.openClientStorage() oid = self._storage.new_oid() oid2 = self._storage.new_oid() # When we create a new storage, it should always do a full # verification self.assertEqual(self._storage.verify_result, "full verification") # do two storages of the object to make sure an invalidation # message is generated revid = self._dostore(oid) revid = self._dostore(oid, revid) # Create a second object and revision to guarantee it doesn't # show up in the list of invalidations sent when perstore restarts. revid2 = self._dostore(oid2) revid2 = self._dostore(oid2, revid2) # sync() is needed to prevent invalidation for oid from arriving # in the middle of the load() call. perstorage.sync() perstorage.load(oid, '') perstorage.close() revid = self._dostore(oid, revid) perstorage = self.openClientStorage(cache="test") self.assertEqual(perstorage.verify_result, "quick verification") self.assertEqual(perstorage._server._last_invals, (revid, [(oid, '')])) self.assertEqual(perstorage.load(oid, ''), self._storage.load(oid, '')) perstorage.close() def checkVerificationWith2ClientsInvqOverflow(self): perstorage = self.openClientStorage(cache="test") self.assertEqual(perstorage.verify_result, "full verification") self._storage = self.openClientStorage() oid = self._storage.new_oid() # When we create a new storage, it should always do a full # verification self.assertEqual(self._storage.verify_result, "full verification") # do two storages of the object to make sure an invalidation # message is generated revid = self._dostore(oid) revid = self._dostore(oid, revid) perstorage.load(oid, '') perstorage.close() # the test code sets invq bound to 2 for i in range(5): revid = self._dostore(oid, revid) perstorage = self.openClientStorage(cache="test") self.assertEqual(perstorage.verify_result, "full verification") t = time.time() + 30 while not perstorage.end_verify.isSet(): perstorage.sync() if time.time() > t: self.fail("timed out waiting for endVerify") self.assertEqual(self._storage.load(oid, '')[1], revid) self.assertEqual(perstorage.load(oid, ''), self._storage.load(oid, '')) perstorage.close() class ReconnectionTests(CommonSetupTearDown): # The setUp() starts a server automatically. In order for its # state to persist, we set the class variable keep to 1. In # order for its state to be cleaned up, the last startServer() # call in the test must pass keep=0. keep = 1 invq = 2 def checkReadOnlyStorage(self): # Open a read-only client to a read-only *storage*; stores fail # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Start a read-only server self.startServer(create=0, index=0, read_only=1, keep=0) # Start a read-only client self._storage = self.openClientStorage(read_only=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) def checkReadOnlyFallbackReadOnlyStorage(self): # Open a fallback client to a read-only *storage*; stores fail # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Start a read-only server self.startServer(create=0, index=0, read_only=1, keep=0) # Start a read-only-fallback client self._storage = self.openClientStorage(read_only_fallback=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) def checkReconnectReadOnly(self): # A read-only client reconnects from a read-write to a # read-only server # Start a client self._storage = self.openClientStorage(read_only=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) # Shut down the server self.shutdownServer() self._servers = [] # Poll until the client disconnects self.pollDown() # Stores should still fail self.assertRaises(ReadOnlyError, self._dostore) # Restart the server self.startServer(create=0, read_only=1, keep=0) # Poll until the client connects self.pollUp() # Stores should still fail self.assertRaises(ReadOnlyError, self._dostore) def checkReconnectFallback(self): # A fallback client reconnects from a read-write to a # read-only server # Start a client in fallback mode self._storage = self.openClientStorage(read_only_fallback=1) # Stores should succeed here self._dostore() # Shut down the server self.shutdownServer() self._servers = [] # Poll until the client disconnects self.pollDown() # Stores should fail now self.assertRaises(ClientDisconnected, self._dostore) # Restart the server self.startServer(create=0, read_only=1, keep=0) # Poll until the client connects self.pollUp() # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) def checkReconnectUpgrade(self): # A fallback client reconnects from a read-only to a # read-write server # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Start a read-only server self.startServer(create=0, read_only=1) # Start a client in fallback mode self._storage = self.openClientStorage(read_only_fallback=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) # Shut down the server self.shutdownServer() self._servers = [] # Poll until the client disconnects self.pollDown() # Stores should fail now self.assertRaises(ClientDisconnected, self._dostore) # Restart the server, this time read-write self.startServer(create=0, keep=0) # Poll until the client sconnects self.pollUp() # Stores should now succeed self._dostore() def checkReconnectSwitch(self): # A fallback client initially connects to a read-only server, # then discovers a read-write server and switches to that # We don't want the read-write server created by setUp() self.shutdownServer() self._servers = [] # Allocate a second address (for the second server) self._newAddr() # Start a read-only server self.startServer(create=0, index=0, read_only=1, keep=0) # Start a client in fallback mode self._storage = self.openClientStorage(read_only_fallback=1) # Stores should fail here self.assertRaises(ReadOnlyError, self._dostore) # Start a read-write server self.startServer(index=1, read_only=0, keep=0) # After a while, stores should work for i in range(300): # Try for 30 seconds try: self._dostore() break except (ClientDisconnected, ReadOnlyError): # If the client isn't connected at all, sync() returns # quickly and the test fails because it doesn't wait # long enough for the client. time.sleep(0.1) else: self.fail("Couldn't store after starting a read-write server") def checkNoVerificationOnServerRestart(self): self._storage = self.openClientStorage() # When we create a new storage, it should always do a full # verification self.assertEqual(self._storage.verify_result, "full verification") self._dostore() self.shutdownServer() self.pollDown() self._storage.verify_result = None self.startServer(create=0, keep=0) self.pollUp() # There were no transactions committed, so no verification # should be needed. self.assertEqual(self._storage.verify_result, "no verification") def checkNoVerificationOnServerRestartWith2Clients(self): perstorage = self.openClientStorage(cache="test") self.assertEqual(perstorage.verify_result, "full verification") self._storage = self.openClientStorage() oid = self._storage.new_oid() # When we create a new storage, it should always do a full # verification self.assertEqual(self._storage.verify_result, "full verification") # do two storages of the object to make sure an invalidation # message is generated revid = self._dostore(oid) self._dostore(oid, revid) perstorage.load(oid, '') self.shutdownServer() self.pollDown() self._storage.verify_result = None perstorage.verify_result = None logging.info('2ALLBEEF') self.startServer(create=0, keep=0) self.pollUp() self.pollUp(storage=perstorage) # There were no transactions committed, so no verification # should be needed. self.assertEqual(self._storage.verify_result, "no verification") self.assertEqual(perstorage.verify_result, "no verification") perstorage.close() self._storage.close() class TimeoutTests(CommonSetupTearDown): timeout = 1 def checkTimeout(self): storage = self.openClientStorage() txn = Transaction() storage.tpc_begin(txn) storage.tpc_vote(txn) time.sleep(2) self.assertRaises(ClientDisconnected, storage.tpc_finish, txn) storage.close() def checkTimeoutOnAbort(self): storage = self.openClientStorage() txn = Transaction() storage.tpc_begin(txn) storage.tpc_vote(txn) storage.tpc_abort(txn) storage.close() def checkTimeoutOnAbortNoLock(self): storage = self.openClientStorage() txn = Transaction() storage.tpc_begin(txn) storage.tpc_abort(txn) storage.close() def checkTimeoutAfterVote(self): raises = self.assertRaises unless = self.failUnless self._storage = storage = self.openClientStorage() # Assert that the zeo cache is empty unless(not list(storage._cache.contents())) # Create the object oid = storage.new_oid() obj = MinPO(7) # Now do a store, sleeping before the finish so as to cause a timeout t = Transaction() storage.tpc_begin(t) revid1 = storage.store(oid, ZERO, zodb_pickle(obj), '', t) storage.tpc_vote(t) # Now sleep long enough for the storage to time out time.sleep(3) storage.sync() unless(not storage.is_connected()) storage._wait() unless(storage.is_connected()) # We expect finish to fail raises(ClientDisconnected, storage.tpc_finish, t) # The cache should still be empty unless(not list(storage._cache.contents())) # Load should fail since the object should not be in either the cache # or the server. raises(KeyError, storage.load, oid, '') def checkTimeoutProvokingConflicts(self): eq = self.assertEqual raises = self.assertRaises require = self.assert_ self._storage = storage = self.openClientStorage() # Assert that the zeo cache is empty. require(not list(storage._cache.contents())) # Create the object oid = storage.new_oid() obj = MinPO(7) # We need to successfully commit an object now so we have something to # conflict about. t = Transaction() storage.tpc_begin(t) revid1a = storage.store(oid, ZERO, zodb_pickle(obj), '', t) revid1b = storage.tpc_vote(t) revid1 = handle_serials(oid, revid1a, revid1b) storage.tpc_finish(t) # Now do a store, sleeping before the finish so as to cause a timeout. obj.value = 8 t = Transaction() storage.tpc_begin(t) revid2a = storage.store(oid, revid1, zodb_pickle(obj), '', t) revid2b = storage.tpc_vote(t) revid2 = handle_serials(oid, revid2a, revid2b) # Now sleep long enough for the storage to time out. # This used to sleep for 3 seconds, and sometimes (but very rarely) # failed then. Now we try for a minute. It typically succeeds # on the second time thru the loop, and, since self.timeout is 1, # it's typically faster now (2/1.8 ~= 1.11 seconds sleeping instead # of 3). deadline = time.time() + 60 # wait up to a minute while time.time() < deadline: if storage.is_connected(): time.sleep(self.timeout / 1.8) storage.sync() else: break storage.sync() require(not storage.is_connected()) storage._wait() require(storage.is_connected()) # We expect finish to fail. raises(ClientDisconnected, storage.tpc_finish, t) # Now we think we've committed the second transaction, but we really # haven't. A third one should produce a POSKeyError on the server, # which manifests as a ConflictError on the client. obj.value = 9 t = Transaction() storage.tpc_begin(t) storage.store(oid, revid2, zodb_pickle(obj), '', t) raises(ConflictError, storage.tpc_vote, t) # Even aborting won't help. storage.tpc_abort(t) storage.tpc_finish(t) # Try again. obj.value = 10 t = Transaction() storage.tpc_begin(t) storage.store(oid, revid2, zodb_pickle(obj), '', t) # Even aborting won't help. raises(ConflictError, storage.tpc_vote, t) # Abort this one and try a transaction that should succeed. storage.tpc_abort(t) storage.tpc_finish(t) # Now do a store. obj.value = 11 t = Transaction() storage.tpc_begin(t) revid2a = storage.store(oid, revid1, zodb_pickle(obj), '', t) revid2b = storage.tpc_vote(t) revid2 = handle_serials(oid, revid2a, revid2b) storage.tpc_finish(t) # Now load the object and verify that it has a value of 11. data, revid = storage.load(oid, '') eq(zodb_unpickle(data), MinPO(11)) eq(revid, revid2) class MSTThread(threading.Thread): __super_init = threading.Thread.__init__ def __init__(self, testcase, name): self.__super_init(name=name) self.testcase = testcase self.clients = [] def run(self): tname = self.getName() testcase = self.testcase # Create client connections to each server clients = self.clients for i in range(len(testcase.addr)): c = testcase.openClientStorage(addr=testcase.addr[i]) c.__name = "C%d" % i clients.append(c) for i in range(testcase.ntrans): # Because we want a transaction spanning all storages, # we can't use _dostore(). This is several _dostore() calls # expanded in-line (mostly). # Create oid->serial mappings for c in clients: c.__oids = [] c.__serials = {} # Begin a transaction t = Transaction() for c in clients: #print "%s.%s.%s begin\n" % (tname, c.__name, i), c.tpc_begin(t) for j in range(testcase.nobj): for c in clients: # Create and store a new object on each server oid = c.new_oid() c.__oids.append(oid) data = MinPO("%s.%s.t%d.o%d" % (tname, c.__name, i, j)) #print data.value data = zodb_pickle(data) s = c.store(oid, ZERO, data, '', t) c.__serials.update(handle_all_serials(oid, s)) # Vote on all servers and handle serials for c in clients: #print "%s.%s.%s vote\n" % (tname, c.__name, i), s = c.tpc_vote(t) c.__serials.update(handle_all_serials(None, s)) # Finish on all servers for c in clients: #print "%s.%s.%s finish\n" % (tname, c.__name, i), c.tpc_finish(t) for c in clients: # Check that we got serials for all oids for oid in c.__oids: testcase.failUnless(c.__serials.has_key(oid)) # Check that we got serials for no other oids for oid in c.__serials.keys(): testcase.failUnless(oid in c.__oids) def closeclients(self): # Close clients opened by run() for c in self.clients: try: c.close() except: pass Zope-2.8.7-final/lib/python/ZEO/tests/deadlock.py0000664000076400007640000000214110435502417020545 0ustar ajungajungimport ZODB from ZODB.POSException import ConflictError from ZEO.ClientStorage import ClientStorage, ClientDisconnected from ZEO.zrpc.error import DisconnectedError import os import random import time L = range(1, 100) def main(): z1 = ClientStorage(('localhost', 2001), wait=1) z2 = ClientStorage(('localhost', 2002), wait=2) db1 = ZODB.DB(z1) db2 = ZODB.DB(z2) c1 = db1.open() c2 = db2.open() r1 = c1.root() r2 = c2.root() while 1: try: try: update(r1, r2) except ConflictError, msg: print msg transaction.abort() c1.sync() c2.sync() except (ClientDisconnected, DisconnectedError), err: print "disconnected", err time.sleep(2) def update(r1, r2): k1 = random.choice(L) k2 = random.choice(L) updates = [(k1, r1), (k2, r2)] random.shuffle(updates) for key, root in updates: root[key] = time.time() transaction.commit() print os.getpid(), k1, k2 if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/tests/testAuth.py0000664000076400007640000001143710435502417020610 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test suite for AuthZEO.""" import os import tempfile import time import unittest from ZEO import zeopasswd from ZEO.Exceptions import ClientDisconnected from ZEO.tests.ConnectionTests import CommonSetupTearDown class AuthTest(CommonSetupTearDown): __super_getServerConfig = CommonSetupTearDown.getServerConfig __super_setUp = CommonSetupTearDown.setUp __super_tearDown = CommonSetupTearDown.tearDown realm = None def setUp(self): self.pwfile = tempfile.mktemp() if self.realm: self.pwdb = self.dbclass(self.pwfile, self.realm) else: self.pwdb = self.dbclass(self.pwfile) self.pwdb.add_user("foo", "bar") self.pwdb.save() self._checkZEOpasswd() self.__super_setUp() def _checkZEOpasswd(self): args = ["-f", self.pwfile, "-p", self.protocol] if self.protocol == "plaintext": from ZEO.auth.base import Database zeopasswd.main(args + ["-d", "foo"], Database) zeopasswd.main(args + ["foo", "bar"], Database) else: zeopasswd.main(args + ["-d", "foo"]) zeopasswd.main(args + ["foo", "bar"]) def tearDown(self): self.__super_tearDown() os.remove(self.pwfile) def getConfig(self, path, create, read_only): return "" def getServerConfig(self, addr, ro_svr): zconf = self.__super_getServerConfig(addr, ro_svr) zconf.authentication_protocol = self.protocol zconf.authentication_database = self.pwfile zconf.authentication_realm = self.realm return zconf def wait(self): for i in range(25): time.sleep(0.1) if self._storage.test_connection: return self.fail("Timed out waiting for client to authenticate") def testOK(self): # Sleep for 0.2 seconds to give the server some time to start up # seems to be needed before and after creating the storage self._storage = self.openClientStorage(wait=0, username="foo", password="bar", realm=self.realm) self.wait() self.assert_(self._storage._connection) self._storage._connection.poll() self.assert_(self._storage.is_connected()) # Make a call to make sure the mechanism is working self._storage.versions() def testNOK(self): self._storage = self.openClientStorage(wait=0, username="foo", password="noogie", realm=self.realm) self.wait() # If the test established a connection, then it failed. self.failIf(self._storage._connection) def testUnauthenticatedMessage(self): # Test that an unauthenticated message is rejected by the server # if it was sent after the connection was authenticated. # Sleep for 0.2 seconds to give the server some time to start up # seems to be needed before and after creating the storage self._storage = self.openClientStorage(wait=0, username="foo", password="bar", realm=self.realm) self.wait() self._storage.versions() # Manually clear the state of the hmac connection self._storage._connection._SizedMessageAsyncConnection__hmac_send = None # Once the client stops using the hmac, it should be disconnected. self.assertRaises(ClientDisconnected, self._storage.versions) class PlainTextAuth(AuthTest): import ZEO.tests.auth_plaintext protocol = "plaintext" database = "authdb.sha" dbclass = ZEO.tests.auth_plaintext.Database realm = "Plaintext Realm" class DigestAuth(AuthTest): import ZEO.auth.auth_digest protocol = "digest" database = "authdb.digest" dbclass = ZEO.auth.auth_digest.DigestDatabase realm = "Digest Realm" test_classes = [PlainTextAuth, DigestAuth] def test_suite(): suite = unittest.TestSuite() for klass in test_classes: sub = unittest.makeSuite(klass) suite.addTest(sub) return suite if __name__ == "__main__": unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZEO/tests/auth_plaintext.py0000664000076400007640000000403310435502417022032 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Implements plaintext password authentication. The password is stored in an SHA hash in the Database. The client sends over the plaintext password, and the SHA hashing is done on the server side. This mechanism offers *no network security at all*; the only security is provided by not storing plaintext passwords on disk. """ import sha from ZEO.StorageServer import ZEOStorage from ZEO.auth import register_module from ZEO.auth.base import Client, Database def session_key(username, realm, password): return sha.new("%s:%s:%s" % (username, realm, password)).hexdigest() class StorageClass(ZEOStorage): def auth(self, username, password): try: dbpw = self.database.get_password(username) except LookupError: return 0 password_dig = sha.new(password).hexdigest() if dbpw == password_dig: self.connection.setSessionKey(session_key(username, self.database.realm, password)) return self.finish_auth(dbpw == password_dig) class PlaintextClient(Client): extensions = ["auth"] def start(self, username, realm, password): if self.stub.auth(username, password): return session_key(username, realm, password) else: return None register_module("plaintext", StorageClass, PlaintextClient, Database) Zope-2.8.7-final/lib/python/ZEO/tests/forker.py0000664000076400007640000001531710435502417020300 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Library for forking storage server and connecting client storage""" import os import sys import time import errno import socket import logging import StringIO import tempfile import logging logger = logging.getLogger('ZEO.tests.forker') class ZEOConfig: """Class to generate ZEO configuration file. """ def __init__(self, addr): self.address = addr self.read_only = None self.invalidation_queue_size = None self.monitor_address = None self.transaction_timeout = None self.authentication_protocol = None self.authentication_database = None self.authentication_realm = None def dump(self, f): print >> f, "" print >> f, "address %s:%s" % self.address if self.read_only is not None: print >> f, "read-only", self.read_only and "true" or "false" if self.invalidation_queue_size is not None: print >> f, "invalidation-queue-size", self.invalidation_queue_size if self.monitor_address is not None: print >> f, "monitor-address %s:%s" % self.monitor_address if self.transaction_timeout is not None: print >> f, "transaction-timeout", self.transaction_timeout if self.authentication_protocol is not None: print >> f, "authentication-protocol", self.authentication_protocol if self.authentication_database is not None: print >> f, "authentication-database", self.authentication_database if self.authentication_realm is not None: print >> f, "authentication-realm", self.authentication_realm print >> f, "" logger = logging.getLogger() print >> f print >> f, "" print >> f, "level", logger.level for handler in logger.handlers: if isinstance(handler, logging.FileHandler): path = handler.baseFilename elif isinstance(handler, logging.StreamHandler): stream = handler.stream if stream.name == "": path = "STDOUT" elif stream.name == "": path = "STDERR" else: # just drop it on the floor; unlikely an issue when testing continue else: # just drop it on the floor; unlikely an issue when testing continue # This doesn't convert the level values to names, so the # generated configuration isn't as nice as it could be, # but it doesn't really need to be. print >> f, "" print >> f, "level", handler.level print >> f, "path ", path if handler.formatter: formatter = handler.formatter if formatter._fmt: print >> f, "format", encode_format(formatter._fmt) if formatter.datefmt: print >> f, "dateformat", encode_format(formatter.datefmt) print >> f, "" print >> f, "" def __str__(self): f = StringIO.StringIO() self.dump(f) return f.getvalue() def encode_format(fmt): # The list of replacements mirrors # ZConfig.components.logger.handlers._control_char_rewrites for xform in (("\n", r"\n"), ("\t", r"\t"), ("\b", r"\b"), ("\f", r"\f"), ("\r", r"\r")): fmt = fmt.replace(*xform) return fmt def start_zeo_server(storage_conf, zeo_conf, port, keep=0): """Start a ZEO server in a separate process. Takes two positional arguments a string containing the storage conf and a ZEOConfig object. Returns the ZEO port, the test server port, the pid, and the path to the config file. """ # Store the config info in a temp file. tmpfile = tempfile.mktemp(".conf") fp = open(tmpfile, 'w') zeo_conf.dump(fp) fp.write(storage_conf) fp.close() # Find the zeoserver script import ZEO.tests.zeoserver script = ZEO.tests.zeoserver.__file__ if script.endswith('.pyc'): script = script[:-1] # Create a list of arguments, which we'll tuplify below qa = _quote_arg args = [qa(sys.executable), qa(script), '-C', qa(tmpfile)] if keep: args.append("-k") d = os.environ.copy() d['PYTHONPATH'] = os.pathsep.join(sys.path) pid = os.spawnve(os.P_NOWAIT, sys.executable, tuple(args), d) adminaddr = ('localhost', port + 1) # We need to wait until the server starts, but not forever. # 30 seconds is a somewhat arbitrary upper bound. A BDBStorage # takes a long time to open -- more than 10 seconds on occasion. for i in range(120): time.sleep(0.25) try: logger.debug('connect %s', i) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(adminaddr) ack = s.recv(1024) s.close() logging.debug('acked: %s' % ack) break except socket.error, e: if e[0] not in (errno.ECONNREFUSED, errno.ECONNRESET): raise s.close() else: logging.debug('boo hoo') raise return ('localhost', port), adminaddr, pid, tmpfile if sys.platform[:3].lower() == "win": def _quote_arg(s): return '"%s"' % s else: def _quote_arg(s): return s def shutdown_zeo_server(adminaddr): # Do this in a loop to guard against the possibility that the # client failed to connect to the adminaddr earlier. That really # only requires two iterations, but do a third for pure # superstition. for i in range(3): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: s.connect(adminaddr) except socket.error, e: if e[0] == errno.ECONNREFUSED and i > 0: break raise try: ack = s.recv(1024) except socket.error, e: if e[0] == errno.ECONNRESET: raise ack = 'no ack received' logger.debug('shutdown_zeo_server(): acked: %s' % ack) s.close() Zope-2.8.7-final/lib/python/ZEO/tests/zeoserver.py0000664000076400007640000001624610435502417021036 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Helper file used to launch a ZEO server cross platform""" import os import sys import time import errno import getopt import socket import signal import asyncore import threading import logging import ThreadedAsync.LoopCallback from ZEO.StorageServer import StorageServer from ZEO.runzeo import ZEOOptions def cleanup(storage): # FileStorage and the Berkeley storages have this method, which deletes # all files and directories used by the storage. This prevents @-files # from clogging up /tmp try: storage.cleanup() except AttributeError: pass logger = logging.getLogger('ZEO.tests.zeoserver') def log(label, msg, *args): message = "(%s) %s" % (label, msg) logger.debug(message, *args) class ZEOTestServer(asyncore.dispatcher): """A server for killing the whole process at the end of a test. The first time we connect to this server, we write an ack character down the socket. The other end should block on a recv() of the socket so it can guarantee the server has started up before continuing on. The second connect to the port immediately exits the process, via os._exit(), without writing data on the socket. It does close and clean up the storage first. The other end will get the empty string from its recv() which will be enough to tell it that the server has exited. I think this should prevent us from ever getting a legitimate addr-in-use error. """ __super_init = asyncore.dispatcher.__init__ def __init__(self, addr, server, keep): self.__super_init() self._server = server self._sockets = [self] self._keep = keep # Count down to zero, the number of connects self._count = 1 self._label ='%d @ %s' % (os.getpid(), addr) self.create_socket(socket.AF_INET, socket.SOCK_STREAM) # Some ZEO tests attempt a quick start of the server using the same # port so we have to set the reuse flag. self.set_reuse_addr() try: self.bind(addr) except: # We really want to see these exceptions import traceback traceback.print_exc() raise self.listen(5) self.log('bound and listening') def log(self, msg, *args): log(self._label, msg, *args) def handle_accept(self): sock, addr = self.accept() self.log('in handle_accept()') # When we're done with everything, close the storage. Do not write # the ack character until the storage is finished closing. if self._count <= 0: self.log('closing the storage') self._server.close_server() if not self._keep: for storage in self._server.storages.values(): cleanup(storage) self.log('exiting') # Close all the other sockets so that we don't have to wait # for os._exit() to get to it before starting the next # server process. for s in self._sockets: s.close() # Now explicitly close the socket returned from accept(), # since it didn't go through the wrapper. sock.close() os._exit(0) self.log('continuing') sock.send('X') self._count -= 1 def register_socket(self, sock): # Register a socket to be closed when server shutsdown. self._sockets.append(sock) class Suicide(threading.Thread): def __init__(self, addr): threading.Thread.__init__(self) self._adminaddr = addr def run(self): # If this process doesn't exit in 330 seconds, commit suicide. # The client threads in the ConcurrentUpdate tests will run for # as long as 300 seconds. Set this timeout to 330 to minimize # chance that the server gives up before the clients. time.sleep(330) log(str(os.getpid()), "suicide thread invoking shutdown") # If the server hasn't shut down yet, the client may not be # able to connect to it. If so, try to kill the process to # force it to shutdown. if hasattr(os, "kill"): os.kill(pid, signal.SIGTERM) time.sleep(5) os.kill(pid, signal.SIGKILL) else: from ZEO.tests.forker import shutdown_zeo_server # Nott: If the -k option was given to zeoserver, then the # process will go away but the temp files won't get # cleaned up. shutdown_zeo_server(self._adminaddr) def main(): global pid pid = os.getpid() label = str(pid) log(label, "starting") # We don't do much sanity checking of the arguments, since if we get it # wrong, it's a bug in the test suite. keep = 0 configfile = None # Parse the arguments and let getopt.error percolate opts, args = getopt.getopt(sys.argv[1:], 'kC:') for opt, arg in opts: if opt == '-k': keep = 1 elif opt == '-C': configfile = arg zo = ZEOOptions() zo.realize(["-C", configfile]) zeo_port = int(zo.address[1]) if zo.auth_protocol == "plaintext": import ZEO.tests.auth_plaintext # Open the config file and let ZConfig parse the data there. Then remove # the config file, otherwise we'll leave turds. # The rest of the args are hostname, portnum test_port = zeo_port + 1 test_addr = ('localhost', test_port) addr = ('localhost', zeo_port) log(label, 'creating the storage server') storage = zo.storages[0].open() mon_addr = None if zo.monitor_address: mon_addr = zo.monitor_address server = StorageServer( zo.address, {"1": storage}, read_only=zo.read_only, invalidation_queue_size=zo.invalidation_queue_size, transaction_timeout=zo.transaction_timeout, monitor_address=mon_addr, auth_protocol=zo.auth_protocol, auth_database=zo.auth_database, auth_realm=zo.auth_realm) try: log(label, 'creating the test server, keep: %s', keep) t = ZEOTestServer(test_addr, server, keep) except socket.error, e: if e[0] <> errno.EADDRINUSE: raise log(label, 'addr in use, closing and exiting') storage.close() cleanup(storage) sys.exit(2) t.register_socket(server.dispatcher) # Create daemon suicide thread d = Suicide(test_addr) d.setDaemon(1) d.start() # Loop for socket events log(label, 'entering ThreadedAsync loop') ThreadedAsync.LoopCallback.loop() if __name__ == '__main__': main() Zope-2.8.7-final/lib/python/ZEO/tests/test_cache.py0000664000076400007640000001401410435502417021103 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Basic unit tests for a multi-version client cache.""" import os import tempfile import unittest import ZEO.cache from ZODB.utils import p64 n1 = p64(1) n2 = p64(2) n3 = p64(3) n4 = p64(4) n5 = p64(5) class CacheTests(unittest.TestCase): def setUp(self): # The default cache size is much larger than we need here. Since # testSerialization reads the entire file into a string, it's not # good to leave it that big. self.cache = ZEO.cache.ClientCache(size=1024**2) self.cache.open() def tearDown(self): if self.cache.path: os.remove(self.cache.path) def testLastTid(self): self.assertEqual(self.cache.getLastTid(), None) self.cache.setLastTid(n2) self.assertEqual(self.cache.getLastTid(), n2) self.cache.invalidate(None, "", n1) self.assertEqual(self.cache.getLastTid(), n2) self.cache.invalidate(None, "", n3) self.assertEqual(self.cache.getLastTid(), n3) self.assertRaises(ValueError, self.cache.setLastTid, n2) def testLoad(self): data1 = "data for n1" self.assertEqual(self.cache.load(n1, ""), None) self.assertEqual(self.cache.load(n1, "version"), None) self.cache.store(n1, "", n3, None, data1) self.assertEqual(self.cache.load(n1, ""), (data1, n3, "")) # The cache doesn't know whether version exists, because it # only has non-version data. self.assertEqual(self.cache.load(n1, "version"), None) self.assertEqual(self.cache.modifiedInVersion(n1), None) def testInvalidate(self): data1 = "data for n1" self.cache.store(n1, "", n3, None, data1) self.cache.invalidate(n1, "", n4) self.cache.invalidate(n2, "", n2) self.assertEqual(self.cache.load(n1, ""), None) self.assertEqual(self.cache.loadBefore(n1, n4), (data1, n3, n4)) def testVersion(self): data1 = "data for n1" data1v = "data for n1 in version" self.cache.store(n1, "version", n3, None, data1v) self.assertEqual(self.cache.load(n1, ""), None) self.assertEqual(self.cache.load(n1, "version"), (data1v, n3, "version")) self.assertEqual(self.cache.load(n1, "random"), None) self.assertEqual(self.cache.modifiedInVersion(n1), "version") self.cache.invalidate(n1, "version", n4) self.assertEqual(self.cache.load(n1, "version"), None) def testNonCurrent(self): data1 = "data for n1" data2 = "data for n2" self.cache.store(n1, "", n4, None, data1) self.cache.store(n1, "", n2, n3, data2) # can't say anything about state before n2 self.assertEqual(self.cache.loadBefore(n1, n2), None) # n3 is the upper bound of non-current record n2 self.assertEqual(self.cache.loadBefore(n1, n3), (data2, n2, n3)) # no data for between n2 and n3 self.assertEqual(self.cache.loadBefore(n1, n4), None) self.cache.invalidate(n1, "", n5) self.assertEqual(self.cache.loadBefore(n1, n5), (data1, n4, n5)) self.assertEqual(self.cache.loadBefore(n2, n4), None) def testException(self): self.assertRaises(ValueError, self.cache.store, n1, "version", n2, n3, "data") self.cache.store(n1, "", n2, None, "data") self.assertRaises(ValueError, self.cache.store, n1, "", n3, None, "data") def testEviction(self): # Manually override the current maxsize maxsize = self.cache.size = self.cache.fc.maxsize = 3395 # 1245 self.cache.fc = ZEO.cache.FileCache(3395, None, self.cache) # Trivial test of eviction code. Doesn't test non-current # eviction. data = ["z" * i for i in range(100)] for i in range(50): n = p64(i) self.cache.store(n, "", n, None, data[i]) self.assertEquals(len(self.cache), i + 1) # The cache now uses 1225 bytes. The next insert # should delete some objects. n = p64(50) self.cache.store(n, "", n, None, data[51]) self.assert_(len(self.cache) < 51) # TODO: Need to make sure eviction of non-current data # and of version data are handled correctly. def testSerialization(self): self.cache.store(n1, "", n2, None, "data for n1") self.cache.store(n2, "version", n2, None, "version data for n2") self.cache.store(n3, "", n3, n4, "non-current data for n3") self.cache.store(n3, "", n4, n5, "more non-current data for n3") path = tempfile.mktemp() # Copy data from self.cache into path, reaching into the cache # guts to make the copy. dst = open(path, "wb+") src = self.cache.fc.f src.seek(0) dst.write(src.read(self.cache.fc.maxsize)) dst.close() copy = ZEO.cache.ClientCache(path) copy.open() # Verify that internals of both objects are the same. # Could also test that external API produces the same results. eq = self.assertEqual eq(copy.getLastTid(), self.cache.getLastTid()) eq(len(copy), len(self.cache)) eq(copy.version, self.cache.version) eq(copy.current, self.cache.current) eq(copy.noncurrent, self.cache.noncurrent) def test_suite(): return unittest.makeSuite(CacheTests) Zope-2.8.7-final/lib/python/ZEO/tests/testConnection.py0000664000076400007640000000740110435502417022002 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test setup for ZEO connection logic. The actual tests are in ConnectionTests.py; this file provides the platform-dependent scaffolding. """ # System imports import unittest # Import the actual test class from ZEO.tests import ConnectionTests, InvalidationTests class FileStorageConfig: def getConfig(self, path, create, read_only): return """\ path %s create %s read-only %s """ % (path, create and 'yes' or 'no', read_only and 'yes' or 'no') class BerkeleyStorageConfig: def getConfig(self, path, create, read_only): return """\ envdir %s read-only %s """ % (path, read_only and "yes" or "no") class MappingStorageConfig: def getConfig(self, path, create, read_only): return """""" class FileStorageConnectionTests( FileStorageConfig, ConnectionTests.ConnectionTests, InvalidationTests.InvalidationTests ): """FileStorage-specific connection tests.""" level = 2 class FileStorageReconnectionTests( FileStorageConfig, ConnectionTests.ReconnectionTests, ): """FileStorage-specific re-connection tests.""" # Run this at level 1 because MappingStorage can't do reconnection tests level = 1 class FileStorageInvqTests( FileStorageConfig, ConnectionTests.InvqTests ): """FileStorage-specific invalidation queue tests.""" level = 1 class FileStorageTimeoutTests( FileStorageConfig, ConnectionTests.TimeoutTests ): level = 2 class BDBConnectionTests( BerkeleyStorageConfig, ConnectionTests.ConnectionTests, InvalidationTests.InvalidationTests ): """Berkeley storage connection tests.""" level = 2 class BDBReconnectionTests( BerkeleyStorageConfig, ConnectionTests.ReconnectionTests ): """Berkeley storage re-connection tests.""" level = 2 class BDBInvqTests( BerkeleyStorageConfig, ConnectionTests.InvqTests ): """Berkeley storage invalidation queue tests.""" level = 2 class BDBTimeoutTests( BerkeleyStorageConfig, ConnectionTests.TimeoutTests ): level = 2 class MappingStorageConnectionTests( MappingStorageConfig, ConnectionTests.ConnectionTests ): """Mapping storage connection tests.""" level = 1 # The ReconnectionTests can't work with MappingStorage because it's only an # in-memory storage and has no persistent state. class MappingStorageTimeoutTests( MappingStorageConfig, ConnectionTests.TimeoutTests ): level = 1 test_classes = [FileStorageConnectionTests, FileStorageReconnectionTests, FileStorageInvqTests, FileStorageTimeoutTests, MappingStorageConnectionTests, MappingStorageTimeoutTests] def test_suite(): suite = unittest.TestSuite() for klass in test_classes: sub = unittest.makeSuite(klass, 'check') suite.addTest(sub) return suite if __name__ == "__main__": unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZEO/tests/multi.py0000664000076400007640000001053110435502417020133 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A multi-client test of the ZEO storage server""" # TODO: This code is currently broken. import ZODB, ZODB.DB, ZODB.FileStorage, ZODB.POSException import persistent import persistent.mapping import transaction from ZEO.tests import forker import os import tempfile import time import types VERBOSE = 1 CLIENTS = 4 RECORDS_PER_CLIENT = 100 CONFLICT_DELAY = 0.1 CONNECT_DELAY = 0.1 CLIENT_CACHE = '' # use temporary cache class Record(persistent.Persistent): def __init__(self, client=None, value=None): self.client = client self.value = None self.next = None def set_next(self, next): self.next = next class Stats(persistent.Persistent): def __init__(self): self.begin = time.time() self.end = None def done(self): self.end = time.time() def init_storage(): path = tempfile.mktemp() if VERBOSE: print "FileStorage path:", path fs = ZODB.FileStorage.FileStorage(path) db = ZODB.DB(fs) root = db.open().root() root["multi"] = persistent.mapping.PersistentMapping() transaction.commit() return fs def start_server(addr): storage = init_storage() pid, exit = forker.start_zeo_server(storage, addr) return pid, exit def start_client(addr, client_func=None): pid = os.fork() if pid == 0: try: import ZEO.ClientStorage if VERBOSE: print "Client process started:", os.getpid() cli = ZEO.ClientStorage.ClientStorage(addr, client=CLIENT_CACHE) if client_func is None: run(cli) else: client_func(cli) cli.close() finally: os._exit(0) else: return pid def run(storage): if hasattr(storage, 'is_connected'): while not storage.is_connected(): time.sleep(CONNECT_DELAY) pid = os.getpid() print "Client process connected:", pid, storage db = ZODB.DB(storage) root = db.open().root() while 1: try: s = root[pid] = Stats() transaction.commit() except ZODB.POSException.ConflictError: transaction.abort() time.sleep(CONFLICT_DELAY) else: break dict = root["multi"] prev = None i = 0 while i < RECORDS_PER_CLIENT: try: size = len(dict) r = dict[size] = Record(pid, size) if prev: prev.set_next(r) transaction.commit() except ZODB.POSException.ConflictError, err: transaction.abort() time.sleep(CONFLICT_DELAY) else: i = i + 1 if VERBOSE and (i < 5 or i % 10 == 0): print "Client %s: %s of %s" % (pid, i, RECORDS_PER_CLIENT) s.done() transaction.commit() print "Client completed:", pid def main(client_func=None): if VERBOSE: print "Main process:", os.getpid() addr = tempfile.mktemp() t0 = time.time() server_pid, server = start_server(addr) t1 = time.time() pids = [] for i in range(CLIENTS): pids.append(start_client(addr, client_func)) for pid in pids: assert type(pid) == types.IntType, "invalid pid type: %s (%s)" % \ (repr(pid), type(pid)) try: if VERBOSE: print "waitpid(%s)" % repr(pid) os.waitpid(pid, 0) except os.error, err: print "waitpid(%s) failed: %s" % (repr(pid), err) t2 = time.time() server.close() os.waitpid(server_pid, 0) # TODO: Should check that the results are consistent! print "Total time:", t2 - t0 print "Server start time", t1 - t0 print "Client time:", t2 - t1 if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/tests/ThreadTests.py0000664000076400007640000001202110435502417021227 0ustar ajungajung############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Compromising positions involving threads.""" import threading import transaction from ZODB.tests.StorageTestBase import zodb_pickle, MinPO import ZEO.ClientStorage ZERO = '\0'*8 class BasicThread(threading.Thread): def __init__(self, storage, doNextEvent, threadStartedEvent): self.storage = storage self.trans = transaction.Transaction() self.doNextEvent = doNextEvent self.threadStartedEvent = threadStartedEvent self.gotValueError = 0 self.gotDisconnected = 0 threading.Thread.__init__(self) self.setDaemon(1) def join(self): threading.Thread.join(self, 10) assert not self.isAlive() class GetsThroughVoteThread(BasicThread): # This thread gets partially through a transaction before it turns # execution over to another thread. We're trying to establish that a # tpc_finish() after a storage has been closed by another thread will get # a ClientStorageError error. # # This class gets does a tpc_begin(), store(), tpc_vote() and is waiting # to do the tpc_finish() when the other thread closes the storage. def run(self): self.storage.tpc_begin(self.trans) oid = self.storage.new_oid() self.storage.store(oid, ZERO, zodb_pickle(MinPO("c")), '', self.trans) self.storage.tpc_vote(self.trans) self.threadStartedEvent.set() self.doNextEvent.wait(10) try: self.storage.tpc_finish(self.trans) except ZEO.ClientStorage.ClientStorageError: self.gotValueError = 1 self.storage.tpc_abort(self.trans) class GetsThroughBeginThread(BasicThread): # This class is like the above except that it is intended to be run when # another thread is already in a tpc_begin(). Thus, this thread will # block in the tpc_begin until another thread closes the storage. When # that happens, this one will get disconnected too. def run(self): try: self.storage.tpc_begin(self.trans) except ZEO.ClientStorage.ClientStorageError: self.gotValueError = 1 class ThreadTests: # Thread 1 should start a transaction, but not get all the way through it. # Main thread should close the connection. Thread 1 should then get # disconnected. def checkDisconnectedOnThread2Close(self): doNextEvent = threading.Event() threadStartedEvent = threading.Event() thread1 = GetsThroughVoteThread(self._storage, doNextEvent, threadStartedEvent) thread1.start() threadStartedEvent.wait(10) self._storage.close() doNextEvent.set() thread1.join() self.assertEqual(thread1.gotValueError, 1) # Thread 1 should start a transaction, but not get all the way through # it. While thread 1 is in the middle of the transaction, a second thread # should start a transaction, and it will block in the tcp_begin() -- # because thread 1 has acquired the lock in its tpc_begin(). Now the main # thread closes the storage and both sub-threads should get disconnected. def checkSecondBeginFails(self): doNextEvent = threading.Event() threadStartedEvent = threading.Event() thread1 = GetsThroughVoteThread(self._storage, doNextEvent, threadStartedEvent) thread2 = GetsThroughBeginThread(self._storage, doNextEvent, threadStartedEvent) thread1.start() threadStartedEvent.wait(1) thread2.start() self._storage.close() doNextEvent.set() thread1.join() thread2.join() self.assertEqual(thread1.gotValueError, 1) self.assertEqual(thread2.gotValueError, 1) # Run a bunch of threads doing small and large stores in parallel def checkMTStores(self): threads = [] for i in range(5): t = threading.Thread(target=self.mtstorehelper) threads.append(t) t.start() for t in threads: t.join(30) for i in threads: self.failUnless(not t.isAlive()) # Helper for checkMTStores def mtstorehelper(self): name = threading.currentThread().getName() objs = [] for i in range(10): objs.append(MinPO("X" * 200000)) objs.append(MinPO("X")) for obj in objs: self._dostore(data=obj) Zope-2.8.7-final/lib/python/ZEO/zeopasswd.py0000664000076400007640000001032110435502417017653 0ustar ajungajung#!python ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Update a user's authentication tokens for a ZEO server. usage: python zeopasswd.py [options] username [password] Specify either a configuration file: -C/--configuration -- ZConfig configuration file or the individual options: -f/--filename -- authentication database filename -p/--protocol -- authentication protocol name -r/--realm -- authentication database realm Additional options: -d/--delete -- delete user instead of updating password """ import getopt import getpass import sys import os import ZConfig import ZEO def usage(msg): print __doc__ print msg sys.exit(2) def options(args): """Password-specific options loaded from regular ZEO config file.""" try: opts, args = getopt.getopt(args, "dr:p:f:C:", ["configure=", "protocol=", "filename=", "realm"]) except getopt.error, msg: usage(msg) config = None delete = 0 auth_protocol = None auth_db = "" auth_realm = None for k, v in opts: if k == '-C' or k == '--configure': schemafile = os.path.join(os.path.dirname(ZEO.__file__), "schema.xml") schema = ZConfig.loadSchema(schemafile) config, nil = ZConfig.loadConfig(schema, v) if k == '-d' or k == '--delete': delete = 1 if k == '-p' or k == '--protocol': auth_protocol = v if k == '-f' or k == '--filename': auth_db = v if k == '-r' or k == '--realm': auth_realm = v if config is not None: if auth_protocol or auth_db: usage("Error: Conflicting options; use either -C *or* -p and -f") auth_protocol = config.zeo.authentication_protocol auth_db = config.zeo.authentication_database auth_realm = config.zeo.authentication_realm elif not (auth_protocol and auth_db): usage("Error: Must specifiy configuration file or protocol and database") password = None if delete: if not args: usage("Error: Must specify a username to delete") elif len(args) > 1: usage("Error: Too many arguments") username = args[0] else: if not args: usage("Error: Must specify a username") elif len(args) > 2: usage("Error: Too many arguments") elif len(args) == 1: username = args[0] else: username, password = args return auth_protocol, auth_db, auth_realm, delete, username, password def main(args=None, dbclass=None): p, auth_db, auth_realm, delete, username, password = options(args) if p is None: usage("Error: configuration does not specify auth protocol") if p == "digest": from ZEO.auth.auth_digest import DigestDatabase as Database elif p == "srp": from ZEO.auth.auth_srp import SRPDatabase as Database elif dbclass: # dbclass is used for testing tests.auth_plaintext, see testAuth.py Database = dbclass else: raise ValueError, "Unknown database type %r" % p if auth_db is None: usage("Error: configuration does not specify auth database") db = Database(auth_db, auth_realm) if delete: db.del_user(username) else: if password is None: password = getpass.getpass("Enter password: ") db.add_user(username, password) db.save() if __name__ == "__main__": main(sys.argv[1:]) Zope-2.8.7-final/lib/python/ZEO/zeoctl.py0000664000076400007640000000061110435502417017135 0ustar ajungajung#!/usr/bin/env python2.3 """Wrapper script for zdctl.py that causes it to use the ZEO schema.""" import os import ZEO import zdaemon.zdctl # Main program def main(args=None): options = zdaemon.zdctl.ZDCtlOptions() options.schemadir = os.path.dirname(ZEO.__file__) options.schemafile = "zeoctl.xml" zdaemon.zdctl.main(args, options) if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/DEPENDENCIES.cfg0000664000076400007640000000010110435502417017564 0ustar ajungajungBTrees ThreadedAsync ZConfig ZODB persistent transaction zdaemon Zope-2.8.7-final/lib/python/ZEO/mkzeoinst.py0000775000076400007640000001540110435502417017666 0ustar ajungajung#!python ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """%(program)s -- create a ZEO instance. Usage: %(program)s home [port] Given an "instance home directory" and some configuration options (all of which have default values), create the following: /etc/zeo.conf -- ZEO config file /var/ -- Directory for data files: Data.fs etc. /log/ -- Directory for log files: zeo.log and zeoctl.log /bin/runzeo -- the zeo server runner /bin/zeoctl -- start/stop script (a shim for zeoctl.py) The script will not overwrite existing files; instead, it will issue a warning if an existing file is found that differs from the file that would be written if it didn't exist. """ # WARNING! Several templates and functions here are reused by ZRS. # So be careful with changes. import os import sys import stat import getopt zeo_conf_template = """\ # ZEO configuration file %%define INSTANCE %(instance_home)s address %(port)d read-only false invalidation-queue-size 100 # pid-filename $INSTANCE/var/ZEO.pid # monitor-address PORT # transaction-timeout SECONDS path $INSTANCE/var/Data.fs level info path $INSTANCE/log/zeo.log program $INSTANCE/bin/runzeo socket-name $INSTANCE/etc/%(package)s.zdsock daemon true forever false backoff-limit 10 exit-codes 0, 2 directory $INSTANCE default-to-interactive true # user zope python %(python)s zdrun %(zodb3_home)s/zdaemon/zdrun.py # This logfile should match the one in the %(package)s.conf file. # It is used by zdctl's logtail command, zdrun/zdctl doesn't write it. logfile $INSTANCE/log/%(package)s.log """ zeoctl_template = """\ #!/bin/sh # %(PACKAGE)s instance control script # The following two lines are for chkconfig. On Red Hat Linux (and # some other systems), you can copy or symlink this script into # /etc/rc.d/init.d/ and then use chkconfig(8) to automatically start # %(PACKAGE)s at boot time. # chkconfig: 345 90 10 # description: start a %(PACKAGE)s server PYTHON="%(python)s" INSTANCE_HOME="%(instance_home)s" ZODB3_HOME="%(zodb3_home)s" CONFIG_FILE="%(instance_home)s/etc/%(package)s.conf" PYTHONPATH="$ZODB3_HOME" export PYTHONPATH INSTANCE_HOME ZEOCTL="$ZODB3_HOME/ZEO/zeoctl.py" exec "$PYTHON" "$ZEOCTL" -C "$CONFIG_FILE" ${1+"$@"} """ runzeo_template = """\ #!/bin/sh # %(PACKAGE)s instance start script PYTHON="%(python)s" INSTANCE_HOME="%(instance_home)s" ZODB3_HOME="%(zodb3_home)s" CONFIG_FILE="%(instance_home)s/etc/%(package)s.conf" PYTHONPATH="$ZODB3_HOME" export PYTHONPATH INSTANCE_HOME RUNZEO="$ZODB3_HOME/ZEO/runzeo.py" exec "$PYTHON" "$RUNZEO" -C "$CONFIG_FILE" ${1+"$@"} """ def main(): ZEOInstanceBuilder().run() print "All done." class ZEOInstanceBuilder: def run(self): try: opts, args = getopt.getopt(sys.argv[1:], "h", ["help"]) except getopt.error, msg: print msg sys.exit(2) program = os.path.basename(sys.argv[0]) if opts: # There's only the help options, so just dump some help: msg = __doc__ % {"program": program} print msg sys.exit() if len(args) not in [1, 2]: print "Usage: %s home [port]" % program sys.exit(2) instance_home = args[0] if not os.path.isabs(instance_home): instance_home = os.path.abspath(instance_home) for entry in sys.path: if os.path.exists(os.path.join(entry, 'ZODB')): zodb3_home = entry break else: print "Can't find the Zope home (not in sys.path)" sys.exit(2) if args[1:]: port = int(args[1]) else: port = 8100 # match example in zope.conf params = self.get_params(zodb3_home, instance_home, port) self.create(instance_home, params) def get_params(self, zodb3_home, instance_home, port): return { "package": "zeo", "PACKAGE": "ZEO", "zodb3_home": zodb3_home, "instance_home": instance_home, "port": port, "python": sys.executable, } def create(self, home, params): makedir(home) makedir(home, "etc") makedir(home, "var") makedir(home, "log") makedir(home, "bin") makefile(zeo_conf_template, home, "etc", "zeo.conf", **params) makexfile(zeoctl_template, home, "bin", "zeoctl", **params) makexfile(runzeo_template, home, "bin", "runzeo", **params) def which(program): strpath = os.getenv("PATH") binpath = strpath.split(os.pathsep) for dir in binpath: path = os.path.join(dir, program) if os.path.isfile(path) and os.access(path, os.X_OK): if not os.path.isabs(path): path = os.path.abspath(path) return path raise IOError, "can't find %r on path %r" % (program, strpath) def makedir(*args): path = "" for arg in args: path = os.path.join(path, arg) mkdirs(path) return path def mkdirs(path): if os.path.isdir(path): return head, tail = os.path.split(path) if head and tail and not os.path.isdir(head): mkdirs(head) os.mkdir(path) print "Created directory", path def makefile(template, *args, **kwds): path = makedir(*args[:-1]) path = os.path.join(path, args[-1]) data = template % kwds if os.path.exists(path): f = open(path) olddata = f.read().strip() f.close() if olddata: if olddata != data.strip(): print "Warning: not overwriting existing file %r" % path return path f = open(path, "w") f.write(data) f.close() print "Wrote file", path return path def makexfile(template, *args, **kwds): path = makefile(template, *args, **kwds) umask = os.umask(022) os.umask(umask) mode = 0777 & ~umask if stat.S_IMODE(os.stat(path)[stat.ST_MODE]) != mode: os.chmod(path, mode) print "Changed mode for %s to %o" % (path, mode) return path if __name__ == "__main__": main() Zope-2.8.7-final/lib/python/ZEO/schema.xml0000664000076400007640000000223110435502417017245 0ustar ajungajung This schema describes the configuration of the ZEO storage server process.

One or more storages that are provided by the ZEO server. The section names are used as the storage names, and must be unique within each ZEO storage server. Traditionally, these names represent small integers starting at '1'.
Zope-2.8.7-final/lib/python/ZEO/StorageServer.py0000664000076400007640000011276310435502417020444 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """The StorageServer class and the exception that it may raise. This server acts as a front-end for one or more real storages, like file storage or Berkeley storage. TODO: Need some basic access control-- a declaration of the methods exported for invocation by the server. """ import asyncore import cPickle import os import sys import threading import time import logging import transaction from ZEO import ClientStub from ZEO.CommitLog import CommitLog from ZEO.monitor import StorageStats, StatsServer from ZEO.zrpc.server import Dispatcher from ZEO.zrpc.connection import ManagedServerConnection, Delay, MTDelay from ZEO.zrpc.trigger import trigger from ZEO.Exceptions import AuthError from ZODB.ConflictResolution import ResolvedSerial from ZODB.POSException import StorageError, StorageTransactionError from ZODB.POSException import TransactionError, ReadOnlyError, ConflictError from ZODB.serialize import referencesf from ZODB.utils import u64, oid_repr from ZODB.loglevels import BLATHER logger = logging.getLogger('ZEO.StorageServer') # TODO: This used to say "ZSS", which is now implied in the logger name. # Can this be either set to str(os.getpid()) (if that makes sense) or removed? _label = "" # default label used for logging. def set_label(): """Internal helper to reset the logging label (e.g. after fork()).""" global _label _label = "%s" % os.getpid() def log(message, level=logging.INFO, label=None, exc_info=False): """Internal helper to log a message.""" label = label or _label if label: message = "(%s) %s" % (label, message) logger.log(level, message, exc_info=exc_info) class StorageServerError(StorageError): """Error reported when an unpickleable exception is raised.""" class ZEOStorage: """Proxy to underlying storage for a single remote client.""" # Classes we instantiate. A subclass might override. ClientStorageStubClass = ClientStub.ClientStorage # A list of extension methods. A subclass with extra methods # should override. extensions = [] def __init__(self, server, read_only=0, auth_realm=None): self.server = server # timeout and stats will be initialized in register() self.timeout = None self.stats = None self.connection = None self.client = None self.storage = None self.storage_id = "uninitialized" self.transaction = None self.read_only = read_only self.locked = 0 self.verifying = 0 self.store_failed = 0 self.log_label = _label self.authenticated = 0 self.auth_realm = auth_realm # The authentication protocol may define extra methods. self._extensions = {} for func in self.extensions: self._extensions[func.func_name] = None def finish_auth(self, authenticated): if not self.auth_realm: return 1 self.authenticated = authenticated return authenticated def set_database(self, database): self.database = database def notifyConnected(self, conn): self.connection = conn # For restart_other() below self.client = self.ClientStorageStubClass(conn) addr = conn.addr if isinstance(addr, type("")): label = addr else: host, port = addr label = str(host) + ":" + str(port) self.log_label = _label + "/" + label def notifyDisconnected(self): # When this storage closes, we must ensure that it aborts # any pending transaction. if self.transaction is not None: self.log("disconnected during transaction %s" % self.transaction) self._abort() else: self.log("disconnected") if self.stats is not None: self.stats.clients -= 1 def __repr__(self): tid = self.transaction and repr(self.transaction.id) if self.storage: stid = (self.storage._transaction and repr(self.storage._transaction.id)) else: stid = None name = self.__class__.__name__ return "<%s %X trans=%s s_trans=%s>" % (name, id(self), tid, stid) def log(self, msg, level=logging.INFO, exc_info=False): log(msg, level=level, label=self.log_label, exc_info=exc_info) def setup_delegation(self): """Delegate several methods to the storage""" self.versionEmpty = self.storage.versionEmpty self.versions = self.storage.versions self.getSerial = self.storage.getSerial self.history = self.storage.history self.load = self.storage.load self.loadSerial = self.storage.loadSerial self.modifiedInVersion = self.storage.modifiedInVersion try: fn = self.storage.getExtensionMethods except AttributeError: # We must be running with a ZODB which # predates adding getExtensionMethods to # BaseStorage. Eventually this try/except # can be removed pass else: d = fn() self._extensions.update(d) for name in d.keys(): assert not hasattr(self, name) setattr(self, name, getattr(self.storage, name)) self.lastTransaction = self.storage.lastTransaction def _check_tid(self, tid, exc=None): if self.read_only: raise ReadOnlyError() if self.transaction is None: caller = sys._getframe().f_back.f_code.co_name self.log("no current transaction: %s()" % caller, level=logging.WARNING) if exc is not None: raise exc(None, tid) else: return 0 if self.transaction.id != tid: caller = sys._getframe().f_back.f_code.co_name self.log("%s(%s) invalid; current transaction = %s" % (caller, repr(tid), repr(self.transaction.id)), logging.WARNING) if exc is not None: raise exc(self.transaction.id, tid) else: return 0 return 1 def getAuthProtocol(self): """Return string specifying name of authentication module to use. The module name should be auth_%s where %s is auth_protocol.""" protocol = self.server.auth_protocol if not protocol or protocol == 'none': return None return protocol def register(self, storage_id, read_only): """Select the storage that this client will use This method must be the first one called by the client. For authenticated storages this method will be called by the client immediately after authentication is finished. """ if self.auth_realm and not self.authenticated: raise AuthError, "Client was never authenticated with server!" if self.storage is not None: self.log("duplicate register() call") raise ValueError, "duplicate register() call" storage = self.server.storages.get(storage_id) if storage is None: self.log("unknown storage_id: %s" % storage_id) raise ValueError, "unknown storage: %s" % storage_id if not read_only and (self.read_only or storage.isReadOnly()): raise ReadOnlyError() self.read_only = self.read_only or read_only self.storage_id = storage_id self.storage = storage self.setup_delegation() self.timeout, self.stats = self.server.register_connection(storage_id, self) def get_info(self): return {'length': len(self.storage), 'size': self.storage.getSize(), 'name': self.storage.getName(), 'supportsUndo': self.storage.supportsUndo(), 'supportsVersions': self.storage.supportsVersions(), 'extensionMethods': self.getExtensionMethods(), } def get_size_info(self): return {'length': len(self.storage), 'size': self.storage.getSize(), } def getExtensionMethods(self): return self._extensions def loadEx(self, oid, version): self.stats.loads += 1 return self.storage.loadEx(oid, version) def loadBefore(self, oid, tid): self.stats.loads += 1 return self.storage.loadBefore(oid, tid) def zeoLoad(self, oid): self.stats.loads += 1 v = self.storage.modifiedInVersion(oid) if v: pv, sv = self.storage.load(oid, v) else: pv = sv = None try: p, s = self.storage.load(oid, '') except KeyError: if sv: # Created in version, no non-version data p = s = None else: raise return p, s, v, pv, sv def getInvalidations(self, tid): invtid, invlist = self.server.get_invalidations(tid) if invtid is None: return None self.log("Return %d invalidations up to tid %s" % (len(invlist), u64(invtid))) return invtid, invlist def verify(self, oid, version, tid): try: t = self.storage.getTid(oid) except KeyError: self.client.invalidateVerify((oid, "")) else: if tid != t: # This will invalidate non-version data when the # client only has invalid version data. Since this is # an uncommon case, we avoid the cost of checking # whether the serial number matches the current # non-version data. self.client.invalidateVerify((oid, version)) def zeoVerify(self, oid, s, sv): if not self.verifying: self.verifying = 1 self.stats.verifying_clients += 1 try: os = self.storage.getTid(oid) except KeyError: self.client.invalidateVerify((oid, '')) # It's not clear what we should do now. The KeyError # could be caused by an object uncreation, in which case # invalidation is right. It could be an application bug # that left a dangling reference, in which case it's bad. else: # If the client has version data, the logic is a bit more # complicated. If the current serial number matches the # client serial number, then the non-version data must # also be valid. If the current serialno is for a # version, then the non-version data can't change. # If the version serialno isn't valid, then the # non-version serialno may or may not be valid. Rather # than trying to figure it whether it is valid, we just # invalidate it. Sending an invalidation for the # non-version data implies invalidating the version data # too, since an update to non-version data can only occur # after the version is aborted or committed. if sv: if sv != os: self.client.invalidateVerify((oid, '')) else: if s != os: self.client.invalidateVerify((oid, '')) def endZeoVerify(self): if self.verifying: self.stats.verifying_clients -= 1 self.verifying = 0 self.client.endVerify() def pack(self, time, wait=1): # Yes, you can pack a read-only server or storage! if wait: return run_in_thread(self._pack_impl, time) else: # If the client isn't waiting for a reply, start a thread # and forget about it. t = threading.Thread(target=self._pack_impl, args=(time,)) t.start() return None def _pack_impl(self, time): self.log("pack(time=%s) started..." % repr(time)) self.storage.pack(time, referencesf) self.log("pack(time=%s) complete" % repr(time)) # Broadcast new size statistics self.server.invalidate(0, self.storage_id, None, (), self.get_size_info()) def new_oids(self, n=100): """Return a sequence of n new oids, where n defaults to 100""" if self.read_only: raise ReadOnlyError() if n <= 0: n = 1 return [self.storage.new_oid() for i in range(n)] # undoLog and undoInfo are potentially slow methods def undoInfo(self, first, last, spec): return run_in_thread(self.storage.undoInfo, first, last, spec) def undoLog(self, first, last): return run_in_thread(self.storage.undoLog, first, last) def tpc_begin(self, id, user, description, ext, tid=None, status=" "): if self.read_only: raise ReadOnlyError() if self.transaction is not None: if self.transaction.id == id: self.log("duplicate tpc_begin(%s)" % repr(id)) return else: raise StorageTransactionError("Multiple simultaneous tpc_begin" " requests from one client.") self.transaction = t = transaction.Transaction() t.id = id t.user = user t.description = description t._extension = ext self.serials = [] self.invalidated = [] self.txnlog = CommitLog() self.tid = tid self.status = status self.store_failed = 0 self.stats.active_txns += 1 def tpc_finish(self, id): if not self._check_tid(id): return assert self.locked self.stats.active_txns -= 1 self.stats.commits += 1 self.storage.tpc_finish(self.transaction) tid = self.storage.lastTransaction() if self.invalidated: self.server.invalidate(self, self.storage_id, tid, self.invalidated, self.get_size_info()) self._clear_transaction() # Return the tid, for cache invalidation optimization return tid def tpc_abort(self, id): if not self._check_tid(id): return self.stats.active_txns -= 1 self.stats.aborts += 1 if self.locked: self.storage.tpc_abort(self.transaction) self._clear_transaction() def _clear_transaction(self): # Common code at end of tpc_finish() and tpc_abort() self.transaction = None self.txnlog.close() if self.locked: self.locked = 0 self.timeout.end(self) self.stats.lock_time = None self.log("Transaction released storage lock", BLATHER) # _handle_waiting() can start another transaction (by # restarting a waiting one) so must be done last self._handle_waiting() def _abort(self): # called when a connection is closed unexpectedly if not self.locked: # Delete (d, zeo_storage) from the _waiting list, if found. waiting = self.storage._waiting for i in range(len(waiting)): d, z = waiting[i] if z is self: del waiting[i] self.log("Closed connection removed from waiting list." " Clients waiting: %d." % len(waiting)) break if self.transaction: self.stats.active_txns -= 1 self.stats.aborts += 1 self.tpc_abort(self.transaction.id) # The public methods of the ZEO client API do not do the real work. # They defer work until after the storage lock has been acquired. # Most of the real implementations are in methods beginning with # an _. def storea(self, oid, serial, data, version, id): self._check_tid(id, exc=StorageTransactionError) self.stats.stores += 1 self.txnlog.store(oid, serial, data, version) # The following four methods return values, so they must acquire # the storage lock and begin the transaction before returning. def vote(self, id): self._check_tid(id, exc=StorageTransactionError) if self.locked: return self._vote() else: return self._wait(lambda: self._vote()) def abortVersion(self, src, id): self._check_tid(id, exc=StorageTransactionError) if self.locked: return self._abortVersion(src) else: return self._wait(lambda: self._abortVersion(src)) def commitVersion(self, src, dest, id): self._check_tid(id, exc=StorageTransactionError) if self.locked: return self._commitVersion(src, dest) else: return self._wait(lambda: self._commitVersion(src, dest)) def undo(self, trans_id, id): self._check_tid(id, exc=StorageTransactionError) if self.locked: return self._undo(trans_id) else: return self._wait(lambda: self._undo(trans_id)) def _tpc_begin(self, txn, tid, status): self.locked = 1 self.timeout.begin(self) self.stats.lock_time = time.time() self.storage.tpc_begin(txn, tid, status) def _store(self, oid, serial, data, version): err = None try: newserial = self.storage.store(oid, serial, data, version, self.transaction) except (SystemExit, KeyboardInterrupt): raise except Exception, err: self.store_failed = 1 if isinstance(err, ConflictError): self.stats.conflicts += 1 self.log("conflict error oid=%s msg=%s" % (oid_repr(oid), str(err)), BLATHER) if not isinstance(err, TransactionError): # Unexpected errors are logged and passed to the client self.log("store error: %s, %s" % sys.exc_info()[:2], logging.ERROR, exc_info=True) # Try to pickle the exception. If it can't be pickled, # the RPC response would fail, so use something else. pickler = cPickle.Pickler() pickler.fast = 1 try: pickler.dump(err, 1) except: msg = "Couldn't pickle storage exception: %s" % repr(err) self.log(msg, logging.ERROR) err = StorageServerError(msg) # The exception is reported back as newserial for this oid newserial = err else: if serial != "\0\0\0\0\0\0\0\0": self.invalidated.append((oid, version)) if newserial == ResolvedSerial: self.stats.conflicts_resolved += 1 self.log("conflict resolved oid=%s" % oid_repr(oid), BLATHER) self.serials.append((oid, newserial)) return err is None def _vote(self): self.client.serialnos(self.serials) # If a store call failed, then return to the client immediately. # The serialnos() call will deliver an exception that will be # handled by the client in its tpc_vote() method. if self.store_failed: return return self.storage.tpc_vote(self.transaction) def _abortVersion(self, src): tid, oids = self.storage.abortVersion(src, self.transaction) inv = [(oid, src) for oid in oids] self.invalidated.extend(inv) return tid, oids def _commitVersion(self, src, dest): tid, oids = self.storage.commitVersion(src, dest, self.transaction) inv = [(oid, dest) for oid in oids] self.invalidated.extend(inv) if dest: inv = [(oid, src) for oid in oids] self.invalidated.extend(inv) return tid, oids def _undo(self, trans_id): tid, oids = self.storage.undo(trans_id, self.transaction) inv = [(oid, None) for oid in oids] self.invalidated.extend(inv) return tid, oids # When a delayed transaction is restarted, the dance is # complicated. The restart occurs when one ZEOStorage instance # finishes as a transaction and finds another instance is in the # _waiting list. # It might be better to have a mechanism to explicitly send # the finishing transaction's reply before restarting the waiting # transaction. If the restart takes a long time, the previous # client will be blocked until it finishes. def _wait(self, thunk): # Wait for the storage lock to be acquired. self._thunk = thunk if self.storage._transaction: d = Delay() self.storage._waiting.append((d, self)) self.log("Transaction blocked waiting for storage. " "Clients waiting: %d." % len(self.storage._waiting)) return d else: self.log("Transaction acquired storage lock.", BLATHER) return self._restart() def _restart(self, delay=None): # Restart when the storage lock is available. if self.txnlog.stores == 1: template = "Preparing to commit transaction: %d object, %d bytes" else: template = "Preparing to commit transaction: %d objects, %d bytes" self.log(template % (self.txnlog.stores, self.txnlog.size()), level=BLATHER) self._tpc_begin(self.transaction, self.tid, self.status) loads, loader = self.txnlog.get_loader() for i in range(loads): # load oid, serial, data, version if not self._store(*loader.load()): break resp = self._thunk() if delay is not None: delay.reply(resp) else: return resp def _handle_waiting(self): # Restart any client waiting for the storage lock. while self.storage._waiting: delay, zeo_storage = self.storage._waiting.pop(0) if self._restart_other(zeo_storage, delay): if self.storage._waiting: n = len(self.storage._waiting) self.log("Blocked transaction restarted. " "Clients waiting: %d" % n) else: self.log("Blocked transaction restarted.") return def _restart_other(self, zeo_storage, delay): # Return True if the server restarted. # call the restart() method on the appropriate server. try: zeo_storage._restart(delay) except: self.log("Unexpected error handling waiting transaction", level=logging.WARNING, exc_info=True) zeo_storage.connection.close() return 0 else: return 1 class StorageServer: """The server side implementation of ZEO. The StorageServer is the 'manager' for incoming connections. Each connection is associated with its own ZEOStorage instance (defined below). The StorageServer may handle multiple storages; each ZEOStorage instance only handles a single storage. """ # Classes we instantiate. A subclass might override. DispatcherClass = Dispatcher ZEOStorageClass = ZEOStorage ManagedServerConnectionClass = ManagedServerConnection def __init__(self, addr, storages, read_only=0, invalidation_queue_size=100, transaction_timeout=None, monitor_address=None, auth_protocol=None, auth_database=None, auth_realm=None): """StorageServer constructor. This is typically invoked from the start.py script. Arguments (the first two are required and positional): addr -- the address at which the server should listen. This can be a tuple (host, port) to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection. A hostname may be a DNS name or a dotted IP address. storages -- a dictionary giving the storage(s) to handle. The keys are the storage names, the values are the storage instances, typically FileStorage or Berkeley storage instances. By convention, storage names are typically strings representing small integers starting at '1'. read_only -- an optional flag saying whether the server should operate in read-only mode. Defaults to false. Note that even if the server is operating in writable mode, individual storages may still be read-only. But if the server is in read-only mode, no write operations are allowed, even if the storages are writable. Note that pack() is considered a read-only operation. invalidation_queue_size -- The storage server keeps a queue of the objects modified by the last N transactions, where N == invalidation_queue_size. This queue is used to speed client cache verification when a client disconnects for a short period of time. transaction_timeout -- The maximum amount of time to wait for a transaction to commit after acquiring the storage lock. If the transaction takes too long, the client connection will be closed and the transaction aborted. monitor_address -- The address at which the monitor server should listen. If specified, a monitor server is started. The monitor server provides server statistics in a simple text format. auth_protocol -- The name of the authentication protocol to use. Examples are "digest" and "srp". auth_database -- The name of the password database filename. It should be in a format compatible with the authentication protocol used; for instance, "sha" and "srp" require different formats. Note that to implement an authentication protocol, a server and client authentication mechanism must be implemented in a auth_* module, which should be stored inside the "auth" subdirectory. This module may also define a DatabaseClass variable that should indicate what database should be used by the authenticator. """ self.addr = addr self.storages = storages set_label() msg = ", ".join( ["%s:%s:%s" % (name, storage.isReadOnly() and "RO" or "RW", storage.getName()) for name, storage in storages.items()]) log("%s created %s with storages: %s" % (self.__class__.__name__, read_only and "RO" or "RW", msg)) for s in storages.values(): s._waiting = [] self.read_only = read_only self.auth_protocol = auth_protocol self.auth_database = auth_database self.auth_realm = auth_realm self.database = None if auth_protocol: self._setup_auth(auth_protocol) # A list of at most invalidation_queue_size invalidations. # The list is kept in sorted order with the most recent # invalidation at the front. The list never has more than # self.invq_bound elements. self.invq = [] self.invq_bound = invalidation_queue_size self.connections = {} self.dispatcher = self.DispatcherClass(addr, factory=self.new_connection) self.stats = {} self.timeouts = {} for name in self.storages.keys(): self.stats[name] = StorageStats() if transaction_timeout is None: # An object with no-op methods timeout = StubTimeoutThread() else: timeout = TimeoutThread(transaction_timeout) timeout.start() self.timeouts[name] = timeout if monitor_address: self.monitor = StatsServer(monitor_address, self.stats) else: self.monitor = None def _setup_auth(self, protocol): # Can't be done in global scope, because of cyclic references from ZEO.auth import get_module name = self.__class__.__name__ module = get_module(protocol) if not module: log("%s: no such an auth protocol: %s" % (name, protocol)) return storage_class, client, db_class = module if not storage_class or not issubclass(storage_class, ZEOStorage): log(("%s: %s isn't a valid protocol, must have a StorageClass" % (name, protocol))) self.auth_protocol = None return self.ZEOStorageClass = storage_class log("%s: using auth protocol: %s" % (name, protocol)) # We create a Database instance here for use with the authenticator # modules. Having one instance allows it to be shared between multiple # storages, avoiding the need to bloat each with a new authenticator # Database that would contain the same info, and also avoiding any # possibly synchronization issues between them. self.database = db_class(self.auth_database) if self.database.realm != self.auth_realm: raise ValueError("password database realm %r " "does not match storage realm %r" % (self.database.realm, self.auth_realm)) def new_connection(self, sock, addr): """Internal: factory to create a new connection. This is called by the Dispatcher class in ZEO.zrpc.server whenever accept() returns a socket for a new incoming connection. """ if self.auth_protocol and self.database: zstorage = self.ZEOStorageClass(self, self.read_only, auth_realm=self.auth_realm) zstorage.set_database(self.database) else: zstorage = self.ZEOStorageClass(self, self.read_only) c = self.ManagedServerConnectionClass(sock, addr, zstorage, self) log("new connection %s: %s" % (addr, repr(c))) return c def register_connection(self, storage_id, conn): """Internal: register a connection with a particular storage. This is called by ZEOStorage.register(). The dictionary self.connections maps each storage name to a list of current connections for that storage; this information is needed to handle invalidation. This function updates this dictionary. Returns the timeout and stats objects for the appropriate storage. """ l = self.connections.get(storage_id) if l is None: l = self.connections[storage_id] = [] l.append(conn) stats = self.stats[storage_id] stats.clients += 1 return self.timeouts[storage_id], stats def invalidate(self, conn, storage_id, tid, invalidated=(), info=None): """Internal: broadcast info and invalidations to clients. This is called from several ZEOStorage methods. This can do three different things: - If the invalidated argument is non-empty, it broadcasts invalidateTransaction() messages to all clients of the given storage except the current client (the conn argument). - If the invalidated argument is empty and the info argument is a non-empty dictionary, it broadcasts info() messages to all clients of the given storage, including the current client. - If both the invalidated argument and the info argument are non-empty, it broadcasts invalidateTransaction() messages to all clients except the current, and sends an info() message to the current client. """ if invalidated: if len(self.invq) >= self.invq_bound: self.invq.pop() self.invq.insert(0, (tid, invalidated)) for p in self.connections.get(storage_id, ()): if invalidated and p is not conn: p.client.invalidateTransaction(tid, invalidated) elif info is not None: p.client.info(info) def get_invalidations(self, tid): """Return a tid and list of all objects invalidation since tid. The tid is the most recent transaction id seen by the client. Returns None if it is unable to provide a complete list of invalidations for tid. In this case, client should do full cache verification. """ if not self.invq: log("invq empty") return None, [] earliest_tid = self.invq[-1][0] if earliest_tid > tid: log("tid to old for invq %s < %s" % (u64(tid), u64(earliest_tid))) return None, [] oids = {} for _tid, L in self.invq: if _tid <= tid: break for key in L: oids[key] = 1 latest_tid = self.invq[0][0] return latest_tid, oids.keys() def close_server(self): """Close the dispatcher so that there are no new connections. This is only called from the test suite, AFAICT. """ self.dispatcher.close() if self.monitor is not None: self.monitor.close() for storage in self.storages.values(): storage.close() # Force the asyncore mainloop to exit by hackery, i.e. close # every socket in the map. loop() will return when the map is # empty. for s in asyncore.socket_map.values(): try: s.close() except: pass def close_conn(self, conn): """Internal: remove the given connection from self.connections. This is the inverse of register_connection(). """ for cl in self.connections.values(): if conn.obj in cl: cl.remove(conn.obj) class StubTimeoutThread: def begin(self, client): pass def end(self, client): pass class TimeoutThread(threading.Thread): """Monitors transaction progress and generates timeouts.""" # There is one TimeoutThread per storage, because there's one # transaction lock per storage. def __init__(self, timeout): threading.Thread.__init__(self) self.setDaemon(1) self._timeout = timeout self._client = None self._deadline = None self._cond = threading.Condition() # Protects _client and _deadline self._trigger = trigger() def begin(self, client): # Called from the restart code the "main" thread, whenever the # storage lock is being acquired. (Serialized by asyncore.) self._cond.acquire() try: assert self._client is None self._client = client self._deadline = time.time() + self._timeout self._cond.notify() finally: self._cond.release() def end(self, client): # Called from the "main" thread whenever the storage lock is # being released. (Serialized by asyncore.) self._cond.acquire() try: assert self._client is not None assert self._client is client self._client = None self._deadline = None finally: self._cond.release() def run(self): # Code running in the thread. while 1: self._cond.acquire() try: while self._deadline is None: self._cond.wait() howlong = self._deadline - time.time() if howlong <= 0: # Prevent reporting timeout more than once self._deadline = None client = self._client # For the howlong <= 0 branch below finally: self._cond.release() if howlong <= 0: client.log("Transaction timeout after %s seconds" % self._timeout) self._trigger.pull_trigger(lambda: client.connection.close()) else: time.sleep(howlong) def run_in_thread(method, *args): t = SlowMethodThread(method, args) t.start() return t.delay class SlowMethodThread(threading.Thread): """Thread to run potentially slow storage methods. Clients can use the delay attribute to access the MTDelay object used to send a zrpc response at the right time. """ # Some storage methods can take a long time to complete. If we # run these methods via a standard asyncore read handler, they # will block all other server activity until they complete. To # avoid blocking, we spawn a separate thread, return an MTDelay() # object, and have the thread reply() when it finishes. def __init__(self, method, args): threading.Thread.__init__(self) self._method = method self._args = args self.delay = MTDelay() def run(self): try: result = self._method(*self._args) except (SystemExit, KeyboardInterrupt): raise except Exception: self.delay.error(sys.exc_info()) else: self.delay.reply(result) Zope-2.8.7-final/lib/python/ZEO/component.xml0000664000076400007640000001003310435502417020006 0ustar ajungajung The content of a ZEO section describe operational parameters of a ZEO server except for the storage(s) to be served. The address at which the server should listen. This can be in the form 'host:port' to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection (at least one '/' is required). A hostname may be a DNS name or a dotted IP address. If the hostname is omitted, the platform's default behavior is used when binding the listening socket ('' is passed to socket.bind() as the hostname portion of the address). Flag indicating whether the server should operate in read-only mode. Defaults to false. Note that even if the server is operating in writable mode, individual storages may still be read-only. But if the server is in read-only mode, no write operations are allowed, even if the storages are writable. Note that pack() is considered a read-only operation. The storage server keeps a queue of the objects modified by the last N transactions, where N == invalidation_queue_size. This queue is used to speed client cache verification when a client disconnects for a short period of time. The address at which the monitor server should listen. If specified, a monitor server is started. The monitor server provides server statistics in a simple text format. This can be in the form 'host:port' to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection (at least one '/' is required). A hostname may be a DNS name or a dotted IP address. If the hostname is omitted, the platform's default behavior is used when binding the listening socket ('' is passed to socket.bind() as the hostname portion of the address). The maximum amount of time to wait for a transaction to commit after acquiring the storage lock, specified in seconds. If the transaction takes too long, the client connection will be closed and the transaction aborted. The name of the protocol used for authentication. The only protocol provided with ZEO is "digest," but extensions may provide other protocols. The path of the database containing authentication credentials. The authentication realm of the server. Some authentication schemes use a realm to identify the logical set of usernames that are accepted by this server. The full path to the file in which to write the ZEO server's Process ID at startup. If omitted, $INSTANCE/var/ZEO.pid is used. $INSTANCE/var/ZEO.pid (or $clienthome/ZEO.pid) Zope-2.8.7-final/lib/python/persistent/0000775000076400007640000000000010435610645017033 5ustar ajungajungZope-2.8.7-final/lib/python/persistent/SETUP.cfg0000664000076400007640000000121010435502406020401 0ustar ajungajung# Extension information for zpkg. # Mark an "exported" header for use from other packages. # This is not needed for headers only used within the package. # header cPersistence.h # This is included by cPersistence.h, so all users of cPersistence.h # have to be able to include this indirectly. # header ring.h source cPersistence.c source ring.c depends-on cPersistence.h depends-on ring.h source cPickleCache.c source ring.c depends-on cPersistence.h depends-on ring.h source TimeStamp.c Zope-2.8.7-final/lib/python/persistent/mapping.py0000664000076400007640000001014110435502406021030 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Python implementation of persistent base types $Id: mapping.py 66122 2006-03-22 15:40:32Z tseaver $""" import persistent from UserDict import UserDict class PersistentMapping(UserDict, persistent.Persistent): """A persistent wrapper for mapping objects. This class allows wrapping of mapping objects so that object changes are registered. As a side effect, mapping objects may be subclassed. A subclass of PersistentMapping or any code that adds new attributes should not create an attribute named _container. This is reserved for backwards compatibility reasons. """ # UserDict provides all of the mapping behavior. The # PersistentMapping class is responsible marking the persistent # state as changed when a method actually changes the state. At # the mapping API evolves, we may need to add more methods here. __super_delitem = UserDict.__delitem__ __super_setitem = UserDict.__setitem__ __super_clear = UserDict.clear __super_update = UserDict.update __super_setdefault = UserDict.setdefault __super_pop = UserDict.pop __super_popitem = UserDict.popitem def __delitem__(self, key): self.__super_delitem(key) self._p_changed = 1 def __setitem__(self, key, v): self.__super_setitem(key, v) self._p_changed = 1 def clear(self): self.__super_clear() self._p_changed = 1 def update(self, b): self.__super_update(b) self._p_changed = 1 def setdefault(self, key, failobj=None): # We could inline all of UserDict's implementation into the # method here, but I'd rather not depend at all on the # implementation in UserDict (simple as it is). if not self.has_key(key): self._p_changed = 1 return self.__super_setdefault(key, failobj) def pop(self, key, *args): self._p_changed = 1 return self.__super_pop(key, *args) def popitem(self): self._p_changed = 1 return self.__super_popitem() # __iter__ was added in ZODB 3.4.2, but should have been added long # before. We could inherit from Python's IterableUserDict instead # (which just adds __iter__ to Python's UserDict), but that class isn't # documented, and it would add another level of lookup for all the # other methods. def __iter__(self): return iter(self.data) # If the internal representation of PersistentMapping changes, # it causes compatibility problems for pickles generated by # different versions of the code. Compatibility works in both # directions, because an application may want to share a database # between applications using different versions of the code. # Effectively, the original rep is part of the "API." To provide # full compatibility, the getstate and setstate must read and # write objects using the old rep. # As a result, the PersistentMapping must save and restore the # actual internal dictionary using the name _container. def __getstate__(self): state = dict([x for x in self.__dict__.items() if not x[0].startswith('_v_')]) state['_container'] = state['data'] del state['data'] return state def __setstate__(self, state): if state.has_key('_container'): self.data = state['_container'] del state['_container'] elif not state.has_key('data'): self.data = {} self.__dict__.update(state) Zope-2.8.7-final/lib/python/persistent/ring.c0000664000076400007640000000346410435502406020140 0ustar ajungajung/***************************************************************************** Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #define RING_C "$Id: ring.c 25186 2004-06-02 15:07:33Z jim $\n" /* Support routines for the doubly-linked list of cached objects. The cache stores a doubly-linked list of persistent objects, with space for the pointers allocated in the objects themselves. The cache stores the distinguished head of the list, which is not a valid persistent object. The next pointers traverse the ring in order starting with the least recently used object. The prev pointers traverse the ring in order starting with the most recently used object. */ #include "Python.h" #include "ring.h" void ring_add(CPersistentRing *ring, CPersistentRing *elt) { assert(!elt->r_next); elt->r_next = ring; elt->r_prev = ring->r_prev; ring->r_prev->r_next = elt; ring->r_prev = elt; } void ring_del(CPersistentRing *elt) { elt->r_next->r_prev = elt->r_prev; elt->r_prev->r_next = elt->r_next; elt->r_next = NULL; elt->r_prev = NULL; } void ring_move_to_head(CPersistentRing *ring, CPersistentRing *elt) { elt->r_prev->r_next = elt->r_next; elt->r_next->r_prev = elt->r_prev; elt->r_next = ring; elt->r_prev = ring->r_prev; ring->r_prev->r_next = elt; ring->r_prev = elt; } Zope-2.8.7-final/lib/python/persistent/__init__.py0000664000076400007640000000227310435502406021143 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Provide access to Persistent and PersistentMapping. $Id: __init__.py 25186 2004-06-02 15:07:33Z jim $ """ from cPersistence import Persistent, GHOST, UPTODATE, CHANGED, STICKY from cPickleCache import PickleCache from cPersistence import simple_new import copy_reg copy_reg.constructor(simple_new) # Make an interface declaration for Persistent, # if zope.interface is available. try: from zope.interface import classImplements except ImportError: pass else: from persistent.interfaces import IPersistent classImplements(Persistent, IPersistent) Zope-2.8.7-final/lib/python/persistent/README.txt0000664000076400007640000000131310435502407020523 0ustar ajungajung=================== Persistence support =================== (This document is under construction. More basic documentation will eventually appear here.) Overriding __getattr__, __getattribute__, __setattr__, and __delattr__ ----------------------------------------------------------------------- Subclasses can override the attribute-management methods. For the __getattr__ method, the behavior is like that for regular Python classes and for earlier versions of ZODB 3. For __getattribute__, __setattr__, and __delattr__, it is necessary to call certain methods defined by persistent.Persistent. Detailed examples and documentation is provided in the test module, persistent.tests.test_overriding_attrs. Zope-2.8.7-final/lib/python/persistent/list.py0000664000076400007640000000556110435502406020362 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Python implementation of persistent list. $Id: list.py 25186 2004-06-02 15:07:33Z jim $""" import persistent from UserList import UserList class PersistentList(UserList, persistent.Persistent): __super_setitem = UserList.__setitem__ __super_delitem = UserList.__delitem__ __super_setslice = UserList.__setslice__ __super_delslice = UserList.__delslice__ __super_iadd = UserList.__iadd__ __super_imul = UserList.__imul__ __super_append = UserList.append __super_insert = UserList.insert __super_pop = UserList.pop __super_remove = UserList.remove __super_reverse = UserList.reverse __super_sort = UserList.sort __super_extend = UserList.extend def __setitem__(self, i, item): self.__super_setitem(i, item) self._p_changed = 1 def __delitem__(self, i): self.__super_delitem(i) self._p_changed = 1 def __setslice__(self, i, j, other): self.__super_setslice(i, j, other) self._p_changed = 1 def __delslice__(self, i, j): self.__super_delslice(i, j) self._p_changed = 1 def __iadd__(self, other): L = self.__super_iadd(other) self._p_changed = 1 return L def __imul__(self, n): L = self.__super_imul(n) self._p_changed = 1 return L def append(self, item): self.__super_append(item) self._p_changed = 1 def insert(self, i, item): self.__super_insert(i, item) self._p_changed = 1 def pop(self, i=-1): rtn = self.__super_pop(i) self._p_changed = 1 return rtn def remove(self, item): self.__super_remove(item) self._p_changed = 1 def reverse(self): self.__super_reverse() self._p_changed = 1 def sort(self, *args): self.__super_sort(*args) self._p_changed = 1 def extend(self, other): self.__super_extend(other) self._p_changed = 1 # This works around a bug in Python 2.1.x (up to 2.1.2 at least) where the # __cmp__ bogusly raises a RuntimeError, and because this is an extension # class, none of the rich comparison stuff works anyway. def __cmp__(self, other): return cmp(self.data, self._UserList__cast(other)) Zope-2.8.7-final/lib/python/persistent/cPersistence.c0000664000076400007640000007333110435502406021630 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ static char cPersistence_doc_string[] = "Defines Persistent mixin class for persistent objects.\n" "\n" "$Id: cPersistence.c 29446 2005-03-11 22:07:51Z tim_one $\n"; #include "cPersistence.h" #include "structmember.h" struct ccobject_head_struct { CACHE_HEAD }; /* These two objects are initialized when the module is loaded */ static PyObject *TimeStamp, *py_simple_new; /* Strings initialized by init_strings() below. */ static PyObject *py_keys, *py_setstate, *py___dict__, *py_timeTime; static PyObject *py__p_changed, *py__p_deactivate; static PyObject *py___getattr__, *py___setattr__, *py___delattr__; static PyObject *py___slotnames__, *copy_reg_slotnames, *__newobj__; static PyObject *py___getnewargs__, *py___getstate__; static int init_strings(void) { #define INIT_STRING(S) \ if (!(py_ ## S = PyString_InternFromString(#S))) \ return -1; INIT_STRING(keys); INIT_STRING(setstate); INIT_STRING(timeTime); INIT_STRING(__dict__); INIT_STRING(_p_changed); INIT_STRING(_p_deactivate); INIT_STRING(__getattr__); INIT_STRING(__setattr__); INIT_STRING(__delattr__); INIT_STRING(__slotnames__); INIT_STRING(__getnewargs__); INIT_STRING(__getstate__); #undef INIT_STRING return 0; } #ifdef Py_DEBUG static void fatal_1350(cPersistentObject *self, const char *caller, const char *detail) { char buf[1000]; PyOS_snprintf(buf, sizeof(buf), "cPersistence.c %s(): object at %p with type %.200s\n" "%s.\n" "The only known cause is multiple threads trying to ghost and\n" "unghost the object simultaneously.\n" "That's not legal, but ZODB can't stop it.\n" "See Collector #1350.\n", caller, self, self->ob_type->tp_name, detail); Py_FatalError(buf); } #endif static void ghostify(cPersistentObject*); /* Load the state of the object, unghostifying it. Upon success, return 1. * If an error occurred, re-ghostify the object and return -1. */ static int unghostify(cPersistentObject *self) { if (self->state < 0 && self->jar) { PyObject *r; /* Is it ever possibly to not have a cache? */ if (self->cache) { /* Create a node in the ring for this unghostified object. */ self->cache->non_ghost_count++; ring_add(&self->cache->ring_home, &self->ring); Py_INCREF(self); } /* set state to CHANGED while setstate() call is in progress to prevent a recursive call to _PyPersist_Load(). */ self->state = cPersistent_CHANGED_STATE; /* Call the object's __setstate__() */ r = PyObject_CallMethod(self->jar, "setstate", "O", (PyObject *)self); if (r == NULL) { ghostify(self); return -1; } self->state = cPersistent_UPTODATE_STATE; Py_DECREF(r); if (self->cache && self->ring.r_next == NULL) { #ifdef Py_DEBUG fatal_1350(self, "unghostify", "is not in the cache despite that we just " "unghostified it"); #else PyErr_Format(PyExc_SystemError, "object at %p with type " "%.200s not in the cache despite that we just " "unghostified it", self, self->ob_type->tp_name); return -1; #endif } } return 1; } /****************************************************************************/ static PyTypeObject Pertype; static void accessed(cPersistentObject *self) { /* Do nothing unless the object is in a cache and not a ghost. */ if (self->cache && self->state >= 0 && self->ring.r_next) ring_move_to_head(&self->cache->ring_home, &self->ring); } static void unlink_from_ring(cPersistentObject *self) { /* If the cache has been cleared, then a non-ghost object isn't in the ring any longer. */ if (self->ring.r_next == NULL) return; /* if we're ghostifying an object, we better have some non-ghosts */ assert(self->cache->non_ghost_count > 0); self->cache->non_ghost_count--; ring_del(&self->ring); } static void ghostify(cPersistentObject *self) { PyObject **dictptr; /* are we already a ghost? */ if (self->state == cPersistent_GHOST_STATE) return; /* Is it ever possible to not have a cache? */ if (self->cache == NULL) { self->state = cPersistent_GHOST_STATE; return; } if (self->ring.r_next == NULL) { /* There's no way to raise an error in this routine. */ #ifdef Py_DEBUG fatal_1350(self, "ghostify", "claims to be in a cache but isn't"); #else return; #endif } /* If we're ghostifying an object, we better have some non-ghosts. */ assert(self->cache->non_ghost_count > 0); self->cache->non_ghost_count--; ring_del(&self->ring); self->state = cPersistent_GHOST_STATE; dictptr = _PyObject_GetDictPtr((PyObject *)self); if (dictptr && *dictptr) { Py_DECREF(*dictptr); *dictptr = NULL; } /* We remove the reference to the just ghosted object that the ring * holds. Note that the dictionary of oids->objects has an uncounted * reference, so if the ring's reference was the only one, this frees * the ghost object. Note further that the object's dealloc knows to * inform the dictionary that it is going away. */ Py_DECREF(self); } static int changed(cPersistentObject *self) { if ((self->state == cPersistent_UPTODATE_STATE || self->state == cPersistent_STICKY_STATE) && self->jar) { PyObject *meth, *arg, *result; static PyObject *s_register; if (s_register == NULL) s_register = PyString_InternFromString("register"); meth = PyObject_GetAttr((PyObject *)self->jar, s_register); if (meth == NULL) return -1; arg = PyTuple_New(1); if (arg == NULL) { Py_DECREF(meth); return -1; } Py_INCREF(self); PyTuple_SET_ITEM(arg, 0, (PyObject *)self); result = PyEval_CallObject(meth, arg); Py_DECREF(arg); Py_DECREF(meth); if (result == NULL) return -1; Py_DECREF(result); self->state = cPersistent_CHANGED_STATE; } return 0; } static PyObject * Per__p_deactivate(cPersistentObject *self) { if (self->state == cPersistent_UPTODATE_STATE && self->jar) { PyObject **dictptr = _PyObject_GetDictPtr((PyObject *)self); if (dictptr && *dictptr) { Py_DECREF(*dictptr); *dictptr = NULL; } /* Note that we need to set to ghost state unless we are called directly. Methods that override this need to do the same! */ ghostify(self); } Py_INCREF(Py_None); return Py_None; } static PyObject * Per__p_activate(cPersistentObject *self) { if (unghostify(self) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static int Per_set_changed(cPersistentObject *self, PyObject *v); static PyObject * Per__p_invalidate(cPersistentObject *self) { signed char old_state = self->state; if (old_state != cPersistent_GHOST_STATE) { if (Per_set_changed(self, NULL) < 0) return NULL; ghostify(self); } Py_INCREF(Py_None); return Py_None; } static PyObject * pickle_slotnames(PyTypeObject *cls) { PyObject *slotnames; slotnames = PyDict_GetItem(cls->tp_dict, py___slotnames__); if (slotnames) { Py_INCREF(slotnames); return slotnames; } slotnames = PyObject_CallFunctionObjArgs(copy_reg_slotnames, (PyObject*)cls, NULL); if (slotnames && !(slotnames == Py_None || PyList_Check(slotnames))) { PyErr_SetString(PyExc_TypeError, "copy_reg._slotnames didn't return a list or None"); Py_DECREF(slotnames); return NULL; } return slotnames; } static PyObject * pickle_copy_dict(PyObject *state) { PyObject *copy, *key, *value; char *ckey; int pos = 0; copy = PyDict_New(); if (!copy) return NULL; if (!state) return copy; while (PyDict_Next(state, &pos, &key, &value)) { if (key && PyString_Check(key)) { ckey = PyString_AS_STRING(key); if (*ckey == '_' && (ckey[1] == 'v' || ckey[1] == 'p') && ckey[2] == '_') /* skip volatile and persistent */ continue; } if (PyObject_SetItem(copy, key, value) < 0) goto err; } return copy; err: Py_DECREF(copy); return NULL; } static char pickle___getstate__doc[] = "Get the object serialization state\n" "\n" "If the object has no assigned slots and has no instance dictionary, then \n" "None is returned.\n" "\n" "If the object has no assigned slots and has an instance dictionary, then \n" "the a copy of the instance dictionary is returned. The copy has any items \n" "with names starting with '_v_' or '_p_' ommitted.\n" "\n" "If the object has assigned slots, then a two-element tuple is returned. \n" "The first element is either None or a copy of the instance dictionary, \n" "as described above. The second element is a dictionary with items \n" "for each of the assigned slots.\n" ; static PyObject * pickle___getstate__(PyObject *self) { PyObject *slotnames=NULL, *slots=NULL, *state=NULL; PyObject **dictp; int n=0; slotnames = pickle_slotnames(self->ob_type); if (!slotnames) return NULL; dictp = _PyObject_GetDictPtr(self); if (dictp) state = pickle_copy_dict(*dictp); else { state = Py_None; Py_INCREF(state); } if (slotnames != Py_None) { int i; slots = PyDict_New(); if (!slots) goto end; for (i = 0; i < PyList_GET_SIZE(slotnames); i++) { PyObject *name, *value; char *cname; name = PyList_GET_ITEM(slotnames, i); if (PyString_Check(name)) { cname = PyString_AS_STRING(name); if (*cname == '_' && (cname[1] == 'v' || cname[1] == 'p') && cname[2] == '_') /* skip volatile and persistent */ continue; } /* Unclear: Will this go through our getattr hook? */ value = PyObject_GetAttr(self, name); if (value == NULL) PyErr_Clear(); else { int err = PyDict_SetItem(slots, name, value); Py_DECREF(value); if (err < 0) goto end; n++; } } } if (n) state = Py_BuildValue("(NO)", state, slots); end: Py_XDECREF(slotnames); Py_XDECREF(slots); return state; } static int pickle_setattrs_from_dict(PyObject *self, PyObject *dict) { PyObject *key, *value; int pos = 0; if (!PyDict_Check(dict)) { PyErr_SetString(PyExc_TypeError, "Expected dictionary"); return -1; } while (PyDict_Next(dict, &pos, &key, &value)) { if (PyObject_SetAttr(self, key, value) < 0) return -1; } return 0; } static char pickle___setstate__doc[] = "Set the object serialization state\n\n" "The state should be in one of 3 forms:\n\n" "- None\n\n" " Ignored\n\n" "- A dictionary\n\n" " In this case, the object's instance dictionary will be cleared and \n" " updated with the new state.\n\n" "- A two-tuple with a string as the first element. \n\n" " In this case, the method named by the string in the first element will be\n" " called with the second element.\n\n" " This form supports migration of data formats.\n\n" "- A two-tuple with None or a Dictionary as the first element and\n" " with a dictionary as the second element.\n\n" " If the first element is not None, then the object's instance dictionary \n" " will be cleared and updated with the value.\n\n" " The items in the second element will be assigned as attributes.\n" ; static PyObject * pickle___setstate__(PyObject *self, PyObject *state) { PyObject *slots=NULL; if (PyTuple_Check(state)) { if (!PyArg_ParseTuple(state, "OO:__setstate__", &state, &slots)) return NULL; } if (state != Py_None) { PyObject **dict; dict = _PyObject_GetDictPtr(self); if (dict) { if (!*dict) { *dict = PyDict_New(); if (!*dict) return NULL; } } if (*dict) { PyDict_Clear(*dict); if (PyDict_Update(*dict, state) < 0) return NULL; } else if (pickle_setattrs_from_dict(self, state) < 0) return NULL; } if (slots && pickle_setattrs_from_dict(self, slots) < 0) return NULL; Py_INCREF(Py_None); return Py_None; } static char pickle___reduce__doc[] = "Reduce an object to contituent parts for serialization\n" ; static PyObject * pickle___reduce__(PyObject *self) { PyObject *args=NULL, *bargs=NULL, *state=NULL, *getnewargs=NULL; int l, i; getnewargs = PyObject_GetAttr(self, py___getnewargs__); if (getnewargs) { bargs = PyObject_CallFunctionObjArgs(getnewargs, NULL); Py_DECREF(getnewargs); if (!bargs) return NULL; l = PyTuple_Size(bargs); if (l < 0) goto end; } else { PyErr_Clear(); l = 0; } args = PyTuple_New(l+1); if (args == NULL) goto end; Py_INCREF(self->ob_type); PyTuple_SET_ITEM(args, 0, (PyObject*)(self->ob_type)); for (i = 0; i < l; i++) { Py_INCREF(PyTuple_GET_ITEM(bargs, i)); PyTuple_SET_ITEM(args, i+1, PyTuple_GET_ITEM(bargs, i)); } state = PyObject_CallMethodObjArgs(self, py___getstate__, NULL); if (!state) goto end; state = Py_BuildValue("(OON)", __newobj__, args, state); end: Py_XDECREF(bargs); Py_XDECREF(args); return state; } /* Return the object's state, a dict or None. If the object has no dict, it's state is None. Otherwise, return a dict containing all the attributes that don't start with "_v_". The caller should not modify this dict, as it may be a reference to the object's __dict__. */ static PyObject * Per__getstate__(cPersistentObject *self) { /* TODO: Should it be an error to call __getstate__() on a ghost? */ if (unghostify(self) < 0) return NULL; /* TODO: should we increment stickyness? Tim doesn't understand that question. S*/ return pickle___getstate__((PyObject*)self); } /* The Persistent base type provides a traverse function, but not a clear function. An instance of a Persistent subclass will have its dict cleared through subtype_clear(). There is always a cycle between a persistent object and its cache. When the cycle becomes unreachable, the clear function for the cache will break the cycle. Thus, the persistent object need not have a clear function. It would be complex to write a clear function for the objects, if we needed one, because of the reference count tricks done by the cache. */ static void Per_dealloc(cPersistentObject *self) { if (self->state >= 0) unlink_from_ring(self); if (self->cache) cPersistenceCAPI->percachedel(self->cache, self->oid); Py_XDECREF(self->cache); Py_XDECREF(self->jar); Py_XDECREF(self->oid); self->ob_type->tp_free(self); } static int Per_traverse(cPersistentObject *self, visitproc visit, void *arg) { int err; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ return err; \ } VISIT(self->jar); VISIT(self->oid); VISIT(self->cache); #undef VISIT return 0; } /* convert_name() returns a new reference to a string name or sets an exception and returns NULL. */ static PyObject * convert_name(PyObject *name) { #ifdef Py_USING_UNICODE /* The Unicode to string conversion is done here because the existing tp_setattro slots expect a string object as name and we wouldn't want to break those. */ if (PyUnicode_Check(name)) { name = PyUnicode_AsEncodedString(name, NULL, NULL); } else #endif if (!PyString_Check(name)) { PyErr_SetString(PyExc_TypeError, "attribute name must be a string"); return NULL; } else Py_INCREF(name); return name; } /* Returns true if the object requires unghostification. There are several special attributes that we allow access to without requiring that the object be unghostified: __class__ __del__ __dict__ __of__ __setstate__ */ static int unghost_getattr(const char *s) { if (*s++ != '_') return 1; if (*s == 'p') { s++; if (*s == '_') return 0; /* _p_ */ else return 1; } else if (*s == '_') { s++; switch (*s) { case 'c': return strcmp(s, "class__"); case 'd': s++; if (!strcmp(s, "el__")) return 0; /* __del__ */ if (!strcmp(s, "ict__")) return 0; /* __dict__ */ return 1; case 'o': return strcmp(s, "of__"); case 's': return strcmp(s, "setstate__"); default: return 1; } } return 1; } static PyObject* Per_getattro(cPersistentObject *self, PyObject *name) { PyObject *result = NULL; /* guilty until proved innocent */ char *s; name = convert_name(name); if (!name) goto Done; s = PyString_AS_STRING(name); if (unghost_getattr(s)) { if (unghostify(self) < 0) goto Done; accessed(self); } result = PyObject_GenericGetAttr((PyObject *)self, name); Done: Py_XDECREF(name); return result; } /* Exposed as _p_getattr method. Test whether base getattr should be used */ static PyObject * Per__p_getattr(cPersistentObject *self, PyObject *name) { PyObject *result = NULL; /* guilty until proved innocent */ char *s; name = convert_name(name); if (!name) goto Done; s = PyString_AS_STRING(name); if (*s != '_' || unghost_getattr(s)) { if (unghostify(self) < 0) goto Done; accessed(self); result = Py_False; } else result = Py_True; Py_INCREF(result); Done: Py_XDECREF(name); return result; } /* TODO: we should probably not allow assignment of __class__ and __dict__. */ static int Per_setattro(cPersistentObject *self, PyObject *name, PyObject *v) { int result = -1; /* guilty until proved innocent */ char *s; name = convert_name(name); if (!name) goto Done; s = PyString_AS_STRING(name); if (strncmp(s, "_p_", 3) != 0) { if (unghostify(self) < 0) goto Done; accessed(self); if (strncmp(s, "_v_", 3) != 0 && self->state != cPersistent_CHANGED_STATE) { if (changed(self) < 0) goto Done; } } result = PyObject_GenericSetAttr((PyObject *)self, name, v); Done: Py_XDECREF(name); return result; } static int Per_p_set_or_delattro(cPersistentObject *self, PyObject *name, PyObject *v) { int result = -1; /* guilty until proved innocent */ char *s; name = convert_name(name); if (!name) goto Done; s = PyString_AS_STRING(name); if (strncmp(s, "_p_", 3)) { if (unghostify(self) < 0) goto Done; accessed(self); result = 0; } else { if (PyObject_GenericSetAttr((PyObject *)self, name, v) < 0) goto Done; result = 1; } Done: Py_XDECREF(name); return result; } static PyObject * Per__p_setattr(cPersistentObject *self, PyObject *args) { PyObject *name, *v, *result; int r; if (!PyArg_ParseTuple(args, "OO:_p_setattr", &name, &v)) return NULL; r = Per_p_set_or_delattro(self, name, v); if (r < 0) return NULL; result = r ? Py_True : Py_False; Py_INCREF(result); return result; } static PyObject * Per__p_delattr(cPersistentObject *self, PyObject *name) { int r; PyObject *result; r = Per_p_set_or_delattro(self, name, NULL); if (r < 0) return NULL; result = r ? Py_True : Py_False; Py_INCREF(result); return result; } static PyObject * Per_get_changed(cPersistentObject *self) { if (self->state < 0) { Py_INCREF(Py_None); return Py_None; } return PyBool_FromLong(self->state == cPersistent_CHANGED_STATE); } static int Per_set_changed(cPersistentObject *self, PyObject *v) { int deactivate = 0, true; if (!v) { /* delattr is used to invalidate an object even if it has changed. */ if (self->state != cPersistent_GHOST_STATE) self->state = cPersistent_UPTODATE_STATE; deactivate = 1; } else if (v == Py_None) deactivate = 1; if (deactivate) { PyObject *res, *meth; meth = PyObject_GetAttr((PyObject *)self, py__p_deactivate); if (meth == NULL) return -1; res = PyObject_CallObject(meth, NULL); if (res) Py_DECREF(res); else { /* an error occured in _p_deactivate(). It's not clear what we should do here. The code is obviously ignoring the exception, but it shouldn't return 0 for a getattr and set an exception. The simplest change is to clear the exception, but that simply masks the error. This prints an error to stderr just like exceptions in __del__(). It would probably be better to log it but that would be painful from C. */ PyErr_WriteUnraisable(meth); } Py_DECREF(meth); return 0; } true = PyObject_IsTrue(v); if (true == -1) return -1; else if (true) return changed(self); if (self->state >= 0) self->state = cPersistent_UPTODATE_STATE; return 0; } static PyObject * Per_get_oid(cPersistentObject *self) { PyObject *oid = self->oid ? self->oid : Py_None; Py_INCREF(oid); return oid; } static int Per_set_oid(cPersistentObject *self, PyObject *v) { if (self->cache) { int result; if (v == NULL) { PyErr_SetString(PyExc_ValueError, "can't delete _p_oid of cached object"); return -1; } if (PyObject_Cmp(self->oid, v, &result) < 0) return -1; if (result) { PyErr_SetString(PyExc_ValueError, "can not change _p_oid of cached object"); return -1; } } Py_XDECREF(self->oid); Py_XINCREF(v); self->oid = v; return 0; } static PyObject * Per_get_jar(cPersistentObject *self) { PyObject *jar = self->jar ? self->jar : Py_None; Py_INCREF(jar); return jar; } static int Per_set_jar(cPersistentObject *self, PyObject *v) { if (self->cache) { int result; if (v == NULL) { PyErr_SetString(PyExc_ValueError, "can't delete _p_jar of cached object"); return -1; } if (PyObject_Cmp(self->jar, v, &result) < 0) return -1; if (result) { PyErr_SetString(PyExc_ValueError, "can not change _p_jar of cached object"); return -1; } } Py_XDECREF(self->jar); Py_XINCREF(v); self->jar = v; return 0; } static PyObject * Per_get_serial(cPersistentObject *self) { return PyString_FromStringAndSize(self->serial, 8); } static int Per_set_serial(cPersistentObject *self, PyObject *v) { if (v) { if (PyString_Check(v) && PyString_GET_SIZE(v) == 8) memcpy(self->serial, PyString_AS_STRING(v), 8); else { PyErr_SetString(PyExc_ValueError, "_p_serial must be an 8-character string"); return -1; } } else memset(self->serial, 0, 8); return 0; } static PyObject * Per_get_mtime(cPersistentObject *self) { PyObject *t, *v; if (unghostify(self) < 0) return NULL; accessed(self); if (memcmp(self->serial, "\0\0\0\0\0\0\0\0", 8) == 0) { Py_INCREF(Py_None); return Py_None; } t = PyObject_CallFunction(TimeStamp, "s#", self->serial, 8); if (!t) return NULL; v = PyObject_CallMethod(t, "timeTime", ""); Py_DECREF(t); return v; } static PyObject * Per_get_state(cPersistentObject *self) { return PyInt_FromLong(self->state); } static PyGetSetDef Per_getsets[] = { {"_p_changed", (getter)Per_get_changed, (setter)Per_set_changed}, {"_p_jar", (getter)Per_get_jar, (setter)Per_set_jar}, {"_p_mtime", (getter)Per_get_mtime}, {"_p_oid", (getter)Per_get_oid, (setter)Per_set_oid}, {"_p_serial", (getter)Per_get_serial, (setter)Per_set_serial}, {"_p_state", (getter)Per_get_state}, {NULL} }; static struct PyMethodDef Per_methods[] = { {"_p_deactivate", (PyCFunction)Per__p_deactivate, METH_NOARGS, "_p_deactivate() -- Deactivate the object"}, {"_p_activate", (PyCFunction)Per__p_activate, METH_NOARGS, "_p_activate() -- Activate the object"}, {"_p_invalidate", (PyCFunction)Per__p_invalidate, METH_NOARGS, "_p_invalidate() -- Invalidate the object"}, {"_p_getattr", (PyCFunction)Per__p_getattr, METH_O, "_p_getattr(name) -- Test whether the base class must handle the name\n" "\n" "The method unghostifies the object, if necessary.\n" "The method records the object access, if necessary.\n" "\n" "This method should be called by subclass __getattribute__\n" "implementations before doing anything else. If the method\n" "returns True, then __getattribute__ implementations must delegate\n" "to the base class, Persistent.\n" }, {"_p_setattr", (PyCFunction)Per__p_setattr, METH_VARARGS, "_p_setattr(name, value) -- Save persistent meta data\n" "\n" "This method should be called by subclass __setattr__ implementations\n" "before doing anything else. If it returns true, then the attribute\n" "was handled by the base class.\n" "\n" "The method unghostifies the object, if necessary.\n" "The method records the object access, if necessary.\n" }, {"_p_delattr", (PyCFunction)Per__p_delattr, METH_O, "_p_delattr(name) -- Delete persistent meta data\n" "\n" "This method should be called by subclass __delattr__ implementations\n" "before doing anything else. If it returns true, then the attribute\n" "was handled by the base class.\n" "\n" "The method unghostifies the object, if necessary.\n" "The method records the object access, if necessary.\n" }, {"__getstate__", (PyCFunction)Per__getstate__, METH_NOARGS, pickle___getstate__doc }, {"__setstate__", (PyCFunction)pickle___setstate__, METH_O, pickle___setstate__doc}, {"__reduce__", (PyCFunction)pickle___reduce__, METH_NOARGS, pickle___reduce__doc}, {NULL, NULL} /* sentinel */ }; /* This module is compiled as a shared library. Some compilers don't allow addresses of Python objects defined in other libraries to be used in static initializers here. The DEFERRED_ADDRESS macro is used to tag the slots where such addresses appear; the module init function must fill in the tagged slots at runtime. The argument is for documentation -- the macro ignores it. */ #define DEFERRED_ADDRESS(ADDR) 0 static PyTypeObject Pertype = { PyObject_HEAD_INIT(DEFERRED_ADDRESS(&PyPersist_MetaType)) 0, /* ob_size */ "persistent.Persistent", /* tp_name */ sizeof(cPersistentObject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)Per_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ (getattrofunc)Per_getattro, /* tp_getattro */ (setattrofunc)Per_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)Per_traverse, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Per_methods, /* tp_methods */ 0, /* tp_members */ Per_getsets, /* tp_getset */ }; /* End of code for Persistent objects */ /* -------------------------------------------------------- */ typedef int (*intfunctionwithpythonarg)(PyObject*); /* Load the object's state if necessary and become sticky */ static int Per_setstate(cPersistentObject *self) { if (unghostify(self) < 0) return -1; self->state = cPersistent_STICKY_STATE; return 0; } static PyObject * simple_new(PyObject *self, PyObject *type_object) { return PyType_GenericNew((PyTypeObject *)type_object, NULL, NULL); } static PyMethodDef cPersistence_methods[] = { {"simple_new", simple_new, METH_O, "Create an object by simply calling a class's __new__ method without " "arguments."}, {NULL, NULL} }; static cPersistenceCAPIstruct truecPersistenceCAPI = { &Pertype, (getattrofunc)Per_getattro, /*tp_getattr with object key*/ (setattrofunc)Per_setattro, /*tp_setattr with object key*/ changed, accessed, ghostify, (intfunctionwithpythonarg)Per_setstate, NULL /* The percachedel slot is initialized in cPickleCache.c when the module is loaded. It uses a function in a different shared library. */ }; void initcPersistence(void) { PyObject *m, *s; PyObject *copy_reg; if (init_strings() < 0) return; m = Py_InitModule3("cPersistence", cPersistence_methods, cPersistence_doc_string); Pertype.ob_type = &PyType_Type; Pertype.tp_new = PyType_GenericNew; if (PyType_Ready(&Pertype) < 0) return; if (PyModule_AddObject(m, "Persistent", (PyObject *)&Pertype) < 0) return; cPersistenceCAPI = &truecPersistenceCAPI; s = PyCObject_FromVoidPtr(cPersistenceCAPI, NULL); if (!s) return; if (PyModule_AddObject(m, "CAPI", s) < 0) return; if (PyModule_AddIntConstant(m, "GHOST", cPersistent_GHOST_STATE) < 0) return; if (PyModule_AddIntConstant(m, "UPTODATE", cPersistent_UPTODATE_STATE) < 0) return; if (PyModule_AddIntConstant(m, "CHANGED", cPersistent_CHANGED_STATE) < 0) return; if (PyModule_AddIntConstant(m, "STICKY", cPersistent_STICKY_STATE) < 0) return; py_simple_new = PyObject_GetAttrString(m, "simple_new"); if (!py_simple_new) return; copy_reg = PyImport_ImportModule("copy_reg"); if (!copy_reg) return; copy_reg_slotnames = PyObject_GetAttrString(copy_reg, "_slotnames"); if (!copy_reg_slotnames) { Py_DECREF(copy_reg); return; } __newobj__ = PyObject_GetAttrString(copy_reg, "__newobj__"); if (!__newobj__) { Py_DECREF(copy_reg); return; } if (!TimeStamp) { m = PyImport_ImportModule("persistent.TimeStamp"); if (!m) return; TimeStamp = PyObject_GetAttrString(m, "TimeStamp"); Py_DECREF(m); /* fall through to immediate return on error */ } } Zope-2.8.7-final/lib/python/persistent/interfaces.py0000664000076400007640000002346710435502407021540 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Persistence Interfaces $Id: interfaces.py 30127 2005-04-24 01:28:57Z jim $ """ from zope.interface import Interface from zope.interface import Attribute class IPersistent(Interface): """Python persistent interface A persistent object can be in one of several states: - Unsaved The object has been created but not saved in a data manager. In this state, the _p_changed attribute is non-None and false and the _p_jar attribute is None. - Saved The object has been saved and has not been changed since it was saved. In this state, the _p_changed attribute is non-None and false and the _p_jar attribute is set to a data manager. - Sticky This state is identical to the saved state except that the object cannot transition to the ghost state. This is a special state used by C methods of persistent objects to make sure that state is not unloaded in the middle of computation. In this state, the _p_changed attribute is non-None and false and the _p_jar attribute is set to a data manager. There is no Python API for detecting whether an object is in the sticky state. - Changed The object has been changed. In this state, the _p_changed attribute is true and the _p_jar attribute is set to a data manager. - Ghost the object is in memory but its state has not been loaded from the database (or its state has been unloaded). In this state, the object doesn't contain any application data. In this state, the _p_changed attribute is None, and the _p_jar attribute is set to the data manager from which the object was obtained. In all the above, _p_oid (the persistent object id) is set when _p_jar first gets set. The following state transactions are possible: - Unsaved -> Saved This transition occurs when an object is saved in the database. This usually happens when an unsaved object is added to (e.g. as an attribute or item of) a saved (or changed) object and the transaction is committed. - Saved -> Changed Sticky -> Changed This transition occurs when someone sets an attribute or sets _p_changed to a true value on a saved or sticky object. When the transition occurs, the persistent object is required to call the register() method on its data manager, passing itself as the only argument. - Saved -> Sticky This transition occurs when C code marks the object as sticky to prevent its deactivation. - Saved -> Ghost This transition occurs when a saved object is deactivated or invalidated. See discussion below. - Sticky -> Saved This transition occurs when C code unmarks the object as sticky to allow its deactivation. - Changed -> Saved This transition occurs when a transaction is committed. After saving the state of a changed object during transaction commit, the data manager sets the object's _p_changed to a non-None false value. - Changed -> Ghost This transition occurs when a transaction is aborted. All changed objects are invalidated by the data manager by an abort. - Ghost -> Saved This transition occurs when an attribute or operation of a ghost is accessed and the object's state is loaded from the database. Note that there is a separate C API that is not included here. The C API requires a specific data layout and defines the sticky state. About Invalidation, Deactivation and the Sticky & Ghost States The sticky state is intended to be a short-lived state, to prevent an object's state from being discarded while we're in C routines. It is an error to invalidate an object in the sticky state. Deactivation is a request that an object discard its state (become a ghost). Deactivation is an optimization, and a request to deactivate may be ignored. There are two equivalent ways to request deactivation: - call _p_deactivate() - set _p_changed to None There are two ways to invalidate an object: call the _p_invalidate() method (preferred) or delete its _p_changed attribute. This cannot be ignored, and is used when semantics require invalidation. Normally, an invalidated object transitions to the ghost state. However, some objects cannot be ghosts. When these objects are invalidated, they immediately reload their state from their data manager, and are then in the saved state. """ _p_jar = Attribute( """The data manager for the object. The data manager implements the IPersistentDataManager interface. If there is no data manager, then this is None. """) _p_oid = Attribute( """The object id. It is up to the data manager to assign this. The special value None is reserved to indicate that an object id has not been assigned. Non-None object ids must be hashable and totally ordered. """) _p_changed = Attribute( """The persistent state of the object This is one of: None -- The object is a ghost. false but not None -- The object is saved (or has never been saved). true -- The object has been modified since it was last saved. The object state may be changed by assigning or deleting this attribute; however, assigning None is ignored if the object is not in the saved state, and may be ignored even if the object is in the saved state. Note that an object can transition to the changed state only if it has a data manager. When such a state change occurs, the 'register' method of the data manager must be called, passing the persistent object. Deleting this attribute forces invalidation independent of existing state, although it is an error if the sticky state is current. """) _p_serial = Attribute( """The object serial number. This member is used by the data manager to distiguish distinct revisions of a given persistent object. This is an 8-byte string (not Unicode). """) def __getstate__(): """Get the object data. The state should not include persistent attributes ("_p_name"). The result must be picklable. """ def __setstate__(state): """Set the object data. """ def _p_activate(): """Activate the object. Change the object to the saved state if it is a ghost. """ def _p_deactivate(): """Deactivate the object. Possibly change an object in the saved state to the ghost state. It may not be possible to make some persistent objects ghosts, and, for optimization reasons, the implementation may choose to keep an object in the saved state. """ def _p_invalidate(): """Invalidate the object. Invalidate the object. This causes any data to be thrown away, even if the object is in the changed state. The object is moved to the ghost state; further accesses will cause object data to be reloaded. """ class IPersistentNoReadConflicts(IPersistent): def _p_independent(): """Hook for subclasses to prevent read conflict errors. A specific persistent object type can define this method and have it return true if the data manager should ignore read conflicts for this object. """ # TODO: document conflict resolution. class IPersistentDataManager(Interface): """Provide services for managing persistent state. This interface is used by a persistent object to interact with its data manager in the context of a transaction. """ def setstate(object): """Load the state for the given object. The object should be in the ghost state. The object's state will be set and the object will end up in the saved state. The object must provide the IPersistent interface. """ def oldstate(obj, tid): """Return copy of 'obj' that was written by transaction 'tid'. The returned object does not have the typical metadata (_p_jar, _p_oid, _p_serial) set. I'm not sure how references to other peristent objects are handled. Parameters obj: a persistent object from this Connection. tid: id of a transaction that wrote an earlier revision. Raises KeyError if tid does not exist or if tid deleted a revision of obj. """ def register(object): """Register an IPersistent with the current transaction. This method must be called when the object transitions to the changed state. A subclass could override this method to customize the default policy of one transaction manager for each thread. """ # Maybe later: ## def mtime(object): ## """Return the modification time of the object. ## The modification time may not be known, in which case None ## is returned. If non-None, the return value is the kind of ## timestamp supplied by Python's time.time(). ## """ Zope-2.8.7-final/lib/python/persistent/cPickleCache.c0000664000076400007640000007727510435502406021512 0ustar ajungajung /***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /* Objects are stored under three different regimes: Regime 1: Persistent Classes Persistent Classes are part of ZClasses. They are stored in the self->data dictionary, and are never garbage collected. The klass_items() method returns a sequence of (oid,object) tuples for every Persistent Class, which should make it possible to implement garbage collection in Python if necessary. Regime 2: Ghost Objects There is no benefit to keeping a ghost object which has no external references, therefore a weak reference scheme is used to ensure that ghost objects are removed from memory as soon as possible, when the last external reference is lost. Ghost objects are stored in the self->data dictionary. Normally a dictionary keeps a strong reference on its values, however this reference count is 'stolen'. This weak reference scheme leaves a dangling reference, in the dictionary, when the last external reference is lost. To clean up this dangling reference the persistent object dealloc function calls self->cache->_oid_unreferenced(self->oid). The cache looks up the oid in the dictionary, ensures it points to an object whose reference count is zero, then removes it from the dictionary. Before removing the object from the dictionary it must temporarily resurrect the object in much the same way that class instances are resurrected before their __del__ is called. Since ghost objects are stored under a different regime to non-ghost objects, an extra ghostify function in cPersistenceAPI replaces self->state=GHOST_STATE assignments that were common in other persistent classes (such as BTrees). Regime 3: Non-Ghost Objects Non-ghost objects are stored in two data structures: the dictionary mapping oids to objects and a doubly-linked list that encodes the order in which the objects were accessed. The dictionary reference is borrowed, as it is for ghosts. The list reference is a new reference; the list stores recently used objects, even if they are otherwise unreferenced, to avoid loading the object from the database again. The doubly-link-list nodes contain next and previous pointers linking together the cache and all non-ghost persistent objects. The node embedded in the cache is the home position. On every attribute access a non-ghost object will relink itself just behind the home position in the ring. Objects accessed least recently will eventually find themselves positioned after the home position. Occasionally other nodes are temporarily inserted in the ring as position markers. The cache contains a ring_lock flag which must be set and unset before and after doing so. Only if the flag is unset can the cache assume that all nodes are either his own home node, or nodes from persistent objects. This assumption is useful during the garbage collection process. The number of non-ghost objects is counted in self->non_ghost_count. The garbage collection process consists of traversing the ring, and deactivating (that is, turning into a ghost) every object until self->non_ghost_count is down to the target size, or until it reaches the home position again. Note that objects in the sticky or changed states are still kept in the ring, however they can not be deactivated. The garbage collection process must skip such objects, rather than deactivating them. */ static char cPickleCache_doc_string[] = "Defines the PickleCache used by ZODB Connection objects.\n" "\n" "$Id: cPickleCache.c 29891 2005-04-06 22:35:05Z tim_one $\n"; #define DONT_USE_CPERSISTENCECAPI #include "cPersistence.h" #include "structmember.h" #include #include #undef Py_FindMethod /* Python string objects to speed lookups; set by module init. */ static PyObject *py__p_changed; static PyObject *py__p_deactivate; static PyObject *py__p_jar; static PyObject *py__p_oid; static cPersistenceCAPIstruct *capi; /* This object is the pickle cache. The CACHE_HEAD macro guarantees that layout of this struct is the same as the start of ccobject_head in cPersistence.c */ typedef struct { CACHE_HEAD int klass_count; /* count of persistent classes */ PyObject *data; /* oid -> object dict */ PyObject *jar; /* Connection object */ int cache_size; /* target number of items in cache */ /* Most of the time the ring contains only: * many nodes corresponding to persistent objects * one 'home' node from the cache. In some cases it is handy to temporarily add other types of node into the ring as placeholders. 'ring_lock' is a boolean indicating that someone has already done this. Currently this is only used by the garbage collection code. */ int ring_lock; /* 'cache_drain_resistance' controls how quickly the cache size will drop when it is smaller than the configured size. A value of zero means it will not drop below the configured size (suitable for most caches). Otherwise, it will remove cache_non_ghost_count/cache_drain_resistance items from the cache every time (suitable for rarely used caches, such as those associated with Zope versions. */ int cache_drain_resistance; } ccobject; static int cc_ass_sub(ccobject *self, PyObject *key, PyObject *v); /* ---------------------------------------------------------------- */ #define OBJECT_FROM_RING(SELF, HERE) \ ((cPersistentObject *)(((char *)here) - offsetof(cPersistentObject, ring))) /* Insert self into the ring, following after. */ static void insert_after(CPersistentRing *self, CPersistentRing *after) { assert(self != NULL); assert(after != NULL); self->r_prev = after; self->r_next = after->r_next; after->r_next->r_prev = self; after->r_next = self; } /* Remove self from the ring. */ static void unlink_from_ring(CPersistentRing *self) { assert(self != NULL); self->r_prev->r_next = self->r_next; self->r_next->r_prev = self->r_prev; } static int scan_gc_items(ccobject *self, int target) { /* This function must only be called with the ring lock held, because it places non-object placeholders in the ring. */ cPersistentObject *object; CPersistentRing *here; CPersistentRing before_original_home; int result = -1; /* guilty until proved innocent */ /* Scan the ring, from least to most recently used, deactivating * up-to-date objects, until we either find the ring_home again or * or we've ghosted enough objects to reach the target size. * Tricky: __getattr__ and __del__ methods can do anything, and in * particular if we ghostify an object with a __del__ method, that method * can load the object again, putting it back into the MRU part of the * ring. Waiting to find ring_home again can thus cause an infinite * loop (Collector #1208). So before_original_home records the MRU * position we start with, and we stop the scan when we reach that. */ insert_after(&before_original_home, self->ring_home.r_prev); here = self->ring_home.r_next; /* least recently used object */ while (here != &before_original_home && self->non_ghost_count > target) { assert(self->ring_lock); assert(here != &self->ring_home); /* At this point we know that the ring only contains nodes from persistent objects, plus our own home node. We know this because the ring lock is held. We can safely assume the current ring node is a persistent object now we know it is not the home */ object = OBJECT_FROM_RING(self, here); if (object->state == cPersistent_UPTODATE_STATE) { CPersistentRing placeholder; PyObject *method; PyObject *temp; int error_occurred = 0; /* deactivate it. This is the main memory saver. */ /* Add a placeholder, a dummy node in the ring. We need to do this to mark our position in the ring. It is possible that the PyObject_GetAttr() call below will invoke a __getattr__() hook in Python. Also possible that deactivation will lead to a __del__ method call. So another thread might run, and mutate the ring as a side effect of object accesses. There's no predicting then where in the ring here->next will point after that. The placeholder won't move as a side effect of calling Python code. */ insert_after(&placeholder, here); method = PyObject_GetAttr((PyObject *)object, py__p_deactivate); if (method == NULL) error_occurred = 1; else { temp = PyObject_CallObject(method, NULL); Py_DECREF(method); if (temp == NULL) error_occurred = 1; } here = placeholder.r_next; unlink_from_ring(&placeholder); if (error_occurred) goto Done; } else here = here->r_next; } result = 0; Done: unlink_from_ring(&before_original_home); return result; } static PyObject * lockgc(ccobject *self, int target_size) { /* This is thread-safe because of the GIL, and there's nothing * in between checking the ring_lock and acquiring it that calls back * into Python. */ if (self->ring_lock) { Py_INCREF(Py_None); return Py_None; } self->ring_lock = 1; if (scan_gc_items(self, target_size) < 0) { self->ring_lock = 0; return NULL; } self->ring_lock = 0; Py_INCREF(Py_None); return Py_None; } static PyObject * cc_incrgc(ccobject *self, PyObject *args) { int obsolete_arg = -999; int starting_size = self->non_ghost_count; int target_size = self->cache_size; if (self->cache_drain_resistance >= 1) { /* This cache will gradually drain down to a small size. Check a (small) number of objects proportional to the current size */ int target_size_2 = (starting_size - 1 - starting_size / self->cache_drain_resistance); if (target_size_2 < target_size) target_size = target_size_2; } if (!PyArg_ParseTuple(args, "|i:incrgc", &obsolete_arg)) return NULL; if (obsolete_arg != -999 && (PyErr_Warn(PyExc_DeprecationWarning, "No argument expected") < 0)) return NULL; return lockgc(self, target_size); } static PyObject * cc_full_sweep(ccobject *self, PyObject *args) { int dt = -999; /* TODO: This should be deprecated; */ if (!PyArg_ParseTuple(args, "|i:full_sweep", &dt)) return NULL; if (dt == -999) return lockgc(self, 0); else return cc_incrgc(self, args); } static PyObject * cc_minimize(ccobject *self, PyObject *args) { int ignored = -999; if (!PyArg_ParseTuple(args, "|i:minimize", &ignored)) return NULL; if (ignored != -999 && (PyErr_Warn(PyExc_DeprecationWarning, "No argument expected") < 0)) return NULL; return lockgc(self, 0); } static int _invalidate(ccobject *self, PyObject *key) { static PyObject *_p_invalidate = NULL; PyObject *meth, *v; v = PyDict_GetItem(self->data, key); if (v == NULL) return 0; if (_p_invalidate == NULL) { _p_invalidate = PyString_InternFromString("_p_invalidate"); if (_p_invalidate == NULL) { /* It doesn't make any sense to ignore this error, but the caller ignores all errors. TODO: and why does it do that? This should be fixed */ return -1; } } if (v->ob_refcnt <= 1 && PyType_Check(v)) { /* This looks wrong, but it isn't. We use strong references to types because they don't have the ring members. The result is that we *never* remove classes unless they are modified. We can fix this by using wekrefs uniformly. */ self->klass_count--; return PyDict_DelItem(self->data, key); } meth = PyObject_GetAttr(v, _p_invalidate); if (meth == NULL) return -1; v = PyObject_CallObject(meth, NULL); Py_DECREF(meth); return v == NULL ? -1 : 0; } static PyObject * cc_invalidate(ccobject *self, PyObject *inv) { PyObject *key, *v; int i = 0; if (PyDict_Check(inv)) { while (PyDict_Next(inv, &i, &key, &v)) { if (_invalidate(self, key) < 0) return NULL; } PyDict_Clear(inv); } else { if (PyString_Check(inv)) { if (_invalidate(self, inv) < 0) return NULL; } else { int l, r; l = PyObject_Length(inv); if (l < 0) return NULL; for (i=l; --i >= 0; ) { key = PySequence_GetItem(inv, i); if (!key) return NULL; r = _invalidate(self, key); Py_DECREF(key); if (r < 0) return NULL; } /* Dubious: modifying the input may be an unexpected side effect. */ PySequence_DelSlice(inv, 0, l); } } Py_INCREF(Py_None); return Py_None; } static PyObject * cc_get(ccobject *self, PyObject *args) { PyObject *r, *key, *d = NULL; if (!PyArg_ParseTuple(args, "O|O:get", &key, &d)) return NULL; r = PyDict_GetItem(self->data, key); if (!r) { if (d) r = d; else r = Py_None; } Py_INCREF(r); return r; } static PyObject * cc_items(ccobject *self) { return PyObject_CallMethod(self->data, "items", ""); } static PyObject * cc_klass_items(ccobject *self) { PyObject *l,*k,*v; int p = 0; l = PyList_New(0); if (l == NULL) return NULL; while (PyDict_Next(self->data, &p, &k, &v)) { if(PyType_Check(v)) { v = Py_BuildValue("OO", k, v); if (v == NULL) { Py_DECREF(l); return NULL; } if (PyList_Append(l, v) < 0) { Py_DECREF(v); Py_DECREF(l); return NULL; } Py_DECREF(v); } } return l; } static PyObject * cc_debug_info(ccobject *self) { PyObject *l,*k,*v; int p = 0; l = PyList_New(0); if (l == NULL) return NULL; while (PyDict_Next(self->data, &p, &k, &v)) { if (v->ob_refcnt <= 0) v = Py_BuildValue("Oi", k, v->ob_refcnt); else if (! PyType_Check(v) && (v->ob_type->tp_basicsize >= sizeof(cPersistentObject)) ) v = Py_BuildValue("Oisi", k, v->ob_refcnt, v->ob_type->tp_name, ((cPersistentObject*)v)->state); else v = Py_BuildValue("Ois", k, v->ob_refcnt, v->ob_type->tp_name); if (v == NULL) goto err; if (PyList_Append(l, v) < 0) goto err; } return l; err: Py_DECREF(l); return NULL; } static PyObject * cc_lru_items(ccobject *self) { PyObject *l; CPersistentRing *here; if (self->ring_lock) { /* When the ring lock is held, we have no way of know which ring nodes belong to persistent objects, and which a placeholders. */ PyErr_SetString(PyExc_ValueError, ".lru_items() is unavailable during garbage collection"); return NULL; } l = PyList_New(0); if (l == NULL) return NULL; here = self->ring_home.r_next; while (here != &self->ring_home) { PyObject *v; cPersistentObject *object = OBJECT_FROM_RING(self, here); if (object == NULL) { Py_DECREF(l); return NULL; } v = Py_BuildValue("OO", object->oid, object); if (v == NULL) { Py_DECREF(l); return NULL; } if (PyList_Append(l, v) < 0) { Py_DECREF(v); Py_DECREF(l); return NULL; } Py_DECREF(v); here = here->r_next; } return l; } static void cc_oid_unreferenced(ccobject *self, PyObject *oid) { /* This is called by the persistent object deallocation function when the reference count on a persistent object reaches zero. We need to fix up our dictionary; its reference is now dangling because we stole its reference count. Be careful to not release the global interpreter lock until this is complete. */ PyObject *v; /* If the cache has been cleared by GC, data will be NULL. */ if (!self->data) return; v = PyDict_GetItem(self->data, oid); assert(v); assert(v->ob_refcnt == 0); /* Need to be very hairy here because a dictionary is about to decref an already deleted object. */ #ifdef Py_TRACE_REFS /* This is called from the deallocation function after the interpreter has untracked the reference. Track it again. */ _Py_NewReference(v); /* Don't increment total refcount as a result of the shenanigans played in this function. The _Py_NewReference() call above creates artificial references to v. */ _Py_RefTotal--; assert(v->ob_type); #else Py_INCREF(v); #endif assert(v->ob_refcnt == 1); /* Incremement the refcount again, because delitem is going to DECREF it. If it's refcount reached zero again, we'd call back to the dealloc function that called us. */ Py_INCREF(v); /* TODO: Should we call _Py_ForgetReference() on error exit? */ if (PyDict_DelItem(self->data, oid) < 0) return; Py_DECREF((ccobject *)((cPersistentObject *)v)->cache); ((cPersistentObject *)v)->cache = NULL; assert(v->ob_refcnt == 1); /* Undo the temporary resurrection. Don't DECREF the object, because this function is called from the object's dealloc function. If the refcnt reaches zero, it will all be invoked recursively. */ _Py_ForgetReference(v); } static PyObject * cc_ringlen(ccobject *self) { CPersistentRing *here; int c = 0; for (here = self->ring_home.r_next; here != &self->ring_home; here = here->r_next) c++; return PyInt_FromLong(c); } static struct PyMethodDef cc_methods[] = { {"items", (PyCFunction)cc_items, METH_NOARGS, "Return list of oid, object pairs for all items in cache."}, {"lru_items", (PyCFunction)cc_lru_items, METH_NOARGS, "List (oid, object) pairs from the lru list, as 2-tuples."}, {"klass_items", (PyCFunction)cc_klass_items, METH_NOARGS, "List (oid, object) pairs of cached persistent classes."}, {"full_sweep", (PyCFunction)cc_full_sweep, METH_VARARGS, "full_sweep() -- Perform a full sweep of the cache."}, {"minimize", (PyCFunction)cc_minimize, METH_VARARGS, "minimize([ignored]) -- Remove as many objects as possible\n\n" "Ghostify all objects that are not modified. Takes an optional\n" "argument, but ignores it."}, {"incrgc", (PyCFunction)cc_incrgc, METH_VARARGS, "incrgc() -- Perform incremental garbage collection\n\n" "This method had been depricated!" "Some other implementations support an optional parameter 'n' which\n" "indicates a repetition count; this value is ignored."}, {"invalidate", (PyCFunction)cc_invalidate, METH_O, "invalidate(oids) -- invalidate one, many, or all ids"}, {"get", (PyCFunction)cc_get, METH_VARARGS, "get(key [, default]) -- get an item, or a default"}, {"ringlen", (PyCFunction)cc_ringlen, METH_NOARGS, "ringlen() -- Returns number of non-ghost items in cache."}, {"debug_info", (PyCFunction)cc_debug_info, METH_NOARGS, "debug_info() -- Returns debugging data about objects in the cache."}, {NULL, NULL} /* sentinel */ }; static int cc_init(ccobject *self, PyObject *args, PyObject *kwds) { int cache_size = 100; PyObject *jar; if (!PyArg_ParseTuple(args, "O|i", &jar, &cache_size)) return -1; self->jar = NULL; self->data = PyDict_New(); if (self->data == NULL) { Py_DECREF(self); return -1; } /* Untrack the dict mapping oids to objects. The dict contains uncounted references to ghost objects, so it isn't safe for GC to visit it. If GC finds an object with more referents that refcounts, it will die with an assertion failure. When the cache participates in GC, it will need to traverse the objects in the doubly-linked list, which will account for all the non-ghost objects. */ PyObject_GC_UnTrack((void *)self->data); self->jar = jar; Py_INCREF(jar); self->cache_size = cache_size; self->non_ghost_count = 0; self->klass_count = 0; self->cache_drain_resistance = 0; self->ring_lock = 0; self->ring_home.r_next = &self->ring_home; self->ring_home.r_prev = &self->ring_home; return 0; } static void cc_dealloc(ccobject *self) { Py_XDECREF(self->data); Py_XDECREF(self->jar); PyObject_GC_Del(self); } static int cc_clear(ccobject *self) { int pos = 0; PyObject *k, *v; /* Clearing the cache is delicate. A non-ghost object will show up in the ring and in the dict. If we deallocating the dict before clearing the ring, the GC will decref each object in the dict. Since the dict references are uncounted, this will lead to objects having negative refcounts. Freeing the non-ghost objects should eliminate many objects from the cache, but there may still be ghost objects left. It's not safe to decref the dict until it's empty, so we need to manually clear those out of the dict, too. We accomplish that by replacing all the ghost objects with None. */ /* We don't need to lock the ring, because the cache is unreachable. It should be impossible for anyone to be modifying the cache. */ assert(! self->ring_lock); while (self->ring_home.r_next != &self->ring_home) { CPersistentRing *here = self->ring_home.r_next; cPersistentObject *o = OBJECT_FROM_RING(self, here); if (o->cache) { Py_INCREF(o); /* account for uncounted reference */ if (PyDict_DelItem(self->data, o->oid) < 0) return -1; } o->cache = NULL; Py_DECREF(self); self->ring_home.r_next = here->r_next; o->ring.r_prev = NULL; o->ring.r_next = NULL; Py_DECREF(o); here = here->r_next; } Py_XDECREF(self->jar); while (PyDict_Next(self->data, &pos, &k, &v)) { Py_INCREF(v); if (PyDict_SetItem(self->data, k, Py_None) < 0) return -1; } Py_XDECREF(self->data); self->data = NULL; self->jar = NULL; return 0; } static int cc_traverse(ccobject *self, visitproc visit, void *arg) { int err; CPersistentRing *here; /* If we're in the midst of cleaning up old objects, the ring contains * assorted junk we must not pass on to the visit() callback. This * should be rare (our cleanup code would need to have called back * into Python, which in turn triggered Python's gc). When it happens, * simply don't chase any pointers. The cache will appear to be a * source of external references then, and at worst we miss cleaning * up a dead cycle until the next time Python's gc runs. */ if (self->ring_lock) return 0; #define VISIT(SLOT) \ if (SLOT) { \ err = visit((PyObject *)(SLOT), arg); \ if (err) \ return err; \ } VISIT(self->jar); here = self->ring_home.r_next; /* It is possible that an object is traversed after it is cleared. In that case, there is no ring. */ if (!here) return 0; while (here != &self->ring_home) { cPersistentObject *o = OBJECT_FROM_RING(self, here); VISIT(o); here = here->r_next; } #undef VISIT return 0; } static int cc_length(ccobject *self) { return PyObject_Length(self->data); } static PyObject * cc_subscript(ccobject *self, PyObject *key) { PyObject *r; r = PyDict_GetItem(self->data, key); if (r == NULL) { PyErr_SetObject(PyExc_KeyError, key); return NULL; } Py_INCREF(r); return r; } static int cc_add_item(ccobject *self, PyObject *key, PyObject *v) { int result; PyObject *oid, *object_again, *jar; cPersistentObject *p; /* Sanity check the value given to make sure it is allowed in the cache */ if (PyType_Check(v)) { /* Its a persistent class, such as a ZClass. Thats ok. */ } else if (v->ob_type->tp_basicsize < sizeof(cPersistentObject)) { /* If it's not an instance of a persistent class, (ie Python classes that derive from persistent.Persistent, BTrees, etc), report an error. TODO: checking sizeof() seems a poor test. */ PyErr_SetString(PyExc_TypeError, "Cache values must be persistent objects."); return -1; } /* Can't access v->oid directly because the object might be a * persistent class. */ oid = PyObject_GetAttr(v, py__p_oid); if (oid == NULL) return -1; if (! PyString_Check(oid)) { PyErr_Format(PyExc_TypeError, "Cached object oid must be a string, not a %s", oid->ob_type->tp_name); return -1; } /* we know they are both strings. * now check if they are the same string. */ result = PyObject_Compare(key, oid); if (PyErr_Occurred()) { Py_DECREF(oid); return -1; } Py_DECREF(oid); if (result) { PyErr_SetString(PyExc_ValueError, "Cache key does not match oid"); return -1; } /* useful sanity check, but not strictly an invariant of this class */ jar = PyObject_GetAttr(v, py__p_jar); if (jar == NULL) return -1; if (jar==Py_None) { Py_DECREF(jar); PyErr_SetString(PyExc_ValueError, "Cached object jar missing"); return -1; } Py_DECREF(jar); object_again = PyDict_GetItem(self->data, key); if (object_again) { if (object_again != v) { PyErr_SetString(PyExc_ValueError, "A different object already has the same oid"); return -1; } else { /* re-register under the same oid - no work needed */ return 0; } } if (PyType_Check(v)) { if (PyDict_SetItem(self->data, key, v) < 0) return -1; self->klass_count++; return 0; } else { PerCache *cache = ((cPersistentObject *)v)->cache; if (cache) { if (cache != (PerCache *)self) /* This object is already in a different cache. */ PyErr_SetString(PyExc_ValueError, "Cache values may only be in one cache."); return -1; } /* else: This object is already one of ours, which is ok. It would be very strange if someone was trying to register the same object under a different key. */ } if (PyDict_SetItem(self->data, key, v) < 0) return -1; /* the dict should have a borrowed reference */ Py_DECREF(v); p = (cPersistentObject *)v; Py_INCREF(self); p->cache = (PerCache *)self; if (p->state >= 0) { /* insert this non-ghost object into the ring just behind the home position. */ self->non_ghost_count++; ring_add(&self->ring_home, &p->ring); /* this list should have a new reference to the object */ Py_INCREF(v); } return 0; } static int cc_del_item(ccobject *self, PyObject *key) { PyObject *v; cPersistentObject *p; /* unlink this item from the ring */ v = PyDict_GetItem(self->data, key); if (v == NULL) { PyErr_SetObject(PyExc_KeyError, key); return -1; } if (PyType_Check(v)) { self->klass_count--; } else { p = (cPersistentObject *)v; if (p->state >= 0) { self->non_ghost_count--; ring_del(&p->ring); /* The DelItem below will account for the reference held by the list. */ } else { /* This is a ghost object, so we haven't kept a reference count on it. For it have stayed alive this long someone else must be keeping a reference to it. Therefore we need to temporarily give it back a reference count before calling DelItem below */ Py_INCREF(v); } Py_DECREF((PyObject *)p->cache); p->cache = NULL; } if (PyDict_DelItem(self->data, key) < 0) { PyErr_SetString(PyExc_RuntimeError, "unexpectedly couldn't remove key in cc_ass_sub"); return -1; } return 0; } static int cc_ass_sub(ccobject *self, PyObject *key, PyObject *v) { if (!PyString_Check(key)) { PyErr_Format(PyExc_TypeError, "cPickleCache key must be a string, not a %s", key->ob_type->tp_name); return -1; } if (v) return cc_add_item(self, key, v); else return cc_del_item(self, key); } static PyMappingMethods cc_as_mapping = { (inquiry)cc_length, /*mp_length*/ (binaryfunc)cc_subscript, /*mp_subscript*/ (objobjargproc)cc_ass_sub, /*mp_ass_subscript*/ }; static PyObject * cc_cache_data(ccobject *self, void *context) { return PyDict_Copy(self->data); } static PyGetSetDef cc_getsets[] = { {"cache_data", (getter)cc_cache_data}, {NULL} }; static PyMemberDef cc_members[] = { {"cache_size", T_INT, offsetof(ccobject, cache_size)}, {"cache_drain_resistance", T_INT, offsetof(ccobject, cache_drain_resistance)}, {"cache_non_ghost_count", T_INT, offsetof(ccobject, non_ghost_count), RO}, {"cache_klass_count", T_INT, offsetof(ccobject, klass_count), RO}, {NULL} }; /* This module is compiled as a shared library. Some compilers don't allow addresses of Python objects defined in other libraries to be used in static initializers here. The DEFERRED_ADDRESS macro is used to tag the slots where such addresses appear; the module init function must fill in the tagged slots at runtime. The argument is for documentation -- the macro ignores it. */ #define DEFERRED_ADDRESS(ADDR) 0 static PyTypeObject Cctype = { PyObject_HEAD_INIT(DEFERRED_ADDRESS(&PyType_Type)) 0, /* ob_size */ "persistent.PickleCache", /* tp_name */ sizeof(ccobject), /* tp_basicsize */ 0, /* tp_itemsize */ (destructor)cc_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ &cc_as_mapping, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ 0, /* tp_doc */ (traverseproc)cc_traverse, /* tp_traverse */ (inquiry)cc_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ cc_methods, /* tp_methods */ cc_members, /* tp_members */ cc_getsets, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)cc_init, /* tp_init */ }; void initcPickleCache(void) { PyObject *m; Cctype.ob_type = &PyType_Type; Cctype.tp_new = &PyType_GenericNew; if (PyType_Ready(&Cctype) < 0) { return; } m = Py_InitModule3("cPickleCache", NULL, cPickleCache_doc_string); capi = (cPersistenceCAPIstruct *)PyCObject_Import( "persistent.cPersistence", "CAPI"); if (!capi) return; capi->percachedel = (percachedelfunc)cc_oid_unreferenced; py__p_changed = PyString_InternFromString("_p_changed"); if (!py__p_changed) return; py__p_deactivate = PyString_InternFromString("_p_deactivate"); if (!py__p_deactivate) return; py__p_jar = PyString_InternFromString("_p_jar"); if (!py__p_jar) return; py__p_oid = PyString_InternFromString("_p_oid"); if (!py__p_oid) return; if (PyModule_AddStringConstant(m, "cache_variant", "stiff/c") < 0) return; /* This leaks a reference to Cctype, but it doesn't matter. */ if (PyModule_AddObject(m, "PickleCache", (PyObject *)&Cctype) < 0) return; } Zope-2.8.7-final/lib/python/persistent/dict.py0000664000076400007640000000541210435502407020326 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Python implementation of persistent container type $Id: dict.py 40331 2005-11-22 21:07:44Z tlotze $ """ import persistent from UserDict import IterableUserDict __metaclass__ = type class PersistentDict(persistent.Persistent, IterableUserDict): """A persistent wrapper for mapping objects. This class allows wrapping of mapping objects so that object changes are registered. As a side effect, mapping objects may be subclassed. """ # IterableUserDict provides all of the mapping behavior. The # PersistentDict class is responsible marking the persistent # state as changed when a method actually changes the state. At # the mapping API evolves, we may need to add more methods here. __super_delitem = IterableUserDict.__delitem__ __super_setitem = IterableUserDict.__setitem__ __super_clear = IterableUserDict.clear __super_update = IterableUserDict.update __super_setdefault = IterableUserDict.setdefault __super_pop = IterableUserDict.pop __super_popitem = IterableUserDict.popitem __super_p_init = persistent.Persistent.__init__ __super_init = IterableUserDict.__init__ def __init__(self, dict=None): self.__super_init(dict) self.__super_p_init() def __delitem__(self, key): self.__super_delitem(key) self._p_changed = True def __setitem__(self, key, v): self.__super_setitem(key, v) self._p_changed = True def clear(self): self.__super_clear() self._p_changed = True def update(self, b): self.__super_update(b) self._p_changed = True def setdefault(self, key, failobj=None): # We could inline all of UserDict's implementation into the # method here, but I'd rather not depend at all on the # implementation in UserDict (simple as it is). if not self.has_key(key): self._p_changed = True return self.__super_setdefault(key, failobj) def pop(self, key, *args): self._p_changed = True return self.__super_pop(key, *args) def popitem(self): self._p_changed = True return self.__super_popitem() Zope-2.8.7-final/lib/python/persistent/wref.py0000664000076400007640000002010210435502406020336 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ZODB-based persistent weakrefs $Id: wref.py 29450 2005-03-11 23:53:09Z tim_one $ """ from persistent import Persistent WeakRefMarker = object() class WeakRef(object): """Persistent weak references Persistent weak references are used much like Python weak references. The major difference is that you can't specify an object to be called when the object is removed from the database. Here's an example. We'll start by creating a persistent object and a refernce to it: >>> import persistent.list >>> import ZODB.tests.util >>> ob = persistent.list.PersistentList() >>> ref = WeakRef(ob) >>> ref() is ob True The hash of the ref if the same as the hash of the referenced object: >>> hash(ref) == hash(ob) True Two refs to the same object are equal: >>> WeakRef(ob) == ref True >>> ob2 = persistent.list.PersistentList([1]) >>> WeakRef(ob2) == ref False Lets save the reference and the referenced object in a database: >>> db = ZODB.tests.util.DB() >>> conn1 = db.open() >>> conn1.root()['ob'] = ob >>> conn1.root()['ref'] = ref >>> ZODB.tests.util.commit() If we open a new connection, we can use the reference: >>> conn2 = db.open() >>> conn2.root()['ref']() is conn2.root()['ob'] True >>> hash(conn2.root()['ref']) == hash(conn2.root()['ob']) True But if we delete the referenced object and pack: >>> del conn2.root()['ob'] >>> ZODB.tests.util.commit() >>> ZODB.tests.util.pack(db) And then look in a new connection: >>> conn3 = db.open() >>> conn3.root()['ob'] Traceback (most recent call last): ... KeyError: 'ob' Trying to dereference the reference returns None: >>> conn3.root()['ref']() Trying to get a hash, raises a type error: >>> hash(conn3.root()['ref']) Traceback (most recent call last): ... TypeError: Weakly-referenced object has gone away Always explicitly close databases: :) >>> db.close() """ # We set _p_oid to a marker so that the serialization system can # provide special handling of weakrefs. _p_oid = WeakRefMarker def __init__(self, ob): self._v_ob = ob self.oid = ob._p_oid self.dm = ob._p_jar def __call__(self): try: return self._v_ob except AttributeError: try: self._v_ob = self.dm[self.oid] except KeyError: return None return self._v_ob def __hash__(self): self = self() if self is None: raise TypeError('Weakly-referenced object has gone away') return hash(self) def __eq__(self, other): self = self() if self is None: raise TypeError('Weakly-referenced object has gone away') other = other() if other is None: raise TypeError('Weakly-referenced object has gone away') return self == other class PersistentWeakKeyDictionary(Persistent): """Persistent weak key dictionary This is akin to WeakKeyDictionaries. Note, however, that removal of items is extremely lazy. See below. We'll start by creating a PersistentWeakKeyDictionary and adding some persistent objects to it. >>> d = PersistentWeakKeyDictionary() >>> import ZODB.tests.util >>> p1 = ZODB.tests.util.P('p1') >>> p2 = ZODB.tests.util.P('p2') >>> p3 = ZODB.tests.util.P('p3') >>> d[p1] = 1 >>> d[p2] = 2 >>> d[p3] = 3 We'll create an extra persistent object that's not in the dict: >>> p4 = ZODB.tests.util.P('p4') Now we'll excercise iteration and item access: >>> l = [(str(k), d[k], d.get(k)) for k in d] >>> l.sort() >>> l [('P(p1)', 1, 1), ('P(p2)', 2, 2), ('P(p3)', 3, 3)] And the containment operator: >>> [p in d for p in [p1, p2, p3, p4]] [True, True, True, False] We can add the dict and the referenced objects to a database: >>> db = ZODB.tests.util.DB() >>> conn1 = db.open() >>> conn1.root()['p1'] = p1 >>> conn1.root()['d'] = d >>> conn1.root()['p2'] = p2 >>> conn1.root()['p3'] = p3 >>> ZODB.tests.util.commit() And things still work, as before: >>> l = [(str(k), d[k], d.get(k)) for k in d] >>> l.sort() >>> l [('P(p1)', 1, 1), ('P(p2)', 2, 2), ('P(p3)', 3, 3)] >>> [p in d for p in [p1, p2, p3, p4]] [True, True, True, False] Likewise, we can read the objects from another connection and things still work. >>> conn2 = db.open() >>> d = conn2.root()['d'] >>> p1 = conn2.root()['p1'] >>> p2 = conn2.root()['p2'] >>> p3 = conn2.root()['p3'] >>> l = [(str(k), d[k], d.get(k)) for k in d] >>> l.sort() >>> l [('P(p1)', 1, 1), ('P(p2)', 2, 2), ('P(p3)', 3, 3)] >>> [p in d for p in [p1, p2, p3, p4]] [True, True, True, False] Now, we'll delete one of the objects from the database, but *not* from the dictionary: >>> del conn2.root()['p2'] >>> ZODB.tests.util.commit() And pack the database, so that the no-longer referenced p2 is actually removed from the database. >>> ZODB.tests.util.pack(db) Now if we access the dictionary in a new connection, it no longer has p2: >>> conn3 = db.open() >>> d = conn3.root()['d'] >>> l = [(str(k), d[k], d.get(k)) for k in d] >>> l.sort() >>> l [('P(p1)', 1, 1), ('P(p3)', 3, 3)] It's worth nothing that that the versions of the dictionary in conn1 and conn2 still have p2, because p2 is still in the caches for those connections. Always explicitly close databases: :) >>> db.close() """ # TODO: It's expensive trying to load dead objects from the database. # It would be helpful if the data manager/connection cached these. def __init__(self, adict=None, **kwargs): self.data = {} if adict is not None: keys = getattr(adict, "keys", None) if keys is None: adict = dict(adict) self.update(adict) if kwargs: self.update(kwargs) def __getstate__(self): state = Persistent.__getstate__(self) state['data'] = state['data'].items() return state def __setstate__(self, state): state['data'] = dict([ (k, v) for (k, v) in state['data'] if k() is not None ]) Persistent.__setstate__(self, state) def __setitem__(self, key, value): self.data[WeakRef(key)] = value def __getitem__(self, key): return self.data[WeakRef(key)] def __delitem__(self, key): del self.data[WeakRef(key)] def get(self, key, default=None): """D.get(k[, d]) -> D[k] if k in D, else d. >>> import ZODB.tests.util >>> key = ZODB.tests.util.P("key") >>> missing = ZODB.tests.util.P("missing") >>> d = PersistentWeakKeyDictionary([(key, 1)]) >>> d.get(key) 1 >>> d.get(missing) >>> d.get(missing, 12) 12 """ return self.data.get(WeakRef(key), default) def __contains__(self, key): return WeakRef(key) in self.data def __iter__(self): for k in self.data: yield k() def update(self, adict): if isinstance(adict, PersistentWeakKeyDictionary): self.data.update(adict.update) else: for k, v in adict.items(): self.data[WeakRef(k)] = v # TODO: May need more methods, and tests. Zope-2.8.7-final/lib/python/persistent/cPersistence.h0000664000076400007640000000736010435502407021635 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #ifndef CPERSISTENCE_H #define CPERSISTENCE_H #include "Python.h" #include "ring.h" #define CACHE_HEAD \ PyObject_HEAD \ CPersistentRing ring_home; \ int non_ghost_count; struct ccobject_head_struct; typedef struct ccobject_head_struct PerCache; /* How big is a persistent object? 12 PyGC_Head is two pointers and an int 8 PyObject_HEAD is an int and a pointer 12 jar, oid, cache pointers 8 ring struct 8 serialno 4 state + extra (52) so far 4 dict ptr 4 weaklist ptr ------------------------- 64 only need 62, but obmalloc rounds up to multiple of eight Even a ghost requires 64 bytes. It's possible to make a persistent instance with slots and no dict, which changes the storage needed. */ #define cPersistent_HEAD \ PyObject_HEAD \ PyObject *jar; \ PyObject *oid; \ PerCache *cache; \ CPersistentRing ring; \ char serial[8]; \ signed char state; \ unsigned char reserved[3]; #define cPersistent_GHOST_STATE -1 #define cPersistent_UPTODATE_STATE 0 #define cPersistent_CHANGED_STATE 1 #define cPersistent_STICKY_STATE 2 typedef struct { cPersistent_HEAD } cPersistentObject; typedef void (*percachedelfunc)(PerCache *, PyObject *); typedef struct { PyTypeObject *pertype; getattrofunc getattro; setattrofunc setattro; int (*changed)(cPersistentObject*); void (*accessed)(cPersistentObject*); void (*ghostify)(cPersistentObject*); int (*setstate)(PyObject*); percachedelfunc percachedel; } cPersistenceCAPIstruct; #define cPersistenceType cPersistenceCAPI->pertype #ifndef DONT_USE_CPERSISTENCECAPI static cPersistenceCAPIstruct *cPersistenceCAPI; #endif #define cPersistanceModuleName "cPersistence" #define PER_TypeCheck(O) PyObject_TypeCheck((O), cPersistenceCAPI->pertype) #define PER_USE_OR_RETURN(O,R) {if((O)->state==cPersistent_GHOST_STATE && cPersistenceCAPI->setstate((PyObject*)(O)) < 0) return (R); else if ((O)->state==cPersistent_UPTODATE_STATE) (O)->state=cPersistent_STICKY_STATE;} #define PER_CHANGED(O) (cPersistenceCAPI->changed((cPersistentObject*)(O))) #define PER_GHOSTIFY(O) (cPersistenceCAPI->ghostify((cPersistentObject*)(O))) /* If the object is sticky, make it non-sticky, so that it can be ghostified. The value is not meaningful */ #define PER_ALLOW_DEACTIVATION(O) ((O)->state==cPersistent_STICKY_STATE && ((O)->state=cPersistent_UPTODATE_STATE)) #define PER_PREVENT_DEACTIVATION(O) ((O)->state==cPersistent_UPTODATE_STATE && ((O)->state=cPersistent_STICKY_STATE)) /* Make a persistent object usable from C by: - Making sure it is not a ghost - Making it sticky. IMPORTANT: If you call this and don't call PER_ALLOW_DEACTIVATION, your object will not be ghostified. PER_USE returns a 1 on success and 0 failure, where failure means error. */ #define PER_USE(O) \ (((O)->state != cPersistent_GHOST_STATE \ || (cPersistenceCAPI->setstate((PyObject*)(O)) >= 0)) \ ? (((O)->state==cPersistent_UPTODATE_STATE) \ ? ((O)->state=cPersistent_STICKY_STATE) : 1) : 0) #define PER_ACCESSED(O) (cPersistenceCAPI->accessed((cPersistentObject*)(O))) #endif Zope-2.8.7-final/lib/python/persistent/TimeStamp.c0000664000076400007640000002446210435502406021105 0ustar ajungajung/***************************************************************************** Copyright (c) 2001, 2004 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "Python.h" #include PyObject *TimeStamp_FromDate(int, int, int, int, int, double); PyObject *TimeStamp_FromString(const char *); static char TimeStampModule_doc[] = "A 64-bit TimeStamp used as a ZODB serial number.\n" "\n" "$Id: TimeStamp.c 41596 2006-02-11 21:03:06Z tseaver $\n"; typedef struct { PyObject_HEAD unsigned char data[8]; } TimeStamp; /* The first dimension of the arrays below is non-leapyear / leapyear */ static char month_len[2][12]={ {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31} }; static short joff[2][12] = { {0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334}, {0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335} }; static double gmoff=0; /* TODO: May be better (faster) to store in a file static. */ #define SCONV ((double)60) / ((double)(1<<16)) / ((double)(1<<16)) static int leap(int year) { return year % 4 == 0 && (year % 100 != 0 || year % 400 == 0); } static int days_in_month(int year, int month) { return month_len[leap(year)][month]; } static double TimeStamp_yad(int y) { double d, s; y -= 1900; d = (y - 1) * 365; if (y > 0) { s = 1.0; y -= 1; } else { s = -1.0; y = -y; } return d + s * (y / 4 - y / 100 + (y + 300) / 400); } static double TimeStamp_abst(int y, int mo, int d, int m, int s) { return (TimeStamp_yad(y) + joff[leap(y)][mo] + d) * 86400 + m * 60 + s; } static int TimeStamp_init_gmoff(void) { struct tm *t; time_t z=0; t = gmtime(&z); if (t == NULL) { PyErr_SetString(PyExc_SystemError, "gmtime failed"); return -1; } gmoff = TimeStamp_abst(t->tm_year+1900, t->tm_mon, t->tm_mday - 1, t->tm_hour * 60 + t->tm_min, t->tm_sec); return 0; } static void TimeStamp_dealloc(TimeStamp *ts) { PyObject_Del(ts); } static int TimeStamp_compare(TimeStamp *v, TimeStamp *w) { int cmp = memcmp(v->data, w->data, 8); if (cmp < 0) return -1; if (cmp > 0) return 1; return 0; } static long TimeStamp_hash(TimeStamp *self) { register unsigned char *p = (unsigned char *)self->data; register int len = 8; register long x = *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= 8; if (x == -1) x = -2; return x; } typedef struct { /* TODO: reverse-engineer what's in these things and comment them */ int y; int m; int d; int mi; } TimeStampParts; static void TimeStamp_unpack(TimeStamp *self, TimeStampParts *p) { unsigned long v; v = (self->data[0] * 16777216 + self->data[1] * 65536 + self->data[2] * 256 + self->data[3]); p->y = v / 535680 + 1900; p->m = (v % 535680) / 44640 + 1; p->d = (v % 44640) / 1440 + 1; p->mi = v % 1440; } static double TimeStamp_sec(TimeStamp *self) { unsigned int v; v = (self->data[4] * 16777216 + self->data[5] * 65536 + self->data[6] * 256 + self->data[7]); return SCONV * v; } static PyObject * TimeStamp_year(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyInt_FromLong(p.y); } static PyObject * TimeStamp_month(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyInt_FromLong(p.m); } static PyObject * TimeStamp_day(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyInt_FromLong(p.d); } static PyObject * TimeStamp_hour(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyInt_FromLong(p.mi / 60); } static PyObject * TimeStamp_minute(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyInt_FromLong(p.mi % 60); } static PyObject * TimeStamp_second(TimeStamp *self) { return PyFloat_FromDouble(TimeStamp_sec(self)); } static PyObject * TimeStamp_timeTime(TimeStamp *self) { TimeStampParts p; TimeStamp_unpack(self, &p); return PyFloat_FromDouble(TimeStamp_abst(p.y, p.m - 1, p.d - 1, p.mi, 0) + TimeStamp_sec(self) - gmoff); } static PyObject * TimeStamp_raw(TimeStamp *self) { return PyString_FromStringAndSize((const char*)self->data, 8); } static PyObject * TimeStamp_str(TimeStamp *self) { char buf[128]; TimeStampParts p; int len; TimeStamp_unpack(self, &p); len =sprintf(buf, "%4.4d-%2.2d-%2.2d %2.2d:%2.2d:%09.6f", p.y, p.m, p.d, p.mi / 60, p.mi % 60, TimeStamp_sec(self)); return PyString_FromStringAndSize(buf, len); } static PyObject * TimeStamp_laterThan(TimeStamp *self, PyObject *obj) { TimeStamp *o = NULL; TimeStampParts p; unsigned char new[8]; int i; if (obj->ob_type != self->ob_type) { PyErr_SetString(PyExc_TypeError, "expected TimeStamp object"); return NULL; } o = (TimeStamp *)obj; if (memcmp(self->data, o->data, 8) > 0) { Py_INCREF(self); return (PyObject *)self; } memcpy(new, o->data, 8); for (i = 7; i > 3; i--) { if (new[i] == 255) new[i] = 0; else { new[i]++; return TimeStamp_FromString((const char*)new); } } /* All but the first two bytes are the same. Need to increment the year, month, and day explicitly. */ TimeStamp_unpack(o, &p); if (p.mi >= 1439) { p.mi = 0; if (p.d == month_len[leap(p.y)][p.m - 1]) { p.d = 1; if (p.m == 12) { p.m = 1; p.y++; } else p.m++; } else p.d++; } else p.mi++; return TimeStamp_FromDate(p.y, p.m, p.d, p.mi / 60, p.mi % 60, 0); } static struct PyMethodDef TimeStamp_methods[] = { {"year", (PyCFunction)TimeStamp_year, METH_NOARGS}, {"minute", (PyCFunction)TimeStamp_minute, METH_NOARGS}, {"month", (PyCFunction)TimeStamp_month, METH_NOARGS}, {"day", (PyCFunction)TimeStamp_day, METH_NOARGS}, {"hour", (PyCFunction)TimeStamp_hour, METH_NOARGS}, {"second", (PyCFunction)TimeStamp_second, METH_NOARGS}, {"timeTime",(PyCFunction)TimeStamp_timeTime, METH_NOARGS}, {"laterThan", (PyCFunction)TimeStamp_laterThan, METH_O}, {"raw", (PyCFunction)TimeStamp_raw, METH_NOARGS}, {NULL, NULL}, }; static PyTypeObject TimeStamp_type = { PyObject_HEAD_INIT(NULL) 0, "persistent.TimeStamp", sizeof(TimeStamp), 0, (destructor)TimeStamp_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ (cmpfunc)TimeStamp_compare, /* tp_compare */ (reprfunc)TimeStamp_raw, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ (hashfunc)TimeStamp_hash, /* tp_hash */ 0, /* tp_call */ (reprfunc)TimeStamp_str, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ TimeStamp_methods, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ }; PyObject * TimeStamp_FromString(const char *buf) { /* buf must be exactly 8 characters */ TimeStamp *ts = (TimeStamp *)PyObject_New(TimeStamp, &TimeStamp_type); memcpy(ts->data, buf, 8); return (PyObject *)ts; } #define CHECK_RANGE(VAR, LO, HI) if ((VAR) < (LO) || (VAR) > (HI)) { \ return PyErr_Format(PyExc_ValueError, \ # VAR " must be between %d and %d: %d", \ (LO), (HI), (VAR)); \ } PyObject * TimeStamp_FromDate(int year, int month, int day, int hour, int min, double sec) { TimeStamp *ts = NULL; int d; unsigned int v; if (year < 1900) return PyErr_Format(PyExc_ValueError, "year must be greater than 1900: %d", year); CHECK_RANGE(month, 1, 12); d = days_in_month(year, month - 1); if (day < 1 || day > d) return PyErr_Format(PyExc_ValueError, "day must be between 1 and %d: %d", d, day); CHECK_RANGE(hour, 0, 23); CHECK_RANGE(min, 0, 59); /* Seconds are allowed to be anything, so chill If we did want to be pickly, 60 would be a better choice. if (sec < 0 || sec > 59) return PyErr_Format(PyExc_ValueError, "second must be between 0 and 59: %f", sec); */ ts = (TimeStamp *)PyObject_New(TimeStamp, &TimeStamp_type); v = (((year - 1900) * 12 + month - 1) * 31 + day - 1); v = (v * 24 + hour) * 60 + min; ts->data[0] = v / 16777216; ts->data[1] = (v % 16777216) / 65536; ts->data[2] = (v % 65536) / 256; ts->data[3] = v % 256; sec /= SCONV; v = (unsigned int)sec; ts->data[4] = v / 16777216; ts->data[5] = (v % 16777216) / 65536; ts->data[6] = (v % 65536) / 256; ts->data[7] = v % 256; return (PyObject *)ts; } PyObject * TimeStamp_TimeStamp(PyObject *obj, PyObject *args) { char *buf = NULL; int len = 0, y, mo, d, h = 0, m = 0; double sec = 0; if (PyArg_ParseTuple(args, "s#:TimeStamp", &buf, &len)) { if (len != 8) { PyErr_SetString(PyExc_ValueError, "8-character string expected"); return NULL; } return TimeStamp_FromString(buf); } PyErr_Clear(); if (!PyArg_ParseTuple(args, "iii|iid", &y, &mo, &d, &h, &m, &sec)) return NULL; return TimeStamp_FromDate(y, mo, d, h, m, sec); } static PyMethodDef TimeStampModule_functions[] = { {"TimeStamp", TimeStamp_TimeStamp, METH_VARARGS}, {NULL, NULL}, }; void initTimeStamp(void) { PyObject *m; if (TimeStamp_init_gmoff() < 0) return; m = Py_InitModule4("TimeStamp", TimeStampModule_functions, TimeStampModule_doc, NULL, PYTHON_API_VERSION); if (m == NULL) return; TimeStamp_type.ob_type = &PyType_Type; TimeStamp_type.tp_getattro = PyObject_GenericGetAttr; } Zope-2.8.7-final/lib/python/persistent/ring.h0000664000076400007640000000512010435502406020134 0ustar ajungajung/***************************************************************************** Copyright (c) 2003 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ /* Support routines for the doubly-linked list of cached objects. The cache stores a headed, doubly-linked, circular list of persistent objects, with space for the pointers allocated in the objects themselves. The cache stores the distinguished head of the list, which is not a valid persistent object. The other list members are non-ghost persistent objects, linked in LRU (least-recently used) order. The r_next pointers traverse the ring starting with the least recently used object. The r_prev pointers traverse the ring starting with the most recently used object. Obscure: While each object is pointed at twice by list pointers (once by its predecessor's r_next, again by its successor's r_prev), the refcount on the object is bumped only by 1. This leads to some possibly surprising sequences of incref and decref code. Note that since the refcount is bumped at least once, the list does hold a strong reference to each object in it. */ typedef struct CPersistentRing_struct { struct CPersistentRing_struct *r_prev; struct CPersistentRing_struct *r_next; } CPersistentRing; /* The list operations here take constant time independent of the * number of objects in the list: */ /* Add elt as the most recently used object. elt must not already be * in the list, although this isn't checked. */ void ring_add(CPersistentRing *ring, CPersistentRing *elt); /* Remove elt from the list. elt must already be in the list, although * this isn't checked. */ void ring_del(CPersistentRing *elt); /* elt must already be in the list, although this isn't checked. It's * unlinked from its current position, and relinked into the list as the * most recently used object (which is arguably the tail of the list * instead of the head -- but the name of this function could be argued * either way). This is equivalent to * * ring_del(elt); * ring_add(ring, elt); * * but may be a little quicker. */ void ring_move_to_head(CPersistentRing *ring, CPersistentRing *elt); Zope-2.8.7-final/lib/python/persistent/tests/0000775000076400007640000000000010435610645020175 5ustar ajungajungZope-2.8.7-final/lib/python/persistent/tests/persistent.txt0000664000076400007640000002235010435502406023133 0ustar ajungajungTests for persistent.Persistent =============================== This document is an extended doc test that covers the basics of the Persistent base class. The test expects a class named 'P' to be provided in its globals. The P class implements the Persistent interface. Test framework -------------- The class P needs to behave like ExampleP. (Note that the code below is *not* part of the tests.) class ExampleP(Persistent): def __init__(self): self.x = 0 def inc(self): self.x += 1 The tests use stub data managers. A data manager is responsible for loading and storing the state of a persistent object. It's stored in the _p_jar attribute of a persistent object. >>> class DM: ... def __init__(self): ... self.called = 0 ... def register(self, ob): ... self.called += 1 ... def setstate(self, ob): ... ob.__setstate__({'x': 42}) >>> class BrokenDM(DM): ... def register(self,ob): ... self.called += 1 ... raise NotImplementedError ... def setstate(self,ob): ... raise NotImplementedError >>> from persistent import Persistent Test Persistent without Data Manager ------------------------------------ First do some simple tests of a Persistent instance that does not have a data manager (_p_jar). >>> p = P() >>> p.x 0 >>> p._p_changed False >>> p._p_state 0 >>> p._p_jar >>> p._p_oid Verify that modifications have no effect on _p_state of _p_changed. >>> p.inc() >>> p.inc() >>> p.x 2 >>> p._p_changed False >>> p._p_state 0 Try all sorts of different ways to change the object's state. >>> p._p_deactivate() >>> p._p_state 0 >>> p._p_changed = True >>> p._p_state 0 >>> del p._p_changed >>> p._p_changed False >>> p._p_state 0 >>> p.x 2 Test Persistent with Data Manager --------------------------------- Next try some tests of an object with a data manager. The DM class is a simple testing stub. >>> p = P() >>> dm = DM() >>> p._p_oid = "00000012" >>> p._p_jar = dm >>> p._p_changed 0 >>> dm.called 0 Modifying the object marks it as changed and registers it with the data manager. Subsequent modifications don't have additional side-effects. >>> p.inc() >>> p._p_changed 1 >>> dm.called 1 >>> p.inc() >>> p._p_changed 1 >>> dm.called 1 It's not possible to deactivate a modified object. >>> p._p_deactivate() >>> p._p_changed 1 It is possible to invalidate it. That's the key difference between deactivation and invalidation. >>> p._p_invalidate() >>> p._p_state -1 Now that the object is a ghost, any attempt to modify it will require that it be unghosted first. The test data manager has the odd property that it sets the object's 'x' attribute to 42 when it is unghosted. >>> p.inc() >>> p.x 43 >>> dm.called 2 You can manually reset the changed field to False, although it's not clear why you would want to do that. The object changes to the UPTODATE state but retains its modifications. >>> p._p_changed = False >>> p._p_state 0 >>> p._p_changed False >>> p.x 43 >>> p.inc() >>> p._p_changed True >>> dm.called 3 __getstate__() and __setstate__() --------------------------------- The next several tests cover the __getstate__() and __setstate__() implementations. >>> p = P() >>> state = p.__getstate__() >>> isinstance(state, dict) True >>> state['x'] 0 >>> p._p_state 0 Calling setstate always leaves the object in the uptodate state? (I'm not entirely clear on this one.) >>> p.__setstate__({'x': 5}) >>> p._p_state 0 Assigning to a volatile attribute has no effect on the object state. >>> p._v_foo = 2 >>> p.__getstate__() {'x': 5} >>> p._p_state 0 The _p_serial attribute is not affected by calling setstate. >>> p._p_serial = "00000012" >>> p.__setstate__(p.__getstate__()) >>> p._p_serial '00000012' Change Ghost test ----------------- If an object is a ghost and it's _p_changed is set to True, it should have no effect. >>> p = P() >>> p._p_jar = DM() >>> p._p_oid = 1 >>> p._p_deactivate() >>> p._p_changed >>> p._p_state -1 >>> p._p_changed = True >>> p._p_changed >>> p._p_state -1 Activate, deactivate, and invalidate ------------------------------------ Some of these tests are redundant, but are included to make sure there are explicit and simple tests of _p_activate(), _p_deactivate(), and _p_invalidate(). >>> p = P() >>> p._p_oid = 1 >>> p._p_jar = DM() >>> p._p_deactivate() >>> p._p_state -1 >>> p._p_activate() >>> p._p_state 0 >>> p.x 42 >>> p.inc() >>> p.x 43 >>> p._p_state 1 >>> p._p_invalidate() >>> p._p_state -1 >>> p.x 42 Test failures ------------- The following tests cover various errors cases. When an object is modified, it registers with its data manager. If that registration fails, the exception is propagated and the object stays in the up-to-date state. It shouldn't change to the modified state, because it won't be saved when the transaction commits. >>> p = P() >>> p._p_oid = 1 >>> p._p_jar = BrokenDM() >>> p._p_state 0 >>> p._p_jar.called 0 >>> p._p_changed = 1 Traceback (most recent call last): ... NotImplementedError >>> p._p_jar.called 1 >>> p._p_state 0 Make sure that exceptions that occur inside the data manager's setstate() method propagate out to the caller. >>> p = P() >>> p._p_oid = 1 >>> p._p_jar = BrokenDM() >>> p._p_deactivate() >>> p._p_state -1 >>> p._p_activate() Traceback (most recent call last): ... NotImplementedError >>> p._p_state -1 Special test to cover layout of __dict__ ---------------------------------------- We once had a bug in the Persistent class that calculated an incorrect offset for the __dict__ attribute. It assigned __dict__ and _p_jar to the same location in memory. This is a simple test to make sure they have different locations. >>> p = P() >>> p.inc() >>> p.inc() >>> 'x' in p.__dict__ True >>> p._p_jar Inheritance and metaclasses --------------------------- Simple tests to make sure it's possible to inherit from the Persistent base class multiple times. There used to be metaclasses involved in Persistent that probably made this a more interesting test. >>> class A(Persistent): ... pass >>> class B(Persistent): ... pass >>> class C(A, B): ... pass >>> class D(object): ... pass >>> class E(D, B): ... pass >>> a = A() >>> b = B() >>> c = C() >>> d = D() >>> e = E() Also make sure that it's possible to define Persistent classes that have a custom metaclass. >>> class alternateMeta(type): ... type >>> class alternate(object): ... __metaclass__ = alternateMeta >>> class mixedMeta(alternateMeta, type): ... pass >>> class mixed(alternate, Persistent): ... pass >>> class mixed(Persistent, alternate): ... pass Basic type structure -------------------- >>> Persistent.__dictoffset__ 0 >>> Persistent.__weakrefoffset__ 0 >>> Persistent.__basicsize__ > object.__basicsize__ True >>> P.__dictoffset__ > 0 True >>> P.__weakrefoffset__ > 0 True >>> P.__dictoffset__ < P.__weakrefoffset__ True >>> P.__basicsize__ > Persistent.__basicsize__ True Slots ----- These are some simple tests of classes that have an __slots__ attribute. Some of the classes should have slots, others shouldn't. >>> class noDict(object): ... __slots__ = ['foo'] >>> class p_noDict(Persistent): ... __slots__ = ['foo'] >>> class p_shouldHaveDict(p_noDict): ... pass >>> p_noDict.__dictoffset__ 0 >>> x = p_noDict() >>> x.foo = 1 >>> x.foo 1 >>> x.bar = 1 Traceback (most recent call last): ... AttributeError: 'p_noDict' object has no attribute 'bar' >>> x._v_bar = 1 Traceback (most recent call last): ... AttributeError: 'p_noDict' object has no attribute '_v_bar' >>> x.__dict__ Traceback (most recent call last): ... AttributeError: 'p_noDict' object has no attribute '__dict__' The various _p_ attributes are unaffected by slots. >>> p._p_oid >>> p._p_jar >>> p._p_state 0 If the most-derived class does not specify >>> p_shouldHaveDict.__dictoffset__ > 0 True >>> x = p_shouldHaveDict() >>> isinstance(x.__dict__, dict) True Pickling -------- There's actually a substantial effort involved in making subclasses of Persistent work with plain-old pickle. The ZODB serialization layer never calls pickle on an object; it pickles the object's class description and its state as two separate pickles. >>> import pickle >>> p = P() >>> p.inc() >>> p2 = pickle.loads(pickle.dumps(p)) >>> p2.__class__ is P True >>> p2.x == p.x True We should also test that pickle works with custom getstate and setstate. Perhaps even reduce. The problem is that pickling depends on finding the class in a particular module, and classes defined here won't appear in any module. We could require each user of the tests to define a base class, but that might be tedious. Interfaces ---------- Some versions of Zope and ZODB have the zope.interfaces package available. If it is available, then persistent will be associated with several interfaces. It's hard to write a doctest test that runs the tests only if zope.interface is available, so this test looks a little unusual. One problem is that the assert statements won't do anything if you run with -O. >>> try: ... import zope.interface ... except ImportError: ... pass ... else: ... from persistent.interfaces import IPersistent ... assert IPersistent.implementedBy(Persistent) ... p = Persistent() ... assert IPersistent.providedBy(p) ... assert IPersistent.implementedBy(P) ... p = P() ... assert IPersistent.providedBy(p) Zope-2.8.7-final/lib/python/persistent/tests/test_persistent.py0000664000076400007640000000160310435502406024001 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## from zope.testing import doctest from persistent import Persistent class P(Persistent): def __init__(self): self.x = 0 def inc(self): self.x += 1 def test_suite(): return doctest.DocFileSuite("persistent.txt", globs={"P": P}) Zope-2.8.7-final/lib/python/persistent/tests/__init__.py0000664000076400007640000000001210435502406022272 0ustar ajungajung# package Zope-2.8.7-final/lib/python/persistent/tests/test_list.py0000664000076400007640000001375110435502406022563 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests for PersistentList """ import unittest l0 = [] l1 = [0] l2 = [0, 1] class TestPList(unittest.TestCase): def _getTargetClass(self): from persistent.list import PersistentList return PersistentList def test_volatile_attributes_not_persisted(self): # http://www.zope.org/Collectors/Zope/2052 m = self._getTargetClass()() m.foo = 'bar' m._v_baz = 'qux' state = m.__getstate__() self.failUnless('foo' in state) self.failIf('_v_baz' in state) def testTheWorld(self): # Test constructors pl = self._getTargetClass() u = pl() u0 = pl(l0) u1 = pl(l1) u2 = pl(l2) uu = pl(u) uu0 = pl(u0) uu1 = pl(u1) uu2 = pl(u2) v = pl(tuple(u)) class OtherList: def __init__(self, initlist): self.__data = initlist def __len__(self): return len(self.__data) def __getitem__(self, i): return self.__data[i] v0 = pl(OtherList(u0)) vv = pl("this is also a sequence") # Test __repr__ eq = self.assertEqual eq(str(u0), str(l0), "str(u0) == str(l0)") eq(repr(u1), repr(l1), "repr(u1) == repr(l1)") eq(`u2`, `l2`, "`u2` == `l2`") # Test __cmp__ and __len__ def mycmp(a, b): r = cmp(a, b) if r < 0: return -1 if r > 0: return 1 return r all = [l0, l1, l2, u, u0, u1, u2, uu, uu0, uu1, uu2] for a in all: for b in all: eq(mycmp(a, b), mycmp(len(a), len(b)), "mycmp(a, b) == mycmp(len(a), len(b))") # Test __getitem__ for i in range(len(u2)): eq(u2[i], i, "u2[i] == i") # Test __setitem__ uu2[0] = 0 uu2[1] = 100 try: uu2[2] = 200 except IndexError: pass else: raise TestFailed("uu2[2] shouldn't be assignable") # Test __delitem__ del uu2[1] del uu2[0] try: del uu2[0] except IndexError: pass else: raise TestFailed("uu2[0] shouldn't be deletable") # Test __getslice__ for i in range(-3, 4): eq(u2[:i], l2[:i], "u2[:i] == l2[:i]") eq(u2[i:], l2[i:], "u2[i:] == l2[i:]") for j in range(-3, 4): eq(u2[i:j], l2[i:j], "u2[i:j] == l2[i:j]") # Test __setslice__ for i in range(-3, 4): u2[:i] = l2[:i] eq(u2, l2, "u2 == l2") u2[i:] = l2[i:] eq(u2, l2, "u2 == l2") for j in range(-3, 4): u2[i:j] = l2[i:j] eq(u2, l2, "u2 == l2") uu2 = u2[:] uu2[:0] = [-2, -1] eq(uu2, [-2, -1, 0, 1], "uu2 == [-2, -1, 0, 1]") uu2[0:] = [] eq(uu2, [], "uu2 == []") # Test __contains__ for i in u2: self.failUnless(i in u2, "i in u2") for i in min(u2)-1, max(u2)+1: self.failUnless(i not in u2, "i not in u2") # Test __delslice__ uu2 = u2[:] del uu2[1:2] del uu2[0:1] eq(uu2, [], "uu2 == []") uu2 = u2[:] del uu2[1:] del uu2[:1] eq(uu2, [], "uu2 == []") # Test __add__, __radd__, __mul__ and __rmul__ #self.failUnless(u1 + [] == [] + u1 == u1, "u1 + [] == [] + u1 == u1") self.failUnless(u1 + [1] == u2, "u1 + [1] == u2") #self.failUnless([-1] + u1 == [-1, 0], "[-1] + u1 == [-1, 0]") self.failUnless(u2 == u2*1 == 1*u2, "u2 == u2*1 == 1*u2") self.failUnless(u2+u2 == u2*2 == 2*u2, "u2+u2 == u2*2 == 2*u2") self.failUnless(u2+u2+u2 == u2*3 == 3*u2, "u2+u2+u2 == u2*3 == 3*u2") # Test append u = u1[:] u.append(1) eq(u, u2, "u == u2") # Test insert u = u2[:] u.insert(0, -1) eq(u, [-1, 0, 1], "u == [-1, 0, 1]") # Test pop u = pl([0, -1, 1]) u.pop() eq(u, [0, -1], "u == [0, -1]") u.pop(0) eq(u, [-1], "u == [-1]") # Test remove u = u2[:] u.remove(1) eq(u, u1, "u == u1") # Test count u = u2*3 eq(u.count(0), 3, "u.count(0) == 3") eq(u.count(1), 3, "u.count(1) == 3") eq(u.count(2), 0, "u.count(2) == 0") # Test index eq(u2.index(0), 0, "u2.index(0) == 0") eq(u2.index(1), 1, "u2.index(1) == 1") try: u2.index(2) except ValueError: pass else: raise TestFailed("expected ValueError") # Test reverse u = u2[:] u.reverse() eq(u, [1, 0], "u == [1, 0]") u.reverse() eq(u, u2, "u == u2") # Test sort u = pl([1, 0]) u.sort() eq(u, u2, "u == u2") # Test extend u = u1[:] u.extend(u2) eq(u, u1 + u2, "u == u1 + u2") # Test iadd u = u1[:] u += u2 eq(u, u1 + u2, "u == u1 + u2") # Test imul u = u1[:] u *= 3 eq(u, u1 + u1 + u1, "u == u1 + u1 + u1") def test_suite(): return unittest.makeSuite(TestPList) if __name__ == "__main__": loader = unittest.TestLoader() unittest.main(testLoader=loader) Zope-2.8.7-final/lib/python/persistent/tests/test_mapping.py0000664000076400007640000001073410435502406023241 0ustar ajungajung############################################################################## # # Copyright (c) 2006 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test PersistentMapping """ import unittest l0 = {} l1 = {0:0} l2 = {0:0, 1:1} class MappingTests(unittest.TestCase): def _getTargetClass(self): from persistent.mapping import PersistentMapping return PersistentMapping def test_volatile_attributes_not_persisted(self): # http://www.zope.org/Collectors/Zope/2052 m = self._getTargetClass()() m.foo = 'bar' m._v_baz = 'qux' state = m.__getstate__() self.failUnless('foo' in state) self.failIf('_v_baz' in state) def testTheWorld(self): # Test constructors pm = self._getTargetClass() u = pm() u0 = pm(l0) u1 = pm(l1) u2 = pm(l2) uu = pm(u) uu0 = pm(u0) uu1 = pm(u1) uu2 = pm(u2) class OtherMapping: def __init__(self, initmapping): self.__data = initmapping def keys(self): return self.__data.keys() def items(self): return self.__data.items() v0 = pm(OtherMapping(u0)) vv = pm([(0, 0), (1, 1)]) # Test __repr__ eq = self.assertEqual eq(str(u0), str(l0), "str(u0) == str(l0)") eq(repr(u1), repr(l1), "repr(u1) == repr(l1)") eq(`u2`, `l2`, "`u2` == `l2`") # Test __cmp__ and __len__ def mycmp(a, b): r = cmp(a, b) if r < 0: return -1 if r > 0: return 1 return r all = [l0, l1, l2, u, u0, u1, u2, uu, uu0, uu1, uu2] for a in all: for b in all: eq(mycmp(a, b), mycmp(len(a), len(b)), "mycmp(a, b) == mycmp(len(a), len(b))") # Test __getitem__ for i in range(len(u2)): eq(u2[i], i, "u2[i] == i") # Test get for i in range(len(u2)): eq(u2.get(i), i, "u2.get(i) == i") eq(u2.get(i, 5), i, "u2.get(i, 5) == i") for i in min(u2)-1, max(u2)+1: eq(u2.get(i), None, "u2.get(i) == None") eq(u2.get(i, 5), 5, "u2.get(i, 5) == 5") # Test __setitem__ uu2[0] = 0 uu2[1] = 100 uu2[2] = 200 # Test __delitem__ del uu2[1] del uu2[0] try: del uu2[0] except KeyError: pass else: raise TestFailed("uu2[0] shouldn't be deletable") # Test __contains__ for i in u2: self.failUnless(i in u2, "i in u2") for i in min(u2)-1, max(u2)+1: self.failUnless(i not in u2, "i not in u2") # Test update l = {"a":"b"} u = pm(l) u.update(u2) for i in u: self.failUnless(i in l or i in u2, "i in l or i in u2") for i in l: self.failUnless(i in u, "i in u") for i in u2: self.failUnless(i in u, "i in u") # Test setdefault x = u2.setdefault(0, 5) eq(x, 0, "u2.setdefault(0, 5) == 0") x = u2.setdefault(5, 5) eq(x, 5, "u2.setdefault(5, 5) == 5") self.failUnless(5 in u2, "5 in u2") # Test pop x = u2.pop(1) eq(x, 1, "u2.pop(1) == 1") self.failUnless(1 not in u2, "1 not in u2") try: u2.pop(1) except KeyError: pass else: raise TestFailed("1 should not be poppable from u2") x = u2.pop(1, 7) eq(x, 7, "u2.pop(1, 7) == 7") # Test popitem items = u2.items() key, value = u2.popitem() self.failUnless((key, value) in items, "key, value in items") self.failUnless(key not in u2, "key not in u2") # Test clear u2.clear() eq(u2, {}, "u2 == {}") def test_suite(): return unittest.makeSuite(MappingTests) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/persistent/tests/test_overriding_attrs.py0000664000076400007640000002601410435502406025171 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Overriding attr methods This module tests and documents, through example, overriding attribute access methods. $Id: test_overriding_attrs.py 29891 2005-04-06 22:35:05Z tim_one $ """ from persistent import Persistent import transaction from ZODB.tests.util import DB class SampleOverridingGetattr(Persistent): """Example of overriding __getattr__ """ def __getattr__(self, name): """Get attributes that can't be gotten the usual way The __getattr__ method works pretty much the same for persistent classes as it does for other classes. No special handling is needed. If an object is a ghost, then it will be activated before __getattr__ is called. In this example, our objects returns a tuple with the attribute name, converted to upper case and the value of _p_changed, for any attribute that isn't handled by the default machinery. >>> o = SampleOverridingGetattr() >>> o._p_changed False >>> o._p_oid >>> o._p_jar >>> o.spam ('SPAM', False) >>> o.spam = 1 >>> o.spam 1 We'll save the object, so it can be deactivated: >>> db = DB() >>> conn = db.open() >>> conn.root()['o'] = o >>> transaction.commit() >>> o._p_deactivate() >>> o._p_changed And now, if we ask for an attribute it doesn't have, >>> o.eggs ('EGGS', False) And we see that the object was activated before calling the __getattr__ method. We always close databases after we use them: >>> db.close() """ # Don't pretend we have any special attributes. if name.startswith("__") and name.endswrith("__"): raise AttributeError, name else: return name.upper(), self._p_changed class SampleOverridingGetattributeSetattrAndDelattr(Persistent): """Example of overriding __getattribute__, __setattr__, and __delattr__ In this example, we'll provide an example that shows how to override the __getattribute__, __setattr__, and __delattr__ methods. We'll create a class that stores it's attributes in a secret dictionary within it's instance dictionary. The class will have the policy that variables with names starting with 'tmp_' will be volatile. """ def __init__(self, **kw): self.__dict__['__secret__'] = kw.copy() def __getattribute__(self, name): """Get an attribute value The __getattribute__ method is called for all attribute accesses. It overrides the attribute access support inherited from Persistent. Our sample class let's us provide initial values as keyword arguments to the constructor: >>> o = SampleOverridingGetattributeSetattrAndDelattr(x=1) >>> o._p_changed 0 >>> o._p_oid >>> o._p_jar >>> o.x 1 >>> o.y Traceback (most recent call last): ... AttributeError: y Next, we'll save the object in a database so that we can deactivate it: >>> db = DB() >>> conn = db.open() >>> conn.root()['o'] = o >>> transaction.commit() >>> o._p_deactivate() >>> o._p_changed And we'll get some data: >>> o.x 1 which activates the object: >>> o._p_changed 0 It works for missing attribes too: >>> o._p_deactivate() >>> o._p_changed >>> o.y Traceback (most recent call last): ... AttributeError: y >>> o._p_changed 0 See the very important note in the comment below! We always close databases after we use them: >>> db.close() """ ################################################################# # IMPORTANT! READ THIS! 8-> # # We *always* give Persistent a chance first. # Persistent handles certain special attributes, like _p_ # attributes. In particular, the base class handles __dict__ # and __class__. # # We call _p_getattr. If it returns True, then we have to # use Persistent.__getattribute__ to get the value. # ################################################################# if Persistent._p_getattr(self, name): return Persistent.__getattribute__(self, name) # Data should be in our secret dictionary: secret = self.__dict__['__secret__'] if name in secret: return secret[name] # Maybe it's a method: meth = getattr(self.__class__, name, None) if meth is None: raise AttributeError, name return meth.__get__(self, self.__class__) def __setattr__(self, name, value): """Set an attribute value The __setattr__ method is called for all attribute assignments. It overrides the attribute assignment support inherited from Persistent. Implementors of __setattr__ methods: 1. Must call Persistent._p_setattr first to allow it to handle some attributes and to make sure that the object is activated if necessary, and 2. Must set _p_changed to mark objects as changed. See the comments in the source below. >>> o = SampleOverridingGetattributeSetattrAndDelattr() >>> o._p_changed 0 >>> o._p_oid >>> o._p_jar >>> o.x Traceback (most recent call last): ... AttributeError: x >>> o.x = 1 >>> o.x 1 Because the implementation doesn't store attributes directly in the instance dictionary, we don't have a key for the attribute: >>> 'x' in o.__dict__ False Next, we'll save the object in a database so that we can deactivate it: >>> db = DB() >>> conn = db.open() >>> conn.root()['o'] = o >>> transaction.commit() >>> o._p_deactivate() >>> o._p_changed We'll modify an attribute >>> o.y = 2 >>> o.y 2 which reactivates it, and markes it as modified, because our implementation marked it as modified: >>> o._p_changed 1 Now, if commit: >>> transaction.commit() >>> o._p_changed 0 And deactivate the object: >>> o._p_deactivate() >>> o._p_changed and then set a variable with a name starting with 'tmp_', The object will be activated, but not marked as modified, because our __setattr__ implementation doesn't mark the object as changed if the name starts with 'tmp_': >>> o.tmp_foo = 3 >>> o._p_changed 0 >>> o.tmp_foo 3 We always close databases after we use them: >>> db.close() """ ################################################################# # IMPORTANT! READ THIS! 8-> # # We *always* give Persistent a chance first. # Persistent handles certain special attributes, like _p_ # attributes. # # We call _p_setattr. If it returns True, then we are done. # It has already set the attribute. # ################################################################# if Persistent._p_setattr(self, name, value): return self.__dict__['__secret__'][name] = value if not name.startswith('tmp_'): self._p_changed = 1 def __delattr__(self, name): """Delete an attribute value The __delattr__ method is called for all attribute deletions. It overrides the attribute deletion support inherited from Persistent. Implementors of __delattr__ methods: 1. Must call Persistent._p_delattr first to allow it to handle some attributes and to make sure that the object is activated if necessary, and 2. Must set _p_changed to mark objects as changed. See the comments in the source below. >>> o = SampleOverridingGetattributeSetattrAndDelattr( ... x=1, y=2, tmp_z=3) >>> o._p_changed 0 >>> o._p_oid >>> o._p_jar >>> o.x 1 >>> del o.x >>> o.x Traceback (most recent call last): ... AttributeError: x Next, we'll save the object in a database so that we can deactivate it: >>> db = DB() >>> conn = db.open() >>> conn.root()['o'] = o >>> transaction.commit() >>> o._p_deactivate() >>> o._p_changed If we delete an attribute: >>> del o.y The object is activated. It is also marked as changed because our implementation marked it as changed. >>> o._p_changed 1 >>> o.y Traceback (most recent call last): ... AttributeError: y >>> o.tmp_z 3 Now, if commit: >>> transaction.commit() >>> o._p_changed 0 And deactivate the object: >>> o._p_deactivate() >>> o._p_changed and then delete a variable with a name starting with 'tmp_', The object will be activated, but not marked as modified, because our __delattr__ implementation doesn't mark the object as changed if the name starts with 'tmp_': >>> del o.tmp_z >>> o._p_changed 0 >>> o.tmp_z Traceback (most recent call last): ... AttributeError: tmp_z We always close databases after we use them: >>> db.close() """ ################################################################# # IMPORTANT! READ THIS! 8-> # # We *always* give Persistent a chance first. # Persistent handles certain special attributes, like _p_ # attributes. # # We call _p_delattr. If it returns True, then we are done. # It has already deleted the attribute. # ################################################################# if Persistent._p_delattr(self, name): return del self.__dict__['__secret__'][name] if not name.startswith('tmp_'): self._p_changed = 1 def test_suite(): from zope.testing.doctest import DocTestSuite return DocTestSuite() Zope-2.8.7-final/lib/python/persistent/tests/test_pickle.py0000664000076400007640000001434510435502406023057 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Basic pickling tests $Id: test_pickle.py 29891 2005-04-06 22:35:05Z tim_one $ """ from persistent import Persistent import pickle def print_dict(d): d = d.items() d.sort() print '{%s}' % (', '.join( [('%r: %r' % (k, v)) for (k, v) in d] )) def cmpattrs(self, other, *attrs): for attr in attrs: if attr[:3] in ('_v_', '_p_'): continue c = cmp(getattr(self, attr, None), getattr(other, attr, None)) if c: return c return 0 class Simple(Persistent): def __init__(self, name, **kw): self.__name__ = name self.__dict__.update(kw) self._v_favorite_color = 'blue' self._p_foo = 'bar' def __cmp__(self, other): return cmpattrs(self, other, '__class__', *(self.__dict__.keys())) def test_basic_pickling(): """ >>> x = Simple('x', aaa=1, bbb='foo') >>> print_dict(x.__getstate__()) {'__name__': 'x', 'aaa': 1, 'bbb': 'foo'} >>> f, (c,), state = x.__reduce__() >>> f.__name__ '__newobj__' >>> f.__module__ 'copy_reg' >>> c.__name__ 'Simple' >>> print_dict(state) {'__name__': 'x', 'aaa': 1, 'bbb': 'foo'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.__setstate__({'z': 1}) >>> x.__dict__ {'z': 1} """ class Custom(Simple): def __new__(cls, x, y): r = Persistent.__new__(cls) r.x, r.y = x, y return r def __init__(self, x, y): self.a = 42 def __getnewargs__(self): return self.x, self.y def __getstate__(self): return self.a def __setstate__(self, a): self.a = a def test_pickling_w_overrides(): """ >>> x = Custom('x', 'y') >>> x.a = 99 >>> (f, (c, ax, ay), a) = x.__reduce__() >>> f.__name__ '__newobj__' >>> f.__module__ 'copy_reg' >>> c.__name__ 'Custom' >>> ax, ay, a ('x', 'y', 99) >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ class Slotted(Persistent): __slots__ = 's1', 's2', '_p_splat', '_v_eek' def __init__(self, s1, s2): self.s1, self.s2 = s1, s2 self._v_eek = 1 self._p_splat = 2 class SubSlotted(Slotted): __slots__ = 's3', 's4' def __init__(self, s1, s2, s3): Slotted.__init__(self, s1, s2) self.s3 = s3 def __cmp__(self, other): return cmpattrs(self, other, '__class__', 's1', 's2', 's3', 's4') def test_pickling_w_slots_only(): """ >>> x = SubSlotted('x', 'y', 'z') >>> d, s = x.__getstate__() >>> d >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> d >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ class SubSubSlotted(SubSlotted): def __init__(self, s1, s2, s3, **kw): SubSlotted.__init__(self, s1, s2, s3) self.__dict__.update(kw) self._v_favorite_color = 'blue' self._p_foo = 'bar' def __cmp__(self, other): return cmpattrs(self, other, '__class__', 's1', 's2', 's3', 's4', *(self.__dict__.keys())) def test_pickling_w_slots(): """ >>> x = SubSubSlotted('x', 'y', 'z', aaa=1, bbb='foo') >>> d, s = x.__getstate__() >>> print_dict(d) {'aaa': 1, 'bbb': 'foo'} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> print_dict(d) {'aaa': 1, 'bbb': 'foo'} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ def test_pickling_w_slots_w_empty_dict(): """ >>> x = SubSubSlotted('x', 'y', 'z') >>> d, s = x.__getstate__() >>> print_dict(d) {} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 >>> x.s4 = 'spam' >>> d, s = x.__getstate__() >>> print_dict(d) {} >>> print_dict(s) {'s1': 'x', 's2': 'y', 's3': 'z', 's4': 'spam'} >>> pickle.loads(pickle.dumps(x)) == x 1 >>> pickle.loads(pickle.dumps(x, 0)) == x 1 >>> pickle.loads(pickle.dumps(x, 1)) == x 1 >>> pickle.loads(pickle.dumps(x, 2)) == x 1 """ from zope.testing.doctest import DocTestSuite import unittest def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/persistent/tests/test_wref.py0000664000076400007640000000155010435502406022545 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ $Id: test_wref.py 29891 2005-04-06 22:35:05Z tim_one $ """ import unittest from zope.testing.doctest import DocTestSuite def test_suite(): return DocTestSuite('persistent.wref') if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/persistent/tests/testPersistent.py0000664000076400007640000001701610435502406023607 0ustar ajungajung############################################################################# # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import pickle import time import unittest from persistent import Persistent, GHOST, UPTODATE, CHANGED, STICKY from persistent.cPickleCache import PickleCache from persistent.TimeStamp import TimeStamp from ZODB.utils import p64 class Jar(object): """Testing stub for _p_jar attribute.""" def __init__(self): self.cache = PickleCache(self) self.oid = 1 self.registered = {} def add(self, obj): obj._p_oid = p64(self.oid) self.oid += 1 obj._p_jar = self self.cache[obj._p_oid] = obj def close(self): pass # the following methods must be implemented to be a jar def setklassstate(self): # I don't know what this method does, but the pickle cache # constructor calls it. pass def register(self, obj): self.registered[obj] = 1 def setstate(self, obj): # Trivial setstate() implementation that just re-initializes # the object. This isn't what setstate() is supposed to do, # but it suffices for the tests. obj.__class__.__init__(obj) class P(Persistent): pass class H1(Persistent): def __init__(self): self.n = 0 def __getattr__(self, attr): self.n += 1 return self.n class H2(Persistent): def __init__(self): self.n = 0 def __getattribute__(self, attr): supergetattr = super(H2, self).__getattribute__ try: return supergetattr(attr) except AttributeError: n = supergetattr("n") self.n = n + 1 return n + 1 class PersistenceTest(unittest.TestCase): def setUp(self): self.jar = Jar() def tearDown(self): self.jar.close() def testOidAndJarAttrs(self): obj = P() self.assertEqual(obj._p_oid, None) obj._p_oid = 12 self.assertEqual(obj._p_oid, 12) del obj._p_oid self.jar.add(obj) # Can't change oid of cache object. def deloid(): del obj._p_oid self.assertRaises(ValueError, deloid) def setoid(): obj._p_oid = 12 self.assertRaises(ValueError, setoid) def deloid(): del obj._p_jar self.assertRaises(ValueError, deloid) def setoid(): obj._p_jar = 12 self.assertRaises(ValueError, setoid) def testChangedAndState(self): obj = P() self.jar.add(obj) # The value returned for _p_changed can be one of: # 0 -- it is not changed # 1 -- it is changed # None -- it is a ghost obj.x = 1 self.assertEqual(obj._p_changed, 1) self.assertEqual(obj._p_state, CHANGED) self.assert_(obj in self.jar.registered) obj._p_changed = 0 self.assertEqual(obj._p_changed, 0) self.assertEqual(obj._p_state, UPTODATE) self.jar.registered.clear() obj._p_changed = 1 self.assertEqual(obj._p_changed, 1) self.assertEqual(obj._p_state, CHANGED) self.assert_(obj in self.jar.registered) # setting obj._p_changed to None ghostifies if the # object is in the up-to-date state, but not otherwise. obj._p_changed = None self.assertEqual(obj._p_changed, 1) self.assertEqual(obj._p_state, CHANGED) obj._p_changed = 0 # Now it's a ghost. obj._p_changed = None self.assertEqual(obj._p_changed, None) self.assertEqual(obj._p_state, GHOST) obj = P() self.jar.add(obj) obj._p_changed = 1 # You can transition directly from modified to ghost if # you delete the _p_changed attribute. del obj._p_changed self.assertEqual(obj._p_changed, None) self.assertEqual(obj._p_state, GHOST) def testStateReadonly(self): # make sure we can't write to _p_state; we don't want yet # another way to change state! obj = P() def setstate(value): obj._p_state = value self.assertRaises(TypeError, setstate, GHOST) self.assertRaises(TypeError, setstate, UPTODATE) self.assertRaises(TypeError, setstate, CHANGED) self.assertRaises(TypeError, setstate, STICKY) def testInvalidate(self): obj = P() self.jar.add(obj) self.assertEqual(obj._p_changed, 0) self.assertEqual(obj._p_state, UPTODATE) obj._p_invalidate() self.assertEqual(obj._p_changed, None) self.assertEqual(obj._p_state, GHOST) obj._p_activate() obj.x = 1 obj._p_invalidate() self.assertEqual(obj._p_changed, None) self.assertEqual(obj._p_state, GHOST) def testSerial(self): noserial = "\000" * 8 obj = P() self.assertEqual(obj._p_serial, noserial) def set(val): obj._p_serial = val self.assertRaises(ValueError, set, 1) self.assertRaises(ValueError, set, "0123") self.assertRaises(ValueError, set, "012345678") self.assertRaises(ValueError, set, u"01234567") obj._p_serial = "01234567" del obj._p_serial self.assertEqual(obj._p_serial, noserial) def testMTime(self): obj = P() self.assertEqual(obj._p_mtime, None) t = int(time.time()) ts = TimeStamp(*time.gmtime(t)[:6]) obj._p_serial = repr(ts) self.assertEqual(obj._p_mtime, t) self.assert_(isinstance(obj._p_mtime, float)) def testPicklable(self): obj = P() obj.attr = "test" s = pickle.dumps(obj) obj2 = pickle.loads(s) self.assertEqual(obj.attr, obj2.attr) def testGetattr(self): obj = H1() self.assertEqual(obj.larry, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) self.jar.add(obj) obj._p_deactivate() # The simple Jar used for testing re-initializes the object. self.assertEqual(obj.larry, 1) # The getattr hook modified the object, so it should now be # in the changed state. self.assertEqual(obj._p_changed, 1) self.assertEqual(obj._p_state, CHANGED) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) def testGetattribute(self): obj = H2() self.assertEqual(obj.larry, 1) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) self.jar.add(obj) obj._p_deactivate() # The simple Jar used for testing re-initializes the object. self.assertEqual(obj.larry, 1) # The getattr hook modified the object, so it should now be # in the changed state. self.assertEqual(obj._p_changed, 1) self.assertEqual(obj._p_state, CHANGED) self.assertEqual(obj.curly, 2) self.assertEqual(obj.moe, 3) # TODO: Need to decide how __setattr__ and __delattr__ should work, # then write tests. def test_suite(): return unittest.makeSuite(PersistenceTest) Zope-2.8.7-final/lib/python/persistent/tests/test_PickleCache.py0000664000076400007640000000263110435502406023736 0ustar ajungajung############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Unit tests for PickleCache $Id: test_PickleCache.py 29891 2005-04-06 22:35:05Z tim_one $ """ class DummyConnection: def setklassstate(self, obj): """Method used by PickleCache.""" def test_delitem(): """ >>> from persistent import PickleCache >>> conn = DummyConnection() >>> cache = PickleCache(conn) >>> del cache[''] Traceback (most recent call last): ... KeyError: '' >>> from persistent import Persistent >>> p = Persistent() >>> p._p_oid = 'foo' >>> p._p_jar = conn >>> cache['foo'] = p >>> del cache['foo'] """ from zope.testing.doctest import DocTestSuite import unittest def test_suite(): return unittest.TestSuite(( DocTestSuite(), )) if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/persistent/tests/persistenttestbase.py0000664000076400007640000002541110435502406024500 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import unittest from persistent import Persistent from persistent.interfaces import IPersistent try: import zope.interface except ImportError: interfaces = False else: interfaces = True class Test(unittest.TestCase): klass = None # override in subclass def testSaved(self): p = self.klass() p._p_oid = '\0\0\0\0\0\0hi' dm = DM() p._p_jar = dm self.assertEqual(p._p_changed, 0) self.assertEqual(dm.called, 0) p.inc() self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 1) p.inc() self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 1) p._p_deactivate() self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 1) p._p_deactivate() self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 1) del p._p_changed # deal with current cPersistence implementation if p._p_changed != 3: self.assertEqual(p._p_changed, None) self.assertEqual(dm.called, 1) p.inc() self.assertEqual(p.x, 43) self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 2) p._p_changed = 0 self.assertEqual(p._p_changed, 0) self.assertEqual(dm.called, 2) self.assertEqual(p.x, 43) p.inc() self.assertEqual(p._p_changed, 1) self.assertEqual(dm.called, 3) def testUnsaved(self): p = self.klass() self.assertEqual(p.x, 0) self.assertEqual(p._p_changed, 0) self.assertEqual(p._p_jar, None) self.assertEqual(p._p_oid, None) p.inc() p.inc() self.assertEqual(p.x, 2) self.assertEqual(p._p_changed, 0) p._p_deactivate() self.assertEqual(p._p_changed, 0) p._p_changed = 1 self.assertEqual(p._p_changed, 0) p._p_deactivate() self.assertEqual(p._p_changed, 0) del p._p_changed self.assertEqual(p._p_changed, 0) if self.has_dict: self.failUnless(p.__dict__) self.assertEqual(p.x, 2) def testState(self): p = self.klass() self.assertEqual(p.__getstate__(), {'x': 0}) self.assertEqual(p._p_changed, 0) p.__setstate__({'x':5}) self.assertEqual(p._p_changed, 0) if self.has_dict: p._v_foo = 2 self.assertEqual(p.__getstate__(), {'x': 5}) self.assertEqual(p._p_changed, 0) def testSetStateSerial(self): p = self.klass() p._p_serial = '00000012' p.__setstate__(p.__getstate__()) self.assertEqual(p._p_serial, '00000012') def testDirectChanged(self): p = self.klass() p._p_oid = 1 dm = DM() p._p_jar = dm self.assertEqual(p._p_changed, 0) self.assertEqual(dm.called, 0) p._p_changed = 1 self.assertEqual(dm.called, 1) def testGhostChanged(self): # An object is a ghost, and it's _p_changed it set to True. # This assignment should have no effect. p = self.klass() p._p_oid = 1 dm = DM() p._p_jar = dm p._p_deactivate() self.assertEqual(p._p_changed, None) p._p_changed = True self.assertEqual(p._p_changed, None) def testRegistrationFailure(self): p = self.klass() p._p_oid = 1 dm = BrokenDM() p._p_jar = dm self.assertEqual(p._p_changed, 0) self.assertEqual(dm.called, 0) try: p._p_changed = 1 except NotImplementedError: pass else: raise AssertionError("Exception not propagated") self.assertEqual(dm.called, 1) self.assertEqual(p._p_changed, 0) def testLoadFailure(self): p = self.klass() p._p_oid = 1 dm = BrokenDM() p._p_jar = dm p._p_deactivate() # make it a ghost try: p._p_activate() except NotImplementedError: pass else: raise AssertionError("Exception not propagated") self.assertEqual(p._p_changed, None) def testActivate(self): p = self.klass() dm = DM() p._p_oid = 1 p._p_jar = dm p._p_changed = 0 p._p_deactivate() # Unsure: does this really test the activate method? p._p_activate() self.assertEqual(p._p_changed, 0) self.assertEqual(p.x, 42) def testDeactivate(self): p = self.klass() dm = DM() p._p_oid = 1 p._p_deactivate() # this deactive has no effect self.assertEqual(p._p_changed, 0) p._p_jar = dm p._p_changed = 0 p._p_deactivate() self.assertEqual(p._p_changed, None) p._p_activate() self.assertEqual(p._p_changed, 0) self.assertEqual(p.x, 42) if interfaces: def testInterface(self): self.assert_(IPersistent.implementedBy(Persistent), "%s does not implement IPersistent" % Persistent) p = Persistent() self.assert_(IPersistent.providedBy(p), "%s does not implement IPersistent" % p) self.assert_(IPersistent.implementedBy(P), "%s does not implement IPersistent" % P) p = self.klass() self.assert_(IPersistent.providedBy(p), "%s does not implement IPersistent" % p) def testDataManagerAndAttributes(self): # Test to cover an odd bug where the instance __dict__ was # set at the same location as the data manager in the C type. p = P() p.inc() p.inc() self.assert_('x' in p.__dict__) self.assert_(p._p_jar is None) def testMultipleInheritance(self): # make sure it is possible to inherit from two different # subclasses of persistent. class A(Persistent): pass class B(Persistent): pass class C(A, B): pass class D(object): pass class E(D, B): pass def testMultipleMeta(self): # make sure it's possible to define persistent classes # with a base whose metaclass is different class alternateMeta(type): pass class alternate(object): __metaclass__ = alternateMeta class mixedMeta(alternateMeta, type): pass class mixed(alternate,Persistent): __metaclass__ = mixedMeta def testSlots(self): # Verify that Persistent classes behave the same way # as pure Python objects where '__slots__' and '__dict__' # are concerned. class noDict(object): __slots__ = ['foo'] class shouldHaveDict(noDict): pass class p_noDict(Persistent): __slots__ = ['foo'] class p_shouldHaveDict(p_noDict): pass self.assertEqual(noDict.__dictoffset__, 0) self.assertEqual(p_noDict.__dictoffset__, 0) self.assert_(shouldHaveDict.__dictoffset__ <> 0) self.assert_(p_shouldHaveDict.__dictoffset__ <> 0) def testBasicTypeStructure(self): # test that a persistent class has a sane C type structure # use P (defined below) as simplest example self.assertEqual(Persistent.__dictoffset__, 0) self.assertEqual(Persistent.__weakrefoffset__, 0) self.assert_(Persistent.__basicsize__ > object.__basicsize__) self.assert_(P.__dictoffset__) self.assert_(P.__weakrefoffset__) self.assert_(P.__dictoffset__ < P.__weakrefoffset__) self.assert_(P.__basicsize__ > Persistent.__basicsize__) # Unsure: Can anyone defend/explain the test below? The tests classes defined # here don't define __call__, so this weird test will always pass, but to what # end? If a klass is given that happens to define __call__, the test *may* # mysteriously fail. Who cares? ## def testDeactivateErrors(self): ## p = self.klass() ## p._p_oid = '\0\0\0\0\0\0hi' ## dm = DM() ## p._p_jar = dm ## def typeerr(*args, **kwargs): ## self.assertRaises(TypeError, p, *args, **kwargs) ## typeerr(1) ## typeerr(1, 2) ## typeerr(spam=1) ## typeerr(spam=1, force=1) ## p._p_changed = True ## class Err(object): ## def __nonzero__(self): ## raise RuntimeError ## typeerr(force=Err()) class P(Persistent): def __init__(self): self.x = 0 def inc(self): self.x += 1 class P2(P): def __getstate__(self): return 42 def __setstate__(self, v): self.v = v class B(Persistent): __slots__ = ["x", "_p_serial"] def __init__(self): self.x = 0 def inc(self): self.x += 1 def __getstate__(self): return {'x': self.x} def __setstate__(self, state): self.x = state['x'] class DM: def __init__(self): self.called = 0 def register(self, ob): self.called += 1 def setstate(self, ob): ob.__setstate__({'x': 42}) class BrokenDM(DM): def register(self,ob): self.called += 1 raise NotImplementedError def setstate(self,ob): raise NotImplementedError class PersistentTest(Test): klass = P has_dict = 1 def testPicklable(self): import pickle p = self.klass() p.inc() p2 = pickle.loads(pickle.dumps(p)) self.assertEqual(p2.__class__, self.klass) # verify that the inc is reflected: self.assertEqual(p2.x, p.x) # This assertion would be invalid. Interfaces # are compared by identity and copying doesn't # preserve identity. We would get false negatives due # to the differing identities of the original and copied # PersistentInterface: # self.assertEqual(p2.__dict__, p.__dict__) def testPicklableWCustomState(self): import pickle p = P2() p2 = pickle.loads(pickle.dumps(p)) self.assertEqual(p2.__class__, P2); self.assertEqual(p2.__dict__, {'v': 42}) class BasePersistentTest(Test): klass = B has_dict = 0 Zope-2.8.7-final/lib/python/persistent/DEPENDENCIES.cfg0000664000076400007640000000012510435502406021333 0ustar ajungajung# the following are needed by the tests transaction ZODB zope.interface zope.testing Zope-2.8.7-final/lib/python/transaction/0000775000076400007640000000000010435610645017160 5ustar ajungajungZope-2.8.7-final/lib/python/transaction/__init__.py0000664000076400007640000000244010435502414021263 0ustar ajungajung############################################################################ # # Copyright (c) 2001, 2002, 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################ """Exported transaction functions. $Id: __init__.py 30129 2005-04-24 01:29:02Z jim $ """ from transaction._transaction import Transaction from transaction._manager import TransactionManager, ThreadTransactionManager manager = ThreadTransactionManager() get = manager.get begin = manager.begin commit = manager.commit abort = manager.abort savepoint = manager.savepoint def get_transaction(): from ZODB.utils import deprecated36 deprecated36(""" use transaction.get() instead of get_transaction(). transaction.commit() is a shortcut spelling of transaction.get().commit(), and transaction.abort() of transaction.get().abort().""") return get() Zope-2.8.7-final/lib/python/transaction/README.txt0000664000076400007640000000105310435502414020647 0ustar ajungajungThis package is currently a facade of the ZODB.Transaction module. It exists to support: - Application code that uses the ZODB 4 transaction API - ZODB4-style data managers (transaction.interfaces.IDataManager) Note that the data manager API, transaction.interfaces.IDataManager, is syntactically simple, but semantically complex. The semantics were not easy to express in the interface. This could probably use more work. The semantics are presented in detail through examples of a sample data manager in transaction.tests.test_SampleDataManager. Zope-2.8.7-final/lib/python/transaction/interfaces.py0000664000076400007640000003304410435502414021653 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Transaction Interfaces $Id: interfaces.py 30534 2005-05-27 18:07:18Z tim_one $ """ import zope.interface class ITransactionManager(zope.interface.Interface): """An object that manages a sequence of transactions. Applications use transaction managers to establish transaction boundaries. """ def begin(): """Begin a new transaction. If an existing transaction is in progress, it will be aborted. The newTransaction() method of registered synchronizers is called, passing the new transaction object. """ def get(): """Get the current transaction. """ def commit(): """Commit the current transaction. """ def abort(): """Abort the current transaction. """ def savepoint(optimistic=False): """Create a savepoint from the current transaction. If the optimistic argument is true, then data managers that don't support savepoints can be used, but an error will be raised if the savepoint is rolled back. An ISavepoint object is returned. """ def registerSynch(synch): """Register an ISynchronizer. Synchronizers are notified about some major events in a transaction's life. See ISynchronizer for details. """ def unregisterSynch(synch): """Unregister an ISynchronizer. Synchronizers are notified about some major events in a transaction's life. See ISynchronizer for details. """ class ITransaction(zope.interface.Interface): """Object representing a running transaction. Objects with this interface may represent different transactions during their lifetime (.begin() can be called to start a new transaction using the same instance, although that example is deprecated and will go away in ZODB 3.6). """ user = zope.interface.Attribute( """A user name associated with the transaction. The format of the user name is defined by the application. The value is of Python type str. Storages record the user value, as meta-data, when a transaction commits. A storage may impose a limit on the size of the value; behavior is undefined if such a limit is exceeded (for example, a storage may raise an exception, or truncate the value). """) description = zope.interface.Attribute( """A textual description of the transaction. The value is of Python type str. Method note() is the intended way to set the value. Storages record the description, as meta-data, when a transaction commits. A storage may impose a limit on the size of the description; behavior is undefined if such a limit is exceeded (for example, a storage may raise an exception, or truncate the value). """) def commit(): """Finalize the transaction. This executes the two-phase commit algorithm for all IDataManager objects associated with the transaction. """ def abort(): """Abort the transaction. This is called from the application. This can only be called before the two-phase commit protocol has been started. """ def savepoint(optimistic=False): """Create a savepoint. If the optimistic argument is true, then data managers that don't support savepoints can be used, but an error will be raised if the savepoint is rolled back. An ISavepoint object is returned. """ def join(datamanager): """Add a data manager to the transaction. `datamanager` must provide the transactions.interfaces.IDataManager interface. """ def note(text): """Add text to the transaction description. This modifies the `.description` attribute; see its docs for more detail. First surrounding whitespace is stripped from `text`. If `.description` is currently an empty string, then the stripped text becomes its value, else two newlines and the stripped text are appended to `.description`. """ def setUser(user_name, path="/"): """Set the user name. path should be provided if needed to further qualify the identified user. This is a convenience method used by Zope. It sets the .user attribute to str(path) + " " + str(user_name). This sets the `.user` attribute; see its docs for more detail. """ def setExtendedInfo(name, value): """Add extension data to the transaction. name is the name of the extension property to set, of Python type str; value must be pickleable. Multiple calls may be made to set multiple extension properties, provided the names are distinct. Storages record the extension data, as meta-data, when a transaction commits. A storage may impose a limit on the size of extension data; behavior is undefined if such a limit is exceeded (for example, a storage may raise an exception, or remove `` pairs). """ def beforeCommitHook(hook, *args, **kws): """Register a hook to call before the transaction is committed. The specified hook function will be called after the transaction's commit method has been called, but before the commit process has been started. The hook will be passed the specified positional and keyword arguments. Multiple hooks can be registered and will be called in the order they were registered (first registered, first called). This method can also be called from a hook: an executing hook can register more hooks. Applications should take care to avoid creating infinite loops by recursively registering hooks. Hooks are called only for a top-level commit. A subtransaction commit does not call any hooks. If the transaction is aborted, hooks are not called, and are discarded. Calling a hook "consumes" its registration too: hook registrations do not persist across transactions. If it's desired to call the same hook on every transaction commit, then beforeCommitHook() must be called with that hook during every transaction; in such a case consider registering a synchronizer object via a TransactionManager's registerSynch() method instead. """ def getBeforeCommitHooks(): """Return iterable producing the registered beforeCommit hooks. A triple (hook, args, kws) is produced for each registered hook. The hooks are produced in the order in which they were registered. """ class ITransactionDeprecated(zope.interface.Interface): """Deprecated parts of the transaction API.""" def begin(info=None): """Begin a new transaction. If the transaction is in progress, it is aborted and a new transaction is started using the same transaction object. """ # TODO: deprecate this for 3.6. def register(object): """Register the given object for transaction control.""" class IDataManager(zope.interface.Interface): """Objects that manage transactional storage. These objects may manage data for other objects, or they may manage non-object storages, such as relational databases. For example, a ZODB.Connection. Note that when some data is modified, that data's data manager should join a transaction so that data can be committed when the user commits the transaction. """ transaction_manager = zope.interface.Attribute( """The transaction manager (TM) used by this data manager. This is a public attribute, intended for read-only use. The value is an instance of ITransactionManager, typically set by the data manager's constructor. """) def abort(transaction): """Abort a transaction and forget all changes. Abort must be called outside of a two-phase commit. Abort is called by the transaction manager to abort transactions that are not yet in a two-phase commit. """ # Two-phase commit protocol. These methods are called by the ITransaction # object associated with the transaction being committed. The sequence # of calls normally follows this regular expression: # tpc_begin commit tpc_vote (tpc_finish | tpc_abort) def tpc_begin(transaction): """Begin commit of a transaction, starting the two-phase commit. transaction is the ITransaction instance associated with the transaction being committed. """ def commit(transaction): """Commit modifications to registered objects. Save changes to be made persistent if the transaction commits (if tpc_finish is called later). If tpc_abort is called later, changes must not persist. This includes conflict detection and handling. If no conflicts or errors occur, the data manager should be prepared to make the changes persist when tpc_finish is called. """ def tpc_vote(transaction): """Verify that a data manager can commit the transaction. This is the last chance for a data manager to vote 'no'. A data manager votes 'no' by raising an exception. transaction is the ITransaction instance associated with the transaction being committed. """ def tpc_finish(transaction): """Indicate confirmation that the transaction is done. Make all changes to objects modified by this transaction persist. transaction is the ITransaction instance associated with the transaction being committed. This should never fail. If this raises an exception, the database is not expected to maintain consistency; it's a serious error. """ def tpc_abort(transaction): """Abort a transaction. This is called by a transaction manager to end a two-phase commit on the data manager. Abandon all changes to objects modified by this transaction. transaction is the ITransaction instance associated with the transaction being committed. This should never fail. """ def sortKey(): """Return a key to use for ordering registered DataManagers. ZODB uses a global sort order to prevent deadlock when it commits transactions involving multiple resource managers. The resource manager must define a sortKey() method that provides a global ordering for resource managers. """ # Alternate version: #"""Return a consistent sort key for this connection. # #This allows ordering multiple connections that use the same storage in #a consistent manner. This is unique for the lifetime of a connection, #which is good enough to avoid ZEO deadlocks. #""" class ISavepointDataManager(IDataManager): def savepoint(): """Return a data-manager savepoint (IDataManagerSavepoint). """ class IDataManagerSavepoint(zope.interface.Interface): """Savepoint for data-manager changes for use in transaction savepoints. Datamanager savepoints are used by, and only by, transaction savepoints. Note that data manager savepoints don't have any notion of, or responsibility for, validity. It isn't the responsibility of data-manager savepoints to prevent multiple rollbacks or rollbacks after transaction termination. Preventing invalid savepoint rollback is the responsibility of transaction rollbacks. Application code should never use data-manager savepoints. """ def rollback(): """Rollback any work done since the savepoint. """ class ISavepoint(zope.interface.Interface): """A transaction savepoint. """ def rollback(): """Rollback any work done since the savepoint. InvalidSavepointRollbackError is raised if the savepoint isn't valid. """ valid = zope.interface.Attribute( "Boolean indicating whether the savepoint is valid") class InvalidSavepointRollbackError(Exception): """Attempt to rollback an invalid savepoint. A savepoint may be invalid because: - The surrounding transaction has committed or aborted. - An earlier savepoint in the same transaction has been rolled back. """ class ISynchronizer(zope.interface.Interface): """Objects that participate in the transaction-boundary notification API. """ def beforeCompletion(transaction): """Hook that is called by the transaction at the start of a commit. """ def afterCompletion(transaction): """Hook that is called by the transaction after completing a commit. """ def newTransaction(transaction): """Hook that is called at the start of a transaction. This hook is called when, and only when, a transaction manager's begin() method is called explictly. """ Zope-2.8.7-final/lib/python/transaction/_transaction.py0000664000076400007640000006165410435502414022224 0ustar ajungajung############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################ """Transaction objects manage resources for an individual activity. Compatibility issues -------------------- The implementation of Transaction objects involves two layers of backwards compatibility, because this version of transaction supports both ZODB 3 and ZODB 4. Zope is evolving towards the ZODB4 interfaces. Transaction has two methods for a resource manager to call to participate in a transaction -- register() and join(). join() takes a resource manager and adds it to the list of resources. register() is for backwards compatibility. It takes a persistent object and registers its _p_jar attribute. TODO: explain adapter Subtransactions --------------- Note: Suntransactions are deprecated! Use savepoint/rollback instead. A subtransaction applies the transaction notion recursively. It allows a set of modifications within a transaction to be committed or aborted as a group. A subtransaction is a strictly local activity; its changes are not visible to any other database connection until the top-level transaction commits. In addition to its use to organize a large transaction, subtransactions can be used to optimize memory use. ZODB must keep modified objects in memory until a transaction commits and it can write the changes to the storage. A subtransaction uses a temporary disk storage for its commits, allowing modified objects to be flushed from memory when the subtransaction commits. The commit() and abort() methods take an optional subtransaction argument that defaults to false. If it is a true, the operation is performed on a subtransaction. Subtransactions add a lot of complexity to the transaction implementation. Some resource managers support subtransactions, but they are not required to. (ZODB Connection is the only standard resource manager that supports subtransactions.) Resource managers that do support subtransactions implement abort_sub() and commit_sub() methods and support a second argument to tpc_begin(). The second argument to tpc_begin() indicates that a subtransaction commit is beginning (if it is true). In a subtransaction, there is no tpc_vote() call, because sub-transactions don't need 2-phase commit. If a sub-transaction abort or commit fails, we can abort the outer transaction. The tpc_finish() or tpc_abort() call applies just to that subtransaction. Once a resource manager is involved in a subtransaction, all subsequent transactions will be treated as subtransactions until abort_sub() or commit_sub() is called. abort_sub() will undo all the changes of the subtransactions. commit_sub() will begin a top-level transaction and store all the changes from subtransactions. After commit_sub(), the transaction must still call tpc_vote() and tpc_finish(). If the resource manager does not support subtransactions, nothing happens when the subtransaction commits. Instead, the resource manager is put on a list of managers to commit when the actual top-level transaction commits. If this happens, it will not be possible to abort subtransactions. Two-phase commit ---------------- A transaction commit involves an interaction between the transaction object and one or more resource managers. The transaction manager calls the following four methods on each resource manager; it calls tpc_begin() on each resource manager before calling commit() on any of them. 1. tpc_begin(txn) 2. commit(txn) 3. tpc_vote(txn) 4. tpc_finish(txn) Subtransaction commit --------------------- Note: Subtransactions are deprecated! When a subtransaction commits, the protocol is different. 1. tpc_begin() is passed a second argument, which indicates that a subtransaction is being committed. 2. tpc_vote() is not called. Once a subtransaction has been committed, the top-level transaction commit will start with a commit_sub() call instead of a tpc_begin() call. Before-commit hook --------------- Sometimes, applications want to execute some code when a transaction is committed. For example, one might want to delay object indexing until a transaction commits, rather than indexing every time an object is changed. Or someone might want to check invariants only after a set of operations. A pre-commit hook is available for such use cases, just use beforeCommitHook() passing it a callable and arguments. The callable will be called with its arguments at the start of the commit (but not for substransaction commits). Error handling -------------- When errors occur during two-phase commit, the transaction manager aborts all the resource managers. The specific methods it calls depend on whether the error occurs before or after the call to tpc_vote() on that transaction manager. If the resource manager has not voted, then the resource manager will have one or more uncommitted objects. There are two cases that lead to this state; either the transaction manager has not called commit() for any objects on this resource manager or the call that failed was a commit() for one of the objects of this resource manager. For each uncommitted object, including the object that failed in its commit(), call abort(). Once uncommitted objects are aborted, tpc_abort() or abort_sub() is called on each resource manager. Synchronization --------------- You can register sychronization objects (synchronizers) with the tranasction manager. The synchronizer must implement beforeCompletion() and afterCompletion() methods. The transaction manager calls beforeCompletion() when it starts a top-level two-phase commit. It calls afterCompletion() when a top-level transaction is committed or aborted. The methods are passed the current Transaction as their only argument. """ import logging import sys import thread import warnings import weakref import traceback from cStringIO import StringIO from zope import interface from transaction import interfaces # Sigh. In the maze of __init__.py's, ZODB.__init__.py takes 'get' # out of transaction.__init__.py, in order to stuff the 'get_transaction' # alias in __builtin__. So here in _transaction.py, we can't import # exceptions from ZODB.POSException at top level (we're imported by # our __init__.py, which is imported by ZODB's __init__, so the ZODB # package isn't well-formed when we're first imported). # from ZODB.POSException import TransactionError, TransactionFailedError _marker = object() # The point of this is to avoid hiding exceptions (which the builtin # hasattr() does). def myhasattr(obj, attr): return getattr(obj, attr, _marker) is not _marker class Status: # ACTIVE is the initial state. ACTIVE = "Active" COMMITTING = "Committing" COMMITTED = "Committed" # commit() or commit(True) raised an exception. All further attempts # to commit or join this transaction will raise TransactionFailedError. COMMITFAILED = "Commit failed" class Transaction(object): interface.implements(interfaces.ITransaction, interfaces.ITransactionDeprecated) # Assign an index to each savepoint so we can invalidate later savepoints # on rollback. The first index assigned is 1, and it goes up by 1 each # time. _savepoint_index = 0 # If savepoints are used, keep a weak key dict of them. This maps a # savepoint to its index (see above). _savepoint2index = None # Remember the savepoint for the last subtransaction. _subtransaction_savepoint = None # Meta data. ._extension is also metadata, but is initialized to an # emtpy dict in __init__. user = "" description = "" def __init__(self, synchronizers=None, manager=None): self.status = Status.ACTIVE # List of resource managers, e.g. MultiObjectResourceAdapters. self._resources = [] # Weak set of synchronizer objects to call. if synchronizers is None: from ZODB.utils import WeakSet synchronizers = WeakSet() self._synchronizers = synchronizers self._manager = manager # _adapters: Connection/_p_jar -> MultiObjectResourceAdapter[Sub] self._adapters = {} self._voted = {} # id(Connection) -> boolean, True if voted # _voted and other dictionaries use the id() of the resource # manager as a key, because we can't guess whether the actual # resource managers will be safe to use as dict keys. # The user, description, and _extension attributes are accessed # directly by storages, leading underscore notwithstanding. self._extension = {} self.log = logging.getLogger("txn.%d" % thread.get_ident()) self.log.debug("new transaction") # If a commit fails, the traceback is saved in _failure_traceback. # If another attempt is made to commit, TransactionFailedError is # raised, incorporating this traceback. self._failure_traceback = None # Holds (hook, args, kws) triples added by beforeCommitHook. # TODO: in Python 2.4, change to collections.deque; lists can be # inefficient for FIFO access of this kind. self._before_commit = [] # Raise TransactionFailedError, due to commit()/join()/register() # getting called when the current transaction has already suffered # a commit/savepoint failure. def _prior_operation_failed(self): from ZODB.POSException import TransactionFailedError assert self._failure_traceback is not None raise TransactionFailedError("An operation previously failed, " "with traceback:\n\n%s" % self._failure_traceback.getvalue()) def join(self, resource): if self.status is Status.COMMITFAILED: self._prior_operation_failed() # doesn't return if self.status is not Status.ACTIVE: # TODO: Should it be possible to join a committing transaction? # I think some users want it. raise ValueError("expected txn status %r, but it's %r" % ( Status.ACTIVE, self.status)) # TODO: the prepare check is a bit of a hack, perhaps it would # be better to use interfaces. If this is a ZODB4-style # resource manager, it needs to be adapted, too. if myhasattr(resource, "prepare"): # TODO: deprecate 3.6 resource = DataManagerAdapter(resource) self._resources.append(resource) if self._savepoint2index: # A data manager has joined a transaction *after* a savepoint # was created. A couple of things are different in this case: # # 1. We need to add its savepoint to all previous savepoints. # so that if they are rolled back, we roll this one back too. # # 2. We don't actually need to ask the data manager for a # savepoint: because it's just joining, we can just abort it to # roll back to the current state, so we simply use an # AbortSavepoint. datamanager_savepoint = AbortSavepoint(resource, self) for transaction_savepoint in self._savepoint2index.keys(): transaction_savepoint._savepoints.append( datamanager_savepoint) def savepoint(self, optimistic=False): if self.status is Status.COMMITFAILED: self._prior_operation_failed() # doesn't return, it raises try: savepoint = Savepoint(self, optimistic, *self._resources) except: self._cleanup(self._resources) self._saveCommitishError() # reraises! if self._savepoint2index is None: self._savepoint2index = weakref.WeakKeyDictionary() self._savepoint_index += 1 self._savepoint2index[savepoint] = self._savepoint_index return savepoint # Remove and invalidate all savepoints we know about with an index # larger than `savepoint`'s. This is what's needed when a rollback # _to_ `savepoint` is done. def _remove_and_invalidate_after(self, savepoint): savepoint2index = self._savepoint2index index = savepoint2index[savepoint] # use items() to make copy to avoid mutating while iterating for savepoint, i in savepoint2index.items(): if i > index: savepoint.transaction = None # invalidate del savepoint2index[savepoint] # Invalidate and forget about all savepoints. def _invalidate_all_savepoints(self): for savepoint in self._savepoint2index.keys(): savepoint.transaction = None # invalidate self._savepoint2index.clear() def register(self, obj): # The old way of registering transaction participants. # # register() is passed either a persisent object or a # resource manager like the ones defined in ZODB.DB. # If it is passed a persistent object, that object should # be stored when the transaction commits. For other # objects, the object implements the standard two-phase # commit protocol. manager = getattr(obj, "_p_jar", obj) if manager is None: raise ValueError("Register with no manager") adapter = self._adapters.get(manager) if adapter is None: adapter = MultiObjectResourceAdapter(manager) adapter.objects.append(obj) self._adapters[manager] = adapter self.join(adapter) else: # TODO: comment out this expensive assert later # Use id() to guard against proxies. assert id(obj) not in map(id, adapter.objects) adapter.objects.append(obj) def begin(self): from ZODB.utils import deprecated36 deprecated36("Transaction.begin() should no longer be used; use " "the begin() method of a transaction manager.") if (self._resources or self._synchronizers): self.abort() # Else aborting wouldn't do anything, except if _manager is non-None, # in which case it would do nothing besides uselessly free() this # transaction. def commit(self, subtransaction=False): if self._savepoint2index: self._invalidate_all_savepoints() if subtransaction: # TODO deprecate subtransactions self._subtransaction_savepoint = self.savepoint(1) return if self.status is Status.COMMITFAILED: self._prior_operation_failed() # doesn't return self._callBeforeCommitHooks() self._synchronizers.map(lambda s: s.beforeCompletion(self)) self.status = Status.COMMITTING try: self._commitResources() except: self._saveCommitishError() # This raises! self.status = Status.COMMITTED if self._manager: self._manager.free(self) self._synchronizers.map(lambda s: s.afterCompletion(self)) self.log.debug("commit") def _saveCommitishError(self): self.status = Status.COMMITFAILED # Save the traceback for TransactionFailedError. ft = self._failure_traceback = StringIO() t, v, tb = sys.exc_info() # Record how we got into commit(). traceback.print_stack(sys._getframe(1), None, ft) # Append the stack entries from here down to the exception. traceback.print_tb(tb, None, ft) # Append the exception type and value. ft.writelines(traceback.format_exception_only(t, v)) raise t, v, tb def getBeforeCommitHooks(self): return iter(self._before_commit) def beforeCommitHook(self, hook, *args, **kws): self._before_commit.append((hook, args, kws)) def _callBeforeCommitHooks(self): # Call all hooks registered, allowing further registrations # during processing. while self._before_commit: hook, args, kws = self._before_commit.pop(0) hook(*args, **kws) def _commitResources(self): # Execute the two-phase commit protocol. L = list(self._resources) L.sort(rm_cmp) try: for rm in L: rm.tpc_begin(self) for rm in L: rm.commit(self) self.log.debug("commit %r" % rm) for rm in L: rm.tpc_vote(self) self._voted[id(rm)] = True try: for rm in L: rm.tpc_finish(self) except: # TODO: do we need to make this warning stronger? # TODO: It would be nice if the system could be configured # to stop committing transactions at this point. self.log.critical("A storage error occured during the second " "phase of the two-phase commit. Resources " "may be in an inconsistent state.") raise except: # If an error occurs committing a transaction, we try # to revert the changes in each of the resource managers. t, v, tb = sys.exc_info() try: self._cleanup(L) finally: self._synchronizers.map(lambda s: s.afterCompletion(self)) raise t, v, tb def _cleanup(self, L): # Called when an exception occurs during tpc_vote or tpc_finish. for rm in L: if id(rm) not in self._voted: try: rm.abort(self) except Exception: self.log.error("Error in abort() on manager %s", rm, exc_info=sys.exc_info()) for rm in L: try: rm.tpc_abort(self) except Exception: self.log.error("Error in tpc_abort() on manager %s", rm, exc_info=sys.exc_info()) def abort(self, subtransaction=False): if subtransaction: # TODO deprecate subtransactions. if not self._subtransaction_savepoint: raise interfaces.InvalidSavepointRollbackError if self._subtransaction_savepoint.valid: self._subtransaction_savepoint.rollback() # We're supposed to be able to call abort(1) multiple # times without additional effect, so mark the subtxn # savepoint invalid now. self._subtransaction_savepoint.transaction = None assert not self._subtransaction_savepoint.valid return if self._savepoint2index: self._invalidate_all_savepoints() self._synchronizers.map(lambda s: s.beforeCompletion(self)) tb = None for rm in self._resources: try: rm.abort(self) except: if tb is None: t, v, tb = sys.exc_info() self.log.error("Failed to abort resource manager: %s", rm, exc_info=sys.exc_info()) if self._manager: self._manager.free(self) self._synchronizers.map(lambda s: s.afterCompletion(self)) self.log.debug("abort") if tb is not None: raise t, v, tb def note(self, text): text = text.strip() if self.description: self.description += "\n\n" + text else: self.description = text def setUser(self, user_name, path="/"): self.user = "%s %s" % (path, user_name) def setExtendedInfo(self, name, value): self._extension[name] = value # TODO: We need a better name for the adapters. class MultiObjectResourceAdapter(object): """Adapt the old-style register() call to the new-style join(). With join(), a resource mananger like a Connection registers with the transaction manager. With register(), an individual object is passed to register(). """ def __init__(self, jar): self.manager = jar self.objects = [] self.ncommitted = 0 def __repr__(self): return "<%s for %s at %s>" % (self.__class__.__name__, self.manager, id(self)) def sortKey(self): return self.manager.sortKey() def tpc_begin(self, txn): self.manager.tpc_begin(txn) def tpc_finish(self, txn): self.manager.tpc_finish(txn) def tpc_abort(self, txn): self.manager.tpc_abort(txn) def commit(self, txn): for o in self.objects: self.manager.commit(o, txn) self.ncommitted += 1 def tpc_vote(self, txn): self.manager.tpc_vote(txn) def abort(self, txn): tb = None for o in self.objects: try: self.manager.abort(o, txn) except: # Capture the first exception and re-raise it after # aborting all the other objects. if tb is None: t, v, tb = sys.exc_info() txn.log.error("Failed to abort object: %s", object_hint(o), exc_info=sys.exc_info()) if tb is not None: raise t, v, tb def rm_cmp(rm1, rm2): return cmp(rm1.sortKey(), rm2.sortKey()) def object_hint(o): """Return a string describing the object. This function does not raise an exception. """ from ZODB.utils import oid_repr # We should always be able to get __class__. klass = o.__class__.__name__ # oid would be great, but may this isn't a persistent object. oid = getattr(o, "_p_oid", _marker) if oid is not _marker: oid = oid_repr(oid) return "%s oid=%s" % (klass, oid) # TODO: deprecate for 3.6. class DataManagerAdapter(object): """Adapt zodb 4-style data managers to zodb3 style Adapt transaction.interfaces.IDataManager to ZODB.interfaces.IPureDatamanager """ # Note that it is pretty important that this does not have a _p_jar # attribute. This object will be registered with a zodb3 TM, which # will then try to get a _p_jar from it, using it as the default. # (Objects without a _p_jar are their own data managers.) def __init__(self, datamanager): self._datamanager = datamanager # TODO: I'm not sure why commit() doesn't do anything def commit(self, transaction): # We don't do anything here because ZODB4-style data managers # didn't have a separate commit step pass def abort(self, transaction): self._datamanager.abort(transaction) def tpc_begin(self, transaction): # We don't do anything here because ZODB4-style data managers # didn't have a separate tpc_begin step pass def tpc_abort(self, transaction): self._datamanager.abort(transaction) def tpc_finish(self, transaction): self._datamanager.commit(transaction) def tpc_vote(self, transaction): self._datamanager.prepare(transaction) def sortKey(self): return self._datamanager.sortKey() class Savepoint: """Transaction savepoint. Transaction savepoints coordinate savepoints for data managers participating in a transaction. """ interface.implements(interfaces.ISavepoint) valid = property(lambda self: self.transaction is not None) def __init__(self, transaction, optimistic, *resources): self.transaction = transaction self._savepoints = savepoints = [] for datamanager in resources: try: savepoint = datamanager.savepoint except AttributeError: if not optimistic: raise TypeError("Savepoints unsupported", datamanager) savepoint = NoRollbackSavepoint(datamanager) else: savepoint = savepoint() savepoints.append(savepoint) def rollback(self): transaction = self.transaction if transaction is None: raise interfaces.InvalidSavepointRollbackError transaction._remove_and_invalidate_after(self) try: for savepoint in self._savepoints: savepoint.rollback() except: # Mark the transaction as failed. transaction._saveCommitishError() # reraises! class AbortSavepoint: def __init__(self, datamanager, transaction): self.datamanager = datamanager self.transaction = transaction def rollback(self): self.datamanager.abort(self.transaction) class NoRollbackSavepoint: def __init__(self, datamanager): self.datamanager = datamanager def rollback(self): raise TypeError("Savepoints unsupported", self.datamanager) Zope-2.8.7-final/lib/python/transaction/_manager.py0000664000076400007640000001167610435502414021310 0ustar ajungajung############################################################################ # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################ """A TransactionManager controls transaction boundaries. It coordinates application code and resource managers, so that they are associated with the right transaction. """ import thread from transaction._transaction import Transaction # We have to remember sets of synch objects, especially Connections. # But we don't want mere registration with a transaction manager to # keep a synch object alive forever; in particular, it's common # practice not to explicitly close Connection objects, and keeping # a Connection alive keeps a potentially huge number of other objects # alive (e.g., the cache, and everything reachable from it too). # Therefore we use "weak sets" internally. # # Obscure: because of the __init__.py maze, we can't import WeakSet # at top level here. # Call the ISynchronizer newTransaction() method on every element of # WeakSet synchs. # A transaction manager needs to do this whenever begin() is called. # Since it would be good if tm.get() returned the new transaction while # newTransaction() is running, calling this has to be delayed until after # the transaction manager has done whatever it needs to do to make its # get() return the new txn. def _new_transaction(txn, synchs): if synchs: synchs.map(lambda s: s.newTransaction(txn)) # Important: we must always pass a WeakSet (even if empty) to the Transaction # constructor: synchronizers are registered with the TM, but the # ISynchronizer xyzCompletion() methods are called by Transactions without # consulting the TM, so we need to pass a mutable collection of synchronizers # so that Transactions "see" synchronizers that get registered after the # Transaction object is constructed. class TransactionManager(object): def __init__(self): from ZODB.utils import WeakSet self._txn = None self._synchs = WeakSet() def begin(self): if self._txn is not None: self._txn.abort() txn = self._txn = Transaction(self._synchs, self) _new_transaction(txn, self._synchs) return txn def get(self): if self._txn is None: self._txn = Transaction(self._synchs, self) return self._txn def free(self, txn): assert txn is self._txn self._txn = None def registerSynch(self, synch): self._synchs.add(synch) def unregisterSynch(self, synch): self._synchs.remove(synch) def commit(self, sub=False): self.get().commit(sub) def abort(self, sub=False): self.get().abort(sub) def savepoint(self, optimistic=False): return self.get().savepoint(optimistic) class ThreadTransactionManager(TransactionManager): """Thread-aware transaction manager. Each thread is associated with a unique transaction. """ def __init__(self): # _threads maps thread ids to transactions self._txns = {} # _synchs maps a thread id to a WeakSet of registered synchronizers. # The WeakSet is passed to the Transaction constructor, because the # latter needs to call the synchronizers when it commits. self._synchs = {} def begin(self): tid = thread.get_ident() txn = self._txns.get(tid) if txn is not None: txn.abort() synchs = self._synchs.get(tid) if synchs is None: from ZODB.utils import WeakSet synchs = self._synchs[tid] = WeakSet() txn = self._txns[tid] = Transaction(synchs, self) _new_transaction(txn, synchs) return txn def get(self): tid = thread.get_ident() txn = self._txns.get(tid) if txn is None: synchs = self._synchs.get(tid) if synchs is None: from ZODB.utils import WeakSet synchs = self._synchs[tid] = WeakSet() txn = self._txns[tid] = Transaction(synchs, self) return txn def free(self, txn): tid = thread.get_ident() assert txn is self._txns.get(tid) del self._txns[tid] def registerSynch(self, synch): tid = thread.get_ident() ws = self._synchs.get(tid) if ws is None: from ZODB.utils import WeakSet ws = self._synchs[tid] = WeakSet() ws.add(synch) def unregisterSynch(self, synch): tid = thread.get_ident() ws = self._synchs[tid] ws.remove(synch) Zope-2.8.7-final/lib/python/transaction/tests/0000775000076400007640000000000010435610645020322 5ustar ajungajungZope-2.8.7-final/lib/python/transaction/tests/__init__.py0000664000076400007640000000000210435502414022415 0ustar ajungajung# Zope-2.8.7-final/lib/python/transaction/tests/savepointsample.py0000664000076400007640000001563410435502414024111 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Savepoint data manager implementation example. Sample data manager implementation that illustrates how to implement savepoints. See savepoint.txt in the transaction package. $Id: savepointsample.py 33273 2005-07-11 22:04:18Z tim_one $ """ import UserDict from zope import interface import transaction.interfaces class SampleDataManager(UserDict.DictMixin): """Sample implementation of data manager that doesn't support savepoints This data manager stores named simple values, like strings and numbers. """ interface.implements(transaction.interfaces.IDataManager) def __init__(self, transaction_manager=None): if transaction_manager is None: # Use the thread-local transaction manager if none is provided: transaction_manager = transaction.manager self.transaction_manager = transaction_manager # Our committed and uncommitted data: self.committed = {} self.uncommitted = self.committed.copy() # Our transaction state: # # If our uncommitted data is modified, we'll join a transaction # and keep track of the transaction we joined. Any commit # related messages we get should be for this same transaction self.transaction = None # What phase, if any, of two-phase commit we are in: self.tpc_phase = None ####################################################################### # Provide a mapping interface to uncommitted data. We provide # a basic subset of the interface. DictMixin does the rest. def __getitem__(self, name): return self.uncommitted[name] def __setitem__(self, name, value): self._join() # join the current transaction, if we haven't already self.uncommitted[name] = value def __delitem__(self, name): self._join() # join the current transaction, if we haven't already del self.uncommitted[name] def keys(self): return self.uncommitted.keys() # ####################################################################### ####################################################################### # Transaction methods def _join(self): # If this is the first change in the transaction, join the transaction if self.transaction is None: self.transaction = self.transaction_manager.get() self.transaction.join(self) def _resetTransaction(self): self.transaction = None self.tpc_phase = None def abort(self, transaction): """Throw away changes made before the commit process has started """ assert ((transaction is self.transaction) or (self.transaction is None) ), "Must not change transactions" assert self.tpc_phase is None, "Must be called outside of tpc" self.uncommitted = self.committed.copy() self._resetTransaction() def tpc_begin(self, transaction): """Enter two-phase commit """ assert transaction is self.transaction, "Must not change transactions" assert self.tpc_phase is None, "Must be called outside of tpc" self.tpc_phase = 1 def commit(self, transaction): """Record data modified during the transaction """ assert transaction is self.transaction, "Must not change transactions" assert self.tpc_phase == 1, "Must be called in first phase of tpc" # In our simple example, we don't need to do anything. # A more complex data manager would typically write to some sort # of log. def tpc_vote(self, transaction): assert transaction is self.transaction, "Must not change transactions" assert self.tpc_phase == 1, "Must be called in first phase of tpc" # This particular data manager is always ready to vote. # Real data managers will usually need to take some steps to # make sure that the finish will succeed self.tpc_phase = 2 def tpc_finish(self, transaction): assert transaction is self.transaction, "Must not change transactions" assert self.tpc_phase == 2, "Must be called in second phase of tpc" self.committed = self.uncommitted.copy() self._resetTransaction() def tpc_abort(self, transaction): assert transaction is self.transaction, "Must not change transactions" assert self.tpc_phase is not None, "Must be called inside of tpc" self.uncommitted = self.committed.copy() self._resetTransaction() # ####################################################################### ####################################################################### # Other data manager methods def sortKey(self): # Commit operations on multiple data managers are performed in # sort key order. This important to avoid deadlock when data # managers are shared among multiple threads or processes and # use locks to manage that sharing. We aren't going to bother # with that here. return str(id(self)) # ####################################################################### class SampleSavepointDataManager(SampleDataManager): """Sample implementation of a savepoint-supporting data manager This extends the basic data manager with savepoint support. """ interface.implements(transaction.interfaces.ISavepointDataManager) def savepoint(self): # When we create the savepoint, we save the existing database state. return SampleSavepoint(self, self.uncommitted.copy()) def _rollback_savepoint(self, savepoint): # When we rollback the savepoint, we restore the saved data. # Caution: without the copy(), further changes to the database # could reflect in savepoint.data, and then `savepoint` would no # longer contain the originally saved data, and so `savepoint` # couldn't restore the original state if a rollback to this # savepoint was done again. IOW, copy() is necessary. self.uncommitted = savepoint.data.copy() class SampleSavepoint: interface.implements(transaction.interfaces.IDataManagerSavepoint) def __init__(self, data_manager, data): self.data_manager = data_manager self.data = data def rollback(self): self.data_manager._rollback_savepoint(self) Zope-2.8.7-final/lib/python/transaction/tests/test_SampleDataManager.py0000664000076400007640000002510710435502414025240 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample objects for use in tests $Id: test_SampleDataManager.py 29891 2005-04-06 22:35:05Z tim_one $ """ class DataManager(object): """Sample data manager This class provides a trivial data-manager implementation and doc strings to illustrate the the protocol and to provide a tool for writing tests. Our sample data manager has state that is updated through an inc method and through transaction operations. When we create a sample data manager: >>> dm = DataManager() It has two bits of state, state: >>> dm.state 0 and delta: >>> dm.delta 0 Both of which are initialized to 0. state is meant to model committed state, while delta represents tentative changes within a transaction. We change the state by calling inc: >>> dm.inc() which updates delta: >>> dm.delta 1 but state isn't changed until we commit the transaction: >>> dm.state 0 To commit the changes, we use 2-phase commit. We execute the first stage by calling prepare. We need to pass a transation. Our sample data managers don't really use the transactions for much, so we'll be lazy and use strings for transactions: >>> t1 = '1' >>> dm.prepare(t1) The sample data manager updates the state when we call prepare: >>> dm.state 1 >>> dm.delta 1 This is mainly so we can detect some affect of calling the methods. Now if we call commit: >>> dm.commit(t1) Our changes are"permanent". The state reflects the changes and the delta has been reset to 0. >>> dm.state 1 >>> dm.delta 0 """ def __init__(self): self.state = 0 self.sp = 0 self.transaction = None self.delta = 0 self.prepared = False def inc(self, n=1): self.delta += n def prepare(self, transaction): """Prepare to commit data >>> dm = DataManager() >>> dm.inc() >>> t1 = '1' >>> dm.prepare(t1) >>> dm.commit(t1) >>> dm.state 1 >>> dm.inc() >>> t2 = '2' >>> dm.prepare(t2) >>> dm.abort(t2) >>> dm.state 1 It is en error to call prepare more than once without an intervening commit or abort: >>> dm.prepare(t1) >>> dm.prepare(t1) Traceback (most recent call last): ... TypeError: Already prepared >>> dm.prepare(t2) Traceback (most recent call last): ... TypeError: Already prepared >>> dm.abort(t1) If there was a preceeding savepoint, the transaction must match: >>> rollback = dm.savepoint(t1) >>> dm.prepare(t2) Traceback (most recent call last): ,,, TypeError: ('Transaction missmatch', '2', '1') >>> dm.prepare(t1) """ if self.prepared: raise TypeError('Already prepared') self._checkTransaction(transaction) self.prepared = True self.transaction = transaction self.state += self.delta def _checkTransaction(self, transaction): if (transaction is not self.transaction and self.transaction is not None): raise TypeError("Transaction missmatch", transaction, self.transaction) def abort(self, transaction): """Abort a transaction The abort method can be called before two-phase commit to throw away work done in the transaction: >>> dm = DataManager() >>> dm.inc() >>> dm.state, dm.delta (0, 1) >>> t1 = '1' >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) The abort method also throws away work done in savepoints: >>> dm.inc() >>> r = dm.savepoint(t1) >>> dm.inc() >>> r = dm.savepoint(t1) >>> dm.state, dm.delta (0, 2) >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) If savepoints are used, abort must be passed the same transaction: >>> dm.inc() >>> r = dm.savepoint(t1) >>> t2 = '2' >>> dm.abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> dm.abort(t1) The abort method is also used to abort a two-phase commit: >>> dm.inc() >>> dm.state, dm.delta (0, 1) >>> dm.prepare(t1) >>> dm.state, dm.delta (1, 1) >>> dm.abort(t1) >>> dm.state, dm.delta (0, 0) Of course, the transactions passed to prepare and abort must match: >>> dm.prepare(t1) >>> dm.abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> dm.abort(t1) """ self._checkTransaction(transaction) if self.transaction is not None: self.transaction = None if self.prepared: self.state -= self.delta self.prepared = False self.delta = 0 def commit(self, transaction): """Complete two-phase commit >>> dm = DataManager() >>> dm.state 0 >>> dm.inc() We start two-phase commit by calling prepare: >>> t1 = '1' >>> dm.prepare(t1) We complete it by calling commit: >>> dm.commit(t1) >>> dm.state 1 It is an error ro call commit without calling prepare first: >>> dm.inc() >>> t2 = '2' >>> dm.commit(t2) Traceback (most recent call last): ... TypeError: Not prepared to commit >>> dm.prepare(t2) >>> dm.commit(t2) If course, the transactions given to prepare and commit must be the same: >>> dm.inc() >>> t3 = '3' >>> dm.prepare(t3) >>> dm.commit(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '3') """ if not self.prepared: raise TypeError('Not prepared to commit') self._checkTransaction(transaction) self.delta = 0 self.transaction = None self.prepared = False def savepoint(self, transaction): """Provide the ability to rollback transaction state Savepoints provide a way to: - Save partial transaction work. For some data managers, this could allow resources to be used more efficiently. - Provide the ability to revert state to a point in a transaction without aborting the entire transaction. In other words, savepoints support partial aborts. Savepoints don't use two-phase commit. If there are errors in setting or rolling back to savepoints, the application should abort the containing transaction. This is *not* the responsibility of the data manager. Savepoints are always associated with a transaction. Any work done in a savepoint's transaction is tentative until the transaction is committed using two-phase commit. >>> dm = DataManager() >>> dm.inc() >>> t1 = '1' >>> r = dm.savepoint(t1) >>> dm.state, dm.delta (0, 1) >>> dm.inc() >>> dm.state, dm.delta (0, 2) >>> r.rollback() >>> dm.state, dm.delta (0, 1) >>> dm.prepare(t1) >>> dm.commit(t1) >>> dm.state, dm.delta (1, 0) Savepoints must have the same transaction: >>> r1 = dm.savepoint(t1) >>> dm.state, dm.delta (1, 0) >>> dm.inc() >>> dm.state, dm.delta (1, 1) >>> t2 = '2' >>> r2 = dm.savepoint(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> r2 = dm.savepoint(t1) >>> dm.inc() >>> dm.state, dm.delta (1, 2) If we rollback to an earlier savepoint, we discard all work done later: >>> r1.rollback() >>> dm.state, dm.delta (1, 0) and we can no longer rollback to the later savepoint: >>> r2.rollback() Traceback (most recent call last): ... TypeError: ('Attempt to roll back to invalid save point', 3, 2) We can roll back to a savepoint as often as we like: >>> r1.rollback() >>> r1.rollback() >>> r1.rollback() >>> dm.state, dm.delta (1, 0) >>> dm.inc() >>> dm.inc() >>> dm.inc() >>> dm.state, dm.delta (1, 3) >>> r1.rollback() >>> dm.state, dm.delta (1, 0) But we can't rollback to a savepoint after it has been committed: >>> dm.prepare(t1) >>> dm.commit(t1) >>> r1.rollback() Traceback (most recent call last): ... TypeError: Attempt to rollback stale rollback """ if self.prepared: raise TypeError("Can't get savepoint during two-phase commit") self._checkTransaction(transaction) self.transaction = transaction self.sp += 1 return Rollback(self) class Rollback(object): def __init__(self, dm): self.dm = dm self.sp = dm.sp self.delta = dm.delta self.transaction = dm.transaction def rollback(self): if self.transaction is not self.dm.transaction: raise TypeError("Attempt to rollback stale rollback") if self.dm.sp < self.sp: raise TypeError("Attempt to roll back to invalid save point", self.sp, self.dm.sp) self.dm.sp = self.sp self.dm.delta = self.delta def test_suite(): from zope.testing.doctest import DocTestSuite return DocTestSuite() if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/transaction/tests/test_transaction.py0000664000076400007640000003602110435502414024254 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Test tranasction behavior for variety of cases. I wrote these unittests to investigate some odd transaction behavior when doing unittests of integrating non sub transaction aware objects, and to insure proper txn behavior. these tests test the transaction system independent of the rest of the zodb. you can see the method calls to a jar by passing the keyword arg tracing to the modify method of a dataobject. the value of the arg is a prefix used for tracing print calls to that objects jar. the number of times a jar method was called can be inspected by looking at an attribute of the jar that is the method name prefixed with a c (count/check). i've included some tracing examples for tests that i thought were illuminating as doc strings below. TODO add in tests for objects which are modified multiple times, for example an object that gets modified in multiple sub txns. $Id: test_transaction.py 30534 2005-05-27 18:07:18Z tim_one $ """ import unittest import transaction from ZODB.utils import positive_id class TransactionTests(unittest.TestCase): def setUp(self): mgr = self.transaction_manager = transaction.TransactionManager() self.sub1 = DataObject(mgr) self.sub2 = DataObject(mgr) self.sub3 = DataObject(mgr) self.nosub1 = DataObject(mgr, nost=1) # basic tests with two sub trans jars # really we only need one, so tests for # sub1 should identical to tests for sub2 def testTransactionCommit(self): self.sub1.modify() self.sub2.modify() self.transaction_manager.commit() assert self.sub1._p_jar.ccommit_sub == 0 assert self.sub1._p_jar.ctpc_finish == 1 def testTransactionAbort(self): self.sub1.modify() self.sub2.modify() self.transaction_manager.abort() assert self.sub2._p_jar.cabort == 1 def testTransactionNote(self): t = self.transaction_manager.get() t.note('This is a note.') self.assertEqual(t.description, 'This is a note.') t.note('Another.') self.assertEqual(t.description, 'This is a note.\n\nAnother.') t.abort() # repeat adding in a nonsub trans jars def testNSJTransactionCommit(self): self.nosub1.modify() self.transaction_manager.commit() assert self.nosub1._p_jar.ctpc_finish == 1 def testNSJTransactionAbort(self): self.nosub1.modify() self.transaction_manager.abort() assert self.nosub1._p_jar.ctpc_finish == 0 assert self.nosub1._p_jar.cabort == 1 def BUGtestNSJSubTransactionCommitAbort(self): """ this reveals a bug in transaction.py the nosub jar should not have tpc_finish called on it till the containing txn ends. sub calling method commit nosub calling method tpc_begin sub calling method tpc_finish nosub calling method tpc_finish nosub calling method abort sub calling method abort_sub """ self.sub1.modify(tracing='sub') self.nosub1.modify(tracing='nosub') self.transaction_manager.commit(1) assert self.sub1._p_jar.ctpc_finish == 1 # bug, non sub trans jars are getting finished # in a subtrans assert self.nosub1._p_jar.ctpc_finish == 0 self.transaction_manager.abort() assert self.nosub1._p_jar.cabort == 1 assert self.sub1._p_jar.cabort_sub == 1 ### Failure Mode Tests # # ok now we do some more interesting # tests that check the implementations # error handling by throwing errors from # various jar methods ### # first the recoverable errors def testExceptionInAbort(self): self.sub1._p_jar = SubTransactionJar(errors='abort') self.nosub1.modify() self.sub1.modify(nojar=1) self.sub2.modify() try: self.transaction_manager.abort() except TestTxnException: pass assert self.nosub1._p_jar.cabort == 1 assert self.sub2._p_jar.cabort == 1 def testExceptionInCommit(self): self.sub1._p_jar = SubTransactionJar(errors='commit') self.nosub1.modify() self.sub1.modify(nojar=1) try: self.transaction_manager.commit() except TestTxnException: pass assert self.nosub1._p_jar.ctpc_finish == 0 assert self.nosub1._p_jar.ccommit == 1 assert self.nosub1._p_jar.ctpc_abort == 1 def testExceptionInTpcVote(self): self.sub1._p_jar = SubTransactionJar(errors='tpc_vote') self.nosub1.modify() self.sub1.modify(nojar=1) try: self.transaction_manager.commit() except TestTxnException: pass assert self.nosub1._p_jar.ctpc_finish == 0 assert self.nosub1._p_jar.ccommit == 1 assert self.nosub1._p_jar.ctpc_abort == 1 assert self.sub1._p_jar.ctpc_abort == 1 def testExceptionInTpcBegin(self): """ ok this test reveals a bug in the TM.py as the nosub tpc_abort there is ignored. nosub calling method tpc_begin nosub calling method commit sub calling method tpc_begin sub calling method abort sub calling method tpc_abort nosub calling method tpc_abort """ self.sub1._p_jar = SubTransactionJar(errors='tpc_begin') self.nosub1.modify() self.sub1.modify(nojar=1) try: self.transaction_manager.commit() except TestTxnException: pass assert self.nosub1._p_jar.ctpc_abort == 1 assert self.sub1._p_jar.ctpc_abort == 1 def testExceptionInTpcAbort(self): self.sub1._p_jar = SubTransactionJar( errors=('tpc_abort', 'tpc_vote')) self.nosub1.modify() self.sub1.modify(nojar=1) try: self.transaction_manager.commit() except TestTxnException: pass assert self.nosub1._p_jar.ctpc_abort == 1 # last test, check the hosing mechanism ## def testHoserStoppage(self): ## # It's hard to test the "hosed" state of the database, where ## # hosed means that a failure occurred in the second phase of ## # the two phase commit. It's hard because the database can ## # recover from such an error if it occurs during the very first ## # tpc_finish() call of the second phase. ## for obj in self.sub1, self.sub2: ## j = HoserJar(errors='tpc_finish') ## j.reset() ## obj._p_jar = j ## obj.modify(nojar=1) ## try: ## transaction.commit() ## except TestTxnException: ## pass ## self.assert_(Transaction.hosed) ## self.sub2.modify() ## try: ## transaction.commit() ## except Transaction.POSException.TransactionError: ## pass ## else: ## self.fail("Hosed Application didn't stop commits") class DataObject: def __init__(self, transaction_manager, nost=0): self.transaction_manager = transaction_manager self.nost = nost self._p_jar = None def modify(self, nojar=0, tracing=0): if not nojar: if self.nost: self._p_jar = NoSubTransactionJar(tracing=tracing) else: self._p_jar = SubTransactionJar(tracing=tracing) self.transaction_manager.get().join(self._p_jar) class TestTxnException(Exception): pass class BasicJar: def __init__(self, errors=(), tracing=0): if not isinstance(errors, tuple): errors = errors, self.errors = errors self.tracing = tracing self.cabort = 0 self.ccommit = 0 self.ctpc_begin = 0 self.ctpc_abort = 0 self.ctpc_vote = 0 self.ctpc_finish = 0 self.cabort_sub = 0 self.ccommit_sub = 0 def __repr__(self): return "<%s %X %s>" % (self.__class__.__name__, positive_id(self), self.errors) def sortKey(self): # All these jars use the same sort key, and Python's list.sort() # is stable. These two return self.__class__.__name__ def check(self, method): if self.tracing: print '%s calling method %s'%(str(self.tracing),method) if method in self.errors: raise TestTxnException("error %s" % method) ## basic jar txn interface def abort(self, *args): self.check('abort') self.cabort += 1 def commit(self, *args): self.check('commit') self.ccommit += 1 def tpc_begin(self, txn, sub=0): self.check('tpc_begin') self.ctpc_begin += 1 def tpc_vote(self, *args): self.check('tpc_vote') self.ctpc_vote += 1 def tpc_abort(self, *args): self.check('tpc_abort') self.ctpc_abort += 1 def tpc_finish(self, *args): self.check('tpc_finish') self.ctpc_finish += 1 class SubTransactionJar(BasicJar): def abort_sub(self, txn): self.check('abort_sub') self.cabort_sub = 1 def commit_sub(self, txn): self.check('commit_sub') self.ccommit_sub = 1 class NoSubTransactionJar(BasicJar): pass class HoserJar(BasicJar): # The HoserJars coordinate their actions via the class variable # committed. The check() method will only raise its exception # if committed > 0. committed = 0 def reset(self): # Calling reset() on any instance will reset the class variable. HoserJar.committed = 0 def check(self, method): if HoserJar.committed > 0: BasicJar.check(self, method) def tpc_finish(self, *args): self.check('tpc_finish') self.ctpc_finish += 1 HoserJar.committed += 1 def test_join(): """White-box test of the join method The join method is provided for "backward-compatability" with ZODB 4 data managers. The argument to join must be a zodb4 data manager, transaction.interfaces.IDataManager. >>> from ZODB.tests.sampledm import DataManager >>> from transaction._transaction import DataManagerAdapter >>> t = transaction.Transaction() >>> dm = DataManager() >>> t.join(dm) The end result is that a data manager adapter is one of the transaction's objects: >>> isinstance(t._resources[0], DataManagerAdapter) True >>> t._resources[0]._datamanager is dm True """ def test_beforeCommitHook(): """Test the beforeCommitHook. Let's define a hook to call, and a way to see that it was called. >>> log = [] >>> def reset_log(): ... del log[:] >>> def hook(arg='no_arg', kw1='no_kw1', kw2='no_kw2'): ... log.append("arg %r kw1 %r kw2 %r" % (arg, kw1, kw2)) Now register the hook with a transaction. >>> import transaction >>> t = transaction.begin() >>> t.beforeCommitHook(hook, '1') We can see that the hook is indeed registered. >>> [(hook.func_name, args, kws) ... for hook, args, kws in t.getBeforeCommitHooks()] [('hook', ('1',), {})] When transaction commit starts, the hook is called, with its arguments. >>> log [] >>> t.commit() >>> log ["arg '1' kw1 'no_kw1' kw2 'no_kw2'"] >>> reset_log() A hook's registration is consumed whenever the hook is called. Since the hook above was called, it's no longer registered: >>> len(list(t.getBeforeCommitHooks())) 0 >>> transaction.commit() >>> log [] The hook is only called for a full commit, not for subtransactions. >>> t = transaction.begin() >>> t.beforeCommitHook(hook, 'A', kw1='B') >>> t.commit(subtransaction=True) >>> log [] >>> t.commit() >>> log ["arg 'A' kw1 'B' kw2 'no_kw2'"] >>> reset_log() If a transaction is aborted, no hook is called. >>> t = transaction.begin() >>> t.beforeCommitHook(hook, "OOPS!") >>> transaction.abort() >>> log [] >>> transaction.commit() >>> log [] The hook is called before the commit does anything, so even if the commit fails the hook will have been called. To provoke failures in commit, we'll add failing resource manager to the transaction. >>> class CommitFailure(Exception): ... pass >>> class FailingDataManager: ... def tpc_begin(self, txn, sub=False): ... raise CommitFailure ... def abort(self, txn): ... pass >>> t = transaction.begin() >>> t.join(FailingDataManager()) >>> t.beforeCommitHook(hook, '2') >>> t.commit() Traceback (most recent call last): ... CommitFailure >>> log ["arg '2' kw1 'no_kw1' kw2 'no_kw2'"] >>> reset_log() Let's register several hooks. >>> t = transaction.begin() >>> t.beforeCommitHook(hook, '4', kw1='4.1') >>> t.beforeCommitHook(hook, '5', kw2='5.2') They are returned in the same order by getBeforeCommitHooks. >>> [(hook.func_name, args, kws) #doctest: +NORMALIZE_WHITESPACE ... for hook, args, kws in t.getBeforeCommitHooks()] [('hook', ('4',), {'kw1': '4.1'}), ('hook', ('5',), {'kw2': '5.2'})] And commit also calls them in this order. >>> t.commit() >>> len(log) 2 >>> log #doctest: +NORMALIZE_WHITESPACE ["arg '4' kw1 '4.1' kw2 'no_kw2'", "arg '5' kw1 'no_kw1' kw2 '5.2'"] >>> reset_log() While executing, a hook can itself add more hooks, and they will all be called before the real commit starts. >>> def recurse(txn, arg): ... log.append('rec' + str(arg)) ... if arg: ... txn.beforeCommitHook(hook, '-') ... txn.beforeCommitHook(recurse, txn, arg-1) >>> t = transaction.begin() >>> t.beforeCommitHook(recurse, t, 3) >>> transaction.commit() >>> log #doctest: +NORMALIZE_WHITESPACE ['rec3', "arg '-' kw1 'no_kw1' kw2 'no_kw2'", 'rec2', "arg '-' kw1 'no_kw1' kw2 'no_kw2'", 'rec1', "arg '-' kw1 'no_kw1' kw2 'no_kw2'", 'rec0'] >>> reset_log() """ def test_suite(): from zope.testing.doctest import DocTestSuite return unittest.TestSuite(( DocTestSuite(), unittest.makeSuite(TransactionTests), )) if __name__ == '__main__': unittest.TextTestRunner().run(test_suite()) Zope-2.8.7-final/lib/python/transaction/tests/test_savepoint.py0000664000076400007640000000353610435502414023744 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tests of savepoint feature $Id: test_savepoint.py 30165 2005-04-25 16:29:28Z jim $ """ import unittest from zope.testing import doctest def testRollbackRollsbackDataManagersThatJoinedLater(): """ A savepoint needs to not just rollback it's savepoints, but needs to rollback savepoints for data managers that joined savepoints after the savepoint: >>> import transaction.tests.savepointsample >>> dm = transaction.tests.savepointsample.SampleSavepointDataManager() >>> dm['name'] = 'bob' >>> sp1 = transaction.savepoint() >>> dm['job'] = 'geek' >>> sp2 = transaction.savepoint() >>> dm['salary'] = 'fun' >>> dm2 = transaction.tests.savepointsample.SampleSavepointDataManager() >>> dm2['name'] = 'sally' >>> 'name' in dm True >>> 'job' in dm True >>> 'salary' in dm True >>> 'name' in dm2 True >>> sp1.rollback() >>> 'name' in dm True >>> 'job' in dm False >>> 'salary' in dm False >>> 'name' in dm2 False """ def test_suite(): return unittest.TestSuite(( doctest.DocFileSuite('../savepoint.txt'), doctest.DocTestSuite(), )) if __name__ == '__main__': unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/transaction/tests/test_register_compat.py0000664000076400007640000000753710435502414025130 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test backwards compatibility for resource managers using register(). The transaction package supports several different APIs for resource managers. The original ZODB3 API was implemented by ZODB.Connection. The Connection passed persistent objects to a Transaction's register() method. It's possible that third-party code also used this API, hence these tests that the code that adapts the old interface to the current API works. These tests use a TestConnection object that implements the old API. They check that the right methods are called and in roughly the right order. Common cases ------------ First, check that a basic transaction commit works. >>> cn = TestConnection() >>> cn.register(Object()) >>> cn.register(Object()) >>> cn.register(Object()) >>> transaction.commit() >>> len(cn.committed) 3 >>> len(cn.aborted) 0 >>> cn.calls ['begin', 'vote', 'finish'] Second, check that a basic transaction abort works. If the application calls abort(), then the transaction never gets into the two-phase commit. It just aborts each object. >>> cn = TestConnection() >>> cn.register(Object()) >>> cn.register(Object()) >>> cn.register(Object()) >>> transaction.abort() >>> len(cn.committed) 0 >>> len(cn.aborted) 3 >>> cn.calls [] Error handling -------------- The tricky part of the implementation is recovering from an error that occurs during the two-phase commit. We override the commit() and abort() methods of Object to cause errors during commit. Note that the implementation uses lists internally, so that objects are committed in the order they are registered. (In the presence of multiple resource managers, objects from a single resource manager are committed in order. I'm not sure if this is an accident of the implementation or a feature that should be supported by any implementation.) The order of resource managers depends on sortKey(). >>> cn = TestConnection() >>> cn.register(Object()) >>> cn.register(CommitError()) >>> cn.register(Object()) >>> transaction.commit() Traceback (most recent call last): ... RuntimeError: commit >>> len(cn.committed) 1 >>> len(cn.aborted) 3 Clean up: >>> transaction.abort() """ import transaction class Object(object): def commit(self): pass def abort(self): pass class CommitError(Object): def commit(self): raise RuntimeError("commit") class AbortError(Object): def abort(self): raise RuntimeError("abort") class BothError(CommitError, AbortError): pass class TestConnection: def __init__(self): self.committed = [] self.aborted = [] self.calls = [] def register(self, obj): obj._p_jar = self transaction.get().register(obj) def sortKey(self): return str(id(self)) def tpc_begin(self, txn): self.calls.append("begin") def tpc_vote(self, txn): self.calls.append("vote") def tpc_finish(self, txn): self.calls.append("finish") def tpc_abort(self, txn): self.calls.append("abort") def commit(self, obj, txn): obj.commit() self.committed.append(obj) def abort(self, obj, txn): obj.abort() self.aborted.append(obj) from zope.testing import doctest def test_suite(): return doctest.DocTestSuite() Zope-2.8.7-final/lib/python/transaction/tests/abstestIDataManager.py0000664000076400007640000000376610435502414024545 0ustar ajungajung############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test cases for objects implementing IDataManager. This is a combo test between Connection and DB, since the two are rather incestuous and the DB Interface is not defined that I was able to find. To do a full test suite one would probably want to write a dummy storage that will raise errors as needed for testing. I started this test suite to reproduce a very simple error (tpc_abort had an error and wouldn't even run if called). So it is *very* incomplete, and even the tests that exist do not make sure that the data actually gets written/not written to the storge. Obviously this test suite should be expanded. $Id: abstestIDataManager.py 29961 2005-04-13 18:28:03Z tim_one $ """ from unittest import TestCase class IDataManagerTests(TestCase, object): def setUp(self): self.datamgr = None # subclass should override self.obj = None # subclass should define Persistent object self.txn_factory = None def get_transaction(self): return self.txn_factory() ################################ # IDataManager interface tests # ################################ def testCommitObj(self): tran = self.get_transaction() self.datamgr.prepare(tran) self.datamgr.commit(tran) def testAbortTran(self): tran = self.get_transaction() self.datamgr.prepare(tran) self.datamgr.abort(tran) Zope-2.8.7-final/lib/python/transaction/tests/test_SampleResourceManager.py0000664000076400007640000002734610435502414026165 0ustar ajungajung############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Sample objects for use in tests $Id: test_SampleResourceManager.py 26003 2004-06-29 21:22:58Z tim_one $ """ class ResourceManager(object): """Sample resource manager. This class provides a trivial resource-manager implementation and doc strings to illustrate the protocol and to provide a tool for writing tests. Our sample resource manager has state that is updated through an inc method and through transaction operations. When we create a sample resource manager: >>> rm = ResourceManager() It has two pieces state, state and delta, both initialized to 0: >>> rm.state 0 >>> rm.delta 0 state is meant to model committed state, while delta represents tentative changes within a transaction. We change the state by calling inc: >>> rm.inc() which updates delta: >>> rm.delta 1 but state isn't changed until we commit the transaction: >>> rm.state 0 To commit the changes, we use 2-phase commit. We execute the first stage by calling prepare. We need to pass a transation. Our sample resource managers don't really use the transactions for much, so we'll be lazy and use strings for transactions. The sample resource manager updates the state when we call tpc_vote: >>> t1 = '1' >>> rm.tpc_begin(t1) >>> rm.state, rm.delta (0, 1) >>> rm.tpc_vote(t1) >>> rm.state, rm.delta (1, 1) Now if we call tpc_finish: >>> rm.tpc_finish(t1) Our changes are "permanent". The state reflects the changes and the delta has been reset to 0. >>> rm.state, rm.delta (1, 0) """ def __init__(self): self.state = 0 self.sp = 0 self.transaction = None self.delta = 0 self.txn_state = None def _check_state(self, *ok_states): if self.txn_state not in ok_states: raise ValueError("txn in state %r but expected one of %r" % (self.txn_state, ok_states)) def _checkTransaction(self, transaction): if (transaction is not self.transaction and self.transaction is not None): raise TypeError("Transaction missmatch", transaction, self.transaction) def inc(self, n=1): self.delta += n def tpc_begin(self, transaction): """Prepare to commit data. >>> rm = ResourceManager() >>> rm.inc() >>> t1 = '1' >>> rm.tpc_begin(t1) >>> rm.tpc_vote(t1) >>> rm.tpc_finish(t1) >>> rm.state 1 >>> rm.inc() >>> t2 = '2' >>> rm.tpc_begin(t2) >>> rm.tpc_vote(t2) >>> rm.tpc_abort(t2) >>> rm.state 1 It is an error to call tpc_begin more than once without completing two-phase commit: >>> rm.tpc_begin(t1) >>> rm.tpc_begin(t1) Traceback (most recent call last): ... ValueError: txn in state 'tpc_begin' but expected one of (None,) >>> rm.tpc_abort(t1) If there was a preceeding savepoint, the transaction must match: >>> rollback = rm.savepoint(t1) >>> rm.tpc_begin(t2) Traceback (most recent call last): ,,, TypeError: ('Transaction missmatch', '2', '1') >>> rm.tpc_begin(t1) """ self._checkTransaction(transaction) self._check_state(None) self.transaction = transaction self.txn_state = 'tpc_begin' def tpc_vote(self, transaction): """Verify that a data manager can commit the transaction. This is the last chance for a data manager to vote 'no'. A data manager votes 'no' by raising an exception. transaction is the ITransaction instance associated with the transaction being committed. """ self._checkTransaction(transaction) self._check_state('tpc_begin') self.state += self.delta self.txn_state = 'tpc_vote' def tpc_finish(self, transaction): """Complete two-phase commit >>> rm = ResourceManager() >>> rm.state 0 >>> rm.inc() We start two-phase commit by calling prepare: >>> t1 = '1' >>> rm.tpc_begin(t1) >>> rm.tpc_vote(t1) We complete it by calling tpc_finish: >>> rm.tpc_finish(t1) >>> rm.state 1 It is an error ro call tpc_finish without calling tpc_vote: >>> rm.inc() >>> t2 = '2' >>> rm.tpc_begin(t2) >>> rm.tpc_finish(t2) Traceback (most recent call last): ... ValueError: txn in state 'tpc_begin' but expected one of ('tpc_vote',) >>> rm.tpc_abort(t2) # clean slate >>> rm.tpc_begin(t2) >>> rm.tpc_vote(t2) >>> rm.tpc_finish(t2) Of course, the transactions given to tpc_begin and tpc_finish must be the same: >>> rm.inc() >>> t3 = '3' >>> rm.tpc_begin(t3) >>> rm.tpc_vote(t3) >>> rm.tpc_finish(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '3') """ self._checkTransaction(transaction) self._check_state('tpc_vote') self.delta = 0 self.transaction = None self.prepared = False self.txn_state = None def tpc_abort(self, transaction): """Abort a transaction The abort method can be called before two-phase commit to throw away work done in the transaction: >>> rm = ResourceManager() >>> rm.inc() >>> rm.state, rm.delta (0, 1) >>> t1 = '1' >>> rm.tpc_abort(t1) >>> rm.state, rm.delta (0, 0) The abort method also throws away work done in savepoints: >>> rm.inc() >>> r = rm.savepoint(t1) >>> rm.inc() >>> r = rm.savepoint(t1) >>> rm.state, rm.delta (0, 2) >>> rm.tpc_abort(t1) >>> rm.state, rm.delta (0, 0) If savepoints are used, abort must be passed the same transaction: >>> rm.inc() >>> r = rm.savepoint(t1) >>> t2 = '2' >>> rm.tpc_abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> rm.tpc_abort(t1) The abort method is also used to abort a two-phase commit: >>> rm.inc() >>> rm.state, rm.delta (0, 1) >>> rm.tpc_begin(t1) >>> rm.state, rm.delta (0, 1) >>> rm.tpc_vote(t1) >>> rm.state, rm.delta (1, 1) >>> rm.tpc_abort(t1) >>> rm.state, rm.delta (0, 0) Of course, the transactions passed to prepare and abort must match: >>> rm.tpc_begin(t1) >>> rm.tpc_abort(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> rm.tpc_abort(t1) This should never fail. """ self._checkTransaction(transaction) if self.transaction is not None: self.transaction = None if self.txn_state == 'tpc_vote': self.state -= self.delta self.txn_state = None self.delta = 0 def savepoint(self, transaction): """Provide the ability to rollback transaction state Savepoints provide a way to: - Save partial transaction work. For some resource managers, this could allow resources to be used more efficiently. - Provide the ability to revert state to a point in a transaction without aborting the entire transaction. In other words, savepoints support partial aborts. Savepoints don't use two-phase commit. If there are errors in setting or rolling back to savepoints, the application should abort the containing transaction. This is *not* the responsibility of the resource manager. Savepoints are always associated with a transaction. Any work done in a savepoint's transaction is tentative until the transaction is committed using two-phase commit. >>> rm = ResourceManager() >>> rm.inc() >>> t1 = '1' >>> r = rm.savepoint(t1) >>> rm.state, rm.delta (0, 1) >>> rm.inc() >>> rm.state, rm.delta (0, 2) >>> r.rollback() >>> rm.state, rm.delta (0, 1) >>> rm.tpc_begin(t1) >>> rm.tpc_vote(t1) >>> rm.tpc_finish(t1) >>> rm.state, rm.delta (1, 0) Savepoints must have the same transaction: >>> r1 = rm.savepoint(t1) >>> rm.state, rm.delta (1, 0) >>> rm.inc() >>> rm.state, rm.delta (1, 1) >>> t2 = '2' >>> r2 = rm.savepoint(t2) Traceback (most recent call last): ... TypeError: ('Transaction missmatch', '2', '1') >>> r2 = rm.savepoint(t1) >>> rm.inc() >>> rm.state, rm.delta (1, 2) If we rollback to an earlier savepoint, we discard all work done later: >>> r1.rollback() >>> rm.state, rm.delta (1, 0) and we can no longer rollback to the later savepoint: >>> r2.rollback() Traceback (most recent call last): ... TypeError: ('Attempt to roll back to invalid save point', 3, 2) We can roll back to a savepoint as often as we like: >>> r1.rollback() >>> r1.rollback() >>> r1.rollback() >>> rm.state, rm.delta (1, 0) >>> rm.inc() >>> rm.inc() >>> rm.inc() >>> rm.state, rm.delta (1, 3) >>> r1.rollback() >>> rm.state, rm.delta (1, 0) But we can't rollback to a savepoint after it has been committed: >>> rm.tpc_begin(t1) >>> rm.tpc_vote(t1) >>> rm.tpc_finish(t1) >>> r1.rollback() Traceback (most recent call last): ... TypeError: Attempt to rollback stale rollback """ if self.txn_state is not None: raise TypeError("Can't get savepoint during two-phase commit") self._checkTransaction(transaction) self.transaction = transaction self.sp += 1 return SavePoint(self) def discard(self, transaction): pass class SavePoint(object): def __init__(self, rm): self.rm = rm self.sp = rm.sp self.delta = rm.delta self.transaction = rm.transaction def rollback(self): if self.transaction is not self.rm.transaction: raise TypeError("Attempt to rollback stale rollback") if self.rm.sp < self.sp: raise TypeError("Attempt to roll back to invalid save point", self.sp, self.rm.sp) self.rm.sp = self.sp self.rm.delta = self.delta def discard(self): pass def test_suite(): from doctest import DocTestSuite return DocTestSuite() if __name__ == '__main__': unittest.main() Zope-2.8.7-final/lib/python/transaction/DEPENDENCIES.cfg0000664000076400007640000000002410435502414021455 0ustar ajungajungZODB zope.interface Zope-2.8.7-final/lib/python/transaction/savepoint.txt0000664000076400007640000001773410435502414021737 0ustar ajungajungSavepoints ========== Savepoints provide a way to save to disk intermediate work done during a transaction allowing: - partial transaction (subtransaction) rollback (abort) - state of saved objects to be freed, freeing on-line memory for other uses Savepoints make it possible to write atomic subroutines that don't make top-level transaction commitments. Applications ------------ To demonstrate how savepoints work with transactions, we've provided a sample data manager implementation that provides savepoint support. The primary purpose of this data manager is to provide code that can be read to understand how savepoints work. The secondary purpose is to provide support for demonstrating the correct operation of savepoint support within the transaction system. This data manager is very simple. It provides flat storage of named immutable values, like strings and numbers. >>> import transaction.tests.savepointsample >>> dm = transaction.tests.savepointsample.SampleSavepointDataManager() >>> dm['name'] = 'bob' As with other data managers, we can commit changes: >>> transaction.commit() >>> dm['name'] 'bob' and abort changes: >>> dm['name'] = 'sally' >>> dm['name'] 'sally' >>> transaction.abort() >>> dm['name'] 'bob' Now, let's look at an application that manages funds for people. It allows deposits and debits to be entered for multiple people. It accepts a sequence of entries and generates a sequence of status messages. For each entry, it applies the change and then validates the user's account. If the user's account is invalid, we roll back the change for that entry. The success or failure of an entry is indicated in the output status. First we'll initialize some accounts: >>> dm['bob-balance'] = 0.0 >>> dm['bob-credit'] = 0.0 >>> dm['sally-balance'] = 0.0 >>> dm['sally-credit'] = 100.0 >>> transaction.commit() Now, we'll define a validation function to validate an account: >>> def validate_account(name): ... if dm[name+'-balance'] + dm[name+'-credit'] < 0: ... raise ValueError('Overdrawn', name) And a function to apply entries. If the function fails in some unexpected way, it rolls back all of its changes and prints the error: >>> def apply_entries(entries): ... savepoint = transaction.savepoint() ... try: ... for name, amount in entries: ... entry_savepoint = transaction.savepoint() ... try: ... dm[name+'-balance'] += amount ... validate_account(name) ... except ValueError, error: ... entry_savepoint.rollback() ... print 'Error', str(error) ... else: ... print 'Updated', name ... except Exception, error: ... savepoint.rollback() ... print 'Unexpected exception', error Now let's try applying some entries: >>> apply_entries([ ... ('bob', 10.0), ... ('sally', 10.0), ... ('bob', 20.0), ... ('sally', 10.0), ... ('bob', -100.0), ... ('sally', -100.0), ... ]) Updated bob Updated sally Updated bob Updated sally Error ('Overdrawn', 'bob') Updated sally >>> dm['bob-balance'] 30.0 >>> dm['sally-balance'] -80.0 If we provide entries that cause an unexpected error: >>> apply_entries([ ... ('bob', 10.0), ... ('sally', 10.0), ... ('bob', '20.0'), ... ('sally', 10.0), ... ]) Updated bob Updated sally Unexpected exception unsupported operand type(s) for +=: 'float' and 'str' Because the apply_entries used a savepoint for the entire function, it was able to rollback the partial changes without rolling back changes made in the previous call to apply_entries: >>> dm['bob-balance'] 30.0 >>> dm['sally-balance'] -80.0 If we now abort the outer transactions, the earlier changes will go away: >>> transaction.abort() >>> dm['bob-balance'] 0.0 >>> dm['sally-balance'] 0.0 Savepoint invalidation ---------------------- A savepoint can be used any number of times: >>> dm['bob-balance'] = 100.0 >>> dm['bob-balance'] 100.0 >>> savepoint = transaction.savepoint() >>> dm['bob-balance'] = 200.0 >>> dm['bob-balance'] 200.0 >>> savepoint.rollback() >>> dm['bob-balance'] 100.0 >>> savepoint.rollback() # redundant, but should be harmless >>> dm['bob-balance'] 100.0 >>> dm['bob-balance'] = 300.0 >>> dm['bob-balance'] 300.0 >>> savepoint.rollback() >>> dm['bob-balance'] 100.0 However, using a savepoint invalidates any savepoints that come after it: >>> dm['bob-balance'] = 200.0 >>> dm['bob-balance'] 200.0 >>> savepoint1 = transaction.savepoint() >>> dm['bob-balance'] = 300.0 >>> dm['bob-balance'] 300.0 >>> savepoint2 = transaction.savepoint() >>> savepoint.rollback() >>> dm['bob-balance'] 100.0 >>> savepoint2.rollback() Traceback (most recent call last): ... InvalidSavepointRollbackError >>> savepoint1.rollback() Traceback (most recent call last): ... InvalidSavepointRollbackError >>> transaction.abort() Databases without savepoint support ----------------------------------- Normally it's an error to use savepoints with databases that don't support savepoints: >>> dm_no_sp = transaction.tests.savepointsample.SampleDataManager() >>> dm_no_sp['name'] = 'bob' >>> transaction.commit() >>> dm_no_sp['name'] = 'sally' >>> savepoint = transaction.savepoint() Traceback (most recent call last): ... TypeError: ('Savepoints unsupported', {'name': 'bob'}) >>> transaction.abort() However, a flag can be passed to the transaction savepoint method to indicate that databases without savepoint support should be tolerated until a savepoint is rolled back. This allows transactions to proceed if there are no reasons to roll back: >>> dm_no_sp['name'] = 'sally' >>> savepoint = transaction.savepoint(1) >>> dm_no_sp['name'] = 'sue' >>> transaction.commit() >>> dm_no_sp['name'] 'sue' >>> dm_no_sp['name'] = 'sam' >>> savepoint = transaction.savepoint(1) >>> savepoint.rollback() Traceback (most recent call last): ... TypeError: ('Savepoints unsupported', {'name': 'sam'}) Failures -------- If a failure occurs when creating or rolling back a savepoint, the transaction state will be uncertain and the transaction will become uncommitable. From that point on, most transaction operations, including commit, will fail until the transaction is aborted. In the previous example, we got an error when we tried to rollback the savepoint. If we try to commit the transaction, the commit will fail: >>> transaction.commit() # doctest: +ELLIPSIS Traceback (most recent call last): ... TransactionFailedError: An operation previously failed, with traceback: ... TypeError: ('Savepoints unsupported', {'name': 'sam'}) We have to abort it to make any progress: >>> transaction.abort() Similarly, in our earlier example, where we tried to take a savepoint with a data manager that didn't support savepoints: >>> dm_no_sp['name'] = 'sally' >>> dm['name'] = 'sally' >>> savepoint = transaction.savepoint() Traceback (most recent call last): ... TypeError: ('Savepoints unsupported', {'name': 'sue'}) >>> transaction.commit() # doctest: +ELLIPSIS Traceback (most recent call last): ... TransactionFailedError: An operation previously failed, with traceback: ... TypeError: ('Savepoints unsupported', {'name': 'sue'}) >>> transaction.abort() After clearing the transaction with an abort, we can get on with new transactions: >>> dm_no_sp['name'] = 'sally' >>> dm['name'] = 'sally' >>> transaction.commit() >>> dm_no_sp['name'] 'sally' >>> dm['name'] 'sally' Zope-2.8.7-final/lib/python/reStructuredText/0000775000076400007640000000000010435610645020173 5ustar ajungajungZope-2.8.7-final/lib/python/reStructuredText/__init__.py0000664000076400007640000001344310435500327022304 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Wrapper to integrate reStructuredText into Zope This implementation requires docutils 0.3.4+ from http://docutils.sf.net/ """ try: import docutils except ImportError: raise ImportError, 'Please install docutils 0.3.3+ from http://docutils.sourceforge.net/#download.' version = docutils.__version__.split('.') if version < ['0', '3', '3']: raise ImportError, """Old version of docutils found: Got: %(version)s, required: 0.3.3+ Please remove docutils from %(path)s and replace it with a new version. You can download docutils at http://docutils.sourceforge.net/#download. """ % {'version' : docutils.__version__, 'path' : docutils.__path__[0] } import sys, os, locale from App.config import getConfiguration from docutils.core import publish_parts # get encoding default_enc = sys.getdefaultencoding() default_output_encoding = getConfiguration().rest_output_encoding or default_enc default_input_encoding = getConfiguration().rest_input_encoding or default_enc # starting level for elements (default behaviour inside Zope is

) default_level = 3 initial_header_level = getConfiguration().rest_header_level or default_level # default language used for internal translations and language mappings for DTD # elements default_lang = 'en' default_language_code = getConfiguration().rest_language_code or default_language class Warnings: def __init__(self): self.messages = [] def write(self, message): self.messages.append(message) def render(src, writer='html4css1', report_level=1, stylesheet='default.css', input_encoding=default_input_encoding, output_encoding=default_output_encoding, language_code=default_language_code, initial_header_level = initial_header_level, settings = {}): """get the rendered parts of the document the and warning object """ # Docutils settings: settings = settings.copy() settings['input_encoding'] = input_encoding settings['output_encoding'] = output_encoding settings['stylesheet'] = stylesheet if language_code: settings['language_code'] = language_code settings['language_code'] = language_code settings['file_insertion_enabled '] = 0 # starting level for elements: settings['initial_header_level'] = initial_header_level + 1 # set the reporting level to something sane: settings['report_level'] = report_level # don't break if we get errors: settings['halt_level'] = 6 # remember warnings: settings['warning_stream'] = warning_stream = Warnings() parts = publish_parts(source=src, writer_name=writer, settings_overrides=settings, config_section='zope application') return parts, warning_stream def HTML(src, writer='html4css1', report_level=1, stylesheet='default.css', input_encoding=default_input_encoding, output_encoding=default_output_encoding, language_code=default_language_code, initial_header_level = initial_header_level, warnings = None, settings = {}): """ render HTML from a reStructuredText string - 'src' -- string containing a valid reST document - 'writer' -- docutils writer - 'report_level' - verbosity of reST parser - 'stylesheet' - Stylesheet to be used - 'input_encoding' - encoding of the reST input string - 'output_encoding' - encoding of the rendered HTML output - 'report_level' - verbosity of reST parser - 'language_code' - docutils language - 'initial_header_level' - level of the first header tag - 'warnings' - will be overwritten with a string containing the warnings - 'settings' - dict of settings to pass in to Docutils, with priority """ parts, warning_stream = render(src, writer = writer, report_level = report_level, stylesheet = stylesheet, input_encoding = input_encoding, output_encoding = output_encoding, language_code=language_code, initial_header_level = initial_header_level, settings = settings) header = '%(title)s\n' % { 'level': initial_header_level, 'title': parts['title'], } subheader = '%(subtitle)s\n' % { 'level': initial_header_level+1, 'subtitle': parts['subtitle'], } body = '%(docinfo)s%(body)s' % { 'docinfo': parts['docinfo'], 'body': parts['body'], } output = '' if parts['title']: output = output + header if parts['subtitle']: output = output + subheader output = output + body warnings = ''.join(warning_stream.messages) return output.encode(output_encoding) __all__ = ("HTML", 'render') Zope-2.8.7-final/lib/python/reStructuredText/README.txt0000664000076400007640000000005110435500327021660 0ustar ajungajungSee doc/RESTRUCTUREDTEXT.txt for details Zope-2.8.7-final/lib/python/reStructuredText/reStructuredText.txt0000664000076400007640000030323510435500327024275 0ustar ajungajung======================================= reStructuredText Markup Specification ======================================= :Author: David Goodger :Contact: goodger@users.sourceforge.net :Revision: $Revision: 1.64 $ :Date: $Date: 2004/07/24 18:35:08 $ :Copyright: This document has been placed in the public domain. .. Note:: This document is a detailed technical specification; it is not a tutorial or a primer. If this is your first exposure to reStructuredText, please read `A ReStructuredText Primer`_ and the `Quick reStructuredText`_ user reference first. .. _A ReStructuredText Primer: ../../user/rst/quickstart.html .. _Quick reStructuredText: ../../user/rst/quickref.html reStructuredText_ is plaintext that uses simple and intuitive constructs to indicate the structure of a document. These constructs are equally easy to read in raw and processed forms. This document is itself an example of reStructuredText (raw, if you are reading the text file, or processed, if you are reading an HTML document, for example). The reStructuredText parser is a component of Docutils_. Simple, implicit markup is used to indicate special constructs, such as section headings, bullet lists, and emphasis. The markup used is as minimal and unobtrusive as possible. Less often-used constructs and extensions to the basic reStructuredText syntax may have more elaborate or explicit markup. reStructuredText is applicable to documents of any length, from the very small (such as inline program documentation fragments, e.g. Python docstrings) to the quite large (this document). The first section gives a quick overview of the syntax of the reStructuredText markup by example. A complete specification is given in the `Syntax Details`_ section. `Literal blocks`_ (in which no markup processing is done) are used for examples throughout this document, to illustrate the plaintext markup. .. contents:: ----------------------- Quick Syntax Overview ----------------------- A reStructuredText document is made up of body or block-level elements, and may be structured into sections. Sections_ are indicated through title style (underlines & optional overlines). Sections contain body elements and/or subsections. Some body elements contain further elements, such as lists containing list items, which in turn may contain paragraphs and other body elements. Others, such as paragraphs, contain text and `inline markup`_ elements. Here are examples of `body elements`_: - Paragraphs_ (and `inline markup`_):: Paragraphs contain text and may contain inline markup: *emphasis*, **strong emphasis**, `interpreted text`, ``inline literals``, standalone hyperlinks (http://www.python.org), external hyperlinks (Python_), internal cross-references (example_), footnote references ([1]_), citation references ([CIT2002]_), substitution references (|example|), and _`inline internal targets`. Paragraphs are separated by blank lines and are left-aligned. - Five types of lists: 1. `Bullet lists`_:: - This is a bullet list. - Bullets can be "-", "*", or "+". 2. `Enumerated lists`_:: 1. This is an enumerated list. 2. Enumerators may be arabic numbers, letters, or roman numerals. 3. `Definition lists`_:: what Definition lists associate a term with a definition. how The term is a one-line phrase, and the definition is one or more paragraphs or body elements, indented relative to the term. 4. `Field lists`_:: :what: Field lists map field names to field bodies, like database records. They are often part of an extension syntax. :how: The field marker is a colon, the field name, and a colon. The field body may contain one or more body elements, indented relative to the field marker. 5. `Option lists`_, for listing command-line options:: -a command-line option "a" -b file options can have arguments and long descriptions --long options can be long also --input=file long options can also have arguments /V DOS/VMS-style options too There must be at least two spaces between the option and the description. - `Literal blocks`_:: Literal blocks are either indented or line-prefix-quoted blocks, and indicated with a double-colon ("::") at the end of the preceding paragraph (right here -->):: if literal_block: text = 'is left as-is' spaces_and_linebreaks = 'are preserved' markup_processing = None - `Block quotes`_:: Block quotes consist of indented body elements: This theory, that is mine, is mine. -- Anne Elk (Miss) - `Doctest blocks`_:: >>> print 'Python-specific usage examples; begun with ">>>"' Python-specific usage examples; begun with ">>>" >>> print '(cut and pasted from interactive Python sessions)' (cut and pasted from interactive Python sessions) - Two syntaxes for tables_: 1. `Grid tables`_; complete, but complex and verbose:: +------------------------+------------+----------+ | Header row, column 1 | Header 2 | Header 3 | +========================+============+==========+ | body row 1, column 1 | column 2 | column 3 | +------------------------+------------+----------+ | body row 2 | Cells may span | +------------------------+-----------------------+ 2. `Simple tables`_; easy and compact, but limited:: ==================== ========== ========== Header row, column 1 Header 2 Header 3 ==================== ========== ========== body row 1, column 1 column 2 column 3 body row 2 Cells may span columns ==================== ====================== - `Explicit markup blocks`_ all begin with an explicit block marker, two periods and a space: - Footnotes_:: .. [1] A footnote contains body elements, consistently indented by at least 3 spaces. - Citations_:: .. [CIT2002] Just like a footnote, except the label is textual. - `Hyperlink targets`_:: .. _Python: http://www.python.org .. _example: The "_example" target above points to this paragraph. - Directives_:: .. image:: mylogo.png - `Substitution definitions`_:: .. |symbol here| image:: symbol.png - Comments_:: .. Comments begin with two dots and a space. Anything may follow, except for the syntax of footnotes/citations, hyperlink targets, directives, or substitution definitions. ---------------- Syntax Details ---------------- Descriptions below list "doctree elements" (document tree element names; XML DTD generic identifiers) corresponding to syntax constructs. For details on the hierarchy of elements, please see `The Docutils Document Tree`_ and the `Docutils Generic DTD`_ XML document type definition. Whitespace ========== Spaces are recommended for indentation_, but tabs may also be used. Tabs will be converted to spaces. Tab stops are at every 8th column. Other whitespace characters (form feeds [chr(12)] and vertical tabs [chr(11)]) are converted to single spaces before processing. Blank Lines ----------- Blank lines are used to separate paragraphs and other elements. Multiple successive blank lines are equivalent to a single blank line, except within literal blocks (where all whitespace is preserved). Blank lines may be omitted when the markup makes element separation unambiguous, in conjunction with indentation. The first line of a document is treated as if it is preceded by a blank line, and the last line of a document is treated as if it is followed by a blank line. Indentation ----------- Indentation is used to indicate, and is only significant in indicating: - multi-line contents of list items, - multiple body elements within a list item (including nested lists), - the definition part of a definition list item, - block quotes, - the extent of literal blocks, and - the extent of explicit markup blocks. Any text whose indentation is less than that of the current level (i.e., unindented text or "dedents") ends the current level of indentation. Since all indentation is significant, the level of indentation must be consistent. For example, indentation is the sole markup indicator for `block quotes`_:: This is a top-level paragraph. This paragraph belongs to a first-level block quote. Paragraph 2 of the first-level block quote. Multiple levels of indentation within a block quote will result in more complex structures:: This is a top-level paragraph. This paragraph belongs to a first-level block quote. This paragraph belongs to a second-level block quote. Another top-level paragraph. This paragraph belongs to a second-level block quote. This paragraph belongs to a first-level block quote. The second-level block quote above is inside this first-level block quote. When a paragraph or other construct consists of more than one line of text, the lines must be left-aligned:: This is a paragraph. The lines of this paragraph are aligned at the left. This paragraph has problems. The lines are not left-aligned. In addition to potential misinterpretation, warning and/or error messages will be generated by the parser. Several constructs begin with a marker, and the body of the construct must be indented relative to the marker. For constructs using simple markers (`bullet lists`_, `enumerated lists`_, footnotes_, citations_, `hyperlink targets`_, directives_, and comments_), the level of indentation of the body is determined by the position of the first line of text, which begins on the same line as the marker. For example, bullet list bodies must be indented by at least two columns relative to the left edge of the bullet:: - This is the first line of a bullet list item's paragraph. All lines must align relative to the first line. [1]_ This indented paragraph is interpreted as a block quote. Because it is not sufficiently indented, this paragraph does not belong to the list item. .. [1] Here's a footnote. The second line is aligned with the beginning of the footnote label. The ".." marker is what determines the indentation. For constructs using complex markers (`field lists`_ and `option lists`_), where the marker may contain arbitrary text, the indentation of the first line *after* the marker determines the left edge of the body. For example, field lists may have very long markers (containing the field names):: :Hello: This field has a short field name, so aligning the field body with the first line is feasible. :Number-of-African-swallows-required-to-carry-a-coconut: It would be very difficult to align the field body with the left edge of the first line. It may even be preferable not to begin the body on the same line as the marker. Escaping Mechanism ================== The character set universally available to plaintext documents, 7-bit ASCII, is limited. No matter what characters are used for markup, they will already have multiple meanings in written text. Therefore markup characters *will* sometimes appear in text **without being intended as markup**. Any serious markup system requires an escaping mechanism to override the default meaning of the characters used for the markup. In reStructuredText we use the backslash, commonly used as an escaping character in other domains. A backslash followed by any character (except whitespace characters) escapes that character. The escaped character represents the character itself, and is prevented from playing a role in any markup interpretation. The backslash is removed from the output. A literal backslash is represented by two backslashes in a row (the first backslash "escapes" the second, preventing it being interpreted in an "escaping" role). Backslash-escaped whitespace characters are removed from the document. This allows for character-level `inline markup`_. There are two contexts in which backslashes have no special meaning: literal blocks and inline literals. In these contexts, a single backslash represents a literal backslash, without having to double up. Please note that the reStructuredText specification and parser do not address the issue of the representation or extraction of text input (how and in what form the text actually *reaches* the parser). Backslashes and other characters may serve a character-escaping purpose in certain contexts and must be dealt with appropriately. For example, Python uses backslashes in strings to escape certain characters, but not others. The simplest solution when backslashes appear in Python docstrings is to use raw docstrings:: r"""This is a raw docstring. Backslashes (\) are not touched.""" Reference Names =============== Simple reference names are single words consisting of alphanumerics plus isolated (no two adjacent) internal hyphens, underscores, and periods; no whitespace or other characters are allowed. Footnote labels (Footnotes_ & `Footnote References`_), citation labels (Citations_ & `Citation References`_), `interpreted text`_ roles, and some `hyperlink references`_ use the simple reference name syntax. Reference names using punctuation or whose names are phrases (two or more space-separated words) are called "phrase-references". Phrase-references are expressed by enclosing the phrase in backquotes and treating the backquoted text as a reference name:: Want to learn about `my favorite programming language`_? .. _my favorite programming language: http://www.python.org Simple reference names may also optionally use backquotes. Reference names are whitespace-neutral and case-insensitive. When resolving reference names internally: - whitespace is normalized (one or more spaces, horizontal or vertical tabs, newlines, carriage returns, or form feeds, are interpreted as a single space), and - case is normalized (all alphabetic characters are converted to lowercase). For example, the following `hyperlink references`_ are equivalent:: - `A HYPERLINK`_ - `a hyperlink`_ - `A Hyperlink`_ Hyperlinks_, footnotes_, and citations_ all share the same namespace for reference names. The labels of citations (simple reference names) and manually-numbered footnotes (numbers) are entered into the same database as other hyperlink names. This means that a footnote (defined as "``.. [1]``") which can be referred to by a footnote reference (``[1]_``), can also be referred to by a plain hyperlink reference (1_). Of course, each type of reference (hyperlink, footnote, citation) may be processed and rendered differently. Some care should be taken to avoid reference name conflicts. Document Structure ================== Document -------- Doctree element: document. The top-level element of a parsed reStructuredText document is the "document" element. After initial parsing, the document element is a simple container for a document fragment, consisting of `body elements`_, transitions_, and sections_, but lacking a document title or other bibliographic elements. The code that calls the parser may choose to run one or more optional post-parse transforms_, rearranging the document fragment into a complete document with a title and possibly other metadata elements (author, date, etc.; see `Bibliographic Fields`_). Specifically, there is no way to indicate a document title and subtitle explicitly in reStructuredText. Instead, a lone top-level section title (see Sections_ below) can be treated as the document title. Similarly, a lone second-level section title immediately after the "document title" can become the document subtitle. The rest of the sections are then lifted up a level or two. See the `DocTitle transform`_ for details. Sections -------- Doctree elements: section, title. Sections are identified through their titles, which are marked up with adornment: "underlines" below the title text, or underlines and matching "overlines" above the title. An underline/overline is a single repeated punctuation character that begins in column 1 and forms a line extending at least as far as the right edge of the title text. Specifically, an underline/overline character may be any non-alphanumeric printable 7-bit ASCII character [#]_. When an overline is used, the length and character used must match the underline. Underline-only adornment styles are distinct from overline-and-underline styles that use the same character. There may be any number of levels of section titles, although some output formats may have limits (HTML has 6 levels). .. [#] The following are all valid section title adornment characters:: ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ Some characters are more suitable than others. The following are recommended:: = - ` : . ' " ~ ^ _ * + # Rather than imposing a fixed number and order of section title adornment styles, the order enforced will be the order as encountered. The first style encountered will be an outermost title (like HTML H1), the second style will be a subtitle, the third will be a subsubtitle, and so on. Below are examples of section title styles:: =============== Section Title =============== --------------- Section Title --------------- Section Title ============= Section Title ------------- Section Title ````````````` Section Title ''''''''''''' Section Title ............. Section Title ~~~~~~~~~~~~~ Section Title ************* Section Title +++++++++++++ Section Title ^^^^^^^^^^^^^ When a title has both an underline and an overline, the title text may be inset, as in the first two examples above. This is merely aesthetic and not significant. Underline-only title text may *not* be inset. A blank line after a title is optional. All text blocks up to the next title of the same or higher level are included in a section (or subsection, etc.). All section title styles need not be used, nor need any specific section title style be used. However, a document must be consistent in its use of section titles: once a hierarchy of title styles is established, sections must use that hierarchy. Each section title automatically generates a hyperlink target pointing to the section. The text of the hyperlink target (the "reference name") is the same as that of the section title. See `Implicit Hyperlink Targets`_ for a complete description. Sections may contain `body elements`_, transitions_, and nested sections. Transitions ----------- Doctree element: transition. Instead of subheads, extra space or a type ornament between paragraphs may be used to mark text divisions or to signal changes in subject or emphasis. (The Chicago Manual of Style, 14th edition, section 1.80) Transitions are commonly seen in novels and short fiction, as a gap spanning one or more lines, with or without a type ornament such as a row of asterisks. Transitions separate other body elements. A transition should not begin or end a section or document, nor should two transitions be immediately adjacent. The syntax for a transition marker is a horizontal line of 4 or more repeated punctuation characters. The syntax is the same as section title underlines without title text. Transition markers require blank lines before and after:: Para. ---------- Para. Unlike section title underlines, no hierarchy of transition markers is enforced, nor do differences in transition markers accomplish anything. It is recommended that a single consistent style be used. The processing system is free to render transitions in output in any way it likes. For example, horizontal rules (``
``) in HTML output would be an obvious choice. Body Elements ============= Paragraphs ---------- Doctree element: paragraph. Paragraphs consist of blocks of left-aligned text with no markup indicating any other body element. Blank lines separate paragraphs from each other and from other body elements. Paragraphs may contain `inline markup`_. Syntax diagram:: +------------------------------+ | paragraph | | | +------------------------------+ +------------------------------+ | paragraph | | | +------------------------------+ Bullet Lists ------------ Doctree elements: bullet_list, list_item. A text block which begins with a "-", "*", or "+", followed by whitespace, is a bullet list item (a.k.a. "unordered" list item). List item bodies must be left-aligned and indented relative to the bullet; the text immediately after the bullet determines the indentation. For example:: - This is the first bullet list item. The blank line above the first list item is required; blank lines between list items (such as below this paragraph) are optional. - This is the first paragraph in the second item in the list. This is the second paragraph in the second item in the list. The blank line above this paragraph is required. The left edge of this paragraph lines up with the paragraph above, both indented relative to the bullet. - This is a sublist. The bullet lines up with the left edge of the text blocks above. A sublist is a new list so requires a blank line above and below. - This is the third item of the main list. This paragraph is not part of the list. Here are examples of **incorrectly** formatted bullet lists:: - This first line is fine. A blank line is required between list items and paragraphs. (Warning) - The following line appears to be a new sublist, but it is not: - This is a paragraph continuation, not a sublist (since there's no blank line). This line is also incorrectly indented. - Warnings may be issued by the implementation. Syntax diagram:: +------+-----------------------+ | "- " | list item | +------| (body elements)+ | +-----------------------+ Enumerated Lists ---------------- Doctree elements: enumerated_list, list_item. Enumerated lists (a.k.a. "ordered" lists) are similar to bullet lists, but use enumerators instead of bullets. An enumerator consists of an enumeration sequence member and formatting, followed by whitespace. The following enumeration sequences are recognized: - arabic numerals: 1, 2, 3, ... (no upper limit). - uppercase alphabet characters: A, B, C, ..., Z. - lower-case alphabet characters: a, b, c, ..., z. - uppercase Roman numerals: I, II, III, IV, ..., MMMMCMXCIX (4999). - lowercase Roman numerals: i, ii, iii, iv, ..., mmmmcmxcix (4999). The following formatting types are recognized: - suffixed with a period: "1.", "A.", "a.", "I.", "i.". - surrounded by parentheses: "(1)", "(A)", "(a)", "(I)", "(i)". - suffixed with a right-parenthesis: "1)", "A)", "a)", "I)", "i)". While parsing an enumerated list, a new list will be started whenever: - An enumerator is encountered which does not have the same format and sequence type as the current list (e.g. "1.", "(a)" produces two separate lists). - The enumerators are not in sequence (e.g., "1.", "3." produces two separate lists). It is recommended that the enumerator of the first list item be ordinal-1 ("1", "A", "a", "I", or "i"). Although other start-values will be recognized, they may not be supported by the output format. A level-1 [info] system message will be generated for any list beginning with a non-ordinal-1 enumerator. Lists using Roman numerals must begin with "I"/"i" or a multi-character value, such as "II" or "XV". Any other single-character Roman numeral ("V", "X", "L", "C", "D", "M") will be interpreted as a letter of the alphabet, not as a Roman numeral. Likewise, lists using letters of the alphabet may not begin with "I"/"i", since these are recognized as Roman numeral 1. The second line of each enumerated list item is checked for validity. This is to prevent ordinary paragraphs from being mistakenly interpreted as list items, when they happen to begin with text identical to enumerators. For example, this text is parsed as an ordinary paragraph:: A. Einstein was a really smart dude. However, ambiguity cannot be avoided if the paragraph consists of only one line. This text is parsed as an enumerated list item:: A. Einstein was a really smart dude. If a single-line paragraph begins with text identical to an enumerator ("A.", "1.", "(b)", "I)", etc.), the first character will have to be escaped in order to have the line parsed as an ordinary paragraph:: \A. Einstein was a really smart dude. Nested enumerated lists must be created with indentation. For example:: 1. Item 1. a) Item 1a. b) Item 1b. Example syntax diagram:: +-------+----------------------+ | "1. " | list item | +-------| (body elements)+ | +----------------------+ Definition Lists ---------------- Doctree elements: definition_list, definition_list_item, term, classifier, definition. Each definition list item contains a term, an optional classifier, and a definition. A term is a simple one-line word or phrase. An optional classifier may follow the term on the same line, after an inline " : " (space, colon, space). A definition is a block indented relative to the term, and may contain multiple paragraphs and other body elements. There may be no blank line between a term line and a definition block (this distinguishes definition lists from `block quotes`_). Blank lines are required before the first and after the last definition list item, but are optional in-between. For example:: term 1 Definition 1. term 2 Definition 2, paragraph 1. Definition 2, paragraph 2. term 3 : classifier Definition 3. Inline markup is parsed in the term line before the term/classifier delimiter (" : ") is recognized. The delimiter will only be recognized if it appears outside of any inline markup. A definition list may be used in various ways, including: - As a dictionary or glossary. The term is the word itself, a classifier may be used to indicate the usage of the term (noun, verb, etc.), and the definition follows. - To describe program variables. The term is the variable name, a classifier may be used to indicate the type of the variable (string, integer, etc.), and the definition describes the variable's use in the program. This usage of definition lists supports the classifier syntax of Grouch_, a system for describing and enforcing a Python object schema. Syntax diagram:: +---------------------------+ | term [ " : " classifier ] | +--+------------------------+--+ | definition | | (body elements)+ | +---------------------------+ Field Lists ----------- Doctree elements: field_list, field, field_name, field_body. Field lists are used as part of an extension syntax, such as options for directives_, or database-like records meant for further processing. They may also be used for two-column table-like structures resembling database records (label & data pairs). Applications of reStructuredText may recognize field names and transform fields or field bodies in certain contexts. For examples, see `Bibliographic Fields`_ below, or the "image" and "meta" directives in `reStructuredText Directives`_. Field lists are mappings from field names to field bodies, modeled on RFC822_ headers. A field name is made up of one or more letters, numbers, whitespace, and punctuation, except colons (":"). Inline markup is parsed in field names. Field names are case-insensitive when further processed or transformed. The field name, along with a single colon prefix and suffix, together form the field marker. The field marker is followed by whitespace and the field body. The field body may contain multiple body elements, indented relative to the field marker. The first line after the field name marker determines the indentation of the field body. For example:: :Date: 2001-08-16 :Version: 1 :Authors: - Me - Myself - I :Indentation: Since the field marker may be quite long, the second and subsequent lines of the field body do not have to line up with the first line, but they must be indented relative to the field name marker, and they must line up with each other. :Parameter i: integer The interpretation of individual words in a multi-word field name is up to the application. The application may specify a syntax for the field name. For example, second and subsequent words may be treated as "arguments", quoted phrases may be treated as a single argument, and direct support for the "name=value" syntax may be added. Standard RFC822_ headers cannot be used for this construct because they are ambiguous. A word followed by a colon at the beginning of a line is common in written text. However, in well-defined contexts such as when a field list invariably occurs at the beginning of a document (PEPs and email messages), standard RFC822 headers could be used. Syntax diagram (simplified):: +--------------------+----------------------+ | ":" field name ":" | field body | +-------+------------+ | | (body elements)+ | +-----------------------------------+ Bibliographic Fields ```````````````````` Doctree elements: docinfo, author, authors, organization, contact, version, status, date, copyright, field, topic. When a field list is the first non-comment element in a document (after the document title, if there is one), it may have its fields transformed to document bibliographic data. This bibliographic data corresponds to the front matter of a book, such as the title page and copyright page. Certain registered field names (listed below) are recognized and transformed to the corresponding doctree elements, most becoming child elements of the "docinfo" element. No ordering is required of these fields, although they may be rearranged to fit the document structure, as noted. Unless otherwise indicated below, each of the bibliographic elements' field bodies may contain a single paragraph only. Field bodies may be checked for `RCS keywords`_ and cleaned up. Any unrecognized fields will remain as generic fields in the docinfo element. The registered bibliographic field names and their corresponding doctree elements are as follows: - Field name "Author": author element. - "Authors": authors. - "Organization": organization. - "Contact": contact. - "Address": address. - "Version": version. - "Status": status. - "Date": date. - "Copyright": copyright. - "Dedication": topic. - "Abstract": topic. The "Authors" field may contain either: a single paragraph consisting of a list of authors, separated by ";" or ","; or a bullet list whose elements each contain a single paragraph per author. ";" is checked first, so "Doe, Jane; Doe, John" will work. In some languages (e.g. Swedish), there is no singular/plural distinction between "Author" and "Authors", so only an "Authors" field is provided, and a single name is interpreted as an "Author". If a single name contains a comma, end it with a semicolon to disambiguate: ":Authors: Doe, Jane;". The "Address" field is for a multi-line surface mailing address. A specialized form of line block`_ (see `reStructuredText Directives`_), newlines and whitespace will be preserved. The "Dedication" and "Abstract" fields may contain arbitrary body elements. Only one of each is allowed. They become topic elements with "Dedication" or "Abstract" titles (or language equivalents) immediately following the docinfo element. This field-name-to-element mapping can be replaced for other languages. See the `DocInfo transform`_ implementation documentation for details. Unregistered/generic fields may contain one or more paragraphs or arbitrary body elements. RCS Keywords ```````````` `Bibliographic fields`_ recognized by the parser are normally checked for RCS [#]_ keywords and cleaned up [#]_. RCS keywords may be entered into source files as "$keyword$", and once stored under RCS or CVS [#]_, they are expanded to "$keyword: expansion text $". For example, a "Status" field will be transformed to a "status" element:: :Status: $keyword: expansion text $ .. [#] Revision Control System. .. [#] RCS keyword processing can be turned off (unimplemented). .. [#] Concurrent Versions System. CVS uses the same keywords as RCS. Processed, the "status" element's text will become simply "expansion text". The dollar sign delimiters and leading RCS keyword name are removed. The RCS keyword processing only kicks in when both of these conditions hold: 1. The field list is in bibliographic context (first non-comment construct in the document, after a document title if there is one). 2. The field name is a recognized bibliographic field name. Option Lists ------------ Doctree elements: option_list, option_list_item, option_group, option, option_string, option_argument, description. Option lists are two-column lists of command-line options and descriptions, documenting a program's options. For example:: -a Output all. -b Output both (this description is quite long). -c arg Output just arg. --long Output all day long. -p This option has two paragraphs in the description. This is the first. This is the second. Blank lines may be omitted between options (as above) or left in (as here and below). --very-long-option A VMS-style option. Note the adjustment for the required two spaces. --an-even-longer-option The description can also start on the next line. -2, --two This option has two variants. -f FILE, --file=FILE These two options are synonyms; both have arguments. /V A VMS/DOS-style option. There are several types of options recognized by reStructuredText: - Short POSIX options consist of one dash and an option letter. - Long POSIX options consist of two dashes and an option word; some systems use a single dash. - Old GNU-style "plus" options consist of one plus and an option letter ("plus" options are deprecated now, their use discouraged). - DOS/VMS options consist of a slash and an option letter or word. Please note that both POSIX-style and DOS/VMS-style options may be used by DOS or Windows software. These and other variations are sometimes used mixed together. The names above have been chosen for convenience only. The syntax for short and long POSIX options is based on the syntax supported by Python's getopt.py_ module, which implements an option parser similar to the `GNU libc getopt_long()`_ function but with some restrictions. There are many variant option systems, and reStructuredText option lists do not support all of them. Although long POSIX and DOS/VMS option words may be allowed to be truncated by the operating system or the application when used on the command line, reStructuredText option lists do not show or support this with any special syntax. The complete option word should be given, supported by notes about truncation if and when applicable. Options may be followed by an argument placeholder, whose role and syntax should be explained in the description text. Either a space or an equals sign may be used as a delimiter between options and option argument placeholders; short options ("-" or "+" prefix only) may omit the delimiter. Option arguments may take one of two forms: - Begins with a letter (``[a-zA-Z]``) and subsequently consists of letters, numbers, underscores and hyphens (``[a-zA-Z0-9_-]``). - Begins with an open-angle-bracket (``<``) and ends with a close-angle-bracket (``>``); any characters except angle brackets are allowed internally. Multiple option "synonyms" may be listed, sharing a single description. They must be separated by comma-space. There must be at least two spaces between the option(s) and the description. The description may contain multiple body elements. The first line after the option marker determines the indentation of the description. As with other types of lists, blank lines are required before the first option list item and after the last, but are optional between option entries. Syntax diagram (simplified):: +----------------------------+-------------+ | option [" " argument] " " | description | +-------+--------------------+ | | (body elements)+ | +----------------------------------+ Literal Blocks -------------- Doctree element: literal_block. A paragraph consisting of two colons ("::") signifies that the following text block(s) comprise a literal block. The literal block must either be indented or quoted (see below). No markup processing is done within a literal block. It is left as-is, and is typically rendered in a monospaced typeface:: This is a typical paragraph. An indented literal block follows. :: for a in [5,4,3,2,1]: # this is program code, shown as-is print a print "it's..." # a literal block continues until the indentation ends This text has returned to the indentation of the first paragraph, is outside of the literal block, and is therefore treated as an ordinary paragraph. The paragraph containing only "::" will be completely removed from the output; no empty paragraph will remain. As a convenience, the "::" is recognized at the end of any paragraph. If immediately preceded by whitespace, both colons will be removed from the output (this is the "partially minimized" form). When text immediately precedes the "::", *one* colon will be removed from the output, leaving only one colon visible (i.e., "::" will be replaced by ":"; this is the "fully minimized" form). In other words, these are all equivalent (please pay attention to the colons after "Paragraph"): 1. Expanded form:: Paragraph: :: Literal block 2. Partially minimized form:: Paragraph: :: Literal block 3. Fully minimized form:: Paragraph:: Literal block All whitespace (including line breaks, but excluding minimum indentation for indented literal blocks) is preserved. Blank lines are required before and after a literal block, but these blank lines are not included as part of the literal block. Indented Literal Blocks ``````````````````````` Indented literal blocks are indicated by indentation relative to the surrounding text (leading whitespace on each line). The minimum indentation will be removed from each line of an indented literal block. The literal block need not be contiguous; blank lines are allowed between sections of indented text. The literal block ends with the end of the indentation. Syntax diagram:: +------------------------------+ | paragraph | | (ends with "::") | +------------------------------+ +---------------------------+ | indented literal block | +---------------------------+ Quoted Literal Blocks ````````````````````` Quoted literal blocks are unindented contiguous blocks of text where each line begins with the same non-alphanumeric printable 7-bit ASCII character [#]_. A blank line ends a quoted literal block. The quoting characters are preserved in the processed document. .. [#] The following are all valid quoting characters:: ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ Note that these are the same characters as are valid for title adornment of sections_. Possible uses include literate programming in Haskell and email quoting:: John Doe wrote:: >> Great idea! > > Why didn't I think of that? You just did! ;-) Syntax diagram:: +------------------------------+ | paragraph | | (ends with "::") | +------------------------------+ +------------------------------+ | ">" per-line-quoted | | ">" contiguous literal block | +------------------------------+ Block Quotes ------------ Doctree element: block_quote, attribution. A text block that is indented relative to the preceding text, without markup indicating it to be a literal block, is a block quote. All markup processing (for body elements and inline markup) continues within the block quote:: This is an ordinary paragraph, introducing a block quote. "It is my business to know things. That is my trade." -- Sherlock Holmes If the final block of a block quote begins with "--", "---", or a true em-dash (flush left within the block quote), it is interpreted as an attribution. If the attribution consists of multiple lines, the left edges of the second and subsequent lines must align. Blank lines are required before and after a block quote, but these blank lines are not included as part of the block quote. Syntax diagram:: +------------------------------+ | (current level of | | indentation) | +------------------------------+ +---------------------------+ | block quote | | (body elements)+ | | | | -- attribution text | | (optional) | +---------------------------+ Doctest Blocks -------------- Doctree element: doctest_block. Doctest blocks are interactive Python sessions cut-and-pasted into docstrings. They are meant to illustrate usage by example, and provide an elegant and powerful testing environment via the `doctest module`_ in the Python standard library. Doctest blocks are text blocks which begin with ``">>> "``, the Python interactive interpreter main prompt, and end with a blank line. Doctest blocks are treated as a special case of literal blocks, without requiring the literal block syntax. If both are present, the literal block syntax takes priority over Doctest block syntax:: This is an ordinary paragraph. >>> print 'this is a Doctest block' this is a Doctest block The following is a literal block:: >>> This is not recognized as a doctest block by reStructuredText. It *will* be recognized by the doctest module, though! Indentation is not required for doctest blocks. Tables ------ Doctree elements: table, tgroup, colspec, thead, tbody, row, entry. ReStructuredText provides two syntaxes for delineating table cells: `Grid Tables`_ and `Simple Tables`_. As with other body elements, blank lines are required before and after tables. Tables' left edges should align with the left edge of preceding text blocks; if indented, the table is considered to be part of a block quote. Once isolated, each table cell is treated as a miniature document; the top and bottom cell boundaries act as delimiting blank lines. Each cell contains zero or more body elements. Cell contents may include left and/or right margins, which are removed before processing. Grid Tables ``````````` Grid tables provide a complete table representation via grid-like "ASCII art". Grid tables allow arbitrary cell contents (body elements), and both row and column spans. However, grid tables can be cumbersome to produce, especially for simple data sets. The `Emacs table mode`_ is a tool that allows easy editing of grid tables, in Emacs. See `Simple Tables`_ for a simpler (but limited) representation. Grid tables are described with a visual grid made up of the characters "-", "=", "|", and "+". The hyphen ("-") is used for horizontal lines (row separators). The equals sign ("=") may be used to separate optional header rows from the table body (not supported by the `Emacs table mode`_). The vertical bar ("|") is used for vertical lines (column separators). The plus sign ("+") is used for intersections of horizontal and vertical lines. Example:: +------------------------+------------+----------+----------+ | Header row, column 1 | Header 2 | Header 3 | Header 4 | | (header rows optional) | | | | +========================+============+==========+==========+ | body row 1, column 1 | column 2 | column 3 | column 4 | +------------------------+------------+----------+----------+ | body row 2 | Cells may span columns. | +------------------------+------------+---------------------+ | body row 3 | Cells may | - Table cells | +------------------------+ span rows. | - contain | | body row 4 | | - body elements. | +------------------------+------------+---------------------+ Some care must be taken with grid tables to avoid undesired interactions with cell text in rare cases. For example, the following table contains a cell in row 2 spanning from column 2 to column 4:: +--------------+----------+-----------+-----------+ | row 1, col 1 | column 2 | column 3 | column 4 | +--------------+----------+-----------+-----------+ | row 2 | | +--------------+----------+-----------+-----------+ | row 3 | | | | +--------------+----------+-----------+-----------+ If a vertical bar is used in the text of that cell, it could have unintended effects if accidentally aligned with column boundaries:: +--------------+----------+-----------+-----------+ | row 1, col 1 | column 2 | column 3 | column 4 | +--------------+----------+-----------+-----------+ | row 2 | Use the command ``ls | more``. | +--------------+----------+-----------+-----------+ | row 3 | | | | +--------------+----------+-----------+-----------+ Several solutions are possible. All that is needed is to break the continuity of the cell outline rectangle. One possibility is to shift the text by adding an extra space before:: +--------------+----------+-----------+-----------+ | row 1, col 1 | column 2 | column 3 | column 4 | +--------------+----------+-----------+-----------+ | row 2 | Use the command ``ls | more``. | +--------------+----------+-----------+-----------+ | row 3 | | | | +--------------+----------+-----------+-----------+ Another possibility is to add an extra line to row 2:: +--------------+----------+-----------+-----------+ | row 1, col 1 | column 2 | column 3 | column 4 | +--------------+----------+-----------+-----------+ | row 2 | Use the command ``ls | more``. | | | | +--------------+----------+-----------+-----------+ | row 3 | | | | +--------------+----------+-----------+-----------+ Simple Tables ````````````` Simple tables provide a compact and easy to type but limited row-oriented table representation for simple data sets. Cell contents are typically single paragraphs, although arbitrary body elements may be represented in most cells. Simple tables allow multi-line rows (in all but the first column) and column spans, but not row spans. See `Grid Tables`_ above for a complete table representation. Simple tables are described with horizontal borders made up of "=" and "-" characters. The equals sign ("=") is used for top and bottom table borders, and to separate optional header rows from the table body. The hyphen ("-") is used to indicate column spans in a single row by underlining the joined columns. A simple table begins with a top border of equals signs with one or more spaces at each column boundary (two or more spaces recommended). Regardless of spans, the top border *must* fully describe all table columns. There must be at least two columns in the table (to differentiate it from section headers). The last of the optional header rows is underlined with '=', again with spaces at column boundaries. There may not be a blank line below the header row separator; it would be interpreted as the bottom border of the table. The bottom boundary of the table consists of '=' underlines, also with spaces at column boundaries. For example, here is a truth table, a three-column table with one header row and four body rows:: ===== ===== ======= A B A and B ===== ===== ======= False False False True False False False True False True True True ===== ===== ======= Underlines of '-' may be used to indicate column spans by "filling in" column margins to join adjacent columns. Column span underlines must be complete (they must cover all columns) and align with established column boundaries. Text lines containing column span underlines may not contain any other text. A column span underline applies only to one row immediately above it. For example, here is a table with a column span in the header:: ===== ===== ====== Inputs Output ------------ ------ A B A or B ===== ===== ====== False False False True False True False True True True True True ===== ===== ====== Each line of text must contain spaces at column boundaries, except where cells have been joined by column spans. Each line of text starts a new row, except when there is a blank cell in the first column. In that case, that line of text is parsed as a continuation line. For this reason, cells in the first column of new rows (*not* continuation lines) *must* contain some text; blank cells would lead to a misinterpretation. An empty comment ("..") is sufficient and will be omitted from the processed output (see Comments_ below). Also, this mechanism limits cells in the first column to only one line of text. Use `grid tables`_ if this limitation is unacceptable. Underlines of '-' may also be used to visually separate rows, even if there are no column spans. This is especially useful in long tables, where rows are many lines long. Blank lines are permitted within simple tables. Their interpretation depends on the context. Blank lines *between* rows are ignored. Blank lines *within* multi-line rows may separate paragraphs or other body elements within cells. The rightmost column is unbounded; text may continue past the edge of the table (as indicated by the table borders). However, it is recommended that borders be made long enough to contain the entire text. The following example illustrates continuation lines (row 2 consists of two lines of text, and four lines for row 3), a blank line separating paragraphs (row 3, column 2), and text extending past the right edge of the table:: ===== ===== col 1 col 2 ===== ===== 1 Second column of row 1. 2 Second column of row 2. Second line of paragraph. 3 - Second column of row 3. - Second item in bullet list (row 3, column 2). ===== ===== Explicit Markup Blocks ---------------------- An explicit markup block is a text block: - whose first line begins with ".." followed by whitespace (the "explicit markup start"), - whose second and subsequent lines (if any) are indented relative to the first, and - which ends before an unindented line. Explicit markup blocks are analogous to bullet list items, with ".." as the bullet. The text immediately after the explicit markup start determines the indentation of the block body. Blank lines are required between explicit markup blocks and other elements, but are optional between explicit markup blocks where unambiguous. The explicit markup syntax is used for footnotes, citations, hyperlink targets, directives, substitution definitions, and comments. Footnotes ````````` Doctree elements: footnote, label. Each footnote consists of an explicit markup start (".. "), a left square bracket, the footnote label, a right square bracket, and whitespace, followed by indented body elements. A footnote label can be: - a whole decimal number consisting of one or more digits, - a single "#" (denoting `auto-numbered footnotes`_), - a "#" followed by a simple reference name (an `autonumber label`_), or - a single "*" (denoting `auto-symbol footnotes`_). If the first body element within a footnote is a simple paragraph, it may begin on the same line as the footnote label. Other elements must begin on a new line, consistently indented (by at least 3 spaces) and left-aligned. Footnotes may occur anywhere in the document, not only at the end. Where or how they appear in the processed output depends on the processing system. Here is a manually numbered footnote:: .. [1] Body elements go here. Each footnote automatically generates a hyperlink target pointing to itself. The text of the hyperlink target name is the same as that of the footnote label. `Auto-numbered footnotes`_ generate a number as their footnote label and reference name. See `Implicit Hyperlink Targets`_ for a complete description of the mechanism. Syntax diagram:: +-------+-------------------------+ | ".. " | "[" label "]" footnote | +-------+ | | (body elements)+ | +-------------------------+ Auto-Numbered Footnotes ....................... A number sign ("#") may be used as the first character of a footnote label to request automatic numbering of the footnote or footnote reference. The first footnote to request automatic numbering is assigned the label "1", the second is assigned the label "2", and so on (assuming there are no manually numbered footnotes present; see `Mixed Manual and Auto-Numbered Footnotes`_ below). A footnote which has automatically received a label "1" generates an implicit hyperlink target with name "1", just as if the label was explicitly specified. .. _autonumber label: `autonumber labels`_ A footnote may specify a label explicitly while at the same time requesting automatic numbering: ``[#label]``. These labels are called _`autonumber labels`. Autonumber labels do two things: - On the footnote itself, they generate a hyperlink target whose name is the autonumber label (doesn't include the "#"). - They allow an automatically numbered footnote to be referred to more than once, as a footnote reference or hyperlink reference. For example:: If [#note]_ is the first footnote reference, it will show up as "[1]". We can refer to it again as [#note]_ and again see "[1]". We can also refer to it as note_ (an ordinary internal hyperlink reference). .. [#note] This is the footnote labeled "note". The numbering is determined by the order of the footnotes, not by the order of the references. For footnote references without autonumber labels (``[#]_``), the footnotes and footnote references must be in the same relative order but need not alternate in lock-step. For example:: [#]_ is a reference to footnote 1, and [#]_ is a reference to footnote 2. .. [#] This is footnote 1. .. [#] This is footnote 2. .. [#] This is footnote 3. [#]_ is a reference to footnote 3. Special care must be taken if footnotes themselves contain auto-numbered footnote references, or if multiple references are made in close proximity. Footnotes and references are noted in the order they are encountered in the document, which is not necessarily the same as the order in which a person would read them. Auto-Symbol Footnotes ..................... An asterisk ("*") may be used for footnote labels to request automatic symbol generation for footnotes and footnote references. The asterisk may be the only character in the label. For example:: Here is a symbolic footnote reference: [*]_. .. [*] This is the footnote. A transform will insert symbols as labels into corresponding footnotes and footnote references. The number of references must be equal to the number of footnotes. One symbol footnote cannot have multiple references. The standard Docutils system uses the following symbols for footnote marks [#]_: - asterisk/star ("*") - dagger (HTML character entity "†", Unicode U+02020) - double dagger ("‡"/U+02021) - section mark ("§"/U+000A7) - pilcrow or paragraph mark ("¶"/U+000B6) - number sign ("#") - spade suit ("♠"/U+02660) - heart suit ("♥"/U+02665) - diamond suit ("♦"/U+02666) - club suit ("♣"/U+02663) .. [#] This list was inspired by the list of symbols for "Note Reference Marks" in The Chicago Manual of Style, 14th edition, section 12.51. "Parallels" ("||") were given in CMoS instead of the pilcrow. The last four symbols (the card suits) were added arbitrarily. If more than ten symbols are required, the same sequence will be reused, doubled and then tripled, and so on ("**" etc.). .. Note:: When using auto-symbol footnotes, the choice of output encoding is important. Many of the symbols used are not encodable in certain common text encodings such as Latin-1 (ISO 8859-1). The use of UTF-8 for the output encoding is recommended. An alternative for HTML and XML output is to use the "xmlcharrefreplace" `output encoding error handler`__. __ ../../user/config.html#output-encoding-error-handler Mixed Manual and Auto-Numbered Footnotes ........................................ Manual and automatic footnote numbering may both be used within a single document, although the results may not be expected. Manual numbering takes priority. Only unused footnote numbers are assigned to auto-numbered footnotes. The following example should be illustrative:: [2]_ will be "2" (manually numbered), [#]_ will be "3" (anonymous auto-numbered), and [#label]_ will be "1" (labeled auto-numbered). .. [2] This footnote is labeled manually, so its number is fixed. .. [#label] This autonumber-labeled footnote will be labeled "1". It is the first auto-numbered footnote and no other footnote with label "1" exists. The order of the footnotes is used to determine numbering, not the order of the footnote references. .. [#] This footnote will be labeled "3". It is the second auto-numbered footnote, but footnote label "2" is already used. Citations ````````` Citations are identical to footnotes except that they use only non-numeric labels such as ``[note]`` or ``[GVR2001]``. Citation labels are simple `reference names`_ (case-insensitive single words consisting of alphanumerics plus internal hyphens, underscores, and periods; no whitespace). Citations may be rendered separately and differently from footnotes. For example:: Here is a citation reference: [CIT2002]_. .. [CIT2002] This is the citation. It's just like a footnote, except the label is textual. .. _hyperlinks: Hyperlink Targets ````````````````` Doctree element: target. These are also called _`explicit hyperlink targets`, to differentiate them from `implicit hyperlink targets`_ defined below. Hyperlink targets identify a location within or outside of a document, which may be linked to by `hyperlink references`_. Hyperlink targets may be named or anonymous. Named hyperlink targets consist of an explicit markup start (".. "), an underscore, the reference name (no trailing underscore), a colon, whitespace, and a link block:: .. _hyperlink-name: link-block Reference names are whitespace-neutral and case-insensitive. See `Reference Names`_ for details and examples. Anonymous hyperlink targets consist of an explicit markup start (".. "), two underscores, a colon, whitespace, and a link block; there is no reference name:: .. __: anonymous-hyperlink-target-link-block An alternate syntax for anonymous hyperlinks consists of two underscores, a space, and a link block:: __ anonymous-hyperlink-target-link-block See `Anonymous Hyperlinks`_ below. There are three types of hyperlink targets: internal, external, and indirect. 1. _`Internal hyperlink targets` have empty link blocks. They provide an end point allowing a hyperlink to connect one place to another within a document. An internal hyperlink target points to the element following the target. For example:: Clicking on this internal hyperlink will take us to the target_ below. .. _target: The hyperlink target above points to this paragraph. Internal hyperlink targets may be "chained". Multiple adjacent internal hyperlink targets all point to the same element:: .. _target1: .. _target2: The targets "target1" and "target2" are synonyms; they both point to this paragraph. If the element "pointed to" is an external hyperlink target (with a URI in its link block; see #2 below) the URI from the external hyperlink target is propagated to the internal hyperlink targets; they will all "point to" the same URI. There is no need to duplicate a URI. For example, all three of the following hyperlink targets refer to the same URI:: .. _Python DOC-SIG mailing list archive: .. _archive: .. _Doc-SIG: http://mail.python.org/pipermail/doc-sig/ An inline form of internal hyperlink target is available; see `Inline Internal Targets`_. 2. _`External hyperlink targets` have an absolute or relative URI or email address in their link blocks. For example, take the following input:: See the Python_ home page for info. `Write to me`_ with your questions. .. _Python: http://www.python.org .. _Write to me: jdoe@example.com After processing into HTML, the hyperlinks might be expressed as:: See the Python home page for info. Write to me with your questions. An external hyperlink's URI may begin on the same line as the explicit markup start and target name, or it may begin in an indented text block immediately following, with no intervening blank lines. If there are multiple lines in the link block, they are stripped of leading and trailing whitespace and concatenated. The following external hyperlink targets are equivalent:: .. _one-liner: http://docutils.sourceforge.net/rst.html .. _starts-on-this-line: http:// docutils.sourceforge.net/rst.html .. _entirely-below: http://docutils. sourceforge.net/rst.html If an external hyperlink target's URI contains an underscore as its last character, it must be escaped to avoid being mistaken for an indirect hyperlink target:: This link_ refers to a file called ``underscore_``. .. _link: underscore\_ It is possible (although not generally recommended) to include URIs directly within hyperlink references. See `Embedded URIs`_ below. 3. _`Indirect hyperlink targets` have a hyperlink reference in their link blocks. In the following example, target "one" indirectly references whatever target "two" references, and target "two" references target "three", an internal hyperlink target. In effect, all three reference the same thing:: .. _one: two_ .. _two: three_ .. _three: Just as with `hyperlink references`_ anywhere else in a document, if a phrase-reference is used in the link block it must be enclosed in backquotes. As with `external hyperlink targets`_, the link block of an indirect hyperlink target may begin on the same line as the explicit markup start or the next line. It may also be split over multiple lines, in which case the lines are joined with whitespace before being normalized. For example, the following indirect hyperlink targets are equivalent:: .. _one-liner: `A HYPERLINK`_ .. _entirely-below: `a hyperlink`_ .. _split: `A Hyperlink`_ If a reference name contains a colon followed by whitespace, either: - the phrase must be enclosed in backquotes:: .. _`FAQTS: Computers: Programming: Languages: Python`: http://python.faqts.com/ - or the colon(s) must be backslash-escaped in the link target:: .. _Chapter One\: "Tadpole Days": It's not easy being green... See `Implicit Hyperlink Targets`_ below for the resolution of duplicate reference names. Syntax diagram:: +-------+----------------------+ | ".. " | "_" name ":" link | +-------+ block | | | +----------------------+ Anonymous Hyperlinks .................... The `World Wide Web Consortium`_ recommends in its `HTML Techniques for Web Content Accessibility Guidelines`_ that authors should "clearly identify the target of each link." Hyperlink references should be as verbose as possible, but duplicating a verbose hyperlink name in the target is onerous and error-prone. Anonymous hyperlinks are designed to allow convenient verbose hyperlink references, and are analogous to `Auto-Numbered Footnotes`_. They are particularly useful in short or one-off documents. However, this feature is easily abused and can result in unreadable plaintext and/or unmaintainable documents. Caution is advised. Anonymous `hyperlink references`_ are specified with two underscores instead of one:: See `the web site of my favorite programming language`__. Anonymous targets begin with ".. __:"; no reference name is required or allowed:: .. __: http://www.python.org As a convenient alternative, anonymous targets may begin with "__" only:: __ http://www.python.org The reference name of the reference is not used to match the reference to its target. Instead, the order of anonymous hyperlink references and targets within the document is significant: the first anonymous reference will link to the first anonymous target. The number of anonymous hyperlink references in a document must match the number of anonymous targets. For readability, it is recommended that targets be kept close to references. Take care when editing text containing anonymous references; adding, removing, and rearranging references require attention to the order of corresponding targets. Directives `````````` Doctree elements: depend on the directive. Directives are an extension mechanism for reStructuredText, a way of adding support for new constructs without adding new primary syntax (directives may support additional syntax locally). All standard directives (those implemented and registered in the reference reStructuredText parser) are described in the `reStructuredText Directives`_ document, and are always available. Any other directives are domain-specific, and may require special action to make them available when processing the document. For example, here's how an image may be placed:: .. image:: mylogo.jpeg A figure (a graphic with a caption) may placed like this:: .. figure:: larch.png The larch. An admonition (note, caution, etc.) contains other body elements:: .. note:: This is a paragraph - Here is a bullet list. Directives are indicated by an explicit markup start (".. ") followed by the directive type, two colons, and whitespace (together called the "directive marker"). Directive types are case-insensitive single words (alphanumerics plus internal hyphens, underscores, and periods; no whitespace). Two colons are used after the directive type for these reasons: - Two colons are distinctive, and unlikely to be used in common text. - Two colons avoids clashes with common comment text like:: .. Danger: modify at your own risk! - If an implementation of reStructuredText does not recognize a directive (i.e., the directive-handler is not installed), a level-3 (error) system message is generated, and the entire directive block (including the directive itself) will be included as a literal block. Thus "::" is a natural choice. The directive block is consists of any text on the first line of the directive after the directive marker, and any subsequent indented text. The interpretation of the directive block is up to the directive code. There are three logical parts to the directive block: 1. Directive arguments. 2. Directive options. 3. Directive content. Individual directives can employ any combination of these parts. Directive arguments can be filesystem paths, URLs, title text, etc. Directive options are indicated using `field lists`_; the field names and contents are directive-specific. Arguments and options must form a contiguous block beginning on the first or second line of the directive; a blank line indicates the beginning of the directive content block. If either arguments and/or options are employed by the directive, a blank line must separate them from the directive content. The "figure" directive employs all three parts:: .. figure:: larch.png :scale: 50 The larch. Simple directives may not require any content. If a directive that does not employ a content block is followed by indented text anyway, it is an error. If a block quote should immediately follow a directive, use an empty comment in-between (see Comments_ below). Actions taken in response to directives and the interpretation of text in the directive content block or subsequent text block(s) are directive-dependent. See `reStructuredText Directives`_ for details. Directives are meant for the arbitrary processing of their contents, which can be transformed into something possibly unrelated to the original text. It may also be possible for directives to be used as pragmas, to modify the behavior of the parser, such as to experiment with alternate syntax. There is no parser support for this functionality at present; if a reasonable need for pragma directives is found, they may be supported. Directives do not generate "directive" elements; they are a *parser construct* only, and have no intrinsic meaning outside of reStructuredText. Instead, the parser will transform recognized directives into (possibly specialized) document elements. Unknown directives will trigger level-3 (error) system messages. Syntax diagram:: +-------+-------------------------------+ | ".. " | directive type "::" directive | +-------+ block | | | +-------------------------------+ Substitution Definitions ```````````````````````` Doctree element: substitution_definition. Substitution definitions are indicated by an explicit markup start (".. ") followed by a vertical bar, the substitution text, another vertical bar, whitespace, and the definition block. Substitution text may not begin or end with whitespace. A substitution definition block contains an embedded inline-compatible directive (without the leading ".. "), such as an image. For example:: The |biohazard| symbol must be used on containers used to dispose of medical waste. .. |biohazard| image:: biohazard.png It is an error for a substitution definition block to directly or indirectly contain a circular substitution reference. `Substitution references`_ are replaced in-line by the processed contents of the corresponding definition (linked by matching substitution text). Matches are case-sensitive but forgiving; if no exact match is found, a case-insensitive comparison is attempted. Substitution definitions allow the power and flexibility of block-level directives_ to be shared by inline text. They are a way to include arbitrarily complex inline structures within text, while keeping the details out of the flow of text. They are the equivalent of SGML/XML's named entities or programming language macros. Without the substitution mechanism, every time someone wants an application-specific new inline structure, they would have to petition for a syntax change. In combination with existing directive syntax, any inline structure can be coded without new syntax (except possibly a new directive). Syntax diagram:: +-------+-----------------------------------------------------+ | ".. " | "|" substitution text "| " directive type "::" data | +-------+ directive block | | | +-----------------------------------------------------+ Following are some use cases for the substitution mechanism. Please note that most of the embedded directives shown are examples only and have not been implemented. Objects Substitution references may be used to associate ambiguous text with a unique object identifier. For example, many sites may wish to implement an inline "user" directive:: |Michael| and |Jon| are our widget-wranglers. .. |Michael| user:: mjones .. |Jon| user:: jhl Depending on the needs of the site, this may be used to index the document for later searching, to hyperlink the inline text in various ways (mailto, homepage, mouseover Javascript with profile and contact information, etc.), or to customize presentation of the text (include username in the inline text, include an icon image with a link next to the text, make the text bold or a different color, etc.). The same approach can be used in documents which frequently refer to a particular type of objects with unique identifiers but ambiguous common names. Movies, albums, books, photos, court cases, and laws are possible. For example:: |The Transparent Society| offers a fascinating alternate view on privacy issues. .. |The Transparent Society| book:: isbn=0738201448 Classes or functions, in contexts where the module or class names are unclear and/or interpreted text cannot be used, are another possibility:: 4XSLT has the convenience method |runString|, so you don't have to mess with DOM objects if all you want is the transformed output. .. |runString| function:: module=xml.xslt class=Processor Images Images are a common use for substitution references:: West led the |H| 3, covered by dummy's |H| Q, East's |H| K, and trumped in hand with the |S| 2. .. |H| image:: /images/heart.png :height: 11 :width: 11 .. |S| image:: /images/spade.png :height: 11 :width: 11 * |Red light| means stop. * |Green light| means go. * |Yellow light| means go really fast. .. |Red light| image:: red_light.png .. |Green light| image:: green_light.png .. |Yellow light| image:: yellow_light.png |-><-| is the official symbol of POEE_. .. |-><-| image:: discord.png .. _POEE: http://www.poee.org/ The "image" directive has been implemented. Styles [#]_ Substitution references may be used to associate inline text with an externally defined presentation style:: Even |the text in Texas| is big. .. |the text in Texas| style:: big The style name may be meaningful in the context of some particular output format (CSS class name for HTML output, LaTeX style name for LaTeX, etc), or may be ignored for other output formats (such as plaintext). .. @@@ This needs to be rethought & rewritten or removed: Interpreted text is unsuitable for this purpose because the set of style names cannot be predefined - it is the domain of the content author, not the author of the parser and output formatter - and there is no way to associate a style name argument with an interpreted text style role. Also, it may be desirable to use the same mechanism for styling blocks:: .. style:: motto At Bob's Underwear Shop, we'll do anything to get in your pants. .. style:: disclaimer All rights reversed. Reprint what you like. .. [#] There may be sufficient need for a "style" mechanism to warrant simpler syntax such as an extension to the interpreted text role syntax. The substitution mechanism is cumbersome for simple text styling. Templates Inline markup may be used for later processing by a template engine. For example, a Zope_ author might write:: Welcome back, |name|! .. |name| tal:: replace user/getUserName After processing, this ZPT output would result:: Welcome back, name! Zope would then transform this to something like "Welcome back, David!" during a session with an actual user. Replacement text The substitution mechanism may be used for simple macro substitution. This may be appropriate when the replacement text is repeated many times throughout one or more documents, especially if it may need to change later. A short example is unavoidably contrived:: |RST| is a little annoying to type over and over, especially when writing about |RST| itself, and spelling out the bicapitalized word |RST| every time isn't really necessary for |RST| source readability. .. |RST| replace:: reStructuredText_ .. _reStructuredText: http://docutils.sourceforge.net/rst.html Substitution is also appropriate when the replacement text cannot be represented using other inline constructs, or is obtrusively long:: But still, that's nothing compared to a name like |j2ee-cas|__. .. |j2ee-cas| replace:: the Java `TM`:super: 2 Platform, Enterprise Edition Client Access Services __ http://developer.java.sun.com/developer/earlyAccess/ j2eecas/ Comments ```````` Doctree element: comment. Arbitrary indented text may follow the explicit markup start and will be processed as a comment element. No further processing is done on the comment block text; a comment contains a single "text blob". Depending on the output formatter, comments may be removed from the processed output. The only restriction on comments is that they not use the same syntax as any of the other explicit markup constructs: substitution definitions, directives, footnotes, citations, or hyperlink targets. To ensure that none of the other explicit markup constructs is recognized, leave the ".." on a line by itself:: .. This is a comment .. _so: is this! .. [and] this! .. this:: too! .. |even| this:: ! A explicit markup start followed by a blank line and nothing else (apart from whitespace) is an "empty comment". It serves to terminate a preceding construct, and does **not** consume any indented text following. To have a block quote follow a list or any indented construct, insert an unindented empty comment in-between. Syntax diagram:: +-------+----------------------+ | ".. " | comment | +-------+ block | | | +----------------------+ Implicit Hyperlink Targets ========================== Implicit hyperlink targets are generated by section titles, footnotes, and citations, and may also be generated by extension constructs. Implicit hyperlink targets otherwise behave identically to explicit `hyperlink targets`_. Problems of ambiguity due to conflicting duplicate implicit and explicit reference names are avoided by following this procedure: 1. `Explicit hyperlink targets`_ override any implicit targets having the same reference name. The implicit hyperlink targets are removed, and level-1 (info) system messages are inserted. 2. Duplicate implicit hyperlink targets are removed, and level-1 (info) system messages inserted. For example, if two or more sections have the same title (such as "Introduction" subsections of a rigidly-structured document), there will be duplicate implicit hyperlink targets. 3. Duplicate explicit hyperlink targets are removed, and level-2 (warning) system messages are inserted. Exception: duplicate `external hyperlink targets`_ (identical hyperlink names and referenced URIs) do not conflict, and are not removed. System messages are inserted where target links have been removed. See "Error Handling" in `PEP 258`_. The parser must return a set of *unique* hyperlink targets. The calling software (such as the Docutils_) can warn of unresolvable links, giving reasons for the messages. Inline Markup ============= In reStructuredText, inline markup applies to words or phrases within a text block. The same whitespace and punctuation that serves to delimit words in written text is used to delimit the inline markup syntax constructs. The text within inline markup may not begin or end with whitespace. Arbitrary `character-level inline markup`_ is supported although not encouraged. Inline markup cannot be nested. There are nine inline markup constructs. Five of the constructs use identical start-strings and end-strings to indicate the markup: - emphasis_: "*" - `strong emphasis`_: "**" - `interpreted text`_: "`" - `inline literals`_: "``" - `substitution references`_: "|" Three constructs use different start-strings and end-strings: - `inline internal targets`_: "_`" and "`" - `footnote references`_: "[" and "]_" - `hyperlink references`_: "`" and "\`_" (phrases), or just a trailing "_" (single words) `Standalone hyperlinks`_ are recognized implicitly, and use no extra markup. The inline markup start-string and end-string recognition rules are as follows. If any of the conditions are not met, the start-string or end-string will not be recognized or processed. 1. Inline markup start-strings must start a text block or be immediately preceded by whitespace or one of:: ' " ( [ { < - / : 2. Inline markup start-strings must be immediately followed by non-whitespace. 3. Inline markup end-strings must be immediately preceded by non-whitespace. 4. Inline markup end-strings must end a text block or be immediately followed by whitespace or one of:: ' " ) ] } > - / : . , ; ! ? \ 5. If an inline markup start-string is immediately preceded by a single or double quote, "(", "[", "{", or "<", it must not be immediately followed by the corresponding single or double quote, ")", "]", "}", or ">". 6. An inline markup end-string must be separated by at least one character from the start-string. 7. An unescaped backslash preceding a start-string or end-string will disable markup recognition, except for the end-string of `inline literals`_. See `Escaping Mechanism`_ above for details. For example, none of the following are recognized as containing inline markup start-strings: - asterisks: * "*" '*' (*) (* [*] {*} 1*x BOM32_* - double asterisks: ** a**b O(N**2) etc. - backquotes: ` `` etc. - underscores: _ __ __init__ __init__() etc. - vertical bars: | || etc. It may be desirable to use inline literals for some of these anyhow, especially if they represent code snippets. It's a judgment call. These cases *do* require either literal-quoting or escaping to avoid misinterpretation:: *4, class_, *args, **kwargs, `TeX-quoted', *ML, *.txt The inline markup recognition rules were devised intentionally to allow 90% of non-markup uses of "*", "`", "_", and "|" *without* resorting to backslashes. For 9 of the remaining 10%, use inline literals or literal blocks:: "``\*``" -> "\*" (possibly in another font or quoted) Only those who understand the escaping and inline markup rules should attempt the remaining 1%. ;-) Inline markup delimiter characters are used for multiple constructs, so to avoid ambiguity there must be a specific recognition order for each character. The inline markup recognition order is as follows: - Asterisks: `Strong emphasis`_ ("**") is recognized before emphasis_ ("*"). - Backquotes: `Inline literals`_ ("``"), `inline internal targets`_ (leading "_`", trailing "`"), are mutually independent, and are recognized before phrase `hyperlink references`_ (leading "`", trailing "\`_") and `interpreted text`_ ("`"). - Trailing underscores: Footnote references ("[" + label + "]_") and simple `hyperlink references`_ (name + trailing "_") are mutually independent. - Vertical bars: `Substitution references`_ ("|") are independently recognized. - `Standalone hyperlinks`_ are the last to be recognized. Character-Level Inline Markup ----------------------------- It is possible to mark up individual characters within a word with backslash escapes (see `Escaping Mechanism`_ above). Backslash escapes can be used to allow arbitrary text to immediately follow inline markup:: Python ``list``\s use square bracket syntax. The backslash will disappear from the processed document. The word "list" will appear as inline literal text, and the letter "s" will immediately follow it as normal text, with no space in-between. Arbitrary text may immediately precede inline markup using backslash-escaped whitespace:: Possible in *re*\ ``Structured``\ *Text*, though not encouraged. The backslashes and spaces separating "re", "Structured", and "Text" above will disappear from the processed document. .. CAUTION:: The use of backslash-escapes for character-level inline markup is not encouraged. Such use is ugly and detrimental to the unprocessed document's readability. Please use this feature sparingly and only where absolutely necessary. Emphasis -------- Doctree element: emphasis. Start-string = end-string = "*". Text enclosed by single asterisk characters is emphasized:: This is *emphasized text*. Emphasized text is typically displayed in italics. Strong Emphasis --------------- Doctree element: strong. Start-string = end-string = "**". Text enclosed by double-asterisks is emphasized strongly:: This is **strong text**. Strongly emphasized text is typically displayed in boldface. Interpreted Text ---------------- Doctree element: depends on the explicit or implicit role and processing. Start-string = end-string = "`". Interpreted text is text that is meant to be related, indexed, linked, summarized, or otherwise processed, but the text itself is typically left alone. Interpreted text is enclosed by single backquote characters:: This is `interpreted text`. The "role" of the interpreted text determines how the text is interpreted. The role may be inferred implicitly (as above; the "default role" is used) or indicated explicitly, using a role marker. A role marker consists of a colon, the role name, and another colon. A role name is a single word consisting of alphanumerics plus internal hyphens, underscores, and periods; no whitespace or other characters are allowed. A role marker is either a prefix or a suffix to the interpreted text, whichever reads better; it's up to the author:: :role:`interpreted text` `interpreted text`:role: Interpreted text allows extensions to the available inline descriptive markup constructs. To emphasis_, `strong emphasis`_, `inline literals`_, and `hyperlink references`_, we can add "title reference", "index entry", "acronym", "class", "red", "blinking" or anything else we want. Only pre-determined roles are recognized; unknown roles will generate errors. A core set of standard roles is implemented in the reference parser; see `reStructuredText Interpreted Text Roles`_ for individual descriptions. In addition, applications may support specialized roles. Inline Literals --------------- Doctree element: literal. Start-string = end-string = "``". Text enclosed by double-backquotes is treated as inline literals:: This text is an example of ``inline literals``. Inline literals may contain any characters except two adjacent backquotes in an end-string context (according to the recognition rules above). No markup interpretation (including backslash-escape interpretation) is done within inline literals. Line breaks are *not* preserved in inline literals. Although a reStructuredText parser will preserve runs of spaces in its output, the final representation of the processed document is dependent on the output formatter, thus the preservation of whitespace cannot be guaranteed. If the preservation of line breaks and/or other whitespace is important, `literal blocks`_ should be used. Inline literals are useful for short code snippets. For example:: The regular expression ``[+-]?(\d+(\.\d*)?|\.\d+)`` matches floating-point numbers (without exponents). Hyperlink References -------------------- Doctree element: reference. - Named hyperlink references: - Start-string = "" (empty string), end-string = "_". - Start-string = "`", end-string = "\`_". (Phrase references.) - Anonymous hyperlink references: - Start-string = "" (empty string), end-string = "__". - Start-string = "`", end-string = "\`__". (Phrase references.) Hyperlink references are indicated by a trailing underscore, "_", except for `standalone hyperlinks`_ which are recognized independently. The underscore can be thought of as a right-pointing arrow. The trailing underscores point away from hyperlink references, and the leading underscores point toward `hyperlink targets`_. Hyperlinks consist of two parts. In the text body, there is a source link, a reference name with a trailing underscore (or two underscores for `anonymous hyperlinks`_):: See the Python_ home page for info. A target link with a matching reference name must exist somewhere else in the document. See `Hyperlink Targets`_ for a full description). `Anonymous hyperlinks`_ (which see) do not use reference names to match references to targets, but otherwise behave similarly to named hyperlinks. Embedded URIs ````````````` A hyperlink reference may directly embed a target URI inline, within angle brackets ("<...>") as follows:: See the `Python home page `_ for info. This is exactly equivalent to:: See the `Python home page`_ for info. .. _Python home page: http://www.python.org The bracketed URI must be preceded by whitespace and be the last text before the end string. With a single trailing underscore, the reference is named and the same target URI may be referred to again. With two trailing underscores, the reference and target are both anonymous, and the target cannot be referred to again. These are "one-off" hyperlinks. For example:: `RFC 2396 `__ and `RFC 2732 `__ together define the syntax of URIs. Equivalent to:: `RFC 2396`__ and `RFC 2732`__ together define the syntax of URIs. __ http://www.rfc-editor.org/rfc/rfc2396.txt __ http://www.rfc-editor.org/rfc/rfc2732.txt If reference text happens to end with angle-bracketed text that is *not* a URI, the open-angle-bracket needs to be backslash-escaped. For example, here is a reference to a title describing a tag:: See `HTML Element: \`_ below. The reference text may also be omitted, in which case the URI will be duplicated for use as the reference text. This is useful for relative URIs where the address or file name is also the desired reference text:: See ``_ or ``__ for details. .. CAUTION:: This construct offers easy authoring and maintenance of hyperlinks at the expense of general readability. Inline URIs, especially long ones, inevitably interrupt the natural flow of text. For documents meant to be read in source form, the use of independent block-level `hyperlink targets`_ is **strongly recommended**. The embedded URI construct is most suited to documents intended *only* to be read in processed form. Inline Internal Targets ------------------------ Doctree element: target. Start-string = "_`", end-string = "`". Inline internal targets are the equivalent of explicit `internal hyperlink targets`_, but may appear within running text. The syntax begins with an underscore and a backquote, is followed by a hyperlink name or phrase, and ends with a backquote. Inline internal targets may not be anonymous. For example, the following paragraph contains a hyperlink target named "Norwegian Blue":: Oh yes, the _`Norwegian Blue`. What's, um, what's wrong with it? See `Implicit Hyperlink Targets`_ for the resolution of duplicate reference names. Footnote References ------------------- Doctree element: footnote_reference. Start-string = "[", end-string = "]_". Each footnote reference consists of a square-bracketed label followed by a trailing underscore. Footnote labels are one of: - one or more digits (i.e., a number), - a single "#" (denoting `auto-numbered footnotes`_), - a "#" followed by a simple reference name (an `autonumber label`_), or - a single "*" (denoting `auto-symbol footnotes`_). For example:: Please RTFM [1]_. .. [1] Read The Fine Manual Citation References ------------------- Doctree element: citation_reference. Start-string = "[", end-string = "]_". Each citation reference consists of a square-bracketed label followed by a trailing underscore. Citation labels are simple `reference names`_ (case-insensitive single words, consisting of alphanumerics plus internal hyphens, underscores, and periods; no whitespace). For example:: Here is a citation reference: [CIT2002]_. See Citations_ for the citation itself. Substitution References ----------------------- Doctree element: substitution_reference, reference. Start-string = "|", end-string = "|" (optionally followed by "_" or "__"). Vertical bars are used to bracket the substitution reference text. A substitution reference may also be a hyperlink reference by appending a "_" (named) or "__" (anonymous) suffix; the substitution text is used for the reference text in the named case. The processing system replaces substitution references with the processed contents of the corresponding `substitution definitions`_ (which see for the definition of "correspond"). Substitution definitions produce inline-compatible elements. Examples:: This is a simple |substitution reference|. It will be replaced by the processing system. This is a combination |substitution and hyperlink reference|_. In addition to being replaced, the replacement text or element will refer to the "substitution and hyperlink reference" target. Standalone Hyperlinks --------------------- Doctree element: reference. Start-string = end-string = "" (empty string). A URI (absolute URI [#URI]_ or standalone email address) within a text block is treated as a general external hyperlink with the URI itself as the link's text. For example:: See http://www.python.org for info. would be marked up in HTML as:: See http://www.python.org for info. Two forms of URI are recognized: 1. Absolute URIs. These consist of a scheme, a colon (":"), and a scheme-specific part whose interpretation depends on the scheme. The scheme is the name of the protocol, such as "http", "ftp", "mailto", or "telnet". The scheme consists of an initial letter, followed by letters, numbers, and/or "+", "-", ".". Recognition is limited to known schemes, per the `Official IANA Registry of URI Schemes`_ and the W3C's `Retired Index of WWW Addressing Schemes`_. The scheme-specific part of the resource identifier may be either hierarchical or opaque: - Hierarchical identifiers begin with one or two slashes and may use slashes to separate hierarchical components of the path. Examples are web pages and FTP sites:: http://www.python.org ftp://ftp.python.org/pub/python - Opaque identifiers do not begin with slashes. Examples are email addresses and newsgroups:: mailto:someone@somewhere.com news:comp.lang.python With queries, fragments, and %-escape sequences, URIs can become quite complicated. A reStructuredText parser must be able to recognize any absolute URI, as defined in RFC2396_ and RFC2732_. 2. Standalone email addresses, which are treated as if they were absolute URIs with a "mailto:" scheme. Example:: someone@somewhere.com Punctuation at the end of a URI is not considered part of the URI, unless the URI is terminated by a closing angle bracket (">"). Backslashes may be used in URIs to escape markup characters, specifically asterisks ("*") and underscores ("_") which are vaid URI characters (see `Escaping Mechanism`_ above). .. [#URI] Uniform Resource Identifier. URIs are a general form of URLs (Uniform Resource Locators). For the syntax of URIs see RFC2396_ and RFC2732_. ---------------- Error Handling ---------------- Doctree element: system_message, problematic. Markup errors are handled according to the specification in `PEP 258`_. .. _reStructuredText: http://docutils.sourceforge.net/rst.html .. _Docutils: http://docutils.sourceforge.net/ .. _The Docutils Document Tree: ../doctree.html .. _Docutils Generic DTD: ../docutils.dtd .. _transforms: http://docutils.sourceforge.net/docutils/transforms/ .. _Grouch: http://www.mems-exchange.org/software/grouch/ .. _RFC822: http://www.rfc-editor.org/rfc/rfc822.txt .. _DocTitle transform: .. _DocInfo transform: http://docutils.sourceforge.net/docutils/transforms/frontmatter.py .. _getopt.py: http://www.python.org/doc/current/lib/module-getopt.html .. _GNU libc getopt_long(): http://www.gnu.org/software/libc/manual/html_node/Getopt-Long-Options.html .. _doctest module: http://www.python.org/doc/current/lib/module-doctest.html .. _Emacs table mode: http://table.sourceforge.net/ .. _Official IANA Registry of URI Schemes: http://www.iana.org/assignments/uri-schemes .. _Retired Index of WWW Addressing Schemes: http://www.w3.org/Addressing/schemes.html .. _World Wide Web Consortium: http://www.w3.org/ .. _HTML Techniques for Web Content Accessibility Guidelines: http://www.w3.org/TR/WCAG10-HTML-TECHS/#link-text .. _reStructuredText Directives: directives.html .. _reStructuredText Interpreted Text Roles: roles.html .. _RFC2396: http://www.rfc-editor.org/rfc/rfc2396.txt .. _RFC2732: http://www.rfc-editor.org/rfc/rfc2732.txt .. _Zope: http://www.zope.com/ .. _PEP 258: ../../peps/pep-0258.html .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: Zope-2.8.7-final/lib/python/reStructuredText/tests/0000775000076400007640000000000010435610645021335 5ustar ajungajungZope-2.8.7-final/lib/python/reStructuredText/tests/__init__.py0000664000076400007640000000003210435500327023434 0ustar ajungajung"""reST tests package.""" Zope-2.8.7-final/lib/python/reStructuredText/tests/testReST.py0000664000076400007640000000112210435500327023413 0ustar ajungajungimport unittest from reStructuredText import HTML class TestReST(unittest.TestCase): def testRoman(self): # Make sure we can import the rst parser from docutils.parsers import rst def testWithSingleSubtitle(self): input = ''' title ----- subtitle ++++++++ text ''' expected='''

title

subtitle

text

''' output = HTML(input) self.assertEquals(output, expected) def test_suite(): from unittest import TestSuite, makeSuite return TestSuite((makeSuite(TestReST),)) Zope-2.8.7-final/lib/python/ZTUtils/0000775000076400007640000000000010435610645016211 5ustar ajungajungZope-2.8.7-final/lib/python/ZTUtils/CHANGES.txt0000664000076400007640000000142510435500330020011 0ustar ajungajungZTUtils changes This file contains change information for the current release. Change information for previous versions can be found in the file HISTORY.txt. After Version 1.1.4 Features Added - Added 'sequence_length' attribute to batches. Version 1.1.4 Features Added - Under Python 2.2, Iterator both accepts and produces Python iterator objects. - first() and last() methods allow you to tell whether the current element is different from the next or previous element. This is most useful when the sequence is sorted. Bugs Fixed - Handle both string and class Unauthorized exceptions. - Batch construction masked sequence errors, such as Unauthorized. Zope-2.8.7-final/lib/python/ZTUtils/__init__.py0000664000076400007640000000206410435500330020311 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Package of template utility classes and functions. $Id: __init__.py 28963 2005-01-27 14:56:22Z shh $ """ from Iterator import Iterator from Tree import encodeExpansion, decodeExpansion, a2b, b2a from SimpleTree import SimpleTreeMaker __allow_access_to_unprotected_subobjects__ = 1 __roles__ = None from ZTUtils.Zope import Batch, TreeMaker, SimpleTreeMaker, LazyFilter from ZTUtils.Zope import url_query, make_query, make_hidden_input Zope-2.8.7-final/lib/python/ZTUtils/Tree.py0000664000076400007640000002447110435500330017457 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tree manipulation classes $Id: Tree.py 24764 2004-05-17 06:13:48Z philikon $ """ from Acquisition import Explicit from ComputedAttribute import ComputedAttribute from types import ListType, TupleType class TreeNode(Explicit): __allow_access_to_unprotected_subobjects__ = 1 state = 0 # leaf height = 1 size = 1 def __init__(self): self._child_list = [] def _add_child(self, child): 'Add a child which already has all of its children.' self._child_list.append(child) self.height = max(self.height, child.height + 1) self.size = self.size + child.size def flat(self): 'Return a flattened preorder list of tree nodes' items = [] self.walk(items.append) return items def walk(self, f, data=None): 'Preorder walk this tree, passing each node to a function' if data is None: f(self) else: f(self, data) for child in self._child_list: child.__of__(self).walk(f, data) def _depth(self): return self.aq_parent.depth + 1 depth = ComputedAttribute(_depth, 1) def __getitem__(self, index): return self._child_list[index].__of__(self) def __len__(self): return len(self._child_list) _marker = [] class TreeMaker: '''Class for mapping a hierachy of objects into a tree of nodes.''' __allow_access_to_unprotected_subobjects__ = 1 _id = 'tpId' _values = 'tpValues' _assume_children = 0 _values_filter = None _values_function = None _state_function = None _expand_root = 1 _cached_children = None def setChildAccess(self, attrname=_marker, filter=_marker, function=_marker): '''Set the criteria for fetching child nodes. Child nodes can be accessed through either an attribute name or callback function. Children fetched by attribute name can be filtered through a callback function. ''' if function is _marker: self._values_function = None if attrname is not _marker: self._values = str(attrname) if filter is not _marker: self._values_filter = filter else: self._values_function = function def setIdAttr(self, id): """Set the attribute or method name called to get a unique Id. The id attribute or method is used to get a unique id for every node in the tree, so that the state of the tree can be encoded as a string using Tree.encodeExpansion(). The returned id should be unique and stable across Zope requests. If the attribute or method isn't found on an object, either the objects persistence Id or the result of id() on the object is used instead. """ self._id = id def setExpandRoot(self, expand): """Set wether or not to expand the root node by default. When no expanded flag or mapping is passed to .tree(), assume the root node is expanded, and leave all subnodes closed. The default is to expand the root node. """ self._expand_root = expand and True or False def setAssumeChildren(self, assume): """Set wether or not to assume nodes have children. When a node is not expanded, when assume children is set, don't determine if it is a leaf node, but assume it can be opened. Use this when determining the children for a node is expensive. The default is to not assume there are children. """ self._assume_children = assume and True or False def setStateFunction(self, function): """Set the expansion state function. This function will be called to determine if a node should be open or collapsed, or should be treated as a leaf node. The function is passed the current object, and the intended state for that object. It should return the actual state the object should be in. State is encoded as an integer, meaning: -1: Node closed. Children will not be processed. 0: Leaf node, cannot be opened or closed, no children are processed. 1: Node opened. Children will be processed as part of the tree. """ self._state_function = function def tree(self, root, expanded=None, subtree=0): '''Create a tree from root, with specified nodes expanded. "expanded" must be false, true, or a mapping. Each key of the mapping is the id of a top-level expanded node, and each value is the "expanded" value for the children of that node. ''' node = self.node(root) child_exp = expanded if not simple_type(expanded): # Assume a mapping expanded = expanded.has_key(node.id) child_exp = child_exp.get(node.id) expanded = expanded or (not subtree and self._expand_root) # Set state to 0 (leaf), 1 (opened), or -1 (closed) state = self.hasChildren(root) and (expanded or -1) if self._state_function is not None: state = self._state_function(node.object, state) node.state = state if state > 0: for child in self.getChildren(root): node._add_child(self.tree(child, child_exp, 1)) if not subtree: node.depth = 0 return node def node(self, object): node = TreeNode() node.object = object node.id = b2a(self.getId(object)) return node def getId(self, object): id_attr = self._id if hasattr(object, id_attr): obid = getattr(object, id_attr) if not simple_type(obid): obid = obid() return obid if hasattr(object, '_p_oid'): return str(object._p_oid) return id(object) def hasChildren(self, object): if self._assume_children: return 1 # Cache generated children for a subsequent call to getChildren self._cached_children = (object, self.getChildren(object)) return not not self._cached_children[1] def getChildren(self, object): # Check and clear cache first if self._cached_children is not None: ob, children = self._cached_children self._cached_children = None if ob is object: return children if self._values_function is not None: return self._values_function(object) children = getattr(object, self._values) if not (isinstance(children, ListType) or isinstance(children, TupleType)): # Assume callable; result not useful anyway otherwise. children = children() return self.filterChildren(children) def filterChildren(self, children): if self._values_filter: return self._values_filter(children) return children def simple_type(ob, is_simple={type(''):1, type(0):1, type(0.0):1, type(0L):1, type(None):1 }.has_key): return is_simple(type(ob)) import base64 from string import translate, maketrans import zlib a2u_map = maketrans('+/=', '-._') u2a_map = maketrans('-._', '+/=') def b2a(s): '''Encode a value as a cookie- and url-safe string. Encoded string use only alpahnumeric characters, and "._-". ''' return translate(base64.encodestring(str(s)), a2u_map).replace('\n', '') def a2b(s): '''Decode a b2a-encoded string.''' return base64.decodestring(translate(s, u2a_map)) def encodeExpansion(nodes, compress=1): '''Encode the expanded node ids of a tree into a string. Accepts a list of nodes, such as that produced by root.flat(). Marks each expanded node with an expansion_number attribute. Since node ids are encoded, the resulting string is safe for use in cookies and URLs. ''' steps = [] last_depth = -1 n = 0 for node in nodes: if node.state <=0: continue dd = last_depth - node.depth + 1 last_depth = node.depth if dd > 0: steps.append('_' * dd) steps.append(node.id) node.expansion_number = n n = n + 1 result = ':'.join(steps) if compress and len(result) > 2: zresult = ':' + b2a(zlib.compress(result, 9)) if len(zresult) < len(result): result = zresult return result def decodeExpansion(s, nth=None, maxsize=8192): '''Decode an expanded node map from a string. If nth is an integer, also return the (map, key) pair for the nth entry. ''' if len(s) > maxsize: # Set limit to avoid DoS attacks. raise ValueError('Encoded node map too large') if s[0] == ':': # Compressed state dec = zlib.decompressobj() s = dec.decompress(a2b(s[1:]), maxsize) if dec.unconsumed_tail: raise ValueError('Encoded node map too large') del dec map = m = {} mstack = [] pop = 0 nth_pair = None if nth is not None: nth_pair = (None, None) for step in s.split(':'): if step.startswith('_'): pop = len(step) - 1 continue if pop < 0: mstack.append(m) m[obid] = {} m = m[obid] elif map: m[obid] = None if len(step) == 0: return map obid = step if pop > 0: m = mstack[-pop] del mstack[-pop:] pop = -1 if nth == 0: nth_pair = (m, obid) nth = None elif nth is not None: nth = nth - 1 m[obid] = None if nth == 0: return map, (m, obid) if nth_pair is not None: return map, nth_pair return map Zope-2.8.7-final/lib/python/ZTUtils/Iterator.py0000664000076400007640000001126610435500330020347 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Iterator class Unlike the builtin iterators of Python 2.2+, these classes are designed to maintain information about the state of an iteration. The Iterator() function accepts either a sequence or a Python iterator. The next() method fetches the next item, and returns true if it succeeds. $Id: Iterator.py 24764 2004-05-17 06:13:48Z philikon $ """ class Iterator: '''Simple Iterator class''' __allow_access_to_unprotected_subobjects__ = 1 nextIndex = 0 def __init__(self, seq): self.seq = seq if iterInner._supports(seq, self): self._inner = iterInner self._prep_next = iterInner.prep_next return raise TypeError, "Iterator does not support %s" % `seq` def __getattr__(self, name): try: inner = getattr(self._inner, 'it_' + name) except AttributeError: raise AttributeError, name return inner(self) def next(self): if not (hasattr(self, '_next') or self._prep_next(self)): return 0 self.index = i = self.nextIndex self.nextIndex = i+1 self._advance(self) return 1 def _advance(self, it): self.item = self._next del self._next del self.end self._advance = self._inner.advance self.start = 1 def number(self): return self.nextIndex def even(self): return not self.index % 2 def odd(self): return self.index % 2 def parity(self): if self.index % 2: return 'odd' return 'even' def letter(self, base=ord('a'), radix=26): index = self.index s = '' while 1: index, off = divmod(index, radix) s = chr(base + off) + s if not index: return s def Letter(self): return self.letter(base=ord('A')) def Roman(self, rnvalues=( (1000,'M'),(900,'CM'),(500,'D'),(400,'CD'), (100,'C'),(90,'XC'),(50,'L'),(40,'XL'), (10,'X'),(9,'IX'),(5,'V'),(4,'IV'),(1,'I')) ): n = self.index + 1 s = '' for v, r in rnvalues: rct, n = divmod(n, v) s = s + r * rct return s def roman(self): return self.Roman().lower() def first(self, name=None): if self.start: return 1 return not self.same_part(name, self._last, self.item) def last(self, name=None): if self.end: return 1 return not self.same_part(name, self.item, self._next) def length(self): return len(self.seq) def same_part(self, name, ob1, ob2): if name is None: return ob1 == ob2 no = [] return getattr(ob1, name, no) == getattr(ob2, name, no) is not no def __iter__(self): return IterIter(self) class InnerBase: '''Base Inner class for Iterators''' # Prep sets up ._next and .end def prep_next(self, it): it.next = self.no_next it.end = 1 return 0 # Advance knocks them down def advance(self, it): it._last = it.item it.item = it._next del it._next del it.end it.start = 0 def no_next(self, it): return 0 def it_end(self, it): if hasattr(it, '_next'): return 0 return not self.prep_next(it) class IterInner(InnerBase): '''Iterator inner class for objects with Python iterators''' def _supports(self, ob, it): try: it.iter = iter(ob) return 1 except: return 0 def prep_next(self, it): try: it._next = it.iter.next() except StopIteration: it._prep_next = self.no_next it.end = 1 return 0 it.end = 0 return 1 class IterIter: def __init__(self, it): self.it = it self.skip = it.nextIndex > 0 and not it.end def next(self): it = self.it if self.skip: self.skip = 0 return it.item if it.next(): return it.item raise StopIteration iterInner = IterInner() Zope-2.8.7-final/lib/python/ZTUtils/SimpleTree.py0000664000076400007640000000401710435500330020623 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Simple Tree classes $Id: SimpleTree.py 24764 2004-05-17 06:13:48Z philikon $ """ from Tree import TreeMaker, TreeNode, b2a from cgi import escape class SimpleTreeNode(TreeNode): def branch(self): if self.state == 0: return {'link': None, 'img': '  '} if self.state < 0: setst = 'expand' exnum = self.aq_parent.expansion_number img = 'pl' else: setst = 'collapse' exnum = self.expansion_number img = 'mi' base = self.aq_acquire('baseURL') obid = self.id pre = self.aq_acquire('tree_pre') return {'link': '?%s-setstate=%s,%s,%s#%s' % \ (pre, setst[0], exnum, obid, obid), 'img': '%s' % \ (escape(base, 1), img, setst)} class SimpleTreeMaker(TreeMaker): '''Generate Simple Trees''' def __init__(self, tree_pre="tree"): self.tree_pre = tree_pre def node(self, object): node = SimpleTreeNode() node.object = object node.id = b2a(self.getId(object)) return node def tree(self, root, expanded=None, subtree=0): node = TreeMaker.tree(self, root, expanded, subtree) if not subtree: node.tree_pre = self.tree_pre node.baseURL = root.REQUEST['BASEPATH1'] return node Zope-2.8.7-final/lib/python/ZTUtils/Zope.py0000664000076400007640000002462410435500330017475 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Zope-specific versions of ZTUTils classes $Id: Zope.py 37986 2005-08-18 11:03:22Z andreasjung $ """ import sys, cgi, urllib, cgi from Tree import encodeExpansion, decodeExpansion, TreeMaker from SimpleTree import SimpleTreeMaker from Batch import Batch from Products.ZCatalog.Lazy import Lazy from AccessControl import getSecurityManager from DateTime import DateTime try: from AccessControl.ZopeGuards import guarded_getitem except ImportError: Unauthorized = 'Unauthorized' def guarded_getitem(object, index): v = object[index] if getSecurityManager().validate(object, object, index, v): return v raise Unauthorized, 'unauthorized access to element %s' % `i` else: from AccessControl import Unauthorized class LazyFilter(Lazy): # A LazyFilter that checks with the security policy def __init__(self, seq, test=None, skip=None): self._seq=seq self._data=[] self._eindex=-1 self._test=test if not (skip is None or str(skip) == skip): raise TypeError, 'Skip must be None or a string' self._skip = skip def __getitem__(self,index): data=self._data try: s=self._seq except AttributeError: return data[index] i=index if i < 0: i=len(self)+i if i < 0: raise IndexError, index ind=len(data) if i < ind: return data[i] ind=ind-1 test=self._test e=self._eindex skip = self._skip while i > ind: e = e + 1 try: try: v = guarded_getitem(s, e) except Unauthorized, vv: if skip is None: self._eindex = e msg = '(item %s): %s' % (index, vv) raise Unauthorized, msg, sys.exc_info()[2] skip_this = 1 else: skip_this = 0 except IndexError: del self._test del self._seq del self._eindex raise IndexError, index if skip_this: continue if skip and not getSecurityManager().checkPermission(skip, v): continue if test is None or test(v): data.append(v) ind=ind+1 self._eindex=e return data[i] class TreeSkipMixin: '''Mixin class to make trees test security, and allow skipping of unauthorized objects. ''' skip = None def setSkip(self, skip): self.skip = skip return self def getChildren(self, object): return LazyFilter(self._getChildren(object), skip=self.skip) def filterChildren(self, children): if self._values_filter: return self._values_filter(LazyFilter(children, skip=self.skip)) return children class TreeMaker(TreeSkipMixin, TreeMaker): _getChildren = TreeMaker.getChildren class SimpleTreeMaker(TreeSkipMixin, SimpleTreeMaker): _getChildren = SimpleTreeMaker.getChildren def cookieTree(self, root_object, default_state=None): '''Make a tree with state stored in a cookie.''' tree_pre = self.tree_pre state_name = '%s-state' % tree_pre set_name = '%s-setstate' % tree_pre req = root_object.REQUEST state = req.get(state_name) if state: setst = req.form.get(set_name) if setst: st, pn, expid = setst.split(',') state, (m, obid) = decodeExpansion(state, int(pn)) if m is None: pass elif st == 'e': if m[obid] is None: m[obid] = {expid: None} else: m[obid][expid] = None elif st == 'c' and m is not state and obid==expid: del m[obid] else: state = decodeExpansion(state) else: state = default_state tree = self.tree(root_object, state) rows = tree.flat() req.RESPONSE.setCookie(state_name, encodeExpansion(rows)) return tree, rows # Make the Batch class test security, and let it skip unauthorized. _Batch = Batch class Batch(Batch): def __init__(self, sequence, size, start=0, end=0, orphan=0, overlap=0, skip_unauthorized=None): sequence = LazyFilter(sequence, skip=skip_unauthorized) _Batch.__init__(self, sequence, size, start, end, orphan, overlap) # These functions are meant to be used together in templates that use # trees or batches. For example, given a batch with a 'bstart' query # argument, you would use "url_query(request, omit='bstart')" to get # the base for the batching links, then append # "make_query(bstart=batch.previous.first)" to one and # "make_query(bstart=batch.end)" to the other. def make_query(*args, **kwargs): '''Construct a URL query string, with marshalling markup. If there are positional arguments, they must be dictionaries. They are combined with the dictionary of keyword arguments to form a dictionary of query names and values. Query names (the keys) must be strings. Values may be strings, integers, floats, or DateTimes, and they may also be lists or namespaces containing these types. Names and string values should not be URL-quoted. All arguments are marshalled with complex_marshal(). ''' d = {} for arg in args: d.update(arg) d.update(kwargs) uq = urllib.quote qlist = complex_marshal(d.items()) for i in range(len(qlist)): k, m, v = qlist[i] qlist[i] = '%s%s=%s' % (uq(k), m, uq(str(v))) return '&'.join(qlist) def make_hidden_input(*args, **kwargs): '''Construct a set of hidden input elements, with marshalling markup. If there are positional arguments, they must be dictionaries. They are combined with the dictionary of keyword arguments to form a dictionary of query names and values. Query names (the keys) must be strings. Values may be strings, integers, floats, or DateTimes, and they may also be lists or namespaces containing these types. All arguments are marshalled with complex_marshal(). ''' d = {} for arg in args: d.update(arg) d.update(kwargs) hq = cgi.escape qlist = complex_marshal(d.items()) for i in range(len(qlist)): k, m, v = qlist[i] qlist[i] = ('' % (hq(k), m, hq(str(v)))) return '\n'.join(qlist) def complex_marshal(pairs): '''Add request marshalling information to a list of name-value pairs. Names must be strings. Values may be strings, integers, floats, or DateTimes, and they may also be lists or namespaces containing these types. The list is edited in place so that each (name, value) pair becomes a (name, marshal, value) triple. The middle value is the request marshalling string. Integer, float, and DateTime values will have ":int", ":float", or ":date" as their marshal string. Lists will be flattened, and the elements given ":list" in addition to their simple marshal string. Dictionaries will be flattened and marshalled using ":record". ''' i = len(pairs) while i > 0: i = i - 1 k, v = pairs[i] m = '' sublist = None if isinstance(v, str): pass elif hasattr(v, 'items'): sublist = [] for sk, sv in v.items(): if isinstance(sv, list): for ssv in sv: sm = simple_marshal(ssv) sublist.append(('%s.%s' % (k, sk), '%s:list:record' % sm, ssv)) else: sm = simple_marshal(sv) sublist.append(('%s.%s' % (k, sk), '%s:record' % sm, sv)) elif isinstance(v, list): sublist = [] for sv in v: sm = simple_marshal(sv) sublist.append((k, '%s:list' % sm, sv)) else: m = simple_marshal(v) if sublist is None: pairs[i] = (k, m, v) else: pairs[i:i + 1] = sublist return pairs def simple_marshal(v): if isinstance(v, str): return '' if isinstance(v, bool): return ':boolean' if isinstance(v, int): return ':int' if isinstance(v, float): return ':float' if isinstance(v, DateTime): return ':date' return '' def url_query(request, req_name="URL", omit=None): '''Construct a URL with a query string, using the current request. request: the request object req_name: the name, such as "URL1" or "BASEPATH1", to get from request omit: sequence of name of query arguments to omit. If a name contains a colon, it is treated literally. Otherwise, it will match each argument name that starts with the name and a period or colon. ''' base = request[req_name] qs = request.get('QUERY_STRING', '') if qs and omit: qsparts = qs.split('&') if isinstance(omit, str): omits = {omit: None} else: omits = {} for name in omit: omits[name] = None omitted = omits.has_key unq = urllib.unquote for i in range(len(qsparts)): name = unq(qsparts[i].split('=', 1)[0]) if omitted(name): qsparts[i] = '' name = name.split(':', 1)[0] if omitted(name): qsparts[i] = '' name = name.split('.', 1)[0] if omitted(name): qsparts[i] = '' qs = '&'.join(filter(None, qsparts)) # We alway append '?' since arguments will be appended to the URL return '%s?%s' % (base, qs) Zope-2.8.7-final/lib/python/ZTUtils/tests/0000775000076400007640000000000010435610645017353 5ustar ajungajungZope-2.8.7-final/lib/python/ZTUtils/tests/__init__.py0000664000076400007640000000007710435500330021455 0ustar ajungajung# This file is needed to make this directory a Python package. Zope-2.8.7-final/lib/python/ZTUtils/tests/testZope.py0000664000076400007640000000441010435500330021526 0ustar ajungajungimport os, sys from unittest import TestCase, makeSuite, main import string import urllib from ZTUtils.Zope import make_query, complex_marshal from DateTime import DateTime class QueryTests(TestCase): def testMarshallLists(self): '''Test marshalling lists''' test_date = DateTime() list_ = [1, test_date, 'str'] result = complex_marshal([('list',list_),]) assert result == [('list', ':int:list', 1), ('list', ':date:list', test_date), ('list', ':list', 'str')] def testMarshallRecords(self): '''Test marshalling records''' test_date = DateTime() record = {'arg1': 1, 'arg2': test_date, 'arg3': 'str'} result = complex_marshal([('record',record),]) assert result == [('record.arg1', ':int:record', 1), ('record.arg2', ':date:record', test_date), ('record.arg3', ':record', 'str')] def testMarshallListsInRecords(self): '''Test marshalling lists inside of records''' test_date = DateTime() record = {'arg1': [1, test_date, 'str'], 'arg2': 1} result = complex_marshal([('record',record),]) assert result == [('record.arg1', ':int:list:record', 1), ('record.arg1', ':date:list:record', test_date), ('record.arg1', ':list:record', 'str'), ('record.arg2', ':int:record', 1)] def testMakeComplexQuery(self): '''Test that make_query returns sane results''' test_date = DateTime() quote_date = urllib.quote(str(test_date)) record = {'arg1': [1, test_date, 'str'], 'arg2': 1} list_ = [1, test_date, 'str'] date = test_date int_ = 1 str_ = 'str' query = make_query(date=test_date, integer=int_, listing=list_, record=record, string=str_) assert query == 'date:date=%s&integer:int=1&listing:int:list=1&listing:date:list=%s&listing:list=str&string=str&record.arg1:int:list:record=1&record.arg1:date:list:record=%s&record.arg1:list:record=str&record.arg2:int:record=1'%(quote_date,quote_date,quote_date) def test_suite(): return makeSuite(QueryTests) if __name__=='__main__': main() Zope-2.8.7-final/lib/python/ZTUtils/tests/testTree.py0000664000076400007640000001671510435500330021523 0ustar ajungajungimport unittest from ZTUtils import Tree class Item: children = () id = '' def __init__(self, id, children=()): self.id = id self.children = children def tpId(self): return self.id def tpValues(self): return self.children class TreeTests(unittest.TestCase): def setUp(self): self.tm = Tree.TreeMaker() self.root = Item('a', ( Item('b', ( Item('d'), Item('e'))), Item('c', ( Item('f', ( Item('h'), Item('i'))), Item('g'))))) self.items = { 'a': self.root, 'b': self.root.children[0], 'c': self.root.children[1], 'd': self.root.children[0].children[0], 'e': self.root.children[0].children[1], 'f': self.root.children[1].children[0], 'g': self.root.children[1].children[1], 'h': self.root.children[1].children[0].children[0], 'i': self.root.children[1].children[0].children[1], } self.expansionmap = {Tree.b2a('a'): {Tree.b2a('c'): None}} def testBaseTree(self): treeroot = self.tm.tree(self.root) self.assertEqual(len(treeroot), 2) self.assertEqual(treeroot.size, 3) self.assertEqual(treeroot.height, 2) self.assertEqual(treeroot.depth, 0) self.assertEqual(treeroot.state, 1) self.assert_(treeroot.object is self.root) i = 'b' for subnode in treeroot: self.assertEqual(len(subnode), 0) self.assertEqual(subnode.size, 1) self.assertEqual(subnode.height, 1) self.assertEqual(subnode.depth, 1) self.assertEqual(subnode.state, -1) self.assert_(subnode.object is self.items[i]) i = chr(ord(i) + 1) expected_set = [self.items['a'], self.items['b'], self.items['c']] set = treeroot.flat() self.assertEqual(len(set), treeroot.size) self.assertEqual([s.object for s in set], expected_set) set = [] def collect(node, set=set): set.append(node.object) treeroot.walk(collect) self.assertEqual(len(set), treeroot.size) self.assertEqual(set, expected_set) def testExpandedTree(self): treeroot = self.tm.tree(self.root, 1) self.assertEqual(len(treeroot), 2) self.assertEqual(treeroot.size, len(self.items)) self.assertEqual(treeroot.height, 4) self.assertEqual(treeroot.depth, 0) self.assertEqual(treeroot.state, 1) self.assert_(treeroot.object is self.root) items = self.items expected_set = [items['a'], items['b'], items['d'], items['e'], items['c'], items['f'], items['h'], items['i'], items['g']] set = treeroot.flat() self.assertEqual(len(set), treeroot.size) self.assertEqual([s.object for s in set], expected_set) set = [] def collect(node, set=set): set.append(node.object) treeroot.walk(collect) self.assertEqual(len(set), treeroot.size) self.assertEqual(set, expected_set) leaves = ('d', 'e', 'g', 'h', 'i') for node in treeroot.flat(): if node.object.id in leaves: self.assertEqual(node.state, 0) else: self.assertEqual(node.state, 1) self.assertEqual(node.size, len(node.flat())) def testExpansionMap(self): treeroot = self.tm.tree(self.root, self.expansionmap) self.assertEqual(treeroot.size, 5) self.assertEqual(treeroot.state, 1) self.assertEqual(treeroot[0].state, -1) self.assertEqual(treeroot[1].state, 1) self.assertEqual(treeroot[1][0].state, -1) self.assertEqual(treeroot[1][1].state, 0) def testAssumeChildren(self): self.tm.setAssumeChildren(True) treeroot = self.tm.tree(self.root, self.expansionmap) self.assertEqual(treeroot[1][1].state, -1) def testNoExpandRoot(self): self.tm.setExpandRoot(False) treeroot = self.tm.tree(self.root) self.assertEqual(treeroot.state, -1) self.assertEqual(len(treeroot), 0) def testIdAttribute(self): treeroot = self.tm.tree(self.root) self.tm.setIdAttr('id') treeroot2 = self.tm.tree(self.root) self.assertEqual(treeroot.id, treeroot2.id) def testChildrenAttribute(self): self.tm.setChildAccess(attrname='children') treeroot = self.tm.tree(self.root) self.assertEqual(len(treeroot), 2) def testChildrenFilter(self): def filter(children): return [c for c in children if c.id in ('b', 'd')] self.tm.setChildAccess(filter=filter) treeroot = self.tm.tree(self.root, 1) self.assertEqual(treeroot.size, 3) self.assertEqual(len(treeroot), 1) self.assertEqual(len(treeroot[0]), 1) expected_set = [self.items['a'], self.items['b'], self.items['d']] set = [] def collect(node, set=set): set.append(node.object) treeroot.walk(collect) self.assertEqual(set, expected_set) def testChildrenFunction(self): def childrenFunction(object): return object.children self.tm.setChildAccess(function=childrenFunction) treeroot = self.tm.tree(self.root) self.assertEqual(len(treeroot), 2) def testStateFunction(self): def stateFunction(object, state): if object.id == 'b': return 1 if object.id == 'd': return -1 return state self.tm.setStateFunction(stateFunction) treeroot = self.tm.tree(self.root) self.assertEqual(treeroot.size, 5) self.assertEqual(treeroot.state, 1) self.assertEqual(treeroot[0].state, 1) self.assertEqual(treeroot[0][0].state, -1) self.assertEqual(treeroot[0][1].state, 0) self.assertEqual(treeroot[1].state, -1) def testEncodeDecode(self): treeroot1 = self.tm.tree(self.root, self.expansionmap) encoded = Tree.encodeExpansion(treeroot1.flat()) self.failIf(encoded.find('\n') != -1) decodedmap = Tree.decodeExpansion(encoded) treeroot2 = self.tm.tree(self.root, decodedmap) self.assertEqual(treeroot1.size, treeroot2.size) self.assertEqual(len(treeroot1), len(treeroot2)) def testEncodedExpansionIdWithDot(self): # Regression test for Collector issue #603 # An encoded node ID with a first character with the first 6 bits set. item = Item('\xfcberbug!', (Item('b'),)) # 'uberbug!' with u-umlaut. treeroot1 = self.tm.tree(item) encoded = Tree.encodeExpansion(treeroot1.flat()) decodedmap = Tree.decodeExpansion(encoded) treeroot2 = self.tm.tree(item, decodedmap) self.assertEqual(treeroot1.size, treeroot2.size) self.assertEqual(len(treeroot1), len(treeroot2)) def testDecodeInputSizeLimit(self): self.assertRaises(ValueError, Tree.decodeExpansion, 'x' * 10000) def testDecodeDecompressedSizeLimit(self): import zlib from ZTUtils.Tree import b2a, a2b, encodeExpansion, decodeExpansion big = b2a(zlib.compress('x' * (1024*1100))) self.assert_(len(big) < 8192) # Must be under the input size limit self.assertRaises(ValueError, Tree.decodeExpansion, ':' + big) def test_suite(): return unittest.makeSuite(TreeTests) if __name__=='__main__': unittest.main() Zope-2.8.7-final/lib/python/ZTUtils/tests/testBatch.py0000664000076400007640000000326610435500330021642 0ustar ajungajungimport os, sys from unittest import TestCase, makeSuite, main import string from ZTUtils import Batch class BatchTests(TestCase): def testEmpty(self): '''Test empty Batch''' b = Batch([], 5) assert b.previous is None assert b.next is None assert len(b) == b.start == b.end == 0, (len(b), b.start, b.end) def testSingle(self): '''Test single Batch''' for bsize in range(1, 6): seq = range(bsize) b = Batch(seq, 5) assert b.previous is None assert b.next is None assert b.start == 1, b.start assert len(b) == b.end == bsize assert b.sequence_length == len(seq) for i in seq: assert b[i] == i, (b[i], i) neg = -1 - i assert b[neg] == (bsize + neg), (b[neg], (bsize + neg)) def testOrphan(self): '''Test orphan collection''' for bsize in (6, 7): b = Batch(range(bsize), 5, orphan=3) assert b.next is None assert len(b) == bsize assert b[bsize - 1] == bsize - 1 assert b.sequence_length == bsize b = Batch(range(8), 5) assert len(b) == 5 assert b.sequence_length == 8 assert len(b.next) == 3 def testLengthEqualsSizePlusOrphans(self): '''Test limit case where batch length is equal to size + orphans''' for bsize, length in ((12,11), (13,12), (14,13), (15,10)): b = Batch(range(bsize), size=10, start=1, end=0, orphan=3, overlap=0) assert length == b.length def test_suite(): return makeSuite(BatchTests) if __name__=='__main__': main() Zope-2.8.7-final/lib/python/ZTUtils/tests/run.py0000775000076400007640000000063610435500330020526 0ustar ajungajung#!/usr/bin/env python """Run all tests.""" import unittest import glob import os import sys def test_suite(): suite = unittest.TestSuite() for mname in glob.glob(os.path.join(sys.path[0], 'test*.py')): mname = os.path.split(mname)[1][:-3] m = __import__(mname) suite.addTest(m.test_suite()) return suite if __name__ == "__main__": unittest.main(defaultTest='test_suite') Zope-2.8.7-final/lib/python/ZTUtils/tests/testIterator.py0000664000076400007640000001124110435500330022402 0ustar ajungajungfrom __future__ import generators import os, sys, unittest from ZTUtils import Iterator try: iter do_piter_test = 1 except NameError: do_piter_test = 0 class itemIterator: 'Ignore the __getitem__ argument in order to catch non-monotonic access.' def __init__(self, n): self.n = n self.i = 0 def __getitem__(self, i): if self.i >= self.n: raise IndexError i = self.i self.i = self.i + 1 return i class genIterator: 'Generator-based iteration' def __init__(self, n): self.n = n def __iter__(self): for i in range(self.n): yield i class IteratorTests(unittest.TestCase): def testIterator0(self): it = Iterator(()) assert not it.next(), "Empty iterator" def testIterator1(self): it = Iterator((1,)) assert it.next() and not it.next(), "Single-element iterator" def testIteratorMany(self): it = Iterator('text') for c in 'text': assert it.next(), "Multi-element iterator" assert not it.next(), "Multi-element iterator" def testStart(self): for size in range(4): it = Iterator(range(size+1)) it.next() assert it.start, "Start true on element 1 of %s" % (size + 1) el = 1 while it.next(): el = el + 1 assert not it.start, ( "Start false on element %s of %s" % (el, size+1)) def testEnd(self): for size in range(4): size = size + 1 it = Iterator(range(size)) el = 0 while it.next(): el = el + 1 if el == size: assert it.end, "End true on element %s" % size else: assert not it.end, ( "End false on element %s of %s" % (el, size)) def assertRangeMatch(self, ob, n): it = Iterator(ob) for el in range(n): assert it.next(), "Iterator stopped too soon" assert it.index == el, "Incorrect index" assert it.number() == el + 1, "Incorrect number" assert it.item == el, "Incorrect item" def testIndex(self): self.assertRangeMatch(range(5), 5) self.assertRangeMatch((0,1,2,3,4), 5) self.assertRangeMatch({0:0, 1:1, 2:2, 3:3, 4:4}, 5) self.assertRangeMatch(itemIterator(5), 5) self.assertRangeMatch(genIterator(5), 5) def testFirstLast(self): it = Iterator([1]) it.next() assert it.first() == it.last() == 1, "Bad first/last on singleton" four = range(4) for a in 2,3: for b in four: for c in four: s = 'a' * a + 'b' * b + 'c' * c it = Iterator(s) it.next() assert it.first(), "First element not first()" last = s[0] lastlast = it.last() while it.next(): assert ((it.item != last) == it.first()), ( "first() error") assert ((it.item != last) == lastlast), ( "last() error" % (it.item, last, lastlast)) last = it.item lastlast = it.last() assert lastlast, "Last element not last()" if do_piter_test: def testIterOfIter(self): for i in range(4): r = range(i) it1 = Iterator(r) it2 = Iterator(iter(r)) while it1.next() and it2.next(): assert it1.item == it2.item, "Item mismatch with iter()" assert it1.index == it2.index, ( "Index mismatch with iter()") assert not (it1.next() or it2.next()), ( "Length mismatch with iter()") def testIterIter(self): wo_iter = map(lambda x:(x, x), range(4)) for i in range(4): r = range(i) w_iter = [] it = Iterator(r) for x in it: w_iter.append((x, it.index)) assert w_iter == wo_iter[:i], ( "for-loop failure on full iterator") it = Iterator(range(4)) it.next(); it.next(); it.next() w_iter = [] for x in it: w_iter.append((x, it.index)) assert w_iter == wo_iter[2:], "for-loop failure on half iteration" def test_suite(): return unittest.makeSuite(IteratorTests) if __name__=='__main__': main() Zope-2.8.7-final/lib/python/ZTUtils/Batch.py0000664000076400007640000000766610435500330017610 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Batch class, for iterating over a sequence in batches $Id: Batch.py 25080 2004-05-28 13:40:38Z andreasjung $ """ from ExtensionClass import Base class LazyPrevBatch(Base): def __of__(self, parent): return Batch(parent._sequence, parent._size, parent.first - parent._size + parent.overlap, 0, parent.orphan, parent.overlap) class LazyNextBatch(Base): def __of__(self, parent): try: parent._sequence[parent.end] except IndexError: return None return Batch(parent._sequence, parent._size, parent.end - parent.overlap, 0, parent.orphan, parent.overlap) class LazySequenceLength(Base): def __of__(self, parent): parent.sequence_length = l = len(parent._sequence) return l class Batch(Base): """Create a sequence batch""" __allow_access_to_unprotected_subobjects__ = 1 previous = LazyPrevBatch() next = LazyNextBatch() sequence_length = LazySequenceLength() def __init__(self, sequence, size, start=0, end=0, orphan=0, overlap=0): '''Encapsulate "sequence" in batches of "size". Arguments: "start" and "end" are 0-based indexes into the sequence. If the next batch would contain no more than "orphan" elements, it is combined with the current batch. "overlap" is the number of elements shared by adjacent batches. If "size" is not specified, it is computed from "start" and "end". Failing that, it is 7. Attributes: Note that the "start" attribute, unlike the argument, is a 1-based index (I know, lame). "first" is the 0-based index. "length" is the actual number of elements in the batch. "sequence_length" is the length of the original, unbatched, sequence ''' start = start + 1 start,end,sz = opt(start,end,size,orphan,sequence) self._sequence = sequence self.size = sz self._size = size self.start = start self.end = end self.orphan = orphan self.overlap = overlap self.first = max(start - 1, 0) self.length = self.end - self.first if self.first == 0: self.previous = None def __getitem__(self, index): if index < 0: if index + self.end < self.first: raise IndexError, index return self._sequence[index + self.end] if index >= self.length: raise IndexError, index return self._sequence[index+self.first] def __len__(self): return self.length def opt(start,end,size,orphan,sequence): if size < 1: if start > 0 and end > 0 and end >= start: size=end+1-start else: size=7 if start > 0: try: sequence[start-1] except IndexError: start=len(sequence) if end > 0: if end < start: end=start else: end=start+size-1 try: sequence[end+orphan] except IndexError: end=len(sequence) elif end > 0: try: sequence[end-1] except IndexError: end=len(sequence) start=end+1-size if start - 1 < orphan: start=1 else: start=1 end=start+size-1 try: sequence[end+orphan-1] except IndexError: end=len(sequence) return start,end,size Zope-2.8.7-final/lib/python/ZTUtils/HISTORY.txt0000664000076400007640000000364310435500330020106 0ustar ajungajungZTUtils history This file contains change information for previous versions of ZTUtils. Change information for the current release can be found in the file CHANGES.txt. Version 1.1.3 Brown-bag bugfix release. Version 1.1.2 Bugs Fixed - Orphans defaulted to 3, which was confusing and out of sync with DTML-In. - Orphan batches were broken. Version 1.1.1 Bugs Fixed - Python 1.5.2-incompatible changes crept in. Version 1.1.0 Features Added - TreeMakers have a setChildAccess() method that you can use to control tree construction. Child nodes can be accessed through either an attribute name or callback function. Children fetched by attribute name can be filtered through a callback function. - A new LazyFilter class allows you to filter a sequence using Zope security and an optional filter callback function. The security and filter tests are lazy, meaning they are performed as late as possible. The optional 'skip' argument determines the reaction when access to a sequence element is refused by the Zope security policy. The default (None) is to raise the 'Unauthorized' exception. If a string is passed, such elements are skipped. If the string is non-empty, it is treated as a permission name, and the element is skipped if the user doesn't have that permission on the element. - The Zope versions of TreeMaker, SimpleTreeMaker, and Batch now use LazyFilter. The TreeMakers have a setSkip() method that can be used to set the 'skip' value. Batch has an optional 'skip_unauthorized' argument that is passed to LazyFilter as 'skip'. - Utility functions make_query(), url_query(), and make_hidden_input() have been added. Zope-2.8.7-final/lib/python/HelpSys/0000775000076400007640000000000010435610645016222 5ustar ajungajungZope-2.8.7-final/lib/python/HelpSys/HelpSys.py0000664000076400007640000002320510435500317020157 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import Acquisition from OFS.SimpleItem import Item from OFS.ObjectManager import ObjectManager from Globals import Persistent, DTMLFile, HTML from Products.ZCatalog.ZCatalog import ZCatalog from Products.ZCatalog.Lazy import LazyCat from cgi import escape import Products import HelpTopic import Globals class HelpSys(Acquisition.Implicit, ObjectManager, Item, Persistent): """ Zope Help System Provides browsing and searching of Zope Product Help. """ meta_type='Help System' manage_options=( {'label' : 'Contents', 'action' : 'menu'}, {'label' : 'Search', 'action' : 'search'}, ) __ac_permissions__=( ('View', ('__call__', 'searchResults', 'HelpButton', '', 'index_html', 'menu', 'search', 'results', 'main', 'helpLink')), ('Access contents information', ('helpValues',)), ) def __init__(self, id='HelpSys'): self.id=id def helpValues(self, spec=None): "ProductHelp objects of all Products that have help" hv=[] for product in self.Control_Panel.Products.objectValues(): productHelp=product.getProductHelp() # only list products that actually have help if productHelp.helpValues(): hv.append(productHelp) return hv # Seaching does an aggregated search of all ProductHelp # objects. Only Help Topics for which the user has permissions # are returned. def __call__(self, REQUEST=None, **kw): "Searchable interface" if REQUEST is not None: perms=[] user=REQUEST.AUTHENTICATED_USER for p in self.ac_inherited_permissions(): if user.has_permission(p[0], self): perms.append(p[0]) REQUEST.set('permissions',perms) results=[] for ph in self.helpValues(): results.append(apply(getattr(ph, '__call__'), (REQUEST,) , kw)) return LazyCat(results) searchResults=__call__ index_html=DTMLFile('dtml/frame', globals()) menu=DTMLFile('dtml/menu', globals()) search=DTMLFile('dtml/search', globals()) results=DTMLFile('dtml/results', globals()) main=HTML("""""") standard_html_header=DTMLFile('dtml/menu_header', globals()) standard_html_footer=DTMLFile('dtml/menu_footer', globals()) button=DTMLFile('dtml/button', globals()) def HelpButton(self, topic, product): """ Insert a help button linked to a help topic. """ return self.button(self, self.REQUEST, product=product, topic=topic) helpURL=DTMLFile('dtml/helpURL',globals()) def helpLink(self, product='OFSP', topic='ObjectManager_Contents.stx'): # Generate an tag linking to a help topic. This # is a little lighter weight than the help button approach. basepath=self.REQUEST['BASEPATH1'] help_url='%s/Control_Panel/Products/%s/Help/%s' % ( basepath, product, topic ) help_url='%s?help_url=%s' % (self.absolute_url(), help_url) script="window.open('%s','zope_help','width=600,height=500," \ "menubar=yes,toolbar=yes,scrollbars=yes,resizable=yes');" \ "return false;" % escape(help_url, 1).replace("'", "\\'") h_link='Help!' % ( escape(help_url, 1), script ) return h_link def tpValues(self): """ Tree protocol - returns child nodes Aggregates Product Helps with the same title. """ helps={} for help in self.helpValues(): if helps.has_key(help.title): helps[help.title].append(help) else: helps[help.title]=[help] cols=[] for k,v in helps.items(): cols.append(TreeCollection(k,v,0)) return cols Globals.default__class_init__(HelpSys) class TreeCollection: """ A temporary wrapper for a collection of objects objects, used for help topic browsing to make a collection of objects appear as a single object. """ def __init__(self, id, objs, simple=1): self.id=self.title=id self.objs=objs self.simple=simple def tpValues(self): values=[] if self.simple: values=self.objs else: for obj in self.objs: values=values + list(obj.tpValues()) # resolve overlap ids={} for value in values: if ids.has_key(value.id): ids[value.id].append(value) else: ids[value.id]=[value] results=[] for k,v in ids.items(): if len(v)==1: results.append(v[0]) else: values=[] for topic in v: values=values + list(topic.tpValues()) results.append(TreeCollection(k, values)) results.sort(lambda x, y: cmp(x.id, y.id)) return results def tpId(self): return self.id class ProductHelp(Acquisition.Implicit, ObjectManager, Item, Persistent): """ Manages a collection of Help Topics for a given Product. Provides searching services to HelpSystem. """ meta_type='Product Help' icon='p_/ProductHelp_icon' lastRegistered=None meta_types=({'name':'Help Topic', 'action':'addTopicForm', 'permission':'Add Documents, Images, and Files'}, ) manage_options=( ObjectManager.manage_options + Item.manage_options ) __ac_permissions__=( ('Add Documents, Images, and Files', ('addTopicForm', 'addTopic')), ) def __init__(self, id='Help', title=''): self.id=id self.title=title c=self.catalog=ZCatalog('catalog') # clear catalog for index in c.indexes(): c.delIndex(index) for col in c.schema(): c.delColumn(col) c.addIndex('SearchableText', 'TextIndex') c.addIndex('categories', 'KeywordIndex') c.addIndex('permissions', 'KeywordIndex') c.addColumn('categories') c.addColumn('permissions') c.addColumn('title_or_id') c.addColumn('url') c.addColumn('id') addTopicForm=DTMLFile('dtml/addTopic', globals()) def addTopic(self, id, title, REQUEST=None): "Add a Help Topic" topic=HelpTopic.DTMLDocumentTopic( HelpTopic.default_topic_content, __name__=id) topic.title=title self._setObject(id, topic) if REQUEST is not None: return self.manage_main(self, REQUEST, manage_tabs_message='Help Topic added.') def helpValues(self, REQUEST=None): """ Lists contained Help Topics. Help Topics for which the user is not authorized are not listed. """ topics=self.objectValues('Help Topic') if REQUEST is None: return topics return filter( lambda ht, u=REQUEST.AUTHENTICATED_USER: ht.authorized(u), topics) def tpValues(self): """ Tree protocol - child nodes """ topics=[] apitopics=[] dtmltopics=[] zpttopics=[] for topic in self.objectValues('Help Topic'): if hasattr(topic,'isAPIHelpTopic') and topic.isAPIHelpTopic: apitopics.append(topic) else: try: if callable(topic.id): id=topic.id() else: id=topic.id if id[:5]=='dtml-': dtmltopics.append(topic) if (id[:5] in ('metal', 'tales') and id[5] in ('.', '-')) or \ (id[:3]=='tal' and id[3] in ('.', '-')): zpttopics.append(topic) else: topics.append(topic) except ImportError: # Don't blow up if we have references to non-existant # products laying around pass if dtmltopics: topics = topics + [TreeCollection(' DTML Reference', dtmltopics)] if apitopics: topics = topics + [TreeCollection(' API Reference', apitopics)] if zpttopics: topics = topics + [TreeCollection(' ZPT Reference', zpttopics)] return topics def all_meta_types(self): f=lambda x: x['name'] in ('Image', 'File') return filter(f, Products.meta_types) + self.meta_types def __call__(self, *args, **kw): """ Searchable interface """ return apply(self.catalog.__call__, args, kw) standard_html_header=DTMLFile('dtml/topic_header', globals()) standard_html_footer=DTMLFile('dtml/topic_footer', globals()) Globals.default__class_init__(ProductHelp) Zope-2.8.7-final/lib/python/HelpSys/HelpUtil.py0000664000076400007640000002716610435500317020330 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Help system support module""" __version__='$Revision: 1.12 $'[11:-2] import Globals, Acquisition import StructuredText.StructuredText import sys, os, re stx_class=StructuredText.StructuredText.HTML class HelpBase(Acquisition.Implicit): """ """ def __bobo_traverse__(self, REQUEST, name=None): # A sneaky trick - we cant really _have_ an index_html # because that would often hide the index_html of a # wrapped object ;( if name=='index_html': return self.hs_index return getattr(self, name) def __len__(self): return 1 class object(Acquisition.Implicit): def __init__(self, name, ob, op=None): self._name=name self._obj_=ob self._obp_=op def __getattr__(self, name): return getattr(self.__dict__['_obj_'], name) def __len__(self): return 1 def get_id(self): return id(self._obj_) def get_name(self): return self._name def get_type(self): return type(self._obj_).__name__ def get_value(self): return self._obj_ def get_docstring(self): if hasattr(self._obj_, '__doc__'): doc=self._obj_.__doc__ if not doc: doc='' return doc return '' def get_docstring_html(self): doc=self.get_docstring() if doc.find('\n\n') > -1: doc=doc.split('\n\n') if len(doc) > 1: doc[1]=doc[1].strip() doc='\n\n'.join(doc) return str(stx_class(doc)) def version(self): if hasattr(self._obj_, '__version__'): return self._obj_.__version__ tpId =get_name tpURL =get_name __str__=get_name class moduleobject(object): def get_file(self): if hasattr(self._obj_, '__file__'): return self._obj_.__file__ def get_modules(self): data=[] for name, ob in self._obj_.__dict__.items(): if is_module(ob) and _chModule(name, ob): data.append(moduleobject(name, ob, self)) return data def get_classes(self): data=[] for name, ob in self._obj_.__dict__.items(): if is_class(ob) and _chClass(name, ob): data.append(classobject(name, ob, self)) return data class classobject(object): def get_metatype(self): try: return self._obj_.meta_type except: t, v = sys.exc_info()[:2] return '%s %s' % (t, v) def get_module(self): if hasattr(self._obj_, '__module__'): module=sys.modules[self._obj_.__module__] return moduleobject(module.__name__, module) def get_file(self): return self.get_module().get_file() def get_bases(self): bases=[] if hasattr(self._obj_, '__bases__'): for base in self._obj_.__bases__: bases.append(classobject(base.__name__, base)) return bases def get_base_list(self, list=None): if list is None: list=[] list.append(self) for base in self.get_bases(): list=base.get_base_list(list) return list def get_methods(self): keys=self._obj_.__dict__.keys() dict=self._obj_.__dict__ keys.sort() methods=[] for name in keys: ob=dict[name] if is_method(ob) and _chMethod(name, ob): methods.append(methodobject(name, ob, self)) return methods def get_method_dict(self, dict=None): if dict is None: dict={} dup=dict.has_key for method in self.get_methods(): name=method.get_name() if not dup(name): dict[name]=method for base in self.get_bases(): dict=base.get_method_dict(dict) return dict def get_method_list(self): dict=self.get_method_dict() keys=dict.keys() keys.sort() list=[] for key in keys: list.append(dict[key]) del dict return list ## def obAttributes(self): ## # Return list of class attributes ## keys=self._obj_.__dict__.keys() ## dict=self._obj_.__dict__ ## keys.sort() ## attrs=[] ## for name in keys: ## ob=dict[name] ## if _isAttribute(ob) and _chAttribute(name, ob): ## attrs.append(AttributeObject(name, ob, self)) ## return attrs ## def obAttributeDict(self, dict=None): ## # Return dict of attrs in class and superclasses ## if dict is None: ## dict={} ## root=1 ## else: root=0 ## dup=dict.has_key ## for attr in self._obj_Attributes(): ## name=attr.obName() ## if not dup(name): ## dict[name]=attr ## for base in self._obj_Bases(): ## dict=base.obAttributeDict(dict) ## return dict ## def obAttributeList(self): ## # Return list of attrs in class and superclasses ## dict=self._obj_AttributeDict() ## keys=dict.keys() ## keys.sort() ## list=[] ## append=list.append ## for name in keys: ## append(dict[name]) ## del dict ## return list # needs to be tested !!! The conversion of reconvert.convert looks suspicious sig_match=re.compile(r'[\w]*\([^)]*\)').match # matches "f(arg1, arg2)" pre_match=re.compile(r'[\w]*\([^)]*\)[ -]*').match # with ' ' or '-' included class methodobject(object): def get_class(self): return self._obp_ def get_module(self): return self.get_class().get_module() def get_file(self): return self.get_module().get_file() def get_docstring(self): func=self._obj_ doc='' if hasattr(func, 'im_func'): func=func.im_func if hasattr(func, '__doc__'): doc=func.__doc__ if not doc: doc='' doc=doc.strip() if hasattr(func, 'func_code'): if hasattr(func.func_code, 'co_varnames'): return doc mo=pre_match(doc) if mo is not None: return doc[mo.end(0):] return doc def get_signaturex(self): name=self._name func=self._obj_ method=None if hasattr(func, 'im_func'): method=1 func=func.im_func # Normal functions if hasattr(func, 'func_code'): if hasattr(func.func_code, 'co_varnames'): args=map(lambda x: x, func.func_code.co_varnames[:func.func_code.co_argcount]) ndefaults=func.func_defaults ndefaults=ndefaults and len(ndefaults) or 0 if '__ick__' in args: nick=len(args)-args.index('__ick__') args=args[:-nick] ndefaults=ndefaults-nick if ndefaults > 0: args[-ndefaults]='['+args[-ndefaults] args[-1]=args[-1]+']' if method: args=args[1:] if name=='__call__': name='Call Operation' return '%s(%s)' % (name, ', '.join(args)) # Other functions - look for something that smells like # a signature at the beginning of the docstring. if hasattr(func, '__doc__'): doc=func.__doc__ if not doc: doc='' doc=doc.strip() mo=sig_match(doc) if mo is not None: return doc[:mo.end(0)] return '%s()' % name def get_signature(self): try: return self.get_signaturex() except: t, v = sys.exc_info()[:2] return '%s %s' % (t, v) ## class AttributeObject(_ob_): ## def obClass(self): ## return self.op ## def obModule(self): ## return self.obClass().obModule() ## def obFile(self): ## return self.obModule().obFile() ## class InstanceObject(_ob_): ## def obClass(self): ## # Return the class for this instance ## c=self._obj_.__class__ ## return ClassObject(c.__name__, c) ## def obClassList(self): ## # Return list of all superclasses ## return self._obj_Class().obClassList() ## def obMethods(self): ## # Return list of instance methods ## keys=self._obj_.__dict__.keys() ## dict=self._obj_.__dict__ ## keys.sort() ## methods=[] ## for name in keys: ## ob=dict[name] ## if _isMethod(ob) and _chMethod(name, ob): ## methods.append(MethodObject(name, ob)) ## return methods ## def obMethodDict(self): ## # Return dict of instance and superclass methods ## dict=self._obj_Class().obMethodDict() ## for method in self._obj_Methods(): ## dict[method.obName()]=method ## return dict ## def obMethodList(self): ## # Return list of instance and superclass methods ## dict=self._obj_MethodDict() ## keys=dict.keys() ## keys.sort() ## list=[] ## append=list.append ## for name in keys: ## append(dict[name]) ## return list ## def obAttributes(self): ## # Return list of instance attributes ## keys=self._obj_.__dict__.keys() ## dict=self._obj_.__dict__ ## keys.sort() ## attrs=[] ## for name in keys: ## ob=dict[name] ## if _isAttribute(ob) and _chAttribute(name, ob): ## attrs.append(AttributeObject(name, ob, self)) ## return attrs ## def obAttributeDict(self): ## # Return dict of instance and superclass attributes ## dict=self._obj_Class().obAttributeDict() ## for attr in self._obj_Attributes(): ## dict[attr.obName()]=attr ## return dict ## def obAttributeList(self): ## # Return list of instance and superclass attributes ## dict=self._obj_AttributeDict() ## keys=dict.keys() ## keys.sort() ## list=[] ## append=list.append ## for name in keys: ## append(dict[name]) ## return list _classtypes=(type(Globals.HTML), type(Globals.Persistent), ) _methodtypes=(type([].sort), type(Globals.default__class_init__), type(Globals.HTML.manage_edit), type(Globals.HTML.__changed__), type(Globals.MessageDialog.manage_edit), ) def is_module(ob): return type(ob)==type(sys) def is_class(ob): return type(ob) in _classtypes def is_method(ob): if type(ob) in _methodtypes or hasattr(ob, 'func_code'): return 1 return 0 def is_attribute(ob): return not is_method(ob) def _chModule(name, ob): if name[0]=='_': return 0 return 1 def _chClass(name, ob): if name[0]=='_': return 0 return 1 def _chMethod(name, ob): if name[0]=='_': return 0 return 1 def _chAttribute(name, ob): if name[0]=='_': return 0 return 1 Zope-2.8.7-final/lib/python/HelpSys/__init__.py0000664000076400007640000000133010435500317020322 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## # backward compatibility import HelpSys import sys sys.modules['HelpSys.HelpSystem']=HelpSys Zope-2.8.7-final/lib/python/HelpSys/dtml/0000775000076400007640000000000010435500317017154 5ustar ajungajungZope-2.8.7-final/lib/python/HelpSys/dtml/search.dtml0000664000076400007640000000045310435500317021305 0ustar ajungajung

Search terms

Zope-2.8.7-final/lib/python/HelpSys/dtml/topic_footer.dtml0000664000076400007640000000001710435500317022530 0ustar ajungajung Zope-2.8.7-final/lib/python/HelpSys/dtml/helpsys_menu.dtml0000664000076400007640000000176110435500317022556 0ustar ajungajung Z Object Publishing Environment Help
Z Online Help Z Online Help
click to view this item click to view this item &dtml-hs_title; Zope-2.8.7-final/lib/python/HelpSys/dtml/topic_header.dtml0000664000076400007640000000067410435500317022473 0ustar ajungajung &dtml-title; Zope-2.8.7-final/lib/python/HelpSys/dtml/results.dtml0000664000076400007640000000065210435500317021542 0ustar ajungajung

Help topics matching &dtml-SearchableText;:

&dtml-title_or_id;
No matches.

Zope-2.8.7-final/lib/python/HelpSys/dtml/attributeView.dtml0000664000076400007640000000012210435500317022667 0ustar ajungajung

&dtml-name; = &dtml-value;

Zope-2.8.7-final/lib/python/HelpSys/dtml/menu_header.dtml0000664000076400007640000000011510435500317022307 0ustar ajungajung &dtml-title; Zope-2.8.7-final/lib/python/HelpSys/dtml/frame.dtml0000664000076400007640000000071310435500317021131 0ustar ajungajung Zope Help System SRC="&dtml.url_quote-help_url;" SRC="main" NAME="help_main" MARGINWIDTH="2" MARGINHEIGHT="0" SCROLLING="auto"> Management interfaces require the use of a <B>frames-capable</B> web browser. Zope-2.8.7-final/lib/python/HelpSys/dtml/objectitem.dtml0000664000076400007640000000346310435500317022171 0ustar ajungajung Object Reference

Object Reference

&dtml-meta_type;

&dtml-meta_type; methods

&dtml-get_name;
 		&dtml-get_name;
&dtml-get_signature;
Permission: &dtml-permission;

No documentation for this method

 top

Back to Object Reference

Zope-2.8.7-final/lib/python/HelpSys/dtml/helpsys.dtml0000664000076400007640000000106510435500317021527 0ustar ajungajung Z Object Publishing Environment Help This item requires the use of a <em>frames-capable</em> web browser. Zope-2.8.7-final/lib/python/HelpSys/dtml/addTopic.dtml0000664000076400007640000000110310435500317021560 0ustar ajungajung Add Help Topic

Add Help Topic

Id
Title
Zope-2.8.7-final/lib/python/HelpSys/dtml/helpURL.dtml0000664000076400007640000000011010435500317021341 0ustar ajungajung&dtml-BASEPATH1;/Control_Panel/Products/&dtml-product;/Help/&dtml-topic;Zope-2.8.7-final/lib/python/HelpSys/dtml/helpsys_main.dtml0000664000076400007640000000363510435500317022540 0ustar ajungajung Z Object Publishing Environment Help

Z Object Publishing Environment Help

The Z online help system provides links to in-depth product guides, programming references and other resources.

General Documentation

General documentation for the use and management of built-in and add-on products, as well as an extensive DTML manual are available on the Zope website at http://www.zope.org.

The Zope Guides are a great place to start learning Zope.

Programming References

There is currently an online object reference that documents the services provided by high-level objects such as Documents and Folders. Other references will be added soon.

Other Resources

One of the best resources available is the Zope mailing list. You can subscribe to the mailing list by sending an email to zope-request@zope.org and including the word "subscribe" in the subject line.

There is also a Zope documentation project working on a Zope FAQ

Zope-2.8.7-final/lib/python/HelpSys/dtml/button.dtml0000664000076400007640000000116110435500317021350 0ustar ajungajung
Zope-2.8.7-final/lib/python/HelpSys/dtml/APIHelpView.dtml0000664000076400007640000000204510435500317022114 0ustar ajungajung

API Documentation

1 or (apis and funcs)">

Classes

&dtml-name; ,

1 or (apis and funcs)">

Functions

&dtml-name; ,

Classes

Functions
Zope-2.8.7-final/lib/python/HelpSys/dtml/APIView.dtml0000664000076400007640000000117410435500317021305 0ustar ajungajung

class &dtml-name; ( &dtml-sequence-key;, )

Product Constructor

Methods

 Zope-2.8.7-final/lib/python/HelpSys/dtml/menu_footer.dtml0000664000076400007640000000002010435500317022350 0ustar ajungajung Zope-2.8.7-final/lib/python/HelpSys/dtml/menu.dtml0000664000076400007640000000050610435500317021003 0ustar ajungajung &dtml-title_or_id; &dtml-title; Zope-2.8.7-final/lib/python/HelpSys/dtml/objectref.dtml0000664000076400007640000000152610435500317022005 0ustar ajungajung Object Reference

Object Reference

The object reference documents the interfaces of objects which are built in to the system or have been installed as add-on products. This reference focuses on those object services useful in DTML scripting.

Back to Help

Zope-2.8.7-final/lib/python/HelpSys/dtml/methodView.dtml0000664000076400007640000000070510435500317022153 0ustar ajungajung

&dtml-name;(&dtml-sequence-item; , , &dtml-sequence-key;=&dtml-sequence-item;, , *&dtml-varargs; , **&dtml-kwargs;):

Zope-2.8.7-final/lib/python/HelpSys/HelpTopic.py0000664000076400007640000002007710435500317020463 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import Acquisition from ComputedAttribute import ComputedAttribute from OFS.SimpleItem import Item from Globals import Persistent, HTML, DTMLFile, ImageFile from OFS.DTMLDocument import DTMLDocument from OFS.PropertyManager import PropertyManager import os.path import Globals class HelpTopicBase: "Mix-in Help Topic support class" _properties=( {'id':'title', 'type':'string', 'mode':'w'}, {'id':'categories', 'type':'multiple selection', 'select_variable':'categories_values', 'mode':'w'}, {'id':'permissions', 'type':'multiple selection', 'select_variable':'permissions_values', 'mode':'w'}, ) # default values categories=('Content Manager Information',) permissions=('View',) def _permissions_values(self): perms=[] for m in self.permission_settings(): perms.append(m['name']) return perms permissions_values=ComputedAttribute(_permissions_values, 1) categories_values=( 'Content Manager Information', 'DTML Programmer Information', 'Python Programmer Information', ) def helpValues(self, REQUEST=None): return () def authorized(self, user): "Is a given user authorized to view this Help Topic?" if not self.permissions: return 1 for perm in self.permissions: if user.has_permission(perm, self): return 1 return 0 # Indexable methods # ----------------- def SearchableText(self): "The full text of the Help Topic, for indexing purposes" raise NotImplementedError def url(self): "URL for indexing purposes" return '/'.join(self.getPhysicalPath()) # Private indexing methods # ------------------------ def manage_afterAdd(self, item, container): self.index_object() def manage_afterClone(self, item): self.index_object() def manage_beforeDelete(self, item, container): self.unindex_object() def _setPropValue(self, id, value): setattr(self,id,value) self.reindex_object() def index_object(self, prefix=''): self.get_catalog().catalog_object(self, prefix + self.url()) def unindex_object(self, prefix=''): self.get_catalog().uncatalog_object(prefix + self.url()) def reindex_object(self): self.unindex_object() self.index_object() def get_catalog(self): c = self.catalog # Migrate HelpSys catalog (Zope 2.8+) if not hasattr(c, '_migrated_280'): c.manage_convertIndexes() return c class HelpTopic(Acquisition.Implicit, HelpTopicBase, Item, PropertyManager, Persistent): """ Abstract base class for Help Topics """ meta_type='Help Topic' icon='p_/HelpTopic_icon' _v_last_read = 0 manage_options=( {'label':'Properties', 'action':'manage_propertiesForm'}, {'label':'View', 'action':'index_html'}, ) __ac_permissions__=( ('View', ('index_html', 'SearchableText', 'url')), ('Access contents information', ('helpValues',)), ) def _set_last_read(self, filepath): try: mtime = os.stat(filepath)[8] except: mtime = 0 self._v_last_read = mtime def _check_for_update(self): if Globals.DevelopmentMode: try: mtime=os.stat(self.file)[8] except: mtime=0 if mtime != self._v_last_read: fileob = open(self.file) self.obj = fileob.read() fileob.close() self._v_last_read=mtime self.reindex_object() def index_html(self, REQUEST, RESPONSE): "View the Help Topic" raise NotImplementedError class DTMLDocumentTopic(HelpTopicBase, DTMLDocument): """ A user addable Help Topic based on DTML Document. """ meta_type='Help Topic' icon='p_/HelpTopic_icon' def munge(self,*args, **kw): apply(DTMLDocument.munge, (self,) + args, kw) self.reindex_object() def SearchableText(self): return '%s %s' % (self.title, self.read()) default_topic_content="""\

This is the Help Topic.

""" class DTMLTopic(HelpTopic): """ A basic Help Topic. Holds a HTMLFile object. """ def __init__(self, id, title, file, permissions=None, categories=None): self.id=id self.title=title file,ext=os.path.splitext(file) prefix,file=os.path.split(file) self.index_html=DTMLFile(file,prefix) if permissions is not None: self.permissions=permissions if categories is not None: self.categories=categories def SearchableText(self): "The full text of the Help Topic, for indexing purposes" return '%s %s' % (self.title, self.index_html.read()) class TextTopic(HelpTopic): """ A basic Help Topic. Holds a text file. """ index_html = None def __init__(self, id, title, file, permissions=None, categories=None): self.id=id self.title=title self.file = file self.obj=open(file).read() self._set_last_read(file) if permissions is not None: self.permissions=permissions if categories is not None: self.categories=categories def __call__(self, REQUEST=None): "View the Help Topic" self._check_for_update() return self.obj def SearchableText(self): "The full text of the Help Topic, for indexing purposes" return '%s %s' % (self.title, self.obj) class STXTopic(TextTopic): """ A structured-text topic. Holds a HTMLFile object. """ index_html = None def __call__(self, REQUEST=None): """ View the STX Help Topic """ self._check_for_update() return self.htmlfile(self, REQUEST) htmlfile = HTML("""\ """) class ReSTTopic(TextTopic): """ A reStructuredText [1]_ topic. Similar to STXTopic, it uses a simle DTML construct to render its contents - this time using the *reStructuredText* language. .. [1] reStructuredText (http://docutils.sourceforge.net/rst.html) """ index_html = None def __call__(self, REQUEST=None): """ Renders the ReST Help Topic """ self._check_for_update() return self.htmlfile(self, REQUEST) htmlfile = HTML("""\ """) class ImageTopic(HelpTopic): """ A image Help Topic. Holds an ImageFile object. """ meta_type='Help Image' def __init__(self, id, title, file, categories=None, permissions=None): self.id=id self.title=title self.file = file self.obj=open(file).read() self._set_last_read(file) dir, file=os.path.split(file) self.image=ImageFile(file, dir) if permissions is not None: self.permissions=permissions if categories is not None: self.categories=categories def index_html(self, REQUEST, RESPONSE): "View the Help Topic" self._check_for_update() return self.image.index_html(REQUEST, RESPONSE) def SearchableText(self): "The full text of the Help Topic, for indexing purposes" return '' Zope-2.8.7-final/lib/python/HelpSys/APIHelpTopic.py0000664000076400007640000002234410435500317021014 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ API documentation help topics. $Id: APIHelpTopic.py 24763 2004-05-17 05:59:28Z philikon $ """ import types from AccessControl import ClassSecurityInfo from Globals import DTMLFile from Globals import InitializeClass from Globals import Persistent import HelpTopic _ignore_objects = {} try: import Interface _ignore_objects.update(Interface.__dict__) except ImportError: pass class APIHelpTopic(HelpTopic.HelpTopic): """ Provides API documentation. """ isAPIHelpTopic=1 funcs=() # for backward compatibility def __init__(self, id, title, file): self.id=id self.title=title dict={} execfile(file, dict) self.doc=dict.get('__doc__','') self.apis=[] self.funcs=[] for k, v in dict.items(): if (not _ignore_objects.has_key(k) or _ignore_objects[k] is not v): if type(v)==types.ClassType: # A class. self.apis.append(APIDoc(v, 0)) elif (hasattr(v, 'isImplementedByInstancesOf')): # A scarecrow interface. self.apis.append(APIDoc(v, 1)) elif type(v)==types.FunctionType: # A function self.funcs.append(MethodDoc(v, 0)) # try to get title from first non-blank line # of module docstring if not self.title: lines=self.doc.split('\n') while 1: line=lines[0].strip() if line: # get rid of anything after a colon in the line self.title=line.split(':')[0] break lines.pop(0) if not lines: break # otherwise get title from first class name if not self.title: self.title=self.apis[0].name index_html=DTMLFile('dtml/APIHelpView', globals()) def SearchableText(self): "The full text of the Help Topic, for indexing purposes" text="%s %s" % (self.title, self.doc) for api in self.apis + self.funcs: try: # not all api's provide SearchableText() text="%s %s" % (text, api.SearchableText()) except AttributeError: pass return text class APIDoc(Persistent): """ Describes an API. """ security = ClassSecurityInfo() security.setDefaultAccess( {'attributes': True, 'constructor': True, 'doc': True, 'extends': True, 'name': True, 'methods': True} ) extends=() def __init__(self, klass, isInterface=0): if isInterface: self._createFromInterface(klass) else: self._createFromClass(klass) def _createFromInterface(self, klass): # Creates an APIDoc instance given an interface object. self.name=klass.__name__ self.doc=trim_doc_string(klass.__doc__) # inheritence information self.extends=[] ## for base in klass.getBases(): ## names = string.split(base.__name__, '.') ## url="%s/Help/%s.py#%s" % (names[0], names[1], names[2]) ## self.extends.append((names[2], url)) # constructor information ## if hasattr(klass, '__constructor__'): ## self.constructor=MethodDoc(klass.__constructor__) # Get info on methods and attributes, ignore special items self.attributes=[] self.methods=[] from Interface.Method import Method for k,v in klass.namesAndDescriptions(): if hasattr(v, 'getSignatureInfo'): self.methods.append(MethodDoc(v, 1)) else: self.attributes.append(AttributeDoc(k, v.__doc__)) def _createFromClass(self, klass): # Creates an APIDoc instance given a python class. # the class describes the API; it contains # methods, arguments and doc strings. # # The name of the API is deduced from the name # of the class. # # The base APIs are deduced from the __extends__ # attribute. self.name=klass.__name__ self.doc=trim_doc_string(klass.__doc__) # inheritence information if hasattr(klass,'__extends__'): self.extends=[] for base in klass.__extends__: names=base.split( '.') url="%s/Help/%s.py#%s" % (names[0], names[1], names[2]) self.extends.append((names[2], url)) # constructor information if hasattr(klass, '__constructor__'): self.constructor=MethodDoc(klass.__constructor__) # Get info on methods and attributes, ignore special items self.attributes=[] self.methods=[] for k,v in klass.__dict__.items(): if k not in ('__extends__', '__doc__', '__constructor__'): if type(v)==types.FunctionType: self.methods.append(MethodDoc(v, 0)) else: self.attributes.append(AttributeDoc(k, v)) def SearchableText(self): """ The full text of the API, for indexing purposes. """ text="%s %s" % (self.name, self.doc) for attribute in self.attributes: text="%s %s" % (text, attribute.name) for method in self.methods: text="%s %s %s" % (text, method.name, method.doc) return text view=DTMLFile('dtml/APIView', globals()) InitializeClass(APIDoc) class AttributeDoc(Persistent): """ Describes an attribute of an API. """ security = ClassSecurityInfo() security.setDefaultAccess( {'name': True, 'value': True} ) def __init__(self, name, value): self.name=name self.value=value view=DTMLFile('dtml/attributeView', globals()) InitializeClass(AttributeDoc) class MethodDoc(Persistent): """ Describes a method of an API. required - a sequence of required arguments optional - a sequence of tuples (name, default value) varargs - the name of the variable argument or None kwargs - the name of the kw argument or None """ security = ClassSecurityInfo() security.setDefaultAccess( {'doc': True, 'kwargs': True, 'name': True, 'optional': True, 'required': True, 'varargs': True} ) varargs=None kwargs=None def __init__(self, func, isInterface=0): if isInterface: self._createFromInterfaceMethod(func) else: self._createFromFunc(func) def _createFromInterfaceMethod(self, func): self.name = func.__name__ self.doc = trim_doc_string(func.__doc__) self.required = func.required opt = [] for p in func.positional[len(func.required):]: opt.append((p, func.optional[p])) self.optional = tuple(opt) if func.varargs: self.varargs = func.varargs if func.kwargs: self.kwargs = func.kwargs def _createFromFunc(self, func): if hasattr(func, 'im_func'): func=func.im_func self.name=func.__name__ self.doc=trim_doc_string(func.__doc__) # figure out the method arguments # mostly stolen from pythondoc CO_VARARGS = 4 CO_VARKEYWORDS = 8 names = func.func_code.co_varnames nrargs = func.func_code.co_argcount if func.func_defaults: nrdefaults = len(func.func_defaults) else: nrdefaults = 0 self.required = names[:nrargs-nrdefaults] if func.func_defaults: self.optional = tuple(map(None, names[nrargs-nrdefaults:nrargs], func.func_defaults)) else: self.optional = () varargs = [] ix = nrargs if func.func_code.co_flags & CO_VARARGS: self.varargs=names[ix] ix = ix+1 if func.func_code.co_flags & CO_VARKEYWORDS: self.kwargs=names[ix] view=DTMLFile('dtml/methodView', globals()) InitializeClass(MethodDoc) def trim_doc_string(text): """ Trims a doc string to make it format correctly with structured text. """ text=text.strip() text=text.replace( '\r\n', '\n') lines=text.split('\n') nlines=[lines[0]] if len(lines) > 1: min_indent=None for line in lines[1:]: if not line: continue indent=len(line) - len(line.lstrip()) if indent < min_indent or min_indent is None: min_indent=indent for line in lines[1:]: nlines.append(line[min_indent:]) return '\n'.join(nlines) Zope-2.8.7-final/lib/python/HelpSys/images/0000775000076400007640000000000010435500317017461 5ustar ajungajungZope-2.8.7-final/lib/python/HelpSys/images/hs_larrow.gif0000775000076400007640000000150310435500317022152 0ustar ajungajungGIF89a 111p!,  ( L!Ç:|HA /b ;Zope-2.8.7-final/lib/python/HelpSys/images/hs_cbook.gif0000775000076400007640000000160410435500317021743 0ustar ajungajungGIF89aPPP@@@@@!,aH,A :tĉ+0Əvb#IRH1%I'[z,(g&|͌v|1P 04iDN80 ;Zope-2.8.7-final/lib/python/HelpSys/images/hs_rarrow.gif0000775000076400007640000000150310435500317022160 0ustar ajungajungGIF89a 111p!,  Hp  DHpC#FbB/;Zope-2.8.7-final/lib/python/HelpSys/images/hs_darrow.gif0000775000076400007640000000150310435500317022142 0ustar ajungajungGIF89a 111p!,  8 ÆB\(1!Ŋ1J ;Zope-2.8.7-final/lib/python/HelpSys/images/hs_dnode.gif0000775000076400007640000000160410435500317021737 0ustar ajungajungGIF89aPPP@@@@@!,a@AxჅ*tpÄ)f8" A"\dIR鱡J&c"x@`eǖ(mNiΌ-k9SMF'92ӧ/J5 ;Zope-2.8.7-final/lib/python/HelpSys/images/hs_uarrow.gif0000775000076400007640000000150410435500317022164 0ustar ajungajungGIF89a 111p!, ! $`AXpaA Ň f\;Zope-2.8.7-final/lib/python/HelpSys/images/hs_obook.gif0000775000076400007640000000162410435500317021761 0ustar ajungajungGIF89aPPP@@@@@!,qH *D B<0"B d!BrI"H!I6x!.$leʚ+?ft kN@N@]@":5FjU*U= ;Zope-2.8.7-final/lib/python/HelpSys/images/helpTopic.gif0000664000076400007640000000017510435500317022102 0ustar ajungajungGIF89aPPP!,@BX*ܮ@!rKfjty٥m] R|U\`CK>$e+\0z;Zope-2.8.7-final/lib/python/HelpSys/images/productHelp.gif0000664000076400007640000000021110435500317022433 0ustar ajungajungGIF89aPPP!,@NxJ@Y'jD&m),ePfEWZ@jC(`@qiM,h= * 'h7);Zope-2.8.7-final/lib/python/HelpSys/ObjectRef.py0000664000076400007640000001036610435500317020437 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Object Reference implementation""" __version__='$Revision: 1.10 $'[11:-2] import sys, os, Globals, Acquisition from HelpUtil import HelpBase, classobject from HelpUtil import is_class, is_module from Globals import DTMLFile from urllib import quote class ObjectItem(HelpBase, classobject): """ """ __roles__=None hs_main=DTMLFile('dtml/objectitem', globals()) hs_cicon='HelpSys/hs_dnode' hs_eicon='HelpSys/hs_dnode' def hs_id(self): return self._obj_.meta_type def hs_url(self): return quote(self._obj_.meta_type) hs_title=hs_id def __getattr__(self, name): if name in ('isDocTemp', '__call__'): raise AttributeError, name return getattr(self.__dict__['_obj_'], name) def get_method_list(self): rdict=classobject.get_method_dict(self) perms=self.__ac_permissions__ mdict={} mlist=[] for p in self.__ac_permissions__: pname=p[0] fnames=p[1] for fname in fnames: if rdict.has_key(fname): fn=rdict[fname] fn.permission=pname mdict[fname]=fn keys=mdict.keys() keys.sort() for key in keys: fn=mdict[key] if not hasattr(fn._obj_, '__doc__'): continue doc=fn._obj_.__doc__ if hasattr(fn._obj_, '__class__') and \ fn._obj_.__class__.__doc__ is doc: continue mlist.append(mdict[key]) del rdict del mdict return mlist hs_objectvalues__roles__=None def hs_objectvalues(self): return [] class ObjectRef(HelpBase): """ """ __names__=None __roles__=None hs_main=DTMLFile('dtml/objectref', globals()) hs_cicon='HelpSys/hs_cbook' hs_eicon='HelpSys/hs_obook' hs_id ='ObjectRef' hs_title='Object Reference' hs_url =hs_id def hs_deferred__init__(self): # This is necessary because we want to wait until all # products have been installed (imported). dict={} for k, v in sys.modules.items(): if v is not None and k != '__builtins__': dict=self.hs_search_mod(v, dict) keys=dict.keys() keys.sort() __traceback_info__=(`dict`,) for key in keys: setattr(self, key, dict[key]) self.__names__=keys def hs_search_mod(self, mod, dict): # Root through a module for things that look like # createable object classes. hidden=('Control Panel', 'Principia Draft', 'simple item', 'Broken Because Product is Gone') for k, v in mod.__dict__.items(): if is_class(v) and hasattr(v, 'meta_type') and \ hasattr(v, '__ac_permissions__'): if callable(v.meta_type): try: meta_type=v.meta_type() except: # Ack. probably a ZClass :( meta_type=None else: meta_type=v.meta_type if (meta_type is not None) and (meta_type not in hidden): dict[meta_type]=ObjectItem(k, v) if is_module(v) and hasattr(v, '__path__'): dict=self.hs_search_mod(v, dict) return dict hs_objectvalues__roles__=None def hs_objectvalues(self): if self.__names__ is None: self.hs_deferred__init__() items=[] for id in self.__names__: items.append(getattr(self, id)) return items def __getitem__(self, key): return self.__dict__[key].__of__(self) Zope-2.8.7-final/lib/Components/0000775000076400007640000000000010435500331015425 5ustar ajungajungZope-2.8.7-final/lib/Components/BTree/0000775000076400007640000000000010435500331016426 5ustar ajungajungZope-2.8.7-final/lib/Components/BTree/intSet.c0000775000076400007640000003133710435500331020052 0ustar ajungajung/***************************************************************************** Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ static char intSet_module_documentation[] = "" "\n$Id: intSet.c,v 1.26 2003/10/19 22:26:29 tseaver Exp $" ; #include "cPersistence.h" #define UNLESS(E) if(!(E)) #define RETURN_NONE Py_INCREF(Py_None); return Py_None #define MIN_INTSET_ALLOC 8 #define INTSET_DATA_TYPE int #define INTSET_DATA_FLAG "l" typedef struct { cPersistent_HEAD int size, len; INTSET_DATA_TYPE *data; } intSet; staticforward PyExtensionClass intSetType; #define OBJECT(O) ((PyObject*)(O)) #define INTSET(O) ((intSet*)(O)) /* We want to be sticky most of the time */ #define PER_RETURN(O,R) R #define PER_INT_RETURN(O,R) R #undef PER_ALLOW_DEACTIVATION #define PER_ALLOW_DEACTIVATION(O) static PyObject * intSet_has_key(intSet *self, PyObject *args) { int min, max, i, l; INTSET_DATA_TYPE k, key; UNLESS(PyArg_ParseTuple(args,INTSET_DATA_FLAG,&key)) return NULL; PER_USE_OR_RETURN(self, NULL); for(min=0, max=self->len, i=max/2, l=max; i != l; l=i, i=(min+max)/2) { k=self->data[i]; if(k == key) return PER_RETURN(self, PyInt_FromLong(1)); if(k > key) max=i; else min=i; } return PER_RETURN(self, PyInt_FromLong(0)); } static int intSet_grow(intSet *self, int l) { int g; INTSET_DATA_TYPE *data; if(self->data) { g=self->size*2; if(g < l) g=l; UNLESS(data=realloc(self->data, sizeof(INTSET_DATA_TYPE)*g)) { PyErr_NoMemory(); return -1; } self->data=data; self->size=g; } else { g=l < MIN_INTSET_ALLOC ? MIN_INTSET_ALLOC : l; UNLESS(self->data=malloc(sizeof(INTSET_DATA_TYPE)*g)) { PyErr_NoMemory(); return -1; } self->size=g; } return 0; } static INTSET_DATA_TYPE intSet_modify(intSet *self, INTSET_DATA_TYPE ikey, int add) { int min, max, i, l; INTSET_DATA_TYPE *data, k; PER_USE_OR_RETURN(self, -1); data=self->data; for(min=0, max=self->len, i=max/2, l=max; i != l; l=i, i=(min+max)/2) { k=data[i]; if(k == ikey) { if(! add) { data+=i; self->len--; if(i < (self->len)) memmove(data, data+1, (self->len-i)*sizeof(INTSET_DATA_TYPE)); if(PER_CHANGED(self) < 0) return PER_INT_RETURN(self, -1); } return PER_INT_RETURN(self, 0); } if(k > ikey) max=i; else min=i; } if(!add) return PER_INT_RETURN(self, 0); if(self->len >= self->size && intSet_grow(self,self->len+1) < 0) return PER_INT_RETURN(self, -1); if(max != i) i++; data=self->data+i; if(self->len > i) memmove(data+1,data,(self->len-i)*sizeof(INTSET_DATA_TYPE)); *data=ikey; self->len++; if(PER_CHANGED(self) < 0) return PER_INT_RETURN(self, -1); return PER_INT_RETURN(self, ikey); } static PyObject * intSet_insert(intSet *self, PyObject *args) { INTSET_DATA_TYPE key; UNLESS(PyArg_ParseTuple(args,INTSET_DATA_FLAG,&key)) return NULL; if(intSet_modify(self, key, 1) < 0) return NULL; RETURN_NONE; } static PyObject * intSet_remove(intSet *self, PyObject *args) { INTSET_DATA_TYPE key; UNLESS(PyArg_ParseTuple(args,INTSET_DATA_FLAG,&key)) return NULL; if(intSet_modify(self, key, 0) < 0) return NULL; RETURN_NONE; } static PyObject * intSet_clear(intSet *self, PyObject *args) { self->len=0; if(PER_CHANGED(self) < 0) return PER_RETURN(self, NULL); RETURN_NONE; } static PyObject * intSet___getstate__(intSet *self, PyObject *args) { PyObject *r=0; int i, l; char *c; INTSET_DATA_TYPE *d; PER_USE_OR_RETURN(self, NULL); l=self->len; UNLESS(r=PyString_FromStringAndSize(NULL,l*4)) goto err; UNLESS(c=PyString_AsString(r)) goto err; d=self->data; for(i=0; i < l; i++, d++) { *c++ = (int)( *d & 0xff); *c++ = (int)((*d >> 8) & 0xff); *c++ = (int)((*d >> 16) & 0xff); *c++ = (int)((*d >> 24) & 0xff); } return PER_RETURN(self, r); err: Py_DECREF(r); return PER_RETURN(self, NULL); } static PyObject * intSet___setstate__(intSet *self, PyObject *args) { PyObject *data; int i, l, v; char *c; PER_PREVENT_DEACTIVATION(self); UNLESS(PyArg_ParseTuple(args,"O",&data)) return PER_RETURN(self, NULL); UNLESS(c=PyString_AsString(data)) return PER_RETURN(self, NULL); if((l=PyString_Size(data)) < 0) return PER_RETURN(self, NULL); l/=4; intSet_clear(self, NULL); if(l > self->size && intSet_grow(self,l) < 0) return PER_RETURN(self, NULL); PyErr_Clear(); for(i=0; i < l; i++) { v = ((int)(unsigned char)*c++) ; v |= ((int)(unsigned char)*c++) << 8; v |= ((int)(unsigned char)*c++) << 16; v |= ((int)(unsigned char)*c++) << 24; self->data[i]=v; } self->len=l; Py_INCREF(Py_None); return PER_RETURN(self, Py_None); } static PyObject * intSet_set_operation(intSet *self, PyObject *other, int cpysrc, int cpyboth, int cpyoth) { intSet *r=0, *o; int i, l, io, lo; INTSET_DATA_TYPE *d, *od, v, vo; if(other->ob_type != self->ob_type) { PyErr_SetString(PyExc_TypeError, "intSet set operations require same-type operands"); return NULL; } o=INTSET(other); PER_USE_OR_RETURN(self, NULL); PER_USE_OR_RETURN((intSet*)other, NULL); od=o->data; d=self->data; UNLESS(r=INTSET(PyObject_CallObject(OBJECT(self->ob_type), NULL))) goto err; for(i=0, l=self->len, io=0, lo=o->len; i < l && io < lo; ) { v=d[i]; vo=od[io]; if(v < vo) { if(cpysrc) { if(r->len >= r->size && intSet_grow(r,0) < 0) goto err; r->data[r->len]=v; r->len++; } i++; } else if(v==vo) { if(cpyboth) { if(r->len >= r->size && intSet_grow(r,0) < 0) goto err; r->data[r->len]=v; r->len++; } i++; io++; } else { if(cpyoth) { if(r->len >= r->size && intSet_grow(r,0) < 0) goto err; r->data[r->len]=vo; r->len++; } io++; } } if(cpysrc && i < l) { l-=i; if(r->len+l > r->size && intSet_grow(r,r->len+l) < 0) goto err; memcpy(r->data+r->len, d+i, l*sizeof(INTSET_DATA_TYPE)); r->len += l; } else if(cpyoth && io < lo) { lo-=io; if(r->len+lo > r->size && intSet_grow(r,r->len+lo) < 0) goto err; memcpy(r->data+r->len, od+io, lo*sizeof(INTSET_DATA_TYPE)); r->len += lo; } PER_ALLOW_DEACTIVATION(self); PER_ALLOW_DEACTIVATION(o); return OBJECT(r); err: PER_ALLOW_DEACTIVATION(self); PER_ALLOW_DEACTIVATION(o); Py_DECREF(r); return NULL; } static PyObject * intSet_add(intSet *self, PyObject *other) { return intSet_set_operation(self,other,1,1,1); } static PyObject * intSet_union(intSet *self, PyObject *args) { PyObject *other; UNLESS(PyArg_ParseTuple(args,"O",&other)) return NULL; return intSet_set_operation(self,other,1,1,1); } static PyObject * intSet_intersection(intSet *self, PyObject *args) { PyObject *other; UNLESS(PyArg_ParseTuple(args,"O",&other)) return NULL; return intSet_set_operation(self,other,0,1,0); } static PyObject * intSet_difference(intSet *self, PyObject *args) { PyObject *other; UNLESS(PyArg_ParseTuple(args,"O",&other)) return NULL; return intSet_set_operation(self,other,1,0,0); } static PyObject * intSet__p___reinit__(intSet *self, PyObject *args) { PyObject *dict; if(self->state==cPersistent_UPTODATE_STATE && HasInstDict(self) && (dict=INSTANCE_DICT(self))) { PyDict_Clear(dict); } PER_GHOSTIFY(self); Py_INCREF(Py_None); return Py_None; } static struct PyMethodDef intSet_methods[] = { {"has_key", (PyCFunction)intSet_has_key, METH_VARARGS, "has_key(id) -- Test whether the set has the given id"}, {"insert", (PyCFunction)intSet_insert, METH_VARARGS, "insert(id,[ignored]) -- Add an id to the set"}, {"remove", (PyCFunction)intSet_remove, METH_VARARGS, "remove(id) -- Remove an id from the set"}, {"clear", (PyCFunction)intSet_clear, METH_VARARGS, "clear() -- Remove all of the ids from the set"}, {"union", (PyCFunction)intSet_union, METH_VARARGS, "union(other) -- Return the union of the set with another set"}, {"intersection", (PyCFunction)intSet_intersection, METH_VARARGS, "intersection(other) -- " "Return the intersection of the set with another set"}, {"difference", (PyCFunction)intSet_difference, METH_VARARGS, "difference(other) -- Return the difference of the set with another set" }, {"__getstate__", (PyCFunction)intSet___getstate__, METH_VARARGS, "__getstate__() -- get the persistent state"}, {"__setstate__", (PyCFunction)intSet___setstate__, METH_VARARGS, "__setstate__() -- set the persistent state"}, {"_p___reinit__", (PyCFunction)intSet__p___reinit__, METH_VARARGS, "_p___reinit__(oid,jar,copy) -- Reinitialize from a newly created copy"}, {"_p_deactivate", (PyCFunction)intSet__p___reinit__, METH_VARARGS, "_p_deactivate(oid,jar,copy) -- Reinitialize from a newly created copy"}, {NULL, NULL} /* sentinel */ }; static void intSet_dealloc(intSet *self) { free(self->data); PER_DEL(self); Py_DECREF(self->ob_type); PyMem_DEL(self); } static PyObject * intSet_getattr(intSet *self, PyObject *name) { return Py_FindAttr((PyObject *)self, name); } /* Code to handle accessing intSet objects as sequence objects */ static int intSet_length(intSet *self) { PER_USE_OR_RETURN(self,-1); return PER_INT_RETURN(self,self->len); } static PyObject * intSet_repeat(intSet *self, int n) { PyErr_SetString(PyExc_TypeError, "intSet objects do not support repetition"); return NULL; } static PyObject * intSet_item(intSet *self, int i) { PyObject *e; PER_USE_OR_RETURN(self,NULL); if(i >= 0 && i < self->len) return PER_RETURN(self,PyInt_FromLong(self->data[i])); UNLESS(e=PyInt_FromLong(i)) goto err; PyErr_SetObject(PyExc_IndexError, e); Py_DECREF(e); err: PER_ALLOW_DEACTIVATION(self) return NULL; } static PyObject * intSet_slice(intSet *self, int ilow, int ihigh) { PyErr_SetString(PyExc_TypeError, "intSet objects do not support slicing"); return NULL; } static int intSet_ass_item(intSet *self, int i, PyObject *v) { PyErr_SetString(PyExc_TypeError, "intSet objects do not support item assignment"); return -1; } static int intSet_ass_slice(PyListObject *self, int ilow, int ihigh, PyObject *v) { PyErr_SetString(PyExc_TypeError, "intSet objects do not support slice assignment"); return -1; } static PySequenceMethods intSet_as_sequence = { (inquiry)intSet_length, /*sq_length*/ (binaryfunc)intSet_add, /*sq_concat*/ (intargfunc)intSet_repeat, /*sq_repeat*/ (intargfunc)intSet_item, /*sq_item*/ (intintargfunc)intSet_slice, /*sq_slice*/ (intobjargproc)intSet_ass_item, /*sq_ass_item*/ (intintobjargproc)intSet_ass_slice, /*sq_ass_slice*/ }; static PyExtensionClass intSetType = { PyObject_HEAD_INIT(NULL) 0, /*ob_size*/ "intSet", /*tp_name*/ sizeof(intSet), /*tp_basicsize*/ 0, /*tp_itemsize*/ /* methods */ (destructor)intSet_dealloc, /*tp_dealloc*/ (printfunc)0, /*tp_print*/ (getattrfunc)0, /*obsolete tp_getattr*/ (setattrfunc)0, /*obsolete tp_setattr*/ (cmpfunc)0, /*tp_compare*/ (reprfunc)0, /*tp_repr*/ 0, /*tp_as_number*/ &intSet_as_sequence, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ (hashfunc)0, /*tp_hash*/ (ternaryfunc)0, /*tp_call*/ (reprfunc)0, /*tp_str*/ (getattrofunc)intSet_getattr, /*tp_getattro*/ 0, /*tp_setattro*/ /* Space for future expansion */ 0L,0L, "A set of integers", METHOD_CHAIN(intSet_methods), PERSISTENCE_FLAGS, }; static struct PyMethodDef module_methods[] = { {NULL, NULL} /* sentinel */ }; void initintSet(void) { PyObject *m, *d; UNLESS(ExtensionClassImported) return; if((cPersistenceCAPI=PyCObject_Import("cPersistence","CAPI"))) { intSetType.methods.link=cPersistenceCAPI->methods; intSetType.tp_getattro=cPersistenceCAPI->getattro; intSetType.tp_setattro=cPersistenceCAPI->setattro; } else return; /* Create the module and add the functions */ m = Py_InitModule4("intSet", module_methods, intSet_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); PyExtensionClass_Export(d,"intSet",intSetType); /* Check for errors */ if (PyErr_Occurred()) Py_FatalError("can't initialize module intSet"); } Zope-2.8.7-final/lib/Components/initgroups/0000775000076400007640000000000010435500331017630 5ustar ajungajungZope-2.8.7-final/lib/Components/initgroups/initgroups.c0000664000076400007640000000276310435500331022207 0ustar ajungajung/***************************************************************************** Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved. This software is subject to the provisions of the Zope Public License, Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE ****************************************************************************/ #include "Python.h" #if defined(__unix__) || defined(unix) || defined(__NetBSD__) || defined(__MACH__) /* Mac OS X */ #include #include #include static PyObject * initgroups_initgroups(PyObject *self, PyObject *args) { char *username; unsigned int igid; gid_t gid; if (!PyArg_ParseTuple(args, "sI:initgroups", &username, &igid)) return NULL; gid = igid; if (initgroups(username, gid) == -1) return PyErr_SetFromErrno(PyExc_OSError); Py_INCREF(Py_None); return Py_None; } static PyMethodDef InitgroupsMethods[] = { {"initgroups", initgroups_initgroups, METH_VARARGS}, {NULL, NULL} }; #else /* This module is empty on non-UNIX systems. */ static PyMethodDef InitgroupsMethods[] = { {NULL, NULL} }; #endif /* defined(__unix__) || defined(unix) */ void initinitgroups(void) { Py_InitModule("initgroups", InitgroupsMethods); } Zope-2.8.7-final/setup.py0000664000076400007640000005051310435500314014251 0ustar ajungajung #!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Distutils setup for Zope In-place building This builds extension modules in-place, much like build_extensions.py does. Use 'setup.py' like this:: python setup.py build_ext -i Installation This builds extension modules, compiles python modules, and installs everything needed to support Zope instances in the directory of your choosing. For example, to use '/usr/local/lib/zope':: python setup.py install \ --home=/usr/local/lib/zope \ --install-platlib=/usr/local/lib/zope \ --install-purelib=/usr/local/lib/zope """ import glob import os import sys import distutils.core # Override install_data to install into module directories, and to support # globbing on data_files. from distutils.command.install import install from distutils.command.install_data import install_data from distutils.util import convert_path class ZopeInstallData(install_data): def finalize_options(self): self.set_undefined_options('install', ('install_purelib', 'install_dir'), ('root', 'root'), ('force', 'force'), ) def run(self): self.mkpath(self.install_dir) for f in self.data_files: if isinstance(f, str): # it's a simple file, so copy it f = convert_path(f) gl = glob.glob(f) if len(gl) == 0: raise distutils.core.DistutilsFileError, \ "can't copy '%s': no matching files" % f for g in gl: if os.path.isfile(g): if self.warn_dir: self.warn("setup script did not provide a " "directory for '%s' -- installing " "right in '%s'" % (g, self.install_dir)) (out, _) = self.copy_file(g, self.install_dir) self.outfiles.append(out) else: # it's a tuple with path to install to and a list of files dir = convert_path(f[0]) if not os.path.isabs(dir): dir = os.path.join(self.install_dir, dir) elif self.root: dir = change_root(self.root, dir) self.mkpath(dir) for data in f[1]: data = convert_path(data) gl = glob.glob(data) if len(gl) == 0: raise distutils.core.DistutilsFileError, \ "can't copy '%s': no matching files" % data for g in gl: if os.path.isfile(g): (out, _) = self.copy_file(g, dir) self.outfiles.append(out) # We create a custom "install scheme" that works the same way on all # platforms. We do this in order to prevent distutils from trying to # guess where to put our files on a per-platform basis. ZOPE_INSTALL_SCHEME = { 'purelib': '$base/lib/python', 'platlib': '$base/lib/python', 'headers': '$base/lib/python', 'scripts': '$base/bin', 'data' : '$base/lib/python', } class ZopeInstall(install): # give distutils install step knowledge about build file placement options user_options = install.user_options + [ ('build-base=', None, 'base directory for build library'), ('build-lib=', None, 'build directory for all distribution'), ('build-scripts=', None, 'build directory for scripts'), ('build-temp=', None, 'temporary build directory'), ] build_scripts = None build_temp = None def run(self): """ Override run to pass along build location info so we can use custom build directories """ build = self.distribution.get_command_obj('build') build.build_base = self.build_base build.build_lib = self.build_lib build.build_scripts = self.build_scripts build.build_temp = self.build_temp install.run(self) def select_scheme(self, name): """ Override the default platform installation schemes, ignoring whatever 'name' is passed in. For our purposes, we want to put all library, header, and data into [install_base]/lib/python. Comment this method out to achieve distutils-standard platform-specific behavior for 'setup.py install'. This is most useful if you set the [install-base] by using the '--prefix' or '--home' flags on the setup.py install command line. Otherwise, all Zope software will probably be installed to your Python's 'lib/python' directory. """ scheme = ZOPE_INSTALL_SCHEME import distutils.command.install for key in distutils.command.install.SCHEME_KEYS: attrname = 'install_' + key if getattr(self, attrname) is None: setattr(self, attrname, scheme[key]) class ZopeDistribution(distutils.core.Distribution): def __init__(self, attrs): distutils.core.Distribution.__init__(self, attrs) self.cmdclass["install"] = ZopeInstall self.cmdclass["install_data"] = ZopeInstallData # presumes this script lives in the base dir BASE_DIR=os.path.dirname(os.path.abspath(sys.argv[0])) AUTHOR = 'Zope Corporation and Contributors' # Most modules are in lib/python in the source distribution PACKAGES_ROOT = os.path.join(BASE_DIR, 'lib', 'python') os.chdir(PACKAGES_ROOT) # Most of this is ripped from the Zope 3 setup.py. from distutils import dir_util from distutils.command.build import build as buildcmd from distutils.command.build_ext import build_ext from distutils.command.install_lib import install_lib as installcmd from distutils.core import setup from distutils.dist import Distribution from distutils.extension import Extension # This class serves multiple purposes. It walks the file system looking for # auxiliary files that distutils doesn't install properly, and it actually # copies those files (when hooked into by distutils). It also walks the file # system looking for candidate packages for distutils to install as normal. # The key here is that the package must have an __init__.py file. class Finder: def __init__(self, ignore, prefix): self._files = [] self._pkgs = {} self._ignore = ignore # We're finding packages in lib/python in the source dir, but we're # copying them directly under build/lib.. So we need to lop off # the prefix when calculating the package names from the file names. self._plen = len(prefix) + 1 def visit(self, dir, files): # First see if this is one of the packages we want to add, or if # we're really skipping this package. if '__init__.py' in files: aspkg = dir[self._plen:].replace(os.sep, '.') self._pkgs[aspkg] = True ignore = ('.py',) + self._ignore else: ignore = self._ignore # Add any extra files we're interested in for file in files: base, ext = os.path.splitext(file) if ext not in ignore: self._files.append(os.path.join(dir, file)) def copy_files(self, cmd, outputbase): for file in self._files: dest = os.path.join(outputbase, file[self._plen:]) # Make sure the destination directory exists dir = os.path.dirname(dest) if not os.path.exists(dir): dir_util.mkpath(dir) cmd.copy_file(file, dest) def get_packages(self): return self._pkgs.keys() def remove_stale_bytecode(arg, dirname, names): names = map(os.path.normcase, names) for name in names: if name.endswith(".pyc") or name.endswith(".pyo"): srcname = name[:-1] if srcname not in names: fullname = os.path.join(dirname, name) print "Removing stale bytecode file", fullname os.unlink(fullname) # Create the finder instance, which will be used in lots of places. `finder' # is the global we're most interested in. IGNORE_EXTS = ('.pyc', '.pyo', '.c', '.h', '.so', '.cfg') finder = Finder(IGNORE_EXTS, PACKAGES_ROOT) for dirpath, dirnames, filenames in os.walk(PACKAGES_ROOT): if not '.svn' in dirpath: finder.visit(dirpath, filenames) packages = finder.get_packages() # Distutils hook classes class MyBuilder(buildcmd): def run(self): os.path.walk(os.curdir, remove_stale_bytecode, None) buildcmd.run(self) finder.copy_files(self, self.build_lib) class MyExtBuilder(build_ext): # Override the default build_ext to remove stale bytecodes. # Technically, removing bytecode has nothing to do with # building extensions, but Zope's the build_ext -i variant # is used to build Zope in place. def run(self): os.path.walk(os.curdir, remove_stale_bytecode, None) build_ext.run(self) class MyLibInstaller(installcmd): def run(self): installcmd.run(self) finder.copy_files(self, self.install_dir) class MyDistribution(Distribution): # To control the selection of MyLibInstaller and MyPyBuilder, we # have to set it into the cmdclass instance variable, set in # Distribution.__init__(). def __init__(self, *attrs): Distribution.__init__(self, *attrs) self.cmdclass['install'] = ZopeInstall self.cmdclass['build'] = MyBuilder self.cmdclass['build_ext'] = MyExtBuilder self.cmdclass['install_lib'] = MyLibInstaller EXTENSIONCLASS_INCLUDEDIRS = ['ExtensionClass'] DCPYEXPAT_INCLUDEDIRS = ['Shared/DC/xml/pyexpat/expat/xmlparse', 'Shared/DC/xml/pyexpat/expat/xmltok'] DCPYEXPAT_DIR = 'Shared/DC/xml/pyexpat' # All extension modules must be listed here. ext_modules = [ # AccessControl Extension(name='AccessControl.cAccessControl', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['Acquisition'], sources=['AccessControl/cAccessControl.c'], depends=['ExtensionClass/ExtensionClass.h', 'ExtensionClass/pickle/pickle.c', 'Acquisition/Acquisition.h']), # BTrees Extension(name='BTrees._OOBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], sources=['BTrees/_OOBTree.c']), Extension(name='BTrees._OIBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], sources=['BTrees/_OIBTree.c']), Extension(name='BTrees._IIBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], define_macros=[('EXCLUDE_INTSET_SUPPORT', None)], sources=['BTrees/_IIBTree.c']), Extension(name='BTrees._IOBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], define_macros=[('EXCLUDE_INTSET_SUPPORT', None)], sources=['BTrees/_IOBTree.c']), Extension(name='BTrees._IFBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], define_macros=[('EXCLUDE_INTSET_SUPPORT', None)], sources=['BTrees/_IFBTree.c']), Extension(name='BTrees._fsBTree', include_dirs=EXTENSIONCLASS_INCLUDEDIRS+['persistent'], define_macros=[('EXCLUDE_INTSET_SUPPORT', None)], sources=['BTrees/_fsBTree.c']), # DocumentTemplate Extension(name='DocumentTemplate.cDocumentTemplate', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=['DocumentTemplate/cDocumentTemplate.c']), # ExtensionClass Extension(name='ExtensionClass._ExtensionClass', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["ExtensionClass/_ExtensionClass.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='Acquisition._Acquisition', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["Acquisition/_Acquisition.c"], depends=["ExtensionClass/ExtensionClass.h", "Acquisition/Acquisition.h"]), Extension(name='MethodObject._MethodObject', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["MethodObject/_MethodObject.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='MultiMapping._MultiMapping', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["MultiMapping/_MultiMapping.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='ThreadLock._ThreadLock', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["ThreadLock/_ThreadLock.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='Missing._Missing', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["Missing/_Missing.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='Record._Record', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["Record/_Record.c"], depends=["ExtensionClass/ExtensionClass.h"]), Extension(name='ComputedAttribute._ComputedAttribute', include_dirs=EXTENSIONCLASS_INCLUDEDIRS, sources=["ComputedAttribute/_ComputedAttribute.c"], depends=["ExtensionClass/ExtensionClass.h"]), # dcpyexpat Extension(name='Shared.DC.xml.pyexpat.dcpyexpat', include_dirs=DCPYEXPAT_INCLUDEDIRS, define_macros=[('XML_NS', None)], sources=[DCPYEXPAT_DIR + '/expat/xmlparse/xmlparse.c', DCPYEXPAT_DIR + '/expat/xmlparse/hashtable.c', DCPYEXPAT_DIR + '/expat/xmltok/xmlrole.c', DCPYEXPAT_DIR + '/expat/xmltok/xmltok.c', DCPYEXPAT_DIR + '/dcpyexpat.c']), # initgroups Extension(name='initgroups', sources=['../Components/initgroups/initgroups.c']), # indexes Extension(name='Products.PluginIndexes.TextIndex.Splitter.ZopeSplitter.ZopeSplitter', sources=['Products/PluginIndexes/TextIndex/Splitter/ZopeSplitter/src/ZopeSplitter.c']), Extension(name='Products.PluginIndexes.TextIndex.Splitter.ISO_8859_1_Splitter.ISO_8859_1_Splitter', sources=['Products/PluginIndexes/TextIndex/Splitter/ISO_8859_1_Splitter/src/ISO_8859_1_Splitter.c']), Extension(name='Products.PluginIndexes.TextIndex.Splitter.UnicodeSplitter.UnicodeSplitter', sources=['Products/PluginIndexes/TextIndex/Splitter/UnicodeSplitter/src/UnicodeSplitter.c']), Extension(name='Products.ZCTextIndex.stopper', sources=['Products/ZCTextIndex/stopper.c']), Extension(name='Products.ZCTextIndex.okascore', sources=['Products/ZCTextIndex/okascore.c']), #ZODB Extension(name = 'persistent.cPersistence', include_dirs = ['persistent'], sources= ['persistent/cPersistence.c', 'persistent/ring.c'], depends = ['persistent/cPersistence.h', 'persistent/ring.h', 'persistent/ring.c'] ), Extension(name = 'Persistence._Persistence', include_dirs = ['persistent', 'ExtensionClass'], sources = ['Persistence/_Persistence.c'], depends = ['persistent/cPersistence.h', 'ExtensionClass/ExtensionClass.h'] ), Extension(name = 'persistent.cPickleCache', include_dirs = ['persistent'], sources= ['persistent/cPickleCache.c', 'persistent/ring.c'], depends = ['persistent/cPersistence.h', 'persistent/ring.h', 'persistent/ring.c'] ), Extension(name = 'persistent.TimeStamp', sources= ['persistent/TimeStamp.c'] ), Extension(name = 'ZODB.winlock', sources = ['ZODB/winlock.c'] ), #zope Extension("zope.proxy._zope_proxy_proxy", ["zope/proxy/_zope_proxy_proxy.c"], include_dirs = ["zope/proxy"], depends = ["zope/proxy/proxy.h"]), Extension("zope.security._proxy", ["zope/security/_proxy.c"], include_dirs = ["zope/proxy"], depends = ["zope/proxy/proxy.h"]), Extension("zope.security._zope_security_checker", ["zope/security/_zope_security_checker.c"], include_dirs = [], depends = []), Extension("zope.interface._zope_interface_coptimizations", ["zope/interface/_zope_interface_coptimizations.c"]), Extension("zope.hookable._zope_hookable", ["zope/hookable/_zope_hookable.c"]), Extension("zope.thread._zope_thread", ["zope/thread/_zope_thread.c"]), Extension("zope.app.container._zope_app_container_contained", ["zope/app/container/_zope_app_container_contained.c"], include_dirs = ["persistent", "zope/proxy", "zope/app/container"], depends = [ "persistent/cPersistence.h", "zope/proxy/_zope_proxy_proxy.c", ]), ] # We're using the module docstring as the distutils descriptions. doclines = __doc__.split("\n") setup(name='Zope', author=AUTHOR, version="2.8", maintainer="Zope Corporation", maintainer_email="zope-dev@zope.org", url = "http://www.zope.org/", ext_modules = ext_modules, license = "http://www.zope.org/Resources/ZPL", platforms = ["any"], description = doclines[0], long_description = "\n".join(doclines[2:]), packages = packages, distclass = MyDistribution, ) # The rest of these modules live in the root of the source tree os.chdir(BASE_DIR) IGNORE_NAMES = ( 'CVS', '.svn', # Revision Control Directories ) def skel_visit(skel, dirname, names): for ignore in IGNORE_NAMES: if ignore in names: names.remove(ignore) L = [] for name in names: if os.path.isfile(os.path.join(dirname, name)): L.append("%s/%s" % (dirname, name)) skel.append(("../../" + dirname, L)) installed_data_files = [ ["../../doc", ['doc/*.txt']], ["../../import", ['import/*.zexp']], ["../../bin", ['utilities/README.txt']], ] os.path.walk("skel", skel_visit, installed_data_files) setup( name='Zope', author=AUTHOR, data_files=installed_data_files, scripts=["utilities/mkzeoinstance.py", "utilities/mkzopeinstance.py", "utilities/check_catalog.py", "utilities/load_site.py", "utilities/requestprofiler.py", "utilities/zpasswd.py", "utilities/copyzopeskel.py", "utilities/reindex_catalog.py", "utilities/compilezpy.py", "utilities/decompilezpy.py", "utilities/ZODBTools/timeout.py", "utilities/ZODBTools/analyze.py", "utilities/ZODBTools/checkbtrees.py", "utilities/ZODBTools/fsdump.py", "utilities/ZODBTools/fsrefs.py" , "utilities/ZODBTools/fstail.py", "utilities/ZODBTools/fstest.py", "utilities/ZODBTools/migrate.py", "utilities/ZODBTools/netspace.py", "utilities/ZODBTools/parsezeolog.py", "utilities/ZODBTools/repozo.py", "utilities/ZODBTools/space.py", "utilities/ZODBTools/timeout.py", "utilities/ZODBTools/zeopack.py", "utilities/ZODBTools/zeoqueue.py", "utilities/ZODBTools/zeoreplay.py", "utilities/ZODBTools/zeoserverlog.py", "utilities/ZODBTools/zeoup.py", "utilities/ZODBTools/zodbload.py", "test.py"], distclass=ZopeDistribution, ) Zope-2.8.7-final/README.txt0000664000076400007640000000252510435500314014235 0ustar ajungajungWelcome to The Zope Source Release Zope is an open-source web application server. This document provides some general information about the Zope source release and provides links to other documents. Installation information can be found in "doc/INSTALL.txt". Other documentation is also in the "doc" directory and in the Zope.org documentation section at http://www.zope.org/Documentation/. General Zope information is available at http://www.zope.org/ Report problems with this release on the Zope mailing list (zope@zope.org) To subscribe to the list send mail to zope-request@zope.org with "subscribe" in the subject line. Introduction The source release is intended for tinkerers, those who want to use Zope components separately, people who want to use their own Python, and people who work on platforms that are not supported by a binary distribution. License The Zope License is included in "doc/LICENSE.txt" Send your feedback about the license to zope-license@zope.org. Installing Zope Follow the instructions in "doc/INSTALL.txt" to install Zope. Reporting bugs Bugs reports should be made through the Zope bugtracker at http://collector.zope.org/Zope. A bug report should contain detailed information about how to reproduce the bug, error messages (see /error_log within the ZMI or var/event.log) Zope-2.8.7-final/Extensions/0000775000076400007640000000000010435500315014673 5ustar ajungajungZope-2.8.7-final/Extensions/README.txt0000664000076400007640000000011310435500315016364 0ustar ajungajungPlace files containing code for External Method objects in this directory. Zope-2.8.7-final/inst/0000775000076400007640000000000010435610645013521 5ustar ajungajungZope-2.8.7-final/inst/compilezpy.py0000664000076400007640000000376010435500331016262 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import compileall, os, sys class Shutup: def write(*args): pass # :) class NoteErr: wrote = 0 def write(self, *args): self.wrote = 1 apply(stderr.write, args) def compile_non_test(dir): """Byte-compile all modules except those in test directories.""" success = compileall.compile_dir(dir, maxlevels=0) try: names = os.listdir(dir) except os.error: print "Can't list", dir names = [] names.sort() for name in names: fullname = os.path.join(dir, name) if (name != os.curdir and name != os.pardir and os.path.isdir(fullname) and not os.path.islink(fullname) and name != 'test' and name != 'tests' and name != 'skins'): success = success and compile_non_test(fullname) return success print print '-'*78 print 'Compiling python modules' stdout = sys.stdout stderr = sys.stderr try: try: success = 0 sys.stdout = Shutup() sys.stderr = NoteErr() success = compile_non_test(os.getcwd()) finally: success = success and not sys.stderr.wrote sys.stdout = stdout sys.stderr = stderr except: success = 0 import traceback traceback.print_exc() if not success: print print '!' * 78 print 'There were errors during Python module compilation.' print '!' * 78 print sys.exit(1) Zope-2.8.7-final/inst/install.py0000664000076400007640000001355410435500331015537 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Generic file and directory installer. Typically called when installing Zope via the Makefile written by 'configure.py' in the 'make install' step. """ import getopt import os import re import shutil import stat import sys # RE that, if a pathname's base name matches, causes it to be ignored # when copying that file or directory. default_omitpattern = r'(\..*|CVS|.*~)$' def main(src, dst, dirmode=0755, fmode=0644, omitpattern=default_omitpattern, retain_xbit=1): """ Copy a file or directory named by src to a file or directory named by dst, normalizing mode bit settings as necessary. Recursively copies directory trees using shutil.copy2(). Errors are reported to standard output. - 'dirmode' is the directory creation mode. All directories are created with this mode. - 'fmode' is the default file creation mode. This mode is modified by the status of the source file. If the source file is executable, mod the fmode to be +wgo executable. - omitpattern is a Python-style regex pattern. If a file or directory name matches this pattern, it will never be copied. - if the dst directory already exists, don't raise an error. """ try: if os.path.isdir(src): copydir(src, dst, dirmode, fmode, omitpattern, retain_xbit) else: names = omit([src], omitpattern) names and copyfile(names[0], dst, fmode, retain_xbit) except (IOError, os.error), why: print "Can't copy %s to %s: %s" % (`src`, `dst`, str(why)) def copydir(src, dst, dirmode, fmode, omitpattern, retain_xbit): names = omit(os.listdir(src), omitpattern) try: # always create directories with dirmode os.makedirs(dst, dirmode) except os.error, why: if why[0] == 17: # directory already exists pass else: raise for name in omit(names, omitpattern): srcname = os.path.join(src, name) dstname = os.path.join(dst, name) if os.path.isdir(srcname): copydir(srcname, dstname, dirmode,fmode,omitpattern, retain_xbit) else: copyfile(srcname, dstname, fmode, retain_xbit) def copylink(src, dst): linkto = os.readlink(src) os.symlink(linkto, dst) def copyfile(src, dst, mode, retain_xbit): shutil.copy2(src, dst) # change dest file mode to fmode but # make +wgo executable if source file is executable dstmode = mode st = os.stat(src) srcmode = st[stat.ST_MODE] if retain_xbit and (srcmode & stat.S_IEXEC): dstmode = (mode | 0111) if os.path.isdir(dst): # if dst is a directory, copy the file in to it os.chmod(os.path.join(dst, os.path.split(src)[-1]), dstmode) else: os.chmod(dst, dstmode) omitcache = {} def omit(names, omitpattern): return [ n for n in names if not re.match(omitpattern, os.path.basename(n)) ] def usage(): print "%s [opts] source dest" % sys.argv[0] print print "Copies a file or directory specified by 'source' to 'dest'" print "normalizing mode bit settings as necessary." print print "If src is a file and dst is a directory, the file will be" print "copied into the dst directory. However, if src is a directory" print "and dst is a directory, the contents of src will be copied into" print "dst." print print "opts: --dirmode=mode --fmode=mode --omitpattern=patt" print print " --dontcopyxbit when copying a file marked as executable," print " don't make the copy executable." print " --dirmode mode bit settings of dest dirs (e.g. '755')" print " --fmode mode bit settings of dest files (e.g. '644')" print " (modified wgo+x when dontcopyxbit is not" print " specified)" print " --omitpattern a Python-style regex pattern. File and" print " directory names which match this pattern will " print " not be copied. The default omitpattern is" print " '%s'" % default_omitpattern if __name__ == '__main__': if len(sys.argv) < 3: print "too few arguments" usage() sys.exit(2) dirmode = 0755 fmode = 0644 omitpattern = default_omitpattern retain_xbit = 1 longopts = ['dirmode=', 'fmode=', 'omitpattern=', 'help', 'copyxmode' ] try: opts, args = getopt.getopt(sys.argv[1:], 'h', longopts) except getopt.GetoptError, v: print v usage() sys.exit(2) try: source, dest = args except: print "wrong number of arguments" usage() sys.exit(2) for o, a in opts: if o in ('-h', '--help'): usage() sys.exit() if o == '--dirmode': if not a.startswith('0'): a = '0%s' % a dirmode = eval(a) if o == '--fmode': if not a.startswith('0'): a = '0%s' % a fmode = eval(a) if o == '--omitpattern': omitpattern = a if o == '--dontcopyxbit': retain_xbit = 0 main(source, dest, dirmode, fmode, omitpattern, retain_xbit) Zope-2.8.7-final/inst/WinBuilders/0000775000076400007640000000000010435500331015736 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/0000775000076400007640000000000010435500331020063 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/0000775000076400007640000000000010435500331021021 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/README.txt0000664000076400007640000000005310435500331022515 0ustar ajungajungThis is an instance home for a ZEO server. Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/etc/0000775000076400007640000000000010435500331021574 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/etc/zeo.conf.in0000664000076400007640000000055410435500331023651 0ustar ajungajung# ZEO configuration file %define INSTANCE_HOME <> address 8100 read-only false invalidation-queue-size 100 # monitor-address PORT # transaction-timeout SECONDS path $INSTANCE_HOME\var\Data.fs level info path $INSTANCE_HOME\log\zeo.log Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/etc/zeoctl.conf.in0000664000076400007640000000124310435500331024350 0ustar ajungajung# zeoctl configuration file %define INSTANCE_HOME <> %define SOFTWARE_HOME <> program $INSTANCE_HOME/bin/runzeo socket-name $INSTANCE_HOME/etc/zeo.zdsock daemon true forever false backoff-limit 10 exit-codes 0, 2 directory $INSTANCE_HOME default-to-interactive true # user zope python <> zdrun $SOFTWARE_HOME\ZEO\runzeo.py # This logfile should match the one in the zeo.conf file. # It is used by zdctl's logtail command, zdrun/zdctl doesn't write it. logfile $INSTANCE_HOME\log\zeo.log level info path $INSTANCE_HOME\log\zeoctl.log Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/bin/0000775000076400007640000000000010435500331021571 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/bin/runzeo.bat.in0000664000076400007640000000046410435500331024214 0ustar ajungajung@set PYTHON=<> @set ZEO_HOME=<> @set SOFTWARE_HOME=%ZEO_HOME%\lib\python @set INSTANCE_HOME=<> @set CONFIG_FILE=%INSTANCE_HOME%\etc\zeo.conf @set PYTHONPATH=<> @set ZEO_RUN=%SOFTWARE_HOME%\ZEO\runzeo.py "%PYTHON%" "%ZEO_RUN%" -C "%CONFIG_FILE%" %1 %2 %3 %4 %5 Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/bin/zeoservice.py.in0000664000076400007640000000761310435500331024735 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ A Windows NT service frontend. Usage: Installation The service should be installed by the Windows installer. You can manually install, uninstall the service from the commandline. ntservice.py [options] install|update|remove|start [...] |stop|restart [...]|debug [...] Options for 'install' and 'update' commands only: --username domain\username : The Username the service is to run under --password password : The password for the username --startup [manual|auto|disabled] : How the service starts, default = manual Commands install : Installs the service update : Updates the service, use this when you change the service class implementation remove : Removes the service start : Starts the service, this can also be done from the services control panel stop : Stops the service, this can also be done from the services control panel restart : Restarts the service debug : Runs the service in debug mode You can view the usage options by running ntservice.py without any arguments. Note: you may have to register the Python service program first, win32\PythonService.exe /register Starting Start the service by clicking the 'start' button in the services control panel. You can set the service to automatically start at boot time by choosing 'Auto' startup by clicking the 'statup' button. Stopping Stop the service by clicking the 'stop' button in the services control panel. Event logging Events are logged to the NT application event log. Use the event viewer to keep track of events. Note: to successfully run this script, the software home needs to be on the PYTHONPATH. """ import os.path from os.path import dirname as dn import sys # these are replacements from mkzeoinstance PYTHON = r'<>' SOFTWARE_HOME=r'<>' INSTANCE_HOME = r'<>' ZEO_HOME = r'<>' ZEO_RUN = r'%s\ZEO\runzeo.py' % SOFTWARE_HOME CONFIG_FILE= os.path.join(INSTANCE_HOME, 'etc', 'zeo.conf') PYTHONSERVICE_EXE=r'%s\bin\PythonService.exe' % ZEO_HOME sys.path.insert(0, SOFTWARE_HOME) # Setup the environment, so sub-processes see these variables os.environ["PYTHONPATH"] = os.environ.get("PYTHONPATH", "") + ";" + SOFTWARE_HOME os.environ["INSTANCE_HOME"] = INSTANCE_HOME from nt_svcutils import service servicename = 'ZEO_%s' % str(hash(INSTANCE_HOME.lower())) class InstanceService(service.Service): start_cmd = '"%s" "%s" -C "%s"' % (PYTHON, ZEO_RUN, CONFIG_FILE) _svc_name_ = servicename _svc_display_name_ = 'ZEO instance at %s' % INSTANCE_HOME # If the exe we expect is not there, let the service framework search # for it. This will be true for people running from source builds and # relying on pre-installed pythonservice.exe. # Note this is only used at install time, not runtime. if os.path.isfile(PYTHONSERVICE_EXE): _exe_name_ = PYTHONSERVICE_EXE process_runner = PYTHON process_args = '"%s" -C "%s"' % (ZOPE_RUN, CONFIG_FILE) evtlog_name = 'ZEO' if __name__ == '__main__': import win32serviceutil win32serviceutil.HandleCommandLine(InstanceService) Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/log/0000775000076400007640000000000010435500331021602 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/log/README.txt0000664000076400007640000000007410435500331023301 0ustar ajungajungThis directory contains logfiles generated by a ZEO server. Zope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/var/0000775000076400007640000000000010435500331021611 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/skel/var/README.txt0000664000076400007640000000016210435500331023306 0ustar ajungajungThis directory contains files related to a ZEO server instance such as database files, lock files, and pid files. Zope-2.8.7-final/inst/WinBuilders/zeo_addons/utilities/0000775000076400007640000000000010435500331022076 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/zeo_addons/utilities/compilezpy.py0000664000076400007640000000400610435500331024643 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import compileall, os, sys class Shutup: def write(*args): pass # :) class NoteErr: wrote = 0 def write(self, *args): self.wrote = 1 apply(stderr.write, args) def compile_non_test(dir): """Byte-compile all modules except those in test directories.""" success = compileall.compile_dir(dir, maxlevels=0) try: names = os.listdir(dir) except os.error: print "Can't list", dir names = [] names.sort() for name in names: fullname = os.path.join(dir, name) if (name != os.curdir and name != os.pardir and os.path.isdir(fullname) and not os.path.islink(fullname) and name != 'test' and name != 'tests' and name != 'skins'): success = success and compile_non_test(fullname) return success print print '-'*78 print 'Compiling python modules' stdout = sys.stdout stderr = sys.stderr try: try: success = 0 sys.stdout = Shutup() sys.stderr = NoteErr() success = compile_non_test(os.getcwd()) finally: success = success and not sys.stderr.wrote sys.stdout = stdout sys.stderr = stderr except: success = 0 import traceback traceback.print_exc() if not success: print print '!' * 78 print 'There were errors during Python module compilation.' print '!' * 78 print sys.exit(1) Zope-2.8.7-final/inst/WinBuilders/zeo_addons/utilities/copyskel.py0000664000076400007640000002214410435500331024304 0ustar ajungajung#! python ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """%(program)s: Create a target directory from a skeleton directory. usage: %(program)s [options] Options: -h/--help -- print this help text -s/--sourcedir -- the skeleton source directory -t/--targetdir -- the directory to which the skeleton files will be copied -u/--uid -- the username/uid of the user who will own the target files -g/--gid -- the groupname/gid of the group who will own the target files -r/--replace -- specify replacement value for .in file This script may be used to install a custom Zope instance home skeleton directory. It is most useful when used to install a skeleton which does not follow the standard 'one-directory-as-instance-home' paradigm used by the stock Zope source install. The value of --targetdir should be the directory where you would like to copy the skeleton hierarchy. For many packagers, this will be "/" or "/usr" or "/usr/local". The value of --sourcedir should be a directory which contains a custom skeleton hierarchy. For many packagers, the skeleton source directory may contain directories like "usr" and "bin" and these directories will contain files and other directories, comprising an installation hierarchy suitable for your platform. The skeleton source hierarchy may contain any kind of file. Files in the skeleton hierarchy that end with a ".in" extension will go through textual substitution before they are placed in the target directory. When they are placed in the target directory, the ".in" extension is removed. Specify textual replacement values by passing one or more --replace= options to the script. The value of each replace option needs to be in the following format: --replace=key:value. 'key' is the value that will be replaced (minus the "<<" and ">>" values expected by the replacement). 'value' is the value that should be used for replacement. Files which do not have an ".in" extension are copied without substitution. All file mode bits from files/dirs in the skeleton source directory are copied along with the file/directory itself. If the --uid and/or --gid flags are used, all directories and files created by this script will be provided with this owner and/or group. Otherwise, the uid and group owner of the files will be the executing user's. Existing directory structures and files are left unchanged. If a file already exists in the target directory, it is left unchanged and the source file is not copied. If a directory already exists in the target directory, its ownership information and mode bit settings are left unchanged. """ import os import shutil import sys import getopt def main(): try: opts, args = getopt.getopt(sys.argv[1:], "hs:t:u:g:r:", ["help", "sourcedir=", "targetdir=", "uid=", "gid=", "replace="] ) except getopt.GetoptError, msg: usage(sys.stderr, msg) sys.exit(2) script = os.path.abspath(sys.argv[0]) sourcedir = None targetdir = None uid = None gid = None replacements = {} for opt, arg in opts: if opt in ("-h", "--help"): usage(sys.stdout) sys.exit() if opt in ("-s", "--sourcedir"): sourcedir = os.path.abspath(os.path.expanduser(arg)) if not sourcedir: usage(sys.stderr, "sourcedir must not be empty") sys.exit(2) if opt in ("-t", "--targetdir"): targetdir = os.path.abspath(os.path.expanduser(arg)) if not targetdir: usage(sys.stderr, "targetdir must not be empty") sys.exit(2) if opt in ("-u", "--uid"): if not arg: usage(sys.stderr, "uid must not be empty") sys.exit(2) try: if os.getuid() != 0: usage(sys.stderr, "You must be root to specify a uid") sys.exit(2) try: uid = int(arg) except: try: import pwd uid = pwd.getpwnam(arg)[2] if not gid: gid = pwd.getpwnam(arg)[3] except KeyError: usage(sys.stderr, "The user indicated by uid does not exist on " "your system") sys.exit(2) except (ImportError, AttributeError): usage(sys.stderr, "Your system does not support the gid or uid options") sys.exit(2) if opt in ("-g", "--gid"): if not arg: usage(sys.stderr, "gid must not be empty") sys.exit(2) try: if os.getuid() != 0: usage(sys.stderr, "You must be root to specify a gid") sys.exit(2) try: gid = int(arg) except: try: import pwd gid = pwd.getpwnam(arg)[3] except KeyError: usage(sys.stderr, "The user indicated by gid does not exist on " "your system") sys.exit(2) except (ImportError, AttributeError): usage(sys.stderr, "Your system does not support the gid or uid options") sys.exit(2) if opt in ("-r", "--replace"): if not arg: continue k, v = arg.split(':', 1) replacements[k] = v if not sourcedir: usage(sys.stderr, "Must specify sourcedir") sys.exit(2) if not targetdir: usage(sys.stderr, "Must specify targetdir") sys.exit(2) copyskel(sourcedir, targetdir, uid, gid, **replacements) def copyskel(sourcedir, targetdir, uid, gid, **replacements): """ This is an independent function because we'd like to import and call it from mkzopeinstance """ # Create the top of the instance: if not os.path.exists(targetdir): os.makedirs(targetdir) # This is fairly ugly. The chdir() makes path manipulation in the # walk() callback a little easier (less magical), so we'll live # with it. pwd = os.getcwd() os.chdir(sourcedir) try: try: os.path.walk(os.curdir, copydir, (targetdir, replacements, uid, gid)) finally: os.chdir(pwd) except (IOError, OSError), msg: print >>sys.stderr, msg sys.exit(1) CVS = os.path.normcase("CVS") def copydir((targetdir, replacements, uid, gid), sourcedir, names): # Don't recurse into CVS directories: for name in names[:]: if os.path.normcase(name) == CVS: names.remove(name) elif os.path.isfile(os.path.join(sourcedir, name)): # Copy the file: sn, ext = os.path.splitext(name) if os.path.normcase(ext) == ".in": dst = os.path.join(targetdir, sourcedir, sn) if os.path.exists(dst): continue copyin(os.path.join(sourcedir, name), dst, replacements, uid, gid) if uid is not None: os.chown(dst, uid, gid) else: src = os.path.join(sourcedir, name) dst = os.path.join(targetdir, src) if os.path.exists(dst): continue shutil.copyfile(src, dst) shutil.copymode(src, dst) if uid is not None: os.chown(dst, uid, gid) else: # Directory: dn = os.path.join(targetdir, sourcedir, name) if not os.path.exists(dn): os.mkdir(dn) shutil.copymode(os.path.join(sourcedir, name), dn) if uid is not None: os.chown(dn, uid, gid) def copyin(src, dst, replacements, uid, gid): ifp = open(src) text = ifp.read() ifp.close() for k in replacements: text = text.replace("<<%s>>" % k, replacements[k]) ofp = open(dst, "w") ofp.write(text) ofp.close() shutil.copymode(src, dst) if uid is not None: os.chown(dst, uid, gid) def usage(stream, msg=None): if msg: print >>stream, msg print >>stream program = os.path.basename(sys.argv[0]) print >>stream, __doc__ % {"program": program} if __name__ == '__main__': main() Zope-2.8.7-final/inst/WinBuilders/zeo_addons/utilities/mkzeoinstance.py0000664000076400007640000000700110435500331025320 0ustar ajungajung#! python ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """%(program)s: Create a ZEO instance home. usage: %(program)s [options] Options: -h/--help -- print this help text -d/--dir -- the dir in which the instance home should be created -u/--user NAME:PASSWORD -- set the user name and password of the initial user -s/--skelsrc -- the dir from which skeleton files should be copied When run without arguments, this script will ask for the information necessary to create a ZEO instance home. """ import getopt import os import shutil import sys import copyskel def main(): try: opts, args = getopt.getopt(sys.argv[1:], "hu:d:s:", ["help", "user=", "dir=", "skelsrc="] ) except getopt.GetoptError, msg: usage(sys.stderr, msg) sys.exit(2) script = os.path.abspath(sys.argv[0]) user = None password = None skeltarget = None skelsrc = None for opt, arg in opts: if opt in ("-d", "--dir"): skeltarget = os.path.abspath(os.path.expanduser(arg)) if not skeltarget: usage(sys.stderr, "dir must not be empty") sys.exit(2) if opt in ("-s", "--skelsrc"): skelsrc = os.path.abspath(os.path.expanduser(arg)) if not skelsrc: usage(sys.stderr, "skelsrc must not be empty") sys.exit(2) if opt in ("-h", "--help"): usage(sys.stdout) sys.exit() if not skeltarget: # interactively ask for skeltarget and initial user name/passwd. # cant set custom instancehome in interactive mode, we default # to skeltarget. skeltarget = instancehome = os.path.abspath( os.path.expanduser(get_skeltarget()) ) instancehome = skeltarget zeohome = os.path.dirname(os.path.dirname(script)) softwarehome = os.path.join(zeohome, "lib", "python") if skelsrc is None: # default to using stock ZEO skeleton source skelsrc = os.path.join(zeohome, "skel") inituser = os.path.join(instancehome, "inituser") kw = { "PYTHON": sys.executable, "INSTANCE_HOME": instancehome, "SOFTWARE_HOME": softwarehome, "ZEO_HOME": zeohome, } copyskel.copyskel(skelsrc, skeltarget, None, None, **kw) def usage(stream, msg=None): if msg: print >>stream, msg print >>stream program = os.path.basename(sys.argv[0]) print >>stream, __doc__ % {"program": program} def get_skeltarget(): print 'Please choose a directory in which you\'d like to install' print 'ZEO "instance home" files such as database files, configuration' print 'files, etc.' print while 1: skeltarget = raw_input("Directory: ").strip() if skeltarget == '': print 'You must specify a directory' continue else: break return skeltarget if __name__ == "__main__": main() Zope-2.8.7-final/inst/WinBuilders/zeo_addons/utilities/decompilezpy.py0000664000076400007640000000177010435500331025161 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os import sys def main(dirname): os.path.walk(dirname, rmpycs, None) def rmpycs(arg, dirname, names): for name in names: path = os.path.join(dirname, name) if ( name.endswith('.pyc') or name.endswith('.pyo') and os.path.isfile(path) ): os.unlink(path) if __name__ == '__main__': main(sys.argv[1]) Zope-2.8.7-final/inst/WinBuilders/Makefile.python0000664000076400007640000000116710435500331020723 0ustar ajungajung# Zope/ZEO build and install Makefile for Python (gnumake-style). # Puts binaries into build/bin # See README.txt for system requirements. # See python.mk for the input files required in tmp/. # do "make -f build" # then "make -f where type is one of "zope", "python", or "zeo". Alternatively, you can avoid the bash shell completely, and from a Windows command prompt type: bash WinBuilders/buildout Everything should work! * For 'zope', the buildout populates the "build" directory with a Windows executable installer. * For python, the buildout populates the "build" directory with a Python laid out for Zope and/or ZEO. [Tim isn't sure this target has ever been tested/used.] * For 'zeo', the buildout populates the "build" directory with a Windows executable installer. [Tim isn't sure this target has ever been tested/used.] An 'inst\src' directory is also created. The makefiles don't even pretend to do a good job of keeping track of dependencies; best practice is to blow away the 'build' and 'src' directories between runs. Note that the 'tmp' directory should _not_ be deleted -- it's purely an input to this process. If the build fails: If the Windows drive you are working on is not C: (or Inno isn't installed there!), try executing the following: WinBuilders/buildout CYGROOT=/cygdrive/{your_drive_letter} If you see errors relating to MSVC not being installed, or the build process failing to find MSVC, it may be necessary to bring up the MSVC gui at least once (MSDev doesn't finish writing all the registry keys it should until the GUI is first launched). If may also be necessary to run VCVARS.bat to set up the VC++ environment (but generally is not.) See below for Win98SE. If you see any make errors with references to any of the files required in tmp/ (see 'Setup Environment' above), it's because a later version is now required, or the files you've downloaded are not in 'tmp'. Testing Zope ------------ The test suite can be run from inst\build\: - Open a native (not Cygwin) DOS box. We want to test with the Python the Zope installer includes. - cd to inst\build - Copy log.ini from the root of the Zope checkout. This isn't necessary for the tests to pass, but if you don't do it a great many spurious log messages will be displayed on the console, some of which "look like" errors (some of the tests deliberately provoke errors). - Enter bin\python bin\test.py -v --all or whatever variation you like best. All tests should pass. Also run the Windows installer, and play with the Zope it installs. All platform notes ------------------ - Depending on your MSVC installation options, you have to run vcvars32.bat to set up envars for MSVC. Running that from a bash shell doesn't have any effect on the Cygwin PATH. This works: + Open a native DOS box. + Run vcvars32.bat. + Start a bash shell from the same box (== run cygwin.bat, found in the root of your Cygwin installation -- the same thing the Cygwin shell desktop shortcut resolves to, so you can get the exact path by looking at the icon's Properties). Win98SE notes ------------- - Every time a makefile runs xcopy, there's a segfault in kernel32.dll, which hangs the bash shell with an endless succession of error boxes. The only way I found to break out of this was to bring up the debugger, close it, then type Ctrl+C at the hung bash shell. The bash shell appears to be fine at that point, but you can never close it (short of killing it via the task manager). Same thing if xcopy32 is used instead. xcopy works OK directly from a bash shell. The segfaults occur if it's run via a makefile, or via a shell script. Guessing a problem with I/O redirection, since some other apps can't see keyboard input before the hung stuff is killed. Workaround: xxcopy works fine ; free for personal use, but not for commercial use. Rename it to xcopy.exe and get it into your path before the native xcopy, or fiddle the XCOPY defn in common.mk to use xxcopy instead of xcopy. Zope-2.8.7-final/inst/WinBuilders/etc/0000775000076400007640000000000010435500331016511 5ustar ajungajungZope-2.8.7-final/inst/WinBuilders/etc/zeo.iss.in0000664000076400007640000001421210435500331020433 0ustar ajungajung[Setup] AppName=ZEO <> AppVerName=ZEO <> AppPublisher=Zope Corporation AppPublisherURL=http://www.zope.org AppSupportURL=http://www.zope.org AppUpdatesURL=http://www.zope.org DefaultDirName={pf}\ZEO-<> DefaultGroupName=ZEO <> OutputBaseFilename=ZEO-<>-win32 WizardImageFile=<>\etc\zlogo_left.bmp WizardSmallImageFile=<>\etc\zlogo_top.bmp ; Starting w/ Inno 4.1.3, Inno decided to stretch the .bmp files in various ; ways. Hard to know why, but it looks terrible on my pretty vanilla box. ; Luckily, 4.1.3 also added WizardImageStretch to turn that off. WizardImageStretch=no SourceDir=. OutputDir=. [Components] Name: main; Description: "Main ZEO files"; Types: full compact custom; Flags: fixed Name: instance; Description: "ZEO 'instance home' files (required to run ZEO immediately)"; Types: full compact custom; ExtraDiskSpaceRequired: 1453056 [Tasks] Name: service; Description: "Run your ZEO instance as a Windows service (start ZEO automatically at system startup)"; MinVersion: 4.0,4.0; Components: instance; Check: IsAdministrator [Files] ; Main ZEO files Source:"bin\*.*"; DestDir: "{app}\bin"; Flags: ignoreversion recursesubdirs Source:"doc\*.*"; DestDir: "{app}\doc"; Flags: ignoreversion recursesubdirs Source:"lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion recursesubdirs Source:"skel\*.*"; DestDir: "{app}\skel"; Flags: ignoreversion recursesubdirs ; these are required to be put into the bin directory for proper function of NT services Source:"bin\Lib\site-packages\PythonService.exe"; DestDir: "{app}\bin"; Flags: ignoreversion Source:"bin\Lib\site-packages\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion [Icons] Name: "{group}\Run ZEO In Console"; Filename: "{code:GetDataDir}\bin\runzeo.bat"; Check: DoInstanceHome Name: "{group}\Uninstall ZEO"; Filename: "{uninstallexe}"; Name: "{group}\Make ZEO Instance"; Filename: "{app}\bin\python.exe"; Parameters: """{app}\bin\mkzeoinstance.py""" [Run] Filename: "{app}\bin\pythonw.exe"; Parameters: """{app}\bin\compilezpy.py"" ""{app}"" "; Description: "Byte-compiling Python files"; StatusMsg: "Byte-compiling Python files..." Filename:"{app}\bin\pythonw.exe"; Parameters: """{app}\bin\mkzeoinstance.py"" ""--dir={code:GetDataDir}"" "; Description: "Creating default instance"; StatusMsg: "Creating default instance..."; Check: DoInstanceHome Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zeoservice.py"" --startup auto install"; Check: DoService Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zeoservice.py"" start"; Description: "Launch ZEO"; Flags: nowait postinstall skipifsilent; Check: DoService Filename: "{code:GetDataDir}\bin\runzeo.bat"; Description: "Launch ZEO"; Flags: nowait postinstall skipifsilent; Check: DontDoService [UninstallRun] Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zeoservice.py"" stop"; Check: DoService Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zeoservice.py"" remove"; Check: DoService Filename: "{app}\bin\pythonw.exe"; Parameters: """{app}\bin\decompilezpy.py"" ""{app}""" [Code] var PasswordPrompts, PasswordValues : array of String; PasswordChars : array of char; DataDirValues: array of String; Password : string; DataDir : String; function InitializeSetup(): Boolean; begin { set up password data structures } SetArrayLength(PasswordPrompts, 1); PasswordPrompts[0] := 'Password:'; SetArrayLength(PasswordValues, 1); PasswordValues[0] := ''; SetArrayLength(PasswordChars, 1); PasswordChars[0] := '*'; Password := ''; { set up data dir data structures } SetArrayLength(DataDirValues, 1); DataDir := ''; Result := True; end; function CollectInstanceDir(): Boolean; var Next: Boolean; DirOk: Boolean; begin DirOk := True; ScriptDlgPageSetSubCaption1('Select where ZEO instance files should be installed'); ScriptDlgPageSetSubCaption2('Select the folder to which you would like Setup to install ZEO "instance" files, then click Next.'); if DataDir = '' then DataDir:= 'C:\ZEO-Instance'; if DataDirValues[0] <> '' then DataDirValues[0]:= ''; { Ask for a dir until the user has approved one or clicked Back or Cancel } Next:= InputDir(DataDirValues[0], DataDir); if Next and FileOrDirExists(DataDir) then DirOk := False; while Next and not DirOk do begin if DataDir = '' then begin DirOk := False; MsgBox(SetupMessage(msgInvalidPath), mbError, MB_OK); end; if FileOrDirExists(DataDir) then begin DirOk := MsgBox('Directory Exists' #13#13 'The directory ' + DataDir + ' already exists. Would you like to create instance files in that directory anyway?', mbConfirmation, MB_YESNO) = idYes; end; if not DirOk then Next := InputDir(DataDirValues[0], DataDir); end; Result:=Next; end; function DoInstanceHome():Boolean; var S : String; begin S := WizardSelectedComponents(False); Result := Pos('instance', S) <> 0; end; function DoService(): Boolean; var S : String; begin S := WizardSelectedTasks(False); Result := Pos('service', S) <> 0; end; function DontDoService(): Boolean; begin Result := not DoService(); end; function ScriptDlgPages(CurPage: Integer; BackClicked: Boolean): Boolean; var Next : Boolean; CurSubPage : Integer; begin Next:=True; if ( (not BackClicked and (CurPage = wpSelectTasks)) or (BackClicked and (CurPage = wpReady)) ) and DoInstanceHome() then begin ScriptDlgPageOpen(); ScriptDlgPageSetCaption('Instance Setup'); Next:=CollectInstanceDir(); if not BackClicked then Result := Next else Result := not Next; ScriptDlgPageClose(not Result); end; Result:=Next; end; function NextButtonClick(CurPage: Integer): Boolean; begin Result := ScriptDlgPages(CurPage, False); end; function BackButtonClick(CurPage: Integer): Boolean; begin Result := ScriptDlgPages(CurPage, True); end; function GetPassword(Default: String): String; begin Result := Password; end; function GetDataDir(Default : String):String; begin Result := DataDir; end; { GetInstanceDir } function IsAdministrator(): Boolean; begin Result := IsAdminLoggedOn(); end; Zope-2.8.7-final/inst/WinBuilders/etc/sitecustomize.py0000664000076400007640000000046110435500331021773 0ustar ajungajung""" Add Zope packages in Windows binary distro to sys.path automagically """ import sys import os try: sp = __file__ except: sp = None if sp: dn = os.path.dirname swhome = os.path.join(dn(dn(dn(dn(sp)))), 'lib', 'python') if os.path.exists(swhome): sys.path.insert(0, swhome) Zope-2.8.7-final/inst/WinBuilders/etc/zlogo_top.bmp0000664000076400007640000001014610435500331021227 0ustar ajungajungBMf6(2<0 Üi||{ڒI峁]Մ2չttt332```z"̫IIHqiqޜ[%%%ߡcѳُDe齒m rZZZb繊xuܙVԀ,dmZȤ׉;`ƠUUUiii ،?j}'ۖP躍~ؾӶl fֆ7Xj dfxgv#V淇ޟ_cdc_c>>>pppdddONNoϯߠ_f)GGE)Eif~~~~~~~~~~i)QD~~~~g~~g\~~~~<$$f~~~~~<4QGG{f~\~~~fEq~~~\f4$)E"~~~~)D~~~D~~~~"Dg~~D)D\~~D7~~~E"~\"qg~~G~~~~~~)q~~~E~~~Q44GGG$EGEGGGG4"4""'~~~Di~~~~g~g~~~\g~~~~~~~gg~~~g~g~~~~~~~Q&f~~~~~~~g~~~~~~~~~~~~~~~~~~~g~~~~"~~~~~~~~~fiiDiDf7fffD~~> AppVerName=Zope <> AppPublisher=Zope Corporation AppPublisherURL=http://www.zope.org AppSupportURL=http://www.zope.org AppUpdatesURL=http://www.zope.org DefaultDirName={pf}\Zope-<> DefaultGroupName=Zope <> OutputBaseFilename=Zope-<>-win32 WizardImageFile=<>\etc\zlogo_left.bmp WizardSmallImageFile=<>\etc\zlogo_top.bmp ; Starting w/ Inno 4.1.3, Inno decided to stretch the .bmp files in various ; ways. Hard to know why, but it looks terrible on my pretty vanilla box. ; Luckily, 4.1.3 also added WizardImageStretch to turn that off. WizardImageStretch=no SolidCompression=yes SourceDir=. OutputDir=. [Components] Name: main; Description: "Main Zope files"; Types: full compact custom; Flags: fixed Name: instance; Description: "Zope 'instance home' files (required to run Zope immediately)"; Types: full compact custom; ExtraDiskSpaceRequired: 1453056 [Tasks] Name: service; Description: "Run your Zope instance as a Windows service (start Zope automatically at system startup)"; MinVersion: 4.0,4.0; Components: instance; Check: IsAdministrator [Files] ; Main Zope files Source: "<>\etc\README.html"; DestDir: "{app}"; Flags: ignoreversion Source:"bin\*.*"; DestDir: "{app}\bin"; Flags: ignoreversion recursesubdirs Source:"doc\*.*"; DestDir: "{app}\doc"; Flags: ignoreversion recursesubdirs Source:"import\*.*"; DestDir: "{app}\import"; Flags: ignoreversion recursesubdirs Source:"lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion recursesubdirs Source:"skel\*.*"; DestDir: "{app}\skel"; Flags: ignoreversion recursesubdirs ; these are required to be put into the bin directory for proper function of NT services Source:"bin\Lib\site-packages\win32\PythonService.exe"; DestDir: "{app}\bin"; Flags: ignoreversion Source:"bin\Lib\site-packages\pywin32_system32\PyWinTypes23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion Source:"bin\Lib\site-packages\pywin32_system32\PythonCOM23.dll"; DestDir: "{app}\bin"; Flags: ignoreversion ; This is a helper module for manging registry entries at uninstall time Source: "<>\bin\fixreg.py"; DestDir: "{app}\bin"; Flags: ignoreversion [Icons] Name: "{group}\Run Zope In Console"; Filename: "{code:GetDataDir}\bin\runzope.bat"; Check: DoInstanceHome Name: "{group}\Uninstall Zope"; Filename: "{uninstallexe}"; Name: "{group}\Make Zope Instance"; Filename: "{app}\bin\python.exe"; Parameters: """{app}\bin\mkzopeinstance.py""" [Run] Filename: "{app}\bin\pythonw.exe"; Parameters: """{app}\bin\compilezpy.py"" ""{app}"" "; Description: "Byte-compiling Python files"; StatusMsg: "Byte-compiling Python files..." Filename:"{app}\bin\pythonw.exe"; Parameters: """{app}\bin\mkzopeinstance.py"" --user=""admin:{code:GetPassword}"" --dir=""{code:GetDataDir}"" "; Description: "Creating default instance"; StatusMsg: "Creating default instance..."; Check: DoInstanceHome Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zopeservice.py"" --startup auto install"; Check: DoService Filename: "{app}\README.html"; Flags: shellexec skipifdoesntexist postinstall skipifsilent; Description: "View 'QuickStart' document in a web browser" Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zopeservice.py"" start"; Description: "Launch Zope"; Flags: nowait postinstall skipifsilent; Check: DoService Filename: "{code:GetDataDir}\bin\runzope.bat"; Description: "Launch Zope"; Flags: nowait postinstall skipifsilent; Check: DontDoService [UninstallRun] Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zopeservice.py"" --wait=30 stop"; Check: DoService Filename: "{app}\bin\pythonw.exe"; Parameters: """{code:GetDataDir}\bin\zopeservice.py"" remove"; Check: DoService Filename: "{app}\bin\pythonw.exe"; Parameters: """{app}\bin\decompilezpy.py"" ""{app}""" Filename: "{app}\bin\pythonw.exe"; Parameters: """{app}\bin\fixreg.py"" ""<>""" [Registry] Root: HKLM; Subkey: "Software\Zope Corporation\Zope"; ValueType: string; ValueName: "PreviousVersion"; ValueData: "{reg:HKLM\Software\Zope Corporation\Zope,CurrentVersion|}" Root: HKLM; Subkey: "Software\Zope Corporation\Zope"; Flags: uninsdeletekeyifempty; ValueType: string; ValueName: "CurrentVersion"; ValueData: "<>" Root: HKLM; Subkey: "Software\Zope Corporation\Zope\<>"; Flags: uninsdeletekey; ValueType: string; ValueName: "Home"; ValueData: "{app}" Root: HKLM; Subkey: "Software\Zope Corporation\Zope\<>"; Flags: uninsdeletekey; ValueType: string; ValueName: "Python"; ValueData: "{app}\bin\python.exe" Root: HKLM; Subkey: "Software\Zope Corporation\Zope\<>"; Flags: uninsdeletekey; ValueType: string; ValueName: "Pythonw"; ValueData: "{app}\bin\pythonw.exe" [Code] var PasswordPrompts, PasswordValues : array of String; PasswordChars : array of char; DataDirValues: array of String; Password : string; DataDir : String; function InitializeSetup(): Boolean; begin { set up password data structures } SetArrayLength(PasswordPrompts, 1); PasswordPrompts[0] := 'Password:'; SetArrayLength(PasswordValues, 1); PasswordValues[0] := ''; SetArrayLength(PasswordChars, 1); PasswordChars[0] := '*'; Password := ''; { set up data dir data structures } SetArrayLength(DataDirValues, 1); DataDir := ''; Result := True; end; function CollectInstanceDir(): Boolean; var Next: Boolean; DirOk: Boolean; begin DirOk := True; ScriptDlgPageSetSubCaption1('Select where Zope instance files should be installed'); ScriptDlgPageSetSubCaption2('Select the folder to which you would like Setup to install Zope "instance" files, then click Next.'); if DataDir = '' then DataDir:= 'C:\Zope-Instance'; if DataDirValues[0] <> '' then DataDirValues[0]:= ''; { Ask for a dir until the user has approved one or clicked Back or Cancel } Next:= InputDir(False, DataDirValues[0], DataDir); if Next and FileOrDirExists(DataDir) then DirOk := False; while Next and not DirOk do begin if DataDir = '' then begin DirOk := False; MsgBox(SetupMessage(msgInvalidPath), mbError, MB_OK); end; if FileOrDirExists(DataDir) then begin DirOk := MsgBox('Directory Exists' #13#13 'The directory ' + DataDir + ' already exists. Would you like to create instance files in that directory anyway?', mbConfirmation, MB_YESNO) = idYes; end; if not DirOk then Next := InputDir(False, DataDirValues[0], DataDir); end; Result:=Next; end; function CollectPassword(): Boolean; var Next: Boolean; begin ScriptDlgPageSetSubCaption1('Specify administrator password'); ScriptDlgPageSetSubCaption2('The login name for your Zope administrator account is "admin". When you first connect to the Zope management interface, you will need to login using the "admin" username and the password you specify below.'); Next := InputQueryArrayEx(PasswordPrompts, PasswordChars, PasswordValues); while Next and (PasswordValues[0] = '') do begin MsgBox('You must enter an administrator password', mbError, MB_OK) Next := InputQueryArrayEx(PasswordPrompts, PasswordChars, PasswordValues); end; Password := PasswordValues[0]; Result:=Next; end; function DoInstanceHome():Boolean; var S : String; begin S := WizardSelectedComponents(False); Result := Pos('instance', S) <> 0; end; function DoService(): Boolean; var S : String; begin S := WizardSelectedTasks(False); Result := Pos('service', S) <> 0; end; function DontDoService(): Boolean; begin Result := not DoService(); end; function ScriptDlgPages(CurPage: Integer; BackClicked: Boolean): Boolean; var Next : Boolean; CurSubPage : Integer; begin Next:=True; if ( (not BackClicked and (CurPage = wpSelectTasks)) or (BackClicked and (CurPage = wpReady)) ) and DoInstanceHome() then begin if not BackClicked then CurSubPage:=0 else CurSubPage:=1; ScriptDlgPageOpen(); ScriptDlgPageSetCaption('Instance Setup'); while (CurSubPage >=0) and (CurSubPage <=1) and not Terminated do begin case CurSubPage of 0: Next:=CollectInstanceDir(); 1: Next:=CollectPassword(); end; if Next then CurSubPage := CurSubPage +1 else CurSubPage := CurSubPage -1; end; if not BackClicked then Result := Next else Result := not Next; ScriptDlgPageClose(not Result); end; Result:=Next; end; function NextButtonClick(CurPage: Integer): Boolean; begin Result := ScriptDlgPages(CurPage, False); end; function BackButtonClick(CurPage: Integer): Boolean; begin Result := ScriptDlgPages(CurPage, True); end; function GetPassword(Default: String): String; begin Result := Password; end; function GetDataDir(Default : String):String; begin Result := DataDir; end; { GetInstanceDir } function IsAdministrator(): Boolean; begin Result := IsAdminLoggedOn(); end; Zope-2.8.7-final/inst/WinBuilders/etc/zlogo_left.bmp0000664000076400007640000004746610435500331021376 0ustar ajungajungBM6O6(}K^^^ѳkՄ2666}ݜ\ֻlkkޡbhuz"溌EEE轑؎CǢܙVsQQQpytttȥn ͬۖQْJ涆qԁ-ڕNŞ)ߣh,,,ʨ׉;###v崃u|{zؿkxʩ岀،?淈l JJJLIFِGړKWVTgoaa`˪ϰӶֆ6@>=㲁㴃Ը̬}f}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}t|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||t}}||}}||}}||}}||}}||}}|l*1 |eh 2;2|}}|}}}}}}}}}}}}}}}}}}}}}}}}}}&fi}}bt&b}"}}}}}}}}}}}}}}}}};;|}}|2"}}}}}}}}}}}}}}}}}}}}}}}}}2)b}b*2yD}}ts}}}}}}HHHH"}}}}}}*;;|}}|;b}}}} e;| &HknH}}D:b}b|}}}}}}2 e yy |}}|)}}}}n"}}`Ab}}`}}}}}}|}}|}}}}|f"}}}}}`}}}}}}|}}|fb}}}b.t}}"f)}}}1}}}}}}|}}|}}}}}}}:2"}}bf}}}}}}|}}|`}}}}:D}}"21}}}}}}}}}|}}|2H}}}b.}}}A"}}b }}}}}}|}}|s}}}}}}}}}}z}}}}}}|}}|:}}}}|H}}} *}}}t}}}}}}|}}|}}}b2}}}}l}}}b}}}}}}|}}|f"}}}DAe}}}Hb}}} }}}2 A}}}|}}|}}}})}}}HH}}}}}}"}}}}"ti}}}|}}| }}}}p}}}HH}}}}}}&t"}}}D}}}D"} |}}|2H}}}"2}}}bH}}}}}}`}}}t2}}}}}}}}}}}}} |}}|b}}}` }}}}2b}}}e}}}b}}H2}}} eA|}}|:}}}}yb}}}eA}}}}2}}}}}}k}}}|}}|[}}}bn[}}}e}}}D}}}}}b }}}|}}|f"}}}:}}}[}}})}}}}}}}}}|}}|*}}}})2b}}b2[}}}}}}e}}} }}}|}}| }}}}p }}}i b}}t}}} }}}}}}|}}|2D}}}"22"}}"2}}};}}}e}}}}}}|}}|}}}}\*}}}`*}}})}}}}}}1}}}|}}||}}}}:1}}}^"}}`}}}}}}}}}|}}|}}}bb}}|*b}}k}}})}}}t}}}|}}|.b}}}H2H}}; }}"}}}2*\}}}Hf}}}| &D |}}|:}}}}\A1"}}D)*e ;1[}}" 2}}}HHHHHb}}}}H|2}}}}}}}}}}}}}}A|}}|}}}}:A"}}}}}}}H2iO*.il|}}|fe .b}}}b^2e A|}}|}}}bH ||||||D}}}}"f|}}|}}}}}}}}}}}}}}}}}}}}}}}}[|}}|)|}}||}}||}}||}}||}}|GXPE'_ 3 J0o?|}}|P%/ ~~~~~~~~~~~~~~~~ /90,|}}|Mr ~~~~~~~~~~~~~~~~~~~~~~~~~aZ6U|}}|7'/~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ rx5|}}|A-a~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%K$|}}|0~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~B|}}|q~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|}}|, Z~~~~~~~~~~~~~~~~8F 'MMMYC3~~~~~~~~~~~~~~~~ar|}}|>~~~~~~~~~~~~~~Z<6XBcV-C~~~~~~~~~~~~~~Q( |}}|A ~~~~~~~~~~~~Qo cda~~~~~~~~~~~~FLc|}}|Xr~~~~~~~~~~~aRBWQ~~~~~~~~~~~ZJc|}}|?8~~~~~~~~~~~9!#Z~~~~~~~~~~ A|}}|8~~~~~~~~~~%J M ~~~~~~~~~~~3 |}}| ~~~~~~~~~QNUdS~~~~~~~~~~3 |}}|Ta~~~~~~~~~>P~~~~~~~~~~<|}}|cK~~~~~~~~~a Acx%~~~~~~~~~<7|}}|~~~~~~~~~Z!>~~~~~~~~~# |}}|?j~~~~~~~~B ~~~~~~~~~C |}}|{%~~~~~~~~~~~~~~~~~|}}|W~~~~~~~~r$'~~~~~~~~QL|}}|N~~~~~~~~U7'~~~~~~~~R |}}|oa~~~~~~~%d~~~~~~~~#|}}|,~~~~~~~Z r~~~~~~~~m|}}|c#~~~~~~~Qmc BU~~~~~~~%V|}}|Q~~~~~~~8]__<# 'TEPM+UgggAggh{?P=u#<__] r9R%8~~~~~~~~<|}}|r~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~aQ%R/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Q|}}|o~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~]|}}|7%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~aP|}}|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7|}}|A8~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~P|}}|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ c|}}|7>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~aQ8 ~~~~~~Z,|}}|?a~~~~~~~~~~~~~~~~~~~~~~Q ' 9R8/Q ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Z%9 'EP{UhB~~~~~~~6|}}|~~~~~~~~~~~~~~~~~~~~~~~~ZEc7777 5VMqo0E-JKT(((((((6=E0PG 777$8~~~~~~|}}|5~~~~~~~~~~~~~~~~~~~~~~~~~~K7C~~~~~~~|}}|w~~~~~~~~~~~~~~~~~~~~~~~~~~~~ a~~~~~~|}}|C~~~~~~ 9~~~~~~~~~~~~~~~~~~~~~F,%~~~~~~9|}}|~~~~~~>o ~~~~~~~~~~~~~~~~~~~~~LN~~~~~~Z,|}}|V~~~~~~~ dW~~~~~~~~~~~~~~~~~~~~~~%P~~~~~~~!|}}|@~~~~~~~xL ~~~~~~~~~~~~~~~~~~~~~~  ~~~~~~|}}|N~~~~~~/{I~~~~~~~~~~~~~~~~~~~~~~~~ c<~~~~~~9|}}| ~~~~~~ >~~~~~~~~~~~~~~~~~~~~~~~~F$(~~~~~~%|}}|d~~~~~~3{j~~~~~~~~~~~~~~~~~~~~~~~~RMm~~~~~~ZX|}}|A ~~~~~~J!a~~~~~~~~~~~~~~~~~~~~~~~~j~~~~~~~q|}}|X~~~~~~amd'~~~~~~~~~~~~~~~~~~~~~~~~~ Ch ~~~~~~J|}}|q~~~~~~ d~~~~~~~~~~~~~~~~~~~~~~~~~~]+~~~~~~|}}|=~~~~~~/V]~~~~~~~~~~~~~~~~~~~~~~~~~~?<~~~~~~ |}}|~~~~~~/$~~~~~~~~~~~~~~~~~~~~~~~~~~Qd3~~~~~~3|}}|K~~~~~~8,>~~~~~~~~~~~~~~~~~~~~~~~~~~~ $#~~~~~~ |}}|T~~~~~~8+9~~~~~~~~~~~~~~~~~~~~~~~~~~~#~~~~~~ |}}|T~~~~~~8,~~~~~~~~~~~~~~~~~~~~~~~~~~~jJ#~~~~~~ |}}|T~~~~~~8$W~~~~~~~~~~~~~~~~~~~~~~~~~~~a#~~~~~~ |}}|T~~~~~~j 3~~~~~~~~~~~~~~~~~~~~~~~~~~~~3~~~~~~_|}}|6~~~~~~/V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~QYA<~~~~~~|}}|~~~~~~ ?I ~~~~~~~~~~~~~~~~~~~~~~~~~~~~]+~~~~~~'|}}|~~~~~~a0JZ~~~~~~~~~~~~~~~~~~~~~~~~~~~~%=h ~~~~~~|}}|+~~~~~~~@q%~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #V~~~~~~~x|}}|7 ~~~~~~#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Mo~~~~~~ |}}|d~~~~~~3a~~~~~~~~~~~~~~~~~~~~~~~~~~~~~' ~~~~~~%U|}}| ~~~~~~8+(Q~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>Pc~~~~~~>c|}}|6~~~~~~~o8~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Q'cQ~~~~~~#|}}|~~~~~~~'r~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~9P~~~~~~~@|}}|j~~~~~~rd ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ZNT~~~~~~ X|}}|c3~~~~~~ dmQ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>M%~~~~~~Rc|}}|@~~~~~~~{9~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~aMa~~~~~~'|}}|GZ~~~~~~>5d'Q~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~0~~~~~~~|}}|d<~~~~~~~qw~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8~~~~~~%d|}}|x~~~~~~~]c$ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~j6~~~~~~~|}}|~~~~~~aLm~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~R-~~~~~~ V|}}|(~~~~~~~#B ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ F~~~~~~~ c|}}|+8~~~~~~~w~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ?d$~~~~~~~!|}}|N~~~~~~~FdIZ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~_wAc~~~~~~~>5|}}|$j~~~~~~a@r~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>'/~~~~~~~=|}}|~~~~~~~%, J%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>A|}}|~~~~~~~~~~~~~~~~~V|}}|6 ~~~~~~~~~'K ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~9'(~~~~~~~~~RV|}}|c6a~~~~~~~~=7Y#R~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~j<( ]~~~~~~~~~r|}}|da~~~~~~~~a L!I<Q~~~~~~~~~~~~~~~~~~~~~~~~%'6L$d'Z~~~~~~~~~3V|}}|q8~~~~~~~~~a,ccdAVYTT4>RRR9r TT@ddJ%~~~~~~~~~a |}}|~~~~~~~~~~ c~~~~~~~~~~Z7|}}|_a~~~~~~~~~~a4!$7?%~~~~~~~~~~~M|}}|c/~~~~~~~~~~~~#0+wIr~~~~~~~~~~~~~#+|}}|V_~~~~~~~~~~~~~~8>K!c0@ Ra~~~~~~~~~~~~~jJ |}}|7!~~~~~~~~~~~~~~~~ j%R(((F8Qa~~~~~~~~~~~~~~~~_|}}| Yj~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~rc|}}|$'%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !d|}}|a~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Z_d|}}|  R~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 9v{|}}|C>a~~~~~~~~~~~~~~~~~~~~~~ZF? |}}|!W%~~~~~~~~~~~~~ 4U|}}|cdBLq=KTT6-{Ad|}}||}}||}}t|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||t}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}Zope-2.8.7-final/inst/WinBuilders/etc/README.html0000664000076400007640000000575510435500331020350 0ustar ajungajung Zope Windows Binary Post-Installation QuickStart

Welcome to Zope, a high-performance object-oriented platform for building dynamic Web applications. Here are some quick pointers to get you started:

  • If you've launched your Zope instance, and you'd like to get started right away, you can go directly to the Zope Management Interface. Use the username "admin" and the password you specified during installation process to log in. Please be patient as it may take a few moments for your Zope instance to start before you can connect to it successfully.

  • If you've not started your Zope instance, you can do so by choosing the "Run Zope" command from the Zope program group in your Start Menu. If you installed Zope as a Windows 'service' (NT/2000/XP only), you may use the Control Panel --> Administrative Tools --> Services applet to start your Zope instance as well. The service will be named "Zope instance at <instance home directory name>".

  • If you've not yet created a Zope instance, perform the following steps to do so:

    1. Visit a console window (type "cmd" (NT/2000/XP) or "command" (95/98/ME) in the Run box from the Start Menu).
    2. Change to the "bin" directory directory of your Zope software home using the "cd" command, e.g. "cd c:\Program Files\Zope-2.7\bin".
    3. Type "python.exe mkzopeinstance.py"
    4. Follow the instructions provided by 'mkzopeinstance.py'.

    To start the resulting instance, visit the directory you specified during your interaction with 'mkzopeinstance.py' and type "bin\runzope.bat".

  • For the most complete "starter" documentation, Read The Fine Manual. This document guides you through the whole process of learning Zope, from logging in for the first time to creating your own web applications.

  • For further documentation, go to the main Documentation Overview on Zope.org. Here you will find pointers to official and community contributed documentation.

  • To get personal assistance with Zope, look at the various Mailing Lists about Zope. The Mailing Lists are where you can get quick, accurate, friendly help from a large community of Zope users from around the world.

  • If you wish, you can find out about Zope Corporation, the publishers of Zope.

Zope-2.8.7-final/inst/WinBuilders/MANIFEST.txt0000664000076400007640000000355410435500331017714 0ustar ajungajungManifest and description of included files |-- Makefile.python: the Makefile used to create a Python binary from source |-- Makefile.zeo: the Makefile used to create a ZEO installer package |-- Makefile.zope: the Makefile used to create a Zope installer package |-- README.txt |-- bin | |-- compilezpy.py: compiles a tree of Python scripts into .pyc bytecode | |-- decompilezpy.py: deletes a tree full of .pyc bytecode files | |-- escape.sh: utility script that converts backslashes between unix & win | |-- fixreg.py: utility to manage Zope Corporation Win registry entries | |-- makezope.bat: utility script to build Zope, getting around NMAKE "bug" | `-- runzeo.bat: script which starts a ZEO instance |-- buildout: the main script used to build Zope and/or ZEO installers |-- doc | `-- ZC_PY_DIST_README.txt: readme for the Zope Corp. Python distro |-- etc | |-- README.html: custom template used for Zope Quick Start page. | |-- sitecustomize.py: adds Zope dirs to sys.path automagically in distro | |-- zeo.iss.in: the Innosetup file to create a ZEO installer | |-- zlogo_left.bmp: used by Innosetup | |-- zlogo_top.bmp: used by Innosetup | `-- zope.iss.in: the Innosetup file to create a Zope installer |-- mk | |-- common.mk: common make routines | |-- python.mk: routines for creating a Python distro | |-- zeo.mk: routines for creating a ZEO server distro | `-- zope.mk: routines for creating a Zope distro `-- zeo_addons: a custom INSTANCE_HOME for ZEO |-- skel | |-- README.txt | |-- bin | | |-- runzeo.bat.in | | `-- zeoservice.py.in | |-- etc | | |-- zeo.conf.in | | `-- zeoctl.conf.in | |-- log | | `-- README.txt | `-- var | `-- README.txt `-- utilities |-- compilezpy.py |-- copyskel.py |-- decompilezpy.py `-- mkzeoinstance.py Zope-2.8.7-final/inst/WinBuilders/buildout0000664000076400007640000000152210435500331017510 0ustar ajungajung#! /bin/sh # # buildout []* # # should be one of 'squid', 'zope', or 'zeo' (or some # other server instance for whom we have a specific makefile) # # are additional arguments to pass to make, e.g. # 'compile', 'install', etc. # reldir=`dirname $0` reldir=`realpath $reldir` server_type=$1 shift if [ -n "$PYTHONHOME" -o -n "$PYTHONPATH" ]; then echo "** Python build is likely to malfunction if either of" 1>&2 echo "** PYTHONHOME ('$PYTHONHOME') or PYTHONPATH ('$PYTHONPATH')" 1>&2 echo "** is set - please unset them before doing the buildout." 1>&2 exit 1 fi if [ -f $reldir/Makefile.$server_type ]; then /usr/bin/make -f $reldir/Makefile.$server_type \ -I $reldir MAKEFILEDIR=$reldir $@ else echo "buildout [make_args]*" fi Zope-2.8.7-final/inst/WinBuilders/Makefile.zeo0000664000076400007640000000175010435500331020175 0ustar ajungajung# Zope2 build and install Makefile for win32 (gnumake-style). # Creates an installable executable of ZEO/ZODB3. # Use a Win2k/XP machine with Cygwin, VC++, and InnoSetup4 # Install Inno Setup 4 into Program Files\Inno Setup 4 # also obtain and situate the following supporting files before # attempting to run the makefile: # # obtain the source distribution of ZODB 3.X.X and put the tar.gz file # in tmp\ZODB3-3.2.tar.gz # see python.mk for files needed to build Python # To produce the executable: # do "make -f build" # then "make -f build" # then "make -f "$(BUILD_DIR)/zeo.iss" # Remove CVS directories from the build tree. find $(BUILD_DIR) -name CVS -type d -exec $(RMRF) {} \; -prune # Convert text files to Windows line ends. unix2dos has the nice # property that it leaves lines with \r\n alone, so it doesn't hurt # to do this on files already converted to Windows convention. find $(BUILD_DIR) -name "*.py" -o -name "*.txt" -o -name "*.bat" | \ xargs unix2dos # Build the Inno installer. $(CD) "$(BUILD_DIR)";"$(ISS_COMPILER)" /cc "$(WIN_BUILD_DIR)\zeo.iss" $(BUILD_DIR)/lib/python/ExtensionClass.pyd: WIN_TMPDIR=tmp\zeotmp $(BUILD_DIR)/lib/python/ExtensionClass.pyd: install_python $(MKDIR) $(WIN_TMPDIR) $(CD) $(SRC_DIR)/$(ZODBDIRNAME); \ $(BUILD_DIR)/bin/python.exe setup.py install \ --prefix="$(WIN_BASE_DIR)\$(WIN_TMPDIR)" --no-compile $(MKDIR) $(@D) $(XCOPY) "$(WIN_TMPDIR)\Lib\site-packages\*.py" "$(shell cygpath -w $(@D))" $(XCOPY) "$(WIN_TMPDIR)\Lib\site-packages\*.pyd" "$(shell cygpath -w $(@D))" $(XCOPY) "$(WIN_TMPDIR)\Lib\site-packages\*.txt" "$(shell cygpath -w $(@D))" $(XCOPY) "$(WIN_TMPDIR)\Lib\site-packages\*.xml" "$(shell cygpath -w $(@D))" $(XCOPY) "$(WIN_TMPDIR)\Lib\site-packages\*.conf" "$(shell cygpath -w $(@D))" $(XCOPY) "$(WIN_TMPDIR)\Include\*.h" "$(WIN_BUILD_DIR)\bin\Include" $(XCOPY) "$(WIN_TMPDIR)\Scripts\*.py" "$(WIN_BUILD_DIR)\bin" $(RMRF) $(shell cygpath -u $(WIN_TMPDIR)) $(BUILD_DIR)/lib/python/nt_svcutils/__init__.py: CVS_REPOSITORY=$(ZOPE_CVS_REPOSITORY) $(BUILD_DIR)/lib/python/nt_svcutils/__init__.py: CVS_MODULE=Zope/lib/python/nt_svcutils $(BUILD_DIR)/lib/python/nt_svcutils/__init__.py: CVS_TAG=-r HEAD $(BUILD_DIR)/lib/python/nt_svcutils/__init__.py: ${CD} ${BUILD_DIR}/lib/python && \ ${CVS} -d ${CVS_REPOSITORY} export ${CVS_TAG} -d \ $(shell basename $(@D)) ${CVS_MODULE} $(BUILD_DIR)/skel/README.txt: $(MKDIR) "$(@D)" $(CPR) $(MAKEFILEDIR)/zeo_addons/skel/* $(@D) $(TOUCH) "$@" $(BUILD_DIR)/bin/mkzeoinstance.py: $(MKDIR) "$(@D)" $(CPR) $(MAKEFILEDIR)/zeo_addons/utilities/* $(@D) $(TOUCH) "$(@)" $(BUILD_DIR)/lib/python/version.txt: $(BUILD_DIR)/lib/python/ExtensionClass.pyd $(BUILD_DIR)/lib/python/nt_svcutils/__init__.py $(BUILD_DIR)/skel/README.txt $(BUILD_DIR)/bin/mkzeoinstance.py @echo ZEO $(ZODBVERSION) > "$(BUILD_DIR)/lib/python/version.txt" $(TOUCH) $@ src/$(ZODBDIRNAME)/setup.py: $(MKDIR) "$(SRC_DIR)" $(CD) "$(SRC_DIR)" && $(TAR) xvzf ../tmp/$(ZODBDIRNAME).tar.gz $(TOUCH) $@ Zope-2.8.7-final/inst/WinBuilders/mk/zope.mk0000664000076400007640000000400510435502543017661 0ustar ajungajungZOPE_REQUIRED_FILES=tmp/$(ZOPEDIRNAME).tgz REQUIRED_FILES=$(PYTHON_REQUIRED_FILES)\ $(ZOPE_REQUIRED_FILES) ZOPEVERSION=2.8.7-final ZOPEDIRNAME=Zope-$(ZOPEVERSION) MAKEZOPE="$(MAKEFILEDIR)/bin/makezope.bat" "$(WIN_BUILD_DIR)" # run the Zope tests test_zope: $(CD) "$(BASE_DIR)/src/Zope" "$(PYPCBUILDDIR)/python.exe" utilities/testrunner.py -a $(CD) "$(BASE_DIR)" clean_zope: $(RMRF) src/$(ZOPEDIRNAME) install_zope: src/$(ZOPEDIRNAME)/inst/configure.py \ $(BUILD_DIR)/lib/python/version.txt install_python \ $(BUILD_DIR)/Zope-$(ZOPEVERSION)-win32.exe ESCAPED=$(shell sh $(MAKEFILEDIR)/bin/escape.sh '$(WIN_MAKEFILEDIR)') SEDSCRIPT="s@<>@$(ZOPEVERSION)@g;s@<>@$(ESCAPED)@g" $(BUILD_DIR)/Zope-$(ZOPEVERSION)-win32.exe: $(BUILD_DIR)/lib/python/version.txt $(SED) $(SEDSCRIPT) < "$(MAKEFILEDIR)/etc/zope.iss.in" | unix2dos > "$(BUILD_DIR)/zope.iss" # Remove CVS directories from the build tree. find $(BUILD_DIR) -name CVS -type d -exec $(RMRF) {} \; -prune # Convert text files to Windows line ends. unix2dos has the nice # property that it leaves lines with \r\n alone, so it doesn't hurt # to do this on files already converted to Windows convention. find $(BUILD_DIR) -name "*.py" -o -name "*.txt" -o -name "*.bat" | \ xargs unix2dos # Build the Inno installer. $(CD) "$(BUILD_DIR)";"$(ISS_COMPILER)" /cc "$(WIN_BUILD_DIR)\zope.iss" $(BUILD_DIR)/lib/python/Zope2/Startup/run.py: $(CD) "$(BUILD_DIR)"; \ bin/python.exe \ "$(WIN_SRC_DIR)\$(ZOPEDIRNAME)\inst\configure.py" \ --prefix="$(WIN_BUILD_DIR)" --no-compile $(MAKEZOPE) $(TOUCH) "$(BUILD_DIR)/lib/python/Zope2/Startup/run.py" $(BUILD_DIR)/lib/python/version.txt: $(BUILD_DIR)/lib/python/Zope2/Startup/run.py @echo Zope $(ZOPEVERSION) > "$(BUILD_DIR)/lib/python/version.txt" $(TOUCH) "$(BUILD_DIR)/lib/python/version.txt" src/$(ZOPEDIRNAME)/inst/configure.py: $(MKDIR) "$(SRC_DIR)" $(CD) "$(SRC_DIR)" && $(TAR) xvzf ../tmp/$(ZOPEDIRNAME).tgz \ && $(TOUCH) $(ZOPEDIRNAME)/inst/configure.py Zope-2.8.7-final/inst/WinBuilders/mk/python.mk0000664000076400007640000001211210435500331020214 0ustar ajungajung# The Python and pywin32 versions. For Python, both the source tarball # and the Windows installer must be in tmp/. For pywin32 (previously known # as win32all), the Windows installer must be in tmp/. Nothing beyond those # is required to build Python, and you don't even need a compiler. PYVERSION_MAJOR=2 PYVERSION_MINOR=3 PYVERSION_PATCH=5 PYVERSION=$(PYVERSION_MAJOR).$(PYVERSION_MINOR).$(PYVERSION_PATCH) W32ALLVERSION=205 # CAUTION: Extracting files from Wise installers doesn't really do what # you expect. While a Wise installer is a zip file, the zip file # structure is flat (Wise reconstructs the intended directory structure # from metadata stored in proprietary FILEnnnn.DAT files also in the # zip file). Consequently, the package structure of Python packages is # lost, and if there's more than one file with the same name, you only # get "the last one" to be extracted (all files are extracted to the # same directory). # # For Python, this doesn't matter, because we're only sucking out the # precompiled .pyd and .exe files from the Python installer -- there # are no name clashes in that set, and it's a pretty safe bet there never # will be (else Python wouldn't be able to decide which to use!). We # use the Python source tarball to get all the non-executable parts we # need. # # pywin32 doesn't have this problem as it now uses # a standard distutils 'bdist_wininst' installation .exe. These executables are # valid .zip files with a "PLATLIB" directory being the complete directory # structure as installed into "site-packages". These recent pywin32 builds have # no dependencies on registry settings etc so will work directly as copied out of # the .exe. The only concerns are the pywintypes/pythoncom dlls, which is # handled by the Inno installer PYDIRNAME=Python-$(PYVERSION) # Standard bdist_wininst name - eg: pywin32-203.win32-py2.3[.exe] W32ALLDIRNAME=pywin32-$(W32ALLVERSION).win32-py$(PYVERSION_MAJOR).$(PYVERSION_MINOR) W32EXCLUDE=*.chm # The Python tarball is extracted to PYSRCDIR. # The contents of the Python installer get extracted to PYEXTRACTDIR. # The " " " win32all " " " " W32EXTRACTDIR. PYSRCDIR=$(BASE_DIR)/src/$(PYDIRNAME) PYEXTRACTDIR=$(BASE_DIR)/src/$(PYDIRNAME)-extract W32EXTRACTDIR=$(BASE_DIR)/src/$(W32ALLDIRNAME) WIN_PYSRCDIR=$(shell cygpath -w $(PYSRCDIR)) WIN_PYEXTRACTDIR=$(shell cygpath -w $(PYEXTRACTDIR)) WIN_W32EXTRACTDIR=$(shell cygpath -w $(W32EXTRACTDIR)) PYTHON_REQUIRED_FILES=tmp/$(W32ALLDIRNAME).exe \ tmp/$(PYDIRNAME).tgz \ tmp/$(PYDIRNAME).exe # Arbitrary files from each of the installers and tarballs, to use as # targets to force them to get unpacked. ARB_PYSRCDIR=$(PYSRCDIR)/PCbuild/pcbuild.dsw ARB_PYEXTRACTDIR=$(PYEXTRACTDIR)/zlib.pyd ARB_W32EXTRACTDIR=$(W32EXTRACTDIR)/PLATLIB # Building Python just consists of extracting files. build_python: $(ARB_PYSRCDIR) $(ARB_PYEXTRACTDIR) $(ARB_W32EXTRACTDIR) # Installing Python consists of copying oodles of files into # $(BUILD_DIR). install_python: $(BUILD_DIR)/bin/python.exe clean_python: $(RMRF) $(PYSRCDIR) $(RMRF) $(PYEXTRACTDIR) clean_libs: $(RMRF) $(W32EXTRACTDIR) $(ARB_PYSRCDIR): tmp/$(PYDIRNAME).tgz $(MKDIR) "$(SRC_DIR)" $(CD) "$(SRC_DIR)" && $(TAR) xvzf ../tmp/$(PYDIRNAME).tgz $(TOUCH) "$(ARB_PYSRCDIR)" $(ARB_PYEXTRACTDIR): tmp/$(PYDIRNAME).exe $(MKDIR) "$(PYEXTRACTDIR)" "tmp/$(PYDIRNAME).exe" /S /X "$(WIN_PYEXTRACTDIR)" $(TOUCH) "$(ARB_PYEXTRACTDIR)" # unzip warns about .exe not being exactly a .zip, then succeeds in # extracting the files, then returns with exit != 0 - ignore exit code $(ARB_W32EXTRACTDIR): tmp/$(W32ALLDIRNAME).exe -$(UNZIP) -o "-d$(W32EXTRACTDIR)" "tmp/$(W32ALLDIRNAME).exe" cd "$(WIN_W32EXTRACTDIR)\PLATLIB" ; $(RMRF) $(W32EXCLUDE) $(TOUCH) "$(ARB_W32EXTRACTDIR)" $(BUILD_DIR)/bin/python.exe: $(MKDIR) "$(BUILD_DIR)" $(MKDIR) "$(BUILD_DIR)/doc" $(CP) "$(MAKEFILEDIR)/doc/ZC_PY_DIST_README.txt" "$(BUILD_DIR)/doc" $(CP) "$(PYSRCDIR)/LICENSE" "$(BUILD_DIR)/doc/PYTHON_LICENSE.txt" unix2dos "$(BUILD_DIR)/doc/PYTHON_LICENSE.txt" $(MKDIR) "$(BUILD_DIR)/bin/DLLs" $(XCOPY) "$(WIN_PYEXTRACTDIR)\*.pyd" "$(WIN_BUILD_DIR)\bin\DLLs" $(MKDIR) "$(BUILD_DIR)/bin/Lib" $(XCOPY) "$(WIN_PYSRCDIR)\Lib\*.py" "$(WIN_BUILD_DIR)\bin\Lib" $(MKDIR) "$(BUILD_DIR)/bin/Lib/site-packages" $(CP) "$(PYSRCDIR)/Lib/site-packages/README" \ "$(BUILD_DIR)/bin/Lib/site-packages" $(XCOPY) "$(WIN_W32EXTRACTDIR)\PLATLIB" \ "$(WIN_BUILD_DIR)\bin\Lib\site-packages" $(CP) "$(MAKEFILEDIR)/etc/sitecustomize.py" \ "$(BUILD_DIR)/bin/Lib/site-packages" $(MKDIR) "$(BUILD_DIR)/bin/Include" $(XCOPY) "$(WIN_PYSRCDIR)\Include\*.h" "$(WIN_BUILD_DIR)\bin\Include" $(XCOPY) "$(WIN_PYSRCDIR)\PC\*.h" "$(WIN_BUILD_DIR)\bin\Include" $(MKDIR) "$(BUILD_DIR)/bin/libs" $(CP) "$(PYEXTRACTDIR)/python23.lib" "$(BUILD_DIR)/bin/libs" $(MKDIR) "$(BUILD_DIR)/bin" $(CP) "$(PYEXTRACTDIR)/pythonw.exe" "$(BUILD_DIR)/bin" $(CP) "$(PYEXTRACTDIR)/w9xpopen.exe" "$(BUILD_DIR)/bin" $(CP) "$(PYEXTRACTDIR)/python23.dll" "$(BUILD_DIR)/bin" $(CP) "$(PYEXTRACTDIR)/python.exe" "$(BUILD_DIR)/bin" $(TOUCH) "$(BUILD_DIR)/bin/python.exe" Zope-2.8.7-final/inst/WinBuilders/mk/spread.mk0000664000076400007640000000366210435500331020163 0ustar ajungajung# Justs repackages the Spread binaries. # Puts binaries into build/bin, and some docs into build/doc. # See README.txt for system requirements. # Download the binary distribution of Spread, from http://www.spread.org/. # Store it in tmp/. NOTE: May need to rename it to .tgz (depends on how # it's downloaded). The name is of the form spread-bin-n.nn.n.tgz SPREAD_VERSION=3.17.1 SPREAD_DIR=spread-bin-$(SPREAD_VERSION) SPREAD_ARCHIVE=$(SPREAD_DIR).tgz SPREAD_REQUIRED_FILES=tmp/$(SPREAD_ARCHIVE) REQUIRED_FILES=$(SPREAD_REQUIRED_FILES) # Unpack the tarball into src/. build_spread: $(SRC_DIR)/$(SPREAD_DIR)/win/spread.exe $(SRC_DIR)/$(SPREAD_DIR)/win/spread.exe : tmp/$(SPREAD_ARCHIVE) $(MKDIR) "$(SRC_DIR)" $(TAR) -C "$(SRC_DIR)" -xvzf tmp/$(SPREAD_ARCHIVE) $(TOUCH) $@ # A dumb trick so we don't unpack the tarball multiple times. $(SRC_DIR)/$(SPREAD_DIR)/license.txt \ $(SRC_DIR)/$(SPREAD_DIR)/Readme.txt \ $(SRC_DIR)/$(SPREAD_DIR)/sample.spread.conf \ : $(SRC_DIR)/$(SPREAD_DIR)/win/spread.exe $(TOUCH) $@ # Copy the Windows stuff from src/ into build/bin/ and build/doc install_spread: $(BUILD_DIR)/bin/spread.exe \ $(BUILD_DIR)/bin/sprecv.exe \ $(BUILD_DIR)/bin/spsend.exe \ $(BUILD_DIR)/bin/sptuser.exe \ $(BUILD_DIR)/bin/spflooder.exe \ $(BUILD_DIR)/doc/SPREAD_LICENSE.txt \ $(BUILD_DIR)/doc/SPREAD_README.txt \ $(BUILD_DIR)/doc/sample.spread.conf $(BUILD_DIR)/bin/spread.exe \ $(BUILD_DIR)/bin/sprecv.exe \ $(BUILD_DIR)/bin/spsend.exe \ $(BUILD_DIR)/bin/sptuser.exe \ $(BUILD_DIR)/bin/spflooder.exe : $(BUILD_DIR)/bin/% : \ $(SRC_DIR)/$(SPREAD_DIR)/win/% $(MKDIR) $(@D) $(CP) $< $@ $(TOUCH) $@ $(BUILD_DIR)/doc/SPREAD_LICENSE.txt: $(SRC_DIR)/$(SPREAD_DIR)/license.txt $(COPY_AND_WINDOWIZE_LINEENDS) $(BUILD_DIR)/doc/SPREAD_README.txt: $(SRC_DIR)/$(SPREAD_DIR)/Readme.txt $(COPY_AND_WINDOWIZE_LINEENDS) $(BUILD_DIR)/doc/sample.spread.conf: $(SRC_DIR)/$(SPREAD_DIR)/sample.spread.conf $(COPY_AND_WINDOWIZE_LINEENDS) Zope-2.8.7-final/inst/WinBuilders/mk/common.mk0000664000076400007640000000333510435500331020172 0ustar ajungajungBASE_DIR=$(shell pwd) WIN_BASE_DIR=$(shell cygpath -w $(BASE_DIR)) BUILD_DIR=$(BASE_DIR)/build WIN_BUILD_DIR=$(shell cygpath -w $(BUILD_DIR)) SRC_DIR=$(BASE_DIR)/src WIN_SRC_DIR=$(shell cygpath -w $(SRC_DIR)) TMP_DIR=$(BASE_DIR)/tmp WIN_TMP_DIR=$(shell cygpath -w $(TMP_DIR)) WIN_MAKEFILEDIR=$(shell cygpath -w $(MAKEFILEDIR)) # Root of the Windows drive you're working on. The setting here is for # the C: drive and using a default out-of-the-box Cygwin. CYGROOT=/cygdrive/c RM=rm -f RMRF=rm -rf CD=cd XCOPY=xcopy /i /s /e /y CPR=cp -r CP=cp MKDIR=mkdir -p CVS="$(MAKEFILEDIR)/bin/cvs.exe" TAR=tar SED=sed TOUCH=touch NMAKE=nmake CSCRIPT=cscript ECHO=echo ISS_DIR=$(CYGROOT)/Progra~1/Inno Setup 4 ISS_COMPILER=$(ISS_DIR)/Compil32.exe # We need a version that understands cygwin paths, so /bin/ UNZIP=/bin/unzip $(REQUIRED_FILES): if [ -z "$(BASE_DIR)/$@" ]; then echo. &echo. & echo \ You must download $(@F) and place it in "$(BASE_DIR)/tmp" &\ error 1; fi PRODUCTS_DIR=build/instance/Products PYLIBDIR=build/instance/lib/python ZOPE_CVS_REPOSITORY=:pserver:anonymous@cvs.zope.org:/cvs-repository CVS=/usr/bin/cvs -z7 -q CVS_UPDATE=${CVS} update -dP CVSROOT=:ext:korak.zope.com:/cvs-turbointranet # Use COPY_AND_WINDOWIZE_LINEENDS like so: # # destination_path: source_path # $(COPY_AND_WINDOWIZE_LINEENDS) # # Any directories needed to hold destination_path are created. The file # is copied from source_path to there, and unix2dos is run on it in its # new home. The paths must give file names, not directories; this reflects # that renaming of files is often needed in these makefiles. define COPY_AND_WINDOWIZE_LINEENDS $(MKDIR) $(@D) $(CP) $< $@ unix2dos $@ $(TOUCH) $@ endefZope-2.8.7-final/inst/file_from_infile.py0000664000076400007640000000611710435500331017356 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Reads a file named by 'src', performs textual replacements on the file based on sed-style markup, and writes the result to the file named by 'dst' unless 'dst' already exists. """ import getopt, os, sys from os.path import abspath, split, dirname import shutil import versions default_map = { 'PYTHON' : sys.executable, 'BASE_DIR' : abspath(split(dirname(sys.argv[0]))[0]), 'ZOPE_MAJOR_VERSION' : versions.ZOPE_MAJOR_VERSION, 'ZOPE_MINOR_VERSION' : versions.ZOPE_MINOR_VERSION, 'ZOPE_BRANCH_NAME' : versions.ZOPE_BRANCH_NAME, 'VERSION_RELEASE_TAG' : versions.VERSION_RELEASE_TAG, } def main(source, dest, map, force): if not force and os.path.exists(dest): print '%s exists, so I left it alone' % dest else: txt = open(source, 'rb').read() for k, v in map.items(): txt = txt.replace('<<%s>>' % k, v) outfile = open(dest, 'wb') outfile.write(txt) outfile.close() shutil.copystat(source, dest) print "Wrote %s from %s" % (dest, source) def usage(): print "%s [opts] src dst" % sys.argv[0] print print "Reads a file named by 'src', performs textual replacements on " print "the file based on sed-style markup embedded in the infile, and " print "and writes the result to the file named by 'dst' unless 'dst'." print "already exists. The generated file will have the same permissions" print "and other mode bit settings as the source file." print print "Options:" print print " --force Force replacement of dst even if it already exists." for name, value in default_map.items(): print (" --%s=value controls text replacement, default '%s'" % (name, value)) if __name__ == '__main__': if len(sys.argv) < 3: usage() sys.exit(127) map = default_map.copy() force = 0 try: longopts = ['help', 'force'] for name in default_map.keys(): longopts.append('%s=' % name) opts, args = getopt.getopt(sys.argv[1:], 'h', longopts) except getopt.GetoptError, v: print v usage() sys.exit(1) try: source, dest = args except: usage() sys.exit(1) for o, a in opts: if o in ('-h', '--help'): usage() sys.exit() if o == '--force': force = 1 if o in map.keys(): map[o] = a main(source, dest, map, force) Zope-2.8.7-final/inst/walkandscrub.py0000664000076400007640000000314210435500331016541 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os, sys DEBUG = 0 if os.name in ('posix', 'nt', 'dos'): EXCLUDED_NAMES=['..', '.'] else: EXCLUDED_NAMES=[] # extend EXCLUDED_NAMES here manually with filenames ala "asyncore.pyc" for # files that are only distributed in compiled format (.pyc, .pyo) # if necessary (not currently necessary in 2.3.1 AFAIK) - chrism def walkandscrub(path): path = os.path.expandvars(os.path.expanduser(path)) print print '-'*78 sys.stdout.write( "Deleting '.pyc' and '.pyo' files recursively under %s...\n" % path ) os.path.walk(path, scrub, []) sys.stdout.write('Done.\n') def scrub(list, dirname, filelist): for name in filelist: if name in EXCLUDED_NAMES: continue prefix, ext = os.path.splitext(name) if ext == '.pyo' or ext == '.pyc': full = os.path.join(dirname, name) os.unlink(full) if DEBUG: print full if __name__ == '__main__': DEBUG = 1 walkandscrub(os.getcwd()) Zope-2.8.7-final/inst/make_resource.py0000664000076400007640000000254010435500331016706 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Build a PCGI resource file. You must be in the directory containing this script. """ import os from do import * def main(cwd=os.getcwd(), name='Zope', user='', group=''): python=sys.executable print '-'*78 print 'Writing the pcgi resource file (ie cgi script), %s' % name cwd=os.environ.get('ZDIR',cwd) open(name,'w').write('''#!%(cwd)s/pcgi/pcgi-wrapper PCGI_NAME=Zope PCGI_MODULE_PATH=%(cwd)s/lib/python/Zope PCGI_PUBLISHER=%(cwd)s/pcgi/pcgi_publisher.py PCGI_EXE=%(python)s PCGI_SOCKET_FILE=%(cwd)s/var/pcgi.soc PCGI_PID_FILE=%(cwd)s/var/pcgi.pid PCGI_ERROR_LOG=%(cwd)s/var/pcgi.log PCGI_DISPLAY_ERRORS=1 BOBO_REALM=%(name)s BOBO_DEBUG_MODE=1 INSTANCE_HOME=%(cwd)s ''' % vars()) ch(name, user, group, 0755) Zope-2.8.7-final/inst/Makefile.win.in0000664000076400007640000000404110435500331016347 0ustar ajungajung# Zope2 build and install Makefile for win32 (nmake-style). # We do as much as possible in Python in order to avoid needing to # learn autoconf or some other awful thing. ;-) NAME=Zope MAJOR_VERSION=<> MINOR_VERSION=<> RELEASE_TAG=<> PACKAGE_NAME=$(NAME)-$(MAJOR_VERSION).$(MINOR_VERSION)-$(RELEASE_TAG) PYTHON="<>" PREFIX=<> BASE_DIR=<> BUILD_BASE=<> DISTUTILS_OPTS=<> INSTALL_FLAGS=<> TESTOPTS=-v BUILD_FLAGS=--build-base="$(BUILD_BASE)" \ --build-lib="$(BUILD_BASE)\build-lib" \ --build-scripts="$(BUILD_BASE)\build-scripts" \ --build-temp="$(BUILD_BASE)\build-temp" RM=del /f /q !IF ("$(OS)" == "Windows_NT") RMRF=rmdir /s /q !ELSE RMRF=deltree /y !ENDIF CD=cd XCOPY=xcopy /i /s /e /y COPY=copy default: build # default: The default step (invoked when make is called without a target) @ echo. @ echo Zope built. Next, do 'nmake install'. @ echo. .PHONY: clean install build unbuild .PHONY: default # build: Do whatever 'setup.py build' implies build: $(PYTHON) "$(BASE_DIR)\setup.py" \ $(DISTUTILS_OPTS) build $(BUILD_FLAGS) # unbuild: Remove the build directory (undo the make build step) unbuild: -$(RMRF) $(BUILD_BASE) # install: Install a software home. install: build version_txt $(PYTHON) "$(BASE_DIR)\setup.py" $(DISTUTILS_OPTS) install \ --prefix="$(PREFIX)" $(BUILD_FLAGS) $(INSTALL_FLAGS) @ echo. @ echo Zope binaries installed successfully. @ echo Now run '$(PYTHON) $(PREFIX)\bin\mkzopeinstance.py' # version_txt: create a version file in lib/python/version.txt version_txt: echo Zope $(MAJOR_VERSION).$(MINOR_VERSION)-$(RELEASE_TAG) >\ "$(BASE_DIR)/lib/python/version.txt" # clean: Delete the build files and any binaries/bytecode files in # the source directory for good measure. clean: unbuild $(CD) "$(BASE_DIR) -$(RM) /s *.pyc *.pyo *.dll *.o *.obj *.pyd -$(RM) (BASE_DIR)\lib\python\version.txt Zope-2.8.7-final/inst/Makefile.in0000664000076400007640000001075610435500331015565 0ustar ajungajung# Zope2 build and install Makefile. # We do as much as possible in Python in order to avoid needing to # learn autoconf or some other awful thing. ;-) NAME=Zope MAJOR_VERSION=<> MINOR_VERSION=<> RELEASE_TAG=<> PACKAGE_NAME=${NAME}-${MAJOR_VERSION}.${MINOR_VERSION}-${RELEASE_TAG} PYTHON="<>" TMPDIR=/tmp PREFIX=<> BASE_DIR=<> BUILD_BASE=<> DISTUTILS_OPTS=<> INSTALL_FLAGS=<> TESTOPTS=-v BUILD_FLAGS=--build-base="${BUILD_BASE}" \ --build-lib="${BUILD_BASE}/build-lib" \ --build-scripts="${BUILD_BASE}/build-scripts"\ --build-temp="${BUILD_BASE}/build-temp" RM=rm -f RMRF=rm -rf FIND=find XARGS=xargs CD=cd LN=ln -sfn CP=cp TAR=tar MKDIR=mkdir -p .PHONY : clean install instance untestinst testinst build unbuild .PHONY : default # default: The default step (invoked when make is called without a target) default: build @echo @echo Zope built. Next, do \'make install\' \(or \'make instance\' @echo to run a Zope instance directly from the build directory\). @echo # build: Do whatever 'setup.py build' implies build: ${PYTHON} "${BASE_DIR}/setup.py" \ ${DISTUTILS_OPTS} build ${BUILD_FLAGS} # unbuild: Remove the build directory (undo the make build step) unbuild: ${RMRF} ${BUILD_BASE} # install: Install a software home. install: build version_txt ${PYTHON} "${BASE_DIR}/setup.py" ${DISTUTILS_OPTS} install \ --home="${PREFIX}" ${BUILD_FLAGS} ${INSTALL_FLAGS} [ -f ${PREFIX}/bin/python ] || ${LN} ${PYTHON} ${PREFIX}/bin/python @echo @echo Zope binaries installed successfully. @echo Now run \'${PREFIX}/bin/mkzopeinstance.py\' # inplace: Install a software home into to the source directory. # # Note: We used to run 'build_ext -i' for 'inplace', but that was # suboptimal because it had a tendency to try to rebuild all of the # (possibly already-built) extensions that might be built during a # previous 'make' step. built_ext doesn't understand '--build-base' # and friends so we can't stop it from doing this easily. So instead, # we rely on the stock install step and name the prefix as the current # directory. This is a little less efficient than just building the # extensions because it also compiles bytecode, but it's more intuitive and # less expensive in the common case than letting distutils # potentially rebuild the binaries when we've done that already. inplace: PREFIX=${BASE_DIR} inplace: install # instance: Do an inplace build and create an instance home in the resulting # software home. instance: inplace ${PYTHON} "${BASE_DIR}/bin/mkzopeinstance.py" ${MKZ_FLAGS} \ --dir="${BASE_DIR}" # uninstance: Remove the instance files made by make instance (w/ prejudice) uninstance: ${RMRF} "${BASE_DIR}/bin" ${RMRF} "${BASE_DIR}/etc" ${RMRF} "${BASE_DIR}/log" ${RMRF} "${BASE_DIR}/var" ${RMRF} "${BASE_DIR}/Products" # testinst: Perform an inplace build and create an instance home in the # resulting software home without asking questions. Useful when # performing automated testing. testinst: MKZ_FLAGS=--user=admin:admin testinst: instance # test: Do an inplace build and run the Zope test suite. test: inplace ${PYTHON} "${BASE_DIR}/test.py" ${TESTOPTS} # clean: Delete the build files and any binaries/bytecode files in # the source directory for good measure. clean: unbuild ${FIND} "${BASE_DIR}" \ -name '*.py[co]' -o -name '*.so' -o -name '*.o' | ${XARGS} ${RM} # version_txt: create a version file in lib/python/version.txt version_txt: printf "Zope ${MAJOR_VERSION}.${MINOR_VERSION}-${RELEASE_TAG}" >\ "${BASE_DIR}/lib/python/version.txt" # sdist: Create a source distribution file (implies clobber). # sdist: clobber sdist_tgz # sdist_tgz: Create a tgz archive file as a source distribution. # sdist_tgz: version_txt ${MKDIR} ${TMPDIR} ${CD} ${TMPDIR} && ${LN} ${BASE_DIR} ${PACKAGE_NAME} && \ ${TAR} czfh ${BASE_DIR}/${PACKAGE_NAME}.tgz \ --exclude=${PACKAGE_NAME}.tgz\ --exclude=.svn\ --exclude=makefile \ --exclude=build-base \ --exclude=*~ \ --exclude=.#* ${PACKAGE_NAME} ${RMRF} ${TMPDIR}/${PACKAGE_NAME} # clobber: Make the source tree 'pristine' again. clobber: clean uninstance # distclean: Make the source tree *really* 'pristine' again. distclean: clobber ${RM} makefile Makefile ${RMRF} build-base Zope-2.8.7-final/inst/versions.py0000664000076400007640000000032510435502526015741 0ustar ajungajungZOPE_MAJOR_VERSION = '2.8' ZOPE_MINOR_VERSION = '7' ZOPE_BRANCH_NAME = '$Name$'[6:] or 'no-branch' # always start prerelease branches with '0' to avoid upgrade # issues in RPMs VERSION_RELEASE_TAG = 'final' Zope-2.8.7-final/inst/configure.py0000664000076400007640000002030310435500331016040 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ Create a Makefile for building and installing Zope. """ import getopt import os import sys import versions QUIET=0 if sys.platform == 'win32': PREFIX = 'c:\\Zope-' + versions.ZOPE_MAJOR_VERSION IN_MAKEFILE = 'Makefile.win.in' MAKE_COMMAND='the Visual C++ batch file "VCVARS32.bat" and then "nmake"' else: PREFIX = '/opt/Zope-' + versions.ZOPE_MAJOR_VERSION IN_MAKEFILE = 'Makefile.in' MAKE_COMMAND='make' def main(): # below assumes this script is in the BASE_DIR/inst directory global PREFIX BASE_DIR=os.path.abspath(os.path.dirname(os.path.dirname(sys.argv[0]))) BUILD_BASE=os.path.join(os.getcwd(), 'build-base', 'python-%s.%s' % sys.version_info[:2]) PYTHON=sys.executable MAKEFILE=open(os.path.join(BASE_DIR, 'inst', IN_MAKEFILE)).read() REQUIRE_LF_ENABLED = 1 REQUIRE_ZLIB = 1 REQUIRE_EXPAT = 1 INSTALL_FLAGS = '' DISTUTILS_OPTS = '' try: longopts = ['help', 'ignore-largefile', 'ignore-zlib', 'ignore-expat', 'prefix=', 'build-base=', 'optimize', 'no-compile', 'quiet'] opts, args = getopt.getopt(sys.argv[1:], 'h', longopts) except getopt.GetoptError, v: print v usage() sys.exit(1) for o, a in opts: if o in ('-h', '--help'): usage() sys.exit() if o == '--prefix': PREFIX=os.path.abspath(os.path.expanduser(a)) if o == '--ignore-largefile': REQUIRE_LF_ENABLED=0 if o == '--ignore-zlib': REQUIRE_ZLIB=0 if o == '--ignore-expat': REQUIRE_EXPAT=0 if o == '--optimize': INSTALL_FLAGS = '--optimize=1 --no-compile' if o == '--no-compile': INSTALL_FLAGS = '--no-compile' if o == '--build-base': BUILD_BASE = a if o == '--quiet': DISTUTILS_OPTS = '-q' global QUIET QUIET = 1 if REQUIRE_LF_ENABLED: test_largefile() if REQUIRE_ZLIB: test_zlib() if REQUIRE_EXPAT: test_expat() out(' - Zope top-level binary directory will be %s.' % PREFIX) if INSTALL_FLAGS: out(' - Distutils install flags will be "%s"' % INSTALL_FLAGS) idata = { '<>':PYTHON, '<>':PREFIX, '<>':BASE_DIR, '<>':BUILD_BASE, '<>':INSTALL_FLAGS, '<>':versions.ZOPE_MAJOR_VERSION, '<>':versions.ZOPE_MINOR_VERSION, '<>':versions.VERSION_RELEASE_TAG, '<>':DISTUTILS_OPTS, } for k,v in idata.items(): MAKEFILE = MAKEFILE.replace(k, v) f = open(os.path.join(os.getcwd(), 'makefile'), 'w') f.write(MAKEFILE) out(' - Makefile written.') out('') out(' Next, run %s.' % MAKE_COMMAND) out('') def usage(): usage = (""" %(program)s configures and writes a Makefile for Zope. Defaults for options are specified in brackets. Configuration: -h, --help display this help and exit Options: --quiet suppress nonessential output --ignore-zlib allow configuration to proceeed if Python zlib module is not found. --ignore-largefile allow configuration to proceed without Python large file support. --ignore-expat allow configuration to proceed if the expat XML parsing module is not found. --optimize compile Python files as .pyo files instead of as .pyc files --no-compile don't compile Python files Directories: --build-base=DIR use DIR to store temporary build files --prefix=DIR install Zope files in DIR [%(PREFIX)s] By default, 'make install' will install Zope software files in '%(PREFIX)s' You can specify an alternate location for these files by using '--prefix', for example: '--prefix=$HOME/zope'. """ % ({'program':sys.argv[0], 'PREFIX':PREFIX}) ) print usage def test_expat(): try: import xml.parsers.expat except ImportError: print ( """ The Python interpreter you are using does not appear to have the 'pyexpat' library module installed. For many Zope features to work properly, including Zope Page Templates, you must install a Python interpreter which includes the pyexpat module, or install the pyexpat library into your Python interpreter manually. The file which represents the library is named 'pyexpat.so' (UNIX) or 'pyexpat.dll' (Windows) and is typically located in the 'lib-dynload' directory of your Python's library directory. Some Python packagers ship the pyexpat module as a separate installable binary. If you are using a system-provided Python installation, you may want to look for a 'python-xml' or 'python-pyexpat' package (or something like it) and install it to make the pyexpat module available to Zope. If you've compiled your Python interpreter from source, you may need to recompile and reinstall it after installing James Clark's expat libraries and development packages (look for libexpat.so and expat.h). Typically, these come as part of your operating system's libexpat and libexpat-dev packages, respectively. Run the configure script with the --ignore-expat option to prevent this warning with the understanding that some Zope features may not work properly until you've installed the pyexpat module. """ ) sys.exit(1) def test_zlib(): try: import zlib except ImportError: print ( """ The Python interpreter you are using does not appear to have the 'zlib' library module installed. For Zope to be able to run, you must install a Python interpreter which includes the zlib module, or install the zlib library into your Python interpreter manually. The file which represents the library is named 'zlib.so' (UNIX) or 'zlib.dll' (Windows) and is typically located in the 'lib-dynload' directory of your Python's library directory. Some Python packagers ship the zlib module as a separate installable binary. If you are using a system-provided Python installation, you may want to look for a 'python-zlib' package (or something like it) and install it to make the Python zlib module available to Zope. Run the configure script with the --ignore-zlib option to prevent this warning with the understanding that Zope will not start properly until you've installed the zlib module. """ ) sys.exit(1) except: print 'An error occurred while trying to import zlib!' import traceback; traceback.print_exc() sys.exit(1) def test_largefile(): OK=0 f = open(sys.argv[0], 'r') try: # 2**31 == 2147483648 f.seek(2147483649L) f.close() OK=1 except (IOError, OverflowError): f.close() if OK: return print ( """ This Python interpreter does not have 'large file support' enabled. Large file support is required to allow the default Zope ZODB database to grow larger than 2GB on most platforms. Either install a Python interpreter with large file support (see http://www.python.org/doc/current/lib/posix-large-files.html) or run this program again with the --ignore-largefile option to prevent this warning, with the understanding that your Zope may fail if the ZODB database size ever exceeds 2GB. """ ) sys.exit(1) def out(s): if not QUIET: print s if __name__ == '__main__': main() Zope-2.8.7-final/configure0000775000076400007640000001571010435500314014446 0ustar ajungajung#!/bin/sh # Zope configure script # $Id: configure 29125 2005-02-12 15:52:06Z andreasjung $ ##################################################################### # BEGIN EDITABLE PARAMETERS # ##################################################################### # Place the Zope major version number below. ZOPE_VERS=2.8 # Place the optimal target version number for Zope (as returned by sys.version) # below TARGET="2.3.5" # Order a list of "acceptable" python version numbers (as returned by # sys.version) below in "best" to "worst" order, not including the # target version. Up to six acceptable python versions are allowed. # Do not include the target version number in this list! ACCEPTABLE="" # provide the executable names for all the acceptable versions # (and the target version) below EXENAMES="python python2 python2.3" ##################################################################### # END EDITABLE PARAMETERS # ##################################################################### # where are we? HERE=`dirname $0` # should we be quiet? QUIET="" usage() { echo echo "configure [--help] [--quiet] [--with-python=path] [--prefix=path] " echo " [--build-base=path] [--ignore-largefile] [--ignore-zlib]" echo " [--optimize]" echo echo " Creates a Makefile suitable for building and installing Zope" echo echo " Options: " echo " --help shows usage and quits" echo " --quiet suppress nonessential output" echo " --with-python specify a path to a Python interpreter to use" echo " --prefix specify an installation path for binary data" echo " --build-base specify a temporary path for build files" echo " --ignore-largefile ignore large file support warnings" echo " --ignore-expat ignore warnings about expat/pyexpat" echo " --ignore-zlib ignore warnings about zlib" echo " --optimize optimize compiled Python bytecode" echo " --no-compile Dont compile Python bytecode" echo echo " Given no options, configure will search your PATH for a suitable" echo " Python interpreter and will use '/opt/Zope-$ZOPE_VERS' as a prefix." echo } # bootstrap ourselves by finding a Python interpreter if necessary get_python() { OLDIFS="$IFS" IFS=":" FOUND="" VERSION="" FOUNDLIST="" out "Testing for an acceptable Python interpreter..." out "" for DIR in $PATH; do IFS="$OLDIFS" for EXECUTABLE in $EXENAMES; do FULL="$DIR/$EXECUTABLE" if [ -x "$FULL" -a ! -d "$FULL" ]; then CMD="import string,sys;a=string.split(sys.version)[0]" # Strip trailing + from version number CMD="$CMD;a=(a[-1]=='+')and(a[:-1])or(a);print a" VERSION=`"$FULL" -c "$CMD"` out " Python version $VERSION found at $FULL" if [ "$VERSION" = "$TARGET" ]; then FOUND="$FULL" FOUNDVERSION=$VERSION break 2 else i=1; for ACC in $ACCEPTABLE; do i=`expr $i + 1` for SLOT in $FOUNDLIST; do if [ $SLOT -eq $i ]; then # slot "i" already populated. This means we've # already found this particular version of # python. Continue the for ACC in # $ACCEPTABLE loop and don't overwrite the # one we already found (interpreters first # on the path win). continue 2 fi done if [ "$VERSION" = "$ACC" ]; then FOUNDLIST="$FOUNDLIST $i" eval "FOUND$i=$FULL" eval "FOUNDVERSION$i=$VERSION" fi done fi fi done done if [ "$VERSION" = "$TARGET" ]; then out "" out " The optimum Python version ($TARGET) was found at $FOUND." elif [ -z "$FOUND1" ] && [ -z "$FOUND2" ] && [ -z "$FOUND3" ] && [ -z "$FOUND4" ] && [ -z "$FOUND5" ] && [ -z "$FOUND6" ] ; then out "" out " No suitable Python version found. You should install" out " Python version $TARGET before continuing." if [ "$ACCEPTABLE" ]; then out " Versions $ACCEPTABLE also work, but not as optimally." fi exit 1 else if [ -n "$FOUND1" ]; then FOUND=$FOUND1 FOUNDVERSION=$FOUNDVERSION1 elif [ -n "$FOUND2" ]; then FOUND=$FOUND2 FOUNDVERSION=$FOUNDVERSION2 elif [ -n "$FOUND3" ]; then FOUND=$FOUND3 FOUNDVERSION=$FOUNDVERSION3 elif [ -n "$FOUND4" ]; then FOUND=$FOUND4 FOUNDVERSION=$FOUNDVERSION4 elif [ -n "$FOUND5" ]; then FOUND=$FOUND5 FOUNDVERSION=$FOUNDVERSION5 elif [ -n "$FOUND6" ]; then FOUND=$FOUND6 FOUNDVERSION=$FOUNDVERSION6 fi out "" out " !! WARNING !! " out " An acceptable, but non-optimal Python version ($FOUNDVERSION) " out " was found at '$FOUND'." out " But consider installing version '$TARGET' before running " out " 'make'. If this isn't the Python version or interpreter " out " instance you wish to use, you may specify a Python interpreter" out " manually by rerunning the ./configure script with the " out " '--with-python' option." fi out "" } out() { if [ -z "$QUIET" ]; then echo $1 fi } NEWOPTS="" for OPT in $@; do case "$OPT" in --h* | -h*) usage exit 0 ;; --with-python=*) # pop this argument from the arglist, it is not valid to # pass this along to the Python configurator. shift; FOUND=`echo $OPT | sed -e 's/--with-python\=//'` # use eval to do tilde expansion below eval "FOUND='$FOUND'" out "" out "Using Python interpreter at $FOUND" ;; --with-python) # in case someone passed in a --with-python without a value, # we raise an error instead of passing it along to configure.py # (which would raise an inscrutable error were it to receive this # option). out "--with-python argument requires an option" exit 1 ;; --quiet* | -q*) QUIET="true" NEWOPTS="$NEWOPTS $OPT" ;; *) NEWOPTS="$NEWOPTS $OPT" ;; esac done out "" out "Configuring Zope installation" out "" if [ -z "$FOUND" ]; then get_python fi # run the Python configurator "$FOUND" "$HERE/inst/configure.py" $NEWOPTS Zope-2.8.7-final/doc/0000775000076400007640000000000010435502510013300 5ustar ajungajungZope-2.8.7-final/doc/FAQ.txt0000664000076400007640000002646410435500316014466 0ustar ajungajungZope Installation Frequently Asked Questions -------------------------------------------- Installing Zope 1. On certain Unix platforms (usually Solaris) I get a "checksum error" untarring the Zope distribution. Whats the deal? Zope is built and distributed using GNU tar. Some vendor-supplied tar implementations have problems with long path names as they appear in GNU tar created files. This is a problem that occurs in a number of software distributions created using GNU tar, not just Zope. We suggest that you install a GNU tar on your system to avoid this problem (not just with Zope, but with other software distributions as well). 2. I am trying to build Zope on my BSD based system, but I keep getting "symbol not found" errors and the build scripts stop. What do I do? On some BSD based systems (BSDI, BSD/OS), you need to edit the Makefile.pre.in files in your Zope installation to add -rdynamic to the LDSHARED makefile variable. On some systems it may also be necessary to change the file named 'do.py' located in the 'inst' directory of your Zope installation, changing the line: do('make') to: do('make -k'). 3. I am having trouble building Zope on my OpenBSD system. I get some wierd compilation errors. Make sure you unpack Zope using GNU tar; you will not get any error messages with OpenBSD's standard tar, but your installation will be subtly broken due to problems with long path names. 4. I am having trouble building Zope on a libc 5 Linux machine. Zope requires Python be built with glibc 2.x (libc 6) on Linux. 5. I am trying to install Zope on a Windows NT/2000 machine, but I keep getting a dialog saying "corrupt installation detected". Is something wrong with the distribution file? No, to install Windows NT / 2000 you should be logged in with "Administrator" privileges on the machine. If you are seeing the "corrupt installation detected" dialog, you probably don't have the required administrator privileges (your guess is as good as ours as to why you get that particular error message). Starting/Accessing your Zope installation 1. After installing Zope, I go to the URL in my web browser but I get a "404 Not Found" or "Server Error" message Is the URL really correct? Are you running Zope behind another web server? If so, this message is coming from your web server, not Zope. It usually means that your web server is misconfigured. Look in the error log for your web server for details of the error. 2. I have installed Zope, and when I go to the URL in my web browser, I get a password prompt. Although I type in the correct username and password, I keep getting prompted. What is going on? Are you running another web server in front of Zope? Zope manages access control internally, but it relies on your web server software to relay any authentication information in web requests. Some web servers automatically filter this information out of requests passed to cgi programs. If you keep getting password prompts that never accept your password, this means that your web server is not properly configured to pass authentication info on to Zope. See the doc/WEBVSERVER.txt file for information on correctly configuring your web server to pass authentication information. 3. I have installed Zope, and when I go to the URL in my web browser, I get a "Connection refused" error. Probably this means that Zope has failed to start. You should try to start Zope in the foreground and look for error messages. On Linux / Unix / OSX, you can do this by giving the command "./bin/runzope" or "./bin/zopectl fg" in your zope instance (they are equivalent). On Windows, navigate to the Zope folder in your Start Menu and click on "Run Zope in Console". A console window with process startup information will be displayed. If there are errors, the last error shown should explain what went wrong. 4. When I go to a URL for an application that isn't running, it takes quite a while to startup. Why is that? The application has to get into memory and then get the database loaded. Depending on database file size, this might take a while. Packing your Zope database at regular intervals can reduce startup time. 5. I have Zope with the Oracle Database adapter installed, and can't get it running. Why? If you have Oracle installed, you need to ensure that the environment variable "ORACLE_HOME" is in the environment of the running application. You can do this by adding a line to the section of etc/zope.conf, such as: ORACLE_HOME /foo/bar/ 6. I have forgotten the only password used to access the site or I have modified the security settings in such a way that even I can't get access. How do I fix it? Assuming you have write access to the directory where Zope is installed, you can create a temporary "emergency user" using the 'zpasswd.py' script:: python bin/zpasswd.py access Follow the prompts to enter a user name and password. 'zpasswd.py' will write the file named 'access'. Zope will look for this file the next time it starts. After restarting, browse to Zope's management interface and use the name and password you entered. As the emergency user, you are allowed to create user folders and user accounts as well as adjust security settings but you are not allowed to create objects like DTML methods or folders. This is a safety precaution. See more details in the file SECURITY.TXT. Managing the Zope process 1. How do I find the application server in my process listing? When running Zope, there will be a .pid file in var/Z2.pid which contains two process IDs. 2. If I have more than one Zope process running on the same machine, how do I tell them apart in "ps" listings? When running under ZServer, the Zope processes will show up in the listings with 'run.py'. (You may need to do a long listing to see it.) Look for the pids listed in var/Z2.pid. 3. Is it OK to use 'kill' to shut down the process? If you started zope with the command "zopectl start", the command "zopectl stop" will shut down Zope. You can also shut down the process using the Shutdown button in the Control Panel. Alternatively, the URL: 'Control_Panel/manage_shutdown' will, with a suitable username and password, shutdown the application. If you need a non-interactive way to shutdown your application, e.g. from a script, then write a program that sends a web request to that URL with the appropriate username and password. If you must, you can shut down the application by killing it. The respective var/Z2.pid file will have the pids. Upgrading from Earlier Versions of Zope 1. How can I upgrade to Zope 2.8? The usual procedure when installing a new version of Zope is: - Create a new instance (see doc/INSTALL.txt). - Copy the var/Data.fs file from your old instance into your new instance's var/ directory. It is best to do this while the old instance is not running; alternatively, you can use repozo.py as described at http://www.zope.org/Wikis/ZODB/FileStorageBackup to make a live backup, then create a new Data.fs from that backup in your new instance. Keep your old Data.fs in the old instance as a backup. - Copy the Products and Extensions directories from your old instance into the new instance. Leave the old instance directories intact as a backup. - Update etc/zope.conf with any changes you made in your old instance. (Likewise for zeo.conf if you are using zeo.) - Start the new instance. (Be sure the old instance is not running.) - Update any ZCatalogs as described below. - Update any Python Scripts. This is not strictly necessary, but it will make Zope restart somewhat faster. Visit this URL in your browser: http://.../manage_addProduct/PythonScripts/recompile 2. After moving my old Data.fs to Zope 2.8 I get an "AttributeError, _length" when trying to use the ZCatalog? In Zope 2.8 we cleaned up the index implementations and therefore you need to migrate every single ZCatalog manually. Notice: **YOU SHOULD BACKUP YOUR var/Data.fs file before starting the migration**, see http://www.zope.org/Wikis/ZODB/FileStorageBackup for backup instructions. To perform the migration: - go to the ZMI - click on your ZCatalog instance (or portal_catalog inside your CMF or Plone instance) - you will get an "AttributeError, _length"...don't panic - call the "manage_convertIndexes" method of the ZCatalog (by replacing the "manage_main" part of the URL with "manage_convertIndexes") - depending on the size of your ZCatalog the migration may take a while - after the conversion a message will appear: "Indexes converted and reindexed" 3. CMF doesn't work! Check that you have a recent stable version of CMF. As of Zope 2.8.0, you need at least CMF-1.4.8. CMF is available at http://www.zope.org/Products/CMF/ 4. Plone doesn't work! As of the release of Zope 2.8.0, Plone does not support Zope 2.8 yet. Check http://www.plone.org for updates. General 1. My browser is giving a message, 'Sorry, an error occurred.' How do I find out what the real error is? Step number one in these scenarios is to check the error log on your server. The standard error page should provide you with a link directly to the log entry for that error. If something has gone wrong with the standard error page itself, or if you are using a modified error page, this link may be missing. If this happens, you can visit /event_log/manage_main and browse the recent errors. In general, error_log will show the *traceback* for errors. This traceback explains the error message. Occassionally you won't see any error message. Rather, the HTML source will contain the last good HTML returned by the request before the error occurred. This is due to caching by the browser. Clear your cache and view the HTML source again. 2. I'm using Python 2.4 and I'm having a problem ... As of the release of Zope 2.8, Python 2.4 has not been approved for use with any version of Zope yet. Please use Python 2.3. The correct version of Python should always be automatically detected and used when you configure and install Zope. In general, you should let the configure script do its job and don't try to force an unsupported version of Python! Zope-2.8.7-final/doc/CHANGES.txt0000664000076400007640000013024510435502510015116 0ustar ajungajungZope Changes This file contains change information for the current Zope release. Change information for previous versions of Zope can be found in the file HISTORY.txt. To-do - Add cyclic-garbage collection support to C extension classes, especially to acquisition wrappers. N.B: ExtensionClassType already declares that it supports GC (via the Py_TPFLAGS_HAVE_GC flag), but does not appear to conform to the rules for such a type laid out in the Python docs: http://docs.python.org/api/supporting-cycle-detection.html Zope 2.8.7 (unreleased) Features added: - ZODB 3.4.3 Bugs fixed - Collector #1429: Fix name/value traversal of ZSQL methods (backport from 2.9). - Collector #1447: When editing content on a virtual-hosted zope, AcceleratedHTTPCacheManager now purges the correct URL. - Collector #2072: Applied patch to fix problem with overly restrictive __bobo_traverse__ security and tests. - Applied patch by Yoshinori Okuji to fix some XML export/import problems - Collector #2037: fixed broken ACTUAL_URL for '/' - Collector #2039: 'ZPublisher.HTTPRequest.HTTPRequest._authUserPW' choked on passwords which contained colons. - Missing import of NotFound in webdav.Resource. - OFS.Image: 'Image.update_data' did not refresh the Etag. - Collector #940: PageTemplateFile: Open files with universal line-endings support to avoid line-endings problems within python expressions. - OFS.PropertyManager: Provided missing security declarations. Zope 2.8.6 (2006/02/25) Bugs Fixed - Collector #1819: fixed signature of MountedObject.SimpleTrailblazer._construct( - ZPublisher.BaseRequest: The publisher would happily publish attributes of type 'bool' and 'complex', as well as Python 2.4's 'set' and 'frozenset'. - Collector #1991: ZPublisher did not deal properly with a trailing %20 in the URL - Collector #2013: improved XHTML conformance of error messages, some of which did not close '

' tags. - Collector #1999: fixed broken FTP rename functionality (RNFR now returns 350 as status code instead 250) - Collector #2002: fixed broken 'ls -R' functionality (didn't recurse properly subclasses of OFS.Folder) Zope 2.8.5 (2005/12/19) Bugs Fixed - Collector #1939: When running as a service, Zope could potentially collect too much log output filling the NT Event Log. When that happened, a 'print' during exception handling would cause an IOError in the restart code causing the service not to restart automatically. Problem is that a service/pythonw.exe process *always* has an invalid sys.stdout. But due to the magic of buffering, small "print" statements would not fail - but once the file actually got written to, the error happened. Never a problem when debugging, as the process has a console, and hence a valid stdout. - Collector #1976: FTP STOR command would load the file being uploaded in memory. Changed to use a TemporaryFile. - Collector #1904: On Mac OS X avoid a spurious OSError when zopectl exits. - CopySupport: Reverted workaround in '_verifyObjectPaste'. 'checkPermission' now respects proxy roles, so the warkaround introduced to fix http://www.zope.org/Collectors/Zope/78 is no longer needed. Meta types listed in all_meta_types need a 'permission' key. - Collector #1774: Harmonize the implementation of AccessControl.ZopeSecurityPolicy.checkPermission with 'validate', checking ownership and proxy roles if an executable object is on the stack. - AccessControl.SecurityInfo: Fixed problem with setPermissionDefault when the permission wasn't used anywhere else in the class to protect methods. - Collector #1957: Made ZPublisher.HTTPResponse._error_html return conformant XHTML. - Collector #1891: Backported changes to ZCatalog regression tests, removing use of 'whrandom' (and its 'seed' function). - Collector #1621, #1894: Added BBB alias for 'whrandom' in AccessControl/DTML.py and RestrictedPython/Utilities.py. The alias will be removed in Zope 2.10. - Collector #1954: DocumentTemplate.DT_String: remove non-XHTML wart from error message. - Fixed unclear security declarations. Warn when an attempt is made to have a security declaration on a nonexistent method. - OFS Application: While deprecated since years, old-style product metadata in the __init__.py did not show deprecation warnings. Added warnings and converted ZGadflyDA/__init__.py and ZSQLMethods/__init__.py to use registerClass instead. Other - The SiteErrorLog now copies exceptions to the event log by default. - ObjectManager now has an hasObject method to test presence. This brings it in line with BTreeFolder. - Made 'zopectl test' work for software homes which do not have an "inplace" build (it used to require that test.py be in $ZOPE_HOME/bin/; now it will use $ZOPE_HOME as a fallback). - Improved logging of ConflictErrors. All conflict errors are logged at INFO, with counts of how many occurred and how many were resolved. Tracebacks for all conflicts are logged a DEBUG level, although these won't help anyone much. If a conflict error is unresolved, it will now bubble up to error_log and standard_error_message. - Suppressed output of deprecation warning in 'filepath' test for PythonScripts. Zope 2.8.4 (2005/10/26) Bugs Fixed - Collector #1927: Modified ZReST not to store rendered / warnings as persistent attributes, using volatile attributes instead as a cache. - Collector #1926: fixed a typo in _doAddUser when password encryption is enabled. - If a content object implemented any in-place numeric operators, untrusted code could call them, thus modifying the content. - If Python 2.4 is used, despite the fact that Python 2.4 is unsupported, untrusted code could use generator expressions to gain access to container items. Zope 2.8.3 (2005/10/18) Other - ZSQLMethod.manage_main: Moved the error message that warns of a non-existing or closed database connection next to the Connection ID dropdown and present it using red to increase its visibility. - Update to Docutils 0.3.9 (forgotten in Zope 2.8.2) Zope 2.8.2 (2005/10/13) Features added - Collector #1118: Added syntax to dtml-sqlgroup to support flexible generation of 'UPDATE' statements (bounty sponsored by Logicalware). - The '@' character is now allowed in object ids (RFC 1738 allows it). Bugs Fixed - OFS.Image.manage_FTPget() would str() it's .data attribute, potentially loading the whole file in memory as a string. Changed to use RESPONSE.write() iterating through the Pdata chain, just like index_html(). - When PageTemplates have a syntax error, show the traceback output in the rendered error message. - Collector #1914: Hardened 'call_with_ns' (in 'Products.PageTemplates.ZRPythonExpr') against namespaces from other callers than page templates. - Collector #1490: Added a new zope.conf option to control the character set used to encode unicode data that reaches ZPublisher without any specified encoding. - Collector #1895: omit 'var' folder from recursive traversal causing trouble with DirectoryStorage. - disabled ".. include" directive for all the ZReST product and the reStructuredText package - Collector #1888: Adjust call to 'engine.translate' to accomodate change in its signature. - Collector #1863: Prevent possibly sensitive information to leak via the TransientObject's __repr__ method. - Repaired 'handle_errors' usage for doctests, along with the supporting 'debug' argument passed to 'ZPublisher.Test.publish_module'. - Collector #1879: applied patch by Dieter Maurer to fix a bug in ac_aquire() ignoring the default argument - Collector #1864, #1906: fixed header normalization in appendHeader() - Collector #1899: fixed migration issue when using export/import for ZCatalog instances - 'ZPublisher.Test.publish' now takes a 'done_string' argument, which is written to standard error when the request completes (forward ported from Zope 2.7 branch). - Collector #556: now returns 'null' instead of 'None' for values which are None in Python (sponsored by a bounty from Logicalware). - Collector #1182: BBB Forward port fix from 2.7 branch (19 months ago!), reverting 'guarded_getitem' to pass the 'index' argument as the name to 'validate'. This change is *not* propagated to the trunk, because the resolution of #1182 specifies that the reverted behavior (i.e., passing None for item accces) is to become the standard implementation as of 2.9. - Collector #1877: skel/Products/README.txt inappropriately copied from CMF. - Collector #1871: Applied patch to support lists with records using ZTUtils.make_query() - AccessControl: creating a new user through "zpasswd inituser" did not work properly with a top-level user folder with enabled password encryption. - ZCatalog: refreshCatalog() could not be called safely from a ZEO client script - Catalog.clear(): fixed handling of _length attribute (caused import problems for some .zexp files e.g. Squishdot instances) - DateIndex now properly removes documents from both indexes if the value is None - Collector #1888: Some parts of the TALInterpreter would not pass a default when translating, yet expect a string back. This would cause an error (usually "NoneType has no attribute 'replace'") in the case the message was not translated. Zope 2.8.1 (2005/08/11) Features added - Interface: Added Z3 -> Z2 bridge utilities. This allows to migrate interfaces to Zope 3 style interfaces and bridge them back to oldstyle interfaces for backwards compatibility. Bugs Fixed - Zope2.Startup.zopectl: fork before execv when running unit tests (don't exit the shell, if run from there). - TAL: MassageIDs are now handled the same way as in zope.tal. - DocumentTemplate: ustr no longer mangles MassageIDs. Custom string types are now returned unchanged. - As developed in a long thread starting at http://mail.zope.org/pipermail/zope/2005-July/160433.html there appears to be a race bug in the Microsoft Windows socket implementation, rarely visible in ZEO and/or in ZServer/medusa/thread/select_trigger.py when multiple processes try to create an "asyncore trigger" simultaneously, most often (in stress tests) manifesting as a hung process. Windows-specific trigger code in both changed to work around this bug when it occurs. - Collector #1807: fixed memory leak in cAccessControl.guarded_getattr() Zope 2.8.1 b1 (2005/07/28) Features Added - PluginIndexes, ZCTextIndex and ZCatalog: Added some z3 interfaces. - Verbose security exception reporting has been folded into Zope, removing the need for the VerboseSecurity product. See the documentation for the "verbose-security" option in zope.conf. - "TemporaryStorage" (the storage that is used mainly to back the default sessioning database) is now MVCC capable, which essentially means that its usage will no longer generate ZODB ReadConflictErrors. Bugs Fixed - Collector #1852: fixed wrong URL construction in webdav.davcmds - Collector #1844: fixed whitespace handling in the ZMI "Find" tab - Collector #1813: removed spurious inclusion of CMFBTreeFolder. in Products/BTreeFolder2 (CMFCore will include it after 1.5, with an appropriate module alias for backward compatibility). - Replaced all transaction.commit(1) calls by transaction.savepoint() - Collector #1832: UnIndex swallowed ConflictErrors. - Collector #1815: ZCTextIndex accepts (again) sequences of strings to be indexed. - Collector #1812: Fixed key error in ZSQL ZMI/Test - Fixed CMFBTreeFolder for CMF 1.5+ - WebDAV COPY and MOVE did not call '_notifyOfCopyTo' and '_postCopy' hooks like it was done in OFS.CopySupport. Additionally added 'manage_changeOwnershipType' to make MOVE behave even closer to OFS.CopySupport. - Collector #1548: Fix 'httplib' usage in ZPublisher.Client. - Collector #1808: manage_convertIndexes no longer tries to change the index types causing some trouble with CMF. - manage_convertIndexes did not treat DateRangeIndexes and PathIndexes properly. - Updated Zope X3 to bugfix release 3.0.1 - Updated Five to bugfix release 1.0.2 (see Products/Five/CHANGES.txt) Zope 2.8.0 (2005/06/11) Bugs Fixed - Collector #1792: applied patch for broken ZClasses - doc/FAQ.txt updated: should bear some resemblance to reality now. (PCGI stuff removed; error information updated; PID information updated; upgrade procedure added; some common version questions added.) - Collector #1770: Fixed RestructuredText subtitle - Collector #1803: Fixed InitializeClass for some corner case. - Collector #1798, issue 1: ZopeTestCase no longer tries to install products that were installed by Zope during startup. - Collector #1799: Avoid lying about parent's refcount when calling back into Python code. - Collector #889: made 'and' operator for KeywordIndexes actually restrict results as expected (thanks to 'aroda' for the patch!). - Collector #1323: applied patch to fix umask problem in zdctl - Updated Five to bugfix release 1.0.1 (see Products/Five/CHANGES.txt) Zope 2.8.0 b2 (2005/05/22) Features added - Made WebDAV server distinguishable from the default HTTP server both in the ZMI and in event.log. - Included BTreeFolder2 Bugs fixed - Collector #1507/1728: Server addresses are now handled the same way on all platforms. This fixes the default binding on Windows. - Collector #1781: made 'create_mount_points' ZConfig option actually work (thanks to Dieter Maurer for the patch). - Collector #1780: DateTime.strftime() now handles dates <= 1900 or >= 2038 - Collector #1775: turning off debug mode by default - Collector #1784: fixed handling of multiple attributes in ZCTextIndex - Don't copy '.svn' directories from skeleton into an instance (thanks to Dale Hirt for the patch). - Collector #1776: Improved setup.py. The Finder class is now used for the complete lib/python tree and has a blacklist instead of a whitelist for file extensions. So there should no longer be a need to update setup.py if modules or files are added or removed in lib/python. - Collector #1751: Improved error reporting reporting during the startup phase - Collector #1745: Fixed ZSQL error KeyError 'query' - Collector #1735: fixed UnicodeDecodeError in Loader.py Zope 2.8b1 (2005/04/24) Features added - Added lazy: TAL expression and fixed defer: expression for python expression - ZCatalog.CatalogBrains: An _unrestrictedGetObject method has been added. - ZODB transactions now support savepoints. See transaction/savepoint.txt. These will replace subtransactions. Bugs fixed - Collector #1754: Fixed import of 'transaction' in 'zopectl adduser' (which wasy dying with a NameError). - Collector #1750: StructuredText: fixed handling of image URLs with query string - Collector #1748: Fixed SIGSEGV in Acquisition - Hotfix_20050405: classes defined in untrusted code could shadow the roles of methods defined as protected by their bases. - Collector #1656: Fixed enumeration within untrusted code (forward-port from 2.7 branch). - Collector #1721: Fixed handling of an empty indexed_attrs parameter Zope 2.8a2 (2005/04/02) Features added - ZCatalog.CatalogBrains: 'getObject' now raises errors, rather than returning None, in cases where the path points either to a nonexistent object (in which case it raises NotFound) or to one which the user cannot access (raising Unauthorized). Sites which rely on the old behavior can restore setting a new zope.conf option, 'catalog-getObject-raises', to "off". This compatibility option will be removed in Zope 2.10. - PluginIndexes: the ZCatalog's "Indexes" tab now show the number of distinct values indexed by each index instead of a mixture of indexed objects versus number of distinct values. Indexes derived from UnIndex show both values within their own ZMI screen. In addition most indexes have now a "Browse" tab to browse through the list of indexed values and their occurrences. - FTPServer: a RNFR (rename from) request is now being responded with a 550 error code if the source file does not exist - Fixed ObjectManager to not swallow exceptions during object deletion (in debug mode and if the user is not Manager). This allows for better debugging, while still keeping the possibility for a Manager to delete buggy objects. - Added a ZConfig directive 'large-file-threshold' to control the request content-size threshold at which a temporary file gets created. Use the same value for deciding between reading the whole request in memory or just a chunk inside webdav.NullResource.PUT(). - RAMCacheManager: Allow invalidation of a cache entry from the Statistics view in the ZMI - Collector #1454/OFS.File: Accept content types ending with "javascript" as editable through the File edit form, just like text/ types - Zope X3 3.0.0's 'src/zope' package is included now. - Five (Zope 3 integration technology for Zope 2) is included now in Products/Five. Bugs fixed - Collector #1460: guarded_apply was too restrictive. - OFS.Traversable still used a string 'NotFound' exception. - ZPublisher would fail to recognize a XML-RPC request if the content-type header included a 'charset' parameter. - Forward-ported 'aq_acquire'-related fix and associated tests from Zope 2.7.4. - Collector #1730: XML page templates couldn't call aq_parent in path expressions. - Fixed brain.getObject() to correctly traverse to an object even if one of its parents is not accessible, to be close to what the Publisher does. - Forward ported fix for OFS.CopySupport tests which corrected signature of a faux security policy's 'validate' method. - 'setup.py' did not install the 'Zope' compatibility module (the old 'Zope' package has been renamed to 'Zope2'). - Fixed Shared.DC.ZRDB.Results to behave with the new-style ExtensionClass. Added a test. - 'setup.py' did not install the new 'Zope' compatibility module (the 'Zope' package has been renamedd to 'Zope2'). - Collector #1507: Zope now binds again to all available IP addresses if ip-address is unset - Use 'del' instead of 'list.remove()' in Catalog.delColumn(). There can be only one column with the same name, and it could potentially break catalog metadata as remove() may remove more than one element from the list if they have the same value. Also, we already have the list index we are interested in deleting so it doesn't make sense to look up the value and call 'list.remove()' on it. - Collector #1628: FTP server has been broken (directory listings did not work) - Collector #1705: CopySource._postCopy is never called - Collector #1617: Fixed crash in ZPT code (caused by improper checks in cAccessControl) - Collector #1683: fixing batching in the DA "Test" tab - Collector #1648: Fix bug in Medusa FTP - Collector #1667: allow 'max-number-of-session-objects 0' to have the same effect as setting the value via the web interface (i.e., make the number of session objects unlimited, rather than falling back to the default). - Collector: #1651: removed compiler warning - Collector #1661: make 'python-check-interval' setting in zope.conf actually work as documented. This setting allows for important tuning opportunities for production Zope servers. - Collector #1657: Don't break host-based virtual hosting when purging an HTTP accelerator. - DTML Methods were not interoperable with the new filestream_iterator and caches based on it (FileCacheManager). - Collector #1655: fixed severe memory leak in TemporaryStorage - Collector #1407: fixed XML escaping problem introduced in 2.7.4 b1 - Collector #1151: HTTP compression was broken on error pages - The REQUEST now contains a new entry ACTUAL_URL which contains the full URL without query string as it appears within the location bar of the browser. The key has been added to provide a single key that is available for vhosted and non-vhosted installations. - Collector #1605: VHM did not quote URLs - webdav.Resource: during COPY, manage_afterClone was called way too early, thus the object wasn't bound to the database and couldn't find a context. Changed to behave the same way as CopySupport. - RAMCacheManager: opimized performance by using cPickle instead of pickle and by using the highest pickle protocol available instead of using ASCII pickles (patch by Dieter Maurer) - Collector #631: Image URLs in StructuredText containing port numbers were not rendered correctly - Collector #1498: Don't choke on malformed cookies. Cookies of the form "foo=bar; hmm; baz=gee" will give an empty value for 'hmm' instead of silently discarding it and the rest of the string. (Thanks to 'sirilyan' for the patch.) - bin/zopectl test now uses os.execv, instead os os.system, so that options with characters that needs shell quoting doesn't break the command. - Collector #1219: Make XML export sane again. - Collector #945: Allow adding empty PythonScript instances programmatically. - Updated doc/UNITTEST.txt and lib/python/Testing/README.txt to reflect progress made since UNITTEST.txt was originally written. - Removed Version objects from the add menu. Versions are agreed to be a feature that should not be used as it is not well implemented and allows for data loss. - Collector #1510: Allow encoding of application/xhtml+xml pages according to the charset specified in the Content-Type header (thanks to Jacek Konieczny for the patch). - Collector #1599: made sqltest work with unicode strings (thanks to Peter Sabaini for the patch). - zopectl: fixed handling of child processes (patch by Dieter Maurer) - Collector #1593: fixed dumb _get_id() implementation in OFS.CopySupport that produced copy_of_copy_of....files (thanks to Alexandre Boeglin for the patch). - Collector #1450: files in utilities/ZODBTools are now installed during the installation process in the 'bin' directory - Collector #1003: added new 'http-header-max-length' directive to zope.conf to specific the maximum length of a HTTP request header before it is considered as a possible DoS attack and discarded. - Collector #1371: added new 'cgi-maxlen' directive to zope.conf to limit the amount of form data being processed by Zope to prevent DoS attacks - Collector #1407: changed WebDAV display name for objects to title_or_id() - the 'trusted-proxy' directive in zope.conf now also accepts hostnames instead of IP addresses only (patch by Dieter Maurer) - Fixed test.py to not over-resolve symbolic links. Needed to run tests when the Products directory and a product are symlinks. - Collector #1583/ZReST: Fixed handling of the title attribute for non-ascii characters. - Collector #1577: Fixed cryptic error message in ZPublisher if a non-ASCII string is passed to a date, int, long or float property. - Collector #1576: Fixed Z Search Interface to use proper HTML. - Collector #1127: strftime did not take timezone into account. - Collector #1569/DateTime: Added a new ISO8601-method that will return correctly formatted ISO 8601-representations to augment the ISO method which isn't compliant with ISO 8601. - ZPublisher: changed some hardcoded 'latin1' arguments to 'iso-8859-15' since latin1 is obsolete. - Collector #1566: Installation of Zope on some older Solaris versions could fail due to a broken "echo" implementation causing the creation of a borked version.txt file. - Collector #934: Image and File objects are now always internally split into small chunks even when initialized from a string. - docutils: updated to V 0.3.5. The Zope core now contains a full copy of the docutils package except some GPLed files which can not be included with the Zope distribution due to license constraints on svn.zope.org. - docutils: moved from lib/python/docutils to lib/python/third_party/docutils - Collector #1557/OFS.Image: Introducing new 'alt' property. The 'alt' attribute is no longer taken from the 'title' property but from the new 'alt' property. The border="0" attribute is no longer part of the HTML output except specified otherwise. - Set a default value of '' for the new 'alt' property as not to break existing content. - Collector #1511: made IPCServer show up in the Control Panel under "Network Services" - Collector #1443: Applied patch by Simon Eisenmann that reimplements the XML parser used in WebDAV fixing a memory leak. - Always unescape element contents on webdav.xmltools - Use saxutils to escape/unescape values for/from PROPFIND/PROPPATCH. - Make OFS.PropertySheet use the escaping function from webdav.xmltools. - Escape/unescape " and ' - Don't escape properties stored as XML (ie: having a __xml_attrs__ metadata set by PROPPATCH) when building a PROPFIND response. - If a PROPPATCH element value contains only a CDATA section, store the CDATA contents only. - Catch AttributeErrors and KeyErrors raised from __bobo_traverse__ and convert them to NotFound. In debug mode a more verbose error message is issued, the same way it's done on attribute/item traversal. - Collector #1523: replace the text field for importing .zexp/.xml files with a selection list - Stitch newly-created object into it's container *before* calling it's PUT() method. This fixes an issue with OFS.File/OFS.Image that would result into reading the whole file in memory and wrapping it into a *single* Pdata object. - Import ZServer.CONNECTION_LIMIT variable *inside* the method that uses it. Before this, the variable was imported at the module level, thus binding it too early which would cause the ZConfig handler to have no real effect. Zope 2.8a1 (2004/10/17) Features added - Included Stefan Holek's ZopeTestCase 0.9 - The SiteErrorLog allows you to acknowledge (or delete) exceptions, so you can reduce or clear the list without restarting your Zope server. Additionally the SiteErrorLog is covered by unit tests now. - Unit tests added for the SiteErrorLog. - UI improvement for the ZCatalog. The "catalog contents" allow you to filter the cataloged objects by path now. - Made test.py follow symbolic links on POSIX systems. - added utilities/reindex_catalog.py to perform ZCatalog maintenance operations from the command line (through zopectl) - RESPONSE.setBody and RESPONSE.setStatus now accept lock parameters in the same way as RESPONSE.redirect. These prevent further calls to the methods from overwriting the previous value. This is useful when writing http proxies. - DateTime: new DateTime instance can be constructed from a given DateTime instance: d_new = DateTime(d_old) - The DateTime parser now throws a SyntaxError upon any parsing errors. - ZCatalog: added a new configuration option in the "Advanced" tab to provide optional logging of the progress of long running reindexing or recataloging operations. - made Zope.configure return the starter instance to enable other methods to be called, such as starter.setupConfiguredLoggers() - Improved Unicode handling in Page Templates. Template contents and title will now be saved as a Unicode string if the management_page_charset variable can be acquired and is true. The character set of an uploaded file can now be specified. - zopectl now accepts the -m argument to set a umask for files created by the managed process (e.g. -m 002 or --umask 002). - AccessControl/permission_settings() now has a new optional parameter 'permission' to retrieve the permission settings for a particular permission. - The obsolete 'SearchIndex' package has been removed - Traversal now supports a "post traversal hook" that get's run after traversal finished and the security context is established. - Using "_usage" parameters in a ZCatalog query is deprecated and logged as DeprecationWarning. - MailHost now has two additional properties, a user id and a password. These are used to attempt ESMTP authentication before sending a mail. - Folder listings in FTP now include "." as well as "..". - When a VHM is activated, it adds the mapping 'VIRTUAL_URL_PARTS': (SERVER_URL, BASEPATH1, virtual_url_path) to the request's 'other' dictionary. If BASEPATH1 is empty, it is omitted from the tuple. The joined parts are also added under the key 'VIRTUAL_URL'. Since the parts are evaluated before traversal continues, they will not reflect modifications to the path during traversal or by the addition of a default method such as 'index_html'. - Extension Classes, a key Zope foundation, have been totally rewritten based on Python new-style classes. This change provides a number of advantages: o Use of new-style class features (e.g. slots, descriptors, etc.) in Zope objects. Support for object protocols (special __ methods) added since Python 1.4. o Support for cyclic garbage collection. o Ability to use new-style classes as base classes of Zope objects. o Pave the way for sharing code between Zope 2 and Zope 3. Note -- Extension classes with __of__ methods are made into Python read descriptors. If an extension classes is used to implement a descriptor, indirectly by implementing __of__ or directly by implementing __get__, the behavior of the descriptor will differ from ordinary descriptors in an important way. The descriptors __get__ method will be called *even if* the descriptor is stored on an instance of an extension class. Normally descritor __get__ methods are called only of the descriptor is stored in a class. - ZODB 3.3 This is the first version of ZODB that does not require ExtensionClass. - Add 'parity' method to ZTUtils Iterators. - Allow untrusted code to mutate ZPublisher record objects. - Added a "mime-types" configuration value which names a file giving additional MIME type to filename extension mappings. The "mime-types" setting may be given more than once in the configuration file; the files have the same format at the mime.types file distributed with Apache. - Changed the ZEO server and control process to work with a single configuration file; this is now the default way to configure these processes. (It's still possible to use separate configuration files.) The ZEO configuration file can now include a "runner" section used by the control process and ignored by the ZEO server process itself. If present, the control process can use the same configuration file. - ZConfig was updated to version 2.0. The new version includes two new ways to perform schema extension; of particular interest in Zope is the ability for a configuration file to "import" new schema components to allow 3rd-party components (such as storages, databases, or logging handlers) to be used. - The testrunner.py script has been replaced with test.py which is now installed into the 'bin' folder. Bugs fixed - Removed Python 2.3.3 as valid option. ZODB 3.3 requires Python 2.3.4 or later. - Collector #1332: Added in-place migration of the Catalog.__len__ attribute to avoid new-style class caching problems. Instances of ZCatalog or instances of classes with ZCatalog as base class will be migrated automatically. Instances of Catalog or classes with Catalog as base class must be migrated manually by calling the migrate__len__() method on the every instance. In addition old BTree migration code (for pre-Zope 2.5 instances) has been removed. If you want to migrate from such an old version to Zope 2.8, you need to clear and reindex your ZCatalog). - Collector #1595: same as in Collector #1132 for indexes derived from UnIndex. Exisiting ZCatalog instances must be converted manually by calling the "manage_convertIndexes" method through-the-web for every single ZCatalog instance. See also doc/FAQ.txt (Installation, question #4) - Collector #1457: ZCTextIndex's QueryError and ParseError are now available for import from untrusted code. - Collector #1473: zpasswd.py can now accept --username without --password - Collector #1491: talgettext.py did not create a proper header for the generated .pot file if multiple pagetemplate files were processed. - Collector #1477: TaintedString.strip() now implements the same signature as str.strip() - TAL: tal:on-error does not trap ConflictError anymore. - OFS.CopySupport: Enforced "Delete objects" permission during move (CMF Collector #259). - Removed DWIM'y attempt to filter acquired-but-not-aceessible results from 'guarded_getattr'. - Collector #1267: applied patch to fix segmentation faults on x86_64 systems - ZReST: the charset used in the rendered HTML was not set to the corresponding output_encoding property of the ZReST instance. In addition changing the encodings through the Properties tab did not re-render the HTML. - Collector #1234: an exception triple passed to LOG() was not propagated properly to the logging module of Python - Collector #1441: Removed headers introduced to make Microsoft webfolders and office apps happy, since they make a lot of standards-compliant things unhappy AND they trick MS Office into trying to edit office files stored in Zope via WebDAV even when the user isn't allowed to edit them and is only trying to download them. - Collector #1445: Fixed bad interaction between -p and -v(v) options to test.py that resulted in exceptions being printed when they shouldn't have been. - Collector #729: manage_main doesn't display the correct page title most of the time. It is not completely fixed but using title_or_id makes folders display the correct id as a fallback. - Collector #1370: Fixed html generated by Z Search interface. - Collector #1295: Fixed minor niglet with the Elvis tutorial. - added "version.txt" to setup.py to avoid untrue "unreleased version" messages within the control panel - Collector #1436: applied patch to fix a memory leak in cAccessControl. - Collector #1431: fixed NetBSD support in initgroups.c - Collector #1406: fixed segmentation fault by acquisition - Collector #1392: ExternalMethod ignored management_page_charset - unrestrictedTraverse() refactored to remove hasattr calls (which mask conflict errors) and for greater readability and maintainability. - Zope can now be embedded in C/C++ without exceptions being raised in zdoptions. - Collector #1213: Fixed wrong labels of cache parameters - Collector #1265: Fixed handling of orphans in ZTUtil.Batch - Collector #1293: missing 'address' parameters within one of the server sections raise an exception. - Collector #1345: AcceleratedHTTPCacheManager now sends the Last-Modified header. - Collector #1126: ZPublisher.Converters.field2lines now using splitlines() instead of split('\n'). - Collector #1322: fixed HTML quoting problem with ZSQL methods in DA.py - Collector #1124: The ZReST product now uses the same reST encoding parameters from zope.conf as the low-level reStructuredText implementation. - Collector #1259: removed the "uninstall" target from the Makefile since the uninstall routine could also remove non-Zope files. Because this was to dangerous it has been removed completely. - Collector #1299: Fixed bug in sequence.sort() - Collector #1159: Added test for __MACH__ to initgroups.c so the initgroups method becomes available on Mac OS X. - Collector #1004: text,token properties were missing in PropertyManager management page. - Display index name on error message when index can't be used as 'sort_on'. - PUT would fail if the created object had a __len__ = 0 (eg: BTreeFolder2) and fallback to _default_put_factory. Fix by checking if the returned object is None instead. - Collector #1160: HTTPResponse.expireCookie() potentially didn't when an 'expires' keyword argument was passed. - Collector #1289: Allow ZSQL methods to be edited via WebDAV. - WebDAV property values were not being properly escaped on 'propstat'. - WebDAV 'supportedlock' was not checking if the object did implement the WriteLockInterface before returning it's value. - reStructuredText ignored the encoding settings in zope.conf - ObjectManager no longer raises string exceptions - Collector #1260: Testing/__init__.py no longer changes the INSTANCE_HOME. - App.config.setConfiguration() did not update the legacy source for debug_mode, Globals.DevelopmentMode. - Script (Python) objects now have a _filepath attribute, also used as the '__file__' global at runtime. This prevents an import problem caused by the fix to #1074. - Minor usability tweaks: * Increased FindSupport meta type selection widgets height to 8 lines - The DateTime module did not recognize the settings for "datetime-format". - Stop testrunner.py from recursing into the 'build-base' directory created by setup.py. - Collector #1074: Change Scripts' __name__ to None - Range searches with KeywordIndexes did not work with record-style query parameters - Item_w__name__ now has a working getId() method - PageTemplateFile now using Item_w__name__ mixin, fixing its getId() and absolute_url() methods. - Only one VirtualHostMonster is allowed per container. - Collector #1133: TreeTag choked on Ids of type long. - Collector #1012: A carefully crafted compressed tree state could violate size limit. Limit is no longer hardcoded. - Collector #1139: tal:attributes didn't escape double quotes. - Management interface of TopicIndexes has been completely broken - Collector #1129: Improper parsing of ISO8601 in DateTime. - Removed pervasive use of string exceptions (some may still be hiding in the woodwork, but all raise's with string literals are gone). - AccessControl.User used a misleading string exeception, 'NotImplemented', which shadowed the Python builtin. - Collector #426: Inconsistent, undocumented error() method. - Collector #799: Eliminate improper uses of SCRIPT_NAME. - Collector #445: Add internal global declaration for Script bindings. - Collector #616: Make CONTEXTS available to TALES Python expressions. - Collector #1074: Give Script execution context a __name__ - Collector #1095: Allow TAL paths starting with '/varname' as a preferred spelling for 'CONTEXTS/varname'. - Collector #391: Cut and paste now requires delete permissions. - Collector #331: Referenses to URL in manage_tabs was changed to REQUEST.URL to prevent accidental overriding. - Made the control panel properly reflect the cache-size setting of ZODB's object cache once again. - ConflictError was swallowed in ObjectManager by manage_beforeDelete and _delObject. This could break code expecting to do cleanups before deletion. - Python 2.3 BooleanType wasn't handled properly by ZTUtils marshalling and ZPublisher's converters. - Collector #1065: bin/ scripts didn't export HOME envars. - Collector #572: WebDAV GET protected by 'FTP Access' permission. Two new methods have been added to WebDAV resources, "manage_DAVget" and "listDAVObjects". These are now used by WebDAV instead of the earlier "manage_FTPget" and "objectValues". This separates the permissions, and allows WebDAV specific overriding of these methods. - Collector #904: Platform specific signals in zdaemon/Daemon.py (fixed by removing the "fossil" module from 2.7 branch and head). - Workaround for Collector #1081: The 'title' property for objects derived from OFS.Folder or PropertyManager can now be removed and replaced with a ustring property. This allows the usage of non-ISO-8859-1 or ASCII charsets - Collector #951: DateTime(None) is now equal to DateTime() - Collector #1056: aq_acquire() ignored the default argument - Collector #1087: ZPT: "repeat/item/length" did not work as documented in the Zope Book. - Collector #721: Entities in tal:attribute values weren't properly escaped. - Collector #851: Traversable.py: A bare try..except shadowed conflict errors - Collector #1058: Several fixes for PropertySheets when used outside ZClasses - Collector #1053: parseIndexRequest turned empty sequence of search terms into unrestricted search. - manage_tabs had a namespace problem with the acquisition of names from the manage_options variable resulting to acquire "target" and "action" from objects above in the hierachy. - PathIndex and TopicIndex are now using a counter for the number of indexed objects instead of using a very expensive calculation based on the keys of their indexes. - Collector #1039: Whitespace problem in Z2.log fixed - changed some bare try: except:'s in Shared.DC.ZRDB.Connection so that they now log exceptions that occur. - ObjectManager will now attempt to set Owner local role keyed to the user's id, rather than username. Zope-2.8.7-final/doc/SECURITY.txt0000664000076400007640000000676710435500316015332 0ustar ajungajungSetting the initial user name and password Because Zope is managed through the web, user names and passwords must be used to assure that only authorized people can make changes to a Zope installation. Some user name and password is needed to "bootstrap" the creation of normal managers of your Zope site. This is accomplished through the use of the file 'inituser'. The first time Zope starts, it will detect that no users have been defined in the root user folder. It will search for the 'inituser' file and, if it exists, will add the user defined in the file to the root user folder. Normally, 'inituser' is created by the Zope install scripts. Either the installer prompts for the password or a randomly generated password is created and displayed at the end of the build script. You can use the 'zpasswd.py' script to create 'inituser' yourself. Execute 'zpasswd.py' like this:: python zpasswd.py inituser The script will prompt you for the name, password, and allowed domains. The default is to encode the password with SHA, so please remember this password as there is no way to recover it (although 'zpasswd.py' lets you reset it.) In some situations you may need to bypass normal security controls because you have lost your password or because the security settings have been mixed up. Zope provides a facility called an "emergency user" so that you can reset passwords and correct security settings. The emergency user password must be defined outside the application user interface. It is defined in the 'access' file located in the Zope directory. It should be readable only by the user as which your web server runs. To create the emergency user, use 'zpasswd.py' to create the 'access' file like this:: python zpasswd.py access In order to provide a somewhat higher level of security, various encoding schemes are supported which provide access to either SHA-1 encryption or the standard UNIX crypt facility if it has been compiled into Python. Unless you have some special requirements (see below), you should use the SHA-1 facility, which is the default. Format of 'inituser' and 'access' A password file should consist of a single line of the form: name:password Note that you may also add an optional third component to the line in the access file to restrict access by domain. For example, the line: mario:nintendoRules:*.mydomain.com in your 'access' file will only allow permit emergency user access from *.mydomain.com machines. Attempts to access the system from other domains will fail, even if the correct emergency user name and password are used. Please note that if you use the ZServer monitor capability, you will need to run with a clear text password. Setting permissions on the var directory. You need to set permissions on the Zope var directory. Zope needs to read and write data from its var directory. Before running Zope you should ensure that you give adequate permissions to the Zope var directory for the userid Zope will run under. Depending on how you choose to run Zope you will need to give different permissions to the var directory. If you use Zope with an existing web server, it will probably run Zope as 'nobody'. In this case 'nobody' needs read and write permissions to the var directory. If you change the way you run Zope you may need to modify the permissions of the var directory and the files in it to allow Zope to read and write under its changed userid. Zope-2.8.7-final/doc/SETUID.txt0000664000076400007640000000267710435500316015054 0ustar ajungajungZope effective user support Zope can bind its network service to low ports such as 21 (FTP) and 80 (HTTP). In order to bind to low ports, Zope must be started as the root user. However, Zope will only run as root long enough to bind to these low ports. It will then attempt to setuid to a less privileged user. You must specify the user to which Zope will attempt to setuid by changing the 'effective-user' parameter in the zope.conf configuration file to an existing username or UID. All runtime files will be written as this user. If you do not specify an 'effective-user' in the configuration file, and you attempt to start Zope, it will refuse to start. Zope additionally emits a warning if you specify 'nobody' as the 'effective-user'. The rationale for this warning stems from the fact that, historically, many other UNIX services dropped privileges to the 'nobody' account after starting as root. Any security defects in these services could cause someone to gain access as the 'nobody' account on your system. If someone was to gain control of your 'nobody' account they could compromise your Zope files. The most important thing to remember about effective user support is that you don't have to start Zope as root unless you want to listen for requests on low ports (ports beneath 1024). In fact, if you don't have this need, you are much better off just starting Zope under a dedicated user account. Zope-2.8.7-final/doc/SIGNALS.txt0000664000076400007640000000157210435500316015150 0ustar ajungajungSignals (POSIX only) Signals are a POSIX inter-process communications mechanism. If you are using Windows then this documentation does not apply. Zope responds to signals which are sent to the process id specified in the file '$INSTANCE_HOME/var/Z2.pid':: SIGHUP - close open database connections, then restart the server process. A idiom for restarting a Zope server is: kill -HUP `cat $INSTANCE_HOME/var/z2.pid` SIGTERM - close open database connections then shut down. A common idiom for shutting down Zope is: kill -TERM `cat $INSTANCE_HOME/var/Z2.pid` SIGINT - same as SIGTERM SIGUSR2 - close and re-open all Zope log files (z2.log, event log, detailed log.) A common idiom after rotating Zope log files is: kill -USR2 `cat $INSTANCE_HOME/var/z2.pid` Zope-2.8.7-final/doc/LICENSE.txt0000664000076400007640000000447610435500316015140 0ustar ajungajungZope Public License (ZPL) Version 2.0 ----------------------------------------------- This software is Copyright (c) Zope Corporation (tm) and Contributors. All rights reserved. This license has been certified as open source. It has also been designated as GPL compatible by the Free Software Foundation (FSF). Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions in source code must retain the above copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name Zope Corporation (tm) must not be used to endorse or promote products derived from this software without prior written permission from Zope Corporation. 4. The right to distribute this software or to use it for any purpose does not give you the right to use Servicemarks (sm) or Trademarks (tm) of Zope Corporation. Use of them is covered in a separate agreement (see http://www.zope.com/Marks). 5. If any files are modified, you must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. Disclaimer THIS SOFTWARE IS PROVIDED BY ZOPE CORPORATION ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL ZOPE CORPORATION OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. This software consists of contributions made by Zope Corporation and many individuals on behalf of Zope Corporation. Specific attributions are listed in the accompanying credits file. Zope-2.8.7-final/doc/ZODB.txt0000664000076400007640000000627410435500316014612 0ustar ajungajungThe Zope Object Database, ZODB, version 3.0 ZODB is the next generation of our object database architecture. It provides a number of advantages over or existing databases: - Support for concurrency, - Well-defined storage management interface that will allow many different storage management strategies to be used, from file storage, to RDBMS storage, to memory storage, - More robust file storage format, - Much better version support and integrations of versions with the transaction system. For example, it is possible to commit from one version to another, to undo version commit and abort, and to use "temporary versions" to reduce memory use when manipulating large quantities of data. - Support for multiple databases in the same object system. - Support for multi-process storage managers, although the standard distribution will not include any multiple process storage managers. Note: Using ZODB 3 from Python In ZODB 2, you could access the top-level application object with:: import Main app=Main.app You could then navigate from the top-level object by getting attributes, calling methods, etc.. In ZODB 3, you get the top-level object like this:: import Zope app=Zope.app() ... do stuff with objects # and when you're done: app._p_jar.close() You have to import the Zope application module, which uses ZODB 3 rather than ZODB 2. In ZODB 3, you have to get a connection to a database before you access an object. The 'app()' function combines opening a connection and getting the top level object. When your done using the top-level object (or any objects accessible from it) you can close the database connection by calling the 'close' method on the '_p_jar' attribute, which is the database connection. You don't need to close the connection if you are going to exit Python without doing any more work. Don't forget to:: import transaction transaction.commit() If you want any changes to made to be saved. Note: Converting ZODB 2 (aka BoboPOS) data files to ZODB 3. The bbb.py script in utilities can be used to convert data files from ZODB 2 to ZODB 3 format:: utilities/bbb.py -f output_file input_file Here's a example:: utilities/bbb.py -f var/Data.fs var/Data.bbb You can also convert export files from ZODB 2 by inclding the -x option:: utilities/bbb.py -x -f output.zexp input.bbe ZODB 3 and Zope Database Adapters Most database adapters are *currently* likely to be problematic unless the underlying extensions and libraries: - allow multiple simultaneous database connections, - are thread safe, - give up the Python global interpreter lock when making database calls. This is only a problem when running Zope 2 with multiple threads. ZODB 3 Futures These are features that are lkely to wait for releases after 2.0. - Multiple database support - OPTIMIZATION: FileStorage will get a more efficient data structure for maintaining index information and key methods in the ZODB framework will move to C. Zope-2.8.7-final/doc/RESTRUCTUREDSTEXT.txt0000664000076400007640000000250710435500316016632 0ustar ajungajungIntegration of reStructuredText (reST) in Zope: Zope 2.7 or higher integrates reST as part of the Python docutils package. The syntax of reST is defined under http://docutils.sf.net/spec/rst/introduction.html Usage inside DTML: Usage inside ZPT: Usage inside PythonScripts: from Products.PythonScripts.standard import restructured_text rendered_html = restructured_test(rest_txt) return rendered_html Usage inside Zope products: from reStructuredText import HTML rendered_html = HTML(rest_txt) ... Character set issues: reST processes the reST document internally using unicode. A reST document is converted using Pythons default encoding to unicode and converted back from unicode to the default encoding on the output side. This means you must ensure that Python default encoding is properly. You can customize the default encoding by creating a file sitecustomize.py somewhere in yout PYTHONPATH: import sys sys.setdefaultencoding("iso-8859-1") This version of Zope also includes the ZReST product written by Richard Jones. ZRest is a standalone Zope product to handle reStructuredText documents. Zope-2.8.7-final/doc/DEBUGGING.txt0000664000076400007640000000544510435500316015346 0ustar ajungajung Running Zope in Debug Mode If you wish to run Zope in debug mode, set the 'debug-mode' configuration file parameter to 'on' (this is the default). This will have the following effects:: o On UNIX, Zope will not detach from the controlling terminal. o The Z_DEBUG_MODE environment variable gets set, which causes behavioral changes to Zope appropriate for software development. See the configuration file description of 'debug-mode' for more information. Using 'zopectl debug' In Zopes 2.7 and better, a utility known as 'zopectl' is installed into generated instance homes. You can use it to inspect a Zope instance's running state via an interactive Python interpreter by passing zopectl the 'debug' parameter on the command line. The 'top-level' Zope object (the root folder) will be bound to the name 'app' within the interpreter. You can then use normal Python method calls against app and use the Python interpreter normally to inspect results:: [chrism@james Trunk]$ bin/zopectl debug Starting debugger (the name "app" is bound to the top-level Zope object) >>> app.objectIds() ['acl_users', 'Control_Panel', 'temp_folder', 'browser_id_manager', 'session _data_manager', 'error_log', 'index_html', 'standard_error_message', 'standa rd_html_footer', 'standard_html_header', 'standard_template.pt'] >>> Using the Medusa Monitor NOTE: as of Zope 2.7, the Medusa monitor client is known to have operational problems due to lack of maintenance. It may work, it may not. If it doesn't work as advertised, please try 'zopectl debug' instead. If you're running Zope with a medusa monitor port, you can connect and interact via a python prompt with a running Zope session, while others interact with it over the web. Some constraints: o You must connect to a server running on the local host. o You must authenticate as superuser. o At the time of writing the superuser password (in the access file) must be plain text, not encrypted. To use the monitor, start it from the shell prompt like so (as if from the Zope root): python lib/python/ZServer/medusa/monitor_client.py localhost 8099 You'll get back a python prompt. To connect with the running Zope session: >>> import Zope >>> app = Zope.app() >>> dir(app.Control_Panel.Products) ['ExternalMethod', 'MIMETools', 'MailHost', 'OFSP', 'ZCatalog', 'ZGadflyDA', 'ZSQLMethods', '_objects'] >>> To see ODB changes caused by access concurrent with your debug session (eg, those web connections that can still be happening, or other debug sessions!), you may have to do a bit of magic: >>> app._p_jar.sync() To commit changes you make, before exiting, you may have to do another bit of magic: >>> import transaction >>> transaction.commit() Zope-2.8.7-final/doc/WINDOWS.txt0000664000076400007640000000477310435500316015210 0ustar ajungajungHow to build and install Zope from source code on Windows. ---------------------------------------------------------- These instructions appear to work for 2.7 and the trunk: * Ensure you have the correct MSVC version installed for the version of Python you will be using. * Install (or build from sources) Python http://www.python.org * Install (or build from sources) the Python for Windows extensions http://sourceforge.net/projects/pywin32/ * Unpack the Zope source distribution or pull from CVS. Change to that directory. * Execute: % python.exe inst\configure.py It should say something like: > > - Zope top-level binary directory will be c:\Zope-2.7. > - Makefile written. > > Next, run the Visual C++ batch file "VCVARS32.bat" and then "nmake". (run 'configure.py --help' to see how to change things) * 'makefile' will have ben created. As instructed, execute 'nmake'. If the build succeeds, the last message printed should be: > Zope built. Next, do 'nmake install'. * As instructed, execute 'nmake install'. A few warnings will be generated, but they can be ignored. The last message in the build process should be: > Zope binaries installed successfully. * If you are running from CVS, the build may fail: See http://collector.zope.org/Zope/1530 > running install_data > error: can't copy 'version.txt': no matching files > NMAKE : fatal error U1077: '"e:\src\python-2.3-cvs\pcbuild\python.exe"' : return code '0x1' > Stop. If you see this error, edit setup.py and comment the line referencing 'version.txt' * Zope itself has now been installed. We need to create an instance. Run: % python.exe {install_path}\bin\mkzopeinstance.py We will be prompted, via the console, for the instance directory and username/password for the admin user. * We are now ready to start zope. Run: % {zope_instance}\run_zope.bat. Zope should start with nice log messages being printed to stdout. When Zope is ready, you should see: > ------ > 2004-10-13T12:27:58 INFO(0) Zope Ready to handle requests Press Ctrl+C to stop this instance of the server. * Optionally, install as a Windows service. Execute: % python {zope_instance}\zope_service.py to see the valid options. You may want something like: % python {zope_instance}\zope_service.py --startup=auto install Once installed, it can be started any number of ways: - python {zope_instance}\zope_service.py start - Control Panel - net start service_short_name (eg, "net start Zope_-1227678699" Zope-2.8.7-final/doc/TODO.txt0000664000076400007640000002060710435500316014615 0ustar ajungajungThe following is a TODO list dealing with installation and startup changes proposed for the Zope trunk. It is organized into these categories: "BEFORE 2.7 FINAL" (things that need to be done before we're finished, but not currently on the critical path) "MAYBE NEVER BUT NICE TO HAVE" (items that are not on any critical path) "COMMUNITY CONCERNS" (uncategorized concerns dealing with features missing from HEAD and prior 2.7 releases) ---------------------------------- BEFORE RELEASE OF ZOPE 2.7 FINAL ---------------------------------- Provide more intuitive command line option overrides Currently, you can override config file options by using the -X command line switch to runzope.py, followed by key/value pairs separated by an equal sign. This isn't explained anywhere and I'm not sure what limitations it has (can you change any value in the config file this way?). Zope-specific command-line options that don't require the -X (e.g. --event-log-file=something) should be provided to the user via changes to Zope/Startup/options/ZopeOptions. Make 'runzope -h' work and give better error messages when a configuration fails to make the grade. Currently runzope -h just emits a traceback, and the message printed by run.py says to run 'run.py -h' which can't work because things that it needs to import aren't on the PYTHONPATH. Config file needs better inline docs The Zope ZConfig config file has some inline docs. They need to be completed. Additional docs may come as a result of writing a schema-to-HTML translator. Make as many things defaultable as feasible Maybe we can allow a config file in which everything is commented out. We'll see. Write some more unit tests Write unit tests for Zope.Startup packages. What to do about envvars? Envvars are still used "under the hood" in ZConfig handlers as the result of particular configuration declarations in order to make Zope do the right thing (e.g. INSTANCE_HOME, SOFTWARE_HOME, DTML_REQUEST_AUTOQUOTE, SESSION_TIMEOUT_MINUTES and other envvars are set in ZConfig handlers for their respective keys). But envvars should not be used to try to configure Zope, as the handlers overwrite existing envvars with prejudice. We need to come down on one side or the other about envvars.. either they should be respected at startup as they always have been or they should be explicitly not respected. Currently they are not respected. We need to communicate this decision to developers and update doc/ENVIRONMENT.txt as necessary. win32-eventlog needs testing The "win32-eventlog" log handler (which is creatable via the config file) needs to be tested. Note: As of Jul 6, 2003, it has been confirmed to not work. The service name that it is logging under ("Zope") is not registered with the NT event service properly, causing a warning to be written to the log every time a log call is made. Server construction errors need to be better When a server is constructed, it attempts to bind to a port. If the port cannot be bound, an error is raised. The error currently doesn't include the port number, and should. I propose that we add two more options to the config file: Create import-directory and extensions-directory directives These would both be multikeys which specify some number of directories that contained importable zexp files and external methods, respectively. This would allow us to not require any fixed instance home directory. Instead, each path required by each subsystem is specifiable by itself in the config file. I'm sure that utilizing these options in the config file will break things that rely on having a monolithic INSTANCE_HOME such as products that attempt to do something like "import_dir = os.path.join(INSTANCE_HOME, 'import'). So I propose that the stock Zope instance home install continue to follow the old pattern (where everything is installed into a single instance home directory), but we provide the advanced config file options for roll-your-own packagers and advanced users. I would like to do the same thing for the software home, but I haven't thought much about it yet. Review the Zope Book 2.6 Edition chapters and come up with revisions or at least create a Zope 2.7 Install HowTo The 2.6 edition Zope Book at http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition has three chapters which detail installation (Installing and Starting Zope), maintenance (Maintaining Zope) and ZEO (Scalability and ZEO). These chapters should be reviewed for inaccuracies with respect to the forthcoming trunk changes and changes should be made "offline" to allow a Zope Book 2.7 edition. At least create a HowTo which summarizes the differences between installing 2.6 and installing 2.7. ------------------------------ MAYBE NEVER BUT NICE TO HAVE ------------------------------ ZConfig defaults We deferred several issues that we recognized as areas for improvement in ZConfig that might make it possible to avoid writing nasty procedural code for default handling unnecessary (e.g. Zope.Startup.handlers.root_handler). See http://my.zope.com/CPM/CPM/issues/4 and http://my.zope.com/CPM/CPM/issues/3. Not necessary for merge, but useful to think about for future. ZConfig should keep enough state to be able to reconstitute the textual representation of the configuration. It would be nice if ZConfig kept enough state to be able to reconstitute the configuration in textual representation (to aid GUI builders and to make it possible to have a meaningful 'zopectl showconfig' or somesuch). ---------------------------------- COMMUNITY CONCERNS (uncategorized) ---------------------------------- Status: Request for comment sent to the community: http://mail.zope.org/pipermail/zope-dev/2003-March/018999.html Lots of discussion! - ZConfig is too complex when compared with envvar parsing. - Somewhat unrelated, but the features of "debug mode" should be disentangled from one another and specifiable individually. - Need to support arbitrary storage types. - Provide --swhome and ---with-python switch to mkzopeinstance, which will allow folks to create an instance that points to particular (known) Python versions and software homes. XXX This doesn't make sense. mkzopeinstance is part of the software home; if you want to use a different one, use the mkzopeinstance from the software home you want to use. The same goes for the Python to use; that's "built-in" to a software home. Using a different Python doesn't make sense given that the software home includes compiled modules. AAA This is to service to-be-chrooted installs. - Explain how to set up and use a ZEO server using mkzeoinst included in 2.7's ZEO. (Use the installed bin/mkzeoinstance script; use --help for more information.) Richard Jones has done work to allow you to create a ZEO instance in a way similar to creating a Zope instance (yay!) after creating a software home. - Give ZConfig replacement access to the environment or shell somehow. For instance, some folks use the same 'start' script in all of their instances right now (under 2.6). The script does things based on the value of an envvar that can be used to distinguish the config values of the instance from other instances. We could allow for the same sort of behavior E.g.: %define HOSTNAME `hostname` (assuming `hostname` resolves to a hostname) lockfile-name /var/lock/$HOSTNAME-lockfile - Give installaler an option to put libs in a user-specifiable directory at software home installation time. - Give installer an option to put docs in a user-specifiable directory at software home installation time. - Make it possible to install Zope-related Python libraries to The site-packages of the Python used to invoke setup.py. - Offer to install software home 'bin' scripts into a directory separate from the software home 'bin' directory. - Allow for the installation of platform-dependent files (basically Python extensions) to be installed to a place separate than that of platform independent files (as requested by Luca DeVitis). - Upon failure of Windows service startup, it's possible for the reason for the failure to not be logged anywhere. This is because we carefully wait til late in the startup process to write logfiles so UNIX has a chance to setuid. This is unnecessary for Windows. Zope-2.8.7-final/doc/HELPSYS.txt0000664000076400007640000001116310435500316015174 0ustar ajungajungZope Help System The Zope Help System provides context-sensitive on-line help for Zope users. The system is flexible and can provide help for Python and ZClass-based Zope Products. In the future the Help System will be expanded to provide additional help including API documentation. Using the Help System Every standard Zope management screen should include a help button which provides access to help for that screen. Additionally all the installed help topics can be browsed and searched. Architecture The Help System is based around Zope Products. When a product is installed, its help objects are installed along with it. All help content is associated with a product. Help content is provided by 'Help Topic' objects. These objects live inside Product folders within a special container object called a 'Product Help' object. When you browse a Product folder in the Control Panel you will see these 'Product Help' objects and their 'Help Topics'. In general you get access to the Help System through the help system object which has methods for drawing help buttons. This object lives in the Zope application object and has an id of 'HelpSys'. Writing Help for ZClasses Suppose you've created an addable type of object with ZClasses. You'd like the management screens of your objects to have help buttons just like the standard Zope management screens. First create some Help Topics though the web which document your management screens. Do this by going to your ZClass's Product and creating new Help Topics inside the Product Help object. Next go to your ZClass and click on the 'Views' management tab. On this screen you define your object's management views. Each view has a name, a method, and optionally a help topic. If you select a help topic for a view, a help button will be drawn on that management view and it will be linked to the help topic you select. Writing Help for Python Products To support help your Python product needs to register help topics during product registration, and it needs to indicate which help topics should be associated with which management screens. Registering Help Topics To register help topics use the 'registerHelp' method on the ProductContext object. For example:: def initialize(context): ... context.registerHelp() This method will create help topics for all files found in the 'help' subdirectory of the product. Supported file types include: .html, .htm, .txt, .stx, .dtml, .gif, .jpg, .png. Appropriate classes of help topics are used depending on the suffix of the help files. If you want more control over how your help topics are created you can use the 'registerHelpTopic' method that takes an id and a help topic object as arguments. For example:: from mySpecialHelpTopics import MyTopic def initialize(context): ... context.registerHelpTopic('myTopic', MyTopic()) Associating Help Topics with Management Screens The chief way to bind a help topic to a management screen is to include information about the help topic in the class's 'manage_options' structure. For example:: manage_options=( {'label':'Edit', 'action':'editMethod', 'help':('productId','topicId')}, ) In this example, 'productId' refers to the name of the Zope Product in which the class is defined, and 'topicId' refers to the id of the Help Topic associated with this management view. When Zope draws the management view it will automatically include a help button pointing to the right help topic if you provide this information in the 'manage_options' structure. Note: sometimes Zope gets confused and defaults to highlighting the first management tab in place of the correct one. To fix this, set the 'management_view' variable to the name of the correct view. If the wrong view is hilighted, then the wrong help button will be drawn. To draw a help button on a management screen that is not a view, use the 'HelpButton' method of the 'HelpSys' object like so:: This will draw a help button linked to the specified help topic. If you prefer to draw your own help button you can use the helpURL method instead like so:: This will give you a URL to the help topic. You can choose to draw whatever sort of button or link you wish. Zope-2.8.7-final/doc/INSTALL.txt0000664000076400007640000001461010435500316015153 0ustar ajungajungBuilding and installing Zope from source ---------------------------------------- Welcome to Zope! This document describes building and installing Zope on UNIX and Linux. See WINDOWS.txt for information about Windows. See the PLATFORMS directory for notes about various other platforms. System requirements when building from source bash or another Bourne shell variant Python 2.3.5 or later installed somewhere in the system PATH (Note: Python 2.4 is officially *not* supported because there has not been a security audit for Python 2.4 and Zope 2 so far. If you use Python 2.4 with Zope then you use it at your own risk from the security perspective). GNU make A C compiler (gcc recommended) Recommendations - You are recommended to build and install Zope as a non-root user. Building Zope To build Zope, run the conventional UNIX build sequence from within the Zope source tree:: ./configure --prefix=/where/to/install/zope make If you do not specify a '--prefix' option, during a later step, Zope will be installed into a default location. If the configure script cannot find a suitable Python interpreter for use with Zope, it will complain with an informative error message. If this happens, you may use the '--with-python' command-line option to 'configure' to specify which Python interpreter to use. Run './configure --help' to see other command-line options available via the configure script. Optional: Building and Installing Zope 'In-Place' Older versions of Zope were typically run directly from the 'source' directory itself. This is useful for development purposes. You can regain that behavior by performing the following sequence of commands within a Zope source directory: ./configure make instance This command also creates an administrative user with the specified username and password. WARNING: "make instance" doesn't work on FreeBSD 5.0 and presumably other platforms. You should either use GNU make (gmake instance), or use "make install" instead. Making an "In-Place" instance builds the binary files and creates the files necessary for a Zope instance to be run directly within the build directory (e.g. 'in-place'). You may start Zope by running:: ./bin/runzope Optional: Building Zope Using The "Clean-Source-Tree" Method You can run the 'configure' command from outside of the Zope source tree. If you do so, the makefile will be written to your current directory. The files generated by the build process (via 'make') will be written to the directory from which you run 'configure'. You can then use 'make install' to install these files to their canonical locations. This feature is to support source locations which are not writable. Installing Zope To install Zope to the place you've specified via the '--prefix' option (or to the default location if you didn't specify a prefix), type:: make install Creating a Zope Instance Home Once you've performed the install step, to begin actually using Zope, you will need to create an "instance home", which is a directory that contains configuration and data for a Zope server process. The instance home is created using the 'mkzopeinstance.py' script:: /where/to/install/zope/bin/mkzopeinstance.py You will be asked to provide a user name and password for an administrator's account during 'mkzopeinstance'. Command-line options to 'mkzopeinstance' are available, and can be investigated by running 'mkzopeinstance.py --help'. Starting Zope Once an instance home has been created, the Zope server can now be started using this command: /location/of/zope/instance/bin/runzope If you get errors indicating that addresses are in use, then you will have to supply arguments to runzope to change the ports used for HTTP or FTP. The default HTTP and FTP ports used by Zope are 8080 and 8021 respectively. You can change the ports used by specifying the "port-base" parameter to runzope. For example, to run Zope on ports 9080 and 9021:: ./bin/runzope -X port-base=1000 There is also an interactive Zope command shell named 'zopectl' that may be invoked by running 'bin/zopectl'. By default, 'zopectl start' will start a background process that manages Zope and 'zopectl stop' will stop the background process. Logging In To Zope Once you've started Zope, you can then connect to the Zope webserver by directing your browser to:: http://yourhost:8080/manage .. where 'yourhost' is the DNS name or IP address of the machine running Zope. If you changed the HTTP port as described, use a port number of 8080 + the port-base value. You will be prompted for a user name and password. Use the user name and password you provided in response to the prompts issued during the "make instance" process. Now you're off and running! You should be looking at the Zope management screen which is divided into two frames. On the left you can navigate between Zope objects and on the right you can edit them by selecting different management functions with the tabs at the top of the frame. If you haven't used Zope before, you should head to the Zope web site and read some documentation. The Zope Book is a good place to start. You can access the Zope Book at: http://www.zope.org/Documentation/Books/ZopeBook Integrating Zope With An Existing Webserver Zope doesn't require any existing webserver to run, but you can integrate it with other webservers as necessary. See the WEBSERVERS.txt file for more information about configuring Zope with an existing web server. There is also information about integrating Zope with existing webservers on the Zope.org website. Troubleshooting - This version of Zope requires Python 2.3.4 or better. - The Python you run Zope with *must* have threads compiled in, which is the case for a vanilla build. Warning: Zope will not run with a Python version that uses libpth. You *must* use libpthread. - To build Python extensions you need to have Python configuration information available. If your Python comes from an RPM you may need the python-devel (or python-dev) package installed too. If you built Python from source all the configuration information should already be available. - See CHANGES.txt for important notes on this version of Zope. Zope-2.8.7-final/doc/UNITTEST.txt0000664000076400007640000004073610435500316015334 0ustar ajungajungZope Unit Testing Zope Testing If you encounter a directory named "tests" in a package within within the Zope source code, it most likely indicates that the directory contains test code used to ensure that the code owned by the package works as it was designed. Many of the test scripts contained within Zope "tests" directories will be scripts which contain "unit tests". What Unit Tests Are A "unit" may be defined as a piece of code with a single intended purpose. A "unit test" is defined as a piece of code which exists to codify the intended behavior of a unit and to compare its intended behavior against its actual behavior. Unit tests are a way for developers and quality assurance engineers to quickly ascertain whether independent units of code are working as expected. Unit tests are generally written at the same time as the code they are intended to test. A unit testing framework allows a collection of unit tests to be run without human intervention, producing a minimum of output if all the tests in the collection are successful. What Unit Tests Are Not It's very useful to define unit tests in terms of what they are not. From the "Extreme Programming Enthusiast" website (http://c2.com/cgi/wiki?UnitTestsDefined):: Unit tests are not: - Manually operated. - Automated screen-driver tests that simulate user input (these are "functional tests"). - Interactive. They run "no questions asked." - Coupled. They run without dependencies except those native to the thing being tested. - Complicated. Unit test code is typically straightforward procedural code that simulates an event. Unit Testing Frameworks A unit testing framework is generally employed to collect related unit tests together in order to make it easier to run them as a group. When used with a unit testing framework, unit tests live outside of the modules of code they're intended to test. How Unit Tests Help In The Development Process Unit tests should be written at the same time the code they test is written. A short, healthy cycle of "code/write test/run test/repeat" can help a developer code more quickly by reducing "backtracking" effort and by helping the developer focus on the actual problem at hand. Additonally, the unit tests generated at initial development time can serve as later assurance that maintenance and refactoring performed on code does not break any of its intended functionality or behavior. The results of unit tests may additionally be used as a metric by quality assurance personnel along with the results of other tests to gauge code quality before before it is "shipped." Basic Unit Testing Philosophies Write unit tests at the same time that you write the code. Nothing's worse than being faced with the prospect of writing tests against a huge chunk of existing code because you "have to." Writing unit tests post-facto can be boring and also robs you of the main benefits that unit testing can provide. Writing unit tests at the same time you write the code helps you focus on the task at hand. Writing unit tests in conjuction with code can be fun and satisfying, and can help you improve the quality of your code while its goals are fresh in your mind. Used properly, unit testing may also help you write code faster, because you will need to "backtrack" less. Some "Extreme Programming" enthusiasts posit that you should write a test before you write its associated code, and then develop the code until it passes the test. Unit tests should be developed against as small and specific a subset of a system's or subsystem's functionality as possible. For instance, a one unit test may test that a unique id generator produces ids of a specific length, while another unit test in the same group may ensure that a thousand ids from the same unique id generator do not contain the same value. Writing a single unit test which tests a broad swath of a system's capabilities is counterproductive, because it does not allow for a fine enough granularity when attempting to figure out "what went wrong," requiring the developer to "backtrack". Unit testing is capable of helping to help reduce backtracking, but only if used properly. A unit test does not produce any output unless it fails. If a unit test fails, it should print something useful, but short. A unit test should never fill the screen with output or otherwise produce output that needs to be manually examined for "clues". This is the realm of other testing methodologies. If unit tests are written at sufficiently granular level, it is often enough just to know the name of the unit test that failed. "It is better to have tried to test and failed than to not have tried to test at all" aka "test the riskist things first." If the prospect of writing a series of unit tests for an existing system seems daunting, it's important to remember that no matter how many unit tests you write, you cannot prove that your software does not have bugs. Therefore, you cannot possibly test every case while developing. You should plan to write tests against code based on the risks involved in not testing that code. Don't feel that you need to write a test case for every "corner case" (although do try to hit the riskiest ones). In the worst case, it's better to have a test module with one lonely unit test in it than none at all. "Test fixtures" are employed by unit tests. Test fixtures are bits of state and environment that allow the unit test to perform its job properly. An example of a test fixture might be a file, an instance of a class, or a row in a database table. Any part of the environment needed by a unit test besides the unit testing framework itself is considered a test fixture. In general, the common fixtures required by individual tests in a testing group should be more or less identical. If the fixtures needed by a single test are radically different than the fixtures required by the rest of the tests, or if the test does not require the fixtures provided to the other tests, it probably belongs in another (or its own) group of tests. When a unit test in a group modifies the state of a test fixture, the test fixture should be restored to a known state before the next unit test in the group is run. Often, this means "rolling back" changes to a transactional database or restoring the state of a string so the next unit test can inherit a known state of the same fixtures. The unit testing framework has capabilities which allow you to automate most of this work by providing a "set up" method which gets called before each test is run and a "tear down" method that gets called after a test is finished. Unit tests should play nicely with the unit testing framework they employ. Unit tests should not call "sys.exit()" or do similar things which effect their ability to be run as part of a group of tests. The testing framework attempts to deal with misbehaved unit tests, but it's better just to make them behave nicely in the first place. Unit tests should "grow" with the code that they're intended to test. For example, if a group of unit tests is intended to verify the behavior of all of the routines within a module, additional unit tests should be added to the test group when new functionality is added to that module. Writing Unit Tests For The Zope Core If you're writing core code, you probably don't need to listen to any more of this. :-) The rules for writing tests for Zope core code are simple: - The testing code should make use of PyUnit (/lib/python/unittest.py). Instructions for using PyUnit are available at http://pyunit.sourceforge.net. - Tests must be placed in a "tests" subdirectory of the package or directory in which the core code you're testing lives. - Test modules should be named something which represents the functionality they test, and should begin with the prefix "test." E.g., a test module for BTree should be named testBTree.py. - An individual test module should take no longer than 60 seconds to complete. Writing Unit Tests For Applications Based On Zope Zope uses the PyUnit unit testing framework, the documentation for which is available at http://pyunit.sourceforge.net. The lib/python/unittest.py module is the framework. You may establish your own conventions for naming and placement of test modules, but using the same rules as for Zope Core is recommended, and the standard used by most. Writing unit tests against applications based on Zope can be difficult. Zope is a collection of related modules, some with non-trivial interdependencies. Running its code successfully also in some cases depends on state provided only in the context of a web request, so calling Zope methods directly may not work as you expect. If you're not intimately familiar with Zope, implementing unit tests can be fustrating. For example, for the common case, before you are able to write code which tests a Zope SQL Method, you must establish a test fixture which represents your entire Zope site. This is made easier by the ZopeTestCase package, included with Zope from Zope 2.8. It is located in the directory lib/python/Testing/ZopeTestCase and the documentation in the doc subdirectory. In principle, subclassing from Testing.ZopeTestCase.ZopeTestCase instead of subclassing from unittest.TestCase will mean that you have a zope test fixture set up for you. There are several examples of how to use ZopeTestCase in the ZopeTestCase directory. If you don't want to do that, you can still set up the fixture yourself. - add the 'lib/python' directory of your Zope installation to the PYTHONPATH (via sys.path.insert()) - 'import ZODB' (may not be required, but just put it in for good measure) - 'import Zope' - 'app = Zope.app()' - operate on the Zope instance space by calling methods from the root object (bound to 'app'), e.g.: app.acl_users.manage_addUser() - a transaction will not be committed to the Zope object database until you call "transaction.commit()", so changes made to the ZODB are not visible to succeeding tests unless that function is called explicitly. - As a part of your tearDown, make sure to call "app._p_jar.close()". This closes the database connection cleanly. For more information on operating on Zope programatically by "importing" it, see Michel Pelletier's "The Debugger Is Your Friend" at http://www.zope.org/Members/michel/HowTos/TheDebuggerIsYourFriend Running the unit tests The basic command to run unit tests is:: bin/python bin/test.py --config-file etc/zope.conf or on windows:: C:\Path\To\Python.exe bin\test.py --config-file etc\zope.conf This will run all unit tests in lib/python and below (that is, it will run the zope core unit tests. To run the unit tests located in the Products directory you need to add two parameters --libdir Products so that test.py will look for modules in the Products directory, and --dir Products to look for tests in the Products directory and below. You can of course specify --dir even closer, so a typical command to run the tests for the "Myproduct" product would be bin/python bin/test.py -v --config-file etc/zope.conf --libdir Products \ --dir Products/Myproduct This is rather long, and on unix you can shorten this to:: bin/zopectl test --dir Products/Myproduct These commands are equivalent. bin/zopectl will just run test.py with some useful defaults for you:: -v increases verbosity. You can add -vv for even more output. --config-file etc/zope.conf reads in the Zope configuration file,so that important paths are set up. --libdir Products tells test.py to include Products as a module path and to include the tests there. To see the rest of the command options you can run it with --help:: bin/zopectl test --help The test output should look something like this:: Running unit tests at level 1 Running unit tests from /home/zope/Products/CMFCore/tests Parsing /home/zope/etc/zope.conf ..................... ---------------------------------------------------------------------- Ran 21 tests in 0.130s OK Speeding up the tests Not all unit tests will need a zope test fixture. If you have many tests that do not need it, it can be a good idea to separate them into different test files, so that you can run the tests that do not need a fixture separately, since setting up the fixture takes time. This will not save you time when running all your tests, but it can save time while developing, since you can skip loading Zope when running some tests (also see Functional testing, below). Also, one of the most effective tools for facilitating unit testing is ZEO (http://www.zope.org/Products/ZEO). ZEO is an open-source clustering solution for Zope which makes it possible to front-end a single "storage server" which manages a Zope object database with multiple Zope clients that run a "client storage". The reason ZEO is interesting for unit testing is mostly an unintended side-effect of how it works as compared to Zope without ZEO. Zope without ZEO commonly uses a "FileStorage" to hold its object database. When Zope is started with a FileStorage, the FileStorage code processes an "index" file. This takes time. Zope using a ClientStorage as with ZEO does not process an index file, making startup faster. Fast startup of Zope is critical to effective unit testing. It is recommended that you implement ZEO if you're heavy in to unit testing, as it really speeds things up. It's not strictly required, however. Emulating requests Sometimes, just importing Zope isn't enough. For example, it's often not possible to obtain the results of a DTML or Python method by simply calling it from your running code without doing lots of trickery by establishing fake REQUEST object to pass in to it, as Zope expects a REQUEST object to exist in the context of a web request (which a DTML method is somewhat logically designed to serve). There are two ways of doing this. The old and complicated way is using Zope.debug() (documented below), but the new way is known as "functional tests". Support for this is included in ZopeTestCase, see lib/python/Testing/ZopeTestCase/doc/FunctionalTesting.stx for more information. It does almost the same thing as the "debug" method, but it returns a RESPONSE object, so you can easily check that the output is correct, and not only that the request finished without raising any exceptions. Zope.debug() Zope.debug() allows you to simulate a web request, which generally provides all the state necessary to run methods which depend on web requests, and returns the results of the web request as it would be seen in by a web browser. To use the Zope debug method, do the following: - add the lib/python path to your PYTHONPATH (via sys.path.insert()) - 'import ZODB' - 'import Zope' - 'Zope.debug('/a/url/representing/a/method?with=a?couple=arguments', u='username:password', s='silent', e={'some':'environment', 'variable':'settings'}) The "silent" option causes Zope not to print anything. You can set your python's stdout to a file or a file-like object to capture the output if you do not set the silent flag. Administrivia Unit test scripts found in the Zope source code make use of Pythons PyUnit unit testing framework, written by Stephen Purcell (thanks Stephen!). PyUnit is based on the JUnit testing framework for Java (written by Kent Beck and Erich Gamma), which in turn was based on a testing framework designed for Smalltalk (also written by Kent Beck). Unit testing is a primary tenet of "Extreme Programming", a software development methodology designed to faciliate the rapid production of high quality code with a minimum of developmental ceremony. For more information on unit tests as they relate to Extreme Programming, see http://c2.com/cgi/wiki?UnitTestsDefined. Although Zope Corporation has not embraced the entire spectrum of Extreme Programming methodologies in its software development process, we've found unit tests a way to speed development and produce higher-quality code. ZopeTestCase was written by Stefan H. Holek, and is included with Zope from Zope 2.8. (Thanks Stefan!) Zope-2.8.7-final/doc/WEBSERVER.txt0000664000076400007640000004327310435500316015420 0ustar ajungajungUsing Zope with an existing web server -------------------------------------- While Zope comes with a web server, you may wish to use it with an existing web server. Use Persistent CGI (PCGI) or FastCGI to allow your existing web server to work with Zope on Unix and Windows. Roughly, PCGI is a protocol for translating CGI requests from a web server into Zope requests. CGI requests are traditionally one shot events; the web server handles a request for a CGI script by spawning a new process to handle the requests, returning the results of the request when the process dies. Zope is a long running process, which means that it does not start up and die on each request like traditional CGI programs. If it did, each request would take far too long due to the time to start and stop an application server like Zope. Thus, PCGI is one of the options for gatewaying from the separate CGI processes to the Zope long running process. FastCGI is very similar to PCGI. Where PCGI has a "wafer-thin" external CGI program to connect Apache and Zope, FastCGI has an Apache module. This saves the so called "fork tax" which is incurred when Apache forks to execute the PCGI CGI program. Using Zope in multi-threaded mode with ZServer ---------------------------------------------- ZServer is a general purpose TCP server for publishing Zope objects over various transport protocols. For Zope to run multi-threaded, you must run ZServer. ZServer is based on Sam Rushing's Medusa software. The benefit of using Medusa as the ZServer core is that it is not protocol specific (Medusa provides libraries to program for any protocol) and it is easily extensible. Because Medusa is written in Python, is extremely high performance by design, and comes with an HTTP and FTP server, we chose it for the Zope core. It is not necessary, however, for ZServer to actually listen for incoming HTTP requests. If you want Apache to do the actual listening and serving, then you can use ZServer's PCGI or FastCGI components to communicate with Apache. PCGI PCGI is deprecated and should not be used anymore. To install PCGI, please get the PCGI support package from the Zope 2.6 tarball (it no longer ships with Zope 2.7), and read the pcgi/README file from that package. To run ZServer with PCGI, you must configure Zope with a PCGI server. You can do this by adding a section to the etc/zope.conf file like this:: address /path/to/PCGI/resource/file Note, you must have gone through the directions in 'INSTALL.txt' for this to work. This command will cause Zope to start with a PCGI server as well as any other servers you specify in the zope.conf file. For PCGI to work, the webserver and Zope must agree on a PCGI resource file. Now the Zope long running process is started up, and the PCGI component is loaded and ready to receive CGI requests from your webserver. The installation process should create a 'Zope.cgi' PCGI file. Copy the 'Zope.cgi' file to your web server's 'cgi-bin' directory. On Unix you can also create a symbolic link to 'Zope.cgi' from your cgi-bin directory. For example:: ln -s /home/amos/Zope/Zope.cgi /usr/local/apache/cgi-bin/Zope At this point you should perform any other steps you web server requires to install and configure a CGI script. Note: For more information on PCGI check out Jeff Bauer's "PCGI pages", http://starship.python.net/crew/jbauer/persistcgi/. When your Zope.cgi file is correctly configured as a CGI script with your web server, you are ready to access Zope through the web. You should point your browser at: 'http://youmachine.example.com:8998/cgi-bin/Zope.cgi/manage' (Your URL maybe be different depending on how your web server is configured.) You should be prompted to enter a username and password. Enter the Zope "super manager" name and password. Note: Apache requires some tricky configuration to get it to pass the HTTP authentication header information to Zope. See the section 'Zope authentication with existing web servers'. There is an Apache module for Zope/PCGI: http://www.zope.org/Members/phd/mod_pcgi2/ One can easily compile and configure it. It passes the HTTP authentication header and other environment variables. It supports virtual hosting. FastCGI FastCGI is relatively easy to set up if you have installed an Apache module before. If you haven't, don't worry, it isn't difficult. It comes with straightforward instructions. You will need to download the mod_fastcgi module from FastCGI.com: http://www.fastcgi.com/ A version more recent than 2.2.2 is needed. Extract the archive and follow the directions in its 'INSTALL' file. Using the 'DSO' method is usually easiest. Next, Zope must be set up to use FastCGI. You must decide whether you want to use FastCGI via a Unix domain socket or a TCP port. You will have to use a TCP port if you are using a Win32 box, or any other operating system that does not provide Unix domain sockets (sockets in the filesystem). It is also the method you must use if Apache and Zope reside on different computers. You will need to select a port number. This is arbitrary, but it must not already be in use, and if you are not the super-user, it must be a sufficiently high number. For illustrative purposes, I will use 8889. Unix domain sockets do not open a TCP port, which may be significant in some tight-security situation, but it requires slightly more setup, and so it is slightly less fool-proof. You will need to select a location and filename for the socket. Likely options are '/tmp/zope.soc' or within Zope's 'var' directory. Make sure that both Zope and Apache will have adequate file permissions to read and write to the socket. For illustrative purposes, I will use '/tmp/zope.soc'. This can be configured by adding a fast-cgi section in your etc/zope.conf file. The section should look like this for a TCP port:: address localhost:8889 To use a Unix domain socket instead, specify the path of the socket file instead of a [hostname:]port:: address /tmp/zope.soc You can start Zope at this point by running the 'runzope' script:: $ ./bin/runzope Next, Apache must be set up to pass FastCGI requests to Zope. This is done with a 'FastCgiExternalServer' directive in Apache's httpd.conf. By using 'FastCgiExternalServer' we are telling FastCGI that we'll be launching and killing the long-running process manually. It goes at the beginning of 'Section 2'. This directive will look like this for a TCP port configuration:: FastCgiExternalServer /PATH/TO/apache/htdocs/zope \ -host localhost:8889 \ -pass-header Authorization ...or like this for a socket configuration:: FastCgiIpcDir /tmp FastCgiExternalServer /PATH/TO/apache/htdocs/zope \ -socket zope.soc \ -pass-header Authorization The FastCgiIpcDir directive is needed to tell FastCGI where to find the socket file. By default, FastCGI will look in /tmp/fcgi. You *cannot* specify a full path with the '-socket' option, only a filename within the FastCgiIpcDir. The first argument after the directive name is confusing. Apache maps URLs to directories and files within your 'DocumentRoot'. You must specify the name of a file within your DocumentRoot directory, but that file need not (and probably should not) actually exist. When Apache tries to serve that file, it will instead send the request through FastCGI. Make sense? Say, for instance, you want zope to be accessible from this URL: http://myserver.org/myzope Suppose further that you have Apache installed at /usr/local/apache. The directive would have to look like this:: FastCgiExternalServer /usr/local/apache/htdocs/myzope \ ... ... The next argument depends on whether you are using a TCP port or a socket. Note that the -host (port) argument requires the hostname. In the example, it's the same host as Apache runs on, but this need not be the case. If you are using a socket, make sure Apache has the appropriate privileges to find it. The last argument, '-pass-header Authorization', tells FastCGI to pass the Authorization headers to Zope. Without this, you would not be able to use the Zope web management interface, or anything else that requires authorization. (Note: This feature was not in mod_fastcgi 2.2.2) One more item must be added to the httpd.conf. Below the 'FastCgiExternServer' directive, enter the following:: SetHandler fastcgi-script ...changing '/zope' as appropriate. Using the 'http://myserver.org/myzope' example from above, it would look like this:: SetHandler fastcgi-script This creates a special case for the location specified, telling Apache to let FastCGI handle the request. That is all that is needed. Make sure Zope is running, start or restart Apache, and you are finished. Using Zope in single-threaded mode with pcgi_publisher ------------------------------------------------------ The installation process should create a 'Zope.cgi' PCGI file. Copy the 'Zope.cgi' file to your web server's cgi-bin directory. On Unix you can also create a symbolic link to 'Zope.cgi' from your cgi-bin directory. For example:: ln -s /home/amos/Zope/Zope.cgi /usr/local/apache/cgi-bin/Zope At this point you should perform any other steps you web server requires to install and configure a CGI script. Note: For more information on PCGI check out Jeff Bauer's "PCGI pages", http://starship.python.net/crew/jbauer/persistcgi/. When your Zope.cgi file is correctly configured as a CGI script with your web server, you are ready to access Zope through the web. You should point your browser at: 'http://youmachine.example.com:8998/cgi-bin/Zope.cgi/manage' (Your URL maybe be different depending on how your web server is configured.) You should be prompted to enter a username and password. Enter the Zope "super manager" name and password. Zope authentication with existing web servers --------------------------------------------- Zope normally performs both authentication and authorization of users. Some web servers don't pass authentication information to CGI scripts. If you keep getting rejected when you try to access Zope through the web with the "super manager" user name and password, there is a good chance that your web server is not passing authentication information to Zope. Getting Apache to pass authentication headers Before attempting to use your own Apache with Zope, it is highly recommended that you look at Zap. Zap is a preconfigured and precompiled version of Apache for Linux that drops right into Zope with no hassles. Even if you want to use your own Apache, or if you use it on a different platform than Linux, it is very helpful to have Zap's 'httpd.conf' file to guide you through configuring Apache. Zap can be found on the Zope website at: http://www.zope.org/Download/Releases/Zap-1.1.0 If you are using Apache you will need to convince Apache to pass authentication headers to Zope. The easiest way to do this with Apache 1.3 and above is to use mod_rewrite. Here is an example of configuration information which you would place in an Apache conf file:: # Zope configuration maps /Zope/ to the Zope.cgi CGI script RewriteEngine on RewriteCond %{HTTP:Authorization} ^(.*) RewriteRule ^/Zope(.*) /usr/local/apache/cgi-bin/Zope.cgi/$1 [e=HTTP_CGI_AUTHORIZATION:%1,t=application/x-httpd-cgi,l] Note that the RewriteRule should be one long line, and that the last character is the letter l, not the number 1. For Apache servers version 1.3b4 and above, there is an alternate way to get the server to pass through authorization headers, but you must have the ability to recompile your Apache server binary. If you pass the flag -DSECURITY_HOLE_PASS_AUTHORIZATION when compiling the server, the resulting Apache binary will allow authorization headers to pass through to CGI programs and you can avoid using the Rewrite rules described above. Allowing your server to handle authentication itself Sometimes you may prefer to handle authentication outside Zope, for example if your web server already does complex authorization, or if it seems too difficult to convince it to pass authentication information to Zope. As of 1.9, Zope began supporting a mode that allowed the web server to handle authentication. The 'REMOTE_USER' environment variable is then matched to the identity of a user object in Zope. The following provide step-by-step instructions for setting this up in Apache, allowing both the Zope "super manager" defined in the Zope 'access' file and users defined in Zope User Folders to be authenticated via the web server. Get Apache to authenticate /cgi-bin/Zope Add a directive in your Apache configuration file such as:: AuthType Basic AuthName Zope-realm AuthUserFile /usr/local/etc/httpd/conf/ru_users require valid-user Then send Apache a '-1' signal to tell it to re-read its configuration files. *Note*: The above presumes that '/cgi-bin/Zope' has been made executable by some other Apache directive in the configuration file. Ensure Apache has 'superuser' Using Apache's tools for managing a user database, make sure that the 'AuthUserFile' defined above has a valid user called 'superuser'. Get Zope to use Apache's authentication Change Zope's access file to contain just the superuser id followed by a colon, as in:: superuser: Note that this can be any value, including spaces. The only restriction is that the value must match a user defined in Apache's user database. N.B.: removing the password in the access file also enables access to the monitor for any user connecting through the localhost interface -- DISABLE THE MONITOR if using this option on any box which allows untrusted logins. Shut down Zope by doing:: kill `cat var/Main.pid` from the Zope directory. Configure Zope At this point you are able to log in using the "superuser" identity. If you want other people defined in the Apache user database to have identities in Zope, you need to add them to a User Folder (the object whose ID is acl_users). Either click on the pre-defined acl_users in the top folder or add a User Folder object to a subfolder. Specific web servers Apache * As mentioned above, Apache does not pass authorization information to CGI scripts by default. See above for information on how to deal with this situation. * There is Apache module to support PCGI (mod_pcgi2). See the section "PCGI" above for details. Netscape Servers * Like Apache, Netscape does not pass HTTP Authorization information to CGI scripts. We have a plugin at our website that addresses this. http://www.zope.org * Alternatively, you can allow the web server to perform the authentication step. See above for more information. IIS * You must turn off Windows NT Challenge/Response authentication. To do this, go to IIS Manager, right-click on the server, select Service Properties, and deselect *both* 'Windows NT Challenge/Response' and, strangely, Basic Authentication from the Password Authentication area of the Service tabbed worksheet. * IIS kindly throws out PATH_INFO when writing to its logs, so if you want to log which Zope objects are actually being accessed, you will need to investigate an ISAPI filter * An ISAPI module to support PCGI is under development. * IIS 4.0 throws away Zope's error messages by default. This behavior can create quite a few problems, including authentication problems. Microsoft prides itself on the clear error messages that IIS 4.0 presents, when the user makes a mistake. These error messages are implemented in the form of Custom Error handlers, that return a file, or URL to a user when a certain error occurs. This means, that when you forget to fill in an Id when you want to create, say, a new SQL Method, Zope's clear error message is replaced by IIS's totally irrelevant error message. Also, it completely breaks authentication when the user uses IE5.0 when trying to log into a secure area of the server. These Custom Error handlers are enabled by default. Luckily, the handlers can quite easily be switched off: Open the IIS website in the Management Console, and navigate to the folder you put the PCGI executable in. If you named your Zope installation 'Zope', it will be called 'Zope.exe'. Double-click on that file. A property page will appear. Select the 'Custom Errors' tab. Now, select every HTTP Error code in the listbox that doesn't have type 'Default', and click on the 'Set to Default' button for each one. This will disable IIS overriding the error message returned by Zope. Click OK, and voila, Zope is allowed to tell the world what it thinks went wrong. Zope-2.8.7-final/doc/ZEO/0000775000076400007640000000000010435500334013737 5ustar ajungajungZope-2.8.7-final/doc/ZEO/trace.txt0000664000076400007640000001661310435500334015605 0ustar ajungajungZEO Client Cache Tracing ======================== An important question for ZEO users is: how large should the ZEO client cache be? ZEO 2 (as of ZEO 2.0b2) has a new feature that lets you collect a trace of cache activity and tools to analyze this trace, enabling you to make an informed decision about the cache size. Don't confuse the ZEO client cache with the Zope object cache. The ZEO client cache is only used when an object is not in the Zope object cache; the ZEO client cache avoids roundtrips to the ZEO server. Enabling Cache Tracing ---------------------- To enable cache tracing, you must use a persistent cache (specify a ``client`` name), and set the environment variable ZEO_CACHE_TRACE to a non-empty value. The path to the trace file is derived from the path to the persistent cache file by appending ".trace". If the file doesn't exist, ZEO will try to create it. If the file does exist, it's opened for appending (previous trace information is not overwritten). If there are problems with the file, a warning message is logged. To start or stop tracing, the ZEO client process (typically a Zope application server) must be restarted. The trace file can grow pretty quickly; on a moderately loaded server, we observed it growing by 7 MB per hour. The file consists of binary records, each 34 bytes long if 8-byte oids are in use; a detailed description of the record lay-out is given in stats.py. No sensitive data is logged: data record sizes (but not data records), and binary object and transaction ids are logged, but no object pickles, object types or names, user names, transaction comments, access paths, or machine information (such as machine name or IP address) are logged. Analyzing a Cache Trace ----------------------- The stats.py command-line tool is the first-line tool to analyze a cache trace. Its default output consists of two parts: a one-line summary of essential statistics for each segment of 15 minutes, interspersed with lines indicating client restarts, followed by a more detailed summary of overall statistics. The most important statistic is the "hit rate", a percentage indicating how many requests to load an object could be satisfied from the cache. Hit rates around 70% are good. 90% is excellent. If you see a hit rate under 60% you can probably improve the cache performance (and hence your Zope application server's performance) by increasing the ZEO cache size. This is normally configured using key ``cache_size`` in the ``zeoclient`` section of your configuration file. The default cache size is 20 MB, which is small. The stats.py tool shows its command line syntax when invoked without arguments. The tracefile argument can be a gzipped file if it has a .gz extension. It will be read from stdin (assuming uncompressed data) if the tracefile argument is '-'. Simulating Different Cache Sizes -------------------------------- Based on a cache trace file, you can make a prediction of how well the cache might do with a different cache size. The simul.py tool runs a simulation of the ZEO client cache implementation based upon the events read from a trace file. A new simulation is started each time the trace file records a client restart event; if a trace file contains more than one restart event, a separate line is printed for each simulation, and a line with overall statistics is added at the end. Example, assuming the trace file is in /tmp/cachetrace.log:: $ python simul.py -s 4 /tmp/cachetrace.log CircularCacheSimulation, cache size 4,194,304 bytes START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE Jul 22 22:22 39:09 3218856 1429329 24046 41517 44.4% 40776 99.8 This shows that with a 4 MB cache size, the cache hit rate is 44.4%, the percentage 1429329 (number of cache hits) is of 3218856 (number of load requests). The cache simulated 40776 evictions, to make room for new object states. At the end, 99.8% of the bytes reserved for the cache file were in use to hold object state (the remaining 0.2% consists of "holes", bytes freed by object eviction and not yet reused to hold another object's state). Let's try this again with an 8 MB cache:: $ python simul.py -s 8 /tmp/cachetrace.log CircularCacheSimulation, cache size 8,388,608 bytes START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE Jul 22 22:22 39:09 3218856 2182722 31315 41517 67.8% 40016 100.0 That's a huge improvement in hit rate, which isn't surprising since these are very small cache sizes. The default cache size is 20 MB, which is still on the small side:: $ python simul.py /tmp/cachetrace.log CircularCacheSimulation, cache size 20,971,520 bytes START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE Jul 22 22:22 39:09 3218856 2982589 37922 41517 92.7% 37761 99.9 Again a very nice improvement in hit rate, and there's not a lot of room left for improvement. Let's try 100 MB:: $ python simul.py -s 100 /tmp/cachetrace.log CircularCacheSimulation, cache size 104,857,600 bytes START TIME DURATION LOADS HITS INVALS WRITES HITRATE EVICTS INUSE Jul 22 22:22 39:09 3218856 3218741 39572 41517 100.0% 22778 100.0 It's very unusual to see a hit rate so high. The application here frequently modified a very large BTree, so given enough cache space to hold the entire BTree it rarely needed to ask the ZEO server for data: this application reused the same objects over and over. More typical is that a substantial number of objects will be referenced only once. Whenever an object turns out to be loaded only once, it's a pure loss for the cache: the first (and only) load is a cache miss; storing the object evicts other objects, possibly causing more cache misses; and the object is never loaded again. If, for example, a third of the objects are loaded only once, it's quite possible for the theoretical maximum hit rate to be 67%, no matter how large the cache. The simul.py script also contains code to simulate different cache strategies. Since none of these are implemented, and only the default cache strategy's code has been updated to be aware of MVCC, these are not further documented here. Simulation Limitations ---------------------- The cache simulation is an approximation, and actual hit rate may be higher or lower than the simulated result. These are some factors that inhibit exact simulation: - The simulator doesn't try to emulate versions. If the trace file contains loads and stores of objects in versions, the simulator treats them as if they were loads and stores of non-version data. - Each time a load of an object O in the trace file was a cache hit, but the simulated cache has evicted O, the simulated cache has no way to repair its knowledge about O. This is more frequent when simulating caches smaller than the cache used to produce the trace file. When a real cache suffers a cache miss, it asks the ZEO server for the needed information about O, and saves O in the client cache. The simulated cache doesn't have a ZEO server to ask, and O continues to be absent in the simulated cache. Further requests for O will continue to be simulated cache misses, although in a real cache they'll likely be cache hits. On the other hand, the simulated cache doesn't need to evict any objects to make room for O, so it may enjoy further cache hits on objects a real cache would have evicted. Zope-2.8.7-final/doc/ZEO/README.txt0000664000076400007640000000106210435500334015434 0ustar ajungajungZEO Documentation ================= This directory contains ZEO documentation. howto.txt The first place to look. It provides a high-level overview of ZEO features and details on how to install and configure clients and servers. Includes a configuration reference. cache.txt Explains how the client cache works. trace.txt Describe cache trace tool used to determine ideal cache size. ZopeREADME.txt A somewhat dated description of how to integrate ZEO with Zope. It provides a few hints that are not yet in howto.txt. Zope-2.8.7-final/doc/ZEO/howto.txt0000664000076400007640000004302110435500334015640 0ustar ajungajung========================== Running a ZEO Server HOWTO ========================== Introduction ------------ ZEO (Zope Enterprise Objects) is a client-server system for sharing a single storage among many clients. Normally, a ZODB storage can only be used by a single process. When you use ZEO, the storage is opened in the ZEO server process. Client programs connect to this process using a ZEO ClientStorage. ZEO provides a consistent view of the database to all clients. The ZEO client and server communicate using a custom RPC protocol layered on top of TCP. There are several configuration options that affect the behavior of a ZEO server. This section describes how a few of these features working. Subsequent sections describe how to configure every option. Client cache ~~~~~~~~~~~~ Each ZEO client keeps an on-disk cache of recently used objects to avoid fetching those objects from the server each time they are requested. It is usually faster to read the objects from disk than it is to fetch them over the network. The cache can also provide read-only copies of objects during server outages. The cache may be persistent or transient. If the cache is persistent, then the cache files are retained for use after process restarts. A non-persistent cache uses temporary files that are removed when the client storage is closed. The client cache size is configured when the ClientStorage is created. The default size is 20MB, but the right size depends entirely on the particular database. Setting the cache size too small can hurt performance, but in most cases making it too big just wastes disk space. The document "Client cache tracing" describes how to collect a cache trace that can be used to determine a good cache size. ZEO uses invalidations for cache consistency. Every time an object is modified, the server sends a message to each client informing it of the change. The client will discard the object from its cache when it receives an invalidation. These invalidations are often batched. Each time a client connects to a server, it must verify that its cache contents are still valid. (It did not receive any invalidation messages while it was disconnected.) There are several mechanisms used to perform cache verification. In the worst case, the client sends the server a list of all objects in its cache along with their timestamps; the server sends back an invalidation message for each stale object. The cost of verification is one drawback to making the cache too large. Note that every time a client crashes or disconnects, it must verify its cache. Every time a server crashes, all of its clients must verify their caches. The cache verification process is optimized in two ways to eliminate costs when restarting clients and servers. Each client keeps the timestamp of the last invalidation message it has seen. When it connects to the server, it checks to see if any invalidation messages were sent after that timestamp. If not, then the cache is up-to-date and no further verification occurs. The other optimization is the invalidation queue, described below. Invalidation queue ~~~~~~~~~~~~~~~~~~ The ZEO server keeps a queue of recent invalidation messages in memory. When a client connects to the server, it sends the timestamp of the most recent invalidation message it has received. If that message is still in the invalidation queue, then the server sends the client all the missing invalidations. This is often cheaper than perform full cache verification. The default size of the invalidation queue is 100. If the invalidation queue is larger, it will be more likely that a client that reconnects will be able to verify its cache using the queue. On the other hand, a large queue uses more memory on the server to store the message. Invalidation messages tend to be small, perhaps a few hundred bytes each on average; it depends on the number of objects modified by a transaction. Transaction timeouts ~~~~~~~~~~~~~~~~~~~~ A ZEO server can be configured to timeout a transaction if it takes too long to complete. Only a single transaction can commit at a time; so if one transaction takes too long, all other clients will be delayed waiting for it. In the extreme, a client can hang during the commit process. If the client hangs, the server will be unable to commit other transactions until it restarts. A well-behaved client will not hang, but the server can be configured with a transaction timeout to guard against bugs that cause a client to hang. If any transaction exceeds the timeout threshold, the client's connection to the server will be closed and the transaction aborted. Once the transaction is aborted, the server can start processing other client's requests. Most transactions should take very little time to commit. The timer begins for a transaction after all the data has been sent to the server. At this point, the cost of commit should be dominated by the cost of writing data to disk; it should be unusual for a commit to take longer than 1 second. A transaction timeout of 30 seconds should tolerate heavy load and slow communications between client and server, while guarding against hung servers. When a transaction times out, the client can be left in an awkward position. If the timeout occurs during the second phase of the two phase commit, the client will log a panic message. This should only cause problems if the client transaction involved multiple storages. If it did, it is possible that some storages committed the client changes and others did not. Monitor server ~~~~~~~~~~~~~~ The ZEO server updates several counters while it is running. It can be configured to run a separate monitor server that reports the counter values and other statistics. If a client connects to the socket, the server send a text report and close the socket immediately. It does not read any data from the client. An example of a monitor server report is included below:: ZEO monitor server version 2.1a1 Fri Apr 4 16:57:42 2003 Storage: 1 Server started: Fri Apr 4 16:57:37 2003 Clients: 0 Clients verifying: 0 Active transactions: 0 Commits: 0 Aborts: 0 Loads: 0 Stores: 0 Conflicts: 0 Conflicts resolved: 0 Connection management ~~~~~~~~~~~~~~~~~~~~~ A ZEO client manages its connection to the ZEO server. If it loses the connection, it starts a thread that attempts to reconnect. While it is disconnected, it can satisfy some reads by using its cache. The client can be configured to wait a connection when it is created or to return immediately and provide data from its persistent cache. It usually simplifies programming to have the client wait for a connection on startup. When the client is disconnected, it polls periodically to see if the server is available. The rate at which it polls is configurable. The client can be configured with multiple server addresses. In this case, it assumes that each server has identical content and will use any server that is available. It is possible to configure the client to accept a read-only connection to one of these servers if no read-write connection is available. If it has a read-only connection, it will continue to poll for a read-write connection. This feature supports the Zope Replication Services product, http://www.zope.com/Products/ZopeProducts/ZRS. In general, it could be used to with a system that arranges to provide hot backups of servers in the case of failure. Authentication ~~~~~~~~~~~~~~ ZEO supports optional authentication of client and server using a password scheme similar to HTTP digest authentication (RFC 2069). It is a simple challenge-response protocol that does not send passwords in the clear, but does not offer strong security. The RFC discusses many of the limitations of this kind of protocol. Note that this feature provides authentication only. It does not provide encryption or confidentiality. The challenge-response also produces a session key that is used to generate message authentication codes for each ZEO message. This should prevent session hijacking. Guard the password database as if it contained plaintext passwords. It stores the hash of a username and password. This does not expose the plaintext password, but it is sensitive nonetheless. An attacker with the hash can impersonate the real user. This is a limitation of the simple digest scheme. The authentication framework allows third-party developers to provide new authentication modules. Installing software ------------------- ZEO is distributed as part of the ZODB3 package and with Zope, starting with Zope 2.7. You can download it from: - http://www.zope.org/Products/ZODB3.2, or - http://www.zope.org/Products/Zope To use ZEO with Zope 2.6, download ZODB3.2 and install it into your Zope software home. ZODB3 comes with a distutils setup.py script. You can use the --home option to setup.py install to the software in custom location. For example, if Zope is installed in /home/zope, then this command will install the new ZEO and ZODB: python setup.py install --home /home/zope The install command should create a /home/zope/lib/python/ZEO directoy. Configuring server ------------------ The script runzeo.py runs the ZEO server. The server can be configured using command-line arguments or a config file. This document describes only describes the config file. Run runzeo.py -h to see the list of command-line arguments. The runzeo.py script imports the ZEO package. ZEO must either be installed in Python's site-packages directory or be in a directory on PYTHONPATH. The configuration file specifies the underlying storage the server uses, the address it binds, and a few other optional parameters. An example is:: address zeo.example.com:8090 monitor-address zeo.example.com:8091 path /var/tmp/Data.fs path /var/tmp/zeo.log format %(asctime)s %(message)s This file configures a server to use a FileStorage from /var/tmp/Data.fs. The server listens on port 8090 of zeo.example.com. It also starts a monitor server that lists in port 8091. The ZEO server writes its log file to /var/tmp/zeo.log and uses a custom format for each line. Assuming the example configuration it stored in zeo.config, you can run a server by typing:: python /usr/local/bin/runzeo.py -C zeo.config A configuration file consists of a section and a storage section, where the storage section can use any of the valid ZODB storage types. It may also contain an eventlog configuration. See the document "Configuring a ZODB database" for more information about configuring storages and eventlogs. The zeo section must list the address. All the other keys are optional. address The address at which the server should listen. This can be in the form 'host:port' to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection (at least one '/' is required). A hostname may be a DNS name or a dotted IP address. If the hostname is omitted, the platform's default behavior is used when binding the listening socket ('' is passed to socket.bind() as the hostname portion of the address). read-only Flag indicating whether the server should operate in read-only mode. Defaults to false. Note that even if the server is operating in writable mode, individual storages may still be read-only. But if the server is in read-only mode, no write operations are allowed, even if the storages are writable. Note that pack() is considered a read-only operation. invalidation-queue-size The storage server keeps a queue of the objects modified by the last N transactions, where N == invalidation_queue_size. This queue is used to speed client cache verification when a client disconnects for a short period of time. monitor-address The address at which the monitor server should listen. If specified, a monitor server is started. The monitor server provides server statistics in a simple text format. This can be in the form 'host:port' to signify a TCP/IP connection or a pathname string to signify a Unix domain socket connection (at least one '/' is required). A hostname may be a DNS name or a dotted IP address. If the hostname is omitted, the platform's default behavior is used when binding the listening socket ('' is passed to socket.bind() as the hostname portion of the address). transaction-timeout The maximum amount of time to wait for a transaction to commit after acquiring the storage lock, specified in seconds. If the transaction takes too long, the client connection will be closed and the transaction aborted. authentication-protocol The name of the protocol used for authentication. The only protocol provided with ZEO is "digest," but extensions may provide other protocols. authentication-database The path of the database containing authentication credentials. authentication-realm The authentication realm of the server. Some authentication schemes use a realm to identify the logic set of usernames that are accepted by this server. Configuring client ------------------ The ZEO client can also be configured using ZConfig. The ZODB.config module provides several function for opening a storage based on its configuration. - ZODB.config.storageFromString() - ZODB.config.storageFromFile() - ZODB.config.storageFromURL() The ZEO client configuration requires the server address be specified. Everything else is optional. An example configuration is:: server zeo.example.com:8090 To use a ZEO client from Zope, write a configuration file and load it from custom_zodb.py:: from ZODB.config import storageFromURL Storage = storageFromURL("/path/to/client.txt") The other configuration options are listed below. storage The name of the storage that the client wants to use. If the ZEO server serves more than one storage, the client selects the storage it wants to use by name. The default name is '1', which is also the default name for the ZEO server. cache-size The maximum size of the client cache, in bytes. name The storage name. If unspecified, the address of the server will be used as the name. client Enables persistent cache files. The string passed here is used to construct the cache filenames. If it is not specified, the client creates a temporary cache that will only be used by the current object. var The directory where persistent cache files are stored. By default cache files, if they are persistent, are stored in the current directory. min-disconnect-poll The minimum delay in seconds between attempts to connect to the server, in seconds. Defaults to 5 seconds. max-disconnect-poll The maximum delay in seconds between attempts to connect to the server, in seconds. Defaults to 300 seconds. wait A boolean indicating whether the constructor should wait for the client to connect to the server and verify the cache before returning. The default is true. read-only A flag indicating whether this should be a read-only storage, defaulting to false (i.e. writing is allowed by default). read-only-fallback A flag indicating whether a read-only remote storage should be acceptable as a fallback when no writable storages are available. Defaults to false. At most one of read_only and read_only_fallback should be true. realm The authentication realm of the server. Some authentication schemes use a realm to identify the logic set of usernames that are accepted by this server. A ZEO client can also be created by calling the ClientStorage constructor explicitly. For example:: from ZEO.ClientStorage import ClientStorage storage = ClientStorage(("zeo.example.com", 8090)) Running the ZEO server as a daemon ---------------------------------- In an operational setting, you will want to run the ZEO server a daemon process that is restarted when it dies. The zdaemon package provides two tools for running daemons: zdrun.py and zdctl.py. The document "Using zdctl and zdrun to manage server processes" (Doc/zdctl.txt) explains how to use these scripts to manage daemons. Rotating log files ~~~~~~~~~~~~~~~~~~ ZEO will re-initialize its logging subsystem when it receives a SIGUSR2 signal. If you are using the standard event logger, you should first rename the log file and then send the signal to the server. The server will continue writing to the renamed log file until it receives the signal. After it receives the signal, the server will create a new file with the old name and write to it. Tools ----- There are a few scripts that may help running a ZEO server. The zeopack.py script connects to a server and packs the storage. It can be run as a cron job. The zeoup.py script attempts to connect to a ZEO server and verify that is is functioning. The zeopasswd.py script manages a ZEO servers password database. Diagnosing problems ------------------- If an exception occurs on the server, the server will log a traceback and send an exception to the client. The traceback on the client will show a ZEO protocol library as the source of the error. If you need to diagnose the problem, you will have to look in the server log for the rest of the traceback. Zope-2.8.7-final/doc/ZEO/ZopeREADME.txt0000664000076400007640000000713010435500334016274 0ustar ajungajungZope Enterprise Objects (ZEO) Installation ZEO 2.0 requires Zope 2.4 or higher and Python 2.1 or higher. If you use Python 2.1, we recommend the latest minor release (2.1.3 as of this writing) because it includes a few bug fixes that affect ZEO. Put the package (the ZEO directory, without any wrapping directory included in a distribution) in your Zope lib/python. The setup.py script in the top-level ZEO directory can also be used. Run "python setup.py install --home=ZOPE" where ZOPE is the top-level Zope directory. You can test ZEO before installing it with the test script:: python test.py -v Run the script with the -h option for a full list of options. The ZEO 2.0b2 release contains 122 unit tests on Unix. Starting (and configuring) the ZEO Server To start the storage server, go to your Zope install directory and run:: python lib/python/ZEO/start.py -p port_number This run the storage sever under zdaemon. zdaemon automatically restarts programs that exit unexpectedly. The server and the client don't have to be on the same machine. If they are on the same machine, then you can use a Unix domain socket:: python lib/python/ZEO/start.py -U filename The start script provides a number of options not documented here. See doc/start.txt for more information. Running Zope as a ZEO client To get Zope to use the server, create a custom_zodb module, custom_zodb.py, in your Zope install directory, so that Zope uses a ClientStorage:: from ZEO.ClientStorage import ClientStorage Storage = ClientStorage(('', port_number)) You can specify a host name (rather than '') if you want. The port number is, of course, the port number used to start the storage server. You can also give the name of a Unix domain socket file:: from ZEO.ClientStorage import ClientStorage Storage = ClientStorage(filename) There are a number of configuration options available for the ClientStorage. See doc/ClientStorage.txt for details. If you want a persistent client cache which retains cache contents across ClientStorage restarts, you need to define the environment variable, ZEO_CLIENT, or set the client keyword argument to the constructor to a unique name for the client. This is needed so that unique cache name files can be computed. Otherwise, the client cache is stored in temporary files which are removed when the ClientStorage shuts down. For example, to start two Zope processes with unique caches, use something like:: python z2.py -P8700 ZEO_CLIENT=8700 python z2.py -P8800 ZEO_CLIENT=8800 Zope product installation Normally, Zope updates the Zope database during startup to reflect product changes or new products found. It makes no sense for multiple ZEO clients to do the same installation. Further, if different clients have different software installed, the correct state of the database is ambiguous. Zope will not modify the Zope database during product installation if the environment variable ZEO_CLIENT is set. Normally, Zope ZEO clients should be run with ZEO_CLIENT set so that product initialization is not performed. If you do install new Zope products, then you need to take a special step to cause the new products to be properly registered in the database. The easiest way to do this is to start Zope once with the environment variable FORCE_PRODUCT_LOAD set. The interaction between ZEO and Zope product installation is unfortunate. Zope-2.8.7-final/doc/ZEO/cache.txt0000664000076400007640000000360210435500334015544 0ustar ajungajungZEO Client Cache The client cache provides a disk based cache for each ZEO client. The client cache allows reads to be done from local disk rather than by remote access to the storage server. The cache may be persistent or transient. If the cache is persistent, then the cache file is retained for use after process restarts. A non- persistent cache uses a temporary file. The client cache is managed in a single file, of the specified size. The life of the cache is as follows: - The cache file is opened (if it already exists), or created and set to the specified size. - Cache records are written to the cache file, as transactions commit locally, and as data are loaded from the server. - Writes are to "the current file position". This is a pointer that travels around the file, circularly. After a record is written, the pointer advances to just beyond it. Objects starting at the current file position are evicted, as needed, to make room for the next record written. A distinct index file is not created, although indexing structures are maintained in memory while a ClientStorage is running. When a persistent client cache file is reopened, these indexing structures are recreated by analyzing the file contents. Persistent cache files are created in the directory named in the ``var`` argument to the ClientStorage, or if ``var`` is None, in the current working directory. Persistent cache files have names of the form:: client-storage.zec where: client -- the client name, as given by the ClientStorage's ``client`` argument storage -- the storage name, as given by the ClientStorage's ``storage`` argument; this is typically a string denoting a small integer, "1" by default For example, the cache file for client '8881' and storage 'spam' is named "8881-spam.zec". Zope-2.8.7-final/doc/ZOPE3.txt0000775000076400007640000002035410435500316014712 0ustar ajungajungUsing Zope 3 Components in Zope 2 Applications Background Zope 3 is the next major revision of Zope. It is designed to be more 'programmer-centric' and easier to learn, use and extend. Zope 3 introduces an interface-centric component architecture that makes it easier to develop and deploy components without requiring developers to learn and understand the entire Zope framework. Much more information on Zope 3 is available on the Zope 3 area of the zope.org Website at http://dev.zope.org/Zope3/. As of Zope 2.8, the "Five" project has been integrated into the Zope 2 core. The "Five" project implements a compatibility layer that allows many Zope 3 components and patterns to be used in new and existing Zope 2 applications. This provides a number of benefits:: - availability of Zope 3 technologies in Zope 2 like the component architecture and declarative configuration - you can gradually evolve your Zope 2 projects to better plan for migration to Zope 3 - you start learning about Zope 3 right now, preparing yourself better for the future. Since Zope 3 is open to contributions, you could even influence your future for the better ;) Features The Five integration layer provides support for Zope 3 interfaces, Zope Configuration Markup Language (ZCML), adapters, views, utilities, schema-driven content, and Zope 3 add and edit forms. Note that the Five layer does *not* attempt to provide a ZMI user interface for Zope 3 components. Zope 2.8+ includes the essential Zope 3 packages, so it is not necessary to install Zope 3. Zope 3 Interfaces Everything in the zope.interface package should work. Zope 3 interfaces are the foundation of the component architecture, and also the foundation of schemas. Historically, Zope 2 has used the ``__implements__`` class attribute for interface declarations. Zope 2 cannot detect Zope 3 interfaces and the Zope 3 machinery cannot detect Zope 2 interfaces. This is a good thing, as Zope 2 has no way to deal with Zope 3 interfaces, and Zope 3 cannot work with Zope 2 interfaces. This means you can safely make a class declare both a Zope 2 and Zope 3 interface independently from each other. It's a rare case where you need this - you're usually better off just switching to ``implements()`` for your application if possible. Switching from Zope 2 interfaces to Zope 3 interfaces is easy -- just make your interfaces inherit from ``zope.interface.Interface`` instead of ``Interface.Interface`` (or ``Interface.Base``). Next, change all ``__implements__`` to ``implements()``. You will also have to change any calls to ``isImplementedBy`` and such in your application to ``providedBy``, as ``isImplementedBy`` has been deprecated (you'll see the DeprecationWarnings in your Zope log). ZCML ZCML is the Zope Configuration Markup Language, an XML application. Zope 3 code consists of a lot of components that can be plugged together using ZCML. If you put a 'site.zcml' in the home directory of your Zope instance, this is the root of the ZCML tree. An example of 'site.zcml' is in 'site.zcml.in'. If you don't place a site.zcml, Five falls back on 'fallback.zcml'. ZCML in Zope 2 with Five has a special directive, 'five:loadProducts', to load the ZCML ('meta.zcml', 'configure.zcml') of all installed Zope 2 products, if available. Another special directive, 'five:loadProductsOverrides' is available to load any overriding ZCML (overrides.zcml) in these products. In the 'overrides.zcml' you can override existing views or adapters, in this or in other products. The Five integration layer tries to use the Zope 3 ZCML directives where possible, though it does sometimes support only a subset of the possible attributes. It also introduces a few directives of its own under the ``five`` namespace. The supported directives are listed per namespace in alphabetic order below. zope ``http://namespaces.zope.org/zope`` ======================================== adapter ------- Hook an adapter factory to an interface. content ------- Declare interface and permissions on content object. Declares Zope 2 permissions. permission ---------- Way to make Zope 2 permissions available to Five, ``title`` is permission name. redefinePermission ------------------ Redefine a permission in included ZCML as another one. service ------- Declare a global service serviceType ----------- Declare a type of service. skin ---- Declare a skin, consisting of layers. utility ------- Declare a global utility. browser ``http://namespaces.zope.org/browser`` ============================================== page ---- Declare a page view for an interface. Permission is a Zope 2 permission. pages ----- Declare multiple page views for an interface. Permissions are Zope 2 permissions. defaultView ----------- Declare the name of the view that should be used for the default when viewing the object; i.e. when the object is traversed to without a view. defaultSkin ----------- Declare the default skin used. interface --------- Define an interface in ZCML. layer ----- Declare a layer. menu ---- Declare a menu menuItem, menuItems ------------------- Declare menuItems five ``http://namespaces.zope.org/five`` ======================================== implements ---------- Make a class declare it implements an interface. loadProducts ------------ Loads ZCML in all Zope 2 products. First processes all ``meta.zcml`` files, then processes all ``configure.zcml`` files. loadProductsOverrides --------------------- Loads overriding ZCML in all products (``overrides.zcml``). traversable ----------- Make a Zope 2 content class traversable in the Zope 3 manner using Five. This is used to attached views, resources and other things to Zope 2 objects. defaultViewable --------------- Make a Zope 2 content class use a Zope 3 default view when looking at it without any paths appended to it. This works then instead of ``index_html`` in Zope 2. pagesFromDirectory ------------------ Load all *.pt files in a directory as pages. Useful when you want to share templates between Five and CMF, so you can declare pages like this is a similar way to setting up skin folders in portal_skins. browser:editform ---------------- Create an edit form based on a schema. browser:addform --------------- Create an add form based on a schema. Adapters Generally, adapters can now be used in Zope 2 just as they are used in Zope 3. Zope 3 Views Zope 3 views work in Zope 2 with Five, including layers and skins. To make them work however, you need to make a Zope 2 class "traversable". This can be done by using the 'five:traversable' directive in ZCML. Schemas and Forms With the Five layer, it is possible to use schema-based content using the standard Zope 3 patterns. Security declarations Five aims to eradicate declareProtected, ClassSecurityInfo and initializeClass from your Zope 2 code. In order to do this, Five provides the Zope 3 way of declaring permissions from ZCML, but uses the Zope 2 mechanisms to actually set them. To declare permissions for methods and templates on views you use the permission attribute on the browser:page directive, and specify a Zope 2 permission (given a Zope 3 name). You can find a list of these permissions in permissions.zcml in Five. The permission check takes place before the view is executed. The content directive can also be used to declare permissions on Zope 2 content classes. Note however that these permissions will be ignored by views anyway, as they are trusted -- it only serves to protect directly exposed methods on content classes (the python scripts and the ZPublisher). Zope-2.8.7-final/doc/CREDITS.txt0000664000076400007640000000422610435500316015144 0ustar ajungajungCredits The Zope software receives contributions from far and wide. Here's the Zope Hall of Fame: o Stephen Purcell allows us to distribute his PyUnit unit testing framework with Zope. o Jeff Bauer is Zope Dude Number One. Jeff took over PCGI and kept pushing it forward through the years. o Sam Rushing worked with us at Digital Creations to make Medusa the publishing platform for ZServer and the concurrency of Zope2. o A subset of windows guru Mark Hammond's win32 extensions are bundled with win32 binary distributions of Zope. o Martijn Pieters and Brian Hooper contributed the #in reverse attribute. o Phillip Eby contributed the DTML 'let' tag and many other useful ideas, including the inspiration for the DTML 'call', 'with' and 'return' tags. o The DateTime module was based on work from Ted Horst. o Jordan Baker contributed the 'try' tag, something we've wanted for a long, long time. o Martijn Pieters chipped in with a safe range function. o Michael Hauser came up with the name "Zope". o Eric Kidd from Userland contributed to ZPublisher's support for XML-RPC. o Andrew M. Kuchling wrote the initial version of mod_pcgi, making him extremely cool in our book. o Oleg Broytmann has taken up the standard of mod_pcgi and moving it to be a really amazing thing, and ready for prime time. o Jephte CLAIN made some patches to European ZopeTime. o Thanks to Gregor Hoffleit for his work in getting Zope into the Debian distribution. o All the other Zopistas far and wide that stuck with us during the Bobo/Principia days and politely push us to make the best damn app server on this or any other planet. o Of course the list of credits would be quite incomplete without mentioning Guido van Rossum, benevolent dictator of Python and long-time friend of Digital Creations. Zope Power is Python Power. o Special thanks to Richard Stallman and the Free Software Foundation for their assistance and feedback on the GPL-compatible 2.0 version of the Zope Public License. Zope-2.8.7-final/doc/PLATFORMS/0000775000076400007640000000000010435500316014651 5ustar ajungajungZope-2.8.7-final/doc/PLATFORMS/README.txt0000664000076400007640000000022510435500316016346 0ustar ajungajungThis directory is to collect notes that people send us on using Zope on Platforms that we (Digital Creations) don't have and can't support directly. Zope-2.8.7-final/doc/PLATFORMS/AIX.txt0000664000076400007640000000156410435500316016041 0ustar ajungajung from Thiebaut Champenier "python wo_pcgi.py" doesn't build the extensions correctly due to linking problems. To fix it, I just SymLinked ld_so_aix, makexp_aix and python.exp (found in /Modules/) into each of the dirs in /python/lib that contain .c files... and it worked just great. But I'm sure it's not the best solution from Wolfgang Roesner (reporting segfaults and a fix) The default stack size for a pthread on AIX (at least in my configuration) is 96kB. On Solaris it's 1M. As soon as I explicitly set the initial stack size to 1M the problem went away and it appears as if Zope is up and running. Unfortunately I really had to change "thread_pthread.h" in Python which obviously is not a nice thing to do as I now have a need for my own private version of Python. But at least I have a running version of Zope on AIX. Zope-2.8.7-final/doc/PLATFORMS/BSD.txt0000664000076400007640000000161010435500316016020 0ustar ajungajungSteps to building python for use with Zope on FreeBSD to avoid segmentation faults (by Andrew Sawyers for Pythons up to 2.3). The default thread stack size for FreeBSD is too small; when building Python for use with Zope it is often necessary to increase the thread stack size to avoid segmentation faults. There are two ways which you can increase the default python thread stack size. Way #1: In your Python source directory: * >cd Python * >edit thread_pthreads.h at the top of the header file, add #define THREAD_STACK_SIZE 1000000 * >save * >cd ..; ./configure Way #2: In your Python source directory: * >./configure * >make OPT="-DNDEBUG -g -O3 -Wall -Wstrict-prototypes -DTHREAD_STACK_SIZE=1000000" Passing in the define or adding the define to your source solves all experiences I have experienced with Zope segfaulting while running on my FreeBSD platforms. Zope-2.8.7-final/doc/HISTORY.txt0000775000076400007640000114407310435500316015221 0ustar ajungajungZope History This file contains change information for previous versions of Zope. Change information for the current release can be found in the file CHANGES.txt. Zope 2.7.5 RC 2 Bugs fixed - Renamed utility methods in zdaemon/tests/test_zdrun.py to avoid a clash with unittest.TestCase methods (the clash surfaced only under Python 2.4.1c1, due to subtle changes in MRO behavior). - Removed redundant 'getIndexSourceNames' implementation in lib/python/Products/PluginIndexes/common/UnIndex.py. - Hardened a locale-sensitive unit test in Zope/Startup/tests/testStarter.py against platform-specific return type of 'getlocale'. - Updated doc/INSTALL.txt to match currently preferred and acceptable Python versions. - Use 'del' instead of 'list.remove()' in Catalog.delColumn(). There can be only one column with the same name, and it could potentially break catalog metadata as remove() may remove more than one element from the list if they have the same value. Also, we already have the list index we are interested in deleting so it doesn't make sense to look up the value and call 'list.remove()' on it. Zope 2.7.5 RC 1 (2005/03/09) Bugs fixed - Collector #1721: Fixed handling of the 'extra' parameter in UnIndex.py Zope 2.7.5 beta 1 (2005/02/28) Features added - Collector #533: ZMI Find now searches in text Files. Bugs fixed - Collector #1705: CopySource._postCopy is never called - guarded_getattr: Restored ability to aquire "through" unprotected contexts, broken through overzealous cleanup in Zope 2.7.3. - ZEO/zrpc: Fixed several thread and asyncore races in ZEO's connection dance. - Collector #1683: fixing batching in the DA "Test" tab - Got rid of import order dependencies in PageTemplates, TAL, and ZTUtils. This also means that the Z2 implementation of PageTemplates no longer works outside of Zope. - Collector #1648: Fix bug in Medusa FTP - OFS.OrderSupport: Added tpValues method. Trees like the ZMI menu did not show subfolders in the defined order. (http://collector.zope.org/Zope/1339) - The IStreamIterator interface had inappropriate "self" arguments in its method definitions. - Clarified implementation of getUserById. The method should return None if the requested user does not exist. - Collector #1667: allow 'max-number-of-session-objects 0' to have the same effect as setting the value via the web interface (i.e., make the number of session objects unlimited, rather than falling back to the default). - Collector #1661: make 'python-check-interval' setting in zope.conf actually work as documented. This setting allows for important tuning opportunities for production Zope servers. - Collector #1657: Don't break host-based virtual hosting when purging an HTTP accelerator cache. - Collector: #1651: removed compiler warning Zope 2.7.4 final (2005/01/15) Bugs fixed - Restored test.py to the version shipping with Zope 2.7.3. - Collector #1655: fixed severe memory leak in TemporaryStorage - Collector #1656: Several iterator operations were broken, particularly enumerate(). - DTML Methods were not interoperable with the new filestream_iterator and caches based on it (FileCacheManager). Zope 2.7.4 RC 2 (2005/01/07) Bugs fixed - Collector #1407: fixed XML escaping problem introduced in 2.7.4 b1 - moved Docutils back to lib/python/docutils - removed sitecustomize.py file introduced in 2.7.4b1 - updated Docutils to V 0.3.7 Zope 2.7.4 RC 1 (2004/12/23) Bugs fixed - Added a workaround that allows MSIE's (buggy) WebFolders client to display modified and created dates properly for Zope objects. - Running Zope as a service under Windows did not work. - Collector #1626: webdav.NullResource raised Forbidden as a last resort for any exception type, which masked genuine errors and also caused problems for clients legitimiately expecting an Unauthorized response. Fixed to raise Unauthorized if we get a CopyError, and let everything else pass through. - Collector #1151: HTTP compression was broken on error pages Zope 2.7.4 beta 2 (2004/12/15) Bugs fixed - Collector #1617: fixed unchecked buffer accesses and unchecked Python API calls in cAccessControl.c (thanks to Tim Peters) Zope 2.7.4 beta 1 (2004/12/02) Features added - RAMCacheManager: Allow invalidation of a cache entry from the Statistics view in the ZMI - Collector #1454/OFS.File: Accept content types ending with "javascript" as editable through the File edit form, just like text/ types Bugs fixed - The REQUEST now contains a new entry ACTUAL_URL which contains the full URL without query string as it appears within the location bar of the browser. The key has been added to provide a single key that is available for vhosted and non-vhosted installations. - Collector #1605: VHM did not quote URLs - webdav.NullResource: doing a PUT of a large file could potentially bring your box to a halt, because the whole file would be read into memory just to detect the content-type. Added a 'large-file-threshold' directive to control the boundary where a file gets read completely into memory. The same directive controls the creation of a tmpfile vs. reading the whole request into memory on ZServer as well. - The 'connection-limit' directive was not taking effect as it modified a module-level global that was already bound to the functions that used it by the time the directive took effect. Modified the functions so that they import the variable in the function body instead of at the top of the module. - webdav.Resource: during COPY, manage_afterClone was called way too early, thus the object wasn't bound to the database and couldn't find a context. Changed to behave the same way as CopySupport. - RAMCacheManager: opimized performance by using cPickle instead of pickle and by using the highest pickle protocol available instead of using ASCII pickles (patch by Dieter Maurer) - Collector #631: Image URLs in StructuredText containing port numbers were not rendered correctly - Collector #1003: added new 'http-header-max-length' directive to zope.conf to specific the maximum length of a HTTP request header before it is considered as a possible DoS attack and discarded. - bin/zopectl test now uses os.execv, instead os os.system, so that options with characters that needs shell quoting doesn't break the command. - Updated doc/UNITTEST.txt and lib/python/Testing/README.txt to reflect progress made since UNITTEST.txt was originally written. - Collector #1498: Don't choke on malformed cookies. Cookies of the form "foo=bar; hmm; baz=gee" will give an empty value for 'hmm' instead of silently discarding it and the rest of the string. (Thanks to 'sirilyan' for the patch.) - Collector #945: Couldn't programmatically add "empty" PythonScripts. - Collector #1219: XML export is working again (Thanks Tres) - Collector #1510: Allow encoding of application/xhtml+xml pages according to the charset specified in the Content-Type header (thanks to Jacek Konieczny for the patch). - Collector #1599: made sqltest work with unicode strings (thanks to Peter Sabaini for the patch). - zopectl: fixed handling of child processes (patch by Dieter Maurer) - Collector #1593: fixed dumb _get_id() implementation in OFS.CopySupport that produced copy_of_copy_of....files (thanks to Alexandre Boeglin for the patch). - Collector #1450: files in utilities/ZODBTools are now installed during the installation process in the 'bin' directory - Collector #1371: added new 'cgi-maxlen' directive to zope.conf to limit the amount of form data being processed by Zope to prevent DoS attacks - Collector #1407: changed WebDAV display name for objects to title_or_id() - the 'trusted-proxy' directive in zope.conf now also accepts hostnames instead of IP addresses only (patch by Dieter Maurer) - Fixed test.py to not over-resolve symbolic links. Needed to run tests when the Products directory and a product are symlinks. - Collector #1583/ZReST: Fixed handling of the title attribute for non-ascii characters. - Collector #1577: Fixed cryptic error message in ZPublisher if a non-ASCII string is passed to a date, int, long or float property. - Collector #1576: Fixed Z Search Interface to use proper HTML. - Collector #1569/DateTime: Added a new ISO8601-method that will return correctly formatted ISO 8601-representations to augment the ISO method which isn't compliant with ISO 8601. - Collector #1127: strftime did not take timezone into account. - ZPublisher: changed some hardcoded 'latin1' arguments to 'iso-8859-15' since latin1 is obsolete. - Collector #1566: Installation of Zope on some older Solaris versions could fail due to a broken "echo" implementation causing the creation of a borked version.txt file. - Collector #934: Image and File objects are now always internally split into small chunks even when initialized from a string. - docutils: updated to V 0.3.5. The Zope core now contains a full copy of the docutils package except some GPLed files which can not be included with the Zope distribution due to license constraints on cvs.zope.org. - docutils: moved from lib/python/docutils to lib/python/third_party/docutils - Collector #1557/OFS.Image: Introducing new 'alt' property. The 'alt' attribute is no longer taken from the 'title' property but from the new 'alt' property. The border="0" attribute is no longer part of the HTML output except specified otherwise. - Set a default value of '' for the new 'alt' property as not to break existing content. - Collector #1511: made IPCServer show up in the Control Panel under "Network Services" - Collector #1443: Applied patch by Simon Eisenmann that reimplements the XML parser used in WebDAV fixing a memory leak. - Always unescape element contents on webdav.xmltools - Use saxutils to escape/unescape values for/from PROPFIND/PROPPATCH. - Make OFS.PropertySheet use the escaping function from webdav.xmltools. - Escape/unescape " and ' - Don't escape properties stored as XML (ie: having a __xml_attrs__ metadata set by PROPPATCH) when building a PROPFIND response. - If a PROPPATCH element value contains only a CDATA section, store the CDATA contents only. - Catch AttributeErrors and KeyErrors raised from __bobo_traverse__ and convert them to NotFound. In debug mode a more verbose error message is issued, the same way it's done on attribute/item traversal. Zope 2.7.3 final (2004/10/24) Bugs fixed Zope 2.7.3 beta 2 (2004/10/11) Bugs fixed - PageTemplates/TAL: Enabled i18n namespace for XML mode. http://collector.zope.org/Zope/1101 - Re-added the DAV header (Collector #1518) which had been removed in beta 1. - Added a "reset" button to the TOC management page and a reset argument to the associated target method. This allows users to decide to ditch the contents of their TOC and "start over". (Useful for people having problems with upgrades from older Zope versions). - Make TransientObjectContainer __setstate__ actually work. Upgrades should be flawless from Zope 2.7.0, 2.7.1, and 2.7.2. Upgrades from 2.6.3 will work, but data may not be retained. - Changed default resolution for new TransientObjectContainers to 60 seconds (this is helpful to prevent certain classes of conflicts, see http://www.plope.com/Members/dunny/conflicts for more info). Zope 2.7.3 beta 1 (2004/09/26) Features added - Made test.py follow symbolic links on POSIX systems. - added a set of unit tests for the SiteErrorLog. - Add a knob (not exposed to UI) to TransientObjectContainer to turn off "inband" housekeeping Housekeeping can now optionally be done using an external scheduling facility by calling the "housekeep" method regularly. Bugs fixed - Collector #1425: correct example for trace logger in zope.conf. - Collector #1457: ZCTextIndex's QueryError and ParseError are now available for import from untrusted code. - Collector #1491: talgettext.py did not create a proper header for the generated .pot file if multiple pagetemplate files were processed. - A composite of sessioning/ZODB/publisher bugfixes should help various KeyError/SystemError symptoms when using Plone / PTS or under very high sessioning load See http://zope.org/Collectors/Zope/1401 and http://collector.zope.org/Zope/789 . - TransientObject conflict resolver could potentially fail; when it failed, the conflict resolution machinery could resolve the TransientObject to None. (never reported) - Collector #1477: TaintedString.strip() now implements the same signature as str.strip() - Collector 789: Add Michael Dunstan's "explicit abort after error propagates into publisher" patch as documented in http://zope.org/Collectors/Zope/789 and as required by recent changes to ZODB which prevent a connection from being cleanly closed if modifications are extant in that connection. (Reports exist of improved sessioning behavior as a result also.) - If an object implementing the IStreamIterator interface (ala filestream_iterator) was returned by Zope to the publisher and for whatever reason, the ZServer response object was leaked, a reference to the stream iterator was kept around, which could result in a symptom of many files being held open by Zope after a large number of response objects had leaked. The response now explicitly removes the reference that it has to the stream producer after response output, sealing this variety of leak. - OFS: Fixed 'Last Modified' reverse sorting in main.dtml. - TAL: tal:on-error does not trap ConflictError anymore. - OFS.CopySupport: Enforced "Delete objects" permission during move (CMF Collector #259). - Removed DWIM'y attempt to filter acquired-but-not-aceessible results from 'guarded_getattr'. - Collector #1267: applied patch to fix segmentation faults on x86_64 systems - ZReST: the charset used in the rendered HTML was not set to the corresponding output_encoding property of the ZReST instance. In addition changing the encodings through the Properties tab did not re-render the HTML. - Collector #1234: an exception triple passed to LOG() was not propagated properly to the logging module of Python - Python 2.3.4 is now the recommended Python version, Python 2.3.3 is still a valid choice - Collector #1441: Removed headers introduced to make Microsoft webfolders and office apps happy, since they make a lot of standards-compliant things unhappy AND they trick MS Office into trying to edit office files stored in Zope via WebDAV even when the user isn't allowed to edit them and is only trying to download them. If you NEED this functionality then uncomment the lines prefixed with: # XXMSXX ...in the following files: lib/python/webdav/Resource.py lib/python/webdav/Collection.py lib/python/ZServer/HTTPResponse.py AND file an issue in http://www.zope.org/Collectors/Zope since these headers are getting totally deleted in Zope 2.8 unless you do! - Collector #1445: Fixed bad interaction between -p and -v(v) options to test.py that resulted in exceptions being printed when they shouldn't have been. - Collector #1370: Fixed html generated by Z Search interface. - Collector #729: manage_main doesn't display the correct page title most of the time. It is not completely fixed but using title_or_id makes folders display the correct id as a fallback. - Collector #1295: Fixed minor niglet with the Elvis tutorial. - Collector #1436: applied patch to fix a memory leak in cAccessControl. - Collector #1435: fixed mis-spelled variable name in zopeservice.py - Collector #1431: fix in initgroups.c for NetBSD - Added Stefan Holek's changes to test.py that allow for tests to be run from an instance - added "version.txt" to setup.py to avoid untrue "unreleased version" messages within the control panel Zope 2.7.2 final (2004/07/21) Bugs fixed - ZEO/ClientStorage: fixed check for temporary cache files (patch by Dieter Maurer) Zope 2.7.2 RC1 (2004/07/14) Bugs fixed - Collector #1403: Transience._getCallback raised NameError when a path naming an onAdd or onDelete method pointed to a nonexistent object. - Collector #1384: Fixed LazyCat length calculation after all contained sequences have been fully accessed. - Collector #1386: Fixed ISO 1386, making the colon in the timezone offset optional. - Collector #1392: ExternalMethod ignored management_page_charset - Fix for issue 233 in the Zope 3 collector. This corresponds to the Hotfix_2004-07-13 product. Zope 2.7.1 final (2004/06/21) Bugs fixed - reST support has been broken due to a failed import Zope 2.7.1 beta2 (2004/06/12) Bugs fixed - Zope can now be embedded in C/C++ without exceptions being raised in zdoptions. - Sessioning machinery "delete notifier" could in many cases be called much later than the expiration of the actual session data object it was called upon. It is now guaranteed to be called on the first exercising of the session machinery after object expiration + 'session-resolution-seconds'. - Collector #1293: missing 'address' parameters within one of the server sections raise an exception. - Collector #1265: Fixed handling of orphans in ZTUtil.Batch - Collector #1213: Fixed wrong labels of cache parameters - Collector #596: Page Templates can now be added via ZPublisher.Client or any other situation in which REQUEST is passed without a file and a text arg is provided. Features Added - Collector #1317: FindSupport.ZopeFind now searches in SearchableText() if available (i.e. CMF content) Zope 2.7.1 beta1 (2004/05/23) Features Added - Collector #1233: Scripts written before the new configuration behavior in Zope 2.7 used to be able to simply do "import Zope; app = Zope.app()" to get a hold of the root Zope object. With the introduction of the new configuration machinery, an extra step needed to be taken, so the equivalent becomes "import Zope; Zope.configure('/path/to/configfile'); app=Zope.app()". This feature "fixes" that by allowing an environment variable "ZOPE_CONFIG" to be set. If the ZOPE_CONFIG envvar is set, the "Zope.configure(..)" line can be omitted, as the startup process will take the vaule of ZOPE_CONFIG as the config file name. - New top-level configuration file directive: 'session-esolution-seconds'. This represents an integer value specifying the number of seconds to be used as the "timeout resolution" of the '/temp_folder/session_data' transient object container in Zope's object database. - New knob for transient object containers: 'timeout resolution (in seconds). Setting this higher than the default (20 seconds) allows the transience machinery to do fewer "writes" at the expense of causing items to time out later than the "Data object timeout value" by a factor of (at most) this many seconds. This number must divide evenly into the number of timeout seconds ("Data object timeout value" * 60) and cannot be set higher than the timeout value in seconds. - New "transience" implementation which is more stable under high load. This fixes some sessioning problems. Much gratitude to Michael Dunstan for providing a function testing rig for sessioning that allowed for the reproduction of various failures at will! - Recataloging a ZCatalog instance is now more safe and predictable. Indexing errors are catched and logged. In addition the progress of the recataloging operation is logged. So one can see how much documents are already processed and how much documents are remaining. - AccessControl/Role.py: permission_settings() has a new optional parameter 'permission' to retrieve the settings for a specific permission. - OFS.OrderSupport: Added optional 'subset_ids' argument to move methods. This is useful in combination with views that filter out some hidden sub-objects. - Zope application code can now return a "stream iterator" object to ZPublisher. If a stream iterator is returned from application code, it will be unwound by Zope's networking code and its data will be rendered as the body of the response to a client. An example: def method_returning_a_stream_iterator(self): import os, stat from ZPublisher.Iterators import filestream_iterator path = '/var/zope/Z2.log' size = os.stat(path)[stat.ST_SIZE] self.REQUEST.RESPONSE.setHeader('Content-Length', size) return filestream_iterator('/var/zope/Z2.log', 'r') 'filestream_iterator' is a class which implements the "IStreamIterator" interface, which just signifies a "normal" Python iterator that is guaranteed to iterate over a stream of bytes. This interface is defined within the ZPublisher.Iterators module. Any instance of a class which implements this interface may be returned to ZPublisher. Before a stream iterator is returned from Zope code the 'Content-Length' header of the response *must* be set. The major use for this feature is to allow application code to serve static files from the filesystem without first needing to read all file data into memory or explicitly chunking data from static files out via 'RESPONSE.write' (both of which are slower than just letting medusa itself handle the output via a native producer). http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing has more information. - OFS.Image's index_html method now calls its own ZCacheable_set method with the single argument None. Existing cache managers such as HTTPAcceleratedCacheManager and RAMCacheManager will do nothing with this value, but other cache managers are free to inspect the caller and cache its data. - OFS.Image's index_html method now attempts to retrieve data from a cache manager via its own ZCacheable_get method. Previously, this method did call in to ZCacheable_get, but did nothing with the data that it retrieved (under the assumption that there are no cache manager implementations which could handle large file data). Existing cache managers will return a null value from ZCacheable_get, but future cache managers will be free to return a stream iterator, which can be returned directly to the publisher. For a sample implementation of such a cache manager, see cvs.zope.org:/Products/FileCacheManager. - OFS.Image's manage_FTPget now attempts to get data from a cache manager before sending back data out of the database. - New restructured text implementation with an updated docutils package v0.3.4. - New zope.conf directive called rest-header-level with a default value of 3. It's setting the initial header level for rest like structured-text-header-level for stx. - Collector #1233: Introducing a new environment variable ZOPE_CONFIG has can be used to specify the location of the zope.conf file for scripts importing the 'Zope' module directly. In earlier versions it has been necessary to call Zope.configure explict. Example:: export ZOPE_CONFIG=$INSTANCE_HOME/etc/zope.conf From Python script: import Zope # Zope.configure('etc/zope.conf') <-- no longer needed app = Zope.app() - The testrunner.py has been removed since it is replaced with the test.py script. Bugs Fixed - Collector #789: Zope's transaction behavior flawed. Historically, if an error bubbled up to Zope's publishing machinery, execution of the error handler code would be performed in a context that was essentially "between" two transactions. This caused problems for applications (like sessioning) which want to be able to write to the database during an error message. That need may sound insane, but since the transaction is aborted after the error handler has executed, it has the same effect as the "old" behavior inasmuch as no inconsistent state will ever be committed to the database as a result of this behavior unless someone really wants to shoot themselves by calling "transaction.commit()" during error handling code. - TransientObjects (session data objects) may have lost state at ZODB commit time after they were mutated via their __setitem__, __delitem__, update, or clear methods because they did not signal to the ZODB persistence machinery that they had been modified. - Collector #852: Bug in the error reporting of some failure cases in transactions meant that the cause of the failure was not logged. - Collector #1345: AcceleratedHTTPCacheManager now sends the Last-Modified header. - Fixed inconsistency between the way MultiHook and traverse() call traversal hooks. - Collector #1304: zopectl doesn't allow specification of zdrun effective user. zopectl now supports the -u switch which tells it to setuid to the specified username/uid when running the daemon manager. - Collector #1341: TemporaryStorage was not usable under a ZEO server. - Collector #777: 'HTTPRequest.__str__' was perfectly happy to display the values of password fields; the issue notes their presence in the error log, but '' or '

' had the same effect. - Fixed bug in catalog merging which could cause a KeyError when merging results sorted by an index with few keys. - Collector #1126: ZPublisher.Converters.field2lines now using splitlines() instead of split('\n'). - Collector #1322: fixed HTML quoting problem with ZSQL methods in DA.py - TemporaryStorage bug fixed where KeyErrors could be reported coming from TemporaryStorage.load (most often reported seen during sessioning usage). - Collector #1261: fspack.py neglected to import the oid_repr() function, but used it to produce error output. - Collector #1124: The ZReST product now uses the same reST encoding parameters from zope.conf as the low-level reStructuredText implementation. - Collector #1259: removed the "uninstall" target from the Makefile since the uninstall routine could also remove non-Zope files. Because this was to dangerous it has been removed completely. - Collector #1299: Fixed bug in sequence.sort() - Collector #1309: The reference counts reported by DB.cacheExtremeDetails() for ghosts were one too small. - Collector #1159: Added test for __MACH__ to initgroups.c so the initgroups method becomes available on Mac OS X. - Collector #1226: ZTUtils.Tree encoding could include carriage returns, which would result in a malformed cookie HTTP header. - Collector #1222: Fixed broken API help by adding missing security declarations in APIHelpTopic.py. - Collector #1004: text,token properties were missing in PropertyManager management page. - Collector #553: sporadic Zope crashes BTree bucket conflict resolution could segfault if fed three empty buckets. A BTree had to start out empty, and two overlapping transactions had to each add something to the tree and delete it again before committing, so this was unlikely and hard to reproduce. - Collector #1298, ZSQLMethods now have a __traceback_supplement__ so they are not rendered in the site error log. - Collector #562: pDocumentTemplate is now disconnected because it has been broken for some time. It will no longer get imported if cDocumentTemplate isn't there. It is left only as an implementation reference for the C version. - Collector #1300, Fixed persistence of dtml-tree state which was caused by an overly cautious unpickler. - Collector #544: Remove Content-Length header for 304 responses from images/files which violated RFC 2616. This was a workaround for a since fixed Apache proxy-cache bug. - Display index name on error message when index can't be used as 'sort_on'. - PUT would fail if the created object had a __len__ = 0 (eg: BTreeFolder2) and fallback to _default_put_factory. Fix by checking if the returned object is None instead. - Fix performance bug when calculating the number of objects in a large ZCatalog. This made clicking on the indexes tab of such a catalog extremely expensive. - Collector #1289: Allow ZSQL methods to be edited via WebDAV. - Collector #1283: DTML Method "standard_error_message" with uppercase html tag - WebDAV property values were not being properly escaped on 'propstat'. - WebDAV 'supportedlock' was not checking if the object did implement the WriteLockInterface before returning it's value. - FTP download speed was very slow because the buffer size used for the feeding of data into asyncore was very small. Increasing it to a "normal" amount sped up FTP downloads by ~ 100X. - OFS.Image's insanely long index_html method was factored out into several parts. - ZCatalog result/brain methods getPath() and getObject() now properly propagate database conflict errors which should eliminate spurious missing results on busy servers. - Collector #1160: HTTPResponse.expireCookie() potentially didn't when an 'expires' keyword argument was passed. - reStructuredText ignored the encoding parameters in zope.conf - ObjectManager no longer raises string exceptions. - Collector #1260: Testing/__init__.py no longer changes the INSTANCE_HOME. - App.config.setConfiguration() did not update the legacy source for debug_mode, Globals.DevelopmentMode. - Collector #1255: getWrappedOwner() must return None if the object is owned by the UnownableOwner. - Collector 434: meta-refresh on web-based restart was set too low, and would often try to hit the server too soon on slower machines, displaying a 500 error, which caused people to panic. It was 5 seconds, it's now 10. - Minor usability tweaks * Made selection widgets of FindSupport forms 8 lines high. - Stop testrunner.py from recursing into the 'build-base' directory created by setup.py. - Pass along command line args to scripts called via "zopectl run". This allows you to use scripts which require arguments using zopectl run. If you do "zopectl run scriptname.py arg1 arg2", the value of sys.argv within the script will consist of ['scriptname.py', 'arg1', 'arg2']. - The security-policy-implementation directive had no effect. It was not possible to switch from the C implementation. - DeprecationWarnings in Scripts were converted into mysterious exceptions. Scripts now gain a '_filepath' attribute, used at runtime as the '__file__' global value. - zopectl under daemon mode on MacOS X 10.3, Solaris, Linux 2.6 kernels emitted a traceback upon startup. This has been fixed. See http://zope.org/Collectors/Zope/1235 for more info. - mkzeoinst.py complained if there was already a process listening on the port that the new instance was told to listen on, and would not install instance files until the process was shut down. This has been fixed. - ZPublisher/Test.py publish() function now conveys publisher-specific environment variables on when explicitly specified in the -e env variable. This change should be very backwards compatable, since it was previously futile to provide these values. Also, publish_module() now notices when it gets a mapping-like argument named 'form' and puts it on the request - a handy way to get form data to the publisher. This change may be suprising if you're used to having your forms ignored. I think these Zope.debug() enhancments are pretty safe, since up to now it's been a tool good for not much more than poking and prodding. Now it should be more useful for more thorough testing purposes. - ObjectManager will now attempt to set Owner local role keyed to the user's id, rather than username. Zope 2.7.0 (2004/02/11) (no changes from rc2) Zope 2.7.0 rc 2 (2004/01/30) Bugs Fixed - One of the 2.6.3 fixes added a previously missing security check when binding 'context' and 'container' to Python Scripts. Because many existing scripts appear in containers that users cannot access, this caused a lot of unauthorized errors in existing (mostly CMF) sites, since 'container' is bound by default. The fix has been adjusted so that an unauthorized is only raised if the bound name is actually used in a script, making backward compatibility much better. - Collector #1154 / # 615: interaction with sessions could cause the security context to be discarded, potentially breaking scripts that depend on proxy roles. - Stopped overriding the version of xmlrpclib in the Python standard library. As of Python 2.3, the standard xmlrpclib now includes all the features Zope needs, plus bugfixes and integration with new Python types. - A pre-existing bug that could cause subtly different results when calling the C vs. Python version of validate() through the authorize() method of a UserFolder was found and fixed. - An inadvertant change to the behavior of Owned.getOwner was reverted. - Fix for bug 1207: errors building in read-only directories. Zope 2.7.0 rc1 (2004/01/19) Bugs Fixed - Collector #1182: Functions for handling decisions about unprotected subobjects were not passed "names" when doing unnamed (item) access. In 2.6.3 we changed access checks when doing item access to pass None rather than the key value when validating access. This broke some existing applications. We have reverted these changes. - Collector #1186: some globals required to support certain operations in restricted code (e.g., list comprehensions in DTML expressions) were not propagated to all the right places. - Collector #1074: Change Scripts' __name__ to None, added unit tests for the effect of __name__ on class definitions and imports. - README.txt (and other file-like objects or directories without an __init__.py) on the Products path are no longer considered to be Products. A warning message is no longer raised about duplicate README.txt "products" upon startup in a default installation. - XMLRPC queries failed due to a missing import. - Forward-ported Toby's unicode encoding hacks for propertysheets from the 2.6 branch. - Some potential refcount issues in cAccessControl.c were fixed. - Fixed bug in ZEO server's monitor option. The -m / --monitor option used to specify the monitor address did not work. Zope 2.7.0b4 (2004/01/09) Default config file changes - Zope no longer creates "default" ZODB databases if none exist in the zope config file. At least one database (the root database, at mount-point /) must be specified in zope.conf for Zope to start properly now. In zope.conf files generated by older 2.7 betas, just uncomment the "main" and "temporary" zodb_db definitions in the zope.conf file to be in parity with what would have been generated in 2.7b4 and beyond. - Zope no longer configures a "default" set of servers if no servers are specified in the config file. If no servers are specified in the config file, no servers are started. Features added - Folder listings in FTP now include "." as well as "..". - Using "_usage" parameters in a ZCatalog query is deprecated and logged as DeprecationWarning. - Added a "mime-types" configuration value which names a file giving additional MIME type to filename extension mappings. The "mime-types" setting may be given more than once in the configuration file; the files have the same format at the mime.types file distributed with Apache. - Added a "warnfilter" config directive which allows for the specification of Python warning filters from within the Zope configuration file. - When a VHM is activated, it adds the mapping 'VIRTUAL_URL_PARTS': (SERVER_URL, BASEPATH1, virtual_url_path) to the request's 'other' dictionary. If BASEPATH1 is empty, it is omitted from the tuple. The joined parts are also added under the key 'VIRTUAL_URL'. Since the parts are evaluated before traversal continues, they will not reflect modifications to the path during traversal or by the addition of a default method such as 'index_html'. - Added VHM regression tests to the SiteAccess product. Also see Collector #809. - Updated the interactive Zope tutorial to use ZPT in place of DTML. - When installing, symlink the Python used to build Zope into the installed 'bin' directory (unless it has a 'python' already). - Python 2.3 builtins 'sum', 'dict', and 'enumerate' exposed to guarded code. Bugs fixed - Collector #1140: setting the access control implementation from the configuration file didn't work. The ZOPE_SECURITY_POLICY environment variable is no longer honored. - Browsers that do not escape html in query strings such as Internet Explorer 5.5 could potentially send a script tag in a query string to the ZSearch interface for cross-site scripting. - FilteredSets (used within TopicIndex) are defined via an expression, which was naievely eval'ed. - The ZTUtils SimpleTree decompressed tree state data from the request without checking for final size, which could allow for certain types of DoS attacks. - Inadequate security assertions on administrative "find" methods could potentially be abused. - Some improper security assertions on DTMLDocument objects could potentially allow access to members that should be protected. - Class security was not properly intialized for PythonScripts, potentially allowing access to variables that should be protected. It turned out that most of the security assertions were in fact activated as a side effect of other code, but this fix is still appropriate to ensure that all security declarations are properly applied. - The dtml-tree tag used an "eval" of user-supplied data; its efforts to prevent abuse were ineffective. - XML-RPC marshalling of class instances used the instance __dict__ to marshal the object, and could include attributes prefixed with an underscore name. These attributes are considered private in Zope and should generally not be disclosed. - Some property types were stored in a mutable data type (list) which could potentially allow untrusted code to effect changes on those properties without going through appropriate security checks in particular scenarios. - Inadequate type checking could allow unicode values passed to RESPONSE.write() to be passed into deeper layers of asyncore, where an exception would eventually be generated at a level that would cause the Zserver main loop to terminate. - The variables bound to page templates and Python scripts such as "context" and "container" were not checked adequately, allowing a script to potentially access those objects without ensuring the necessary permissions on the part of the executing user. - Iteration over sequences could in some cases fail to check access to an object obtained from the sequence. Subsequent checks (such as for attributes access) of such an object would still be performed, but it should not have been possible to obtain the object in the first place. - List and dictionary instance methods such as the get method of dictionary objects were not security aware and could return an object without checking access to that object. Subsequent checks (such as for attributes access) of such an object would still be performed, but it should not have been possible to obtain the object in the first place. - Use of 'import as. in Python scripts could potentially rebind names in ways that could be used to avoid appropriate security checks. - A number of newer built-ins (min, max, enumerate, iter, sum) were either unavailable in untrusted code or did not perform adequate security checking. - Unpacking via function calls, variable assignment, exception variables and other contexts did not perform adequate security checks, potentially allowing access to objects that should have been protected. - DTMLMethods with proxy rights could incorrectly transfer those rights via acquisition when traversing to a parent object. - Range searches with KeywordIndexes did not work with record-style query parameters - Collector #484: Depending on the server's timezone (east of Greenwich) and os (Windows) _calcTimezoneName didn't work with 1970/01/01 and some other special dates. - Item_w__name__ now has a working getId() method - PageTemplateFile now using Item_w__name__ mixin, fixing its getId() and absolute_url() methods. - PCGI, FCGI, and HTTP servers now check for unicode types in their close methods. - Only one VirtualHostMonster is allowed per container. - Collector #1133: TreeTag choked on Ids of type long. - Collector #1012: A carefully crafted compressed tree state could violate size limit. Limit is no longer hardcoded. - Collector #1139: tal:attributes didn't escape double quotes. - Collector #809: Added and documented Traversable.py methods absolute_url_path and virtual_url_path, and reverted earlier change to absolute_url behaviour. - Collector #927: Raise a more specific exception than BadRequest when disconnected. - Collector #1129: Improper parsing of ISO8601 in DateTime. - updated docutils packages (fixes several problems with RestructuredText) - Fixed a bug in the ZEO server's getInvalidations() method. If it had the requested invalidations, it would send all the invalidations in its queue rather than just the requested ones. This didn't affect correctness of the cache, but did cause it to throw out valid data. Zope 2.7.0b3 (2003/11/18) Features added - Add 'parity' method to ZTUtils Iterators. - Allow untrusted code to mutate ZPublisher record objects. - Changed the ZEO server and control process to work with a single configuration file; this is now the default way to configure these processes. (It's still possible to use separate configuration files.) The ZEO configuration file can now include a "runner" section used by the control process and ignored by the ZEO server process itself. If present, the control process can use the same configuration file. - ZConfig was updated to version 2.0. The new version includes two new ways to perform schema extension; of particular interest in Zope is the ability for a configuration file to "import" new schema components to allow 3rd-party components (such as storages, databases, or logging handlers) to be used. - Add an update_metadata keyword argument to instances of a Catalog's catalogObject method and an ZCatalog's catalog_object method. This argument defaults to true. If this argument is specified false, the catalog update will not update the metadata for the object. If the obejct has never been cataloged before, the metadata is always added and the flag has no effect. - The standard ProperyManager now includes the same label functionality as the CMF SimpleItemWithProperties object. By adding a 'label' entry to the property definition you can now display a prettier label for the property. Bugs fixed - Removed all cases where string literals were raised as exceptions, and where named strings caused deprecation messages during unit tests (some named strings may still be hiding in the woodwork). - AccessControl.User used a misleading string exeception, 'NotImplemented', which shadowed the Python builtin. - Collector #1112: logfile reopening didn't work. - Collector #1110: Under Python 2.3, some DateIndex tests were failing. - Collector #426: Inconsistent, undocumented error() method. - Collector #799: Eliminate improper uses of SCRIPT_NAME. - Collector #445: Add internal global declaration for Script bindings. - Collector #616: Make CONTEXTS available to TALES Python expressions. - Collector #1074: Give Script execution context a __name__ - Collector #1095: Allow TAL paths starting with '/varname' as a preferred spelling for 'CONTEXTS/varname'. - Collector #391: Cut and paste now requires delete permissions. - Collector #331: Referenses to URL in manage_tabs was changed to REQUEST.URL to prevent accidental overriding. - Made the control panel properly reflect the cache-size setting of ZODB's object cache once again. - ConflictError was swallowed in ObjectManager by manage_beforeDelete and _delObject. This could break code expecting to do cleanups before deletion. - Python 2.3 BooleanType wasn't handled properly by ZTUtils marshalling and ZPublisher's converters. - Collector #1065: bin/ scripts didn't export HOME envars. - Collector #1034: METAL macros with content-type text/xml raised Unauthorized errors inappropriately. - Collector #572: WebDAV GET protected by 'FTP Access' permission. Two new methods have been added to WebDAV resources, "manage_DAVget" and "listDAVObjects". These are now used by WebDAV instead of the earlier "manage_FTPget" and "objectValues". This separates the permissions, and allows WebDAV specific overriding of these methods. - Workaround for Collector #1081: The 'title' property for objects derived from OFS.Folder or PropertyManager can now be removed and replaced with a ustring property. This allows the usage of non-ISO-8859-1 or ASCII charsets - profile-publisher-file config file directive now works properly (thanks to Andy McKay for the report). - Collector #904: Platform specific signals in zdaemon/Daemon.py (fixed by removing the "fossil" module from 2.7 branch and head). - Collector #951: DateTime(None) now equal to DateTime() - Collector #1056: aq_acquire() ignored the default argument - Collector #1087: ZPT: "repeat/item/length" did not work as documented in the Zope Book. - Collector #1079: Fixed BaseRequest's traversal loop to avoid some infinite loop scenarios. - The View tab in the ZMI in some cases pointed back to manage_workspace. - Added the name "context" as a synonym for "here" in page templates, since "context" is better understood. This unifies the convention in templates, scripts, and Zope 3. - Disassociate zeo-client-name from enable-product-installation ZConfig keys. In the past, if you ran an appserver as ZEO client with a persistent cache, the startup logic prevented products from being loaded. This was a hack that got baked in via the weird interaction between the FORCE_PRODUCT_LOAD and ZEO_CLIENT environment variables and was carried over into Zope 2.7 by inertia. Now they have nothing to do with each other and can be specified independently. - Monitor server would start even if an emergency user didn't exist. - Collector #721: Entities in tal:attribute values weren't properly escaped. - Collector #851: Traversable.py: A bare try..except shadowed conflict errors - Collector #1058: Several fixes for PropertySheets when used outside ZClasses - Collector #1053: parseIndexRequest turned empty sequence of search terms into unrestricted search. - manage_tabs had a namespace problem with the acquisition of names from the manage_options variable resulting to acquire "target" and "action" from objects above in the hierachy. - PathIndex and TopicIndex are now using a counter for the number of indexed objects instead of using a very expensive calculation based on the keys of their indexes. - Collector #1039: Whitespace problem in Z2.log fixed - Collector #78: Proxy roles not respected when calling manage_pasteObjects. - changed some bare try: except:'s in Shared.DC.ZRDB.Connection so that they now log exceptions that occur. - Responses from Zope's FTP server could sometimes be HTML! Zope 2.7.0b2 (2003/08/26) Features added - New zopectl feature: adduser. Instead of using the inituser mechanism, "zopectl adduser " will add a user with the 'Manager' role to your site. This is useful both interactively and for packagers. - New top-level package: nt_svcutils. This takes the place of the Zope.Startup.nt package to allow for reuse outside of Zope. - Renamed the "service file" for a Zope instance under Windows from ntservice.py to zopeservice.py to allow a ZEO server instance to share the same instance home as a Zope instance (the ZEO service file will be called zeoservice.py). - The TemporaryStorage module was moved from Products/TemporaryFolder to a new top-level package named tempstorage. - Split functionality of mkzopeinstance into two separate modules: mkzopeinstance.py and copyzopeskel.py. copyzopeskel.py can be used directly by packagers to install custom skeleton directories or mkzopeinstance can use a custom skeleton directory via the '--skelsrc' flag. - Added 'environment' and 'path' directives to config file directive list. 'environment' is a section which allows the user to specify environment variables within the config file. 'path' allows the user to specify additional directories to insert into sys.path when configuration is processed. - Moved temporary storage related files into their own module within lib/python, out of Products.TemporaryStorage. This was necessary in order to prevent the configuration machinery from prematurely importing some modules that it didn't need to during startup (the transitive set of all imported modules within Products.TemporaryStorage). - Add a check to the configurator to ensure that the Python found has an expat parser. Bugs fixed - The Products directory in an instance home was not automatically added to the Products.__path__ (it needed to be specified manually). It is now automatically added at instance startup. - The Windows runzope.bat file generated by mkzopeinstance from the default skeleton directory now uses python.exe instead of pythonw.exe, so you can actually see output from the Zope process when you run it interactively. - backport of TAL fixes from z3 * i18n and metal interactions * fix handling of nested translations with tal:content/replace and i18n:name - Collector #1017: reST has been broken - backward compatibility for i18n:attributes if items are separated with spaces and there are more than two items, all are treated as attributes plus deprecation warning if items are separated with spaces and there are two items : * i18n:attributes="value msg_id;" (with semicolon) will always be treated as * i18n:attributes="value title" will be treated as as long as title exist as attribute (both as static or in tal:attributes) plus deprecation warning - deny attributes being both part of tal:attributes and having a messageid in i18n:attributes - Fixed failing Zope.Startup tests caused by import of nonexistent Startup.getSchema function. - VHM: manage_edit has been broken - The container-class keyword to database sections didn't work. - ZConfig didn't report a line number and munged capitalization of replacement keys when a replacement error was provided. - log-to-stderr on startup will now emit messages to the console at the lowest logging level defined by any of the handlers in the eventlog section. - entirely removed warning when the starting user's umask is "too permissive". it wasn't clear that it added any value under normal operations. - debug-mode config file option did not work. - ZClasses that subclassed ObjectManager that were created in earlier versions of Zope would not load under 2.6, due to the new Interfaces package. Added back a simple stub module and changed a constructor to allow these ZClasses to work. - Bugfix: if zopectl is run as the root user, the debug, run, and adduser commands will cause the Python process which performs those actions to switch users to the effective user. This prevents ZODB index files, log files, etc. from being written as root, potentially preventing later startup by the effective user. - Bugfix: allow the zopectl process to ascertain the program that zdrun should run from the Zope config file. This is necessary if the file is moved out of a "single-directory-as-instance" instance home. - Various fixes to content of ZopeTutorial to deal with URLs moving on new.zope.org (thanks to Stefane Fermigier). - Added optparse and textwrap modules to docutils package to prevent ReST barfage (AJ). - Added a sample zeoclient and tempstorage stanzas to the default config file (zope.conf.in). - Various tweaks to the dbtab configuration code (Shane). - Various fixes/enhancements to ZODB/ZEO (PythonLabs). - Removed 'inst/Zope.spec' Zope RPM spec file in favor of placing it in a separate internal ZC repository. It will be available in the SRPM. - Fix for Collector 823 (XML RPC exception values were always converted to strings and thus turned into a Fault object). Thanks to Sandor Palfy for the patch. - Remove --zeo/-z flags from mkzopeinstance.py, as custom_zodb.py is no longer the preferred method of configuring custom storages now that we have DBTab in the core, and you should be able to create a custom skeleton dir with the right zope.conf.in that has a zeo client storage set up if you need to. - 'make sdist' now creates a file with an extension of '.tgz' rather than .'.tar.gz' (mainly to meet the expectations of Zope.org about file names). - moved 'zpasswd.py' into utilities directory in source tree. Zope 2.7.0b1 (2003/07/21) Features added - zopectl command now has 'debug' and 'run' options. - the zodb_db section in zope.conf now accepts a 'connection-class' key that accepts a python dotted-path-name to use as the connection class for the database. - The ReST input and output encodings are now configured via the 'rest-input-encoding' and 'rest-output-encoding' config file directives rather than the REST_INPUT_ENCODING and REST_OUTPUT_ENCODING environment variables. - Datetime-format settings are now configured via the 'datetime-format' configuration file directive rather than the DATETIME_FORMAT environment variable. - Trusted proxies are now configured via the 'trusted-proxy' configuration file directive rather than the ZOPE_TRUSTED_PROXIES environment variable. - The maximum number of sockets that ZServer will open in order to service incoming connections can now be specified via the max-listen-sockets conf file parameter. - Shane Hathaway's DBTab product has been integrated. The Control_Panel -> Databases tab now allows you to control all mounted databases. You can add a mount point by editing the zope.conf file, adding a zodb_db section for the main database and other mounted databases, and choosing "ZODB Mount Point" from the Zope "Add" list. - DTML Methods and Documents supply a traceback supplement when called. - Windows installer contains NT/2K/XP service support on a per-instance-home basis. - zopectl command now has 'debug' and 'run' options. - the zodb_db section in zope.conf now accepts a 'connection-class' key that accepts a python dotted-path-name to use as the connection class for the database. Bugs Fixed - mkzopeinstance did not expand tildes in directory name input. - The 'configure' script did not work under Solaris sh. - The SiteErrorLog object did not ignore NotFound errors, which caused (mainly) useless messages to be kept in the log. - The addition of a linefeed to version.txt in lib/python caused Apache proxies to choke because the string was injected into a header including the linefeed. This has been fixed. - Collector: #964: standard_error_message refers to looking into the HTML code for more information which is deprecated. Referring to the error log now. - Collector #893: Mailhost: munge_header has been broken for addresses containing the recipients full name - Zope's setup.py didn't include the ZEO.auth package. - Collector #628: Applied patch to fix several textarea resize problems. - Collector #953: fixed namespace collision with form_title in ZMI - Collector #342: Avoiding insertion of a BASE tag for file objects with content-type text/html - Windows installer properly deletes pyc/pyo files on uninstall. - Windows binary post-install quickstart page is now more informative. - Fixed a potential bug in ZTUtils.Tree.decodeExpansion where a potentially empty string was tested for it's first character; used .startswith for safety. Zope 2.7.0a1 (2003/07/03) Features added - OFS: OrderSupport and OrderedFolder added. OrderSupport is a mixin class that adds the IOrderedContainer interface to ObjectManagers. OrderedFolder - meta_type 'Folder (Ordered)' - is a new Folder class using OrderSupport. (thanks to Stephan Richter for inspiration and some code) - Folder: Constructor now takes an optional 'id' argument. - Show known medusa servers in Control_Panel. - New startup and installation regime (./configure; make; make install instances use a config file, daemon is now a separate process). See http://dev.zope.org/Wikis/DevSite/Proposals/InstallationAndConfiguration for more information. - New module: App.config. New API for getting a configuration object. This should be the preferred way for all code in Zope to get configured values for many settings. For settings made available via this module, alternate locations are deprecated, though will to be supported for Zope 2.7. - Collector #435: Support for passwords encoded using MySQL's PASSWORD() function add to lib/python/AccessControl/AuthEncoding.py. - Collector #167: Support __getattr__ on cAccessControl PermissionRole objects to allow gathering of permission names for products like DocFinder and VerboseSecurity. - Added a new REQUEST method, getClientAddr(), to determine a clients IP address. Address restrictions in the user folder have been changed to use this method. By default the value returned by this method is the same as the initial value of REQUEST['REMOTE_ADDR']. Zope also has support for obtaining a more useful value from other headers if a front-end proxy is in use. See doc/ENVIRONMENT.txt for details. - DateTime module: added support to parse international dateformats. The Datetime constructor has a new "datefmt" parameter to enforce the parsing of a date as "us" or "international" date. The new field descriptor field descriptor "date_international" can be used to enforce this behaviour inside the ZPublisher. See also doc/ENVIRONMENT.txt to check with the DATETIME_FORMAT - KeywordIndex, FieldIndex and ZCTextIndex are now able to index more than one attribute of an object. This removes the ties between the indexes ID and the attribute name to be indexed. - Integration of reStructuredText (reST) and the ZReST product by Richard Jones. See doc/RESTRUCTUREDTEXT.txt for details. - Objects locked through WebDAV are now marked with a lock icon inside the ZMI. - Collector #741: Applied patch to provide better FTP error messages. - Made all PluginIndexes and ZCTextIndex use 'safe_callable', which is aware of extension classes that fill 'tp_callable' but don't define '__call__'. - Made KeywordIndex be more robust about receiving a value that is not a string or an iterable type. Bugs Fixed - Collector #954: clear() method of TopicIndex removed all filter sets instead of clearing them. - Collector #939: Fixed typo in TopicIndexes - Collector #937: UnicodeError exception available within PythonScripts - Collector #902: recursive Scripts were broken due to shared globals. - Product initialization would only consult a file named "version.txt" to read version information. Now it will check version.txt, VERSION.txt and VERSION.TXT. - Collector #928: DateIndex ignored timezones when indexing and querying - Collector #892: misleading error msg when initializing an OIBTree from a dict with a float value. The message claimed that the dict's items didn't consist of 2-element tuples, but of course they do. The TypeError now says "expected integer value". - Collector #882: delPropertySheet was broken due to a typo - Collector #683: WeDAV request without XML preamble were not recognized by Zope and raised an exception. - Index constructors were called without acquisition context of the calling object. This caused caller.getPhysicalPath() to fail for indexes depending on this information. - Collector #814: PathIndexes now uses IITreeSet instead of IISet. This should result in less memory usage. - Fixed a whitespace problem in Z2.log (AWStats failed to process Zope logfiles) - Collector #816: dtml-sendmail corrupted header if message was left blank. - Collector #790: WebDAV access is granted by default only to Managers and Authenticated users instead to Anonymous users. This restricts access to the Control_Panel and especially to the Products management to trusted users. - Rename, Cut & Delete operations on locked objects (WebDAV) are no longer permitted and will raise an exception. Copies of locked objects are copied without lock. - Collector #634: Image objects can now be rendered without border attribute by calling "image.tag(border='')". - Collector #695: Object IDs "." and ".." are no longer permitted. - Collector #771: ZCatalog failed to index DTML Document if the name of a catalog metadata was identical with the name of an acquired object. - Collector #686: intSets no longer lose their values. - Collector #685: Improved documentation explaining how, where and why security assertions should be placed in: lib/python/Products/PythonScripts/README.txt lib/python/Products/PythonScripts/module_access_examples.py - The ZEO unit tests and wo_pcgi.py didn't run on Windows if the path to the python executable included a space. - Some calls to os.system('chmod') has been replaced with the more portable os.chmod() call, to make install work properly on Windows. - Fixed an isinstance() check in SimpleItem on standard_error_message handling that would always break if the first argument was not an extension class because the second argument was a type. - Any write request could be tricked into writing into a version be setting a version cookie or by including a version name in the request. Now we require the user to globally have permission to join or leave versions to run a request in a version. Backward incompatabilities - We no longer honor local security settings that would allow someone to join or leave versions unless the location of the settings is a folder directly or indirectly containing the user's user folder. Bugs Fixed - Collector #956: automatically installing Examples at startup could be a security risk. Examples now must be installed explicitly by the user (thanks to Jamie Heilman and day0). - Collector #954: clear() method of TopicIndex removed all filter sets instead of clearing them. - Collector #882: Fixed typo in PropertySheets - Collector #939: Fixed typo in TopicIndexes - Collector #937: UnicodeError exception available within PythonScripts - Collector #902: recursive Scripts were broken due to shared globals. - Product initialization would only consult a file named "version.txt" to read version information. Now it will check version.txt, VERSION.txt and VERSION.TXT. - Make ZCTextIndex much less prone to generating conflict errors. Previously *any* concurrent updates would provoke a conflict. - Fix query performance and scalability bug in ZCTextIndex. - Collector #928: DateIndex ignored timezones when indexing and querying - Any write request could be tricked into writing into a version be setting a version cookie or by including a version name in the request. Now we require the user to globally have permission to join or leave versions to run a request in a version. - Fixed a problem with potentially mis-acquiring 'func_code' in publisher BeforeTraverse hook. - Fix for issue 683: Image cache manager headers were not sent when an image request returned a 304 (in response to an if-mod-since request). - Made all PluginIndexes and ZCTextIndex use 'safe_callable', which is aware of extension classes that fill 'tp_callable' but don't define '__call__'. - Made KeywordIndex be more robust about receiving a value that is not a string or an iterable type. - Fixed incorrect docstring in OFSP/Image help. - Fixed unhelpful signal description. Zope 2.6.2 beta 2 Bugs Fixed - TemporaryStorage (which is used by TemporaryFolder, and thus the default sessioning configuration) no longer uses a "LowConflictConnection" database connection. This fixes a bug in which data structures used for session housekeeping data could become desynchronized; the symptom for this was KeyErrors being raised from TransientObjectContainer's 'get' method. As a result, many more conflicts will be raised under high session load, but desynchronization will not occur. - Fix potential performance bug in PathIndex. - Scored result sets from catalog (i.e., from text indexes) can now be merged and sorted together across queries like unscored results. - Fixed a memory leak in TALES. If an exception propagated from a tal:repeat block, an uncollectable cycle held a reference to everything in the TALES context. - If you're running in development mode, Zope will now raise an exception if a product cannot be initialized properly instead of silently continuing. This is to prevent a debugging frenzy in which you spelunk through one more more half-initialized modules wondering why the class you wanted isn't part of the module namespace. If not in development mode, the process continues silently. - Though Python 2.2.2 is not officially supported, a potential issue was found when running under 2.2.2. Some built-in types gained docstrings in the 2.2.2 release, making them publishable where they weren't publishable before. A fix has been added to the publisher to ensure that the types of objects that are publishable do not change between 2.1 and 2.2. Zope 2.6.1 beta 2 Features added - DateTime objects now have a tzoffset() method that returns the objects timezones offset from GMT in seconds. Bugs Fixed - Collector #740: DateTime now handles positive numerical timezones correcly. - Collector #763: There was no error when you had a sendmail-tag without specifying a mailhost or smpthost. Also added a missing import. - Work around potential BTrees key enumeration bugs in Transience package by checking explicitly for error cases. - Collector #736: ZPublisher now allows marshalling tags to contain a '-'. This is the first step towards a fix for Collector #737 - Collector #714: CopySupport's manage_clone now calls manage_afterClone in the saem way that manage_pasteObjects does. - Collector #697: Multiple selection properties were incorrectly marshalled. note than any non-ascii multiple selection properties modified in versions without this fix will have been corrupted in the zodb. - Collector #256: Added a check in _doChangeUser to make sure passwords isn't encrypted twice. - Added a sortKey() method to Shared.DC.ZRDB.TM to silence warnings from updated ZODB that DAs dont have that method. Zope 2.6.1 beta 1 Bugs Fixed - VirtualHostMonster handles empty Mapping paths properly. - Deadlock prevention code added. It was possible for earlier versions of ZODB to deadlock when using multiple storages. If multiple transactions committed concurrently and both transactions involved two or more shared storages, deadlock was possible. This problem has been fixed by introducing a sortKey() method to the transaction and storage APIs that is used to define an ordering on transaction participants. This solution will prevent deadlocks provided that all transaction participants that use locks define a valid sortKey() method. A warning is raised if a participant does not define sortKey(). For backwards compatibility, BaseStorage provides a sortKey() that uses __name__. - Fixed bug in FileStorage related to object uncreation. An attempt to load an uncreated object now raises KeyError. - Fixed a couple bugs in FileStorage recover() that wrote incorrect backpointers. - Fixed data_txn attribute of iterator data records to use the transaction id of the previous transaction, even if it also has a data_txn field. - Fixed conflict resolution bug that raised a NameError when a class involved in a conflict could not be loaded. - Fixed C extensions that included standard header files before Python.h, which is not allowed. - Added code to ThreadedAsync/LoopCallback.py to work around a bug in asyncore.py: a handled signal can cause unwanted reads to happen. - Collector #651: WebDAV Lock Manager was broken. - Collector #646: metal:slot was lost during the I18n merge. - Collector #640: Fix security assertion on ZCTextIndex query method. - Delayed opening the ZODB until after the "Zope" module has been imported, fixing a deadlock issue involving ZEO. The "Zope" module now has a "startup()" function. - Fixed a NameError in the recent change to DateTime.rfc822(). - Made DateTime.rfc822() simpler and independent of local timezone. - Fixed bug in Transience reported by kedai which caused spurious KeyErrors under heavy sessioning usage. - Fixed bug in the Interface Verify package; base interfaces were not included in an interface compliancy test. - Collector #650: Fixed implicit list marshalling for lists where the first two values are tainted. - Collector #671: HTTP Ranges were broken for files and images whose length wasn't exactly divisible by 2**16. - ModuleSecurityInfo declarations could be lost if further declarations were made after the Info object already had been applied. Such additional declarations could take place in Python trusted code run after Zope strartup or during a Product refresh. - Collector #699: MailHosts created in 2.5 breaks in 2.6. - Collector #694: dtml-sendmail mailto specification replaces "To:" header. - Collector #702: DateTime.rfc822() fails without daylight saving - Collector #703: KeyErrors raised when unindexing a PathIndex (and TopicIndexes) should be swallowed and logged. Zope 2.6.0 Bugs Fixed - Caused many places throughout the code base to use calls to user.getId() rather than user.getUserName(). With most (all?) user folder implementations today, this will have no behavioral change, as getId is always alised to getUserName. However, this makes it possible to write user folder implementations which make the distinction between the user's id and the user's name. These user folders will allow users to change names independent of their identity. - WebDAV Lock Manager actually gives the user a chance to specify a starting path **before** searching for locks, shortening query times and memory usage in large Zope instances. - PageTemplateFiles were previously owned by whatever object contained them. This resulted in very hard bugs if the user who owned the container was removed. Since PageTemplateFiles come from the filesystem, they are now "unowned", similar to DTMLFiles. Security is still applied, but now it is applied correctly. - Collector #411: DateTime.rfc822 is not rfc822 compliant Zope 2.6.0 beta 2 Bugs Fixed - The ability to add multiple select properties to PropertyManagers was broken (issue 612). - Removed the signal handler hung off USR1 for packing the database. This feature proved dangerous as the pack operation would happen in the main thread, causing all asyncore operations to stop until it was finished. - Collector #372: tal:attributes failed when combined with tal:replace. - Don't try to close network connections in the signal handler for shutdown. This hosed ZEO clients. - Collector #292: PythonScript.write() didn't properly refresh bindings. - Dumb bug in zdaemon fixed in which it would try to kill process numbers 1, 2, 3, 10, 12, and 15 when it caught a signal related to any of these signal numbers. Instead, it actually tries now to kill its child process with the same signal. - Write pidfiles out with trailing newlines. - Fix setVirtualRoot in the face of unicode paths (such as occur during an XML-RPC request. - Collector #539: Fixed rendering of TAL namespace tags with an 'on-error' statement. - Collector #586: Generated 'start' scripts had a nonsensical export of an "INST_HOME" environment variable. - Collector #580: TALES evaluateBoolean() was squishing 'default'. - Collector #581: TALES Path traversal should not special-case a blank string in the second element position. It now skips directly to item access when a path element is blank or has a leading '_'. - Fixed inconsistent attribute access in TALES Paths. - Deprecated hasRole alias failed to return result. - Collector #538: Hybrid path expressions no longer attempt to call a value returned by the final, non-path alternate. - Collector #573: ZTUtils Iterator didn't catch AttributeError. - Collector #517: The properties page incorrectly rendered properties with non-latin1 values if there were no unicode properties defined, and incorrectly processed properties with non-ascii names. - ZTUtils.SimpleTree could not build a tree with a root other than the ZODB root object. Also, filter functions didn't work at all, let alone in accordance with the documentation in the code. - Collector #603: ZTUtils.Tree.encodeExpansion encoded depth with '.' characters, but decodeExpansion could possibly see an encoded node id as an encoded depth when that encoded id started with a '.'. - Collector #605: ZTUtils.Tree.decodeExpansion set no limits on the string to be decoded, allowing for a DoS attack with very large strings. - The fix for issue #144 broke the ability to create an empty Image or File object. This functionality is now reenabled again. - ZTUtils.Zope.TreeSkipMixin allows you to skip unauthorized objects in the tree, but the filter wasn't applied when trying to filter candidate child nodes through a custom setChildAccess filter. - Emails sent through MailHost now automatically include a Date header if not already present, in compliance with RFC822 and RFC2822. Features Added - Add optional 'relative' argument to getURL the method in CatalogBrains. This allows it to generate site relative URLs like absolute_url can. - ZTUtils.Tree.encodeExpansion now will use zlib compression by default, allowing for a far larger number of open tree states to be encoded. decodeExpansion handles compressed expansion states automatically. - ZTUtils.Tree.TreeMaker now has additional methods for setting various flags and attributes that influence how the tree is built, making these aspects accessible to PythonScripts. - ZTUtils.Tree.TreeMaker has a new method setStateFunction, which allows you to set a callback function that can influence the state (open, closed, leaf) of each node in the tree. - Pidfile handling improved. When Zope is started under zdaemon, it no longer writes its own pidfile. Instead, it passes in the path to Z2.pid to zdaemon as its pidfile name. The 'zProcessManager.pid' file is no longer ever written. This caused a change to the -Z option of z2.py which should be mostly backwards-compatible (unless people were relying on zProcessManager.pid to be written). Now the -Z option is a boolean. -Z1 means use a daemon. -Z0 means dont. The default is -Z1. Zope 2.6.0 beta 1 Bugs Fixed - Collector #587: fixed wrong migration to string methods in DTMLMethod.py - Collector #583: Searching for '/' with PathIndexes failed. - Fixed bug in manage_editProperties which used an incorrect default for several types of property when they were not found in the REQUEST. - Collector #574: Fixed write on HEAD requests caused by overzealous ETag support. - Fixed bug in z2.py where it would eat certain socket error exceptions at startup. - Collector #550: Exceptions in XML-RPC requests no longer envoke standard_error_message. Plain text error messages are instead added to the fault string. In debug mode, a full traceback is also included since access to the error log is not a given for XML-RPC developers. - Collector #512,541: Fixed broken WebDAV compatiblity with Cadaver 0.20.X due to a missing Lock-Token header. - Zope Page Templates set a 'content-type' header even if the result of their execution was not rendered to the browser. We now check to make sure a content-type header is not already set before allowing a page template to set its own. - The title_or_id attribute of browser id managers and session data managers is now accessible publically. - Collector #510: When Python scripts and other "Script" objects were acquired during URL traversal, the __before_publishing_traverse__ code did not properly stop traversal at the script and populate traverse_subpath with the remaining url path elements. - Collector #238: Version Save and Discard buttons were too close to each other in Version management screens. - The "Add Browser ID Manager" permission was renamed to "Add Browser Id Manager". - Collector #437: dtml-sqltest now renders 'v not in (b,c)' when used as . Previously, a sqltest for inequality would render 'v <> b' when a single value was submitted, but would render 'a in (b,c)' when multiple values were present and the 'multiple' switch was set. - Collector #478: Z Search Interfaces with no parameters are now generating correct HTML. - Collector #448: Z Search Interfaces created as PageTemplates have a correct title, not a fragment of dtml. - Fixed brokenness of session data manager hasSessionData method. The old method created a session data object as a result of the call; it does not now. - Collector #458: Fixed broken reindex_all in CatalogAwareness classes. - The default "start" script now causes the event log to be sent to standard output unless the "EVENT_LOG_FILE" or "STUPID_LOG_FILE" environment variable is found in the environment. - The much-hated name "STUPID_LOG_FILE" now has a preferred alias: "EVENT_LOG_FILE". - Collector #454: The "default" session_data transient object container was not created if an object named "session_data" existed in the root. - Restored behavior of ZCatalog when arguments with empty string are passed in to searchResults. These values are now ignored. If only empty string values are passed to searchResults, then it returns all results (it is assuming what was passed is essentially an empty filter). - Collector #160: Allow TemporaryStorages to participate when a version is active. - Collector #446: Fixed management security assertions on ZCatalogIndexes class. - The BTree module functions weightedIntersection() and weightedUnion() now treat negative weights as documented. It's hard to explain what their effects were before this fix, as the sign bits were getting confused with an internal distinction between whether the result should be a set or a mapping. - New "Transience" (session data storage) implementation. More reliable under high load. - Collector #402: PythonScript recompile utility should only be usable by Manager to prevent abuse. - Collector #433: Fixed broken Splitter backwards compatiblity issue caused by code cleanup. - Collector #151: The Python 2.1 / 2.2 fcntl compatibility hacks were bypassed when using medusa directly without importing ZServer first (as when using monitor_client.py). - Collector #72: Start on Windows 95 machines with no network devices installed. - Collector #79: Don't swallow App.FindHomes exceptions. - The set operation difference(X, None) was returning None instead of returning X, contradicting the docs and common sense. difference(None, X) continues to return None. - Fix bug in ISO_8859_1 splitter which corruped storage on initialization. - Collector #421: Storage leak in cAccessControl - FileLibrary and GuestBook example applications gave anonymous users the Manager proxy role when uploading files - a potential vulnerability on production servers. - Exceptions that use untrusted information from a REQUEST object in the exception message now html-quote that information. - Stop leaking FastCGI Authorization header in environment to prevent password compromise - #178: Don't compile PythonScripts in skins directories - Fixed the help registration system and Zope tutorial to honor the environment variables, FORCE_PRODUCT_LOAD, and ZEO_CACHE, that affect whether products are installed in the database at application startup. - Collector #547: xmlrpclib SlowParser should also handle CDATA sections. - Collector #525: Don't mask Unautorized exceptions as XML-RPC faults. Fix based on patch from Brad Clements. - Collector #465: Allow XML-RPC requests with no tag. - Collector #528: Don't clear REQUEST_METHOD for XML-RPC requests; instead check for an XML-RPC Response objetc in BaseRequest.traverse. Features Added - Browser ids can now be encoded in the URL and Zope can be instructed to automatically include the browser id in its generated URLs. - Browser Id Managers now provide a saner way to obtain a hidden form element which encodes the browser id name and browser id. An interface method named "getHiddenFormField" on browser id managers now exists which returns a snippet of HTML as a hidden form field that encodes these values. - A Site Error Log object is now created in the root at Zope startup time. - Added 'url_unquote' and 'url_unquote_plus' modifiers to DTML (also fmt=url-unquote and fmt=url-unquote-plus), and made the same functions available in the PythonScripts.standard module. - Collector #186: Added urlencode to the standard importables for Python scripts. - and &dtml.-name; will now automatically HTML-quote unsafe data taken implictly from the REQUEST object. Data taken explicitly from the REQUEST object is not affected, as well as any other data not originating from REQUEST. This can be disabled (at your own risk!) by setting the environment variable ZOPE_DTML_REQUEST_AUTOQUOTE to one of 'no', '0', or 'disabled'. - ZCatalog index management ui is now integrated into ZCatalog rather than being a subobject managment screen with different tabs. - ZCTextIndexes can now be instantiated without constructing a silly "extra" record object if desired. - SimpleItem class now passes a new argument "error_log_url" to the standard_error_message template on error. If the site contains an error log object, this will contain the url to the applicable log entry for the error. - The IOBTree module also supports multiunion() now. - BTrees and TreeSets are complex objects, with parent->child pointers, sibling pointers, and multi-level parent->descendant pointers. About half the pointers are formally redundant, but speed operations. BTrees and TreeSets now support a ._check() method, which does a thorough job of examining all these pointers for consistency. It raises AssertionError if it finds any problems, else returns None. In Zope 2.5, in rare cases a key deletion could leave these internal pointers in an inconsistent state (what was supposed to be redundant information became conflicting information). The most likely symptom was that tree.keys() would yield an object that disgreed with the tree about how many keys there are. tree._check() can be used if you suspect such a problem (and if you find one, rebuilding the tree is the best solution for now). - Added support for the ZOPE_HOME environment variable, which points to the Zope root, where the ZServer package and default imports may be found. - Collector #516 -- "title" property on image tags - Collector #117 -- change External Method DTML to name="id" vs unquoted id - Collector #61 -- now manage_PasteObjects return a list of dictionaries containing {'id':original_id,'new_id':newly_pasted_obj_id} when called with REQUEST=None - Changed FORCE_PRODUCT_LOAD so that if it is set, it determines whether products are installed regardless of whether ZEO_CACHE is set. This means that you can disable product installation by setting FORCE_PRODUCT_LOAD to an empty string even if you are not using a ZEO persistent cache. Documented FORCE_PRODUCT_LOAD - xmlrpclib has been updated to the Python 2.2 version, which includes support for the Expat parser for unmarshalling data, which speeds up things considerably. - Binary builds for Linux are now built against glibc 2.1.3 with large file support enabled. - Binary builds for Solaris are now built against Solaris 8 with large file support enabled. - Added i18n support in TAL processing Zope 2.6.0 alpha 1 Features Added - The IIBTree module has a new multiunion function. It accepts a sequence of sets, treesets, etc, and returns the union of the keys of these objects, as an IISet. It's designed for peak speed when the input sequence contains many objects. - Set the default sys checkinterval to a higher value (500) to take better advantage of faster processors. Since there is no way to scientifically determine a number that works best for everyone, this at least should err on the side of better performance "out of the box" for higher-end production systems. Note that you can always use the -i argument to z2 to change the check interval. - Added support for gzip content compression for clients that support it. See lib/python/ZPublisher/HTTPResponse.py for more details. - Added ZCTextIndex plug-in index product. A replacement for TextIndex. - Removed the venerable but senile QuickStart folder from the default FileStorage. "Alas, poor Yorick! I knew him, Horatio." - Signal handling and log rotation All Zope process will respond to signals in the specified manner: SIGHUP - close open database connections and sockets, then restart the process SIGTERM - close open database connections and sockets, then shut down. SIGINT - same as SIGTERM SIGUSR2 - rotate all Zope log files (z2.log, event log, detailed log) The common idiom for doing automated logfile rotation will become: kill -USR2 `cat /path/to/var/z2.pid` The common idiom for doing "prophylactic" restarts will become: kill -HUP `cat /path/to/var/z2.pid` When a process is interrupted via ctrl-C or via a TERM signal (INT, TERM), all open database connections and sockets will be closed before the process dies. This will speed up restart time for sites that use a FileStorage as its index will be written to the filesystem before shutdown. Unspecified signals kill the process without doing cleanup. - ZCatalog no longer has a hand in managing text index vocabularies. The cruft associated with this functionality has been exorcised. No default indexes or metadata elements are defined for you when you create a new ZCatalog. Since we now have many new kinds of plug-in indexes it no longer made sense to do this anymore. - A new permission "Copy or Move" was added. This permission may be used respective to an object to prevent objects from being copyable or movable while within the management interface. The "old" behavior stipulated that users whom possessed the "View management screens" permission to an object's container could copy or move the object arbitrarily, even if they had limited access to the object itself. Once the object was moved or copied, the user became the owner of the new object, allowing them to see potentially sensitive information in the management interface for the object itself. This permission is granted to Manager and Anonymous by default, and must be revoked on an object-by-object basis if site managers intend to provide management screen access to folders which contain sensitive subobjects. This patch came as a result of Collector #376 (thanks to Chris Deckard). - Structured Text's "DocumentWithImages" class did not recognize image filenames with underscores. - The getElementsByTagName method of STDOM (used by Structured Text) would croak on most documents, especially those containing unwrapped text nodes. Fixed. - FileUpload objects now evaluate false when the have an empty file name. Making it easier to check for omitted file upload form fields. - ZClasses now use a python script as their constructor method instead of a DTML method. Also, ZClasses inherit from CatalogPathAwareness now instead of CatalogAwareness. - added browser_default hook to ZPublisher. This allows objects to specify the path to the default method that the publisher calls when the object is published. The default for objects not defining browser_default is still 'index_html' for bw compatibility. A ZMI configurable browser_default implementation has been added to ObjectManager. You can configure browser_default for OMs via a new "settings" management tab. - added TopicIndexes: a TopicIndex is a container for so-called FilteredSet. A FilteredSet consists of an expression and a set of internal ZCatalog document identifiers that represent a pre-calculated result list for performance reasons. Instead of executing the same query on a ZCatalog multiple times it is much faster to use a TopicIndex instead. - requestprofiler: added new --daysago option and added support for reading gzipped detailed logfiles - DateTime: new functions JulianDay() and week() to perform calculation of the week number based on the Julian calendar. - WebDAV: the new environment variable WEBDAV_SOURCE_PORT_CLIENTS enables retrieval of the document source for dedicated WebDAV clients (see ENVIRONMENT.txt for usage) - Collector #272: Optimizations for RESPONSE.write - Collector #271: New environment variables are now used to send the access log into syslog. ZSYSLOG_ACCESS, ZSYSLOG_ACCESS_FACILITY, and SYSLOG_ACCESS_SERVER now do the same job as the old environment variables without _ACCESS in their name. Those old environment variables still do the same job of sending the event log to syslog. - When run as a daemon on Unix, Zope will now redirect stdin/stdout/stderr to /dev/null - Nicer formatting for the increasingly tall permissions table. - TextIndex: Enhanced splitter functionality now allows the TextIndex to index numbers, single characters. It is also possible to enable case-sensitive indexing. The new configuration options are available through the addForm of the Vocabulary object. - ICP server support. For more information see http://www.zope.org/Members/htrd/icp/intro - STXNG: added new env. variable STX_DEFAULT_LEVEL to change the default level for elements (see doc/ENVIRONMENT.txt) - Collector #304: several catalog optimisations - New implementation of ZODB object cache. The new implemenation is more likely to keep the size of the object cache close to the target size. This change means that memory consumption may be reduced. Some users will need to increase the default cache size, because a too small setting is more likely to hurt performance than it did in the past. Third-party C extensions that use the persistence API must be recompiled, and may need to be updated to work correctly with the new cache; see PER_GHOSTIFY(). - The ZODB Connection is now resposible for registering changed objects with the current transaction. - Implementation of RestrictedCreation fishbowl proposal; Product registration can now include a function used to determine whether that product constructor want to allow objects to be created in the specified container object. - Collector 196: manage_page_style.css is now cacheable. Added freshness information to ImageFile, to improve cacheability of management interface - Collector 358: added a new parameter no_push_item to dtml-in, to inhibit automatically pushing sequence-item onto the namespace stack. - STXNG: Structured Text now supports images by default by using the HTMLWithImages class (has been disabled prior to Zope 2.6) - new option --force-http-connection-close for z2.py to prevent clients from maintaing pipelined connections to the Zope server (Collector #412) - Updated the Interface package to be compatible with Zope 3 Interfaces. This included changing some interface APIs that may affect existing products. - Added a database activity monitoring graph to the control panel, making it easier to tune the ZODB cache size. Bugs Fixed - External methods didn't properly setup func_defaults and func_code when they were first loaded. This meant mapply couldn't properly map arguments on the first try. - Fixed bug #96: Narrower/Wider buttons now work on both CSS and non-CSS compliant browsers. This allows better control for browsers that have a hard time knowing what 100% means. - Fix for Collector #319: filtered_manage_options didn't correctly filter tabs based on permission. - Made repr of an HTTPRequest.record eval'able as a dict (Collector #89). - Fixed bug #144: Upload button on dtml, py scripts, images, files and pts now raises an error if the file is not specified rather than clearing the source. - Fixed bug #275: setPermissionDefault didn't actually set the right permission -> role mappings. - Fixed bug reported on maillist during EWOULDBLOCK when using FTP server (http:// lists.zope.org/pipermail/zope/2002-March/111521.html). - App/FindHomes.py now computes the "real" path for SOFTWARE_HOME and INSTANCE_HOME, resolving any symlinks in any element within paths passed in via the INSTANCE_HOME or SOFTWARE_HOME envvars. Paths that are computed by "dead reckoning" from os.getcwd and module paths are also "realpathed". So for instance, if you use '/home/chrism/Instance' as your INSTANCE_HOME, and '/home/chrism' is a symlink to '/other/home/chrism', your INSTANCE_HOME will be computed as '/other/home/chrism/Instance'. This is necessary to avoid weirdnesses while using "dead reckoning" from INSTANCE_HOME and SOFTWARE_HOME in other parts of the code. POSIX systems only. - Fixed PropertyManager/PropertySheets so that you can safely add a property named 'ids' without breaking your properties page. - Removed spurious 'self' from scarecrow interfaces; updated method-generation in Interface package to ignore self when source is a method (rather than a function). - Collector #32: Use difflib instead of ndiff - Fixed long standing bug in PythonScript where get_size returned the incorrect length. This broke editing using EMACS via FTP or WebDAV. Thanks to John Glavin at South River Technologies for help finding the bug. - Collector #207: fixed problem with inner links in STXNG - Collector #210: HTML() function of StructuredText produced wrong tags. - Collector #166: ObjectManger.all_meta_types() implemented only an incomplete filter based on interfaces. - FTP: Downloading files through FTP has been broken since 2.4.0 because the downloaded file has been stored with a HTTP header at the beginning of the file. Fixed! - FTP: Spaces in usernames inside a FTP file listing are now replaced by underscores to avoid confusion with some FTP clients. - Collector #227: improved handling of unicode string in TextIndex.py with unmodified default encoding in site.py. - Collector #227: z2.py, TextIndex/dtml/manage_vocab.dtml modified to display unicode strings in the vocabulary properly (now using UTF-8 encoding for display purposes) - Collector #250: applied several patches for TextIndex for better unicode support for the GlobbingLexicon - Collector #254: return owner object from getOwner wrapped in its context - Collector #259: walkandscrub.py did not delete all .pyc and .pyo files during installation. Fixed. - Collector #231: BTrees ignoring errors from comparison function - Collector #278: DocumentWithImages could not handle URLs with underscores - Collector #279: changed exception handling for safegmtime() to provide a more intuitive traceback for operating systems with a limited gmtime() implementations - Collector #285: Zope changes its working directory to the var directory at startup - WebDAV: removing an non-existing property now returns a HTTP 200-OK response instead of 404 (now compliant with RFC 2518) - Fixed a bug in TM.py that would cause database adapters to hang on errors in the second phase of the two-phase commit. - Collector #291: ZCatalog not unindexing deleted properties - Collector #266: Retried requests losing track of http request headers, causing Connection:Close requests to stall - Collector #17: Fixed broken links in StandardCacheManagers help - Collector #1: UNIX security fixes: make starting Zope as 'root' secure, stop using 'nobody', warn of insecure umasks - Collector #303: Properties of type 'long' got truncated - Collector #325: adding a new TextIndex to an existing Catalog cleared the standard Vocabulary. - Collector #373: content_type property for Image objects are no longer deletable to prevent malfunction. - Collector #343: The ZCatalogs 'Indexes' view showed the wrong number of indexed objects for FieldIndexes. - FTP server: replaced 'System_Process' by 'Sysproc' to avoid breaking some FTP clients and the output format with overlong usernames. - Fixed a potential bug with cAccessControl's permission role deallocator which would try to decref things which may not have been set, due to a change in the initializer (which will bail out if it doesnt get called with a tuple argument) - Collector #185, 341: PCGIServer and FCGIServer logs corrected and now output extended information like HTTPServer does. - Propertysheets: Ids like 'values' and 'items' are now forbidden as they break WebDAV functionality. Existing Propertysheets are not affected - Collector #348: decapitate() now recognizes both \r\n and \n\n to be compliant with the HTTP RFC - Collector #386: workaround for hanging FTP connections with NcFTP - Collector #419: repaired off-by-1 errors and IndexErrors when slicing BTree-based data structures. For example, an_IIBTree.items()[0:0] had length 1 (should be empty) if the tree wsan't empty. Zope 2.5.1 beta 1 Bugs Fixed - Fixed failed incref in cAccessControl that caused problems when ghosting occured. Zope 2.5.1 New Features - Added the fsIndex feature. This adds a new BTree variant, fsBTree, a module fsIndex.py that uses it, and some changes to FileStorage.py to use fsIndex if it exists so that the oid-to-offset index (pindex) is a BTree rather than a straight dict. - ZODB changes necessary to support StandaloneZODB were merged - requestprofiler: added new --daysago option and added support for reading gzipped detailed logfiles - HTTP_TRACEBACK_STYLE environment variable controls the appearance of error tracebacks when not in debug mode. Recognized values are 'none' (no traceback), 'js' (§ that expands into traceback when clicked), and 'plain' (plain text). The default is to include the traceback in an HTML comment. Bugs Fixed - A fix for the issue addressed by Hotfix 2002-04-15 was incorporated into the core. - Fixed table regex parsing bug in ClassicStructuredText thanks to Albert Ting via Zope maillist. - Register CatalogPathAware as a ZClass base class in response to Collector #33. - Fix for Collector #319: filtered_manage_options didn't correctly filter tabs based on permission. - Fix for Collector # 275: setPermissionDefault was not being propagated to actual security settings. - Fixed bug where TTW code that called: del REQUEST.SESSION['key'] Would fail with a slice error (due to the security machinery). Assigning __garded_delitem__ to __delitem__ solves this problem, and allows del to be called against a SESSION key value. - Fixed a bug that could cause spurious AttributeErrors and other strange failures when trying to work with methods acquired from within the context of a transient object obtained via REQUEST.SESSION. - A last modified attribute has been added to transient objects. The conflict resolution method of transient objects now uses the last_modified time when deciding which if any of the three states to return. Objects with a later last modified time are preferred. Thanks to Tres for finding a corner case that identified the problem with the "old" conflict resolution machinery and suggesting this fix. - Collector #236: recursive ownership changes could be short-circuited prematurely if a user already owned a subobject. - Page templates will now not expand if expand=0 is passed to them. - ZTUtils.Zope.Batch now has the same default orphan parameter as ZTUtils.Batch.Batch. Calculation of previous batches with non-zero orphans is fixed. - Python-based Scripts that need to be recompiled for efficiency now emit a single, much more explanatory log entry, per run. The recompile() utility function works properly now. - Caused sqltest implementation to match help system docs Expression tags now work. "multiple" attribute now works. type="nb" attr now works. (thanks to Christian Theune for the patch) - Fixed cAccessControl to incref __roles__ in imPermissionRole_of - Fixed long standing bug in PythonScript where get_size returned the incorrect length. This broke editing using EMACS via FTP or WebDAV. Thanks to John Glavin at South River Technologies for help finding the bug. - Collector #207: fixed problem with inner links in STXNG - Collector #210: HTML() function of StructuredText produced wrong tags. - Collector #227: improved handling of unicode string in TextIndex.py with unmodified default encoding in site.py. - Collector #166: ObjectManger.all_meta_types() implemented only an incomplete filter based on interfaces. - FTP: Downloading files through FTP has been broken since 2.4.0 because the downloaded file has been stored with a HTTP header at the beginning of the file. Fixed! - FTP: Spaces in usernames inside a FTP file listing are now replaced by underscores to avoid confusion with some FTP clients. - Added UnicodeSplitter to Windows build process - STXNG: underlined text at the end of line has not been recognized properly - Collector #250: applied several patches for TextIndex for better unicode support for the GlobbingLexicon - Fixed compatibility of the Interface package with newer versions of Python. - Collector #259: walkandscrub.py did not delete all .pyc and .pyo files during installation. Fixed. - Collector #254, make getOwner wrap the owner object in its context - DTML: unicode conversion inside DTML to other character sets has been broken (workaround added to z2.py) - Collector #278: DocumentWithImages could not handle URLs with underscores - WebDAV: removing an non-existing property now returns a HTTP 200-OK response instead of 404 (now compliant with RFC 2518) - Updated to zlib 1.1.4. - Fixed a bug in TM.py that would cause database adapters to hang on errors in the second phase of the two-phase commit. - Collector #17: Fixed broken links in StandardCacheManagers help - Collector #303: Properties of type 'long' got truncated - Collector #325: adding a new TextIndex to an existing Catalog cleared the standard Vocabulary. - Collector #343: The ZCatalogs 'Indexes' view showed the wrong number of indexed objects for FieldIndexes. - FTP server: replaced 'System_Process' by 'Sysproc' to avoid breaking some FTP clients and the output format with overlong usernames. - Propertysheets: Ids like 'values','keys' and 'values' are now forbidden as they break WebDAV functionality. Existing Propertysheets are not affected - FTP: some passive FTP connections were blocking. Fixed. - Fix potential segfault with cAccessControl's permission role deallocator when the permisson role is constructed with a non-tuple argument. - Collector #185, 341: PCGIServer and FCGIServer logs corrected and now output extended information like HTTPServer does. - Collector #167: superValues() now includes items with duplicate IDs Zope 2.5.1 beta 1 Bugs Fixed - Fixed failed incref in cAccessControl that caused problems when ghosting occured. Zope 2.5.0 Bugs Fixed - Btree module and UnicodeSplitter could potentially fail to incref Py_None on return - Setting proxy roles on DTMLMethods and Documents has been broken - Collector #99: items(), keys() and values() methods of IIBucket objects has been broken when called with min/max parameters. - WebDAV: The long outstanding read-only problem with WinWord and .html is finally solved. Zope sends now an additional ETag header for DTMLMethods and DTMLDocuments. Many thanks to Joachim Schmitz for this hint. - PathIndex: fixed minor bugs, enhanced testsuite - DTML-MIME tag: creates now a 'Mime-Version: 1.0' to make some fussy email programs happy - utilities/requestprofiler - fixed bugs in timed mode which could show nonsensical results. Features Added - TextIndex/Splitters: the constructor of all three splitters has now three new optional parameters: 'maxlen'=(1-256) - to specify the maximum length of splitted words 'singlechar'=(1|0) - allows single characters to be indexed 'indexnumbers'=(1|0)- allows numbers to be indexed The default values of all parameters reflect the standard behaviour. - Enhancements to utilites/requestprofiler.py: Added readstats and writestats features which allow for saves and reuse of profile stats between runs of the requestprofiler. Added urlfocus mode. Urlfocus mode presents a summary of activity within a period of time before and after the invocation of a particular URL as it was recorded in the big M log. This can be useful when trying to track down a problem which you believe is related to some series of URL invocations as opposed to a single URL invocation. - added doc/ENVIRONMENT.txt to document all used environment variables Zope is using - Provided a much more robust tool for recovering data from damaged FileStorage files: Allow recovery of data when: - either transaction or data records are damaged and when - multiple parts of a file are damaged The interface has changed to not modify in place. Other features: - Progress indicator - Verbose output - optional packing - index creation - The last entry in the breadcrumb path in the ZMI is now an underlined hyperlink. - Added a new script "utitilies/check_catalog.py" to perform some consistency checks on a ZCatalog instance - Added the following new products related to sessioning: - browser id manager - session data manager - transient object container - temporary folder - Zope will create default sessioning objects on startup - Zope will create an Examples folder on startup. This folder contains a collection of simple example applications. Bugs Fixed - STXNG: added '+' as allowed character inside URLs - enhanced the checks for 'Domains' field in the UserFolder - Webdav: fixed broken MOVE and COPY commands - Updated DTMLMethod validation support patch with version by Steve Alexander - did not work properly under some acquisition related reaons. - pre-2.5 installation could not properly migrated to 2.5 because of changes in the Splitter API - ZODB conflict errors were logged at a severity that caused tracebacks to be logged to the stupid log file "out of the box". Because conflict errors happen under normal operations, and their appearance in the log frequently alarms folks, typical conflict error logging verbiage has been reduced, and traceback logging has been changed to happen only at BLATHER log level, which means that conflict tracebacks will only be logged if STUPID_LOG_SEVERITY is set to -100 or lower. Zope 2.5 Alpha 2 Bugs Fixed - New accelerated security management bugfix in App/Manager.py, changing calling sequence to positional parameters. - STXNG: re-enabled inner now work also through DTML and not only when using STXNG standalone - STXNG: generated HTML is now more HTML4/XHTML compliant (quoted attributes, inner links) Zope 2.5 Alpha 1 Features Added - re-enabled inner links for STXNG documents. - Zope installations upgraded from pre-2.4 installations did not show the WebDAV LockManager entry in the control panel. - "requestprofiler.py" script in utilities now does better analysis, including: "active" in detailed output is now slightly more meaningful. It is > 0 only if other requests started after it started but before it finished. The greater the active count, the more likely it is that something was going on at the time in which the request ran that caused a slowdown. Multiple files may be analyzed at the same time. Also, script recognizes "U" opcode in big M logs as meaning a restart. - Added a "preview" field to the edit form for Image objects. This provides a preview of the actual image data on the form to make it easier to know what you are working on. The preview image is scaled if necessary to be useful without being obtrusive. - Added the ability to edit File object content in a textarea. If the content of a File is a text type and the size is less than 64K (the max size for text area data in some Web browsers), then you can edit the content as you would a DTML or other text-based object. - New user management API for user folder objects was implemented. As of Zope 2.5, manage_addUser, manage_editUser and manage_delUsers form the official API for user management. The old grotesque way of using manage_users is now deprecated. The default implementation of these API methods simply call the _doXXX versions of the methods that user folder authors have already implemented, which means that these APIs will work for current user folder implementations without any action on the part of the author. User folder authors that implement the new manage_XXX API can get rid of the old _doXXX versions of the methods, which are no longer required (we only use them if the new api is not directly implemented). API documentation for the User Folder API was also added to the help system. - Python compiler in RestrictedPython brought in sync with main distribution. Made to be compatible with Python 2.2. - Added links in the "debugging information" control panel for viewing the contents of the ZODB cache. - Added user password encryption capability, fulfilling the needs described in the proposal at: http://dev.zope.org/Wikis/DevSite/Proposals/EncryptedUserfolderPasswords There is now a "properties" tab in user folders where you can select whether passwords are stored encrypted. - Locale support in STXNG has been broken (since 2.4.0) - Added SOFTWARE_HOME, INSTANCE_HOME, and CLIENT_HOME to the control panel to make it easer to tell what configuration of Zope is running. - Re-applied Medusa patch to allow proxy-like HTTP requests (GET http://hostname:port/ HTTP/1.0) - Objects indexed by the PathIndex can now provide a hook or an attribute with the same name as the index name to provide a customized path information other than using getPhysicalPath(). - Selecting "Clear Catalog" from the ZCatalog "Advanced" did not clear the vocabulary associated with a TextIndex. - Added updated and enhanced testsuite for STXNG. - added getEntryForObject() for PathIndexes. Reworked PathIndex's internal inverse index. - API help topics can now document functions as well as classes. - Security accelerations in c - Accelerated C Document Template handling; Document Templates can now do additional rendering in C, improving performance. - Unicode support for ZCatalog: - added new UnicodeSplitter - ZCatalog now allows unicode strings as attributes of objects to be indexed using a TextIndex (see lib/python/ZCatalog/README.txt) Bugs fixed - WebDAV: Zope escaped nested object properties derived from internal dav__* functions in PropertySheets.py although they are considered to be safe and do not need any escaping. This caused Zope to be completely incomplete with Windows XP. Fixed ! - WebDAV: '(' and ')' are now allowed in Ids for Zope objects. This is needed when one tries to duplicate files using cut&paste through Microsoft webfolders. - Collector #2532: ZCatalog.availableSplitters is now protected by security mechanism. - Collector #2412: a read-only FileStorage has not been closed properly. - Collector #2390: Objects of type 'Help Image' were not properly re-registered inside registerHelp(). - Fixed broken FTP download for larger files. - Collector # 2396: StructuredText did not allow URLs containing "%" - Collector # 2397: StructuredText could not handle underlined text properly. Also will no longer produce .. and .. - Collector #2438: Using a slice operation like [30:] on a ZCatalog search result caused a MemoryError because the __getslice__ implementation used range() instead of xrange(). - Collector #2423: Searching a FieldIndexes for documents with a blank string has been broken. - WebDAV Lockmanager was not working due to a Python 2.1 incompatibility. - Collector #2482: A COPY operation through WebDAV on a locked resource left the destination resource in a locked state so any WebDAV client was unable to unlock the destination object. Locks are now cleared from the destination object. - Error message AttributeError/_v_blocks when a DTMLfile is not present or could not be read replaced by a more informative message. - Collector #2497: SERVER_PROTOCOL variable is now compliant with the CGI specification and looks like "HTTP/1.1" instead of "1.1" - Creation of a TextIndex ignored the vocabulary setting. - Fixed broken aquisition of vocabularies from a Catalog by a TextIndex. - Collector #2504: level parameter has not been passed to HTMLClass constructor - default for 'orphan' attribute of is now 0 instead 3. - Fixed conflict resolution problem in BTrees (BTreeTemplace/ _p_resolveConflict) - Fixed many subtle bugs in the BTrees, including some that could silently corrupt the BTree and render some data inaccessible. - Collector #2524: Medusa sent "HTTP/None..." as response header when then HTTP version could not be determined from the HTTP request. Now sending "HTTP/1.0..." - queries for the PathIndex can now specified as tuple (path,level). the level parameter inside a query overrides the optional 'level' parameter for a complete search request. - Collector #2561: XXBucket.values() returned keys instead of values - Fixed the API docs for user objects. - Fixed bad interaction between ZCatalog and dtml-in (submitted by Steve Alexander) - Multiple links in a paragraph with mixed link notation (quotation+colon+URL or quotation+comma+whitespace+URL) did not work properly (thanks to Alastair Burt for reporting and submitting the patch). - Fixed case where DTMLMethod.py complained when it tried to remove the accelerated DTML security validation routine after recursive entry. Zope 2.4 Bugs fixed - Some of the import shenanigans in ZService.py were still trying to load Python 1.5 versions of support dlls for running as a service under win32 (meaning you effectively couldn't). - Collector #2335: older products like GadflyDA that did not use registerClass() were registered but their visibility flag has not been set to Global. So these products did not appear in the ZMF. Zope 2.4 beta 1 Features Added - Added a new lesson to the Zope tutorial which shows how to insert data into a relational database. - Made the meta-type in the upper-left corner of ZClass instances' management interface into a link to the ZClass definition. - Collector #2316: introduced new environment variable ZSYSLOG_FACILITY to override the default syslog facility 'user' Bugs fixed - Updated the Zope tutorial a bit. Now it references the Zope Book and the online help's DTML reference. Replaced ZopeTime examples with DateTime in lesson 9. - The manage_afterClone hook was called too early for copied objects, before the cloned object was reattached to its context. This caused problems with CatalogAware objects reindexing themselves on copy. - Fixed a situation that could slow down lazy processing of sorted search results. - Product import ordering has been changed. Now, the total set of all Products are initialized in ascending alphabetical order by product name, regardless of whether an INSTANCE_HOME exists. If two products with the same name exist in both SOFTWARE_HOME and INSTANCE_HOME, the order in which they appear in Application.Products.__path__ determines whether they are loaded first or second. - Collector #2318: html_quote function re-aliased into DT_Util so as not to break existing user code. - Corrections to the code that adds interface information to the meta type registry. - Collector #2274: fixed typo in Transaction.py. Made Connection.py a bit more noisy in case of failure - Collector #2273: fixed bug in STXNG (unable to handle :img: commands with absolute URLs) - Collector #2272: fixed problem in Transaction.py (Objects with id==None could not print themselves) - fixed Python module import problems causing XML export/import too be completely broken - Collector #2270: fixed import problem when $INSTANCE_HOME was set - Collector #2269: default script for PythonScript is no longer compiled during installation - Updated to zlib 1.1.3. - fixed parenthesis handling for TextIndex queries - Collector #2283: PersistentMapping class keys() method now returns a *copy* of the cached keys list instead of the cached list of keys itself. This prevents mutation of the list. - DateTime constructor now accepts ISO 8601 compliant dates - Collector #2254: added hook for webdav/ftp MKCOL - Collector #2247: put values of properties into CDATA section to prevent non XML compliant documents to be sent to a webdav client. - Collector #2287: fixed missing import of 'render_blocks' in SendMailTag.py - Collector #2289: leading \n\r inside messageText broke mail headers - Collector #2290: better handling of subject header in Mailhost.py - Collector #2291: fixed cookie path in Zope Version Control - Collector #2286: Could not create ZClasses. coptimizations.c had been corrected erroneously. Fixed. - It was discovered that the implicit names of certain DTMLMethods were ambiguous, making security assertions for them unpredictable. Code was added to detect the condition and correct it in the Zope core. The affected DTML methods did not pose a security risk. Instead, most were overprotective. - "print" with a trailing comma (to avoid a newline) caused spurious errors in Python Scripts. It turned out the compiler module was in error. Fixed. - Collector #2306: Fixed broken glossary of Zope Tutorial - Collector #2305: A comma inside an optional parameter of the parameter string broke the ZScriptHTML_tryParams function of PythonScripts. - Restored performance of security checks in DTML and Scripts (Python) by replacing read guards with function calls to guarded_getattr() and guarded_getitem() of ZopeGuards. - Restricted the target of PythonScript print statements. - Python 2.1 reopened the old "AttributeError: __call__" bug. Checked in a fix that should finally solve it correctly. Zope 2.4 alpha 1 Features Added - Zope now requires Python 2.1. See the Zope 2.4 migration document for details: http://www.zope.org/Products/Zope/2.4.0/Zope24MigrationGuide.html - The 'Authenticated' role has been added as one of Zope's standard roles. A user's possession of this role indicates that he or she has been authenticated by the Zope security machinery. It is an implicit role, and cannot be provided to users within the user management screens or the local roles management screens. It is provided to all authenticated users. Another way to think of the 'Authenticated' role is that it is possessed by all users *except* the Anonymous User. - A module 'requestprofiler.py' is now part of the utilities directory. When run from the command line, this utility allows you to generate profiling information from Zope's detailed request log (the "-M" log). - Restricted code can import AccessControl.getSecurityManager, and use 'validate', 'validateValue', 'checkPermission', 'getUser', and 'calledByExecutable'. - Zope's WebDAV support now includes exclusive write locking. - Data.fs.in index_html now shows zope_quick_start instead of old, inaccurate content. - Changed index_html, standard_html_header, standard_html_footer, and standard_error_message in Data.fs.in to use "new" DTML syntax (as opposed to SSI-style syntax). - Added check for 'FORCE_PRODUCT_LOAD' environ var to bits which try to detect whether we're a ZEO client or not in order not to skip product-loading tasks. This has the consequence that a ZEO client with the "ZEO_CLIENT" env var set will cause product loading tasks to occur iff his "FORCE_PRODUCT_LOAD" environment var is set as well. - Further optimizations in the lexicon of the Catalog as well as the final merge code integration for Text indexes. This should reduce the bloat when things are reindexed. - StructuredText: old StructuredText replaces by StructuredTextNG. Added support for locale settings. Several bugfixes and code cleanup. - Image and File objects now support the HTTP Range and If-Range headers, enabeling partial downloads of such objects. This can be used by clients to restart a broken download. Downloads of these objects through FTP can also be restarted. - Update of Medusa tree to latest version - If an INSTANCE_HOME is defined and has a 'lib/python' subdirectory, it will be added to the front of the Python path. - DTML-In and DTML-Tree now have optional "prefix" attributes that can be used to make friendlier tag variable names. - Added product reloading capability, formerly provided by the "Refresh" product. This enables developers to see the effect of changes to their products without restarting Zope. - Added new 'sequence' module for underscore namespace to provide extended sorting functionality for sequence (implements ExtendedDTMLSorting proposal) - Fixed a long-standing bug in FileStorage that made it so versions were only partially committed. - Rewrote the complete indexing infrastructure according to the DropinIndex proposal (see lib/python/Products/PlugginIndexes/README.txt for detailed informations). - Adopted ZCatalog to new indexing infrastructure. - added text() method to HTTPRequest object to provide a plain text representation of the request (Collector #2264) Bugs Fixed - TextIndexes which called methods expecting an argument failed with a TypeError. This was fixed by extending an exception handler. - A security issue having to do with setting permission mappings on ZClass methods was fixed (this supersedes Hotfix-2001-05-01) - Automatic reloading of Help topics while running in debug mode was fixed. - Fixed problem adding propertynames with spaces (Collector #2206) - Plugged a memory leak in extensions. There are often circular references in code generated by Python's "exec" statement but with a small change extensions now break those references. - Hardened ZMI contents view against subobjects w/ flaky 'get_size' (Collector #1900). - Fixed OFS.CopySupport's _verifyObjectPaste to utilize the permission stored in the metatype registry, if available (Collector #1975) - Fixed a long-standing bug in FileStorage that made it so versions were only partially committed. Zope 2.3.2 Bugs accidentally fixed :) - Fixed missing html_quote attribute inside some DTML methods (Collector #2170) - Fixed logging output to the compatible with the Common Logfile Format specs (Collector #2188) Zope 2.3.2 beta 2 Bugs fixed - Fixed Catalog "object does not support item deletion" bug. Zope 2.3.2 beta 1 Bugs Fixed - Catalog field index bug fixed. - Fixed several places where filesystem paths were assembled in a non-portable way. - Fixed some places in SimpleItem that assumed that the object was an Acquirer. - Fixed layout problem in StandardCacheManagers/dtml/propsRCM.dtml (Collector #2152) - Fix bug in AcceleratedHTTPCacheManager. Sended HTTP headers were depending on locale settings (Collector #2142) - GoLIVE is unable to handle some empty properties send from the Zope's WebDAV server. (getlastmodified is no more send when they are not available for an object) (Collector #2150) - added globbing support for the FTP server - added support for fetching recursive listings from the FTP server (needed by ncftp) - fixed handling of broken objects in the WebDAV server - An ambiguous timezone alias (CDT) was removed from DateTime. - It was not possible to pass 'class=...' to the tag() method of Image objects because 'class' is a Python reserved word. Image.tag() now accepts an optional 'css_class' argument that is turned into a 'class' attribute on the resulting tag to work around this. - Fixed various places where the right tabs were not highlighted after form submissions. - The code that produces the data for the security settings form included a misleading default ('Manager') that could make it looks as though Manager had a permission in a subobject when that permission had actually been denied in a higher level object. This only affected the form generated, not the actual security settings in effect. - The textarea resizing buttons for DTML objects and scripts were not preserving work done in the text area during the resize. - Some changes were made to the implemenation of User.allowed(), which will make it less expensive to do local role matching and also resolves an issue noted in the UserProgrammableSecurityObjects proposal on dev.zope.org. - A problem in the bytecode munging done by Python scripts that could cause a core dump was fixed. Zope 2.3.1 Bugs Fixed - Fixed catalog length error bug. - Fixed textindex search queries with parenthesis (which haven't worked since at least 2.2.4). - Added logic to use the value of 'textindex_operator' as passed in the request to determine which query operator to use (and, near, andnot, or). Valid values to pass in to textindex_operator are 'and', 'or', 'near', and 'andnot' (capitalization is ignored). This is a near-term workaround for the inability to specify a default text index query operator on a per-index basis. It provides the ability to override the currently module-defined default 'Or' operator for textindexes on a per-search basis. An example of the utility of textindex_operator used with a ZCatalog instance: zcatalog.searchResults(atextindex='foo', textindex_operator='and') - The import machinery did not correctly find import files located in SOFTWARE_HOME/import if SOFTWARE_HOME / INSTANCE_HOME setups were in use. - The default view for "broken" objects was broken :) - The FTP server now provides for more informative error handling through a setMessage() method on the ftp response object (thanks to Richard Jones). - The title of HTML help files is now parsed out and used by the help engine (thanks to Richard Jones). - Objects wrapped with Explicit acquisition wrappers were not correctly handling the __nonzero__ protocol (as well as other numeric protocols). - A bug in the sendmail tag that could cause "len of unsized object" errors has been fixed. - Some bits of DateTime parsing were not raising the same errors claimed in the documentation. - Handling of HTTP "Destination" headers for WebDAV MOVE and COPY was not tolerant of complete URLs (including scheme, server, etc.) - Keyword indexes generated spurious error logs when objects without keywords were added. - Fixed a memory error in the new BTree buckets that surfaced during testing on ia64. - Fixed a multi-arg append() call in the ClassicDocumentClass in StructuredText. Zope 2.3.1 beta 3 Bugs Fixed - ImageFile objects were not using os.path.join to build the local path to an image file, which caused oddities on win32. - Some missing casts in Splitter.c prevented the splitter from properly working with some international characters. - Fixed documentation of dtml-var tag. Improved discussion of entity syntax and fmt and url attributes. - Fixed manage_convertBTrees method of ZCatalogs (this method was failing due to a missed copying of the length value, preventing Catalogs from showing their "old" items once converted). - Changed the presentation of last-modified dates in ZMI listings to an ISO format that is a bit shorter and a better common format. - Added a patch to load_site.py to handle entity references. - Added a patch to load_site.py to use the new dtml syntax when adding header and footer lines. - Handling of the __nonzero__ protocol by Acquisition wrappers was fixed. Zope 2.3.1 beta 2 Bugs Fixed - Fixed formatting bugs in the DTML reference. Also added a note to the dtml-tree tag reference page explaining the leaves attribute better. - Fixed broken Image and File downloads when using an Apache server set up to cache such data. In certain circumstances a content-length of 0 would be sent, which made Internet Explorer not read any more than that, resulting in broken images and downloads. - Fixed a doc string display bug in API help topics. Also slightly improved the display of help topics which include more than one class. - Fixed inaccuracies in 'dtml-in' DTML reference help topic. (Collector #1972) - Merged a bug fix that was missed in the logging call when an object that is not in a Text Index tries to unindex itself. - Removed a duplicate list item from the copyright screen. - Fixed a bug in the docstring of DemoStorage.py - Added ZPL to ZPublisher/BeforeTraverse.py - The tabs behaved oddly in the "database management" part of the control panel. - The icon for Python script objects has been updated. Thanks to Chris Withers for sending it in. - Added an entry to doc/FAQ.txt about the "checksum error" that some vendor-supplied tar implementations (Solaris) produce due to long pathnames in an archive produced by GNU tar. - Using the 'scale' argument to the 'tag' method of an Image object could produce invalid height and width values in the resulting HTML tag (floating point values). - The tab corner .gifs for inactive tabs were not transparent in places where they should have been. - Corrected mistakes in the logic that retries the request when a ConflictError occurs. The behavior was a little erratic. - The problems ('attribute error: commit_sub') that some people were having with certain kinds of objects (like database connections) not playing correctly with subtransactions have been fixed. - Applied changes from Hotfix_2001-02-23. Some ZClass methods weren't protected properly and the ObjectManager, PropertyManager, and PropertySheets classes were returning mutable objects that could be exploited by restricted code. - A malformed html option tag made the add list broken for Mozilla. - The permission declaration for 'manage_main' on ZSQLMethod objects was missing, making editing effectively only available to the Manager role (the Change Database Methods permission had no effect). - A missing '/' at the end of the contents form action url caused some clients to behave strangely. - A long-standing problem with traversal of object names with spaces in them from WebDAV tools has been resolved. Part of the DAV machinery was in some cases double-url-quoting DAV:href values. - A long-languishing patch from Dieter Maurer to fix batching info in dtml-in has been applied. - z2.py failed to correctly pass an address:port specification for the ZSYSLOG_SERVER environment variable to the medusa syslog logger. - The value passed for the ZSYSLOG environment variable was not being used in the same way by logging machinery in ZServer and the Zope core. The file in doc/LOGGING.txt has been updated to reflect that the value passed for ZSYSLOG should be the path to the UNIX domain socket to log to (usually '/dev/log'). - Using the strftime method of DateTime objects (often via the fmt attribute of a dtml tag) produced incorrect results for DateTime objects outside of the range of the native C strftime implementation. A change has been made so that an exception is raised in those situations rather than silently return the wrong answer. - Corrected the aq_inContextOf() method so it always follows containment wrappers. - The headers attribute of FileUpload objects did not have any security assertions, which meant that it could not be accessed from DTML as advertised. - A number of Collector patches to fix DateTime issues (mostly regarding timezone handling) were applied. - Key errors were returned from catalog searches due to bugs in unindexing objects. - Objects that should have been included in search results were excluded do to bugs in indexing objects. - The database grew too fast when objects were routinely indexed one at a time by the catalog. This was due to inefficiencies in the underlying BTree implementation that caused too many objects to be changed when indexing incrementally. In addition, the set implementation, used for keeping track of documents in field indexes, when the number of documents per key was large, caused large database updates when a document was added or removed, because the entire set was stored in a single database record. A new BTree and set implementation has been provided to address these problems. - The catalog is the most common source of conflict errors due to a number of "hot spots" in the implementation and due to the large number of objects that are typically updated when indexing. The hot spots have now been removed and the number of objects modified when indexing has been dramatically reduced. In addition, conflicts detected during transaction commit can now usually be resolved using new conflict-resolution protocols and conflict-resolution support added to the catalog. These changes will not avoid all conflicts, but will reduce conflicts significantly in many cases. Specific information on how to update existing Catalogs to take advantage of the data structures available can be found at: http://www.zope.org/Members/mcdonc/HowTos/UpgradeToNewCatalog Zope 2.3.1 beta 1 Bugs Fixed - PUT on NullResource is now a public method that checks whether the user can create the type of object returned by a PUT_Factory. - dtml-in sorting using unhashable types (like lists) has been fixed. - A new environment variable "FORCE_PRODUCT_LOAD" when set on ZEO clients (clients which also define the "ZEO_CLIENT" env var) causes product loading to be "forced" by that ZEO client. Formerly, if all ZEO clients had the "ZEO_CLIENT" environment variable set, it was impossible to force a product load without undefining the "ZEO_CLIENT" env var on one of the ZEO clients. Now, if one of the ZEO clients has both the ZEO_CLIENT env var and the FORCE_PRODUCT_LOAD env var defined, that client will push products into the ZODB. It is advisable to only set FORCE_PRODUCT_LOAD on one ZEO client if it's set at all. - A typo in CopySupport caused the wrong http error to be raised in manage_renameObjects (500 Server Error instead of Bad Request). - A bit of PermissionMapping logic was calling manage_access with too many arguments, causing an error when trying to make changes on the Security tab of ZClasses. - The Catalog no longers throws an AttributeError if you pass it a 'sort_on' parameter that isn't useful. Instead, it throws a ValueError with an instructional string. - An error in the edit form for Permission objects made it impossible to edit Permission objects in Products through the Web. - The script source in Python scripts was not being html quoted on the Web edit form. - Fixed a bug in Mount.py that made it difficult to see the reason connections failed. - Fixed a bug which made it impossible to cut and paste ZCatalog instances. - Processes spawned from a Zope process could have a bad effect if one tried to kill the Zope process while a spawned process was still active. Because the spawned process had inherited the listening sockets of the servers across the fork(), it was not possible to restart the Zope instance until the spawned process had died and release the server sockets :( - Added a "tempfile.py" module to lib/python. This tempfile module is r1.27 from the Python CVS MAIN branch. It fixes two race condition bugs that could appear under heavy load. This module will be used by Zope instead of the tempfile.py that ships with the Python source or with a Zope binary distribution. It is probably unnecessary under Python 2.0+, but won't hurt. - Removed lib/python/App/manage.dtml, which wasn't removed when dtml files were moved to a separate directory. - Fixed two mistakes in the RAM cache cleanup code. - Added code to handle a failed BeforeTraverse hook gracefully. - Added help system docs for Script (Python) instance management screens. - Checked in Stephen Purcell's PyUnit 1.3, replacing PyUnit 1.2. - Hardened ZMI contents view against subobjects w/ flaky 'get_size' (Collector #1900). - Corrected a bug that could leave a cacheable object associated with the wrong cache manager after switching cache managers. - Changed the html title tag in the manage frameset to use BASE0 instead of SERVER_NAME, providing more useful info for virtual hosted sites (patch from Chris Withers). - The highlighted active tab wasn't correct for some security views. - Changed undo error message wording in FileStorage, DemoStorage, and POSException. Confusing error reports that claim a transaction could not be undone because the transaction was "undoable" now claim that the transaction was "non-undoable". - Some conflict errors failed to properly quote object ids, making the ids unreadable and introducting binary text into output. - Changed Product init/list code to accept all of "VERSION.TXT", "VERSION.txt", and "version.txt" as the version.txt file for the Product. Additionally accept any of "README.txt", "README.TXT", or "readme.txt" as the readme filename. - When the root index_html was missing, an inappropriate error was displayed to a visitor when viewing an object that didn't have its own index_html. (Collector 1954, thanks to Chris Withers for the bugreport). - A bug prevented unchecking the "cache anonymous connections only" checkbox for Accelerated HTTP Cache Managers. - Some code that tried to workaround a DAV client bug dealing with ports in Host headers caused problems for virtual hosting setups. That code has been disabled until we decide that we care enough about the buggy client to work around it in a better way. - The isCurrent(Month|Day|Hour|Minute) methods of DateTime objects returned incorrect answers. Thanks to Casey Duncan for the patch. - The "help" links did not work if javascript was disabled in the client. - ZCatalog getobject now uses unrestrictedTraverse during getobject instead of restrictedTraverse. This emulates Zope 2.2 behavior. - A bug in the HelpTopic implementation for STX help topics caused them to be inaccessible unless Anonymous Users had the View permission. - Structured Text did not correctly recognize CRLFs generated by windows editors as paragraph dividers. - A bit of code that ran after an import where ZClass objects were added was removed. It was designed to fixup the ZClass registry after import to resolve ZClass dependencies that could get broken if a system were moved in separate imports. It turned out to be expensive under ZEO, and the benefit is not that compelling (if a failed dependency does get introduced, it will be spotted at the next startup). - The month name recorded in Z2.log was affected by the current locale setting, which caused problems for various logfile analysis tools. - Product object in the control panel had a useless View tab by an accident of inheritance. - Manual restarts and shutdowns weren't logged. - Unix: If processes were restarted too frequently, the daemon process incorrectly inferer a startup problem and shut itself down. - Unix: On restart, the watcher daemon restarted as well, causing the watchers process id to change. - The method for enabling modules for use with Python scripts was not documented and a bit harder than it needed to be. A helper function has been added in a Utility.py in the PythonScripts product to make this easier and the process is documented in the README.txt in the PythonScripts package. - Fixed OFS.CopySupport's _verifyObjectPaste to utilize the permission stored in the metatype registry, if available (Collector #1975). - In certain situations using restrictedTraverse failed with the wrong error if called on the application object with the name of a nonexistent object. - Subtransactions couldn't be used if some data managers didn't support subtransactions. This was especially painful when using RDBMS user authentication in combination with large image or file uploads or catalog rebuilding, which use subtransactions. Now allow subtransaction commit (but not abort) even when some participating data managers don't understand subtransactions. New Features - The API for cataloging objects changed slightly: o If the object being cataloged has a 'getPhysicalPath' method, as Zope objects typically (always) do, then it is no longer necessary to pass a unique ID to the catalog when catalling the 'catalog_object' method. It is recommended to not pass a unique id and let the catalog figure out the unique id on it's own. o If a unique id is passed to the catalog 'catalog_object' method, it **must** be a string. Zope 2.3.0 Bugs Fixed - The authentication machinery now correctly returns a 400 (Bad Request) if an invalid authentication token (bad base64 encoding) is sent by a client. - ZClasses with very minimal base classes could end up without a '_setId' method, which createInObjectManager expects. - Fixed a bug that caused the ExtensionClass __call_method__ hook to fail when used with unbound C methods. - Fixed a bug in the management interface which caused the "Paste" button to not show up after a copy or cut operation on the first showing of manage_main. - Final lexicon optimizations that provide additional performance over previous releases. In addition, the number of objects that have to be updated is frequently reduced. - Merge code for Catalog Text indexes has been integrated. This will now merge the changes in, rather than replacing them. This should reduce the number of objects that has to be updated. In addition, when nothing has changed, the object's indexes won't be touched, saving enormous amounts of space for some applications. - Flow of the Catalog management screens cleaned up so that pages are refreshed correctly. Buttons on the Advanced tab refresh to the Advanced tab now. - Further management interface cleanup of the Lexicon to bring in line with the normal ZMI. Zope 2.3.0 beta 3 Bugs Fixed - The import / export button did not show up if a folder was empty. - A problem in acquisition wrapping of users obtained though the SecurityManager caused certain ownership operations to fail (this manifested itself as a report about broken DAV MOVE operations). - The Zope management screens no longer try to set a default charset with the content-type. - Certain security related operations were failing due to argument mismatch errors (too many arguments). - Passing unicode data to html_quote could cause problems since html_quote was trying to screen out two characters that many browsers are willing to accept as html special characters (to prevent "cross-site scripting" attacks). This has been moved out of html_quote and into the RESPONSE object, where the chars will be quoted only if no charset is defined for the content-type or the charset is an alias for Latin-1. - Rename via FTP was not supported. - Changed index_html, standard_html_header, standard_html_footer, and standard_error_message in Data.fs.in to use "new" DTML syntax (as opposed to SSI-style syntax). - meta_type of all DTML Methods in Data.fs.in object manager "_objects" lists is now "DTML Method". It had been "Document", which caused inaccurate superValues results if 'spec' was used. - Make ZClasses navigable to FTP/WebDAV; implement 'PUT_factory' hook to create PythonScripts (for MIMEtype 'text/x-python') and DTMLMethods (for other 'text' MIMEtypes) (Collector #998). - Calling manage_addProperty with a list value and a type of 'lines' caused a string representation of the list to be stored. - Submitting the proxy roles form without selecting any roles to be used as proxy roles caused objects with proxy role support to silently become unexecutable (have effectively empty proxy roles) rather than raising an error. The proxy role api now requires that at least one role be passed in or an error will be raised. - Mechanisms in the underbelly of the Catalog and Globbing Lexicon (which is the default for all new Catalogs) has been overhauled given substantial performance increases. On simple queries, performance should double (or more) in many situations, whereas with globbed queries it may increase by substantially more. - A method in SQLMethod objects had been removed but the reference to it in __ac_permissions__ had not, which caused failure on attempting to set permissions on SQLMethods. - A bit of exception handing in the dtml-in tag implementation was too general and could hide subsequent rendering exceptions (thanks to Richard Jones for the patch). - Cacheability was not fully enabled for DTML Documents. Zope 2.3.0 beta 2 Bugs Fixed - Changed management style sheet to explicitly set the http content-type to avoid a rendering problem on resize in NS browsers. - Data.fs.in index_html now shows zope_quick_start instead of old, inaccurate content. - Changed index_html, standard_html_header, standard_html_footer, and standard_error_message in Data.fs.in to use "new" DTML syntax (as opposed to SSI-style syntax). - The way that the default management tree view imposed sorting in its tree tag dtml made it hard for custom objects to provide a sorting that would be more appropriate for the custom object. The management tree view now preserves whatever ordering is returned from tpValues. The default tpValues implementation in the ObjectManager class sorts by id by default. - Disallowed object IDs that start with "aq_". - Changed the default support for "domain authentication mode" in UserFolder to be disabled by default. Domain auth mode was implemented for a very specific case long ago and causes a lot of overhead for anonymous accesses that are needless for the 99% case. People who actually want domain auth mode turned on may call a new 'setDomainAuthenticationMode' method to enable it if they wish. - Changed the implementation of emergency_user to be backward compatible with the expectations of third-party user folders. Third party user folders should now work with Zope 2.3 without modification. - A bug in the search interface generation for ZCatalogs was fixed. - An integrity check for the global product registry has been added at startup to mitigate registry consistency problems caused by things like missing base classes that cannot be detected by Zope (like removing a Product that another Product depends upon). If a problem is detected, the global registry is automatically rebuilt and the action is logged. - A bug in the rendering of 'record' type form variables when rendering a request object was fixed. - A bug that cause setting of proxy roles for Python Scripts to fail was fixed. Zope 2.3.0 beta 1 Features Added - Added a hook that allows user folders to provide a logout action. - Added a browser preferences screen to allow people to tweak the management UI to their liking. For the folks who complained that they didn't like the new top frame, they can (among other things) turn it off from the browser preferences screen. - Added Michel's new QuickStart material. I haven't quite decided whether the old QuickStart should go away or stay around as a source of examples. - The logout function has been implemented in a fairly minimal way. We may try to make this nicer by final if we get time. - The ZCatalog interface is now cleaned up and matches the new interface look and feel better. In addition some logical reorganization was made to move things onto an Advanced tab. - Result sets from the Catalog are now much Lazier, and will do concatenation with eachother in a lazy fashion. This was suggested by Casey Duncan in Collector #1712. Bugs Fixed - Added a deprecated alias to UnrestrictedUser, Super, for use by user folder products that depend on the old class name. - Fixed path for management interface files used for CatalogPathAwareness and Aqueduct. - Fixed a NameError in HTTPRequest. - Made manage_page_style.css correctly available to all. - ZCatalog objects now show up in the Add List in the same naming convention that was used for all other Z* objects. This does *not* affect the meta_type that is actually used for the object itself. - (Collector #1835, 1820, 1826) Eliminated errors in both Field and Keyword indexes where old keys might show up in 'uniqueValuesFor()' because of the way the data structures were kept around. - (Collector #1823)Eliminated situation where if the Catalog did not have a metadata record for 'meta_type' the Cataloged Objects view would be incorrect and list everything as a 'ZCatalog'. Now it simply lists it as 'Unknown'. - (Collector #1844) On the brains returned from ZCatalog queries, 'getObject()' now tries to resolve URLs as well as paths. This should catch more cases. - Tags generated for ImageFile objects attempted to use title_or_id(), which is not defined for those objects. - Mounting now fails gracefully in when getId() is not available in the mounted object. Zope 2.3.0 alpha 2 Features Added - The install machinery for source release has been modified to allow Zope to build out of the box for Python 2.0. Note however, that Python 2.0 is still not officially supported. You may see quite a few warnings from the extension builder when compiling for Python 2. - A new module, AccessControl.Permissions has been added to make it easier to use the new security assertion spelling. The new module provides consistent symbolic constants for the standard Zope permissions. - Cache manager support added. This allows site administrators to ease the burden on their site in a very configurable way. It also provides an API for developers to follow when experimenting with caching strategies. - The ZPublisher 'method' form variable type has been deprecated in favor of 'action'. The behavior is the same, only the official (and documented in the Zope book) name has changed. The 'method' name is still supported for backward compatibility. - The 'objectIds' and 'objectValues' methods of ObjectManager derived objects are no longer directly Web-accessible. This is a topic that has come up over and over on the lists. Some (xml-rpc, mostly) users may depend on this behavior - applications that need access to this information remotely should be modified so that a Python Script or DTML Method can explicitly pass the data. - The Image.tag() and ZopeAttributionButton methods now return an image tag that is XHTML compatible; a space and a slash have been added. - SQLMethods can now be edited via FTP and WebDAV tools. Thanks to Anthony Baxter for his FTP support patches. - The Catalog has been slightly overhauled to manage object paths instead of URLs in its tables. This should not cause any backward compatability concern, but everyone upgrading should read the web pages on the zope.org site at: http://dev.zope.org/Wikis/DevSite/Projects/ZCatalogVirtualHostFix/UpgradeFAQ this will provide information about how to upgrade and new features on the result sets, like getObject and getPath. These are very important. - SiteAccess 2.0 has been added, to enable virtual hosting. - The StandardCacheManagers product has been added as a primary product, making it easier to get started with caching. - The class DTMLFile has been added alongside of HTMLFile. It supports name bindings, ignores positional parameters, and puts the container on top of the namespace by default. Most HTMLFiles should work the same (or more securely) if converted to a DTMLFile. Most management interface methods should be converted by the final release of 2.3. - Added a variable called PUBLISHED to REQUEST. From now on, this variable should be used instead of PARENTS for user validation. - The inituser file is now read even when one user has been created. This provides a way to reset the password after a new user installs Zope but ignores the generated password. - ZCatalogs have a reduced number of management interface tabs. - ZCatalog keyword and field indexes have been modified to use a merge strategy when existing indexes are updated. When an existing object is indexed, the contents of field and keyword indexes are merged with the changes detected between the existing contents of the index and the new content. - CatalogPathAware class added. This will eventually replace CatalogAware. - The ManagementInterfaceQuickFix project was merged in. The Zope management interface has been tweaked in various ways to improve productivity and consistency and is now at least slightly less ugly :) Bugs Fixed - A misspelled function name which prevented the addition of properties was corrected. - Caused PropertySheets to restrict IDs the same way ObjectManager does. - (Collector #1586) Fixed situation where the Catalog would attempt to loop over a bucket as if it were a list, which won't work. This was reported by Steve Alexander with a patch. - Corrected local role computation (Hotfix 2000-12-15) - The basic user folder implementation in User.py was changed to use the Zope security policy machinery. see http://dev.zope.org/Wikis/DevSite/Proposals/ ChangeUserFoldersToUseSecurityPolicyAPI for details. - Trying to cut or copy with no items selected now returns a nicer error message. - A roles keyword argument was added to ZopeSecurityPolicy.validate to enable callers to pass in roles as opposed to allowing the machinery to figure it out for itself. - Some product context initialization related to setting roles was updated. Zope 2.3.0 alpha 1 Features Added - Python Scripts are now part of the Zope core. Big whopping kudos to Evan Simpson for all of the work he has put into this! Having Python Scripts in the core will allow people to much more easily separate logic and presentation (and get that logic out of DTML!) More information and prototype documentation for Python Scripts can be found in the dev.zope.org project: http://dev.zope.org/Wikis/DevSite/Projects/PythonMethods - Added the __replaceable__ property support to ObjectManager. This is currently documented only in the Wiki. - Added unit tests for the DateTime module. - Added new BASEPATHn and URLPATHn variables in the REQUEST object, and changed Zope core DTML files to use BASEPATH1 instead of SCRIPT_NAME. - Added new getId() method to SimpleItem.Item. This should now be used instead of referencing 'object.id' directly, as it is guaranteed to always be a method and to always return the right thing regardless of how the id of the object is stored internally. - Improved Ownership controls. Now you simply choose whether or not to take ownership of sub-objects when taking ownership. There is no need to control implicit/explicit ownership. - Changed the Zope installation procedure so it is only necessary to create one user account and that user is stored in the ZODB. The user created at startup now is simply a normal intial "Manager", not the "superuser". It is no longer necessary to login, create an initial manager, logout and log back in! Woohoo! - Implemented the "emergency user" concept, which is the new name for what was called the superuser. The emergency user doesnt even exist now until you explicitly create it. - Added new "WebDAV source view" HTTP handler, enabled by new '-W' (note uppercase) switch to z2.py. This handler is *not* enabled by default. - Implemented "hookable PUT creation" (allows containers to override webdav.NullResource's guess at the type of object to create when PUT is done to an unknown ID). - Added testrunner.py to the utilities directory. The testrunner is a basic utility for running PyUnit based unit tests. It can be used to run all tests found in the Zope tree, all test suites in a given directory or in specific files. The testrunner will be used to ensure that all checked in tests pass before releases are made. For more information, see the docstring of the actual testrunner.py module. - The Interface scarecrow package has been checked in - more work will likely be done on it before it goes into wide use. See Michel's "Zope Interfaces" project on dev.zope.org for details: http://www.zope.org/Wikis/Interfaces/FrontPage - PyUnit has been checked into the core. Along with the testrunner, this provides enough infrastructure for us to incrementally begin accumulating (and running!) test suites for various parts of the Zope core. - The new security assertion support has been checked in. For more information and an updated version of the "Zope security for developers" guide see the project on dev.zope.org: http://dev.zope.org/Wikis/DevSite/Projects/DeclarativeSecurity Bugs Fixed - Removed some cruft in OFS/content_types.py (an old data structure was being constructed but was going unused in favor of a newer structure used in conjunction with the mimetypes module). - (Collector #1650)Where the underlying object does not define its own '__cmp__()', comparisons of acquisition-wrapped objects fall back to comparing the identities *of the wrappers* . Fixed to unwrap the object (both, if needed) before comparing identities. - (Collector #1687 Products which register base classes for ZClasses typically defer creating them until product registration; the derived ZClass needs them to be available immediately after import. Deprecated 'ProductContext.registerZClass' and 'ProductContext.registerBaseClass' in favor of a new function, 'ZClasses.createZClassForBase' (because none of the machinery needed a ProductContext instance anyway). - (Collector #1355) Fixed overlapping HTTP POST requests in ZServer which could have been corrupted. Thanks to Jeff Ragsdale. - Undid a bug fix that caused the DateTime unit tests to fail. - Removed the requirement that an "access" file exist. "access" is now only needed to create an emergency user account. - Disabled the monitor port by default because, initially, there is no emergency user, and thus no password that can be used to protect the port. - Secured the hole that was patched by Hotfix_2000-12-08. - Disallowed object IDs that end with two underscores. - Caused PropertyManager to restrict id's the same way ObjectManager does. Zope 2.2.5 Bugs fixed - Corrected a mounted database connection leak. The reference to the sub-connection was occasionally being garbage collected, meaning the sub-connection never get closed and Zope eventually used up all connections from the pool. Zope 2.2.5 beta 1 Bugs fixed - Secured the hole that was patched by Hotfix_2000-12-08. - A bug in ZServer caused concurrent POST requests to overwrite each others incoming data, leading to 'AttributeError: data' errors. - The ZSQLMethods code to handle query caching had a bug that could cause a KeyError in certain cicumstances. - A dumb algorithm in the basic user folder interface caused a reverse dns lookup to be done for each user in a user folder if domain-based matching was being used. This could cause a real lag for sites with large user folders doing domain matching. - The "View" management page for Image objects generated an incorrect link to the image object (though it often happened to work thanks to acquisition). - Import only looked for files in INSTANCE_HOME/import. Now, if the file is not found there it looks in SOFTWARE_HOME/import. - Fixed situation where the Catalog would attempt to loop over a bucket as if it were a list, which won't work. This was reported by Steve Alexander with a patch (#1586). - It was not possible to access the 'manage_access' screen for a custom propertysheet on an instance. Thanks to Steve Alexander for this one as well. - The checkboxes on the "Add ZClass" form were being reset after every base class addition. - A WebDAV support bug that caused authentication to fail for MOVE and COPY (and possibly other) requests was fixed. - The WebDAV support was setting an extra HTTP 'Connection' header when Zope was running under ZSever, because ZServer didn't handle that correctly back when the DAV code was written. - Several other DAV-related fixes were made to address bugs discovered in the course of using Zope with several DAV clients. The DAV support now generates a faux creation date property to make Adobe GoLive happy. - A bug in state diffing in the Tree tag when a tuple was passed in instead of a list was fixed. - Corrected local role computation (Hotfix 2000-12-15). - Fixed protection of the update_data method of File / Image objects (Hotfix 2000-12-18). - The content_type argument passed to manage_addImage or manage_addFile was ineffective. - The Image.tag() and ZopeAttributionButton methods now return an image tag that is XHTML compatible; a space and a slash have been added. - Long running processes spawned via os.system() from Zope code could hang the request that spawned them due to inheritance of open file descriptors. Thanks to Dieter Maurer for a fix for ZServer to set the close-on-exec flag on ZServer sockets. - Some of ZServer's medusa underpinnings were outputting various 'WARNING' messages that should have been ignorable and caused confusion for new Zope users. The startup messages have been fixed to avoid spewing things that scare newbies unnecessarily. - SQLAlias objects were leaking when a database column was accessed via an aliased name. This was tracked down to a subtle bug in ExtensionClass. - A packing bug could, on rare occasions, cause corruption of transactions committed while the database was being packed. This corruption wouldn't show up until the database was scanned, either on startup without an index, or on subsequent packs. The actual data records were correct in most cases, but contained invalid file pointers to their transaction or other record data. The problem could be corrected using the FileStorage recovery tool, fsrecover.py. Zope 2.2.4 Bugs fixed - There was a mistyped permission in HelpSys.py ("Add Documents, Files, and Images" should be "Add Documents, Images, and Files"). - The caching code added for ImageFile objects had a dumb cut-n-paste error (fixed for 2.2.4 final). ImageFile caching was verified after the fix with the Cacheability Engine at: http://www.web-caching.com/cacheability.html - Fixed and tested a problem in the way that headers are extracted from the message text in the dtml-sendmail tag. Windows text editors could introduce ^M characters that threw off the rfc822 message parser if they were not cleaned off before parsing. Zope 2.2.4 beta 1 Bugs fixed - Zope 2.2.4 includes all of the 2.2.3 changes plus a fix for a bug introduced while fixing another bug that caused acquisition of permissions to fail (whew!). - Fixed a bug in computation of object roles (from permissions) that could cause roles to come from non-containment context. - ImageFile objects (e.g. icons in the mgmt interface) did not set or recognize HTTP caching headers, which caused some browsers to never cache them. - The function rfc1123_date was mis-formatting http dates. - The registerHelp() method of ProductContext objects would blow up if a product did not have a help directory. Zope 2.2.3 Bugs fixed - Resolved known condition in the Catalog where objects are not guaranteed to be unindexed before indexing happens again. Now objects are guaranteed to be unindexed correctly first. This should resolve all outstanding KeyErrors with Catalog, although currently broken Catalogs will require a reindex to resolve them. - Fixed a race condition and a possible memory leak in mounted databases. - Added the DTML Reference to the online help system. - Applied Toby Dickenson's patch that fixes aq_inContextOf. Now it is more thorough in discerning whether an object is in the context of another object. - Fixed keyword args for aq_acquire. - Plugged a memory leak that occurred when objects wrapped through acquisition but which do not have a "__call__" method are invoked. The arguments passed to the call would be leaked. - DateTime did not recognize timezone strings like '+0200'. - Security for Images and Files could not be changed due to an old name left in the permissions structure (__call__). - Removed some ancient references to Principia from the ZGadflyDA user interface. - Fixed bug in Client.py which opened filesystem file without using read-binary mode, which caused file corruption when uploading binary files using ZPublisher.Client from Windows NT. - The '_getOb' method of ObjectManagers did not correctly restrict access to private attributes (beginning with an '_'). - The XML import/export machinery was fixed to properly deal with 'long' datatypes. - Uploading new contents for a File or Image object without a content-type header caused the old (possibly now incorrect) content_type attribute to be retained. - There was an extra slash in the rewrite rule for using PCGI in single-threaded mode in WEBSERVERS.txt - The PARENTS list passed to the validate method of User Folders did not include the published object, which could lead to incorrect security handling in certain cases (this was the 2000-10-02 Hotfix). - (Collector #1687) Products which register base classes for ZClasses typically defer creating them until product registration; the derived ZClass needs them to be available immediately after import. Deprecated 'ProductContext.registerZClass' and 'ProductContext.registerBaseClass' in favor of a new function, 'ZClasses.createZClassForBase' (because none of the machinery needed a ProductContext instance anyway). Update OFSP and ZCatalog products to use this machinery, instead of 'registerBaseClass'. - The manage_edit of the Connection base class used by database connection tried to coerce the connection string to an actual string. This was problematic because some DAs (Sybase apparently) store the connection string as an HTTPRequest.record object. - BASE tags generated by Zope were not XHTML compliant. - Part of the PropertyManager interface incorrectly assumed a wrapped object. - HTTP date headers were generated by strftime in medusa; this meant that using a different locale caused medusa to generate invalide Date headers re: the RFC. The date header generation has been changed to use english even if a different locale is in effect. - Dates generated by WebDAV code also used strftime and were therefore made invalid when a non-english locale was in use. - Headings generated in HTML by StructuredText did not generate a closing paragraph tag. - StructuredText did not recognize URLs with ampersands in them. - A bit of the SecurityManagement code expected threads to be available, which broke the "single-threaded-mode" that some folks use with pcgi. - DateTime arithmetic was broken if you tried to subtract a DateTime from another DateTime where one of them was on daylight savings time and the other was not. - The infamous "__call__" bug that could surface in situations where documents called other documents has been stamped out. And there was much rejoicing. - The redirect target after adding a ZSQLMethod had too many path elements in it. Zope 2.2.2 Bugs Fixed - Applied Dieter Maurers patch to prevent log entries from being dropped when clients disconnect prematurely. - The manage_test method of database connection objects were not checking for RDB-format results, which caused the test queries to fail from the test tab of some older DAs. - Z2.py did not allow passing numeric UIDs for the -u option. - Fixed a bug in the dtml-var tag handling of the 'null' attribute. Before null would only work correctly if there was also a 'fmt' attribute. Now null correctly works with non-zero false values when there isn't a 'fmt' option. - Corrected a factory bug that prevented creation of ZClass instances within objects owned by users defined in a different user folder than the root acl_users. - Made ZopeAttributionButton method of Application object public. - Factory dispatchers inappropriately acquired ownership in certain cases, causing problems creating objects for users not defined in the root user folder. - The DestinationURL method of FactoryDispatchers did not work correctly if the dispatcher was obtained through the unrestrictedTraverse method. - Several calls to socket.bind() were fixed for Python 1.6/2.0 compatibility in PCGI and Gadfly. - The sqlvar tag could fail on the trailing 'L' of the string representation of longs when coercing to int or float. - Under some circumstances, versions were only partially committed (or aborted) in FileStorages causing objects to be left locked in a version with no way to unlock them. - FileStorages opened read-only incorrectly truncated data files when incomplete transactions were discovered. - Read transactions were aborted due to conflict errors. This was an attempt to catch write conflicts early, but was far too pessimistic. - The document_src method of DTML objects required a REQUEST and RESPONSE argument, making it difficult to use from Python code. They are now optional. - The "test" tab of SQL Methods did not include the rendered query in the error message if the test failed due to a database-level error (such as a malformed query). - A signed/unsigned buglet in the code for Splitter objects caused recognition of national characters to fail on some platforms (FreeBSD/x86). - A bug in ZODB database objects could cause ZEO clients to fail to start without deleting the ZEO client cache. - The 'lock' keyword argument to RESPONSE.redirect was not in the API documentation. - Some missing SCRIPT_NAME variables were added to the IMG SRC paths for the navigator pane. Zope 2.2.1 Bugs Fixed - Correction of ZClass-in-ZClass bugfix which prevented import of .zexp's containing Factories. - Made acquisition in the URL and XML-RPC work together. - A bug in FTP downloading was fixed. - A bug in the "views" screen for ZClasses could cause strange IndexErrors depending upon whether help topics were created. - Improved behavior when non-persistent objects are added to an ObjectManager. - Fixed an obscure bug that could cause Zope to crash when there were errors in computed attributes that depended on acquisition. - Fixed very obscure bug in database invalidation when committing from one version to another. This could cause source version data to fail to get updated. - Fixed a condition that prevented error messages from being displayed when performing a redirect. This means that standard_error_message documents that redirect now need to add a "lock=1" argument to the redirect call. - A bug in the stupid file logger consumed exception info for certain types of errors (such as failed product import). - A bug in integration with the new security machinery caused copy / paste operations to fail in certain cases. - Changed permission assignments for 'manage_editForm', 'manage', 'manage_main', 'manage_uploadForm', 'manage_historyCopy', 'manage_beforeHistoryCopy' and 'manage_afterHistoryCopy' to 'Change DTML Documents' (for DTML Documents) and 'Change DTML Methods' (for DTML Methods) which is more correct and makes view filtering work properly. - The medusa code that produced the server log file was writing the formatting the date in GMT but adding a local timezone identifier (+0100, etc.) It has been fixed to format the date using localtime. - The DTML "tree" tag did not understand the common shorthand spellings for name= and expr=. - The HTML4 formatting method on DateTime objects did not return the datetime in UTC as it was supposed to. - A bug in cgi.py caused an entire extra copy of large file uploads and input data to be kept in memory, defeating the work the module does to avoid doing exactly that ;^) - A bug in the cgi module could cause memory to spike on large inputs, as it kept a complete copy of the input in memory that was never used :( - A packing bug could cause data written by saved versions to be inaccessible. This is actually a bug that was fixed in Zope 2.1, but the fix never got applied to the main code tree. :( - The behavior of the tag and __str__ methods of images has changed slightly; if no 'alt' argument is given, it will use the title of the image (if defined) for the alt attribute. Otherwise it will generate 'alt=""'. Also, it now generates 'border="0"' unless a different value for border is given as a keyword argument. - The database control screen in the control panel did not work with storages that returned string "sizes". Some storages return strings to indicate units other than bytes. - Fixed the broken Product Help icon. - Fixed ZClass view help setting, thanks to Jeff Sasmor. Zope 2.2.1 beta 1 Bugs Fixed - FileStorage-based databases became corrupted when their size grew beyond 2GB. - FileStorage-based databases could be corrupted if transaction descriptions or transaction extended meta-data exceeded 64KB in size. - Some relational database adapters use an older version of a results class in Shared.DC.ZRDB. The RDB-format based results class was missing a needed security assertion to allow access to result data. - Missing security assertions in the ZDOM module caused unauthorized errors for things trying to use ZDOM (such as XMLDocument). - A bug in the registration of help content caused unnecessary transactions to be written at startup. - The machinery for guessing content types ignored default values passed in explicitly. - A problem in the Find machinery removed the acquisition context from subobjects, causing permission checking to fail. - There was a hard-coded unix path separator in the logic for initializing special dtml objects from files. - The ordering of tabs on some management screens was changed inadvertantly during the 2.2 development cycle. The ordering has been tweaked to conform more closely to earlier releases. - The strftime() method of DateTime objects was changed so that it formats based on the current timezone representation of a given DateTime object. This is now also the case when formatting DateTimes from DTML using the fmt="..." construct (which uses the strftime method under the hood). Previously, strftime converted to GMT, which was inconsistent with other formatting methods and caused problems for people. Those who actually do want the date to be formatted in GMT can use: myDate.toZone('GMT') to get a version of the object represented in GMT and then call the formatting methods on that object. - Permissions were not properly defined for the new history vew in DTMLMethods and DTMLDocuments. This caused many useful features to be available only to managers and others to be available only to those with undo priveledges. A new permission, "View History" has been added for viewing historical changes and you can copy historical versions to the present if yo can edit DTMLDocuments or DTMLMethods. - The new traversal method, restrictedTraverse was private, but should have been public. It also didn't check access to the root object when an absolute path was used. - The roles (and only the roles) "Manager" and "Anonymous" had the "Access contents information" permission on the root application object regardless of permission settings made through the management interface. - A ZCatalog bug which was symptomized by a "KeyError" during a searchResults query was squished by changing some exception handling in the cataloging machinery. - Version's chose their names based on absolute_url, which depends on access method, rather than physical path. - A fix for socket.connect() calls in preparation for Python 2.0 was accidentally applied to a wrong part of Client.py in ZPublisher. - The "distribution" tab for ZClass-based products has been changed to allow creation of redistributable products. - DAV property sheets were not correctly associated with permissions, which cause Unauthorized errors when iterating over propertysheets under the new security model. - Some patches to PCGI added in the 2.2 development cycle to support things like syslog connectivity were backed out due to problems on certain platforms. The patches may make it back in the future after the issues are resolved and broader platform testing can be done. - The new security machinery did not allow REQUEST to be accessed when it was acquired. This didn't prevent access to REQUEST, but made the access eccessively expensive. - Fixed the display order of the Zope tutorial topics. - Fixed the help system menu frame to not use standard html header and footer. - Products that have not been updated to define an 'initialize' method to do product initialization were not filtered correctly by permission in the filtered object add list. - Exception logging was added to catalog to catch "shouldnt happen" sorts of events. - Text indexes did not recognize boolean connectives (and, not) if they were mixed-case. - Fixed Tutorial bug that caused win98 to crash with adding a tutorial. - Fixed ZService.py to avoid overwriting existing service start parameters at install time on win32. - The deprecated alias getSize() was added back to Image and DTML objects (existing products still depend on it). - Cleaned up a DocumentTemplate namespace issue. Fixed QuickStart. - Removed old validation code in cDocumentTemplate. - In ZCatalog, it was possible for stemming on single-word search queries to be handled incorrectly and produce no results. - A problem in the url traversal machinery caused strange errors in cases where the root of the site was requested but had no index_html instead of the expected Not Found error. - ZCatalog objects did not implement the searchable object interface correctly. - XML (database) import sometimes failed, especially with ZClass instances. - XML (database) export didn't properly handle strings that contained the string "\n". - The tag insertion machinery failed if the request result had content-type with a charset attribute (thanks to Dieter Maurer). - A long-standing bug that caused a "resource not found" error when trying to instantiate a ZClass inside of another ZClass was fixed. Hooray for Chris McDonough :^) - In certain circumstances it was possible for the manage_afterAdd and other manage_after* methods to get into an infinite loop by essentially acquiring themselves in cases where subobjects did not implement those protocols. - The sqlvar tag produced misleading error messages when an expression caused an exception during rendering. - RFC1123 dates were not formatted correctly in the webdav.client implementation. - Made the ObjectManager class inherit from Traversable (which exposes unrestrictedTraverse). ObjectManager-type items which didn't inherit from SimpleItem.Item had problems with Management.Tabs producing the right views for manage_options before this was in there. - FileStorage-based databases got into an unusable state if available disk space was exceeded. This problem could lead to database corruption. Zope 2.2.0 Bugs Fixed - The add form for MailHost objects described fields that are no longer needed (descriptions have been removed). - User databases did not correctly handle situations where a browser sends an authentication token naming a user that can't be found while accessing a publicly available resource. - Minor bug in ZCatalog which prevented deletion of indexes fixed. - XML-RPC requests could not be completed via https. - Several uses of multi-argument append() calls (that will be illegal in Python 2.0) were fixed. - The UNLOCK method for DAV support was missing some needed argument declarations. - Corrected some error handling in the database connection code. - Fixed support for strftime day-of-week formatting in DateTime. - Fixed Image content-type detection to also use the binary "nonces" in the file to determine the real content-type. This should catch GIF/JPEG/PNGs that are uploaded without the correct information associated with them. - Added missing permission declarations to make the methods of PropertySheets objects accessible. Zope 2.2.0 beta 4 Bugs Fixed - The History tab is now protected by the 'Undo changes' permission, and setting this permission actually has effect now. - A bug in validation in filtered_manage_tabs made tabs misbehave in some cases. - Fix an AttributeError bug in the Lexicon objects used by ZCatalog. - FileUpload objects did not have the required security assertion to allow access to attributes such as filename and headers. - Broken objects weren't deletable due to a recent change in the object deletion logic added to support mountable databases. - Some formatting bugs in the status method of the ZServer HTTP server were fixed (thanks to Jeff Rush). - The _read_data method of Image and File objects didn't correctly handle being passed an actual file object. - A new behavior in 2.2 is that after calling REQUEST.redirect the HTTP status can no longer be changed. This caused problems for some people who were calling redirect() and then trying to set the HTTP status to 301 Moved Permanantly instead of the default 302 Moved Temporarily status. To make this possible, a new optional 'status' argument is now recognized by the redirect method so that RESPONSE.redirect('/foo', status=301) can be used - Corrected the text of some kinds of 'unauthorized' error messages. - Fixed a bug that caused dtml-try to throw a KeyError. Related to the recent changes made to DT_Util.namespace(). - Packing to a time earlier that a previous pack could lead to serious data lossage. - Attempts to install product information into the Zope database on startup made no sense and could cause problems for ZEO clients. Product adtabase updates are suppressed when the ZEO_CLIENT environment variable is set. - A missing security assertion made DTML code that tried to create instances using manage_addProduct['foo'] constructs fail. - Fixed a couple typos in the Tutorial thanks to Alastair Burt. - Fixed a Javascript error in the Tutorial thanks to Luke Tymowski. - Revised online help content quite a bit. - Revised API docs quite a bit. - Fixed a bug in Logging docs thanks to Andy McKay. - Added code that will reveal bugs in calls to DT_Util.namespace(). - The way that the values of selection and multiple selection properties were looked up on the properties form was changed so that the values can now be either properties on the same object or found through the DTML '_' namespace to support situations where the selection value is acquired or otherwise not findable in the object's properties. - The If-Modified-Since handling in Image and File objects did not gracefully handle malformed date strings in that header (which seem to be sent by certain proxy servers). Malformed date strings are now ignored as if the header was not present rather than raising an error. - Corrected a recent problem with boolean properties. - Permissions were not being set properly on factory methods (constructors) loaded from Python (disk) products. This bug existed in 2.1, but was not very visible until recent fixes were made to the security machinery. - The ownership management code incorrectly was willing to delete ownership information in the class rather than only in object instances. - A bug that made packing always complain that you were trying to pack to an earlier time than a previous pack was fixed. - The security context stack wasn't handled correctly for DTMLDocuments. - DateTime objects did not support GMT(+/-)xx30 timezones. Zope 2.2.0 beta 3 Bugs Fixed - Handling of "broken" products was, er, broken :) This is why some folks reported seeing objects "spontaneously change into Folders". In reality, the Broken objects that should have been created in place of objects whose Product was missing or broken weren't behaving correctly due to an interaction with the new overridable __getattr__ support in cPersistence. - FTP support was broken for Folders containing Broken objects. - ZServer returned None as the HTTP version if no version was given in the HTTP request. - It was possible to rename or move Versions, which was plain bad. - A bug that caused SQLMethods to have problems unpickling was fixed. This bug also made it impossible to add Zope Tutorial instances. - A bug in the lookup order for Product extensions was fixed. - Fixed the tutorial to ensure that a gadfly directory has been created before attempting to import the tutorial info. Zope 2.2.0 beta 2 Bugs Fixed - Fixed Tutorial bug that caused win98 to crash with adding a tutorial. - Fixed ZService.py to avoid overwriting existing service start parameters at install time on win32. - The deprecated alias getSize() was added back to Image and DTML objects (existing products still depend on it). - The "Host" headers sent by ZPublisher/Client.py did not include the port number if given. - Cleaned up a DocumentTemplate namespace issue. Fixed QuickStart. - Removed old validation code in cDocumentTemplate. - Added checks to only allow legal Python name characters in ZClass ids. - Selection and multiple selection properties were displayed incorrectly on ZClass propertysheet forms. - getPropertyType was not supported for PropertySheet objects. It appeared to work but always returned None because it was being acquired :) - The 'send' method of MailHost objects was broken if explicit to, from or subject arguments were given. - The constructor for MailHost objects was changed to have a non- backward compatible signature in the alpha period. The 'localhost' and 'timeout' arguments were added back to the MailHost constructor for backward code compatibility for 2.2 beta 2 (though these args are now effectively ignored). - An insufficiently protected method deep in DocumentTemplate was fixed to prevent a security issue that could allow unauthorized changes to the sources of DTML objects. - Insertion of the BASE tag in responses violated the HTML 4 spec, which states that BASE must come first in the element of a document so that relative references thereafter will work. It now conforms to the spec. - A bug in the SendMail tag if an existing mailhost was named has been fixed. - New and improved content has been added to the help system. - Some problems with certain management screens having wrong help or no help have been fixed. - A bug in PropertyManager allowed properties defined in products as read-only to be changed. - Added an entry in the content_types registry for RealAudio file extensions. - At some point z2.py stopped accepting "-P 0" as a valid option. That has been fixed. - Fixed a bug that caused instances of ZClasses which use DTMLMethod as a base class to have an ID of "". Unfortunately, existing instances will still have the wrong ID. The effect is it's not possible to use the clipboard on them. They will need to be re-created. - Fixed a bug in PermissionMapping that could in a certain case cause determination of whether an object is a ZClass instance method to throw an exception. - A few tweaks were needed to the way that SQLMethods made use of the new security machinery. This fixes the authorization problems some users were seeing when calling SQLMethods from inside of other SQLMethods. - Medusa did not handle absolute URLs given in HTTP commands quite correctly. It turns out that WebTV (and possibly others) send such commands, and the HTTP spec allows them, so we now support it. Zope 2.2.0 beta 1 Features Changed - Split the old CHANGES.txt into two files. CHANGES.txt will now contain only change information for the current Zope release. Change information for older versions is now in the file HISTORY.txt. - Added basic internal support for mountable databases. A separate product will still be required to make use of this ability. - Added a new "history" tab to selected objects (DTML methods and documents for now) that provides access to previous versions through the web. - The property management screens now include the types for existing properties. - Added support at Python level for user-defined __get/set/delattr__ methods on persistent objects. These have essentially the same semantics as for Python, except that overridden __set/delattr__ methods must explicitly signal changes that should be persistent. - Better error messages and syslog capabilities were added to pcgi-wrapper (thanks to Jeff Rush). - Calls to the ZLogger system were added to the user authentication process to help sys admins diagnose login problems (thanks again to Jeff Rush). - The Undo view shows *only* transactions performed in the place where undo was performed. (As before, it also shows only transactions performed by users defined in the place where the undo user is defined.) - The way that transaction logging is done was changed to make transaction logs based on "physical" object and user paths. This was necessary to make undo work properly in the presense of virtual hosts. - A number of hooks have been added and changes made to support products that implement virtual hosts. Keep an eye on http://www.zope.org/Members/michel/Projects/Interfaces/ImplementingVirtualHosts - The distributions now have an Extensions directory by default. Bugs Fixed - Added type checking to the constructor and edit methods of builtin Zope object types to prevent inappropriate passing of acquisition-wrapped objects for standard attributes (like title, etc.). A similar restriction was added to prevent the adding of wrapped objects as properties. - Fixed a naming bug in MailHost that caused simple_send to fail. - Fixed a buglet in the rename form that caused the form to submit to the old manage_renameObject if the user just hit the enter key in the browser on a single item rename. - A bug that caused the tabs to disappear after performing cache management operations was fixed. - A problem that prevented multiple Zope instances from running at the same time on win32 was fixed. - An algorithm in medusa's max_sockets.py code was updated to improve startup time on certain platforms. - The behavior of manage_editProperties on PropertyManagers and PropertySheets has been fixed so that existing properties that are not found in the update request are not reset to empty values. - A problem that disallowed access to the individual records in a result set from a SQL method was fixed. - A problem that caused the query template for an SQL method to be rendered twice when used from the "test" tab has been fixed. - Fix for bug #1270: Netscape Image Problems. Added casts to long to avoid an overflow error caused when trying to convert dates sent by certain versions of Netscape in the If-Modified-Since header. - Corrected DateTime.py to recognize years specified as 00 through 69 as 2000 through 2069. - A bug that made objects named in dtml namespaces always fail security validation was fixed. This was noticed by folks trying to use manage_tabs from ZClasses who hit the bug because the manage_tags document uses a namespace: - A change to pcgi_publisher was made to support FreeBSD, where send() will send only 8192 bytes at a time. - A bug that broke import in the alpha was fixed. - A bug in the transaction handling logic could cause infinite loops if objects were registered incorrectly. - A bug in the transaction machinery could cause objects to be aborted multiple times if errors occurred. - Logging via syslog now correctly captures the process id. - Adding User Folders and MailHosts from the management screens didn't redirect to the correct URL after adding the object. - A form problem that prevented joining or leaving versions has been fixed. - A more reasonable default content type is now used for HEAD requests on DTML objects if the object does not have a content_type attribute or a file-extension-like id that can be used to determine content type. - HTTP HEAD handling was inconsistent for collections, depending on whether the "default document" (index_html) was acquired or not. Zope 2.2.0 alpha 1 Features Changed - Added a new security policy architecture and object ownership to address the server side trojan issue. The new architecture cleaned up various places where Zope code did authorization checks. A side effect of the new policy is that it is a bit more strict than it used to be - objects without explicit protection to which access was previously allowed will now be denied. There is a new declarative method for providing access to such objects, which has been applied to certain previously unprotected Zope objects that DTML writers are accustomed to having access to. See the SecurityPolicy wiki pages on Zope.org for further information. - Added a new online help system. Help is now available for standard Zope objects. Zope developers can add help for their Python Products and Control Panel Products. See HELPSYS.txt - Added logic to increase the Python interpreter "check interval" that should provide at least a 20% performance improvement for most Zope sites. Also added a new -i option to z2.py so that Zope users can pass in alternate values (the default is 120) for the check interval. This lets users experiment and tune the interval for the best results in their particular system environments. - The message returned when an empty result set is returned from a ZSQL Method has been clarified. - In the load_site utility: o Added logic to try and figure out a path to ZPublisher if one was not provided. o Added an option, -I, to automatically add an index_html method that redirects to index.html or index.htm. - Record objects returned by ':record' form-marshalling tag have been enhanced to act like mapping objects. - The ZServer startup script, z2.py, now allows all servers to be disabled with a single option, - The ZServer startup script, z2.py, now allows all multiple HTTP, FTP, or monitor servers to be run. - The ZServer startup script, z2.py, now allows separate IP addresses to be specified for each HTTP, FTP, or monitor server. - Added 'urlparam' attribute to tree tag to allow propagation of extra parameters through tree URL's (Collector #1045). - When renaming objects, the old id is provided as the default value for the new id. - Allow non-sliceable keys in PersistentMappings (Collector #1064). - Use smtplib in implementing MailHost/sendmail (Collector #1005). - Allow call to get() w/o explicit default in PersistentMapping (Collector #1182). - Allow rename of multiple items (Collector #1065). - Allow expr syntax in for type, disposition, encode, name, filename; add skip_expr (Collector #892). Bugs Fixed - Fixed problem with extra PermissionMapping objects being in the aq_parents hierarchy during traversal when a method was being accessed a ZClass defined instance method. - Fixed ZCatalog 'Find to ZCatalog' with expressions problem. - Fixed broken links in ZCatalog 'Cataloged objects' view. - FTP directory listings were not sorted. - Improved FCGI support in ZServer. - Improved ZServer streaming, including fixing chunking and keep-alive support. Chunking is now done by default with HTTP 1.1 clients. Fixed possible problems with large responses. Thanks to Toby Dickenson. - ZServer's FTP server no longer binds to all interfaces. - Fixed a bug which caused File objects of type 'text/html' be have the wrong Content-Length set. - A bug in SQL methods could caused results to be returned without an acquisition context. This could cause methods provided via ZClass brains to be inaccessable due to the dependence of ZClass method security on a correct acquisition environment. - Fixed tree tag so that expand_all and branches_expr play nice (Collector #919). - Suppressed "private" names from catalog indexes/meta-data (Collector #1076). - Allowed strings with linebreaks as data in Gadfly queries (Collector #972). - Added note to WEBSERVER.txt pointing out security issues with REMOTE_USER mode (Collector #733). - Fixed colspan counting in tree tag (Collector #913). - Logging enabled with the -D option to the z2.py script did not include traceback information and was not performed during Zope startup. This made finding startup errors difficult. - The logic for reparsing the dtml files used for the Zope management screens while in development mode was broken - it was reparsing system dtml files on every request. - Correct DateTime docstring to remove spurious 'negative offset' language (Collector #1074). - Clean up pointer cast to suppress compiler warning in zlib.c (Collector #1190). - Guarantee null-terminated buffer in Record_init() so Record_compare() doesn't UMR (Collector #1012). - Clean up use of PData for large images (Collector #1061). - Fix overflow exception in statistics of (Collector #1089). - Check for invalid characters in whole id, not just first character (Collector #1131). - Prevent unintentional exceptions from blocking manage_beforeDelete; added OFS.ObjectManager.BeforeDeleteException to allow an object to veto its deletion (Collector #1183). Zope 2.1.6 Bugs Fixed - Some bits from the 2.2 line inadvertantly got into the ZRDB package during the SQL Methods update which caused errors for certain database adapters. - A fix to the logic in Acquisition for aq_acquire caused a bug in the handling of inner ZClasses. Zope 2.1.5 Bugs Fixed - Fixed a problem with the permission declarations for the Image and File classes. The problem made it impossible to change the 'View' permission for File objects. - Added logic to setgid() to the user's primary group if z2.py is run by root. - Fixed a bug in TimeStamp objects that produced a wrong date/time representation for bobobase_modification_time. - Changed _checkId in ObjectManager to disallow REQUEST as an object id. - Fixed a bug that could allow someone with a lot of Zope zen to change the apparent AUTHENTICATED_USER to access things that they shouldn't. - Fixed a bug in ZServer that could cause server hangs under certain heavy load conditions. - Fixed a remaining '.' form target that we missed in the Zope rename form. This caused rename to fail for Zope installations running behind Netscape servers. - Fixed a potential buffer bug in PCGI reported by Larry Luther. - In the load_site utility, binary files were misshandled on windows. - Fixed manage_renameObject to be willing to get the REQUEST via acquisition, making it easier to use from DTML. - Changed the expireCookie method of FTPResponse to fail gracefully when attempting to delete a cookie that does not exist. - Fixed a bug that caused a traceback when all members of a multiple select property were deselected. - The _validTime method of DateTime objects rejected time values with fractional seconds between 59 and 60 (which caused problems with some database date conversions). - Changed id checking logic to disallow '/' in an object id. - Structured Text had problems with '~' characters in URLs. - Fixed a potential security hole that could allow users with permission to add Folders and edit DTML (and a who have a lot of Zope zen) to get access to things that they shouldn't. - The internal templates used by SQL methods weren't correctly applying the same access control constraints as standard DTML objects. - Fixed ZopeAttributionButton to open in the top-level of the web browser. - Fixed a problem with using the "scale" arguments to the tag method of Image objects. - Fixed a problem in the Acquisition module that could cause objects accessed via aq_acquire to not be wrapped correctly. - Fixed If-Modified-Since header handling for Images and Files. Zope 2.1.4 Bugs Fixed - Removed the "feature" that allowed the REQUEST object to be traversed through the web. While useful for debugging, this could be a security issue. Zope 2.1.3 Bugs Fixed - A race condition in the logic for managing Zope database connections caused Zope to hang on very busy sites. - A bug in the packing code that caused records to be nreadable after: o someone did work in a version o Someone did an (unrelated) undo o the version was committed and the database was packed to a time before the work was done in the version. - Fixed a bug that caused packing to raise an error in the following situation: o someone modifies and then deletes an object in a version. o they commit the version o the database is packed between the time the object is deleted and the time the version is committed. - Fixed a bug that caused Zope to sometimes hang instead of shutting down or restarting when accessed over a fast network. - It wasn't possible to use a ZClass instance as a method of a ZClass. Zope 2.1.2 Bugs Fixed - Thanks to Kevin Littlejohn's sleuthing, a sizable problem in the security machinery in DTML has been brought to our attention and resolved. The problem is most acute in situations where untrusted people can edit DTML documents or methods. Zope 2.1.1 Bugs Fixed - Conflict errors (multiple threads trying to modify the same object) were not handled correctly for requests with bodies (e.g. POST requests). - ZODB FileStorage index files could become out of date after an undo. This could lead to strange results and apparent database corruption if Zope was shut down ungracefully and restarted. Now index files are removed when undoing records. - Saving or discarding versions failed if a transaction that added a new object in the version was undone. - There was circumstantial evidence that a failure of an OS to correctly delay writing to data to disk after Zope had written it led to a corrupted database. Zope now makes an 'fsync' system call (where available on Unix systems) to force data to be written to phusical storage when transactions are committed. - Some applicataions were writing empty transactions. The FileStorage now checks for and avoids writing empty transactions in this case. - Multiple-selection properties didn't work in regular property views, even though they did work in ZClass property sheets. - The Win32 WISE installer did not create a default Zope.cgi PCGI resource file. - The Win32 WISE installer script is now included in the /inst directory of win32 installations by popular demand. - An import problem caused the name "BTree" to be incorrectly overwritten in TextIndex.py and UnTextIndex.py in the SearchIndex package, which caused problems for some third- party product authors. - There were a number of problems with the load-site utility (utilities/load_site.py): o A recent change caused HTML files to have their headers and footers replaced with var tags for standard headers and footers. While this is sometimes very useful, it is sometimes disastrous. This feature is now enabled with the -D option. o Handling of files with a '.dtml' extension was broken. These are now handled correctly. Files with '.dtml' suffixes now get uploaded as methods. o A workaround for old sites that had a bug in Document (aka DTMLMethod) upload has been disabled. This workaround is now enabled when the -9 option is used. The following bug fixes were accidentially excluded from 2.1.0: - Redefining the "views" on a ZClass could incorrectly affect the views available on instances of base classes of the ZClass. - The new 'url' option to DTML 'var' tags did not work properly for DTML and other callable objects. Zope 2.1.0 Features Changed - Enabled preliminary support for FastCGI to ZServer. This allows Zope to communicate with web servers that support FastCGI. Use z2.py's -F switch to configure FastCGI support in ZServer. See WEBSERVER.txt for more information. Bugs Fixed - Redefining the "views" on a ZClass could incorrectly affect the views available on instances of base classes of the ZClass. - The new 'url' option to DTML 'var' tags did not work properly for DTML and other callable objects. Zope 2.1.0 beta 2 Features Added - Added Oleg Broytmann's patches to utilities/load_site.py that provide automatic parsing of title and property information and inclusion of std headers and footers. - Added Martijn Pieters' patches to DT_Try.py to implement try/except/else and try/finally constructs. - Virtual Hosting and SiteObjects. Virtual Hosting allows a server that can multi-host, like Apache, to serve different hosts from different Zope folders. 'SiteObjects' act as a placeholder for a virtualhost. Bugs Fixed - Some problems with the product distribution machinery that prevented product distributions from installing correctly have been fixed. - Description lines on the Undo screen were not HTML quoted, which could cause problems rendering the Undo log if any of the descriptions contained stray HTML tags. - Bugs in the way that ZClass instances were pickled have been fixed. - Some long-standing indentation problems with the tree tag have been fixed. - The Zope database lock file was not properly closed. - Linux and Solaris binary distributions did not contain the python _locale module required for locale support. Zope 2.1.0 beta 1 Features Added - The ZCatalog now supports relevance ranking. Instead of search results being returned in an essentially random order, results are now sorted by 'score'. For text indexes, the score is the number of occurrences of the search term in a document. The score can be accessed by an attribute on Catalog results 'data_record_score_' and 'data_record_normalized_score_' contain the score and normalized score for the result object. - Added Keyword Indexes to the Catalog. Keyword indexes allow you to index a sequence of 'keywords' as an atomic property of an object. This is useful for buiding catagorical hierarchies. - Z SQL Methods arguments list is now a TEXTAREA widget to ease editing long aruments lists. - SQL Database Adapters now have a 'test' tab where arbitrary SQL may be entered and executed immediately through the connection object. DTML is not supported in this tab. - SQL Methods now support an option 'op' parameter on the DTML method which allows you to choose the operator used for comparison. - WebDAV support has been modified to support MS Office 2000 applications. The changes _emulate_ aspects of DAV level 2 locking but do _not_ provide actual locking support at the present time. - Files and Images now can be served without loading themselves into memory. This is automatically done for large Files and Images. As a by product, RESPONSE.write now works better in ZServer. - Most if not all of the default DTML generated by Zope and its built-in products is now generated in the new syntax. The DTML sources for Zope's built-in management interfaces have also been converted to the new syntax. - DTML entity references can now include dot-separated var tag *options*, as in: '&dtml.url_quote-foo;' - There is a new 'url' option on the DTML var tag that causes absolute_url to be called on the displayed value. This allows entity references like: '&dtml.url-foo;' - There is a new 'url_quote_plus' option on the DTML var tag that acts like the standard url_quote option but uses '+' to encode blank spaces. - There is a new 'missing' option in the DTML var tag that allows you to provide a value to be used if the variable is missing, rather than raising a KeyError. This will work in the entity reference syntax as well. If it is used without an argument, it will provide an empty string. - The DTML built-in function, getattr, now accepts an additional argument providing a default value. - The tag() method of Image objects now support optional 'scale', 'xscale' and 'yscale' arguments that will automatically scale the output height and width tag attributes. - The Products folder in the Control Panel now has an 'Import/Export' tab to allow easier importing and exporting of Control Panel Products. Features Changed - ZCatalog uses subtransactions to keep memory consuption low at the expense of indexing speed. Subtransactions can now be disabled through the management interface for use with non subtransactions compatible objects like ZSQL Methods (or for raw speed if you have lots of RAM). - Added a coptimizations module. The first optimization is to provide an implementation of persistent_id in C. This routine, which is called extremely often while pickling was found to be a significant bottleneck. - More flexible cache management methods were added to ZODB Connection objects so that apps that want to move objects out of memory can force aggressive cache behavior. - Added the beginnings of internationalization support to Zope. The Zope startup script (z2.py) now supports a "-L" option that can be used to pass a locale name (such as "en" or "de") to Zope at startup. If specified, Zope will attempt to set the locale using the locale module if it is available. This change will allow Zope to better handle international characters in a number of places such as searching and indexing. - Logging from ZServer modified to produce Common Log Format in a Combined format, where user-agent and referer are also logged. - xmlrpclib was updated to relect the current version (0.9.8) from Pythonware. - Add button returned to the management interface, although the Javascript auto-select is still in place. This resolves some problems with browsers on certain platforms. - Added small explanation on Image/File add form that if the 'id' is not provided, it will be autogenerated based on the file name. Bugs Fixed - The build procedure for source checkouts now sets the directory structure permissions to something (775) readable by anyone, including, e.g., the web server running under a different user id and group than the installer... - A bug in the DTML tree tag that prevented expand_all from working correctly has been fixed. - ZCatalog now actually saves memory when using subtransactions. Plugged a couple memory leaks doing this. - Added ALT text fixed nested anchor tags produced by the Tree tag to ensure that valid HTML is produced, and fixed the reverse option to the Tree tag. - A bug in FTP cookie authentication was fixed. - Fixed a bug that caused External Methods to be reloaded every time they were used in development mode. - Fixed a permission typo in ZCatalog and a sort_on bug. - Fix for missing REQUEST in a method signature in the CatalogAwareness module. - Changed to propagate errors raised by _begin for thunked database adapters. Also changed completion code to check registered status and avoid completion logic if not registered, which would be the case if _begin failed. - A memory leak in cPersistence has been fixed, and a bug in object deactivation that prevented deactivation of BTrees was fixed. - A problem in BTrees that caused unintended db writes on Catalog searches was fixed. - Some unnecessary debug mode overhead eliminated - external methods are now automatically reloaded only when the .py files are changed. - A thread-safety problem with PCGI handling was fixed. - A bug that caused installation of through-the-web product distributions to fail. - A bug in the handling of long int properties was fixed. - Some bugs in the generation and verification of CRYPT based passwords was fixed. - A bug in the earliestTime method of DateTime objects was fixed. - A bug in determination of the ZServer version string was fixed. - Height and width detection for some PNG images did not work properly. - A bug that caused proxy role machinery to ignore local roles was fixed. - Added a fix for PCGI to make sure binary IO is used on win32. - A javascript fix for IE5 was made for the folder add list. - A bug in sequence multiplication in DTML was fixed. - A ZServer bug that prevented running multiple Zope2 installations on non-posix systems was fixed. - An External Method error handling bug was fixed. - The local 'Owner' role could be overwritten when an object was copied, moved or renamed. - Documented and fixed the creation of 'selection' and 'multiple selection' properties. - SECURITY: Fixed bug that allowed users with local roles gain priviledges on acquired objects. getRolesInContext did not used the correct context, as defined by the inner-most object wrapping. - SECURITY: The permissions for Find support methods were not being properly initialized. - SECURITY: Viewing existing user no long displays current password, only space for putting in the new password. This will be more critical as we move toward fully encrypted passwords. - SECURITY: A bug that prevented setting permissions on External Methods in ZClasses was fixed. Zope 2.0.1 Bugs Fixed - Fixed a bug which allowed unauthorized options to be displayed when adding a Folder. Now you only see options to add an index_document and/or a user folder if you have adequate permissions. Zope 2.0 Features Added - Database conflict errors are now logged. Although conflict errors are normally handled by retrying requests, frequent conflict errors can degrade performance and should be dealt with by redesigning applications. Features Changed - The process id displayed in the control panel now inclused the thread id, in parentheses. - Upload of image or file data *always* used sub-transactions. Unfortunately, database adapters don't support subtransactions. Images and files now use subtransactions only of the image or file is greater than 128K bytes in size. Also, subtransactions are not used if data are uploaded as a string by using the ':string' ZPublisher type in the upload form. More advanced DA's could, and probably should support sub-transactions. Bugs Fixed - Attempting to cut an object then paste it into itself or one of its subobjects would cause a recursion problem. - Certain ZCatalog reindex operations could fail due to an unintended transaction abort. - Database connections were leaked when database conflict errors occurred and handled by reexecuting requests. When 7 connections leaked, the site would accept no more web requests. - The transaction-integration for relational databases didn't work properly. Aborting Zope transactions didn't abort relational database transactions. What's worse, transactions were not committed after a Zope transaction using a relational database was aborted. This explains the reported problems with Gadfly databases and transactions. - Fixed a bug which prevented ZPublisher.Client from uploading files to ZServer. - Added some logic to the win32 binary installer to fix some dll loading problems people were getting on international versions of NT. - Fixed a bug in cDocumentTemplate that could cause AttributeError instances to be leaked in certain situations. - 'reverse' option for the 'in' tag as well as the 'tree' tag has been changed to deal with a copy of the data, rather than the original. The database was being modified in place. The side effect is that in large list situations (with ZCatalog for example), sizable memory usage may result. - Missing values (as returned from SQL methods) were output as empty strings by the DTML var tag, even when a numeric format (e.g. "%5.3f") is used. Using a numeric format with a missing value should generate an error, unless the null attribute is used. Also, the null attribute had no effect when outputing missing values. - It was impossible to map some permissions to themselves, or to other inherited, but not registered permissions in products and in class methods. When this failed, a silly error message ("Waaa" ;) was output. - SQL methods had an unused method, getFindContent allowed anonymous users to read SQL method source. This method was renamed to PrincipiaSearchSource, and protected with a permission. - SQL methods could not be found with the find mechanism by searching their source. - The PrincipiaSearchSource method of DTML documents and methods was not protected, so anonymous users could read their source. - Fixed a bug in ZOM which caused getElementsByTagName to fail when using XML Document. Thanks to Andrew Kuchling. - It was possible to hijak 'self' arguments for DTML and External methods with request variables. This was especially problematic for XML-RPC, which uses positional arguments. - It was difficult or impossible to call External methods with 'self' arguments from XML-RPC. Zope 2.0 beta 6 Features Added - The control panel process id display now also includes the thread id. - The Zope start script, 'start', is now written in a location-independent manner, making it a little easier to move Zope sites around after installation. Features Changed Zope 2.0 introduces a new HTML DTML syntax that, among other things, allows simple 'var' tags to be entered using the entity-reference syntax, as in:: The entity reference syntax is mainly intended for use when a var tag is used to insert text into an HTML tag. In these cases, the text needs to be html-quoted. For this reason, use of the entity reference syntax now implies html quoting. For example, the DTML snippet above is equivalent to:: (Note that when inserting text to be used in a URL, as in an A tag HREF attribute, we need to url-quote the text. Zope 2.1 will provide a means to do this with the entity-reference syntax.) Bugs Fixed - Missing values were not formatted correctly with numeric formats and did not work properly with the 'null' attribute of the DTML 'var' tag. - It wasn't possible to map some permissions of objects in Products. For example, it wasn't possible to enable a permission by mapping it to itself. Attempts to do so lead to a rather silly uninformative error message. - When providing default values for input fields, few of the built-in DTML methods used html quoting. This lead to incorrect behavior when default values included markup or quotes. - View filtering failed to show tabs for views in some cases where it should. - Single-character variable names were'nt allowed in the entity-reference DTML syntax. - DTML batch processing didn't work properly. - The DTML functions, '_.getattr', and '_.hasattr' incorrectly acquired attributes even when explicit acquisition was used. - Image dimensions were not autodetected for PNG 1.2 files. - The "View" view didn't display correctly for images with ids with characters that had special meaning in URLs. - The 'optional' attribute in the 'sqltest' tag had no effect. - With browsers (like lynx) that correctly implement RFC 1867 for file-upload support, non-file fields in file-upload requests were treated as file fields. Zope 2.0 beta 5 Features Removed - The help system, which was primarily geared to DTML programmers was removed. Bugs fixed - Database packing failed in certain cases where the data set contained many version commits. - Database packing generated spurious error logs indicating that there were bad references for objects created in ncommitted versions. - A memory leak due to errors in managing request data. - Zope sites became unresponsive (a.k.a. "Hung") due to an error in managing ZODB 3 database connections. - Only the top-level folder's standard_error_message was used. - There were thread-safety and resouce consumption problems in the implementation of thread-safe regex objects. - Fixed bug where ZCatalog's not in the root folder did not recognize the proper URL of objects using the 'Find' machinery. This caused all the objects to mysteriously disapear when you updated the catalog (because their paths were wrong). The Catalog now works correctly anywhere in the object hierarchy. - Fixed a bug that caused error messages to be formatted incorrectly if the standard_html_header did not begin with an HTML tag. - The undo form was accessible even to people who couldn't do undo. This made it look like we were allowing undo in cases where we were'nt supposed to. - The undo form did not restrict undoable transactions to just those transactions performed by users that were authenticated at the same level. This made the new undo less restrictive than the Zope 1 undo. Undo should probably be *more* restrictive, but that will have to wait for 2.1 or later. - Zope crashed under load for applications using ZCatalog due to an ancient bug in handling loading of empty BTrees from the database. This bug has only become effective with ZCatalog. Zope 2.0 beta 4 Features Removed - Draft objects are no longer supported. Features added - The source release install script now checks to make sure that the Python used has thread support. - FileStorages can now habe quotas. Bugs fixed - This release fixes a socket listen-queue depth problem on Win32 systems that caused poor performance and connection refusals in ZServer. - Fixes to the win32 installation were made to prevent a potential conflict on systems that have an existing installation of the Python win32 extensions. - Some of the management views, including the security/permission mapping views were not visible for ExternalMethods. - New SQL methods could bot be created. - New mailhost objects could not be created. - New Draft objects could be created. - Draft objects could not be deleted. - ZClasses could not be added as methods. - Disk full errors could leave the object system in an inconsistent state. The handling of this error was made more robust. Also, when serious errors like this occur late in the transaction commit process, the transaction system switches to read-only mode. The File-storage recovery process needs to change to make disk full errors detectable earlier in the two-phase commit process so that the system can recover without switching to read-only mode. This change, which is expected in 2.1 will also provide better write performance. - Errors during commit of sub-transactions were not handled correctly, leading to inconsistent object data. - Packing not not mark all transactions performed before the pack time as "packed". Undoing these transactions lead to a broken database. - On win9x systems, random data seems to be added to the end of the file on system crashes, causing apparent file corruption. Code has been added to check for this case and recover by truncating the file with a log message. - The Zope version was not shown in the control panel. - The binary release had the PCGI wrapper program in the wrong directory so PCGI installations were not successful. Zope 2.0 beta 3 Features added - Added a 'getobject' method to ZCatalog. This allows you to more easily get at a cataloged object from 'searchResults'. For example::
Bugs fixed - This release fixes a bug in acquisition that caused creation of ZClass instances to fail under some (common) circumstances. - Copy and Paste validation machinary now takes local roles into account. - ZDOM now adheres more closely to the DOM spec with respect to Document and DOMImplementation nodes. - The main management screen will no longer show the 'Delete' or 'Export' buttons if the user doesn't have adequate permissions to delete or export objects respectively. Zope 2.0 beta 2 Features added - Added a '-r' switch to z2.py for ZServer read-only mode. - The 'null' attribute in the DTML 'var' tag is now used when the value being output if the Python value 'None'. Some people considered this to be a bug. - Permission objects are automatically created for new ZClass objects as part of the ZClass wizard. - For Python programmers, there is a new convenience function for debugging in the Zope package. For example, to debug method spam in the Zope debugger:: python import Zope Zope.debug('spam', d=1) Bugs fixed - Changed the logic used to detect binary content types to only look for control characters. The previous version incorrectly considered certain Latin-1 characters as binary. - The ZService.py used to implement Zope as a Win32 service on NT has been updated to handle a difference in the way .dll files are found on internationalized versions of NT. - The configure.in file used to build PCGI has been updated. - Several fixes to DateTime were added to provide better behavior on machines with unrecognized time zone names. - Conflict errors, which occur when simultaneous writes occur to the same object in multiple database connections, are now handled correctly by re-executing (up to three times) the requests that lost the race condition. - Error formatting with standard_error_message has been moved to the object publisher via a new published module hook, 'zpublisher_exception_hook'. In addition to enabling conflict error handling, this fixed two other bugs: - Error handling in ExternalMethods and in DTML methods or documents called from other DTML documents or methods caused errors to be propagated in ways that defeated use of the try tag and other forms of error handling. - Errors raised before or after calling published objects were not formatted. - Errors in processing form variables, such as missing or miss-typed values resulted in server errors, rather than in formatted error reports. - Database browsing didn't work for ZGadflyDA database adapters. - Transaction meta data were not captured correctly by ZPublisher, which made the undo log look odd wrt the user who performed the transaction. - Permissions for version operations were not set correctly. This made it hard to delegate work to be done in versions and to separate version participation from version finalization. To fix this, it was necessary to split the "Join/Leave" view into separate "Join/Leave" and "Save/Discard" views. - No remark could be entered when discarding versions. - Fixed a ZServer kill_zombies bug which kept ZServer from killing hung connections. - Improved ZServer's handling of slightly broken HTTP requests. - Fixed a properties setting permission bug. - Fixed a product permissions registration bug that allowed product permission data to get lost. - Updated the MailHost product to use the new product initialization procedure. This fixed a MailHost adding permissions bug. - Made Z Class icons visible to everyone. This way you can view a Z Class icon without having and permissions on the Z Class. - Objects were sometimes incorrectly acquired do a bug in detecting attribute errors during acquisition. This led to errors getting covered up or obscured. - There was a memory leak in acquisition. - Some thread-safety problems were fixed. - DTML entity references (e.g. '&dtml-spam;') with missing semicolons (e.g. '&dtml-spam') caused infinite loops. - Extra spaces around expressions in DTML caused syntax errors. - Broken objects, which are used as surrogates for objects that can no longer be instantiated due to missing product installations, no longer use HTML markup in their text representation, as this caused odd effects in HTML titles. - A number of fixes were made to the new DOM support. - Versions could be deleted by users working in the deleted versions. This led to strange paradoxical effects. Attempts to delete a version or an object containing a version by someone working in a version now lead to errors. - ZGadflyDA has been updated to use the current (as of this release) version of Gadfly. (Note that this had been incorrectly reported as a 2.0 beta 1 feature.) - Using an sql method in a version sometimes caused the method to be locked, making it unusable by non-version users. This was due to the way that column meta-data was managed. Now column meta-data, used by report creation wizards, is only safed when sql methods are tested. - In StructuredText, list items could not directly introduce example code. - Undoing transactions that discarded versions did not cause versions to appear non-empty. - Temporary files were created for file storages that were opened in read-only mode. - A bug in FileStorage and DemoStorage didn't show you more than the first 20 previous transactions in the Undo log. - An incorrect contents view was returned after adding a File object. - On the cache parameters page, the number of bytes, rather than the number of objects in the database was shown. Note: The section for Zope 2.0 alpha 4 did not mention that ZCatalog is now shipped with the Zope core. Zope 2.0 beta 1 Backward incompatibility - ZODB 2 is no longer supported or included! - The utility script bbb.py, no longer supports xml output. Known problems - Handling of conflict errors hasn't been implemented. When two threads try to update the same object, one will get a conflict error. By beta release 2, requests that get conflict errors will be retried until they succeed (or fail without a conflict error) or until a maximum number of retries is exceeded. Features added - The default ports used by ZServer have been changed. The new default ports are: 8080 for HTTP, 8021 for FTP and 8099 for the monitor server. - ZServer now uses the zLOG logging mechanism for informational and warning messages. - ZServer now includes a fairly stable NT service ZService.py - 'access' file now can use SHA-1 hashing, and now does by default - 'zpasswd.py' added to top directory to manage this new 'access' file - A number of changes have been made to enhance security control in through-the-web-developed products: - There are new "Permission" objects that can be added to Products. Permission objects provide a mechanism for defining new permissions. These new permissions show up in folder security screens. - Factory objects can have permission settings. This affects whether items show up in add lists and who can use factories. - Objects in products now have permission mapping rather than direct permission settings just like ZClass methods. - New permissions cannot be defined in ZClasses any longer. Rather, ZClasses can select from global permissions, including permissions defined with permission objects. Here's a sample scenario: 1. Add a ZClass, "Container" with the option to create a factory and constructor methods. 2. Add a permission object with the permission "Manage Containers". 3. Change the permission of the factory created in step 1 to "Manage Containers". 4. Visit the ZClass created in step 1, view the "Define Permissions" view and map the "Create class instances" permission to "Manage Containers". 5. Visit the constructor DTML Methods created in step 1, view the "Define Permissions" view and map the "View" permission to "Manage Containers". In this scenario, we created a ZClass and made it addable only by users (roles) that have the "Manage Containers" permission. - There is a new module, Shared.DC.THUNK (lib/python/Shared/DC/THUNK.py) that provides a mix-in class that implements ZODB 3 transaction protocols and that provides course-grained locking logic that should allow many non-thread-safe database adapters to be usable in multi-threaded Zope applications. The Gadfly database adapter has been updated to use this mix-in class. - The transaction management mix-in classes (Shared.DC.TM.TM and Shared.DC.THUNK.THUNKED_TM) now provide a lower-level transaction management protocol. Database adapters that mix these in should: - Call 'self._register()' when performing database queries, and - If necessary, override methods: _begin() -- Which is called to begin a transaction, _finish() -- Which is called to finish a transaction, and save changes, or _abort() -- Which is called to abort (rollback) a transaction, and discard changes, or - Object export now provides an XML export option and import files may be in Zope export format or XML format. - Zope objects that support the Zope management framework now support the DOM 1.0 read interfaces. This allows DOM read methods and attributes to be usable with Zope objects and Python DOM tools, like FourThought's 4XSL to work with Zope objects. Note that DOM-defined attributes are defined with "get" methods. For example, to get the DOM-defined "parentNode" attribute, use: 'object.getParentNode()'. - The pyexpat extension module for using James Clark's expat library for parsing XML is included in the distribution as Shared.DC.pyexpat. Bugs Fixed - You can no longer cd to an acquired directory in ZServer FTP. - A number of changes were made for thread safety. This was not as hard as it sounds, since persistent objects are always thread-safe. (Well, only one thread accesses any particular copy of a persistent object.) The biggest issue here is mutable global variables, mutable objects stored in classes, mutable default Python function arguments, and global variables in extensions. Zope 2.0 alpha 4 Backward incompatibility - The old ZopeHTTPServer is no longer supported or included. Use ZServer. Features added - The let and return tags were added to DTML. - Several new input form types were added: :default -- allows you to specify default values :record -- allows you to combine multiple form variables into a single input variable. For example:: will result in a single variable, 'date', with attributes 'year', 'month', and 'day'. :records -- allows you to input a list of records (ie a table) :ignore-empty -- If the field if the value is an empty string. For more information, see the section on "Form variable types" in http://www.zope.org/Documentation/Reference/Trinkets. - In the startup script, z2.py: - The startup script, z2.py, has additional documentation on providing empty strings as arguments. - Added a -P option to the startup script, z2.py, to specify a base port for the various servers. For example:: python z2.py -P8000 is equivalent to: python z2.py -w8080 -f8021 -m8099 This is a convenient short-hand for systems with many zserver instances running. - Added an -l option to the startup script, z2.py, to specify the location of the ZServer hit log. If the location is a relative path, then it is assumed to be rooted in the 'var' directory. - A progress indicator was added to the database conversion utility, utilities/bbb.py. - The PCGI installer now writes the PCGI info file to use ZODB 3 rather than ZODB 2. - The default content is provided in both ZODB 2 and ZODB 3 format. - There is a new logging framework for reporting errors from Python-level code. See the module zLOG.py for details. For the time being, to turn on logging, set the environment variable STUPID_LOG_FILE to a file name or to an empty string to log to standard error. We will eventually provide a more formal and functional logging configuration interface. - The hooks used by objects that modify their environment were changed. The new hooks are: manage_addHook -- called after an object has been added manage_beforeDelete -- called before an object is deleted manage_afterClone -- called after an object has been copied. This should only be used by object that capture their identity is some way other than their id attribute or persistent id. (It's really for ZClasses, which have global ids.) - Pack is now supported for ZODB3 File Storages. - The header output by manage_tabs now shows whether the current object has been modified or locked by a version and shows the version if the version isn't the current working version. - The locked-in-version decoration in the object management contents view shows the locking version if the version isn't the current working version. - Several demonstration ZODB Storage implementations have been added. See lib/python/ZODB/DemoStorage.py and lib/python/ZODB/dbmStorage.py. - Subtransactions have been added (instead of temporary versions). If a true argument is given in a transaction commit or abort call (e.g. 'transaction.commit(1)'), then the commit or abort applies to a subtransaction of the current transaction. This is useful in a number of ways, including: o Subtransaction commits on requests that work with lots of data can reduce memory usage because data can be removed from memory after changes are saved in a subtransaction, o You can rollback changes due to a local error without rolling back *all* of the changes made in a long transaction. - There is an advanced interface for specifying alternate storages for Zope. Zope will try to import the module 'custom_zodb' from INSTANCE_HOME and use it's Storage attribute. For example, to use a Demo storage based on the standard File storage, you might define a 'custom_zodb' module with:: import Globals, ZODB.FileStorage, ZODB.DemoStorage name='%s/Data.fs' % Globals.data_dir base=ZODB.FileStorage.FileStorage(name, read_only=1) Storage=ZODB.DemoStorage.DemoStorage("Demo (%s)" % name, base) - Find can now search text of SQL methods. - DTML tree tags now have a reverse option. - ZClasses now have class ids. These class ids are registered in a central registry and are used when unpicking instances. - Database connections now have a sync method for synchronizing the connection with saved data. This is useful for interactive connections to see database changes made by other threads. - Errors encountered when trying to unpickle object state are now logged, if logging is enabled. - Added an extra checkpoint to ZODB 3 FileStorage Transaction commit to better handle system crashes. - ZPublisher request objects now have a BODYFILE key to get the request body as a file for non-form non-GET requests. - Cookie support was added to the ZServer FTP server to make FTP access compatible with cookie-based authentication. - ZServer now includes a NT Service, 'ZServer.py' This service requires the Python win32 extensions. It is not yet complete and not yet integrated with the Zope installer. It does however include a large doc string. - ZServer is officially using the z2.py start script now. The old start.py and zinit.py were removed. Note: z2.py will call zdaemon.py for you, if you want to use ZServer in daemon mode on Unix. - Added a 'has_permission' method to user objects. This facilitates finding out if a user has access to a given object. For example:: - The product add list is now filtered. This should keep objects out of the product add list for which the user does not have adequate permissions. This is implemented with a new method, 'filtered_meta_types' which returns all the meta_types for which a given user has permissions. Note: Right now ZClasses do not associate permissions with Products. Thus all Control Panel Products appear in the product add list, no matter what roles the user has. - HTTP REQUEST objects now use HTML in their string representation. - The REQUEST keys of the form BASEx and URLx, where x is an integer now allow x to be greater than 9. - index_html is not longer used automatically for XML-RPC requests. - ZPublisher debugger, 'ZPublisher.test' now accepts a 'extra' keyword argument with a dictionary giving extra variables to be added to the request. This is handy for testing methods that require large amounts of input data, such as file uploads. - The load_site utility ('utilities/load_site.py') now accepts a '-9' switch to make it work with Zope (and Principia) sites at revisions 1.9 and lower. Bugs fixed - Got rid of some DOS line-endings in xmlrpclib.py that caused install-time module compilation to fail. - Fixed DOS line-endings in medusa/max_sockets.py - Fixed a NameError (missing TupleType) definition in the export/import (copy/paste) machinery. - The startup script failed on Windows because it imported the posix module without checking the os. - ZServer crashed when the number of simultaneous connections exceeded a system limit, especially on Windows. ZServer now stops accepting new connections when the number of active connections exceeds a threshold. - Accessing cached results of items() (or values() or keys()) calls on BTrees caused core dumps if affected elements were deleted. For example:: for k in abtree.keys(): del abtree[k] Note that this code is still incorrect due to the fact that the objects returned from abtree.items() (or values() or keys()) have reference, rather than copy semantics. - Compiling Acquisition.c caused (bogus) compiler errors on SGI. - zdeemun wus mispeled - Local roles were not handled correctly when filtering management views. - Copying and pasting ZClass instances copied and pasted their ZClasses too, effectively disconnecting them from their managed ZClasses. The same thing happened when ZClass instances were exported and imported. - The Database area in the control panel was broken. - Copy and paste of multiple objects was broken. - Copy and paste of ZClass methods was broken. - Copy and paste of ZClass properysheets was enabled and broken. It is now disabled. - Accidental import of BoboPOS in processes using ZODB (3) caused strange errors. - A bug in batch processing caused batch processing to hang or give incorrect results. - DocumentTemplate-based classes, like DTMLDocument and DTMLMethod could not be subclassed correctly. - There was a memeory leak in the DTML try tag. - When adding folders with user databases, no check was made to make sure the user was permitted to add a user database. - Broken objects weren't created for objects whose class could not be found when using ZODB 3. - The 'self' argument wasn't properly passed to external methods used in ZClasses. - The Setup file in the SearchIndex package had an unneeded -I option that caused compilation errors on some platforms. - ZGadflyDA transactions were not committed. - ZClasses didn't provide an interface to edit their titles. - Propertysheets were broken for ZClasses that inherited from built-in classes, like Folder and DTMLDocument. - Propertysheets were disabled for instances by default. Now, propertysheets have the "manage properties" permission by default. - The navigation pane wasn't updated when classes were added. - Changes in ZClasses were not properly propagated to all database connections using ZODB3. - Database connections were not closed correctly under heavy load when the number of application threads exceeded the number of database connections in the connection pool. This lead to Zope locking up under load if the application thread count was increased using the z2.py -t option. - undo in ZODB 3 didn't properly refresh objects. - export and import was broken for ZODB 3. - Fixed ZServer HTTP zombie killing. Now hung connections are successfully closed. - A bug in handling the case when a file was created on open without the create flag caused weird errors on windows. - The REQUEST BASE1 key was not computed correctly when SCRIPT_NAME was an empty string. - It was impossible to return an empty list or dictionary to an XML-RPC request. - The database conversion script fails when the first record in the input file lacked properly-formatted transaction data. The conversion script also didn't work on some systems due to an incorrect 'flush' call on the *input* file. - Factories that were also ZClass methods didn't get unregistered properly when they were deleted. Zope 2.0 alpha 3 Features added - XML-RPC support All Zope objects are now XML-RPC servers. - ZODB 2 (aka BoboPOS) to ZODB 3 FileStorage database conversion An option has been added to the bbb.py script in the utilities directory to convert ZODB 2 database files to ZODB 3 format. The bbb.py script now accepts an '-f' option that takes an argument specifying an output file name. For example, to convert the ZODB 2 data file, Data.bbb, in the var directory, to ZODB 3 FileStorage format:: python utilities/bbb.py -f var/Data.fs var/Data.bbb Note that version data are not converted. - In Z Classes, Methods and Property Sheets now have a "Define Permissions" management tab to map permissions. This tab used to be labeled "Security". - ZServer's FTP server now attempts to display an object's owner as defined in local roles. Bugs fixed - The cache management screens were broken - Local roles on DTML Methods and Documents were not persistent. - ZServer's select trigger has been updated and now shouldn't give errors when running ZServer as another users. - The tree tag used persistent object ids in ways that were not compatible with ZODB 3. This caused strange behavior like weird content-types and invalid state messages. - The DTML In tag no longer handled batch processing correctly, and sometimes got into infinite loops. This was due to a very old bug that was made effective by Python 1.5.2. - Objects that could not be loaded because their implementing modules (or packages) had been removed were not handled properly as broken objects when ZODB 3 was used. - The DTML Python string format did not allow expressions for simple variable insertion. Expression insertion is now allowed if the var tag name if the var tag name is supplied, as in:: %(var expr="x+1")s Note that, because of this change, the var tag name must also be supplied when a variable named 'var' is inserted, as in:: %(var var)s Zope 2.0 alpha 2 Features added - There is a new optional EXPERIMENTAL HTML DTML syntax. As an alternative to SSI syntax, you can now use ordinary HTML syntax, as in: You can also use an entity-reference syntax for the var tag as an alternative to the tag syntax. For example:: &dtml-foo; is equivalent to:: and is especially handy in attributes:: Both the new syntax and the SSI syntax are allowed in the same document. In fact, if you were really twisted, you could: IMPORTANT This feature really is EXPERIMENTAL. It could disappear someday, depending on how people like it. We'd really like to get feedback on how useful this feature is. - reverse options were added to the in and tree tags. - The startup script, z2.py, has a -D option to turn on debugging mode. - Made environment variable settings work with the http server. So now you can do things like ./z2.py -w 8888 SCRIPT_NAME=Zope and actually have the specified SCRIPT_NAME override the normal SCRIPT_NAME. This is Useful for using ZServer behind a proxy. - Added support for the Medusa monitor server. - There's a first cut at a ZServer NT service. This is Zope on NT's answer to zdeamon.py It actually seems to work, though there's still a fair amount missing. - The error messages for missing or miss-formatted access files was improved. - A new debugging method, manage_debug, has been added to Control_Panel to monitor the Zope process for possible memory leaks. - The name, 'this', has been added to the namespace available to construction methods to provide a function for computing the destination object. The name, 'this' is essentially equivalent to 'Destination' and 'Destination' will be deprecated eventually. - The title of the main Zope management interface now includes a full URL. - Folders that have (or acquire) index_html methods now have view tabs for viewing index_html. - On browsers that support Java Script, the "Add" button is omitted and the add form is visited as soon as a selection is made from the select list. - The python-pickle format used to export data from ZODB 2 was updated to handle string data differently. - Update syslog clients to try for SOCK_DGRAM first, since newer syslogd won't accept SOCK_STREAM connections (Collector #888). Bugs fixed - A number of bugs and features were left out of the release notes for Zope 2.0 alpha 1. The notes for Zope 2.0 alpha 1 have been updated substantially. - There was a serious memory leak introduced when a security bug was fixed in 1.10.0. - There was a memory leak in ZODB 3 persistent objects, which now manage object IDs as persistent objects. - The Python 1.5.2 cgi module didn't work with Zope. Zope now includes its own fixed version of the module. This bug caused FTP, Netscape Publishing and WebDAV to fail. - A bug in transaction management caused Gadfly requests to fail. Other database adapters need to be updated. - Running ZServer as nobody failed due to the order in which the log file was opened. - Added fix contributed by Brian Hooper for HTTP Clients which don't always spell their HTTP headers correctly. - A number of classes didn't get there reference-counts reduced when instances were destroyed. This confused the new memory allocation monitoring tool. - New user folders had no effect because they did not get properly registered in their containing folders. - Changes made to factory objects were lost when a process was restarted. - Clicking on the 'Help' caused the help information to be displayed in the current frame, rather than in a new window. - Shutting down Zope attempted to save the database index even when Zope was run in read-only mode. - A number of fixes were made in the Date-Time support. - Batch processing in the in tag sometimes failed. - Sites with custom roles in the top-level application object did not pick up the new 'Owner' role. - Content-type detection had problems in DTML documents and methods. - Some cookie expiration dates used 2-digit years. - Error values for errors raised by ExternalMethods were lost. - Values returned by absolute_url were not URL quoted. - Extra newlines were prepended to documents when they were edited. - Objects ids were not properly quoted in a number of places, because id quoting was unnecessary until recent relaxation of the rules for ids. - ExternalMethods are now automatically reloaded when running in debug mode. - Search interfaces generated by the search interface wizard no longer include a "Cancel" button. - Errors were masked by a bug in handling ZODB3 transaction aborts. - The ZODB 3 cache manager deactivated objects too quickly, causing performance to degrade. - BASEx request variables were not computed correctly when SCRIPT_NAME was not empty. - The URLx variables couldn't be computed in some cases when using ZServer. Zope 2.0 alpha 1 BACKWARD INCOMPATIBILITY - A change was made to the way ZClasses are constructed. Previously, the __call__ method was an alias for the index_html method which took took an id and a request and created a new instance and added the instance to the destination. Now, the __call__ method simply calls the underlying managed class with the supplied arguments to create and return an instance. The instance is not added to a folder. The index_html method is now an alias for the createInObjectManager method, which does what the index_html method did before. There is a new method, fromRequest, that takes an id and a request and creates a new instance. It does not at the instance to a destination. Features Added - Integration of ZODB 3 - The ZClass construction interface has been cleaned up to make construction of ZClasses a little easier from Python. - When adding ZClasses, there is now an option to automatically create construction methods and a factory. - "Management" screens now *only* show tabs a user permitted to see. Used in combination with carefully selected roles, this change provides an adaptable management interface. - Instance homes can now have local Products directories. - There is a new ZServer start up script that can be used without modification and that integrates process management. This script replaces the ZServer 'start.py' and 'zinit.py' scripts. - When the new startup script is used, then the control panel shows a "Restart" button which causes the Zope process to exit and be restarted and a "Shutdown" button that causes Zope to exit and not be restarted. - There is a *very* rough cut at a centralized version manager in the control panel. This is only functional when the ZODB3 database is used. This feature will mature significantly before the final 2.0 release. - A new Zope icon was added, and the "powered by Zope" icon has been updated. - Made DTMLDocuments, DTMLMethods, Images, Files and Folders subclassable by ZClasses. - ZClass property sheets can now have read and modify permissions. - An "add list" name can be specified when creating a ZClass. If specified, then construction methods and a factory are created. This makes the class immediately instantiatable and provides construction method examples. - The ZClass basic property sheet now shows base classes. - Changed the string representation of HTTPRequest so that is a lot friendlier. - Added a Zope debugging short cut. Now:: import ZPublisher ZPublisher.Zope(path) is a shortcut for:: import ZPublisher ZPublisher.test('Zope', path) Bugs Fixed - ZClass permission setting interfaces didn't show inherited permissions. - A number of bugs arising from attempts to set caching headers were fixed. - Some default permissions were not set correctly. - The unused non-functioning resource-consuming Scheduler has been removed. - Access was given to objects without roles even if access would be denied to the container. - Fix submitted by Martijn Pieters: Any property type int or long, set to 0, showed up in the Properties Management tab as blank. This resulted in errors when trying to change values of other properties, etc. - Version names were computed incorrectly. - Delete and rename were broken for ZClasses. - It wasn't possible to override inherited attributes with methods in ZClasses. - Property values were not removed from ZClasses when property sheets were deleted. This made it impossible to re-add a property from a deleted property sheet. - Generated property view and edit forms for ZClass property sheets had a number of problems. - The propertysheets property sheet should only allow addition of property sheets. Zope 1.11.0 pr1, which should have been named Zope 2.0.0 alpha 0. Lots has changed in this release. The major news is ZClasses and WebDAV. Features Added - Added WebDAV support. See http://webdav.zope.org/ for details. - Added Z Classes. Documentation is forthcoming. ZClasses might be viewed as somewhat experimental, although we're using them aggressively for internal projects. - Added Property Sheets. Documentation is forthcoming. - ZServer now included in distribution. - Added new product registration interface. Documentation is forthcoming. - Added local roles which allow for permission settings for users on individual objects. This allows for 'ownership' of Zope objects. Added 'Owner' role to default roles, and automatic setting of the 'Owner' role on a created object to the user who created it. - Added hooks for XML-RPC support to ZPublisher. See the source for details. XML-RPC support is not yet finished. - Added automatic height/width detection for JPEG images. - Revised Zope error messages. - A new DTML 'try' tag added based on work by Jordan B. Baker. See the source for details. Documentation is forthcoming. - A new DTML 'mime' tag has been added. See the MIMETools README.txt for more information. - A new 'range' function added the the '_' DTML variable. - A new 'utilities' directory is included in the distribution. - A new flag attribute, 'only', of the DTML 'with' tag prunes the DTML namespace to only the namespace created with the with tag. - Added contributed support for encoding options in the SendMail tag and MailHost send method. - Management tabs are now prettier and allow status messages with the manage_tabs_message variable. - Added a method of notifying users of changes without using a MessageDialog. This makes editing DTML Documents and Methods easier. - Added more documentation to Product add screens. - New icons for many Zope objects. - Sessions are now known as Versions. - Changed from using 'manage_main' as default management method to use 'manage_workspace' which is a reserved method that dispatches to the default management interface which is the first interface defined in manage_options - Permission settings are now inherited by Products from their base classes. This means when you build a Product you only need to specify its particular permission settings. - Changed ZPublisher.publish_module's interface so that instead of providing a stdin, stdout, and stderr, you can provide a Request and a Response object. ZServer makes extensive use of this feature. - Added methods to HTTPRequest to allow access to ZPublishers object traversal. HTTPRequest.clone and HTTPRequest.resolve_url. See the source for more details. - Added a new "builtin" function reorder to DTML. reorder(s, [with, without]]) Reorder the items in s according to the order given in with and with items mentioned in without removed. Items from s not mentioned in with are removed. s, with, and without are all either sequences if strings or sequences of key-value tuples, with ordering done on the keys. This function is useful for constructing ordered select lists. Bugs Fixed - Added small fixes to ZopeHTTPServer in anticipation of DAV. - Construction, upload, and edit methods did not consistently support either string or file data. This caused really weird behavior when file upload was used with out including the ':string' suffix in the argument name when doing a file upload. This bug was previously (and incorrectly) reported as a ZPublisher.Client problem. - Errors in ExternalMethods were masked by bugs in error reporting logic. The bugs were made far more effective by Python 1.5.1. - Databases (or export files) created in binary distributions were not usable in source distributions due to cPickle limitations in stock Python 1.5.1. We now link/copy the cPickle extension to BoboPOS to make sure that it uses the latest version. - Fixed registration of MailHost settings and incorrect default quoted-printable encoding of message bodies. - Added contributed patch that fixes bug in copied-object id generation. - DTML Documents and Methods now attempt to determine their content-type rather than assuming they are text/html. - Fixed PUT bug in determining content-type of binary files. - Added changes that allow the top level userfolder to be replaced by a UserDB or other userfolder-like object. - Fixed an FTP support bug that sometimes kept simple items from producing FTP listings. - Added URL1 instead of "." as the form target for the workspace forms -- using "." causes NS servers to intercept the request and report an error, making Zope effectively unusable. - Added ZPL 1.0 and credits to copyright link in management screen. - Fixed a bug that prevented validate() from working if the top-level object were accessed and no default object (index_html) existed. - Fixed a logic bug in security validation that allowed access to unacquired objects that didn't have __roles__ even if the containing object was protected. Zope 1.10.2 Bugs Fixed - Fixed a permission problem with DTMLMethod objects that caused errors when attempting to commit permission changes. - Factories could not determine the available methods when they were being edited, making them uneditable. Zope 1.10.1 This is a Python 1.5.2 beta 2 compatibility release. A Python C API change in Python 1.5.2 beta 2 broke Zope. This release contains a fix. Zope should work correctly with Python 1.5, 1.5.1 and beta versions of 1.5.2. Zope 1.10.0 New Features - Added Import/Export interfaces to Folders. Bugs Fixed - Fixed a bug in ZopeHTTPServer which caused it to fail on POST requests without content-type headers. - PUT and file uploads would fail if the content-type could not be determined. Now objects created via PUT and file upload will default to either 'application/octet-stream' for binary files or 'text/plain' for non-binary files if no content-type can be determined from headers or filename extensions. - Fixed a permission bug left from a change to the Find support. - Fixed misc typos in docstrings and docs. - Fixed a bug that kept objects from being freed from the cache! Basically, there was never a check for inactive objects. - MailHost objects didn't do appropriate CRLF conversion or recognize 4xx server responses. Zope 1.10.0 pr1 New Features - Documents generated by the search interface wizard no longer use tab characters. - Added a -s option to ZopeHTTPServer. This is useful for emulating older sites when generating screen shots for creating or updating documentation. - Default bobobase now includes a Zope button along with a link to the Zope site. This satisfies the ZPL attribution requirement. The button is created by a new builtin method 'ZopeAttributionButton'. Also, both the source and the binary distributions now have the same default bobobase. - Made the logic for getting SOFTWARE_HOME a bit more robust, so that it handles relative paths and paths starting with '.' and '..'. This is useful when simply importing Main from the Python prompt in the lib/python directory. - The ThreadLock module has been changed to reflect changes in the Python API in Python 1.5.2. - The '_' variable in DTML now has the random module as an attribute and DateTime.DateTime as the attribute DateTime. - Added keys() and items() methods to the REQUEST object. - Added get_header() method to the REQUEST object for retrieving HTTP headers. - New SearchIndex package and BTree component have been added to the distribution, providing low-level support for products with built-in searching such as Tabula and Confera. - Documents have now been split into two distinct object types: DTML Documents and DTML Methods. DTML Methods behave exactly as Document objects did. DTML Documents are similar except that the "client" or "self" of a DTML Document is the DTML Document itself rather than its parent, and DTML Documents can have properties. - DTML Documents, Image and File objects can now have properties. - A new PropertyManager mixin class allows developers to easily add properties and property management to non-Folder objects. - Added a new absolute_url method which can generate an absolute url for the object on which the method is called. - GIF and PNG Images now figure out their own dimensions when uploaded. Image height and width properties are now editable. - Images now generate themselves as html IMG tags, using an absolute url and correct height and width tags if possible. - There is a new form input type, boolean, for inputing boolean data. - Added a read_only option to the SimpleDB.MultipleRevision and PickleDictionary constructors to open a database in read_only mode. - Added an environment variable, ZOPE_READ_ONLY, used by the Zope framework. If this variable is set, then the database is opened in read only mode. If this variable is set to a string parsable by DateTime.DateTime, then the database is opened read-only as of the time given. Note that changes made by another process after the database has been opened are not visible. - Added a database_quota option to the SimpleDB.MultipleRevision and PickleDictionary constructors and a SimpleDB.MultipleRevision.set_quota method to set a database quota. The quota is given as either an integer number of bytes, or a function taking an integer number of bytes argument and returning whether the quota has been exceeded. - Added an environment variable, ZOPE_DATABASE_QUOTA, used by the Zope framework. If this variable is set, it should be set to an integer number of bytes. Additions to the database are not allowed if the database size exceeds the quota. - Added size and last modification time to DTML Method and DTML Document edit interface. - Added a new module, BoboPOS.winlock, that provides file locking on windows. This should prevent many cases of data corruption. - Added FTP support to Documents, Files, Images, and Folders. FTP interface is detailed in OFS/FTPInterface.py though it may change. Basic FTP support was also added to SimpleItem to make it easier for Zope objects to support FTP. Bugs Fixed - Permissions for manage_clone method were wrong, meaning that essentially only managers could use it. - File paths that showed up in tracebacks reflected the paths the files had when they were compiled (at distribution build-time). The compiled-in paths have been changed to be relative to the Zope installation directory. - Fixed a bug in ZPublisher's special handling the "cancel" submit buttons. The test for the cancel button was case sensitive. I changed it so it would accept any case, as well as extra spaces. Cancel buttons should probably be handled using the new :method form type, but this would require the presense of an appropriate redirect method. Perhaps there should be a :redirect form type. - Date strings with years < 31 were not accepted, even through the documentation says they should be. - If Zope crashed or was unable to completely write a database record, then on restart, a data corruption error was reported. Now, truncated records are removed if necessary on start up. If necessary, other records are removed as well to avoid partially-written transactions. - Lack of sys.argv in the win32 service version of Zope caused it to raise errors under certain conditions. - Several HTTP PUT related bugs were fixed. - Simple items did not have an objectValues method, which caused odd behavior in certain tree tags since the method would be acquired from the parent of the simple item. - The Find form in the management interface depended on javascript to function properly. - Added a fix to compensate for the fact that Apache servers may rename the entire cgi environment when mod_rewrite rules are used in .htaccess files. - If system clocks were changed, so that database records had records in the "future", then database corruption could either occur or be incorrectly reported. Now this situation results in an informative error message and corruption does not occur. Zope 1.9.0 New Features - Zope now provides better information about products. If a Product has a README.txt, then it will be readable through the product management interface. If a product encounters an error during startup, the traceback is available through the product management interface. - Cache statistics are displayed in minutes, rather than seconds. - Products can define the attribute '__module_aliases__' in their '__init__' modules that specifies aliases for modules used in the product. This is useful if module names have changed and the database contains pickles refering to the old modules. The attribute should be a sequence of tuples. Each tuple has a (dotted) module name and a module. Bugs Fixed - Development on ZopeHTTPServer has been frozen in anticipation of the switch to Medusa. Therefor ZopeHTTPServer still does support PUT. Sorry. - ZPublisher.Client used the obsolete rand module. - Errors occuring during product import caused Zope to fail to come up. - Product version information was not displayed correctly. - ZGadflyDA cached table meta-data which lead to missleading information when browsing databases. Zope 1.9 beta 3 New Features - Z SQL Methods (FKA Aqueduct) and the Z Gadfly Database Adapter are now included. - The argument handling for ZopeHTTPServer has been changed to be a bit more flexible wrt the order of argumements. While options must still come first, environment settings can be given in any order. This makes startup scripts loke serve.py quite a bit simpler. - ZopeHTTPServer now sports preliminary support for PUT publishing. - ZopeHTTPServer now serves from port 9673 by default, not 8080. Bugs fixed - The start.bat file used to start ZopeHTTPServer on win32 was misnamed and didn't correctly pass command-line arguments. - Win32 machines without an existing Python installation had problems due to PYTHONPATH bootstrapping. - The included ZopeHTTPServer was failing to start correctly on win32 machines that were not connected to the network. Zope 1.9 beta 2 In addition to various documentation updates, this release includes: New Features - Changed serve.py to accept ZopeHTTPServer command-line options. - Added a -P option to the ZopeHTTPServer to specify a file to contain the process ID. This allows a script to stop the process on Unix. - Added support for the environment variables, Z_REALM and Z_DEBUG_MODE to set an authentication realm and to enable debug mode. Bugs fixed - External methods were broken due to differences in Python 1.4 and Python 1.5 comparison semantics. - ZopeHTTPServer command-line environment settings were not available in os.environ. - Some class references were wrong in examples in the ExtensionClass documentation. - INSTANCE_HOME was not set correctly when the debugger was run with a relative path to the published module. - The #! line in the ZopeHTTPServer was hidden by the copyright statement and didn't have the right path. - A bug in pcgi_publisher.py caused file uploads larger than 1 MB to fail. Zope 1.9 beta 1 Bugs fixed: - pcgi did not build correctly on Solaris 2.6 - links in headers were broken when Zope was reached with /, as is the case when the ZopeHTTPServer is used. - Python sources used tabs and spaces for indentation, which made some people unhappy. - Copyright statements were missing or out of date. - Added better instructions for building and running Zope. - Added new Zope License and attribution information. - Merged BoboHTTPServer updates with ZopeHTTPServer. Including socket changes to allow easier stopping and starting. - Shutting down the ZopeHTTPServer from the application caused an empty document to be returned. - The install scripts failed if run more than once when attempting to create a var directory. Features - Added (and old version of) the zlib module to make installation simpler. - Zope requires an access file. The install scripts now create one. Zope 1.9 alpha 1 This was the inital Zope test release. Zope-2.8.7-final/testing.log0000664000076400007640000003754010435523133014730 0ustar ajungajung------ 2006-05-26T08:33:54 WARNING ZODB.FileStorage temp.fs Database records 899 seconds in the future ------ 2006-05-26T08:33:54 CRITICAL ZODB.FileStorage temp.fs Database records 3599 seconds in the future ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 8 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 9 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 10 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 11 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 12 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 13 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 14 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 CRITICAL ZODB.DB DB.open() has 15 open connections with a pool_size of 7 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 3 open connections with a pool_size of 2 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 4 open connections with a pool_size of 2 ------ 2006-05-26T08:34:04 CRITICAL ZODB.DB DB.open() has 5 open connections with a pool_size of 2 ------ 2006-05-26T08:34:04 WARNING ZODB.DB DB.open() has 7 open connections with a pool_size of 6 ------ 2006-05-26T08:34:06 WARNING ZODB.DB DB.open() has 4 open connections with a pool_size of 3 ------ 2006-05-26T08:34:06 WARNING ZODB.DB DB.open() has 5 open connections with a pool_size of 3 ------ 2006-05-26T08:34:06 WARNING ZODB.DB DB.open() has 6 open connections with a pool_size of 3 ------ 2006-05-26T08:35:11 ERROR zope.app.generations Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/generations/generations.py", line 338, in evolve manager.evolve(context, generation) File "", line 12, in evolve raise ValueError(generation) ValueError: 4 ------ 2006-05-26T08:35:11 ERROR zope.app.generations Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/generations/generations.py", line 338, in evolve manager.evolve(context, generation) File "", line 12, in evolve raise ValueError(generation) ValueError: 4 ------ 2006-05-26T08:35:18 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:35:18 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:35:21 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:35:21 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:35:21 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:35:22 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed HTMLParser.HTMLParseError: unexpected char 'S' in internal subset, at line 1, column 18 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:36:37 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:36:52 ERROR foo.bar eek None ------ 2006-05-26T08:36:52 INFO foo.bar blah blah ------ 2006-05-26T08:52:51 WARNING ZODB.FileStorage temp.fs Database records 899 seconds in the future ------ 2006-05-26T08:52:51 CRITICAL ZODB.FileStorage temp.fs Database records 3599 seconds in the future ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 8 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 9 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 10 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 11 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 12 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 13 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 14 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 CRITICAL ZODB.DB DB.open() has 15 open connections with a pool_size of 7 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 3 open connections with a pool_size of 2 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 4 open connections with a pool_size of 2 ------ 2006-05-26T08:52:58 CRITICAL ZODB.DB DB.open() has 5 open connections with a pool_size of 2 ------ 2006-05-26T08:52:58 WARNING ZODB.DB DB.open() has 7 open connections with a pool_size of 6 ------ 2006-05-26T08:52:59 WARNING ZODB.DB DB.open() has 4 open connections with a pool_size of 3 ------ 2006-05-26T08:52:59 WARNING ZODB.DB DB.open() has 5 open connections with a pool_size of 3 ------ 2006-05-26T08:52:59 WARNING ZODB.DB DB.open() has 6 open connections with a pool_size of 3 ------ 2006-05-26T08:54:09 ERROR zope.app.generations Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/generations/generations.py", line 338, in evolve manager.evolve(context, generation) File "", line 12, in evolve raise ValueError(generation) ValueError: 4 ------ 2006-05-26T08:54:09 ERROR zope.app.generations Failed to evolve database to generation 4 for app1 Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/generations/generations.py", line 338, in evolve manager.evolve(context, generation) File "", line 12, in evolve raise ValueError(generation) ValueError: 4 ------ 2006-05-26T08:54:16 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:54:16 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:54:18 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:54:18 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:54:18 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:54:19 ERROR SiteError Error while reporting an error to the ErrorLogging service Traceback (most recent call last): File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/app/publication/zopepublication.py", line 209, in _logErrorWithErrorReportingService errService = zapi.getService(zapi.servicenames.ErrorLogging) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/__init__.py", line 68, in getService return getServices(context).getService(name) File "/develop/sandboxes/Zope-2.8/Zope/lib/python/zope/component/service.py", line 109, in getService raise ComponentLookupError(name) ComponentLookupError: 'ErrorLogging' ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed HTMLParser.HTMLParseError: unexpected char 'S' in internal subset, at line 1, column 18 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 0 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:55:05 ERROR root PageTemplateFile: Error in template: Compilation failed xml.parsers.expat.ExpatError: unclosed token: line 1, column 1 ------ 2006-05-26T08:55:23 ERROR foo.bar eek None ------ 2006-05-26T08:55:23 INFO foo.bar blah blah Zope-2.8.7-final/test.py0000664000076400007640000007772410435500314014105 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """ test.py [-abcCdDfgGhLmpqrtTuv] [modfilter [testfilter]] Find and run tests written using the unittest module. The test runner searches for Python modules that contain test suites. It collects those suites, and runs the tests. There are many options for controlling how the tests are run. There are options for using the debugger, reporting code coverage, and checking for refcount problems. The test runner uses the following rules for finding tests to run. It searches for packages and modules that contain "tests" as a component of the name, e.g. "frob.tests.nitz" matches this rule because tests is a sub-package of frob. Within each "tests" package, it looks for modules that begin with the name "test." For each test module, it imports the module and calls the test_suite() method, which must return a unittest TestSuite object. (If a package contains a file named .testinfo, it will not be searched for tests. Really.) -a level --all Run the tests at the given level. Any test at a level at or below this is run, any test at a level above this is not run. Level 0 runs all tests. The default is to run tests at level 1. --all is a shortcut for -a 0. -b Run "python setup.py build_ext -i" before running tests, where "python" is the version of python used to run test.py. Highly recommended. Tests will be run from the build directory. (Note: In Python < 2.3 the -q flag is added to the setup.py command line.) -c Use pychecker --config-file filename Configure Zope by loading the specified configuration file (zope.conf). -C filename Shortcut for --config-file filename. -d Instead of the normal test harness, run a debug version which doesn't catch any exceptions. This is occasionally handy when the unittest code catching the exception doesn't work right. Unfortunately, the debug harness doesn't print the name of the test, so Use With Care. --dir directory Option to limit where tests are searched for. This is important when you *really* want to limit the code that gets run. For example, if refactoring interfaces, you don't want to see the way you have broken setups for tests in other packages. You *just* want to run the interface tests. -D Works like -d, except that it loads pdb when an exception occurs. -f Run functional tests instead of unit tests. -g threshold Set the garbage collector generation0 threshold. This can be used to stress memory and gc correctness. Some crashes are only reproducible when the threshold is set to 1 (agressive garbage collection). Do "-g 0" to disable garbage collection altogether. -G gc_option Set the garbage collection debugging flags. The argument must be one of the DEBUG_ flags defined bythe Python gc module. Multiple options can be specified by using "-G OPTION1 -G OPTION2." --import-testing Import the Testing package to setup the test ZODB. Useful for running tests that forgot to "import Testing". --libdir test_root Search for tests starting in the specified start directory (useful for testing components being developed outside the main "src" or "build" trees). Note: This directory will be prepended to sys.path! --keepbytecode Do not delete all stale bytecode before running tests -L Keep running the selected tests in a loop. You may experience memory leakage. --nowarnings Install a filter to suppress warnings emitted by code. -t Time the individual tests and print a list of the top 50, sorted from longest to shortest. -p Show running progress. It can be combined with -v or -vv. -r Look for refcount problems. This requires that Python was built --with-pydebug. -T Use the trace module from Python for code coverage. XXX This only works if trace.py is explicitly added to PYTHONPATH. The current utility writes coverage files to a directory named `coverage' that is parallel to `build'. It also prints a summary to stdout. --coverage Use the coverage.py module from Gareth Rees (http://www.nedbatchelder.com/code/modules/coverage.html) to collect data for code coverage. This will output trace data in a file called '.coverage'. You will need to call coverage.py after the test run to analyse the data (-r for a line count report, -a for annotated file). --profile Use the profile module from the standard library to collect profiling data. This will output data in a file called '.profile'. You will need to use the pstats module from the standard library after the test run to analyse the data. --hotshot Use the hotshot module from the standard library to collect profiling data. This will output data in a file called '.hotshot'. You will need to use the hotshot.pstats module from the standard library after the test run to analyse the data. You may also use the hotshot2cg script from the KCacheGrind sources to create data suitable for analysis by KCacheGrind. -v -q Verbose output. With one -v, unittest prints a dot (".") for each test run. With -vv, unittest prints the name of each test (for some definition of "name" ...). With no -v (or with -q), unittest is silent until the end of the run, except when errors occur. -u -m Use the PyUnit GUI instead of output to the command line. The GUI imports tests on its own, taking care to reload all dependencies on each run. The debug (-d), verbose (-v), and Loop (-L) options will be ignored. The testfilter filter is also not applied. -m starts the gui minimized. Double-clicking the progress bar will start the import and run all tests. modfilter testfilter Case-sensitive regexps to limit which tests are run, used in search (not match) mode. In an extension of Python regexp notation, a leading "!" is stripped and causes the sense of the remaining regexp to be negated (so "!bc" matches any string that does not match "bc", and vice versa). By default these act like ".", i.e. nothing is excluded. modfilter is applied to a test file's path, starting at "build" and including (OS-dependent) path separators. testfilter is applied to the (method) name of the unittest methods contained in the test files whose paths modfilter matched. Extreme (yet useful) examples: test.py -vvb . "^checkWriteClient$" Builds the project silently, then runs unittest in verbose mode on all tests whose names are precisely "checkWriteClient". Useful when debugging a specific test. test.py -vvb . "!^checkWriteClient$" As before, but runs all tests whose names aren't precisely "checkWriteClient". Useful to avoid a specific failing test you don't want to deal with just yet. test.py -m . "!^checkWriteClient$" As before, but now opens up a minimized PyUnit GUI window (only showing the progress bar). Useful for refactoring runs where you continually want to make sure all tests still pass. """ import gc import os import re import pdb import sys import time import traceback import unittest from distutils.util import get_platform PLAT_SPEC = "%s-%s" % (get_platform(), sys.version[0:3]) def callers(n): callers = [] f = sys._getframe(2) while f: co = f.f_code callers.append((co.co_filename, co.co_name)) f = f.f_back n -= 1 if not n: break return callers class ImmediateTestResult(unittest._TextTestResult): __super_init = unittest._TextTestResult.__init__ __super_startTest = unittest._TextTestResult.startTest __super_printErrors = unittest._TextTestResult.printErrors def __init__(self, stream, descriptions, verbosity, debug=False, count=None, progress=False): self.__super_init(stream, descriptions, verbosity) self._debug = debug self._progress = progress self._progressWithNames = False self.count = count self._testtimes = {} if progress and verbosity == 1: self.dots = False self._progressWithNames = True self._lastWidth = 0 self._maxWidth = 80 try: import curses except ImportError: pass else: import curses.wrapper def get_max_width(scr, self=self): self._maxWidth = scr.getmaxyx()[1] try: curses.wrapper(get_max_width) except curses.error: pass self._maxWidth -= len("xxxx/xxxx (xxx.x%): ") + 1 def stopTest(self, test): self._testtimes[test] = time.time() - self._testtimes[test] if gc.garbage: print "The following test left garbage:" print test print gc.garbage # XXX Perhaps eat the garbage here, so that the garbage isn't # printed for every subsequent test. def print_times(self, stream, count=None): results = self._testtimes.items() results.sort(lambda x, y: cmp(y[1], x[1])) if count: n = min(count, len(results)) if n: print >>stream, "Top %d longest tests:" % n else: n = len(results) if not n: return for i in range(n): print >>stream, "%6dms" % int(results[i][1] * 1000), results[i][0] def _handle_problem(self, err, test, errlist): if self._debug: raise err[0], err[1], err[2] if errlist is self.errors: prefix = 'Error' else: prefix = 'Failure' tb = "".join(traceback.format_exception(*err)) if self._progress: self.stream.writeln("\r") self.stream.writeln("%s in test %s" % (prefix,test)) self.stream.writeln(tb) self._lastWidth = 0 elif self.showAll: self._lastWidth = 0 self.stream.writeln(prefix.upper()) elif self.dots: self.stream.write(prefix[0]) errlist.append((test, tb)) def startTest(self, test): if self._progress: self.stream.write("\r%4d" % (self.testsRun + 1)) if self.count: self.stream.write("/%d (%5.1f%%)" % (self.count, (self.testsRun + 1) * 100.0 / self.count)) if self.showAll: self.stream.write(": ") elif self._progressWithNames: # XXX will break with multibyte strings name = self.getShortDescription(test) width = len(name) if width < self._lastWidth: name += " " * (self._lastWidth - width) self.stream.write(": %s" % name) self._lastWidth = width self.stream.flush() self.__super_startTest(test) self._testtimes[test] = time.time() def getShortDescription(self, test): s = self.getDescription(test) if len(s) > self._maxWidth: pos = s.find(" (") if pos >= 0: w = self._maxWidth - (pos + 5) if w < 1: # first portion (test method name) is too long s = s[:self._maxWidth-3] + "..." else: pre = s[:pos+2] post = s[-w:] s = "%s...%s" % (pre, post) return s[:self._maxWidth] def addError(self, test, err): self._handle_problem(err, test, self.errors) def addFailure(self, test, err): self._handle_problem(err, test, self.failures) def printErrors(self): if self._progress and not (self.dots or self.showAll): self.stream.writeln() self.__super_printErrors() def printErrorList(self, flavor, errors): for test, err in errors: self.stream.writeln(self.separator1) self.stream.writeln("%s: %s" % (flavor, self.getDescription(test))) self.stream.writeln(self.separator2) self.stream.writeln(err) class ImmediateTestRunner(unittest.TextTestRunner): __super_init = unittest.TextTestRunner.__init__ def __init__(self, **kwarg): debug = kwarg.get("debug") if debug is not None: del kwarg["debug"] progress = kwarg.get("progress") if progress is not None: del kwarg["progress"] self.__super_init(**kwarg) self._debug = debug self._progress = progress # Create the test result here, so that we can add errors if # the test suite search process has problems. The count # attribute must be set in run(), because we won't know the # count until all test suites have been found. self.result = ImmediateTestResult( self.stream, self.descriptions, self.verbosity, debug=self._debug, progress=self._progress) def _makeResult(self): # Needed base class run method. return self.result def run(self, test): self.result.count = test.countTestCases() return unittest.TextTestRunner.run(self, test) # setup list of directories to put on the path class PathInit: def __init__(self, build, libdir=None): # Calculate which directories we're going to add to sys.path. self.libdir = os.path.join('lib', 'python') # Hack sys.path self.home = os.path.dirname(os.path.realpath(sys.argv[0])) # test.py lives in $ZOPE_HOME/bin when installed ... dir, file = os.path.split(self.home) if file == 'bin': self.home = dir sys.path.insert(0, os.path.join(self.home, self.libdir)) self.cwd = os.path.realpath(os.getcwd()) # Hack again for external products. if libdir: self.libdir = os.path.realpath(os.path.join(self.cwd, libdir)) else: self.libdir = os.path.realpath(os.path.join(self.cwd, self.libdir)) if self.libdir not in sys.path: sys.path.insert(0, self.libdir) # Determine where to look for tests if test_dir: self.testdir = os.path.abspath(os.path.join(self.cwd, test_dir)) else: self.testdir = self.libdir kind = functional and "functional" or "unit" print "Running %s tests from %s" % (kind, self.testdir) def match(rx, s): if not rx: return True if rx[0] == "!": return re.search(rx[1:], s) is None else: return re.search(rx, s) is not None class TestFileFinder: def __init__(self, prefix): self.files = [] self._plen = len(prefix) if not prefix.endswith(os.sep): self._plen += 1 global functional if functional: self.dirname = "ftests" else: self.dirname = "tests" # dirs maps directories to a boolean indicating whether # the directory is a package. Bootstrap dirs with prefix; # it isn't actually a package, but it contains packages. self.dirs = {prefix: True} def is_package(self, dir): # Return true if dir contains a testable package. bool = self.dirs.get(dir) if bool is not None: return bool files = os.listdir(dir) if ".testinfo" in files or "__init__.py" not in files: self.dirs[dir] = False return False parent, dir = os.path.split(dir) bool = self.is_package(parent) self.dirs[dir] = bool return bool def visit(self, rx, dir, files): if os.path.split(dir)[1] != self.dirname: # Allow tests module rather than package. if "tests.py" in files: path = os.path.join(dir, "tests.py") if match(rx, path): self.files.append(path) return return if not self.is_package(dir): return # Put matching files in matches. If matches is non-empty, # then make sure that the package is importable. matches = [] for file in files: if file.startswith('test') and os.path.splitext(file)[-1] == '.py': path = os.path.join(dir, file) if match(rx, path): matches.append(path) # ignore tests when the package can't be imported, possibly due to # dependency failures. pkg = dir[self._plen:].replace(os.sep, '.') try: __import__(pkg) # We specifically do not want to catch ImportError since that's useful # information to know when running the tests. except RuntimeError, e: if VERBOSE: print "skipping %s because: %s" % (pkg, e) return else: self.files.extend(matches) def module_from_path(self, path): """Return the Python package name indicated by the filesystem path.""" assert path.endswith(".py") path = path[self._plen:-3] mod = path.replace(os.sep, ".") return mod def find_tests(rx): global finder finder = TestFileFinder(pathinit.libdir) walk_with_symlinks(pathinit.testdir, finder.visit, rx) return finder.files def package_import(modname): mod = __import__(modname) for part in modname.split(".")[1:]: mod = getattr(mod, part) return mod class PseudoTestCase: """Minimal test case objects to create error reports. If test.py finds something that looks like it should be a test but can't load it or find its test suite, it will report an error using a PseudoTestCase. """ def __init__(self, name, descr=None): self.name = name self.descr = descr def shortDescription(self): return self.descr def __str__(self): return "Invalid Test (%s)" % self.name def get_suite(file, result): modname = finder.module_from_path(file) try: mod = package_import(modname) return mod.test_suite() except AttributeError: result.addError(PseudoTestCase(modname), sys.exc_info()) return None def filter_testcases(s, rx): new = unittest.TestSuite() for test in s._tests: # See if the levels match dolevel = (level == 0) or level >= getattr(test, "level", 0) if not dolevel: continue if isinstance(test, unittest.TestCase): name = test.id() # Full test name: package.module.class.method name = name[1 + name.rfind("."):] # extract method name if not rx or match(rx, name): new.addTest(test) else: filtered = filter_testcases(test, rx) if filtered: new.addTest(filtered) return new def gui_runner(files, test_filter): utildir = os.path.join(os.getcwd(), "utilities") sys.path.append(utildir) import unittestgui suites = [] for file in files: suites.append(finder.module_from_path(file) + ".test_suite") suites = ", ".join(suites) minimal = (GUI == "minimal") unittestgui.main(suites, minimal) class TrackRefs: """Object to track reference counts across test runs.""" def __init__(self): self.type2count = {} self.type2all = {} def update(self): import types obs = sys.getobjects(0) type2count = {} type2all = {} classes = [] for o in obs: all = sys.getrefcount(o) t = type(o) if t is types.ClassType: classes.append((all, o)) if t in type2count: type2count[t] += 1 type2all[t] += all else: type2count[t] = 1 type2all[t] = all ct = [(type2count[t] - self.type2count.get(t, 0), type2all[t] - self.type2all.get(t, 0), t) for t in type2count.iterkeys()] ct.sort() ct.reverse() for delta1, delta2, t in ct: if delta1 or delta2: print "%-55s %8d %8d" % (t, delta1, delta2) classes.sort() classes.reverse() for n, c in classes[:10]: print n, c self.type2count = type2count self.type2all = type2all def runner(files, test_filter, debug): runner = ImmediateTestRunner(verbosity=VERBOSE, debug=debug, progress=progress) result = runner.result suite = unittest.TestSuite() for file in files: s = get_suite(file, runner.result) # See if the levels match dolevel = (level == 0) or level >= getattr(s, "level", 0) if s is not None and dolevel: s = filter_testcases(s, test_filter) suite.addTest(s) try: r = runner.run(suite) if timesfn: r.print_times(open(timesfn, "w")) if VERBOSE: print "Wrote timing data to", timesfn if timetests: r.print_times(sys.stdout, timetests) numbad = len(result.failures) + len(result.errors) return numbad except: if debugger: pdb.post_mortem(sys.exc_info()[2]) else: raise def walk_with_symlinks(path, visit, arg): """Like os.path.walk, but follows symlinks on POSIX systems. This could theoretically result in an infinite loop, if you create symlink cycles in your Zope sandbox, so don't do that. """ try: names = os.listdir(path) except os.error: return visit(arg, path, names) exceptions = (os.curdir, os.pardir, 'var') for name in names: if name not in exceptions: name = os.path.join(path, name) if os.path.isdir(name): walk_with_symlinks(name, visit, arg) def remove_stale_bytecode(arg, dirname, names): names = map(os.path.normcase, names) for name in names: if name.endswith(".pyc") or name.endswith(".pyo"): srcname = name[:-1] if srcname not in names: fullname = os.path.join(dirname, name) print "Removing stale bytecode file", fullname, try: os.unlink(fullname) except (OSError, IOError), e: print ' --> %s (errno %d)' % (e.strerror, e.errno) else: print def main(module_filter, test_filter, libdir): global pathinit global config_file configure_logging() # Initialize the path and cwd pathinit = PathInit(build, libdir) if not keepStaleBytecode: walk_with_symlinks(pathinit.home, remove_stale_bytecode, None) # Load configuration if config_file: config_file = os.path.realpath(config_file) print "Parsing %s" % config_file import Zope2 Zope2.configure(config_file) if not keepStaleBytecode: from App.config import getConfiguration softwarehome = os.path.realpath(getConfiguration().softwarehome) instancehome = os.path.realpath(getConfiguration().instancehome) softwarehome = os.path.normcase(softwarehome) if not softwarehome.startswith(os.path.normcase(instancehome)): walk_with_symlinks(instancehome, remove_stale_bytecode, None) # Import Testing package to setup the test ZODB if import_testing: import Testing files = find_tests(module_filter) files.sort() if GUI: gui_runner(files, test_filter) elif LOOP: if REFCOUNT: rc = sys.gettotalrefcount() track = TrackRefs() while True: numbad = runner(files, test_filter, debug) gc.collect() if gc.garbage: print "GARBAGE:", len(gc.garbage), gc.garbage return if REFCOUNT: prev = rc rc = sys.gettotalrefcount() print "totalrefcount=%-8d change=%-6d" % (rc, rc - prev) track.update() else: numbad = runner(files, test_filter, debug) return numbad def configure_logging(): """Initialize the logging module.""" import logging.config # Get the log.ini file from the current directory instead of possibly # buried in the build directory. XXX This isn't perfect because if # log.ini specifies a log file, it'll be relative to the build directory. # Hmm... logini = os.path.abspath("log.ini") if os.path.exists(logini): logging.config.fileConfig(logini) else: logging.basicConfig() if os.environ.has_key("LOGGING"): level = int(os.environ["LOGGING"]) logging.getLogger().setLevel(level) def process_args(argv=None): import getopt global module_filter global test_filter global VERBOSE global LOOP global GUI global TRACE global REFCOUNT global debug global debugger global build global level global libdir global timesfn global timetests global progress global keepStaleBytecode global functional global test_dir global config_file global import_testing global no_warnings if argv is None: argv = sys.argv module_filter = None test_filter = None VERBOSE = 0 LOOP = False GUI = False TRACE = False COVERAGE = False PROFILE = False HOTSHOT = False REFCOUNT = False debug = False # Don't collect test results; simply let tests crash debugger = False build = False gcthresh = None gcdebug = 0 gcflags = [] level = 1 libdir = None progress = False timesfn = None timetests = 0 keepStaleBytecode = 0 functional = False test_dir = None config_file = None import_testing = False no_warnings = False try: opts, args = getopt.getopt(argv[1:], "a:bcC:dDfg:G:hLmpqrtTuv", ["all", "help", "libdir=", "times=", "keepbytecode", "nowarnings", "dir=", "config-file=", "import-testing", "coverage", "profile", "hotshot"]) except getopt.error, msg: print msg print "Try `python %s -h' for more information." % argv[0] sys.exit(2) for k, v in opts: if k == "-a": level = int(v) elif k == "--all": level = 0 elif k == "-b": build = True elif k == "-c": # make sure you have a recent version of pychecker if not os.environ.get("PYCHECKER"): os.environ["PYCHECKER"] = "-q" import pychecker.checker elif k == "-d": debug = True elif k == "-D": debug = True debugger = True elif k == "-f": functional = True elif k in ("-h", "--help"): print __doc__ sys.exit(0) elif k == "-g": gcthresh = int(v) elif k == "-G": if not v.startswith("DEBUG_"): print "-G argument must be DEBUG_ flag, not", repr(v) sys.exit(1) gcflags.append(v) elif k == '--keepbytecode': keepStaleBytecode = 1 elif k == '--libdir': libdir = v elif k == "-L": LOOP = 1 elif k == "-m": GUI = "minimal" elif k == "-p": progress = True elif k == "-r": if hasattr(sys, "gettotalrefcount"): REFCOUNT = True else: print "-r ignored, because it needs a debug build of Python" elif k == "-T": TRACE = True elif k == "--coverage": COVERAGE = True elif k == "--profile": PROFILE = True elif k == "--hotshot": HOTSHOT = True elif k == "-t": if not timetests: timetests = 50 elif k == "-u": GUI = 1 elif k == "-v": VERBOSE += 1 elif k == "-q": VERBOSE = 0 elif k == "--times": try: timetests = int(v) except ValueError: # must be a filename to write timesfn = v elif k == '--dir': test_dir = v elif k == '--config-file' or k == '-C': config_file = v elif k == '--import-testing': import_testing = True elif k == '--nowarnings': no_warnings = True if no_warnings: import warnings warnings.simplefilter('ignore', Warning, append=1) if gcthresh is not None: if gcthresh == 0: gc.disable() print "gc disabled" else: gc.set_threshold(gcthresh) print "gc threshold:", gc.get_threshold() if gcflags: val = 0 for flag in gcflags: v = getattr(gc, flag, None) if v is None: print "Unknown gc flag", repr(flag) print gc.set_debug.__doc__ sys.exit(1) val |= v gcdebug |= v if gcdebug: gc.set_debug(gcdebug) if build: # Python 2.3 is more sane in its non -q output if sys.hexversion >= 0x02030000: qflag = "" else: qflag = "-q" cmd = sys.executable + " setup.py " + qflag + " build_ext -i" if VERBOSE: print cmd sts = os.system(cmd) if sts: print "Build failed", hex(sts) sys.exit(1) if VERBOSE: kind = functional and "functional" or "unit" if level == 0: print "Running %s tests at all levels" % kind else: print "Running %s tests at level %d" % (kind, level) if args: if len(args) > 1: test_filter = args[1] module_filter = args[0] try: if TRACE: # if the trace module is used, then we don't exit with # status if on a false return value from main. coverdir = os.path.join(os.getcwd(), "coverage") import trace tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix], trace=0, count=1) tracer.runctx("main(module_filter, test_filter, libdir)", globals=globals(), locals=vars()) r = tracer.results() r.write_results(show_missing=True, summary=True, coverdir=coverdir) elif COVERAGE: try: from coverage import the_coverage except: print "You need to install coverage.py from " print "http://www.nedbatchelder.com/code/modules/coverage.html" sys.exit() the_coverage.start() main(module_filter, test_filter, libdir) elif PROFILE: import profile profile.runctx("main(module_filter, test_filter, libdir)", globals=globals(), locals=vars(), filename=".profile") elif HOTSHOT: import hotshot profile = hotshot.Profile(".hotshot") profile.runctx("main(module_filter, test_filter, libdir)", globals=globals(), locals=vars()) else: bad = main(module_filter, test_filter, libdir) if bad: sys.exit(1) except ImportError, err: print err print sys.path raise if __name__ == "__main__": process_args() Zope-2.8.7-final/utilities/0000775000076400007640000000000010435500415014550 5ustar ajungajungZope-2.8.7-final/utilities/compilezpy.py0000775000076400007640000000401210435500331017312 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import compileall, os, sys class Shutup: def write(*args): pass # :) class NoteErr: wrote = 0 def write(self, *args): self.wrote = 1 apply(stderr.write, args) def compile_non_test(dir): """Byte-compile all modules except those in test directories.""" success = compileall.compile_dir(dir, maxlevels=0) try: names = os.listdir(dir) except os.error: print "Can't list", dir names = [] names.sort() for name in names: fullname = os.path.join(dir, name) if (name != os.curdir and name != os.pardir and os.path.isdir(fullname) and not os.path.islink(fullname) and name != 'test' and name != 'tests' and name != 'skins'): success = success and compile_non_test(fullname) return success print print '-'*78 print 'Compiling python modules' stdout = sys.stdout stderr = sys.stderr try: try: success = 0 sys.stdout = Shutup() sys.stderr = NoteErr() success = compile_non_test(os.getcwd()) finally: success = success and not sys.stderr.wrote sys.stdout = stdout sys.stderr = stderr except: success = 0 import traceback traceback.print_exc() if not success: print print '!' * 78 print 'There were errors during Python module compilation.' print '!' * 78 print sys.exit(1) Zope-2.8.7-final/utilities/check_catalog.py0000664000076400007640000001052010435500331017664 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Script to check consistency of a ZCatalog $Id: check_catalog.py 28991 2005-01-31 15:45:46Z jim $ """ import Zope2 import os,sys,re,getopt from types import IntType from BTrees.IIBTree import IISet,difference,intersection def checkCatalog(path,indexes): """ perform some consistency checks on a ZCatalog instance""" root = Zope2.app() try: catalog = root.unrestrictedTraverse(path) except AttributeError: print 'Error: catalog object not found' sys.exit(1) # get Catalog instance _cat = catalog._catalog # check Catalog internal BTrees l_data = list(_cat.data.keys()) l_data.sort() l_uids = list(_cat.uids.values()) l_uids.sort() l_paths = list(_cat.data.keys()) l_paths.sort() print "Checking catalog internal BTrees" print "\tINFO: Mapping data: %d entries" % len(l_data) print "\tINFO: Mapping uids: %d entries" % len(l_uids) print "\tINFO: Mapping paths: %d entries" % len(l_paths) if l_data == l_uids: print "\tOK: Mapping data equals Mapping uids" else: print "\tERR: Mapping data does not equal Mapping uids" if l_data == l_paths: print "\tOK: Mapping data equals Maaping paths" else: print "\tERR: Mapping data does not equal Maaping paths" # check BTrees of indexes for id,idx in _cat.indexes.items(): if indexes and not idx.meta_type in indexes: continue print "Checking index '%s' (type: %s)" % (id, idx.meta_type) if idx.meta_type in ['FieldIndex','KeywordIndex']: # check forward entries RIDS = IISet() for key, rids in idx._index.items(): if isinstance(rids,IntType): RIDS.insert( rids ) else: map(RIDS.insert , rids.keys()) diff = difference(RIDS, IISet(_cat.data.keys())) if len(diff)!=0: print '\tERR: Problem with forward entries' print '\tERR: too much forward entries:', diff else: print '\tOK: Forward entries (%d entries)' % (len(RIDS)) elif idx.meta_type in ['PathIndex']: RIDS = IISet() for rids in map(None,idx._index.values()): map(RIDS.insert , rids.values()[0]) diff = difference(RIDS, IISet(_cat.data.keys())) if len(diff)!=0: print '\tERR: Problem with forward entries' print '\tERR: too much forward entries:', diff else: print '\tOK: Forward entries (%d entries)' % (len(RIDS)) if idx.meta_type in ['FieldIndex','KeywordIndex','PathIndex']: # check backward entries RIDS = IISet(idx._unindex.keys()) diff = difference(RIDS, IISet(_cat.data.keys())) if len(diff)!=0: print '\tERR: Problem with backward entries' print '\tERR: too much backward entries:', diff else: print '\tOK: Backward entries (%d entries)' % (len(RIDS)) def usage(): print "Usage: %s [--FieldIndex|KeywordIndex|PathIndex] /path/to/ZCatalog" % \ os.path.basename(sys.argv[0]) print print "This scripts checks the consistency of the internal" print "BTrees of a ZCatalog and its indexes." sys.exit(1) def main(): opts,args = getopt.getopt(sys.argv[1:],'h',\ ['help','FieldIndex','KeywordIndex','PathIndex']) indexes = [] for o,v in opts: if o in ['-h','--help']: usage() if o in ['--FieldIndex','--KeywordIndex','--PathIndex']: indexes.append(o[2:]) checkCatalog(args,indexes) if __name__=='__main__': main() Zope-2.8.7-final/utilities/mkzopeinstance.py0000775000076400007640000001360610435500331020162 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """%(program)s: Create a Zope instance home. usage: %(program)s [options] Options: -h/--help -- print this help text -d/--dir -- the dir in which the instance home should be created -u/--user NAME:PASSWORD -- set the user name and password of the initial user -s/--skelsrc -- the dir from which skeleton files should be copied When run without arguments, this script will ask for the information necessary to create a Zope instance home. """ import getopt import os import shutil import sys import copyzopeskel def main(): try: opts, args = getopt.getopt(sys.argv[1:], "hu:d:s:", ["help", "user=", "dir=", "skelsrc="] ) except getopt.GetoptError, msg: usage(sys.stderr, msg) sys.exit(2) script = os.path.abspath(sys.argv[0]) user = None password = None skeltarget = None skelsrc = None for opt, arg in opts: if opt in ("-d", "--dir"): skeltarget = os.path.abspath(os.path.expanduser(arg)) if not skeltarget: usage(sys.stderr, "dir must not be empty") sys.exit(2) if opt in ("-s", "--skelsrc"): skelsrc = os.path.abspath(os.path.expanduser(arg)) if not skelsrc: usage(sys.stderr, "skelsrc must not be empty") sys.exit(2) if opt in ("-h", "--help"): usage(sys.stdout) sys.exit() if opt in ("-u", "--user"): if not arg: usage(sys.stderr, "user must not be empty") sys.exit(2) if not ":" in arg: usage(sys.stderr, "user must be specified as name:password") sys.exit(2) user, password = arg.split(":", 1) if not skeltarget: # interactively ask for skeltarget and initial user name/passwd. # cant set custom instancehome in interactive mode, we default # to skeltarget. skeltarget = instancehome = os.path.abspath( os.path.expanduser(get_skeltarget()) ) instancehome = skeltarget zopehome = os.path.dirname(os.path.dirname(script)) softwarehome = os.path.join(zopehome, "lib", "python") configfile = os.path.join(instancehome, 'etc', 'zope.conf') if skelsrc is None: # default to using stock Zope skeleton source skelsrc = os.path.join(zopehome, "skel") inituser = os.path.join(instancehome, "inituser") if not (user or os.path.exists(inituser)): user, password = get_inituser() # we need to distinguish between python.exe and pythonw.exe under # Windows. Zope is always run using 'python.exe' (even for services), # however, it may be installed via pythonw.exe (as a sub-process of an # installer). Thus, sys.executable may not be the executable we use. # We still provide both PYTHON and PYTHONW, but PYTHONW should never # need be used. psplit = os.path.split(sys.executable) exedir = os.path.join(*psplit[:-1]) pythonexe = os.path.join(exedir, 'python.exe') pythonwexe = os.path.join(exedir, 'pythonw.exe') if ( os.path.isfile(pythonwexe) and os.path.isfile(pythonexe) and (sys.executable in [pythonwexe, pythonexe]) ): # we're using a Windows build with both python.exe and pythonw.exe # in the same directory PYTHON = pythonexe PYTHONW = pythonwexe else: # we're on UNIX or we have a nonstandard Windows setup PYTHON = PYTHONW = sys.executable kw = { "PYTHON":PYTHON, "PYTHONW":PYTHONW, "INSTANCE_HOME": instancehome, "SOFTWARE_HOME": softwarehome, "ZOPE_HOME": zopehome, } copyzopeskel.copyskel(skelsrc, skeltarget, None, None, **kw) if user and password: write_inituser(inituser, user, password) def usage(stream, msg=None): if msg: print >>stream, msg print >>stream program = os.path.basename(sys.argv[0]) print >>stream, __doc__ % {"program": program} def get_skeltarget(): print 'Please choose a directory in which you\'d like to install' print 'Zope "instance home" files such as database files, configuration' print 'files, etc.' print while 1: skeltarget = raw_input("Directory: ").strip() if skeltarget == '': print 'You must specify a directory' continue else: break return skeltarget def get_inituser(): import getpass print 'Please choose a username and password for the initial user.' print 'These will be the credentials you use to initially manage' print 'your new Zope instance.' print user = raw_input("Username: ").strip() if user == '': return None, None while 1: passwd = getpass.getpass("Password: ") verify = getpass.getpass("Verify password: ") if verify == passwd: break else: passwd = verify = '' print "Password mismatch, please try again..." return user, passwd def write_inituser(fn, user, password): import binascii import sha fp = open(fn, "w") pw = binascii.b2a_base64(sha.new(password).digest())[:-1] fp.write('%s:{SHA}%s\n' % (user, pw)) fp.close() os.chmod(fn, 0644) if __name__ == "__main__": main() Zope-2.8.7-final/utilities/reindex_catalog.py0000664000076400007640000000775710435500331020267 0ustar ajungajung############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """ A utility to perform external reindex operations for ZCatalogs Usage: zopectl run reindex_catalog.py [options] Use --help to get a list of all options. Author: Andreas Jung (andreas@andreas-jung.com) $Id$ """ import sys from optparse import OptionParser import transaction from Products.ZCatalog.ProgressHandler import StdoutHandler def path2catalog(path): """ lookup catalog by path """ catalog = app.restrictedTraverse(path, None) if not catalog: raise ValueError('No catalog found at %s' % path) return catalog def getHandler(options): """ return a progress handler """ if options.silent: return None else: return StdoutHandler(options.steps) def listIndexes(options, args): """ print a list of all indexes to stdout """ catalog = path2catalog(options.catalog) indexes = catalog._catalog.indexes print 'Listing of all indexes at %s' % options.catalog print for id, idx in indexes.items(): print '%-20s %-50s %d' % (id, idx.meta_type, idx.numObjects()) def reindexIndexes(optioons, args): """ reindex given list of indexes """ catalog = path2catalog(options.catalog) handler = getHandler(options) for id in args: print 'Reindexing index %s at %s' % (id, options.catalog) catalog.reindexIndex(id, None, handler) transaction.commit() def refreshMetadata(options, args): """ reindex metadata """ catalog = path2catalog(options.catalog) handler = getHandler(options) print 'Refresh metadata at %s' % options.catalog catalog.refreshMetadata(handler) transaction.commit() def reindexAll(options, args): """ reindex complete catalog """ catalog = path2catalog(options.catalog) handler = getHandler(options) print 'Reindexing complete ZCatalog at %s' % options.catalog catalog.refreshCatalog(options.clearCatalog, handler) transaction.commit() if __name__ == '__main__': parser = OptionParser() parser.add_option('-c', '--catalog', dest='catalog', help='path to ZCatalog') parser.add_option('-i', '--indexes', action="store_true", dest="listIndexes", help='list all indexes') parser.add_option('-C', '--clear', action="store_true", dest="clearCatalog", default=False, help='clear catalog before reindexing the complete catalog (--all only)') parser.add_option('-a', '--all', action="store_true", dest="reindexAll", help='reindex the complete catalog and update all metadata') parser.add_option('-r', '--reindex', action="store_true", dest="reindexIndexes", help='reindex specified indexes') parser.add_option('-m', '--metadata', action="store_true", dest="refreshMetadata", help='refresh all metadata') parser.add_option('-n', '--steps', dest="steps", default=100, type="int", help='log progress every N objects') parser.add_option('-s', '--silent', action="store_true", dest="silent", default=False, help='do not log reindexing progress to stdout') options,args = parser.parse_args() if options.listIndexes: listIndexes(options, args) if options.reindexIndexes: reindexIndexes(options, args) if options.refreshMetadata: refreshMetadata(options, args) if options.reindexAll: reindexAll(options, args) Zope-2.8.7-final/utilities/requestprofiler.py0000664000076400007640000006530010435500331020356 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Request log profiler script $Id: requestprofiler.py 24764 2004-05-17 06:13:48Z philikon $ """ import sys, time, getopt, math, cPickle from types import StringType try: import gzip except: pass class ProfileException(Exception): pass class Request: def __init__(self): self.url = None self.start = None self.method = None self.t_recdinput = None self.isize = None self.t_recdoutput = None self.osize = None self.httpcode = None self.t_end = None self.elapsed = "I" self.active = 0 def put(self, code, t, desc): if code not in ('A', 'B', 'I', 'E'): raise "unknown request code %s" % code if code == 'B': self.start = t self.method, self.url = desc.strip().split() elif code == "I": self.t_recdinput = t self.isize = desc.strip() elif code == "A": self.t_recdoutput = t self.httpcode, self.osize = desc.strip().split() elif code == 'E': self.t_end = t self.elapsed = int(self.t_end - self.start) def isfinished(self): return not self.elapsed == "I" def prettystart(self): if self.start is not None: t = time.localtime(self.start) return time.strftime('%Y-%m-%dT%H:%M:%S', t) else: return "NA" def shortprettystart(self): if self.start is not None: t = time.localtime(self.start) return time.strftime('%H:%M:%S', t) else: return "NA" def win(self): if self.t_recdinput is not None and self.start is not None: return self.t_recdinput - self.start else: return "NA" def wout(self): if self.t_recdoutput is not None and self.t_recdinput is not None: return self.t_recdoutput - self.t_recdinput else: return "NA" def wend(self): if self.t_end is not None and self.t_recdoutput is not None: return self.t_end - self.t_recdoutput else: return "NA" def endstage(self): if self.t_end is not None: stage = "E" elif self.t_recdoutput is not None: stage = "A" elif self.t_recdinput is not None: stage = "I" else: stage = "B" return stage def total(self): stage = self.endstage() if stage == "B": return 0 if stage == "I": return self.t_recdinput - self.start if stage == "A": return self.t_recdoutput - self.start if stage == "E": return self.elapsed def prettyisize(self): if self.isize is not None: return self.isize else: return "NA" def prettyosize(self): if self.osize is not None: return self.osize else: return "NA" def prettyhttpcode(self): if self.httpcode is not None: return self.httpcode else: return "NA" def __str__(self): body = ( self.prettystart(), self.win(), self.wout(), self.wend(), self.total(), self.endstage(), self.prettyosize(), self.prettyhttpcode(), self.active, self.url ) return self.fmt % body fmt = "%19s %4s %4s %4s %3s %1s %7s %4s %4s %s" def getheader(self): body = ('Start', 'WIn', 'WOut', 'WEnd', 'Tot', 'S', 'OSize', 'Code', 'Act', 'URL') return self.fmt % body class StartupRequest(Request): def endstage(self): return "U" def total(self): return 0 class Cumulative: def __init__(self, url): self.url = url self.times = [] self.hangs = 0 self.allelapsed = None def put(self, request): elapsed = request.elapsed if elapsed == "I": self.hangs = self.hangs + 1 self.times.append(elapsed) def all(self): if self.allelapsed == None: self.allelapsed = [] for elapsed in self.times: self.allelapsed.append(elapsed) self.allelapsed.sort() return self.allelapsed def __str__(self): body = ( self.hangs, self.hits(), self.total(), self.max(), self.min(), self.median(), self.mean(), self.url ) return self.fmt % body def getheader(self): return self.fmt % ('Hangs', 'Hits', 'Total', 'Max', 'Min', 'Median', 'Mean', 'URL') fmt = "%5s %5s %5s %5s %5s %6s %5s %s" def hits(self): return len(self.times) def max(self): return max(self.all()) def min(self): return min(self.all()) def mean(self): l = len(self.times) if l == 0: return "I" else: t = self.total() if t == "I": return "I" return t/l def median(self): all = self.all() l = len(all) if l == 0: return "I" else: if l == 1: return all[0] elif l % 2 != 0: i = l/2 + 1 return all[i] else: i = l/2 - 1 i2 = i + 1 v1 = all[i] v2 = all[i2] if isinstance(v1, StringType) or isinstance(v2, StringType): return "I" else: return (v1 + v2) / 2 def total(self): t = 0 all = self.all() for elapsed in all: if elapsed == "I": continue t = t + elapsed return t def parsebigmlogline(line): tup = line.split(None, 3) if len(tup) == 3: code, id, timestr = tup return code, id, timestr, '' elif len(tup) == 4: return tup else: return None def get_earliest_file_data(files): temp = {} earliest_fromepoch = 0 earliest = None retn = None for file in files: line = file.readline() if not line: continue linelen = len(line) line = line.strip() tup = parsebigmlogline(line) if tup is None: print "Could not interpret line: %s" % line continue code, id, timestr, desc = tup timestr = timestr.strip() fromepoch = getdate(timestr) temp[file] = linelen if earliest_fromepoch == 0 or fromepoch < earliest_fromepoch: earliest_fromepoch = fromepoch earliest = file retn = [code, id, fromepoch, desc] for file, linelen in temp.items(): if file is not earliest: file.seek(file.tell() - linelen) return retn def get_requests(files, start=None, end=None, statsfname=None, writestats=None, readstats=None): finished = [] unfinished = {} if readstats: fp = open(statsfname, 'r') u = cPickle.Unpickler(fp) requests = u.load() fp.close() del u del fp else: while 1: tup = get_earliest_file_data(files) if tup is None: break code, id, fromepoch, desc = tup if start is not None and fromepoch < start: continue if end is not None and fromepoch > end: break if code == 'U': finished.extend(unfinished.values()) unfinished.clear() request = StartupRequest() request.url = desc request.start = int(fromepoch) finished.append(request) continue request = unfinished.get(id) if request is None: if code != "B": continue # garbage at beginning of file request = Request() for pending_req in unfinished.values(): pending_req.active = pending_req.active + 1 unfinished[id] = request t = int(fromepoch) try: request.put(code, t, desc) except: print "Unable to handle entry: %s %s %s"%(code, t, desc) if request.isfinished(): del unfinished[id] finished.append(request) finished.extend(unfinished.values()) requests = finished if writestats: fp = open(statsfname, 'w') p = cPickle.Pickler(fp) p.dump(requests) fp.close() del p del fp return requests def analyze(requests, top, sortf, start=None, end=None, mode='cumulative', resolution=60, urlfocusurl=None, urlfocustime=60): if mode == 'cumulative': cumulative = {} for request in requests: url = request.url stats = cumulative.get(url) if stats is None: stats = Cumulative(url) cumulative[url] = stats stats.put(request) requests = cumulative.values() requests.sort(sortf) write(requests, top) elif mode=='timed': computed_start = requests[0].start computed_end = requests[-1].t_end if start and end: timewrite(requests,start,end,resolution) if start and not end: timewrite(requests,start,computed_end,resolution) if end and not start: timewrite(requests,computed_start,end,resolution) if not end and not start: timewrite(requests,computed_start,computed_end,resolution) elif mode == 'urlfocus': requests.sort(sortf) urlfocuswrite(requests, urlfocusurl, urlfocustime) else: requests.sort(sortf) write(requests, top) def urlfocuswrite(requests, url, t): l = [] i = 0 for request in requests: if request.url == url: l.append(i) i = i + 1 before = {} after = {} x = 0 for n in l: x = x + 1 r = requests[n] start = r.start earliest = start - t latest = start + t print 'URLs invoked %s seconds before and after %s (#%s, %s)' % \ (t, url, x, r.shortprettystart()) print '---' i = -1 for request in requests: i = i + 1 if request.start < earliest: continue if request.start > latest: break if n == i: # current request print '%3d' % (request.start - start), print '%s' % (request.shortprettystart()), print request.url continue if request.start <= start: if before.get(i): before[i] = before[i] + 1 else: before[i] = 1 if request.start > start: if after.get(i): after[i] = after[i] + 1 else: after[i] = 1 print '%3d' % (request.start - start), print '%s' % (request.shortprettystart()), print request.url print print ('Summary of URLs invoked before (and at the same time as) %s ' '(times, url)' % url) before = before.items() before.sort() for k,v in before: print v, requests[k].url print print 'Summary of URLs invoked after %s (times, url)' % url after = after.items() after.sort() for k,v in after: print v, requests[k].url def write(requests, top=0): if len(requests) == 0: print "No data.\n" return i = 0 header = requests[0].getheader() print header for stat in requests: i = i + 1 if verbose: print str(stat) else: print str(stat)[:78] if i == top: break def getdate(val): try: val = val.strip() year, month, day = int(val[:4]), int(val[5:7]), int(val[8:10]) hour,minute,second=int(val[11:13]),int(val[14:16]),int(val[17:19]) t = time.mktime((year, month, day, hour, minute, second, 0, 0, -1)) return t except: raise ProfileException, "bad date %s" % val def getTimeslice(period, utime): low = int(math.floor(utime)) - period + 1 high = int(math.ceil(utime)) + 1 for x in range(low, high): if x % period == 0: return x def timewrite(requests, start, end, resolution): print "Start: %s End: %s Resolution: %d secs" % \ (tick2str(start), tick2str(end), resolution) print "-" * 78 print print "Date/Time #requests requests/second" d = {} max = 0 min = None for r in requests: t = r.start slice = getTimeslice(resolution,t) if slice > max: max = slice if (min is None) or (slice < min): min = slice if d.has_key(slice): d[slice] = d[slice] + 1 else: d[slice] = 1 num = 0 hits = 0 avg_requests = None max_requests = 0 for slice in range(min, max, resolution): num = d.get(slice, 0) if num>max_requests: max_requests = num hits = hits + num if avg_requests is None: avg_requests = num else: avg_requests = (avg_requests + num) / 2 s = tick2str(slice) s = s + " %6d %4.2lf" % (num,num*1.0/resolution) print s print '='*78 print " Peak: %6d %4.2lf" % \ (max_requests,max_requests*1.0/resolution) print " Avg: %6d %4.2lf" % \ (avg_requests,avg_requests*1.0/resolution) print "Total: %6d n/a " % (hits) def tick2str(t): return time.strftime('%Y-%m-%dT%H:%M:%S', time.localtime(t)) def codesort(v1, v2): v1 = v1.endstage() v2 = v2.endstage() if v1 == v2: return 0 if v1 == "B": return -1 # v1 is smaller than v2 if v1 == "I": if v2 == "B": return 1 # v1 is larger than v2 else: return -1 if v1 == "A": if v2 in ['B', 'I']: return 1 else: return -1 if v1 == "E": return 1 class Sort: def __init__(self, fname, ascending=0): self.fname = fname self.ascending = ascending def __call__(self, i1, i2): f1 = getattr(i1, self.fname) f2 = getattr(i2, self.fname) if callable(f1): f1 = f1() if callable(f2): f2 = f2() if f1 < f2: if self.ascending: return -1 else: return 1 elif f1 == f2: return 0 else: if self.ascending: return 1 else: return -1 def detailedusage(): details = usage(0) pname = sys.argv[0] details = details + """ Reports are of four types: cumulative, detailed, timed, or urlfocus. The default is cumulative. Data is taken from one or more Zope detailed request logs (-M logs, aka 'big M' logs) or from a preprocessed statistics file. For cumulative reports, each line in the profile indicates information about a Zope method (URL) collected via a detailed request log. For detailed reports, each line in the profile indicates information about a single request. For timed reports, each line in the profile indicates informations about the number of requests and the number of requests/second for a period of time. For urlfocus reports, ad-hoc information about requests surrounding the specified url is given. Each 'filename' is a path to a '-M' log that contains detailed request data. Multiple input files can be analyzed at the same time by providing the path to each file. (Analyzing multiple big M log files at once is useful if you have more than one Zope client on a single machine and you'd like to get an overview of all Zope activity on that machine). If you wish to make multiple analysis runs against the same input data, you may want to use the --writestats option. The --writestats option creates a file which holds preprocessed data representing the specfified input files. Subsequent runs (for example with a different sort spec) will be much faster if the --readstats option is used to specify a preprocessed stats file instead of actual input files because the logfile parse step is skipped. If a 'sort' value is specified, sort the profile info by the spec. The sort order is descending unless indicated. The default cumulative sort spec is 'total'. The default detailed sort spec is 'start'. For cumulative reports, the following sort specs are accepted: 'hits' -- the number of hits against the method 'hangs' -- the number of unfinished requests to the method 'max' -- the maximum time in secs taken by a request to this method 'min' -- the minimum time in secs taken by a request to this method 'mean' -- the mean time in secs taken by a request to this method 'median' -- the median time in secs taken by a request to this method 'total' -- the total time in secs across all requests to this method 'url' -- the URL/method name (ascending) For detailed (non-cumulative) reports, the following sort specs are accepted: 'start' -- the start time of the request to ZServer (ascending) 'win' -- the num of secs ZServer spent waiting for input from client 'wout' -- the secs ZServer spent waiting for output from ZPublisher 'wend' -- the secs ZServer spent sending data to the client 'total' -- the secs taken for the request from begin to end 'endstage' -- the last successfully completed request stage (B, I, A, E) 'osize' -- the size in bytes of output provided by ZPublisher 'httpcode' -- the HTTP response code provided by ZPublisher (ascending) 'active' -- total num of requests pending at the end of this request 'url' -- the URL/method name (ascending) For timed and urlfocus reports, there are no sort specs allowed. If the 'top' argument is specified, only report on the top 'n' entries in the profile (as per the sort). The default is to show all data in the profile. If the 'verbose' argument is specified, do not trim url to fit into 80 cols. If the 'today' argument is specified, limit results to hits received today. If the 'daysago' argument is specified, limit results to hits received n days ago. The 'resolution' argument is used only for timed reports and specifies the number of seconds between consecutive lines in the report (default is 60 seconds). The 'urlfocustime' argument is used only for urlfocus reports and specifies the number of seconds to target before and after the URL provided in urlfocus mode. (default is 10 seconds). If the 'start' argument is specified in the form 'DD/MM/YYYY HH:MM:SS' (UTC), limit results to hits received after this date/time. If the 'end' argument is specified in the form 'DD/MM/YYYY HH:MM:SS' (UTC), limit results to hits received before this date/time. 'start' and 'end' arguments are not honored when request stats are obtained via the --readstats argument. Examples: %(pname)s debug.log Show cumulative report statistics for information in the file 'debug.log', by default sorted by 'total'. %(pname)s debug.log --detailed Show detailed report statistics sorted by 'start' (by default). %(pname)s debug.log debug2.log --detailed Show detailed report statistics for both logs sorted by 'start' (by default). %(pname)s debug.log --cumulative --sort=mean --today --verbose Show cumulative report statistics sorted by mean for entries in the log which happened today, and do not trim the URL in the resulting report. %(pname)s debug.log --cumulative --sort=mean --daysago=3 --verbose Show cumulative report statistics sorted by mean for entries in the log which happened three days ago, and do not trim the URL in the resulting report. %(pname)s debug.log --urlfocus='/manage_main' --urlfocustime=60 Show 'urlfocus' report which displays statistics about requests surrounding the invocation of '/manage_main'. Focus on the time periods 60 seconds before and after each invocation of the '/manage_main' URL. %(pname)s debug.log --detailed --start='2001/05/10 06:00:00' --end='2001/05/11 23:00:00' Show detailed report statistics for entries in 'debug.log' which begin after 6am UTC on May 10, 2001 and which end before 11pm UTC on May 11, 2001. %(pname)s debug.log --timed --resolution=300 --start='2001/05/10 06:00:00' --end='2001/05/11 23:00:00' Show timed report statistics for entries in the log for one day with a resolution of 5 minutes %(pname)s debug.log --top=100 --sort=max Show cumulative report of the the 'top' 100 methods sorted by maximum elapsed time. %(pname)s debug.log debug2.log --writestats='requests.stat' Write stats file for debug.log and debug2.log into 'requests.stat' and show default report. %(pname)s --readstats='requests.stat' --detailed Read from 'requests.stat' stats file (instead of actual -M log files) and show detailed report against this data.""" % {'pname':pname} return details def usage(basic=1): usage = ( """ Usage: %s filename1 [filename2 ...] [--cumulative | --detailed | [--timed --resolution=seconds]] [--sort=spec] [--top=n] [--verbose] [--today | [--start=date] [--end=date] | --daysago=n ] [--writestats=filename | --readstats=filename] [--urlfocus=url] [--urlfocustime=seconds] [--help] Provides a profile of one or more Zope "-M" request log files. """ % sys.argv[0] ) if basic == 1: usage = usage + """ If the --help argument is given, detailed usage docs are provided.""" return usage if __name__ == '__main__': if len(sys.argv) == 1: print usage() sys.exit(0) if sys.argv[1] == '--help': print detailedusage(); sys.exit(0) mode = 'cumulative' sortby = None trim = 0 top = 0 verbose = 0 start = None end = None resolution=60 urlfocustime=10 urlfocusurl=None statsfname = None readstats = 0 writestats = 0 files = [] i = 1 for arg in sys.argv[1:]: if arg[:2] != '--': if arg[-3:] == '.gz' and globals().has_key('gzip'): files.append(gzip.GzipFile(arg,'r')) else: files.append(open(arg)) sys.argv.remove(arg) i = i + 1 try: opts, extra = getopt.getopt( sys.argv[1:], '', ['sort=', 'top=', 'help', 'verbose', 'today', 'cumulative', 'detailed', 'timed','start=', 'end=','resolution=', 'writestats=','daysago=', 'readstats=','urlfocus=','urlfocustime='] ) for opt, val in opts: if opt=='--readstats': statsfname = val readstats = 1 elif opt=='--writestats': statsfname = val writestats = 1 if opt=='--sort': sortby = val if opt=='--top': top=int(val) if opt=='--help': print detailedusage(); sys.exit(0) if opt=='--verbose': verbose = 1 if opt=='--resolution': resolution=int(val) if opt=='--today': now = time.localtime(time.time()) # for testing - now = (2001, 04, 19, 0, 0, 0, 0, 0, -1) start = list(now) start[3] = start[4] = start[5] = 0 start = time.mktime(start) end = list(now) end[3] = 23; end[4] = 59; end[5] = 59 end = time.mktime(end) if opt=='--daysago': now = time.localtime(time.time() - int(val)*3600*24 ) # for testing - now = (2001, 04, 19, 0, 0, 0, 0, 0, -1) start = list(now) start[3] = start[4] = start[5] = 0 start = time.mktime(start) end = list(now) end[3] = 23; end[4] = 59; end[5] = 59 end = time.mktime(end) if opt=='--start': start = getdate(val) if opt=='--end': end = getdate(val) if opt=='--detailed': mode='detailed' d_sortby = sortby if opt=='--cumulative': mode='cumulative' if opt=='--timed': mode='timed' if opt=='--urlfocus': mode='urlfocus' urlfocusurl = val if opt=='--urlfocustime': urlfocustime=int(val) validcumsorts = ['url', 'hits', 'hangs', 'max', 'min', 'median', 'mean', 'total'] validdetsorts = ['start', 'win', 'wout', 'wend', 'total', 'endstage', 'isize', 'osize', 'httpcode', 'active', 'url'] if mode == 'cumulative': if sortby is None: sortby = 'total' assert sortby in validcumsorts, (sortby, mode, validcumsorts) if sortby in ['url']: sortf = Sort(sortby, ascending=1) else: sortf = Sort(sortby) elif mode == 'detailed': if sortby is None: sortby = 'start' assert sortby in validdetsorts, (sortby, mode, validdetsorts) if sortby in ['start', 'url', 'httpcode']: sortf = Sort(sortby, ascending=1) elif sortby == 'endstage': sortf = codesort else: sortf = Sort(sortby) elif mode=='timed': sortf = None elif mode=='urlfocus': sortf = Sort('start', ascending=1) else: raise 'Invalid mode' req=get_requests(files, start, end, statsfname, writestats, readstats) analyze(req, top, sortf, start, end, mode, resolution, urlfocusurl, urlfocustime) except AssertionError, val: a = "%s is not a valid %s sort spec, use one of %s" print a % (val[0], val[1], val[2]) sys.exit(0) except getopt.error, val: print val sys.exit(0) except ProfileException, val: print val sys.exit(0) except SystemExit: sys.exit(0) except: import traceback traceback.print_exc() print usage() sys.exit(0) Zope-2.8.7-final/utilities/README.txt0000664000076400007640000000166610435500331016254 0ustar ajungajungThis directory contains utility scripts and modules that augment Zope. To get detailed usage information, run any of these scripts without arguments: load_site.py -- Load a Zope site from files and directories This script illustrates used of the Zope RPC mechanism, ZPublisher.Client. It provides some examples of pitfalls and their work-arounds. check_catalog.py -- Perform some consistency tests on a ZCatalog instance mkzopeinstance.py -- create a Zope instance home copyzopeskel.py -- copy a Zope instance home skeleton directory to target mkzeoinstance.py -- create a ZEO instance home requestprofiler.py -- parse and analyze the Zope "detailed" log file zpasswd.py -- generate "access" or "inituser" files for use with Zope compilezpy.py -- compile all .py files to .pyc files in the current directory and below decompilezpy.py -- remove all .py[co] files in the current directory and below Zope-2.8.7-final/utilities/load_site.py0000664000076400007640000002037510435500331017071 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Load a Zope site from a collection of files or directories """ usage=""" [options] url file ..... where options are: -D For HTML documents, replace the start of the content, up to and including the opening body tag with a DTML var tag that inserts the standard header. Also replace the closing body and html tag with a DTML var tag that inserts the standard footer. -I For each index.html, add an index_html that redirects. -p path Path to ZPublisher. If not provided, load_site will make an attempt to figure it out. -u user:password Credentials -v Run in verbose mode. -9 Use *old* zope method names. """ import sys, getopt, os, string ServerError='' verbose=0 old=0 doctor=0 index_html=0 def main(): user, password = 'superuser', '123' opts, args = getopt.getopt(sys.argv[1:], 'p:u:DIv9') global verbose global old global doctor global index_html havepath=None for o, v in opts: if o=='-p': d, f = os.path.split(v) if f=='ZPublisher': sys.path.insert(0,d) else: sys.path.insert(0,v) havepath=1 elif o=='-u': v = string.split(v,':') user, password = v[0], string.join(v[1:],':') elif o=='-D': doctor=1 elif o=='-I': index_html=1 elif o=='-v': verbose=1 elif o=='-9': old=1 if not args: print sys.argv[0]+usage sys.exit(1) if not havepath: here=os.path.split(sys.argv[0])[0] if os.path.exists(os.path.join(here,'ZPublisher')): sys.path.insert(0,here) else: here=os.path.split(here)[0] here=os.path.join(here,'lib','python') if os.path.exists(os.path.join(here,'ZPublisher')): sys.path.insert(0,here) url=args[0] files=args[1:] import ZPublisher.Client global ServerError ServerError=ZPublisher.Client.ServerError object=ZPublisher.Client.Object(url, username=user, password=password) for f in files: upload_file(object, f) def call(f, *args, **kw): # Call a function ignoring redirect bci errors. try: apply(f,args, kw) except ServerError, v: if str(v)[:1] != '3': raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2] def upload_file(object, f): if os.path.isdir(f): return upload_dir(object, f) dir, name = os.path.split(f) root, ext = os.path.splitext(name) if ext in ('file', 'dir'): ext='' else: ext=string.lower(ext) if ext and ext[0] in '.': ext=ext[1:] if ext and globals().has_key('upload_'+ext): if verbose: print 'upload_'+ext, f return globals()['upload_'+ext](object, f) if verbose: print 'upload_file', f, ext call(object.manage_addFile, id=name, file=open(f,'rb')) def upload_dir(object, f): if verbose: print 'upload_dir', f dir, name = os.path.split(f) call(object.manage_addFolder, id=name) object=object.__class__(object.url+'/'+name, username=object.username, password=object.password) for n in os.listdir(f): upload_file(object, os.path.join(f,n)) # ----- phd ----- # Modified by Oleg Broytmann from sgmllib import SGMLParser def join_attrs(attrs): attr_list = [] for attrname, value in attrs: attr_list.append('%s="%s"' % (attrname, string.strip(value))) if attr_list: s = " " + string.join(attr_list, " ") else: s = "" return s class HeadParser(SGMLParser): def __init__(self): SGMLParser.__init__(self) self.seen_starthead = 0 self.seen_endhead = 0 self.seen_startbody = 0 self.head = "" self.title = "" self.accumulator = "" def handle_data(self, data): if data: self.accumulator = self.accumulator + data def handle_charref(self, ref): self.handle_data("&#%s;" % ref) def handle_entityref(self, ref): self.handle_data("&%s;" % ref) def handle_comment(self, data): if data: self.accumulator = self.accumulator + "" % data def start_head(self, attrs): if not self.seen_starthead: self.seen_starthead = 1 self.head = "" self.title = "" self.accumulator = "" def end_head(self): if not self.seen_endhead: self.seen_endhead = 1 self.head = self.head + self.accumulator self.accumulator = "" def start_title(self, attrs): self.head = self.head + self.accumulator self.accumulator = "" def end_title(self): self.title = self.accumulator self.accumulator = "" def start_body(self, attrs): if not self.seen_startbody: self.seen_startbody = 1 self.accumulator = "" def end_body(self): pass # Do not put and def end_html(self): pass # into output stream # Pass other tags unmodified def unknown_starttag(self, tag, attrs): self.accumulator = self.accumulator + "<%s%s>" % (string.upper(tag), join_attrs(attrs)) def unknown_endtag(self, tag): self.accumulator = self.accumulator + "" % string.upper(tag) def parse_html(infile): parser = HeadParser() while 1: line = infile.readline() if not line: break parser.feed(line) parser.close() infile.close() return (string.strip(parser.title), string.strip(parser.head), string.strip(parser.accumulator)) def upload_html(object, f): dir, name = os.path.split(f) f=open(f) if doctor: title, head, body = parse_html(f) if old: body = ("\n\n" + body + "\n\n") else: body = ("\n\n" + body + "\n\n") else: if old: f=f.read() title, head, body = '', '', f if old: call(object.manage_addDocument, id=name, file=body) if index_html and name in ('index.html', 'index.htm'): call(object.manage_addDocument, id='index_html', file=('' '/%s' '' % name )) else: call(object.manage_addDTMLDocument, id=name, title=title, file=body) if index_html and name in ('index.html', 'index.htm'): call(object.manage_addDTMLMethod, id='index_html', file=('' '/%s' '' % name )) # Now add META and other tags as property if head: object=object.__class__(object.url+'/'+name, username=object.username, password=object.password) call(object.manage_addProperty, id="loadsite-head", type="text", value=head) # ----- /phd ----- upload_htm=upload_html def upload_dtml(object, f): dir, name = os.path.split(f) f=open(f) if old: f=f.read() call(object.manage_addDocument, id=name, file=f) else: call(object.manage_addDTMLMethod, id=name, file=f) def upload_gif(object, f): dir, name = os.path.split(f) call(object.manage_addImage, id=name, file=open(f,'rb')) upload_jpg=upload_gif upload_png=upload_gif if __name__=='__main__': main() Zope-2.8.7-final/utilities/zpasswd.py0000775000076400007640000001417410435500331016624 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001,2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Zope user bootstrap system Usage: %(PROGRAM)s [options] filename If this program is called without command-line options, it will prompt for all necessary information. The available options are: -u / --username= Set the username to be used for the initial user or the emergency user -p / --password= Set the password -e / --encoding= Set the encryption/encoding rules. Defaults to SHA-1. OPTIONAL -d / --domains= Set the domain names that the user user can log in from. Defaults to any. OPTIONAL. -h / --help Print this help text and exit. Filename is required and should be the name of the file to store the information in (usually "inituser" or "access"). """ import sys, sha, binascii, random, getopt, getpass, os try: from crypt import crypt except ImportError: crypt = None PROGRAM = sys.argv[0] COMMASPACE = ', ' def generate_salt(): """Generate a salt value for the crypt function.""" salt_choices = ("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789./") return random.choice(salt_choices)+random.choice(salt_choices) def generate_passwd(password, encoding): encoding=encoding.upper() if encoding == 'SHA': pw = '{SHA}' + binascii.b2a_base64(sha.new(password).digest())[:-1] elif encoding == 'CRYPT': pw = '{CRYPT}' + crypt(password, generate_salt()) elif encoding == 'CLEARTEXT': pw = password return pw def write_generated_password(home, ac_path, username): pw_choices = ("ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789!") acfile=open(ac_path, 'w') pw = '' for i in range(8): pw = pw + random.choice(pw_choices) acfile.write('%s:%s\n' % (username, generate_passwd(pw, 'SHA'))) acfile.close() os.chmod(ac_path, 0644) return pw def write_access(home, user='', group=''): ac_path=os.path.join(home, 'access') if not os.path.exists(ac_path): print '-'*78 print 'creating default access file' pw = write_generated_password(home, ac_path, 'emergency') print """Note: The emergency user name and password are 'emergency' and '%s'. You can change the emergency name and password with the zpasswd script. To find out more, type: %s zpasswd.py """ % (pw, sys.executable) import do; do.ch(ac_path, user, group) def get_password(): while 1: password = getpass.getpass("Password: ") verify = getpass.getpass("Verify password: ") if verify == password: return password else: password = verify = '' print "Password mismatch, please try again..." def write_inituser(home, user='', group=''): ac_path=os.path.join(home, 'inituser') if not os.path.exists(ac_path): print '-'*78 print 'creating default inituser file' pw = write_generated_password(home, ac_path, 'admin') print """Note: The initial user name and password are 'admin' and '%s'. You can change the name and password through the web interface or using the 'zpasswd.py' script. """ % pw import do; do.ch(ac_path, user, group) def usage(code, msg=''): print >> sys.stderr, __doc__ % globals() if msg: print >> sys.stderr, msg sys.exit(code) def main(): shortopts = 'u:p:e:d:h' longopts = ['username=', 'password=', 'encoding=', 'domains=', 'help'] try: opts, args = getopt.getopt(sys.argv[1:], shortopts, longopts) except getopt.error, msg: usage(1, msg) # Defaults username = password = None domains = '' encoding = 'SHA' for opt, arg in opts: if opt in ('-h', '--help'): usage(0) elif opt in ('-u', '--username'): username = arg elif opt in ('-p', '--password'): password = arg elif opt in ('-e', '--encoding'): encoding = arg elif opt in ('-d', '--domains'): domains = ':' + arg # Extra command line arguments? if len(args) == 0: usage(1, 'filename is required') elif len(args) == 1: access_file = open(args[0], 'w') else: usage(1, 'Extra command line arguments: ' + COMMASPACE.join(args)) if opts: # There were some command line args, so verify if username is not None and password is None: password = get_password() else: # No command line args, so prompt while 1: username = raw_input("Username: ") if username != '': break password = get_password() while 1: print """ Please choose a format from: SHA - SHA-1 hashed password (default) CRYPT - UNIX-style crypt password CLEARTEXT - no protection """ encoding = raw_input("Encoding: ") if encoding == '': encoding = 'SHA' break if encoding.upper() in ['SHA', 'CRYPT', 'CLEARTEXT']: break domains = raw_input("Domain restrictions: ") if domains: domains = ":" + domains # Done with prompts and args access_file.write(username + ":" + generate_passwd(password, encoding) + domains) # If called from the command line if __name__=='__main__': main() Zope-2.8.7-final/utilities/mkzeoinstance.py0000775000076400007640000000175310435500331020002 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## import os import sys mydir = os.path.dirname(os.path.abspath(sys.argv[0])) zopehome = os.path.dirname(mydir) softwarehome = os.path.join(zopehome, "lib", "python") if softwarehome not in sys.path: sys.path.insert(0, softwarehome) from ZEO.mkzeoinst import ZEOInstanceBuilder if __name__ == "__main__": ZEOInstanceBuilder().run() Zope-2.8.7-final/utilities/copyzopeskel.py0000775000076400007640000002223210435500331017652 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """%(program)s: Create a target directory from a skeleton directory. usage: %(program)s [options] Options: -h/--help -- print this help text -s/--sourcedir -- the skeleton source directory -t/--targetdir -- the directory to which the skeleton files will be copied -u/--uid -- the username/uid of the user who will own the target files -g/--gid -- the groupname/gid of the group who will own the target files -r/--replace -- specify replacement value for .in file This script may be used to install a custom Zope instance home skeleton directory. It is most useful when used to install a skeleton which does not follow the standard 'one-directory-as-instance-home' paradigm used by the stock Zope source install. The value of --targetdir should be the directory where you would like to copy the skeleton hierarchy. For many packagers, this will be "/" or "/usr" or "/usr/local". The value of --sourcedir should be a directory which contains a custom skeleton hierarchy. For many packagers, the skeleton source directory may contain directories like "usr" and "bin" and these directories will contain files and other directories, comprising an installation hierarchy suitable for your platform. The skeleton source hierarchy may contain any kind of file. Files in the skeleton hierarchy that end with a ".in" extension will go through textual substitution before they are placed in the target directory. When they are placed in the target directory, the ".in" extension is removed. Specify textual replacement values by passing one or more --replace= options to the script. The value of each replace option needs to be in the following format: --replace=key:value. 'key' is the value that will be replaced (minus the "<<" and ">>" values expected by the replacement). 'value' is the value that should be used for replacement. Files which do not have an ".in" extension are copied without substitution. All file mode bits from files/dirs in the skeleton source directory are copied along with the file/directory itself. If the --uid and/or --gid flags are used, all directories and files created by this script will be provided with this owner and/or group. Otherwise, the uid and group owner of the files will be the executing user's. Existing directory structures and files are left unchanged. If a file already exists in the target directory, it is left unchanged and the source file is not copied. If a directory already exists in the target directory, its ownership information and mode bit settings are left unchanged. """ import os import shutil import sys import getopt def main(): try: opts, args = getopt.getopt(sys.argv[1:], "hs:t:u:g:r:", ["help", "sourcedir=", "targetdir=", "uid=", "gid=", "replace="] ) except getopt.GetoptError, msg: usage(sys.stderr, msg) sys.exit(2) script = os.path.abspath(sys.argv[0]) sourcedir = None targetdir = None uid = None gid = None replacements = {} for opt, arg in opts: if opt in ("-h", "--help"): usage(sys.stdout) sys.exit() if opt in ("-s", "--sourcedir"): sourcedir = os.path.abspath(os.path.expanduser(arg)) if not sourcedir: usage(sys.stderr, "sourcedir must not be empty") sys.exit(2) if opt in ("-t", "--targetdir"): targetdir = os.path.abspath(os.path.expanduser(arg)) if not targetdir: usage(sys.stderr, "targetdir must not be empty") sys.exit(2) if opt in ("-u", "--uid"): if not arg: usage(sys.stderr, "uid must not be empty") sys.exit(2) try: if os.getuid() != 0: usage(sys.stderr, "You must be root to specify a uid") sys.exit(2) try: uid = int(arg) except: try: import pwd uid = pwd.getpwnam(arg)[2] if not gid: gid = pwd.getpwnam(arg)[3] except KeyError: usage(sys.stderr, "The user indicated by uid does not exist on " "your system") sys.exit(2) except (ImportError, AttributeError): usage(sys.stderr, "Your system does not support the gid or uid options") sys.exit(2) if opt in ("-g", "--gid"): if not arg: usage(sys.stderr, "gid must not be empty") sys.exit(2) try: if os.getuid() != 0: usage(sys.stderr, "You must be root to specify a gid") sys.exit(2) try: gid = int(arg) except: try: import pwd gid = pwd.getpwnam(arg)[3] except KeyError: usage(sys.stderr, "The user indicated by gid does not exist on " "your system") sys.exit(2) except (ImportError, AttributeError): usage(sys.stderr, "Your system does not support the gid or uid options") sys.exit(2) if opt in ("-r", "--replace"): if not arg: continue k, v = arg.split(':', 1) replacements[k] = v if not sourcedir: usage(sys.stderr, "Must specify sourcedir") sys.exit(2) if not targetdir: usage(sys.stderr, "Must specify targetdir") sys.exit(2) copyskel(sourcedir, targetdir, uid, gid, **replacements) def copyskel(sourcedir, targetdir, uid, gid, **replacements): """ This is an independent function because we'd like to import and call it from mkzopeinstance """ # Create the top of the instance: if not os.path.exists(targetdir): os.makedirs(targetdir) # This is fairly ugly. The chdir() makes path manipulation in the # walk() callback a little easier (less magical), so we'll live # with it. pwd = os.getcwd() os.chdir(sourcedir) try: try: os.path.walk(os.curdir, copydir, (targetdir, replacements, uid, gid)) finally: os.chdir(pwd) except (IOError, OSError), msg: print >>sys.stderr, msg sys.exit(1) CVS_DIRS = [os.path.normcase("CVS"), os.path.normcase(".svn")] def copydir((targetdir, replacements, uid, gid), sourcedir, names): # Don't recurse into CVS directories: for name in names[:]: if os.path.normcase(name) in CVS_DIRS: names.remove(name) elif os.path.isfile(os.path.join(sourcedir, name)): # Copy the file: sn, ext = os.path.splitext(name) if os.path.normcase(ext) == ".in": dst = os.path.join(targetdir, sourcedir, sn) if os.path.exists(dst): continue copyin(os.path.join(sourcedir, name), dst, replacements, uid, gid) if uid is not None: os.chown(dst, uid, gid) else: src = os.path.join(sourcedir, name) dst = os.path.join(targetdir, src) if os.path.exists(dst): continue shutil.copyfile(src, dst) shutil.copymode(src, dst) if uid is not None: os.chown(dst, uid, gid) else: # Directory: dn = os.path.join(targetdir, sourcedir, name) if not os.path.exists(dn): os.mkdir(dn) shutil.copymode(os.path.join(sourcedir, name), dn) if uid is not None: os.chown(dn, uid, gid) def copyin(src, dst, replacements, uid, gid): ifp = open(src) text = ifp.read() ifp.close() for k in replacements: text = text.replace("<<%s>>" % k, replacements[k]) ofp = open(dst, "w") ofp.write(text) ofp.close() shutil.copymode(src, dst) if uid is not None: os.chown(dst, uid, gid) def usage(stream, msg=None): if msg: print >>stream, msg print >>stream program = os.path.basename(sys.argv[0]) print >>stream, __doc__ % {"program": program} if __name__ == '__main__': main() Zope-2.8.7-final/utilities/decompilezpy.py0000775000076400007640000000177410435500331017637 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001 Zope Corporation and Contributors. All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## import os import sys def main(dirname): os.path.walk(dirname, rmpycs, None) def rmpycs(arg, dirname, names): for name in names: path = os.path.join(dirname, name) if ( name.endswith('.pyc') or name.endswith('.pyo') and os.path.isfile(path) ): os.unlink(path) if __name__ == '__main__': main(sys.argv[1]) Zope-2.8.7-final/utilities/ZODBTools/0000775000076400007640000000000010435500417016331 5ustar ajungajungZope-2.8.7-final/utilities/ZODBTools/migrate.py0000775000076400007640000002576110435500417020351 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001, 2002, 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """A script to gather statistics while doing a storage migration. This is very similar to a standard storage's copyTransactionsFrom() method, except that it's geared to run as a script, and it collects useful pieces of information as it's working. This script can be used to stress test a storage since it blasts transactions at it as fast as possible. You can get a good sense of the performance of a storage by running this script. Actually it just counts the size of pickles in the transaction via the iterator protocol, so storage overheads aren't counted. Usage: %(PROGRAM)s [options] [source-storage-args] [destination-storage-args] Options: -S sourcetype --stype=sourcetype This is the name of a recognized type for the source database. Use -T to print out the known types. Defaults to "file". -D desttype --dtype=desttype This is the name of the recognized type for the destination database. Use -T to print out the known types. Defaults to "file". -o filename --output=filename Print results in filename, otherwise stdout. -m txncount --max=txncount Stop after committing txncount transactions. -k txncount --skip=txncount Skip the first txncount transactions. -p/--profile Turn on specialized profiling. -t/--timestamps Print tids as timestamps. -T/--storage_types Print all the recognized storage types and exit. -v/--verbose Turns on verbose output. Multiple -v options increase the verbosity. -h/--help Print this message and exit. Positional arguments: source-storage-args: Semicolon separated list of arguments for the source storage, as key=val pairs. E.g. "file_name=Data.fs;read_only=1" destination-storage-args: Comma separated list of arguments for the source storage, as key=val pairs. E.g. "name=full;frequency=3600" """ import re import sys import time import getopt import marshal import profile from ZODB import utils from ZODB import StorageTypes from ZODB.TimeStamp import TimeStamp PROGRAM = sys.argv[0] ZERO = '\0'*8 try: True, False except NameError: True = 1 False = 0 def usage(code, msg=''): print >> sys.stderr, __doc__ % globals() if msg: print >> sys.stderr, msg sys.exit(code) def error(code, msg): print >> sys.stderr, msg print "use --help for usage message" sys.exit(code) def main(): try: opts, args = getopt.getopt( sys.argv[1:], 'hvo:pm:k:D:S:Tt', ['help', 'verbose', 'output=', 'profile', 'storage_types', 'max=', 'skip=', 'dtype=', 'stype=', 'timestamps']) except getopt.error, msg: error(2, msg) class Options: stype = 'FileStorage' dtype = 'FileStorage' verbose = 0 outfile = None profilep = False maxtxn = -1 skiptxn = -1 timestamps = False options = Options() for opt, arg in opts: if opt in ('-h', '--help'): usage(0) elif opt in ('-v', '--verbose'): options.verbose += 1 elif opt in ('-T', '--storage_types'): print_types() sys.exit(0) elif opt in ('-S', '--stype'): options.stype = arg elif opt in ('-D', '--dtype'): options.dtype = arg elif opt in ('-o', '--output'): options.outfile = arg elif opt in ('-p', '--profile'): options.profilep = True elif opt in ('-m', '--max'): options.maxtxn = int(arg) elif opt in ('-k', '--skip'): options.skiptxn = int(arg) elif opt in ('-t', '--timestamps'): options.timestamps = True if len(args) > 2: error(2, "too many arguments") srckws = {} if len(args) > 0: srcargs = args[0] for kv in re.split(r';\s*', srcargs): key, val = kv.split('=') srckws[key] = val destkws = {} if len(args) > 1: destargs = args[1] for kv in re.split(r';\s*', destargs): key, val = kv.split('=') destkws[key] = val if options.stype not in StorageTypes.storage_types.keys(): usage(2, 'Source database type must be provided') if options.dtype not in StorageTypes.storage_types.keys(): usage(2, 'Destination database type must be provided') # Open the output file if options.outfile is None: options.outfp = sys.stdout options.outclosep = False else: options.outfp = open(options.outfile, 'w') options.outclosep = True if options.verbose > 0: print 'Opening source database...' modname, sconv = StorageTypes.storage_types[options.stype] kw = sconv(**srckws) __import__(modname) sclass = getattr(sys.modules[modname], options.stype) srcdb = sclass(**kw) if options.verbose > 0: print 'Opening destination database...' modname, dconv = StorageTypes.storage_types[options.dtype] kw = dconv(**destkws) __import__(modname) dclass = getattr(sys.modules[modname], options.dtype) dstdb = dclass(**kw) try: t0 = time.time() doit(srcdb, dstdb, options) t1 = time.time() if options.verbose > 0: print 'Migration time: %8.3f' % (t1-t0) finally: # Done srcdb.close() dstdb.close() if options.outclosep: options.outfp.close() def doit(srcdb, dstdb, options): outfp = options.outfp profilep = options.profilep verbose = options.verbose # some global information largest_pickle = 0 largest_txn_in_size = 0 largest_txn_in_objects = 0 total_pickle_size = 0L total_object_count = 0 # Ripped from BaseStorage.copyTransactionsFrom() ts = None ok = True prevrevids = {} counter = 0 skipper = 0 if options.timestamps: print "%4s. %26s %6s %8s %5s %5s %5s %5s %5s" % ( "NUM", "TID AS TIMESTAMP", "OBJS", "BYTES", # Does anybody know what these times mean? "t4-t0", "t1-t0", "t2-t1", "t3-t2", "t4-t3") else: print "%4s. %20s %6s %8s %6s %6s %6s %6s %6s" % ( "NUM", "TRANSACTION ID", "OBJS", "BYTES", # Does anybody know what these times mean? "t4-t0", "t1-t0", "t2-t1", "t3-t2", "t4-t3") for txn in srcdb.iterator(): skipper += 1 if skipper <= options.skiptxn: continue counter += 1 if counter > options.maxtxn >= 0: break tid = txn.tid if ts is None: ts = TimeStamp(tid) else: t = TimeStamp(tid) if t <= ts: if ok: print >> sys.stderr, \ 'Time stamps are out of order %s, %s' % (ts, t) ok = False ts = t.laterThan(ts) tid = `ts` else: ts = t if not ok: print >> sys.stderr, \ 'Time stamps are back in order %s' % t ok = True if verbose > 1: print ts prof = None if profilep and (counter % 100) == 0: prof = profile.Profile() objects = 0 size = 0 newrevids = RevidAccumulator() t0 = time.time() dstdb.tpc_begin(txn, tid, txn.status) t1 = time.time() for r in txn: oid = r.oid objects += 1 thissize = len(r.data) size += thissize if thissize > largest_pickle: largest_pickle = thissize if verbose > 1: if not r.version: vstr = 'norev' else: vstr = r.version print utils.U64(oid), vstr, len(r.data) oldrevid = prevrevids.get(oid, ZERO) result = dstdb.store(oid, oldrevid, r.data, r.version, txn) newrevids.store(oid, result) t2 = time.time() result = dstdb.tpc_vote(txn) t3 = time.time() newrevids.tpc_vote(result) prevrevids.update(newrevids.get_dict()) # Profile every 100 transactions if prof: prof.runcall(dstdb.tpc_finish, txn) else: dstdb.tpc_finish(txn) t4 = time.time() # record the results if objects > largest_txn_in_objects: largest_txn_in_objects = objects if size > largest_txn_in_size: largest_txn_in_size = size if options.timestamps: tidstr = str(TimeStamp(tid)) format = "%4d. %26s %6d %8d %5.3f %5.3f %5.3f %5.3f %5.3f" else: tidstr = utils.U64(tid) format = "%4d. %20s %6d %8d %6.4f %6.4f %6.4f %6.4f %6.4f" print >> outfp, format % (skipper, tidstr, objects, size, t4-t0, t1-t0, t2-t1, t3-t2, t4-t3) total_pickle_size += size total_object_count += objects if prof: prof.create_stats() fp = open('profile-%02d.txt' % (counter / 100), 'wb') marshal.dump(prof.stats, fp) fp.close() print >> outfp, "Largest pickle: %8d" % largest_pickle print >> outfp, "Largest transaction: %8d" % largest_txn_in_size print >> outfp, "Largest object count: %8d" % largest_txn_in_objects print >> outfp, "Total pickle size: %14d" % total_pickle_size print >> outfp, "Total object count: %8d" % total_object_count # helper to deal with differences between old-style store() return and # new-style store() return that supports ZEO import types class RevidAccumulator: def __init__(self): self.data = {} def _update_from_list(self, list): for oid, serial in list: if not isinstance(serial, types.StringType): raise serial self.data[oid] = serial def store(self, oid, result): if isinstance(result, types.StringType): self.data[oid] = result elif result is not None: self._update_from_list(result) def tpc_vote(self, result): if result is not None: self._update_from_list(result) def get_dict(self): return self.data if __name__ == '__main__': main() Zope-2.8.7-final/utilities/ZODBTools/fsdump.py0000664000076400007640000000030610435500417020200 0ustar ajungajung#!/usr/bin/env python2.3 """Print a text summary of the contents of a FileStorage.""" from ZODB.FileStorage.fsdump import fsdump if __name__ == "__main__": import sys fsdump(sys.argv[1]) Zope-2.8.7-final/utilities/ZODBTools/netspace.py0000664000076400007640000000632410435500417020512 0ustar ajungajung#!/usr/bin/env python2.3 """Report on the net size of objects counting subobjects. usage: netspace.py [-P | -v] data.fs -P: do a pack first -v: print info for all objects, even if a traversal path isn't found """ import ZODB from ZODB.FileStorage import FileStorage from ZODB.utils import U64, get_pickle_metadata from ZODB.referencesf import referencesf def find_paths(root, maxdist): """Find Python attribute traversal paths for objects to maxdist distance. Starting at a root object, traverse attributes up to distance levels from the root, looking for persistent objects. Return a dict mapping oids to traversal paths. TODO: Assumes that the keys of the root are not themselves persistent objects. TODO: Doesn't traverse containers. """ paths = {} # Handle the root as a special case because it's a dict objs = [] for k, v in root.items(): oid = getattr(v, '_p_oid', None) objs.append((k, v, oid, 0)) for path, obj, oid, dist in objs: if oid is not None: paths[oid] = path if dist < maxdist: getattr(obj, 'foo', None) # unghostify try: items = obj.__dict__.items() except AttributeError: continue for k, v in items: oid = getattr(v, '_p_oid', None) objs.append(("%s.%s" % (path, k), v, oid, dist + 1)) return paths def main(path): fs = FileStorage(path, read_only=1) if PACK: fs.pack() db = ZODB.DB(fs) rt = db.open().root() paths = find_paths(rt, 3) def total_size(oid): cache = {} cache_size = 1000 def _total_size(oid, seen): v = cache.get(oid) if v is not None: return v data, serialno = fs.load(oid, '') size = len(data) for suboid in referencesf(data): if seen.has_key(suboid): continue seen[suboid] = 1 size += _total_size(suboid, seen) cache[oid] = size if len(cache) == cache_size: cache.popitem() return size return _total_size(oid, {}) keys = fs._index.keys() keys.sort() keys.reverse() if not VERBOSE: # If not running verbosely, don't print an entry for an object # unless it has an entry in paths. keys = filter(paths.has_key, keys) fmt = "%8s %5d %8d %s %s.%s" for oid in keys: data, serialno = fs.load(oid, '') mod, klass = get_pickle_metadata(data) refs = referencesf(data) path = paths.get(oid, '-') print fmt % (U64(oid), len(data), total_size(oid), path, mod, klass) if __name__ == "__main__": import sys import getopt PACK = 0 VERBOSE = 0 try: opts, args = getopt.getopt(sys.argv[1:], 'Pv') path, = args except getopt.error, err: print err print __doc__ sys.exit(2) except ValueError: print "expected one argument, got", len(args) print __doc__ sys.exit(2) for o, v in opts: if o == '-P': PACK = 1 if o == '-v': VERBOSE += 1 main(path) Zope-2.8.7-final/utilities/ZODBTools/fsrefs.py0000664000076400007640000001353410435500417020201 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Check FileStorage for dangling references. usage: fsrefs.py [-v] data.fs fsrefs.py checks object sanity by trying to load the current revision of every object O in the database, and also verifies that every object directly reachable from each such O exists in the database. It's hard to explain exactly what it does because it relies on undocumented features in Python's cPickle module: many of the crucial steps of loading an object are taken, but application objects aren't actually created. This saves a lot of time, and allows fsrefs to be run even if the code implementing the object classes isn't available. A read-only connection to the specified FileStorage is made, but it is not recommended to run fsrefs against a live FileStorage. Because a live FileStorage is mutating while fsrefs runs, it's not possible for fsrefs to get a wholly consistent view of the database across the entire time fsrefs is running; spurious error messages may result. fsrefs doesn't normally produce any output. If an object fails to load, the oid of the object is given in a message saying so, and if -v was specified then the traceback corresponding to the load failure is also displayed (this is the only effect of the -v flag). Three other kinds of errors are also detected, when an object O loads OK, and directly refers to a persistent object P but there's a problem with P: - If P doesn't exist in the database, a message saying so is displayed. The unsatisifiable reference to P is often called a "dangling reference"; P is called "missing" in the error output. - If the current state of the database is such that P's creation has been undone, then P can't be loaded either. This is also a kind of dangling reference, but is identified as "object creation was undone". - If P can't be loaded (but does exist in the database), a message saying that O refers to an object that can't be loaded is displayed. fsrefs also (indirectly) checks that the .index file is sane, because fsrefs uses the index to get its idea of what constitutes "all the objects in the database". Note these limitations: because fsrefs only looks at the current revision of objects, it does not attempt to load objects in versions, or non-current revisions of objects; therefore fsrefs cannot find problems in versions or in non-current revisions. """ import traceback import types from ZODB.FileStorage import FileStorage from ZODB.TimeStamp import TimeStamp from ZODB.utils import u64, oid_repr, get_refs, get_pickle_metadata from ZODB.POSException import POSKeyError VERBOSE = 0 # There's a problem with oid. 'data' is its pickle, and 'serial' its # serial number. 'missing' is a list of (oid, class, reason) triples, # explaining what the problem(s) is(are). def report(oid, data, serial, missing): from_mod, from_class = get_pickle_metadata(data) if len(missing) > 1: plural = "s" else: plural = "" ts = TimeStamp(serial) print "oid %s %s.%s" % (hex(u64(oid)), from_mod, from_class) print "last updated: %s, tid=%s" % (ts, hex(u64(serial))) print "refers to invalid object%s:" % plural for oid, info, reason in missing: if isinstance(info, types.TupleType): description = "%s.%s" % info else: description = str(info) print "\toid %s %s: %r" % (oid_repr(oid), reason, description) print def main(path): fs = FileStorage(path, read_only=1) # Set of oids in the index that failed to load due to POSKeyError. # This is what happens if undo is applied to the transaction creating # the object (the oid is still in the index, but its current data # record has a backpointer of 0, and POSKeyError is raised then # because of that backpointer). undone = {} # Set of oids that were present in the index but failed to load. # This does not include oids in undone. noload = {} for oid in fs._index.keys(): try: data, serial = fs.load(oid, "") except (KeyboardInterrupt, SystemExit): raise except POSKeyError: undone[oid] = 1 except: if VERBOSE: traceback.print_exc() noload[oid] = 1 inactive = noload.copy() inactive.update(undone) for oid in fs._index.keys(): if oid in inactive: continue data, serial = fs.load(oid, "") refs = get_refs(data) missing = [] # contains 3-tuples of oid, klass-metadata, reason for info in refs: try: ref, klass = info except (ValueError, TypeError): # failed to unpack ref = info klass = '' if ref not in fs._index: missing.append((ref, klass, "missing")) if ref in noload: missing.append((ref, klass, "failed to load")) if ref in undone: missing.append((ref, klass, "object creation was undone")) if missing: report(oid, data, serial, missing) if __name__ == "__main__": import sys import getopt opts, args = getopt.getopt(sys.argv[1:], "v") for k, v in opts: if k == "-v": VERBOSE += 1 path, = args main(path) Zope-2.8.7-final/utilities/ZODBTools/zeoreplay.py0000664000076400007640000002045710435500417020725 0ustar ajungajung#!/usr/bin/env python2.3 """Parse the BLATHER logging generated by ZEO, and optionally replay it. Usage: zeointervals.py [options] Options: --help / -h Print this message and exit. --replay=storage -r storage Replay the parsed transactions through the new storage --maxtxn=count -m count Parse no more than count transactions. --report / -p Print a report as we're parsing. Unlike parsezeolog.py, this script generates timestamps for each transaction, and sub-command in the transaction. We can use this to compare timings with synthesized data. """ import re import sys import time import getopt import operator # ZEO logs measure wall-clock time so for consistency we need to do the same #from time import clock as now from time import time as now from ZODB.FileStorage import FileStorage #from BDBStorage.BDBFullStorage import BDBFullStorage #from Standby.primary import PrimaryStorage #from Standby.config import RS_PORT from ZODB.Transaction import Transaction from ZODB.utils import p64 datecre = re.compile('(\d\d\d\d-\d\d-\d\d)T(\d\d:\d\d:\d\d)') methcre = re.compile("ZEO Server (\w+)\((.*)\) \('(.*)', (\d+)") class StopParsing(Exception): pass def usage(code, msg=''): print __doc__ if msg: print msg sys.exit(code) def parse_time(line): """Return the time portion of a zLOG line in seconds or None.""" mo = datecre.match(line) if mo is None: return None date, time_ = mo.group(1, 2) date_l = [int(elt) for elt in date.split('-')] time_l = [int(elt) for elt in time_.split(':')] return int(time.mktime(date_l + time_l + [0, 0, 0])) def parse_line(line): """Parse a log entry and return time, method info, and client.""" t = parse_time(line) if t is None: return None, None, None mo = methcre.search(line) if mo is None: return None, None, None meth_name = mo.group(1) meth_args = mo.group(2) meth_args = [s.strip() for s in meth_args.split(',')] m = meth_name, tuple(meth_args) c = mo.group(3), mo.group(4) return t, m, c class StoreStat: def __init__(self, when, oid, size): self.when = when self.oid = oid self.size = size # Crufty def __getitem__(self, i): if i == 0: return self.oid if i == 1: return self.size raise IndexError class TxnStat: def __init__(self): self._begintime = None self._finishtime = None self._aborttime = None self._url = None self._objects = [] def tpc_begin(self, when, args, client): self._begintime = when # args are txnid, user, description (looks like it's always a url) self._url = args[2] def storea(self, when, args, client): oid = int(args[0]) # args[1] is "[numbytes]" size = int(args[1][1:-1]) s = StoreStat(when, oid, size) self._objects.append(s) def tpc_abort(self, when): self._aborttime = when def tpc_finish(self, when): self._finishtime = when # Mapping oid -> revid _revids = {} class ReplayTxn(TxnStat): def __init__(self, storage): self._storage = storage self._replaydelta = 0 TxnStat.__init__(self) def replay(self): ZERO = '\0'*8 t0 = now() t = Transaction() self._storage.tpc_begin(t) for obj in self._objects: oid = obj.oid revid = _revids.get(oid, ZERO) # BAW: simulate a pickle of the given size data = 'x' * obj.size # BAW: ignore versions for now newrevid = self._storage.store(p64(oid), revid, data, '', t) _revids[oid] = newrevid if self._aborttime: self._storage.tpc_abort(t) origdelta = self._aborttime - self._begintime else: self._storage.tpc_vote(t) self._storage.tpc_finish(t) origdelta = self._finishtime - self._begintime t1 = now() # Shows how many seconds behind (positive) or ahead (negative) of the # original reply our local update took self._replaydelta = t1 - t0 - origdelta class ZEOParser: def __init__(self, maxtxns=-1, report=1, storage=None): self.__txns = [] self.__curtxn = {} self.__skipped = 0 self.__maxtxns = maxtxns self.__finishedtxns = 0 self.__report = report self.__storage = storage def parse(self, line): t, m, c = parse_line(line) if t is None: # Skip this line return name = m[0] meth = getattr(self, name, None) if meth is not None: meth(t, m[1], c) def tpc_begin(self, when, args, client): txn = ReplayTxn(self.__storage) self.__curtxn[client] = txn meth = getattr(txn, 'tpc_begin', None) if meth is not None: meth(when, args, client) def storea(self, when, args, client): txn = self.__curtxn.get(client) if txn is None: self.__skipped += 1 return meth = getattr(txn, 'storea', None) if meth is not None: meth(when, args, client) def tpc_finish(self, when, args, client): txn = self.__curtxn.get(client) if txn is None: self.__skipped += 1 return meth = getattr(txn, 'tpc_finish', None) if meth is not None: meth(when) if self.__report: self.report(txn) self.__txns.append(txn) self.__curtxn[client] = None self.__finishedtxns += 1 if self.__maxtxns > 0 and self.__finishedtxns >= self.__maxtxns: raise StopParsing def report(self, txn): """Print a report about the transaction""" if txn._objects: bytes = reduce(operator.add, [size for oid, size in txn._objects]) else: bytes = 0 print '%s %s %4d %10d %s %s' % ( txn._begintime, txn._finishtime - txn._begintime, len(txn._objects), bytes, time.ctime(txn._begintime), txn._url) def replay(self): for txn in self.__txns: txn.replay() # How many fell behind? slower = [] faster = [] for txn in self.__txns: if txn._replaydelta > 0: slower.append(txn) else: faster.append(txn) print len(slower), 'laggards,', len(faster), 'on-time or faster' # Find some averages if slower: sum = reduce(operator.add, [txn._replaydelta for txn in slower], 0) print 'average slower txn was:', float(sum) / len(slower) if faster: sum = reduce(operator.add, [txn._replaydelta for txn in faster], 0) print 'average faster txn was:', float(sum) / len(faster) def main(): try: opts, args = getopt.getopt( sys.argv[1:], 'hr:pm:', ['help', 'replay=', 'report', 'maxtxns=']) except getopt.error, e: usage(1, e) if args: usage(1) replay = 0 maxtxns = -1 report = 0 storagefile = None for opt, arg in opts: if opt in ('-h', '--help'): usage(0) elif opt in ('-r', '--replay'): replay = 1 storagefile = arg elif opt in ('-p', '--report'): report = 1 elif opt in ('-m', '--maxtxns'): try: maxtxns = int(arg) except ValueError: usage(1, 'Bad -m argument: %s' % arg) if replay: storage = FileStorage(storagefile) #storage = BDBFullStorage(storagefile) #storage = PrimaryStorage('yyz', storage, RS_PORT) t0 = now() p = ZEOParser(maxtxns, report, storage) i = 0 while 1: line = sys.stdin.readline() if not line: break i += 1 try: p.parse(line) except StopParsing: break except: print 'input file line:', i raise t1 = now() print 'total parse time:', t1-t0 t2 = now() if replay: p.replay() t3 = now() print 'total replay time:', t3-t2 print 'total time:', t3-t0 if __name__ == '__main__': main() Zope-2.8.7-final/utilities/ZODBTools/README.txt0000664000076400007640000001143410435500417020032 0ustar ajungajungThis directory contains a collection of utilities for managing ZODB databases. Some are more useful than others. If you install ZODB using distutils ("python setup.py install"), fsdump.py, fstest.py, repozo.py, and zeopack.py will be installed in /usr/local/bin. Unless otherwise noted, these scripts are invoked with the name of the Data.fs file as their only argument. Example: checkbtrees.py data.fs. analyze.py -- a transaction analyzer for FileStorage Reports on the data in a FileStorage. The report is organized by class. It shows total data, as well as separate reports for current and historical revisions of objects. checkbtrees.py -- checks BTrees in a FileStorage for corruption Attempts to find all the BTrees contained in a Data.fs, calls their _check() methods, and runs them through BTrees.check.check(). fsdump.py -- summarize FileStorage contents, one line per revision Prints a report of FileStorage contents, with one line for each transaction and one line for each data record in that transaction. Includes time stamps, file positions, and class names. fsoids.py -- trace all uses of specified oids in a FileStorage For heavy debugging. A set of oids is specified by text file listing and/or command line. A report is generated showing all uses of these oids in the database: all new-revision creation/modifications, all references from all revisions of other objects, and all creation undos. fstest.py -- simple consistency checker for FileStorage usage: fstest.py [-v] data.fs The fstest tool will scan all the data in a FileStorage and report an error if it finds any corrupt transaction data. The tool will print a message when the first error is detected an exit. The tool accepts one or more -v arguments. If a single -v is used, it will print a line of text for each transaction record it encounters. If two -v arguments are used, it will also print a line of text for each object. The objects for a transaction will be printed before the transaction itself. Note: It does not check the consistency of the object pickles. It is possible for the damage to occur only in the part of the file that stores object pickles. Those errors will go undetected. space.py -- report space used by objects in a FileStorage usage: space.py [-v] data.fs This ignores revisions and versions. netspace.py -- hackish attempt to report on size of objects usage: netspace.py [-P | -v] data.fs -P: do a pack first -v: print info for all objects, even if a traversal path isn't found Traverses objects from the database root and attempts to calculate size of object, including all reachable subobjects. parsezeolog.py -- parse BLATHER logs from ZEO server This script may be obsolete. It has not been tested against the current log output of the ZEO server. Reports on the time and size of transactions committed by a ZEO server, by inspecting log messages at BLATHER level. repozo.py -- incremental backup utility for FileStorage Run the script with the -h option to see usage details. timeout.py -- script to test transaction timeout usage: timeout.py address delay [storage-name] This script connects to a storage, begins a transaction, calls store() and tpc_vote(), and then sleeps forever. This should trigger the transaction timeout feature of the server. zeopack.py -- pack a ZEO server The script connects to a server and calls pack() on a specific storage. See the script for usage details. zeoreplay.py -- experimental script to replay transactions from a ZEO log Like parsezeolog.py, this may be obsolete because it was written against an earlier version of the ZEO server. See the script for usage details. zeoup.py usage: zeoup.py [options] The test will connect to a ZEO server, load the root object, and attempt to update the zeoup counter in the root. It will report success if it updates to counter or if it gets a ConflictError. A ConflictError is considered a success, because the client was able to start a transaction. See the script for details about the options. zodbload.py -- exercise ZODB under a heavy synthesized Zope-like load See the module docstring for details. Note that this script requires Zope. New in ZODB3 3.1.4. zeoserverlog.py -- analyze ZEO server log for performance statistics See the module docstring for details; there are a large number of options. New in ZODB3 3.1.4. fsrefs.py -- check FileStorage for dangling references fstail.py -- display the most recent transactions in a FileStorage usage: fstail.py [-n nxtn] data.fs The most recent ntxn transactions are displayed, to stdout. Optional argument -n specifies ntxn, and defaults to 10. migrate.py -- do a storage migration and gather statistics See the module docstring for details. zeoqueue.py -- report number of clients currently waiting in the ZEO queue See the module docstring for details. Zope-2.8.7-final/utilities/ZODBTools/zeoup.py0000775000076400007640000000711510435500417020054 0ustar ajungajung#!/usr/bin/env python2.3 """Make sure a ZEO server is running. usage: zeoup.py [options] The test will connect to a ZEO server, load the root object, and attempt to update the zeoup counter in the root. It will report success if it updates the counter or if it gets a ConflictError. A ConflictError is considered a success, because the client was able to start a transaction. Options: -p port -- port to connect to -h host -- host to connect to (default is current host) -S storage -- storage name (default '1') -U path -- Unix-domain socket to connect to --nowrite -- Do not update the zeoup counter. -1 -- Connect to a ZEO 1.0 server. You must specify either -p and -h or -U. """ import getopt import socket import sys import time from persistent.mapping import PersistentMapping import transaction import ZODB from ZODB.POSException import ConflictError from ZODB.tests.MinPO import MinPO from ZEO.ClientStorage import ClientStorage ZEO_VERSION = 2 def check_server(addr, storage, write): t0 = time.time() if ZEO_VERSION == 2: # TODO: should do retries w/ exponential backoff. cs = ClientStorage(addr, storage=storage, wait=0, read_only=(not write)) else: cs = ClientStorage(addr, storage=storage, debug=1, wait_for_server_on_startup=1) # _startup() is an artifact of the way ZEO 1.0 works. The # ClientStorage doesn't get fully initialized until registerDB() # is called. The only thing we care about, though, is that # registerDB() calls _startup(). if write: db = ZODB.DB(cs) cn = db.open() root = cn.root() try: # We store the data in a special `monitor' dict under the root, # where other tools may also store such heartbeat and bookkeeping # type data. monitor = root.get('monitor') if monitor is None: monitor = root['monitor'] = PersistentMapping() obj = monitor['zeoup'] = monitor.get('zeoup', MinPO(0)) obj.value += 1 transaction.commit() except ConflictError: pass cn.close() db.close() else: data, serial = cs.load("\0\0\0\0\0\0\0\0", "") cs.close() t1 = time.time() print "Elapsed time: %.2f" % (t1 - t0) def usage(exit=1): print __doc__ print " ".join(sys.argv) sys.exit(exit) def main(): host = None port = None unix = None write = 1 storage = '1' try: opts, args = getopt.getopt(sys.argv[1:], 'p:h:U:S:1', ['nowrite']) for o, a in opts: if o == '-p': port = int(a) elif o == '-h': host = a elif o == '-U': unix = a elif o == '-S': storage = a elif o == '--nowrite': write = 0 elif o == '-1': ZEO_VERSION = 1 except Exception, err: s = str(err) if s: s = ": " + s print err.__class__.__name__ + s usage() if unix is not None: addr = unix else: if host is None: host = socket.gethostname() if port is None: usage() addr = host, port check_server(addr, storage, write) if __name__ == "__main__": try: main() except SystemExit: raise except Exception, err: s = str(err) if s: s = ": " + s print err.__class__.__name__ + s sys.exit(1) Zope-2.8.7-final/utilities/ZODBTools/manual_tests/0000775000076400007640000000000010435500417021030 5ustar ajungajungZope-2.8.7-final/utilities/ZODBTools/manual_tests/testfstest.py0000664000076400007640000001264610435500417023623 0ustar ajungajung"""Verify that fstest.py can find errors. Note: To run this test script fstest.py must be on your PYTHONPATH. """ from cStringIO import StringIO import os import re import struct import tempfile import unittest import fstest from fstest import FormatError, U64 class TestCorruptedFS(unittest.TestCase): f = open('test-checker.fs', 'rb') datafs = f.read() f.close() del f def setUp(self): self._temp = tempfile.mktemp() self._file = open(self._temp, 'wb') def tearDown(self): if not self._file.closed: self._file.close() if os.path.exists(self._temp): try: os.remove(self._temp) except os.error: pass def noError(self): if not self._file.closed: self._file.close() fstest.check(self._temp) def detectsError(self, rx): if not self._file.closed: self._file.close() try: fstest.check(self._temp) except FormatError, msg: mo = re.search(rx, str(msg)) self.failIf(mo is None, "unexpected error: %s" % msg) else: self.fail("fstest did not detect corruption") def getHeader(self): buf = self._datafs.read(16) if not buf: return 0, '' tl = U64(buf[8:]) return tl, buf def copyTransactions(self, n): """Copy at most n transactions from the good data""" f = self._datafs = StringIO(self.datafs) self._file.write(f.read(4)) for i in range(n): tl, data = self.getHeader() if not tl: return self._file.write(data) rec = f.read(tl - 8) self._file.write(rec) def testGood(self): self._file.write(self.datafs) self.noError() def testTwoTransactions(self): self.copyTransactions(2) self.noError() def testEmptyFile(self): self.detectsError("empty file") def testInvalidHeader(self): self._file.write('SF12') self.detectsError("invalid file header") def testTruncatedTransaction(self): self._file.write(self.datafs[:4+22]) self.detectsError("truncated") def testCheckpointFlag(self): self.copyTransactions(2) tl, data = self.getHeader() assert tl > 0, "ran out of good transaction data" self._file.write(data) self._file.write('c') self._file.write(self._datafs.read(tl - 9)) self.detectsError("checkpoint flag") def testInvalidStatus(self): self.copyTransactions(2) tl, data = self.getHeader() assert tl > 0, "ran out of good transaction data" self._file.write(data) self._file.write('Z') self._file.write(self._datafs.read(tl - 9)) self.detectsError("invalid status") def testTruncatedRecord(self): self.copyTransactions(3) tl, data = self.getHeader() assert tl > 0, "ran out of good transaction data" self._file.write(data) buf = self._datafs.read(tl / 2) self._file.write(buf) self.detectsError("truncated possibly") def testBadLength(self): self.copyTransactions(2) tl, data = self.getHeader() assert tl > 0, "ran out of good transaction data" self._file.write(data) buf = self._datafs.read(tl - 8) self._file.write(buf[0]) assert tl <= 1<<16, "can't use this transaction for this test" self._file.write("\777\777") self._file.write(buf[3:]) self.detectsError("invalid transaction header") def testDecreasingTimestamps(self): self.copyTransactions(0) tl, data = self.getHeader() buf = self._datafs.read(tl - 8) t1 = data + buf tl, data = self.getHeader() buf = self._datafs.read(tl - 8) t2 = data + buf self._file.write(t2[:8] + t1[8:]) self._file.write(t1[:8] + t2[8:]) self.detectsError("time-stamp") def testTruncatedData(self): # This test must re-write the transaction header length in # order to trigger the error in check_drec(). If it doesn't, # the truncated data record would also caught a truncated # transaction record. self.copyTransactions(1) tl, data = self.getHeader() pos = self._file.tell() self._file.write(data) buf = self._datafs.read(tl - 8) hdr = buf[:15] ul, dl, el = struct.unpack(">HHH", hdr[-6:]) self._file.write(buf[:15 + ul + dl + el]) data = buf[15 + ul + dl + el:] self._file.write(data[:24]) self._file.seek(pos + 8, 0) newlen = struct.pack(">II", 0, tl - (len(data) - 24)) self._file.write(newlen) self.detectsError("truncated at") def testBadDataLength(self): self.copyTransactions(1) tl, data = self.getHeader() self._file.write(data) buf = self._datafs.read(tl - 8) hdr = buf[:7] # write the transaction meta data ul, dl, el = struct.unpack(">HHH", hdr[-6:]) self._file.write(buf[:7 + ul + dl + el]) # write the first part of the data header data = buf[7 + ul + dl + el:] self._file.write(data[:24]) self._file.write("\000" * 4 + "\077" + "\000" * 3) self._file.write(data[32:]) self.detectsError("record exceeds transaction") if __name__ == "__main__": unittest.main() Zope-2.8.7-final/utilities/ZODBTools/manual_tests/testzeopack.py0000664000076400007640000000645610435500417023751 0ustar ajungajung# Some simple tests for zeopack.py # For this to work, zeopack.py must by on your PATH. from ZODB.FileStorage import FileStorage from ZODB.tests.StorageTestBase import StorageTestBase from ZEO.tests import forker import ZODB import os import socket import tempfile import threading import time import unittest # TODO: The forker interface isn't clearly defined. It's different on # different branches of ZEO. This will break someday. # TODO: Only handle the Unix variant of the forker. Just to give Tim # something to do. class PackerTests(StorageTestBase): def setUp(self): self.started = 0 def start(self): self.started =1 self.path = tempfile.mktemp(suffix=".fs") self._storage = FileStorage(self.path) self.db = ZODB.DB(self._storage) self.do_updates() self.pid, self.exit = forker.start_zeo_server(self._storage, self.addr) def do_updates(self): for i in range(100): self._dostore() def tearDown(self): if not self.started: return self.db.close() self._storage.close() self.exit.close() try: os.kill(self.pid, 9) except os.error: pass try: os.waitpid(self.pid, 0) except os.error, err: ##print "waitpid failed", err pass for ext in '', '.old', '.lock', '.index', '.tmp': path = self.path + ext try: os.remove(path) except os.error: pass def set_inet_addr(self): self.host = socket.gethostname() self.port = forker.get_port() self.addr = self.host, self.port def testPack(self): self.set_inet_addr() self.start() status = os.system("zeopack.py -h %s -p %s" % (self.host, self.port)) assert status == 0 assert os.path.exists(self.path + ".old") def testPackDays(self): self.set_inet_addr() self.start() status = os.system("zeopack.py -h %s -p %s -d 1" % (self.host, self.port)) # Since we specified one day, nothing should get packed assert status == 0 assert not os.path.exists(self.path + ".old") def testAF_UNIXPack(self): self.addr = tempfile.mktemp(suffix=".zeo-socket") self.start() status = os.system("zeopack.py -U %s" % self.addr) assert status == 0 assert os.path.exists(self.path + ".old") def testNoServer(self): status = os.system("zeopack.py -p 19") assert status != 0 def testWaitForServer(self): self.set_inet_addr() def delayed_start(): time.sleep(11) self.start() t = threading.Thread(target=delayed_start) t.start() status = os.system("zeopack.py -h %s -p %s -W" % (self.host, self.port)) t.join() assert status == 0 assert os.path.exists(self.path + ".old") class UpTest(unittest.TestCase): def testUp(self): status = os.system("zeoup.py -p 19") # There is no ZEO server on port 19, so we should see non-zero # exit status. assert status != 0 if __name__ == "__main__": unittest.main() Zope-2.8.7-final/utilities/ZODBTools/manual_tests/testrepozo.py0000664000076400007640000001052010435500417023616 0ustar ajungajung#!/usr/bin/env python ############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test repozo.py. This is a by-hand test. It succeeds iff it doesn't blow up. Run it with its home directory as the current directory. It will destroy all files matching Data.* and Copy.* in this directory, and anything in a subdirectory of name 'backup'. """ import os import random import time import glob import sys import shutil import ZODB from ZODB import FileStorage import transaction PYTHON = sys.executable + ' ' def cleanup(): for fname in glob.glob('Data.*') + glob.glob('Copy.*'): os.remove(fname) if os.path.isdir('backup'): for fname in os.listdir('backup'): os.remove(os.path.join('backup', fname)) os.rmdir('backup') class OurDB: def __init__(self): from BTrees.OOBTree import OOBTree self.getdb() conn = self.db.open() conn.root()['tree'] = OOBTree() transaction.commit() self.close() def getdb(self): storage = FileStorage.FileStorage('Data.fs') self.db = ZODB.DB(storage) def gettree(self): self.getdb() conn = self.db.open() return conn.root()['tree'] def pack(self): self.getdb() self.db.pack() def close(self): if self.db is not None: self.db.close() self.db = None # Do recovery to time 'when', and check that it's identical to correctpath. def check(correctpath='Data.fs', when=None): if when is None: extra = '' else: extra = ' -D ' + when cmd = PYTHON + '../repozo.py -vRr backup -o Copy.fs' + extra os.system(cmd) f = file(correctpath, 'rb') g = file('Copy.fs', 'rb') fguts = f.read() gguts = g.read() f.close() g.close() if fguts != gguts: raise ValueError("guts don't match\n" " correctpath=%r when=%r\n" " cmd=%r" % (correctpath, when, cmd)) def mutatedb(db): # Make random mutations to the btree in the database. tree = db.gettree() for dummy in range(100): if random.random() < 0.6: tree[random.randrange(100000)] = random.randrange(100000) else: keys = tree.keys() if keys: del tree[keys[0]] transaction.commit() db.close() def main(): cleanup() os.mkdir('backup') d = OurDB() # Every 9th time thru the loop, we save a full copy of Data.fs, # and at the end we ensure we can reproduce those too. saved_snapshots = [] # list of (name, time) pairs for copies. for i in range(100): # Make some mutations. mutatedb(d) # Pack about each tenth time. if random.random() < 0.1: print "packing" d.pack() d.close() # Make an incremental backup, half the time with gzip (-z). if random.random() < 0.5: os.system(PYTHON + '../repozo.py -vBQr backup -f Data.fs') else: os.system(PYTHON + '../repozo.py -zvBQr backup -f Data.fs') if i % 9 == 0: copytime = '%04d-%02d-%02d-%02d-%02d-%02d' % (time.gmtime()[:6]) copyname = os.path.join('backup', "Data%d" % i) + '.fs' shutil.copyfile('Data.fs', copyname) saved_snapshots.append((copyname, copytime)) # Make sure the clock moves at least a second. time.sleep(1.01) # Verify current Data.fs can be reproduced exactly. check() # Verify snapshots can be reproduced exactly. for copyname, copytime in saved_snapshots: print "Checking that", copyname, "at", copytime, "is reproducible." check(copyname, copytime) # Tear it all down. cleanup() print 'Test passed!' if __name__ == '__main__': main() Zope-2.8.7-final/utilities/ZODBTools/manual_tests/test-checker.fs0000664000076400007640000000144210435500417023744 0ustar ajungajungFS21?4\B3 initial database creation?4\B39(cPersistence PersistentMapping qNt.}qU _containerq}s.?4" ?4"4Y((U PersistenceqUPersistentMappingqtqNt.}qU _containerq}qUaUThis is a test.qss.?5\w1 ?5\wA((U PersistenceqUPersistentMappingqtqNt.}qU _containerq}q(UaUThis is a test.qK(Uq(hUPersistentMappingq ttQus.?5\wAB((U PersistenceqUPersistentMappingqtqNt.}qU _containerq}qs.1?5.5w ?5.5wz_((U PersistenceqUPersistentMappingqtqNt.}qU _containerq}qUaUThis is another test.qss.Zope-2.8.7-final/utilities/ZODBTools/parsezeolog.py0000664000076400007640000000647610435500417021252 0ustar ajungajung#!/usr/bin/env python2.3 """Parse the BLATHER logging generated by ZEO2. An example of the log format is: 2002-04-15T13:05:29 BLATHER(-100) ZEO Server storea(3235680, [714], 235339406490168806) ('10.0.26.30', 45514) """ import re import time rx_time = re.compile('(\d\d\d\d-\d\d-\d\d)T(\d\d:\d\d:\d\d)') def parse_time(line): """Return the time portion of a zLOG line in seconds or None.""" mo = rx_time.match(line) if mo is None: return None date, time_ = mo.group(1, 2) date_l = [int(elt) for elt in date.split('-')] time_l = [int(elt) for elt in time_.split(':')] return int(time.mktime(date_l + time_l + [0, 0, 0])) rx_meth = re.compile("zrpc:\d+ calling (\w+)\((.*)") def parse_method(line): pass def parse_line(line): """Parse a log entry and return time, method info, and client.""" t = parse_time(line) if t is None: return None, None mo = rx_meth.search(line) if mo is None: return None, None meth_name = mo.group(1) meth_args = mo.group(2).strip() if meth_args.endswith(')'): meth_args = meth_args[:-1] meth_args = [s.strip() for s in meth_args.split(",")] m = meth_name, tuple(meth_args) return t, m class TStats: counter = 1 def __init__(self): self.id = TStats.counter TStats.counter += 1 fields = ("time", "vote", "done", "user", "path") fmt = "%-24s %5s %5s %-15s %s" hdr = fmt % fields def report(self): """Print a report about the transaction""" t = time.ctime(self.begin) if hasattr(self, "vote"): d_vote = self.vote - self.begin else: d_vote = "*" if hasattr(self, "finish"): d_finish = self.finish - self.begin else: d_finish = "*" print self.fmt % (time.ctime(self.begin), d_vote, d_finish, self.user, self.url) class TransactionParser: def __init__(self): self.txns = {} self.skipped = 0 def parse(self, line): t, m = parse_line(line) if t is None: return name = m[0] meth = getattr(self, name, None) if meth is not None: meth(t, m[1]) def tpc_begin(self, time, args): t = TStats() t.begin = time t.user = args[1] t.url = args[2] t.objects = [] tid = eval(args[0]) self.txns[tid] = t def get_txn(self, args): tid = eval(args[0]) try: return self.txns[tid] except KeyError: print "uknown tid", repr(tid) return None def tpc_finish(self, time, args): t = self.get_txn(args) if t is None: return t.finish = time def vote(self, time, args): t = self.get_txn(args) if t is None: return t.vote = time def get_txns(self): L = [(t.id, t) for t in self.txns.values()] L.sort() return [t for (id, t) in L] if __name__ == "__main__": import fileinput p = TransactionParser() i = 0 for line in fileinput.input(): i += 1 try: p.parse(line) except: print "line", i raise print "Transaction: %d" % len(p.txns) print TStats.hdr for txn in p.get_txns(): txn.report() Zope-2.8.7-final/utilities/ZODBTools/space.py0000664000076400007640000000303710435500417020001 0ustar ajungajung#!/usr/bin/env python2.3 """Report on the space used by objects in a storage. usage: space.py data.fs The current implementation only supports FileStorage. Current limitations / simplifications: Ignores revisions and versions. """ from ZODB.FileStorage import FileStorage from ZODB.utils import U64, get_pickle_metadata def run(path, v=0): fs = FileStorage(path, read_only=1) # break into the file implementation if hasattr(fs._index, 'iterkeys'): iter = fs._index.iterkeys() else: iter = fs._index.keys() totals = {} for oid in iter: data, serialno = fs.load(oid, '') mod, klass = get_pickle_metadata(data) key = "%s.%s" % (mod, klass) bytes, count = totals.get(key, (0, 0)) bytes += len(data) count += 1 totals[key] = bytes, count if v: print "%8s %5d %s" % (U64(oid), len(data), key) L = totals.items() L.sort(lambda a, b: cmp(a[1], b[1])) L.reverse() print "Totals per object class:" for key, (bytes, count) in L: print "%8d %8d %s" % (count, bytes, key) def main(): import sys import getopt try: opts, args = getopt.getopt(sys.argv[1:], "v") except getopt.error, msg: print msg print "usage: space.py [-v] Data.fs" sys.exit(2) if len(args) != 1: print "usage: space.py [-v] Data.fs" sys.exit(2) v = 0 for o, a in opts: if o == "-v": v += 1 path = args[0] run(path, v) if __name__ == "__main__": main() Zope-2.8.7-final/utilities/ZODBTools/timeout.py0000775000076400007640000000325310435500417020377 0ustar ajungajung#!/usr/bin/env python2.3 """Transaction timeout test script. This script connects to a storage, begins a transaction, calls store() and tpc_vote(), and then sleeps forever. This should trigger the transaction timeout feature of the server. usage: timeout.py address delay [storage-name] """ import sys import time from ZODB.Transaction import Transaction from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_pickle from ZEO.ClientStorage import ClientStorage ZERO = '\0'*8 def main(): if len(sys.argv) not in (3, 4): sys.stderr.write("Usage: timeout.py address delay [storage-name]\n" % sys.argv[0]) sys.exit(2) hostport = sys.argv[1] delay = float(sys.argv[2]) if sys.argv[3:]: name = sys.argv[3] else: name = "1" if "/" in hostport: address = hostport else: if ":" in hostport: i = hostport.index(":") host, port = hostport[:i], hostport[i+1:] else: host, port = "", hostport port = int(port) address = (host, port) print "Connecting to %s..." % repr(address) storage = ClientStorage(address, name) print "Connected. Now starting a transaction..." oid = storage.new_oid() version = "" revid = ZERO data = MinPO("timeout.py") pickled_data = zodb_pickle(data) t = Transaction() t.user = "timeout.py" storage.tpc_begin(t) storage.store(oid, revid, pickled_data, version, t) print "Stored. Now voting..." storage.tpc_vote(t) print "Voted; now sleeping %s..." % delay time.sleep(delay) print "Done." if __name__ == "__main__": main() Zope-2.8.7-final/utilities/ZODBTools/repozo.py0000775000076400007640000004121110435500417020223 0ustar ajungajung#!/usr/bin/env python2.3 # repozo.py -- incremental and full backups of a Data.fs file. # # Originally written by Anthony Baxter # Significantly modified by Barry Warsaw """repozo.py -- incremental and full backups of a Data.fs file. Usage: %(program)s [options] Where: Exactly one of -B or -R must be specified: -B / --backup Backup current ZODB file. -R / --recover Restore a ZODB file from a backup. -v / --verbose Verbose mode. -h / --help Print this text and exit. -r dir --repository=dir Repository directory containing the backup files. This argument is required. The directory must already exist. You should not edit the files in this directory, or add your own files to it. Options for -B/--backup: -f file --file=file Source Data.fs file. This argument is required. -F / --full Force a full backup. By default, an incremental backup is made if possible (e.g., if a pack has occurred since the last incremental backup, a full backup is necessary). -Q / --quick Verify via md5 checksum only the last incremental written. This significantly reduces the disk i/o at the (theoretical) cost of inconsistency. This is a probabilistic way of determining whether a full backup is necessary. -z / --gzip Compress with gzip the backup files. Uses the default zlib compression level. By default, gzip compression is not used. Options for -R/--recover: -D str --date=str Recover state as of this date. Specify UTC (not local) time. yyyy-mm-dd[-hh[-mm[-ss]]] By default, current time is used. -o filename --output=filename Write recovered ZODB to given file. By default, the file is written to stdout. """ import os import sys import md5 import gzip import time import errno import getopt from ZODB.FileStorage import FileStorage program = sys.argv[0] BACKUP = 1 RECOVER = 2 COMMASPACE = ', ' READCHUNK = 16 * 1024 VERBOSE = False def usage(code, msg=''): outfp = sys.stderr if code == 0: outfp = sys.stdout print >> outfp, __doc__ % globals() if msg: print >> outfp, msg sys.exit(code) def log(msg, *args): if VERBOSE: # Use stderr here so that -v flag works with -R and no -o print >> sys.stderr, msg % args def parseargs(): global VERBOSE try: opts, args = getopt.getopt(sys.argv[1:], 'BRvhf:r:FD:o:Qz', ['backup', 'recover', 'verbose', 'help', 'file=', 'repository=', 'full', 'date=', 'output=', 'quick', 'gzip']) except getopt.error, msg: usage(1, msg) class Options: mode = None # BACKUP or RECOVER file = None # name of input Data.fs file repository = None # name of directory holding backups full = False # True forces full backup date = None # -D argument, if any output = None # where to write recovered data; None = stdout quick = False # -Q flag state gzip = False # -z flag state options = Options() for opt, arg in opts: if opt in ('-h', '--help'): usage(0) elif opt in ('-v', '--verbose'): VERBOSE = True elif opt in ('-R', '--recover'): if options.mode is not None: usage(1, '-B and -R are mutually exclusive') options.mode = RECOVER elif opt in ('-B', '--backup'): if options.mode is not None: usage(1, '-B and -R are mutually exclusive') options.mode = BACKUP elif opt in ('-Q', '--quick'): options.quick = True elif opt in ('-f', '--file'): options.file = arg elif opt in ('-r', '--repository'): options.repository = arg elif opt in ('-F', '--full'): options.full = True elif opt in ('-D', '--date'): options.date = arg elif opt in ('-o', '--output'): options.output = arg elif opt in ('-z', '--gzip'): options.gzip = True else: assert False, (opt, arg) # Any other arguments are invalid if args: usage(1, 'Invalid arguments: ' + COMMASPACE.join(args)) # Sanity checks if options.mode is None: usage(1, 'Either --backup or --recover is required') if options.repository is None: usage(1, '--repository is required') if options.mode == BACKUP: if options.date is not None: log('--date option is ignored in backup mode') options.date = None if options.output is not None: log('--output option is ignored in backup mode') options.output = None else: assert options.mode == RECOVER if options.file is not None: log('--file option is ignored in recover mode') options.file = None return options # afile is a Python file object, or created by gzip.open(). The latter # doesn't have a fileno() method, so to fsync it we need to reach into # its underlying file object. def fsync(afile): afile.flush() fileobject = getattr(afile, 'fileobj', afile) os.fsync(fileobject.fileno()) # Read bytes (no more than n, or to EOF if n is None) in chunks from the # current position in file fp. Pass each chunk as an argument to func(). # Return the total number of bytes read == the total number of bytes # passed in all to func(). Leaves the file position just after the # last byte read. def dofile(func, fp, n=None): bytesread = 0L while n is None or n > 0: if n is None: todo = READCHUNK else: todo = min(READCHUNK, n) data = fp.read(todo) if not data: break func(data) nread = len(data) bytesread += nread if n is not None: n -= nread return bytesread def checksum(fp, n): # Checksum the first n bytes of the specified file sum = md5.new() def func(data): sum.update(data) dofile(func, fp, n) return sum.hexdigest() def copyfile(options, dst, start, n): # Copy bytes from file src, to file dst, starting at offset start, for n # length of bytes. For robustness, we first write, flush and fsync # to a temp file, then rename the temp file at the end. sum = md5.new() ifp = open(options.file, 'rb') ifp.seek(start) tempname = os.path.join(os.path.dirname(dst), 'tmp.tmp') if options.gzip: ofp = gzip.open(tempname, 'wb') else: ofp = open(tempname, 'wb') def func(data): sum.update(data) ofp.write(data) ndone = dofile(func, ifp, n) assert ndone == n ifp.close() fsync(ofp) ofp.close() os.rename(tempname, dst) return sum.hexdigest() def concat(files, ofp=None): # Concatenate a bunch of files from the repository, output to `outfile' if # given. Return the number of bytes written and the md5 checksum of the # bytes. sum = md5.new() def func(data): sum.update(data) if ofp: ofp.write(data) bytesread = 0 for f in files: # Auto uncompress if f.endswith('fsz'): ifp = gzip.open(f, 'rb') else: ifp = open(f, 'rb') bytesread += dofile(func, ifp) ifp.close() if ofp: ofp.close() return bytesread, sum.hexdigest() def gen_filename(options, ext=None): if ext is None: if options.full: ext = '.fs' else: ext = '.deltafs' if options.gzip: ext += 'z' t = time.gmtime()[:6] + (ext,) return '%04d-%02d-%02d-%02d-%02d-%02d%s' % t # Return a list of files needed to reproduce state at time options.date. # This is a list, in chronological order, of the .fs[z] and .deltafs[z] # files, from the time of the most recent full backup preceding # options.date, up to options.date. import re is_data_file = re.compile(r'\d{4}(?:-\d\d){5}\.(?:delta)?fsz?$').match del re def find_files(options): when = options.date if not when: when = gen_filename(options, '') log('looking for files between last full backup and %s...', when) all = filter(is_data_file, os.listdir(options.repository)) all.sort() all.reverse() # newest file first # Find the last full backup before date, then include all the # incrementals between that full backup and "when". needed = [] for fname in all: root, ext = os.path.splitext(fname) if root <= when: needed.append(fname) if ext in ('.fs', '.fsz'): break # Make the file names relative to the repository directory needed = [os.path.join(options.repository, f) for f in needed] # Restore back to chronological order needed.reverse() if needed: log('files needed to recover state as of %s:', when) for f in needed: log('\t%s', f) else: log('no files found') return needed # Scan the .dat file corresponding to the last full backup performed. # Return # # filename, startpos, endpos, checksum # # of the last incremental. If there is no .dat file, or the .dat file # is empty, return # # None, None, None, None def scandat(repofiles): fullfile = repofiles[0] datfile = os.path.splitext(fullfile)[0] + '.dat' fn = startpos = endpos = sum = None # assume .dat file missing or empty try: fp = open(datfile) except IOError, e: if e.errno <> errno.ENOENT: raise else: # We only care about the last one. lines = fp.readlines() fp.close() if lines: fn, startpos, endpos, sum = lines[-1].split() startpos = long(startpos) endpos = long(endpos) return fn, startpos, endpos, sum def do_full_backup(options): # Find the file position of the last completed transaction. fs = FileStorage(options.file, read_only=True) # Note that the FileStorage ctor calls read_index() which scans the file # and returns "the position just after the last valid transaction record". # getSize() then returns this position, which is exactly what we want, # because we only want to copy stuff from the beginning of the file to the # last valid transaction record. pos = fs.getSize() fs.close() options.full = True dest = os.path.join(options.repository, gen_filename(options)) if os.path.exists(dest): print >> sys.stderr, 'Cannot overwrite existing file:', dest sys.exit(2) log('writing full backup: %s bytes to %s', pos, dest) sum = copyfile(options, dest, 0, pos) # Write the data file for this full backup datfile = os.path.splitext(dest)[0] + '.dat' fp = open(datfile, 'w') print >> fp, dest, 0, pos, sum fp.flush() os.fsync(fp.fileno()) fp.close() def do_incremental_backup(options, reposz, repofiles): # Find the file position of the last completed transaction. fs = FileStorage(options.file, read_only=True) # Note that the FileStorage ctor calls read_index() which scans the file # and returns "the position just after the last valid transaction record". # getSize() then returns this position, which is exactly what we want, # because we only want to copy stuff from the beginning of the file to the # last valid transaction record. pos = fs.getSize() fs.close() options.full = False dest = os.path.join(options.repository, gen_filename(options)) if os.path.exists(dest): print >> sys.stderr, 'Cannot overwrite existing file:', dest sys.exit(2) log('writing incremental: %s bytes to %s', pos-reposz, dest) sum = copyfile(options, dest, reposz, pos - reposz) # The first file in repofiles points to the last full backup. Use this to # get the .dat file and append the information for this incrementatl to # that file. fullfile = repofiles[0] datfile = os.path.splitext(fullfile)[0] + '.dat' # This .dat file better exist. Let the exception percolate if not. fp = open(datfile, 'a') print >> fp, dest, reposz, pos, sum fp.flush() os.fsync(fp.fileno()) fp.close() def do_backup(options): repofiles = find_files(options) # See if we need to do a full backup if options.full or not repofiles: log('doing a full backup') do_full_backup(options) return srcsz = os.path.getsize(options.file) if options.quick: fn, startpos, endpos, sum = scandat(repofiles) # If the .dat file was missing, or was empty, do a full backup if (fn, startpos, endpos, sum) == (None, None, None, None): log('missing or empty .dat file (full backup)') do_full_backup(options) return # Has the file shrunk, possibly because of a pack? if srcsz < endpos: log('file shrunk, possibly because of a pack (full backup)') do_full_backup(options) return # Now check the md5 sum of the source file, from the last # incremental's start and stop positions. srcfp = open(options.file, 'rb') srcfp.seek(startpos) srcsum = checksum(srcfp, endpos-startpos) srcfp.close() log('last incremental file: %s', fn) log('last incremental checksum: %s', sum) log('source checksum range: [%s..%s], sum: %s', startpos, endpos, srcsum) if sum == srcsum: if srcsz == endpos: log('No changes, nothing to do') return log('doing incremental, starting at: %s', endpos) do_incremental_backup(options, endpos, repofiles) return else: # This was is much slower, and more disk i/o intensive, but it's also # more accurate since it checks the actual existing files instead of # the information in the .dat file. # # See if we can do an incremental, based on the files that already # exist. This call of concat() will not write an output file. reposz, reposum = concat(repofiles) log('repository state: %s bytes, md5: %s', reposz, reposum) # Get the md5 checksum of the source file, up to two file positions: # the entire size of the file, and up to the file position of the last # incremental backup. srcfp = open(options.file, 'rb') srcsum = checksum(srcfp, srcsz) srcfp.seek(0) srcsum_backedup = checksum(srcfp, reposz) srcfp.close() log('current state : %s bytes, md5: %s', srcsz, srcsum) log('backed up state : %s bytes, md5: %s', reposz, srcsum_backedup) # Has nothing changed? if srcsz == reposz and srcsum == reposum: log('No changes, nothing to do') return # Has the file shrunk, probably because of a pack? if srcsz < reposz: log('file shrunk, possibly because of a pack (full backup)') do_full_backup(options) return # The source file is larger than the repository. If the md5 checksums # match, then we know we can do an incremental backup. If they don't, # then perhaps the file was packed at some point (or a # non-transactional undo was performed, but this is deprecated). Only # do a full backup if forced to. if reposum == srcsum_backedup: log('doing incremental, starting at: %s', reposz) do_incremental_backup(options, reposz, repofiles) return # The checksums don't match, meaning the front of the source file has # changed. We'll need to do a full backup in that case. log('file changed, possibly because of a pack (full backup)') do_full_backup(options) def do_recover(options): # Find the first full backup at or before the specified date repofiles = find_files(options) if not repofiles: if options.date: log('No files in repository before %s', options.date) else: log('No files in repository') return if options.output is None: log('Recovering file to stdout') outfp = sys.stdout else: log('Recovering file to %s', options.output) outfp = open(options.output, 'wb') reposz, reposum = concat(repofiles, outfp) if outfp <> sys.stdout: outfp.close() log('Recovered %s bytes, md5: %s', reposz, reposum) def main(): options = parseargs() if options.mode == BACKUP: do_backup(options) else: assert options.mode == RECOVER do_recover(options) if __name__ == '__main__': main() Zope-2.8.7-final/utilities/ZODBTools/fstail.py0000664000076400007640000000276510435500417020177 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Tool to dump the last few transactions from a FileStorage.""" from ZODB.fstools import prev_txn import binascii import getopt import sha import sys def main(path, ntxn): f = open(path, "rb") f.seek(0, 2) th = prev_txn(f) i = ntxn while th and i > 0: hash = sha.sha(th.get_raw_data()).digest() l = len(str(th.get_timestamp())) + 1 th.read_meta() print "%s: hash=%s" % (th.get_timestamp(), binascii.hexlify(hash)) print ("user=%r description=%r length=%d" % (th.user, th.descr, th.length)) print th = th.prev_txn() i -= 1 if __name__ == "__main__": ntxn = 10 opts, args = getopt.getopt(sys.argv[1:], "n:") path, = args for k, v in opts: if k == '-n': ntxn = int(v) main(path, ntxn) Zope-2.8.7-final/utilities/ZODBTools/zeoserverlog.py0000664000076400007640000003473010435500417021440 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Tools for analyzing ZEO Server logs. This script contains a number of commands, implemented by command functions. To run a command, give the command name and it's arguments as arguments to this script. Commands: blocked_times file threshold Output a summary of episodes where thransactions were blocked when the episode lasted at least threshold seconds. The file may be a file name or - to read from standard input. The file may also be a command: script blocked_times 'bunzip2 = 0 if blocking and waiting == 1: t1 = time(line) t2 = t1 if not blocking and last_blocking: last_wait = 0 t2 = time(line) cid = idre.search(line).group(1) if waiting == 0: d = sub(t1, time(line)) if d >= thresh: print t1, sub(t1, t2), cid, d t1 = t2 = cid = blocking = waiting = last_wait = max_wait = 0 last_blocking = blocking connidre = re.compile(r' zrpc-conn:(\d+.\d+.\d+.\d+:\d+) ') def time_calls(f): f, thresh = f if f == '-': f = sys.stdin else: f = xopen(f) thresh = float(thresh) t1 = None maxd = 0 for line in f: line = line.strip() if ' calling ' in line: t1 = time(line) elif ' returns ' in line and t1 is not None: d = sub(t1, time(line)) if d >= thresh: print t1, d, connidre.search(line).group(1) maxd = max(maxd, d) t1 = None print maxd def xopen(f): if f == '-': return sys.stdin if ' ' in f: return os.popen(f, 'r') return open(f) def time_tpc(f): f, thresh = f if f == '-': f = sys.stdin else: f = xopen(f) thresh = float(thresh) transactions = {} for line in f: line = line.strip() if ' calling vote(' in line: cid = connidre.search(line).group(1) transactions[cid] = time(line), elif ' vote returns None' in line: cid = connidre.search(line).group(1) transactions[cid] += time(line), 'n' elif ' vote() raised' in line: cid = connidre.search(line).group(1) transactions[cid] += time(line), 'e' elif ' vote returns ' in line: # delayed, skip cid = connidre.search(line).group(1) transactions[cid] += time(line), 'd' elif ' calling tpc_abort(' in line: cid = connidre.search(line).group(1) if cid in transactions: t1, t2, vs = transactions[cid] t = time(line) d = sub(t1, t) if d >= thresh: print 'a', t1, cid, sub(t1, t2), vs, sub(t2, t) del transactions[cid] elif ' calling tpc_finish(' in line: if cid in transactions: cid = connidre.search(line).group(1) transactions[cid] += time(line), elif ' tpc_finish returns ' in line: if cid in transactions: t1, t2, vs, t3 = transactions[cid] t = time(line) d = sub(t1, t) if d >= thresh: print 'c', t1, cid, sub(t1, t2), vs, sub(t2, t3), sub(t3, t) del transactions[cid] newobre = re.compile(r"storea\(.*, '\\x00\\x00\\x00\\x00\\x00") def time_trans(f): f, thresh = f if f == '-': f = sys.stdin else: f = xopen(f) thresh = float(thresh) transactions = {} for line in f: line = line.strip() if ' calling tpc_begin(' in line: cid = connidre.search(line).group(1) transactions[cid] = time(line), [0, 0] if ' calling storea(' in line: cid = connidre.search(line).group(1) if cid in transactions: transactions[cid][1][0] += 1 if not newobre.search(line): transactions[cid][1][1] += 1 elif ' calling vote(' in line: cid = connidre.search(line).group(1) if cid in transactions: transactions[cid] += time(line), elif ' vote returns None' in line: cid = connidre.search(line).group(1) if cid in transactions: transactions[cid] += time(line), 'n' elif ' vote() raised' in line: cid = connidre.search(line).group(1) if cid in transactions: transactions[cid] += time(line), 'e' elif ' vote returns ' in line: # delayed, skip cid = connidre.search(line).group(1) if cid in transactions: transactions[cid] += time(line), 'd' elif ' calling tpc_abort(' in line: cid = connidre.search(line).group(1) if cid in transactions: try: t0, (stores, old), t1, t2, vs = transactions[cid] except ValueError: pass else: t = time(line) d = sub(t1, t) if d >= thresh: print t1, cid, "%s/%s" % (stores, old), \ sub(t0, t1), sub(t1, t2), vs, \ sub(t2, t), 'abort' del transactions[cid] elif ' calling tpc_finish(' in line: if cid in transactions: cid = connidre.search(line).group(1) transactions[cid] += time(line), elif ' tpc_finish returns ' in line: if cid in transactions: t0, (stores, old), t1, t2, vs, t3 = transactions[cid] t = time(line) d = sub(t1, t) if d >= thresh: print t1, cid, "%s/%s" % (stores, old), \ sub(t0, t1), sub(t1, t2), vs, \ sub(t2, t3), sub(t3, t) del transactions[cid] def minute(f, slice=16, detail=1, summary=1): f, = f if f == '-': f = sys.stdin else: f = xopen(f) cols = ["time", "reads", "stores", "commits", "aborts", "txns"] fmt = "%18s %6s %6s %7s %6s %6s" print fmt % cols print fmt % ["-"*len(col) for col in cols] mlast = r = s = c = a = cl = None rs = [] ss = [] cs = [] as = [] ts = [] cls = [] for line in f: line = line.strip() if (line.find('returns') > 0 or line.find('storea') > 0 or line.find('tpc_abort') > 0 ): client = connidre.search(line).group(1) m = line[:slice] if m != mlast: if mlast: if detail: print fmt % (mlast, len(cl), r, s, c, a, a+c) cls.append(len(cl)) rs.append(r) ss.append(s) cs.append(c) as.append(a) ts.append(c+a) mlast = m r = s = c = a = 0 cl = {} if line.find('zeoLoad') > 0: r += 1 cl[client] = 1 elif line.find('storea') > 0: s += 1 cl[client] = 1 elif line.find('tpc_finish') > 0: c += 1 cl[client] = 1 elif line.find('tpc_abort') > 0: a += 1 cl[client] = 1 if mlast: if detail: print fmt % (mlast, len(cl), r, s, c, a, a+c) cls.append(len(cl)) rs.append(r) ss.append(s) cs.append(c) as.append(a) ts.append(c+a) if summary: print print 'Summary: \t', '\t'.join(('min', '10%', '25%', 'med', '75%', '90%', 'max', 'mean')) print "n=%6d\t" % len(cls), '-'*62 print 'Clients: \t', '\t'.join(map(str,stats(cls))) print 'Reads: \t', '\t'.join(map(str,stats( rs))) print 'Stores: \t', '\t'.join(map(str,stats( ss))) print 'Commits: \t', '\t'.join(map(str,stats( cs))) print 'Aborts: \t', '\t'.join(map(str,stats( as))) print 'Trans: \t', '\t'.join(map(str,stats( ts))) def stats(s): s.sort() min = s[0] max = s[-1] n = len(s) out = [min] ni = n + 1 for p in .1, .25, .5, .75, .90: lp = ni*p l = int(lp) if lp < 1 or lp > n: out.append('-') elif abs(lp-l) < .00001: out.append(s[l-1]) else: out.append(int(s[l-1] + (lp - l) * (s[l] - s[l-1]))) mean = 0.0 for v in s: mean += v out.extend([max, int(mean/n)]) return out def minutes(f): minute(f, 16, detail=0) def hour(f): minute(f, 13) def day(f): minute(f, 10) def hours(f): minute(f, 13, detail=0) def days(f): minute(f, 10, detail=0) new_connection_idre = re.compile(r"new connection \('(\d+.\d+.\d+.\d+)', (\d+)\):") def verify(f): f, = f if f == '-': f = sys.stdin else: f = xopen(f) t1 = None nv = {} for line in f: if line.find('new connection') > 0: m = new_connection_idre.search(line) cid = "%s:%s" % (m.group(1), m.group(2)) nv[cid] = [time(line), 0] elif line.find('calling zeoVerify(') > 0: cid = connidre.search(line).group(1) nv[cid][1] += 1 elif line.find('calling endZeoVerify()') > 0: cid = connidre.search(line).group(1) t1, n = nv[cid] if n: d = sub(t1, time(line)) print cid, t1, n, d, n and (d*1000.0/n) or '-' def recovery(f): f, = f if f == '-': f = sys.stdin else: f = xopen(f) last = '' trans = [] n = 0 for line in f: n += 1 if line.find('RecoveryServer') < 0: continue l = line.find('sending transaction ') if l > 0 and last.find('sending transaction ') > 0: trans.append(line[l+20:].strip()) else: if trans: if len(trans) > 1: print " ... %s similar records skipped ..." % ( len(trans) - 1) print n, last.strip() trans=[] print n, line.strip() last = line if len(trans) > 1: print " ... %s similar records skipped ..." % ( len(trans) - 1) print n, last.strip() if __name__ == '__main__': globals()[sys.argv[1]](sys.argv[2:]) Zope-2.8.7-final/utilities/ZODBTools/zodbload.py0000664000076400007640000007441710435500417020516 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2003 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE. # ############################################################################## """Test script for testing ZODB under a heavy zope-like load. Note that, to be as realistic as possible with ZEO, you should run this script multiple times, to simulate multiple clients. Here's how this works. The script starts some number of threads. Each thread, sequentially executes jobs. There is a job producer that produces jobs. Input data are provided by a mail producer that hands out message from a mailbox. Execution continues until there is an error, which will normally occur when the mailbox is exhausted. Command-line options are used to provide job definitions. Job definitions have perameters of the form name=value. Jobs have 2 standard parameters: frequency=integer The frequency of the job. The default is 1. sleep=float The number os seconds to sleep before performing the job. The default is 0. Usage: loadmail2 [options] Options: -edit [frequency=integer] [sleep=float] Define an edit job. An edit job edits a random already-saved email message, deleting and inserting a random number of words. After editing the message, the message is (re)cataloged. -insert [number=int] [frequency=integer] [sleep=float] Insert some number of email messages. -index [number=int] [frequency=integer] [sleep=float] Insert and index (catalog) some number of email messages. -search [terms='word1 word2 ...'] [frequency=integer] [sleep=float] Search the catalog. A query is givem with one or more terms as would be entered into a typical seach box. If no query is given, then queries will be randomly selected based on a set of built-in word list. -setup Set up the database. This will delete any existing Data.fs file. (Of course, this may have no effect, if there is a custom_zodb that defined a different storage.) It also adds a mail folder and a catalog. -options file Read options from the given file. Th efile should be a python source file that defines a sequence of options named 'options'. -threads n Specify the number of threads to execute. If not specified (< 2), then jobs are run in a single (main) thread. -mbox filename Specify the mailbox for getting input data. There is a (lame) syntax for providing options within the filename. The filename may be followed by up to 3 integers, min, max, and start: -mbox 'foo.mbox 0 100 10000' The messages from min to max will be read from the mailbox. They will be assigned message numbers starting with start. So, in the example above, we read the first hundred messages and assign thgem message numbers starting with 10001. The maxmum can be given as a negative number, in which case, it specifies the number of messages to read. The start defaults to the minimum. The following two options: -mbox 'foo.mbox 300 400 300' and -mbox 'foo.mbox 300 -100' are equivalent $Id: zodbload.py 29533 2005-03-17 19:11:11Z tim_one $ """ import mailbox import math import os import random import re import sys import threading import time import transaction class JobProducer: def __init__(self): self.jobs = [] def add(self, callable, frequency, sleep, repeatp=0): self.jobs.extend([(callable, sleep, repeatp)] * int(frequency)) random.shuffle(self.jobs) def next(self): factory, sleep, repeatp = random.choice(self.jobs) time.sleep(sleep) callable, args = factory.create() return factory, callable, args, repeatp def __nonzero__(self): return not not self.jobs class MBox: def __init__(self, filename): if ' ' in filename: filename = filename.split() if len(filename) < 4: filename += [0, 0, -1][-(4-len(filename)):] filename, min, max, start = filename min = int(min) max = int(max) start = int(start) if start < 0: start = min if max < 0: # negative max is treated as a count self._max = start - max elif max > 0: self._max = start + max - min else: self._max = 0 else: self._max = 0 min = start = 0 if filename.endswith('.bz2'): f = os.popen("bunzip2 <"+filename, 'r') filename = filename[-4:] else: f = open(filename) self._mbox = mb = mailbox.UnixMailbox(f) self.number = start while min: mb.next() min -= 1 self._lock = threading.Lock() self.__name__ = os.path.splitext(os.path.split(filename)[1])[0] self._max = max def next(self): self._lock.acquire() try: if self._max > 0 and self.number >= self._max: raise IndexError(self.number + 1) message = self._mbox.next() message.body = message.fp.read() message.headers = list(message.headers) self.number += 1 message.number = self.number message.mbox = self.__name__ return message finally: self._lock.release() bins = 9973 #bins = 11 def mailfolder(app, mboxname, number): mail = getattr(app, mboxname, None) if mail is None: app.manage_addFolder(mboxname) mail = getattr(app, mboxname) from BTrees.Length import Length mail.length = Length() for i in range(bins): mail.manage_addFolder('b'+str(i)) bin = hash(str(number))%bins return getattr(mail, 'b'+str(bin)) def VmSize(): try: f = open('/proc/%s/status' % os.getpid()) except: return 0 else: l = filter(lambda l: l[:7] == 'VmSize:', f.readlines()) if l: l = l[0][7:].strip().split()[0] return int(l) return 0 def setup(lib_python): try: os.remove(os.path.join(lib_python, '..', '..', 'var', 'Data.fs')) except: pass import Zope2 import Products import AccessControl.SecurityManagement app=Zope2.app() Products.ZCatalog.ZCatalog.manage_addZCatalog(app, 'cat', '') from Products.ZCTextIndex.ZCTextIndex import PLexicon from Products.ZCTextIndex.Lexicon import Splitter, CaseNormalizer app.cat._setObject('lex', PLexicon('lex', '', Splitter(), CaseNormalizer()) ) class extra: doc_attr = 'PrincipiaSearchSource' lexicon_id = 'lex' index_type = 'Okapi BM25 Rank' app.cat.addIndex('PrincipiaSearchSource', 'ZCTextIndex', extra) transaction.commit() system = AccessControl.SpecialUsers.system AccessControl.SecurityManagement.newSecurityManager(None, system) app._p_jar.close() def do(db, f, args): """Do something in a transaction, retrying of necessary Measure the speed of both the compurartion and the commit """ from ZODB.POSException import ConflictError wcomp = ccomp = wcommit = ccommit = 0.0 rconflicts = wconflicts = 0 start = time.time() while 1: connection = db.open() try: transaction.begin() t=time.time() c=time.clock() try: try: r = f(connection, *args) except ConflictError: rconflicts += 1 transaction.abort() continue finally: wcomp += time.time() - t ccomp += time.clock() - c t=time.time() c=time.clock() try: try: transaction.commit() break except ConflictError: wconflicts += 1 transaction.abort() continue finally: wcommit += time.time() - t ccommit += time.clock() - c finally: connection.close() return start, wcomp, ccomp, rconflicts, wconflicts, wcommit, ccommit, r def run1(tid, db, factory, job, args): (start, wcomp, ccomp, rconflicts, wconflicts, wcommit, ccommit, r ) = do(db, job, args) start = "%.4d-%.2d-%.2d %.2d:%.2d:%.2d" % time.localtime(start)[:6] print "%s %s %8.3g %8.3g %s %s\t%8.3g %8.3g %s %r" % ( start, tid, wcomp, ccomp, rconflicts, wconflicts, wcommit, ccommit, factory.__name__, r) def run(jobs, tid=''): import Zope2 while 1: factory, job, args, repeatp = jobs.next() run1(tid, Zope2.DB, factory, job, args) if repeatp: while 1: i = random.randint(0,100) if i > repeatp: break run1(tid, Zope2.DB, factory, job, args) def index(connection, messages, catalog, max): app = connection.root()['Application'] for message in messages: mail = mailfolder(app, message.mbox, message.number) if max: # Cheat and use folder implementation secrets # to avoid having to read the old data _objects = mail._objects if len(_objects) >= max: for d in _objects[:len(_objects)-max+1]: del mail.__dict__[d['id']] mail._objects = _objects[len(_objects)-max+1:] docid = 'm'+str(message.number) mail.manage_addDTMLDocument(docid, file=message.body) # increment counted getattr(app, message.mbox).length.change(1) doc = mail[docid] for h in message.headers: h = h.strip() l = h.find(':') if l <= 0: continue name = h[:l].lower() if name=='subject': name='title' v = h[l+1:].strip() type='string' if name=='title': doc.manage_changeProperties(title=h) else: try: doc.manage_addProperty(name, v, type) except: pass if catalog: app.cat.catalog_object(doc) return message.number class IndexJob: needs_mbox = 1 catalog = 1 prefix = 'index' def __init__(self, mbox, number=1, max=0): self.__name__ = "%s%s_%s" % (self.prefix, number, mbox.__name__) self.mbox, self.number, self.max = mbox, int(number), int(max) def create(self): messages = [self.mbox.next() for i in range(self.number)] return index, (messages, self.catalog, self.max) class InsertJob(IndexJob): catalog = 0 prefix = 'insert' wordre = re.compile(r'(\w{3,20})') stop = 'and', 'not' def edit(connection, mbox, catalog=1): app = connection.root()['Application'] mail = getattr(app, mbox.__name__, None) if mail is None: time.sleep(1) return "No mailbox %s" % mbox.__name__ nmessages = mail.length() if nmessages < 2: time.sleep(1) return "No messages to edit in %s" % mbox.__name__ # find a message to edit: while 1: number = random.randint(1, nmessages-1) did = 'm' + str(number) mail = mailfolder(app, mbox.__name__, number) doc = getattr(mail, did, None) if doc is not None: break text = doc.raw.split() norig = len(text) if norig > 10: ndel = int(math.exp(random.randint(0, int(math.log(norig))))) nins = int(math.exp(random.randint(0, int(math.log(norig))))) else: ndel = 0 nins = 10 for j in range(ndel): j = random.randint(0,len(text)-1) word = text[j] m = wordre.search(word) if m: word = m.group(1).lower() if (not wordsd.has_key(word)) and word not in stop: words.append(word) wordsd[word] = 1 del text[j] for j in range(nins): word = random.choice(words) text.append(word) doc.raw = ' '.join(text) if catalog: app.cat.catalog_object(doc) return norig, ndel, nins class EditJob: needs_mbox = 1 prefix = 'edit' catalog = 1 def __init__(self, mbox): self.__name__ = "%s_%s" % (self.prefix, mbox.__name__) self.mbox = mbox def create(self): return edit, (self.mbox, self.catalog) class ModifyJob(EditJob): prefix = 'modify' catalog = 0 def search(connection, terms, number): app = connection.root()['Application'] cat = app.cat n = 0 for i in number: term = random.choice(terms) results = cat(PrincipiaSearchSource=term) n += len(results) for result in results: obj = result.getObject() # Apparently, there is a bug in Zope that leads obj to be None # on occasion. if obj is not None: obj.getId() return n class SearchJob: def __init__(self, terms='', number=10): if terms: terms = terms.split() self.__name__ = "search_" + '_'.join(terms) self.terms = terms else: self.__name__ = 'search' self.terms = words number = min(int(number), len(self.terms)) self.number = range(number) def create(self): return search, (self.terms, self.number) words=['banishment', 'indirectly', 'imprecise', 'peeks', 'opportunely', 'bribe', 'sufficiently', 'Occidentalized', 'elapsing', 'fermenting', 'listen', 'orphanage', 'younger', 'draperies', 'Ida', 'cuttlefish', 'mastermind', 'Michaels', 'populations', 'lent', 'cater', 'attentional', 'hastiness', 'dragnet', 'mangling', 'scabbards', 'princely', 'star', 'repeat', 'deviation', 'agers', 'fix', 'digital', 'ambitious', 'transit', 'jeeps', 'lighted', 'Prussianizations', 'Kickapoo', 'virtual', 'Andrew', 'generally', 'boatsman', 'amounts', 'promulgation', 'Malay', 'savaging', 'courtesan', 'nursed', 'hungered', 'shiningly', 'ship', 'presides', 'Parke', 'moderns', 'Jonas', 'unenlightening', 'dearth', 'deer', 'domesticates', 'recognize', 'gong', 'penetrating', 'dependents', 'unusually', 'complications', 'Dennis', 'imbalances', 'nightgown', 'attached', 'testaments', 'congresswoman', 'circuits', 'bumpers', 'braver', 'Boreas', 'hauled', 'Howe', 'seethed', 'cult', 'numismatic', 'vitality', 'differences', 'collapsed', 'Sandburg', 'inches', 'head', 'rhythmic', 'opponent', 'blanketer', 'attorneys', 'hen', 'spies', 'indispensably', 'clinical', 'redirection', 'submit', 'catalysts', 'councilwoman', 'kills', 'topologies', 'noxious', 'exactions', 'dashers', 'balanced', 'slider', 'cancerous', 'bathtubs', 'legged', 'respectably', 'crochets', 'absenteeism', 'arcsine', 'facility', 'cleaners', 'bobwhite', 'Hawkins', 'stockade', 'provisional', 'tenants', 'forearms', 'Knowlton', 'commit', 'scornful', 'pediatrician', 'greets', 'clenches', 'trowels', 'accepts', 'Carboloy', 'Glenn', 'Leigh', 'enroll', 'Madison', 'Macon', 'oiling', 'entertainingly', 'super', 'propositional', 'pliers', 'beneficiary', 'hospitable', 'emigration', 'sift', 'sensor', 'reserved', 'colonization', 'shrilled', 'momentously', 'stevedore', 'Shanghaiing', 'schoolmasters', 'shaken', 'biology', 'inclination', 'immoderate', 'stem', 'allegory', 'economical', 'daytime', 'Newell', 'Moscow', 'archeology', 'ported', 'scandals', 'Blackfoot', 'leery', 'kilobit', 'empire', 'obliviousness', 'productions', 'sacrificed', 'ideals', 'enrolling', 'certainties', 'Capsicum', 'Brookdale', 'Markism', 'unkind', 'dyers', 'legislates', 'grotesquely', 'megawords', 'arbitrary', 'laughing', 'wildcats', 'thrower', 'sex', 'devils', 'Wehr', 'ablates', 'consume', 'gossips', 'doorways', 'Shari', 'advanced', 'enumerable', 'existentially', 'stunt', 'auctioneers', 'scheduler', 'blanching', 'petulance', 'perceptibly', 'vapors', 'progressed', 'rains', 'intercom', 'emergency', 'increased', 'fluctuating', 'Krishna', 'silken', 'reformed', 'transformation', 'easter', 'fares', 'comprehensible', 'trespasses', 'hallmark', 'tormenter', 'breastworks', 'brassiere', 'bladders', 'civet', 'death', 'transformer', 'tolerably', 'bugle', 'clergy', 'mantels', 'satin', 'Boswellizes', 'Bloomington', 'notifier', 'Filippo', 'circling', 'unassigned', 'dumbness', 'sentries', 'representativeness', 'souped', 'Klux', 'Kingstown', 'gerund', 'Russell', 'splices', 'bellow', 'bandies', 'beefers', 'cameramen', 'appalled', 'Ionian', 'butterball', 'Portland', 'pleaded', 'admiringly', 'pricks', 'hearty', 'corer', 'deliverable', 'accountably', 'mentors', 'accorded', 'acknowledgement', 'Lawrenceville', 'morphology', 'eucalyptus', 'Rena', 'enchanting', 'tighter', 'scholars', 'graduations', 'edges', 'Latinization', 'proficiency', 'monolithic', 'parenthesizing', 'defy', 'shames', 'enjoyment', 'Purdue', 'disagrees', 'barefoot', 'maims', 'flabbergast', 'dishonorable', 'interpolation', 'fanatics', 'dickens', 'abysses', 'adverse', 'components', 'bowl', 'belong', 'Pipestone', 'trainees', 'paw', 'pigtail', 'feed', 'whore', 'conditioner', 'Volstead', 'voices', 'strain', 'inhabits', 'Edwin', 'discourses', 'deigns', 'cruiser', 'biconvex', 'biking', 'depreciation', 'Harrison', 'Persian', 'stunning', 'agar', 'rope', 'wagoner', 'elections', 'reticulately', 'Cruz', 'pulpits', 'wilt', 'peels', 'plants', 'administerings', 'deepen', 'rubs', 'hence', 'dissension', 'implored', 'bereavement', 'abyss', 'Pennsylvania', 'benevolent', 'corresponding', 'Poseidon', 'inactive', 'butchers', 'Mach', 'woke', 'loading', 'utilizing', 'Hoosier', 'undo', 'Semitization', 'trigger', 'Mouthe', 'mark', 'disgracefully', 'copier', 'futility', 'gondola', 'algebraic', 'lecturers', 'sponged', 'instigators', 'looted', 'ether', 'trust', 'feeblest', 'sequencer', 'disjointness', 'congresses', 'Vicksburg', 'incompatibilities', 'commend', 'Luxembourg', 'reticulation', 'instructively', 'reconstructs', 'bricks', 'attache', 'Englishman', 'provocation', 'roughen', 'cynic', 'plugged', 'scrawls', 'antipode', 'injected', 'Daedalus', 'Burnsides', 'asker', 'confronter', 'merriment', 'disdain', 'thicket', 'stinker', 'great', 'tiers', 'oust', 'antipodes', 'Macintosh', 'tented', 'packages', 'Mediterraneanize', 'hurts', 'orthodontist', 'seeder', 'readying', 'babying', 'Florida', 'Sri', 'buckets', 'complementary', 'cartographer', 'chateaus', 'shaves', 'thinkable', 'Tehran', 'Gordian', 'Angles', 'arguable', 'bureau', 'smallest', 'fans', 'navigated', 'dipole', 'bootleg', 'distinctive', 'minimization', 'absorbed', 'surmised', 'Malawi', 'absorbent', 'close', 'conciseness', 'hopefully', 'declares', 'descent', 'trick', 'portend', 'unable', 'mildly', 'Morse', 'reference', 'scours', 'Caribbean', 'battlers', 'astringency', 'likelier', 'Byronizes', 'econometric', 'grad', 'steak', 'Austrian', 'ban', 'voting', 'Darlington', 'bison', 'Cetus', 'proclaim', 'Gilbertson', 'evictions', 'submittal', 'bearings', 'Gothicizer', 'settings', 'McMahon', 'densities', 'determinants', 'period', 'DeKastere', 'swindle', 'promptness', 'enablers', 'wordy', 'during', 'tables', 'responder', 'baffle', 'phosgene', 'muttering', 'limiters', 'custodian', 'prevented', 'Stouffer', 'waltz', 'Videotex', 'brainstorms', 'alcoholism', 'jab', 'shouldering', 'screening', 'explicitly', 'earner', 'commandment', 'French', 'scrutinizing', 'Gemma', 'capacitive', 'sheriff', 'herbivore', 'Betsey', 'Formosa', 'scorcher', 'font', 'damming', 'soldiers', 'flack', 'Marks', 'unlinking', 'serenely', 'rotating', 'converge', 'celebrities', 'unassailable', 'bawling', 'wording', 'silencing', 'scotch', 'coincided', 'masochists', 'graphs', 'pernicious', 'disease', 'depreciates', 'later', 'torus', 'interject', 'mutated', 'causer', 'messy', 'Bechtel', 'redundantly', 'profoundest', 'autopsy', 'philosophic', 'iterate', 'Poisson', 'horridly', 'silversmith', 'millennium', 'plunder', 'salmon', 'missioner', 'advances', 'provers', 'earthliness', 'manor', 'resurrectors', 'Dahl', 'canto', 'gangrene', 'gabler', 'ashore', 'frictionless', 'expansionism', 'emphasis', 'preservations', 'Duane', 'descend', 'isolated', 'firmware', 'dynamites', 'scrawled', 'cavemen', 'ponder', 'prosperity', 'squaw', 'vulnerable', 'opthalmic', 'Simms', 'unite', 'totallers', 'Waring', 'enforced', 'bridge', 'collecting', 'sublime', 'Moore', 'gobble', 'criticizes', 'daydreams', 'sedate', 'apples', 'Concordia', 'subsequence', 'distill', 'Allan', 'seizure', 'Isadore', 'Lancashire', 'spacings', 'corresponded', 'hobble', 'Boonton', 'genuineness', 'artifact', 'gratuities', 'interviewee', 'Vladimir', 'mailable', 'Bini', 'Kowalewski', 'interprets', 'bereave', 'evacuated', 'friend', 'tourists', 'crunched', 'soothsayer', 'fleetly', 'Romanizations', 'Medicaid', 'persevering', 'flimsy', 'doomsday', 'trillion', 'carcasses', 'guess', 'seersucker', 'ripping', 'affliction', 'wildest', 'spokes', 'sheaths', 'procreate', 'rusticates', 'Schapiro', 'thereafter', 'mistakenly', 'shelf', 'ruination', 'bushel', 'assuredly', 'corrupting', 'federation', 'portmanteau', 'wading', 'incendiary', 'thing', 'wanderers', 'messages', 'Paso', 'reexamined', 'freeings', 'denture', 'potting', 'disturber', 'laborer', 'comrade', 'intercommunicating', 'Pelham', 'reproach', 'Fenton', 'Alva', 'oasis', 'attending', 'cockpit', 'scout', 'Jude', 'gagging', 'jailed', 'crustaceans', 'dirt', 'exquisitely', 'Internet', 'blocker', 'smock', 'Troutman', 'neighboring', 'surprise', 'midscale', 'impart', 'badgering', 'fountain', 'Essen', 'societies', 'redresses', 'afterwards', 'puckering', 'silks', 'Blakey', 'sequel', 'greet', 'basements', 'Aubrey', 'helmsman', 'album', 'wheelers', 'easternmost', 'flock', 'ambassadors', 'astatine', 'supplant', 'gird', 'clockwork', 'foxes', 'rerouting', 'divisional', 'bends', 'spacer', 'physiologically', 'exquisite', 'concerts', 'unbridled', 'crossing', 'rock', 'leatherneck', 'Fortescue', 'reloading', 'Laramie', 'Tim', 'forlorn', 'revert', 'scarcer', 'spigot', 'equality', 'paranormal', 'aggrieves', 'pegs', 'committeewomen', 'documented', 'interrupt', 'emerald', 'Battelle', 'reconverted', 'anticipated', 'prejudices', 'drowsiness', 'trivialities', 'food', 'blackberries', 'Cyclades', 'tourist', 'branching', 'nugget', 'Asilomar', 'repairmen', 'Cowan', 'receptacles', 'nobler', 'Nebraskan', 'territorial', 'chickadee', 'bedbug', 'darted', 'vigilance', 'Octavia', 'summands', 'policemen', 'twirls', 'style', 'outlawing', 'specifiable', 'pang', 'Orpheus', 'epigram', 'Babel', 'butyrate', 'wishing', 'fiendish', 'accentuate', 'much', 'pulsed', 'adorned', 'arbiters', 'counted', 'Afrikaner', 'parameterizes', 'agenda', 'Americanism', 'referenda', 'derived', 'liquidity', 'trembling', 'lordly', 'Agway', 'Dillon', 'propellers', 'statement', 'stickiest', 'thankfully', 'autograph', 'parallel', 'impulse', 'Hamey', 'stylistic', 'disproved', 'inquirer', 'hoisting', 'residues', 'variant', 'colonials', 'dequeued', 'especial', 'Samoa', 'Polaris', 'dismisses', 'surpasses', 'prognosis', 'urinates', 'leaguers', 'ostriches', 'calculative', 'digested', 'divided', 'reconfigurer', 'Lakewood', 'illegalities', 'redundancy', 'approachability', 'masterly', 'cookery', 'crystallized', 'Dunham', 'exclaims', 'mainline', 'Australianizes', 'nationhood', 'pusher', 'ushers', 'paranoia', 'workstations', 'radiance', 'impedes', 'Minotaur', 'cataloging', 'bites', 'fashioning', 'Alsop', 'servants', 'Onondaga', 'paragraph', 'leadings', 'clients', 'Latrobe', 'Cornwallis', 'excitingly', 'calorimetric', 'savior', 'tandem', 'antibiotics', 'excuse', 'brushy', 'selfish', 'naive', 'becomes', 'towers', 'popularizes', 'engender', 'introducing', 'possession', 'slaughtered', 'marginally', 'Packards', 'parabola', 'utopia', 'automata', 'deterrent', 'chocolates', 'objectives', 'clannish', 'aspirin', 'ferociousness', 'primarily', 'armpit', 'handfuls', 'dangle', 'Manila', 'enlivened', 'decrease', 'phylum', 'hardy', 'objectively', 'baskets', 'chaired', 'Sepoy', 'deputy', 'blizzard', 'shootings', 'breathtaking', 'sticking', 'initials', 'epitomized', 'Forrest', 'cellular', 'amatory', 'radioed', 'horrified', 'Neva', 'simultaneous', 'delimiter', 'expulsion', 'Himmler', 'contradiction', 'Remus', 'Franklinizations', 'luggage', 'moisture', 'Jews', 'comptroller', 'brevity', 'contradictions', 'Ohio', 'active', 'babysit', 'China', 'youngest', 'superstition', 'clawing', 'raccoons', 'chose', 'shoreline', 'helmets', 'Jeffersonian', 'papered', 'kindergarten', 'reply', 'succinct', 'split', 'wriggle', 'suitcases', 'nonce', 'grinders', 'anthem', 'showcase', 'maimed', 'blue', 'obeys', 'unreported', 'perusing', 'recalculate', 'rancher', 'demonic', 'Lilliputianize', 'approximation', 'repents', 'yellowness', 'irritates', 'Ferber', 'flashlights', 'booty', 'Neanderthal', 'someday', 'foregoes', 'lingering', 'cloudiness', 'guy', 'consumer', 'Berkowitz', 'relics', 'interpolating', 'reappearing', 'advisements', 'Nolan', 'turrets', 'skeletal', 'skills', 'mammas', 'Winsett', 'wheelings', 'stiffen', 'monkeys', 'plainness', 'braziers', 'Leary', 'advisee', 'jack', 'verb', 'reinterpret', 'geometrical', 'trolleys', 'arboreal', 'overpowered', 'Cuzco', 'poetical', 'admirations', 'Hobbes', 'phonemes', 'Newsweek', 'agitator', 'finally', 'prophets', 'environment', 'easterners', 'precomputed', 'faults', 'rankly', 'swallowing', 'crawl', 'trolley', 'spreading', 'resourceful', 'go', 'demandingly', 'broader', 'spiders', 'Marsha', 'debris', 'operates', 'Dundee', 'alleles', 'crunchier', 'quizzical', 'hanging', 'Fisk'] wordsd = {} for word in words: wordsd[word] = 1 def collect_options(args, jobs, options): while args: arg = args.pop(0) if arg.startswith('-'): name = arg[1:] if name == 'options': fname = args.pop(0) d = {} execfile(fname, d) collect_options(list(d['options']), jobs, options) elif options.has_key(name): v = args.pop(0) if options[name] != None: raise ValueError( "Duplicate values for %s, %s and %s" % (name, v, options[name]) ) options[name] = v elif name == 'setup': options['setup'] = 1 elif globals().has_key(name.capitalize()+'Job'): job = name kw = {} while args and args[0].find("=") > 0: arg = args.pop(0).split('=') name, v = arg[0], '='.join(arg[1:]) if kw.has_key(name): raise ValueError( "Duplicate parameter %s for job %s" % (name, job) ) kw[name]=v if kw.has_key('frequency'): frequency = kw['frequency'] del kw['frequency'] else: frequency = 1 if kw.has_key('sleep'): sleep = float(kw['sleep']) del kw['sleep'] else: sleep = 0.0001 if kw.has_key('repeat'): repeatp = float(kw['repeat']) del kw['repeat'] else: repeatp = 0 jobs.append((job, kw, frequency, sleep, repeatp)) else: raise ValueError("not an option or job", name) else: raise ValueError("Expected an option", arg) def find_lib_python(): for b in os.getcwd(), os.path.split(sys.argv[0])[0]: for i in range(6): d = ['..']*i + ['lib', 'python'] p = os.path.join(b, *d) if os.path.isdir(p): return p raise ValueError("Couldn't find lib/python") def main(args=None): lib_python = find_lib_python() sys.path.insert(0, lib_python) if args is None: args = sys.argv[1:] if not args: print __doc__ sys.exit(0) print args random.seed(hash(tuple(args))) # always use the same for the given args options = {"mbox": None, "threads": None} jobdefs = [] collect_options(args, jobdefs, options) mboxes = {} if options["mbox"]: mboxes[options["mbox"]] = MBox(options["mbox"]) # Perform a ZConfig-based Zope initialization: zetup(os.path.join(lib_python, '..', '..', 'etc', 'zope.conf')) if options.has_key('setup'): setup(lib_python) else: import Zope2 Zope2.startup() #from ThreadedAsync.LoopCallback import loop #threading.Thread(target=loop, args=(), name='asyncore').start() jobs = JobProducer() for job, kw, frequency, sleep, repeatp in jobdefs: Job = globals()[job.capitalize()+'Job'] if getattr(Job, 'needs_mbox', 0): if not kw.has_key("mbox"): if not options["mbox"]: raise ValueError( "no mailbox (mbox option) file specified") kw['mbox'] = mboxes[options["mbox"]] else: if not mboxes.has_key[kw["mbox"]]: mboxes[kw['mbox']] = MBox[kw['mbox']] kw["mbox"] = mboxes[kw['mbox']] jobs.add(Job(**kw), frequency, sleep, repeatp) if not jobs: print "No jobs to execute" return threads = int(options['threads'] or '0') if threads > 1: threads = [threading.Thread(target=run, args=(jobs, i), name=str(i)) for i in range(threads)] for thread in threads: thread.start() for thread in threads: thread.join() else: run(jobs) def zetup(configfile_name): from Zope.Startup.options import ZopeOptions from Zope.Startup import handlers as h from App import config opts = ZopeOptions() opts.configfile = configfile_name opts.realize(args=[]) h.handleConfig(opts.configroot, opts.confighandlers) config.setConfiguration(opts.configroot) from Zope.Startup import dropPrivileges dropPrivileges(opts.configroot) if __name__ == '__main__': main() Zope-2.8.7-final/utilities/ZODBTools/analyze.py0000775000076400007640000001015610435500417020354 0ustar ajungajung#!/usr/bin/env python2.3 # Based on a transaction analyzer by Matt Kromer. import pickle import re import sys import types from ZODB.FileStorage import FileStorage class Report: def __init__(self): self.OIDMAP = {} self.TYPEMAP = {} self.TYPESIZE = {} self.FREEMAP = {} self.USEDMAP = {} self.TIDS = 0 self.OIDS = 0 self.DBYTES = 0 self.COIDS = 0 self.CBYTES = 0 self.FOIDS = 0 self.FBYTES = 0 def shorten(s, n): l = len(s) if l <= n: return s while len(s) + 3 > n: # account for ... i = s.find(".") if i == -1: # In the worst case, just return the rightmost n bytes return s[-n:] else: s = s[i + 1:] l = len(s) return "..." + s def report(rep): print "Processed %d records in %d transactions" % (rep.OIDS, rep.TIDS) print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS) print ("Average transaction size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.TIDS)) print "Types used:" fmt = "%-46s %7s %9s %6s %7s" fmtp = "%-46s %7d %9d %5.1f%% %7.2f" # per-class format fmts = "%46s %7d %8dk %5.1f%% %7.2f" # summary format print fmt % ("Class Name", "Count", "TBytes", "Pct", "AvgSize") print fmt % ('-'*46, '-'*7, '-'*9, '-'*5, '-'*7) typemap = rep.TYPEMAP.keys() typemap.sort() cumpct = 0.0 for t in typemap: pct = rep.TYPESIZE[t] * 100.0 / rep.DBYTES cumpct += pct print fmtp % (shorten(t, 46), rep.TYPEMAP[t], rep.TYPESIZE[t], pct, rep.TYPESIZE[t] * 1.0 / rep.TYPEMAP[t]) print fmt % ('='*46, '='*7, '='*9, '='*5, '='*7) print "%46s %7d %9s %6s %6.2fk" % ('Total Transactions', rep.TIDS, ' ', ' ', rep.DBYTES * 1.0 / rep.TIDS / 1024.0) print fmts % ('Total Records', rep.OIDS, rep.DBYTES / 1024.0, cumpct, rep.DBYTES * 1.0 / rep.OIDS) print fmts % ('Current Objects', rep.COIDS, rep.CBYTES / 1024.0, rep.CBYTES * 100.0 / rep.DBYTES, rep.CBYTES * 1.0 / rep.COIDS) if rep.FOIDS: print fmts % ('Old Objects', rep.FOIDS, rep.FBYTES / 1024.0, rep.FBYTES * 100.0 / rep.DBYTES, rep.FBYTES * 1.0 / rep.FOIDS) def analyze(path): fs = FileStorage(path, read_only=1) fsi = fs.iterator() report = Report() for txn in fsi: analyze_trans(report, txn) return report def analyze_trans(report, txn): report.TIDS += 1 for rec in txn: analyze_rec(report, rec) def get_type(record): try: classinfo = pickle.loads(record.data)[0] except SystemError, err: s = str(err) mo = re.match('Failed to import class (\S+) from module (\S+)', s) if mo is None: raise else: klass, mod = mo.group(1, 2) return "%s.%s" % (mod, klass) if isinstance(classinfo, types.TupleType): mod, klass = classinfo return "%s.%s" % (mod, klass) else: return str(classinfo) def analyze_rec(report, record): oid = record.oid report.OIDS += 1 if record.data is None: # No pickle -- aborted version or undo of object creation. return try: size = len(record.data) # Ignores various overhead report.DBYTES += size if oid not in report.OIDMAP: type = get_type(record) report.OIDMAP[oid] = type report.USEDMAP[oid] = size report.COIDS += 1 report.CBYTES += size else: type = report.OIDMAP[oid] fsize = report.USEDMAP[oid] report.FREEMAP[oid] = report.FREEMAP.get(oid, 0) + fsize report.USEDMAP[oid] = size report.FOIDS += 1 report.FBYTES += fsize report.CBYTES += size - fsize report.TYPEMAP[type] = report.TYPEMAP.get(type, 0) + 1 report.TYPESIZE[type] = report.TYPESIZE.get(type, 0) + size except Exception, err: print err if __name__ == "__main__": path = sys.argv[1] report(analyze(path)) Zope-2.8.7-final/utilities/ZODBTools/fstest.py0000664000076400007640000001502610435500417020217 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2001, 2002 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """Simple consistency checker for FileStorage. usage: fstest.py [-v] data.fs The fstest tool will scan all the data in a FileStorage and report an error if it finds any corrupt transaction data. The tool will print a message when the first error is detected, then exit. The tool accepts one or more -v arguments. If a single -v is used, it will print a line of text for each transaction record it encounters. If two -v arguments are used, it will also print a line of text for each object. The objects for a transaction will be printed before the transaction itself. Note: It does not check the consistency of the object pickles. It is possible for the damage to occur only in the part of the file that stores object pickles. Those errors will go undetected. """ # The implementation is based closely on the read_index() function in # ZODB.FileStorage. If anything about the FileStorage layout changes, # this file will need to be udpated. import string import struct import sys class FormatError(ValueError): """There is a problem with the format of the FileStorage.""" class Status: checkpoint = 'c' undone = 'u' packed_version = 'FS21' TREC_HDR_LEN = 23 DREC_HDR_LEN = 42 VERBOSE = 0 def hexify(s): """Format an 8-bite string as hex""" l = [] for c in s: h = hex(ord(c)) if h[:2] == '0x': h = h[2:] if len(h) == 1: l.append("0") l.append(h) return "0x" + string.join(l, '') def chatter(msg, level=1): if VERBOSE >= level: sys.stdout.write(msg) def U64(v): """Unpack an 8-byte string as a 64-bit long""" h, l = struct.unpack(">II", v) if h: return (h << 32) + l else: return l def check(path): file = open(path, 'rb') file.seek(0, 2) file_size = file.tell() if file_size == 0: raise FormatError("empty file") file.seek(0) if file.read(4) != packed_version: raise FormatError("invalid file header") pos = 4L tid = '\000' * 8 # lowest possible tid to start i = 0 while pos: _pos = pos pos, tid = check_trec(path, file, pos, tid, file_size) if tid is not None: chatter("%10d: transaction tid %s #%d \n" % (_pos, hexify(tid), i)) i = i + 1 def check_trec(path, file, pos, ltid, file_size): """Read an individual transaction record from file. Returns the pos of the next transaction and the transaction id. It also leaves the file pointer set to pos. The path argument is used for generating error messages. """ h = file.read(TREC_HDR_LEN) if not h: return None, None if len(h) != TREC_HDR_LEN: raise FormatError("%s truncated at %s" % (path, pos)) tid, stl, status, ul, dl, el = struct.unpack(">8s8scHHH", h) tmeta_len = TREC_HDR_LEN + ul + dl + el if tid <= ltid: raise FormatError("%s time-stamp reduction at %s: %s <= %s" % (path, pos, hexify(tid), hexify(ltid))) ltid = tid tl = U64(stl) # transaction record length - 8 if pos + tl + 8 > file_size: raise FormatError("%s truncated possibly because of" " damaged records at %s" % (path, pos)) if status == Status.checkpoint: raise FormatError("%s checkpoint flag was not cleared at %s" % (path, pos)) if status not in ' up': raise FormatError("%s has invalid status '%s' at %s" % (path, status, pos)) if tmeta_len > tl: raise FormatError("%s has an invalid transaction header" " at %s" % (path, pos)) tpos = pos tend = tpos + tl if status != Status.undone: pos = tpos + tmeta_len file.read(ul + dl + el) # skip transaction metadata i = 0 while pos < tend: _pos = pos pos, oid = check_drec(path, file, pos, tpos, tid) if pos > tend: raise FormatError("%s has data records that extend beyond" " the transaction record; end at %s" % (path, pos)) chatter("%10d: object oid %s #%d\n" % (_pos, hexify(oid), i), level=2) i = i + 1 file.seek(tend) rtl = file.read(8) if rtl != stl: raise FormatError("%s has inconsistent transaction length" " for undone transaction at %s" % (path, pos)) pos = tend + 8 return pos, tid def check_drec(path, file, pos, tpos, tid): """Check a data record for the current transaction record""" h = file.read(DREC_HDR_LEN) if len(h) != DREC_HDR_LEN: raise FormatError("%s truncated at %s" % (path, pos)) oid, serial, _prev, _tloc, vlen, _plen = ( struct.unpack(">8s8s8s8sH8s", h)) prev = U64(_prev) tloc = U64(_tloc) plen = U64(_plen) dlen = DREC_HDR_LEN + (plen or 8) if vlen: dlen = dlen + 16 + vlen file.seek(8, 1) pv = U64(file.read(8)) file.seek(vlen, 1) # skip the version data if tloc != tpos: raise FormatError("%s data record exceeds transaction record " "at %s: tloc %d != tpos %d" % (path, pos, tloc, tpos)) pos = pos + dlen if plen: file.seek(plen, 1) else: file.seek(8, 1) # _loadBack() ? return pos, oid def usage(): print __doc__ sys.exit(-1) if __name__ == "__main__": import getopt try: opts, args = getopt.getopt(sys.argv[1:], 'v') if len(args) != 1: raise ValueError, "expected one argument" for k, v in opts: if k == '-v': VERBOSE = VERBOSE + 1 except (getopt.error, ValueError): usage() try: check(args[0]) except FormatError, msg: print msg sys.exit(-1) chatter("no errors detected") Zope-2.8.7-final/utilities/ZODBTools/checkbtrees.py0000775000076400007640000000601310435500417021170 0ustar ajungajung#!/usr/bin/env python2.3 """Check the consistency of BTrees in a Data.fs usage: checkbtrees.py data.fs Try to find all the BTrees in a Data.fs, call their _check() methods, and run them through BTrees.check.check(). """ from types import IntType import ZODB from ZODB.FileStorage import FileStorage from BTrees.check import check # Set of oids we've already visited. Since the object structure is # a general graph, this is needed to prevent unbounded paths in the # presence of cycles. It's also helpful in eliminating redundant # checking when a BTree is pointed to by many objects. oids_seen = {} # Append (obj, path) to L if and only if obj is a persistent object # and we haven't seen it before. def add_if_new_persistent(L, obj, path): global oids_seen getattr(obj, '_', None) # unghostify if hasattr(obj, '_p_oid'): oid = obj._p_oid if not oids_seen.has_key(oid): L.append((obj, path)) oids_seen[oid] = 1 def get_subobjects(obj): getattr(obj, '_', None) # unghostify sub = [] try: attrs = obj.__dict__.items() except AttributeError: attrs = () for pair in attrs: sub.append(pair) # what if it is a mapping? try: items = obj.items() except AttributeError: items = () for k, v in items: if not isinstance(k, IntType): sub.append(("", k)) if not isinstance(v, IntType): sub.append(("[%s]" % repr(k), v)) # what if it is a sequence? i = 0 while 1: try: elt = obj[i] except: break sub.append(("[%d]" % i, elt)) i += 1 return sub def main(fname): fs = FileStorage(fname, read_only=1) cn = ZODB.DB(fs).open() rt = cn.root() todo = [] add_if_new_persistent(todo, rt, '') found = 0 while todo: obj, path = todo.pop(0) found += 1 if not path: print "", repr(obj) else: print path, repr(obj) mod = str(obj.__class__.__module__) if mod.startswith("BTrees"): if hasattr(obj, "_check"): try: obj._check() except AssertionError, msg: print "*" * 60 print msg print "*" * 60 try: check(obj) except AssertionError, msg: print "*" * 60 print msg print "*" * 60 if found % 100 == 0: cn.cacheMinimize() for k, v in get_subobjects(obj): if k.startswith('['): # getitem newpath = "%s%s" % (path, k) else: newpath = "%s.%s" % (path, k) add_if_new_persistent(todo, v, newpath) print "total", len(fs._index), "found", found if __name__ == "__main__": import sys try: fname, = sys.argv[1:] except: print __doc__ sys.exit(2) main(fname) Zope-2.8.7-final/utilities/ZODBTools/fsstats.py0000775000076400007640000001303710435500417020401 0ustar ajungajung#!/usr/bin/env python2.3 """Print details statistics from fsdump output.""" import re import sys rx_txn = re.compile("tid=([0-9a-f]+).*size=(\d+)") rx_data = re.compile("oid=([0-9a-f]+) class=(\S+) size=(\d+)") def sort_byhsize(seq, reverse=False): L = [(v.size(), k, v) for k, v in seq] L.sort() if reverse: L.reverse() return [(k, v) for n, k, v in L] class Histogram(dict): def add(self, size): self[size] = self.get(size, 0) + 1 def size(self): return sum(self.itervalues()) def mean(self): product = sum([k * v for k, v in self.iteritems()]) return product / self.size() def median(self): # close enough? n = self.size() / 2 L = self.keys() L.sort() L.reverse() while 1: k = L.pop() if self[k] > n: return k n -= self[k] def mode(self): mode = 0 value = 0 for k, v in self.iteritems(): if v > value: value = v mode = k return mode def make_bins(self, binsize): maxkey = max(self.iterkeys()) self.binsize = binsize self.bins = [0] * (1 + maxkey / binsize) for k, v in self.iteritems(): b = k / binsize self.bins[b] += v def report(self, name, binsize=50, usebins=False, gaps=True, skip=True): if usebins: # Use existing bins with whatever size they have binsize = self.binsize else: # Make new bins self.make_bins(binsize) maxval = max(self.bins) # Print up to 40 dots for a value dot = max(maxval / 40, 1) tot = sum(self.bins) print name print "Total", tot, print "Median", self.median(), print "Mean", self.mean(), print "Mode", self.mode(), print "Max", max(self) print "One * represents", dot gap = False cum = 0 for i, n in enumerate(self.bins): if gaps and (not n or (skip and not n / dot)): if not gap: print " ..." gap = True continue gap = False p = 100 * n / tot cum += n pc = 100 * cum / tot print "%6d %6d %3d%% %3d%% %s" % ( i * binsize, n, p, pc, "*" * (n / dot)) print def class_detail(class_size): # summary of classes fmt = "%5s %6s %6s %6s %-50.50s" labels = ["num", "median", "mean", "mode", "class"] print fmt % tuple(labels) print fmt % tuple(["-" * len(s) for s in labels]) for klass, h in sort_byhsize(class_size.iteritems()): print fmt % (h.size(), h.median(), h.mean(), h.mode(), klass) print # per class details for klass, h in sort_byhsize(class_size.iteritems(), reverse=True): h.make_bins(50) if len(filter(None, h.bins)) == 1: continue h.report("Object size for %s" % klass, usebins=True) def revision_detail(lifetimes, classes): # Report per-class details for any object modified more than once for name, oids in classes.iteritems(): h = Histogram() keep = False for oid in dict.fromkeys(oids, 1): L = lifetimes.get(oid) n = len(L) h.add(n) if n > 1: keep = True if keep: h.report("Number of revisions for %s" % name, binsize=10) def main(path): txn_objects = Histogram() # histogram of txn size in objects txn_bytes = Histogram() # histogram of txn size in bytes obj_size = Histogram() # histogram of object size n_updates = Histogram() # oid -> num updates n_classes = Histogram() # class -> num objects lifetimes = {} # oid -> list of tids class_size = {} # class -> histogram of object size classes = {} # class -> list of oids MAX = 0 tid = None f = open(path, "rb") for i, line in enumerate(f): if MAX and i > MAX: break if line.startswith(" data"): m = rx_data.search(line) if not m: continue oid, klass, size = m.groups() size = int(size) obj_size.add(size) n_updates.add(oid) n_classes.add(klass) h = class_size.get(klass) if h is None: h = class_size[klass] = Histogram() h.add(size) L = lifetimes.setdefault(oid, []) L.append(tid) L = classes.setdefault(klass, []) L.append(oid) objects += 1 elif line.startswith("Trans"): if tid is not None: txn_objects.add(objects) m = rx_txn.search(line) if not m: continue tid, size = m.groups() size = int(size) objects = 0 txn_bytes.add(size) f.close() print "Summary: %d txns, %d objects, %d revisions" % ( txn_objects.size(), len(n_updates), n_updates.size()) print txn_bytes.report("Transaction size (bytes)", binsize=1024) txn_objects.report("Transaction size (objects)", binsize=10) obj_size.report("Object size", binsize=128) # object lifetime info h = Histogram() for k, v in lifetimes.items(): h.add(len(v)) h.report("Number of revisions", binsize=10, skip=False) # details about revisions revision_detail(lifetimes, classes) class_detail(class_size) if __name__ == "__main__": main(sys.argv[1]) Zope-2.8.7-final/utilities/ZODBTools/zeopack.py0000775000076400007640000000616110435500417020346 0ustar ajungajung#!/usr/bin/env python2.3 """Connect to a ZEO server and ask it to pack. Usage: zeopack.py [options] Options: -p port -- port to connect to -h host -- host to connect to (default is current host) -U path -- Unix-domain socket to connect to -S name -- storage name (default is '1') -d days -- pack objects more than days old -1 -- Connect to a ZEO 1 server -W -- wait for server to come up. Normally the script tries to connect for 10 seconds, then exits with an error. The -W option is only supported with ZEO 1. You must specify either -p and -h or -U. """ import getopt import socket import sys import time from ZEO.ClientStorage import ClientStorage WAIT = 10 # wait no more than 10 seconds for client to connect def connect(storage): # The connect-on-startup logic that ZEO provides isn't too useful # for this script. We'd like to client to attempt to startup, but # fail if it can't get through to the server after a reasonable # amount of time. There's no external support for this, so we'll # expose the ZEO 1.0 internals. (consenting adults only) t0 = time.time() while t0 + WAIT > time.time(): storage._call.connect() if storage._connected: return raise RuntimeError, "Unable to connect to ZEO server" def pack1(addr, storage, days, wait): cs = ClientStorage(addr, storage=storage, wait_for_server_on_startup=wait) if wait: # _startup() is an artifact of the way ZEO 1.0 works. The # ClientStorage doesn't get fully initialized until registerDB() # is called. The only thing we care about, though, is that # registerDB() calls _startup(). cs._startup() else: connect(cs) cs.invalidator = None cs.pack(wait=1, days=days) cs.close() def pack2(addr, storage, days): cs = ClientStorage(addr, storage=storage, wait=1, read_only=1) cs.pack(wait=1, days=days) cs.close() def usage(exit=1): print __doc__ print " ".join(sys.argv) sys.exit(exit) def main(): host = None port = None unix = None storage = '1' days = 0 wait = 0 zeoversion = 2 try: opts, args = getopt.getopt(sys.argv[1:], 'p:h:U:S:d:W1') for o, a in opts: if o == '-p': port = int(a) elif o == '-h': host = a elif o == '-U': unix = a elif o == '-S': storage = a elif o == '-d': days = int(a) elif o == '-W': wait = 1 elif o == '-1': zeoversion = 1 except Exception, err: print err usage() if unix is not None: addr = unix else: if host is None: host = socket.gethostname() if port is None: usage() addr = host, port if zeoversion == 1: pack1(addr, storage, days, wait) else: pack2(addr, storage, days) if __name__ == "__main__": try: main() except Exception, err: print err sys.exit(1) Zope-2.8.7-final/utilities/ZODBTools/zeoqueue.py0000775000076400007640000002553510435500417020562 0ustar ajungajung#!/usr/bin/env python2.3 """Report on the number of currently waiting clients in the ZEO queue. Usage: %(PROGRAM)s [options] logfile Options: -h / --help Print this help text and exit. -v / --verbose Verbose output -f file --file file Use the specified file to store the incremental state as a pickle. If not given, %(STATEFILE)s is used. -r / --reset Reset the state of the tool. This blows away any existing state pickle file and then exits -- it does not parse the file. Use this when you rotate log files so that the next run will parse from the beginning of the file. """ import os import re import sys import time import errno import getopt import cPickle as pickle COMMASPACE = ', ' STATEFILE = 'zeoqueue.pck' PROGRAM = sys.argv[0] try: True, False except NameError: True = 1 False = 0 tcre = re.compile(r""" (?P \d{4}- # year \d{2}- # month \d{2}) # day T # separator (?P \d{2}: # hour \d{2}: # minute \d{2}) # second """, re.VERBOSE) ccre = re.compile(r""" zrpc-conn:(?P\d+.\d+.\d+.\d+:\d+)\s+ calling\s+ (?P \w+) # the method \( # args open paren \' # string quote start (?P \S+) # first argument -- usually the tid \' # end of string (?P .*) # rest of line """, re.VERBOSE) wcre = re.compile(r'Clients waiting: (?P\d+)') def parse_time(line): """Return the time portion of a zLOG line in seconds or None.""" mo = tcre.match(line) if mo is None: return None date, time_ = mo.group('ymd', 'hms') date_l = [int(elt) for elt in date.split('-')] time_l = [int(elt) for elt in time_.split(':')] return int(time.mktime(date_l + time_l + [0, 0, 0])) class Txn: """Track status of single transaction.""" def __init__(self, tid): self.tid = tid self.hint = None self.begin = None self.vote = None self.abort = None self.finish = None self.voters = [] def isactive(self): if self.begin and not (self.abort or self.finish): return True else: return False class Status: """Track status of ZEO server by replaying log records. We want to keep track of several events: - The last committed transaction. - The last committed or aborted transaction. - The last transaction that got the lock but didn't finish. - The client address doing the first vote of a transaction. - The number of currently active transactions. - The number of reported queued transactions. - Client restarts. - Number of current connections (but this might not be useful). We can observe these events by reading the following sorts of log entries: 2002-12-16T06:16:05 BLATHER(-100) zrpc:12649 calling tpc_begin('\x03I\x90((\xdbp\xd5', '', 'QueueCatal... 2002-12-16T06:16:06 BLATHER(-100) zrpc:12649 calling vote('\x03I\x90((\xdbp\xd5') 2002-12-16T06:16:06 BLATHER(-100) zrpc:12649 calling tpc_finish('\x03I\x90((\xdbp\xd5') 2002-12-16T10:46:10 INFO(0) ZSS:12649:1 Transaction blocked waiting for storage. Clients waiting: 1. 2002-12-16T06:15:57 BLATHER(-100) zrpc:12649 connect from ('10.0.26.54', 48983): 2002-12-16T10:30:09 INFO(0) ZSS:12649:1 disconnected """ def __init__(self): self.lineno = 0 self.pos = 0 self.reset() def reset(self): self.commit = None self.commit_or_abort = None self.last_unfinished = None self.n_active = 0 self.n_blocked = 0 self.n_conns = 0 self.t_restart = None self.txns = {} def iscomplete(self): # The status report will always be complete if we encounter an # explicit restart. if self.t_restart is not None: return True # If we haven't seen a restart, assume that seeing a finished # transaction is good enough. return self.commit is not None def process_file(self, fp): if self.pos: if VERBOSE: print 'seeking to file position', self.pos fp.seek(self.pos) while True: line = fp.readline() if not line: break self.lineno += 1 self.process(line) self.pos = fp.tell() def process(self, line): if line.find("calling") != -1: self.process_call(line) elif line.find("connect") != -1: self.process_connect(line) # test for "locked" because word may start with "B" or "b" elif line.find("locked") != -1: self.process_block(line) elif line.find("Starting") != -1: self.process_start(line) def process_call(self, line): mo = ccre.search(line) if mo is None: return called_method = mo.group('method') # Exit early if we've got zeoLoad, because it's the most # frequently called method and we don't use it. if called_method == "zeoLoad": return t = parse_time(line) meth = getattr(self, "call_%s" % called_method, None) if meth is None: return client = mo.group('addr') tid = mo.group('tid') rest = mo.group('rest') meth(t, client, tid, rest) def process_connect(self, line): pass def process_block(self, line): mo = wcre.search(line) if mo is None: # assume that this was a restart message for the last blocked # transaction. self.n_blocked = 0 else: self.n_blocked = int(mo.group('num')) def process_start(self, line): if line.find("Starting ZEO server") != -1: self.reset() self.t_restart = parse_time(line) def call_tpc_begin(self, t, client, tid, rest): txn = Txn(tid) txn.begin = t if rest[0] == ',': i = 1 while rest[i].isspace(): i += 1 rest = rest[i:] txn.hint = rest self.txns[tid] = txn self.n_active += 1 self.last_unfinished = txn def call_vote(self, t, client, tid, rest): txn = self.txns.get(tid) if txn is None: print "Oops!" txn = self.txns[tid] = Txn(tid) txn.vote = t txn.voters.append(client) def call_tpc_abort(self, t, client, tid, rest): txn = self.txns.get(tid) if txn is None: print "Oops!" txn = self.txns[tid] = Txn(tid) txn.abort = t txn.voters = [] self.n_active -= 1 if self.commit_or_abort: # delete the old transaction try: del self.txns[self.commit_or_abort.tid] except KeyError: pass self.commit_or_abort = txn def call_tpc_finish(self, t, client, tid, rest): txn = self.txns.get(tid) if txn is None: print "Oops!" txn = self.txns[tid] = Txn(tid) txn.finish = t txn.voters = [] self.n_active -= 1 if self.commit: # delete the old transaction try: del self.txns[self.commit.tid] except KeyError: pass if self.commit_or_abort: # delete the old transaction try: del self.txns[self.commit_or_abort.tid] except KeyError: pass self.commit = self.commit_or_abort = txn def report(self): print "Blocked transactions:", self.n_blocked if not VERBOSE: return if self.t_restart: print "Server started:", time.ctime(self.t_restart) if self.commit is not None: t = self.commit_or_abort.finish if t is None: t = self.commit_or_abort.abort print "Last finished transaction:", time.ctime(t) # the blocked transaction should be the first one that calls vote L = [(txn.begin, txn) for txn in self.txns.values()] L.sort() for x, txn in L: if txn.isactive(): began = txn.begin if txn.voters: print "Blocked client (first vote):", txn.voters[0] print "Blocked transaction began at:", time.ctime(began) print "Hint:", txn.hint print "Idle time: %d sec" % int(time.time() - began) break def usage(code, msg=''): print >> sys.stderr, __doc__ % globals() if msg: print >> sys.stderr, msg sys.exit(code) def main(): global VERBOSE VERBOSE = 0 file = STATEFILE reset = False # -0 is a secret option used for testing purposes only seek = True try: opts, args = getopt.getopt(sys.argv[1:], 'vhf:r0', ['help', 'verbose', 'file=', 'reset']) except getopt.error, msg: usage(1, msg) for opt, arg in opts: if opt in ('-h', '--help'): usage(0) elif opt in ('-v', '--verbose'): VERBOSE += 1 elif opt in ('-f', '--file'): file = arg elif opt in ('-r', '--reset'): reset = True elif opt == '-0': seek = False if reset: # Blow away the existing state file and exit try: os.unlink(file) if VERBOSE: print 'removing pickle state file', file except OSError, e: if e.errno <> errno.ENOENT: raise return if not args: usage(1, 'logfile is required') if len(args) > 1: usage(1, 'too many arguments: %s' % COMMASPACE.join(args)) path = args[0] # Get the previous status object from the pickle file, if it is available # and if the --reset flag wasn't given. status = None try: statefp = open(file, 'rb') try: status = pickle.load(statefp) if VERBOSE: print 'reading status from file', file finally: statefp.close() except IOError, e: if e.errno <> errno.ENOENT: raise if status is None: status = Status() if VERBOSE: print 'using new status' if not seek: status.pos = 0 fp = open(path, 'rb') try: status.process_file(fp) finally: fp.close() # Save state statefp = open(file, 'wb') pickle.dump(status, statefp, 1) statefp.close() # Print the report and return the number of blocked clients in the exit # status code. status.report() sys.exit(status.n_blocked) if __name__ == "__main__": main() Zope-2.8.7-final/utilities/ZODBTools/fsoids.py0000664000076400007640000000446310435500417020201 0ustar ajungajung#!/usr/bin/env python2.3 ############################################################################## # # Copyright (c) 2004 Zope Corporation and Contributors. # All Rights Reserved. # # This software is subject to the provisions of the Zope Public License, # Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution. # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS # FOR A PARTICULAR PURPOSE # ############################################################################## """FileStorage oid-tracer. usage: fsoids.py [-f oid_file] Data.fs [oid]... Display information about all occurrences of specified oids in a FileStorage. This is meant for heavy debugging. This includes all revisions of the oids, all objects referenced by the oids, and all revisions of all objects referring to the oids. If specified, oid_file is an input text file, containing one oid per line. oids are specified as integers, in any of Python's integer notations (typically like 0x341a). One or more oids can also be specified on the command line. The output is grouped by oid, from smallest to largest, and sub-grouped by transaction, from oldest to newest. This will not alter the FileStorage, but running against a live FileStorage is not recommended (spurious error messages may result). See testfsoids.py for a tutorial doctest. """ import sys from ZODB.FileStorage.fsoids import Tracer def usage(): print __doc__ def main(): import getopt try: opts, args = getopt.getopt(sys.argv[1:], 'f:') if not args: usage() raise ValueError, "Must specify a FileStorage" path = None for k, v in opts: if k == '-f': path = v except (getopt.error, ValueError): usage() raise c = Tracer(args[0]) for oid in args[1:]: as_int = int(oid, 0) # 0 == auto-detect base c.register_oids(as_int) if path is not None: for line in open(path): as_int = int(line, 0) c.register_oids(as_int) if not c.oids: raise ValueError("no oids specified") c.run() c.report() if __name__ == "__main__": main() Zope-2.8.7-final/log.ini0000664000076400007640000000111210435500314014010 0ustar ajungajung# This file configures the logging module for the test harness: # critical errors are logged to testing.log; everything else is # ignored. # Documentation for the file format is at # http://www.red-dove.com/python_logging.html#config [logger_root] level=CRITICAL handlers=normal [handler_normal] class=FileHandler level=NOTSET formatter=common args=('testing.log', 'a') filename=testing.log mode=a [formatter_common] format=------ %(asctime)s %(levelname)s %(name)s %(message)s datefmt=%Y-%m-%dT%H:%M:%S [loggers] keys=root [handlers] keys=normal [formatters] keys=common