Debugging Zope Python Code

Created by . Last modified on 2003/08/05.

This is really a work in progress: i'm putting it out now because it's pretty juicy, rather than waiting 'til it's presentable. I hope to polish it, and add some juice, over time.

Debugging Zope Python Code

Here are some hints about python-level (external method, product, and core) zope debugging. These are the techniques i depend on - ones that i could not do without.

(See also Michel Pelletier's more recent The Debugger is Your Friend, with more details about some of the stuff i cover here, and no details about some of the other stuff...)

All entail some kind of privileged access - either to the code and in some cases to the to the host installation or to the superuser password. Ie, these techniques are for tightly coupled debugging, close to the server. They are also for recent zopes - 2.0 or later. Make sure you're using python 1.5.2 everywhere...)

Using Tracebacks to do the equivalent of print-flag debugging

When working on a web site - particularly a remote one! - you can't just put prints in your code to narrow down on where and how bugs, particularly fatal ones, are occurring. Ie, you don't have the luxury of an output stream, or even files, where you can send debugging flag output. Zope has some provisions, however, that can be just about as good, or even better, provided you know about them.

First of all, when running Zope normally it hides on the error pages that the web server produces the python traceback for the error, as an HTML "" comment. So if you an error page, do a "View source" to see the traceback.

You get lots of extra mileage for debugging when you run Zope under ZServer [which docs should i ref?] with "debugging mode" activated. To do so on Unix, you have to include the -D option in the ZServer start script - either by passing it on the command line or by including it in the text for the script. In Win32 you have to run Zope from a DOS command, eg with start.bat, not as a service. I'll say more about this below - among other things, it exposes the tracebacks on error pages, so you don't have to do a View Souce.

Whether or not running debug mode, in lotsa situations you can use these tracebacks for a form of flag output by putting, eg:

      raise "DEBUG", "Issue %s has %s items" % (self.id, self.values())

to see the value of x in the middle of a computation.

The string "DEBUG" is not special to Zope - i typically use it for this sort of thing so i can tell what it is if i happen to leave one in accidentally, and it makes it easy to search through for "DEBUG", etc. (Note that your program must not be catching this exception - though it's unlikely to be doing an except "DEBUG":, it may be doing a bare except: and suppressing it. I would be suspicious of such code - in many or even most cases this is bad programming practice, because it can hide bugs that you should be identifying.)

One particularly nice - and important - nuance here is that the Zope publisher will notice the uncaught exception, passing it along but first aborting the current transaction. This means that any changes cause by the current request up to the point of the raised exception will be nullified (in most cases - obviously, if your program sends an email before the exception is raised, there's no way of calling it back). This means that you can redo an activity again and again with no accumulated effects, until your remove your debugging exception!

Under strategic conditions - eg, when other people are accessing the site - you may want to condition the debugging flag according to your login:

       if AUTHENTICATED_USER.getUserName() == 'klm':
           raise "DEBUG", "Issue %s has %s items" % (self.id, self.values())

As mentioned above, there are more benefits to running ZServer in debug mode.

• In addition to exposing the tracebacks on the error pages, it also sets zope so it notices changes to External Methods files and automagically reloads them - so your debugging flags or code fixes take effect without having to restart the site.
• The tracebacks are html_quoted. Objects that have mapping behaviors - eg, python dictionaries, and most notably the Zope REQUEST object - are displayed in a nice key/value table layout, making it easy to examine the REQUEST and other environment settings. Eg:

           raise "DEBUG", (REQUEST.AUTHENTICATED_USER.getUserName(), REQUEST)
  

will first show the user and then show a nicely formatted display of the request - yay!

• Zope tracebacks pay special attention, printing out the value of a variable named __traceback_info__ if it finds it in the stack frame where the error occurred. Eg:

          __traceback_info__ = (x, y, z)
  

to snapshot the values of vars x, y, and z, in case an exception does occur in the function/class/method, even if you don't put an explicit raise in yourself.

• Unlike the exception values, you have to explicitly html_quote values that include special html entities - the classic case is python types, including instance types, which begin with a <, so that the succeeding characters would be invisible until the closing >. So, eg:

           from DocumentTemplate import html_quote
           ...
           raise "DEBUG", html_quote(`self`)
  

In general, for complex objects it's often a good idea to stringify the values::

__traceback_info__ = (`x`, `y`, `z`)

• On Unix type systems the start script won't fork in the background, and it reports the port connections it establishes. (In Win32 you have to run Zope from a DOS command, eg with start.bat, not as a service.) This is crucial for interacting with Python's pdb debugging from a running Zope - see below.
• In unix-type systems, a [Restart] button is added to control panel. Restart enables you to pick up changes to the core python and compiled lower-level code without having to reinvoke any external commands.

[What are the drawbacks, besides exposing the tracebacks to everyone?]

Stepping in a Running Zope with Python's PDB Debugger

[I haven't thought about platform concerns here, only about doing it from Unix - i'm not sure if or how this works under Win32.]

One unexpected benefit of running Zope with ZServer in debug mode (above) you is that it enables you to enter Python's PDB debugger and step through the active code! Briefly, to do this:

• You need to be running ZServer in debug mode
• Where you want to start debugging, put:

         import pdb; pdb.set_trace()
  

• If you do this in an external method, zope will notice the changed file, reload it, and enter the debugger next time you hit that point in the code.
• If in a product or other (eg, core) Zope code, you'll have to restart zope - go to the control panel and hit the restart button.

When it hits that line, the shell running zope will show the pdb prompt, and you can step in, over, and around your code and the underlying Zope core python code. Note that ZServer will be blocked on your activity - other people visiting the site will not get a response until you finish!

Ran out of time for now - in case my raw outline notes help:

   . using the medusa monitor to connect with a running zope
    ; Prerequisites:
     , You must be running ZServer, which must be configured (i think 
       it's default) to start the medusa monitor server.  If you run
       with -D, as above, it will report the port you need to know.
     , You must know your superuser password, and it must be encoded
       in the access file as plain text, not encrypted!
     , You have to be on the host - despite specifying the host in
       the monitor_client command.
    ; in a recent zope checkout (or 2.0 distribution) on the host:
    ; cd ZServer/medusa
    ; python monitor_client.py localhost <monitor port>
     , the port is reported in the ZServer startup
     , must be python 1.5.2 or later
    ; >>> import Zope
    ; >>> app = Zope.app()
    ; now you can access objects in the ZODB and poke around with them:
     , >>> control_panel = app.Control_Panel
     , >>> control_panel.Database.manage_pack(days=2)
     , Whee!  You just packed your database...

   . What about using ZClient??

Suggestions, corrections, inspirations to Ken Manheimer .