ZopeTestingGuidelines

Jeremy Hylton in response to MartijnF?'s query about the testing committee:

The committee included a bunch of people: Jim, Jeremy, Jeffrey, Fred, Evan, Chris, Brian.

The simple stuff is really simple:

• Use the standard unittest module.
• Put tests in a subpackage named tests under the package being tested.
• Never write tests that depend on being executed from a particular location.
• If you need to store data in the tests directory, get at it using imports. Example: from ZODB.tests import MinPO? dir = os.path.dirname(MinPO?.__file__) - The test runner (person or program) is responsible for making sure the Python path is set correctly to find all the needed modules.
• Do not modify sys.path!
• Be aware that some tests need to work with Zope and StandaloneZODB?. If you modify those tests, make sure they work with both checkouts.

  packages: ZODB, BTrees?, bsddb3Storage, ZEO, ExtensionClass?, Persistence, ThreadedAsync?, zLOG, zdaemon

There was more stuff about the Testing package in Zope/lib/python. I'm not sure I understand the details, but you need to import it to use it.

There are a bunch of examples that execfile() something called framework.py, but that's deprecated. I believe Tres has started removing it.

Later:

I remembered one more rule (and probably still need to remember a few more). Each test module should define a function test_suite() that returns a unittest TestSuite? object. The affords some flexibility in selecting test cases for a particular environment. The ZEO tests, for example, are different on Win32 than on Unix.

Commentary from MartijnF?:

Tres is indeed on a hunt to remove framework.py. If you indeed take care to put /path/to/Zope/lib/python on your pythonpath (such as through the environment variable PYTHONPATH) yourself somehow I've found things work fine, and the advantage is that the tests become very straightforward to read and maintain.

One thing I've noticed is that to test many modules (i.e. in ParsedXML?) you need to import ZODB to make Persistence work, even though the testing code does not directly refer to Persistence. That's not a testing guideline; perhaps there's a better way, just an observation for folks writing unit tests.

Update on path manipulation

It turns out sys.path manipulation may be okay under some circumstances.

Example code that triggered the discussion:

      if __name__=='__main__':
          import sys
          sys.path.insert(0, '../../..')
          sys.path.insert(0, '..')

The question was: "I thought doing things like this were frowned upon now?"

The key is the condition "__name__ == '__main__': sys.path only gets munged if that module is run as a standalone script, e.g.:

      $ cd lib/python/Products/FooProduct/tests
      $ python checkSomeGnarlyFeature.py

For "regression tests", like the multithread tests ChrisM? put in a separate module yesterday, assuming that the test will be run directly from the command line is fine. ![In fact, it's a feature.]

If, however, the test is a unit test, then the second path insertion is likely to mean that test will fail when run via testrunner, as the parent package won't be in the path at that point. In such a case, the appropriate fix is to do imports from fully-qualified package/module names.