DoctestIdeas

• Review (already checked in to Python)

  Current Z3 changes

  • cleanup of DocTestUnit - DONE
  • Setup/teardown - DONE
  • DocTestFile (could/should be done better; signature is off) - DONE

• New doctest gimmicks >= 2.3 need documentation.

• postmortem debugging - DONE

  produce

      code
      __compare_output()
      

  as one code block, so the code is still active if the actual output doesn't match.

  Scratch that -- too difficult, for "technical reasons". Instead a debugging-aware "the output is different" reporter needs access to the globals the failing example ran in. It can exec a statement that raises an exception, using those globals as the exec's context. Then the debugger will have access to all the bindings the failing example used.

• set_trace - DONE

  monkey-patch pdb.set_trace, to first restore stdout

• Pass globals to DocTestSuite - DONE
  • globs + extraglobs - DONE
  • augment/override flag?
  • make copy of globals (or module globals if globs is None); "merge" extraglobs if given - DONE
  • The intent of extraglobs is to make a doctest somewhat parameterizable, so that the same test can be invoked multiple times with different bindings for some of its globals. For example, a doctest written for a base class may be usable for (partially) testing subclasses too, if the test uses a generic name for the class being tested and this name is bound to the actual class being tested via extraglobs.

• Refactor doctest - DONE
  • make more pluggable.

    Should this be done with callbacks, or by defining a class that can be subclassed, with methods overridden? -edloper

    DONE, via subclassing.

    • comparison - DONE
    • output - DONE
    • which objects to test

      Unclear what this is really about. There's the now-deprecated "private name" gimmick. Filtering a DocTestFinder list seems better than building in obscure options.

    • callbacks for
      • comparison failed - DONE
      • unexpected exception - DONE
  • get rid of umpteen module-level functions with 15 args each - DONE
  • use common code to find things to test (why did Jim write his own?) - DONE
  • the "private name" idea was/is stupid, and should go away - DONE as far as possible for 2.4, by raising DeprecationWarning; can't go away until 2.5 at earliest

• blank line gimmick - DONE, via <BLANKLINE> markup, enabled by default

• multiline traceback message - DONE, multi-line exception detail works now

  doctest insists on 1-line msg now

  (does this mean that I no longer will have my doctests cluttered with "Traceback (most recent call last):"?)

  • I don't think so -- I'm assuming that this refers to the fact that doctest can't currently deal with things like "raise ValueError?(x\ny\nz)", where the exception message (value) spans multiple lines. But if there's demand, we might be able to add an additional string to signal an exception (we already accept both "most recent call last" and "innermost last" variants of the Traceback line).) -edloper

• better diff output when test fails - DONE, context and unified diffs are options now.

  esp./only multi-line expected output; it can be very hard to see what's different (same thing w/ unittest's assertEqual).

• elided parts - DONE, via ELLIPSIS option

  generalization of traceback junk-skipping

• pretty printer

  Unclear there's a strong need for this anymore. The new NORMALIZE_WHITESPACE comparison option goes a long way toward meeting the use cases I had (like showing expected output for range(1000) on a 100 lines instead of 1 line).

... --[ManuelG]?, 2004/07/22 19:30 EST reply

• currently if you assign a shorter name, i.e., Punter = NewImprovedPrinterPunter?, NewImprovedPrinterPunter?'s test get run twice

• doctest and unittest work well together, I make passing doctest one of tests in unittest, sometimes the only test is checking doctest strings

  when a doctest is starting to look hairy, I usually have a bunch of tests that could efficiently use some ad-hoc framework, and the perfect place for that kind of code is in a unittest test

• temporary debugging print messages confuse doctest

  it would be nice if I could put a prefix to all my temp debug print strings, and doctest would just ignore those lines anywhere they might pop up

... --[Zen]?, 2004/07/25 18:42 EST reply
- Fix unittest.main to accept a TestSuite? as an argument allowing DocTestSuite? and TestCase? instances to coexist better in the one file.

• Create a DocTestSuite? from a text file, allowing us to easily test examples in documentation (currently doing ``def test(): pass; test.__doc__ = open(README.txt).read()``)

Walkthrough for Current Implementation --[edloper]?, 2004/07/29 12:54 EST reply
While doing a thourough read-through of the code, I wrote up a walkthrough for doctest.py. It should be useful for anyone else who wants to start looking at the code, and/or as a reference to the code. It's at Doctest136ImplementationSummary.

SpoofOut? vs StringIO? --[edloper]?, 2004/07/30 16:11 EST reply
Is there a good reason to define a new _SpoofOut class, rather than just using StringIO??

hiding imported objects --[edloper]?, 2004/07/30 18:44 EST reply
doctest.py goes to great lengths to make sure that imported modules and objects don't appear in its namespace, or that they're named with leading underscores. E.g., the "re" module is deleted after it's used; _run_examples_inner does its imports within the function body; and imported objects are renamed (via "as") to begin with a leading underscore.

Is this useful, given that the module also defines __all__?

extraglobs --[edloper]?, 2004/07/30 20:33 EST reply
What's the motivation for giving an extra set of globals that should be merged in (extraglobs)? (Under "Pass globals to DocTestSuite?, above.)

hiding imported objects --[tim_one]?, 2004/08/02 15:19 EST reply
doctest existed before __all__ did; that's why it goes to such pain. As you say, not necessary any more.

extraglobs --[tim_one]?, 2004/08/02 15:46 EST reply
Jim wants a way to "parameterize" doctests. For example, write a doctest for a base class, and then reuse it verbatim to test any number of subclasses. The doctest would use a generic name for the actual class getting tested, and extraglobs would map the generic name to the specific class to be tested.

SpoofOut? vs StringIO? --[tim_one]?, 2004/08/02 15:54 EST reply
I don't know whether StringIO? would work. The primary concern at the time was to make sure the implementation could be fiddled in whatever ways proved needed so that the same code would work under Jython too. I'd rather not spend time investigating the current state of Jython now, so "ain't broke, don't fix" rules this one.

doctest & debug printfs --[edloper]?, 2004/08/04 22:52 EST reply
ManuelG? said: "it would be nice if I could put a prefix to all my temp debug print strings, and doctest would just ignore those lines anywhere they might pop up"

I think the best solution here is to explicitly write your debug printfs to either sys.stderr, or to the real sys.stdout (eg with sys.__stdout__). E.g., try debug statements like this:

    if debug: print >>sys.stderr, 'testing'

Aliases --[tim_one]?, 2004/08/06 15:15 EST reply
Manuel G said "currently if you assign a shorter name, i.e., Punter = NewImprovedPrinterPunter, NewImprovedPrinterPunter's test get run twice".

The doctest in 2.4a2 weeds out duplicates, by keeping a set of the addresses (id()s) of the objects it's tested, and skipping an object if it's seen again.

Doctests from text files --[tim_one]?, 2004/08/06 15:18 EST reply
Zen said "Create a DocTestSuite from a text file"

Zope3 has code to do this, and Jim Fulton is adding DocFileSuite() to CVS Python doctest.py. It wasn't ready in time for 2.4a2; it will be in the next Python 2.4 alpha.

Compare return values by ``==`` --[cben]?, 2004/08/14 20:55 EST reply
Doctests like

>>> foo() == {1: 2, 3: 4}
True

One case where you don't usually want comparison by value is in testing stdout (think about debugging a pretty-printer). Fortunately, doctest knows what the tested code returned and what it printed. The printed part should match a [start of]? the expected output as now. The rest of the expected output should be passed through `eval` and the result compared by value to the return value. If `eval` raises, it should be compared by repr as now.

"Comparing by value" should be done by ``res==expected``. Perhaps also requiring ``type(res) is type(expected)`` would be useful but that would not allow `True` in place of `1`. Perhaps ``isinstance(res, type(expected))``?

Of course all this should be pluggable via callbacks. Probably one compare callback that gets the stdout, return value and expected string (for both stdout and return value) from the doctest block would be best.