wvZopeFilter/0040755000076500001440000000000007552350012012350 5ustar abeluserswvZopeFilter/README0100644000076500001440000001603007552345724013243 0ustar abeluserswvZopeFilter - reduce wvWare/unrtf HTML output wvZopeFilter tries to remove many purely "appearance-related" HTML tags from HTML files written by wvWare or unrtf and to keep only structural information from the documents. While the goal of wvWare and unrtf to extract as much information as possible from Word doc resp. RTF files is admirable and quite often useful, there are situations, where "clean" HTML code is preferable. I am using wvWare and unrtf to allow users to upload text files to a Zope server, and web designers would go crazy, if all the nice fonts like Futura or Gill specified by well-meaning users without real knowledge about design issues in general and especially about HTML design would appear in or CSS tags on the web site. The idea of wvZopeFilter is to extract only structural information from wvWare- or unrtf-produced HTML files, and to use information about the Word style sheets to replace certain

tags with other tags like H1, possibly together with a CSS style reference. Additionally, it removes many redundant tag sequences like , which appear quite frequently in the output of wvWare and unrtf. This can reduce the size of the HTML file by 30%. Configuration wvZopeFilter uses the program wmf2gd from the wvWare package to convert WMF files written by unrtf into PNG files. The path to wmf2gd is currently "hard coded" in the variable WMF2GD near the top of wvZopeFilter.py. If you want to use wvZopeFilter as a stand alone program, you will most likely need to change the dictionary tagDict (defined near the end of wvZopeFilter.py) in order to specify, how Word style sheets shall be transformed into HTML tags and CSS style sheet names. For details about tagDict, see below. Installation You need a Python interpreter, of course. The Python code contains a few constructs like "a += 3" which require a Python version >= 2.0. Getting the "best" version of wvWare and unrtf is a bit more difficult. You can use version "wvWare 0.72":http://sourceforge.net/projects/wvware. In this case, you should use the "conversion description file" wvZope-wv-unpatched.xml. This works fine, but wvWare 0.72 cannot tell us, if a Word style set the font to bold or italic. In order to get this information, you need to patch the wvWare source files with diff output from this package. With this patch installed, you can (and should) use wvZope.xml. This patch includes another patch contributed by Martin Junius to the wvWare project, which implements XML support. Similary, "unrtf 0.18.1":http://www.gnu.org/directory/text/doc/GNUunRTF.html needs a patch in order to provide the Word style sheets names in the output data. You should run the patched version of unrtf with new parameter --xml. Usage To write a HTML file from a Word Doc file, call wvWare with the parameter "-x xmlfile", where xmlfile is either the XML file wvZope.xml or wvZope-wv-unpatched.xml from this package. This ensures, that all

tags will contain a class attribute (the Word style sheet name), which is used by wvZopeFilter to write a certain HTML tag and possibly a CSS style name. If you want to convert an RTF file, you'll need to patch the unrtf sources, so that the unrtf output contains class attributes in the

tags. Call the patched unrtf with the option "--xml". wvZopeFiler.py as a stand-alone program You can use wvZopeFilter as a stand alone program. Simply pass the name of an HTML file written by WvWare or unrtf on the command line. The output is written to stdout. Usage from other Python programs Copy the lines below "if __name__ == '__main__':" and change them to fit your needs ;) Instantiate the class wvZopeFilter; the constructor requires a "tagdict" as a parameter. This is a dictionary which specifies, how the

tags and their Word style names are converted to HTML tags and CSS styles. A dictionary key is the name of a Word style; a dictionary value is a tuple with one or two values. The first value specifies the the HTML tag, which will replace the tag

; the second value, if present, specifies a CSS style name for the HTML tag. The method wvZopeFilter.feed(self, string) is inherited from sgmllib.SGMLPArser. The parameter string is the HTML input. You may call wvZopeFilter.feed more than once with consecutive chunks of HTML input. For details refer to the sgmllib documentation. wvZopeFilter.removeRedundant(self) removes tag sequences like or \s. The return value is the number of removed tags. wvZopeFilter.mergeSameTags(self) merges adjacent , , , 

 tags, i.e. sequences like "text1 text2" are
      replaced by "text1 text2". The return value is the number of 
      "mergers".
      
      wvZopeFilter.removeWordStyleBoldItalic(self) is only useful, if a
      patch is applied to wvWare. If a Word style sheet defines
      bold or italic text, wvWare and unrtf enclose the entire paragraph in
      ... resp. ... tags. Keeping these tags would not allow
      a CSS style sheet to sucessfully define a non-italic or non-bold
      style. The patches add the attributes 'bold="0|1"' and 
      'italic="0|1"' to the 
 tag in the wvWare/unrtf output. The method
      wvZopeFilter.removeWordStyleBoldItalic removes all  resp.  tags
      in paragraphs which have the attribute bold resp. italic set to 1.
      Calling this method for HTML data written by unrtf has currently no
      effect. This does not work currently not for unrtf.
      
      wvZopeFilter.changeTags(self) replaces 
 tags with the tags 
      specified by the dicitonary passed as a parameter to the constructor.
      Word style names which do not appear in the tagdict, are recorded in
      an internal class attribute. A list of these style names can
      be retrieved by calling wvZopeFilter.getUnkownStyles.
      
      wvZopeFilter.convertImages(self) tries to convert WMF files into
      PNG files. I have tested wvZopeFilter only with simple pixel images;
      the conversion works fine for these files, but I expect problems,
      if a WMF file contains text with "fancy" fonts.
      
      wvZopeFilter.removeTags(self, taglist) removes the tags listed in the
      list taglist. This is useful to remove  or 
+
+
wvZopeFilter/wvZope-wv-unpatched.xml0100644000076500001440000002613007552063467017005 0ustar  abelusers


XML




<?xml version="1.0" ?>
<!DOCTYPE doc SYSTEM "wvXml.dtd">







<!--Section Begins-->
<!--Section Ends-->




left
right

center

left
<!--Could Someone who sees this tag tell me what was is this type of justification, asian languages only i thing-->



type="1"
type="I"
type="i"
type="A"
type="a"



none
solid
solid
double
double
solid
dotted
dashed
dotted
dotted
double
dashed
dashed
dashed
dashed
dashed
dashed
dashed
dashed
dashed
solid
double
sashed
dashed
ridge
groove
ridge






















0
0
0
0
0
0


<table width="%" border="1" cols="" rows=""></table>







<tr>

</tr>



<td bgcolor="" width="%" rowspan="" colspan="">


</td>














<b></b>
<i></i>

 
<s></s><a href="#author">[Author ID: at  ]</a>














<u></u><a href="#author">[Author ID: at ]</a>





<s></s>





<s></s>



<sup></sup>
<sub></sub>

<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>

 
<a href="#author">--&gt;</a><a href="#author">[Author ID: at ]</a>





Black
Blue
Cyan
Green
Magenta
Red
Yellow
White
DarkBlue
DarkCyan
DarkGreen
DarkMagenta
DarkRed
#8b8b00
DarkGray
LightGrey



<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>



<a href="#author">--&gt;</a><a href="#author">[Author ID: at  ()]</a>
















<a href="#comment ">--&gt;</a>

<a href="#comment ">[Author:]</a>







wvZopeFilter/wvZope.xml0100644000076500001440000002623407551617434014405 0ustar  abelusers


XML




<?xml version="1.0" ?>
<!DOCTYPE doc SYSTEM "wvXml.dtd">







<!--Section Begins-->
<!--Section Ends-->




left
right

center

left
<!--Could Someone who sees this tag tell me what was is this type of justification, asian languages only i thing-->



type="1"
type="I"
type="i"
type="A"
type="a"



none
solid
solid
double
double
solid
dotted
dashed
dotted
dotted
double
dashed
dashed
dashed
dashed
dashed
dashed
dashed
dashed
dashed
solid
double
sashed
dashed
ridge
groove
ridge






















0
0
0
0
0
0


<table width="%" border="1" cols="" rows=""></table>







<tr>

</tr>



<td bgcolor="" width="%" rowspan="" colspan="">


</td>














<b></b>
<i></i>

 
<s></s><a href="#author">[Author ID: at  ]</a>














<u></u><a href="#author">[Author ID: at ]</a>





<s></s>





<s></s>



<sup></sup>
<sub></sub>

<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>

 
<a href="#author">--&gt;</a><a href="#author">[Author ID: at ]</a>





Black
Blue
Cyan
Green
Magenta
Red
Yellow
White
DarkBlue
DarkCyan
DarkGreen
DarkMagenta
DarkRed
#8b8b00
DarkGray
LightGrey



<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>
<blink></blink>



<a href="#author">--&gt;</a><a href="#author">[Author ID: at  ()]</a>
















<a href="#comment ">--&gt;</a>

<a href="#comment ">[Author:]</a>







wvZopeFilter/wvZopeFilter.py0100644000076500001440000005167707552343027015407 0ustar  abelusers
""" wvZopeGilter -- filters layout-specific and redundant HTML data 
    written by wvWare and unrtf

    Copyright (C) 2002 Abel Deuring 

    This program is free software; you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by the
    Free Software Foundation; either version 2, or (at your option) any
    later version.
     
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.
     
    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
    USA.
"""

import sgmllib
import sys, os
import re


WMF2GD = '/usr/local/bin/wmf2gd --maxwidth=%s --maxheight=%s --maxsize -t png %s.wmf > %s.png'

_character_entities=(('&', '&'),
                     ("<", '<' ),
                     (">", '>' ),
                     ('"', '"'))

_debugLevel = 0
_warnLevel = 1
_testLevel = 50
def WARN(s):
    if _debugLevel >= _warnLevel:
        sys.stderr.write(s)

def TEST(s):
    if _debugLevel >= _testLevel:
        sys.stderr.write(s)


class elem:
    """ abstract base class for data and elements """
    def redundant(self):
        """ returns:
              0 -> not redundant
              1 -> redundant
              other: value which can replace the current tag.
        """
        return 0
    
    def removeRedundant(self):
        return 0
    
    def isspace(self):
        return 0
    
    def mergeSameTags(self, inPreTag=0):
        return 0
    
class text(elem):
    """ stores regular HTML text """
    
    _isspace = re.compile('^\s*$')
    
    def __init__(self, t):
        self.data = t
    
    def writeContents(self, file):
        res = self.data
        for (old, new) in _character_entities:
            res = res.replace(old, new)
        file.write(res)
    
    def isspace(self):
        return self._isspace.match(self.data)

class text_notransform(elem):
    """ stores the a href tag """
    def __init__(self, t):
        self.data = t
    
    def writeContents(self, file):
        file.write(self.data)

class ahref_text(text_notransform):
    pass

class container(elem):
    """ abstract class for HTML tags which may contain sub-tags """
    def __init__(self):
        self.contents = []
        self.dangling = {}
    
    def add(self, elem):
        self.contents.append(elem)
    
    def writeContents(self, file):
        for e in self.contents:
            e.writeContents(file)
    
    def redundant(self):
        """ this tag can be omitted, if it has no sub tags 
            Or it can be replaced with a whitespace data elemet,
            if the contents consists only of whitespaces
        """
        if len(self.contents) == 0:
            return 1
        else:
            for e in self.contents:
                if not e.isspace():
                    return 0
            TEST("redundant %s >%s<\n" % (self.tag, e))
            return e
    
    def removeRedundant(self):
        removed = 1
        total = 0
        test = getattr(self, "inRemove", 0)
        # this should not happen:
        if test: raise "recursion detected! %s" % self.tag
        self.inRemove = 1
        while removed:
            removed = 0
            i = len(self.contents) - 1
            while i >= 0:
                removed += self.contents[i].removeRedundant()
                test = self.contents[i].redundant()
                if test == 1:
                    self.contents.pop(i)
                    removed += 1
                elif test != 0: 
                    """ element has signaled that it can be replaced by
                        its contents
                    """
                    TEST("replaced elem %s with its contents\n" % self.contents[i].tag)
                    self.contents[i] = test
                i -= 1
            total += removed
        self.inRemove = 0
        return total
    
    _mayMerge = ('i', 'b', 'font', 'pre')
    def mergeSameTags(self, inPreTag=0):
        # first merge things like  ...  ... 
        total = 0
        i = len(self.contents) - 1
        if self.tag == 'pre':
            inPreTag = 1
        while i > 0:
            e2 = self.contents[i]
            e1 = self.contents[i-1]
            if     e1.__class__.__name__ == 'startStopTag' \
               and e2.__class__.__name__ == 'startStopTag' \
               and e1.tag in self._mayMerge                \
               and e1.tag == e2.tag                        \
               and e1.attr == e2.attr:
                e1.contents += e2.contents
                self.contents.pop(i)
                total += 1
            elif     e1.__class__.__name__ == 'text' \
                 and e2.__class__.__name__ == 'text':
                e1.data += e2.data
                self.contents.pop(i)
                # don't replace inside 
...
 
                if (not inPreTag) and e1.isspace(): 
                    e1.data = ' ' 
                total += 1
            i -= 1
        for e in self.contents:
            total += e.mergeSameTags(inPreTag)
        return total
            
    def removeWordStyleBoldItalic(self, removeBold=0, removeItalic=0):
        """ If a word style is by default bold or italic
            (i.e., if an input  
 tag has the attributes bold="1"
            or italic="1"), replace any "inner"  resp.  tags
            with their contents
            
            This method is currently only useful, if a patched version
            wvWare is used. Does not work with unrtf.
        """
        if self.tag == "p":
            newAttr = []
            for k,v in self.attr:
                if k == "bold":
                    if v == "1":
                        removeBold = 1
                elif k == "italic":
                    if v == "1":
                        removeItalic = 1
                else: newAttr.append((k,v))
            self.attr = newAttr
        i = len(self.contents) - 1
        while i >= 0:
            e = self.contents[i]
            if hasattr(e, 'contents'):
                e.removeWordStyleBoldItalic(removeBold, removeItalic)
                if    (removeBold and e.tag == 'b') \
                   or (removeItalic and e.tag == 'i'):
                    self.contents = self.contents[0:i] + e.contents \
                                  + self.contents[i+1:]
            i -= 1
            
    
    def convertImages(self):
        """ try to convert all wmf files to png """
        for e in self.contents:
            if getattr(e, 'tag', None) == 'img':
                attr = {}
                for (k, v) in e.attr:
                    attr[k] = v
                if attr['src'][-4:] == '.wmf':
                    fname = attr['src'][:-4]
                    cmd = WMF2GD % (attr['width'], attr['height'], fname, fname)
                    os.system(cmd)
                    attr['src'] = attr['src'][:-4] + '.png'
                    e.attr = []
                    for k in attr.keys():
                        e.attr.append((k, attr[k]))
            else:
                if hasattr(e, 'convertImages'):
                    e.convertImages()
    
class simpleTag(elem):
    """ for 
,  etc """
    def __init__(self, tag, attr):
        self.tag = tag
        self.attr = attr
    
    def writeContents(self, file):
        file.write('<' + self.tag)
        for kv in self.attr:
            file.write(' %s="%s"' % kv)
        file.write('>')

class startStopTag(container, simpleTag):
    """ for ... 
    """
    def __init__(self, tag, attr):
        simpleTag.__init__(self, tag, attr)
        container.__init__(self)
    
    def writeContents(self, file):
        simpleTag.writeContents(self, file)
        container.writeContents(self, file)
        file.write('' % self.tag)
    
# listTag knows how to deal with qrong-matched