Zope is the most robot friendly dynamic web publishing environment. There are a couple of robot caveats. Robots usually refuse to index pages that contain 'cgi-bin', '?' and other indication of dynamic pages because they don't want to suck down an entire database or flood a dynamic page with infinite requests. Since ZSQLMethods allow direct traversal to database recrods, we may wish to prevent a robot from indexing these objects. This can be done using a robots.txt file.   # http://www.yoursite.com/robots.txt must be located at your site root.
  # Mail [email protected] with constructive criticism.
  User-agent: *
  Disallow: /path/to/database/objs
Another way to prevent robots from indexing is to include robot indexing information in a page's meta data.   <meta name="robots" content="noindex"> # prevents indexing of the page
  <meta name="robots" content="nofollow"> # prevents indexing links on this page
Relative URLs can also confuse robots or even cause the robot to 'infinite loop' on a relative URL and bring down your machine or flood your network. The same technique can be used to prevent indexing of relative URLs, but its probably better to specify absolute URLs. Avoid these:
  </a href="relative/index_html">Relative URL</a>
Promote these:
  </a href="http://yoursite.com/absolute/index_html">Absolute URL</a>
Finally, the specification for the robots.txt file is here.
http://info.webcrawler.com/mak/projects/robots/norobots.html