FETCHSITE - A recursive web mirroring tool
==========================================

Fetchsite can be used like wget to mirror web-trees for static
viewing. Fetchsites primary design goal was, to create a tool to
generate static versions of dynamic Zope websites. As such, fetchsite
is considered to be a Zope developers tool.

Fetchsite has a number of special features, which distinguish it from
comparable tools. Fetchsite does

  1. recursive fetch a complete site. Save all documents to normalized
     paths

  2. intelligently process directory default views (aka index.html)

  3. interpret CSS and JavaScript code

  4. corretly process the <BASE> tag and HTML redirects

  5. support file redirection

  6. fetchsite is XHTML-clean

Fetchsite also has a number of Shortcomings, which make it suboptimal
as a general wget replacement:

  1. It only works on valid HTML pages. fetchsite will break,
     sometimes horribly, if confronted with incorrect HTML

  2. Fetchsite is specialized for Usage with Zope: it does recognize
     index_html as index document instead of index.<whatever>

  3. It chokes on xml directives (<? ... ?> constructs)

  4. It has more Bugs than wget :-)

Several features are on the TODO list:

  o parse bobo_exception headers

  o remove duplicate files on redirects using the rewrite engine

  o better JavaScript parsing

  o somehow handle xml directives

  o better error handling

  o make filename normalization configurable


1. Pathname normalization
-------------------------

Fetchsite stores all downloaded dokuments in a directory hierarchy
mirroring the structure of the website. All destination paths are
normalized in the following manner:

  o If a file redirection applies to the source URL, it is applied

  o If no file redirection applies, every path component is processed
    in the following way:
   
     o The component is folded to lowercase and truncated to 8.3
       characters

     o The component is uniquified by appending letters or changing
       the last letters of the name until it is unique

     o The file extension is adjusted according to the content-type of
       the document


2. Directory default views
--------------------------

If during pathname normalization a name collision document <->
directory is encountered, we assume, that the document is the
directories default view and save the document as index document in
the directory. For example, when downloading

    http://no.where.org/this/is/a/test
    http://no.where.org/this/is/a/test/again

fetchsite will find a document <-> directory collision at
'this/is/a/test' and will therefore store:

    http://no.where.org/this/is/a/test       -> this/is/a/test/index.html
    http://no.where.org/this/is/a/test/again -> this/is/a/test/again.html

(assuming, both are html documents)

3. CSS and JavaScript paring
----------------------------

fetchsite employs very somple CSS and JavaScript parsing using
regular expressions. In CSS this is not realy a problem, since CSS
very seldom contains regions of plain text. In JavaScript, this
approach might uncorrectly scramble JavaScript code.

Therefore, JavaScript statements are *only* processed in <script>
blocks and event tags, *not* in JavaScript files. CSS code on the
other hand is interpreted everywhere including style attributes,
<style> blocks and external style files.

In CSS, the construct

   url(<url>)

is interpreted. The <url> is added to the list of URL's to download
and changed to point to the correct downloaded target file.

JavaScript code does not have any clear URL syntax per se. Therefore,
fetchsite introduces a few oft used commands and interprets those. It
is possible to add additional commands using the --parse-js command
line argument.

  preloadImages('<url>','<url>',...)
    This command presumably will preload images used for mouse over
    effects or such. fetchiste will add all <url>'s to its download
    list and correctly adjust the code, to point to the local files.

  gotoUrl(frame,'<url>')
    This command should open <url> in the given frame. The <url> is
    added to the download list and the code adjusted to point to the
    downloaded file.

  IsURL('<url>')
    This command should be a no-op and is used, to mark any string in
    JavaScript code as a URL to be processed. This URL will *not* be
    added to the download list, but it will be adjusted to point to
    the correct local file *or* directory if possible. This can also
    be used on *incomplete* URL's, e.g. the path of the images
    Folder.

4. Redirects and the <BASE> tag
-------------------------------

fetchsite will follow redirects and interpret the <BASE> tag. When a
redirect occurs, the file is processed, as if it had been found using
the original URL, *not* the redirected URL. This is necessary to allow
using the path corresponding to the original URL on the downloaded
directory hierarchy. If for example http://no.such.org/ redirects to
http://no.such.org/the/main/page, the file will still be stored under
'/index.html'. If this were not the case, the homepage would only be
accessible using 'the/main/page.html' in the mirrored directory
tree.

This behaviour can lead to duplicate document download, if the same
file is accessed directly. This is necessary, to keep all URL's
working.

<BASE> tags are correctly interpreted and removed from the saved
documents.

5. File redirection
-------------------

This feature allows to specify any target path for a given URL. File
redirection can be used to control the directory layout of the mirror
tree.

Example: You have 3 seperate areas in your site and want to fetch
theese areas using seperate fetchsite commands. This will enable you
to selectivly update only one area at a time.

Your main layout uses a number of images and css files. Theese are of
course identical for the three areas. If you simply download the 3
areas seperatly you will end up with three copies of the image
files. Using file redirection, you can save thees images in a shared
image folder:

  fetchsite.py --cut-dirs 1 --dest area1 --redirect images:images \
      http://test.domain/area1
  fetchsite.py --cut-dirs 1 --dest area2 --redirect images:images \
      http://test.domain/area2
  fetchsite.py --cut-dirs 1 --dest area3 --redirect images:images \
      http://test.domain/area3

  
Usage:
------

    fetchsite.py [ --cut-dirs | -c n ] [ --dest | -d dir ] [ --help | -h ] 
                 [ --parse-js | -S cmd:arg:recurse ]... [ --quiet | -q ] 
                 [ --redirect | -R src:dst ]... url

Recursively fetch a complete site from the given <url>. 'fetchsite' will parse
HTML documents to recursively find all URL's. The files are than stored in a
local mirror tree. Within the files, all URL references are changed to point
to the local files.

In addition to ordinary URL references in HTML files (anchors, links, script
or CSS references) 'fetchsite' also interprets CSS and JavaScript:

   CSS:style="... :url(<url>)..."
       This adds <url> to the download list. The url will be changed to point
       to the local file

   JavaScript:preloadImages('<img>','<img>', ..._)
       This adds all the images referenced to the download list and changes the
       image references accordingly

   JavaScript:gotoUrl(frame,'<url>')
       This adds <url> to the download list and again changes the url reference

   JavaScript:... IsURL('<url>') ...
       This construct will *not* add <url> to the download list *but* the url
       will be adjusted to point to the local file or directory referenced if
       possible.

To use above JavaScript extensions, the following functions need to be defined
*before* their first use in every page:

    function preloadImages()
    {
        if (!document.preloadedImages) document.preloadedImages = new Array();
        var i,j; j = document.preloadedImages.length;
        for (i=0; i<preloadImages.arguments.length; ++i) {
    	    document.preloadedImages[j] = new Image();
    	    document.preloadedImages[j++].src = preloadImages.arguments[i];
        }
    }

    function gotoUrl(frame,url) { frame.location.href = url }

    function IsURL(url) { return url; }

All JavaScript commands are recognized in either <script> blocks or event
handler attributes but *not* in JavaScript files. All CSS constructs are
recognized in CSS files (marked with content-type 'text/css'), <style> blocks
or 'style=' attributes.

'fetchsite' will first download all files to the temporary directory
'.fetchsite.temp' in the current directory. After all pages have been
downloaded, the pages will be processed and moved to their final destination.

LIMITATIONS

There may be several I am not aware of. At this time, I can think of only one
Limitation, which will be removed in a future Version:

  o the query string (the part after the first ? in the URL) must not contain
    unescaped slash characters

Options:
    --cut-dirs n                number of leading directories to remoave. a
    -c n                        file with the url 'http://host/path/to/object'
                                will have the first <n> directory names removed
                                from '/path/to/object' to find the target
                                path before normalization (i.e. case-folding,
                                truncation and uniquification). If the url has
                                fewer than <n> directories, the target path is
                                'object'

    --dest dir                  store in subdirectory <dir> of current
    -d dir                      ectory. This will prepend all normalized target
                                paths with <dir> *except* the <dst> arguments
                                of all --redirect specification

    --help                      Show help information
    -h                          

    --parse-js cmd:arg:recurse  add new JavaScript command <cmd> interpreting
    -S cmd:arg:recurse          the <arg>'th argument as a URL, or all
                                arguments, if <arg> == 0. if <recurse> is 1,
                                add the url to the download list if 0, only
                                change the url reference. So the predefined
                                JavaScript commands are declared with
                                
                                   --parse-js preloadImages:0:1
                                   --parse-js gotoUrl:2:1
                                   --parse-js IsURL:1:0

    --quiet                     be quiet
    -q                          

    --redirect src:dst          save the object found under path <src> at
    -R src:dst                  <dst>. <src> is the complete pathname of the
                                url used to download the object. If the url
                                shall refer to a directories default view
                                append '/index' to the url. <dst> is the
                                *normalized* target path to use for storing
                                <src>. If <src> refers to a directory this will
                                affect *all* files with urls below this
                                directory.