Item19: Optimize harvester to remove duplicates

The harvester currently does not distinguish between urls or urls with a trailing slash. It also does not distinguish between www.host or host.

Solution:

• strip http:// and slash at end. Remove www too, compare what's left. Put www back if it existed. Put http:// in front

-- Erik Borra - 28 Feb 2008

Decided to not strip www, did the rest.

-- Erik Borra - 09 Apr 2008