You are here: Foswiki>Dmi Web>DmiTools (29 Oct 2012, ErikBorra)Edit Attach

Digital Methods: Tools and Utilities

The various scrapers, crawlers, etc. listed below may be useful for different digital methods. For convenience, the list is loosely divided into tools and utilities, with the former denoting more complex operations that are either method- or device-specific, and the latter referring to relatively simple actions with multiple applications. Please note that of the prescribed uses for these tools and utilities, none are required.


Issue Crawler and Allied Tool Set

  • Issue Crawler - Enter URLs and the Issue Crawler performs co-link analysis in 1-3 iterations, and outputs a cluster graph. See Issue Crawler instructions of use, scenarios of use and FAQ.
  • Extract URLs - Extracts URLs from an Issuecrawler .xml file. Useful for retrieving starting points.
  • Actor Profiler - Calculates the top 10 nodes of an Issue Crawler network map, generates graphics of each node, showing types of inlinks received and outlinks given. It also queries Google for the issue you specify, and shows, on the graphic, the PageRank for those top 10 nodes. The colors for the actor profile are taken from the color scheme on the Issue Crawler map. >howto<
  • pict_geo.gifIssue Geographer - Geo-locates the organizations on an Issue Crawler map, using whois information, and visualizes the organizations' registered locations on a geographical map.
  • Compare networks over time - Compares Issue Crawler networks over time, and displays ranked actor lists. The over time module is best used in tandem with the Issue Crawler scheduler. The results may be plotted to line graphs, in software such as Excel.
  • Google Network Cloud - Extracts the URLs from an Issue Crawler result, and allows the user to query all the URLs for key words. It outputs a tag cloud. Good for the analysis of the content of a network.
  • Triangulation tool - Enter two or more lists of URLs or other items, and the triangulation tool finds commonalities among them.
  • Harvester - Enter text that contains URLs (e.g., copy-and-pasted Google results, or Webpage source code) and the harvester extracts the URLs, and creates a clean URL list.

PageRank Tools

  • Pagerank - Discover a website's Google Pagerank per issue/query.
  • Issue Dramaturg - Scheduled list of websites' Pagerank per issue.

Google Scrapers

  • googlescraper-1.pngGoogle Scraper - Query Google for a particular keyword/issue, possibly in particular sites.
  • Google Scraper Frequency Tool - Use a results file from the Google Scraper and discover the frequency with which the original query appears in each individual host. (Tip: visualize these results in a tag cloud.)
  • Google Teaser Text Ripper - Use a results file from the Google Scraper to get unique phrases from the teaser (or lead) text for each google search return.
  • YouTube video discovery - Use a results file from the Google Scraper to discover, count, and rank YouTube, Ikbis, and Google Video links in the descriptions. Obtain such a result file by querying a set of sites for e.g. '', '', 'videoplay?'.
  • Split Results - Split the results from Google Scraper into two lists (e.g. blocked and unblocked sites).
  • Google News Scraper - Query* with one or more keywords. It's only possible to scrape articles of the last 30 days.
  • Google Images Scraper - Query with one or more keywords, and/or use to query specific sites for images.
  • Scrape Google Blogsearch (Open Kapow) - Scrapes titles and URLs for a query.

Technorati Scrapers

Other Tools and Scrapers

  • Open Calais - Discovers the most relevant words and phrases among a set of websites, within a text, or within an issue network (i.e. an Issue Crawler .xml file). This tool is based on Reuters Open Calais.
  • Issue Discovery - Discovers the most relevant words and phrases among a set of websites, within a text, or within an issue network (i.e. an Issue Crawler .xml file). >howto<
  • Significance Measure - Enter a set of keywords. They will be scraped in various search engines and their specificity to eachother will be returned.
  • RSS discovery - Discovers RSS/ATOM/RDF feeds in websites.

  • YouTube Response Retworks - enter a <nop>YouTube movie and get all the responses to that video.
  • Surfer Issue Pathways - Building upon Alexa's related sites feature, this tool determines which sites are likely to be in the actual surfer paths of other sites related to the same issue.
  • Wikipedia Network Analysis - Find a hyperlink network around a Wikipedia topic (see here for more information).
  • Related Tags Cloud Generator - Visualizes already scraped tag listings. (Create a tag cloud showing URLs and tags related to a specific issue or keyword.)
  • WeScrape (Meta-Tool) - A howto guide for building your own scraper.
  • Issue Feed - summarizes the content of a set of sites. Enter a list of Websites and give the list a name. finds the sites' feeds, analyzes them, and outputs a substantive summary of all the sites in key words, ranked according to specificity and frequency measures. It works for sites in English or Dutch.


External Tools and Wish List

Using the tools

Tools categorized

ToolDatabase (work in progress)

create new tag
, view all tags
Topic attachments
I Attachment Action Size Date Who Comment
e-cultuur.pngpng e-cultuur.png manage 12 K 02 Dec 2008 - 15:23 RichardRogers Dorling Map Example
googlescraper-1.pngpng googlescraper-1.png manage 21 K 02 Dec 2008 - 15:21 RichardRogers google scraper icon
googlescraper_with_stylesheets.gifgif googlescraper_with_stylesheets.gif manage 42 K 27 Aug 2007 - 15:44 UnknownUser  
pict_geo.gifgif pict_geo.gif manage 620 bytes 02 Dec 2008 - 15:26 RichardRogers Issue Geographer Icon
Topic revision: r36 - 29 Oct 2012, ErikBorra
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback