DMI Tools firefox extension

The DMI toolbar is a Firefox extension that provides extra functionality to the DMI tools. Currently it provides off-loading of HTTP requests to the client browser. This means that the requests do not happen through the DMI server but from the computer using the Firefox extension.

Important notes

This extension and the HTTP off-loading are an experimental way of performing digital methods research. Use caution when contemplating the results. To avoid researching personalized results, make sure to follow the 'research browser' guidelines below before each scrape.

Installation

  • Install and start Firefox
  • Within Firefox, click on and install the DMI Firefox extension and restart Firefox. You also can right-click and 'save as' to desktop, and drag and drop the file, dmitools.xpi, into Firefox.
  • Make sure to allow pop-ups from tools.digitalmethods.net

The research browser

Depending on your research needs, use a research browser to avoid bias by stored cookies, personalisation settings and other factors that may change the results returned.

The DMI toolbar works in combination with the Google Scraper or Lippmannian Device, amongst other tools. Here we explain how the researcher can be disentangled from Google.

When using the Google scraper or Lippmannian Device with our Firefox toolbar, the researcher needs to take a few steps to ensure that day to day activities do not interfere with research.

FAQ

Why is there a need for a DMI toolbar?

Traditionally, all our scrapers would connect to websites from DMI's servers. As some services such as Google do not like automated requests, they would temporarily block all requests coming from the IP-address performing thoses requests (i.e. DMI's servers). Before Google blocks an IP-address for a longer time, it will offer a CAPTCHA to the user in order to verify that the requests are made by a human. This is not something our servers could confirm.

By offloading the requests to the researcher's browser 1) less requests originate from the same IP, and 2) if Google finds there are too many requests, the CAPTCHA can be filled in by the researcher.

Can I get blocked?

We have tested the toolbar for many requests and were confronted with multiple CAPTCHAs. Using this technique our IP-addresses were never blocked. Experience learns that Google only blocks IP-addresses temporarily.

Nothing happens after filling in the CAPTCHA returned by the Google Scraper.

Make sure to close the tab where you filled in the CAPTCHA

Tell me more about how Google personalizes my results.

Academic research

Please note that this plugin is provided as is, without any guarantee about its functionality or proper use. This plugin is only provided for use in academic research.
Topic revision: r14 - 07 May 2017, RichardRogers
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback