DMI Tools firefox extension

The DMI toolbar is a Firefox extension that provides extra functionality to the DMI tools. Currently it provides off-loading of HTTP requests to the client browser. This means that the requests do not happen through the DMI server but from the computer using the Firefox extension.

Important notes

This extension and the HTTP off-loading are an experimental way of performing digital methods research. Use caution when contemplating the results. To avoid researching personalized results, make sure to follow the 'research browser' guidelines below before each scrape.

Newer versions of Firefox, starting with Firefox 57 (Quantum) require you to re-install the extension. Automatic updates do not work.

Installation

• Install and start Firefox
• Within Firefox, click on and install the DMI Firefox extension and restart Firefox. You also can right-click and 'save as' to desktop, and drag and drop the file, dmitools.xpi, into Firefox.
• Make sure to allow pop-ups from tools.digitalmethods.net

The research browser

Depending on your research needs, use a research browser to avoid bias by stored cookies, personalisation settings and other factors that may change the results returned.

The DMI toolbar works in combination with the Google Scraper or Lippmannian Device, amongst other tools. Here we explain how the researcher can be disentangled from Google.

When using the Google scraper or Lippmannian Device with our Firefox toolbar, the researcher needs to take a few steps to ensure that day to day activities do not interfere with research.

• Consider installing a separate version of Firefox, a so called research browser, used solely for research purposes.
  • Alternatively use a specially created Firefox profile or install a separate version of Firefox on a USB stick.
  • See our video on setting up a research browser.
• In the research browser, make sure to log out of any Google services. See our video on setting up Google for research
• Even when logged out, Google personalizes search results based on a cookie in your browser. After logging out, go to https://www.google.com/history/optout and make sure 'Customization based on signed-out search activity is disabled.' Doing this
  • removes personalization on web-history
  • removes personalization on (social) network
  • retains localization on geography (e.g. .nl)

FAQ

Why is there a need for a DMI toolbar?

Traditionally, all our scrapers would connect to websites from DMI's servers. As some services such as Google do not like automated requests, they would temporarily block all requests coming from the IP-address performing thoses requests (i.e. DMI's servers). Before Google blocks an IP-address for a longer time, it will offer a CAPTCHA to the user in order to verify that the requests are made by a human. This is not something our servers could confirm.

By offloading the requests to the researcher's browser 1) less requests originate from the same IP, and 2) if Google finds there are too many requests, the CAPTCHA can be filled in by the researcher.

Can I get blocked?

We have tested the toolbar for many requests and were confronted with multiple CAPTCHAs. Using this technique our IP-addresses were never blocked. Experience learns that Google only blocks IP-addresses temporarily.

Nothing happens after filling in the CAPTCHA returned by the Google Scraper.

Make sure to close the tab where you filled in the CAPTCHA

Tell me more about how Google personalizes my results.

• Things Internet Researchers Should Know About Google
• Turn off search history personalization
• 180 days of search activity are linked to an anonymous cookie, even when logged out.
• Martin Feuz empirically tested Google's personalization in 2009/2010. His article can be found online at Personal Web searching in the age of semantic capitalism: Diagnosing the mechanisms of personalization.

Academic research

Please note that this plugin is provided as is, without any guarantee about its functionality or proper use. This plugin is only provided for use in academic research.