From: Stevan Harnad <[log in to unmask]>
Date: Fri, 23 Aug 2013 01:17:19 -0400

This is a response to a query regarding Eric Archambault's report on
OA Growth by Adam G Dunn in Science Insider: "I find it difficult to
believe that the authors of the study managed to create a harvester
that could identify and verify the pdfs linked to by Google Scholar
when Google Scholar actively blocks IP addresses when they identify
crawling."

Our own "harvester" attempts to gather the all-important data on OA
growth were blocked by Google.

It is completely understandable and justifiable that Google shields
its increasingly vital global database and search mechanisms from the
countless and incessant worldwide attempts at exploitation by
commercial interests, spammers, and malware that could bring Google to
its knees if not rigorously and relentlessly blocked.

But in the very special (and tiny) case of scientific research
articles it would not only be a great help to the worldwide research
community but to Google (and Google Scholar) itself if Google granted
special individual exemptions for important international studies like
Eric Archambault's, which was commissioned by the European Union to
monitor the global growth rate of open access to research.

Google and Google Scholar would become all the richer as research
databases if data like Eric's (and our own) were not made so
excruciatingly difficult and time-consuming to gather by Google's
blanket blockage of automated data-mining.

(We do not trawl books, so Google's agreements with publishers are not
violated or at issue in any way. We just want to trawl for articles
whose metadata match the the metadata from Web of Science or SCOPUS
and have been made freely accessible on the web; nor do we want their
full-texts: just to check whether they are there!)

Stevan Harnad