[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]
Re: st: collecting raw data from the web via browser automation
On May 22, 2006, at 10:54 PM, Michael Blasnik wrote:
I'm not sure if any of these tools can actually solve the problem
Yes, they can. Both curl and wget support authentication, cookies,
SSL, and the use of HTTP POST (in addition to GET) to submit a
request. And with either Python or Perl, you can script an entire
web session, including passing through multiple forms, with each
subsequent request dependent on the result(s) returned from the last.
As a later post indicated, you can use Stata's -copy- to retrieve a
page using GET (i.e., parameters encoded in the actual URL), and in
this way initiate a search with Google Scholar. However, the URL in
the original posting resulted from clicking on the "Import into..."
link corresponding to a single item from the list of items returned
by a search. I'm not sure how this selection would be made
programmatically, or, if the intention was to grab the information on
all of the top n items (note that depending upon how large n is, this
might be spread across multiple results pages, due to the way results
are batched). Moreover, the format of the data returned by the
original URL depends upon how your "Scholar Preferences" are set
(i.e., which bibliographic format), and these preferences are
probably stored in a cookie. Finally, regardless of the export
format chosen, you may still need to do some post-processing before
reading the "data" into Stata. Thus, even though the initial search
can be triggered with -copy-, one of the other suggested tools may
well be necessary to complete the entire task (or at least to do so
in an efficient way).
On May 22, 2006, at 4:21 PM, Austin Nichols wrote:
Google Scholar has a nice way to set Preferences so that links to
bibliographic info are generated in the search results, but I don't
use BibTeX or EndNote or any of those things--I use Stata, and I
want to automate the whole process of seaching and saving those
data (which look like http://scholar.google.com/scholar.bib?
q=info:nmXVGJVxYjQJ:scholar.google.com/&output=citation by the way)
and infiling them into Stata so I can have a nice database of
articles made for me on any set of search terms I put in.
I meant to comment on this before, but forgot. As much as I love to
see new uses for Stata, I would strongly urge you to look at one of
the free programs available for managing BibTex files (e.g.,
tkbibtex, BibTool, or Bibcursed (multi-platform), BibDesk (my
personal favorite; OS X only), or BibEdit or BibDB (Windows only)).
Also, many text editors provide tools for working with files in
BibTeX format. As you know, you can export directly into BibTeX
format from Scholar. Even if you don't actually use BibTeX when
writing, these tools may permit you to accomplish what you need, and
may suggest other things you hadn't thought of.
* For searches and help try: