Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Re: Web APIs and JSON file format reading


From   Eric Booth <[email protected]>
To   "<[email protected]>" <[email protected]>
Subject   Re: st: Re: Web APIs and JSON file format reading
Date   Mon, 20 Dec 2010 03:11:53 +0000

<>


Hi Andrew:

I've written a couple of do/ado files to copy data from argument-based APIs (mostly Google APIs) to a txt file, then -infile-/-intext-(from SSC)/-insheet-  them into Stata, and then  parse them using a combination of string_functions (though nothing in Mata), but I gave up on the JSON parsing program I was writing because I found Google Refine and it's  ".parseJSON" class made quick work of getting the data I needed.  

See this example:
http://code.google.com/p/google-refine/wiki/FetchingURLsFromWebServices
(yahoo uses JSON too:  http://developer.yahoo.com/javascript/json.html)

My opinion is that it'd be useful to have a JSON (or feed-XML) parsing program for Stata, but as I've started a couple of these types of programs, my limited understanding of  JSON had me kludging together a site-specific program that would likely fail for other websites'/APIs with different query structures.   

To answer your questions:  I don't think it's been done, but I think it's possible (or at least it is via the [-copy-, -infile-, parse] approach I've taken  -- I have no idea about using Mata for something like this).  If you do get something written, I'd have an interest in using it -- I like to have all my workflow in one program rather than having to go to something external (like Google Refine) that breaks up the workflow.   The Worldbank API page explains their query structure very clearly, so it might be a good place to start : http://data.worldbank.org/node/11

A final tip:  you can now get most of Google's Data APIs in JSON instead of XML by using the 'alt=json' parameter , see this link for examples: http://code.google.com/apis/gdata/samples.html  


- Eric

__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
[email protected]
Office: +979.845.6754



On Dec 19, 2010, at 6:40 PM, Andrew Dyck wrote:

> Hi Stata users,
> 
> I'm interested in using Stata to access data from web APIs such as
> data.worldbank.org or Twiter. I use 64-bit Stata MP for Windows and
> Linux.
> 
> I'm aware of the -net-, -copy- and -webuse- commands, however, my
> searches so far haven't uncovered any tools for reading in data
> directly from an API. At this time it seems that this may not be
> possible in Stata without using an external python or perl script.
> However, if you have any experience in using JSON with Stata I'd be
> very happy to hear from you.
> 
> I have begun research into writing a native Stata module (using mata)
> for reading JSON data from a web API but I thought I'd tap the
> Statalist knowledge bank first to:
> 
> 1) stop me early if it's just not possible or already done;
> 2) suggest any strategies for how to go about this; and
> 3) see if there is any interest in such a feature or if users are
> already using some other methods that would work better than something
> native to Stata.
> 
> Any help, advice or links to more info would be greatly appreciated.
> 
> Thanks,
> Andrew
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/




*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index