Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Reading large data sets in Stata


From   Michael Norman Mitchell <Michael.Norman.Mitchell@gmail.com>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Reading large data sets in Stata
Date   Mon, 22 Feb 2010 12:43:03 -0800

To add to Kit's excellent answer, I would add the link to the "interactive dataset calculator" he referred to...

http://www.stata.com/support/faqs/data/datacalc.cgi

That might have appeared in a different posting that I missed.

Michael N. Mitchell
See the Stata tidbit of the week at...
http://www.MichaelNormanMitchell.com
Visit me on Facebook at...
http://www.facebook.com/MichaelNormanMitchell

On 2010-02-22 12.19 PM, Christopher Baum wrote:
<>
Stas said

That's 13Gb of data, right? If you really want to put everything into memory, then you would probably need a computer with 24Gb of RAM. I don't really know if you can buy anything like that in the desktop format, and what kind of OS you would need to look at, although I am sure there are clusters with much larger memory capacities. If you only need subsets of that data set, then you could use <list of the variables that you REALLY need> if <subsetting to the conditions you REALLY want to analyze> using <this huge data set name> That way, you may have a data set of a more realistic 2Gb size that you can work with on a 4Gb RAM machine.

That's not necessarily 13 Gb of data. Using the interactive calculator on the FAQ, if you assume all 37 variables can be held in 4 bytes each, it's under 7 Gb. If on average they only need 3 bytes each, it's under 6 Gb. Stat/Transfer can optimize the dataset as it converts it to Stata format. Stas' suggestions are well taken, but one more is important--if any of these variables are 0/1 indicators, or integers taking on values 1..5, etc. they need not chew up nearly as much memory. I don't know if you can get it down to a 2 Gb size, though. To use more than 2 Gb, you need a 64-bit machine (almost all machines are these days), and Stata 11 will automatically install the 64-bit version on such a machine.

Kit Baum | Boston College Economics and DIW Berlin | http://ideas.repec.org/e/pba1.html An Introduction to Stata Programming | http://www.stata-press.com/books/isp.html An Introduction to Modern Econometrics Using Stata | http://www.stata-press.com/books/imeus.html

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index