Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Machine spec for 70GB data


From   Joerg Luedicke <joerg.luedicke@gmail.com>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Machine spec for 70GB data
Date   Sat, 22 Oct 2011 09:25:40 -0400

What are "a few millions"? If by that you mean like a handful then you
must have a ton of variables. If you do not need all of them for your
analyses, you can read the data in in chunks, set up the variables you
need, and eventually put it together again. However, in my experience
it seems difficult to fit more complicated multilevel models in Stata
when sample size becomes large. I find this to be especially true in
the case of models with crossed random effects. So just beware, even
if you get all the data you want into memory, you may not be able to
run the model you propose.

J.

On Sat, Oct 22, 2011 at 7:00 AM, Gindo Tampubolon
<Gindo.Tampubolon@manchester.ac.uk> wrote:
> Dear all,
>
> I need to process a large data file [70GB; a few millions obs] with Stata 12 MP8. Mainly to do cross-random effects,individuals and hospitals, where the outcome is length of stay [controlling for no more than a handful of covariates to begin with]. As an approximation, the outcome is treated as continuous i.e. linear mixed models.
>
> What kind of machine spec would be needed? Any ideas, information, experience? Would operating system make any difference? I'm open to consider Windows, Linux, OS X.
>
> Many thanks,
> Gindo
> University of Manchester
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index