Re: st: Working with really large Datasets

[Fernando Rios Avila <f.rios.a@gmail.com>]

Richard and Justin,
Thank you very much for the suggestions.
Best regards

On Mon, Oct 15, 2012 at 8:09 PM, Richard Williams
<richardwilliams.ndu@gmail.com> wrote:
> At 05:52 PM 10/15/2012, Fernando Rios Avila wrote:
>>
>> Dear stata listers,
>> I wonder if any one here can share some experience on working with
>> really large datasets. I m working with a panel dataset (census type
>> of data) for workers and firms over time. The total number of
>> observations is about 70 million. I want to estimate  two way fixed
>> effects models, manually including dummies for regions time and
>> industries. However with the size of the dataset, the results become
>> unmanageable.
>> Does anyone know or can direct me to an strategy to deal with "too much
>> data"?
>> I was thinking about obtaining random samples (say 5%), picking
>> individuals at random, and keeping them along the whole time they
>> appear on the sample, and then combining all the results, in a similar
>> fashion as it is done with Multiple Imputation datasets. But im not
>> sure how valid would that procedure be.
>> Any suggestions are welcome,
>> Thank you.
>
>
> This FAQ by Cox & Merryman might give you some ideas.
>
> http://www.stata.com/support/faqs/data-management/sampling-clusters/
>
>
> -------------------------------------------
> Richard Williams, Notre Dame Dept of Sociology
> OFFICE: (574)631-6668, (574)631-6463
> HOME:   (574)289-5227
> EMAIL:  Richard.A.Williams.5@ND.Edu
> WWW:    http://www.nd.edu/~rwilliam
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/