Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

RE: st: selecting obs while reading in huge data set


From   "Steve Stillman" <stillman@motu.org.nz>
To   <statalist@hsphsun2.harvard.edu>
Subject   RE: st: selecting obs while reading in huge data set
Date   Thu, 19 Aug 2004 20:15:53 +1200

Sascha,
I have a recommendation that I wouldn't usually make.  I have been
recently doing work with matched employer-employee data with over 30
million obs, so we have been running into the same problem as you.  SAS
is much better for large dataset merges than Stata.  In particular, proc
SQL is remarkably fast at doing these types of merges (likely because
SQL is written with this type of operation in mind).

Well there is was, likely the last time I will recommend SAS over Stata.

Cheers,
Steve

-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu]On Behalf Of Sascha O.
Becker
Sent: Thursday, August 19, 2004 6:59 PM
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: selecting obs while reading in huge data set


Dear Daniel,

thanks for your reply!

You suggested:

****
Perhaps you can read the employee and firm ID only?

    .insheet empid firmid using mydata

This is only 1/5th the variables, so it might fit in your computer
memory.
Then merge the result with the firm dataset, keeping only matched
records, then merge again with employee dataset, keeping only matched
records.
****

This last step is actually identical to the original problem. "The
employee dataset" is the full dataset with all variables. In order to
merge this to anything, it needs to be in memory at least once, and this

is exactly the problem.

There seems to be no way round some kind of looping, either over
observations, or over subsets of variables that I would merge against
the firm data set and then append/merge those sub-datasets.

Cheers, Sascha
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index