Re: st: file size problems

[Jeph Herrin <junk@spandrel.net>]

This is very helpful - guess I've been -use-ing data for so many years,
it didn't occur to me to check -help use-.  My oversight.

Thanks,
Jeph




Scott Merryman wrote:
> Regarding the third question:  rather than subset the data after it is in
> memory try loading only the relevant subset, 
>  -use first.dta if month<mdy(1,1,2000)-
>
> For example:
>
> . set memory 1m
>
> Current memory allocation
>
>                     current                                 memory usage
>     settable          value     description                 (1M = 1024k)
>     --------------------------------------------------------------------
>     set maxvar         5000     max. variables allowed           1.733M
>     set memory            1M    max. data space                  1.000M
>     set matsize         400     max. RHS vars in models          1.254M
>                                                             -----------
>                                                                  3.987M
>
> . sysuse auto
> (1978 Automobile Data)
>
> . expand 275
> (20276 observations created)
>
> . keep if fore ==1
> no room to add more variables due to width
>     An attempt was made to add a variable that would have increased the
> memory required to store an observation beyond what is currently possible.
> You have the following alternatives:
>
>      1.  Store existing variables more efficiently; see help compress.
>
>      2.  Drop some variables or observations; see help drop.  (Think of
> Stata's data area as the area of a rectangle; Stata can trade off width and
>          length.)
>
>      3.  Increase the amount of memory allocated to the data area using the
> set memory command; see help memory.
> r(902);
>
> . save "C:\Documents and Settings\Scott_2\Desktop\temp.dta", replace
> file C:\Documents and Settings\Scott_2\Desktop\temp.dta saved
>
> . clear
>
> . use "C:\Documents and Settings\Scott_2\Desktop\temp.dta" if fore == 1
> (1978 Automobile Data)
>
>
> Scott
>
>
>
> > -----Original Message-----
> > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-
> > statalist@hsphsun2.harvard.edu] On Behalf Of Jeph Herrin
> > Sent: Sunday, August 20, 2006 10:50 AM
> > To: statalist@hsphsun2.harvard.edu
> > Subject: Re: st: file size problems
> >
> > Sorry, perhaps I should have been more clear. I have 3 distinct questions:
> >
> > 1) Why does a 500mb file require 800mb of Stata memory?
> > 2) How to get around "op. sys. refuses to supply memory" error
> >     when using -set mem 1000m- in winxp (when xp should be able
> >     to access at least 2.5gb, and no other apps are running)?
> > 3) How to drop observations once the file is in memory, when
> >     -keep if mycondition- gives a "no room to add more obs" error
> >     (and there is no way to increase memory)?
> >
> > thanks,
> > Jeph
>
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/