Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down on April 23, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
"Nick Cox" <n.j.cox@durham.ac.uk> |

To |
<statalist@hsphsun2.harvard.edu> |

Subject |
RE: st: RE: Large data set that won't open |

Date |
Thu, 24 Jun 2010 11:04:26 +0100 |

Sure. There is a difference between -ls-, the Stata command, and -ls- as an OS command or utility used outside Stata. In my experience, the routine use of -ls- usually signals a Unix mindset if not a Unix machine. Nick n.j.cox@durham.ac.uk Martin Weiss " I think that John signalled indirectly through an example using an -ls- that he is using some flavour of Unix." -ls- works on my Windows machine w/o problems, though... Nick Cox Also to the point: 1. If the dataset is too big for Stata, it will hardly fit into Excel (unless very recent versions of Excel have suddenly acquired greatly increased capacities). 2. I think that John signalled indirectly through an example using an -ls- that he is using some flavour of Unix. In that case, he is unlikely to have access to Excel any way. (Either way, everyone being clear about what OS they are using assists communication in this area.) Nick n.j.cox@durham.ac.uk Martin Weiss But why bother to use spreadsheets here, given that Stata can, as I mentioned earlier, -use- part of a file? Apart from -if- and -in-, a -varlist- can also be supplied, so the "split on variables" can be achieved within Stata. Kay Walker Possibly if you can't get into Stata to edit the data file to make it smaller, drop it into Excel, split the file on variables not cases,(and copy the identifiers to the "new" half) so that you can analyse only those variables and cases you NEED in a particular context. You can always swap relevant variables in and out of the working data file if you want to perform further analyses later. I think it's the fact that the RAM has to hold the program and the relevant part of the datafile in memory for analysis that is causing the problem, not merely the size of the datafile vs the program. Martin Weiss wrote: > The size of your RAM is hard to tell from your current memory allocation - > the latter is just a lower bound for the former. But I guess you do not have > 6G of RAM, do you? You would still need additional RAM for computations, > then. > > Note you can -use- a dataset with -if- and -in- qualifiers, so it is still > possible to get a subset into mem... John Antonakis > I have a rather large data set that I cannot open in Stata; here are the > particulars: > > ls "D:\My Documents\STATA\upstata.dta", > 5375.8M 6/23/10 10:44 upstata.dta > > When I try to open the data set I get: > > . use "D:\My Documents\STATA\upstata.dta", clear > no room to add more observations > An attempt was made to increase the number of observations beyond > what is currently > possible. You have the following alternatives: > > 1. Store your variables more efficiently; see help compress. > (Think of Stata's data > area as the area of a rectangle; Stata can trade off width and > length.) > > 2. Drop some variables or observations; see help drop. > > 3. Increase the amount of memory allocated to the data area using > the set memory > command; see help memory. > r(901); > > Here is my current memory allocation: > > . q memory > > Current memory allocation > > current memory usage > settable value description (1M = 1024k) > -------------------------------------------------------------------- > set maxvar 5000 max. variables allowed 1.909M > set memory 1024M max. data space 1,024.000M > set matsize 1000 max. RHS vars in models 7.713M > ----------- > 1,033.622M > > > It seems that I just don't have enough RAM to open this file. Is that > correct? * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/

**Follow-Ups**:**Re: st: RE: Large data set that won't open***From:*Neil Shephard <nshephard@gmail.com>

**References**:**st: Large data set that won't open***From:*John Antonakis <john.antonakis@unil.ch>

**st: RE: Large data set that won't open***From:*"Martin Weiss" <martin.weiss1@gmx.de>

**Re: st: RE: Large data set that won't open***From:*Kay Walker <kay.walker@internode.on.net>

**RE: st: RE: Large data set that won't open***From:*"Martin Weiss" <martin.weiss1@gmx.de>

**RE: st: RE: Large data set that won't open***From:*"Nick Cox" <n.j.cox@durham.ac.uk>

**RE: st: RE: Large data set that won't open***From:*"Martin Weiss" <martin.weiss1@gmx.de>

- Prev by Date:
**st: RE: Simple panel data mean plot** - Next by Date:
**Re: st: RE: Large data set that won't open** - Previous by thread:
**RE: st: RE: Large data set that won't open** - Next by thread:
**Re: st: RE: Large data set that won't open** - Index(es):