Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: file too big to import


From   Eric Booth <ebooth@ppri.tamu.edu>
To   "<statalist@hsphsun2.harvard.edu>" <statalist@hsphsun2.harvard.edu>
Subject   Re: st: file too big to import
Date   Wed, 20 Apr 2011 12:00:07 +0000

<>

I guess that's assuming you don't have Small Stata (which is limited to 1200 observations -- see -help limits-).  If you're still having issues getting this imported, more information about your system (version of Stata, OS, physical RAM) would be helpful for others to answer your question.

- Eric

__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
ebooth@ppri.tamu.edu
Office: +979.845.6754


On Apr 20, 2011, at 6:54 AM, Eric Booth wrote:

> <>
> 
> First, try increasing your memory -- if you're setting your mem with  "set mem 10000", that's setting it to 10000K, not 10000M (examine  -query mem- or -memory- for details).   Even so, try "stepping up" your memory as much as possible.  Here is a good guide to the amount of mem you'll need: http://www.stata.com/support/faqs/data/howbig.html  Most modern machines (even 32-bit) should be able to handle data the size you are describing.
> 
> Second, you can try using -infile- (see this thread on some tips: http://www.stata.com/statalist/archive/2003-09/msg00122.html).  Also, the user program -chunky- (from SSC) may help.  Finally, you could always try the software StatTransfer (http://www.stattransfer.com/).
> 
> - Eric
> 
> 
> __
> Eric A. Booth
> Public Policy Research Institute
> Texas A&M University
> ebooth@ppri.tamu.edu
> 
> 
> On Apr 20, 2011, at 5:57 AM, King, Carina wrote:
> 
>> Hi,
>> 
>> I am having problems importing a file. It is stored as csv file and contains (I think, as I haven't been able to see the full data yet) over 1,400,000 records with 25 variables. So far I have tried the following:
>> 
>> Opening it with insheet using "D:\datasets\mr_final.csv" (not enough memory), so and I increased my memory (which is now set to 10,000) and it's still not enough. 
>> 
>> I tried opening it in excel and wordpad so I could separate it out into smaller chunks, but it crashes the programmes when I try that. 
>> 
>> I attempted to use infile and specify "in 1/500000" but it comes with the error message 'file does not contain dictionary'
>> 
>> So, is there a way to make a dictionary? Or change the file type so it takes less space? Or use the 'in' option with insheet? Or any other ideas?? I'm totally stuck! Thanks!
>> 
>> Kind Regards,
>> Carina
>> 


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index