Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: file too big to import
From
Eric Booth <[email protected]>
To
"<[email protected]>" <[email protected]>
Subject
Re: st: file too big to import
Date
Wed, 20 Apr 2011 11:54:17 +0000
<>
First, try increasing your memory -- if you're setting your mem with "set mem 10000", that's setting it to 10000K, not 10000M (examine -query mem- or -memory- for details). Even so, try "stepping up" your memory as much as possible. Here is a good guide to the amount of mem you'll need: http://www.stata.com/support/faqs/data/howbig.html Most modern machines (even 32-bit) should be able to handle data the size you are describing.
Second, you can try using -infile- (see this thread on some tips: http://www.stata.com/statalist/archive/2003-09/msg00122.html). Also, the user program -chunky- (from SSC) may help. Finally, you could always try the software StatTransfer (http://www.stattransfer.com/).
- Eric
__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
[email protected]
On Apr 20, 2011, at 5:57 AM, King, Carina wrote:
> Hi,
>
> I am having problems importing a file. It is stored as csv file and contains (I think, as I haven't been able to see the full data yet) over 1,400,000 records with 25 variables. So far I have tried the following:
>
> Opening it with insheet using "D:\datasets\mr_final.csv" (not enough memory), so and I increased my memory (which is now set to 10,000) and it's still not enough.
>
> I tried opening it in excel and wordpad so I could separate it out into smaller chunks, but it crashes the programmes when I try that.
>
> I attempted to use infile and specify "in 1/500000" but it comes with the error message 'file does not contain dictionary'
>
> So, is there a way to make a dictionary? Or change the file type so it takes less space? Or use the 'in' option with insheet? Or any other ideas?? I'm totally stuck! Thanks!
>
> Kind Regards,
> Carina
>
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/