Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: file too big to import


From   Eric Booth <ebooth@ppri.tamu.edu>
To   "<statalist@hsphsun2.harvard.edu>" <statalist@hsphsun2.harvard.edu>
Subject   Re: st: file too big to import
Date   Wed, 20 Apr 2011 11:54:17 +0000

<>

First, try increasing your memory -- if you're setting your mem with  "set mem 10000", that's setting it to 10000K, not 10000M (examine  -query mem- or -memory- for details).   Even so, try "stepping up" your memory as much as possible.  Here is a good guide to the amount of mem you'll need: http://www.stata.com/support/faqs/data/howbig.html  Most modern machines (even 32-bit) should be able to handle data the size you are describing.

Second, you can try using -infile- (see this thread on some tips: http://www.stata.com/statalist/archive/2003-09/msg00122.html).  Also, the user program -chunky- (from SSC) may help.  Finally, you could always try the software StatTransfer (http://www.stattransfer.com/).

- Eric


__
Eric A. Booth
Public Policy Research Institute
Texas A&M University
ebooth@ppri.tamu.edu


On Apr 20, 2011, at 5:57 AM, King, Carina wrote:

> Hi,
> 
> I am having problems importing a file. It is stored as csv file and contains (I think, as I haven't been able to see the full data yet) over 1,400,000 records with 25 variables. So far I have tried the following:
> 
> Opening it with insheet using "D:\datasets\mr_final.csv" (not enough memory), so and I increased my memory (which is now set to 10,000) and it's still not enough. 
> 
> I tried opening it in excel and wordpad so I could separate it out into smaller chunks, but it crashes the programmes when I try that. 
> 
> I attempted to use infile and specify "in 1/500000" but it comes with the error message 'file does not contain dictionary'
> 
> So, is there a way to make a dictionary? Or change the file type so it takes less space? Or use the 'in' option with insheet? Or any other ideas?? I'm totally stuck! Thanks!
> 
> Kind Regards,
> Carina
> 
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index