Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Importing subset of a pipe delimited textfile


From   Brendan Halpin <brendan.halpin@ul.ie>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Importing subset of a pipe delimited textfile
Date   Wed, 17 Oct 2012 21:51:29 +0100

On Wed, Oct 17 2012, Rob Shaw wrote:

> I have a very large (around 4Gb) text file that has been pipe
> delimited. It won't all fit in memory so I want to process it in
> parts.
>
> For fixed datasets I would use infile with the in 1/10000000 option
> then 10000001/2000000 etc. However, this dataset has been pipe
> delimited so I would need to use insheet, but insheet doesn't seem to
> permit the "in" option.

If you're on Mac, Linux or Solaris, the well-worn unix text utilities
will deal with this neatly. In particular -split- will create subfile
(and sed will change | to \t, but filefilter copes with that)

Something like

split --numeric-suffixes --lines=10000 bigfile.raw 

should create multiple files containing 10000 lines, with a numeric
suffix (bigfile.raw01 and so on, I think), which you can then handle
cleanly in Stata.

If you're on that other OS, there's always cygwin.


Brendan
-- 
Brendan Halpin,   Department of Sociology,   University of Limerick,   Ireland
Tel: w +353-61-213147  f +353-61-202569  h +353-61-338562;  Room F1-009 x 3147
mailto:brendan.halpin@ul.ie    ULSociology on Facebook: http://on.fb.me/fjIK9t
http://teaching.sociology.ul.ie/bhalpin/wordpress         twitter:@ULSociology
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index