Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: compressed data and named pipes on linux


From   Daniel Feenberg <feenberg@nber.org>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: compressed data and named pipes on linux
Date   Sun, 19 Aug 2012 17:24:42 -0400 (EDT)


On Sat, 18 Aug 2012, James Sams wrote:

I wish to compress my stata datasets and use them via a named pipe. I was

...
$ mkfifo tempfile
$ zcat 2667.dta.gz > tempfile &
$ stata -q use tempfile.
Segmentation fault (core dumped)

FWIW, Doing this works just fine:
$ mkfifo tempfile
$ zcat 2667.dta.gz > tempfile &
$ cat tempfile > 2667.dta && stata -q use 2667.dta
$ stata -q use 2667.dta



Although the Stata documentation refers to using "mknod" rather than "mkfifo", both seem to work in the same way.

To read a .dta file from a pipe here, I needed to change tempfile to tempfile.dta, or the use statement complains "file tempfile.dta not found". I also have to change ">tempfile" to ">>tempfile.dta", or the shell complains "file exists", based on the fifo existing. The latter depends on the shell you use, but I am not sure why you manage to avoid the former.

The segmentation error is something we used to get in the distant past with any dataset that was larger than a few kilobytes, but as far as I can tell reading from a pipe does work in Stata 11+, even for very large .dta files.

So I think there is something special about 2667.dta that causes -use- to try some random access I/O, which won't work with a pipe and will cause a segmentation error. Have you tried this with toy datasets, perhaps ones lacking labels or other features? Perhaps Stata tech support could help. Can you send them an example using one of the standard example datasets?

If you do find out what is causing the segmentation fault, I hope you will post the information or a workaround here.

daniel feenberg

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index