Re: st: compressed data and named pipes on linux

[Daniel Feenberg <feenberg@nber.org>]

On Sat, 18 Aug 2012, James Sams wrote:

> I wish to compress my stata datasets and use them via a named pipe. I was

...
> $ mkfifo tempfile
> $ zcat 2667.dta.gz > tempfile &
> $ stata -q use tempfile.
> Segmentation fault (core dumped)
>
> FWIW, Doing this works just fine:
> $ mkfifo tempfile
> $ zcat 2667.dta.gz > tempfile &
> $ cat tempfile > 2667.dta && stata -q use 2667.dta
> $ stata -q use 2667.dta

Although the Stata documentation refers to using "mknod" rather than 
"mkfifo", both seem to work in the same way.

To read a .dta file from a pipe here, I needed to change tempfile to 
tempfile.dta, or the use statement complains "file tempfile.dta not 
found". I also have to change ">tempfile" to ">>tempfile.dta", or the 
shell complains "file exists", based on the fifo existing. The latter 
depends on the shell you use, but I am not sure why you manage to avoid 
the former.

The segmentation error is something we used to get in the distant past 
with any dataset that was larger than a few kilobytes, but as far as I can 
tell reading from a pipe does work in Stata 11+, even for very large .dta 
files.

So I think there is something special about 2667.dta that causes -use- to 
try some random access I/O, which won't work with a pipe and will cause a 
segmentation error. Have you tried this with toy datasets, perhaps ones 
lacking labels or other features? Perhaps Stata tech support could help. 
Can you send them an example using one of the standard example datasets?

If you do find out what is causing the segmentation fault, I hope you will 
post the information or a workaround here.

daniel feenberg

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/