Re: st: A bug in egen and gen?

[Nick Cox <njcoxstata@gmail.com>]

As said, do read Bill Gould's blog entries on precision.

Nick

On Thu, Feb 17, 2011 at 8:03 PM, Liao, Junlin <junlin-liao@uiowa.edu> wrote:
> Thanks for reminding me that. I'll try that and report back.
>
> Junlin Liao
> Surgery Finance, 1422 JCP
> Phone: (319) 356-2588
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Sarah Edgington
> Sent: Thursday, February 17, 2011 2:01 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> Junlin,
> Have you tried this experiment with something other than a large integer?  I think Nick's point was that you only regain the space using compress if you're dealing exclusively with integers.  You've demonstrated that compressing a double variable in the case where all observations are integers gets you the same storage size as if you'd started out in float.
> Is the same true if all observations are not integers?
> -Sarah
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
> Sent: Thursday, February 17, 2011 11:53 AM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> Nick,
>
> I had experimented with Stata in terms of storage. Here are the results:
>
> I generate one variable with a single value of 83085733 for 1000,000 times.
> The different sizes are
>                                 Original file after -compress-
> Float                      3907KB
> 3907KB (Long)
> Double                 7813KB
> 3907KB (Long)
>
> I can see that if the variable is of type double, it requires twice as much storage space comparing to float. The storage space for float is as much as for long. There is no difference after the files are compressed to the final appropriate data type. Therefore, my recommendation for Stata to use double as default calculation and finally select the appropriate type to store data is sensible.
>
> It is also desirable to simply set type to double and compress whenever saving data.
>
> Thanks,
>
> Junlin
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
> Sent: Thursday, February 17, 2011 1:29 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> I'm confused here now. Isn't the type of variables determines storage spaces? I'll do some experiments to see your point here. Your attention and quick responses are greatly appreciated.
>
> Junlin
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
> Sent: Thursday, February 17, 2011 1:23 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: A bug in egen and gen?
>
> You're saying, in effect, that nearly doubling storage would not typically bite users. That will be true in some cases but not all.
>
> There is no need to wonder. The help for -save- says there is no such option. But -compress- before -save- is naturally a very good choice and you could program your own wrapper for -save- that always did it.
>
> Here is a sketch:
>
> program jlsave
> version 8
> compress
> save `0'
> end
>
> But if you do what you just said you wanted to do, -set type double-, using
> -compress- is not going to give you back more than a fraction of the extra storage you spend. The fraction will depend on how much you deal with strings, always integer variables, etc.
>
> On Thu, Feb 17, 2011 at 7:11 PM, Liao, Junlin <junlin-liao@uiowa.edu> wrote:
>
> Storage wouldn't be a problem if we perform -compress- command regularly.
> I'm wondering if Stata can let you select an option whenever it saves data, it compresses. It will surely be handy to solve this problem.
>
> Nick Cox
>
>> In practice, if StataCorp always warned you of everything that could
>> bite
> you, the help would be much, much longer.
>>
>> Your last suggestion would typically leave -double-s in place unless
>> it so
> happened that the result was integers in every observation. To see why, study Bill Gould's recent postings on the StataCorp blog.
>> That would on average nearly double your storage. If you don't mind,
>> you
> might as follow Stas' suggestion and -set type double-.
>>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/