Re: st: A bug in egen and gen?

[Nick Cox <njcoxstata@gmail.com>]

You're discovering for yourself what is already documented e.g. at
help datatypes. The issue for you remains very simple. Using -double-s
wherever you would otherwise use -float-s or -long-s will in respect
of those variables double your storage. You can get some of that back
by -compress-, but not all. In fact, if you are fairly typical in your
Stata use, my guess is not much.
You should also remember that often Stata creates temporary variables
on your behalf, so the storage you need may temporarily be much larger
than you realise.

Most users really are better off with the default. That is why it is
the default. Otherwise, StataCorp would have changed it long since.

Nick

On Thu, Feb 17, 2011 at 8:13 PM, Liao, Junlin <junlin-liao@uiowa.edu> wrote:
> I expanded my experiment to include one more variable that contains a string ("Experiment for testing Stata Options"). The results are the same:
>
> No compress, with option float: 39,064KB
> No compress, with option double: 42,970KB
> Compressed with either option: 39,064KB
>
> Basically float=long in terms of storage. The option for numeric variables has no impact on string variables - as expected.
>
> Junlin Liao
> Surgery Finance, 1422 JCP
> Phone: (319) 356-2588
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Sarah Edgington
> Sent: Thursday, February 17, 2011 2:01 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> Junlin,
> Have you tried this experiment with something other than a large integer?  I think Nick's point was that you only regain the space using compress if you're dealing exclusively with integers.  You've demonstrated that compressing a double variable in the case where all observations are integers gets you the same storage size as if you'd started out in float.
> Is the same true if all observations are not integers?
> -Sarah
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
> Sent: Thursday, February 17, 2011 11:53 AM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> Nick,
>
> I had experimented with Stata in terms of storage. Here are the results:
>
> I generate one variable with a single value of 83085733 for 1000,000 times.
> The different sizes are
>                                 Original file after -compress-
> Float                      3907KB
> 3907KB (Long)
> Double                 7813KB
> 3907KB (Long)
>
> I can see that if the variable is of type double, it requires twice as much storage space comparing to float. The storage space for float is as much as for long. There is no difference after the files are compressed to the final appropriate data type. Therefore, my recommendation for Stata to use double as default calculation and finally select the appropriate type to store data is sensible.
>
> It is also desirable to simply set type to double and compress whenever saving data.
>
> Thanks,
>
> Junlin
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Liao, Junlin
> Sent: Thursday, February 17, 2011 1:29 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: RE: st: A bug in egen and gen?
>
> I'm confused here now. Isn't the type of variables determines storage spaces? I'll do some experiments to see your point here. Your attention and quick responses are greatly appreciated.
>
> Junlin
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
> Sent: Thursday, February 17, 2011 1:23 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: A bug in egen and gen?
>
> You're saying, in effect, that nearly doubling storage would not typically bite users. That will be true in some cases but not all.
>
> There is no need to wonder. The help for -save- says there is no such option. But -compress- before -save- is naturally a very good choice and you could program your own wrapper for -save- that always did it.
>
> Here is a sketch:
>
> program jlsave
> version 8
> compress
> save `0'
> end
>
> But if you do what you just said you wanted to do, -set type double-, using
> -compress- is not going to give you back more than a fraction of the extra storage you spend. The fraction will depend on how much you deal with strings, always integer variables, etc.
>
> On Thu, Feb 17, 2011 at 7:11 PM, Liao, Junlin <junlin-liao@uiowa.edu> wrote:
>
> Storage wouldn't be a problem if we perform -compress- command regularly.
> I'm wondering if Stata can let you select an option whenever it saves data, it compresses. It will surely be handy to solve this problem.
>
> Nick Cox
>
>> In practice, if StataCorp always warned you of everything that could
>> bite
> you, the help would be much, much longer.
>>
>> Your last suggestion would typically leave -double-s in place unless
>> it so
> happened that the result was integers in every observation. To see why, study Bill Gould's recent postings on the StataCorp blog.
>> That would on average nearly double your storage. If you don't mind,
>> you
> might as follow Stas' suggestion and -set type double-.
>>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
> ________________________________
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
> ________________________________
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
> ________________________________
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/