Re: st: upper limit on fweights? overflowing into missing values?

[Richard Williams <richardwilliams.ndu@gmail.com>]

I agree that the documentation could be clearer. However, for -help weights-, it says for fweights that "the result of the command is the same as if you duplicated each observation however many times and then ran the command unweighted." So, the implication would seem to be that the fweighted number of cases can be no greater than Stata's 2 billion+ limit. And that in fact is born out by my examples.

I don't know why there is a 2 billion limit -- and yes, it seems like fweighting could be a way around it -- but apparently it isn't. My guess is that the 2 billion limit is rarely a problem and hence StataCorp hasn't worried too much about it. Or, maybe you would need a super long data type to handle anything bigger. After all, the sample size does enter into various formulas, and they may not be set up to handle anything more than 2 billion plus.

Sent from my iPad

On Jul 28, 2013, at 7:53 PM, Nick Cox <njcoxstata@gmail.com> wrote:

> This is interesting, but in principle I don't see that Stata's limit
> on # of observations has any bearing on how big frequency weights can
> be. I can imagine people wanting to use frequency weights to subvert
> the limit on number of observations.
> 
> A different point is that if there is a limit on how big weights can
> be it should be documented e.g. at -help limits-.
> Nick
> njcoxstata@gmail.com
> 
> 
> On 29 July 2013 00:46, Richard Williams <richardwilliams.ndu@gmail.com> wrote:
>> According to -help limits-, the maximum number of observations is 2,147,483,647. Your weights give you more than 4 billion cases, well above that. Further, the help also says that this is a theoretical maximum; memory availability will certainly impose a smaller maximum.
>> 
>> On my computer, I specified [fw = 1073741823] on the pwcorr command and it ran. Then I specified [fw = 1073741824] and it did not run. These numbers put you just below and just above the maximum number of cases that Stata allows.
>> 
>> So in short, it appears that your fweighted cases can't exceed the 2 billion+ that Stata allows, and memory restrictions may hold you to even less than that.
>> 
>> Also, you probably need to specify that the fweight variable is type long, e.g.
>> 
>> input y x long fw
>> 
>> Sent from my iPad
>> 
>> On Jul 27, 2013, at 12:36 PM, László Sándor <sandorl@gmail.com> wrote:
>> 
>>> Hi,
>>> If you care, here is an example that silently produces missing values.
>>> I notified Stata Support.
>>> 
>>> input y x fw
>>> 2 1 2147483621
>>> 1 2 2147483621
>>> end
>>> de
>>> pwcorr y x [fw=fw]
>>> exit
>>> 
>>> Thanks,
>>> 
>>> Laszlo
>>> 
>>> On Sun, Jul 21, 2013 at 5:08 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>> I'd suggest documenting your problems with a reproducible example and
>>>> sending Stata tech support.
>>>> 
>>>> 
>>>> Nick
>>>> njcoxstata@gmail.com
>>>> 
>>>> 
>>>> On 21 July 2013 21:55, László Sándor <sandorl@gmail.com> wrote:
>>>>> Hi,
>>>>> in Stata/MP 12.1 I am getting missing values with using -pwcorr- with
>>>>> -fweights- though the feature works fine with other data or if I scale
>>>>> my weights down. Is it possible to simply have too large fweights,
>>>>> e.g. if they cannot be of type -long- anymore?
>>>>> 
>>>>> If so, why doesn't Stata warn me about this?
>>>>> 
>>>>> I vaguely remember some Statalist of Stata blog discussion of this,
>>>>> but I could not even Google it up, and Stata still did not warn me…
>>>>> 
>>>>> Actually, why didn't Stata complain that I did not have integer
>>>>> fweights if obviously the variable wasn't of type byte, int or long?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Laszlo
>>>>> 
>>>>> *
>>>>> *   For searches and help try:
>>>>> *   http://www.stata.com/help.cgi?search
>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>> 
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>> 
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> 
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/