Re: st: upper limit on fweights? overflowing into missing values?

[Richard Williams <richardwilliams.ndu@gmail.com>]

Incidentally, some commands work when the freighted number of cases is 4 billion, e.g. -sum-, -tab2-. pwcorr, regress and corr do not. My guess is that it has something to do with the types of calculations involved.

Sent from my iPad

On Jul 28, 2013, at 7:53 PM, Nick Cox <njcoxstata@gmail.com> wrote:

> This is interesting, but in principle I don't see that Stata's limit
> on # of observations has any bearing on how big frequency weights can
> be. I can imagine people wanting to use frequency weights to subvert
> the limit on number of observations.
> 
> A different point is that if there is a limit on how big weights can
> be it should be documented e.g. at -help limits-.
> Nick
> njcoxstata@gmail.com
> 
> 
> On 29 July 2013 00:46, Richard Williams <richardwilliams.ndu@gmail.com> wrote:
>> According to -help limits-, the maximum number of observations is 2,147,483,647. Your weights give you more than 4 billion cases, well above that. Further, the help also says that this is a theoretical maximum; memory availability will certainly impose a smaller maximum.
>> 
>> On my computer, I specified [fw = 1073741823] on the pwcorr command and it ran. Then I specified [fw = 1073741824] and it did not run. These numbers put you just below and just above the maximum number of cases that Stata allows.
>> 
>> So in short, it appears that your fweighted cases can't exceed the 2 billion+ that Stata allows, and memory restrictions may hold you to even less than that.
>> 
>> Also, you probably need to specify that the fweight variable is type long, e.g.
>> 
>> input y x long fw
>> 
>> Sent from my iPad
>> 
>> On Jul 27, 2013, at 12:36 PM, László Sándor <sandorl@gmail.com> wrote:
>> 
>>> Hi,
>>> If you care, here is an example that silently produces missing values.
>>> I notified Stata Support.
>>> 
>>> input y x fw
>>> 2 1 2147483621
>>> 1 2 2147483621
>>> end
>>> de
>>> pwcorr y x [fw=fw]
>>> exit
>>> 
>>> Thanks,
>>> 
>>> Laszlo
>>> 
>>> On Sun, Jul 21, 2013 at 5:08 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>> I'd suggest documenting your problems with a reproducible example and
>>>> sending Stata tech support.
>>>> 
>>>> 
>>>> Nick
>>>> njcoxstata@gmail.com
>>>> 
>>>> 
>>>> On 21 July 2013 21:55, László Sándor <sandorl@gmail.com> wrote:
>>>>> Hi,
>>>>> in Stata/MP 12.1 I am getting missing values with using -pwcorr- with
>>>>> -fweights- though the feature works fine with other data or if I scale
>>>>> my weights down. Is it possible to simply have too large fweights,
>>>>> e.g. if they cannot be of type -long- anymore?
>>>>> 
>>>>> If so, why doesn't Stata warn me about this?
>>>>> 
>>>>> I vaguely remember some Statalist of Stata blog discussion of this,
>>>>> but I could not even Google it up, and Stata still did not warn me…
>>>>> 
>>>>> Actually, why didn't Stata complain that I did not have integer
>>>>> fweights if obviously the variable wasn't of type byte, int or long?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Laszlo
>>>>> 
>>>>> *
>>>>> *   For searches and help try:
>>>>> *   http://www.stata.com/help.cgi?search
>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>> 
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>> 
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> 
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/