Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: xtlogit: panel data transformation's recast to double makes model incomputable
From
Nick Cox <[email protected]>
To
"[email protected]" <[email protected]>
Subject
Re: st: xtlogit: panel data transformation's recast to double makes model incomputable
Date
Wed, 3 Apr 2013 18:13:16 +0100
Update on -ladder-, following some experiments:
-ladder- breaks down for extreme sample sizes and/or extreme sample
skewness and kurtosis. Tom has both. Essentially either case yields
very low P-values, sometimes beyond the approximations that Stata
uses. The most fragile part is the default adjustment, which in fact
is carried out by -sktest-.
As reasonable transformations are not difficult to guess otherwise,
this isn't really a roadblock.
On 3 April 2013 17:10, Nick Cox <[email protected]> wrote:
> So, it is not a problem with zero or negative values. I don't have an
> explanation for your difficulties with -ladder- otherwise.
>
> I don't think anyone needs a test to choose a transformation with 2
> million values that are manifestly highly skew.
>
> Your -summarize- results give min, max and nine quantiles in between.
> Transformations of interest are here all monotonic, so preserve order,
> so it is sufficient just to fire up Mata to look at what quantiles
> become, e.g.
>
> : y
> 1
> +----------+
> 1 | .02 |
> 2 | .62 |
> 3 | 1.66 |
> 4 | 3.09 |
> 5 | 8.66 |
> 6 | 18.68 |
> 7 | 33.48 |
> 8 | 57.38 |
> 9 | 74.08 |
> 10 | 123.49 |
> 11 | 785.37 |
> +----------+
>
> : log10(y)
> 1
> +----------------+
> 1 | -1.698970004 |
> 2 | -.2076083105 |
> 3 | .220108088 |
> 4 | .4899584794 |
> 5 | .937517892 |
> 6 | 1.271376872 |
> 7 | 1.524785449 |
> 8 | 1.758760544 |
> 9 | 1.869700974 |
> 10 | 2.091631791 |
> 11 | 2.895074308 |
> +----------------+
>
> : sqrt(y)
> 1
> +---------------+
> 1 | .1414213562 |
> 2 | .7874007874 |
> 3 | 1.288409873 |
> 4 | 1.757839583 |
> 5 | 2.942787794 |
> 6 | 4.322036557 |
> 7 | 5.786190457 |
> 8 | 7.574958746 |
> 9 | 8.606973917 |
> 10 | 11.11260546 |
> 11 | 28.02445361 |
> +---------------+
>
> A glance shows that roots are not strong enough, but logarithms do
> much better, which matches a lot in economic thinking. However, it's
> one thing to make a distribution more symmetric; the question of how
> it behaves in a model remains.
>
> Nick
>
> On 3 April 2013 16:09, Tom <[email protected]> wrote:
>> "close" is a price, and therefore it is always positive. -sum close,
>> detail- follows (after dropping some outliers in which I am not
>> interested):
>> http://pastebin.com/25gm94i0
>>
>> Because close is always positive I would like to return to the -ladder
>> close- issue. It turns out that when I supply -noadjust-, i.e. execute
>> -ladder close, noadjust- it does return some details:
>> http://pastebin.com/m8Z4YmKd
>>
>> -noadjust- is simply the -noadjust- option of -sktest-, used by
>> -ladder-. The help file says that this is what it does:
>>
>> " noadjust suppresses the empirical adjustment made by Royston
>> (1991) to the overall chi-squared and its significance level and
>> presents the unaltered test as described by D'Agostino, Balanger, and
>> D'Agostino (1990)."
>>
>> But does anyone have a clue why this adjustment would prevent ladder
>> from returning anything? Also see the results of -sktest close- versus
>> -sktest close, noadjust-:
>> http://pastebin.com/tPUf3xE6
>>
>> Tom
>>
>> On Wed, Apr 3, 2013 at 4:54 PM, Nick Cox <[email protected]> wrote:
>>> The transformations that -ladder- tries are typically only defined for
>>> positive variables. As I recall, many of your variables are variously
>>> positive and negative.
>>>
>>> The usual transformations for right-skewed variables, such as square
>>> root and logarithmic, are thus no use in your case.
>>>
>>> This is precisely why cube root and inverse hyperbolic functions were
>>> suggested earlier/
>>>
>>> On the face of it that is not an explanation of why all the
>>> transformations fail, but I'd not be inclined to think that -ladder-
>>> will help your problem.
>>>
>>> I'm afraid that you have keep reminding us of basics -- e.g. what is
>>> -close- and what are its descriptive statistics -- as even those
>>> following your threads can't be expected to keep track of all earlier
>>> posts.
>>>
>>> Nick
>>>
>>> On 3 April 2013 15:33, Tom <[email protected]> wrote:
>>>
>>>> Anyway, I think I may have a very important pointer that could help in
>>>> finding the cause of this issue:
>>>> when I run -ladder close- it returns absolutely nothing, i.e.:
>>>>
>>>> ladder close
>>>>
>>>> Transformation formula chi2(2) P(chi2)
>>>> ------------------------------------------------------------------
>>>> cubic close^3 . .
>>>> square close^2 . .
>>>> identity close . .
>>>> square root sqrt(close) . .
>>>> log log(close) . .
>>>> 1/(square root) 1/sqrt(close) . .
>>>> inverse 1/close . .
>>>> 1/square 1/(close^2) . .
>>>> 1/cubic 1/(close^3) . .
>>>>
>>>> I believe this is very much related to the problems I am having with
>>>> this variable, close. If I can determine why ladder refuses to give me
>>>> any transformations, then I may be able to figure out what is the
>>>> source of this issue.
>>>>
>>>> I am not just trying to get an answer, I am interested in why ladder
>>>> is not returning any transformations because this may give me
>>>> information on the root cause of the problem.
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>> * http://www.ats.ucla.edu/stat/stata/
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>> * http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> Nick
> [email protected]
--
Nick
[email protected]
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/