Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down at the end of May, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
Nick Cox <njcoxstata@gmail.com> |

To |
"statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu> |

Subject |
Re: st: xtlogit: panel data transformation's recast to double makes model incomputable |

Date |
Wed, 3 Apr 2013 18:13:16 +0100 |

Update on -ladder-, following some experiments: -ladder- breaks down for extreme sample sizes and/or extreme sample skewness and kurtosis. Tom has both. Essentially either case yields very low P-values, sometimes beyond the approximations that Stata uses. The most fragile part is the default adjustment, which in fact is carried out by -sktest-. As reasonable transformations are not difficult to guess otherwise, this isn't really a roadblock. On 3 April 2013 17:10, Nick Cox <njcoxstata@gmail.com> wrote: > So, it is not a problem with zero or negative values. I don't have an > explanation for your difficulties with -ladder- otherwise. > > I don't think anyone needs a test to choose a transformation with 2 > million values that are manifestly highly skew. > > Your -summarize- results give min, max and nine quantiles in between. > Transformations of interest are here all monotonic, so preserve order, > so it is sufficient just to fire up Mata to look at what quantiles > become, e.g. > > : y > 1 > +----------+ > 1 | .02 | > 2 | .62 | > 3 | 1.66 | > 4 | 3.09 | > 5 | 8.66 | > 6 | 18.68 | > 7 | 33.48 | > 8 | 57.38 | > 9 | 74.08 | > 10 | 123.49 | > 11 | 785.37 | > +----------+ > > : log10(y) > 1 > +----------------+ > 1 | -1.698970004 | > 2 | -.2076083105 | > 3 | .220108088 | > 4 | .4899584794 | > 5 | .937517892 | > 6 | 1.271376872 | > 7 | 1.524785449 | > 8 | 1.758760544 | > 9 | 1.869700974 | > 10 | 2.091631791 | > 11 | 2.895074308 | > +----------------+ > > : sqrt(y) > 1 > +---------------+ > 1 | .1414213562 | > 2 | .7874007874 | > 3 | 1.288409873 | > 4 | 1.757839583 | > 5 | 2.942787794 | > 6 | 4.322036557 | > 7 | 5.786190457 | > 8 | 7.574958746 | > 9 | 8.606973917 | > 10 | 11.11260546 | > 11 | 28.02445361 | > +---------------+ > > A glance shows that roots are not strong enough, but logarithms do > much better, which matches a lot in economic thinking. However, it's > one thing to make a distribution more symmetric; the question of how > it behaves in a model remains. > > Nick > > On 3 April 2013 16:09, Tom <tommedema@gmail.com> wrote: >> "close" is a price, and therefore it is always positive. -sum close, >> detail- follows (after dropping some outliers in which I am not >> interested): >> http://pastebin.com/25gm94i0 >> >> Because close is always positive I would like to return to the -ladder >> close- issue. It turns out that when I supply -noadjust-, i.e. execute >> -ladder close, noadjust- it does return some details: >> http://pastebin.com/m8Z4YmKd >> >> -noadjust- is simply the -noadjust- option of -sktest-, used by >> -ladder-. The help file says that this is what it does: >> >> " noadjust suppresses the empirical adjustment made by Royston >> (1991) to the overall chi-squared and its significance level and >> presents the unaltered test as described by D'Agostino, Balanger, and >> D'Agostino (1990)." >> >> But does anyone have a clue why this adjustment would prevent ladder >> from returning anything? Also see the results of -sktest close- versus >> -sktest close, noadjust-: >> http://pastebin.com/tPUf3xE6 >> >> Tom >> >> On Wed, Apr 3, 2013 at 4:54 PM, Nick Cox <njcoxstata@gmail.com> wrote: >>> The transformations that -ladder- tries are typically only defined for >>> positive variables. As I recall, many of your variables are variously >>> positive and negative. >>> >>> The usual transformations for right-skewed variables, such as square >>> root and logarithmic, are thus no use in your case. >>> >>> This is precisely why cube root and inverse hyperbolic functions were >>> suggested earlier/ >>> >>> On the face of it that is not an explanation of why all the >>> transformations fail, but I'd not be inclined to think that -ladder- >>> will help your problem. >>> >>> I'm afraid that you have keep reminding us of basics -- e.g. what is >>> -close- and what are its descriptive statistics -- as even those >>> following your threads can't be expected to keep track of all earlier >>> posts. >>> >>> Nick >>> >>> On 3 April 2013 15:33, Tom <tommedema@gmail.com> wrote: >>> >>>> Anyway, I think I may have a very important pointer that could help in >>>> finding the cause of this issue: >>>> when I run -ladder close- it returns absolutely nothing, i.e.: >>>> >>>> ladder close >>>> >>>> Transformation formula chi2(2) P(chi2) >>>> ------------------------------------------------------------------ >>>> cubic close^3 . . >>>> square close^2 . . >>>> identity close . . >>>> square root sqrt(close) . . >>>> log log(close) . . >>>> 1/(square root) 1/sqrt(close) . . >>>> inverse 1/close . . >>>> 1/square 1/(close^2) . . >>>> 1/cubic 1/(close^3) . . >>>> >>>> I believe this is very much related to the problems I am having with >>>> this variable, close. If I can determine why ladder refuses to give me >>>> any transformations, then I may be able to figure out what is the >>>> source of this issue. >>>> >>>> I am not just trying to get an answer, I am interested in why ladder >>>> is not returning any transformations because this may give me >>>> information on the root cause of the problem. >>> * >>> * For searches and help try: >>> * http://www.stata.com/help.cgi?search >>> * http://www.stata.com/support/faqs/resources/statalist-faq/ >>> * http://www.ats.ucla.edu/stat/stata/ >> * >> * For searches and help try: >> * http://www.stata.com/help.cgi?search >> * http://www.stata.com/support/faqs/resources/statalist-faq/ >> * http://www.ats.ucla.edu/stat/stata/ > > > > -- > Nick > njcoxstata@gmail.com -- Nick njcoxstata@gmail.com * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/

**References**:**st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Maarten Buis <maartenlbuis@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**AW: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"Klaus Pforr" <kpforr@googlemail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Tom <tommedema@gmail.com>

**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable***From:*Nick Cox <njcoxstata@gmail.com>

- Prev by Date:
**Re: st: playing a media sound (.wav) from Stata in Windows 7** - Next by Date:
**st: Newey-west errors for ordered probit?** - Previous by thread:
**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable** - Next by thread:
**Re: st: xtlogit: panel data transformation's recast to double makes model incomputable** - Index(es):