Statalist The Stata Listserver


[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

Re: st: Bonferroni and Sidak adjusted p-values


From   khigbee@stata.com
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Bonferroni and Sidak adjusted p-values
Date   Wed, 08 Nov 2006 11:23:15 -0600

Thomas J. Steichen <SteichT@rjrt.com> asks:

> I have noted on a number of occasions that the Bonferroni and Sidak 
> adjusted p-values from multiple comparisons by the -oneway- command 
> differ slightly (but more than simple rounding error) from what my 
> hand calculations suggest. Has anyone else noted this?
> 
> I'll give some example output below (using Sidak adjustment) and 
> also show my hand calculations: 
> 
> .. oneway vol id, sidak t
> 
>             |         Summary of Volume
>          ID |        Mean   Std. Dev.       Freq.
> ------------+------------------------------------
>          AA |   934.42349   207.71981         132
>          AB |     953.325   201.66785         140
>          AC |   1049.2007   232.39271         137
>          AD |   1003.5008   212.34701         132
>          AE |   931.34357   193.50937         140
>          AF |   856.44392   183.51134         148
> ------------+------------------------------------
>       Total |   953.14089   213.68692         829
> 
>                       Analysis of Variance
>     Source            SS         df      MS          F     Prob > F
> --------------------------------------------------------------------
> Between groups    3095545.81      5   619109.162   14.68     0.0000
>  Within groups    34712674.1    823   42178.2188
> --------------------------------------------------------------------
>     Total         37808219.9    828   45662.1013
> 
> Bartlett's test for equal variances: chi2(5) = 9.3237 Prob>chi2 = 0.097
> 
>                           Comparison of Volume by ID
>                                    (Sidak)
> Row Mean-|
> Col Mean |         AA         AB         AC         AD         AE
> ---------+-------------------------------------------------------
>       AB |    18.9015
>          |      1.000
>          |
>       AC |    114.777    95.8757
>          |      0.000      0.002
>          |
>       AD |    69.0773    50.1758      -45.7
>          |      0.092      0.494      0.655
>          |
>       AE |   -3.07991   -21.9814   -117.857   -72.1572
>          |      1.000      0.999      0.000      0.057
>          |
>       AF |   -77.9796   -96.8811   -192.757   -147.057   -74.8997
>          |      0.023      0.001      0.000      0.000      0.030
> 
> 
> As an example, calculate p for the AD-AA comparison (shown as 
> 0.092 above).
> 
> First, calculate t as t = diff / (s * sqrt( 1/n1 + 1/n4))
>    where diff is in the above table, s = sqrt(within MS) and
>          n1 and n4 are the Freq's from the first table above.
> 
> .. di 69.0773 / (sqrt(42178.2188) * sqrt(1/132 +1/132))
> 2.7325191
> 
> Now calculate 1-tailed t probability
> 
> .. di ttail(132+132-2, 2.7325191)
> ..00335642
> 
> Note: there are n = 15 comparisons to be adjusted for
> 
> Calculate Sidak adjustment (using 2-tailed probability)
> 
> .. di min(1, 1 - (1 - .00335642 * 2)^15)
> ..09609597
> 
> Note: this differs from the reported 0.092
> 
> 
> Calculate Bonferroni adjustment (using 2-tailed probability)
> 
> .. di min(1, .00335642*2*15)
> ..1006926
> 
> Note: the Bonferroni adjusted p is reported as 0.096
> 
> 
> Calculate Scheffe adjustment (using t value)
> 
> .. di  Ftail(5,823,(2.7325191^2)/5)
> ..18949275
> 
> Note: the Scheffe adjusted p is reported as 0.189
>       (this one is usually OK within round-off)
> 
> 
> If someone can confirm this problem -- or point out the
> error in my calculations -- I would appreciate it.


You are using 132+132-2 as the degrees of freedom.  However,
s=sqrt(within MS) is computed from the pooled groups (all 6
groups in your case) and the degrees of freedom for that pooled s
is 823.  So for your example, to get the Sidak adjusted p-value,
you would

    . di min(1, 1-(1-ttail(823,
           69.0773/(sqrt(42178.2188)*sqrt(1/132+1/132)))*2)^15)

    .09208965

which agrees with what -oneway- produced for the Sidak adjusted
p-value for the comparison of AA and AD.  Likewise for the
Bonferroni adjusted p-value, you need to use 823 instead of
132+132-2 for the degrees of freedom in your call to -ttail()-.
For Scheffe you were already using the 823 degrees of freedom in
the call to -Ftail()-.  That is why it was in agreement with
-oneway-s output.

The Bartlett test reported under the anova table in -oneway- can
help you decide if pooling the groups to obtain a common within
MS (and hence a common s) was a reasonable thing to do or not.
If not, then -oneway- is not appropriate.

Ken Higbee    khigbee@stata.com
StataCorp     1-800-STATAPC
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index