Thomas J. Steichen <[email protected]> asks:
> I have noted on a number of occasions that the Bonferroni and Sidak
> adjusted p-values from multiple comparisons by the -oneway- command
> differ slightly (but more than simple rounding error) from what my
> hand calculations suggest. Has anyone else noted this?
>
> I'll give some example output below (using Sidak adjustment) and
> also show my hand calculations:
>
> .. oneway vol id, sidak t
>
> | Summary of Volume
> ID | Mean Std. Dev. Freq.
> ------------+------------------------------------
> AA | 934.42349 207.71981 132
> AB | 953.325 201.66785 140
> AC | 1049.2007 232.39271 137
> AD | 1003.5008 212.34701 132
> AE | 931.34357 193.50937 140
> AF | 856.44392 183.51134 148
> ------------+------------------------------------
> Total | 953.14089 213.68692 829
>
> Analysis of Variance
> Source SS df MS F Prob > F
> --------------------------------------------------------------------
> Between groups 3095545.81 5 619109.162 14.68 0.0000
> Within groups 34712674.1 823 42178.2188
> --------------------------------------------------------------------
> Total 37808219.9 828 45662.1013
>
> Bartlett's test for equal variances: chi2(5) = 9.3237 Prob>chi2 = 0.097
>
> Comparison of Volume by ID
> (Sidak)
> Row Mean-|
> Col Mean | AA AB AC AD AE
> ---------+-------------------------------------------------------
> AB | 18.9015
> | 1.000
> |
> AC | 114.777 95.8757
> | 0.000 0.002
> |
> AD | 69.0773 50.1758 -45.7
> | 0.092 0.494 0.655
> |
> AE | -3.07991 -21.9814 -117.857 -72.1572
> | 1.000 0.999 0.000 0.057
> |
> AF | -77.9796 -96.8811 -192.757 -147.057 -74.8997
> | 0.023 0.001 0.000 0.000 0.030
>
>
> As an example, calculate p for the AD-AA comparison (shown as
> 0.092 above).
>
> First, calculate t as t = diff / (s * sqrt( 1/n1 + 1/n4))
> where diff is in the above table, s = sqrt(within MS) and
> n1 and n4 are the Freq's from the first table above.
>
> .. di 69.0773 / (sqrt(42178.2188) * sqrt(1/132 +1/132))
> 2.7325191
>
> Now calculate 1-tailed t probability
>
> .. di ttail(132+132-2, 2.7325191)
> ..00335642
>
> Note: there are n = 15 comparisons to be adjusted for
>
> Calculate Sidak adjustment (using 2-tailed probability)
>
> .. di min(1, 1 - (1 - .00335642 * 2)^15)
> ..09609597
>
> Note: this differs from the reported 0.092
>
>
> Calculate Bonferroni adjustment (using 2-tailed probability)
>
> .. di min(1, .00335642*2*15)
> ..1006926
>
> Note: the Bonferroni adjusted p is reported as 0.096
>
>
> Calculate Scheffe adjustment (using t value)
>
> .. di Ftail(5,823,(2.7325191^2)/5)
> ..18949275
>
> Note: the Scheffe adjusted p is reported as 0.189
> (this one is usually OK within round-off)
>
>
> If someone can confirm this problem -- or point out the
> error in my calculations -- I would appreciate it.
You are using 132+132-2 as the degrees of freedom. However,
s=sqrt(within MS) is computed from the pooled groups (all 6
groups in your case) and the degrees of freedom for that pooled s
is 823. So for your example, to get the Sidak adjusted p-value,
you would
. di min(1, 1-(1-ttail(823,
69.0773/(sqrt(42178.2188)*sqrt(1/132+1/132)))*2)^15)
.09208965
which agrees with what -oneway- produced for the Sidak adjusted
p-value for the comparison of AA and AD. Likewise for the
Bonferroni adjusted p-value, you need to use 823 instead of
132+132-2 for the degrees of freedom in your call to -ttail()-.
For Scheffe you were already using the 823 degrees of freedom in
the call to -Ftail()-. That is why it was in agreement with
-oneway-s output.
The Bartlett test reported under the anova table in -oneway- can
help you decide if pooling the groups to obtain a common within
MS (and hence a common s) was a reasonable thing to do or not.
If not, then -oneway- is not appropriate.
Ken Higbee [email protected]
StataCorp 1-800-STATAPC
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
