# st: Bonferroni and Sidak adjusted p-values

 From "Steichen, Thomas J." <[email protected]> To <[email protected]> Subject st: Bonferroni and Sidak adjusted p-values Date Tue, 7 Nov 2006 10:47:22 -0500

```Statalisters,

I have noted on a number of occasions that the Bonferroni and Sidak
adjusted p-values from multiple comparisons by the -oneway- command
differ slightly (but more than simple rounding error) from what my
hand calculations suggest. Has anyone else noted this?

I'll give some example output below (using Sidak adjustment) and
also show my hand calculations:

.. oneway vol id, sidak t

|         Summary of Volume
ID |        Mean   Std. Dev.       Freq.
------------+------------------------------------
AA |   934.42349   207.71981         132
AB |     953.325   201.66785         140
AC |   1049.2007   232.39271         137
AE |   931.34357   193.50937         140
AF |   856.44392   183.51134         148
------------+------------------------------------
Total |   953.14089   213.68692         829

Analysis of Variance
Source            SS         df      MS          F     Prob > F
--------------------------------------------------------------------
Between groups    3095545.81      5   619109.162   14.68     0.0000
Within groups    34712674.1    823   42178.2188
--------------------------------------------------------------------
Total         37808219.9    828   45662.1013

Bartlett's test for equal variances: chi2(5) = 9.3237 Prob>chi2 = 0.097

Comparison of Volume by ID
(Sidak)
Row Mean-|
Col Mean |         AA         AB         AC         AD         AE
---------+-------------------------------------------------------
AB |    18.9015
|      1.000
|
AC |    114.777    95.8757
|      0.000      0.002
|
|      0.092      0.494      0.655
|
AE |   -3.07991   -21.9814   -117.857   -72.1572
|      1.000      0.999      0.000      0.057
|
AF |   -77.9796   -96.8811   -192.757   -147.057   -74.8997
|      0.023      0.001      0.000      0.000      0.030

As an example, calculate p for the AD-AA comparison (shown as
0.092 above).

First, calculate t as t = diff / (s * sqrt( 1/n1 + 1/n4))
where diff is in the above table, s = sqrt(within MS) and
n1 and n4 are the Freq's from the first table above.

.. di 69.0773 / (sqrt(42178.2188) * sqrt(1/132 +1/132))
2.7325191

Now calculate 1-tailed t probability

.. di ttail(132+132-2, 2.7325191)
..00335642

Note: there are n = 15 comparisons to be adjusted for

Calculate Sidak adjustment (using 2-tailed probability)

.. di min(1, 1 - (1 - .00335642 * 2)^15)
..09609597

Note: this differs from the reported 0.092

Calculate Bonferroni adjustment (using 2-tailed probability)

.. di min(1, .00335642*2*15)
..1006926

Note: the Bonferroni adjusted p is reported as 0.096

Calculate Scheffe adjustment (using t value)

.. di  Ftail(5,823,(2.7325191^2)/5)
..18949275

Note: the Scheffe adjusted p is reported as 0.189
(this one is usually OK within round-off)

If someone can confirm this problem -- or point out the
error in my calculations -- I would appreciate it.

Tom

-----------------------------------------
Thomas J. Steichen
[email protected]
-----------------------------------------

-----------------------------------------
CONFIDENTIALITY NOTE: This e-mail message, including any
attachment(s), contains information that may be confidential,
protected by the attorney-client or other legal privileges, and/or
proprietary non-public information. If you are not an intended
recipient of this message or an authorized assistant to an intended
then delete it from your system. Use, dissemination, distribution,
or reproduction of this message and/or any of its attachments (if
any) by unintended recipients is not authorized and may be
unlawful.

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```