Statalist The Stata Listserver


[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: Bonferroni and Sidak adjusted p-values


From   "Steichen, Thomas J." <SteichT@rjrt.com>
To   <statalist@hsphsun2.harvard.edu>
Subject   st: Bonferroni and Sidak adjusted p-values
Date   Tue, 7 Nov 2006 10:47:22 -0500

Statalisters,

I have noted on a number of occasions that the Bonferroni and Sidak 
adjusted p-values from multiple comparisons by the -oneway- command 
differ slightly (but more than simple rounding error) from what my 
hand calculations suggest. Has anyone else noted this?

I'll give some example output below (using Sidak adjustment) and 
also show my hand calculations: 

.. oneway vol id, sidak t

            |         Summary of Volume
         ID |        Mean   Std. Dev.       Freq.
------------+------------------------------------
         AA |   934.42349   207.71981         132
         AB |     953.325   201.66785         140
         AC |   1049.2007   232.39271         137
         AD |   1003.5008   212.34701         132
         AE |   931.34357   193.50937         140
         AF |   856.44392   183.51134         148
------------+------------------------------------
      Total |   953.14089   213.68692         829

                      Analysis of Variance
    Source            SS         df      MS          F     Prob > F
--------------------------------------------------------------------
Between groups    3095545.81      5   619109.162   14.68     0.0000
 Within groups    34712674.1    823   42178.2188
--------------------------------------------------------------------
    Total         37808219.9    828   45662.1013

Bartlett's test for equal variances: chi2(5) = 9.3237 Prob>chi2 = 0.097

                          Comparison of Volume by ID
                                   (Sidak)
Row Mean-|
Col Mean |         AA         AB         AC         AD         AE
---------+-------------------------------------------------------
      AB |    18.9015
         |      1.000
         |
      AC |    114.777    95.8757
         |      0.000      0.002
         |
      AD |    69.0773    50.1758      -45.7
         |      0.092      0.494      0.655
         |
      AE |   -3.07991   -21.9814   -117.857   -72.1572
         |      1.000      0.999      0.000      0.057
         |
      AF |   -77.9796   -96.8811   -192.757   -147.057   -74.8997
         |      0.023      0.001      0.000      0.000      0.030


As an example, calculate p for the AD-AA comparison (shown as 
0.092 above).

First, calculate t as t = diff / (s * sqrt( 1/n1 + 1/n4))
   where diff is in the above table, s = sqrt(within MS) and
         n1 and n4 are the Freq's from the first table above.

.. di 69.0773 / (sqrt(42178.2188) * sqrt(1/132 +1/132))
2.7325191

Now calculate 1-tailed t probability

.. di ttail(132+132-2, 2.7325191)
..00335642

Note: there are n = 15 comparisons to be adjusted for

Calculate Sidak adjustment (using 2-tailed probability)

.. di min(1, 1 - (1 - .00335642 * 2)^15)
..09609597

Note: this differs from the reported 0.092


Calculate Bonferroni adjustment (using 2-tailed probability)

.. di min(1, .00335642*2*15)
..1006926

Note: the Bonferroni adjusted p is reported as 0.096


Calculate Scheffe adjustment (using t value)

.. di  Ftail(5,823,(2.7325191^2)/5)
..18949275

Note: the Scheffe adjusted p is reported as 0.189
      (this one is usually OK within round-off)


If someone can confirm this problem -- or point out the
error in my calculations -- I would appreciate it.

Tom

-----------------------------------------
Thomas J. Steichen
steicht@rjrt.com
-----------------------------------------


-----------------------------------------
CONFIDENTIALITY NOTE: This e-mail message, including any
attachment(s), contains information that may be confidential,
protected by the attorney-client or other legal privileges, and/or
proprietary non-public information. If you are not an intended
recipient of this message or an authorized assistant to an intended
recipient, please notify the sender by replying to this message and
then delete it from your system. Use, dissemination, distribution,
or reproduction of this message and/or any of its attachments (if
any) by unintended recipients is not authorized and may be
unlawful.


*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index