Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: st: strange -multproc- results
From
"Feiveson, Alan H. (JSC-SK311)" <[email protected]>
To
"[email protected]" <[email protected]>
Subject
RE: st: strange -multproc- results
Date
Wed, 17 Mar 2010 16:26:04 -0500
Hi Roger - Thanks for your response - it certainly does help.
Al
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Roger Newson
Sent: Wednesday, March 17, 2010 3:29 PM
To: [email protected]
Subject: Re: st: strange -multproc- results
The paper to consult on these methods is Newson (2003), which gives a
survey of all the wierd and wonderful assumptions, formulas, and
features of all these procedures. The Simes procedure is less
conservative than most, but the Krieger procedure is usually even less
conservative, because it estimates the prior probability that a null
hypothesis is true. For this to be feasible, the Krieger procedure
assumes that the P-values are independent, so consistent estimation can
be done. The Storey procedure is probably even less conservative than
the Krieger, and is even more complicated.
I hope this helps.
Best wishes
Roger
References
Newson R. Multiple-test procedures and smile plots. The Stata Journal
2003; 3(2): 100-132. Download from
http://www.stata-journal.com/article.html?article=st0035
On 17/03/2010 20:07, Feiveson, Alan H. (JSC-SK311) wrote:
> Hi - I have been using -multproc- to control the false discovery rate (FDR) on 27 significance tests. As given in the help file, there are several methods to chose from for controlling the FDR: liu1,liu2,simes,yekutieli, and krieger.
>
>
> method() Step type FWER/FDR Definition or source
> userspecified One-step Either pcor() option
> bonferroni One-step FWER pcor=puncor/m
> sidak One-step FWER pcor=1-(1-puncor)^(1/m)
> (or Sidak, 1967)
> holm Step-down FWER Holm, 1979
> holland Step-down FWER Holland and Copenhaver, 1987
> liu1 Step-down FDR Benjamini and Liu, 1999a
> liu2 Step-down FDR Benjamini and Liu, 1999b
> hochberg Step-up FWER Hochberg, 1988
> rom Step-up FWER Rom, 1990
> simes Step-up FDR Benjamini and Hochberg, 1995 (or
> Benjamini and Yekutieli, 2001
> (first method))
> yekutieli Step-up FDR Benjamini and Yekutieli, 2001
> (second method)
> krieger Step-up FDR Benjamini, Krieger and Yekutieli, 2001
>
> So I tried liu1, liu2, simes, yekutieli, and krieger to see what difference it would make with a specified FDR of 0.05. The two liu's and yekutieli were about the same (4 rejections, critical P-value about 0.002. But the simes and krieger were completely different (simes: 14 rejections, critical p-vlaue = 0.026) and (krieger: 15 rejections, critical p-value = 0.051). The latter two look too good to be true, especially the Krieger, where the critical p-value is actually higher than the specified FDR rate.
>
> Anyone know what's going on here? Am I doing this correctly? What assumptions are there for Krieger, for example, that do not hold for the first three?
>
> Al Feiveson
>
> If anyone wants to try it - here's the data:
>
> h se z pv
> 1761.754 419.83 4.196352 .0000271
> .0613758 .0171379 3.58129 .0003419
> .0431283 .0134256 3.212402 .0013163
> .0503242 .0159218 3.160711 .0015738
> .0662939 .0223807 2.962102 .0030555
> .0388915 .0133944 2.903562 .0036894
> .0793423 .0274955 2.885645 .0039061
> .0353006 .0129654 2.722682 .0064754
> 868.2667 323.8542 2.681042 .0073393
> .0491057 .0184865 2.6563 .0079003
> 893.4875 341.7166 2.614703 .0089305
> .0310786 .0131919 2.355878 .018479
> .034885 .0150222 2.322223 .0202209
> .032349 .0144647 2.236412 .0253248
> .0302972 .0139816 2.166937 .0302396
> .0295493 .0165115 1.789618 .0735153
> -.0201654 .0129031 -1.562831 .1180923
> .0255772 .017776 1.438857 .1501909
> .0150236 .0122768 1.223738 .2210511
> .0187261 .0165989 1.128154 .2592548
> -.013579 .0127488 -1.065118 .2868224
> .0142208 .0142817 .9957331 .3193798
> .0099778 .0117025 .8526207 .3938697
> .0071485 .0095379 .7494824 .4535665
> .0130484 .021209 .6152284 .5384039
> -.0067718 .0137157 -.4937261 .6214996
> .0028293 .0090162 .3138045 .7536695
>
> . multproc ,method(liu1) pvalue(pv) puncor(.05)
>
> Method: liu1
> Uncorrected overall critical P-value: .05
> Number of P-values: 27
> Corrected overall critical P-value: .00262649
> Number of rejected P-values: 4
>
> . multproc ,method(liu2) pvalue(pv) puncor(.05)
>
> Method: liu2
> Uncorrected overall critical P-value: .05
> Number of P-values: 27
> Corrected overall critical P-value: .00255198
> Number of rejected P-values: 4
>
> . multproc ,method(simes) pvalue(pv) puncor(.05)
>
> Method: simes
> Uncorrected overall critical P-value: .05
> Number of P-values: 27
> Corrected overall critical P-value: .02592593
> Number of rejected P-values: 14
>
> . multproc ,method(yekutieli) pvalue(pv) puncor(.05)
>
> Method: yekutieli
> Uncorrected overall critical P-value: .05
> Number of P-values: 27
> Corrected overall critical P-value: .00190351
> Number of rejected P-values: 4
>
> . multproc ,method(krieger) pvalue(pv) puncor(.05)
>
> Method: krieger
> Uncorrected overall critical P-value: .05
> Number of P-values: 27
> Corrected overall critical P-value: .05102041
> Number of rejected P-values: 15
>
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
--
Roger B Newson BSc MSc DPhil
Lecturer in Medical Statistics
Respiratory Epidemiology and Public Health Group
National Heart and Lung Institute
Imperial College London
Royal Brompton Campus
Room 33, Emmanuel Kaye Building
1B Manresa Road
London SW3 6LR
UNITED KINGDOM
Tel: +44 (0)20 7352 8121 ext 3381
Fax: +44 (0)20 7351 8322
Email: [email protected]
Web page: http://www.imperial.ac.uk/nhli/r.newson/
Departmental Web page:
http://www1.imperial.ac.uk/medicine/about/divisions/nhli/respiration/popgenetics/reph/
Opinions expressed are those of the author, not of the institution.
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/