[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

From |
Roger Newson <[email protected]> |

To |
[email protected] |

Subject |
RE: st: logarithmic scales |

Date |
Mon, 01 Dec 2003 16:42:36 +0000 |

At 15:31 01/12/03 +0000, Nick Cox wrote:

A good discussion of this issue is given in Subsection 35.7 of Kirkwood and Sterne (2003), which is a basic text aimed mostly at non-mathematicians. This uses a Bayesian heuristic, based on the well-known result that the posterior odds between 2 hypotheses after the data analysis is equal to the prior odds between the same 2 hypotheses multiplied by the likelihood ratio between the 2 hypotheses. It is argued that a P-value below 0.003 is good enough for most of the people most of the time, because, *if* the prior odds are as bad as 100:1 against a nonzero population difference, *and* the power to detect a difference significant at P<=0.001 is as low as 0.5, *then* the posterior odds in favour of a nonzero population difference, given a P-value <=0.001, will be 5:1 in favour.I'd assert, perhaps very rashly, that beyond some threshold, very low P-values are practically indistinguishable. I suppose that log P-value of -20 is often appealing as a kind of thermonuclear demolition of a null hypothesis, but I wonder if anyone would think differently of (say) -6. Also, as is well known, the further you go out into the tail the more you depend on everything being as it be (model assumptions, data without measurement error, numerical analysis...). On the other hand, there are situations in which an overwhelming P-value is needed for any ensuing decision.

This heuristic seems to make sense to me, if the P-value is for the parameter of prior interest in the study design protocol, because not many grant-awarding bodies will pay for a study for which they consider the prior odds of an interesting difference to be worse than 100:1 against. On the other hand, in the real world, with today's technology, it is nearly always cheaper to torture the data until they confess than to collect more data. Therefore, a lot of people's colleagues expect them to do "subset analyses from hell", and are reluctant to write up negative results as such. Therefore, an honest scientist who wants to accumulate publications is often not a data miner, but a "data lawyer", cross-examoining the data on the moral equivalent of a "no-win no-fee" contract. Under these conditions, a lot of statistically-minded scientists will forget what they learned at college, and do what they are told, and torture the data. If the P-value is from one of a sequence of subset analyses, and is undertaken posterior to a main analysis which found nothing, then, arguably, the "prior odds" against an interesting difference might reasonably be worse than 100:1 against.

Roger

References

Kirkwood BR, Sterne JAC. Essential medical statistics. Second edition. Oxford, UK: Blackwell Science; 2003.

--

Roger Newson

Lecturer in Medical Statistics

Department of Public Health Sciences

King's College London

5th Floor, Capital House

42 Weston Street

London SE1 3QD

United Kingdom

Tel: 020 7848 6648 International +44 20 7848 6648

Fax: 020 7848 6620 International +44 20 7848 6620

or 020 7848 6605 International +44 20 7848 6605

Email: [email protected]

Website: http://www.kcl-phs.org.uk/rogernewson

Opinions expressed are those of the author, not the institution.

*

* For searches and help try:

* http://www.stata.com/support/faqs/res/findit.html

* http://www.stata.com/support/statalist/faq

* http://www.ats.ucla.edu/stat/stata/

**References**:**Re: st: logarithmic scales***From:*Roger Newson <[email protected]>

**RE: st: logarithmic scales***From:*"Nick Cox" <[email protected]>

- Prev by Date:
**Re: st: Framingham equation** - Next by Date:
**st: Re: RE: What does contribution to chi-square mean in tabchii** - Previous by thread:
**RE: st: logarithmic scales** - Next by thread:
**RE: st: logarithmic scales** - Index(es):

© Copyright 1996–2024 StataCorp LLC | Terms of use | Privacy | Contact us | What's new | Site index |