Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Bootstrap sampling for evaluating hypothesis tests

From   Margaret MacDougall <>
Subject   Re: st: Bootstrap sampling for evaluating hypothesis tests
Date   Sat, 16 Mar 2013 08:21:29 +0000

Dear Maarten

Thanks for so kindly offering such a comprehensive reply. I look forward to exploring your suggestions.

Best wishes


Dr Margaret MacDougall
Medical Statistician and Researcher in Education
Centre for Population Health Sciences
University of Edinburgh Medical School
Teviot Place
Edinburgh EH8 9AG

Tel:  +44 (0)131 650 3211
Fax:  +44 (0)131 650 6909

On 14/03/2013 10:28, Maarten Buis wrote:
On Wed, Mar 13, 2013 at 4:04 PM, Margaret MacDougall wrote:
I would value receiving recommendations on literature explaining the
application of bootstrap sampling to assess robustness to Type I errors of a
proposed new hypothesis test. Better still, if the recommended references
contain corresponding computer syntax!
In terms of literature references, I would look at bootstrap tests. A
bootstrap changes the data such that the null hypothesis is true and
looks at the proportion of replictions in which the test statistic is
more extreme than the one observed in the original data. In bootstrap
tests these can be used as an estimate of the p-value(*), but you can
compare it with the asymptotic p-value returned by your tests and see
if they correspond. It is useful to also consider the Monte Carlo
confidence interval, which captures the variability you can expect in
the proportion due to the fact that it is based on a random process.
Say you find 1000 out of 20000 replications in which the test
statistic was more extreme than the one in the original sample, than
the Monte Carlo confidence interval can be computed by typing in
Stata: -cii 20000 1000-

If you save the p-values from all replications you can look at the
distribution of the p-values, as I did in the examples I gave earlier.

Nice introductions to bootstrap tests can be found in Chapter 4 of
(Davison & Hinkley 1997) and Chapter 16 of (Efron & Tibshirani 1993).
They are both good introductory texts, but I found that they
complement one another well, so it is useful to look at both of them.

You can also find more Stata code examples in the manual entry of
-bootstrap-: Under "Remarks" go to the section titled "Achieved
significance level", and it will give an example of how to use
-bootstrap- to do a bootstrap test.

Hope this helps,

A.C. Davison and D.V. Hinkley (1997) Bootstrap Methods and their
Applications. Cambridge: Cambridge University Press.

B. Efron and R.J. Tibshirani (1993) An Introduction to the Bootstrap.
Boca Raton: Chapman & Hall/CRC

(*) Alternatively, for testing purposes it makes sense to use the (
number of replictions in which the test statistic is more extreme than
the one observed in the original data + 1 ) / ( the number of
replications +1 ), see:  Chapter 4 of (Davison & Hinkley 1997). Though
for a large number of replications the difference with the simple
proportion is trivial.

Maarten L. Buis
Reichpietschufer 50
10785 Berlin
*   For searches and help try:

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

*   For searches and help try:

© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index