Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.

# Re: st: Bootstrap sampling for evaluating hypothesis tests

 From Maarten Buis To statalist@hsphsun2.harvard.edu Subject Re: st: Bootstrap sampling for evaluating hypothesis tests Date Thu, 14 Mar 2013 11:28:19 +0100

```On Wed, Mar 13, 2013 at 4:04 PM, Margaret MacDougall wrote:
> I would value receiving recommendations on literature explaining the
> application of bootstrap sampling to assess robustness to Type I errors of a
> proposed new hypothesis test. Better still, if the recommended references
> contain corresponding computer syntax!

In terms of literature references, I would look at bootstrap tests. A
bootstrap changes the data such that the null hypothesis is true and
looks at the proportion of replictions in which the test statistic is
more extreme than the one observed in the original data. In bootstrap
tests these can be used as an estimate of the p-value(*), but you can
compare it with the asymptotic p-value returned by your tests and see
if they correspond. It is useful to also consider the Monte Carlo
confidence interval, which captures the variability you can expect in
the proportion due to the fact that it is based on a random process.
Say you find 1000 out of 20000 replications in which the test
statistic was more extreme than the one in the original sample, than
the Monte Carlo confidence interval can be computed by typing in
Stata: -cii 20000 1000-

If you save the p-values from all replications you can look at the
distribution of the p-values, as I did in the examples I gave earlier.

Nice introductions to bootstrap tests can be found in Chapter 4 of
(Davison & Hinkley 1997) and Chapter 16 of (Efron & Tibshirani 1993).
They are both good introductory texts, but I found that they
complement one another well, so it is useful to look at both of them.

You can also find more Stata code examples in the manual entry of
-bootstrap-: Under "Remarks" go to the section titled "Achieved
significance level", and it will give an example of how to use
-bootstrap- to do a bootstrap test.

Hope this helps,
Maarten

A.C. Davison and D.V. Hinkley (1997) Bootstrap Methods and their
Applications. Cambridge: Cambridge University Press.

B. Efron and R.J. Tibshirani (1993) An Introduction to the Bootstrap.
Boca Raton: Chapman & Hall/CRC

(*) Alternatively, for testing purposes it makes sense to use the (
number of replictions in which the test statistic is more extreme than
the one observed in the original data + 1 ) / ( the number of
replications +1 ), see:  Chapter 4 of (Davison & Hinkley 1997). Though
for a large number of replications the difference with the simple
proportion is trivial.

---------------------------------
Maarten L. Buis
WZB
Reichpietschufer 50
10785 Berlin
Germany

http://www.maartenbuis.nl
---------------------------------
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/
```