The -oneway- program gives you the option of doing Scheffe, Bonferroni and
Sidak tests. Some other commands offer similar options, e.g. -test-. Is
there any consensus as to which is best? Are there any situations in which
one is clearly preferable to another? My experience has been that results
tend to be similar, and the examples from the Stata reference guide also
show little difference.
Also, as a practical matter, how often do these tests get done in
practice? I see them rarely in the stuff I read and no reviewer has ever
asked me to provide this info. I understand the rationale for these tests,
but if you take that rationale to its logical conclusion it seems like
every coefficient in every model should have its p-value adjusted to
reflect the fact that multiple tests are going on. The tests also seem
like they can be excessively conservative -- all differences between pairs
of groups could be significant at the .05 level, but after making one of
these adjustments none of the differences could be significant.
