Nick Cox said:
"I'm still puzzled by the over-arching question and curious
as to how this fits into any research project. ... Why is the state mean
vs other states mean comparison the focus here? ... is this
really what other researchers want to know, given the
many other possibilities such as confidence intervals,
graphs, tables?"
My comments are well beyond a Stata question, so if this is the wrong
forum, please forgive and ignore what follows.
We provide the state of Colorado with evaluation reports regarding
programs to prevent and reduce tobacco use. Our audience isn't
researchers but public and government data consumers. Sometimes we
analyze population-level data collected at the U.S. national level
through state-stratified designs, which support calculation of both
state and national estimates. Other times, we analyze Colorado-level
data collected under complex-sample designs that allow for estimation at
sub-state levels.
In most cases, we are asked to present results comparatively, i.e.,
between state and rest of nation, or sub-state area and rest-of-state.
We typically declare (non)significance using design-adjusted hypothesis
tests of two rates, although we don't necessarily include p-values. So
we do indeed focus on comparing means (proportions) by
single-state-of-interest vs. rest-of-states, because it's exactly what
we are asked to do (but not by other researchers).
A second but related issue: We're thinking of dropping
hypothesis-testing and relying instead on non-overlapping confidence
bounds as the criterion for statistical difference. I think (but am not
sure) that this change would be neutral or conservative in all instances
(i.e., that non-overlap is always at least as stringent as appropriate
hypothesis tests).
My questions:
1. Does this context for focusing on mean comparisons by
state-vs-other-states seem reasonable?
2. Are non-overlapping confidence intervals always more conservative
than appropriate hypothesis tests?
3. What are the implications of switching to CIs for
difference-determination (other than potential "customer" disappointment
that fewer comparisons appear significantly different)?
Thx...arnold
Arnold H. Levinson, Ph.D.
Director, Tobacco Program Evaluation Group (TPEG)
Assistant Professor of Preventive Medicine
University of Colorado at Denver & Health Sciences Center
AMC Cancer Research Center
1600 Pierce Street, Lakewood CO 80214
[email protected]
303-239-3402
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
