Statalist The Stata Listserver


[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: RE: Re: Statistical significance vs the 95% confidence intervals -- how should i interpret these


From   "Jason Yackee" <[email protected]>
To   <[email protected]>
Subject   st: RE: Re: Statistical significance vs the 95% confidence intervals -- how should i interpret these
Date   Fri, 8 Dec 2006 11:37:49 -0800

As a follow-up, because I am dealing with this issue in my own work,
I have seen it said a number of times, in published articles and in
online "how to" guides, that when the confidence interval spans the null
value that this indicates statistical insignificance.  Here is an
example from a website:

"When you see a confidence interval in a published medical report, you
should look for two things. First, does the interval contain a value
that implies no change or no effect? For example, with a confidence
interval for a difference look to see whether that interval includes
zero. With a confidence interval for a ratio, look to see whether that
interval contains one. The interval shown below implies no statistically
significant change."

The graphic shows a confidence interval spanning the zero point.

So my question is: when we have this case -- a confidence interval
spanning the zero point -- is it both technically correct and not unduly
misleading to the average reader to say that the effect is not
"statistically significant"?

Jason

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Michael
Blasnik
Sent: Friday, December 08, 2006 9:54 AM
To: [email protected]
Subject: st: Re: Statistical significance vs the 95% confidence
intervals -- how should i interpret these

Neither of your approaches is correct.  In your first approach, you are 
being overly conservative by just checking whether confidence intervals 
overlap (but you aren't very far off).  In your second approach,
-ttesti- is 
looking for standard deviations, not standard errors.  Look at the
output 
and you will see how ttest has created a new standard error that is much

smaller (approxiately s.d./sqrt(n)).

Your question would be best answered by using whatever estimation method
you 
used to come up with the estimates and standard errors and then use 
Stata's -test- or -lincom- commands to test hypotheses that properly
account 
for potential covariance.  If you can't do that for some reason, then
you 
would need to make an assumption about the covariance.  If you assume
that 
the two estimates are independent, then you can calculate a t-statistic
on 
the difference based on the difference in the estimates and the standard

error of this difference:

di (.484721-.4556235)/sqrt(.015637^2+.0138994^2)
1.3907943

So t=1.39.  You can get a p-value from:

di ttail(2398,(.484721-.4556235)/sqrt(.015637^2+.0138994^2))
.08220843

(or use whatever d.f. are appropriate if 2398 is not, but it won't make
any 
real difference here)

So you could conclude that, if the two estimates are independent, then
the 
difference between year1 and year3 is not quite statistically
significant at 
the .05 level.


Michael Blasnik

----- Original Message ----- 
From: "Columbia & Belmont Apt" <[email protected]>
To: <[email protected]>
Sent: Friday, December 08, 2006 12:25 PM
Subject: st: Statistical significance vs the 95% confidence intervals --
how 
should i interpret these


> Dear Statalist Colleagues,
> This might be a simple question but I cannot reconcile the following 
> results.
>
> The question is "Are the estimates for year 1 and 3 statistically 
> different?
> year    estimate    st.error        95% conf interval
> year1    .484721     .015637     .454068    .5153741
> year2   .4128893    .0145616    .3843443    .4414343
> year3   .4556235    .0138994    .4283765    .4828704
>
> 1. Using the 95% confidence interval that I obtained:
>
> Year 1 and year 3 confidence intervals overlap thus year 1&3 estimates
are 
> not statistically different at the 95% level.
>
> 2. Using ttesti they are statistically different with 99% confidence:
>
> . ttesti 2398 .484721  .015637 2399 .4556235 .0138994
> Two-sample t test with equal variances
>
------------------------------------------------------------------------
------
>         |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. 
> Interval]
>
---------+--------------------------------------------------------------
------
>       x |    2398     .484721    .0003193     .015637    .4840948 
> .4853472
>       y |    2399    .4556235    .0002838    .0138994     .455067 
> .45618
>
---------+--------------------------------------------------------------
------
> combined |    4797    .4701692    .0002996    .0207488    .4695819 
> .4707565
>
---------+--------------------------------------------------------------
------
>    diff |            .0290975    .0004272                  .02826 
> .029935
>
------------------------------------------------------------------------
------
> Degrees of freedom: 4795
>                      Ho: mean(x) - mean(y) = diff = 0
>     Ha: diff < 0               Ha: diff != 0              Ha: diff > 0
>       t =  68.1143                t =  68.1143              t =
68.1143
>   P < t =   1.0000          P > |t| =   0.0000          P > t =
0.0000
>
>
> Which one is the correct? What am I not interpreting right?

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2024 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index