Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.

# Re: st: reliability with -icc- and -estat icc-

 From "JVerkuilen (Gmail)" To statalist@hsphsun2.harvard.edu Subject Re: st: reliability with -icc- and -estat icc- Date Tue, 26 Feb 2013 20:40:49 -0500

On Tue, Feb 26, 2013 at 8:31 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
> Yes. I want to know how consistent the raters are in their scoring
> and/or ranking.
> The Applications are Fixed Effects.  The raters are Random Effects.
>
> Any help would be appreciated.
>
> I have a colleague who works in SAS and did proc corr alpha.  I'm not
> sure if that is the correct way to do it, and I'm not sure that method
> is possible in STATA.

It's absolutely possible. I just ran the following model, which I
believe (but am not 100% sure) is what you want. This has a random
intercept for Rator and fixed effects for application. The ICC is
massively inflated by rater 4, who is clearly anchored very
differently and has a massively lower response variance. HUGE outlier.
If you'd be willing I'd love to use it as an example for my Bayesian
ICC estimator paper.

. xtmixed Score i.Application if Rator != 4, || Rator:,
covariance(independent)  difficult

Mixed-effects ML regression                     Number of obs      =        33
Group variable: Rator                           Number of groups   =         3

Obs per group: min =        11
avg =      11.0
max =        11

Wald chi2(10)      =     95.51
Log likelihood = -77.750467                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
Score |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
Application |
2  |  -7.333333   1.954361    -3.75   0.000    -11.16381   -3.502855
3  |         -2   1.954361    -1.02   0.306    -5.830478    1.830478
4  |         -2   1.954361    -1.02   0.306    -5.830478    1.830478
5  |        -12   1.954361    -6.14   0.000    -15.83048   -8.169522
6  |  -9.333333   1.954361    -4.78   0.000    -13.16381   -5.502855
7  |  -9.333333   1.954361    -4.78   0.000    -13.16381   -5.502855
8  |         -4   1.954361    -2.05   0.041    -7.830478   -.1695221
9  |         -6   1.954361    -3.07   0.002    -9.830478   -2.169522
10  |          1   1.954361     0.51   0.609    -2.830478    4.830478
11  |  -8.333333   1.954361    -4.26   0.000    -12.16381   -4.502855
|
_cons |         14   1.565624     8.94   0.000     10.93143    17.06857
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
Rator: Identity              |
sd(_cons) |   1.274458   .6891609      .4416111    3.677996
-----------------------------+------------------------------------------------
sd(Residual) |   2.393594   .3090117      1.858493    3.082763
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =     3.99 Prob >= chibar2 = 0.0229

. estat icc

Residual intraclass correlation

------------------------------------------------------------------------------
Level |        ICC   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
Rator |   .2208793   .1946277      .0299686    .7223361
------------------------------------------------------------------------------
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/