Re: st: reliability with -icc- and -estat icc-

["JVerkuilen (Gmail)" <jvverkuilen@gmail.com>]

On Tue, Feb 26, 2013 at 8:31 PM, Lenny Lesser <lenny3200@gmail.com> wrote:
> Yes. I want to know how consistent the raters are in their scoring
> and/or ranking.
> The Applications are Fixed Effects.  The raters are Random Effects.
>
> Any help would be appreciated.
>
> I have a colleague who works in SAS and did proc corr alpha.  I'm not
> sure if that is the correct way to do it, and I'm not sure that method
> is possible in STATA.

It's absolutely possible. I just ran the following model, which I
believe (but am not 100% sure) is what you want. This has a random
intercept for Rator and fixed effects for application. The ICC is
massively inflated by rater 4, who is clearly anchored very
differently and has a massively lower response variance. HUGE outlier.
If you'd be willing I'd love to use it as an example for my Bayesian
ICC estimator paper.



. xtmixed Score i.Application if Rator != 4, || Rator:,
covariance(independent)  difficult

Mixed-effects ML regression                     Number of obs      =        33
Group variable: Rator                           Number of groups   =         3

                                                Obs per group: min =        11
                                                               avg =      11.0
                                                               max =        11


                                                Wald chi2(10)      =     95.51
Log likelihood = -77.750467                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
       Score |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
 Application |
          2  |  -7.333333   1.954361    -3.75   0.000    -11.16381   -3.502855
          3  |         -2   1.954361    -1.02   0.306    -5.830478    1.830478
          4  |         -2   1.954361    -1.02   0.306    -5.830478    1.830478
          5  |        -12   1.954361    -6.14   0.000    -15.83048   -8.169522
          6  |  -9.333333   1.954361    -4.78   0.000    -13.16381   -5.502855
          7  |  -9.333333   1.954361    -4.78   0.000    -13.16381   -5.502855
          8  |         -4   1.954361    -2.05   0.041    -7.830478   -.1695221
          9  |         -6   1.954361    -3.07   0.002    -9.830478   -2.169522
         10  |          1   1.954361     0.51   0.609    -2.830478    4.830478
         11  |  -8.333333   1.954361    -4.26   0.000    -12.16381   -4.502855
             |
       _cons |         14   1.565624     8.94   0.000     10.93143    17.06857
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
Rator: Identity              |
                   sd(_cons) |   1.274458   .6891609      .4416111    3.677996
-----------------------------+------------------------------------------------
                sd(Residual) |   2.393594   .3090117      1.858493    3.082763
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =     3.99 Prob >= chibar2 = 0.0229

. estat icc

Residual intraclass correlation

------------------------------------------------------------------------------
                       Level |        ICC   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
                       Rator |   .2208793   .1946277      .0299686    .7223361
------------------------------------------------------------------------------
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/