Re: st: Interrater Agreement and Kap Command

[Frank Gallo <policepsychology@gmail.com>]

Hi Morten,

Thank you for your response. Let me clarify my situation. The ratings
are interval-like. The data file is in the following format: columns
(raters, n = 94) and rows (variables, n = 49). The raters gave
responses that ranged from 1 to 9 on the variables. I am looking for
something similar to the ICC (model: two-way mixed, alpha; type:
consistency) available in SPSS. I hope this helps.

Best,
Frank


On Tue, Jul 13, 2010 at 5:53 PM, Morten Hesse <mh@crf.au.dk> wrote:
> Dear Frank
> You wouldn't want to use the ICC if your values are not interval scale. What
> I know of SPSS that would be the ICC for an interval scale. There is a
> similar statistic for logistic regression (the rho statistic), but in my
> experience, in cases where kappa values suggest moderate agreement between
> any two observations, the rho statistic derived from xtlogit or MLE logistic
> regression in other software will show very high values of agreement, and I
> do not believe that they represent reliability estimates that are realistic,
> although I do not have the expertise to say so (any thoughts, anyone else).
> Therefore, you would want to use Fleiss's kappa. I am not sure if that or a
> similar statistic is available for Stata.
> Percent agreement should be fairly easy to estimate using an excell
> spreadsheet.
> One question: If you say 49 variables, would that refer to different
> outcomes pertaining to the same units of observation?
> Greetings
> Morten
>
> Frank Gallo skrev:
>>
>> Dear All:
>>
>> I am a beginner with Stata, and I prefer to use Stata instead of SPSS
>> to address the following situation. I want to evaluate interrater
>> agreement. I have more than two raters (94), more than two ratings (49
>> variables), and varying number of raters (some missing values on the
>> variables). The data are in the form to use the kap command, however,
>> there is no statistic for testing kappa > 0 and no results reported
>> regarding the percentage of agreement. Is there some alternative that
>> I can use in Stata to obtain these results? I looked through the
>> archives and could not find one. SPSS offers the ICC option. I
>> appreciate any thoughts.
>>
>> Best,
>> Frank
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/