Re: st: reliability with ordinal data-Kendall's w?

[Morten Hesse <mh@crf.au.dk>]

I believe this one of those instances where your field may have its 
conventions, and that is what you should go by.
In my field, the statistic of choice would be the weighted kappa.
The Bland-Altman plot would not be very informative with four steps on 
the scale. In my view, the meaningful thing would be to analyze the 
kappa statistic for each category. That could give you an indication of 
whether the high or low values are more or less reliable, which is one 
thing that the BA-plot does. The precision of agreement (another nice 
property of the BA-plot) is really less meaningful with 4 levels. You  
cannot deviate more than 3 levels, and a 4*4 table of all ratings would 
be much more informative. (basically: tab r1 r2, where r1 is the first 
column containing ratings, and r2 is the second).
Hope this is useful
Morten


Den 13-12-2010 21:01, Ploutz-Snyder, Robert (JSC-SK)[USRA] skrev:
> I need to perform a reliability analysis among two raters who rated the same observations independently.  Observations are rated on an ordinal scale, taking 4 distinct values in increasing order.  Neither rater is assumed the "gold standard."  There are a LOT of ties in my dataset (a good thing in terms of rater reliability).
>
> I came across some web hits about Kendall's W, which is different from the more familiar tau.  Can anyone attest to whether this is the best way to go about a reliability analysis with ordinal data?   Is there a way to implement on Stata?? How does W differ from tau, and Somer's d??
>
> Are there other/better methods that I should be considering too?
>
> Also--for those knowledgeable about reliability analyses... I think that eventually I would like to graph these data with something akin to the Bland-Altman plots.  Do you take issue with that approach given that the data are not continuous?  Are the BA Limits of Agreement valid in this situation??
>
> Rob
>
>
>
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/