Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: reliability with ordinal data-Kendall's w?


From   Morten Hesse <mh@crf.au.dk>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: reliability with ordinal data-Kendall's w?
Date   Tue, 14 Dec 2010 10:51:01 +0100

I believe this one of those instances where your field may have its conventions, and that is what you should go by.
In my field, the statistic of choice would be the weighted kappa.
The Bland-Altman plot would not be very informative with four steps on the scale. In my view, the meaningful thing would be to analyze the kappa statistic for each category. That could give you an indication of whether the high or low values are more or less reliable, which is one thing that the BA-plot does. The precision of agreement (another nice property of the BA-plot) is really less meaningful with 4 levels. You cannot deviate more than 3 levels, and a 4*4 table of all ratings would be much more informative. (basically: tab r1 r2, where r1 is the first column containing ratings, and r2 is the second).
Hope this is useful
Morten


Den 13-12-2010 21:01, Ploutz-Snyder, Robert (JSC-SK)[USRA] skrev:
I need to perform a reliability analysis among two raters who rated the same observations independently.  Observations are rated on an ordinal scale, taking 4 distinct values in increasing order.  Neither rater is assumed the "gold standard."  There are a LOT of ties in my dataset (a good thing in terms of rater reliability).

I came across some web hits about Kendall's W, which is different from the more familiar tau.  Can anyone attest to whether this is the best way to go about a reliability analysis with ordinal data?   Is there a way to implement on Stata?? How does W differ from tau, and Somer's d??

Are there other/better methods that I should be considering too?

Also--for those knowledgeable about reliability analyses... I think that eventually I would like to graph these data with something akin to the Bland-Altman plots.  Do you take issue with that approach given that the data are not continuous?  Are the BA Limits of Agreement valid in this situation??

Rob





*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index