Re: st: Kappa weights & category detection

[James Kirkbride <jbk25@cam.ac.uk>]

Dear Ronan

Many thanks for your feedback. I agree with you entirely and we are in 
fact using collapsed categories of diagnostic groups which make clinical 
sense. This is part of a training program to ensure our psychiatrists 
(rating case notes using opcrit criteria) are doing roughly the same thing.

Thanks very much for your salient advice on how to proceed.

Part of my concern with kappa in stata is whether it acknowledges all 
the categories (including those not rated) -- it was unclear to me 
whether this was taken into account.

Thanks again

James



On 11/02/2011 5:43 AM, Ronan Conroy wrote:
> The main limitation is that of your data. The potential range of psychiatric diagnosis is large, and you can tap very little of this with a small study.
>
> That said, rather than weighting Kappa, which would produce an arbitrary scoring system, I would report my results under
>
> 1. Agreement on headline diagnosis
> Under this header, only clinically significant disagreements would count. So if you are using DSM-IV, if both raters diagnosed depressive disorder of any sort (any 369 code) the sub-classification would be ignored.
>
> 2. Agreement in detail
> Precise agreement on diagnosis
>
> Rather than subsuming all disagreement into a global statistic, it would be interesting to see what were the commonest disagreements (you would be surprised by substantial disagreement about obsessive compulsive disorder, but I think we would all expect some variation in mood disorders).
>
>
>
> Sometimes an overall statistic can obscure the interesting questions.
>
> r
>
>
> On 10 Feb 2011, at 20:25, James Kirkbride wrote:
>
> > Hello Stata Users
> >
> > I am trying to calculate some Kappa values for two raters who have each
> > rated the same 10 case notes to produce a mental health diagnosis. Their
> > diagnosis can be one of 16 possible categories. Obviously not all 16
> > categories are used, so I specify a weight matrix to let stata know
> > there are 16 possible categories that could have been chosen.
> >
> > However I am unsure stata has taken this into account as when I run the
> > analysis I get the same kappa statistic with or without the weight
> > matrix/absolute command specified (output below).
> >
> > Any help on this would be greatly appreciated. Is stata taking into
> > account the full possibility of all 16 categories?
> >
> > (I realise 10 is a small number of subjects, but this particular study
> > is resource limited).
> Ronán Conroy
> rconroy@rcsi.ie
> Associate Professor
> Division of Population Health Sciences
> Royal College of Surgeons in Ireland
> Beaux Lane House
> Dublin 2
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/

--

James Kirkbride Ph.D.

Sir Henry Wellcome Research Fellow
Department of Psychiatry
University of Cambridge
www.sepea.org
jbk25@cam.ac.uk
01223 336 798


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/