Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: RE: inter and intra rater


From   Stephen McKay <S.McKay@bristol.ac.uk>
To   statalist@hsphsun2.harvard.edu
Subject   st: RE: inter and intra rater
Date   Tue, 23 Sep 2003 10:36:47 +0100

Emma

You should take a look at 
http://www.stata.com/support/faqs/stat/anova2.html

The problem may lie with your data structure.  Consider using a long 
rather than wide format, with subjects forming the rows.  Each row 
should be something like

subjectid judgeid occasionid score

i.e some rehape-ing may be helpful.
Then score is the appropriate outcome variable for the anova.
You may also find the contributed command wsanova (within-subjects 
anova) easier to comprehend than anova, though I don't think it gives 
icc by default.

Hope some of this helps!

Steve

Date: Mon, 22 Sep 2003 16:47:54 +0100
From: Emma <rmhaemh@ucl.ac.uk>
Subject: Re: st: RE: inter and intra rater

Dear Nick, (and statalist)

Many thanks for your prompt response to my initial question.

Earlier I sent the question below to statalist, but somewhat 
confusingly I 
only sent a subset of the data for the small disease group, below is 
the 
data for the entire set. where rater is the individual who took the 
measure, 6 individuals, subject 1 had small disease, subject 2 had 
moderate 
disease and subject 3 had large disease. and repeats 1 and 2 were the 
two 
repeated measurements taken by the raters.

ratersubject 1subject 1subject 2subject 2subject 3subject 3
repeat1repeat2repeat1repeat2repeat1repeat2
JT22.522.535.5356867.5
KD22.522.536.535.567.568.5
WD22.522.536.536.56768
NC222335.5366767
RP22.52236.536.56868.5
ES21.52336356867.5
I wondered if anyone has any suggestions to look at the three subjects 
together, whilst checking for inter rater and intra rater reliability,

Kindest regards and thanks,
Emma

At 16:18 22/09/03 +0100, you wrote:
>Emma
>
> > I wonder if someone could help me with the following, I am
> > looking to
> > calculate intra rater and inter rater correlations on the
> > following data
> > set. A continuous variable has been measured by 6 trainees on three
> > different subjects (the three subjects have differing
> > degrees of disease:
> > small,  moderate and large). All 6 trainees repeated the
> > measurements on
> > the three subjects on two separate occasions.
> >
> > I have applied ICC as a measure for the agreement between
> > the 6 raters;
> > however STATA outputs 0* for the ICC whereas the data
> > appears to show a
> > reasonable agreement in measures between raters:
> >
> > rater   time1          time2
> > JT    22.5    22.5
> > KD    22.5    22.5
> > WD    22.5    22.5
> > NC    22      23
> > RP    22.5    22
> > ES       21.5            23
> >
> >
> > . loneway var1 rater
> >
> >                  One-way Analysis of Variance for var1: measure2
> >
> >                                                Number of
> > obs =        12
>
> >        R-squared =    0.0870
> >
> >      Source                SS         df      MS
> > F     Prob > F
> > ------------------------------------------------------------
> > -------------
> > Between rater          .16666667      5    .03333333
> > 0.11     0.9845
> > Within rater                1.75      6    .29166667
> > ------------------------------------------------------------
> > -------------
> > Total                  1.9166667     11    .17424242
> >
> >           Intraclass       Asy.
> >           correlation      S.E.       [95% Conf. Interval]
> >           ------------------------------------------------
> >              0.00000*    0.42817       0.00000     0.83921
> >
> >           Estimated SD of rater effect                   .
> >           Estimated SD within rater               .5400617
> >           Est. reliability of a rater mean         0.00000*
> >                (evaluated at n=2.00)
> >
> > (*) Truncated at zero.
> >
> >
> > Also I wondered what measure I should use to consider the
> > agreement between
> > the repeat measures (should this be ICC also, and if so how
> > should the data
> > be set up, apologies to ask a basic question). Finally, is
> > there anyway to
> > consider the three subjects data together, i.e to combine
> > the info for the
> > small,  moderate and large.
>
>One comment only, as I am mostly in the dark
>here: it is not clear to me how the subset of data
>you give is related to the analysis you
>report or to the problem you describe.
>
>Nevertheless, focusing on that subset alone,
>although the "agreement" between raters is close
>in the sense that all ratings are 21.5-23, which presumably
>is some small fraction of the possible range, the
>correlation (classic sense) is nevertheless
>strong and _negative_.
>
>However, another way of thinking about it is that
>your data points may collectively be one big blob.
>
>I guess wildly that this is consistent
>with small ICC.
>
>Nick
>n.j.cox@durham.ac.uk

----------------------
Stephen McKay
PFRC, School of Geographical Sciences

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2021 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index