Re: st: RE: inter and intra rater

 From Emma To statalist@hsphsun2.harvard.edu Subject Re: st: RE: inter and intra rater Date Mon, 22 Sep 2003 16:47:54 +0100

Dear Nick, (and statalist)

Many thanks for your prompt response to my initial question.

Earlier I sent the question below to statalist, but somewhat confusingly I only sent a subset of the data for the small disease group, below is the data for the entire set. where rater is the individual who took the measure, 6 individuals, subject 1 had small disease, subject 2 had moderate disease and subject 3 had large disease. and repeats 1 and 2 were the two repeated measurements taken by the raters.

ratersubject 1subject 1subject 2subject 2subject 3subject 3
repeat1repeat2repeat1repeat2repeat1repeat2
JT22.522.535.5356867.5
KD22.522.536.535.567.568.5
WD22.522.536.536.56768
NC222335.5366767
RP22.52236.536.56868.5
ES21.52336356867.5
I wondered if anyone has any suggestions to look at the three subjects together, whilst checking for inter rater and intra rater reliability,

Kindest regards and thanks,
Emma

At 16:18 22/09/03 +0100, you wrote:

```Emma

> I wonder if someone could help me with the following, I am
> looking to
> calculate intra rater and inter rater correlations on the
> following data
> set. A continuous variable has been measured by 6 trainees on three
> different subjects (the three subjects have differing
> degrees of disease:
> small,  moderate and large). All 6 trainees repeated the
> measurements on
> the three subjects on two separate occasions.
>
> I have applied ICC as a measure for the agreement between
> the 6 raters;
> however STATA outputs 0* for the ICC whereas the data
> appears to show a
> reasonable agreement in measures between raters:
>
> rater   time1          time2
> JT    22.5    22.5
> KD    22.5    22.5
> WD    22.5    22.5
> NC    22      23
> RP    22.5    22
> ES       21.5            23
>
>
> . loneway var1 rater
>
>                  One-way Analysis of Variance for var1: measure2
>
>                                                Number of
> obs =        12

>        R-squared =    0.0870
>
>      Source                SS         df      MS
> F     Prob > F
> ------------------------------------------------------------
> -------------
> Between rater          .16666667      5    .03333333
> 0.11     0.9845
> Within rater                1.75      6    .29166667
> ------------------------------------------------------------
> -------------
> Total                  1.9166667     11    .17424242
>
>           Intraclass       Asy.
>           correlation      S.E.       [95% Conf. Interval]
>           ------------------------------------------------
>              0.00000*    0.42817       0.00000     0.83921
>
>           Estimated SD of rater effect                   .
>           Estimated SD within rater               .5400617
>           Est. reliability of a rater mean         0.00000*
>                (evaluated at n=2.00)
>
> (*) Truncated at zero.
>
>
> Also I wondered what measure I should use to consider the
> agreement between
> the repeat measures (should this be ICC also, and if so how
> should the data
> be set up, apologies to ask a basic question). Finally, is
> there anyway to
> consider the three subjects data together, i.e to combine
> the info for the
> small,  moderate and large.

One comment only, as I am mostly in the dark
here: it is not clear to me how the subset of data
you give is related to the analysis you
report or to the problem you describe.

Nevertheless, focusing on that subset alone,
although the "agreement" between raters is close
in the sense that all ratings are 21.5-23, which presumably
is some small fraction of the possible range, the
correlation (classic sense) is nevertheless
strong and _negative_.

However, another way of thinking about it is that
your data points may collectively be one big blob.

I guess wildly that this is consistent
with small ICC.

Nick
n.j.cox@durham.ac.uk

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```
```
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```