Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.

# RE: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science

 From "Allan Reese (Cefas)" To Subject RE: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science Date Thu, 28 Feb 2013 10:42:35 -0000

```Lenny Lesser posed a problem (copied below) on Statalist and has
received several replies to the question posed on ICC method.  Nick Cox
commented "A scatter plot matrix is instructive" and in a follow-up
message added, "it is well to know what patterns do or do not exist
before you start quantifying them.  This is just to underline that
repeating the graphs with ranks underlines how much information is
Stata commands, including JVerkuilen's, "my gut impression is that you
really should use the scores. The context you cite is no doubt correct,
but for comparing raters with each other the scores they gave are
essential."

With no disrespect to anyone, I see this as a classic example of
"Mathsworld" - a mindset where because you are presented with numbers
you do sums, and the context is ignored.  LOOK AT THE DATA.

In the first place, the score values are not skewed; they are just all
low range.  The highest score is 18/100, adjustable to A* in GCSE but
indicating to me a general problem with the exercise.  The "outlier"
rater 4 has awarded values ONLY of 0/1/2/3, from which I'd guess she has
ranked the apps anyway and never awarded scores.

. table Score Rater

----------------------------------
|         Rater
Score |    1     2     3     4
----------+-----------------------
0 |                      1
1 |          1           8
2 |    1     1     1     1
3 |          2           1
5 |    1     1     1
6 |    2           3
7 |    2     1     1
8 |          1
9 |          1
10 |          1
11 |    1           1
12 |          2
13 |    1           1
15 |                1
16 |    1           1
17 |    1           1
18 |    1
----------------------------------

Perhaps everyone noticed that, and has treated this as purely an
exercise in modelling?  However, it is my impression that people
sometimes love their models more than the data they "explain".

Allan

------------------------------------------------------------------------
--------
```