Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science


From   "Allan Reese (Cefas)" <allan.reese@cefas.co.uk>
To   <statalist@hsphsun2.harvard.edu>
Subject   RE: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science
Date   Thu, 28 Feb 2013 10:42:35 -0000

Lenny Lesser posed a problem (copied below) on Statalist and has
received several replies to the question posed on ICC method.  Nick Cox
commented "A scatter plot matrix is instructive" and in a follow-up
message added, "it is well to know what patterns do or do not exist
before you start quantifying them.  This is just to underline that
repeating the graphs with ranks underlines how much information is
thereby discarded."  There have been various comments on methods and
Stata commands, including JVerkuilen's, "my gut impression is that you
really should use the scores. The context you cite is no doubt correct,
but for comparing raters with each other the scores they gave are
essential."

With no disrespect to anyone, I see this as a classic example of
"Mathsworld" - a mindset where because you are presented with numbers
you do sums, and the context is ignored.  LOOK AT THE DATA.

In the first place, the score values are not skewed; they are just all
low range.  The highest score is 18/100, adjustable to A* in GCSE but
indicating to me a general problem with the exercise.  The "outlier"
rater 4 has awarded values ONLY of 0/1/2/3, from which I'd guess she has
ranked the apps anyway and never awarded scores. 

. table Score Rater

----------------------------------
          |         Rater         
    Score |    1     2     3     4
----------+-----------------------
        0 |                      1
        1 |          1           8
        2 |    1     1     1     1
        3 |          2           1
        5 |    1     1     1      
        6 |    2           3      
        7 |    2     1     1      
        8 |          1            
        9 |          1            
       10 |          1            
       11 |    1           1      
       12 |          2            
       13 |    1           1      
       15 |                1      
       16 |    1           1      
       17 |    1           1      
       18 |    1                  
----------------------------------

Perhaps everyone noticed that, and has treated this as purely an
exercise in modelling?  However, it is my impression that people
sometimes love their models more than the data they "explain". 

Allan


------------------------------------------------------------------------
--------


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index