Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down on April 23, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
Nick Cox <njcoxstata@gmail.com> |

To |
statalist@hsphsun2.harvard.edu |

Subject |
Re: st: reliability with -icc- and -estat icc- |

Date |
Wed, 27 Feb 2013 01:39:41 +0000 |

There is an -alpha- command in Stata [NB] and -search alpha- would have pointed you to it. It strikes me that there is little or no point in any calculation that includes rater 4. Nick On Wed, Feb 27, 2013 at 1:31 AM, Lenny Lesser <lenny3200@gmail.com> wrote: > Yes. I want to know how consistent the raters are in their scoring > and/or ranking. > The Applications are Fixed Effects. The raters are Random Effects. > > Any help would be appreciated. > > I have a colleague who works in SAS and did proc corr alpha. I'm not > sure if that is the correct way to do it, and I'm not sure that method > is possible in STATA. > > On Tue, Feb 26, 2013 at 12:15 PM, JVerkuilen (Gmail) > <jvverkuilen@gmail.com> wrote: >> Just to be clear, you'd like to know the reliability (ICC) of the >> raters using the Applications as fixed effects? So basically you have >> four observations per application, but want to know about how >> consistent the raters are? I'm just trying to make sure I can >> replicate the problem, because when I tried to fit one model I got one >> answer and another blew up. >> >> I ask because I'm writing a paper on ICC estimation and have been >> considering problems very similar to the one you have here. >> >> >> On Tue, Feb 26, 2013 at 2:35 PM, Nick Cox <njcoxstata@gmail.com> wrote: >>> A scatter plot matrix is instructive. >>> >>> Warning: destroys your data. >>> >>> reshape wide Score rank , i(Application) j(Rator) >>> graph matrix Score? >>> >>> #4 really is oddball. >>> >>> Another interesting plot is >>> >>> parplot Score?, tr(raw) >>> >>> where -parplot- must be installed from SSC first. >>> >>> Nick >>> >>> On Tue, Feb 26, 2013 at 7:24 PM, Lenny Lesser <lenny3200@gmail.com> wrote: >>>> Hi Rebecca, >>>> Thank you for your help. As a clarification: >>>> We used scores originally, but Rater 4's scores were all very low. >>>> Thus, when we ranked them, there were a lot of ties. As seen below, 8 >>>> of the 11 apps got a rank of "2" by rater. >>>> Lenny >>>> >>>> Application Rator Score rank >>>> 5 1 2 1 >>>> 7 1 5 2 >>>> 2 1 6 3 >>>> 9 1 6 3 >>>> 11 1 7 4 >>>> 6 1 7 4 >>>> 8 1 11 5 >>>> 3 1 13 6 >>>> 4 1 16 7 >>>> 10 1 17 8 >>>> 1 1 18 9 >>>> 6 2 1 1 >>>> 5 2 2 2 >>>> 11 2 3 3 >>>> 7 2 3 3 >>>> 4 2 5 4 >>>> 1 2 7 5 >>>> 8 2 8 6 >>>> 2 2 9 7 >>>> 3 2 10 8 >>>> 10 2 12 9 >>>> 9 2 12 9 >>>> 5 3 2 1 >>>> 2 3 5 2 >>>> 7 3 6 3 >>>> 6 3 6 3 >>>> 9 3 6 3 >>>> 11 3 7 4 >>>> 8 3 11 5 >>>> 3 3 13 6 >>>> 4 3 15 7 >>>> 10 3 16 8 >>>> 1 3 17 9 >>>> 7 4 0 1 >>>> 1 4 1 2 >>>> 9 4 1 2 >>>> 6 4 1 2 >>>> 8 4 1 2 >>>> 4 4 1 2 >>>> 5 4 1 2 >>>> 3 4 1 2 >>>> 11 4 1 2 >>>> 2 4 2 3 >>>> 10 4 3 4 >>>> >>>> On Tue, Feb 26, 2013 at 9:54 AM, Rebecca Pope <rebecca.a.pope@gmail.com> wrote: >>>>> Lenny, >>>>> I was just addressing your syntax error, not your underlying data >>>>> issues. Why would you expect a ratio to increase when you've made the >>>>> numerator 0? If you are getting an ICC close to 0, you should think >>>>> about what that is telling you about your data. >>>>> >>>>> If you look at e.g. judges.dta (example for -icc-), you'll see that >>>>> the results for the ICC is the same regardless of the method that you >>>>> use. >>>>> >>>>> webuse judges >>>>> icc rating target judge, mixed >>>>> xtmixed rating i.judge || _all: R.target, reml var >>>>> nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2) >>>>> >>>>> The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed- >>>>> will never give you a negative value though. >>>>> >>>>> An aside: "I'm using the ranks (within an individual) instead of the >>>>> actual scores." >>>>> >>>>> If you are using rankings (1-11 presumably) within individual rather >>>>> than actual scores it isn't clear to me how rater 4 could be "off the >>>>> charts" regardless of actual scores assigned. By converting scores to >>>>> rankings, you've wiped out the correlation of scores within rater. You >>>>> seem to be interested instead in how e.g. app 1 is rated by all 4 >>>>> raters (correlation within app). If raters 1, 2, 3 all give it a score >>>>> of 1 (their preferred app) & rater 4 gives it a 6, you don't want to >>>>> drop that info. That is what you are analyzing. >>>>> >>>>> Regards, >>>>> Rebecca >>>>> >>>>> >>>>> >>>>> On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote: >>>>>> Thanks Rebecca, >>>>>> With that code, I get the same problem when I eliminate one rater. >>>>>> >>>>>> the var(rater) goes to zero, which makes my ICC 0, rather go up to a >>>>>> higher number as I expected. >>>>>> >>>>>> >>>>>> ---------- Forwarded message ---------- >>>>>> From: Rebecca Pope <rebecca.a.pope@gmail.com> >>>>>> Date: Tue, Feb 26, 2013 at 7:08 AM >>>>>> Subject: Re: st: reliability with -icc- and -estat icc- >>>>>> To: statalist@hsphsun2.harvard.edu >>>>>> >>>>>> >>>>>> Lenny, >>>>>> I don't think you've got the correct syntax for -xtmixed- if you are >>>>>> trying to duplicate ANOVA results, which is the type of analysis that >>>>>> -icc- appears to conduct (documentation is still limited, so I won't >>>>>> swear to anything). >>>>>> >>>>>> Use this syntax for -xtmixed-: >>>>>> xtmixed rank i.Application || _all: R.Rater, reml var >>>>>> >>>>>> -estat icc- is not a valid post-estimation command after this >>>>>> specification. However, you can just use the definition that ICC = >>>>>> Var(Rater)/(Var(Rater)+Var(Residual)). >>>>>> >>>>>> You might also want to take a look at >>>>>> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you >>>>>> instructions for using -xtmixed- to conduct ANOVA-type analyses (using >>>>>> Stata 10, so you'll need to modify somewhat). >>>>>> >>>>>> Regards, >>>>>> Rebecca >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote: >>>>>>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications. >>>>>>> The data is skewed right, as they all got low scores. I'm using the >>>>>>> ranks (within an individual) instead of the actual scores. I want to >>>>>>> know the correlation in ranking between the different raters. >>>>>>> >>>>>>> I've tried the two commands: >>>>>>> >>>>>>> -xtmixed rank Application || Rater: , reml >>>>>>> -estat icc >>>>>>> >>>>>>> (icc=0.19) >>>>>>> >>>>>>> and >>>>>>> >>>>>>> -icc rank Rater Application, mixed consistency >>>>>>> >>>>>>> (icc=0.34) >>>>>>> >>>>>>> They give me two different answers. Which one is correct? >>>>>>> >>>>>>> >>>>>>> Next, we found out that rater 4 was off the charts, and we want to >>>>>>> eliminate her and rerun the analysis. When we do this we get wacky >>>>>>> ICCs. In the first method we get an ICC of 2e-26. In the 2nd method >>>>>>> (-icc), we get -.06. Eliminating any of the other raters gives us >>>>>>> ICCs close to the original ICC. Why are we getting such a crazy >>>>>>> number when we eliminate this 4th rater? >>>>>>> >>>>>>> >>>>>>> I'm guessing this might be instability in the model, but I'm not sure >>>>>>> how to get around it. * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/

**Follow-Ups**:**Re: st: reliability with -icc- and -estat icc-***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**References**:**st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Rebecca Pope <rebecca.a.pope@gmail.com>

**Fwd: st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Rebecca Pope <rebecca.a.pope@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

- Prev by Date:
**Re: st: reliability with -icc- and -estat icc-** - Next by Date:
**Re: st: reliability with -icc- and -estat icc-** - Previous by thread:
**Re: st: reliability with -icc- and -estat icc-** - Next by thread:
**Re: st: reliability with -icc- and -estat icc-** - Index(es):