Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down on April 23, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
Nick Cox <njcoxstata@gmail.com> |

To |
statalist@hsphsun2.harvard.edu |

Subject |
Re: st: reliability with -icc- and -estat icc- |

Date |
Tue, 26 Feb 2013 19:35:01 +0000 |

A scatter plot matrix is instructive. Warning: destroys your data. reshape wide Score rank , i(Application) j(Rator) graph matrix Score? #4 really is oddball. Another interesting plot is parplot Score?, tr(raw) where -parplot- must be installed from SSC first. Nick On Tue, Feb 26, 2013 at 7:24 PM, Lenny Lesser <lenny3200@gmail.com> wrote: > Hi Rebecca, > Thank you for your help. As a clarification: > We used scores originally, but Rater 4's scores were all very low. > Thus, when we ranked them, there were a lot of ties. As seen below, 8 > of the 11 apps got a rank of "2" by rater. > Lenny > > Application Rator Score rank > 5 1 2 1 > 7 1 5 2 > 2 1 6 3 > 9 1 6 3 > 11 1 7 4 > 6 1 7 4 > 8 1 11 5 > 3 1 13 6 > 4 1 16 7 > 10 1 17 8 > 1 1 18 9 > 6 2 1 1 > 5 2 2 2 > 11 2 3 3 > 7 2 3 3 > 4 2 5 4 > 1 2 7 5 > 8 2 8 6 > 2 2 9 7 > 3 2 10 8 > 10 2 12 9 > 9 2 12 9 > 5 3 2 1 > 2 3 5 2 > 7 3 6 3 > 6 3 6 3 > 9 3 6 3 > 11 3 7 4 > 8 3 11 5 > 3 3 13 6 > 4 3 15 7 > 10 3 16 8 > 1 3 17 9 > 7 4 0 1 > 1 4 1 2 > 9 4 1 2 > 6 4 1 2 > 8 4 1 2 > 4 4 1 2 > 5 4 1 2 > 3 4 1 2 > 11 4 1 2 > 2 4 2 3 > 10 4 3 4 > > On Tue, Feb 26, 2013 at 9:54 AM, Rebecca Pope <rebecca.a.pope@gmail.com> wrote: >> Lenny, >> I was just addressing your syntax error, not your underlying data >> issues. Why would you expect a ratio to increase when you've made the >> numerator 0? If you are getting an ICC close to 0, you should think >> about what that is telling you about your data. >> >> If you look at e.g. judges.dta (example for -icc-), you'll see that >> the results for the ICC is the same regardless of the method that you >> use. >> >> webuse judges >> icc rating target judge, mixed >> xtmixed rating i.judge || _all: R.target, reml var >> nlcom exp(_b[lns1_1_1:_cons])^2/(exp(_b[lnsig_e:_cons])^2+exp(_b[lns1_1_1:_cons])^2) >> >> The two ICCs are nearly equal (to 6 decimal places). Using -xtmixed- >> will never give you a negative value though. >> >> An aside: "I'm using the ranks (within an individual) instead of the >> actual scores." >> >> If you are using rankings (1-11 presumably) within individual rather >> than actual scores it isn't clear to me how rater 4 could be "off the >> charts" regardless of actual scores assigned. By converting scores to >> rankings, you've wiped out the correlation of scores within rater. You >> seem to be interested instead in how e.g. app 1 is rated by all 4 >> raters (correlation within app). If raters 1, 2, 3 all give it a score >> of 1 (their preferred app) & rater 4 gives it a 6, you don't want to >> drop that info. That is what you are analyzing. >> >> Regards, >> Rebecca >> >> >> >> On Tue, Feb 26, 2013 at 10:16 AM, Lenny Lesser <lenny3200@gmail.com> wrote: >>> Thanks Rebecca, >>> With that code, I get the same problem when I eliminate one rater. >>> >>> the var(rater) goes to zero, which makes my ICC 0, rather go up to a >>> higher number as I expected. >>> >>> >>> ---------- Forwarded message ---------- >>> From: Rebecca Pope <rebecca.a.pope@gmail.com> >>> Date: Tue, Feb 26, 2013 at 7:08 AM >>> Subject: Re: st: reliability with -icc- and -estat icc- >>> To: statalist@hsphsun2.harvard.edu >>> >>> >>> Lenny, >>> I don't think you've got the correct syntax for -xtmixed- if you are >>> trying to duplicate ANOVA results, which is the type of analysis that >>> -icc- appears to conduct (documentation is still limited, so I won't >>> swear to anything). >>> >>> Use this syntax for -xtmixed-: >>> xtmixed rank i.Application || _all: R.Rater, reml var >>> >>> -estat icc- is not a valid post-estimation command after this >>> specification. However, you can just use the definition that ICC = >>> Var(Rater)/(Var(Rater)+Var(Residual)). >>> >>> You might also want to take a look at >>> http://www.ats.ucla.edu/stat/stata/faq/xtmixed.htm which will give you >>> instructions for using -xtmixed- to conduct ANOVA-type analyses (using >>> Stata 10, so you'll need to modify somewhat). >>> >>> Regards, >>> Rebecca >>> >>> >>> >>> On Mon, Feb 25, 2013 at 10:56 PM, Lenny Lesser <lenny3200@gmail.com> wrote: >>>> I have 4 raters that gave a score of 0-100 on 11 smartphone applications. >>>> The data is skewed right, as they all got low scores. I'm using the >>>> ranks (within an individual) instead of the actual scores. I want to >>>> know the correlation in ranking between the different raters. >>>> >>>> I've tried the two commands: >>>> >>>> -xtmixed rank Application || Rater: , reml >>>> -estat icc >>>> >>>> (icc=0.19) >>>> >>>> and >>>> >>>> -icc rank Rater Application, mixed consistency >>>> >>>> (icc=0.34) >>>> >>>> They give me two different answers. Which one is correct? >>>> >>>> >>>> Next, we found out that rater 4 was off the charts, and we want to >>>> eliminate her and rerun the analysis. When we do this we get wacky >>>> ICCs. In the first method we get an ICC of 2e-26. In the 2nd method >>>> (-icc), we get -.06. Eliminating any of the other raters gives us >>>> ICCs close to the original ICC. Why are we getting such a crazy >>>> number when we eliminate this 4th rater? >>>> >>>> >>>> I'm guessing this might be instability in the model, but I'm not sure >>>> how to get around it. * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/

**Follow-Ups**:**Re: st: reliability with -icc- and -estat icc-***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*"JVerkuilen (Gmail)" <jvverkuilen@gmail.com>

**References**:**st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Rebecca Pope <rebecca.a.pope@gmail.com>

**Fwd: st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Rebecca Pope <rebecca.a.pope@gmail.com>

**Re: st: reliability with -icc- and -estat icc-***From:*Lenny Lesser <lenny3200@gmail.com>

- Prev by Date:
**Re: st: reliability with -icc- and -estat icc-** - Next by Date:
**st: passing argument(s) to Mata constructors** - Previous by thread:
**Re: st: reliability with -icc- and -estat icc-** - Next by thread:
**Re: st: reliability with -icc- and -estat icc-** - Index(es):