Re: statalist-digest V4 #4807 (st: reliability with -icc- ) - Statistics as APPLIED science

[Lenny Lesser <lenny3200@gmail.com>]

Hi,
Thank you all for your comments.
Surely their is a methods and conceptual issue.
There is also a STATA question, which I was mostly discussing here.

The problem with our scale is that is wasn't meant for smartphone
apps.  We actually developed another shorter scale for this purpose
that we tested along side this.  The shorter scale had more agreement
between raters and ranked the apps similarly.  The reason the first
scale was so poor is that it asked detailed questions that were easy
to miss in looking at a smartphone app.  They were also repetitive and
similar.

For the STATA question, it appears that we've found the right solution.
Thank you all,
Lenny

On Thu, Feb 28, 2013 at 5:37 AM, JVerkuilen (Gmail)
<jvverkuilen@gmail.com> wrote:
> On Thu, Feb 28, 2013 at 6:10 AM, Nick Cox <njcoxstata@gmail.com> wrote:
>> I do agree broadly with Allan, whether or not that is surprising.
>
> Me too, actually, though I think both Nick and I were misquoted and
> taken out of context. I know the first thing I did with the raw data
> was qnorm and graph box....
>
>
>> A wilder idea is that rater 4 who gave no score higher than 3 either
>> never knew or somehow forgot that scores could be up to 100 and just
>> used a 5-point scale. Even if #4 did know that, #4 is so out-of-line
>> that including them remains dubious, although doing computations with
>> and without #4 remains manageable.
>>
>> In any case if the highest score is 18, then something else is going
>> on that needs to be spelled out, if only as context.
>
> 100% agree and that's certainly consistent with a good bit of the
> discussion on that thread, but there were quite a bit of discussions
> not strictly aimed at Lenny's original problem but at the broader
> question of ICC estimation using Stata, which is the nature of this
> listserv.
>
>
>
>> Only the original poster can add more context than we already have. In
>> any field that I know about this dataset would be too small to be
>> publishable, except as a toy dataset to make points about method
>> (which I take it is Jay's motive here).
>
> Yes, that's my exact motive. I'm writing a paper on "small sample"
> problems with estimation of reliability coefficients, which are quite
> common in practice.
>
>
> --
> JVVerkuilen, PhD
> jvverkuilen@gmail.com
>
> "It is like a finger pointing away to the moon. Do not concentrate on
> the finger or you will miss all that heavenly glory." --Bruce Lee,
> Enter the Dragon (1973)
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/