Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Area under the ROC curve less than 0.5?


From   Roger Newson <[email protected]>
To   "[email protected]" <[email protected]>
Subject   Re: st: Area under the ROC curve less than 0.5?
Date   Fri, 21 Jan 2011 13:54:56 +0000

In reply to your first query, the area under the ROC can be anything from 0 to 1. Predictors with a ROC area less than 0.5 are negative prwedictors, and predictors with a ROC area between 0.5 and 1 are positive predictors.

In reply to your second query, it is possible for the difference between 2 ROC areas to be statistically non-significant in a small sample, even if one ROC area is less than half the other. This is because the confidence interval for the difference between 2 ROC areas may be wide, and include a zero difference and a range of positive and negative differences. To get a confidence interval for the difference between 2 ROC areas (also known as Harrell's c statistics), use the -somersd- package, which you can download from SSC and which has 3 .pdf manuals, distributed with the package as ancillary files. The on-line help contains hyper-references to the Stata Journal articles explaining the -somersd- package, which you can use with -lincom- to get confidence intervals for the difference between 2 Harrell's c statistics.

I hope this helps.

Best wishes

Roger


Roger B Newson BSc MSc DPhil
Lecturer in Medical Statistics
Respiratory Epidemiology and Public Health Group
National Heart and Lung Institute
Imperial College London
Royal Brompton Campus
Room 33, Emmanuel Kaye Building
1B Manresa Road
London SW3 6LR
UNITED KINGDOM
Tel: +44 (0)20 7352 8121 ext 3381
Fax: +44 (0)20 7351 8322
Email: [email protected]
Web page: http://www.imperial.ac.uk/nhli/r.newson/
Departmental Web page:
http://www1.imperial.ac.uk/medicine/about/divisions/nhli/respiration/popgenetics/reph/

Opinions expressed are those of the author, not of the institution.

On 21/01/2011 09:56, [email protected] wrote:
Hi!

I just want to ask, after running a logistic regression on a development
sample, I ran the lroc command. The corresponding area under the ROC
curve is .9226. When I wished to check the robustness of the model on a
holdout sample, I keyed in "lroc if sample==2". The software was able to
run it on this separate sample, however, the resulting AROC is now
.4060. I thought AROC values are only from 0.5 to 1.0, or was I
mistaken?

Also, when I tried running roccomp on the two samples, Stata was again
able to churn out the same ROCs (.9226 and .4060), but the chi-square
p-value is .0663, indicating that at alpha=.05, the null hypothesis of
equal areas under the curves cannot be rejected.  Does it make sense
that even though the AROC of the second sample is less than half of the
first sample, the test still says that the 2 ROC curves have equal
areas? Is this test perhaps affected by the number of observations?
(sample 1 : 26 positive, 174 negative, while sample 2 contains 5(+) and
102(-)).

Would really appreciate anyone's help. Thanks in advance!


- Reggae




*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index