Receiver operating characteristics (ROC)

Stata’s suite for ROC analysis consists of six commands: roctab, roccomp, rocfit, rocgold, rocreg, and rocregplot.

Stata’s roctab command provides nonparametric estimation of the ROC curve, and produces Bamber and Hanley confidence intervals for the area under the ROC curve.

Stata’s roccomp command provides tests of equality of ROC areas. It can estimate nonparametric and parametric binormal ROC curves.

rocfit fits maximum likelihood models for a single classifier, an indicator of the latent binormal variable for the true status.

rocgold performs tests of equality of ROC area, against a “gold standard” ROC curve, and can adjust significance levels for multiple tests across classifiers via Sidak’s correction.

rocreg performs ROC regression, that is, it can adjust both sensitivity and specifity for prognotic factors such as age and gender; it is by far the most general of all the ROC commands.

rocregplot draws ROC curves as modeled by rocreg. ROC curves may be drawn across covariate values, across classifiers, and both.

Norton et al. (2000) examined a neo-natal audiology study on hearing impairment. A hearing test was applied to children aged 30 to 53 months. It is believed that the classifier y1 (DPOAE 65 at 2kHz) becomes more accurate at older ages.

We use rocreg to fit a maximum likelihood model for this situation. The extra effect of current age on y1 when the child has hearing impairment is estimated by specifying roccov(). The control population effect of current age and gender of the child is estimated with the ctrlcov() option.

The results show us that current age has a borderline significant positive effect on the ROC curve (p-value = 0.045). We now use rocregplot to draw the ROC curves for ages of 50 and 40 months, and add some graph options to make the legend pretty and place it inside the graph.

rocregplot, at1(currage=40) at2(currage=50) ///
legend(order(3 "reference" 1 "40 mos." 2 "50 mos.") ring(0) rows(3) pos(5)) ///
title("ROC, by age") xsize(4) ysize(4)

The graph indicates that the area under the curve (AUC) for 50 months is clearly larger than that for 40 months, and this can be formally verified by using the postestimation command testnl after rocreg; see [R] rocregplot for a related example.

Two other classifiers were examined in the study, y2 (TEOAE 80 at 2kHz) and y3 (ABR). We will use rocgold to compare the ROC areas of y2 and y3, assuming a “gold standard” classifier of y1 (DPOAE 65 at 2kHz). The sidak option provides adjusted p-values, reflecting the two tests that are being performed.

We cannot reject the hypotheses that y2 and y3 have the same area as y1. Both the adjusted and unadjusted p-values support this.

Wieand et. al. (1989) examined a pancreatic cancer study. No covariates were recorded, and the study was a case–control study.

We use rocreg to estimate the ROC curve for the classifier y2 (CA 125) that was examined. A nonparametric estimate is used, and we bootstrap to obtain standard errors. We estimate the sensitivity for the specificity value of .6 through the roc() option, which takes argument 1-specificity. The partial area under the curve (pAUC), the area under the ROC curve up to a given 1-specificity value, is estimated for the specificity of .4 with the pauc() option. The case–control sampling of the study is indicated to rocreg via the bootcc option.

We can use rocregplot to see the ROC curve for y2 (CA 125). We also ask for normal-based confidence band for ROC value at the specificity of .6.

rocregplot, plot1opts(msymbol(i)) ///
legend(order(2 "reference" 1 "CA 125") ring(0) rows(2) pos(5)) ///
xsize(4) ysize(4) title("ROC, CA 125")

See New in Stata 12 for more about what was added in Stata Release 12.

References

Norton, S. J., M. P. Gorga, J. E. Widen, R. C. Folsom, Y. Sininger B. Cone-Wesson, B. R. Vohr, K. Mascher, and K. Fletcher. 2000.

Identification of neonatal hearing impairment: Evaluation of transient evoked otoacoustic emission, distortion product otoacoustic emission, and auditory brain stem response test performance. Ear and Hearing 21: 508–528.

Wieand, S., M. H. Gail, B. R. James, and K. L. James. 1989.

A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76: 585–592.

Pepe, M. S. 2003.

The Statistical Evaluation of Medical Tests for Classification and Prediction. New York: Oxford University Press.