Re: st: Combine bootstrapped results of imputed data - does this work?

[roland andersson <rolandersson@gmail.com>]

This is a followup to this thread.

I contacted Yulia Marchenko at Stata. She has written a FAQ example
which I used, but I had made a mistake which she helped me to correct.
I hope all details are mentioned in this correspondence.
This works - we can do roctab from imputed variables and calculate
bootstrapped CI.
best regards
Roland E AnderssonJönköping. Sweden



---------- Forwarded message ----------From: roland andersson
<rolandersson@gmail.com>Date: 2011/12/7Subject: Re: Fwd: st: Combine
bootstrapped results of imputed data -does this work?- Dölj citerad
text -To: Yulia Marchenko <ymarchenko@stata.com>Kopia:
anderssonmanne@gmail.com

Dear Yulia
Thank you very much. This helps us very much.
I think I understand my error. I saved the bootstrapped SE from
theroctab, but your solution saves the SE of the bootstrapped ROC
area? Ijust want you to check out my program so there is no
misunderstanding.
"cap program drop ebootsraproctabprogram ebootstraproctab, eclass
 version 11.0       /* Step 1: perform ROC analysis */       args
refvar classvar       bootstrap r(area): roctab  `refvar' `classvar'
       /* Step 2: save estimate and its variance in temporary
matrices*/       tempname b V       mat `b' = _b[_bs_1]       mat `V'
= e(V)       local N =e(N)
       /* Step 3: make column names and row names consistent*/
mat colnames `b' = AUC       mat colnames `V' = AUC       mat rownames
`V' = AUC
       /*Step 4: post results to e()*/       ereturn post `b' `V',
obs(`N')       ereturn local cmd "ebootstraproctab"       ereturn
local title "ROC area"
end"
Regards
Roland Andersson


2011/12/7 Yulia Marchenko <ymarchenko@stata.com>:> Dear Roland,>> I
have a couple of corrections for your -ebootstraproctab- program.  For
a> moment, let's forget you have imputed data.> When you use
-bootstrap r(area): roctab ...-, the bootstrapped variance of> your
ROC estimate is saved in the -e(V)- matrix.  The bootstrapped
variance> (or standard error) of a ROC estimate (or any estimate) is
not a> bootstrapped estimate of the model variance (or standard error)
of the ROC> estimate.  So, you should not be bootstraping -r(se)-.
You will need to> modify the corresponding lines of your
-ebootstraproctab- program as> follows:>>  . bootstrap r(area): roctab
 `refvar' `classvar'  // in Step 1>  . mat `V' = e(V) // in Step 2>>
Regarding looping over the imputed data, the -mi estimate- command
takes> care of it.  When you use -ebootstraproctab- with -mi
estimate-, -mi> estimate- executes -ebootstraproctab- on each imputed
data and so bootstrap> is performed on each imputed data.>> Best
regards,>> Yulia>>>>> Dear Yulia Marchenko>>>> Excuse me for
contacting you directly. I have posted a question to>> Statalist a few
times and have not received any response, and I am>> convinced that
you can give me a clear answer.  I have adapted one of>> your programs
from the stata website and wonder if this will do what I>> want.>>>> I
have constructed a clinical diagnostic score. I want to analyse the>>
bias corrected ROC area of the score by bootstrapping. The problem
is>> that I had missing values in the original data and have therefore
used>> multiple imputation. So I want to use bootstrapping for
analysing the>> the area in 20 imputed datasets.>>>> I do not know how
to combine the bootstrapped ROC area and SE. I found>> this useful
program>> http://www.stata.com/support/faqs/stat/mi_combine.html which
I adapted>> for bootstrapping (see below).>>>> My command is>> . mi
estimate, cmdok: ebootstraproctab appendicitdgn  clinicalscore>>>> The
program produces results, but I want to be sure that the produced>>
results are correct, ie represents the combined bootstrapped results>>
from 20 imputed datasets.>>>> I am not sure that the program will loop
over all the imputed>> datasets, ie how can bootstrap know which
dataset to use each time?>>>> Here is my>>>> "cap program drop
ebootsraproctab>> program ebootstraproctab, eclass>>      version
11.0>>      /* Step 1: perform ROC analysis */>>      args refvar
classvar>>      bootstrap r(area) r(se): roctab  `refvar'
`classvar'>>>>      /* Step 2: save estimate and its variance in
temporary matrices*/>>      tempname b V>>      mat `b' = _b[_bs_1]>>
    mat `V' = _b[_bs_2]^2>>      local N =e(N)>>>>      /* Step 3:
make column names and row names consistent*/>>      mat colnames `b' =
AUC>>      mat colnames `V' = AUC>>      mat rownames `V' = AUC>>>>
  /*Step 4: post results to e()*/>>      ereturn post `b' `V',
obs(`N')>>      ereturn local cmd "ebootstraproctab">>      ereturn
local title "ROC area">>>> end">>>> My sincerest regards>>>> Roland
Andersson>> Jönköping, Sweden>>>>>>
2011/12/6 roland andersson <rolandersson@gmail.com>:
> Dear Statalist
>
> I have constructed a clinical diagnostic score. I want to analyse the
> bias corrected ROC area of the score by bootstrapping. The problem is
> that I had missing values in the original data and have therefore used
> multiple imputation. So I want to use bootstrapping for analysing the
> the area in 20 imputed datasets.
>
> I do not know how to combine the bootstrapped ROC area and SE. I found
> this useful program
> http://www.stata.com/support/faqs/stat/mi_combine.html which I adapted
> for bootstrapping (see below).
>
> My command is
> . mi estimate, cmdok: ebootstraproctab appendicitdgn  clinicalscore
>
> The program produces results, but I want to be sure that the produced
> results are correct, ie represents the combined bootstrapped results
> from 20 imputed datasets.
>
> All comments are wellcome.
>
>
> "cap program drop ebootsraproctab
> program ebootstraproctab, eclass
>       version 11.0
>       /* Step 1: perform ROC analysis */
>       args refvar classvar
>       bootstrap r(area) r(se): roctab  `refvar' `classvar'
>
>       /* Step 2: save estimate and its variance in temporary matrices*/
>       tempname b V
>       mat `b' = _b[_bs_1]
>       mat `V' = _b[_bs_2]^2
>       local N =e(N)
>
>       /* Step 3: make column names and row names consistent*/
>       mat colnames `b' = AUC
>       mat colnames `V' = AUC
>       mat rownames `V' = AUC
>
>       /*Step 4: post results to e()*/
>       ereturn post `b' `V', obs(`N')
>       ereturn local cmd "ebootstraproctab"
>       ereturn local title "ROC area"
>
> end"
>
> Roland Andersson
> Jönköping, Sweden
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/