Re: st: deriving a bootstrap estimate of a difference between two weighted regressions

[Steve Samuels <sjsamuels@gmail.com>]

The difference I saw in my last email was due to the small number of
bootstrap replicates I had run for my test cases  When I upped the
number of replications to 200  the standard errors that included the
logistic regression were about 19% higher than the standard errors
that did not for ATE and ATT, and about the same for ATC.  This is
still important, I think.
  The lack of difference for ATC makes sense, as the weights based on
the propensity scores affected only the 2% of the sample in the
"treated" group.

Steve



On Mon, Aug 9, 2010 at 9:39 PM, Steve Samuels <sjsamuels@gmail.com> wrote:
> To finish off this topic, perhaps:
>
> I had occasion recently to examine the causal association between a
> continuous outcome and a two category "treatment" variable.  The
> "treated" group constituted about 2% of the sample. The method of
> analysis was OLS, and propensity scores had been estimated by logistic
> regression.  I computed bootstrap standard errors that included and
> excluded the logistic step.
>
> The result: standard errors for  average treatment effects ATE,  ATT,
> and ATC  that included the logistic regression were roughly double(!)
> the standard errors that did not.
>
> Steve
>
>
>
> On Wed, Aug 4, 2010 at 1:07 PM, Ariel Linden, DrPH
> <ariel.linden@gmail.com> wrote:
>> Thank you, Stas. I will take your suggestions under advisement!
>>
>>
>>
>> Date: Tue, 3 Aug 2010 14:37:51 -0500
>> From: Stas Kolenikov <skolenik@gmail.com>
>> Subject: Re: st: deriving a bootstrap estimate of a difference between two
>> weighted regressions
>>
>> On Tue, Aug 3, 2010 at 1:20 PM, Ariel Linden, DrPH <ariel.linden@gmail.com>
>> wrote:
>>> Thank you Stas and Steve for your comments!
>>>
>>> When I stated that the first model's weight would be ATT and the next
>>> ATC, it was already after running the propensity score model and
>>> establishing
>> the
>>> weights for each subject:
>>> ATT = cond(treatvar, 1, propvar/(1- propvar)), and ATC =
>>> cond(treatvar, (1-propvar)/propvar, 1)
>>>
>>> Under these conditions, there should be no negative weights, so that
>>> is
>> not
>>> a concern.
>>
>> The negative weights would come out of Steve's suggestion to entertain the
>> difference in weights (as that's what your procedure boils down to).
>>
>>> I am thinking that the code would look something like this, but I
>>> would appreciate your input:
>>>
>>> 1. bootstrap _b[treatvar] from first regression with [pw=ATT] 2. save
>>> 10,000 samples to file (or tempfile) 3. bootstrap _b[treatvar] from
>>> second regression with [pw=ATC] 4. save 10,000 samples to file (or
>>> tempfile) 5. gen difference = treatvar1-treatvar2 6. bootstrap
>>> r(mean): sum  difference, to get bootstrapped CIs
>>>
>>> Does this make sense?
>>
>> 1-2 will produce something very similar to the _se[treatvar] in your basic
>> regression with ATT weights (probably with -robust- option), and
>> 3-4 will produce something very similar to _se[treatvar] in the regression
>> with ATC weights. I outlined the code for you in the previous message -- you
>> need to bootstrap the whole estimation procedure = { the propensity
>> regression (leading to the weights) + two main regressions with two sets of
>> weights }. In other words, for each bootstrap sample, you would need to run
>> everything in the curly brackets to produce your "difference" estimate.
>>
>> I cannot comment on the scientific validity of this procedure; other people
>> more knowledgeable in treatment effect estimation could do that.
>>
>>
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>
>
>
> --
> Steven Samuels
> sjsamuels@gmail.com
> 18 Cantine's Island
> Saugerties NY 12477
> USA
> Voice: 845-246-0774
> Fax:    206-202-4783
>



-- 
Steven Samuels
sjsamuels@gmail.com
18 Cantine's Island
Saugerties NY 12477
USA
Voice: 845-246-0774
Fax:    206-202-4783

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/