Re: st: Re: psmatch2 question

[Steve Samuels <sjsamuels@gmail.com>]

I agree with the -psmatch2-  authors.  I recently bootstrapped an ATT
with and without the  logistic regression that estimated the
propensity score.  The SE that included the propensity score step was
about 19% higher than the SE that ignored that step.   Ignoring the
uncertainty in the propensity score would have been equivalent to
pretending that the sample size was about 40% higher than it actually
was  (1.19^2 =1.416).

Here is code similar to what I used.  The propensity score estimation
actually required  much more than one  -logistic- command: variable
selection, balance checking, model checking and revision, calibration
of scores, and restriction to common support.  For simplicity, the
-bootstrap- ignored all that and so still under-estimates the actual
standard error.

Steve
-- 
Steven Samuels
sjsamuels@gmail.com
18 Cantine's Island
Saugerties NY 12477
USA
Voice: 845-246-0774
Fax:    206-202-4783


**************************CODE BEGINS**************************
scalar drop _all

capture program drop attboot
use d01, clear
program define attboot, rclass

preserve
qui logistic treatment v1-v10
predict probx, pr
gen w_att = cond(treatment, 1, probx/(1- probx))  //weight

reg outcome treatment [pw= w_att]
local att = _b[treatment]
return scalar att = `att'

restore
end

bootstrap  att = r(att), reps(5000): attboot
estat bootstrap, all
***************************CODE ENDS***************************

On Tue, Aug 24, 2010 at 1:57 PM, Joseph Coveney <jcoveney@bigplanet.com> wrote:
> anna bargagliotti wrote:
>
> I am a bit confused on how to read  the output for the psmatch2 command.  I run
> the following:
>
> psmatch2   bootcamp2008 sex race classlevel_2008 engineering_2008
> retorgradstem_2009, outcome(ucumgpa_2009) cal(.01)
>
> [output excerpted]
>
> My question is related to the T-stats reported (ie -.78 and .07).  I am  unsure
> how to interpret these.  What are these exactly?  I was under the  impression
> that the T-stat for ATT (.07) is the statistic to use in  determining whether
> the treatment had an effect on ucumgpa_2009,  however, the output does not give
> a significance level.  To check, I  tried to recreate this T-stat by computing
> it by brute force but I could  not do so.
>
> I tried the following:
> 1.  reduced the sample to only those students who were in the treatment  and
> their matches
> 2. created repeated samples for students who were matches more than one  time
> 3. Performed the ttest comparing the ucumgpa_2009 of treatment vs.  control
> (indicated by the bootcamp2008 variable)
>
> The tvalue I get is no where near the T-stat given by psmatch2.   Have I
> misunderstood the output given by psmatch2?  How do I tell  whether the
> bootcamp2008 had a significant effect on ucumgpa_2009?
>
> --------------------------------------------------------------------------------
>
> You can reconstruct the reported t statistic for ATT from the returned scalars
> r(att) and r(seatt).  I'm guessing, but I believe that the command's authors
> have declined computing the associated p-value for the same reason that they
> have the output warn, "Note: S.E. for ATT does not take into account that the
> propensity score is estimated."  That is, any computed p-value would be an
> overstatement.  The help file for -psmatch2- suggests bootstrapping as an
> alternative approach (with a caveat about its validity) to computing the
> t statistic.
>
> Joseph Coveney
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>



-- 
Steven Samuels
sjsamuels@gmail.com
18 Cantine's Island
Saugerties NY 12477
USA
Voice: 845-246-0774
Fax:    206-202-4783

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/