Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: RE: Output problem attnd , attr, atts
From
"Jason Zarmulski" <[email protected]>
To
[email protected]
Subject
Re: st: RE: Output problem attnd , attr, atts
Date
Thu, 30 Jun 2011 14:27:51 +0200
Thanks Jan
--- Ursprüngliche Nachricht ---
Von: Jan Bryla <[email protected]>
Datum: 28.06.2011 21:00:07
An: "[email protected]" <[email protected]>
Betreff: st: RE: Output problem attnd , attr, atts
> Jason,
>
> It seems to me that the interpretation is standard to the treatment literature
> (average treatment effect on the treated). Differences arise because you
> are applying different "methods". In that respect it is not surprising
> to me, that you obtain different results. I guess the underlying reason the
> methods produce rather dissimilar results has to do with the distribution
> of the outcome of interest and the propensity score.
>
> If memory serves -pstest- works only after -psmatch2-, but maybe other listers
> can confirm/deny?
>
> Regarding robustness I would definetely check if these results are robust
> to various potential problems.
>
> Hope it helps - there are good introductions to these methods around on the
> web.
>
> Jan Bryla
>
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Jason Zarmulski
> Sent: 28. juni 2011 11:10
> To: [email protected]
> Subject: st: Output problem attnd , attr, atts
>
> Hi Statalists,
> >
> >
> > Now I have the problem with interpreting the different outcomes.
>
> My Questions:
> >
> > -How can I Interpret the different Outcomes?
> > -Why isnt the .pstest method working with these outcomes?
> > -Are these Outcomes my the final ones, or do I have to do some robustness
>
> > tests as well? (If yes, which one?)
> >
> >
>
> > My new Propensity Score and the different outcomes are:
> >
> >
> > The balancing property is satisfied
> >
> >
> > This table shows the inferior bound, the number of treated
> > and the number of controls for each block
> >
> > Inferior |
> > of block | REL(Dummy)
> > of pscore | 0 1 | Total
> > -----------+----------------------+----------
> > .2 | 5 6 | 11
> > .4 | 16 21 | 37
> > .6 | 20 17 | 37
> > .7 | 14 47 | 61
> > .8 | 19 110 | 129
> > .9 | 9 223 | 232
> > -----------+----------------------+----------
> > Total | 83 424 | 507
> >
> >
> >
> > . attnd stockprice2010 reldummy, pscore(mypscore)comsup
> >
> >
> > The program is searching the nearest neighbor of each treated unit.
>
> > This operation may take a while.
> >
> >
> >
> > ATT estimation with Nearest Neighbor Matching method
> > (random draw version)
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat. n. contr. ATT Std. Err. t
> > ---------------------------------------------------------
> >
> > 424 65 -0.572 6.828 -0.084
> >
> > ---------------------------------------------------------
> > Note: the numbers of treated and controls refer to actual
> > nearest neighbour matches
> >
> >
> >
> > . attr stockprice2010 reldummy, pscore(mypscore) radius(0.1) comsup
>
> >
> >
> > The program is searching for matches of treated units within radius.
>
> > This operation may take a while.
> >
> >
> >
> > ATT estimation with the Radius Matching method
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat. n. contr. ATT Std. Err. t
> > ---------------------------------------------------------
> >
> > 424 83 1.354 3.588 0.377
> >
> > ---------------------------------------------------------
> > Note: the numbers of treated and controls refer to actual
> > matches within radius
> >
> >
> > . atts stockprice2010 reldummy, pscore(mypscore) comsup blockid(myblock)
>
> >
> >
> >
> >
> > ATT estimation with the Stratification method
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat. n. contr. ATT Std. Err. t
> > ---------------------------------------------------------
> >
> > 424 83 0.273 2.520 0.109
> >
> > ---------------------------------------------------------
> >
> >Thanks
> Jason
> >
> >
> >
> > --- Ursprüngliche Nachrsht ---
> > Von: Jan Bryla <[email protected]>
> > Datum: 21.06.2011 15:04:02
> > An: "[email protected]" <[email protected]>
>
> >
> > Betreff: RE: st: RE: Ousout problem psmatch2
> >
> > > Ah, yes. You are right - my mistake.
> > >
> > > Regarding -attnd- I see a few reasons why you can get different
> results:
> >
> > >
> > > - using -psmatch2- you "restrict" to 10 neighbours. What
> is
> > the
> > > number og nearest neighbours selected by -attnd-?
> > > - you set caliper to 0.1 using psmatch2. But I don't think a similar
>
> > "restriction"
> > > is imposed in attnd?
> > >
> > > Maybe this is a start...
> > >
> > > /Jan
> > >
> > >
> > > -----Original Message-----
> > > From: [email protected] [mailto:[email protected]]
>
> >
> > > On Behalf Of Jason Zarmulski
> > > Sent: 21. juni 2011 12:23
> > > To: [email protected]
> > > Subject: Re: st: RE: Ousout problem psmatch2
> > >
> > > Hi Jan,
> > >
> > > thanks a lot for your support, I changed it and my new output is:
>
> > > :-)
> > >
> > > . psmatch2 reldummy , outcome( stockprice2010) pscore(mypscore)
> neighbor(10)
> >
> > > caliper(0.1) ate
> > > ----------------------------------------------------------------------------------------
>
> >
> > >
> > > Variable Sample | Treated Controls Difference
>
> > > S.E. T-stat
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > stockprice2010 Unmatched | 20.6236969 21.7278571 -1.10416018
>
> > 2.08916252
> > > -0.53
> > > ATT | 20.6236969 20.4730466 .150650303
>
> > 4.16715274
> > > 0.04
> > > ATU | 21.7278571 17.8625509 -3.86530621
>
> > > . .
> > > ATE | -.513405498
>
> >
> > > . .
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > Note: S.E. does not take into account that the propensity score
> is estimated.
> >
> > >
> > >
> > > | psmatch2:
> > > psmatch2: | Common
> > > Treatment | support
> > > assignment | On suppor | Total
> > > -----------+-----------+----------
> > > Untreated | 84 | 84
> > > Treated | 424 | 424
> > > -----------+-----------+----------
> > > Total | 508 | 508
> > >
> > >
> > > To the attnd problem : I didnt install a package for attnd. Maybe
> it
> > was
> > > with the"nnmatch ado" package
> > >
> > > The differences between the psmatch2 and the attnd are still there....
>
> >
> > >
> > > This is the help attnd file:
> > >
> > >
> > > Calculate the average treatment effect on the treated using nearest
>
> > neighbor
> > > matching
> > >
> > > attnd outcome treatment [varlist] [weight] [if exp]
> [in
> > range]
> > > [ ,
> > > pscore(scorevar) logit index comsup detail bootstrap
>
> > reps(#)
> > >
> > > noisily dots ]
> > >
> > > fweights, iweights, and pweights are allowed; see help
> weights.
> >
> > >
> > >
> > > Description
> > >
> > > attnd estimates the average treatment effect on the treated
> (ATT)
> > using
> > > nearest
> > > neighbor matching. attnd should be run after the correct propensity
>
> > score
> > >
> > > specification; i.e., the one satisfying the balancing property
> has
> > been
> > > found
> > > using, for example, pscore. If users do not provide a variable
> name
> > for
> > > the
> > > propensity score, the propensity score is estimated based on
> the
> > specification
> > >
> > > in varlist. Note that in this case the balancing property is
> not
> > tested.
> > >
> > >
> > > It is left under the responsibility of the user to select the
> comsup
> >
> > > option if
> > > the user provided propensity score has been estimated on a
> common
> > support
> > > for
> > > treated and controls. Otherwise, the ATT is estimated using
> also
> > the
> > >
> > > observations outside the common support for which the propensity
>
> > score
> > > may not
> > > be balanced.
> > >
> > > To save on computing time, nearest neighbors are not determined
>
> > by comparing
> > >
> > > treated observations to every single control, but by first
> sorting
> > all
> > > records
> > > by the estimated propensity score and then searching forward
> and
> > backward
> > > for
> > > the closest control unit(s). If a treated unit forward and
> backward
> > matches
> > >
> > > happen to be equally good, this program randomly draws (hence
> the
> > letters
> > > "nd"
> > > for Nearest neighbor and random Draw) either the forward or
> backward
> >
> > > matches.
> > > This approach is one of two computationally feasible options
> to
> > obtain
> > >
> > > analytical standard errors while at the same time exploiting
> the
> > very
> > > fast
> > > forward and backward search strategy. The second possibility
> is
> > based
> > > on giving
> > > equal weight to the groups of forward and backward matches
> in case
> > of
> > > equally
> > > good forward and backward matches and is performed by attnw.
> In
> > practice,
> > > the
> > > case of multiple nearest neighbors should be very rare. In
> particular,
> >
> > > if the
> > > set of X's contains continuous variables, in which case, both
> attnd
> > and
> > > attnw
> > > should give equal results (except for bootstrapped standard
> errors).
> >
> > > The
> > > likelihood of multiple nearest neighbors is further reduced
> if the
> > propensity
> > >
> > > score is estimated and saved in double precision, which is
> what
> > pscore
> > > does by
> > > default.
> > >
> > > The ATT is computed by averaging over the unit-level treatment
> effects
> >
> > > of the
> > > treated where the control(s) matched to a treated observation
> is/are
> >
> > > those
> > > observations in the control group that have the closest propensity
>
> > score.
> > > If
> > > there are multiple nearest neighbors, the average outcome of
> those
> > controls
> > > is
> > > used.
> > >
> > >
> > > Options
> > >
> > > pscore(scorevar) specifies the name of the user-provided variable
>
> > name
> > > for the
> > > estimated propensity score. If no name is provided the
> propensity
> >
> > > score is
> > > estimated based on the specification in varlist.
> > >
> > > logit uses a logit model to estimate the propensity score instead
>
> > of
> > > the default
> > > probit model when the option pscore(scorevar) is not specified
>
> > by
> > > the user.
> > > Otherwise, no effect is produced.
> > >
> > > index requires the use of the linear index as the propensity
> score
> > when
> > > the
> > > option pscore(scorevar) is not specified by the user.
> Otherwise,
> >
> > > no effect
> > > is produced.
> > >
> > > comsup restricts the computation of the ATT to the region of
> common
> > support.
> > >
> > >
> > > detail displays more detailed output documenting the steps
> performed
> >
> > > to obtain
> > > the final results.
> > >
> > > bootstrap bootstraps the standard error of the treatment effect.
>
> >
> > >
> > > reps(#) specifies the number of bootstrap replications to be
> performed.
> >
> > > The
> > > default is 50. This option produces an effect only if
> the bootstrap
> >
> > > option
> > > is specified.
> > >
> > > noisily requests that any output from the replications be displayed.
>
> >
> > > This
> > > option produces an effect only if the bootstrap option
> is specified.
> >
> > >
> > >
> > > dots requests that a dot be placed on the screen at the beginning
>
> > of
> > > each
> > > replication. This option produces an effect only if the
> bootstrap
> >
> > > option is
> > > specified.
> > >
> > >
> > > Remarks
> > >
> > > Please remember to use the update query command before running
> this
> > program
> > > to
> > > make sure you have an up-to-date version of Stata installed.
> Otherwise,
> >
> > > this
> > > program may not run properly.
> > >
> > > The treatment has to be binary.
> > >
> > > When users do not specify their own previously estimated propensity
>
> > score,
> > > the
> > > bootstrap encompasses the estimation of the propensity score
> based
> > on
> > > the
> > > specification given by varlist. This procedure is actually
> recommended
> >
> > > to
> > > account for the uncertainty associated with the estimation
> of the
> > propensity
> > >
> > > score. Even more so when the comsup option is specified because
>
> > in this
> > > case
> > > the region of common support changes with every bootstrap sample,
>
> > and
> > >
> > > bootstrapped standard errors pick up this uncertainty as well.
> So,
> > typically
> > >
> > > users would first identify a specification satisfying the balancing
>
> > property
> > > --
> > > using pscore -- and then provide exactly this specification
> in varlist
> >
> > > and use
> > > bootstrapped standard errors.
> > >
> > >
> > > Saved results
> > >
> > > The program stores the estimated treatment effect, its standard
>
> > error,
> > > and the t
> > > statistic respectively in the scalars r(attnd), r(seattnd),
> and
> > r(tsattnd).
> > >
> > >
> > > The number of treated and the number of controls are stored
> respectively
> >
> > > in the
> > > scalars r(ntnd) and r(ncnd).
> > >
> > > The bootstrapped standard error and t statistic are stored
> respectively
> >
> > > in the
> > > scalars r(bseattnd) and r(btsattnd).
> > >
> > >
> > > Examples
> > >
> > > . attnd wage training age age2 exp exp2
> > >
> > > . attnd wage training age age2 exp exp2, boot reps(100) dots
>
> > >
> > > . attnd wage training age age2 exp exp2, logit boot reps(100)
>
> > >
> > > . attnd wage training age age2 exp exp2, comsup boot reps(100)
>
> > >
> > >
> > > Authors
> > >
> > > Sascha O. Becker
> > > Center for Economic Studies, University of Munich
> > >
> > > Andrea Ichino
> > > Department of Economics, European University Institute,
> Florence
> >
> > >
> > >
> > > Email [email protected] or [email protected] if you observe any
> problems.
> >
> > >
> > >
> > >
> > > Acknowledgments
> > >
> > > The way to implement the propensity score estimation in the
> bootstrap
> >
> > > procedure
> > > has been adapted from the psmatch program written by Barbara
> Sianesi
> >
> > > (University
> > > College London and Institute for Fiscal Studies) Email: [email protected].
>
> >
> > >
> > >
> > >
> > > Also see
> > >
> > > Online: help for pscore, atts, attr, attk, attnw (if installed),
>
> > and
> > > bs.
> > >
> > > Further details on the analytical formulas and on
> the
> > algorithms
> > > used
> > > in these programs can be found under http://www.sobecker.de
>
> >
> > > or
> > > http://www.iue.it/Personal/Ichino.
> > >
> > > Thanks
> > > Jason
> > >
> > >
> > > --- Ursprüngliche Nachricht ---
> > > Von: Jan Bryla <[email protected]>
> > > Datum: 20.06.2011 20:14:03
> > > An: "[email protected]" <[email protected]>
>
> >
> > >
> > > Betreff: st: RE: Ousout problem psmatch2
> > >
> > > > Jason, your first question seems easy to solve: I think your
> treatment
> >
> > > variable
> > > > and the outcome variable are identical. Recall the syntax
> for psmatch2,
> >
> > > see
> > > > -help psmatch2-.
> > > >
> > > > The second point left me a bit confused. Searching for -attnd-
>
> > using
> > > -findit
> > > > attnd- didn't really turn out any hints. Is -attnd- available
> from
> > SSC?
> > > Maybe
> > > > you can clarify your steps there? Maybe differences are due
> to
> > the issue
> > >
> > > > with the first question.
> > > >
> > > > Hope it helps
> > > > Jan Bryla
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: [email protected] [mailto:[email protected]]
>
> >
> > >
> > > > On Behalf Of Jason Zarmulski
> > > > Sent: 20. juni 2011 16:45
> > > > To: [email protected]
> > > > Subject: st: Ousout problem psmatch2
> > > >
> > > > Dear Statalists,
> > > > I got two problems, first one:
> > > >
> > > > I have a problem understanding the Output of my psmatch2 results.
>
> > I'm
> > > new
> > > > to
> > > > this so it could be a trivial error, but I'm not sure. I wanted
>
> > to do
> > > a
> > > > nearest neighbor matching with replacement.
> > > > My results are:
> > > >
> > > > . psmatch2 stockprice2010, outcome( stockprice2010) pscore(mypscore)
>
> >
> > >
> > > > neighbor(10) caliper(0.1) ate
> > > > ----------------------------------------------------------------------------------------
>
> >
> > >
> > > >
> > > > Variable Sample | Treated Controls Difference
>
> >
> > > > S.E. T-stat
> > > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > >
> > > > stockprice2010 Unmatched | 1.628 .479999997 .99964329
>
> >
> > > > .000849677 1176.50
> > > > ATT | 1.62 .689999992 .930000013
>
> >
> > >
> > > > .399478823 2.33
> > > > ATU | .479999997 1.79500002 1.31500002
>
> >
> > > > . .
> > > > ATE | 1.05833335
>
> >
> > >
> > > > . .
> > > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > >
> > > > Note: S.E. does not take into account that the propensity
> score
> > is
> > > > estimated.
> > > >
> > > > psmatch2: | psmatch2: Common
> > > > Treatment | support
> > > > assignment | Off suppo On suppor | Total
> > > > -----------+----------------------+----------
> > > > Untreated | 0 2 | 2
> > > > Treated | 1 4 | 5
> > > > 2 | 0 11 | 11
> > > > 3 | 0 12 | 12
> > > > 4 | 0 17 | 17
> > > > 5 | 0 34 | 34
> > > > 6 | 0 15 | 15
> > > > 7 | 0 19 | 19
> > > > 8 | 0 18 | 18
> > > > 9 | 0 17 | 17
> > > > 10 | 0 18 | 18
> > > > 11 | 0 13 | 13
> > > > 12 | 0 13 | 13
> > > > 13 | 0 20 | 20
> > > > 14 | 0 23 | 23
> > > > 15 | 0 23 | 23
> > > > 16 | 0 10 | 10
> > > > 17 | 0 13 | 13
> > > > 18 | 0 10 | 10
> > > > 19 | 0 11 | 11
> > > > 20 | 0 11 | 11
> > > > 21 | 0 10 | 10
> > > > 22 | 0 10 | 10
> > > > 23 | 0 11 | 11
> > > > 24 | 0 10 | 10
> > > > 25 | 0 7 | 7
> > > > 26 | 0 9 | 9
> > > > 27 | 0 6 | 6
> > > > 28 | 0 7 | 7
> > > > 29 | 0 4 | 4
> > > > 30 | 0 7 | 7
> > > > 31 | 0 6 | 6
> > > > 32 | 0 7 | 7
> > > > 33 | 0 7 | 7
> > > > 34 | 0 5 | 5
> > > > 35 | 0 6 | 6
> > > > 36 | 0 6 | 6
> > > > 37 | 0 6 | 6
> > > > 38 | 0 5 | 5
> > > > 39 | 0 4 | 4
> > > > 40 | 0 4 | 4
> > > > 41 | 0 9 | 9
> > > > 42 | 0 7 | 7
> > > > 43 | 0 5 | 5
> > > > 44 | 0 1 | 1
> > > > 45 | 0 1 | 1
> > > > 46 | 0 1 | 1
> > > > 47 | 0 1 | 1
> > > > 48 | 0 3 | 3
> > > > 49 | 0 4 | 4
> > > > 50 | 0 1 | 1
> > > > 51 | 0 1 | 1
> > > > 52 | 0 1 | 1
> > > > 53 | 0 1 | 1
> > > > 55 | 0 1 | 1
> > > > 56 | 0 3 | 3
> > > > 57 | 0 1 | 1
> > > > 59 | 0 1 | 1
> > > > 60 | 0 2 | 2
> > > > 62 | 0 1 | 1
> > > > 63 | 0 2 | 2
> > > > 69 | 0 1 | 1
> > > > 70 | 0 1 | 1
> > > > 78 | 0 1 | 1
> > > > 83 | 0 1 | 1
> > > > 84 | 0 2 | 2
> > > > 92 | 0 1 | 1
> > > > -----------+----------------------+----------
> > > > Total | 1 505 | 506
> > > >
> > > > Question: Why arent there only the untreated and treated in
> the
> > table
> > > and
> > > >
> > > > what ist the meaning of the numbers 2-92?
> > > >
> > > >
> > > > Second problem:
> > > >
> > > > I've done the nearest neighbor matching of the same data sample
>
> > with
> > > the
> > > >
> > > > attnd function.
> > > > My results are:
> > > >
> > > > . attnd stockprice2010 reldummy, pscore(mypscore)
> > > >
> > > >
> > > > The program is searching the nearest neighbor of each treated
>
> > unit.
> > >
> > > > This operation may take a while.
> > > >
> > > >
> > > >
> > > > ATT estimation with Nearest Neighbor Matching method
> > > > (random draw version)
> > > > Analytical standard errors
> > > >
> > > > ---------------------------------------------------------
>
> > > > n. treat. n. contr. ATT Std. Err. t
>
> > > > ---------------------------------------------------------
>
> > > >
> > > > 424 63 -2.255 6.070 -0.371
>
> > > >
> > > > ---------------------------------------------------------
>
> > > > Note: the numbers of treated and controls refer to actual
>
> > > > nearest neighbour matches
> > > >
> > > > Why is this result for ATT so different to the one of psmatch2?
>
> >
> > > > Is the attnd matching with replacement?
> > > >
> > > > Any suggestions would be much appreciated
> > > >
> > > > Thanks
> > > >
> > > > Jason
> > > >
> > > > --
> > > > View this message in context: http://statalist.1588530.n2.nabble.com/Ousout-problem-psmatch2-tp6496035p6496035.html
>
> >
> > >
> > > >
> > > > Sent from the Statalist mailing list archive at Nabble.com.
>
> > > > *
> > > > * For searches and help try:
> > > > * http://www.stata.com/help.cgi?search
> > > > * http://www.stata.com/support/statalist/faq
> > > > * http://www.ats.ucla.edu/stat/stata/
> > > >
> > > > *
> > > > * For searches and help try:
> > > > * http://www.stata.com/help.cgi?search
> > > > * http://www.stata.com/support/statalist/faq
> > > > * http://www.ats.ucla.edu/stat/stata/
> > >
> > > *
> > > * For searches and help try:
> > > * http://www.stata.com/help.cgi?search
> > > * http://www.stata.com/support/statalist/faq
> > > * http://www.ats.ucla.edu/stat/stata/
> > >
> > > *
> > > * For searches and help try:
> > > * http://www.stata.com/help.cgi?search
> > > * http://www.stata.com/support/statalist/faq
> > > * http://www.ats.ucla.edu/stat/stata/
> >
> > *
> > * For searches and help try:
> > * http://www.stata.com/help.cgi?search
> > * http://www.stata.com/support/statalist/faq
> > * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/