Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
st: RE: Output problem attnd , attr, atts
From
Jan Bryla <[email protected]>
To
"[email protected]" <[email protected]>
Subject
st: RE: Output problem attnd , attr, atts
Date
Tue, 28 Jun 2011 21:00:07 +0200
Jason,
It seems to me that the interpretation is standard to the treatment literature (average treatment effect on the treated). Differences arise because you are applying different "methods". In that respect it is not surprising to me, that you obtain different results. I guess the underlying reason the methods produce rather dissimilar results has to do with the distribution of the outcome of interest and the propensity score.
If memory serves -pstest- works only after -psmatch2-, but maybe other listers can confirm/deny?
Regarding robustness I would definetely check if these results are robust to various potential problems.
Hope it helps - there are good introductions to these methods around on the web.
Jan Bryla
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jason Zarmulski
Sent: 28. juni 2011 11:10
To: [email protected]
Subject: st: Output problem attnd , attr, atts
Hi Statalists,
>
>
> Now I have the problem with interpreting the different outcomes.
My Questions:
>
> -How can I Interpret the different Outcomes?
> -Why isnt the .pstest method working with these outcomes?
> -Are these Outcomes my the final ones, or do I have to do some robustness
> tests as well? (If yes, which one?)
>
>
> My new Propensity Score and the different outcomes are:
>
>
> The balancing property is satisfied
>
>
> This table shows the inferior bound, the number of treated
> and the number of controls for each block
>
> Inferior |
> of block | REL(Dummy)
> of pscore | 0 1 | Total
> -----------+----------------------+----------
> .2 | 5 6 | 11
> .4 | 16 21 | 37
> .6 | 20 17 | 37
> .7 | 14 47 | 61
> .8 | 19 110 | 129
> .9 | 9 223 | 232
> -----------+----------------------+----------
> Total | 83 424 | 507
>
>
>
> . attnd stockprice2010 reldummy, pscore(mypscore)comsup
>
>
> The program is searching the nearest neighbor of each treated unit.
> This operation may take a while.
>
>
>
> ATT estimation with Nearest Neighbor Matching method
> (random draw version)
> Analytical standard errors
>
> ---------------------------------------------------------
> n. treat. n. contr. ATT Std. Err. t
> ---------------------------------------------------------
>
> 424 65 -0.572 6.828 -0.084
>
> ---------------------------------------------------------
> Note: the numbers of treated and controls refer to actual
> nearest neighbour matches
>
>
>
> . attr stockprice2010 reldummy, pscore(mypscore) radius(0.1) comsup
>
>
> The program is searching for matches of treated units within radius.
> This operation may take a while.
>
>
>
> ATT estimation with the Radius Matching method
> Analytical standard errors
>
> ---------------------------------------------------------
> n. treat. n. contr. ATT Std. Err. t
> ---------------------------------------------------------
>
> 424 83 1.354 3.588 0.377
>
> ---------------------------------------------------------
> Note: the numbers of treated and controls refer to actual
> matches within radius
>
>
> . atts stockprice2010 reldummy, pscore(mypscore) comsup blockid(myblock)
>
>
>
>
> ATT estimation with the Stratification method
> Analytical standard errors
>
> ---------------------------------------------------------
> n. treat. n. contr. ATT Std. Err. t
> ---------------------------------------------------------
>
> 424 83 0.273 2.520 0.109
>
> ---------------------------------------------------------
>
>Thanks
Jason
>
>
>
> --- Ursprüngliche Nachrsht ---
> Von: Jan Bryla <[email protected]>
> Datum: 21.06.2011 15:04:02
> An: "[email protected]" <[email protected]>
>
> Betreff: RE: st: RE: Ousout problem psmatch2
>
> > Ah, yes. You are right - my mistake.
> >
> > Regarding -attnd- I see a few reasons why you can get different results:
>
> >
> > - using -psmatch2- you "restrict" to 10 neighbours. What is
> the
> > number og nearest neighbours selected by -attnd-?
> > - you set caliper to 0.1 using psmatch2. But I don't think a similar
> "restriction"
> > is imposed in attnd?
> >
> > Maybe this is a start...
> >
> > /Jan
> >
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]]
>
> > On Behalf Of Jason Zarmulski
> > Sent: 21. juni 2011 12:23
> > To: [email protected]
> > Subject: Re: st: RE: Ousout problem psmatch2
> >
> > Hi Jan,
> >
> > thanks a lot for your support, I changed it and my new output is:
> > :-)
> >
> > . psmatch2 reldummy , outcome( stockprice2010) pscore(mypscore) neighbor(10)
>
> > caliper(0.1) ate
> > ----------------------------------------------------------------------------------------
>
> >
> > Variable Sample | Treated Controls Difference
> > S.E. T-stat
> > ----------------------------+-----------------------------------------------------------
>
> >
> > stockprice2010 Unmatched | 20.6236969 21.7278571 -1.10416018
> 2.08916252
> > -0.53
> > ATT | 20.6236969 20.4730466 .150650303
> 4.16715274
> > 0.04
> > ATU | 21.7278571 17.8625509 -3.86530621
> > . .
> > ATE | -.513405498
>
> > . .
> > ----------------------------+-----------------------------------------------------------
>
> >
> > Note: S.E. does not take into account that the propensity score is estimated.
>
> >
> >
> > | psmatch2:
> > psmatch2: | Common
> > Treatment | support
> > assignment | On suppor | Total
> > -----------+-----------+----------
> > Untreated | 84 | 84
> > Treated | 424 | 424
> > -----------+-----------+----------
> > Total | 508 | 508
> >
> >
> > To the attnd problem : I didnt install a package for attnd. Maybe it
> was
> > with the"nnmatch ado" package
> >
> > The differences between the psmatch2 and the attnd are still there....
>
> >
> > This is the help attnd file:
> >
> >
> > Calculate the average treatment effect on the treated using nearest
> neighbor
> > matching
> >
> > attnd outcome treatment [varlist] [weight] [if exp] [in
> range]
> > [ ,
> > pscore(scorevar) logit index comsup detail bootstrap
> reps(#)
> >
> > noisily dots ]
> >
> > fweights, iweights, and pweights are allowed; see help weights.
>
> >
> >
> > Description
> >
> > attnd estimates the average treatment effect on the treated (ATT)
> using
> > nearest
> > neighbor matching. attnd should be run after the correct propensity
> score
> >
> > specification; i.e., the one satisfying the balancing property has
> been
> > found
> > using, for example, pscore. If users do not provide a variable name
> for
> > the
> > propensity score, the propensity score is estimated based on the
> specification
> >
> > in varlist. Note that in this case the balancing property is not
> tested.
> >
> >
> > It is left under the responsibility of the user to select the comsup
>
> > option if
> > the user provided propensity score has been estimated on a common
> support
> > for
> > treated and controls. Otherwise, the ATT is estimated using also
> the
> >
> > observations outside the common support for which the propensity
> score
> > may not
> > be balanced.
> >
> > To save on computing time, nearest neighbors are not determined
> by comparing
> >
> > treated observations to every single control, but by first sorting
> all
> > records
> > by the estimated propensity score and then searching forward and
> backward
> > for
> > the closest control unit(s). If a treated unit forward and backward
> matches
> >
> > happen to be equally good, this program randomly draws (hence the
> letters
> > "nd"
> > for Nearest neighbor and random Draw) either the forward or backward
>
> > matches.
> > This approach is one of two computationally feasible options to
> obtain
> >
> > analytical standard errors while at the same time exploiting the
> very
> > fast
> > forward and backward search strategy. The second possibility is
> based
> > on giving
> > equal weight to the groups of forward and backward matches in case
> of
> > equally
> > good forward and backward matches and is performed by attnw. In
> practice,
> > the
> > case of multiple nearest neighbors should be very rare. In particular,
>
> > if the
> > set of X's contains continuous variables, in which case, both attnd
> and
> > attnw
> > should give equal results (except for bootstrapped standard errors).
>
> > The
> > likelihood of multiple nearest neighbors is further reduced if the
> propensity
> >
> > score is estimated and saved in double precision, which is what
> pscore
> > does by
> > default.
> >
> > The ATT is computed by averaging over the unit-level treatment effects
>
> > of the
> > treated where the control(s) matched to a treated observation is/are
>
> > those
> > observations in the control group that have the closest propensity
> score.
> > If
> > there are multiple nearest neighbors, the average outcome of those
> controls
> > is
> > used.
> >
> >
> > Options
> >
> > pscore(scorevar) specifies the name of the user-provided variable
> name
> > for the
> > estimated propensity score. If no name is provided the propensity
>
> > score is
> > estimated based on the specification in varlist.
> >
> > logit uses a logit model to estimate the propensity score instead
> of
> > the default
> > probit model when the option pscore(scorevar) is not specified
> by
> > the user.
> > Otherwise, no effect is produced.
> >
> > index requires the use of the linear index as the propensity score
> when
> > the
> > option pscore(scorevar) is not specified by the user. Otherwise,
>
> > no effect
> > is produced.
> >
> > comsup restricts the computation of the ATT to the region of common
> support.
> >
> >
> > detail displays more detailed output documenting the steps performed
>
> > to obtain
> > the final results.
> >
> > bootstrap bootstraps the standard error of the treatment effect.
>
> >
> > reps(#) specifies the number of bootstrap replications to be performed.
>
> > The
> > default is 50. This option produces an effect only if the bootstrap
>
> > option
> > is specified.
> >
> > noisily requests that any output from the replications be displayed.
>
> > This
> > option produces an effect only if the bootstrap option is specified.
>
> >
> >
> > dots requests that a dot be placed on the screen at the beginning
> of
> > each
> > replication. This option produces an effect only if the bootstrap
>
> > option is
> > specified.
> >
> >
> > Remarks
> >
> > Please remember to use the update query command before running this
> program
> > to
> > make sure you have an up-to-date version of Stata installed. Otherwise,
>
> > this
> > program may not run properly.
> >
> > The treatment has to be binary.
> >
> > When users do not specify their own previously estimated propensity
> score,
> > the
> > bootstrap encompasses the estimation of the propensity score based
> on
> > the
> > specification given by varlist. This procedure is actually recommended
>
> > to
> > account for the uncertainty associated with the estimation of the
> propensity
> >
> > score. Even more so when the comsup option is specified because
> in this
> > case
> > the region of common support changes with every bootstrap sample,
> and
> >
> > bootstrapped standard errors pick up this uncertainty as well. So,
> typically
> >
> > users would first identify a specification satisfying the balancing
> property
> > --
> > using pscore -- and then provide exactly this specification in varlist
>
> > and use
> > bootstrapped standard errors.
> >
> >
> > Saved results
> >
> > The program stores the estimated treatment effect, its standard
> error,
> > and the t
> > statistic respectively in the scalars r(attnd), r(seattnd), and
> r(tsattnd).
> >
> >
> > The number of treated and the number of controls are stored respectively
>
> > in the
> > scalars r(ntnd) and r(ncnd).
> >
> > The bootstrapped standard error and t statistic are stored respectively
>
> > in the
> > scalars r(bseattnd) and r(btsattnd).
> >
> >
> > Examples
> >
> > . attnd wage training age age2 exp exp2
> >
> > . attnd wage training age age2 exp exp2, boot reps(100) dots
> >
> > . attnd wage training age age2 exp exp2, logit boot reps(100)
> >
> > . attnd wage training age age2 exp exp2, comsup boot reps(100)
> >
> >
> > Authors
> >
> > Sascha O. Becker
> > Center for Economic Studies, University of Munich
> >
> > Andrea Ichino
> > Department of Economics, European University Institute, Florence
>
> >
> >
> > Email [email protected] or [email protected] if you observe any problems.
>
> >
> >
> >
> > Acknowledgments
> >
> > The way to implement the propensity score estimation in the bootstrap
>
> > procedure
> > has been adapted from the psmatch program written by Barbara Sianesi
>
> > (University
> > College London and Institute for Fiscal Studies) Email: [email protected].
>
> >
> >
> >
> > Also see
> >
> > Online: help for pscore, atts, attr, attk, attnw (if installed),
> and
> > bs.
> >
> > Further details on the analytical formulas and on the
> algorithms
> > used
> > in these programs can be found under http://www.sobecker.de
>
> > or
> > http://www.iue.it/Personal/Ichino.
> >
> > Thanks
> > Jason
> >
> >
> > --- Ursprüngliche Nachricht ---
> > Von: Jan Bryla <[email protected]>
> > Datum: 20.06.2011 20:14:03
> > An: "[email protected]" <[email protected]>
>
> >
> > Betreff: st: RE: Ousout problem psmatch2
> >
> > > Jason, your first question seems easy to solve: I think your treatment
>
> > variable
> > > and the outcome variable are identical. Recall the syntax for psmatch2,
>
> > see
> > > -help psmatch2-.
> > >
> > > The second point left me a bit confused. Searching for -attnd-
> using
> > -findit
> > > attnd- didn't really turn out any hints. Is -attnd- available from
> SSC?
> > Maybe
> > > you can clarify your steps there? Maybe differences are due to
> the issue
> >
> > > with the first question.
> > >
> > > Hope it helps
> > > Jan Bryla
> > >
> > >
> > > -----Original Message-----
> > > From: [email protected] [mailto:[email protected]]
>
> >
> > > On Behalf Of Jason Zarmulski
> > > Sent: 20. juni 2011 16:45
> > > To: [email protected]
> > > Subject: st: Ousout problem psmatch2
> > >
> > > Dear Statalists,
> > > I got two problems, first one:
> > >
> > > I have a problem understanding the Output of my psmatch2 results.
> I'm
> > new
> > > to
> > > this so it could be a trivial error, but I'm not sure. I wanted
> to do
> > a
> > > nearest neighbor matching with replacement.
> > > My results are:
> > >
> > > . psmatch2 stockprice2010, outcome( stockprice2010) pscore(mypscore)
>
> >
> > > neighbor(10) caliper(0.1) ate
> > > ----------------------------------------------------------------------------------------
>
> >
> > >
> > > Variable Sample | Treated Controls Difference
>
> > > S.E. T-stat
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > stockprice2010 Unmatched | 1.628 .479999997 .99964329
>
> > > .000849677 1176.50
> > > ATT | 1.62 .689999992 .930000013
>
> >
> > > .399478823 2.33
> > > ATU | .479999997 1.79500002 1.31500002
>
> > > . .
> > > ATE | 1.05833335
>
> >
> > > . .
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > Note: S.E. does not take into account that the propensity score
> is
> > > estimated.
> > >
> > > psmatch2: | psmatch2: Common
> > > Treatment | support
> > > assignment | Off suppo On suppor | Total
> > > -----------+----------------------+----------
> > > Untreated | 0 2 | 2
> > > Treated | 1 4 | 5
> > > 2 | 0 11 | 11
> > > 3 | 0 12 | 12
> > > 4 | 0 17 | 17
> > > 5 | 0 34 | 34
> > > 6 | 0 15 | 15
> > > 7 | 0 19 | 19
> > > 8 | 0 18 | 18
> > > 9 | 0 17 | 17
> > > 10 | 0 18 | 18
> > > 11 | 0 13 | 13
> > > 12 | 0 13 | 13
> > > 13 | 0 20 | 20
> > > 14 | 0 23 | 23
> > > 15 | 0 23 | 23
> > > 16 | 0 10 | 10
> > > 17 | 0 13 | 13
> > > 18 | 0 10 | 10
> > > 19 | 0 11 | 11
> > > 20 | 0 11 | 11
> > > 21 | 0 10 | 10
> > > 22 | 0 10 | 10
> > > 23 | 0 11 | 11
> > > 24 | 0 10 | 10
> > > 25 | 0 7 | 7
> > > 26 | 0 9 | 9
> > > 27 | 0 6 | 6
> > > 28 | 0 7 | 7
> > > 29 | 0 4 | 4
> > > 30 | 0 7 | 7
> > > 31 | 0 6 | 6
> > > 32 | 0 7 | 7
> > > 33 | 0 7 | 7
> > > 34 | 0 5 | 5
> > > 35 | 0 6 | 6
> > > 36 | 0 6 | 6
> > > 37 | 0 6 | 6
> > > 38 | 0 5 | 5
> > > 39 | 0 4 | 4
> > > 40 | 0 4 | 4
> > > 41 | 0 9 | 9
> > > 42 | 0 7 | 7
> > > 43 | 0 5 | 5
> > > 44 | 0 1 | 1
> > > 45 | 0 1 | 1
> > > 46 | 0 1 | 1
> > > 47 | 0 1 | 1
> > > 48 | 0 3 | 3
> > > 49 | 0 4 | 4
> > > 50 | 0 1 | 1
> > > 51 | 0 1 | 1
> > > 52 | 0 1 | 1
> > > 53 | 0 1 | 1
> > > 55 | 0 1 | 1
> > > 56 | 0 3 | 3
> > > 57 | 0 1 | 1
> > > 59 | 0 1 | 1
> > > 60 | 0 2 | 2
> > > 62 | 0 1 | 1
> > > 63 | 0 2 | 2
> > > 69 | 0 1 | 1
> > > 70 | 0 1 | 1
> > > 78 | 0 1 | 1
> > > 83 | 0 1 | 1
> > > 84 | 0 2 | 2
> > > 92 | 0 1 | 1
> > > -----------+----------------------+----------
> > > Total | 1 505 | 506
> > >
> > > Question: Why arent there only the untreated and treated in the
> table
> > and
> > >
> > > what ist the meaning of the numbers 2-92?
> > >
> > >
> > > Second problem:
> > >
> > > I've done the nearest neighbor matching of the same data sample
> with
> > the
> > >
> > > attnd function.
> > > My results are:
> > >
> > > . attnd stockprice2010 reldummy, pscore(mypscore)
> > >
> > >
> > > The program is searching the nearest neighbor of each treated
> unit.
> >
> > > This operation may take a while.
> > >
> > >
> > >
> > > ATT estimation with Nearest Neighbor Matching method
> > > (random draw version)
> > > Analytical standard errors
> > >
> > > ---------------------------------------------------------
> > > n. treat. n. contr. ATT Std. Err. t
> > > ---------------------------------------------------------
> > >
> > > 424 63 -2.255 6.070 -0.371
> > >
> > > ---------------------------------------------------------
> > > Note: the numbers of treated and controls refer to actual
> > > nearest neighbour matches
> > >
> > > Why is this result for ATT so different to the one of psmatch2?
>
> > > Is the attnd matching with replacement?
> > >
> > > Any suggestions would be much appreciated
> > >
> > > Thanks
> > >
> > > Jason
> > >
> > > --
> > > View this message in context: http://statalist.1588530.n2.nabble.com/Ousout-problem-psmatch2-tp6496035p6496035.html
>
> >
> > >
> > > Sent from the Statalist mailing list archive at Nabble.com.
> > > *
> > > * For searches and help try:
> > > * http://www.stata.com/help.cgi?search
> > > * http://www.stata.com/support/statalist/faq
> > > * http://www.ats.ucla.edu/stat/stata/
> > >
> > > *
> > > * For searches and help try:
> > > * http://www.stata.com/help.cgi?search
> > > * http://www.stata.com/support/statalist/faq
> > > * http://www.ats.ucla.edu/stat/stata/
> >
> > *
> > * For searches and help try:
> > * http://www.stata.com/help.cgi?search
> > * http://www.stata.com/support/statalist/faq
> > * http://www.ats.ucla.edu/stat/stata/
> >
> > *
> > * For searches and help try:
> > * http://www.stata.com/help.cgi?search
> > * http://www.stata.com/support/statalist/faq
> > * http://www.ats.ucla.edu/stat/stata/
>
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/statalist/faq
> * http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/