Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: RE: Output problem attnd , attr, atts


From   "Jason Zarmulski" <Jason.Zarmulski@mtox.de>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: RE: Output problem attnd , attr, atts
Date   Thu, 30 Jun 2011 14:27:51 +0200

Thanks Jan

--- Ursprüngliche Nachricht ---
Von: Jan Bryla <JBR@finansraadet.dk>
Datum: 28.06.2011 21:00:07
An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
Betreff: st: RE: Output problem attnd , attr, atts

> Jason,
>
> It seems to me that the interpretation is standard to the treatment literature
> (average treatment effect on the treated). Differences arise because you
> are applying different "methods". In that respect it is not surprising
> to me, that you obtain different results. I guess the underlying reason the
> methods produce rather dissimilar results has to do with the distribution
> of the outcome of interest and the propensity score.
>
> If memory serves -pstest- works only after -psmatch2-, but maybe other listers
> can confirm/deny?
>
> Regarding robustness I would definetely check if these results are robust
> to various potential problems.
>
> Hope it helps - there are good introductions to these methods around on the
> web.
>
> Jan Bryla
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu]
> On Behalf Of Jason Zarmulski
> Sent: 28. juni 2011 11:10
> To: statalist@hsphsun2.harvard.edu
> Subject: st: Output problem attnd , attr, atts
>
>  Hi Statalists,
> >
> >
> > Now I have the problem with interpreting the different outcomes.
>
> My Questions:
> >
> > -How can I Interpret the different Outcomes?
> > -Why isnt the .pstest method working with these outcomes?
> > -Are these Outcomes my the final ones, or do I have to do some robustness
>
> > tests as well? (If yes, which one?)
> >
> >
>
> > My new Propensity Score and the different outcomes are:
> >
> >
> > The balancing property is satisfied
> >
> >
> > This table shows the inferior bound, the number of treated
> > and the number of controls for each block
> >
> >   Inferior |
> >   of block |      REL(Dummy)
> > of pscore  |         0          1 |     Total
> > -----------+----------------------+----------
> >         .2 |         5          6 |        11
> >         .4 |        16         21 |        37
> >         .6 |        20         17 |        37
> >         .7 |        14         47 |        61
> >         .8 |        19        110 |       129
> >         .9 |         9        223 |       232
> > -----------+----------------------+----------
> >      Total |        83        424 |       507
> >
> >
> >
> > . attnd stockprice2010 reldummy, pscore(mypscore)comsup
> >
> >
> >  The program is searching the nearest neighbor of each treated unit.
>
> >  This operation may take a while.
> >
> >
> >
> > ATT estimation with Nearest Neighbor Matching method
> > (random draw version)
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat.   n. contr.         ATT   Std. Err.           t
> > ---------------------------------------------------------
> >
> >       424          65      -0.572       6.828      -0.084
> >
> > ---------------------------------------------------------
> > Note: the numbers of treated and controls refer to actual
> > nearest neighbour matches
> >
> >
> >
> > . attr stockprice2010 reldummy, pscore(mypscore) radius(0.1) comsup
>
> >
> >
> >  The program is searching for matches of treated units within radius.
>
> >  This operation may take a while.
> >
> >
> >
> > ATT estimation with the Radius Matching method
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat.   n. contr.         ATT   Std. Err.           t
> > ---------------------------------------------------------
> >
> >       424          83       1.354       3.588       0.377
> >
> > ---------------------------------------------------------
> > Note: the numbers of treated and controls refer to actual
> > matches within radius
> >
> >
> > . atts stockprice2010 reldummy, pscore(mypscore) comsup blockid(myblock)
>
> >
> >
> >
> >
> > ATT estimation with the Stratification method
> > Analytical standard errors
> >
> > ---------------------------------------------------------
> > n. treat.   n. contr.         ATT   Std. Err.           t
> > ---------------------------------------------------------
> >
> >       424          83       0.273       2.520       0.109
> >
> > ---------------------------------------------------------
> >
> >Thanks
> Jason
> >
> >
> >
> > --- Ursprüngliche Nachrsht ---
> > Von: Jan Bryla <JBR@finansraadet.dk>
> > Datum: 21.06.2011 15:04:02
> > An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
>
> >
> > Betreff: RE: st: RE: Ousout problem psmatch2
> >
> > > Ah, yes. You are right - my mistake.
> > >
> > > Regarding -attnd- I see a few reasons why you can get different
> results:
> >
> > >
> > > - using -psmatch2- you "restrict" to 10 neighbours. What
> is
> > the
> > > number og nearest neighbours selected by -attnd-?
> > > - you set caliper to 0.1 using psmatch2. But I don't think a similar
>
> > "restriction"
> > > is imposed in attnd?
> > >
> > > Maybe this is a start...
> > >
> > > /Jan
> > >
> > >
> > > -----Original Message-----
> > > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu]
>
> >
> > > On Behalf Of Jason Zarmulski
> > > Sent: 21. juni 2011 12:23
> > > To: statalist@hsphsun2.harvard.edu
> > > Subject: Re: st: RE: Ousout problem psmatch2
> > >
> > > Hi Jan,
> > >
> > > thanks a lot for your support, I changed it and my new output is:
>
> > > :-)
> > >
> > > . psmatch2 reldummy , outcome( stockprice2010) pscore(mypscore)
> neighbor(10)
> >
> > > caliper(0.1) ate
> > > ----------------------------------------------------------------------------------------
>
> >
> > >
> > >         Variable     Sample |    Treated     Controls   Difference
>
> > >   S.E.   T-stat
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > >   stockprice2010  Unmatched | 20.6236969   21.7278571  -1.10416018
>
> >  2.08916252
> > >    -0.53
> > >                         ATT | 20.6236969   20.4730466   .150650303
>
> >  4.16715274
> > >     0.04
> > >                         ATU | 21.7278571   17.8625509  -3.86530621
>
> > >      .        .
> > >                         ATE |                          -.513405498
>
> >
> > >      .        .
> > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > Note: S.E. does not take into account that the propensity score
> is estimated.
> >
> > >
> > >
> > >            | psmatch2:
> > >  psmatch2: |   Common
> > >  Treatment |  support
> > > assignment | On suppor |     Total
> > > -----------+-----------+----------
> > >  Untreated |        84 |        84
> > >    Treated |       424 |       424
> > > -----------+-----------+----------
> > >      Total |       508 |       508
> > >
> > >
> > > To the attnd problem : I didnt install a package for attnd. Maybe
> it
> > was
> > > with the"nnmatch ado" package
> > >
> > > The differences between the psmatch2 and the attnd are still there....
>
> >
> > >
> > > This is the help attnd file:
> > >
> > >
> > > Calculate the average treatment effect on the treated using nearest
>
> > neighbor
> > > matching
> > >
> > >             attnd outcome treatment [varlist] [weight] [if exp]
> [in
> > range]
> > > [ ,
> > >                   pscore(scorevar) logit index comsup detail bootstrap
>
> > reps(#)
> > >
> > >                   noisily dots ]
> > >
> > >         fweights, iweights, and pweights are allowed; see help
> weights.
> >
> > >
> > >
> > > Description
> > >
> > >     attnd estimates the average treatment effect on the treated
> (ATT)
> > using
> > > nearest
> > >     neighbor matching. attnd should be run after the correct propensity
>
> > score
> > >
> > >     specification; i.e., the one satisfying the balancing property
> has
> > been
> > > found
> > >     using, for example, pscore. If users do not provide a variable
> name
> > for
> > > the
> > >     propensity score, the propensity score is estimated based on
> the
> > specification
> > >
> > >     in varlist. Note that in this case the balancing property is
> not
> > tested.
> > >
> > >
> > >     It is left under the responsibility of the user to select the
> comsup
> >
> > > option if
> > >     the user provided propensity score has been estimated on a
> common
> > support
> > > for
> > >     treated and controls.  Otherwise, the ATT is estimated using
> also
> > the
> > >
> > >     observations outside the common support for which the propensity
>
> > score
> > > may not
> > >     be balanced.
> > >
> > >     To save on computing time, nearest neighbors are not determined
>
> > by comparing
> > >
> > >     treated observations to every single control, but by first
> sorting
> > all
> > > records
> > >     by the estimated propensity score and then searching forward
> and
> > backward
> > > for
> > >     the closest control unit(s). If a treated unit forward and
> backward
> > matches
> > >
> > >     happen to be equally good, this program randomly draws (hence
> the
> > letters
> > > "nd"
> > >     for Nearest neighbor and random Draw) either the forward or
> backward
> >
> > > matches.
> > >     This approach is one of two computationally feasible options
> to
> > obtain
> > >
> > >     analytical standard errors while at the same time exploiting
> the
> > very
> > > fast
> > >     forward and backward search strategy. The second possibility
> is
> > based
> > > on giving
> > >     equal weight to the groups of forward and backward matches
> in case
> > of
> > > equally
> > >     good forward and backward matches and is performed by attnw.
> In
> > practice,
> > > the
> > >     case of multiple nearest neighbors should be very rare. In
> particular,
> >
> > > if the
> > >     set of X's contains continuous variables, in which case, both
> attnd
> > and
> > > attnw
> > >     should give equal results (except for bootstrapped standard
> errors).
> >
> > > The
> > >     likelihood of multiple nearest neighbors is further reduced
> if the
> > propensity
> > >
> > >     score is estimated and saved in double precision, which is
> what
> > pscore
> > > does by
> > >     default.
> > >
> > >     The ATT is computed by averaging over the unit-level treatment
> effects
> >
> > > of the
> > >     treated where the control(s) matched to a treated observation
> is/are
> >
> > > those
> > >     observations in the control group that have the closest propensity
>
> > score.
> > > If
> > >     there are multiple nearest neighbors, the average outcome of
> those
> > controls
> > > is
> > >     used.
> > >
> > >
> > > Options
> > >
> > >     pscore(scorevar) specifies the name of the user-provided variable
>
> > name
> > > for the
> > >         estimated propensity score. If no name is provided the
> propensity
> >
> > > score is
> > >         estimated based on the specification in varlist.
> > >
> > >     logit uses a logit model to estimate the propensity score instead
>
> > of
> > > the default
> > >         probit model when the option pscore(scorevar) is not specified
>
> > by
> > > the user.
> > >         Otherwise, no effect is produced.
> > >
> > >     index requires the use of the linear index as the propensity
> score
> > when
> > > the
> > >         option pscore(scorevar) is not specified by the user.
> Otherwise,
> >
> > > no effect
> > >         is produced.
> > >
> > >     comsup restricts the computation of the ATT to the region of
> common
> > support.
> > >
> > >
> > >     detail displays more detailed output documenting the steps
> performed
> >
> > > to obtain
> > >         the final results.
> > >
> > >     bootstrap bootstraps the standard error of the treatment effect.
>
> >
> > >
> > >     reps(#) specifies the number of bootstrap replications to be
> performed.
> >
> > >  The
> > >         default is 50.  This option produces an effect only if
> the bootstrap
> >
> > > option
> > >         is specified.
> > >
> > >     noisily requests that any output from the replications be displayed.
>
> >
> > >  This
> > >         option produces an effect only if the bootstrap option
> is specified.
> >
> > >
> > >
> > >     dots requests that a dot be placed on the screen at the beginning
>
> > of
> > > each
> > >         replication.  This option produces an effect only if the
> bootstrap
> >
> > > option is
> > >         specified.
> > >
> > >
> > > Remarks
> > >
> > >     Please remember to use the update query command before running
> this
> > program
> > > to
> > >     make sure you have an up-to-date version of Stata installed.
>  Otherwise,
> >
> > > this
> > >     program may not run properly.
> > >
> > >     The treatment has to be binary.
> > >
> > >     When users do not specify their own previously estimated propensity
>
> > score,
> > > the
> > >     bootstrap encompasses the estimation of the propensity score
> based
> > on
> > > the
> > >     specification given by varlist. This procedure is actually
> recommended
> >
> > > to
> > >     account for the uncertainty associated with the estimation
> of the
> > propensity
> > >
> > >     score.  Even more so when the comsup option is specified because
>
> > in this
> > > case
> > >     the region of common support changes with every bootstrap sample,
>
> > and
> > >
> > >     bootstrapped standard errors pick up this uncertainty as well.
> So,
> > typically
> > >
> > >     users would first identify a specification satisfying the balancing
>
> > property
> > > --
> > >     using pscore -- and then provide exactly this specification
> in varlist
> >
> > > and use
> > >     bootstrapped standard errors.
> > >
> > >
> > > Saved results
> > >
> > >     The program stores the estimated treatment effect, its standard
>
> > error,
> > > and the t
> > >     statistic respectively in the scalars r(attnd), r(seattnd),
> and
> > r(tsattnd).
> > >
> > >
> > >     The number of treated and the number of controls are stored
> respectively
> >
> > > in the
> > >     scalars r(ntnd) and r(ncnd).
> > >
> > >     The bootstrapped standard error and t statistic are stored
> respectively
> >
> > > in the
> > >     scalars r(bseattnd) and r(btsattnd).
> > >
> > >
> > > Examples
> > >
> > >     . attnd wage training age age2 exp exp2
> > >
> > >     . attnd wage training age age2 exp exp2, boot reps(100) dots
>
> > >
> > >     . attnd wage training age age2 exp exp2, logit boot reps(100)
>
> > >
> > >     . attnd wage training age age2 exp exp2, comsup boot reps(100)
>
> > >
> > >
> > > Authors
> > >
> > >         Sascha O. Becker
> > >         Center for Economic Studies, University of Munich
> > >
> > >         Andrea Ichino
> > >         Department of Economics, European University Institute,
> Florence
> >
> > >
> > >
> > >     Email so.b@gmx.net or andrea.ichino@iue.it if you observe any
> problems.
> >
> > >
> > >
> > >
> > > Acknowledgments
> > >
> > >     The way to implement the propensity score estimation in the
> bootstrap
> >
> > > procedure
> > >     has been adapted from the psmatch program written by Barbara
> Sianesi
> >
> > > (University
> > >     College London and Institute for Fiscal Studies) Email: barbara_s@ifs.org.uk.
>
> >
> > >
> > >
> > >
> > > Also see
> > >
> > >     Online:  help for pscore, atts, attr, attk, attnw (if installed),
>
> > and
> > > bs.
> > >
> > >               Further details on the analytical formulas and on
> the
> > algorithms
> > > used
> > >               in these programs can be found under http://www.sobecker.de
>
> >
> > > or
> > >               http://www.iue.it/Personal/Ichino.
> > >
> > > Thanks
> > > Jason
> > >
> > >
> > > --- Ursprüngliche Nachricht ---
> > > Von: Jan Bryla <JBR@finansraadet.dk>
> > > Datum: 20.06.2011 20:14:03
> > > An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
>
> >
> > >
> > > Betreff: st: RE: Ousout problem psmatch2
> > >
> > > > Jason, your first question seems easy to solve: I think your
> treatment
> >
> > > variable
> > > > and the outcome variable are identical. Recall the syntax
> for psmatch2,
> >
> > > see
> > > > -help psmatch2-.
> > > >
> > > > The second point left me a bit confused. Searching for -attnd-
>
> > using
> > > -findit
> > > > attnd- didn't really turn out any hints. Is -attnd- available
> from
> > SSC?
> > > Maybe
> > > > you can clarify your steps there? Maybe differences are due
> to
> > the issue
> > >
> > > > with the first question.
> > > >
> > > > Hope it helps
> > > > Jan Bryla
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu]
>
> >
> > >
> > > > On Behalf Of Jason Zarmulski
> > > > Sent: 20. juni 2011 16:45
> > > > To: statalist@hsphsun2.harvard.edu
> > > > Subject: st: Ousout problem psmatch2
> > > >
> > > > Dear Statalists,
> > > > I got two problems, first one:
> > > >
> > > > I have a problem understanding the Output of my psmatch2 results.
>
> > I'm
> > > new
> > > > to
> > > > this so it could be a trivial error, but I'm not sure. I wanted
>
> > to do
> > > a
> > > > nearest neighbor matching with replacement.
> > > > My results are:
> > > >
> > > > . psmatch2 stockprice2010, outcome( stockprice2010) pscore(mypscore)
>
> >
> > >
> > > > neighbor(10) caliper(0.1) ate
> > > > ----------------------------------------------------------------------------------------
>
> >
> > >
> > > >
> > > >         Variable     Sample |    Treated     Controls   Difference
>
> >
> > > > S.E.   T-stat
> > > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > >
> > > >   stockprice2010  Unmatched |      1.628   .479999997    .99964329
>
> >
> > > > .000849677  1176.50
> > > >                         ATT |       1.62   .689999992   .930000013
>
> >
> > >
> > > > .399478823     2.33
> > > >                         ATU | .479999997   1.79500002   1.31500002
>
> >
> > > > .        .
> > > >                         ATE |                           1.05833335
>
> >
> > >
> > > > .        .
> > > > ----------------------------+-----------------------------------------------------------
>
> >
> > >
> > > >
> > > > Note: S.E. does not take into account that the propensity
> score
> > is
> > > > estimated.
> > > >
> > > >  psmatch2: |   psmatch2: Common
> > > >  Treatment |        support
> > > > assignment | Off suppo  On suppor |     Total
> > > > -----------+----------------------+----------
> > > >  Untreated |         0          2 |         2
> > > >    Treated |         1          4 |         5
> > > >          2 |         0         11 |        11
> > > >          3 |         0         12 |        12
> > > >          4 |         0         17 |        17
> > > >          5 |         0         34 |        34
> > > >          6 |         0         15 |        15
> > > >          7 |         0         19 |        19
> > > >          8 |         0         18 |        18
> > > >          9 |         0         17 |        17
> > > >         10 |         0         18 |        18
> > > >         11 |         0         13 |        13
> > > >         12 |         0         13 |        13
> > > >         13 |         0         20 |        20
> > > >         14 |         0         23 |        23
> > > >         15 |         0         23 |        23
> > > >         16 |         0         10 |        10
> > > >         17 |         0         13 |        13
> > > >         18 |         0         10 |        10
> > > >         19 |         0         11 |        11
> > > >         20 |         0         11 |        11
> > > >         21 |         0         10 |        10
> > > >         22 |         0         10 |        10
> > > >         23 |         0         11 |        11
> > > >         24 |         0         10 |        10
> > > >         25 |         0          7 |         7
> > > >         26 |         0          9 |         9
> > > >         27 |         0          6 |         6
> > > >         28 |         0          7 |         7
> > > >         29 |         0          4 |         4
> > > >         30 |         0          7 |         7
> > > >         31 |         0          6 |         6
> > > >         32 |         0          7 |         7
> > > >         33 |         0          7 |         7
> > > >         34 |         0          5 |         5
> > > >         35 |         0          6 |         6
> > > >         36 |         0          6 |         6
> > > >         37 |         0          6 |         6
> > > >         38 |         0          5 |         5
> > > >         39 |         0          4 |         4
> > > >         40 |         0          4 |         4
> > > >         41 |         0          9 |         9
> > > >         42 |         0          7 |         7
> > > >         43 |         0          5 |         5
> > > >         44 |         0          1 |         1
> > > >         45 |         0          1 |         1
> > > >         46 |         0          1 |         1
> > > >         47 |         0          1 |         1
> > > >         48 |         0          3 |         3
> > > >         49 |         0          4 |         4
> > > >         50 |         0          1 |         1
> > > >         51 |         0          1 |         1
> > > >         52 |         0          1 |         1
> > > >         53 |         0          1 |         1
> > > >         55 |         0          1 |         1
> > > >         56 |         0          3 |         3
> > > >         57 |         0          1 |         1
> > > >         59 |         0          1 |         1
> > > >         60 |         0          2 |         2
> > > >         62 |         0          1 |         1
> > > >         63 |         0          2 |         2
> > > >         69 |         0          1 |         1
> > > >         70 |         0          1 |         1
> > > >         78 |         0          1 |         1
> > > >         83 |         0          1 |         1
> > > >         84 |         0          2 |         2
> > > >         92 |         0          1 |         1
> > > > -----------+----------------------+----------
> > > >      Total |         1        505 |       506
> > > >
> > > > Question: Why arent there only the untreated and treated in
> the
> > table
> > > and
> > > >
> > > > what ist the meaning of the numbers 2-92?
> > > >
> > > >
> > > > Second problem:
> > > >
> > > > I've done the nearest neighbor matching of the same data sample
>
> > with
> > > the
> > > >
> > > > attnd function.
> > > > My results are:
> > > >
> > > > . attnd  stockprice2010 reldummy, pscore(mypscore)
> > > >
> > > >
> > > >  The program is searching the nearest neighbor of each treated
>
> > unit.
> > >
> > > >  This operation may take a while.
> > > >
> > > >
> > > >
> > > > ATT estimation with Nearest Neighbor Matching method
> > > > (random draw version)
> > > > Analytical standard errors
> > > >
> > > > ---------------------------------------------------------
>
> > > > n. treat.   n. contr.         ATT   Std. Err.           t
>
> > > > ---------------------------------------------------------
>
> > > >
> > > >       424          63      -2.255       6.070      -0.371
>
> > > >
> > > > ---------------------------------------------------------
>
> > > > Note: the numbers of treated and controls refer to actual
>
> > > > nearest neighbour matches
> > > >
> > > > Why is this result for ATT so different to the one of psmatch2?
>
> >
> > > > Is the attnd matching with replacement?
> > > >
> > > > Any suggestions would be much appreciated
> > > >
> > > > Thanks
> > > >
> > > > Jason
> > > >
> > > > --
> > > > View this message in context: http://statalist.1588530.n2.nabble.com/Ousout-problem-psmatch2-tp6496035p6496035.html
>
> >
> > >
> > > >
> > > > Sent from the Statalist mailing list archive at Nabble.com.
>
> > > > *
> > > > *   For searches and help try:
> > > > *   http://www.stata.com/help.cgi?search
> > > > *   http://www.stata.com/support/statalist/faq
> > > > *   http://www.ats.ucla.edu/stat/stata/
> > > >
> > > > *
> > > > *   For searches and help try:
> > > > *   http://www.stata.com/help.cgi?search
> > > > *   http://www.stata.com/support/statalist/faq
> > > > *   http://www.ats.ucla.edu/stat/stata/
> > >
> > > *
> > > *   For searches and help try:
> > > *   http://www.stata.com/help.cgi?search
> > > *   http://www.stata.com/support/statalist/faq
> > > *   http://www.ats.ucla.edu/stat/stata/
> > >
> > > *
> > > *   For searches and help try:
> > > *   http://www.stata.com/help.cgi?search
> > > *   http://www.stata.com/support/statalist/faq
> > > *   http://www.ats.ucla.edu/stat/stata/
> >
> > *
> > *   For searches and help try:
> > *   http://www.stata.com/help.cgi?search
> > *   http://www.stata.com/support/statalist/faq
> > *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index