Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
| From | "Jason Zarmulski" <Jason.Zarmulski@mtox.de> |
| To | statalist@hsphsun2.harvard.edu |
| Subject | Re: st: RE: Output problem attnd , attr, atts |
| Date | Thu, 30 Jun 2011 14:27:51 +0200 |
Thanks Jan --- Ursprüngliche Nachricht --- Von: Jan Bryla <JBR@finansraadet.dk> Datum: 28.06.2011 21:00:07 An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu> Betreff: st: RE: Output problem attnd , attr, atts > Jason, > > It seems to me that the interpretation is standard to the treatment literature > (average treatment effect on the treated). Differences arise because you > are applying different "methods". In that respect it is not surprising > to me, that you obtain different results. I guess the underlying reason the > methods produce rather dissimilar results has to do with the distribution > of the outcome of interest and the propensity score. > > If memory serves -pstest- works only after -psmatch2-, but maybe other listers > can confirm/deny? > > Regarding robustness I would definetely check if these results are robust > to various potential problems. > > Hope it helps - there are good introductions to these methods around on the > web. > > Jan Bryla > > > -----Original Message----- > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] > On Behalf Of Jason Zarmulski > Sent: 28. juni 2011 11:10 > To: statalist@hsphsun2.harvard.edu > Subject: st: Output problem attnd , attr, atts > > Hi Statalists, > > > > > > Now I have the problem with interpreting the different outcomes. > > My Questions: > > > > -How can I Interpret the different Outcomes? > > -Why isnt the .pstest method working with these outcomes? > > -Are these Outcomes my the final ones, or do I have to do some robustness > > > tests as well? (If yes, which one?) > > > > > > > My new Propensity Score and the different outcomes are: > > > > > > The balancing property is satisfied > > > > > > This table shows the inferior bound, the number of treated > > and the number of controls for each block > > > > Inferior | > > of block | REL(Dummy) > > of pscore | 0 1 | Total > > -----------+----------------------+---------- > > .2 | 5 6 | 11 > > .4 | 16 21 | 37 > > .6 | 20 17 | 37 > > .7 | 14 47 | 61 > > .8 | 19 110 | 129 > > .9 | 9 223 | 232 > > -----------+----------------------+---------- > > Total | 83 424 | 507 > > > > > > > > . attnd stockprice2010 reldummy, pscore(mypscore)comsup > > > > > > The program is searching the nearest neighbor of each treated unit. > > > This operation may take a while. > > > > > > > > ATT estimation with Nearest Neighbor Matching method > > (random draw version) > > Analytical standard errors > > > > --------------------------------------------------------- > > n. treat. n. contr. ATT Std. Err. t > > --------------------------------------------------------- > > > > 424 65 -0.572 6.828 -0.084 > > > > --------------------------------------------------------- > > Note: the numbers of treated and controls refer to actual > > nearest neighbour matches > > > > > > > > . attr stockprice2010 reldummy, pscore(mypscore) radius(0.1) comsup > > > > > > > The program is searching for matches of treated units within radius. > > > This operation may take a while. > > > > > > > > ATT estimation with the Radius Matching method > > Analytical standard errors > > > > --------------------------------------------------------- > > n. treat. n. contr. ATT Std. Err. t > > --------------------------------------------------------- > > > > 424 83 1.354 3.588 0.377 > > > > --------------------------------------------------------- > > Note: the numbers of treated and controls refer to actual > > matches within radius > > > > > > . atts stockprice2010 reldummy, pscore(mypscore) comsup blockid(myblock) > > > > > > > > > > > ATT estimation with the Stratification method > > Analytical standard errors > > > > --------------------------------------------------------- > > n. treat. n. contr. ATT Std. Err. t > > --------------------------------------------------------- > > > > 424 83 0.273 2.520 0.109 > > > > --------------------------------------------------------- > > > >Thanks > Jason > > > > > > > > --- Ursprüngliche Nachrsht --- > > Von: Jan Bryla <JBR@finansraadet.dk> > > Datum: 21.06.2011 15:04:02 > > An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu> > > > > > Betreff: RE: st: RE: Ousout problem psmatch2 > > > > > Ah, yes. You are right - my mistake. > > > > > > Regarding -attnd- I see a few reasons why you can get different > results: > > > > > > > > - using -psmatch2- you "restrict" to 10 neighbours. What > is > > the > > > number og nearest neighbours selected by -attnd-? > > > - you set caliper to 0.1 using psmatch2. But I don't think a similar > > > "restriction" > > > is imposed in attnd? > > > > > > Maybe this is a start... > > > > > > /Jan > > > > > > > > > -----Original Message----- > > > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] > > > > > > On Behalf Of Jason Zarmulski > > > Sent: 21. juni 2011 12:23 > > > To: statalist@hsphsun2.harvard.edu > > > Subject: Re: st: RE: Ousout problem psmatch2 > > > > > > Hi Jan, > > > > > > thanks a lot for your support, I changed it and my new output is: > > > > :-) > > > > > > . psmatch2 reldummy , outcome( stockprice2010) pscore(mypscore) > neighbor(10) > > > > > caliper(0.1) ate > > > ---------------------------------------------------------------------------------------- > > > > > > > > > Variable Sample | Treated Controls Difference > > > > S.E. T-stat > > > ----------------------------+----------------------------------------------------------- > > > > > > > > > stockprice2010 Unmatched | 20.6236969 21.7278571 -1.10416018 > > > 2.08916252 > > > -0.53 > > > ATT | 20.6236969 20.4730466 .150650303 > > > 4.16715274 > > > 0.04 > > > ATU | 21.7278571 17.8625509 -3.86530621 > > > > . . > > > ATE | -.513405498 > > > > > > . . > > > ----------------------------+----------------------------------------------------------- > > > > > > > > > Note: S.E. does not take into account that the propensity score > is estimated. > > > > > > > > > > > | psmatch2: > > > psmatch2: | Common > > > Treatment | support > > > assignment | On suppor | Total > > > -----------+-----------+---------- > > > Untreated | 84 | 84 > > > Treated | 424 | 424 > > > -----------+-----------+---------- > > > Total | 508 | 508 > > > > > > > > > To the attnd problem : I didnt install a package for attnd. Maybe > it > > was > > > with the"nnmatch ado" package > > > > > > The differences between the psmatch2 and the attnd are still there.... > > > > > > > > > This is the help attnd file: > > > > > > > > > Calculate the average treatment effect on the treated using nearest > > > neighbor > > > matching > > > > > > attnd outcome treatment [varlist] [weight] [if exp] > [in > > range] > > > [ , > > > pscore(scorevar) logit index comsup detail bootstrap > > > reps(#) > > > > > > noisily dots ] > > > > > > fweights, iweights, and pweights are allowed; see help > weights. > > > > > > > > > > > Description > > > > > > attnd estimates the average treatment effect on the treated > (ATT) > > using > > > nearest > > > neighbor matching. attnd should be run after the correct propensity > > > score > > > > > > specification; i.e., the one satisfying the balancing property > has > > been > > > found > > > using, for example, pscore. If users do not provide a variable > name > > for > > > the > > > propensity score, the propensity score is estimated based on > the > > specification > > > > > > in varlist. Note that in this case the balancing property is > not > > tested. > > > > > > > > > It is left under the responsibility of the user to select the > comsup > > > > > option if > > > the user provided propensity score has been estimated on a > common > > support > > > for > > > treated and controls. Otherwise, the ATT is estimated using > also > > the > > > > > > observations outside the common support for which the propensity > > > score > > > may not > > > be balanced. > > > > > > To save on computing time, nearest neighbors are not determined > > > by comparing > > > > > > treated observations to every single control, but by first > sorting > > all > > > records > > > by the estimated propensity score and then searching forward > and > > backward > > > for > > > the closest control unit(s). If a treated unit forward and > backward > > matches > > > > > > happen to be equally good, this program randomly draws (hence > the > > letters > > > "nd" > > > for Nearest neighbor and random Draw) either the forward or > backward > > > > > matches. > > > This approach is one of two computationally feasible options > to > > obtain > > > > > > analytical standard errors while at the same time exploiting > the > > very > > > fast > > > forward and backward search strategy. The second possibility > is > > based > > > on giving > > > equal weight to the groups of forward and backward matches > in case > > of > > > equally > > > good forward and backward matches and is performed by attnw. > In > > practice, > > > the > > > case of multiple nearest neighbors should be very rare. In > particular, > > > > > if the > > > set of X's contains continuous variables, in which case, both > attnd > > and > > > attnw > > > should give equal results (except for bootstrapped standard > errors). > > > > > The > > > likelihood of multiple nearest neighbors is further reduced > if the > > propensity > > > > > > score is estimated and saved in double precision, which is > what > > pscore > > > does by > > > default. > > > > > > The ATT is computed by averaging over the unit-level treatment > effects > > > > > of the > > > treated where the control(s) matched to a treated observation > is/are > > > > > those > > > observations in the control group that have the closest propensity > > > score. > > > If > > > there are multiple nearest neighbors, the average outcome of > those > > controls > > > is > > > used. > > > > > > > > > Options > > > > > > pscore(scorevar) specifies the name of the user-provided variable > > > name > > > for the > > > estimated propensity score. If no name is provided the > propensity > > > > > score is > > > estimated based on the specification in varlist. > > > > > > logit uses a logit model to estimate the propensity score instead > > > of > > > the default > > > probit model when the option pscore(scorevar) is not specified > > > by > > > the user. > > > Otherwise, no effect is produced. > > > > > > index requires the use of the linear index as the propensity > score > > when > > > the > > > option pscore(scorevar) is not specified by the user. > Otherwise, > > > > > no effect > > > is produced. > > > > > > comsup restricts the computation of the ATT to the region of > common > > support. > > > > > > > > > detail displays more detailed output documenting the steps > performed > > > > > to obtain > > > the final results. > > > > > > bootstrap bootstraps the standard error of the treatment effect. > > > > > > > > > reps(#) specifies the number of bootstrap replications to be > performed. > > > > > The > > > default is 50. This option produces an effect only if > the bootstrap > > > > > option > > > is specified. > > > > > > noisily requests that any output from the replications be displayed. > > > > > > This > > > option produces an effect only if the bootstrap option > is specified. > > > > > > > > > > > dots requests that a dot be placed on the screen at the beginning > > > of > > > each > > > replication. This option produces an effect only if the > bootstrap > > > > > option is > > > specified. > > > > > > > > > Remarks > > > > > > Please remember to use the update query command before running > this > > program > > > to > > > make sure you have an up-to-date version of Stata installed. > Otherwise, > > > > > this > > > program may not run properly. > > > > > > The treatment has to be binary. > > > > > > When users do not specify their own previously estimated propensity > > > score, > > > the > > > bootstrap encompasses the estimation of the propensity score > based > > on > > > the > > > specification given by varlist. This procedure is actually > recommended > > > > > to > > > account for the uncertainty associated with the estimation > of the > > propensity > > > > > > score. Even more so when the comsup option is specified because > > > in this > > > case > > > the region of common support changes with every bootstrap sample, > > > and > > > > > > bootstrapped standard errors pick up this uncertainty as well. > So, > > typically > > > > > > users would first identify a specification satisfying the balancing > > > property > > > -- > > > using pscore -- and then provide exactly this specification > in varlist > > > > > and use > > > bootstrapped standard errors. > > > > > > > > > Saved results > > > > > > The program stores the estimated treatment effect, its standard > > > error, > > > and the t > > > statistic respectively in the scalars r(attnd), r(seattnd), > and > > r(tsattnd). > > > > > > > > > The number of treated and the number of controls are stored > respectively > > > > > in the > > > scalars r(ntnd) and r(ncnd). > > > > > > The bootstrapped standard error and t statistic are stored > respectively > > > > > in the > > > scalars r(bseattnd) and r(btsattnd). > > > > > > > > > Examples > > > > > > . attnd wage training age age2 exp exp2 > > > > > > . attnd wage training age age2 exp exp2, boot reps(100) dots > > > > > > > . attnd wage training age age2 exp exp2, logit boot reps(100) > > > > > > > . attnd wage training age age2 exp exp2, comsup boot reps(100) > > > > > > > > > > Authors > > > > > > Sascha O. Becker > > > Center for Economic Studies, University of Munich > > > > > > Andrea Ichino > > > Department of Economics, European University Institute, > Florence > > > > > > > > > > > Email so.b@gmx.net or andrea.ichino@iue.it if you observe any > problems. > > > > > > > > > > > > > > Acknowledgments > > > > > > The way to implement the propensity score estimation in the > bootstrap > > > > > procedure > > > has been adapted from the psmatch program written by Barbara > Sianesi > > > > > (University > > > College London and Institute for Fiscal Studies) Email: barbara_s@ifs.org.uk. > > > > > > > > > > > > > > > Also see > > > > > > Online: help for pscore, atts, attr, attk, attnw (if installed), > > > and > > > bs. > > > > > > Further details on the analytical formulas and on > the > > algorithms > > > used > > > in these programs can be found under http://www.sobecker.de > > > > > > or > > > http://www.iue.it/Personal/Ichino. > > > > > > Thanks > > > Jason > > > > > > > > > --- Ursprüngliche Nachricht --- > > > Von: Jan Bryla <JBR@finansraadet.dk> > > > Datum: 20.06.2011 20:14:03 > > > An: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu> > > > > > > > > > Betreff: st: RE: Ousout problem psmatch2 > > > > > > > Jason, your first question seems easy to solve: I think your > treatment > > > > > variable > > > > and the outcome variable are identical. Recall the syntax > for psmatch2, > > > > > see > > > > -help psmatch2-. > > > > > > > > The second point left me a bit confused. Searching for -attnd- > > > using > > > -findit > > > > attnd- didn't really turn out any hints. Is -attnd- available > from > > SSC? > > > Maybe > > > > you can clarify your steps there? Maybe differences are due > to > > the issue > > > > > > > with the first question. > > > > > > > > Hope it helps > > > > Jan Bryla > > > > > > > > > > > > -----Original Message----- > > > > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] > > > > > > > > > > On Behalf Of Jason Zarmulski > > > > Sent: 20. juni 2011 16:45 > > > > To: statalist@hsphsun2.harvard.edu > > > > Subject: st: Ousout problem psmatch2 > > > > > > > > Dear Statalists, > > > > I got two problems, first one: > > > > > > > > I have a problem understanding the Output of my psmatch2 results. > > > I'm > > > new > > > > to > > > > this so it could be a trivial error, but I'm not sure. I wanted > > > to do > > > a > > > > nearest neighbor matching with replacement. > > > > My results are: > > > > > > > > . psmatch2 stockprice2010, outcome( stockprice2010) pscore(mypscore) > > > > > > > > > > neighbor(10) caliper(0.1) ate > > > > ---------------------------------------------------------------------------------------- > > > > > > > > > > > > > > Variable Sample | Treated Controls Difference > > > > > > > S.E. T-stat > > > > ----------------------------+----------------------------------------------------------- > > > > > > > > > > > > > > stockprice2010 Unmatched | 1.628 .479999997 .99964329 > > > > > > > .000849677 1176.50 > > > > ATT | 1.62 .689999992 .930000013 > > > > > > > > > > .399478823 2.33 > > > > ATU | .479999997 1.79500002 1.31500002 > > > > > > > . . > > > > ATE | 1.05833335 > > > > > > > > > > . . > > > > ----------------------------+----------------------------------------------------------- > > > > > > > > > > > > > > Note: S.E. does not take into account that the propensity > score > > is > > > > estimated. > > > > > > > > psmatch2: | psmatch2: Common > > > > Treatment | support > > > > assignment | Off suppo On suppor | Total > > > > -----------+----------------------+---------- > > > > Untreated | 0 2 | 2 > > > > Treated | 1 4 | 5 > > > > 2 | 0 11 | 11 > > > > 3 | 0 12 | 12 > > > > 4 | 0 17 | 17 > > > > 5 | 0 34 | 34 > > > > 6 | 0 15 | 15 > > > > 7 | 0 19 | 19 > > > > 8 | 0 18 | 18 > > > > 9 | 0 17 | 17 > > > > 10 | 0 18 | 18 > > > > 11 | 0 13 | 13 > > > > 12 | 0 13 | 13 > > > > 13 | 0 20 | 20 > > > > 14 | 0 23 | 23 > > > > 15 | 0 23 | 23 > > > > 16 | 0 10 | 10 > > > > 17 | 0 13 | 13 > > > > 18 | 0 10 | 10 > > > > 19 | 0 11 | 11 > > > > 20 | 0 11 | 11 > > > > 21 | 0 10 | 10 > > > > 22 | 0 10 | 10 > > > > 23 | 0 11 | 11 > > > > 24 | 0 10 | 10 > > > > 25 | 0 7 | 7 > > > > 26 | 0 9 | 9 > > > > 27 | 0 6 | 6 > > > > 28 | 0 7 | 7 > > > > 29 | 0 4 | 4 > > > > 30 | 0 7 | 7 > > > > 31 | 0 6 | 6 > > > > 32 | 0 7 | 7 > > > > 33 | 0 7 | 7 > > > > 34 | 0 5 | 5 > > > > 35 | 0 6 | 6 > > > > 36 | 0 6 | 6 > > > > 37 | 0 6 | 6 > > > > 38 | 0 5 | 5 > > > > 39 | 0 4 | 4 > > > > 40 | 0 4 | 4 > > > > 41 | 0 9 | 9 > > > > 42 | 0 7 | 7 > > > > 43 | 0 5 | 5 > > > > 44 | 0 1 | 1 > > > > 45 | 0 1 | 1 > > > > 46 | 0 1 | 1 > > > > 47 | 0 1 | 1 > > > > 48 | 0 3 | 3 > > > > 49 | 0 4 | 4 > > > > 50 | 0 1 | 1 > > > > 51 | 0 1 | 1 > > > > 52 | 0 1 | 1 > > > > 53 | 0 1 | 1 > > > > 55 | 0 1 | 1 > > > > 56 | 0 3 | 3 > > > > 57 | 0 1 | 1 > > > > 59 | 0 1 | 1 > > > > 60 | 0 2 | 2 > > > > 62 | 0 1 | 1 > > > > 63 | 0 2 | 2 > > > > 69 | 0 1 | 1 > > > > 70 | 0 1 | 1 > > > > 78 | 0 1 | 1 > > > > 83 | 0 1 | 1 > > > > 84 | 0 2 | 2 > > > > 92 | 0 1 | 1 > > > > -----------+----------------------+---------- > > > > Total | 1 505 | 506 > > > > > > > > Question: Why arent there only the untreated and treated in > the > > table > > > and > > > > > > > > what ist the meaning of the numbers 2-92? > > > > > > > > > > > > Second problem: > > > > > > > > I've done the nearest neighbor matching of the same data sample > > > with > > > the > > > > > > > > attnd function. > > > > My results are: > > > > > > > > . attnd stockprice2010 reldummy, pscore(mypscore) > > > > > > > > > > > > The program is searching the nearest neighbor of each treated > > > unit. > > > > > > > This operation may take a while. > > > > > > > > > > > > > > > > ATT estimation with Nearest Neighbor Matching method > > > > (random draw version) > > > > Analytical standard errors > > > > > > > > --------------------------------------------------------- > > > > > n. treat. n. contr. ATT Std. Err. t > > > > > --------------------------------------------------------- > > > > > > > > > 424 63 -2.255 6.070 -0.371 > > > > > > > > > --------------------------------------------------------- > > > > > Note: the numbers of treated and controls refer to actual > > > > > nearest neighbour matches > > > > > > > > Why is this result for ATT so different to the one of psmatch2? > > > > > > > Is the attnd matching with replacement? > > > > > > > > Any suggestions would be much appreciated > > > > > > > > Thanks > > > > > > > > Jason > > > > > > > > -- > > > > View this message in context: http://statalist.1588530.n2.nabble.com/Ousout-problem-psmatch2-tp6496035p6496035.html > > > > > > > > > > > > > > Sent from the Statalist mailing list archive at Nabble.com. > > > > > * > > > > * For searches and help try: > > > > * http://www.stata.com/help.cgi?search > > > > * http://www.stata.com/support/statalist/faq > > > > * http://www.ats.ucla.edu/stat/stata/ > > > > > > > > * > > > > * For searches and help try: > > > > * http://www.stata.com/help.cgi?search > > > > * http://www.stata.com/support/statalist/faq > > > > * http://www.ats.ucla.edu/stat/stata/ > > > > > > * > > > * For searches and help try: > > > * http://www.stata.com/help.cgi?search > > > * http://www.stata.com/support/statalist/faq > > > * http://www.ats.ucla.edu/stat/stata/ > > > > > > * > > > * For searches and help try: > > > * http://www.stata.com/help.cgi?search > > > * http://www.stata.com/support/statalist/faq > > > * http://www.ats.ucla.edu/stat/stata/ > > > > * > > * For searches and help try: > > * http://www.stata.com/help.cgi?search > > * http://www.stata.com/support/statalist/faq > > * http://www.ats.ucla.edu/stat/stata/ > > * > * For searches and help try: > * http://www.stata.com/help.cgi?search > * http://www.stata.com/support/statalist/faq > * http://www.ats.ucla.edu/stat/stata/ > > * > * For searches and help try: > * http://www.stata.com/help.cgi?search > * http://www.stata.com/support/statalist/faq > * http://www.ats.ucla.edu/stat/stata/ * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/