Re: Re: st: Significance test with -pscore- and -attnd-

[Lukas Borkowski <570722@soas.ac.uk>]

The first email did not go through.

Ariel,

thank you for the advice on how to check the statistical significance of ATT following the user-written -attnd- command. The reference I was referring to is Khandker, S. R., Koolwal, G. B. and Samad, H. A. (2010). Handbook on Impact Evaluation. Quantitative Methods and Practices. The World Bank. Washington, D. C. Apologies for the inconvenience.

Coming back to estimating ATT using panel data, I am aware that a Difference in Difference estimation with propensity weights (ps/(1-ps)) could be a suitable approach. However, I would like to compare these results with "simple" ATT results, possibly derived with -attnd-. My question is on how to implement -attnd- in a panel structure. In a two-waved perfectly balanced panel, is it reasonable to convert my panel into a wide dataset using -reshape- to run -pscore- using suitable covariates from the first wave and then -attnd- on the outcome variable in the second wave? 

Thanks for your advice. I really appreciate it!

Best, Lukas

#
Lukas Borkowski
University of London, School of Oriental and African Studies (SOAS)

M: 570722@soas.ac.uk



On 10.08.2013, at 16:39, "Ariel Linden, DrPH" <ariel.linden@gmail.com> wrote:

> Lucas,
> 
> You are asking some very basic questions that would suggest you need to
> learn more about propensity score matching (or matching in a more broad
> sense). I suggest you read Stuart (2010) and Caliendo & Kopeinig (2008) for
> general guidance in this area.:
> 
> Here are some more specific responses to your questions, but as you can see,
> they require you to know what you're asking (and thus, I suggest you read
> the papers)
> 
> - keep only matched observations? 
> 
> * it depends on what you plan on doing next, and how you limit your
> observations for analysis
> 
> - keep only observations within the common support?
> 
> * it depends on what you plan on doing next, and how you limit your
> observations for analysis
> 
> - estimate the ATT with -attnd- using both panel rounds or only the post
> round?
> 
> * This question suggests that you need to learn about the estimators and how
> you use them. Briefly, the baseline characteristics are used as a
> pre-processing step for matching. Once matches are derived (and you have
> checked for covariate balance), you move on to the analysis stage. In an
> analysis, you evaluate the effect of the "treatment" on the outcome,
> controlling for baseline characteristics. In this case, you have controlled
> for baseline characteristics via the matching process, so the analysis is on
> the outcome (in your terminology, the post round).
> 
> - Do I have to convert my dataset from long to wide to estimate the ATT?
> 
> * If you want to use this procedure, then you should have the data in wide
> format. If you have multiple waves, you have other matching and outcomes
> analysis approaches to consider.
> 
> - How can I see whether the ATT is significant (as, for example, shown but
> not explained in Khandker et al., 2010)?
> 
> *First off, you didn't provide us with the complete reference. Should we
> guess, or are you assuming that we all know exactly what you are
> referencing? What if Khandker wrote 10 papers in 2010? 
> 
> *Second, you have (at least) two ways of finding the level of significance:
> first, you know that p = 0.05 is approximately equal to t = 1.96, so in this
> case with a t score of > 1.96, you can be reasonably assured that the value
> is statistically significant. A more specific approach to use, following
> -attnd- (a user written program -findit attnd-) looks something like this
> (after running some example code): 
> 
> . return list
> 
> scalars:
>              r(ncnd) =  12144
>              r(ntnd) =  185
>           r(tsattnd) =  -12.52377338537646
>           r(seattnd) =  587.1491980157692
>             r(attnd) =  -7353.32349935502
> 
> * estimate t(df,t)
> . di t(r(ncnd)+r(ntnd)-2,r(tsattnd))
> 4.567e-36
> 
> I hope this helps
> 
> Ariel
> 
> Stuart, E.A. (2010) Matching methods for causal inference: a review and a
> look forward. Statistical Science, 25(1), 1–21.
> 
> Caliendo, M. Kopeinig, S. (2008) Some practical guidance for the
> implementation of propensity score matching. Journal of Economic Surveys,
> 22, 31-72. 
> 
> ________________________________________
> From
>  Lukas Borkowski <570722@soas.ac.uk>
> To
>  "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
> Subject
>  st: Significance test with -pscore- and -attnd-
> Date
>  Fri, 9 Aug 2013 17:32:23 +0200
> ________________________________________
> Dear Statalist,
> 
> I am using Stata 12.1 and run an impact evaluation on a two-period panel
> dataset using -pscore- and related commands (written by Sascha O. Becker and
> Andrea Ichino). I got myself a bit confused and would really appreciate your
> help:
> 
> After running -pscore- to estimate the propensity score using the first
> panel round (baseline data) only, should I 
> 
> - keep only matched observations?
> - keep only observations within the common support?
> - estimate the ATT with -attnd- using both panel rounds or only the post
> round?
> - Do I have to convert my dataset from long to wide to estimate the ATT?
> 
> Moreover, the -attnd- command provides the standard error and a t-statistic.
> How can I see whether the ATT is significant (as, for example, shown but not
> explained in Khandker et al., 2010)?
> 
> Thank your very much for your help!
> 
> Best,
> 
> Lukas
> 
> #
> Lukas Borkowski
> University of London, School of Oriental and African Studies (SOAS)
> 
> 
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/


#
Lukas Borkowski
University of London, School of Oriental and African Studies (SOAS)

M: 570722@soas.ac.uk




*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/