Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
simone ferro <simone.ferro88@gmail.com> |

To |
statalist@hsphsun2.harvard.edu |

Subject |
Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow? |

Date |
Sun, 13 Jan 2013 12:21:06 +0100 |

Thank you so much for your answer, it was very clear and helpful, I followed your suggestion and I got better results, so thank you very much, by the way I'm using a kernel matching based on a probit regression, variables are mainly continuous. Now I get a better balance, but as expected t-tests refuse the null-hypotesis of equal means, and chi2 test refuse the null hypotesis of balance as well, looking at the means it seems to me that this is a good balance, do you think I can simply don't look at t-tests and chi2 test? --------------------------------------------------------- | Mean | t-test Variable | Treated Control %bias | t p>|t| -------------+--------------------------+---------------- cost_per_e~l | 60.95 60.053 4.4 | -3.48 0.001 asset | 193.35 260.33 -5.3 | -1.41 0.159 employees | 398.82 551.06 -7.9 | -0.97 0.333 av_age | 54.424 54.951 -6.9 | 1.69 0.092 donne_perc | .29114 .28214 4.3 | 3.86 0.000 laureati_p~c | .23952 .23494 1.9 | 0.11 0.914 bluecollar~c | .35553 .37312 -5.8 | -2.94 0.004 occupaz_min | .16463 .13253 9.1 | -0.13 0.895 donne_perc | .29114 .28214 4.3 | 3.86 0.000 ---------------------------------------------------------- Pseudo R2 LR chi2 p>chi2 MeanB MedB ---------------------------------------------------------- 0.163 86.83 0.000 5.6 5.3 ---------------------------------------------------------- thank you again, Regards, Simone Il giorno 13/gen/2013, alle ore 05:18, Adam Olszewski <adam.olszewski@gmail.com> ha scritto: > Hi Simone, > t-test based comparisons after PS matching are highly controversial. > t-test makes a lot of usually untenable assumptions (are the variables > normally distributed?), and moreover is too sensitive to sample size. > A non-significant test might just mean a small sample, while a > minuscule difference in means might be "significant" in a very large > sample. > Standardized differences of means seem to be more accepted, although > for continuous covariates comparing actual distributions may be most > persuasive. > From your message it is not clear what type of matching you used and > whether all the variables are continuous or some are categorical. If > you question the test statistics calculation, you can run the t-test > manually and check. While I'm not knowledgeable about practices in > sociology and economics, you may want to look at SDM's rather than the > pstest results. See e.g.: > Austin, P. C. (2009). "Balance diagnostics for comparing the > distribution of baseline covariates between treatment groups in > propensity-score matched samples." Statistics in Medicine 28(25): > 3083-3107. > > Good luck, > Adam Olszewski > > On Sat, Jan 12, 2013 at 7:37 AM, simone ferro <simone.ferro88@gmail.com> wrote: >> dear Statalist, >> >> I would please need some clarifications about the interpretation of the command pstest after running psmatch2: >> I report a random output just as an example. >> >> pstest ebitda_marg asset employees av_age donne_perc laureati_perc bluecollar_perc, both >> >> ------------------------------------------------------------------------------ >> Unmatched | Mean %reduct | t-test >> Variable Matched | Treated Control %bias |bias| | t p>|t| >> --------------------------+----------------------------------+---------------- >> ebitda_marg Unmatched | 11.404 8.2304 36.8 | 3.25 0.001 >> Matched | 10.746 10.555 2.2 94.0 | -2.33 0.020 >> | | >> asset Unmatched | 395.52 34.389 28.6 | 2.48 0.014 >> Matched | 115.36 89.157 2.1 92.7 | -2.10 0.037 >> | | >> employees Unmatched | 641.98 508.48 4.6 | 0.42 0.677 >> Matched | 474.12 704.43 -8.0 -72.5 | 0.18 0.857 >> | | >> av_age Unmatched | 53.369 56.714 -45.0 | -4.06 0.000 >> Matched | 53.39 53.051 4.6 89.9 | 3.84 0.000 >> | | >> donne_perc Unmatched | .34711 .37372 -11.8 | -1.06 0.291 >> Matched | .34805 .33374 6.3 46.2 | 0.16 0.874 >> | | >> laureati_perc Unmatched | .26656 .18165 35.4 | 3.15 0.002 >> Matched | .26815 .23665 13.2 62.9 | -1.52 0.129 >> | | >> bluecollar_~c Unmatched | .30019 .3349 -11.2 | -1.00 0.317 >> Matched | .30425 .33592 -10.2 8.7 | -0.28 0.776 >> | | >> ------------------------------------------------------------------------------ >> If I understood well, reported t-.tests' null hypothesis is that the two covariates are equal in treated and control group, >> so I should look at t-tests to check if the groups are well balanced, >> In some tutorial instead I've read that the right approach is to look at the bias percentage that should be under 10 to be considered ok, >> >> Which one of the two approaches is the right one?it's fundamental for me to understand because they provide totally different interpretations. >> indeed if I look at t-tests, I find problems with ebitda_margin, asset and av_age, because they are significantly different in the two groups, >> while if I look at the bias percentage, I find problem with laureati_perc and blecollar, because their bias% are bigger than 10. >> >> I also would appreciate your confirmation of the interpretation of the two indicators(%bias and t-tests), because with this interpretation I find the two values contradictory. >> looking for example at the variable av_age, I find a very little bias, and the means after matching of treated and control group are almost identical(53.39 and 53.051), by the way the t-test reports a p-value of 0.000! >> So it seems like I have misunderstood the meaning of the t.test because I don't think that 53.39 and 53.051 can be statistically different with a t-stat of 3.84, also given the nature of the variable(the average age of the managers of a firm) that should infact be quite variable. >> The same happens with the variable ebitda, which reports a bias% of 2.2% and almost identical values(10.746 and 10.555), but t-stat is -2.33 and p-value 2%! >> Can you please help me? >> thanks in advance for the help, >> Regards, >> Simone Ferro >> * >> * For searches and help try: >> * http://www.stata.com/help.cgi?search >> * http://www.stata.com/support/faqs/resources/statalist-faq/ >> * http://www.ats.ucla.edu/stat/stata/ > > * > * For searches and help try: > * http://www.stata.com/help.cgi?search > * http://www.stata.com/support/faqs/resources/statalist-faq/ > * http://www.ats.ucla.edu/stat/stata/ * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/

**References**:**st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?***From:*simone ferro <simone.ferro88@gmail.com>

**Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?***From:*Adam Olszewski <adam.olszewski@gmail.com>

- Prev by Date:
**[no subject]** - Next by Date:
**st: cointegration - value of the speed of adjustment** - Previous by thread:
**Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?** - Next by thread:
**RE: Re: st: problem with the interpretation of pstest after psmatch2, t-tests and percentage of bias provide conflicting results, which one should I follow?** - Index(es):