[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

RE: st: Testing for program effectiveness with heckman

From   Maarten buis <>
Subject   RE: st: Testing for program effectiveness with heckman
Date   Fri, 30 May 2008 09:57:00 +0100 (BST)

--- "Riemer, Richard A CIV DMDC" <>
> Maarten, Thank you for your reply.  I can see the distinction you are
> making.  However, I wanted to use heckman because I thought it would
> do a better job at explaining self-selection of test-preparation
> rather than simple moderated regression where there could be
> correlated errors between the two equations.  Following the example
> of wage of women, we could say that 'afqt after test prep' is missing
> on sample members who do not engage in test prep and that those
> sample members would have scored lower than average if they would
> have engaged in test prep.

What you are running into is the fundamental problem with causal
analysis. A causal effect can be thought of as a counterfactual
experiment: You want to compare the test score of someone that prepared
for the test with the test score of that same person when (s)he did not
prepare for the test. The problem is that you cannot have a person that
is both prepared and unprepared at the same time. An alternative way of
thingking about this is that you are looking for another person that is
the same in every respect except that that person did not prepare. Such
a person obviously does not exist.  

The information we do have is a comparison of groups. You can use
regression / ANOVA to control for other observed variables. An
alternative method of controlling for observed characteristics is
propensity score matching. Some would call these estimates biased
because they expect that students who know that they won't do well are
less likely to prepare, thus leading to an overestimation of the effect
of preparetion; the students who did not prepare are expected to gain
less from preparation then the students that did prepare. I would not
call the group comparisons biased, but I would call them the empirical
information that you use in your model. Once you have presented the
empirical information, you can start adding assumptions, for instance
by using -treatreg-, thus sacrificing empirical content to get closer
to the theoretical concept you are interested in. If anything group
comparisons are, if interpreted correctly as the difference between
groups, likely to be much less biased than so called causal models.
This is not because one type of model is inherently better than the
other, but because the group comparison models try to solve a much
easier problem. 

-- Maarten

Maarten L. Buis
Department of Social Research Methodology
Vrije Universiteit Amsterdam
Boelelaan 1081
1081 HV Amsterdam
The Netherlands

visiting address:
Buitenveldertselaan 3 (Metropolitan), room Z434

+31 20 5986715

Sent from Yahoo! Mail.
A Smarter Email
*   For searches and help try:

© Copyright 1996–2017 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index