Re: st: Relative Importance of predictors in regression

[David Hoaglin <dchoaglin@gmail.com>]

Hi, Sam.

No "wordplay" is involved.  Regression coefficients are a type of
slope, so it is natural to talk about "increase" and "change," which
most people would consider to be continuous, whereas "difference"
often comes across as discrete.

Earlier, when I mentioned subspaces and projections, I had in mind a
description based on linear algebra.  Since you teach regression, you
should be familiar with partial-regression plots (also called
added-variable plots), and I can use that setting to describe the main
idea.

Suppose Y = b1X1 + b2X2 + ... + bpXp.  The observed values of Y
usually include disturbances (or noise or errors), but I am using the
true values of Y in order to describe the interpretation of the true
regression coefficients.  The description applies with minor
modifications when Y includes noise.  I have not numbered the
predictors from 0, so X1 will often be the constant predictor, and I
will focus on b2.

Regress Y on X1, X3, ..., Xp, producing the fitted regression c1X1 +
c3X3 + ... + cpXp and the residuals Y - (c1X1 + c3X3 + ... + cpXp).
This step adjusts Y by removing the contributions of X1, X3, ..., Xp.

Regress X2 on X1, X3, ..., Xp, producing the fitted regression a1X1 +
a3X3 + ... + apXp and the residuals X2 - (a1X1 + a3X3 + ... + apXp).
This step adjusts X2 for simultaneous linear change in X1, X3, ...,
Xp.

Regress the Y-residuals on the X2-residuals.  The resulting regression
line (through the origin) is
Y - (c1X1 + c3X3 + ... + cpXp) = b2[X2 - (a1X1 + a3X3 + ... + apXp)].
That is, the slope of this regression line is the coefficient of X2 in
the original model.  And, because the values of Y do not include
noise, the points (X2-residual, Y-residual) lie on that line through
the origin.

I hope this explanation will help you to explain the correct general
interpretation in your teaching, and steer your students away from the
"held constant" interpretation.  I realize that it may be awkward if
the textbook uses the "held constant" interpretation (students will
wonder what else the book gets wrong), but the message is important.
Some books do get multiple regression right.  One example is Weisberg
(1985).  And Cook and Weisberg (1982, Section 2.3.2) give a
straightforward proof for the usual partial-regression plot.

Regards,

David Hoaglin

Cook, R. D., and S. Weisberg (1982).  Residuals and Influence in
Regression.  New York: Chapman & Hall.

Weisberg, S. (1985).  Applied Linear Regression, second edition.  New
York: John Wiley & Sons.

On Mon, Nov 4, 2013 at 9:13 PM, Lucas <lucaselastic@gmail.com> wrote:
> Well, just two things:
>
> 1)The word "change" means change (Xa becomes Xb); the word
> "difference" means difference (Xa is not the same as Xb).  Why
> wordplay?  If difference is always an accurate statement of the model,
> and "change" is only accurate sometimes (as your response implicitly
> admits), why not say, "Wow, my bad.  Yes, difference is generally
> accurate, and that of course is a better way to describe the model in
> general, and especially in a context with people from dozens of
> fields."?
>
> 2)You indicated that "Straightforward mathematics" proved the point
> you have often articulated.  Now you indicate is too difficult to
> express in plain text, which certainly does not sound like
> straightforward mathematics.  But, okay.  Still, I would love to see
> this expression.  I'm not joking or baiting--I teach this material and
> if there's an expression that would make your point I'll integrate it
> into my teaching. Others on the list might do so as well.  That would
> be a major service to the disciplines. Perhaps instead of trying to
> write it in plain text, you can provide a citation to something that
> clearly makes your point. That could be helpful.
>
> So, at this point I, and perhaps others, look forward to receiving
> that citation.  Thanks in advance.
>
> Take care.
> Sam
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/