Maarten L. Buis wrote:
>>All the logics that Nick mentrioned are at work in the same way
regardless whether you are doing a univariate, bivariate, or
multivariate analysis. So there are good reasons to use significance
tests (perferable with a footnote saying that the logic is a bit
different from what most people are used to), and there are good rasons
for not using significance tests in this situation. There is however no
law that states that you should always use tests when doing a multiple
regression.<<
There are also other sources of inferential unreliability besides
sampling error that the covariance matrix of the parameters might help
diagnose, namely unreliability of the measurements themselves (a big
issue with behavioral science data as someone else mentioned),
multicollinearity, and how well the model approximates the data simply
as a data reduction device. In the population case, where multiple
regression (and related procedures) are applied simply as data reduction
devices, the projective geometry logic holds sway and the various
quantities used for inference are signs of ill-fit. The sampling model
that underlies frequentist significance levels fails but to the extent
that the significance tests approximate Bayesian quantities, they can be
put on a "sound" logical footing ("sound" in quotes to indicate that
some people buy the Bayesian argument and others do not). Their
frequentist properties are bogus in this case, but that doesn't mean
they're not useful. We apply significance tests a lot when the
underlying frequentist logic can't possibly hold.
I seem to recall that there's some discussion of this point in the
permutation literature, which I recall reading about in the
correspondence analysis literature, where permutations and bootstrapping
are often used to get a sense of how "stable" the results are,
irrespective of the underlying sampling model. This is probably
discussed (somewhere) in the Gifi (1990) book---good luck finding it, as
anyone who's familiar with the book can attest.
Jay
--
J. Verkuilen
Assistant Professor of Educational Psychology
City University of New York-Graduate Center
365 Fifth Ave.
New York, NY 10016
Email: [email protected]
Office: (212) 817-8286
FAX: (212) 817-1516
Cell: (217) 390-4609
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
