Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: st: choosing between two non nested models

From   "Craig, Benjamin M." <>
To   <>
Subject   RE: st: choosing between two non nested models
Date   Wed, 1 Sep 2010 08:44:15 -0400

This is an impossible question, because of its dependence on your
personal perspective.

If you were a Bayesian, Bayes factors should decide model selection;
however, BIC is proportional to the factors in large samples, and is
much easier to estimate.

If you are frequentist, it is messier. AIC is entirely arbitrary in its
trade-off between bias and variance, but it is justifiable if you agree
with its normative assumptions. Alternatively, if you just care about
prediction, you could apply out-of-sample predictions techniques (see
Copas test). 

As a last resort, frequentists may try re-sampling techniques, where
errors from one model are incorporated with predictions from another to
construct psuedo data to see if the model results are numerically
nested, even if their formulae are not (e.g., logit vs. probit). These
more complicated techniques require greater sophistication for the
analyst and audience than may be required.

As an econometrician, I recommend that you state your model first, then
estimate. Any other model is an exploratory analysis, and rejecting a
model using data may take a sample size beyond that available. Theory
should drive your model selection, not data. This is no right or wrong
answer, just evidence.


Benjamin M. Craig, PhD

Assistant Member, Health Outcomes & Behavior, Moffitt Cancer Center

Associate Professor, Economics, University of South Florida 

12902 Magnolia Dr, MRC-CANCONT, Tampa, FL 33612-9416

Phone (813) 745-6710; Fax (813) 745-6525 

-----Original Message-----
[] On Behalf Of Richard
Sent: Wednesday, September 01, 2010 8:55 AM
Subject: Re: st: choosing between two non nested models

At 05:32 AM 9/1/2010, Michael N. Mitchell wrote:
>I wonder if AIC and/or BIC may be of help? You can see
>. help estat ic
>for more information. I think this is a longshot, but hope it may pay

I don't think those stats are legitimate if the observations are
different.  But I wonder why they are different in the first place?  I
think you would usually want to be analyzing the same cases if you want
to compare two models.  It would be easy enough to do a listwise
deletion of all cases that have missing data in the two models.

Richard Williams, Notre Dame Dept of Sociology
OFFICE: (574)631-6668, (574)631-6463
HOME:   (574)289-5227
EMAIL:  Richard.A.Williams.5@ND.Edu

*   For searches and help try:

This transmission may be confidential or protected from disclosure and is only for review and use by the intended recipient. Access by anyone else is unauthorized. Any unauthorized reader is hereby notified that any review, use, dissemination, disclosure or copying of this information, or any act or omission taken in reliance on it, is prohibited and may be unlawful. If you received this transmission in error, please notify the sender immediately. Thank you.

*   For searches and help try:

© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index