Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
From | Nick Cox <njcoxstata@gmail.com> |
To | statalist@hsphsun2.harvard.edu |
Subject | Re: st: How to compare RMSE of two models |
Date | Tue, 27 Nov 2012 13:14:53 +0000 |
That seems to me the same question re-worded, but not more clearly. As often happens, I started writing a reply to your first post broadly similar to Maarten's, but he got there first. You can use RMSE as a criterion, in which case the model with the lower RMSE is the better model. Naturally many would want to emphasise here that e.g. a scatter plot with fitted regression line would be as or more informative about which model is "better". Recasting that comparison as a signficance test seems to raise more questions than it solves, not least quite what sense can be given to a significance test given bootstrap results for RMSE for two models with a shared response and different predictors. If people don't understand what you want and/or think it seems unsound in principle, they won't usually pass to the next question of suggesting Stata commands to do it. Nick On Tue, Nov 27, 2012 at 12:49 PM, rasool.bux <rasool.bux@aku.edu> wrote: > Dear Maarten, > > Thanks for reply. May be I am not conveying my question clearly. I would like to compare the two models against the same yvariable, which model is performing better and how we could say that model 1 is better than model 2 i.e. in terms of p-values of RMSE. > > Best regards > Rasool Bux > > -----Original Message----- > From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Maarten Buis > Sent: Tuesday, November 27, 2012 2:45 PM > To: statalist@hsphsun2.harvard.edu > Subject: Re: st: How to compare RMSE of two models > > On Tue, Nov 27, 2012 at 10:32 AM, rasool.bux wrote: >> How to statistical compare RMSE of two equations/models in Stata? Kindly let me know which test used for it and how to report it. Same for IQR (95%CIs) reporting as single value i.e. p75 - p25. >> >> bootstrap "regress yvar xvar1" _b rmse1=e(rmse), reps(1000) >> saving(rmse1) bootstrap "regress yvar xvar2" _b rmse2=e(rmse), >> reps(1000) saving(rmse2) > > The first thing to consider when you want to perform a statistical test is to think about the null hypothesis. In this case I don't think what you want to do involves a meaningful null hypothesis. In that case you just don't want to perform that test. > >> bootstrap iqr_v1=(r(p75)-r(p25)), rep(1000) seed(12345789) nodots >> saving(iqr_v1,replace): summarize v1, detail > > Again consider the null hypothesis, and think whether you want to test that hypothesis. Again, I don't think you want to do that. * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/