Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.

# Re: st: Adjusted R-squared comparison

 From Nick Cox To statalist@hsphsun2.harvard.edu Subject Re: st: Adjusted R-squared comparison Date Wed, 6 Feb 2013 12:41:36 +0000

```There is an extra dimension here. John's bootstrap example is a nice
simple example of a model applied to non-panel, non-time series data.
-bootstrap-ping panel data that are time series too is trickier, to
say the least.

Nick

On Wed, Feb 6, 2013 at 12:30 PM, John Antonakis <John.Antonakis@unil.ch> wrote:
> Hi Panagiotis:
>
> In fact, the result you get is the mean and SD of the bootstrap.
>
> Specifically:
>
> sysuse auto
> bootstrap e(r2), seed(123) reps(50) : reg price mpg weight
>
> gives:
>
>
> Bootstrap replications (50)
> ----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
> ..................................................    50
>
> Linear regression                               Number of obs =        74
>                                                 Replications =        50
>
>       command:  regress price mpg weight
>         _bs_1:  e(r2)
>
> ------------------------------------------------------------------------------
>              |   Observed   Bootstrap Normal-based
>              |      Coef.   Std. Err.      z    P>|z|     [95% Conf.
> Interval]
> -------------+----------------------------------------------------------------
>        _bs_1 |   .2933891    .074451     3.94   0.000 .1474678    .4393104
> ------------------------------------------------------------------------------
>
>
> .2933891 is the mean of the bootstrapped r-squares and .07215 is the SD.
>
> If you wish to check this save the bootstrap estimates (using saving) and
> check the mean and SD manually.
>
> So, with these two values from both samples, I guess you could do a t-test
> for the difference if this is what you are looking for.
>
> Let's see what others might say.
>
>
> Best,
> J.
>
>
> __________________________________________
>
> John Antonakis
> Professor of Organizational Behavior
> Director, Ph.D. Program in Management
>
> Faculty of Business and Economics
> University of Lausanne
> Internef #618
> CH-1015 Lausanne-Dorigny
> Switzerland
> Tel ++41 (0)21 692-3438
> Fax ++41 (0)21 692-3305
> http://www.hec.unil.ch/people/jantonakis
>
> Associate Editor
> __________________________________________
>
> On 06.02.2013 12:57, Panagiotis Manganaris wrote:
>> Unfortunately Nick and John, I must use adj r-squared because it
>> represents a specific metric in the field of accounting. More specifically,
>> I use a model where returns are the dependent variable and earnings, along
>> with the change in earnings, are the independent variables. In this model
>> the adjusted r-squared represents the value relevance of the earnings (this
>> is what I am trying to gauge). Therefore, I am obliged to use r2.
>> Thank you for the procedure you mention John, but I had already tried it
>> in the past. It is helpful, but only in a vague way. It does not provide the
>> mean and the variance of r2, so I could use them to test the significance.
>> For instance, the intervals almost always overlap when I use this method.
>> That does not provide concrete evidence of statistical significance or
>> non-significance. If I don't prove that there is (or there is not) a
>> statistically significant difference, I cannot show whether my metric (value
>> relevance) has been altered between the two periods.
>>
>>
>>
>> 2013/2/6 John Antonakis <John.Antonakis@unil.ch>
>> Can't agree more with you Nick.  We should care more about having
>> consistent estimators than high r-squares (i.e., Panagiotis, what I mean
>> here is that we can still estimate the slope consistently even if we don't
>> have a tight fitting regression line).  So, I don't know why you are
>> interested in this comparison, Panagiotis. I would think you would be more
>> interested in comparing estimates, as in a Chow test (Chow, G. C. (1960).
>> Tests of equality between sets of coefficients in two linear regressions.
>> Econometrica, 28(3), 591-605). If you are using fixed-effects models, you
>> can model the fixed-effects with dummies and then do a Chow test via
>> suest....see -help suest-.
>>
>>
>> Best,
>> J.
>>
>> __________________________________________
>>
>> John Antonakis
>> Professor of Organizational Behavior
>> Director, Ph.D. Program in Management
>>
>> Faculty of Business and Economics
>> University of Lausanne
>> Internef #618
>> CH-1015 Lausanne-Dorigny
>> Switzerland
>> Tel ++41 (0)21 692-3438
>> Fax ++41 (0)21 692-3305
>> http://www.hec.unil.ch/people/jantonakis
>>
>> Associate Editor
>> __________________________________________
>>
>> On 06.02.2013 11:40, Nick Cox wrote:
>>
>> My own other idea is that adjusted R-squares are a lousy basis to
>> compare two models, even of the same kind. They leave out too much
>> information.
>>
>> Nick
>>
>> On Wed, Feb 6, 2013 at 10:37 AM, John Antonakis <John.Antonakis@unil.ch>
>> wrote:
>> I think that the only think you can do is to bootstrap the r-squares and
>> see
>> if their confidence intervals overlap.
>>
>> To bootstrap you just do:
>>
>> E.g.,
>>
>> sysuse auto
>> bootstrap e(r2), seed(123) reps(1000) : reg price mpg weight
>>
>> You will be interested in either:
>>
>>        e(r2_w)             R-squared within model
>>        e(r2_o)             R-squared overall model
>>        e(r2_b)             R-squared between model
>>
>> See help xtreg with respect to saved results.
>>
>> Let's see if others have other ideas.
>> On 06.02.2013 10:22, Panagiotis Manganaris wrote:
>>
>> I need to compare two adjusted r-squared of the same model for two
>> different periods of time (each one spans for a period of years). So far,
>> I
>> have split my data in two groups, those that belong to the period
>> 1998-2004
>> and those that belong to the period 2005-2011. Then I used xtreg on the
>> same
>> model for each group of data. I've derived their adjusted r-squared and I
>> want to know if those two adjusted r-squared are significantly different
>> from each other.
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/
```