# Re: st: testing equality of parameter estimates across two models--suest followed by test

 From "Tim Wade" To statalist@hsphsun2.harvard.edu Subject Re: st: testing equality of parameter estimates across two models--suest followed by test Date Thu, 26 Jun 2008 08:32:03 -0400

```Hi Brent, I'm afraid I can't provide much intuition here since it
would probably require familiarity with the data.  You could verify
the result using likelihood tests, or interaction terms and compare
the result to a suest result to see if your result is reasonable. Of
course you would have to do this comparison without the survey
estimators I think but it could help see if the suest result is
reasonable. You should get the same results for interaction terms as
you do for suest, for example note the p-value for the coefficient on
the interaction terms for weight and headroom is the same as that when
you test the coefficients after suest.

Tim

. sysuse auto.dta
(1978 Automobile Data)

. gen mpgcat=cond(1, mpg>=20, 0)

. logit foreign weight headroom if mpg<20

Iteration 0:   log likelihood = -14.354071
Iteration 1:   log likelihood = -8.6195982
Iteration 2:   log likelihood = -6.3389471
Iteration 3:   log likelihood =  -5.776879
Iteration 4:   log likelihood =  -5.655399
Iteration 5:   log likelihood = -5.6441813
Iteration 6:   log likelihood =  -5.644033
Iteration 7:   log likelihood =  -5.644033

Logistic regression                               Number of obs   =         35
LR chi2(2)      =      17.42
Prob > chi2     =     0.0002
Log likelihood =  -5.644033                       Pseudo R2       =     0.6068

------------------------------------------------------------------------------
foreign |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
weight |  -.0094147   .0054628    -1.72   0.085    -.0201216    .0012922
headroom |  -.1793897    .869445    -0.21   0.837    -1.883471    1.524691
_cons |    29.9695   17.37471     1.72   0.085    -4.084307    64.02331
------------------------------------------------------------------------------

. est store m1

. logit foreign weight headroom if mpg>=20

Iteration 0:   log likelihood = -26.711343
Iteration 1:   log likelihood = -20.190873
Iteration 2:   log likelihood =  -19.24142
Iteration 3:   log likelihood = -19.157028
Iteration 4:   log likelihood = -19.156039
Iteration 5:   log likelihood = -19.156039

Logistic regression                               Number of obs   =         39
LR chi2(2)      =      15.11
Prob > chi2     =     0.0005
Log likelihood = -19.156039                       Pseudo R2       =     0.2829

------------------------------------------------------------------------------
foreign |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
weight |  -.0032841   .0011223    -2.93   0.003    -.0054837   -.0010846
headroom |   .1883558   .6371348     0.30   0.768    -1.060405    1.437117
_cons |   7.107416   3.061682     2.32   0.020     1.106629     13.1082
------------------------------------------------------------------------------

. est store m2

. suest m1 m2

Simultaneous results for m1, m2

Number of obs   =         74

------------------------------------------------------------------------------
|               Robust
|      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
m1           |
weight |  -.0094147    .004095    -2.30   0.022    -.0174408   -.0013885
headroom |  -.1793897   .4468522    -0.40   0.688    -1.055204    .6964245
_cons |    29.9695   13.53867     2.21   0.027     3.434191    56.50481
-------------+----------------------------------------------------------------
m2           |
weight |  -.0032841   .0010726    -3.06   0.002    -.0053864   -.0011819
headroom |   .1883558   .6625353     0.28   0.776    -1.110189    1.486901
_cons |   7.107416   3.342944     2.13   0.033      .555367    13.65947
------------------------------------------------------------------------------

. test [m1]weight=[m2]weight

( 1)  [m1]weight - [m2]weight = 0

chi2(  1) =    2.10
Prob > chi2 =    0.1476

( 1)  [m1]headroom - [m2]headroom = 0

chi2(  1) =    0.21
Prob > chi2 =    0.6454

. xi: logit foreign i.mpgcat*weight i.mpgcat*headroom, vce(robust)
i.mpgcat          _Impgcat_0-1        (naturally coded; _Impgcat_0 omitted)
i.mpgcat*weight   _ImpgXweigh_#       (coded as above)

note: _Impgcat_1 dropped because of collinearity
Iteration 0:   log pseudolikelihood =  -45.03321
Iteration 1:   log pseudolikelihood = -29.341451
Iteration 2:   log pseudolikelihood = -26.225121
Iteration 3:   log pseudolikelihood = -25.164491
Iteration 4:   log pseudolikelihood = -24.856965
Iteration 5:   log pseudolikelihood = -24.802901
Iteration 6:   log pseudolikelihood = -24.800082
Iteration 7:   log pseudolikelihood = -24.800072

Logistic regression                               Number of obs   =         74
Wald chi2(5)    =      15.57
Prob > chi2     =     0.0082
Log pseudolikelihood = -24.800072                 Pseudo R2       =     0.4493

------------------------------------------------------------------------------
|               Robust
foreign |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
weight |  -.0094147   .0040949    -2.30   0.021    -.0174406   -.0013888
_ImpgXweig~1 |   .0061305   .0042331     1.45   0.148    -.0021661    .0144272
_Impgcat_1 |  -22.86209   13.94489    -1.64   0.101    -50.19356    4.469389
headroom |  -.1793897   .4468517    -0.40   0.688    -1.055203    .6964235
_ImpgXhead~1 |   .3677455   .7991429     0.46   0.645    -1.198546    1.934037
_cons |    29.9695   13.53826     2.21   0.027     3.434993    56.50401
------------------------------------------------------------------------------

Tim

On Wed, Jun 25, 2008 at 6:42 PM, Brent Fulton <fultonb@berkeley.edu> wrote:
> Hi Tim, thank for your response. I agree the covariance term could matter so
> I used your suggested command -matlist e(V)-
>
> The cov([m1a]hispanic, [m1b]hispanic) was 0.12, so standard error of the
> difference 0.017 becomes quite small (0.011), making the p-value of 0.0991
> make sense.
>
> se of difference: sqrt( (0.347001)^2 + (0.34728)^2 - 2(0.120451))= 0.011
>
> Now I realize the covariance term really mattered, but I don't know quite
> how to explain it. The standard error of each estimate was quite wide at
> 0.347, yet the standard error of the estimate of the difference between
> these estimators was only 0.011. The models were very similar; only one
> independent variable was added to the second model.
>
> I'd appreciate any intuition on how to explain how the standard error of the
> difference is very small relative to the standard error of each parameter
> estimate.
>
> Thanks,
> Brent
>
>
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Tim Wade
> Sent: Wednesday, June 25, 2008 2:38 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: testing equality of parameter estimates across two
> models--suest followed by test
>
> Brent,
>
> I've never used suest with survey estimators and I'm not sure how the
> F-statistic is obtained.  But I don't think you can rely only on the
> standard errors for the coefficients to assess the precision of the
> estimator. Unless the two models are independent you need to take into
> account the cross-model covariance, which you can list  after you
> "suest" your models using "matlist e(V)". If they have a positive
> covariance this will reduce the variance of the (a-b) estimate since:
> Var(a-b)=Var(a)+Var(b)-2*cov(a, b)
>
>
> for example:
>
>
> . sysuse auto.dta
> (1978 Automobile Data)
>
> . logistic foreign mpg
>
> Logistic regression                               Number of obs   =
> 74
>                                                  LR chi2(1)      =
> 11.49
>                                                  Prob > chi2     =
> 0.0007
> Log likelihood =  -39.28864                       Pseudo R2       =
> 0.1276
>
> ----------------------------------------------------------------------------
> --
>     foreign | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf.
> Interval]
> -------------+--------------------------------------------------------------
> --
>         mpg |   1.173232   .0616972     3.04   0.002     1.058331
> 1.300608
> ----------------------------------------------------------------------------
> --
>
> . est store m1
>
> . logistic foreign mpg headroom
>
> Logistic regression                               Number of obs   =
> 74
>                                                  LR chi2(2)      =
> 13.39
>                                                  Prob > chi2     =
> 0.0012
> Log likelihood =  -38.34058                       Pseudo R2       =
> 0.1486
>
> ----------------------------------------------------------------------------
> --
>     foreign | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf.
> Interval]
> -------------+--------------------------------------------------------------
> --
>         mpg |   1.139888   .0623668     2.39   0.017     1.023977
> 1.268919
>    headroom |    .598804    .227678    -1.35   0.177     .2842103
> 1.261623
> ----------------------------------------------------------------------------
> --
>
> . est store m2
>
> . suest m1 m2
>
> Simultaneous results for m1, m2
>
>                                                  Number of obs   =
> 74
>
> ----------------------------------------------------------------------------
> --
>             |               Robust
>             |      Coef.   Std. Err.      z    P>|z|     [95% Conf.
> Interval]
> -------------+--------------------------------------------------------------
> --
> m1           |
>         mpg |   .1597621   .0514512     3.11   0.002     .0589197
> .2606046
>       _cons |  -4.378866   1.181215    -3.71   0.000    -6.694005
> -2.063727
> -------------+--------------------------------------------------------------
> --
> m2           |
>         mpg |   .1309298   .0546012     2.40   0.016     .0239135
> .2379461
>    headroom |   -.512821   .3168964    -1.62   0.106    -1.133927
> .1082846
>       _cons |  -2.284788   1.838875    -1.24   0.214    -5.888917
> 1.319342
> ----------------------------------------------------------------------------
> --
>
> . matlist e(V)
>
>             | m1                   | m2
>             |       mpg      _cons |       mpg   headroom      _cons
> -------------+----------------------+---------------------------------
> m1           |                      |
>         mpg |  .0026472            |
>       _cons | -.0590473   1.395269 |
> -------------+----------------------+---------------------------------
> m2           |                      |
>         mpg |  .0026541  -.0605161 |  .0029813
>    headroom |  .0012744  -.0545147 |  .0064742   .1004233
>       _cons | -.0627595   1.580435 | -.0860442  -.4539693   3.381462
>
> . test [m1]mpg=[m2]mpg
>
>  ( 1)  [m1]mpg - [m2]mpg = 0
>
>           chi2(  1) =    2.60
>         Prob > chi2 =    0.1071
>
> . scalar num=0.1597621-0.1309298
>
> . scalar denom=0.0026472+0.0029813-(2*0.0026541)
>
> . di chi2tail(1, ((num)/sqrt(denom))^2)
> .10717545
>
> Hope this helps, Tim
>
>
> On Wed, Jun 25, 2008 at 2:51 PM, Brent Fulton <fultonb@berkeley.edu> wrote:
>> Hi,
>>
>> I am using Stata 9.2 and wanted to test whether the parameter estimate for
>> the variable "hispanic" was statistically different in model 1 as compared
>> to model 2. I assume that -suest- followed by -test- is the appropriate
>> procedure, but it is giving me puzzling results. How is it possible when
> the
>> estimates only differ by 0.017 that the p-value of the Wald test is
>> remarkably low at 0.0991? It must be the case that the standard error of
> the
>> 0.017 statistic is very low, but I'm not sure how that's possible given
> that
>> the standard errors of the parameter estimates in both models were 0.347.
>>
>>
>> code:
>> svy, subpop(if sample1==1): logit adhd_d \$ind1a `geodummies'
>> estimates store m1a
>> svy, subpop(if sample1==1): logit adhd_d \$ind1b `geodummies'
>> estimates store m1b
>> suest m1a m1b, svy
>> test [m1a]hispanic=[m1b]hispanic
>>
>> output:
>> m1a--hispanic parameter, (se), (p-value): -0.310 (0.347) (0.372)
>> m1b--hispanic parameter, (se), (p-value): -0.327 (0.347) (0.346)
>>
>> Adjusted Wald test (testing equality of hispanic parameter estimates)
>> F(  1, 78762) =    2.72
>> Prob > F =    0.0991
>>
>>
>> Thanks,
>> Brent Fulton
>>
>> p.s. note that is a similar question to the link below, but no answer was
>> posted
>> http://www.stata.com/statalist/archive/2007-10/msg00870.html
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/support/faqs/res/findit.html
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```