Search
   >> Home >> Resources & support >> FAQs >> Interpreting quadchk results
The following material is based on a question and answer that appeared on Statalist.

How should I interpret changing quadchk results?

Title   Interpreting quadchk results
Author Vince Wiggins, StataCorp
Date April 2001; minor revisions June 2013

Question:

We are using quadchk after fitting a random-effects logistic regression model using xtlogit.

Using the default (12) quadrature points and running quadchk on 8 and 16 points, we are getting a relative difference for the lnsig2u parameter of 0.34. The manual indicates that this means that all parameter estimates are unreliable. The manual also suggests that there is nothing to be gained by increasing the number of quadrature points. However, if we increase the number of points to 24 and run quadchk (on 20 and 28), all relative differences are now <1%, which the manual suggests might be OK.

If, by increasing the number of quadrature points the relative differences do decrease (even if this is not guaranteed) below an acceptable level (say, 1%), can we now use the results with confidence, or does the fact that there were problems with a smaller number of points mean that we shouldn’t trust any of the results?

Answer:

These are difficult questions, and there are not any definitive answers. They are difficult because estimators like xtlogit layer a nonlinear optimization method (Stata’s ml) atop an approximation to a likelihood (using quadrature). Convergence of nonlinear optimization is a difficult enough question. Convergence paths are dependent on the optimization method, the data, and the model, and all of these factors can interact. That the criterion function is approximated by quadrature and that this approximation interacts with the parameter estimates during optimization compounds the problem.

Any generalizations about such estimators are bound to have limited application, and special situations will be common. These are cutting-edge estimators using methods that often tax numerical computation, and they place a much larger burden on the user than do any of Stata’s other estimators; see, for example, the related FAQ on xttobit.

quadchk takes a conservative approach in assessing the stability of quadrature. That strikes me as the right approach; it is better to question and look hard than to blindly accept a possibly unstable solution.

What about the questioners’ results? Let’s look at the output of their quadchks:

. xtset id . quietly xtlogit mantoux i.hiv, re intmethod(ghermite) . quadchk, nooutput Refitting model intpoints() = 8 Refitting model intpoints() = 16 Quadrature check Fitted Comparison Comparison quadrature quadrature quadrature 12 points 8 points 16 points
Log -194.26031 -194.24843 -194.25817
likelihood .01188804 .00214396 Difference
-.0000612 -.00001104 Relative difference
mantoux: -1.1489135 -1.1477097 -1.1505496
Ihiv_2 .00120379 -.00163613 Difference
-.00104776 .00142407 Relative difference
mantoux: 1.1986231 1.1975941 1.2000441
_cons -.00102897 .00142102 Difference
-.00085846 .00118555 Relative difference
lnsig2u: -.03456185 -.02280296 -.03370975
_cons .01175889 .0008521 Difference
-.34022753 -.02465438 Relative difference
. quietly xtlogit mantoux i.hiv, re intpoints(24) intmethod(ghermite) . quadchk, nooutput Refitting model intpoints() = 20 Refitting model intpoints() = 28 Quadrature check Fitted Comparison Comparison quadrature quadrature quadrature 24 points 20 points 28 points
Log -194.25723 -194.25735 -194.25722
likelihood -.00011997 8.854e-06 Difference
6.176e-07 -4.558e-08 Relative difference
mantoux: -1.1508081 -1.1508083 -1.1508011
Ihiv_2 -1.386e-07 7.005e-06 Difference
1.205e-07 -6.087e-06 Relative difference
mantoux: 1.2002655 1.2002679 1.2002592
_cons 2.462e-06 -6.275e-06 Difference
2.051e-06 -5.228e-06 Relative difference
lnsig2u: -.03248611 -.03270534 -.03246283
_cons -.00021923 .00002328 Difference
.0067484 -.00071648 Relative difference

By the time the second quadchk has been run, we have estimated the model using 6 different numbers of quadrature points: 8, 12, 16, 20, 24, and 28. The only place where there is any substantial difference is in the estimate of the log of the variance of the random component, lnsig2u, and this occurs only with 8 quadrature points. All 5 other estimates are close for all of the parameters. At this point, I would feel pretty comfortable with all of the estimates other than those from 8 quadrature points.

What’s more, the rest of the parameters appear to be relatively insensitive to the estimate of lnsig2u. With 8 quadrature points, all the other parameters are still very close to the estimates with more quadrature points. We might visualize a ridge in the likelihood where lnsig2u can vary over some bounded range while the likelihood changes very little, so long as the other parameters remain the same.

We do not have all of the output, but I would guess that the inferences about all of the parameters other than lnsig2u differ very little across the six estimates—even the estimates using quadrature points. I also suspect that users would see little difference in their inferences if they just estimate a marginal model and allow for intra-id correlation by specifying clustering.

. logit mantoux i.hiv, cluster(id)
The Stata Blog: Not Elsewhere Classified Find us on Facebook Follow us on Twitter LinkedIn Google+ Watch us on YouTube