Stata | FAQ: Interpreting quadchk results

We are using quadchk after fitting a random-effects logistic regression model using xtlogit.

Using the default (12) quadrature points and running quadchk on 8 and 16 points, we are getting a relative difference for the lnsig2u parameter of 0.34. The manual indicates that this means that all parameter estimates are unreliable. The manual also suggests that there is nothing to be gained by increasing the number of quadrature points. However, if we increase the number of points to 24 and run quadchk (on 20 and 28), all relative differences are now <1%, which the manual suggests might be OK.

If, by increasing the number of quadrature points the relative differences do decrease (even if this is not guaranteed) below an acceptable level (say, 1%), can we now use the results with confidence, or does the fact that there were problems with a smaller number of points mean that we shouldn’t trust any of the results?

Answer:

These are difficult questions, and there are not any definitive answers. They are difficult because estimators like xtlogit layer a nonlinear optimization method (Stata’s ml) atop an approximation to a likelihood (using quadrature). Convergence of nonlinear optimization is a difficult enough question. Convergence paths are dependent on the optimization method, the data, and the model, and all of these factors can interact. That the criterion function is approximated by quadrature and that this approximation interacts with the parameter estimates during optimization compounds the problem.

Any generalizations about such estimators are bound to have limited application, and special situations will be common. These are cutting-edge estimators using methods that often tax numerical computation, and they place a much larger burden on the user than do any of Stata’s other estimators; see, for example, the related FAQ on xttobit.

[XT] quadchk takes a conservative approach in assessing the stability of quadrature. That strikes me as the right approach; it is better to question and look hard than to blindly accept a possibly unstable solution.

What about the questioners’ results? Let’s look at the output of their quadchks:

. quietly xi: xtlogit mantoux i.hiv, re i(id) intmethod(ghermite) . quadchk, nooutput Refitting model intpoints() = 8 Refitting model intpoints() = 16 Quadrature check Fitted Comparison Comparison quadrature quadrature quadrature 12 points 8 points 16 points ----------------------------------------------------- Log -194.26031 -194.24843 -194.25817 likelihood .01188804 .00214396 Difference -.0000612 -.00001104 Relative difference ----------------------------------------------------- mantoux: -1.1489135 -1.1477097 -1.1505496 Ihiv_2 .00120379 -.00163613 Difference -.00104776 .00142407 Relative difference ----------------------------------------------------- mantoux: 1.1986231 1.1975941 1.2000441 _cons -.00102897 .00142102 Difference -.00085846 .00118555 Relative difference ----------------------------------------------------- lnsig2u: -.03456185 -.02280296 -.03370975 _cons .01175889 .0008521 Difference -.34022753 -.02465438 Relative difference ----------------------------------------------------- . quietly xi: xtlogit mantoux i.hiv, re i(id) intpoints(24) intmethod(ghermite) . quadchk, nooutput Refitting model intpoints() = 20 Refitting model intpoints() = 28 Quadrature check Fitted Comparison Comparison quadrature quadrature quadrature 24 points 20 points 28 points ----------------------------------------------------- Log -194.25723 -194.25735 -194.25722 likelihood -.00011997 8.854e-06 Difference 6.176e-07 -4.558e-08 Relative difference ----------------------------------------------------- mantoux: -1.1508081 -1.1508083 -1.1508011 Ihiv_2 -1.386e-07 7.005e-06 Difference 1.205e-07 -6.087e-06 Relative difference ----------------------------------------------------- mantoux: 1.2002655 1.2002679 1.2002592 _cons 2.462e-06 -6.275e-06 Difference 2.051e-06 -5.228e-06 Relative difference ----------------------------------------------------- lnsig2u: -.03248611 -.03270534 -.03246283 _cons -.00021923 .00002328 Difference .0067484 -.00071648 Relative difference -----------------------------------------------------

By the time the second quadchk has been run, we have estimated the model using 6 different numbers of quadrature points: 8, 12, 16, 20, 24, and 28. The only place where there is any substantial difference is in the estimate of the log of the variance of the random component, lnsig2u, and this occurs only with 8 quadrature points. All 5 other estimates are close for all of the parameters. At this point, I would feel pretty comfortable with all of the estimates other than those from 8 quadrature points.

What’s more, the rest of the parameters appear to be relatively insensitive to the estimate of lnsig2u. With 8 quadrature points, all the other parameters are still very close to the estimates with more quadrature points. We might visualize a ridge in the likelihood where lnsig2u can vary over some bounded range while the likelihood changes very little, so long as the other parameters remain the same.

We do not have all of the output, but I would guess that the inferences about all of the parameters other than lnsig2u differ very little across the six estimates—even the estimates using quadrature points. I also suspect that users would see little difference in their inferences if they just estimate a marginal model and allow for intra-id correlation by specifying clustering.

How should I interpret changing quadchk results?

Question:

Answer: