Note: The following material is based on a question and answer
that appeared on Statalist.

Title | Interpreting quadchk results | |

Author | Vince Wiggins, StataCorp |

We are using **quadchk**
after fitting a random-effects logistic regression
model using **xtlogit**.

Using the default (12) quadrature points and running **quadchk** on 8 and
16 points, we are getting a relative difference for the **lnsig2u**
parameter of 0.34. The manual indicates that this means that all parameter
estimates are unreliable. The manual also suggests that there is nothing to
be gained by increasing the number of quadrature points. However, if we
increase the number of points to 24 and run **quadchk** (on 20 and 28),
all relative differences are now <1%, which the manual suggests might be
OK.

If, by increasing the number of quadrature points the relative differences do decrease (even if this is not guaranteed) below an acceptable level (say, 1%), can we now use the results with confidence, or does the fact that there were problems with a smaller number of points mean that we shouldn’t trust any of the results?

These are difficult questions, and there are not any definitive
answers. They are difficult because estimators like **xtlogit** layer a
nonlinear optimization method (Stata’s
**ml**) atop an
approximation to a likelihood (using quadrature). Convergence of nonlinear
optimization is a difficult enough question. Convergence paths are
dependent on the optimization method, the data, and the model, and all of
these factors can interact. That the criterion function is approximated by
quadrature and that this approximation interacts with the parameter
estimates during optimization compounds the problem.

Any generalizations about such estimators are bound to have limited
application, and special situations will be common. These are cutting-edge
estimators using methods that often tax numerical computation, and they
place a much larger burden on the user than do any of Stata’s other
estimators; see, for example, the related FAQ on
**xttobit**.

**quadchk** takes a conservative approach in assessing the stability
of quadrature. That strikes me as the right approach; it is better to
question and look hard than to blindly accept a possibly unstable solution.

What about the questioners’ results? Let’s look at the output
of their **quadchk**s:

Log -194.26031 -194.24843 -194.25817 | |

likelihood .01188804 .00214396 | Difference |

-.0000612 -.00001104 | Relative difference |

mantoux: -1.1489135 -1.1477097 -1.1505496 | |

Ihiv_2 .00120379 -.00163613 | Difference |

-.00104776 .00142407 | Relative difference |

mantoux: 1.1986231 1.1975941 1.2000441 | |

_cons -.00102897 .00142102 | Difference |

-.00085846 .00118555 | Relative difference |

lnsig2u: -.03456185 -.02280296 -.03370975 | |

_cons .01175889 .0008521 | Difference |

-.34022753 -.02465438 | Relative difference |

Log -194.25723 -194.25735 -194.25722 | |

likelihood -.00011997 8.854e-06 | Difference |

6.176e-07 -4.558e-08 | Relative difference |

mantoux: -1.1508081 -1.1508083 -1.1508011 | |

Ihiv_2 -1.386e-07 7.005e-06 | Difference |

1.205e-07 -6.087e-06 | Relative difference |

mantoux: 1.2002655 1.2002679 1.2002592 | |

_cons 2.462e-06 -6.275e-06 | Difference |

2.051e-06 -5.228e-06 | Relative difference |

lnsig2u: -.03248611 -.03270534 -.03246283 | |

_cons -.00021923 .00002328 | Difference |

.0067484 -.00071648 | Relative difference |

By the time the second **quadchk** has been run, we have estimated the
model using 6 different numbers of quadrature points: 8, 12, 16, 20, 24,
and 28. The only place where there is any substantial difference is in the
estimate of the log of the variance of the random component, **lnsig2u**,
and this occurs only with 8 quadrature points. All 5 other estimates are
close for all of the parameters. At this point, I would feel pretty
comfortable with all of the estimates other than those from 8 quadrature
points.

What’s more, the rest of the parameters appear to be relatively
insensitive to the estimate of **lnsig2u**. With 8 quadrature points,
all the other parameters are still very close to the estimates with more
quadrature points. We might visualize a ridge in the likelihood where
**lnsig2u** can vary over some bounded range while the likelihood changes
very little, so long as the other parameters remain the same.

We do not have all of the output, but I would guess that the inferences
about all of the parameters other than **lnsig2u** differ very little
across the six estimates—even the estimates using quadrature points.
I also suspect that users would see little difference in their inferences if
they just estimate a marginal model and allow for intra-id correlation by
specifying clustering.