st: tricks to speed up -xtmelogit-

[Jeph Herrin <stata@spandrel.net>]

All,

I am trying to estimate a series of models using 6 million observations;
the observations are nested within 3000 groups, and the dichotomous
outcome is somewhat rare, occurring in about 0.5% of observations.
There are about 150 independent variables, and so my basic model looks
like this:

 . xtmelogit Y x1-x150 || group:

This took approximately 3 weeks to converge on a high end machine
(3.2GHz, Intel Core i7, 24GB RAM). I saved the estimation result

 . est save main

but now would like to estimate some related models of the form

 . xtmelogit Y x1-x150 z1 z2 || group:

and would like to think I can shave some considerable time off the
estimation using the prior information available. I tried

 . est use main
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

but this gave me an error that the likelihood was flat and nothing
proceed. So I've thought of some other approaches, but am not sure what
I expect to be most efficient, and would prefer not to spend weeks
figuring it out.

One idea was to use a sample, estimate the big model, and then use
that as a starting point:

 . est use main
 . matrix b = e(b)
 . gen byte sample = (uniform()*1000)<1
 . xtmelogit Y x1-x150 z1 z2 if sample || group:, from(b)
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

Another was to first use Laplace iteration, and start with that result:

 . est use main
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 if sample || group:, from(b) laplace
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

I'd appreciate any insight into which of these approaches might shave
a meaningful amount of time off of getting the final estimates, or if
there is another that I could try.

thanks,
Jeph



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/