Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

st: tricks to speed up -xtmelogit-


From   Jeph Herrin <stata@spandrel.net>
To   statalist@hsphsun2.harvard.edu
Subject   st: tricks to speed up -xtmelogit-
Date   Tue, 21 Dec 2010 14:15:12 -0500

All,

I am trying to estimate a series of models using 6 million observations;
the observations are nested within 3000 groups, and the dichotomous
outcome is somewhat rare, occurring in about 0.5% of observations.
There are about 150 independent variables, and so my basic model looks
like this:

 . xtmelogit Y x1-x150 || group:

This took approximately 3 weeks to converge on a high end machine
(3.2GHz, Intel Core i7, 24GB RAM). I saved the estimation result

 . est save main

but now would like to estimate some related models of the form

 . xtmelogit Y x1-x150 z1 z2 || group:

and would like to think I can shave some considerable time off the
estimation using the prior information available. I tried

 . est use main
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

but this gave me an error that the likelihood was flat and nothing
proceed. So I've thought of some other approaches, but am not sure what
I expect to be most efficient, and would prefer not to spend weeks
figuring it out.

One idea was to use a sample, estimate the big model, and then use
that as a starting point:

 . est use main
 . matrix b = e(b)
 . gen byte sample = (uniform()*1000)<1
 . xtmelogit Y x1-x150 z1 z2 if sample || group:, from(b)
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

Another was to first use Laplace iteration, and start with that result:

 . est use main
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 if sample || group:, from(b) laplace
 . matrix b = e(b)
 . xtmelogit Y x1-x150 z1 z2 || group:, from(b) refineopts(iterate(0))

I'd appreciate any insight into which of these approaches might shave
a meaningful amount of time off of getting the final estimates, or if
there is another that I could try.

thanks,
Jeph



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index