Statalist


[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: xtlogit is slow


From   ymarchenko@stata.com (Yulia Marchenko, StataCorp LP)
To   statalist@hsphsun2.harvard.edu, "Mitchell F. Berman" <mfb1@columbia.edu>
Subject   st: xtlogit is slow
Date   Wed, 09 Jul 2008 17:40:28 -0500

On Sunday July 06, 2008 "Mitchell F. Berman" <mfb1@columbia.edu> reported a
speed issue for -xtlogit, pa-: 

> I've noticed that xtlogit is slow compared to SAS genmod to get logistic
> regressions using an exchangeable correlation model.  For large data sets
> (300,000 rows-15 variables) xtlogit takes 45 minutes, SAS takes 1.5 minutes.
> The results are identical.
> ...

The issue is actually associated with -xtgee-, which is used by -xtlogit- to
fit population-averaged models.

After analyzing Mitchell's data, we've determined that (as some contributors
on this thread have speculated) slow downs are not necessarily caused by large
datasets.  The slow downs are caused by large panels -- Mitchell's data has
panels as large as 2,600 or so observations.

-xtgee- works through matrix manipulations involving working correlation
matrices at the panel level.  In Mitchell's case -xtgee- is working with
square matrices of dimension as high as 2,600.  -xtgee- in its current
implementation performs manipulations directly on these matrices, whereas an
implementation that takes into account the structure (and possible sparseness)
of these working correlations (e.g., exchangeable) would be much more
efficient.

As such, this is something that we will seriously look into adding.


-- Yulia Marchenko				-- Bobby Gutierrez
ymarchenko@stata.com				rgutierrez@stata.com
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index