Re: st: too good to be true : lr test in mlogit?

[John Litfiba <cariboupad@gmx.fr>]

Dear Marten and Joerg,

Thank you for this idea : it can be a interesting solution indeed!
Maybe I shall add 2 remarks here :

1) The log likelihood doesnt converge when I try to fit a random or
fixed effect with xtlogit on my entire dataset..
I have to chose a very "small" (well, compared to the total size of
the sample) of about 10000 observations in order to see the results...
otherwise I get an error message after 3 or 4 iterations

2) The idea of running lets say M regressions over randomly chose
samples could be a solution, but it is statistically valid ? I mean if
I obtain the distribution of the parameters across my M simulation can
I infer something on the parameters of the simulation that should have
been done on the entire dataset ?

Have a good day
Best Regards,

On 13 May 2011 15:11, Joerg Luedicke <joerg.luedicke@gmail.com> wrote:
> On Fri, May 13, 2011 at 6:56 AM, Maarten Buis <maartenlbuis@gmail.com> wrote:
>> On Fri, May 13, 2011 at 11:21 AM, John Litfiba wrote:
>>> But after various tentatives I think that it is not computationally
>>> feasible (at least on my PC) to fit a random or fixed effect logit on
>>> such a large dataset
>>
>> I would not give up on that yet. I would select my model on a sample
>> of your data (I would sample higher level units, rather than
>> individual observations),
>
> This is what I would do, too. In addition, I would be interested in
> the variation of results from different samples. I would recommend
> setting up a program that is drawing a random sample, say 1%, running
> the model, and saving the results. Then let that program run a 1000 or
> whatever times. (How this can be done is described here:
> http://www.stata.com/statalist/archive/2011-04/msg01220.html) Small
> variation in results could then be indicative of the fact that it is
> pointless anyway to have a sample size of > 2mio. In that case, i.e.
> low variation of results, you can also compare your xtlogit model with
> the model you have now etc.  Large variation could be an indication of
> the fact that a very large sample size may be appropriate. You could
> also play with different sample sizes (say, 1%, 3%, 5%) and see how
> the variation of results changes across sample size. At the very
> least, this will provide useful information about your sample and
> might be helpful with respect to the specification of your model.
>
> J.
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/