Re: st: Significance of between-cluster variance in xtlogit

[Austin Nichols <austinnichols@gmail.com>]

Michael Josten <mj.josten@googlemail.com>:
A confidence interval for the log of variance including zero means
that you cannot reject that the variance is one, not that you cannot
reject that the variance is zero.

I note you are making a strong assumption in your regression about the
distribution of interviewer effects under both modes, which may not be
justified--have you graphed the distribution of residuals (say,
interviewer-level mean residuals) from a model that does not impose
such an assumption?

On Thu, Mar 15, 2012 at 11:14 AM, Michael Josten
<mj.josten@googlemail.com> wrote:
> Hi there,
>
> I am currently dealing with multilevel logistic regression models to
> investigate interviewer and interview mode effects. I mainly use
> xtlogit for this and everything is working fine except for the
> interpretation of the actual significance of the observed
> between-cluster variance caused by the interviewers. Throughout all of
> my models, from the random intercept only model to the full random
> intercept model with all respondent and interviewer variables, rho
> continuously is considerably high (0.3-0.2) which is indicative of a
> remarkable interviewer effect. The provided likelihood ratio test for
> the null hypothesis that the residual between-cluster variance is zero
> indicates that rho indeed is highly statistically significant for each
> single model. For me this is a sign that the multilevel design is
> appropriate because otherwise I would get biased estimators.
>
> However, a second result is contradictory to this interpretation: the
> log cluster variance - indicated as “lnsig2u” - doesn’t seem to be
> significant anymore at the moment I control for the interview mode
> CATI or CAPI. I can see that looking at the confidence interval which
> embraces zero, but I can’t comprehend what kind of statistical test is
> performed here even though there is some standard error provided.
> Stata manual doesn’t give information about this procedure. However,
> this would be a sign that it isn’t the interviewers in general who are
> cheating but it is only the difference between CATI and
> CAPI-Interviewer.
>
> Even though this makes sense as there are different incentives in the
> two modes for interviewers to cheat, there are still two contradictory
> findings: on the one hand rho as the proportion of between-variance of
> the total variance indicates that there is a significant
> between-cluster variance, on the other hand in the same model some
> sort of untransparent test comes to the result that exactly this
> between-cluster variance isn’t significant for its own. Does anyone
> have experience with this kind of problem and can help me as I am not
> quite sure on which result to rely on for my interpretation as their
> messages are totally different. Thanks!
>
> Best,
> Michael

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/