# Re: st: Very low sensitivity of fixed-effects logit

 From Steven Samuels To statalist@hsphsun2.harvard.edu Subject Re: st: Very low sensitivity of fixed-effects logit Date Thu, 5 Jun 2008 21:57:46 -0400

Leda's fixed-effects probabilities would also have been distorted if she had specified the default prediction after fixed-effects - xtlogit-. The default is the probability of a positive response, conditional on there being exactly *one* positive response in the cluster.

-Steve

On Jun 5, 2008, at 9:13 PM, Steven Samuels wrote:

I can think of several reasons why the predictions differ. One is primary: predictions from the fixed-effects model are nonsensical. The model omits a constant (see the output), and the predictions omit fixed panel effects. On the other hand, the random effects model includes a constant and the predictions (with the pu0 optioons) omit a zero-average random panel effect.

Some other thoughts:
1. iweights cannot vary within a panel (cluster), so they are ordinarily a poor substitute for pweights.

2. I don't know what you mean about wanting to exclude "the effect of certain unobserved cluster
variables, as geographical access, in X4". If you want to know if there are cluster differences, a more direct approach would be to fit the random effects model and test whether the between-cluster SD (sigma) is zero. You can also include cluster-level variables in the prediction equation.

Good luck!

-Steve

On Jun 5, 2008, at 7:22 PM, Leda Inga wrote:

Thank you very much. I've used the following commands:

Fixed-effects models:

# delimit;
xi: xtlogit Y X1 X2 X3 i.X4 if [rural==1] [iw=pwi], fe i(V001) nolog or
;
delimit cr

Random-effects models.

# delimit;
xi: xtlogit Y X1 X2 X3 i.X4 region3 region5 if [rural==1] [iw=pwi],
re i(V001) nolog or
;
delimit cr

I didn't use svy: because it didn't allow me to use xtlogit (I'm
interested in excluding the effect of certain unobserved cluster
variables, as geographical access, in X4 and get a consistent
estimator of its impact on Y). I know that when using survey data it's
not correct to use the "if", but svy, subpop( www). Nevertheless, I
couldn't think of another way.

To calcute the predictions I used:

predict Yhat if e(sample), pu0

I didn't use lsens because it can't be used after xtlogit.

I hope this information helps to clarify my problem.

Leda,

You need to tell us exactly which logistic commands you ran. - svy: logit-
would be appropriate for fixed-effects logistic regression with stratified,
weighted, and clustered, data. -logistic- itself will not take stratum
variables, but will take weights and clustering.

-xtmelogit- cannot handle survey weights, but would otherwise be the proper
program for random effects logistic regression. The user-written package
-gllamm- can do multi-level logistic regression with weights. With random
effects models, several types of predictions are possible, depending on
whether random-effects are permitted in the prediction.

The user-written command -rocss- was written for Stata Version 8. It does
not accept weights and shouldn't be used with weighted survey data. After
-logistic-, -logit-, or -svy: logistic-, you should try Stata's built-in
commands: -lsens- and -lroc- .

After -gllamm- , you could compute your own tables of specificity and
sensitivity by printing out the weighted tables of probabilities. If there
are too many distinct values, round the probabilities to the nearest 0.01
before tabling.

Steve

On Jun 5, 2008, at 5:40 PM, Leda Inga wrote:

Hi,

I'm runnig a fixed-effects and random-effects logit with DHS
(Demographic Health Survey) data. The groups are the clusters within
which each female belongs to. Given the recomendation in a previous
statalist mail
```(http://www.stata.com/statalist/archive/2007-06/msg00818.html
```
), I calculated the percentege correctly predicted for both models.
Nevertheless, I'm very surprised because the sensitivity (% correctly
predicted of positive outcomes) is very low for the fixed-effects
logit while the specificity (% correctly predicted of negative
outcomes: zeros) very high. On the other hand, both measures are more
acceptable for the random effects models. Besides, the pvalue of the
hausman test is zero.
Are this measures (sensititvity and specificity) the most appropiate
for measuring the quality of the results of this kind of models? And
are the results I've gotten frequent when comparing a fixed effects
versus a random effects model?

Here are the results given by rocss, a command the calculates the
sensitivity (sens) and specificity (spec) for different cutoffs:

Fixed-effects logit:

cutoff sens spec omspec cclass carea
------------------------------------------------------
1. 0.000 1.0000 0.0000 1.0000 47.5149 0.0000
2. 0.100 0.7250 0.6656 0.3344 69.3837 0.5741
3. 0.200 0.4250 0.9026 0.0974 67.5660 0.7104
4. 0.300 0.2313 0.9637 0.0363 61.5734 0.7304
5. 0.400 0.1219 0.9908 0.0092 57.7961 0.7352
6. 0.500 0.0580 0.9951 0.0049 54.9844 0.7356
7. 0.600 0.0209 0.9995 0.0005 53.4507 0.7358
8. 0.700 0.0078 1.0000 0.0000 52.8543 0.7358
9. 0.800 0.0036 1.0000 0.0000 52.6555 0.7358
10. 0.900 0.0006 1.0000 0.0000 52.5135 0.7358
11. 1.000 0.0000 1.0000 0.0000 52.4851 0.7358

```+------------------------------------------------------+
```
```Random-effects logit:

cutoff     sens     spec   omspec    cclass    carea
------------------------------------------------------
1.    0.000   1.0000   0.0000   1.0000   48.9190   0.0000
2.    0.100   0.9791   0.1930   0.8070   57.7572   0.1910
3.    0.200   0.9291   0.3791   0.6209   64.8135   0.3685
4.    0.300   0.8490   0.5321   0.4679   68.7099   0.5046
5.    0.400   0.7528   0.6726   0.3274   71.1808   0.6171
6.    0.500   0.6353   0.7809   0.2191   70.9670   0.6923
7.    0.600   0.5036   0.8721   0.1279   69.1851   0.7442
8.    0.700   0.3871   0.9293   0.0707   66.4053   0.7697
9.    0.800   0.2448   0.9674   0.0326   61.3923   0.7817
10.    0.900   0.0971   0.9930   0.0070   55.4764   0.7861
11.    1.000   0.0000   1.0000   0.0000   51.0810   0.7864

Any help would be very appreciated.
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

```
```*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

```
```*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```
```*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```
```*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
```