Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Generalized lineal models with survey data


From   Paolina Medina <[email protected]>
To   [email protected]
Subject   Re: st: Generalized lineal models with survey data
Date   Tue, 27 Jul 2010 12:32:01 -0500

Thank you very much for your useful and enlightening answer!!

If you let me, i would like to ask you just 2 more questions:

Why do you think this very large quasi log likelihood value is
arising? Could it be just because i am using a lot of dummy variables?
Or may be just my regressors are not the best predictors for the
dependent variable?

And, finally if you were to choose between the nbreg regression that i
just posted and the following poisson regression with the same
regressors, which one would you choose and why? Or which test would
you run to answer that question? The thing is that i think there is no
gof test available for this regressions with survey data in stata. Or
is it?

Again, thank you very very much for your patience and everything.

Poisson regression:

 svy: poisson  ncels resmay6 numradios nTVs tfijo tpaga luz ncompus
internet prim2 sec2 prepa2 normal2 tec2 pro2 m2 doc2 tra
> bajadores e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22 e23 e24 e25 e26 e27 e28 e29 e30 e3
> 1 e32 estrato1 estrato2 estrato3 estrato4, log
(running poisson on estimation sample)
note: e3 dropped due to collinearity
note: estrato2 dropped due to collinearity

Iteration 0:   log pseudolikelihood =  -32297663
Iteration 1:   log pseudolikelihood =  -32264546
Iteration 2:   log pseudolikelihood =  -32264459
Iteration 3:   log pseudolikelihood =  -32264459

Poisson regression                                Number of obs   =       6089
                                                  LR chi2(51)     =  1.761e+07
                                                  Prob > chi2     =     0.0000
Log pseudolikelihood =  -32264459                 Pseudo R2       =     0.2143

------------------------------------------------------------------------------
       ncels |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     resmay6 |  -.1189499   .0003869  -307.48   0.000    -.1197081   -.1181917
   numradios |   .0516967   .0002443   211.65   0.000      .051218    .0521754
        nTVs |   .1765135   .0001923   918.12   0.000     .1761367    .1768903
       tfijo |  -.1133193   .0004484  -252.70   0.000    -.1141982   -.1124404
       tpaga |   .1687625   .0004486   376.22   0.000     .1678833    .1696417
         luz |    .760426   .0046124   164.87   0.000     .7513859    .7694661
     ncompus |   .1401203   .0003325   421.45   0.000     .1394686    .1407719
    internet |  -.0047513   .0005739    -8.28   0.000    -.0058761   -.0036265
       prim2 |   .1220794   .0004182   291.90   0.000     .1212596    .1228991
        sec2 |   .1243139   .0002344   530.28   0.000     .1238544    .1247733
      prepa2 |   .0985697   .0002857   344.95   0.000     .0980097    .0991298
     normal2 |  -.0754923   .0007591   -99.45   0.000    -.0769801   -.0740045
        tec2 |   .0036673   .0007941     4.62   0.000      .002111    .0052237
        pro2 |   .0751998   .0004025   186.83   0.000     .0744109    .0759886
          m2 |   .0292362   .0007304    40.03   0.000     .0278047    .0306678
        doc2 |  -.1606452    .002035   -78.94   0.000    -.1646338   -.1566567
trabajadores |   .1317087   .0001916   687.47   0.000     .1313332    .1320842
          e1 |   .0016868    .002384     0.71   0.479    -.0029858    .0063595
          e2 |   .0648902    .002022    32.09   0.000     .0609271    .0688533
          e4 |  -.0181873   .0026577    -6.84   0.000    -.0233962   -.0129784
          e5 |  -.1868012   .0021267   -87.83   0.000    -.1909695   -.1826329
          e6 |   .1466339   .0026388    55.57   0.000      .141462    .1518058
          e7 |  -.3935259   .0021961  -179.20   0.000    -.3978301   -.3892217
          e8 |  -.1245186   .0020253   -61.48   0.000    -.1284882    -.120549
          e9 |  -.2299049   .0019112  -120.29   0.000    -.2336509    -.226159
         e10 |   -.083798   .0022478   -37.28   0.000    -.0882035   -.0793924
         e11 |  -.3531031   .0020698  -170.60   0.000    -.3571598   -.3490464
         e12 |  -.4600155     .00238  -193.28   0.000    -.4646802   -.4553507
         e13 |  -.3038536    .002392  -127.03   0.000    -.3085418   -.2991653
         e14 |  -.1274274   .0019299   -66.03   0.000      -.13121   -.1236447
         e15 |  -.3696341   .0019058  -193.95   0.000    -.3733694   -.3658987
         e16 |  -.0019455   .0020393    -0.95   0.340    -.0059423    .0020514
         e17 |  -.1287657   .0023403   -55.02   0.000    -.1333526   -.1241788
         e18 |   -.106628   .0026009   -41.00   0.000    -.1117257   -.1015304
         e19 |  -.1713332   .0019934   -85.95   0.000    -.1752401   -.1674263
         e20 |  -.3297292   .0023891  -138.02   0.000    -.3344117   -.3250467
         e21 |  -.2838863   .0020318  -139.72   0.000    -.2878685   -.2799042
         e22 |   -.079044   .0022371   -35.33   0.000    -.0834287   -.0746593
         e23 |   .1517949   .0022013    68.96   0.000     .1474803    .1561094
         e24 |  -.2631791   .0022394  -117.52   0.000    -.2675682     -.25879
         e25 |   .0587084   .0021346    27.50   0.000     .0545247    .0628921
         e26 |   .0442957   .0020797    21.30   0.000     .0402195    .0483719
         e27 |   .1651334   .0021816    75.69   0.000     .1608575    .1694092
         e28 |  -.0281828   .0020233   -13.93   0.000    -.0321484   -.0242172
         e29 |  -.6082313    .002915  -208.66   0.000    -.6139446   -.6025181
         e30 |   -.144252   .0020102   -71.76   0.000    -.1481918   -.1403121
         e31 |   .0250983   .0022023    11.40   0.000     .0207819    .0294146
         e32 |  -.1352466   .0025166   -53.74   0.000    -.1401789   -.1303142
    estrato1 |    .189495   .0005579   339.69   0.000     .1884016    .1905883
    estrato3 |   -.322916   .0010167  -317.62   0.000    -.3249087   -.3209234
    estrato4 |  -.4792419   .0008096  -591.98   0.000    -.4808286   -.4776552
       _cons |  -1.493952   .0049766  -300.19   0.000    -1.503706   -1.484198
------------------------------------------------------------------------------

Computing scores...

Survey results:

Survey: Poisson regression

Number of strata   =         4                  Number of obs      =      6089
Number of PSUs     =       837                  Population size    =  27782772
                                                Design df          =       833
                                                F(  51,    783)    =     50.61
                                                Prob > F           =    0.0000

------------------------------------------------------------------------------
             |             Linearized
       ncels |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
     resmay6 |  -.1189499   .0302713    -3.93   0.000    -.1783669   -.0595329
   numradios |   .0516967    .019889     2.60   0.010     .0126581    .0907352
        nTVs |   .1765135   .0210548     8.38   0.000     .1351867    .2178402
       tfijo |  -.1133193   .0389423    -2.91   0.004    -.1897558   -.0368828
       tpaga |   .1687625   .0330961     5.10   0.000     .1038009     .233724
         luz |    .760426   .4341483     1.75   0.080    -.0917272    1.612579
     ncompus |   .1401203   .0232643     6.02   0.000     .0944568    .1857837
    internet |  -.0047513   .0434178    -0.11   0.913    -.0899725    .0804699
       prim2 |   .1220794   .0353386     3.45   0.001     .0527162    .1914425
        sec2 |   .1243139   .0272578     4.56   0.000     .0708118     .177816
      prepa2 |   .0985697   .0253691     3.89   0.000     .0487748    .1483647
     normal2 |  -.0754923   .0678928    -1.11   0.266    -.2087533    .0577688
        tec2 |   .0036673   .0752994     0.05   0.961    -.1441316    .1514663
        pro2 |   .0751998    .038429     1.96   0.051    -.0002293    .1506288
          m2 |   .0292362     .04043     0.72   0.470    -.0501204    .1085929
        doc2 |  -.1606452   .0938375    -1.71   0.087     -.344831    .0235405
trabajadores |   .1317087   .0193343     6.81   0.000      .093759    .1696583
          e1 |   .0016868   .1667689     0.01   0.992    -.3256499    .3290235
          e2 |   .0648902    .147677     0.44   0.660    -.2249727    .3547531
          e4 |  -.0181873   .1666332    -0.11   0.913    -.3452575    .3088829
          e5 |  -.1868012   .1583675    -1.18   0.239    -.4976475    .1240451
          e6 |   .1466339   .1557807     0.94   0.347     -.159135    .4524028
          e7 |  -.3935259   .2460116    -1.60   0.110    -.8764014    .0893497
          e8 |  -.1245186   .1549826    -0.80   0.422    -.4287209    .1796838
          e9 |  -.2299049   .1487182    -1.55   0.123    -.5218113    .0620015
         e10 |   -.083798   .2982254    -0.28   0.779    -.6691596    .5015637
         e11 |  -.3531031   .1536798    -2.30   0.022    -.6547482   -.0514579
         e12 |  -.4600155   .1721164    -2.67   0.008    -.7978484   -.1221826
         e13 |  -.3038536   .1661253    -1.83   0.068     -.629927    .0222199
         e14 |  -.1274274   .1614396    -0.79   0.430    -.4443035    .1894488
         e15 |  -.3696341    .151954    -2.43   0.015    -.6678918   -.0713763
         e16 |  -.0019455   .2189627    -0.01   0.993    -.4317289    .4278379
         e17 |  -.1287657   .1711779    -0.75   0.452    -.4647564     .207225
         e18 |   -.106628    .162161    -0.66   0.511    -.4249202    .2116641
         e19 |  -.1713332   .1540608    -1.11   0.266    -.4737261    .1310597
         e20 |  -.3297292    .247216    -1.33   0.183    -.8149688    .1555104
         e21 |  -.2838863   .1542873    -1.84   0.066    -.5867239    .0189512
         e22 |   -.079044   .1822687    -0.43   0.665     -.436804     .278716
         e23 |   .1517949   .1548563     0.98   0.327    -.1521596    .4557493
         e24 |  -.2631791    .179553    -1.47   0.143    -.6156086    .0892503
         e25 |   .0587084    .176449     0.33   0.739    -.2876286    .4050453
         e26 |   .0442957   .1611852     0.27   0.784    -.2720812    .3606725
         e27 |   .1651334    .168757     0.98   0.328    -.1661055    .4963723
         e28 |  -.0281828   .1619115    -0.17   0.862    -.3459853    .2896197
         e29 |  -.6082313   .1689908    -3.60   0.000    -.9399292   -.2765335
         e30 |   -.144252    .190234    -0.76   0.448    -.5176462    .2291423
         e31 |   .0250983   .1760391     0.14   0.887    -.3204341    .3706306
         e32 |  -.1352466   .2260741    -0.60   0.550    -.5789884    .3084952
    estrato1 |    .189495   .0642525     2.95   0.003     .0633791    .3156108
    estrato3 |   -.322916    .111296    -2.90   0.004    -.5413697   -.1044624
    estrato4 |  -.4792419   .1214264    -3.95   0.000    -.7175797   -.2409042
       _cons |  -1.493952    .462237    -3.23   0.001    -2.401238   -.5866655
------------------------------------------------------------------------------










On Tue, Jul 27, 2010 at 12:02 PM, Stas Kolenikov <[email protected]> wrote:
> No, you don't have any problems with the degrees of freedom, which is
> #PSUs - #strata = 837-4 = 833, and is reported as such. So I tend to
> believe in Steven's story about empirical underidentification of the
> overdispersion parameter: the likelihood is so flat in alpha that the
> curvature (inverse of the variance) of the likelihood wrt this
> parameter cannot be estimated with numeric accuracy that Stata would
> find acceptable to report. And yes, this is an indication that
> overdispersion is not such a great problem: coniditioning on
> covariates and taking weights into account seems to make your data
> approximately OK.
>
> As for the general convergence problems, they may be caused by the
> scale of weights. Note that your log pseudo-likelihood has 8 digits
> before the decimal point, and typically Stata wants to optimize things
> down to 7 or so digits after the decimal point, that is, you need to
> have about 15 reliable digits to declare convergence. That's too much
> to ask for, as 15 digits is the accuracy limit of the -datatype-
> double. In this situation (and in this situation only), it would be OK
> to relax the convergence criteria by specifying something like
> -ltolerance(1e-3)- instead of the default 1e-7; or rescale the weights
> so that they sum up to say sample size rather than the population
> size.
>
> On Tue, Jul 27, 2010 at 5:30 PM, Paolina Medina
> <[email protected]> wrote:
>> Thank you both, very much.
>> So this almost zero alpha, without a confidence interval can be taken
>> to indicate that there is no overdispersion in the model?
>> Here is my svyset statement and the complete output..
>> I am using 52 regressors (including the constant), i really dont know
>> how many are the design degrees of freedom... But in fact whenever i
>> take any of these regressors i get a lot of troubles with convergence
>> in the survey results (not concave or backed up) and i have to throw
>> away many other regressors to get convergence again.
>> Do you know anything i can do to fix this?
>
> --
> Stas Kolenikov, also found at http://stas.kolenikov.name
> Small print: I use this email account for mailing lists only.
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>



-- 
Paolina Medina Palma

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index