Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down at the end of May, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
Eduardo Andrés Alfonso Sierra <ealfonsosierra@googlemail.com> |

To |
statalist@hsphsun2.harvard.edu |

Subject |
Re: st: Bivariate probit model with partial observability and survey data |

Date |
Sun, 21 Aug 2011 17:19:44 +0200 |

Hi Thanks for the suggestion to use the noisily option of the svy. It reveals that the model, when using svy, does not seem to converge. Here's how the output looks like: It starts Fitting comparison model and it finished after 339 iterations. Then, in Fitting full model, it keeps on iterating, quite in the same log likelihood value after 700 iterations and always showing backed up. Now, after more than 4 days running!, it keeps on going. Exactly the same specification with the same data but without the svy, converged after a couple of hours (although it seems quite a lot of time, it sounds reasonable since the model has a lot of explanatory variables -more than 20- and the dataset is not a small one -147,096 obs-). So, as Professor Kolenikov said, "With weights and such, the computation may take even longer than with i.i.d. settings, although it should be in the order of single to low double digit %, rather than a factor of 2 or 3" ¹. In this case however, the computation time with weights is taking so much longer (a factor of 48 and counting!!!). I am aware of the poor convergence properties of this model, but I really don´t get why the weights and adjustments for survey design might change it so radically, so any idea on how to proceed would be greatly appreciated. Additionally, I have already followed the suggestion in Stata help, which says that "This particular model commonly has poor convergence properties, so we recommend that you use the difficult option if you want to fit the Poirier partial observability model" ². I have tried with the difficult option and the problem persists (without svy it converges after a while, but with svy it is still iterating). Furthermore, I found in Statalist that "Changes in Stata in version 11 have occasionally produced results like this" ³, referring to a case in which a probit model converged in Stata 10 but not in Stata 11, showing a similar iteration process as in my case ("after about five iterations it says (backed up), and then it gets stuck on the same log psuedolikelihood, forever it appears" ³). So I also tried running my model in Stata 10 and once again, the biprobit without svy converges after a while but using svy, it seems it does not. Finally, Professor Kolenikov pointed out that in my original email I forgot to provide the complete reference of the paper I cited. He is right and I apologize for that. Here´s the reference. Poirier, D. J. (1980). Partial observability in bivariate probit models. Journal of Econometrics, 12(2), 209-217. Thanks in advance for any help or suggestion. Eduardo ¹ Reply from Professor Kolenikov to my original post. See previous post in this thread. ² http://www.stata.com/help.cgi?biprobit, accessed 21-Aug-2011 ³ http://www.stata.com/statalist/archive/2010-08/msg01331.html and http://www.stata.com/statalist/archive/2010-08/msg01273.html, accessed 21-Aug-2011 On Fri, Aug 19, 2011 at 4:20 PM, Stas Kolenikov <skolenik@gmail.com> wrote: > -svy- likes to take over the formatting of output, and, in particular, > it suppresses the iteration log. If you type -svy, noisily: biprobit > (whatever)-, you'd probably see more. With weights and such, the > computation may take even longer than with i.i.d. settings, although > it should be in the order of single to low double digit %, rather than > a factor of 2 or 3. > > BTW, you need to provide full references to whatever you are citing. > And the issue that you have is not with the bivariate probit, but with > -svy-, so non-econometricians would have just skipped your email. > > 2011/8/18 Eduardo Andrés Alfonso Sierra <ealfonsosierra@googlemail.com>: >> Hello everyone >> >> I am estimating a bivariate probit with partial observability >> following Poirier (1980), hoping to disentangle the determinants of >> the decision to participate in a health related program and the >> decision made by the program to accept or reject a person (which, for >> some variables might be in opposite directions). Hence, I have two >> decision functions that looks something like this: >> >> Y1 = X1*b + X2*c + Ui >> Y2 = X1*d + + Vi >> >> The first one, models the decision made by the person whether or not >> to apply to the program. >> The second one, models the decision made by the people who run the >> program whether to accept or reject the person (and the exclusion >> restriction comes from the fact that the people running the program >> cannot observe several characteristics of the person applying, which >> might actually influence his decision to apply - X2). >> >> Partial observability arises from the fact that I can only observe if >> the person participates in the program (Y=Y1*Y2), but not the >> decisions Y1 and Y2 independently. >> >> So I am doing something like this in Stata: >> >> biprobit (apply = age male educ healthstatus riskfactors riskaversion) >> (accept = age male educ healthstatus), partial >> >> Where apply and accept are both equals to the participation status >> (Y=Y1*Y2), as explained in >> http://www.stata.com/support/faqs/stat/biprobit.html >> >> After trying several variables, a reasonable model converged (although >> it takes quite a lot of time). >> >> Now my problem is that I also have to take into account the survey >> design (quite complex I think - two stages, stratified, ...). >> According to the survey design, for estimating proportions and other >> parameters, I successfully configured the survey data in Stata as >> follows: >> >> svyset mysvypsu [pweight=expfact], fpc(mysvyfpc) strata(mysvystrata) >> singleunit(certainty) || segment >> >> So, after that I tried running the biprobit with the survey prefix >> (svy:biprobit) and it seems it does nothing. It just freezes showing >> the message: (running biprobit on estimation sample). >> >> I know that the the biprobit works quite fine with survey data (svy >> prefix), because I have used it before, but without the issue of >> partial observability (and also with significant fewer independent >> variables), but in this case, it is just not working at all! >> >> Any suggestion? Any help or idea would be greatly appreciated! >> >> Thanks in advance, >> >> Eduardo >> * >> * For searches and help try: >> * http://www.stata.com/help.cgi?search >> * http://www.stata.com/support/statalist/faq >> * http://www.ats.ucla.edu/stat/stata/ >> > > > > -- > Stas Kolenikov, also found at http://stas.kolenikov.name > Small print: I use this email account for mailing lists only. > > * > * For searches and help try: > * http://www.stata.com/help.cgi?search > * http://www.stata.com/support/statalist/faq > * http://www.ats.ucla.edu/stat/stata/ > * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/

**Follow-Ups**:**Re: st: Bivariate probit model with partial observability and survey data***From:*Stas Kolenikov <skolenik@gmail.com>

**References**:**st: Bivariate probit model with partial observability and survey data***From:*Eduardo Andrés Alfonso Sierra <ealfonsosierra@googlemail.com>

**Re: st: Bivariate probit model with partial observability and survey data***From:*Stas Kolenikov <skolenik@gmail.com>

- Prev by Date:
**Re: st: mac editor for big files** - Next by Date:
**Re: st: Bivariate probit model with partial observability and survey data** - Previous by thread:
**Re: st: Bivariate probit model with partial observability and survey data** - Next by thread:
**Re: st: Bivariate probit model with partial observability and survey data** - Index(es):