[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

From |
"Scott Merryman" <smerryman@kc.rr.com> |

To |
<statalist@hsphsun2.harvard.edu> |

Subject |
st: Re: How much can we trust Stata's non-linear solver(s)? |

Date |
Tue, 26 Aug 2003 20:57:11 -0500 |

----- Original Message ----- From: "joachim Wagner" <wagner@uni-lueneburg.de> To: <statalist@hsphsun2.harvard.edu> Sent: Monday, August 25, 2003 2:46 AM Subject: st: How much can we trust Stata's non-linear solver(s)? > Dear Listmembers, > > at the weekend I read a paper by McCullough and Vinod on "Verifying the > solution from a nonlinear solver: A case study" in the June 2003 issue of > the American Economic Review. For those of you who are not in economics, > this is an absolute top-journal in our discipline, and the authors are > considered to be top-experts in the field. I was shocked (really, believe > me!) to learn how much can go wrong and goes wrong when one tries to find > the ML solution of such day-to-day problems as the parameters of a probit > model using a PC and several programs. The authors give examples, and they > argue that "the researcher's job is not done when the program reports > convergence - it is only beginning" (p. 876), and ask us to examine the > gradient, inspect the solution path, evaluate the Hessian, including an > eigensystem analysis, and profile the likelihood. Uff. Do I really have to > learn how to do all this? Do I have to use three or more different packages > to compare the solutions? Shall I demand this when I sit down to write my > next referee report later this week? Must I be aware that the next report I > receive will demand me to do all this? > > Unfortunately, the authors do not name horses and riders (as we say in > Germany - in German, of course), so I have no idea whether Stata is among > the programs one can trust or not. Obviously, I want to know, and I am sure > many of the list-members share my view. Given that the acknowledgement > footnote of the paper mentions W. Gould, I hope he can tell us more. > > From a slightly different perspective, would it be a good idea to > implement the steps suggested by the authors for a "post-convergence" check > in Stata ? > > Given that I am no expert in numerical computing and all the related > difficulties, I would like to learn your opinions, experiences etc. > > Best, > > Joachim > For what it is worth, when McCullough and Vinod write (referring to a paper by Houston Stokes "On the Advantage of Using Two or More Econometric Software Systems to Solve the Same Problem") "Observe that nonexistence was no impediment to five solvers declaring not only that a solution existed, but that the solution had been located. For sake of completeness we note that another two packages did refuse to produce an answer because they recognized that the maximum likelihood estimator did not exist--the developers of these packages had written their programs to examine the data for possible signs of nonexistence of the solution." (McCullough and Vinod, page 873) One of the two statistical packages that recognized that a solution did not exist was Stata, the other was TSP. A preprint copy of Stokes' can be found (in Word format) at http://tigger.uic.edu/~hhstokes/papers.html In the paper, while Stokes finds that Stata recognizes that a solution does not exist, he does find Stata's results 'strange and confusing' -- but I am not sure I follow. He writes: "As a final test of packages, Stata version 7.0 for Linux was tried. Stata reported "note: d2~=0 predicts success perfectly. d2 dropped and 15 obs not used." The Stata results are shown in Table 2, column 5. The reported coefficients and log likelihood ( -8.64038) for this "reduced model" were found to be nearly the same as was found by the other systems for the complete period including D2. What Stata appeared to have done is to just not report the coefficients for D2. The cryptic Stata message night cause some confusion to the average researcher since what Stata reports is not Maddala's model (2), which is a probit model estimated for all 44 observations without D2 on the right hand side. If the Maddala equation (2) is run, all programs replicate the results reported by Maddala." (page 4.) "Sorting the dataset with respect to D1, the dependent variable, reveals that when D1=0, D2 is always 0 but then D1 =1, D2 can be 0 or 1. This means that if we find that D2 =1, we know for certain that D1 = 1. There were 15 such cases. Once the sort was studied, the formerly cryptic message in Stata was understood. All 15 such cases were detected by Stata. What is strange is that if Stata really believed its own message, it would have either reported nothing (the best decision) or reported the subset model in equation (2). Just not reporting the D2 coefficient but leaving all other coefficients in the model is a strange and confusing choice. " (page 6) "What started as a replication of probit results reported in Maddala [16] was extended when different results were obtained with the software systems B34S, SAS, RATS, LIMDEP and Stata. The differences were traced to the effect of alternative convergence tolerances on both the rank of the variance covariance matrix and the point estimate of one variable that was caused by Quasicomplete Separation. Only two software systems, Stata and TSP, were able to detect the underlying problem that for this model and these data, the maximum likelihood estimate does not exist. " (page 12) Scott * * For searches and help try: * http://www.stata.com/support/faqs/res/findit.html * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/

**References**:**st: How much can we trust Stata's non-linear solver(s)?***From:*joachim Wagner <wagner@uni-lueneburg.de>

- Prev by Date:
**Re: Re: st: SP/RM[1,2] EMS** - Next by Date:
**st: RE: out-of-sample prediction of shares** - Previous by thread:
**st: How much can we trust Stata's non-linear solver(s)?** - Next by thread:
**Re: Re: st: SP/RM[1,2] EMS** - Index(es):

© Copyright 1996–2015 StataCorp LP | Terms of use | Privacy | Contact us | What's new | Site index |