Re: st: Improving processing speed

[Nick Cox <njcoxstata@gmail.com>]

One detail is that you have lines in couples like this

keep if sizef==`i'
nl gom3: P_WOR ageyear if sizef==`i'

The second -if sizeof==`i'- is redundant given the first. But that may
not help much.

If this is still running, you probably just have to be patient. But
the key question is what is the slowest part? I'd guess wildly it's
one or more of your -nl- calls.

Nick

On Tue, May 29, 2012 at 7:14 AM, Sebastian Galarza
<sebastian@theicct.org> wrote:
> Hi all,
>
> I have the following code and it is taking a very long time to loop through the dataset. In summary, I am running different regressions and retrieving a matrix with the beta values and r-squared for each regression. I am far from being an expert user and would like to know if there are any improvements that I can make to increase the processing speed as it is took over 20 hours to get through the first loop the second loop is still running after about the same amount of time. As a side note I originally had forvalues i=1/17,000 but given the processing time I decided to use a sample to obtain preliminary results.
>
> Any help would be greatly appreciated,
>
> clear all
>
> *START - Exponential Regression
> forvalues i=1/376 {
> use "/Users/Sebastian/Desktop/Matrix/Pool_Sizef_Sample.dta", clear
> keep if sizef==`i'
> nl (P_WOR = {b0=0.1}*(1 * exp({b1=0.05}* (ageyear)))) if sizef==`i'
> matrix beta = e(b)
> matrix r2=e(r2)
> matrix list beta
> matrix list r2
> matrix D = beta, r2
> svmat D, names(D)
> keep fieldnumber poolnumber sizef D1 D2 D3
> keep if D1!=.
> duplicates drop
> save "/Users/Sebastian/Desktop/Matrix/Sample/exponential/pool`i'.dta", replace
> clear matrix
> clear
> program drop _all
> }
>
> use "/Users/Sebastian/Desktop/Matrix/Sample/exponential/pool1.dta", clear
> forvalues i=2/376 {
> append using "/Users/Sebastian/Desktop/Matrix/Sample/exponential/pool`i'.dta"
> sort sizef
> save "/Users/Sebastian/Desktop/Matrix/Sample/exponential/exponential.dta", replace
> }
>
> clear
> program drop _all
>
> *END - Exponential Regression
>
> clear all
> *START - Logistical Function
> forvalues i=1/376 {
> use "/Users/Sebastian/Desktop/Matrix/Pool_Sizef_Sample.dta", clear
> keep if sizef==`i'
> nl log3: P_WOR ageyear if sizef==`i'
> matrix beta = e(b)
> matrix r2=e(r2)
> matrix list beta
> matrix list r2
> matrix D = beta, r2
> svmat D, names(D)
> keep fieldnumber poolnumber sizef D1 D2 D3 D4
> keep if D1!=.
> duplicates drop
> save "/Users/Sebastian/Desktop/Matrix/Sample/logistical/pool`i'.dta", replace
> clear matrix
> clear
> program drop _all
> }
>
> use "/Users/Sebastian/Desktop/Matrix/Sample/logistical/pool1.dta", clear
> forvalues i=2/376 {
> append using "/Users/Sebastian/Desktop/Matrix/Sample/logistical/pool`i'.dta"
> sort sizef
> save "/Users/Sebastian/Desktop/Matrix/Sample/logistical/logistical.dta", replace
> }
> clear
> program drop _all
> * END - Logistical Function
>
>
> *START - Gompertz Function
> forvalues i=1/376 {
> use "/Users/Sebastian/Desktop/Matrix/Pool_Sizef_Sample.dta", clear
> keep if sizef==`i'
> nl gom3: P_WOR ageyear if sizef==`i'
> matrix beta = e(b)
> matrix r2=e(r2)
> matrix list beta
> matrix list r2
> matrix D = beta, r2
> svmat D, names(D)
> keep fieldnumber poolnumber sizef D1 D2 D3 D4
> keep if D1!=.
> duplicates drop
> save "/Users/Sebastian/Desktop/Matrix/Sample/gompertz/pool`i'.dta", replace
> clear
> clear matrix
> program drop _all
> }
>
> use "/Users/Sebastian/Desktop/Matrix/Sample/gompertz/pool1.dta", clear
> forvalues i=2/376 {
> append using "/Users/Sebastian/Desktop/Matrix/Sample/gompertz/pool`i'.dta"
> sort sizef
> save "/Users/Sebastian/Desktop/Matrix/Sample/gompertz/gompertz.dta", replace
> }
> clear
> program drop _all
> * END - Gompertz Function
>
> clear
> program drop _all
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/