Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
st: Improving code speed
st: Improving code speed
Wed, 22 May 2013 19:25:02 +0200
Dear statalist users,
I am running into a "loop efficiency problem" in that I have to
construct a variable using many iterations and I am not sure whether I
am being as efficient as possible. Given the number of observations
that I have and with my current code, I have to wait days for my code
to finish running! Here's my problem:
I have a total of 50000 observations and need to construct a variable
Y that will be computed using different subsamples of these
observations. In particular,
Y=Y1 when the subsample contains only the first observation,
Y=Y2 when the subsample contains observations 1 and 2,
Y=Y3 when the subsample contains observations 1, 2 and 3 etc until
The idea is therefore to loop over the sample and define the subsample
which contains observations 1 until k and construct the variable
Y`k'=Yk if id==k and Y`k'=0 if id!=k. Then sum the variables Y`k'
after each loop to end up with the final variable Y.
To further complicate things, the variable Y needs to be the average
of 100 simulations that depend on draws taken from a normal
distribution. Hence I need to do a loop within the initial loop in
order to do the 100 simulations.
My code therefore looks like this:
local reps=100 \\ define the number of simulations
gen epsilon=rnormal() \\ generate the random var for the simulations
gen subs=(id<=`k') \\ Define the subsample to be used
gen Y`k'=0 \\ gen the intermediate Y`k'
gen x`i'=z \\ generate simulated variable
gen x`i'=z + epsilon[`i',1] if id==`k' \\ Add the random part
bysort subs: egen tsum=sum(x`i')
gen Y_`i'=t/tsum if id ==`k' \\ Construct Y for simulation i
replace Y_`i'=0 if id!=`k'
replace Y`k'=Y`k' + Y_`i'
replace Y`k'=0 if id!=`k'
drop Y_`i' t tsum x`i'
replace Y`k'=Y`k'/`reps' // average Y from the 100 simulations
replace Y= Y + Y`k'
drop Y`k' subs
The code runs fine, but I takes a lot of time since it has to
construct 100 variables for each of the 50000 iterations. I have tried
many different possibilities and I can't think of another way of
Any tip or suggestion that would help improve the efficiency of my
code would be greatly appreciated!!!
Many thanks in advance!
* For searches and help try: