Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: how can i make my loop run faster?


From   Austin Nichols <[email protected]>
To   [email protected]
Subject   Re: st: how can i make my loop run faster?
Date   Tue, 20 Sep 2011 09:49:57 -0400

Stefano Rossi <[email protected]>:
You can sort your data and use -in- instead of -if- to reduce
processing time considerably.  You can generate regression coefs and
predictions without ever firing up -regress-, probably using lag
operators or a wide date structure, which can also speed things up.

See e.g.
http://www.stata.com/statalist/archive/2008-10/msg00136.html
and for some related material:
http://www.stata.com/statalist/archive/2011-01/msg00662.html
http://www.stata.com/statalist/archive/2008-04/msg00967.html

On Tue, Sep 20, 2011 at 1:24 AM, Partho Sarkar
<[email protected]> wrote:
> Hello Stefanlo
>
> I am not quite sure I understand what you are proposing, but I suppose
> this ought to be do-able,  though I can't offhand think of any direct
> way to do this.  I think this would also be an extremely cumbersome
> way to do this, and almost certainly  even slower than any of the
> others we have talked about.  However, I had thought of another
> possible approach:
>
> Since you have a "short" panel- "only" 200 periods, but many more
> firms, I would think dividing up your sample period into, say, 10
> (non-overlapping) subsets, and doing a statsby regression for each
> would give you the results for all (or selected subset of) firms
> within each time sub-period.  You would still have to
> combine(merge/append) the results.
>
> But why not try the rolling loop as suggested in the thread I cited,
> first splitting the firms into manageable subsets (which I think you
> would have to do in any case, if you want to run this routine- unless
> you have access to a super-computer!)  ?
>
> By the way, I am curious to know what exactly the source of your data is!
>
> Hope this helps
>
> Partho
>
>
> On Tue, Sep 20, 2011 at 10:20 AM, Stefano Rossi <[email protected]> wrote:
>> Dear Partho,
>>
>> many thanks for this, which is very useful.  I can see how "rolling" works, and I can see how it can generate efficiency gains, but I agree the whole procedure may still be quite slow and require splitting the sample into subsets to get a faster procedure in some way.
>>
>> I am currently considering a different path, namely generating a cross-section of observations by firm-period, whereby each firm-period unit contains 12 observations, from -1 to -12 (I would also have a separate data by +1 to +12).  This procedure would effectively produce a dataset 12 times larger than my current one.  This procedure would get around the "rolling" issue, and would allow me to use the "statsby" (or equivalent) command without worrying of the length of the estimation sample, with potentially large efficiency improvements (i.e., no "ifs").
>>
>> Provided my intuition is correct, my one concern here is how to create such dataset, which is 12 times bigger than the current one.  Is there a built-in Stata command that allows to do this efficiently?
>>
>> Many thanks for your support.
>>
>> Kind regards,
>>
>> Stefano
>>
>>
>> ________________________________________
>> From: [email protected] [[email protected]] On Behalf Of Partho Sarkar [[email protected]]
>> Sent: Tuesday, September 20, 2011 12:33 AM
>> To: [email protected]
>> Subject: Re: st: how can i make my loop run faster?
>>
>> I guess Stefano might have solved his problem by now, but just to
>> complete this, here is a post by  Brian R. Landy from an older thread
>> which gives the complete code for -rolling-, including merging the
>> results files.
>>
>> http://www.stata.com/statalist/archive/2009-09/msg01239.html
>>
>> The thread also points out the speed problems with rolling for panel data.
>>
>> P.Sarkar

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index