Re: st: Calculate "running" standard deviation

[Austin Nichols <austinnichols@gmail.com>]

Christopher <cpoliquin@hbs.edu> :
No need to generate variables inside your loop.
-in- is usually faster than -if- but you might want to exploit -tsset-
if you are worried about gaps in data.
Also, it's a bad idea to name a program -test- since that is a useful
official Stata command.
Try this slight modification (further improvements are possible):

clear all
input str3 country year value
"GBR" 2001 10
"GBR" 2002 20
"GBR" 2003 25
"USA" 2001 30
"USA" 2002 10
"USA" 2003 15
end
g sdev=.
program putsd, byable(recall)
 syntax varlist(max=1) [, sdv(varlist min=1 max=1)]
 local first = _byn1()
 local last = _byn2()
 forv x=`first'/`last' {
  qui su `varlist' in `first'/`x'
  replace `sdv' = r(sd) in `x'
  }
end
bys country (year): putsd value, sdv(sdev)
list

On Thu, Jan 20, 2011 at 10:01 AM, Poliquin, Christopher
<cpoliquin@hbs.edu> wrote:
> Hi,
>
> I am attempting to calculate a "running" standard deviation on some time series data.  The idea is each row has a standard deviation that is based solely on past observations.  The data looks like...
>
> COUNTRY   YEAR   VALUE   SDEV
> +---------------------------------------------------------------------+
> | GBR     2001    10
> | GBR     2002    20     this one based on GBR-2002 and GBR-2001
> | GBR     2003    25     this one based on all the GBR observations
> +---------------------------------------------------------------------+
> | USA     2001    30     this one based only on this row
> | USA     2002    10     this one based on USA-2001 and USA-2002
> | USA     2003    15
> +---------------------------------------------------------------------+
>
> So I need to fill in the SDEV column, with the stat grouped by country, and based only on years prior to the year in the YEAR column.
>
> I think I have a solution, but I can't believe it's the best way to do this.  Here is my code...
>
> sort COUNTRY YEAR
> gen obs = _n
> program test, byable(recall)
>        local first = _byn1()
>        local last = _byn2()
>        forvalues x=`first'(1)`last' {
>                quietly gen y = VALUE if obs <= `x' & obs >= `first'
>                quietly egen w = sd(y)
>                replace SDEV = w in `x'
>                drop y w
>        }
> end
> by COUNTRY : test
>
>
> Could the experts please give a critique of the above?  Especially considering that I am working with about 80,000 rows, not 6.
>
> Also, there must be better words to describe what I want.  "Running standard deviation" doesn't sound like the best term.
>
>
> Best wishes,
> Chris
> ------------------------
> Research Associate
> Harvard Business School

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/