[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: RE: How to adjust the content of a local macro?

From   Joachim Landström <>
Subject   Re: st: RE: How to adjust the content of a local macro?
Date   Wed, 09 Dec 2009 23:28:21 +0100

I am amazed with the level of help that you get from here. Let me start off expressing my gratitude for this.

To answer some of Nick's points:

1. The loop consists of repeated -drop-ping of observations not desired, working with the remaining subset and then a -restore- of the original. It is difficult to say in general what is most efficient and what most elegant but for a situation like that below I'd normally just add an extra condition excluding the observations not wanted, rather than repeatedly doing major surgery on the dataset. However, others could equally point out that applying -if- on a very large dataset can be time-consuming.

Answer #1: I have been using ?preserve- & -restore, preserve- in loops because of system limitations. I have a dataset as below

Contains data from us_data_ret.dta
  obs:    17,516,468
 vars:             5                          10 Nov 2009 07:02
 size:   525,494,040 (56.6% of memory free)
              storage  display     value
variable name   type   format      label      variable label
id              int    %8.0g                  ID
dscd            str6   %9s                    DSCD
date            float  %td
year            int    %8.0g
totalReturn     double %10.0g
Sorted by:  id  date

that I join with annual market data as below:

Contains data from C:\Users\Joachim\AppData\Local\Temp\ST_02000005.tmp
obs: 1,461 Total returns for value-weighted ave S/B & H/L BM portf and their diffs. See Fam
 vars:             5                          9 Dec 2009 23:08
 size:        46,752 (99.9% of memory free)
              storage  display     value
variable name   type   format      label      variable label
date            float  %td
SMBtr float %9.0g Small-Minus-Big Total Returns. See Fama & French (1993). HMLtr float %9.0g High-Minus-Low Total Returns. See Fama & French (1993).
riskFreereturn  float  %9.0g                  US T-Bill 3 month, middle rate
marketReturn double %9.0g Frank Russel 3000, Total Return Index
Sorted by:

The first dataset above is in "long format" and has 10826 unique ids (US firms). The date variable ranges from 03Jan1978 to 30dec2008 with weekly stock market data.

Then I wish to do a Fama & French 3F CAPM calculation per id and then, based on the 3F CAPM I wish to calculate the expected annual return for each firm-year when each firm is traded on the stock market. Now I know I may be using too much lingo, but this boils down to an OLS regression per id provided that enough data is available (for me that implies three years of weekly data).

Since I do not have enough computing power/memory (I am using a Windows 32-bit system) to add more variables into the joined dataset I have resorted use the ?preserve- & -restore, preserve- method to drop all but a single id and then do the necessary operations on that subset. Then I save the data into a tempfile. Later (after 6-7 hours), when the loop is done I use ?append- to build the dataset anew (having dropped unnecessary data).

Any suggestions to more elegant methods are greatly appreciated.

2. If only the minimum and maximum are needed from a -summarize- it is best just to use a -meanonly- option. (The name -meanonly- is misleading, as I've had occasion to remark before.)

I will have the ?meanonly- option in mind. Did not know about it. :)

3. Code like

	local `minDate' = r(min)
	<stuff> if <stuff> date >= ``minDate''

looks legal but odd. You are probably using more levels of macros than you need. It's hard to tell because the code isn't completely self-contained (that's not a criticism; it wasn't necessary for your question).

I am indeed confused with the Stata usage of ` ' and have been having problems with this. minDate is a ?tempname- so if I just write `minDate' I don't get the value, just the tempname. Doing it as ``minDate'' provides me with the value. But I am sure there are smarter ways to do this. . .

4. Code in which you loop over the contents of a local macro and change that macro within the loop can be tricky. Watch out!

My computer at the university is running a test this evening with the revised code and I was intuitively also worried to change ?panelVar- inside the loop which it guides. I have done it as Martin proposed and added a ?adjustedPanelVar- that I modify. I need it for a second loop where I append the dataset once the first loop is done.

5. The -if- condition in

	summarize totalReturn if totalReturn != .

is unnecessary as -summarize- always ignores missings.

That is always good to know.

6. To get minimum and maximum dates in a panel, no looping is necessary as

egen mindate = min(date), by(id)
egen maxdate = max(date), by(id)

will do it. Similarly it looks as if your main problem does not need any looping either, as it should yield to -egen- operations. Look at -egen, count()- in particular.

Yes I have used ?min()-elsewhere but I cannot easily add more variables to the complete database due to memory limitations. Memory is a luxury to me. . .

7. More generally, it is not always positive to know too many other languages if they lead you to seek a Stata equivalent of other code when there's a Stataish way to do it without any real programming.

I completely agree. Coming from SAS I am used to ?proc sql- and SAS' -proc iml-. Had I just known about STATA 1998 this would not have been a problem. Now I have to "delearn" the SAS version of sql while learning STATA and that is like a baby learning to walk. I try my best but keep bruising myself. . . :)

I will respond to Martin's second post tomorrow by posting the code and an an excerpt of the data (beware that it looks much like crochet since I have mainly been trying to get the code to run without exiting with errors).


*   For searches and help try:

© Copyright 1996–2017 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index