st: RE: RE: RE: How to loop a <clean> script over multiple files?

[Nick Cox <n.j.cox@durham.ac.uk>]

"All this" being building a macro with a filelist from one or more directories. 

Nick 
n.j.cox@durham.ac.uk 


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: 23 January 2012 18:55
To: 'statalist@hsphsun2.harvard.edu'
Subject: st: RE: RE: How to loop a <clean> script over multiple files?

As already pointed out, -fs- (SSC) does all this for you. 

Nick 
n.j.cox@durham.ac.uk 

Impavido, Gregorio

If the files can be differentiated by a specific element try this for point 1 of your question:

local wkdir "/Users/gregorio/Documents/Stata" // or whatever is your working directory
local filelist : dir . file "`wkdir'/eu??.txt"  // builds macro with all relevant files of type "eu01.txt"

you can then run something like 
 
foreach file of local filelist {
	insheet using etc.....
}

-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Brandon Olszewski
Sent: Friday, January 20, 2012 7:38 PM
To: statalist@hsphsun2.harvard.edu
Subject: st: How to loop a <clean> script over multiple files?

Hi Statalist:
I want to clean about 24 data files (all with the same variables)
elegantly, without pasting the clean code for each one separately. I'm
starting with .txt files organized by month, will clean them, save
them as .dta files in a temp directory (mkdir  ${datpath}temp), merge
them into a wide file, then reshape them into a long file. To
simplify, let's say the cleaning code is this:
>>>
insheet using ${datpath}data_file_1.txt
rename var1 id
format id %9.0g
renvars _all, postfix(1)
rename id id
save ${datpath}temp/data_file_1.dta
>>>

I want to run that code on 24 files (representing 2 years' worth of
monthly data), creating files that have a constant id at each time,
but variables that are marked by time (hence, the renvars/postfix).
Obviously, I know how to set up a global macro for my data path, but I
don't know enough to apply that to my problem here. It seems like I
need to:
1. Write a macro that identifies the 24 files (maybe with
"${datpath}*.txt" ?) so that the .do file knows what files to insheet
and clean;
2. Write a macro that tells Stata to add the correct <postfix> to each
file - I imagine that's done with something like:
local i = 1
renvars _all, postfix(i+1)
...but I know that won't work, because it will tag all the variables
with a "2" at the end or something similar.

So, I'm stumped. I've looked on the list, for instance at
http://www.stata.com/statalist/archive/2011-12/msg00055.html and at
the <Programming Stata> FAQ page at
http://www.stata.com/support/faqs/lang/, but I don't even know where
to begin. I write <foreach> loops all the time, but I don't understand
the mechanics well enough to apply those ideas to this problem. Thanks
for any help, in advance.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/