Notice: On March 31, it was **announced** that Statalist is moving from an email list to a **forum**. The old list will shut down on April 23, and its replacement, **statalist.org** is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From |
Michael Stewart <michaelstewartresearch@gmail.com> |

To |
statalist <statalist@hsphsun2.harvard.edu> |

Subject |
Re: st: Need help with working with multiple files |

Date |
Tue, 28 May 2013 09:45:19 -0400 |

Thanks a lot ,Nick.The entire loop works like a charm.I think, I will take courses on stata programming . Thanks again Sincerely Mike On Tue, May 28, 2013 at 9:30 AM, Nick Cox <njcoxstata@gmail.com> wrote: > has a inner loop using the same macro name for loop element as the > outer loop. > > The villain in the piece is > > foreach x of varlist price mpg rep78 trunk{ > > which has the side-effect of zapping local macro x once it has finished. > > The rule is that a local macro defined by a loop is undefined once > that loop finishes. It is, doubly, local. > > Thus `x' is defined <here> but not <there> in something like > > foreach x of local X { > > * here `x' is OK > > foreach x of local Y { > > } > > * there `x' has been undefined > > } > > Remedy: Don't do that. Use a different name, e.g. > > foreach y of local Y > > Nick > njcoxstata@gmail.com > > > On 28 May 2013 14:03, Michael Stewart <michaelstewartresearch@gmail.com> wrote: >> Dear Statalist user , >> I am having problem with saving files after damamanagement tasks using >> a locals in foreach loop and facing difficulty.(As I try to work with >> three datasets, the output is only one dataset).I have tried many >> options, sought help of statlist but able to correct the problem >> partially but there is a bug somewhere. I am here with posting the >> code and stata output with "traceon".May be any one could please help >> me. >> >> MY CODE / DO FILE IS AS FOLLOWS >> local ds1 "D:\_Dummy\multiple\1\auto_1.dta" >> local ds2 "D:\_Dummy\multiple\2\auto_2.dta" >> local ds3 "D:\_Dummy\multiple\3\auto_3.dta" >> local all ds1 ds2 ds3 >> foreach x in `all'{ >> use ``x'',clear >> di "``x''" >> include "D:\_Dummy\multiple\4\mpg.do" >> di "``x''" >> include "D:\_Dummy\multiple\5\sum.do" >> save ``x''_new.dta,replace >> } >> >> mpg DO FILE IS AS FOLLOWS ( NO FOREACH LOOP IN THIS DOFILE) >> gen mean=sum(mpg) >> >> sum DOFILE IS AS FOLLOWS (FOREACH LOOP PRESENT IN THIS DOFILE) >> foreach x of varlist price mpg rep78 trunk{ >> gen x_`x'=`x'==3 >> } >> >> >> PROBLEM: instead of generating three new datasets , it is generating >> one dataset _new. >> >> MY STATA OUTPUT IS AS FOLLOWS(WITH TRACE ON AND DEBUGGNG STEP di >> ``x''); LOOKS LIKE AFTER EXECUTUNG THE SECOND DOFILE (WITH FOREACH >> LOOP) di ``x'' shows nothing!!!!! >> >> **************************************************************************************************************************** >> local ds1 "D:\_Dummy\multiple\1\auto_1.dta" >> >> . local ds2 "D:\_Dummy\multiple\2\auto_2.dta" >> >> . local ds3 "D:\_Dummy\multiple\3\auto_3.dta" >> >> . local all ds1 ds2 ds3 >> >> . foreach x in `all'{ >> 2. use ``x'',clear >> 3. di "``x''" >> 4. include "D:\_Dummy\multiple\4\mpg.do" >> 5. di "``x''" >> 6. include "D:\_Dummy\multiple\5\sum.do" >> 7. save ``x''_new.dta,replace >> 8. } >> - foreach x in `all'{ >> = foreach x in ds1 ds2 ds3{ >> - use ``x'',clear >> = use D:\_Dummy\multiple\1\auto_1.dta,clear >> (1978 Automobile Data) >> - di "``x''" >> = di "D:\_Dummy\multiple\1\auto_1.dta" >> D:\_Dummy\multiple\1\auto_1.dta >> - include "D:\_Dummy\multiple\4\mpg.do" >> >> . gen mean=sum(mpg) >> >> . >> . - di "``x''" >> = di "D:\_Dummy\multiple\1\auto_1.dta" >> D:\_Dummy\multiple\1\auto_1.dta >> - include "D:\_Dummy\multiple\5\sum.do" >> >> . gen xx= sum(weight) >> >> . foreach x of varlist price mpg rep78 trunk{ >> 2. gen x_`x'=`x'==3 >> 3. } >> - foreach x of varlist price mpg rep78 trunk{ >> - gen x_`x'=`x'==3 >> = gen x_price=price==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_mpg=mpg==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_rep78=rep78==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_trunk=trunk==3 >> - } >> >> . >> . - save ``x''_new.dta,replace >> = save _new.dta,replace >> file _new.dta saved >> - } >> - use ``x'',clear >> = use D:\_Dummy\multiple\2\auto_2.dta,clear >> (1978 Automobile Data) >> - di "``x''" >> = di "D:\_Dummy\multiple\2\auto_2.dta" >> D:\_Dummy\multiple\2\auto_2.dta >> - include "D:\_Dummy\multiple\4\mpg.do" >> >> . gen mean=sum(mpg) >> >> . >> . - di "``x''" >> = di "D:\_Dummy\multiple\2\auto_2.dta" >> D:\_Dummy\multiple\2\auto_2.dta >> - include "D:\_Dummy\multiple\5\sum.do" >> >> . gen xx= sum(weight) >> >> . foreach x of varlist price mpg rep78 trunk{ >> 2. gen x_`x'=`x'==3 >> 3. } >> - foreach x of varlist price mpg rep78 trunk{ >> - gen x_`x'=`x'==3 >> = gen x_price=price==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_mpg=mpg==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_rep78=rep78==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_trunk=trunk==3 >> - } >> >> . >> . - save ``x''_new.dta,replace >> = save _new.dta,replace >> file _new.dta saved >> - } >> - use ``x'',clear >> = use D:\_Dummy\multiple\3\auto_3.dta,clear >> (1978 Automobile Data) >> - di "``x''" >> = di "D:\_Dummy\multiple\3\auto_3.dta" >> D:\_Dummy\multiple\3\auto_3.dta >> - include "D:\_Dummy\multiple\4\mpg.do" >> >> . gen mean=sum(mpg) >> >> . >> . - di "``x''" >> = di "D:\_Dummy\multiple\3\auto_3.dta" >> D:\_Dummy\multiple\3\auto_3.dta >> - include "D:\_Dummy\multiple\5\sum.do" >> >> . gen xx= sum(weight) >> >> . foreach x of varlist price mpg rep78 trunk{ >> 2. gen x_`x'=`x'==3 >> 3. } >> - foreach x of varlist price mpg rep78 trunk{ >> - gen x_`x'=`x'==3 >> = gen x_price=price==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_mpg=mpg==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_rep78=rep78==3 >> - } >> - gen x_`x'=`x'==3 >> = gen x_trunk=trunk==3 >> - } >> >> . >> . - save ``x''_new.dta,replace >> = save _new.dta,replace >> file _new.dta saved >> - } >> >> . >> end of do-file >> **************************************************************************************************************************** >> >> >> So as you have noticed a, after executing dofile "sum", the naleof >> the file is being removed and di "" shows nothing. >> >> Can anyone of the users help fix the bug. >> >> Thank you >> Mike >> >> >> >> >> On Tue, May 28, 2013 at 7:37 AM, Nick Cox <njcoxstata@gmail.com> wrote: >>> It is best to think that you are addressing Statalist rather than >>> think that one person is necessarily going to answer. Assume that >>> people at the other end are just as capricious as you might be and get >>> bored, need to meet partners, attend to the day job, or write their >>> novel instead. On the whole the list is quite good at picking up >>> answerable questions but we don't have a buddy system. >>> >>> From your example it seems likeliest that one of the extra loops you >>> are including is somehow zapping the local macro x. I don't have >>> another guess. Inner loops are not in principle fatal to outer loops >>> -- otherwise programming would hardly be possible -- but it seems that >>> your code is stomping on itself. >>> >>> Your code is longer and more complicated than you care to show and you >>> can't win easily here. Post too much code or too little; either can be >>> wrong in terms of not getting a useful reply. >>> >>> But I would check that you are not using the same macro name for >>> different loops. >>> >>> Also, you can add debugging steps >>> >>> di "`mymacro'" >>> >>> to get Stata to show what it thinks -mymacro- is (or is not). >>> >>> Example where nested loops are fine. >>> >>> . foreach x in a b c { >>> 2. di "first time `x'" >>> 3. foreach y in 1 2 3 { >>> 4. di "`y'" >>> 5. } >>> 6. di "second time `x'" >>> 7. } >>> first time a >>> 1 >>> 2 >>> 3 >>> second time a >>> first time b >>> 1 >>> 2 >>> 3 >>> second time b >>> first time c >>> 1 >>> 2 >>> 3 >>> second time c >>> Nick >>> njcoxstata@gmail.com >>> >>> >>> On 28 May 2013 11:24, Michael Stewart <michaelstewartresearch@gmail.com> wrote: >>>> HI Nick >>>> Sorry to keep bothering you as I wanted to clarify the last few >>>> sentences of your reply "save the file within >>>> the loop _or_ otherwise use the name you want directly." >>>> >>>> Here in my situation, the code would be >>>> >>>> clear >>>> local ds1 c\data\top/dataset >>>> local ds2 c\data\date/news/dataset >>>> ... >>>> ... >>>> .. >>>> local ds13 c\data\data/datedataset- >>>> local all ds1 ....ds13 >>>> foreach x in `all'{ >>>> use ``x'',clear >>>> datamagement tasks / contains foreaach loop >>>> datamagement tasks / contains foreaach loop >>>> datamagement tasks / contains foreaach loop >>>> >>>> save ????? >>>> } >>>> >>>> Here while saving the dataset , when I use save ``x''_new,replace--> >>>> this saves the dataset as _new and we get one dataset instead of the >>>> 13 dataset that we started with >>>> >>>> According to your recommendation "otherwise use the name you want >>>> directly", how can I fit 13 names here in the loop . >>>> Am I missing some important concept here?? Please help me with the >>>> problem and understand the concept. >>>> >>>> Thanks >>>> MIke >>>> >>>> >>>> On Tue, May 28, 2013 at 3:40 AM, Nick Cox <njcoxstata@gmail.com> wrote: >>>>> If you refer to a macro that doesn't exist Stata doesn't regard that >>>>> as an error: it just substitutes nothing, or equivalently ignores the >>>>> reference. >>>>> >>>>> Nested macro references don't affect this principle and indeed must be >>>>> consistent with it. So if local macro x does not exist as a string to >>>>> be used neither do any of >>>>> >>>>> `x' >>>>> ``x'' >>>>> ```x'''' >>>>> >>>>> and so forth. You could say that they define empty strings if you >>>>> like, but the effect is the same: a non-existent string and an empty >>>>> string look identical. >>>>> >>>>> From what you say that is precisely what you did. It was the -foreach- >>>>> loop that defined x as what the loop used -- and (an extra principle >>>>> here) that macro is not defined outside the loop, and indeed is doubly >>>>> local, to the loop as well as in the usual sense. >>>>> >>>>> The user manual [U] contains a basic introduction to macros in >>>>> >>>>> [U] Chapter 18 . . . . . . . . . . . . . . . . . . . . Programming Stata >>>>> (help program, macro, quotes, syntax, examplehelpfile) >>>>> >>>>> From your earlier posts it seems that you need to save the file within >>>>> the loop _or_ otherwise use the name you want directly. >>>>> >>>>> Nick >>>>> njcoxstata@gmail.com >>>>> >>>>> On 28 May 2013 06:00, Michael Stewart <michaelstewartresearch@gmail.com> wrote: >>>>>> Dear Steve and Nick , >>>>>> I am having problem with saving the files >>>>>> I am using ``x'' and the dofile runs through multiple dofiles like a charm >>>>>> However, after completetion of the datamanagement tasks, the each >>>>>> file is being stored as "_new" in the stata working directory. >>>>>> And the interesting part is that this happens only if the individual >>>>>> tasks have foreach loop in it. If the individual tasks dont have >>>>>> foreach loop in it, then the files are saved in respective folders >>>>>> i.e., save ``x''_new, replace --> works >>>>>> I am not sure is there is a way to circumvent this problem >>>>>> Thanks >>>>>> Mike >>>>>> >>>>>> >>>>>> On Mon, May 27, 2013 at 10:46 PM, Michael Stewart >>>>>> <michaelstewartresearch@gmail.com> wrote: >>>>>>> HI , >>>>>>> First of all, thank you very much Steve and Nick.It works like a charm >>>>>>> and I can automate stata to replicate many tasks >>>>>>> I work on Windows .Using ``x'' works.I havent tested `x'.But it >>>>>>> Works.Thats great . >>>>>>> Thank you Steve and Nick >>>>>>> Thank you , >>>>>>> Yours Sincerely, >>>>>>> Mike. >>>>>>> >>>>>>> >>>>>>> On Mon, May 27, 2013 at 9:46 PM, Steve Samuels <sjsamuels@gmail.com> wrote: >>>>>>>> >>>>>>>> Well, I was concerned about that, but the code worked for me in >>>>>>>> Mac OS X, Here was my test do file: >>>>>>>> >>>>>>>> >>>>>>>> ********************************** >>>>>>>> local ds1 "~Downloads/testsim01" >>>>>>>> local ds2 "~Downloads/dtest01" >>>>>>>> >>>>>>>> local all ds1 ds2 >>>>>>>> di `"`all'"' >>>>>>>> foreach x in `all' { >>>>>>>> use `x', clear >>>>>>>> gen nv =1 >>>>>>>> save `x'_new, replace >>>>>>>> } >>>>>>>> ******************************** >>>>>>>> >>>>>>>> Steve >>>>>>>> >>>>>>>> On May 27, 2013, at 8:50 PM, Nick Cox wrote: >>>>>>>> >>>>>>>> This won't work, as when Stata hits the loop >>>>>>>> >>>>>>>> foreach x in `all' >>>>>>>> >>>>>>>> it will then see >>>>>>>> >>>>>>>> foreach x in ds1 ds2 >>>>>>>> >>>>>>>> and will fail at >>>>>>>> >>>>>>>> use ds1, clear >>>>>>>> >>>>>>>> There are two levels of macro referencing in Steve's code, but only >>>>>>>> one level going backwards. >>>>>>>> >>>>>>>> use ``x'', clear >>>>>>>> >>>>>>>> is one way to fix it. >>>>>>>> >>>>>>>> Nick >>>>>>>> njcoxstata@gmail.com >>>>>>>> >>>>>>>> >>>>>>>> On 28 May 2013 01:23, Steve Samuels <sjsamuels@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Michael Steward (michaelstewartresearch@gmail.com): >>>>>>>>> >>>>>>>>> You can loop, as Nick suggests. Put the data management tasks in >>>>>>>>> external do files, e.g. tasks1.do, tasks2.do, tasks3.do. Then something >>>>>>>>> like the following should work (not tested in Windows). See the help for >>>>>>>>> -include-. The big problem I anticipate is the mounds of output that >>>>>>>>> this could generate. >>>>>>>>> >>>>>>>>> >>>>>>>>> ********************************* >>>>>>>>> local ds1 "c/data/dataset-1" >>>>>>>>> local ds2 "c/data/newdata/dataset-2" // etc. >>>>>>>>> >>>>>>>>> local all ds1 ds2 >>>>>>>>> >>>>>>>>> foreach x in `all' { >>>>>>>>> use `x', clear >>>>>>>>> >>>>>>>>> include tasks1 >>>>>>>>> include tasks2 >>>>>>>>> include tasks3 >>>>>>>>> >>>>>>>>> save `x'_new, replace >>>>>>>>> } >>>>>>>>> ********************************* >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> >>>>>>>>> On May 27, 2013, at 3:19 PM, Michael Stewart wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> I am trying to find a way to work multiple files successively. >>>>>>>>> I have multiple large files(really large) placed in different directories >>>>>>>>> I am trying to find a way to load each file into stata, complete bunch >>>>>>>>> of same datamagement tasks,create a new file from it and places them >>>>>>>>> in a common folder >>>>>>>>> >>>>>>>>> I could use a dofile and do something like >>>>>>>>> >>>>>>>>> clear >>>>>>>>> use c\data\dataset-1 >>>>>>>>> datamagement tasks >>>>>>>>> datamagement tasks >>>>>>>>> datamagement tasks >>>>>>>>> save c\data\dataset-1_new >>>>>>>>> clear >>>>>>>>> use c\data\newdata\dataset-2 >>>>>>>>> datamagement tasks >>>>>>>>> datamagement tasks >>>>>>>>> datamagement tasks >>>>>>>>> save c\data\newdata\dataset-2_new >>>>>>>>> clear >>>>>>>>> >>>>>>>>> But is there an elegant and simpler way to do it given that I have >>>>>>>>> nearly 15 files to do data management tasks. >>>>>>>>> >>>>>>>>> Can we use foreach loop / forvalues loop or anything like that .Please suggest >>>>> * >>>>> * For searches and help try: >>>>> * http://www.stata.com/help.cgi?search >>>>> * http://www.stata.com/support/faqs/resources/statalist-faq/ >>>>> * http://www.ats.ucla.edu/stat/stata/ >>>> >>>> >>>> >>>> -- >>>> Thank you , >>>> Yours Sincerely, >>>> Mike. >>>> * >>>> * For searches and help try: >>>> * http://www.stata.com/help.cgi?search >>>> * http://www.stata.com/support/faqs/resources/statalist-faq/ >>>> * http://www.ats.ucla.edu/stat/stata/ >>> * >>> * For searches and help try: >>> * http://www.stata.com/help.cgi?search >>> * http://www.stata.com/support/faqs/resources/statalist-faq/ >>> * http://www.ats.ucla.edu/stat/stata/ >> >> >> >> -- >> Thank you , >> Yours Sincerely, >> Mike. >> * >> * For searches and help try: >> * http://www.stata.com/help.cgi?search >> * http://www.stata.com/support/faqs/resources/statalist-faq/ >> * http://www.ats.ucla.edu/stat/stata/ > * > * For searches and help try: > * http://www.stata.com/help.cgi?search > * http://www.stata.com/support/faqs/resources/statalist-faq/ > * http://www.ats.ucla.edu/stat/stata/ -- Thank you , Yours Sincerely, Mike. * * For searches and help try: * http://www.stata.com/help.cgi?search * http://www.stata.com/support/faqs/resources/statalist-faq/ * http://www.ats.ucla.edu/stat/stata/

**References**:**st: Need help with working with multiple files***From:*Michael Stewart <michaelstewartresearch@gmail.com>

**Re: st: Need help with working with multiple files***From:*Steve Samuels <sjsamuels@gmail.com>

**Re: st: Need help with working with multiple files***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: Need help with working with multiple files***From:*Steve Samuels <sjsamuels@gmail.com>

**Re: st: Need help with working with multiple files***From:*Michael Stewart <michaelstewartresearch@gmail.com>

**Re: st: Need help with working with multiple files***From:*Michael Stewart <michaelstewartresearch@gmail.com>

**Re: st: Need help with working with multiple files***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: Need help with working with multiple files***From:*Michael Stewart <michaelstewartresearch@gmail.com>

**Re: st: Need help with working with multiple files***From:*Nick Cox <njcoxstata@gmail.com>

**Re: st: Need help with working with multiple files***From:*Michael Stewart <michaelstewartresearch@gmail.com>

**Re: st: Need help with working with multiple files***From:*Nick Cox <njcoxstata@gmail.com>

- Prev by Date:
**Re: st: ldecomp question** - Next by Date:
**Re: st: Need help with working with multiple files** - Previous by thread:
**Re: st: Need help with working with multiple files** - Next by thread:
**Re: st: Need help with working with multiple files** - Index(es):