Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Need help with working with multiple files


From   Nick Cox <njcoxstata@gmail.com>
To   "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
Subject   Re: st: Need help with working with multiple files
Date   Tue, 28 May 2013 14:30:51 +0100

has a inner loop using the same macro name for loop element as the
outer loop.

The villain in the piece is

foreach x of varlist price mpg rep78 trunk{

which has the side-effect of zapping local macro x once it has finished.

The rule is that a local macro defined by a loop is undefined once
that loop finishes. It is, doubly, local.

Thus `x' is defined <here> but not <there> in something like

foreach x of local X {

* here `x' is OK

     foreach x of local Y {

    }

* there `x' has been undefined

}

Remedy: Don't do that. Use a different name, e.g.

foreach y of local Y

Nick
njcoxstata@gmail.com


On 28 May 2013 14:03, Michael Stewart <michaelstewartresearch@gmail.com> wrote:
> Dear Statalist user ,
> I am having problem with saving files after damamanagement tasks using
> a locals in foreach loop and facing difficulty.(As I try to work with
> three datasets, the output is only one dataset).I have tried many
> options, sought help of statlist but able to correct the problem
> partially but there is a bug somewhere.  I am here with posting the
> code and stata output with "traceon".May be any one could please help
> me.
>
> MY CODE / DO FILE IS AS FOLLOWS
> local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
> local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
> local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
> local all ds1 ds2 ds3
> foreach x in `all'{
> use ``x'',clear
> di "``x''"
> include "D:\_Dummy\multiple\4\mpg.do"
> di "``x''"
> include "D:\_Dummy\multiple\5\sum.do"
> save ``x''_new.dta,replace
> }
>
> mpg DO FILE IS AS  FOLLOWS ( NO FOREACH LOOP IN THIS DOFILE)
> gen mean=sum(mpg)
>
> sum DOFILE IS AS FOLLOWS  (FOREACH LOOP PRESENT  IN THIS DOFILE)
> foreach x of varlist price mpg rep78 trunk{
> gen x_`x'=`x'==3
> }
>
>
> PROBLEM: instead of generating three new  datasets , it is generating
> one dataset _new.
>
> MY STATA OUTPUT IS AS FOLLOWS(WITH TRACE ON AND DEBUGGNG STEP di
> ``x''); LOOKS LIKE AFTER EXECUTUNG THE SECOND DOFILE (WITH FOREACH
> LOOP) di ``x''  shows nothing!!!!!
>
> ****************************************************************************************************************************
>  local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
>
> . local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
>
> . local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
>
> . local all ds1 ds2 ds3
>
> . foreach x in `all'{
>   2. use ``x'',clear
>   3. di "``x''"
>   4. include "D:\_Dummy\multiple\4\mpg.do"
>   5. di "``x''"
>   6. include "D:\_Dummy\multiple\5\sum.do"
>   7. save ``x''_new.dta,replace
>   8. }
> - foreach x in `all'{
> = foreach x in ds1 ds2 ds3{
> - use ``x'',clear
> = use D:\_Dummy\multiple\1\auto_1.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\1\auto_1.dta"
> D:\_Dummy\multiple\1\auto_1.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\1\auto_1.dta"
> D:\_Dummy\multiple\1\auto_1.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
>   2. gen x_`x'=`x'==3
>   3. }
>   - foreach x of varlist price mpg rep78 trunk{
>   - gen x_`x'=`x'==3
>   = gen x_price=price==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_mpg=mpg==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_rep78=rep78==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_trunk=trunk==3
>   - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
> - use ``x'',clear
> = use D:\_Dummy\multiple\2\auto_2.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\2\auto_2.dta"
> D:\_Dummy\multiple\2\auto_2.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\2\auto_2.dta"
> D:\_Dummy\multiple\2\auto_2.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
>   2. gen x_`x'=`x'==3
>   3. }
>   - foreach x of varlist price mpg rep78 trunk{
>   - gen x_`x'=`x'==3
>   = gen x_price=price==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_mpg=mpg==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_rep78=rep78==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_trunk=trunk==3
>   - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
> - use ``x'',clear
> = use D:\_Dummy\multiple\3\auto_3.dta,clear
> (1978 Automobile Data)
> - di "``x''"
> = di "D:\_Dummy\multiple\3\auto_3.dta"
> D:\_Dummy\multiple\3\auto_3.dta
> - include "D:\_Dummy\multiple\4\mpg.do"
>
> . gen mean=sum(mpg)
>
> .
> . - di "``x''"
> = di "D:\_Dummy\multiple\3\auto_3.dta"
> D:\_Dummy\multiple\3\auto_3.dta
> - include "D:\_Dummy\multiple\5\sum.do"
>
> . gen xx= sum(weight)
>
> . foreach x of varlist price mpg rep78 trunk{
>   2. gen x_`x'=`x'==3
>   3. }
>   - foreach x of varlist price mpg rep78 trunk{
>   - gen x_`x'=`x'==3
>   = gen x_price=price==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_mpg=mpg==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_rep78=rep78==3
>   - }
>   - gen x_`x'=`x'==3
>   = gen x_trunk=trunk==3
>   - }
>
> .
> . - save ``x''_new.dta,replace
> = save _new.dta,replace
> file _new.dta saved
> - }
>
> .
> end of do-file
> ****************************************************************************************************************************
>
>
> So as you have noticed a, after executing dofile  "sum", the naleof
> the file is being removed and di "" shows nothing.
>
> Can anyone of the users help fix the bug.
>
> Thank you
> Mike
>
>
>
>
> On Tue, May 28, 2013 at 7:37 AM, Nick Cox <njcoxstata@gmail.com> wrote:
>> It is best to think that you are addressing Statalist rather than
>> think that one person is necessarily going to answer. Assume that
>> people at the other end are just as capricious as you might be and get
>> bored, need to meet partners, attend to the day job, or write their
>> novel instead. On the whole the list is quite good at picking up
>> answerable questions but we don't have a buddy system.
>>
>> From your example it seems likeliest  that one of the extra  loops you
>> are including is somehow zapping the local macro x. I don't have
>> another guess. Inner loops are not in principle fatal to outer loops
>> -- otherwise programming would hardly be possible -- but it seems that
>> your code is stomping on itself.
>>
>> Your code is longer and more complicated than you care to show and you
>> can't win easily here. Post too much code or too little; either can be
>> wrong in terms of not getting a useful reply.
>>
>> But I would check that you are not using the same macro name for
>> different loops.
>>
>> Also, you can add debugging steps
>>
>> di "`mymacro'"
>>
>> to get Stata to show what it thinks -mymacro- is (or is not).
>>
>> Example where nested loops are fine.
>>
>> . foreach x in a b c {
>>   2.     di "first time   `x'"
>>   3.         foreach y in 1 2 3 {
>>   4.                 di "`y'"
>>   5.         }
>>   6.         di "second time  `x'"
>>   7. }
>> first time   a
>> 1
>> 2
>> 3
>> second time  a
>> first time   b
>> 1
>> 2
>> 3
>> second time  b
>> first time   c
>> 1
>> 2
>> 3
>> second time  c
>> Nick
>> njcoxstata@gmail.com
>>
>>
>> On 28 May 2013 11:24, Michael Stewart <michaelstewartresearch@gmail.com> wrote:
>>> HI Nick
>>> Sorry to keep bothering you as I wanted to clarify the last few
>>> sentences of your reply "save the file within
>>> the loop _or_ otherwise use the name you want directly."
>>>
>>> Here in my situation, the code would be
>>>
>>> clear
>>> local ds1 c\data\top/dataset
>>> local ds2  c\data\date/news/dataset
>>> ...
>>> ...
>>> ..
>>> local ds13  c\data\data/datedataset-
>>> local all ds1 ....ds13
>>> foreach x in `all'{
>>> use ``x'',clear
>>> datamagement tasks   / contains foreaach loop
>>> datamagement tasks  / contains foreaach loop
>>> datamagement tasks  / contains foreaach loop
>>>
>>> save ?????
>>> }
>>>
>>> Here while saving the dataset , when I use  save ``x''_new,replace-->
>>> this saves the dataset as _new and we get one dataset instead of the
>>> 13 dataset that we started with
>>>
>>> According to your recommendation "otherwise use the name you want
>>> directly", how can I fit 13 names here in the loop .
>>> Am I missing some  important concept here?? Please help me with  the
>>> problem and understand the concept.
>>>
>>> Thanks
>>> MIke
>>>
>>>
>>> On Tue, May 28, 2013 at 3:40 AM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>> If you refer to a macro that doesn't exist Stata doesn't regard that
>>>> as an error: it just substitutes nothing, or equivalently ignores the
>>>> reference.
>>>>
>>>> Nested macro references don't affect this principle and indeed must be
>>>> consistent with it. So if  local macro x does not exist as a string to
>>>> be used neither do any of
>>>>
>>>> `x'
>>>> ``x''
>>>> ```x''''
>>>>
>>>> and so forth. You could say that they define empty strings if you
>>>> like, but the effect is the same: a non-existent string and an empty
>>>> string look identical.
>>>>
>>>> From what you say that is precisely what you did. It was the -foreach-
>>>> loop that defined x as what the loop used -- and (an extra principle
>>>> here) that macro is not defined outside the loop, and indeed is doubly
>>>> local, to the loop as well as in the usual sense.
>>>>
>>>> The user manual [U] contains a basic introduction to macros in
>>>>
>>>> [U]     Chapter 18  . . . . . . . . . . . . . . . . . . . .  Programming Stata
>>>>         (help program, macro, quotes, syntax, examplehelpfile)
>>>>
>>>> From your earlier posts it seems that you need to save the file within
>>>> the loop _or_ otherwise use the name you want directly.
>>>>
>>>> Nick
>>>> njcoxstata@gmail.com
>>>>
>>>> On 28 May 2013 06:00, Michael Stewart <michaelstewartresearch@gmail.com> wrote:
>>>>> Dear Steve and Nick ,
>>>>> I am having problem with saving the files
>>>>> I am using ``x'' and the dofile runs through multiple dofiles like a charm
>>>>> However, after completetion of the datamanagement tasks, the  each
>>>>> file is being stored as "_new" in the stata working directory.
>>>>> And the interesting part is that  this happens only if the individual
>>>>> tasks have foreach loop in it. If the individual tasks dont have
>>>>> foreach loop in it, then the files are saved in respective folders
>>>>> i.e., save ``x''_new, replace --> works
>>>>> I am not sure is there is a way to circumvent this problem
>>>>> Thanks
>>>>> Mike
>>>>>
>>>>>
>>>>> On Mon, May 27, 2013 at 10:46 PM, Michael Stewart
>>>>> <michaelstewartresearch@gmail.com> wrote:
>>>>>> HI ,
>>>>>> First of all, thank you very much Steve and Nick.It works like a charm
>>>>>> and I can automate stata to replicate many tasks
>>>>>> I work on Windows .Using ``x'' works.I havent tested `x'.But it
>>>>>> Works.Thats great .
>>>>>> Thank you Steve and Nick
>>>>>> Thank you ,
>>>>>> Yours Sincerely,
>>>>>> Mike.
>>>>>>
>>>>>>
>>>>>> On Mon, May 27, 2013 at 9:46 PM, Steve Samuels <sjsamuels@gmail.com> wrote:
>>>>>>>
>>>>>>> Well, I was concerned about that, but the code worked for me in
>>>>>>> Mac OS X, Here was my test do file:
>>>>>>>
>>>>>>>
>>>>>>> **********************************
>>>>>>> local ds1 "~Downloads/testsim01"
>>>>>>> local ds2 "~Downloads/dtest01"
>>>>>>>
>>>>>>> local all ds1 ds2
>>>>>>> di `"`all'"'
>>>>>>> foreach x in `all' {
>>>>>>> use `x', clear
>>>>>>> gen nv =1
>>>>>>> save `x'_new, replace
>>>>>>> }
>>>>>>> ********************************
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>> On May 27, 2013, at 8:50 PM, Nick Cox wrote:
>>>>>>>
>>>>>>> This won't work, as when Stata hits the loop
>>>>>>>
>>>>>>> foreach x in `all'
>>>>>>>
>>>>>>> it will then see
>>>>>>>
>>>>>>> foreach x in ds1 ds2
>>>>>>>
>>>>>>> and will fail at
>>>>>>>
>>>>>>> use ds1, clear
>>>>>>>
>>>>>>> There are two levels of macro referencing in Steve's code, but only
>>>>>>> one level going backwards.
>>>>>>>
>>>>>>> use ``x'', clear
>>>>>>>
>>>>>>> is one way to fix it.
>>>>>>>
>>>>>>> Nick
>>>>>>> njcoxstata@gmail.com
>>>>>>>
>>>>>>>
>>>>>>> On 28 May 2013 01:23, Steve Samuels <sjsamuels@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Michael Steward (michaelstewartresearch@gmail.com):
>>>>>>>>
>>>>>>>> You can loop, as Nick suggests. Put the data management tasks in
>>>>>>>> external do files, e.g. tasks1.do, tasks2.do, tasks3.do. Then something
>>>>>>>> like the following should work (not tested in Windows). See the help for
>>>>>>>> -include-. The big problem I anticipate is the mounds of output that
>>>>>>>> this could generate.
>>>>>>>>
>>>>>>>>
>>>>>>>> *********************************
>>>>>>>> local ds1 "c/data/dataset-1"
>>>>>>>> local ds2 "c/data/newdata/dataset-2" // etc.
>>>>>>>>
>>>>>>>> local all ds1 ds2
>>>>>>>>
>>>>>>>> foreach x in `all' {
>>>>>>>>   use `x', clear
>>>>>>>>
>>>>>>>>   include tasks1
>>>>>>>>   include tasks2
>>>>>>>>   include tasks3
>>>>>>>>
>>>>>>>>   save `x'_new, replace
>>>>>>>> }
>>>>>>>> *********************************
>>>>>>>>
>>>>>>>> Steve
>>>>>>>>
>>>>>>>>
>>>>>>>> On May 27, 2013, at 3:19 PM, Michael Stewart wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>> I am trying to find a way to work multiple files successively.
>>>>>>>> I have multiple large files(really large) placed in different directories
>>>>>>>> I am trying to find a way to load each file into stata, complete bunch
>>>>>>>> of same datamagement tasks,create  a new file from it and places them
>>>>>>>> in a common folder
>>>>>>>>
>>>>>>>> I could use a dofile and do something like
>>>>>>>>
>>>>>>>> clear
>>>>>>>> use c\data\dataset-1
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> save c\data\dataset-1_new
>>>>>>>> clear
>>>>>>>> use c\data\newdata\dataset-2
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> datamagement tasks
>>>>>>>> save c\data\newdata\dataset-2_new
>>>>>>>> clear
>>>>>>>>
>>>>>>>> But is there an elegant and simpler way to do it given that I have
>>>>>>>> nearly 15 files to do data management tasks.
>>>>>>>>
>>>>>>>> Can we use foreach loop / forvalues loop or anything like that .Please suggest
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>
>>>
>>>
>>> --
>>> Thank you ,
>>> Yours Sincerely,
>>> Mike.
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> Thank you ,
> Yours Sincerely,
> Mike.
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index