Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: Need help with working with multiple files
From
Michael Stewart <[email protected]>
To
statalist <[email protected]>
Subject
Re: st: Need help with working with multiple files
Date
Tue, 28 May 2013 09:45:19 -0400
Thanks a lot ,Nick.The entire loop works like a charm.I think, I will
take courses on stata programming .
Thanks again
Sincerely
Mike
On Tue, May 28, 2013 at 9:30 AM, Nick Cox <[email protected]> wrote:
> has a inner loop using the same macro name for loop element as the
> outer loop.
>
> The villain in the piece is
>
> foreach x of varlist price mpg rep78 trunk{
>
> which has the side-effect of zapping local macro x once it has finished.
>
> The rule is that a local macro defined by a loop is undefined once
> that loop finishes. It is, doubly, local.
>
> Thus `x' is defined <here> but not <there> in something like
>
> foreach x of local X {
>
> * here `x' is OK
>
> foreach x of local Y {
>
> }
>
> * there `x' has been undefined
>
> }
>
> Remedy: Don't do that. Use a different name, e.g.
>
> foreach y of local Y
>
> Nick
> [email protected]
>
>
> On 28 May 2013 14:03, Michael Stewart <[email protected]> wrote:
>> Dear Statalist user ,
>> I am having problem with saving files after damamanagement tasks using
>> a locals in foreach loop and facing difficulty.(As I try to work with
>> three datasets, the output is only one dataset).I have tried many
>> options, sought help of statlist but able to correct the problem
>> partially but there is a bug somewhere. I am here with posting the
>> code and stata output with "traceon".May be any one could please help
>> me.
>>
>> MY CODE / DO FILE IS AS FOLLOWS
>> local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
>> local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
>> local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
>> local all ds1 ds2 ds3
>> foreach x in `all'{
>> use ``x'',clear
>> di "``x''"
>> include "D:\_Dummy\multiple\4\mpg.do"
>> di "``x''"
>> include "D:\_Dummy\multiple\5\sum.do"
>> save ``x''_new.dta,replace
>> }
>>
>> mpg DO FILE IS AS FOLLOWS ( NO FOREACH LOOP IN THIS DOFILE)
>> gen mean=sum(mpg)
>>
>> sum DOFILE IS AS FOLLOWS (FOREACH LOOP PRESENT IN THIS DOFILE)
>> foreach x of varlist price mpg rep78 trunk{
>> gen x_`x'=`x'==3
>> }
>>
>>
>> PROBLEM: instead of generating three new datasets , it is generating
>> one dataset _new.
>>
>> MY STATA OUTPUT IS AS FOLLOWS(WITH TRACE ON AND DEBUGGNG STEP di
>> ``x''); LOOKS LIKE AFTER EXECUTUNG THE SECOND DOFILE (WITH FOREACH
>> LOOP) di ``x'' shows nothing!!!!!
>>
>> ****************************************************************************************************************************
>> local ds1 "D:\_Dummy\multiple\1\auto_1.dta"
>>
>> . local ds2 "D:\_Dummy\multiple\2\auto_2.dta"
>>
>> . local ds3 "D:\_Dummy\multiple\3\auto_3.dta"
>>
>> . local all ds1 ds2 ds3
>>
>> . foreach x in `all'{
>> 2. use ``x'',clear
>> 3. di "``x''"
>> 4. include "D:\_Dummy\multiple\4\mpg.do"
>> 5. di "``x''"
>> 6. include "D:\_Dummy\multiple\5\sum.do"
>> 7. save ``x''_new.dta,replace
>> 8. }
>> - foreach x in `all'{
>> = foreach x in ds1 ds2 ds3{
>> - use ``x'',clear
>> = use D:\_Dummy\multiple\1\auto_1.dta,clear
>> (1978 Automobile Data)
>> - di "``x''"
>> = di "D:\_Dummy\multiple\1\auto_1.dta"
>> D:\_Dummy\multiple\1\auto_1.dta
>> - include "D:\_Dummy\multiple\4\mpg.do"
>>
>> . gen mean=sum(mpg)
>>
>> .
>> . - di "``x''"
>> = di "D:\_Dummy\multiple\1\auto_1.dta"
>> D:\_Dummy\multiple\1\auto_1.dta
>> - include "D:\_Dummy\multiple\5\sum.do"
>>
>> . gen xx= sum(weight)
>>
>> . foreach x of varlist price mpg rep78 trunk{
>> 2. gen x_`x'=`x'==3
>> 3. }
>> - foreach x of varlist price mpg rep78 trunk{
>> - gen x_`x'=`x'==3
>> = gen x_price=price==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_mpg=mpg==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_rep78=rep78==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_trunk=trunk==3
>> - }
>>
>> .
>> . - save ``x''_new.dta,replace
>> = save _new.dta,replace
>> file _new.dta saved
>> - }
>> - use ``x'',clear
>> = use D:\_Dummy\multiple\2\auto_2.dta,clear
>> (1978 Automobile Data)
>> - di "``x''"
>> = di "D:\_Dummy\multiple\2\auto_2.dta"
>> D:\_Dummy\multiple\2\auto_2.dta
>> - include "D:\_Dummy\multiple\4\mpg.do"
>>
>> . gen mean=sum(mpg)
>>
>> .
>> . - di "``x''"
>> = di "D:\_Dummy\multiple\2\auto_2.dta"
>> D:\_Dummy\multiple\2\auto_2.dta
>> - include "D:\_Dummy\multiple\5\sum.do"
>>
>> . gen xx= sum(weight)
>>
>> . foreach x of varlist price mpg rep78 trunk{
>> 2. gen x_`x'=`x'==3
>> 3. }
>> - foreach x of varlist price mpg rep78 trunk{
>> - gen x_`x'=`x'==3
>> = gen x_price=price==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_mpg=mpg==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_rep78=rep78==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_trunk=trunk==3
>> - }
>>
>> .
>> . - save ``x''_new.dta,replace
>> = save _new.dta,replace
>> file _new.dta saved
>> - }
>> - use ``x'',clear
>> = use D:\_Dummy\multiple\3\auto_3.dta,clear
>> (1978 Automobile Data)
>> - di "``x''"
>> = di "D:\_Dummy\multiple\3\auto_3.dta"
>> D:\_Dummy\multiple\3\auto_3.dta
>> - include "D:\_Dummy\multiple\4\mpg.do"
>>
>> . gen mean=sum(mpg)
>>
>> .
>> . - di "``x''"
>> = di "D:\_Dummy\multiple\3\auto_3.dta"
>> D:\_Dummy\multiple\3\auto_3.dta
>> - include "D:\_Dummy\multiple\5\sum.do"
>>
>> . gen xx= sum(weight)
>>
>> . foreach x of varlist price mpg rep78 trunk{
>> 2. gen x_`x'=`x'==3
>> 3. }
>> - foreach x of varlist price mpg rep78 trunk{
>> - gen x_`x'=`x'==3
>> = gen x_price=price==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_mpg=mpg==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_rep78=rep78==3
>> - }
>> - gen x_`x'=`x'==3
>> = gen x_trunk=trunk==3
>> - }
>>
>> .
>> . - save ``x''_new.dta,replace
>> = save _new.dta,replace
>> file _new.dta saved
>> - }
>>
>> .
>> end of do-file
>> ****************************************************************************************************************************
>>
>>
>> So as you have noticed a, after executing dofile "sum", the naleof
>> the file is being removed and di "" shows nothing.
>>
>> Can anyone of the users help fix the bug.
>>
>> Thank you
>> Mike
>>
>>
>>
>>
>> On Tue, May 28, 2013 at 7:37 AM, Nick Cox <[email protected]> wrote:
>>> It is best to think that you are addressing Statalist rather than
>>> think that one person is necessarily going to answer. Assume that
>>> people at the other end are just as capricious as you might be and get
>>> bored, need to meet partners, attend to the day job, or write their
>>> novel instead. On the whole the list is quite good at picking up
>>> answerable questions but we don't have a buddy system.
>>>
>>> From your example it seems likeliest that one of the extra loops you
>>> are including is somehow zapping the local macro x. I don't have
>>> another guess. Inner loops are not in principle fatal to outer loops
>>> -- otherwise programming would hardly be possible -- but it seems that
>>> your code is stomping on itself.
>>>
>>> Your code is longer and more complicated than you care to show and you
>>> can't win easily here. Post too much code or too little; either can be
>>> wrong in terms of not getting a useful reply.
>>>
>>> But I would check that you are not using the same macro name for
>>> different loops.
>>>
>>> Also, you can add debugging steps
>>>
>>> di "`mymacro'"
>>>
>>> to get Stata to show what it thinks -mymacro- is (or is not).
>>>
>>> Example where nested loops are fine.
>>>
>>> . foreach x in a b c {
>>> 2. di "first time `x'"
>>> 3. foreach y in 1 2 3 {
>>> 4. di "`y'"
>>> 5. }
>>> 6. di "second time `x'"
>>> 7. }
>>> first time a
>>> 1
>>> 2
>>> 3
>>> second time a
>>> first time b
>>> 1
>>> 2
>>> 3
>>> second time b
>>> first time c
>>> 1
>>> 2
>>> 3
>>> second time c
>>> Nick
>>> [email protected]
>>>
>>>
>>> On 28 May 2013 11:24, Michael Stewart <[email protected]> wrote:
>>>> HI Nick
>>>> Sorry to keep bothering you as I wanted to clarify the last few
>>>> sentences of your reply "save the file within
>>>> the loop _or_ otherwise use the name you want directly."
>>>>
>>>> Here in my situation, the code would be
>>>>
>>>> clear
>>>> local ds1 c\data\top/dataset
>>>> local ds2 c\data\date/news/dataset
>>>> ...
>>>> ...
>>>> ..
>>>> local ds13 c\data\data/datedataset-
>>>> local all ds1 ....ds13
>>>> foreach x in `all'{
>>>> use ``x'',clear
>>>> datamagement tasks / contains foreaach loop
>>>> datamagement tasks / contains foreaach loop
>>>> datamagement tasks / contains foreaach loop
>>>>
>>>> save ?????
>>>> }
>>>>
>>>> Here while saving the dataset , when I use save ``x''_new,replace-->
>>>> this saves the dataset as _new and we get one dataset instead of the
>>>> 13 dataset that we started with
>>>>
>>>> According to your recommendation "otherwise use the name you want
>>>> directly", how can I fit 13 names here in the loop .
>>>> Am I missing some important concept here?? Please help me with the
>>>> problem and understand the concept.
>>>>
>>>> Thanks
>>>> MIke
>>>>
>>>>
>>>> On Tue, May 28, 2013 at 3:40 AM, Nick Cox <[email protected]> wrote:
>>>>> If you refer to a macro that doesn't exist Stata doesn't regard that
>>>>> as an error: it just substitutes nothing, or equivalently ignores the
>>>>> reference.
>>>>>
>>>>> Nested macro references don't affect this principle and indeed must be
>>>>> consistent with it. So if local macro x does not exist as a string to
>>>>> be used neither do any of
>>>>>
>>>>> `x'
>>>>> ``x''
>>>>> ```x''''
>>>>>
>>>>> and so forth. You could say that they define empty strings if you
>>>>> like, but the effect is the same: a non-existent string and an empty
>>>>> string look identical.
>>>>>
>>>>> From what you say that is precisely what you did. It was the -foreach-
>>>>> loop that defined x as what the loop used -- and (an extra principle
>>>>> here) that macro is not defined outside the loop, and indeed is doubly
>>>>> local, to the loop as well as in the usual sense.
>>>>>
>>>>> The user manual [U] contains a basic introduction to macros in
>>>>>
>>>>> [U] Chapter 18 . . . . . . . . . . . . . . . . . . . . Programming Stata
>>>>> (help program, macro, quotes, syntax, examplehelpfile)
>>>>>
>>>>> From your earlier posts it seems that you need to save the file within
>>>>> the loop _or_ otherwise use the name you want directly.
>>>>>
>>>>> Nick
>>>>> [email protected]
>>>>>
>>>>> On 28 May 2013 06:00, Michael Stewart <[email protected]> wrote:
>>>>>> Dear Steve and Nick ,
>>>>>> I am having problem with saving the files
>>>>>> I am using ``x'' and the dofile runs through multiple dofiles like a charm
>>>>>> However, after completetion of the datamanagement tasks, the each
>>>>>> file is being stored as "_new" in the stata working directory.
>>>>>> And the interesting part is that this happens only if the individual
>>>>>> tasks have foreach loop in it. If the individual tasks dont have
>>>>>> foreach loop in it, then the files are saved in respective folders
>>>>>> i.e., save ``x''_new, replace --> works
>>>>>> I am not sure is there is a way to circumvent this problem
>>>>>> Thanks
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>> On Mon, May 27, 2013 at 10:46 PM, Michael Stewart
>>>>>> <[email protected]> wrote:
>>>>>>> HI ,
>>>>>>> First of all, thank you very much Steve and Nick.It works like a charm
>>>>>>> and I can automate stata to replicate many tasks
>>>>>>> I work on Windows .Using ``x'' works.I havent tested `x'.But it
>>>>>>> Works.Thats great .
>>>>>>> Thank you Steve and Nick
>>>>>>> Thank you ,
>>>>>>> Yours Sincerely,
>>>>>>> Mike.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, May 27, 2013 at 9:46 PM, Steve Samuels <[email protected]> wrote:
>>>>>>>>
>>>>>>>> Well, I was concerned about that, but the code worked for me in
>>>>>>>> Mac OS X, Here was my test do file:
>>>>>>>>
>>>>>>>>
>>>>>>>> **********************************
>>>>>>>> local ds1 "~Downloads/testsim01"
>>>>>>>> local ds2 "~Downloads/dtest01"
>>>>>>>>
>>>>>>>> local all ds1 ds2
>>>>>>>> di `"`all'"'
>>>>>>>> foreach x in `all' {
>>>>>>>> use `x', clear
>>>>>>>> gen nv =1
>>>>>>>> save `x'_new, replace
>>>>>>>> }
>>>>>>>> ********************************
>>>>>>>>
>>>>>>>> Steve
>>>>>>>>
>>>>>>>> On May 27, 2013, at 8:50 PM, Nick Cox wrote:
>>>>>>>>
>>>>>>>> This won't work, as when Stata hits the loop
>>>>>>>>
>>>>>>>> foreach x in `all'
>>>>>>>>
>>>>>>>> it will then see
>>>>>>>>
>>>>>>>> foreach x in ds1 ds2
>>>>>>>>
>>>>>>>> and will fail at
>>>>>>>>
>>>>>>>> use ds1, clear
>>>>>>>>
>>>>>>>> There are two levels of macro referencing in Steve's code, but only
>>>>>>>> one level going backwards.
>>>>>>>>
>>>>>>>> use ``x'', clear
>>>>>>>>
>>>>>>>> is one way to fix it.
>>>>>>>>
>>>>>>>> Nick
>>>>>>>> [email protected]
>>>>>>>>
>>>>>>>>
>>>>>>>> On 28 May 2013 01:23, Steve Samuels <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> Michael Steward ([email protected]):
>>>>>>>>>
>>>>>>>>> You can loop, as Nick suggests. Put the data management tasks in
>>>>>>>>> external do files, e.g. tasks1.do, tasks2.do, tasks3.do. Then something
>>>>>>>>> like the following should work (not tested in Windows). See the help for
>>>>>>>>> -include-. The big problem I anticipate is the mounds of output that
>>>>>>>>> this could generate.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *********************************
>>>>>>>>> local ds1 "c/data/dataset-1"
>>>>>>>>> local ds2 "c/data/newdata/dataset-2" // etc.
>>>>>>>>>
>>>>>>>>> local all ds1 ds2
>>>>>>>>>
>>>>>>>>> foreach x in `all' {
>>>>>>>>> use `x', clear
>>>>>>>>>
>>>>>>>>> include tasks1
>>>>>>>>> include tasks2
>>>>>>>>> include tasks3
>>>>>>>>>
>>>>>>>>> save `x'_new, replace
>>>>>>>>> }
>>>>>>>>> *********************************
>>>>>>>>>
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On May 27, 2013, at 3:19 PM, Michael Stewart wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>> I am trying to find a way to work multiple files successively.
>>>>>>>>> I have multiple large files(really large) placed in different directories
>>>>>>>>> I am trying to find a way to load each file into stata, complete bunch
>>>>>>>>> of same datamagement tasks,create a new file from it and places them
>>>>>>>>> in a common folder
>>>>>>>>>
>>>>>>>>> I could use a dofile and do something like
>>>>>>>>>
>>>>>>>>> clear
>>>>>>>>> use c\data\dataset-1
>>>>>>>>> datamagement tasks
>>>>>>>>> datamagement tasks
>>>>>>>>> datamagement tasks
>>>>>>>>> save c\data\dataset-1_new
>>>>>>>>> clear
>>>>>>>>> use c\data\newdata\dataset-2
>>>>>>>>> datamagement tasks
>>>>>>>>> datamagement tasks
>>>>>>>>> datamagement tasks
>>>>>>>>> save c\data\newdata\dataset-2_new
>>>>>>>>> clear
>>>>>>>>>
>>>>>>>>> But is there an elegant and simpler way to do it given that I have
>>>>>>>>> nearly 15 files to do data management tasks.
>>>>>>>>>
>>>>>>>>> Can we use foreach loop / forvalues loop or anything like that .Please suggest
>>>>> *
>>>>> * For searches and help try:
>>>>> * http://www.stata.com/help.cgi?search
>>>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>> * http://www.ats.ucla.edu/stat/stata/
>>>>
>>>>
>>>>
>>>> --
>>>> Thank you ,
>>>> Yours Sincerely,
>>>> Mike.
>>>> *
>>>> * For searches and help try:
>>>> * http://www.stata.com/help.cgi?search
>>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> * http://www.ats.ucla.edu/stat/stata/
>>> *
>>> * For searches and help try:
>>> * http://www.stata.com/help.cgi?search
>>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>>> * http://www.ats.ucla.edu/stat/stata/
>>
>>
>>
>> --
>> Thank you ,
>> Yours Sincerely,
>> Mike.
>> *
>> * For searches and help try:
>> * http://www.stata.com/help.cgi?search
>> * http://www.stata.com/support/faqs/resources/statalist-faq/
>> * http://www.ats.ucla.edu/stat/stata/
> *
> * For searches and help try:
> * http://www.stata.com/help.cgi?search
> * http://www.stata.com/support/faqs/resources/statalist-faq/
> * http://www.ats.ucla.edu/stat/stata/
--
Thank you ,
Yours Sincerely,
Mike.
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/