Re: st: file cannot be opened in loop appending many datasets

[Sergiy Radyakin <serjradyakin@gmail.com>]

Adrien,
1. can you provide an example with capture that does not work? fileop
rethrows the same error code outside if it exceeds the number of
attempts permitted. you can still capture that error code.
2. the 603 error in practice often occurs because there is another
software that reacts to a new file, most commonly an antivirus
program, that scans a new file immediately upon creation. during the
time the av is reading this file it remains unavailable for other
programs. the RAM/HDD sync is likely not a contributor.
Best,Sergiy

On Thu, Jul 25, 2013 at 9:55 AM, adrien bouguen
<abouguen@povertyactionlab.org> wrote:
> Thanks Sergiy.
>
> Though, your program seems not to be working with a "capture" as it is
> the case in my loop.
>
>  I have resolved the problem by adding a "sleep 2000" after saving the
> data. But it is true Stefan that it slows down the process a lot.
>
> To understand what's going on from a non-IT specialist person, the
> error is caused by the fact that data are stored in a hard drive while
> it is manipulated in the RAM. Since the RAM is way faster than the
> hard drive it messes things up. Is that correct?
>
> Thanks
>
> 2013/7/25 Stefan Bernhard <stefanbernhard88@gmail.com>:
>> Sergiy, thanks so much for your little program!
>> I had the same problem with looping and appending over several
>> thousand, and this speeds up my process by some 1000%, as I had added
>> a -sleep 500- after each loop to reduce the likelihood of failure
>> (which it still did several times after a few hundred loops).
>>
>> On a sidenote, isn't it somehow possible to tell Stata to use the RAM
>> for -local tempfile-s instead of the HDD? It seems to be a wasteful
>> process to write to the mechanic disc tens of thousands of times
>> during such loops as the here mentioned, instead of using the
>> non-mechanical RAM.
>>
>> On a second sidenote, I tried to add a -noisily- to your -display-
>> line in fileop, but the quitely of my own loop suppresses the output.
>> What are the mechanics behind this? Can I somehow make noisily go back
>> up through all 'layers' of quietly? That being without adding
>> -noisily- before the -fileop- in the respective -save- operation,
>> because I don't want to read the saving info.
>>
>> i.e. during:
>>
>> qui {
>> foreach something {
>> fileop save "test.dta", replace
>> }
>> }
>>
>> I want to know when fileop did its job and read the line "Encountered
>> error `ecode'.`emsg'"
>>
>>
>> Best Regards, Stefan Bernhard,
>>
>> 2013/7/24 Sergiy Radyakin <serjradyakin@gmail.com>:
>>> See if -fileop- can help you.
>>>
>>> radyakin.org/statalist/2013072401/fileop.htm
>>>
>>> Best, Sergiy Radyakin
>>>
>>>
>>>
>>>
>>> On Wed, Jul 24, 2013 at 11:15 AM, adrien bouguen
>>> <abouguen@povertyactionlab.org> wrote:
>>>> Dear all,
>>>>
>>>> I am trying to append a large number of datasets using 2 loops and I am
>>>> getting a strange error. After a while, the loop stops with a r(603) error
>>>> saying that the data cannot be opened whereas the data is there and was not
>>>> opened before.
>>>>
>>>> The error seems to be affected datasets randomly : it is never the same
>>>> dataset that can't be opened and it is never the same loop number (here
>>>> count) that poses the problem.
>>>>
>>>> I suspect a problem with the memory but I have tried this command in
>>>> different computers without success. See below for the code.
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks so much for your help
>>>>
>>>>
>>>> clear
>>>> set mem 2g
>>>> set virtual on
>>>> global dta "Q:/numerique/dta"
>>>> global text "Q:/numerique/text"
>>>>
>>>> clear
>>>> set mem 1000
>>>> set virtual on
>>>> local files : dir "$dta/enfants" files "*"
>>>> dis `files'
>>>> global count=0
>>>> foreach file in `files' {
>>>> local z=subinstr("`file'",".dta","",.)
>>>> use `"$var/`z'_v1.dta"', clear
>>>> global count=$count+1
>>>> forval j=3(2)31 {
>>>> dis "`j'"
>>>> cap append using `"$var/`z'_v`j'.dta"'
>>>> if _rc==0 {
>>>> local y=`j'+1
>>>> drop if v1=="" & v`j'==""
>>>> replace v2=v`y' if v1==""
>>>> replace v1=v`j' if v1==""
>>>> drop v`j' v`y'
>>>> }
>>>> }
>>>> save `"$col/`z'_colonne.dta"', replace
>>>> clear
>>>> }
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> Adrien Bouguen
> Ecole d'économie de Paris - J-PAL
> 66 bis avenue Jean Moulin
> 75014 Paris
> tel: +33 (0)1 71 19 40 83
> port: +33 (0)6 69 34 12 02
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/