Re: st: RE: RE: joinby command and memory issues

["Weichle, Thomas" <Thomas.Weichle@va.gov>]

Does this demonstrate that using this method is limited by my system?

The max memory appears to be right around 1050m.  I read in the original
datasets, drop unnecessary variables, compress the data, and then save
them.  After that, I perform the joinby and still see the error code.

. set memory 1100m
op. sys. refuses to provide memory
r(909);

end of do-file

r(909);

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. set memory 1060m
op. sys. refuses to provide memory
r(909);

end of do-file

r(909);

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. set memory 1050m

Current memory allocation

                    current                                 memory usage
    settable          value     description                 (1M = 1024k)
    --------------------------------------------------------------------
    set maxvar         5000     max. variables allowed           1.909M
    set memory         1050M    max. data space              1,050.000M
    set matsize         400     max. RHS vars in models          1.254M
                                                            -----------
                                                             1,053.163M

. 
end of do-file

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. use "G:\ESA_Cancer\ESA_DATA\ESA_USE\epo0209.dta", clear

. keep study_id epo_dt

. sort study_id epo_dt

. compress
epo_dt was long now int
study_id was double now long

. save "G:\ESA_Cancer\ESA_DATA\ESA_USE\epo0209.dta", replace
file G:\ESA_Cancer\ESA_DATA\ESA_USE\epo0209.dta saved

. 
end of do-file

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. use "G:\ESA_Cancer\ESA_DATA\ESA_USE\hgb0209.dta", clear

. keep study_id ord_date result

. sort study_id ord_date result

. compress
ord_date was long now int
study_id was double now long

. save "G:\ESA_Cancer\ESA_DATA\ESA_USE\hgb0209.dta", replace
file G:\ESA_Cancer\ESA_DATA\ESA_USE\hgb0209.dta saved

. 
end of do-file

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. ******** Open Dataset ******** 
. use "G:\ESA_Cancer\ESA_DATA\ESA_USE\epo0209.dta", clear

. unique study_id
Number of unique values of study_id is  36381
Number of records is  406400

. 
. describe, fullnames

Contains data from G:\ESA_Cancer\ESA_DATA\ESA_USE\epo0209.dta
  obs:       406,400                          
 vars:             2                          8 Oct 2010 16:15
 size:     5,689,600 (99.6% of memory free)
------------------------------------------------------------------------
---------------------------
              storage  display     value
variable name   type   format      label      variable label
------------------------------------------------------------------------
---------------------------
study_id        long   %12.0g                 Study ID
epo_dt          int    %d                     
------------------------------------------------------------------------
---------------------------
Sorted by:  study_id  epo_dt

. 
end of do-file

. do "C:\DOCUME~1\VH6F19~1\LOCALS~1\Temp\6\STD00000000.tmp"

. * Pairwise combinations
. joinby study_id using "G:\ESA_Cancer\ESA_DATA\ESA_USE\hgb0209.dta",
unmatched(none) _merge(_merge
> )
no room to add more variables because of width
    An attempt was made to add a variable that would have increased the
memory required to store an
    observation beyond what is currently possible.  You have the
following alternatives:

     1.  Store existing variables more efficiently; see help compress.

     2.  Drop some variables or observations; see help drop.  (Think of
Stata's data area as the
         area of a rectangle; Stata can trade off width and length.)

     3.  Increase the amount of memory allocated to the data area using
the set memory command; see
         help memory.
r(902);

end of do-file

r(902);


Tom Weichle
Math Statistician
Center for Management of Complex Chronic Care (CMC3)
Hines VA Hospital, Bldg 1, C202
708-202-8387 ext. 24261
Thomas.Weichle@va.gov 


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/