Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

st: collapse is too memory demanding


From   Oliver Jones <ojones@wiwi.uni-bielefeld.de>
To   statalist@hsphsun2.harvard.edu
Subject   st: collapse is too memory demanding
Date   Mon, 26 Jul 2010 20:43:11 +0200

Hi everybody,
this is my first posting and I hope to ask a good question...
How much additional free memory do I need to perfom a -collapse-?

Problem setting:
I have a dataset containing information about ~20 million people living in ~180 different regions and working in ~330 different jobs. The information is given by ~70 zero/one dummy variables, like male[yes/no], female[yes/no], age20-25[yes/no], ...
When I try to collapse it like this I get the error that I need more free memory

**********
* begin excerpt code
*
* m_total is a dummy variable taking the value 1 if the person is male
* f_total is a dummy variable taking the value 1 if the person is female
*
collapse (sum) m_total f_total ...(68 more dummy variables), by(aoaa beruford) fast
*
*
.
.
no room to add more variables because of width
An attempt was made to add a variable that would have increased the memory required to store an observation beyond what is currently possible. You have the
    following alternatives:

     1.  Store existing variables more efficiently; see help compress.

2. Drop some variables or observations; see help drop. (Think of Stata's data area as the area of a rectangle; Stata can trade off width and length.)

3. Increase the amount of memory allocated to the data area using the set memory command; see help memory.
r(902);
*
*
memory

.         memory
                                                  bytes
--------------------------------------------------------------------
Details of set memory usage
    overhead (pointers)                     159,493,976        8.45%
    data                                  1,455,382,531       77.11%
                                        ----------------------------
    data + overhead                       1,614,876,507       85.56%
    free                                    272,560,293       14.44%
                                        ----------------------------
    Total allocated                       1,887,436,800      100.00%
--------------------------------------------------------------------
Other memory usage
    set maxvar usage                          2,041,738
    set matsize usage                         1,315,200
    programs, saved results, etc.                37,424
                                        ---------------
    Total                                     3,394,362
-------------------------------------------------------
Grand total                               1,890,831,162

*
*
* end code excerpt
**********



I am grateful for any help.

Kind regards
Oliver



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index