Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, is already up and running.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

st: collapse is too memory demanding

From   Oliver Jones <>
Subject   st: collapse is too memory demanding
Date   Mon, 26 Jul 2010 20:43:11 +0200

Hi everybody,
this is my first posting and I hope to ask a good question...
How much additional free memory do I need to perfom a -collapse-?

Problem setting:
I have a dataset containing information about ~20 million people living in ~180 different regions and working in ~330 different jobs. The information is given by ~70 zero/one dummy variables, like male[yes/no], female[yes/no], age20-25[yes/no], ...
When I try to collapse it like this I get the error that I need more free memory

* begin excerpt code
* m_total is a dummy variable taking the value 1 if the person is male
* f_total is a dummy variable taking the value 1 if the person is female
collapse (sum) m_total f_total ...(68 more dummy variables), by(aoaa beruford) fast
no room to add more variables because of width
An attempt was made to add a variable that would have increased the memory required to store an observation beyond what is currently possible. You have the
    following alternatives:

     1.  Store existing variables more efficiently; see help compress.

2. Drop some variables or observations; see help drop. (Think of Stata's data area as the area of a rectangle; Stata can trade off width and length.)

3. Increase the amount of memory allocated to the data area using the set memory command; see help memory.

.         memory
Details of set memory usage
    overhead (pointers)                     159,493,976        8.45%
    data                                  1,455,382,531       77.11%
    data + overhead                       1,614,876,507       85.56%
    free                                    272,560,293       14.44%
    Total allocated                       1,887,436,800      100.00%
Other memory usage
    set maxvar usage                          2,041,738
    set matsize usage                         1,315,200
    programs, saved results, etc.                37,424
    Total                                     3,394,362
Grand total                               1,890,831,162

* end code excerpt

I am grateful for any help.

Kind regards

*   For searches and help try:

© Copyright 1996–2016 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index