Re: st: Mata Memory

[Olesya Kotlyachkov <vovko@mcmaster.ca>]

On 07/23/2012, Natarajan Balasubramanian wrote:

> Hi,
>    
> I am having trouble with memory allocation in Stata (I tried both IC Stata 11X64 and IC Stata 12X64, both).
>
> Here is the command and the error:
> ************
> felsdvreg y x, ivar(f1) jvar(f2) feff(feff) peff(peff) mover(mover)  group(group) xb(xb) res(res) mnum(mnum) pobs(pobs)
>
> Memory requirement for moment matrices in GB:
>     8.045968656
>                        J():  3900  unable to allocate real <tmp>[31712,31712]
>                decompreg():     -  function returned error
>                    <istmt>:     -  function returned error
> r(3900);
>
> end of do-file
>
> r(3900);
> ************
> Note that felsdvreg estimates regressions with two high-dimensional fixed effects. In this case, I have about 35,000 of each fixed effect on a dataset of about 200,000.
>
> For the r(3900), this is what the Stata manual says , "You must take up the problem with your  operating system". Based on an older post, (http://www.stata.com/statalist/archive/2006-10/msg00938.html   ), a 64 bit machine (which is what I am using) is better. 8GB is more than the RAM on my machine (4GB) but is there any way to fix this without upgrading the RAM. Note that with a smaller sample (10% of the current one), the same program runs correctly.
Hi -
I am having exactly the same issue with the felsdvreg 
command. felsdvreg is producing the same error when reaching 
step 11:
********
Step 11 - Fit restricted models for F-tests Start Mata 
environment
Memory requirement for moment matrices in GB:
.15945408

decompreg(): 3900 unable to allocate real <tmp>[275922,4415]
<istmt>: - function returned error r(3900);
 ********
Following Stata Manual advice, I took up the problem with my 
operating system.
I am running 4 core Stata-12 on a 64 bit Windows 7 machine 
with 4 cores, 16 Gb physical RAM, and a large hard drive 
with system managed page file. I ran the Process Explorer to 
see what's happening with the system and the memory. 
Whatever felsdvreg is doing, Stata/Mata take up all 16 gig 
of RAM, ask for more - and crash. Normally Windows would 
automatically add the virtual memory when the physical limit 
is reached, however in this case it does not. Any idea why?
I have also tried to set min_memory in Stata right away - 
clean system after reboot, min_memory of 24 Gb sets fine. 
Then I start the do-file - and the command does not even 
run, I only see the felsdvreg command line and no output at 
all - not even the first step. It does not really freeze, it 
just does not do anything.

Has anyone seen this happen and especially if you have 
resolved it - please help me figure it out! I'm interested 
in both issues - reaching physical RAM limit while in Mata, 
and setting min_memory and Mata. Any help would be greatly 
appreciated!

Thanks!
Olesya Kotlyachkov
McMaster University
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/