Dear all,
this is a repost of a question regarding a problem that is still
bothering us, there might not be any obvious answer, apart from "buy
more RAM"...
A colleague of mine has a question about memory allocation using Mata:
---------------------------
Hello,
I am trying to do matrix calculations in Mata in a very large data
set (around 3 million observations and roughly 25 relevant variables).
My operating system won't allow me to allocate more than 930M of memory
(I have 1gb), and I have been using views where possible. My problems
seem to be with memory management, although I'm open to alternative
interpretations.
The first problem is when I do a calculation of the sort
"x=cross(x',u)" with x (3,188,029 by 22) being a view on a dataset with
no extraneous variables and u being a small (22 by 22) matrix, I get the
message "<istmt>: 3900 unable to allocate real <tmp>[22,3188029]"
which I take to mean that Mata can not allocate the memory necessary for
storing a temporary matrix used in the calculation, even if there is no
problem if the calculation replaces the initial data with the result of
the matrix multiplication. Is there any way to get around this other
than looping over each observation (or block of observations) in the
data set?
Second, I am again using a view onto a (now smaller) data set with 3
variables and 3,188,029 observations and need to do some row-by-row
calculations with another 3,188,029 by 22 matrix that I stored
previously (the one at the origin of problem 1). When I try to
fgetmatrix the stored matrix, I get a similar error message, namely
"fgetmatrix(): 3900 unable to allocate real <tmp>[3188029,22]". I have
no clue how to get around this one, since looping won't work if I can't
even get the matrix into memory and I don't know how to read an
arbitrary row of a stored matrix (the calculations do not necessarily
draw martix rows sequentially).
I'm currently running this on Stata9 and could switch to Stata9 SE,
but my understanding of the differences leads me to believe that the
solution doesn't lie in that direction, and in any case I'm hoping to
turn this (eventually) into a public-access ADO for which requiring
users to have Stata9 SE seems to be a little harsh.
Thank you for any help you can give.
-----------------------------------
Best,
Antoine
*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/
