Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Save asarray


From   Phil Schumm <[email protected]>
To   Statalist Statalist <[email protected]>
Subject   Re: st: Save asarray
Date   Sat, 18 Jan 2014 10:28:41 -0600

On Jan 17, 2014, at 6:29 PM, Sergiy Radyakin <[email protected]> wrote:
> I am also interested in the limits of associative arrays (e.g. storing ~10mln keys of type: "2014_01_17_100_075_XYZ" and values of type double). I am fine if it takes ~300mb in size, but will it be working ok? (in my actual experiment the memory grew very fast and I had to abort the experiment when >4GB where allocated and swapping became intolerable.)


I haven't tried something like this with associative arrays in Mata, so I can't really comment apart from pointing out that pre-specifying the size of the array (if you know it ahead of time) and turning off upsizing/downsizing may help.  If it were I, I might be tempted to try storing the data in a MongoDB instance and accessing them through a Java plugin (via the MongoDB Java driver).  Certainly that would scale well, if the access time was fast enough.


-- Phil


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index