Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Save asarray


From   Sergiy Radyakin <[email protected]>
To   "[email protected]" <[email protected]>
Subject   Re: st: Save asarray
Date   Tue, 21 Jan 2014 12:13:12 -0500

Dear Phil,

thank you very much for your advice. Saving associative arrays with
fputmatrix works, but creates files of unexpectedly large size (e.g.
3.8kb in your example of three keys). I really appreciate you taking
time to provide an example.

The help file for fputmatrix says that: "fputmatrix() writes matrices
in a compact, efficient, and portable format;", which makes me think
that the memory footprint of the asarray (once loaded) would be at
least the same as it is in the file. This explains my Stata exhausting
the memory so fast.

Given this, associative arrays probably wouldn't work for my current
project, and I will have to explore the other options, including the
plugins (perhaps classic plugins to maintain compatibility with Stata
versions prior to 13, where Java first appeared).

Thank you,
    Sergiy Radyakin




On Sat, Jan 18, 2014 at 11:28 AM, Phil Schumm <[email protected]> wrote:
> On Jan 17, 2014, at 6:29 PM, Sergiy Radyakin <[email protected]> wrote:
>> I am also interested in the limits of associative arrays (e.g. storing ~10mln keys of type: "2014_01_17_100_075_XYZ" and values of type double). I am fine if it takes ~300mb in size, but will it be working ok? (in my actual experiment the memory grew very fast and I had to abort the experiment when >4GB where allocated and swapping became intolerable.)
>
>
> I haven't tried something like this with associative arrays in Mata, so I can't really comment apart from pointing out that pre-specifying the size of the array (if you know it ahead of time) and turning off upsizing/downsizing may help.  If it were I, I might be tempted to try storing the data in a MongoDB instance and accessing them through a Java plugin (via the MongoDB Java driver).  Certainly that would scale well, if the access time was fast enough.
>
>
> -- Phil
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/faqs/resources/statalist-faq/
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index