Re: st: Op. sys. refuses to provide memory - a cautionary tale

[Alan Riley <ariley@stata.com>]

Ben Craig (Benjamin.Craig@moffitt.org) started a discussion thread
regarding Stata/MP 11 and memory allocation.

We're continuing to work with Ben; we think the memory allocation error
he reported is easily resolved.  We will report back to the list when
we have a resolution.

Later in the thread, Ben posted a response from Stata Technical Support.
Some parts of that response could easily be misconstrued.

To clarify:

   All flavors of Stata, including Stata/MP, can take advantage of the
   larger amount of memory 64-bit operating systems provide as compared
   to 32-bit systems.

   The only differences between running Stata on 32-bit and 64-bit
   systems are the amount of memory that can be allocated and the fact
   that 64-bit systems can move data somewhat faster.

   Stata 11's factor variables REDUCE the amount of memory required
   by Stata.  In fact, sometimes they reduce it by a lot.  The indicator
   variables for the levels of factor variables and their interactions
   are virtual.  They are not created in your dataset.

   When a user encounters a memory allocation error, setting Stata's
   version to an earlier number is not the answer.

There are four general memory allocation problems that can arise:

   Q1) variations on "no room to add more variables" and "no room to add more
       observations"

   Q2) "matsize too small"

   Q3) "maxvar too small"

   Q4) "operating system refuses to provide memory"


We recommend the following approaches when you encounter one of
these messages.  Note that the approach differs for each message.

   Q1) variations on "no room to add more variables" and "no room to add more
       observations".

       A1) Increase the amount of memory allocated to Stata's data area with
           -set memory-.  The goal is to allocate enough memory to hold
           your dataset AND any variables you may create or that Stata
           commands may create in performing your analysis.  There is no
           need to allocate more than is necessary for this.  See the following
           FAQ for advice on how to calculate the proper setting:

              http://www.stata.com/support/faqs/data/setmemory.html

           Allocating more than is necessary ties up memory that may be
           needed elsewhere.


   Q2) "matsize too small"

       A2) Increase the matsize setting with -set matsize-.  Generally,
           matsize needs to be no larger than the maximum number of
           covariates you expect to use in an estimation, or, in a multi-
           equation setting, the maximum number of parameters you expect
           to estimate.

           Allocating more than is necessary ties up memory that may be
           needed elsewhere.


   Q3) "maxvar too small"

       A3) Stata/MP and Stata/SE support up to 32767 variables.  However,
           at startup, to conserve memory which may be needed elsewhere, the 
           maximum number of allowed variables is set to 5000.  If you
           see the message above, increase this number by typing
           -set maxvar- and setting it to the maximum number of variables
           you expect to have in the current Stata session.

           Allocating more than is necessary ties up memory that may be
           needed elsewhere.


   Q4) "operating system refuses to provide memory"

       A4) This message means exactly what it says.  Stata asked the
           operating system for some amount of memory, and the operating
           system told Stata that amount of memory was not available.

           This message usually occurs when you are trying to resolve
           one of Q1 through Q3, and you get this error in response to
           the associated -set- command.  At this point, you need to
           increase the amount of memory the operating system is willing
           to make available to Stata.  For very large problems, that may
           mean you need a 64-bit system.  Also, on 32-bit Windows XP,
           it may mean you need to update to Service Pack 3 to fix a problem
           in Windows XP's memory manager.

           First, however, make sure that you are not asking for more
           memory than you need.
           
           Rarely, Stata will issue this error message while executing a
           command, as seen by Ben.  This usually arises only when one
           of the -set- commands discussed above has allocated almost
           all of the memory available on the system.

           To solve this problem, try REDUCING some of the other
           settings.  This may seem counter-intuitive, but what Stata is
           trying to tell you is that it needs some memory outside of
           that allocated by the above settings.



-- Alan                         -- Vince
   ariley@stata.com                vwiggins@stata.com

Addendum: With regard to A4, official Stata commands rarely ask Mata to
allocate much memory.  You, however, may use Mata in ways that allocate
a large amount of memory.  Be aware that all of Mata's memory allocation
is independent from the memory allocated by the three -set- commands above
and must come from free memory outside those areas.

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/