Re: st: Re: Memory, Stata 10, and 32-bit XP

[Alan Riley <ariley@stata.com>]

There has been a recent discussion about memory allocation in Stata
on various platforms and various versions of Stata.  Michael Blasnik
(michael.blasnik@verizon.net) described his experience moving from
32-bit Windows XP to 32-bit Windows Vista:

> With Stata 9/SE, I was able to allocate 1200m-1350m under WinXP32 but can 
> now allocate about 1750m under Vista32 (with 4GB installed).  It is perhaps 
> the only reason to think about switching to Vista (it does take a couple of 
> weeks to figure out how to disable all the stuff you don't want to use, and 
> there are still compatibility problems with some older applications).
> 
> I haven't yet moved to Stata 10 and I wonder if other people find that v10 
> can't load as large datasets as v9 under Windows?

Stata 10 for Windows does have a memory footprint that is bigger than
Stata 9 for Windows.  That is, Stata 10, before the user allocates
memory to data with -set memory-, takes more memory for itself.  This
is to be expected: each new version of Stata in general will have
a larger memory footprint than the previous version due to new
features added to it.  (That is an oversimplification, but it will
do for the purpose of this email.)

Under 32-bit Windows, the change from Stata 9 to Stata 10 is about
200 MB due to a particular graphics library being used in the new
Graph Editor.  Thus, if Michael can allocate 1750 MB under 32-bit Vista
using Stata 9, I would expect him to be able to allocate about 1550 MB
under 32-bit Vista using Stata 10.


With regards to the recent discussion about memory allocation in
Stata for Windows, here is a summary:

   64-bit Linux or Unix on 64-bit hardware:
        memory allocation limited only by amount of memory on the
        computer

   32-bit Linux, Unix, or Macintosh:
        memory allocation limited in theory to 2 GB; limited in
        practice to very close to 2 GB  (it is even possible
        to sometimes get very slightly more than 2 GB, depending
        on what the operating system will allow)

   64-bit Windows (any version) on 64-bit Itanium hardware:
        memory allocation limited only by amount of memory on the
        computer

   64-bit Windows (any version) on 64-bit x86-64 hardware:
        memory allocation limited only by amount of memory on the
        computer
         
   32-bit Windows Vista:
        memory allocation limited in theory to 2 GB; limited in
        practice to roughly 1500 MB to 1800 MB.

   32-bit Windows XP:
        memory allocation limited in theory to 2 GB; limited in
        practice to roughly 900 MB to 1200 MB.

You will note that the numbers under 32-bit Windows XP are low.  This
is because of a problem inherent in the memory manager in Windows XP
(not in Stata) exacerbated by changes made by Microsoft in Windows XP
Service Pack 2 (SP2).  A hotfix from Microsoft is available which can
help the user resolve the problem introduced by Windows XP SP2, as
detailed here:

    http://www.stata.com/support/faqs/win/winmemory.html

Stata needs to allocate its memory for data as a single contiguous
block of memory.  The problem with the 32-bit Windows XP memory
manager is that it can (but does not always) fragment the memory
available to an application which sometimes makes it impossible for
Stata to allocate as much memory as a user would like.  The memory
manager in 32-bit Windows Vista does not suffer from this problem.

As I wrote above, 32-bit Windows Vista has a better memory manager and
does not suffer from this problem.  In addition, Microsoft has
promised that Service Pack 3 (SP3) for Windows XP is coming very soon.
We have installed an early trial version of SP3 available only to
software developers and our initial tests indicate that SP3 resolves
the same problem as the hotfix.

Because we know a large number of users are still using Windows
XP, in addition to waiting for SP3 to be available, we are looking
into a way for Stata to work around the problem with the Windows XP
memory manager.


Alan Riley
(ariley@stata.com)

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/