Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: RE: RE: RE: Re: memory management in Win2000


From   "Steven Stillman (LMPG)" <Steven.Stillman@lmpg.dol.govt.nz>
To   "'statalist@hsphsun2.harvard.edu'" <statalist@hsphsun2.harvard.edu>
Subject   st: RE: RE: RE: Re: memory management in Win2000
Date   Tue, 15 Jul 2003 13:21:48 +1200

Thanks to a bit more testing, I have figured out that the problem is being
caused because I am opening datasets stored on a network drive. 
Steve

> -----Original Message-----
> From:	Ed Bassin [SMTP:edbassin@profsoft-health.com]
> Sent:	Monday, July 14, 2003 10:10 PM
> To:	statalist@hsphsun2.harvard.edu
> Subject:	st: RE: RE: Re: memory management in Win2000
> 
> I don't have a solution to your issue but I regularly open 300-500MB
> datasets using Stata 7 on Win2K at rates around 150-200MB/minute.  The
> machines on which that occurs have RAID 5 arrays and 2GB or more of
> physical
> memory, but that shouldn't explain this big of a difference.
> 
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Steven Stillman
> (LMPG)
> Sent: Monday, July 14, 2003 1:21 AM
> To: 'statalist@hsphsun2.harvard.edu'
> Subject: st: RE: Re: memory management in Win2000
> 
> Sorry, I should have been a bit clearer here.  I am using stata 7 with
> stata
> 7 format datasets.  I am not using virtual memory with at least 100m of
> physical memory available after opening a 240m dataset.  The problem is
> with
> the initial allocation of physical memory that occurs when I open stata
> for
> the first time.  This is just plain slow taking approx. 1min to allocate
> each 20m.  I am assuming this has to do with win2000 and my particular
> motherboard, etc. and nothing to do with Stata but I was hoping that some
> experts out there might have a solution for speeding things up.
> 
> Thanks,
> Steve
> 
> PS I am about to install stata 8 so we will soon see if this helps speeds
> things up.
> 
> > -----Original Message-----
> > From:	kturner@stata.com [SMTP:kturner@stata.com]
> > Sent:	Saturday, July 12, 2003 3:46 AM
> > To:	statalist@hsphsun2.harvard.edu
> > Subject:	st: Re: memory management in Win2000
> > 
> > Steven Stillman <Steven.Stillman@lmpg.dol.govt.nz> wrote: 
> > 
> > >>This is more of a windows questions than a stata question.  I 
> > >>frequently work on large datasets (150-200m) on a Pent IIII PC w/ 
> > >>Win2000 and 512m
> > ram.
> > >>I find that it takes a long time (3-5 minutes) for these large 
> > >>datasets
> > to
> > >>initial open (stata is very fast once the datasets are loaded).  
> > >>Does
> > anyone
> > >>have any suggestions on how to get Win2000 to allocate memory 
> > >>quicker to stata (this seems to be what holds things up)?  I was 
> > >>thinking that ala
> > Unix
> > >>there might be a way to assign a fixed block of ram to stata instead 
> > >>of
> > it
> > >>being dynamically reallocated every time a dataset is opened.
> > 
> > And Friedrich Huebler <huebler@rocketmail.com> had this suggestion:
> > 
> > >Do you use Stata 8 with a dataset from an earlier version of Stata?
> > >Try to convert the dataset to Stata 8 format and see if it takes less 
> > >time to open.
> > 
> > This suggestion is a very good one. Using the latest format of dataset 
> > for your version of Stata is always faster.
> > 
> > Another suggestion:
> > Due to the fact that it takes so long and that the size of the dataset 
> > is already a large percentage of the physical RAM (200/512), the odds 
> > are that Windows is being forced to use virtual memory. You can verify 
> > this by using the Windows Task Manager (typically you can right-click 
> > on the task bar to bring up the Task Manager; Ctrl-Alt-Del is another 
> > way to get to it). On the Task Manager, there is a tab for Performance 
> > and on this tab is a couple of tables showing memory usage. If your 
> > Total Commit Charge memory is higher than your Total Physical memory, 
> > you are in a state where Windows is forced to use virtual memory.
> > 
> > You can *probably* witness the problem by viewing the Available 
> > Physical Memory _during_ the load of the Stata dataset -- it should 
> > hover around 6-3Mb as Windows continually swaps memory from Physical 
> > to Virtual space.
> > 
> > Solutions:
> > 	1. Buy more RAM. 
> > 	2. Close all unnecessary applications while you are working with 
> > these
> > 	   large datasets.
> > 	3. There are a few more odd tricks that can be done to speed up 
> > 	   virtual memory, but may not fix the entire problem: 
> > 	   http://www.pureperformance.com/js/showtip.asp?id=22
> > 
> > 
> > Hope this helps,
> > 
> > --Kevin
> > kturner@stata.com
> > *
> > *   For searches and help try:
> > *   http://www.stata.com/support/faqs/res/findit.html
> > *   http://www.stata.com/support/statalist/faq
> > *   http://www.ats.ucla.edu/stat/stata/
> 
> The information contained in this document is intended only for the
> addressee and is not necessarily the views nor the official communication
> of
> the Department of Labour.  All final/official papers which are sent from
> the
> Department will be sent by non-electronic means, on appropriate
> letterhead,
> signed by authorised personnel.
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/

The information contained in this document is intended only for the
 addressee and is not necessarily the views nor the official 
communication of the Department of Labour.  All final/official papers 
which are sent from the Department will be sent by non-electronic
means, on appropriate letterhead, signed by authorised personnel.
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index