Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Op. sys. refuses to provide memory - a cautionary tale


From   Sergiy Radyakin <serjradyakin@gmail.com>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Op. sys. refuses to provide memory - a cautionary tale
Date   Tue, 24 Aug 2010 14:45:56 -0400

Martin, this is very vogue. What is the "total amount of memory on
your machine"?
Is this the amount of hardware sticks of DIMMs (or whatever
technology) that you've
paid your dollars (euros, pounds) for? or is it the amount of memory
that Windows
can see and make use of?

The answer should probably be "the least of ...." followed by some
parameters of the
hardware and software.

Most importantly, even some 64-bit flavors of Windows face the 2GB
physical memory
limit (e.g. Win7 Starter Edition), which means that if you are running
such a system,
installing more hardware memory will not do you any good. 16GB is a
limit for several
64-bit flavors of Windows: e.g. Windows Vista Home Premium.

Let me put a link to the Microsoft's website, as this is very volatile
information:
http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx

The "availability" is also technology-dependent. Refer for example to
the Stata 1.0
manual. It says, in particular: "When STATA is started, the program
examines your PC,
determines how much memory is available, and then lays claim to all of
it. Thus the
size of the largest data set that you can use depends upon the
hardware configuration
of your PC". If Stata 1.0 was available now, and you ran it in Windows
7, would it claim
all the GBs installed and available? No. However the statement I
quoted from the manual
was true, in that sense Stata will make use of all the "memory that is
made available to it",
which may differ however large from "memory that is put into the
computer box". In the
case of Stata 1.0 it would mean that it will honestly make use of 640k
out of perhaps 8GB
or whatever you have in your desktop. [anyone still having Stata 1.0
can give it a shot, and
let me use an opportunity and ask here if anyone still has a copy of
Stata for DOS? or for
that matter, what is the earliest version of Stata that is currently
in operation?]

2Benjamin: I understand your frastration, however, if you are in
charge of getting your own
hardware and software, configuring it all and making it work together,
find some time to
learn about the potential problems, incompatibilities, limits and
adjust your expectations
accordingly. Working with large datasets with Stata is possible, and
all the necessary
information is available and made public, and nobody intentionally
hides it from you. If you
don't have time to check this out, it is always possible to delegate
such tasks to others,
or simply to ask around for ready solutions. Perhaps you only need to
upgrade the Windows
since the hardware itself has been 64-bit for years now.

"Does any listserv member think that I should go from 6 to 12 cores?"
Yes! Why not? What do we know about your expected returns from the
project, availability
of funds? Do you even face a budget constraint? If not, go for the
32-cores (current max for
Stata/MP11)...

2Eric: Quote: "however it will get excruciatingly slow if you set your
memory to 500m and
use a 1G dataset." I don't think it is going to be slow, it just not
going to work. Stata will say
"not enough memory". I'd rather agree to the following: "if you
install 512mb of physical
memory into the computer, but allocate 1GB in Stata it will slow down"
(and note: regardless (!)
of the size of the dataset).

Best,
 Sergiy Radyakin


2010/8/24 Martin Weiss <martin.weiss1@gmx.de>:
>
> <>
>
>
> " but what is the final word? How much memory can a 64bit system handle?"
>
>
>
>
> " You are limited only by the total amount of memory on your machine", see
> http://www.stata.com/products/64bitintro.html
>
>
>
>
> HTH
> Martin
>
>
> -----Ursprüngliche Nachricht-----
> Von: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] Im Auftrag von Kiss Sándor
> Csanád
> Gesendet: Dienstag, 24. August 2010 11:53
> An: statalist@hsphsun2.harvard.edu
> Betreff: RE: st: Op. sys. refuses to provide memory - a cautionary tale
>
> Sorry guys, I read all this, and also this
>
> http://www.stata.com/statalist/archive/2010-01/msg00409.html
>
> but what is the final word? How much memory can a 64bit system handle?
>
> Thanks
>
>
>
>           Kiss Sándor Csanád
>           Elemző közgazdász / Economist
>          Office of the Fiscal Council, Hungary
>           1055 Budapest, Honvéd utca 20.
>           T: (+36 1) 510 3025
>           Mobil: (+36 30) 703 1024
>           Fax: (+36 1) 510 3099
>           Web: www.mkkt.hu
>
>
> -----Original Message-----
> From: owner-statalist@hsphsun2.harvard.edu
> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Jeph Herrin
> Sent: Monday, August 23, 2010 7:30 PM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>
> I took the plunge on Windows 7 64bit and have been very
> satisfied with it relative to XP 64bit, which is what that machine
> was running. Though with only 16gb RAM, I haven't really put
> it to the test you envision with 192gb.
>
> For what it's worth, I shop my PCs from a custom builder that
> caters to engineers and CAD/CAM types; they were dead set against
> selling Vista but said that Win7 had so far proved itself for
> their customers and they were recommending it for those who
> needed not-Linux.
>
> cheers,
> Jeph
>
>
> On 8/23/2010 12:45 PM, Jay Tuthill wrote:
>> Hi All,
>>
>> I'm also in the middle of upgrading my hardware as my current 32 bit
>> computers will not let me run very efficiently the datasets I want. I'm
>> considering the new Intel workstation system SC5650SCWSR which offers
>> dual Xeon processors and up to 192GB of RAM. Am curious if anyone has
>> compared the Windows 7 64bit versus Window Server 2008 R2 64bit
>> operating systems. (I have built my own computers for several years and
>> have a wide latitude in how I configure them.)
>>
>> Thanks...Jay
>>
>> -----Original Message-----
>> From: owner-statalist@hsphsun2.harvard.edu
>> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Eric Booth
>> Sent: Saturday, August 21, 2010 5:24 PM
>> To:<statalist@hsphsun2.harvard.edu>
>> Subject: Re: st: Op. sys. refuses to provide memory - a cautionary tale
>>
>> <>
>>
>> On Aug 21, 2010, at 4:19 PM, Eric Booth wrote:
>>
>>>   but beyond this limit Stata won't slow down as you add or allocate
>> more RAM.
>>
>> Clarification:
>> That should say, "but UP TO (or before you reach this memory limit)
>> Stata won't slow down as you add or allocate more RAM..."
>>
>> On Aug 21, 2010, at 4:19 PM, Eric Booth wrote:
>>
>>> <>
>>> On Aug 20, 2010, at 4:07 PM, Tony wrote:
>>>> Too much RAM will slow it down.
>>>
>>> Stata will certainly slow down if you set and use more memory in Stata
>> than is physically available on your machine because you start using
>> virtual memory, but beyond this limit Stata won't slow down as you add
>> or allocate more RAM.
>>> That is, it will take the same time to run a do-file on a 1G dataset
>> whether you allocate 2G, 8G, or 20G of memory to Stata; however it will
>> get excruciatingly slow if you set your memory to 500m and  use a 1G
>> dataset.
>>>
>>>
>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:
>>>
>>>>> The purpose is real world speed, so has anyone actually noticed if
>> going
>>>>> up to 24GB RAM, solid state drive expedited your jobs
>>>
>>>
>>> I haven't tested the idea of getting a SSD drive, but I think the
>> speed advantage would be evident mainly when you were opening(reading)
>> or saving(writing) a large dataset since your using the data in memory
>> the rest of the time.  I do want to try out installing a SSD drive for
>> working with data that is larger than my physical RAM and requires me to
>> use virtual memory to work with it (I've maxed out my physical RAM with
>> 8G sticks in each of the slots).  I've read about moving your page/swap
>> file to a SSD which should speed things up when working in virtual
>> memory (but since SSDs wear out faster with more read/writes, this might
>> be a concern).  Also, you could move the location of the tempfiles that
>> Stata creates to that path by setting your OS system temp file location
>> (see:  http://www.stata.com/statalist/archive/2009-05/msg00416.html).
>> Maybe someone here has tried working with SSD and large datasets ?
>>>
>>> Again, more RAM is always better IMO--but it only speeds you up in the
>> sense that it prevents you from using page file.  There are also speeds
>> associated with RAM (mine is 1066 DDR3), but I don't know much
>> differences in memory speed matters.
>>>
>>>
>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M. wrote:
>>>>> Does any listserv member think that I should go from 6 to 12 cores?
>>>>>
>>>>> Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>> Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>
>>> Depends on what you are doing.  If you've got a time intensive
>> procedure that you're running on your 6 core machine, try running it
>> with your -set processors- at 1, 2, 4, and 6 and see what kind of speed
>> increase you observe, e.g.:
>>>
>>> *****!
>>> timer clear
>>> forval n = 1(2)6 {
>>> clear all
>>> set mem 32g
>>> set processors `n'
>>> timer on `n'
>>>      <your command goes here>
>>> timer off `n'
>>>      }
>>> timer list
>>> *****!
>>>
>>> ~ Eric
>>>
>>>
>>> __
>>> Eric A. Booth
>>> Public Policy Research Institute
>>> Texas A&M University
>>> ebooth@ppri.tamu.edu
>>> Office: +979.845.6754
>>>
>>>
>>>>
>>>> On Fri, Aug 20, 2010 at 1:28 PM, Craig, Benjamin M.
>>>> <Benjamin.Craig@moffitt.org>  wrote:
>>>>> Thanks Nick, I have learned that to truly take advantage of the
>> latest
>>>>> version of Stata, 64-bits and 4 or more cores is required. To be a
>> bit
>>>>> more specific, lets assume I am using 6-core Stata MP on Windows 7
>>>>> Professional, 64-bit for computationally intensive simulation
>> analyses.
>>>>>
>>>>> Does any listserv member think that I should go from 6 to 12 cores?
>>>>>
>>>>> Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>> Dual Six Core Processor,X5680,3.33GHz,12M,6.4GT/s
>>>>>
>>>>> Is it worthwhile to upgrade from RAM and Hard drive? For example,
>>>>>
>>>>> 12GB DDR3 ECC SDRAM Memory,1333MHz,6X2GB
>>>>>
>>>>> 1TB SATA 3.0Gb/s,7200 RPM HardDrive with 32MB DataBurst Cache
>>>>>
>>>>> The purpose is real world speed, so has anyone actually noticed if
>> going
>>>>> up to 24GB RAM, solid state drive expedited your jobs? In theory, it
>>>>> should, but I am hoping that someone has purchase a computer
>> recently to
>>>>> test this hypothesis.
>>>>>
>>>>> Cheers,
>>>>> Ben
>>>>>
>>>>>
>>>>> Benjamin M. Craig, PhD
>>>>>
>>>>> Assistant Member, Health Outcomes&  Behavior, Moffitt Cancer Center
>>>>>
>>>>> Associate Professor, Economics, University of South Florida
>>>>>
>>>>> 12902 Magnolia Dr, MRC-CANCONT, Tampa, FL 33612-9416
>>>>>
>>>>> Phone (813) 745-6710; Fax (813) 745-6525
>>>>>
>>>>> -----Original Message-----
>>>>> From: owner-statalist@hsphsun2.harvard.edu
>>>>> [mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
>>>>> Sent: Friday, August 20, 2010 5:00 AM
>>>>> To: 'statalist@hsphsun2.harvard.edu'
>>>>> Subject: RE: st: Op. sys. refuses to provide memory - a cautionary
>> tale
>>>>>
>>>>> I think this partially answers itself in that I don't think that it
>> can
>>>>> fairly be expected that a company website is a proper place for a
>>>>> company, in this case StataCorp, to offer opinions about anything
>> that
>>>>> is currently controversial.
>>>>>
>>>>> That said, Benjamin's question is obviously practical and a fair one
>> for
>>>>> members of this list to venture opinions and comment from
>> experience. In
>>>>> addition, presumably people other than econometricians are not
>> excluded.
>>>>>
>>>>>
>>>>> Nick
>>>>> n.j.cox@durham.ac.uk
>>>>>
>>>>> Craig, Benjamin M.
>>>>>
>>>>> To clarify, the website answers essential questions (which systems
>> are
>>>>> supported?) and provides some advice and education. I need to know
>> more
>>>>> details on current controversies relating to multiple core, 64-bit
>> and
>>>>> drive speeds in the real world. In this sense, it is a bit
>> incomplete to
>>>>> say that more core, more bits, and faster drives are preferable. As
>> was
>>>>> previously post, I had thought that a 32-bit dual core desktop was
>> good
>>>>> enough for my needs, and was woefully wrong. Others seem to
>> follow...
>>>>>
>>>>> http://www.stata.com/statalist/archive/2010-07/msg01337.html
>>>>>
>>>>> If a listserv member has tried STATA MP on multiple machines, I
>> would
>>>>> like to know what worked best so that I can buy one. There is an
>> obvious
>>>>> caveat: speed depends on the task. However, I would counter that
>> some
>>>>> data are better than none. For example, I recently bootstrapped a ML
>>>>> with 1000 iteration and inequality constraints. It took 4 weeks. Do
>> you
>>>>> think your machine can do better? Personally, I do not use stata for
>>>>> database management, and doubt that many econometricians do. If
>> someone
>>>>> has a good analytics machine, and he/she thinks that it works well,
>> I'd
>>>>> like to know its components. Maybe a consensus will emerge. Maybe
>> not.
>>>>>
>>>>> Martin Weiss
>>>>>
>>>>> In which respect is the Stata website "incomplete"? There is advice
>> at
>>>>> http://www.stata.com/products/opsysmp.html, and how is the website
>>>>> supposed to give more detailed advice? It does not know your
>> specific
>>>>> setup, hence the reluctance to go into greater depth...
>>>>>
>>>>> Craig, Benjamin M.
>>>>>
>>>>> Okay, I give up... I need a new machine. Due to institutional
>> policies,
>>>>> I need to buy a Dell. Otherwise, I would very much like any advice
>> on
>>>>> this purchase. My best guess is a 64-bit 8-core desktop for a 6-core
>>>>> version of Stata MP. I don't need a rocket, just a racecar.
>>>>>
>>>>> If you have any specifications that you would like to share with me,
>>>>> please send them directly or post them on the listserv for others
>>>>> looking to upgrade. I have read the Stata website, which seems
>>>>> incomplete.
>>>
>>>
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/statalist/faq
>>> *   http://www.ats.ucla.edu/stat/stata/
>>
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index