Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Overriding a loop if 0 observations using tabstat


From   Jeph Herrin <junk@spandrel.net>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Overriding a loop if 0 observations using tabstat
Date   Wed, 28 Apr 2010 10:45:44 -0400

Agree, this is not intuitive. My first run was with
50mb allocated:

	t=48.90; t=60.45; t=72.30.

but when I allocate 1gb (on an 8gb machine)

	t=79.66; t=105.07; t=121.07

?



Robert Picard wrote:
I don't understand. Under both scenario (-set memory 1g- or -set
memory 10m-), the dataset size and everything else is the same. On my
computer with 12GB of RAM, a 1g allocation should not make a
difference and none of it should be paged out to virtual memory. In
fact, Stata does not even allocated to itself 1GB of real or virtual
memory (when I look at the Activity Monitor) unless I actually create
or load a dataset which requires 1GB of RAM.

The reason why I ask is that the lesson appears to be that when
running Stata, you should always aim for the smallest memory
allocation possible for maximum efficiency at the price of finding
out, hours later when you encounter an insufficient memory error that
you should have used a larger -set memory-.

Robert

On Tue, Apr 27, 2010 at 3:42 PM, Martin Weiss <martin.weiss1@gmx.de> wrote:
<>

The additional 990m for the 1g allocation decrease the amount available for
computations, so this is what I would expect to happen.


HTH
Martin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Robert Picard
Sent: Dienstag, 27. April 2010 21:39
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Overriding a loop if 0 observations using tabstat

Do you guys see a difference if you try under a different memory
allocation? I'm running Stata/MP 11 (4 cores) on a Mac Pro 2.93GHz
Quad-Core with 12GB of RAM and get:

with 1g allocation: t=17.49; t=64.09; t=71.18
with 10m allocation: t=10.93; t=43.35; t=47.68

Just curious,

Robert

On Tue, Apr 27, 2010 at 2:59 PM, Jeph Herrin <junk@spandrel.net> wrote:
This is 64bit MP 2 on Windows 7 with 8G ram.
The processor is an AMD Phenom II with 3.20GHz clock speed.

cheers,
J


Martin Weiss wrote:
<>

Jeph, out of curiosity, what kind of equipment is it that throws up these
numbers? Mine is 64 bit MP 4 on Windows 7 with 4G Ram.


HTH
Martin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Jeph Herrin
Sent: Dienstag, 27. April 2010 20:27
To: statalist@hsphsun2.harvard.edu
Subject: Re: st: Overriding a loop if 0 observations using tabstat

t=48.90; t=60.45; t=72.30. :>


Martin Weiss wrote:
<>

t=100.28; t=207.58; t=241.55. :-)


HTH
Martin


-----Original Message-----
From: owner-statalist@hsphsun2.harvard.edu
[mailto:owner-statalist@hsphsun2.harvard.edu] On Behalf Of Nick Cox
Sent: Dienstag, 27. April 2010 19:08
To: statalist@hsphsun2.harvard.edu
Subject: RE: st: Overriding a loop if 0 observations using tabstat

Good question. I decided to do some timings to support -- or rebut -- my
feeling that -count- which just counts should be faster than -summarize,
meanonly- which does other stuff too and in turn than -summarize- which
does
other stuff too. But although that's the order the timings are closer
than
I
guessed. Still, doing anything the quickest way does no harm and may
give
valuable speed-up for large problems.
Here is one test script. Compare your experiences:
clear
set obs 100000
set seed 2803
gen y = runiform()
set rmsg on

qui forval i = 1/10000 {
       count if y > 0.5
}

qui forval i = 1/10000 {
       su y if y > 0.5, meanonly
}

qui forval i = 1/10000 {
       su y if y > 0.5
}

My timings were t=187.49; 254.49; 313.38, which no doubt shows up the
Mesolithic age of my machine.
Nick n.j.cox@durham.ac.uk
Martin Weiss

" As a small detail of efficiency, I would always recommend -count-
rather
than -summarize- for the purpose here."

My earlier code did use -count-... What makes this thing more efficient,
though? Both are built-in, so they probably enjoy a big advantage over
everybody else anyway. So I guess the reason for your preference is the
fact
that -count- calculates fewer results than -su, mean-?

Nick Cox

A secondary theme here is that this kind of code gets very difficult to
read, which makes it difficult to maintain and debug.
I note that the condition
intab1 == 1 & admit_ic == 1 & btwg < .
is common to all the -summarize- and -tabstat- commands. That being so,
you
could get that out of the way like this
preserve keep if intab1 == 1 & admit_ic == 1 & btwg < .
<stuff> restore
Your -tabstat- options that are constant can be put in a little bag:
local opts stat(n mean median p25 p75 min max) col(stat) f(%9.0g) notot
nosep

Now <stuff> can be rewritten
forv i = 0/5 {
       foreach y in male singlet {
               forv s = 0/1 {
                       di "myga==`i' & `y'==`s'"
                       qui su bwtg if myga==`i' & `y'
                       if r(N) != 0 {
                               tabstat bwtg if myga==`i', `opts' by(`y')
                      }
               }
       }
}

Now it is easier to see what is going on. I added some cosmetic changes
too,
which this horrible mailer may well reverse.
One puzzle: Did you mean to add the condition "& `y'" to the
-summarize-?
It
means the same as
& `y' != 0
-- which may or may not be what you want.
As a small detail of efficiency, I would always recommend -count- rather
than -summarize- for the purpose here.
Nick n.j.cox@durham.ac.uk
sara khan

Many thanks Maarten for your advice. I managed to resolve it with the
following code:

forv i=0/5 {
foreach y in male singlet{
forv s=0/1{
di "myga==`i' & `y'==`s'"
qui su bwtg if myga==`i' & intab1==1 & admit_ic==1 & bwtg<. & `y'
       if r(N)!=0{
tabstat bwtg if myga==`i' & intab1==1 & admit_ic==1 & bwtg<., stat(n
mean median p25 p75 min max ) by(`y') col(stat) f(%9.0g) notot nosep

}
}
}
}


On Tue, Apr 27, 2010 at 12:56 PM, Maarten buis <maartenbuis@yahoo.co.uk>
wrote:
--- On Tue, 27/4/10, sara khan wrote:
I just tried this but the output only shows the display
results and nothing from tabstat.
<snip>

-capture- works for me:

*----------------- begin example ---------------------
sysuse auto, clear
forvalues i = 0/5 {
      capture noisily tabstat mpg if rep78== `i', ///
              s(n mean) by(foreign)
}
*-------------------- end example -------------------

In order to debug your loop I would build it step by step:
step 1: no looping, no locals, no -if- just a single -tatstat- command
step 2: add -capture noisily-
step 3: add some -if- conditions
step 4: build a single loop (e.g. over i but not over y)
etc. etc.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index