Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

Re: st: announcing statsbyfast - a faster statsby


From   vwiggins@stata.com (Vince Wiggins, StataCorp)
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: announcing statsbyfast - a faster statsby
Date   Thu, 17 Jul 2003 15:23:21 -0500

Michael Blasnik <michael.blasnik@verizon.net> notes that a faster version of
-statsby- that uses -in- rather than -if-, -statsbyfast-, is now available.

> [...]
> The new ado -statsbyfast- is now posted on SSC for downloading.
> statsbyfast is statsby with fewer than 10 lines of changed code and
> it works identically as far as I can tell.

I will note that two of us here at StataCorp had been following this thread
and considered changing -statsby- to use -in- for most cases.  I haven't
thought carefully about Michal's solution, but a quick look shows that it is
different than what we were considering and much more ingenious.

It will, however, suffer from one potential problem that our solution also
suffered.  If the command being executed for each group changes the sort order
of the data, then the results will become indeterminate, and generally not
what you want.  This will never happen with official Stata estimators or for
the vast majority of Stata commands, because they are "sort stable".  It is
incredibly easy for users to also make there commands sort stable, just add
the -sortpreserve-, option to the end of the -program XYZ- statement.

Even so, many old user-written commands will not be sort stable.

Since Michael is apparently much faster at responding to this than we are, he
might consider adding the "fix" we had in mind for the sorting problem.  (We
were going to warn Michael that we might change -statsby-, but he was so fast
that before we finished considering our options he had posted a darn good
solution.)

This will make perfect sense to Michael since he is already using most of
these tricks.  First, keep the old PostGroups subprogram and rename it say
PostGroupsSlowly.  After each run of the command check that the sort order is
what it was when you started.  After the first run, this is probably a waste
of time, but it is a fast comparison of two, usually short, string macros.  If
it is not, throw away everything and call -statsbyfast- again with a special
hidden option.  When the hidden option is seen, reroute through
PostGroupsSlowly, rather than the now faster PostGroups.

We are very likely to add this speed up to -statsby-, but in our own good
time.


-- Vince
   vwiggins@stata.com

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index