# Re: st: Rounding error (?) from substr to (double) real

 From jlinhart@stata.com (Jean Marie Linhart, StataCorp.) To statalist@hsphsun2.harvard.edu Subject Re: st: Rounding error (?) from substr to (double) real Date Tue, 25 Mar 2003 14:48:54 -0600

Nick Cox <n.j.cox@durham.ac.uk> wrote:

> 1. To keep every digit in a numeric identifier that is interpretable
> as an integer, use -long- not -double-. The very large numbers which
> can be approximately be held in a -double- obscure the fact that
> even 8-digit integers cannot all be held exactly, giving rise to
> anomalies such as those you experienced.

I think Nick mistyped here.  He meant that doubles cannot hold 16
digit integers.  They do just fine with 8 digit integers.

Why is this?

If I can explain this coherently and without any typos, IEEE double
precision numbers have 64 total bits (binary digits) broken down into:
1 bit for the sign, 11 bits for the binary exponent and 52 bits for
the binary fraction.  It is the binary fraction that determines the
precision.  The binary fraction is intended to represent a binary
number between 1 and 2, i.e., there is an assumed 1 at the front, we
really have 1.F where F is the fractional part that is stored in the
52 bits.  Any nonzero number can be written this way by choosing the
correct exponent.  This gives us a precision of 1/2^53.  Since 1e-15 >
1/2^53 > 1e-16, this means we expect to get 15 digits.  Sometimes we
will get 16, but not always.

Or do web searches on "IEEE floating point"

--Jean Marie  "Math and dialog girl, woo hoo!"
jlinhart@stata.com
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/