[Proj] Numerical Precision in Proj, WKT, and everywhere

[Craig Bruce]

Gerald I. Evenden <geraldi.evenden at gmail.com> wrote:

> BCD? Binary Coded Decimal? Just that I have not heard the term in maybe
> 25 years. Seriously. I am not being sarcastic.

All the common database systems use BCD to represent the very common
NUMBER data type.  This allows them to represent terminating decimal
fractions exactly and SQL allows numbers to have up to 38 decimal digits.
See the section "Packed decimal numbers" near the bottom of:

http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.ibm.db2.luw.sql.ref.doc/doc/r0001030.html

The down side is that calculations are slower (not preformed directly
by hardware), but this slowness is hidden by the large overhead costs of
database access.

> As I recall, the only place you demonstrated precision problems was
> an example that went to full mantissa precision where I admit there
> may be problems.  *But* I know of no place in cartography where such
> precision is required.

I'm speaking much more generally than cartography.  I'm saying that all
doubles should be displayed in the way I suggest in general by default
unless you know the values have errors in them or the application
domain suggests using fewer digits.  (Most of the code I write is very
general purpose.)  Otherwise, you're introducing arbitrary errors into
the numbers, causing a greater divergence between a binary->binary and
a binary->text->binary round trip if such a round trip can't be avoided.
You need to use 17 digits to get a completely clean round trip, but this
exposes the eyesore patterns in the IEEE encoding.

--------------------------+----------------------+--------------------------
Dr. Craig S. Bruce        | Ph 819-771-8303 x205 |             CubeWerx Inc.
Senior Software Developer |   Fax 819-771-8388   |  Gatineau, Québec, Canada
csbruce at cubewerx.com      |  http://csbruce.com/ |  http://www.cubewerx.com/
--------------------------+----------------------+--------------------------
    "The purpose of The Law is to protect criminals from being lynched."
                [What, you though it was to protect *you*?]