As for accuracy, you’re still confusing the statistical/scientific/engineering meaning of significant DECIMAL digits with binary floating point mantissa bits. Except for exact sums of negative powers of 2, e.g., #.5, #.75, #.203125 (# + 13/64), binary floating point can’t store DECIMAL fractional values exactly. The best binary floating point can do is use all available binary bits to store the closest value possible. Such software then uses binary floating point math for calculations using those numbers and the results are in turn stored using all available binary bits to store the closest value possible. These values are simply the result of unavoidable imprecision in the decimal-to-binary-to-decimal conversions. THEY ARE ACCURATE (or as accurate as possible), just not in the same sense as spurious decimal places beyond the number of significant (DECIMAL) digits in statistical/… context.

You persist in believing Excel can (or should) perform decimal arithmetic. It can if you use precision as displayed. Otherwise, it can’t.

Nah.. I don’t thinkt that is going to change fzz. Afaik MS has always supported the IEEE 754 standard and my arguments are not that important. The chances of anyone running into problems because of this rounding behaviour is very small. Rewriting Excel is most likely not an option. But despite all that I still think it’s silly to display that much decimals if they are not accurate.

As for the calculation speed, with todays computer that probably isn’t that much of a problem. But you are correct ofcourse that there is a performance penalty for using exact calculations if that were possible.

Charles: that assumption isn’t used at all except in scientific calculations. Fault calculations are commonly used to cover for measuring/calibrating errors for example.

You obviously want Excel to use arbitrary of specified precision decimal arithmetic. That’d make Excel VERY, VERY SLOW. Others prefer Excel to be relatively fast if a bit of a pain to work with in terms of truncation error.

IOW, Excel is what it is. It’s NOT going to change (at least not in this way). You’re going to be very disappointed expecting your arguments will change Excel.

Q: Is there a way to determine how precise a number is after you’ve converted to binary and back? For instance, if you subtract 57 from 57.8, is there an algorithm to determine how precise the answer is?

The degree of error will depend on the method used to convert from decimal to binary and how many binary bits are used.
For Excel I guess the simplest way would be to compare the results of its slightly modified IEEE scheme with the results using an extended precision addin like
http://digilander.libero.it/foxes/MultiPrecision.htm
which claims to be able to handle up to 250 significant digits.

Rembo,

“the maximum error in 4.0 – 3.0 is 0.099999? – this is only true if you make the assumption that digits that are not shown could be anything and ignore any errors caused by the calculation method used (in this case converting to binary and back).
I do not think this assumption is widely used, for instance certainly not in Finance/banking. Even in physical sciences I would have thought it was more usual to state a measurement +-value.

.Value2
Is faster than .Value for numbers because it does not do the Currency and Date implicit type conversions, as well as avoiding the truncation problems.

Jon; I counted the number of decimals in the WoW example. Not sure if I counted them right though, I was more concerned with result of the sum.

Fzz: you look at this from a computer science point of view. That is a great way to explain the problem and why it occurs. Indeed all software that uses floating point calculations following the IEEE 754 standard (as Excel does) will make calcuation errors due to rounding. I can’t argue with that.
But I’m an Excel user and I look at Excel as such. It doesn’t make any sense to me to display 26 digits when oly about 15 digits are accurate (read this on Chip Pearsons page).

I had thought that by changing the formatting to “currency”, this not-quite-zero-sum situation that was being discussed here might have *actually* returned zero. But, alas, it doesn’t because it is merely the object model that is doing a conversion to Currency behind the scenes and not actually within the cells themselves.

As for what *could be*, I think your assessment is exactly right: improving this situation for precision would definitely mean a rather large penalty in calculation time. Probably the easiest way to implement better precision would be to utilize the Decimal data type. It still would not have the range of a double, but it would still have an excellent range and has greater precision (and more flexible precision) than Currency; however, its processing is not native to the CPU and so its calculation speed would probably be about 10x slower than calculating doubles.

Still, it might be neat to be able to set the data type (perhaps via the number format) and be willing to make that trade-off on the fly. I’m not saying that it’s convenient for Excel or the calculation engine to be able to do this, but it’s an interesting idea which could have a lot of value in certain situations. And this is essentially what I mistakenly believed was going on with the “currency” number formatting.

I was basically fooled by the following code, which returns “Currency”:

I’m not sure what else I should have assumed, but I thought that the underlying type actually *was* Currency. But I was wrong. If I was a C++/XLL programmer, one who is used to using an XLOPER on a daily basis, then I would have known better.

Still, I think we learned something here. Perhaps it is “off topic”, I don’t know, but the way Excel VBA is handling Currency data types for cells with “currency” number formatting is definitely interesting. And the fact that the following code truncates to two decimal places is also surprising and worth being aware of:

It certainly makes me want to use .Value2 a little more often.

Even if Excel did use the Variant/Currency data type for cells formatted as currency, that’d only solve truncation error in addition and subtraction. Multiplication and division would remain problematic. Also, it’s less flexible than precision as displayed. If you have a cell (A1) formatted as 0.000 and enter 1.001 in it, it’s squared value should be 1.002001, but currency format will truncate this at 4 decimal places whereas using precision as displayed you could format the cell as 0.000000, and the full value would be preserved.

As long as you have only FINITE precision, you’d still have to handle values that lack finite representations in any base, e.g., SQRT(2). If you want arbitrary precision and symbolic processing, there are software packages available, e.g., Mathematica, MatLab, Maple, Maxima, Gauss, MuPad and maybe others, but AFAIK there are no spreadsheets providing such functionality in no small part because such functionality means SLOW calculations.

Finite precision binary floating point has this advantage: it’s currently the fastest way of performing fairly accurate computer arithmetic. If you believe there would be better (either faster or as fast but more accurate) ways for computers to perform arithmetic calculations, where are they? Wouldn’t you suppose there’d be a huge market for chips implementing such alternatives? Perhaps the absence of such chips is evidence that no one has yet been able to come up with anything better.