Floating point inaccuracy examples
I am working with parsing xlsx files and came across this behaviour. I don’t know how to explain it without describing the steps to reproduce the issue.
1. Create new blank Excel file (xlsx) 2. Type '8.8' into cell A1 3. Save file 4. Close Excel, rename file with .zip extension and open 5. Find sheet1.xml and open 6. Value for 8.8 in A1 will be saved as '8.8000000000000007'
This creates a problem for me, because when I parse Excel files, I want to get the value typed in, ‘8.8’ not ‘8.8000000000000007’.
Does anyone know why Excel does this?
EDIT: And how does Excel know when I open that file to only display ‘8.8’ and not ‘8.8000000000000007’? Does Excel round at a specific decimal place?
A floating point number which has a finite decimal representation doesn’t necessarily have a finite binary representation. The conversion is a lossy one.
Because 8.8 is a fraction whose denominator (5) isn’t a power of 2, its binary floating-point representation is infinite. The underlying data type the application uses to store it (Excel in this case) is of finite precision, and the representation is thus truncated.