Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1226 questions
3313
votes
30 answers

Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
Cato Johnston
  • 38,149
  • 10
  • 35
  • 42
404
votes
11 answers

How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x == 0) { // do important operation } Isn't…
Proud Member
  • 38,700
  • 43
  • 143
  • 225
283
votes
6 answers

Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
dplanet
  • 4,883
  • 9
  • 26
  • 42
181
votes
21 answers

What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run static analysis, I get: "JAVA0078 Floating point…
user128807
  • 10,179
  • 16
  • 49
  • 72
161
votes
4 answers

Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values actually have ugly decimal representations: >>>…
Aivar
  • 6,112
  • 3
  • 40
  • 69
131
votes
12 answers

Is it possible to get 0 by subtracting two unequal floating point numbers?

Is it possible to get division by 0 (or infinity) in the following example? public double calculation(double a, double b) { if (a == b) { return 0; } else { return 2 / (a - b); } } In normal cases it…
Thirler
  • 18,868
  • 13
  • 58
  • 86
108
votes
5 answers

Dealing with float precision in Javascript

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying floating point precision? For example: 0.2 +…
Jeroen
  • 28,942
  • 33
  • 116
  • 190
100
votes
28 answers

Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I don't need the value rounded off. Hope this is possible…
kcssm
  • 1,397
  • 3
  • 13
  • 19
91
votes
4 answers

What's the closest double to 1.0, that isn't 1.0?

Is there a way to programmatically get the double that is closest to 1.0, but isn't actually 1.0? One hacky way to do this would be to memcpy the double to a same-sized integer, and then subtract one. The way IEEE754 floating-point formats work,…
jorgbrown
  • 1,628
  • 12
  • 20
89
votes
8 answers

PHP - Floating Number Precision

$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
dcmoody
  • 1,225
  • 1
  • 11
  • 14
75
votes
3 answers

Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include int main() { printf("%d\n", (0.1 + 0.2 != 0.3)); return…
Stas
  • 10,615
  • 6
  • 36
  • 55
69
votes
6 answers

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and .Equals() is value comparison. (Edit: I know…
LZW
  • 1,025
  • 1
  • 10
  • 13
67
votes
6 answers

Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5, the value of jsonParsed is the number rounded: Object…
Jaanus
  • 17,268
  • 14
  • 62
  • 102
58
votes
4 answers

Inaccuracy of decimal in .NET

Yesterday during debugging something strange happened to me and I can't really explain it: So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but shouldn't the results be the same?
10rotator01
  • 605
  • 6
  • 14
54
votes
5 answers

negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True. Is there any difference between 0.0 and -0.0 (I…
max
  • 40,904
  • 40
  • 170
  • 328
1
2 3
…
81 82