Questions tagged [rounding-error]

rounding errors are when the number that is stored is not exactly the number expected

This error occurs because decimal numbers may not have an exact representation in binay.

Example: 0.210 = 0.0011001100110011001100110011001100110011...2

Because the number has to be truncated, as computers have a finite capacity for storage, the rounding error occurs.

if the example 0.210 is held in 16 bits, the value becomes 0.0011001100110012, and multiplying the number by 510, it will become 0.1111111111111012, instead of 1.0000000000000002, and comparing 0.210 * 510 to 1 would result in inequality, because of the rounding error

303 questions
-4
votes
1 answer

When I round x in python it returns 0.0

When I try to run print(round(x, 2)) it shows as 0.0.
-4
votes
2 answers

Java Rounding to 15 decimal place

I have the below codes round the forward rate to 15 decimal place. When _ForwardRate is 13,555.0, the result return is wrong. public double round(double Number, int Decimal_Place) { if (Number==0) return 0; double _plug = 0.000001; …
bittersour
  • 813
  • 1
  • 8
  • 23
-6
votes
2 answers

SQL Server Rounding issue

I have a problem in data accuracy when I store data in variables/columns. Consider the following example DECLARE @w NUMERIC(38, 15) = 0.458441, @a NUMERIC(38, 15) = 10.000000, @p NUMERIC(38, 15) = 5.000000 select (1+0.458441)*(1+…
Pரதீப்
  • 85,687
  • 16
  • 112
  • 148
1 2 3
20
21