Stack Overflow
Stack Overflow

Questions tagged [rounding-error]

rounding errors are when the number that is stored is not exactly the number expected

This error occurs because decimal numbers may not have an exact representation in binay.

Example: 0.210 = 0.0011001100110011001100110011001100110011...2

Because the number has to be truncated, as computers have a finite capacity for storage, the rounding error occurs.

if the example 0.210 is held in 16 bits, the value becomes 0.0011001100110012, and multiplying the number by 510, it will become 0.1111111111111012, instead of 1.0000000000000002, and comparing 0.210 * 510 to 1 would result in inequality, because of the rounding error

303 questions
58
votes
3 answers

BigDecimal multiply by zero

I am performing a simple multiplication with BigDecimal and I have found some strange behaviour when multiplying by zero (multiplying by zero is correct in this use-case). Basic maths tells me that anything multiplied by zero will equal zero…
Richard
  • 9,114
  • 4
  • 24
  • 31
38
votes
9 answers

How is floating point stored? When does it matter?

In follow up to this question, it appears that some numbers cannot be represented by floating point at all, and instead are approximated. How are floating point numbers stored? Is there a common standard for the different sizes? What kind of gotchas…
Adam Davis
  • 87,598
  • 55
  • 254
  • 328
33
votes
5 answers

Rounding Standards - Financial Calculations

I am curious about the existence of any "rounding" standards" when it comes to the calculation of financial data. My initial thoughts are to perform rounding only when the data is being presented to the user (presentation layer). If "rounded" data…
Guamez
  • 335
  • 1
  • 4
  • 6
33
votes
6 answers

What class to use for money representation?

What class should I use for representation of money to avoid most rounding errors? Should I use Decimal, or a simple built-in number? Is there any existing Money class with support for currency conversion that I could use? Any pitfalls that I should…
Esteban Küber
  • 33,970
  • 13
  • 78
  • 96
17
votes
2 answers

C# Rounding MidpointRounding.ToEven vs MidpointRounding.AwayFromZero

In C# Is there any difference in the accuracy of the two decimal rounding strategies MidpointRounding.ToEven and MidpointRounding.AwayFromZero? I mean do both ensure an even distribution amongst the numbers that are rounded to, or is one rounding…
Carlo V. Dango
  • 11,816
  • 15
  • 63
  • 107
16
votes
2 answers

Is there a datatype "Decimal" in R?

I read data stored in the format DECIMAL from a MySQL-Table. I want to do calculations on those numbers within R. I used to cast them to a numeri represaentation using as.numeric(), but the documentation says: numeric is identical to double (and…
R_User
  • 9,332
  • 22
  • 68
  • 115
13
votes
2 answers

Entity Framework Code First truncating my decimals

I am using Entity Framework 6.x using the Code First approach on an MVC 5 application. In this particular situation my model (among other things) contains two properties named Latitude and Longitude: [Required, Range(-90, +90)] public decimal…
Lord of Scripts
  • 3,420
  • 5
  • 35
  • 59
12
votes
5 answers

php intval() and floor() return value that is too low?

Because the float data type in PHP is inaccurate, and a FLOAT in MySQL takes up more space than an INT (and is inaccurate), I always store prices as INTs, multipling by 100 before storing to ensure we have exactly 2 decimal places of precision.…
Josh
  • 10,711
  • 11
  • 62
  • 103
12
votes
4 answers

Specifying DPI of a GDI Device Context

I have an application that generates metafiles (EMFs). It uses the reference device (aka the screen) to render these metafiles, so the DPI of the metafile changes depending on what machine the code is running on. Let's say my code is intending to…
Nicholas Piasecki
  • 23,869
  • 5
  • 74
  • 91
11
votes
6 answers

PHP rounding error

I'm using PHP 5.2.13 on my linux server. I'm getting weird error when rounding numbers. This is my test case:
PiotrL
  • 187
  • 1
  • 1
  • 11
10
votes
2 answers

Delphi Roundto and FormatFloat Inconsistency

I'm getting a rounding oddity in Delphi 2010, where some numbers are rounding down in roundto, but up in formatfloat. I'm entirely aware of binary representation of decimal numbers sometimes giving misleading results, but in that case I would expect…
Robbie Matthews
  • 1,140
  • 10
  • 18
10
votes
4 answers

Rounding differences on Windows vs Unix based system in sprintf

I have problem on UNIX based systems sprintf does not round up properly value. For example double tmp = 88888888888885.875 char out[512]; Thats 88,888,888,888,885.875 just to be easier on eyes. I am giving such specific and big example because it…
grobartn
  • 3,000
  • 9
  • 35
  • 52
10
votes
4 answers

If two languages follow IEEE 754, will calculations in both languages result in the same answers?

I'm in the process of converting a program from Scilab code to C++. One loop in particular is producing a slightly different result than the original Scilab code (it's a long piece of code so I'm not going to include it in the question but I'll try…
Paul Warnick
  • 853
  • 2
  • 10
  • 24
10
votes
4 answers

Detecting precision loss when converting from double to float

I am writing a piece of code in which i have to convert from double to float values. I am using boost::numeric_cast to do this conversion which will alert me of any overflow/underflow. However i am also interested in knowing if that conversion…
Yogesh Arora
  • 2,116
  • 1
  • 24
  • 35
10
votes
4 answers

How do deal with infinitely repeating numbers as decimals?

When division results in an infinitely repeating number, the number obviously gets truncated to fit into the size of the decimal. So something like 1/3 becomes something like 0.3333333333333333333. If we then multiply that number by 3, we get…
GBleaney
  • 1,907
  • 2
  • 19
  • 36
1
2 3
…
20 21