Stack Overflow
Stack Overflow

Questions tagged [double-precision]

Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.

243 questions
7
votes
3 answers

Accuracy of double precision multiplication in java?

What is the guaranteed accuracy of multiplication operator for double values in java? For example, 2.2 * 100 is 220.00000000000003, but 220 is a double number. 220.00000000000003 is the next double after 220.
ililit
  • 1,259
  • 1
  • 12
  • 18
7
votes
3 answers

Help in double precision addition

I was testing some of my code, in javascript I added .1+.2 and it gives me .30000000000000004 instead of .3 . I don't understand this. But when I added .1+.3 it gives me .4. I googled it and find its something about Double Precision addition. But I…
Vaibhav Jain
  • 31,558
  • 43
  • 103
  • 159
7
votes
6 answers

How can I 'trim' a C# double to the value it will be stored as in an sqlite database?

I noticed that when I store a double value such as e.g. x = 0.56657011973046234 in an sqlite database, and then retrieve it later, I get y = 0.56657011973046201. According to the sqlite spec and the .NET spec (neither of which I originally bothered…
fostandy
  • 3,892
  • 4
  • 32
  • 40
7
votes
3 answers

Java - maximum loss of precision in one double addition/subtraction

Is it possible to establish, even roughly, what the maximum precision loss would be when dealing with two double values in java (adding/subtracting)? Probably the worst case scenario is when two numbers cannot be represented exactly, and then an…
Bober02
  • 13,918
  • 28
  • 83
  • 157
6
votes
6 answers

float to double assignment

Consider the following code snippet float num = 281.583f; int amount = (int) Math.round(num*100f); float rounded = amount/100.0f; double dblPrecision = rounded; double dblPrecision2 = num; System.out.println("num : " + num + " amount: " + amount + "…
Prabhu R
  • 12,624
  • 19
  • 75
  • 107
6
votes
2 answers

Is there an accurate approximation of the acos() function?

I need an acos() function with double precision within a compute shader. Since there is no built-in function of acos() in GLSL with double precision, I tried to implement my own. At first, I implemented a Taylor series like the equation from Wiki -…
DanceIgel
  • 646
  • 7
  • 24
6
votes
8 answers

How can I trust casting from double to integer?

I've been working on some simple code for creating histograms and found that following code: double value = 1.2; double bucketSize = 0.4; double bucketId = value / bucketSize; std::cout << "bucketId as double: " << bucketId << std::endl; std::cout…
Moomin
  • 1,606
  • 5
  • 25
  • 45
6
votes
3 answers

Java float is more precise than double?

Code: class Main { public static void main (String[] args) { System.out.print("float: "); System.out.println(1.35f-0.00026f); System.out.print("double: "); System.out.println(1.35-0.00026); } } Output: float:…
Mustafa
  • 803
  • 1
  • 13
  • 22
6
votes
3 answers

Formatting Double to print without scientific notation using DecimalFormat

I have been reading timestamp values from sensor readings, but since they are provided in nanoseconds, I thought I would cast them to double and make the conversion. The resulting number is a 17 digit value, plus the separator. Trying to print it…
ravemir
  • 905
  • 2
  • 11
  • 27
6
votes
2 answers

when to use expm1 instead of exp in java

I am confused about using expm1 function in java The Oracle java doc for Math.expm1 says: Returns exp(x) -1. Note that for values of x near 0, the exact sum of expm1(x) + 1 is much closer to the true result of ex than exp(x). but this page…
Masood_mj
  • 1,036
  • 8
  • 21
5
votes
5 answers

Tiny numbers in place of zero?

I have been making a matrix class (as a learning exercise) and I have come across and issue whilst testing my inverse function. I input a arbitrary matrix as such: 2 1 1 1 2 1 1 1 2 And got it to calculate the inverse and I got the correct…
Rarge
  • 221
  • 1
  • 4
  • 13
5
votes
3 answers

host to network double?

I'd like to send some double precision floating point numbers over the network. (standard C, standard sockets) There is no htond or ntohd to convert the data to and from network byte order. What should I do? I have a couple solutions in my head but…
dec-vt100
  • 190
  • 2
  • 9
5
votes
2 answers

Fortran double precision program with a simple MKL BLAS routine

In trying to mix precision in a simple program - using both real and double - and use the ddot routine from BLAS, I'm coming up with incorrect output for the double precision piece. Here's the code: program test !! adding this statement narrowed…
Shamster
  • 1,852
  • 4
  • 21
  • 26
5
votes
2 answers

IEEE-754 floating-point precision: How much error is allowed?

I'm working on porting the sqrt function (for 64-bit doubles) from fdlibm to a model-checker tool I'm using at the moment (cbmc). As part of my doings, I read a lot about the ieee-754 standard, but I think I didn't understand the guarantees of…
sascha
  • 27,544
  • 6
  • 57
  • 97
5
votes
6 answers

C# High double precision

I'm writing a function that calculates the value of PI, and returns it as a double. So far so good. But once the function gets to 14 digits after the decimal place, it can't hold any more. I'm assuming this is because of the double's limited…
Entity
  • 7,263
  • 19
  • 69
  • 117
1
2
3
…
16 17