Comparing 1 million integers in an array without sorting it first

Question

I have a task to find the difference between every integer in an array of random numbers and return the lowest difference. A requirement is that the integers can be between 0 and int.maxvalue and that the array will contain 1 million integers.

I put some code together which works fine for a small amount of integers but it takes a long time (so long most of the time I give up waiting) to do a million. My code is below, but I'm looking for some insight on how I can improve performance.

for(int i = 0; i < _RandomIntegerArray.Count(); i++) {
  for(int ii = i + 1; ii < _RandomIntegerArray.Count(); ii++) {
    if (_RandomIntegerArray[i] == _RandomIntegerArray[ii]) continue;

    int currentDiff = Math.Abs(_RandomIntegerArray[i] - _RandomIntegerArray[ii]);

    if (currentDiff < lowestDiff) {
      Pairs.Clear();
    }

    if (currentDiff <= lowestDiff) {
      Pairs.Add(new NumberPair(_RandomIntegerArray[i], _RandomIntegerArray[ii]));
      lowestDiff = currentDiff;
    }
  }
}

Apologies to everyone that has pointed out that I don't sort; unfortunately sorting is not allowed.

Answer 1

Imagine that you have already found a pair of integers a and b from your random array such that a > b and a-b is the lowest among all possible pairs of integers in the array.

Does an integer c exist in the array such that a > c > b, i.e. c goes between a and b? Clearly, the answer is "no", because otherwise you'd pick the pair {a, c} or {c, b}.

This gives an answer to your problem: a and b must be next to each other in a sorted array. Sorting can be done in O(N*log N), and the search can be done in O(N) - an improvement over O(N²) algorithm that you have.

Answer 2

As per @JonSkeet try sorting the array first and then only compare consecutive array items, which means that you only need to iterate the array once:

    Array.Sort(_RandomIntegerArray);
    for (int i = 1; i < _RandomIntegerArray.Count(); i++)
    {
        int currentDiff = _RandomIntegerArray[i] - _RandomIntegerArray[i-1];
        if (currentDiff < lowestDiff)
        {
            Pairs.Clear();
        }

        if (currentDiff <= lowestDiff)
        {
            Pairs.Add(new NumberPair(_RandomIntegerArray[i], _RandomIntegerArray[i-1]));
            lowestDiff = currentDiff;
        }

    }

In my testing this results in < 200 ms elapsed for 1 million items.

Answer 3

You've got a million integers out of a possible 2.15 or 4.3 billion (signed or unsigned). That means the largest possible min distance is either about 2150 or 4300. Let's say that the max possible min distance is D.

Divide the legal integers into groups of length D. Create a hash h keyed on integers with arrays of ints as values. Process your array by taking each element x, and adding it to h[x/D].

The point of doing this is that any valid pair of points is either contained in h(k) for some k, or collectively in h(k) and h(k+1).

Find your pair of points by going through the keys of the hash and checking the points associated with adjacent keys. You can sort if you like, or use a bitvector, or any other method but now you're dealing with small arrays (on average 1 element per array).

Answer 4

As elements of the array are b/w 0 to int.maxvalue, so I suppose maxvalue will be less than 1 million. If it is so you just need to initialise the array[maxvalue] to 0 and then as you read 1 million values increment the value in your array.

Now read this array and find the lowest value as described by others as if all the values were sorted. If at any element is present more than 1 than its value will be >1 so you could easily say that min. difference is 0.

NOTE- This method is efficient only if you do not use sorting and more importantly int.maxvalue<<<<<(less than) 10^6(1 million).

Answer 5

It helps a little if you do not count on each iteration

int countIntegers = _RandomIntegerArray.Count();
for(int i = 0; i < countIntegers; i++) {
    //...
    for(int ii = i + 1; ii < countIntegers; ii++) {
        //...

Given that Count() is only returning the number of Ints in an array on each successful count and not modifying the array or caching output until modifications.

Answer 6

How about splitting up the array into arraysize/number of processors sized chunks and running each chunk in a different thread. (Neil)

Assume three parts A, B and C of size as close as possible.
For each part, find the minimum "in-part" difference and that of pairs with the first component from the current part and the second from the next part (A being the next from C).
With a method taking O(n²) time, n/3 should take one ninth, done 2*3 times, this amounts to two thirds plus change for combining the results.
This calls to be applied recursively - remember Карацу́ба/Karatsuba multiplication?
Wait - maybe use two parts after all, for three fourth of the effort - very close to "Karatsuba". (When not seeing how to use an even number of parts, I was thinking multiprocessing with every processor doing "the same".)