14

Generating Prime numbers is a toy problem that I often attempt from time to time, especially when experimenting with a new programming language, platform or style.

I was thinking of attempting to write a Prime Number Generation algorithm or a Prime Number Test Algorithm using Hadoop (Map Reduce).

I thought I'd post this question to get tips, references, to algorithms, approaches.

Although my primary interest is a Map Reduce based algorithm I wouldn't mind looking at new Hadoop programming models or for example looking at using PiCloud

I have seems some interesting questions here on Prime Number Generation: here, here and here, but nothing related to a Parallel approach caught my eye.

Thanks in advance.

Community
  • 1
  • 1
user1172468
  • 4,910
  • 6
  • 29
  • 56
  • check out http://www.mersenne.org and http://en.wikipedia.org/wiki/Great_Internet_Mersenne_Prime_Search, there is also a chinese group that finds very large prime numbers and i'm pretty sure they both would have a decent parallel algorithm(MPI/Distributed) to do so – pyCthon Sep 24 '12 at 23:54

1 Answers1

14

Here's an algorithm that is built on mapping and reducing (folding). It expresses the sieve of Eratosthenes

     P = {3,5,7, ...} \ U {{p2, p2+2p, p2+4p, ...} | p in P}

for the odd primes (i.e without the 2). The folding tree is indefinitely deepening to the right, like this:

enter image description here

where each prime number marks a stream of odd multiples of that prime, e.g. for 7: 49, 49+14, 49+28, ... , which are all merged to get all the composite numbers, and then primes are found in the gaps between the composite numbers. It is in Haskell, so the timing is taken care of implicitly by the lazy evaluation mechanism (and the algorithmic adjustment where each comparing node always lets through the first number from the left without demanding a number from the right, because it is guaranteed to be bigger anyway).

Odds can be used instead of odd primes as seeds for multiples generation, to simplify things (with obvious performance implications).

The work can naturally be divided into segments between consecutive primes' squares. Haskell code follows, but we can regard it as an executable pseudocode too (where : is a list node lazy constructor, a function call f(x) is written f x, and parentheses are used for grouping only):

primes = 2 : g []
  where
    g ps = 3 : minus [5,7..] (_U [[p*p, p*p+2*p..] | p <- g ps])
    _U ((x:xs):t) = x : union xs (_U (pairs t))
    pairs ((x:xs):ys:t) = (x : union xs ys) : pairs t

union (x:xs) (y:ys) = case compare x y of 
    LT -> x : union  xs (y:ys) 
    EQ -> x : union  xs    ys 
    GT -> y : union (x:xs) ys

minus (x:xs) (y:ys) = case compare x y of
    LT -> x : minus  xs (y:ys) 
    EQ ->     minus  xs    ys 
    GT ->     minus (x:xs) ys

A discussion is here. More sophisticated, lazier scheduling is here. Also this SO answer shows approximate translation of (related) Haskell code in terms of generators; this one in Python.

Will Ness
  • 62,652
  • 8
  • 86
  • 167
  • just a side note the complexity of this would be big O (n ln(ln(n)) ? – pyCthon Sep 25 '12 at 01:39
  • 1
    I don't think so, no. `n log(log n)` is predicated on the ability to use value as *address*, to mark in O(1) time the entry corresponding to a multiple, as composite. But here the values are compared and duplicates removed. That's like the distinction between integer sorting and comparison sorts, which carry (at least) the extra `log n` factor, always. [Empirically](http://en.wikipedia.org/wiki/Analysis_of_algorithms#Empirical_orders_of_growth) it [runs at `~ n^1.2 .. 1.25`](http://ideone.com/p0e81) (see _TMWE_ entry there) to produce first few millions of primes, and worsening. – Will Ness Sep 25 '12 at 11:36
  • 1
    ... but working in segments between squares of consecutive primes, using direct-access arrays for the "merging", it should be close to `n log(log n)`. But then it looses its "tree" structure - _one_ array would have to be used in such setting, to achieve the speedup. – Will Ness Sep 25 '12 at 11:38
  • 1
    @pyCthon correction: one array per each segment would have to be used, but the work could be done for several segments at once, in parallel, following the "segmented" sieve of Eratosthenes setup. That way the overall complexity could be reduced. The cost of sieving each separate segment from `a` to `b`, when all primes are already available, is `~ (b-a) log(log(b)/2)`, and the cost of calculating the starting points for all multiples streams per segment is `~ pi(sqrt(b)) ~ 2 sqrt(b) / log(b)`. – Will Ness Sep 28 '12 at 11:16