How to solve this memory error - python

Question

I have the following code

    def gen_primes():

        D = {}  

        q = 2  

        while True:
            if q not in D:         
                yield q        
                D[q * q] = [q]
            else:           
                for p in D[q]:
                    D.setdefault(p + q, []).append(p)
                del D[q]

            q += 1


    f = open("primes1.txt","w")

    filen = 1
    ran1 = 1
    ran2 = 10000000

    k = 1
    for i in gen_primes():

        if (k >= ran1) and (k <= ran2):

            f.write(str(i) + "\n")
            if k%1000000 == 0:
                print k
            k = k + 1
        else:
            ran1 = ran2 + 1
            ran2 = ran2 + 10000000
            f.close()
            filen = filen + 1;
            f = open("primes" + str(filen) + ".txt","w")

        if k > 100000000:           
            break
    f.close()

The prime generation algorithm is taken from Simple Prime Generator in Python

This program is giving memory error

Traceback (most recent call last):
  File "C:\Python25\Projects\test.py", line 43, in <module>
    for i in gen_primes():
  File "C:\Python25\Projects\test.py", line 30, in gen_primes
    D.setdefault(p + q, []).append(p)
MemoryError

I am trying to store consecutive 10,000,000 primes in one file.

See also that question: http://stackoverflow.com/questions/2211990/ — tzot, Oct 26 '11 at 14:00

Omnifarious · Accepted Answer · 2011-05-16T15:00:00.397

1

This prime generator doesn't use very much memory. It's also not very fast.

def gcd(a, b):
    rem = a % b
    while rem != 0:
        a = b
        b = rem
        rem = a % b
    return b

def primegen():
    yield 2
    yield 3
    yield 5
    yield 7
    yield 11
    accum = 2*3*5*7
    out = file('tmp_primes.txt', 'w')
    inp = file('tmp_primes.txt', 'r+')
    out.write('0x2\n0x3\n0x5\n0x7\n0xb\n')
    inp.read(20)
    inpos = inp.tell()
    next_accum = 11
    next_square = 121
    testprime = 13
    while True:
        if gcd(accum, testprime) == 1:
            accum *= testprime # It's actually prime!
            out.writelines((hex(testprime), '\n'))
            yield testprime
        testprime += 2
        if testprime >= next_square:
            accum *= next_accum
            nextline = inp.readline()
            if (len(nextline) < 1) or (nextline[-1] != '\n'):
                out.flush()
                inp.seek(inpos)
                nextline = inp.readline()
            inpos = inp.tell()
            next_accum = int(nextline, 16)
            next_square = next_accum * next_accum

def next_n(iterator, n):
    """Returns the next n elements from an iterator.

    >>> list(next_n(iter([1,2,3,4,5,6]), 3))
    [1, 2, 3]
    >>> list(next_n(primegen(), 10))
    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
    """
    while n > 0:
        yield iterator.next()
        n -= 1

edited May 16 '11 at 15:00

answered May 16 '11 at 06:06

Omnifarious

50,447
15
117
181

This can be improved greatly if one is willing to be a bit clever about not multiplying a prime into the accumulator until its absolutely necessary and using a temporary file that holds a list of the primes discovered so far so the prime to be multiplied into the accumulator next can be read in easily without storing them all in memory. – Omnifarious May 16 '11 at 06:52
You are using Euclidean algorithm for gcd.Can u tell me how can I use this code for prime number generation.I couldn't understand the working of next_n. – Vinod May 16 '11 at 07:27
@Vinod: The `primegen` function is a generator, and will generate an infinite number of prime numbers, like `gen_primes` in your example code. The `next_n` function is a generator that simply returns the next `n` elements from an iterator. It's very useful if you have an infinite length iterator but only want 100 elements from it. – Omnifarious May 16 '11 at 07:29
@Vinod: I modified it slightly to try to make it clearer exactly what's going on. It might help to know that `next_n` is purely a useful utility function, and not at all necessary to the generation of primes. – Omnifarious May 16 '11 at 07:41
@Vinod: I've modified `primegen` as I talked about in my comments. It should now be faster and use less memory. It accomplishes this by by storing the list of generated primes in a file and reading them in as needed to update the 'accumulator', which is sort of a list of primes to be checked against. – Omnifarious May 16 '11 at 15:02

Federico Vera · Answer 2 · 2011-05-16T03:12:02.537

0

Try using this generator: http://code.activestate.com/recipes/366178-a-fast-prime-number-list-generator/

Is incredibly fast (10000000 primes in a couple of seconds) and not that memory consuming

to save in a file perhaps it's easier simply to do something like:

interval_start = 100
interval_length = 10000000
f = open("primes1.txt","w")

for prime in primes(interval_start + interval_length)[interval_start::]:
   f.write(str(prime) + "\n")

f.close()

edited May 16 '11 at 03:12

answered May 16 '11 at 02:59

Federico Vera

1,318
1
12
17

I need first 100 million primes.This is causing memory problems. – Vinod May 16 '11 at 03:14
just add an extra zero to interval_lenght, I just test it and it created a nice 48,7Mb file with 0.1e^9 lines... – Federico Vera May 16 '11 at 03:21
@Vinod, how large is your memory？It works well in my computer with 2GB memory. – xiao 啸 May 16 '11 at 06:16
@xiao My laptop has 2.5GB memory. – Vinod May 16 '11 at 07:33
@Vinod are you using the 64-bit version of python? I tried the code in a 32-bit python installation and caught a MemoryError, although it works fine under python 64 – Federico Vera May 16 '11 at 12:25

score 0 · Answer 3 · answered May 16 '11 at 03:55

I've run slightly modified code on my machine 10^6 primes in ~30 sec (I am running Python 3.2)

Here is the code:

def gen_primes():  
    D = {}  
    # The running integer that's checked for primeness
    q = 2  

    while True:
        if q not in D:
            yield q        
            D[q * q] = [q]
        else:
            for p in D[q]:
                D.setdefault(p + q, []).append(p)
            del D[q]

        q += 1


def main():
    j = 0
    f = open("primes1.txt","w")
    for i in gen_primes():
        j += 1
        #print(j, i)
        f.write(str(i) + "\n")
        if (j > 10000000): break
    f.close()

if __name__ == "__main__":
    main()

I cannot generate 100million primes with this. The memory problem persists. — Vinod, May 16 '11 at 11:17

score 0 · Answer 4 · answered May 16 '11 at 07:05

0

Install the package gmpy and you can write the file without requiring much memory at all

import gmpy
p=2
with open("primes.txt","w") as f:
    for n in xrange(100000000):
        print >> f, p
        p = gmpy.next_prime(p)

answered May 16 '11 at 07:05

John La Rooy

263,347
47
334
476

I have problems in installing gmpy in Windows.I have installed gmp also. – Vinod May 16 '11 at 11:16

How to solve this memory error - python

4 Answers4