How to make itertools.permutations() to write each example on a file by modifying the code - Python

Question

This is the code of itertools.permutations:

def permutations(iterable, r=None):
    # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
    # permutations(range(3)) --> 012 021 102 120 201 210
    pool = tuple(iterable)
    n = len(pool)
    r = n if r is None else r
    if r > n:
        return
    indices = list(range(n))
    cycles = list(range(n, n-r, -1))
    yield tuple(pool[i] for i in indices[:r])
    while n:
        for i in reversed(range(r)):
            cycles[i] -= 1
            if cycles[i] == 0:
                indices[i:] = indices[i+1:] + indices[i:i+1]
                cycles[i] = n - i
            else:
                j = cycles[i]
                indices[i], indices[-j] = indices[-j], indices[i]
                yield tuple(pool[i] for i in indices[:r])
                break
        else:
            return

I want it instead of adding the permutations into a list, to write each permutation it finds to a .txt file immediately not by looping the list and writing them into the file.

Okay, cool. What's stopping you from doing that? Do you know how to open files? Do you know how to write to them? Please read [ask]. — Chris, Nov 18 '18 at 15:55
the code is a `copy&paste` from https://docs.python.org/3/library/itertools.html#itertools.permutations ... what did you code? — Patrick Artner, Nov 18 '18 at 15:58
@mkrieger1 hoppla - youre right ... r = 0 wouldn't make any sense though - but it should return [] and not them perms of n. — Patrick Artner, Nov 18 '18 at 15:58
I don't know, but I think the Python devs did know what they were doing. — mkrieger1, Nov 18 '18 at 16:00

timgeb · Answer 1 · 2018-11-18T17:06:14.330

3

The whole point of itertools is that values are produced lazily and not as lists. You don't need to change the code of permutations, all you need to do is loop over the permutations as they are produced and then write them to a file.

for p in permutations(your_iterable):
    # your code to write p to a file here

Note that at no point there's a list involved in this code. permutations(your_iterable) is an iterator you can explicitly pull values out of with next or implicitly by employing a for loop.

it is a very large amount of permutations and causes a memory error

I am trying to tell you that permutations(your_iterable) can never produce a memory error in any non-artificially-contrived scenario if you don't explicitly make a list out of permutations(your_iterable) yourself. The iterator produces one permutation at a time which you can write to a file without ever holding all permutations in RAM.

I am passing a list of length 110 to itertools.permuations

This is not a problem with the standard library or values in RAM.

Your code will produce

15882455415227429404253703127090772871724410234473563207581748318444567162948183030959960131517678520479243672638179990208521148623422266876757623911219200000000000000000000000000

permutations. There is no file system on the planet with enough storage space to write these permutations to.

Also, there's not enough time left until the heat death of the universe to compute all these permutations, even if you could compute one in a nanosecond.

>>> from math import factorial
>>> factorial(110)/(10E9*3600*24*365)
>>> 5.0362935740827724e+160 # years

4 is the second parameter for permutations

Ok, in that case your file would require around 1.7 GiB of space if you stored each permutation as a string of minimum length 13 akin to '(0, 1, 2, 3)\n' with one byte per character.

>>> perms = sum(1 for _ in permutations(list(range(110)), 4))
>>> perms*13/(2**30)
>>> 1.677529625594616

That's manageable - but it could require much more space if the objects in your list had longer string representations. We can't tell you why you get a memory error without seeing your code.

edited Nov 18 '18 at 17:06

answered Nov 18 '18 at 15:56

timgeb

64,821
18
95
124

I want to write them when they are produced not by looping. – Johnny P. Nov 18 '18 at 15:59
1

@JohnnyP. this code writes each permutation to a file as soon as it is produced. "Not by looping" makes no sense here, you have to have a loop somewhere if you want to do the same thing (writing) for multiple other things (permutations). – timgeb Nov 18 '18 at 16:00
@JohnnyP., it looks like this is an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). Why are you preoccupied with whether there's a loop or not? What are you _actually_ trying to accomplish? – Chris Nov 18 '18 at 16:04
@JohnnyP. https://stackoverflow.com/questions/1756096/understanding-generators-in-python – mkrieger1 Nov 18 '18 at 16:04
Because it is a very large amount of permutations and causes a memory error. – Johnny P. Nov 18 '18 at 16:05
@JohnnyP., then you should be posting _your code_ and asking about _the memory error_. This isn't a problem with Python's standard library code, it's a problem with _your_ code. – Chris Nov 18 '18 at 16:08
It reurns a memory error because there are billions of permutations. – Johnny P. Nov 18 '18 at 16:09
@JohnnyP., not because of the code you're asking about. It's a problem with how you're using the standard library, which you aren't showing us. – Chris Nov 18 '18 at 16:09
@JohnnyP. well has your hard drive enough space for billions of permutations? ... – timgeb Nov 18 '18 at 16:09
How much space do I want? – Johnny P. Nov 18 '18 at 16:11
...how should _we_ know how much space you want? Again, you _haven't shown us **anything**_ useful to understand your problem. Show us _your_ code, tell us about your data set, ask about the memory error, and then maybe we can help you. – Chris Nov 18 '18 at 16:12
@timgeb you wrote 'The iterator produces one permutation at a time which you can write to a file without ever holding all permutations in RAM.', but how? – Johnny P. Nov 18 '18 at 16:12
I use itertools.permutations(a list of lenght 110, 4) – Johnny P. Nov 18 '18 at 16:14
1

@JohnnyP., I'm going to say this one more time, then I'm going to leave. **You _must_ show us _your_ actual code.** "A list of length 110, 4" doesn't make any sense. Lists have one length, not two. Are you actually doing something with numpy arrays? We literally _**cannot** help you_ the way you're asking. The problem isn't in the standard library. Please read [ask], then try again. – Chris Nov 18 '18 at 16:16
4 is the second parameter for permutations. – Johnny P. Nov 18 '18 at 16:18
1

@JohnnyP. "but how" – timgeb Nov 18 '18 at 16:19
@JohnnyP. are you aware that you are trying to store factorial(110) permutations? – timgeb Nov 18 '18 at 16:32
@timgeb The second parameter of permutation is 4 so I want to store 110*109*108*107. – Johnny P. Nov 18 '18 at 17:00

How to make itertools.permutations() to write each example on a file by modifying the code - Python

1 Answers1