6

In the Effective Python book, the author recommends using assignment expressions to avoid redundancy in comprehensions, for example:

def fun(i):
    return 2 * i

result = {x: y for x in [0, 1, 2, 3] if (y := fun(x)) > 3}

instead of

result = {x: fun(x) for x in [0, 1, 2, 3] if fun(x) > 3}

result has the value {2: 4, 3: 6}.

The author states that

If a comprehension uses the walrus operator in the value part of the comprehension and doesn’t have a condition, it’ll leak the loop variable into the containing scope. [...] It’s better not to leak loop variables, so I recommend using assignment expressions only in the condition part of a comprehension.

However, in the example above, y is set to 6 at the end of the program. So, the variable in the assignment expression leaked, although it is defined in the condition.

The same thing happens for list comprehensions:

>>> _ = [(x, leak) for x in range(4) if (leak := 2 * x) > 3]
>>> leak
6

And even for generator expressions:

>>> it = ((x, leak) for x in range(4) if (leak := 2 * x) > 3)
>>> next(it)
(2, 4)
>>> leak
4
>>> next(it)
(3, 6)
>>> leak
6

What am I missing? Is there any way to avoid leaking in assignment expressions in comprehensions at all?

Kilian Batzner
  • 5,420
  • 2
  • 32
  • 44
  • 2
    Python doesn't have block scoping for variables, the only scopes are function and class definitions. So the explanation in the text doesn't really make sense to me. – Barmar Oct 04 '20 at 09:29
  • Generators and list comprehensions don't establish a new scope, either. – Barmar Oct 04 '20 at 09:31
  • 1
    This is old, but there is some updates regarding python 3.X: https://stackoverflow.com/questions/4198906/list-comprehension-rebinds-names-even-after-scope-of-comprehension-is-this-righ – David S Oct 04 '20 at 09:33
  • @Barmar In fact they do: `x = 1; print([locals() for y in range(1)])` prints `y` and does not print `x` from the outer scope. – kaya3 Oct 07 '20 at 15:51
  • @kaya3 You're right. More to the point: `[y for y in range(1)]` doesn't leak `y` out to the containing scope. I didn't realize that until your comment – Barmar Oct 07 '20 at 17:34

1 Answers1

0

In Python, it's impossible not to leak loop variables

Unlike other languages such as C or Java, Python has no separate scope within if and for blocks. So when you use the := operator in an if statement, a for loop, or a list comprehension, the assigned variable will be in scope throughout the remainder of the function or class definition. This also means that after every for loop, the loop variable will still be in scope and contain the value of the loop's last iteration.

I disagree with the author of of Effective Python if he thinks that's a bad thing. "Leaking" loop variables can be very useful! Consider the following example:

while line := f.readLine():
    if 'Kilian' in line:
        break

print('This is the first line that contains your name: ', line)

However, there is one exception to this rule: implicit assignments made in list comprehensions have their own scope:

>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

This exception is probably where your confusion stems from. It's a special case only, and not applicable when using the := inside a list comprehension.

Jaap Joris Vens
  • 2,552
  • 1
  • 16
  • 30
  • 1
    This is not wrong in terms of the code's behaviour, but conceptually it is misleading. Comprehensions are evaluated in their own stack frames, so "implicit assignments" aren't leaked for the same reason local variables in functions aren't leaked to the caller. You can check this by calling `locals()` inside a comprehension. The walrus operator requires a special case to that behaviour in order to deliberately leak the variable. – kaya3 Oct 07 '20 at 15:42