Python 3: send method of generators

Question

I can't understand the send method. I understand that it is used to operate the generator. But the syntax is here: generator.send(value).

I somehow can't catch why the value should become the result of the current yield expression. I prepared an example:

def gen():
    for i in range(10):
        X = yield i
        if X == 'stop':
            break
        print("Inside the function " + str(X))

m = gen()
print("1 Outside the function " + str(next(m)) + '\n')
print("2 Outside the function " + str(next(m)) + '\n')
print("3 Outside the function " + str(next(m)) + '\n')
print("4 Outside the function " + str(next(m)) + '\n')
print('\n')
print("Outside the function " + str(m.send(None)) + '\n') # Start generator
print("Outside the function " + str(m.send(77)) + '\n')
print("Outside the function " + str(m.send(88)) + '\n')
#print("Outside the function " + str(m.send('stop')) + '\n')
print("Outside the function " + str(m.send(99)) + '\n')
print("Outside the function " + str(m.send(None)) + '\n')

The result is:

1 Outside the function 0

Inside the function None
2 Outside the function 1

Inside the function None
3 Outside the function 2

Inside the function None
4 Outside the function 3



Inside the function None
Outside the function 4

Inside the function 77
Outside the function 5

Inside the function 88
Outside the function 6

Inside the function 99
Outside the function 7

Inside the function None
Outside the function 8

Well, frankly speaking, it is astonishing me.

1. In the documentation we can read that when a yield statement is executed, the state of the generator is frozen and the value of expression_list is returned to next‘s caller. Well, it doesn't seem to have happened. Why can we execute if statement and print function inside gen().
2. How can I understand why X inside and outside the function differs? Ok. Let us assume that send(77) transmits 77 into m. Well, yield expression becomes 77. Then what is X = yield i? And how 77 inside the function converts into 5 when occurs outside?
3. Why the first result string doesn't reflect anything that is going on inside the generator?

Anyway, could you somehow comment on these send and yield statements?

Answer 1

When you use send and expression yield in a generator, you're treating it as a coroutine; a separate thread of execution that can run sequentially interleaved but not in parallel with its caller.

When the caller executes R = m.send(a), it puts the object a into the generator's input slot, transfers control to the generator, and waits for a response. The generator receives object a as the result of X = yield i, and runs until it hits another yield expression e.g. Y = yield j. Then it puts j into its output slot, transfers control back to the caller, and waits until it gets resumed again. The caller receives j as the result of R = m.send(a), and runs until it hits another S = m.send(b) statement, and so on.

R = next(m) is just the same as R = m.send(None); it's putting None into the generator's input slot, so if the generator checks the result of X = yield i then X will be None.

As a metaphor, consider a dumb waiter:

When the server gets an order from a customer, they put the pad in the dumb waiter, send it to the kitchen, and wait by the hatch for the dish:

R = kitchen.send("Ham omelette, side salad")

The chef (who's been waiting by the hatch) picks up the order, prepares the dish, yields it to the restaurant, and waits for the next order:

next_order = yield [HamOmelette(), SideSalad()]

The server (who's been waiting by the hatch) takes the dish to the customer and returns with another order, etc.

Because both the server and chef wait by the hatch after sending an order or yielding a dish, there's only one person doing anything at any one time i.e. the process is single threaded. Both sides can use normal control flow, as the generator machinery (the dumb waiter) takes care of interleaving execution.

Answer 2

The most confusing part should be this line X = yield i, specially when you call send() on the generator. Actually the only thing you need to know is:

in the lexical level: next() is equal to send(None)

in the interpreter level: X = yield i equals to below lines(ORDER MATTERS):

yield i
# won't continue until next() or send() is called
# and this is also the entry point of next() or send()
X = the_input_of_send

and, the 2 lines of comment is the exact reason, why we need to call send(None) for the first time, because the generator will return i (yield i) before assign the value to X

Answer 3

def gen():
    i = 1
    while True:
        i += 1
        x = yield i
        print(x)

m = gen()
next(m)
next(m)
m.send(4)

result

None
4

look at more simplified codes above.
I think the thing leaded to your confusion is 'x = yield i' statment, this statment is not saying value accepted from send() method assgined to i then i assgined to x. Instead, value i is returned by yield statment to generator, x is assgined by send() method.One statement does two thing at same time.

Answer 4

Since you asked even for comments, consider the following case:

def lambda_maker():
    def generator():
        value = None
        while 1:
            value = yield value
            value= value[0][1]
    f = generator()
    next(f)  # skip the first None
    return f.send  # a handy lambda value: value[0][1]

Now the following two lines are equivalent:

a_list.sort(key=lambda a: a[0][1])
a_list.sort(key=lambda_maker())

(Incidentally, in the current (2018-05-26, day 1 post-GDPR ☺) CPython2 and CPython3 implementations, the second line runs faster than the first, but that's a detail related to frame object initialization overhead on every function call.)

What happens here? lambda_maker calls f=generator() and gets a generator; calling the initial next(f) starts running the generator and consumes the initial None value, and pauses at the yield line. Then it returns the bound method f.send to its caller. From this point on, every time this bound method is called, the generator.value local receives the argument of the bound method, recalculates value and then loops back yielding the current value of value and waits for the next .send to get another value.

The generator object stays in-memory and all it does in the loop is:

• yield the current result (initially None)
• receive another value (whatever someone used as argument to .send)
• re-calculate current result based on received value
• loop back

Answer 5

Note:
For simplicity is my answer restricted to the case when the generator has at most 1 yield commands in every line.

TL;DR:

• The .send() method:

  • sends a value to the currently suspended command, but
  • receives a value from the next, oncoming suspended command

• Sent value is “receives” the yield expression itself.
  It means that the expression yield 7

  • yields value 7, but
  • its own value, i.e. the value of the (yield 7), may be for example "hello".
    (Parentheses are generally mandatory except the simplest cases.)

---

In detail:

Preface:

Let g = gen() is an instance of a generator iterator gen().

• The command next(g) behaves exactly as g.send(None).

  • So in the next explanation I will fully omit the next() function.

• Sending a not-None value is only allowed if the instance g is suspended at the statement with the yield command:
  

  • Why? Because the .send() method sends a value directly to the suspended yield expression (see the point 4. in the “Step by step” section bellow).

  So before sending a not-None value, we must put the generator instance to such a suspended state by sending to it the None value. It may be as simple as g.send(None):

• But just before g becomes suspended, it yields the value of the yield command. This yielded value becomes the return value of the .send() method:

  We probably want to use this received value or save it in a variable for the later use, so instead of the two previous pictures let's start our journey with this:

---

Step by step:

1. The first .send() starts the instance g. Instance g begin performing its commands up to the first yield statement, which yields its value:

   It means, that in the variable from_iterator_1 will be the string "first_from_iterator".

    

2. Now, after yielding its first value, we have g in the suspended state

   which allows us sending to g something useful, other than None — e.g. the number 1.

    

3. So let's send the number 1 to g:

    

4. As g was suspended at the expression yield "first_from_iterator", the value of this expression (itself) will become 1.

   (Yes, the yield "first_from_iterator" is an expression, likewise a + b is.)

   Recall that at this moment the value "first_from_iterator" is long time ago already yielded.

    

5. The instance g then wakes up, and — in turn — the g.send() now waits for a returned value.

    

6. The previously suspended, now woken statement will be performed.
   (Before the suspension, it was not performed, it only yielded a value.) In our simple case (the woken statement is yield "first_from_iterator") there remains nothing to perform, but what about

   • saving the sent value (1) to a variable for the later use instead?

     received_1 = yield "first_from_iterator"      
     

   • or performing a more complicated computation with it instead?

     result = math.cos((yield "first_from_iterator") * math.pi)         # result: -1
     

    

7. All consequent statements in g will be performed, but only up to the next statement with the yield command in it.

    

8. That next statement (with the yield command in it) yields a value

   which suspends g again and wakes up the waiting .send() method (by providing it the expected — yielded — return value).

    

9. It allows performing next commands after it:

    

10. Now we are in the same situation as in the point 2. — just before performing the (next) .send() method — so the story will be repeated.

    Note:
    It will be repeated with the same issue as in the last point of the “Preface” section above — we probably don't want to throw out the yielded value, so instead of the commands

    g.send(1)                          # Not very appropriate
    

    is better to use something as

    from_iterator_2 = g.send(1)        # Saving the 2nd yielded value
    

    (and similarly for the next g.send(2) command).

---

The big picture:

(The first send (with the None parameter) starts the generator instance performing its commands.)