This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines. It is basically a combination of ideas from these two PEPs, which may be considered redundant if this PEP is accepted:
Coroutines are a natural way of expressing many algorithms, such as simulations, games, asynchronous I/O, and other forms of event-driven programming or co-operative multitasking. Python’s generator functions are almost coroutines – but not quite – in that they allow pausing execution to produce a value, but do not provide for values or exceptions to be passed in when execution resumes. They also do not allow execution to be paused within the try portion of try/finally blocks, and therefore make it difficult for an aborted coroutine to clean up after itself.
Also, generators cannot yield control while other functions are executing, unless those functions are themselves expressed as generators, and the outer generator is written to yield in response to values yielded by the inner generator. This complicates the implementation of even relatively simple use cases like asynchronous communications, because calling any functions either requires the generator to block (i.e. be unable to yield control), or else a lot of boilerplate looping code must be added around every needed function call.
However, if it were possible to pass values or exceptions into a generator at the point where it was suspended, a simple co-routine scheduler or trampoline function would let coroutines call each other without blocking – a tremendous boon for asynchronous applications. Such applications could then write co-routines to do non-blocking socket I/O by yielding control to an I/O scheduler until data has been sent or becomes available. Meanwhile, code that performs the I/O would simply do something like this:
in order to pause execution until the nonblocking_read() coroutine produced a value.
In other words, with a few relatively minor enhancements to the language and to the implementation of the generator-iterator type, Python will be able to support performing asynchronous operations without needing to write the entire application as a series of callbacks, and without requiring the use of resource-intensive threads for programs that need hundreds or even thousands of co-operatively multitasking pseudothreads. Thus, these enhancements will give standard Python many of the benefits of the Stackless Python fork, without requiring any significant modification to the CPython core or its APIs. In addition, these enhancements should be readily implementable by any Python implementation (such as Jython) that already supports generators.
By adding a few simple methods to the generator-iterator type, and with two minor syntax adjustments, Python developers will be able to use generator functions to implement co-routines and other forms of co-operative multitasking. These methods and adjustments are:
A prototype patch implementing all of these changes against the current Python CVS HEAD is available as SourceForge patch #1223381 (https://bugs.python.org/issue1223381).
A new method for generator-iterators is proposed, called send(). It takes exactly one argument, which is the value that should be sent in to the generator. Calling send(None) is exactly equivalent to calling a generator’s next() method. Calling send() with any other value is the same, except that the value produced by the generator’s current yield expression will be different.
Because generator-iterators begin execution at the top of the generator’s function body, there is no yield expression to receive a value when the generator has just been created. Therefore, calling send() with a non-None argument is prohibited when the generator iterator has just started, and a TypeError is raised if this occurs (presumably due to a logic error of some kind). Thus, before you can communicate with a coroutine you must first call next() or send(None) to advance its execution to the first yield expression.
As with the next() method, the send() method returns the next value yielded by the generator-iterator, or raises StopIteration if the generator exits normally, or has already exited. If the generator raises an uncaught exception, it is propagated to send()’s caller.
The yield-statement will be allowed to be used on the right-hand side of an assignment; in that case it is referred to as yield-expression. The value of this yield-expression is None unless send() was called with a non-None argument; see below.
A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. So
are all legal, but
are all illegal. (Some of the edge cases are motivated by the current legality of yield 12, 42.)
Note that a yield-statement or yield-expression without an expression is now legal. This makes sense: when the information flow in the next() call is reversed, it should be possible to yield without passing an explicit value (yield is of course equivalent to yield None).
When send(value) is called, the yield-expression that it resumes will return the passed-in value. When next() is called, the resumed yield-expression will return None. If the yield-expression is a yield-statement, this returned value is ignored, similar to ignoring the value returned by a function call used as a statement.
In effect, a yield-expression is like an inverted function call; the argument to yield is in fact returned (yielded) from the currently executing function, and the return value of yield is the argument passed in via send().
Note: the syntactic extensions to yield make its use very similar to that in Ruby. This is intentional. Do note that in Python the block passes a value to the generator using send(EXPR) rather than return EXPR, and the underlying mechanism whereby control is passed between the generator and the block is completely different. Blocks in Python are not compiled into thunks; rather, yield suspends execution of the generator’s frame. Some edge cases work differently; in Python, you cannot save the block for later use, and you cannot test whether there is a block or not. (XXX - this stuff about blocks seems out of place now, perhaps Guido can edit to clarify.)
Let a generator object be the iterator produced by calling a generator function. Below, g always refers to a generator object.
The syntax for generator functions is extended to allow a yield-statement inside a try-finally statement.
g.throw(type, value, traceback) causes the specified exception to be thrown at the point where the generator g is currently suspended (i.e. at a yield-statement, or at the start of its function body if next() has not been called yet). If the generator catches the exception and yields another value, that is the return value of g.throw(). If it doesn’t catch the exception, the throw() appears to raise the same exception passed it (it falls through). If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw() call. In summary, throw() behaves like next() or send(), except it raises an exception at the suspension point. If the generator is already in the closed state, throw() just raises the exception it was passed without executing any of the generator’s code.
The effect of raising the exception is exactly as if the statement:
was executed at the suspension point. The type argument must not be None, and the type and value must be compatible. If the value is not an instance of the type, a new exception instance is created using the value, following the same rules that the raise statement uses to create an exception instance. The traceback, if supplied, must be a valid Python traceback object, or a TypeError occurs.
Note: The name of the throw() method was selected for several reasons. Raise is a keyword and so cannot be used as a method name. Unlike raise (which immediately raises an exception from the current execution point), throw() first resumes the generator, and only then raises the exception. The word throw is suggestive of putting the exception in another location, and is already associated with exceptions in other languages.
Alternative method names were considered: resolve(), signal(), genraise(), raiseinto(), and flush(). None of these seem to fit as well as throw().
A new standard exception is defined, GeneratorExit, inheriting from Exception. A generator should handle this by re-raising it (or just not catching it) or by raising StopIteration.
g.close() is defined by the following pseudo-code:
g.__del__() is a wrapper for g.close(). This will be called when the generator object is garbage-collected (in CPython, this is when its reference count goes to zero). If close() raises an exception, a traceback for the exception is printed to sys.stderr and further ignored; it is not propagated back to the place that triggered the garbage collection. This is consistent with the handling of exceptions in __del__() methods on class instances.
If the generator object participates in a cycle, g.__del__() may not be called. This is the behavior of CPython’s current garbage collector. The reason for the restriction is that the GC code needs to break a cycle at an arbitrary point in order to collect it, and from then on no Python code should be allowed to see the objects that formed the cycle, as they may be in an invalid state. Objects hanging off a cycle are not subject to this restriction.
Note that it is unlikely to see a generator object participate in a cycle in practice. However, storing a generator object in a global variable creates a cycle via the generator frame’s f_globals pointer. Another way to create a cycle would be to store a reference to the generator object in a data structure that is passed to the generator as an argument (e.g., if an object has a method that’s a generator, and keeps a reference to a running iterator created by that method). Neither of these cases are very likely given the typical patterns of generator use.
Also, in the CPython implementation of this PEP, the frame object used by the generator should be released whenever its execution is terminated due to an error or normal exit. This will ensure that generators that cannot be resumed do not remain part of an uncollectable reference cycle. This allows other code to potentially use close() in a try/finally or with block (per PEP 343) to ensure that a given generator is properly finalized.
An earlier draft of this PEP proposed a new continue EXPR syntax for use in for-loops (carried over from PEP 340), that would pass the value of EXPR into the iterator being looped over. This feature has been withdrawn for the time being, because the scope of this PEP has been narrowed to focus only on passing values into generator-iterators, and not other kinds of iterators. It was also felt by some on the Python-Dev list that adding new syntax for this particular feature would be premature at best.
Discussion on python-dev has revealed some open issues. I list them here, with my preferred resolution and its motivation. The PEP as currently written reflects this preferred resolution.
I originally chose TypeError because it represents gross misbehavior of the generator function, which should be fixed by changing the code. But the with_template decorator class in PEP 343 uses RuntimeError for similar offenses. Arguably they should all use the same exception. I’d rather not introduce a new exception class just for this purpose, since it’s not an exception that I want people to catch: I want it to turn into a traceback which is seen by the programmer who then fixes the code. So now I believe they should both raise RuntimeError. There are some precedents for that: it’s raised by the core Python code in situations where endless recursion is detected, and for uninitialized objects (and for a variety of miscellaneous conditions).
However, looking more closely at the consumer interface, it seems that the desired semantics for feed() are different than for send(), because send() can’t be meaningfully called on a just-started generator. Also, the consumer interface as currently defined doesn’t include handling for StopIteration.
Therefore, it seems like it would probably be more useful to create a simple decorator that wraps a generator function to make it conform to the consumer interface. For example, it could warm up the generator with an initial next() call, trap StopIteration, and perhaps even provide reset() by re-invoking the generator function.
A prototype patch implementing all of the features described in this PEP is available as SourceForge patch #1223381 (https://bugs.python.org/issue1223381).
This patch was committed to CVS 01-02 August 2005.
Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first formally proposed the ideas of communicating values or exceptions into generators, and the ability to close generators. Timothy Delaney suggested the title of this PEP, and Steven Bethard helped edit a previous version. See also the Acknowledgements section of PEP 340.
TBD.
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0342.rst
Last modified: 2025-02-01 08:59:27 UTC