The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic.
It is proposed to make coroutines a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible.
This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses yield as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed.
We believe that the changes proposed here will help keep Python relevant and competitive in a quickly growing area of asynchronous programming, as many other languages have adopted, or are planning to adopt, similar features: [2], [5], [6], [7], [8], [10].
This change was implemented based primarily due to problems encountered attempting to integrate support for native coroutines into the Tornado web server (reported in [18]).
Before 3.5.2, __aiter__ was expected to return an awaitable resolving to an asynchronous iterator. Starting with 3.5.2, __aiter__ should return asynchronous iterators directly.
If the old protocol is used in 3.5.2, Python will raise a PendingDeprecationWarning.
In CPython 3.6, the old __aiter__ protocol will still be supported with a DeprecationWarning being raised.
In CPython 3.7, the old __aiter__ protocol will no longer be supported: a RuntimeError will be raised if __aiter__ returns anything but an asynchronous iterator.
Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the yield from syntax introduced in PEP 380. This approach has a number of shortcomings:
This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring.
Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new async with statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new async for statement makes it possible to perform asynchronous calls in iterators.
This proposal introduces new syntax and semantics to enhance coroutine support in Python.
This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the “Cofunctions” proposal (PEP 3152, now rejected in favor of this specification).
From this point in this document we use the word native coroutine to refer to functions declared using the new syntax. generator-based coroutine is used where necessary to refer to coroutines that are based on generator syntax. coroutine is used in contexts where both definitions are applicable.
The following new syntax is used to declare a native coroutine:
Key properties of coroutines:
A new function coroutine(fn) is added to the types module. It allows interoperability between existing generator-based coroutines in asyncio and native coroutines introduced by this PEP:
The function applies CO_ITERABLE_COROUTINE flag to generator-function’s code object, making it return a coroutine object.
If fn is not a generator function, it is wrapped. If it returns a generator, it will be wrapped in an awaitable proxy object (see below the definition of awaitable objects).
Note, that the CO_COROUTINE flag is not applied by types.coroutine() to make it possible to separate native coroutines defined with new syntax, from generator-based coroutines.
The following new await expression is used to obtain a result of coroutine execution:
await, similarly to yield from, suspends execution of read_data coroutine until db.fetch awaitable completes and returns the result data.
It uses the yield from implementation with an extra step of validating its argument. await only accepts an awaitable, which can be one of:
Any yield from chain of calls ends with a yield. This is a fundamental mechanism of how Futures are implemented. Since, internally, coroutines are a special kind of generators, every await is suspended by a yield somewhere down the chain of await calls (please refer to PEP 3156 for a detailed explanation).
To enable this behavior for coroutines, a new magic method called __await__ is added. In asyncio, for instance, to enable Future objects in await statements, the only change is to add __await__ = __iter__ line to asyncio.Future class.
Objects with __await__ method are called Future-like objects in the rest of this PEP.
It is a TypeError if __await__ returns anything but an iterator.
It is a SyntaxError to use await outside of an async def function (like it is a SyntaxError to use yield outside of def function).
It is a TypeError to pass anything other than an awaitable object to an await expression.
await keyword is defined as follows:
where “primary” represents the most tightly bound operations of the language. Its syntax is:
See Python Documentation [12] and Grammar Updates section of this proposal for details.
The key await difference from yield and yield from operators is that await expressions do not require parentheses around them most of the times.
Also, yield from allows any expression as its argument, including expressions like yield from a() + b(), that would be parsed as yield from (a() + b()), which is almost always a bug. In general, the result of any arithmetic operation is not an awaitable object. To avoid this kind of mistakes, it was decided to make await precedence lower than [], (), and ., but higher than ** operators.
| yield x, yield from x | Yield expression |
| lambda | Lambda expression |
| if – else | Conditional expression |
| or | Boolean OR |
| and | Boolean AND |
| not x | Boolean NOT |
| in, not in, is, is not, <, <=, >, >=, !=, == | Comparisons, including membership tests and identity tests |
| | | Bitwise OR |
| ^ | Bitwise XOR |
| & | Bitwise AND |
| <<, >> | Shifts |
| +, - | Addition and subtraction |
| *, @, /, //, % | Multiplication, matrix multiplication, division, remainder |
| +x, -x, ~x | Positive, negative, bitwise NOT |
| ** | Exponentiation |
| await x | Await expression |
| x[index], x[index:index], x(arguments...), x.attribute | Subscription, slicing, call, attribute reference |
| (expressions...), [expressions...], {key: value...}, {expressions...} | Binding or tuple display, list display, dictionary display, set display |
Valid syntax examples:
| if await fut: pass | if (await fut): pass |
| if await fut + 1: pass | if (await fut) + 1: pass |
| pair = await fut, 'spam' | pair = (await fut), 'spam' |
| with await fut, open(): pass | with (await fut), open(): pass |
| await foo()['spam'].baz()() | await ( foo()['spam'].baz()() ) |
| return await coro() | return ( await coro() ) |
| res = await coro() ** 2 | res = (await coro()) ** 2 |
| func(a1=await coro(), a2=0) | func(a1=(await coro()), a2=0) |
| await foo() + await bar() | (await foo()) + (await bar()) |
| -await foo() | -(await foo()) |
Invalid syntax examples:
| await await coro() | await (await coro()) |
| await -coro() | await (-coro()) |
An asynchronous context manager is a context manager that is able to suspend execution in its enter and exit methods.
To make this possible, a new protocol for asynchronous context managers is proposed. Two new magic methods are added: __aenter__ and __aexit__. Both must return an awaitable.
An example of an asynchronous context manager:
A new statement for asynchronous context managers is proposed:
which is semantically equivalent to:
As with regular with statements, it is possible to specify multiple context managers in a single async with statement.
It is an error to pass a regular context manager without __aenter__ and __aexit__ methods to async with. It is a SyntaxError to use async with outside of an async def function.
With asynchronous context managers it is easy to implement proper database transaction managers for coroutines:
Code that needs locking also looks lighter:
instead of:
An asynchronous iterable is able to call asynchronous code in its iter implementation, and asynchronous iterator can call asynchronous code in its next method. To support asynchronous iteration:
An example of asynchronous iterable:
A new statement for iterating through asynchronous iterators is proposed:
which is semantically equivalent to:
It is a TypeError to pass a regular iterable without __aiter__ method to async for. It is a SyntaxError to use async for outside of an async def function.
As for with regular for statement, async for has an optional else clause.
With asynchronous iteration protocol it is possible to asynchronously buffer data during iteration:
Where cursor is an asynchronous iterator that prefetches N rows of data from a database after every N iterations.
The following code illustrates new asynchronous iteration protocol:
then the Cursor class can be used as follows:
which would be equivalent to the following code:
The following is a utility class that transforms a regular iterable to an asynchronous one. While this is not a very useful thing to do, the code illustrates the relationship between regular and asynchronous iterators.
Coroutines are still based on generators internally. So, before PEP 479, there was no fundamental difference between
and
And since PEP 479 is accepted and enabled by default for coroutines, the following example will have its StopIteration wrapped into a RuntimeError
The only way to tell the outside code that the iteration has ended is to raise something other than StopIteration. Therefore, a new built-in exception class StopAsyncIteration was added.
Moreover, with semantics from PEP 479, all StopIteration exceptions raised in coroutines are wrapped in RuntimeError.
This section applies only to native coroutines with CO_COROUTINE flag, i.e. defined with the new async def syntax.
The behavior of existing *generator-based coroutines* in asyncio remains unchanged.
Great effort has been made to make sure that coroutines and generators are treated as distinct concepts:
An attempt to use __iter__ or __next__ on a native coroutine object will result in a TypeError.
Coroutines are based on generators internally, thus they share the implementation. Similarly to generator objects, coroutines have throw(), send() and close() methods. StopIteration and GeneratorExit play the same role for coroutines (although PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380, and Python Documentation [11] for details.
throw(), send() methods for coroutines are used to push values and raise errors into Future-like objects.
A common beginner mistake is forgetting to use yield from on coroutines:
For debugging this kind of mistakes there is a special debug mode in asyncio, in which @coroutine decorator wraps all functions with a special object with a destructor logging a warning. Whenever a wrapped generator gets garbage collected, a detailed logging message is generated with information about where exactly the decorator function was defined, stack trace of where it was collected, etc. Wrapper object also provides a convenient __repr__ function with detailed information about the generator.
The only problem is how to enable these debug capabilities. Since debug facilities should be a no-op in production mode, @coroutine decorator makes the decision of whether to wrap or not to wrap based on an OS environment variable PYTHONASYNCIODEBUG. This way it is possible to run asyncio programs with asyncio’s own functions instrumented. EventLoop.set_debug, a different debug facility, has no impact on @coroutine decorator’s behavior.
With this proposal, coroutines is a native, distinct from generators, concept. In addition to a RuntimeWarning being raised on coroutines that were never awaited, it is proposed to add two new functions to the sys module: set_coroutine_wrapper and get_coroutine_wrapper. This is to enable advanced debugging facilities in asyncio and other frameworks (such as displaying where exactly coroutine was created, and a more detailed stack trace of where it was garbage collected).
In order to allow better integration with existing frameworks (such as Tornado, see [13]) and compilers (such as Cython, see [16]), two new Abstract Base Classes (ABC) are added:
Note that generator-based coroutines with CO_ITERABLE_COROUTINE flag do not implement __await__ method, and therefore are not instances of collections.abc.Coroutine and collections.abc.Awaitable ABCs:
To allow easy testing if objects support asynchronous iteration, two more ABCs are added:
To avoid backwards compatibility issues with async and await keywords, it was decided to modify tokenizer.c in such a way, that it:
This approach allows for seamless combination of new syntax features (all of them available only in async functions) with any existing code.
An example of having “async def” and “async” attribute in one piece of code:
This proposal preserves 100% backwards compatibility.
asyncio module was adapted and tested to work with coroutines and new statements. Backwards compatibility is 100% preserved, i.e. all existing code will work as-is.
The required changes are mainly:
Because plain generators cannot yield from native coroutine objects (see Differences from generators section for more details), it is advised to make sure that all generator-based coroutines are decorated with @asyncio.coroutine before starting to use the new syntax.
There is no use of await names in CPython.
async is mostly used by asyncio. We are addressing this by renaming async() function to ensure_future() (see asyncio section for details).
Another use of async keyword is in Lib/xml/dom/xmlbuilder.py, to define an async = False attribute for DocumentLS class. There is no documentation or tests for it, it is not used anywhere else in CPython. It is replaced with a getter, that raises a DeprecationWarning, advising to use async_ attribute instead. ‘async’ attribute is not documented and is not used in CPython code base.
Grammar changes are fairly minimal:
async and await names will be softly deprecated in CPython 3.5 and 3.6. In 3.7 we will transform them to proper keywords. Making async and await proper keywords before 3.7 might make it harder for people to port their code to Python 3.
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines (called “cofunctions”). Some key points:
Differences from this proposal:
The following code:
would look like:
With async for keyword it is desirable to have a concept of a coroutine-generator – a coroutine with yield and yield from expressions. To avoid any ambiguity with regular generators, we would likely require to have an async keyword before yield, and async yield from would raise a StopAsyncIteration exception.
While it is possible to implement coroutine-generators, we believe that they are out of scope of this proposal. It is an advanced concept that should be carefully considered and balanced, with a non-trivial changes in the implementation of current generator objects. This is a matter for a separate PEP.
async/await is not a new concept in programming languages:
This is a huge benefit, as some users already have experience with async/await, and because it makes working with many languages in one project easier (Python with ECMAScript 7 for instance).
PEP 492 was accepted in CPython 3.5.0 with __aiter__ defined as a method, that was expected to return an awaitable resolving to an asynchronous iterator.
In 3.5.2 (as PEP 492 was accepted on a provisional basis) the __aiter__ protocol was updated to return asynchronous iterators directly.
The motivation behind this change is to make it possible to implement asynchronous generators in Python. See [19] and [20] for more details.
While it is possible to just implement await expression and treat all functions with at least one await as coroutines, this approach makes APIs design, code refactoring and its long time support harder.
Let’s pretend that Python only has await keyword:
If useful() function is refactored and someone removes all await expressions from it, it would become a regular python function, and all code that depends on it, including important() would be broken. To mitigate this issue a decorator similar to @asyncio.coroutine has to be introduced.
For some people bare async name(): pass syntax might look more appealing than async def name(): pass. It is certainly easier to type. But on the other hand, it breaks the symmetry between async def, async with and async for, where async is a modifier, stating that the statement is asynchronous. It is also more consistent with the existing grammar.
async is an adjective, and hence it is a better choice for a statement qualifier keyword. await for/with would imply that something is awaiting for a completion of a for or with statement.
async keyword is a statement qualifier. A good analogy to it are “static”, “public”, “unsafe” keywords from other languages. “async for” is an asynchronous “for” statement, “async with” is an asynchronous “with” statement, “async def” is an asynchronous function.
Having “async” after the main statement keyword might introduce some confusion, like “for async item in iterator” can be read as “for each asynchronous item in iterator”.
Having async keyword before def, with and for also makes the language grammar simpler. And “async def” better separates coroutines from regular functions visually.
Transition Plan section explains how tokenizer is modified to treat async and await as keywords only in async def blocks. Hence async def fills the role that a module level compiler declaration like from __future__ import async_await would otherwise fill.
New asynchronous magic methods __aiter__, __anext__, __aenter__, and __aexit__ all start with the same prefix “a”. An alternative proposal is to use “async” prefix, so that __anext__ becomes __async_next__. However, to align new magic methods with the existing ones, such as __radd__ and __iadd__ it was decided to use a shorter version.
An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an async keyword to their declarations:
This approach has the following downsides:
The vision behind existing generator-based coroutines and this proposal is to make it easy for users to see where the code might be suspended. Making existing “for” and “with” statements to recognize asynchronous iterators and context managers will inevitably create implicit suspend points, making it harder to reason about the code.
Syntax for asynchronous comprehensions could be provided, but this construct is outside of the scope of this PEP.
Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP.
This proposal introduces no observable performance impact. Here is an output of python’s official set of benchmarks [4]:
There is no observable slowdown of parsing python files with the modified tokenizer: parsing of one 12Mb file (Lib/test/test_binop.py repeated 1000 times) takes the same amount of time.
The following micro-benchmark was used to determine performance difference between “async” functions and generators:
The result is that there is no observable performance difference:
Note that depth of 19 means 1,048,575 calls.
The reference implementation can be found here: [3].
While the list of changes and new things is not short, it is important to understand, that most users will not use these features directly. It is intended to be used in frameworks and libraries to provide users with convenient to use and unambiguous APIs with async def, await, async for and async with syntax.
All concepts proposed in this PEP are implemented [3] and can be tested.
PEP 492 was accepted by Guido, Tuesday, May 5, 2015 [14].
The implementation is tracked in issue 24017 [15]. It was committed on May 11, 2015.
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, Łukasz Langa, Greg Ewing, Stephen J. Turnbull, Jim J. Jewett, Brett Cannon, Alyssa Coghlan, Steven D’Aprano, Paul Moore, Nathaniel Smith, Ethan Furman, Stefan Behnel, Paul Sokolovsky, Victor Petrovykh, and many others for their feedback, ideas, edits, criticism, code reviews, and discussions around this PEP.
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0492.rst
Last modified: 2025-02-01 08:55:40 UTC