We propose to extend the iterator protocol with a new __(a)iterclose__ slot, which is called automatically on exit from (async) for loops, regardless of how they exit. This allows for convenient, deterministic cleanup of resources held by iterators without reliance on the garbage collector. This is especially valuable for asynchronous generators.
In practical terms, the proposal here is divided into two separate parts: the handling of async iterators, which should ideally be implemented ASAP, and the handling of regular iterators, which is a larger but more relaxed project that can’t start until 3.7 at the earliest. But since the changes are closely related, and we probably don’t want to end up with async iterators and regular iterators diverging in the long run, it seems useful to look at them together.
Python iterables often hold resources which require cleanup. For example: file objects need to be closed; the WSGI spec adds a close method on top of the regular iterator protocol and demands that consumers call it at the appropriate time (though forgetting to do so is a frequent source of bugs); and PEP 342 (based on PEP 325) extended generator objects to add a close method to allow generators to clean up after themselves.
Generally, objects that need to clean up after themselves also define a __del__ method to ensure that this cleanup will happen eventually, when the object is garbage collected. However, relying on the garbage collector for cleanup like this causes serious problems in several cases:
Fortunately, Python provides a standard tool for doing resource cleanup in a more structured way: with blocks. For example, this code opens a file but relies on the garbage collector to close it:
and recent versions of CPython will point this out by issuing a ResourceWarning, nudging us to fix it by adding a with block:
But there’s a subtlety here, caused by the interaction of with blocks and generators. with blocks are Python’s main tool for managing cleanup, and they’re a powerful one, because they pin the lifetime of a resource to the lifetime of a stack frame. But this assumes that someone will take care of cleaning up the stack frame… and for generators, this requires that someone close them.
In this case, adding the with block is enough to shut up the ResourceWarning, but this is misleading – the file object cleanup here is still dependent on the garbage collector. The with block will only be unwound when the read_newline_separated_json generator is closed. If the outer for loop runs to completion then the cleanup will happen immediately; but if this loop is terminated early by a break or an exception, then the with block won’t fire until the generator object is garbage collected.
The correct solution requires that all users of this API wrap every for loop in its own with block:
This gets even worse if we consider the idiom of decomposing a complex pipeline into multiple nested generators:
In general if you have N nested generators then you need N+1 with blocks to clean up 1 file. And good defensive programming would suggest that any time we use a generator, we should assume the possibility that there could be at least one with block somewhere in its (potentially transitive) call stack, either now or in the future, and thus always wrap it in a with. But in practice, basically nobody does this, because programmers would rather write buggy code than tiresome repetitive code. In simple cases like this there are some workarounds that good Python developers know (e.g. in this simple case it would be idiomatic to pass in a file handle instead of a path and move the resource management to the top level), but in general we cannot avoid the use of with/finally inside of generators, and thus dealing with this problem one way or another. When beauty and correctness fight then beauty tends to win, so it’s important to make correct code beautiful.
Still, is this worth fixing? Until async generators came along I would have argued yes, but that it was a low priority, since everyone seems to be muddling along okay – but async generators make it much more urgent. Async generators cannot do cleanup at all without some mechanism for deterministic cleanup that people will actually use, and async generators are particularly likely to hold resources like file descriptors. (After all, if they weren’t doing I/O, they’d be generators, not async generators.) So we have to do something, and it might as well be a comprehensive fix to the underlying problem. And it’s much easier to fix this now when async generators are first rolling out, than it will be to fix it later.
The proposal itself is simple in concept: add a __(a)iterclose__ method to the iterator protocol, and have (async) for loops call it when the loop is exited, even if this occurs via break or exception unwinding. Effectively, we’re taking the current cumbersome idiom (with block + for loop) and merging them together into a fancier for. This may seem non-orthogonal, but makes sense when you consider that the existence of generators means that with blocks actually depend on iterator cleanup to work reliably, plus experience showing that iterator cleanup is often a desirable feature in its own right.
PEP 525 proposes a set of global thread-local hooks managed by new sys.{get/set}_asyncgen_hooks() functions, which allow event loops to integrate with the garbage collector to run cleanup for async generators. In principle, this proposal and PEP 525 are complementary, in the same way that with blocks and __del__ are complementary: this proposal takes care of ensuring deterministic cleanup in most cases, while PEP 525’s GC hooks clean up anything that gets missed. But __aiterclose__ provides a number of advantages over GC hooks alone:
then you can’t refactor this into an async generator without changing its semantics, and vice-versa. This seems very unpythonic. (It also leaves open the question of what exactly class-based async iterators are supposed to do, given that they face exactly the same cleanup problems as async generators.) __aiterclose__, on the other hand, is defined at the protocol level, so it’s duck-type friendly and works for all iterators, not just generators.
Arguably in regular code one can get away with skipping the with block around for loops, depending on how confident one is that one understands the internal implementation of the generator. But here we have to cope with arbitrary response handlers, so without __aiterclose__, this with construction is a mandatory part of every middleware.
__aiterclose__ allows us to eliminate the mandatory boilerplate and an extra level of indentation from every middleware:
So the __aiterclose__ approach provides substantial advantages over GC hooks.
This leaves open the question of whether we want a combination of GC hooks + __aiterclose__, or just __aiterclose__ alone. Since the vast majority of generators are iterated over using a for loop or equivalent, __aiterclose__ handles most situations before the GC has a chance to get involved. The case where GC hooks provide additional value is in code that does manual iteration, e.g.:
If we go with the GC-hooks + __aiterclose__ approach, this generator will eventually be cleaned up by GC calling the generator __del__ method, which then will use the hooks to call back into the event loop to run the cleanup code.
If we go with the no-GC-hooks approach, this generator will eventually be garbage collected, with the following effects:
The solution here – as the warning would indicate – is to fix the code so that it calls __aiterclose__, e.g. by using a with block:
Basically in this approach, the rule would be that if you want to manually implement the iterator protocol, then it’s your responsibility to implement all of it, and that now includes __(a)iterclose__.
GC hooks add non-trivial complexity in the form of (a) new global interpreter state, (b) a somewhat complicated control flow (e.g., async generator GC always involves resurrection, so the details of PEP 442 are important), and (c) a new public API in asyncio (await loop.shutdown_asyncgens()) that users have to remember to call at the appropriate time. (This last point in particular somewhat undermines the argument that GC hooks provide a safe backup to guarantee cleanup, since if shutdown_asyncgens() isn’t called correctly then I think it’s possible for generators to be silently discarded without their cleanup code being called; compare this to the __aiterclose__-only approach where in the worst case we still at least get a warning printed. This might be fixable.) All this considered, GC hooks arguably aren’t worth it, given that the only people they help are those who want to manually call __anext__ yet don’t want to manually call __aiterclose__. But Yury disagrees with me on this :-). And both options are viable.
Several commentators on python-dev and python-ideas have suggested that a pattern to avoid these problems is to always pass resources in from above, e.g. read_newline_separated_json should take a file object rather than a path, with cleanup handled at the top level:
This works well in simple cases; here it lets us avoid the “N+1 with blocks problem”. But unfortunately, it breaks down quickly when things get more complex. Consider if instead of reading from a file, our generator was reading from a streaming HTTP GET request – while handling redirects and authentication via OAUTH. Then we’d really want the sockets to be managed down inside our HTTP client library, not at the top level. Plus there are other cases where finally blocks embedded inside generators are important in their own right: db transaction management, emitting logging information during cleanup (one of the major motivating use cases for WSGI close), and so forth. So this is really a workaround for simple cases, not a general solution.
The semantics of __(a)iterclose__ are somewhat inspired by with blocks, but context managers are more powerful: __(a)exit__ can distinguish between a normal exit versus exception unwinding, and in the case of an exception it can examine the exception details and optionally suppress propagation. __(a)iterclose__ as proposed here does not have these powers, but one can imagine an alternative design where it did.
However, this seems like unwarranted complexity: experience suggests that it’s common for iterables to have close methods, and even to have __exit__ methods that call self.close(), but I’m not aware of any common cases that make use of __exit__’s full power. I also can’t think of any examples where this would be useful. And it seems unnecessarily confusing to allow iterators to affect flow control by swallowing exceptions – if you’re in a situation where you really want that, then you should probably use a real with block anyway.
This section describes where we want to eventually end up, though there are some backwards compatibility issues that mean we can’t jump directly here. A later section describes the transition plan.
Generally, __(a)iterclose__ implementations should:
And generally, any code which starts iterating through an iterable with the intention of exhausting it, should arrange to make sure that __(a)iterclose__ is eventually called, whether or not the iterator is actually exhausted.
The core proposal is the change in behavior of for loops. Given this Python code:
we desugar to the equivalent of:
where the “traditional-for statement” here is meant as a shorthand for the classic 3.5-and-earlier for loop semantics.
Besides the top-level for statement, Python also contains several other places where iterators are consumed. For consistency, these should call __iterclose__ as well using semantics equivalent to the above. This includes:
In addition, a yield from that successfully exhausts the called generator should as a last step call its __iterclose__ method. (Rationale: yield from already links the lifetime of the calling generator to the called generator; if the calling generator is closed when half-way through a yield from, then this will already automatically close the called generator.)
We also make the analogous changes to async iteration constructs, except that the new slot is called __aiterclose__, and it’s an async method that gets awaited.
Generator objects (including those created by generator comprehensions):
Async generator objects (including those created by async generator comprehensions):
QUESTION: should file objects implement __iterclose__ to close the file? On the one hand this would make this change more disruptive; on the other hand people really like writing for line in open(...): ..., and if we get used to iterators taking care of their own cleanup then it might become very weird if files don’t.
The operator module gains two new functions, with semantics equivalent to the following:
The itertools module gains a new iterator wrapper that can be used to selectively disable the new __iterclose__ behavior:
Example usage (assuming that file objects implements __iterclose__):
Python ships a number of iterator types that act as wrappers around other iterators: map, zip, itertools.accumulate, csv.reader, and others. These iterators should define a __iterclose__ method which calls __iterclose__ in turn on their underlying iterators. For example, map could be implemented as:
In some cases this requires some subtlety; for example, itertools.tee should not call __iterclose__ on the underlying iterator until it has been called on all of the clone iterators.
The payoff for all this is that we can now write straightforward code like:
and be confident that the file will receive deterministic cleanup without the end-user having to take any special effort, even in complex cases. For example, consider this silly pipeline:
If our file contains a document where doc["key"] turns out to be an integer, then the following sequence of events will happen:
(The details above assume that we implement file.__iterclose__; if not then add a with block to read_newline_separated_json and essentially the same logic goes through.)
Of course, from the user’s point of view, this can be simplified down to just:
1. int.upper() raises an AttributeError 1. The file object is closed. 2. The AttributeError propagates out of list
So we’ve accomplished our goal of making this “just work” without the user having to think about it.
While the majority of existing for loops will continue to produce identical results, the proposed changes will produce backwards-incompatible behavior in some cases. Example:
This code used to be correct, but after this proposal is implemented will require an itertools.preserve call added to the first for loop.
[QUESTION: currently, if you close a generator and then try to iterate over it then it just raises Stop(Async)Iteration, so code the passes the same generator object to multiple for loops but forgets to use itertools.preserve won’t see an obvious error – the second for loop will just exit immediately. Perhaps it would be better if iterating a closed generator raised a RuntimeError? Note that files don’t have this problem – attempting to iterate a closed file object already raises ValueError.]
Specifically, the incompatibility happens when all of these factors come together:
So the problem is how to manage this transition, and those are the levers we have to work with.
First, observe that the only async iterables where we propose to add __aiterclose__ are async generators, and there is currently no existing code using async generators (though this will start changing very soon), so the async changes do not produce any backwards incompatibilities. (There is existing code using async iterators, but using the new async for loop on an old async iterator is harmless, because old async iterators don’t have __aiterclose__.) In addition, PEP 525 was accepted on a provisional basis, and async generators are by far the biggest beneficiary of this PEP’s proposed changes. Therefore, I think we should strongly consider enabling __aiterclose__ for async for loops and async generators ASAP, ideally for 3.6.0 or 3.6.1.
For the non-async world, things are harder, but here’s a potential transition path:
In 3.7:
Our goal is that existing unsafe code will start emitting warnings, while those who want to opt-in to the future can do that immediately:
In 3.8:
In 3.9:
I believe that this satisfies the normal requirements for this kind of transition – opt-in initially, with warnings targeted precisely to the cases that will be effected, and a long deprecation cycle.
Probably the most controversial / risky part of this is the use of stack introspection to make the iterable-consuming functions sensitive to a __future__ setting, though I haven’t thought of any situation where it would actually go wrong yet…
Thanks to Yury Selivanov, Armin Rigo, and Carl Friedrich Bolz for helpful discussion on earlier versions of this idea.
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0533.rst
Last modified: 2025-02-01 08:59:27 UTC