This document proposes an iteration interface that objects can provide to control the behaviour of for loops. Looping is customized by providing a method that produces an iterator object. The iterator provides a get next value operation that produces the next item in the sequence each time it is called, raising an exception when no more items are available.
In addition, specific iterators over the keys of a dictionary and over the lines of a file are proposed, and a proposal is made to allow spelling dict.has_key(key) as key in dict.
Note: this is an almost complete rewrite of this PEP by the second author, describing the actual implementation checked into the trunk of the Python 2.2 CVS tree. It is still open for discussion. Some of the more esoteric proposals in the original version of this PEP have been withdrawn for now; these may be the subject of a separate PEP in the future.
A new exception is defined, StopIteration, which can be used to signal the end of an iteration.
A new slot named tp_iter for requesting an iterator is added to the type object structure. This should be a function of one PyObject * argument returning a PyObject *, or NULL. To use this slot, a new C API function PyObject_GetIter() is added, with the same signature as the tp_iter slot function.
Another new slot, named tp_iternext, is added to the type structure, for obtaining the next value in the iteration. To use this slot, a new C API function PyIter_Next() is added. The signature for both the slot and the API function is as follows, although the NULL return conditions differ: the argument is a PyObject * and so is the return value. When the return value is non-NULL, it is the next value in the iteration. When it is NULL, then for the tp_iternext slot there are three possibilities:
The higher-level PyIter_Next() function clears the StopIteration exception (or derived exception) when it occurs, so its NULL return conditions are simpler:
Iterators implemented in C should not implement a next() method with similar semantics as the tp_iternext slot! When the type’s dictionary is initialized (by PyType_Ready()), the presence of a tp_iternext slot causes a method next() wrapping that slot to be added to the type’s tp_dict. (Exception: if the type doesn’t use PyObject_GenericGetAttr() to access instance attributes, the next() method in the type’s tp_dict may not be seen.) (Due to a misunderstanding in the original text of this PEP, in Python 2.2, all iterator types implemented a next() method that was overridden by the wrapper; this has been fixed in Python 2.3.)
To ensure binary backwards compatibility, a new flag Py_TPFLAGS_HAVE_ITER is added to the set of flags in the tp_flags field, and to the default flags macro. This flag must be tested before accessing the tp_iter or tp_iternext slots. The macro PyIter_Check() tests whether an object has the appropriate flag set and has a non-NULL tp_iternext slot. There is no such macro for the tp_iter slot (since the only place where this slot is referenced should be PyObject_GetIter(), and this can check for the Py_TPFLAGS_HAVE_ITER flag directly).
(Note: the tp_iter slot can be present on any object; the tp_iternext slot should only be present on objects that act as iterators.)
For backwards compatibility, the PyObject_GetIter() function implements fallback semantics when its argument is a sequence that does not implement a tp_iter function: a lightweight sequence iterator object is constructed in that case which iterates over the items of the sequence in the natural order.
The Python bytecode generated for for loops is changed to use new opcodes, GET_ITER and FOR_ITER, that use the iterator protocol rather than the sequence protocol to get the next value for the loop variable. This makes it possible to use a for loop to loop over non-sequence objects that support the tp_iter slot. Other places where the interpreter loops over the values of a sequence should also be changed to use iterators.
Iterators ought to implement the tp_iter slot as returning a reference to themselves; this is needed to make it possible to use an iterator (as opposed to a sequence) in a for loop.
Iterator implementations (in C or in Python) should guarantee that once the iterator has signalled its exhaustion, subsequent calls to tp_iternext or to the next() method will continue to do so. It is not specified whether an iterator should enter the exhausted state when an exception (other than StopIteration) is raised. Note that Python cannot guarantee that user-defined or 3rd party iterators implement this requirement correctly.
The StopIteration exception is made visible as one of the standard exceptions. It is derived from Exception.
A new built-in function is defined, iter(), which can be called in two ways:
Iterator objects returned by either form of iter() have a next() method. This method either returns the next value in the iteration, or raises StopIteration (or a derived exception class) to signal the end of the iteration. Any other exception should be considered to signify an error and should be propagated normally, not taken to mean the end of the iteration.
Classes can define how they are iterated over by defining an __iter__() method; this should take no additional arguments and return a valid iterator object. A class that wants to be an iterator should implement two methods: a next() method that behaves as described above, and an __iter__() method that returns self.
The two methods correspond to two distinct protocols:
Container-like objects usually support protocol 1. Iterators are currently required to support both protocols. The semantics of iteration come only from protocol 2; protocol 1 is present to make iterators behave like sequences; in particular so that code receiving an iterator can use a for-loop over the iterator.
which is equivalent to
which is equivalent to, but much faster than
as long as the restriction on modifications to the dictionary (either by the loop or by another thread) are not violated.
This means that for x in dict is shorthand for for x in dict.iterkeys().
Other mappings, if they support iterators at all, should also iterate over the keys. However, this should not be taken as an absolute rule; specific applications may have different requirements.
The following proposal is useful because it provides us with a good answer to the complaint that the common idiom to iterate over the lines of a file is ugly and slow.
as a shorthand for
which is equivalent to, but faster than
This also shows that some iterators are destructive: they consume all the values and a second iterator cannot easily be created that iterates independently over the same values. You could open the file for a second time, or seek() to the beginning, but these solutions don’t work for all file types, e.g. they don’t work when the open file object really represents a pipe or a stream socket.
Because the file iterator uses an internal buffer, mixing this with other file operations (e.g. file.readline()) doesn’t work right. Also, the following code:
doesn’t work as you might expect, because the iterator created by the second for-loop doesn’t take the buffer read-ahead by the first for-loop into account. A correct way to write this is:
(The rationale for these restrictions are that for line in file ought to become the recommended, standard way to iterate over the lines of a file, and this should be as fast as can be. The iterator version is considerable faster than calling readline(), due to the internal buffer in the iterator.)
If all the parts of the proposal are included, this addresses many concerns in a consistent and flexible fashion. Among its chief virtues are the following four – no, five – no, six – points:
The following topics have been decided by consensus or BDFL pronouncement.
Arguments against __next__(): while many iterators are used in for loops, it is expected that user code will also call next() directly, so having to write __next__() is ugly; also, a possible extension of the protocol would be to allow for prev(), current() and reset() operations; surely we don’t want to use __prev__(), __current__(), __reset__().
Arguments against __call__() (the original proposal): taken out of context, x() is not very readable, while x.next() is clear; there’s a danger that every special-purpose object wants to use __call__() for its most common operation, causing more confusion than clarity.
(In retrospect, it might have been better to go for __next__() and have a new built-in, next(it), which calls it.__next__(). But alas, it’s too late; this has been deployed in Python 2.2 since December 2001.)
Resolution: iter() it is.
Resolution: the builtin iter() takes an optional argument, which is the sentinel to look for.
Resolution: once StopIteration is raised, calling it.next() continues to raise StopIteration.
Note: this was in fact not implemented in Python 2.2; there are many cases where an iterator’s next() method can raise StopIteration on one call but not on the next. This has been remedied in Python 2.3.
Resolution: tentatively rejected (though there are still people arguing for this).
While some of these are useful, many of these cannot easily be implemented for all iterator types without adding arbitrary buffering, and sometimes they can’t be implemented at all (or not reasonably). E.g. anything to do with reversing directions can’t be done when iterating over a file or function. Maybe a separate PEP can be drafted to standardize the names for such operations when they are implementable.
Resolution: rejected.
should assign x the successive keys, values, or items of the dictionary. The symmetry between if x in y and for x in y suggests that it should iterate over keys. This symmetry has been observed by many independently and has even been used to “explain” one using the other. This is because for sequences, if x in y iterates over y comparing the iterated values to x. If we adopt both of the above proposals, this will also hold for dictionaries.
The argument against making for x in dict iterate over the keys comes mostly from a practicality point of view: scans of the standard library show that there are about as many uses of for x in dict.items() as there are of for x in dict.keys(), with the items() version having a small majority. Presumably many of the loops using keys() use the corresponding value anyway, by writing dict[x], so (the argument goes) by making both the key and value available, we could support the largest number of cases. While this is true, I (Guido) find the correspondence between for x in dict and if x in dict too compelling to break, and there’s not much overhead in having to write dict[x] to explicitly get the value.
For fast iteration over items, use for key, value in dict.iteritems(). I’ve timed the difference between
and
and found that the latter is only about 7% faster.
Resolution: By BDFL pronouncement, for x in dict iterates over the keys, and dictionaries have iteritems(), iterkeys(), and itervalues() to return the different flavors of dictionary iterators.
The iterator protocol has been discussed extensively in a mailing list on SourceForge:
Initially, some of the discussion was carried out at Yahoo; archives are still accessible:
This document is in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0234.rst
Last modified: 2025-02-01 08:55:40 UTC