Several modern programming languages have so-called “null-coalescing” or “null- aware” operators, including C# [1], Dart [2], Perl, Swift, and PHP (starting in version 7). There are also stage 3 draft proposals for their addition to ECMAScript (a.k.a. JavaScript) [3] [4]. These operators provide syntactic sugar for common patterns involving null references.
This PEP proposes three None-aware operators for Python, based on the definitions and other language’s implementations of those above. Specifically:
See the Grammar changes section for specifics and examples of the required grammar changes.
See the Examples section for more realistic examples of code that could be updated to use the new operators.
The None object denotes the lack of a value. For the purposes of these operators, the lack of a value indicates that the remainder of the expression also lacks a value and should not be evaluated.
A rejected proposal was to treat any value that evaluates as “false” in a Boolean context as not having a value. However, the purpose of these operators is to propagate the “lack of value” state, rather than the “false” state.
Some argue that this makes None special. We contend that None is already special, and that using it as both the test and the result of these operators does not change the existing semantics in any way.
See the Rejected Ideas section for discussions on alternate approaches.
The following rules of the Python grammar are updated to read:
The coalesce rule provides the ?? binary operator. Unlike most binary operators, the right-hand side is not evaluated until the left-hand side is determined to be None.
The ?? operator binds more tightly than other binary operators as most existing implementations of these do not propagate None values (they will typically raise TypeError). Expressions that are known to potentially result in None can be substituted for a default value without needing additional parentheses.
Some examples of how implicit parentheses are placed when evaluating operator precedence in the presence of the ?? operator:
Particularly for cases such as a ?? 2 ** b ?? 3, parenthesizing the sub-expressions any other way would result in TypeError, as int.__pow__ cannot be called with None (and the fact that the ?? operator is used at all implies that a or b may be None). However, as usual, while parentheses are not required they should be added if it helps improve readability.
An augmented assignment for the ?? operator is also added. Augmented coalescing assignment only rebinds the name if its current value is None. If the target name already has a value, the right-hand side is not evaluated. For example:
The maybe-dot and maybe-subscript operators are added as trailers for atoms, so that they may be used in all the same locations as the regular operators, including as part of an assignment target (more details below). As the existing evaluation rules are not directly embedded in the grammar, we specify the required changes below.
Assume that the atom is always successfully evaluated. Each trailer is then evaluated from left to right, applying its own parameter (either its arguments, subscripts or attribute name) to produce the value for the next trailer. Finally, if present, await is applied.
For example, await a.b(c).d[e] is currently parsed as ['await', 'a', '.b', '(c)', '.d', '[e]'] and evaluated:
When a None-aware operator is present, the left-to-right evaluation may be short-circuited. For example, await a?.b(c).d?[e] is evaluated:
Note
await will almost certainly fail in this context, as it would in the case where code attempts await None. We are not proposing to add a None-aware await keyword here, and merely include it in this example for completeness of the specification, since the atom_expr grammar rule includes the keyword. If it were in its own rule, we would have never mentioned it.
Parenthesised expressions are handled by the atom rule (not shown above), which will implicitly terminate the short-circuiting behaviour of the above transformation. For example, (a?.b ?? c).d?.e is evaluated as:
When used as an assignment target, the None-aware operations may only be used in a “load” context. That is, a?.b = 1 and a?[b] = 1 will raise SyntaxError. Use earlier in the expression (a?.b.c = 1) is permitted, though unlikely to be useful unless combined with a coalescing operation:
For the maybe-dot and maybe-subscript operators, the intention is that expressions including these operators should be read and interpreted as for the regular versions of these operators. In “normal” cases, the end results are going to be identical between an expression such as a?.b?[c] and a.b[c], and just as we do not currently read “a.b” as “read attribute b from a if it has an attribute a or else it raises AttributeError”, there is no need to read “a?.b” as “read attribute b from a if a is not None” (unless in a context where the listener needs to be aware of the specific behaviour).
For coalescing expressions using the ?? operator, expressions should either be read as “or … if None” or “coalesced with”. For example, the expression a.get_value() ?? 100 would be read “call a dot get_value or 100 if None”, or “call a dot get_value coalesced with 100”.
Note
Reading code in spoken text is always lossy, and so we make no attempt to define an unambiguous way of speaking these operators. These suggestions are intended to add context to the implications of adding the new syntax.
This section presents some examples of common None patterns and shows what conversion to use None-aware operators may look like.
Using the find-pep505.py script [5] an analysis of the Python 3.7 standard library discovered up to 678 code snippets that could be replaced with use of one of the None-aware operators:
Some of these are shown below as examples before and after converting to use the new operators.
From bisect.py:
After updating to use the ??= augmented assignment statement:
From calendar.py:
After updating to use the ?? operator:
From email/generator.py (and importantly note that there is no way to substitute or for ?? in this situation):
After updating:
From asyncio/subprocess.py:
After updating to use the ?. operator:
From asyncio/tasks.py:
After updating to use the ?. operator:
From ctypes/_aix.py:
After updating:
From os.py:
After updating to use the ?. operator:
From importlib/abc.py:
After partially updating:
After extensive updating (arguably excessive, though that’s for the style guides to determine):
From dis.py:
After updating to use the ?[] and ?? operators:
This example is from a Python web crawler that uses the Flask framework as its front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client:
Both first_seen and last_seen are allowed to be null in the database, and they are also allowed to be null in the JSON response. JSON does not have a native way to represent a datetime, so the server’s contract states that any non-null date is represented as an ISO-8601 string.
Without knowing the exact semantics of the first_seen and last_seen attributes, it is impossible to know whether the attribute can be safely or performantly accessed multiple times.
One way to fix this code is to replace each conditional expression with an explicit value assignment and a full if/else block:
This adds ten lines of code and four new code paths to the function, dramatically increasing the apparent complexity. Rewriting using the None-aware attribute operator results in shorter code with more clear intent:
The next example is from a Python scraping library called Grab:
This example contains several good examples of needing to provide default values. Rewriting to use conditional expressions reduces the overall lines of code, but does not necessarily improve readability:
The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ctype if it has a value and use the string literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved.
Rewriting using the None coalescing operator:
This syntax has an intuitive ordering of the operands. In find_content_type, for example, the preferred value ctype appears before the fallback value. The terseness of the syntax also makes for fewer lines of code and less code to visually parse, and reading from left-to-right and top-to-bottom more accurately follows the execution flow.
The first three ideas in this section are oft-proposed alternatives to treating None as special. For further background on why these are rejected, see their treatment in PEP 531 and PEP 532 and the associated discussions.
The operators could be generalised to user-defined types by defining a protocol to indicate when a value represents “no value”. Such a protocol may be a dunder method __has_value__(self) that returns True if the value should be treated as having a value, and False if the value should be treated as no value.
With this generalization, object would implement a dunder method equivalent to this:
NoneType would implement a dunder method equivalent to this:
In the specification section, all uses of x is None would be replaced with not x.__has_value__().
This generalization would allow for domain-specific “no-value” objects to be coalesced just like None. For example, the pyasn1 package has a type called Null that represents an ASN.1 null:
Similarly, values such as math.nan and NotImplemented could be treated as representing no value.
However, the “no-value” nature of these values is domain-specific, which means they should be treated as a value by the language. For example, math.nan.imag is well defined (it’s 0.0), and so short-circuiting math.nan?.imag to return math.nan would be incorrect.
As None is already defined by the language as being the value that represents “no value”, and the current specification would not preclude switching to a protocol in the future (though changes to built-in objects would not be compatible), this idea is rejected for now.
This suggestion is fundamentally the same as adding a no-value protocol, and so the discussion above also applies.
Similar behavior to the ?? operator can be achieved with an or expression, however or checks whether its left operand is false-y and not specifically None. This approach is attractive, as it requires fewer changes to the language, but ultimately does not solve the underlying problem correctly.
Assuming the check is for truthiness rather than None, there is no longer a need for the ?? operator. However, applying this check to the ?. and ?[] operators prevents perfectly valid operations applying
Consider the following example, where get_log_list() may return either a list containing current log messages (potentially empty), or None if logging is not enabled:
If ?. is checking for true values rather than specifically None and the log has not been initialized with any items, no item will ever be appended. This violates the obvious intent of the code, which is to append an item. The append method is available on an empty list, as are all other list methods, and there is no reason to assume that these members should not be used because the list is presently empty.
Further, there is no sensible result to use in place of the expression. A normal lst.append returns None, but under this idea lst?.append may result in either [] or None, depending on the value of lst. As with the examples in the previous section, this makes no sense.
As checking for truthiness rather than None results in apparently valid expressions no longer executing as intended, this idea is rejected.
Arguably, the reason to short-circuit an expression when None is encountered is to avoid the AttributeError or TypeError that would be raised under normal circumstances. As an alternative to testing for None, the ?. and ?[] operators could instead handle AttributeError and TypeError raised by the operation and skip the remainder of the expression.
This produces a transformation for a?.b.c?.d.e similar to this:
One open question is which value should be returned as the expression when an exception is handled. The above example simply leaves the partial result, but this is not helpful for replacing with a default value. An alternative would be to force the result to None, which then raises the question as to why None is special enough to be the result but not special enough to be the test.
Secondly, this approach masks errors within code executed implicitly as part of the expression. For ?., any AttributeError within a property or __getattr__ implementation would be hidden, and similarly for ?[] and __getitem__ implementations.
Similarly, simple typing errors such as {}?.ietms() could go unnoticed.
Existing conventions for handling these kinds of errors in the form of the getattr builtin and the .get(key, default) method pattern established by dict show that it is already possible to explicitly use this behaviour.
As this approach would hide errors in code, it is rejected.
The None-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might be written as foo?(), where foo is only called if it is not None.
This has been deferred on the basis of the proposed operators being intended to aid traversal of partially populated hierarchical data structures, not for traversal of arbitrary class hierarchies. This is reflected in the fact that none of the other mainstream languages that already offer this syntax have found it worthwhile to support a similar syntax for optional function invocations.
A workaround similar to that used by C# would be to write maybe_none?.__call__(arguments). If the callable is None, the expression will not be evaluated. (The C# equivalent uses ?.Invoke() on its callable type.)
To generalize the None-aware behavior and limit the number of new operators introduced, a unary, postfix operator spelled ? was suggested. The idea is that ? might return a special object that could would override dunder methods that return self. For example, foo? would evaluate to foo if it is not None, otherwise it would evaluate to an instance of NoneQuestion:
With this new operator and new type, an expression like foo?.bar[baz] evaluates to NoneQuestion if foo is None. This is a nifty generalization, but it’s difficult to use in practice since most existing code won’t know what NoneQuestion is.
Going back to one of the motivating examples above, consider the following:
The JSON serializer does not know how to serialize NoneQuestion, nor will any other API. This proposal actually requires lots of specialized logic throughout the standard library and any third party library.
At the same time, the ? operator may also be too general, in the sense that it can be combined with any other operator. What should the following expressions mean?:
This degree of generalization is not useful. The operators actually proposed herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns.
Haskell has a concept called Maybe that encapsulates the idea of an optional value without relying on any special keyword (e.g. null) or any special instance (e.g. None). In Haskell, the purpose of Maybe is to avoid separate handling of “something” and nothing”.
A Python package called pymaybe provides a rough approximation. The documentation shows the following example:
The function maybe() returns either a Something instance or a Nothing instance. Similar to the unary postfix operator described in the previous section, Nothing overrides dunder methods in order to allow chaining on a missing value.
Note that or_else() is eventually required to retrieve the underlying value from pymaybe’s wrappers. Furthermore, pymaybe does not short circuit any evaluation. Although pymaybe has some strengths and may be useful in its own right, it also demonstrates why a pure Python implementation of coalescing is not nearly as powerful as support built into the language.
The idea of adding a builtin maybe type to enable this scenario is rejected.
Another common way to initialize default values is to use the ternary operator. Here is an excerpt from the popular Requests package:
This particular formulation has the undesirable effect of putting the operands in an unintuitive order: the brain thinks, “use data if possible and use [] as a fallback,” but the code puts the fallback before the preferred value.
The author of this package could have written it like this instead:
This ordering of the operands is more intuitive, but it requires 4 extra characters (for “not “). It also highlights the repetition of identifiers: data if data, files if files, etc.
When written using the None coalescing operator, the sample reads:
This document has been placed in the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0505.rst
Last modified: 2025-02-01 08:55:40 UTC