Add Limited C API support for extending some types with opaque data by allowing code to only deal with data specific to a particular (sub)class.
This mechanism is required to be usable with PyHeapTypeObject.
This PEP does not propose allowing to extend non-dynamically sized variable sized objects such as tuple or int due to their different memory layout and perceived lack of demand for doing so. This PEP leaves room to do so in the future via the same mechanism if ever desired.
The motivating problem this PEP solves is attaching C-level state to custom types — i.e. metaclasses (subclasses of type).
This is often needed in “wrappers” that expose another type system (e.g. C++, Java, Rust) as Python classes. These typically need to attach information about the “wrapped” non-Python class to the Python type object.
This should be possible to do in the Limited API, so that the language wrappers or code generators can be used to create Stable ABI extensions. (See PEP 652 for the benefits of providing a stable ABI.)
Extending type is an instance of a more general problem: extending a class while maintaining loose coupling – that is, not depending on the memory layout used by the superclass. (That’s a lot of jargon; see Rationale for a concrete example of extending list.)
In the Limited API, most structs are opaque: their size and memory layout are not exposed, so they can be changed in new versions of CPython (or alternate implementations of the C API).
This means that the usual subclassing pattern – making the struct used for instances of the base type be the first element of the struct used for instances of the derived type – does not work. To illustrate with code, the example from the tutorial extends PyListObject (list) using the following struct:
This won’t compile in the Limited API, since PyListObject is opaque (to allow changes as features and optimizations are implemented).
Instead, this PEP proposes using a struct with only the state needed in the subclass, that is:
The subclass can now be completely decoupled from the memory layout (and size) of the superclass.
This is possible today. To use such a struct:
However, this has disadvantages:
To make this easy (and even best practice for projects that choose loose coupling over maximum performance), this PEP proposes an API to:
This will be specified by a negative PyType_Spec.basicsize: -sizeof(SubListState).
The base class is not limited to PyListObject, of course: it can be used to extend any base class whose instance struct is opaque, unstable across releases, or not exposed at all – including type (PyHeapTypeObject) or third-party extensions (for example, NumPy arrays [1]).
For cases where no additional state is needed, a zero basicsize will be allowed: in that case, the base’s tp_basicsize will be inherited. (This currently works, but lacks explicit documentation and tests.)
The tp_basicsize of the new class will be set to the computed total size, so code that inspects classes will continue working as before.
Additional considerations are needed to subclass variable-sized objects while maintaining loose coupling: the variable-sized data can collide with subclass data (SubListState in the example above).
Currently, CPython doesn’t provide a way to prevent such collisions. So, the proposed mechanism of extending opaque classes (negative base->tp_itemsize) will fail by default.
We could stop there, but since the motivating type — PyHeapTypeObject — is variable sized, we need a safe way to allow subclassing it. A bit of background first:
There are two main memory layouts for variable-sized objects.
In types such as int or tuple, the variable data is stored at a fixed offset. If subclasses need additional space, it must be added after any variable-sized data:
In other types, like PyHeapTypeObject, variable-sized data always lives at the end of the instance’s memory area:
The first layout enables fast access to the items array. The second allows subclasses to ignore the variable-sized array (assuming they use offsets from the start of the object to access their data).
Since this PEP focuses on PyHeapTypeObject, it proposes an API to allow subclassing for the second variant. Support for the first can be added later as an API-compatible change (though your PEP author doubts it’d be worth the effort).
This PEP proposes a type flag, Py_TPFLAGS_ITEMS_AT_END, which will indicate the PyHeapTypeObject-like layout. This can be set in two ways:
This flag will be necessary to extend a variable-sized type using negative basicsize.
An alternative to a flag would be to require subclass authors to know that the base uses a compatible layout (e.g. from documentation). A past version of this PEP proposed a new PyType_Slot for it. This turned out to be hard to explain, and goes against the idea of decoupling the subclass from the base layout.
The new flag will be used to allow safely extending variable-sized types: creating a type with spec->basicsize < 0 and base->tp_itemsize > 0 will require the flag.
Additionally, this PEP proposes a helper function to get the variable-sized data of a given instance, if it uses the new Py_TPFLAGS_ITEMS_AT_END flag. This hides the necessary pointer arithmetic behind an API that can potentially be adapted to other layouts in the future (including, potentially, a VM-managed layout).
To make it easier to verify that all cases are covered, here’s a scary-looking big-picture decision tree.
Note
The individual cases are easier to explain in isolation (see the reference implementation for draft docs).
Setting spec->itemsize < 0 is always an error. This PEP does not propose any mechanism to extend tp->itemsize rather than just inherit it.
One more piece of the puzzle is PyMemberDef.offset. Extensions that use a subclass-specific struct (SubListState above) will get a way to specify “relative” offsets (offsets based from this struct) rather than “absolute” ones (based off the PyObject struct).
One way to do it would be to automatically assume “relative” offsets when creating a class using the new API. However, this implicit assumption would be too surprising.
To be more explicit, this PEP proposes a new flag for “relative” offsets. At least initially, this flag will serve only as a check against misuse (and a hint for reviewers). It must be present if used with the new API, and must not be used otherwise.
In the code blocks below, only function headers are part of the specification. Other code (the size/offset calculations) are details of the initial CPython implementation, and subject to change.
The basicsize member of PyType_Spec will be allowed to be zero or negative. In that case, its absolute value will specify how much extra storage space instances of the new class require, in addition to the basicsize of the base class. That is, the basicsize of the resulting class will be:
where _align rounds up to a multiple of alignof(max_align_t).
When spec->basicsize is zero, basicsize will be inherited directly instead, i.e. set to base->tp_basicsize without aligning. (This already works; explicit tests and documentation will be added.)
On an instance, the memory area specific to a subclass – that is, the “extra space” that subclass reserves in addition its base – will be available through a new function, PyObject_GetTypeData. In CPython, this function will be defined as:
Another function will be added to retrieve the size of this memory area:
The result may be higher than requested by -basicsize. It is safe to use all of it (e.g. with memset).
The new *Get* functions come with an important caveat, which will be pointed out in documentation: They may only be used for classes created using negative PyType_Spec.basicsize. For other classes, their behavior is undefined. (Note that this allows the above code to assume cls->tp_base is not NULL.)
When spec->itemsize is zero, tp_itemsize will be inherited from the base. (This already works; explicit tests and documentation will be added.)
A new type flag, Py_TPFLAGS_ITEMS_AT_END, will be added. This flag can only be set on types with non-zero tp_itemsize. It indicates that the variable-sized portion of an instance is stored at the end of the instance’s memory.
The default metatype (PyType_Type) will set this flag.
A new function, PyObject_GetItemData, will be added to access the memory reserved for variable-sized content of types with the new flag. In CPython it will be defined as:
This function will initially not be added to the Limited API.
Extending a class with positive base->itemsize using negative spec->basicsize will fail unless Py_TPFLAGS_ITEMS_AT_END is set, either on the base or in spec->flags. (See Extending variable-size objects for a full explanation.)
Extending a class with positive spec->itemsize using negative spec->basicsize will fail.
In types defined using negative PyType_Spec.basicsize, the offsets of members defined via Py_tp_members must be relative to the extra subclass data, rather than the full PyObject struct. This will be indicated by a new flag in PyMemberDef.flags: Py_RELATIVE_OFFSET.
In the initial implementation, the new flag will be redundant. It only serves to make the offset’s changed meaning clear, and to help avoid mistakes. It will be an error to not use Py_RELATIVE_OFFSET with negative basicsize, and it will be an error to use it in any other context (i.e. direct or indirect calls to PyDescr_NewMember, PyMember_GetOne, PyMember_SetOne).
CPython will adjust the offset and clear the Py_RELATIVE_OFFSET flag when initializing a type. This means that:
The following new functions/values are proposed.
These will be added to the Limited API/Stable ABI:
These will be added to the public C API only:
No backwards compatibility concerns are known.
The implementation assumes that an instance’s memory between type->tp_base->tp_basicsize and type->tp_basicsize offsets “belongs” to type (except variable-length types). This is not documented explicitly, but CPython up to version 3.11 relied on it when adding __dict__ to subclasses, so it should be safe.
None known.
The author of pybind11 originally requested solving the issue (see point 2 in this list), and has been verifying the implementation.
Florian from the HPy project said that the API looks good in general. (See below for a possible solution to performance concerns.)
The initial implementation will include reference documentation and a What’s New entry, which should be enough for the target audience – authors of C extension libraries.
A reference implementation is in the extend-opaque branch in the encukou/cpython GitHub repo.
The proposed implementation may waste some space if instance structs need smaller alignment than alignof(max_align_t). Also, dealing with alignment makes the calculation slower than it could be if we could rely on base->tp_basicsize being properly aligned for the subtype.
In other words, the proposed implementation focuses on safety and ease of use, and trades space and time for it. If it turns out that this is a problem, the implementation can be adjusted without breaking the API:
A flag like Py_TPFLAGS_ITEMS_AT_END could be added to signal the “tuple-like” layout described in Extending variable-size objects, and all mechanisms this PEP proposes could be adapted to support it. Other layouts could be added as well. However, it seems there’d be very little practical benefit, so it’s just a theoretical possibility.
Instead of a negative spec->basicsize, a new PyType_Spec flag could’ve been added. The effect would be the same to any existing code accessing these internals without up to date knowledge of the change as the meaning of the field value is changing in this situation.
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0697.rst
Last modified: 2025-02-01 08:55:40 UTC