This PEP proposes to standardize a new pickle protocol version, and accompanying APIs to take full advantage of it:
The PEP guarantees unchanged behaviour for anyone not using the new APIs.
The pickle protocol was originally designed in 1995 for on-disk persistency of arbitrary Python objects. The performance of a 1995-era storage medium probably made it irrelevant to focus on performance metrics such as use of RAM bandwidth when copying temporary data before writing it to disk.
Nowadays the pickle protocol sees a growing use in applications where most of the data isn’t ever persisted to disk (or, when it is, it uses a portable format instead of Python-specific). Instead, pickle is being used to transmit data and commands from one process to another, either on the same machine or on multiple machines. Those applications will sometimes deal with very large data (such as Numpy arrays or Pandas dataframes) that need to be transferred around. For those applications, pickle is currently wasteful as it imposes spurious memory copies of the data being serialized.
As a matter of fact, the standard multiprocessing module uses pickle for serialization, and therefore also suffers from this problem when sending large data to another process.
Third-party Python libraries, such as Dask [1], PyArrow [4] and IPyParallel [3], have started implementing alternative serialization schemes with the explicit goal of avoiding copies on large data. Implementing a new serialization scheme is difficult and often leads to reduced generality (since many Python objects support pickle but not the new serialization scheme). Falling back on pickle for unsupported types is an option, but then you get back the spurious memory copies you wanted to avoid in the first place. For example, dask is able to avoid memory copies for Numpy arrays and built-in containers thereof (such as lists or dicts containing Numpy arrays), but if a large Numpy array is an attribute of a user-defined object, dask will serialize the user-defined object as a pickle stream, leading to memory copies.
The common theme of these third-party serialization efforts is to generate a stream of object metadata (which contains pickle-like information about the objects being serialized) and a separate stream of zero-copy buffer objects for the payloads of large objects. Note that, in this scheme, small objects such as ints, etc. can be dumped together with the metadata stream. Refinements can include opportunistic compression of large data depending on its type and layout, like dask does.
This PEP aims to make pickle usable in a way where large data is handled as a separate stream of zero-copy buffers, letting the application handle those buffers optimally.
To keep the example simple and avoid requiring knowledge of third-party libraries, we will focus here on a bytearray object (but the issue is conceptually the same with more sophisticated objects such as Numpy arrays). Like most objects, the bytearray object isn’t immediately understood by the pickle module and must therefore specify its decomposition scheme.
Here is how a bytearray object currently decomposes for pickling:
This is because the bytearray.__reduce_ex__ implementation reads morally as follows:
In turn it produces the following pickle code:
(the call to pickletools.optimize above is only meant to make the pickle stream more readable by removing the MEMOIZE opcodes)
We can notice several things about the bytearray’s payload (the sequence of bytes b'abc'):
What we really want is something like the following:
We see that several conditions are required for the above to work:
We are introducing a new type pickle.PickleBuffer which can be instantiated from any buffer-supporting object, and is specifically meant to be returned from __reduce__ implementations:
PickleBuffer is a simple wrapper that doesn’t have all the memoryview semantics and functionality, but is specifically recognized by the pickle module if protocol 5 or higher is enabled. It is an error to try to serialize a PickleBuffer with pickle protocol version 4 or earlier.
Only the raw data of the PickleBuffer will be considered by the pickle module. Any type-specific metadata (such as shapes or datatype) must be returned separately by the type’s __reduce__ implementation, as is already the case.
The PickleBuffer class supports a very simple Python API. Its constructor takes a single PEP 3118-compatible object. PickleBuffer objects themselves support the buffer protocol, so consumers can call memoryview(...) on them to get additional information about the underlying buffer (such as the original type, shape, etc.). In addition, PickleBuffer objects have the following methods:
raw()
release()
On the C side, a simple API will be provided to create and inspect PickleBuffer objects:
PyObject *PyPickleBuffer_FromObject(PyObject *obj)
PyPickleBuffer_Check(PyObject *obj)
const Py_buffer *PyPickleBuffer_GetBuffer(PyObject *picklebuf)
int PyPickleBuffer_Release(PyObject *picklebuf)
PickleBuffer can wrap any kind of buffer, including non-contiguous buffers. However, it is required that __reduce__ only returns a contiguous PickleBuffer (contiguity here is meant in the PEP 3118 sense: either C-ordered or Fortran-ordered). Non-contiguous buffers will raise an error when pickled.
This restriction is primarily an ease-of-implementation issue for the pickle module but also other consumers of out-of-band buffers. The simplest solution on the provider side is to return a contiguous copy of a non-contiguous buffer; a sophisticated provider, though, may decide instead to return a sequence of contiguous sub-buffers.
pickle.Pickler.__init__ and pickle.dumps are augmented with an additional buffer_callback parameter:
pickle.Unpickler.__init__ and pickle.loads are augmented with an additional buffers parameter:
Three new opcodes are introduced:
When pickling encounters a PickleBuffer, that buffer can be considered in-band or out-of-band depending on the following conditions:
An in-band buffer is serialized as follows:
An out-of-band buffer is serialized as follows:
The distinction between readonly and writable buffers is motivated below (see “Mutability”).
Even in-band pickling can be improved by returning a PickleBuffer instance from __reduce_ex__, as one copy is avoided on the serialization path [10] [12].
PEP 3118 buffers can be readonly or writable. Some objects, such as Numpy arrays, need to be backed by a mutable buffer for full operation. Pickle consumers that use the buffer_callback and buffers arguments will have to be careful to recreate mutable buffers. When doing I/O, this implies using buffer-passing API variants such as readinto (which are also often preferable for performance).
If you pickle and then unpickle an object in the same process, passing out-of-band buffer views, then the unpickled object may be backed by the same buffer as the original pickled object.
For example, it might be reasonable to implement reduction of a Numpy array as follows (crucial metadata such as shapes is omitted for simplicity):
Then simply passing the PickleBuffer around from dumps to loads will produce a new Numpy array sharing the same underlying memory as the original Numpy object (and, incidentally, keeping it alive):
This won’t happen with the traditional pickle API (i.e. without passing buffers and buffer_callback parameters), because then the buffer view is serialized inside the pickle stream with a copy.
The pickle persistence interface is a way of storing references to designated objects in the pickle stream while handling their actual serialization out of band. For example, one might consider the following for zero-copy serialization of bytearrays:
This mechanism has two drawbacks:
(the Python 2 cPickle module supported an undocumented inst_persistent_id() hook that was only called on non-built-in types; it was added in 1997 in order to alleviate the performance issue of calling persistent_id, presumably at ZODB’s request)
By passing a sequence of buffers, rather than a single buffer, we would potentially save on function call overhead in case a large number of buffers are produced during serialization. This would need additional support in the Pickler to save buffers before calling the callback. However, it would also prevent the buffer callback from returning a boolean to indicate whether a buffer is to be serialized in-band or out-of-band.
We consider that having a large number of buffers to serialize is an unlikely case, and decided to pass a single buffer to the buffer callback.
If we were to allow serializing a PickleBuffer in protocols 4 and earlier, it would actually make a supplementary memory copy when the buffer is mutable. Indeed, a mutable PickleBuffer would serialize as a bytearray object in those protocols (that is a first copy), and serializing the bytearray object would call bytearray.__reduce_ex__ which returns a bytes object (that is a second copy).
To prevent __reduce__ implementors from introducing involuntary performance regressions, we decided to reject PickleBuffer when the protocol is smaller than 5. This forces implementors to switch to __reduce_ex__ and implement protocol-dependent serialization, taking advantage of the best path for each protocol (or at least treat protocol 5 and upwards separately from protocols 4 and downwards).
The PEP was initially implemented in the author’s GitHub fork [6]. It was later merged into Python 3.8 [7].
A backport for Python 3.6 and 3.7 is downloadable from PyPI [8].
Support for pickle protocol 5 and out-of-band buffers was added to Numpy [11].
Support for pickle protocol 5 and out-of-band buffers was added to the Apache Arrow Python bindings [9].
Dask.distributed implements a custom zero-copy serialization with fallback to pickle [2].
PyArrow implements zero-copy component-based serialization for a few selected types [5].
PEP 554 proposes hosting multiple interpreters in a single process, with provisions for transferring buffers between interpreters as a communication scheme.
Thanks to the following people for early feedback: Alyssa Coghlan, Olivier Grisel, Stefan Krah, MinRK, Matt Rocklin, Eric Snow.
Thanks to Pierre Glaser and Olivier Grisel for experimenting with the implementation.
This document has been placed into the public domain.
Source: https://github.com/python/peps/blob/main/peps/pep-0574.rst
Last modified: 2025-02-01 08:59:27 UTC