Add a C API to configure the Python initialization without relying on C structures and the ability to make ABI-compatible changes in the future.
Complete PEP 587 API by adding PyInitConfig_AddModule() which can be used to add a built-in extension module; feature previously referred to as the “inittab”.
Add PyConfig_Get() and PyConfig_Set() functions to get and set the current runtime configuration.
PEP 587 “Python Initialization Configuration” unified all the ways to configure the Python initialization. This PEP unifies also the configuration of the Python preinitialization and the Python initialization in a single API. Moreover, this PEP only provides a single choice to embed Python, instead of having two “Python” and “Isolated” choices (PEP 587), to simplify the API further.
The lower level PEP 587 PyConfig API remains available for use cases with an intentionally higher level of coupling to CPython implementation details (such as emulating the full functionality of CPython’s CLI, including its configuration mechanisms).
PEP 587 has no API to get the current runtime configuration, only to configure the Python initialization.
For example, the global configuration variable Py_UnbufferedStdioFlag was deprecated in Python 3.12 and using PyConfig.buffered_stdio is recommended instead. It only works to configure Python, there is no public API to get PyConfig.buffered_stdio.
Users of the limited C API are asking for a public API to get the current runtime configuration.
Cython needs to get the optimization_level configuration option: issue.
When global configuration variables were deprecated in 2022, Marc-André Lemburg requested a C API to access these configuration variables at runtime (not only during Python initialization).
To fix CVE-2020-10735, a denial-of-service when converting a very large string to an integer (in base 10), it was discussed to add a new PyConfig member to stable branches which affects the ABI.
Gregory P. Smith proposed a different API using text based configuration file to not be limited by PyConfig members: FR: Allow private runtime config to enable extending without breaking the PyConfig ABI (August 2022).
In the end, it was decided to not add a new PyConfig member to stable branches, but only add a new PyConfig.int_max_str_digits member to the development branch (which became Python 3.12). A dedicated private global variable (unrelated to PyConfig) is used in stable branches.
The Python preinitialization uses the PyPreConfig structure and the Python initialization uses the PyConfig structure. Both structures have four duplicated members: dev_mode, parse_argv, isolated and use_environment.
The redundancy is caused by the fact that the two structures are separated, whereas some PyConfig members are needed by the preinitialization.
Examples:
On Linux, FreeBSD and macOS, applications are usually either statically linked to a libpython, or load dynamically a libpython . The libpython shared library is versioned, example: libpython3.12.so for Python 3.12 on Linux.
The vim project can target the stable ABI. Usually, the “system Python” version is used. It’s not currently possible to select which Python version to use. Users would like the ability to select a newer Python on demand.
On Linux, another approach to deploy an application embedding Python, such as GIMP, is to include Python in Flatpack, AppImage or Snap “container”. In this case, the application brings its own copy of Python version with the container.
Examples:
These utilities create standalone applications, they are not linked to libpython.
Marc-André Lemburg requested a C API to set the value of some configuration options at runtime:
Previously, it was possible to set directly global configuration variables:
But these configuration flags were deprecated in Python 3.12 and are scheduled for removal in Python 3.14.
Add C API functions and structure to configure the Python initialization:
Add C API functions to get and set the current runtime configuration:
The C API uses null-terminated UTF-8 encoded strings to refer to a configuration option name.
These C API functions are excluded from the limited C API.
The PyInitConfig structure is implemented by combining the three structures of the PyConfig API and has an inittab member as well:
The PyStatus status is no longer separated, but part of the unified PyInitConfig structure, which makes the API easier to use.
Configuration options are named after PyPreConfig and PyConfig structure members. See the PyPreConfig documentation and the PyConfig documentation.
Deprecating and removing configuration options is out of the scope of the PEP and should be discussed on a case by case basis.
Following options can be get by PyConfig_Get() and set and PyConfig_Set().
| argv | list[str] | API: sys.argv. |
| base_exec_prefix | str | API: sys.base_exec_prefix. |
| base_executable | str | API: sys._base_executable. |
| base_prefix | str | API: sys.base_prefix. |
| bytes_warning | int | API: sys.flags.bytes_warning. |
| exec_prefix | str | API: sys.exec_prefix. |
| executable | str | API: sys.executable. |
| inspect | bool | API: sys.flags.inspect (int). |
| int_max_str_digits | int | API: sys.flags.int_max_str_digits, sys.get_int_max_str_digits() and sys.set_int_max_str_digits(). |
| interactive | bool | API: sys.flags.interactive. |
| module_search_paths | list[str] | API: sys.path. |
| optimization_level | int | API: sys.flags.optimize. |
| parser_debug | bool | API: sys.flags.debug (int). |
| platlibdir | str | API: sys.platlibdir. |
| prefix | str | API: sys.base_prefix. |
| pycache_prefix | str | API: sys.pycache_prefix. |
| quiet | bool | API: sys.flags.quiet (int). |
| stdlib_dir | str | API: sys._stdlib_dir. |
| use_environment | bool | API: sys.flags.ignore_environment (int). |
| verbose | int | API: sys.flags.verbose. |
| warnoptions | list[str] | API: sys.warnoptions. |
| write_bytecode | bool | API: sys.flags.dont_write_bytecode (int) and sys.dont_write_bytecode (bool). |
| xoptions | dict[str, str] | API: sys._xoptions. |
Some option names are different than sys attributes, such as optimization_level option and sys.flags.optimize attribute. PyConfig_Set() sets the corresponding sys attribute.
The xoptions is a list of strings in PyInitConfig where each string has the format key (value is True implicitly) or key=value. In the current runtime configuration, it becomes a dictionary (key: str → value: str | True).
Following options can be get by PyConfig_Get(), but cannot be set by PyConfig_Set().
| allocator | int | |
| buffered_stdio | bool | |
| check_hash_pycs_mode | str | |
| code_debug_ranges | bool | |
| coerce_c_locale | bool | |
| coerce_c_locale_warn | bool | |
| configure_c_stdio | bool | |
| configure_locale | bool | |
| cpu_count | int | API: os.cpu_count() (int | None). |
| dev_mode | bool | API: sys.flags.dev_mode. |
| dump_refs | bool | |
| dump_refs_file | str | |
| faulthandler | bool | API: faulthandler.is_enabled(). |
| filesystem_encoding | str | API: sys.getfilesystemencoding(). |
| filesystem_errors | str | API: sys.getfilesystemencodeerrors(). |
| hash_seed | int | |
| home | str | |
| import_time | bool | |
| install_signal_handlers | bool | |
| isolated | bool | API: sys.flags.isolated (int). |
| legacy_windows_fs_encoding | bool | Windows only. |
| legacy_windows_stdio | bool | Windows only. |
| malloc_stats | bool | |
| orig_argv | list[str] | API: sys.orig_argv. |
| parse_argv | bool | |
| pathconfig_warnings | bool | |
| perf_profiling | bool | API: sys.is_stack_trampoline_active(). |
| program_name | str | |
| run_command | str | |
| run_filename | str | |
| run_module | str | |
| run_presite | str | need a debug build. |
| safe_path | bool | |
| show_ref_count | bool | |
| site_import | bool | API: sys.flags.no_site (int). |
| skip_source_first_line | bool | |
| stdio_encoding | str | API: sys.stdin.encoding, sys.stdout.encoding and sys.stderr.encoding. |
| stdio_errors | str | API: sys.stdin.errors, sys.stdout.errors and sys.stderr.errors. |
| tracemalloc | int | API: tracemalloc.is_tracing() (bool). |
| use_frozen_modules | bool | |
| use_hash_seed | bool | |
| user_site_directory | bool | API: sys.flags.no_user_site (int). |
| utf8_mode | bool | |
| warn_default_encoding | bool | |
| _pystats | bool | API: sys._stats_on(), sys._stats_off(). Need a Py_STATS build. |
It must be freed with PyInitConfig_Free().
Return NULL on memory allocation failure.
void PyInitConfig_Free(PyInitConfig *config):Free memory of an initialization configuration.The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
int PyInitConfig_HasOption(PyInitConfig *config, const char *name):Test if the configuration has an option called name.Return 1 if the option exists, or return 0 otherwise.
int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value):Get an integer configuration option.On success, the string must be released with free(value).
int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items):Get a string list configuration option as an array of null-terminated UTF-8 encoded strings.On success, the string list must be released with PyInitConfig_FreeStrList(length, items).
void PyInitConfig_FreeStrList(size_t length, char **items):Free memory of a string list created by PyInitConfig_GetStrList().The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
Some configuration options have side effects on other options. This logic is only implemented when Py_InitializeFromInitConfig() is called, not by the “Set” functions below. For example, setting dev_mode to 1 does not set faulthandler to 1.
int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value):Set an integer configuration option.The new module can be imported by the name name, and uses the function initfunc as the initialization function called on the first attempted import.
If Python is initialized multiple times, PyInitConfig_AddModule() must be called at each Python initialization.
Similar to the PyImport_AppendInittab() function.
See PyInitConfig_GetExitcode() for the exitcode case.
An error message is an UTF-8 encoded string.
If config has an exit code, format the exit code as an error message.
The error message remains valid until another PyInitConfig function is called with config. The caller doesn’t have to free the error message.
int PyInitConfig_GetExitcode(PyInitConfig* config, int *exitcode):Get the config exit code.Only the Py_InitializeFromInitConfig() function can set an exit code if the parse_argv option is non-zero.
An exit code can be set when parsing the command line failed (exit code 2) or when a command line option asks to display the command line help (exit code 0).
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
PyObject* PyConfig_Get(const char *name):Get the current runtime value of a configuration option as a Python object.The object type depends on the option: see Configuration Options tables.
Other options are get from internal PyPreConfig and PyConfig structures.
The caller must hold the GIL. The function cannot be called before Python initialization nor after Python finalization.
int PyConfig_GetInt(const char *name, int *value):Similar to PyConfig_Get(), but get the value as an integer.Set an exception and return NULL on error.
The caller must hold the GIL.
PyObject* PyConfig_Set(const char *name, PyObject *value):Set the current runtime value of a configuration option.Read-only configuration options cannot be set.
The caller must hold the GIL. The function cannot be called before Python initialization nor after Python finalization.
The behavior of options, the default option values, and the Python behavior can change at each Python version: they are not “stable”.
Moreover, configuration options can be added, deprecated and removed following the usual PEP 387 deprecation process.
The lower level PEP 587 PyPreConfig and PyConfig APIs remain available and fully supported. As noted in the Abstract, they remain the preferred approach for embedding use cases that are aiming to closely emulate the behaviour of the full CPython CLI, rather than just making a Python runtime available as part of a larger application.
The PyPreConfig APIs may be used in combination with the initialization API in this PEP. In such cases, the read-only vs read/write restrictions for preconfiguration settings apply to PyInitConfig_SetInt in addition to PyConfig_Set once the interpreter has been preconfigured (specifically, only use_environment may be updated, attempting to update any of the other preconfiguration variables will report an error).
Example initializing Python, set configuration options of various types, return -1 on error:
Example increasing the bytes_warning option of an initialization configuration:
Example getting the current runtime value of the configuration option verbose:
On error, the function silently ignores the error and returns -1. In practice, getting the verbose option cannot fail, unless a future Python version removes the option.
Changes are fully backward compatible. Only new APIs are added.
Existing API such as the PyConfig C API (PEP 587) are left unchanged.
It was proposed to provide the configuration as text to make the API compatible with the stable ABI and to allow custom options.
Example:
The API would take the configuration as a string, not as a file. Example with a hypothetical PyInit_SetConfig() function:
The example ignores error handling to make it easier to read.
The problem is that generating such configuration text requires adding quotes to strings and to escape quotes in strings. Formatting an array of strings becomes non-trivial.
Providing an API to format a string or an array of strings is not really worth it, whereas Python can provide directly an API to set a configuration option where the value is passed directly as a string or an array of strings. It avoids giving special meaning to some characters, such as newline characters, which would have to be escaped.
Using strings to refer to a configuration option requires comparing strings which can be slower than comparing integers.
Use integers, similar to type “slots” such as Py_tp_doc, to refer to a configuration option. The const char *name parameter is replaced with int option.
Accepting custom options is more likely to cause conflicts when using integers, since it’s harder to maintain “namespaces” (ranges) for integer options. Using strings, a simple prefix with a colon separator can be used.
Integers also requires maintaining a list of integer constants and so make the C API and the Python API larger.
Python 3.13 only has around 62 configuration options, and so performance is not really a blocker issue. If better performance is needed later, a hash table can be used to get an option by its name.
If getting a configuration option is used in hot code, the value can be read once and cached. By the way, most configuration options cannot be changed at runtime.
Eric Snow expressed concerns that this proposal might reinforce with embedders the idea that initialization is a single monolithic step. He argued that initialization involves 5 distinct phases and even suggested that the API should reflect this explicitly. Eric proposed that, at the very least, the implementation of initialization should reflect the phases, in part for improved code health. Overall, his explanation has some similarities with PEP 432 and PEP 587.
Another of Eric’s key points relevant to this PEP was that, ideally, the config passed to Py_InitializeFromConfig() should be complete before that function is called, whereas currently initialization actually modifies the config.
While Eric wasn’t necessarily suggesting an alternative to PEP 741, any proposal to add a granular initialization API around phases is effectively the opposite of what this PEP is trying to accomplish. Such API is more complicated, it requires adding new public structures and new public functions. It makes the Python initialization more complicated, rather than this PEP tries to unify existing APIs and make them simpler (the opposite). Having multiple structures for similar purpose can lead to duplicate members, similar issue than duplicated members between existing PyPreConfig and PyConfig structures.
Accepting strings encoded to the locale encoding and accepting wide strings (wchar_t*) in the PyInitConfig API was deferred to keep the PyInitConfig API simple and avoid the complexity of the Python preinitialization. These features are also mostly needed when emulating the full CPython CLI behaviour, and hence better served by the lower level PEP 587 API.
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0741.rst
Last modified: 2024-09-03 13:37:25 UTC