PEP 623 – Remove wstr from Unicode | peps.python.org Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

Python »
PEP Index »
PEP 623

PEP 623 – Remove wstr from Unicode

Author: Inada Naoki <songofacandy at gmail.com> BDFL-Delegate: Victor Stinner <vstinner at python.org> Discussions-To: Python-Dev thread Status: Final Type: Standards Track Created: 25-Jun-2020 Python-Version: 3.10 Resolution: Python-Dev thread Table of Contents

Abstract

PEP 393 deprecated some unicode APIs, and introduced wchar_t *wstr, and Py_ssize_t wstr_length in the Unicode structure to support these deprecated APIs.

This PEP is planning removal of wstr, and wstr_length with deprecated APIs using these members by Python 3.12.

Deprecated APIs which doesn’t use the members are out of scope because they can be removed independently.

Motivation

Memory usage

str is one of the most used types in Python. Even most simple ASCII strings have a wstr member. It consumes 8 bytes per string on 64-bit systems.

Runtime overhead

To support legacy Unicode object, many Unicode APIs must call PyUnicode_READY().

We can remove this overhead too by dropping support of legacy Unicode object.

Simplicity

Supporting legacy Unicode object makes the Unicode implementation more complex. Until we drop legacy Unicode object, it is very hard to try other Unicode implementation like UTF-8 based implementation in PyPy.

Rationale

Python 4.0 is not scheduled yet

PEP 393 introduced efficient internal representation of Unicode and removed border between “narrow” and “wide” build of Python.

PEP 393 was implemented in Python 3.3 which is released in 2012. Old APIs were deprecated since then, and the removal was scheduled in Python 4.0.

Python 4.0 was expected as next version of Python 3.9 when PEP 393 was accepted. But the next version of Python 3.9 is Python 3.10, not 4.0. This is why this PEP schedule the removal plan again.

Py_UNICODE_WSTR_LENGTH()
PyUnicode_GET_SIZE()
PyUnicode_GetSize()
PyUnicode_GET_DATA_SIZE()
PyUnicode_AS_UNICODE()
PyUnicode_AS_DATA()
PyUnicode_AsUnicode()
_PyUnicode_AsUnicode()
PyUnicode_AsUnicodeAndSize()
PyUnicode_FromUnicode()

Python 3.10

Following macros, enum members are marked as deprecated. Py_DEPRECATED(3.10) macro are used as possible. But they are deprecated only in comment and document if the macro can not be used easily.
- PyUnicode_WCHAR_KIND
- PyUnicode_READY()
- PyUnicode_IS_READY()
- PyUnicode_IS_COMPACT()
PyUnicode_FromUnicode(NULL, size) and PyUnicode_FromStringAndSize(NULL, size) emit DeprecationWarning when size > 0.
PyArg_ParseTuple() and PyArg_ParseTupleAndKeywords() emit DeprecationWarning when u, u#, Z, and Z# formats are used.

Python 3.12

Following members are removed from the Unicode structures:
- wstr
- wstr_length
- state.compact
- state.ready
The PyUnicodeObject structure is removed.
Following macros and functions, and enum members are removed:
- Py_UNICODE_WSTR_LENGTH()
- PyUnicode_GET_SIZE()
- PyUnicode_GetSize()
- PyUnicode_GET_DATA_SIZE()
- PyUnicode_AS_UNICODE()
- PyUnicode_AS_DATA()
- PyUnicode_AsUnicode()
- _PyUnicode_AsUnicode()
- PyUnicode_AsUnicodeAndSize()
- PyUnicode_FromUnicode()
- PyUnicode_WCHAR_KIND
- PyUnicode_READY()
- PyUnicode_IS_READY()
- PyUnicode_IS_COMPACT()
PyUnicode_FromStringAndSize(NULL, size)) raises RuntimeError when size > 0.
PyArg_ParseTuple() and PyArg_ParseTupleAndKeywords() raise SystemError when u, u#, Z, and Z# formats are used, as other unsupported format character.