bpo-43510: Implement PEP 597 opt-in EncodingWarning.#19481

methane

See PEP 597.

Add -X warn_default_encoding and PYTHONWARNDEFAULTENCODING.
Add EncodingWarning
Add io.text_encoding()
open(), TextIOWrapper() emits EncodingWarning when encoding is omitted and warn_default_encoding is enabled.
_pyio.TextIOWrapper() uses UTF-8 as fallback default encoding used when failed to import locale module. (used during building Python)
bz2, configparser, gzip, lzma, pathlib, tempfile modules use io.text_encoding().
Add documents
Add what's new entry

bpo-43510

CAM-Gerlach

Thanks @methane ! I've replied to your comments with updated suggestions to reflect your feedback.

methane

I will merge this PR next week if there are no objections.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO + PGO 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/96/builds/910) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/96/builds/910

Failed tests:

test_tools
test_ftplib

Failed subtests:

test_storlines - test.test_ftplib.TestFTPClass
test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 38 sec
test_shelve: 2 min 30 sec
test_mailbox: 1 min 57 sec
test_dbm: 1 min 52 sec
test_tarfile: 1 min 36 sec
test_multiprocessing_spawn: 1 min 34 sec
test_tokenize: 1 min 33 sec
test_dbm_gnu: 1 min 28 sec
test_unparse: 1 min 28 sec
test_asyncio: 1 min 17 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

2 re-run tests:
test_ftplib test_tools

Total duration: 6 min 19 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 623, in test_storlines
    self.check_data(self.server.handler_instance.last_received_data, RETR_DATA)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_ftplib.py", line 494, in check_data
    self.assertEqual(len(received), len(expected))
AssertionError: 12019 != 12018


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto-pgo/build/Lib/test/bad_coding.py']

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 LTO 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/507/builds/896) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/507/builds/896

Failed tests:

test_tools

Failed subtests:

test_reindent_file_with_bad_encoding - test.test_tools.test_reindent.ReindentTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_mailbox: 4 min 33 sec
test_peg_generator: 3 min 37 sec
test_concurrent_futures: 2 min 56 sec
test_shelve: 2 min 27 sec
test_multiprocessing_spawn: 1 min 50 sec
test_tokenize: 1 min 49 sec
test_unparse: 1 min 37 sec
test_lib2to3: 1 min 25 sec
test_asyncio: 1 min 21 sec
test_multiprocessing_forkserver: 1 min 9 sec

1 test failed:
test_tools

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_tools

Total duration: 7 min 15 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/test_tools/test_reindent.py", line 29, in test_reindent_file_with_bad_encoding
    rc, out, err = assert_python_ok(self.script, '-r', bad_coding_path)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/python', '-X', 'faulthandler', '-I', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Tools/scripts/reindent.py', '-r', '/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64.lto/build/Lib/test/bad_coding.py']

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 4827483.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/146/builds/1014) and take a look at the build logs.
Check if the failure is related to this commit (4827483) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/146/builds/1014

Failed tests:

test_locale

Failed subtests:

test_getsetlocale_issue1813 - test.test_locale.TestMiscellaneous

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

395 tests OK.

10 slowest tests:

test_mmap: 9 min 37 sec
test_largefile: 5 min 57 sec
test_io: 5 min 39 sec
test_unparse: 3 min 12 sec
test_multiprocessing_spawn: 2 min 16 sec
test_concurrent_futures: 2 min 3 sec
test_tokenize: 1 min 46 sec
test_capi: 1 min 46 sec
test_peg_generator: 1 min 39 sec
test_asyncio: 1 min 31 sec

1 test failed:
test_locale

31 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxlimited test_xxtestfuzz test_zipfile64

1 re-run test:
test_locale

Total duration: 19 min 35 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_locale.py", line 567, in test_getsetlocale_issue1813
    locale.setlocale(locale.LC_CTYPE, loc)
  File "D:\buildarea\3.x.bolen-windows10\build\lib\locale.py", line 610, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

methane

test_locale fail is unrelating to this PR. See bpo-37945.

CAM-Gerlach

Thanks @methane ! Excited to see this in Python 3.10.

methane added the DO-NOT-MERGE label Apr 12, 2020

the-knights-who-say-ni added the CLA signed label Apr 12, 2020

bedevere-bot added the awaiting core review label Apr 12, 2020

methane force-pushed the open-encoding branch 4 times, most recently from c37295e to 6f72b38 Compare April 13, 2020 09:22

methane added 3 commits January 29, 2021 17:36

Raise a warning when encoding is omitted …

a3c014b

The warning is raised only in dev mode. pathlib uses the new `io.text_encoding` helper function. Other libraries will follow.

add test

050bd1b

wrap encoding=None with text_encoding.

939f4a0

methane force-pushed the open-encoding branch from e44dc5b to 939f4a0 Compare January 29, 2021 08:37

Add io.LOCALE_ENCODING = "locale"

3c99777

methane changed the title PEP 597: Raise a warning when encoding is omitted PEP 597: Emit EncodingWarning when encoding is omitted Jan 30, 2021

methane added 2 commits January 30, 2021 17:52

Add EncodingWarning. …

4016278

Copied from python#13938

Add sys.warn_default_encoding

c5c556c

methane force-pushed the open-encoding branch from db537b9 to c5c556c Compare January 30, 2021 08:52

methane added 11 commits January 30, 2021 18:04

shorten option names

d9a08c2

EncodingWarning extends Warning

772648e

make clinic

1a8e305

fix test

20966cd

remove wrong test case

2b80f42

fix exception_hierarchy.txt

760308c

Make sys.flags.encoding_warning int

a95dff2

Fix text_embed.

31fb411

Fix test_pickle

096a0a3

configparser: use io.text_encoding()

99fc938

Rename option names …

6fdbcbc

sys.flags.encoding_warning -> warn_default_encoding -X warn_encoding -> warn_default_encoding PYTHONWARNENCODING -> PYTHONWARNDEFAULTENCODING

methane changed the title PEP 597: Emit EncodingWarning when encoding is omitted bpo-43510: Add encoding="locale" and optional EncodingWarning Mar 16, 2021

methane removed the DO-NOT-MERGE label Mar 16, 2021

Merge remote-tracking branch 'upstream/master' into open-encoding

3f362bc

methane and others added 8 commits March 23, 2021 14:48

Update Doc/library/io.rst …

54c7dc6

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

Update Doc/library/io.rst …

5b2830b

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

Apply suggestions from code review …

14f2a6e

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

Move EncodingWarnings

06e2a32

fix comment

27d49d2

fix text_encoding() docstring

80f4644

update what's new

6ad0e7f

fix doc build

73b27f1

CAM-Gerlach reviewed Mar 23, 2021

View reviewed changes

Comment thread Lib/_pyio.py Outdated Show resolved Hide resolved

Update Doc/library/io.rst …

c149d65

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

methane commented Mar 24, 2021

View reviewed changes

Comment thread Lib/_pyio.py Outdated Show resolved Hide resolved

Apply suggestions from code review …

4eb7655

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

methane commented Mar 24, 2021

View reviewed changes

Comment thread Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved

Apply suggestions from code review …

e3bce76

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

CAM-Gerlach reviewed Mar 24, 2021

View reviewed changes

Comment thread Doc/library/io.rst Outdated Show resolved Hide resolved

Update Doc/library/io.rst …

c089fd7

Co-authored-by: CAM Gerlach <CAM.Gerlach@Gerlach.CAM>

methane merged commit 4827483 into python:master Mar 29, 2021

methane deleted the open-encoding branch March 29, 2021 03:28

bedevere-bot removed the awaiting core review label Mar 29, 2021

alexdelorenzo mentioned this pull request Apr 16, 2021

Python 3.10 alpha 7 introduces new changes to pathlib alexdelorenzo/aiopath#4

Open

chris-allan mentioned this pull request Oct 10, 2022

Update to a suitable version of the concurrent-log-handler ome/omero-web#415

Merged

AlexWaygood mentioned this pull request Oct 26, 2023

text_encoding() is missing from io.__all__ #111356

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-43510: Implement PEP 597 opt-in EncodingWarning.#19481