This may break old Python 2 in subprocess.

I'm sorry, my environment was wrong.
I accidently test above in Lib/ directory in cpython's checkout.

It still raises a good question though, as that setting does affect Python 2 differently from the way it affects Python 3 - it changes the implicit encoding step on stdout, but stdin still relies on passing the raw bytes through without interpretation:

It's also a potential problem that 'surrogateescape' doesn't exist in Python 2, so it may be better to just use Py_SetStandardStreamEncoding in PEP 538, and leave enabling surrogateescape in subprocesses as well to PEP 540 (via PYTHONUTF8=1 in the parent environment).

It also turns out that LANG=C python2 is an easy way to demonstrate that GNU readline just plain doesn't handle UTF-8 properly in the C locale - attempting to edit the print(u"こんにちは") line at the interactive prompt to remove the u prefix or add it back results in nonsense:

Am I reading this right? It seems odd that setting PYTHONCOERCECLOCALE would disable coercing the legacy locale. The sense is exactly opposite of what I'd expect.

Should the envar be named PYTHONNOCOERCECLOCALE?

Also, what if the environment variable is set to "0"? Given the above description, that should still skip coercion.

Oops, that's a holdover from when the setting was PYTHONALLOWCLOCALE - presumably I changed the title of the section, but then got distracted by something else before updating the body.

Fixed in 1c3a270

Does the value of the envar matter or not? It just needs to be any string value, right?

Docs bug - the warning matches the behaviour, the docs don't.

Docs fixed in 1c3a270

Does the order matter? If not, what about using for env_var in base_var_dict:?

Done in d12b412

And here...

Done in d12b412

Doesn't this contradict the documentation at the top of this PR?

Docs fixed in 1c3a270

What about using self.addCleanup(os.chdir, self.oldcwd) in the setUp() instead and getting rid of the tearDown()?

Good point - I stole the basic structure from test_capi (simplified a bit since this doesn't need to run on Windows), so it's missing some modern niceties. With addCleanup, we wouldn't even need self.oldcwd as a record, we can just add the cleanup operation before changing the directory:

Done in d12b412

Does it make any sense to run this without universal_newlines also? Since the locale changes should only affect text mode, that additional test would just ensure that nothing's changed when running subprocesses in bytes mode.

For this particular test, I don't think so, as it's just checking that the runtime warning gets emitted when we bypass the locale coercion in the standard Python CLI, rather than doing anything with the standard streams.

However, it does make sense for test_locale_coercion itself (which I should rename to test_c_locale_coercion): that should check the behaviour when the Python subprocess is run with the -u option for unbuffered binary streams.

So it turns out that even -u doesn't change anything of interest to this PEP any more, and there's just an error in the --help docs that makes it sound like it still does: http://bugs.python.org/issue28647

However, now that the PEP potentially also affects the way the standard streams are configured, I expanded on the test cases to also cover that: 4e6d502

There's been some discussion about the "legacy C locale" nomenclature. What about "bare C locale"?

Even glibc devs consider 7-bit ASCII a legacy encoding at this point, it's just an enormous ecosystem to try to migrate to a new default way of doing things.

My view would likely be different if we were planning to invest time and energy in getting Python 3 to "work properly" in the C locale, but the underlying assumptions are sufficiently incompatible that it isn't even clear what "working properly" would even mean. (PEP 540 gets closer than the status quo, but it's definition of "working properly" is "ignoring the declared locale encoding as almost certainly being wrong")

This seems to come up in every project I work on. Do we use UK or US spelling? :)

I think in Python we use US spellings.

Yeah, I'll typically use my native spellings when writing PEPs and email, but anything in the actual code or docs should be using the US spelling.

Fixed in ccfc83f

You do this same check in two places. Would it make sense to refactor that into a call or macro so there's no chance in getting out of sync? (See my question at the top of the PR.)

An alternative would be to check a C global here, rather than checking the environment variable directly, and have a _Py_DisableCLocaleWarning() helper that the CLI calls when PYTHONCOERCECLOCALE=0 is set.

The downside of that approach would be that it means embedding applications can't use PYTHONCOERCECLOCALE=0 to silence the warning without changing the configured locale, and I don't want to define a new public embedding configuration API just to avoid checking the environment variable twice.

So I'll unconditionally export a private helper function from pylifecycle.c (so the ABI doesn't depend on the configure flag setting), and re-use that from the CLI implementation.

These LC_ALL should be updated.

@zooba This is the bit that I suspect may be unintentionally changing the interpreter startup behaviour on Windows and thus causing the Appveyor test failures: there's currently nothing restricting this particular code path to only run on non-Windows systems, so I suspect it may be triggering and activating surrogateescape by default on the Windows standard streams, even though Windows uses its own code page system, rather than the POSIX locale system. If I'm right, then the entire body of this function should be inside some kind of #ifdef, such that the Windows case degenerates to return NULL;

Possibly. The only thing that is available for coercion on Windows is locale.getpreferredencoding(), and there are no results from setlocale() that you can reasonably interpret as "wrong".

When/if the "force UTF-8" option comes along, that should apply to Windows, but I don't think we need to determine the error handler here at all for Windows.

Comparing this to the old code (as a result of the test_sys failure), it seems we were setting surrogateescape by default in the C locale, even on Windows. So I've had a second go at fixing this by keeping that, and only skipping the new checks for the coercion target locales.

It turned out this wasn't the problem, but rather the earlier call to setlocale(LC_CTYPE, ""), where I had removed a HAVE_SETLOCALE guard as being redundant in 2017. It turns out that even though Windows does have setlocale available:

1. Passing an empty string doesn't work the same way it does in glibc (presumably it syncs the active C locale with the active Windows code page rather than looking at environment variables)
2. The Windows builds don't define HAVE_SETLOCALE, so that guard was previously serving as a cryptic #ifndef MS_WINDOWS check

So rather than restoring the original guard, I replaced it with an explicit #ifndef MS_WINDOWS check instead.

Per @zooba: I need to explain the (lack of) security implications here, since:

• the envvar can only be used to turn off locale coercion, it doesn't do anything else (i.e. it's a boolean toggle, not input data)
• if an attacker is in a position to corrupt the behaviour of the ascii encoding, they're likely to be able to attack the utf-8 encoding just as effectively

👍

This appears to be an unrelated change.

You're right that it's technically unrelated now, but an earlier iteration of the patch failed in debug mode without it - that iteration was using Py_SetStandardStreamEncoding, and when you do that, you allocate some memory that gets freed during interpreter startup after the default allocator has been configured. Without this change, that free attempt failed in debug mode due to the allocator mismatch.

So these guards mean that the "safe to call before Py_Initialize" APIs actually are safe to call from the CLI entry point.