I am not a fan of this solution as this is coupling the tokenizer and the grammar, which is a known problematic behaviour.

In particular, I don't like the fact that we need to manually check against every keyword explicitly.

Cannot we just error if we find any character that is not a whitespace or a valid numeric literal?

We should try to fix this in a way they is keyword-agnostic

Checking against the list of valid keywords is a temporary code. It is because we want to do gradual deprecation and only emit a warning for currently valid code. At the end, there will be only syntax error and no list of keywords.

If emit a warning unconditionally, this will break some tests for invalid numeric literals (test_bad_numerical_literals in test_grammar.py and test_literals_with_leading_zeroes in test_compile.py). Some test cases will produce warning and then error, so we would need to rewrite these tests significantly, and several releases later after converting warnings to error restore original tests. Double warning-error can also confuse users.

Checking against the list of valid keywords is a temporary code. It is because we want to do gradual deprecation and only emit a warning for currently valid code. At the end, there will be only syntax error and no list of keywords.

If emit a warning unconditionally, this will break some tests for invalid numeric literals (test_bad_numerical_literals in test_grammar.py and test_literals_with_leading_zeroes in test_compile.py). Some test cases will produce warning and then error, so we would need to rewrite these tests significantly, and several releases later after converting warnings to error restore original tests. Double warning-error can also confuse users.

Thanks for the clarification. Could you add some comments regarding this plans to the source code of the change to make sure this context is there when reading the code?

In that case I assume that the main thing to do here is to all agree on the plan.

I personally find the plan reasonable so I'm on board, but I would go directly to SyntaxWarning and in 1 or 2 releases making it a SyntaxError.

Sure. I added a comment in verify_end_of_number(), but seems it is not enough. I'll expand by adding explanation of the intention of this behavior.

As for SyntaxWarning vs DeprecationWarning, we still emit DeprecationWarning for invalid character escapes (like in "C:\Program Files"). Initially SyntaxWarning was emitted, but after public discussion it was changed to DeprecationWarning.

@pablogsal, do you want to add these warnings in 3.10 or 3.11 if add them at all?

@pablogsal, do you want to add these warnings in 3.10 or 3.11 if add them at all?

I would like them to be included in Python 3.10 so we can make them illegal sooner. What do you think?

@serhiy-storchaka: Status check is done, and it's a success ✅ .

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

GH-26614 is a backport of this pull request to the 3.10 branch.