Symbols and functions exported by the <token.h> header file cannot be used outside Python internals. There is no public C API to tokenize a Python source file. I propose to simply remove this header file. It's not included by Python.h.
In Python 3.10, I removed the C API related to AST and the parser, since it was not possible to use it outside Python internals. See: https://docs.python.org/dev/whatsnew/3.10.html#id4
Reactions are currently unavailable
Symbols and functions exported by the <token.h> header file cannot be used outside Python internals. There is no public C API to tokenize a Python source file. I propose to simply remove this header file. It's not included by Python.h.
In Python 3.10, I removed the C API related to AST and the parser, since it was not possible to use it outside Python internals. See: https://docs.python.org/dev/whatsnew/3.10.html#id4