Skip to content

generate_tokens starts to give SyntaxError #105549

Description

@sylee957

Bug report

generate_tokens gives error in python 3.12

from tokenize import generate_tokens
from io import StringIO
list(generate_tokens(StringIO('01234').readline))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sylee957/.pyenv/versions/3.12.0b1/lib/python3.12/tokenize.py", line 451, in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
  File "/home/sylee957/.pyenv/versions/3.12.0b1/lib/python3.12/tokenize.py", line 542, in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
  File "<string>", line 1
    01234
    ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

While in Python 3.11, it successfully returns some result

[TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='01234'), TokenInfo(type=2 (NUMBER), string='1234', start=(1, 1), end=(1, 5), line='01234'), TokenInfo(type=4 (NEWLINE), string='', start=(1, 5), end=(1, 6), line=''), TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

This is related to some issues in SymPy
sympy/sympy#25185

Your environment

  • CPython versions tested on: 3.12.0b1
  • Operating system and architecture: Ubuntu 22.04.2 LTS

Linked PRs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

3.12only security fixes3.13only security fixestype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions