Skip to content

gh-155525: Cover quadratic f-string tokenization regression - #156756

Merged
pablogsal merged 3 commits into
python:mainfrom
gwosti:perf/fstring-expression-span
Sep 19, 2026
Merged

pablogsal merged 3 commits into
python:mainfrom
gwosti:perf/fstring-expression-span

Conversation

@gwosti

@gwosti gwosti commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Add regression coverage for modules containing many f-string or t-string replacement fields, expression text after tokenizer buffer growth, and != followed by a format specifier.

The tokenizer refactor in gh-156484 already removed the quadratic copying on main. This PR now retains that implementation and adds the original regression tests. The release-branch backports also need the original expression-span fix, adapted to their older tokenizer layout.

Closes gh-155525.

@python-cla-bot

python-cla-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Comment thread Parser/lexer/string.c Outdated
Comment thread Lib/test/test_fstring.py Outdated
Comment thread Parser/lexer/state.h Outdated
@gwosti
gwosti requested a review from pablogsal September 19, 2026 19:40
@pablogsal pablogsal changed the title gh-155525: Fix quadratic complexity in f-string tokenization gh-155525: Cover quadratic f-string tokenization regression Sep 19, 2026
@pablogsal pablogsal added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Sep 19, 2026
@pablogsal
pablogsal enabled auto-merge (squash) September 19, 2026 21:35
@pablogsal
pablogsal merged commit c1df684 into python:main Sep 19, 2026
58 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @gwosti for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

GH-157828 is a backport of this pull request to the 3.15 branch.

@miss-islington-app

Copy link
Copy Markdown

Sorry, @gwosti and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker c1df6843d36233ec1da71a1d5f9b74dd6ebc9b97 3.13

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 19, 2026
@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

GH-157829 is a backport of this pull request to the 3.14 branch.

@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

GH-157828 is a backport of this pull request to the 3.15 branch.

@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

GH-157829 is a backport of this pull request to the 3.14 branch.

@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

GH-157830 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Sep 19, 2026
pablogsal added a commit that referenced this pull request Sep 19, 2026
…57830)

[3.13] gh-155525: Cover quadratic f-string tokenization regression (GH-156756)

* gh-155525: Avoid quadratic f-string tokenization

* fixup! gh-155525: Avoid quadratic f-string tokenization

---------
(cherry picked from commit c1df684)

Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
pablogsal added a commit that referenced this pull request Sep 19, 2026
…57829)

* [ 3.14 ] gh-155525: Avoid quadratic f-string tokenization

Adapt the original expression-span fix to the release tokenizer and retain interactive buffers while a formatted string is open.

Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>

* gh-155525: Cover quadratic f-string tokenization regression (GH-156756)

* gh-155525: Avoid quadratic f-string tokenization

* fixup! gh-155525: Avoid quadratic f-string tokenization

---------
(cherry picked from commit c1df684)

Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

---------

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
pablogsal added a commit that referenced this pull request Sep 19, 2026
…57828)

* [ 3.15 ] gh-155525: Avoid quadratic f-string tokenization

Adapt the original expression-span fix to the release tokenizer and retain interactive buffers while a formatted string is open.

Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>

* gh-155525: Cover quadratic f-string tokenization regression (GH-156756)

* gh-155525: Avoid quadratic f-string tokenization

* fixup! gh-155525: Avoid quadratic f-string tokenization

---------
(cherry picked from commit c1df684)

Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

---------

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Non-Debug 3.13 (tier-1) has failed when building commit 3dcc324.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1419/builds/1334) and take a look at the build logs.
  4. Check if the failure is related to this commit (3dcc324) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1419/builds/1334

Failed tests:

  • test_int

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "b:\uildarea\3.13.ware-win11.nondebug\build\Lib\test\test_int.py", line 729, in test_denial_of_service_prevented_str_to_int
    self.assertLessEqual(sw_fail_huge.seconds, sw_convert.seconds/2)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0.015625 not less than or equal to 0.0078125


Traceback (most recent call last):
  File "b:\uildarea\3.13.ware-win11.nondebug\build\Lib\test\test_int.py", line 740, in test_denial_of_service_prevented_str_to_int
    self.assertLessEqual(sw_fail_extra_huge.seconds, sw_convert.seconds/2)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0.015625 not less than or equal to 0.0078125

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants