gh-155525: Cover quadratic f-string tokenization regression - #156756
Conversation
|
Thanks @gwosti for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15. |
|
GH-157828 is a backport of this pull request to the 3.15 branch. |
|
Sorry, @gwosti and @pablogsal, I could not cleanly backport this to |
|
GH-157829 is a backport of this pull request to the 3.14 branch. |
|
GH-157828 is a backport of this pull request to the 3.15 branch. |
|
GH-157829 is a backport of this pull request to the 3.14 branch. |
|
GH-157830 is a backport of this pull request to the 3.13 branch. |
…57830) [3.13] gh-155525: Cover quadratic f-string tokenization regression (GH-156756) * gh-155525: Avoid quadratic f-string tokenization * fixup! gh-155525: Avoid quadratic f-string tokenization --------- (cherry picked from commit c1df684) Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
…57829) * [ 3.14 ] gh-155525: Avoid quadratic f-string tokenization Adapt the original expression-span fix to the release tokenizer and retain interactive buffers while a formatted string is open. Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com> * gh-155525: Cover quadratic f-string tokenization regression (GH-156756) * gh-155525: Avoid quadratic f-string tokenization * fixup! gh-155525: Avoid quadratic f-string tokenization --------- (cherry picked from commit c1df684) Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
…57828) * [ 3.15 ] gh-155525: Avoid quadratic f-string tokenization Adapt the original expression-span fix to the release tokenizer and retain interactive buffers while a formatted string is open. Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com> * gh-155525: Cover quadratic f-string tokenization regression (GH-156756) * gh-155525: Avoid quadratic f-string tokenization * fixup! gh-155525: Avoid quadratic f-string tokenization --------- (cherry picked from commit c1df684) Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: gwosti <322952417+gwosti@users.noreply.github.com>
|
Add regression coverage for modules containing many f-string or t-string replacement fields, expression text after tokenizer buffer growth, and
!=followed by a format specifier.The tokenizer refactor in gh-156484 already removed the quadratic copying on main. This PR now retains that implementation and adds the original regression tests. The release-branch backports also need the original expression-span fix, adapted to their older tokenizer layout.
Closes gh-155525.