gh-153319: Read turtledemo scripts as UTF-8 - #153321
Conversation
…ncoding The demo viewer read each script with open() and no encoding argument, i.e. the locale default encoding. That emits an EncodingWarning under -X warn_default_encoding and can mis-decode a script on a non-UTF-8 locale. Read the source with tokenize.open() instead, which decodes using the encoding detected from the file (its coding cookie, else UTF-8), matching how Python read the module when it was imported just above.
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
LGTM. But did you consider using inspect.getsource()?
serhiy-storchaka
added
needs backport to 3.14
|
AFAIK, turtledemo examples that are not all ascii (are their any?) are utf-8 encoded. Why not just add this to the open call? |
Per review, open the demo scripts with encoding='utf-8' instead of tokenize.open(); the scripts are UTF-8.
|
Good point, done. The demo scripts are UTF-8, so I switched to |
terryjreedy
left a comment
terryjreedy
left a comment
There was a problem hiding this comment.
It would be a bug if anyone merged a not-all-ascii script coded as anything other than utf8.
|
Re-running un-related failure of Docs check. |
|
Repeat failure: File "/home/runner/work/cpython/cpython/Doc/venv/lib/python3.14/site-packages/sphinx/builders/changes.py", line 61, in write_documents
ttext = self.typemap[changeset.type]
~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'soft-deprecated'
The full traceback has been saved in:
/tmp/sphinx-err-mf7e_eg3.log
To report this error to the developers, please open an issue at <https://github.com/sphinx-doc/sphinx/issues/>. Thanks!
Please also report this if it was a user error, so that a better error message can be provided next time.
make: *** [Makefile:56: build] Error 2
make: Leaving directory '/home/runner/work/cpython/cpython/Doc'
Error: Process completed with exit code 2.Updating branch since something in doc processing may have been updated since July. |
terryjreedy
changed the title
|
Thanks @tonghuaroot for the PR, and @terryjreedy for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15. |
|
GH-157872 is a backport of this pull request to the 3.15 branch. |
bedevere-app
Bot
removed
the
needs backport to 3.15
|
GH-157873 is a backport of this pull request to the 3.14 branch. |
|
GH-157874 is a backport of this pull request to the 3.13 branch. |
|
GH-157873 is a backport of this pull request to the 3.14 branch. |
|
GH-157874 is a backport of this pull request to the 3.13 branch. |
Stop using locale default encoding. All stdlib .py files (except for testing) are utf-8 encoded. --------- (cherry picked from commit de0d976) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
gh-153319: Read turtledemo scripts with the source file's own encoding (GH-153321) Stop using locale default encoding. All stdlib .py files (except for testing) are utf-8 encoded. --------- (cherry picked from commit de0d976) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
|
This is a small change, but is this actually needed for 3.15+? All the 19 demos are pure ASCII, and from 3.15 UTF-8 mode is the default (PEP 686). This would only be a problem if:
(And for 3.14 and earlier, would be a problem without step 3. Which I think is already quite rare.) The last demo was added in 2015, and before that, 2009. This PR feels both unnecessary, and the sort of churn we try to avoid. |
4 participants
Stop using locale default encoding. All stdlib .py files (except for testing) are utf-8 encoded.