Conversation
The generated modules are checked in, and `make gen-check` asserts that regenerating them reproduces the committed artifact. Two host dependencies broke that: - `gen_schema.py:179` and `gen_meta.py:59` interpolate a `Path` into the generated header, so the path uses `os.sep` and a non-POSIX host emits `schema\schema.json` where the committed artifact says `schema/schema.json`. - both generators run ruff via `subprocess.run(..., text=True)` without an encoding, so the child's UTF-8 output is decoded with `locale.getpreferredencoding(False)`. When ruff fails, the decode fails first and the message becomes "stream did not contain valid UTF-8", discarding the actual reason -- and hiding the header mismatch, since that error is raised before the artifact comparison. `as_posix()` emits `/` on every host, and `encoding="utf-8"` is what both files already pass to every read_text/write_text they perform. tests/test_gen_all.py goes from 1 failed / 20 passed to 21 passed; reverting only these two scripts restores the failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
The schema code generator's output depends on the host OS, in two places. The generated modules are checked in and
make gen-check(also exercised bytests/test_gen_all.py::test_codegen_check_is_clean_and_read_only) asserts that regenerating them reproduces the committed artifact — so a generator that is not a pure function of its inputs defeats the check it is there to satisfy.1. The header embeds
os.sep.scripts/gen_schema.py:179andscripts/gen_meta.py:59interpolate aPathinto an f-string, which stringifies with the platform separator. The committed header saysschema/schema.json; regenerating on a non-POSIX host produces:2. The ruff pipe is decoded with the ambient locale. Both generators run ruff through
subprocess.run(..., text=True)withoutencoding.text=Trueuseslocale.getpreferredencoding(False), so the child's UTF-8 output is decoded as whatever the host locale says — on this machinecp936:When ruff exits non-zero, the decode fails before the message is read, and the diagnostic is replaced:
The real reason ruff failed is discarded. That is the error path of a build script, so the loss is the whole point of the message. This also hides defect 1: the decode error is raised before the artifact comparison, so the header mismatch is never reached.
The change
Four lines, no behaviour change on a host whose locale is already UTF-8:
input=source, text=True, + encoding="utf-8", capture_output=True,encoding="utf-8"is what the rest of both files already does for every file they read or write (gen_schema.py:64,81,91,180,gen_meta.py:26,43), so this makes the subprocess consistent with its own module rather than introducing a convention.as_posix()emits/on every host, which is what the committed artifact already contains.Verification
tests/test_gen_all.pyis the repository's own check for exactly this, and it needs no new test:Reverting only the two scripts and keeping everything else restores
1 failed, 20 passed, so the change is what the test detects rather than an incidental pass.The rest of the suite is unchanged by this:
--doctest-modulesgoes from 11 failed / 346 passed to 10 failed / 347 passed, the one difference being this test.ruff checkandruff format --checkare clean on both files.Scope
I did not touch the remaining failures in this checkout. They are in
tests/http/*andtests/real_user/test_stdio_limits.pyand I have not established whether they are host-specific or real, so I am not claiming anything about them. CI runs onubuntu-latestonly, where the header separator is already/and the locale is already UTF-8 — which is why neither defect is visible there, and why this is a reproducibility fix rather than a platform feature.