Skip to content

fix: make the schema codegen host-independent - #151

Open
PerryLink wants to merge 1 commit into
agentclientprotocol:mainfrom
PerryLink:fix/host-independent-codegen
Open

PerryLink wants to merge 1 commit into
agentclientprotocol:mainfrom
PerryLink:fix/host-independent-codegen

Conversation

@PerryLink

Copy link
Copy Markdown

The defect

The schema code generator's output depends on the host OS, in two places. The generated modules are checked in and make gen-check (also exercised by tests/test_gen_all.py::test_codegen_check_is_clean_and_read_only) asserts that regenerating them reproduces the committed artifact — so a generator that is not a pure function of its inputs defeats the check it is there to satisfy.

1. The header embeds os.sep. scripts/gen_schema.py:179 and scripts/gen_meta.py:59 interpolate a Path into an f-string, which stringifies with the platform separator. The committed header says schema/schema.json; regenerating on a non-POSIX host produces:

-# Generated from schema/schema.json. Do not edit by hand.
+# Generated from schema\\schema.json. Do not edit by hand.

2. The ruff pipe is decoded with the ambient locale. Both generators run ruff through subprocess.run(..., text=True) without encoding. text=True uses locale.getpreferredencoding(False), so the child's UTF-8 output is decoded as whatever the host locale says — on this machine cp936:

locale.getpreferredencoding(False) == 'cp936'

When ruff exits non-zero, the decode fails before the message is read, and the diagnostic is replaced:

RuntimeError: ruff check --fix failed:
ruff failed
  Cause: stream did not contain valid UTF-8

The real reason ruff failed is discarded. That is the error path of a build script, so the loss is the whole point of the message. This also hides defect 1: the decode error is raised before the artifact comparison, so the header mismatch is never reached.

The change

Four lines, no behaviour change on a host whose locale is already UTF-8:

-    lines = [f"# Generated from {schema_json.relative_to(ROOT)}. Do not edit by hand."]
+    lines = [f"# Generated from {schema_json.relative_to(ROOT).as_posix()}. Do not edit by hand."]
             input=source,
             text=True,
+            encoding="utf-8",
             capture_output=True,

encoding="utf-8" is what the rest of both files already does for every file they read or write (gen_schema.py:64,81,91,180, gen_meta.py:26,43), so this makes the subprocess consistent with its own module rather than introducing a convention. as_posix() emits / on every host, which is what the committed artifact already contains.

Verification

tests/test_gen_all.py is the repository's own check for exactly this, and it needs no new test:

                        before            after
tests/test_gen_all.py   1 failed, 20 passed    21 passed

Reverting only the two scripts and keeping everything else restores 1 failed, 20 passed, so the change is what the test detects rather than an incidental pass.

The rest of the suite is unchanged by this: --doctest-modules goes from 11 failed / 346 passed to 10 failed / 347 passed, the one difference being this test. ruff check and ruff format --check are clean on both files.

Scope

I did not touch the remaining failures in this checkout. They are in tests/http/* and tests/real_user/test_stdio_limits.py and I have not established whether they are host-specific or real, so I am not claiming anything about them. CI runs on ubuntu-latest only, where the header separator is already / and the locale is already UTF-8 — which is why neither defect is visible there, and why this is a reproducibility fix rather than a platform feature.

The generated modules are checked in, and `make gen-check` asserts that regenerating
them reproduces the committed artifact. Two host dependencies broke that:

- `gen_schema.py:179` and `gen_meta.py:59` interpolate a `Path` into the generated
  header, so the path uses `os.sep` and a non-POSIX host emits `schema\schema.json`
  where the committed artifact says `schema/schema.json`.
- both generators run ruff via `subprocess.run(..., text=True)` without an encoding,
  so the child's UTF-8 output is decoded with `locale.getpreferredencoding(False)`.
  When ruff fails, the decode fails first and the message becomes "stream did not
  contain valid UTF-8", discarding the actual reason -- and hiding the header
  mismatch, since that error is raised before the artifact comparison.

`as_posix()` emits `/` on every host, and `encoding="utf-8"` is what both files
already pass to every read_text/write_text they perform.

tests/test_gen_all.py goes from 1 failed / 20 passed to 21 passed; reverting only
these two scripts restores the failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant