CI recurrence study: Tier 0 fingerprinting, collector, and harness - #10
Closed
chrishonson wants to merge 2 commits into
Closed
chrishonson wants to merge 2 commits into
chrishonson wants to merge 2 commits into
Conversation
Schema, Tier 0 normalizer, and Tier 1 worker prompt for the CI failure recurrence study, plus the collector and the model-free analysis path. Tier 0 is deterministic regex normalization + sha1 fingerprinting: free, reproducible, and hashable, which an LLM label is not. Tiers 1/2 are only ever asked about what Tier 0 leaves as a singleton. Verified with known-answer fixtures (20/20 in test_pipeline.py): the normalizer collapses one bug across differing paths, line numbers, and float values while keeping distinct test files and distinct exit codes apart; the chronological walk reproduces a known rate; both flake shapes are detected; the ingest path handles annotation evidence with a step-name fallback. Two bugs found and fixed by that testing, both of which would have corrupted the headline rate: - a port rule fired on four-digit line numbers ahead of the line/col rule, shattering collapse across every large file - the absolute-path rule mangled repo-relative annotation paths, which are identical on every runner and therefore signal No recurrence rate is computed yet. This session cannot reach the Actions API for either target repo (403 per-repository authorization; the one reachable repo has zero CI failures), so collection stopped rather than building the Tier 1 loop on an unmeasured Tier 0. See findings.md. Stdlib only: no LangChain, DSPy, Celery, or Redis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJvuGv9fvnpNGXmJFp2HzQ
The README asked for a PAT with `public_repo`. That scope grants write to
every public repo the holder can see and buys this study nothing: both
targets are public and all three endpoints used here (actions/runs,
runs/{id}/jobs, check-runs/{id}/annotations) are public-readable without
any scope.
The token exists only to lift the rate limit from 60/hr unauthenticated
to 5,000/hr, which any valid token does regardless of scopes. So: a
classic PAT with zero scopes ticked, or a fine-grained PAT set to
"Public repositories (read-only)" -- noting a fine-grained token cannot
be scoped to a repo you do not own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJvuGv9fvnpNGXmJFp2HzQ
Owner
Author
|
Closing unmerged: the recurrence study was completed outside this repository, so The work is preserved and stays reachable after the branch is deleted. Head commit is Note for anyone deleting branches here: PR #11 (Secret Manager) was stacked on top of these two commits. It has been rebased onto Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Schema, Tier 0 normalizer, and Tier 1 worker prompt for the CI failure
recurrence study, plus the collector and the model-free analysis path.
Tier 0 is deterministic regex normalization + sha1 fingerprinting: free,
reproducible, and hashable, which an LLM label is not. Tiers 1/2 are only
ever asked about what Tier 0 leaves as a singleton.
Verified with known-answer fixtures (20/20 in test_pipeline.py): the
normalizer collapses one bug across differing paths, line numbers, and
float values while keeping distinct test files and distinct exit codes
apart; the chronological walk reproduces a known rate; both flake shapes
are detected; the ingest path handles annotation evidence with a
step-name fallback.
Two bugs found and fixed by that testing, both of which would have
corrupted the headline rate:
rule, shattering collapse across every large file
are identical on every runner and therefore signal
No recurrence rate is computed yet. This session cannot reach the Actions
API for either target repo (403 per-repository authorization; the one
reachable repo has zero CI failures), so collection stopped rather than
building the Tier 1 loop on an unmeasured Tier 0. See findings.md.
Stdlib only: no LangChain, DSPy, Celery, or Redis.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01DJvuGv9fvnpNGXmJFp2HzQ