Summarize visual regression failures on pull requests - #1188
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Collaborator
Author
|
/update-snapshots |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Artifact validation occurs after extraction, and metadata is read from the base revision instead of the tested PR head.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Adds sticky PR comments summarizing failed Playwright visual regression tests.
Changes:
- Resolves failed visual jobs and downloads their artifacts.
- Parses, sanitizes, groups, and publishes failure details.
- Adds unit tests and contributor documentation.
| File | Description |
|---|---|
.github/workflows/pr-screenshot-summary.yaml |
Integrates failure-comment generation. |
.github/scripts/update_visual_failure_comment.py |
Parses artifacts and manages comments. |
.github/scripts/test_update_visual_failure_comment.py |
Tests parsing, rendering, and comment updates. |
tests/visual/README.md |
Documents failure comments. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
|
📸 Regenerated visual baselines per your |
JAORMX
approved these changes
Oct 1, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Post a compact, sticky pull request comment when Playwright visual regression tests fail. The comment groups light and dark failures by screen, reports the route, viewport, changed-pixel count, and ratio, and links directly to the failed job and complete Playwright artifact. A later run updates the same comment, and a passing visual run removes it.
The read-only pull request job remains unchanged. The trusted
workflow_runfollow-up downloads and validates the artifact, checks that the run still matches the current pull request head, sanitizes artifact-derived text, and executes only the helper script from the trusted base revision.The parser was replayed against the artifact from run 36173514905 and produced this summary row:
/ai-gatewayInline before/after images remain deferred until generated images have a suitable hosting location.
Type of change
Related issues/PRs
Screenshots
Not applicable.
Submitter checklist
Content and formatting
Reviewer checklist
Content