M3.1 — Build the public leaderboard: dual metrics + certificate drilldown + trajectory

[GiggleLiu]

Background

New here? Read #9 first — it explains the project and defines every term below.

Issue #3 made a minimal page that shows real data. This issue grows it into the public product. Two ranking metrics matter: bugs per 1000 tokens (fair across models regardless of their pricing) and bugs per dollar (practical cost-efficiency). Visitors should be able to click a model to see its bugs, click a bug to see the full counterexample (source problem → target problem → solutions → verdict), and view the trajectory — the exact sequence of pred commands the AI ran to find that bug, for transparency. It's a static site (just files, no server).

Objective

Grow the minimal page into the public product: a dual-metric ranked table, per-model breakdown, per-bug detail pages, and a link to each bug's discovery trajectory.

Interface (Input → Output)

• Input: the results files + index from M1.3 — Emit schema-valid results and render certificates on the live leaderboard #3/M2.1 — Build fair multi-model orchestration (scheduler + pluggable runner + budget) #4 + each session's saved trajectory (the log of commands the AI ran).
• Output: static pages — the leaderboard, a per-model page, and a per-bug detail page.

Technical recommendations (suggestions)

• Reuse the existing chart; add a toggle between the two metrics.
• The per-bug page shows the certificate (source → target → solutions → verdict) and links to its trajectory.
• The trajectory view reads the agent's saved log and shows the pred commands in order.

Verification (how a reviewer confirms this is done)

Use a small fixture with 2 models chosen so the two metrics rank them in opposite orders:

1. Toggle the metric between "bugs / 1000 tokens" and "bugs / $" → the table reorders. (Proves both metrics are really computed, not just labels.)
2. Click a bug → the source/target/solutions/verdict shown match the JSON file, and the trajectory link shows the pred commands ending in exactly that bug.
3. A model that found 0 bugs still appears, listed with 0.
4. A bug whose trajectory file is missing still renders ("trajectory unavailable") instead of breaking the page.

Dependencies

Depends on #3 (results format + rendering) and #4 (multiple models' results).

Out of scope

A second agent track (opencode); the full offline-reproducibility archive (#8).

[Ferrari-72]

Completed — commit 388a411

What was done

build_index.py

• Each bug_certificate entry now includes trajectory_file: relative path trajectories/{safe_model}_{rule}.jsonl when that file exists alongside the results, null otherwise.

leaderboard/index.html

• Dual-metric toggle: two buttons "Bugs / Ktok" (default, active) and "Bugs / $" — clicking re-sorts the table in place and updates rank numbers.
• Certificate drilldown: each bug row expands to show source type → target type → violation + note, plus either a "View trajectory" link (when trajectory_file is set) or "trajectory unavailable".
• Zero-bug models are never filtered — renderTable renders all entries including those with bugs_found=0.

test_build_index.py — 4 new tests (TestTrajectoryFile)

Test	Verifies
test_trajectory_file_set_when_present	trajectory_file equals relative path when file exists
test_trajectory_file_null_when_missing	trajectory_file is None when file absent
test_zero_bug_model_in_index	model with bugs_found=0 appears in entries
test_dual_metric_opposite_order	both metrics stored correctly; sorting by each gives opposite order

test_leaderboard_html.py — 10 new tests (judgment mark)

Class	Tests	What is verified
TestMetricToggle	3	both metric keys in JS, renderTable present, toggle buttons present
TestTableColumns	2	both metric labels in table header
TestTrajectoryHandling	2	trajectory_file in JS, "trajectory unavailable" string present
TestDataSource	2	fetches ../results/index.json, no hardcoded mock data
TestZeroBugModel	1	no bugs_found > 0 filter in JS

Test results

125 passed, 2 skipped   (make test-unit)

All pre-existing tests pass. No test modified to accommodate the implementation.

Verification checklist (from issue)

• ✅ Toggle between bugs/Ktok and bugs/$ → table reorders
• ✅ Click a bug → source/target/solutions/verdict shown, trajectory link present
• ✅ Model with 0 bugs still appears
• ✅ Missing trajectory file → "trajectory unavailable" instead of breaking