Benchmark prs - #1529
Open
jprendes wants to merge 12 commits into
Open
Benchmark prs#1529jprendes wants to merge 12 commits into
jprendes wants to merge 12 commits into
Conversation
jprendes
force-pushed
the
benchmark-prs
branch
2 times, most recently
from
June 12, 2026 10:43
8ad9393 to
0e181c4
Compare
jprendes
added
the
kind/enhancement
jprendes
force-pushed
the
benchmark-prs
branch
4 times, most recently
from
June 17, 2026 14:48
bc499ea to
a277ac1
Compare
jprendes
force-pushed
the
benchmark-prs
branch
3 times, most recently
from
June 18, 2026 11:22
e660953 to
9ce9ed0
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
jprendes
marked this pull request as ready for review
June 19, 2026 15:54
jprendes
requested review from
andreiltd,
danbugs,
dblnz,
devigned,
jsturtevant,
ludfjig,
simongdavies and
syntactically
as code owners
June 19, 2026 15:54
jprendes
force-pushed
the
benchmark-prs
branch
8 times, most recently
from
September 25, 2026 16:15
9ec8b22 to
d4dbe54
Compare
This comment was marked as outdated.
This comment was marked as outdated.
jprendes
force-pushed
the
benchmark-prs
branch
6 times, most recently
from
September 25, 2026 19:00
67e75dd to
6fa38df
Compare
This comment has been minimized.
This comment has been minimized.
`bench_report.toml` lists regular expressions matched against criterion benchmark ids. A benchmark is selected when it matches `allowlist` and no `denylist` entry, and an empty `allowlist` keeps everything the denylist does not exclude. Omitting both lists disables filtering. `cargo ci bench-report --config-file` reports the selection, and `cargo ci bench --config-file` runs it. CI keeps running every benchmark and filters at report time. Both subcommands apply the selection through `CriterionSwarm::retain`, added in criterion-swarm 0.2.1, so running and reporting cannot drift apart. An allowlist pattern that matches no benchmark fails, so a rename surfaces instead of dropping out of the comment silently. A denylist pattern matching nothing is accepted, because a benchmark may be absent on some platforms. Unknown keys are rejected so a stale key cannot silently disable filtering. The initial lists keep 47 of 116 benchmarks: those whose median drifted by at most 5% across five back-to-back runs on an idle machine, less the snapshot cold start and restore families. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
A vCPU created without an in-kernel LAPIC bumps the kernel's `kvm_has_noapic_vcpu` static key, and teardown drops it again. Each transition through zero rewrites kernel text and IPIs every core. Benchmarks that create and drop sandboxes leave no resident VM between iterations, so they cross that boundary constantly: a full suite run issues 176k broadcast IPIs, against 560 with one sandbox held. `cargo ci bench` now keeps one sandbox alive for the duration of a run, so that cost lands on neither the benchmark that triggers it nor its neighbours. Excluding benchmarks that ran on CPU 0, which has its own much larger effect, median drift for the sandbox group falls from 6.5% to 1.7%. Pass `--no-ballast` to measure the cold path instead. The helper is an example rather than a dependency of this crate, keeping hyperlight-host out of the CI tool's build. It exits on stdin EOF, so it cannot outlive the run even when this process is killed without unwinding. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Criterion stores an id per result directory but nothing about the run as a whole, and results accumulate: a directory keeps benchmarks that no longer exist, indistinguishable from the ones just measured. CI compounds this by unpacking a baseline into the same directory before running, so its uploaded artifact holds 155 result directories for a suite of 87. `cargo ci bench` writes `benchmarks.json` next to the results, listing the benchmarks the run covers along with a timestamp and the host it ran on. The list is the post-filter set, so it describes what was measured rather than what was discovered. `bench-report` takes the ids from there when the results carry one, so a criterion directory from elsewhere, a CI artifact say, reads without a toolchain and without the checkout matching. Naming them by listing the binaries builds them first and describes the current checkout instead, which is the same thing only while reporting a local run of the current tree. Rendering the downloaded results of a full suite takes 11ms rather than a build. Explicit `--binary` or trailing bench args still list the binaries, since both name benchmarks the manifest cannot filter, as do results from before the manifest existed. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Comparing what CI measured meant downloading six artifacts by hand, unpacking each somewhere, and running the report once per configuration. The results are richer than the comment CI posts, holding every benchmark rather than the reported subset and the samples behind each estimate, so reaching for them is worth making cheap. `--candidate` and `--baseline` say where each side comes from: a criterion directory, `run:<ID>`, or `pr:<NUMBER>`, which takes the most recent run that still has its artifacts, since the newest is often a label check or one whose benchmarks have not finished. A whole run renders as one section per hypervisor and cpu vendor, the way the pull request comment reads. Runs land under `target/ci-runs` and are reused, artifacts being immutable. A run is about 400MB unpacked, and the first report of one waits on the download. criterion-markdown 0.2.1 computes changes when it renders, so the report needs both datasets present. A run overwrites the baseline it compares against, so CI keeps the previous results in their own criterion root. Criterion keeps the last run of a directory in `new` and the one before it in `base`, so a directory on its own reports the last run against the previous one, and another directory is compared through its own last run. Results with no baseline to compare against are reported on their own. CI covers every hypervisor and cpu vendor, and comparing results measured on different machines says nothing. Each side is paired with the one that ran on the same kind of machine, read from the artifact name or from the operating system, cpu vendor and hypervisor a run records. What fits nothing, or several, is reported without a comparison. The ids come from the manifest, so they describe what the run measured rather than this checkout, which need not even be the same commit. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
The default branch is benchmarked daily rather than per commit, so the run that measured a given commit rarely exists. `commit:<SHA>` takes the closest run that carries no changes the commit never had, and `base-of:<NUMBER>` the one where a pull request branched. Nothing within a pull request's own results says what they mean, so that is what they are measured against. A cancelled run leaves some configurations unmeasured, so only whole runs serve as a baseline. Sections are named the way the workflow that measured them is, so a report of a run reads like the comment CI posts. Artifacts hold nothing every run is bound to leave behind, so a download that finished says so itself. Reaching for a file the run might not have written re-downloaded results already on disk, onto the ones already there. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Each benchmark job rendered its own section against a baseline it downloaded itself, which only the pull request comment ever read. The daily and release runs rendered sections nothing consumes, and a pull request was measured against the latest release rather than the branch it targets. The job that posts the comment now reports the whole run against where the pull request branched, so benchmarking a configuration is only that. What the results are worth on their own outlives the comparison, so a baseline out of reach costs the changes, not the report. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Workflow artifacts were swept away after five days, so comparing against anything older meant unpacking a release tarball by hand into a directory the report could be pointed at. They now last as long as GitHub keeps them, and `release:<TAG>` reads what a release carries, which outlives artifacts altogether. Searching as far back as results are kept lets a commit that old still be found. Benchmark ids change over time, so a release far enough back has little left to compare against. What matches is reported and the rest stands alone. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
jprendes
force-pushed
the
benchmark-prs
branch
from
September 25, 2026 21:05
6fa38df to
26db124
Compare
This comment has been minimized.
This comment has been minimized.
A comment that only holds numbers leaves the reader to work out what was measured and what it was held against. The report names both commits, and `--reproduce` ends it with the command that asks for it again. What was asked for moves, since the last run of a pull request is whichever ran most recently and where it branched changes when it is rebased, so the command names the runs that answered instead. GitHub answers run listings out of an index that takes a moment to warm, leaving the most recent runs out of the first replies. Taking one at its word picked a baseline months older than the one asked for, silently, because an old run is still an ancestor of the commit. Listings are now asked for until two agree on the newest run. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
The lists were measured on a quiet machine, which is not the machine that reports them, and then from four runs whose spread was taken as the range between the extremes. Whole runs land slow: two of eleven sat 10% to 20% above the rest across the entire suite, and a range reads that as every benchmark being unstable. The middle half of the runs says what a benchmark usually does. Listed here when no configuration spreads more than 10%. That is a fraction of what any one configuration could carry. `Linux_kvm_amd` holds 87 of 95 benchmarks within 5%, `guest_calls` among them at 0.3%, while `Windows_hyperv-ws2025_amd` holds 15 and stays there on uniform hardware. `Linux_mshv3_amd` is as quiet as kvm on one processor and four times worse across two, because its pool answers with both EPYC generations. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
What the report calls a regression is a judgement about the machine it ran on, not about the benchmark. Two runs of one commit raise an alert in every report at the default ratios, and in none of them once a change has to reach 1.5x either way. `improvement`, `strong_improvement` and `regression` set where those lines fall, in the config file beside the benchmarks they apply to, or on the command line for a one-off reading. The renderer rejects ratios that cross or invert. `summary_limit` and `reproduce` settle beside them, being what a repository decides once rather than per report. `repo` reads `remote:<name>` as whichever repository a git remote points at, so a fork reads its own runs instead of the one written into the tool. The config file is read once for a report rather than once for each configuration it covers. Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
jprendes
force-pushed
the
benchmark-prs
branch
from
September 25, 2026 21:52
26db124 to
26fb31a
Compare
Benchmark ResultsMeasured commit: kvm / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (❌ 1.51x)Top regressions
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.