Derive the parallel worker count from the file count - #6467
SanderMuller wants to merge 2 commits into
Conversation
9cfa18c to
5d93cef
Compare
A warm run of a few dozen changed files uses one worker today, on any machine.
jobSize 20 and minimumNumberOfJobsPerProcess 2 together ask for 40 files before
a second worker is allowed, so an edit-and-rerun cycle is single threaded while
the other cores idle.
Behind featureToggles.adaptiveParallelWorkerCount, off by default and on in
bleedingEdge, the worker count comes from the file count instead:
workers = clamp(round(0.5 * sqrt(files)), files >= 9 ? 2 : 1, cores)
A worker's startup is a fixed cost, so the number of files it needs to earn its
keep grows with the size of the run. sqrt(files) encodes that: about 12 files
per worker at 25 files, about 40 at 400, saturating at the usable cores from
roughly 800 files upward. Full runs therefore keep the schedule they have today,
jobs and workers both, which the tests assert at 800 and 4524 files.
The job count follows the worker count, because the spawn loop stops when the
queue runs dry and a worker without a job of its own never starts.
Warm runs of phpstan-src, one cache snapshot per arm, 3 rounds, medians, with a
duplicate control arm (0.5 to 3.7 pct):
files default adaptive wall CPU
9 1w 4.32s 2w 4.19s -3.0% +25%
25 1w 4.82s 3w 4.41s -8.5% +44%
50 1w 6.93s 4w 5.11s -26.3% +50%
100 2w 7.56s 5w 5.96s -21.2% +35%
200 5w 8.21s 7w 7.50s -8.6% +14%
The trade is CPU for latency, so it is opt-in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5d93cef to
abc3ff1
Compare
|
in the 2 tables of the PR description, it would be interessing how many processes were used in parallel (before the PR; and after the PR) edit: I just realized what I am asking for is this "5w" in the description? |
|
Yes, that is what the The counts are observed rather than derived. Each arm ran once with the live worker processes counted while it ran, before the timed rounds. Deriving them from the formula is how I got an earlier run wrong. |
staabm
left a comment
staabm
left a comment
There was a problem hiding this comment.
I think it makes sense.
other recent changes to the scheduler did not use a feature flag, thats why I think we don't need one here either.
my computers all have 14 cores, so I cannot provide new numbers ;-).
ondrej has a different spec and can test on different hardware.
maybe @VincentLanglet does also have different hardware available to test it
|
Two things that come to my mind before I think about this in depth:
When forking workers there is essentially no cost. Run with |
sqrt(files) dips below the existing job-count formula between roughly 400 and 800 files, so the rule quietly took workers away from large runs. On 600 warm files it scheduled 12 where the default schedules 14, and measured 13.2% slower against a 2.2% control. The rule exists to stop small runs being starved, never to reduce a large one. Take the maximum of the two, so the adaptive count can only ever raise parallelism. Small runs are unchanged: 9, 25, 50, 100 and 200 files still schedule 2, 3, 4, 5 and 7 workers. The test asserts the invariant across 13 file counts rather than trusting a table, because the dip only shows up in a band neither end of the range covers. Reported by @staabm on the pull request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both points land, and the second one corrects a claim I made. Forking. You are right. Decreasing
Your shape and mine are 1.6% apart against a 4.7% control, so they are the same thing. The
I think your framing is the better one, for two reasons beyond it being a smaller change. It cannot regress a large run. A smaller chunk only ever increases the job count, so It also keeps the tuning in one place. Two things it does not cover. At 9 and 25 files the sqrt chunk lands one worker lower than mine. Nine files measured -3.0% against a 0.5% control, so there is something there worth keeping. And at a fixed worker count I could not find a consistent Happy to rebuild it as a |
3 participants
A warm run of a few dozen changed files uses one worker, on any machine.
jobSize: 20andminimumNumberOfJobsPerProcess: 2together ask for 40 files before a second worker is allowed. The edit-and-rerun cycle is therefore single threaded while the rest of the cores idle.This PR is a proposal, not a finished answer. It is off by default. What I am asking for is measurements on other machines and other corpora, because my numbers come from one 14-core laptop.
What it does
Behind
featureToggles.adaptiveParallelWorkerCount, off by default and on inbleedingEdge, the worker count comes from the file count:and never below what the job count already justifies, so it can only ever raise parallelism.
The shape is fitted to the measurements below, not derived from a model. What the data shows is that the number of files a worker needs before it earns its place grows with the size of the run.
From 381 files upward the schedule is identical, jobs and processes both. I walked every file count from 1 to 900 to find that boundary rather than reading it off the table.
Measurements
Warm runs of this repository, one result-cache snapshot per arm, 3 rounds, medians. A duplicate control arm sizes the noise on every row.
@staabm asked for the range above 200, up to where the process count reaches the machine's 14. Those rows are weaker and I am reporting them as measured:
The process counts are observed, not derived: each arm was run once with the live worker processes counted while it ran, before the timed rounds.
A regression that range found
The first version of this PR used
sqrt()alone. It dips below the existing job-count formula between roughly 400 and 800 files, so it quietly took workers away from large runs. At 600 warm files it scheduled 12 processes where the default schedules 14, and measured 13.2% slower against a 2.2% control.Fixed by taking the maximum of the two formulas, so the adaptive count can only ever raise parallelism. A test asserts that invariant across 13 file counts, because the dip appears in a band that neither end of the range covers.
Thanks to @staabm for asking for exactly the numbers that exposed it.
The cost
It trades CPU for latency, 14 to 50 pct more CPU, which is why it is opt-in. On a laptop that is fans and battery on every run.
The shape was fitted on 18 points across five codebases. Those are this repository plus four real applications of 1957, 4339, 6395 and 11897 files, at levels 5 through max. Against measured worker counts the worst case was +0.0 pct and the mean -9.3 pct. The coefficient 0.5 was fitted on the same points it was scored against, so treat it as a starting value rather than a settled one. Its neighbours 0.4 and 0.6 score about the same, so it is not knife edge.
Two traps if you benchmark this
Both cost me an invalid run.
composer dump-autoloadafter switching to this branch. The autowired parameter is read from the generatedvendor/attributes.php, so without it the toggle silently stays off and both arms measure the same thing.What I have not tested
One machine, 14 cores, where
diagnosereportsMechanism: fork (pcntl_fork). Whether the same shape suits a spawning setup is open, and it is the reason I am asking for measurements elsewhere. The coefficient was never tested against a different core count. The small-file evidence is warm runs only.maximumNumberOfProcessesstill applies on top, so an explicit cap keeps working.