Skip to content

Reduce idle FATE worker polling interval - #6557

Merged
DomGarguilo merged 1 commit into
apache:mainfrom
DomGarguilo:fateWorkerBug
Sep 24, 2026
Merged

DomGarguilo merged 1 commit into
apache:mainfrom
DomGarguilo:fateWorkerBug

Conversation

@DomGarguilo

Copy link
Copy Markdown
Member

Related to #5827

On a 5 node test cluster I hit the cost of this issue testing 4.0.0-alpha. The cluster was idle with no fate transactions and the fate workers were still burning 24% of a core on one manager and 72% on another. ~26 workers all waking up 1000 times a second on an empty queue. Measured from thread dumps 60s apart.

#5825 replaced the poll(100, MILLISECONDS) with poll() + Thread.sleep(1) so 100x the wakeups. Sleeping for 50 or 100ms would get most of that back now without waiting for JDK 21. Working on a PR for that now.

Originally posted by @DomGarguilo in #5827

This PR increases the sleep interval to 50ms to vastly decrease this idle usage described above. The new interval is derived from the tryTransfer timeout (100ms) so that it is still guaranteed the worker will poll at least once while the work finder is waiting to hand off.

@DomGarguilo DomGarguilo added this to the 4.0.0 milestone Sep 23, 2026
@DomGarguilo DomGarguilo self-assigned this Sep 23, 2026
@DomGarguilo DomGarguilo added the bug This issue has been verified to be a bug. label Sep 23, 2026
@DomGarguilo
DomGarguilo merged commit 67c5c00 into apache:main Sep 24, 2026
12 of 13 checks passed
@DomGarguilo
DomGarguilo deleted the fateWorkerBug branch September 24, 2026 15:21
@ctubbsii ctubbsii modified the milestones: 4.0.0-alpha-1, 4.0.0 Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug This issue has been verified to be a bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants