drivers: do not exit the ttl fiber on an error - #266
Open
maksimuimin wants to merge 2 commits into
Open
maksimuimin wants to merge 2 commits into
maksimuimin wants to merge 2 commits into
Conversation
fifottl and limfifottl used an unbuffered channel to stop the ttl fiber, but the fiber read it only in ro mode. So stop() blocked forever in rw mode and drop() did not stop the fiber at all, leaking one fiber per dropped tube. utubettl used a buffered channel: stop() returned, but the fiber kept running and crashed on the dropped space. Now the fiber works while it is the fiber registered in the tube: stop() deregisters it, wakes it up and joins it, and fifottl got a drop() which stops the fiber before dropping the space. limfifottl forwards start()/stop()/drop() to the parent driver so that the registration lives in a single object. Closes tarantool#262 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Any error inside a ttl iteration (a user on_task_change callback raising, a transaction conflict, a failed WAL write) terminated the ttl fiber for good: start() saw the dead fiber in self.fiber and did nothing, so ttl/ttr/delay processing stopped until the next ro -> rw switch, and with the unbuffered channel of fifottl stop() blocked the queue state machine in ENDING forever. Now the fiber logs the error, backs off for a second and retries the iteration. It still exits when it is cancelled or stopped. Closes tarantool#263 Part of tarantool#238 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #263, related to #238
Based on #265 (the first commit is that PR); only the second commit is new here.
Problem
Any error inside a ttl iteration made the fiber
return 1, leaving the dead fiber inself.fiber. Sostart()did nothing, ttl/ttr/delay processing stopped until the next ro -> rw switch, and forfifottlthe nextstop()blocked the queue state machine inENDINGforever. A useron_task_changecallback raising once was enough to trigger it, see the repro in #263.Fix
The fiber no longer exits on an unexpected error: it logs the error, backs off for one second and retries the iteration. Transient causes (a failed WAL write, a transaction conflict, a user callback) recover on their own; a persistent one is logged once per second instead of silently disabling ttl for the tube. The fiber still exits when it is cancelled or stopped (#265).
This also gives #238 a working answer: after
stop()the fiber can always be brought back withstart().New test
t/270-ttl-fiber-errors.t: a callback raising on ttl, the fiber survives, the next expired task is deleted, and an ro -> rw cycle brings the queue back toRUNNING. Full suite passes locally (Tarantool 3.8, memtx and vinyl).