Repository navigation
Conversation
…of exiting When login or init failed, the daemon exited and container / service restart policies started it again, logging in to the server on every restart. Frequently failing deployments potentially generated thousands of such logins. The daemon now retries login and init within the same process, reusing the existing Mergin client. Consecutive failures back off exponentially; a successful sync resets the wait to sleep_time. New daemon option 'max_retries' limits consecutive failed retries of the start or of unexpected errors, after which the daemon exits as before and always sends a notification email. Sync errors after a successful start are still retried indefinitely.
This mitigates the very frequent unnecessary logins to server. After login, the auth token is stored in <working_dir>/.mergin_auth.json. Rejected tokens (401) are removed and a new login is done. Stored token is kept even when the working directory is removed by --force-init cleaning.
The server may reject a token before it expires. Failed logins are retried with longer backoff, as rejected credentials can not be fixed by retrying, to avoid locking the account.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Sometimes there are db-syncs logging in very frequently (e.g. every ~2 s). By design, db-sync logs in once per process, so the cause is processes being started over and over: a failing daemon restarted by Docker or another service manager, or
--single-runcalled frequently from cron. Each new process logged in again.Fixes
1. Retry failed start in-process instead of exiting
--force-initcleaning and init failures are retried inside the daemon, with exponential backoff fromsleep_timeup to 10 min. An existing client is reused, so a failed init doesn't log in again.DbSyncError.daemon.max_retriesoption (default 10,0= never exit). The daemon exits after this many consecutive failed retries of the start or of unexpected errors, and always sends a notification email when it gives up. Sync errors after a successful start are still retried indefinitely.2. Reuse auth token across restarts
<working_dir>/.mergin_auth.json.--force-initcleaning.3. Handle tokens rejected by the server