fix: keep the job worker sidecar out of module upgrades (hide-menus hook, healthcheck, grace period)

[kneckinator]

Companion to OpenSPP/odoo-job-worker#33, which pauses the job worker while modules are installed or upgraded. This PR covers the OpenSPP side of running the worker as a sidecar during deploys.

1. spp_hide_menus_base: re-hide menus only in the process that updated modules

_register_hook called hide_menus() at the end of every registry load. That covered every upgrade path, which is why it lives there. But it also ran in every other process that reloads its registry after another one signals a change: each HTTP worker, each cron worker, and the job worker. All of them made the same writes to the same ir.ui.menu.group_ids and spp.hide.menu rows, racing the upgrading process for them during deploys (lock waits, duplicate-key and serialization errors).

Only a load that installed or updated modules can have reset group_ids, and Odoo lists exactly those modules in registry.updated_modules. The hook now runs hide_menus() only when that list is non-empty. The CLI -u, the upgrade wizard and immediate install all end in such a load, so hiding is still re-applied on every path the original comment names.

• Behaviour change: a plain restart with no module update no longer re-hides menus. Nothing resets group_ids on a plain restart, so there is nothing to re-apply.
• Tests:
  • new: no update → hide_menus() is not called;
  • new: update → it is called once;
  • the existing re-hide-after-reset test now sets updated_modules explicitly. Before, it passed only because the test registry itself had been loaded with -i.
• 27/27 module tests pass. Bumped to 19.0.2.1.1, with a HISTORY entry.

Checked the other _register_hooks. spp_cel_domain, spp_cel_vocabulary and spp_disability_registry are in-memory only, and spp_audit reads rules and patches classes without writing. None need changing.

2. Compose: a healthcheck that works, and a grace period

• docker/docker-compose.production.yml: the queue-worker healthcheck ran pgrep -f 'odoo.addons.job_worker.cli', but the container runs python …/job_worker_runner.py, so the pattern never matched. The runner's degraded signal (a quarantined database, stuck database-error recovery, an upgrade pause that never ends) never reached the orchestrator. It now runs job_worker_healthcheck.py, which checks the runner's heartbeat file.
• Root docker-compose.yml: jobworker had no healthcheck; it now has the same one. The comment claiming databases are rediscovered "every 5 minutes" was also wrong, so it now describes the upgrade pause instead.
• docker/docker-compose.nginx.yml: queue-worker had no healthcheck and no grace period; it now has both, as in the production file.
• All three files: stop_grace_period: 60s. Docker's 10s default SIGKILLed the worker before the runner's 30s wait for running jobs, and every killed job lost a retry attempt when it was reclaimed.

3. docker/README.md

• The stop-worker → one-shot -u → start procedure for upgrades;
• what the job_worker upgrade gate does when the worker is left running;
• what the queue worker's healthcheck checks.

Verified end to end

Root compose, ui profile, with this branch and odoo-job-worker#33 mounted:

1. Installed job_worker,spp_hide_menus_base with a one-shot -i beside the running jobworker.
2. Recreated openspp with ODOO_UPDATE_MODULES=all, leaving jobworker running.

Results:

• The worker log shows the pause and the resume: Job worker paused for database jwe2e: modules-in-transition [auth_passkey_portal:to upgrade, …], then resumed … after 5s; reloading the registry.
• No "Transient module states were reset" in either container.
• No duplicate-key or serialization errors.
• jobworker stayed healthy under the new heartbeat healthcheck.

Deployment notes

• The pause needs job_worker 19.0.1.3.0 (fix(worker): pause the job worker while modules are installed or upgraded odoo-job-worker#33, merged). Images built with the default ODOO_JOB_WORKER_REF=19.0 pick it up. A build pinned to an older commit does not. The hook fix and the compose changes stand on their own.
• Existing deployments may not pick up new addon code. Both production stacks mount the odoo_addons named volume over /mnt/extra-addons, so the addon code there, including the job worker and job_worker_healthcheck.py, may predate the image. docker/README.md now says how to check, and how to re-seed the volume.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.04%. Comparing base (1a3c591) to head (3a60630).
⚠️ Report is 1 commits behind head on 19.0.

Additional details and impacted files

@@            Coverage Diff             @@
##             19.0     #577      +/-   ##
==========================================
+ Coverage   76.91%   77.04%   +0.13%     
==========================================
  Files         704      727      +23     
  Lines       45774    47165    +1391     
==========================================
+ Hits        35205    36338    +1133     
- Misses      10569    10827     +258     

Flag	Coverage Δ
spp_analytics	93.25% <ø> (ø)
spp_api_v2_change_request	73.37% <ø> (ø)
spp_api_v2_cycles	71.03% <ø> (ø)
spp_api_v2_data	77.77% <ø> (ø)
spp_api_v2_entitlements	70.23% <ø> (ø)
spp_api_v2_gis	74.60% <ø> (ø)
spp_api_v2_programs	92.22% <ø> (ø)
spp_approval	50.85% <ø> (ø)
spp_area	80.16% <ø> (?)
spp_area_hdx	81.60% <ø> (?)
spp_base_common	91.07% <ø> (ø)
spp_case_cel	89.50% <ø> (ø)
spp_case_demo	94.82% <ø> (ø)
spp_case_entitlements	100.00% <ø> (ø)
spp_case_programs	100.00% <ø> (ø)
spp_hide_menus_base	95.34% <100.00%> (?)
spp_programs	67.56% <ø> (-0.03%)	⬇️
spp_registry	89.00% <ø> (ø)
spp_security	69.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
spp_hide_menus_base/models/ir_module_module.py	91.17% <100.00%> (ø)

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gonzalesedwin1123]

Thanks. The core fix is correct, and I checked it against the Odoo 19 source:

• registry.updated_modules is a fresh list for every Registry (orm/registry.py). It is only appended for packages with an install or upgrade operation (modules/loading.py, load_module_graph).
• A reload after another process signals (check_signaling() → Registry.new(db)) starts with an empty list, so HTTP, cron and job-worker reloads skip the pass.
• CLI -u/-i, the base.module.upgrade wizard, button_immediate_* and the base.partially_updated_database recovery path all run an update load, so hiding is still re-applied on every path that can reset group_ids. With -u under prefork, the master's preload runs the hook once, before forking.
• Your _register_hook inventory is complete.
• job_worker_healthcheck.py exists in the image at that path, and it exits 1 when there's no heartbeat. The old pgrep pattern could never match job_worker_runner.py.
• The new no-update test would fail against the old code.

Three changes before merge:

1. docker/docker-compose.nginx.yml needs the same fix. Its queue-worker has no healthcheck and no stop_grace_period. Deployments on the Nginx stack keep Docker's 10s SIGKILL, and each killed job loses a retry attempt. Please add the same healthcheck and stop_grace_period: 60s blocks. Its /tmp is a tmpfs, which suits the heartbeat file.

2. The README upgrade snippet expands $DB_NAME in the host shell.

$COMPOSE run --rm odoo odoo -d "$DB_NAME" -u <modules> --stop-after-init --no-http

DB_NAME normally lives in .env. Compose uses that file for interpolation but never exports it to the operator's shell, so this usually becomes -d "". The passthrough command already gets the database from the generated /etc/odoo/odoo.conf (ODOO_RC, db_name = ${DB_NAME}), so dropping -d "$DB_NAME" is enough.

3. Mark the README text that depends on OpenSPP/odoo-job-worker#33. That PR hasn't merged. Until it does, this README describes behaviour job_worker 19.0 doesn't have: the worker "pauses by itself", the JOB_WORKER_UPGRADE_* / JOB_WORKER_UPGRADE_PAUSE_UNHEALTHY_AFTER settings, the one-hour-unhealthy rule, and the "Upgrading modules while the worker runs" doc section. A short "requires job_worker with the upgrade gate (odoo-job-worker#33)" on those paragraphs would do. So would keeping only the stop → -u → start procedure here and adding the pause text once #33 lands.

Non-blocking suggestions:

• The named-volume caveat in the PR description should go in docker/README.md, where operators will see it: odoo_addons over /mnt/extra-addons means existing deployments won't pick up job_worker_healthcheck.py or #33 from a new image.
• The snippet stops only the worker, so odoo keeps serving users and running crons during the one-shot -u. If that's intended, a sentence saying so would help. Otherwise, stop odoo too.
• On stop, the runner joins worker threads one after another, 30s each (_cleanup → _stop_worker_thread), so 60s covers one or two databases. The compose comment could say "per database".
• The first paragraph of the _register_hook comment still says hiding is re-applied "at the end of every registry load (startup, …)", which now contradicts the gate. Also, updated_modules lasts as long as the updating process's registry. A later _setup_models__ in that process, such as adding a custom field, re-runs the hook. That is harmless, but "a load that updated modules" is not quite the exact condition.
• Operator-facing behaviour change: menu drift from anything other than a module update no longer self-heals on restart. A HISTORY line saying -u spp_hide_menus_base restores hidden menus would help.

[kneckinator]

Thanks for checking this against the Odoo source. All eight points are addressed in 501d1aa and 3a60630.

Blocking

1. docker-compose.nginx.yml: queue-worker now has the same heartbeat healthcheck and stop_grace_period: 60s. The healthcheck comment notes the heartbeat file lives in the /tmp tmpfs.
2. $DB_NAME in the host shell: I dropped -d "$DB_NAME". The one-shot takes the database from the generated /etc/odoo/odoo.conf (ODOO_RC, db_name = ${DB_NAME}), and the README now says so.
3. Text that depends on fix(spp_change_request_v2): add no_create/no_open to lookup fields in CR detail views #33: it merged on 2026-09-29 as job_worker 19.0.1.3.0. The pause and health paragraphs now say they need 19.0.1.3.0 or later. They also say that images built with the default ODOO_JOB_WORKER_REF=19.0 include it, while a pinned older commit doesn't, and neither does an odoo_addons volume seeded before it.

Suggestions

• Named-volume caveat: moved into docker/README.md as its own subsection under Updating. It includes a command to check the job_worker version the queue worker actually runs, and how to re-seed the volume.
• Stopping only the worker: that wasn't intended. The snippet now stops queue-worker and odoo, says why (requests and crons hit the same changing schema), and says users see downtime while the upgrade runs.
• "Per database": all three compose comments now say the runner waits up to 30s per database, one after another, and to raise the grace period for workers that serve more than one or two databases.
• The _register_hook comment: rewritten. The pass runs in a registry that installed or updated modules. Because updated_modules lasts for that registry's lifetime, a later _setup_models__ in the same process repeats the pass, and that's harmless because hide_menus() is idempotent. You're right that "a load that updated modules" wasn't the exact condition.
• HISTORY: added a line saying menu drift from anything other than a module update no longer self-heals on restart, and that -u spp_hide_menus_base re-applies hiding.

spp_hide_menus_base: 27/27 tests pass. Pre-commit passes, and semgrep is clean.

[gonzalesedwin1123]

Re-reviewed at 3a60630. All three asks and the five suggestions are addressed, and I re-checked the new text against the image and the compose files:

• docker-compose.nginx.yml: healthcheck and stop_grace_period match the production file; the heartbeat file defaults to /tmp/job_worker_heartbeat (job_worker/cli/heartbeat.py), which that service's tmpfs covers.
• The one-shot without -d is right: the entrypoint treats any argument other than a bare odoo as a passthrough, generates /etc/odoo/odoo.conf from docker/odoo.conf.template first (db_name = ${DB_NAME}), and the image sets ODOO_RC to it. Both production stacks use that template (docker/nginx/odoo.conf.template is an nginx config, not an Odoo one).
• Named-volume caveat, odoo stopped during the one-shot, per-database grace-period comments, the _register_hook comment and the HISTORY recovery line all read correctly.

One precision point, non-blocking, fix now or in a follow-up as you prefer:

"job_worker 19.0.1.3.0 or later" doesn't identify the gate. odoo-job-worker#33 didn't bump the manifest; 19.0.1.3.0 was set by #32 on 2026-08-20 and the CHANGELOG still lists it as unreleased, so any image or odoo_addons volume built from 19.0 between 2026-08-20 and 2026-09-29 reports 19.0.1.3.0 without the gate. That's exactly the "volume seeded before it" case the README's grep '"version"' check is meant to catch, and it can't. Something like

docker compose -f docker/docker-compose.production.yml exec queue-worker \
  test -f /mnt/extra-addons/odoo-job-worker/job_worker/cli/upgrade_gate.py && echo gate present

answers it directly, and the two "19.0.1.3.0 or later" sentences could say "job_worker with the upgrade gate (odoo-job-worker#33, merged 2026-09-29; the manifest version did not change)". Alternatively a bump to 19.0.1.4.0 on odoo-job-worker would make the version check meaningful.

Approving; the merge is Edwin's call. Our #532 rebases onto this once it lands (renumber to 19.0.2.1.2).