fix(server): retry provider attachment CAS conflicts

[drew]

Summary

Retry transient sandbox resource-version conflicts during server-owned provider attach and detach. Explicit client resource versions continue to fail on conflict so clients can detect concurrent edits.

Related Issue

No issue required: localized reliability fix for observed provider attachment flakes. The stopped-sandbox runtime identity changes originally in this PR are superseded by #3485 and have been removed.

Changes

• Retry bounded server-owned sandbox CAS mutations for provider attach/detach, preserving the existing client-driven CAS contract.
• Keep deterministic concurrency coverage for server-owned retries and the existing explicit-version tests.
• Update the gateway architecture overview to describe the retry exception.
• Integrate current main without rewriting the PR branch history.

Testing

• mise run pre-commit passes
• mise run test passes
• Focused server-owned CAS concurrency test passes
• OPENSHELL_E2E_DOCKER_TEST=provider_readiness mise run e2e:docker passes, including Docker conformance
• mise run ci passes locally: unrelated Go SDK gateway-list tests find this host's /etc/openshell/gateways/default and fail their empty-directory assertions. The PR's hosted CI will run on the updated head.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[github-actions]

Label test:e2e applied for ea707d2. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

gator-agent

PR Review Status

This localized sandbox reliability fix is project-valid, and the independent full-diff review found no blocking issues. The required E2E workflow has not started with the new label, so pipeline monitoring cannot begin yet.

Action required: A maintainer must open the existing E2E run for this head and choose Re-run all jobs so the test:e2e suite executes with the label set.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Maintainer-authored, localized server reliability fix for observed sandbox lifecycle flakes
• Docs: Not needed because the patch changes internal retry behavior without changing user-facing behavior
• Checks: Branch Checks, Helm Lint, and Trivy Changes are green on the current head
• E2E: test:e2e is applied; the label helper requires Re-run all jobs on the existing run before the suite is dispatched
• Head SHA: ea707d22973f06cb3eec23386ebf3a02836bb16f
• Base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Merge base SHA: 29e89a2f2289ad538c195e136baaaac2a92a3a2e
• Patch ID: 2e63f6a2c343f28901543396718033eb8e6c442c
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[elezar]

PR #3485 supersedes the stopped-sandbox runtime-identity portion of this change and should land first. After #3485 merges, the expected next steps here are:

• Rebase this PR onto main.
• Retain the bounded server-owned CAS retries for provider attach/detach, including update_sandbox_cas and its concurrency coverage.
• Drop the mint_next_runtime_authentication / persist_next_runtime_identity changes and the tests tied to that split pre-lock identity flow.
• If useful, port the unrelated-status-update scenario into a test of the compute-side atomic start transition introduced by fix(server): serialize sandbox restart authentication #3485.

The reason is that this PR fences writers racing from the same persisted identity, but it still leaves identity rotation separate from the lifecycle lock and Starting transition. A later start can rotate the identity again in that window. #3485 closes that window by committing the winning identity and Starting phase atomically, then reusing that identity for idempotent driver retries.

[drew]

gator-agent

Blocker Follow-Up Nudge

This PR is still blocked after more than 48 business hours. @elezar's maintainer direction is now actionable because #3485 has merged; I checked #3501 and it currently conflicts with main while retaining the superseded runtime-identity work.

Next action: @drew, rebase onto main, resolve the conflict, retain the bounded provider attach/detach CAS retries and concurrency coverage, and drop the superseded runtime-identity changes and their tests as @elezar requested. Push the updated head so Gator can review the author-only delta and continue E2E dispatch.

[drew]

Updated this PR against current main and resolved the merge conflict. The stopped-sandbox runtime identity changes and their tests were removed because #3485 supersedes them.

The remaining diff keeps bounded CAS retries for server-owned provider attach/detach mutations, preserves fail-fast behavior for explicit client resource versions, and retains the concurrency test. I also updated the gateway architecture doc to describe this retry exception.

Verification on the updated head: mise run pre-commit, mise run test, the focused CAS concurrency test, and Docker conformance plus provider_readiness E2E all passed. Hosted Branch Checks and E2E are running. Local mise run ci still hits unrelated Go gateway-list tests because this host has /etc/openshell/gateways/default.

[elezar]

Reviewed the CAS retry behavior, explicit-version contract, concurrency coverage, and architecture update. No actionable issues found.