Skip to content

docs: clarify Kubernetes OOM and scheduling failures - #771

Draft
warp-agent-staging[bot] wants to merge 9 commits into
mainfrom
factory/k8s-resource-troubleshooting
Draft

warp-agent-staging[bot] wants to merge 9 commits into
mainfrom
factory/k8s-resource-troubleshooting

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Clarifies how operators diagnose and remediate self-hosted Kubernetes task failures without confusing worker-daemon resources, task-container OOMs, and scheduler capacity failures. Closes #769.

Dependency

This PR is stacked on #770 (docs/self-hosted-kubernetes-troubleshooting at e568ac4251d1411d25271edf0724fb4701a03d96, based on main). Its GitHub base is that branch, and the #770 head is an ancestor of this branch. Merge #770 before this PR; restack if #770's head changes.

Changes

platform/self-hosting/troubleshooting.mdx

  • Replaces a dense Kubernetes failure list with short symptom, verify, and fix paths for FailedScheduling, OOMKilled, eviction, and exit code 143.
  • Preserves the distinction between SIGTERM and an OOM report, and between failures before scheduling and failures in a running container.
  • Reduces metrics troubleshooting from seven steps to five and removes detail that operators do not need to choose the next action.

platform/self-hosting/managed-kubernetes.mdx

  • Distinguishes the long-running worker Deployment's resources from task Job resources.
  • Clarifies the chart's daemon resource defaults and that it sets no daemon limits.
  • Documents task sizing with pod_template and workload-specific runner instance shapes, including task-only shape precedence and the generated-init-container limitation.

Publishable sizing guidance

  • The chart's 100m CPU and 128Mi memory defaults are requests for the worker daemon Deployment; the chart sets no daemon limits by default.
  • Task containers have no worker-defined CPU or memory defaults without pod_template resources or an explicit runner instance shape. Cluster policies can still inject defaults.
  • An explicit runner instance shape sets the task container's CPU and memory requests equal to its limits for that run and overrides matching pod_template values. A pod_template can size operator-defined containers but does not alter worker-generated setup or materialization init containers.
  • Operators should size by workload, cap concurrency when aggregate requests exceed capacity, preserve node headroom, and use compatible autoscaling rather than overprovisioning every worker.

Intentional omissions

  • No universal task size or blanket overprovisioning recommendation was added because the product defines no workload-independent baseline.
  • Internal retry counts and policies are omitted. The docs describe only the observable Job and Pod state operators can act on.

Content design plan

  • Reader and job: A Kubernetes operator diagnosing why a self-hosted task stopped or never started, then sizing the workload without changing unrelated workers.
  • Gap today: Existing troubleshooting combines resource failures, does not cover OOMKilled or exit 143, and does not distinguish daemon resources from task resources.
  • Change: Add symptom-first OOM, eviction, SIGTERM, and scheduling paths plus concise sizing guidance. Exclude universal sizing numbers, internal retry behavior, and unverified future behavior.

Source evidence

  • warpdotdev/oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef: internal/worker/kubernetes.go, internal/types/messages.go, internal/metrics/metrics.go, Helm values.yaml, and Kubernetes tests for resource merging, termination classification, scheduling, and cleanup.
  • warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74: runner instance-shape resolution and self-hosted assignment in runner_instance_shape.go, workers/common/task_utils.go, and workers/selfhosted/websocket.go.
  • PR docs: Improve self-hosted Kubernetes troubleshooting #770 at e568ac4251d1411d25271edf0724fb4701a03d96 for current public terminology, configuration names, and page ownership.

Verification

  • npm run build - passed.
  • python3 .agents/skills/check_for_broken_links/check_links.py --internal-only - passed; 4,244 internal links checked, 0 broken.
  • python3 .agents/skills/style_lint/style_lint.py --changed - passed; 5 branch-changed files scanned, 0 issues.
  • python3 .agents/skills/doc_quality_policy/check_compression_contract.py - passed for troubleshooting.mdx and the three parent-PR pages. managed-kubernetes.mdx is 2,010 words, above the 1,500-word feature-doc budget. The 1,941-word baseline is inherited from docs: Improve self-hosted Kubernetes troubleshooting #770; this PR keeps its focused resource guidance next to the canonical pod_template and instance-shape behavior.
  • git diff --check - passed.
  • Trunk CLI - unavailable in the runner, so trunk check was not run.

Unverified claims

None — all new behavior, configuration, default, and metric claims were verified against current source.

Documentation risk

Risk: engineering-review-required
Rationale: Adds technical claims about self-hosted Kubernetes task resources, runner shapes, and failure classification.
Source files consulted: oz-agent-worker@530bbe934a632910dfe8818ab7b4c6027a3473ef: internal/worker/kubernetes.go and charts/oz-agent-worker/values.yaml; warp-server@2916aa67b511e8ef1f556766aa3d2dfe42a42f74: logic/ai/ambient_agents/workers/common/task_utils.go and logic/ai/ambient_agents/workers/selfhosted/websocket.go
Docs override: none

Co-Authored-By: Oz oz-agent@warp.dev

warp-agent-staging Bot and others added 2 commits September 19, 2026 18:53
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Sep 19, 2026
@vercel

vercel Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 20, 2026 5:38pm UTC

Request Review

@warp-agent-staging warp-agent-staging Bot added factory:docs-factory Label associated to the "docs-factory" factory warpy-factory Opened by the Warp factory agents labels Sep 19, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

View run View conversation View origin

Co-Authored-By: Oz <oz-agent@warp.dev>
warp-agent-staging Bot and others added 3 commits September 20, 2026 17:19
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
@warp-agent-staging
warp-agent-staging Bot changed the base branch from main to docs/self-hosted-kubernetes-troubleshooting September 20, 2026 17:28
Co-Authored-By: Oz <oz-agent@warp.dev>
warp-agent-staging Bot and others added 2 commits September 20, 2026 17:32
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
Base automatically changed from docs/self-hosted-kubernetes-troubleshooting to main September 25, 2026 02:53

This branch was successfully deployed

1 active deployment
Preview — 973e7ad1 Deployed Sep 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed factory:docs-factory Label associated to the "docs-factory" factory warpy-factory Opened by the Warp factory agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document self-hosted Kubernetes OOM vs scheduling failures

0 participants