Skip to content

(WIP)CKS: reconcile node capacity after live resize - #14223

Draft
andrijapanicsb wants to merge 6 commits into
mainfrom
cks-live-resize-capacity-reconciliation
Draft

andrijapanicsb wants to merge 6 commits into
mainfrom
cks-live-resize-capacity-reconciliation

Conversation

@andrijapanicsb

@andrijapanicsb andrijapanicsb commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Why this follow-up is needed

This draft follows #13226, which removed the CKS restriction that prevented changing node compute offerings while a KVM cluster was running. That work builds on #12975, the KVM live-resize support for fixed service offerings.

The VM resize itself can complete successfully: CloudStack records the new offering and libvirt exposes the additional CPU or RAM. Kubernetes may still be advertising the old values, though. In practice, the missing operator step was to restart kubelet on the resized node and check that the Node object's capacity and allocatable had caught up.

#14213 tracks that gap. This PR moves that reconciliation into the CKS offering-change workflow, so a successful scale operation means CloudStack, the guest, and Kubernetes agree on the node resources.

What this PR does

For a CPU-count or RAM increase on a running, Cloud-managed KVM worker or control-plane node, CKS now handles one node at a time:

  1. Record the node's current schedulability and Kubernetes resource values.
  2. Cordon the node when it was previously schedulable. Existing Pods are left alone.
  3. Resize the VM using the existing CloudStack workflow.
  4. Check the guest's online CPU count and MemTotal.
  5. Restart kubelet, wait for it to become active, then wait for the Kubernetes Node to be Ready again.
  6. Read the Node as JSON and verify its capacity and allocatable reflect the resized guest.
  7. Uncordon only a node that CKS cordoned itself, then move on to the next node.

The target-node SSH connection is resolved from its actual port-forwarding rule for isolated/VPC networks, rather than assuming that every node is reachable on port 2222. Direct-access networks use the VM's private address on port 22.

Deliberately not done here

  • This is a cordon-only workflow. A kubelet restart does not require an unconditional drain, so resident workloads are not evicted.
  • Created, Stopped, cap-only, external-node, and non-KVM paths keep their existing behaviour.
  • Dedicated etcd live resize is rejected for now. It needs explicit etcd health/quorum handling rather than Kubernetes-node logic.

Draft / QA notes

The focused reconciler and scale-worker tests pass locally. During the first real QA resize, the new reconciler was present in the management JAR but was not visible to the scale worker: the worker gets dependencies through ComponentContext, while the reconciler had only been declared in the plugin Spring context. This draft now also registers it as a component, matching the existing CKS helper pattern.

The dedicated KVM environment has been prepared with a current public CKS ISO, dynamically scalable SystemVM template, and a running two-vCPU/two-GiB worker. The same resize will be repeated against this commit once management-host access is restored, with before/after Node JSON captured for CPU-only, RAM-only, combined resize, pre-cordoned nodes, and forced kubelet/SSH failures.

@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 34.28571% with 138 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.92%. Comparing base (602d9ec) to head (40b50b0).

Files with missing lines Patch % Lines
...er/actionworkers/KubernetesClusterScaleWorker.java 6.57% 70 Missing and 1 partial ⚠️
...s/KubernetesClusterNodeCapacityReconcilerImpl.java 46.55% 39 Missing and 23 partials ⚠️
...utils/KubernetesClusterNodeCapacityReconciler.java 72.22% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main   #14223    +/-   ##
==========================================
  Coverage     19.91%   19.92%            
- Complexity    20200    20232    +32     
==========================================
  Files          6373     6375     +2     
  Lines        577230   577435   +205     
  Branches      70696    70750    +54     
==========================================
+ Hits         114974   115048    +74     
- Misses       449690   449797   +107     
- Partials      12566    12590    +24     
Flag Coverage Δ
uitests 3.71% <ø> (ø)
unittests 21.19% <34.28%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andrijapanicsb andrijapanicsb changed the title CKS: reconcile node capacity after live resize (WIP)CKS: reconcile node capacity after live resize Sep 22, 2026
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

ACSHomeBot commented Sep 22, 2026

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform
PASS RPM EL (EL8/9/10)
PASS DEB Ubuntu, Debian

Test packages are available at:

@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown

🔴 Test Coverage Grade: D — Marginal

Metric Value
Line coverage 24.84%
Branch coverage 19.07%

Grade Scale

Grade Line Coverage Meaning
🟢 A ≥ 80% Excellent - this code sleeps well at night 😴
🟡 B 60-79% Good - almost there, don't stop now 😉
🟠 C 40-59% Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D 20-39% Marginal - boldly shipping where no test has gone before 🖖
⛔ F < 20% Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
28.9% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube Cloud

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants