Kubernetes / control plane / autoscaling
Scale Up and Scale Down Are Two Different Programs
That the cluster autoscaler is one feedback loop that sizes the cluster to the load. It is two programs that never speak, and both of them reason about requests rather than usage — the second half being inherited from the scheduler and covered in the scheduling lesson. What is specific to the autoscaler is the asymmetry. Scale-up is a simulation against a node that does not exist: fork the snapshot, add one template node, take Pods[0] out of each equivalence group and run CheckPredicates on it. If that sample pod does not fit the template, it emits NotTriggerScaleUp and does nothing, forever, on every scan — and it never considers whether two new nodes would help, because there is only ever one test node in the snapshot. Scale-down never looks at pending pods at all: it compares the larger of the CPU and memory request ratios against --scale-down-utilization-threshold, and a node under the line still needs to clear five ordered blocking rules. The consequence people cannot predict is that both can be true at once — pods stuck Pending because no template fits them, while a node is drained and deleted for being quiet. The safety valve usually cited, --scale-down-delay-after-add, does not help, because it is armed by a scale-up and no scale-up happened.
The cluster autoscaler runs two unrelated algorithms under one name. Scale-up forks its cluster snapshot, adds one hypothetical node built from a node group's template, and asks the scheduler's own predicates whether a sample pod would fit on it. Scale-down walks real nodes comparing the sum of their pods' requests to allocatable, then runs an ordered drainability gate that has nothing to do with capacity. Neither loop reads the other's inputs, so a cluster can be removing a node and unable to schedule a pod in the same ten-second scan.
Where this is already explained
- The HPA Does Nothing Until You Are 10% Past Target
That a HorizontalPodAutoscaler with `averageUtilization: 80` keeps the workload near 80%, so a workload sitting above 80% and not scaling means the metrics pipeline is broken. The controller applies a dead band: `tolerances.isWithin(usageRatio)` returns the current replica count unchanged whenever the ratio of current to target is inside 1.0 +/- 0.1, so at a target of 80 nothing happens until the average crosses 88%, and nothing scales down until it falls under 72%. Worse, the second half of the same function re-adds every pod that has no metric yet at a usage of 0% whenever the answer would have been a scale-up, which routinely pushes the recomputed ratio back inside the dead band — so an HPA that has just added pods will refuse to add more until those pods start reporting, no matter how hot the running ones are.
- A High PriorityClass Does Not Get Your Pod Scheduled
That a high PriorityClass is a guarantee — the scheduler will evict whatever it must to run your pod. It is not, for two separate reasons that produce the same Pending pod. First, preemption is per-node and all-or-nothing: SelectVictimsOnNode removes every eligible victim from one node and re-runs the filters, and if the pod still does not fit, that node is dropped. Free capacity spread across several nodes is never combined, so a 6-core pod on 4-core nodes stays Pending with 'Preemption is not helpful for scheduling' while the cluster holds 20 cores of batch work. Second, preemption frees space, it does not reserve it: the successful result is a nominatedNodeName written to the pod's status, which is a hint for the next scheduling cycle, not a lock, and the victims still take their full terminationGracePeriodSeconds to leave. The related belief that a PodDisruptionBudget protects a pod from preemption is also false — the scheduler prefers victims that do not violate a PDB and prefers nodes with fewer violations, but it will violate one rather than give up, which the Kubernetes documentation states as 'PodDisruptionBudget is supported, but not guaranteed'.
- How kube-scheduler Places a Pod
That a pod goes Pending because the cluster is out of capacity, so the fix is a bigger or extra node. The scheduler never reads utilisation: it compares the pod's requests against allocatable minus the sum of every already-placed pod's requests, so a node running at 12% CPU can be 100% requested and refuse everything. Worse, because placement is greedy and per-pod with no backtracking, the default LeastAllocated scoring spreads small pods evenly and leaves every node with a hole too small for the next big one — the cluster has the CPU free, just never in one place.
3 published lessons depend on this concept, which is what moves it up the writing queue. Nothing is hidden behind this page — it has not been written.