How kube-scheduler Places a Pod

The scheduler has never looked at your CPU graphs. When it decides whether a pod fits on a node it adds up the requests written in the specs of the pods already on that node and subtracts the total from the node's allocatable capacity. What those pods are actually doing with the CPU is not an input. A node sitting at 12% utilisation can refuse every pod you send it.

That much most people know, or half-know. The part that surprises them is what happens next. The scheduler places one pod at a time, and it never revisits a decision. Each pod is filtered against every node, the survivors are scored, and the winner is bound before the next pod is even considered. The default scoring rule prefers the emptiest node — so a batch of small pods spreads itself evenly across the cluster and leaves every node holding an identical hole, each one slightly too small for the large pod that comes next.

Below is the real thing: filtering with the actual reason strings from fit.go, then scoring with the actual LeastAllocated and BalancedAllocation formulas. The default cluster is six 4-core nodes running eighteen web pods. Ten and a half cores are unrequested and the whole cluster is running at 13% CPU. Now watch the indexer — one pod asking for 2 CPUs — go Pending anyway. Then flip placement order and watch it fit, with nothing else changed.

At the defaults the event reads 0/6 nodes are available: 6 Insufficient cpu. Every node has 1750m free and the indexer wants 2000m, so every node fails by 250 millicores — a quarter of a core — while 10,500m sits unclaimed across the cluster and the machines are 13% busy. Now set placement order to the indexer, then 18 web pods. Pending drops to zero. The pods are the same, the nodes are the same, the arithmetic is the same; only the order in which the scheduler saw them changed.

The one control that does nothing

Drag actual CPU used, as % of request from 5 all the way to 100. The usage bars grow, the "cluster CPU actually used" readout climbs from 3% to 53%, and not one placement changes. Pending stays where it was. The same nodes pass and fail the same filters.

This is not a simplification in the model. The filter is nine lines of fitsRequest in pkg/scheduler/framework/plugins/noderesources/fit.go, and the comparison it makes is:

podRequest.MilliCPU > nodeInfo.GetAllocatable().GetMilliCPU() − nodeInfo.GetRequested().GetMilliCPU()

GetRequested() is a running sum the scheduler's cache maintains by adding up spec.containers[].resources.requests for every pod assigned to that node. There is no term in that expression for what any process is doing. The kubelet does report usage, cAdvisor does export it, Prometheus does store it — and the scheduler subscribes to none of it.

So a request is a reservation. Setting requests.cpu: 2 on a pod that peaks at 200 millicores removes two whole cores from the cluster's schedulable pool for as long as that pod exists, and the only place that shows up is here — as some other pod going Pending. It does not show up on a utilisation dashboard, because from the dashboard's point of view nothing happened.

The reverse mistake is just as common. A pod that requests 100m and uses 3 cores schedules onto anything, then flattens whatever it lands next to. Requests are what the cluster plans with; limits are what the kernel enforces at runtime, and the scheduler reads only the first of the two. Raising a limit will never move a pod.

Filtering: what "allocatable" actually is

Set cores per node to 4 and untick both kube-reserved and DaemonSets. Each node offers a clean 4000m. Tick them back on and the same node offers 3550m — and 450m of that difference is invisible in kubectl get nodes, which shows capacity.

Allocatable is defined by the kubelet, not the scheduler:

Allocatable = Capacity − kube-reserved − system-reserved − eviction-threshold

kube-reserved is set aside for the kubelet and the container runtime, system-reserved for sshd and the rest of the OS, and the eviction threshold — evictionHard, typically memory.available<100Mi or a percentage — is held back so that the kubelet has room to act before the kernel does. The Kubernetes documentation is blunt about the consequence: "the scheduler does not over-subscribe Allocatable." Managed clusters set these for you on a sliding scale, so a 4-core node and a 64-core node in the same cluster surrender quite different fractions.

Then the DaemonSets get there first. A CNI agent, a log shipper, a node exporter and a CSI driver are on every node before any of your workloads, and their requests are in GetRequested() like anyone else's. In the simulation that is a flat 200m and 256Mi; a real cluster with a service mesh and a security agent can be several times that.

Two more filter rules are worth knowing because they produce error text that looks nothing like a resource problem:

• Too many pods. Drag kubelet maxPods down to 4. Nodes with capacity to spare start refusing pods on a pod count, because len(nodeInfo.GetPods()) + 1 > allowedPodNumber is checked before any resource is. The kubelet default is 110. On AWS with the VPC CNI the real ceiling is often lower and derived from how many IP addresses the instance type can hold, which is why the number in your cluster may be 29 or 58.
• Unresolvable. Push indexer requests.cpu above a node's whole allocatable — 4000m against a 4-core node whose allocatable is 3750m — and the log switches from "short by 250m" to UnschedulableAndUnresolvable. The distinction matters: a resolvable failure means evicting something might help, so preemption is worth attempting and the pod is retried whenever a pod elsewhere is deleted. An unresolvable one means no amount of freeing up will ever make this node work, and the cluster autoscaler will report pod didn't trigger scale-up (it wouldn't fit if a new node is added) rather than buying you a machine.

The event line assembles itself from those verdicts. Every rejected node contributes its reasons, the reasons are counted, each count is formatted as "<n> <reason>", and the strings are sorted — sorted as strings, which is why you sometimes see 10 Insufficient memory listed before 9 Insufficient cpu. So in 0/12 nodes are available: 3 Insufficient cpu, 9 Insufficient memory. the numbers are counts of nodes, not of cores or gigabytes. That is the single most common misreading of this message, and it matters because the two counts tell you whether you have one problem or two.

Scoring: the default rule spreads, and spreading is what fragments

Filtering says which nodes can hold the pod. Scoring picks. Every scoring plugin returns 0–100 for each feasible node, the scheduler multiplies by that plugin's weight and sums, and the highest total wins. Two of the default plugins respond to resource requests, and both are in the log:

• NodeResourcesFit, weight 1. Its default strategy is LeastAllocated, and the score for one resource is ((allocatable − requested) × 100) / allocatable with the incoming pod's request already added in. CPU and memory each carry weight 1, so the plugin's score is their mean. More free space scores higher.
• NodeResourcesBalancedAllocation, weight 1. It measures how far apart the CPU fraction and the memory fraction are: (1 − |fCPU − fMem| / 2) × 100, computed with and without the pod, then recombined so the result sits between 50 and 100. It rewards a pod that makes the node's two dimensions more even. It is why a CPU-heavy pod is nudged towards a node that is already memory-heavy.

The others contribute nothing here. ImageLocality scores 0 on every node when nobody has pulled the image. TaintToleration (weight 3), NodeAffinity (weight 2) and InterPodAffinity (weight 2) return the same value on every node when the pod declares no preferences, and a constant cannot break a tie. What is left, at the defaults, is a rule that says: put the pod on the emptiest node.

Now follow what that does over eighteen pods. Pod 1 sees six identical nodes and lands somewhere. Pod 2 sees five nodes at 200m and one at 800m, so it avoids the one that just got a pod. By pod 18 the cluster is perfectly level — three web pods on every node, 1750m free on every node, and not one node with room for a 2000m pod. The scheduler did not fail. It did exactly what it was configured to do, eighteen times in a row, and the outcome nobody asked for is the sum of eighteen locally correct decisions.

Two things about the algorithm make this permanent rather than temporary. It is greedy: each pod is bound before the next is looked at, and the scheduler has no idea the indexer is coming. There is no backtracking: once web-4 is on node-01 the scheduler will not move it to make room, not now and not in an hour. Nothing in core Kubernetes ever re-packs a running cluster. Rescheduling is a separate, non-default component (the descheduler) that works by evicting pods so the scheduler gets another go.

Switch scoringStrategy to MostAllocated and watch the picture invert. The web pods pile onto node-01 until it is full, then node-02, and two nodes are left almost empty — so the indexer fits, and Pending goes to zero. That is a real, supported setting:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated

It is also a trade, not an upgrade — see the next section.

One caveat about the model. Real clusters also run PodTopologySpread with built-in default constraints — maxSkew: 3 across kubernetes.io/hostname and maxSkew: 5 across topology.kubernetes.io/zone, both ScheduleAnyway — which adds another spreading pressure for pods that belong to the same ReplicaSet. It only scores nodes that carry those labels, and it is a soft preference rather than a filter, so it changes which node wins a close call without changing anything about the arithmetic above. The simulation leaves it out so that the two resource plugins are legible on their own.

Where each fix stops working

Add nodes. Take nodes from 6 to 8 with everything else at the defaults. The indexer schedules — but not because the cluster gained capacity. It schedules because eighteen pods no longer divide evenly, so two nodes end up with two web pods instead of three and one of them has a big enough hole. Go to 9 nodes and it still works; go to 12 and it works with room to spare. This is the fix everyone reaches for and it does work, which is exactly the problem: it works by accident, at the price of a permanently over-provisioned cluster, and the next uneven workload puts you back where you started.

Use bigger nodes. Take cores per node from 4 to 8 and leave the node count at 6. Now each node holds more web pods and the leftover hole is bigger, so the indexer fits easily. Bigger nodes genuinely do reduce fragmentation, because the wasted remainder is a smaller fraction of a large node than of a small one. The boundary is visible from the other end: drag cores down to 2. Allocatable collapses to 1750m, seven pods go Pending instead of one, and the 2000m indexer is now Unresolvable on every node in the cluster — no number of 2-core nodes will ever hold it. Somewhere between those two is your real constraint, and it is the ratio of your largest pod to your node size — not your total core count.

Switch to MostAllocated. It fixes fragmentation and it costs you two things. Packing pods tightly onto a few nodes means every node failure takes down a larger share of every workload, and it means the nodes that are full are really full — so a pod that bursts above its request has no slack on the machine to burst into, and its neighbours feel it. Bin-packing is the right default for batch and the wrong default for latency-sensitive services, which is why the shipped default is the other one.

Let the autoscaler handle it. The cluster autoscaler watches for Pending pods and adds nodes, and for the default case here it would work. Set indexer CPU to 3600m against 4-core nodes and it stops working, in the specific way the logs describe: the autoscaler runs the same filter against a simulated new node of each node group's shape, sees the pod fail there too, and emits pod didn't trigger scale-up (it wouldn't fit if a new node is added). It is a scheduler simulation, not a capacity oracle, so anything that fails on an empty node of your largest shape will sit Pending forever while the autoscaler declines to act.

Preempt something. Give the indexer a higher priorityClassName and the postFilter phase will look for a set of lower-priority pods whose removal would make one node feasible. This is the only mechanism in core Kubernetes that can undo fragmentation on demand — and it is not free: the victims are deleted with their terminationGracePeriodSeconds honoured, and they then go back into the queue and may fragment the cluster somewhere else. Note also what the event says when there is nothing to take: 0/6 nodes are available: 6 No preemption victims found for incoming pod.

Shrink the requests. Drop web requests.cpu from 600m to 500m and everything fits with room over. This is usually the correct fix and it is the one with the sharpest boundary, because you are now trading a scheduling problem for a runtime one. The request is also the pod's weight under contention and its floor under node-pressure eviction — cut it below what the pod actually needs and you have moved the failure from "Pending, obvious, at deploy time" to "slow and evicted first, at 3 a.m." Right-sizing means measuring the p95 of real usage and requesting that, not requesting whatever makes the scheduler stop complaining.

Reading it on a real cluster

The single most useful command is the bottom of kubectl describe node. It prints exactly the numbers the filter uses, and nothing else does:

Capacity:
  cpu:                4
  memory:             16374624Ki
  pods:               110
Allocatable:
  cpu:                3920m
  memory:             15360Mi
  pods:               110
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource   Requests     Limits
  cpu        2000m (51%)  4 (102%)
  memory     1792Mi (11%) 6Gi (40%)

Three things to read there. Capacity versus Allocatable is your reservation overhead. The Requests column is the number the scheduler subtracts — if it says 51% then 49% of allocatable is schedulable, whatever top says. And the parenthetical warning that limits may exceed 100% is the giveaway that limits are not a scheduling input: the API server will happily accept a node's worth of limits three times over, because nothing ever adds them up for placement.

To find fragmentation across the whole cluster rather than one node at a time, the quantity you want is the largest free block, not the total:

kubectl get nodes -o json | jq -r '
  .items[] | [.metadata.name, .status.allocatable.cpu] | @tsv'

kubectl describe nodes | grep -A5 "Allocated resources"

If total free CPU is comfortable and the largest single free block is smaller than your biggest pod, you have this problem and adding nodes is treating a symptom.

For a specific Pending pod, the event is authoritative and it expires:

kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp
kubectl describe pod <name> | sed -n '/Events:/,$p'

Read the histogram before doing anything. 12 Insufficient cpu on a twelve-node cluster is a uniform problem — every node is equally full, which is the fragmentation signature. 2 Insufficient cpu, 10 node(s) had untolerated taint is a completely different bug and no amount of CPU will fix ten of those nodes.

On the scheduler itself, two metrics are worth an alert. scheduler_pending_pods is split by queue, and the unschedulable queue growing while active stays flat means pods are being parked rather than churned. scheduler_pod_scheduling_attempts is a histogram — a long tail there means pods are being retried repeatedly, which is what fragmentation plus an autoscaler that cannot help looks like from the control plane. scheduler_schedule_attempts_total{result="unschedulable"} separates "could not place" from result="error", which is a different and much rarer problem.

One scaling note that surprises people running large clusters: above roughly 100 nodes the scheduler stops looking at all of them. The percentageOfNodesToScore setting defaults to a value computed from cluster size, with a hard floor of 5%, and the scheduler stops filtering as soon as it has found enough feasible nodes to score. So on a 5,000-node cluster the "best" node the scheduler picks is the best of a sample, and two identical pods can land very differently. Below 100 nodes every node is always considered and this setting does nothing.

Next: what happens to that pod once it lands — the request becomes a scheduling weight and the limit becomes a hard quota, and why the memory limit kills instead of throttling.