A High PriorityClass Does Not Get Your Pod Scheduled

A PriorityClass does not buy your pod a node. It buys your pod one extra question, asked once per node, in isolation: "if I delete every pod on this node with a lower priority than you, would you fit here?" If the answer is no on every node, nothing is deleted and your pod stays Pending — and it will stay Pending with a cluster that has plenty of low-priority work in it, because the scheduler never adds up free space across two machines.

Preemption is the PostFilter phase of the ordinary scheduling cycle. It runs only after filtering has already failed on every node, and its job is to find a single node it can make room on. It is not a cluster-wide negotiation, it does not move pods, and it does not defragment anything. It is the same greedy, one-node-at-a-time reasoning as the filter and score phases, with one extra step.

Below is a five-node cluster. Each node has 4 CPUs of allocatable capacity and is completely full: five batch pods at priority 100 requesting 500m each, and two web pods at priority 1000 requesting 750m each. A new web pod arrives — priority 1000, requesting 3 CPUs. Read the readouts before you touch anything. The cluster holds 12.5 CPUs of preemptible batch work. The incoming pod wants 3. It is Pending.

Then move incoming pod priority one notch, from 1000 to 10000, and watch what has to die.

Why 12.5 free CPUs were unreachable

At the defaults every node reports the same thing:

node-01: removing all 5 eligible pods frees 2500m and the pod requests 3000m.

Five times over. The scheduler never adds those five 2500m figures together, and there is no place in the code where it could. This is SelectVictimsOnNode, which takes exactly one node:

// As the first step, remove all pods eligible for preemption from the node and
// check if the given pod can be scheduled without them present.
for _, pi := range nodeInfo.GetPods() {
    if pl.isPreemptionAllowed(nodeInfo, pi, pod) {
        potentialVictims = append(potentialVictims, pi)
    }
}
…
if status := pl.fh.RunFilterPluginsWithNominatedPods(ctx, state, pod, nodeInfo); !status.IsSuccess() {
    return nil, 0, status
}

That return is the whole story. If emptying this node of every lower-priority pod is not enough, the node is discarded and the loop moves to the next one. A node is either sufficient on its own or it is worthless. Because that is true of all five nodes here, the pod's event is:

0/5 nodes are available: 5 Insufficient cpu.
preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.

"Preemption is not helpful for scheduling" is the string worth memorising. It does not mean "the cluster is full". It means "there is no single node I could clear that would hold this pod". They are very different diagnoses with very different fixes: the first calls for more nodes, the second calls for bigger nodes, or a smaller pod, or a higher priority that widens the victim pool.

Try the last of those. Move incoming pod priority from 1000 to 10000. The web pods, at priority 1000, are now eligible victims too — preemptible CPU cluster-wide goes from 12.5 to 20.0, the reachable figure on one node goes from 2.5 to 4.0, and the pod schedules. That is what raising a priority actually does: it enlarges the set of pods the scheduler is allowed to kill on each node, one node at a time. It never makes two nodes into one.

Now move it the other way, to 100 — the same priority as the batch pods. Every node reports No preemption victims found for incoming pod and preemptible CPU cluster-wide reads 0.0. The rule in isPreemptionAllowed is strict inequality:

return corev1helpers.PodPriority(victim.GetPod()) < corev1helpers.PodPriority(preemptor) &&
    pl.IsEligiblePod(nodeInfo, victim, preemptor)

Equal priority is not enough. A cluster where every workload shares one PriorityClass — which is what you get if someone set globalDefault: true on a class and stopped there — has preemption switched off in practice, and the only symptom is Pending pods.

The PodDisruptionBudget got your web tier killed

Set incoming pod priority to 10000 and leave everything else alone. The scheduler kills five pods: three batch and two web pods at priority 1000, and the readout says two of the five were past their PodDisruptionBudget anyway.

Now untick a PodDisruptionBudget covers the batch pods. The scheduler kills six pods: five batch and one web. Zero PDB violations, obviously, since there is no PDB.

Read those two results next to each other. With the PDB in place, two production web pods died that would otherwise have lived, two batch pods died in violation of the budget that was supposed to protect them, and the total body count went down by one. The budget did not prevent a disruption. It redirected it onto a workload with a higher priority and no budget of its own.

This is not a bug in the model; it falls straight out of the order the victims are put back in. After the scheduler has emptied the node, it reprieves as many pods as it can, and it tries the PDB-violating candidates first:

violatingVictims, nonViolatingVictims := filterPodsWithPDBViolation(potentialVictims, pdbs)
…
for _, p := range violatingVictims {
    if fits, err := reprievePod(p); err != nil { … } else if !fits {
        numViolatingVictim++
    }
}
// Now we try to reprieve non-violating victims.
for _, p := range nonViolatingVictims { … }

Reprieving is first-come, first-served against a shrinking slack: the node has 4000m of eligible pods and the incoming pod needs 3000m, so only 1000m worth can be handed back. Whoever is offered a reprieve first gets one. Putting the PDB-violating batch pods at the front of that queue means they consume the slack, and the pods still standing when the slack runs out — the web pods, in the non-violating list — become the victims.

Two more properties of that code are worth knowing before you rely on a PDB to survive a preemption.

The budget is counted per node, from scratch, on every dry run. filterPodsWithPDBViolation starts each node's evaluation by copying pdb.Status.DisruptionsAllowed into a local counter. Five nodes are dry-run with the same budget of one. Nothing reconciles them, and the number the scheduler read may already be stale — that race between the scheduler and the disruption controller is the mechanism behind kubernetes/kubernetes#91492, open since 2020 with forty-one comments.

A violated budget is a tie-break, not a veto. In pickOneNodeForPreemption the PDB count is the first of five score functions, in this order: fewest PDB violations, then lowest highest-priority victim, then smallest sum of victim priorities, then fewest victims, then latest-started victims. Watch the criterion rows in the simulation's log as you move the controls — each one only runs to break the tie the one above it left. When every candidate node violates the budget equally, criterion one drops out entirely and the choice is made on priority. Nothing anywhere refuses to preempt. The Kubernetes documentation is explicit about this, and it is a section heading rather than a footnote: "PodDisruptionBudget is supported, but not guaranteed."

What the scheduler picked, and why it picked that

The victim set on a node is minimal, not arbitrary. Having emptied the node, reprievePod adds each candidate back and re-runs the filters; if the incoming pod still fits with that candidate restored, the candidate lives. So the scheduler kills the fewest pods that clear the space, and within the PDB split it offers reprieve in descending order of importance — MoreImportantPod, which is higher priority first and, on a tie, the older pod first.

Between nodes, the five tie-breaks in pickOneNodeForPreemption decide. Two of them surprise people:

Criterion four is "fewest victims", not "least CPU". Given a choice between killing one 4-CPU pod and four 1-CPU pods with the same priorities, the scheduler kills the one big pod. It is counting objects, not resources.

Criterion five prefers to kill the pods that started most recently. latestStartTimeScoreFunc takes the earliest start time among each node's victims and picks the node where that is latest. The intent is to preserve long-running work; the effect on a cluster that deploys frequently is that the freshest replica of whatever you just rolled out is the one selected.

Notice what is not on that list. Nothing considers whether a victim will be able to reschedule somewhere else, which is the substance of the open issue kubernetes/kubernetes#141227: a DaemonSet pod and a Deployment pod look identical to the victim chooser, and preempting a pod that has nowhere else to go simply relocates the Pending pod rather than resolving it.

And nothing rolls. All the victims on the chosen node are deleted together, in parallel, as soon as the decision is made — kubernetes/kubernetes#133102 is that report. If four replicas of one Deployment happen to share a node, preemption takes all four at once regardless of what its rolling update strategy says, because a rolling update is a controller's behaviour and this is a delete.

What preemption does not give you

It does not reserve the space it freed. The successful return value is NewPostFilterResultWithNominatedNode(bestCandidate.Name()), which writes status.nominatedNodeName on your pod. That is a hint. The pod then goes back through an ordinary scheduling cycle and competes for the node it just cleared, against every other pending pod, and the space is accounted for as if the pod were already there only through the NominatedPods mechanism the filters consult. A pod that sits with nominatedNodeName populated for minutes is usually waiting for the victims to actually die, and it will not preempt anywhere else while that is true — PodEligibleToPreemptOthers returns "not eligible due to a terminating pod on the nominated node."

It does not shorten a graceful shutdown. The victims are deleted with util.DeletePod and a normal delete, so each one gets its full terminationGracePeriodSeconds. A batch job with a 30-minute grace period is a 30-minute delay on the pod that preempted it, and there is nothing in the scheduler that overrides this.

It does not protect a running pod from anything else. Priority is a scheduling input. Once the pod is running, what matters under node pressure is the QoS class and the kubelet's eviction ranking, which is a different mechanism with a different order — the kubelet considers whether a container has exceeded its requests before it considers priority. Nor does priority affect CFS throttling or the cgroup CPU weight: that comes from the request. Every number preemption reasons about is a request, and limits are as invisible here as they are in filtering.

On a large cluster, it does not even look at every node. GetOffsetAndNumCandidates picks a random starting offset and dry-runs at most calculateNumCandidates(numNodes) nodes, which is MinCandidateNodesPercentage (default 10) percent of the nodes, floored at MinCandidateNodesAbsolute (default 100). Below 100 feasible nodes every node is examined and this does nothing. Above it, two identical pods can get different victim sets on different nodes, and a node that would have been the best candidate may simply not have been in the sample.

Reading it on a real cluster

Start with the second half of the FailedScheduling event. kubectl describe pod gives you both halves, and only the second is about preemption:

Warning  FailedScheduling  ...  0/40 nodes are available: 3 node(s) had untolerated
taint, 37 Insufficient cpu. preemption: 0/40 nodes are available:
37 Preemption is not helpful for scheduling.

Three readings. Preemption is not helpful for scheduling on every node means no single node can be cleared enough — right-size the pod or get bigger nodes. No preemption victims found for incoming pod means everything on those nodes is at your priority or above — the fix is priority, or realising that your "high priority" class is the same one everything else uses. And note that the taint-failing nodes are excluded from the preemption count entirely: findCandidates only considers nodes whose filter status was Unschedulable, never UnschedulableAndUnresolvable, so preemption can only ever fix a resource-shaped failure. It will not evict anything to get you past a nodeSelector, a taint or a volume-zone conflict.

Find out what you actually preempted. The event is written on the victim, not the preemptor:

kubectl get events --field-selector reason=Preempted -A --sort-by=.lastTimestamp
# Preempted by Pod default/indexer-7c9 on node ip-10-0-3-14

The victim also carries a pod condition — type DisruptionTarget, reason PreemptionByScheduler, message "<scheduler>: preempting to accommodate a higher priority pod". If something in your fleet is restarting for no reason you can find, that condition on the terminated pod is the answer, and it is the one signal that distinguishes preemption from a node-pressure eviction or an OOM kill.

Watch the two scheduler metrics that matter. scheduler_preemption_attempts_total counting up while scheduler_preemption_victims stays flat is the signature of the default state in this simulation: the scheduler is trying and failing on every cycle. A histogram of scheduler_preemption_victims with a long tail means single pods are clearing whole nodes, which is usually a pod that is too large for the node shape you bought.

Audit your PriorityClasses before you audit anything else. kubectl get priorityclass and look at three things: whether any class has globalDefault: true (only one may, and it silently becomes the priority of every pod that does not name a class), whether the values are spread out enough to be meaningful, and whether anything user-defined is close to 1000000000, which is HighestUserDefinablePriority. Above that are the two reserved system classes — system-cluster-critical at 2000000000 and system-node-critical at 2000001000 — and you want a real gap between your workloads and them, because a workload that can preempt kube-proxy will eventually preempt kube-proxy.

If you want the priority without the killing, say so. preemptionPolicy: Never on the PriorityClass keeps the queue ordering — a high-priority pod is still considered before low-priority ones, which is the whole benefit for a workload that can afford to wait — and turns off PostFilter for it entirely. Tick it in the simulation: the reason string changes to not eligible due to preemptionPolicy=Never and nothing dies. This is the right setting for most "important" batch work, and almost nobody sets it.

The next question this raises is what happens when preemption genuinely cannot help and the honest answer is "buy a node" — which is a decision made by a completely separate component, with its own reasons for saying no.