Kubernetes / node / resource management / cgroups
cgroup v2: The Limit That Kills You Is Four Levels Above Your Container
That migrating a node to cgroup v2 renames files and changes nothing — memory.limit_in_bytes becomes memory.max, cpu.shares becomes cpu.weight, same behaviour. Two structural changes bite. First, the single hierarchy means your container's memory limit is only the innermost of three or four limits it is charged against, and when the failing one is kubepods.slice — which the kubelet limits by default — the OOM domain becomes every pod on the node, the badness denominator becomes node allocatable rather than your limit, and oom_score_adj stops cancelling out and starts outweighing resident memory by three orders of magnitude, so the pod that dies is the Burstable pod with the smallest memory request rather than the pod that allocated. Second, since Kubernetes 1.28 the kubelet sets memory.oom.group=1 on container cgroups, so the kernel kills every process in the victim's container instead of the single fattest one.
Your container has a memory limit. It is not the limit that kills it. On cgroup v2 every page your process touches is charged against your container and your pod and the slice that holds every pod on the node, in that order, and the first one of those to reach its ceiling is the one that runs the out-of-memory (OOM) killer — the kernel routine that picks a process and kills it when a memory request cannot be satisfied. Which one it is decides who dies — and it is routinely a container three levels away from the one that asked for memory.
A control group, or cgroup, is a kernel object that accounts and limits the
resources of the processes inside it. cgroup v1 gave every resource its own
separate tree: one hierarchy for memory, another for CPU, another for
pids, and a process could sit in unrelated places in each.
cgroup v2 collapses all of them into a single hierarchy — this is what
"unified" means — and it is now effectively the only option: Kubernetes
deprecated cgroup v1 in v1.35, and from that release the kubelet refuses to
start on a v1 node unless an administrator sets failCgroupV1
to false.
The migration is usually described as a rename. memory.limit_in_bytes
becomes memory.max, cpu.shares becomes
cpu.weight, cpu.cfs_quota_us and
cpu.cfs_period_us collapse into one
cpu.max file. That part is true and it is the least interesting
thing about it. The change that shows up in your incident channel is
structural: because there is one tree, your container is now four levels
deep in it, and every level above you is a limit you did not set and an
OOM domain you are a candidate in.
Below is that tree, running. Six pods on one 16 GiB node. Every one of them
was admitted by the scheduler, because the scheduler adds up
requests, and the requests come to 5,376 MiB. The
limits come to 21,632 MiB, on a slice the kubelet has capped at
15,360. Your pod is web, it is using 320 MiB of a 1,024 MiB
limit, and it is going to have a bad time. Start with worker processes in
your container: drag it from 3 down to 1.
web
Also fixed on the node: redis, Guaranteed at 2048Mi and
using 1900; and your pod's otel-agent companion
container, 128Mi request and limit, one process, using 96. It is an
ordinary second entry in spec.containers, not a native
sidecar — see the note in the request section for why that matters.
Three minutes of wall clock, stepped at one second, on a node with 16 GiB
of RAM. Each second every container tries to charge what it needs. The
charge walks up the tree exactly as page_counter_try_charge()
does, and the first ancestor whose memory.max would be
exceeded becomes mem_over_limit: reclaim runs in
that cgroup's subtree, and if reclaim cannot free enough, the OOM
killer scans that cgroup's subtree. Sizes and growth rates are
illustrative; the arithmetic on top of them — the walk, the reclaim
target, the badness formula, the group kill — is the kernel's own.
One row per level of the single hierarchy. The bar is
memory.current against memory.max.
the level the charge
actually failed at ·
admitted the charge ·
no limit set, so it can
never fail.
oom_badness() ranks processes, not containers:
points are the task's resident pages plus
oom_score_adj × totalpages / 1000, and for a cgroup OOM
totalpages is that cgroup's own memory.max.
oom_score_adj is a per-process integer from −1000 to 1000
that the kubelet writes from the pod's
Quality of Service (QoS) class — the Guaranteed /
Burstable / BestEffort label in the second column, which Kubernetes
assigns from how the pod's requests compare to its limits.
Each row is the largest process in a container, modelled as an equal
share of the container's anonymous memory. The model leaves out page
tables and swap entries, which the kernel also counts and which are
under a percent here.
charges admitted · a charge failed and reclaim covered it · reclaim could not, and something was killed.
At three worker processes your container is never killed, and it is
second in the queue. Read the badness board: your
server, holding 300 MiB, scores 15,352 points. The
checkout-b container holding 3,482 MiB — eleven times as
much, and the container that actually asked for the memory — scores 15,278
and survives. Drag worker processes to 1 and your container starts
dying: four kills, and 735 MiB of your page cache thrown away on the way
there. Drag it to 12 and the kills stop. You did not change how much
memory your pod uses by a single byte.
One tree, four limits, and only one of them is yours
The rows in the tree panel are not a diagram. They are the actual directory
structure under /sys/fs/cgroup on a node using the systemd
cgroup driver, and each of them holds a real memory.max file
that the kubelet writes.
/, the root cgroup. Nomemory.maxfile exists here — you cannot limit the machine below itself. Physical memory is the ceiling, and reaching it means the global OOM killer rather than a cgroup one. The simulation draws it as a 16,384 MiB limit so you can watch the failure move up to it.kubepods.slice. Everything Kubernetes runs. The kubelet writesmemory.maxhere equal to node capacity minuskubeReservedminussystemReserved, and it does this by default:enforceNodeAllocatabledefaults to[pods]. On the simulated node that is 16,384 − 1,024 = 15,360 MiB. The kubelet writes this file whether or not you enable enforcement — turning enforcement off raises the number to full node capacity rather than removing it, which the last section returns to.kubepods-burstable.slice. One per QoS class — Guaranteed pods live directly underkubepods.slice, Burstable and BestEffort get their own slice. The kubelet setscpu.weighthere but leavesmemory.maxatmaxunless you turn on the alphaQOSReservedfeature gate and pass--qos-reserved. This is why the row is grey and can never fail.- The pod cgroup.
memory.maxis the sum of the containers' memory limits — and only if every container declares one. Miss a limit on one sidecar and the whole pod cgroup goes unlimited. - The container cgroup. Your
limits.memory. The only number in the whole tree that appears in your YAML.
The nesting is not a Kubernetes preference. cgroup v2 has a rule the kernel
documentation calls the no-internal-process constraint: a non-root cgroup
may distribute resources to its children only if it has no
processes of its own. So the moment the kubelet wants a limit on "all pods"
it must create kubepods.slice as a pure container of children
and push every process one level deeper. Controllers themselves are enabled
top-down through cgroup.subtree_control, and the kernel
enforces that a child can only enable controllers its parent has already
enabled. That is the whole reason your container is at depth four rather
than depth one.
And this is where the rename story breaks. On cgroup v1 the memory
controller had its own hierarchy and so did the CPU controller, and while
memory limits were hierarchical there too, the tree they were hierarchical
in was a different tree from the one deciding CPU. On v2 there is
one tree, so the level that stops your memory allocation is the same level
that owns your CPU weight, your pids.max, and your I/O weight.
One structure, one set of ancestors, one place for all of them to go
wrong.
What the charge actually does
When a process first touches a page, the kernel charges it to the process's
memory cgroup by calling page_counter_try_charge(). That
function's own comment says what it does: "try to hierarchically charge
pages… Returns true on success, or false and @fail if the
counter or one of its ancestors has hit its configured limit." The loop is
literally for (c = counter; c; c = c->parent). Every ancestor
is incremented; if any of them would go over its max, the
increment is rolled back and a pointer to that counter is returned.
try_charge_memcg() then takes that pointer and names it
mem_over_limit. Everything after this point uses
mem_over_limit and not your container:
try_to_free_mem_cgroup_pages(mem_over_limit, …)— reclaim scans the failing cgroup's whole subtree.mem_cgroup_oom(mem_over_limit, …)— the OOM killer's candidate set is the failing cgroup's whole subtree.oc->totalpages = mem_cgroup_get_max(oc->memcg)— the denominator in the badness formula is the failing cgroup's limit.
Three consequences, and each one is a different thing going wrong. Take them in order.
totalpages is the whole trick
oom_badness() is four lines of arithmetic:
points = rss + swapents + pagetables; points += oom_score_adj × (totalpages / 1000)
oom_score_adj is a per-process integer from −1000 to 1000 that
the kubelet sets from the pod's QoS class: −997 for Guaranteed, 1000 for
BestEffort, and for Burstable
1000 − 1000 × containerMemoryRequest / nodeCapacity, clamped
into 3…999. Across that whole range it is a bias on the ranking, not a veto.
The one value that is a veto is −1000, and Kubernetes never writes
it: oom_badness() returns LONG_MIN the moment
adj == OOM_SCORE_ADJ_MIN, and the cgroup v2 documentation says
such tasks "are treated as an exception and are never killed" even under a
group kill. The kubelet's most protective value is −997, which is a very
large bias and still a bias.
The important part is that totalpages is the OOMing cgroup's
own limit, so the same oom_score_adj is worth a completely
different number of megabytes depending on which cgroup failed. Move
server: memory in use to 1,200 MiB, above its 1,024 MiB limit,
and watch the board collapse to a single row: the OOM domain is now your
own container, totalpages is 1,024 MiB, and one point of
oom_score_adj buys 1.0 MiB. Put it back to 320 and the domain
is kubepods.slice, totalpages is 15,360 MiB, and
one point buys 15.4 MiB. Fifteen times the leverage, from a number
nobody changed.
That factor is what produces the default run's result. At the first kill:
batch/worker— BestEffort, 2,639 MiB in two processes, so 1,319 MiB in its largest.oom_score_adj1000, worth +15,360 MiB. Total 16,679. Killed.web/server— that is you — Burstable, 300 MiB in three processes, 100 MiB in the largest.oom_score_adj993, worth +15,252 MiB. Total 15,352. Second in line.checkout-b/app— the container that actually asked for the memory. 3,482 MiB in four processes, 870 MiB in the largest.oom_score_adj938, worth +14,408 MiB. Total 15,278. Fourth. Survives.
Your 300 MiB container outranks a 3,482 MiB one. Two things did that. The
smaller one is the request: 128Mi against 1024Mi is 55 points of
oom_score_adj, and at this level 55 points is 845 MiB. The
larger one is process count, and it is the one nobody expects.
The kernel ranks processes; Kubernetes bills containers
select_bad_process() calls
mem_cgroup_scan_tasks(oc->memcg, oom_evaluate_task, oc). Tasks,
not cgroups. A container that forks four workers presents four candidates
of a quarter the size each, and each of them is scored on its own
rss. The kubelet's own source says so, in a comment above the
Burstable formula: "Note that this is a heuristic, it won't work if a
container has many small processes."
Drag worker processes in your container and watch the board reorder while every byte of memory stays where it was:
- 1 process — your largest task is the whole 300 MiB. 15,552 points, still second, but now the scores above and below you shift enough that your container is killed four times in the run and gives up 735 MiB of page cache.
- 3 processes — 100 MiB each, 15,352 points, second and never killed.
- 12 processes — 25 MiB each, 15,277 points. You drop below
checkout-b/appto fourth and stay there.
This is a genuinely perverse incentive and it is worth saying plainly: on a
node where the binding limit is kubepods.slice, splitting your
heap across more processes makes you harder to kill, and it does so without
freeing a single page. It also degrades gracefully in the wrong direction —
see the group-kill section below, where more processes means a bigger blast
radius when you are picked anyway.
Guaranteed is a 30 GiB handicap, at one level only
Set server: requests.memory equal to server: limits.memory —
both at 1024Mi — and the pod becomes Guaranteed, because
otel-agent already has matching request and limit. Your
oom_score_adj goes from 993 to −997 and your score goes from
+15,352 to −15,214: last but one, below a redis holding
1,894 MiB. The swing is 1,990 points at 15.36 MiB each, which is 30,566 MiB
of head start on a node that only has 16,384.
At the node level, therefore, QoS is not a tie-breaker. It is decisive, and Guaranteed pods are effectively unkillable by a neighbour's allocation. Now push server: memory in use past your own limit while still Guaranteed. The domain becomes your container, every process in it carries the same −997, the constant cancels out of a ranking that only compares processes inside one container, and you die exactly as fast as you would have at any other QoS. That asymmetry is the single most useful thing to know here: Guaranteed protects you from your neighbours, never from your own total.
The part with no metric: reclaim
Long before anything is killed, the failing charge does something quieter.
try_to_free_mem_cgroup_pages(mem_over_limit, …) walks the
failing cgroup's subtree looking for reclaimable pages, and the cheapest
reclaimable pages on a node are clean file-backed pages — page cache. It
does not care which container faulted them in, and it certainly does not
care which container asked for the memory that triggered the reclaim.
At the simulation's defaults your web pod is never killed and
never comes near its own limit, and it still gives up 213 MiB of page
cache over three minutes. It never held more than 141 MiB at once: that 213
is a running total, the pod being stripped, refilling, and being stripped
again, and it ends the run at zero cache. The your page cache
reclaimed readout is that running total, and the reason it is worth
staring at
is that no Kubernetes signal reports it. Your
container_memory_working_set_bytes goes down. Your
limit is untouched. Your restart count is zero. Your p99 doubles, because
every read that used to hit cache now hits the disk.
Drag share of each pod that is page cache across its range and watch the two effects trade off:
- 0% — nothing to reclaim, so every failed charge goes straight to the OOM killer. First kill at t+75 s, 19 kills in the run, and your pod loses only 41 MiB.
- 35% — first kill at t+80 s, 18 kills, 213 MiB of your cache gone.
- 80% — first kill at t+93 s, only 10 kills, and your pod surrenders 1,321 MiB of cache to buy that quiet.
Page cache is the shock absorber, and the bill is paid by whoever has the
most of it, not by whoever caused the pressure. That is the honest reading
of the memory.stat counters: pgscan and
pgsteal climbing on a container that is doing nothing wrong.
cgroup v2 does give you an instrument for this that v1 never had. Pressure
Stall Information (PSI) exposes memory.pressure in every
cgroup, reporting the share of wall-clock time in which tasks were stalled
waiting on memory — some for at least one task,
full for all of them. The kernel raises it around exactly this
reclaim path: try_charge_memcg() wraps the reclaim call in
psi_memstall_enter() and psi_memstall_leave().
Reading memory.pressure on kubepods.slice is the
only cheap way to see the node-level charge failures at all.
memory.high, and why Guaranteed pods do not get one
cgroup v2 also adds memory.high, which the kernel documentation
describes as a "memory usage throttle limit": crossing it puts the cgroup
under heavy reclaim pressure and stalls the allocator, but it "never invokes
the OOM killer." It is the middle setting that cgroup v1's memory controller
did not have.
Kubernetes reaches it through the MemoryQoS feature gate, which was alpha
and off from v1.22 and becomes beta and on by default in v1.37. When it is
on, the kubelet computes
memory.high = floor((requests.memory + factor × (limits.memory −
requests.memory)) / pageSize) × pageSize from
memoryThrottlingFactor. It can also write the protection side of
the pair — memory.min for Guaranteed pods,
memory.low for Burstable ones, both taken from the request — but
only if you ask: that branch is gated on
memoryReservationPolicy == TieredReservation, and the kubelet
defaults the policy to None, whose else-branch writes literal
0 to both files. Through v1.36, then, turning MemoryQoS on gets
you the throttle — memoryThrottlingFactor defaults to 0.9 — and
none of the protection. Check both settings on your own kubelet rather than
assuming the gate did what its name suggests. Read the guard on the throttle
itself in
kuberuntime_container_linux.go, too:
memoryRequest != memoryLimitSpec || memoryRequest == 0. A
Guaranteed container has request equal to limit, so it is skipped. The
safest QoS class is the one with no throttle between healthy and dead.
Group kill: the 1.28 change nobody announced to your team
memory.oom.group is a cgroup v2 file with no cgroup v1
equivalent. The kernel documentation says that when it is set, "all tasks
belonging to the cgroup or to its descendants … are killed together or not
at all." Its kernel default is 0.
Kubernetes overrides that default. Pull request #117793, "use the cgroup
aware OOM killer if available", landed in the v1.28 milestone and
writes "memory.oom.group": "1" into the container's
Unified map on every cgroup v2 node. Its release note is one
sentence: processes within the cgroup "will be treated as a unit and killed
simultaneously in the event of an OOM kill on any process in the cgroup."
On cgroup v1 one worker died and the container carried on, usually still
passing its readiness probe. On v2 from 1.28, the container is gone.
Untick memory.oom.group on containers and the run
changes shape completely: 84 kill events instead of 18. Each
single-process kill frees a quarter of a container, the supervisor respawns
the worker, the worker refills, and the node is back where it was seconds
later. The container never restarts, so
kubectl get pods shows RESTARTS 0 throughout and
the pod stays Ready while it is being dismembered. That is the cgroup v1
behaviour people describe as gentler, and it is why the group kill was
introduced.
It is also why the escape hatch exists. Kubernetes v1.32 added the
singleProcessOOMKill kubelet option, whose documentation reads:
"if true, will prevent the memory.oom.group flag from being set
for container cgroups in cgroups v2… the behavior aligns with the behavior
of cgroups v1." Its default is false, and on cgroup v1 nodes
only null or true is even accepted. It took two
attempts and ten months: pull request #122813 proposed it in January
2024, ran to forty-eight comments, and was closed unmerged that May; #126096
reproposed the same option and merged in November 2024, for v1.32. That is a
reasonable proxy for how much of a behaviour change 1.28 actually shipped.
One detail that decides the blast radius. mem_cgroup_get_oom_group()
walks from the victim's cgroup up to the OOMing cgroup and picks
the highest ancestor with oom.group set. Kubernetes sets
it on container cgroups and nowhere else. So when
kubepods.slice is the OOM domain, the walk goes
container → pod → QoS slice → kubepods.slice, finds the flag
only at the bottom, and kills exactly one container. Your other containers
survive, your pod stays Running, and the only evidence is
RESTARTS ticking on one container. Issue #124253 is people
asking for the flag at the pod level instead; it is not there.
Where each fix stops working
Raise your memory limit. Drag server: limits.memory from
1024Mi to 8192Mi and compare the board. It is byte-for-byte identical: same
order, same points, same victim. limits.memory is not an input
to oom_badness() at any level above your own container, and at
the node level it is not an input at all. What it does change is the pod
cgroup's memory.max, which climbs to 8,320 MiB and is now even
further from being the binding constraint. You have bought protection
against exactly one failure mode — your own container overrunning — and none
against the one the simulation is showing you.
Raise your memory request. This one does something, and the amount is
computable. Each point of oom_score_adj is worth
totalpages/1000, and each 1000th of node capacity of request is
worth one point. On a 16 GiB node that is 16.4 MiB of request per point and
15.36 MiB of head start per point at the kubepods.slice level:
a 1 GiB request buys 62 points, or 952 MiB. Take the request from
128Mi to 2048Mi with the limit at 8192Mi and your score drops from 15,352 to
13,540 and you fall below your own otel-agent, which did not
move. Note the denominator, though: it is nodeCapacity, not
allocatable and not anything you control. The same 1 GiB request on a 64 GiB
node buys 15 points instead of 62, so this lever gets weaker exactly as you
move to the larger nodes where the node-level limit is most likely to bind.
One carve-out, because it changes this answer entirely. If
otel-agent were a native sidecar — an entry in
spec.initContainers carrying
restartPolicy: Always, which is what Kubernetes now means by
the word — GetContainerOOMScoreAdjust would take a different
branch for it. A native sidecar's score is clamped down to the score implied
by the smallest memory request among the pod's regular containers,
so raising your own request would drag the sidecar's protection along with
it instead of leaving it behind. The simulation models the ordinary case, a
second entry in spec.containers, where the two are scored
independently. If your observability agent is a native sidecar, expect a tie
rather than the overtake shown here.
Go Guaranteed. The strongest available move at the node level, worth
30,566 MiB of ranking as shown above, and it costs you the whole point of
the Burstable class: you now pay for your peak all the time, and
the scheduler reserves your limit rather than your average.
It is also worth nothing inside your own container, and — with MemoryQoS on
— it removes your memory.high cushion.
Turn off node-allocatable enforcement. Untick
enforceNodeAllocatable: [pods] and watch the
kubepods.slice row: its memory.max does not
disappear, it goes from 15,360 to 16,384 MiB. This surprises people,
so it is worth reading the kubelet's own reason for it.
enforceNodeAllocatableCgroups() runs whether or not you asked
for enforcement, and the comment above the line says why: "We need to update
limits on node allocatable cgroup no matter what because default cpu shares
on cgroups are low and can cause cpu starvation." The only thing your
setting changes is which number gets written —
nodeAllocatable := cm.internalCapacity, the machine's full
memory, instead of capacity minus the reserves.
A limit equal to the whole machine is not the same as no limit, but it is
close, and the difference is who else is on the machine. The root cgroup
holds the pods and the node's daemons; kubepods.slice
holds only the pods. Both now have a ceiling of 16,384 MiB, so the root
reaches it first by exactly the amount the daemons are using, and the charge
walk — which tests kubepods.slice before the root — sails
through the slice and fails at the root. That is what the tree shows:
kubepods.slice at 15,422 of 16,384, the root at 16,190 of
16,384, and the failure marked on the root. Drag kube- +
system-reserved to 0 with enforcement still off and the failure
snaps back down to kubepods.slice, because with no daemons
holding anything the two rows carry identical usage and the lower one is
tested first.
So the rescue is much smaller than it looks. The first kill moves from
t+80 s to t+82 s. totalpages goes from 15,360 to 16,384, which
makes oom_score_adj slightly more powerful, not less.
The same container dies. What actually changes is the reclaim scope, and the
last line of the decision log reports it. With enforcement on, the node's own
daemons give up 0 MiB of page cache all run, because
system.slice is not in the failing cgroup's subtree. With it
off they give up 256 MiB — all of it — because now they are.
Enforcement does not decide who dies. It decides whether sshd, containerd
and the kubelet are inside the blast radius of your pods' memory pressure.
The two candidate rows the root cgroup adds explain why the daemons do not
simply die instead: kubelet.service scores −16,163 because the
kubelet sets its own oom_score_adj to −999, and
system.slice scores 28 — its 563 MiB spread over twenty
daemons, with an oom_score_adj of 0 adding nothing — against
pods scoring five figures. Every pod on the node is a more attractive victim
than sshd by roughly the whole size of the machine. That is
oom_score_adj doing its job.
Reduce kube-reserved. Drag kube- + system-reserved to 0 and
kubepods.slice gains 1,024 MiB: the first kill moves from
t+80 s to t+92 s and the run has 7 kills instead of 18. Drag it to
2048Mi and the first kill arrives at t+69 s with 22. Every megabyte you
reserve for system daemons is a megabyte the pods provably cannot have, and
the Kubernetes documentation describes this enforcement purely as
"evicting pods whenever the overall usage across all pods exceeds
'Allocatable'" — which is true and incomplete. Eviction is the kubelet's
reaction, on its own polling interval. The cgroup limit underneath it is the
kernel's reaction, and it happens in the page fault.
Set a pod-level limit. spec.resources.limits.memory —
Kubernetes Enhancement Proposal (KEP) 2837, beta and on by default since
v1.34 — writes the pod cgroup's
memory.max directly instead of summing containers. Set
pod-level limits.memory to 512Mi and server: memory in use to
700, and a fourth OOM domain appears: the pod cgroup, with two candidates
and totalpages of 512 MiB. Now one point of
oom_score_adj is worth 0.5 MiB, both containers carry the same
993, so the adjustment cancels and the ranking is pure resident size:
server's largest process at 152 MiB scores 660 against
otel-agent's 67 MiB at 576. Eight kills of your own container in
three minutes, from a limit no container in the pod ever exceeded.
Get a bigger node. This one genuinely works, and it is worth understanding why it is not a trick: the sum of limits stops exceeding the slice. It is also the only fix on this list that costs money rather than resilience, which is presumably why it is the one everyone reaches for.
Checking it on a real node
First, confirm which version you are on. On the node:
stat -fc %T /sys/fs/cgroup/→cgroup2fsfor v2,tmpfsfor v1. cgroup v2 support has been stable since Kubernetes v1.25 and cgroup v1 is deprecated as of v1.35, where the kubelet refuses to start on it unlessfailCgroupV1is set tofalse.
Then read the ceiling that is not in anybody's YAML. Every path below assumes
the systemd cgroup driver, which is what puts
.slice on the end of every directory name, and which
kubeadm-provisioned and most managed clusters use. It is not the kubelet's
built-in default — that is still cgroupfs, which lays the same
cgroups out without the suffixes:
/sys/fs/cgroup/kubepods/ rather than
/sys/fs/cgroup/kubepods.slice/. Check which one you have with
kubectl get --raw /api/v1/nodes/<node>/proxy/configz and
look for cgroupDriver before copying any path from here.
cat /sys/fs/cgroup/kubepods.slice/memory.maxandmemory.current. Compare the first againstkubectl get node -o jsonpath='{.status.allocatable.memory}': the cgroup number is larger, by exactly yourevictionHard: memory.available, because the scheduler's Allocatable subtracts the eviction threshold and the cgroup limit does not.cat /sys/fs/cgroup/kubepods.slice/memory.events→low,high,max,oom,oom_kill,oom_group_kill. A non-zeromaxhere is the single most under-read number on a Kubernetes node: it counts the times a charge failed at the node level and had to reclaim. It rises long beforeoom_killdoes.cat /sys/fs/cgroup/kubepods.slice/memory.pressure→some avg10=… full avg10=….fullabove a few percent means every task on the node was stalled on memory at once.
When something is killed, the kernel log settles it in one line. The victim
dump prints oom_memcg= — the cgroup that hit its limit — and
task_memcg= — the cgroup the victim was in:
checkout invoked oom-killer: gfp_mask=0x1100cca, order=0, oom_score_adj=938
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-9f3….scope,
mems_allowed=0,oom_memcg=/kubepods.slice,
task_memcg=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71….slice/cri-containerd-2ae….scope,
task=worker,pid=41207,uid=0
Tasks in /kubepods.slice/…/cri-containerd-2ae….scope are going to be killed due to memory.oom.group set
Memory cgroup out of memory: Killed process 41207 (worker) total-vm:1584312kB, anon-rss:1350784kB,
file-rss:12608kB, shmem-rss:0kB, UID:0 pgtables:2784kB oom_score_adj:1000
Four things to read off it, in order of usefulness:
- If
oom_memcgandtask_memcgdisagree, you were killed for somebody else's allocation. Hereoom_memcg=/kubepods.sliceand the task was in a BestEffort pod's container: the node-level limit failed and the kernel picked the highest-badness process on the whole node. No amount of tuning that container's limit would have changed the outcome. - The first line names the invoker, not the victim.
checkout invoked oom-killer … oom_score_adj=938against a victim withoom_score_adj:1000— two different processes in two different pods. When they match, it is an ordinary container-limit OOM and the container's own limit is the thing to fix. - The
Tasks in … are going to be killed due to memory.oom.group setline is the 1.28 behaviour, printed bymem_cgroup_print_oom_group(). Its absence on a v2 node means somebody setsingleProcessOOMKill: true. oom_score_adjon the victim line tells you its QoS. 1000 is BestEffort, −997 is Guaranteed, anything in between is Burstable and the number decodes to a request:request ≈ (1000 − adj) × nodeCapacity / 1000.
In Prometheus, cAdvisor exports the node-level cgroup with an empty pod label, so the ceiling and the usage are both graphable:
container_memory_working_set_bytes{id="/kubepods.slice"} / container_spec_memory_limit_bytes{id="/kubepods.slice"}
Alert on that ratio, not on per-pod usage. A node can sit at 96% of
kubepods.slice with every single pod under 40% of its own limit,
and every per-pod dashboard you own will look healthy right up to the
kill. The corresponding cgroup file, and the one that will not lie to you
about scrape intervals, is the max counter in
memory.events.
Finally, the two structural questions worth asking of any pod spec before
you ship it. Does every container declare limits.memory? If not
the pod cgroup has no limit at all, and the next ceiling above your
container is the node's. And how many processes does your container run? On
a node whose binding limit is kubepods.slice that number is a
survival parameter, and on
the CPU side of the same tree it is the
parameter that decides how fast you burn a quota. It is the same number
pulling in two directions, which is a fair summary of what the unified
hierarchy did to capacity planning.
A Burstable pod with requests.memory: 256Mi,
limits.memory: 4Gi is OOMKilled while using 900 MiB. The
kernel log shows oom_memcg=/kubepods.slice and
task_memcg=/kubepods.slice/kubepods-burstable.slice/…. What
changes the outcome?
Next, the two limits that live in this same tree and behave nothing like each other: the memory limit that kills the container and the CPU limit that only stops it.