Memory Limits, OOMKill and Exit Code 137

A CPU limit slows your container down. A memory limit kills it. There is no middle setting, because there is nothing to slow down: the kernel cannot give a process nine tenths of a page. When the container's memory charge reaches the limit and the kernel cannot free enough to continue, it picks a process inside that container and sends it SIGKILL. No exception, no stack trace, no chance to shed load. Exit code 137, and the container starts again from zero on the same node.

The Kubernetes documentation says this, in one sentence people read past: memory limits "are enforced by the kernel with out of memory (OOM) kills" and are "enforced reactively", while CPU limits are "a hard limit the kernel enforces". Those two clauses describe opposite mechanisms. A CPU limit becomes a CFS quota — a budget that refills every 100 milliseconds, and spending it early means being stopped until it refills. A memory limit becomes one number in one file, memory.max, and crossing it is terminal.

Below is a container running for five minutes against a 1 GiB limit. The application holds 320 MiB of live data — the objects it genuinely needs at any moment — and the JVM is started with -Xmx960m, which looks like a sensible 64 MiB of margin. Watch it die anyway, then drag limits.memory and see which failures the limit fixes and which it only relocates.

At the defaults the container is killed 51 seconds in, and again, and again: four restarts in five minutes, with the gaps growing 10, 20, 40, 80 seconds. The heap never overflowed. The JVM was given 960 MiB and its committed heap peaked at 833, so -Xmx was never reached and no OutOfMemoryError was thrown. What crossed the limit was the sum: 833 MiB of heap, plus 48 thread stacks at a megabyte each, plus 72 MiB of metaspace and code cache, plus 46 MiB of collector bookkeeping, plus 44 MiB of the VM's own native memory. That is 1043 MiB against a limit of 1024. The ledger prints the arithmetic. Nothing in -Xmx covers the lines below the first one.

The request is not a reservation

Drag resources.requests.memory across its entire range. Not one number in the simulation moves except the QoS readout. That is not a simplification — it is the whole behaviour of the field on a default cluster.

The kubelet writes exactly one memory number into the container's cgroup: the limit, as memory.max on cgroup v2 or memory.limit_in_bytes on v1. The request is written nowhere. There is no kernel file that says "this container is entitled to 512 MiB", and no mechanism that keeps those 512 MiB available for you once the node fills up. The request does three things, all of them outside the container:

• It is what the scheduler adds up. A node is full when the sum of requests reaches allocatable, so the request decides which node you land on and nothing else about placement.
• It sets oom_score_adj, which biases the node-level OOM killer, the one that runs when the machine itself is out of memory. The kubelet's formula is in pkg/kubelet/qos/policy.go: −997 if the pod is Guaranteed, 1000 if BestEffort, and otherwise 1000 − (1000 × containerMemoryRequest) / nodeMemoryCapacity, floored at 3. A 512 MiB request on a 16 GiB node gives 969.
• It decides the QoS class together with the limit, which is what the eviction ranking uses. That is the subject of the next lesson, and it is a node-level mechanism, not a container-level one.

None of those help the container in the simulation, because the kill in the simulation is a cgroup OOM, not a node OOM. The kernel's oom_badness() makes the distinction explicit. For a memory cgroup OOM it sets totalpages to mem_cgroup_get_max() — your own limit — rather than the machine's memory, and every process in your container carries the same oom_score_adj. An identical constant added to every candidate changes no ranking. Your Guaranteed pod's −997 does not protect it from itself.

There is one way to make the request mean something to the kernel, and whether you already have it depends on your version. The MemoryQoS feature gate was alpha and off by default from v1.22 through v1.36; in pkg/features/kube_features.go on release-1.37 it gains a second entry, {Version: 1.37, Default: true, PreRelease: Beta}, so from v1.37 it is on unless you turn it off. Check yours rather than assuming — the behaviour below is the same either way, but which of your clusters is doing it is not. With the gate enabled the kubelet sets memory.high for Burstable containers at requests + 0.9 × (limits − requests), and with memoryReservationPolicy: TieredReservation it also sets memory.min to the request for Guaranteed pods and memory.low for Burstable ones. Tick MemoryQoS in the simulation at the defaults: the container is still killed, at the same second. memory.high throttles allocation and applies reclaim pressure; the kernel documentation says plainly that going over it "never invokes the OOM killer". It buys latency, not survival. It helps a container with reclaimable pages and does nothing for one whose growth is anonymous memory it is still using.

Your runtime does not know how big its container is, unless it was told

Select JVM, container default and leave everything else alone. The kills stop. The log fills instead with java.lang.OutOfMemoryError: Java heap space, and the container keeps running the whole five minutes. You changed nothing about the container, the limit or the data. You changed which of two completely different deaths you get.

Since JDK 10, backported to 8u191, the JVM reads the cgroup limit under -XX:+UseContainerSupport, which is on by default, and sizes the heap as a percentage of it. That percentage is MaxRAMPercentage, and its default in HotSpot is 25.0. So a 1 GiB limit means a 256 MiB maximum heap, and a live set of 320 MiB simply does not fit. You get a Java exception with a stack trace, thrown by a process that is alive and using a third of its container.

Now drag the limit to 2048 MiB. Everything is fine: the heap ceiling moves to 512 MiB, the live set fits, peak charge is 696 MiB. Note what just happened. The limit is an input to your program's memory use, not only a bound on it. Keep dragging — at 4096 MiB the same 320 MiB of data sits in a container whose peak charge is 1229 MiB, because the collector has been told it may let the heap reach a gigabyte before collecting. Raising a limit to stop OOM kills works, and it silently raises what every replica actually consumes.

Switch back to JVM, -Xmx set by hand, which is what most teams do precisely to avoid that. The default here is -Xmx960m against a 1024 MiB limit: 64 MiB of margin, which feels generous. The ledger shows where it goes.

• 48 MiB of thread stacks. One megabyte per thread is the -Xss default on 64-bit Linux, and 48 threads is a small servlet pool. Set the limit to 1536 and drag threads / workers from 48 to 256: headroom falls from 359 MiB to 99. Of the 260 MiB that vanished, 208 is stack and the other 52 is per-thread native structures. Nobody sizing that pool thought they were spending a quarter of a gigabyte.
• 72 MiB of metaspace and code cache. Class metadata and JIT-compiled code live outside the heap and are not bounded by -Xmx at all. Metaspace is unbounded by default; it is capped only if you set -XX:MaxMetaspaceSize.
• 46 MiB of collector bookkeeping at that heap size — card tables, remembered sets, mark bitmaps — which grows with the heap you configured, not the heap you use.
• Native allocation outside all of the above: the VM's own C heap, glibc malloc arenas, direct byte buffers from any NIO or Netty code path. Direct buffers are capped by -XX:MaxDirectMemorySize, whose default is the maximum heap size all over again.

That is the arithmetic behind the most-upvoted version of this question on Stack Overflow, "Java using much more memory than heap size", at 115,000 views. The rule that falls out of it: -Xmx is not a container budget. It is one line of a container budget, and on a service with a large thread pool it can be well under half of it. The same accounting problem shows up one layer up in a Spark executor, where the overhead fraction exists exactly because someone has to pay for these lines.

Select Go and drag threads / workers from 2 to 256. The total barely moves — a goroutine starts with an 8 KiB stack, so 256 of them cost two megabytes where 256 Java threads cost 256. Go's exposure is the other direction: with the default GOGC=100 the collector runs when the heap reaches twice the live set, so a container holding 320 MiB of live data needs 640 MiB of heap plus overhead before a collection is even attempted. The Go runtime does not read your cgroup limit for this.

GOMEMLIMIT exists to tell it, and the slider labelled -Xmx / GOMEMLIMIT is that variable while Go is selected. Push it to its maximum, 4096, so that it is far above the container limit and therefore not binding — that is the same as not setting it at all — and raise live data to 600 MiB. The container is killed at 0:43 and four times over. Now bring GOMEMLIMIT back to 960: no kills, 56 MiB of headroom, same data, same limit. The collector simply ran more often.

Two cautions on that. The garbage collector guide is explicit that the limit is soft: "the Go runtime makes no guarantees that it will maintain this memory limit under all circumstances; it only promises some reasonable amount of effort." And its own recommendation is to leave 5–10% of headroom below the container limit, because GOMEMLIMIT governs memory the runtime knows about and there is always some it does not.

Note the asymmetry with CPU. Go 1.25 made GOMAXPROCS default to the cgroup CPU limit, and the JVM has read the CPU quota since JDK 10. For memory, only the JVM reads the limit, only for the heap, and only to a quarter of it.

Who dies, and what survives

Select Python, gunicorn, set threads / workers to 96 — which is 16 gunicorn workers — and untick memory.oom.group. The kernel kills a process 26 times over the five minutes and RESTARTS stays at zero. This is the failure that wastes the most time, because every dashboard a team normally looks at says the pod is healthy.

The kernel's OOM killer picks a process, not a container. Its ranking function oom_badness() is resident pages plus page tables plus swap, with oom_score_adj scaled by totalpages/1000 added on. In a gunicorn container the master process holds almost nothing and each worker holds its own copy of its share of the data, so the largest worker always outranks the master. The kernel kills a worker. The master notices a dead child and forks a replacement. PID 1 never died, so the container never exited, so the kubelet has nothing to report — the pod's restart count does not move and there is no OOMKilled anywhere in kubectl describe. The only evidence is a counter in memory.events and a line in the node's kernel log. This is the practical content of Kubernetes issue 69676, "Log something about OOMKilled containers", open since 2018 with 123 reactions.

Tick memory.oom.group back on and the same run reports one restart and 17 kills — one group kill, in which all sixteen workers and the master died in the same second, and the container therefore exited and was reported. memory.oom.group is a cgroup v2 file whose kernel documentation says the cgroup is "treated as an indivisible workload by the OOM killer" and that all its tasks "are killed together or not at all". Kubernetes 1.32 made the kubelet set it on container cgroups by default; the escape hatch is the kubelet option singleProcessOOMKill: true, which is the only permitted value on cgroup v1 because v1 has no group kill. So the answer to "why did my multi-process container get partially killed" is very often "because the node is still on cgroup v1, or on a kubelet older than 1.32".

When PID 1 does die, watch what the kubelet does, because it is the opposite of what the situation seems to call for. The container is restarted in place: same pod, same node, same cgroup, same limit, empty heap. Nothing reschedules. Nothing consults the node's free memory, which may be enormous — this is the "OOMKilled with apparently plenty of memory left in node" question, at 39,000 views, and the answer is that the limit is enforced per-cgroup and the node was never involved.

So the loop is deterministic. The restart resets the warm-up, the application reloads the same working set, and it crosses the same threshold at roughly the same age. The gaps between attempts are the kubelet's crash loop back-off, which starts at 10 seconds and doubles to a ceiling of 300; the constants are initialCrashLoopBackOff = 10s and MaxCrashLoopBackOff in pkg/kubelet/kubelet.go, and the counter is forgotten after 600 seconds without a restart. An alpha gate, ReduceDefaultCrashLoopBackOffDecay, lowers those to 1 second and 60. Count the restarts in the simulation at the defaults: four in five minutes, killed at 0:51, 1:52, 2:58 and 4:29, restarted at 1:01, 2:12 and 3:38, and the fifth attempt would not begin until 80 seconds after the last one. What looks like a system slowly recovering is a system taking longer and longer to fail the same way.

Contrast that with eviction, which is the other way a pod loses to memory and behaves in the way people expect an OOMKill to. An eviction is the kubelet's decision, not the kernel's; it fires when the node is short of memory, it terminates every container in the pod, it sets the pod's phase to Failed, and the replacement pod is usually created somewhere else. Two mechanisms, two verdicts, and the word "OOM" is used for both. If kubectl describe pod says OOMKilled with exit code 137, that is the kernel and your limit. If it says Evicted with a message about memory.available, that is the kubelet and your node.

Where each fix stops working

Raise the limit. At the defaults, 1216 MiB is enough and the kills stop. This works and is usually right, and it has two boundaries. The first is the one above: if the runtime sizes itself from the limit, the limit is not headroom, and raising it raises usage with it. The second is that you have moved the failure from your container to the node — memory you are permitted to use but did not request is memory that is not reserved for you anywhere, so under node pressure you become an eviction candidate the moment you exceed your request.

Set requests equal to limits. Drag requests up to 1024 with the limit at 1024: the QoS readout flips to Guaranteed and oom_score_adj to −997. The kill happens at exactly the same second. Guaranteed protects you from other pods and from the node; it does not protect you from your own arithmetic, because the constant is applied equally to every process in your own cgroup and cancels out of the ranking.

Read files and blame the cache. Leave the limit at 1024, select JVM, container default, and set file reads to 40 MiB/s. From 0:20 onward the charge sits at exactly 100% of the limit, the bars stay amber for the rest of the run, and nothing is ever killed — the log shows the kernel reclaiming a few tens of megabytes of page cache every second and the allocation succeeding every time. Page cache is charged to your cgroup — that is why memory.current and every "container memory usage" graph run hot on any container that touches a file — but clean page cache is reclaimable, so the kernel drops it and the allocation proceeds. Your dashboard shows 100% of the limit and your container is fine.

That is also why an alert on memory usage is close to useless and an alert on the kill is exact. It is worth knowing where the exception is: page cache that cannot be dropped — dirty pages waiting on a slow disk, pages in a tmpfs or an emptyDir: {medium: Memory} volume, or a file the container has locked — is charged like anonymous memory and will kill you. An emptyDir memory volume in particular is a RAM disk billed to your container's limit, and it is the most common surprising line on this list.

The node-level version of the same confusion is Kubernetes issue 43916, "kubelet counts active page cache against memory.available", the most-reacted memory issue in the repository at 223 reactions and 144 comments, open since 2017. The eviction signal subtracts inactive_file from the node's usage but not active_file, so file pages that are recently touched and perfectly droppable are counted as used, and the kubelet can start evicting pods to reclaim memory the kernel would have handed back for free.

Give the runtime the right number. This is the fix that addresses the mechanism. Put the limit back to 1024, keep -Xmx set by hand, and drag -Xmx from 960 down. At 832 it still dies. At 768 the container survives all five minutes with 50 MiB of headroom, and at 704 with 117. You did not give the container more memory. You stopped promising the heap memory that the stacks, the metaspace and the collector had already spent.

Its boundary is visible three drags away. Keep pulling -Xmx down, to 320: now the heap is smaller than the live set and you are back to java.lang.OutOfMemoryError. Between "the container dies with no stack trace" and "the application dies with one" there is a window, and the width of that window is the limit minus everything that is not heap. If that window is negative — a large thread pool against a small limit — no value of -Xmx works, and the honest answers are fewer threads, a smaller -Xss, or a bigger limit.

Checking it on a real pod

Start with the verdict, because it is unambiguous and takes one command:

kubectl describe pod api-7d9f -n prod | grep -A5 "Last State"
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 13 Aug 2026 09:14:02 +0000
      Finished:     Thu, 13 Aug 2026 09:14:53 +0000

Reason: OOMKilled with exit code 137 means the kernel SIGKILLed a process in that container's cgroup. Subtract Started from Finished: if it is tens of seconds every time, you have a working set that does not fit and no amount of restarting will change it. If it is hours or days, you have a leak, and the limit is a detector rather than a cause.

Then get the numbers the kubelet does not report, from inside the container on cgroup v2:

• cat /sys/fs/cgroup/memory.max → the limit in bytes, or max. On v1 it is /sys/fs/cgroup/memory/memory.limit_in_bytes.
• cat /sys/fs/cgroup/memory.current → what is charged right now, page cache included.
• cat /sys/fs/cgroup/memory.stat → the breakdown that makes the ledger above real. anon is the part that can kill you, file is mostly not, slab and kernel_stack are charged to you as well and appear in no language runtime's accounting.
• cat /sys/fs/cgroup/memory.events → oom counts the times an allocation was about to fail; oom_kill counts processes actually killed; oom_group_kill counts group kills. A rising oom_kill with a flat pod restart count is the silent-worker-death case exactly.
• cat /proc/1/oom_score_adj → confirms which QoS class the kubelet actually assigned, without trusting the YAML you think you applied.

In Prometheus the useful signal is the kill, not the usage:

increase(kube_pod_container_status_restarts_total[1h]) joined against kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}

For sizing, the number to compare against the limit is container_memory_working_set_bytes, which is cAdvisor's usage minus inactive_file — that is, with the obviously-droppable page cache removed. Do not use container_memory_usage_bytes, which includes all page cache and will tell you that every container reading a file is nearly full. Take the maximum, not the average:

max_over_time(container_memory_working_set_bytes{container!=""}[7d]) / on(pod,container) kube_pod_container_resource_limits{resource="memory"}

A ratio consistently above roughly 0.9 is a pod that will be killed by any slightly unusual day. A ratio below 0.3 is a pod paying for memory it never touches — and if it is Guaranteed, paying for it out of the scheduler's allocatable pool as well.

Finally, read the node. The kernel writes a full report on every cgroup OOM and it names the victim, which is the one thing kubectl will not tell you:

Memory cgroup out of memory: Killed process 4213 (java) total-vm:4192384kB,
  anon-rss:1180416kB, file-rss:23044kB, shmem-rss:0kB, UID:1000
  pgtables:2604kB oom_score_adj:969
memory: usage 1048576kB, limit 1048576kB, failcnt 812

anon-rss is the process's own anonymous memory, and comparing it to the limit on the next line tells you immediately whether one process was the whole story or whether the container died of the sum of its parts.

Next: what Guaranteed actually guarantees, and why the class that decides eviction order is not an input to the eviction decision — QoS classes.