When zram Makes Kubernetes Nodes Read From Disk
Kubernetes nodes whose image ships with a zram swap device can produce a continuous stream of major page faults under memory pressure. Nodes without zram do not show it.
The cause is two settings interacting on the same node, and the fix is a one-line change to one of them.
Diagnosis
The metric that exposes the behavior is node_vmstat_pgmajfault. A major page fault means that a page was not in memory and had to be fetched from a backing store. Under memory pressure, the rate can become continuously elevated rather than appearing as occasional spikes.
At the same time, disk reads increase substantially. At first this seems like a direct consequence of the major faults: if each fault requires reading a 4 KiB page, the fault rate should roughly correspond to the amount of data being read from disk.
It doesn’t. The fault rate implies significantly more reads than the node is actually performing. Most of the pages are coming from somewhere other than disk.
The reason is zram. Unlike nodes without swap, these nodes have a zram device enabled by the node image. zram keeps swapped pages compressed in RAM rather than writing them to disk. A swap-in therefore does not generate disk I/O, but the kernel still counts it as a major page fault.
That explains the excess faults, but not the disk reads. Swap-out and swap-in traffic can continue at roughly the same rate while zram remains only partially occupied: pages are moving through swap rather than accumulating there.
That leaves three questions:
- Why is the same memory continuously swapped in and out?
- Why doesn’t reclaim settle, despite zram having plenty of capacity?
- Why does this produce more disk IOPS than nodes without zram?
Answering them means looking at how Linux decides what memory to reclaim.
How Linux Reclaims Memory
Memory can be broadly divided into two types that Linux reclaims differently:
- File-backed pages, or page cache: executables, shared libraries, container image layers, and other data that already exists on a filesystem. A clean page can be reclaimed simply by dropping it; if it is needed again, Linux reads it back from the filesystem.
- Anonymous pages: process state such as heap and stack memory that has no backing copy on a filesystem. To reclaim an anonymous page, Linux must first write it to a swap device. Without swap, anonymous memory cannot be reclaimed.
On these nodes, that swap device is zram: a block device that keeps swapped pages compressed in RAM. Swapping out compresses a page into zram; swapping it back in decompresses it. Neither operation reaches disk.
Linux reclaims memory when available memory comes under pressure. When free memory drops below the low watermark, kswapd wakes and reclaims pages until free memory rises above the high watermark, then goes back to sleep. If allocations outrun it, the allocating process can perform reclaim itself and block while doing so.
When reclaim runs, Linux has to choose between dropping file-backed pages and swapping out anonymous pages. vm.swappiness controls that trade-off: higher values favor anonymous reclaim through swap, while lower values favor file-backed pages.
The important detail on a Kubernetes node is that this decision is made per memory cgroup. Kubelet sets the swap limit to zero for the containers it starts, so pod anonymous memory cannot normally be swapped out. For pods, reclaim is therefore effectively limited to file-backed memory.
The node image sets vm.swappiness=200. That is the maximum the kernel accepts, and at the maximum the file share of the reclaim scan is zero. In the node’s system and runtime cgroups, where swap is available, reclaim takes anonymous memory and passes over their file cache.
The result is two different reclaim policies on the same node: pods reclaim file cache, while system and runtime processes preferentially reclaim anonymous memory into zram.
1. Why is the Same Memory Continuously Swapped In and Out?
Only the node’s system and runtime cgroups can swap at all, and what they hold is a small amount of anonymous memory belonging to processes that are running. Reclaim takes those pages because at swappiness 200 they are the only thing it scans there. The processes touch them again shortly afterwards, the kernel faults them back in, and memory is still short, so the next pass selects the same pages again.
The same working set is written out and read back continuously. That is why swap traffic stays high while occupancy stays low: nothing is parked in the device, everything is passing through it.
2. Why Doesn’t Reclaim Settle, Despite zram Having Plenty of Capacity?
Reclaim stops when free memory rises above the high watermark, and here what it frees does not stay free. A page written to zram is not fully released, because the compressed copy still occupies memory, and the pages that come back are consumed again immediately. The pods’ page cache behaves the same way: evicted, then read straight back.
The one pool that would end it is the cold file cache held by containerd and kubelet: image layers and binaries read when a container starts and rarely touched afterwards, so dropping them frees memory that stays free. At swappiness=200 its share of the reclaim scan is zero.
Linux has a rule that would override that split: when a cgroup holds plenty of inactive cache and that cache is not being faulted back in, reclaim takes file pages first and skips anonymous memory, whatever swappiness says. The rule is cleared as soon as the kernel counts file pages being faulted back in, which it reads as the working set changing rather than as cache going cold. The pods’ refaults do exactly that, so the mechanism that would have trimmed the cold cache is disabled by the thrashing it exists to prevent.
Reclaim therefore keeps freeing memory that is consumed again immediately, so kswapd finishes a pass, fails to balance, and is woken again. Capacity was never the constraint.
3. Why Does This Produce More Disk IOPS Than Nodes Without zram?
The swap traffic itself never reaches disk. The disk reads come from the page cache that reclaim repeatedly evicts.
For pods, page cache is the only reclaimable memory. When reclaim drops a page containing an executable or a shared library, it must be read back from disk when the process touches it again. Each refault is typically a small read, producing high IOPS with little throughput.
The difference is what happens when the node has no swap device. Anonymous memory is unreclaimable everywhere, so reclaim is forced to take file-backed memory, including the large cold cache held by containerd and kubelet. Those pages are rarely touched again, so the memory they free stays free, reclaim reaches its target, and stops.
With zram and swappiness=200, reclaim can avoid that cold cache by swapping anonymous memory instead. When that fails to make lasting progress, it keeps reaching into the pods’ page cache, producing the refaults that drive disk IOPS.
The Fix
The fix is to lower vm.swappiness from 200 to 60.
At 60, the node’s cold file cache is considered normally during reclaim. Most of those pages are never read again, so the memory they free stays free and reclaim reaches the high watermark and stops.
Nothing about the pods changes: their anonymous memory remains unreclaimable and their page cache remains reclaimable. The difference is that reclaim no longer repeatedly evicts their working set, removing most of the refaults that were driving the disk IOPS.
Results
The change produced a clear break in behavior. The same metrics, queries, and resolution show:
- Major fault rate fell by four orders of magnitude, at both the median and p99.
- Read IOPS fell 33× at the median and 100× at p99.
- Peak read IOPS fell from above the provisioned limit to roughly 5% of it.
- Pages were swapped out in fewer than 10% of samples.
The fault rate falls further than the IOPS because most of those faults were zram swap-ins that never reached a disk; only the smaller share coming from file refaults produced read I/O.
Other application and node metrics remained healthy and within their normal ranges, indicating that the change affected memory reclaim rather than the workload.
Lowering vm.swappiness stopped the swap churn and cut the disk I/O it was producing by one to two orders of magnitude.