Skip to content

RSS vs Heap Target: Why Container Memory Exceeds GC Goal

1. Problem Encountered

  • Context: A Go service runs in a container with an 8 GiB memory limit. A dynamic GC tuner is active, configuring the target heap size to 70% of the container limit.
  • Observation: The container's Resident Set Size (RSS) consistently remains at approximately 79.5%, exceeding the 70% target by 9.5%.
  • Objective: Determine whether the excess memory is idle/reusable (held by the Go runtime) or active/unreclaimable (non-heap overhead or memory leak), and assess the Out-Of-Memory (OOM) risk during QPS spikes.

2. Analysis Approach

  • Invalidation of Heap Profiling: inuse_space pprof is insufficient. It measures only live heap objects (approximately 55 MB) and excludes idle memory retained by the runtime, as well as non-heap operational overhead.
  • Runtime Scavenger Mechanism: Without an explicit GOMEMLIMIT, the Go runtime Scavenger operates with a hardcoded retainExtraPercent = 10 rule. It retains an extra 10% of memory above the target heap goal to serve as a buffer against frequent OS system calls.
  • Verification Strategy: Utilize go_memstats_* metrics to compute the exact byte-level memory ledger and validate the physical allocation state.

3. Data Verification

Key metrics extracted from the Prometheus endpoint:

go_memstats_heap_idle_bytes 2.24239616e+09
go_memstats_heap_released_bytes 1.7104896e+07
go_memstats_heap_sys_bytes 6.48282112e+09
go_memstats_next_gc_bytes 6.012246789e+09
go_memstats_sys_bytes 6.86632924e+09

Calculations are based on the 8 GiB (8,589,934,592 bytes) container limit.

Target Goal Accuracy

go_memstats_next_gc_bytes (6.01 GB) / 8 GiB = 70.00%

The 70% heap target is strictly enforced.

RSS Ledger Reconciliation

Total memory held by Go = go_memstats_sys_bytes (6.86 GB) - go_memstats_heap_released_bytes (0.017 GB) = 6.84 GB
Physical RSS Watermark = 6.84 GB / 8 GiB = 79.73%

This aligns with the observed 79.5% RSS.

Memory Segmentation

  • Idle Memory (Reusable buffer): heap_idle_bytes (2.24 GB) - released (0.017 GB) = 2.22 GB

  • This confirms the active retention of the 10% Scavenger buffer.

  • Non-Heap Overhead (Active usage): sys (6.86 GB) - heap_sys (6.48 GB) = 384 MB

  • Goroutine stacks: 180 MB
  • MSpan management: 87 MB
  • GC metadata: 79 MB

4. Conclusion

The 9.5% RSS deviation is the intended behavior of the Go runtime. It consists of a 2.22 GB idle memory buffer and 384 MB of required non-heap overhead. There is no memory leak.

During QPS spikes, the Go runtime will allocate from the 2.22 GB idle buffer before requesting new pages from the OS. The current RSS level does not pose an immediate OOM risk.