Scavenging¶

Sweeping marks memory as free within Go's runtime, but it doesn't return that memory to the operating system. Scavenging is the complementary process that releases unused physical pages back to the OS, reducing the application's RSS (Resident Set Size) memory footprint.

This article explores Go's scavenging implementation, explaining when and how memory is returned to the OS.

Why Scavenging Matters¶

Go's memory allocator operates on two levels:

Why separate sweeping from scavenging?

• Sweep: Fast, frequent, maintains free lists for reuse
• Scavenge: Slower, OS syscall overhead, reduces RSS

Sweeping ensures memory is available for future allocations. Scavenging ensures the OS knows memory isn't needed, which is critical for: - Memory-constrained environments: Containers with limits, embedded systems - Multi-tenant systems: Returning memory allows other processes to use it - Cloud costs: Lower RSS = smaller resource bills

Scavenge Target Calculation¶

The scavenger maintains a target high-water mark—the amount of physical memory the runtime aims to retain.

Without GOMEMLIMIT¶

When GOMEMLIMIT is not set, the formula is lenient:

scavenge_target = ((retainExtraPercent + 100) / 100) ×
                  (heapGoal / lastHeapGoal) ×
                  lastHeapInUse

Where:
retainExtraPercent = 10 (default)

Rationale: This allows heap to grow before scavenging kicks in aggressively. If the application is growing (heap increasing), we retain more memory. If shrinking, we scavenge more aggressively.

With GOMEMLIMIT¶

When GOMEMLIMIT is set, the formula becomes aggressive:

scavenge_target = ((100 - reduceExtraPercent) / 100) × memoryLimit

Where:
reduceExtraPercent = 10

Rationale: The runtime must respect the memory limit, so it scavenges down to 90% of GOMEMLIMIT to ensure headroom for fluctuations.

Fragmentation Considerations¶

The scavenger targets physical pages (OS-managed), not logical spans (Go-managed). This distinction is crucial because Go is a non-moving collector:

OS Pages: [Page1][Page2][Page3][Page4][Page5]
Spans:    [Span A: Pages 1-2] [Span B: Page 3] [Span C: Pages 4-5]

After GC:
- Span A: Fully live (keep)
- Span B: Fully dead (return)
- Span C: Fully live (keep)

Free pages list: [Page 3] (1 page)

Next allocation: Needs 2 contiguous pages
Available: Only [Page 3] (not enough!)

Must: Request new page from OS even though Page 3 is free!

Implication: Using heapInUse (actively used spans) underestimates memory needs because fragmentation creates "holes" that can't be used for large allocations.

The scavenger instead estimates based on historical fragmentation patterns, maintaining extra pages to accommodate allocation patterns.

Two Modes of Scavenging¶

Asynchronous Scavenger¶

Similar to sweep, the runtime has a background goroutine that periodically scavenges:

func bgScavenger() {
    for {
        // Calculate how much to scavenge
        target := scavengeTarget()
        current := heapInUse

        if current > target {
            // Need to release memory
            toRelease := current - target
            scavenge(toRelease)
        }

        // Wait before next check
        sleep(scavengeSleepTime)
    }
}

Characteristics: - Runs asynchronously, doesn't block allocation - Opportunistic—scavenges when resources are idle - Best effort—doesn't guarantee RSS will be reduced by next cycle

Synchronous Scavenger¶

When allocation pressure is high and we're approaching GOMEMLIMIT, the runtime scavenges synchronously during allocation:

func allocSpan(npages uintptr) *mspan {
    // Try to allocate from free lists
    if span := mheap_.allocFromCache(npages); span != nil {
        return span
    }

    // No contiguous pages available!
    // We have fragmented free pages but need larger span

    // Before requesting from OS:
    // 1. Scavenge small free pages
    // 2. Request new pages
    // 3. Hope OS coalesces them into contiguous range

    // Example: Need 4 pages, have [1][2][3] fragmented
    scavengeAndCoalesce(1, 2, 3)  // Return to OS

    // Now request 4 new pages from OS
    // OS gives us [4][5][6][7] (contiguous!)
    return mheap_.allocFromOS(npages)
}

When this happens: - Allocation is very fast (high allocation rate) - Approaching GOMEMLIMIT (OOM risk) - Heap is fragmented (many small free pages)

Cost: Allocation pauses briefly while scavenging occurs, but this prevents worse outcome: OOM kill.

Memory Return Mechanism: MADV_DONTNEED¶

Go 1.23 uses MADV_DONTNEED on Linux to return pages to the OS:

func sysUnusedOS(v unsafe.Pointer, n uintptr) {
    // Advise OS that memory range [v, v+n) is not needed
    err := madvise(v, n, MADV_DONTNEED)

    // OS immediately marks pages as not resident
    // Future accesses will cause page faults
    // Pages are zero-filled on fault
}

Characteristics of MADV_DONTNEED:

Why MADV_DONTNEED instead of alternatives?

• MADV_FREE: Lazily returns memory (better for reuse), but Go's design favors deterministic behavior
• MADV_DONTNEED: Immediately returns memory (better for RSS monitoring, co-located containers)

Go's design assumes scavenging is careful enough that memory is returned only when truly not needed, so the immediate cost of MADV_DONTNEED is acceptable.

Interaction with GC Cycle¶

The scavenger updates its target at the end of each GC cycle:

func gcMarkTermination() {
    // ... GC termination work ...

    // Calculate new target based on heap metrics
    newTarget := calculateScavengeTarget(
        heapGoal,
        lastHeapGoal,
        heapInUse,
        gomeMLimit,
    )

    atomic.Store(&scavengeGoal, newTarget)

    // Background scavenger reads this value
}

Key insight: The scavenger reacts to GC cycle outcomes. If GC just ran and heap shrank significantly, scavenger becomes more aggressive. If heap grew, scavenger becomes conservative.

Edge Cases and Tricky Behavior¶

Fragmentation is Unavoidable¶

Because Go is a non-moving collector, fragmentation accumulates over time:

• Short-lived objects: Cause churn, creating internal fragmentation within spans
• Mixed allocation sizes: Small allocations break up large contiguous ranges
• Peak-and-shrink: Heaps that grow then leave "holes" when shrinking

The scavenger compensates by maintaining extra pages, but can't eliminate fragmentation entirely.

RSS Inflation During Growth¶

When heap grows rapidly, RSS temporarily exceeds logical heap size:

Time:  T0     T1     T2     T3     T4
Heap:  100MB  200MB  300MB  400MB  300MB
RSS:   110MB  220MB  350MB  450MB  350MB  ← Lags behind scavenging

The scavenger is lazy—it waits to see if growth continues before returning memory. This causes RSS inflation during growth spurts.

Container OOM with Low Heap Usage¶

A surprising scenario: Application has low heap usage but high RSS, causing container OOM:

Container limit: 512MB
Application heap: 200MB
Application RSS: 500MB  ← Oops!

Cause: Previous peak was 450MB, scavenger hasn't returned memory yet

Solution: Set GOMEMLIMIT to force aggressive scavenging.

Summary¶

Scavenging is the bridge between Go's logical memory management and the OS's physical memory:

• Asynchronous mode: Background goroutine opportunistically reduces RSS
• Synchronous mode: Urgent scavenging during allocation to prevent OOM
• Target calculation: Based on heap goals or GOMEMLIMIT, adjusted for fragmentation
• Mechanism: MADV_DONTNEED returns pages immediately, zeroing on next access
• Fragmentation: Unavoidable in non-moving collectors, compensated by extra retention

Understanding scavenging is essential for: - Running Go in memory-constrained containers - Diagnosing "high RSS, low heap" issues - Tuning GOMEMLIMIT for predictable memory usage

Further Reading¶

• Madvise Documentation
• Go Memory Management
• Allocation and GC (Previous Article)
• STW to Scan Stack (Next Article)