GC tuning checklist#

Go’s GC is guided by the GOGC environment variable (default 100). When you see GC runs every few milliseconds (debug.gc traces), experiment with lowering GOGC (e.g., GOGC=80), which triggers earlier GC but keeps heap smaller. Conversely, if CPU steal is the problem, increase GOGC (e.g., 200) to reduce GC frequency and let the heap grow.

Monitor runtime.ReadMemStats() fields such as NumGC, PauseTotalNs, and HeapInuse. Use Go’s trace tool (go tool trace) for more detail if GC pauses look unexpectedly long.

For services that allocate aggressively (e.g., JSON APIs), consider adding pooling via sync.Pool, reusing bytes.Buffer, or marshalling into preallocated slices. Just be careful: pools must be sized relative to expected concurrency to avoid contention.

Scheduling and GOMAXPROCS#

By default, Go sets GOMAXPROCS = runtime.NumCPU(). For containers under CPU quotas (like Docker), runtime.NumCPU() may exceed the stripped quota, causing more goroutines than allowed cores. Set GOMAXPROCS explicitly (GOMAXPROCS=2 for a 2-core quota) or rely on runtime.GOMAXPROCS(runtime.NumCPU()) early in main when you have visibility into cgroup limits.

Use go test -c to generate a binary and run GODEBUG=schedtrace=1000 to monitor scheduler transitions during load: look for high runq-sz or sysmon overhead indicating goroutines are starving.

CPU hotspots vs locking#

pprof’s web output shows call graphs with edges representing mutual locking or channel waits. If a lock dominates (e.g., sync.Mutex.Lock accounts for most CPU), evaluate:

• Changing to sync.RWMutex when reads dominate.
• Sharding the resource (e.g., per-shard map) to limit lock contention.
• Switching to lock-free data structures when appropriate.

For channel-heavy code, check if goroutines block on <-ch while the producer is slow. Use pprof's goroutine profile to see stacks waiting on channels or mutexes.

Allocations and zero-copy patterns#

Slice growth and string conversions are typical allocation culprits. Look for patterns like:

This allocates every request. Replace with strconv.AppendInt into a reusable buffer or preformatted strings when possible.

When parsing JSON, avoid map[string]any and prefer typed structs. Use encoding/json.Decoder with DisallowUnknownFields to avoid reflection overhead.

Validating changes with load tests#

After you craft a fix (pooling, fewer allocations, locking optimization), rerun the profile under load. Keep the same traffic shape and check:

• CPU percent vs baseline (use /debug/pprof/trace to visualize).
• heap growth between requests (should plateau faster).
• GC pause histograms (look for reduced pause times or counts).

Capture diffs: pprof diff shows the delta between two profiles.

Summary checklist#

1. Capture CPU + heap + goroutine profiles from realistic load.
2. Tune GOGC and GOMAXPROCS relative to cgroup limits.
3. Identify hotspots, locking issues, and allocation churn using pprof commands like top, list, and web.
4. Apply concrete changes (buffer reuse, map sharding, better data structures).
5. Rerun load tests and compare profiles to confirm the improvement.
6. Automate profiled traces periodically if production behavior drifts.

This workflow keeps Go services fast and predictable even under heavy concurrency.