> GC-LLM: Garbage Collection for LLM Inference Memory

Systems & Infrastructure 专家 2026-04-01 06:41 2026-04-01
#LLM inference #memory management #garbage collection #KV cache #PagedAttention #real-time systems

Applying GC principles (reference counting, generational collection, memory pools) to GPU memory in Transformer inference. 37-53% memory reduction vs PyTorch, formal WCET bounds (2.5ms young-gen GC), and EDF schedulability guarantees for edge deployment.

// TABLE_OF_CONTENTS

GC-LLM: Garbage Collection for Inference

The Exact GC Mapping

Classical GC LLM Inference
Young generation Activations (per-op lifetime)
Middle generation KV cache (per-request)
Old generation Model weights (permanent)

Memory Reduction

Scenario Standard PyTorch GC-LLM Reduction
Long context (8K) 8.2 GB 5.1 GB 37.8%
High concurrency (32 req) 12.5 GB 7.8 GB 37.6%
Shared prefix 6.8 GB 3.2 GB 52.9%

Shared prefix uses copy-on-write PagedAttention with reference counting — architecturally impossible without per-tensor lifecycle tracking.

Policy Benchmark

Policy Pool Hit Rate Throughput Best For
IMMEDIATE 99.8% 45K t/s Churn workloads
NAIVE 0% 421K t/s Max throughput (no protection)
ADAPTIVE Dynamic Dynamic Production (switches at 60%/85% pressure)

Real-Time Guarantees

Theorem 7: Young-gen WCET ≤ 200 tensors × 10μs + 0.5ms = 2.5ms — well within 20-100ms inter-token budgets.

Theorem 8: EDF schedulability for Jetson Orin Nano running Qwen2.5-1.5B: N_max = 3 concurrent requests with deterministic latency.

Theory vs Measurement

Metric Predicted Measured
GC speedup 10-15x 12.28x
Prefix cache hit >1000x >6000x
Compression ratio 2-3x 2.24x
< BACK_TO_NOTES()