Presentation outlines low-overhead instrumentation using atomic primitives, per-CPU sharding, lock-free histograms, and eBPF

Engineers face a trade-off between system visibility and performance overhead when adding metrics. The presentation notes that insufficient instrumentation hides performance problems, while excessive instrumentation degrades throughput. Brian Martin frames the challenge as needing 'fearless' instrumentation that does not sacrifice speed.

He outlines four techniques drawn from his work at IOP Systems: atomic primitives to avoid locks, per-CPU sharding to distribute work, lock-free histograms for efficient aggregation, and eBPF to probe kernel behavior without modifying it. Each technique aims to reduce the cost of collecting metrics in hot paths. The talk positions these techniques as ways to maintain visibility without sacrificing performance.

Adoption requires integrating the metriken library, which uses a link-time hack to discover metrics, and optionally deploying the Rezolus agent for eBPF-based telemetry. The approach assumes a Rust-based toolchain and familiarity with low-level synchronization primitives. This integration path is intended for performance-critical paths where traditional metrics would be too costly.

The talk does not detail the limits of these methods; it does not specify conditions under which overhead re-emerges or where accuracy degrades. Consequently, engineers must evaluate the techniques in their own environments to determine applicability. The presentation focuses on strategies drawn from work at IOP Systems.