> Continuously capturing low-overhead performance profiles in production
It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
> Continuously capturing low-overhead performance profiles in production
It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.
Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens inside GPUs now!
[1] https://github.com/open-telemetry/opentelemetry-ebpf-profile...
[2] https://zymtrace.com/article/zero-friction-gpu-profiler/
> For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.
This is not an accurate summary of what they developed.
Using .eh_frame to unwind stacks without frame pointers is not novel - it is exactly what it is for and perf has had an implementation doing it since ~2010. The problem is the kernel support for this was repeatedly rejected so the kernel samples kilobytes of stack and then userspace does the unwind
What they developed is an implementation of unwinding from an eBPF program running in the kernel using data from eh_frame.
True, I should have been more specific about the context:
Their invention is about pushing down the .eh_frame walking to kernel space, so you don't need to ship large chunks of stack memory to userspace for post-processing. And eBPF code is the executor of that "pushed down" .eh_frame walking.
The GitHub page mentions a patent on this too: https://patents.google.com/patent/US11604718B1/en
OTel Profiling SIG maintainer here: I understand your concern, but we’ve tried our best to make things efficient across the protocol and all involved components.
Please let us know if you find any issues with what we are shipping right now.
Anything to actually add?