Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload.
You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything, when you get to the 10-30% range.
The only way you get to “majority” is if your trace implementation is slower than time travel debugging and provides less information, but then why choose something worse in every dimension.
I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace spans, which are ~millions of times more expensive than function calls. Even eBPF uprobe/uretprobe is 100x more expensive than XRay, at a minimum. HotSpot's JFR is like a miracle compared to what people suffer through to diagnose Rust+Tokio.
1: https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?
Huh, it seems xray puts blank trampolines all over your binary? That sounds pretty nifty but I would expect it to be pretty language agnostic, ish? Adding support should be doable for Rust as well, right? Anyways, pretty nifty.
I am by no means an expert, but I've recently improved performance for some code and used tracy. They have rust bindings as well. It's pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations/aquisition tool.
Edit: apparently rust already supports xray natively on the nightly.
Not even an analog: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/...
It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.
afaik, C/C++ have the same problem.
The difference is nobody in the C++ community believes that a dominant asynchronous executor library exists, and there is not a pervasive belief that it would be helpful.
The "C++ community", if it even exists, barely believes in sharing code let alone any library being "dominant." They'd have to agree on a build system first, after all.
But honestly that's a mischaracterization of the situation in Rust. Tokio is popular for networked service backends. If that's the wheelhouse you're in then yea it might look "dominant."
You don't need a build system to share code.
You can share with header files and respective (shared) object files regardless of the build system you're using. Likewise you could just share the source. None of this needs a build system.
I was just being a bit sardonic because the C++ ecosystem is so fragmented that something like tokio couldn't really exist. It would be one of three executors in boost, abseil, or folly, and you would never see the kind of downstream ecosystem build on top of them because C++ shops are allergic to external dependencies.
Any organisation that cares about security should be allergic to external dependencies, that is why companies like Nexus and JFrog exist, with companies paying to keep internal repos infrastructure in shape.
One just doesn't install willy nilly from the Internet into the CI/CD pipeline.
Well, they do, and then spend a few late nights when there is a bunch of CVE to fix.
Using an artifactory instance as the origin is not functionally different from installing something 'willy nilly' from the internet. It addresses a narrow range of threats while predominantly being more reliable and faster than public repositories.
It doesn't fix the actual problems with C++, which is that it's significantly more difficult to get and use external dependencies because of the compilation and linkage model of C++ libraries.
If C++ were as easy to build and link as modern programming languages you'd see the same kinds of tools as cargo, and the same kinds of ecosystem evolution as rust, like tokio. But you don't, because C++ code sucks to build, package, distribute, update, and reuse.
(I'm aware/have used conan/meson/vcpkg/etc - doesn't change my opinion).
It surely does, because in most companies that care about security it isn't a mirror, rather the only third party packages that developers are allowed to use beyond the standard library.
Additionally, making new packages available for consumption requires approval from IT and possibly legal, before they become available for consumption.
What is hard is people educated in scripting languages not wanting to learn about toolchains.
The moment Rust depends on other programming languages, we get a build.rs spaghetti file, depending on the knowledge of those writing it, or people throwing away Cargo altogether, and replacing it with Bazel, buck2 and co.
Of course we do, it is done via OS package managers, commercial libraries and SDKs.
More recently, via vcpkg and conan.
I'm not sure how other people are using LLMs for instrumentation, but IMO the layer you want running in prod is very different from what you want running for a one-off test. E.g., I have some code floating around which burns a pinned core on increasing a counter, with a little wrapper code around grabbing real timestamps at the beginning and end of a session and converting between the two units of time. It's helpful when microbenchmarking a very small unit of code as it actually behaves in a larger program (not perfect -- obviously tweaks the icache and pipeline behavior at a minimum -- but no measurement has zero tradeoffs, and you're always choosing which set of tradeoffs you prefer). An LLM can quickly instrument the call path I care about while I study this or that intervention. The ability to bang out a large amount of throwaway code is delightful.
One legitimately great thing about LLMs is that it makes it feasible to add these kind of tracing instrumentations temporarily for profiling and then throw them away so they never reach source control let alone production.
I can get an LLM to trace my incomprehensible Tokio application which was also written by an LLM, which is why I don't understand its behavior. Truly the future we were promised.
I guess you should adopt RFCs or ADRs to help clarify the Tokio application, like this https://github.com/brunoarueira/thoth-mesh/tree/main/docs/ad.... This project is vibe coded, but I had put the effort to create issues, roadmap and ADRs, so later I can understand the project without going deep on the code!
Reaching source control is fine as long as there is a compile time flag to disable the whole thing, which tokio-tracing does
Was this in a specific application? I wouldn't necessarily expect that histogram to be particularly bad for most applications.
Reading the clock every time you jump into a closure is in fact incredibly wasteful, and is exacerbated by chopping work up into tiny chunks for questionable reasons.
Just curious, why? Is this true even if you did something like a per-CPU histogram that uses atomic ops to increment?
If you have a per-cpu metric there would not be a reason to use atomic instructions to mutate it.
In general your unpinned userspace threads will hit the same CPU 99.99% of the time, but not 100%.
Sure. You get the pointer, you lock the mutex, 99.99% of the time that is uncontended, then you set all the metrics and release it.
Taking the mutex uses (uncontended) atomic ops.
If you’re not using eBPF to trace your app you’re doing it wrong.
Doesn’t that only work on Linux? And then only for things that make syscalls? Presumably people have to trace other slow paths sometime.
The low cost of eBPF tracing is another myth.
1) Its no myth, but you can definitely foot-bullet into doing it wrong, and 2) it's a far better path to take than in-app telemetry.