Ah, yes, I misunderstood. And I have seen cases where a direct CPU alert makes sense, but 99 times out of 100 times I see it, it's nothing but trouble. Worse, I tend to see the cpu alert when there are no end to end synthetic alerts, 500 alerts, queue depth alerts, etc.
If your requests are fast and cheap, you can probe frequently relative to your goals, but often that's not really possible (think, long SQL queries, or scheduling a container/pod). There you need several datapoints, or possible fewer augmented with other signals.
Yeah very true.
Talking about long SQL queries, I quite like throwing CPU alerts on database servers. They'll be a low priority alert (ie no out of hours "pagers") so just something that goes into a slack channel. But they're a good indicator of when developers have poorly optimized SQL, or the DB schema is poorly defined (eg missing indexes), or the DB server itself is poorly sized.
This wouldn't be something you'd expect to need in production and definitely not something you'd rely on as a notice of a production outage. But it is an example of one of those 1% occasions where a CPU alert does add value to the overall observability of the application.
But this also ties into your excellent point about how you'd use CPU and other data points to build a picture of what's happening in your application.
Oh, I was thinking about it as the person running SQL as a service. People run queries that go on for days....
idle CPU is often wasted CPU