On-call problems are mostly just a result of Kubernetes' "cattle not pets" brainrot ideology where people throw local simulate-ability out the window because the Rube-Goldberg web of pods and operators and cluster state gets too complex to simulate locally or keep in your head (esp. if code is located in 50+ microservice repos).
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
>Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen".
Uh, I can literally see thousands of crashes happening all the time. You think there isn't a .01% chance of hardware going bad or hitting kernel bugs?
It's not that crashes don't happen, it's just that they don't "randomly" happen. The error codes from bad hardware are predictable (against kernel bugs... well I use an "allnoconfig" kernel, which disables like 95% of the kernel). Bad hardware, bad network, bad RAM, all of these are simulateable in QEMU or Linux (tc netem / blkdebug).
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
i'm not questioning your setup, I'm saying that if you need a 10,000 machines to run a system you're going to have kernel bugs and hardware bugs and you have no choice to treat machines like cattle.