1. Containers aren't a security boundary. Yes they can be used as such, but there is too much overhead (privilege vs unprivileged, figuring out granular capabilities, mount permissions, SELinux/AppArmor/Seccomp, gVisor) and the whole thing is just too brittle.

2. lxd VMs are QEMU-based and very heavy. Great when you need full desktop virtualization, but not for this use case. They also don't work on macOS.

Using Apple virtualization framework (which natively supports lightweight containers) on macOS and a more barebones virtualization stack like Firecracker on Linux is really the sweet spot. You get boot times in milliseconds and the full security of a VM.

qemu has a microvm machine profile, also boots in ms.

There are also tooling on Linux to do containers as microvm's, long before Apple containers were a thing.

And yet Amazon spent a ton of time and money writing Firecracker from scratch for their workloads. Why is that?

Multiple reasons:

1. Firecracker is still a smaller more deliberate surface area 2. qemu didn't have a microvm type at the time. Firecracker was the impetus for it