I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this.
If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it.
But EC2 is not like your home server. There are some nontrivial considerations and requirements to offer nested virtualization at cloud scale:
1. Ensuring virtualized networking (VPC) works with nested VMs as well as with the primary VM
2. Making sure the environment (VMM etc) is sufficiently hardened to meet AWS's incredibly stringent security standards so that nesting doesn't pose unintended threats or weaken EC2's isolation properties. EC2 doesn't use libvirt or an off-the-shelf KVM. See https://youtu.be/cD1mNQ9YbeA?si=hcaZaV2W_hcEIn9L&t=1095 and https://youtu.be/hqqKi3E-oG8?si=liAfollyupYicc_L&t=501
3. Ensuring performance and reliability meets customer standards
4. Building a rock-solid control plane around it all
It's not a trivial matter of flipping a bit.
There's no better way to get good information that is right, than to say something that is misguided and/or wrong.
Thanks for the well-reasoned response.
All that sounds like it would better be a contribution to KVM from the get go rather than invent stuff that eventually showed up in KVM anyway
I always enjoy the color you add to these conversations. Thanks!
I always enjoy the color you add to these conversations in your newsletter.
It's provided many a chuckle.
Thanks!
Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?
Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)
I guess I should have peeked at the source code when I was there…
No need, at least when I was there when the day was still one, before the pandemic. And well, Firecracker is open source.
A few of the best technical presentations that I've watched were at a pre-SKO event. Nitro, Graviton and Firecracker.
Great engineering pieces, the three of them.
Since I don't work for AWS I'm allowed to say that at the scale of millions/billions of microVMs you're better off running them on bare metal instances to avoid the overhead of nested virtualization.
I used to work for AWS and I’m allowed to say the same thing. ;-)
Nitro is very interesting stuff