How much speed are you actually sacrificing? I'd assume modern hardware can decrypt/encrypt TLS very fast.

Side note: I remember working for a company in early 2000's that had Sparc III 1U servers we had to buy crypto accelerators to do this until I had to convince them to use dell/linux.