My company just did this. Still using AWS S3 but everything else is on-prem with kubernetes.

That sounds insanely expensive. AWS S3 only makes sense if your workload is also in AWS. Using AWS S3 while having your compute elsewhere is a recipe for an insane egress bill.

Dependa how you use it. If it's only for backups then it's fine.

How much of your public facing workloads rely on S3 if I can ask?

There's plenty of OSS drop-in replacements for S3.