Our internally provided certs of various CAs have a TTL of 72 hours and should be renewed every 48 hours.
It's been a huge pain as we have encountered a ton of bugs and missing features in libraries and applications to reload certs like this. And we have some really ugly workarounds in place, because some applications place a "reload a consul client" on the same level of "reload all config, including opening new sockets, adjusting socket parameters, doing TCP connection handover" - all to rebuild a stateless client throwing a few parameters at a standard http client. But oh well.
But I refuse to back down. Reload your certs and your secrets. If we encounter a situation in which we have to mass-revoke and mass-reissue internal certs, it'll be easy for those who do. I don't have time for everyone else.
> Our internally provided certs of various CAs have a TTL of 72 hours and should be renewed every 48 hours.
Do you promise to come back and tell us the story about when someone went on vacation and the certs issued on a Thursday didn't renew over the weekend and come Monday everything broke and no one could authenticate or get into the building?
At least that sounds like it would be a more interesting story than the one where the person who quit a year ago didn't document all the places they manually installed the 2-year certificate.
I will. We've been betting Postgres connectivity for a few hundred applications on this over the past three years. If this fucks up, it'll be known without me.
I'm curious what requirement drove you to such arbitrarily small TTL, other than "because we can" dick-measuring geekery.
I applaud you for sticking to your guns though.
At the end of the day, we were worried about exactly these issues - if an application has to reload certs once every 2 years, it will always end up a mess.
And the conventional wisdom for application management and deployments is - if it's painful, do it more. Like this, applications in the container infrastructure are forced to get certificate deployment and reloading right on day 1.
And yes, some older application that were migrated to the infrastructure went ahead and loaded their credentials and certificates for other dependencies into their database or something like that and then ended up confused when this didn't work at all. Now it's fixed.
Why would the cert renewal be manual?
That's how it used to be done. Buy a certificate with a 2 year expiry and manually install it on your server (you only had 1; it was fine).
I can tell you that there are still quite a few of us out here that are doing the once a year manual renewal. I have suggested a plan to use Let's Encrypt with automated renewal, but for some companies, they are using old technology and/or old processes that "seniors" are comfortable with since they understand them and suggesting a better process isn't always looked favorably upon (especially if your job relies on the manual renewal process as one of those cryptic things only IT can do).
Some of this rhymes with Colm MacCárthaigh's case against mTLS.
https://news.ycombinator.com/item?id=25380301
This has been our issue too. We've had mandates for rotating OAuth secrets (client ID & client secret).
Except there are no APIs to rotate those. The infrastructure doesn't exist yet.
And refreshing those automatically does not validate ownership, unlike certificates where you can do a DNS check or an HTTP check.
Microsoft has some technology that next to these tokens they also have a per-machine certificate that is used to sign requests, and those certificates can't leave the machine.
We've also felt the pain for OAuth secrets. Current mandates for us are 6 months.
Because we run on Azure / AKS, switching to federated credentials ("workload identities") with the app registrations made most of the pain go away because MS manages all the rotations (3 months) etc. If you're on managed AKS the OIDC issuer side is also automagic. And it's free. I think GCP offers something similar.
https://learn.microsoft.com/en-us/entra/workload-id/workload...