And if you ingest patches in seconds, you now have an issue of checking that the patch is not poisonous. Many a modern outage comes from changes deployed too fast. Damned if you do, damned if you don't.
if a CVE drops at 23pm, attackers have at least 8 hours before someone in the office has a possibility to take a look at it. nowadays, that seems to be plenty.
I think "good security/engineering practices" still has a chance.
Things like--
* Minimizing attack surfaces by not running lots of software on the same server and not bloating feature sets beyond the minimum
* Limiting access so that most things can't be done by most users, there are *no* super-users, and escalating access is temporary ans requires multiple accounts to authorize
* Encrypting data at rest
All software has flaws and it's now easier than ever to find and exploit them, but this disproportionately hurts people who were only putting on a show of security and relying on a single layer or two of defense instead of developing a robust insider threat model (which helps with both insider threats and threats that get inside) and defense in depth.
Completely agree there is lots you can do to shut down exposed surface area.
Is encrypting data at rest useful when the app must have access to decrypt the data? What sort of scenarios do you use it for and where do the keys live?
Full zero trust internally. Both inbound and outbound proxies (different machines) to carefully limit access, and provide a place to mitigate attacks. Etc.
If your plan is to quickly apply upstream patches, your plan is bad.
I try so hard for this, but don't even get approval to change an internal site to use https instead of http.
I get we gotta keep the new features and workflows coming, but a single resource dedicated to low hanging fruit of security could fix a lot in a very short time.
It requires management buy-in, both to allow some number of engineers to be allocated to such "non-production" work, and also to allow the friction security creates to continue to exist and "slow down" the "real work", both by vetoing things that would weaken security and by forcing everyone to use more cumbersome workflows.
That usually doesn't happen unless the company has suffered a large financial or reputational loss as a result of poor security practices. Sometimes not even then, if the company doesn't get scared they'll start losing money or get in some sort of legal trouble.
Have on-call sysadmins or have them work in a three-shift schedule. That way there's always someone there to fix stuff as it happens, including deploying security patches or worst case taking affected systems offline until they can be patched
This used to be common before devops and cloud took over. Looks like it might make a comeback. Maybe with a new fancier job title
There's a really simple solution that people will hate: server goes offline at night.
I've seen this a few times, and it's pretty annoying as a user, but ultimately it's not that bad usually. If you are ordering something at 3am, the physical shipment still occurs during daylight hours. Many banking operations are not actually instantaneous, they settle in a batch job overnight.
And this doesn't have to be binary. Providing a read-only interface provides lots of value while having a reduced attack surface.
You can have guardrails and still ask for approval. There is a difference between a guy who just has to approve a CVE fix and someone that has to take the time to understand what's going on, etc.
Yeah. There's a huge difference between how a 5-10 person start up or a 50-100 person business can handle this, compared to a tech giant or hyperscaler who's already got 1000 engineers in a dozen or more timezones.
It's easy, if you already have teams in the US, UK, India, and Australia - it's always normal office hours in one of those location.
When seconds matter, it is impossible to patch if you haven't patched already. If you need to patch, or attempt to contain against an adversary who is prepared and sophisticated enough to succeed within minutes, you've already lost. Defenders need to win every time. Offense only needs to win once.
And if you ingest patches in seconds, you now have an issue of checking that the patch is not poisonous. Many a modern outage comes from changes deployed too fast. Damned if you do, damned if you don't.
if a CVE drops at 23pm, attackers have at least 8 hours before someone in the office has a possibility to take a look at it. nowadays, that seems to be plenty.
i don't have a solution.
I think "good security/engineering practices" still has a chance.
Things like--
All software has flaws and it's now easier than ever to find and exploit them, but this disproportionately hurts people who were only putting on a show of security and relying on a single layer or two of defense instead of developing a robust insider threat model (which helps with both insider threats and threats that get inside) and defense in depth.Completely agree there is lots you can do to shut down exposed surface area.
Is encrypting data at rest useful when the app must have access to decrypt the data? What sort of scenarios do you use it for and where do the keys live?
Full zero trust internally. Both inbound and outbound proxies (different machines) to carefully limit access, and provide a place to mitigate attacks. Etc.
If your plan is to quickly apply upstream patches, your plan is bad.
> Full zero trust internally
I try so hard for this, but don't even get approval to change an internal site to use https instead of http.
I get we gotta keep the new features and workflows coming, but a single resource dedicated to low hanging fruit of security could fix a lot in a very short time.
It’s a hard transition, but possible. Obviously less work to do it from the start.
It requires management buy-in, both to allow some number of engineers to be allocated to such "non-production" work, and also to allow the friction security creates to continue to exist and "slow down" the "real work", both by vetoing things that would weaken security and by forcing everyone to use more cumbersome workflows.
That usually doesn't happen unless the company has suffered a large financial or reputational loss as a result of poor security practices. Sometimes not even then, if the company doesn't get scared they'll start losing money or get in some sort of legal trouble.
I’ve worked places that did a lot of this (not everything, but a lot) from very early on.
Have on-call sysadmins or have them work in a three-shift schedule. That way there's always someone there to fix stuff as it happens, including deploying security patches or worst case taking affected systems offline until they can be patched
This used to be common before devops and cloud took over. Looks like it might make a comeback. Maybe with a new fancier job title
There's a really simple solution that people will hate: server goes offline at night.
I've seen this a few times, and it's pretty annoying as a user, but ultimately it's not that bad usually. If you are ordering something at 3am, the physical shipment still occurs during daylight hours. Many banking operations are not actually instantaneous, they settle in a batch job overnight.
And this doesn't have to be binary. Providing a read-only interface provides lots of value while having a reduced attack surface.
Do you know about timezones?
"We have fully automated multi-cloud follow-the-sun platform migrations. All your code runs in datacenters where it's currently office hours."
People are gonna hate this but: we’ll need agents ready and waiting to autonomously patch and deploy instantly
Then you just prompt-inject the agent to convince it to deploy a malicious patch. This isn't a solution.
You can have guardrails and still ask for approval. There is a difference between a guy who just has to approve a CVE fix and someone that has to take the time to understand what's going on, etc.
It’s agents all the way down.
We need to deploy essential services at least twice, apply CVE ‘fixes’ to a subset of them, and have tooling to diff their outputs.
That tooling should be shut off from the internet [1], and if it uses a LLM that should be at least a few years old.
[1] technically impossible if it also has the ability to veto responses, but the surface can be very small.
A published CVE is kind of late in the game even.
low attack surface applications is the future.
23 pm is what, 11:00 am?
23 pm where? Simple answer is distributed teams.
You are going to provide me with a budget for that solution?
Easy to drop a sentence in a discussion. Harder to implement.
Yeah. There's a huge difference between how a 5-10 person start up or a 50-100 person business can handle this, compared to a tech giant or hyperscaler who's already got 1000 engineers in a dozen or more timezones.
It's easy, if you already have teams in the US, UK, India, and Australia - it's always normal office hours in one of those location.
I think Apple still hasn't patched the broken root certificate on macOS cli tooling, right?
It has been days now.
When seconds matter, it is impossible to patch if you haven't patched already. If you need to patch, or attempt to contain against an adversary who is prepared and sophisticated enough to succeed within minutes, you've already lost. Defenders need to win every time. Offense only needs to win once.
Citation:
Binance hot wallet signed 50 EIP-7702 delegations 3 minutes before the drain - https://news.ycombinator.com/item?id=50306715 - October 2026
Binance hot wallet signed 50 authorizations 3 minutes before the drain - https://808bits.com/articles/binance-hot-wallet-eip-7702-del... - October 30th, 2026
[dead]