Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
> fluid, self-modifying things w/ contracts
This smells like worm food. Those contracts had better be airtight.
What would you prefer: 99% of user requests to succeed but w/ 5% risk of getting O-day'd, or have 90% of user requests succeed but w/ 0.1% risk of getting 0-day'd.
The first and it isn't even close. Normal operation of the system is more important than preventing attacks, and trading it away at a 2:1 ratio is a bad trade.
You're trading it away at a 50:1 ratio though.
The point of the example is that with the first choice you have 5% legitimate customers and 5% malicious "customers" and by letting all of them in aka all 10%, you get maximum customer satisfaction including satisfaction of the malicious customers.
Alternatively you reject the malicious customers and as collateral damage accept that some customers do not get their request approved.
You're also confusing "normal operation" with what a user request is. A user request is a request to change something, so by definition it is not the continuation of the existing system. You're thinking that a user request is a HTTP request which is not what's being discussed.
Personally - for banking - I'd be plenty happy dealing w/ 90% uptime if it gave me an order of magnitude more security against getting robbed. This tradeoff may not have been realistic a decade ago, but it seems to be where things are trending.
It also is de-facto the case today when logging in as per usual sometimes requires a captcha or additional verification.
I could imagine it going so far as to have an LLM re-impl core libs every time a request is processed. Lightning never strikes the same place twice.