And when the system fails for whatever reason?

Just because AI exists doesn't mean we can neglect basic design principles.

If we throw everything out the window, why don't we just name every file as a hash of its content? Why bother with ASCII names at all?

Fundamentally, it's the human that needs to maintain the system and fix it when it breaks, and that becomes significantly easier if it's designed in a way a human would interact with it. Take the AI away, and you still have a perfectly reasonable data store that a human can continue using.