I think the main reason you see this happening over and over again is because we're teaching this whole discipline wrong. By 1960 most of the problems in software development were known and had one or more solutions. Knuth spent decades just cataloging what was mostly already known (and moved the field forward in quite a few occasions as well).
And yet, you can't go a day without someone declaring that now is the time to do it right, this time it will be different. And then they proceed to do one thing after another for which the outcome is already known, just not to them. I think the best way to teach would be to start off with a fairly detailed history of what had gone before, just to give people a map and some basic awareness of the degree to which things have already been done, rather than to find new and interesting ways to shoot themselves in the foot (again).
Unlike actual engineering, software "engineering" as a field has decided to reinvent itself every generation - worse, every turn of the trends, even every project and person. Majority of the practitioners are in it for superficial reasons, unaware of its rich history and culture.
With ignorance comes arrogance of an individualist intellectual, thinking their unique revolutionary contribution will wow the public and move the field forward. Except inevitably they're not only reinventing the wheel but the entire automobile, without knowing basic principles and the work of predecessors. It has a lot in common with modern art.
> we're teaching this whole discipline wrong
I sometimes think languages after C, like C++ and Java, were misguided in some ways. Sure they provided business value, brought new ideas, and the software worked - but their popularity came at a cost of leaving countless great thoughts behind in history, and resulted in a poverty of software culture, education and imagination.
There are optimistic signs of people returning to the roots, re-learning the lessons and re-discovering ideas. I think many are coming to realize the need for a reformation of sorts.