I brought this up during our AI workshops, but I called it the “confident idiot”

Seeing the idea explored in such depth is great, I really am concerned about this.