Replacing a human tutor with an AI that will wildly hallucinate is wrong, and will directly contribute to the dumbing down of society that people here often bemoan.
This is the equivalent of "parenting" by putting a kid in front of YouTube Kids for half the day
"That will wildly hallucinate" feels wildly hyperbolic considering how fast models are improving in their abilities, especially with proper safeguards built into the model harness.
I was making a dessert recipe last week where everything including utensils needed to stay cold, and Gemini suggested I freeze my hands in an ice bath before getting started.
It's fairly common to briefly cool your hands with cold water or in an ice bath before working on certain recipes.
It used the words "freeze your hands"
I think any reasonable person would come to the conclusion it meant for you to chill your hands in ice water.