Generalization from few examples is something transformers are bad at. They memorize outright. This is probably in part the objective, but humans have multiple readings of something they see, so generalization.

Transformers can also become confused be texts that no humans become confused by. I wrote some stories that I've used as test material, where I deliberately refuse to say who is speaking, or whose perspective we see, but where it is obvious to a human who it must be, and LLMs can make huge screwups in those texts. Mixing up an old guy with a young guy who he, when he was a similar age, was similar to, mixing up a kid with the kid's mother, that sort of thing.

I think the first part has no know solution. The second part is probably solvable, but not with a transformer-- maybe if they could make notes or output reasoning traces during prefill.

Both problems are hard and unsolved though.