Ask each of those if they can spell seventeen, and then ask them how many e's are in the word.

They're now over fitted on strawberry has 3 R's. So ask them how many R's are in srrarbrarry. There are, obviously, 3 R's in srrarbrarry too.

Ask a color blind person to spell "red" and "green", then ask them how many red apples are on the table.

LLMs work on tokens. They literally cannot see individual letters. Yet, unlike humans, they can instantly create a tool to solve this task for them and never be stumped by it again.

I'm so tired of this brain-dead gotcha comment.

It is important to understand how this technology works. It isn't magic. Your example is solid. Similar to the Chinese Room.

Do you think your legislators or most users understand tokens and predictions and what is going on? Do you think that those who bombed a school using it know this?

Until frontier models can account for the disparity between what they can do and what the person (or other agent) is asking, we will continue to suffer bad information used like good information.

That tokens cannot count letters like that is not a brain-dead gotcha. Ignoring how humans are using these models is lacking in critical understanding of the real world. People are literally going to die due to humans not understanding how this works (and already have). True brain-dead, and body-dead.