Even in casual Scrabble-like games, I expect using a restricted set of words would create a lot of "come on, that's totally a real word, why can't I use it" moments. Most people know at least a few uncommon words that most other people don't (because it's different words for each person).

The Wordle list of legal guesses is not substantially curated; AFAIK basically all five-letter words legal in Scrabble are on it (except on offensiveness grounds, which was a highly controversial decision). If this were not the case, I predict you'd get user dissatisfaction as per above. Wordle's list of possible answers is much more curated, but that's my point; it can err on the side of conservatism, because users won't notice if a word that they'd expect to be on there is missing, whereas they will notice if such a word is not allowed as a guess.