For starters, the average person doesn't use LLMs in any capacity beyond "ChatGPT" or whatever chatbot interface they happen to have latched on to.
I don't know the financial penalties of a faulty Tax filing in America since this is just done automatically for individuals in Sweden by our IRS-equivalent. But I assume this is something you don't want to mess up. If so, apply an general stochasticity argument for why AI isn't end-to-end responsible for much at all where details and correctness really matters.
This is a common misconception. The IRS is generally reasonable. They will send you notices about mistakes on your tax forms. I think it's the tax filing industry that perpetuates the myth that making mistakes on your tax form can lead to you going to jail.
Having said that, I used Claude to do my taxes this year. I generally use the free fillable forms provided by the IRS so I don't use a program that tries to abstract it all away. It was much easier to do with Claude than manually this year.
Interesting! As I said I've only heard complaints from Americans since I have no insight into how the tax filing works. If just general Claude is capable of doing it, then that is just as satisfying of an answer to why there isn't specialized AI tax software.