This is a common misconception. The IRS is generally reasonable. They will send you notices about mistakes on your tax forms. I think it's the tax filing industry that perpetuates the myth that making mistakes on your tax form can lead to you going to jail.
Having said that, I used Claude to do my taxes this year. I generally use the free fillable forms provided by the IRS so I don't use a program that tries to abstract it all away. It was much easier to do with Claude than manually this year.
Interesting! As I said I've only heard complaints from Americans since I have no insight into how the tax filing works. If just general Claude is capable of doing it, then that is just as satisfying of an answer to why there isn't specialized AI tax software.