I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this: "If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?" LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.
You just need to give an instruction something along the lines of "any initial calculation or derived value must be computed through the use of a $LANG program."
Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.
In my experience agents will reach for numpy for any non trivial math without being instructed
It's the judgement of "trivial" where I find issues.
Most calculations are trivial.
It is also trivial to specify deterministic tool calling for all math
Exactly. That's why I specify it that way.
[flagged]
As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project.
“The scale reads −(100 g + the weight of the empty container).”
I literally gave it your question. What models did you test?
[dead]
[dead]