> nearly as accurate as a human book keeper
Anything to avoid using the metric system.
Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.
> nearly as accurate as a human book keeper
Anything to avoid using the metric system.
Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.
Hey, author of the blog post here.
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.
It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.
You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.
Why would I want both a less accurate book keeper and to incur all of the liability of doing the books myself?!
It's not less accurate. As commented above, the control knew they were being tested against the machine, so made sure to be super careful.
In everyday life the human is less careful, and the machine costs 1% of the human.
Presumably your current book keeper is not your slave, and you have to pay them...
...again: liability is the key issue. Cost savings a not exactly an isolated issue here.
Then why only have one human bookkeeper? Surely two would be better, since you can compare their results. But then, perhaps you should hire three, so you can figure out which one is right.
Regardless of how correct they are, they assume liability: which is a metric that you do not improve with more bookkeepers.
But what if the third bookkeeper is malicious? You need at least four bookkeepers to achieve byzantine fault tolerance with f=1.
does this assume a maximum of one malicious bookkeeper?
Yes, you need 3f+1, so 4 for f=1, 7 for f=2.
https://lamport.azurewebsites.net/pubs/byz.pdf
Got some bad news for you about that "liability" thing: it's always been on you. Read the fine print on your tax forms sometime.
Because it's way cheaper. Not saying it's a good tradeoff, but that appears to be their pitch.