> One team suggested that instead of text-to-SQL, we should build semantic business logic layers, “show me Q4 revenue” should map to a verified calculation, not raw SQL generation.
Okay, how would that work though? Verified by who and calculated by what?
I need deets.
They're saying that someone should implement the CalculateQuarterRevenue(year, quarter) function somewhere in a manner that has been verified (e.g. run it against previous quarters to make sure it works correctly) then rather than using the LLM to generate SQL you use it to decide what domain function should be called. Which to me seems to mean that someone on the panel was gently taking the piss out of the idea. Since if you've done all the hardwork anyway presenting this in a deterministic way with a nice UX is straightforward bit of front end work.
It also removes a lot of the value of the LLM. They're perceived as being smart, and the interface (open-ended text) implies they are capable of more than executing pre-defined functions.
So if you have a "CalculateQuarterRevenue(year, quarter)" function, you'll soon find your users asking for the data per-month. Or just for the last six weeks. Or just for a specific client. And they'll be confused when it doesn't work.
The conversational user interface is misleading then, isn't it? It can't make you a sandwich either, though it allows you to submit this request.
Yes. The sandwich example is contrived, but the basis is "discoverability." It's very opaque to the user what actually can be done and how reliable the result is.
Compare this to basically any website you've ever been to. It's the "GUIs vs. CLIs" discussion all over again, except even CLIs had man pages for discoverability.
In other words, there should be a list of predefined queries, or possibly subqueries, that the user can request. This is basically how products used to work before AI. The difference is now you can request which query you want verbally.
edit: I'm serious. I'm just answering the question, not making a value judgement.
I assume you're being tongue in check but I've watched a lot of people use software and they really just don't know anything about it. Being able to verbally request something is something they can learn to do while googling how do I normalize the scores in my rubric to add up to 100 is something they couldn't.
Verbal queries is the solution for the world we have even if it's not optimal.
Your last sentence sums it up. This is what users want.
The main killer app, I think, boils down really expensive speech-to-text (and vice versa) with a reasonable number of seemingly authoritative querying details in fairly plain language. It's a new, 'better' search engine, just with different pitfalls people need to get up to speed on. And that may be enough, because employing humans to fill the same role as effectively is more expensive still.
So simple classification problem. Big deal.
On one side, you have an agent calculating the revenue.
On the other side, you have an SQL that calculates the revenue
Compare the two. If the two disagree, get the AI to try again. If the AI is still wrong after 10 tries, just use the SQL output.
so you have an answer and then you throw compute at trying to produce the answer in a different way.
What I hear is a billion dollar AI startup in the making!
Well put...lol :-)
A simple way is perhaps implement a text-to-metrics system where metrics could be defined as SQL functions.
psychedelics