As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not?
I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
I think it’s sort of self-defined. If a model is able to use bash well enough to not need specific tools.
The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance.
Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
> Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.
This has been my conclusion as well, and I'm doing my best to try and back this up quantitatively. In a perfect world, I could smite all of the internal MCP servers in my Corp environment and replace them with REST/GraphQL. No one using any of these servers is hitting them with models that would perform worse orchestrating with bash- and some of these folks are running harnesses with awful MCP clients.
I am still wondering about the effectiveness of using grep/awk (and ad-hoc python scripts) in code bases, as opposed to more sophisticated LSP and the like?
I think it just means “is there some parser program that consumes the LLM token stream and spawns shell processes with the detected command strings”.
Im not sure how this could make sense, considering the content of the paper.