I've found LLMs to be bad at balancing parenthesis. I've also found them to be less likely to hallucinate library types in dynamic languages, they tend to hallucinate arguments to library functions/methods instead.
I've found LLMs to be bad at balancing parenthesis. I've also found them to be less likely to hallucinate library types in dynamic languages, they tend to hallucinate arguments to library functions/methods instead.
> LLMs to be bad at balancing parenthesis
Because you treating Lisp just like any other (non-homoiconic) PL. Give an agent a true Lisp REPL to mess around, and you'd be surprised. Things get very interesting. I still don't understand why more people don't do that - isn't that obvious first thing anyone should figure out? Like I can't even imagine working with Lisp without a REPL and structural editing - I'd immediately fail at balancing parens. Why do you expect a [dumber] machine would do any better?
Yeah. In my GToolkit setup, I started with giving it eval. At first, it tried to "rebuild the world" for each task, but we gradually settled on a set of images for specific tasks that can be rebuilt as a separate step and are otherwise cached. That gave us fast eval, and it kinda snowballed from there. I thought I would need to implement some IPC into a live image, but the startup is fast enough that it doesn't matter too much. The agent now has both the textual source (in Tonel) on disk and can easily query a live image via CLI.
What do you think about making the agent write type annotations? There are built-in forms in CL, and in Smalltalk, I settled for pragmas for now. They are not checked, but since I started using them, I think the rate of one- or two-shotting solutions has gone up.
> I've found LLMs to be bad at balancing parenthesis.
I think that was true last year. In my experience, it’s no longer the case with Claude Code or Codex.
assuming you're using state of the art llms, when they start failing with paren balancing it means they're approaching the limits of their context window and a new session should be started.