I use LLM_based coding agents frequently with Lisp languages and my way of working is different with Lisp languages: 1) if generated code ever has a syntax error, I like to quickly fix the syntax error myself. 2) for some reason I usually prefer to run tests myself in another terminal (with Python, Typescript, etc. I let the coding harness run tests).
I haven’t let coding harnesses run REPLs. When I do let a coding harness run tests I specify in, for examp,e, my Common Lisp skills file to run ‘sbcl -load …” so bash test commands are one liners.
It would be interesting to work on skills and a harness to use REPLs - nothing bad about that idea, I just haven’t tried it.
Yes, to a certain degree, although it does take initial effort to make it happen.
LLMs do somewhat okay job when you use them with Lisp, treating it just like any other PL, which is roughly the Unix/pipe model - batch-style. Agent spawns process -> reads stdout/stderr -> spawns next process. State lives in-between the calls and in files. Each tool invocation is stateless.
Things get far more interesting when you give an LLM a true Lisp REPL. LLM stops guessing and starts empirically analyzing current state of things and produces working solution faster, costing far less tokens. And you get to watch it solve things interactively, e.g. I often let AI poke through our UI (via Playwright-driven Clojurescript REPL), while monitoring situation in k8s - through nrepl port, connected to Clojure REPL. It literally interactively walks the DOM, finds the selectors, clicks buttons, etc. All without restarts, complex state management and all.
Depends very much on your harness and effective use of tools. For vibe-coding, stick to Python.
It can get good with the right setup. I made Codex work with GToolkit (similarly underrepresented, but technically impressive and a seemingly good fit for LLMs), but it took a lot of tweaking of the project structure, extensive instructions in AGENTS.md, and some custom skills (some borrowed from Gt4Llm, the GT's built-in model harness). Out of the box, it burned tokens and took way too long to implement even basic things.
YMMV - as with everything LLM-related - but I think without a similar setup, an agent instructed to write CL/Clj/Scheme/Racket will have the same issues. It might be better in an established, large project - but starting from an empty Git repo, I suspect you'll have to fight your way to productivity.
Current SOTA models work great with Emacs Lisp and Common Lisp. Giving access to the REPL enables them to do an amazing job but they still work well with Unix input/output static code processing paradigm.
I've found LLMs to be bad at balancing parenthesis. I've also found them to be less likely to hallucinate library types in dynamic languages, they tend to hallucinate arguments to library functions/methods instead.
Because you treating Lisp just like any other (non-homoiconic) PL. Give an agent a true Lisp REPL to mess around, and you'd be surprised. Things get very interesting. I still don't understand why more people don't do that - isn't that obvious first thing anyone should figure out? Like I can't even imagine working with Lisp without a REPL and structural editing - I'd immediately fail at balancing parens. Why do you expect a [dumber] machine would do any better?
Yeah. In my GToolkit setup, I started with giving it eval. At first, it tried to "rebuild the world" for each task, but we gradually settled on a set of images for specific tasks that can be rebuilt as a separate step and are otherwise cached. That gave us fast eval, and it kinda snowballed from there. I thought I would need to implement some IPC into a live image, but the startup is fast enough that it doesn't matter too much. The agent now has both the textual source (in Tonel) on disk and can easily query a live image via CLI.
What do you think about making the agent write type annotations? There are built-in forms in CL, and in Smalltalk, I settled for pragmas for now. They are not checked, but since I started using them, I think the rate of one- or two-shotting solutions has gone up.
assuming you're using state of the art llms, when they start failing with paren balancing it means they're approaching the limits of their context window and a new session should be started.
yes if you run a repl server like slynk the llms will happily communicate with your running program to develop and test things. It's often quite creepy to witness.
I use LLM_based coding agents frequently with Lisp languages and my way of working is different with Lisp languages: 1) if generated code ever has a syntax error, I like to quickly fix the syntax error myself. 2) for some reason I usually prefer to run tests myself in another terminal (with Python, Typescript, etc. I let the coding harness run tests).
I haven’t let coding harnesses run REPLs. When I do let a coding harness run tests I specify in, for examp,e, my Common Lisp skills file to run ‘sbcl -load …” so bash test commands are one liners.
It would be interesting to work on skills and a harness to use REPLs - nothing bad about that idea, I just haven’t tried it.
Yes, to a certain degree, although it does take initial effort to make it happen.
LLMs do somewhat okay job when you use them with Lisp, treating it just like any other PL, which is roughly the Unix/pipe model - batch-style. Agent spawns process -> reads stdout/stderr -> spawns next process. State lives in-between the calls and in files. Each tool invocation is stateless.
Things get far more interesting when you give an LLM a true Lisp REPL. LLM stops guessing and starts empirically analyzing current state of things and produces working solution faster, costing far less tokens. And you get to watch it solve things interactively, e.g. I often let AI poke through our UI (via Playwright-driven Clojurescript REPL), while monitoring situation in k8s - through nrepl port, connected to Clojure REPL. It literally interactively walks the DOM, finds the selectors, clicks buttons, etc. All without restarts, complex state management and all.
Depends very much on your harness and effective use of tools. For vibe-coding, stick to Python.
It can get good with the right setup. I made Codex work with GToolkit (similarly underrepresented, but technically impressive and a seemingly good fit for LLMs), but it took a lot of tweaking of the project structure, extensive instructions in AGENTS.md, and some custom skills (some borrowed from Gt4Llm, the GT's built-in model harness). Out of the box, it burned tokens and took way too long to implement even basic things.
YMMV - as with everything LLM-related - but I think without a similar setup, an agent instructed to write CL/Clj/Scheme/Racket will have the same issues. It might be better in an established, large project - but starting from an empty Git repo, I suspect you'll have to fight your way to productivity.
Current SOTA models work great with Emacs Lisp and Common Lisp. Giving access to the REPL enables them to do an amazing job but they still work well with Unix input/output static code processing paradigm.
I've found LLMs to be bad at balancing parenthesis. I've also found them to be less likely to hallucinate library types in dynamic languages, they tend to hallucinate arguments to library functions/methods instead.
> LLMs to be bad at balancing parenthesis
Because you treating Lisp just like any other (non-homoiconic) PL. Give an agent a true Lisp REPL to mess around, and you'd be surprised. Things get very interesting. I still don't understand why more people don't do that - isn't that obvious first thing anyone should figure out? Like I can't even imagine working with Lisp without a REPL and structural editing - I'd immediately fail at balancing parens. Why do you expect a [dumber] machine would do any better?
Yeah. In my GToolkit setup, I started with giving it eval. At first, it tried to "rebuild the world" for each task, but we gradually settled on a set of images for specific tasks that can be rebuilt as a separate step and are otherwise cached. That gave us fast eval, and it kinda snowballed from there. I thought I would need to implement some IPC into a live image, but the startup is fast enough that it doesn't matter too much. The agent now has both the textual source (in Tonel) on disk and can easily query a live image via CLI.
What do you think about making the agent write type annotations? There are built-in forms in CL, and in Smalltalk, I settled for pragmas for now. They are not checked, but since I started using them, I think the rate of one- or two-shotting solutions has gone up.
> I've found LLMs to be bad at balancing parenthesis.
I think that was true last year. In my experience, it’s no longer the case with Claude Code or Codex.
assuming you're using state of the art llms, when they start failing with paren balancing it means they're approaching the limits of their context window and a new session should be started.
yes if you run a repl server like slynk the llms will happily communicate with your running program to develop and test things. It's often quite creepy to witness.
Why is this being downvoted? It's a reasonable question.
It is fashionable to misunderstand Lisp and hate AI.
I spent years (on my free time) coding Lisp, even had the aspiration (but I was younger then) to rewrite P4V (Perforce Client GUI) in it :) -
https://github.com/malkia/p4bee
but haven't touched it in years....