Yes, to a certain degree, although it does take initial effort to make it happen.

LLMs do somewhat okay job when you use them with Lisp, treating it just like any other PL, which is roughly the Unix/pipe model - batch-style. Agent spawns process -> reads stdout/stderr -> spawns next process. State lives in-between the calls and in files. Each tool invocation is stateless.

Things get far more interesting when you give an LLM a true Lisp REPL. LLM stops guessing and starts empirically analyzing current state of things and produces working solution faster, costing far less tokens. And you get to watch it solve things interactively, e.g. I often let AI poke through our UI (via Playwright-driven Clojurescript REPL), while monitoring situation in k8s - through nrepl port, connected to Clojure REPL. It literally interactively walks the DOM, finds the selectors, clicks buttons, etc. All without restarts, complex state management and all.