> So why Lisp (or when)
> [...] among its extensibility, its interactive environment, the REPL, and a lot of other features we haven’t touched yet. It is the combination of all of them that makes Lisp programming what it is.
Agreed. However, although the alternatives are few, they do exist. Today, I'd like to convince you (whether you're OP or a commenter) to give one of them a try. I'm talking about GToolkit[1]: Smalltalk/Pharo-based reimagining of Smalltalk as a productive environment for modern system design, analysis, and implementation. It's based on Pharo and its VM, but with GT-specific extensions and replacements, developed on GitHub in both Smalltalk and Rust.
I used both Common Lisp and Smalltalk over the years for some of my side projects. Technically, the environments are comparable: image-based, live, interactive development is central to both. Lisp is easier to fit into modern workflows: it's still just files on disk. If you ignore the REPL and treat the image as a bundler, you can have a Java- or Python-like development workflow. You can gradually adopt the more interactive ways of working with the codebase. Smalltalk was historically more of an "all or nothing" approach, but nowadays it supports Git-based workflows as first-class, with GToolkit providing additional tools directly. GT is tied to a single IDE, but that IDE is genuinely powerful, pragmatic, and easy to customize: the entire IDE is Smalltalk code[2] that lives in the image beside your code, so you can live-edit any part of it at any time.
The languages differ, most obviously in the object model (single inheritance and message sends in Smalltalk, and multiple inheritance with multimethods in CLOS), but I don't think one is strictly better than the other for the vast majority of code. You just need to structure your code differently. Both systems are very dynamic, so neither is like Java or C++. After quite a few projects in both, I am convinced that this difference doesn't matter.
Both languages have very simple syntax. Smalltalk doesn't have macros; instead, it has an extensible/replaceable compiler for method bodies. This capability is used, for example, for compiling grammars into parsers/lexers in SmaCC. It's not as convenient for control-flow abstraction as CL's macros, but Smalltalk tends to make them out of blocks (lexical closures with non-local exits) plus actions on thisContext. In practice, I never found either language too limiting in what I could express, and both have small, regular basic syntax.
TL;DR: Common Lisp and GToolkit Smalltalk offer comparable technical merits, with the most pronounced difference being GT's built-in, high-quality IDE, which open source Lisp lacks.
The IDE (and the capability to build cross-platform, but natively rendered GUI apps) is the major selling point of GToolkit, but the bigger reason to consider it is social. Lisp had a bit of a renaissance from 2005 to 2015, but ~it has since died down~ (EDIT: After checking a bit, it's more like there are similar levels of development now, but it's just varied, and I personally hit a few areas where nothing new happened for a while. Or in other words, the hopes of CL getting catapulted to mainstream by the wave of adoption didn't materialize, and the adoption seems to have largely plateaued, with a slight upward trend since then.) A lot of great things for Lisp appeared in that time: ASDF and Quicklisp being prime examples. Unfortunately, the momentum was lost, and since then, the rate of development has plummeted. It's still a solid proposition because of its stability, which means a lot of old code still works perfectly well, but it's a double-edged sword: it would take a hundred miracles in a row for CL to get a plausible M:N concurrency story, for example. This isn't true for Smalltalk, and doubly so.
There's a small but active community around Pharo, mostly academics and hobbyists (IME; and it's from a decade ago, so YMMV). Pharo itself is already a "Smalltalk-like" language, and it regularly gets features that are not in original Smalltalk (stateful traits and slots (reification of class and instance variables) are good examples). It continues to evolve, and each version brings notable improvements to the language, the VM, and the class library. In CL, since there are many implementations maintained by various groups and the standard is set in stone, adopting extensions to the standard is incredibly hard and time-consuming. You can, of course, commit to a single implementation, but there are only 2 or 3 implementations that are actively developed and trying to do something "new"; they are all experimental and incomplete. And while SBCL has quite a few nice features, its codebase was a bit hard for me to grok (read: days on end of banging my head against a wall, then giving up; I'm much more comfortable messing with Smalltalk bytecode compiler).
Secondly, GToolkit is driven by a group of coders[3] who have established a profitable consultancy. It's being developed around a single vision that the authors swear helps them achieve their projects' goals. Even if you're not sold on moldable development[4] as a methodology, GToolkit gives you all the tools you might need in a modern development and then adds tons of domain-specific tools and utilities that might be useful after just a bit of adjustment. The whole environment is built to make such adjustments as painless as possible, too.
The only problem I experience with GToolkit is the lack of structured, book-like documentation. The GToolkit book is closer to the PHP wiki or a bundle of tutorials than something like the Rust book. It's not that the docs aren't there - they often are, but finding them effectively is a bit challenging. I found that simply cloning all the repos that GT consists of (tens, if not hundreds, but there's automation for that) and pointing an agent to the directory with them is often enough to quickly find what I'm looking for (if the built-in Spotter fails and I'm too lazy to construct an in-image search).
TL;DR: GToolkit/Pharo move much faster than the CL world, and while the number of maintainers might be similar (honestly hard to estimate, though), the effort on the GT side is more centralized and, in my opinion, heads in a better direction. If you want an extensible, live, dynamic environment that shares many of Common Lisp's strengths, but is more polished, more actively maintained, and feels more modern, take a look at GToolkit.
(Disclaimer: no affiliation, just a programmer who likes to explore unpopular languages).
[2] Some parts are implemented as Rust dynamic libraries and called from Smalltalk via FFI.
All three major programming environments at Xerox PARC, shared similar concepts.
Interlisp-D, Smalltalk, Mesa (XDE) which evolved into Cedar.
If you read Xerox papers about all of them, there are several quotes on how relevant it was to share the same programming experience across environments.
Which is why, given their linage, JVM and CLR are the closest big mindshare ecosystems that somehow still have traces of those features when using their IDEs and runtimes, even without being a proper Smalltalk or Lisp.
> how relevant it was to share the same programming experience across environments.
Right. If you view the language as just a part of a bigger "programming experience", you can do with a worse language with better tooling (pathological example: (lack of) namespaces in many Smalltalks to this day). If you focus on language design above all else, you end up with either something pretty on paper (but never implemented) or something practical and (kind of) elegant, but it'll take 40 years to get the tooling to where you want it (and still won't cover many important capabilities).
Also, it's easier to swap languages if you focus more on tooling. From what I read, many Lisp and Smalltalk IDEs/environments were (and some still are) polyglot. I sadly didn't have access to them at the time, but I imagine switching from Turbo Pascal to Turbo C++ was easier than going from Turbo C++ to Visual C++? That said, tooling tends to break if the language actively fights it, so it's not like PL design is completely irrelevant.
> Which is why, given their linage, JVM and CLR are the closest big mindshare ecosystems that somehow still have traces of those features when using their IDEs and runtimes
Yes. The only problem is that those concepts/features become niche black magic (somewhat unavoidable for advanced features in "big mindshare ecosystem"). You can definitely compile Java classes dynamically at runtime, install them, and use them normally; however, outside of frameworks/IDEs/JVM languages, the ClassLoader is used mostly for loading static assets, and even simple reflection is viewed with suspicion.
Using Smalltalk or Lisp directly puts those powerful concepts "front and center", which a) is good for education; and more importantly, b) feels good when things click :) It could be an advantage for small groups of programmers, but from what I understand, it becomes a liability in large organizations. Pharo is especially bad at it, and GToolkit doesn't fix everything (they do a good job at providing tools, but don't want to maintain too large a patch set for Pharo, which is a good trade-off given their limited resources). But I'm replying to "Why Lisp (or When)", and in this context, Smalltalk is (IMO) the better direct alternative than JVM/CLR.