> Except this LLM would have a real shot of beating humans in open-domain problem solving.
At some point we need to start recognizing LLMs for what they are and stop making outlandish claims like this. A moment of reflection ought to reveal that “open domain problem solving” is not what an LLM does.
An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data), in the absence of a prior formulation of these laws in the training set.
TFA describes a need for scoring, at scale, qualitative results to human queries. Certainly that’s important (it’s what Google is built upon), but we don’t need to make outlandish claims about LLM capabilities to achieve it.
Or maybe we do if our next round of funding depends upon it.
As a function of energy, it’s provably impossible for a next word predictor with a constant energy per token to come up with anything that’s not in its training. (I think Yann LeCun came up with this?)
It seems to me RL was quite revolutionary (especially with protein folding/AlphaGo) - but using a minimal form of it to solve a training (not prediction) problem seems rather like bringing a bazooka to a banana fight.
Using explore/exploit methods to search potential problem spaces might really help propel this space forward. But the energy requirements do not favor the incumbents as things are now scaled to the current classic LLM format.
> An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data)
You could use Symbolic Regression instead, and the LLM will write the code. Under the hood it would use a genetic programming library like with SymbolicRegressor.
Found a reference:
> AI-Descartes, an AI scientist developed by researchers at IBM Research, Samsung AI, and the University of Maryland, Baltimore County, has reproduced key parts of Nobel Prize-winning work, including Langmuir’s gas behavior equations and Kepler’s third law of planetary motion. Supported by the Defense Advanced Research Projects Agency (DARPA), the AI system utilizes symbolic regression to find equations fitting data, and its most distinctive feature is its logical reasoning ability. This enables AI-Descartes to determine which equations best fit with background scientific theory. The system is particularly effective with noisy, real-world data and small data sets. The team is working on creating new datasets and training computers to read scientific papers and construct background theories to refine and expand the system’s capabilities.
https://scitechdaily.com/ai-descartes-a-scientific-renaissan...