Is that with or without the LLM being able to assemble + profile its hypotheses itself? Because “optimize this code to execute in fewer CPU cycles under the test harness” seems like one of those perfect self-contained problems for LLMs (their equivalent of an “embarrassingly parallel problem”: an “embarrassingly-easily-explored solution space with embarrassingly-easily-measured objective success criteria.”)

Sure, they might make all sorts of dumb hypotheses at first, but as long as the results of those stay in their context, they do seem to eventually “run out of ways to be stupid.” (Which is to say, LLMs seem to experience in-context learning even via self-directed trial-and-error, if given a sufficiently-large number of iterations and no way to cheat.)