will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.
You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?
I can't wait for qwen4 flash next or something like a 150b moe from qwen.