Removing meaningless chatter can be helpful, but a non reasoning LLM needs to generate text to "think". If you force a non reasoning LLM to produce a single boolean result, then it's just a coin flip.
Had a little luck with having it do an impression of the Star Trek computer, although at the cost of having it try to insert star-trek themed hallucinations like warp engine status.
Removing meaningless chatter can be helpful, but a non reasoning LLM needs to generate text to "think". If you force a non reasoning LLM to produce a single boolean result, then it's just a coin flip.
In my experience the accuracy was fine but actually reading the output was so annoying I removed it.
Had a little luck with having it do an impression of the Star Trek computer, although at the cost of having it try to insert star-trek themed hallucinations like warp engine status.