A little tip for LLM-as-classifier that worked for me.

Use binary questions rather than multi-class. Then you can use the consistency as another signal of your pipeline working or not, on top of accuracy.