I think an important comparison is the LLM with the supervised training data in context. I wonder how the LLM would perform if you provided all, or a representative subset of, the data used to fine tune the CNN in the prompt and then asked it to classify a new case.

This strategy is how LLM-like models have achieved leading performance on tabular prediction tasks (TabPFN).