Ironically would have been better to use LLM to classify posts.

My thoughts exactly. Or at least curate a sample of true positives and use zero shot classification with semantic embeddings + keywords.