"confidence": 0, so I guess you could threshold it

Only if the confidences are calibrated, and they probably aren't. Any idea how the confidence is derived?