It's previous recipients synced over iCloud.

Run this and then send an email to see it: log stream --predicate 'subsystem == "com.apple.cloudkit"' --info --debug

Verified - and the actual payload to the iCloud endpoint is e2e encrypted ciphertext.