shitty CAPTCHA’s and simple checksums have been ruled to be “effective measures”, so this would probably be too.
However, Section 1201(f) is designed to allow developers leeway in reverse engineering to make "independently created computer programs" talk to other "computer programs". But this usually distinguishes between talking to a binary (good) and reading a media file (bad).
My guess is the encoder is probably legal under 1201(f) because it’s reverse engineering a DMCA covered application to create new media files for it to use. But the decoder would be illegal because it reverse engineers a DMCA covered application to extract copyrighted media.
No, on appeal the simple checksum was ruled to NOT be an effective measure. [0]
And while courts might have ruled that a CAPTCHA might count as a "technological measure" they haven't gotten as far as ruling them as "effective" yet.
But in general yes. The protection scheme doesn't need to be well designed or free of design flaws to count as "effective". But from what I can tell, it does need to be a valid attempt at some cryptographic scheme requiring a secret known only to the copyright holder.
[0] https://law.justia.com/cases/federal/appellate-courts/F3/387...