Decompiling binaries and building something very similar is surely an act of piracy.

That’s only really done for video games where you want an identical clone. The vibecoded photoshops aren’t decomps. They are just asking LLMs to build photoshop from scratch in rust.

You can't prove that point,because model training data is opaque.

It might be, depending on what the LLM used the binary for. Patented algorithms? Yes. File format compatibility? Quite possibly not.

But what about when they're only given the user manual and told to implement it?

And extra "fun", given that "hack HuggingFace" wasn't part of the thing being tested when OpenAI's LLM did exactly that: what happens when the LLM decides the best way to implement the user manual is to decompile the binary?