If not for copyright, you could always do that and copyleft wouldn't be needed.

Just because something is copyleft doesn't mean the person who gave you the binary you're using has to supply you with the code the used to build it. That's what the GPL does.