If you're an amazing, top 1% software engineer, and you switch jobs to become a penetration tester, you're not going to be top 1% day one. There's knowledge specific to penetration testing you'll need to research and absorb and practice before you're good.
For the models, it's always day one. They don't learn from experience. Whatever they know after training is all they'll ever know. So if you don't teach them penetration testing in training, they won't know it. Maybe they could theoretically derive everything from first principles, and so could a top 1% software engineer, but that's a very slow and inefficient way to do things.
Im not sure if transfer learning is still applicable to these models, but if it is then we can just train the last few layers and get the model to ace in a related domain ( going from SWE -> Cyber ).
> Whatever they know after training is all they'll ever know.
Feeding enough source material as context can absolutely make them better.
Having something dumb that just cycles through the state of the art for exploits is already absolutely incredible security tooling (if people bother to make it into a process which I'm less optimistic about)