The issue is that many optimization opportunities only appear after monomorphization, inlining, de-virtualization, etc. etc.

Not that you couldn't do source level analysis as you suggest... it just wouldn't be effective in many cases.

It would also be 'unstable' in the sense that it might depend on architecture, etc.