Yeah, except the CPUs are complex, you might not understand ALL the details of a working program (like randomising memory layout).
Plus not all speed-ups are made equal. Some speedups of a part of program can have no or even negative impact (as in making part of program faster slows down overall program execution).
No doubt, they are complex and you can't factor for everything, but I suspect that just even a cursory understanding of some broad characteristics across hardware will help you to design faster software when using a low-level language (and to some extent when using higher level languages as well), and your software will probably achieve must faster speeds than focusing on exotic design patterns in your software, for example.
The truth is really a middle ground and the fact that you won't get the design right the first time around until you have built that hardware programming experience and learned from past mistakes. But this statement doesn't make for a good marketing stunt.