I mean...you should always design with speed in mind (In that Jeff Dean sense :) but what 'premature optimization' is referring to, is more like localized speed optimizations/hacks. Don't do those until a) you know you'll need it and b) you know where it will help.