The issue is likely that readers for whom DHH sounds racist have (1) read the accusative fluff about him, and (2) then read some sentences in the blog post(s). If you read more, watched interviews or asked him directly, you'd get that it's less about race and more about merit. It's just that these attributes correlate (which is not racism).
I strongly disagree. I listened to a few of his interviews (one was the 5 hour long lex interview), and read the entirety of his more inflammatory blog posts. In one of them he specifically says he doesn't like modern London because it's not full of native british people, but what he really means is he doesn't like it because it's not full of white british people. Of course he doesn't say this outright but if you look up the statistics he sites you can see for yourself that he excludes asian, black and mixed british people from the statistic he lists. This is just one specific example.
I do think he also supports merit based immigration, but that doesn't mean he's not racist. He also very openly supports very extreme right wing political figures that promote things like ethnic cleansing. If you look into it more, he is pretty blatantly racist.