What plausible mechanism is there to even approach the idea that skin color influences intelligence? Why not study the effects on intelligence of eye color, or blood type?
This is especially clearly just a racist concept once you realize that none of the studies of IQ vs race actually do any genetic testing, they only rely on phenotype. Which is absurd given the fact that the child of a black-skinned person born in sub-Saharan Africa and a white-skinned person born in a village in Ireland is extremely likely to be black-skinned, but is genetically just as much part of both populations. It's only the history of racism that assigns this child as black more so than white.
We know that people with red hair tend to metabolize anesthesia differently. It is not completely implausible that there is also a correlation between looks and other traits, although it seems almost impossible to me to untangle cultural, environmental and genetic effects.
Of course, any two traits can be correlated. But unless there is an obvious causal mechanism ("whiter skinned people get sun burns more easily", "darker skinned people tend to have vitamin D deficiencies at higher latitudes"), there's no reason to assume it and start looking for it to prove or disprove. It's a priori just as likely that there is a genetic link between toe shape and intelligence as skin color and intelligence. Yet no one is doing the science of measuring toe length correlation to intelligence, just skin color ("race").
As far as I know phrenology measured all kinds of things and claimed correlations. But they used highly dubious data (even for their time) and the field is now too unpopular to try (something similar) again with better methodology.
And I would say that the racist claim is that there is observable correlation and that genetic explanations are about as probable as other explanations. But I suppose many non-genetic explanations (eg cultural differences) would also be counted as racism, since in these kinds of discussions race and culture are often jumbled together.