The dose makes the poison, and with refined sugar the dose required to be harmful is quite small compared to sugar embedded in while fruits.