But you can't legalize all crime. Drugs are special because they are (sort of) victimless.

Drugs are absolutely not victimless, except perhaps for caffeine.