No it's not. Your computer creates a unique ID and you have to accept that on the unlocked phone the first time (or every time if you choose to).

So even when adb is on an attacker can't just plug into your phone and use it. Besides, I just switch it off when I don't use it