Twitter has settings for who can reply to tweets, which are configurable per post. You can make it so that only people you follow can reply.