rmmod just tells you it's not loaded; you'd have to delete the module to prevent it auto-loading.