Does the model quality become degraded in other ways?

no because we freeze the main weights and only train our added weights.