That sounds sensible to me. But I am likewise NAL.

If you've refused to answer questions and giving them the password, and they don't have a warrant, taking your case off and inputting the mysterious number they found in there seems a lot more like them destroying your (data) property by oafishly trying to force access to a phone they have no business messing with, rather than you ostensibly destroying evidence of a theoretical undiscovered crime.

I'm sure they would still try to mess with you in a dozen ways for annoying them, but at least you'd be on decently solid ground in court as far as that goes.