So? What's the threat model? How does having phone numbers help?

You're making assertions about what Signal needs, and you're doing it without knowing their threat model.