Such a nice design on server-side.. and yet on the client side, it uses system address book - something that Google backs up on the server, and many carriers back up too, and many apps (like Whatsapp) save it too.
"Hey, _we_ don't store your contacts, we are good! Instead you have to manage them yourself, and in process share your presumably "secure" Signal contact list with Apple, Google, Facebook phone carriers and everyone else. But it's not on our servers, so we don't care"