Signal claims to know nothing. However, for a few years now users' contact lists are uploaded to Signal's servers with the notoriously insecure Intel Secure Enclave being the only protection. It is likely that a state actor has access to that, which is already highly desirable information for mass surveillance.
Is it really?
I once worked for a company that happened to find itself in possession of a nearly complete social graph of one of the rich countries. The goal of the project was a different one, that graph was a kind of side effect. The graph was never actually used, but the company did have it.
Producing the graph was neither difficult nor expensive. I believe the complete project cost only a little over €1M.
If you want to gather data like that, you can do it without any expensive intelligence operations or attacks. You can spend a million on writing a desirable free smartphone app that needs contact permissions and another few hundred thousand on promoting the app, then sit back while the data is uploaded to your servers. To me that appraoch sounds a lot simpler and cheaper than breaking into Signal, Intel/SGX or a DC hoster.
I suggest that attacking anyone to get their contact data isn't really desirable.
> the notoriously insecure Intel Secure Enclave being the only protection
While I share your concerns about Intel SGX, your statement is not exactly true: SGX is only meant as an additional measure to secure insecure PINs.