I found a solution I am happy with. Based on my research into this it is a problem I'm not alone with. I do concede that writing your own analyzers is unusual (which is why i wrote that it's fine for simple setups). At the same time I deem having a common library to be referred by more than one analyzer project something that should be possible without running into strange errors.

If a tool (dotnet build) tells me something is wrong I am fine with it. If the same tool works after I added something to the project description and then fails at a random time after without having changed the referenced stuff, then I will happily blame the tool. Especially when commenting out the reference, recompiling until error, and then uncommenting it fixes the issue. While this behavior doesn't necessitate an issue with the files per-se, there is only entity consuming it, so to me there is no distinction.