I had this problem at a previous job. I spent almost all of my time taking care of the behind the scenes administrative work (scheduling meetings, making sure that people had the information they needed to come into the meetings prepared, etc.). However, when performance review came around I was told that the only thing that they cared about was that I hadn't completed many story points because I was too busy keeping things from falling apart.

So I stopped doing all the administrative work and focused on just completing story points. A week or two later my manager asks the team "how come all of our meetings are falling apart now? We get into a meeting and no one knows what's going on."