There is no FOSS modem. The baseband is a separate computer operating on a lower level than the OS.

Your provider can run arbitrary code there.

I thought the PinePhone community had succeeded in making custom firmware for the modem in the PinePhone, though I'm not sure of the legality of actually using it. Plus both PinePhone and Librem 5 had a killswitch to disconnect the modem at least.