Linux being... Linux, it's not easy to use, but it can do what you want.
1. Use `choom` to give your Firefox PIDs a score of +1000, so they always get reaped first
2. Use systemd to create a Control Group to limit firefox and reap it first (https://dev.to/msugakov/taking-firefox-memory-usage-under-co...)
3. Enable vm.oom_kill_allocating_task to kill the task that asked for too much memory
4. Nuclear option: change how all overcommiting works (https://www.kernel.org/doc/html/v5.1/vm/overcommit-accountin...)
3. vm.oom_kill_allocating_task is a footgun. It kills the last task that asked for memory and it could be any random task in the system.
4. disabling overcommit is another footgun, it makes malloc fail long before the memory is exhausted. See for a detailed explanation https://unix.stackexchange.com/a/797888/1027