OpenBSD and the rest have a limits file where you can set RAM limits per user and sometimes per process, so is not a big issue.
On GNU/Linux and the rest not supporting dynamic swap files, you can swap into anything ensembling a file, even into virtual disk images.
Also set up ZRAM as soon as possible. 1/3 of the physical RAM for ZRAM it's perfect, it will almost double your effective RAM size with ease.
I think zswap is the better option because it's not a fixed RAM storage, it merely compresses pages in RAM up to a variable limit and then writes to swap space when needed, which is more efficient.
It worked very well with my preceding laptop limited to 4GB of RAM.
You can do this with cgroups but you aren't allowed to use cgroups if you use systemd, because it messes up systemd.
systemd is the primary user of cgroups. Your comment doesn't make sense.
systemd allows setting cgroup memory limits.