> I also enabled UFW (which I should have done ages ago)
I disrecommend UFW.
firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can.
firewall-cmd --persistent --set-default-zone=block
firewall-cmd --persistent --zone=block --add-service=ssh
firewall-cmd --persistent --zone=block --add-service=https
firewall-cmd --persistent --zone=block --add-port=80/tcp
firewall-cmd --reload
Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports.
Newer versions of firewalld gives an easy way to configure this via StrictForwardPorts=yes in /etc/firewalld/firewalld.conf.
If you can, do not expose ports like this 8080:8080, but do this "192.168.0.1:8080:8080" so its bound to a private IP. Then use any old method expose only what you want to the world.
In my own use I have 10.0.10.11 on the vm that I host docker stuff. It doesn't even have its own public IP meaning I could actually expose to 0.0.0.0 if I wanted to but things might change in the future so it's a precaution. That IP is only accessible via wireguard and by the other machines that share the same subnet so reverse proxying with caddy on a public IP is super easy.
It's really a trap. I'm surprised they never changed the default to 127.0.0.1 instead of 0.0.0.0. So you would need to explicitly specify it, if you want to bind to all interfaces.
The reason is convenience. There would be a lot more friction if they didn't do it like this for everything other than local development.
Docker also has more traps and not quite as obvious as this. For example, it can change the private IP block its using without telling you. I got hit by this once due to a clash with a private block I was using for some other purpose. There's a way to fix it in the config but it won't affect already created containers.
By the way. While we're here. A public service announcement. You probably do NOT need the userland-proxy and can disable it.
/etc/docker/daemon.json
{ "userland-proxy": false }
Is there a guide that lists some common options / gotchas in Docker like this?
Some quick searching yields generic advice about keeping everything updated or running in rootless mode.
Not that I'm aware of. Sorry. Here's one my daemon.json files though. It tames the log file size and sets its format. And fixes the IP block so it won't change like I mentioned above.
Yup the regular "8080:8080" bind resulted in a ransom note in my database on day 1. Bound it to localhost only now.
I had the same experience (postgres/postgres on default port). It took me a few days to find out, because the affected database was periodically re-built from another source. I just noticed that for some periods the queries failed until the next rebuild.
Yea plenty of bots scouting the internet for these kinda vulnerabilities, good learning experience, wont happen again :D
one thing I always forget about, is that you have a whole network of 127.0.0.0/8 , not just one IP.
So you can create multiple addresses with multiple separate "domains" mapped statically in /etc/hosts, and allow multiple apps to listen on "the same" port without conflicts.
Unlike IPv6 localhost that's just the [::1] address. I'm not sure if you can abuse IPv4 in IPv6 to do the same
I never thought of using localhost like that, I'm surprised that works actually. Typically, if you want a private /8 you would use 10.0.0.0/8 but the standard 192.168.0.0/16 gives you a lot of address space ( 255^2 - 2 IPs (iirc) ) too.
..actually this is very weird. Are you saying you can bind to 127.0.0.2:80 without adding a virtual IP to the NIC? So the concept of "localhost" is really an entire class A network? That sounds like a network stack bug to me heh.
edit: yeah my route table on osx confirms it. very strange (at least to me)
That was deliberate. Works on Linux and Windows as well. I think this is the current RFC: https://datatracker.ietf.org/doc/html/rfc5735
You can do:
python3 -m http.server -b 127.0.0.1 8080
python3 -m http.server -b 127.0.0.2 8080
python3 -m http.server -b 127.0.0.3 8080
and all will be available.
Private network ranges don't really have the same purpose, they can be routed, you have to always consider conflicts and so on. But here with 127/8 you are in your own world and you don't worry about anything. You can also do tests where you need to expose more than 65k ports :)
You have to also remember these are things established likely before even DNS was a thing, IP space was considered so big that anyone could have a huge chunk of it, and it was mostly managed manually.
I didn't really know the mechanism of how this worked but if you check your resolv file you might find that the nameserver IP for your localhost is 127.0.0.53 . It is so in recent Linux distros. (Probably a systemd thing)
I keep reading comments by podman fans asking to drop Docker and yet every time I have tried to use podman it failed on me miserably. IMHO it would be better if podman was not designed and sold as a docker drop in replacement but its own thing.
That sucks, I never had any problem running a Dockerfile in podman. I don't know what I do differently, but I would as a principle filter out any container that messes with stuff like docker in docker. Podman doesn't need these kind of shenegians.
Also the Docker Compose tool is a well-know exception to the compatibility story. (There is some unofficial podman compose tool, but that is not feature complete and quadlets are better anyway.)
I agree with approaching podman as its own thing though. Yes, you can build a Dockerfile, but buildah lets you build an efficient OCI image from scratch without needing root. For those interested, this document¹ explains how buildah compares to podman and docker.
1. https://github.com/containers/buildah/tree/main/docs/contain...
There's a real dearth of blog posts explaining how to use quadlets for the local dev experience, and actually most guides I've seen seem to recommend using podman/Docker compose. Do you use quadlets for local dev and testing?
Quadlets aren't what I'd personally use for local dev. They are good for running a local headless persistent service. So I wouldn't use it for your service-under-test but they can be a good fit for supporting dev tools like a local package registry, proxy or VPN gateway.
The docs you need for quadlets are basically here: https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
The one gotcha I can think of not mentioned there is that if you run it as a non-root user and want it to run without logging in as that user, you need to: `sudo loginctl enable-linger $USER`.
If you don't vibe with quadlets, it's equally fine to do a normal systemd .service file with `ExecStart=podman run ...`, which quadlets are just convenience sugar for. I'd start there and then return to quadlets if/when you find that becomes too messy. Don't add new abstraction layers just because you can if they don't help.
If you have a more complex service consisting of multiple containers you want to schedule as a single unit, it's also totally fine to combine systemd and compose by having `ExecStart=podman compose up ...`.
Do you want it to run silently in the background with control over autorestarts and log to system journal? Quadlets/systemd.
Do you want to have multiple containers scheduled together (or just prefer it)? Compose.
Do you want to manually invoke it and have the output in a terminal by default? CLI run or compose.
A side-effect of running rootless and daemonless is that containers stop on user log out, and I can't believe how all this is to be expected for a newcomer to parse. Because I thought the whole point of containers in production was for them to keep running when you log out.
Of course, when you think about it, nobody expects a command to just survive logging out, but coming from docker, you still have that expectation. And I wonder, am I supposed to be running this on a tmux like the old days? No, you need to do a bunch of systemd/linger/stuff. So being that we are already in systemd land, you keep searching and end up in quadlets, which are a new-ish thing with (last I checked) bad docs, replacing whatever was used before (which has good docs). Docs, being said, that give k8s ptsd. Quadlet, podlet and pods.
It seems that when podman deviates from docker, it does in the least ergonomic way possible. Or maybe I have been lobotomized by years and years of using docker, or maybe my patience threshold is very low nowadays. But this has been my experience. I felt very stupid when I deployed something and it stopped after 5 minutes. I was ready to use podman, because it worked locally. And then it failed in production. Thanks no.
A side-effect of running rootless and daemonless is that containers stop on user log out
This is not a side effect of running rootless, it's a side effect of running systemd (or rather, systemd-logind).
Yes, but I want it to only apply to podman, not any running task.
You should compare it imho to ssh. If you break your connection, your session is gone. So if you only want certain parts of your session survive, which ones should? Because maybe your container depends on avahi on the host, or cups, or...?
Just a random thought, but if you can create a user on the host that just has the most minimal set of systemd services enabled your container needs, you could apply it to that user.
But still, on a server that wouldn't make much sense imho, as the default user is usually the service user having a minimal set of services enabled. On a desktop, your default user is logged in anyways. So I think this isn't a real problem tbh.
I just use Podman's Kubernetes yaml as a compose substitute when running everything locally. This way it is fairly similar to production. Docker compose seems very niche to me now.
podman is not a drop-in replacement for Docker. You can replace it with podman but expect to encounter minor inconsistencies and some major differences, especially if you use Docker Compose or you want to use podman in rootless mode. It's far from just being a matter of `alias docker=podman`.
The only non-deprecated way of having your Compose services restart automatically is with Quadlet files which are systemd unit files with extra options specific to containers. You need to manually translate your docker-compose.yml into one or more Quadlet files. Documentation for those leaves a lot to be desired too, it's just one huge itemized man page.
This affects podman too.
Not if you run it in rootless mode, which is more of a first class citizen in Podman compared to Docker.
> Not if you run it in rootless mode.
Same as for docker, yes?
https://docs.docker.com/engine/security/rootless/
Rootless exists in Docker, yes, but as OP said, it's not first-class. The setup process is clunky, things break more often. In podman it just works, and podman is leading with features like quadlets, which make docker services just services like any other.
No one wants, nor asked for, quadlets.
nope. You should look at https://docs.docker.com/engine/network/
Networking is just better in podman.
> nope. You should look at https://docs.docker.com/engine/network/
That page does not address rootless Docker, which can be installed (not just run) without root, so it would not have the ability to clobber firewall rules.
Nothing in the article talked about podman or podman vs docker. Umami with its NexJS and React CVE vulnerability was the issue. BTW, I use Docker because it works extremely well and because there is so much astroturfing from the podman gang I wouldn't use it if my life depended on it until that shit calms down.
In docker, simply clearly define the interface (ip) and port. It can be 0.0.0.0:80 for example. No bypass happens.
No, I'm happy with Docker, Docker works very well.
it doesn't matter what netfilter frontend you use if you allow outbound connections from any binary.
In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries.
This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers.
On the other hand, removing exec permissions to /tmp, /var/tmp and /dev/shm is also useful.
> On the other hand, removing exec permissions to /tmp, /var/tmp and /dev/shm is also useful.
Sadly that's more of a duck tape or plaster, because any serious malware will launch their scripts with the proper '/bin/bash /path/to/dropped/payload' invocation. A non-exec mount works reasonably well only against actual binaries dropped into the paths, because it's much less common to launch them with the less known '/bin/ld.so /path/to/my/binary' stanza.
I've also at one time suggested that Debian installer should support configuring a read-only mount for /tmp, but got rejected. Too many packaging scripts depend on being able to run their various steps from /tmp (or more correctly, $TMPDIR).
I agree. That's why I said that it's also useful. It won't work in all scenarios, but in most of the cryptomining attacks, files dropped to /tmp are binaries.
It is really unfortunate that a lot of services expect to have write access to their config files, so you can tweak settings with a web UI.
If this weren't the case, plenty of containers could probably have a fully read-only filesystem.
Is there an automated way of doing this?
Two paths:
- Configuration management (ansible, salt, chef, puppet)
- Preconfigured images (NixOS, packer, Guix, atomic stuffs)
For a one-off: pssh
restricting outbound connections by binary: OpenSnitch .
You can also restrict outbound connections to cryptomining pools and malicious IPs. For example by using IOCs from VirusTotal or urlhaus.bazaar.ch
Wasn’t there that npm malware thing a while ago that trashed your home folder if it couldn’t phone home?
I strongly disagree, firewall-cmd is way too complicated. I mean it's probably fine if your main job is being a firewall administrator, but for us who just need to punch a hole in a firewall as a tiny necessary prerequisite for what we actually want to do, it's just too much.
On ufw systems, I know what to do: if a port I need open is blocked, I run 'ufw allow'. It's dead simple, even I can remember it. And if I don't, I run 'ufw --help' and it tells me to run 'ufw allow'.
Firewall-cmd though? Any time I need to punch a hole in the firewall on a Fedora system, I spend way too long reading the extremely over-complicated output of 'firewall-cmd --help' and the monstrous man page, before I eventually give up and run 'sudo systemctl disable --now firewalld'. This has happened multiple times.
If firewalld/firewall-cmd works for you, great. But I think it's an absolutely terrible solution for anyone whose needs are, "default deny all incoming traffic, open these specific ports, on this computer". And it's wild that it's the default firewall on Fedora Workstation.
Personally I find just using nftables.conf straightforward enough that I don't really understand the need for anything additional. With iptables, it was painful, but iptables has been deprecated for a while now.
Same here, I'm surprised most linux users I know like to install firewalld, UFW, or some other overlaying firewall rather than just editing the nftables config directly. It's not very difficult, although I've never really dug deep into the weeds of iptables. I suspect many people who have used iptables long ago in the past assume nftables is samilar and avoid interacting with it directly out of habit.
With nftables you need to learn a lot before you cam be partially sure of wbat you do.
With ufw gui you need a single checkbox - block incoming connections.
Not sure what you find difficult about it, but I just took the "workstation" config from the gentoo wiki and used it on my laptop.
Perhaps if you're doing more complicated things like bridging interfaces or rerouting traffic it would be more difficult to use than the alternatives, but for a simple whitelist it's extremely easy to configure and modify.
> Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports.
This sounds like great news. I followed some of the open issues about this on GitHub and it never really got a satisfactory fix. I found some previous threads on this "StrictForwardPorts": https://news.ycombinator.com/item?id=42603136.
Hetzner has a free firewall service outside of your machine. You can use that as the first line of defence.
It's a good idea. At OCI, I have the VCN firewall enabled and ufw firewall enabled within my VPS's.
The problem with Hetzner's firewall service is it nukes network performance especially on ipv6.
It also killed my docker networking, so portainer stopped working.
That's what I use. Is it enough? Or should I also install a firewall on my machine?
Do both. Using provider's firewall service adds another level of defence. But hiccups may occur and firewall rules may briefly disappear (sync issues, upgrades, vm mobility issues) and you services then may become exposed. Happened to me in the past, were "lucky" enough so no damage was taken.
Security in layers, I'd do both.
The problem with firewalld is that it has the worst UX of any program I know. Completely unintuitive options, the program itself doesn’t provide any useful help or hint if you get anything wrong and the documentation is so awful you have to consult the Red Hat manuals that have thankfully been written for those companies that pay thousands per month in support.
It’s not like iptables was any better, but it was more intuitive as it spoke about IPs and ports, not high-level arbitrary constructs such as zones and services defined in some XML file. And since firewalld uses iptables/nftables underneath, I wonder why do I need a worse leaky abstraction on top of what I already know.
I truly hate firewalld.
Coming from FreeBSD and pf, all Linux firewalls I’ve tried feels clunky _at best_ UX-wise.
I’d love a Linux firewall configured with a sane config file and I think BSD really nailed it. It’s easy to configure and still human readable, even for more advanced firewall gateway setups with many interfaces/zones.
A have no doubt that Linux can do all the same stuff feature-wise, but oh god the UX :/
I completely agree.
I have been using for many decades both Linux and FreeBSD, on many kinds of computers.
When comparing Linux with FreeBSD, I probably do not find anything more annoying on Linux than its networking configuration tools.
While I am using Linux on my laptops and desktops and on some servers with computational purposes, on the servers that host networking services I much prefer FreeBSD, for the ease of administration.
nftables is configured like that https://wiki.nftables.org/wiki-nftables/index.php/Simple_rul...
Have you tried nftables? It is so much nicer than iptables.
Yeah, I'm already using nftables and I agree that it's better than eg. iptables (or the numerous frontends for iptables) and probably the best bet we have at this point - but honestly, it's still far from the UX I get from pf - unfortunately :/
Hate it as well. Why should I bother learning about zones and abstract away ports, adresses, interfaces etc. only to find out pretty soon that my baremetal server actually always needs fine grained rules at least from the firewalld's point of view.
Why people use these abstractions over nftables? Use nftables, it's easy to learn, effective and if you know it you know everything about firewalls.
UFW and Firewall-CMD both just use iptables in that context though. The real upgrade is in switching to nftables. I know I'm going to need to learn eBpf as the next step too, but for now nftables is readable and easy to grok especially after you rip out the iptables stuff, but technically nftables is still using netfilter.
And ufw supports nftables btw. I think the real lesson is write your own firewalls and make them non-permissive - then just template that shit with CaC.
Do either of them work for container-to-container traffic?
Imagine container A which exposes tightly-coupled services X and Y. Container B should be able to access only X, container C should be able to accesd only Y.
For some reason there just isn't a convenient way to do this with Docker or Podman. Last time I looked into it, it required having to manually juggle the IP addressed assigned to the container and having the service explicitly bind to it - which is just needlessly complicated. Can firewalls solve this?
I can't answer your question about Docker or Podman, but in Kubernetes there is the NetworkPolicy API which is designed for exactly this use-case. I'm sure it uses Linux native tooling (iptables, nftables, etc) under the hood, so it's at least within the real of feasible that those tools can be used for this purpose.
also docker bypasses ufw
You might be interested in ufw-docker: https://github.com/chaifeng/ufw-docker
I've not used firewalld but I have used ufw on my desktops and servers going back to 2006 and can guarantee I have no plans to change from it.
One of those rare HN comments that's just pure gold.
The don't use ufw but use firewalld comment? I disagree. I've used ufw since 2006 and have no plans to change. Works great.
I’ll just mention Foomuuri here. Its bit of a spiritual successor to shorewall and has firewalld emulation to work with tools compatible with firewalld
Thanks! Would be cool to have it packaged for alpine since firewalld requires D-Bus. There is awall but that's still on iptables and IMO at bit clunky to set up.
Foomuuri is ALMOST there.
I mean there are some payload over payload like GRE VPE/VXLAN/VLAN or IPSec that needs to be written in raw nft if using Foomuuni but it works!.
But I love the Shorewall approach and your configuration gracefully encapsulated Shorewall mechanic.
Disclaimer: I maintain vim-syntax-nftables syntax highlighter repo at Github.
Does it have a gui?
..backed by xml?