Monday, January 25

debugging fail2ban

Trying to document a basic fail2ban install on CentOS 7.

There’s an /etc/fail2ban/jail.conf. You’re not supposed to edit this directly. According to official docs, my understanding of what I should do is create a jail.local that looks something like this:

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

[sshd]
enabled = true

…where those values will override things already defined in jail.conf, which contains a whole bunch of defaults. This doesn’t quite seem to work—fail2ban starts, logs things, and reports that it’s banning IPs, but as far as I can tell, the iptables firewall I expect it to be configuring isn’t touched, and bans are definitely not happening.

Whereas if I do:

$ sudo cp jail.conf jail.local

…then add enabled = true to the [sshd] section, and restart fail2ban:

$ sudo systemctl restart fail2ban

…then iptables -L shows me more or less what I’d expect.

I don’t think my mental model of how these config files fit together is correct. What’s different between the two scenarios?

I wonder if you can get fail2ban-client to dump configuration values, and it turns out that you can:

$ sudo fail2ban-client -d

With this it’s easy enough to see what changes between the two scenarios:

$ sudo fail2ban-client -d > one
# switch from sparse jail.local to full copy of jail.conf
$ sudo fail2ban-client -d > two
$ diff -u one two > jails.diff

It’d be ugly to inline here, but here’s jails.diff. As you can tell, a bunch of stuff referring to firewallcmd-ipset is replaced by stuff referring to iptables-multiport. That makes some intuitive sense—firewalld, not iptables, is standard on CentOS. My question is still ok, so why are the configuration values differing? Does some section of jail.conf not get executed if jail.local is present? Or am I just totally wrong about how this fits together?

To be continued!

Continued: Ok, so on first inspection, the fail2ban configuration stuff appears to be an impenetrable thicket of everything-lives-somewhere-else OO spaghetti. Eventually, however, I grep my way to realizing that /etc/fail2ban/jail.d/00-firewalld.conf looks like so:

[root@centos7-fail2ban fail2ban]# cat jail.d/00-firewalld.conf 
# This file is part of the fail2ban-firewalld package to configure the use of
# the firewalld actions as the default actions.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not use firewalld
[DEFAULT]
banaction = firewallcmd-ipset

And the documented execution order of config files turns out to go:

  1. jail.conf
  2. jail.d/*.conf (in alphabetical order)
  3. jail.local
  4. jail.d/*.local (in alphabetical order).

…which means that unless jail.local overrides it, the jail.d/00-firewalld.conf will set banaction to firewallcmd-ipset. The fail2ban-firewalld package seems to be a hard dependency of fail2ban, so I have a bit more puzzling to do, but in short I was being an idiot.

I land on this version of jail.local:

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

# Override firewalld banning:
banaction = iptables-multiport

[sshd]
enabled = true

In sysadmin-land, you’re very often looking for one of two moments:

  1. I am being an idiot.
  2. Someone else is wrong.

Always bet on #1, if you have to bet.