The problem#

I use Ubiquiti-branded network products for my switches and wireless access points and use the UniFi Network controller software to configure them.¹ I noticed the web interface wasn't working and checked the status the service and saw the logs showed it starting the service and no further information (and looking at the log files didn't show it generating any errors either):

$ sudo systemctl status unifi.service
[...]
Aug 16 14:32:13 host systemd[1]: Starting unifi.service - unifi...

Specifically, this was on version 9.0.114-28033-1 of the unifi Debian package provided by Ubiquiti.

The solution#

After multiple steps of investigation, I finallly got the Unifi service running by changing the ownership of all of its data files to the unifi user/group:

sudo chown -R unifi:unifi /usr/lib/unifi/{data,logs}

But this might not have been the best fix: the first thing it did when I opened the web interface was tell me there's no database and offered to restore from a backup. Luckily I don't change the settings much and it did regular automatic backups (in addition to a slightly older manual backup I had), so I didn't lose anything. Also, it's possible the data loss was due to some other troubleshooting step, not that one. As always, keep backups and before making a change consider making an extra copy of the directory before the change (or making a snapshot if you're using ZFS or another filesystem that supports them).

The details#

The problem#

I had recently done an apt upgrade that included upgrading ZFS and noticed zpool status showed a weird "(non-allocating)" message, which seemed concerning:

$ zpool status
  pool: tank
 state: ONLINE
config:

    NAME         STATE     READ WRITE CKSUM
    tank         ONLINE       0     0     0
      mirror-0   ONLINE       0     0     0
        ata-***  ONLINE       0     0     0  (non-allocating)
        ata-***  ONLINE       0     0     0  (non-allocating)

errors: No known data errors

The solution#

This forum thread suggested the error may be due to a version mismatch between the ZFS tools and the kernel module. I confirmed there was a mismatch:

$ zpool --version
zfs-2.2.3-2
zfs-kmod-2.1.14-1

The easy way to load the new version of a kernel module after an update is to reboot the computer. But if you don't want to do that, here's the general outline of the commands I ran to unload and reload ZFS (run as root):

# Stop using ZFS
$ zfs umount -a
$ zpool export tank
$ service zfs-zed stop
# Remove modules
$ rmmod zfs
$ rmmod spl
# will show error: rmmod: ERROR: Module spl is in use by: ...
# repeatedly rmmod dependencies until spl is removed.

# Reload ZFS
$ modprobe zfs
$ service zfs-zed start
$ zpool import tank

The details#