The problem#

I use Ubiquiti-branded network products for my switches and wireless access points and use the UniFi Network controller software to configure them.¹ I noticed the web interface wasn't working and checked the status the service and saw the logs showed it starting the service and no further information (and looking at the log files didn't show it generating any errors either):

$ sudo systemctl status unifi.service
[...]
Aug 16 14:32:13 host systemd[1]: Starting unifi.service - unifi...

Specifically, this was on version 9.0.114-28033-1 of the unifi Debian package provided by Ubiquiti.

The solution#

After multiple steps of investigation, I finallly got the Unifi service running by changing the ownership of all of its data files to the unifi user/group:

sudo chown -R unifi:unifi /usr/lib/unifi/{data,logs}

But this might not have been the best fix: the first thing it did when I opened the web interface was tell me there's no database and offered to restore from a backup. Luckily I don't change the settings much and it did regular automatic backups (in addition to a slightly older manual backup I had), so I didn't lose anything. Also, it's possible the data loss was due to some other troubleshooting step, not that one. As always, keep backups and before making a change consider making an extra copy of the directory before the change (or making a snapshot if you're using ZFS or another filesystem that supports them).

The details#

A buggy program#

Consider the following (contrived) program¹ which starts a background process to create a file and then waits while the background process is still running before checking to see if the file exists:

#!/bin/sh

# Make sure file doesn't exist.
rm -f file

# Create file in a background process.
touch file &
# While there is a touch process running...
while ps -C "touch" > /dev/null
do
    # ... wait one second for it to complete.
    sleep 1
done
# Check if file was created.
if [ -f file ]
then
    echo "Of course it worked."
else
    echo "Huh? File wasn't created."
    # Wait for background tasks to complete.
    wait
    if [ -f file ]
    then
        echo "Now it's there!"
    else
        echo "File never created."
    fi
fi

# Clean up.
rm -f file

Naturally, it will always output "Of course it worked.", right? Run it in a terminal yourself to confirm this. But I claimed this program is buggy; there's more going on.