A Weird Imagination

Unifi network controller failing to start

The problem#

I use Ubiquiti-branded network products for my switches and wireless access points and use the UniFi Network controller software to configure them.1 I noticed the web interface wasn't working and checked the status the service and saw the logs showed it starting the service and no further information (and looking at the log files didn't show it generating any errors either):

$ sudo systemctl status unifi.service
[...]
Aug 16 14:32:13 host systemd[1]: Starting unifi.service - unifi...

Specifically, this was on version 9.0.114-28033-1 of the unifi Debian package provided by Ubiquiti.

The solution#

After multiple steps of investigation, I finallly got the Unifi service running by changing the ownership of all of its data files to the unifi user/group:

sudo chown -R unifi:unifi /usr/lib/unifi/{data,logs}

But this might not have been the best fix: the first thing it did when I opened the web interface was tell me there's no database and offered to restore from a backup. Luckily I don't change the settings much and it did regular automatic backups (in addition to a slightly older manual backup I had), so I didn't lose anything. Also, it's possible the data loss was due to some other troubleshooting step, not that one. As always, keep backups and before making a change consider making an extra copy of the directory before the change (or making a snapshot if you're using ZFS or another filesystem that supports them).

The details#

Finding the logs#

Service logs#

systemctl status only shows a few lines of the service log by default. To view more of the log, you can use the --full option or get more control with journalctl:

$ sudo journalctl -u unifi.service -b

The -b option limits the logs to the current boot. It does show some errors that were relevant:

Aug 09 02:43:20 host systemd[1]: Starting unifi.service - unifi...
Aug 09 02:43:20 host unifi-network-service-helper[2203]: grep: /var/lib/unifi/system.properties: Permission denied
[...]
Aug 09 02:43:22 host java[2330]: 15:04:22,286 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[tasks_log] - openFile(logs/tasks.log,true) call failed. java.io.FileNotFoundException: logs/tasks.log (Permission denied)
[...]
Aug 09 02:58:21 host systemd[1]: unifi.service: start operation timed out. Terminating.
Aug 09 02:58:21 host systemd[1]: unifi.service: Failed with result 'timeout'.
Aug 09 02:58:21 host systemd[1]: Failed to start unifi.service - unifi.
Aug 09 02:58:21 host systemd[1]: unifi.service: Consumed 1min 45.767s CPU time, 543.2M memory peak.
Aug 09 02:58:22 host systemd[1]: unifi.service: Scheduled restart job, restart counter is at 1.
Aug 09 02:58:22 host systemd[1]: Starting unifi.service - unifi...

It continues to loop, outputting a similar log every 15 seconds. The full log even for one startup of the service is quite verbose so I've put the log for the first startup of the boot in the a footnote.2

Log files#

Services often provide more detailed logs than appear in journalctl in log files, which are usually under /var/log/ in a subdirectory for the program. unifi does in fact follow this convention, so they were exactly where I expected at /var/log/unifi/. Notably, I later found /usr/lib/unifi/ which has symlinks to the data directories it uses under /var/ including a logs link to that directory.

There's a lot of different log files there:

$ sudo ls /var/log/unifi
access.log    inform_request.log  server.log    tasks.log
access.log.1  migration.log       server.log.1  tasks.log.1
gc.log        mongod.log          server.log.2  unifi_crash.log
gc.log.0      mongod.log.1        server.log.3
gc.log.1      mongod.log.6.gz     startup.log
hotspot.log   remote              state.log

So I used ls -ltr (-l="long", -t="sort by time", -r="reverse sort") to see which ones had actually been written recently, which let me filter out some logs that hadn't been touched in the past day.

First attempt to fix permissions#

The journalctl output above highlighted two files as having permissions problems: /var/lib/unifi/system.properties and logs/tasks.log (apparently relative to /usr/lib/unifi/). Note the Java error FileNotFoundException has the detail of "Permission denied" indicating the file was found, it just couldn't be opened. Looking at those files with ls -l, I saw they somehow had been assigned to being owned by root instead of the unifi user, so I ran chown unifi:unifi3 on both of them. Then those errors went away. Success! Or not. The service didn't actually start, it just didn't show any error messages.

I had noticed that other files in those directories were also owned by root, but I wanted to investigate further what was going on before I just brute-force changed all of them to be owned by unifi.

MongoDB restarting#

Since unifi wasn't outputting any logs past the "starting..." message, I did some looking around to try figure out what it actually was doing. I looked at the process tree in htop4 to see what it was running and also looked at all of the files under the directories under /var pointed to by /usr/lib/unifi/ and noticed something odd: /usr/lib/unifi/run/mongod.pid was getting a fresh modified time every few seconds, suggesting it was starting and immediately stopping. In htop I could see that unifi sometimes had a MongoDB child process.

To investigate the child process, I looked under the /proc filesystem entry for the process based on the PID file, although I had to try a few times since it sometimes exited before the command completed:

# ll /proc/$(cat /var/run/unifi/mongod.pid)/fd
ls: cannot access '/proc/820934/fd': No such file or directory
# ll /proc/$(cat /var/run/unifi/mongod.pid)/fd
ls: cannot access '/proc/820934/fd': No such file or directory
# ll /proc/$(cat /var/run/unifi/mongod.pid)/fd
total 0
lr-x------ 1 unifi unifi 64 Aug 16 13:43 0 -> pipe:[109441095]|
l-wx------ 1 unifi unifi 64 Aug 16 13:43 1 -> pipe:[109441096]|
lrwx------ 1 unifi unifi 64 Aug 16 13:43 10 -> anon_inode:[timerfd]
lrwx------ 1 unifi unifi 64 Aug 16 13:43 11 -> socket:[109439211]=
lrwx------ 1 unifi unifi 64 Aug 16 13:43 12 -> socket:[109439212]=
l-wx------ 1 unifi unifi 64 Aug 16 13:43 2 -> pipe:[109441097]|
lr-x------ 1 unifi unifi 64 Aug 16 13:43 3 -> /dev/urandom
lr-x------ 1 unifi unifi 64 Aug 16 13:43 4 -> /dev/urandom
l-wx------ 1 unifi unifi 64 Aug 16 13:43 5 -> /var/log/unifi/mongod.log
lr-x------ 1 unifi unifi 64 Aug 16 13:43 6 -> /dev/urandom
lrwx------ 1 unifi unifi 64 Aug 16 13:43 7 -> /var/lib/unifi/db/mongod.lock
lrwx------ 1 unifi unifi 64 Aug 16 13:43 8 -> anon_inode:[eventfd]
lrwx------ 1 unifi unifi 64 Aug 16 13:43 9 -> anon_inode:[eventpoll]

Which really just told me that the mongodb.log file in the logs directory I already knew about was worth looking at:

# tail /var/log/unifi/mongod.log
2025-08-16T13:43:40.594-0700 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2025-08-16T13:43:40.594-0700 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2025-08-16T13:43:40.594-0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31619M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),cache_size=256M
2025-08-16T13:43:41.050-0700 E STORAGE  [initandlisten] WiredTiger error (13) [1754313101:50837][820958:0x7fe76b04da40], wiredtiger_open: __posix_open_file, 715: /usr/lib/unifi/data/db/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1754313101:50837][820958:0x7fe76b04da40], wiredtiger_open: __posix_open_file, 715: /usr/lib/unifi/data/db/WiredTiger.turtle: handle-open: open: Permission denied
2025-08-16T13:43:41.050-0700 E -        [initandlisten] Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 488
2025-08-16T13:43:41.051-0700 I STORAGE  [initandlisten] exception in initAndListen: Location28595: 13: Permission denied, terminating
2025-08-16T13:43:41.051-0700 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2025-08-16T13:43:41.051-0700 I NETWORK  [initandlisten] removing socket file: /usr/lib/unifi/run/mongodb-27117.sock
2025-08-16T13:43:41.051-0700 I CONTROL  [initandlisten] now exiting
2025-08-16T13:43:41.051-0700 I CONTROL  [initandlisten] shutting down with code:100

That log gives another permission denied error, this one on /usr/lib/unifi/data/db/WiredTiger.turtle. Once again, that file existed but was also owned by root not unifi.

At this point I just ran the chown -R unifi:unifi command above to change the ownership on all of the files.

Data loss#

Once I changed the file ownership, the service actually started. But it showed me a message saying it didn't have any data and offered to restore from a backup, listing automatic backups it had made in addition to letting me provide a manual backup. I'm not sure how the database got corrupted, but the backup was recent enough that I didn't think I had changed any settings since then, so I didn't make any attempt to recover the database before restoring the backup.

How did this happen?#

As all of the new files unifi created are owned by unifi, it's unclear how any of the files got marked as owned by root. Presumably either an old version did things differently or, more likely, an update somehow migrated from an older version wrong. But I assume I would have opened it immediately after running an update to confirm the update didn't break anything, so that doesn't quite make sense either.


  1. They also sell devices running this software that they brand as "Cloud Gateways" with varying capabilities. But I already have a desktop that I leave always-on and don't need any of the extra features those provide as they are more intended for larger commercial networks. 

  2. The full log:

    Aug 09 02:43:20 host systemd[1]: Starting unifi.service - unifi...
    Aug 09 02:43:20 host unifi-network-service-helper[2203]: grep: /var/lib/unifi/system.properties: Permission denied
    Aug 09 02:43:20 host unifi-network-service-helper[2219]: grep: /var/lib/unifi/system.properties: Permission denied
    Aug 09 02:43:20 host unifi-network-service-helper[2248]: grep: /var/lib/unifi/system.properties: Permission denied
    Aug 09 02:43:20 host unifi-network-service-helper[2268]: grep: /var/lib/unifi/system.properties: Permission denied
    Aug 09 02:43:20 host unifi[2289]: init complete...
    Aug 09 02:43:20 host unifi-network-service-helper[2289]: <13>Aug 09 02:43:20 unifi: init complete...
    Aug 09 02:43:20 host unifi-network-service-helper[2321]: <13>Aug 09 02:43:20 unifi: Skipping init-uos...
    Aug 09 02:43:20 host unifi[2321]: Skipping init-uos...
    Aug 09 02:43:22 host java[2330]: WARN Unable to load properties from '/usr/lib/unifi/data/system.properties' - /usr/lib/unifi/data/system.properties (Permission denied)
    Aug 09 02:43:22 host java[2330]: 15:04:21,985 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.14
    Aug 09 02:43:22 host java[2330]: 15:04:21,987 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Here is a list of configurators discovered as a service, by rank:
    Aug 09 02:43:22 host java[2330]: 15:04:21,987 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 -   org.springframework.boot.logging.logback.RootLogLevelConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:21,987 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - They will be invoked in order until ExecutionStatus.DO_NOT_INVOKE_NEXT_IF_ANY is returned.
    Aug 09 02:43:22 host java[2330]: 15:04:21,988 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Constructed configurator of type class org.springframework.boot.logging.logback.RootLogLevelConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:22,002 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - org.springframework.boot.logging.logback.RootLogLevelConfigurator.configure() call lasted 1 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
    Aug 09 02:43:22 host java[2330]: 15:04:22,002 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:22,002 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:22,007 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.scmo]
    Aug 09 02:43:22 host java[2330]: 15:04:22,007 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.scmo]
    Aug 09 02:43:22 host java[2330]: 15:04:22,007 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 5 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
    Aug 09 02:43:22 host java[2330]: 15:04:22,007 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:22,008 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
    Aug 09 02:43:22 host java[2330]: 15:04:22,012 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback-test.xml] at [jar:file:/usr/lib/unifi/lib/ace.jar!/logback-test.xml]
    Aug 09 02:43:22 host java[2330]: 15:04:22,020 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@1c852c0f - URL [jar:file:/usr/lib/unifi/lib/ace.jar!/logback-test.xml] is not of type file
    Aug 09 02:43:22 host java[2330]: 15:04:22,205 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [server_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,205 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,228 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@a37aefe - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,232 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,270 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[server_log] - Active log file name: logs/server.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,270 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[server_log] - File property is set to [logs/server.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,271 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [state_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,271 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,272 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5d99c6b5 - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,274 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,275 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[state_log] - Active log file name: logs/state.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,276 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[state_log] - File property is set to [logs/state.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,276 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [migration_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,276 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,277 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@266374ef - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,278 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,279 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[migration_log] - Active log file name: logs/migration.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,279 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[migration_log] - File property is set to [logs/migration.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,279 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [tasks_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,279 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,280 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@13b3d178 - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,280 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,281 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[tasks_log] - Active log file name: logs/tasks.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,281 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[tasks_log] - File property is set to [logs/tasks.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,286 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[tasks_log] - openFile(logs/tasks.log,true) call failed. java.io.FileNotFoundException: logs/tasks.log (Permission denied)
    Aug 09 02:43:22 host java[2330]:         at java.io.FileNotFoundException: logs/tasks.log (Permission denied)
    Aug 09 02:43:22 host java[2330]:         at         at java.base/java.io.FileOutputStream.open0(Native Method)
    Aug 09 02:43:22 host java[2330]:         at         at java.base/java.io.FileOutputStream.open(FileOutputStream.java:289)
    Aug 09 02:43:22 host java[2330]:         at         at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:230)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:26)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:206)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.FileAppender.start(FileAppender.java:126)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:104)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.model.processor.AppenderModelHandler.postHandle(AppenderModelHandler.java:84)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.model.processor.DefaultProcessor.secondPhaseTraverse(DefaultProcessor.java:257)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.model.processor.DefaultProcessor.secondPhaseTraverse(DefaultProcessor.java:253)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.model.processor.DefaultProcessor.traversalLoop(DefaultProcessor.java:90)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.model.processor.DefaultProcessor.process(DefaultProcessor.java:106)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.joran.GenericXMLConfigurator.processModel(GenericXMLConfigurator.java:208)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:170)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:122)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:65)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.util.DefaultJoranConfigurator.configureByResource(DefaultJoranConfigurator.java:68)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.util.DefaultJoranConfigurator.configure(DefaultJoranConfigurator.java:35)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.util.ContextInitializer.invokeConfigure(ContextInitializer.java:128)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:103)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:66)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.spi.LogbackServiceProvider.initializeLoggerContext(LogbackServiceProvider.java:52)
    Aug 09 02:43:22 host java[2330]:         at         at ch.qos.logback.classic.spi.LogbackServiceProvider.initialize(LogbackServiceProvider.java:41)
    Aug 09 02:43:22 host java[2330]:         at         at org.slf4j.LoggerFactory.bind(LoggerFactory.java:199)
    Aug 09 02:43:22 host java[2330]:         at         at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:186)
    Aug 09 02:43:22 host java[2330]:         at         at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:496)
    Aug 09 02:43:22 host java[2330]:         at         at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:482)
    Aug 09 02:43:22 host java[2330]:         at         at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:431)
    Aug 09 02:43:22 host java[2330]:         at         at com.ubnt.service.system.returnclass.õ00000(Unknown Source)
    Aug 09 02:43:22 host java[2330]:         at         at com.ubnt.service.system.returnclass.ø00000(Unknown Source)
    Aug 09 02:43:22 host java[2330]:         at         at com.ubnt.service.system.returnclass.<clinit>(Unknown Source)
    Aug 09 02:43:22 host java[2330]:         at         at com.ubnt.ace.Launcher.<clinit>(Unknown Source)
    Aug 09 02:43:22 host java[2330]: 15:04:22,286 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [hotspot_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,286 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,287 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@6a57ae10 - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,287 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,288 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[hotspot_log] - Active log file name: logs/hotspot.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,288 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[hotspot_log] - File property is set to [logs/hotspot.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,288 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [inform_request_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,288 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,288 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@766653e6 - No compression will be used
    Aug 09 02:43:22 host java[2330]: 15:04:22,290 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,293 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[inform_request_log] - Active log file name: logs/inform_request.log
    Aug 09 02:43:22 host java[2330]: 15:04:22,293 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[inform_request_log] - File property is set to [logs/inform_request.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,293 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [startup_log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,293 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.FileAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,299 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,300 |-INFO in ch.qos.logback.core.FileAppender[startup_log] - File property is set to [logs/startup.log]
    Aug 09 02:43:22 host java[2330]: 15:04:22,300 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [InMemoryAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,300 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [com.ubnt.ace.logs.InMemoryAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,302 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
    Aug 09 02:43:22 host java[2330]: 15:04:22,303 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [AnalyticsAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,303 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [com.ubnt.service.trace.logerror.AnalyticsAppender]
    Aug 09 02:43:22 host java[2330]: 15:04:22,645 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [java] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [javax] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [javax.jmdns] to OFF
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [sun] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [httpclient.wire] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [net.schmizz] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.codahale] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.tomcat] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.commons] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.catalina] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.hibernate.validator] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.mongodb] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.mongodb] to ERROR
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.springframework] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [de.javawi.jstun] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.ubnt] to INFO
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.ubiquiti] to INFO
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [com.amazonaws.internal] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [springfox] to WARN
    Aug 09 02:43:22 host java[2330]: 15:04:22,646 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [state_log] to Logger[state]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [migration_log] to Logger[migration]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [tasks] to DEBUG
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [tasks] to false
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [tasks_log] to Logger[tasks]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [hotspot_log] to Logger[guest]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [inform_request] to false
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [inform_request_log] to Logger[inform_request]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [startup_log] to Logger[startup]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [server_log] to Logger[ROOT]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [InMemoryAppender] to Logger[ROOT]
    Aug 09 02:43:22 host java[2330]: 15:04:22,647 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [AnalyticsAppender] to Logger[ROOT]
    Aug 09 02:43:22 host java[2330]: 15:04:22,648 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@4e07b95f - End of configuration.
    Aug 09 02:43:22 host java[2330]: 15:04:22,650 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@28b46423 - Registering current configuration as safe fallback point
    Aug 09 02:43:22 host java[2330]: 15:04:22,650 |-INFO in ch.qos.logback.classic.util.ContextInitializer@62833051 - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 642 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
    Aug 09 02:58:21 host systemd[1]: unifi.service: start operation timed out. Terminating.
    Aug 09 02:58:21 host systemd[1]: unifi.service: Failed with result 'timeout'.
    Aug 09 02:58:21 host systemd[1]: Failed to start unifi.service - unifi.
    Aug 09 02:58:21 host systemd[1]: unifi.service: Consumed 1min 45.767s CPU time, 543.2M memory peak.
    Aug 09 02:58:22 host systemd[1]: unifi.service: Scheduled restart job, restart counter is at 1.
    Aug 09 02:58:22 host systemd[1]: Starting unifi.service - unifi...
    

  3. Since these directories are set to be not readable by most users, it was easier to just use sudo -i to get a shell signed in as root instead of using sudo for every command and not being able to tab-complete filenames because my normal user account couldn't see the files. 

  4. pstree can also be used for this. See this previous blog post

Comments

Have something to add? Post a comment by sending an email to comments@aweirdimagination.net. You may use Markdown for formatting.

There are no comments yet.