The problem#

I will often make copies of important files onto multiple devices, and then later make backups of all of those devices onto the same drive. At which point, I now have multiple redundant copies of those files within my backup. Tools like rdfind, fdupes, and jdupes exist to deal with the general problem of searching a collection of files for duplicates efficiently, but none of them support only checking if files are identical if their filenames and/or paths match, so they end up doing a lot of extra work in this case.

The solution#

Download the script I wrote, hardlink-dups-by-name.sh and run it as follows:

hardlink-dups-by-name.sh a_backup/ another_backup/

Then all files like a_backup/some/path that are identical to the corresponding file another_backup/some/path will get hard-linked together so there will only be one copy of the data taking up space.

The details#

The problem#

I've been playing Codenames online a lot lately (using my fork of codenames.plus), and a friend suggested it might be fun to have themed word lists. Specifically, they suggested Star Trek as a theme as it's a fandom that's fairly widely known. They left it up to me to figure out what should be in a Star Trek themed word list.

The solution#

If you just want to play Codenames with the list, go to my Codenames web app and select one or both of the Star Trek card packs. If you just want the word lists, you can download the Star Trek: The Next Generation words and the Star Trek: Deep Space 9 words.

To generate a word list yourself (I used this source for the Star Trek scripts), you will need a common words list like en_50k.txt which I mentioned in my previous post on anagram games, and then pipe the corpus through the following script (which you will likely have to modify for the idiosyncrasies of your data):

#!/bin/bash
set -euo pipefail

NUM_COMMON=2000 # Filter out the most common 2000 words
COMMON_WORDS="$(mktemp)"
<en_50k.txt head "-$NUM_COMMON" | cut -d' ' -f1 |\
    sort | tr '[:lower:]' '[:upper:]' >"$COMMON_WORDS"

# Select only dialogue lines (in Star Trek scripts)
grep -aP '^\t\t\t[^\t]' |\
    # Split words
    tr ' .,:()\[\]!?;"/\t[:cntrl:]' '[\n*]' |\
    sed 's/--/\n/' |\
    # Strip whitespace
    sed 's/^\s\+//' | sed 's/\s\+$//' |\
    grep -av '^\s*$' |\
    # Strip quotes
    sed "s/^'//" | sed "s/'$//" |\
    # Filter out numbers
    grep -av '^[[:digit:]]*$' |\
    tr '[:lower:]' '[:upper:]' |\
    # Fix for contractions not being in wordlist
    sed "s/'\(S\|RE\|VE\|LL\|M\|D\)$//" |\
    grep -av "'T$" |\
    # Remove some more non-words
    grep -avF '-' |\
    grep -avF '&' |\
    # Count
    sort | uniq -c |\
    # Only keep words with >25 occurrences
    awk '{ if ($1 > 25) { print } }' |\
    # Remove common words
    join -v2 -22 -o 2.1,2.2 "$COMMON_WORDS" - |\
    # Sort most common words first
    sort -rn

rm "$COMMON_WORDS"

The output of the script will require some manual effort to decide which words really belong in the final list, but it's a good start.

The details#

Most desktop environments provide options for customizing keyboard shortcuts. In XFCE, there's settings panels for both for window manager shortcuts and application shortcuts. While the term "application shortcuts" suggests using them for launching applications, and many keyboards do have special keys for launching a music player or a calculator that I do have set up, I don't find myself using those much. I have buttons on my panel for applications that I launch often; if I'm going to be clicking away into a new application, I don't find clicking on the panel to be an additional inconvenience.

On the other hand, "application shortcuts" can be used for launching arbitrary scripts, including ones don't involve switching contexts.

Keys to use#

Many keyboards have extra keys intended for global commands labeled with various symbols. If you have them, you can be creative about what you want them to mean and even combine them with modifiers (Shift, Ctrl, etc.) to get more inputs. On the other hand, if you have a more traditional keyboard layout (which is likely the case on a laptop), your choices are more limited. To avoid confusion, it's generally best to use the Windows key (usually called the Super key in Linux) for global shortcuts as it is not usually used for anything else.

Shortcut ideas#

The problem#

I want my computer to act differently when I'm actively using it as opposed to away from. I almost always lock the screen when I step away from my computer, so I want to have the same signal do more than just start the screensaver.

The solution#

Save the follow script which is slightly modified from the example in the man page for xscreensaver-command as watch-xscreensaver.pl:

#!/usr/bin/perl

my $blanked = 0;
open (IN, "xscreensaver-command -watch |");
while (<IN>) {
    print;
    if (m/^(BLANK|LOCK)/) {
        if (!$blanked) {
            system "on-xscreensaver-lock";
            $blanked = 1;
        }
    } elsif (m/^UNBLANK/) {
        system "on-xscreensaver-unlock";
        $blanked = 0;
    }
}
if ($blanked) {
    system "on-xscreensaver-unlock";
}

Either call it from your ~/.xsessionrc file or just manually run from a terminal in your X session. I run it from a screen session so I can reattach to it and see the output:

screen -d -m -S xscreensaver-watch watch-xscreensaver.pl

My on-xscreensaver-lock and on-xscreensaver-unlock scripts are below and may be a good starting place, but yours will probably be different depending on your needs.

The details#

The problem#

Which window I have focused is a signal to the computer for the state I want it to be in. For instance, I normally leave my speaker muted so, for example, I don't accidentally play sound from a website with unexpected videos. But this means that when I do want sound, I need to manually unmute the sound, even though I've already told the computer that I want to watch Netflix, which always involves turning on the sound.

Of course, for the particular problem of unmuting the sound, adding a keyboard shortcut and rereading xkcd 1205: Is It Worth the Time? probably would have been a more appropriate solution. But I wanted a general solution to the problem.

The solution#

Download x11_watch_active_window.py. Then the following script will unmute the speakers if Netflix is focused:

#!/bin/sh
x11_watch_active_window.py | while read -r FocusApp
do
    if [ "Netflix - Google Chrome" = "$FocusApp" ]
    then
        echo Netflix is focused, unmuting.
        pactl set-sink-mute 0 0
    fi
done

The details#

The problem#

In last week's post, I discussed using cpulimit on multiple processes in the special case of web browsers, but I wanted a more general solution.

The solution#

cpulimit-all.sh is a wrapper around cpulimit which will call cpulimit many times to cover multiple processes of the same name and subprocesses.

Using that script, the follow is the equivalent of the script from last week to limit all browser processes to 10% CPU:

cpulimit-all.sh --limit=10 --max-depth=1 \
    -e firefox -e firefox-esr -e chromium -e chrome

But also, we can add a couple options to include any grandchild processes and check for new processes to limit every minute:

cpulimit-all.sh --limit=10 --max-depth=2 \
    -e firefox -e firefox-esr -e chromium -e chrome \
    --watch-interval=1m

The details#

The problem#

cpulimit is a useful utility for stopping a program from wasting CPU, but it only limits a single process. As all modern web browsers use process isolation, limiting just a single process doesn't do very much, we actually want to limit all of the browser processes.

The solution#

The following script will limit the CPU usage of all browser processes to $LIMIT percent CPU. Note that the limit is per process not total over all processes, so you may want to set it quite low to actually have an effect.

LIMIT=10 # Hard-code a limit of 10% CPU as an example.

# Kill child processes (stop limiting CPU) on script exit.
for sig in INT QUIT HUP TERM; do
  trap "
    pkill -P $$
    trap - $sig EXIT
    kill -s $sig "'"$$"' "$sig"
done
trap cleanup EXIT

# Find and limit all child processes of all browsers.
for name in firefox firefox-esr chromium chrome
do
    for ppid in $(pgrep "$name")
    do
        cpulimit --pid="$ppid" --limit="$LIMIT" &
        for pid in "$ppid" $(pgrep --parent "$ppid")
        do
            cpulimit --pid="$pid" --limit="$LIMIT" &
        done
    done
done

The details#

The problem#

My previous post got the video from my smartphone to show up as a camera device on my desktop, but for a video chat, we probably also want audio. So, now the question is: how to build GStreamer pipelines that will allow minimal-webrtc-gstreamer to use virtual microphone and speaker devices that I can point a voice/video chat application at, allowing me to use my smartphone's microphone and speaker for applications on my desktop.

The solution#

The following requires that you are using PulseAudio as your sound server and have downloaded minimal-webrtc-gstreamer:

pactl load-module module-null-sink sink_name=virtspk \
    sink_properties=device.description=Virtual_Speaker
pactl load-module module-null-sink sink_name=virtmic \
    sink_properties=device.description=Virtual_Microphone_Sink
pactl load-module module-remap-source \
    master=virtmic.monitor source_name=virtmic \
    source_properties=device.description=Virtual_Microphone
./minimal-webrtc-host.py\
    --url "https://apps.aweirdimagination.net/camera/"\
    --receiveAudioTo device=virtmic\
    --sendAudio "pulsesrc device=virtspk.monitor"\
    --sendVideo false --receiveVideo false

You can reset your PulseAudio configuration by killing PulseAudio:

pulseaudio -k

You can make the PulseAudio settings permanent by following these instructions to put them in your default.pa file.

The details#

The problem#

I want to make my smartphone's camera appear as an actual camera device on my desktop so any application (primarily Discord) can use it like it were a normal USB web cam.

My previous post introduced minimal-webrtc-gstreamer, which got as far as getting the video stream from any web browser into a GStreamer pipeline, which reduces the problem to outputting a GStreamer pipeline into a virtual web cam device.

The solution#

Download minimal-webrtc-gstreamer and install v4l2loopback. Then run

sudo modprobe v4l2loopback video_nr="42"\
    'card_label=virtcam'\
    exclusive_caps=1 max_buffers=2
./minimal-webrtc-host.py\
    --url "https://apps.aweirdimagination.net/camera/"\
    --receiveVideoTo /dev/video42\
    --sendAudio false

You can test by watching the stream with

gst-launch-1.0 v4l2src device=/dev/video42 ! autovideosink

Note that some applications, including the current desktop release of Discord may not support the virtual camera, showing a solid black square or failing to connect to it at all. It should work in the latest Chromium/Chrome browser, including for the Discord web app.

When done, remove the virtual camera device:

sudo modprobe -r v4l2loopback

The details#

The problem#

When writing a shell script that starts background jobs, sometimes running those jobs past the lifetime of the script doesn't make sense. (Of course, sometimes background jobs really should keeping going after the script completes, but that's not the case this post is concerned with.) In the case that either the background jobs are used to do some background computation relevant to the script or the script can conceptually be thought of as a collection of processes, it makes sense for killing the script to also kill any background jobs it started.

The solution#

At the start of the script, add

cleanup() {
    # kill all processes whose parent is this process
    pkill -P $$
}

for sig in INT QUIT HUP TERM; do
  trap "
    cleanup
    trap - $sig EXIT
    kill -s $sig "'"$$"' "$sig"
done
trap cleanup EXIT

If you really want to kill only jobs and not all child processes, use the kill_child_jobs() function from all.sh or look at the other versions in the kill-child-jobs repository.

The details#