The problem#

My personal desktop runs Debian Unstable ("Sid")¹. The nature of running a bleeding edge distro is that things break sometimes. I use Debian Testing/Stable or Ubuntu on my other machines to make my life easier, but I often want access to the latest version of some piece of software and running Debian Unstable is one way to do that. Admittedly, I also do it partially just because fixing things that break is a good way of learning how things work.

The most common kind of problem I run into is that upgrades are not straightforward. For their unstable distro, Debian doesn't make any promises about package dependencies not changing. This is less of a problem when there's an additional package that needs to be installed, but can be complicated when there's conflicts which require removing packages to get an upgrade to go through.

Recently I ran into an extreme version of this problem: trying to upgrade, it proposed uninstalling nearly everything I had installed. Worse, trying to resolve the issue, I got a scary sounding warning that I had uninstalled libssl3:

dpkg: libssl3:amd64: dependency problems, but removing anyway as you requested:
 [...]
 systemd depends on libssl3 (>= 3.0.0).
 sudo depends on libssl3 (>= 3.0.0).
 [...]

Both of those sound important.

The solution#

Luckily, it wasn't as bad as it sounded. Looking at the message, it turned out I had replaced libssl3 with libssl3t64. The latter of which is actually the exact same thing, although the package manager doesn't know that. The reason for the different package name is part of the Debian project to transition to 64-bit time_t, which is required to fix the Year 2038 problem. While on AMD64 and other 64-bit architectures, everything already uses 64-bit time_t, that's not true of all platforms that Debian supports. The way Debian handles ABI transitions like this is to rename the library packages with a suffix (t64 for this one) to ensure the old and new ABI don't get mixed accidentally. Since all of the architectures share the package names, the rename also happens on AMD64 even though there's actual change to match the rename on other platforms where the ABI did change.

Presumably the upgrade will be smoother when done between stable versions, but it really confused apt (which I usually use via wajig):

$ wajig install libssl-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libegl1 : Depends: libegl-mesa0 but it is not going to be installed
 libreoffice-core : Depends: libgstreamer-plugins-base1.0-0 (>= 1.0.0) but it is not going to be installed
                    Depends: libgstreamer1.0-0 (>= 1.4.0) but it is not going to be installed
                    Depends: liborcus-0.18-0 (>= 0.19.2) but it is not going to be installed
                    Depends: liborcus-parser-0.18-0 (>= 0.19.2) but it is not going to be installed
 wine-development : Depends: wine64-development (>= 8.21~repack-1) but it is not going to be installed or
                             wine32-development (>= 8.21~repack-1)
                    Depends: wine64-development (< 8.21~repack-1.1~) but it is not going to be installed or
                             wine32-development (< 8.21~repack-1.1~)
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

Yeah, no idea what libegl1, libreoffice-core, or wine-development have to do with upgrading libssl-dev, but apt was showing those same packages in the error messages no matter what I tried to upgrade and trying to upgrade those packages didn't work either. Luckily, aptitude was able to handle it somewhat better:

$ sudo aptitude install libssl-dev
The following packages will be upgraded:
  libssl-dev{b}
1 packages upgraded, 0 newly installed, 0 to remove and 1459 not upgraded.
Need to get 2,699 kB of archives. After unpacking 1,122 kB will be used.
The following packages have unmet dependencies:
 libssl-dev : Depends: libssl3t64 (= 3.2.1-3) but it is not going to be installed
The following actions will resolve these dependencies:

     Remove the following packages:
1)     libssl3 [3.1.4-2 (now)]
2)     libssl3:i386 [3.1.4-2 (now)]

     Install the following packages:
3)     libssl3t64 [3.2.1-3 (testing, unstable)]
4)     libssl3t64:i386 [3.2.1-3 (testing, unstable)]



Accept this solution? [Y/n/q/?] y
The following NEW packages will be installed:
  libssl3t64{a} libssl3t64:i386{a}
The following packages will be REMOVED:
  libssl3{a} libssl3:i386{a}
The following packages will be upgraded:
  libssl-dev
1 packages upgraded, 2 newly installed, 2 to remove and 1457 not upgraded.
Need to get 7,177 kB of archives. After unpacking 2,294 kB will be used.
Do you want to continue? [Y/n/?]

Getting the packages to upgrade involved a lot of calls to aptitude that looked like that: removing a list of libraries and a installing a matching list of new libraries whose names were identical to those removed except with t64 at the end.

The details#

The problem#

I wanted to find a specific Unicode character I had used somewhere in some previous blog post. But by the nature of not knowing exactly what character it was, I wasn't sure how to search for it.

The solution#

Instead, I wrote a script based on this post to simply list all of the characters appearing in any file in a given directory:

cat * | sed 's/./&\n/g' | sort -u

Although the output is small, to further reduce the noise, this version strips out the English letters, numbers, and common symbols:

cat * \
    | tr -d 'a-zA-Z0-9!@#$%^&*()_+=`~,./?;:"[]{}<>|\\'"'-" \
    | sed 's/./&\n/g' | sort -u

The details#

The problem#

For various reasons you might want graphical access to another computer, since some things can't be done over a text interface, including actually designing and troubleshooting what the graphical interface looks like. The other computer might be in a remote location across the internet, in a different room, or simply have a less convenient form factor like a tablet or television, so it's easier to use your desktop's keyboard, mouse, and monitor.

The solution#

The standard solution for this is VNC, specifically the x11vnc VNC server.

To keep a VNC server open to the current X11 session:

x11vnc -usepw -nevershared -forever -localhost -loop &
#... (run one or more graphical applications that block)
# When done, kill everything.
rkill $$

Then to connect to it, assuming the hostname is tablet and you're set up to connect to it via SSH:

$ vncviewer -via tablet -passwd ~/.vnc/tab-passwd localhost

This assumes you've created a ~/.vnc/passwd password file on the server by running

$ x11vnc -storepasswd

and entering something at the prompt from your favorite password generator. No need to save the password anywhere as the file itself is the actual password; just copy it to the client at ~/.vnc/tab-passwd to match the path used in the example above.

The details#

You know, like "lions and tigers and bears, oh my"… okay, not funny, moving on…

The problem#

There's a lot of different ways to transmit streams of bytes between applications on the same host or different hosts with various reasons you might want to use each one. And sometimes the two endpoints might disagree on which one they want to be using.

The solution#

As it turns out, there actually is a single answer to bridging any two byte streams: socat. The documentation has plenty of examples. Here's a few I made up involving named pipes and Unix sockets to go along with my recent posts:

Bridge a pair of named pipes to a Unix socket#

socat UNIX-LISTEN:test.sock 'PIPE:pipe_in!!PIPE:pipe_out'

Builds a bridge such that a client sees a Unix socket test.sock and the server communicates through two named pipes, pipe-in to send data over the socket and pipe_out to read the data received over the socket.

Connect to Unix socket HTTP server via TCP#

socat TCP-LISTEN:8042,fork,bind=localhost \
    UNIX-CONNECT:http.sock

For an HTTP server accepting connections via the Unix socket http.sock, makes it also accept connections via the TCP socket localhost:8042.

Forward a Unix socket over an SSH connection#

socat EXEC:"ssh remote 'socat UNIX-CLIENT:service.sock -'" \
    UNIX-LISTEN:proxy-to-remote.sock

Note ssh can do the same without socat (including supporting either side being a TCP port):

ssh -N -L ./proxy-to-remote.sock:./service.sock remote

But that demonstrates combining socat and ssh for getting access to streams only accessible from a remote computer.

The details#

The problem#

Previously, I wrote about using named pipes for IPC to allow controlling a process by another process running on the same computer possibly as a different user, with the access control set by file permissions. But I observed that the restricted unidirectional communication mechanism limited how useful it could be, suggesting another design might be better in settings where bidirectional communication including confirmation of commands may be useful.

Is there a good general solution to this problem without losing the convenience of access control via file permissions?

The solution#

Let's use everyone's favorite RPC mechanism: HTTP. But HTTP normally runs over TCP, and even if we bind to localhost, the HTTP server would still be accessible to any user on the same computer and require selecting a port number that's not already in use. Instead, we can bind the HTTP server to a Unix socket, which similar to named pipes, look a lot like a file, but allow communication like a network socket.

Python's built-in HTTP server doesn't directly support binding to a Unix socket, but the following is slightly modified from an example I found of how to get it to:

import http.server
import json
import os
import socket
import sys
import traceback

def process_cmd(cmd, *args):
    print(f"In process_cmd({cmd}, {args})...")

class HTTPHandler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        size = int(self.headers.get('Content-Length', 0))
        body = self.rfile.read(size)
        args = json.loads(body) if body else []
        try:
            result = process_cmd(self.path[1:], *args)
            self.send(200, result or 'Success')
        except Exception:
            self.send(500, str(traceback.format_exc()))

    def do_GET(self):
        self.do_POST()

    def send(self, code, reply):
        # avoid exception in server.py address_string()
        self.client_address = ('',)
        self.send_response(code)
        self.end_headers()
        self.wfile.write(reply.encode('utf-8'))

sock_file = sys.argv[1]
try:
    os.remove(sock_file)
except OSError:
    pass
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind(sock_file)
sock.listen(0)

server = http.server.HTTPServer(sock_file, HTTPHandler,
                                False)
server.socket = sock
server.serve_forever()

sock.shutdown(socket.SHUT_RDWR)
sock.close()
os.remove(sock_file)

Then you can query the server using curl:

$ ./server.py http.socket &
$ curl --unix-socket http.socket http://anyhostname/foo
[GET response...]
$ curl --unix-socket http.socket http://anyhostname/foo \
    --json '["some", "args"]'
[POST reponse...]

or in Python, using requests-unixsocket:

import requests_unixsocket
session = requests_unixsocket.Session()
host = "http+unix//http.socket/"
r = session.get(host + "foo")
# Inspect r.status_code and r.text or r.json()
r = session.post(host + "foo", json=["some", "args"])
# Inspect r.status_code and r.text or r.json()

The details#

The problem#

Firefox Marionette only allows a single client to connect a time, so I'd like to have a program in charge of holding that connection that can communicate with the other parts of the system that know what I want Firefox to actually do. While a common way of handling this is to run an HTTP server, that seems pretty heavyweight and would allow access to any user on the machine.

This can be generalized to any case where we want to be able to control one program from another on the same computer. Some reasons why we might not be able to simply act from the first program is if the latter has different access permissions or is holding onto some state like, as mentioned, an open socket.

The solution#

The two programs can communicate over a named pipe, also known as a FIFO. The mkfifo command creates one on the filesystem:

$ mkfifo a_pipe
$ chown server_uid:clients_gid a_pipe
$ chmod 620 a_pipe

Then you can use tail -f a_pipe to watch the pipe and echo something > pipe to write to the pipe. To add a bit more structure, here's a very simple server and client in Python where the client sends one command per line as JSON and the server processes the commands one at a time:

# server.py
import fileinput
import json

def process_cmd(cmd, *args):
    print(f"In process_cmd({cmd})...")

for line in fileinput.input():
    try:
        process_cmd(*json.loads(line))
    except Exception as ex:
        print("Command failed:")
        print(ex, flush=True)

# client.py
import json
import sys

print(json.dumps(sys.argv[1:]))

# Run the server.
$ tail -f a_pipe | python server.py

# Send some commands using the client.
$ python client.py cmd foo bar > a_pipe
$ python client.py another_cmd baz > a_pipe

Then the server will print out

In process_cmd(cmd)...
In process_cmd(another_cmd)...

The details#

The problem#

Previously, I showed how to get Firefox to show just the web content without any of the window borders or toolbars. But there's an obvious problem: those UI elements are actually useful for doing things with the browser. We can give a single URL as an argument when we start the browser, and that may be good enough for some use-cases, but what if we want to have more control over what the browser is displaying?

The solution#

Firefox has a feature for running automated tests called Marionette which we can use for automating Firefox outside of the context of running tests. There's an official Python client:

$ pip install marionette_driver
$ firefox --marionette &
$ python
>>> from marionette_driver.marionette import Marionette
>>> client = Marionette('localhost', port=2828)
>>> client.start_session()
{'browserName': 'firefox', ... }
>>> client.navigate('https://example.com/')

If it works, you should see Firefox load the URL https://example.com/.

You can find more information on the available commands on the basics page and the documentation.

The details#

The problem#

For my recent posts on ZFS, I wanted to quickly try out a bunch of variants of my proposed operations without worrying about accidentally modifying my real ZFS filesystems. Specifically, I wanted to know which ways of copying files would result in more efficiently reusing blocks from existing snapshots where possible.

The solution#

WARNING: The instructions below will modify the ZFS pool tank, which is the default name used in many ZFS examples, and therefore may be a real ZFS pool on your computer.

I strongly recommend doing all of this inside a VM to be sure you are not affecting any real filesystems. I used a VirtualBox VM that I installed Debian on and used the guest additions to share a directory between the VM and my actual machine.

First create a 1 GiB virtual (i.e. in a file instead of a physical device) ZFS pool to run tests on:

fallocate -l 1G /root/tank
zpool create tank /root/tank

Then perform various filesystem operations and inspect the result of zfs list -o space to determine if they were using more (or less) space than you expect. In order to make sure I was being consistent and make it easier to test out multiple variations, I wrote some scripts:

git clone https://git.aweirdimagination.net/perelman/zfs-test.git
cd zfs-test/bin
# dump logs from create-/copy-all- and-measure into ../logs/
./measure-all
# read ../logs/ and print space used as Markdown table
./logs-to-table --links

The details#

The problem#

ZFS datasets are a powerful way to organize your filesystems. At first glance, datasets look a lot like filesystems, so you may default to just one or at most a handful per pool. But unlike with traditional filesystems where you have to decide how much of your disk space each one gets when it's created, ZFS datasets share the space available to the entire pool. Since datasets are the granularity at which ZFS operations like snapshots and zfs send/recv work, having more datasets can give you better control over having different backup policies for different subsets of your data, and ZFS scales just fine to hundreds or thousands of datasets, so you don't have to really worry about creating too many.

But if you're me (well, not just me) and you realize this after you already have months of snapshots of a few terabytes of data, how do you reorganize your ZFS pool into more datasets without either losing the snapshot history or ending up wasting a lot of disk space on redundant copies of data?

The solution#

Before doing anything with real data, make backups and confirm you can restore from them.

I do not have a one-size-fits-all solution here; instead I'll outline the general process and recommend you continually review at each step to make sure things look correct and be ready to zfs rollback and retry if you make a mistake or notice a way you could have done something in a more space-efficient manner.

1. Create the new dataset hierarchy. I'll refer to the old dataset as tank/old and the new dataset root as tank/new.
2. Do an initial copy of the earliest snapshot you want to keep from the .zfs directory. If it's @first, then the copy command will be rsync -avhxPHS /tank/old/.zfs/snapshot/first/ /tank/new/.
3. Check your work and possibly delete or dedup files.
4. zfs snapshot -r tank/new@first
5. Do an incremental copy of the next snapshot. If it's @second, this may be as simple as rsync -avhxPHS@-1 --delete /tank/old/.zfs/snapshot/second/ /tank/new/, but that will waste space if you have moved files or modified small sections of large files.
6. Check your work, and make any necessary changes.
7. zfs snapshot -r tank/new@second
8. Repeat steps 5-7 for each snapshot you want to keep.
9. zfs rename tank/old tank/legacy && zfs rename tank/new tank/old

The details#

The problem#

When doing an incremental backup, any moved file on the source filesystem usually results in recopying the file to the destination filesystem. For a large file this can both be slow and possibly waste space if the destination keeps around deleted files (e.g. ZFS holding on to old snapshots). If both sides are ZFS, then you can get zfs send/recv to handle all of the details efficiently. But if only the source filesystem is ZFS or the ZFS datasets are not at the same granularity on both sides, that doesn't apply.

zfs diff gives the information about file moves from a snapshot, but its output format is a little awkward for scripting.

The solution#

Download the script I wrote, zfs-diff-move.sh and run it like

zfs-diff-move.sh /path/ /tank/dataset/ tank/dataset@base @new

The following is an abbreviated version of it:

#!/bin/bash
zfs diff -H "$3" "$4" | grep '^R' | while read -r line
do
  get_path() {
    path="$(echo -e "$(echo "$line" | cut -d$'\t' "-f$3")")"
    echo "${path/#$2/$1}"
  }

  from="$(get_path "$1" "$2" 2)"
  to="$(get_path "$1" "$2" 3)"
  mkdir -vp -- "$(dirname "$to")"
  mv -vn -- "$from" "$to" || echo "Unable to move $from"
done

The details#