You know, like "lions and tigers and bears, oh my"… okay, not funny, moving on…

The problem#

There's a lot of different ways to transmit streams of bytes between applications on the same host or different hosts with various reasons you might want to use each one. And sometimes the two endpoints might disagree on which one they want to be using.

The solution#

As it turns out, there actually is a single answer to bridging any two byte streams: socat. The documentation has plenty of examples. Here's a few I made up involving named pipes and Unix sockets to go along with my recent posts:

Bridge a pair of named pipes to a Unix socket#

socat UNIX-LISTEN:test.sock 'PIPE:pipe_in!!PIPE:pipe_out'

Builds a bridge such that a client sees a Unix socket test.sock and the server communicates through two named pipes, pipe-in to send data over the socket and pipe_out to read the data received over the socket.

Connect to Unix socket HTTP server via TCP#

socat TCP-LISTEN:8042,fork,bind=localhost \
    UNIX-CONNECT:http.sock

For an HTTP server accepting connections via the Unix socket http.sock, makes it also accept connections via the TCP socket localhost:8042.

Forward a Unix socket over an SSH connection#

socat EXEC:"ssh remote 'socat UNIX-CLIENT:service.sock -'" \
    UNIX-LISTEN:proxy-to-remote.sock

Note ssh can do the same without socat (including supporting either side being a TCP port):

ssh -N -L ./proxy-to-remote.sock:./service.sock remote

But that demonstrates combining socat and ssh for getting access to streams only accessible from a remote computer.

The details#

I use a Nokia N9 as my cell phone, largely because its MeeGo operating system is Linux based and in fact the command-line can be used very similarly to any other Debian system. This also means Linux sysadmining skills can be used to work around bugs in this sadly no longer supported platform.

The N9 stores a lot of its state including contacts, messages, and call logs in an SQLite database called tracker. It turns out many people have had trouble with it failing, resulting in the contacts app showing the error Can't import contacts and the messaging the phone apps also showing no data. Those threads offer various solutions on how to get your phone back to a working state. In my case, I followed the instructions, and my phone worked fine for several months before failing in the same way again.

I followed the instructions a second time but noticed that it was giving disk full errors. On further inspection, it was clear that the disk wasn't actually full: it was actually out of inodes. After some work which led to my previous blog post, I found /home/user/.cache/telepathy/avatars/gabble/jabber/ had hundreds of thousands of files (and I don't have that many friends). Simply deleting them freed up all of the inodes and I haven't had any troubles since, although I've been making regular backups just in case.

Recovering (some) lost data#

While the files have been deleted, they may not have been overwritten yet, so there may be some hope of a partial recovery. The data for tracker is stored in /home/user/.cache/tracker/. df has the useful side effect of revealing which filesystem a directory is on:

$ df /home/user/.cache/tracker/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mmcblk0p3         2064208    793712   1165640  41% /home

Attempting data recovery on a mounted partition is a bad idea as the unused space might get overwritten by new files; it's best to make a copy of it. Now that we know where the filesystem is, we can copy it using dd:

dd if=/dev/mmcblk0p3 | ssh $hostname dd of=$file

Then we can examine the partition offline. Particularly strings and grep with its -A and -B options can search for known strings like names and phone numbers and nearby content. For example, searching for a phone number without spaces should find at least some of the associated text messages:

strings partdump | grep -A3 -B3 -F '+19175551212'

Unfortunately, this method is slow and unreliable. I've used it to recover a few text messages and a few phone numbers, but there's no clear way to automate it, so do not expect to recover all of your contacts and text messages this way.

The problem#

Transferring many small files is much slower than you would expect given their total size.

The solution#

tar c directory | pv -abrt | ssh target 'cd destination; tar x'

or

cd destination; ssh source tar c directory | pv -abrt | tar x

The details#