The problem#

In a web app where the display should be constantly up-to-date, the client needs some way to get up-to-date information from the server. One of the simplest ways to do so is to regularly (every few seconds) query the server asking if there is new information. This involves making a lot of requests and is wasteful of bandwidth and processor time on both the client and server (the latter can be improved with caching).

If updates are rare, it makes much more sense for the server to notify the client when they occur, but HTTP is designed around the client making requests to the server, not the other way around. And, furthermore, the Django web framework (like many web frameworks) is built around that model.

The solution#

Of course, this is a well-understood problem and there are a wide variety of APIs and libraries to solve it discussed on the Wikipedia page for Comet. The main workarounds are WebSockets which is a very flexible technology for two-way communication in a web browser and long polling which is a simpler technique which involves merely having the server not answer a request immediately and instead wait until it actually has an update to reply with.

In the rest of this blog post, I discuss the changes I made to convert a Django-based web app that I originally wrote to use a basic polling pattern and hosted using uWSGI to instead use long polling and be hosted using Gunicorn/Uvicorn. I also cover nginx configuration including hosting the app in a subdirectory.

The details#

The problem#

The monitor-lock.py script in my previous blog post uses python-xlib, which currently mainly relies on manually porting Xlib functions to Python. This is why it is missing the barrier-related functions I needed in that post. There is work on automating this process, but it appears to be abandoned. I started trying to pick up where they had left off before finding the python-xcffib project which provides auto-generated bindings for libxcb and therefore gives full support for interacting with X at a low level from Python.

python-xcffib (named after the cffi library it uses for binding to the C XCB library) gives a slightly lower-level API than python-xlib, but they are both fairly thin wrappers over the X protocol, so the differences are minor. It was fairly straightforward to port my script from the previous post to use python-xcffib, available as monitor-lock-xcb.py.

Unfortunately, I ran into a bug in python-xcffib:

Traceback (most recent call last):
...
  File "./monitor-lock-xcb.py", line 38, in main
    devices = conn.xinput.XIQueryDevice(xcffib.xinput.Device.AllMaster).reply().infos
...
  File "/usr/lib/python3/dist-packages/xcffib/__init__.py", line 139, in _resize
    assert self.size + increment <= self.known_max
AssertionError

The solution#

I've submitted the fix upstream, so most likely you will not encounter this error. Updating to the latest version (after v0.8.1) should be sufficient to fix the problem.

The fix I applied was to modify the module's __init__.py (the location, which may be different on your machine, is in the stack trace). Specifically, on line 108 in the function Unpacker.unpack(), in the call to struct.calcsize(), change fmt to "=" + fmt.

The details#

The problem#

I hosted a LAN party¹ a little while ago and ended up needing to loan out multiple computers to guests in the interest of having no one try to lug their desktop over. As it turns out, I don't keep multiple of spare gaming-ready laptops around, so I needed to get more computers somehow.

The solution#

My desktop has three screens attached to it (two monitors plus a projector), so given an extra keyboard and mouse (or two), it should be possible to run multiple instances of the game on it at the same time to let multiple people play using the same computer.

The script from this forum post makes it easy to set up multi-pointer X so a second keyboard and mouse will get its own mouse cursor. Then each keyboard and mouse pair can interact with its own instance of the game.

As an additional aid, I wrote monitor-lock.py which allows you to assign a mouse to a monitor, so it cannot be moved off that monitor to prevent accidentally interacting with the other player's instance of the game.

The basic usage is that you first run it with no arguments to get the available screens and pointers getting an output something like this:

$ ./monitor-lock.py 
...
Available screens:
screen 0: {'x': 0, 'y': 0, 'width': 3840, 'height': 2160}
screen 1: {'x': 3840, 'y': 0, 'width': 1920, 'height': 1200}
screen 2: {'x': 3840, 'y': 1200, 'width': 1920, 'height': 1080}

Available pointers:
device 2: Virtual core pointer
device 17: second pointer

USAGE: ./monitor-lock.py [device] [screen]

and then in a screen session (so you don't have to worry about accidentally doing this on a monitor you've locked your pointer away from), run

./monitor-lock.py 2 0

and

./monitor-lock.py 17 1

to lock the primary pointer to the first screen and the second pointer to the second screen.

Just use Ctrl+C to kill the process when you want the pointer to be able to move freely again.

The details#

The problem#

The Xbox 360 controller has become the defacto standard controller in PC gaming in recent years, likely due to both the popularity of the Xbox and the fact that the controller can easily be used with a computer. One downside of this is that some games assume you have one. If the game supports it and is running through Steam, then Steam's controller settings will let you use any controller, but that doesn't work for all games, and you might not be using Steam. The game that prompted this blog post actually does have Steam controller support promised in the future, but it's in early access and they are busy developing other parts of the game.¹

xboxdrv#

The solution is xboxdrv, the userspace Xbox controller driver. In addition to supporting actual Xbox controllers, it can also simulate Xbox controllers based on inputs from other devices like a PlayStation controller or some less common controller.

Too many input devices#

On a modern computer there are often many input devices,

$ ls /dev/input/event* | wc -l
28

They are just identified by numbers, so it can be difficult to choose the right one and trial-and-error can get tiresome with so many. There is some help from the by-id and by-path listings:

$ ls -go --time-style=+ /dev/input/by-id/
...
lrwxrwxrwx 1 10  usb-045e_0291-if06-event-joystick -> ../event26
lrwxrwxrwx 1  6  usb-045e_0291-if06-joystick -> ../js6
lrwxrwxrwx 1 10  usb-0b43_0003-event-if00 -> ../event20
lrwxrwxrwx 1 10  usb-0b43_0003-event-joystick -> ../event19
lrwxrwxrwx 1  6  usb-0b43_0003-joystick -> ../js1
lrwxrwxrwx 1  9  usb-BTC_USB_Multimedia_Keyboard-event-if01 -> ../event2
lrwxrwxrwx 1  9  usb-BTC_USB_Multimedia_Keyboard-event-kbd -> ../event1
...

$ ls -go --time-style=+ /dev/input/by-path/
...
lrwxrwxrwx 1  9  pci-0000:00:1a.2-usb-0:2:1.0-event-kbd -> ../event1
lrwxrwxrwx 1  9  pci-0000:00:1a.2-usb-0:2:1.1-event -> ../event2
...
lrwxrwxrwx 1 10  pci-0000:00:1d.0-usb-0:1:1.6-event-joystick -> ../event26
lrwxrwxrwx 1  6  pci-0000:00:1d.0-usb-0:1:1.6-joystick -> ../js6
lrwxrwxrwx 1 10  pci-0000:00:1d.0-usb-0:2:1.0-event -> ../event20
lrwxrwxrwx 1 10  pci-0000:00:1d.0-usb-0:2:1.0-event-joystick -> ../event19
...

But, for the most part, those names aren't very helpful, especially since many joystick devices support 2 or 4 joysticks connected to the same device.

identify_evdev.py#

Enter identify_evdev.py:

$ identify_evdev.py
/dev/input/event22

Where /dev/input/event22 is the device of the joystick I touched after running identify_evdev.py.

Calculating π the hard way#

In honor of Pi Day, I was going to try to write a script that computed π in shell, but given the lack of floating point support, I decided it would be too messy. If you want to see hard to follow code to generate π, I highly recommend the IOCCC entry westley.c from 1998, the majority of which is an ASCII art circle which calculates its own area and radius in order to estimate π. The hint file suggests looking at the output of

$ cc -E westley.c

The 2012 entry, endoh2 is also a pretty amazing π calculator.

Getting π#

Instead, I will just generate π the shell way: using another program.

$ python -c 'import math; print(math.pi)'
3.14159265359

Pelican's default Makefile includes an option make regenerate which uses Pelican's -r/--autoreload option to regenerate the site whenever a file is modified. Combined with the Firefox extension Auto Reload, this makes it easy to keep an eye on how a blog post will be rendered as you author it and to quickly preview theme changes.

The problem#

With just thirty articles, Pelican already takes several seconds to regenerate the site. For publishing a site, this is plenty fast, but for tweaking formatting in a blog post or theme, this is too slow.

The quick solution#

Pelican has an option, --write-selected, which makes it only write out the files listed. Writing just one file takes about half a second on my computer, even though it still has to do some processing for all of the files in order to determine what to write. To use --write-selected, you have to determine the output filename of the article you are editing:

$ pelican -r content -o output -s pelicanconf.py \
    --relative-urls \
    --write-selected output/draft/in-progress-article.html

The right solution#

Optimally, we wouldn't have to tell Pelican which file to output; instead, it would figure out which files could be affected by a change and regenerate only those files.

Podcasts are a simple extension to RSS: in addition to text and a link, posts can include a file to be downloaded. In the case of podcasts, this is an audio file. Due to this, while many specialized podcast applications exist, any news aggregator will work, although it might not have the best interface for that use case.

My normal workflow for podcasts is to keep track of them in a news aggregator and explicitly download the files to a local folder. While this isn't overly onerous for a weekly podcast, it is a repetitive task that could be automated. More importantly, downloading an entire backlog of dozens of episodes that way would take a while.

The problem#

Given a file which contains a list of floating point numbers in IEEE 754 single-precision format stored in big endian byte order, how do you view and manipulate this data using command-line tools? This is an actual problem one of my officemates had.

The solution#

$ od --endian=big -f file
0000000   1.7155696e-07   1.0432226e-08    4.563314e+30    6.162976e-33

Liferea is a desktop news aggregator (sometimes called an RSS reader). Unlike the late Google Reader or most of its alternatives like the open-source Tiny Tiny RSS which are web-based and run on a server to be accessed via a web browser, Liferea is a separate desktop application and uses an embedded browser to view content.

The problem#

Sometimes you don't actually care about all of the items in a feed and the site provides no filtering mechanism. If the uninteresting items are rare enough, you can just ignore them, but a news aggregator is most useful if it only notifies you of news items you actually might want to read.

The solution#

Luckily, Liferea is very flexible. It supports running a command on a feed which it calls a conversion filter. I wrote some python scripts to filter feeds by title locally.

For instance, I wanted to follow only the changelog posts in the forum feed http://braceyourselfgames.com/forums/feed.php, but it includes changes to all forum topics, so I checked the Use conversion filter option and set the conversion filter to

/path/to/atom_filter_title.py --whitelist "Re: Change log"

The problem#

My previous blog post has a footnote in the first sentence. Due to the way footnotes are handled, the footnote reference is a link to #fn:prg, which works fine if the footnote is actually on the page, but on the blog main page (or any other listing of multiple articles) the footnote is not present because it's after the Read more… link. The result is that on those pages, all footnote references are broken links. These broken links should either be repaired such that they point to the article page or removed.

First attempt#

Unable to find an existing solution, I decided to write my own plugin, summary_footnotes. I started by finding another plugin, clean_summary that modifies summary and based my code off of it. That plugin uses Beautiful Soup to parse the summary and rewrite it. A quick look at the docs and I was able to figure out how to select the footnote links and rewrite them, which got me this version of the plugin.