The problem#

The monitor-lock.py script in my previous blog post uses python-xlib, which currently mainly relies on manually porting Xlib functions to Python. This is why it is missing the barrier-related functions I needed in that post. There is work on automating this process, but it appears to be abandoned. I started trying to pick up where they had left off before finding the python-xcffib project which provides auto-generated bindings for libxcb and therefore gives full support for interacting with X at a low level from Python.

python-xcffib (named after the cffi library it uses for binding to the C XCB library) gives a slightly lower-level API than python-xlib, but they are both fairly thin wrappers over the X protocol, so the differences are minor. It was fairly straightforward to port my script from the previous post to use python-xcffib, available as monitor-lock-xcb.py.

Unfortunately, I ran into a bug in python-xcffib:

Traceback (most recent call last):
...
  File "./monitor-lock-xcb.py", line 38, in main
    devices = conn.xinput.XIQueryDevice(xcffib.xinput.Device.AllMaster).reply().infos
...
  File "/usr/lib/python3/dist-packages/xcffib/__init__.py", line 139, in _resize
    assert self.size + increment <= self.known_max
AssertionError

The solution#

I've submitted the fix upstream, so most likely you will not encounter this error. Updating to the latest version (after v0.8.1) should be sufficient to fix the problem.

The fix I applied was to modify the module's __init__.py (the location, which may be different on your machine, is in the stack trace). Specifically, on line 108 in the function Unpacker.unpack(), in the call to struct.calcsize(), change fmt to "=" + fmt.

The details#

The bug#

The original code#

The Unpacker class is a helper for unpacking binary data formats used in the X protocol. It keeps track of the position in a stream of bytes and helps with converting that stream into a collection of values.

The Unpacker.unpack() function reads the next one or more values from the stream and updates the offset into the stream so the next operation works with the bytes after those values. It uses the Python struct module to do the actual work of dealing with the binary format. The values to be read are specified using a struct format string.

Here's the original source code of the function:

def unpack(self, fmt, increment=True):
    size = struct.calcsize(fmt)
    if size > self.size - self.offset:
        self._resize(size)
    ret = struct.unpack_from("=" + fmt, self.buf, self.offset)

    if increment:
        self.offset += size
    return ret

Format string mismatch#

The issue is that the struct.unpack_from() and struct.calcsize() calls are given different format strings, so they don't act on the same format. The = prefix tells the struct module to use standardized sizes and alignments when interpreting the format string as opposed to the default (which can also be explicitly requested with @) of using the native sizes and alignments. Which means the difference between fmt and "=" + fmt is only important if the native settings disagree with the standardized ones in a relevant way. For instance, on my machine:

>>> struct.calcsize('HI')
8
>>> struct.calcsize('=HI')
6

Looking at the chart of format strings, HI means an unsigned short (2 bytes) followed by an unsigned int (4 bytes). Obviously, 2+4=6, so it seems reasonable to expect the second answer. But it comes out to 8 instead due to the 4-byte value being 4-byte aligned. That is, 2 bytes of padding are added after the 2-byte value so the 4-byte value ends up at an offset which is a multiple of 4.

Presumably, the native alignment is different on the original developer's machine. Or format strings for which it matters simply don't appear very often, so they never ran into this issue.

Troubleshooting Python#

One of the reasons I like using libraries written in Python is that it's very easy to modify them as there's no explicit compilation step required: it's as simple as finding the installed copy of the library and editing it in-place. So I started my troubleshooting simply by editing the code in the stack trace to add print statements to get more information.

Finding the bug#

As I had ported the code over from a version that used python-xlib, I had two copies of the code that should be reading the exact same values. I added print statements to both trying to see where they differed. I was also looking at xinput.h and /usr/share/xcb/xinput.xml (in the xcb-proto package) trying to determine where python-xcffib was not following the protocol.

Printing out a value that differed#

In the python-xlib version, I printed out the correct result of

xinput.query_device(window, xinput.AllMasterDevices).devices

and in the xcffib source of xinput.py, I added print statements until I found the first difference. The first one that displayed a difference was in the class DeviceClass under the if self.type == DeviceClassType.Valuator branch:

print("Valuator (size=%d): number=%d, label=%d, min=%s, max=%s, value=%s, resolution=%d, mode=%d" %
        (unpacker.offset - base, self.number, self.label,
            self.min, self.max, self.value,
            self.resolution, self.mode))

which output

Valuator (size=48): number=0, label=164, min=<xcffib.xinput.FP3232 object at 0x7f78f7bb4a90>, max=<xcffib.xinput.FP3232 object at 0x7f78f7bb4b70>, value=<xcffib.xinput.FP3232 object at 0x7f78f7bb4ba8>, resolution=0, mode=2

Looking for the corresponding data from python-xlib, I found

{'type': 2, 'length': 11, 'sourceid': 12, 'number': 0, 'label': 164, 'min': -1.0, 'max': -1.0, 'value': 2157.6072313010227, 'resolution': 1, 'mode': 0}

The resolution and mode are different, which is enough to know there's a problem.

Narrowing to the first value that differed#

In order to see the difference in the min/max/value fields, I had to implement a __str__() method for xcffib.xinput.FP3232, which I did by copying python-xlib's FP3232.parse_value() and modifying it slightly:

def __str__(self):
    ret = float(self.integral)
    # optimised math.ldexp(float(frac), -32)
    ret += float(self.frac) * (1.0 / (1 << 32))
    return str(ret)

Then rerunning, I got the output

Valuator (size=48): number=0, label=164, min=65535.99998474121, max=4.7637149691581726e-07, value=-125790330.0, resolution=0, mode=2

showing that the issue occurs between label and min. Furthermore, the size of 48 when it should be 44 according to xinput.xml meant that I knew an extra 4 bytes of padding was being added somewhere.

Examining the sizes and offsets#

Looking up a few lines, I saw

if self.type == DeviceClassType.Valuator:
    self.number, self.label = unpacker.unpack("HI")
    self.min = FP3232(unpacker)
    unpacker.pad(FP3232)

Commenting out unpacker.pad(FP3232) get the size down to 46 but didn't get the right values computed:

Valuator (size=46): number=0, label=164, min=65535.99998474121, max=65535.0280456543, value=2022703104.000018, resolution=0, mode=0

And it's after the read of min, so it can't be the only problem anyway. After some fiddling around with what FP3232 did, I added print statements between each line of print(unpacker.offset - base):

Before number: 6
After label: 14
After min: 22
After pad after min: 24

Which showed that the line

self.number, self.label = unpacker.unpack("HI")

was advancing the offset by 14-6=8 bytes instead of expected 6 bytes for a 2-byte value followed by a 4-byte value.

To the bug#

That's when I went to check how unpacker.unpack() was computing the offset and saw that the formats were different. I added some additional print statements there to determine that struct.calcsize() was indeed computing 8 as the size when I expected 6. After reading the documentation on struct format strings, I reached the solution described above.