The problem#

In my previous posts on minimal-webrtc, I set up a peer-to-peer connection between the web browsers on two different devices. For more flexibility, including making the remote camera and microphone appear as local camera and microphone devices, we need to handle the WebRTC connection outside of a web browser.

The solution#

minimal-webrtc-gstreamer is a command-line client for minimal-webrtc written in Python using the GStreamer library. It's mostly a modification of the webrtc-sendrecv.py demo script to use minimal-webrtc as the signaling server to make it easier for me to tinker with.

Run as follows:

./minimal-webrtc-host.py\
    --url "https://apps.aweirdimagination.net/camera/"\
    --receiveAudio --receiveVideo any

It will output a URL as text and QR code for the other device to connect to. With those options, the output from that device's camera will be shown on screen and the output from its microphone will be played through your speakers. That device will be sent video and audio test patterns. See ./minimal-webrtc-host.py --help for more information.

The details#

GStreamer support for WebRTC#

GStreamer is a very flexible multimedia framework that allows for building pipelines of components that express pretty much any audio/video input, output, conversion, or mutation you can think of, including a couple years ago adding support for WebRTC with its webrtcbin plugin. WebRTC isn't a straightforward "input" or "output" as it supports two-directional communication and requires a custom signaling server, so using it requires writing some code. Of course, the release included examples; I chose to use this Python one as my starting point.

As GStreamer has a lot of modules, they group them by quality into three buckets (defined in their README): "good" (supported and expected to work well), "ugly" (supported but may have issues), and "bad" (unsupported: pull requests welcome). The WebRTC support falls into the "bad" bucket, so some rough edges are expected.

Using the example signaling server#

The example code includes a web page and a signaling server which I used as a starting point on the general principle that it's easier to modify working code than to try to get something working from scratch. It has the same basic functionality as the example command-line I gave above with few issues that make it frustrating to work with:

1. The default set up is to have the WebSockets go through the signaling server hosted on a separate port from the server hosting the website. When running them locally with self-signed certificates, this means you have to manually go to the WebSocket web server just to mark its certificate as trusted.
2. The server times out when the WebSocket is unused for 30 seconds, and once the WebRTC connection is set up, nothing more is sent over the WebSocket. The clients interpret the disconnection as an error and shut down the WebRTC connection as well.
3. The workflow is to open the website which shows a number to enter into the command-line for running the demo. Before running the demo app, you can type in the desired audio/video constraints as JSON.

Of course, those could be worked around with minor modifications, but I had already written minimal-webrtc so changing the code to connect to that instead seemed more straightforward.

WebRTC incompatibilities#

It turns out GStreamer's WebRTC implementation is a lot more particular about how the connection setup is done than the Firefox or Chromium implementations are, so it took some time to figure out how to adjust the logic to match what GStreamer would accept.

Trickle ICE only#

GStreamer does not have a way to wait until all of the ICE candidates are ready and send them all at once in the offer/answer. This is a known bug which has been fixed as of version 1.17.1. But that version is currently only available in Debian under experimental, so I haven't tried installing it. I did try compiling the latest GStreamer from source and was still having problems, but I'm not sure I actually compiled and installed it correctly.

This wasn't too big a problem as I had never actually removed the code from minimal-webrtc for accepting ICE candidates as separate messages, but it does mean that serverless mode isn't really an option.

No data channel#

While GStreamer has an API for data channels, I couldn't figure out how to get it to actually connect. Since the previous issue with ICE already meant that serverless mode wouldn't work, this wasn't really a major concern.

While I haven't investigated in more detail, I suspect to get a data channel connected, you probably need to define it in the pipeline somehow. Possibly using the appsrc element as in this example which appears to be for a different WebRTC plugin for GStreamer than the one that's included with GStreamer.

This theory is based on noticing that receiving an audio/video stream doesn't work unless there is a stream of that kind being transmitted, which is why my implementation sends silence or a solid color instead of omitting the stream entirely. On the other hand, this StackOverflow answer suggests an alternative of registering to receive a stream of that type.

Cannot accept offer in response to offer#

SDP messages for setting up a WebRTC connection can be labeled as "offer" or "answer". Between two web browsers, getting an offer in response to an offer seems to be fine; they'll exchange multiple offers before settling down into the final connection setup. But GStreamer just gives an error. To fix this, I adjusted the logic so that when talking to GStreamer, minimal-webrtc will wait to respond to an offer until it finishes setting up the audio/video streams that it will send back, instead of replying as soon as the ICE candidates are ready and then sending another offer later.

Figuring out the right correction to logic took carefully looking at the example's Javascript to figure out what it was actually doing differently.

Displaying the QR code#

As minimal-webrtc-host.py is acting as the host and it's the host's job to display a QR code for the client to use to connect, I needed to be able to display the QR code from Python. Luckily, the utility I use to display QR codes on the command-line, qr, is implemented in Python, so it's easy to use it within a Python program:

print(url)
import qrcode
qr = qrcode.QRCode()
qr.add_data(url)
qr.print_ascii(tty=True)

To be continued...#

At this point, I have almost exactly the same functionality as the original demo in a slightly easier to use form, but I do have basis for adding more features. Future blog posts will explore using this to make the remote device's camera and microphone appear as local devices that can be used by other applications.