The problem#
In my previous posts on minimal-webrtc, I set up a peer-to-peer connection between the web browsers on two different devices. For more flexibility, including making the remote camera and microphone appear as local camera and microphone devices, we need to handle the WebRTC connection outside of a web browser.
The solution#
minimal-webrtc-gstreamer
is a command-line
client for minimal-webrtc
written in Python using
the GStreamer library. It's mostly a modification of
the webrtc-sendrecv.py
demo script to use
minimal-webrtc
as the signaling server to make it easier for me to
tinker with.
Run as follows:
./minimal-webrtc-host.py\
--url "https://apps.aweirdimagination.net/camera/"\
--receiveAudio --receiveVideo any
It will output a URL as text and QR code for the other device to connect
to. With those options, the output from that device's camera will be
shown on screen and the output from its microphone will be played
through your speakers. That device will be sent video and audio test
patterns. See ./minimal-webrtc-host.py --help
for more information.
The details#
GStreamer support for WebRTC#
GStreamer is a very flexible multimedia framework that
allows for building pipelines of components that express pretty much any
audio/video input, output, conversion, or mutation you can think of,
including a couple years ago
adding support for WebRTC
with its webrtcbin
plugin. WebRTC isn't a
straightforward "input" or "output" as it supports two-directional
communication and requires a custom signaling server, so using it
requires writing some code. Of course, the release included examples; I
chose to use this Python one as my starting point.
As GStreamer has a lot of modules, they group them by quality into three buckets (defined in their README): "good" (supported and expected to work well), "ugly" (supported but may have issues), and "bad" (unsupported: pull requests welcome). The WebRTC support falls into the "bad" bucket, so some rough edges are expected.
Using the example signaling server#
The example code includes a web page and a signaling server which I used as a starting point on the general principle that it's easier to modify working code than to try to get something working from scratch. It has the same basic functionality as the example command-line I gave above with few issues that make it frustrating to work with:
- The default set up is to have the WebSockets go through the signaling server hosted on a separate port from the server hosting the website. When running them locally with self-signed certificates, this means you have to manually go to the WebSocket web server just to mark its certificate as trusted.
- The server times out when the WebSocket is unused for 30 seconds, and once the WebRTC connection is set up, nothing more is sent over the WebSocket. The clients interpret the disconnection as an error and shut down the WebRTC connection as well.
- The workflow is to open the website which shows a number to enter into the command-line for running the demo. Before running the demo app, you can type in the desired audio/video constraints as JSON.
Of course, those could be worked around with minor modifications, but I
had already written minimal-webrtc
so changing the code to connect to
that instead seemed more straightforward.
WebRTC incompatibilities#
It turns out GStreamer's WebRTC implementation is a lot more particular about how the connection setup is done than the Firefox or Chromium implementations are, so it took some time to figure out how to adjust the logic to match what GStreamer would accept.
Trickle ICE only#
GStreamer does not have a way to wait until all of the ICE candidates are ready and send them all at once in the offer/answer. This is a known bug which has been fixed as of version 1.17.1. But that version is currently only available in Debian under experimental, so I haven't tried installing it. I did try compiling the latest GStreamer from source and was still having problems, but I'm not sure I actually compiled and installed it correctly.
This wasn't too big a problem as I had never actually removed the code
from minimal-webrtc
for accepting ICE candidates as separate messages,
but it does mean that serverless mode isn't really an option.
No data channel#
While GStreamer has an API for data channels, I couldn't figure out how to get it to actually connect. Since the previous issue with ICE already meant that serverless mode wouldn't work, this wasn't really a major concern.
While I haven't investigated in more detail, I suspect to get a data
channel connected, you probably need to define it in the pipeline
somehow. Possibly using the appsrc
element as in this
example which appears to be for a different
WebRTC plugin for GStreamer than the one that's included with GStreamer.
This theory is based on noticing that receiving an audio/video stream doesn't work unless there is a stream of that kind being transmitted, which is why my implementation sends silence or a solid color instead of omitting the stream entirely. On the other hand, this StackOverflow answer suggests an alternative of registering to receive a stream of that type.
Cannot accept offer in response to offer#
SDP messages for setting up a WebRTC connection can be labeled as
"offer" or "answer". Between two web browsers, getting an offer in
response to an offer seems to be fine; they'll exchange multiple offers
before settling down into the final connection setup. But GStreamer just
gives an error. To fix this, I adjusted the logic
so that when talking to GStreamer, minimal-webrtc
will wait to respond
to an offer until it finishes setting up the audio/video streams that it
will send back, instead of replying as soon as the ICE candidates are
ready and then sending another offer later.
Figuring out the right correction to logic took carefully looking at the example's Javascript to figure out what it was actually doing differently.
Displaying the QR code#
As minimal-webrtc-host.py
is acting as the host and it's the host's job
to display a QR code for the client to use to connect, I needed to be
able to display the QR code from Python. Luckily, the utility I use to
display QR codes on the command-line, qr
, is implemented
in Python, so it's easy to use it within a Python program:
print(url)
import qrcode
qr = qrcode.QRCode()
qr.add_data(url)
qr.print_ascii(tty=True)
To be continued...#
At this point, I have almost exactly the same functionality as the original demo in a slightly easier to use form, but I do have basis for adding more features. Future blog posts will explore using this to make the remote device's camera and microphone appear as local devices that can be used by other applications.
Comments
Have something to add? Post a comment by sending an email to comments@aweirdimagination.net. You may use Markdown for formatting.
There are no comments yet.