My previous post got the video from my smartphone
to show up as a camera device on my desktop, but for a video
chat, we probably also want audio. So, now the question is:
how to build GStreamer pipelines that will allow
minimal-webrtc-gstreamer to use virtual
microphone and speaker devices that I can point a voice/video chat
application at, allowing me to use my smartphone's microphone and
speaker for applications on my desktop.
The following requires that you are using
PulseAudio as your sound server and have downloaded
pactl load-module module-null-sink sink_name=virtspk \ sink_properties=device.description=Virtual_Speaker pactl load-module module-null-sink sink_name=virtmic \ sink_properties=device.description=Virtual_Microphone_Sink pactl load-module module-remap-source \ master=virtmic.monitor source_name=virtmic \ source_properties=device.description=Virtual_Microphone ./minimal-webrtc-host.py\ --url "https://apps.aweirdimagination.net/camera/"\ --receiveAudioTo device=virtmic\ --sendAudio "pulsesrc device=virtspk.monitor"\ --sendVideo false --receiveVideo false
You can reset your PulseAudio configuration by killing PulseAudio:
You can make the PulseAudio settings permanent by following
these instructions to put them in your
Unix pipe PulseAudio module
This answer gives directions for setting up a virtual
microphone device that takes its input from a Unix pipe using
module-pipe-source, but not instructions for
using it with GStreamer.
GStreamer supports redirecting to a file using the
filesink element. As usually when outputting to a file,
the appropriate thing to do is to write to it as fast as possible,
that behavior has to be overridden using the
Additionally, the proper capabilities have to be set to match what
PulseAudio expects to be coming on that pipe... which I don't think I
got correct since I never managed to get it to sound right with this
method. Here's the code I had which doesn't quite work:
caps = Gst.Caps.from_string("audio/x-raw,format=S16LE,channels=1") capsfilter = Gst.ElementFactory.make("capsfilter", "afilter") capsfilter.set_property("caps", caps) sink = Gst.ElementFactory.make('filesink') sink.set_property('location', 'path/to/virtmic') sink.set_property('sync', 'true')
Null output PulseAudio module
With further research, I found multiple posts talking about making virtual microphones in order to combine microphone and game/media output into the same stream. This one is the most detailed one I found.
The core concept is PulseAudio's
module. In PulseAudio, a "sink" is usually a speaker or other physical
device that actually plays the sound in the real world. A "null"
sink is a device that can be set as a sound output for a program but
discards the sound instead of sending it to a real device (equivalent
to redirecting a pipe to
/dev/null). But it's still useful because
every sink has an associated "monitor" source that plays back all sound
received on the corresponding sink.
We use null sinks to define two new devices which correspond to the remote speaker and microphone:
pactl load-module module-null-sink sink_name=virtspk \ sink_properties=device.description=Virtual_Speaker pactl load-module module-null-sink sink_name=virtmic \ sink_properties=device.description=Virtual_Microphone_Sink
(Note the use of underscores ("
_") instead of spaces in the device
descriptions; I couldn't figure out how to properly escape spaces in
those even though they're clearly allowed as existing devices have
spaces in their descriptions.)
While the null sink automatically includes a "monitor" source, many
programs know to exclude monitors when listing microphones. To work
around that, the
lets us clone that source to another one not labeled as being a monitor:
pactl load-module module-remap-source \ master=virtmic.monitor source_name=virtmic \ source_properties=device.description=Virtual_Microphone
Connecting to minimal-webrtc-gstreamer.py
We want to pipe the received audio (from the remote microphone) to the
virtual microphone device (
virtmic) and send to the remote speaker the
monitor of the virtual speaker device (
./minimal-webrtc-host.py\ --url "https://apps.aweirdimagination.net/camera/"\ --receiveAudioTo device=virtmic\ --sendAudio "pulsesrc device=virtspk.monitor"\ --sendVideo false --receiveVideo false
Connecting more audio streams with loopback module
The tutorial I linked above discusses using
module-loopback to pipe audio among various
devices, so, for example, the monitor of your actual speakers could be
sent to the remote microphone and/or speakers.
pavucontrol makes it easy to select which devices
different programs and modules use so you don't need to know the
PulseAudio device names and you can change which device is being used
even if the program doesn't have an interface to do so.