So, hi everyone, I'm Lorenzo Minier, and just before we start, you may see that I have
a small contraption over there, it's because I am an open source developer, but I also
am a very help-based musician, and I'm managing the music production that room tomorrow.
So, of course, I had to bring a food controller that is a click for my own slides.
I mean, this is me, the controlling the slides, this is just fun.
So, and some of you may know me as the main author of Janus, which is a Webarty-C server,
and for questionable-choising and picking pictures, of course.
But today, I will not be talking much about Webarty-C.
I will mention Webarty-C a little, but I'll focus more on another project instead that is
called I am quick.
So, I'll be talking a bit about the quick protocol and the efforts that have been going
on to do real-time media on top of quick as well, whether it is possible with our
the existing solutions and so on, and my efforts in that direction.
And I won't have much time to talk about quick in general, just to be very quick crash
course if you have nothing about it.
It's basically a new transport protocol that does not leave in the kernel, it leaves
in the user space because it goes on top of UDP as the actual transport protocol itself.
And, basically, it was conceived in order to try and get the best out of both the TCP
and UDP words together.
So, if you maybe a very informal definition may be TCP on steroids over UDP, if you
want something like that.
And while it was initially conceived, I was sorry, to fast.
While it was initially designed mostly for HTTP trees, so to optimize over the different
iterations of HTTP, you can put, you can basically build any application on top of that.
It's a transport protocol itself, and it has built in capability, especially encryption,
which helps cut down latency and so on, and one other important feature is that it supports
multi-stream out of the box, which means that unlike TCP, you can have multiple streams
over the same quick connection, which helps avoiding all those nasty out-of-line blocking
issues that you typically have with TCP.
So, again, it tries to solve many problems that TCP had, and also using UDP, it should allow
for more things that TCP allowed, which brings to the next question.
So, can we use quick for real-time media, because, of course, it works for HTTP 3, it works
great, but real-time media is an entirely different beast.
And of course, we know that WebRTC fits that bill.
There are several properties that it does that makes it possible.
There are some parallels that we can drive for quick as well, in terms of the different
functionality that are there and so on, mostly because, again, quick is its own transport protocol,
but it goes over UDP, which means that it can take some gains in terms of latency from there,
as well.
There are some currently big differences in terms of the final step, because, of course, with WebRTC,
we have all the media stock that is provided in the browser or in our library and so on.
If we want to do the same with quick, we have WebTransports to use quick and a browser,
but the whole media stock we have to write ourselves using Web Codex and other things.
So, that is, of course, challenging, but interior, it gives you more control to do, to do
call things.
So, this means that at least interior, quick is a potential candidate for real-time media,
but let's see what are the actual protocols that should make it possible.
And, of course, the first thing that comes to mind is why not put RTP over quick itself.
We all know about RTP, it works well for our use cases, and there is indeed a draft that
basically explains a bit how you can put RTP over quick to different mappings, and it focuses
more on all the things that you don't really need to do with quick, because, for instance,
many of the RTCP stats that were really provided to you by quick out of the box, and it focuses
more on framing, mostly because, with quick, you are free to act a bit like UDP when you
use data grants, and a bit more like TCP when you use streams, and so there are different
ways by which you can multiply RTP packets over there, different choices depending on what
you need to do.
And in terms of framing, it looks a bit like this.
One important functionality is that each RTP packet is also tagged with a flow identifier,
and what this flow identifier is entirely up to the application, it can be flow ID3 is
my video, or flow ID3 is all my RTP packets, no matter what they are.
It's really up to you, and it's negotiating in different ways, and depending on the multiplexing
that you're using this flow identifier, you may need to write for every packet, or just
for the first one in a sequence, for instance, if you're using stream.
Another important distinction is that if you're using streams, a stream is basically like
a mini TCP, which means that you need to provide framing exactly as we do need to do
that for RTP over TCP, which means that we need to provide a length attribute, and we don't
need to do that for data grant.
And I mean, RTP or a quick is interesting, but it's not really what the ITF is really
interested in, because it's again an attempt to make an existing protocol and just show
it on top of quick, which works for some use cases, but it's not really the optimal solution.
So they are starting to work on a different approach that is called media over quick.
And with media over quick, they are trying to build a new protocol that is aware of quick
and can try and take advantage of all its strengths, effectively.
And so they are designing a generic ingest distribution kind of protocol that whose aim is mainly
configurable latency, so that it's really as generic as possible, you can use it for
comprenting or broadcasting or even video on demand if you want or something like that.
From a architectural perspective, it does look a bit like, if you are familiar with WebRTCS
a fuse, it's basically the same kind of approach where you have a pub sub and a mechanism
media that is being published and media that can be distributed accordingly.
And all of that being very aware of quick as its foundations, so trying to take advantage
of all the different things that it has and most importantly, being independent of video
formats and media formats in general, and explain why it is in a second.
And if we look at these graph for here, which is the graph that we usually pretty much show
no in terms of depending on the latency that we are willing to accept will see the best protocol
for the job, we know that higher latency is HLS dash, so usually what you go for, if you want
really low latency, what back to C is going to lead the only solution, and so mock initially
tried to cover as more gap in there, but they soon found out that they wanted something like
these instead. So basically a single protocol to rule them all, the configurable latency that
I mentioned initially, so I may want to watch something in real time, or with a 30 seconds delay,
or maybe watch something from yesterday using the same protocol dynamically, basically.
And for an architectural perspective, as I mentioned, again, very similar, if you're familiar
with WebRTC, this will not surprise you, you have a publisher, pushing objects to a relay,
and the relay will push it to all the subscribers that are interested in it, and architecture
can be as complex as you won't, depending on for instance, if you're doing some sort of
CDN distribution, it can get quite complex that everything is specified quite nicely in the draft.
And an interesting property of mock is that, again, it is completely independent of formats
of whatever. So with WebRTC, we have specific rules for how to read video,
all those specific columns and so on, with mock, there is nothing like that.
Everything is just an object. So we carry objects around objects of data, and what those objects
are, is entirely up to the application, there are ways to basically signal it so that people are
ready to process food is coming in. But in general, we have objects, we can group objects together,
and with video frames, as it makes sense, if you start from a keyframe, and all the other objects
are the differences from that keyframe, the group of pictures, basically, a sequence of group
of objects is a track, which can be an other track, a video track, a subtitle track, whatever you
won't kind of track. And there is much more to say about mock in general, but I don't have much
time to wait all that for with a bit more on the, on some of the implementation aspects. And
even though it is a new protocol and actually a very active one, changes are happening over,
there are a lot of implementations out there in different languages by some very big companies,
as well. A lot of companies are invested in that, and you can find some more information in
their week if you want to know a bit more about those. And that person is started working on
these since a year or more or less in a new library that I call Diane Quick, and if your
career is about the name, you can ask me later. And basically the idea was to, exactly. And
basically the idea was to write a new open source library. So unlike Janus, which is a service
type component, I wasn't in something a bit more flexible this time, so I implemented these
as a quick library so that I could implement different things on top of that. And it is the
open source, you can find the repo there and an introductory blog post as well. With the idea that
this library would be generic, so you can use it as a generic quick or web transport library,
but I also wanted to build in support for those media protocols that we've seen before. So both
RTP or Quick and Media or Quick mostly because those are the protocols that I wanted to experiment with.
And so the idea was to provide higher level APIs to basically hide all the quick complexity,
which may or may not be possible in the wrong run. I'm exactly not sure at the moment.
And of course, I'm very far away from done. So it is very much a personal experiment so far,
basically everything is written from scratch, even the quick start, which is probably a big mistake.
And I'll explain later why, but it has been very interesting as a test by then sandbox to experiment
with all these new protocols, with new ways of exchanging media and stuff like that.
For instance, it allowed me to play a bit with web transport, seeing if I could,
for instance, talk from a browser to an application using my library, just using web transfers,
that way, experiment, a bit also, I put the library inside genres as an experiment so that I could
get to a data channels to web transport and have them in chat with each other somehow. So just
just to see if I could, not in groundbreaking in that, but it basically allowed me to do some kind
of interesting experiments. And of course, I wanted to experiment with those media protocols as
well. So I started with RTP or a quick because it was the easier of the two. Of course, I just
needed to figure out the framing. And basically, I designed a higher level API where you just,
basically, send and receive RTP objects using this API and don't really have to care about all
the quick aspects in the background. This was the domain idea. And I used that to basically
implement a basic server, a basic client, and a basic integration in genres so that I could,
for instance, bridge RTP coming from quick to WebRTC and vice versa, just to see if it was possible.
And again, I won't bother you too much with all the different experiments that I did there,
also because it's just images as demos, they don't make much sense. This, for instance, was just
the way where I could, basically, show of RTP or have RTP or a quick on one side and then show
it as WebRTC on the WebRTC side or vice versa. Or even at WebRTC and WebRTC on both sides,
and an RTP or a quick tunnel in the middle, which is probably the only place I see RTP over
quick adding some kind of a success because it allows you to multiply some multiple RTP sessions
over the same connection. It makes sense for tracking for boy P, for instance, or something like this.
But again, as I mentioned, RTP or a quick is not really that much of interest because
even the seeped people don't seem to be very interested in that, at least to the moment,
at least for the moment. And as I mentioned, media or a quick is really gaining the attention
in the sunlightization bodies. And so I was very much interested in that as well, coming from WebRTC
myself. And so I started doing pretty much the same kind of effort there. So again,
exposing APIs that map to the protocol semantics. It's a ways to publish, with subscribe,
wait to advertise your presence and so on. And then the library itself hides all the quick
quick stuff that is needed in order to make that happen basically in a nutshell.
And using that functionality, I implanted a few sample applications like a basic publisher, a basic
subscriber, a proof of concept relay that I want to expand. I want to build a stronger relay for
the job. So something that could be seen as the mock Janus Equivalent if you will. And a few different
ways, including a basic integration within Janus itself that I'll talk about in a few minutes.
And again, I wanted to think that is important to mention, especially if you at the end of
presentation you'll be interested in media or quick in general or this implementation.
Basically, all the participants in both the specification and the different implementations
are in a dedicated Slack channel that is the mock channel on the quick dev Slack group. So quick dev
is the Slack community that hosts everything related to quick development and the mock channel
is where the mock activity is typically happening. And this is where all the implementers also
engage with each other to organize inter-obsessions and stuff like this or to figure out
new updates to the specification for instance. And basically, I did a lot of different tests to
to do different things. Again, images that don't make much sense without context. There are some
links to different blog posts where I go much more in detail if you're interested. I'll go
quickly to those because I want to show something in particular. There is one really point
in particular that is interesting because meta was being very active in the mock specification.
As basically provided, as an open source repository is the ways to have both a mock publisher
and a mock subscriber within a browser, which means they implemented all that media stack that I mentioned,
all the web code exploration. The mock stack has part of a browser itself, which means that
from a browser you can create a mock publisher, connect to a relay and a subscriber can receive
that media. And I mean, this is pretty much the same as a million different web artists see
them as that we use ourselves for many times. And so this encouraged me to try and check whether or
not I could indeed bridge mock and web artists see to each other somehow. So to see if there
was some meta ground, how much effort it would be to translate between the two.
And basically the process that I followed was basically since I am quick, my implementation is a
library. I can use it within genres somehow. So I created a new plugin in genres that would allow
me to refer to I am quick and implement the mock parts, so both the publisher and the subscriber.
And so starting from the publisher it meant that I could use meta's demo as a publisher to send
media to a relay, which could be very well my own relay for that matter. And then within genres
I can subscribe to the mock objects and then figure out how to translate them so that whoever
to see users can consume them. And it turns out there were basically three different things that I
need to do. First of all, in terms of protocol, I mentioned how mock is media independent and whatever,
but for audio and video, in particular, they are using a specific format that is called the
lower ed container lock for short that is basically, you could see it as basically the equivalent
of RTP provides some timing information and some basic information that tell you what the object is.
So first of all, I need to translate the lock timing to the RTP timing, then with mock every object
in for video is a video frame, which means I can get a very big video frame and I need to
chop it up when I send it over RTP. That's another thing that I had to do. So split in multiple RTP
packets and then for video itself, web code accidentally do a BCC, RTP only does an XB. I had to
basically translate the way they are talking niche to 64 in there, which doesn't mean transcoding,
it just means replacing some bits here and there as you know, but that's basically what I needed to
do. And that allowed me to have a working mock publisher to about the CB work kind of thing
to do the other way around. I basically had to do the reverse. A web RTC publisher will send video frames
over multiple RTP packets. I need to reconstruct the video frame out of that. I need to do the same
thing, replace an XB with a BCC because that's what web code X want. And I need to translate
whatever timing information is in RTP to the lock format and then once I did that, everything
would work in in mock as well. Which means that I could do something like like this. So basically
web RTC mock publisher and web RTC viewer or vice versa. Even though of course I mean this
were a nice experiment, but I wouldn't want to do something closer to what Ali beg and did here. So
it basically explained how many demos recently showing how you can do a live streaming of a sports
game. In this case, a basketball game using media over quick using dynamic latency and so on.
And using that timeline down there, you can also go back and watch a highlight that you missed
for instance all over the same connection, which is kind of cool. It could be a game changer for
the for this specific scenario for instance. And since I am a football fan and most importantly,
I am an apoly fan, I want to do exactly the same thing, but with things that I like. So I don't know how
far I am, but hopefully we'll get there. Which brings us to, I don't know if I'm right on time,
or time's up. So I'll summarize briefly briefly briefly. The quicksack is something that I
want to replace. I wrote it from scratch. It works fine on my laptop. It probably works
awful when I want to do something more interesting. So I'll use NGTCP to in the future,
something that I want to do. And done it yet. I need to keep up with the specification. It's very
active. I want to work on media, football, not only we are there. And of course, I brought these
to your attention. So first of all, to make you aware of the market, if you are in Torrey,
I know also about the library if you are curious about that. And this brings this is the end.
And if you are here tomorrow as well, I'll be managing the music production in the room. Finally,
we have music. Open media was very nice to host some music presentations in the past, so maybe
we may be interested as well. So thank you. I don't know if you have time for questions,
or if we have to look at it short. Okay. I don't know if there is anybody who has questions,
or do you have questions? So perhaps you said this, but you have to have some of my
questions, so I'm sorry you haven't. What's the relationship between Microsoft and Web Transport?
Yeah. So the question is, what's the relation between mock and web transport? And basically,
web transport is basically really to keep it short. It's basically the equivalent of
what web socket is to TCP and HTTP web transport is to quick. So basically, you have quick. And then
out of quick, you can create a web transport session, which has custom APIs that you create
out of an HTTP 3 session. So you create an HTTP 3 session. You use connect in HTTP to say,
I want a web transport session. And after that, it's basically the same as quick,
we just very few information for framing purposes for the streams and so on. And that gives you
access direct access to quick in the browser. And then mock is just the application layer on top of
that. So you can either use mock on real quick, or you can use mock on top of web transport.
I think the, yeah. What is this, as you see, see, do you think you mentioned that, yeah,
sorry, which, uh, you're talking about something with, uh, quick, NG TCP 2. Yeah, NG TCP 2 is the
name of one of the libraries that provide you, uh, direct quick functionality. So it's one of the
libraries that you, that you use if you want to have quick support in your application. And so it's
a lower level implementation of the protocol itself. And since I have, I implemented quick on my own way
for my stack because it was also learning process. I want to replace the quick part and put
NG TCP there. And then keep all the application level stuff that I implemented, including the
RTP or quick and media or quick integration on top of this new layer. So this will just replace the
quick part. And then, the last thing is, yeah, if they heard that I still call it media over
quick, they would probably shout at me and throw things at me. They don't like the media part anymore.
So they, uh, now, they don't like the media part anymore. So they, uh, now, they don't
use the network. I mean, if they heard that I still call it media over quick, they would probably
shout at me and throw things at me. They don't like the media part anymore. So they,
now, officially mock is just an acronym. So it's already happening that basically whatever you put on
mock is up to you. There are ways, there are drafts that specify how you can
containerize specific kind of media, but that will be completely, uh, orthogonal to the
specification of the protocol. The protocol will always remain generic enough. So, and I'm very late.
So sorry again. Thank you.