All right, time for the next talk, just quiet, quiet everyone, all right, so every
year we try to find, you know, a presentation and sometimes they get submitted, sometimes
we have to find, you know, people introducing a new and exciting inference framework because
I think it's all about those types of things these days, like again, you know, you can sort
of do a lot of things with PyTorch, but that is super boring and it's not really inference
optimized, so last year it was the ML this year it's tracked, so Julian and Matthew take it away
Hey, no, no, okay, well, thank you very much for attending this presentation,
Julian and myself are both walking as software engineer developers at Sonos voice contouring
Paris and we are here today to introduce two open source libraries, one has been in
April while it's tracked and the other is a companion project that we open source last summer.
So, just quite seeing the surface, we have just a couple of minutes to introduce tracked,
so what you have to remember it is it's a generic neural inference neural network inference
library. One of these characteristics is that it's tracked all the way down, it goes from passing
your model to executing it, so it contains the matrix multiplication stuff and optimized
routine that we need. It's not new, it has been around for nearly 10 years and it's been used
extensively in Sonos, but some of the company have been using it. It's written in reverse,
it's meant to be very integration friendly and it's released under a very permissible essence.
So, just to give you an idea of what it looks like to use tracked, there's a bunch of
here, but very little is actually tracked code. The first two lines show how to
instantiate the NNF transfer of tracked, the second line show how to load the model,
transform it into a renewable form where, which you can actually run, which means finding out
the right order of operations stuff like that, so that's the first two lines at the top of the screen.
And then you have a bunch of stuff which is just getting an input ready for processing and
finally you get a model.run where you actually run some model on the image and the rest is
post processing and displaying to make the example walk, so very simple to use.
Another facet of tracked is it's common line tool, so it's designed to be able to first
dump a model, so that here you can see the beginning of a mobile net model, you can see the
convolution with some details, so it's useful when you're given a model and need to figure out
how you can make it run, that's your first stop. But then once it's running, you can actually
get insight on the performance, here you can see a profile of the same mobile net network
with, you can see that, for instance, the device convolution takes nearly 10% of the entire
execution time and you also get some insight on the arithmetic's intensity of the operator
and so you can design if something is what you expect or if you need to investigate it.
As I said before, it's meant to be very easy to integrate, so one of the key point is it doesn't have any
big cumbersome sort party, it's very easy to cross compile because we are using a rest.
It has support for resin, so you can actually run some model in your browser and we have a
bunch of targets that we optimized for, so I'm 32 and I'm 64, I'll abuse, because well, so
knows, but also Intel and wasum, so wasum works with CMED, I mean all of them are CMED,
but that was wasum CMED, we are also optimizing for metal and cuida, tracked as a CAPI and on top of
this CAPI was some Python bindings. Recently, we've worked on adding to tracks on GPU support
and we still want it to be very easy, so you can see the difference in the first two lines of code
and so one in the middle of the screen, what it means to go from CPU to GPU and everything goes
right on track, it's still all the fact that we're still working on support, but you can see that
in this case we get a nice improvement on the performance and you can also see I mean in
cell phone occasion to show how the command link can be used to bench a model.
Here I want you to show some of the web thing or thing that you might find weird if you play with
track and on this instance we are showing the dump of Yamamoto and one thing that I want you to
look at is the source as this S thing which means that track supports symbolic dimension of
shapes, so track likes to know everything about to network before running it so we need to know the
rank of each tensor and we need to know the shapes of each dimension, I mean the length of each
dimension, but there is a twist you can use symbol, so we don't have to have the exact value,
we need a symbolic expression to know how to reason about your model and I have another example
here, so that's in the same model, it's a Yamamoto, the dynamic key value catch operator,
what you can see here is the shape is now S plus P as being the number of token in the
current terms of Antoine if you are doing text generation and the P is the best sequence,
so here you can have all you prompt or everything accumulated before.
And one of the things which is different about track compared to the engine is that the state,
I mean track can do statement management, so here what you can see is that the dynamic key
value catch operator just have one input and one input it doesn't have to be plugged into
extra model input and outputs for storm management, for statement management, so state can be managed
by track inside a state object that track API exposed. So all of the examples so far that I have shown
about NNF and we come back to that, but track also has some support for I mean it has
first classities and for Onix, we support like 85% of the operator set and it pans out to
more model than that, we have to do some adjustment because of the knowing all about the tensors
before hand thing and the track protocol for format doesn't contain everything we need,
so here you can see the set input fact line four or five where we actually load the
model in two stage, first we load some kind of protocol, then we add some type information,
shape information here to specify exactly the shape of one input and that's enough in that case
to help track everything else about the shapes in the model, so it's really
analog to when you add, when you need to add some type information in the C++ program or in
the rest program for to add the compile of figure out everything.
More recently, the other hand thing happened and it came with NGIN that we are no longer generic
but we are taking a new approach where basically you have got the model in the NGIN and then you
provide the tensors on the side and that doesn't work for track because track wants the model as
an input, so for some time we try to manage with Onix but we like that we want not to go
going anywhere with this and we add this internal tool that is called Torch2NNF and that is what
Trulian is going to introduce now. Thank you, Matthew. So we will discuss about now shipping
neural network from Torch2NNF format and NNF format which is highly compatible with the
tracked inference and giant which is discussed. So before going on we just want to set
what is a model set for us, so something which is completely aside from a neural NGIN is a list of
non-sorn names and a graph of computation that I have been defined correctly which inform you
the set of transformation that go from input to output and so this is what you need to have something
independent that you can consider an asset. So what is NNF? NNF stands for neural network,
action format, it addresses the same core problem as NNX and was specified by the coronavirus group.
It's something that didn't match as good as for the community as it could be because it landed
one year after UNNX and it didn't have a proper inference on giant to run on but it has some
very interesting key features namely it have a readable graph structure that you can see on the right
there so it's very easy to read if you can just it's textual so you can directly use it with your
text editor and it have composition available directly so as you know neural network are repeating
blocks over and over again and so you don't have to repeat over and over again this especially for us
what was interesting is that it's an extensible tensor format as well that is defined in the
specification which are as well to have extended quantization logic and so now what is dodge
to NNF? So dodge to NNF it's a project that was born in 2022 out of frustration trying to
export or NNX quantized model for internal projects and it aimed to be a strong bridge between
PyTorch and NNF and more precisely tracked NNF extended version with check upfront at export time
that the equivalence between the two and friends on giant but are the PyTorch one and the
tracked one are equivalent for some specific example and so you can see here an example where you
export a model which is very similar from the API of NNX you may be familiar from PyTorch
except now you have this inference target which is there and which allow you to specify any
and friends on giant but here tracked with a specific version which allow you to unlock some specific
features and so why total training is so as we discussed it unlock some specific quantization
and the quantization function which are more advanced it can relieve you control specific module
say realization if you wish so you can take the and on some specific stuff you can add additional
expression for the symbolics which is discussed about tracked directly backed in the model
assets and finally it's more focused on single based neural networks so it will work better
compared to NNX to export this model for tracked so thanks for listening if you have any question
fit free to reach us on a GitHub or just ask us now areas of the condition as well if you wish to
contribute or use the project thank you very much we plan to stay around the room if you have
questions stuff I also jumped into the matrix group so thank you so much guys I don't see any
questions so I guess we'll we'll just let you hang out with the people