So, hello everyone, I am Gabbard Vorskoch, I am going to talk about name resolution
in package managers from Lab Produce Beat Perspective.
This is going to be rather practical and like side-by-side comparison of three prominent
package managers and it's not going to be so theoretical, I live this to the next presenter.
And, yeah, just a few words about me, so as a day-to-day work at Roch, in the Rochlinux
team, we are providing the B&B distribution for medical devices, I also work as a geeks developer
for roughly a decade now, which is a functional package manager and as a hobby, I like
astronomy and I like sailing.
So let's get into it and see what we have.
So let's first have like a few definitions cleared up, how I am going to handle packages
in this talk is that we are thinking about them as a three-plat, the three-plat of package
name, a package version and the actual data that is comprising this package.
This is not always like 100% true, for example, you might end up in the unfortunate situation
that the same name and version is actually designating multiple different packages.
This is not so good.
Yeah, then what is the reproducibility?
The reproducibility is a property that when you are like building the same source then you
get the same binary.
This is pretty straightforward saying that we would like to achieve, it is like not always
working like this, but why is this important, why do we want to have this?
That's because if you think about the complexity of contemporary software, just by
having a look at its binary representation, it is very hard to come to any conclusion
about what it actually does.
That's why we are preferring to work with them in source form.
But then viewed like to establish a correspondence that, okay, this binary is really corresponding
to this source code.
But now with the current tooling that we have at our disposal, the only way to actually
do this is to rebuild it from source and then see that we are coming up with the same result.
Right now going backwards does not work, yeah it works for like small stuff, but for
typical software, it does not fly.
So yeah, when we are talking about reproducibility, we are talking about different layers
of reproducibility.
We can have like a layer on package level, that okay, this package is reproducible, if
I built it from source and I get the same package.
We can also talk about this at different levels.
For example, if we are talking about reproducibility as a system level, then what we expect
is that from the same definition of the operating system, I got the same set of files
on the carbrive of the system.
Before this to happen, we actually have to pin the whole transitive dependency graph of
the system image, and because if there is like just one version is much anywhere, it is
not going to be a reproducible system image anymore.
Yeah, why do we have a name resolute in packaging systems is that we don't really want
to encode this whole transitive dependency graph with all the versions for every package
definition.
Most probably in the first place, we don't have enough information to do this, and then
we would duplicate also like an immersive amount of information.
So we have to simplify somehow, and be able to express the dependencies in a way that we
are referring to like not the complete replete of the package, but we are just, let's
say we are just speaking the name or the name and the version of something like a smaller
piece of information to identify this package.
Yeah, we would like to essentially have something like this that we have a package called
a name, and this depends on this and this and this package.
And also, because we want to be reproducible, we want to have this process to be deterministic.
In most cases, this is not deterministic, there are a few approaches that are taken, but
usually it involves like creating the candidates set of packages that we can resolve the
dependency to, and then somehow selecting one from them, usually let's say the latest version.
Yeah, but then this in a latest version might depend, for example, when the package repository
update was run the last time on the system and this kind of things, and this is not really
acceptable if you want to have a reproducible build of the system, so somehow you have to
circumvent this thing.
And the other thing is that we don't have to, we don't want to hold the packages that
are like existing in the world because that would be like super annoying, we want to
share them somehow.
So we want to have a place where we share that, okay, this software can be packaged like
this, and from this package definition, you can somehow build it, or you have a binary
that you can download, that is going to correspond to that particular piece of software,
and you can install it on your system.
Yeah, so we are having like repositories, which is like called in a different way by every
package managerizing that exists, we already had this, that for example, in Rustware, calling
these, the packages we are calling creates, and the repositories we are calling registries,
but these are conceptually the same thing.
Yeah, okay, so right now what we are going to look at is how we are defining this process.
First, we have like a bunch of configuration that is like the active configuration of
the package manager, one piece of this information is which of these repositories are
used.
Yeah, so where I am sourcing the package definition from, and this is like common piece of
information, it is done very similarly in each package manager, you have usually one
or more configuration files with information like this.
This is an example for a deviant source definition, a reason example for cargo, how you define
a registry there, this is an example in git, how you define a channel there.
And then all of these repositories have some kind of underlying data structure.
So here in Rust Cargo, I am pretty sure that there are a bunch of people who is like
much more knowledgeable than I am, because cargo I am using only as a user, this is roughly
what I know about it, that you are pointing your URL there, and that's more of the information
you pass there.
In the beginning, there I know a bit more about this, and I would say that I am a poor
user, so here you have this sign by, stands up which is basically pointing to GPG key,
and then in the deviant repository you actually have a release file which is signed by this
key, and it is containing an integrated database of the actual state of the repository.
And in Giggs, what we have is actually each of these channels are Git repositories.
And then basically there we are using the security model of the underlying Git repository.
When we are accessing it, we have of course the option to have like TLS, and then in the
content of the Git repository, we have the Git commit signing, and all these additional
security properties that Giggs gives us.
And what is interesting about this is that when you are thinking about this, then the cargo
and the deviant registries are like not really having their full history at your disposal.
On the other hand, what Giggs channels do, because they are Git repositories, they have
the full history available.
So that means that basically you can time travel backwards and forwards along the channel
definition, and then you can have, let's say, a package from like five years ago pretty
easily pulled in.
Yeah, and also what this means there is it's pretty easy to actually pin down the whole repo
state, because you just notify which channels are in use, and you write up like the
Git commits that they are at, and then you have a full, fully reproducible repository
state.
And in the other cases, you have to actually resort to some kind of endlessly information
to do this.
Yeah, and then how does a package definition look like, because this is also important
from the reproducible perspective.
So let's have a look at how deviant does this, deviant packages are like defined by quite
a few files.
The control file is basically giving you like the name of the package, and it also tells
you is dependencies.
Yeah, and then you also have the change logo, the change logo is essentially telling
you the version of the package, more or less.
And then you have basically the rules as the building formation, the source format, because
deviant is able to build packages in different ways based on the structure of the actual
package definition on the deviant side, and this is roughly building up a package.
Yeah, so most of these code are snippets, they are not full definitions, they are just
the parts that are like important from this perspective.
Yeah, so in Rust you do something like this, this is in a single file, and then you
have the name and the version, and then the dependencies.
So this format is like way more compact, and here in the dependencies you actually have
a bit of linions, you can say that it's like bigger than this, smaller than that, but
smaller.
Yeah, so you can have like a specification of the versions that are allowed for the resolution.
In this form it is actually tying the versions, but you can have like more in for you.
And then this is how it looks in geeks.
What you see here is that the package name is basically the placated.
So the name of the package, after this name stands up, is what you are referring to the
package as on the command line.
And the top one is actually defining a variable, and this is how you are referring to the
packaging API.
And then in the inputs, this name that you have there is corresponding to these variables.
And then the name resolute in geeks is basically the resolution of the variable being
in the underlying programming language.
That means that when you put this inputs list geoc, then you already pinned the version to
the exact one, like a very similar definition to this one.
And this is always like fully pinned, there is literally no way to express anything besides
this binding construct.
That means that this structure is completely reproducible from the state of the repository.
So a few observations, debim packaging requires quite a bit of work, I think everyone
who ever did this knows that this is not like a trivial feat, it can be done pretty quickly,
but it can be a complicated list.
And the geeks packages, basically this is what I told about, is that the variable binding
resolute is already giving you a full graph of type dependencies.
Then there is usually a last step, which is missing from geeks, is that once you have
the candidates, how you actually find out which ones to manifest on the system.
So normally in the BM-based system, you are manifesting the latest version that is available
in all the repositories for the given package, except if you are having the conflict or
if you are defining pinning.
So that means that if you pin everything in a WN system, you get reproducible build, then
you have like one of these pin definitions for each and every package on your system.
This is perfectly doable, but it is pretty inconvenient.
And in cargo, we are creating a lock file.
This is like pretty typical thing, a bunch of other package managers are doing this thing,
that basically when you are building, it is like resolving the dependencies, drops that into
a 5 for you, and then that file is like taking precedence for the future builds, until
you say that you want to update it.
This is giving you like a reproducible set of sources to build from.
And in geeks, we already resolve basically at the finish on time, by this trick that we
are actually putting the 2d flash package objects into the graph.
Yeah, then next thing is that when we have these repositories, then there are a few
problems to take for about.
So sometimes the repository state is affecting your build.
You can work on this in all of the systems.
It is like pretty much dependent on how convenient to work on this if you want to a reproducible
build.
And then the other thing is that the user or the client side of the thing is not having
control over this state.
So if you get in control over this state, then there is no problem, but usually this is
not the case.
Then let's say the WMP people that were pleased, don't publish the tricks anymore.
This just doesn't fly.
Yeah?
And there are a bit of a problem, so we will look 5 base things.
The biggest problem there is that you are not like require to publish your look file.
And then there is like a very hard to figure out, for example, what your test suit was
run against.
Yeah?
Because it gets detached, you cannot like reproduce the same thing because you are missing
this information.
So one way to improve here is to encourage committing the look file into the source
control.
Then there is a third thing.
This is affecting all the systems, sometimes the like upstream just goes away.
And then you don't have a way to rebuild it anymore.
Yeah?
So, what is the benefit that is like provided by having central managed repository?
Is that the repository managers can enforce certain properties?
So this is, for example, where crazy.io can shine.
Because they can say that, okay, let's say we are not allowing to remove stuff.
Or we are not allowing to push a different binary corresponding to the name and version
and these things.
And if this is like centralized, then this is giving you the possibility to do this.
And then the situation is very better than like on the wild west where everyone is pushing
everything into the repository.
Yeah, but this still requires some coordination.
Yeah?
So approach is that can be done when there is no central management is basically too
full.
Either you take over the repository to your side and define it state completely, which
is one approach that is like typically applied.
And then the other one is that you can add additional configuration information to always
resolve the same name.
That's happening.
Yeah?
But this is usually like a lot of work.
And then it does not really mitigate against this package removal problem.
Yeah, actually taking over the repository does because there you just say it, you don't
remove it and that's it.
And then in functional package managers, what we have instead is that this is like completely
tied the repository state.
It is always at the trouble to the actual build.
So you can have a provision in file that is like telling you that it was built by having
the package manager pointing to these commits on these channels and that's it.
And then we also do in Geeks something very interesting about the removal of the upstreams
because there we are still relying on the source at least.
There are like a bunch of systems for example software heritage is allowing you to send
the source code to them and then they are archiving it.
And before the upstream, you are as are going down, software heritage still have a copy
from which you can build.
And also, basically in this functional package managers, the binary packages are just
the cache.
There is a promise, let's say, that you can build the exact same thing from source if you
want, but we are providing in our build infrastructure, something that we build ourselves,
feel free to track it, otherwise you can add our signature and then trust the binary
that we build for you.
Yeah, so packaging effort.
So usually for language package managers, it's relatively low.
They are made to be frictionless with that particular programming language.
But there are two things here.
One of them is the diamond problem when you are having like two packages depending on the
conflicting versions of something that is like single package.
This is a 3K1 and then it can result in all kind of interesting situations.
You can avoid this, for example, by running the whole thing in a container, then they are
not touching each other and then we are happy.
And then there is the infrastructure problem with language specific package managers, which
means that basically the things that you build are still depending on the actual versions
of the tool chain whatsoever, that is like not defined in the package definition.
And yeah, basically here you want to have an isolation and then the effort is usually
like you create the metadata build instructions and then you test the package and then
you can release it.
It's very similar to functional package managers.
And then there is a maintenance effort which goes into like how hard it is to update
this.
And once we have like all of this, then we are ending up with like some interesting
data, like how well this thing actually works.
And what we get is that now we are having like a 95% of reproducibility for almost every
thing, which is really nice.
There is one missing point which is really bad as of now, is that most of the time we
are going to build from really Starbucks on the distribution level.
And then the creation of the really stable from the sources usually not reproducible.
This is something that we want to address in the near future.
And then my key point about the whole thing is if you make this simple for everyone, then
everyone is willing to have reproducibility.
So let's go here and try to make reproducible build easy.
Let's see if I can make those write about one of our initials.
And then the I can activate which is what I am going to say.
Do you understand what you say?
That is really nice.
So what do you say about Filipinos?
next speaker now? Yeah, so it doesn't seem to be the case. Thank you very much.