Yeah, so there we want to present a good project of us, where we try to kind of revamp
the way we do a cross-compilation using basal and LLVM.
So you all know a cross-compilation is quite hard topic, it's quite arcane, mysterious, you
have to master a lot of pieces to get it right, and it gets harder and harder the more platform
you have, you add.
So you know, you know, existing solutions are mostly sisterhood based, which you know are
great, one stop shop, but also sometimes kind of a black box, you just trust what's
in there.
If you want to configure things differently, you can have projects like that, they are really
great of the shelf solution, will cross-compile for any targets and do it quite well,
using these routes, using pre-bilt.
So all of these tools exist, which was a lot, you know, way beyond the scope of this project
that is not just cross-compiling user programs, but entire Linux built, or as we so a couple
presentations before stuff that also do from source, entirely from source, but very specific
to one specific target.
Then you have stuff like XXCC, which actually this project is heavy based on, as in,
we don't use XCC, but we use a lot of the pre-sept in that, they do everything from source,
and that's what we wanted to achieve with this project.
Then you have stuff like Nix, which I'm not going to talk so much about, but it's cross-compilation
is very well supported, but it is, makes a lot of assumption about where it's going
to run, so it's also a great solution, just not what we're going to talk about.
So what we want to do here is just cross-compile absolutely everything from source, because
if you can cross-compile a very simple source, it means that using clung, which is a cross-compiler,
we can target any of the platforms that LLVM supports, and essentially, allows us to
cross-compile to any possible target within the realm of LLVM supports.
So what we need for that, LLVM pre-built whole chain, source code for LVM, and all of
the other one times, like G-lipsy, LLVM, which is a muscle, any kind of other one times,
or esoteric, one times that you can think of, we need the source, and of course, we need
basal, because that's what the talk is about, basal for dos, who not familiar, it's a
build system, open source, build by Google, and one of the philosophy is that it drives you
to declare absolutely everything as an input.
So absolutely everything is an input, on variable, on variable, on input, tools, on input,
every single header, and source, file is an input, and it allows us to do a much more
correct build.
So we use our own pre-built LLVM tool chain, with a slight twist, is that we use the very
very cool tool from the LLVM project, the David Hunt, the other day, called the LLVM
driver, that basically allows you to use lots of the LLVM tools, but as a single binary,
pretty much like the Busy Box works, and this is interesting for us, because it allows
us to ship a very, very minimal, a very, very small tool chains, with all of the
clang, LLD, what else, object, object, object, object properties, and yeah, sort of
megabytes, and that's pretty much all we need, plus, during time sources.
So once we have all of that, and all honorable mention for David Hunt, for upstreaming a lot of
things to LLVM, to make that possible, and to support a lot of tools that were not
supported before.
So with all of that, what we have is this kind of complex graph, not the complex one,
when you know it, but basically the tool chain that we build is a multi-stage one, because
the LLVM cannot depend on the LLVM, so we have to build the LLVM, or the parts of the
LLVM, a very specific way, LLVM winds, same LLVM plus plus, same, all different stages,
all different kind of dependencies, and at the end, we have a final, a complete tool chain
that has all of the runtime dependencies that the normal program requires, and all of that
built from source.
The cool part is that, because everything is, is from source, and everything has been,
like, download it, everything is aromatic, we can actually pass this flag, which is kind of
unusual for cross-compilation, but we found it was pretty cool to add and to just prove
that there really is no C's route.
What happens next is a 100% aromatic link where absolutely everything has been cross-compile
from source.
So, all of the runtime is including the CRTs, and that's indian, allows us to
to build a user program, fully air-metically compiled
and linked for any target that we can support.
So yeah, any host, any target, as long as we can build,
as long as the target is supported by LLVM,
and we have the build sources for that,
then we can target absolutely everything.
So far, we support these whole matrix and it's growing
and it's getting easier and easier for us
because the big work was at the beginning
while you're in everything.
Now, adding targets is mostly writing build files
and then adding that to the graph
and we can support more targets.
Yeah, the problem is we need to write the build files
that's kind of the hard part there
between David and I, we implemented a lot of that.
So it's a lot of auditing on how all of those runtime
is built, how to do it properly.
In the special case of the GDBC, we go we offer
every single version up to a cutting point
that is the oldest LTS of RATGL,
which we start with a good minimal version to support.
So we have to adapt the build files for the versions.
The problem also with the GDBC is a kind of a specific case
because it is very, very hard to compile.
It's quite hard to understand, yeah, I need to go fast, sorry.
It's very hard, lots of McFiles cross-compiler.
So we decided, OK, we're going to generate a stub library.
We're all of the version symbols and then links against it.
So we generate basically a stub assembly file
with all of the versions per version of the GDBC.
We compile it and we link against it.
All of the symbols are null, but that's fully all right
for all the satisfied linker.
We actually took this from ZIG, if you want to check out,
it's a very cool tool called ABI list tool,
which we have in your best disk project for the GDBC.
Quick demo before I give the mic to David.
Where is he working?
So here, we're going to cross-compile for Linux from Mac,
Open SSL, and so it builds absolutely everything
as we mentioned, all of the runtime is all of the compiler RT.
It's a craft custom resource, dear.
It crafts basically everything to hand over to the plan driver
with do not pass any kind of additional flags to it.
As you can see, it's dynamically linked Linux RTC4.
And if you run it, Linux is just basically
Docker to be able to run from Mac and it works.
All right, so let's talk about some of the other components
that we built.
We added support for sanitizers.
They're built from source as part of the build graph.
So that's cool.
If you pass dash dash config equals UBCAN,
your binary gets built with UBCAN across all the platforms
we support.
We can link it statically dynamically, whatever.
But cool other cool thing is, because the tool
changed part of the build, we can rebuild LVM with UBCAN,
and then build your binary with that.
So if folks are working on debugging LVM or something,
that's like a cool, fun, handy thing.
So we support Windows.
So Windows, we built from source.
So we use the MNGW project, which is great.
But we wrote build rules to build a CRT's.
We build main works, which is like this extension library,
and a couple other like C libraries, which
generate import lives from all the.dev files.
And then there's a bunch of like MRI recipes
for assembling these things into the way
that the normal MNGW assist route layout looks like.
So we replicate all of that and do it on the fly,
and prepare all that for the linker to do the final link.
So demo.
That's so cool.
Is that everyone?
Yeah, it up.
Yeah, cool.
So here, we're going to change the platform slide
to target Windows AMB64, but we also support ARM64,
and we're going to go ahead and build that from source as well.
So right now, it's creating all the import libraries
compiling with libc++, targeting Windows.
And then it'll finish in the sec.
And we have a, oh, there we go.
We have our Windows file.
We're going to run it on the line, and it's
a little world.
That's kind of neat.
That better.
Cool.
So why do we do all this other than the fact
that we're nerds and we love doing crazy stuff like this?
So this lets you do remote builds with zero setup,
because everything is defined from source
as part of your build graph.
All of your build actions are able to be
scheduled for remotely across a ton of cores.
So we'll show a demo of that in a bit.
This also means you get a working cross compilation
for C and a cross linker for other languages.
So you can plug this in as a linker for rust,
for go, et cetera, for go requires a patch
to go tool chain, what we submitted.
But for rust, it kind of just works out in the box.
Big shout out to our friends at build buddy.
Build buddy is a remote execution provider for basal builds.
We kind of abuse them a ton to run a bunch of these things,
because as you can imagine, we are building LVM from source
a lot, and it's not that fast to build, except for us.
It's very fast to build.
I don't know if you can see this, but it's just showing
a profile of the graph.
There's like 700, 800 cores working on this build.
So if you just kind of sharded out, it's OK.
Let's show them the sample of that.
What is the example?
What are we showing, Ashley?
Same but we're most.
Oh, yes, this is remote build of a thing we should
produce, see with open SSL.
So it's going to load, it's going to build, it's done building.
That's kind of nice.
And then build buddy has a nice UI for like visualizing the graph.
That's the same thing.
Cool.
The final thing is bootstrapping.
So once you have this like hermatic cross tool chain,
you can build LVM itself, which
is what we do as part of our release process.
But we also integrated it into the build graph.
So if you pass dash, dash, config, it was bootstrapping.
Then what a single build command.
We're going to build you a fresh compiler,
and then use that compiler to compile your user code.
And it's basically the graph de-plicated,
because that's a bootstrapping, because right?
Yep, it's a single build.
Cool, let's show a demo of that as well.
So here we pass the extra bootstrapping, bootstrap flag.
And we're going to build LVM from source.
And I'm going to build OpenSSL using the new LVM from source.
This is all running remotely, because otherwise,
it's going to take like 20 minutes or whatever, right?
But if you throw it off, of course, at it,
it's actually kind of pretty fast.
Can see the 7,000 actions, which I have 7,000 files to combine.
So that LVM's done.
Now we're building up in a social.
If you ever are developing LVM, again,
you can make a change to LVM in your local checkout,
and then iterate on the entire thing
and to end the single build command.
Cool.
OK, I don't know if we need that.
Cool.
So what's next?
So right now, we build LVM.
We build with LTO and everything,
but we would love to instrument it and do FDU.
So the best way to instrument a compiler is to compile some stuff.
So our build graph will look like this.
We're going to build LVM from source.
We're going to use that new LVM to build an optimized
and instrument to the LVM.
We're going to use that to compile LVM
to get your profiling data.
And then we're going to combine all of that
into like a final optimized compiler.
It's going to take a while, but it's OK.
What I also want to do, we are not support
for more targets, more run times.
If you have other ones, send us patches, come talk to us.
We kind of want to be like complete and serve
a variety of use cases.
We already have some folks who submitted some stuff
for Windows.
We're looking at trying to get Cosmo LVM
to see working.
We're going to be pretty neat to have a single compiler binary
that works across various platforms,
because especially with like remote execution,
you can share your action inputs and get cache hits,
no matter which platform the actual compiler ran on
if it's the same exact binary.
So this little bit of a science project for now,
but making some progress on it.
And the final thing that we think
would be really neat is to be able to eject from basil
and kind of use basil to produce a normal cross compiler,
or a normal sys route, and then use that to power other projects.
So we're not quite sure how this would look yet.
It might be like this, where you actually create the cross
end, and then use that, or it might be like a make cc approach,
which we'll have like a shell script that uses basil
under the hood to create it, and then run the filter to the build.
We're going to play with it, see what feels good, see what works well.
But yeah, definitely trying to bring the power
basil to non-basely uses as well if possible.
Cool.
As all we have, thanks for listening.
Here's our repo, everything's up there.
And yeah, any questions?
Yeah.
Yeah, thanks for the architects.
It's very exciting.
I actually did something similar, so this time here.
But my size is to work with Jesus, and therefore I cannot set it
in English.
But how I understood what basically you make
sederanostasis or rapid electro bloodstains
is like an organ Orchestra with Find so
that you'll just make a GF C and use the
Korean Devil so we will be kind of having a feito
idea and something so.
They're really can speaking visolo's over which is see.
A part from March, and you'll actually will be
G-lip-C, the water is going to be made to work or G-C-C.
So I think we like simplify it a little bit.
We don't build G-lip-C itself from source.
We kind of build the headers in like an interface library.
And then when we dynamically link,
we end up using the system G-lip-C at one time.
If that makes sense.
But I don't know if you want to say more about that.
Yeah, and for G-C-C, as you mentioned,
so far, I don't think I have the energy to try to
specify G-C-C.
We have plenty to actually specify LB-C-C++
and the bunch of other new projects.
But G-C-C itself is just out of RAM, really.
Let's start.
OK.
Let's imagine you have a way to run a possible binary.
And then, could you rearrange the fields,
the whole thing, and specializing, or what rule?
Maybe you're already doing that?
Yeah, the question is, can you run the cross-compiled binary
to gather PGF statistics in like recompile?
Yeah, so Bazel already has support
for running a compilation with like a pre-uptained workflow.
And you can kind of do it out of the box.
It would be two separate build commands.
Like, you would build a gathered profile
than build again.
We only get to cheat because our workload is compilation.
So we can make it a very fancy build graph.
But that's also kind of us, like, showing off.
But I do think, like, they go the same approach.
You could, if you encoded your like profile gathering
thing as a build action, like it could also be wired
together in similar way.
Got it?
Do you plan to have many, many packages to do?
Do you make your show real best, Zella?
Let's say that I won't use this for a complex project,
but you want to kind of have that kind of basically
Yeah, the question is do we intend to become a package manager?
The answer is no, because Bazel has something called
Bable Center Registry, which is repository of build rules
for a bunch of packages.
Not everything, but there's a fair amount growing like day by day.
So the idea is, this is a tool chain that you bring into your project
with like three lines of config.
And then you can bring in all of those packages,
as well, through the Bazel module system,
and then basically compose those.
So as long as they have build rules that build the thing
and don't, some things only compile in certain platforms,
but as long as they haven't really done that,
we'll provide all the run times and all the standard library
and everything, so you can kind of combine them.
And this does work off the shelf for most things we tried.
Yeah, good.
So the question was, how do we deal with, if we need to depend on
an SO library, like, pre-built SO library?
So actually, Bazel already provides such a feature
called CC in port.
And for us, it's going to be armetic,
because you reference the file as far as Bazel knows.
It will know about that file.
It will come into the links and box and then Nicky.
So it just works.
And we actually do that a lot.
The file should exist on the platform, or yes.
If not, I mean, I do that on another project.
We just take them out of the wherever they are,
and we just CC in port in Bazel, in Bazel.
Yeah, Bazel also has really good support for bringing in remote files.
You're going to address files by URL and hash.
So you don't have to go and download things yourself.
You can just say, hey, I depend on this file.
That's hosted wherever.
And then, you know, write a rule, bring that into your build,
the SCC import.
So that way, like, if I build a project this way,
and then you go to load my repo, you don't have to go to download
stuff, you can just run the build.
Come, combine this outside after?
APPLAUSE