Welcome to day. So this is the first time garage is presenting at Boston. Usually it's
either a counter or a lax. This time it's me. I'm not part of the original developer group.
I'm actually more like a contributor and part of the Duffler Association, which is a shifurding
the garage project. So I can't answer deep chord questions, but I've been handling the
community channel for quite a while for people who know me. And I think it's interesting
for us to have a point a checkpoint today after five years of garage on the way we are
and what you can do with it. So let's get going. The fur just a small word on it. We are
a French association. So it's just a group of people. We try to promote digital sovereignty
and privacy. And of course that goes so hard to word self-hosting as an alternative to
big crowd providers. This means that we, because we do self-hosting, we need resiliency. And that
means that we get good at time and relatively low supervision and the low manpower that goes
with volunteer work. For anybody that needs to catch up, why is garage exactly? Why is it
different? Garage is a self-hosted drop-in replacement for the AWS S3 protocol. I mean,
AWS Object Storage General, but also there's a ton of product now that user protocol.
It is a bit different because it's mainly implemented resiliency through geographical
resiliency. So the idea was to use not to put stuff in it at usenther. We've heard an
power, we've heard an connectivity and instead to use smaller machines in a location that
do not have redundancy, like re-opartment or like a shared location. And to implement the resiliency
on top of that, so meaning that you have multiple geographically copy of it out of it,
and then if one location goes down, then you can just access it in another location.
So you can use it to host a website. We host a defler, more than 600 websites on garage
for quite a while now. You can store media, pictures, videos, as well. You can use it for peer
as a backend, and a ton of people are using it as well as a backup target. So classical S3
protocol usage. Why is Garage different? There's a ton of over-esprimplementation
order. Garage is a bit different because it's coordination free. Meaning that we don't have
a central elected master. There is no election going on. Every request, basically, we need to
get consensus from the cluster, and that makes it extremely resilient to latency. I'm
going to talk a bit about that later, but that's the main difference. We do not have a raft,
we do not have elections, and that also creates some constraints on how the clusters are managed.
But the benefit is that it's extremely beneficial for a situation where you have high latency
between nodes, because the lack of elections will mean that basically every request is going to
give you the base latency. This is a quite old graph at this point. I didn't rerun the testing,
but the point still stands. Basically, here what is shows you is that when you operate on
object in Garage, then you're going to get latency when you have high not-to-not-latency,
and we're talking about a hundred milliseconds, so this is not that a central latency, this is
like almost Europe, a US latency. You get much better latency on some operations, and that's what
really where architecture is about, because other solutions are much more optimized if you want to
just put stuff in Iraq, and then have just like horizontal scaling, that is very located,
physically speaking. The consistency model is not acid. We only have read after write consistency,
which is still better than eventual consistency. I think that's as well what's as free,
had initially, originally. I think they improved a bit for models since then, but they still
say that they have read after write consistency, it might be better. We talked about geographical
deployments. Garage has a zone concept that allows you to spread your deployments across
different zones, which are geographical locations, and then you can have multiple nodes per
location, so you can have, for example, three locations, and then have two nodes in one location,
three nodes in another locations, allows you to scale a bit for cluster as well.
What makes Garage a bit different, so I really talked about it, it's really the geographical
distribution aspect. Here we can see an example that you could have with multiple zones, one in
the UK, one in France, one in Belgium, one in Germany, and one in Switzerland,
and you can see that the chunks of data are actually spread, so this cluster has a replication
factor set to free, and you can see that the blue chunk is actually replicated in Belgium in France
and in Switzerland, but the red chunk is actually replicated in Germany in France and in the UK.
So on top of the cluster, you have a replication factor, and then that replication factor would
basically govern all of the copies of the data that happens in the cluster itself.
This is a quite old compatibility list, we have more now, but these are some of the software
that you can find that are using S3 for storage, and that can be configured to use Garage.
Again, because some software have hard-coated dependency to Amazon S3, they are not always
compatible, but more and more software now are becoming compatible with self-hosted S3 alternative.
So this is a very exciting. You can find a more up-to-date list on the website. We have a section
in here called the integrations, where you can basically list all of the software that has been
tested with configuration samples for them to help you get started.
So nothing was new at this point. All of this was from over Garage Talks.
Now this is the new content. This is just a short history. We have a roadmap that we had so far.
So the initial beta release was in 2021. We have the actual first release version that we
talked about to use in 2022, and then we had the first stable release in 2024.
There was a nice talk done by Alex at first them. That was an onswing this.
The one has been here for quite a while. We decided I was talking with Alex yesterday.
We decided to deprecate it this summer. One year after we released the 2.0 version.
But right now we're still maintaining it. And then the 2.0 was released a couple of months ago.
It actually doesn't have a ton of breaking changes, but it has a brand new HTTP administration API,
which is one of the reason why we had to release a new major version.
And then we changed the old replication modes setting to split it between the verification factor
and the consistency policy. So that made things a bit more configurable for you to pick
your replication and your consistency factor differently. We are thinking about releasing a free
of zero version. We are really thinking about versioning support. Garage does not support right
no, it's free versioning, which is a huge issue for a lot of software that relies on it.
So there's a couple of things like that. Versionings, tax on buckets, object retention policies.
Those are things we are thinking about. We are not making any promises, but they are
or objectives for the freedom of the version.
Okay, so now one thing I wanted to do for quite a while is to spend some time on best practices
for guard deployments. We have ton of people asking your questions on it's sometime bit clunky.
So I have a couple of slides for that and then we can either do a quick demo or I can take questions.
So the things you should know, if you don't know garage yet, we don't have to support within Garage.
So Garage allows you to configure a website in point and that website in point is finished to
be. It is by design. We intend you to do your own certificate management and outside of Garage.
And so it means that you have to set up your own proxy. We have the commutation for engineers,
for caddy and a couple of hours. Usually it's not really a big issue, but it's something that
we made that trust constantly back then. We don't have yet a free anonymous access, which
course for issue for some softwares. Basically, we advise you to use a website in point for that.
But not also for managed to use that way. So it's something that we're freaking about. I think
there is an issue right now about it and I was discussing with Alex. We might have a PR coming in
couple of months and it might require us to release any major version. We're not through yet.
Due to the lack of, like I said, like customer management or elections, we don't have a way
to kind of like automatically configure the cluster. And so that rolls always fall back to the
operator. So that means that when you create a cluster, the nodes can discover themselves,
but they will not autoconfigure themselves because the cluster layouts that you will configure,
which determines which nodes does what, is something that needs to be done by some from a 30,
but we don't have. So this can be node. Sorry, this can be done by interacting with any node.
But it falls back basically on an operator to do that. And that operator can be either
human. You can do that manually with a CLI. Or you can also build your own scripts or operator in
the company's sort of way to do that for you if you want. A couple of people in the community have done that.
And then you're going to make the cluster set up by the way.
The replication factor is something that is defined at cluster creation. And that we
stronger advise not to change. They are in supported procedures to do that. But please don't try it.
We had issues in the past. And it's very annoying to recover a cluster in that state. So basically
what we advise is really think about your replication factor first. And then if you need to change it,
just slip in a cluster and copy your data. Come on, pitfall. The default region in garage is garage.
Not your ST1. Very common issues. And then we had a discussion about access well yesterday.
And he was saying that really from his own experience as well, salvaging clusters from people
that may or after disasters do not use the degraded consistency policy really only use it for that
I recovery. It don't use it for like a regular cluster.
Well, I had a lot of questions along the last two years on what hardware people should use.
We really build garage to tell her any hardware, but it works best on some hardware, at least.
Those things are pretty obvious. I think for most people in the storage community.
But they might be not necessarily obvious for people that just want an S3 storage store and they don't
pretty know a lot about storage. So the first rule when you start to your storage is to not put your
storage back in on something like an FS or an S&B. It's a reciprocal disaster. So please don't do that.
Especially for the metadata folder. The base on S&B yarn on the FS don't tend to
perform really well. And sometimes they have bizarre issues, especially when you cut the network.
So don't do that, please. Get a right-intensive flash disk for the metadata folder. To be honest,
like anything flash should work for like very, very small use case. But if you have any
service use, and if you have a relatively big cluster, especially if you have nodes that come and go,
garage actually use the metadata database to store queues of blocks that needs to be received.
And those queues can be very, very right-intensive. I can tell you that at the third, we had normal SSDs.
Like some some like retail SSDs. We wore them out in less than two years. They basically
we thought they were leveling. So try to find something a bit flash. Sorry. Like geared towards a
right-intensive workload. And keep an eye on the word leveling of your drives.
With vise too, as well use red one for that. And to use a copy on the right-fast system, to do
stop plotting. You can also enable compression. You get a bit more latency, but it basically
compressively well, most of the base does. Get HDDs or flash for your data folder,
use XFS for your data folder, garage creates a lot of small files. So you need something that has a lot of
eye nodes. And here you have a choice. We have a multi HDD mode for the best performance. So
garage will basically take each drive independently. And then your application can be basically
geographical. Like you don't need to have multiple replicas in the same zone. If you can
retrieve them from another location. If you really don't want to think about it, you can use red.
And you kind of red below it, ZFS or NDDA mox best. But you will leave a lot of
performance on the table, because here I absolutely basically be limited by VRA.
But on the CPU and VRAM, we don't need that much. Of course, depends on the use case.
If you have a cluster with one petabyte of data, having a bit of RAM will help for metadata.
About metadata, we have multiple metadata engine in garage. Basically, the metadata is free things
that all of the mapping between the files and the blocks. It's also the bucket, the bucket,
the access on all of that stuff. But also any file that you upload to the cluster, well,
our below three kilobytes will actually be stored in the metadata engine itself and not on disk.
So it's something good to know. If you store tons of small files, you might not see those blocks
in the data folder. There would be a metadata side. We had multiple engine. Along with yours,
sled has been removed. After one.0, if you haven't done that, you need we need to migrate.
We have the tools for that. It's basically take your no-down, execute the migration, and then
put it back up. The two means that we have right now are SQLites, which is the safest one.
It is recommended for smart clusters. It has a couple of performance issues, but it works.
It's pretty rock solid in terms of power loss and data consistency. And then anandiby is a bit
faster than SQLites. It's there for big clusters. However, we had a couple of plenty issues
a long time ago. And if you have a renewable power, then I strongly suggest you use a UPS with
anandiby over to backup frequently. Anandiby has also a limit for the key size that has been
hidden by a couple of people using backup services. So the maximum length of a usable key is
480 bytes. So if you need long keys, you might have to use a SQLite. And then we had a
newcomer, which is FJL, which is experimental at this point, but it's already in the code base.
So you can try it and let us know what you think. The metadata engine can be download
par node, like there is no really restrictions. You can have a couple of nodes using a SQLite
and a couple of nodes using anandiby. We really thought garage to be something that was meant to
be george-distributed, but following the minio people turned off heart. It's good like that.
We had a ton of people asking us for single load deployments, which are a bit clunky,
because it's really meant to be a cluster. So we, of course, if you have a single load, you will
lose resiliency. Make sure you have backup for a metadata. If you use anandiby with single load
and you have a poor cut, you might end up with a corrupted database. And it's very annoying to
recover from that. We really don't have a way to rebuild the metadata from the data. So you will
have to basically, you will potentially lose some data if you do that. Like you will at this
news all of the file names and any file that might be stored. So we advise you to use SQLite,
if you do that, and potentially put a UPS on top, some people really have super flaky power,
and they are very not really crushed like every two days, and the chances for you to have
a corruption with that is right if they hide. Right now, the experience to set up garage in a single
load is a bit long. There's a ton of command and set up buttons to be down. Someone from
the community actually put together a bit of a wrapper for that. We might actually integrate that
into garage itself. So I was discussing with Alex yesterday and we were thinking about integrating
like a garage, serve, dash dash, single load that would basically do a single load setup. So that might
come into following them. With the load deployments, Joe distributed zones try to have at least
free. Keeping mine your network and Io, we had people built a very, very big clusters on top of
DSL effectively, and then they lose a node and they have to rebuild it and it takes it's only
a half a month. So be careful about that. You can easily accumulate data and then keep an eye
on how long you take for you to recover from that. Please set up monitoring. If you don't have
that, it's very hard to understand what's happening. You can get some insight, but it's difficult.
To give you a bit of an ID, Duffer has been earning a nine-terbat cluster, which is three nodes,
three nodes, and two nodes, so three locations, replication set to three. Over regular fiber
across Europe, we are between Lyon and Brazil and the bit of Britannia as well. We heard that
were tons of bigger clusters out here, up to the petapites, but we don't really have a lot of data
because we don't have any telemetry in the project by design, so we don't know. Tell us if you
have big clusters, we're interested. I'm going to go a bit faster, even. I don't think we're going
to have time for a demo. But we, on deploying, really use whatever you want, garage is a single binary.
We compile it, we went to Great Land to compile it statically, so you can really just drop it on
the system. You can also use Docker. We have Kubernetes and console integrations for not discovery,
but you will have to set them alive manually, or try to script on an operator to do that.
You can use something called Gateway nodes, which don't store anything, but can kind of like
geared your request toward the best node for load balancing. And if you find the default settings
or default missing time to be a bit conservative, you can adjust the missing tranquility,
which is basically how much time a garage sleeps between resting operations to avoid overloading
vector styles at all. There's a Kubernetes storage controller that is in the work as well
from the community, so you can take a look at that if you run into Kubernetes.
We have someone from the community that created a UI this year. It's already usable, it leveraged
the admin API, so thanks to them, I forgot to put the name in here, I should have actually.
But it's not here, you can find it, and then we also have an official UI coming later this year,
but it will be embedded within the product itself. So those are mockups for now, but
this is a very actively being worked on things to some funds that you got from the net this year.
So Valam, I went to do a short pass on the garage metrics, so this is what you get if you
version the default dashboard, really the two things that you need to be aware of are those two things
here, so the RisingQ length and the RisingT Error blocks. It goes to shut up, it means that you are
issues because your clusters cannot sync blocks correctly across the clusters.
The rest are not necessarily an important, but they give you more insight on the information.
One thing that a lot of people don't know is that garage for safety reasons don't delete blocks right away if you
delete a file. So if you delete a file in this API, garage will actually wait almost 24 hours before
actually removing the block from storage. So if you create and remove a lot of files, then potentially you might
see your storage going up for a while, because garage will not remove those yet. So the the
table gcq length has a bit of that as well, like it was showing you the blocks as very remove
black hours across time. One thing is going wrong, enable the debug logs, we add a ton of people
with authentication issues because of the S3 way of signing a request, so be careful about
your reverse proxy configuration, check your tranquility if you think the RisingSerslow, and then
LMDB has a bad tendency to kind of like being blown it up a time goes on. You can just shut
down your garage node, do an MDB copy, we've the C parameter to compact it, and then it will go back
to normal size. We take questions on the garage matrix channel in English, and you can log issues as
well on gith.deflared.fr, which is for digital instance, we also have a mirror record on github,
but it's basically just here for the people that want it on github, and we don't take issues there.
And then if you open an issue, if you have any problem, please provide the output of
garage status, and then garage status as well. If you're moving from main to main, I don't think
I will have time to do a demo, but basically we add in is that you need to list your brackets on keys,
and then you create those brackets on the garage cluster. Right now you will have to create new keys,
because garage enforces that the keys start with the specific string. We will put a PR
soon to basically add a parameter to skip that, and use garage key import to basically import your
key, and then you just loop over the buckets with our clone, and then you just copy for that up.
I found a blog post would actually be better for this, as I don't have time for a demo here.
And that's it, I think. I can do some demo.
Does that think with questions? Does that think with questions, or?
Okay, so that's five minute for questions.
Yeah, so the question is, do you need gateway nodes within a cluster?
The response is no. Any nodes that can take requests, gate windows are only here for helping,
basically they have a copy of the metadata in the cluster, and they are able to direct.
So what might happen if you hit any nodes is that you will bounce, like the node might not have a copy of it
at all, and it will might go elsewhere. And so you lose one RTT in latency. What you can do is you can run a gateway
locally, and then that way you can just talk to the gateway locally, and the gateway will basically
ban you directly to the correct nodes. So that's much faster in that regard.
One caveat vote is that gateway nodes are fully part of the cluster itself, and so they basically
have full access to the entire cluster. So that means that, while if you're running a gateway locally,
then you have to trust that that compute, because it basically has full embedded access to the entire cluster.
Yes?
You mentioned that it's more than a consumer, I'm guessing, because it's doing also synchronize.
Do you have any inform or set-up sites for those sites, so you can optimize other sites?
Yeah.
Okay, so the question is, we're doing a lot of block trites, and sequence, we're a block trite here,
a hug to optimize that, especially if you have something like ZFS that has a block size.
So that's a good question. I don't know what LNDB used by default. I would assume it's something
that's 16k or similar. That's usually common for databases. For the metadata, so that depends
a bit on what you pick. For the data itself, so garage has the block size setting, and it will effectively
split any incoming file up to that block size. So if you have a block size, says to I think
the default is one megabyte. Any file that is below one megabyte will start in the single chunk.
So it will appear as one file on your data folder, and if you have a file that is above that,
like then it will be split in multiple chunks. We actually advise to raise that block size,
because there's no round consequences. The only consequences are the additional buffers,
because when garage, one does block, it's 111 by one. So the bigger you block, the more memory you need,
and also like within RP cycles, because it's brought by block, then your RP cycles will last longer,
but that's more or less it. So big clusters really go up in in block size,
between sometimes like 10 megabytes or even 15 megabytes. If you store a lot of big content,
like people storing video content, it can go very high, and that gives in a lot of more bandwidth for that.
So the question is what is the metadata folder used for? And it's really only used for that,
and a couple of static configuration for like the layout configuration as well.
Yeah, so the metadata folder needs to be basically tuned for the metadata engine that you're using.
So the question is, blocks are not necessarily deleted right away when you delete the file in this
free. Is it possible to recover that? Right now, don't think it is. I mean, technically,
the blocks are still on disk, but the metadata has been deleted. So I mean, one thing though,
is if you have a metadata snapshot, then you would be able to restore it, and then to just give
a block to garage, and then we scan the blocks, and it will just connect again.
So the question is, did we consider implementing a recycle bin? This basically would be implemented by
S3 versioning. Yes? If you are implementing versioning, we also implement object.
So the question is, if we implement versioning, we will, we will be asked for implementing
implementing object locking. I need to look into it. I think it's something that we haven't done yet,
but that's something that's required, but is high in demand by a lot of backups of
her, that is it for a ransomware protection. So yeah. And the time's up. You can find me. I will be
hanging around. You can find me for reference.