WEBVTT

00:00.000 --> 00:10.200
Work there for a short time amount of my life, so for me it's quite interesting, I hope

00:10.200 --> 00:15.000
for you as well and it will be great, it will be awesome, it will be the best talk in your

00:15.000 --> 00:16.000
life.

00:16.000 --> 00:19.000
So with that, no pressure at all, enjoy your talk.

00:30.000 --> 00:32.000
Two main microphones.

00:32.000 --> 00:37.000
Really close.

00:37.000 --> 00:42.000
Hello, and welcome to Running with Pedia on Environmental open source and a healthy dose

00:42.000 --> 00:46.000
of caching.

00:46.000 --> 00:49.000
Some before we start, there are a few declamers and caveants.

00:49.000 --> 00:54.000
So this talk reflects the hard work of loads of brilliant people, staff and volunteers

00:54.000 --> 00:59.000
alike, and I'm just the one just drew some graphs here, and it prioritizes simplicity

00:59.000 --> 01:01.000
of our accuracy.

01:01.000 --> 01:07.000
I've hidden a lot of the underlying complexity and important and hidden important components,

01:07.000 --> 01:11.000
like analytics, wicked data, and machine learnings of ability cloud services a lot.

01:11.000 --> 01:15.000
Now, keep in mind, if any of this thinks I'm Greek to you, there are a few Greeks in the house,

01:15.000 --> 01:19.000
just grab the one who's nearest, they'll help you out.

01:19.000 --> 01:23.000
And lastly, views are my own, not my employers.

01:23.000 --> 01:28.000
So, did you know that the infrastructure has to include Pedia and this is a project?

01:28.000 --> 01:32.000
A run by the Whitney differentation founded 22 years ago.

01:32.000 --> 01:35.000
All content is managed by volunteers.

01:35.000 --> 01:39.000
It supports over 300 languages and it turns 25 this year.

01:39.000 --> 01:44.000
Lastly, we've been training Salalam since we were just since there were just call statistics.

01:44.000 --> 01:50.000
But most importantly, we have some truly extraordinary content, which shape by what matters

01:50.000 --> 01:53.000
and what's happening in the world.

01:53.000 --> 01:59.000
For example, during major global events like the 2014 Ebola outbreak, or the COVID-19 pandemic,

01:59.000 --> 02:05.000
relevant articles where completely overhauled and translated into hundreds of languages.

02:05.000 --> 02:07.000
And that's the power of the global community.

02:07.000 --> 02:14.000
And from academics, we go to biographies of important people, scientific articles, political articles.

02:14.000 --> 02:17.000
I mean, you can't make this stuff up.

02:17.000 --> 02:22.000
And finally, the definitive answer to who is the queen of pop.

02:22.000 --> 02:27.000
So, or who is the real king.

02:27.000 --> 02:35.000
Or you can just keep all that and click on the person-of-life section of your favorite supervillain.

02:35.000 --> 02:45.000
So, besides Wikipedia, the foundation hosts the vast number of smaller yet interesting projects like Wikipedia code.

02:45.000 --> 02:48.000
We can use, and of course, Wikipedia.

02:48.000 --> 02:55.000
Hi, I'm Effy. I use the cell and settler PDFs at small shops before I became a reliable engineer at the foundation.

02:55.000 --> 03:01.000
I've also have a degree in physics, which is a great excuse for saying, I do not know this, I just am just a physicist.

03:01.000 --> 03:03.000
Thank you for coming.

03:03.000 --> 03:08.000
Why do you think I will chat about today around Wikimedia's infrastructure?

03:08.000 --> 03:14.000
We'll start with a bare metal part, move on to the application layout and open source, and lastly, to the rise of the machines.

03:14.000 --> 03:23.000
What I would like for you is to leave this room with a rough understanding of what's power and Wikipedia, and what are the current challenges.

03:23.000 --> 03:28.000
All right, the bare metal part.

03:28.000 --> 03:36.000
We have two core data centers in the US and five caching points, or pops, the last one being in São Paulo.

03:36.000 --> 03:44.000
Serving about 25 billion pages, page views per month, and with a little over 2200 bare metal servers.

03:44.000 --> 03:55.000
It's important to know that we do own our infrastructure, and therefore good reasons for that, like we want to have control over who can access it, physically and remotely.

03:55.000 --> 04:01.000
We buy, we destroy our hardware ourselves, and we do that for privacy and cost savings.

04:01.000 --> 04:05.000
But this comes with a complex game of constraints and trade-offs.

04:05.000 --> 04:14.000
Capacity planning can be a nightmare, market conditions can be unpredictable, like run prices this days, and recon servers is also an interesting challenge.

04:14.000 --> 04:23.000
Now, about those two core data centers, they are located in the United States, and they host all our data, including user data.

04:23.000 --> 04:31.000
For read operations, they are both active active, however, for write operations, and a given time only one of them is active.

04:31.000 --> 04:39.000
And we alternate between the two, every six months, reprocesses, through a process we call the data centers, whichever.

04:39.000 --> 04:48.000
The details of which are a bit complicated, however, the really short version is that it happens within the span of three days.

04:48.000 --> 04:56.000
And you will use it to perform like disruptive maintenance tasks, and it's just good hygiene to check your processes.

04:56.000 --> 05:11.000
It used to be much heavier, 10 engineers and around 45 minutes of read only mode, and this has been cut down to two engineers, two minutes time, two minutes of read only mode, or 3.5 if I'm performing it.

05:11.000 --> 05:15.000
The content delivery part.

05:15.000 --> 05:24.000
By large, when we are not talking to chat bots, and we are actually clicking on pages, we hit some public webcash, a CDN in most cases.

05:24.000 --> 05:34.000
And if this content with request is not cash, it will be fetched from the backend, which of course is not different than when you're visiting Wikipedia, right?

05:34.000 --> 05:41.000
And according to hack and use, this is an accurate depiction of our infrastructure, which you can learn from your own laptop.

05:41.000 --> 05:48.000
So, I think I'm done.

05:48.000 --> 05:52.000
So, jokes aside, things are more complicated.

05:52.000 --> 05:57.000
So, if we zoom into the CDN, you won't be surprised to find just a bunch of cashing servers.

05:57.000 --> 05:59.000
Now, what is in the cashing server?

05:59.000 --> 06:02.000
There are many layers of HTTP purposes.

06:02.000 --> 06:10.000
The outer layer is AHA proxy, responsible for TLS elimination, and HTTP 2, throttling, rate limiting, the lot.

06:10.000 --> 06:16.000
And then we have varnish, which cash is cash is objects in memory.

06:16.000 --> 06:21.000
So, most frequently stuff are live there, and it's doing like a lot of the heavy lifting.

06:21.000 --> 06:28.000
And lastly, the HES layer, the Apache traffic server, storing objects on desk this time.

06:28.000 --> 06:33.000
And on misses, HES will fetch content from the application layer.

06:33.000 --> 06:38.000
Now, all those resources coexist in a single bare metal host.

06:38.000 --> 06:46.000
And I would like to also add that there is a lot of cashing logic and routing complexity there.

06:46.000 --> 06:52.000
One last thing to mention here is that each pop has two sets of cashing servers.

06:52.000 --> 06:58.000
One set is responsible for text, HTML, JS, and all that.

06:58.000 --> 07:03.000
And the other one is dedicated to what we call upload, which is technically media stuff.

07:03.000 --> 07:08.000
And before we zoom into the CDN, let's go through some fun facts.

07:08.000 --> 07:15.000
Zip's flow is in a empirical observation saying that your most popular item is twice as popular as your second one.

07:15.000 --> 07:24.000
So, if we are hosting pictures of 100 animals, the dog has 20 requests per second, and the cat has 10.

07:24.000 --> 07:36.000
So, and this lead gets as easy and the outcome is a part of the distribution, which means that 8% of your traffic comes from only 20% of your content.

07:36.000 --> 07:41.000
So, 25 miles would be generating the vast of our traffic.

07:41.000 --> 07:46.000
Now, in some parts of the world, maybe dingos are more popular than dogs.

07:46.000 --> 07:51.000
So, content cash, CDNs have regional content characteristics.

07:51.000 --> 08:02.000
And lastly, paid CDNs may also provide services to enforce privacy laws or block content in regions based due to licensing or sensitive.

08:03.000 --> 08:13.000
Right. Now, before we get to the application layer, we need to make a quick side quest to talk about communities and media wiki and a promise it's going to be short.

08:13.000 --> 08:18.000
So, Kubernetes is a platform to run containerized applications that's all you need to do.

08:18.000 --> 08:27.000
The smallest deployment unit is a pod, and in a pod there's a main application, which is surrounded by some other containers which just help it out.

08:27.000 --> 08:37.000
A deployment manages a group of application, making sure that they run like the supervisor, and in order to talk to those pods, you need a service, which is basically the receptionist.

08:37.000 --> 08:43.000
So, traffic comes in. The receptionist shows you to a pod that will serve you.

08:43.000 --> 08:46.000
All right, out of the way. Now, media wiki.

08:47.000 --> 08:57.000
Media wiki has been a core application since 2001, sold enough to have his own beer now, and it's powerful, scalable, and it's built on PHP and my SQL.

08:57.000 --> 09:01.000
We used to use it like literally for everything.

09:01.000 --> 09:11.000
Now, to run media wiki traditionally, we used to have role-based clusters, sizing each one, based on the functionality and workload it was serving.

09:11.000 --> 09:18.000
So, the cluster serving, the API was differently sized from the cluster serving, the web.

09:18.000 --> 09:36.000
But, being a wiki also has friends. It's surrounded by a number of microservices, which it talks to you through extensions, such as side-toids generating citation data, muffled, mentoring, muff functions, thumbboard, doing image scaling, and whatnot.

09:36.000 --> 09:44.000
Next, there are some note-worlding milestones of this infrastructure. In 2014, we adopted the microservices model.

09:44.000 --> 09:53.000
In 2016, we migrated from Apache Mod PHP to Facebook's HHVM, because Facebook promised, it will sold, or our problems.

09:53.000 --> 10:00.000
In 2019, we migrated from HHVM to PHP VPN, because you should make promises you can't keep.

10:01.000 --> 10:06.000
It was decided that HHVM wouldn't support PHP in future versions.

10:06.000 --> 10:21.000
And, 2025, 2025, founders, concluding a marathon, which roadblock, by roadblock, by roadblock, we got our old monolith and migrated to Kubernetes.

10:21.000 --> 10:28.000
So, for the same reasons, and we did that for the same reason that most organizations have.

10:28.000 --> 10:36.000
By making this transition, we'd not have road-based classes anymore, but commit as deployments.

10:36.000 --> 10:39.000
Okay, let's go to the interesting bits.

10:39.000 --> 10:48.000
The sit-in section, we mentioned we have two types of requests, text and upload, and on a cashmess, HHVM will fall where the request is a vocational layer.

10:48.000 --> 10:53.000
HHVM has a number of rules, which would determine where you're going to be routed to.

10:53.000 --> 10:58.000
And if this is a non-medium request, you will most certainly hit a Kubernetes service.

10:58.000 --> 11:04.000
And if we zoom in that, the Kubernetes service is going to send you to the appropriate ports of a deployment.

11:04.000 --> 11:11.000
And if we zoom further dip it down, like MWWeb here, for example, we'll find a bunch of media weeky ports.

11:11.000 --> 11:17.000
To understand what is inside the media weeky port, we're going to have one monicide quest.

11:17.000 --> 11:24.000
You're about to learn more about PHP that you've broken for, so I'm really sorry for that.

11:25.000 --> 11:32.000
When a user initiates a request, it is first received by the web server, which could be like Apache and GeneX Minus.

11:32.000 --> 11:40.000
The web server forwards that to PHPFPM, application server that manages pull-up workup processes, FPM.

11:40.000 --> 11:50.000
This is this.

11:50.000 --> 11:55.000
FPM delegates, that was the best part.

11:55.000 --> 12:05.000
The request to the ZendEngine, the ZendEngine analyzes and compiles the PHP code into UpCodes, which then cached and then executed.

12:06.000 --> 12:14.000
Now, during execution, the script, my fetch data from MAPCU, which is an in-memory shared key value storage, holding stuff like, you know,

12:14.000 --> 12:23.000
list of countries, configuration settings, variables, a lot, or it might fetch from the distributed cache from MAMCASH in our case.

12:23.000 --> 12:27.000
And if that's a miss, there's no other option, but to create the database.

12:27.000 --> 12:40.000
So, to update this request, this flow, and make it observable, here are the containers of a media weeky pod.

12:40.000 --> 12:47.000
And void is a proxy that is like a gatekeeper, and it handles communication in Microsoft service architectures.

12:47.000 --> 12:50.000
Then there's a web server container, Apache in our case.

12:50.000 --> 12:56.000
The media week container, where we run PHPFPM, and Macrota to write requests to the distributed cache.

12:56.000 --> 13:00.000
There are a few more containers for observability and logging purposes.

13:00.000 --> 13:06.000
So, you're ready for your next PHP gig, Haragon? Let's go to our data stores.

13:06.000 --> 13:09.000
Last time, it's a last time, promise.

13:09.000 --> 13:14.000
So, focusing on MAMCASH and the database.

13:14.000 --> 13:22.000
I'm not going to try to sell you MAMCASH, but we need to mention some high-level stuff that makes it the building blocks basically of the internet.

13:22.000 --> 13:26.000
The lid developers demand them, and frankly, it's an amazing piece of software.

13:26.000 --> 13:30.000
When MAMCASH breaks, it's you. It's not MAMCASH.

13:30.000 --> 13:37.000
And at its core, it's a key value store. It does clever memory management using slabs, and of course, it's how are you.

13:37.000 --> 13:43.000
This is where we saw it, a familiar data, like article metadata, a range of pages, and all that.

13:43.000 --> 13:49.000
So, to talk to MAMCASH, we're using Facebook's Macrota because we never learned.

13:49.000 --> 13:55.000
It is a MAMCASH proxy that sits between the PHP processes and the class in the MAMCASH class.

13:55.000 --> 13:59.000
And it is what makes our distributed cash distributed.

13:59.000 --> 14:05.000
Some great features are sharding, replication, failover support, and of course, it's battle tested.

14:05.000 --> 14:08.000
I've broken this part of the infrastructure twice.

14:08.000 --> 14:12.000
The first time it was DNS, the second time it wasn't DNS.

14:13.000 --> 14:20.000
So, now, for storing the real permanent data, we've been using MariaDB since 2013.

14:20.000 --> 14:28.000
Auditor-based clusters are divided into sections, which have primary and secondary and replicants.

14:28.000 --> 14:32.000
Think of sections as like statically assigned sharding mechanism.

14:32.000 --> 14:37.000
So, the English Wikipedia is section one, the Italian is section two, and all.

14:37.000 --> 14:43.000
As you may have meguest, media Wiki reads from the replicas, writes in the primaries.

14:43.000 --> 14:48.000
And we do cross data, center, replication, and we have three main clusters.

14:48.000 --> 14:56.000
The Wiki text that holds the actual Wiki text blobs, the metadata one, telling us where the blobs are, and lastly, parser cache.

14:56.000 --> 15:04.000
To make this happen, we have 270 DB servers serving about 600,000 queries per second.

15:05.000 --> 15:08.000
Now, the next two sections are a bit short, but important.

15:08.000 --> 15:12.000
The first one is where we store basically comments.

15:12.000 --> 15:20.000
So, when we are making a request, and it's a media file, CDN will lead you to Swift.

15:20.000 --> 15:27.000
So, openStance Swift is an object storage system that stores and retrieves data via HTTP.

15:27.000 --> 15:30.000
It's scalable because it's built for distributed system.

15:30.000 --> 15:37.000
So, you send a request to the frontends, the backends do the actual, the actual storing and retrieving.

15:37.000 --> 15:49.000
To give you a sense of scale, we have about 161 objects taking up to 874 terabytes of this wealth without thumbnails.

15:49.000 --> 15:56.000
But if we include thumbnails, this is going to balloon to 2 billion objects and full petabyte of data.

15:56.000 --> 16:01.000
Oh, concluding with events.

16:01.000 --> 16:08.000
I'm not going to try to explain our event platform, primarily because I don't think anyone can.

16:08.000 --> 16:20.000
But I'm going to mention Kafka, which is named, we're using Apache Kafka, and it is named after a famous author who is famous for depicting hopelessness and absurdity.

16:20.000 --> 16:23.000
And that would be it at that.

16:23.000 --> 16:29.000
It's a streaming process platform that fetches data feeds, that for real-time feeds,

16:29.000 --> 16:35.000
and we're using it for sending events like we could text with time-plating refresh.

16:35.000 --> 16:40.000
Edgecash is purging, cross-we-key links, and all that.

16:40.000 --> 16:44.000
And, cheers for bearing with me. This is the last section.

16:45.000 --> 16:49.000
The unhealthy cash part. So, some of this content here is not mine.

16:49.000 --> 16:54.000
It has been presented by the people who are actually building our front light defenses in this.

16:54.000 --> 16:59.000
Whether we realize it or not, caching is part of our everyday life, right?

16:59.000 --> 17:04.000
You keep posted with your Wi-Fi passport on your fridge for your guests.

17:04.000 --> 17:07.000
You keep your Netflix password on your mom's fridge.

17:07.000 --> 17:09.000
And, not forget, cash secured.

17:09.000 --> 17:12.000
The bugs, they have given us all the best logos and tech history.

17:12.000 --> 17:15.000
And, for a while, the researchers was like, all the right, right?

17:15.000 --> 17:17.000
They were so cool.

17:17.000 --> 17:24.000
But, jokes aside, there's some common strategies to warm cashes up and invalidate them.

17:24.000 --> 17:29.000
Warm them up is like fetching an cash miss, and pre-warm, or pre-warm,

17:29.000 --> 17:34.000
like fetch items before the expire, or when data changes.

17:34.000 --> 17:39.000
Well, for invalidation, we've got several approaches with the most common ones being

17:39.000 --> 17:45.000
TTL expiration, or invalidating on specific events.

17:45.000 --> 17:49.000
Now, just like any other web shop, edge caching is crucial to us too.

17:49.000 --> 17:55.000
Over 90% of our requests are actually completed right there on the edge.

17:55.000 --> 18:01.000
Now, browsers and CDNs cash response URLs, cash responses for URLs,

18:01.000 --> 18:05.000
and application layers caching, application layer caches,

18:05.000 --> 18:09.000
store the data needed to build those responses.

18:09.000 --> 18:12.000
So, caching systems are optimized to improve performance,

18:12.000 --> 18:16.000
and they do that by analyzing human behavior.

18:16.000 --> 18:21.000
So, we all have optimized our systems for human traffic.

18:21.000 --> 18:24.000
As users, when we are browsing, we are scrolling down,

18:24.000 --> 18:26.000
we are clicking things, we're going down,

18:26.000 --> 18:30.000
and in winter out in a semi predictable way.

18:30.000 --> 18:36.000
And from our end, we are budging capacity to handle traffic spikes, right?

18:36.000 --> 18:41.000
So, for us, the latest paper announcement, broker all-time record,

18:41.000 --> 18:45.000
with traffic peeking at 800,000 requests per second,

18:45.000 --> 18:48.000
that's like six times our usual traffic.

18:48.000 --> 18:51.000
And bandwidth hitting a 134 gigabits per second,

18:51.000 --> 18:54.000
that's like twice our traffic, and we survived that.

18:54.000 --> 18:58.000
Which is a proof that we've built our systems to bend, but not,

18:58.000 --> 19:02.000
when the circumstances demanded, demanded, but not break.

19:02.000 --> 19:08.000
Since January 20, 2014, we saw a 50% increase of bandwidth usage,

19:08.000 --> 19:10.000
mostly used by bots.

19:10.000 --> 19:13.000
Now, this is where it all falls apart.

19:13.000 --> 19:17.000
LLM crawlers and AI agents, they are rapidly increasing web traffic,

19:17.000 --> 19:21.000
and it feels like every AI starter has been like hovering data for data,

19:21.000 --> 19:24.000
and they do not just scrape raw HTML.

19:24.000 --> 19:27.000
They target reach media with detailed metadata,

19:27.000 --> 19:29.000
like what we have in common.

19:29.000 --> 19:32.000
It's like they're prime training material.

19:32.000 --> 19:34.000
We have found ourselves in all situations,

19:34.000 --> 19:38.000
where our cross data center links are saturated,

19:38.000 --> 19:42.000
because bots are scraping and cash content from the edges.

19:42.000 --> 19:48.000
And they are so wild, like nobody's business.

19:49.000 --> 19:52.000
So the thing is that, unlike older bots, they do not play nice.

19:52.000 --> 19:55.000
They are distributed, they are coming from all over the place.

19:55.000 --> 19:56.000
They're unpredictable.

19:56.000 --> 19:58.000
They are scraping horizontally.

19:58.000 --> 20:00.000
One day's letter A is like,

20:00.000 --> 20:02.000
Bollywood actors the second.

20:02.000 --> 20:06.000
And I don't think one to review this films afterwards.

20:08.000 --> 20:12.000
But most of all, they are trying to pretend they are something else.

20:12.000 --> 20:15.000
Like the Google bot, like other LLM's,

20:16.000 --> 20:20.000
and they're trying hard to impersonate humans.

20:20.000 --> 20:23.000
So residential proxies are the new black.

20:23.000 --> 20:27.000
For those not aware, residential proxies

20:27.000 --> 20:31.000
route traffic through residential internet connections,

20:31.000 --> 20:34.000
which is pretty shady, because it's often unclear

20:34.000 --> 20:36.000
if the user actually gave consent.

20:36.000 --> 20:38.000
As far as the name user knows,

20:38.000 --> 20:43.000
they are just downloaded solitaire game for the phone.

20:43.000 --> 20:45.000
And sadly, we are paying for it.

20:45.000 --> 20:48.000
We are traffickers, computational costs,

20:48.000 --> 20:51.000
particle assumption, monitoring fatigue from people

20:51.000 --> 20:54.000
getting page and paying a welcome old game

20:54.000 --> 20:56.000
to find the next bot.

20:56.000 --> 20:59.000
And it's same for everyone else on the internet.

20:59.000 --> 21:02.000
Articles, blogs, media, news cell,

21:02.000 --> 21:04.000
sites, code review, platforms,

21:04.000 --> 21:08.000
even testing systems and our users themselves.

21:08.000 --> 21:11.000
So let's cash everything.

21:12.000 --> 21:14.000
Another pre-let idea.

21:14.000 --> 21:16.000
The dynamic response says,

21:16.000 --> 21:19.000
your parameters may change the rendering of a page.

21:19.000 --> 21:21.000
Each page exists in multiple formats.

21:21.000 --> 21:24.000
Like, jizz it and compress mobile,

21:24.000 --> 21:26.000
best of languages, locals.

21:26.000 --> 21:28.000
So we wouldn't be caching one thing,

21:28.000 --> 21:31.000
but rather caching many variants over the same thing.

21:31.000 --> 21:33.000
And there's not stop there.

21:33.000 --> 21:35.000
Data is constantly changing,

21:35.000 --> 21:37.000
which means that we're invalidating

21:37.000 --> 21:40.000
and warming up the caches over and over again.

21:40.000 --> 21:43.000
It has regionality.

21:43.000 --> 21:47.000
It's like asking a peter using Italy

21:47.000 --> 21:50.000
to make it to every day bake pineapple pizzas.

21:50.000 --> 21:53.000
Just in case some tourists will pop by.

21:53.000 --> 21:55.000
It will turn up and just ask for one.

21:55.000 --> 21:57.000
It's like a waste of pizzas pineapple and

21:57.000 --> 21:59.000
angry chatty.

21:59.000 --> 22:00.000
All right.

22:00.000 --> 22:02.000
Can we strike that?

22:02.000 --> 22:03.000
Kind of.

22:03.000 --> 22:07.000
Some quick tips are binding the dumb crawlers.

22:07.000 --> 22:10.000
I mean, like ancient user agents.

22:10.000 --> 22:11.000
In terms of explore six,

22:11.000 --> 22:14.000
not going to be doing any science soon.

22:14.000 --> 22:16.000
IP reputation services,

22:16.000 --> 22:18.000
as well as known crawler databases,

22:18.000 --> 22:20.000
can help us categorize traffic

22:20.000 --> 22:22.000
and apply relevant limits.

22:22.000 --> 22:23.000
And finally,

22:23.000 --> 22:25.000
we can have some browser level checks.

22:25.000 --> 22:27.000
JavaScript headers.

22:27.000 --> 22:30.000
J-A-J-4.

22:30.000 --> 22:31.000
J-A-J-4.

22:31.000 --> 22:33.000
J-D-D-3 and J-4 fingerprinting.

22:33.000 --> 22:34.000
Sorry.

22:34.000 --> 22:36.000
J-A fingerprinting is a method

22:36.000 --> 22:40.000
that we identify through TLS and H-T-B hung shakes,

22:40.000 --> 22:42.000
like cyphosuits and extensions

22:42.000 --> 22:45.000
to identify unique clients.

22:45.000 --> 22:48.000
They are all great ways to separate browsers.

22:48.000 --> 22:52.000
Basically what I am describing is web application

22:52.000 --> 22:54.000
firewalls.

22:54.000 --> 22:58.000
So when are my going with this?

22:58.000 --> 23:00.000
SkyNet is going to scrape our websites.

23:00.000 --> 23:02.000
They were round out of resources.

23:02.000 --> 23:04.000
Out of jobs.

23:04.000 --> 23:06.000
Or out of all this impressionators.

23:06.000 --> 23:08.000
Thank you so much.

23:08.000 --> 23:17.000
APPLAUSE

23:17.000 --> 23:19.000
So awkward because we have now to share

23:19.000 --> 23:20.000
microphone.

23:20.000 --> 23:22.000
Since we have a small present.

23:22.000 --> 23:24.000
So nice.

23:24.000 --> 23:28.000
So there will be no questions for this talk.

23:28.000 --> 23:29.000
Do you want to do something?

23:29.000 --> 23:31.000
There are a few inconveniences in the room.

23:31.000 --> 23:32.000
There is staff.

23:32.000 --> 23:33.000
There are volunteers.

23:33.000 --> 23:35.000
Can you please stand out?

23:35.000 --> 23:36.000
Please stand out.

23:36.000 --> 23:37.000
Come on.

23:37.000 --> 23:38.000
Past employees as well.

23:38.000 --> 23:39.000
Past staff, please.

23:39.000 --> 23:46.000
APPLAUSE

23:46.000 --> 23:49.000
This is your work.

23:49.000 --> 23:50.000
Yeah.

23:50.000 --> 23:51.000
I don't count.

23:51.000 --> 23:52.000
Anyways.

23:52.000 --> 23:54.000
So that's it for this talk.

23:54.000 --> 23:56.000
So you can stay for the next one.

23:56.000 --> 23:59.000
If you want to learn something about ruby genes.

23:59.000 --> 24:02.000
Otherwise, have a nice foster.

24:29.000 --> 24:31.000
Thank you.

24:59.000 --> 25:09.000
APPLAUSE