WEBVTT 00:00.000 --> 00:09.840 Oh, my CPU is sometimes slow, and I don't know why, maybe because it's too low power, 00:09.840 --> 00:13.440 maybe because my code is hardly inefficient. 00:13.440 --> 00:17.080 So I actually invited an old colleague of mine, which is nothing to do with why he was 00:17.080 --> 00:18.080 accepted. 00:18.080 --> 00:23.280 But with an amazing talk, he's working on databases in his main life of Grafana, and 00:23.280 --> 00:27.520 it's now going to talk about this, why CPU is a slow and how to fix this? 00:27.520 --> 00:29.600 Right, let's talk last. 00:29.600 --> 00:36.000 Yeah, thank you, everyone. 00:36.000 --> 00:38.400 So yeah, thank you, Mark, for the introduction. 00:38.400 --> 00:44.520 So I'm going to want to show you some research I've done, at some point, started computer 00:44.520 --> 00:48.840 science and forgot everything about it, so it's always good to, I guess, remember some 00:48.840 --> 00:54.560 of the principles, how the CPU is run, and right now I'm working on Grafana Power 00:54.560 --> 00:57.840 scope, which is a database for profiling data. 00:57.840 --> 01:05.040 So it is kind of related to the topic here, so we obviously know Go has P-Prof, and we collect 01:05.040 --> 01:11.960 CPU usage there, and this kind of just shows that we can go deeper if we have to. 01:11.960 --> 01:19.400 I'm not saying that I have to do this a lot, or maybe, ever, but I always had on my list, 01:19.640 --> 01:25.120 to basically figure out a bit more out from the CPU performance metrics, it gives. 01:25.120 --> 01:32.120 So quick overview, over today, so first, we're going to look at how CPUs kind of evolved 01:32.120 --> 01:39.600 over the last couple of years, and basically, I'd like to look at the cache architecture. 01:39.600 --> 01:43.320 I think it fits quite well with the green tea, got a bunch of collected talk a few here 01:43.320 --> 01:50.320 for longer, and then I want to take a look how branch prediction happens in your CPU, 01:50.320 --> 01:56.680 and why sometimes that might happen to you, and finally I want to show how you could 01:56.680 --> 02:04.240 measure this, and what kind of tools are available, and hopefully, a quick demo on something 02:04.480 --> 02:10.440 benchmark that I run remotely, it's going to see it, probably. 02:10.440 --> 02:16.920 Here's maybe just a quick history, so I actually own the two on the left side in my laptops. 02:16.920 --> 02:25.920 So, 20 years ago, a core to do, had two cores, and if we now kind of look over the years, 02:25.920 --> 02:32.800 we are now more at the 20 core mark, if you buy a new laptop this year, so that's very 02:32.880 --> 02:38.960 other metrics I put on, I guess the summary is maybe we have roughly 10x most strengths 02:38.960 --> 02:45.840 in those 20 years until now, so I guess something notable, the kind of frequency seemed to 02:45.840 --> 02:53.040 hit some ceiling, so we're still talking about a base frequency of trigger heads, but it 02:53.040 --> 02:58.240 can obviously go a lot higher, and also the cache become bigger, so you can see that in the 02:58.320 --> 03:10.800 kind of very recently the level 1 has increased from 64 kB to 192, I think, so basically all 03:10.800 --> 03:16.720 in all CPUs are faster than 20 years ago, what I would say has not really changed, 03:16.720 --> 03:22.000 it's that kind of like how they relate to each other different timings in how the CPU 03:22.000 --> 03:29.600 access its memory, so I guess if I go to the level 1 cache, we are in the kind of area of 03:29.600 --> 03:37.440 one nanosecond, so I don't know the cycle is maybe 0.2 nanoseconds, and so basically like 03:37.440 --> 03:43.840 over that time it might have gotten smaller, but what didn't really change is that roughly 10x 03:43.840 --> 03:49.440 is like what the last level cache has in terms of latency, and then if you go to RAM, you probably 03:49.440 --> 04:00.480 are the 100 x of the level 1 cache latency time, and then obviously the further you go away from 04:00.480 --> 04:07.600 from your CPU, the longer this latency will take, I guess an NVME is more in 10 microseconds like 04:08.480 --> 04:14.240 milliseconds for HDDs, I don't know, I put a confiner dashboard to put it in perspective, 04:14.240 --> 04:18.720 it's maybe more 10 seconds that you want to auto reload it, and I guess if you want to get 04:19.600 --> 04:28.720 at first term, it's probably some outside of this room, so let's look at this CPU cache architecture 04:28.720 --> 04:38.960 a bit more in detail, so like when you look at a very simple 2 core machine, you will find 04:39.840 --> 04:46.640 that each core has its dedicated cache in level 1 and level 2, and then the last level cache 04:46.640 --> 04:54.000 is typically shared, so you might have very beefy instances where it's not shared across all 04:54.000 --> 04:59.760 cores with different sockets of CPU, but that is really typical in the kind of consume CPUs, 04:59.760 --> 05:05.040 and also in the architecture that we mostly work with, or I mostly work with a cache line would be 05:05.040 --> 05:11.520 64 bytes long, so I guess no matter what you read one byte, you read at least the 64 05:12.320 --> 05:18.880 of the cache line, so that's the minimum unit, and so the way I guess the CPU operates on those 05:18.880 --> 05:24.880 cache, the CPU tries to figure out what you're up to, and tries to pre-fetch the data, so it's 05:24.880 --> 05:30.720 ready in the cache, so that it doesn't have to go through this, like long cycle of 100 nanoseconds 05:31.360 --> 05:43.920 to get it from memory, and so in order for your code to kind of help the CPU, I guess CPUs are 05:43.920 --> 05:49.600 optimized for this locality, so it's either the temporal locality, you access a variable now, 05:49.600 --> 05:55.120 it's very likely you're going to come back to this variable, and add another one to it, or add a 05:55.120 --> 06:01.200 like, yeah, store some intermediate results and access them very soon after, but then it's also 06:01.200 --> 06:10.480 very likely when you read, I guess in a memory it offset zero, you likely go through that array 06:10.480 --> 06:19.920 and go to the next element or with an offset of 10 to the one after, so basically like as long as 06:19.920 --> 06:25.680 you allow your code to kind of move in these ways, I guess you're going to help the CPU to kind of 06:26.400 --> 06:33.440 predict and allow the CPU to pre-fetch the memory, which will make it easier to not have to 06:33.440 --> 06:42.880 have that latency penalties, yeah, at the time when you want to access memory, so this is obviously 06:42.880 --> 06:47.280 very important for your data structures, I guess we sort of have kind of example, like if you have 06:47.920 --> 06:54.880 either linked list or a tree with a lot of pointers that are very randomly jumping memory space, 06:54.880 --> 07:00.560 then you will face this penalty heavily, because obviously there's no way for the CPU to predict 07:00.560 --> 07:08.720 where the layout of your tree, and therefore you need to kind of think a bit harder about how 07:08.720 --> 07:15.600 you can organize this, and like I guess one very common example where I face that a bit, it's like 07:15.600 --> 07:20.720 I guess go maps when you iterate over them, they come back very randomly and so it might be really 07:20.720 --> 07:27.600 worth I guess considering to sort them, if you have indexes for example coming out of a map, 07:30.480 --> 07:38.480 like I guess another very typical I guess go example is this kind of, I don't know, you treat 07:38.560 --> 07:44.560 a struct a bit like, I don't know, object oriented, so you have, in this example I have a 07:44.560 --> 07:51.360 person database, you might have like millions of persons in your database, and obviously if you're 07:51.360 --> 07:58.880 application and your access pattern is only ever one person, then this kind of array slice of 07:58.880 --> 08:04.560 structs, I guess a person each is ideal, because obviously you only operate on one person, 08:04.640 --> 08:10.320 it's where we local, but if your database is more kind of comparing names with each other, and 08:11.280 --> 08:19.200 I guess never really reading the whole person, it's a lot better to actually design a struct 08:19.200 --> 08:25.760 with a lot of arrays in there, so the data is tightly together, I guess this, obviously it has 08:25.760 --> 08:32.080 a lot to do with a cash line length, I guess I think I had a from in 64 or in 64, but it's 08:32.160 --> 08:37.760 we have the 8 bytes, and then obviously the cash line has 56 bytes, you lose, so if you read through 08:37.760 --> 08:45.120 the, I guess, slice struct, you have a lot of overhead and waste a lot of memory and with off your, 08:47.520 --> 08:56.400 when you when you can run that application, like something CPUs have to do as well, 08:57.360 --> 09:02.960 is they need to kind of understand where your code goes, so when you have a branch, like an if 09:02.960 --> 09:12.240 condition, I guess it is something where basically the CPU wants to understand which branch you're 09:12.240 --> 09:17.680 going to take next, and in order to kind of understand how that works, I'm going to repeat, 09:18.400 --> 09:25.760 I guess some of a very kind of old CPU design, but it's pretty much unchanged with a lot more extra 09:25.840 --> 09:32.800 complexities, but what CPUs do, they do pipelining, and the reason for that is, there obviously 09:34.240 --> 09:39.600 have a lot of different circuits that do different things, and you want to make best use of them, 09:40.240 --> 09:45.920 and I guess what you also need to think about, one instruction is an obron in one cycle, so you run 09:45.920 --> 09:52.400 part of an instruction in one cycle, and I guess to order or like to to make best use of your circuits, 09:52.960 --> 09:58.560 that you have in hardware, you do some kind of pipelining, I guess, it's a bit like a car manufacturing 09:58.560 --> 10:07.520 assembly lines, so you basically have like this pattern where you, so an instruction first step 10:07.520 --> 10:12.960 is usually like fetching the instruction, obviously you have the circuit for it, and in cycle one, 10:12.960 --> 10:18.160 I guess you use it here for the first instruction, and then in cycle two, the first instruction 10:18.160 --> 10:24.080 obviously goes into the next stage, which is the instruction decoding, and then if you also want to 10:24.080 --> 10:28.880 use your circuit for fetching, you can do that with the next instruction, and basically like this 10:28.880 --> 10:35.360 gives this kind of pipelining, and rather than having five cycles per instruction, you can kind of 10:35.360 --> 10:44.320 do four instructions in eight cycles, seven cycles, and when you now look at more modern CPUs, 10:44.320 --> 10:50.960 they have more like a stage depth of like I think 10 to 20, so obviously this becomes a lot 10:50.960 --> 10:57.120 longer, and also modern CPUs are super scalears, so like they would have multiple instruction 10:57.120 --> 11:04.720 fetch circuits, so they could fetch for at the time or six, and so you can see how this kind of 11:04.720 --> 11:11.200 gets a lot more complex to build up that pipeline and start the pipeline, and now when you 11:11.200 --> 11:17.840 think about a branch, obviously like I tried to kind of annotate the same graph here, 11:17.840 --> 11:24.560 so obviously if this would be your kind of if condition, then obviously you like by that point 11:24.560 --> 11:28.720 where you want to start the next instruction, you actually don't know that I would come 11:28.720 --> 11:33.520 off the if condition yet, and so you need to know kind of what your best guess is what your most 11:33.600 --> 11:42.080 likely outcome that is, and so I guess in this case the CPU was predicting the true branch 11:42.080 --> 11:46.720 to be the one that that is the likelyest outcome, and therefore started running it before even 11:46.720 --> 11:53.680 knowing if it acts as really true, and then I guess in the best case somewhere here, it turns out 11:53.680 --> 12:00.800 it true, it's true, and so the CPU kind of won and was able through the branch prediction to save some 12:00.800 --> 12:09.200 time and have more efficiency, I guess at worst it needs to basically stop and kind of go back 12:09.200 --> 12:17.680 replant the work and start the pipeline again, in order to I guess continue at the force 12:18.960 --> 12:27.840 side of things, so that more than CPUs can get quite complex in how they do this prediction, 12:27.920 --> 12:34.800 so I guess most of the time there is like a similar kind of structure of level one level two, 12:34.800 --> 12:43.440 but it's still branches that they've taken, so obviously like you might have code that branches 12:43.440 --> 12:48.000 quite a lot, and so you add a lot of pressure to that, and so you might forget this decisions, 12:48.880 --> 12:55.440 therefore like this if you were not able to kind of get this prediction right or wrong, 12:56.000 --> 13:04.560 I'm having a kind of simple go code as well, and by the way all my examples they should have a link 13:04.560 --> 13:09.760 in the PDF and you can run them yourself and maybe figure out that the benchmark showed that as well, 13:10.880 --> 13:17.200 so in this case we have an input of int and we want to decide if the negative of positive 13:17.200 --> 13:22.640 and count up how many positives we've seen, how many negatives we've seen, so you can see the 13:22.640 --> 13:32.400 code is obviously very simple and straightforward, and depending on your input, your input could 13:32.400 --> 13:37.520 be very kind of ordered, so you might just have all the minuses first and then at some point, 13:37.520 --> 13:44.800 your kind of number switch from minus one to one, or whatever the kind of bit this way of jumps, 13:44.800 --> 13:50.080 you might have a branch mistake, but everything else is just straight pipeline being very 13:50.080 --> 13:56.400 tied and a lot of instructions around, if you don't have random numbers, obviously you 13:57.360 --> 14:05.520 I guess at worst will face like 50% of branch misses because it could happen for any of the 14:05.520 --> 14:12.800 the conditions that I guess the CPU made the wrong choice, I guess when we're creating that example, 14:12.800 --> 14:20.560 I actually spend a bit of time looking at the assembly because sometimes the compiler does 14:20.560 --> 14:26.560 small things, I think the go compiler is maybe a bit less liable to do this, but if you want to 14:26.560 --> 14:30.800 figure out if there's a branch, it's the easiest to look into the assembly and see the 14:30.800 --> 14:40.000 kind of instructions that it generates, and so if you want now want to avoid those branch misses, 14:40.880 --> 14:48.560 I guess the main two ways are either making the data predictable, so we have very good prediction 14:49.440 --> 14:57.040 or getting rid of the branch encode, so in this case, because it is a very simple example, 14:57.040 --> 15:04.160 I guess somewhat straightforward to think of an solution that will basically like do with the other 15:04.160 --> 15:11.440 metrics, the same thing that the branch would achieve, so in this case we do some shifting of the 15:11.440 --> 15:20.560 negative bits, and if the negative bit is obviously one, the negative bit is one, when it's 15:20.560 --> 15:26.160 a negative number, so we add that to the negative count, and if not, then we added to the positive, 15:26.160 --> 15:34.880 so we kind of swapped our branching for a bit more algorithmistics, and therefore I guess the code 15:34.880 --> 15:42.320 can run somewhat straightforward. Also, I think there's been a lot of research, I went researching that 15:42.320 --> 15:47.600 at the talk, I had to really stop myself a lot to not read this paper, about whatever neural 15:47.600 --> 15:52.960 network for our prediction, and so I think it's a very interesting thing to dive a bit more into, 15:53.040 --> 15:59.280 but modern like Ryzen CPUs for example, they use a second stage where they have a neural network 15:59.280 --> 16:07.920 that I think a lot of that is not very open, but people kind of were able to prove from looking 16:09.920 --> 16:18.240 running code on those CPUs, so obviously now we come to the more what I wanted to do in this talk, 16:18.320 --> 16:23.280 so how can I actually use that, and measure that proof that this is actually what happens, 16:23.280 --> 16:31.920 and so CPUs have ways to tell you, so there's a performance monitoring unit that has counters, 16:31.920 --> 16:39.360 and you are able to read those out, so very typical counters are how many cycles are run, 16:39.360 --> 16:45.520 how many instructions have been run, then there's a really detailed information about the cache 16:45.520 --> 16:52.960 misses and hits, do you read or write from cache, and the branching has, as well, 16:52.960 --> 16:59.680 like it counts the branches, it counts the branch misses, and it always is a bit CPU vendor, 16:59.680 --> 17:05.680 depending when you want to go more into customer events, so you can give it a hex code, 17:05.680 --> 17:10.000 and you can basically find in the manual of your CPU vendor what you use, 17:10.640 --> 17:16.560 like obviously one thing to do that is just get the counter for your test, and you have an idea, 17:16.560 --> 17:21.280 I guess how it performs to maybe another test case or a benchmark where they code change, 17:22.320 --> 17:31.520 but it's also possible to profile and actually get the instruction that kind of 17:31.520 --> 17:38.480 was very close to that point branch miss happened, that can be some timing issues, 17:38.480 --> 17:43.520 so it might be that the instruction that that shows up there is actually a bit after when it 17:43.520 --> 17:49.520 really happened, it's just CPUs are not getting that timing right, especially the more consumer ones, 17:51.120 --> 17:58.720 and there's also methods to kind of reduce that in accuracy, but yeah it was really very CPU 17:58.720 --> 18:03.520 dependency, you need to check kind of, so don't be surprised when an instruction that 18:04.000 --> 18:10.240 that is seemingly not related as some high counters, you basically just need to look very close to 18:10.240 --> 18:17.920 when the execution apps are there. I guess probably the biggest problem around PMUs is 18:18.640 --> 18:24.400 for virtual machines, it's kind of tricky to emulate them, because I guess they are in hardware, and if you're 18:25.600 --> 18:33.040 having very small non-dediated course in your VMs, it is kind of impossible to 18:34.000 --> 18:44.720 create that data in the kind of hypervisor level, so in Grafana we kind of run on the big cloud providers, 18:44.720 --> 18:50.000 and the bigger the instances, the more likely you get those PMUs, in GCP, you need to see 18:50.000 --> 18:56.000 set a special flag, and I found we run quite a lot of workloads on ARM, and they seem to have 18:56.960 --> 19:03.360 like even the smaller instances have pretty good PMU pass through, these are links, 19:03.360 --> 19:09.440 so you can kind of read which instance types are kind of providing that data for you. 19:11.280 --> 19:16.160 So I guess one of the biggest problems when I'm doing benchmark, the benchmark loop, changing 19:16.160 --> 19:23.120 code, run benchmark, compared to results, my name files really badly, and I lose all the code all 19:23.120 --> 19:29.600 the time and to get stash and whatever, and so I kind of lost November, I think I was fed up, 19:29.600 --> 19:34.080 and I wrote a little too much basic stores to benchmark results while I run, and some kind of 19:34.080 --> 19:44.320 gets committed to code together, and as I always wanted to kind of go a bit more into those 19:44.320 --> 19:49.120 PMUs and kind of actually like basically measure the benchmark in understand better, 19:49.120 --> 19:55.200 why something is better or not, and if my assumptions are right, I used the opportunity to just 19:55.200 --> 20:02.560 submit this talk, and then for the typical way for me to get something done is to submit a talk, 20:02.560 --> 20:09.120 and then I forced myself to kind of implement this. So I basically wrote a little wrapper around 20:09.200 --> 20:17.120 go tests, and the Perth tool on Linux, which is able to get the information out of the PMUs 20:17.120 --> 20:23.520 of your CPU, present them, and it also fixes that problem of storing the benchmark together with 20:23.520 --> 20:28.400 the code changes and what I've run. So I guess one additional problem for me, so you can see 20:28.400 --> 20:33.200 the MacBook, so there's no Perth available, I think there's some Mac based solution which 20:33.280 --> 20:42.480 fits a lot of XML, which wasn't really easy to kind of make sense of, and so when I tried to do 20:42.480 --> 20:49.120 this before, I always had to compile and ship into Linux instance, and so I also integrated the way to 20:49.840 --> 20:57.360 do an SSH remote run off of benchmark, and like one of the other goals was I want to use PProph to 20:57.440 --> 21:04.240 look at the profiles that Perth generates. Yeah, I can navigate Perth's report, but like I'm 21:04.240 --> 21:09.040 definitely a less proficient as in looking at Flamerafs and using PProph for that. 21:10.400 --> 21:15.920 And so yeah, you will find it on GitHub, the link should be on the on the speak information. 21:15.920 --> 21:20.800 I mostly have two modes, the test mode, which runs, there's a go test with a wrapper around it 21:20.800 --> 21:26.000 to collect those metrics, and the other one is a touch mode where you can run a port, 21:26.000 --> 21:33.840 given you are able to create a privilege port, you basically are able to run that in your Kubernetes 21:33.840 --> 21:38.000 cluster. You then still need to have access to the PMU from your cloud provider that we 21:38.000 --> 21:44.720 end problem now mentioned, but yeah, no, that's basically like a pretty useful, I guess, to look 21:44.720 --> 21:51.040 in how things are running in production. And so I know attempting a demo, so think is crossed. 21:56.000 --> 21:57.000 Oops, oops, oops, oops. 21:57.000 --> 21:58.600 Why Shitone very complicated! 21:58.600 --> 22:00.080 I guess it wasn't that simple? 22:00.080 --> 22:10.840 Unfortunately, this wasn't a really simple step on other words, so like, we are actually trying nil 22:10.840 --> 22:15.340 to create a query per search on the internet so now we can kind of create some content not 22:15.340 --> 22:21.100 more complex and securely describe this in the way we run a kind of special shape. 22:21.100 --> 22:31.100 So, I kind of know that typing and presenting is not super easy, so I kind of automate it a little bit. 22:31.100 --> 22:39.100 So, I'm going to run the Perhance prediction example from earlier, and I guess as a start, let's maybe look quickly. 22:39.100 --> 22:45.100 Remember, the kind of implementation, so we had that example with negative numbers counting up. 22:45.100 --> 22:55.100 With here with the branch first, and then in varying two, it is the branchless approach with the additional arrow matrix. 22:55.100 --> 23:03.100 So, now, if I want to run a benchmark, I guess this is kind of roughly how I run benchmarks and go. 23:03.100 --> 23:13.100 I have four benchmarks, first of branch version with predictable data, then with random data, and then the branchless with those two data variants. 23:13.100 --> 23:17.100 So, I guess I run it here on my MacBook for now. 23:17.100 --> 23:31.100 So, you can see or not see, you can see the four benchmarks, and I guess the fastest one is the one with a branch, and with predictable data. 23:31.100 --> 23:40.100 Then in the second line, you can see the one with random data, so which is like slower, but maybe not as slow as you would expect. 23:40.100 --> 23:55.100 So, this has to do with arm, and the less deep kind of complexity of the CPU, and therefore less stages in the pipeline, and so the cost is not as high, but it is still a penalty of, I guess, around 50% extra. 23:55.100 --> 24:06.100 And then we have the branchless version, which is not as good as the branch version, but it is very static, I guess, for both of the random and the predictable data input. 24:06.100 --> 24:15.100 And so, if I now want to run the same thing with that perf go tool, you basically just need to run perf in front of go. 24:15.100 --> 24:24.100 I try to keep the variables everything kind of gets past through, and this runs on my machine, and basically does just the same thing. 24:24.100 --> 24:34.100 What do you get from here? We are going to be able to list all the runs you did, and get the kibbit hash, and it also saves the test binary. 24:34.100 --> 24:40.100 If you ever wanted to run it again, you could really run it fairly easily. 24:40.100 --> 24:59.100 So, now, in order to do kind of the first assessment of those metrics from the CPU, perf has the static command, and that gives you the counters for the whole of the process you run, or for everything running on the node, depending on the flags you give it. 24:59.100 --> 25:13.100 In this case, we just run this kind of the worst case, the one with the branch and random data, and we run it with stat, and so you can't configure it, or I'm on a MacBook, there's no perf basically. 25:13.100 --> 25:30.100 So, my desktop computer at home is hopefully online, so I kind of added remote holes, and that could be just any SSH, like target as long as it's kind of connectable. 25:30.100 --> 25:51.100 It basically will do exactly the same, but it will compile it for the correct target, so we are connected, kind of figures out, yeah, this is Linux in D64, and basically send the right binary rapid with perf, run the perf command, and that's kind of the perf command output. 25:51.100 --> 26:11.100 So, you can kind of see the metrics it shows by default, I guess, pretty wants to start with, so I'm not super familiar with any more beyond this, and I definitely think you find quite a lot of stack overflow information, or even the agents know quite a lot about how to interpret it, but also perf gives you an indication of the color. 26:11.100 --> 26:40.100 So, you can see that the front end of the CPU, I guess, is that bit with the instruction fetch and instruction decode, and you can see that's kind of where the stalls happen, but what you also see is branch with the 14% and that's kind of the highest I could generate a benchmark for, so definitely like naively would expect 50% as the worst you would get, but there's a lot of smartness going on that you can't get that far. 26:40.100 --> 27:00.100 So typically you would see more like in the low zero point branch misses, whenever I profile something running, so this is really obviously the absolute worst case where we don't nothing else than just this loop, and so obviously this is the obvious next thing to look into from this output. 27:00.100 --> 27:18.100 Now you can basically like profile to get the functions, the line numbers, I guess everything that is related to what I changed here is I did profile, and I added the event file branch misses. 27:18.100 --> 27:26.100 You can collect multiple events, but there are hard valid limitations, and then perf will kind of switch out the events and as you run it. 27:26.100 --> 27:38.100 Typically I think there's like six in the CPU, six event types that you can collect, and so yeah it can only do six at a time if you give it more than six. 27:38.100 --> 27:47.100 But in this case, I also decided to run the whole benchmark suite, so I'm no longer filtering on this one benchmark just so you can see the comparison in the profile. 27:47.100 --> 28:06.100 So I run the same command, just I guess been profile, and you can see also a bit how this one takes a lot longer on the horizon CPU, because it's an AMD 64 versus the arm, and now we should have the profile already. 28:06.100 --> 28:16.100 So it basically passes the part of output generates a p-prof, and I need to see. 28:16.100 --> 28:43.100 So I guess this is the p-prof that, I don't know what's going on. 28:43.100 --> 29:04.100 So you can kind of see the unit, we collect should be branch misses, I can't really read it now. 29:04.100 --> 29:11.100 And then you can see that 100% should be close to 100%. 29:11.100 --> 29:24.100 But a very huge part goes to this benchmark branch with random, and if you would want to look into the source code. 29:24.100 --> 29:40.100 And you can kind of see this executed here, I guess in those lines, you get the instructions, I guess although the goodies you get from p-prof you haven't, or you can upload them to the same tools. 29:40.100 --> 29:47.100 I don't know if you can see the better of benchmark time. 29:47.100 --> 29:53.100 I think time is over, what is it? One minute. 29:53.100 --> 30:11.100 And you don't know what I think, maybe done a quick jump back to the market. 30:11.100 --> 30:20.100 And just showing that all the mode quickly, so with attach, this is a Kubernetes cluster that I'm hopefully also connected to. 30:20.100 --> 30:26.100 So you can select the port, you can tell it, collect me branch misses for this port. 30:26.100 --> 30:33.100 And it kind of does the same, obviously you don't have to compile a test, the port is already running, it finds out the PIDs and touches them. 30:33.100 --> 30:40.100 To the perfect command, collect the profile, runs it back, it steals the binary from the remote side. 30:40.100 --> 30:44.100 It also makes a lot of this manual work that I had before. 30:44.100 --> 30:57.100 And yeah, I guess you can take a look, it's not a bit longer depending on internet speed, but it basically runs for 10 seconds, collect the branch misses, and for me, if you would then see depending on what you do. 30:57.100 --> 31:06.100 Well, you're running, maybe a query, iterators have a lot of branch misses, and yeah, I guess you can then decide if this is both optimizing or not. 31:06.100 --> 31:10.100 Okay. 31:10.100 --> 31:19.100 While we're waiting across. 31:19.100 --> 31:25.100 And wow, not only live them away, but also trusting the first them wide variety work, amazing. 31:25.100 --> 31:30.100 Yeah, five key, five key.