WEBVTT 00:00.000 --> 00:09.840 Oh, my CPU is sometimes slow, and I don't know why, maybe because it's too low power, 00:09.840 --> 00:13.440 maybe because my code is hardly inefficient. 00:13.440 --> 00:17.080 So I actually invited an old colleague of mine, which is nothing to do with why he was 00:17.080 --> 00:18.080 accepted. 00:18.080 --> 00:23.280 But with an amazing talk, he's working on databases in his main life of Grafana, and 00:23.280 --> 00:27.520 it's now going to talk about this, why CPU is a slow and how to fix this? 00:27.520 --> 00:29.600 Right, let's talk last. 00:29.600 --> 00:36.000 Yeah, thank you, everyone. 00:36.000 --> 00:38.400 So yeah, thank you, Mark, for the introduction. 00:38.400 --> 00:44.520 So I'm going to want to show you some research I've done, at some point, started computer 00:44.520 --> 00:48.840 science and forgot everything about it, so it's always good to, I guess, remember some 00:48.840 --> 00:54.560 of the principles, how the CPU is run, and right now I'm working on Grafana Power 00:54.560 --> 00:57.840 scope, which is a database for profiling data. 00:57.840 --> 01:05.040 So it is kind of related to the topic here, so we obviously know Go has P-Prof, and we collect 01:05.040 --> 01:11.960 CPU usage there, and this kind of just shows that we can go deeper if we have to. 01:11.960 --> 01:19.400 I'm not saying that I have to do this a lot, or maybe, ever, but I always had on my list, 01:19.640 --> 01:25.120 to basically figure out a bit more out from the CPU performance metrics, it gives. 01:25.120 --> 01:32.120 So quick overview, over today, so first, we're going to look at how CPUs kind of evolved 01:32.120 --> 01:39.600 over the last couple of years, and basically, I'd like to look at the cache architecture. 01:39.600 --> 01:43.320 I think it fits quite well with the green tea, got a bunch of collected talk a few here 01:43.320 --> 01:50.320 for longer, and then I want to take a look how branch prediction happens in your CPU, 01:50.320 --> 01:56.680 and why sometimes that might happen to you, and finally I want to show how you could 01:56.680 --> 02:04.240 measure this, and what kind of tools are available, and hopefully, a quick demo on something 02:04.480 --> 02:10.440 benchmark that I run remotely, it's going to see it, probably. 02:10.440 --> 02:16.920 Here's maybe just a quick history, so I actually own the two on the left side in my laptops. 02:16.920 --> 02:25.920 So, 20 years ago, a core to do, had two cores, and if we now kind of look over the years, 02:25.920 --> 02:32.800 we are now more at the 20 core mark, if you buy a new laptop this year, so that's very 02:32.880 --> 02:38.960 other metrics I put on, I guess the summary is maybe we have roughly 10x most strengths 02:38.960 --> 02:45.840 in those 20 years until now, so I guess something notable, the kind of frequency seemed to 02:45.840 --> 02:53.040 hit some ceiling, so we're still talking about a base frequency of trigger heads, but it 02:53.040 --> 02:58.240 can obviously go a lot higher, and also the cache become bigger, so you can see that in the 02:58.320 --> 03:10.800 kind of very recently the level 1 has increased from 64 kB to 192, I think, so basically all 03:10.800 --> 03:16.720 in all CPUs are faster than 20 years ago, what I would say has not really changed, 03:16.720 --> 03:22.000 it's that kind of like how they relate to each other different timings in how the CPU 03:22.000 --> 03:29.600 access its memory, so I guess if I go to the level 1 cache, we are in the kind of area of 03:29.600 --> 03:37.440 one nanosecond, so I don't know the cycle is maybe 0.2 nanoseconds, and so basically like 03:37.440 --> 03:43.840 over that time it might have gotten smaller, but what didn't really change is that roughly 10x 03:43.840 --> 03:49.440 is like what the last level cache has in terms of latency, and then if you go to RAM, you probably 03:49.440 --> 04:00.480 are the 100 x of the level 1 cache latency time, and then obviously the further you go away from 04:00.480 --> 04:07.600 from your CPU, the longer this latency will take, I guess an NVME is more in 10 microseconds like 04:08.480 --> 04:14.240 milliseconds for HDDs, I don't know, I put a confiner dashboard to put it in perspective, 04:14.240 --> 04:18.720 it's maybe more 10 seconds that you want to auto reload it, and I guess if you want to get 04:19.600 --> 04:28.720 at first term, it's probably some outside of this room, so let's look at this CPU cache architecture 04:28.720 --> 04:38.960 a bit more in detail, so like when you look at a very simple 2 core machine, you will find 04:39.840 --> 04:46.640 that each core has its dedicated cache in level 1 and level 2, and then the last level cache 04:46.640 --> 04:54.000 is typically shared, so you might have very beefy instances where it's not shared across all 04:54.000 --> 04:59.760 cores with different sockets of CPU, but that is really typical in the kind of consume CPUs, 04:59.760 --> 05:05.040 and also in the architecture that we mostly work with, or I mostly work with a cache line would be 05:05.040 --> 05:11.520 64 bytes long, so I guess no matter what you read one byte, you read at least the 64 05:12.320 --> 05:18.880 of the cache line, so that's the minimum unit, and so the way I guess the CPU operates on those 05:18.880 --> 05:24.880 cache, the CPU tries to figure out what you're up to, and tries to pre-fetch the data, so it's 05:24.880 --> 05:30.720 ready in the cache, so that it doesn't have to go through this, like long cycle of 100 nanoseconds 05:31.360 --> 05:43.920 to get it from memory, and so in order for your code to kind of help the CPU, I guess CPUs are 05:43.920 --> 05:49.600 optimized for this locality, so it's either the temporal locality, you access a variable now, 05:49.600 --> 05:55.120 it's very likely you're going to come back to this variable, and add another one to it, or add a 05:55.120 --> 06:01.200 like, yeah, store some intermediate results and access them very soon after, but then it's also 06:01.200 --> 06:10.480 very likely when you read, I guess in a memory it offset zero, you likely go through that array 06:10.480 --> 06:19.920 and go to the next element or with an offset of 10 to the one after, so basically like as long as 06:19.920 --> 06:25.680 you allow your code to kind of move in these ways, I guess you're going to help the CPU to kind of 06:26.400 --> 06:33.440 predict and allow the CPU to pre-fetch the memory, which will make it easier to not have to 06:33.440 --> 06:42.880 have that latency penalties, yeah, at the time when you want to access memory, so this is obviously 06:42.880 --> 06:47.280 very important for your data structures, I guess we sort of have kind of example, like if you have 06:47.920 --> 06:54.880 either linked list or a tree with a lot of pointers that are very randomly jumping memory space, 06:54.880 --> 07:00.560 then you will face this penalty heavily, because obviously there's no way for the CPU to predict 07:00.560 --> 07:08.720 where the layout of your tree, and therefore you need to kind of think a bit harder about how 07:08.720 --> 07:15.600 you can organize this, and like I guess one very common example where I face that a bit, it's like 07:15.600 --> 07:20.720 I guess go maps when you iterate over them, they come back very randomly and so it might be really 07:20.720 --> 07:27.600 worth I guess considering to sort them, if you have indexes for example coming out of a map, 07:30.480 --> 07:38.480 like I guess another very typical I guess go example is this kind of, I don't know, you treat 07:38.560 --> 07:44.560 a struct a bit like, I don't know, object oriented, so you have, in this example I have a 07:44.560 --> 07:51.360 person database, you might have like millions of persons in your database, and obviously if you're 07:51.360 --> 07:58.880 application and your access pattern is only ever one person, then this kind of array slice of 07:58.880 --> 08:04.560 structs, I guess a person each is ideal, because obviously you only operate on one person, 08:04.640 --> 08:10.320 it's where we local, but if your database is more kind of comparing names with each other, and 08:11.280 --> 08:19.200 I guess never really reading the whole person, it's a lot better to actually design a struct 08:19.200 --> 08:25.760 with a lot of arrays in there, so the data is tightly together, I guess this, obviously it has 08:25.760 --> 08:32.080 a lot to do with a cash line length, I guess I think I had a from in 64 or in 64, but it's 08:32.160 --> 08:37.760 we have the 8 bytes, and then obviously the cash line has 56 bytes, you lose, so if you read through 08:37.760 --> 08:45.120 the, I guess, slice struct, you have a lot of overhead and waste a lot of memory and with off your, 08:47.520 --> 08:56.400 when you when you can run that application, like something CPUs have to do as well, 08:57.360 --> 09:02.960 is they need to kind of understand where your code goes, so when you have a branch, like an if 09:02.960 --> 09:12.240 condition, I guess it is something where basically the CPU wants to understand which branch you're 09:12.240 --> 09:17.680 going to take next, and in order to kind of understand how that works, I'm going to repeat, 09:18.400 --> 09:25.760 I guess some of a very kind of old CPU design, but it's pretty much unchanged with a lot more extra 09:25.840 --> 09:32.800 complexities, but what CPUs do, they do pipelining, and the reason for that is, there obviously 09:34.240 --> 09:39.600 have a lot of different circuits that do different things, and you want to make best use of them, 09:40.240 --> 09:45.920 and I guess what you also need to think about, one instruction is an obron in one cycle, so you run 09:45.920 --> 09:52.400 part of an instruction in one cycle, and I guess to order or like to to make best use of your circuits, 09:52.960 --> 09:58.560 that you have in hardware, you do some kind of pipelining, I guess, it's a bit like a car manufacturing 09:58.560 --> 10:07.520 assembly lines, so you basically have like this pattern where you, so an instruction first step 10:07.520 --> 10:12.960 is usually like fetching the instruction, obviously you have the circuit for it, and in cycle one, 10:12.960 --> 10:18.160 I guess you use it here for the first instruction, and then in cycle two, the first instruction 10:18.160 --> 10:24.080 obviously goes into the next stage, which is the instruction decoding, and then if you also want to 10:24.080 --> 10:28.880 use your circuit for fetching, you can do that with the next instruction, and basically like this 10:28.880 --> 10:35.360 gives this kind of pipelining, and rather than having five cycles per instruction, you can kind of 10:35.360 --> 10:44.320 do four instructions in eight cycles, seven cycles, and when you now look at more modern CPUs, 10:44.320 --> 10:50.960 they have more like a stage depth of like I think 10 to 20, so obviously this becomes a lot 10:50.960 --> 10:57.120 longer, and also modern CPUs are super scalears, so like they would have multiple instruction 10:57.120 --> 11:04.720 fetch circuits, so they could fetch for at the time or six, and so you can see how this kind of 11:04.720 --> 11:11.200 gets a lot more complex to build up that pipeline and start the pipeline, and now when you 11:11.200 --> 11:17.840 think about a branch, obviously like I tried to kind of annotate the same graph here, 11:17.840 --> 11:24.560 so obviously if this would be your kind of if condition, then obviously you like by that point 11:24.560 --> 11:28.720 where you want to start the next instruction, you actually don't know that I would come 11:28.720 --> 11:33.520 off the if condition yet, and so you need to know kind of what your best guess is what your most 11:33.600 --> 11:42.080 likely outcome that is, and so I guess in this case the CPU was predicting the true branch 11:42.080 --> 11:46.720 to be the one that that is the likelyest outcome, and therefore started running it before even 11:46.720 --> 11:53.680 knowing if it acts as really true, and then I guess in the best case somewhere here, it turns out 11:53.680 --> 12:00.800 it true, it's true, and so the CPU kind of won and was able through the branch prediction to save some 12:00.800 --> 12:09.200 time and have more efficiency, I guess at worst it needs to basically stop and kind of go back 12:09.200 --> 12:17.680 replant the work and start the pipeline again, in order to I guess continue at the force 12:18.960 --> 12:27.840 side of things, so that more than CPUs can get quite complex in how they do this prediction, 12:27.920 --> 12:34.800 so I guess most of the time there is like a similar kind of structure of level one level two, 12:34.800 --> 12:43.440 but it's still branches that they've taken, so obviously like you might have code that branches 12:43.440 --> 12:48.000 quite a lot, and so you add a lot of pressure to that, and so you might forget this decisions, 12:48.880 --> 12:55.440 therefore like this if you were not able to kind of get this prediction right or wrong, 12:56.000 --> 13:04.560 I'm having a kind of simple go code as well, and by the way all my examples they should have a link 13:04.560 --> 13:09.760 in the PDF and you can run them yourself and maybe figure out that the benchmark showed that as well, 13:10.880 --> 13:17.200 so in this case we have an input of int and we want to decide if the negative of positive 13:17.200 --> 13:22.640 and count up how many positives we've seen, how many negatives we've seen, so you can see the 13:22.640 --> 13:32.400 code is obviously very simple and straightforward, and depending on your input, your input could 13:32.400 --> 13:37.520 be very kind of ordered, so you might just have all the minuses first and then at some point, 13:37.520 --> 13:44.800 your kind of number switch from minus one to one, or whatever the kind of bit this way of jumps, 13:44.800 --> 13:50.080 you might have a branch mistake, but everything else is just straight pipeline being very 13:50.080 --> 13:56.400 tied and a lot of instructions around, if you don't have random numbers, obviously you 13:57.360 --> 14:05.520 I guess at worst will face like 50% of branch misses because it could happen for any of the 14:05.520 --> 14:12.800 the conditions that I guess the CPU made the wrong choice, I guess when we're creating that example, 14:12.800 --> 14:20.560 I actually spend a bit of time looking at the assembly because sometimes the compiler does 14:20.560 --> 14:26.560 small things, I think the go compiler is maybe a bit less liable to do this, but if you want to 14:26.560 --> 14:30.800 figure out if there's a branch, it's the easiest to look into the assembly and see the 14:30.800 --> 14:40.000 kind of instructions that it generates, and so if you want now want to avoid those branch misses, 14:40.880 --> 14:48.560 I guess the main two ways are either making the data predictable, so we have very good prediction 14:49.440 --> 14:57.040 or getting rid of the branch encode, so in this case, because it is a very simple example, 14:57.040 --> 15:04.160 I guess somewhat straightforward to think of an solution that will basically like do with the other 15:04.160 --> 15:11.440 metrics, the same thing that the branch would achieve, so in this case we do some shifting of the 15:11.440 --> 15:20.560 negative bits, and if the negative bit is obviously one, the negative bit is one, when it's 15:20.560 --> 15:26.160 a negative number, so we add that to the negative count, and if not, then we added to the positive, 15:26.160 --> 15:34.880 so we kind of swapped our branching for a bit more algorithmistics, and therefore I guess the code 15:34.880 --> 15:42.320 can run somewhat straightforward. Also, I think there's been a lot of research, I went researching that 15:42.320 --> 15:47.600 at the talk, I had to really stop myself a lot to not read this paper, about whatever neural 15:47.600 --> 15:52.960 network for our prediction, and so I think it's a very interesting thing to dive a bit more into, 15:53.040 --> 15:59.280 but modern like Ryzen CPUs for example, they use a second stage where they have a neural network 15:59.280 --> 16:07.920 that I think a lot of that is not very open, but people kind of were able to prove from looking 16:09.920 --> 16:18.240 running code on those CPUs, so obviously now we come to the more what I wanted to do in this talk, 16:18.320 --> 16:23.280 so how can I actually use that, and measure that proof that this is actually what happens, 16:23.280 --> 16:31.920 and so CPUs have ways to tell you, so there's a performance monitoring unit that has counters, 16:31.920 --> 16:39.360 and you are able to read those out, so very typical counters are how many cycles are run, 16:39.360 --> 16:45.520 how many instructions have been run, then there's a really detailed information about the cache 16:45.520 --> 16:52.960 misses and hits, do you read or write from cache, and the branching has, as well, 16:52.960 --> 16:59.680 like it counts the branches, it counts the branch misses, and it always is a bit CPU vendor, 16:59.680 --> 17:05.680 depending when you want to go more into customer events, so you can give it a hex code, 17:05.680 --> 17:10.000 and you can basically find in the manual of your CPU vendor what you use, 17:10.640 --> 17:16.560 like obviously one thing to do that is just get the counter for your test, and you have an idea, 17:16.560 --> 17:21.280 I guess how it performs to maybe another test case or a benchmark where they code change, 17:22.320 --> 17:31.520 but it's also possible to profile and actually get the instruction that kind of 17:31.520 --> 17:38.480 was very close to that point branch miss happened, that can be some timing issues, 17:38.480 --> 17:43.520 so it might be that the instruction that that shows up there is actually a bit after when it 17:43.520 --> 17:49.520 really happened, it's just CPUs are not getting that timing right, especially the more consumer ones, 17:51.120 --> 17:58.720 and there's also methods to kind of reduce that in accuracy, but yeah it was really very CPU 17:58.720 --> 18:03.520 dependency, you need to check kind of, so don't be surprised when an instruction that 18:04.000 --> 18:10.240 that is seemingly not related as some high counters, you basically just need to look very close to 18:10.240 --> 18:17.920 when the execution apps are there. I guess probably the biggest problem around PMUs is 18:18.640 --> 18:24.400 for virtual machines, it's kind of tricky to emulate them, because I guess they are in hardware, and if you're 18:25.600 --> 18:33.040 having very small non-dediated course in your VMs, it is kind of impossible to 18:34.000 --> 18:44.720 create that data in the kind of hypervisor level, so in Grafana we kind of run on the big cloud providers, 18:44.720 --> 18:50.000 and the bigger the instances, the more likely you get those PMUs, in GCP, you need to see 18:50.000 --> 18:56.000 set a special flag, and I found we run quite a lot of workloads on ARM, and they seem to have 18:56.960 --> 19:03.360 like even the smaller instances have pretty good PMU pass through, these are links, 19:03.360 --> 19:09.440 so you can kind of read which instance types are kind of providing that data for you. 19:11.280 --> 19:16.160 So I guess one of the biggest problems when I'm doing benchmark, the benchmark loop, changing 19:16.160 --> 19:23.120 code, run benchmark, compared to results, my name files really badly, and I lose all the code all 19:23.120 --> 19:29.600 the time and to get stash and whatever, and so I kind of lost November, I think I was fed up, 19:29.600 --> 19:34.080 and I wrote a little too much basic stores to benchmark results while I run, and some kind of 19:34.080 --> 19:44.320 gets committed to code together, and as I always wanted to kind of go a bit more into those 19:44.320 --> 19:49.120 PMUs and kind of actually like basically measure the benchmark in understand better, 19:49.120 --> 19:55.200 why something is better or not, and if my assumptions are right, I used the opportunity to just 19:55.200 --> 20:02.560 submit this talk, and then for the typical way for me to get something done is to submit a talk, 20:02.560 --> 20:09.120 and then I forced myself to kind of implement this. So I basically wrote a little wrapper around 20:09.200 --> 20:17.120 go tests, and the Perth tool on Linux, which is able to get the information out of the PMUs 20:17.120 --> 20:23.520 of your CPU, present them, and it also fixes that problem of storing the benchmark together with 20:23.520 --> 20:28.400 the code changes and what I've run. So I guess one additional problem for me, so you can see 20:28.400 --> 20:33.200 the MacBook, so there's no Perth available, I think there's some Mac based solution which 20:33.280 --> 20:42.480 fits a lot of XML, which wasn't really easy to kind of make sense of, and so when I tried to do 20:42.480 --> 20:49.120 this before, I always had to compile and ship into Linux instance, and so I also integrated the way to 20:49.840 --> 20:57.360 do an SSH remote run off of benchmark, and like one of the other goals was I want to use PProph to 20:57.440 --> 21:04.240 look at the profiles that Perth generates. Yeah, I can navigate Perth's report, but like I'm 21:04.240 --> 21:09.040 definitely a less proficient as in looking at Flamerafs and using PProph for that. 21:10.400 --> 21:15.920 And so yeah, you will find it on GitHub, the link should be on the on the speak information. 21:15.920 --> 21:20.800 I mostly have two modes, the test mode, which runs, there's a go test with a wrapper around it 21:20.800 --> 21:26.000 to collect those metrics, and the other one is a touch mode where you can run a port, 21:26.000 --> 21:33.840 given you are able to create a privilege port, you basically are able to run that in your Kubernetes 21:33.840 --> 21:38.000 cluster. You then still need to have access to the PMU from your cloud provider that we 21:38.000 --> 21:44.720 end problem now mentioned, but yeah, no, that's basically like a pretty useful, I guess, to look 21:44.720 --> 21:51.040 in how things are running in production. And so I know attempting a demo, so think is crossed. 21:56.000 --> 22:16.620 So I kind of know that type of movement. 22:16.620 --> 22:29.300 So I kind of know the typing and the sentence is not super easy, so I kind of 22:29.300 --> 22:33.580 automate it a little bit, and so I'm going to run the Perhance prediction example from 22:33.580 --> 22:42.060 earlier, and I guess as a start let's maybe look quickly, remember the kind of implementation, 22:42.060 --> 22:47.340 so we had that example with negative numbers counting up, with here with the branch first, 22:47.340 --> 22:55.620 and then in varying two, it is the branchless approach with the additional arrow matrix. 22:55.620 --> 23:02.020 So now if I want to run a benchmark I guess this is kind of roughly how I run benchmarks 23:02.020 --> 23:10.100 and go, I have four benchmarks, first of branch version with predictable data and with random 23:10.100 --> 23:15.460 data and then the branchless with those two data variants, so I guess I run it here on my 23:15.460 --> 23:24.820 MacBook for now, so you can see or not see, you can see the four benchmarks and I guess the 23:24.820 --> 23:32.820 fastest one is the one with a branch and with predictable data. Then in the second line you 23:32.820 --> 23:39.460 can see the one with random data, so which is like slower, but maybe not as slow as you would 23:39.460 --> 23:46.020 expect, so this has to do with arm and the the the less deep kind of complexity of the CPU and 23:46.020 --> 23:50.980 therefore less stages in the pipeline and so the cost is not as high, but it is still a penalty 23:50.980 --> 23:58.820 of I guess around 50% extra, and then we have the branchless version which is not as good as the 23:58.820 --> 24:05.220 branch version, but it is a very static I guess for both of the random and the predictable data 24:05.300 --> 24:13.300 can put, and so if I now want to run the same thing with that perf go tool, you basically 24:13.300 --> 24:20.100 just need to run perf in front of go, I try to keep the variables everything kind of gets past through 24:20.100 --> 24:26.580 and this runs on my machine and basically does just the same thing, what do you get from here? 24:27.620 --> 24:33.620 We are can be able to list all the runs you did and get the kibbit hash and it also saves the 24:33.700 --> 24:38.820 test binary, if you ever wanted to run it again, you could really run it fairly easily. 24:40.980 --> 24:47.140 So now in order to do kind of the first assessment of those metrics from the CPU, 24:47.940 --> 24:54.660 perf has the static command and that gives you the counters for the whole of the process you run 24:54.660 --> 25:00.580 or for everything running on the note, depending on the flux you give it. In this case we just run 25:00.580 --> 25:08.180 this kind of the worst case, the one with branch and random data and we run it with stat, 25:08.180 --> 25:16.500 and so you can't configure it or I'm on a MacBook there's no perf basically. So my desktop computer 25:16.500 --> 25:23.300 at home is hopefully online, so I kind of added remote holes and that could be just any 25:23.300 --> 25:31.860 SSH like target as long as it's kind of connectable, it basically will do exactly the same 25:31.860 --> 25:37.940 it will compile it for the correct target, so we are connected, kind of figures out yeah this is Linux 25:37.940 --> 25:49.460 in D64 and basically send the right binary rapid with perf, run the perf command and that's 25:49.460 --> 25:54.740 kind of the perf command output. So you can kind of see the metrics it shows by default I guess 25:54.740 --> 26:00.660 pretty wants to start with, so I'm not super familiar with any more beyond this and I definitely 26:00.660 --> 26:05.460 think you find quite a lot of stack overflow information or even the agents know quite a lot 26:05.460 --> 26:12.180 about how to interpret it but also perf gives you an indication of the color, so you can see that 26:13.140 --> 26:19.940 the front end of the CPU I guess is that bit with the instruction fetch instruction decode and you 26:19.940 --> 26:28.180 can see that's kind of where the stalls happen but what you also see is branch with the 14% and that's 26:28.180 --> 26:33.540 kind of the highest I could generate a benchmark for, so definitely like naively you would expect 26:33.540 --> 26:38.820 50% as the worst you would get, but there's a lot of smartness going on that you can't get that far, 26:38.900 --> 26:47.140 so typically you would see more like in the low zero point branch misses whenever I profile 26:47.140 --> 26:51.140 something running so this is really obviously the absolute worst case where we don't nothing else 26:51.140 --> 26:59.060 than just this loop and so obviously this is the obvious next thing to look into from this output 26:59.780 --> 27:09.780 and now you can basically like profile to get the functions the line numbers I guess everything 27:09.780 --> 27:17.700 that is related so what I changed here is I did profile and I added the event type branch misses 27:17.700 --> 27:22.660 so you can collect multiple events but there are hardware limitations and then purple kind of 27:22.660 --> 27:30.340 switch out the events and as you run it typically I think there's like six in the CPU six event 27:30.340 --> 27:36.340 types that you can collect and so yeah it can only obviously do six at a time if you give it more 27:36.340 --> 27:43.220 than six but in this case I also decided to run the whole benchmark suite so I'm no longer filtering 27:43.220 --> 27:49.620 on this one benchmark just so you can see the comparison in the profile so I run the the same 27:49.620 --> 27:56.820 command just I guess been profile and you can see also a bit how this one takes a lot longer 27:57.620 --> 28:05.220 on the horizon CPU because it's an AMD 64 versus the arm and now we should have the profile already 28:06.020 --> 28:14.900 so it basically passes the profile could generate the p-prof and I need to see 28:19.940 --> 28:21.620 I'm going 28:35.780 --> 28:41.300 so I guess this is the p-prof that I don't know what's going on 28:50.260 --> 29:01.940 it's easy to kind of go so you can kind of see the unit we collect should be branch misses I 29:01.940 --> 29:10.980 can't really read it but and then you can see that 100% should be close to 100% maybe not 29:11.460 --> 29:20.260 but a very huge part goes to this bent smart branch with random and if you would want to 29:20.260 --> 29:29.060 look into the source curve you can kind of see this actually muted here I guess in those lines you 29:29.060 --> 29:34.500 get the instructions I guess although the goodies you get from p-prof you haven't and or you can 29:34.500 --> 29:44.820 upload them to the same tools yeah I don't know if you can see the better of benchmark time 29:47.780 --> 29:58.660 I think time is over one minute and you know I think maybe done a quick jump back to the 30:04.900 --> 30:18.900 and just showing that all the mode quickly so with attach this is a Kubernetes cluster that I'm 30:18.900 --> 30:24.100 hopefully also connected to so you can select the port can tell it collect me branch misses for this port 30:26.340 --> 30:30.180 and it kind of does the same obviously it doesn't have to compile a test the port is already running 30:30.180 --> 30:36.820 it finds out the PIDs attached just now to the perfect command collect the profile runs it back 30:36.820 --> 30:42.660 gets steals the binary from the remote so it it also makes a lot of this manual work that I had before 30:44.020 --> 30:48.660 and yeah I guess you can take a look it's not a bit take a bit longer depending on internet speed but 30:50.500 --> 30:54.740 it basically runs for 10 seconds collect the branch misses and and for me if you would then see 30:54.740 --> 31:02.100 depending on what you're running maybe a query iterators have a lot of branch misses and yeah 31:02.100 --> 31:06.820 I guess you can then decide if this is both optimizing or not okay 31:10.020 --> 31:12.020 while we're waiting across 31:12.020 --> 31:23.780 and wow not only live them away but also trusting the first them why far to work amazing