WEBVTT 00:00.000 --> 00:10.440 Red Hat for the OpenShift and Kubernetes ecosystem platforms. 00:10.440 --> 00:16.560 Today we're going to discuss how we improved the performance of Lama CPP, running on containers 00:16.560 --> 00:19.280 on macOS. 00:19.280 --> 00:26.240 This work was led and implemented by Kevin Pujé, and fortunately, Kevin was 00:26.400 --> 00:31.040 able to come to date, so you guys are stuck with me. 00:31.040 --> 00:37.800 But Kevin is on the matrix channel to answer all the top questions. 00:37.800 --> 00:39.360 So let's get started. 00:39.360 --> 00:44.600 As you guys know, it's not possible to run containers natively on macOS. 00:44.600 --> 00:51.560 The containers are some have a Linux technology, and in order to run containers on Mac OS, 00:51.560 --> 00:54.120 we need a virtualization layer. 00:54.120 --> 01:02.280 So both podman and Docker, and more recently, I think last year, Apple released the new 01:02.280 --> 01:08.560 Apple container framework, and they all use virtual machines to run the containers on top 01:08.560 --> 01:09.560 of them. 01:09.560 --> 01:14.000 I think Docker and podman, they have this one virtual machine to run all the containers 01:14.000 --> 01:19.080 and Apple follows some have a slightly different approach where they have one micro 01:19.080 --> 01:21.640 virtual machine per each of the containers. 01:21.640 --> 01:26.440 But the truth is, we need a think virtualization layer in macOS in order to be able 01:26.440 --> 01:32.680 to run the containers. 01:32.680 --> 01:39.640 So in the case of podman and podman desktop, we need a backend for run this virtual 01:39.640 --> 01:41.240 machines. 01:41.240 --> 01:48.760 For some time now, I think this became defaulting like in podman 5.6, like last summer, 01:48.760 --> 01:51.520 the backend that we use is Libcare Run. 01:51.520 --> 01:58.960 Libcare Run is a dynamic library that allows us to run virtual machines in user space. 01:58.960 --> 02:06.000 Very fast to create these virtual machines is very focused for this use case of running containers 02:06.000 --> 02:08.600 on top of virtual machines. 02:08.600 --> 02:14.360 Libcare Run leverages the native, I provide some framework of each of the platform. 02:14.440 --> 02:18.480 So if you use Libcare Run Linux, you get a KVM acceleration. 02:18.480 --> 02:26.680 And if you run Libcare Run on macOS, you get a Apple hypervisor utilization framework. 02:26.680 --> 02:36.240 Libcare Run is also relevant in other context as well for the story of micro virtual machines 02:36.240 --> 02:37.240 in Linux. 02:37.240 --> 02:42.000 Not only it is a very important piece of the confidential computing history, it has 02:42.000 --> 02:47.480 support for this trusted platform extensions for Intel and AMD. 02:47.480 --> 02:55.480 So very nice tool, and very relevant for this talk, Libcare Run was the first backend 02:55.480 --> 03:03.800 that supported GPUs on macOS, and we are going to discuss how this was possible now. 03:03.800 --> 03:09.640 I think it is important for us to understand why we would like to run AI models on top 03:09.640 --> 03:10.640 of containers. 03:10.640 --> 03:17.320 But containers nowadays are the default for packaging, distributing, and managing applications. 03:17.320 --> 03:20.880 So that is why we want containers for running models. 03:20.880 --> 03:26.720 It also makes much easier to transition from local development to for instance Kubernetes, 03:26.720 --> 03:29.560 clusters or production environments. 03:29.560 --> 03:33.760 So I think the justification is clear. 03:33.760 --> 03:39.640 There are a couple of tools that make running containers and models on top of containers very 03:39.640 --> 03:40.640 convenient. 03:40.640 --> 03:48.400 One of them is the podnone AI lab, it is a catalog of models and they are easy to leverage 03:48.400 --> 03:53.920 to make quick proof of concepts and eventually transition to Kubernetes and open shift 03:53.920 --> 03:57.520 another production environments. 03:57.520 --> 04:05.840 There is also the RamaLama project for people that familiar with Docker and podnone is 04:05.840 --> 04:09.000 very convenient, because we have exactly the same API. 04:09.000 --> 04:14.000 The same Docker commands that we are familiar with, Docker podnone, Docker podnone, 04:14.000 --> 04:20.840 Docker podnone, and we can use them to run models with exactly the same syntaxes. 04:20.840 --> 04:27.600 It also supports this bench subcommand which is very convenient for this specific testing 04:27.600 --> 04:33.320 that we conducted during this work. 04:33.560 --> 04:36.200 So what about the GPU acceleration? 04:36.200 --> 04:42.400 How this works when we have this visualization layer in the middle of the containers and 04:42.400 --> 04:43.400 the host? 04:43.400 --> 04:51.440 So in Linux it is very easy, we just expose the depth, the corresponding depth, files to the container 04:51.440 --> 04:57.480 and we automatically get this GPU acceleration within the container. 04:57.480 --> 05:01.840 The issue with virtual GPU acceleration is that we have the virtual machine in the middle. 05:01.840 --> 05:06.560 There are different strategies that we can follow here. 05:06.560 --> 05:12.520 First strategy would be full device password with the GPU, the whole GPU to the virtual 05:12.520 --> 05:13.520 machine. 05:13.520 --> 05:17.680 Unfortunately, this is not possible in Mac, we only have one GPU and the GPU is being used 05:17.680 --> 05:23.160 to render the screen, so that's not an option here. 05:23.160 --> 05:26.480 Hardware devices have a system device password. 05:26.480 --> 05:31.360 So there are enterprise GPUs, Nokia, and the enterprise GPUs that support like 05:31.360 --> 05:36.560 it's lies in the GPU, so you can deliver as small as lies of the GPU to the virtual machine. 05:36.560 --> 05:42.600 The Apple GPUs that doesn't support this kind of hardware assistance. 05:42.600 --> 05:51.400 Virtual device emulation is out of question for hyperforman inference workloads, so we are 05:51.400 --> 05:55.640 left with a parameter to analyze the device option, and this is the option that leave 05:55.640 --> 05:59.840 the run currently implemented. 05:59.840 --> 06:07.160 So in order to make sure that we have a good performance when running containers with 06:07.160 --> 06:12.720 leave the run, there's this virtual driver called the GPU, and there's a collaboration 06:12.720 --> 06:18.080 between the guest kernel in the virtual machine and the host running the supervisor. 06:18.080 --> 06:22.560 In the right side of the diagram, you can see the virtual machine space. 06:22.560 --> 06:25.760 In the userland, we have Lama CPP. 06:25.760 --> 06:30.840 We have the kernel of the guest and the virtual GPU driver, and in the left side of the right 06:30.840 --> 06:35.600 on the left side of the screen, we can see the host side, so we have the partner next 06:35.600 --> 06:42.040 to publication, manage in the containers, leave the run as the lie type advisor lay you're 06:42.040 --> 06:48.040 running a small virtual machine, and the real renderer that makes the translation between 06:48.040 --> 06:55.560 the GPU calls from the guest and the host. 06:55.560 --> 07:02.360 This is a little bit more detailed, we are using Vulcan as the GPU API, and we have like 07:02.360 --> 07:09.560 this layer that translates between Vulcan and the native FP, framework which is called 07:09.560 --> 07:12.120 APL, APL, metal framework. 07:12.120 --> 07:21.000 So we have this little translation layer in the middle to be able to reach the GPU. 07:21.000 --> 07:25.200 And this is the current state of the art, this is what Kevin found itself like six months 07:25.200 --> 07:28.960 ago when he started with his project. 07:28.960 --> 07:31.960 And let's take a look at the numbers. 07:31.960 --> 07:37.040 So first of all, on an optimistic note, the inference performance on containers on Mac OS 07:37.120 --> 07:42.320 has been growing, increasing and proving over time. 07:42.320 --> 07:48.440 So in 2024, unfortunately, we didn't have this CPU support from Limke Ranson, the only 07:48.440 --> 07:53.920 driver available was the CPU one, and as you can see, we were able to generate like 07:53.920 --> 07:59.360 half token per second. 07:59.360 --> 08:12.880 Early last year, there was this compute framework for GML, GMLS, the token is the kernel 08:12.880 --> 08:15.600 back in the giant for Lama CPP. 08:15.600 --> 08:21.480 And with that, we were able to somehow really improve the performance of inference, and 08:21.480 --> 08:30.160 recently, with a new Vulcan native backend, we were able to reach like 40 times faster 08:30.160 --> 08:32.520 results that with just CPU. 08:32.520 --> 08:41.160 So this is an optimistic evolution of running inference workloads on Lama CPP on containers. 08:41.160 --> 08:44.600 However, there is room for improvement. 08:44.600 --> 08:50.340 In this graph, we can see in the first column, we can see the performance of native Lama 08:50.380 --> 08:54.260 CPP Ranson with FAPIL-METAL framework. 08:54.260 --> 08:56.780 So this is somehow what we would like to achieve. 08:56.780 --> 09:01.620 This is the ideal performance, the maximum performance that we can get. 09:01.620 --> 09:07.700 And in the second, in the third column, we can see the performance using the Vulcan tensor 09:07.700 --> 09:09.700 backend. 09:09.700 --> 09:13.460 As you can see, there's no much difference between running this natively on the Mac OS 09:13.460 --> 09:17.980 as a process or running the solution inside of it or machine. 09:17.980 --> 09:23.220 So the performance between the second and the third column are almost identical. 09:23.220 --> 09:27.100 What we can see here is the virtualization was not the bottleneck. 09:27.100 --> 09:33.300 We suspect the bottleneck may be related to this translation between Vulcan and April 09:33.300 --> 09:35.740 framework instructions. 09:35.740 --> 09:44.420 And there is a new meta implementation that probably could improve these results at the 09:44.420 --> 09:49.500 time, at this effort, started the solution wasn't like finalized, so we don't have 09:49.500 --> 09:54.500 numbers for this, but I understand it sounds promising as well. 09:54.500 --> 09:59.820 So this is the gap that we tried to cover in this work. 09:59.820 --> 10:03.820 So this was the state of the art before this work started. 10:03.820 --> 10:10.620 So how we can close the gap between this Vulcan virtual machine solution and the native 10:10.620 --> 10:11.620 framework? 10:11.620 --> 10:19.100 One first of your solutions, so let's use the meta API framework. 10:19.100 --> 10:21.260 This is the most performant one in Mac OS, right? 10:21.260 --> 10:22.660 Why don't we use it? 10:22.660 --> 10:26.800 Because of several reasons, first of all, Linux doesn't have the drivers for April, for 10:26.800 --> 10:29.900 April GPU. 10:29.900 --> 10:32.020 It's very specific to Mac OS. 10:32.020 --> 10:37.420 I'm not sure, but maybe there could be some legal aspects to supporting them and implement 10:37.420 --> 10:38.740 Windows in the kernel. 10:38.740 --> 10:45.260 So that didn't look like a reasonable solution. 10:45.260 --> 10:55.740 So the approach that was found was to use API remoteing to use the native metal tensor 10:55.740 --> 10:58.180 back end from La Masipiti. 10:58.180 --> 11:07.060 So the host uses the native metal back end and the calls are triggered from La Masipiti. 11:08.060 --> 11:11.540 This is somehow the solution that was found. 11:11.540 --> 11:18.180 So as you can see, now the application itself La Masipiti can send commands and instructions 11:18.180 --> 11:24.060 directly to the back end, which is in the host side, translating these instructions directly 11:24.060 --> 11:26.540 to the Apple Metal framework. 11:26.540 --> 11:38.780 So somehow we avoided the use of the Vulcan tensor back end and we made some how the application 11:38.780 --> 11:49.380 which is La Masipiti aware of the incapable of running these commands on the host side. 11:49.380 --> 11:52.940 So the idea is that we intercept the API calls in the guest. 11:52.940 --> 11:59.700 We further the parameters to the host, the host runs the function in the GPU and retams 11:59.700 --> 12:02.100 the data back to the guest. 12:02.100 --> 12:07.380 It's the same strategy used for the Vulcan tensor communication, but we move the strategy 12:07.380 --> 12:11.660 one layer up to the application to La Masipiti. 12:11.660 --> 12:13.580 This is not a generic solution. 12:13.580 --> 12:17.380 This is a specific to the La Masipiti use case. 12:17.380 --> 12:20.220 The Vulcan solution was generic. 12:20.300 --> 12:25.100 We'll see the result now, but it's not very high. 12:25.100 --> 12:31.100 We pay the light price for the for the for the for the hyper calls latency and the synchronization 12:31.100 --> 12:33.740 between the guest and the host. 12:33.740 --> 12:38.580 But the price is not so high because we use parameterization, the host share memory pages 12:38.580 --> 12:47.140 to avoid the double copying issue and sharing data between the guest and the host. 12:47.140 --> 12:55.700 Here we can take a look at the results, the first column, this is by the way packet 12:55.700 --> 13:01.460 processing, processing, inference workload. 13:01.460 --> 13:07.740 So the blue column represents the native performance, this is what we would like to achieve. 13:07.740 --> 13:14.060 And the red column represents the performance of the new solution using this API remote 13:14.220 --> 13:16.220 strategy. 13:16.220 --> 13:24.940 The third column is the the the classic solution using the the Vulcan framework. 13:24.940 --> 13:29.500 So as you can see with the API remote strategy and for this very particular case of La Masipiti 13:29.500 --> 13:36.460 inference, we were able to achieve almost identical performance as the native one. 13:36.700 --> 13:49.260 We run this results in different hardware, so this is a token generation, a token generation 13:49.260 --> 13:57.180 workload, the first graph represents different hardware from M1 to M4. 13:57.180 --> 14:01.100 The second graph here in the right represents different models. 14:01.100 --> 14:06.700 And finally the third graph in the bottom represents different model sizes. 14:06.700 --> 14:11.820 So as you can see in general the the the performance is very good and very reasonable 14:11.820 --> 14:13.580 very close to the native one. 14:13.580 --> 14:19.660 Obviously there's there's a slight performance overhead, but is is very very acceptable 14:19.660 --> 14:21.260 and within limits. 14:21.260 --> 14:26.940 We observe that this overhead is greater in smaller models, which makes sense because 14:26.940 --> 14:33.980 the overhead of the of the hypercolt is bigger, but in general in general the the performance 14:33.980 --> 14:40.860 is really close to the to the to the native performance considering this is running in a container 14:40.860 --> 14:46.620 within a virtual machine, I think the performance is is amazing actually. 14:49.340 --> 14:56.060 So these the main work from an implementation side to happening in a couple of projects, 14:56.140 --> 14:59.420 the biggest code changes are in Lama CPP to implement this new 15:00.540 --> 15:02.700 attention interface backend. 15:03.900 --> 15:10.380 This was merged earlier this week very timely for the confidence and there are also some changes 15:10.380 --> 15:19.100 required in the real GL renderer for the for the API remote in communication that we are waiting 15:19.100 --> 15:21.100 for for some reviews on on that side. 15:22.060 --> 15:34.060 So there's this article in in the developer fret hat blog which explains this in more detail and 15:34.060 --> 15:40.380 with more technical references the work is is available to try. 15:40.380 --> 15:47.420 So we can try this in in macOS we can try this in Linux with with K run and we are we are working 15:47.420 --> 15:53.500 on enable this this workload on Windows as well and we expect as the the remaining PR get 15:53.500 --> 15:58.940 met to have this implemented in postman desktop through this year probably in the next quarter. 16:04.540 --> 16:11.260 Some security considerations so as I said they provide the ransom on user space the virtual 16:11.340 --> 16:18.780 machine manager so with user privileges and the backend and the GPU on the execute calls that 16:18.780 --> 16:24.700 comes from Lama CPP different from the Vulcan model work where we have like more possibilities 16:24.700 --> 16:28.140 of running and trusted code in the in the GPU of the host. 16:28.860 --> 16:32.940 But there are currently limitations and then they are very related to the multi-tenancy aspect. 16:33.500 --> 16:39.340 So the the backend library leaves in the in the advisor addresses space so 16:39.340 --> 16:44.940 a crash within the backend library that translates the the calls to the GPU between the 16:44.940 --> 16:49.260 guests and the host could bring down the advisor and all the virtual machines that run on the 16:49.260 --> 16:54.540 advisor and also the backend of multiple container running the same process. 16:55.420 --> 17:02.780 So we don't have like per user GPU isolation so we don't consider business sign like multi-tenancy 17:02.780 --> 17:09.260 for production ready I think it's great to run locally workloads on macOS but it's not something 17:10.140 --> 17:14.540 that could be extrapolated to production or or other scenarios. 17:17.900 --> 17:21.260 We also have a nightly CI where we run all these scenarios. 17:22.140 --> 17:28.140 We compare native to Vulcan to to IPI remote performance in a daily basis. 17:28.780 --> 17:36.220 We were able to to find some regressions on Rama Lama or Lama CPP so so the the CI has proven 17:36.300 --> 17:41.340 very useful and we plan to extend it and to maintain it with with other use cases. 17:43.820 --> 17:49.660 And I think that's all from our side there are some extra slides that I will upload with some 17:49.660 --> 17:54.700 cuts snippets some code commands that you can use to replicate the experiment on your local machines. 17:55.260 --> 18:00.620 And this this is also the email of of Kevin and my email review guys have any further 18:00.620 --> 18:07.020 questions thanks a lot for your attention and Kevin is happy to take any questions on matrix. 18:07.660 --> 18:10.780 Thank you very much. 18:16.220 --> 18:24.540 Okay so can everyone hear me? Okay so one good question that was asked is this is an amazing work 18:24.540 --> 18:29.260 which I don't think is a question just a statement which by the way I absolutely agree with. 18:29.340 --> 18:36.940 The question is can some of the same techniques be used on VLOM sort of running on macOS and containers? 18:37.660 --> 18:45.100 Probably yes I I think one of the limitation of this work was very specific to Lama CPP so very targeted 18:45.100 --> 18:50.220 and the downside of Kevin and application level approach is that in order to support other 18:50.220 --> 18:56.540 applications you will need to implement this this work on other on any of LLM or other frameworks as well. 18:57.260 --> 19:02.060 So this is the downside of not having like a generic solution like the Vulcan one very targeted 19:03.260 --> 19:05.820 performance goal here with Lama CPP. 19:07.180 --> 19:12.460 Yeah and at what's pointed out VLLM is largely based on PyTorch-like approaches and again then you would 19:12.460 --> 19:17.820 have to have you know to have to have that back end so you know that's kind of goes without saying. 19:17.820 --> 19:23.900 So no other questions so again thank you so much.