WEBVTT 00:00.000 --> 00:12.560 So let's get started. Thanks everyone for coming. Today I'm going to talk about restartable 00:12.560 --> 00:17.880 confidential guests under Kimo. This project that I've been working on for the last few 00:17.880 --> 00:24.840 months. My name is Annie and I'm a software engineer at Red Hat. So let's get started. 00:24.840 --> 00:29.240 So this is the background of what we're going to discuss today. It usually will give you 00:29.240 --> 00:33.240 background of the work, the context, and then we'll talk about the implementation 00:33.240 --> 00:40.280 choices and the challenges. I'll try to give it a more today. A live one if time permits. 00:40.280 --> 00:44.760 And then I'll talk about the current state of the patzer that I've posted already on the 00:44.760 --> 00:52.520 Kimo devil mailing list and the revealing challenges. So the background. So talking 00:52.520 --> 00:59.000 about the different kinds of guests that runs at a Kimo. We have non-confidential virtual 00:59.000 --> 01:03.640 machines. We have confidential virtual machines. And I'm a confidential virtual machines. We have 01:03.640 --> 01:10.600 the SCV VMs. We have ACV ES. We have ACV ES and PVMs. They're all in the technology and for 01:10.600 --> 01:18.040 Intel we have the TDX VMs. But the only ones that are actually restartable are the non-confidential 01:18.040 --> 01:24.920 VMs and the confidential SCV VMs. And none of the others are currently restartable. And the 01:24.920 --> 01:35.320 reason for this is being is that for SCV SMP and SCV ES and the TDX VMs. They encrypt 01:35.320 --> 01:41.160 their state of the CPU racers. And once it is encrypted it's locked in. And when they get 01:41.160 --> 01:48.280 boots the hypervisor is not able to modify the state of those CPU racers. So in order to do 01:48.360 --> 01:56.040 the reset the hypervisor has to reset the state of the racers which since they have 01:56.040 --> 02:04.200 the racers can't the the the guest has to terminate. For ACV VMs they only encrypt their state 02:04.200 --> 02:11.320 the memory. They don't touch the CPU racers. So they are restartable like just any other 02:11.400 --> 02:20.360 novel guests. So a little bit of the context of why this is actually we're doing this. 02:20.920 --> 02:27.080 First of all we want to bring all guests at part with these other both confidential or different 02:27.080 --> 02:32.120 kinds of confidential guests as well as the confidential guest. But there is also another reason for this 02:32.680 --> 02:37.560 which is the work that we have been driving at red art which is the Fooky work and I gave 02:37.560 --> 02:44.840 a presentation for that last year at fourth of them in the vote dev dev room. So you can take a 02:44.840 --> 02:51.000 look at that and there I presented this particular slide and you see that there are two states 02:51.000 --> 02:55.480 of these confidential VMs. The one on the left is actually running with a stock firmware from the 02:55.480 --> 03:01.560 cloud provider. And then one on the right is actually running with a firmware that the end user brings 03:01.640 --> 03:08.200 with them. And ultimately we want to go from this state of the confidential VM to the state of the 03:08.200 --> 03:15.400 confidential VM. So in order to do that there is a reset process. And this at least we can reset the 03:15.400 --> 03:21.080 guest from this state to the state and do all the administration and everything. An encryption 03:21.080 --> 03:29.320 attestation all of it again for this guy. We are not able to implement this. So the main motivation 03:29.320 --> 03:35.400 for me getting involved with this is to implement this reset here so that we can do this. 03:38.520 --> 03:47.640 So in terms of implementation choices. So as I said the the CPU racer states are actually 03:47.640 --> 03:55.400 encrypted. So one option we could do is ask KVM in the kernel to save the pre encrypted states. 03:55.480 --> 04:03.000 The initial state of the memory, the piece table, the CPU racer, everything. And then upon reset 04:03.000 --> 04:10.840 restore that unencrypted state and then do the whole thing again. The problem with this there are two 04:10.840 --> 04:19.080 problems with this one is that what if we wanted to reset to a different launch state. So for example 04:19.080 --> 04:25.960 we could imagine what we were trying to do with this is that here we are changing the firmware. 04:25.960 --> 04:34.760 So our launch state is actually different from what we initially launched with. So that will not 04:34.760 --> 04:41.000 be possible if you are already always resetting with the same launch state. And the second is that 04:41.000 --> 04:48.760 we potentially need a new KVM API which introduces all kinds of complications. And so we 04:48.760 --> 04:58.520 kind of didn't want to get into that. So the other option that we did go for is this one where 04:59.080 --> 05:07.320 what we do is we throw away the initial states of the encrypted state of the confidential gas 05:07.960 --> 05:14.200 and we closed the old KVM file descriptor and we create a new one. And with when we are creating 05:14.200 --> 05:20.920 a new KVM file descriptor for the virtual machine, KVM predictors are the new guests. It is a different 05:20.920 --> 05:29.240 guest. And then with that it does the whole encryption process and the locking in of the 05:29.240 --> 05:37.000 initial launch state. All of it happens again. And then we transition from an unencrypted 05:37.160 --> 05:45.000 encrypted state. And the good thing about this is that we do not need a new KVM API and it's 05:45.000 --> 05:53.880 just transparent to it's just kind of the user does not use any defense and the old KVM process 05:53.880 --> 06:04.040 keeps running. So this is the implementation that we went for. So basically at the end it kind of 06:04.040 --> 06:11.960 looks like this where we have the initial confidential guest that encrypted memory registers 06:11.960 --> 06:18.200 etc. starts running and then it initializes initiates a reboot. And as a part of the reboot 06:18.200 --> 06:25.880 we do a device reset. And at that time we can bring in, we can use the same IDVM or we can 06:25.960 --> 06:34.200 do like we can bring in a different IDVM which is the idea for Fookie or we can use the same 06:34.200 --> 06:42.200 IDVM to state the initials to set the initial state. And then we close the old KVM VM file descriptor 06:42.200 --> 06:49.880 and then we create a new one and we go to the entire sequence again. So it's like basically 06:49.880 --> 06:58.440 initializing the confidential guest all over again with the new VMLD. So what are the challenges 06:58.440 --> 07:05.640 here? So changing KVM VM file descriptor is not being balanced. They're really not done for 07:06.200 --> 07:12.120 obviously for normal guests it's not needed. So it's like pulling the carpet under your feet 07:12.120 --> 07:21.320 and replacing with another. So here we know that all the so this is to understand the challenges 07:21.320 --> 07:26.600 we try to understand the structure of how the different modules work. So you have these accelerators 07:26.600 --> 07:33.880 and then the KVM accelerator implements all architecture diagnostics stuff in in this in KVM 07:33.880 --> 07:39.960 all dot c and then from there it calls into more architecture specific stuff. And then also as a 07:40.040 --> 07:46.760 part of this creates the VCPUs against the VM file descriptors and those VCPU initialization 07:46.760 --> 07:53.640 calls architecture diagnostics stuff as well as architecture specific stuff here and other devices 07:53.640 --> 07:59.720 and other modules and KVM also does the same thing. They have the new all kinds of idol calls 07:59.720 --> 08:06.040 to the VM file descriptor here to do both architecture specific stuff and non-architecture specific 08:06.360 --> 08:13.480 stuff also they do sometimes initialization is the different states of the VCPUs so they call 08:13.480 --> 08:25.720 into the VCPU file descriptor as well. So this is so basically this is the broad picture of how it 08:25.720 --> 08:31.960 looks like like I said there is a architecture non-system then there is architecture specific 08:31.960 --> 08:39.560 stuff that initializes the confidential competing module which does some general initialization 08:39.560 --> 08:45.240 and then it does some architecture specific initializations here depending on what technology 08:45.240 --> 08:51.480 we're using for the confidential cast and then there is also the VCPU file descriptors that are 08:51.480 --> 08:59.320 created against the VM file descriptors and they do all kinds of initialization. So how to make sure 08:59.320 --> 09:04.920 that all these happen again as a part of the reset process because we're changing the VM file descriptor. 09:04.920 --> 09:12.920 So what we do here we create different notifiers one notifier is before the VM file descriptor change 09:13.960 --> 09:18.920 and then the other is after the VM file descriptor change so basically those module that want to 09:20.840 --> 09:28.760 save the states from KVM into KMO before the VM file descriptor changes can use this notifier 09:29.480 --> 09:35.400 and then this notifier is used after the VM file descriptor changes to do all the initializations 09:35.400 --> 09:42.600 as well as pushing in the states from KMO to KVM again if needed not in all cases it is needed. 09:42.600 --> 09:48.440 Migration for example is not supported today so we don't deal with migration cases but there are 09:48.440 --> 09:57.000 other cases where we need to push in the states from the KMO to the KVM so there we can use 09:57.080 --> 10:04.520 this mechanism also since the VCP initialization needs to be done again we need to actually 10:04.520 --> 10:10.680 have this notifier to notify subsystems that VCP will have been changed and you need to do the 10:10.680 --> 10:16.520 VCP initialization stuff again more architected specific as well as architected elastic stuff. 10:17.000 --> 10:28.840 So that's basically what I just described so how do we actually do this in terms of implementing 10:28.840 --> 10:38.360 and figuring out the issues and actually getting to do this stuff so we use stress explicitly 10:39.720 --> 10:45.880 to figure out all the different ioptocals and the code paths from where they're called 10:46.680 --> 10:51.240 and then we need to cover all the code paths we need to understand the code where they're being 10:51.240 --> 10:56.360 called why they're being called what needs to be done again what need not be done again because 10:56.360 --> 11:01.960 sometimes the initializations happens once the data structures and the tables gets initialized like 11:01.960 --> 11:08.280 IQ routing for example they get initialized once within KMO and that remains and now 11:08.280 --> 11:16.360 only thing we need to do is push that information back into KVM once we have the new KVM VM file descriptor 11:16.360 --> 11:22.120 so we need to understand what's really going on and do the stuff that's needed to be done again 11:22.120 --> 11:27.720 some functions for example are not written in a way that can be called again because they're 11:27.720 --> 11:34.920 meant to be done once because confidential gas signalize once and then you know since reset was not 11:34.920 --> 11:41.000 supported they they were not written in a way so that those functions can be called again so 11:41.080 --> 11:48.840 we need to fix those things there were a lot of debug cycles and many of these debugging can 11:48.840 --> 11:58.520 actually happen in non-copen environment because if you look at this slide here this this part 11:58.520 --> 12:04.520 is all confidential specific but this parts can all be exercised even in non-confinition 12:04.520 --> 12:10.680 environments so all of this we can do the debugging of all of this we can do any any any 12:10.760 --> 12:18.920 host we don't need a confidential hardware or a confidential setup to actually do the investigations 12:20.600 --> 12:23.960 so I'm sorry I'm going in the wrong direction 12:26.520 --> 12:31.880 okay so demo I'll give in the end because in case I run out of time I'm not going to take a risk 12:31.880 --> 12:38.840 but we I have already uploaded some demo videos on my top page so those are the confidential 12:38.920 --> 12:45.160 specific demos because I do not have a confidential hardware available right now so the non-copo 12:45.160 --> 12:51.400 part I'll try to if I have time I'll show you it's it's pretty easy so let's let's 12:51.400 --> 12:58.360 discuss the current state of the process it's actually currently we're only addressing 12:58.360 --> 13:06.840 exit six only and we're not talking about arm or other platforms I just posted the version 13:06.840 --> 13:12.440 three of the parts that earlier last week so it's in the mailing list I have a look at that 13:14.840 --> 13:22.040 we have tested this on all these confidential guests as well as non-confinential hosts 13:22.840 --> 13:29.400 with Zen staff as well key muci pipeline is good although key muci pipeline does not 13:29.400 --> 13:36.120 actually there's not enable all caveum specific functional tests because they're all skipped 13:36.120 --> 13:44.440 because dev caveum is not available so the key muci pipeline only proves that there are no 13:44.440 --> 13:49.960 regressions but it doesn't actually exercise any of the new changes that I'm talking about 13:52.440 --> 13:58.280 but maybe we'll fix this maybe maybe we'll enable the caveum tests in the c i and then we can 13:58.280 --> 14:05.720 we can run this as well the test scripts are public here so we can take a look at those mostly 14:05.720 --> 14:11.320 the delivery scripts and some stress scripts etc there is a functional test also has a part 14:11.320 --> 14:19.160 of the V3 of the my padset and that is what I'm kind of going to show you if I have time I just 14:19.400 --> 14:26.760 practice it into a functional test with Zen and normal cases it's kind of nice to be able to 14:26.760 --> 14:33.800 actually exercise the whole thing on your on your laptop or your desktop or whatever if 14:33.800 --> 14:42.520 if you don't have a confidential host currently we're targeting chemo version 11 if everything 14:42.600 --> 14:53.640 gets done well and fast but if not then it will be 11.1 so whatever the remaining challenges 14:54.600 --> 15:00.440 we need people to actually review our our our padset we need people who have specific knowledge 15:00.440 --> 15:08.440 in subsystems like Zen or other devices to look at the changes and let us know if there is 15:08.440 --> 15:15.480 anything that need to be done or not done because there are a lot of subsystems that we touch 15:15.480 --> 15:25.080 which I do not know that all of it I just try to make the best judgment and so we need eyes on those 15:25.080 --> 15:36.520 things and in general look at the padset and see if there are any issues with it so in terms of 15:36.520 --> 15:44.120 focus going back to the focus question what else remains so let's let's go back to that picture 15:44.120 --> 15:54.600 that I had before this one so first of all if we are talking about IGVM then IGVM needs to work with 15:54.600 --> 16:04.600 this reset process and that so IGVM support is already in chemo today but IGVM does not currently 16:04.920 --> 16:11.880 work with the reset in place so for that we are already doing some work and good has posted some 16:11.880 --> 16:21.560 patches on the mailing list so please have a look at this padset and we need to make sure that 16:21.560 --> 16:30.520 this actually works with the reset in place and then we need to have a mechanism for focus specifically 16:30.520 --> 16:38.680 we need to have a mechanism for end users to be able to bring in their own IGVM implementation 16:38.680 --> 16:48.920 or the some way to actually push the IGVM bundle back to the to the hypervisor so that the new IGVM 16:48.920 --> 16:57.080 can set a new initial state of the launch state of the VM and then reset into that so so that work 16:57.160 --> 17:03.480 is yet to be done and I posted an old version of it here but it's not relevant anymore 17:05.320 --> 17:13.480 I thank very much for follow for guidance throughout the process of this work and God as well 17:13.480 --> 17:19.880 God is also deeply involved with the fuki now so he's also doing a lot of the IGVM set of the work 17:20.600 --> 17:26.040 the chemo community in general the reviewers so I've already reviewed my padset and I tagged them and 17:26.040 --> 17:32.920 we I welcome other reviewers and comments on my patches and red hat for supporting me with this work 17:32.920 --> 17:42.120 and giving me all the resources to be able to continue this work as part of fuki so now I'm just 17:42.120 --> 17:55.400 going to quickly see if I can give demo here okay let's see anyway so this is too big actually 17:55.400 --> 18:03.800 but you can see this is my script what it actually does is it creates a chemo VM and what I'm going 18:03.800 --> 18:11.720 to emphasize is this debug flag right here which enables the VM file descriptor change on resets 18:14.600 --> 18:27.080 and rest of it is you have I use the chemo accelerator and everything and we use a uky and we use 18:27.080 --> 18:35.160 dash kernel mechanism in the chemo this uky packages a kernel and a basket as in it so we'll see 18:35.160 --> 18:43.800 what happens when I kind of run this so let's run this so this is non-confidential case because this 18:43.800 --> 18:52.120 is not a confidential hardware so it will just do the VM file descriptor flip and if we do a reset so 18:53.080 --> 19:03.720 another thing that I actually forgot to mention here is that we need to monitor this guy here 19:05.160 --> 19:11.080 so I'm monitoring this chemo dot log and it should speed out a name function that's 19:11.080 --> 19:17.400 it's called in the reset so let's do this 19:17.480 --> 19:33.560 all right so let's resetting if you go back and hopefully 19:33.800 --> 19:40.440 there is 19:48.520 --> 19:56.600 right so there is there is this log that says virtual fine accelerator file that's with the 19:56.600 --> 20:04.040 Tvm file descriptor has changed and hopefully there is this there is a Tvm reset VMFD is 20:04.040 --> 20:10.760 actually the the function call that I enabled on the tracing so you see this on the tracing log here 20:11.640 --> 20:23.080 so that's it and that's it for me do you have any questions do we have time for questions 20:27.560 --> 20:38.600 a couple minutes couple minutes yeah so yes yes 20:51.640 --> 20:55.800 so basically the idea was okay so the question was that 20:57.160 --> 21:02.120 what did I do for testing whether I had any specific functional test in mind so 21:02.120 --> 21:08.760 the idea was to test all the modules that are actually so all the devices and all the modules 21:08.760 --> 21:14.680 where we are using this extra initialization after the VM file descriptor picture so 21:14.680 --> 21:21.960 the the the intent is to use those devices and those configurations in the chemo and then 21:21.960 --> 21:27.480 try and do a reset so that to make sure that all that code is actually being exercised 21:28.440 --> 21:33.640 through the notifies and everything so I added trace point so basically you can you can 21:33.640 --> 21:37.720 enable tracing on those functions and do a reset and see out okay those functions are getting 21:37.720 --> 21:55.160 called everything is fine so any other questions yes yes yeah yeah yeah yeah yeah yeah 21:55.160 --> 22:01.640 this is just for timing or is it the time to move and in general in the tech issue whether yes 22:01.640 --> 22:08.360 this is in the state where you need this change so okay so the question is the 22:08.360 --> 22:15.480 device that I used in my demo whether it is actually needed in the final state or not so 22:17.240 --> 22:21.880 so normal for normal gets non-confinancial guests you don't need this right for 22:21.880 --> 22:29.560 confidential guests it is it happens automatically because that's that's our solution right so 22:29.560 --> 22:35.000 so if you you want to see if actually this is getting called you can you can actually check the 22:35.000 --> 22:40.600 log that I just show or you can enable tracing to see that those functions are actually 22:40.600 --> 22:45.240 getting called or not so the debuts like that I introduced was only for non-coco case 22:45.800 --> 22:51.880 because that's the only way to force a VMFD change but also make sure that in the normal case 22:51.880 --> 22:56.760 when you are running in production environment that's not used it's only used for your debugging 22:56.760 --> 23:00.120 and testing purpose