WEBVTT 00:00.000 --> 00:21.560 Okay, so before going on, it's a brief interaction, my name is Rodlan, and I work as a 00:21.560 --> 00:28.640 architect at IBM, before I spent a while in the bunch of companies, before joining Red Hat in 00:28.640 --> 00:35.240 2022, to work for orchestration team, and then I moved to IBM, and since then I'm working 00:35.240 --> 00:41.320 as part of the CFADN as core contributor. 00:41.320 --> 00:48.800 Um, with agenda of the presentation, basically we'll see what are the motivations that lead 00:48.800 --> 00:56.560 us to introduce the CFADN management gateway, like the legacy issues and the legacy architecture 00:56.560 --> 01:02.720 documentation, then we'll see the architecture of the new management gateway, how it works, 01:02.720 --> 01:08.200 what are the new components and CFADN services that it's in introduces, we'll see the benefits 01:08.200 --> 01:18.360 of this approach, and then we'll see a little bit roadmap and the future work that we have. 01:18.360 --> 01:24.040 We're going dependent in the presentation, I would like to present this slide, because 01:24.040 --> 01:30.960 it shows clearly how the CFADN internal services evolve it, and we can see clearly that 01:30.960 --> 01:38.440 starting from 20, 60 and more or less, we started having more and more services, as you 01:38.440 --> 01:45.440 can see you introduce it, the monitoring stack, it's a proxy keep alive for a high availability, 01:45.440 --> 01:52.360 and more and more services coming around, and it is becoming harder to manage and to access 01:52.360 --> 01:55.800 all of them easily. 01:55.800 --> 02:03.440 This slide we can see the legacy architecture of the CFADN, of the CFADN monitoring and the 02:03.440 --> 02:09.320 management stack, in the sensor point, we can see the CFADN dashboard, which runs as manager 02:09.320 --> 02:15.800 model, the manager itself normally runs in HA mode, where only one manager is active, 02:15.800 --> 02:22.160 the rest are stunned by, and then in the lower part we can see the stack, the monitoring 02:22.160 --> 02:29.080 stack, which is based on the promissius, normally the promissius scraped the metrics from 02:29.080 --> 02:35.200 the manager promissius model and from other providers, like the CFADN and the Nordic 02:35.200 --> 02:41.600 Sporter, or them provide the metrics in point, and then once it's straight the metrics, 02:41.600 --> 02:44.960 it sends them to the other manager, then the other manager process, the different rules 02:44.960 --> 02:50.800 it has to see if it has to fire some alerts, these alerts are sent to the CFADN dashboard 02:50.800 --> 02:57.360 through the web hook task, and then internal is CFADN dashboard, it has an embedded graph 02:57.360 --> 03:05.400 plan, frame, which uses the promQL language to scrape the data from promissius, so this 03:05.400 --> 03:14.840 is a simplified vision of the architecture, so in practice, at least in the old releases, 03:14.840 --> 03:19.800 the authentical, before the interaction of the management gateway, the user, in other 03:19.800 --> 03:25.040 practices, these different services, he has to know on which node and on which sport each 03:25.040 --> 03:32.600 service is running, like for example, we had the dashboard running in the 8443 port promissius 03:32.600 --> 03:37.840 others manager, all of them can be run in the front nodes, the front ports, so we have 03:37.840 --> 03:45.640 to know where the services are deployed in order to access them, by default, the monitoring 03:45.640 --> 03:54.480 stack is deployed without SSL support, so it just uses plain HTTP, starting from 03:54.480 --> 03:59.480 reef, when you reduce it, this secure monitoring stack, which is new flag that you can 03:59.480 --> 04:09.480 turn on, and once you turn it on, at least you get some basic security for the different 04:09.480 --> 04:17.200 services running the monitoring stack, the problem with this is each service has its own 04:17.200 --> 04:25.760 user and password, so it is fragmented, this really have to maintain, we also have some support 04:25.760 --> 04:31.200 for the single sign-on, but it only works for the dashboard, the rest of the services in 04:31.200 --> 04:36.320 the legacy architecture didn't have any supports for single sign-on, so as you can see, 04:36.320 --> 04:44.200 there are a lot of pain points, most important, the default multi-point and points to access 04:44.200 --> 04:49.720 the different services, different URLs, different ports, in constant authentication methods, 04:49.720 --> 04:55.800 single sign-on support, only reduced for the dashboard, and it uses some which is kind of 04:55.800 --> 05:04.160 legacy protocol nowadays, so in addition to that, just mention the dashboard is based on 05:04.160 --> 05:15.800 terrapi framework, which is good to create SSL applications, but it is not really good to 05:15.800 --> 05:25.040 have it as the interface to expose it to an internet, because it is not a hydrant enough for 05:25.040 --> 05:34.760 attacks or to be in the front, so that is another issue facing in the past. 05:34.760 --> 05:39.840 In addition, there is no really high availability for the dashboard, so once you file over 05:39.840 --> 05:45.360 the manager, the dashboard moves to the standby manager, it works, but the IP changes and 05:45.360 --> 05:50.960 then point changes, you have to go to the new IP to access the dashboard, so it is not transparent, 05:50.960 --> 05:57.520 there is some kind of HAA, but it is not transparent to the user, and finally, there is no 05:57.520 --> 06:08.680 high availability support for the monitoring stack, so in order to address all these issues, 06:08.680 --> 06:16.560 I will start thinking on how can we implement something to mitigate all of these problems, 06:16.560 --> 06:24.240 so we came with the idea of the management gateway, and the first important thing we did 06:24.240 --> 06:31.920 is to have a single entry point to the cluster, so no more multi entry points, only one single 06:31.920 --> 06:37.440 entry point, which is the management gateway, which is in fact an NGX acting as reverse proxy, 06:37.440 --> 06:44.200 which will be redirected to the services that are running in the backend, so the management 06:44.200 --> 06:52.200 gateway solves the multiple endpoints a problem, but the management gateway itself, it doesn't 06:52.200 --> 06:57.880 solve the authentication fragmentation issue, because if you only enable the management gateway, 06:57.880 --> 07:06.200 you still have issues for the different services, because you will be still using different 07:06.200 --> 07:12.520 user and password to access all of them, so solve this second issue, when to use a new service, 07:12.920 --> 07:21.960 based on the path to proxy project, which basically of low documentation from the management 07:21.960 --> 07:29.000 gateway to an external entity, in this case, to all the IDPs that are supported by the path to proxy, 07:29.000 --> 07:36.680 so this way, each time the user hits this single entry point of the cluster, it will be redirected 07:36.760 --> 07:45.800 to this external IDP to do the authentication, and once the user is authenticated, then the 07:45.800 --> 07:51.880 after proxy will ignore them and make it everything is okay, and then the user, once authenticated, 07:51.880 --> 08:00.440 can access all the services running in the backend, so that's a proxy is really good, 08:00.440 --> 08:08.760 and it solves the problem of external users, so the cameras I explained in the previous slide 08:08.760 --> 08:15.640 through this path, they authenticate, again, the external IDP, and they are in, but it doesn't solve 08:15.640 --> 08:21.480 the problem of internal authentication, because we wanted also to introduce security internal 08:21.480 --> 08:27.800 in the cluster, so anybody going inside the cluster can't easily sniff the traffic of the 08:27.800 --> 08:34.120 different applications running in the cluster, and for that to introduce a second internal server, 08:34.120 --> 08:39.720 and this one is enforcing until S, by using certificates in this case, denolated by 08:39.720 --> 08:45.080 CFEDM, which in this case acts as the root CA for all the certificates that are 08:46.440 --> 08:52.840 hand-led inside the cluster, so if it's time any of the services need certificates, it acts the 08:52.840 --> 09:00.040 CFEDM and CFEDM generates the certificates needed for this service, so this way we also ensure 09:00.040 --> 09:09.480 the security for the internal communication, if we focus now back on this external server, we can 09:09.480 --> 09:18.040 see that the cornerstone need is the auto proxy project, which authenticate against the external 09:18.040 --> 09:25.880 IDP and idea what IDPs do we support, so by default we inferring support or the IDPs that are 09:25.880 --> 09:33.480 supported by the auto proxy project, because we don't know in extra, francy, processing, we just use 09:33.480 --> 09:44.280 definitely proxy and we configured it to connect to our external IDP, we also introduced and did 09:44.280 --> 09:51.880 some integration with IBM SV, which is the IDP of IBM, and also with the red hat, this is 09:51.880 --> 09:57.480 so, which is in fact the downstream version of cake look, just as extra information for you 09:57.480 --> 10:04.280 to know that these are already supported, for the rest, in theory it should work, but we have already 10:04.280 --> 10:11.320 seen some small issues, but this upstream, as soon as you see some issue, in this IDP, go 10:11.320 --> 10:16.600 up and tick it, you can contribute the protocol directly, if not, it is not just small adjustment 10:16.600 --> 10:23.640 of the decent, which is sent back from the IDP to the dashboard, so normally it's like small 10:23.640 --> 10:33.640 weeks. Okay, so now we have this management gateway in the front of the cluster, which is 10:33.640 --> 10:38.680 good thing, in general, to have a single entry point, so we have our user here, 10:39.640 --> 10:44.760 happy, he has single entry point, he has now the authentication of load throughout to proxy, 10:44.760 --> 10:52.120 with the external IDP, everything is good, everything is okay, nice, but what's happened if some of 10:52.120 --> 10:59.080 these backend services, for example, the node where the service running is down, then you 10:59.080 --> 11:05.960 lose the access to the server, to the service, what happens if the off to proxy itself 11:06.040 --> 11:12.680 goes down, then we can't do authentication anymore and lose access to all the backend services, 11:13.400 --> 11:20.280 worse, what happens if the management gateway service itself goes down, then you lose access to 11:20.280 --> 11:25.800 everything, because you can access the cluster. So in order to access all of these issues, 11:26.760 --> 11:33.080 we came with a high availability model to solve the problems, so the first one to solve is 11:33.080 --> 11:38.520 how can we have multiple instances of the management gateway running, and for this we introduce 11:38.520 --> 11:46.040 this architecture where we have virtual IP in the front, and we have not only one that you can have 11:46.040 --> 11:51.800 as much as instances of management gateway as you like, and each one of them will have a kipa 11:51.800 --> 11:57.480 live D, which is monitoring the health of this instance, so normally the kipa live D will 11:58.440 --> 12:05.960 detect which one is healthy and will automatically wrote the virtual IP to this active instance. 12:07.240 --> 12:14.120 For that we'll proxy a part, again we introduce multiple instances in this case as they are 12:14.120 --> 12:19.320 stateless, we just configure each management gateway with this new list and the management gateway 12:19.400 --> 12:23.720 automatically knows which one of them is healthy and use it. 12:26.520 --> 12:32.760 Just to mention that the IDP as it is external, normally the redundancy of this component is 12:32.760 --> 12:39.560 the responsibility of which deploying the cluster, so this is something that is out of scope of our 12:39.560 --> 12:45.480 control. From practice you will have the user reaching the virtual IP, which is in the front, 12:45.800 --> 12:51.400 will be ready to automatically to the health same management gateway, the management gateway, 12:51.400 --> 12:59.400 then do the authentication for autopopsy, and to solve the backend services issues instead of one 12:59.400 --> 13:05.400 instance we can have many of them, and then the management gateway will automatically detect which one 13:05.400 --> 13:15.080 of them is healthy active and running, and it will pick it for the process of the request. 13:16.120 --> 13:23.400 So basically the benefits we get with this new architecture, now we have a IDC based single sign 13:23.400 --> 13:33.880 on which is modern protocol supported by a lot of cooperates and open source IDPs, so you can 13:33.880 --> 13:40.120 now configure your IDP of choice and you have your safe users instead of using the dashboard local 13:40.120 --> 13:48.600 users, you can use your cooperates users directly to access the safe in more controlled way. 13:49.960 --> 13:56.120 We have a hard not front end, so instead of to repy, now we have in gen X in the front, which is 13:56.120 --> 14:06.360 more secure, more battle tested, and more reliable than the to repy. We do that telestrom 14:06.360 --> 14:13.080 in the management gateway, we have also introduced a mutual tell us for internal communication, 14:13.080 --> 14:18.920 so nobody can easily traffic even if it gets inside the local network of the cluster, 14:19.480 --> 14:26.520 and we have some airbac integration inside the the safe dashboard. 14:29.320 --> 14:33.000 Yeah, in terms of operational benefits, you get single entry point to the cluster, 14:33.000 --> 14:41.080 we don't have to remember any more where its service is running, we get the IDC SSO, 14:41.080 --> 14:46.840 we get high availability both for the safe dashboard, which is management part and also for the 14:46.840 --> 14:55.720 monitoring stack, and it also makes our life easier and it comes to certifications management, 14:55.720 --> 15:05.640 and we have a hardened security with the new stack. Here, a little bit the configuration 15:06.360 --> 15:10.520 of the new service, which are a few guys are from a narrowed self-ideon, 15:10.520 --> 15:17.320 configuration yarn files, so basically it's the same syntax we use for other services, 15:17.320 --> 15:24.680 we just specify the service, the port where you want your management gateway to listen, 15:24.760 --> 15:30.600 and then if you enable this authentication, you have to also configure your off-to-proxy, 15:30.600 --> 15:37.640 you can enable the security or not, obviously it is bad idea to live it without disabling it, 15:38.280 --> 15:45.320 and you can bring also your own certificate or if you don't, then safety DM will automatically 15:45.320 --> 15:52.120 detect that you have enabled this SL, but you don't have certificate, then it will go and generate 15:52.120 --> 15:59.720 the self-signed certificate for you. The what-to-proxy configuration is really easy, 15:59.720 --> 16:03.800 these are like the most important fields, what you need basically is the client ID, 16:03.800 --> 16:08.840 which normally you get once you configure your IDP, and the client secret, and finally the 16:08.840 --> 16:15.400 URL where your IDP is listening, and that's all. This slide represents a little bit like 16:16.360 --> 16:22.680 since now we have different options for security configuration, and this slide we can see all of them, 16:22.680 --> 16:29.640 so by default, if you don't do anything, you have SSL only for dashboard, there is an 16:29.640 --> 16:36.760 ultimate MTLS for internal communication, there is no authentication for the monitoring stack, 16:36.760 --> 16:46.120 Grafana uses only basic authentication. If you enable this slide, if you enable this slide, 16:46.120 --> 16:52.920 which was introducing the reference, you get some basic authentication for the difference, 16:55.160 --> 17:01.080 monitoring services, and for the dashboard, but there is no TLS internal TLS, and you have no 17:01.160 --> 17:07.640 benefit access. If you enable the monitoring gateway, only you get the TLS, you get them TLS for 17:07.640 --> 17:14.040 the internal communication, but you still have to deal with this fragmented authentication issue. 17:14.760 --> 17:23.640 So the access here, I would say is partial, but if you want the full enterprise configuration, 17:24.280 --> 17:31.400 you will need to enable both the monitoring gateway, and also proxy. So this way, 17:31.400 --> 17:39.880 you get TLS enabled for the services, MTLS enabled internally for the internal services communication, 17:39.880 --> 17:47.720 you get SSL access for all the services. So that would be the recommended 17:47.720 --> 17:56.120 configuration for the production deployment, where you care about security, and they recommend one 17:56.120 --> 18:07.400 course. In terms of known limitations, just to make users aware about a bunch of them, 18:07.400 --> 18:12.920 a plenty of use, by default, it doesn't have the concept of cluster, not like for example, 18:13.320 --> 18:21.080 manager. When you have any instances running, and one of them goes down, 18:21.960 --> 18:28.920 then this one will always have gaps in the data. Like if the graphana is using this instance, 18:28.920 --> 18:34.200 it will have this gap and with the current architecture, it is not easy to solve this issue. 18:35.160 --> 18:41.240 In TLS, in this first, Tentacle release is not enforced for all the backend services, 18:41.240 --> 18:50.760 we are working to solve that. Autoproxy is meant for humans, users, it is not 18:50.760 --> 18:55.560 mean for machine-to-machine authentication. So if you want to access the cluster, 18:55.560 --> 19:01.160 externally, for example, to access promises to get some data, if the market makes it 19:01.160 --> 19:08.200 is enabled, it won't let you go in the cluster. We are also exploring how can we 19:08.200 --> 19:15.480 allow this external access in some controlled way, because we have users that need that kind of access. 19:16.760 --> 19:22.280 And finally, in the Tentacle release, we have no monitoring and auditing for the management 19:22.280 --> 19:31.160 get to a service of the auto proxy, but the idea is to introduce the auditing at least, 19:31.160 --> 19:34.760 and monitoring in the next release, which will be umbrella. 19:38.360 --> 19:42.280 Yeah, so that's basically the roadmap. I think I have already gone through all this point, 19:42.280 --> 19:48.120 to extend MTLS to the rest of services. The tech, what possibilities are there, 19:48.120 --> 19:55.400 to add the machine-to-machine support, added monitoring for the management getway, and added 19:55.400 --> 20:03.240 auditing support? Yeah. Unfortunately, we have all of these issues, 20:03.240 --> 20:08.200 they have a demo, but since we're running on some issues, I think we don't have enough time to see 20:08.200 --> 20:10.200 the demo. Sorry for that. 20:10.280 --> 20:21.560 Yeah. Go ahead, please. So, we already set up a proxy in front of the dashboard, 20:22.280 --> 20:28.120 but before dismantle the existing, we don't have any integration, so you still need to log in 20:28.120 --> 20:32.840 to the dashboard. So, how did you start that? Does it sample between off and the dashboard, 20:32.840 --> 20:41.080 or base communication? So, the most important, yeah, sorry. Basically, the question is, 20:41.080 --> 20:45.960 they have already set up auto proxy in the dashboard, because, obviously, 20:45.960 --> 20:51.240 management getway didn't exist in the past. So, it's interesting. Your question, how we solved the 20:51.240 --> 20:58.680 issue of how the dashboard gets the benefit of the authentication. The answer is, it is not 20:58.680 --> 21:04.760 like a direct communication. You get this benefit by going through the NGNX. So, in the NGNX, 21:04.760 --> 21:11.000 which is acting as a reverse proxy, it is in the front. So, any user hitting the entry point, 21:11.880 --> 21:19.880 the NGNX has a model, which is off to off, with checks if the user is authenticated or not. 21:19.880 --> 21:25.640 And if it is not, NGNX leads you to define rules to say, oh, go and use this to do authentication. 21:25.720 --> 21:32.040 And, by default, it comes with auto proxy. So, that's why we don't use HAA proxy, 21:32.040 --> 21:39.000 we use NGNX for this part of the set. And then, also, proxy with this minimal configuration, 21:39.000 --> 21:45.080 you can go to your external IDP to the authentication. And then, it communicates back to the 21:45.800 --> 21:55.160 NGNX that the users have indicated through a token. And then, the NGNX lets you access the cluster. 21:55.960 --> 21:58.600 Yes, how do you ask the credential for the dashboard itself? 21:59.640 --> 22:02.840 You don't need credentials for the dash, but that's the idea. So, with this. 22:05.400 --> 22:10.520 It is, you won't see the dashboard directly. So, the idea once you enable the 22:10.520 --> 22:15.560 demeritment gateway, it will be your only entry point to the cluster. So, you will configure your 22:15.560 --> 22:21.560 firewall, nobody can access another port in the cluster. You only have the port of the 22:21.560 --> 22:29.640 moment gateway as the external port. And then, you go through this workflow. And the dashboard 22:29.640 --> 22:37.640 gets the authentication through the NGNX and the draft proxy. You don't need, so, we will access the 22:37.640 --> 22:45.480 dashboard directly once you do authentication in the IDP. This is some configuration for users. 22:45.560 --> 22:53.960 It has been added recently by the dashboard team. So, you can even configure which users 22:53.960 --> 23:00.600 you want to use for your dashboard access through some additional configuration. But, you don't have 23:00.600 --> 23:06.600 to use local credentials that the idea of the money may get to use your IDP users directly. 23:07.480 --> 23:13.240 Yeah, but the NGNX has to pause the credential to the dashboard right before it gets in the proxy. 23:13.240 --> 23:15.320 So, the NGNX added. Yes. Yes. 23:15.320 --> 23:23.480 As a basic application. No, no, no. It used GWT, which is the way how the draft to proxy handle 23:23.480 --> 23:31.400 the information from the IDP to the dashboard. So, when the user goes to the IDP and does the 23:31.560 --> 23:39.720 authentication, IDP generates a cookie for it. This cookie is sent to a proxy of to proxy 23:39.720 --> 23:46.680 validates the cookie. And then, it's pass it to the NGNX and this information back to the dashboard. 23:46.680 --> 23:53.720 So, dashboard can process this, this cookie to get information of which user is accessing the dashboard. 23:53.720 --> 24:09.320 No, it's IDC. Yes. Now, we support the IDC. Yes. It's modern protocol. So, you can have any IDP 24:09.320 --> 24:11.480 externally configured for your cluster.