WEBVTT 00:00.000 --> 00:13.240 So, this presentation is about reducing technical debt with reproducible shell workflows. 00:13.240 --> 00:19.520 I believe the approach that we take with regard to documenting system is quite different 00:19.520 --> 00:24.680 from most stocks that I've seen here today, and I believe that makes it actually even more 00:24.680 --> 00:28.160 interesting, or at least I hope so. 00:28.160 --> 00:34.240 I'm, well, we first introduce ourselves, maybe Collin, we can do this right to left. 00:34.240 --> 00:46.840 Okay, so, in myself, I'm Collin Hav, an employee at 9, with Hugo, and a lot of people here, 00:46.840 --> 00:55.880 with you, but that's software, and came here to participate to talk about this, and 00:55.880 --> 00:57.880 you go, you can present yourself. 00:57.880 --> 01:02.440 Yeah, I'm here with Collin, I'm in embedded software engineer at mine, I've been working 01:02.440 --> 01:10.040 on mine for seven years now, and I've been working for several projects, for several projects 01:10.040 --> 01:17.080 in Avionics, and one of the more recent projects was for Boeing, where Boeing asked us 01:17.080 --> 01:24.560 to implement a new operating system, Linux operating system, using the built-in system for 01:24.560 --> 01:31.440 the TALOS, the Raptor TALOS II hardware, Raptor TALOS II is, I would say, a very advanced 01:31.440 --> 01:36.880 operating, it's a very performant machine, it's also very expensive, and we encountered 01:36.880 --> 01:44.640 quite a few challenges when implementing this system. 01:44.640 --> 01:49.000 There are two elements to this presentation, so the things, the software artifacts that 01:49.000 --> 01:55.320 we implemented, and then a methodology that we used to implement those software and artifacts 01:55.320 --> 01:58.840 basically to execute the project. 01:58.840 --> 02:05.320 Boeing explicitly asked me to add that the Boeing work was for Raptor TALOS II, Linux power 02:05.320 --> 02:10.240 and I operating system functionality, but that Boeing is also neutral on the workload concepts 02:10.240 --> 02:17.320 and methodology used to carry out the project, which is the main focus of this presentation. 02:17.320 --> 02:26.040 I have been in the main architect of the Neural Simulator called Genesis in computational 02:26.040 --> 02:32.600 biology, and one of the main targets of this simulated was making improving the quality 02:32.600 --> 02:38.000 of scientific publications, and some of you people may know that there's a serious problem 02:38.000 --> 02:43.800 with the quality of many scientific publications, because the results are not reproducible. 02:43.800 --> 02:49.640 So one of the main focus of one main focus of the Genesis Simulator was making these scientific 02:49.640 --> 02:57.600 results reproducible, improving the transparency, and while doing this, we start to make 02:57.600 --> 03:01.680 a distinction between two types of knowledge. 03:01.680 --> 03:06.960 On one hand, there's the process knowledge that you develop while you're carrying out a project, 03:06.960 --> 03:11.760 while you're doing your research, while you're implementing your software system. 03:11.760 --> 03:16.280 In a second, there's the product knowledge, when you do your product delivery, you publish 03:16.280 --> 03:23.480 yourself to it, there's the knowledge that you want to document functionality of the product 03:23.480 --> 03:30.680 that you made, so that are the typical manuals, that are diagrams, using manuals, specification 03:30.680 --> 03:32.440 manuals, and so on. 03:32.440 --> 03:38.640 There's a different fundamental difference between these two types of knowledge. 03:38.640 --> 03:45.440 Similarly, the product knowledge is implicitly consumed by the users, but the process knowledge 03:45.440 --> 03:52.640 is never shown to the user, a user is never exposed to it, nevertheless, if you want 03:52.640 --> 03:58.240 to continue to develop the product, if you want to evolve the product, you will require 03:58.240 --> 04:00.480 that process knowledge. 04:00.480 --> 04:04.800 This presentation has a strong focus on the process knowledge. 04:04.800 --> 04:09.680 There are many good tools for product knowledge, process knowledge, I believe, is underestimated, 04:09.680 --> 04:14.640 nevertheless, enormously important for software development. 04:14.640 --> 04:21.680 So in the Genesis Simulator, we implemented tools that supported the delivery of both process 04:21.680 --> 04:23.560 knowledge and product knowledge. 04:23.560 --> 04:29.040 The product knowledge in a scientific context is quite simple, it's a scientific publication, 04:29.040 --> 04:33.120 but the process knowledge is hardly documented. 04:33.120 --> 04:37.760 As a consequence, results are not always reproducible, especially in embedded systems we face 04:37.760 --> 04:44.200 a similar challenge, reproducing embedded software artifacts is not easy. 04:44.200 --> 04:48.560 There are special build systems like build route, which I will show later on. 04:48.560 --> 04:53.320 We developed several tools for the in the context of Genesis Simulator. 04:53.320 --> 04:59.320 Vigilia has a focus on product knowledge, workflow, the workflow engine that we will present 04:59.360 --> 05:03.600 over here as a focus on process knowledge. 05:03.600 --> 05:10.320 These are open source software tools available from GitHub. 05:10.320 --> 05:16.520 So I'm now going to do a very quick tour about the workflow engine, and especially a focus 05:16.520 --> 05:23.080 on how exactly do you use it, how exactly what is it, how does document work flows. 05:23.080 --> 05:29.880 So I'm going to go to a live demonstration, and maybe the first thing that I will 05:29.880 --> 05:36.120 to hear is show workflow as executable. 05:36.120 --> 05:41.720 So using the workflow engine, you'll implement several projects, or you can implement 05:41.720 --> 05:42.720 several projects. 05:42.720 --> 05:45.640 There's a full isolation between all the different projects. 05:45.640 --> 05:52.200 If I list the projects available on my laptop, I see there's something like 10, quickly 05:52.200 --> 05:55.960 go through a few of them, so there's the build route workflow. 05:55.960 --> 06:02.760 In this case, workflow engine acts as a wrapper around the build route build system. 06:02.760 --> 06:08.640 There is, for instance, over here, the mind workflow. 06:08.640 --> 06:16.080 The mind workflow, that's a workflow engine configuration that assists me in the infrastructure 06:16.080 --> 06:18.560 maintenance of the mind network. 06:18.560 --> 06:23.760 This includes, for instance, the website of mind, where I sometimes do updates, mind workflow 06:23.760 --> 06:31.560 captures, there is a configuration that captures the workflows that I employ when developing 06:31.560 --> 06:34.560 that website. 06:34.560 --> 06:41.200 And then for another example is a fleet miner workflow, fleet miner workflow is a configuration 06:41.200 --> 06:49.600 that assists in the development of an internet of things simulation and maintenance management 06:49.600 --> 06:51.240 system. 06:51.240 --> 06:57.320 And in that case, the workflow engine orchestrates the behavior of many different IoT devices. 06:57.320 --> 07:05.320 Connics into a dashboard, visualize their dynamics. 07:05.320 --> 07:14.720 OK, so what I'm going to do now is I'm going to give a more concrete idea of how 07:14.720 --> 07:15.720 everything works. 07:15.720 --> 07:19.440 I'm quickly going to create a workflow project. 07:19.440 --> 07:28.000 So I make a directory, CD into that directory, there we go, the directory is empty, of 07:28.000 --> 07:29.000 course. 07:29.000 --> 07:32.640 And then I start using the workflow engine. 07:32.640 --> 07:37.880 As you see, I type a few letters, and I use bash completion, bash completion, is a main 07:37.880 --> 07:42.960 element in exploring existing workflow configurations. 07:42.960 --> 07:50.040 If I tap twice here, you see, I will have a project start. 07:50.040 --> 07:55.800 Element, if I tap bash completion again sometimes, the bash completion is used as a kind 07:55.800 --> 08:00.040 of guide for red type of arguments, a common workflow could expect. 08:00.040 --> 08:07.440 In this case, the name of the workflow configuration that I'm about to create, there you 08:07.440 --> 08:11.160 go, workflow engine gives me some additional instructions. 08:11.160 --> 08:16.880 One of the elements here is that I want to install this workflow, making it globally 08:16.880 --> 08:20.880 available on my laptop, there you go. 08:20.880 --> 08:28.840 And from now on, I do have a false-them workflow with bash completion. 08:28.840 --> 08:34.440 The project that I create is based on a template, template has a few examples. 08:34.440 --> 08:39.320 One of these examples is a Python examples, and then Python commands. 08:39.320 --> 08:45.840 So this is just a kind of toy example for you to start with your workflow project. 08:45.840 --> 08:48.760 Just a comment here on terminology. 08:48.760 --> 08:51.640 So you have the false-them workflow executable. 08:51.640 --> 08:58.160 Then Python examples is what is referred to as the target, and Python commands is what 08:58.200 --> 09:02.960 is referred to as the operation, working on that target. 09:02.960 --> 09:09.760 If I execute this workflow, just a very simple toy like example, it echoes a few things to 09:09.760 --> 09:16.520 the screen, if I ask for interactions, it will tell me what it would do if I execute that 09:16.520 --> 09:18.760 workflow. 09:18.760 --> 09:21.840 How do I add a new workflow? 09:21.840 --> 09:22.840 Very simple. 09:22.920 --> 09:30.160 As usual, we have bash completion to assist us to figure out how things work. 09:30.160 --> 09:34.920 We have this operation workflow add. 09:34.920 --> 09:38.200 I type bash completion to see what should be. 09:38.200 --> 09:40.240 How should this work? 09:40.240 --> 09:42.040 I need a target name. 09:42.040 --> 09:46.280 Well, I'm just going to use the target name that I have over here, and an operation 09:46.280 --> 09:47.280 name. 09:47.280 --> 09:51.880 Let's call our new workflow operation one. 09:51.920 --> 09:53.200 There we go. 09:53.200 --> 10:02.080 Create a new workflow in file, and so let's see what's in this file. 10:02.080 --> 10:08.280 So this file, that implements one workflow for the workflow configuration of false-them, 10:08.280 --> 10:09.720 is a regular Python file. 10:09.720 --> 10:12.360 You can import new Python libraries. 10:12.360 --> 10:15.680 You can do anything that you can do with Python in this file. 10:15.680 --> 10:18.520 It's a regular Python file. 10:18.520 --> 10:23.640 And it implements this workflow Python example operation one. 10:23.640 --> 10:36.320 So if I execute this workflow, nothing happens. 10:36.320 --> 10:43.520 If we look at the implementation of this workflow, indeed, nothing happens. 10:43.520 --> 10:49.920 For low and back to this file, this file implements three different Python functions. 10:49.920 --> 10:57.040 So the implementation of the workflow itself, the configuration of the bash completions. 10:57.040 --> 11:02.120 And then here at the end, you'll put the help page. 11:02.120 --> 11:09.520 I'm slowly building towards documenting these workflows. 11:09.520 --> 11:19.480 If we want to add commands to the workflow, we use, for instance, in this case, execute 11:19.480 --> 11:21.720 commands, execute command array. 11:21.720 --> 11:29.200 These are API calls into the workflow engine, and I'm slowly building up how this slowly 11:29.200 --> 11:33.360 building up the complexity of the workflow that I'm about to implement. 11:33.360 --> 11:42.400 So if I write this file, there we go, Python examples operation one. 11:42.400 --> 11:45.800 Now suddenly this operation one workflow is not empty anymore. 11:45.800 --> 11:54.600 The commands are executed, and I can obviously ask for the interactions, there we go. 11:54.600 --> 11:58.760 So this all works nicely together. 11:59.160 --> 12:08.560 With regard to the bash completion, if I tap twice, I get the bash completions. 12:08.560 --> 12:09.960 They're not filled in yet. 12:09.960 --> 12:12.160 How exactly does this work? 12:12.160 --> 12:16.160 So let's add some more meaningful bash completions. 12:16.160 --> 12:21.160 I'm going to choose the typical examples of food. 12:21.160 --> 12:26.160 Say, food is an argument that could be added to this workflow. 12:26.280 --> 12:31.080 After food, we don't expect any further completions. 12:31.080 --> 12:34.600 The second possible argument is bar. 12:34.600 --> 12:43.920 The bar argument could possibly be followed by a bar one, 12:43.920 --> 12:50.480 which is then followed by nothing, and then possibly followed by a bar two argument, 12:50.480 --> 12:53.360 which is followed by nothing. 12:53.440 --> 12:57.240 So I hope I didn't make any typos. 12:57.240 --> 13:02.560 If I go back to the workflow itself, and I tap completion, 13:02.560 --> 13:05.640 there we go bar for the possible completions. 13:05.640 --> 13:08.560 Food isn't followed by anything else. 13:08.560 --> 13:13.840 Bar is followed by bar one or bar two. 13:13.840 --> 13:16.720 If I just execute the workflow, it's still the same. 13:16.720 --> 13:21.280 It's still echoing few commands. 13:21.280 --> 13:27.560 Now there is one option that I would like to show here too. 13:27.560 --> 13:31.760 At the top of the file, we can read the sentence rows are available 13:31.760 --> 13:34.520 as a variable global rows. 13:34.520 --> 13:39.040 And here we see row that's a row that is defined 13:39.040 --> 13:44.040 in the false and configuration, and that can be added to 13:44.040 --> 13:49.440 as an argument on option to these API functions. 13:49.440 --> 13:54.760 So I'm first going to go back to false and workflow, 13:54.760 --> 14:00.840 built in, and then show what rows are available. 14:00.840 --> 14:03.760 If I tap twice here, built in row print, 14:03.760 --> 14:06.600 I see there are several rows available, 14:06.600 --> 14:11.880 and I'm just choose kind of arbitrary, the T-mix rows, 14:11.880 --> 14:12.760 ask the print them. 14:12.760 --> 14:14.640 There are two T-mix rows, one 14, 14:15.200 --> 14:17.920 one 14-mix server. 14:17.920 --> 14:22.680 And what these rows do, they start T-mix sessions, 14:22.680 --> 14:26.360 such that you can execute commands in those T-mix sessions. 14:26.360 --> 14:30.040 Then obviously you can attach a Qmoo engine or a different 14:30.040 --> 14:34.000 or a serial console or something to this T-mix row. 14:37.800 --> 14:43.320 To start the T-mix row, I first need to create, 14:43.320 --> 14:51.600 created. So I start now two t-mux rows. I can attach or t-mux sessions. I can attach 14:52.600 --> 14:54.600 to such a session. 14:54.920 --> 14:56.920 Here we go. 14:57.800 --> 14:59.800 There we go. This is the 15:00.440 --> 15:05.080 Here up is I've a terminal attached to the t-mux session that I just started. 15:06.960 --> 15:08.960 What I can now do 15:10.880 --> 15:12.880 is I can 15:14.280 --> 15:18.400 have some of these commands executed in the t-mux session. 15:19.600 --> 15:25.040 By the time syntax requires me to first declare this variable 15:27.960 --> 15:33.920 global rows and then I define a variable t-mux row equals 15:34.160 --> 15:36.160 t-mux rows. 15:38.640 --> 15:44.640 I have the thing for a moment. I think the name was t-mux device. 15:50.880 --> 15:58.880 Okay, and then here I add the option row equals t-mux row. 15:59.200 --> 16:01.200 t-mux row. 16:02.560 --> 16:05.200 There we go. I write this file. 16:07.920 --> 16:10.720 I then ask again for the interactions. 16:14.240 --> 16:17.440 Python examples operation when interactions 16:18.480 --> 16:23.920 there we go. And we see that the workflow engine would now echo Python 16:24.000 --> 16:29.440 command 1 to the local console and echo these other commands 16:30.800 --> 16:35.120 to the t-mux session. So if I execute this, there we go. Python 16:35.120 --> 16:38.160 command 1 is echoed here in the local console. 16:38.880 --> 16:42.720 And if I go switch to the t-mux session, then I see that Python 16:42.720 --> 16:47.040 command 2, Python command 3 are echoed in that session. 16:47.680 --> 16:52.400 Now, so far these are all toy-like examples just echoing. 16:53.200 --> 16:57.120 Obviously, you can imagine that you attach this to a QMO instance, 16:57.120 --> 17:01.360 that you start to do things with QMO through the telnet port or through an 17:01.360 --> 17:05.680 SSH. This is all possible. It's just about defining the right 17:05.680 --> 17:08.560 rows and combining these things in the right way. 17:13.520 --> 17:20.720 If indeed, we kind of functionally scale up and we look again 17:22.480 --> 17:27.840 at what projects were available over here, then we see that we do have a 17:27.840 --> 17:33.920 tallos to workflow. So this brings us back to the project that we have been 17:33.920 --> 17:48.480 doing for Boeing. And if I then ask for all the workflows of tallos 17:48.480 --> 17:54.560 to, you see that we implemented many different workflows while 17:54.560 --> 18:00.880 executing this project. I'm quickly going to show a few examples 18:00.880 --> 18:11.520 regarding to source code. So the source code of that we, 18:12.560 --> 18:17.680 so if I use a source code target, workflow engine again wraps around 18:18.640 --> 18:22.160 if I ask completion for a configure, what you see over here, 18:22.160 --> 18:26.160 all the different and a difficult fix available for this 18:26.160 --> 18:32.000 particle of source code. We implemented some. This is the tool chain, 18:32.000 --> 18:37.600 so you've got type O with tab completion. So what you see is that the workflow engine 18:37.600 --> 18:42.480 dynamically finds the right difficulties, automatically generates completion, 18:42.480 --> 18:47.120 so there are no type of possible over here. If I just 18:47.120 --> 18:52.160 configure, then you see how the workflow engine exactly wraps around 18:52.160 --> 18:54.720 the build-through build system, it sets the right 18:54.720 --> 18:58.480 environment variables, changes to the right directory and so on. 18:58.480 --> 19:03.840 So now if I would start to build itself like this, I'm not going to 19:03.840 --> 19:09.360 execute this because it's going to take a long time, but then the 19:09.360 --> 19:15.200 build commands would be started to build the tool chain that we need 19:15.200 --> 19:18.080 for the operating for building the operating system for that all of 19:18.080 --> 19:28.640 us to. And this brings us much closer to the real work. 19:28.640 --> 19:38.080 Okay, so I will present quickly the way I've been using 19:38.080 --> 19:44.160 workflow engine to support for go during the development 19:44.160 --> 19:52.800 of a driver of a modification for the driver for the BCM 155719. 19:52.800 --> 19:56.000 And how I use the workflow engine to quickly get on board and 19:56.000 --> 20:03.600 do stuff and accelerate my workflow while keeping it documented. 20:03.600 --> 20:11.200 So a bit of context first, we will need a bit of support to enable the 20:11.200 --> 20:17.600 cross-compilation of this BCM firmware to the target architecture of 20:17.600 --> 20:22.320 PowerPC 64 began then, began then a bit unusual, so there was a lot of 20:22.320 --> 20:28.720 hurdles, we can re-galing that. And so my first step was to just set up the dev 20:28.720 --> 20:34.480 environment, usually it would be done with a Docker container or maybe a giant 20:34.480 --> 20:40.720 batch script or via stuff, but this time it was done with the workflow engine, so 20:40.720 --> 20:46.800 I originally knew I would introduce me to workflow engine very well, yeah. 20:46.800 --> 20:51.840 So I could just pull the workflow configuration, install it, and see what was 20:51.840 --> 20:58.000 available to me to reproduce a basic build. And actually, like a few 20:58.800 --> 21:04.800 hours, I was able to have a native build, which is a good first step for my work. 21:07.200 --> 21:11.040 But the basic, the native build is not what I wanted, I wanted to cross-compile, 21:11.040 --> 21:17.040 so how would I do that? Well, I will be very quickly clear because I have much time 21:17.040 --> 21:23.120 and that assist route, which is basically a way to, to avoid 21:23.120 --> 21:29.360 linking against the native libraries, which would be a program for cross-compiling code. 21:30.320 --> 21:35.680 And I used the deviant and the bootstrap to easily pull full Linux 21:35.680 --> 21:40.080 route system, deviant route system in this case, my computer for the right 21:40.080 --> 21:48.560 architecture. And so that was done for the assist route building, so I could put that into 21:48.640 --> 21:52.240 workflow, so that you could use it and discover it later, and ask me question and 21:52.240 --> 21:57.680 note it. And I generally need to set them really big and long commands over Slack, 21:58.640 --> 22:06.800 as I'm also sorry. And yeah. So next step, I need to fix some problems with the CMake 22:07.840 --> 22:17.920 tool chain of the firmware. Once again very quick, the project builds some of its 22:19.520 --> 22:24.880 build tools in the project, which are there, which if you cross-compile them, then you cross-compile 22:24.880 --> 22:31.360 the build tools, which is not good because you need them to build. So I had to modularize this 22:31.360 --> 22:40.800 CMake system to enable to fix the problems. I also had a bit of an issue with the linker, 22:40.800 --> 22:48.880 would not do some stuff for this architecture, which was sold by pulling and compiling a 22:48.880 --> 23:00.480 specific linker, which brings me to this schematic, where each square is basically a workflow. 23:00.480 --> 23:06.560 I don't need to send a giant bass strip in a pull request. I just have five more workflow 23:06.560 --> 23:16.240 then you go and try iterate and give it feedback on. And later, I can just reuse these workflows 23:16.240 --> 23:25.360 to re-dive into the work that we had done for this really easily in a few comments. 23:25.360 --> 23:35.280 Yeah. Yeah. You go, I give you the flow. Okay. Yeah. That's fine. So what in the end comes out 23:35.280 --> 23:46.960 of this is that we can build a manual for this NICDAR hit. Okay. And so in this case the workflow engine 23:47.680 --> 23:54.080 has produced a PDF document of all the work that Colleen has done. It has an introduction, 23:54.080 --> 23:59.200 you see the same diagram that Colleen just showed, and it has all the work flows, 24:00.000 --> 24:06.240 enumerated, and with all the documentation for how to do the cross compilation of this NIC firmware. 24:13.840 --> 24:19.760 Okay. So all these tools are available. All the repositories that we use are available from 24:19.840 --> 24:30.960 our GitLab repository or our GitLab pages of mind. Now I quickly would like to end with this diagram. 24:30.960 --> 24:35.120 I see at the end already as a smile and he knows how much effort I put in this diagram. 24:36.800 --> 24:42.640 So I showed the workflow engine as a simple wrapper around build route. I've shown the workflow 24:43.200 --> 24:50.800 engine for infrastructure maintenance. I've explained about fleet-minded as a software component 24:50.800 --> 24:59.200 integration system. And I believe with the growing landscape of open source software components, 24:59.840 --> 25:06.320 the technical depth that emerges from the integration, hopefully the useful integration of 25:06.320 --> 25:12.000 some of these software components will greatly increase. And I believe there will be a greater 25:12.000 --> 25:20.480 need for components such as or systems such as the workflow engine to document this technical 25:21.120 --> 25:26.320 and to address that problem. That's it. Thank you. 25:35.120 --> 25:42.080 Is it done for questions? The recording automatically cut off, but if there are any questions, 25:42.400 --> 25:45.920 this is the final top of the day. Now it's your text. 25:48.960 --> 25:59.440 Mark, the workflow configuration is putting a Git repository that can be published to GitHub or GitLab 26:00.080 --> 26:08.320 or whatever medium you want. And then the workflow engine has built in workflows to re-estantiate 26:08.400 --> 26:15.040 that's configuration as a useful configuration. So to make it available on our machine. 26:25.440 --> 26:26.880 Yeah, that's a very good question. 26:26.880 --> 26:37.440 Well, the workflow engine is not a build tool. That's one. I believe all these software 26:37.440 --> 26:41.680 components that we see on the screen now, they have their own dedicated purpose. They're very 26:41.680 --> 26:46.720 good at it. And the workflow engine just wants to integrate them. That's one part of the answer. 26:46.720 --> 26:53.840 The other part of the answer is for us, if you compare to Ansible, there are a few distinguishing 26:53.920 --> 27:00.240 factors. The workflow engine is directly programmable in Python, which I believe, where I believe, 27:00.240 --> 27:05.600 I'm not that familiar with Ansible. I must say, but I believe Ansible has its own external 27:05.600 --> 27:10.400 type of language. Then another, and I believe more important, 27:12.320 --> 27:17.520 difference is that the workflow engine has a focus on documentation and knowledge sharing. 27:18.080 --> 27:22.080 While Ansible is actually a configuration tool. So that's a very 27:22.080 --> 27:27.520 principal's difference. And that's also the reason why we can automatically generate PDF documents 27:27.520 --> 27:35.280 from these workflows. I didn't show that much about that. I apologize for that. But that's definitely 27:35.280 --> 27:45.200 not the purpose of Ansible. The workflow engine executes the code. Yes. So in that sense, you also have 27:45.200 --> 27:50.640 the element that these hell of pages and the code, the workflows themselves are very close to each 27:50.640 --> 28:02.720 other. So it's easy to keep them synchronized. What questions? Okay. Thank you very much.