WEBVTT 00:00.000 --> 00:08.480 So last session of the day, thank you for sticking it out with us. 00:08.480 --> 00:13.240 To close out the day, this final talk is going to look at how AI is reshaping open source 00:13.240 --> 00:14.240 communities. 00:14.240 --> 00:22.880 Please welcome Amanda, Wagner, and David Allen. 00:22.880 --> 00:29.080 Hello, when I asked David earlier, how many people he thought would be in this session, 00:29.080 --> 00:32.200 he said 8 to 80. 00:32.200 --> 00:34.360 So thank you all for being here. 00:34.360 --> 00:40.920 We are talking about everybody's favorite subject, AI, the AI shockwave, and open source 00:40.920 --> 00:41.920 communities. 00:41.920 --> 00:49.360 How AI is reshaping the foundation of open source communities and communities as a whole. 00:49.360 --> 00:56.160 And I think it's really important to note that, to be on theme, AI put the visuals of 00:56.160 --> 00:58.560 this presentation together for us. 00:58.560 --> 01:07.840 So in joy, hi, I'm Amanda, that's me. 01:07.840 --> 01:12.640 I am the senior manager of developer communities at Grafana Labs. 01:12.640 --> 01:19.120 I have over a decade of experience in community strategy, shaping communities, maintaining 01:19.120 --> 01:26.200 them, starting them programs and initiatives, and I'm happy to be with you guys today. 01:26.200 --> 01:27.200 My name is David. 01:27.200 --> 01:28.200 I'm from Grafana. 01:28.200 --> 01:29.200 I do developer advocacy work. 01:29.200 --> 01:34.200 I've been an open source contributor since, before open source was a term, I started 01:34.200 --> 01:39.080 off with free software foundation, package, maintainer, ship. 01:39.080 --> 01:44.640 So to get us started here, what is the shockwave that we're actually talking about? 01:44.640 --> 01:50.080 Within Grafana relative to our community, our open source community, we do regular metrics 01:50.080 --> 01:53.880 to kind of monitor the health and progress of the community. 01:53.880 --> 01:58.400 Because our overall ecosystem is growing, I've gotten a little bit lazy and come to 01:58.400 --> 02:03.320 expect that our community would grow roughly 8 to 18 percent a quarter, almost no matter 02:03.320 --> 02:04.320 what I did. 02:04.320 --> 02:07.920 I mean, I could fall asleep and the community would grow because there's an influx of 02:07.920 --> 02:09.920 people who are interested. 02:09.920 --> 02:14.560 And so what we got our wake-up call when we looked at the numbers one quarter and it basically 02:14.560 --> 02:16.520 went down by 27 percent. 02:16.520 --> 02:17.520 That does not happen. 02:17.520 --> 02:20.800 So we had a lot of internal discussions to be messed up. 02:20.800 --> 02:25.360 The measurement is the data quality bad, something else happened, no, the number of people 02:25.360 --> 02:29.520 entering into the community dropped by 27 percent in one particular quarter. 02:29.520 --> 02:34.560 So we started asking around and we found, like for example, on the website team and also 02:34.560 --> 02:39.680 with the number of other companies that like traffic, organic traffic to the website, was 02:39.680 --> 02:42.440 broadly down by like 30 percent. 02:42.440 --> 02:48.800 And a lot of the reason why boiled down to question and answer behavior, being deferred 02:48.800 --> 02:52.240 to things like chatGPT and LOMs. 02:52.240 --> 02:56.800 And so in the wider marketing world, so to speak, this is not a mystery, this is a broad 02:56.800 --> 02:59.480 trend that everybody is seeing. 02:59.480 --> 03:05.000 The point here is that it's happening to community sign-up pages, your discourse instances, 03:05.000 --> 03:09.320 your slacks, your telegrams, all of those sorts of things. 03:09.320 --> 03:11.920 So this is the trigger. 03:12.920 --> 03:19.520 Yeah, and while this was a shock to us, like David is saying, it's not unique to us. 03:19.520 --> 03:29.320 Most famously, Stack Overflow when chatGPT launched had a 35 to 50 percent decline in traffic. 03:29.320 --> 03:32.240 So that's not really like a party. 03:32.240 --> 03:40.880 So what this is indicating, systemic signals across open source is, again, AI is replacing 03:40.920 --> 03:43.800 these peer-to-peer learning moments. 03:43.800 --> 03:48.600 It's decreasing that human interaction and connection. 03:48.600 --> 03:55.640 And those novice participants who would have formally gone to something like a Q&A 03:55.640 --> 04:04.640 for a mislock channel are now going to AI for that initial touch point. 04:04.640 --> 04:10.320 So the way that we look at our community journey is through these phases. 04:10.320 --> 04:14.160 The phase zero being those novice participants. 04:14.160 --> 04:19.120 It's that first touch point asking dumb questions in public. 04:19.120 --> 04:22.920 And then phase one is their first contribution. 04:22.920 --> 04:24.200 Maybe writing content. 04:24.200 --> 04:25.480 They're building confidence. 04:25.480 --> 04:29.880 They're getting familiar with the community you've made available. 04:29.880 --> 04:34.360 And two is where the magic starts to happen. 04:34.360 --> 04:39.960 This is a pattern of escalation, regular participation, and feeling really integrated into 04:40.000 --> 04:41.280 the community. 04:41.280 --> 04:43.000 And then phase three. 04:43.000 --> 04:47.680 This is the one percent of these community folks. 04:47.680 --> 04:50.960 It is the crem-delacrem of your community. 04:50.960 --> 04:54.840 The champions, the advocates, the maintainers and leaders. 04:54.840 --> 04:59.080 And the really important thing about phase three, while there's Lisa 04:59.080 --> 05:04.400 Mount of them, they're pumping back into other phases by driving content, 05:04.400 --> 05:07.600 creating educational material, mentoring. 05:07.640 --> 05:12.040 A good example of this for us is traditionally in the past. 05:12.040 --> 05:15.760 If you asked a question on our community forums, 05:15.760 --> 05:21.200 there was an incredible likelihood that a champion in our formal champions program 05:21.200 --> 05:23.240 is going to respond to you. 05:23.240 --> 05:26.600 And so they end up feeding back into this funnel. 05:26.600 --> 05:32.080 The key insight here that we've mentioned on these last few slides is 05:32.120 --> 05:38.520 that AI competes most directly with those phase zero and phase one. 05:38.520 --> 05:43.880 But those experts and veterans, they're remaining stable. 05:43.880 --> 05:48.320 And so that's something to take note of. 05:48.320 --> 05:53.720 So the veterans and the experts are remaining stable right now. 05:53.720 --> 06:00.320 If you look at a number of countries in Asia without picking any particular 06:00.360 --> 06:05.760 country, we have seen in global news a kind of trend where when birth rates fall, 06:05.760 --> 06:09.080 you get an inverted demographic pyramid. 06:09.080 --> 06:13.520 And so from the perspective of today, what we're reporting is perhaps not 06:13.520 --> 06:17.920 even that big of a deal, because if you're level two and level three folks 06:17.920 --> 06:22.080 are pretty much stable and still doing their thing, we have sustainability 06:22.080 --> 06:23.120 issues with open source. 06:23.120 --> 06:28.320 As previous speakers have addressed here, but we also know that we've got an 06:28.360 --> 06:30.680 inverted demographic pyramid that's coming. 06:30.680 --> 06:35.160 So from my perspective in the Grafana community, like I could probably ignore 06:35.160 --> 06:41.000 this problem for a couple of quarters, but we know for sure that it's coming 06:41.000 --> 06:41.680 here. 06:41.680 --> 06:46.400 So if you don't ask questions publicly anymore, then you lose opportunities 06:46.400 --> 06:51.160 for accidental mentorship and the social glues starts to break down. 06:51.160 --> 06:55.080 Now we've heard a lot of talks in this room today about concerns like 06:55.080 --> 06:58.760 sustainability and maintain our burnout. 06:58.760 --> 07:02.360 And while we need to do our best to address those issues, there's also natural 07:02.360 --> 07:03.360 attrition. 07:03.360 --> 07:07.560 People come and go in any kind of community and there needs to be replacement 07:07.560 --> 07:10.680 birth rate in effect, right? 07:10.680 --> 07:14.200 So we used to run programs that were like simple welcome. 07:14.200 --> 07:19.000 If you sign up for our, if you sign up for our slack, we were putting a lot of effort 07:19.000 --> 07:23.320 into welcoming you and having you talk to an actual human being when you first show 07:23.320 --> 07:24.320 up. 07:24.320 --> 07:28.920 But such a program, even if it's really well executed, can't do well if the number 07:28.920 --> 07:31.960 of new joiners is going to decline over time. 07:31.960 --> 07:37.000 And the key thing about the inverted demographic pyramid is that your top contributors 07:37.000 --> 07:46.600 and your maintainers, they don't appear magically without earlier social investment. 07:46.600 --> 07:52.280 So this doesn't apply equally to every single community out there. 07:52.280 --> 07:57.080 Back overflow was one of the most heavily hit because if you think about it, it's extremely 07:57.080 --> 08:00.040 QA heavy and it's extremely transactional. 08:00.040 --> 08:06.360 They were uniquely positioned to get messed up the worst by the kind of trends that we're 08:06.360 --> 08:08.240 talking about. 08:08.240 --> 08:12.280 There are resilience factors out there, which is the extent to which you orient to 08:12.280 --> 08:15.400 community around relationships and narratives. 08:15.400 --> 08:18.960 And so some of these factors that are here on the slide are going to start to tell the 08:18.960 --> 08:24.320 difference between not only communities that are going to survive long-term, but also 08:24.320 --> 08:29.200 the ones that are going to be less affected by what we're talking about in this talk. 08:29.200 --> 08:33.840 We'll return to this point in a little bit, but a key moment is basically when you can 08:33.840 --> 08:36.400 get a community member to prove effort. 08:36.400 --> 08:43.280 You see like asking a question is an extractive move and basically chatGPT has intermediated 08:43.280 --> 08:47.560 all of that, the thing that I particularly wonder about most of the people online today 08:47.560 --> 08:52.800 is who they didn't beat today because they were chatting to a robot all day. 08:52.800 --> 08:58.120 If you can move from getting people to do the extractive move as their first touch with 08:58.120 --> 09:06.480 the community to something contributive, that's going to be a very strong resilience factor. 09:06.480 --> 09:12.640 And we have a few more ideas on what you can do. 09:12.640 --> 09:19.280 Number one, I like to think about when I first joined working with communities and somebody 09:19.280 --> 09:24.880 told me the best way to destroy a community is to market and sell them. 09:24.880 --> 09:30.200 I think that counts extra for developer and engineering communities. 09:30.200 --> 09:36.560 And so there is this focus that's really important on authenticity. 09:36.560 --> 09:42.040 So throughout that slide, you're going to hear me say authenticity a lot. 09:42.040 --> 09:49.040 So focusing where humans matter, learning in public, giving opportunities for product 09:49.040 --> 09:56.320 feedback, and then building that mid funnel infrastructure, so creating mentorship cohorts, 09:56.320 --> 10:01.520 office hours, contribution ladders, and structured pathways. 10:01.520 --> 10:06.480 And then with three and four here, I think there's a really important acknowledgement of 10:06.480 --> 10:11.320 this moment, the fluidity of the transition. 10:11.320 --> 10:16.920 So you have the opportunity to establish AI governance early. 10:16.920 --> 10:20.800 So where is AI allowed in your community? 10:20.800 --> 10:27.920 Not swimming against the current because people are going to use AI instead of formal methods 10:27.920 --> 10:30.800 prior to this moment. 10:30.800 --> 10:38.800 But then also defining where that human experience is necessary and required. 10:38.800 --> 10:45.800 And then experiment intentionally with log in Q&A's, people's scientific badges, AI 10:45.800 --> 10:50.880 labels, and that intentionally is very important, right? 10:50.880 --> 10:56.920 Giving the community opportunities to communicate with you and learn from the ways in which 10:56.920 --> 11:02.800 they're acting, given this kind of technological moment. 11:02.800 --> 11:10.960 Okay, we've talked about the big problem. 11:10.960 --> 11:15.080 Folks in this room should want to hear about solutions and we've got a lot of them, much 11:15.080 --> 11:18.320 more than we can fit inside of this talk. 11:18.320 --> 11:22.160 It's difficult to just give you the easy three, do these three things and you're going 11:22.160 --> 11:27.400 to be great because a lot of this has to be tailored to your community context. 11:27.400 --> 11:31.040 If there's only one thing that you would take away from this, it's focused on the proof 11:31.040 --> 11:33.440 of effort moment. 11:33.440 --> 11:39.600 Now overall, I sort of feel like AI in regard to open source communities was a little bit 11:39.600 --> 11:42.800 of a Trojan horse that we rolled in through the front gate. 11:42.800 --> 11:47.680 Now a lot of us are normally tech people and so we're predisposed to be very positive about 11:47.680 --> 11:53.000 tech and it does have wonderful effects if you're coding and doing any number of other 11:53.000 --> 11:54.000 things. 11:54.000 --> 11:59.120 But on the social basis, there's no way to get around the fact that the machine is getting 11:59.120 --> 12:04.800 between two people and chopping off a relationship that could have otherwise existed or 12:04.800 --> 12:09.640 have been formed and that's kind of the Trojan's coming out of the horse, so to speak. 12:09.640 --> 12:14.400 But because we're in the tech industry, we're probably not going to go the lotite approach. 12:14.400 --> 12:18.880 I heard letters and referred, you know, mentioned in one of the earlier sessions, read 12:18.880 --> 12:21.640 the history on that and how it worked out for them. 12:21.640 --> 12:25.280 So basically, the horse is inside the gate, we're going to have to actually deal with 12:25.280 --> 12:26.280 this now. 12:26.280 --> 12:33.000 So the best frame of reference is to basically think about how you can take advantage of 12:33.000 --> 12:37.320 the key opportunities that this provides while rolling with it rather than trying to fight 12:37.320 --> 12:38.600 against the wave. 12:38.600 --> 12:44.440 So as one small example, because AI is invading everybody's community, you also have an 12:44.440 --> 12:48.760 automatic topic that's super, super hot for everybody in your community. 12:48.760 --> 12:55.360 Like within our context, we have an MCP server and we just kind of notice that whenever we talk 12:55.360 --> 12:59.480 about or do anything related to the MCP server, everybody comes out of the woodwork. 12:59.480 --> 13:02.240 Everybody wants to know about the MCP server. 13:02.240 --> 13:07.520 So ironically, since humans can connect about anything, you can connect about the AI instead 13:07.520 --> 13:10.560 of having it just get in the middle of you all the time. 13:10.560 --> 13:14.600 That's the kind of example of lateral thinking that communities need to be doing. 13:14.600 --> 13:19.360 So the last point that I want to make and then we'll open it up for Q&A and thank you for 13:19.360 --> 13:24.200 your attention is the wonderful devrim organizers had me here speaking last year and I got 13:24.240 --> 13:25.200 a cop to something. 13:25.200 --> 13:29.760 So last year I came and I did this talk and it was about appropriate use of LLMs and 13:29.760 --> 13:31.160 communities. 13:31.160 --> 13:35.800 And the key thing that I was trying to say last year was you've got to watch out and make 13:35.800 --> 13:39.080 sure that the robot doesn't get in between community members. 13:39.080 --> 13:41.960 You've got to make sure that it doesn't intermediate the relationship. 13:41.960 --> 13:46.160 Yeah, you can add it into an opt-in channel and people can know that they're talking 13:46.160 --> 13:50.920 to an LLM and they can choose to do that, but don't auto respond to your community members 13:50.920 --> 13:51.920 with robots. 13:51.920 --> 13:57.640 So we knew the principle, we didn't want AI to get in between human relationships and 13:57.640 --> 14:03.840 then it happened this year despite the fact that we knew that that was going to be a bad 14:03.840 --> 14:04.840 thing. 14:04.840 --> 14:10.720 So last year I was like, hey, you know, you can use LLMs to automate certain things and 14:10.720 --> 14:15.880 make things much easier and while maintaining respect and human connection. 14:15.880 --> 14:19.040 And well, the first part of that talk was right. 14:19.040 --> 14:23.280 The second part of the talk got taken away from me by other factors. 14:23.280 --> 14:27.080 And so now we're going to have to actually cope with that. 14:27.080 --> 14:31.560 And I really appreciate your attention and we would love any kind of questions that you 14:31.560 --> 14:32.560 have. 14:32.560 --> 14:34.600 I came to Faustem in person. 14:34.600 --> 14:40.000 I chose to be here and the first thing that we do about this kind of thing is that 14:40.000 --> 14:42.160 we actually connect with each other in person. 14:42.160 --> 14:48.160 Oh, I was just going to add, I wonder what we'll have to cop to next year. 14:48.160 --> 14:54.040 Do you need a term? 14:54.040 --> 15:03.760 Thank you very much. 15:03.760 --> 15:04.760 Questions? 15:04.760 --> 15:07.960 Do we have time for questions? 15:07.960 --> 15:08.960 Oh. 15:08.960 --> 15:09.960 Anybody have a question? 15:09.960 --> 15:10.960 You got a question? 15:10.960 --> 15:11.960 Okay. 15:11.960 --> 15:16.800 Yeah, thank you for the talk. 15:16.800 --> 15:22.080 You mentioned earlier that the worst way to run a community is to market and sell them. 15:22.080 --> 15:25.240 Could you go a bit deeper into what's specifically? 15:25.240 --> 15:41.640 Yeah, I think that just means using, taking advantage of that organic interaction to sell a 15:41.640 --> 15:42.640 product. 15:42.720 --> 15:46.960 I mean, we see it and it registers a lot of the time that's spam. 15:46.960 --> 15:53.960 There are organizations where the sole purpose of creating places for people to connect 15:53.960 --> 15:58.960 is to eventually sell them, is like giving information to sales. 15:58.960 --> 16:04.760 And I can speak from our experience, like we try to be punk rock as hell about this. 16:04.760 --> 16:13.440 And have a certain level of protection for the sake of collaboration and community. 16:13.440 --> 16:19.760 So that's kind of what I'm noting about marketing and sales. 16:19.760 --> 16:22.440 Yeah. 16:22.440 --> 16:26.920 So in any human relationship, the connection is fundamentally about the connection. 16:26.920 --> 16:31.120 And the problem with sales and marketing is that it makes the connection about something else. 16:31.160 --> 16:35.080 So imagine going out to your growing out with a friend for beers. 16:35.080 --> 16:38.480 And the first thing the friend says to you is, really glad that you came out tonight. 16:38.480 --> 16:42.160 It gives me the opportunity to tell you about a revolutionary new technology. 16:42.160 --> 16:43.160 You know? 16:43.160 --> 16:44.160 All right. 16:44.160 --> 16:45.160 Why is that Iki? 16:45.160 --> 16:48.480 That's Iki because the connection is about the connection and he's making it about 16:48.480 --> 16:52.480 something else. 16:52.480 --> 16:54.480 Any other questions? 16:55.480 --> 16:56.160 Yeah. 16:56.160 --> 16:58.160 Do you want to do this? 17:02.960 --> 17:05.880 I have a good very nasty question. 17:05.880 --> 17:13.000 I think you are missing one slide from your presentation to present your new products that you have. 17:13.000 --> 17:19.600 Maybe mapping the existing and extinguishing current products would show something else. 17:19.600 --> 17:21.840 Did you analyze this kind of things? 17:25.480 --> 17:31.480 I am not sure I understood the question to be a perfectly honest. 17:31.480 --> 17:37.480 I think if you're asking us to sell, I politely decline. 17:37.480 --> 17:44.480 Would you like to clarify? 17:44.480 --> 17:51.480 Because I've been using graphana for years and recently I've tried to use some new products. 17:51.480 --> 17:58.480 Unfortunately I have problems with trying to jump to them because they were completely different. 17:58.480 --> 18:04.480 Others were expecting to and some of them were removed deprecated and so on. 18:04.480 --> 18:12.480 So I was wondering if you have some kind of information about how people leave the community regarding that. 18:12.480 --> 18:14.480 So you got it now, right? 18:14.480 --> 18:19.480 So I don't need to run again. 18:20.480 --> 18:25.480 Well, so the issue is that I think without the mic, I'm not playing to the live stream. 18:25.480 --> 18:30.480 I can talk loud enough so that you can hear me. 18:30.480 --> 18:36.480 The reasons why people leave communities are shattered into a lot of different reasons. 18:36.480 --> 18:40.480 Some people have dissatisfactions with any product in any community. 18:40.480 --> 18:48.480 Some people like in some of the previous presentations have excessive expectations that can't be met by anybody. 18:48.480 --> 18:51.480 Some people have disagreements on any number of different things. 18:51.480 --> 18:55.480 I personally do not have broken down data on why people leave the community. 18:55.480 --> 19:00.480 The issue there is that when people leave a community, they leave. 19:00.480 --> 19:04.480 And you can't gather data because they're already gone. 19:04.480 --> 19:13.480 What we can do and what we do do is like observability surveys and community surveys to try to get a sense of how people are feeling before they go. 19:13.480 --> 19:16.480 When they go, it's probably too late to know why. 19:16.480 --> 19:23.480 At scale anyway. 19:23.480 --> 19:40.480 No, I haven't seen any ranting on the forum, but if you know of some and you want to bring it to my attention, I would be happy to take a look. 19:40.480 --> 19:43.480 Was there a question in here? 19:43.480 --> 19:45.480 Okay, maybe not. 19:45.480 --> 19:51.480 Oh, in the back, sorry. 19:51.480 --> 19:53.480 Thank you so much for your talk. 19:53.480 --> 19:57.480 I had a question about the different phases you talked about. 19:57.480 --> 20:06.480 So you talked about phase zero, which is people interacting with a community to get some expertise basically. 20:06.480 --> 20:14.480 We try to ask questions to understand the product and eventually they end up helping in phase two. 20:14.480 --> 20:21.480 Do you feel like there might be an opportunity where people jump straight to phase two with LLMs? 20:21.480 --> 20:25.480 As in, thanks to an LLM, I can contribute much quicker. 20:25.480 --> 20:27.480 I don't need to ask questions. 20:27.480 --> 20:31.480 I can just fix my problem myself. 20:31.480 --> 20:36.480 And then that's my entry into talking to humans. 20:36.480 --> 20:41.480 I'm going to deliver this to Amanda to answer. 20:41.480 --> 20:44.480 I feel like we're really using a lot of the room. 20:44.480 --> 20:49.480 Yeah, I mean, I think it can escalate faster. 20:49.480 --> 20:55.480 It's a very individual experience, right? 20:55.480 --> 21:00.480 In short, yeah, I think you can go through kind of that funnel quicker. 21:00.480 --> 21:11.480 The thing about phase zero is in a lot of ways you can't skip it because it's just the majority of people are going to have that like initial touch point. 21:11.480 --> 21:17.480 And then inevitably it drops, you know. 21:17.480 --> 21:24.480 But if they're persistent and want to contribute more, it does make some of the process smoother. 21:24.480 --> 21:31.480 I'll be the runner. 21:31.480 --> 21:42.480 We're on our way. 21:42.480 --> 21:44.480 Thanks. 21:44.480 --> 21:47.480 Okay, I'm not going to. 21:47.480 --> 21:52.480 If AI is taking over phase zero and one, 21:52.480 --> 21:59.480 could we integrate AI's into chat rooms and discussions and like where other people are. 21:59.480 --> 22:03.480 So that they happen despite AI being there. 22:03.480 --> 22:06.480 I mean, it's not going to solve the whole problem because. 22:06.480 --> 22:11.480 But we're still going to go to chat GPT to other private things, 22:11.480 --> 22:17.480 but we could have AI inside some room at least. 22:17.480 --> 22:22.480 If you want to add to this, you can, but in our particular community case, AI is already in the chat room. 22:22.480 --> 22:30.480 What we did in part from last year's talk is that we put AI in as an opt-in kind of a thing. 22:30.480 --> 22:35.480 So you can go to a particular Slack channel called Ask AI or a particular discourse category. 22:35.480 --> 22:37.480 So that it's explicit. 22:37.480 --> 22:38.480 There's no humans here. 22:38.480 --> 22:39.480 We're talking to robots. 22:39.480 --> 22:41.480 And anybody can use that. 22:41.480 --> 22:45.480 And then, of course, other people can see what people are asking and what the answer was. 22:45.480 --> 22:48.480 And whether it was you rated useful and so on and so forth. 22:48.480 --> 22:50.480 It's in the website. 22:50.480 --> 22:51.480 You know, it's in the Slack. 22:51.480 --> 22:53.480 It's in the discourse. 22:53.480 --> 22:56.480 Amanda and I, when we were talking and preparing for this talk, 22:56.480 --> 23:00.480 we're like, did we maybe a little bit do this to ourselves? 23:00.480 --> 23:03.480 Because we're talking about, for example, chat GPT, 23:03.480 --> 23:06.480 making it so people don't show up and ask questions and like, 23:06.480 --> 23:09.480 didn't David kind of launch that in like two places last year? 23:09.480 --> 23:11.480 Yeah, he did. 23:12.480 --> 23:17.480 Yeah, I think in short, yes, do those things. 23:17.480 --> 23:28.480 But I think it's also fair to acknowledge that you don't necessarily always have to recreate the wheel. 23:28.480 --> 23:33.480 Like, there's a piece of this that is swimming with a current instead of against it. 23:33.480 --> 23:40.480 And a level of it where we do have to accept that people are going to use. 23:40.480 --> 23:43.480 Things like Gemini and chat GPT. 23:43.480 --> 23:46.480 So while you should make things as available as possible, 23:46.480 --> 23:49.480 an efficient as possible and embrace AI. 23:49.480 --> 23:56.480 A part of that embracing AI is acknowledging what people are using and where they're meeting. 23:56.480 --> 24:02.480 One other thing to add here is in the ideation phase and thinking through this problem. 24:02.480 --> 24:06.480 There's a, there's a very vengeful way of thinking about this problem. 24:06.480 --> 24:08.480 We're saying, ah, stupid AI company.