The tooling and how to use them, and that might be overwhelming for users and users and people
new to that stuff, Ryan will talk about how that might be made easier for the community.
Thank you.
Yeah, there's been a lot of nutrition, it feels like doing the presentation in the morning
is the right one to do, but there's the recordings I suppose as well.
So you've heard a lot about the variety of tools today, I think there was.
Aki Dr. EOS, they're one by TNO, SM, as pipes that are very good, there's variety of tools
that you've heard today, and maybe about five or six different tools, and they probably
constitute on the order of about 200 open-source energy system modeling tools, being really,
really specific here, these are tools for infrastructure planning and operation, open-source
that we know about.
And it means that it's a real big problem to keep track of the tools that are available,
unless you come to every single possible conference for people that are talking about these
things, and maybe you'll pick up one or two each time, or you read review papers, because
actually things often come from the academic literature, or academic community, these
things often developed in academia.
And so you could go back, you could read a review, pretty much every single year, there
will be a review about the latest developments in open-source, or just general energy system
modeling, maybe even in 2022, there was about five or six reviews that came out in the same
journal, that we're all about, what are the latest developments in energy system modeling
in the tools available, and it's effectively a losing battle, because you read these,
you might get the latest developments in the tools that they know are available, they might
grab about 100 maximum of the 200 of there, and they are also probably about two or three
years old by the time you read this paper, right?
So they've started their reviews a PhD student probably, they've taken some time doing
the review, they actually literally, the academic review process, the peer review process,
and then you read it a few years later, and inevitably they go a lot of things wrong.
So I've been a lead developer of one of these 200 tools for about nine years, and I can
tell you, every time I read one of these, they've always got something wrong about what
my tool can do, because they are reading it from an external perspective, and maybe that's
a sign of me and my documentation, but it's true for a lot of the other tools as well
that I know about.
So this is where it comes in a project that we've been undertaking at the Open Energy
Transition, with support from breakthrough energy, grids initiative, to map all of these
tools that are available in a fashion that will stay up to date continuously.
So what you see here is a snapshot of the latest update, it's a monthly update we get,
so this is the January update in terms of a snapshot of the top sort of 15 or so tools of
these 210, and we've got in the inventory.
I can show you, I hope, what that looks like in a bit more detail here, so this is, this
is, it live at OpenMod-Tracker.org, and we've got a couple of caveats at the start, so
I want to be very clear, we believe we're catching most of the open source tools that
are out there, we've probably aren't catching them all, and don't blame us for the data
associated with your tool, right, that's what I'll tell you in a minute, we get most of
this data from elsewhere.
So you've got the table here, I've got a couple of different pieces of information that
might be useful to you, when you come and look at it, you've got the link to the source
code, so all of these have to have openly available, publicly available, source code,
you can get a link to the documentation, and we have a link to all but 60 or so tools documentation.
So this is a really good resource, if you just want to find out how to use a tool, I think
we do a better job than the readmees of most of the repositories of telling you where
their documentation is.
So check that out, and then a variety of metrics associated with the source code repositories
for these tools, so things like when was it created, when was it most recently updated,
how many stars does it have, this is ordered by stars at the moment.
Number of contributors and the development distribution score, I believe was developed in
open sustained tech, and both of those really are sort of a bus factor, it's trying
to tell you, if there's one or two developers and a really, really low DDS, it's telling
you that if they go out of action, they choose to leave the industry, that entire project
will probably die with them.
So you want lots of contributors, you want a high DDS, you want a lot of force, you want
people interacting with it in lots of different ways, and you want to know how often
it's downloaded, and that's a difficult one because we can only get that for verifiable
downloads, things that are indexed on package indexes, usually price and packages, is what
that's possible for, and a couple of other data sets as well.
You can do things like filter, you can say, well, I only want to see ones that have documentation,
maybe ones that are at least five or six years old, I want to have a bit of pedigree,
maybe you want ones that have a certain number of stars associated with them, so you can
do a variety of filtering actions to take you through and really filter down on the tools
that might be of interest to you for whatever project you might be doing, whether that's
as a user, a potential user for a tool, or a potential developer, some of the joining
of the development community for a tool, there's a lot of possibilities there.
You can also deep dive into a few other things, so we've tried to capture all of the users
of these tools across GitHub and GitLab, if they're on those ones, and try to classify
them to get an impression of the types of users of these tools, the people are interacting
with these repositories.
We don't do that in quite a job, right?
So about 65% of the users on GitHub who interact with these tools, of the 50,000 or so
that we're picking up at 16,000 now, we don't get about 65% of them, that's because it's
just that GitHub username and no other information, and we can't help to do any better
than that.
But of those who do interact, we get a lot of people from the academic community, small
for you, if you go through the actual thing, you can have a look at any more detail, academic
community, research organisations, professionals, and industry, they're all interacting
in various ways.
We can look at which organisations they are, NREL now, the National Lab of the Rockies, I think,
is a big one.
We open energy transition and also a big one, a variety of other ones, so you've got things
like Chinese universities or US universities.
You can look at this on a two levels, that was for all tools, I'm just going to look at
one of them here, maybe very good isn't here, that would get very good.
So this is really picking on you, but you come to mind, and we can see various organisations
who've got involved, various countries where the users get involved, so you can see the
international sort of community that might be involved with it.
You can maybe look at another one on side, we have pipes afterwards, so we can add
that in, we can see across those two tools, the global reach in terms of the kinds of users
are getting involved, at least the ones we know about, that 35%.
And we can look at the project development metrics, so we can say, okay, for these
projects, some information about their development cadence, how are they, are they waxing?
Are they waning in terms of their level of development, pull requests, issues being opened,
who are the top contributors, across these, this is the top contributions across all of
the projects, who are the top organisations, and in what way do they contribute?
Do they contribute in terms of opening issues, do they contribute in terms of reviewing
or giving feedback, and what is the type of feedback we're getting?
So times closed issues, time to merge, pull requests, all these information for understanding
how well a project is running, and how much engagement is happening during resolution,
is there good feedback on reviews?
Can you trust the review process, is there enough feedback on it from that perspective?
So all of this is possible from this dashboard that we've created in order to help people
understand the future of energy system modeling or where they might want to go.
So what we do is we have to kind of herd cats to achieve this, we go to a variety of upstream
inventories of these kinds of tools, and we grab them all in order to pull together all
of the tools that we bring.
So we know of these, if you know of more, really happy to know about them, we want to
pull in as many as we can, and we do some triage kind of, we remove duplicates, we filter
things out, exclude some tools that we just know for sure, repositories that we know
for sure are not energy system modeling tools, but they've made their way through to these
upstream inventories, and they also have to have open and public source code repositories
and public licenses, like permissive or copy left licenses, usually.
We tend to find that actually our just getting stuff from academic reviews and building
a manual data set as the best of all of them in terms of unique tools that you don't find
anywhere else, and some inventories don't provide any new tools to the mix, so they have
tools that they've mentioned elsewhere as well, so they don't actually add any new information
for us.
We bring this together with the ecosystem's API, and I'm sure Andrew, and this bit, I think
it is, well, it's probably somewhere, at Fast Stand this weekend, and you should really
find them if you can, because ecosystems are great API, to take you to make you something
with your all of our metrics, and there you can also blame them when the information looks
weird, because we aren't really, we're not doctoring that information, we take it and we just
show it as it's, we also do some scraping to try and find the doctor's pages in a bit more
detail, because we don't get that well from elsewhere. That gives us some stats, and then we go
directly to some diary APIs for GitHub and GitHub, and that gives us the information about our
users and about the repository's interactions over time. This is all run in about an hour and
20 minutes in a GitHub CI, the time is mostly in this GitHub and GitLab API requests, but we run
that just automatically, it's sort of set and forget now that it should happen monthly,
I say that, but the last one failed and I had to fix some bugs, but in theory it should run
set and forget, so instead of you having to wait for the next review next year on the tools that
are available that comes out of academia, you can check this dashboard instead.
So why might you use it? Who's going to it? And we are the open energy transition, we've got
sort of a mission a vision to make open source the norm in energy system planning by 2028. That
means going to energy system operators, transmission system operators, and working with them
to move them away from proprietary tools, using open source tools for their energy planning.
That's our entire goal. And that means helping them understand what are the open source
tools that are available and which ones might be most appropriate to them for a given use case.
And we do this by helping them with the tracker here to produce a long list of possible tools
for them and then we deep dive, we let them deep dive if they want to, into more information
about those tools. You can't hope to look at 200 tools, but you might be able to look at
10 or so. And that helps you create a short list. So with one client recently, back in August,
we started some things, we just had some really broad brush things that has to be documentation.
You know, it has to be easy for them if we hand over the tool to the model to them,
they have to have documentation, they can't rely on us, they have to rely on the tool itself.
It should have an index package. The maintainers should care enough about being easy to download
cross-platform. And usually that means having a downloadable index package and so some monthly
download stats that we can grab. When you have to compile things and you are talking to somebody
who's limited them from a technical knowledge, it's usually Excel on Windows, you have to work
with that, not against it. And things like permissive source code licenses so that if they
internally have a tech team that wants to make some updates, they're not going to be limited by that.
In this case, we also included copyreft. We have one column that is manually prepared because we
got some information on the top 30 or so tools about what kind of modeling can you do. So you can
get some really high-level information about the kind of energy that's just modeling you can do
and you can use that for filtering as well. I should say, so Pythso came up in the earlier
conversation, they do rank the highest on a lot of things production cost modeling. When I mentioned
that I'm in lead down to the tool for the last nine years, it's not Pythso. So it's always
quite galling to talk to clients and be like, actually the tool that's looking best here for you
is Pythso. I kind of just have to say that with a straight face and let it lie. There are other tools
that do quite well in this specific filtering for this specific client. And yes, they do well across
these but can you choose what waiting to give them? Because some might matter a lot more to them
than others and that it's very difficult to put waiting to forks versus DDS. So we went in and we did
a much more detailed analysis and these are the kinds of things I'd love to automate in the dashboard
and if you've got ideas of how to do it, I'd love to find out test coverage. I'd like to find
a way to automate accessing test coverage information because people publish their test coverage
or store it in different ways depending on the language that things written in and other decisions.
The quality of their documentation. I might get to a documentation it's a single page,
this is how you install a good luck. So it looks like they've got documentation but actually they
don't. So that requires some manual stuff. Where can you get community supports? Where can you get
data? We've had a lot of our energy modeling tools and the fact that you need reference data.
If you can get a tool but you don't get any data with it when you wait to access the data
or link it to that tool, that tool is functionally useless to a lot of energy system operators.
So reference databases for tools, other tools that can link into it really easily seamlessly
is really important, etc. So we can go through this process of shortlisting through this and we
did this with them to shortlist to about three or four and that was really good and they found
it really useful to have this sort of verifiable way to how do you shortlist tools so that they
can then go back to their higher ups and say this is why we want to go open source and this is
how we're going to do it and this is why we're choosing this tool. But we need to dig deeper because
we just heard earlier about pipes so it's stochastic optimization. If you asked about pipes
and you want to stochastic optimization six months ago the answer would have been it's not
the tool for you but you wouldn't know that from any of these metrics, right? So we need to dig
deeper into this and look at other features and we went with a proprietary tool feature set
I won't tell you which one because I don't want to get into legal ramifications of it and
looked at what are the features that proprietary open, proprietary energy system modeling tools have?
Theme to them, don't have to remember all this now it's all available in the GitHub repository
grouped them and then created a variety of features. So basically a feature taxonomy of energy
system modeling tools but at one that isn't too technical it needs to be really understandable to
potential users who as I said aren't super technical they just they know they want to achieve
a certain task but they wouldn't necessarily know the technical words what that might be.
That brought us on to a new dashboard dashboard you will probably tell if you go to any of these
are not a front-end developer and actually I've seen a lot of stream that apps today in this room
so I'm guessing that most of the stream are not front-end developers but it does the job and these
are all prototypes and aim to be production ready and this is very similar to actually going to be
an extension of the other page in which it's own page that you can go in you've got your tools here
and then you've got specifically what are the features they've got grouped and most importantly
on this is what you're not going to get that easily in any of these review papers there are
effications of that feature existing all of these little superscript are a link to a page in
their documentation or their source code their tests that proves that particular feature exists
but also help you understand how you activate that feature in that tool if you want to and whether
something's in development rather than so this one here whether it's in development rather than
actually ready. You can use it to also filter it based on the use case you've got and I'm not going
to go into this detail because clearly I won't have enough time to do so but you could say okay
I'm running an integrated resource plan which is usually what is done in the US bit in particular
to make high-level decisions on the future asset capacities that you want and we've gone through
and relatively with some degree of subjectivity we've gone through and said what are the features
needed by this and this is a filter view of those features and then you can look at your tools and
say okay which ones meet those features the best. You can do that side by side specifically on
use cases so you could look at all of these and you could say okay look at it for pipes
uh how is pipes are doing for all of these uh next which other so you look for a single tool
but you look across all the use cases and finally you could look and you could write your own
use cases so you could say okay here's all of the options uh give you a little of a tool
tip as to what we mean by it uh click all of the things you're interested in uh save it I'll just
click a couple more uh save it and then you can go back um that's saved I think uh and then I can
go back to the use case comparison and I can look at it in that context uh or I can look at
the tools comparison I can look at it in uh in that context as well so I can look at my
custom use case and look at it in that context so it's a way for you to be able to and I can see here
you I actually here I can't scroll down so it's clearly a problem there um but it's a way for you
to be able to um discuss with others what are the features we need for our tool and therefore
what are the tools that are most appropriate this is a manual task right it requires the developers
usually themselves to contribute to this so that's why you only see four tools and hoping to see tomorrow
very good on here right yeah uh as soon we'll be here shortly afterwards and then uh I'm sure
that uh gems pie will be here soon too and the value of adding your tool to this is the fact that
then it will be used in the decision making process by energy system operators because we will
be showing them this dashboard right we won't be filtering it for pipes so that's uh it will be
showing it to them vanilla so get it up there as quick as you can
this is a little bit uh what it looks like inside the the feature set so you've got just a simple
animal config um we we took sort of some inspiration from con de forge as to uh feedstock maintainers
so you've got maintainers of these lists they don't have to be the developers of the tool
but you've got maintainers of the list and they will be always tagged if there's some need for them
to do anything uh we've got version schema for all of these so you could have different features
uh feature lists that are tied to different version schemas um and uh this is where i said you've
got these sort of source sources uh to the specific use cases and those will be checked in review
and then similarly for use cases uh the the only addition is that we've got a list of the assumptions
that go into that subjective use case list uh so where are we going uh we want community engagement
that's why i'm here today uh and we want to improve UI and UX we'll probably get a front
end developer to actually look at this and and hopefully don't scream uh we want to add new insights
we want to add things like what are the academic and ideally non academic uses uh for these tools
we're going to rely quite a lot on ecosystems for that and open Alex uh we want longer term download
stats in the last 12 months uh pie pie and anaconda uh and Julia uh there were 3 million downloads
that weren't CI related downloads of all of these 200 tools so last year 3 million downloads across
all of those tools there's not capturing all of them because we're only capturing really Python uh so
we're missing quite a lot to want to get more analysis on that and we're going to rely on ecosystems as
well for um security analyses of dependencies because energy system operators want to be uh
certain about the security of these tools that they're using uh we're going to be doing that variety
ways over the next sort of six months or so so really if you want to get involved get involved soon
and looking at UI improvements we really want to refine this taxonomy make sure it's useful
to everyone useful to developers so they understand what they're missing and useful to um
useful to users so they understand what they need and what they can get from the tools that they're using
just as a sort of a case in point of of how we use this uh so the this this sort of feature gap
analysis that we've got uh here with the we've worked with the pipes a team to look at all
of these things that are missing created a a roadmap on GitHub with issues which if all closed
will close this gap between pipes and uh probably the leading proprietary tool for energy
system modeling so you can go through this process by doing this yourself you can say well what
am I missing was it's what proprietary tools can do and create a roadmap for yourselves uh to be
able to close that great so that is me done these and the two um dashboards but also the
two GitHub repositories associated with them and as I say this was done and with a lot of
ultimately financial but also um inspiration support from breakthrough energy uh that's what she's
this project thank you very much
team and it's right rapid fire
uh there will be tools that do lower medium voltage yes definitely uh but these these are generic
tools right so for the most part low medium voltage is a data problem not a tool problem really
so it's more about how the modeling is done in terms of the tools so yes there are several that
can do low and medium voltage that was the question yep yep yep yep yep we know the eo open source
solutions can unlock and it could have been some integration that you make sure the tool is
findable and it ends up that you'll and kind of log in. so do I know that eo opens or solution
cataloging could there be some Synities there uh I do know of it um this is obviously global
It's not just use specific in terms of its use cases, but yeah, we would want to
Link to it, we should probably actually think about as an upstream inventory more than anything else so that we could potentially get more things from it
That are index on it, but not in the other ones we've got
At the moment we're not thinking of then downstreaming what we've got to other other places
How did you feel throughout the CI related double?
Piper has that automatically in a big query
database, and that's the main one and it kind of doesn't but it's far fewer downloads than Piper
And Julia does it automatically in terms of
removing CI from downloads
Yeah, would this help return on investment building open source?
Yeah, I mean what I'm hoping it does is stop anyone from building a new open source and it's just a modeling tool
Because that is probably the worst result on investment
Yeah, it's already relevant now. We see that, but that might be just because of where we're getting the data from that
We're not capturing the latest the newest ones
We are hoping that people will use this to support the financially support tools based on choosing tools that look like them
Sure enough to be worth supporting and that have strong organizations backing them in terms of working on them
So we're hoping that people will look at this also from a financial backing perspective as well
Not sure how they would work out return on investment
Be able to keep take away is don't start developing a new tool now
Cool, thank you very much