What’s Next for Energy Forecasting? Keeping Pace with a Changing Market

Ivona (00:00)
Hello everyone and a very warm welcome to our Hypercube Energy Forecast. My name is Ivona Voroneckaja, I’m an Engineering Manager, previously Senior Data Scientist at Hypercube Consulting. But today I’m here wearing my forecasting and statistics hat
for this energy forecasting episode. Before I go ahead and introduce our guest on the podcast, who some of you may find a familiar face, I just wanted to tell you a little bit more about myself, my background, and why I love forecasting so much. So I actually come from academia. I’ve spent just over five years at the University of Glasgow, both lecturing and researching statistics and machine learning,
particularly focused on that predictive algorithm development. So for those of you interested, a little snippet. So I developed an automatic architecture selection method for hierarchical mixtures of experts. That included some really cool stuff like adapting Reversible Jump algorithm to handle the notorious low acceptance rates, as well as introducing some new algorithms.
which I called a Gate Swap to make the model even more adaptable. The reason I’m telling you all of this is just to basically have a glimpse and share an insight of like, I really, really love this stuff. Moving on from academia, kind of got into consulting and never really looked back. So I applied predictive modelling across a lot of industries, starting from modelling, the likelihood and probability of somebody defaulting on a
mortgage payment all the way to forecasting short-term PR campaign performances. I really, loved seeing data science in real world and how that made a difference and helped these users of the data make informed decisions. In the past, just over two years, I’ve been more focused in the energy area.
So as an Engineering Manager at Hypercube, I help and contribute to our teams, building world-class forecasting and analytics capabilities for organisations in the energy sector. So, we work on forecasting key indicators such as heat decay, domestic energy consumption, but also work on more complex energy derivatives such as net imbalance volumes, etc.
Overall, our goal is to help our customers navigate energy market complexities and set them up for success, really. So excited, as you can imagine, given my background for this conversation with Duncan. Duncan, would you like to introduce yourself, your role, and tell our listeners a little bit more about your journey into energy?
Duncan Bain (SAS) (02:45)
Absolutely. So I’m super excited to be here returning from the mist of time in podcast number eight. So my name is Duncan Bain. I’m part of the Customer Advisory team here at SAS UK. And I cover primary industries, energy, and manufacturing for Northern Europe. So to give you the kind of idea of the geography.
Ivona (02:50)
Ha ha.
Duncan Bain (SAS) (03:07)
That’s the UK and Ireland and then the Nordic countries. So Sweden, Finland, Norway, Denmark, etc. So that’s what I do now. Previous to this, I was Head of Data and Insight at Scottish Power Energy Retail. And if anyone has seen my LinkedIn profile, you’ll see that the first thing up on that is energy nerd. So I’ve been in the energy industry now for probably almost two decades and prior to that, worked in financial services for a bit. So,
Ivona (03:21)
you
Duncan Bain (SAS) (03:30)
I know a little bit about pensions and investments as well. But I wouldn’t take any investment advice from me based on those criteria.
Ivona (03:38)
You should be careful saying that Duncan. People might just flood to you for expert advice. Amazing, thank you so much for the introduction and as an energy nerd, let’s just dive straight into it. Why do you think energy forecasting is such a critical issue today? Specifically, what’s kind of changed from maybe a market and policy perspective to make it so complex?
Duncan Bain (SAS) (03:47)
Not on that, hopefully.
So a long time ago, when we first envisaged the energy network, we designed it to have big generators in the middle, and then we distributed power out to the edges. And so all you really need to know is how much power was required, kind classical load forecasting. And all of the resources that we’re creating electrons for us to use as consumers, right, were
to quote the old Wallace and Gromit ad for the gas port, right? They’re easily turn off and onable. Okay, so it was relatively compared to now easy to supply against demand. Now, much more recently than that, we’ve started to really try and reverse the flow of electrons through the network. So, you know, all of the assets that we’re building are now at the edge of the network. And we’re pushing all those electrons back from the edge towards the middle.
And the types of assets that we’re now deploying are much less easily turn off and onable because they rely on renewable resources like wind and solar. And so as we see the impact of these technologies deliver more and more of the country and put the globe’s demand for energy, we need much better forecasting to understand.
when we’re going to deliver the output from that installed capacity. And then how we can use that to align additional technologies against that to try and either work to flatten the demand or to store energy for periods when we’re going to generate less. So I think in summary, we’re now in a situation where we’ve created a lot of assets at the network edge that are very different to the assets that we first envisaged when we created the network to begin with.
Not only do we need to do more forecasts, but those forecasts we need to do more often, they’re more complex and they’re against different types of assets.
Ivona (05:54)
Absolutely, and if I can just add on to your last thing that you’ve just said, from sort of data science perspective, energy forecasting is just so fascinating, specifically because of all of these complexities that perhaps cause some of the frustrations on the other side of the coin when it comes to energy forecasting. So we all know there’s like strong seasonality, there’s basic things that you said that we are kind of know how to handle, you know.
In the winter, we heat way more than we do in the summer. Then there’s daily activity cycles. People don’t use as much energy when they’re sleeping. So if we’re just using domestic kind of scenarios as an example here. And that challenge from the modelling perspective comes when we introduce the mix of those stable cycles and very volatile short-term shocks. So things you’ve obviously already mentioned, like weather changes,
renewable intermittency, policy shifts, having, you know, EVs, home solar, all of that comes into play as well. So the forecasting, how it’s done these days to account for all of these complexities, it must balance those long-term patterns and short-term volatility, which makes it a very fascinating and very interesting problem to solve. And then on the other hand, when you forecast, you always kind of get it.
slightly wrong, don’t you? There’s a forecast. So it’s so valuable to look at those errors. So where you went wrong, why you went wrong. So forecasting errors are almost a second stage to modelling what’s happening. So they give us some insights on systematic bias, new trends. And unlike other industries in energy, we absorb such high frequency again,
sort of echoing what you said, such high frequency real time data that we do need those more robust scalable models. So just to kind of continue that discussion, how do you think the growth of renewables and prosumer energy impact forecasting accuracy and methods from your point of view?
Duncan Bain (SAS) (08:00)
The biggest change and kind of what we’ve both kind of talked about, if we tried to sum that up in one word, it would be uncertainty. Right? So, you know, lots of things are happening and we can’t be certain that they will happen. Right? So, I mean, I think you raise a really good point. Forecasts are always wrong. Right? Because as yet we cannot tell the future.
Ivona (08:06)
Yes.
Yes.
Duncan Bain (SAS) (08:22)
Right.
So, you know, the first person to have a forecast as a hundred percent correct, I will be phoning them up and asking them for the winning lottery numbers, because you know, that that’s how difficult it’s get them right. So what we want to do is we want to get the least wrong forecast and we want to be able to, adjust our forecasting methodology based on things that happen as we know them. So, I mean weather is a really good example of this. So,
we’ve just had storm Amy here in the UK. And on the Thursday it was forecast to pretty much come right over the top of my house. So of course on the day before we went out and took trampoline down and did all the things to make sure that all of our garden furniture wasn’t going to end up in next door garden furniture. And then on the day it blew about 50 miles north of us and we just got the kind of edge of it. And so even if you’re pretty sure something’s going to happen,
there’s a good chance it probably won’t. What equally, when it comes to do things like forecasting the weather on a very short term basis, there’s about a 50% chance that the weather tomorrow will be the same as it is today. And so what you’re trying to do is balance all of that uncertainty. Now, so like I said, you know, back in the back in the day when we were largely relying on thermal generation, you could just make it work harder, or make it work less hard. Whereas
when we’re looking at installed capacity in wind and solar, you’ve got issues about whether or not the installed capacity is going to deliver that capacity. And then from a network’s perspective, is it going to deliver it in the right place? Because certainly one of the big challenges we have in the UK is that we’ve built a lot of offshore wind because that’s where it’s windiest.
But the majority of our consumption is not in the North Sea. The majority of our consumption is in the South East of England. So we have to get the power from point A to point B. And so it’s important that we understand not just how much total power we need, but where we’re going to need that power and where it’s going to come from. And then we have to start try matching those forecasts up against each other to say, where are the gaps there? Where are we going to need to potentially
import over the the interconnectors. So it all comes down to uncertainty.
Ivona (10:28)
Yeah, absolutely. Just to touch up on a little bit on what you said, you know the weather tomorrow is very likely to be similar to the weather that we’re having today. Let me just use some examples here just to bring that kind of modelling flavour to the conversation. So take ARIMA, right? That’s your classic stats model that you will probably be the third thing you learn in your forecasting class at the university.
So great for short-term stable patterns, maybe for the next few hours or days. It has a place to be, you know, it’s interpretable, it’s fast, it’s small data friendly, all that kind of thing. And there are extensions to it if you want to get a little bit more complexity out of the model, can do like up to SARIMAX and things like that. However, when we do get all of these…
complex dynamics, nonlinear effects, everything that you’ve been talking about, various levels of uncertainty. We usually turn to slightly newer methods, specifically deep learning. So, if we take an LSTM as an example, that is quite well positioned to handle those complex dynamics, demand shifts, consumer behaviour all the things that we kind of spoken about. It processes data in batches.
So that helps to capture kind of long-term dependencies and allows us to handle the data at bigger volumes. But then we also have our kind of hybrid models like CNN, LSTMs, where CNN takes care of your short-term local pattern extraction. And LSTM will look at that longer term, temporal structure, etc. So, these models then offer you…
handle both noise and cycles and they’re quite popular in the energy sector for a good reason. They’re powerful in energy forecasting where you’ve got all these noisy signals but also some strong long-term cycles within it. Obviously there are trades-offs so if you have loads of data you need computational power. Also depending on how well
your kind of starting point is set. And if you have these like baby algorithms helping you to tune parameters, there may be some tuning needed. Essentially what I’m trying to say is like, there’s no one size fits all model. It depends on the goal and the data that you have. As data scientists for us, it’s so exciting to be able to explore these models of different complexity based on the application.
And I understand that SAS has been working on a solution that allows users to fit those models of specifically varying complexities that I’ve just described. Would you be able to tell us a little bit more about it? It makes me really excited.
Duncan Bain (SAS) (13:07)
Yeah, so you’re absolutely right. There is no magic bullet. And I think along with the fact we’ve got this kind of increased complexity, we’ve got more uncertainty. What we’ve really been focusing on here at SAS is how do we enable forecasters to do more forecasts, do those forecasts more quickly, account for these new artifacts coming into the system.
And how can you do that on a repeatable, reproducible basis and fundamentally for a forecasting team, because we’ve already established all forecasts are wrong, right? They have to be able to defend the decisions that they’re taking based on the forecast they’re generating. So it has to be clearly explainable how you got from point A, which is I have my data through to point B, which is I have my forecast. So, although SAS has been in new energy forecasting for
a long, time, we’re nearly 50 years old now. And I think one of our first customers was in the energy space. So we started off doing energy forecasting probably about 45 years ago. Our newest version of that energy forecasting cloud has been designed from the ground up as a software as a service offering that enables forecasters to take data, run these forecasts, it will run
Ivona (14:10)
Wow.
Duncan Bain (SAS) (14:27)
all sorts of different algorithms and it will choose the best one based on whatever error metrics you choose. However, you don’t have to pick that one because sometimes those models can’t capture things like local knowledge. Sometimes it’s because you forgot to put it in, right? If you’ve got a calendar of events, know, typically I have all the kind of normal holiday things in. But if you look at school holidays, for example, across Scotland,
Ivona (14:41)
Hahaha
Duncan Bain (SAS) (14:50)
lots of different local authorities have their school holidays at different times. And so unless you know that and you’ve captured that, it might be that somebody in your forecasting team says, but this area doesn’t have its holidays then we need to account for Other things like local events or where you’ve got big sporting events, things like that. They probably haven’t happened at exactly the same time in your historic data.
So you need to be able to account for that. And so it enables forecasters to intervene and to plug these events in as custom data items very simply without having to kind of go back to their IT organisation and say, can you update the calendar of events in the data store because we want to add these additional events into it. And so that’s really our focus has been on developing a tool that makes it really simple for forecast teams.
to do more of what they do best.
Ivona (15:42)
Absolutely, that just sounds absolutely great and kind of aligns with what I’ve been sharing before I asked you the question, kind of saying how do you go about trying different things out and still having that input from you as a user if you feel like you want it or just being able to go from A to B if time is of the essence. So, that’s really cool to hear that. I was wondering if you had an example of
you know why that better, more accurate forecasting is actually so important.
Duncan Bain (SAS) (16:13)
Well, ultimately, poor forecasting costs businesses loads of money. And there are are different regulatory settlements across the jurisdictions that I work across and globally. So some organisations are actively penalised by their regulators for having forecasts that don’t match up to reality because it affects the way that they dispatch power across the network. And you
the, if you think about your forecasting across kind of different time horizons, you’ve got the kind of very short term forecasting, you know, which is used for, dispatch and kind of that kind of instantaneous demand response. But you then got kind of medium to longer term forecasts, which are more about planning out the infrastructure that we’re going to need as we go forward into the next kind of 5 or 10 years. And so being able to plug into the system events that haven’t happened yet.
Right, so by using synthetic data, you can say, well, if this was the case, then what would that look like and how do we need to plan for that? So there are different requirements across different time horizons, but fundamentally, better forecasts result in more profitable companies.
Ivona (17:23)
Yeah, absolutely. I just wanted to take a little, like zoom out a little bit, take a step back. And, know, we’ve been talking about picking the right model, having the tools that allow you to test different ones, see which one’s the best. But it’d be nice to also talk about whether that is the first step for me, where you start isn’t kind of picking the model, but really understanding the use case. So,
before I start any forecasting, I want to understand what decision does it support? What’s the value that it adds once produced? What is the tolerable error in actual practical terms? And this will shape the right solution because once that’s clear, we can try and think about how it needs to look like at the end
of it, so it’s actually useful for those decision makers. So communication is key here because forecast is only as useful as the person reading it on the other end and being able to clearly understand it. So, this may mean sometimes using visuals, maybe dashboards, perhaps using statistical tools such as confidence intervals to put your uncertainty into some sort of measurable context.
We had the customer struggling to align requirements of the trading and data science teams for building a new data platform. So in their cases, we actually just run workshops to enable that collaboration and fix misalignment and fix that crucial bit so both teams are actually the users of the final product. Were happy with the end result. So I don’t know if you had any
thoughts on that, the kind of first step evaluation value extracting value from, you know, how do we go from being an Excel spreadsheet with a column of forecasts to extracting value.
Duncan Bain (SAS) (19:15)
Yeah, I mean, I think trust is really important, right? Because, you know, as we said, you’re all forecasts are wrong. So what’s the acceptable level of wrongness and how willing are people to take decisions based on the fact that they know it’s not going to be right first time. Ideally, the closer you get to the point of delivery, you know, those short, really short term forecasts should in theory become more accurate. But
Ivona (19:21)
Right.
Duncan Bain (SAS) (19:38)
because you’re basing them on effects on smaller size, you can actually see error creep into that. So it’s very much about building trust in the process. Right, and part of that is just making it really straightforward to explain how you’re going about doing the forecast and what those results actually mean. And you it kind of reminds me of some of the debates we’ve had previously and kind of in other disciplines around, you know,
What’s worse type one or type two errors? right, so is it better to be under forecast or over forecast? Or are you trying to split the difference between those two things? And that’s gonna depend on largely what’s the financial result of doing that? If you’re trading on it are you gonna have a portfolio that contains too much power?
In which case, what do you do with it? Or are you going to have a portfolio that doesn’t contain enough power? In which case you’re looking at trading at market rates to get that point of delivery, which can be but not always extremely costly. And one of the things that kind of brings us back to uncertainty is if you look at a market like Nordpool, it quite often now trades at negative prices
because of oversupply from renewables. And so being oversupplied in that market is probably less good than being undersupplied. But in other markets, the reverse is true. So it very much depends on, as you say, what is the use case? What are we going to use these forecasts to do? What decisions are we going to take based on these forecasts?
And again, if you’re looking at if you’re looking at longer term forecasts, you really need to have a good idea of what you’re not including in that model. So if you’re if you’re thinking about building out a new network infrastructure, right, if you’re if you’re saying, well, we’re just basing this on our historic data, plus, an idea of what percentage uplift might be in order to say, you know, what’s the increased demand going to be from electric vehicles?
Ivona (21:24)
Absolutely.
Duncan Bain (SAS) (21:42)
then that’s only going to be one part of the picture. If you’ve failed to take into account the fact that we’re also going to see, you know, big growth in home energy storage, you know, that forecast could be quite far off. So you’ve really got to understand what you’re doing it for, who’s going to be using it. And it is absolutely critically right to build trust in the forecasting process. And that’s very much more a kind of human factors.
challenge than it is about the technology itself. So what we’re trying to do is make the technology really easy to use so that people can focus on that value add part, which is really around kind of explaining what’s happening in the forecast, why they are the way they look, understand the decisions that are being taken and supporting that decision making process.
Ivona (22:27)
Absolutely, as you were talking, I wish that I had included directionality of errors as being one of the complications of the modelling process when it comes to the energy industry. So I’m glad it got mentioned in some capacity. Absolutely, agree with everything you’re saying. Listen Duncan, looking 5 to 10 years ahead, what innovations and forecasting do you think will have the biggest impact on the energy transition?
Duncan Bain (SAS) (22:54)
In forecasting specifically, I think in the next kind of 5 to 10 year horizon, we’ll start to see, more use of like really cutting edge techniques like quantum optimisation. So, you know, quantum has been on the horizon for well, since I’ve been in the data game, right? Everyone’s been talking about your quantum, but it’s only relatively recently that we’ve started to see
quantum machines that are available for doing this kind of testing. And so, you know, I think that the first place we’ll see that is in operational research processes where the complexity warrants the use of quantum computing. And I think that one of the next places we’ll see that is as we see more and more complexity and more uncertainty come into forecasting, you we will start to see the potential there.
for quantum techniques, especially where you’ve got relatively small volumes of data to base your prediction on. So one of the things that we’ve shown through our research here at SAS on optimisation problems is that although quantum might not outperform classical techniques in terms of its outputs, it will maintain its accuracy even if you reduce the amount of data that’s available to the model.
And so I definitely think we’ll start to see something in the quantum space coming to energy forecasting, certainly in that kind 5 to 10 year time period.
Ivona (24:17)
That’s exciting Let’s do a quick fire questions if that’s okay with you. Just first thing that comes to your mind. You ready? I’ll try. Forecast the future, what will the energy grid look like in 2030?
Duncan Bain (SAS) (24:19)
Which is exciting, isn’t it?
Okay, easy questions only.
It will look mostly the same as it does today but will have put even more assets at the edge of the network.
Ivona (24:46)
What’s the most surprising forecasting insight you’ve ever come across?
Duncan Bain (SAS) (24:52)
Oh, that’s a good one. I think the most surprising to other people, right? I think is that past performance is not necessarily a guarantee of future performance. Right. And you know, we say that in financial services all the time, but you know, historically, and for a really long time, it was true that past performance was a guarantee of future performance. But you know, over the last kind of 5 or 6 years or so, particularly as we went through,
Ivona (25:01)
you
Duncan Bain (SAS) (25:17)
the kind of the big kind of societal change that COVID brought where, we moved demand around enormously in that period. And, you know, we’re kind of settling into a new normal. So for quite a while, the kind of historic performance really didn’t tell you that much about the future so learning to adapt to that quickly
is where companies had the opportunity to thrive or just to get by. So I think that’s probably the most surprising thing is how people just thought that it would continue on the way it always had done. But in reality, we’ve made a lot of changes in a very short space of time. So even if we deployed all the new assets that we have.
That’s quite a big change. But then to really move all of the demand around very, quickly. And you could observe that in the results that came out. You see, basically on the day everybody got sent home, the day after that, the load profile of the country was completely different. And then probably the second most surprising thing about that period, I think for a lot of people who aren’t in the really deeply into the energy industry, is that a lot of people just kept doing what they were doing before.
For many, many industries, right, you can, if you if you’re a manufacturer of goods, that can only happen in the factory, you can’t send that manufacturing home. So there was a lot of workers who were really hot in all their additional PPE. But they were still getting to work every day, they were still getting up at 6am, doing the commute, going into their place of business, doing their work, coming home. So their load profiles didn’t really change that much.
Whereas, office workers like myself, we instead of only up at six to get a train, you know, we could have an extra half an hour in bed, which is lovely. But it meant that we squashed all that consumption into a much shorter period. So those are my kind of two things in kind of really recently that I think have come as surprise to more than one person when I’ve talked about it.
Ivona (27:11)
And it’s an absolute game changer, isn’t it? Because the way you’ve done things before doesn’t work anymore. And you have to change, you have to adapt, you have to do something differently. And I’m referring to how the forecasts were made in the context of what you just said. So, yeah, thank you so much for that insight. That was interesting. Usually when we wrap up the episode, we ask
people where they can go to find out more about you and energy forecasting. But this time very conveniently, there’s a webinar coming up on the 23rd of October. So if you enjoyed this episode, you can join Adam and Duncan for a deep dive into energy forecasting, as well as showcasing some new tooling from SAS and Hypercube. Depending on when you’re listening to this,
you can either watch it live or there will be a recording available in our show notes as well. But for those people that just want to learn more about you and SAS in general, where should we direct them to?
Duncan Bain (SAS) (28:10)
So you can go to the SAS website, www.sas.com, and you’ll find all sorts of information about all the different things we do there, including energy forecasting. And I’m on LinkedIn. I’m usually at all sorts of different events. So if you spot me out on about one of those, please do stop me and say hello or drop me a line. I’d be very happy to chat.
Ivona (28:31)
Duncan, thank you so much for joining us for the second time at a Hypercube Energy Forecast. It has been an absolute pleasure talking to you about all things forecasting and I hope we can do it again sometime in the future.
Duncan Bain (SAS) (28:44)
Thank you for inviting me back.
One of the things that shows how important this subject area is, that we talked about energy forecasting way back in episode eight. Still talking about it now.
I’m still just as excited about it as I was, you know, way back then. And as excited as I was about it when I first came into the industry, right, because it’s such an important part of the kind of whole ecosystem of delivering power to people’s homes and businesses.
Ivona (29:10)
Absolutely. And I think this is just such a fabulous note to finish the podcast on. So thank you so much.
Duncan Bain (SAS) (29:17)
Thank you.