Season 1 Episode 9: Connor Galbraith, Energy Systems Catapult, on Quantifying the Impact of Retrofitted Energy Efficiency Measures

In this episode of the Hypercube podcast, host Adam Sroka is joined by Connor Galbraith, Consultant Data Scientist at Energy Systems Catapult (ESC). Connor shares insights from the RetroMeter project at ESC.

He explains how using smart meter data and modelling techniques allows for more accurate measurement of the energy savings resulting from retrofitted energy efficiency measures.

LISTEN ON SPOTIFYLISTEN ON APPLE
S1E09 Connor Galbraith

In this episode, we covered:

  • The impact of the gas price spike on consumer behaviour.
  • The RetroMeter project and the potential opportunities it unlocks.
  • Overcoming the challenges of working with gas smart meter data.
  • How to utilise energy data projects, like RetroMeter, in the real-world.
  • Issues with the UK talent pipeline and the knock-on effect on decarbonisation.

The weekly Hypercube podcast sits down with leaders in the energy and utilities sectors to explore how data analytics can help businesses make smarter decisions and accelerate business growth.

CHAPTER MARKERS

[0:39] Connor gives an overview of his current role at ESC.

[2:19] Adam asks Connor to share insights from his time working on the RetroMeter project.

[3:53] Connor explains how RetroMeter can unlock Pay for Performance financing packages for retrofits in the UK.

[5:45] Connor walks through some of the methodologies being used as part of the project.

[12:31] Adam and Connor discuss the quality of the data sets being used as part of RetroMeter.

[15:10] Connor talks about the future of the project and how it can contribute to meeting net zero goals.

[18:27] Connor explores the challenges he sees in academics translating innovative ideas into production-ready code that can be used in industry.

[20:31] Connor emphasises the importance of successful trials to get buy-in for decarbonisation projects.

[23:24] Connor highlights the impact of a limited talent pipeline in the UK on decarbonisation.

[25:25] Connor shares where listeners wanting to know about ESC and RetroMeter can find more information.

TRANSCRIPT

Hypercube Podcast Transcript

Title: Connor Galbraith, Energy Systems Catapult, on Quantifying the Impact of Retrofitted Energy Efficiency Measures

Host: Adam Sroka
Guest: Connor Galbraith

Intro: Welcome to the Hypercube Podcast, where we explore how companies in the energy and utility sector leverage data analytics to make smarter decisions and accelerate business growth. I’m Adam Sroka, founder of Hypercube, a strategic consultancy that supports asset owner-operators, traders, route-to-market providers, and energy services companies to unlock the power of data.

If you’re interested in hearing real-world examples of how data and AI are advancing the energy sector, this is the show for you. 

Adam Sroka: Welcome back to the Hypercube Podcast. I’m delighted today to be joined by Connor Galbraith. Connor, would you like to introduce yourself and tell us what you’re up to for the audience?

Connor Galbraith: Yeah, of course. Thanks, Adam. Good to be on the show. My name’s Connor and I’m a data scientist at Energy Systems Catapult. We’re part of an Innovate UK-funded innovation hub based out of Birmingham, and I work with Samuel Young, who I believe you had on your podcast recently. He heads up the AI and data science team at the Catapult.

Adam Sroka: Yeah, we had a really interesting chat about some of the work that had gone into the direction of travel and where we think AI can have an impact in the space. So, yeah, if people haven’t checked that one out, they should do; it was a really good conversation. But why don’t you tell us a bit about what you’re up to at ESC then?

Connor Galbraith: Yeah, of course. I’ve been with the Catapult now since last August, and most of my time has been focused on working on a project called RetroMeter, which is part of a strategic innovation-funded project from Ofgem. We are working on it in a consortium with Electricity North West, the DNO around the Cumbria region, and several other companies you may have heard of, like EP Group and Carbon Co-op. The whole idea of this project is that we are lacking in the UK the ability to really quantify exactly how effective domestic home retrofits are. There’s a lot of guesswork that goes into trying to disentangle the impacts of things like roof and cavity insulation, heat pumps, and so on—how much energy they actually save once you control for things like behavioural change, temperature, and prices.

Adam Sroka: So, any massive findings you can share at this point, or any insights that people might find unusual?

Connor Galbraith: I think one of the most interesting things is that—the details we’ll get into later on, I’m sure—but at a high level, the impact of the gas price spike we experienced the year before last was dramatically worse than we might intuitively understand. One of the really interesting things about this project is that we were given a rare opportunity to work with a very large set of gas smart meter data from about 16,000 properties. As you probably know, it is very difficult to work with smart meter data in the UK due to privacy and governance reasons. One of the things we found is that despite last winter (the winter of 2022-23) being quite a lot colder than the winter before, the average consumer consumed a lot less gas because of the price spike. This was visible across the entire dataset. While it wasn’t strictly what we were looking for, it was an interesting finding and helped demonstrate why we need a more complex methodology for getting to these metered energy savings than perhaps was originally thought.

Adam Sroka: Do you have some insight into where you think the biggest impact of this body of work will be, and what might be the most exciting follow-on from this?

Connor Galbraith: By far, the biggest use case will likely be unlocking what’s called pay-for-performance financing packages. It’s no secret that retrofits can be quite expensive and invasive. There are many properties in the UK, particularly among the social housing stock, where you have the double whammy of occupants being more sensitive to energy price changes and the properties being the least efficient in terms of holding onto heat. By putting an unambiguous number to metered energy savings, green financing companies can tie the funding they provide to the performance of the retrofit because they are confident in the avoided energy use that their investment has generated. This will be revolutionary in the social housing stock, where entities that own and manage large portfolios of homes 

It is much simpler from a financing perspective to get a big chunk of money to retrofit dozens to hundreds of homes at once, and then tie the avoided energy use of that entire portfolio to a single financial instrument. Now,  this is more what EP Group, part of our consortium, are looking at. They’re looking at the business models and the supply chains that lead to this kind of innovations that take the results of the modelling that we’re doing at Energy Systems Catapult. And taking them to decision-makers and capital providers in the UK. 

Adam Sroka: So this is a bit like an open-source approach, right? So what kind of methodologies were you looking at around that? And did you see any sort of particular challenges? Because that’s no mean feat.

Connor Galbraith:  No, so the approach that we’re taking at the moment is trying to use as much of what is out there currently as we possibly can, and then adapting it to the specific UK use case.

So currently the most popular and most effective energy savings open source model out there is one called Open E Meter that was developed by a US company called Recurve and is now being maintained by the Linux Foundation. energy outfit. And the way that one works are, say you have a property that is going to have a retrofit or has already had a retrofit in the recent past.

You take the year’s worth of smart meter measurements for either electricity or gas. In the US, it’s a lot more common to have electrically powered HVAC systems for both heating and cooling. In the UK, we don’t really have that. So we’re focusing on gas-powered heating. And then it’s basically a very fancy set of linear regressions with a very sophisticated model selection approach that says, okay, we’ve trained a year’s worth of energy consumption data based only on the calendar effects and the external temperature to that property.

And the output of that modelling is what’s called counterfactual energy usage. And this is a word that comes up a lot in metered energy settings. What is a counterfactual? It can be quite an unintuitive concept to wrap your head around, but what it says, post-retrofit, what energy would you have consumed had you not had the retrofit, given the temperature the year after the retrofit? 

So, the model assumes that you haven’t had a retrofit, and says, okay, given the temperature we’re seeing today, this is what you would have consumed. And then, the difference between that counterfactual. And what you actually observe the property consuming after the retrofit is your meta denture savings.

Unfortunately, that’s not the whole story. That’s only step one. Yeah, okay. Because one of the downsides to only using external temperature is you have to make a lot of assumptions about the heating behaviour of the occupier hasn’t changed before and after the retrofit. Their sensitivity to energy prices hasn’t changed.

Also, the energy price itself hasn’t changed. In practice, you can’t really use any of these assumptions without getting very Uncertain results. So we’ve taken open e meters as a black box and have applied that to several thousand properties in the UK that have got two years of gas and temperature readings in order to generate that counterfactual for each of them as if they had had a retrofit.

Due to how well the data has been anonymized, we don’t know which of them  statistically very few of them will have had a retrofit. We have to make that assumption for statistical purposes, but some. As I mentioned earlier, the dramatic shift in gas prices, say, the index more than doubled between the winter of 21 and the winter of 22.

We have to control for that somehow.  Yeah, indeed, because this, from the perspective of an open E meter, it doesn’t think anything’s changed.  So when we applied it on its own to all of these properties, it massively overestimated the energy use because the winter was colder, but in reality, people were consuming less because the prices were higher. 

So this is where the second methodology comes in. And this is where we’ve been a bit bespoke in our approach. And so this is what’s called a comparator methodology, where you say, given a property that’s had a retrofit, we’re going to take five, 10, 25 properties that haven’t had a retrofit but are quite similar to your candidate property.

We’re going to fit counterfactuals to them in the same way we have for the candidate property, but then. deduct the estimated metered energy savings, or to be more specific, the model error from those aggregated comparator properties, from the candida property, to remove the effects of things like price changes and other externalities that affect the model.

all of the houses equally, whether they had a retrofit or not. So once you do that, you should be left with only the impact of the retrofit on the candidate property because you’ve stripped away the effect of price elasticity and other society-wide externalities. 

Adam Sroka: How do you select those comparator properties? Is it random, or is it like the nearest neighbour approach?

Connor Galbraith: So the benchmark is random. That is actually where a lot of the performance of the model is decided. It’s in how you match those comparator properties to the candidate. The way that we found works best is to just look at the raw similarity of the daily consumption profile on the baseline the year before the retrofits between the candidate and the comparator. So in our case, we’ve used a metric called the CVRMSE, the coefficient of variation of the root mean squared error. So in layman’s terms, the higher it is, the more dissimilar two consumption profiles are. So what we’ll do is calculate that metric between the candidate property and the comparator.

All of the other properties in our data set and take those with the lowest CVRMSC, indicating they have the most similar profiles prior to when the retrofit took place. The hypothesis is that if they’re very similar in one year, they’re more likely to be very similar the next year. And this, surprisingly, our hypothesis going into it was that if we match properties based on, you know, their built form, whether they’re a detached, semi-detached terrace, uh, how old they are, their EPC rating, that kind of thing, that that would be enough.

That’s actually not what we found at all. Just going off of the inferences that you can make from the smart meter data is more than enough. Now, one of the consequences of that is there’s no guarantee that you’re going to have access to the smart meter data of thousands of properties from which you can choose effective comparators. 

So what we’re working on now, as we’re wrapping up the project, is How much performance can you squeeze out of the model when you don’t have access to smart meter data prior to the retrofits. And that, that’s a really interesting challenge because going back to the impacts on social housing stock in particular, smart meters are very underrepresented in those kinds of properties.

So this is something that we really have to. Get right the first time before, you know, to make sure that all kinds of properties and occupiers are represented.  

Adam Sroka: So, with all housing data stuff, it’s just really hard, isn’t it? Because the final home, like makes some assumptions to the people in it behaving like normal people. Like, is there a hole in a window somewhere? How’s the loft insulation? It just goes on and on and on and on. So we’ve seen this a few times that it does get really complicated really quickly. 

The bigger challenge though, I think is that can be that smart meter data, you hinted at it earlier, you know, like even integrating and getting data from like smart ECC and the like, how did you find the data set, were there any surprises or challenges with it?

Did you have like quality issues or anything like that? I think that’d be quite interesting.

Connor Galbraith: So, we partnered with a company called Hildebrand, who have been around for quite some time now. They’re probably one of the best for smart meter data and data management in the energy sector. And they gave us a pool of about 16 and a half thousand homes of gas smart meter data for us to work with.

And because they are very good at what they do, there were very few problems. few data quality issues. Great. The only issues that did present themselves is gas smart meters are notoriously buggy compared to electric smart meters because you’re trying to convert the flow of a physical gaseous quantity, you know, literally into an electrical measurement, which for all kinds of reasons is trickier than just measuring voltage over a resistor like with smart meters.

So. In data projects like this, I’m definitely a big believer that if you have six hours to cut down a tree, you spend the first bore sharpening the axe. So, I spent a lot of time doing things like converting them all to nested parquets, using a lot of rust-based tools like polar on the server and that to make sure that we had very fast iteration.

In my experience, that’s probably what makes or breaks a data science project. You can’t plan everything in advance in a project that is mostly research, hypothesis testing, and experimentation. So, make sure that you’ve got the essential foundations, and the data engineering done right the first time so that you can iterate on the data science side quickly, I believe leads to much more fruitful innovation at the end of it, as opposed to just barreling into the, the modelling first thing.  So that’s been really good

Adam Sroka:  A refreshing take. Yeah. And it’s the kind of thing, I think you only make that mistake a few once or twice before you realize, like you mature as a techie and realize that, actually, you just really need to get stuck into the data and the models will take care of themselves at the other end. 

Connor Galbraith: Absolutely. Plumbing is not glamorous, but you miss it when it’s not there.

Adam Sroka: That’s a really interesting project then. If the project was to get extended and go on and do something else, what would that be? And how does that, like, what’s the biggest contribution it could make towards like broader net zero goals, for example, across the country? 

Connor Galbraith: We are We’re most definitely looking for ways in which we can take this further. We’re all very encouraged by the results that we’re seeing, not just at the Catapult, but other members of our consortium as well. 

 

The next step undoubtedly is to roll this out on a trial basis in some sort of local area scheme or some large portfolio of houses in which we can directly point to the verified measuring of metered energy savings on homes that have had a retrofit. That is one of the limitations of this alpha phase is what is called part of the SIF innovation funding.

We don’t have any properties in our data set that have had a retrofit. So our null hypothesis is that the metered energy savings are zero. That’s how we’re able to talk about accuracy and bias against that baseline. But to actually demonstrate that there is a demand for high confidence, metered energy savings measurements.

The next step would involve demonstrating those measurements and getting buy-in from energy suppliers and financial institutions who are willing to tie financial products to this flow of verified savings. And as much as I’ll sit here and harp on about how data science is fantastic and necessary for reaching net zero, ultimately it is capital allocators and policymakers. Who implements the results of what we do. A hundred percent. So being able to engage with those stakeholders and players. And as well, just, just as important as well is engaging with the households that are going to be potentially taking out the funding for these products. Like we’re not in any way advocating for a completely top-down approach where a, an owner of a large group of housing demands that all of the houses have an incredibly invasive retrofit because there’s a nice yield attached to the loan at the end of it. There has to be sympathetic engagement and buy-in with the occupiers themselves, and that’s what forms the basis of a lot of the work that Carbon Co-op are doing on this project. They have their own local schemes running at the moment, and they do a lot of work engaging with homeowners and tenants directly to work out how best to interface the results of projects like RetroMeter with them directly.

Adam Sroka: Yeah, that’s really good. I think because that’s like it’s taking what can be quite a difficult to understand or measure idea and making it more tangible and translating into terms that going to land with the folk because like I’ll get insulation because you’ll save on your energy bill. Okay. Yeah. Great. But how much am I going to save? These are really hard questions to answer. And so this is like a really pivotal piece of work in getting into a better, more precise answer for some people. And to say, actually, we think this is the impact it’s going to have. And this is why there’s some robust modelling behind it. And we’ve done a lot of clever thinking and looking at real data. It’s not just a finger in the air. Absolutely. Yeah.  So, let’s step away from the project a bit there. From what you’ve seen here and elsewhere, and like, from your perspective currently, what do you think the biggest kind of challenges are around applying really good data science techniques and modelling and analysis to the energy system, like, where do you think the biggest challenges are, and the biggest opportunities, I suppose?

Connor Galbraith: Yeah, there’s always two sides to that coin. Challenges and opportunities abound. So in my career, I’ve been on, I’ve been on both sides of academia and industry, and both are crucial in their own way to achieving the UK’s net zero goals and globally as well, decarbonization. But one of the biggest challenges is academia sometimes struggles with translating amazing innovation that they do into what we as data science practitioners would call like production Like there’s a lot of people in academia who are very used to perfectly sanitized datasets, kind of very stream of consciousness Jupyter notebooks. Let’s say I know this because I was one of them. 

Adam Sroka: We’ve all been there. Don’t worry. 

Connor Galbraith: And it’s, I’m in no way advocating for any of that to change because sometimes that’s the way that it’s very free flowing and also where serendipitous breakthroughs come about through not having that commercial restriction.

But I do not think enough work is being done both on the commercial and the university side for really engaging between them where the goal is. production-ready outputs. And there are some great examples of where this is working well, like at the UCL Energy Institute and the Oxford University Energy Centre and the Grantham Institute with the Imperial.

They are really leading the way, but that’s only three institutes.  There’s a lot more going on in the UK that we in the industry are not seeing because of the translation Between them. An industry is just either underfunded or mis-incentivized. And taking another step back, this is probably my personal opinion, but you know, the UK’s energy grid intensity is halved in the last 10 years.

It’s gone down by 80% in the last 70 years, so much of the low-hanging fruit in the energy sector has already been decarbonized. The really difficult stuff is all of the decentralized. Assets like domestic heating, transport, and all of the distribution level electrification that is going to be necessary to support those being decarbonized.

I think is massively underestimated just how technically and logistically complicated that is going to be. And a lot of the innovation in that area and a lot of the suggested solutions for these things. They may not get the trade-off correct between the benefits of decarbonisation of centralised control and the potential limitations that that puts on personal freedom. 

One example may be heating systems that are controlled, kind of, at the centralised level for communities and at the postcode level, telling people that In order to meet decarbonization goals, you have to give up the ability of choosing when to heat your home and to what set point. In the UK, it is a very difficult thing to justify. 

I mean, you only had to look at, yeah, someone else knows the, how to hit home better than you that this, I could certainly just gives me the alert of, you can envision the pain of trying to have that conversation and convince people, and even no matter how much money they’ll save, like you get people that, yeah, that’s just not for them.

Yeah. And the onus is on us to get that balance. Right.  And a lot of it, I think, is going to come from very robust and well popularised trial projects. Like, all over the world, there’s been more and more successful pilot projects for introducing universal basic income. And no matter what your opinion is on universal basic income, you can’t argue with the results of those pilots that show employment doesn’t go down, it actually goes up. Mental health, well being goes up. And the same thing needs to happen with AI and energy, right? The benefits need to be demonstrated in very, not just, you know, Technologically, but emotionally palpable terms like that, in order to get buy-in from the community.

Adam Sroka: It’s always a people problem. It’s always a people problem. Yeah. I talk about this all the time with all our customers, but all the tech will solve itself. You can just get everyone to agree on what it is you need to do and how it needs to be done. So it becomes the easy bit, but cajoling people and different stakeholders and aligning success factors at multiple levels.

That’s the hard bit. I totally get that. What do you think the biggest barrier to progress is? What’s going to be the most impactful thing? Is it more awareness and academic level? Is it more work done with communities? How do you think we’re going to get better analysis and better approaches out into the energy system? 

Connor Galbraith: There are three levels to that, I think. The first is a very UK-specific thing, and that is the talent pipeline. Compared to the US, we pay dramatically less for data, AI, and energy talent than they do. And until that balance is rectified, we’re going to be playing second fiddle to people going over to the US and Singapore, et cetera.

So talent is one barrier for sure. And retaining talent in companies that are powerful enough to actually make a difference in the UK. I think another thing is, I think it’s very important to always remember that, you know, China, the US and India emit as much carbon in six days as the UK does in a year.

That’s not to downplay how incredible our decarbonisation efforts have been in the UK, not at all, but I believe an obstacle that we face in the UK is obsessing over that last little bit to meet our legal net zero obligations. the expense of painting ourselves as a demonstration for the world that you can get to net zero without sacrificing economic growth and prosperity.

And we have not been good at that lately. You only need to look at the ULAs in London and how that manifested in the Uxbridge by-election, right? We’re not doing a very good job at the minute of controlling the narrative that decarbonization and economic growth are both possible at the same time. And until that case is made strongly and with confidence, then it doesn’t matter how much the UK decarbonizes because it’s going to be wiped out in less than a week by the bigger economies of the world.

Adam Sroka:  I think there’s a lot of promise, right? And it’s only going to take things like this and some of the amazing work that you’re doing at ESC to move the needle and really move the needle, like do things that mean something to real people that don’t live and breathe energy but are affected by the decisions we make at all levels.

I guess the final point for me is always, is there anything you want to promote or push or point people to so that they can find out more?

Connor Galbraith: Well, I’d be remiss if I didn’t say we have a webinar coming up, although by the point this podcast goes out, it’ll be over. I would keep an eye out for the results of the Retrometer project that we will be publishing and disseminating through Electricity Northwest and various other blog posts and webinar recordings.

Adam Sroka: It’s been a pleasure to speak to you today, Connor. Thank you very much for joining us.

Connor Galbraith: Thank you. Goodbye.

Outro: And that’s it for this episode of the Hypercube podcast. Thanks for tuning in today. If you have any questions about the topics we covered, you can reach out to us on LinkedIn or check out our website at wearehypercube.com. You can also join Beyond Energy, our Slack community of data leaders from the sector. There’s a link to sign up in the episode description. 

We’re just getting this show off the ground, so if you liked today’s episode, please leave us a rating, review, or subscribe wherever you get your podcasts. It all really helps.

See you next time.