Hypercube Podcast Transcript

Title: Smart Meter Data Usage at the Centre for Net Zero

Host: Adam Sroka
Guest: Gus Chadney and Sheng Chai

Intro: Welcome to the Hypercube podcast, where we explore the cutting edge of data and AI in the energy and utility sectors. I’m your host, Adam Sroka, founder of Hypercube, a strategic technology consultancy, supporting asset owner operators, traders, route to market providers, and energy services companies unlock the power of data and AI.

Join me as I sit down with experts from across the energy industry to uncover how they are navigating today’s data driven challenges. And without further ado, let’s get into today’s episode. 

Adam Sroka: Hello, and welcome back to the Hypercube podcast. Exciting one today. I am joined by both Gus and Sheng from the Centre for Net Zero. Good morning, chaps. Would you like to first, yeah, introduce yourselves for the audience at home? 

Gus Chadney: Yep, sure. Hi Adam, I’m Gus, I’m the Data Lead at Centre for Net Zero. Yeah, joined in July 2023, I think. The goal of managing the data team and overseeing our data and modeling projects that we work on.

Before that, I’ve had various kind of roles, many data focused, but in different industries and started in finance and then marketing had my own little startup journey and then came here. 

Sheng Chai: Cool. Yeah. My name is Sheng. I’m a Senior Data Scientist at Centre for Net Zero. I joined DNZ, I think in June, 2022 and predominantly been working on generating synthetic smart metadata.

And prior to DNZ, I’ve worked across many different industries, mostly in the tech startup and scale up role. So I started working in finance, like Gus, and I worked at Cloud at Monzo before moving to e commerce. And then working on recommendation engines at Gusto, and now I’m working on generating AI stuff at Centre for Net Zero.

Adam Sroka: Really exciting. Yeah, I could talk about smart biodata all day, actually. But before we get into that, it’d be just interesting for those in the audience that maybe don’t know the Centre for Net Zero or aren’t deeply aware. Could you give us a kind of run through of why it exists, what you’re trying to achieve some of the main Outcomes and deliverables.

Gus Chadney: Sure. Yeah. So Centre for Net Zero is a not for profit impact driven energy research institute, which was founded by Octopus Energy Group. It’s one of the largest energy retailers in the UK, but as a industry, we operate autonomously designing and running research and field trials all around the world to Yeah, generates data and insight about the future energy systems.

So we have a team of I think 18 now made up of data scientists, economists, strategists, and policy experts. And yeah, our goal is to accelerate the journey to a fully sustainable Global energy system in terms of the problems we’re trying to solve. So yeah, obviously our energy system is like changing rapidly and we’re looking to decarbonize, introducing new technologies and we think understanding consumer behavior is essential to this transition.

One of the things we’re working on is the data. So a lot of the critical decisions today in energy use data from the past. Sometimes aggregated out of date data, mainly due to privacy or technical concerns. Few organizations have open access to the right energy demand data like us, or the motivation to share it.

So yeah. We’re working to make this data open and use it to figure out how to do accurate kind of future demand prediction. So yeah, we essentially day zero, we’ve got the access to the data, access to the consumers to understand their behavior. And that kind of combination will allow us to.

Yeah. Hopefully, really understand demand in the future. So that’s what we’re trying to do. And yeah, octopus obviously is growing as well. We’re going global and our research is going to go global with it. Ideally. So yeah, that’s us. I hope that gives a. A bit of an introduction.

Adam Sroka: That’s really interesting and it is good to hear there’s a mix of like economists and like it’s not just all techies or it’s not just all sort of energy people. I guess just wanted to lean into the question a bit. How do you prioritize what to go after? Because it’s such a huge remit, right?

How much of it is like driven by The economics and the forward looking the big change stuff versus like technology driven. Like you like talked about generous of AI in the introduction. How much do you say? Actually, there’s opportunity here and we need to be the people to do the. the kind of TRL one, two, three part of that journey?

Gus Chadney: Yeah, great question. I think, yeah, two main things to point out. One, we’ve got to play to our strengths, and that is the relationship that we have with the wider group and the access to the customer data. So we, yeah, we have to really, leverage that data and those customers and make sure our research is fully aligned to that.

I think the second one is, yeah we. One of our missions is to do really impactful stuff that no one else is doing. So we probably were trying to do more of the kind of future looking scenario based stuff rather than the day to day operations. Cause that’s where we see the, the biggest chance of Making and powerful change.

Sheng Chai: Yeah, and to add to what guys are saying as well, because a lot of the data that we have right now are data from the past. But we all full well know that in the future with climate change and with technology adoption, today’s data is a of what’s going to happen. In, say, 10, 20, 30 years time, but I think what seems that what we’re really trying to do is like thinking about how might the world look like in the future, and then actually collecting performing, carrying out trials today to collect data that’s reflective of the world that we may see in the future, and then taking some of the data that we’ve collected with trials and then doing some modeling, actually see, How could we make Net Zero a reality?

So I think that’s what we really can do at sense of Net Zero apart from that’s quite different to many other research institutions out there. 

Adam Sroka: That’s really interesting. Like last week I was in with another major energy provider not the octopus group, but of that scale talking about exactly this smart meter data.

And it was mainly around decarbonization of heat and just Where do you even start? Because you’ve got building physics models, you’ve got retrofit schemes going on. Even having access to the data actually is really difficult, unless you’re one of these scale organizations. I guess it puts you in a really useful position.

Even just the interface and owning all the gubbins that you need to support, DCC queries, and so on and so forth. It’s no mean fee. Yeah. Where do you even start on that modeling, like that models of models kind of journey? 

Gus Chadney: Yeah I’ll actually go to synthetic data a bit more and what its kind of advantages are, but that, that, so that’s been our main angle.

So creating really realistic, synthetic, smart meter data on your points for openness and democratization. One of the things we identified really early on is that we want as many researchers and commercial organizations and, PhD students to have access to this data as possible.

So we, earlier this year, we launched like an open community around synthetic smart metadata called OpenSynth, partnered with Linux foundation energy. And the goal of that, that really is so that they’re just like accelerating the amount of smart meter data in circulation. So not only are we driving our own research forward we’re, allowing other people to do that.

To do that as well. And the community has been really successful so far. We’ve released some synthetic smart meter data and we have other people really interested in doing the same. And yeah, it’s been a good journey so far. We 

Adam Sroka: had someone from the North Sea Transition Authority about six months ago now on the podcast and just similar parallels in the, they became the people that owned all the seismic data for the North Sea.

Did a great job of opening up to researchers and my other organizations. And then the pace of research innovation, like new understanding and people being able to get access and get involved with it has like orders of magnitude change. So a really noble effort and really interesting. But yeah, not possible without synthetic data if you’re dealing with people’s homes and an energy usage, right?

Tell us a bit more about that journey. 

Sheng Chai: Yeah, I think at the start when CSL first started I think in 2021, CSL was partnering quite a few different researchers and it was quite difficult to actually share data with the research community because every Partnership that we do, you need to have a separate legal procedure and you need to do all the engineering stats and clean the data and make sure that it’s anonymized.

And then anonymization is also not foolproof. There has been quite a few instances in the past where people were able to reverse engineer anonymization or aggregation to get back to the raw data. But there are still like lots of risks involved with that and obviously when it comes to smart metadata, lots of people are quite sensitive about it.

So then at 7. 0, we just thought in a, how can we make this process of. Sharing data in a more scalable way and also, yeah, in a more scalable way. And also looked at privacy as a cause for something that we want to consider, because obviously many people are quite sensitive about it. So we then started looking at generating synthetic metadata using generative AI.

And then we put something out there in 2022 and got quite interested. And today now we’ve got like just over a hundred alpha users for our talk on Friday. Yeah. So we’ve got researchers from Oxford, Birmingham, Manchester, Cambridge using Friday, we’ve also got of German deafness have to use Friday.

So yeah, we’ve got people from consultants and researchers to government and regulators using Faraday and then we saw, because obviously Faraday is trained on octopus households in the UK and the world does is just revolved around the UK. There’s many other people in many different countries and all of them are looking at, also crying out for data.

So we thought, how do we accelerate this progress that we’ve done on this front? And we saw how about that startup and openstance is the open source community with Linux Energy Foundation. And hopefully we get a salary generation of granular energy demand data, not just in the UK, but globally as well.

So that when it comes to Net Zero transition, we get. It’s just not, it’s not just one country, the entire world needs to come together. Yeah. So whilst we can make a lot of progress in the UK, we can’t forget that, many other countries actually send me to to get more data and yeah, and that’s how we set up OpenSyn.

To really accelerate the efforts. 

Adam Sroka: So I imagine like the data that you get from meters across other countries aren’t going to like this just not going to meet the same standards and the same sort of granularity like different sampling frequency so on and so forth. So are you, is that posed a challenge yet or are you thinking about is there like an abstract data model that is like smart meters that applies everywhere?

Sheng Chai: Yeah, so that’s probably that’s something that we’re not really that worried about because yeah, maybe different countries might have different sampling methods I think in California might be down to every 15 minutes Whereas you and then I think some European countries also at 15 minutes or at UK is at 30 minutes but I think it doesn’t really matter because I’m It depends on the granularity and what you want to look at Because in the UK, they’re the most reliable guest at 30 minutes interval.

So any research to the UK will be at that interval. But if you’re at other countries, yeah, I think we could generate 15 30 minutes. It’s really up to the people who are doing the smart metadata, what would they want, what they want to do with it. So I think it it’s more of the behavior getting away.

Representative behavior of how the world use energy that we care about, because obviously not everyone’s going to use energy the same way that UK citizens are using, either because of behavioral stuff or because of weather driven consumption behaviors. One, obviously people in Texas would be using energy in a very different way to people in the UK because of climate.

And then it’s about getting your representative data set of the world that we live in. And then we can then look at how we might be able to model consumption in one country and then how the entire world can function together. Recently in the UK, we started investing and in Morocco, which is like hours behind in the UK.

So that while the sun is shining in Morocco. Energy can be, solar energy can be used to meet the demands of a UK evening peak time. And you can only really do this if you look at globally how different, using energy in different ways and how then we can all come together. And it’s about the global representativeness of the dataset that I think is more of a challenge.

That’s the thing we’re trying to prioritize. 

Gus Chadney: That’s the main purpose of the community. So it’s to bring those different holders of disparate raw data, the different methods to then generate synthetic data from them and the different use cases people wanted to use that data in their research, have them all in one big mixing pot so that people can, have one place to find what they need.

Yeah, no, no one size fits all model, unfortunately, but Hopefully, yeah, one community to hold it all basically. 

Adam Sroka: Now that makes sense. And while we’re on the topic, then any sort of tech specifics and that are really interesting or have like in generating synthetic data, what kind of tech stack are we using and are there any surprising finds in the technology, or is it just trying to keep pace with OpenAI and whoever else?

Sheng Chai: Yeah, so I think for OpenStreetMap, the sort of stuff that we’re doing, we are nowhere as advanced as Stuff that open AI is doing. We’re looking at like quite basic stuff. But I think one of the things that we are looking we looked quite closely is on topic of privacy, because the whole point of us trying to create synthetic data is to make sure that, this.

Previously of your regional data set or people’s energy consumption is actually being preserved. And how would you actually know that your data is genuinely synthetic and not actually just regurgitating the original data because it’s doing that and that kind of fits the whole purpose of synthetic data.

And when we were doing literature review of like how to generate and evaluate synthetic data and we also found that there are lots of papers on how you could do it. But many people are also evaluating it in their own ways. There is no common standardized benchmark, so that it’s really hard to compare.

Are we the ones that everyone are worried to? And most importantly as well, previously is often not a topic that many people talk about and how are they actually going about evaluating privacy. But actually on the topic of privacy, there is actually already a lot of research done outside of energy, for example, in healthcare, lots of researchers are looking at synthetic healthcare data sets and how to protect privacy in those data sets.

And recently, since Net Zero, we released a paper called defining good, basically evaluation frameworks for synthetic smart metadata, where we look at things like fidelity, utility, and privacy, and looking at, how do we actually, by the way, So that data is really private, or actually just spitting out your own data.

Yeah, and I think that’s basically, I think the biggest challenge about the metadata is how we actually generate the metadata that is one, highly private, but to still remain useful because the more private it is and the less accurate it is and what is that trade off we’re talking about when it’s coming.

Yeah, when I tried the trade off between privacy and the usefulness of the data set. 

Adam Sroka: And do you find that it’s still quite an immature part of the kind of tech space, because the reason I asked that is from our conversations with a lot of customers and big other organizations, but also their customers, people aren’t as protective or as nervous about this smart meter data until you highlight the value in it to them as they would be with their, like their finance, right?

And a lot of the work I’ve seen in synthetic data is around like financial transactions and organizations like the startup Hazy that’s really exciting doing cool things with like fraud detection. Do you find that it’s still quite early days in that space?

Sheng Chai: I think it depends. I think it depends. I think with the thing that we previously is, you never know the damage until it occurs.

So sometimes it might be more responsible, but better To have things up front rather than wait till bad things happen and everyone panics And then that’s then going to set progress 10 step backwards And it’s much sometimes much better to be proactive about it And there are lots of examples of how things like this happen.

Like I think when I was doing my post graduate in machine learning, one of the classic examples that was told in textbook was there was once where a supermarket was sending marketing emails to people mailboxes, and they started sending pregnancy or baby related to a household with a teenage daughter.

And then the father was like, the parents was like, what’s going on? And turns out that teenage daughter was pregnant and I was looking at pregnancy kids and then that supermarket found out I started sending. Things like this, and then that became an ethical issue. And with privacy, that’s exactly the type of things that we’re trying to prevent up front and looking at and have means to actually quantify privacy and look at how to protect it.

And we don’t want, negative things to happen because, yeah the probability of negative things happening. happening is small, but the impact, especially impacts individual could be quite big. For example, if you manage to reverse engineer smart metadata and identify that the smart metadata belongs to you and then identify that, Oh, you were doing some behavior at this time of the day, so that you’re not, something might be going on.

And it could cause unintended consequences on individuals. And even in the UK, there is quite a high resistance as well to smart meter adoption compared to many other countries worldwide and negative impacts like this is just going to set the progress of energy digitization backwards and personally, I think it’s much more responsible when we’re dealing with data to, to be thinking about these and be more proactive about it rather than, Just wait for things to break and they try to fix them and sometimes, yeah, while they try to fix them, it should have been done.

But yeah. 

Gus Chadney: I think it’s also, it’s quite hard for people to understand how organizations like us can use their smart user data for such like amazing benefits. It’s quite kind of opaque in terms of that link between your, your energy usage ticking along and then actually making real, change in the grid that, that benefits or gets us further to Net Zero.

One of the things we’re going to be working on later this year is to try and identify some of those use cases so that, and yeah maybe publish some of the most important ones so that people can potentially link to it. And then. build that trust that we’re using the data in the right way to drive real change.

And it’s not, for commercial reasons, for example or anything else. Yeah, we think building up that link will be key to getting a bit more trust in synthetics, farming, data, and allaying some of those fears. 

Adam Sroka: And I think that’s a really interesting point because we’re in a generation now that we I think we’ve been trained by Cambridge Analytica, the we now know that our data is valuable.

Don’t quite know why, but I know that it’s valuable. And I know I don’t want you to have it because I can’t really articulate that. And that use case around, I think it was Target that was sending out the pregnancy tests in Hannah Fry’s book, Hello World. She talks about that. And I think it’s a Google.

Quote, she brings up this, something like there’s a creepy line and you want to get as close to it as possible, but not cross it. That’s where all the value is, but crossing it’s like really bad. And it’s such a hard thing because yeah, the damage, especially to a commercial organization like Octopus, where you are in people’s homes and it’s very frontline, there’s a lot of potential to do a lot of harm to the customer trust to that journey and the adoption and people will just revert back to what they know and understand. 

Sheng Chai: Exactly. And I think the thing about synthetic data is for synthetic data to be useful, we don’t want just the smart meter. We actually need information attached to it.

What type of social demographic they belong to, what type of LCD they might have, what type of property they live in. But the problem is the more you condition your smart metadata on this column, then previously becomes more and more of an issue. That’s why. It’s funny, good people we’re looking at now, how do we look at this in a way that we can protect privacy, but also at the same time, make sure that the outputs are still high quality because you can protect privacy by, and why end up with just like completely random noise, then what good is it that data when it comes to then research and modeling?

 

Adam Sroka: Yeah, not an easy challenge. I guess you could totally see a route to commercializing it as well. I know that’s not the point, but it’s so valuable. Like being able to have. Like easy to generate data sets just because again, some of the smaller organizations we are talking to. So a company that have built a startup that’s really exciting around retrofitting old buildings and actually like having all the smart meter data would just be super valuable and they could build really powerful models, but it’s just really difficult for them to get hold of and get the consent at scale and build like the POC for their thing works.

And now the technical burden. To get to the next bit it’s just too big because they’re, the kind of infrastructure that we in the UK have doesn’t quite align. 

Gus Chadney: Yeah, and commercialization, this data is also as important as the research elements, because, a lot of the changes that are happening in the energy transition are economics based, and companies and commercial organizations are making these changes and they need to make a profit as well.

So yeah, democratizing the data is, has to happen at all levels. To drive innovation at commercial level and also for the really complicated research questions that would fund the grid by, by 2050. So yeah, it’s all really important, which is why, yeah we want to make this synthetic data from Perl.

Sheng Chai: Yeah. And the, and interesting you measure it because quite a few of our testers were actually using Faraday for that. Retrofit retrofitting purposes, specifically, they were looking at like financing deals of assuming that, Oh, before you had a safe solar panel, your consumption might be like this.

And after you get solar panel, how your consumption might look like and working out what that difference in consumption might be into working out like the payout period of solar panels. And then figuring out a foundation deal. I think that was consultant that used priority for one of the projects fund up by.

Console not giving out too much information because I don’t know how much I can say about that and specifically Yeah, they were using synthetic data for that study of so for that particular representing study So yeah, so I think these are like just a couple of example use cases where synthetic data it’s making democratizing Excessive synthetic is really useful on commercialized but also more on like actually just advancing a transition.

 

Adam Sroka: But Yeah, no, I totally You See the value in it all, it’s really exciting. I guess you, you mentioned like forward looking and some of the research challenges. Do you have any insight into what you think the next steps for research or implementation of this are going to be? Is there anything to talk about and lead the audience?

Sheng Chai: I think there are lots of new models and algorithms coming out to generate synthetic smart metadata. One of the drawbacks prior to the algorithm we had is right now we are only outputting daily profiles. Part of the reason is Because of privacy reasons, other reasons could be used also due to like computational reasons.

There are new algorithms coming out, specifically things like diffusion where it really blew up last year when I think you have new company or stability, AI with the diffusion models or using deficient algorithms to generate synthetic images and videos. And right now there are quite a few papers coming out about using diffusion.

As well to generate very long range trans series. So that could be a potential technology that might be of interest that we want to look at. And also I think we need to understand, come up with more research that actually look at and evaluate the synthetic smart metadata, because I think lots of skepticism when it comes to synthetic smart metadata is, are they actually accurate?

Can we actually really use them in place of real data? And I think right now the research on that is still quite thin. And then I think having more research on that, say, oh yeah, we compare real business data use case and that use case. And that would just give more evidence that yes, in that data could be a very quick way to democratize data for research and modeling purposes and other stuff without having to share the real data.

And it’s just much quicker and easier. So yeah, more research in that front, I think will hopefully it’ll work. Increase the adoption and reduce, lower the barrier to, to adopt to, to adopting this technology. 

Adam Sroka: It’s been a really interesting conversation. Thank you very much. I always just like to ask if the audience would like to find out more or if you’ve got anything you’d like to plug or promote.

Now’s the time, where should we be sending them? 

Gus Chadney: Yeah, sure. Our website is http://centrefornetzero.org/  and you can find all of our output on there. Our papers, our tools, our four pieces. We’re also on socials, LinkedIn and X search, Centre for Net Z ero. The papers. Sheng mentioned our Linked from our website and details about.

OpenSync, community, also you can find our website that has a mailing list where we reach out with new data sets or updates to the community and the algorithms. So yeah, http://centrefornetzero.org/ is the main place. And that’s it. 

Adam Sroka: Thank you very much. I’ve really enjoyed it. All the best at Centre for Net Zero.

I’ll be checking out the newsletters and things because it’s very pertinent to some of the stuff we’re up to. So great conversation. Thank you. Cool. 

Gus Chadney: Thanks for the chat, Adam. 

Sheng Chai: Thank you. 

Thank you for tuning in to the Hypercube podcast. If you enjoyed today’s episode and want to dive deeper into the AI in the energy sector, be sure to subscribe so you never miss an episode.

We’re always eager to hear from our listeners. So if you have any questions, any topics you’d like us to cover or feedback, you can reach out to us on LinkedIn, or check out our website at wearehypercube. com. And remember, the energy landscape is evolving fast. Stay ahead of the curve with the latest strategies and solutions right here on the Hypercube podcast.

See you next time.