Last week, I had the pleasure of attending Big Data LDN 2024 – the UK’s leading data, analytics and AI conference with over 300 expert speakers and 15,000 attendees. I’ve been an avid attendee since Big Data LDN’s debut in 2016, and always come away with a multitude of takeaways to pepper into mine and my team’s work. Here are my key takeaways from Big Data LDN 2024:

The Star of the Show

There was a definitive shift this year, with one theme standing out loud and clear: Generative AI (GenAI) is here to stay. GenAI is no longer just a shiny new tool – it’s made its way into our everyday lives. Now, as organisations look to increase their use of GenAI, there’s a strong push to be more intentional about its use and where it can add value. A key takeaway from this is: start with small, internal use cases, and scale strategically. It’s tempting to dive into large language models (LLMs) like GPT-4 or GPTo1, but to balance cost and effectiveness organisations must be selective about when and where to apply these tools.

A great example of this came from Nihal Mushtaq Amin (Head of AI at COEO) in his session, “Do you need a multi-modal GenAI strategy?”. Nihal shared his approach to balancing cost and effectiveness by using GPT-3.5 for more basic tasks and reserving GPT-4 for the heavy lifting. I found this idea of being selective about model use quite compelling – you can get the best of both worlds without blowing through your budget.

One critical point that came up again and again was around data quality. GenAI’s usefulness is directly tied to the quality of data you feed it – no amount of model sophistication can compensate for poor data. This drives home the need for organisations to prioritise strong data governance practices from the beginning.

Methods for Mitigating Ethics and Privacy Concerns

Throughout the conference, serious questions were raised around ethics. One that stood out to me during a panel session, “Everything you wanted to know about data engineering but were afraid to ask”, was ‘are our jobs in danger with the rise of AI?’ Short answer – yes, if we don’t have the right mindset and adapt as the technology evolves. As AI has matured, it means Copilots can now write code to a good standard, with the potential to reduce developer overhead and hugely increase efficiencies. However, we still need trained developers with the knowledge to recognise when the code is incorrect, or needs tweaked, to produce high quality code at pace. Without knowledge and experience in the core subject, incorrect code goes undetected and causes bigger issues down the line.

This scenario can be compared to when computers entered the workforce – while this did take away the job of typists, those that adapted and learned how to use computers thrived with their new skillset and secured their positions in the evolving job market.

On the same note, privacy concerns around AI were a common theme. When using external AI models, user data may be shared with third-party providers like OpenAI. Many AI models require access to user data to function effectively, raising questions about data ownership and consent.

How can organisations mitigate these concerns? One way to do this is by deploying private versions of AI models within their own infrastructure, ensuring their data remains secure. Strong data governance practices are also essential in maintaining data quality and compliance while using AI technologies.

Quality Data is Non-Negotiable

Among all the exciting new innovations showcased at Big Data LDN, the constant thread running through all of these developments remains data itself. It all depends on how good your data is. Joe Reis’ session on “Mixed Model Arts” was one of my highlights, solidifying this message: “get the data right, then the rest is easy”.

Data Mesh: Take What Works for You

Joe Reis was also a panellist on the “Everything you wanted to know about data engineering but were afraid to ask” debate. The first question prompted a long debate around the merits of the data mesh.

This is a topic of contention, with some polarising views on its place and value within organisations, however, all panellists ultimately agreed in their conclusion. A quote that stood out for me, was the reasoning that organisations should use data mesh as a “North Star” of modern data architectures, while agreeing with other views that likened it to the Agile movement of the early 2000s. Data mesh, like Agile, a framework that requires significant cultural and organisational change to succeed, has a number of key principles that are valuable, but following the approach step-by-step doesn’t work for every organisation. In fact, this method will work for very few organisations. Instead, organisations should focus on picking the principles that work best for them, and embed this into their existing data strategy to reap the benefits of the mesh.

BI meets GenAI

The rise of generative business intelligence, or GenBI, was a key point of discussion across Big Data LDN this year. This combines GenAI and BI capabilities to automate and enhance the process of analysing data and generating insights. Rather than relying on prebuilt dashboards and reports, GenBI uses generative models to create natural language responses to an organisation’s data queries. These models can also use Retrieval-Augmented Generation (RAG) to pull external data (eg websites and knowledge bases) into the pre-trained LLM to deepen its context and present better-informed responses to queries. On the topic of RAG, a big shoutout to James Kinley from Redpanda, who attempted a live demo demonstrating how you can use RAG in Redpanda streaming pipelines to improve the accuracy of GPT.

This is exciting stuff, and this real-time, automated BI can take decision-making to a new level.

Treat Data Like a Product to Drive Governance

Another session I found valuable was Benny Benford’s (Founder of Datent), “Data Transformation Unlocked: A CDO’s Toolkit for Measurable Success”. This was full of gems on:

• Successfully transforming your data strategy
• The importance of organisations having a growth mindset
• Representing how individuals use data and using this as a driver for change
• Treating data as a product
• Data governance.

The challenge of data governance stood out for me. Often seen as a necessary evil, the importance of data governance is much easier to sell when data is already commoditised and valued within an organisation. When teams are extracting real value from their data, they naturally start caring more about its quality and governance, having felt the pain of poor data.

Until Next Year, Big Data LDN

Big Data LDN 2024 was a great way of keeping in the loop with new ideas, strategies and important conversations in data, analytics and AI. It’s clear that GenAI is here to stay, but to leverage its power we need to be conscious of the quality of our data, the ethical considerations and the costs. We also need to carefully consider how new data architectures like data mesh fit into our organisations without being swept up by the latest trends.

As always, the core of everything is data itself. The performance of the tools, the models, the innovations, all come back to data. How we handle this will determine how successfully we navigate the age of AI and data-driven insights.

Joe is a Principal Data Engineer at Hypercube, where we specialise in data, AI and analytics for the energy sector. Contact us to find out how we can help you realise your data and AI project.