From clicks to code: the rise of Infrastructure as Code (IaC)

⸻ Beyond Energy

Principal Data Engineer Edwin Lizarazo explores the evolution of Infrastructure as Code (IaC).

The data landscape is changing: managing workflows and infrastructure through graphical user interfaces (GUIs) is no longer sufficient, particularly when working across multiple environments or teams. While user interfaces such as those provided by Databricks can be convenient for quick, one-off configurations, they fall short when consistency, automation, and scalability become priorities.

Infrastructure as Code (IaC) offers a modern solution to these challenges by enabling teams to define and manage infrastructure through code and bringing the benefits of software engineering such as version control, reusability and automation to infrastructure management.

The challenges of UI-based workflow configuration

When deploying jobs in platforms like Databricks using the UI, those jobs remain confined to a single workspace. This introduces several limitations:

Lack of version control: Changes made through the UI are not tracked, making it impossible to audit or revert configurations.
Manual repetition: Each environment (development, testing, production) requires a separate, manual setup, which increases the risk of inconsistency and human error.
No automation: Without code, there is no opportunity to integrate deployments into CI/CD pipelines, slowing down delivery and increasing operational overhead.

As data platforms scale, these limitations become more pronounced, impacting both speed and reliability.

The role and purpose of IaC

IaC addresses these issues by allowing teams to define their infrastructure as machine-readable configuration files that include clusters, jobs, notebooks, and schedules using tools such as Terraform with the Databricks Terraform provider.

By codifying workflows into reusable modules, teams can:

Ensure consistency across environments: A single definition can be deployed to development, test, and production environments with identical behaviour.
Enable Git-based collaboration: Changes are tracked, reviewed, and approved via pull requests, ensuring transparency and accountability.
Automate deployments through CI/CD: Integration with pipelines (e.g., GitHub Actions, Azure DevOps) enables reliable, hands-off deployment.
Customise behaviour by environment: For instance, jobs can be set to run automatically in production but executed manually in development, offering a level of control that the UI does not provide.

These capabilities transform the management of infrastructure from a manual, error-prone task into a reliable, repeatable engineering process.

Why IaC is essential for success

The primary value of IaC is its ability to transform infrastructure management from an ad-hoc manual process into a formalised software-driven practice. It improves reliability, speeds up development cycles, and enforces governance, all essential for operating modern data platforms. Managing workflows in Databricks or orchestrating resources on AWS, Azure, or GCP, the message is clear:

Infrastructure belongs in code.

Next steps for data teams:

Assess current infrastructure managed manually via UIs.
Begin codifying simple workflows into Terraform or other IaC tools.
Establish Git-based workflows and CI/CD integration.
Scale modules and practices across environments and teams.

By embracing IaC, organisations unlock a foundation for agility, collaboration, and operational excellence. To keep in the loop with tech insights like these, join Hypercube’s Beyond Energy Community.