HomeOperationsAutomated OperationsWhy the Pipeline is key for Cloud Engineers

Why the Pipeline is key for Cloud Engineers

CI/CD pipelines are a common workflow automation tool for software developers. These pipelines generally build, integrate, and deploy new software versions, while doing code quality checks, security checks, integration tests, and other types of tests.

These automated pipelines help developers go through the integration and delivery process with ease and consistency while executing a battery of tests to increase code quality, security, and performance. Pipelines remove the heavy lifting, so developers are able to release smaller and smaller batches of code to production, increasing the overall stability of a production system by decreasing the time-to-fix if issues occur.

And I hear you thinking: great, I already knew this. I bet you have pipelines set up for most of your applications already, are working to improve them, and ensure that the only way to go to production is through a pipeline. And in this post, we’re not talking about making a software engineer happy with CI/CD. Instead, let’s talk about the Cloud side of things and why the pipeline is key for Cloud engineers.

Source: GitLab
Source: GitLab
This is post 2 of 4 in the Amazic World series sponsored by GitLab
1. Toolchain tax: just the tip of the iceberg?
2. Why the Pipeline is key for Cloud Engineers
3. Changing the Security Paradigm – Why you must shift left
4. GitOps is more than Git (or Ops)

Applying CI/CD to cloud

We’re going to talk about why cloud engineers need to be an integral part of the developer’s pipelines.

In the old days of running your own infrastructure, Ops teams created and pre-provisioned the infrastructure, VM templates, networking policies and storage buckets for developers to consume. In the most ideal situation, developers could choose a-la-carte what infrastructure to use for their applications. More often though, the options were outdated, were incompatible or not the right fit, and it took days or weeks to actually approve an infrastructure request and provision the infrastructure.

Cloud changed everything. From the seemingly endless VM instance types and a vast selection of cloud services for databases and other crucial infrastructure components, software developers suddenly had access to a Michelin-star-level infrastructure. Coupled with the on-demand and self-service aspects of the services public cloud vendors have to offer, the cloud massively shifted how infrastructure is being requested, provisioned and consumed.

Developers now have access and complete control over the infrastructure they provision for their applications, which is great for reducing their dependency on Ops and Cloud Platform teams, but not necessarily great for increasing resilience, stability and performance and decreasing cost. Simply put, Software developers aren’t infrastructure (or Cloud) experts, nor is it their immediate responsibility to work on these non-functional aspects. That’s where the infrastructure and Ops-minded engineers come in. Their experience and skill set is very much aimed at these non-functional aspects.

But without a deep understanding and effective collaboration with an application development team, results will be suboptimal. These two groups need to work together to create a cost-effective, resilient and performant infrastructure that is just right for each application.

The benefits of single application CICD whitepaper

The benefits of single application CI/CD

This ebook shows how a single application optimizes workflows so teams can get the most out of their CI/CD.


Download the ebook to learn how you can utilize CI/CD without the costly integrations or plug-in maintenance.


Download for free.

Shared language creates optimal results

And the best way to collaborate is by sharing the same vocabulary. In engineering terms, that often means using the same way-of-work, the same tools and the same workflows.

For infrastructure, that means taking an infrastructure-as-code approach, which applies the concept of CI/CD tools not to software development, but to cloud infrastructure. By managing and provisioning cloud infrastructure resources through code in a version control system, every infrastructure’s configuration and subsequent changes become visible for software developers, making it easier for them to see what’s going on and make (or request) changes.

And cloud infrastructure makes it possible to integrate an application’s infrastructure configuration completely into that application’s CI/CD pipeline and version control.

By adding specific tests and steps to the application CI/CD pipeline, DevOps engineers can make sure the developer’s code works on the specific infrastructure while maintaining their security posture, performance, cost and resilience for each release.

Replacing the Change Advisory Board with a pipeline

And on a technical level, this is great! Making infrastructure an integral part of the software development lifecycle makes sense, and between the on-demand and self-service Cloud service, Infrastructure-as-code tools (like HashiCorp’s Terraform) and CI/CD vendors like GitLab, we have the technical ability to make this happen.

why the pipeline is key for Cloud engineers
Source: https://midascreative.co.uk/change-in-web-design-old-sites-vs-new/

But organizationally, the potential gains are even bigger. Who doesn’t hate the manual and murky Change Advisory Board processes that changes have to go through to be approved?

Well, integrating infrastructure changes into a trusted pipeline can replace the Change Advisory board, by expressing the manual review that the CAB normally does as a set of codified tests in the pipeline.

Not only does this help with the throughput of the CAB (going from weeks to minutes in lead time, it also makes every change for which there is an automated test a standard change, reducing the workload for the CAB and letting them focus on the tougher, more complex issues.

Now, if this sounds a little vague, the best way to get started is by asking the Change Advisory Board members for ‘toil’ in their review process.

Toil is the kind of work that is manual, repetitive (done for each change), automatable (or easily expressed as an automated test) and specific. In our infrastructure world, the CAB looks at instance type, cost, initial and ongoing scaling, security configuration (like load balancing and changes to internet-facing services) and compliance with security policies (like RBAC), risk associated with the change (for instance, does this change potentially impact an adjacent but core application?), other running or requested changes and availability for SRE teams (to jump in when issues arise).

While not all of these are necessarily easy to express as code, each question does have a kernel of machine-expressible logic that can be written as a test. Start by finding those nuggets, and add tests each time the CAB reviews and approves changes.

That way, test coverage is increased gradually over time, which frees up the CAB to deal with the more complex changes at hand, and frees up DevOps engineers to continuously improve their CI/CD pipelines.

In order to take full advantage of CI/CD when replacing the Change Advisory Board, DevOps engineers should minimize toolchain complexity; traditional CI/CD doesn’t work well in this scenario. While we won’t go into details in this post, I recommend reading ‘the benefits of single-application CI/CD’ to find out more about why the pipeline is key for Cloud engineers.


Receive our top stories directly in your inbox!

Sign up for our Newsletters