Meet Pulumi: infrastructure management with your favorite programming language

Since Infrastructure as Code is now the defacto standard to set up and maintain infrastructure, the number of tools to do this and frameworks grows. Perhaps all organizations know CloudFormation for AWS or HashiCorp Terraform to deploy resources in a multi-cloud environment. These solutions require learning a scripting language, which increases the learning curve. Terraform scripts are written in the HashiCorp Configuration Language (HCL) whereas CloudFormation templates are written in the JSON or YAML format. Pulumi is a relative newcomer in the IaC space and lets you write your infrastructure scripts in your favorite programming language.

What is Pulumi?

Modern infrastructure is not just a set of static of VMs. It is an ever-changing landscape of virtual machines, containers, serverless functions, 3rd party APIs and more.

To manage infrastructure components properly, you need to be aware of different aspects similar to traditional application components. Think of isolated components (like microservices), lifecycle management of them, a branching strategy, etc. And what about re-usability or loose coupling of generic components? While a lot of other popular Infrastructure as Code solutions stop right after the initial provisioning of resources, Pulumi goes on.

Pulumi aims to handle infrastructure like common software applications: use your favorite programming language in your IDE of choice. With this in mind, you can set up a decent architecture for your infrastructure scripts. For example: Pulumi version 2 now supports integration tests and policies to utilize best practices in terms of cost management or security. The gap between your source code in Git and the actual resources narrows. Shifting left and GitOps get a new dimension.

Shift left

It all starts with the developers. Students at universities do not learn to write big JSON files or hard-to-maintain YAML templates. They learn to code in “real programming” languages. At present, Pulumi supports the following languages: Go, Python, TypeScript, JavaScript and C#. Java is on the roadmap.

You can use the same constructs like classes, functions, loops, etc. This reduces the so-called “boilerplate code” and focus on the actual infrastructure logic. Developers do not need to switch to a new IDE or install additional plugins; this frees the way to do similar things when you develop your software components: search for code, refactor it, perform integration tests, lint your code, etc. All of this lowers the barrier for developers setting up infrastructure themselves and removes dependencies on the Ops team to write these scripts for them: Dev and Ops are blended even more.

Deployment of infrastructure resources becomes almost similar to application development. Pulumi supports infrastructure changes from end-to-end: Pull Requests can be reviewed and annotated, add comments before you merge. And when you do actually press the button to deploy, you get insights into who did what and what is the impact of the change was. Deployments can be traced back to the original commit hash. With this, the end-to-end visibility increases, resulting in faster feedback loops. From a compliance perspective, you get your audit trails.

Package and reuse

Writing code for infrastructure should follow architectural principles, it should not just describe individual instances of intended resources. Reproduction of infrastructure is important as well as re-usability.

If not, every single DevOps team ends up recreating the same infrastructure code and reinventing the wheel in other places, producing more boilerplate code. Maintenance becomes harder, bugs need to be fixed in many different places simultaneously, etc. DevOps will be slowed down, the risks for the organization increases.

Pulumi supports the packaging of components. With this, you can test things out in isolation and distribute it across different departments. Furthermore, multiple DevOps teams can use upstream features.

It’s a bit similar to Maven in the Java world. Maven is like a package manager for Java-based components. Maven uses so-called “pom files” to describe dependencies with their versions and it also supports hierarchies. With Pulumi you can do a similar thing. Every package has an owner, independent lifecycle and version. Given these, governance of components becomes a lot easier. Furthermore, it increases visibility across the organization about who develops what and thus preventing the reinvention of the wheel, which is a common problem in large organizations.

3 layers

Pulumi supports three layers to package your (reusable) components. In essence they become Pulumi resources. You can use one programming language for the CLI and deployment workflow of the components you want to create. Viewed from bottom to top, these layers are as follows:

• The foundation layer: support for all major cloud providers and the services they offer. Core providers include AWS, Azure, Google Cloud & Kubernetes (yep, Kubernetes is seen as a “core provider” on the same level). These resource providers are loaded through the use of “plugins” to the package you create. It is important to mention that Pulimi simplifies to build similar infrastructure components for multiple providers. This greatly benefits the organizations which rely on multiple cloud providers.
• The patterns layer: cloud-specific libraries for common patterns and practices. It consists of categories like containers, serverless architecture or “raw infrastructure”. For example, creating components on this layer gives you the option to create a Virtual Network for any of the core cloud providers (excluding Kubernetes) or to create a Docker registry.
• The framework layer: support for different programming languages and best practices to use it. Using this, you will utilize the expertise of the Pulumi creators to apply frameworks on a higher level. Those frameworks in turn, use the other layers to define and handle infrastructure resources. This is a more abstract view on cloud resources.

Pulumi practitioners can pick the level of choice to handle infrastructure or mix and match them. You can use the packages of your native package manager (e.g. NPM for Node.js, Pip for Pythin, etc). When creating infra resources by Pulumi, these packages call the Pulumi SDK and this SDK determines how to communicate to the Pulumi engine. Working with packages is almost similar to how you know it already. The handling of  these resources is the only difference of your native package manager and the way Pulumi registers and maintain them. Given this approach, Pulumi tries to blend in here as much as possible to your existing programming language.

Policies & security

Pulumi supports policies to control what can be deployed and what should be blocked in the enterprise version. For example: have a policy in place that checks a specific version of MySQL of an RDS instance in AWS. Policies can be written in the language of choice and these can also be packed into “policy packs”.

In addition, Policy Packs can be signed to make sure they are not tampered with while they progress in the CI/CD pipeline. Policies should come from a trusted place, this can also be enforced.

In the enterprise version, you can separate the creation and enforcement of policies. Developers can deploy their infrastructure resources, but only security engineers can define the run-policies which enforce the validation of the deployments. Role Based Access Control helps you here.

The enterprise version integrates with SAML SSO and identity systems like Azure Active Directory (AAD), GitHub or Okta. All of this makes sure your infrastructure is set up according the central authentication and authorization system and security controls of your organization.

A common approach

Software development teams should use standards and guidelines as much as possible to guarantee a common way of working. It’s not just the development phase: this also applies to architecture, testing, policies, and infrastructure management. Standardization and using a single approach for all of these aspects is important to minimize duplication of work across teams.

Suppose a development team needs to deliver a new micro-service: It starts with the right application architecture. From here, the Ops team can create “infrastructure packages”. Following is the security team which can create the right policies to enforce certain controls. And right after that, they deliver it back to the developers which can implement the business logic. Collaborating like this helps to break down the silos between these teams.

Developers that use this methodology are empowered and are not hindered by organizational obstacles. From this perspective, the scope of the micro-service is not just the application (components) itself but all of the other aspects which are needed to actually run it in a secure and compliant way.

How to start

Pulumi works on the client-side alongside your other programming languages.

It supports MacOS, Windows and Linux.

• For Linux you need to run curl -fsSL https://get.pulumi.com | sh from the command line. Or you can install the version of choice manually.
• Windows users can use the Chocolatey package manger to install it. Run choco install pulumi to install it.
• MacOS users fire up HomeBrew to install it: brew install pulumi.

From here on you need to setup a new project:

pulumi new kubernetes-python

A project is like a Git repository. It consists of 1 or more stacks. For example a stack can be like a DEV or TEST environment. All is ready now to create your actual resources.

Kubernetes example

Let’s make this more practical by provisioning a Kubernetes cluster based on EKS in AWS. Using this cluster, deploy an application on top of it. Pulumi can handle the infrastructure part of Kubernetes or the resources inside the cluster itself.

Create the Kubernetes cluster

First, you need to create a directory and within it, a Pulumi project:

mkdir eks-hello-world && cd eks-hello-world

pulumi new aws-typescript

The last command creates a so called “stack”. The stack is used to differentiate an environment or development phases. Let’s call the stack “development” in our example. We will use the TypeScript programming language in this example.

Install the required dependencies:

npm install –save @pulumi/eks @pulumi/kubernetes

One of the files being created is the index.ts file. It serves as the main entry-point in the Pulumi program. An example of this file is displayed below:

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

const name = "helloworld";

// Create an EKS cluster with non-default configuration
const vpc = new awsx.ec2.Vpc("vpc", { subnets: [{ type: "public" }] });
const cluster = new eks.Cluster(name, {
vpcId: vpc.id,
subnetIds: vpc.publicSubnetIds,
desiredCapacity: 2,
minSize: 1,
maxSize: 2,
storageClasses: "gp2",
deployDashboard: false,
});

// Export the clusters' kubeconfig.
export const kubeconfig = cluster.kubeconfig

In essence it consists of the resources which needs to be setup for EKS (e.g. the Virtual Private Network & it’s subnets, the Kubernetes cluster itself). Besides this, it specifies whether or not to install the default Kubernetes dashboard and the number of worker nodes to host the Kubernetes components (e.g. Pods, Deployments).

Continue to run:

pulumi up

which gives you a preview of what will being created. Answer with “yes” to actually create the resources itself. This is similar to Terraform.

You are now ready to deploy a simple sample application in the cluster.

Deploy an nginx webserver on Kubernetes

Pulumi uses your Kubeconfig file to connect to your cluster and to maintain any resources within the cluster.

A simple script of an Nginx (webserver) container in it’s own namespace looks like this:

// Create a Kubernetes Namespace
const ns = new k8s.core.v1.Namespace(name, {}, { provider: cluster.provider });

// Export the Namespace name
export const namespaceName = ns.metadata.apply(m => m.name);

// Create a NGINX Deployment
const appLabels = { appClass: name };
const deployment = new k8s.apps.v1.Deployment(name,
    {
        metadata: {
            namespace: namespaceName,
            labels: appLabels,
        },
        spec: {
            replicas: 1,
            selector: { matchLabels: appLabels },
            template: {
                metadata: {
                    labels: appLabels,
                },
                spec: {
                    containers: [
                        {
                            name: name,
                            image: "nginx:latest",
                            ports: [{ name: "http", containerPort: 80 }]
                        }
                    ],
                }
            }
        },
    },
    {
        provider: cluster.provider,
    }
);

// Export the Deployment name
export const deploymentName = deployment.metadata.apply(m => m.name);

// Create a LoadBalancer Service for the NGINX Deployment
const service = new k8s.core.v1.Service(name,
    {
        metadata: {
            labels: appLabels,
            namespace: namespaceName,
        },
        spec: {
            type: "LoadBalancer",
            ports: [{ port: 80, targetPort: "http" }],
            selector: appLabels,
        },
    },
    {
        provider: cluster.provider,
    }
);

// Export the Service name and public LoadBalancer Endpoint
export const serviceName = service.metadata.apply(m => m.name);
export const serviceHostname = service.status.apply(s => s.loadBalancer.ingress[0].hostname)

Append these lines of code to your index.ts file. Pulumi exports the name of your service and the hostname to reference it later on.

All is ready now to deploy your nginx webserver

• pulumi up

Just like Terraform you will see a review of your intended deployment. You will be prompted to proceed. Once you do so, Pulumi creates the intended Kubernetes resources.

Use the following commands to use KubeCTL to view the Kubernetes resouces:

pulumi stack output kubeconfig > kubeconfig

export KUBECONFIG=`pwd`/kubeconfig

export KUBERNETES_VERSION=1.11.5 && sudo curl -s -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v${KUBERNETES_VERSION}/bin/linux/amd64/kubectl && sudo chmod +x /usr/local/bin/kubectl

KubeCTL is ready now to query your cluster (e.g. kubectl get pods, kubectl get deployments). From here it should be familiar to you 🙂

Conclusion

Pulumi utilizes the same concepts and patterns as real programming languages to manage infrastructure in a multi-cloud environment. It aims to break down the walls between Dev, Ops, and Security teams by utilizing a single approach for these disciplines. This article provided an introduction to Pulumi and it gave you some pointers to start with it. I hope you like it as much as I do and you will try it out in a future project.