Those of you who have been following our Terraform and Vault technical posts will have seen the benefits of using them for point solutions. Today we will investigate how to use the HashiCorp full-stack.
Hashicorp has a portfolio of products that can be used to seamlessly create a CI/CD workflow for on-prem and cloud workloads. The products are Terraform, Vault, Nomad, Consul, Vagrant and Packer.
HashiCorp Packer
Packer, as its name suggests, is a tool to build virtual machines from start to application delivery. It uses JSON or HCL files as configuration input and has the concept of Builders (the environment the build is to be carried out against) and provisioners (the actions that are to be carried out, network configuration, partition layout, configuration file upload, application installation, etc), to It is mainly used to build out test and development environments, it also can interact with Containers. To see Packer in action, have a read of this post where we build a VMware ESXi Centos Virtual Machine. Packer can be used to build your machines and containers and then used to move the builds into your desired environment as Templates (VMware, GCP, and Azure), AMI’s (AWS) for later consumption by Terraform.
We now move onto the enterprise products, Terraform, Vault, Consul and Nomad
The Enterprise Stack
Hashicorp has a loosely coupled set of products that can be used by the DevOps team to build out very complicated environments against multiple targets including AWS, GCP, Azure, VMware both locally and in the cloud. The graphic above highlights the interactions between each product layer and how they fit into the stack. By utilizing these products DevOps teams can deploy environments from “tin to skin;” keep applications updated, environments synchronized from Development through to Production.
HashiCorp Terraform: Infrastructure as Code
Terraform is HashiCorp’s Infrastructure as Code (IaC) offering, it is a fully-featured solution that uses HCL (Hashicorp Configuration Language) to provide inputs; according to HashiCorp’s HCL Github repository it is heavily inspired by libucl, Nginx and has a similar structure to JSON; which incidentally can be used as an alternative input method. HCL is a declarative language designed to be easily understandable by both humans and machines.
For an explanation of what a Declarative language is have a review of this article. In the context of Terraform, the language is used to define an infrastructure for automated deployment. Terraform can build out anything from a simple virtual machine to extremely complicated environments across multiple endpoints such as AWS, GCP, Azure, VMware, Oracle Cloud and Alibaba. It can be used to deploy virtual machines, traditional containers, Kubernetes, and serverless functions like Lamba. It can be used to install and configure applications and operating systems. It can deploy pre-configured templates and operationalize them.
HashiCorp Vault – protect your crown jewels
Vault is Hashicorp’s bank. It protects the crown jewels of Identity and Access. Vault can be used as a simple secret store, or as a pseudo-identity provider taking credentials from a provider, anonymizing them and providing a one-time-use user credential and tokens to allow secure and protected access to multiple providers, including AWS, Azure, GCP, vSphere and many other platforms and applications.
HashiCorp Consul: the glue to bind them all
Earlier in this article, we mentioned that Vault can utilize Consul to provide a shared storage environment for high availability are resilience. Consul provides much more than a simple clustered and available storage solution. It is more of a service mesh.
What is a service mesh?
A service mesh is a way to control how different parts of an application share data with one another. Unlike other systems for managing this communication, a service mesh is a dedicated infrastructure layer built right into an app. This infrastructure layer can document how different parts of an application interact, it, therefore, becomes easier to optimize communication and avoid downtime as application requirements grow.
Service Discovery
This is the ability to automatically add new instances of services into the service mesh for consumption by applications, operating systems and containers.
Key Vault
Consul KV is a core feature of Consul and is installed with the Consul agent. Once installed with the agent, it will have sane defaults. Consul KV allows users to store indexed objects, though its main uses are storing configuration parameters and metadata. This is then available to any consul agent or server or consuming service.
Nomad – Application configuration
HashiCorp Nomad is a product that reduces the complexity of automating, scheduling, and rescheduling application deployment, it allows operations engineers and developers to work together more closely, and improve DevOps workflow by providing a layer between the Operating system and the application stack, it can be used to automate the operation tasks, scheduling and transparently removing services from a to be upgraded node, be that a container or a traditional application.
Pulling it all together
At first glance these independent products do not seem to be complementary, however, lets work through them and their use-cases. As already stated Packer can be used to build custom templates. And uploaded to your desired deployment target in a later post we will show how to do this, but the following code below utilized in your config file in a stanza titled “post-processors” coupled with an “export” tag in your builders stanza will take your deployed object and move it to, in our case AWS, your desired target deployment
To be added to the Builders stanza in the Packer JSON file
"shutdown_command": "sudo -S shutdown -P now",
"output_directory": "output-{{user `profile`}}",
"format": "ova"
To be added as a new stanza in the Packer JSON file
"post-processors": [
{
"type": "amazon-import",
"access_key": "AWS Access Code Here",
"secret_key": "AWS Secret Key",
"region": "eu-west-2",
"s3_bucket_name": "centos7packer-template",
"license_type": "BYOL",
"tags": {
"Description": "packer amazon import "
}
}
]
The beauty of this is that you can define your own standards and confirm that they meet your audit and compliance needs. This coupled with the knowledge that you fully know the efficacy of the build process down to individual libraries means that you are fully confident of the base builds.
Also as this is now an AMI in your AWS environment these can now be used as the input for your terraform deploys,
We have already seen Vault and Terraform being used together to provide one-time use passwords for AWS deployment, Vault can provide the same function for GCP, vSphere and multiple other endpoints and applications. Vault can also integrate these same services into Nomad application configuration and management to provide seamless access and control.
Nomad sits on the top, managing the application stack. The beauty of this is that the Nomad Agent can be installed either as part of your packer build or more likely due to environmental reasons during your Terraform deployment. Nomad excels at application management, Nomad can update your applications out of cycle. Their cadence will be quicker than the changes on the underlying infrastructure stack. Also when you deploy a Terraform deployment it will destroy what is already there before creating a new instance. This of course introduces another risk, if you have used Packer or Terraform to install your application stack and you use Nomad to manage the update and patching cycle you run the risk of deploying legacy application code into your environment. By moving Application deployment and Management into Nomad and therefore its own stack this risk is removed, and application cadence can be separately managed. Now your deployment cycle would be Packer -> Terraform -> Nomad.
You will have noticed that Consul has not yet been discussed in this scenario, this is because Consul is the glue that is used to manage the workflow. The Consul Service is a distributed service mesh and partakes in Service Discovery and as such has hooks at many levels; for example Consul can provide DNS and DHCP distributed across several datacenters and cloud regions and availability zones. Consul also has a Key Management Store, this allows for the centralization of declared entities allowing for their distribution across the distributed platform. This obviously aids significantly in producing agnostic deployment capability which is capable of being deployed to multiple disparate endpoints.
Summary
Hopefully this article has shown the power of Hashicorp’s products to create a fully functional “tin to skin” deployment and management capability, which is functional across disparate platforms. HashiCorp is building out a solid DevOps delivery platform that is functional across the whole stack. Yes you will need other tools to fill holes, but a large amount of your work can be done within a single vendor cutting down as the need for understanding multiple products that effectually undertake the same role.
