Vault reaches a milestone as HashiCorp releases Vault 1.4

HashiCorp Vault has just crossed a milestone and released their latest and greatest, version 1.4. It might seem a little odd to have such a huhar about a dot revision, but the features included in this release don’t lie.

What is HashiCorp Vault?

Vault is Hashicorp’s tool for securing and accessing secrets. Secrets are anything that needs access to controlled and audited. For example, API Keys, passwords, or certificates. It also provides a detailed audit log of who accessed what, when, and by whom.

Vault allows the creation of dynamic one-time-use access tokens and credentials which significantly improve the security of your environments.

Modern systems require access to a plethora of secrets; database credentials, API keys for external services, credentials for accessing cloud services (like we did when creating EC2 VMs on AWS), and securing service-to-service communication. Vault fulfills these functions perfectly, has cross-platform capabilities and advanced features like key rolling, secure storage, and much more:

• Secure Secret Storage – anything commited to storage is encrypted prior to writing, so gaining physical access to the store is not enough to gain access to any secrets stored therein.
• Dynamic Secrets – Secrets can be dynamically created on demand. When a user requests access, the request is forwarded to Vault for authorization and then Vault creates an key-pass with the desired permissions, valid for a single use, that is also valid for a set amount of time. Vault will automatically revoke the access once the lease expires.
• Data Encryption – Vault can be used to automatically encrypt and decrypt data without storing it, this allows encrypted data to be stored within otherwise unencrypted stores, like a general purpose SQL database.
• Leasing and Renewal – Vault will automatically revoke secrets at the end of their lease, and renew them via a built-in Renew API.
• Revocation – Vault as support for secret revocation. This is not just revocation of a single secret, for example a certificate or a user, but for trees of secrets. This is especially useful when employees leave the company. The revocation of their user account access will also trickle down to revoke all other system access, and any personal certificates. Revocation can also be used to lock down systems that have been compromised.

As we can see it really is a well-featured solution. So how have HashiCorp improved on this with the release of version 1.4. Let’s look at what’s new.

What’s new in HashiCorp Vault 1.4?

This article will now concentrate on the new features of HashiCorp Vault 1.4 starting with those that are available to the users of the free OSS version and then the new features of their enterprise version.

Open Source

• Integrated Storage: In the past, Vault relied exclusively on external storage, like a Consul cluster. With 1.4, Vault offers integrated storage, removing the external requirement for new deployments.
• Vault Helm Chart: Run Vault Enterprise via the Helm Chart hosting on Kubernetes.
• OpenLDAP Secrets Engine: Support the management of static entities within OpenLDAP.
• Kerberos Authentication Method: Supports authentication of users and applications via Kerberos.

Let’s look a little deeper into the new features,

Vault Integrated Storage

Historically if you needed a resilient Vault environment you needed to deploy a significant amount of infrastructure to support it, a minimum of three Vault servers and an external 3- or even 5-node Consul cluster. This is not a simple process.

With the release of version 1.4 HashiCorp have Integrated Storage as a storage engine built into Vault, removing the requirement for configuring and managing external storage back-ends or services. This simplifies the deployment and ongoing operation of production Vault clusters. This feature is available to the users of both the open-source and enterprise workloads.

Replication and the maintenance of consistency of the secrets between the nodes is handled by the Raft library, which is incidentally also used as the arbiter in their Consul product.

Even better they have also published an Integrated Storage Reference Architecture, a Preflight Checklist for Migrating to Integrated Storage, along with a Migration Guide for moving from HashiCorp Consul to Integrated Storage. Not too bad for day one release.

Vault Helm Chart

With previous versions the official Helm Chart for Vault was only a feature of the enterprise version this has now been expanded to support both open-source and enterprise workloads. By using the Helm Chart, Vault clusters running on Kubernetes can be built out in minutes rather than having to deploy the based server, be that virtual or physical, deploy your OS and pre-req libraries and tools. Helm chart will also be the primary mechanism for setting up future Vault and Kubernetes features.

When using Helm chart, the complexity of a Vault deployment can be greatly reduced when deploying on a Kubernetes platform, another side benefit is that this is a repeatable deployment process which deployed in less time, meaning a quicker value return.

Vault OpenLDAP Secrets Engine

OpenLDAP is a common authentication method similar to Active Directory. With the introduction of the OpenLDAP secret engine, this opensource Identity provider can be enhanced by the addition of rotation credentials across OpenLDAP directories.

The OpenLDAP Secrets Engine is a new secrets engine that allows Vault to manage existing OpenLDAP entities for activities such as rotating credentials across OpenLDAP directories for privileged access management (PAM) workflows.

Adding the OpenLDAP secrets engine to Vault will provide a variety of authentication methods to users for accessing their own LDAP credentials. Your users can then be empowered to manage their own LDAP entries, and their passwords can be configured to be automatically rotated based on a time to live value configured by the administrator.

The obvious by-product is that by making the passwords as short-lived as possible, there is a marked reduction of the chance that a credential is compromised. Additionally if a credential is compromised, it can be revoked and rotated rather than having to change a global set of credentials. HashiCorp’s OpenLDAP Secrets Engine is compliant with OpenLDAP v2.4.

Vault Kerberos Authentication Method

The Kerberos Authentication Method is a new feature in Vault that allows Vault to verify applications and users via an existing Kerberos or SPNEGO environment. With Kerberos authentication, Vault communicates with an external Kerberos system to verify the claims in a Kerberos ticket.

This means that Kerberos will also benefits from the added benefits of rotating tickets, and AD with has been integrated with Vault can not take advantage of using Kerberos rather then having to default to NTLM

Enterprise

Now we will start to look at the new features of Enterprise Vault 1.4, this is their costed version.

This release includes the following features:

• Transform Secrets Engine: Transform Secrets Engine performs secure data transformation for protecting secrets that reside in untrusted or semi-trusted systems outside of Vault.
• NetApp Enterprise Key Management Support: Support enterprise key management for NetApp Full Disk Encryption (FDE) and Volume Level Encryption via the KMIP Secrets Engine.
• Improved Disaster Recovery (DR) Workflow: Support improved workflow for promoting a DR Secondary should the DR Primary be lost.

The first two feature we will be looking at are a part of the Enterprise version’s ADP Module

Vault Transform Secrets Engine

The Transform Secrets Engine is a major new addition to the Vault Enterprise Advanced Data Protection (ADP) module, which allows a Vault deployment to protect secrets that reside in untrusted or semi-trusted systems outside of Vault. This include things such as PPI for example, social security numbers, credit card numbers, as well as other types of compliance-regulated data that reside within systems such as a file system or database but must be protected in the event of their residence system’s compromise. But didn’t we already have this with the Transit secret engine? Yes and no, true data was encrypted as shown below:

Now although it would satisfy an audit in that the inputted PPI was encrypted, it could cause validation errors on the receiving database due to the schema expecting data of a specific type, length or format, also preservation of data format may be a requirement of the overriding industry compliance standard.

The Transform Secret Engine was built to solve this issue and prevent a situation where there needed to be an exception to a compliance stance or a major rewrite of the back-end storage device. The feature supports both one-way masking (this is not recoverable) therefore only the encode option is available in the command

and two-way transformations via data type protection, which allows the resultant change to be reverted to show the original data, as such this option supports both encode and decode options.

The Transform Secrets Engine allows Vault to resolve use cases typically addressed by tokenization, with high-performance cryptography and the full suite of the Vault platform’s high availability and security features.

NetApp Enterprise Key Management Support

Vault Enterprise is now certified to automate key management for NetApp Data ONTAP systems. This allows Vault to serve as an external key manager (KMIP) for Data ONTAP, allowing for Vault to protect keys for full disk encryption (or FDE) via NetApp Storage Encryption and at the volume-level via Volume Level Encryption.

Improved Disaster Recovery (DR) Workflow for Vault

This is a Vault Enterprise-only feature, that said I sincerely hope that this feature flows down to the open source solution. Prior to Vault 1.4, in the event of a Disaster Recovery (DR) Primary cluster going down, promoting the DR Secondary cluster required the generation of a DR operation token which was created by a quorum of recovery/unseal keys. The Requirement for this quorum to be carried out during the critical period that downtime, can and did cause delays in rectifying the resultant outage. Vault now supports the creation of a DR Batch Token which, when created, will allow the promotion of a DR Secondary without the need for the quorum key-holders, thereby saving precious time during critical downtime.

It seems such a little thing but it will streamline the recovery process which is always a plus point when you have users and services that can no longer authenticate.

Other new Vault Features

This release also includes a number of additional new features, secure workflow enhancements, general improvements, and obviously bug fixes, and I am sure a number of new bugs, but we will need to wait and see on that one, a full list in the HashiCorp Vault 1.4 changelog.

Some of the more prominent ones are summarized below:

• MongoDB Atlas Secrets: Generate dynamic credentials for both MongoDB Atlas databases and the Atlas programmatic interface.
• Redshift Database Secrets Engine: This has been enhanced to support both static and dynamic secrets for the Amazon Web Services (AWS) Redshift service.
• Venafi Secrets Engine: Ability to dynamically generate SSL/TLS certificates that serve as machine identities.
• Service Registration Config: A newly introduced service_registration configuration stanza, that allows for service registration to be configured separately from the storage backend.
• Kubernetes Service Discovery: Kubernetes service discovery feature where, if configured, Vault will tag Vault pods with their current status.
• Usage Monitoring: UI controls were added to the web interface for monitoring running count metrics of HTTP Requests, Entities, and Tokens in Vault.

Summary

HashiCorp Vault 1.4 is a significant release with plenty of new functionality and it is worth upgrading just for the integrated storage and the benefits that this simplification will bring to your environment and overall cloud bill, add into this the benefits of Helm chart being added to the Opensource environment and we recommend that you start to plan your upgrade. For those on the Enterprise version the addition of the Transformation engine can further expand your use of Vault into those legacy areas that you de-scoped due to the issues that the transit engine brought to the party. Those that have NetApp storage now have the capability to use Vault as a KMIP to manage their easier rotation of filer certificates.