Where to store the secrets you need for your applications and infrastructure? A lot of sysadmins and developers face this dilemma. Not just once, but with every (new) application which is deployed in the on-prem datacenter or in the (public) cloud. HashiCorp Vault and other secret management tools are here to help.
Different types of secrets
A secret is anything you want to keep private and which you want right people or systems to access. Managing secrets deals not only with the secrets itself, but also with controlling the access to them across all users and systems.
But what kind of secrets we are talking about? Think of username & passwords, (security) tokens, public and private encryption keys and SSL certificates to name a few. There is no application I can think of which has no secrets at all. Even the most simple application like a guestbook which stores its data into a relational database like MySQL requires at least one secret: the password to access the database.
More advanced (enterprise grade) applications with a lot of different application components and integration points require a much more higher amount of secrets to protect.
For example a centralized CI server might require the following secrets:
- A central administrator account (username & password).
- When using a secure web front-end (HTTPS), it requires a certificate and a public & private key.
- When the CI server runs on an encrypted disk, you need to store the encryption and decryption keys.
- If the tool connects to a reporting application, it should also store the credentials to connect to this tool.
- If it hands off some work in the CI process to a testing tool like SonarQube, it needs to store the credentials for the API connection.
This list list is not complete, but it does give you an idea about the different secrets you need to protect. Also, you need to know who can and should access these secrets and how those are stored and protected.
A large scope
As you can see, even a single tool already has a lot of secrets. Consider working in a large enterprise environment with many employees using many tools like the example above. And think from the perspective of DevOps teams which actually create (new) custom applications for end users.
Almost every application uses some sort of a database. DevOps teams might store application database passwords in plain text in a source code management tool like Git. The same is true for private keys and certificates. Configuration files might end up at a server which stores passwords in clear text. If these files are not audited, everyone with enough permissions can access these files unnoticed and login to whatever system it gives access to.
When teams work highly autonomous and in ‘you build it, you run it’ environment, it quickly becomes a big challenge to manage all of these secrets. Since there is no centralized control, it’s difficult to keep sight of all the locations of the secrets being used.
The on-prem datacenter
In the “good old days” the operations team had to take care of the application infrastructure and to keep it secure. Developers were responsible to secure their applications. The line between those teams was clear.
A typical use case helps to explain this.
In the past an application was deployed on a single Virtual Machine. Only the system administrator was able to login to it. Maybe also the developers, but he/she was mainly concerned about deploying their application only, and were not given admin rights to the machine. They are no security specialists and they did not care so much about the protection of that Virtual Machine.
Given this scenario, the Virtual Machine had a fixed IP address. From a firewall point of view, it was fairly easy to protect it, since the IP address did not change (very often). Roughly speaking it was enough to control the access to this Virtual Machine and to protect the Virtual Machine itself.
The Virtual Machine itself was part of a “trusted datacenter” and all of the infrastructure components running inside the datacenter were considered “trusted”, since the company owned and controlled them. Infrastructure and applications were “static” and did not change so often.
The public cloud
In the (public) cloud, this has all changed.
Networks in the public cloud cannot be trusted as much as the networks in the on-prem datacenter. It becomes harder to determine which infrastructure components the organization itself controls and which the cloud provider controls. Think of the Virtual Private Cloud environments (isolated areas) of modern cloud providers. Where does the responsibility of the cloud provider ends and where does the responsibility of the organization start? This becomes even more complicated when taking into account the networks between multiple clouds.
Servers in the (public) cloud are not ours. Organizations hire infrastructure which a cloud provider maintains. The DevOps teams operating the cloud, most of the time, do not know the implementation details of the infrastructure anymore. Not a problem, but it creates new risks when it comes to handle secrets. Risks to manage to keep applications and its data secure.
Back to the example of the Virtual Machine which hosts the application. When practicing Infrastructure as Code, the infrastructure to host the application needs to be setup again whenever there is a new version. As a result the IP address of the Virtual Machine changes. Imagine you whitelisted this IP address in the corporate firewall to allow certain people and systems to access it. Changing the firewall every time the IP address changes is impossible. Dynamic infrastructure like this makes secrets even harder to manage. Enter secrets management for these use cases.
Secrets management
Organizations should know who creates, implements, accesses and revokes secrets to keep an eye on those. If not, it poses a risk since unauthorized persons, applications or other systems might reveal valuable data. Data should kept safe, secrets to access it should be stored in a safe and centralized place. These are just some of the advantages to have a proper secrets management solution in place.
Secrets management is all about the centralized management of secrets in order to prevent secrets from leaking. When a centralized secret management tool in place, it helps the organization to keep track of all of their secrets. There is clear visibility in how secrets are stored, accessed and audited.
Secrets management plays a crucial role in the protection of secrets and systems which are based on “dynamic infrastructure” like Virtual Machines in the cloud, containers and serverless solutions. Traditional management of sensitive information as we know it from the on-prem datacenter is not sufficient to handle all of these use cases.
Secret management tools
Secrets management tools help to make the challenge of handling and controlling secrets easier. A short list of popular secret management tools which exist today: Hashicorp Vault (vaultproject.io), CyberArk, AWS secrets manager and Azure Key Vault.
The main features of the most common secrets management tools are:
- Central storage and management of all kinds of secrets (centralized key management)
- Provide a simple APIs to encrypt the data
- Managing identities: who or what can access secrets and for how long (e.g. life-cycle management of tokens)
- Being able to control, log and audit persons and systems accessing the secrets
- Rotating (dynamic) secrets: don’t use a fixed secret all the time, but “refresh” it based upon a given time interval.
When using these tools, it becomes more easy to handle secrets since the management is done from a centralized point of view. All the DevOps teams can use these tools to protect their applications and infrastructure components.
The AWS and Azure cloud providers offer these features as a service, so from a technical point of view, they maintain it for you. A lot of (big) cloud providers use very strong mechanisms to keep their cloud native services secure, this is especially important when using a secrets management they offer. All in all, your secrets management tool should never ever be compromised.
Conclusion
Every organization uses a lot of secrets and many of them struggle to handle those properly. When working in the (public) cloud, this becomes even more challenging since infrastructure resources change very frequently. Secrets management tools are here to manage your secrets from a centralized and controlled point of view. It helps to keep your secrets secure and to protect your valuable data. I hope these tips and tricks provide a basis for you to get started with your “secrets management” journey.