What is Vault?
Vault is a Secret Management tool offered by HashiCorp. Vault provides Advanced Data Protection for your API keys, certificates, secrets, passwords, authentication tokens, PKIs, SSH keys, and other security elements. You can secure your workloads and protect your data across any infrastructure or systems you use.
With Vault, you can create static or dynamic secrets for your workloads and never have to store sensitive data elsewhere.
Other key features of Vault are:
- Secure Secret Storage,
- Data Encryption,
- Leasing and Renewal for your secrets,
- Revocation for your secrets.
How Vault Works
* High-Level Vault Architecture
* HashiCorp Vault Documentation
This is the High-Level architecture of Vault. Let’s break it down.
There is a Storage Backend for storing all of your data. You can configure different storage backend options for your Vault configuration such as local storage, HashiCorp Consul, and S3.
At the top, you can see the HTTP(S) API layer. That’s where users or machines (computers, applications) access Vault.
At the heart of the Vault, you can see the Barrier section. This is where all your configuration, policies, Vault system backend, authentication methods, audit devices and secret engines live.
What are Seal and Unseal States?
When Vault is initialized, it creates an Encryption Key that protects all your data. A master key is in charge of protecting that encryption key. By default, Vault uses Shamir’s secret sharing algorithm to split the Master Key into key shares. You can configure the number of these shares in your Vault configuration and you can also configure the minimum number of required shares to reconstruct Master Key.
* Vault Shamir Secret Sharing
* HashiCorp Vault Documentation
Every time you start a Vault Server, it starts in a sealed state. When Vault is in the Sealed state, you can not access any data that is in the Vault. Vault doesn’t know how to decrypt data when it’s in a sealed state. To access or modify your data, you need to unseal the Vault.
Unsealing the Vault is the process of retrieving the Master key and using that key to decrypt the data inside the Vault and access it.
Once the Vault is unsealed, it will stay in the Unsealed state until one of these things happen:
- The Vault Server is restarted,
- There is an unrecoverable error in the Vault’s Storage layer,
- It is released via API.
There are three Unsealing methods you can use in Vault. In this blog post, I will compare those three methods. These methods are:
- Manual Unsealing,
- Auto Unseal,
- Transit Auto Unseal
Let’s start with manual unsealing.
Manual Unsealing
This method is the default option for all Vault configurations. It is also the simplest one. It works on any platform and configuration options make it flexible.
When you configure your Vault Server, you can configure the number of Key Shares for Master Key and the required number of Key Shares to construct Master Key. By using this minimum number of Key Shares you can unseal Vault.
To unseal Vault you will need to use the ‘vault operator unseal’ command. When you run this command Vault will ask you to enter Key Shares one at a time. That means you will need to run this command repeatedly. Once you reach the minimum number of Key Shares to reconstruct the key and decrypt the Master Key Vault will unseal itself.
To demonstrate this method I have set up a Vault on an Ubuntu Server running on AWS EC2. I have configured Vault to run as a systemd service. Before unsealing I ran the ‘vault status’ command and here is the result.
As you can see from the screenshot, Vault is not initialized and is sealed. After that, I ran the ‘vault operator init’ command to initialize Vault.
In the output of this command, there are five Unseal Keys, Initial Root Token and a description. I need to use three of these Unseal Keys to unseal Vault. And also, when Vault is initialized for the first time, the only authentication method is Root Token. When I ran the ‘vault status’ command I got this result.
As you can see, Vault is initialized and in the Sealed state right now. But in the output, we can also see our total key shares and the minimum threshold for unsealing Vault.
In order to unseal Vault, I use the ‘vault operator unseal’ command.
I have inserted one of the Unseal Keys from the output of the initialize command. Now, Unseal Progress has started. I need to run the same unseal command until I reach the threshold which is three in this demonstration.
After running the unseal command three times, as you can see from the screenshot above the output has changed. The Sealed section is now false. I can see the version of my Vault installation and some other information about my Vault Server.
Now, I can authenticate with my Vault Server and perform operations. For example, to list the enabled secret engines I use the ‘vault secrets list’ command.
But, you should always consider using encryption with PGP keys in this option because as you can see from the first initialize command’s output, Unseal Keys are received as plain-text. That means the user who runs this command receives all the keys as plain-text. This is of course against the security model of the Vault.
You can initialize Vault with PGP keys and with this approach, Vault will generate Unseal Keys and encrypt those keys immediately with given user’s public PGP keys. Only the person who has the corresponding private key is going to be able to decrypt the value and receive the plain-text Unseal key. You can use OpenPGP-compatible tools like GPG or services like Keybase.io for these operations.
One of the best practices of using this method is to distribute those Key Shares to different employees in your company to store them securely. But the downside of this best practice is whenever you need to unseal Vault you will need to reach out to those people. For example, let’s say you have a maintenance window at 3 a.m and you need to restart your Vault server. You will have to wake some people in the middle of the night. Also, this practice introduces some risks about storing keys.
Auto Unseal
You can always configure your Vault Server to use a Cloud Provider’s KMS Solution or a Hardware HSM for unseal operations. That way you won’t have to worry about unsealing your Vault Server manually after every maintenance operation or downtime. Because when you configure Auto Unseal, your Vault Server will reach out to the solution that you configured for unsealing and automatically perform the operation.
With that, you can just set and forget unsealing in your Vault Server. And also, if you are, for example, running all your applications in AWS and using AWS KMS for Auto Unseal you will have some integration benefits from this configuration as well.
If you configure Auto Unseal, you won’t have Unseal Keys when you first initialize Vault Server. Instead, you will have Recovery Keys and you can use these Keys for authorization. But keep in mind, those Recovery Keys are just for authorization. If your Auto Unseal mechanism doesn’t work you can’t use these Recovery Keys for unsealing Vault.
For this demonstration, I need to change my Vault configuration to use a KMS. Since I am running this Vault Server on AWS, I will use the AWS KMS service. I have configured the KMS service already. Now in the Vault configuration file, I am adding the seal stanza.
After changing the configuration I have restarted the vault service. When I check the vault service with the ‘systemctl status vault’ command, there is some additional information about the unsealing' process in the last line saying post-unseal setup complete.
Now it is time to run the ‘vault status’ command again.
The output is a little bit different from the earlier one. Vault is in an unsealed state and instead of showing Unseal Keys you get Recovery Key Shares as I mentioned before.
With that said, there are some regional requirements for Cloud Provider HSM solutions because those solutions are tied into a cloud region. And also, this approach could cause vendor lock-in.
Transit Auto Unseal
With this method, you can configure your Vault Server or Cluster to automatically unseal with another Vault Server or Cluster. For example, if you have multiple Vault clusters, you can place a central Vault cluster for just transit auto unseal configurations and configure other Vault clusters to use this central cluster for unseal operations.
In this demonstration, I have two Vault Servers. To start this process, I need to configure an Auto Unseal key provider in the first Vault Server.
After enabling the transit secrets engine I will write a key to this path.
Now I will create a policy and a token with this policy attached to it. I will use this token in the second server’s configuration for Auto Unseal.
Now in the second Vault Server, I will add the seal stanza with configuration for Transit Auto Unseal.
After changing the configuration, I run the ‘vault status’ command again.
As you can see in the output, Recovery Seal Type has been changed to transit. I run the ‘vault operator init’ to initialize Vault and the ‘vault status’ again. Here is the output.
Just like in the Auto Unseal configuration, with Transit Auto Unseal, Vault Server is initialized and is in unseal state.
With this configuration, you can also set and forget unsealing for your cluster. This configuration is also platform agnostic. That means you can place your Vault clusters on different platforms and it won’t cause any problems. So, transit auto unseal configurations can become really useful when you run many Vault clusters across Cloud Providers or on-premise data centers.
However, with Transit Auto Unseal, you need to have a centralized Vault cluster and you need to maintain this cluster as well. Most importantly, this central cluster becomes the most crucial Vault cluster in your entire deployment. You have to make sure that this cluster is up and running most of the time.
Conclusion
Vault offers many options for secret management. I have described what Vault is, how Vault works and different unseal methods in this blog post. At the same time, I compared these unseal methods with the pros and cons of each one.
Most of the time, having an automatic configuration for unseal operations is better than having a manual configuration. But having another cluster for just these operations and the need of maintaining this centralized cluster brings more operational burden on people’s shoulders.
Of course, you should always configure this option for your needs and requirements. I recommend using Vault if your design, configuration, and requirements are met for your environments.