What is Vault?
Vault is an identity-based secrets and encryption management system. Secrets are any data you want to strictly control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services controlled by authentication and authorization methods. Using Vault's UI, CLI, or HTTP API, you can securely store and manage secrets, strict control and audit access to sensitive data.
Why Use Vault?
- Implementing password rotation policies is painful
- Former employees who had access to secrets may leak information or retaliate maliciously
- Developers accidentally upload secrets to public code repositories, causing data leaks
- Managing secrets across multiple systems is cumbersome
- Need to encrypt secrets securely without exposing keys to applications, preventing key leakage when applications are compromised
Vault Architecture
Vault only exposes the storage backend and API, while other components are protected. Vault does not trust the backend storage—all stored data is encrypted.
Installation
The following example is for Mac. For other platforms, refer to the official Vault installation guide.
$ brew tap hashicorp/tap
$ brew install hashicorp/tap/vault
Starting Vault (Dev Mode)
$ vault server -dev
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variables:
$ export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: qwerty1234567890abcdefghijklmnopqrstuvwxyzABCD
Root Token: hvs.IKbh5pTGI0Qn08G5QNJb4jPY
Development mode should NOT be used in production installations!
Checking Service Status
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.12.1
Build Date 2022-10-27T12:32:05Z
Storage Type inmem
Cluster Name vault-cluster-c5c1f8a4
Cluster ID 60514532-959b-9540-ea21-5d9c968a21ba
HA Enabled false
Writing Your First Secret
$ vault write kv/project-alpha data="enterprise"
Success! Data written to: kv/project-alpha
If you encounter this error:
vault write kv/project-alpha data="enterprise"
Error writing data to kv/project: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/kv/project
Code: 404. Errors:
- no handler for route 'kv/project'
The KV secrets engine is not enabled. Run the following command:
$ vault secrets enable kv
Reading Data
$ vault read kv/project-alpha
Key Value
--- -----
refresh_interval 768h
data enterprise
Using Consul as Storage Backend
Vault does not store data itself and requires a storage backend. It supports multiple backends.
$ brew tap hashicorp/tap
$ brew install hashicorp/tap/consul
Starting Consul
$ consul agent -dev
==> Starting Consul agent...
Version: '1.14.1'
Build Date: '2022-11-21T16:56:07 +0000 UTC'
Node ID: '6d9bcfc4-60cf-5d9b-c4fa-764127a94af6'
Node name: 'Dev-MacBook.local'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600)
Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Gossip Encryption: false
Auto-Encrypt-TLS: false
HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
==> Log data will now stream in as it occurs:
2022-11-30T16:34:52.604+0800 [WARN] agent: Node name "Dev-MacBook.local" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2022-11-30T16:34:52.609+0800 [WARN] agent.auto_config: Node name "Dev-MacBook.local" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.
2022-11-30T16:34:52.613+0800 [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:6d9bcfc4-60cf-5d9b-c4fa-764127a94af6 Address:127.0.0.1:8300}]"
2022-11-30T16:34:52.614+0800 [INFO] agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader-address= leader-id=
Starting Vault (Production Mode)
Create a configuration file using Consul as the storage backend.
Note: This mode is for production and requires manual key configuration and management.
storage "consul"{
address = "127.0.0.1:8500"
path = "vault/"
}
listener "tcp"{
address = "127.0.0.1:8200"
tls_disable = 1
}
Run the following command to start:
$ vault server -config=config.hcl
WARNING! mlock is not supported on this system! An mlockall(2)-like syscall to
prevent memory from being swapped to disk is not supported on this system. For
better security, only run Vault on systems where this call is supported. If
you are running Vault in a Docker container, provide the IPC_LOCK cap to the
container.
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Go Version: go1.19.2
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: false, enabled: false
Recovery Mode: false
Storage: consul (HA available)
Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
Version Sha: e34f8a14fb7a88af4640b09f3ddbb5646b946d9c
==> Vault server started! Log data will stream in below:
2022-11-30T16:42:59.995+0800 [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2022-11-30T16:43:00.006+0800 [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
Note: Although the service is running, Vault is in a sealed state and unavailable.
Attempting operations will show:
$ vault read kv/project
Error reading kv/project: Error making API request.
URL: GET http://127.0.0.1:8200/v1/kv/project
Code: 503. Errors:
* Vault is sealed
The dev mode automatically handles master key creation and unsealing. Production mode requires manual configuration.
Key Concepts
- All data stored in Vault's backend is encrypted
- The key used by Vault is called the Master Key. By default, Vault uses the Shamir algorithm to split the master key into M shares, and administrators must provide at least N of them to reconstruct the master key (M and N are configurable, with M>=N). In an ideal scenario, these M key shares should be distributed to M different people in the organization, and only after obtaining N people's authorization can Vault successfully decrypt the master key.
Initializing Vault (Production Mode)
The following steps generate 5 key shares and one root token. In real scenarios, these 5 key shares are distributed to 5 different people for management. At least 3 keys must be correctly provided to unseal Vault.
Note: Vault enters a sealed state every time it starts/restarts. Until unsealed, almost no operations can be performed.
$ vault operator init
Unseal Key 1: abcDef123Ghi456Jkl789Mno012Pqr345Stu678Vwx901=
Unseal Key 2: xyz987Uvw654Tsu321Rqp054Onm876Lkj210Ihg543=
Unseal Key 3: Fgh210Ijk543Clm876Dno210Pqr987Mno432Klj765=
Unseal Key 4: Def654Ghi987Jkl210Mno543Pqr876Stu109Uvw432=
Unseal Key 5: Abc345Def678Ghi901Jkl234Mno567Pqr890Stu123=
Initial Root Token: s.o7NjzR6qXEvJEwPYpmVFOt4P
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
Vault defaults to HTTPS, so if you get this error during initialization:
Error authenticating: error looking up token: Get "https://127.0.0.1:8200/v1/auth/token/lookup-self": http: server gave HTTP response to HTTPS client
Set the Vault address environment variable:
export VAULT_ADDR='http://127.0.0.1:8200'
Checking Vault Status
Vault is now in a sealed state (Sealed is true).
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.9.2
Storage Type consul
HA Enabled true
Unsealing Vault
Use the vault operator unseal command. Atleast 3 keys are required.
$ vault operator unseal abcDef123Ghi456Jkl789Mno012Pqr345Stu678Vwx901=
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 9eeaaa92-d673-51d3-b993-d3affe68b5a3
Version 1.9.2
Storage Type consul
HA Enabled true
Note: After entering each key, you see "Unseal Progress 1/3". However, this only indicates how many keys have been entered, not whether the entered keys are correct!
Managing Secrets with Vault
Before proceeding, let's examine what measures Vault provides to protect your secrets:
- Key protection mechanism: Keys are split into 5 shares, and at least 3 must be correctly entered to unseal.
- Policy control (access control): Policies can be set for each path.
- Time-to-live: Every token generated by Vault has an expiration time.
First, let's write two test secrets:
$ vault write kv/project-alpha data="enterprise"
Success! Data written to: kv/project-alpha
$ vault write kv/project-beta data="startup"
Success! Data written to: kv/project-beta
Note:
"kv/project-alpha" is a path. Think of it as a table in a database, not a key in a KV store. You can try storing and reading data this way:
$ vault write kv/project-alpha name="company" category="tech" region="us"
Success! Data written to: kv/project-alpha
$ vault read kv/project-alpha
Key Value
--- -----
refresh_interval 768h
name company
category tech
region us
Creating Policies
Create a Policy File
Enter the following content:
path "kv/project-alpha" {
capabilities = ["read"]
}
Explanation: This policy grants read-only access to the "project-alpha" path.
Apply the Policy
Authenticate to the unsealed Vault using the root token from initialization:
$ vault login s.o7NjzR6qXEvJEwPYpmVFOt4P
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do not need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.o7NjzR6qXEvJEwPYpmVFOt4P
token_accessor tKZyxkkFvrmhQeTsoDOogcoV
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
$ vault policy write alpha-readonly policy.hcl
Success! Uploaded policy: alpha-readonly
"alpha-readonly" is the policy name, which can be customized.
Generating Tokens with Policies
Generate a token with read-only access to "kv/project-alpha":
$ vault token create -policy="alpha-readonly"
Key Value
--- -----
token s.bHzsfREs9MGirYO8yZ2TbAzq
token_accessor nPuAQzNTn28vMMUKcDu0mN8g
token_duration 768h
token_renewable true
token_policies ["alpha-readonly" "default"]
identity_policies []
policies ["alpha-readonly" "default"]
This generates a new token: s.bHzsfREs9MGirYO8yZ2TbAzq, valid for 768 hours (32 days).
Authenticate using this new token (to switch tokens, simply run login again):
$ vault login s.bHzsfREs9MGirYO8yZ2TbAzq
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do not need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.bHzsfREs9MGirYO8yZ2TbAzq
token_accessor nPuAQzNTn28vMMUKcDu0mN8g
token_duration 767h56m19s
token_renewable true
token_policies ["alpha-readonly" "default"]
identity_policies []
policies ["alpha-readonly" "default"]
Testing the New Token
Try reading the secret at kv/project-alpha:
$ vault read kv/project-alpha
Key Value
--- -----
refresh_interval 768h
data enterprise
Success!
Try writing:
$ vault write kv/project-alpha data="updated"
Error writing data to kv/project: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/kv/project-alpha
Code: 403. Errors:
* 1 error occurred:
* permission denied
Can this token read secrets from kv/project-beta?
$ vault read kv/project-beta
Error reading kv/project-beta: Error making API request.
URL: GET http://127.0.0.1:8200/v1/kv/project-beta
Code: 403. Errors:
* 1 error occurred:
* permission denied
No, it cannot!
This covers the basics of Vault. There are many more features to explore—check the documentation below.
Reference Links
- [1]: Vault Official Documentation
- [2]: Vault Tutorials
- [3]: How To Securely Manage Secrets with HashiCorp Vault on Ubuntu 20.04