Managing all your secrets with Vault - Review and Walkthrough

By Martin Rusev read

When Hashicorp released Vault - an open source tool for managing secrets(passwords, API keys, etc.), I was excited to check it out. Managing sensitive information is a hard problem in the devops world. I've seen and tried a lot of different approaches over the years, but none of them stands out as particularly easy or scalable:

My point being - there isn't a simple, single solution that works equally well for a single dev, team, mixed team with technical and non technical members, for infrastructure and for apps. You have to mix and match different methods and that is already a security risk.

Vault promises to be a solution that can solve some of these issues or at least be a solid building block for a custom one.

Getting Started with Vault

Vault is written in Go and distributed as a single static binary - this is great for testing on your local machine.

One inconvenience, you will spot right away - there are no deb/rpm files provided or official repositories. This makes Vault somewhat laborious to deploy in production - you have to find/write/maintain your own init scripts and CAPS provision recipes. Alternatively you can deploy Vault in a Docker container or with Supervisord.

Once you download the binary, you can execute commands right away, no config required. This is what a "hello world" looks like in Vault. We are going to store a DigitalOcean API key and then retrieve the value.

vault server -dev
export VAULT_ADDR='http://127.0.0.1:8200' # Put this in your .bashrc/zshrc file

vault write secret/digitalocean-api-key value=7175b4cfff396de2c136d052bfac6b0d4
Success! Data written to: secret/digitalocean-api-key

vault read secret/digitalocean-api-key
Key             Value
lease_duration  2592000
value           7175b4cfff396de2c136d052bfac6b0d4

# Just the value
value read -field=value secret/digitalocean-api-key
7175b4cfff396de2c136d052bfac6b0d4

To retrieve the DigitalOcean API key from your apps, you have to do something like this:

# Using Python for this example, should be similar in other languages.
import requests
headers = {"X-Vault-Token": "b30ee2a3-ea4b-9da0-3e5c-4189d375cad9"}

r = requests.get('http://127.0.0.1:8200/v1/secret/digitalocean-api-key', headers=headers)
json_response = r.json()
# {u'lease_id': u'', u'warnings': None, u'auth': None, u'lease_duration': 2592000,
# u'data': {u'value': u'7175b4cfff396de2c136d052bfac6b0d4'}, u'renewable': False}

if r.status_code == 200:
    api_key = json_response.get('data',{}).get('value', "")
    # 7175b4cfff396de2c136d052bfac6b0d4

In your apps, you can wrap this in a function:

def vault_get_secret(value):
  request ....
  return value

# settings.py
API_KEY = vault_get_secret('digitalocean-api-key')

Vault Concepts

Sealing/Unsealing the Vault

The information you store in Vault is protected by a master key. Vault uses the master key to encrypt/decrypt secrets. The master key is not available on its own and has to be reconstructed every time when you start/restart Vault.

To do that Vault uses an algorithm, called Samir Secret Sharing. The master key is split in 5 keys by default - this is called number of shares. To reconstruct the key you need to provide any 3 of the 5 keys - this is called a threshold. The number of shares and the threshold are fully customizable, depending on your needs. This is well explained in the docs and I found this article that gets into even more detail how that works - with code examples and video.

Unsealing is an important part of the security layer in Vault and something you have to consider, when deploying in production - you have to unseal every time Vault is stopped/the server was restarted, etc.

Authentication & Tokens

Once you unseal Vault, you have to authenticate. Tokens are the default authentication method. You get your admin/root token when you initialy configure Vault - with vault init or vault server -dev. The root token should be used only for configuration.

# Authenticate with the root token
vaul auth b30ee2a3-ea4b-9da0-3e5c-4189d375cad9

# This is still a root token, can be revoked, doesn't expire
vault token-create

Key             Value
token           f125ca98-a6dc-07da-bc48-a8cafce57889
token_duration  0
token_renewable false
token_policies  [root]

vault token-revoke f125ca98-a6dc-07da-bc48-a8cafce57889

Revocation successful.

# Create a token with limited permissions
vault token-create -policy=djangoapp

Key             Value
token           859a8125-d221-d1dd-b709-4629c1e9591b
token_duration  2592000
token_renewable true
token_policies  [djangoapp default]

vault policies djangoapp

path "secret/djangoapp/*" {
  policy = "read"
}

In Vault 0.4 (The version I am testing), you can't list tokens after creation. This is on the roadmap for future releases, but until then - you have to keep track of your tokens in a third party app.

The important thing to remember about Authentication and Tokens.

Each token can be revoked and has different levels of access controlled by a Policy. Policies are the ACL(Access Control List) in Vault.

Policies

Policies in Vault work like chown/users on Unix - you have the root user with access to everything and individual users with access to a specific path.

# djangoapp.hcl
path "secret/djangoapp/*" {
  policy = "read"
}

# authenticate with the root token
vault auth 8a5dbab4-43cc-8b8d-e6a6-04d5e458dca9

# apply the policy, you can change it at any time and reapply with the same command
vault policy-write djangoapp djangoapp.hcl


# This token can only read secrets from secret/djangoapp/*
vault token-create -policy=djangoapp

Key             Value
token           cc9a61f9-0c40-27ff-74aa-5a082c4d269f
token_duration  2592000
token_renewable true
token_policies  [djangoapp default]

# write some secrets
vault write secret/djangoapp/mongo-auth value="martinrusev:8PV2NW6k@192.168.0.1:27017"
vault write secret/djangoapp/digitalocean-api-key value=7175b4cfff396de2c136d052bfac6b0d4


vault auth cc9a61f9-0c40-27ff-74aa-5a082c4d269f

vault read secret/djangoapp/digitalocean-api-key

Key             Value
lease_duration  2592000
value           7175b4cfff396de2c136d052bfac6b0d4

Backends

Backends are the core components of Vault. You already saw the default generic backend, which is all the data you store in the secret path. They work like the filesystem in Unix - you can mount/unmount different types on a path.


$ vault mounts
Path          Type       Default TTL  Max TTL  Description
secret/       generic    system       system   generic secret storage
sys/          system     n/a          n/a      system endpoints used for control ...

vault mount -path=django -description="Secrets used in a Django project" generic

vault mounts
Path          Type       Default TTL  Max TTL  Description
secret/       generic    system       system   generic secret storage
django/       generic    system       system   Secrets used in a Django project
sys/          system     n/a          n/a      system endpoints used for control ...

# We can write our data in this namespace
vault write django/aws-secret-key value="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
vault write django/aws-access-key value="AKIAIOSFODNN7EXAMPLE"

Each backend servers different purpose:

Deploying Vault

To deploy Vault in production you need an init script, which is not provided. You can write your own, pack Vault in a Docker container(this is the easiest option) or daemonize with Supervisord.

The next step is choosing a storage. This is where Vault stores your sensitive data. You can store data in files, S3, MySQL database, Consul, etc.

I personally find S3 to be a nice balance between ease of use and safety. With Vault 0.4 you can't list your secrets, but you can check them out through the Amazon Console.

# vault.hcl
backend "s3" {
  bucket = "vault"
  access_key = "AKIAIPR2HA"
  secret_key = "nO6TK2NAPpZkBf+iSWrnQAojIp"
  region = "eu-west-1"
}

listener "tcp" {
  address = "127.0.0.1:8200"
   tls_disable = 1
}

vault server -config=vault.hcl

vault init

# Key 1: 4cdc4cccb9e2ddda8a2163e8d1a1b94522e6e9c6d25d6a5a12e9677262547c1401
# Key 2: 8b94a7e4614956778ccce9a185d736e9d4a86dd75ca588056f9c10da60a8f31d02
# Key 3: 42378fbd9549091da0bcf6c14f9287e961e0738eb129b4b28d79538e702ab9db03
# Key 4: b35c835644358269219fd75378507380d9cc76ca47fe5f64ce28b28e9eb5f82c04
# Key 5: 7affab0fb035dd030defc833b215c2806c846893aa7263d32ccdf1da8e37b2ea05
# Initial Root Token: 8a5dbab4-43cc-8b8d-e6a6-04d5e458dca9

# You have to provide any 3 of the 5 keys printed to stdout after the init command
vault unseal

# authenticate with the root token, configure Vault, set policies, etc.
vault auth 8a5dbab4-43cc-8b8d-e6a6-04d5e458dca9

Vault uses TLS to protect the TCP socket it is running on. You have to generate one and the docs don't explain how to do this. Docker uses the same security mechanism and the Dockers docs could serve you well as a reference point.

A simpler option for securing your Vault instance can be a nginx/apache frontend. For example, you can setup basic auth or allow access only from certain IP addresses.

# Full configuration at https://mozilla.github.io/server-side-tls/ssl-config-generator/
# /etc/nginx/sites-enabled/vault
server {
        listen 80;
        listen 443 ssl;
        server_name vault;
        return 301 https://vault.yourdomain.com$request_uri;
    }

        server {
        listen 443 ssl;
        location / {
            proxy_buffering    off;
            proxy_pass         http://127.0.0.1:8200;
            proxy_redirect     off;
            proxy_set_header   Host $http_host;
        }
}

My impressions of Vault

I recently documented my experience with Terraform, another Hashicorp project that I use often. My experience with Vault was very similar - really unique and valuable project, solid addition to the devops arsenal.

The downsides - steep learning curve, insufficient documentation, lack of examples. You have to figure out how things works from google groups, stackoverflow, etc. and that takes a significat amount of time.