Generate mTLS certificates for Nomad using Vault
You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates for your Nomad cluster nodes and Vault Agent to automatically create the appropriate certificate and key files on your nodes. With this method, each node has a unique certificate with a relatively short time-to-live (TTL). This feature, along with automatic certificate rotation, allows you to safely and securely scale your cluster while using mutual TLS (mTLS).
In this tutorial, you will secure your existing Nomad cluster with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use Vault Agent to fetch, renew, and periodically rotate your mTLS certificates on your Nomad nodes.
Prerequisites
Running this tutorial requires the following:
- A Nomad environment with Vault installed. You can use this repository to provision a sandbox environment. This tutorial will assume a cluster with one server node and three client nodes.
Note
This tutorial is for demonstration purposes and is using a single Nomad server with a Vault server configured alongside it. In a production cluster, we recommend 3 or 5 Nomad server nodes along with a separate Vault cluster. Refer to the Vault Reference Architecture page to learn how to securely deploy a Vault cluster.
Prepare Vault
If you already have a Vault environment, skip ahead to Log in to Vault.
Initialize Vault server
Run the following command to initialize the Vault server and receive an unseal key and initial root token. If you are running the environment provided in this guide, the Vault server is co-located with the Nomad server. Be sure to note the unseal key and initial root token as you will need these two pieces of information.
$ vault operator init -key-shares=1 -key-threshold=1
The vault operator init command creates a single Vault unseal key for
convenience. For a production environment, we recommended that you create at
least five unseal key shares and securely distribute them to independent
operators. The command defaults to five key shares and a
key threshold of three. If you have provisioned more than one server, the others will
become standby nodes and remain unsealed.
Unseal Vault
Run the command and provide your unseal key.
$ vault operator unseal
The output looks similar to the following.
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.0.3
Cluster Name vault-cluster-d1b6513f
Cluster ID 87d6d13f-4b92-60ce-1f70-41a66412b0f1
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Log in to Vault
Use the login command to authenticate with Vault using the initial root token from earlier.
$ vault login
Token (will be hidden): <your initial root token>
Successful output looks similar to the following.
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
...
Prepare the PKI environment
This tutorial uses a common and recommended pattern which is to have one mount act as the root CA and the other as the intermediate CA. The root CA signs only the intermediate CA CSRs from other PKI secrets engines. For a higher level of security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA.
Generate the root CA
Enable the root PKI secrets engine at the pki path.
$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.
$ vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/
Generate the root certificate and save the certificate as CA_cert.crt.
Note
If you already have an existing root CA certificate in your organization, you can use that in place of generating a new one.
$ vault write -field=certificate pki/root/generate/internal \
common_name="global.nomad" ttl=87600h > CA_cert.crt
Generate the intermediate CA and CSR
Enable the intermediate PKI secrets engine at the pki_int path.
$ vault secrets enable -path=pki_int pki
Success! Enabled the pki secrets engine at: pki_int/
Tune the PKI secrets engine at the pki_int path to issue certificates with a
maximum time-to-live (TTL) of 43800 hours.
$ vault secrets tune -max-lease-ttl=43800h pki_int
Success! Tuned the secrets engine at: pki_int/
Generate a CSR from your intermediate CA and save it as pki_intermediate.csr.
$ vault write -format=json pki_int/intermediate/generate/internal \
common_name="global.nomad Intermediate Authority" \
ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr
Sign and deploy the intermediate CA certificate
Sign the intermediate CA CSR with the root certificate and save it as intermediate.cert.pem.
$ vault write -format=json pki/root/sign-intermediate \
csr=@pki_intermediate.csr format=pem_bundle \
ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem
Import the signed intermediate certificate into Vault.
$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem
Set up the AppRole auth method
Vault offers many authentication methods including AWS, GCP, Azure, GitHub, LDAP, and Okta. In a production environment running in a cloud provider, we recommend using the machine's identity and the cloud provider's auth method.
The AppRole authentication method allows applications or services to authenticate with Vault-defined roles. For on-premises environments, we recommend AppRoles. In this tutorial, Nomad is using the AppRole auth method to retrieve certificates from Vault with the PKI engine.
Enable the AppRole auth method.
$ vault auth enable approle
Success! Enabled approle auth method at: approle/
Create a policy to access the role endpoint
Create a policy file named tls-policy.hcl, add the following contents to it, and save the file. This adds only the update capability for the pki_int/issue/nomad-cluster path. Refer to the Vault policies page for additional information.
tls-policy.hcl
path "pki_int/issue/nomad-cluster" {
capabilities = ["update"]
}
Write the policy into Vault.
$ vault policy write tls-policy tls-policy.hcl
Success! Uploaded policy: tls-policy
Create an approle
Create an approle named nomad-cluster. Consider increasing or decreasing the TTL and SecretID values to match the your environment.
$ vault write auth/approle/role/nomad-cluster \
token_type=batch \
token_policies="tls-policy" \
secret_id_ttl=10m \
token_ttl=10m \
token_max_ttl=15m \
secret_id_num_uses=10
The output looks similar to the following.
Success! Data written to: auth/approle/role/nomad-cluster
Next, get the role ID and save it to a file.
$ vault read -field="role_id" auth/approle/role/nomad-cluster/role-id > roleid
Then, get the SecretID and save it to a file. The SecretID is similar to a password in that you should keep it secret and confidential. Refer to the best practices page for additional information about working with the SecretID.
$ vault write -field="secret_id" auth/approle/role/nomad-cluster/secret-id > secretid
Create and populate the templates directory
The templates directory contains the files used by Vault Agent to render the certificates and keys on the nodes in your cluster.
Create a directory named templates in /opt/nomad.
$ sudo mkdir /opt/nomad/templates
Create a template file for each configuration below, add the contents to it, and save it with the filename given.
We recommend creating only the server and CLI files for server nodes and client and CLI files for client nodes.
Note
The common_name value in the files below must match the Nomad node names in your cluster which include the node name and region. If you have a different configuration for the nodes in your cluster, be sure to update the values.
/opt/nomad/templates/server.agent.crt.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.Cert }}
{{ end }}
/opt/nomad/templates/server.agent.key.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.Key }}
{{ end }}
/opt/nomad/templates/server.ca.crt.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h"}}
{{ .Data.CA }}
{{ end }}
Create the certificates output directory
Create the rendered-cert-files directory to contain the generated certificate files.
$ sudo mkdir /opt/nomad/rendered-cert-files
Start Vault Agent
Create a directory for the configuration named vault-agent in /opt/vault.
$ sudo mkdir /opt/vault/vault-agent
Create the Vault Agent configuration file, name it vault_agent.conf, add the following contents to it, and save the file in the /opt/vault/vault-agent directory.
Modify this configuration file to match the templates created for each specific node from the previous step. Include or remove the server and client template blocks as necessary.
/opt/vault/vault-agent/vault_agent.conf
pid_file = "./pidfile"
vault {
address = "https://127.0.0.1:8200"
# For demonstration purposes only. In a production environment,
# tls_skip_verify value should NOT be set to true.
tls_skip_verify = true
}
auto_auth {
method {
type = "approle"
config = {
role_id_file_path = "/opt/vault/vault-agent/roleid"
secret_id_file_path = "/opt/vault/vault-agent/secretid"
remove_secret_id_file_after_reading = false
}
}
sink {
type = "file"
wrap_ttl = "30m"
config = {
path = "sink_file_wrapped_1.txt"
}
}
sink {
type = "file"
config = {
path = "sink_file_unwrapped_2.txt"
}
}
}
listener "tcp" {
address = "127.0.0.1:8100"
tls_disable = true
}
template_config {
static_secret_render_interval = "10m"
exit_on_retry_failure = true
max_connections_per_host = 20
}
# Server templates; remove if on client node
template {
source = "/opt/nomad/templates/server.agent.crt.tpl"
destination = "/opt/nomad/rendered-cert-files/server.agent.crt"
}
template {
source = "/opt/nomad/templates/server.agent.key.tpl"
destination = "/opt/nomad/rendered-cert-files/server.agent.key"
}
template {
source = "/opt/nomad/templates/server.ca.crt.tpl"
destination = "/opt/nomad/rendered-cert-files/server.ca.crt"
}
# Client templates; remove if on server node
template {
source = "/opt/nomad/templates/client.agent.crt.tpl"
destination = "/opt/nomad/rendered-cert-files/client.agent.crt"
}
template {
source = "/opt/nomad/templates/client.agent.key.tpl"
destination = "/opt/nomad/rendered-cert-files/client.agent.key"
}
template {
source = "/opt/nomad/templates/client.ca.crt.tpl"
destination = "/opt/nomad/rendered-cert-files/client.ca.crt"
}
# CLI templates
template {
source = "/opt/nomad/templates/cli.crt.tpl"
destination = "/opt/nomad/rendered-cert-files/cli.crt"
}
template {
source = "/opt/nomad/templates/cli.key.tpl"
destination = "/opt/nomad/rendered-cert-files/cli.key"
}
Then, start Vault Agent on each node.
$ sudo vault agent -config=/opt/vault/vault-agent/vault_agent.conf
The output shows the agent rendering the local files with the certificate data.
# ...
2024-11-26T10:09:54.347-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/cli.crt.tpl" => "/opt/nomad/rendered-cert-files/cli.crt"
2024-11-26T10:09:54.367-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/server.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/server.ca.crt"
2024-11-26T10:09:54.405-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/client.agent.key.tpl" => "/opt/nomad/rendered-cert-files/client.agent.key"
2024-11-26T10:09:54.425-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/server.agent.key.tpl" => "/opt/nomad/rendered-cert-files/server.agent.key"
2024-11-26T10:09:54.463-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/cli.key.tpl" => "/opt/nomad/rendered-cert-files/cli.key"
2024-11-26T10:09:54.484-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/client.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/client.agent.crt"
2024-11-26T10:09:54.569-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/server.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/server.agent.crt"
2024-11-26T10:09:54.592-0500 [INFO] agent: (runner) rendered "/opt/nomad/templates/client.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/client.ca.crt"
Configure Nomad to use TLS
Add the following tls stanza to the agent configuration file of each Nomad server and client node in the cluster.
/etc/nomad.d/nomad.hcl
tls {
http = true
rpc = true
rpc_upgrade_mode = true
ca_file = "/opt/nomad/rendered-cert-files/server.ca.crt"
cert_file = "/opt/nomad/rendered-cert-files/server.agent.crt"
key_file = "/opt/nomad/rendered-cert-files/server.agent.key"
verify_server_hostname = true
verify_https_client = true
}
The rpc_upgrade_mode attribute ensures that the Nomad servers accept both TLS and non-TLS connections during the upgrade.
Reload the Nomad configuration on each node.
$ systemctl reload nomad
After the reload, remove the rpc_upgrade_mode = true line from each server agent configuration file and save the file. This instructs the servers to only accept TLS connections.
/etc/nomad.d/nomad.hcl
tls {
http = true
rpc = true
- rpc_upgrade_mode = true
# ...
Then, reload the Nomad configuration on each server node again. Refer to the RPC Upgrade Mode page for additional information.
$ systemctl reload nomad
Configure the Nomad CLI
The Nomad CLI defaults to HTTP instead of HTTPS. Configure the Nomad CLI to connect using TLS by providing paths to the certificate files through environment variables.
export NOMAD_CACERT="/opt/nomad/rendered-cert-files/server.ca.crt"
export NOMAD_CLIENT_CERT="/opt/nomad/rendered-cert-files/cli.crt"
export NOMAD_CLIENT_KEY="/opt/nomad/rendered-cert-files/cli.key"
Run the status command to confirm connectivity from the CLI.
$ nomad status
No running jobs
Encrypt server gossip
All communications over RPC and HTTP in your Nomad cluster are now secure with mTLS.
Nomad servers also communicate with the Serf gossip protocol, which does not use TLS. To learn how to configure gossip encryption, follow the Enable Gossip Encryption for Nomad tutorial.
Next steps
In this tutorial, you configured the Vault PKI engine, installed and used Vault Agent to automatically create certificate files on cluster nodes, and enabled mTLS with those certificates to encrypt traffic between the nodes.
Continue your learning with these resources: