At MyWordPress.io, we cannot say enough good things about Caddy. It’s stable, fast, easy to configure, and a breeze to maintain.
For those that are unaware, one of the major facets of our secrets management involves Vault. Like Caddy, using Vault to manage secrets is a breeze–gone are the days of secrets sprawled all of your infrastructure (not to mention developer laptops!).
The Caddy Vault Storage plugin allows all certificates managed by Caddy’s built-in on-demand certificate issuance to be stored in Vault, facilitating easy (and secure) management of distributed Caddy servers to use the same central “certificate storage”.
Getting Started
Let’s build Caddy to include the Vault Storage plugin. After a while, the build completes, and you can now configure your Vault settings for Caddy.
xcaddy build
#
# WARNING: You will need 'go' installed on your $PATH, which is out of
# scope for this tutorial.
#
kmott@kyle-laptop:/tmp$ xcaddy build --output bin/caddy --with github.com/mywordpress-io/caddy-vault-storage@v0.1.1 --with github.com/mywordpress-io/certmagic-vault-storage@v0.1.1
2024/05/07 17:11:30 [INFO] Temporary folder: /tmp/buildenv_2024-05-07-1711.4062144593
2024/05/07 17:11:30 [INFO] Writing main module: /tmp/buildenv_2024-05-07-1711.4062144593/main.go
package main
import (
caddycmd "github.com/caddyserver/caddy/v2/cmd"
// plug in Caddy modules here
_ "github.com/caddyserver/caddy/v2/modules/standard"
_ "github.com/mywordpress-io/caddy-vault-storage"
_ "github.com/mywordpress-io/certmagic-vault-storage"
)
func main() {
caddycmd.Main()
}
2024/05/07 17:11:30 [INFO] Initializing Go module
2024/05/07 17:11:30 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go mod init caddy
go: creating new go.mod: module caddy
go: to add module requirements and sums:
go mod tidy
2024/05/07 17:11:30 [INFO] Pinning versions
2024/05/07 17:11:30 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go get -d -v github.com/caddyserver/caddy/v2
go: downloading github.com/caddyserver/caddy/v2 v2.7.6
go: downloading github.com/caddyserver/caddy v1.0.5
go: downloading github.com/caddyserver/certmagic v0.20.0
go: downloading github.com/quic-go/quic-go v0.40.0
go: downloading github.com/quic-go/qtls-go1-20 v0.4.1
go: added github.com/beorn7/perks v1.0.1
go: added github.com/caddyserver/caddy/v2 v2.7.6
go: added github.com/caddyserver/certmagic v0.20.0
go: added github.com/cespare/xxhash/v2 v2.2.0
go: added github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
go: added github.com/golang/protobuf v1.5.3
go: added github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1
go: added github.com/google/uuid v1.3.1
go: added github.com/klauspost/cpuid/v2 v2.2.5
go: added github.com/libdns/libdns v0.2.1
go: added github.com/matttproud/golang_protobuf_extensions v1.0.4
go: added github.com/mholt/acmez v1.2.0
go: added github.com/miekg/dns v1.1.55
go: added github.com/onsi/ginkgo/v2 v2.9.5
go: added github.com/prometheus/client_golang v1.15.1
go: added github.com/prometheus/client_model v0.4.0
go: added github.com/prometheus/common v0.42.0
go: added github.com/prometheus/procfs v0.9.0
go: added github.com/quic-go/qpack v0.4.0
go: added github.com/quic-go/qtls-go1-20 v0.4.1
go: added github.com/quic-go/quic-go v0.40.0
go: added github.com/zeebo/blake3 v0.2.3
go: added go.uber.org/mock v0.3.0
go: added go.uber.org/multierr v1.11.0
go: added go.uber.org/zap v1.25.0
go: added golang.org/x/crypto v0.14.0
go: added golang.org/x/exp v0.0.0-20230310171629-522b1b587ee0
go: added golang.org/x/mod v0.11.0
go: added golang.org/x/net v0.17.0
go: added golang.org/x/sys v0.14.0
go: added golang.org/x/term v0.13.0
go: added golang.org/x/text v0.13.0
go: added golang.org/x/tools v0.10.0
go: added google.golang.org/protobuf v1.31.0
2024/05/07 17:11:32 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go get -d -v github.com/mywordpress-io/caddy-vault-storage@v0.1.1 github.com/caddyserver/caddy/v2
go: downloading github.com/mywordpress-io/caddy-vault-storage v0.1.1
go: added github.com/mywordpress-io/caddy-vault-storage v0.1.1
go: added github.com/mywordpress-io/certmagic-vault-storage v0.1.1
go: upgraded github.com/onsi/ginkgo/v2 v2.9.5 => v2.12.1
go: upgraded golang.org/x/mod v0.11.0 => v0.12.0
go: upgraded golang.org/x/tools v0.10.0 => v0.12.0
go: added gopkg.in/resty.v1 v1.12.0
2024/05/07 17:11:33 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go get -d -v github.com/mywordpress-io/certmagic-vault-storage@v0.1.1 github.com/caddyserver/caddy/v2
2024/05/07 17:11:34 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go get -d -v
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.45.0
go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.21.0
go: downloading go.opentelemetry.io/otel v1.21.0
go: downloading go.opentelemetry.io/otel/sdk v1.21.0
go: downloading go.opentelemetry.io/otel/trace v1.21.0
go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.21.0
go: downloading google.golang.org/grpc v1.59.0
go: downloading go.opentelemetry.io/proto/otlp v1.0.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231016165738-49dd2c1f3d0b
go: downloading go.opentelemetry.io/otel/metric v1.21.0
go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.18.0
go: downloading github.com/golang/glog v1.1.2
2024/05/07 17:11:39 [INFO] Build environment ready
2024/05/07 17:11:39 [INFO] Building Caddy
2024/05/07 17:11:39 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go mod tidy
go: downloading cloud.google.com/go/iam v1.1.2
go: downloading google.golang.org/genproto v0.0.0-20231012201019-e917dd12ba7a
2024/05/07 17:11:40 [INFO] exec (timeout=-2562047h47m16.854775808s): /home/kmott/.files/bin/go-1.20.4-linux-amd64/bin/go build -o /tmp/bin/caddy -ldflags -w -s -trimpath
2024/05/07 17:12:09 [INFO] Build complete: bin/caddy
2024/05/07 17:12:09 [INFO] Cleaning up temporary folder: /tmp/buildenv_2024-05-07-1711.4062144593Configuration
#
# File: Caddyfile
#
{
skip_install_trust
auto_https disable_redirects
storage vault http://localhost:8200 {
token dead-beef
#approle_login_path <value>
#approle_logout_path <value>
#approle_role_id <value>
#approle_secret_id <value>
secrets_path secret
path_prefix caddy/certificates
#insecure_skip_verify <value>
#lock_timeout <value>
#lock_polling_interval <value>
}
}
example.com:10443 {
tls internal
respond "Hello, world!"
}Run Vault
Startup Vault in dev mode:
vault server -dev -dev-root-token-id=dead-beef
#
# Startup Vault in dev mode with a static token--DO NOT RUN IN PRODUCTION!
#
kmott@kyle-laptop:/tmp$ vault server -dev -dev-root-token-id=dead-beef
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Go Version: go1.17.5
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: inmem
Version: Vault v1.9.2
Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf
==> Vault server started! Log data will stream in below:
2024-05-08T08:47:09.874-0700 [INFO] proxy environment: http_proxy="\"\"" https_proxy="\"\"" no_proxy="\"\""
2024-05-08T08:47:09.874-0700 [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2024-05-08T08:47:09.877-0700 [INFO] core: Initializing VersionTimestamps for core
2024-05-08T08:47:09.877-0700 [INFO] core: security barrier not initialized
2024-05-08T08:47:09.877-0700 [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
2024-05-08T08:47:09.878-0700 [INFO] core: post-unseal setup starting
2024-05-08T08:47:09.881-0700 [INFO] core: loaded wrapping token key
2024-05-08T08:47:09.881-0700 [INFO] core: Recorded vault version: vault version=1.9.2 upgrade time="2024-05-08 08:47:09.881334307 -0700 PDT m=+0.058396820"
2024-05-08T08:47:09.881-0700 [INFO] core: successfully setup plugin catalog: plugin-directory="\"\""
2024-05-08T08:47:09.881-0700 [INFO] core: no mounts; adding default mount table
2024-05-08T08:47:09.882-0700 [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2024-05-08T08:47:09.883-0700 [INFO] core: successfully mounted backend: type=system path=sys/
2024-05-08T08:47:09.884-0700 [INFO] core: successfully mounted backend: type=identity path=identity/
2024-05-08T08:47:09.886-0700 [INFO] core: successfully enabled credential backend: type=token path=token/
2024-05-08T08:47:09.886-0700 [INFO] rollback: starting rollback manager
2024-05-08T08:47:09.886-0700 [INFO] core: restoring leases
2024-05-08T08:47:09.888-0700 [INFO] expiration: lease restore complete
2024-05-08T08:47:09.888-0700 [INFO] identity: entities restored
2024-05-08T08:47:09.888-0700 [INFO] identity: groups restored
2024-05-08T08:47:09.888-0700 [INFO] core: post-unseal setup complete
2024-05-08T08:47:09.889-0700 [INFO] core: root token generated
2024-05-08T08:47:09.889-0700 [INFO] core: pre-seal teardown starting
2024-05-08T08:47:09.889-0700 [INFO] rollback: stopping rollback manager
2024-05-08T08:47:09.889-0700 [INFO] core: pre-seal teardown complete
2024-05-08T08:47:09.890-0700 [INFO] core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2024-05-08T08:47:09.890-0700 [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2024-05-08T08:47:09.890-0700 [INFO] core: post-unseal setup starting
2024-05-08T08:47:09.890-0700 [INFO] core: loaded wrapping token key
2024-05-08T08:47:09.890-0700 [INFO] core: successfully setup plugin catalog: plugin-directory="\"\""
2024-05-08T08:47:09.891-0700 [INFO] core: successfully mounted backend: type=system path=sys/
2024-05-08T08:47:09.891-0700 [INFO] core: successfully mounted backend: type=identity path=identity/
2024-05-08T08:47:09.891-0700 [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2024-05-08T08:47:09.892-0700 [INFO] core: successfully enabled credential backend: type=token path=token/
2024-05-08T08:47:09.892-0700 [INFO] rollback: starting rollback manager
2024-05-08T08:47:09.892-0700 [INFO] core: restoring leases
2024-05-08T08:47:09.892-0700 [INFO] expiration: lease restore complete
2024-05-08T08:47:09.892-0700 [INFO] identity: entities restored
2024-05-08T08:47:09.892-0700 [INFO] identity: groups restored
2024-05-08T08:47:09.892-0700 [INFO] core: post-unseal setup complete
2024-05-08T08:47:09.892-0700 [INFO] core: vault is unsealed
2024-05-08T08:47:09.894-0700 [INFO] expiration: revoked lease: lease_id=auth/token/root/hff23aea39ae38c3c446eec9b3e1d4af89e6dcb37cbfb3da136c5dc634db8889b
2024-05-08T08:47:09.897-0700 [INFO] core: successful mount: namespace="\"\"" path=secret/ type=kv
2024-05-08T08:47:09.908-0700 [INFO] secrets.kv.kv_7a1cc022: collecting keys to upgrade
2024-05-08T08:47:09.908-0700 [INFO] secrets.kv.kv_7a1cc022: done collecting keys: num_keys=1
2024-05-08T08:47:09.908-0700 [INFO] secrets.kv.kv_7a1cc022: upgrading keys finished
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
$ export VAULT_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: ssiVZNyY3IPxaC7pu74huDzLKO0e7eu9psHpUEXWveI=
Root Token: dead-beef
Development mode should NOT be used in production installations!Run Caddy
Run Caddy with the new Caddyfile:
bin/caddy run –config Caddyfile
#
# Start 'caddy', pointing at our Caddyfile
#
kmott@kyle-laptop:/tmp$ bin/caddy run --config Caddyfile
2024/05/08 15:58:17.025 INFO using provided configuration {"config_file": "Caddyfile", "config_adapter": ""}
2024/05/08 15:58:17.027 INFO admin admin endpoint started {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/05/08 15:58:17.027 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc0007df300"}
2024/05/08 15:58:17.038 WARN http.auto_https automatic HTTP->HTTPS redirects are disabled {"server_name": "srv0"}
2024/05/08 15:58:17.039 INFO pki.ca.local root certificate trust store installation disabled; unconfigured clients may show warnings {"path": "storage:pki/authorities/local/root.crt"}
2024/05/08 15:58:17.039 INFO http enabling HTTP/3 listener {"addr": ":10443"}
2024/05/08 15:58:17.039 INFO failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024/05/08 15:58:17.039 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/08 15:58:17.039 INFO http enabling automatic TLS certificate management {"domains": ["example.com"]}
2024/05/08 15:58:17.041 INFO autosaved config (load with --resume flag) {"file": "/home/kmott/.config/caddy/autosave.json"}
2024/05/08 15:58:17.041 INFO serving initial configuration
2024/05/08 15:58:17.042 INFO tls cleaning storage unit {"storage": {}}
2024/05/08 15:58:17.044 INFO tls finished cleaning storage units
2024/05/08 15:58:17.044 INFO tls.obtain acquiring lock {"identifier": "example.com"}
2024/05/08 15:58:17.045 INFO tls.obtain lock acquired {"identifier": "example.com"}
2024/05/08 15:58:17.045 INFO tls.obtain obtaining certificate {"identifier": "example.com"}
2024/05/08 15:58:17.051 INFO tls.obtain certificate obtained successfully {"identifier": "example.com"}
2024/05/08 15:58:17.051 INFO tls.obtain releasing lock {"identifier": "example.com"}
2024/05/08 15:58:17.056 WARN tls stapling OCSP {"error": "no OCSP stapling for [example.com]: no OCSP server specified in certificate", "identifiers": ["example.com"]}Verify Caddy Certificate
After starting Vault and Caddy, you should be able to make HTTPS requests to localhost for example.com using curl, which will issue a new internal TLS certificate:
#
# When invoking curl, we need to tell it we have a specific hostname
# + IP address that it must connect to so the SNI headers will be
# set correctly.
#
# We also set '-k' because the TLS certificate is internal, and not
# publicly trusted (for testing purposes)
#
kmott@kyle-laptop:/tmp$ curl -ki https://example.com/test123 --connect-to example.com:443:127.0.0.1:10443
HTTP/2 200
alt-svc: h3=":10443"; ma=2592000
content-type: text/plain; charset=utf-8
server: Caddy
content-length: 13
date: Wed, 08 May 2024 16:21:16 GMT
Hello, world!Let’s also inspect the TLS certificate that comes back on the wire from Caddy:
#
# We connect to the Caddy listener using openssl, then parse the certificate
# response as an x509 certificate to display information about the cert.
#
kmott@kyle-laptop:/tmp$ openssl s_client -connect 127.0.0.1:10443 -servername example.com | openssl x509 -noout -text
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify return:1
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
37:0c:47:2f:82:42:66:53:05:a3:b9:6c:11:28:6b:64
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = Caddy Local Authority - ECC Intermediate
Validity
Not Before: May 8 15:58:17 2024 GMT
Not After : May 9 03:58:17 2024 GMT
Subject:
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:b3:71:37:7f:4c:89:46:c9:f5:da:54:d4:a4:69:
88:c7:5a:c8:89:f4:40:fe:1b:d5:13:85:ba:78:a5:
88:6a:7b:2b:9a:68:a2:31:80:e6:9b:2f:a6:25:5e:
85:c1:48:44:f2:eb:d0:e4:9b:79:42:a8:5f:93:73:
3b:ed:50:c1:65
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
92:1C:33:C8:A3:E3:D6:D5:AE:F1:ED:F2:28:3F:33:40:7F:3C:C4:7F
X509v3 Authority Key Identifier:
7A:80:EE:67:1D:13:93:CA:0D:23:05:0E:D0:B0:88:A3:06:D0:16:36
X509v3 Subject Alternative Name: critical
DNS:example.com
Signature Algorithm: ecdsa-with-SHA256
Signature Value:
30:45:02:20:11:a1:41:7f:ba:00:5b:cd:26:cd:69:84:7d:06:
18:5c:02:e8:b4:e5:e5:23:7c:b1:96:d1:b6:a6:7a:27:fc:ab:
02:21:00:dd:a0:50:4c:35:42:0d:32:2a:ab:1b:3a:ab:c0:f5:
59:87:4f:9d:1f:a8:b1:e2:19:3a:fb:93:65:44:18:fa:bfVerify Vault Certificate
This is great, but the whole point of the exercise is to make sure that the issues certificate is stored in Vault. Let’s inspect Vault (which should be stored at secret/caddy/certificate based on our Caddyfile settings:
#
# List the contents of Vault at the certificate storage path
#
kmott@kyle-laptop:/tmp$ export VAULT_ADDR=http://127.0.0.1:8200
kmott@kyle-laptop:/tmp$ export VAULT_TOKEN=dead-beef
kmott@kyle-laptop:/tmp$ vault kv list /secret/caddy/certificates
Keys
----
certificates/
last_clean.json
pki/
kmott@kyle-laptop:/tmp$ vault kv list /secret/caddy/certificates/certificates/local/example.com
Keys
----
example.com.crt
example.com.json
example.com.key
#
# Get the example.com.crt file, which should have fields that matches our previous test connection
# certificate using "curl"
#
kmott@kyle-laptop:/tmp$ vault kv get -format=json "/secret/caddy/certificates/certificates/local/example.com/example.com.crt" | jq -r '.data.data.certmagic.data | @base64d' | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
37:0c:47:2f:82:42:66:53:05:a3:b9:6c:11:28:6b:64
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = Caddy Local Authority - ECC Intermediate
Validity
Not Before: May 8 15:58:17 2024 GMT
Not After : May 9 03:58:17 2024 GMT
Subject:
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:b3:71:37:7f:4c:89:46:c9:f5:da:54:d4:a4:69:
88:c7:5a:c8:89:f4:40:fe:1b:d5:13:85:ba:78:a5:
88:6a:7b:2b:9a:68:a2:31:80:e6:9b:2f:a6:25:5e:
85:c1:48:44:f2:eb:d0:e4:9b:79:42:a8:5f:93:73:
3b:ed:50:c1:65
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
92:1C:33:C8:A3:E3:D6:D5:AE:F1:ED:F2:28:3F:33:40:7F:3C:C4:7F
X509v3 Authority Key Identifier:
7A:80:EE:67:1D:13:93:CA:0D:23:05:0E:D0:B0:88:A3:06:D0:16:36
X509v3 Subject Alternative Name: critical
DNS:example.com
Signature Algorithm: ecdsa-with-SHA256
Signature Value:
30:45:02:20:11:a1:41:7f:ba:00:5b:cd:26:cd:69:84:7d:06:
18:5c:02:e8:b4:e5:e5:23:7c:b1:96:d1:b6:a6:7a:27:fc:ab:
02:21:00:dd:a0:50:4c:35:42:0d:32:2a:ab:1b:3a:ab:c0:f5:
59:87:4f:9d:1f:a8:b1:e2:19:3a:fb:93:65:44:18:fa:bf
More Information
If you run into any issues, find bugs, or have questions, please feel free to contact us using the information below, or create a ticket in the GitHub project. We are happy to help!
- GitHub: https://github.com/mywordpress-io/caddy-vault-storage
- Signup: https://app.mywordpress.io/account/signup
- Documentation: https://docs.mywordpress.io/
- Contact: https://www.mywordpress.io/contact/
- Twitter: https://twitter.com/MyWordPressIO
- Discord: https://discord.gg/H5mWqsktj6
- YouTube: https://www.youtube.com/@MyWordPressIO
