Skip to main content

HashiCorp Vault (vault-infra)

vault-infra is not part of the OLIVE platform

It is provided as-is to make setup easier — not maintained, patched or supported as part of the platform release, and not covered by OLIVE support. HashiCorp Vault is third-party software with its own licence, release cycle and security advisories; tracking and applying those is yours regardless of how you installed it.

What OLIVE does stand behind is the contract below. Meet it however you like.

Why OLIVE needs it​

Vault is the platform's source of truth for secrets. Every tenant's credentials, every connectivity authentication entry, the storage credentials and the Keycloak administrator password live there. Microservices authenticate to Vault with the requesting user's JWT, so Vault — not the application — decides what that caller may read.

Without a Vault that OLIVE can bootstrap tenants into, the platform does not start. This is the one component in this section that is not optional.


What the platform administrator specifies​

This chart has to deliver a Vault that meets OLIVE's contract: a KV v2 mount named secret, the Kubernetes auth method with disable_iss_validation, the bootstrap policy and role, and the address block every OLIVE chart reads. Those are not chart decisions — they are part of the platform, and they are listed in full in Infrastructure Prerequisites → HashiCorp Vault.

Read that first if you are setting Vault up without this chart. With the chart, the bootstrap Job writes all of it for you — but the values below still have to agree with what the platform administrator has specified.


Using the vault-infra chart​

The chart wraps the upstream vault chart (raft HA, three replicas, digest-pinned images) and optionally the Vault Secrets Operator, then runs a bootstrap Job that does everything above: initialise, store the key shares, unseal, form the raft quorum, enable the Kubernetes auth method and the KV v2 mount, and write the bootstrap policy and role. It can also render a CronJob that ships raft snapshots to S3.

Every step checks before it writes, so a re-sync against a healthy Vault is a chain of no-ops.

Wrapper chart​

apiVersion: v2
name: vault-infra-wrapper
description: Helm chart for setting up hashicorp vault
type: application
version: 1.0.0
appVersion: '1.0'
dependencies:
- name: vault-infra
version: 0.7.14
repository: oci://registry.gitlab.com/olive6224315/helm-charts/charts
condition: services.global.vault-infra

Values​

global:
namespace: 'oli-hashicorp-vault'
vault:
namespace: 'oli-hashicorp-vault'
serviceName: 'oli-vault'
port: 8200
scheme: 'http'
serviceMesh:
mode: ambient
vaultMeshed: true
# Node confinement. Set appArmor false on hosts without AppArmor (SELinux —
# RHEL, Rocky, OpenShift — and some managed clouds); seccomp stays on.
appArmor: true
seccomp: true
# Master switch for NetworkPolicy in this namespace. This chart's own
# networkPolicy.enabled is left unset and inherits it.
networkPolicy:
enabled: true
ingressController:
namespace: kube-system # 'traefik' for a separate install
podLabels:
app.kubernetes.io/name: rke2-traefik # 'traefik' upstream — no rke2- prefix

vault-infra:
customCA:
enabled: false
seal:
type: awskms
secretName: 'vault-seal-config'
awskms:
region: 'eu-central-1'
kmsKeyId: '<your-kms-key-id>'
credentials:
source: 'secret'
secretName: 'vault-kms-creds'
accessKeyIdKey: 'AWS_ACCESS_KEY_ID'
secretAccessKeyKey: 'AWS_SECRET_ACCESS_KEY'
vault:
enabled: true
fullnameOverride: 'oli-vault'
server:
extraSecretEnvironmentVars:
- envName: AWS_ACCESS_KEY_ID
secretName: vault-kms-creds
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: vault-kms-creds
secretKey: AWS_SECRET_ACCESS_KEY
rbac:
create: true
clusterRoleName: 'vault-secrets-operator-manager-role'
clusterRoleBindingName: ''
namespace:
create: true
syncOptions: 'Prune=false'
# Kyverno enrolment for this namespace — tenant-setup does not manage it.
# Values must match kyverno-infra's modes.*.labelValue; remove a line to
# opt out of that one policy.
labels:
olive/image-verification: 'audit'
olive/image-registry: 'audit'
olive/pod-privileges: 'audit'

services:
global:
vault-infra: true

vault.fullnameOverride and global.vault.serviceName must be the same string — the chart refuses to render if they disagree, because the upstream chart names the StatefulSet, the Services and every cluster-scoped object after it.

Seal options​

seal.type is shamir (manual unseal with key shares) or awskms (auto-unseal). Other KMS providers are configured the same way Vault documents them.

With credentials.source: secret, create the Secret before the first sync

The StatefulSet references it without optional: true. A pod whose secretKeyRef target is missing sits in CreateContainerConfigError — not CrashLoopBackOff, so there are no logs to read — and never becomes Ready.

kubectl -n oli-hashicorp-vault create secret generic vault-kms-creds \
--from-literal=AWS_ACCESS_KEY_ID=<id> \
--from-literal=AWS_SECRET_ACCESS_KEY=<key>

With credentials.source: instanceRole there is no such Secret and no prerequisite.

Switching an existing Vault from one seal type to another is a migration, not a values change.

Raft snapshots to S3​

Off by default (raftSnapshot.enabled). This is the backup of Vault's data: a CronJob logs in as its own Kubernetes auth role, runs vault operator raft snapshot save, verifies the file with snapshot inspect, and copies it to S3. The Job never deletes anything — retention is a bucket lifecycle rule.

vault-infra:
raftSnapshot:
enabled: true
schedule: '15 */6 * * *' # five-field cron, evaluated in timeZone
timeZone: 'Europe/Vienna'
volumeSize: 2Gi # must exceed the snapshot size
s3:
bucket: 'my-vault-backups'
prefix: 'vault-raft'
region: 'eu-central-1'
endpointUrl: '' # set for MinIO / Ceph RGW
sse: 'aws:kms' # aws:kms | AES256 | '' (none requested)
kmsKeyId: ''
aws:
credentialsMode: 'podIdentity' # podIdentity | irsa | secret
roleArn: '' # irsa only
existingSecret: '' # secret only — you create it

What has to exist outside the chart: the bucket (versioning and Object Lock recommended) with a lifecycle rule on the prefix, and an identity allowed s3:PutObject and s3:ListBucket on that prefix — plus kms:GenerateDataKey when sse: aws:kms. On RKE2 or any non-AWS S3 there is no workload identity, so use credentialsMode: secret with a Secret you create in the Vault namespace.

The Vault side is created by the bootstrap Job from a policy scoped to sys/storage/raft/snapshot and nothing else, bound to the snapshot ServiceAccount in that namespace.

A snapshot is only half of a restore

The other half is what unwraps it: the key shares under Shamir, or the KMS key plus recovery keys under auto-unseal. Neither is in the snapshot, and neither may be stored beside it.

In particular, never schedule the KMS key for deletion while snapshots made under it are still retained — the objects survive and become permanently unreadable.

Enabling this on an existing Vault needs root once

The bootstrap Job's own policy gains the right to create the snapshot policy and role only when raftSnapshot.enabled is true — and it deliberately cannot rewrite its own policy. On a fresh install this costs nothing, because the first run is root anyway. On an existing Vault the Job detects the drift and prints the exact commands rather than failing the sync; run them once with a root token minted from a quorum of key shares, and every later sync is hands-off.

NetworkPolicy​

global.networkPolicy.enabled is the master switch for every chart in the namespace; vault-infra.networkPolicy is left unset and inherits it. Rules are rendered for the bootstrap Job on the Vault port, for the API server (a port-only rule, because kubernetes.default.svc is a ClusterIP backed by node IPs and cannot be selected), and — with raftSnapshot on — for S3 and STS on 443.

The ingress-controller selector has to match real pods

global.networkPolicy.ingressController is rendered as one peer (namespaceSelector AND podSelector), so it means "the Traefik pods", not "every pod in that namespace". The default is RKE2's bundled Traefik in kube-system. Change both values for a separately installed Traefik, and check first:

kubectl get pods -n <traefik-namespace> --show-labels

A selector that matches nothing is not an error — it is zero peers, so every route behind it goes dark the moment default-deny lands.

On EKS, NetworkPolicy is only enforced when the VPC CNI add-on runs with enableNetworkPolicy=true, or Calico/Cilium is installed. Otherwise the policies are accepted and silently do nothing.

The internet-egress rules carve global.nonRoutableCidrs out of 0.0.0.0/0, so an egress rule cannot be satisfied by in-cluster traffic. Set raftSnapshot.networkPolicy.allowAwsPublicEndpoints: false and use extraEgress when S3 is reached through an interface VPC endpoint or an on-prem endpoint.

The namespace​

The chart renders its own Namespace rather than relying on Argo CD's CreateNamespace=true, which creates it bare — nothing labels it, so it is not enrolled in the service mesh even when global.serviceMesh.mode says ambient.

That mismatch never fails loudly: vaultMeshed only ever adds an allowed port to the NetworkPolicies, so an over-declared mesh is silently tolerated. What you actually get is tenant-to-Vault traffic running as plaintext on 8200, carrying Vault tokens and secrets, where you believed it was mTLS.

kubectl get ns oli-hashicorp-vault --show-labels # istio.io/dataplane-mode=ambient must be present
Prune=false on the namespace is load-bearing

Deleting the namespace cascades to everything in it: the Vault PVCs, the raft data, and the Secret holding the recovery keys. Under auto-unseal that is not recoverable from the KMS key alone. The chart refuses a syncOptions value that does not contain Prune=false.

Argo CD AppProject​

The project must list the Vault namespace in spec.destinations — and again for vault-secrets-setup and keycloak-infra, whose Jobs also run in the Vault namespace rather than in their own.

vault-infra also creates cluster-scoped objects, which a restricted project refuses with ... is not permitted in project:

clusterResourceWhitelist:
- group: ''
kind: Namespace # namespace.create: true
- group: 'rbac.authorization.k8s.io'
kind: ClusterRoleBinding # two: the bootstrap Job's, and the
# server's token-review auth delegator
# Only with the agent injector enabled - which it has to be, see below
- group: 'rbac.authorization.k8s.io'
kind: ClusterRole
- group: 'admissionregistration.k8s.io'
kind: MutatingWebhookConfiguration
# Only if this chart installs the Vault Secrets Operator subchart
# (vault-secrets-operator.enabled, which ships false)
- group: 'apiextensions.k8s.io'
kind: CustomResourceDefinition # the secrets.hashicorp.com CRDs

With the VSO subchart on, the project also needs ClusterRole and ClusterRoleBinding regardless of the injector: that subchart renders roughly thirty of them, including the aggregated view/edit roles for each of its CRDs.

Not created at the defaults: no PodSecurityPolicy (vault.global.psp.enable: false, and the Kind is gone from Kubernetes 1.25 anyway) and no CSI provider objects (vault.csi.enabled: false).

Whether the CRDs come from this chart is a decision, not a default

vault-secrets-operator.enabled ships false, so by default this chart installs Vault and not the operator — the operator is expected to already exist, as the one instance the cluster runs. Turning it on makes this chart the owner of the secrets.hashicorp.com CRDs, which is only safe once any separately installed VSO release has been removed and its CRD ownership annotations cleaned up. Decide which it is before writing the project, because the two shapes need different permissions.

Vault Secrets Operator: exactly one​

The Vault Secrets Operator (VSO) materialises Vault entries into Kubernetes Secrets. It has no watch-namespace option: every instance reconciles every VaultConnection, VaultAuth and VaultStaticSecret in the whole cluster, and its CRDs are cluster singletons that each instance re-claims on every sync.

Run one. It does not need duplicating: the Vault address lives on the VaultConnection resource, not on the controller, so a single operator can serve tenants on two different Vaults at once.

The Vault Agent Injector is required, and ships off​

The Vault Agent Injector is also a single cluster-wide mutating webhook, so the same "run exactly one" rule applies — but unlike the VSO it is off in the chart's defaults and has to be switched on:

vault:
injector:
enabled: true # ships as false

A deployment with authentication enabled does not work without it. The CouchDB and SeaweedFS pods read their Vault credential from a file the injected agent renders into the pod, so with no webhook in the cluster the pod is simply never mutated: the file never appears and the container waits for it. Nothing fails at render or sync time — the Helm release is healthy, Argo CD is green, and the only evidence is in the pod's own logs.

Because the webhook is cluster-wide, this is a cluster-operator action in the Vault installation, not something a tenant deployment can do for itself.

# The webhook exists
kubectl get mutatingwebhookconfiguration | grep -i vault

# The injector is running
kubectl -n oli-hashicorp-vault get pods -l app.kubernetes.io/name=vault-agent-injector

# A pod really was mutated — must print "injected"
kubectl -n <tenant-namespace> get pod <couchdb-pod> \
-o jsonpath='{.metadata.annotations.vault\.hashicorp\.com/agent-inject-status}'

The injected agent takes its Vault address from the injector's own configuration rather than from the pod, which is the second reason to run exactly one: with two Vaults in a cluster there is still one injector, and every injected agent dials whichever Vault that injector was configured for.

RBAC​

A cross-namespace binding must let the VSO ServiceAccount be used from tenant namespaces, and the ClusterRole it references must actually exist.

A RoleRef to a missing ClusterRole is accepted and grants nothing

The API server does not validate that the referenced ClusterRole exists. A typo produces a binding that looks correct and silently permits nothing. Verify:

kubectl get clusterrole | grep vault-secrets-operator

Applying the bootstrap policy by hand​

kubectl exec -i <vault-pod> -n <namespace> -- \
sh -c 'cat > /tmp/vault-bootstrap-policy-v2.hcl' < vault-bootstrap-policy.hcl

kubectl exec -it <vault-pod> -n <namespace> -- \
vault policy write vault-bootstrap-policy-v2 /tmp/vault-bootstrap-policy-v2.hcl

Vault reads a policy at request time, so re-applying it takes effect immediately — no restart anywhere, and no already-issued token keeps the old rules. That also means a policy change is safe to roll out while the platform is running.

The chart's own Job identity is a separate policy

vault-infra also writes a second, narrower policy for the bootstrap Job itself, so that the Job stops re-authenticating as root on every sync. That one is an implementation detail of the chart — if you are setting Vault up by hand, you do not need it, because there is no Job. You only need the bootstrap policy and role above.

disable_iss_validation is not optional

Kubernetes issues service-account tokens whose iss claim need not match anything Vault can discover. With strict issuer validation on, every tenant login fails, with an error that points at the token rather than at this setting. It is one of the most expensive hours you can spend on a Vault setup.


First install​

  1. Create the namespace, and the AWS credential Secret if you use credentials.source: secret.
  2. Fill in the values and check the Argo CD AppProject destinations.
  3. Sync.

A fresh cluster looks broken until the bootstrap Job runs. All three pods log security barrier not initialized and failed to get raft challenge … Vault is sealed, because none of them can lead a raft cluster that was never initialised. That is expected. Under auto-unseal, the line that tells you the seal loaded correctly is:

core: stored unseal keys supported, attempting fetch

Nothing needs unsealing by hand under auto-unseal — there are no unseal keys.

If it stays that way:

kubectl -n <namespace> get jobs
kubectl -n <namespace> logs job/vault-bootstrap-<checksum> --all-containers

No Job at all means a sync-wave problem. A Job that ran and failed names its own cause. Pods still sealed after a successful init is the seal itself — check the Vault pod logs for the AWS error.

Two things the Job leaves to you​

Copy the key shares out of the vault-init-keys Secret into your password manager. The Job prints an ACTION REQUIRED banner with the command. Under auto-unseal these are recovery keys: they cannot unseal anything, but a quorum of them mints a root token, so guard them identically.

Create olive-regcred in the new Vault. The Job prints that command too.

Nothing in the chart ever deletes the key shares. A root token is re-mintable from the shares; the shares are re-mintable from nothing.

The initial root token is revoked for you

The root token is no longer a standing credential. From the second sync onwards the bootstrap Job authenticates as its own ServiceAccount, so the root token is needed exactly once — and bootstrap.rootToken.revokeAfterBootstrap (on by default) revokes it at the end of a successful first run and removes its copies from etcd.

That is guarded three ways: it only happens on a run that actually used the root token, only after the Job's own role exists and a test login against it succeeded, and only for entries whose credential is already revoked. A failure at any point leaves everything as it was.

Set it false to keep a root token in the key Secret indefinitely. Note that bootstrap.selfRole.enabled: false requires it to be false as well — the chart refuses the other combination, because it would lock itself out of Vault.