Skip to main content

Secrets Rotation

OLIVE holds two quite different kinds of secret, and they are rotated by different people using different tools. Getting the distinction right is most of the work:

Connectivity secretsInfrastructure secrets
What they areThe credentials your connectivity uses to reach other systems: OIDC and OAuth2 client credentials, basic auth, API keys, tokens, certificatesThe credentials the platform itself runs on: storage credentials, registry pull credentials, the identity provider's administrator credential, Vault's own keys
Who owns themConnectivity engineers and editors, per tenantThe platform administrator
Where they liveHashiCorp Vault, in the tenant's secret spaceHashiCorp Vault for most; a smaller set as conventional Kubernetes Secrets
How they are rotatedIn the connectivity editorPer secret — see the inventory below

HashiCorp Vault is the source of truth for tenant secrets and for most infrastructure secrets. Some infrastructure secrets exist only as Kubernetes Secrets, and for those the important question is not whether a Secret exists in etcd but where its authoritative copy lives — that is what decides whether losing it is harmless, self-healing, or destructive.


Connectivity secrets​

Authentication entries — OIDC, OAuth2, basic auth, API keys and the like — are part of the connectivity definitions. They are created and maintained in the connectivity editor, and rotating one means editing its value there: open the authentication entry, replace the credential, save.

Nothing else is needed. The value is stored in Vault under the tenant's secret space and read at execution time, so the next execution of a datapoint or connector uses the new value. There is no pod restart, no sync, and no administrator involvement.

Two consequences worth passing on to your connectivity engineers:

  • Rotate at the source system first, or in step. OLIVE holds a copy of a credential that some other system issues. Changing it in OLIVE does not change it at the target — and changing it at the target without updating OLIVE breaks every datapoint using it.
  • An exported OCB can carry these values. Authentication data is included in an OLIVE Connectivity Bundle, in plain text unless the export was encrypted with a password. Treat bundles accordingly, and prefer rotating a credential that has travelled in an unencrypted bundle.

See "Authentication and Secrets Management" in the OLIVE connectivity developer handbook.


Infrastructure secrets​

These are the platform administrator's responsibility. The mechanism differs per secret, so the inventory below is organised by how you rotate it rather than by namespace.

Rotated with rotationId — storage credentials​

CouchDB and SeaweedFS credentials are generated once by the secret-setup Job and then left alone, which is what makes it safe to re-sync those charts at any time. rotationId is the switch that asks for a new credential:

couchdb:
credentials:
rotationId: '2026.09.04'

seaweedfs:
credentials:
rotationId: '2026.09.04'

Change the string to any new value — a date, a ticket ID, a counter — and sync. In one pass:

  • The Job sees a rotationId different from the one stored in the Vault secret, generates a new username and password, and writes them to secret/shared-secrets-<tenant>/couchdb-credentials (respectively …/seaweedfs-credentials).
  • The StatefulSet's pod annotation carries the same rotationId, so the pods roll and read the new credential at start.

Leaving the value unchanged means "do nothing", so an ordinary sync is unaffected. An empty value is the pre-rotation state: credentials written before this feature carries no rotationId, and the Job leaves them alone.

There is no account to create or delete anywhere. The service takes the credential from its startup configuration, which is rendered fresh from Vault into a memory-backed volume on every container start; the old credential ceases to exist when the old pod does.

Expect a short outage

The storage service is unavailable for the length of the pod restart, and any consumer still holding the previous credential is rejected until it re-reads Vault. The SDK refetches and retries once on an authentication failure, so a rolling restart of the consumers is not required — but the restart window is a real, if short, outage. Rotate off-peak.

Use a date or a ticket ID rather than an incrementing number, so the value records when and why it was rotated.

Rotated in Vault — credentials mirrored by the Vault Secrets Operator​

These exist as Kubernetes Secrets only as a mirror. Vault holds the authoritative copy, and the Vault Secrets Operator (VSO) re-reads it on its refresh interval.

SecretVault pathRotation
olive-regcred — container registry pull credentialsecret/shared-secrets/olive-regcred (cluster-wide, not tenant-scoped)Issue a new token in the registry, write it to the Vault path; VSO re-syncs the Kubernetes Secret within its refresh interval

Deleting the Kubernetes Secret is harmless here — VSO recreates it. Changing the Kubernetes Secret directly is pointless: it is overwritten on the next sync.

couchdb.credentials.staticSecretRefresh and the equivalent SeaweedFS value control how often the mirror re-reads Vault in non-auth deployments. A shorter interval means a rotation made directly in Vault is picked up sooner.

Renewed automatically — no action​

SecretMechanism
vault-client-token (single-no-auth deployments only)Renewed by the vault-client-token-renew CronJob on a daily schedule. Authenticated deployments do not use it at all — each service logs in to Vault per tenant with the caller's JWT
<svc>-express-hash-token (one per Express service)Generated at first render and then preserved across upgrades. Not an external credential — nothing outside the cluster knows it

To force a new Express hash token, delete the Secret and re-sync: the template only reuses a value it finds, so a new one is generated. Do this deliberately — the token backs CSRF protection and internal hashing, so rotating it invalidates values derived from the old one.

Owned by an operator — do not rotate by hand​

SecretOwnerRotation
*-tls (ingress and host certificates)cert-managerRe-issued automatically via ACME before expiry. To force renewal, delete the Secret — cert-manager re-issues it
*-pg-* (Temporal and Keycloak PostgreSQL)CloudNativePGThe operator rotates its TLS material itself. Do not touch or archive these

Rotated outside OLIVE​

SecretSource of truthRotation
vault-kms-creds (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)AWS IAMRotate the IAM access key in AWS, then recreate the Kubernetes Secret with the new values and restart the Vault pods so they pick it up
This Secret only exists with AWS KMS auto-unseal, using static keys

vault-kms-creds is not part of every OLIVE installation. It exists only where Vault was initialised with AWS KMS auto-unseal and is configured to authenticate with a static IAM access key held in a Kubernetes Secret.

It does not exist, and there is nothing to rotate here, when:

  • Vault authenticates to AWS KMS through an instance role or IRSA instead of static keys — the credential is issued and rotated by AWS, not stored by OLIVE. This is the default.
  • Vault uses a different auto-unseal provider — GCP Cloud KMS, Azure Key Vault, Vault transit. Each has its own credential, held wherever that provider expects it, and rotated per that provider's practice.
  • Vault uses Shamir unseal (manual unseal with key shares) — there is no cloud credential at all; see Vault's own keys.

No chart template renders this Secret; it is created by hand when the deployment needs it. If you are not sure which applies to your installation, check the seal stanza in the Vault configuration before looking for a Secret that may not be there.


Rotated in Keycloak first, then in Vault — the Keycloak administrator credential​

oli-keycloak-admin holds the password of Keycloak's admin user. It is a Vault Secrets Operator mirror of the Vault KV entry at secret/<keycloakVaultPath> (by default keycloak/keycloak-credentials), field password — the same entry the realm-import Job reads when it authenticates for tenant onboarding.

The order matters, and Vault is not where the password actually lives

Keycloak reads KC_BOOTSTRAP_ADMIN_PASSWORD only on the first start against an empty database. After that, the authoritative copy of the password is inside Keycloak's own database. Writing a new value into Vault does not change it — it only makes Vault disagree with Keycloak, and the next onboarding run then fails to authenticate.

Change it in Keycloak first, then write the new value to Vault.

  1. Change the password of the admin user in Keycloak — in the admin console, or through Keycloak's admin REST API.

  2. Write the new value to the Vault KV entry, under the password field:

    vault kv put secret/keycloak/keycloak-credentials password='<new-password>'

    Use the path your deployment sets in global.keycloakVaultPath, relative to the KV mount.

  3. Wait for the Vault Secrets Operator to re-read Vault (vault.refreshAfter, one hour by default) or delete the Kubernetes Secret to force an immediate re-sync.

  4. Verify that a tenant onboarding sync still completes — that Job is the main consumer and the first thing that breaks if the two copies disagree.

No Keycloak restart is needed, and none is triggered. The chart deliberately sets no rollout restart target: restarting Keycloak would cost every in-flight login and change nothing, because the bootstrap environment variable is not read again.

The seed Job cannot rotate this for you

On a fresh install a seed Job creates the KV entry once, with a generated password, using vault kv put -cas=0 and a Vault role that grants create but not update. That is deliberate: the password Keycloak stored on first boot exists nowhere else, so an automated overwrite would lock the onboarding Job out of Keycloak with no way back. Rotation is therefore a manual, two-step operation by design.

Rotated by re-running onboarding — the tenant's auth-proxy credentials​

<tenant>-oauth2-proxy-secret carries five values: client-id (the literal string olive-gateway), client-secret (the olive-gateway client's), cookie-secret, redis-secret, and purge-client-secret (the olive-session-purge service-account client used by the session-purge CronJob, whose only permission is realm-management: view-events on that realm).

The tenant realm has four clients, and only these two have their secrets reconciled into Vault. olive-gateway-user and olive-gateway-admin exist in Keycloak only — regenerating one of those is a Keycloak-side operation with no Vault step, and no keycloak-setup re-sync. See Tenant Administrators → The Clients.

They are not rotated by editing Vault. The realm-import Job reconciles them on every sync: it reads the live client secrets from Keycloak, compares them with what Vault holds, and rewrites the Vault entry when they differ.

To rotate the client secrets:

  1. Regenerate the client secret in Keycloak, in the tenant's realm — olive-gateway, or olive-session-purge, or both (Clients → the client → Credentials → Regenerate).
  2. Re-sync the keycloak-setup Helm release so the realm-import Job runs again. It picks up the live value and writes the whole Vault entry.
  3. The Vault Secrets Operator mirrors it into the <tenant>-oauth2-proxy-secret Kubernetes Secret.
  4. Restart the consumers. All five values are injected as environment variables and are only read at pod start, and no rollout restart is configured: restart the oauth2-proxy and redis Deployments in the tenant namespace. The session-purge CronJob picks the new value up on its next run.

cookie-secret and redis-secret have no switch of their own. The Job regenerates both whenever the olive-gateway client secret has changed, or whenever either is missing from Vault. If only the purge client secret moved, they are deliberately preserved — the Job detects that case explicitly and reuses them.

Note that vault kv put replaces the whole entry, so the Job writes all five keys on any change. That is why a drifted purge-client-secret alone is enough to trigger the write.

Rotating the cookie secret logs everybody out

A new cookie-secret invalidates every active session in that tenant — every user has to log in again. This is why an ordinary no-op sync never regenerates it, and why rotating the olive-gateway client secret is not a quiet operation. Plan it like the storage rotation above: off-peak, announced.

Vault's own keys​

vault-init-keys holds Vault's root token and its unseal or recovery key shares. Rotating these is a Vault operation, not an OLIVE setting: refer to HashiCorp Vault's own documentation and practices for rekeying, unseal-key rotation and root-token generation.

Two things specific to this deployment are worth knowing before you start:

  • etcd may hold the only remaining copy of the key shares. Nothing reconciles this Secret, so any archived copy has to be updated in the same operation.
  • From vault-infra 0.6.5 the bootstrap Job revokes the initial root token once it has created and verified its own Kubernetes auth role, and deletes every copy of it from etcd. The key shares are never touched by automation.

Before you rotate anything​

  1. Know where the authoritative copy lives. Vault, the Helm release, a bootstrap Job, an operator, or the component's own database — the answer decides whether deleting the Kubernetes Secret is harmless, self-healing, or destructive, and whether writing a new value into Vault changes anything at all.
  2. Know whether the consumer re-reads it. Values injected as environment variables are read once, at pod start. Where the tables above say to restart a workload, nothing will restart it for you.
  3. Rotate off-peak where a pod restart or a session reset is involved.
  4. Record it. Use the rotationId value to carry a date or a ticket ID, so the "when and why" is visible in the values file rather than only in someone's memory.
  5. Verify afterwards. Confirm the consuming workloads came back and that the security log shows secret access succeeding (OLISEC1) rather than failing (OLISEC3, OLISEC4).