Troubleshooting Common Failure Modes
Most OLIVE 1.3 deployment problems are one of a small number of known situations, and nearly all of them share a property worth stating up front: they do not produce an error message. A network policy that matches nothing is not invalid, a Kyverno policy that selects no namespace is not broken, a service mesh value that disagrees with the cluster is not rejected. What you see instead is a timeout, a probe that never passes, a Job that waits forever, or a control that appears to be in place and checks nothing.
This page is the entry point. It gives the symptom, the usual cause, and where the detail lives.
It covers OLIVE, and assumes you know your way around Kubernetes and the components OLIVE depends on. Where Keycloak, Vault, Argo CD or a service mesh appear below, it is because an OLIVE requirement on them was not met — not because this is a manual for operating them. See When the problem is not in this list.
When a deployment misbehaves and you suspect the security configuration, work through the controls in the sequence given in Security Configuration. It is ordered by how likely each control is to be the cause and how cheap it is to rule out.
First: is it the platform or the cluster?
Three commands separate the two before you start reading values files.
# Are the pods there at all, and in what state?
kubectl -n <tenant-namespace> get pods
# What did Kubernetes itself object to? Scheduling, probes, admission, image pulls.
kubectl -n <tenant-namespace> get events --sort-by=.lastTimestamp | tail -30
# Is capacity the limit?
kubectl top nodes
A pod that is Pending is a cluster problem: resources, storage, or scheduling. See Platform Scaling. A pod that is Running but not Ready is almost always a probe, and under an ambient mesh with default-deny policies that is almost always the health-probe rule. A pod in CreateContainerConfigError is a missing Secret or ConfigMap — and it produces no logs, which is why it is easy to misread.
kubectl get all does not mean allIt omits custom resources. Certificate, Middleware, VaultStaticSecret and the rest will not appear, so a namespace can look complete while the resources that actually make it work are missing. See Kubernetes Resources.
The platform is up but not working
| Symptom | Usual cause | Where to look |
|---|---|---|
No OLIALIVE1 events for a tenant | The liveness connector is not completing. It uses CouchDB, a file write, a file read and data orchestration, so the failure is in one of those — or the foundational bundle was never imported and the scheduler does not exist | Verify the Deployment |
| A rebuilt tenant onboards halfway and fails identically on every retry | Leftover Vault state from the previous tenant of the same name. Vault outlives a namespace teardown | Uninstall and Teardown |
| A fresh install starts with old content in it | A PersistentVolumeClaim survived the previous deployment and was re-bound | Uninstall and Teardown |
A namespace stays in Terminating | A resource with a finalizer that cannot complete, often something created by a Job rather than rendered by a chart | Uninstall and Teardown |
Network and connectivity
| Symptom | Usual cause | Where to look |
|---|---|---|
| Nothing reaches the namespace after enabling network policies | networkPolicy.ingressController matches no pods — zero peers, and default-deny then closes every route in. On RKE2 the label is app.kubernetes.io/name=rke2-traefik, with the prefix | Network Policies |
| Policies exist but nothing is actually blocked | The CNI does not enforce NetworkPolicy. Cilium and Calico do; not every CNI does | Network Policies |
| An infinite login loop, or every request returning 401, all at once | The JWKS fetch is being dropped. Either keycloakIpBlocks pins an address that has changed, or Keycloak sits behind an internal load balancer and the default empty list carves its private range out. Resolve the Keycloak host from inside the cluster to tell which | Network Policies |
| Service-to-service calls time out after enabling the mesh | global.serviceMesh.mode disagrees with what is installed. The two modes need opposite policy changes, so the wrong value produces policies for the wrong traffic shape | Service Mesh and mTLS |
| Liveness and readiness probes fail right after enrolling a namespace in ambient | istio-cni SNATs node-local probes to 169.254.7.127/32; with default-deny and no rule for it, every probe fails. Dual-stack clusters also need the IPv6 equivalent | Service Mesh and mTLS |
| Pod creation fails in every namespace after turning on strict mTLS | The vault-agent-injector webhook is called by the kube-apiserver, which has no mesh identity. Apply the PERMISSIVE exemption before strict peer authentication | Service Mesh and mTLS |
| A call to an internal address fails, public addresses work | OLIVE restricts calls to internal addresses; the required address is not registered | Contact olive-support@boc-group.com with the use case |
Admission control and images
| Symptom | Usual cause | Where to look |
|---|---|---|
mode: enforce is set and unsigned images still deploy | The mode settings only label the namespace. Without Kyverno installed and the policies deployed, nothing checks anything. Read kubectl get polr -n <namespace> yourself, and ask your cluster operator to confirm the policies are present and active | Admission Policies |
| Policies exist, the namespace looks enrolled, nothing is evaluated | The label the policy selects and the label tenant-setup applies do not match. A typo on either side is silent | Admission Policies |
| A tenant ran fine for weeks, then a node drain broke it | Admission acts on pod creation only. Switching to enforce evicts nothing; the failure arrives at the next rollout, scale-up or drain. A crash-looping container does not recreate the pod object, which is why the delay can be so long | Admission Policies |
Vault, Keycloak and secrets
| Symptom | Usual cause | Where to look |
|---|---|---|
A Vault pod sits in CreateContainerConfigError with no logs | With credentials.source: secret, vault-kms-creds must exist before the first sync; the StatefulSet references it without optional: true | HashiCorp Vault |
A fresh Vault logs security barrier not initialized and Vault is sealed | Expected until the bootstrap Job has run. No Job at all is a sync-wave problem; a Job that ran and failed names its own cause; still sealed after a successful init is the seal itself | HashiCorp Vault |
Tenant-to-Vault traffic is plaintext although the mesh says ambient | The Vault chart renders its own bare Namespace, so nothing enrols it. vaultMeshed only ever adds an allowed port, so an over-declared mesh is tolerated silently | HashiCorp Vault |
| A new tenant onboarding fails to authenticate against Keycloak | KC_BOOTSTRAP_ADMIN_PASSWORD is honoured only on the first start against an empty database. Changing the value in Vault afterwards makes the two disagree rather than changing the password | Keycloak |
| Storage pods fail to authenticate after a re-sync | A changed rotationId generated a new credential; the pods need to pick it up | Secrets Rotation |
| Image pulls fail with an authentication error | The olive-regcred mirror is stale or the registry credential has expired | Secrets Rotation |
Argo CD and sync
| Symptom | Usual cause | Where to look |
|---|---|---|
| A sync hangs forever at one wave, with a Job that never completes | The declared ordering has been overridden. The charts' own sync waves put the NetworkPolicy baseline before the Jobs it must permit; if a hand-rolled layout places a policy after the Job that needs it, Argo CD blocks on the wave until it is healthy and the policy can never arrive — the Job waits for the policy, the policy waits for the Job. Deploy the platform as the olive-platform umbrella chart, which carries the ordering | Install Argo CD |
An Application never reaches Synced, or selfHeal reverts working state | Missing ignoreDifferences. Generated storage credentials, the payload the Vault Secrets Operator writes back, and runtime-injected environment all read as permanent drift | Install Argo CD |
| Argo CD refuses to create a resource | The AppProject allow-list does not permit that kind or namespace | Kubernetes Resources |
Temporal workloads stay in Progressing | A race between the Temporal schema Job and CloudNativePG readiness | Troubleshoot Temporal Deployment |
Resources and scheduling
| Symptom | Usual cause | Where to look |
|---|---|---|
Pods stay Pending | Requests no node can satisfy, or storage that cannot be provisioned | Resource Scaling |
Pods are OOMKilled | limits.memory too low for the actual workload | Resource Scaling |
| A service is slow but healthy | CPU throttling against limits.cpu | Resource Scaling |
When the problem is not in this list
What this manual assumes
This manual documents OLIVE. It assumes a working knowledge of Kubernetes, and of the components OLIVE depends on — Keycloak, HashiCorp Vault, Argo CD, and a service mesh if you run one. Where those components appear here, it is to say what OLIVE requires of them: the Vault bootstrap policy, the Keycloak realm and issuer, the sync waves and ignoreDifferences Argo CD needs, the ports a mesh has to allow. That contract is part of the platform, and you can hold us to it.
How to run the components themselves is not. OLIVE does not develop Keycloak, Vault, Argo CD, Istio or Kyverno, and does not troubleshoot them.
Where a question belongs
| Question | Goes to |
|---|---|
| An OLIVE service misbehaves, or a documented OLIVE value does not do what this manual says | OLIVE support |
| A requirement in this manual is wrong, unclear, or changed between releases without being documented | OLIVE support — that is an OLIVE issue, even though it is about a third-party component |
| An OLIVE security event, log stream or the liveness check | OLIVE support |
| How to seal or unseal Vault, rotate its unseal keys, design its storage or run it in HA | HashiCorp's documentation, and whoever operates your cluster |
| How to federate an identity provider in Keycloak, design realms, or configure MFA flows | Keycloak's documentation |
| How to structure Argo CD projects, repositories or RBAC, or why a sync behaves as it does in general | The Argo CD documentation |
| How to install, tune or debug a service mesh, a CNI, an ingress controller or a storage provider | Those projects, and whoever operates your cluster |
| Anything about the setup charts as software — maintenance, patches, upgrades | Nobody, by design: they are provided as-is. See Infrastructure Charts |
A quick test that resolves most cases: would this still be a problem if OLIVE were not installed? If yes, it is a cluster or component question. If the component behaves exactly as its own documentation says and OLIVE is still unhappy, it is ours.
You are of course welcome to ask either way — we would rather hear about something than have you stuck. But a question about a third-party component will usually come back as a pointer to that project's documentation, and it is faster to start there.
What to send
Collect this before contacting OLIVE support at olive-support@boc-group.com, because it is what any diagnosis starts from:
- The OLIVE version, the deployment mode (
global.deploymentMode), and the security-relevant values:networkPolicy.enabled,serviceMesh.mode,imageVerification.mode,imageRegistry.mode. kubectl -n <tenant-namespace> get podsand the last 30 events.- The logs of the failing pod, or — if it is in
CreateContainerConfigError— thedescribeoutput, since there are no logs. - Whether the cluster runs a mesh, which mode, and whether the CNI enforces
NetworkPolicy. - Which of the components you installed yourself rather than with the setup charts, and at which versions.
OLIVE ships no observability stack. It labels its log streams so that whoever configures your stack can route, retain and restrict them; it does not collect or alert on anything itself. See Security Logging and, for the event catalogue, Log Events and Automated Reactions.