Skip to main content

Troubleshooting Common Failure Modes

Most OLIVE 1.3 deployment problems are one of a small number of known situations, and nearly all of them share a property worth stating up front: they do not produce an error message. A network policy that matches nothing is not invalid, a Kyverno policy that selects no namespace is not broken, a service mesh value that disagrees with the cluster is not rejected. What you see instead is a timeout, a probe that never passes, a Job that waits forever, or a control that appears to be in place and checks nothing.

This page is the entry point. It gives the symptom, the usual cause, and where the detail lives.

It covers OLIVE, and assumes you know your way around Kubernetes and the components OLIVE depends on. Where Keycloak, Vault, Argo CD or a service mesh appear below, it is because an OLIVE requirement on them was not met — not because this is a manual for operating them. See When the problem is not in this list.

Narrow it down in this order

When a deployment misbehaves and you suspect the security configuration, work through the controls in the sequence given in Security Configuration. It is ordered by how likely each control is to be the cause and how cheap it is to rule out.


First: is it the platform or the cluster?​

Three commands separate the two before you start reading values files.

# Are the pods there at all, and in what state?
kubectl -n <tenant-namespace> get pods

# What did Kubernetes itself object to? Scheduling, probes, admission, image pulls.
kubectl -n <tenant-namespace> get events --sort-by=.lastTimestamp | tail -30

# Is capacity the limit?
kubectl top nodes

A pod that is Pending is a cluster problem: resources, storage, or scheduling. See Platform Scaling. A pod that is Running but not Ready is almost always a probe, and under an ambient mesh with default-deny policies that is almost always the health-probe rule. A pod in CreateContainerConfigError is a missing Secret or ConfigMap — and it produces no logs, which is why it is easy to misread.

kubectl get all does not mean all

It omits custom resources. Certificate, Middleware, VaultStaticSecret and the rest will not appear, so a namespace can look complete while the resources that actually make it work are missing. See Kubernetes Resources.


The platform is up but not working​

SymptomUsual causeWhere to look
No OLIALIVE1 events for a tenantThe liveness connector is not completing. It uses CouchDB, a file write, a file read and data orchestration, so the failure is in one of those — or the foundational bundle was never imported and the scheduler does not existVerify the Deployment
A rebuilt tenant onboards halfway and fails identically on every retryLeftover Vault state from the previous tenant of the same name. Vault outlives a namespace teardownUninstall and Teardown
A fresh install starts with old content in itA PersistentVolumeClaim survived the previous deployment and was re-boundUninstall and Teardown
A namespace stays in TerminatingA resource with a finalizer that cannot complete, often something created by a Job rather than rendered by a chartUninstall and Teardown

Network and connectivity​

SymptomUsual causeWhere to look
Nothing reaches the namespace after enabling network policiesnetworkPolicy.ingressController matches no pods — zero peers, and default-deny then closes every route in. On RKE2 the label is app.kubernetes.io/name=rke2-traefik, with the prefixNetwork Policies
Policies exist but nothing is actually blockedThe CNI does not enforce NetworkPolicy. Cilium and Calico do; not every CNI doesNetwork Policies
An infinite login loop, or every request returning 401, all at onceThe JWKS fetch is being dropped. Either keycloakIpBlocks pins an address that has changed, or Keycloak sits behind an internal load balancer and the default empty list carves its private range out. Resolve the Keycloak host from inside the cluster to tell whichNetwork Policies
Service-to-service calls time out after enabling the meshglobal.serviceMesh.mode disagrees with what is installed. The two modes need opposite policy changes, so the wrong value produces policies for the wrong traffic shapeService Mesh and mTLS
Liveness and readiness probes fail right after enrolling a namespace in ambientistio-cni SNATs node-local probes to 169.254.7.127/32; with default-deny and no rule for it, every probe fails. Dual-stack clusters also need the IPv6 equivalentService Mesh and mTLS
Pod creation fails in every namespace after turning on strict mTLSThe vault-agent-injector webhook is called by the kube-apiserver, which has no mesh identity. Apply the PERMISSIVE exemption before strict peer authenticationService Mesh and mTLS
A call to an internal address fails, public addresses workOLIVE restricts calls to internal addresses; the required address is not registeredContact olive-support@boc-group.com with the use case

Admission control and images​

SymptomUsual causeWhere to look
mode: enforce is set and unsigned images still deployThe mode settings only label the namespace. Without Kyverno installed and the policies deployed, nothing checks anything. Read kubectl get polr -n <namespace> yourself, and ask your cluster operator to confirm the policies are present and activeAdmission Policies
Policies exist, the namespace looks enrolled, nothing is evaluatedThe label the policy selects and the label tenant-setup applies do not match. A typo on either side is silentAdmission Policies
A tenant ran fine for weeks, then a node drain broke itAdmission acts on pod creation only. Switching to enforce evicts nothing; the failure arrives at the next rollout, scale-up or drain. A crash-looping container does not recreate the pod object, which is why the delay can be so longAdmission Policies

Vault, Keycloak and secrets​

SymptomUsual causeWhere to look
A Vault pod sits in CreateContainerConfigError with no logsWith credentials.source: secret, vault-kms-creds must exist before the first sync; the StatefulSet references it without optional: trueHashiCorp Vault
A fresh Vault logs security barrier not initialized and Vault is sealedExpected until the bootstrap Job has run. No Job at all is a sync-wave problem; a Job that ran and failed names its own cause; still sealed after a successful init is the seal itselfHashiCorp Vault
Tenant-to-Vault traffic is plaintext although the mesh says ambientThe Vault chart renders its own bare Namespace, so nothing enrols it. vaultMeshed only ever adds an allowed port, so an over-declared mesh is tolerated silentlyHashiCorp Vault
A new tenant onboarding fails to authenticate against KeycloakKC_BOOTSTRAP_ADMIN_PASSWORD is honoured only on the first start against an empty database. Changing the value in Vault afterwards makes the two disagree rather than changing the passwordKeycloak
Storage pods fail to authenticate after a re-syncA changed rotationId generated a new credential; the pods need to pick it upSecrets Rotation
Image pulls fail with an authentication errorThe olive-regcred mirror is stale or the registry credential has expiredSecrets Rotation

Argo CD and sync​

SymptomUsual causeWhere to look
A sync hangs forever at one wave, with a Job that never completesThe declared ordering has been overridden. The charts' own sync waves put the NetworkPolicy baseline before the Jobs it must permit; if a hand-rolled layout places a policy after the Job that needs it, Argo CD blocks on the wave until it is healthy and the policy can never arrive — the Job waits for the policy, the policy waits for the Job. Deploy the platform as the olive-platform umbrella chart, which carries the orderingInstall Argo CD
An Application never reaches Synced, or selfHeal reverts working stateMissing ignoreDifferences. Generated storage credentials, the payload the Vault Secrets Operator writes back, and runtime-injected environment all read as permanent driftInstall Argo CD
Argo CD refuses to create a resourceThe AppProject allow-list does not permit that kind or namespaceKubernetes Resources
Temporal workloads stay in ProgressingA race between the Temporal schema Job and CloudNativePG readinessTroubleshoot Temporal Deployment

Resources and scheduling​

SymptomUsual causeWhere to look
Pods stay PendingRequests no node can satisfy, or storage that cannot be provisionedResource Scaling
Pods are OOMKilledlimits.memory too low for the actual workloadResource Scaling
A service is slow but healthyCPU throttling against limits.cpuResource Scaling

When the problem is not in this list​

What this manual assumes​

This manual documents OLIVE. It assumes a working knowledge of Kubernetes, and of the components OLIVE depends on — Keycloak, HashiCorp Vault, Argo CD, and a service mesh if you run one. Where those components appear here, it is to say what OLIVE requires of them: the Vault bootstrap policy, the Keycloak realm and issuer, the sync waves and ignoreDifferences Argo CD needs, the ports a mesh has to allow. That contract is part of the platform, and you can hold us to it.

How to run the components themselves is not. OLIVE does not develop Keycloak, Vault, Argo CD, Istio or Kyverno, and does not troubleshoot them.

Where a question belongs​

QuestionGoes to
An OLIVE service misbehaves, or a documented OLIVE value does not do what this manual saysOLIVE support
A requirement in this manual is wrong, unclear, or changed between releases without being documentedOLIVE support — that is an OLIVE issue, even though it is about a third-party component
An OLIVE security event, log stream or the liveness checkOLIVE support
How to seal or unseal Vault, rotate its unseal keys, design its storage or run it in HAHashiCorp's documentation, and whoever operates your cluster
How to federate an identity provider in Keycloak, design realms, or configure MFA flowsKeycloak's documentation
How to structure Argo CD projects, repositories or RBAC, or why a sync behaves as it does in generalThe Argo CD documentation
How to install, tune or debug a service mesh, a CNI, an ingress controller or a storage providerThose projects, and whoever operates your cluster
Anything about the setup charts as software — maintenance, patches, upgradesNobody, by design: they are provided as-is. See Infrastructure Charts

A quick test that resolves most cases: would this still be a problem if OLIVE were not installed? If yes, it is a cluster or component question. If the component behaves exactly as its own documentation says and OLIVE is still unhappy, it is ours.

You are of course welcome to ask either way — we would rather hear about something than have you stuck. But a question about a third-party component will usually come back as a pointer to that project's documentation, and it is faster to start there.

What to send​

Collect this before contacting OLIVE support at olive-support@boc-group.com, because it is what any diagnosis starts from:

  1. The OLIVE version, the deployment mode (global.deploymentMode), and the security-relevant values: networkPolicy.enabled, serviceMesh.mode, imageVerification.mode, imageRegistry.mode.
  2. kubectl -n <tenant-namespace> get pods and the last 30 events.
  3. The logs of the failing pod, or — if it is in CreateContainerConfigError — the describe output, since there are no logs.
  4. Whether the cluster runs a mesh, which mode, and whether the CNI enforces NetworkPolicy.
  5. Which of the components you installed yourself rather than with the setup charts, and at which versions.
note

OLIVE ships no observability stack. It labels its log streams so that whoever configures your stack can route, retain and restrict them; it does not collect or alert on anything itself. See Security Logging and, for the event catalogue, Log Events and Automated Reactions.