Skip to main content

Service Mesh (service-mesh-setup)

Optional. Skip this entire page if global.serviceMesh.mode is none.

service-mesh-setup is not part of the OLIVE platform

It is provided as-is to make setup easier — not maintained, patched or supported as part of the platform release, and not covered by OLIVE support. Istio is third-party software with its own licence, release cycle and security advisories; tracking and applying those is yours regardless of how you installed it.

What OLIVE does stand behind is the contract below. Meet it however you like.

Why OLIVE offers it​

Traffic between the cluster and external systems is already encrypted. Traffic between workloads inside the cluster is plaintext unless a service mesh provides mutual TLS. Enabling the mesh is what lets you say data is protected in transit inside the cluster too, not only at the edge.

OLIVE supports Istio, and the evaluated shape is ambient mode — a per-node ztunnel DaemonSet rather than a per-pod sidecar. See Cluster Operator → Service Mesh and mTLS for why.

Ambient is the only tested mode

The charts also render a sidecar policy shape, but no OLIVE release has been validated on a sidecar mesh end to end. If the cluster you are asked to prepare already runs sidecars, say so before the deployment is planned rather than after.

This chart does not install Istio

Install istio-base, istiod, istio-cni and ztunnel yourself first. This chart configures a mesh that is already running.


What the platform administrator specifies​

Almost nothing — this is the most operator-owned of these charts. The Administrator sets global.serviceMesh.mode to match what is actually installed, and vaultMeshed to match whether the Vault namespace is enrolled. A wrong value there renders the policy set for the other mode, silently. See Infrastructure Prerequisites → Service mesh.


What the mesh has to provide​

Whether you use this chart or configure the mesh yourself, these are the things that have to be right. Most of them fail silently.

One mesh-wide mTLS policy, starting PERMISSIVE​

Between two enrolled pods, ztunnel already uses mTLS. PERMISSIVE does not weaken that — what STRICT adds is rejecting plaintext from anything unmeshed. Every exemption exists because some unmeshed component legitimately needs to reach a meshed pod.

Stay PERMISSIVE until every exemption is applied and proven. Better still, phase STRICT in one namespace at a time: set mtls.mesh.enabled: false and list the namespaces in mtls.perNamespace, which renders a PeerAuthentication per namespace instead of one mesh-wide object.

mtls:
mesh:
enabled: true
mode: PERMISSIVE # -> STRICT
perNamespace: []
# - namespace: tenant-a
# mode: STRICT
acknowledgeUnmeshedGateway: false
STRICT with an unenrolled ingress controller is refused

PeerAuthentication is a destination-side policy, so exemptions.traefik governs traffic into Traefik, not Traefik into your workloads. If Traefik is not itself enrolled, STRICT makes every meshed workload refuse its plaintext and the platform goes dark from outside while looking healthy inside.

The chart refuses to render that combination unless you have added a per-workload exemption for every workload Traefik dials and set mtls.acknowledgeUnmeshedGateway: true.

Exemptions for components that are never in the mesh​

ComponentWhy it needs an exemption
The kube-apiserver, calling any admission webhookNever in the mesh, has no SPIFFE identity
vault-agent-injectorA mutating webhook called by the apiserver. Under mesh-wide STRICT, pod creation then fails in every tenant namespace that runs CouchDB or SeaweedFS — which is every tenant, because their pod templates carry the injection annotation
Kyverno's admission webhookSame reason
The ingress controller, if not itself enrolledSTRICT makes every meshed workload refuse Traefik's plaintext, and the platform goes dark from outside while looking healthy inside
The CloudNativePG operator, reaching the Postgres instance status APIcnpg-system is not meshed, so it arrives as plaintext. CNPG then loses visibility of its instances and reports the cluster unhealthy while Postgres keeps serving — which reads as an operator bug
The Vault Secrets Operator, Prometheus and other scrapers, Argo CD, backup jobsAnything that reaches meshed pods from outside the mesh. Check these before STRICT; the chart has an exemptions.extra list for them

Kubelet health probes do not belong here — they go through the istio-cni SNAT path and the baseline NetworkPolicy, not through PeerAuthentication.

The CNPG exemption is port-scoped​

exemptions.cnpg is scoped to port 8000 (the instance status API) so that client connections to Postgres on 5432 still require mTLS. It needs one entry per namespace running a CNPG cluster, and the chart refuses an empty namespaces list rather than rendering nothing silently.

A pooler needs its own selector: pgbouncer pods carry cnpg.io/cluster but not cnpg.io/podRole: instance, so the default selector misses them entirely.

The ingress controller on RKE2 and elsewhere​

The ingress-related values of service-mesh-setup default to RKE2's bundled Traefik. On a cluster with a separately installed Traefik (EKS, AKS, …) change them as follows:

ValueRKE2 bundled Traefik (default)Separately installed Traefik (upstream chart, release traefik)
exemptions.traefik.namespacekube-systemtraefik
exemptions.traefik.selectorapp.kubernetes.io/name: rke2-traefikapp.kubernetes.io/name: traefik
exemptions.traefik.ports[8000, 8443][8000, 8443] — the upstream defaults, unless you changed ports.*.port
traefikEnrollment.methodhelmChartConfigexternal
Traefik identity in AuthorizationPolicy rulescluster.local/ns/kube-system/sa/rke2-traefikcluster.local/ns/<traefik-namespace>/sa/<traefik-serviceaccount>

traefikEnrollment.method: helmChartConfig enrols Traefik by rendering a helm.cattle.io/v1 HelmChartConfig for the bundled chart. That resource type exists only on RKE2 / K3s; elsewhere the sync fails with no matches for kind HelmChartConfig. With method: external the chart renders nothing for Traefik but still treats it as enrolled for its STRICT safety check, and you add the enrollment label in your own Traefik values:

# values of your Traefik installation (upstream chart), ambient mode
deployment:
podLabels:
istio.io/dataplane-mode: ambient

method is available from service-mesh-setup 0.5.0.

L4 authorization for the storage services​

This one is specific to ambient mode and easy to miss.

Under ambient, every meshed connection is tunnelled over HBONE on port 15008, and the real destination port travels inside the HTTP/2 CONNECT header where the CNI cannot see it. A NetworkPolicy allowing 15008 to a workload therefore no longer constrains which port of that workload the caller reaches.

Two concrete regressions:

  • seaweedfs opens only the S3 port and deliberately leaves the filer (8888), master (9333) and volume (8080) ports closed — they are unauthenticated.
  • couchdb scopes the Erlang ports (4369, 9100–9200) to couchdb-to-couchdb traffic.

Once both ends are meshed, neither restriction binds. An AuthorizationPolicy on SPIFFE identity and destination port restores them — which is strictly stronger than the IP-based rule it replaces. Enable it together with storage enrollment, not afterwards.

Health probes​

Once a pod is enrolled, kubelet probes no longer arrive from the node IP: istio-cni SNATs node-local packets to 169.254.7.127/32. With default-deny NetworkPolicies and no rule for it, every liveness and readiness probe fails. This is the most common ambient rollout failure. Dual-stack clusters also need fd16:9254:7127:1337:ffff:ffff:ffff:ffff/128.

Ports​

PortPurpose
TCP 15008HBONE tunnel between ambient workloads
TCP 15012Sidecar proxies reaching istiod — sidecar mode only, which is not a tested OLIVE configuration

Using the service-mesh-setup chart​

Wrapper chart​

apiVersion: v2
name: service-mesh-setup-wrapper
description: Helm chart for configuring service mesh
type: application
version: 1.0.0
appVersion: '1.0'
dependencies:
- name: service-mesh-setup
version: 0.5.3
repository: oci://registry.gitlab.com/olive6224315/helm-charts/charts
condition: services.global.service-mesh-setup

Values​

global:
serviceMesh:
mode: ambient
appArmor: true # false on SELinux hosts
seccomp: true

services:
global:
service-mesh-setup: true

Rendering with stock values produces exactly one object: a PERMISSIVE mesh-wide PeerAuthentication. Everything else — the exemptions, the storage authorization policies, Traefik enrollment — is opt-in.


What the Argo CD project must permit​

This chart is the opposite of the other four: at its defaults it creates no cluster-scoped objects at all. Everything it writes is namespaced — the catch is that almost none of it lands in its own namespace, so for this chart spec.destinations matters more than the whitelist.

Where it writesWhatWhen
istio-system (istioNamespace)security.istio.io PeerAuthentication — the mesh-wide mTLS policy and the exemptionsAlways
The storage namespacessecurity.istio.io AuthorizationPolicy — the L4 rules for CouchDB and SeaweedFSPer entry in authorization.storage
kube-systemhelm.cattle.io HelmChartConfig — enrolling RKE2's bundled TraefiktraefikEnrollment.enabled, which ships false
istio-systemServiceAccount, Role, RoleBinding, ConfigMap, CronJobWith the cold-start reconcile job

Each of those namespaces has to be a destination in the project, and a namespace that is missing fails the sync with ... is not permitted in project rather than anything that mentions the mesh.

The one exception is the cold-start reconcile CronJob. It needs cluster-wide get/list/delete on pods and get/list on nodes — it restarts workloads whose ztunnel was not ready when they started — so turning it on adds two Kinds:

# Only with global.serviceMesh.coldStartReconcile.enabled (or the chart-local
# coldStartReconcile.enabled). Unset means off.
clusterResourceWhitelist:
- group: 'rbac.authorization.k8s.io'
kind: ClusterRole
- group: 'rbac.authorization.k8s.io'
kind: ClusterRoleBinding

It does not create the istio-system namespace or install Istio. Both are prerequisites, and security.istio.io has to exist before the first sync — a missing CRD group presents as an unknown resource and a part-way sync, not as a permissions error:

kubectl get crd | grep security.istio.io

Rollout order​

Never enable an exemption and flip STRICT in the same commit. If the selector is wrong you get STRICT with no working exemption and no signal until traffic is already being refused.

  1. Install Istio. Nothing is enrolled; zero traffic impact.
  2. Deploy this chart with defaults — PERMISSIVE mesh policy, nothing else.
  3. Enrol one tenant namespace. Verify with istioctl ztunnel-config workload -n <ns>.
  4. Enable the storage authorization policies for that namespace, alongside its storage.
  5. Enrol the ingress controller and apply its exemption, in the same step. Test end-to-end from outside the cluster.
  6. Enable the remaining exemptions. Verify each selector matches real pods.
  7. Only then switch the mesh policy to STRICT, or phase it in per namespace.

Verify a selector before depending on it​

A PeerAuthentication whose selector matches nothing is not an error — it is zero coverage, discovered at cutover.

kubectl -n kube-system get pods -l app.kubernetes.io/name=rke2-traefik # or your Traefik namespace / labels
kubectl -n oli-hashicorp-vault get pods -l app.kubernetes.io/name=vault-agent-injector
istioctl ztunnel-config workload

Known sharp edges​

  • Port-level mTLS takes container ports, not host ports. Traefik listens on 8000/8443 inside its network namespace; 80/443 are the host ports. Using the latter matches nothing.
  • ztunnel is L4 only. An AuthorizationPolicy containing L7 attributes — HTTP method, path, headers — that lands on a ztunnel fails safe by becoming a DENY. It does not degrade to allowing traffic. L7 needs a waypoint proxy.
  • ALLOW policies deny what they do not match. Any port a workload legitimately serves must appear in some rule. This is why the storage cluster-replication rules are deliberately not port-scoped: Erlang distribution uses a port range, and the ports field takes a list.
  • Keep Jobs and CronJobs out of the mesh in sidecar mode. A classic sidecar keeps running after the main container exits, so the pod never reaches Completed: the Job hangs forever, CronJobs pile up stuck pods, and an Argo CD sync waiting on the Job never finishes.