Skip to main content

Network Policies

OLIVE ships Kubernetes NetworkPolicy objects with every chart, derived from a full service communication matrix: for each service, which peers it must accept traffic from, which peers it must reach, whether it needs storage, whether it needs Vault, and whether it needs the public internet.

The model is default-deny. The tenant-setup chart lays down a namespace-wide deny baseline (plus DNS and telemetry egress), and each workload chart adds only the connections that service actually needs.

What the policies enforce​

  • Express-based services may talk to each other through a shared pod label, olive/platform: <global.express.sharedNamespace>.
  • Only the services that require internet access get it. CouchDB, SeaweedFS and the bootstrap jobs do not.
  • Only the services that require storage may reach the storage services.
  • Only the services that require Vault may reach Vault.
  • A service in one tenant's namespace cannot reach a service in another tenant's namespace. This is the property that matters most, and it is enforced by the network layer rather than by application logic.

This is deliberately redundant with the application-level egress guard, which blocks outbound calls to internal, Kubernetes-internal and link-local metadata addresses and emits a security event (OLIFETCH1) when it does. A network policy stops the packet; the application check stops the attempt and tells you it happened. Either alone would be a single point of failure.

Activating network policies​

One switch turns policies on for every chart in the namespace:

global:
networkPolicy:
enabled: true
ingressController:
namespace: kube-system
podLabels:
app.kubernetes.io/name: rke2-traefik

Each chart's own networkPolicy.enabled is left unset and inherits the global value. A single chart can still opt out locally:

auth-proxy:
networkPolicy:
enabled: false

Before you turn it on​

Two things will silently break a deployment
  1. The CNI must enforce NetworkPolicy. Cilium and Calico do. If the CNI ignores policies, everything appears to work and nothing is actually enforced — which is worse than having no policy, because you will believe you have one. Confirm with your cluster operator.
  2. ingressController must match reality. The policies admit ingress traffic from the pods this selector matches. A selector that matches nothing is not an error — it is zero peers, and default-deny then removes every route into the namespace at once. On RKE2 the bundled Traefik is a DaemonSet in kube-system whose pods are labelled app.kubernetes.io/name=rke2-traefik. Note the rke2- prefix: the upstream Traefik chart uses plain traefik.

The default values describe RKE2's bundled Traefik. When Traefik is installed separately — on Amazon EKS, AKS, or RKE2 with the bundled ingress disabled — set both the namespace and the labels to that installation:

ClusteringressController.namespaceingressController.podLabels
RKE2, bundled Traefik (default)kube-systemapp.kubernetes.io/name: rke2-traefik
Upstream Traefik Helm chart, release traefik (e.g. EKS)traefik (the release namespace)app.kubernetes.io/name: traefik
global:
networkPolicy:
enabled: true
ingressController:
namespace: traefik
podLabels:
app.kubernetes.io/name: traefik

See Chart Values Reference → Ingress controller location for the full explanation.

Verify the selector before enabling:

# RKE2 bundled Traefik
kubectl -n kube-system get pods -l app.kubernetes.io/name=rke2-traefik
# separately installed Traefik
kubectl -n <traefik-namespace> get pods -l app.kubernetes.io/name=traefik --show-labels

If that returns nothing, find the real label and set it before you enable policies.

Amazon EKS

The VPC CNI enforces NetworkPolicy only when its add-on runs with enableNetworkPolicy=true (or when Calico or Cilium is installed). Without that, the policies are accepted and enforce nothing.

Deactivating network policies​

Set the master switch back:

global:
networkPolicy:
enabled: false

The policy objects are removed on the next sync and traffic reverts to the cluster default (usually allow-all). Do this only in a development environment or to isolate a fault during an incident — and put it back afterwards. In a production deployment network policies are expected to be on, and they are one of the four independent mechanisms that enforce the tenant boundary.

Egress and the keycloakIpBlocks value​

Services that need the public internet get an egress rule to 0.0.0.0/0 minus global.nonRoutableCidrs. That subtraction is what stops an internet rule from also matching in-cluster addresses and quietly bypassing the pod selectors:

global:
nonRoutableCidrs:
- '10.0.0.0/8' # RFC1918
- '172.16.0.0/12' # RFC1918
- '192.168.0.0/16' # RFC1918
- '100.64.0.0/10' # CGNAT
- '169.254.0.0/16' # link-local, incl. metadata endpoint 169.254.169.254
- '127.0.0.0/8' # loopback
- '0.0.0.0/8'
- '224.0.0.0/4'
- '240.0.0.0/4'

Keep the list wide rather than exact. A missing range is the dangerous direction. Note that this also blocks private destinations outside the cluster — peered databases, on-prem systems over VPN, AWS interface endpoints, Azure Private Link. Add those back explicitly under the chart's extraEgress.

Every service needs to reach Keycloak to fetch its JWKS. NetworkPolicy cannot match host names, so that rule is an ipBlock built from global.keycloakIpBlocks, restricted to port 443.

The shipped default is an empty list, and empty is not "no rule". It renders 0.0.0.0/0 with the non-routable, cluster, node and link-local ranges subtracted — so it allows any public host on 443 and nothing inside the cluster.

That matters more than it sounds, because it means the default allows a public Keycloak and blocks a private one.

Which value to use​

The deciding question is not "what is Keycloak's IP" but whether the address the pods resolve is public or private.

Where Keycloak sitsPods resolveSet it to
Behind an internet-facing load balancer — the usual arrangement, where authenticate.<cluster>.<domain> goes through the same ingressA public addressLeave empty. Nothing to set, nothing to know
Behind an internet-facing NLB with Elastic IPsA public, stable addressOptionally the Elastic IPs, /32 each — the only case where pinning a load-balancer address is safe
Behind an internal load balancer, or a private hosted zone pointing at oneA private addressThe VPC CIDR, e.g. 10.0.0.0/16. Required — the default blocks this
Behind an ALB of any kind—Do not pin its addresses. AWS allocates them dynamically and documents that you must not depend on them
global:
keycloakIpBlocks: [] # default: any public host on 443
# keycloakIpBlocks:
# - '10.0.0.0/16' # internal LB: the VPC CIDR

Check which case you are in from inside the cluster, not from a workstation — split-horizon DNS answers differently:

kubectl run dnscheck --rm -it --image=alpine/curl --restart=Never -- \
nslookup auth.<cluster>.<domain>

A 10., 172.16–172.31. or 192.168. answer means you must set the value explicitly.

An internal load balancer is blocked by the default, and the symptom is a login loop

If the Keycloak hostname resolves to a private address, the default drops the JWKS fetch — private ranges are subtracted from the rule. The dropped fetch times out, the service answers 401, and the Traefik redirectErrors middleware turns that into a redirect to /oauth2/start: the user is bounced back into a login they already completed. Keycloak is healthy throughout, and nothing in the symptom mentions a network policy. The full chain is in What keycloakIpBlocks is for.

Writing 0.0.0.0/0 out means the same as leaving the list empty

Both render the except: carve-out, so either is a valid way to say "any public host". A narrower range than /0 but still wide — a /8, a /16 — is rendered bare and needs checking by hand; see What keycloakIpBlocks is for.

If you pin a public address, monitor it

A public load-balancer address changes — when the load balancer is recreated, when the ingress is reconfigured, when the provider reassigns it. When it changes and the pinned value does not, every service loses JWKS at the same moment, with the login loop above. Pin every address the name resolves to, monitor it, and re-check it in the same change as any alteration to the load balancer, the ingress or the Keycloak host name.

A VPC CIDR carries none of this risk — it is fixed for the life of the VPC — which is why it is the right answer for the internal case rather than the load balancer's own address.

The full version of this, with the per-chart override, is in What keycloakIpBlocks is for.

When connectivity needs to reach an internal address​

The application-level egress guard blocks outbound calls to internal address ranges, Kubernetes-internal service names and link-local metadata addresses. This is not switchable, and it is separate from the network policies above.

It is not absolute, however. The platform reaches a set of internal addresses it needs in order to function, and those are registered with the platform rather than carved out of the guard. The same registration can in principle cover an additional internal destination, and only a platform administrator can make it.

Raise the case with OLIVE support rather than building around it

There is currently no public API for this, and the mechanism may change in a future release, so it is deliberately not documented as a pattern for connectivity engineers to build against.

If a connectivity engineer comes to you with a use case that genuinely requires OLIVE to call an internal destination, collect the details — which system, why it cannot be exposed as an external endpoint, and what the connectivity is meant to do — and raise it with OLIVE support at olive-support@boc-group.com. You will get the approach that fits the situation.

These reports are what determine whether the mechanism becomes a documented, supported feature in a future release, so a case raised is worth more than a workaround improvised locally.

The connectivity handbook points engineers at you for exactly this — see the chapter "Security in Connectivity Engineering" in the OLIVE connectivity developer handbook.

Interaction with the service mesh​

If you enable a service mesh, the policies change shape:

  • ambient — traffic between meshed pods is tunnelled over HBONE on TCP 15008, so every rule whose peer is a meshed pod must also allow that port. The charts do this automatically when global.serviceMesh.mode: ambient.
  • sidecar — traffic stays on the application port, so no port changes are needed, but every pod's proxy must reach istiod on 15012.

Because the two modes need opposite changes, global.serviceMesh.mode must match what is installed on the cluster. See Service Mesh and mTLS.

Ambient also changes where kubelet probes come from: istio-cni SNATs node-local packets to a fixed link-local address (169.254.7.127/32) so they can be told apart from pod traffic. With default-deny and no rule for it, every liveness and readiness probe fails. This is the most common ambient rollout failure, and the charts cover it via global.serviceMesh.healthProbeCidrs.

Troubleshooting​

A default-deny CNI usually drops rather than refuses, so the symptom is a timeout, not a connection error.

# What policies exist in the namespace
kubectl -n <namespace> get networkpolicy

# What a specific policy actually allows
kubectl -n <namespace> describe networkpolicy <name>

# Is the ingress controller selector matching anything at all?
# (use your ingressController.namespace and podLabels)
kubectl -n kube-system get pods -l app.kubernetes.io/name=rke2-traefik

Symptoms and their usual cause:

SymptomLikely cause
Nothing reachable from outside right after enablingingressController selector matches no pods — typically the RKE2 default left in place on a cluster with a separately installed Traefik
An infinite login loop, or all requests returning 401keycloakIpBlocks pinned to an address that has changed — the JWKS fetch is dropped, and redirectErrors turns the resulting 401 into a redirect to /oauth2/start
Telemetry stops, no other symptomallowObservability ports do not match the collector's Service
Probes fail after enabling ambientHealth-probe CIDR not allowed, or serviceMesh.mode not set to ambient
A datapoint to an on-prem system times outDestination is inside nonRoutableCidrs; add an explicit extraEgress ipBlock