Skip to main content

Kyverno (kyverno-infra)

Optional. The platform runs without it. What you lose is enforcement: the imageVerification, imageRegistry and podPrivileges modes in the tenant values become labels that nothing reads.

kyverno-infra is not part of the OLIVE platform

It is provided as-is to make setup easier — not maintained, patched or supported as part of the platform release, and not covered by OLIVE support. Kyverno is third-party software with its own licence, release cycle and security advisories; tracking and applying those is yours regardless of how you installed it.

What OLIVE does stand behind is the contract below. Meet it however you like.

The policies are documented separately

You do not need this chart to use the policies. OLIVE Kyverno Policies lists each one — what it does, why it exists, and the manifest in a form you can apply and adapt with your own Kyverno.

There are three policy families, each rendered twice (once for audit, once for enforce), so a full installation is up to six policy objects.

Why OLIVE offers it​

OLIVE signs the container images it builds with cosign and pins every image by digest. Kyverno is what makes that verifiable at admission: it checks the signature before a pod is created, and refuses anything that does not verify or that comes from an unapproved registry.

Without an admission engine, digest pinning still guarantees you run the artefact the manifest named — but nothing checks that the artefact the manifest named is the one OLIVE built.

What the chart enforces​

Policy familyValues blockEnrolment labelDefault
Image signatures — keyless cosign signature on images from the OLIVE registryimageVerificationolive/image-verificationenabled, audit only
Registry allow-list — container images must come from an approved registry prefiximageRegistryAllowListolive/image-registryenabled, audit only
Pod privileges — no privileged mode, no privilege escalation, no hostPath, no host namespacespodSecurityRestrictionsolive/pod-privilegesdisabled

The three are deliberately independent, because they cost different amounts to comply with. Pod privileges is usually the first a tenant can enforce — the pod spec only has to stop asking for host access. The registry allow-list is next; nothing has to be rebuilt. Signatures are last, because they need a signing pipeline for everything the namespace pulls.


What the platform administrator specifies​

The policies are the platform administrator's: which identity is allowed to have signed, which registries are on the allow-list, which label keys enrol a namespace, and which of the three controls a given namespace is held to. Those are listed in Infrastructure Prerequisites → Kyverno, and the policies themselves in OLIVE Kyverno Policies.

This page covers installing the engine and rendering those policies into the cluster — all of which needs cluster rights.


Using the kyverno-infra chart​

Wrapper chart​

apiVersion: v2
name: kyverno-infra-wrapper
description: Helm chart for managing Kyverno
type: application
version: 1.0.0
appVersion: '1.0'
dependencies:
- name: kyverno-infra
version: 1.0.10
repository: oci://registry.gitlab.com/olive6224315/helm-charts/charts
condition: services.global.kyverno-infra

Values​

services:
global:
kyverno-infra: true

global:
clusterName: 'develop'
domain: 'olive-team.com'
imageVerification:
labelKey: 'olive/image-verification'
imageRegistry:
labelKey: 'olive/image-registry'
podPrivileges:
labelKey: 'olive/pod-privileges'
serviceMesh:
mode: none

kyverno-infra:
namespace:
create: true
name: '' # follows the Application's destination namespace
syncOptions: 'Prune=false'

imagePinning:
enforce: false # until the kyverno image tags are digest-pinned

imageVerification:
enabled: true

imageReferences:
- 'registry.gitlab.com/olive6224315/docker-images/*'

keyless:
subject: 'https://gitlab.com/olive6224315/docker-images//.gitlab-ci.yml@refs/heads/main'
issuer: 'https://gitlab.com'
rekorUrl: 'https://rekor.sigstore.dev'

# Must exist in the KYVERNO namespace, not the tenant's. Created out of band.
imagePullSecrets:
- 'olive-regcred'

excludeNamespaces:
- 'kube-system'
- 'kube-public'
- 'kube-node-lease'
- 'oli-kyverno'
- 'argocd'

modes:
audit:
enabled: true
enforce:
enabled: false

# ---- registry allow-list --------------------------------------------
imageRegistryAllowList:
enabled: true

# Every entry is a PREFIX, matched up to the ':tag' or '@sha256:' boundary.
# Docker Hub short names are expanded by the chart to all five equivalent
# spellings; write anything else fully qualified.
allowed:
- 'registry.gitlab.com/olive6224315/docker-images'
- 'gcr.io/distroless/nodejs24-debian13'
- 'gcr.io/distroless/python3-debian13'
- 'node'
- 'quay.io/keycloak/keycloak'
- 'quay.io/oauth2-proxy/oauth2-proxy'
- 'hashicorp/vault'
- 'hashicorp/vault-k8s'
- 'hashicorp/vault-secrets-operator'
- 'couchdb'
- 'redis'
- 'chrislusf/seaweedfs'
- 'temporalio/server'
- 'temporalio/admin-tools'
- 'temporalio/ui'
- 'alpine'
- 'alpine/curl'
- 'alpine/k8s'
- 'docker.io/busybox'
# Needed as soon as the mesh is on in sidecar mode — a blocked sidecar
# image means no pod in an enrolled namespace can start.
- 'docker.io/istio/proxyv2'

# Appended to `allowed` rather than replacing it, so a cluster override
# cannot silently drop the platform's own images.
extraAllowed: []

excludeNamespaces:
- 'kube-system'
- 'kube-public'
- 'kube-node-lease'
- 'oli-kyverno'
- 'argocd'

# Pure spec inspection, no network — so it also evaluates what is already
# running. Leave it on: enrolling a namespace produces a full inventory
# within a minute, without restarting a single workload.
background: true

modes:
audit:
enabled: true
enforce:
enabled: false

# ---- pod privilege restrictions -------------------------------------
podSecurityRestrictions:
enabled: false # off by default — switch on per cluster

# The four rules are individually switchable. hostPath is the one that
# blocks legitimate-looking workloads (log shippers, node exporters, CSI
# helpers) and the one most likely to need turning off for a while.
# At least one must stay on.
rules:
privileged: true
privilegeEscalation: true
hostNamespaces: true
hostPath: true

excludeNamespaces:
- 'kube-system'
- 'kube-public'
- 'kube-node-lease'
- 'oli-kyverno'
- 'argocd'

background: true

modes:
audit:
enabled: true
enforce:
enabled: false

kyverno:
enabled: true
crds:
install: true
# Argo CD applies client-side by default, which stuffs the whole manifest
# into last-applied-configuration. Kyverno's CRDs blow past the 262144-byte
# annotation limit, so those specific objects are applied server-side.
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
# THE SECOND HALF OF THE SAME FIX. Kyverno 3.9 splits its CRDs across two
# subcharts with two separate annotation keys — see the warning below.
kyverno-api:
annotations:
argocd.argoproj.io/sync-options: ServerSideApply=true
config:
preserve: false
admissionController:
replicas: 2
Both CRD annotation blocks are needed, and the half-fixed state looks like success

The Kyverno chart 3.9 ships its CRDs from two subcharts: crds (11 CRDs across three groups — 7 kyverno.io, 2 reports.kyverno.io, 2 wgpolicyk8s.io) and kyverno-api (the 11 policies.kyverno.io types OLIVE's policies use, from a separate chart with its own top-level annotations value).

Setting only kyverno.crds.annotations gets the admission controller healthy — its startup check wants the legacy CRDs — while leaving exactly 11 CRDs permanently OutOfSync with error when patching ... annotations: Too long.

Set both. On the Argo CD Application itself, also set ServerSideApply=true and ServerSideDiff=true, and do not use Replace=true — Kyverno's own installation notes warn against it for existing resources.

Prune=false on the namespace is load-bearing

Deleting this namespace removes the admission controller while its cluster-scoped ValidatingWebhookConfiguration lingers — a webhook with no backend, which either blocks pod creation cluster-wide or silently stops enforcing every policy. The second is worse, because nothing reports it.

Registry credentials, in Kyverno's own namespace​

The cosign signature lives in the registry as an ordinary tag beside the image. The OLIVE registry is private, so Kyverno needs its own pull credentials — the tenant's imagePullSecret is not consulted.

kubectl -n oli-kyverno create secret docker-registry olive-regcred \
--docker-server=registry.gitlab.com \
--docker-username=<user> --docker-password=<token>
Without it, every verification fails in a misleading way

The error is "no signatures found"-shaped. It reads like a signing failure and is in fact a credentials failure. This is the single most common reason a working signing pipeline appears broken.

Namespaces to exclude​

System and platform namespaces should not be evaluated: kube-system, kube-public, kube-node-lease, the Kyverno namespace itself, and argocd.

This matters most for the pod-privilege policy: the CNI, CoreDNS and every CSI driver legitimately run privileged, with hostPath and host namespaces. An enforcing version of that policy reaching kube-system is a cluster that cannot recover a node restart.

The platform's own namespaces enrol themselves​

vault-infra and keycloak-infra now label their own namespaces with all three keys at audit by default (namespace.labels and namespace.extraLabels respectively), because tenant-setup does not manage those namespaces. Remove a label line there to opt that namespace out of that one policy.

Enabling the Vault Secrets Operator needs an allow-list entry

hashicorp/vault-secrets-operator must be on imageRegistryAllowList.allowed before the Vault namespace can move past audit.

Service mesh interaction​

Leave the Kyverno namespace out of the mesh, or apply a PERMISSIVE exemption for its webhook service first. The admission webhook is called by the kube-apiserver, which is never in the mesh and has no SPIFFE identity — a mesh-wide STRICT policy rejects those calls, and then every pod creation in every enrolled namespace fails.


What the Argo CD project must permit​

This is the heaviest of the five in cluster-scoped terms — it installs the policy engine and the policies, and both are cluster-scoped:

clusterResourceWhitelist:
- group: ''
kind: Namespace # namespace.create: true, and the subchart's own
- group: 'apiextensions.k8s.io'
kind: CustomResourceDefinition # 24 of them, see the note below
- group: 'rbac.authorization.k8s.io'
kind: ClusterRole
- group: 'rbac.authorization.k8s.io'
kind: ClusterRoleBinding
# The policies themselves - one entry per Kind the chart actually renders
- group: 'policies.kyverno.io'
kind: ImageValidatingPolicy # imageVerification.enabled: true
- group: 'policies.kyverno.io'
kind: ValidatingPolicy # imageRegistryAllowList.enabled: true,
# and podSecurityRestrictions, which ships false
No webhook configurations, and that is deliberate on Kyverno's part

Nothing in this chart renders a ValidatingWebhookConfiguration or MutatingWebhookConfiguration. Kyverno's admission controller registers its own webhooks at runtime, from the policies it finds, and rewrites them as policies come and go.

Two consequences. The project does not need those Kinds permitted — granting them achieves nothing here. And Argo CD never sees the webhooks, so they are outside the Application: they are not pruned when a policy is removed and not reported as drift when Kyverno changes them. An unexpected webhook is a question for Kyverno's logs, not for the sync status.

The CRDs are ordinary templates, so skipCrds does not reach them

The 24 CRDs come from the chart's crds and kyverno-api subcharts, rendered as normal templates rather than from a crds/ directory. helm.skipCrds therefore has no effect on them, and the project must permit CustomResourceDefinition — there is no install-without-CRDs variant of this chart.

They also have to be applied server-side. Argo CD applies client-side by default, which writes the whole manifest into the last-applied-configuration annotation; Kyverno's CRDs are far too large for that, and the same size is why they need a server-side diff as well — see below.

Argo CD adaptations​

Two adjustments keep Argo CD and Kyverno from fighting each other. Add to the argocd-cm ConfigMap:

# Kyverno creates ClusterRoles with an aggregationRule; the API server fills
# in /rules, which Argo CD reads as drift and selfHeal keeps trying to revert.
resource.customizations.ignoreDifferences.rbac.authorization.k8s.io_ClusterRole: |
jsonPointers:
- /rules

# Kyverno churns report CRs constantly. Without this they flood the Argo CD
# resource tree and make the application view unusable.
resource.exclusions: |
- apiGroups: ["kyverno.io", "reports.kyverno.io"]
kinds:
- AdmissionReport
- BackgroundScanReport
- ClusterAdmissionReport
- ClusterBackgroundScanReport
clusters: ["*"]

In the ApplicationSet, the Kyverno application also needs a server-side diff, because its CRDs are too large for a client-side diff to converge — otherwise the CRD resources report OutOfSync forever even when they are synced:

{{- if eq .path.basename "kyverno-infra" }}
argocd.argoproj.io/compare-options: ServerSideDiff=true
{{- end }}

Rollout​

Start every namespace at audit, on all three labels. The registry allow-list and the pod-privilege policy go quiet within a minute, because they are pure spec inspection and evaluate pods that are already running. Signature verification takes longer, because it has to make a registry call and therefore only evaluates at pod creation.

The usual order of reaching enforce, cheapest first:

  1. olive/pod-privileges — complying needs no rebuild and no signature, only a pod spec that stops asking for host access.
  2. olive/image-registry — nothing has to be rebuilt; an unlisted image has to be replaced or the prefix added.
  3. olive/image-verification — needs every OLIVE image in the namespace to be signed by the pipeline.
enforce fails late

Admission control acts on pod creation. Switching a namespace to enforce evicts nothing and everything looks fine — until the next rollout, scale-up, eviction or node drain. A namespace can sit in a broken state for weeks and discover it during a node reboot at 2 a.m. A crash-looping container does not recreate the pod object and is therefore not re-admitted, which is why the failure is so delayed.

Move to enforce only after the audit report has been empty across a full deployment cycle.