Skip to main content

Monitoring and Observability

The OLIVE Platform is a standard Kubernetes-based application. It can use the same observability patterns as other workloads in your cluster: centralized logging, metrics, alerting, and distributed tracing. OLIVE does not mandate a single vendor stack; you integrate the tools your organization already runs or approves.

Logging and dashboards​

OLIVE produces application and infrastructure logs that you can collect with a typical Kubernetes logging stack. Common choices include:

  • Promtail — collect and forward logs from pods and nodes
  • Loki — centralized, cost-efficient log storage and querying
  • Grafana — dashboards, log filtering by context, and health views

This lets the two roles between them:

  • Build custom dashboards for platform activity and performance — cluster-level ones yours, OLIVE-level ones the platform administrator's; see Who owns what
  • Search and filter logs to find errors and unusual patterns
  • Support debugging by correlating messages across services
  • Configure alerting and notifications for log- or metric-driven conditions

If you need to document or implement automated reactions to log-derived events (including security alerting), see Log Events and Automated Reactions.

Distributed tracing​

OLIVE can emit OpenTelemetry (OTel) traces so you can follow requests across services.

A typical stack:

  • Grafana Alloy — collect and forward OpenTelemetry traces
  • Grafana Tempo — store, query, and navigate traces

Traces help you see full request lifetimes, correlate failures with specific calls, and analyze latency and dependencies.

Metrics and alerts​

OLIVE exposes Prometheus metrics endpoints on its own workloads. That is all it does: it exposes the port, names it, and opens a NetworkPolicy rule so a scraper can reach it. Everything else is yours — the ServiceMonitor and PodMonitor objects that define what gets collected, the scrape agent, the time-series store and its retention, Grafana, and the alert rules.

Metrics are off by default in every chart. The platform administrator switches them on per namespace, as described in Metrics Endpoints. Until that happens there is no endpoint to scrape, and the metric sources under If OLIVE metrics are not enabled are all you have.

Who owns what​

The split is not "infrastructure to the cluster operator, nothing to anyone else". It turns on who knows what a number means: the cluster operator owns the pipeline, the platform administrator owns the interpretation of OLIVE's metrics.

ConcernOwner
Metrics endpoints on OLIVE workloadsOLIVE charts
NetworkPolicy rule that admits the scraperOLIVE charts
Enabling the endpoints per namespace (global.metrics.*)Platform administrator
Metrics operator and its CRDsCluster operator
Scrape agent, time-series store, retentionCluster operator
Grafana itself, data sources, notification routingCluster operator
ServiceMonitor / PodMonitor objects in the tenant namespaceShared — they live in the administrator's namespace but must carry the label the operator's agent selects on. Agree who applies them; either works, as long as one of you does
Dashboards and alert rules about OLIVE — tenant health, per-service latency and errors, Temporal backlog, the OLIALIVE1 dead-man's switchPlatform administrator
Dashboards and alert rules about the cluster — node pressure, volume capacity, scheduling, CNICluster operator
Cardinality and retention budgetCluster operator, agreed with the platform administrator
Why the OLIVE dashboards are not the cluster operator's

A cluster operator can tell that a pod is restarting. They cannot tell whether a rising task_schedule_to_start_latency means workers are saturated, whether route="unmatched" matters, or what normal looks like for a tenant that runs three connectors versus three hundred. That knowledge sits with whoever runs the OLIVE installation.

This matches the responsibility matrix, which already assigns platform-level alerting to the cluster operator and OLIVE-level alert rules and first-line triage to the platform administrator. What the cluster operator provides is a Grafana the administrator can build in, and the data behind it.

The practical consequence: the alert rules further down this page are a starting point for the platform administrator, not a to-do list for the cluster operator. Hand them over rather than implementing them blind.

What the cluster must provide​

Four conditions have to hold before any OLIVE metric reaches you. Each one fails silently — a monitor whose selector matches no operator is created without error and simply never produces a target.

  1. A Prometheus-operator-compatible agent, with the ServiceMonitor and PodMonitor CRDs installed. kube-prometheus-stack, the Victoria Metrics Operator and Grafana Alloy in operator mode all qualify.
  2. A monitor per component you want to collect, created in the tenant namespace and carrying whatever label your agent's serviceMonitorSelector or podMonitorSelector matches. OLIVE creates none of these; the table below is everything you need to write them.
  3. A namespace selector that includes the tenant namespaces. serviceMonitorNamespaceSelector and podMonitorNamespaceSelector must cover them, or your own monitors are never read.
  4. PodMonitor as well as ServiceMonitor. SeaweedFS keeps its metrics port off its Service, which publishes only S3 by design, so it needs a PodMonitor. If your agent handles only ServiceMonitor, ask the administrator for seaweedfs.metrics.serviceEnabled: true, which publishes the port on the Service too.

Two facts have to travel the other way, from you to the platform administrator, or the configuration cannot be written: the namespace your scrape agent runs in, and the pod labels of its pods. Together they become the NetworkPolicy rule that lets your scrape through — and a selector that matches nothing is not an error, it is zero peers.

The endpoints are unauthenticated​

NetworkPolicy is the only access control on a metrics port

The metrics endpoints carry no TLS, no bearer token and no authorization check. Any workload that can reach the port can read them. The NetworkPolicy each chart renders alongside the port is the entire access control, which is why OLIVE's guidance is to enable metrics only in namespaces where network policies are on.

The content is operational rather than business data — request counts, latency histograms, disk and memory figures, queue depths — with no payloads and no credentials. It does reveal route names, internal service names and traffic volumes, which has reconnaissance value.

Two cluster properties decide whether that control is real:

  • The CNI must enforce NetworkPolicy. Cilium and Calico do. On Amazon EKS the AWS VPC CNI enforces it only when the add-on runs with enableNetworkPolicy=true; without that the policies are accepted by the API server and do nothing, and the metrics ports are reachable from anywhere in the cluster. This is the same prerequisite described under Networking.
  • A mesh in ambient mode weakens port scoping. Traffic between meshed pods is tunnelled over HBONE on TCP 15008, so a policy admitting your scraper must admit 15008 too — which admits it to every port of that pod, not only the metrics port. Under ambient, treat the metrics policy as pod-level rather than port-level isolation.

What exposes metrics​

ComponentScrape viaSelectorPort namePortPathMetric prefix
Express services (the OLIVE microservices)Serviceapp.kubernetes.io/component: expressmetrics9464/metricshttp_*, nodejs_*, process_*
CouchDBServiceapp.kubernetes.io/component: couchdbmetrics17986/_node/_local/_prometheuscouchdb_*, erlang_vm_*
SeaweedFSPodapp.kubernetes.io/component: seaweedfsmetrics9324/metricsSeaweedFS_*
Temporal (frontend, internal-frontend, matching, history, worker)Service (the five *-headless Services)app.kubernetes.io/name: temporalmetrics9090/metricsunprefixed — see below
PostgreSQL (CloudNativePG)Podcnpg.io/cluster: <cluster-name>—9187/metricscnpg_*, pg_*

Monitors reference port names, not numbers — a Service port name for a ServiceMonitor, a container port name for a PodMonitor. A name that does not exist yields no target and no error.

Five things about that table are worth knowing before you build dashboards on it:

  • CouchDB does not serve /metrics. Its endpoint is /_node/_local/_prometheus. A monitor pointed at /metrics scrapes a 404 and still reports the target as up, with zero samples.

  • Temporal's metric names are unprefixed — action, cache_size, service_requests. In a shared store they collide with other applications, so prefix them in your own monitor:

    metricRelabelings:
    - sourceLabels: [__name__]
    regex: '(.*)'
    targetLabel: __name__
    replacement: 'temporal_$1'
  • Temporal labels its metrics with namespace, meaning a Temporal namespace. The scrape attaches the Kubernetes namespace under the same name, so Temporal's own value arrives as exported_namespace.

  • PostgreSQL is not covered by OLIVE's metrics flag. The CloudNativePG cluster has its own monitoring.enablePodMonitor, and that one does create a PodMonitor — from the CloudNativePG operator's template, not from OLIVE.

  • Temporal can be the exception, if you want it to be. The Temporal subchart bundled with OLIVE is able to create five ServiceMonitor objects, one per server role. OLIVE leaves that off, because monitors are yours. If you would rather have them from the chart than write your own, ask the administrator to enable them and give them the label your agent selects on.

Cardinality: Temporal is the one to budget for​

Series land in your store, so series count is a platform cost rather than a tenant one.

ComponentSeries scaleDriver
Express serviceslow per service, multiplied by service countroute × status_code × method; route is the route pattern, so it is bounded by the number of routes, not by traffic
CouchDBlowa fixed set per node
SeaweedFSlow to moderategrows with volume count
Temporalhighlatency histograms labelled by operation × Temporal namespace × service, across five server roles

Temporal's timerType defaults to histogram. A summary timer type, or a drop list for the heaviest families in the monitor's metricRelabelings, cuts it substantially. Agree a budget before Temporal metrics are switched on, and measure the result:

count by (job) ({namespace="<tenant-namespace>"})

Verifying a tenant's metrics arrive​

Targets exist and are scraped successfully:

up{namespace="<tenant-namespace>"}

Targets that answer but return nothing — the failure up cannot show. A target with up=1 and zero samples is a wrong path or a non-Prometheus endpoint:

scrape_samples_scraped{namespace="<tenant-namespace>"} == 0

A target reporting down is almost always the NetworkPolicy: either the rule was not rendered, or the agent's pod labels in the tenant's global.metrics.scrape do not match your agent's pods. Your agent's own targets page names the cause directly — connection refused, context deadline exceeded, 404 — which is faster than inferring it from queries. A whole namespace going down at once is a configuration change, not an outage.

Alert rules to start from​

OLIVE ships no alert rules. These are starting points with deliberately conservative thresholds, and they belong to the platform administrator to adopt and tune — the cluster operator's part is that the rules can be loaded and that notifications route somewhere. Tune them per environment.

# A metrics endpoint that was being scraped stops answering.
# Scope it to namespaces where metrics are expected: it also fires on
# scale-to-zero and on an intentional metrics rollback.
- alert: OliveTargetDown
expr: up{namespace=~"tenant-.*"} == 0
for: 10m
labels: { severity: warning }

# Express service returning server errors.
- alert: OliveExpressErrorRate
expr: |
sum by (namespace, job) (rate(http_request_duration_seconds_count{status_code=~"5.."}[5m]))
/ sum by (namespace, job) (rate(http_request_duration_seconds_count[5m])) > 0.05
for: 10m
labels: { severity: warning }

# Express service latency regression.
- alert: OliveExpressLatencyHigh
expr: |
histogram_quantile(0.95, sum by (le, namespace, job) (
rate(http_request_duration_seconds_bucket[5m]))) > 2
for: 15m
labels: { severity: warning }

# Node.js event loop saturated: the service is CPU-bound, not waiting on I/O.
- alert: OliveEventLoopLag
expr: nodejs_eventloop_lag_p99_seconds > 0.5
for: 10m
labels: { severity: warning }

# Object storage filling up.
- alert: OliveSeaweedFsVolumeNearFull
expr: |
kubelet_volume_stats_available_bytes{persistentvolumeclaim=~".*seaweedfs.*"}
/ kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*seaweedfs.*"} < 0.15
for: 30m
labels: { severity: critical }

# Temporal workers not keeping up with scheduled tasks.
- alert: OliveTemporalTaskBacklog
expr: |
histogram_quantile(0.95, sum by (le, namespace) (
rate(temporal_task_schedule_to_start_latency_bucket[5m]))) > 5
for: 15m
labels: { severity: warning }

If OLIVE metrics are not enabled​

Metrics are opt-in, and in some deployments they will stay off. Metric-based alerting is still possible on:

  • Kubernetes-level metrics — pod restarts, resource usage, probe failures and scheduling, from the usual sources (kube-state-metrics, the kubelet and cAdvisor). This covers most availability alerting and is where CPU and memory pressure shows up. See Platform Scaling.
  • The third-party components' own metrics — Vault, Keycloak, PostgreSQL and the rest expose what their projects document, independently of OLIVE's flag.
  • The log stream — OLIVE's security signals are events, not gauges. Rules are written against the event codes; see Log Events and Automated Reactions.

The single most useful OLIVE-specific alert is not a metric at all: the OLIALIVE1 liveness event, which each tenant emits every five minutes, and whose absence means that tenant has stopped working end to end.

Metrics are not an audit trail

Metrics are aggregated, unauthenticated and retained on the observability stack's schedule rather than a compliance one. Access logs, authentication events and audit records belong to the log pipeline — see Log Events and Automated Reactions.

Cluster operator responsibilities​

The cluster operator is expected to have the skills to:

  • Select, install and configure the monitoring and observability tooling
  • Operate log, trace and metrics collection pipelines end to end
  • Own the cluster's own dashboards and alerts — nodes, capacity, scheduling, the CNI
  • Own notification routing, and give the platform administrator a Grafana they can build dashboards in

OLIVE-specific dashboards and alert rules are the platform administrator's, because they depend on knowing what OLIVE's metrics mean and what normal looks like for a given tenant. See Who owns what and the responsibility matrix.

OLIVE stays compatible with common Kubernetes observability tooling; specific product choices remain with your operations team.