Monitoring and Observability
The OLIVE Platform is a standard Kubernetes-based application. It can use the same observability patterns as other workloads in your cluster: centralized logging, metrics, alerting, and distributed tracing. OLIVE does not mandate a single vendor stack; you integrate the tools your organization already runs or approves.
Logging and dashboards
OLIVE produces application and infrastructure logs that you can collect with a typical Kubernetes logging stack. Common choices include:
- Promtail — collect and forward logs from pods and nodes
- Loki — centralized, cost-efficient log storage and querying
- Grafana — dashboards, log filtering by context, and health views
This lets the two roles between them:
- Build custom dashboards for platform activity and performance — cluster-level ones yours, OLIVE-level ones the platform administrator's; see Who owns what
- Search and filter logs to find errors and unusual patterns
- Support debugging by correlating messages across services
- Configure alerting and notifications for log- or metric-driven conditions
If you need to document or implement automated reactions to log-derived events (including security alerting), see Log Events and Automated Reactions.
Distributed tracing
OLIVE can emit OpenTelemetry (OTel) traces so you can follow requests across services.
A typical stack:
- Grafana Alloy — collect and forward OpenTelemetry traces
- Grafana Tempo — store, query, and navigate traces
Traces help you see full request lifetimes, correlate failures with specific calls, and analyze latency and dependencies.
Metrics and alerts
OLIVE exposes Prometheus metrics endpoints on its own workloads. That is all it does: it exposes the port, names it, and opens a NetworkPolicy rule so a scraper can reach it. Everything else is yours — the ServiceMonitor and PodMonitor objects that define what gets collected, the scrape agent, the time-series store and its retention, Grafana, and the alert rules.
Metrics are off by default in every chart. The platform administrator switches them on per namespace, as described in Metrics Endpoints. Until that happens there is no endpoint to scrape, and the metric sources under If OLIVE metrics are not enabled are all you have.
Who owns what
The split is not "infrastructure to the cluster operator, nothing to anyone else". It turns on who knows what a number means: the cluster operator owns the pipeline, the platform administrator owns the interpretation of OLIVE's metrics.
| Concern | Owner |
|---|---|
| Metrics endpoints on OLIVE workloads | OLIVE charts |
| NetworkPolicy rule that admits the scraper | OLIVE charts |
Enabling the endpoints per namespace (global.metrics.*) | Platform administrator |
| Metrics operator and its CRDs | Cluster operator |
| Scrape agent, time-series store, retention | Cluster operator |
| Grafana itself, data sources, notification routing | Cluster operator |
ServiceMonitor / PodMonitor objects in the tenant namespace | Shared — they live in the administrator's namespace but must carry the label the operator's agent selects on. Agree who applies them; either works, as long as one of you does |
Dashboards and alert rules about OLIVE — tenant health, per-service latency and errors, Temporal backlog, the OLIALIVE1 dead-man's switch | Platform administrator |
| Dashboards and alert rules about the cluster — node pressure, volume capacity, scheduling, CNI | Cluster operator |
| Cardinality and retention budget | Cluster operator, agreed with the platform administrator |
A cluster operator can tell that a pod is restarting. They cannot tell whether a rising task_schedule_to_start_latency means workers are saturated, whether route="unmatched" matters, or what normal looks like for a tenant that runs three connectors versus three hundred. That knowledge sits with whoever runs the OLIVE installation.
This matches the responsibility matrix, which already assigns platform-level alerting to the cluster operator and OLIVE-level alert rules and first-line triage to the platform administrator. What the cluster operator provides is a Grafana the administrator can build in, and the data behind it.
The practical consequence: the alert rules further down this page are a starting point for the platform administrator, not a to-do list for the cluster operator. Hand them over rather than implementing them blind.
What the cluster must provide
Four conditions have to hold before any OLIVE metric reaches you. Each one fails silently — a monitor whose selector matches no operator is created without error and simply never produces a target.
- A Prometheus-operator-compatible agent, with the
ServiceMonitorandPodMonitorCRDs installed. kube-prometheus-stack, the Victoria Metrics Operator and Grafana Alloy in operator mode all qualify. - A monitor per component you want to collect, created in the tenant namespace and carrying whatever label your agent's
serviceMonitorSelectororpodMonitorSelectormatches. OLIVE creates none of these; the table below is everything you need to write them. - A namespace selector that includes the tenant namespaces.
serviceMonitorNamespaceSelectorandpodMonitorNamespaceSelectormust cover them, or your own monitors are never read. PodMonitoras well asServiceMonitor. SeaweedFS keeps its metrics port off its Service, which publishes only S3 by design, so it needs aPodMonitor. If your agent handles onlyServiceMonitor, ask the administrator forseaweedfs.metrics.serviceEnabled: true, which publishes the port on the Service too.
Two facts have to travel the other way, from you to the platform administrator, or the configuration cannot be written: the namespace your scrape agent runs in, and the pod labels of its pods. Together they become the NetworkPolicy rule that lets your scrape through — and a selector that matches nothing is not an error, it is zero peers.
The endpoints are unauthenticated
The metrics endpoints carry no TLS, no bearer token and no authorization check. Any workload that can reach the port can read them. The NetworkPolicy each chart renders alongside the port is the entire access control, which is why OLIVE's guidance is to enable metrics only in namespaces where network policies are on.
The content is operational rather than business data — request counts, latency histograms, disk and memory figures, queue depths — with no payloads and no credentials. It does reveal route names, internal service names and traffic volumes, which has reconnaissance value.
Two cluster properties decide whether that control is real:
- The CNI must enforce NetworkPolicy. Cilium and Calico do. On Amazon EKS the AWS VPC CNI enforces it only when the add-on runs with
enableNetworkPolicy=true; without that the policies are accepted by the API server and do nothing, and the metrics ports are reachable from anywhere in the cluster. This is the same prerequisite described under Networking. - A mesh in ambient mode weakens port scoping. Traffic between meshed pods is tunnelled over HBONE on TCP 15008, so a policy admitting your scraper must admit 15008 too — which admits it to every port of that pod, not only the metrics port. Under ambient, treat the metrics policy as pod-level rather than port-level isolation.
What exposes metrics
| Component | Scrape via | Selector | Port name | Port | Path | Metric prefix |
|---|---|---|---|---|---|---|
| Express services (the OLIVE microservices) | Service | app.kubernetes.io/component: express | metrics | 9464 | /metrics | http_*, nodejs_*, process_* |
| CouchDB | Service | app.kubernetes.io/component: couchdb | metrics | 17986 | /_node/_local/_prometheus | couchdb_*, erlang_vm_* |
| SeaweedFS | Pod | app.kubernetes.io/component: seaweedfs | metrics | 9324 | /metrics | SeaweedFS_* |
| Temporal (frontend, internal-frontend, matching, history, worker) | Service (the five *-headless Services) | app.kubernetes.io/name: temporal | metrics | 9090 | /metrics | unprefixed — see below |
| PostgreSQL (CloudNativePG) | Pod | cnpg.io/cluster: <cluster-name> | — | 9187 | /metrics | cnpg_*, pg_* |
Monitors reference port names, not numbers — a Service port name for a ServiceMonitor, a container port name for a PodMonitor. A name that does not exist yields no target and no error.
Five things about that table are worth knowing before you build dashboards on it:
-
CouchDB does not serve
/metrics. Its endpoint is/_node/_local/_prometheus. A monitor pointed at/metricsscrapes a 404 and still reports the target as up, with zero samples. -
Temporal's metric names are unprefixed —
action,cache_size,service_requests. In a shared store they collide with other applications, so prefix them in your own monitor:metricRelabelings:- sourceLabels: [__name__]regex: '(.*)'targetLabel: __name__replacement: 'temporal_$1' -
Temporal labels its metrics with
namespace, meaning a Temporal namespace. The scrape attaches the Kubernetes namespace under the same name, so Temporal's own value arrives asexported_namespace. -
PostgreSQL is not covered by OLIVE's metrics flag. The CloudNativePG cluster has its own
monitoring.enablePodMonitor, and that one does create aPodMonitor— from the CloudNativePG operator's template, not from OLIVE. -
Temporal can be the exception, if you want it to be. The Temporal subchart bundled with OLIVE is able to create five
ServiceMonitorobjects, one per server role. OLIVE leaves that off, because monitors are yours. If you would rather have them from the chart than write your own, ask the administrator to enable them and give them the label your agent selects on.
Cardinality: Temporal is the one to budget for
Series land in your store, so series count is a platform cost rather than a tenant one.
| Component | Series scale | Driver |
|---|---|---|
| Express services | low per service, multiplied by service count | route × status_code × method; route is the route pattern, so it is bounded by the number of routes, not by traffic |
| CouchDB | low | a fixed set per node |
| SeaweedFS | low to moderate | grows with volume count |
| Temporal | high | latency histograms labelled by operation × Temporal namespace × service, across five server roles |
Temporal's timerType defaults to histogram. A summary timer type, or a drop list for the heaviest families in the monitor's metricRelabelings, cuts it substantially. Agree a budget before Temporal metrics are switched on, and measure the result:
count by (job) ({namespace="<tenant-namespace>"})
Verifying a tenant's metrics arrive
Targets exist and are scraped successfully:
up{namespace="<tenant-namespace>"}
Targets that answer but return nothing — the failure up cannot show. A target with up=1 and zero samples is a wrong path or a non-Prometheus endpoint:
scrape_samples_scraped{namespace="<tenant-namespace>"} == 0
A target reporting down is almost always the NetworkPolicy: either the rule was not rendered, or the agent's pod labels in the tenant's global.metrics.scrape do not match your agent's pods. Your agent's own targets page names the cause directly — connection refused, context deadline exceeded, 404 — which is faster than inferring it from queries. A whole namespace going down at once is a configuration change, not an outage.
Alert rules to start from
OLIVE ships no alert rules. These are starting points with deliberately conservative thresholds, and they belong to the platform administrator to adopt and tune — the cluster operator's part is that the rules can be loaded and that notifications route somewhere. Tune them per environment.
# A metrics endpoint that was being scraped stops answering.
# Scope it to namespaces where metrics are expected: it also fires on
# scale-to-zero and on an intentional metrics rollback.
- alert: OliveTargetDown
expr: up{namespace=~"tenant-.*"} == 0
for: 10m
labels: { severity: warning }
# Express service returning server errors.
- alert: OliveExpressErrorRate
expr: |
sum by (namespace, job) (rate(http_request_duration_seconds_count{status_code=~"5.."}[5m]))
/ sum by (namespace, job) (rate(http_request_duration_seconds_count[5m])) > 0.05
for: 10m
labels: { severity: warning }
# Express service latency regression.
- alert: OliveExpressLatencyHigh
expr: |
histogram_quantile(0.95, sum by (le, namespace, job) (
rate(http_request_duration_seconds_bucket[5m]))) > 2
for: 15m
labels: { severity: warning }
# Node.js event loop saturated: the service is CPU-bound, not waiting on I/O.
- alert: OliveEventLoopLag
expr: nodejs_eventloop_lag_p99_seconds > 0.5
for: 10m
labels: { severity: warning }
# Object storage filling up.
- alert: OliveSeaweedFsVolumeNearFull
expr: |
kubelet_volume_stats_available_bytes{persistentvolumeclaim=~".*seaweedfs.*"}
/ kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~".*seaweedfs.*"} < 0.15
for: 30m
labels: { severity: critical }
# Temporal workers not keeping up with scheduled tasks.
- alert: OliveTemporalTaskBacklog
expr: |
histogram_quantile(0.95, sum by (le, namespace) (
rate(temporal_task_schedule_to_start_latency_bucket[5m]))) > 5
for: 15m
labels: { severity: warning }
If OLIVE metrics are not enabled
Metrics are opt-in, and in some deployments they will stay off. Metric-based alerting is still possible on:
- Kubernetes-level metrics — pod restarts, resource usage, probe failures and scheduling, from the usual sources (kube-state-metrics, the kubelet and cAdvisor). This covers most availability alerting and is where CPU and memory pressure shows up. See Platform Scaling.
- The third-party components' own metrics — Vault, Keycloak, PostgreSQL and the rest expose what their projects document, independently of OLIVE's flag.
- The log stream — OLIVE's security signals are events, not gauges. Rules are written against the event codes; see Log Events and Automated Reactions.
The single most useful OLIVE-specific alert is not a metric at all: the OLIALIVE1 liveness event, which each tenant emits every five minutes, and whose absence means that tenant has stopped working end to end.
Metrics are aggregated, unauthenticated and retained on the observability stack's schedule rather than a compliance one. Access logs, authentication events and audit records belong to the log pipeline — see Log Events and Automated Reactions.
Cluster operator responsibilities
The cluster operator is expected to have the skills to:
- Select, install and configure the monitoring and observability tooling
- Operate log, trace and metrics collection pipelines end to end
- Own the cluster's own dashboards and alerts — nodes, capacity, scheduling, the CNI
- Own notification routing, and give the platform administrator a Grafana they can build dashboards in
OLIVE-specific dashboards and alert rules are the platform administrator's, because they depend on knowing what OLIVE's metrics mean and what normal looks like for a given tenant. See Who owns what and the responsibility matrix.
OLIVE stays compatible with common Kubernetes observability tooling; specific product choices remain with your operations team.