Skip to main content

Metrics Endpoints

OLIVE's workloads can expose Prometheus metrics endpoints. They are off by default and are switched on per namespace from values.yaml.

That is the whole of OLIVE's part. OLIVE exposes the port, names it, and opens a NetworkPolicy rule so a scraper can reach it. It does not scrape, store, visualise or alert — and it does not create the ServiceMonitor or PodMonitor objects that tell a scrape agent what to collect.

The pipeline is your cluster operator's; the OLIVE dashboards and alert rules are yours. They provide the scrape agent, the time-series store, retention and a Grafana you can build in; you enable the endpoints, and you build the dashboards and alerts that say whether a tenant is healthy — because that depends on knowing what OLIVE's metrics mean. The split is set out in Cluster Operator → Who owns what.

So this page covers one task: enabling the endpoints, and giving the operator the information they need to point a monitor at them.

Before you start: two values from the operator​

The NetworkPolicy rule that admits the scraper is built from these. Without them the port is exposed but unreachable.

What you needConsequence if wrong
The namespace the scrape agent runs inthe scrape is dropped at the network; the target reports down in an agent you cannot see
The pod labels of the scrape agent's podssame — and a selector that matches nothing is not an error

The operator can read both off their own installation:

kubectl get pods -n <monitoring-namespace> --show-labels

Enable network policies first​

The metrics endpoints are unauthenticated

They carry no TLS, no token and no authorization check. Any workload that can reach the port can read them. The NetworkPolicy each chart renders alongside the port is the only access control there is.

Do not enable metrics in a namespace where global.networkPolicy.enabled is false — the endpoints would be reachable from every pod in the cluster.

The metrics themselves are operational rather than business data: request counts, latency histograms, disk and memory figures, queue depths. No payloads, no credentials. They do reveal route names, internal service names and traffic volumes, which is reconnaissance value and the reason the port is worth restricting.

Enabling network policies is a change with its own blast radius, so treat it as a separate step: turn policies on, confirm the deployment is healthy, then enable metrics. Network Policies covers the two things that silently break it — a CNI that does not enforce policy, and an ingressController selector that matches nothing.

Turning metrics on​

One switch covers every component in the namespace:

global:
networkPolicy:
enabled: true # prerequisite, see above

metrics:
enabled: true

# Who may reach the metrics ports. Rendered as ONE policy peer -
# namespaceSelector AND podSelector - so it means "the agent's pods in
# that namespace", not "every pod in that namespace".
scrape:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
podLabels:
app.kubernetes.io/name: prometheus

scrape is mandatory once metrics are enabled. The charts fail the Helm render rather than open a port with an empty namespaceSelector, which would mean every namespace in the cluster.

A component can be enabled or excluded individually:

couchdb:
metrics:
enabled: true

seaweedfs:
metrics:
enabled: true

workflow-engine-postgres:
metrics:
enabled: true

Deleting these keys entirely is cleaner than setting them to true — they then inherit the global value, and there is one fewer place to contradict yourself later.

What the operator needs in order to scrape​

Hand this table over once metrics are enabled. It is everything required to write a ServiceMonitor or PodMonitor, and the OLIVE side of it does not change between deployments.

ComponentScrape viaSelectorPort namePortPath
Express services (the OLIVE microservices)Serviceapp.kubernetes.io/component: expressmetrics9464/metrics
CouchDBServiceapp.kubernetes.io/component: couchdbmetrics17986/_node/_local/_prometheus
SeaweedFSPodapp.kubernetes.io/component: seaweedfsmetrics9324/metrics
TemporalService (the five *-headless Services)app.kubernetes.io/name: temporalmetrics9090/metrics
PostgreSQL (CloudNativePG)Podcnpg.io/cluster: <cluster-name>—9187/metrics

Monitors reference port names, not numbers — a Service port name for a ServiceMonitor, a container port name for a PodMonitor. A name that does not exist yields no target and no error.

Four points the operator will need spelled out:

  • CouchDB does not serve /metrics. Its endpoint is /_node/_local/_prometheus. A monitor pointed at /metrics scrapes a 404 and still reports the target as up, with zero samples.
  • SeaweedFS keeps its metrics port off the Service by default, because that Service deliberately publishes only S3. It therefore needs a PodMonitor. If the operator's agent handles only ServiceMonitor, OLIVE can publish the port on the Service as well — see below.
  • Temporal's metric names are unprefixed — action, cache_size, service_requests — and collide with other applications in a shared store. The operator should add a metricRelabelings rule in their monitor to prefix them.
  • Temporal labels its metrics with namespace, meaning a Temporal namespace. The scrape attaches the Kubernetes namespace under the same name, so Temporal's own value arrives as exported_namespace.

If the operator's agent supports only ServiceMonitor​

SeaweedFS can publish its metrics port on the ClusterIP Service, which makes it scrapable by a ServiceMonitor:

seaweedfs:
metrics:
enabled: true
serviceEnabled: true # also publish the metrics port on the Service

This widens the Service's surface, so leave it off unless the operator needs it.

Temporal​

Temporal already serves metrics on 9090 regardless of OLIVE's flag; the headless Services publish that port and the containers already carry it. Enabling only opens the NetworkPolicy, so no Temporal pod restarts.

The bundled Temporal subchart can create its own five ServiceMonitor objects, one per server role. OLIVE's default is not to: monitors belong to the operator. If the operator would rather have them from the chart, enable them there and supply the label their agent selects on:

workflow-engine-postgres:
metrics:
enabled: true
temporal:
server:
metrics:
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack # ask the operator for key and value

Otherwise declare that the monitors come from outside the chart, which keeps the render consistent and prevents the same endpoints being scraped twice:

workflow-engine-postgres:
metrics:
enabled: true
externalMonitor: true

Exactly one of the two must be set. Enabling both is refused at render time, because two monitors on one endpoint double every series.

Temporal is the one to discuss before enabling

Temporal emits latency histograms labelled by operation × Temporal namespace × service, across five server roles, and is by far the largest contributor to series count in an OLIVE namespace. Those series land in the operator's shared store, so agree a budget with them first. A summary timer type, or a drop list in the operator's metricRelabelings, cuts it substantially.

PostgreSQL​

The CloudNativePG cluster is not covered by global.metrics.enabled. It has its own switch:

workflow-engine-postgres:
cnpg:
cluster:
monitoring:
enablePodMonitor: true

Note that this one does create a PodMonitor, from the CloudNativePG operator's own template rather than from OLIVE.

Verifying the endpoints are exposed​

This is the part you can verify yourself. Whether the metrics then reach a dashboard is the operator's half.

The ports exist. A missing metrics port means the flag did not take effect:

kubectl get svc -n <namespace> -l app.kubernetes.io/component=express \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.ports[*].name}{"\n"}{end}'

kubectl get svc service-couchdb -n <namespace> -o jsonpath='{.spec.ports[*].name}' ; echo

kubectl get pod service-seaweedfs-0 -n <namespace> \
-o jsonpath='{.spec.containers[*].ports[*].name}' ; echo

The NetworkPolicy admits the scraper. Compare what it allows against the agent's real namespace and pod labels:

kubectl describe networkpolicy -n <namespace> <chart>-... | sed -n '/Allowing ingress/,$p'

The endpoint actually serves. The Express images are distroless and have no shell, so use port-forward rather than kubectl exec:

kubectl port-forward -n <namespace> deploy/<service>-express 19464:9464
curl -s localhost:19464/metrics | head -20

The service logged that the listener started:

kubectl logs -n <namespace> deploy/<service>-express | grep service-util-metrics

If all four pass and metrics still do not appear, the remaining links are the operator's: whether a monitor exists for that component, whether their agent's namespace selector covers your namespace, and what error their target reports.

What the endpoints expose​

Useful when agreeing dashboards and a series budget with the operator.

Express services​

Every OLIVE microservice built on the Express base chart exposes the same set, on a listener separate from the application port. Health-check paths are excluded by default, so probe traffic does not dominate the latency histograms.

MetricTypeLabels
http_request_duration_secondshistogrammethod, route, status_code
http_requests_in_flightgaugemethod
nodejs_*variousevent loop lag, heap, handles, garbage collection
process_*variousCPU, resident memory, file descriptors

Every series carries service and version. There is deliberately no tenant label: on a pooled service the tenant varies per request, so it would multiply every series by the number of customers.

route is the route pattern, not the request URL — /result/:jobID is one series rather than one per job, which is what keeps the series count bounded by the number of routes instead of by traffic. A route="unmatched" value means requests arrived that matched no route.

Request count comes from the histogram's _count series. There is no separate counter metric.

The other components​

ComponentMetric prefixSeries scale
CouchDBcouchdb_*, erlang_vm_*low — a fixed set per node
SeaweedFSSeaweedFS_* (capitalised)low to moderate — grows with volume count
Temporalunprefixed (action, service_requests, …)high — see the note above
PostgreSQLcnpg_*, pg_*low

Example: reading the metrics once they are collected​

The queries below assume the operator's observability stack scrapes your namespace and exposes it through Grafana. They are illustrative — the dashboards themselves are the operator's to build. Replace <namespace> and <service>.

Two things that make queries look broken when they are not. Use a fixed range such as [5m] rather than $__rate_interval, which resolves from the panel's time range and can fall below the scrape interval. And rate() needs at least two samples in the window, so a query run immediately after enabling metrics returns nothing even when everything is correct — wait two scrape intervals.

Listing what is actually there, as an instant query. This is the most reliable starting point for any component:

group by (__name__) ({job=~".*express.*"})

Express services:

# request rate by route and status
sum by (route, status_code) (
rate(http_request_duration_seconds_count{job="<service>-express"}[5m]))

# p95 latency by route
histogram_quantile(0.95, sum by (le, route) (
rate(http_request_duration_seconds_bucket{job="<service>-express"}[5m])))

# error ratio per service across the namespace
sum by (job) (rate(http_request_duration_seconds_count{namespace="<namespace>", status_code=~"5.."}[5m]))
/ sum by (job) (rate(http_request_duration_seconds_count{namespace="<namespace>"}[5m]))

# requests in flight - concurrency, and a stuck-request signal if it never falls
http_requests_in_flight{namespace="<namespace>"}

# event loop lag: high means CPU-bound, not waiting on I/O
nodejs_eventloop_lag_p99_seconds{namespace="<namespace>"}

CouchDB:

# request outcomes - the main health signal
sum by (code) (rate(couchdb_httpd_status_codes{namespace="<namespace>"}[5m]))

# open file handles: CouchDB holds one per shard, and this precedes hard failures
couchdb_open_os_files{namespace="<namespace>"}

# the Erlang VM, which is CouchDB's real resource ceiling
erlang_vm_memory_bytes_total{namespace="<namespace>"}

SeaweedFS — names are capitalised, and PromQL is case-sensitive:

# disk used per volume server
sum by (pod) (SeaweedFS_volumeServer_total_disk_size{namespace="<namespace>"})

# volume count against the configured maximum
SeaweedFS_volumeServer_volumes{namespace="<namespace>"}
SeaweedFS_volumeServer_max_volumes{namespace="<namespace>"}

For remaining capacity, measure the volume rather than SeaweedFS's own view — the configured volume maximum can exceed the PVC size, and the PVC then fills first:

kubelet_volume_stats_available_bytes{namespace="<namespace>", persistentvolumeclaim=~".*seaweedfs.*"}
/ kubelet_volume_stats_capacity_bytes{namespace="<namespace>", persistentvolumeclaim=~".*seaweedfs.*"}

Temporal — add the operator's prefix if they applied one:

# p95 service latency by role
histogram_quantile(0.95, sum by (le, service_name) (
rate(service_latency_bucket{namespace="<namespace>"}[5m])))

# persistence latency - usually the real bottleneck
histogram_quantile(0.95, sum by (le, operation) (
rate(persistence_latency_bucket{namespace="<namespace>"}[5m])))

# task backlog: are workers keeping up
histogram_quantile(0.95, sum by (le, task_type) (
rate(task_schedule_to_start_latency_bucket{namespace="<namespace>"}[5m])))

Of these, task_schedule_to_start_latency is the one that says something about your workflows rather than about Temporal: a rising p95 means tasks are queuing because workers are saturated.

Container CPU and memory come from the cluster's own collectors rather than from OLIVE, so they are available whether or not these endpoints are enabled, and they feed sizing decisions — see Resource Scaling and Platform Scaling.

Turning metrics off​

Set global.metrics.enabled: false, or the per-component flag. The ports and NetworkPolicy rules disappear and the affected pods roll — except Temporal, which keeps serving on 9090 but becomes unreachable to the scraper.

Tell the operator, so they can remove the monitors pointing at the endpoints. A monitor left behind simply reports its targets as down.