Metrics Endpoints
OLIVE's workloads can expose Prometheus metrics endpoints. They are off by default and are switched on per namespace from values.yaml.
That is the whole of OLIVE's part. OLIVE exposes the port, names it, and opens a NetworkPolicy rule so a scraper can reach it. It does not scrape, store, visualise or alert — and it does not create the ServiceMonitor or PodMonitor objects that tell a scrape agent what to collect.
The pipeline is your cluster operator's; the OLIVE dashboards and alert rules are yours. They provide the scrape agent, the time-series store, retention and a Grafana you can build in; you enable the endpoints, and you build the dashboards and alerts that say whether a tenant is healthy — because that depends on knowing what OLIVE's metrics mean. The split is set out in Cluster Operator → Who owns what.
So this page covers one task: enabling the endpoints, and giving the operator the information they need to point a monitor at them.
Before you start: two values from the operator
The NetworkPolicy rule that admits the scraper is built from these. Without them the port is exposed but unreachable.
| What you need | Consequence if wrong |
|---|---|
| The namespace the scrape agent runs in | the scrape is dropped at the network; the target reports down in an agent you cannot see |
| The pod labels of the scrape agent's pods | same — and a selector that matches nothing is not an error |
The operator can read both off their own installation:
kubectl get pods -n <monitoring-namespace> --show-labels
Enable network policies first
They carry no TLS, no token and no authorization check. Any workload that can reach the port can read them. The NetworkPolicy each chart renders alongside the port is the only access control there is.
Do not enable metrics in a namespace where global.networkPolicy.enabled is false — the endpoints would be reachable from every pod in the cluster.
The metrics themselves are operational rather than business data: request counts, latency histograms, disk and memory figures, queue depths. No payloads, no credentials. They do reveal route names, internal service names and traffic volumes, which is reconnaissance value and the reason the port is worth restricting.
Enabling network policies is a change with its own blast radius, so treat it as a separate step: turn policies on, confirm the deployment is healthy, then enable metrics. Network Policies covers the two things that silently break it — a CNI that does not enforce policy, and an ingressController selector that matches nothing.
Turning metrics on
One switch covers every component in the namespace:
global:
networkPolicy:
enabled: true # prerequisite, see above
metrics:
enabled: true
# Who may reach the metrics ports. Rendered as ONE policy peer -
# namespaceSelector AND podSelector - so it means "the agent's pods in
# that namespace", not "every pod in that namespace".
scrape:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
podLabels:
app.kubernetes.io/name: prometheus
scrape is mandatory once metrics are enabled. The charts fail the Helm render rather than open a port with an empty namespaceSelector, which would mean every namespace in the cluster.
A component can be enabled or excluded individually:
couchdb:
metrics:
enabled: true
seaweedfs:
metrics:
enabled: true
workflow-engine-postgres:
metrics:
enabled: true
Deleting these keys entirely is cleaner than setting them to true — they then inherit the global value, and there is one fewer place to contradict yourself later.
What the operator needs in order to scrape
Hand this table over once metrics are enabled. It is everything required to write a ServiceMonitor or PodMonitor, and the OLIVE side of it does not change between deployments.
| Component | Scrape via | Selector | Port name | Port | Path |
|---|---|---|---|---|---|
| Express services (the OLIVE microservices) | Service | app.kubernetes.io/component: express | metrics | 9464 | /metrics |
| CouchDB | Service | app.kubernetes.io/component: couchdb | metrics | 17986 | /_node/_local/_prometheus |
| SeaweedFS | Pod | app.kubernetes.io/component: seaweedfs | metrics | 9324 | /metrics |
| Temporal | Service (the five *-headless Services) | app.kubernetes.io/name: temporal | metrics | 9090 | /metrics |
| PostgreSQL (CloudNativePG) | Pod | cnpg.io/cluster: <cluster-name> | — | 9187 | /metrics |
Monitors reference port names, not numbers — a Service port name for a ServiceMonitor, a container port name for a PodMonitor. A name that does not exist yields no target and no error.
Four points the operator will need spelled out:
- CouchDB does not serve
/metrics. Its endpoint is/_node/_local/_prometheus. A monitor pointed at/metricsscrapes a 404 and still reports the target as up, with zero samples. - SeaweedFS keeps its metrics port off the Service by default, because that Service deliberately publishes only S3. It therefore needs a
PodMonitor. If the operator's agent handles onlyServiceMonitor, OLIVE can publish the port on the Service as well — see below. - Temporal's metric names are unprefixed —
action,cache_size,service_requests— and collide with other applications in a shared store. The operator should add ametricRelabelingsrule in their monitor to prefix them. - Temporal labels its metrics with
namespace, meaning a Temporal namespace. The scrape attaches the Kubernetes namespace under the same name, so Temporal's own value arrives asexported_namespace.
If the operator's agent supports only ServiceMonitor
SeaweedFS can publish its metrics port on the ClusterIP Service, which makes it scrapable by a ServiceMonitor:
seaweedfs:
metrics:
enabled: true
serviceEnabled: true # also publish the metrics port on the Service
This widens the Service's surface, so leave it off unless the operator needs it.
Temporal
Temporal already serves metrics on 9090 regardless of OLIVE's flag; the headless Services publish that port and the containers already carry it. Enabling only opens the NetworkPolicy, so no Temporal pod restarts.
The bundled Temporal subchart can create its own five ServiceMonitor objects, one per server role. OLIVE's default is not to: monitors belong to the operator. If the operator would rather have them from the chart, enable them there and supply the label their agent selects on:
workflow-engine-postgres:
metrics:
enabled: true
temporal:
server:
metrics:
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack # ask the operator for key and value
Otherwise declare that the monitors come from outside the chart, which keeps the render consistent and prevents the same endpoints being scraped twice:
workflow-engine-postgres:
metrics:
enabled: true
externalMonitor: true
Exactly one of the two must be set. Enabling both is refused at render time, because two monitors on one endpoint double every series.
Temporal emits latency histograms labelled by operation × Temporal namespace × service, across five server roles, and is by far the largest contributor to series count in an OLIVE namespace. Those series land in the operator's shared store, so agree a budget with them first. A summary timer type, or a drop list in the operator's metricRelabelings, cuts it substantially.
PostgreSQL
The CloudNativePG cluster is not covered by global.metrics.enabled. It has its own switch:
workflow-engine-postgres:
cnpg:
cluster:
monitoring:
enablePodMonitor: true
Note that this one does create a PodMonitor, from the CloudNativePG operator's own template rather than from OLIVE.
Verifying the endpoints are exposed
This is the part you can verify yourself. Whether the metrics then reach a dashboard is the operator's half.
The ports exist. A missing metrics port means the flag did not take effect:
kubectl get svc -n <namespace> -l app.kubernetes.io/component=express \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.ports[*].name}{"\n"}{end}'
kubectl get svc service-couchdb -n <namespace> -o jsonpath='{.spec.ports[*].name}' ; echo
kubectl get pod service-seaweedfs-0 -n <namespace> \
-o jsonpath='{.spec.containers[*].ports[*].name}' ; echo
The NetworkPolicy admits the scraper. Compare what it allows against the agent's real namespace and pod labels:
kubectl describe networkpolicy -n <namespace> <chart>-... | sed -n '/Allowing ingress/,$p'
The endpoint actually serves. The Express images are distroless and have no shell, so use port-forward rather than kubectl exec:
kubectl port-forward -n <namespace> deploy/<service>-express 19464:9464
curl -s localhost:19464/metrics | head -20
The service logged that the listener started:
kubectl logs -n <namespace> deploy/<service>-express | grep service-util-metrics
If all four pass and metrics still do not appear, the remaining links are the operator's: whether a monitor exists for that component, whether their agent's namespace selector covers your namespace, and what error their target reports.
What the endpoints expose
Useful when agreeing dashboards and a series budget with the operator.
Express services
Every OLIVE microservice built on the Express base chart exposes the same set, on a listener separate from the application port. Health-check paths are excluded by default, so probe traffic does not dominate the latency histograms.
| Metric | Type | Labels |
|---|---|---|
http_request_duration_seconds | histogram | method, route, status_code |
http_requests_in_flight | gauge | method |
nodejs_* | various | event loop lag, heap, handles, garbage collection |
process_* | various | CPU, resident memory, file descriptors |
Every series carries service and version. There is deliberately no tenant label: on a pooled service the tenant varies per request, so it would multiply every series by the number of customers.
route is the route pattern, not the request URL — /result/:jobID is one series rather than one per job, which is what keeps the series count bounded by the number of routes instead of by traffic. A route="unmatched" value means requests arrived that matched no route.
Request count comes from the histogram's _count series. There is no separate counter metric.
The other components
| Component | Metric prefix | Series scale |
|---|---|---|
| CouchDB | couchdb_*, erlang_vm_* | low — a fixed set per node |
| SeaweedFS | SeaweedFS_* (capitalised) | low to moderate — grows with volume count |
| Temporal | unprefixed (action, service_requests, …) | high — see the note above |
| PostgreSQL | cnpg_*, pg_* | low |
Example: reading the metrics once they are collected
The queries below assume the operator's observability stack scrapes your namespace and exposes it through Grafana. They are illustrative — the dashboards themselves are the operator's to build. Replace <namespace> and <service>.
Two things that make queries look broken when they are not. Use a fixed range such as [5m] rather than $__rate_interval, which resolves from the panel's time range and can fall below the scrape interval. And rate() needs at least two samples in the window, so a query run immediately after enabling metrics returns nothing even when everything is correct — wait two scrape intervals.
Listing what is actually there, as an instant query. This is the most reliable starting point for any component:
group by (__name__) ({job=~".*express.*"})
Express services:
# request rate by route and status
sum by (route, status_code) (
rate(http_request_duration_seconds_count{job="<service>-express"}[5m]))
# p95 latency by route
histogram_quantile(0.95, sum by (le, route) (
rate(http_request_duration_seconds_bucket{job="<service>-express"}[5m])))
# error ratio per service across the namespace
sum by (job) (rate(http_request_duration_seconds_count{namespace="<namespace>", status_code=~"5.."}[5m]))
/ sum by (job) (rate(http_request_duration_seconds_count{namespace="<namespace>"}[5m]))
# requests in flight - concurrency, and a stuck-request signal if it never falls
http_requests_in_flight{namespace="<namespace>"}
# event loop lag: high means CPU-bound, not waiting on I/O
nodejs_eventloop_lag_p99_seconds{namespace="<namespace>"}
CouchDB:
# request outcomes - the main health signal
sum by (code) (rate(couchdb_httpd_status_codes{namespace="<namespace>"}[5m]))
# open file handles: CouchDB holds one per shard, and this precedes hard failures
couchdb_open_os_files{namespace="<namespace>"}
# the Erlang VM, which is CouchDB's real resource ceiling
erlang_vm_memory_bytes_total{namespace="<namespace>"}
SeaweedFS — names are capitalised, and PromQL is case-sensitive:
# disk used per volume server
sum by (pod) (SeaweedFS_volumeServer_total_disk_size{namespace="<namespace>"})
# volume count against the configured maximum
SeaweedFS_volumeServer_volumes{namespace="<namespace>"}
SeaweedFS_volumeServer_max_volumes{namespace="<namespace>"}
For remaining capacity, measure the volume rather than SeaweedFS's own view — the configured volume maximum can exceed the PVC size, and the PVC then fills first:
kubelet_volume_stats_available_bytes{namespace="<namespace>", persistentvolumeclaim=~".*seaweedfs.*"}
/ kubelet_volume_stats_capacity_bytes{namespace="<namespace>", persistentvolumeclaim=~".*seaweedfs.*"}
Temporal — add the operator's prefix if they applied one:
# p95 service latency by role
histogram_quantile(0.95, sum by (le, service_name) (
rate(service_latency_bucket{namespace="<namespace>"}[5m])))
# persistence latency - usually the real bottleneck
histogram_quantile(0.95, sum by (le, operation) (
rate(persistence_latency_bucket{namespace="<namespace>"}[5m])))
# task backlog: are workers keeping up
histogram_quantile(0.95, sum by (le, task_type) (
rate(task_schedule_to_start_latency_bucket{namespace="<namespace>"}[5m])))
Of these, task_schedule_to_start_latency is the one that says something about your workflows rather than about Temporal: a rising p95 means tasks are queuing because workers are saturated.
Container CPU and memory come from the cluster's own collectors rather than from OLIVE, so they are available whether or not these endpoints are enabled, and they feed sizing decisions — see Resource Scaling and Platform Scaling.
Turning metrics off
Set global.metrics.enabled: false, or the per-component flag. The ports and NetworkPolicy rules disappear and the affected pods roll — except Temporal, which keeps serving on 9090 but becomes unreachable to the scraper.
Tell the operator, so they can remove the monitors pointing at the endpoints. A monitor left behind simply reports its targets as down.
Related
- Network Policies — the prerequisite, and its own silent failure modes
- Resource Scaling — what to do with what the metrics tell you
- Cluster Operator → Monitoring and Observability — what the operator provides, and how they build monitors against these endpoints
- Cluster Operator → Log Events and Automated Reactions — for security and audit signals, which are events rather than metrics