Skip to main content

Platform Scaling

An OLIVE deployment scales on three levels, and the first question is always which one you are actually short of. They are configured in different places, by different people, and changing the wrong one costs time without changing the symptom.

LevelWhat you changeWhereWho
ServiceReplicas, CPU and memory requests and limits per OLIVE servicevalues.yaml of the deploymentPlatform administrator
TenantNumber of tenant deployments, each with its own data storesInfrastructure repository, one deployment per tenantPlatform administrator
ClusterNodes, node sizes, storage capacity and classesKubernetes clusterCluster operator

This page identifies the level and says who owns it. The values themselves are documented where they are set: service-level scaling belongs with the chart values you edit at install time, so it lives in Resource Scaling.

Which level is the limit?​

Three commands separate them before you change anything.

# Service level: is one service at its limits while the others are not?
kubectl -n <tenant-namespace> top pods

# Cluster level: is there capacity left to give it?
kubectl top nodes

# Both: OOMKilled, failed scheduling, and unbound claims all appear here
kubectl -n <tenant-namespace> get events --sort-by=.lastTimestamp | tail -30

Read the result this way:

What you seeThe levelGo to
One service near its limits while nodes have headroomServiceScale a service
OOMKilled on a specific serviceServiceScale a service
Pods Pending, or requests no node can satisfyClusterScale the cluster
A claim that will not bind, or storage exhaustedClusterScale the cluster
Services healthy, but unrelated workloads contending for one tenant's data storesTenantScale by tenant

Two rules that save more time than any individual value:

  1. Change one thing. Scaling several services at once makes the result unreadable.
  2. Re-measure. Resource limits that were guessed once and never checked are the most common cause of both throttling and wasted capacity.

Scale a Service​

Most scaling work is here: a single service is CPU-throttled, running out of memory, or is a bottleneck under load. Stateless OLIVE services take replicas; the per-tenant data stores (CouchDB and SeaweedFS) are scaled vertically only, with temporal-caller as the exception.

See Resource Scaling for the values, the units, and how to verify the result.

Scale by Tenant​

A tenant is the unit of isolation in OLIVE, not a unit of capacity — but it behaves as one, because each tenant deployment brings its own CouchDB, SeaweedFS and Temporal worker. Splitting unrelated workloads into separate tenants therefore gives each of them its own data stores and its own failure domain, at the cost of that baseline footprint per tenant — 1.075 cores and about 1.7 GiB of requests at the chart defaults, as measured in Per deployment.

note

Do not create tenants to work around a single overloaded service. A tenant is permanent in practice: global.tenant builds the DNS subdomain, appears in log records, and names the Vault and Temporal accounts. Scale the service first.

See Tenant Onboarding and Offboarding for the procedure, and Environment and Deployment Profiles for how the deployment modes differ.

Scale the Cluster​

If pods stay Pending, the cluster — not the OLIVE configuration — is the limit. Requests that no node can satisfy, exhausted storage, or a StorageClass that cannot provision more volumes all surface the same way.

warning

OLIVE does not ship an observability stack and does not autoscale. Replicas and resources are what you set in the values; nothing adjusts them for you, and nothing alerts you when a service is at its limit unless you have configured that yourself.