Resource Scaling
The OLIVE Platform's microservice architecture allows you to scale services both horizontally (adding pod replicas) and vertically (adjusting CPU/memory resources). This guide covers manual scaling through values.yaml configuration.
This is the service level — the values you set when you install or change a deployment. If you are not yet sure that a single service is what you are short of, Platform Scaling identifies which of the three levels is the limit, and which of them are the cluster operator's rather than yours.
For what the charts request before you change anything — per deployment, per tenant and per infrastructure chart — see Measured Resource Footprint.
Scaling Concepts
Horizontal Scaling
Horizontal scaling increases the number of pod replicas running a service. This distributes load across multiple instances and improves availability.
When to use:
- Service is CPU or I/O bound and can process requests in parallel
- You need high availability and fault tolerance
- Load is unpredictable or variable
Services that scale well horizontally (stateless):
data-orchestratormicrofrontend-interpreterschedulercloud-function-registryeditortemplate-data-retrieversession-memoryrepositoryfile-storage-serviceimport-exportui-component-registryinstantiator- Most extension services (prompts, agents, etc.)
Services that scale differently (stateful):
couchdb- Uses clustering, requires special configurationseaweedfs- Uses volume replication, requires special configurationtemporal-caller- Worker for workflow connectors; deployed per tenant and supports both horizontal and vertical scaling (exception to stateful services scaling constraints)
Vertical Scaling
Vertical scaling adjusts the CPU and memory resources allocated to each pod. This is useful when a service needs more processing power or memory to handle its workload.
When to use:
- Service is consistently using most of its allocated resources
- You see OOMKilled (out of memory) errors
- CPU utilization is very high
- A bottleneck is identified in a single service
Scaling Stateful Services Per Tenant
Stateful services in the OLIVE Platform (couchdb and seaweedfs) are deployed for each tenant. Currently, only vertical scaling is supported for stateful services, allowing you to increase CPU and memory resources allocated to each tenant's service pods to handle more complex or resource-intensive operations.
Vertical Scaling for Stateful Services: Increase CPU and memory resources for a stateful service's pods for a specific tenant, allowing it to handle increased workload demands.
Exception: temporal-caller Horizontal Scaling
The temporal-caller service is a stateful worker for workflow connectors that supports both horizontal and vertical scaling per tenant. This makes sense when a tenant has significant "connector workflow" activity:
- Horizontal Scaling: Add more replicas of
temporal-callerfor a specific tenant to increase processing capacity for workflow connector executions - Vertical Scaling: Increase CPU and memory resources for
temporal-callerpods to handle more complex or longer-running connector workflows
Each tenant's temporal-caller worker can be independently scaled to match their specific workflow execution demands.
Storage is also a per-tenant decision, and only at deploy time. couchdb and seaweedfs each claim their own volume per tenant, sized from values that are read when the claim is created. Estimate the tenant's storage need and set it before the first deployment — raising the value afterwards does not resize the volume. See Volume sizes.
Configure Horizontal Scaling
Every Express service is the shared express chart, declared as a dependency under that name. So its values sit under express:, one level below the service:
data-orchestrator:
express:
replicas: 3
replicas directly under the service does nothingdata-orchestrator: { replicas: 3 } is accepted by Helm, renders cleanly, syncs green — and changes nothing, because the express subchart never sees it. The deployment stays at one replica.
There is no replicaCount field either; the value is replicas. The same nesting applies to resources, env and everything else the service chart passes through.
The stateful charts are the exception: couchdb and seaweedfs are not Express services, so their resources sit directly under the chart name — see below.
This creates 3 replicas of the data-orchestrator service. Each replica is an independent pod that can process requests.
Example: Multiple Services with Different Replica Counts
data-orchestrator:
express:
replicas: 3
cloud-function-registry:
express:
replicas: 2
editor:
express:
replicas: 2
microfrontend-interpreter:
express:
replicas: 3
Verify Horizontal Scaling
After deploying or updating values, verify the replicas are running:
kubectl -n <tenant-namespace> get pods -l app.kubernetes.io/name=data-orchestrator
kubectl -n <tenant-namespace> get deployment data-orchestrator -o wide
Expected output shows N replicas in the READY column:
NAME READY UP-TO-DATE AVAILABLE AGE
data-orchestrator-5d4c7b9f8c 3/3 3 3 10m
Configure Vertical Scaling
To adjust CPU and memory resources for a service, add a resources section under express:, with requests and limits. The same nesting rule as for replicas applies, and so does the same silent failure if it is omitted:
Understanding Requests and Limits
Requests: The minimum resources Kubernetes guarantees for the pod. The scheduler uses this to decide which node to place the pod on.
Limits: The maximum resources the pod can use. If exceeded, the pod is throttled (CPU) or killed (memory).
Example: Increase Resources for Data Orchestrator
data-orchestrator:
express:
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
This configuration means:
- Guaranteed: At least 500m CPU and 1Gi memory
- Maximum: Up to 2 CPUs and 4Gi memory
- If the pod uses more than 4Gi memory, it will be OOMKilled
- If the pod tries to use more than 2 CPUs continuously, it will be throttled
Resource Units
- CPU: Measured in millicores (m).
1000m = 1 CPU,500m = 0.5 CPU,100m = 0.1 CPU - Memory: Measured in Mebibytes (Mi) or Gibibytes (Gi).
1Gi = 1024Mi,512Mi = 0.5Gi
Example: Complete Vertical Scaling Configuration
data-orchestrator:
express:
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
cloud-function-registry:
express:
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1
memory: 2Gi
instantiator:
express:
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2
memory: 4Gi
Example: Stateful Services per Tenant (couchdb, seaweedfs)
The stateful services ship with deliberately small defaults, sized for a tenant with little content. Review them per tenant rather than per deployment:
| Chart | Value | Default |
|---|---|---|
couchdb | couchdbPvcSize | 1Gi |
couchdb | resources.requests | 256Mi / 100m |
couchdb | resources.limits | 1024Mi / 500m |
seaweedfs | seaweed.pvcSize | 20Gi |
seaweedfs | resources.requests | 512Mi / 100m |
seaweedfs | resources.limits | 1Gi / 1 |
seaweedfs | seaweed.goMemLimit | 800MiB |
couchdb:
couchdbPvcSize: '5Gi'
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "500m"
seaweedfs:
resources:
requests:
cpu: "100m"
memory: "1Gi"
limits:
cpu: "500m"
memory: "2Gi"
seaweed:
pvcSize: '30Gi'
goMemLimit: '1600MiB' # ~80% of limits.memory
Note that seaweed.pvcSize sits under the seaweed key, while couchdbPvcSize is a top-level value of its chart. SeaweedFS is capped at 32 GB of usable object storage regardless of the volume size, so raising seaweed.pvcSize beyond that adds no usable capacity — see Cluster Operator → Sizing the volumes.
SeaweedFS runs its roles in a single Go process whose soft memory ceiling comes from seaweed.goMemLimit, not from the container limit. Leaving it at the default while raising resources.limits.memory means the process still holds itself to 800MiB and the additional memory is never used. Raise it with the limit, to roughly 80% of it.
couchdb needs no equivalent setting — it uses the container limit directly.
Monitor Resource Usage
After deploying with resource configuration, monitor actual usage to validate your settings:
Check Current Resource Usage
kubectl -n <tenant-namespace> top pods
Output example:
NAME CPU(cores) MEMORY(bytes)
data-orchestrator-5d4c7b9f8c-abc 420m 2800Mi
data-orchestrator-5d4c7b9f8c-def 380m 2650Mi
data-orchestrator-5d4c7b9f8c-ghi 450m 2900Mi
Check Node-Level Resource Usage
kubectl top nodes
View Resource Requests and Limits
kubectl -n <tenant-namespace> describe pod <pod-name> | grep -A 5 "Limits\|Requests"
Detect Resource Pressure
Check for OOMKilled events:
kubectl -n <tenant-namespace> get events | grep OOMKilled
Check pod logs for restart reasons:
kubectl -n <tenant-namespace> describe pod <pod-name> | grep -A 2 "Last State"
Scaling Guidelines
General Principles
- Start Conservative: Begin with the default values and increase incrementally based on observed usage
- Request = Expected Load: Set
requeststo what you expect the service to normally use - Limits = Safety Margin: Set
limitshigher than requests to handle spikes, but not so high that you waste resources - Monitor Continuously: Track metrics after each change to validate your assumptions
Performance Tuning Workflow
- Baseline: Deploy with default values and monitor for 1-2 weeks
- Analyze: Identify services with consistently high resource usage or errors
- Scale: Increase replicas or resources for bottleneck services
- Test: Redeploy and monitor for improvements
- Iterate: Repeat until performance targets are met
Common Issues
Pods Pending (Not Scheduling)
Cause: Insufficient node resources for requests.
Solution:
- Lower the
requestsvalues - Add more nodes to the cluster
- Check current node allocation:
kubectl top nodes
OOMKilled Pods
Cause: Pod exceeded memory limits.
Solution:
- Increase
limits.memory - Enable horizontal scaling to distribute load
- Check for memory leaks in application logs
High Latency / Slow Performance
Cause: Service is CPU-throttled due to low limits.
Solution:
- Increase
limits.cpu - Add replicas for horizontal scaling
- Profile the service to identify hotspots
Related Topics
- Chart Values Reference - Core configuration properties
- Cron Scheduler Configuration - Configure scheduled tasks
- Measured Resource Footprint - What each deployment, tenant and infrastructure chart requests in 1.3.0