Troubleshoot Temporal Deployment
Overview
In rare cases, Temporal workloads may become stuck in a "Progressing" state during OLIVE platform deployment. This issue is caused by a race condition where Temporal attempts to initialize its database schema before the CloudNativePG database is fully ready to accept connections.
Root Cause
The OLIVE platform uses CloudNativePG to initialize and manage the PostgreSQL database that Temporal requires. Under certain conditions, a race condition can occur where:
- The Temporal schema initialization job (
temporal-schema-1) starts before the database is fully available - The connection pooling layer (
temporal-pg-poller-rw) has not yet established connections to the database - Temporal services attempt to start before the schema has been successfully created
This results in workloads being stuck in a "Progressing" state and unable to complete their initialization.
Identify the Issue
To check if your Temporal workloads are stuck:
kubectl get jobs -n <namespace> | grep temporal
Look for jobs with a status that shows they are not completing, particularly:
temporal-schema-1workflow-infra-onboard- Any other Temporal-related jobs that appear stuck
You can also check the pod logs to confirm the issue:
kubectl logs -n <namespace> <stuck-job-pod-name>
Debug with Temporal Web UI
For detailed workflow execution insights, use the Temporal Web UI. It provides visibility into workflow history, task progress, retries, failures, and state transitions.
Activate the Temporal Web UI
The Temporal Web UI is disabled by default, deliberately. It provides no tenant isolation: anyone who can reach it sees the workflow executions of every tenant in that deployment.
Enable it for debugging in a development deployment. In a production deployment it must not be enabled in the shared runtime pool, because that is where the workflow executions of all tenants live.
Connectivity engineers cannot do this themselves — the connectivity handbook tells them to ask you — so the request usually arrives as "please enable the Temporal Web UI on the development deployment".
Which values you set, and where, depends on the deployment mode. The UI itself runs where Temporal runs; the Ingress that exposes it belongs to the deployment that owns the hostname.
Development deployment (single-no-auth)
One deployment holds everything, so both values go into the same values.yaml:
workflow-engine-postgres:
enableUIIngress: true
temporal:
web:
enabled: true
Production deployment (multi-shared + multi-tenant)
Temporal runs in the shared runtime pool, while the hostname belongs to the tenant, so the two values are split across two deployments.
In the values.yaml of the tenant deployment (multi-tenant) — this creates the Ingress under the tenant's own hostname:
workflow-engine-postgres:
enableUIIngress: true
In the values.yaml of the shared runtime pool deployment (multi-shared) — this starts the Web UI component itself:
workflow-engine-postgres:
temporal:
web:
enabled: true
Setting only one of the two produces a silent half-state rather than an error: the tenant-side value alone gives an Ingress with nothing behind it, and the pool-side value alone starts a UI that no hostname reaches.
temporal.web.enabled in the shared runtime pool is not per tenant. Once the UI runs there, every tenant's workflow executions are visible through it, and any tenant that has enableUIIngress: true can reach it. Do not enable it in a shared runtime pool that carries real customer data.
If you need to inspect a production problem, reproduce it in a development deployment rather than opening the UI in production.
After enabling the values, wait a few minutes for the Temporal Web UI components to start.
To turn it off again, set the values back to false and re-sync — in a production deployment on both sides. The Ingress is removed with the tenant-side value; the UI component stops with the pool-side value.
The UI is available at:
https://workflow.<tenant>.<clusterName>.<domain>/
Example:
https://workflow.default-tenant.develop.olive-team.com/
For usage details, refer to the official Temporal documentation: Temporal documentation.
If developers require request-level diagnostics during workflow troubleshooting in authenticated deployments, a platform administrator can temporarily enable GLOBAL_DIAGNOSTICS_ENABLED for data-orchestrator in the shared service deployment values. See Chart Values Reference.
Resolution
Step 1: Delete Stuck Jobs
Delete the stuck Temporal jobs to allow them to restart and execute in the correct order:
kubectl delete job -n <namespace> temporal-schema-1
kubectl delete job -n <namespace> workflow-infra-onboard
If other Temporal-related jobs are stuck, delete them as well.
Step 2: Monitor Workload Startup Sequence
After deleting the stuck jobs, monitor the cluster to ensure workloads start in the correct order:
kubectl get pods -n <namespace> -w | grep temporal
Workload Startup Order
Temporal workloads must initialize in this specific sequence for successful deployment:
temporal-pg— CloudNativePG database deployment (must complete first)temporal-pg-poller-rw— Connection pooler providing read-write connections to the databasetemporal-schema-1— Database schema initialization and configurationworkflow-infra-onboard— Tenant onboarding for Temporal workflows- Temporal services — Core Temporal services (UI, server, worker, etc.)
temporal-caller— Final service layer for Temporal workflow invocation
Verify Startup Order
To verify the workloads are starting in the correct sequence, use:
kubectl get jobs -n <namespace> -o wide
Each job should show a completion status before the next one in the sequence begins.
Prevention
To minimize the risk of this race condition in future deployments:
- Ensure all cluster resources (CPU, memory) are adequate and available
- Verify CloudNativePG configuration is correct before deploying Temporal workloads
- Monitor deployment logs during platform initialization
- If workloads appear to hang during Temporal initialization, apply the resolution steps above early
Additional Support
If the issue persists after following these steps or if you encounter different Temporal-related errors, consult the Monitoring and Observability guide for detailed logging and diagnostic procedures.