Skip to main content

Multi-Replica, High Availability & Concurrency Troubleshooting

This guide addresses common issues encountered when deploying Open WebUI in multi-replica environments (e.g., Kubernetes, Docker Swarm) or when using multiple workers (UVICORN_WORKERS > 1) for increased concurrency.

If you are setting up a scaled deployment for the first time, start with the Scaling Open WebUI guide for a step-by-step walkthrough.

Core Requirements Checklist​

Before troubleshooting specific errors, ensure your deployment meets these absolute requirements for a multi-replica setup. Missing any of these will cause instability, login loops, or data loss.

  1. Shared Secret Key: WEBUI_SECRET_KEY MUST be identical on all replicas.
  2. External Database: You MUST use an external PostgreSQL database (see DATABASE_URL). SQLite is NOT supported for multiple instances.
  3. Redis for WebSockets: ENABLE_WEBSOCKET_SUPPORT=True and WEBSOCKET_MANAGER=redis with a valid WEBSOCKET_REDIS_URL are required.
  4. Shared Storage: A persistent volume (RWX / ReadWriteMany if possible, or ensuring all replicas map to the same underlying storage for data/) is critical for RAG (uploads/vectors) and generated images.
  5. External Vector Database (Required): The default ChromaDB uses a local SQLite-backed PersistentClient that is not safe for multi-worker or multi-replica deployments. SQLite connections are not fork-safe, and concurrent writes from multiple processes will crash workers instantly. You must use a dedicated external Vector DB (e.g., PGVector, MariaDB Vector, Milvus, Qdrant) via VECTOR_DB, or run ChromaDB as a separate HTTP server.
  6. Database Session Sharing (Optional): For PostgreSQL deployments with adequate resources, consider enabling DATABASE_ENABLE_SESSION_SHARING=True to improve performance under high concurrency.
  7. Thread Pool Ceiling: Set THREAD_POOL_SIZE=2000 (or higher). The default concurrency ceiling for blocking operations is only 40; at multi-user scale this is exhausted quickly and the app appears to freeze while CPU/RAM look fine. It is a ceiling, not a thread/CPU count: a high value is not a contention risk. Never lower it.
  8. Throttle Presence Writes: Set DATABASE_USER_ACTIVE_STATUS_UPDATE_INTERVAL=300 (300 to 500s). Unset (the default) means each user's last_active_at is written on essentially every request: a continuous flood of tiny UPDATE/COMMIT transactions that saturates the connection pool at scale for no functional benefit.

Common Issues​

1. Login Loops / 401 Unauthorized Errors​

Symptoms:

  • You log in successfully, but the next click logs you out.
  • You see "Unauthorized" or "401" errors in the browser console immediately after login.
  • "Error decrypting tokens" appears in logs.

Cause: Each replica is using a different WEBUI_SECRET_KEY. When Replica A issues a session token (JWT), Replica B rejects it because it cannot verify the signature with its own different key.

Solution: Set the WEBUI_SECRET_KEY environment variable to the same strong, random string on all backend replicas.

# Example in Kubernetes/Compose
env:
  - name: WEBUI_SECRET_KEY
    value: "your-super-secure-static-key-here"

2. WebSocket 403 Errors / Connection Failures​

Symptoms:

  • Chat stops responding or hangs.
  • Browser console shows WebSocket connection failed: 403 Forbidden or Connection closed.
  • Logs show engineio.server: https://your-domain.com is not an accepted origin.

Cause:

  • CORS: The load balancer or ingress origin does not match the allowed origins.
  • Missing Redis: WebSockets are defaulting to in-memory, so events on Replica A (e.g., LLM generation finish) are not broadcast to the user connected to Replica B.

Solution:

  1. Configure CORS: Ensure CORS_ALLOW_ORIGIN includes your public domain and http/https variations.

    If you see logs like engineio.base_server:_log_error_once:354 - https://yourdomain.com is not an accepted origin, you must update this variable. It accepts a semicolon-separated list of allowed origins.

    Example:

    CORS_ALLOW_ORIGIN="https://chat.yourdomain.com;http://chat.yourdomain.com;https://yourhostname;http://localhost:3000"

    Add all valid IPs, Domains, and Hostnames that users might use to access your Open WebUI.

  2. Enable Redis for WebSockets: Ensure these variables are set on all replicas:

    ENABLE_WEBSOCKET_SUPPORT=True
WEBSOCKET_MANAGER=redis
WEBSOCKET_REDIS_URL=redis://your-redis-host:6379/0

3. "Model Not Found" or Configuration Mismatch​

Symptoms:

  • You enable a model or change a setting in the Admin UI, but other users (or you, after a refresh) don't see the change.
  • Chats fail with "Model not found" intermittently.

Cause:

  • Configuration Sync: Replicas are not synced. Open WebUI uses Redis Pub/Sub to broadcast configuration changes (like toggling a model) to all other instances.
  • Missing Redis: If REDIS_URL is not set, configuration changes stay local to the instance where the change was made.

Solution: Set REDIS_URL to point to your shared Redis instance. This enables the Pub/Sub mechanism for real-time config syncing.

REDIS_URL=redis://your-redis-host:6379/0

For Ollama specifically, a chat sent to a model the replica has not seen no longer fails outright: the replica clears its model cache, re-reads the model list from the Ollama servers, and only reports the model as unknown if it is still missing. A model pulled or added while a replica was holding a stale list therefore becomes usable on the next request instead of after the cache expires.

4. Database Corruption / "Locked" Errors​

Symptoms:

  • Logs show database is locked or severe SQL errors.
  • Data saved on one instance disappears on another.
  • sqlalchemy.exc.TimeoutError: QueuePool limit of size N overflow M reached, connection timed out, timeout 30.00 on every request after a short warm-up, not just at peak load.
  • /api/config, /api/v1/chats/?page=1, OIDC callbacks all stall for 10 s to multiple minutes.
  • PRAGMA journal_mode=WAL logged as starting but never completing.
  • Problems appeared suddenly after the 0.8.x → 0.9.x upgrade without any other change in the deployment.

Cause: Using SQLite with multiple replicas, or with a single replica whose webui.db lives on a network filesystem. SQLite is a file-based database; its file locking does not work reliably over NFS / CIFS / CephFS / Azure Files / any network PVC (this is SQLite upstream's own position, not an Open WebUI policy). With the async backend introduced in 0.9.0, this turns from "slow and occasionally locked" into a hard failure mode because slow network fsyncs hold each pool connection hostage long enough to saturate the connection pool on every request.

For the full mechanism (fsync latency + async concurrency + pool saturation + WAL-on-mmap-on-NFS), see Performance → Disk I/O Latency.

Solution, in order of correctness:

  1. Migrate to PostgreSQL (strongly recommended, required for multi-replica):

    DATABASE_URL=postgresql+asyncpg://user:password@postgres-host:5432/openwebui

    For Kubernetes / Docker Swarm this is effectively mandatory. Postgres manages its own I/O against its own local storage, so the network-fsync pathology disappears entirely. See Scaling → Step 1 for the full migration steps.

  2. If you're on a single instance and can move the DB: put webui.db on a locally-attached SSD/NVMe (host bind mount, node-local volume, ephemeral disk), not on the same NFS/Ceph/EFS mount you use for uploads and RAG files. Your shared storage for /app/backend/data is fine; SQLite specifically is the problem.

  3. Do NOT just increase DATABASE_POOL_SIZE. A bigger pool doesn't fix slow fsync; it just schedules more concurrent slow fsyncs against the same slow storage and moves the breaking point by a few seconds. This is a symptom-whack, not a fix.

  4. Temporary damage-control only (for a deployment you're about to migrate):

    DATABASE_POOL_SIZE=1
DATABASE_SQLITE_PRAGMA_BUSY_TIMEOUT=30000

    Serializes to a single async connection, trading concurrency for stability. Not supported long-term. Also consider ENABLE_AUTOMATIONS=false if the background scheduler's periodic poll is the specific thing tipping you over the edge.

5. Uploaded Files or RAG Knowledge Inaccessible​

Symptoms:

  • You upload a file (for RAG) on one instance, but the model cannot find it later.
  • Generated images appear as broken links.

Cause: The /app/backend/data directory is not shared or is not consistent across replicas. If User A uploads a file to Replica 1, and the next request hits Replica 2, Replica 2 won't have the file physically on disk.

Solution:

  • Kubernetes: Use a PersistentVolumeClaim with ReadWriteMany (RWX) access mode if your storage provider supports it (e.g., NFS, CephFS, AWS EFS).
  • Docker Swarm/Compose: Mount a shared volume (e.g., NFS mount) to /app/backend/data on all containers.

6. Worker Crashes During Document Upload (ChromaDB + Multi-Worker)​

Symptoms:

  • Logs show the following sequence, all within the same second: 
    save_docs_to_vector_db:1619 - adding to collection file-id
    INFO:     Waiting for child process [pid]
    INFO:     Child process [pid] died
  • Workers die immediately during RAG document ingestion.
  • The crash is instant (not a timeout).

Cause: The default ChromaDB configuration uses a local PersistentClient backed by SQLite. When uvicorn forks multiple workers (UVICORN_WORKERS > 1), each worker process inherits a copy of the same SQLite database connection, all pointing at the same file on disk (data/vector_db/).

When two workers attempt to write to the collection simultaneously (e.g., during document upload), SQLite's file-level locking fails across forked processes. The result is either a database lock error or a segfault from corrupted internal state inherited across the fork() call, which kills the worker process instantly.

This is a well-known SQLite limitation: open database connections must not be carried across a fork().

Solution: You must stop using the default local ChromaDB with multiple workers. Pick one of these options:

OptionChangeTradeoff
Keep 1 workerSet UVICORN_WORKERS=1 (the default)Simplest, but limits concurrency
Use ChromaDB HTTP modeSet CHROMA_HTTP_HOST / CHROMA_HTTP_PORT to point to a separate Chroma serverEach worker connects via HTTP instead of SQLite, fully fork-safe
Switch vector DBSet VECTOR_DB to pgvector, mariadb-vector, milvus, qdrant, etc.These are client-server databases, inherently multi-process safe

Recommended fix: run ChromaDB as a separate server:

# Run chroma server separately
chroma run --host 0.0.0.0 --port 8000 --path /data/vector_db

# Then set these env vars for Open WebUI
CHROMA_HTTP_HOST=localhost
CHROMA_HTTP_PORT=8000
UVICORN_WORKERS=4

7. Slow Performance in Cloud vs. Local Kubernetes​

Symptoms:

  • Open WebUI performs well locally but experiences significant degradation or timeouts when deployed to cloud providers (AKS, EKS, GKE).
  • Performance drops sharply under concurrent load despite adequate resource allocation.

Cause: This is typically caused by infrastructure latency (Network Latency to the database or Disk I/O latency for SQLite) that is inherently higher in cloud environments compared to local NVMe/SSD storage and local networks.

Solution: Refer to the Cloud Infrastructure Latency section in the Performance Guide for a detailed breakdown of diagnosis and mitigation strategies.

If you need more tips for performance improvements, check out the full Optimization & Performance Guide.

8. Optimizing Database Performance​

For PostgreSQL deployments with adequate resources, consider these optimizations:

Database Session Sharing​

Enabling session sharing can improve performance under high concurrency:

DATABASE_ENABLE_SESSION_SHARING=true

See DATABASE_ENABLE_SESSION_SHARING for details.

Connection Pool Sizing​

If you experience QueuePool limit reached errors or connection timeouts under high concurrency, increase the pool size:

DATABASE_POOL_SIZE=15 (or higher)
DATABASE_POOL_MAX_OVERFLOW=20 (or higher)

Important: The combined total (DATABASE_POOL_SIZE + DATABASE_POOL_MAX_OVERFLOW) should remain well below your database's max_connections limit. PostgreSQL defaults to 100 max connections, so keep the combined total under 50-80 per Open WebUI instance to leave room for other clients and maintenance operations.

Pool Size Multiplies with Concurrency

Each Open WebUI process maintains its own independent connection pool. This applies to multiple replicas (Kubernetes pods, Docker Swarm replicas) and multiple Uvicorn workers within each replica.

The actual maximum number of database connections is:

Total connections = (DATABASE_POOL_SIZE + DATABASE_POOL_MAX_OVERFLOW) × Total processes

Where Total processes = Number of replicas × UVICORN_WORKERS per replica.

For example, with DATABASE_POOL_SIZE=15, DATABASE_POOL_MAX_OVERFLOW=20, 3 replicas, and 2 workers each, you could open up to 210 connections (35 × 6 processes).

See DATABASE_POOL_SIZE for details. For comprehensive database optimization including caching, session sharing, and content extraction tuning, see the Performance & RAM guide.

9. Function/Tool Dependency Installation Crashes​

Symptoms:

  • Workers crash with AssertionError on startup or when a function/tool is first loaded.
  • Logs show pip locking errors or multiple pip processes competing.

Cause: When a function or tool specifies requirements in its frontmatter, Open WebUI runs pip install at runtime. With multiple workers or replicas, each process attempts the installation independently, causing pip's internal lock to detect the conflict and crash.

Solution: Set ENABLE_PIP_INSTALL_FRONTMATTER_REQUIREMENTS=False to disable runtime pip installs entirely. Then pre-install all required packages at image build time:

FROM ghcr.io/open-webui/open-webui:main

RUN pip install --no-cache-dir python-docx requests beautifulsoup4

Runtime requirements installation is only appropriate for single-worker development or homelab environments.

For more details, see the External Packages section of the Tools documentation.

Deployment Best Practices​

Updates and Migrations​

Critical: Avoid Concurrent Migrations

Always ensure only one process is running database migrations when upgrading Open WebUI versions.

Critical: Never Run Two Versions Against One Database

Old and new Open WebUI instances must never serve traffic against the same database at the same time. Rolling updates, canary rollouts and any "update one replica, scale the rest once it is healthy" strategy are unsupported across a release that changes the schema.

Database migrations run automatically on startup. If multiple replicas (or multiple workers within a single container) start simultaneously with a new version, they may try to run migrations concurrently, potentially leading to race conditions or database schema corruption.

Version skew is a separate and harder failure. Migrations move the schema forward for the whole database at once, and the still-running old instances keep issuing queries written against the old shape. Whether that survives depends entirely on what the release changed. v0.11.0 only adds columns and indexes, so an old instance left running through the upgrade keeps working. A release that renames or drops something does not: v0.10.0 renamed the config table to config_old and replaced it with a per-key table, and any older instance still querying config fails from the moment the migration lands. Releases are not labelled by which kind they are.

Treat every upgrade as the second kind. Stop all replicas and workers, let one instance migrate, then bring the fleet back up on the new version. That costs a short full outage, and there is no general zero-downtime path across a schema change.

Safe Update Procedure:

There are two ways to safely handle migrations in a multi-replica environment:

  1. Identify one pod/replica as the "master" for migrations.
  2. Set ENABLE_DB_MIGRATIONS=True (default) on the master pod.
  3. Set ENABLE_DB_MIGRATIONS=False on all other pods.
  4. When updating, the master pod will handle the database schema update while other pods skip the migration step.

ENABLE_DB_MIGRATIONS only decides which instance migrates. It does not make version skew safe. Roll the new image out to every pod at once, not one at a time. If your platform cannot replace all pods simultaneously, use Option 2 instead.

Option 2: Scale Down During Update​

  1. Scale Down: Set replicas to 1 (and ensure UVICORN_WORKERS=1).
  2. Update Image: Update the image or version.
  3. Wait for Health Check: Wait for the single instance to start fully and complete migrations.
  4. Scale Up: Increase replicas back to your desired count.

Session Affinity (Sticky Sessions)​

While Open WebUI is designed to be stateless with proper Redis configuration, enabling Session Affinity (Sticky Sessions) at your Load Balancer / Ingress level can improve performance and reduce occasional jitter in WebSocket connections.

  • Nginx Ingress: nginx.ingress.kubernetes.io/affinity: "cookie"
  • AWS ALB: Enable Target Group Stickiness.

Compress at the Load Balancer, Not in the App​

By default each Open WebUI worker compresses its own HTTP responses, which profiling shows costs roughly 3–4% CPU per worker — multiplied across all replicas. In multi-replica deployments there is always a load balancer or ingress in front, so enable compression there and disable it in the app with ENABLE_COMPRESSION_MIDDLEWARE=false. WebSocket and SSE streaming traffic is never compressed by this middleware, so chat streaming is unaffected. Pair this with proxy-side caching of the content-hashed static bundles under /_app/immutable/ so they are served from the proxy cache instead of the workers. See ENABLE_COMPRESSION_MIDDLEWARE and Scaling → Offload HTTP Compression.

Use the Faster JSON Encoder​

ENABLE_ORJSON=True

Every Socket.IO event in a multi-replica deployment is published through Redis, and each one is encoded on the way out and decoded on the way in. In clustered deployments that encoding was the single largest cost measured on the workers handling live updates, which is why this setting exists. It switches the whole application from Python's standard-library json to orjson, a Rust implementation several times faster. orjson already ships as a pinned dependency, so there is nothing to install. Available from v0.11.0.

What it covers:

PathWhy it matters at scale
Socket.IO payloads, including the ones published over Redis, plus Redis-backed session and collaborative-document stateThe reason to turn this on. Every live update crossing replicas is encoded and decoded here
Upstream provider responses (OpenAI-compatible and Ollama), including per-chunk parsing of streamed completionsRuns once per arriving token batch, so it scales with streaming volume rather than request count
Incoming HTTP request bodies and outgoing JSONResponse bodiesThe broadest surface, but individually the cheapest

Why it is opt-in. orjson is stricter than the standard library, so the default keeps behaviour byte-for-byte identical to earlier releases. In practice the strictness is handled for you: dictionary keys that are not strings, integers beyond 64 bits, and NaN or Infinity literals are all rejected by orjson and fall back to the standard-library path automatically. Enabling this cannot turn a payload that worked into an error.

Two differences do survive the fallback, neither of which affects a normal deployment:

  • NaN and Infinity floats in a response body serialize as null instead of raising. Starlette's own encoder is configured with allow_nan=False, so a response carrying those values previously failed outright and now goes out with nulls. If you have a custom tool or pipe that can emit them, this is a behaviour change to be aware of rather than a regression.
  • Encoded output carries raw UTF-8 rather than \uXXXX escapes. Both are valid JSON and every client parses them identically, but the exact bytes differ, which matters only if something downstream checksums or string-compares encoded JSON.

When not to bother. On a single-worker, single-user instance the saving is real but invisible next to model latency. The setting earns its keep once Socket.IO traffic is crossing Redis between workers or replicas.

See ENABLE_ORJSON for the variable itself and Scaling → Switch the JSON Encoder to orjson for where it fits in a scaled rollout.

This content is for informational purposes only and does not constitute a warranty, guarantee, or contractual commitment. Open WebUI is provided "as is." See your license for applicable terms.