Skip to main content

Multi-Replica, High Availability & Concurrency Troubleshooting

This guide covers common issues in multi-replica deployments (Docker Compose, Docker Swarm, Kubernetes). Scale with replicas of one worker each; UVICORN_WORKERS above 1 is not recommended and hits the same issues.

To set up a scaled deployment, follow Scaling Open WebUI. For single-instance tuning, see Optimization, Performance & RAM Usage.

Core Requirements Checklist​

Before troubleshooting specific errors, make sure your deployment meets these requirements. Missing any of them causes instability, login loops or data loss.

  1. Shared secret key: WEBUI_SECRET_KEY is identical on all replicas.
  2. PostgreSQL: DATABASE_URL points to PostgreSQL. SQLite is not supported for multiple instances. See Step 1.
  3. Redis: REDIS_URL and WEBSOCKET_MANAGER=redis are set on every replica. See Step 2.
  4. External vector database: VECTOR_DB points to a client-server database, or ChromaDB runs as a separate HTTP server. The embedded ChromaDB crashes under multi-process writes. See Step 4.
  5. Shared file storage: every replica reaches the same uploaded files through object storage or a shared filesystem. See Step 5.
  6. One migrating process: only one process runs database migrations on upgrade. See Updates and Migrations.

For recommended tuning on top of these, see Minimum Environment Variables for Scaled Deployments.


Common Issues​

1. Login Loops / 401 Unauthorized Errors​

Symptoms:

  • You log in successfully, but the next click logs you out.
  • The browser console shows "Unauthorized" or "401" errors right after login.
  • "Error decrypting tokens" appears in logs.

Cause: The replicas use different WEBUI_SECRET_KEY values. Replica B rejects a token signed by Replica A. Without WEBUI_SECRET_KEY, each container generates its own key on first start.

Solution: Set WEBUI_SECRET_KEY to the same strong, random string on all replicas. If you set OAUTH_SESSION_TOKEN_ENCRYPTION_KEY separately, it must match everywhere too.

# Example in Kubernetes/Compose
env:
  - name: WEBUI_SECRET_KEY
    value: "your-super-secure-static-key-here"

2. WebSocket 403 Errors / Connection Failures​

Symptoms:

  • Chat stops responding or hangs.
  • The browser console shows WebSocket connection failed: 403 Forbidden or Connection closed.
  • Logs show https://your-domain.com is not an accepted origin.

Cause:

  • CORS: the origin users reach Open WebUI through is missing from the allowed origins.
  • Missing Redis: without WEBSOCKET_MANAGER=redis, live updates stay on the replica that produced them, so a user connected to another replica never sees the response.

Solution:

  1. Configure CORS: add every public origin to CORS_ALLOW_ORIGIN, a semicolon-separated list:

    CORS_ALLOW_ORIGIN="https://chat.yourdomain.com;http://chat.yourdomain.com;https://yourhostname;http://localhost:3000"

    Include every domain, hostname and IP users might use.

  2. Enable Redis for WebSockets on all replicas:

    REDIS_URL=redis://your-redis-host:6379/0
    WEBSOCKET_MANAGER=redis

3. "Model Not Found" or Configuration Mismatch​

Symptoms:

  • A model enabled in the Admin UI shows up for some requests and not others.
  • Chats fail with "Model not found" intermittently.

Cause: Settings live in the shared database, and every replica reads them from there, so settings changes reach all replicas without Redis. The model list is held per replica unless WEBSOCKET_MANAGER=redis shares it. Without it, each replica only knows the models it has loaded itself.

Solution: Set REDIS_URL and WEBSOCKET_MANAGER=redis on every replica (see Step 2).

For Ollama, a chat sent to a model the replica does not know makes it re-read the model list from the Ollama servers first. A model pulled while a replica held a stale list is usable on the next request.

4. Database Corruption / "Locked" Errors​

Symptoms:

  • Logs show database is locked or severe SQL errors.
  • Data saved on one instance disappears on another.
  • sqlalchemy.exc.TimeoutError: QueuePool limit of size N overflow M reached, connection timed out, timeout 30.00 on nearly every request after a short warm-up, not just at peak load.
  • /api/config, chat list and OIDC callbacks stall for 10 seconds to several minutes.
  • PRAGMA journal_mode=WAL is logged as starting but never completes.

Cause: SQLite with multiple replicas, or with a single replica whose webui.db lives on a network filesystem (NFS, CIFS, CephFS, Azure Files, a network PVC). SQLite's file locking is unreliable over network filesystems, and slow network fsyncs hold each pool connection long enough to exhaust the pool. For the mechanism, see Disk I/O Latency.

Solution, in order of preference:

  1. Migrate to PostgreSQL (required for multiple replicas). See Step 1.
  2. Single instance: move webui.db to a locally attached SSD/NVMe (host bind mount, node-local volume). Uploads and RAG files can stay on the shared storage.
  3. Stopgap while you migrate: fewer connections and a longer lock wait reduce lock contention.
    DATABASE_POOL_SIZE=1
    DATABASE_SQLITE_PRAGMA_BUSY_TIMEOUT=30000

A larger DATABASE_POOL_SIZE makes this worse: it runs more slow fsyncs against the same storage at once.

5. Uploaded Files or RAG Knowledge Inaccessible​

Symptoms:

  • You upload a file on one instance, but the model cannot find it later.
  • Images in a chat intermittently show an Image unavailable placeholder, depending on which replica served the request.

Cause: Uploaded files are not shared across replicas. A file uploaded to Replica 1 is missing on Replica 2.

Solution: Use object storage or a shared filesystem on every replica, as described in Step 5. On Kubernetes, a shared volume needs ReadWriteMany (RWX) access.

6. Worker Crashes During Document Upload (ChromaDB + Multi-Worker)​

Symptoms:

  • Logs show this sequence within the same second:
    save_docs_to_vector_db:1619 - adding to collection file-id
    INFO: Waiting for child process [pid]
    INFO: Child process [pid] died
  • Workers die instantly during RAG document ingestion (not a timeout).

Cause: The default ChromaDB runs embedded in each Open WebUI process and stores its data in local SQLite files (data/vector_db/). Only one process may write to them. With several replicas or workers, concurrent writes during uploads kill the process.

Solution:

OptionChangeTradeoff
Keep 1 workerUVICORN_WORKERS=1 (the default) and scale with replicasStill needs one of the options below once there is more than one replica
Use ChromaDB HTTP modePoint CHROMA_HTTP_HOST / CHROMA_HTTP_PORT to a separate Chroma serverEvery process connects over HTTP to one server
Switch vector DBSet VECTOR_DB to pgvector, mariadb-vector, milvus, qdrant or another client-server databaseBuilt for many clients

To keep ChromaDB, run it as its own server:

# Run the Chroma server separately
chroma run --host 0.0.0.0 --port 8000 --path /data/vector_db

# Then point Open WebUI at it
CHROMA_HTTP_HOST=localhost
CHROMA_HTTP_PORT=8000

Run the Chroma server at the same version as the chromadb client bundled with Open WebUI. Each release pins it in pyproject.toml; v0.13.4 pins chromadb==1.5.9. If uploads start failing with errors raised from the Chroma client after the switch, compare the two versions first.

7. Slow Performance in Cloud vs. Local Kubernetes​

Symptoms:

  • Open WebUI performs well locally but degrades or times out on AKS, EKS or GKE.
  • Performance drops sharply under concurrent load despite adequate resources.

Cause: Usually network latency to the database or disk I/O latency for SQLite, both higher in the cloud than on local NVMe and local networks.

Solution: See Cloud Infrastructure Latency.

8. Optimizing Database Performance​

Symptoms:

  • QueuePool limit of size N overflow M reached errors on PostgreSQL under load.
  • PostgreSQL rejects connections with too many clients already.

Cause: The pool of one process is too small for its load, or the pools of all processes together exceed PostgreSQL's max_connections. Each replica has its own pool.

Solution: Size DATABASE_POOL_SIZE and DATABASE_POOL_MAX_OVERFLOW per process and check the total against max_connections, as described in Step 1. If blocking work queues instead (the app freezes while CPU and RAM look fine), raise THREAD_POOL_SIZE.

9. Function/Tool Dependency Installation Crashes​

Symptoms:

  • Workers crash with AssertionError on startup or when a function or tool first loads.
  • Logs show pip locking errors or several pip processes competing.

Cause: When a function or tool lists requirements in its frontmatter, Open WebUI runs pip install at runtime. With UVICORN_WORKERS above 1, every worker runs the installation into the same environment at once.

Solution: Keep UVICORN_WORKERS=1, set ENABLE_PIP_INSTALL_FRONTMATTER_REQUIREMENTS=False and install the packages into the image at build time:

FROM ghcr.io/open-webui/open-webui:main

RUN pip install --no-cache-dir python-docx requests beautifulsoup4

Runtime installation suits only development or homelab setups. See External Packages.


Deployment Best Practices​

Updates and Migrations​

Critical: Avoid Concurrent Migrations

Only one process may run database migrations when upgrading Open WebUI.

Critical: Never Run Two Versions Against One Database

Old and new Open WebUI instances must never serve traffic against the same database at the same time. Rolling updates, canary rollouts and "update one replica, scale the rest once it is healthy" strategies are unsupported across a release that changes the schema.

Migrations run automatically on startup. If several replicas start a new version at once, they run migrations concurrently and can corrupt the schema.

Stop all replicas, let one instance migrate, then bring the fleet back up on the new version. That costs a short full outage; there is no general zero-downtime path across a schema change.

Back up the database first. Putting the old image back does not undo a migration: the schema and any rewritten rows stay as the migration left them. Restoring a backup taken before the upgrade is the way back. See Backup & Restore.

Safe Update Procedure:

  1. Pick one pod as the migration "master".
  2. Keep ENABLE_DB_MIGRATIONS=True (default) on that pod.
  3. Set ENABLE_DB_MIGRATIONS=False on all other pods.
  4. On update, the master pod migrates the schema while the others skip it.

ENABLE_DB_MIGRATIONS only decides which instance migrates. Roll the new image out to every pod at once. If your platform cannot replace all pods simultaneously, use Option 2.

Option 2: Scale Down During Update​

  1. Scale down: set replicas to 1.
  2. Update the image or version.
  3. Wait until the single instance has started and finished migrating.
  4. Scale up to your desired count.

Session Affinity (Sticky Sessions)​

With Redis configured and WebSockets on (the default), any replica can serve any request, so sticky sessions are not needed. On an unstable network, a blip or reconnect can still disturb how streamed responses or tool calls reach the browser; if you see that, turning them on is worth a try. They are required when ENABLE_WEBSOCKET_SUPPORT=false, because long-polling needs every request of a connection to reach the same replica.

  • Nginx Ingress: nginx.ingress.kubernetes.io/affinity: "cookie"
  • AWS ALB: enable Target Group Stickiness.

Use the Faster JSON Encoder​

ENABLE_ORJSON=True

In a multi-replica deployment every Socket.IO event is published through Redis, encoded on the way out and decoded on the way in, which makes JSON encoding one of the largest costs on the replicas handling live updates. ENABLE_ORJSON (default True) uses orjson, a much faster Rust implementation, across the whole app. It ships with Open WebUI, so there is nothing to install. On a small single instance the difference is mostly felt when saving and opening very long chats.

Payloads orjson rejects (non-string dictionary keys, integers beyond 64 bits) fall back to the standard library automatically, so a working payload never turns into an error. Two differences remain:

  • NaN and Infinity floats in a response body become null. With the standard library, such a response fails. Relevant only to custom tools or pipes that emit these values.
  • Encoded output carries raw UTF-8 instead of \uXXXX escapes. Both parse identically; this matters only to something that checksums encoded JSON. Line separator characters stay escaped so streamed replies are not split. Switching the setting on or off needs no migration.

This content is for informational purposes only and does not constitute a warranty, guarantee, or contractual commitment. Open WebUI is provided "as is." See your license for applicable terms.