Multi-Replica, High Availability & Concurrency Troubleshooting
This guide covers common issues in multi-replica deployments (Docker Compose, Docker Swarm, Kubernetes). Scale with replicas of one worker each; UVICORN_WORKERS above 1 is not recommended and hits the same issues.
To set up a scaled deployment, follow Scaling Open WebUI. For single-instance tuning, see Optimization, Performance & RAM Usage.
Core Requirements Checklist
Before troubleshooting specific errors, make sure your deployment meets these requirements. Missing any of them causes instability, login loops or data loss.
- Shared secret key:
WEBUI_SECRET_KEYis identical on all replicas. - PostgreSQL:
DATABASE_URLpoints to PostgreSQL. SQLite is not supported for multiple instances. See Step 1. - Redis:
REDIS_URLandWEBSOCKET_MANAGER=redisare set on every replica. See Step 2. - External vector database:
VECTOR_DBpoints to a client-server database, or ChromaDB runs as a separate HTTP server. The embedded ChromaDB crashes under multi-process writes. See Step 4. - Shared file storage: every replica reaches the same uploaded files through object storage or a shared filesystem. See Step 5.
- One migrating process: only one process runs database migrations on upgrade. See Updates and Migrations.
For recommended tuning on top of these, see Minimum Environment Variables for Scaled Deployments.
Common Issues
1. Login Loops / 401 Unauthorized Errors
Symptoms:
- You log in successfully, but the next click logs you out.
- The browser console shows "Unauthorized" or "401" errors right after login.
- "Error decrypting tokens" appears in logs.
Cause:
The replicas use different WEBUI_SECRET_KEY values. Replica B rejects a token signed by Replica A. Without WEBUI_SECRET_KEY, each container generates its own key on first start.
Solution:
Set WEBUI_SECRET_KEY to the same strong, random string on all replicas. If you set OAUTH_SESSION_TOKEN_ENCRYPTION_KEY separately, it must match everywhere too.
# Example in Kubernetes/Compose
env:
- name: WEBUI_SECRET_KEY
value: "your-super-secure-static-key-here"2. WebSocket 403 Errors / Connection Failures
Symptoms:
- Chat stops responding or hangs.
- The browser console shows
WebSocket connection failed: 403 ForbiddenorConnection closed. - Logs show
https://your-domain.com is not an accepted origin.
Cause:
- CORS: the origin users reach Open WebUI through is missing from the allowed origins.
- Missing Redis: without
WEBSOCKET_MANAGER=redis, live updates stay on the replica that produced them, so a user connected to another replica never sees the response.
Solution:
-
Configure CORS: add every public origin to
CORS_ALLOW_ORIGIN, a semicolon-separated list:CORS_ALLOW_ORIGIN="https://chat.yourdomain.com;http://chat.yourdomain.com;https://yourhostname;http://localhost:3000"Include every domain, hostname and IP users might use.
-
Enable Redis for WebSockets on all replicas:
REDIS_URL=redis://your-redis-host:6379/0 WEBSOCKET_MANAGER=redis
3. "Model Not Found" or Configuration Mismatch
Symptoms:
- A model enabled in the Admin UI shows up for some requests and not others.
- Chats fail with "Model not found" intermittently.
Cause:
Settings live in the shared database, and every replica reads them from there, so settings changes reach all replicas without Redis. The model list is held per replica unless WEBSOCKET_MANAGER=redis shares it. Without it, each replica only knows the models it has loaded itself.
Solution:
Set REDIS_URL and WEBSOCKET_MANAGER=redis on every replica (see Step 2).
For Ollama, a chat sent to a model the replica does not know makes it re-read the model list from the Ollama servers first. A model pulled while a replica held a stale list is usable on the next request.
4. Database Corruption / "Locked" Errors
Symptoms:
- Logs show
database is lockedor severe SQL errors. - Data saved on one instance disappears on another.
sqlalchemy.exc.TimeoutError: QueuePool limit of size N overflow M reached, connection timed out, timeout 30.00on nearly every request after a short warm-up, not just at peak load./api/config, chat list and OIDC callbacks stall for 10 seconds to several minutes.PRAGMA journal_mode=WALis logged as starting but never completes.
Cause:
SQLite with multiple replicas, or with a single replica whose webui.db lives on a network filesystem (NFS, CIFS, CephFS, Azure Files, a network PVC). SQLite's file locking is unreliable over network filesystems, and slow network fsyncs hold each pool connection long enough to exhaust the pool. For the mechanism, see Disk I/O Latency.
Solution, in order of preference:
- Migrate to PostgreSQL (required for multiple replicas). See Step 1.
- Single instance: move
webui.dbto a locally attached SSD/NVMe (host bind mount, node-local volume). Uploads and RAG files can stay on the shared storage. - Stopgap while you migrate: fewer connections and a longer lock wait reduce lock contention.
DATABASE_POOL_SIZE=1 DATABASE_SQLITE_PRAGMA_BUSY_TIMEOUT=30000
A larger DATABASE_POOL_SIZE makes this worse: it runs more slow fsyncs against the same storage at once.
5. Uploaded Files or RAG Knowledge Inaccessible
Symptoms:
- You upload a file on one instance, but the model cannot find it later.
- Images in a chat intermittently show an Image unavailable placeholder, depending on which replica served the request.
Cause: Uploaded files are not shared across replicas. A file uploaded to Replica 1 is missing on Replica 2.
Solution:
Use object storage or a shared filesystem on every replica, as described in Step 5. On Kubernetes, a shared volume needs ReadWriteMany (RWX) access.
6. Worker Crashes During Document Upload (ChromaDB + Multi-Worker)
Symptoms:
- Logs show this sequence within the same second:
save_docs_to_vector_db:1619 - adding to collection file-id
INFO: Waiting for child process [pid]
INFO: Child process [pid] died - Workers die instantly during RAG document ingestion (not a timeout).
Cause:
The default ChromaDB runs embedded in each Open WebUI process and stores its data in local SQLite files (data/vector_db/). Only one process may write to them. With several replicas or workers, concurrent writes during uploads kill the process.
Solution:
| Option | Change | Tradeoff |
|---|---|---|
| Keep 1 worker | UVICORN_WORKERS=1 (the default) and scale with replicas | Still needs one of the options below once there is more than one replica |
| Use ChromaDB HTTP mode | Point CHROMA_HTTP_HOST / CHROMA_HTTP_PORT to a separate Chroma server | Every process connects over HTTP to one server |
| Switch vector DB | Set VECTOR_DB to pgvector, mariadb-vector, milvus, qdrant or another client-server database | Built for many clients |
To keep ChromaDB, run it as its own server:
# Run the Chroma server separately
chroma run --host 0.0.0.0 --port 8000 --path /data/vector_db
# Then point Open WebUI at it
CHROMA_HTTP_HOST=localhost
CHROMA_HTTP_PORT=8000Run the Chroma server at the same version as the chromadb client bundled with Open WebUI. Each release pins it in pyproject.toml; v0.13.4 pins chromadb==1.5.9. If uploads start failing with errors raised from the Chroma client after the switch, compare the two versions first.
7. Slow Performance in Cloud vs. Local Kubernetes
Symptoms:
- Open WebUI performs well locally but degrades or times out on AKS, EKS or GKE.
- Performance drops sharply under concurrent load despite adequate resources.
Cause: Usually network latency to the database or disk I/O latency for SQLite, both higher in the cloud than on local NVMe and local networks.
Solution: See Cloud Infrastructure Latency.
8. Optimizing Database Performance
Symptoms:
QueuePool limit of size N overflow M reachederrors on PostgreSQL under load.- PostgreSQL rejects connections with
too many clients already.
Cause:
The pool of one process is too small for its load, or the pools of all processes together exceed PostgreSQL's max_connections. Each replica has its own pool.
Solution:
Size DATABASE_POOL_SIZE and DATABASE_POOL_MAX_OVERFLOW per process and check the total against max_connections, as described in Step 1. If blocking work queues instead (the app freezes while CPU and RAM look fine), raise THREAD_POOL_SIZE.
9. Function/Tool Dependency Installation Crashes
Symptoms:
- Workers crash with
AssertionErroron startup or when a function or tool first loads. - Logs show pip locking errors or several pip processes competing.
Cause:
When a function or tool lists requirements in its frontmatter, Open WebUI runs pip install at runtime. With UVICORN_WORKERS above 1, every worker runs the installation into the same environment at once.
Solution:
Keep UVICORN_WORKERS=1, set ENABLE_PIP_INSTALL_FRONTMATTER_REQUIREMENTS=False and install the packages into the image at build time:
FROM ghcr.io/open-webui/open-webui:main
RUN pip install --no-cache-dir python-docx requests beautifulsoup4Runtime installation suits only development or homelab setups. See External Packages.
Deployment Best Practices
Updates and Migrations
Only one process may run database migrations when upgrading Open WebUI.
Old and new Open WebUI instances must never serve traffic against the same database at the same time. Rolling updates, canary rollouts and "update one replica, scale the rest once it is healthy" strategies are unsupported across a release that changes the schema.
Migrations run automatically on startup. If several replicas start a new version at once, they run migrations concurrently and can corrupt the schema.
Stop all replicas, let one instance migrate, then bring the fleet back up on the new version. That costs a short full outage; there is no general zero-downtime path across a schema change.
Back up the database first. Putting the old image back does not undo a migration: the schema and any rewritten rows stay as the migration left them. Restoring a backup taken before the upgrade is the way back. See Backup & Restore.
Safe Update Procedure:
Option 1: Designate a Master Migration Pod (Recommended)
- Pick one pod as the migration "master".
- Keep
ENABLE_DB_MIGRATIONS=True(default) on that pod. - Set
ENABLE_DB_MIGRATIONS=Falseon all other pods. - On update, the master pod migrates the schema while the others skip it.
ENABLE_DB_MIGRATIONS only decides which instance migrates. Roll the new image out to every pod at once. If your platform cannot replace all pods simultaneously, use Option 2.
Option 2: Scale Down During Update
- Scale down: set replicas to
1. - Update the image or version.
- Wait until the single instance has started and finished migrating.
- Scale up to your desired count.
Session Affinity (Sticky Sessions)
With Redis configured and WebSockets on (the default), any replica can serve any request, so sticky sessions are not needed. On an unstable network, a blip or reconnect can still disturb how streamed responses or tool calls reach the browser; if you see that, turning them on is worth a try. They are required when ENABLE_WEBSOCKET_SUPPORT=false, because long-polling needs every request of a connection to reach the same replica.
- Nginx Ingress:
nginx.ingress.kubernetes.io/affinity: "cookie" - AWS ALB: enable Target Group Stickiness.
Use the Faster JSON Encoder
ENABLE_ORJSON=TrueIn a multi-replica deployment every Socket.IO event is published through Redis, encoded on the way out and decoded on the way in, which makes JSON encoding one of the largest costs on the replicas handling live updates. ENABLE_ORJSON (default True) uses orjson, a much faster Rust implementation, across the whole app. It ships with Open WebUI, so there is nothing to install. On a small single instance the difference is mostly felt when saving and opening very long chats.
Payloads orjson rejects (non-string dictionary keys, integers beyond 64 bits) fall back to the standard library automatically, so a working payload never turns into an error. Two differences remain:
NaNandInfinityfloats in a response body becomenull. With the standard library, such a response fails. Relevant only to custom tools or pipes that emit these values.- Encoded output carries raw UTF-8 instead of
\uXXXXescapes. Both parse identically; this matters only to something that checksums encoded JSON. Line separator characters stay escaped so streamed replies are not split. Switching the setting on or off needs no migration.
Related Documentation
- Scaling Open WebUI: step-by-step guide from a single instance to production
- Environment Variable Configuration
- Optimization, Performance & RAM Usage
- Redis WebSocket Support: detailed Redis setup tutorial
- Troubleshooting Connection Errors
- RAG Troubleshooting: document upload and embedding issues
- Logging Open WebUI