Real-Time Systems · 10 min read
WebSocket at Scale: What Changes When You Have 100,000 Concurrent Connections
Maintaining a WebSocket connection is architecturally different from serving HTTP requests. At scale, that difference determines whether your real-time system holds up or becomes your most expensive operational problem.
The Illusion of Simplicity
A WebSocket connection is easy to open. A WebSocket server that handles 100,000 concurrent connections reliably — sustaining low latency, recovering gracefully from failures, and scaling horizontally without coordination overhead — is a different engineering problem.
Most real-time systems are built by teams that are confident about the first and unprepared for the second. The production incident comes later, usually under load, and the root cause is always something that was obvious in retrospect.
This is a practical accounting of what changes when WebSocket traffic reaches real scale.
Connection State Is the Core Architectural Constraint
An HTTP request is stateless. A server can handle any request from any client without any prior knowledge of that client. This is why HTTP services scale horizontally with almost no coordination overhead — route traffic to any instance and it works.
A WebSocket connection is stateful. The server that accepted the connection maintains it. The client's messages go to that server, and messages destined for that client must reach that server. When you add horizontal scaling, you introduce a problem: if the client is connected to Instance A and a message must be delivered to that client, how does Instance B or Instance C — which handled the event that triggered the message — know to send it to Instance A?
The standard solution is a publish-subscribe message bus shared across instances. Redis Pub/Sub is the most common choice for mid-scale systems. When any instance needs to send a message to a specific client, it publishes to a channel keyed to the client ID. The instance holding that connection is subscribed to that channel and delivers the message.
This works, and it eliminates the cross-instance coordination problem. But it introduces a dependency: the Redis cluster is now in the critical path of every message delivery. Its availability and latency directly affect your real-time guarantees.
Load Balancing WebSocket Traffic Is Not Like Load Balancing HTTP
Standard round-robin load balancing does not work for WebSocket connections. A client that opens a WebSocket connection must stay connected to the same backend instance for the duration of the connection. HTTP-layer load balancers that distribute requests across instances will break the connection on reconnect unless they use sticky sessions.
Sticky sessions — binding a client to a specific backend instance — solve the routing problem but create an operational one. When an instance is taken down for deployment or fails unexpectedly, all clients connected to it lose their connection simultaneously. This produces a thundering herd: every affected client attempts to reconnect at the same moment.
Without reconnection backoff, this spike in reconnection traffic arrives at exactly the wrong time — when the system is already dealing with a disruption. Exponential backoff with jitter on the client side is non-negotiable at scale. It converts the thundering herd from a spike that can overwhelm the system into a distributed reconnection wave that the system can absorb.
Memory and File Descriptor Limits Define Your Per-Instance Ceiling
Each WebSocket connection consumes file descriptors and memory on the server. Linux defaults set the maximum open file descriptors per process at 1,024. At that limit, a single server process can handle 1,024 connections — less than nothing at real scale.
Tuning the OS and the application runtime to support high connection counts is the first prerequisite for running a WebSocket server at scale:
ulimit -nand/etc/security/limits.confneed to be configured to allow the number of file descriptors you need per process- TCP socket parameters (
net.core.somaxconn,net.ipv4.tcp_max_syn_backlog) need to match the expected connection rate - Application-level event loops (Node.js, Netty, Go's net/http) handle concurrent connections differently — connection count per instance is bounded by the runtime's concurrency model as well as OS limits
The practical ceiling for a single instance depends on these configurations, the message rate per connection, and what the server does for each message. A chat application where most connections are idle most of the time can handle more connections per instance than a trading platform where every connection receives high-frequency price updates.
Message Fanout Is Where Throughput Breaks Down
Sending one message to one connected client is trivial. Sending one message to 10,000 connected clients — a broadcast — is an operation that can degrade your entire real-time infrastructure if not designed for.
The naive implementation iterates over all recipients and sends each message synchronously. At 10,000 recipients, this is a tight loop that occupies the server for a measurable period. At 100,000 recipients, it is a blocking operation that delays every other message in the queue while it runs.
The correct approach separates the fanout from the delivery. When a message needs to reach a large audience, it is published to a Kafka topic or a Redis stream. Dedicated delivery workers consume from the queue and write to connections. Delivery is now asynchronous and can be parallelised across workers. The throughput of the fanout is limited by the delivery worker fleet, not by the main connection handler.
This architecture also gives you a natural buffer during load spikes. If delivery falls behind, messages queue in Kafka rather than blocking the connection handler. The system degrades gracefully — delivery slows, but connections are not dropped and the handler remains responsive.
Presence and Session Management Add Hidden State
Many real-time features require presence: knowing which users are currently connected. This sounds simple. At scale, it is not.
Presence state is high-churn. Connections open and close constantly. Each connect and disconnect event must update the presence store. If 10,000 users reconnect after an instance restart, the presence store receives 20,000 write operations in a short window.
The presence store must also be consistent across instances. If a user is connected to Instance A, Instance B needs to know that user is online when another user queries their status. This means presence state cannot live in instance memory — it must be in a shared store, typically Redis with TTL-based expiry.
TTL-based expiry is the recommended pattern for presence: each connected client sends a heartbeat at a regular interval (every 30 seconds, for example). The server writes or refreshes a key in Redis with a TTL slightly longer than the heartbeat interval. If the client disconnects without sending a disconnect message — network drop, device shutdown, process crash — the key expires naturally and the user is marked offline. This handles the ungraceful disconnection case without requiring the server to detect every abnormal termination.
Observability for Real-Time Systems Is Different
Standard HTTP observability metrics — request rate, error rate, latency — do not adequately describe the health of a WebSocket system.
The metrics that matter for real-time systems:
Connection counts per instance, with alerting thresholds configured to detect both under-utilization (instance has no connections — is it receiving traffic?) and approaching limits (instance is near its connection ceiling).
Message delivery latency: the time from when a message is generated to when it is written to the client connection. This is the metric that reflects the user experience, and it is often not measured at all.
Message queue depth in the fanout queue. A growing queue depth means delivery is falling behind production. This is a leading indicator of latency degradation.
Reconnection rate: a spike in reconnections indicates either a backend disruption or a client-side connectivity problem. Tracking reconnection rate separately from new connection rate makes these distinguishable.
Real-time systems at scale are primarily an infrastructure and architecture problem, not an application logic problem. The WebSocket protocol is the easy part. Connection distribution, state management, fanout throughput, and graceful failure handling are where the engineering work actually is — and they need to be designed before the load arrives, not after.