In our previous post we mentioned that DropReady does not use WebSockets. Several people asked why. This post is the full, technical answer.
Let's start with an honest overview of both technologies, because WebSocket is a great technology - just not for this use case.
How WebSocket works and when it makes sense
WebSocket is a protocol that establishes a persistent, bidirectional connection between the browser and the server. Instead of the classic HTTP cycle (request, response, connection closed), WebSocket keeps an open channel for the entire duration of the session.
This is absolutely the right choice when you need real-time communication from server to client: live chat, push notifications, multiplayer games, live trading, collaborative editing. Anywhere the server needs to actively "push" data to the client without waiting for a request.
The problem starts when you begin counting resources at scale.
Why WebSocket is the wrong choice for a virtual waiting room
Every open WebSocket connection is an open file descriptor on the server. The default file descriptor limit per process on Linux is 1024, though in practice administrators raise it to 65536 or more. Even so, with 100,000 concurrent users in the queue, managing 100,000 open connections requires precise operating system tuning before you even start writing application code.
But that is not the biggest problem. WebSocket is stateful. Every connection has state: which user, what position in the queue, when the last update was sent. When you want to scale horizontally and add a second server, you have to synchronize this state between nodes. You need sticky sessions or an external state store. The architecture immediately becomes nonlinearly complex.
And there is one more problem that shows up in the real world: mobile connections. A user is on a train, enters a tunnel, loses signal for 30 seconds. The WebSocket connection drops. What now? Does the user lose their position in the queue? Does the server know the connection dropped unexpectedly or intentionally? How long do you wait before you consider a slot freed? These are questions you have to answer in code and every answer adds complexity.
For a virtual waiting room — where the user is waiting anyway and a position update every few seconds is absolutely sufficient — WebSocket is a sledgehammer for a nail, with all the complexity baggage it brings.
Thundering Herd and why naive polling is not enough either
Since WebSocket is out, the first instinct is classic polling: the browser sends an HTTP request every X seconds asking "what is my position?".
The problem is that with a large number of users, naive polling with an identical interval leads to timer synchronization. Imagine 50,000 users who all started polling at the same moment with an identical 3-second interval. Every 3 seconds the server gets a spike of 50,000 requests at once, then nothing for 2.9 seconds, then another spike. Instead of even load you get a sawtooth that destabilizes the server.
Even worse, when the server goes down briefly and comes back, all clients waiting for reconnect try to connect simultaneously. This is the classic Thundering Herd problem.
Adaptive Polling with Jitter - how we mitigate it
Our solution combines three mechanisms:
Adaptive backoff - the polling interval is not fixed. When a user is far back in the queue (position > 100), they poll less frequently, e.g. every 10 seconds. When they approach the front (position < 10), they poll more frequently, e.g. every 2 seconds. The update frequency is proportional to how urgent that information is for the user.
Jitter - we add a random time offset to each polling interval. A user who was supposed to poll every 5 seconds actually polls every 4.2 or 5.8 seconds. With a large number of users, Jitter significantly mitigates timer synchronization and helps distribute load more evenly over time. It does not eliminate Thundering Herd completely - that would require additional mechanisms like a circuit breaker or server-side rate limiting - but it drastically reduces the amplitude of spikes.
Stateless server - every polling request is completely independent. The server stores no state per connection. The client sends its queue token, the server checks the position in KeyDB and responds. Connection closed immediately. You can add any number of nodes behind a load balancer with no state synchronization between them.
Implementation in Go with fasthttp - the devil is in the details
Go was the natural choice for this use case. Goroutines are significantly cheaper than system threads, allowing thousands of concurrent requests to be handled at minimal memory usage. fasthttp instead of the standard net/http library gives an additional performance boost through aggressive connection and buffer reuse.
When implementing high-performance endpoints in fasthttp, there are three things most tutorials do not show.
First, the standard json.Marshal from the encoding/json package is based on reflection and allocates new memory on the heap with every call. At tens of thousands of requests per second, Go's GC will choke collecting that garbage. For simple, predictable payloads like a polling endpoint response, it is much better to write directly to the request buffer via fmt.Fprintf.
Second, every KeyDB query must have a hard timeout. Without context.WithTimeout, when KeyDB hiccups briefly (e.g. during a disk snapshot via BGSAVE), thousands of goroutines block waiting for a response and the server dies from OOM even though the application code is correct.
Third, the response to a database error cannot be HTTP 500. When the Vue frontend receives a 500, it will likely panic and start retrying requests without delay - this way, when KeyDB fails, we bring down our own server with a flood of requests. Instead of an error, we return HTTP 200 with an extended next_poll_interval - this is a Circuit Breaker that forces clients to back off during failures.
func handlePoll(ctx *fasthttp.RequestCtx) {
token := string(ctx.QueryArgs().Peek("token"))
if len(token) < 32 { // Minimal validation before hitting the DB
ctx.SetStatusCode(fasthttp.StatusBadRequest)
return
}
// KeyDB protection: hard timeout on the query
c, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
defer cancel()
position, err := queue.GetPosition(c, token)
if err != nil {
// Circuit Breaker: instead of throwing 500, we tell clients to slow down
ctx.SetContentType("application/json")
ctx.SetBodyString(`{"status":"waiting","next_poll_interval":30000}`)
return
}
if position == 0 {
ctx.SetContentType("application/json")
ctx.SetBodyString(`{"status":"admitted"}`)
return
}
estimatedWait := calculateEstimatedWait(position)
nextPollInterval := calculateAdaptiveInterval(position)
// Zero-Allocation Response
ctx.SetContentType("application/json")
fmt.Fprintf(ctx, `{"status":"waiting","position":%d,"estimated_wait":%d,"next_poll_interval":%d}`,
position, estimatedWait, nextPollInterval)
}
func calculateAdaptiveInterval(position int) int {
base := 5000
if position < 10 {
base = 2000
} else if position > 100 {
base = 10000
}
jitter := rand.Intn(base/5) - base/10
return base + jitter
}Notice the absence of json.Marshal - at this scale, reflection in Go is too expensive, so we format the response directly, avoiding heap allocations. Additionally, on any database error (timeout), we do not return HTTP 500, but an extended next_poll_interval - a Circuit Breaker that forces the crowd to slow down their polling and protects the infrastructure from cascading failure.
What comes next
The next technical post will be about Early Intercept and Ed25519 token verification. How we intercept traffic before it reaches the client's server and why asymmetric cryptography is the right tool here.
Have questions about DropReady's architectural choices? Leave a comment or reach out directly.
FAQ
How does Adaptive Polling differ from WebSocket in the context of a virtual waiting room?
WebSocket maintains a persistent, open connection for the entire time a user is waiting, which at scale requires managing thousands of open file descriptors and synchronizing state between nodes. Adaptive Polling sends short, stateless HTTP requests every few seconds and immediately closes the connection. The server stores no state per user, making it trivially scalable horizontally.
Why does Jitter not eliminate Thundering Herd completely?
Jitter distributes client timers randomly, which drastically reduces spike amplitude during normal operation. However, when the server suddenly restarts or recovers from a failure, all clients whose connections were dropped try to reconnect around the same time. Full protection requires additional mechanisms: exponential backoff on the client side on connection errors, and rate limiting and circuit breaker on the server side.
What is a Zero-Allocation Response and why does it matter?
Standard json.Marshal allocates new memory on the heap with every call. At tens of thousands of requests per second, Go's Garbage Collector has to collect these allocations, causing short pauses (GC stop-the-world) and consuming CPU. By writing the response directly via fmt.Fprintf to the fasthttp buffer, we avoid these allocations and let the GC focus on genuinely needed operations.
What happens to a user in the queue when KeyDB is unavailable?
Instead of returning HTTP 500 (which would cause a flood of immediate retries), the server returns HTTP 200 with an extended next_poll_interval of 30 seconds. The frontend treats this as a normal response and waits 30 seconds before the next request. The user's position in the queue is preserved in KeyDB as long as the data has not expired. When KeyDB comes back, users gradually resume polling in a distributed timeframe.
