Load balancing algorithms explained

Putting several servers behind a load balancer solves only half the problem. The other half is deciding, for every request, which server gets it. That decision is the balancing algorithm, and it changes the latency users feel, how hard each server works, and whether an in-memory session survives.

Each algorithm below comes with its selection rule, the case where it breaks down and its nginx configuration. At the end, a short Python script simulates the logic so you can try it.

What are the main load balancing algorithms?#

How does round robin work?#

Round robin is the simplest: the balancer keeps a pointer into the server list and advances one step per request. With three servers the order is 1, 2, 3, 1, 2, 3. It keeps no state about the servers, so it is cheap and has no surprises.

It works well when servers are the same size and requests cost about the same. It fails when one request holds a server for a long time: round robin keeps sending new work to it, because it counts turns, not load.

In nginx it is the default. The nginx documentation describes the behavior as weighted round robin, and every server has a weight of 1 unless you say otherwise.

How does weighted round robin handle different servers?#

If one server has twice the memory, it should get more traffic. The weight parameter does that. With weights 3, 1 and 1, out of every 5 new requests, 3 go to the first server and 1 to each of the others:

upstream app {
    server 10.0.0.11:3000 weight=3;
    server 10.0.0.12:3000;
    server 10.0.0.13:3000;
}

Servers without weight get the default weight of 1. This is the same example the nginx documentation uses.

How does least connections pick a server?#

The balancer tracks how many active connections each server has and sends the next request to the one with the fewest. Ties are broken by rotation. In nginx the directive takes server weights into account.

It is the best fit when request duration varies: uploads, heavy reports, WebSockets or calls to slow APIs. A server stuck on a long request accumulates connections and stops receiving new ones, while the others absorb the queue. When all requests are short and alike, the gain over round robin is usually small.

upstream app {
    least_conn;
    server 10.0.0.11:3000;
    server 10.0.0.12:3000;
    server 10.0.0.13:3000;
}

When should you use IP hash or consistent hashing?#

In these algorithms the choice depends on a key, not on load. The same key value always leads to the same server. That suits in-memory sessions, local caches, or connections that must return to the same place.

nginx's ip_hash uses the first three octets of the client's IPv4 address (or the whole IPv6 address) as the key:

upstream app {
    ip_hash;
    server 10.0.0.11:3000;
    server 10.0.0.12:3000;
    server 10.0.0.13:3000 down;
}

There are two limits. First, because only three octets enter the hash, clients on the same /24 network, and all clients behind one NAT, land on the same server, which unbalances the load. Second, if you remove a server, the hash is recalculated and many clients change target. That is why the documentation tells you to mark a temporarily removed server with down, which preserves the current mapping.

To pick a different key, such as the URL or a cookie, use hash with consistent. Consistent hashing (ketama) ensures that when a server is added or removed, only a small share of keys moves to a different server. It is the usual choice for caches, where remapping everything would empty the cache at once:

upstream cache {
    hash $request_uri consistent;
    server 10.0.0.21:8080;
    server 10.0.0.22:8080;
    server 10.0.0.23:8080;
}

Are there other algorithms?#

Yes. nginx also offers random, which draws a server while respecting weights. The random two least_conn variant draws two servers and keeps the one with fewer active connections. It stops several independent balancers from all sending traffic to the same idle server at the same moment.

There is also least_time, which prefers the server with the lowest average response time and fewest connections. According to the documentation, before version 1.31.0 it was only part of the commercial subscription (NGINX Plus). Check your version before relying on it.

How can you simulate the algorithms in code?#

This Python 3 script implements round robin, weighted round robin, least connections and IP hash. Save it as lb.py and run python lb.py:

import hashlib
import itertools

SERVERS = ["app-1", "app-2", "app-3"]


def round_robin():
    return itertools.cycle(SERVERS)


def weighted_round_robin(weights):
    # {"app-1": 3, "app-2": 1} becomes app-1, app-1, app-1, app-2
    return itertools.cycle([s for s, w in weights.items() for _ in range(w)])


def least_connections(active):
    # active: open connections per server; ties go to the first in the list
    return min(SERVERS, key=lambda s: active[s])


def ip_hash(client_ip):
    digest = hashlib.sha256(client_ip.encode()).digest()
    return SERVERS[int.from_bytes(digest[:4], "big") % len(SERVERS)]


rr = round_robin()
print([next(rr) for _ in range(6)])
wrr = weighted_round_robin({"app-1": 3, "app-2": 1, "app-3": 1})
print([next(wrr) for _ in range(5)])
print(least_connections({"app-1": 12, "app-2": 3, "app-3": 7}))
print(ip_hash("203.0.113.7") == ip_hash("203.0.113.7"))

The output is ['app-1', 'app-2', 'app-3', 'app-1', 'app-2', 'app-3'], then ['app-1', 'app-1', 'app-1', 'app-2', 'app-3'], then app-2 (the least busy) and True (same IP, same server). Append a fourth server to SERVERS and run it again: the modulo-based ip_hash remaps a good share of clients, which is exactly what consistent hashing avoids. This script's weighted round robin sends the three requests to the first server back to back, while a real balancer may interleave the choices.

How do you choose the right algorithm?#

• Equal servers, short and similar requests: round robin.
• Servers of different sizes: weighted round robin.
• Long or unpredictable requests: least connections.
• Server-side state you cannot move: IP hash. The better fix, though, is to take state out of memory, keep sessions in Redis or a database, and go back to an algorithm that needs no key.
• Distributed cache: consistent hashing.

In every case, balancing only works if the servers are interchangeable and the balancer pulls failing servers out of rotation. The reverse proxy in front of them is what applies the rule.

How to do this on Square Cloud#

On Square Cloud, balancing works per domain. You deploy the same code to several applications and attach the same custom domain to all of them; the squareweb.app subdomain cannot be balanced. You set up the domain by creating, at your registrar, the DNS records the dashboard shows (see how to use a custom domain).

Traffic for the domain is distributed round robin across the applications, with passive failover (no active health check) if one becomes unavailable. There is no per-request charge. Custom domains start on the Standard plan, which allows up to 2 applications per domain. Pro allows up to 5 and Enterprise up to 10.

Because the algorithm is round robin and there are no sticky sessions, each request can land on a different application, and each one has its own disk. Keep sessions in a database or Redis, not in process memory. See the walkthrough in load balancing on Square Cloud.