Streaming AI responses to users (SSE)

A large language model generates text one token at a time, and a long answer can take ten seconds or more to finish. If your backend waits for the whole reply before responding, the user stares at a spinner for that entire time. With streaming, the first words appear in a fraction of a second and the rest follow as they are generated.

The standard way to do this over HTTP is server-sent events (SSE). Below, you build a small Node.js endpoint that streams a model reply and a browser client that renders it live. Then you fix the problems that usually break streaming in production. For background on the model side, see What is a large language model (LLM)?.

What are server-sent events?#

SSE is a plain HTTP response that stays open until the server is done. The server sends Content-Type: text/event-stream and writes text messages as it goes. Each message is one or more field: value lines, followed by a blank line:

data: Hello

event: done
data: {}

The fields that matter:

• data: is the payload. Consecutive data: lines in one message are joined with a newline.
• event: names the event. Without it the browser treats the message as a default message event.
• id: sets the last event ID, which the browser sends back as Last-Event-ID when it reconnects.
• retry: sets the reconnect delay in milliseconds.
• A line starting with : is a comment, handy as a keep-alive.

Because it is ordinary HTTP, SSE works with the normal tooling for auth, logging and TLS. Most LLM providers use SSE for their own streaming APIs, so their SDKs already give you a stream to forward.

SSE or WebSockets for AI responses?#

For a chat reply, SSE is usually enough. Data flows one way, from server to browser, and the user's next message is just another HTTP request. WebSockets add a two-way channel, but connection upgrades, heartbeats and reconnect logic become your job. Pick WebSockets when the client must send data continuously on the same connection, such as voice input. Read What is a WebSocket and how does it work? for that case.

How do you stream tokens from the server?#

Install Express and the Anthropic SDK (the same pattern works with any provider that streams):

npm install express @anthropic-ai/sdk

Create server.mjs. The API key and model come from environment variables, never from the code:

import express from "express";
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY
const app = express();
app.use(express.json());

app.post("/api/chat", async (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache, no-transform",
    Connection: "keep-alive",
    "X-Accel-Buffering": "no", // tells nginx not to buffer this response
  });

  const send = (event, data) =>
    res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

  const stream = client.messages.stream({
    model: process.env.ANTHROPIC_MODEL, // a model id from your provider's docs
    max_tokens: 1024,
    messages: req.body.messages,
  });

  // Stop paying for tokens nobody will read.
  res.on("close", () => {
    if (!res.writableEnded) stream.abort();
  });

  stream.on("text", (text) => send("token", text));

  try {
    await stream.finalMessage();
    send("done", {});
  } catch (err) {
    if (!res.destroyed) send("error", { message: "Generation failed" });
  }
  res.end();
});

app.listen(process.env.PORT || 3000, "0.0.0.0");

Run it with ANTHROPIC_API_KEY=... ANTHROPIC_MODEL=... node server.mjs. A few details are easy to miss:

• JSON.stringify on every chunk keeps newlines inside the text from breaking the data: line format.
• no-transform stops intermediaries from compressing or rewriting the stream.
• The close handler aborts the upstream request when the user closes the tab or presses stop. Without it, you keep generating (and paying for) a reply nobody sees.
• Validate and limit req.body.messages in a real app, and authenticate the request before calling the model.

How do you read the stream in the browser?#

The built-in EventSource only sends GET requests, with no body and no custom headers, so it cannot carry a conversation or an auth token. Use fetch and read the body as a stream. The parser below splits on blank lines, which is how SSE messages end:

async function streamChat(messages, onToken, signal) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;

    let end;
    while ((end = buffer.indexOf("\n\n")) !== -1) {
      const raw = buffer.slice(0, end);
      buffer = buffer.slice(end + 2);

      let event = "message";
      let data = "";
      for (const line of raw.split("\n")) {
        if (line.startsWith("event:")) event = line.slice(6).trim();
        else if (line.startsWith("data:")) data += line.slice(5).trimStart();
      }
      if (event === "token") onToken(JSON.parse(data));
      if (event === "error") throw new Error(JSON.parse(data).message);
      if (event === "done") return;
    }
  }
}

Call it with an AbortController, so a Stop button cancels the request and, through the server's close handler, the model call too:

const controller = new AbortController();
const out = document.querySelector("#answer");

streamChat(
  [{ role: "user", content: "Explain SSE in two sentences." }],
  (token) => (out.textContent += token),
  controller.signal,
);

document.querySelector("#stop").onclick = () => controller.abort();

Setting textContent rather than innerHTML matters: model output is untrusted text. If you render Markdown, sanitize the HTML first.

How do you check that streaming really works?#

Test the endpoint without a browser, using curl -N (the -N flag turns off curl's own buffering):

curl -N -X POST http://localhost:3000/api/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Count to ten."}]}'

You should see event: token lines appear one after another, then event: done. If everything lands at once at the end, your code streams correctly but something along the way is buffering it. Repeat the test against the deployed URL: a stream that works locally and not in production almost always points to a proxy.

Why does the stream arrive all at once?#

These are the usual causes, from most to least common:

1. Proxy buffering. nginx buffers responses by default. Set proxy_buffering off; for the route, or keep the X-Accel-Buffering: no header from the example. See What is a reverse proxy?.
2. Compression. Gzip middleware may hold chunks until it has enough to compress. Exclude text/event-stream from compression.
3. Missing headers. Without Content-Type: text/event-stream some clients and proxies wait for the full body.
4. Idle timeouts. A proxy may close a connection that sends nothing for tens of seconds. Send a comment line such as : ping followed by a blank line every 15 seconds while the model is thinking.
5. Browser connection limits. Over HTTP/1.1, browsers allow about six open SSE connections per domain across all tabs. HTTP/2 raises that to the negotiated stream limit, 100 by default.

If you use EventSource for a GET-only stream, it reconnects automatically after a drop and sends the last id: it saw as Last-Event-ID. A model call cannot resume mid-generation, so for chat the safer pattern is to show an error and let the user retry.

How to do this on Square Cloud#

A streaming endpoint holds each connection open for several seconds, and that is where a lot of hosting falls short: a buffering proxy delivers everything at the end, and a process that goes to sleep drops the request. On Square Cloud, the proxy in front of applications passes SSE through without buffering and does not cut long connections, and applications run in always-on containers with no cold start and no sleep mode.

To publish server.mjs on the Node.js runtime:

1. Upload the project with its package.json. There is no build step: at start, the platform runs npm install when node_modules is missing.
2. Pick server.mjs as the main file. The squarecloud.app file is optional: the dashboard generates it at deploy, or you write it at the project root.
3. Add ANTHROPIC_API_KEY and ANTHROPIC_MODEL to the application's environment variables, through the dashboard, API, CLI or VS Code. A change applies on the next restart, without a rebuild.

The application gets a *.squareweb.app subdomain with HTTPS and listens on 0.0.0.0, on the port in the PORT variable, as the example already does. A web application uses at least 512 MB of the plan's RAM pool; Hobby starts at R$12.49 a month. After deploying, repeat the curl -N test against the subdomain while you watch the logs in real time: if the token events arrive one by one, the whole path works.

One caveat about the AI Gateway, the OpenAI-compatible API for the cubic model, in beta from the Standard plan: it does not stream yet, and a request with stream: true comes back with a 400 error. To show the answer token by token, as here, stay with a provider that streams. The gateway fits calls where the answer can arrive whole, such as background summaries and classification. For the full app around the endpoint, see how to host an AI chatbot and the API hosting page.