How to host an AI chatbot

Hosting an AI chatbot sounds new, but the part you host is familiar: an HTTP API. The "brain" lives at the model provider (Anthropic, OpenAI, Google). Your server receives the message, calls the provider and relays the answer.

That changes what matters. You do not need a GPU or much memory. You need an always-on process, a well-guarded API key and protection against abuse, because every message costs money.

What is the architecture of an AI chatbot?#

Three pieces:

1. Front end: the chat UI (a website, a widget, an app). It holds the conversation and shows text as it arrives.
2. Backend (the part you host): an API with an endpoint like POST /chat. It validates input, adds the system prompt, calls the model and returns the reply.
3. Model provider: the LLM API. To see what happens underneath, read what an LLM is.

The backend exists for one core reason: the API key must never reach the browser. Anyone who opens DevTools can copy it. See also what to do if a key leaks.

What do you need before you start?#

• A Python version that is still supported. The minimum version for the Anthropic SDK, FastAPI and Uvicorn is listed in each project's documentation.
• An API key from the provider. This guide uses Anthropic's, but the structure is the same for any other.
• Basic knowledge of environment variables.

Create the project folder with a requirements.txt:

fastapi
uvicorn
anthropic

How do you write the streaming API?#

Without streaming, the user stares at an empty screen until the model finishes the whole answer. With streaming, text appears as it is generated. The Anthropic SDK exposes this through client.messages.stream and text_stream, and FastAPI relays it with StreamingResponse.

import os
from typing import Literal

from anthropic import AsyncAnthropic
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field

client = AsyncAnthropic()  # reads ANTHROPIC_API_KEY from the environment
MODEL = os.environ["ANTHROPIC_MODEL"]  # without it, the app fails at startup with KeyError
SYSTEM = "You are the assistant for the Example store. Answer briefly."

app = FastAPI()


class Msg(BaseModel):
    role: Literal["user", "assistant"]
    content: str = Field(max_length=4000)


class ChatIn(BaseModel):
    messages: list[Msg] = Field(min_length=1, max_length=20)


@app.post("/chat")
async def chat(body: ChatIn):
    async def stream_text():
        async with client.messages.stream(
            model=MODEL,
            max_tokens=1024,
            system=SYSTEM,
            messages=[m.model_dump() for m in body.messages],
        ) as stream:
            async for text in stream.text_stream:
                yield text

    return StreamingResponse(stream_text(), media_type="text/plain; charset=utf-8")


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "8000")))

Save the code as main.py. Worth noting:

• The API is stateless. The front end sends the whole conversation on every request, and the model remembers nothing between calls. The 20-message and 4,000-character limits cap cost and context size.
• The model is configuration, not code. Anthropic retires models over time, so the ID lives in the ANTHROPIC_MODEL variable with no default: if it is missing, the app stops right at startup instead of failing on the first message. Copy the ID from Anthropic's model list. For high-volume, latency-sensitive tasks, the usual pick is a small, fast, low-cost model, such as the Haiku line.
• max_tokens is required by the API and works as a per-reply cost ceiling.
• The final block lets you start it with python main.py, listening on 0.0.0.0 on the port in the PORT variable (8000 if it is not set).

Test locally:

pip install -r requirements.txt
export ANTHROPIC_API_KEY="paste-your-key-here"
export ANTHROPIC_MODEL="id-of-a-small-model"
uvicorn main:app --port 8000
curl -N -X POST localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hi, how are you?"}]}'

-N turns off curl's buffering, so you see the text arrive piece by piece.

How does the front end read the stream?#

In the browser, fetch already exposes the body as a stream:

async function ask(messages, onText) {
  const res = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });
  if (!res.ok || !res.body) throw new Error(`Error ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    onText(value);
  }
}

Call ask with the history and append each chunk (onText) to the reply bubble. If the front end is on another domain, configure CORS in FastAPI to allow only your site's origin.

How do you protect the key and the bill?#

A public chat endpoint invites abuse. Before sharing the link:

• Keep the key in an environment variable only, never in code or in the repository.
• Rate limit by IP or user, at the edge or with a FastAPI library.
• Set a spending limit in the provider dashboard. It is the safety net if your own limit fails.
• Require authentication if the bot is for internal use or for customers.
• Treat user text as untrusted. Someone may try to make the model ignore the system prompt. Read what prompt injection is before giving the model tools or sensitive data.

To estimate the bill, see how much an LLM API costs.

What does the server need to host this?#

• An always-on process. Platforms that suspend idle apps make the first reply slow. For chat, prefer a container that does not sleep.
• HTTPS. Browsers block insecure calls from HTTPS pages.
• Streaming support. Proxies that buffer the response break the effect. Test with curl -N in the final environment.
• Little RAM. The server only relays text, so usage is low. Measure under real load and read how to size RAM and CPU.
• Logs. You will need them to find out why the provider returned a 429 or 5xx. For the framework itself, see hosting FastAPI, Flask and Django.

What are the common errors?#

• Everything arrives at once: a proxy or compression layer is buffering between the app and the browser. Test the app directly with curl -N.
• 401 from the provider: ANTHROPIC_API_KEY is not set in the production environment.
• 404 or "model not found": the ID in ANTHROPIC_MODEL is wrong or the model was retired. Pick another one from the official list and restart the app, with no code change.
• 422 from FastAPI: the body does not match the Pydantic model (for example role other than user/assistant, or an empty conversation).
• High bill at month end: no per-user limit, or max_tokens set too high. See the protection section.

How to do this on Square Cloud#

The server section asked for an always-on process, HTTPS, unbuffered streaming and logs. On Square Cloud the API runs in an isolated, always-on container with no cold start and no sleep mode, and every web application gets a *.squareweb.app subdomain with HTTPS. The proxy in front of the application does not cut long connections and passes SSE and WebSocket traffic without buffering; still, repeat the curl -N test against your subdomain.

1. Upload main.py and requirements.txt as a ZIP in the dashboard, or use the CLI, GitHub or the VS Code extension. On start, the platform installs requirements.txt. The squarecloud.app file is optional: the dashboard generates it at deploy time, or you write it at the project root.
2. Pick main.py as the main file. The code's final block already listens on 0.0.0.0, on the port in the PORT variable, which the platform sets. A web application uses at least 512 MB from the plan's RAM pool.
3. Set ANTHROPIC_API_KEY and ANTHROPIC_MODEL in the application's environment variables. The change applies on the next restart, with no rebuild.
4. Follow the real-time logs to catch 429 or 5xx errors from the provider.

If you would rather not keep an account with a model provider, the AI Gateway, available from the Standard plan (R$44.99 per month), is an OpenAI-compatible API for the cubic model, in beta with early access at no extra cost. With the openai SDK you set the base URL to https://api.squarecloud.app/v2/ai, use an account key with the ai:chat scope and model="cubic"; the model can also call your functions (up to 32 tools) and search the web on its own. For this chatbot there are two caveats: the gateway does not stream yet, so the reply arrives whole at the end; and Standard serves one request at a time, up to 1,000 a day. A public chat with many simultaneous visitors is still better served by a provider that streams. Limits and plans may change when the beta ends.

Hobby, starting at R$12.49 per month, hosts the API but does not include the gateway. See the plans on API hosting.