What is RAG?

An LLM only knows what it saw in training. It does not know your company's internal handbook, the catalog you updated yesterday or your product's documentation. Ask about those and it either admits it does not know or invents an answer that sounds true.

RAG (retrieval-augmented generation) fixes this without training anything. Before asking the model, your code searches your own data for the most relevant passages and sends them along with the question. The model answers by reading, not by remembering.

How does a RAG system work?#

There are two phases. The first runs once (and again when the data changes); the second runs on every question.

Indexing:

1. Gather the documents (PDFs, pages, help articles, database rows).
2. Split each one into smaller pieces, called chunks.
3. Generate an embedding for each chunk with an embedding model.
4. Store the chunk and its embedding in an index.

Querying:

1. The user asks a question.
2. You generate the question's embedding with the same model.
3. You search the index for the k closest chunks (retrieval).
4. You build a prompt with those chunks as context.
5. You send the prompt to the LLM, which writes the answer.

The "augmented" in the name comes from step 4: the prompt is augmented with retrieved information.

What are chunks and embeddings?#

A chunk is a piece of text small enough to fit in the prompt next to several others, and large enough to make sense on its own. A paragraph or a short section usually works. Chunks that are too small lose context, and chunks that are too large dilute the topic and burn tokens.

An embedding is a list of numbers a model produces to represent the meaning of text. Texts with similar meaning produce nearby vectors, even with no words in common. That is what lets "how do I get my money back" find a document titled "refund policy". Storing these vectors and searching by proximity is the job of a vector database.

What does RAG look like in code?#

The example below shows the whole skeleton in plain Python with no dependencies. To keep it runnable, retrieval uses word overlap instead of embeddings, and the output is the prompt you would send to the LLM.

import math
import re
from collections import Counter

DOCS = [
    "Refunds are accepted within 7 days of purchase.",
    "Support is available Monday to Friday, 9am to 6pm.",
    "To change your password, open Settings and click Security.",
]

def tokens(text):
    return re.findall(r"\w+", text.lower())

def similarity(a, b):
    ca, cb = Counter(tokens(a)), Counter(tokens(b))
    dot = sum(ca[t] * cb[t] for t in ca)
    norm = math.sqrt(sum(v * v for v in ca.values())) * math.sqrt(sum(v * v for v in cb.values()))
    return dot / norm if norm else 0.0

def retrieve(question, k=1):
    return sorted(DOCS, key=lambda d: similarity(question, d), reverse=True)[:k]

def build_prompt(question):
    context = "\n".join(retrieve(question))
    return (
        "Answer using only the context below. "
        "If the answer is not in it, say you do not know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )

print(build_prompt("Within how many days can I get a refund?"))

The output is the prompt with the refund passage as context. Note that word matching is fragile: "refund" and "refunds" are different tokens, and the hit came from "days" and "within". In a real system, you swap this search for embeddings, which measure similarity by meaning. The rest of the flow (building the prompt and calling the LLM) stays the same.

When should you use RAG, and when not?#

RAG works well when:

• the data changes often (prices, documentation, policies);
• the data is private and not in the model's training set;
• you need to cite where an answer came from, because each chunk has a known source;
• the text is too large to fit in the prompt.

RAG is unnecessary when:

• all the content fits in the prompt: just send it all;
• you want to change the style or format of answers, which is a job for prompting or fine-tuning;
• the question needs reasoning over the whole set (like "summarize everything"), because retrieval returns only a few passages.

RAG or fine-tuning?#

Fine-tuning adjusts the model's weights, which suits behavior and style. RAG brings knowledge to the model without changing it, and updating the knowledge base means re-indexing only the documents that changed. Nothing stops you from using both.

What are the limits of RAG?#

• Retrieval sets the quality ceiling. If search returns the wrong passage, the model answers well about the wrong thing. Most RAG tuning is retrieval tuning: chunk size, the value of k, the embedding model.
• Hallucination does not vanish. The model can ignore the context. Clear prompt instructions and testing with real questions reduce the problem.
• Cost and latency grow. Each question generates an embedding and sends more tokens to the LLM. See how that adds up in LLM API cost.
• Sensitive data needs care. Whatever is in the index can reach the prompt. Control who can retrieve what.

How to do this on Square Cloud#

A RAG system is an ordinary application: an API that receives the question, generates the embedding, queries the index and calls the LLM. On Square Cloud you deploy that API on one of several runtimes, such as Python and Node.js, in an isolated, always-on container, with the provider keys in environment variables. See how to publish it in API hosting.

Each stage of the flow lives in its own place:

• Embeddings: an external provider. The Python runtime ships prebuilt torch and transformers wheels, but a local embedding model does not fit in the 1 GB Hobby plan.
• Index: the platform has no vector database, and the hosted PostgreSQL is native PostgreSQL with no extensions (no pgvector). Use an external vector service or, with a few thousand chunks, an in-memory search inside the application.
• Documents and metadata: database hosting from Standard (R$44.99 a month); Hobby (R$12.49) includes no database.
• Generation: from Standard, the AI Gateway answers in the OpenAI chat completions format with the cubic model. Point the OpenAI SDK at the base URL https://api.squarecloud.app/v2/ai and use an API key with the ai:chat scope.

The gateway only handles generation. The retrieved chunks, the history and the question must fit in the plan's context window (up to 16K tokens on Standard, up to 24K on Pro), so pick k and the chunk size with that in mind. It is in beta with early access at no extra cost, and the limits may change when the beta ends. For a full chatbot, see how to host an AI chatbot.