How much does a RAG system cost? Embeddings, database and LLM

A RAG system looks expensive because it has many parts: an embedding model, a vector database, an LLM, a server. In practice the bill usually piles up in just one of them, and knowing which one changes your design decisions. Tuning the vector database while sending 10 huge chunks to an expensive model saves money in the wrong place.

The numbers in this article are hypothetical, chosen to keep the math easy, in US dollars and before taxes. They are not any provider's prices; real rates are on the official pricing pages.

What makes up the cost of RAG?#

RAG has two phases, each with its own costs:

• Indexing (once, and again on every update): split documents into chunks, generate an embedding for each chunk and store the vectors in an index.
• Querying (on every question): embed the question, find the most similar chunks and send the question plus chunks to the LLM, which writes the answer.

That gives four lines on the bill: embeddings, vector storage and search, generation by the LLM and hosting for the application that ties it all together.

How much do embeddings cost?#

Embedding models only charge for input, per million tokens, and cost far less than generation models. The math below uses three hypothetical tiers:

Some providers give each account a no-cost token allowance, and others discount batch mode. Current rates are on the pricing pages of providers such as OpenAI, Voyage AI and Google. The embedding provider does not have to be the one that generates answers: Anthropic's embeddings documentation, for example, points to Voyage AI.

The indexing bill is small. Hypothetical example: a corpus of 10,000 documents of 2,000 tokens adds up to 20 million tokens: $1 on the small embedding, $5 on the premium one and nothing while a no-cost allowance lasts, if the provider offers one. Questions are embedded too, but at about 50 tokens each, 1,000 questions a day add up to 1.5 million tokens a month: seven and a half cents.

How much does storing and searching the vectors cost?#

Each vector's size depends on its dimensions: each one takes 4 bytes in single precision (float32) and 2 bytes in half precision (float16), plus a few bytes of header per vector. With 20 million tokens in 500-token chunks, you get 40,000 chunks:

• 40,000 × (4 × 1,536 + 8) bytes ≈ 246 MB of vectors in single precision.
• Half of that, ≈ 123 MB, in half precision.
• Add the chunk text (which you keep to send to the LLM) and the search index, which takes its own space.

There are two ways to host this:

1. A vector extension on a database you run yourself. Some relational databases accept open source extensions that store and search vectors. The cost is the database server, and 40,000 chunks fit comfortably on a small server.
2. A dedicated vector service. It usually has a no-fee plan with limits on storage, monthly reads and writes and region, plus paid plans with a monthly minimum and charges per GB stored and per million read and write operations. Pinecone's pricing page is one example of how that table is laid out.

For a corpus this size, storage costs less than a dollar either way. What matters is the plan minimum. The trade-offs between database types are in what is a vector database.

How much does LLM generation cost?#

This is most of the bill. Every question sends the model the system prompt, the retrieved chunks and the question. With 5 chunks of 500 tokens, the input reaches 3,000 tokens even when the question is one line long.

The math uses the same three hypothetical price tiers per million tokens (input and output) as the LLM API article: a small model at $0.20 and $1, another small one with pricier output at $0.50 and $4, and a mid-tier one at $4 and $20. Real prices are on each provider's pricing page. How per-token billing works is covered in how much an LLM API costs.

How much does a RAG system with 1,000 questions a day cost?#

Hypothetical example: 10,000 documents (20 million tokens), 500-token chunks, 1,000 questions a day (30,000 a month), 5 chunks per question, a 450-token system prompt, a 50-token question and a 400-token answer. The script below does the math (any current Python 3 release, no dependencies):

"""rag_cost.py: run with python rag_cost.py.
Prices in US$ per million tokens: hypothetical values, swap in current ones."""

EMBEDDING = 0.05  # small embedding (hypothetical)
MODELS = {
    "small-a": (0.20, 1.00),
    "small-b": (0.50, 4.00),
    "mid-tier": (4.00, 20.00),
}

corpus_tokens = 20_000_000  # indexing: 10,000 documents of 2,000 tokens
tokens_per_chunk = 500
dimensions = 1536
chunks = corpus_tokens // tokens_per_chunk
bytes_per_vector = 4 * dimensions + 8  # float32 + header

print(f"Indexing: US$ {corpus_tokens / 1e6 * EMBEDDING:.2f} (one time)")
print(f"Chunks: {chunks}, vectors: {chunks * bytes_per_vector / 1e6:.0f} MB")

questions = 30_000  # 1,000 questions a day
input_tokens = 450 + 5 * tokens_per_chunk + 50  # system + 5 chunks + question
output_tokens = 400
question_embeddings = questions * 50 / 1e6 * EMBEDDING

for name, (price_in, price_out) in MODELS.items():
    llm = questions * (input_tokens * price_in + output_tokens * price_out) / 1e6
    print(f"{name}: US$ {llm + question_embeddings:.2f} a month")

Result:

Switching the generation model moves the bill from $30 to $600 a month; switching the embedding model changes it by a few dollars. Then add hosting for the application and the database, a fixed monthly cost.

What hidden costs show up?#

• Too many chunks. Going from 5 to 10 chunks nearly doubles the input of every question. Test how few chunks still give good answers.
• Re-indexing. Changing the embedding model, the chunk size or how you split documents means generating every vector again. Cheap for 20 million tokens, expensive for billions.
• Conversation history. In a chat, history is added on top of the chunks on every message, and input grows fast.
• Long context. Some models charge more for prompts above a certain size. A RAG system that dumps whole documents into the prompt lands in that tier.
• Rerankers and evaluation. A reranking model improves retrieval, but it is one more paid call per question. Running quality tests also burns tokens.
• Document extraction. Scanned PDFs need OCR before they become text, and that can cost more than the embeddings.
• Abuse. A public endpoint with no per-user limit becomes an open LLM API. See what is prompt injection and enforce limits in code.

How do you lower the cost of RAG?#

1. Start with the smallest generation model and move up only if the answers are not good enough.
2. Retrieve less, and better. Smaller chunks, fewer chunks and a minimum similarity cutoff shrink the input.
3. Cap the output with max_tokens and ask for short answers.
4. Use prompt caching for the fixed part (instructions and examples). The chunks change with every question, so the savings come from the system prompt.
5. Cache answers to repeated questions. An internal FAQ tends to get the same questions over and over.
6. Use batch for indexing when the provider offers a batch-mode discount for embeddings.

How to do this on Square Cloud#

The table above shows that almost the whole bill is generation. On Square Cloud you host the RAG application and, from Standard, you can take that line off your dollar invoice.

The application (the API that receives the question, queries the index and calls the model) runs in an always-on container with no cold start. Among many runtimes, such as Python, Node.js and Go, you pick the one for your stack; see the runtimes available today on Runtimes. Provider keys live in the app's environment variables.

For generation, the AI Gateway, available from Standard (R$44.99 a month), is an OpenAI-compatible API that serves the cubic model. You point the OpenAI SDK at https://api.squarecloud.app/v2/ai, use an account API key with the ai:chat scope and send the prompt with the retrieved chunks, as you would with any provider. The gateway is in beta, with early access at no extra cost, and limits and availability may change when the beta ends. Standard allows up to 2.5 million tokens and up to 1,000 requests a day, with a context window of up to 16k tokens; Pro allows up to 5 million tokens and up to 5,000 requests a day. The example's 1,000 daily questions add up to about 3.4 million tokens: more than Standard allows, within Pro.

The gateway only handles generation. Embeddings stay with a provider like the ones in the table, which cost cents, and the index lives in an external vector service: there is no vector database on the platform, and the hosted PostgreSQL is native PostgreSQL without extensions (no pgvector). Database hosting, useful for documents and history, also starts on Standard, in the same datacenter as your applications, with about 1 ms average latency. Hobby, from R$12.49, includes neither databases nor the gateway and fits a light API that calls external providers. A local embedding model, using the Python runtime's torch and transformers wheels, does not fit in the 1 GB Hobby plan.

See the details on AI Gateway and how to publish the API on API hosting.