What is a large language model (LLM)?

When you ask a chat assistant to summarize a text or fix a bug, an LLM is answering. The acronym stands for large language model. It is the core of almost every assistant, chatbot and coding agent you use today.

Knowing the basics of how it works changes how you use it. You will see why it can be confidently wrong, why conversation history costs money, and why the same question can get different answers.

What is an LLM, in simple terms?#

An LLM is a program that takes some text and returns the most likely continuation. During training, it reads enormous amounts of text and learns to predict what comes next. After doing that billions of times, the model absorbs grammar, facts, styles and even code patterns.

The "large" refers to the number of parameters, the numeric values tuned during training. GPT-3, from 2020, had 175 billion parameters, according to the paper that introduced it. The same paper showed that scaling up the model improves performance on tasks described only through instructions and a few examples in the text, with no retraining.

How does an LLM generate text?#

In three steps, repeated in a loop:

1. Tokenization. The text is cut into tokens: word fragments, whole words or symbols. "Hosting" may become one or two tokens.
2. Prediction. The model reads all the tokens and computes a probability for every possible next token.
3. Selection. One token is sampled or picked from that distribution and appended to the text. The cycle restarts until the model emits an end signal.

A tiny Python example shows the idea. It "trains" by counting which words follow which, then generates sentences:

import random
from collections import defaultdict

corpus = "the cat sleeps on the sofa . the cat eats fish . the dog sleeps on the rug .".split()

following = defaultdict(list)
for current, nxt in zip(corpus, corpus[1:]):
    following[current].append(nxt)

word, sentence = "the", ["the"]
while word != "." and len(sentence) < 12:
    word = random.choice(following[word])
    sentence.append(word)

print(" ".join(sentence))

Run it a few times and the sentence changes: "the cat eats fish .", "the dog sleeps on the sofa .". A real LLM runs the same loop, but with a neural network that looks at the whole text, not just the last word. That is what lets it hold a coherent conversation.

What is a transformer?#

It is the neural network architecture behind modern LLMs, introduced in the 2017 paper Attention Is All You Need. Its core idea is attention: when predicting the next token, the model weighs how relevant each earlier token is. In "the dog chased the cat because it was scared", attention helps decide what "it" refers to.

A transformer also processes many tokens in parallel during training, which is what made today's model sizes possible.

What are a context window, temperature and a hallucination?#

The context window is everything the model considers at once: your question, the history and the answer it generates. Anthropic's documentation calls it the model's "working memory" and lists the window size of each Claude model, which reaches hundreds of thousands of tokens or more on the largest ones. More context does not automatically mean better results: the same page explains that accuracy and recall degrade as context grows.

Temperature is a parameter that controls how the next token is sampled. Low values make answers predictable, which helps when extracting data or generating code. High values make them more varied, which suits brainstorming and creative text. Not every model exposes this control, so check the docs for the one you use.

A hallucination is a fluent, wrong answer: a library function that does not exist, an invented quote, a swapped date. It happens because training rewards plausible text, not verified facts. To reduce the risk, put the correct source in the prompt and tell the model to say when it does not know.

What is an LLM used for day to day?#

• Writing and reviewing code, explaining errors, writing tests.
• Summarizing and classifying text: support tickets, emails, logs.
• Extracting structured data from free text, such as turning an order into JSON.
• Conversation: support chatbots, assistants on Discord, Telegram and WhatsApp.
• Agents: the model decides to call tools (look up data, run a command) and uses the result to continue.

If you want to build something like this, see how to build a Discord AI bot and how to host an AI chatbot.

When should you not use an LLM?#

Skip it when the answer must always be exact and verifiable: financial calculations, fixed business rules, tax ID validation. In those cases, an if or a SQL query is cheaper, faster and always gives the same result. Skip it too when a mistake is costly and nobody reviews the output.

Other practical limits:

• No state. The API does not remember the conversation. Your app resends the history on every call, and those tokens are billed again. See how much an LLM API costs.
• Frozen knowledge. The model knows what it saw up to its training cutoff. For current data, send it in the request or give the model a search tool.
• Non-deterministic output. The same input can produce different outputs. Test with many examples, not one.

What related terms come up with LLMs?#

• Prompt: the input text, with instructions and data.
• Fine-tuning: extra training of a model on your examples. Rarely the first step.
• RAG: retrieving relevant passages from your documents and putting them in the prompt.
• Embedding: a numeric representation of text, used for similarity search.
• Open-weight model: a model whose weights are published, so you can run it yourself. An API model, by contrast, runs on the provider's servers.

How to do this on Square Cloud#

In practice, using an LLM in your project means making an HTTP call: your bot or API builds the prompt with the history, sends it to the model and returns the reply. You host that code, not the model. On Square Cloud it runs in an isolated, always-on container, on one of several runtimes such as Node.js, Python and Go: see the runtimes available today on Runtimes. The Hobby plan starts at R$12.49 a month with 1 GB of RAM. That is enough for code that calls an API, but not for running a local model with torch and transformers.

For the model itself, from the Standard plan (R$44.99 a month) you can use the AI Gateway: an OpenAI-compatible API for the cubic model, the same one behind the dashboard assistant. Point any OpenAI SDK at the base URL https://api.squarecloud.app/v2/ai and use an account API key with the ai:chat scope. Everything in this article still applies:

• No state: you resend the history in messages on every call.
• Context window: up to 16K tokens on Standard, up to 24K on Pro and up to 32K on Enterprise.
• Frozen knowledge: the model has built-in web search, decides on its own when to search and cites the URLs.
• Agents: function calling in the OpenAI format, with up to 32 tools.

The gateway is in beta with early access at no extra cost, and its limits and plan availability may change when the beta ends. It takes text only and does not stream yet. See the details on the AI Gateway page.