Large Language
Models

Attention Is All You Need — and Why It Matters for Agents

Every major LLM in use today — from all frontier providers — is built on the Transformer architecture, introduced by Vaswani et al. in 2017. The core innovation was replacing recurrence (processing tokens one at a time, left to right) with self-attention: a mechanism that lets every token in a sequence directly attend to every other token simultaneously.

For agents, the practical implication is immediate: the model does not "read" your prompt sequentially the way a human does. It computes relationships across the entire input in parallel. This is why long, dense context — hundreds of messages, large tool outputs, multi-document inputs — can be processed in a single forward pass, and why the context window is the primary constraint on what an agent can "see" at once.

How Text Becomes Numbers — and Why It Matters

LLMs do not process characters or words directly. They process tokens — subword units produced by a learned tokenizer. The most common tokenization algorithm used by frontier models is Byte Pair Encoding (BPE), which iteratively merges the most frequent character pairs until it reaches a fixed vocabulary size (typically 32k–100k tokens).

A practical rule of thumb: 1 token ≈ 4 characters of English text, or roughly ¾ of a word. A 1,000-word document is approximately 1,333 tokens. Code and non-English text tokenize less efficiently — a single Unicode character may map to multiple tokens, which has real cost implications when agents process large outputs from code interpreters or foreign-language documents.

The Agent's Working Memory

The context window is the maximum number of tokens a model can process in a single forward pass — both the input (prompt + tool results + history) and the output (generated tokens) combined. It is the hard boundary of what the LLM can "see" and reason over at any moment.

For agents, the context window is their working memory. Everything the agent knows during a given turn — the system prompt, the conversation history, tool results, injected documents, and the current reasoning trace — must fit within it. When the context fills up, information must be compressed, summarized, or evicted, which introduces loss and error.

From Raw Text to Instruction-Following Agent Brain

Modern LLMs go through a multi-stage training pipeline. Understanding these stages tells you why the model behaves the way it does — and what you can and cannot change at inference time.

Constitutional AI (CAI), introduced by Anthropic in 2022, extends RLHF by replacing the human preference labeling step with a set of written principles — a "constitution" — that the model uses to critique and revise its own outputs. This makes the alignment process more scalable and transparent. Anthropic applies CAI to train Claude models.

What You Control at Runtime

When you call an LLM API, you have a set of parameters that control how the model generates its output. These are the levers available to agent builders. Understanding each one prevents common mistakes — like setting temperature to 0 for creative tasks or leaving top-p at default when you need deterministic tool calls.

Structured outputs (also called "constrained generation" or "JSON mode") force the model to produce output that conforms to a schema at the token level — invalid tokens are masked out before sampling. This is how agent frameworks guarantee that tool call arguments are valid JSON even when temperature is non-zero. Anthropic's tool use API enforces this automatically when tools are defined.

Hard Limits You Must Design Around

LLMs are powerful reasoning engines but have well-documented limitations. Agent builders who ignore these ship systems that fail in predictable, embarrassing ways. Design your agents assuming these limitations are permanent — because as of 2026, none have been fully solved.

Verified References

Every claim in this section is grounded in one of these sources. No content is generated from model training data alone.