Context engineering: how to optimise what you send to an LLM

You have an AI agent that works. It has its memory, its tools, its carefully written prompt.

But on every call you send everything. The full history. The instructions. The tool results. And your bill explodes.

Context engineering is the art of choosing what to send to the LLM and what to sacrifice, when the context window cannot hold it all.

It is the topic everyone is talking about in 2025, and it is probably the most underrated skill when building AI agents.

---

The principle: not everything fits

An LLM has a limited context window. It is the maximum number of tokens it can process in a single call.

Even with 128,000 tokens (GPT-4o), your prompt fills up fast:

┌─────────────────────────────────────────┐
│         Context window (100%)           │
│                                         │
│  ┌───────────────────────────────┐      │
│  │ System prompt + instructions  │ 15%  │
│  ├───────────────────────────────┤      │
│  │ Tool descriptions             │ 10%  │
│  ├───────────────────────────────┤      │
│  │ Message history               │ 50%  │  ← the big one
│  ├───────────────────────────────┤      │
│  │ Tool results (RAG, web)       │ 15%  │
│  ├───────────────────────────────┤      │
│  │ Question + room for answer    │ 10%  │
│  └───────────────────────────────┘      │
└─────────────────────────────────────────┘

When it overflows, you have two options:

1. Do nothing: the API truncates or fails
2. Consciously choose what stays and what goes

Context engineering is option 2.

---

Step 1: everything has a priority

Not all messages are equal. In context engineering, you assign a weight to each message type:

system      (weight 10)  →  "who you are, your rules"
developer   (weight 8)   →  "how to answer"
user        (weight 6)   →  "what the user is asking"
assistant   (weight 4)   →  "what you already answered"

The system prompt must never be sacrificed. It defines the agent's behaviour. If you lose it, the LLM forgets its instructions — a classic production bug that almost everyone hits on long conversations.

At the other end, the assistant's old replies are the first to go. They are useful for context, but expendable.

---

Step 2: the token budget

Before you send anything to the LLM, you count.

The basic heuristic for estimating tokens:

1 token ≈ 4 characters

"Hello, how are you?"
= 19 characters
≈ 5 tokens

It is an approximation, but it is enough for budget management. In production you would use an exact tokenizer (like tiktoken for OpenAI), but the principle is the same.

The workflow:

1. Count the tokens of each message
2. Compare against the available budget
3. If it overflows → select by priority
4. Check that it fits
5. Send

The selection algorithm is greedy: you sort messages by descending score, add them one by one as long as the budget allows, then put everything back in chronological order.

---

Step 3: compressing without an LLM

This is the most interesting part. When a block of text is too long — a search result, an old summary, a document — you can compress it without calling an LLM.

How? With extractive compression: you score every line of the text and keep the best ones.

The scoring

Each line gets a score based on simple signals:

Is the line a heading (# or ** or -)?  → +2 points
Does the line contain a number?         → +1 point
Does the line contain a proper noun?    → +1 point

A concrete example

A 12-line meeting report that has to be compressed to 5:

# Meeting report                              → score 2 (heading)
General introduction to the project            → score 0
- Objectives met at 85%                        → score 3 (heading + number)
Various discussions about the team             → score 0
- Budget of 150,000 euros allocated            → score 3 (heading + number)
General comments                               → score 0
# Key decisions                                → score 2 (heading)
- Launch planned for 15 March 2025             → score 3 (heading + number)
Other remarks of no importance                 → score 0
Marie Dubois will be technical lead            → score 1 (proper noun)
- Additional budget of 25,000 euros            → score 3 (heading + number)
General conclusion of the report               → score 0

Result after compression (the 5 best scores, put back in order):

- Objectives met at 85%
- Budget of 150,000 euros allocated
# Key decisions
- Launch planned for 15 March 2025
- Additional budget of 25,000 euros

The filler lines (introduction, comments, conclusion) were dropped. The lines with numbers, headings and names survived.

Why not ask the LLM to summarise?

	Extractive compression	LLM summary
Cost	0, it is just code	A paid API call
Speed	Instant	1 to 3 seconds
Determinism	Same input = same output, always	Can vary on every call
Reliability	Copies lines verbatim, cannot hallucinate	Can distort a number or a name

In production you want the compression layer to be free, fast and predictable. The LLM comes in afterwards, on a context that is already clean.

---

Step 4: cleaning up before sending

Two cleanup operations people often forget:

Deduplication

In a long conversation, the same message can appear several times (retry, client-side bug, agent loop). Every duplicate wastes tokens for nothing.

The rule: walk the messages from the end to the beginning, and keep only the first occurrence (the most recent one). Old duplicates disappear.

Citation traceability

When a message cites a source (a Wikipedia link, an internal document), the following messages that have no source inherit that citation.

Message 3: "According to wikipedia.org, GDP is..."  → source: wikipedia
Message 4: "So GDP per capita would be..."          → no source
           → inherits the source from message 3

This is critical in a company. When a manager asks "where does this number come from?", you want to be able to walk back up the chain. Context engineering includes the provenance of information, not just its content.

---

The full pipeline

The steps form a chain. Here is the order in which the context is prepared before each call to the LLM:

                    Raw conversation
                          │
                          ▼
               ┌─────────────────────┐
           1.  │  Compose the prompt │  System + Developer + User
               └──────────┬──────────┘
                          ▼
               ┌─────────────────────┐
           2.  │  Select the memory  │  Rolling (recent)
               │                     │  + Episodic (important)
               └──────────┬──────────┘
                          ▼
               ┌─────────────────────┐
           3.  │  Clean up           │  Deduplication
               │                     │  + Citations
               │                     │  + Extractive compression
               └──────────┬──────────┘
                          ▼
               ┌─────────────────────┐
           4.  │  Pack under budget  │  Scoring + greedy selection
               │                     │  + back into chronological order
               └──────────┬──────────┘
                          ▼
                    Send to the LLM ✅

Every step shrinks the context while preserving the quality of the information. The LLM only receives what is necessary — clean, prioritised, within budget.

---

The takeaway

You are going to lose information. The context window is limited; that is a physical constraint.

The real question is not "how do I keep everything?" but "do you consciously choose what to keep, or do you let the LLM truncate at random?"

Context engineering is choosing. And it often has more impact than the choice of model itself.