By Romain Juban, Vice President, Data Science, C3 AI and Ivan Robles, Lead Data Scientist, C3 AI
A large language model (LLM) has no memory. Every decision it makes depends entirely on the tokens sitting in its context window at that instant. For a chatbot answering one question, that’s a minor detail. For an enterprise agent running an investigation workflow, a forecasting pipeline, or an operational decision loop across dozens of tool calls, this becomes a larger problem: the agent is only ever as reliable as what it’s currently looking at.
In our experience building enterprise agents, most production failures trace back to this issue. The model was given the wrong information, or given the right information at the wrong moment. Context engineering is the discipline of controlling that deliberately, structuring what an agent sees and when it sees it, rather than letting a conversation history accumulate and hoping the model attends to the right parts.
Prompt engineering optimizes the wording of a single instruction. That was enough when the job was a one-off chat response. It falls short for an agent that plans over dozens of steps, calls tools, and carries state across a long-running task, where the harder problem is deciding what enters the context window in the first place.
Figure 1. Prompt engineering shapes a single, mostly static prompt for a chatbot. Context engineering selects a relevant subset from a large pool of candidate context for each agent action, then feeds the agent’s output back in for the next step.
This blog covers the mechanics that make context engineering necessary: how LLMs process the tokens in their window, why they degrade when that window fills with clutter, and the three failure modes every long-running enterprise agent runs into. Throughout, we define where the C3 Agentic AI Platform already handles this work, so teams don’t have to rebuild the mechanics from scratch in their own deployments.
Context Engineering Governs the Agent’s Working Memory
LLMs are shipping with ever-larger context windows, which makes it tempting to pour everything into the prompt and let the model sort it out. In practice, that approach backfires. Large, unstructured inputs dilute attention, bury the instructions that matter, and make reasoning less reliable. The model sees more and understands less.
Reliability comes from selecting and structuring the right information at the right time. To achieve this, context engineering manages three dimensions of an agent’s working memory:
- Content: which instructions, goals, constraints, prior steps, tool results, and retrieved data to include, and how much of each, so the context stays focused and the agent doesn’t drift.
- Structure: how that information is organized, separated, summarized, and formatted so the model interprets it correctly.
- Placement: the order information appears in, so critical details land where the model attends most, surfaced right before the next reasoning step or tool call.
The result is a deliberate, predictable working memory instead of a loose pile of past messages, and that distinction is what separates context engineering from prompt engineering.
Prompt engineering optimizes what you write. Context engineering optimizes what the agent sees, and when it sees it.
Prompt engineering | Context engineering |
|---|---|
Sharpens the wording of a single prompt | Designs the whole information environment the agent works in |
Enough for simple, one-shot tasks | Needed once an agent plans, calls tools, and carries state |
Optimizes what you write | Optimizes what the agent sees, and when it sees it |
Three Mechanical Constraints Explain Why Agents Need Context Engineering
To see why context must be engineered rather than accumulated, look at the mechanical limits of how an LLM processes tokens. Three of those limits explain most of the failures we see in long-running, tool-using enterprise agents:
- Context rot: reliability decays as the window fills.
- Instruction positioning: where a rule sits within the context window helps determine whether it’s followed.
- Cached tokens: a stable prompt prefix can be reused to cut cost and latency.
Each limit maps to a failure mode that can hit enterprise deployments hard: drift, ignored constraints, inconsistent behavior. The next three sections take each one in turn.
Context Rot: Why More Tokens Can Weaken the Model
LLMs don’t process every token with equal clarity. As context grows, especially past a few thousand tokens, the model gets worse at separating signal from noise. This degradation is called context rot, and it can set in well before the context window is anywhere near full.
Context rot is driven by the volume and quality of information a user includes. Too many details, too many distractors, or too much irrelevant content all make it harder for the model to apply the instructions that actually matter for the output.
Figure 2. Context rot: model reliability starts to decay well before the context window is full.
Why context rot happens
LLMs spread their attention across every token in the prompt, and as you add content, several effects compound.
Cause | What happens | Why it matters |
|---|---|---|
Attention spreads thin | Every extra token is one more place the model has to look, and testing shows retrieval precision slipping as context grows. | The instructions that matter start competing with everything else in the window. |
Noise reads as signal | Logs, unused tool output, and stray examples all get treated as meaningful; even a context full of genuinely relevant material degrades reasoning. | The agent chases the wrong details or leans on stale information. |
Length breaks reasoning | Retrieval can be perfect, but multi-step reasoning still falls apart once the input runs long, from planning to classification. | Output turns brittle, and hallucinations creep in. |
How C3 Generative AI Manages Context Rot Automatically
C3 Generative AI counters context rot by compacting the agent’s working memory on every step instead of replaying a growing transcript, through three mechanisms. Summarization condenses verbose transcripts into small, structured state objects that keep the key constraints and decisions and drop the noise. Pruning removes stale logs and redundant tool outputs so they can’t dilute the model’s attention. Dynamic rebuilding assembles a minimal, fresh context for each step from the current query and the concise state summary. Together, they hold the agent’s adherence to business rules steady across long workflows while keeping cost down.
The effect shows up clearly in a single long-running session. We ran a dynamic agent through six increasingly demanding queries, among them polynomial regression, solar insolation modeling, a generalized Monty Hall simulation, and decagon geometry, each adding more code, output, and explanation than the last. Left unmanaged, that history would have pushed the context past 15,000–20,000 tokens and kept climbing. Compaction and summarization instead held it between roughly 6,000 and 8,600 tokens for the whole session, with each compaction step recovering between about 100 and 2,600 tokens. Summarization did the heaviest lifting, collapsing whole prior steps into a single summary, and the agent kept working without ever approaching a context ceiling.
Instruction Positioning: Where You Place Instructions Shapes Model Behavior
LLMs do not weight every part of the prompt equally. Some regions of the context window get more attention; others are routinely down-weighted or ignored. Where an instruction appears in the context window is often as important as how it’s written.
This behavior, known as instruction positioning, follows a consistent pattern: models favor recent tokens, under-weight content in the middle of the prompt, and lean heavily on the patterns in nearby examples. Poor placement makes the model ignore a constraint, copy the wrong pattern, or drift from its intended behavior.
Figure 3. The lost-in-the-middle effect: instruction-following peaks at the start and end of the prompt and dips in the middle.
Why instruction positioning matters
Figure 3 captures this pattern: a model pays most attention to the very start and very end of its prompt, and least to the middle. Two rules follow. Anchor the agent’s role, goals, safety and compliance constraints, and other elements that must always hold, at the top, where they set behavior from the first token. Rules dropped into the middle are the ones most likely to get missed. Because the model also leans on whatever is most recent and closest to the point of action, step-critical instructions belong near the query and should be re-stated at each decision point.
Examples are the exception. A demonstration steers behavior even against the written rules, so place examples after the rules and use only a few.
How C3 AI Anchors Business Rules Where Models Attend
Enterprise agents fail when a critical rule gets lost in a long conversation. The C3 Agentic AI Platform applies the positioning research directly, through automated prompt construction rather than manual tuning.
Prompt templates place core business rules, agent roles, and compliance constraints at the start of every context, where they anchor the model’s behavior from the first token. Critical instructions are re-injected at decision points, right before the agent acts, through callbacks and templates working together, so a rule is present when it matters most. Reusable skills (pre-built instruction modules that encapsulate domain expertise) load on demand, putting the right guidance in place without bloating the context. Examples and demonstrations are placed after the rules, through built-in memory management, which limits format drift while preserving instruction-following across turns. Because these strategies are built into the framework, teams get them by default.
Figure 4. A positioning-aware prompt layout: core rules and general instructions first, then a single general example, then few-shot examples, with action-specific guidance placed near the user message. Recall is highest at the top and bottom of the prompt.
Cached Tokens: Reusing a Stable Prompt Prefix for Speed and Cost
Normally an LLM re-processes every token of the prompt on every request. Cached tokens change that: providers reuse the model’s internal key-value (KV) cache, the intermediate attention state, for the unchanged portion of the context, so the model no longer re-reads or re-computes those tokens. That lowers cost and speeds up responses.
For long-running agents with stable system prompts, fixed role descriptions, or repeated state, caching can cut both latency and price. You pay full price only for new tokens; cached tokens cost a fraction of that, depending on the provider.
How caching works
When part of a prompt stays the same across requests, the model reuses its stored attention representations for those tokens instead of recomputing them. Providers call this KV caching, prompt caching, or prefill reuse, and the cache behaves like a tree of token prefixes: the longer the identical prefix, the more you save.
Figure 5. How prompt caching works. The stable prefix is tokenized once; its intermediate model state is stored and reused across calls, so only new tokens (red and green) are recomputed.
That makes caching most valuable when a large, stable block sits at the front of the prompt, including:
- Long system prompts
- Agent identity and role blocks
- Long-term goals and safety sections
- Repeated schemas, formats, or instructions
- Multi-step workflows with a large, fixed prefix
Figure 6. The payoff from caching a stable prefix: cached input runs 80–90% cheaper, time-to-first-token drops by up to 87%, and inter-token latency falls 29–69%.
How C3 AI Keeps the Cache Warm with Composable Prompt Templates
Caching rewards a particular prompt structure: the more of the context that stays identical across steps, the more you save. The C3 Agentic AI Platform is built to keep that stable region large and verifiable.
Composable prompt templates lock in the stable prefix (system prompt, roles, rules), so the core context stays byte-identical across calls and the cache isn’t invalidated. Summaries and state are appended rather than rewritten, since editing the middle of a prompt forces an expensive full re-prefill. Observability tools trace token flow so engineers can confirm that prompt sections are ordered correctly and that the cache is being hit in production. The dynamic, uncached portion (the user’s current query) is kept small, so each call stays fast and cheap.
Five Primitives Compose a Context-Engineered Agent
The mechanics above, context rot, instruction positioning, and cached tokens, are constraints. Building around them requires a small set of primitives shared by every robust agent architecture:
- Tools: functions the agent calls to act on the world: query a database, run a calculation, invoke an API.
- Prompts and prompt templates: the structured instructions that carry the agent’s role, rules, and goal, assembled fresh for each step.
- Callbacks: hooks that inject or modify context at specific decision points, so the right rule is present right before the agent acts.
- External storage: durable memory that lives outside the context window (past interactions, long-term state), retrieved only when relevant.
- Application state: the compact, structured record of the current task (goals, constraints, results so far) that stands in for a raw chat history.
A few things become clear once these mechanics and primitives sit together. More context does not equal greater reliability: past a few thousand tokens, added content divides the model’s attention and degrades reasoning, even when every token is relevant and retrieval is perfect. Position rivals wording, too, because models attend most to the start and end of a prompt and lose information in the middle, so where a rule sits helps decide whether it’s followed. Examples carry their own weight here: they steer behavior so strongly that too many of them, or any placed before the rules, lead the model to copy patterns instead of following instructions.
Those constraints also point to where the leverage is. A stable prefix is a cost lever, since keeping the system prompt and rules byte-identical across calls lets providers reuse the KV cache, cutting time-to-first-token by up to 87% in published benchmarks. Underlying all of it, working memory has to be engineered rather than accumulated. Reliable agents rebuild a compact, deliberate context for each step instead of letting a chat history grow unchecked.
Why C3 AI Builds Context Engineering into the C3 Agentic AI Platform
These constraints come from how transformers process tokens, so they show up in every long-running agent, regardless of whatever frontier model sits underneath. C3 AI treats context engineering as shared platform infrastructure for exactly that reason. An agent scheduling maintenance across a utility’s generating fleet, screening transactions for a bank’s compliance team, or planning output for a manufacturer runs into the same three failure modes, and benefits from the same disciplined handling of its working memory.
If you’re building agents that must hold up under real production load, these mechanics are where reliability starts.
Ready to see what production-grade enterprise AI agents can do for your organization? Schedule a demo with our team.
About the Authors
Ivan Robles is a Lead Data Scientist on the Data Science team at C3 AI, where he develops machine learning and optimization solutions across a variety of industries. He has a record of AI Kaggle competitions, where he ranked in the top 1.5% globally. He received his Master of Science in Advanced Chemical Engineering with Process Systems Engineering from Imperial College London.
Romain Juban is a Vice President of Data Science at C3 AI, where he leads the data science development and implementation of Generative AI applications. His work focuses on leveraging cutting-edge technologies such as large language models, embedders, retrieval augmented generation, agentic frameworks, and fine-tuning. He received his Master’s Degree in Civil and Environmental Engineering from Stanford University and his Bachelor’s Degree in Mathematics and Computer Science from Ecole Polytechnique, France.