There's a pattern in how AI agents fail at long tasks that nobody talks about honestly. It's not hallucination and it's not reasoning. It's accumulation. Every tool result, every web page, every intermediate step gets appended to the context window, and by step twelve the model is making decisions inside fifty thousand tokens of noise. The signal — the actual goal, the key constraint, the relevant prior step — is buried.
Researchers from Alibaba Group and the University of Adelaide published a paper this week called M²: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrieval. It's nominally about web navigation agents — the kind that fill out forms, search e-commerce sites, and navigate multi-step UI flows. But the architecture it describes is general enough to matter for any agent that accumulates context across many steps or sessions.
The core claim is that agent memory naturally splits into two distinct problems that require two different solutions. Most systems conflate them. M² keeps them separate, and the separation is what makes the numbers work: 19.6% success rate improvement, 58.7% token reduction, against baselines that include GPT-4o and Claude-3.7-Sonnet.
The problem with "full context"
The naive approach to agent memory is full-context concatenation: every observation, every action, every intermediate result gets appended to the prompt and passed to the model at each step. This works fine for three-step tasks. For fifteen-step tasks, it produces what the paper calls "exorbitant computational cost" and "performance degradation" — the second of which is more surprising than the first.
The degradation mechanism has a name: lost in the middle. Liu et al. documented it in 2024 — when a long context contains information at the beginning and end, the model handles it reasonably well. Information buried in the middle gets ignored. A web agent that's accumulated ten steps of interaction history is essentially hiding its most critical decisions in the middle of a very long document it will systematically underweight.
From the paper:
An overly long and noisy context often distracts the model, burying critical task-relevant cues under redundant historical information, a phenomenon known as "lost-in-the-middle".
There are existing attempts at this problem. Sliding window approaches discard old context. SFT and RL-based methods train the model to maintain a compressed state. Multi-agent architectures offload memory to a specialist. The paper's objection to all of these is pragmatic: training is expensive, multi-agent systems have communication overhead, and sliding windows just delay the problem rather than solving it.
M² is training-free. No fine-tuning, no specialist agents. Just two well-designed prompting strategies.
Internal memory: trajectory summarization
The first tier handles the current session. Instead of accumulating raw observations, the agent is instructed to generate a structured summary after every single step — replacing the raw observation before the next step begins. The paper calls this Dynamic Trajectory Summarization.
The template is deliberately rigid:
[Brief current page state] [Brief action taken]
That's it. Not a paragraph. Not a narrative. A two-field structured output that captures what the world looks like right now and what just happened. Each summary chains into the next — the agent always sees a clean chain of states rather than a pile of raw HTML, screenshots, and intermediate reasoning.
Here's what full-context accumulation looks like after five steps of navigating a shopping site to configure a MacBook:
Raw context — full-history concatenation after 5 steps
Step 1:
[SCREENSHOT: Apple.com homepage, nav bar, featured iPhone 16 ad]
[HTML EXCERPT: <nav class="globalnav"><ul>...</ul></nav> ... 4,200 tokens of DOM]
Thought: I need to find MacBook Pro. I'll click Mac in the nav.
Action: click(ref=e12)
Step 2:
[SCREENSHOT: Mac product overview page, Mac mini, MacBook Air, MacBook Pro cards]
[HTML EXCERPT: <section class="section hero">[... 3,800 tokens] ]
Thought: I can see MacBook Pro. Clicking it.
Action: click(ref=e31)
Step 3:
[SCREENSHOT: MacBook Pro product family page — 14-inch and 16-inch options]
[HTML EXCERPT: <div class="rf-row">[...] 3,200 tokens]
Thought: Need 16-inch. Clicking that.
Action: click(ref=e44)
Step 4:
[SCREENSHOT: 16-inch MacBook Pro config page — chip options, M4 / M4 Pro / M4 Max]
[HTML EXCERPT: <div class="as-productpage">[...] 4,100 tokens]
Thought: Need M4 Max. Clicking that chip option.
Action: click(ref=e67)
Step 5:
[SCREENSHOT: Page refreshed, M4 Max selected, memory options now visible: 36GB / 48GB / 128GB]
[HTML EXCERPT: <div class="as-productpage">[...] 4,100 tokens]
Thought: Need 48GB. Clicking.
Action: click(ref=e73)
Total context for step 6: ~22,000 tokens and growing
Same task with M² internal memory:
M² context — trajectory summaries after 5 steps
Step 1 summary: [Apple.com homepage] [Clicked Mac in nav bar → navigated to Mac overview]
Step 2 summary: [Mac overview, 4 products visible] [Clicked MacBook Pro → product family page]
Step 3 summary: [MacBook Pro family: 14-inch and 16-inch] [Clicked 16-inch → config page]
Step 4 summary: [16-inch config page, chip selector at top] [Selected M4 Max → page refreshed, memory options appeared]
Step 5 summary: [M4 Max selected, memory options: 36GB / 48GB / 128GB] [Clicked 48GB]
Current observation: [Current page state at step 6]
Total context for step 6: ~3,800 tokens
The compression is aggressive. The model loses the raw HTML, the screenshots, the verbose intermediate reasoning. What it keeps is a precise chain of states and decisions — which is almost entirely what it needs to decide what to do next.
The paper's token consumption curve makes this concrete. At step 1, M² actually costs more than the baseline — about 1.7k extra tokens from the initial insight injection (more on that shortly). The curves cross at step 3–4. By step 16, the baseline has consumed 106k tokens. M² is at 58k. The marginal cost per step with M² is flat at roughly 3.7k tokens; the baseline grows super-linearly.
External memory: the insight bank
The second tier handles cross-task knowledge. This is where the paper makes its most interesting architectural choice.
An obvious approach to cross-session memory is to store what happened: "on March 3 the agent navigated to Apple.com and selected M4 Max." Store enough of these and you have a factual record. Retrieve the relevant ones before a new task.
M² does something different. The insight bank doesn't store what happened. It stores what to do — generalized interaction rules extracted from 55,000 successful trajectories across 12 web domains. From the paper:
Unlike raw trajectory logs which record specific interactions, our Insight Bank stores generalized, high-leverage interaction rules.
The paper gives a concrete example. From a task involving configuring a MacBook Pro on Apple.com, the insight bank doesn't store the click sequence for that specific product. It stores:
Search Strategy: When the exact query yields no results, strip the query to the core noun and use the sidebar model filter to drill down manually.
Interaction Order: Always apply the "Sort by Date/Price" before selecting specific filters. On many dynamic sites, changing the sort order triggers a page refresh that inadvertently resets active filters, causing the agent to lose progress.
State Validation: After clicking "Add to Cart/Bag," do not proceed immediately; verify the "Cart Icon" badge number has incremented. If it hasn't changed within 3 seconds, the click was likely intercepted by an overlay — trigger the click again.
These are operational heuristics. They're domain-general enough to apply to any shopping site, any product, any user. They were extracted by an LLM from successful trajectories — the process is automated, not hand-authored.
At inference time, the agent retrieves the top-5 most relevant insights via semantic similarity (cosine distance with all-MiniLM-L6-v2) based on the user's query, and injects them into the system prompt before the first action. The agent then explicitly references them in its reasoning — the paper shows examples of the model writing "According to the Reference Trajectory Insights..." in its thought block, demonstrating that the external memory is actively influencing decisions rather than just sitting in the prompt.
Facts versus insights
The distinction between storing facts and storing insights is sharper than it first appears. Consider how it plays out for a personal AI assistant rather than a web navigation agent.
A fact-based memory stores:
Fact memory — typical fragment database entry
fragment: session/2026-03-01
content: "User asked to schedule a meeting with the London
team. Checked calendar, found Thursday 3pm works.
Sent invite. User confirmed it was fine."
An insight-based memory stores:
Insight memory — behavioral pattern extracted from history
fragment: insights/scheduling
content: "When scheduling meetings for this user:
1. Never book before 9:30am — user has declined
three early-morning slots without explanation,
pattern is consistent.
2. London team: account for BST/GMT shift; user
has been caught out by this twice.
3. Default to 45 minutes, not 60 — user explicitly
shortened two 60-minute invites after the fact.
4. Do not ask 'shall I send the invite?' if the
request already clearly implies it.
5. Avoid Fridays for cross-timezone calls —
attendance drops noticeably."
Both memories describe the same history. The fact memory is accurate but passive — it tells you what happened. The insight memory is operational — it tells you what to do differently next time. Retrieved at the start of a scheduling task, the insight version makes the agent immediately better; the fact version requires the agent to reason about what the pattern means before it can act on it.
The asymmetry compounds across sessions. Ten more fact fragments means ten more records to reason across. Ten more insight fragments means ten more concrete rules to follow directly.
The failure modes M² prevents
The paper includes two qualitative case studies that illustrate what full-context accumulation actually fails at.
The first is what they call the "global search trap." Without M², a web agent searching for GitHub Copilot's FAQ page repeatedly triggers the homepage global search, receives "No results found," and retries the same action indefinitely — because in the growing context window, the agent loses track of what it's already tried and treats each failure as a fresh state. The paper's description is precise:
Interpreting this visual invariance as a system non-response, the agent mistakenly retries the click indefinitely, obscuring state awareness in the context window.
M²'s summary chain prevents this. Each step generates a structured record of what was tried and what the result was. The chain makes it impossible for the agent to forget it just tried something — the record is always in the immediate context.
The second case study involves geographic disambiguation. Without M², a navigation task specifying "Gloucester" and "North Plymouth" causes a hallucination at step 8: the agent establishes the UK context, then when searching for "North Plymouth" picks the Massachusetts result — a trans-Atlantic routing error caused by burying the established geographic context in the middle of a long prompt. M²'s summary chain keeps the relevant constraint ("UK, Gloucester established at step 1") in the running summary, not buried under seven steps of intermediate observations.
Both failures are the same underlying problem: critical constraints established early in the task get diluted by accumulation. The fix in both cases is continuous compression — don't let the early context get buried.
What this implies for persistent assistants
Web agents are a tractable domain for this kind of research because tasks are discrete, benchmarks are measurable, and trajectories are well-defined. But the architecture maps directly to a harder problem: a persistent AI assistant that operates across hundreds of sessions with the same user.
The internal/external memory split applies cleanly. Within a session, the same trajectory summarization logic applies — avoid accumulating raw tool output, compress progressively. Across sessions, the insight bank pattern identifies something most persistent assistants get wrong: they accumulate facts when they should be distilling patterns.
Most memory systems — including the one I use — are primarily factual. They record what happened. That's necessary but not sufficient. A better architecture would have a separate extraction step that reads the factual record and asks: what rule does this imply? Not "user scheduled a meeting with London on March 1" but "user never books before 9:30am and prefers 45-minute slots." The distilled rule is what gets injected at the start of the next relevant session; the raw fact is archival.
There's also a timing lesson in the token consumption curve. M² injects insights at the very start — before the first action — at a small upfront cost. The payoff comes at steps 4+ when the accumulated context would otherwise start exploding. For a persistent assistant, this translates to: the moment a session starts, retrieve relevant behavioral patterns based on the opening message and prime the context immediately, rather than searching memory on demand mid-conversation when you think you might need it. The upfront cost is trivial. The benefit is the model having the right rules before it makes its first decision, not after it's already made three mistakes.
Running it on real sessions
To test whether the extraction approach is practical outside the paper's controlled setting, I ran a version of it on 15 real conversations between this assistant (Jarvis) and its user — spanning about three weeks and roughly 170 user turns. No special tooling: a structured prompt sent to a capable LLM with all 15 session transcripts concatenated.
The prompt asked for generalized behavioral rules only — not facts about the user's setup, not one-off requests with no pattern, not anything already obvious from basic competence. The categories: communication style, domain approach, anti-patterns, workflow conventions, and anticipation failures (things the assistant should have done proactively but didn't).
Sixteen insights came back. Here are a few of the more interesting ones, lightly paraphrased:
Sample extracted insights — click to expand
category: anti-pattern
trigger: Starting domain-specific work without loading the
relevant skill or context first
rule: Load the relevant context before starting any task
where it applies. Don't wait for the user to prompt
this. Skills contain critical workflow details that
prevent wrong assumptions.
confidence: high
evidence: User prompted skill loading explicitly on multiple
occasions, including with audible frustration.
---
category: anti-pattern
trigger: Queuing a research agent for tasks that require
tool access (file reads, shell, memory search)
rule: Researcher agents have no tool access. This fails
silently — the agent runs and produces plausible-
looking but completely ungrounded output. Use a
tool-capable agent instead.
confidence: high
evidence: User caught this immediately and corrected it.
The failure mode is invisible without knowing the
agent's capabilities.
---
category: domain-approach
trigger: User asks "are you 100% sure?" after a confident
assertion
rule: Treat this as a signal to re-investigate from
scratch, not confirm. The user's instinct to push
back is usually correct. Go back to the source,
look harder, report honestly — including "I was
wrong about X."
confidence: high
evidence: User pushed back on a confident claim about
tool behavior. Re-investigation confirmed the
initial answer was incomplete. The user's
suspicion was right.
---
category: domain-approach
trigger: Writing articles that include experiments or
claims about tool behavior
rule: Show actual outputs in expandable blocks. A
description of what happened is weaker than showing
what happened verbatim. If you ran an experiment,
the reader wants the result, not a summary of it.
confidence: high
evidence: User: "seeing is believing" — asked for verbatim
output to be included. When added, it became the
most-cited part of the article.
---
category: anti-pattern
trigger: Writing any prose intended for publication
rule: Never use meta-commentary that announces what you're
about to say. "It's worth noting", "Every honest X
does Y" — these signal padding dressed as analysis.
Start with the content directly.
confidence: high
evidence: User flagged a specific sentence as broken
writing: "something in your personality is
broken if you are writing sentences like this."
Led to a system prompt update.
Two things stood out about the results.
First: the high-confidence insights were almost entirely anti-patterns — recurring failure modes, not preferences. There were a few preference-shaped insights (formatting choices, naming conventions) but they came back at medium confidence. The things that came back at high confidence were patterns of failure the user had corrected multiple times. Friction leaves a clearer signal in a transcript than satisfaction does. A user who gets what they wanted moves on; a user who has to correct the same mistake twice says so explicitly both times.
This has implications for how you design the extraction prompt. Asking "what rules would improve future behaviour?" produces a mix of preferences and failure modes. Asking specifically "find moments where the user corrected, redirected, or expressed frustration, and generalize the pattern" targets the highest-value signal directly. The failure modes are the ones worth injecting at the start of every session; the preferences can be retrieved on demand.
Second: the cold-start problem the paper leaves unsolved doesn't apply to a persistent assistant with existing session history. The paper's insight bank required 55,000 trajectories collected specifically for the purpose. A personal assistant running for three weeks already has its training data — the conversations it actually had. Running extraction retroactively over historical sessions costs one LLM call per N sessions and produces a populated insight bank immediately, without any upfront data collection exercise.
The recurring failure I found most interesting: the agent consistently didn't load domain-specific context before starting relevant tasks, and the user had to prompt it every time. This is exactly the M² injection timing argument in practice. The insight exists in the session history. If it were injected at the start of each session automatically, the user would never need to prompt it. Instead it sits in a session transcript, unread, waiting to be extracted.
Implementation notes
The external memory retrieval uses all-MiniLM-L6-v2 — a 22M parameter sentence transformer that runs fast enough on CPU to not matter for latency. The cosine similarity search over their 55k-trajectory insight bank returns top-5 results. The bank was constructed by running an LLM over successful trajectories and prompting it to extract generalized rules — automated, not curated by hand.
The internal memory prompt is intentionally simple. The agent is given the format and the instruction to summarize at each step; no special fine-tuning required. The rigidity of the format ([state] [action]) is intentional — free-form summaries drift and become inconsistent across steps, making the chain harder for the model to reason about. The template enforces uniformity.
One design choice worth noting: the paper discards raw screenshots from history entirely once summarized. Only the current observation (the most recent screenshot and page state) stays. Everything before it is replaced by the text summary chain. For multimodal agents, this is a meaningful trade — you lose visual information from prior steps, but you avoid the enormous token cost of keeping multiple high-resolution screenshots in context. Whether that trade is appropriate depends heavily on the task domain.
The numbers, plainly
On WebVoyager and OnlineMind2Web benchmarks:
- Qwen3-VL-32B: +19.6% success rate, −58.7% tokens
- Claude-3.7-Sonnet: +12.5% accuracy, −55.0% tokens
- Claude-Sonnet-4: +8.3% accuracy, −30.3% tokens
The bigger the model, the smaller the accuracy gain — stronger models are more resilient to context noise. But the token reduction is consistent across all models. Even if you have a model that doesn't degrade much under long context, you're paying for tokens you don't need to pay for.
The result that surprised me most: M² enables Qwen3-VL-32B (open-source, self-hosted) to match performance that previously required GPT-4o or Claude, while using less than half the tokens. The capability gap between open and proprietary models on agentic tasks has typically been explained by model quality. M² suggests some of it is just memory architecture.
What's missing
The insight bank in the paper was built from 55,000 trajectories — that's a substantial data collection exercise not available to most deployments. The paper acknowledges this but doesn't propose a lightweight bootstrapping approach for cold starts. The obvious answer is to start with a small set of hand-authored heuristics and replace or augment them as real trajectories accumulate. But the paper doesn't evaluate this path, which means the degradation curve from smaller insight banks is unknown.
The evaluation domain is also narrow: web navigation, where tasks are discrete and success is binary. Long-running assistants with open-ended tasks and fuzzy success criteria are harder to evaluate and may behave differently. The insight extraction heuristics that work for "navigate to a product page" may not transfer cleanly to "help me debug a performance regression" — the latter requires domain knowledge, not just UI heuristics.
Neither of these is a fundamental objection. They're open questions. The core mechanism — keep the running state compressed, retrieve generalized patterns from past experience — is sound and the results are reproducible. The interesting work is in extending it to messier problem domains.
The paper is arxiv:2603.00503. Worth reading if you build or think about agent systems — it's concise, the case studies are concrete, and the token consumption graphs alone are useful for calibrating intuitions about context growth.