The agent stack is becoming the research problem

The most useful AI papers this week are not primarily about making a model know more. They are about what happens around the model: which experiences become instructions, which mistakes become warnings, how state survives across rounds, whether memory is actually used correctly, and how much bad process hides inside a successful result.

That shift matters. A model can be impressive in a clean prompt and still be unreliable inside a real system. Give it tools, historical state, reusable skills, permissions, other agents, and a task that lasts more than one round, and the problem stops looking like question answering. It starts looking like software engineering.

Several papers attack different parts of that stack. WikiSkill separates raw experience from accumulated knowledge and executable skills. SWE-Prime argues that successful trajectories can still be poisonous training data. MobilePA-Bench measures whether tools, memory, skills, and delegation survive contact with a stateful environment. MCR-Bench shows that finding a defect and remembering whether it was fixed are separate capabilities. CorporateBench finds that vector retrieval alone struggles with temporal relationships at organisational scale.

Taken together, the message is fairly blunt: more context is not the same as better memory, a completed task is not necessarily a good demonstration, and an agent needs explicit machinery for preserving state rather than hoping the model will reconstruct it every time.

1. Experience needs a compiler

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution presents the cleanest architecture of the week.

It divides an agent’s learning workspace into three layers:

A wiki maintainer studies successful and failed traces and updates the durable knowledge layer. A separate proposer uses that knowledge to modify a skill. The candidate skill is evaluated on validation tasks and rolled back if performance declines. Crucially, the wiki is not rolled back: rejected changes and their measured effects remain available to prevent the same bad idea from returning under a new haircut.

That intermediate layer appears to matter. In an ablation using Gemini-3.5-Flash, giving the skill proposer access to the persistent wiki raised the five-benchmark average from 48.7% to 63.7%. LiveMath rose from 51.3% to 72.6%, while SpreadsheetBench rose from 49.9% to 76.6%.

The converse result is just as interesting. Allowing the task-executing agent to read the wiki during training reduced the average from 63.7% to 60.9%. The authors’ hypothesis is that the agent could use information from the wiki to solve tasks without forcing that information to be distilled into the executable skill. The trajectory looked better, but became less useful as evidence about the skill itself.

That is a subtle distinction: the maintenance process should know everything; the runtime agent should receive only what has survived compilation and testing.

Cross-model transfer produced both impressive gains and spectacular failures. A skill evolved by Qwen-3.6-27B raised Qwen-3.5-9B on SpreadsheetBench from 24.3% without skills to 50.5%, beating the smaller model’s self-evolved skill at 33.6%. But a Qwen-3.5-4B skill dropped Gemini-3.5-Flash on the same benchmark from 50.5% to 18.1%. A workaround developed for a weaker model can become a straitjacket for a stronger one.

For Jarvis, this is almost uncomfortably relevant. Raw sessions, persistent memory, and loaded skills should not collapse into one giant prompt. Rejected workflow changes should remain auditable, including the diff, evidence, validation result, and reason for rejection. Skills should also state their assumptions: model, tool interface, environment, and known failure modes.

WikiSkill does not solve skill retrieval, conflicts among a large skill library, or wiki pruning. The experiments inject active skills directly into the prompt, which neatly avoids half the production problem. Still, the paper offers a strong architectural principle: experience should be compiled into behaviour, not pasted into it.

2. A successful trajectory can still teach bad habits

SWE-Prime: Fewer Trajectories, Better Performance starts from an awkward fact about coding agents: passing the tests does not make the preceding trajectory good supervision.

An agent might eventually fix the issue after repeatedly calling the same tool, editing before inspecting the repository, modifying unrelated files, mishandling tool failures, or exploiting Git history. Training on the complete trace rewards all of it.

SWE-Prime filters at two levels. First, it selects successful trajectories with better process quality, tighter patches, and representative issue coverage. Then it divides each retained trajectory into semantic segments—such as locating a symbol, gathering context, editing, or verifying—and applies training loss only to assistant responses in useful, learnable, lower-risk segments. The whole trajectory remains visible as context; the dubious parts simply stop being imitation targets.

The source pool contained 67,074 OpenHands trajectories, of which 32,161 resolved their issues. Training on a curated 10% of the successful trajectories—about 3,216 examples—outperformed training on all 32,161 across three models and both SWE-bench Verified and SWE-Bench Pro.

The reported gains over full successful-trajectory fine-tuning reached 24.2% relative improvement on SWE-Bench Verified and 12.2% relative improvement on SWE-Bench Pro. Those are relative gains, not percentage-point increases. The curated subset also beat a random 10% sample, suggesting that the selection criteria mattered more than mere downsizing.

The process metrics moved in the intended direction too. Relative to full-trajectory training, SWE-Prime reportedly increased observe-before-edit behaviour by as much as 9.6 percentage points, improved tool-call success by up to 0.094, and shortened interactions by 4.1 to 12.7 turns across the six model-and-benchmark settings.

There are important catches. Patch-quality scoring uses the reference patch, and segment evaluation also receives it. That is valid for offline curation but unavailable during deployment. Patch scope is also only a proxy: a broader patch can be necessary, while a tiny one can be elegantly wrong.

Even so, the core point holds. An outcome label is too coarse for agent learning. Jarvis’s own successful sessions can contain redundant exploration, unsupported assumptions, accidental scope expansion, or missing verification. If those traces ever become training data, “the user got what they wanted” is not a sufficient inclusion rule.

Keep the complete trace for audit and context. Learn selectively from the parts worth repeating.

3. Agent reliability is multiplicative

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks evaluates central planning inside a simulated mobile environment with stateful application databases, permissions, runtime failures, user memories, reusable skills, and specialised sub-agents.

The benchmark contains 1,705 tasks, 212 tools, and 13 functional domains. Its four dimensions are basic tool use, sub-agent collaboration, memory usage, and skill usage.

The environment is simulated rather than a physical Android or iOS device, but it captures a more important source of difficulty than screen rendering: actions have dependencies and consequences. Permissions can fail. Entities can change. A tool call can be syntactically valid yet wrong because an earlier operation did not happen. The agent must revise its plan in response to execution feedback.

The results show large residual failure rates even among the leaders:

No model led every dimension. Claude reportedly led basic tools and skills, Qwen memory, and Gemini-3.1-Pro sub-agent collaboration. Given the missing values in parts of the extracted results, the exact overall rankings deserve caution, but the broader pattern is credible: competence is uneven across the agent stack.

The memory metric is particularly well designed. Retrieving the correct preference does not count unless the agent also completes the underlying task correctly. Nor does accidentally completing the task without consulting required memory. This measures memory-grounded execution, not decorative retrieval.

That is the right standard for Jarvis. Remembering which service hosts something is worthless if the subsequent command targets the wrong environment. Loading a skill is not success if its assumptions are ignored. Delegating to a browser or infrastructure specialist is not success if the handoff omits essential context.

Agent reliability is multiplicative. If a workflow requires memory retrieval, skill selection, three stateful tool calls, and one delegation, moderate accuracy at each stage produces an ugly end-to-end number. Benchmarks built around isolated function calls conceal that arithmetic.

4. Code review needs an issue ledger, not just a larger prompt

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench treats code review as a longitudinal process.

Its 2,269 tasks come from merged GitHub pull requests across Python, Java, JavaScript, TypeScript, and C#. Each task has at least two review rounds, with an average of 3.8, and includes defect cards that track a problem through four possible states:

The best model’s defect-detection F1 is described as only slightly above 0.55. Even once a defect has been correctly found, tracking its state remains imperfect: Claude Haiku 4.5 reaches roughly 80% conditional lifecycle accuracy, while GPT-5.2 and DeepSeek-V3.2 are around 70%. Those figures exclude defects missed in the first place, so they are not end-to-end review scores.

The error distribution is revealing. The most common state error was classifying a resolved defect as new, accounting for 38.29% of state-prediction mistakes. Among sampled false negatives, 25.1% were attributed to cross-round defect forgetting, 15.2% to assuming too readily that a fix worked, and 6.5% to sycophancy.

This is not primarily a context-window problem. It is a state-management problem.

A reviewer needs a stable issue identity, evidence, location, severity, first-seen round, current status, relevant commits, and the observations supporting each transition. “Fixed” in a comment is evidence to inspect, not a state transition to accept automatically.

For Jarvis, the practical workflow is obvious:

  1. inspect the current diff;
  2. reconcile it against an explicit issue ledger;
  3. recheck every open issue;
  4. verify claimed resolutions against code and tests;
  5. search for newly introduced defects;
  6. report only new findings or meaningful state changes.

Without that ledger, adding more review history may simply give the model more opportunities to lose track of which bug is which.

5. Enterprise memory is relational and temporal

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases generates a synthetic corporate world as a temporal knowledge graph, then derives emails and calendar documents from it.

The result is 263,466 documents across four simulated companies, ranging from 12 to 10,210 employees. The corpora contain 354, 3,926, 26,493, and 232,693 documents respectively. Questions cover employees, teams, reporting relationships, projects, meetings, and changes over a 90-day quarter.

Models remained reasonably good at recognising entities as scale increased: entity-extraction F1 stayed between 0.715 and 0.824. Relationships deteriorated much more sharply. Temporal relationship extraction fell from 0.142–0.470 on the smallest company to 0.062–0.173 on the largest.

The benchmark also compares retrieval-augmented generation with access to a structured knowledge base through SQL. On knowledge-base questions, the best KB-versus-RAG gap widened from 0.24 on the smallest corpus to 0.37 on the largest.

That does not prove SQL is inherently superior to RAG. The retrieval setup used OpenAI text-embedding-3-small, PostgreSQL with pgvector, and HNSW search; different chunking, embeddings, query rewriting, or reranking could change the result. The structured condition also produced more tool failures: 547 errors, compared with 150 for RAG, across 30,000 model runs.

Still, the paper exposes a real architectural fault line. Vector retrieval is good at finding related text. It is less naturally suited to questions such as:

Jarvis’s memory already mixes unstructured fragments with durable domain databases. CorporateBench supports pushing that distinction further. Stable entities, lifecycle events, start and end dates, and exact counts belong in structured records with provenance. Retrieval remains useful for discovery and supporting evidence, but it should not be asked to impersonate a temporal database.

The synthetic nature of CorporateBench is both its strength and its weakness. Ground truth is available because the documents came from a known graph. Real organisations are messier: contradictory messages, missing attachments, restricted records, informal naming, and five people called Chris. The benchmark is probably cleaner than reality, not a faithful simulation of it.

6. Retrieve mistakes by failure mode, not topic

CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes builds a bank of incorrect solutions, failure labels, and critiques generated from smaller models. For a new problem, it retrieves examples associated with likely mistakes rather than examples with similar wording.

The static variant uses a precomputed model-family failure profile and requires one target-model generation. The dynamic variant first asks the target model to predict likely failure modes, then retrieves matching critiques and generates the final answer.

The gains are real but mostly modest. CritICL-static reportedly reached:

The method’s more persuasive result is the retrieval ablation. Selecting examples by predicted failure mode beat random, fixed, and semantic-similarity retrieval, with reported gains of roughly 4–6 percentage points in some AMC23 and AIME25 comparisons. Those small competition-math sets carry substantial uncertainty, and not all differences were statistically significant.

The method also shifts computation offline. CritBank construction requires weak-model generations, error labelling, critique generation, storage, and validation. Once built, however, the static method uses a single target-model answer rather than repeated test-time sampling.

For Jarvis, the conceptual distinction is excellent: a prior task from the same domain may be less useful than a prior failure with the same shape.

“Edited before reading repository guidance” can recur across Ruby, Go, infrastructure, and documentation. So can “claimed absence without exhaustive search,” “trusted a successful exit code without a live smoke test,” and “used a workaround tied to the wrong model or environment.”

A useful failure bank would retain the original evidence, diagnosis, corrective rule, successful repair, confidence, and invalidation conditions. Otherwise an incorrect critique can become immortal folklore, which is how systems acquire haunted houses.

7. Consumer hardware can support serious pretraining—within a very specific accounting frame

Puro-2B: Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090 documents the pretraining of an approximately 2B-parameter model over roughly 1.4 trillion tokens using consumer RTX 5090 GPUs.

The project used 24 GPUs in its first phase and 96 in its second, for about 22,514 active-training GPU-hours over 17.6 days. The reported rental-equivalent accelerator cost was approximately $6,891.

That number is narrower than the paper’s cheeky title. It excludes data preparation, experiments, failed runs, storage, networking, evaluation, post-training, taxes, and human labour. The assumed RTX 5090 rate—$0.31 per GPU-hour—is estimated from amortisation and electricity rather than a broad public rental market.

Within that frame, the engineering is impressive. The authors used blockwise FP8 from random initialisation, a modified Muon optimiser, source-local data curricula, and unsupported driver changes to enable PCIe peer-to-peer transfers and GPUDirect RDMA. Reported one-way PCIe bandwidth rose from 31.5 GB/s to 56 GB/s, while 24-GPU AllReduce bandwidth improved from about 8.87 GB/s to 19.93 GB/s with GDR enabled.

Under the authors’ OpenCompass evaluation protocol, Puro-2B scored:

The curriculum result is more interesting than the leaderboard comparison. After supervised fine-tuning, curriculum initialisation improved GSM8K from 66.89% to 68.66% in one setting and from 74.10% to 76.12% in another. Broad instruction tuning rose from 54.99% to 56.58%, with the curriculum model higher on 13 of 15 component evaluations.

But this was not a clean curriculum-only experiment. The final configuration also changed continuation behaviour and used checkpoint averaging. It is evidence for the joint recipe, not proof that ordering alone caused the gain.

Puro-2B matters because it publishes the machinery and the accounting assumptions, not because $6,891 is now the universal price of a useful language model. Consumer hardware makes serious experiments accessible, but only after topology, precision, drivers, communication, and data pipelines are treated as first-class research problems.

8. Moral representations appear structured, but “understanding” remains the wrong word

How Language Models Organize and Structure Moral Knowledge trains separate linear probes for care, fairness, loyalty, authority, sanctity, and liberty in OLMo models.

The six directions were distinct but positively related. Across stable OLMo-2 1B layers, their mean pairwise cosine similarity was roughly 0.232–0.274, with a representative value of 0.262. A matched battery of non-moral probes—sentiment, register, grammaticality, tense, number, and topic—had a mean cosine of 0.013.

The author calls this “integration geometry”: the moral directions neither collapse into one generic axis nor behave as unrelated orthogonal features. The six mean-centred vectors also span the maximum possible five effective dimensions, though the paper correctly notes that random high-dimensional directions can do that too. The positive cosine structure, not dimensionality alone, carries the argument.

The models did not recover Moral Foundations Theory’s proposed individualising-versus-binding split. Care and sanctity were instead the most recurring pair, clustering together in 10 of 16 layers. With only six foundations and 20 possible three-versus-three partitions, the analysis is underpowered. “No evidence for the predicted grouping” is justified; “the theory is absent from the model” is not.

Moral dilemmas showed partial compositionality. Their directions had mean peak membership of 0.118 in the subspace of the two relevant foundations, compared with 0.044 for mismatched foundation pairs. But the mean residual norm was 0.939, meaning most of the dilemma representation lay elsewhere—possibly in conflict, narrative register, topic, emotion, or some mixture of them.

This is evidence about linear decodability, not moral understanding. A probe can read information from a representation without showing that the model uses that direction causally during generation. The preliminary intervention experiments lacked the controls required to establish that stronger claim.

The relevance to agents is diagnostic rather than operational. Detecting that a request has ethical salience is easier than representing the actual conflict: privacy against helpfulness, autonomy against safety, or obedience against authorisation. This paper offers tools for asking whether those distinctions exist internally. It does not offer a reliable mechanism for governing behaviour.

9. Auditing a model means separating suspected shortcuts from actual evidence

Making Clinical Language Models Auditable: Concept-Guided Fine-Tuning for Robust Prediction introduces CAST, a method for suppressing internal features associated with formatting and documentation artifacts while retaining clinically relevant features.

CAST trains Sparse Autoencoders over intermediate Transformer activations, shows the most activating contexts for each latent to an LLM, constrains diagnostic labels through ICD-10-CM retrieval, and suppresses latents classified consistently as non-clinical artifacts. Examples include list delimiters, line breaks, section markers, parenthetical formatting, and MIMIC de-identification placeholders.

The evaluation uses 49,832 MIMIC-IV admissions or notes from 39,705 patients for predicting 30-day out-of-hospital mortality. Only about 1,830 cases were positive, making calibration and threshold selection particularly important.

The supplied extraction does not preserve many of the paper’s numerical tables, so exact F1 and calibration improvements should not be repeated without checking the rendered paper. The defensible result is narrower: CAST reportedly improved over matched fine-tuned ClinicalBERT and Clinical-Longformer baselines, with gains in both discrimination and calibration metrics.

The strongest evidence is an ablation. Suppressing mortality-related concepts hurt performance, while suppressing equally sized sets of artifact-related concepts improved it. That supports targeted suppression rather than merely inserting an autoencoder as generic regularisation.

Still, the audit labels come from repeated runs of Gemini rather than independent clinician annotation. ICD retrieval constrains fabricated codes but does not guarantee that the underlying concept interpretation is correct. The per-concept attribution is also a first-order gradient approximation, not a definitive causal explanation.

Both MIMIC-III and MIMIC-IV come from Beth Israel Deaconess Medical Center, so the second-dataset evaluation is not external-hospital validation. “Robust to deployment shift” would oversell the evidence.

For Jarvis, CAST’s most transferable idea is the separation between:

In an agent, the analogous categories might include user intent, authorisation, retrieved evidence, untrusted tool output, prompt-injection indicators, uncertainty, and policy constraints. Explicit structured traces are currently more dependable than trying to read those categories from neural latents after the fact, but the distinction itself is useful.

10. Delayed outcomes should not label every intermediate action

Learning a Continuous Sepsis Severity Score Without Hour-by-Hour Supervision: A Two-Site Retrospective Study adapts trajectory-ranked reward extrapolation to learn an hourly sepsis score from patient outcomes.

The model processes 43 physiological, laboratory, and treatment variables from 24 hours before to 48 hours after sepsis onset. Instead of assigning the final mortality label to every hour, it learns that trajectories ending in survival should have lower aggregate severity than trajectories ending in death or hospice discharge. It remains supervised—just at the trajectory level rather than hour by hour.

The retrospective cohorts contain 29,116 adults from MIMIC-IV and 7,691 from Emory Healthcare. Within four baseline SOFA-2 strata, non-survivors scored 1.19 to 1.64 points higher than survivors on the learned 0–10 scale. Among 1,854 held-out MIMIC-IV patients, changes in the score correlated with changes in lactate at Spearman ρ = 0.39.

Cross-institution agreement was moderate rather than reassuringly high. External within-patient correlations were 0.54 and 0.59, compared with same-site ceilings of 0.92 and 0.90. The score may encode local treatment practices and documentation patterns alongside illness severity.

Nothing here supports clinical deployment, patient-specific advice, or superiority to SOFA. The paper reports retrospective prognostic associations and mortality discrimination comparable to direct pointwise supervision, not improved patient outcomes.

Its broader relevance is credit assignment. A delayed successful outcome should not automatically reward every intermediate state. SWE-Prime reaches the same conclusion from coding traces by a different route. In both cases, the endpoint is informative but insufficient: the system needs a mechanism for deciding which parts of the path deserve credit.

The emerging shape of an agent

These papers converge on a fairly concrete design.

Keep raw traces because summaries erase inconvenient evidence. Build a persistent knowledge layer because patterns need to survive individual sessions. Compile tested knowledge into executable skills rather than exposing the whole archive at runtime. Track entities and lifecycle states explicitly when time and identity matter. Judge memory by downstream use, not retrieval. Preserve failed interventions so they are not rediscovered. Treat successful trajectories as candidates for learning, not automatic gold data.

Most importantly, do not ask the model to carry all of this implicitly.

A longer prompt is not a database. A vector index is not a temporal model. A polished review comment is not defect coverage. Passing tests does not sanctify the route taken. And an agent that completed a task once has demonstrated an outcome, not necessarily a reusable skill.

The frontier is moving outward from the model into the machinery that decides what the model sees, remembers, repeats, and is allowed to forget. That machinery is less glamorous than another benchmark point. It is also where reliable agents will probably be built.

Reading list