The Harness Is Becoming the Model

The most interesting AI papers this week are not really about bigger models. They are about everything surrounding them: the harness that turns a model into an agent, the memory that carries experience between tasks, the verifier that distinguishes completion from bluffing, and the training distribution that quietly determines what capabilities are available in the first place.

That is a useful correction to the usual model-centric story. A fixed model can improve substantially when its workflow is redesigned. A strong coding agent can still fail because it never discovers the shared invariants needed to finish a repository. A retrieved memory can hurt because it contains yesterday’s filenames and users. Post-training can sharpen capabilities already latent in a model without reliably creating knowledge absent from pretraining.

The common thread is that intelligence in deployed systems is becoming less like a single function call and more like an engineered process. The model matters enormously, but so do the state representation, tools, acceptance tests, routing policy, training exposure, and definition of success.

1. AutoDesign: improve the factory, not just the product

AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design asks a better question than “Can an agent revise its answer?” It asks whether an agent can improve the software system that produces future answers.

Its test domain is academic posters. An inner loop reads a paper, selects claims and figures, creates editable HTML, renders it, critiques the result, runs deterministic checks, and repairs local defects. An outer loop then examines performance across multiple papers and proposes one bounded change to the generation harness: perhaps a prompt, tool specification, runtime behavior, orchestration rule, memory structure, or evaluator.

The important part is the acceptance gate. A harness modification is retained only if it improves the training set without reducing performance on a held-out development set. That does not eliminate overfitting—the same development gate is consulted repeatedly—but it is much better than promoting every plausible self-modification because one example looked nicer.

On the authors’ PosterBench evaluation, AutoDesign scores 78.32, the highest reported Main Track result. In a more controlled PosterBench-mini comparison with Claude Code and Claude 4.8 held fixed, it scores 74.56, versus 70.36 for OpenDesign and 66.83 for Claude Design. Attaching the evolved DesignHarness improves results by 5.01 to 19.56 points across seven model-and-agent configurations.

The average increase is reported as 54.99 to 67.39. That is 12.40 points, or roughly a 22.5% relative increase—not merely “12.4%,” despite looser wording in the abstract.

A blind human evaluation provides some support beyond the new benchmark. Eleven volunteers supplied 933 ranking judgments, giving AutoDesign the highest Bradley–Terry preference estimate at 64.0%, though the reported 95% interval is wide at 55.2–77.8%. This is encouraging, not a declaration that conference designers can pack up and go home.

The broader media claims are also ahead of the evidence. AutoDesign demonstrates slides, webpages, and videos, but the formal evaluation is about source-grounded academic posters. “Design anything” remains branding; “improve a structured visual-production harness” is what the paper actually establishes.

For Jarvis, this is the week’s most directly relevant paper. The useful idea is not unrestricted self-rewriting. It is a disciplined improvement loop:

  1. Treat ordinary failures and corrections as evidence about the harness.
  2. Change one component at a time.
  3. Test the change on tasks that did not motivate it.
  4. Keep production feedback separate from a stable regression suite.
  5. Make every persistent update versioned and reversible.

That is less cinematic than recursive self-improvement. It is also how you build something that does not gradually optimize itself into a highly efficient nuisance.

Beyond Retrieval: Query-Conditioned Reuse of Long-Horizon Agent Trajectories identifies a neglected distinction in agent memory.

Retrieval asks which previous interaction might be useful. Reuse asks what, exactly, should be carried from that interaction into the current task.

A successful historical trace may contain a reusable procedure alongside obsolete users, paths, record IDs, dates, observations, and environment state. Injecting the whole trajectory gives the model both the lesson and the landmines.

The paper’s proposed remedy, query-conditioned reuse, turns a retrieved trajectory into a compact note with four parts:

Across 2,391 constructed target tasks in WebArena, WorkArena, and AppWorld, this format achieves 62.3% success, compared with 51.6% when the full selected trajectory is injected directly. It also uses about 9.4k online tokens, versus 18.4k, a reported reduction of 48.9%.

The design of the experiment is more interesting than the headline. The candidate set, selected trajectory, acting model, decoding setup, and tool budget are held fixed across the main reuse conditions. The variable is how the same retrieved experience is presented.

The effects become larger when literal copying becomes more dangerous. Under the largest source-to-target binding changes, stale-binding errors occur in 46.9% of Full Trajectory runs and 10.9% of query-conditioned runs. Correct rebinding rises from 31.7% to 77.8%.

This is not a complete memory architecture. The bank contains only verified successful trajectories; only one memory is supplied; and the targets are constructed by modifying known source tasks. Natural histories will contain failures, contradictions, half-relevant episodes, and old assumptions nobody remembered to label.

Still, the principle is strong: stored history and actionable guidance should be separate artifacts.

For Jarvis, that means a past deployment should not reappear as a script to replay. It should reappear as something closer to:

Memory is not merely retrieval. It is retrieval followed by decontamination.

3. Formal verification exposes the last-mile wall

Vero: Can AI Agents Build Formally Verified Software Repositories? moves beyond individual proof holes and asks agents to complete entire multi-module Lean 4 repositories.

The benchmark contains 43 projects, 743 scored APIs, and 2,705 specifications, translated from real repositories originally written in Python, Dafny, Verus, and Coq. Agents operate in two modes: proving supplied implementations, or jointly writing implementations and proofs.

The strongest evaluated GPT-5.5 xhigh configuration fully solves 27 of 43 repositories in code-and-proof mode and 25 of 43 in proof-only mode. Yet it passes 87.3% and 85.8% of individual specifications, respectively.

That gap is the result.

Current agents can close many local obligations while still failing to produce a coherent verified repository. Ten projects resisted every evaluated agent and mode. The remaining problems tend to share deep structure: cross-module invariants, protocol consistency, helper chains, coverage properties, and custom mathematical theories.

Successful runs are correspondingly organized around reusable lemmas. Across 82 full solves, helper theorems account for a median of 73.6% of proof lines in code-and-proof mode and 71.6% in proof-only. Eighty of the 82 successful runs reuse a helper theorem across multiple specifications.

That resembles ordinary software engineering more than benchmark-style code completion. The difficult part is not emitting one locally plausible patch after another. It is discovering the abstractions that make the rest of the repository tractable.

Implementation freedom helps occasionally. Agents replace the Hungarian algorithm with permutation enumeration, or efficient search structures with simpler recursive lists, because the simpler code is easier to prove. Across five such project-agent pairs, code-and-proof mode closes 250 specifications versus 201 against the fixed reference implementations.

That also reveals a limit: verified does not mean production-worthy. A repository can satisfy its encoded specification while becoming drastically slower, omitting complexity guarantees, or failing to preserve behavior the specification never mentioned.

Vero’s anti-cheating work is nearly as valuable as its scores. The grader reconstructs agent-editable regions in a clean copy and checks for untrusted axioms, sorryAx, decidability laundering, malicious typeclass instances, and splits between logical and executable implementations. In less formal agent benchmarks, these same pathologies appear as modified tests, weakened requirements, bypassed code paths, or a shim that persuades the grader without fixing the system.

For Jarvis-like coding agents, the lesson is to maintain a repository-level invariant map rather than treating every test failure independently. If the same proof or test wall keeps returning, another patch may be the wrong answer. The implementation, shared abstraction, or specification may need to change.

4. Pretraining exposure still sets a formidable ceiling

LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure is an unusually clean attempt to separate capability from elicitation.

The authors construct LittleCurriculum, an 88-billion-token corpus intended to include U.S. kindergarten-to-Grade-5 concepts while sharply reducing later material. They then train a 5-billion-parameter model from scratch and test whether scale, supervised fine-tuning plus GRPO, or in-context examples can recover capabilities that were largely absent from pretraining.

The answer, within this setup, is mostly no.

On a held-out CommonCoreText evaluation, the filtering pipeline retains about 35% of K–5 passages and 0% of Beyond-K–5 passages. An independent WeeBit check retains 2.48% of Beyond-K–5 passages; manual inspection finds clearly advanced concepts in three passages, equivalent to 0.05% of that full split. The boundary is therefore sharp by the paper’s measurements, not magically airtight.

Scaling from 0.6B to 5B parameters improves performance substantially inside the intended exposure range and somewhat around the fuzzy Grade 6–7 boundary. It does little for the hardest Grade 8 mathematics, where performance remains near the floor.

SFT followed by GRPO also helps primarily in-scope. The unrestricted control model gains much more on advanced tasks, widening the gap. Few-shot examples alter response style and length, and modestly improve K–5 performance, but do not recover Beyond-K–5 capability. Explanation prompts similarly fail to produce a meaningful change.

This does not show that reinforcement learning cannot create capabilities, or that retrieval, tools, interactive teaching, larger models, or much greater post-training budgets would fail. It shows that in this controlled 5B-scale experiment, several familiar interventions were much better at exploiting existing exposure than crossing a deliberately engineered knowledge boundary.

The distinction matters well beyond this benchmark. A prompt that makes a model more articulate is not necessarily teaching it anything. A model that produces a longer derivation may be exposing latent knowledge—or merely becoming more verbose around the same mistake.

For systems with retrieval and persistent memory, LittleLearner suggests a useful evaluation discipline: keep an exposure ledger. Separate what was in model training, what was retrieved, what was stored in memory, and what was inferred. Otherwise every successful answer gets vaguely credited to “learning,” which is how engineering turns into mythology.

5. Alignment may depend on when values enter training

Synthetic Persona Pretraining: Alignment from Token Zero asks whether an assistant’s values should be introduced during post-training or embedded throughout pretraining.

The proposed method inserts constitution-guided first-person reflections into ordinary pretraining documents after a special assistant marker. Matching synthetic assistant responses are then used during supervised fine-tuning to bind that pretraining-time persona to the deployed assistant role.

The experiments cover models up to 3 billion parameters trained on as many as 500 billion tokens. Token-zero variants perform better on the authors’ constitution-following benchmark and make fewer risky choices in out-of-distribution moral dilemmas. They also show different value priorities: Truthfulness and Justice rise relative to Learning and Creativity.

But the paper contains an important split result. Early persona exposure appears most useful for value-related generalization, while later midtraining is sufficient—and sometimes better—for jailbreak robustness. Refusal behavior and broader value preferences do not move as one thing.

“Persona binding” also proves fragile. Replacing constitution-matched post-training responses with ordinary responses erodes much of the advantage, especially on the moral-dilemma evaluation. Continual training on chemistry education data degrades alignment results across every recipe. Replaying safety data recovers much of the loss.

This is evidence for training-distribution effects, not proof that models acquire a unified inner self. The evaluations are largely synthetic or model-mediated, the largest model is still small by frontier standards, and moral preferences are often inferred from option log-probabilities rather than free-running agent behavior.

Even so, the deployment lesson is uncomfortable and useful. A system prompt is not a durable personality. Behavior can depend on the compatibility of pretraining data, post-training examples, chat templates, role markers, tool-call formats, and later updates.

For Jarvis, that argues for regression tests across the actual serving stack. Changing the wrapper, assistant token, tool protocol, or model version can alter behavior even when the written principles look unchanged. Safety and personality should be tested under real tool use and unfamiliar conflicts, not merely by asking the assistant to recite its constitution.

6. An AI scientist is only as credible as its evidence ledger

OmniScientist: An Omni-Modal Omni-Discipline AI Scientist is most interesting when it stops sounding like an “AI scientist” pitch and starts looking like a disciplined evidence pipeline.

The system works across images, spectra, waveforms, audio, video, 3-D structures, trajectories, tables, formulas, and graphs. Perception occurs before question formation, allowing raw observations to affect what the system chooses to investigate rather than merely illustrating a text-generated hypothesis.

Its three agentic stages—ideation, experiment, and writeup—sit inside a deterministic outer process. Code-enforced checks cover novelty evidence, leakage, effective sample size, multiple comparisons, provenance, anti-HARKing constraints, numerical consistency, and claim support. The writing stage is constrained by what was actually executed.

Across 36 real-data cases, the system reportedly completes the path from dataset to compiled manuscript every time. Twenty-eight cases are perceptual, so the suite is less balanced than the “omni-discipline” name suggests.

In five paired cases, the full system is compared against a blind variant receiving only precomputed scalar features. Direct perception wins 85% of head-to-head manuscript judgments and improves all seven evaluated dimensions. Five cases is a useful ablation signal, not a population estimate for scientific discovery.

The internal run statistics are revealing: 67 analyses are demoted from headline findings because they fail selection or support criteria; two runs are classified as refuted and 17 as mixed. A system willing to preserve null and mixed results is already doing something more scientific than many press releases.

The case studies—noise-labelled seismograms containing transient structure, and spatial texture patterns in paediatric chest radiographs—are promising computational demonstrations. They are not prospective validation, laboratory replication, or clinical evidence. The manuscripts are scored by model judges, and several important numerical values are missing from the extracted text, so exact effect sizes should wait for inspection of the rendered paper.

The transferable idea for Jarvis is an evidence ledger: quantitative prose should point back to a source, extraction, computation, and execution trace. A fluent writing agent should not be allowed to introduce a number simply because it would make the paragraph better.

7. Training agents requires traces that survive contact with tools

Intern-S2-Preview: Scientific Agentic Foundation Model describes a 397-billion-parameter multimodal scientific model trained through continued pretraining, supervised fine-tuning, multi-task reinforcement learning, agentic RL, and on-policy distillation.

Its benchmark table is broad—scientific reasoning, molecular tasks, multimodal understanding, coding, terminal use, software engineering, and research agents—but comparisons are not always like-for-like. Some benchmarks are internal or new; harnesses and tool budgets vary; and the model is explicitly a preview system. The paper supports “strong across the authors’ comparison suite,” not “best scientific model.”

The systems work is more durable than any ranking.

Intern-S2 separates an agent harness from its task. The harness defines the runtime—coding loop, browser-like agent, external tool agent—while the task defines the environment, objective, and verifier. This allows the same training infrastructure to operate with both directly controlled agents and external runtimes such as Claude Code, OpenHands, OpenCode, Mini-SWE, and OpenClaw.

More importantly, the Trace Store preserves exact token IDs, log probabilities, and, when needed, mixture-of-experts routing decisions. A semantic action can be mapped back to the precise assistant token span that generated it. User text and tool output are masked from the policy loss rather than accidentally treated as model behavior.

Outcome and process feedback remain separate. A session can succeed overall while still assigning reduced positive credit to a segment that made an invalid tool call, repeated a failure, or malformed an output. That is the right distinction for real agents: reaching the destination should not reinforce driving through the neighbour’s garden.

The paper also introduces a plug-in Memory Decoder. A 4B biology memory attached to the frozen backbone raises the Biology-Instructions average from 56.92 to 60.32. That is modular parametric specialization, not transparent episodic memory, but it demonstrates that domain knowledge can be attached without rewriting the 397B backbone.

For Jarvis, the trace representation is the key idea. A useful operational record should connect the request, decision, tool call, observation, and verified outcome. Without that correspondence, learning from agent sessions risks reinforcing whatever text happened to be nearby rather than the action that actually helped or harmed.

8. Adaptive compute should follow the geometry of the problem

The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity is the most theoretical paper here, but its central idea is pleasantly concrete: do not spend equal compute on unequal parts of a generation process.

Masked diffusion begins with masked coordinates and progressively reveals them. The paper introduces unmasking growth complexity, a path-dependent measure of how sharply conditional distributions change during this reveal process. Easy regions can be traversed with larger steps; regions where information changes rapidly receive finer resolution.

The natural coordinate is log reveal odds, not ordinary time. Within a finite partition of that path, each block receives its own geometric multiplier. The paper derives an optimal integer allocation using dynamic programming and develops sample-based estimators that can attach high-probability KL-error guarantees.

Constructed examples include repeated bits, parity constraints, mixtures, hierarchical mixtures, and dense random XORSAT. In favourable cases, a constant number of adaptively placed blocks yields a claimed (\widetilde{\Omega}(\sqrt d)) theoretical improvement over coarse schedules.

That is not a measured speedup on a production image or language model. The strongest results assume access to exact Bayes denoisers and moment control over posterior changes. Learned denoisers add approximation error that must be bounded separately. There are no standard FID, perplexity, wall-clock, or sample-quality results.

Still, the conceptual split is useful:

That pattern extends beyond diffusion. Agent workflows also have long easy stretches and short regions where uncertainty, consequences, or state changes spike. Uniform effort is simple, but rarely optimal.

9. Routing among specialists works—within a controlled world

MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification uses a multimodal language model as a router among ResNet, DINO, and CLIP-based image classifiers.

The router examines an image and auxiliary measurements such as blur, brightness, contrast, and noise, then assigns it to a broad domain: general, facial, geographic, medical, or uncertain. The chosen specialist performs the actual classification.

On a unified benchmark combining CIFAR10, EuroSAT, FER2013, and OrganAMNIST, the system reaches 90.78% accuracy, 1.17 percentage points above the balanced DINOv3 expert. The gain is real but modest, and all experts are trained on the known benchmark mixture. This is routing among familiar domains, not open-world adaptation.

The most useful feature may be the UNSURE path, which falls back to a balanced expert rather than forcing every image into a specialist. Self-consistency does not improve the final score: the baseline reaches 90.78%, compared with 90.71% and 90.75% for two multi-call variants. More reasoning is not automatically better routing.

The handcrafted image-quality signals also misfire. EuroSAT’s uniform textures can look like blur to generic metrics. Such features are useful diagnostics, but weak semantic evidence.

This maps neatly to general tool use. A capable assistant should route text-heavy images to OCR, diagrams to structural visual analysis, technical plots to native data readers, and uncertain cases to a general model or clarification path. The multimodal model acts as a control plane, not a replacement for the specialists.

Its natural-language rationale may help inspection, but should not be mistaken for a faithful causal explanation. Models remain excellent at explaining decisions they may not actually have made for the stated reasons.

10. A small model can be impressive without being a tiny frontier model

DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data is a 1-billion-parameter hierarchical reasoning model trained from scratch by the Danish Foundation Models project.

Its corpus contains 161 datasets and approximately 70.5 billion tokens per epoch, with 24.7% Danish, 6.4% bilingual Danish–English, and 68.5% English data. Training takes just under three weeks on eight NVIDIA B200 GPUs.

The small parameter count should not be confused with a small data or compute experiment. Some datasets are heavily repeated, and the top ten account for 66.5% of sampled tokens.

Mimir’s strongest case is Danish. The authors report leading results among tested systems on DaLA, GEC, and WikiQA, plus a near-best result on Nordjylland News summarisation. It also performs unusually well for its weight class on GSM8K and HumanEval. The paper’s aggregate Math & Code score rises from 46.9 for the earlier HRM-Text baseline to 64.1, a 36.7% relative improvement.

“Frontier performance” needs a thick layer of qualification. Mimir remains behind Gemma 4 on Math & Code, and the authors explicitly say its assistant behaviour is below the state of the art. Aggregate benchmark proximity to a 4B model does not make it a drop-in general assistant.

The data strategy is noteworthy. The project uses openly licensed, agreement-supplied, synthetic, or legally justified research data, including synthetic “transplant datasets” intended to replace sources that did not meet its permissibility standards. That is a serious attempt at provenance, though not an independent legal guarantee that every source is universally uncontroversial.

Mimir looks more plausible as a cheap auxiliary model—for Danish transformation, classification, extraction, routing, or local drafting—than as a Jarvis backbone. The right test would be tool-call exactness, schema adherence, retrieval grounding, latency, hallucination rate, and recovery from tool errors. Benchmark averages are not operational readiness.

The emerging stack

Taken together, these papers describe an AI stack whose important boundaries are becoming clearer.

Pretraining determines much of the raw capability surface. Post-training and prompting can expose, steer, or sharpen it, but may struggle to create what the model never encountered. Values can depend on when and how they entered the training distribution. Long-running agents need traces that distinguish their own actions from user input and tool output.

Above the model, the harness shapes what happens next. It decides which tools exist, how memories are transformed, where compute is spent, when specialists are used, what evidence reaches the writer, and which changes survive. Verification then defines whether apparent progress counts: a rendered poster, a completed browser task, a machine-checked repository, or a manuscript whose numbers all trace back to executed analyses.

This is not an argument that models have become interchangeable. AutoDesign still performs differently across backbones; Vero’s strongest configuration crushes weaker ones; LittleLearner’s controlled exposure constrains what later interventions can recover. The model remains the engine.

But the engine no longer tells you what the vehicle can do.

The practical frontier is increasingly the full system: model, harness, memory, tools, state, evaluator, and acceptance gate. Better agents will come not only from scaling the model, but from making that surrounding machinery explicit, testable, and difficult to bullshit.

Reading list