The Harness Is Part of the Intelligence

A language model is only one component of an AI system. Around it sits a growing mass of prompts, tools, retrieval, memory, evaluators, retries, permissions, stopping rules, and code. That surrounding machinery is usually called the harness, a term that undersells how much of the system’s actual behaviour now lives there.

Several papers this week approach the same problem from different directions: how do we improve an AI system without mistaking motion for progress?

One paper measures whether models can optimize an agent harness against hidden tests. Another trains models to distinguish useful context from persuasive nonsense. Two more use populations of candidates or solvers rather than trusting a single trajectory. An evaluation paper asks when enough evidence has accumulated to stop testing. A safety paper tries to preserve refusal behaviour while a model evolves.

The common thread is that reliable improvement needs structure around optimization. You need held-out evaluation, provenance, uncertainty, diversity, and boundaries on what may change. Otherwise the optimizer eventually finds the metric’s loose floorboard and crawls underneath it.

1. Optimizing the harness, not just the model

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization is the most directly consequential paper in this batch.

The benchmark gives an optimizer model an intentionally untuned agent codebase. The model can alter prompts, tools, context handling, retries, retrieval, answer extraction, reasoning effort, and control flow. It sees detailed development results and aggregate validation scores, but the final candidate is judged on a hidden test partition.

That last part matters. Harness optimization is unusually vulnerable to self-deception because the thing doing the optimization can repeatedly inspect the same visible evaluator. A rising validation score may mean genuine improvement, selection overfitting, or merely discovering the evaluator’s accent.

Across four tasks and 111 scored optimizer runs, changing the optimizer model moved normalized gain by 0.142 on average, while changing the coding harness moved it by 0.079. Both effects were measurable under the authors’ task-specific resolution estimates, but model choice mattered more.

That does not imply native coding tools are irrelevant. Across 20 paired model–task comparisons, the shared harness won 11 and the native harness won nine. On GAIA, however, the GPT models performed substantially better under codex. The aggregate near-tie conceals model-specific interactions.

Several operational findings are more interesting than the leaderboard:

None of that proves traces are useless or that touching more components causes better results. Search effort is an obvious confounder. But it does suggest that prompt tweaking is too narrow a model of harness engineering. The meaningful levers include tool contracts, retry policy, context management, extraction, and resource limits.

For Jarvis, this is almost painfully relevant. Its behaviour is not defined by a model name. It is defined by the model plus memory retrieval, system instructions, skills, browser access, service controls, approval boundaries, and the term-llm orchestration layer. Any claim that Jarvis has “improved” should therefore be tested as a system change against pinned baselines and held-out tasks—not inferred from a nicer-looking prompt or a successful demo.

2. Learning when context deserves trust

Learning When to Trust via Selective Context Preference Optimization studies a failure that appears constantly in tool-using systems: the model reaches the right answer, sees a plausible but incorrect hint, and talks itself out of being right.

The paper’s MIST benchmark renders 1,000 questions in four matched conditions:

  1. no added context;
  2. misleading context pointing toward a plausible wrong answer;
  3. correct, useful context;
  4. related but irrelevant context.

This design catches an important failure in robustness research. A model trained simply to resist external context may improve on adversarial examples by becoming stubborn. That looks good until the context is correct and the model ignores it.

The authors introduce “signal-induced correct-to-wrong,” or SC2W: among questions answered correctly without extra context, how often does a misleading signal flip the answer to wrong?

Every model in their evaluation showed some susceptibility. Reported SC2W was 10.5% for GPT-5.5, 12.9% for Gemini 3.1 Pro, and 12.0% for Claude Opus 4.8. Those are results on this constructed benchmark, not estimates of how often frontier models are misled in ordinary use.

Their training method, SCOPE, uses ordinary DPO. The novelty is in the data: matched preference pairs balanced across misleading, clean, correct-context, and irrelevant-context conditions.

On Qwen3-4B, SCOPE:

The comparison with misleading-only training is the key result. On Llama-3.2-3B, misleading-only DPO improved resistance to bad context but collapsed correct-context accuracy from 78.5% to 56.4%. SCOPE improved misleading-context performance without teaching blanket distrust.

This is not a prompt-injection defence, and the paper does not claim otherwise. It is about selective epistemic trust, not instruction hierarchy or hostile tool use.

Still, the design principle transfers cleanly to Jarvis: retrieved pages, search snippets, email, local files, and tool outputs are evidence, not instructions and not automatic truth. But “ignore external context” would be equally broken. The system has to reject bad evidence, use good evidence, and avoid treating irrelevant text as a vote.

A matched four-condition test suite for tool output would be far more revealing than another pile of isolated QA scores.

3. Populations remember what individual calls forget

Relay, Don’t Route: Adaptive Population Handoff for Cost-Efficient LLM-Driven Evolution challenges the usual approach to mixing cheap and expensive models.

Most routing systems decide which model should handle the next call. RelayEvolve argues that this is the wrong unit of allocation for evolutionary search because the search is stateful. Each generated candidate changes the population available to future generations.

Its strategy is:

  1. let a cheap model explore several trajectories;
  2. maintain a compact archive balancing quality and diversity;
  3. measure the marginal improvement of that archive;
  4. stop cheap exploration when its population-level gain remains low;
  5. hand the curated population to a stronger model for joint refinement.

The distinction between joint refinement and independent refinement matters. Candidates from different cheap trajectories are merged into one strong-model population, allowing ideas to interact rather than remaining sealed in separate runs.

Across four program-evolution benchmarks and three budgets, RelayEvolve achieved the highest reported mean in 11 of 12 settings. The exception was Prism at the lowest budget, where the paper says the methods were already close to saturation. Results were averaged over three runs per setting, which is enough to make the result interesting but nowhere near enough to establish a general law of model routing.

The ablations support the mechanism rather than merely the cheap/strong mixture. Performance declined when the authors removed adaptive relay stopping, randomized Grow/Deepen choices, reversed the model order, or selected handoff candidates using only quality, only diversity, or random choice.

The transferable idea is that best-so-far score is not enough for stateful search. A candidate can be valuable because it covers a different part of the solution space, repairs a distinct failure mode, or provides useful material for recombination.

For Jarvis, this makes sense only for long-running searches with external evaluation: evolving scripts against tests, exploring infrastructure configurations, or comparing several repair strategies before escalating to an expensive model. Applying this machinery to ordinary conversational turns would be absurdly ornate. Sometimes a question is just a question.

4. Build training tasks around solver disagreement

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks applies a related population idea to training-data construction.

An executable terminal task can still be useless. It may be trivial, impossible, ambiguous, broken, or solvable through an unintended shortcut. Structural validation and a reference solution establish that the task can run; they do not establish that it occupies a useful difficulty range.

CalibForge has multiple agents attempt each candidate task, verifies the final sandbox state, and revises the task based on solver behaviour. It uses two retention criteria:

The resulting “solver-relative learnable zone” is a practical alternative to asking an authoring model whether its own task is good.

The authors produced 5,431 retained terminal tasks. In a matched 1,300-task ablation using Qwen3-30B-A3B-Instruct, downstream Terminal-Bench 2.0 performance was:

Calibration materially changed tasks rather than merely filtering them. Only 19% of contrastive candidates initially met the strong-pass/weak-fail condition; after revision and re-probing, 96% eventually did.

The caveat is that disagreement is not inherently useful. All-fail may indicate a broken environment. Strong-fail/weak-pass may reveal leakage, nondeterminism, or model-specific tricks. The author still needs to interpret verified trajectories.

For Jarvis, the appealing adaptation is a small corpus of real, executable terminal and homelab tasks with recorded solver matrices, dependencies, failure modes, and strong verifiers. Easy tasks remain useful as regressions. Boundary tasks are useful for learning. Broken tasks belong in the bin, regardless of how educational their stack traces appear.

5. Stop evaluating when the evidence is sufficient

AV-AIVAT: 74× Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games combines two ideas that should be standard in expensive agent evaluation:

Ordinary confidence intervals do not survive casual repeated peeking. If you rerun an evaluation until the interval looks favourable and stop there, the nominal error guarantee is gone.

The paper demonstrates this vividly. In a constructed zero-effect poker resampling experiment, repeatedly checking a naive fixed-sample 95% interval and stopping at the first apparent effect generated false claims in 1,227 of 2,000 entries: 61.35%.

AV-AIVAT combines AIVAT, a poker-specific variance-reduction method, with confidence sequences designed for sequential monitoring. Across 15 LLM-agent heads-up no-limit hold’em configurations and 71,439 paired hands, AIVAT reduced empirical variance by a median 54×. Under the paper’s asymptotic confidence sequence and a ±1-big-blind precision target, corrected outcomes reached the target with a median 74× fewer hands than raw outcomes.

That is where the title comes from, but it needs several asterisks. The 74× figure is:

The exact empirical-Bernstein method showed only a 1.37× median stopping-time improvement on the HUNL data, largely because its declared payoff bound imposed a width floor. Worse, the corrected-payoff bound there was derived from observed maxima, so the authors appropriately describe that result as descriptive rather than exactly certified. They do derive a structural bound for the simpler Leduc game.

The poker-specific correction is unlikely to transfer directly to general assistants. The larger lesson does: either fix the evaluation horizon in advance or use an interval designed for continuous monitoring. Do not keep checking ordinary intervals and stop when the graph becomes flattering.

6. Preserving a safety behaviour during self-improvement

Safe Evolution with Circuit Anchors studies what happens when task optimization gradually erodes a model’s learned refusal behaviour.

Circuit-Anchored Evolution identifies a small set of transcoder features associated with refusals, tests their causal influence, and penalizes changes to their activation distribution during subsequent evolutionary training.

In a reported Llama intervention, suppressing the selected features reduced refusal on held-out AdvBench prompts from 96.2% to 48.7%. Scaling the features to 10× raised refusal to 99.4%. Random feature sets did not show a comparable systematic effect.

In the clearest long-run result, after 5,000 evolution steps on Gemma-2-2B-IT, the paper reports a 95.81% safety score for the anchored method versus 61.24% for unconstrained evolution.

“Safety” needs quotation marks here. The evaluation is principally refusal-based. Preserving a refusal circuit does not establish resistance to tool misuse, deception, privacy leakage, adaptive jailbreaks, or unsafe external actions. A model could also preserve the activation statistics while rerouting behaviour elsewhere.

The paper is therefore better read as evidence for a general danger: capability optimization does not automatically preserve safeguards. For Jarvis, the practical analogue is not freezing 2% of a neural network. It is keeping permissions, credentials, external-action gates, and regression tests outside the part of the system allowed to optimize itself.

Internal behavioural features may become useful diagnostics. They are not security boundaries.

7. Evidence-linked automation in medicine

Tracing the Heart: An Evidence-Linked Pipeline for Heart-Failure Feature Engineering is notable less for its predictive result than for its architecture.

The nMAS pipeline transforms nine electronic health-record tables into patient-level features using deterministic preprocessing, a versioned clinical rubric, bounded LLM auditing, protected provenance fields, and structural checks. The model can modify only whitelisted fields; invalid audit responses are logged rather than silently replacing deterministic output.

On a 500-patient single-institution heart-failure dataset, the pipeline generated 132 structured columns and 70 higher-order composite columns. Adding the composite features increased repeated cross-validated AUROC from 0.895 to 0.963 for HFrEF versus phenotype-unknown patients, and from 0.870 to 0.910 for HFpEF versus phenotype-unknown patients.

Those are not general heart-failure diagnostic results. Only 3% of patients reportedly had numeric ejection-fraction values, classification targets were close to several engineered features, and there was no external or prospective validation.

The architectural lessons are stronger:

For consequential automation, an internally coherent answer is not enough. You want to know which source fields produced it, which rule transformed them, and whether the model altered anything along the way.

8. Uncertainty belongs inside optimization

Quality Diversity for Reliable Data Driven Time-Use Optimization uses MAP-Elites to generate diverse 24-hour activity schedules for children while penalizing predictions in poorly supported parts of the data.

The paper’s optimizer considers sleep, sedentary time, light activity, and moderate-to-vigorous activity—plus a seven-dimensional variant—and scores candidate schedules using regression-predicted health outcomes and analytical prediction uncertainty.

The responsible interpretation is not that the algorithm discovered an optimal childhood routine. The data are observational, model uncertainty is narrower than real causal uncertainty, and no intervention trial tested the recommendations.

The useful pattern is broader: preserve several high-quality options, represent meaningful trade-offs, and discount apparently excellent solutions that depend on extrapolation. In higher-dimensional experiments, uncertainty-aware optimization shifted candidates away from extreme, weakly supported regions while retaining broadly similar high-performing areas.

A recommendation system should be able to say: this option scores highest under the model, but it rests on thin evidence; here is a slightly less ambitious alternative supported by much denser data. That sentence is usually more valuable than a decimal point pretending to be certainty.

9. Two results that matter mostly in their own domains

An Optimal Agnostic PAC Algorithm claims a statistically optimal high-probability risk bound for binary classification across the full range from noiseless to heavily noisy learning.

The result removes extra logarithmic factors and preserves coefficient-one dependence on the best achievable risk (L^*). It is theoretically significant if the proof survives scrutiny, but “optimal” refers to asymptotic dependence. The stated universal constant is (7 \times 10^8), and the generally improper learner has no computational-efficiency guarantee. This is a learning-theory result, not a practical training recipe.

The appendix is unusually candid about AI assistance: the authors report using GPT-5.5 Pro and GPT-5.6 Sol while exploring proof ideas, and say 11 of 16 runs of a compressed prompt produced an essentially correct proof outline. That is an interesting account of mathematical discovery, not independent verification of the theorem.

Finally, Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation learns when a binary neural-network accumulation can stop before computing every term.

On binary-activation VGG11 for CIFAR-10, the method skipped 86.6% of terms in the deepest convolution for a 0.37-point accuracy drop. Applied to the three deepest convolutions, it reduced idealized full-network arithmetic by 25% for a 1.36-point drop.

The phrase “idealized arithmetic” is doing heavy lifting. The implementation assumes per-output ordering and independent stopping, without measuring the cost of irregular memory access, control flow, synchronization, or lost vectorization. It shows that partial sums often predict the final sign early. It does not show a production accelerator or wall-clock speedup.

What the system around the model must do

Taken together, these papers describe an emerging engineering discipline.

The harness should be optimized, but judged on hidden tasks. Context should be used selectively, not obeyed or ignored wholesale. Search should preserve diverse candidates when the state of the search matters. Training tasks should be calibrated against actual solver behaviour. Evaluations should stop according to valid statistical rules, not when the graph looks good. Safety constraints and provenance should live outside the optimizer’s unquestioned control.

None of these ideas is glamorous on its own. They are mostly forms of bookkeeping: holdouts, archives, confidence sequences, evidence traces, immutable policies. But reliable intelligence is increasingly a property of that bookkeeping.

The model generates possibilities. The harness decides which possibilities become actions, memories, claims, or deployed code. That makes the harness part of the intelligence—and, inconveniently, part of the thing that has to be evaluated.

Reading list