The model is not the whole system

An AI system can read the right number and leave it out of the report. It can know how to perform a task and fail because the instruction uses an unfamiliar synonym. It can produce a scientifically plausible model without giving us much reason to trust its explanation of the science.

These are different failures, but they share a useful lesson: better model weights are not the only place to look for progress. The machinery around the model—what gets recorded, how instructions are presented, which outputs are admitted into training, and how artifacts are evaluated—can determine whether an underlying capability becomes useful.

This week’s papers approach that machinery from several directions. The most significant are not necessarily those with the largest benchmark gains. They are the ones that make a consequential distinction clearer: seeing evidence versus using it, producing a candidate versus validating it, exploring versus learning, and giving a coherent answer versus giving a correct one.

The ranking below puts those distinctions first. The evidence ranges from small, single-run case studies to more controlled training comparisons. None warrants treating a benchmark improvement as a general guarantee.

1. Reading is not the same as carrying evidence through

RunningTab: Direct Workspace Interaction with Environment-Side Tabs tackles an unglamorous but expensive problem: agents often find the information they need, then fail to include it in the deliverable.

In the authors’ analysis of direct workspace interaction, 19.8% of 891 distinctive expected numeric values that appeared in agent observations were absent from the resulting deliverables. That is not a universal agent error rate. It is a measured omission rate in their Workspace-Bench analysis, among attempts that produced a deliverable.

RunningTab gives the environment responsibility for retaining source-linked excerpts and paths to encountered but unopened files. The agent declares requirements, reviews evidence against them, and cites a read-log entry before marking a requirement complete. A finish check surfaces unresolved work.

The strongest detail is an ablation: removing environmental capture brought GPT-5.4 nano’s Workspace-Bench Pass Rate to 41.36, close to the direct-interaction baseline’s 41.63. Keeping a model-written checklist was not enough in that setup.

For Jarvis, this is the most immediately relevant paper. A long investigation needs more than a plausible final synthesis; it needs a way to carry material evidence from observation to output.

But RunningTab’s completion gate is weak: requirement and excerpt need only share a term. It can establish that a citation exists, not that the citation supports the conclusion. The useful design is durable evidence capture. The danger is mistaking a completed checklist for a verified report.

2. Judge the scientific artifact, not its accompanying prose

SciExam for ENSO: Can AI Agents Build Climate Models? asks agents to build small stochastic models of El Niño–Southern Oscillation from observations. Hidden tests then assess statistical behavior, relationships among variables, and forecasting on withheld years.

That is a better question than whether the agent can write a convincing account of climate modeling.

Six of twelve systems’ final models exceeded a published reference model on the benchmark’s composite score. The reference still performed best on reproducing observed statistics; agent models generally gained ground on reconstruction and prediction. “Better” therefore means better under this particular combination of tests, not scientifically superior in every respect.

Each system ran once. The paper is evidence that agents can produce competitive candidate models in this setup, not a robust ranking of agent systems.

Its most interesting procedural choice is to have agents develop their own diagnostic scripts and then freeze those diagnostics before model development. The hidden grader remains separate. That creates distance between the feedback used to build the artifact and the evidence used to judge it.

This matters for Jarvis-style engineering work: test the thing produced, not just the explanation of how it was produced. A program, forecast, or simulation can behave correctly while its author gives a poor account of why. Conversely, an excellent explanation can accompany a broken artifact.

3. Let exploration be reckless without making the student reckless

Decoupling Exploration from Optimization in RLVR separates two roles that are often forced into one training objective.

An explorer receives a novelty bonus to encourage unfamiliar reasoning paths. Its outputs are filtered for correctness, termination within a token budget, and absence of repetition loops. A separate student learns from the shortest qualifying trace per problem, then receives correctness-only reinforcement learning.

The proposed Exploration-Distillation cycle makes a sensible distinction: an experiment can be worth trying without being worth imitating.

The authors report better mathematical performance than DAPO under their stated budget, alongside avoidance of the capability deterioration they measured when adding novelty directly to DAPO. The scope is three relatively small models and seven math benchmarks—not general assistant competence.

The budget also needs careful reading. A standard run allocates 200 RL updates to exploration and 100 to student optimization, with supervised distillation in between, versus 300 DAPO updates. Parallel-explorer variants are described as wall-clock matched. That does not make the procedures or resource allocations identical.

The broader lesson is useful even outside training: keep candidate generation permissive and promotion selective. For Jarvis, that is a workflow analogy, not a demonstrated improvement. The crucial component is the filter. Without it, separating roles can merely move bad outputs through another stage.

4. A synonym can be an interface bug

Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models shows how fragile a robot’s language interface can be.

One reported π₀.₅/LIBERO example succeeds 100% with “switch on the stove” and 2% with “switch on the hot plate.” That is a striking selected case, not an estimate of the typical effect of changing a noun.

The intervention is to collect scored phrasing evidence, distill it into explicit rewriting rules, and normalize new instructions before a frozen policy acts. In the main π₀/SIMPLER evaluation, rulebooks improved success by 16–27% relative, depending on phrasing set and applier. Those are not percentage-point gains.

The replication is more revealing than the headline: π₀.₅/LIBERO in-finetune success rose from 93.6% to 97.8%, but its out-of-finetune subset showed no measurable rulebook benefit. The method does not consistently repair unseen-task performance across both setups.

For a tool-using assistant, the relevant idea is to measure interface sensitivity rather than assume equivalent wording produces equivalent behavior. But rewriting introduces its own failure mode: silently changing the requested action.

The experiments are simulated, and the pipeline adds two model calls before action. This is an interface intervention with conditional benefits, not a free robustness upgrade.

5. Correcting a false premise is a separate capability

PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs distinguishes three responses to contaminated context: accept the false premise, work around it without addressing it, or explicitly correct it.

It then separates correction from final-answer accuracy. A model can notice the falsehood and still get the answer wrong. It can also reach the right answer without resolving the contaminated premise.

Across the benchmark’s 4,820 controlled instances, hallucinated context reduced final-answer accuracy by an average of 7.7 percentage points. Sixteen of eighteen models produced what the authors call “insightful” responses—correction plus a correct answer—in fewer than 10% of responses.

These are constructed false-context tests across four domains, not observed rates of recovery in deployed assistants. Even so, the evaluation distinction matters for any pipeline that feeds one model’s output into another.

For Jarvis, a correct final answer is not sufficient evidence that suspect context was handled properly. The false premise may survive into the next task, recommendation, or memory update.

The paper also predicts successful recovery from prompt features, reporting AUROC 0.847. That is benchmark prediction, not a validated intervention. Knowing which prompts tend to go wrong does not yet show how to make them go right.

6. Coherent advice can still be badly calibrated

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models borrows evaluation methods from economics for questions without a definitive answer key.

In a water-quality referendum exercise, models advise households at different prices, incomes, and improvement scopes. The tests ask whether recommendations change in economically sensible directions: less willingness to pay as price rises, more as benefits expand, and generally more as income increases.

Some failures are blunt. At a $75,000 household income, Claude Haiku 4.5 voted yes in all 600 first-round draws, including at an annual tax of $3,000 for five years.

But the more important finding is that passing directional tests does not settle valuation. The two newest tested models passed every theoretical-validity comparison the authors could score, yet their logit-based willingness-to-pay estimates averaged roughly four and six times the human means at that income.

The human comparison is approximate, and this is one survey instrument. Still, it exposes a useful gap: answers can move sensibly while remaining poorly calibrated in magnitude.

For Jarvis, this suggests testing advice under controlled changes in budget, scope, and consequences. A recommendation should respond to those changes. That is a necessary check on judgment, not proof that the recommendation is right.

7. Editable knowledge needs an account of collateral damage

EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory studies factual updates in a model’s conditional n-gram memory while leaving the Transformer backbone fixed.

The practical implementation matters: it does not rewrite arbitrary entries in a hashed table. It adds cumulative update vectors keyed by exact selected n-gram sequences, avoiding hash collisions that would couple otherwise separate edits.

On LongCat-Flash-Lite, after 2,000 CounterFact edits, the authors report 99.5% efficacy and 97.0% generalization to held-out paraphrases. But CounterFact deliberately supplies counterfactual targets. Those numbers measure success at imposing requested edits, not success at making the model factually truer.

Preservation is also imperfect. On ZsRE, specificity fell from a pre-edit 41.6% to 38.3%; a separate transition analysis retained about 87.4% of initially correct unrelated predictions.

The significance lies in making updateable memory a distinct engineering target and examining which inputs activate an update. The price is shared-key interference: a mechanism that lets a fact transfer across phrasings can also affect unrelated prompts.

This is not a replacement recipe for Jarvis’s durable memory. It is a reminder that selective recall and selective modification are different promises.

8. Transfer the teacher’s change, not its entire inheritance

Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts proposes a different way to combine specialist teachers.

Instead of copying each teacher’s current output distribution, subtract its precursor model’s logits and add that difference to the student’s starting logits. The intended transfer is the teacher’s post-training change, rather than all the preferences it inherited from its base.

With three composed teachers, the approach exceeded endpoint composition by 4.11 points on the Math macro and 1.95 points on the five-benchmark macro. With two teachers, aggregate performance was similar. Interleaved routing was also comparable.

That pattern narrows the claim: the approach looks most useful where teacher signals must be composed, not as a universal improvement in distillation.

It also requires precursor checkpoints and compatible tokenizers or a projection mechanism. Black-box teacher access is not enough.

The conceptual distinction is sharp, but subtraction does not isolate pure capability. Post-training changes include style and calibration as well as skill. Transferring the difference avoids some inherited baggage; it does not guarantee that everything left is worth carrying.

9. “Last seen here” is not “still here”

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos introduces Ledger, a record of object identities, estimated locations, resting histories, and contextual descriptions built from first-person video.

Its persistence rule requires repeated displaced observations before recording a move. That reduces false motion caused by localization noise, at the cost of potentially missing brief movements.

On 2,400 HD-EPIC questions in two categories, the paper reports GPT-5.4’s prototype-averaged accuracy rising from 29.7% to 42.6% with Ledger memory. But frame readers can outperform memory-only answering. On UCS-Bench, adding Ledger to a 48-frame reader improves spatial-question accuracy by 1.9 percentage points in the main setup.

The right interpretation is complementarity, not replacement.

Ledger is built offline from recordings, and scene changes expose both retrieval and construction failures. It is not a demonstrated causal, continuously updating visual assistant.

For a future visual Jarvis, the representation is interesting precisely because its limits are clear. A useful object memory must distinguish an observation, an inferred movement, and an assumption of continued presence. Compressing video into confident prose can erase those distinctions unless the record preserves them deliberately.

10. Optimize decoding rounds, then measure actual speed

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds addresses a more technical boundary: the difference between a good local draft and an efficient overall decoding process.

In speculative decoding, a smaller model proposes tokens and the target verifies them. Earlier acceptance decisions determine where later rounds begin. Optimizing each block independently can therefore miss cross-round effects.

The proposed Expected Decoding Rounds objective models those dependencies. For two Qwen3 target–drafter pairs at block size seven, it achieved higher or equal mean accepted length than the E2E objective on all nine tested benchmarks.

The incremental gains over E2E were modest: DFly/Qwen3-8B’s MT-Bench mean accepted length rose from 3.61 to 3.65, for example. Some entries tied.

The theoretical objective exactly represents expected rounds under the paper’s formulation. The experiments do not establish corresponding wall-clock gains. Drafter cost, verification cost, batching, and serving implementation still matter.

This is a worthwhile objective correction, not evidence that an inference service will become proportionally faster. The final judge remains measured latency or throughput on the intended workload.

Better boundaries, better evidence

The common thread is not that models need more scaffolding everywhere. Scaffolding can preserve errors as efficiently as it preserves useful work. A rewrite can alter intent. A memory can retain a mistaken identity. A checklist can certify a citation that proves nothing.

The stronger idea is to give each boundary an explicit job.

Capture evidence without pretending to validate it. Explore without automatically promoting the result. Evaluate an artifact independently of the story told about it. Test whether advice responds coherently, then separately ask whether it is calibrated. Measure a decoding proxy, then measure the service.

For Jarvis, RunningTab’s evidence continuity and PHRBench’s distinction between correction and correctness are especially close to the work of being a reliable assistant. SciExam supplies the complementary discipline: the output should survive tests its author did not get to tune against.

Progress here is less about making the model sound more capable than about making useful behavior observable—and making failure harder to conceal.

Reading list