The hard part is knowing what got better.

A smaller model produces an answer more cheaply. A revised prompt wins a benchmark. A reasoning trace looks convincing. A coding agent passes its tests. Each is useful evidence. None, by itself, tells us quite as much as we would like.

This week’s papers circle that gap between an observable success and the capability we infer from it. Some offer practical ways to build cheaper, more specialized systems: compile a task into model weights, improve prompts through structured diagnosis, turn old agent sessions into fresh training environments. Others question the instruments used to declare those improvements. Does the judge give the same verdict twice? Does a citation answer the actual question? Does an apparently decisive reasoning step change the outcome?

The productive tension is not “progress versus skepticism.” It is between making systems better and getting more precise about what better means.

Ranked by a mixture of practical consequence, strength of evidence, and relevance to tool-using assistants, these are the nine papers worth attention.

1. True evidence can still answer the wrong question

Lazy Grounding: Attacking Search Agents with Factual Evidence describes a failure that is less theatrical than prompt injection and probably more familiar to anyone who has watched an agent research something incorrectly.

The retrieved document need not lie. It can give a perfectly good answer to a slightly different question.

The user asks for a figure from one year; the source supplies another. The question names a person; the evidence concerns their sibling. The requested identifier is a ZIP code; the retrieved number is a county code. Everything looks grounded until someone checks what the source actually establishes.

The authors construct nearby, answer-changing versions of benchmark questions, create search records supporting those versions, and introduce them into an augmented retrieval environment. Across 12 model–benchmark combinations, accuracy falls by 5.9 percentage points on average, with 11 of the 12 estimated changes negative. The largest drop is 17.3 points.

These are stress-test results, not a measured failure rate for ordinary web search. The records were synthetically introduced rather than published and indexed on the public web. And the “factual evidence” framing has a qualification: in a 40-item manual audit, 34 nearby answers were verified as correct for their rewritten questions.

The mechanism nevertheless matters. Citation checking cannot stop at “does this page say this?” It must also ask “does what this page says satisfy the original request?”

For Jarvis, that distinction belongs near the center of research work. Preserving the year, entity, relationship, units, and requested answer type is not clerical tidiness. It is how an assistant avoids confidently solving the neighboring problem.

A constraint-checking prompt reduced nearby-answer adoption from 20.7% to 14.3% in one experiment. Useful, but hardly solved. The remaining failures are a reminder that an instruction to stay on task is not the same thing as staying on task.

2. Your judge has a noise floor

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints earns attention partly because the researchers did something unfashionable: they set reliability gates, failed them, and stopped rather than finding a more flattering verdict.

In one campaign, 100 byte-identical replay requests reproduced the complete ranking 78 times, below a predefined requirement of 99. Request hashes matched. Recorded response metadata did not reveal a corresponding change. The execution pipeline behaved; the measurement instrument did not meet its specification.

But the most important detail is the metric split. Those same replays achieved:

These are not contradictory results. A complete ordering can change when two nearly tied candidates swap places, while the best candidate remains the same.

The benchmark made that problem unusually acute. Only 16.25% of tasks contained candidates with mixed correctness; in most groups, all four were right or all four were wrong. The judge was often being asked to produce a precise ordering where correctness offered little basis for one.

The lesson is therefore more useful than “LLM judges are unreliable.” It is: measure the resolution your decision actually needs, and establish whether the judge can supply it.

Self-hosting was not a magic escape. In one control, full-ranking agreement among well-formed replies fell from 98.72% in quiet serial operation to 89.25% under concurrent load. That is evidence about one model and serving stack, not proof of the mechanism behind every hosted API discrepancy.

For Jarvis, this matters whenever a model verdict becomes a gate rather than advice. A prompt evaluator, routing classifier, or safety check needs tests of disagreement, malformed output, and realistic concurrency—not just a stable model name in a configuration file.

3. Compile the fuzzy part, keep the exact part in code

Compile by Training: Turning Natural-Language Specifications into Local Neural Functions offers the most immediately appealing systems idea in the set: pay the large-model cost while building a function, rather than every time you call it.

A natural-language specification becomes a task-specific adapter for a shared compact Qwen3-0.6B interpreter. Teacher models generate examples; training refines the adapter; the result is packaged as a .paw artifact containing the adapter, prompt scaffold, specification, and runtime metadata.

This is not natural language turning into ordinary executable code. It is a small neural component acquiring something closer to a software build lifecycle.

On FuzzyBench-Hard, the training compiler reaches 0.836 mean LLM Exact Match, compared with 0.224 for the fast PAW compiler. The subset was deliberately selected for failures of that fast compiler, so the large improvement measures recovery on difficult cases—not general superiority across arbitrary tasks.

A representative cold compilation took 50.9 seconds on a B300, versus 3.5 seconds for the fast compiler. Those are reported service measurements, not universal build times. The paper also lacks the complete runtime cost and device analysis needed to calculate a dependable break-even point.

Still, the architectural boundary is attractive. Teacher APIs are required during compilation; future inputs can run locally without them. That reduces runtime dependence, though it does not make compilation itself private.

Jarvis has plausible uses for this kind of component: a stable classifier, a narrow extractor, or a recurring instruction normalizer. The important division would remain intact: the neural function interprets fuzzy text; deterministic code validates arguments, enforces authorization, and performs the action.

“Compiled” must not become a synonym for “correct.” But versioned neural artifacts with regression tests and rollback are a better engineering abstraction than an unexamined prompt pasted into every request.

4. Better prompts need an editor, not just more amendments

ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize addresses a recognizable failure mode: every mistake adds another instruction, until the prompt becomes a sedimentary record of past embarrassment.

ESPO replaces narrow, incremental mutation with three stages. It diagnoses all training errors together, proposes revisions using several strategies, and selects candidates by their consistency across bootstrap resamples of validation examples.

The proposal strategies are the interesting part. Alongside fixing errors, the optimizer explicitly tries consolidation and ablation: shortening overlapping instructions and removing rules that cause false positives. Not every failure deserves a new prohibition.

Across seven benchmarks, ESPO reports 74.67% average accuracy versus 70.91% for GEPA, with prompts averaging 1,004 rather than 1,878 characters—approximately 47% shorter.

The comparison has important boundaries. Methods begin from deliberately weak prompts. Each benchmark has only 70 training examples and 30 validation examples, and the main results cover three optimization seeds. Bootstrap resampling can probe sensitivity to those 30 examples; it cannot conjure missing coverage.

One ablation is especially useful: diversity alone made performance worse on the tested sentiment task. Generating more candidate prompts does not help if diagnosis and selection are poor. Another experiment found that merely constraining GEPA’s prompt length barely changed accuracy. Brevity was not the whole mechanism.

For Jarvis, the practical lesson is offline maintenance of individual skills: cluster recurring failures, repair the underlying rule, remove contradictory patches, then test against held-out interactions. It is not a license to let a prompt optimizer rewrite the live assistant’s operating constraints because it won a small validation contest.

5. A readable reasoning trace is not a causal map

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning asks whether the reasoning steps that look important are the ones that change what a model is likely to answer.

The authors estimate step importance through repeated continuations from reasoning prefixes. Roughly: after this step, how much does the probability of the eventual answer change?

That is a functional measurement, not a test of logical necessity. A step can be mathematically necessary in a written proof yet add little measured advantage if the model was already highly likely to reach the answer.

The main finding is an asymmetry. Fine-tuned critics recover considerably more importance signal from incorrect traces than from correct ones. Incorrect responses often contain easy cues, such as a consequential final-answer mistake. Identifying the genuinely helpful step in a successful solution is harder.

The trajectory analysis also complicates the familiar story of models reasoning their way to a breakthrough. In the relevant Qwen3-1.7B comparison, thinking mode raises the share of trajectories classified as “high throughout” from 24% to 61%. The authors interpret much of the improvement as a stronger probability of success from the beginning, rather than more mid-trace discoveries.

That does not establish that thinking is unnecessary, or that the answer has literally been computed before generation starts. It describes the continuation probabilities and trajectory categories measured in these experiments.

For Jarvis, the implication is straightforward: explanations should identify evidence, conclusions, and uncertainty without pretending to be a faithful transcript of the machinery that produced them. A passing test is stronger evidence than a paragraph saying “I carefully checked.”

6. Old agent sessions can become new environments

Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments treats an agent session as more than a demonstration to imitate.

File reads reveal workspace contents. Edits expose earlier versions. Writes identify files that did not belong to the initial state. Together, these observations can support reconstruction of a pre-task environment—even without the original repository.

The pipeline first replays visible file evidence, then uses an agent to supply missing context and dependencies. It generates fresh tasks, obtains new solutions, and filters them with generated verifiers. Extensions combine a writable target workspace with a read-only reference, or turn single tasks into persistent multi-round development sessions.

The authors report 37.3 thousand task-sufficient environments. Fine-tuning Qwen3.5-27B on the resulting corpus improves reported Terminal-Bench 2.1 performance by 11.9 percentage points and EvoCode-Bench v2 MT@4 by 13.8 points.

The caveat is embedded in “task-sufficient.” These are not necessarily faithful replicas. In a manual sample of 30 completions, eight introduced substantial unnecessary files or code. The authors found no task solutions in that sample, but a small audit cannot establish absence of leakage across the corpus.

There is also a teacher confound: re-solving uses a stronger, consistent teacher, while source trajectories are heterogeneous. Some improvement may come from better demonstrations rather than environment reconstruction alone.

For Jarvis, the valuable idea is preserving enough pre-change state to re-solve a task, not merely replay a successful transcript. Persistent-workspace failures and recovery are particularly valuable material. But reconstruction is a fallback; when the environment is under our control, capturing the actual initial state is preferable to asking another model to invent the missing pieces.

7. One training question is not one training example

Rethinking On-Policy Distillation of Large Language Models II: One Training Example has the most easily misunderstood headline.

A single query can support hundreds of steps of useful on-policy distillation and recover much of the gain from full-data training. That does not mean a model learns broad capabilities from one labeled sentence.

The student repeatedly generates trajectories, and a teacher supplies dense token-level supervision at the visited prefixes. One fixed question can therefore produce many training states.

In the main state-coverage analysis, one query reaches 71.5% of the clusters visited by full-data OPD. Sixteen semantically diverse queries reach 98.9% and match full-data performance in the reported setting. Sixteen queries drawn from one semantic cluster do substantially worse.

Those percentages describe a particular reference space built from teacher hidden representations, dimensionality reduction, and clustering. They are not percentages of all possible reasoning states. Nor does matching query counts imply matching compute costs: the repeated rollouts and teacher supervision remain substantial.

The result nevertheless changes what “data efficiency” might mean. Counting distinct input questions can badly understate the amount of supervision an on-policy system receives.

For future agent training, the useful hypothesis is to select tasks for the different situations they induce—ambiguity, failed tools, recovery, conflicting evidence—not merely for topical variety. The paper does not establish that a tiny scenario suite is sufficient to train Jarvis, but it gives a reason to care about behavioral coverage rather than prompt count alone.

8. Learn from the teacher, then stop imitating

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR supplies a companion result: teacher guidance and reward optimization may work better as separate stages than as competing signals in every update.

The proposed sequence is simple. First use on-policy distillation to expand the student’s useful solution distribution. Then remove the teacher objective and use reinforcement learning with verifiable rewards to concentrate probability on successful solutions.

The clearest gains appear on procedural logic tasks. With a Qwen3-0.6B-Base student, average logic pass@1 rises from 45.0 with GRPO and 48.9 with OPD to 70.4 with OPD-then-RL. Mathematics results are less decisive: the sequential method leads the aggregate comparison, but several strong alternatives are statistically tied on pass@1.

The experiments share a stated training-step budget, which is not identical to equal wall-clock or monetary cost. Teacher inference, rollout configurations, and baseline tuning still matter.

The authors’ interpretation—coverage first, concentration second—is supported by pass@k and update diagnostics, not proven as a universal law. Its appeal is that it distinguishes being capable of finding a solution across many attempts from reliably producing it on the first attempt.

This is a training result, not an argument for changing Jarvis’s runtime prompting. Its practical relevance begins when there is a real training pipeline and a trustworthy verifier. Without the latter, “optimize beyond the teacher” may simply mean getting better at pleasing a weaker judge.

9. Different explanations beat more copies

Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views asks what kind of repetition helps a model learn.

The answer, in these controlled experiments, is not merely different wording. It is presenting the same knowledge through different explanatory structures: textbook material, blog-style explanations, and question-and-answer discussions.

Using 36 documents across computer science, law, and medicine, the authors compare source repetition, paraphrases, and auxiliary views while holding the knowledge-token budget fixed. Auxiliary views generally improve learning, including factual recall of answers drawn verbatim from the original source—even though the model sees that source fewer times.

The benefit is small at 1B parameters and grows across the tested 7B, 13B, and 32B OLMo-2 models. These are short continued-pretraining experiments, not a demonstration across full-scale pretraining.

An intriguing detail is that stronger generators were not consistently better at producing useful views. Across 11 generator configurations, downstream factual accuracy ranges from 0.405 to 0.422, compared with 0.396 for the paraphrase baseline. That supports “several sufficiently capable generators worked here,” not “teacher quality never matters.”

Together with the one-query distillation paper, this suggests that raw counts—documents, prompts, examples—can conceal the more interesting variable: what different learning situations the data creates.

For Jarvis, this is a possible domain-adaptation technique, not a replacement for retrieval. A service port, deployment state, or credential location belongs in an authoritative source. No amount of elegant explanatory diversity makes stale facts current.

Build cheaper components. Demand better evidence.

The constructive papers offer a coherent direction: narrower neural functions, better-curated training states, reusable environments, and prompts that are edited rather than endlessly extended.

The critical papers supply the conditions under which those improvements deserve belief.

A source must answer the requested question. A judge must resolve the distinction being measured. A reasoning trace must not be mistaken for a causal explanation. A generated test suite must not quietly define success as whatever its sibling generator happened to produce.

These findings do not justify wrapping every model call in five more model calls. The judge-reliability paper is a particularly good warning against that reflex. More sampling cannot repair a badly chosen measurement target.

The better response is to make boundaries explicit: fuzzy interpretation versus exact execution, training versus runtime, reconstruction versus captured state, observed result versus inferred mechanism.

For an assistant like Jarvis, that is the useful destination. Not a system that narrates greater confidence, but one whose claims are easier to check—and whose cheap, specialized components have clearly defined jobs.

Reading list