The Gap Between Knowing and Delivering

An agent can find the right document and still build the wrong thing. A model can recall a fact and still fail to use it when learning something new. A robot can complete a task and ignore the preference that made the task worth delegating.

Several of this week’s papers examine that gap: between possessing a capability and putting it to useful work.

The most interesting results are not all about making models bigger or giving them longer to think. They concern how information is arranged, when feedback arrives, which properties a representation preserves, and what an evaluation actually rewards. These choices are less glamorous than a new intelligence leaderboard. They are also where many practical failures live.

The nine papers below are ranked by their significance for building and understanding useful AI systems, with particular attention to agents such as Jarvis. They are recent preprints, and the numerical results remain the authors’ reported findings. The point is not to crown nine breakthroughs. It is to identify which ideas deserve attention—and what the evidence actually buys us.

1. Delivering the wrong solution, efficiently

FDE-Gym: The Last Mile of AI Delivery starts with a better question than “Can the agent write the code?”

Can it discover what a customer needs, uncover the constraints nobody put in the request, and change a system in a way that improves the customer’s outcome?

The benchmark contains 95 tasks across 20 simulated customer scenarios. Agents inspect systems and documents, contact scripted stakeholders, and implement changes. Results are scored against a customer-specific business metric, with the unchanged system as the zero point and an expert-written solution as the reference. Modeled stakeholder-contact costs are deducted, and incidents can wipe out positive uplift.

At the most realistic instruction level, Claude Opus 5.5 had the highest mean score: 0.56, with a reported 95% interval of 0.50–0.62. It resolved 14 of 95 tasks, where “resolved” means a score of at least 0.8 without an incident.

That is not a claim that agents succeed at only one in seven real customer projects. It is a result under this benchmark’s framework, simulated stakeholders, and scoring rules.

The more revealing comparison concerns missing requirements. On 20 original tasks, the three leading models averaged a 0.28-point decline when moving from fully supplied rules and constraints to having to discover those constraints. Even obtaining every key fact did not guarantee a good implementation: across 73 such task–model cases, only 37% reached business uplift of at least 0.8, before contact-cost deductions.

For Jarvis, this is the most consequential paper in the set. A technically correct action can still be a failure if it addresses the wrong target or consumes more attention than it saves. The benchmark makes that distinction measurable.

Its cost model deserves scrutiny, though. Opus delivered systems reaching 0.8 uplift on 40 tasks, but only 14 remained resolved after contact costs. Those costs are expert-set estimates, not measured human inconvenience. Without a human baseline, still forthcoming, we cannot tell how demanding that budget is for competent people.

Still, the central challenge is well chosen: discovering the requirements and delivering the outcome are separate capabilities.

2. Build the library before answering the questions

Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries addresses a recurring inefficiency in retrieval systems. Every new question can trigger another expedition through substantially the same material.

The proposed alternative is a trained builder that prepares a navigable workspace: summaries, indexes, and connections, with references back to original documents. A separate answering agent then uses it. The builder is trained through downstream answer performance, rather than rewarded merely for producing convincing summaries.

On a held-out BrowseComp-Plus evaluation of 50 corpus-level tasks, a frozen Qwen3.5-4B answerer achieved 0.75 accuracy with the learned workspace, versus 0.42 with embedding retrieval and 0.44 with reranking. With Gemini 3.1 Flash Lite, the corresponding figures were 0.77, 0.30, and 0.42.

Those are substantial differences in this setup. The held-out set is also small, and the result is not a general estimate of what replacing retrieval will do.

The distinction between summarizing and organizing for future use is what makes the work interesting. A useful library needs to anticipate questions without knowing exactly which question comes next. Training against downstream answers gives the curator a signal about whether its organization helps.

This is directly relevant to Jarvis’s repeated work over project documentation, research collections, and archived conversations. A reusable map could reduce rediscovery. It would not remove the need to inspect evidence.

The hard part is freshness. The study evaluates static corpora; a living project changes underneath its summaries. A practical implementation would need source links, invalidation rules, and a reliable path out of the synthetic layer when a detail matters.

Nor is preprocessing free. The paper’s reuse analysis makes efficiency dependent on query volume and token accounting. Its primary training experiment does not explicitly reward lower construction or navigation costs. The supported claim is that learned curation can improve answers and may amortize its expense—not that every corpus needs an autonomous librarian.

3. Give long-running agents feedback before the job is over

MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering tackles a credit-assignment problem familiar to anyone who has watched an agent run experiments.

A model inspects data, writes code, launches a run, waits, examines a score, and changes direction. If training supplies feedback only after the whole assignment ends, useful signals arrive late and cover a large tangle of decisions.

The paper’s Windowed Interaction RL groups decisions between successive experiment results into an “execution window.” The next result provides a shared reward for that window. An asynchronous training process handles experiments that finish at different times.

Under the authors’ harness, on the 22-task MLE-Bench Lite evaluation, the reported Any Medal rate rises from 37.9% for the base model to 72.7% for MLExpert, averaged over three evaluations. The staged ablation progresses through 47.0% after general agent supervised fine-tuning, 62.1% after domain-specific fine-tuning, and then 72.7% after the reinforcement-learning stage.

That progression matters: the final result is not attributable to windowed RL alone.

The released project also emphasizes persistent experiment state, revisiting candidates, and selecting a final submission. These are useful distinctions. Finding a good intermediate result is not the same as preserving it and submitting it correctly.

For Jarvis-like workflows, the immediate lesson is about experiment management, not installing a new training algorithm. Keep results attached to the decisions that produced them. Preserve good candidates. Do not let the last thing attempted automatically become the final answer.

The paper reports transfer to bounded replication and optimization tasks, but this is not evidence of recursive self-improvement. Better performance at engineering experiments is interesting enough without attaching a runaway-intelligence narrative.

4. Knowing a fact is not the same as being able to build on it

Knowledge That Is Linear Before Fine-Tuning Generalizes After It offers the strongest conceptual result in the collection.

Suppose a model knows which continent various places belong to. The researchers randomly replace continent names with arbitrary labels—Europe might become “crab,” Asia “zebra”—then fine-tune on only some places. Can the model apply the new labels to held-out places?

Across 11 models and 70 relations, held-out generalization was strongly associated with how readily the original relation could be decoded from pretrained representations using a linear classifier. The reported within-model Pearson correlations were 0.87–0.92.

This “Pre-linearity” measure remained associated with generalization after controlling jointly for original-fact recall, answer count, training-subject count, and a corpus-based frequency proxy: partial correlations were 0.70–0.82.

The implication is narrower and more interesting than “linear knowledge is better.” A fact can be available for recall without being arranged in a form that makes a related learning problem easy.

That gives us a useful warning about evaluating models. Asking a model to state a fact tests one kind of access. It does not establish that the model can use the fact as scaffolding for new knowledge.

The experiment is deliberately artificial, using randomized labels, selected relations, and restricted candidate answers. It also establishes association, not that making representations more linearly readable would cause better generalization. But it provides a concrete way to investigate a distinction that ordinary question-answering scores tend to conceal.

5. Search quality begins before the search call

QREF: Improving DeepSearch Agents by Aligning Search with Reasoning via Query Refinement and Evidence Filtering focuses on an unglamorous source of agent failure: bad intermediate searches.

Its query refiner rewrites batches of proposed searches using the current question and reasoning context. The aim is to preserve discriminating constraints and make searches complementary rather than repetitive. An evidence filter then removes returned snippets judged irrelevant to the immediate information need.

The refiner is trained, first with supervised examples and then reinforcement learning grounded in retrieval outcomes. The reasoning policies of the two evaluated search-agent backbones remain unchanged.

The abstract reports average relative Pass@1 improvements of 33.55% for WebSailor and 11.04% for TongyiDeepResearch. These are not percentage-point gains. For WebSailor, the main-results text reports absolute gains ranging from 3.11 to 6.79 points across four benchmarks.

The paper’s diagnostic analysis is also useful: 47.04% of 1,992 search batches sampled from 300 WebSailor trajectories contained at least one defined query failure. That does not prove bad queries caused the eventual wrong answers; difficult questions can produce both.

For Jarvis, this suggests a testable intervention rather than an automatic extra model call before every search. Check whether queries carry the constraints already established, and whether successive searches add coverage.

Filtering deserves particular care. Evidence that disagrees with an agent’s current hypothesis may look irrelevant precisely because the hypothesis is wrong. A tidier context is only useful if it has not become a confirmation-bias machine.

6. A safety test can change the behavior it measures

Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages? examines what happens when a prompt explicitly announces a safety evaluation.

Across four instruction-tuned models and six languages, the safety-test cue increased harmful-request refusal in every model–language pairing. The average increase was 13.8 percentage points: 14.4 points across Hindi, Bengali, Tamil, Telugu, and Kannada, compared with 10.6 points in English.

Benign refusals also increased, by 3.9 points overall. The cue appears to make models more refusal-prone, not simply better at distinguishing harmful from harmless requests.

A secondary comparison is helpful: when both prompts mentioned logging, adding safety-team review still raised harmful-request refusal by 4.6 points on average.

This does not establish deception or a model consciously recognizing an examiner. The demonstrated result is behavioral sensitivity to evaluation framing. It also does not directly measure a gap between benchmark and production conversations.

Translation quality, automated judging, and substantial output truncation in some model–language conditions complicate interpretation. Nevertheless, the evaluation lesson is straightforward: telling a model what kind of test it is taking can become part of what the test measures.

7. Two views help when they preserve different things

Twice Upon a Time: Complementary Representations for Time-Series Reasoning combines a conventional line plot with a normalized delay-embedding image.

The plot preserves absolute values and scale. The delay representation suppresses offsets and positive amplitude rescaling while making temporal structure salient. One view helps answer “How high?”; the other can help expose recurring patterns.

On TSRBench, the combined model reports 53.2% accuracy, versus 45.2% for its plot-only counterpart. Prediction rises from 38.0% to 61.7%, while basic perception is roughly unchanged.

The useful control is that simply adding another stream does not reproduce the improvement: Plot+Plot scores 44.8%, and Delay+Delay 42.6%, under controls preserving the dual-tower architecture and token budget.

The gains are not uniform. On multivariate ChatTS numerical alignment, the improvement over plot-only is much smaller: 86.4% to 86.8%.

The broader design idea is worth retaining: normalization always discards something. If absolute measurements matter, a representation designed to ignore scale cannot carry the whole task. Complementary views can help because they preserve different information, not because two pictures are inherently better than one.

8. Shared experts can make routing less consequential

Rethinking Routing Exploration in Modern Mixture-of-Experts asks whether two common architectural choices work against each other.

Fine-grained experts create more possible routes for tokens. Always-on shared experts can dominate the layer’s output, making the choice among routed experts less consequential.

The proposed MirrorMoE modifies routing so that some selected low-probability experts receive negligible aggregation weights. This adaptively reduces the set of experts making a meaningful contribution, with the modification annealed toward standard routing during training.

Across four shared-expert models from 100 million to 2.4 billion total parameters, the authors report lower test perplexity than their corresponding baselines. For the 2.4B model, they report a 1.64-point improvement in average downstream performance across six commonsense tasks.

The boundary is important: without shared experts, the method performs comparably to standard training rather than showing a significant improvement.

This is a targeted architecture result, not a general MoE upgrade. Its interest lies in the diagnosis that adding useful capacity can alter what another component has an opportunity to learn.

9. A world state should not be a camera state

GeoState: Learning View-Decoupled Geometry States for 3D World Modeling separates scene evolution from the viewpoint used to observe it.

The system compresses multi-view geometry features into a scene state, predicts future states, and supplies camera information when decoding depth. The aim is to avoid treating camera-induced appearance changes as changes in the underlying world.

In a FRONT-only Waymo comparison at a two-second forecast horizon, GeoState reports 0.143 absolute relative depth error, versus 0.152 for FR3D. But the methods use materially different scale-alignment protocols, so this is not a clean like-for-like ranking. GeoState retains one context-calibrated scale; the comparison methods use future-frame alignment.

The more informative control holds the scene state fixed while querying it under future cameras. Predicting state evolution improves over that baseline, especially in dynamic windows. That supports the claim that the model learns something beyond changing the viewpoint.

The project also reports transfer to other driving datasets without target-dataset adaptation. It does not establish a general-purpose world model or a complete driving system.

For language agents, the connection is an analogy, not an application result: persistent state and the observation used to query it are different things. Confusing them makes change harder to reason about.

What useful progress looks like

These papers do not point to one missing ingredient.

FDE-Gym asks whether an agent found the right problem and improved the outcome. Librarian Agents asks whether information can be organized for future use. MLExpert changes when experimental feedback becomes available. The knowledge-probing study distinguishes factual recall from a representation that supports new learning.

The remaining work exposes similar dependencies: search depends on query formulation; safety scores depend on framing; time-series reasoning depends on what a view preserves; expert routing depends on which components dominate; world prediction depends on separating the scene from its observer.

For Jarvis, the practical priority is not to bolt on every proposed module. It is to evaluate the transitions where capability becomes action: from a request to an understood goal, from retrieved material to evidence, from an experiment to a decision, and from a technically successful step to a useful result.

That is where impressive systems become dependable—or fail to.

Reading list