The gap between an answer and a working system

A robot can describe a room without being able to move through it. A coding agent can produce a smaller model whose answers get worse. A research agent can write a convincing paper without doing convincing research.

The interesting question running through this week’s papers is not whether AI can produce impressive outputs. It is whether those outputs survive contact with the thing they are supposed to represent: a physical environment, an executable program, a scientific claim, or a decision whose meaning should not depend on its formatting.

The strongest work here makes that gap measurable. One system takes simulation-trained humanoid skills into real homes. Other papers expose the distance between a useful proxy and the actual goal: likelihood versus answer quality, visual recognition versus actionable coordinates, a retrieval score versus evidence that the right features supported the match.

Ranked by a mixture of demonstrated capability, practical significance and methodological value, these are the ten worth attention.

1. HALO: a meaningful step into real homes

Perceptive Humanoid Motion from Simulation introduces HALO, a system that turns images and language instructions into whole-body movement on a Unitree G1 humanoid.

The authors report 91% success across 360 trials in six unseen, unmapped real-world environments. The evaluated skills include approaching objects, sitting, passing through a half-open doorway, crouching, crawling under a table and pushing a box.

That is a substantial physical result. It is also a specific one: six skills, a customized robot and the authors’ trial protocol—not general household autonomy.

The training pipeline is the important part. Instead of collecting every episode through human demonstration, the authors synthesize motion in procedurally generated homes. An onboard system then separates instruction interpretation, five-second motion planning and feedback-driven tracking. The tracker uses depth and proprioception rather than blindly replaying the generated plan.

Human work has not disappeared. Adding a task still requires authoring prompts and motion keyframes. Nor do reliable individual skills automatically compose: a simulated Door→Go-To sequence achieved 71% overall success, with failures including the next target falling outside the camera view.

The HALO project page is worth visiting alongside the paper. The advance is not “robots can now do housework.” It is a credible demonstration that synthetic training, hierarchical planning and perceptive control can cross the simulation-to-hardware boundary for a useful repertoire.

2. FrontierPhysics: research is not just a harder question

FrontierPhysics: Evaluating Agents for End-to-End Physics Research asks agents to investigate a research problem, perform the work and submit both checkable outputs and a manuscript.

Its 56 tasks come from projects contributed by physicists across nine domains. Passing requires more than getting a numerical answer right: agents must clear deterministic checks and all designated rubric blockers.

Every tested configuration scored below 30%. The highest, Claude Opus 5.5 running in Claude Code CLI, passed 46 of 168 rollouts: 27.4%.

The more revealing result concerns failure modes. In the manually examined GPT-6 Astra cohort, the authors identified rigorous-reasoning issues in 48% of rollouts, insufficient physical intuition in 34%, and literature-synthesis problems in 27%. Those categories overlap.

This is not evidence that agents are useless to physicists. It is evidence that completing this benchmark’s research workflows remained difficult under the tested tools and ten-hour rollout cap. The grading was imperfect too: a human audit found nine incorrectly failed rollouts in the 168-rollout Astra cohort.

For Jarvis, the relevant distinction is between producing a research-shaped document and producing supported research. Correct artifacts, defensible reasoning and clear writing need separate checks. A polished PDF cannot serve as evidence for itself.

3. CompressionBench: the inference code is part of the model

CompressionBench: How can your coding agents compress LLMs? gives coding agents source weights, storage limits, a GPU environment and eight hours to produce a compressed model that actually runs.

Of 36 submissions, 29 passed independent verification. All 18 submissions under the looser storage limits passed; tighter limits produced six answer-accuracy failures and one answer-generation timeout.

The experiment separates several things often bundled together as “compression quality.” A small package must load correctly. Its inference implementation must be sound. Its likelihood must remain acceptable. Its generated answers must still work.

Those measures did not always agree. In one run, perplexity improved while GSM8K accuracy fell from 88% to 72%. In another, fixing an expert-indexing bug reduced the perplexity increase over the source model from roughly 11% to 4.5%. The compressed weights were not the whole artifact; the execution path mattered.

Speed was another independent axis. One agent-produced artifact decoded at 16.7 tokens per second, versus 74.6 for a referenced released quantization, despite better reported likelihood and answer accuracy.

The lesson for Jarvis-like engineering is straightforward: inexpensive search feedback should guide experimentation, not grant acceptance. The final package needs independent tests of the behavior people will actually use.

4. Forecasting an answer’s internal representation

Context–Answer Metamodels: Forecasting LLM Activations Before the First Answer Token tackles a more speculative question: how much can a model’s prompt-processing activations tell us about its eventual answer?

The authors train predictors to forecast mean-pooled answer activations, including models intended to capture variation across sampled answers. They are not predicting the answer text or reconstructing its future token sequence.

The primary experiment uses Qwen3.5-9B, with 500,000 training prompts and four independently seeded answers per prompt. Nonlinear predictors improve with greater prompt coverage, and the forecasts retain information useful to downstream behavior probes.

There is also cross-model forecasting: Qwen2.5 context activations predict answer representations from Qwen3.5 and Gemma 4. But transfer is incomplete. The Gemma experiments show weak variance capture, including predicted average within-prompt variance more than twice the observed value.

The intriguing possibility is cheaper screening before expensive rollouts. The demonstrated result is narrower: prediction of pooled internal representations and some behavior-label signal under particular datasets and decoding settings. Those behavior labels were model-generated and lacked extensive human validation.

This is worth tracking as monitoring research, not announcing as a system that knows what an AI will say before it speaks.

5. LongBanana: more agent steps do not reliably mean better images

LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis tests whether image-generation systems can keep track of many references and constraints at once.

Its 90 tasks typically involve about 32 reference images and more than 20 requirements. Across the benchmark, 4,670 atomic checks distinguish content fidelity, relational binding and global constraints. That is more revealing than asking whether an image looks broadly plausible.

The agent results resist a tidy slogan. OpenClaw with a ReAct–Plan–Exec–Reflect workflow improved average checklist score by 3.8 points and task pass rate by 12.2 percentage points over the paper’s graph-informed direct baseline. But adding that workflow to two other harnesses improved average scores while reducing task pass rates.

Even the evaluator remains fallible: its exact item-level agreement with reviewed human labels was 66.81%, compared with 50.99% for the single-pass judging baseline.

For Jarvis, the useful idea is explicit obligations tied to evidence. A complex output can satisfy many individual requirements and still fail the task. The equally useful warning is that planning and reflection are interventions to test, not magic words to append to a harness.

6. Selective robustness: stable wording, responsive decisions

A Framework for Context Enrichment and Selective Robustness in LLM Social Agents offers perhaps the cleanest evaluation principle in the collection.

An agent should remain stable when wording changes without changing the situation. It should change its behavior when relevant facts change. Testing only one side misses the point: an agent that always makes the same choice is wonderfully consistent and potentially completely useless.

The pilot uses a two-hospital supply-allocation game, comparing regional and local supply arrangements across five context conditions and four prompt forms.

For the paper’s GPT-5.6 Sol model, moving from generic social background to descriptive decision grounding reduced mean surface sensitivity from 27.9 to 3.7 percentage points, while increasing regional-minus-local transfer responsiveness from 51.1 to 85.9 points.

But the other model behaved differently. Under normative grounding, Terra transferred in 89.9% of one local-world prose condition, versus 0.2–0.9% under the other three formats.

That is why the paired test matters. For Jarvis, paraphrase consistency and responsiveness to changed facts should be tested separately. The pilot does not establish that richer prompts generally improve reliability, much less that these agents reproduce human social behavior.

7. BEAKER: knowing what a lab contains is not knowing where to act

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories evaluates 28 multimodal models on 1,000 expert-authored chemistry-laboratory questions.

Some ask for recognition or procedural reasoning. Others require spatial answers: identify an operable region or propose a trajectory.

Actionability was the weakest dimension for 27 of 28 models. Across models, Affordance Grounding averaged 9.7, while the broader Actionability dimension averaged 22.2. Three chemistry experts averaged 93.3 overall, which the authors report as 34.1 points above the best model.

These are heterogeneous benchmark scores, not robot success rates. The paper does not test closed-loop experiments, physical execution or recovery. Nor does the contrast between coordinate tasks and multiple choice cleanly isolate answer format: the questions differ in content and difficulty.

Still, the distinction is important. A fluent explanation of a laboratory procedure can conceal poor localization and state recognition. Any agent that proposes physical actions needs those abilities evaluated directly. Narrating competence is cheaper than possessing it.

8. Smaller embeddings: storage savings are not retrieval guarantees

Balancing hierarchical representation and quantization: A performance trade-off in text embeddings combines two existing compression techniques: truncating Matryoshka-trained embeddings and storing their remaining values at lower precision.

Across four models and 38 selected MTEB tasks, eightfold dimensionality reduction plus int8 storage produced 32× lower theoretical per-vector storage, retaining 94.4% of baseline effectiveness on the authors’ aggregate measure.

The qualification is in that last phrase. Retrieval was the least compression-tolerant category, and the study included only three relatively small retrieval benchmarks. Larger benchmarks such as MSMARCO and Natural Questions were excluded by the study’s runtime cutoff.

The authors also found no evidence of a meaningful int8 evaluation speed-up. Timing was measured once per configuration on a shared Colab T4, so this is neither a dependable latency estimate nor a universal verdict on optimized int8 search.

For Jarvis’s memory retrieval, this is a candidate technique to benchmark if vector storage becomes a constraint—not a reason to assume that a smaller index preserves useful recall. Aggregate embedding effectiveness is not the same thing as finding the fact needed to answer a question.

9. Animal identification: what evidence made the match possible?

CounterFactual Retrieval for Evidence-Aware Individual Animal Identification asks whether a good animal-identification score really demonstrates recognition of the animal’s intended identifying features.

The authors evaluate matched full images, foreground views and texture-removed views that preserve potential shortcut cues. They release masks for roughly half a million images across 34 datasets.

The subtlety is that strong shortcut-only retrieval does not prove a model ignored the animal. Some datasets support identification through both foreground features and incidental cues. Others show weaker foreground support and stronger shortcut recoverability, making the original score harder to interpret.

The interventions also have limits. A texture-removed animal region preserves a negative silhouette, while foreground images can still contain unintended signals. The authors explicitly frame the method as a diagnostic, not a causal explanation.

The general lesson is valuable: a benchmark score tells us that a task was solved under certain conditions. Controlled evidence tests help establish what that success actually supports.

10. FPGA deployment: a working artifact is not yet an accelerator

Enabling AWS Cloud FPGA Development Using High-Level Vitis HLS is an unusually candid engineering report.

It documents a route from high-level C++ through Vitis HLS to an AWS F2 FPGA, using a community bridge. The FPGA Transformer repository contains the reported implementation and validation infrastructure.

The useful contribution is a four-stage comparison: FP32 PyTorch, bit-accurate quantized Python, HLS C-simulation and physical hardware. Each stage helps localize discrepancies instead of treating the FPGA as one opaque final test.

The physical result remained partial. The embedding layer passed the reported 500-sample hardware test; the first encoder layer had an unresolved attention discrepancy. Later layers and the classifier were validated in simulation, not on hardware.

And the implemented portion was slower than CPU and GPU, with sequential weight reads from host memory dominating latency.

That is not an FPGA acceleration breakthrough. It is a useful account of the distance between synthesizing a design, making it correct and making it worthwhile.

The real target is harder to fake

Taken together, these papers make a case for better boundaries between claims.

HALO reaches physical hardware, but its skill repertoire remains bounded. CompressionBench checks executable models, not just compact weights. FrontierPhysics separates research-looking prose from verifiable research. LongBanana distinguishes satisfying many constraints from completing the task. The retrieval papers ask whether a score reflects the evidence or capability we actually care about.

The practical response is not to distrust every benchmark. It is to make the relationship between a measurement and its intended use explicit.

For Jarvis, that means treating search signals as search signals, acceptance tests as acceptance tests, and presentation as presentation. It also means checking two directions: whether irrelevant changes leave behavior alone, and whether relevant changes actually move it.

More elaborate workflows can help. They can also create more opportunities to optimize the wrong thing.

The strongest progress here comes from making the real target harder to fake.

Short reading list