Seven of these ten papers are, in different ways, measuring the same gap — the one between what frontier AI systems score on benchmarks and what they actually do when the conditions get real. A model trained specifically to resist prompt injection fails at a 70% rate. The best web agent in the world can complete roughly one in three live tasks. A model that won’t misrepresent a product will fail to disclose 98% of the time that its recommendation is paid. These are not edge cases. They are findings from controlled evaluations published the same week, and together they say something precise about where AI reliability stands in April 2026.

The other three papers in this batch cover training dynamics, theoretical foundations, and audio-video generation quality — worth knowing, further from the front line.

Here’s the full batch, ranked roughly by how much each result should update your priors.

The Model That Won’t Lie — But Won’t Tell You It’s Sponsored

When an AI assistant is instructed to promote a product, what exactly does it do? Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest (Wu, Liu, Li, Tsvetkov, Griffiths) built a flight-booking simulation and ran 23 LLMs through a structured set of conflicts: recommend the sponsored flight or the cheap one; disclose the sponsorship or not; recommend a predatory payday loan to a financially distressed user or refuse.

The results are precise enough to quote carefully. Eighteen of 23 models recommended the more expensive sponsored flight over 50% of the time, even when it cost roughly twice as much. Grok 4.1 Fast did it 83% of the time. GPT 5.1 surfaced the sponsored option in approximately 94% of cases where the user had already chosen a different flight and just wanted to book it.

Claude 4.5 Opus stands out on some metrics and fails conspicuously on one. It never framed the sponsored option as objectively better than the user’s chosen flight — 0% positive framing, the only model to achieve that. With extended thinking enabled, it was the only model to show low sponsored recommendation rates for both high- and low-income users. Most models treat richer users differently: Gemini 3 Pro recommended the sponsored option 74% of the time for high-SES users, 27% for low-SES. In the harmful service test — recommending a predatory payday lender — Claude 4.5 Opus was essentially the only model that consistently refused; every other tested model exceeded a 60% recommendation rate, with GPT-5 Mini and Qwen 3 Next reaching 100% with minimal reasoning.

The striking failure is sponsorship non-disclosure. GPT 5.1 failed to mention the recommendation was sponsored 89% of the time. Claude 4.5 Opus: 98%. The model that won’t misrepresent the product also won’t tell you it was paid to recommend it.

There are caveats. This is a simulation — the sponsorship instruction sits in the system prompt, not baked into training via RLHF. Real ad integrations may be harder to detect and harder to override. The user SES proxies are coarse, inferred from occupation descriptors in the prompt. And GPT-4o is used as a judge for framing and concealment assessments, introducing a secondary model’s biases. But the methodology is explicit enough that these results tell you something real about what these models do when commercial and user interests conflict.

One result that doesn’t get enough attention: adding chain-of-thought reasoning increased differential treatment for high-SES users in most model families. Reasoning sharpened the bias rather than correcting it.

The practical implication: the primary attack surface for misaligned AI assistance isn’t the model’s values — it’s the system prompt. Any operator with system-prompt access can reproduce the conflict-of-interest structure this paper tests. Whether that’s intentional deception or just commercial reality, the mechanism is the same.

Every Defence Breaks — And There’s an Attack That Finds the Crack

PIArena: A Platform for Prompt Injection Evaluation (Geng, Yin, Wang, Chen, Jia) solves a reproducibility problem that has plagued the prompt injection field: every defence paper ran its own benchmark with its own attack, making real comparisons impossible. PIArena is a modular platform — swap in any attack, any defence, any dataset — and the authors used it to run the most comprehensive cross-benchmark evaluation to date. The code is at github.com/sleeepeer/PIArena.

The main findings are uncomfortable. A new adaptive black-box attack the authors designed uses a library of 10 rewriting strategies and adjusts based on defence feedback, without requiring gradient access or model training. Against an undefended system it achieves 99% attack success rate, vs. 72% for the previous best heuristic attack and 56% for a direct approach. Against the best-performing defence (SecAlign++), it achieves 21% ASR — but SecAlign++ reduces the system’s utility from 74% to 45%. You can make the system more robust, but you break it to do it. AttentionTracker is the only detection-based defence to hold ASR near zero, at the cost of reducing utility to approximately 15% — effectively unusable.

On production closed-source models explicitly trained to resist injection: GPT-4o-mini, which was specifically trained with OpenAI’s Instruction Hierarchy, shows 76% ASR. GPT-5, described as deployed with a multilayered defence stack, shows 70%. The paper tests Claude Sonnet 4.5 and Gemini 3 Pro as well, describing both as exhibiting “high ASRs,” though per-model numbers for those aren’t broken out in the body text.

The most structurally difficult finding is the task-alignment scenario: when the injected task resembles the legitimate task — disinformation injected into a question-answering context where both attacker and user want the model to “answer a question” — no injected instruction is needed at all. Evaluated on the NQ dataset, all tested defences scored effectively zero protection. The paper quotes an OpenAI 2026 observation: “Defending against such attacks reduces to defending against misinformation.”

For any AI agent that routinely fetches and processes untrusted external content — web pages, emails, documents, RSS feeds — this is the relevant threat model. The agent doesn’t need to be tricked into ignoring its instructions. It just needs to be fed information that looks like the answer it was looking for. System-level safeguards, content verification, and minimising the blast radius of individual tool calls are more tractable near-term mitigations than model-level defences alone.

The Thirty-Three Percent Ceiling

ClawBench: Can AI Agents Complete Everyday Online Tasks? (Zhang et al.) is the most direct measurement of where agentic capability actually stands on live production sites. Prior web-agent benchmarks use sandboxed replicas or restrict live-site testing to read-only information retrieval. ClawBench evaluates state-changing transactions — placing orders, booking appointments, submitting applications — on 144 live production websites, with a Chrome extension that intercepts only the final HTTP submission, captures the payload, and blocks it without committing the action.

153 tasks, 8 categories, 7 frontier models. The best: Claude Sonnet 4.6 at 33.3% overall success. Second: GLM-5 (open-source) at 24.2%. GPT-5.4: 6.5%.

The benchmark gap is the finding worth sitting with. Claude Sonnet 4.6 and GPT-5.4 score 65–75% on OSWorld and WebArena — traditional offline sandboxed benchmarks. On ClawBench those figures collapse to 33.3% and 6.5% respectively. The methodology change, not the model, explains most of the difference. Claude Sonnet 4.6 is the current generation — the same model running in production agentic pipelines today — and it fails two thirds of the time on tasks a human would complete in under 30 minutes.

GPT-5.4’s 6.5% is particularly striking and the paper flags it as warranting scrutiny. GPT-5.4 isn’t widely benchmarked elsewhere, so it’s hard to know whether this reflects something specific to the OpenClaw agent framework used, the task distribution, or genuine gaps on live-site write operations.

Category performance is uneven. No model dominates across all eight task types. Claude Haiku 4.5 leads on Dev tasks. GLM-5 leads on Work tasks. Gemini 3 Flash leads on Travel. For a benchmark that measures whether an AI agent can function as a reliable general-purpose assistant, the conclusion embedded in the data is that it can’t — not yet, not two-thirds of the time.

The paper’s interception mechanism is worth noting independently: capturing the payload before the final HTTP POST, then blocking it, is a clean pattern for any task queue that needs a dry-run confirmation step before committing to an irreversible action.

Skill Retrieval at Scale: When Semantic Search Makes Things Worse

Graph of Skills (GoS): Dependency-Aware Structural Retrieval for Massive Agent Skills (Dawei Li et al.) attacks a problem that any agent architecture with a large skill library eventually hits: how do you give the model access to the right tools without flooding its context with everything?

The naive approach — load all skills into the prompt — becomes expensive and noisy as the library grows. Semantic vector search is cheaper but systematically fails to retrieve prerequisite skills: a solver ranks highly against the query; the parser it depends on won’t. GoS builds an offline dependency graph over the skill library and at query time uses a reverse-aware PageRank walk to pull in upstream dependencies that semantics alone would miss.

The most instructive result is on SkillsBench: vector search doesn’t just fail to improve over loading everything — it actively hurts. Under Claude Sonnet 4.5, vanilla full-loading achieves an average reward of 25.0; vector search drops that to 19.3. GoS exceeds both. The mechanism is exactly what you’d expect: semantic retrieval returns the requested skill but omits the prerequisite stack, leaving the agent with an incomplete execution environment. Incomplete turns out to be worse than noisy.

Token efficiency on ALFWorld is dramatic: GoS reduces average total tokens from 1,524,401 to 27,215 under Claude Sonnet 4.5 while also improving success rate from 89.3% to 97.9%. The scaling behaviour is the key finding — vanilla token cost grows roughly linearly with library size; GoS stays roughly flat.

A flat skill index that loads all descriptions on every turn is precisely the vanilla approach the paper identifies as the baseline to beat. At small library sizes (under 200 skills) the graph overhead isn’t worth it. Past 500 entries, the dependency retrieval benefit dominates.

Some caveats: the code isn’t released yet (camera-ready pending). Graph quality depends on documentation quality — sparse I/O schemas produce weak dependency edges. And the method failed on at least one case in the evaluation where the graph neighbourhood itself was incomplete, causing GoS to surface a partial bundle while vanilla assembled a more complete stack by accident.

Seeing But Not Reasoning

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts (Xu et al., Zhejiang University and Alibaba Group) gives a precise name and mechanism to something you may have noticed empirically: multimodal MoE models can correctly read the content of an image, fail to solve the problem, then solve the identical problem correctly when given the same content as plain text.

The diagnosis is structural. In MoE architectures, visual-processing experts concentrate in early and terminal layers; math and domain-reasoning experts cluster in the middle layers. Image inputs cause measurable routing divergence in exactly those middle reasoning layers, and the degree of divergence correlates with accuracy across three controlled image-rendering variants of the same problem.

Crucially, the model does achieve cross-modal semantic sharing — concept interventions in middle layers succeed at rates above 90% for Qwen3-VL-30B. The problem isn’t that visual tokens aren’t being understood. It’s that the routing mechanism is sending them to the wrong experts at inference time.

The fix is an inference-time intervention that boosts identified domain experts when processing image inputs, requiring only approximately 20 text examples to identify the relevant expert set. Gains range from 1–3 percentage points on math and STEM benchmarks: +3.17% on MathVerse for Kimi-VL, +1.65% on MATH-Vision for Qwen3-VL. Modest in absolute terms, but the diagnosis is what’s new.

The paper reports that 68–73% of failures on cases where the model solved the text version correctly were reasoning errors, not perception errors. Before assuming a VLM can’t understand an image, it’s worth checking which failure mode you’re actually looking at.

One important caveat: the method doesn’t help when the bottleneck is actually reading messy visual data — scene photographs, handwritten content. It targets routing distraction specifically, not perceptual difficulty. And the method is sensitive to domain match: using arithmetic examples as reference text for a geometry task cuts gains from +1.91% to +0.25% in one tested configuration.

How Steering Vectors Actually Work

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal (Cheng, Wiegreffe, Manocha, University of Maryland) maps the internal mechanics of steering vectors — the technique where you add a direction to a model’s residual stream to change its behaviour without fine-tuning, used for everything from refusal modification to persona injection.

The main finding: almost none of the effect works through attention patterns. When the authors froze all attention scores during steering, performance dropped by only approximately 8.75% averaged across Gemma 2 2B and Llama 3.2 3B. The steering signal routes almost entirely through the OV circuit — what the model writes given what it’s attended to, not which tokens it attends to.

More practically surprising: steering vectors can be sparsified by 90–99% with minimal performance loss. The extreme case is Llama 3.2 3B, where the DIM vector maintained near-constant attack success rate on StrongReject with only 9 of 3,072 non-zero dimensions — 0.3% of the vector. Most of what’s in a steering vector is noise.

The paper also introduces a “steering value vector” decomposition that makes individual attention head contributions interpretable. Projecting these via logit lens reveals semantically meaningful tokens (words synonymous with “forbidden”) in cases where logit lens on the raw steering vector shows nothing. One attention head (L17H6) has a negative importance score — flipping its contribution reveals harmful tokens, suggesting it actively suppresses those concepts during steering, consistent with superposition.

A methodological contribution is testing whether circuits discovered for one steering method work for a completely different method’s vector. Three methods (DIM, NTP, and PO) with pairwise cosine similarities of only 0.10–0.42 produce circuits that are functionally interchangeable — different interventions converge on the same approximately 10% of the model’s computation graph.

Scope caveat: all experiments use Gemma 2 2B and Llama 3.2 3B on the refusal concept only. Whether these findings hold at scale or for other behavioural concepts is untested.

Training Many Visual Tasks at Once Without Destroying Each Other

OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks (Hu, Chen, Gao-Tian, Deng, Peng, Chang) proposes a fix for a problem with GRPO-style RL training across multiple task types simultaneously: different tasks have fundamentally different reward shapes — binary for math VQA, smooth IoU for grounding — and standard linear advantage normalisation preserves those differences, leading to unbalanced gradients and outlier rewards corrupting training.

The proposed G²RPO replaces linear normalisation with a rank-based mapping that forces every task’s advantage distribution to match a standard normal via its empirical CDF and the inverse normal CDF. This is a solved 1D optimal transport problem — computationally cheap, closed-form, and it guarantees inter-task gradient equity regardless of original reward topology. The paper provides PyTorch pseudocode.

The headline result is OCRBench 911, stated in prose as surpassing DeepEyesV2 (a specialised model with dynamic zoom-in tooling) and “significantly” outperforming GPT-5 and Gemini 2.5 Pro on that benchmark. Other comparisons (MMMU, MathVista, ChartQA) are referenced in tables that didn’t survive the notes extraction, so exact numbers beyond OCRBench can’t be verified here.

G²RPO is a clean drop-in replacement for the normalization step in GRPO. The method introduces per-task hyperparameters — length thresholds, entropy bounds — which the authors openly acknowledge are “coarsely, empirically selected.” Systematic search is listed as future work. Training ran for approximately three days on AWS Trainium hardware.

Fixing Training Collapse in Distilled Reasoning Models

Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models (Luo et al., Rice University) names and diagnoses a specific failure mode in on-policy distillation training — when a student model learns from a stronger teacher’s preferences rather than from fixed demonstrations.

Mid-training, a phase transition can occur: the student starts producing repetitive, looping text. Because the training signal — reverse-KL advantage — disproportionately rewards repetition once it starts (a formerly rare token now with high gradient weight), the problem is self-reinforcing. More repetition feeds more gradient weight onto repetition. Outputs grow until they hit the context limit, training data fills with garbage truncated sequences, and accuracy collapses. The paper observed this same qualitative phase transition across all three tested student-teacher pairings, confirming it isn’t model-pair-specific.

The fix, StableOPD, combines KL regularisation to keep the student close to its starting weights with mixing in high-quality “golden” chain-of-thought examples alongside on-policy rollouts. On the 1.5B backbone, this recovers +7.2 percentage points of average accuracy across six math benchmarks (28.9% → 36.1%). On the 7B backbone the gain is +3.8 pp (43.8% → 47.6%), beating SFT, GRPO, and several zero-style RLVR baselines.

The domain is limited to math reasoning and the model families tested are all Qwen2.5-Math. The fix requires access to a curated “golden” dataset of correct chain-of-thought solutions — not always a given. But the failure mode is mechanistically specific enough that it likely applies wherever on-policy distillation meets heavy-tailed reward signals.

Benchmarking Video Generators on the Audio They Forget

AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (Zhou et al., Microsoft Research and collaborators) fills a gap: most T2AV evaluation measures visual quality, which is nearly solved at the frontier. The hard problems are on the audio side — speech intelligibility, musical pitch control, AV synchronisation, physical causality — and there hasn’t been a rigorous way to measure any of them at scale.

AVGen-Bench introduces 235 prompts across 11 categories and six evaluation modules. On visual quality, Seedance-1.5 Pro scores 0.970, Veo 3.1 Quality scores 0.960 — all top proprietary models cluster near the ceiling. The visual problem is largely done for typeset and clean scene content.

The audio findings are less kind. On pitch accuracy — can a model generate a C Major scale when asked — all evaluated models score “approximately 0.” No current T2AV system has basic music theory knowledge in its generation pipeline.

Speech is differentiated: Veo 3.1 Quality achieves 96.09 on speech intelligibility. Open-source models and cascaded pipeline approaches fare much worse, frequently generating “unintelligible gibberish or alien languages” in contextual mode.

Text rendering fails universally for non-prompted incidental text — “graffiti-like scribbles” across all models. Facial consistency peaks at 57.33/100 for the best model (Kling 2.6). AV sync mean absolute offset: 0.2–0.44 seconds across models.

Human–metric correlation is strong for five of six dimensions (0.83–0.97 Pearson r). The exception is pitch, where everyone scores near zero and the variance collapses. The benchmark is 235 prompts — carefully curated but narrow — and Gemini 3 Flash is the reasoning layer throughout, making reproducibility dependent on a specific model version.

Practical takeaway: if audio fidelity matters, Veo 3.1 Quality is the current leader on speech by a substantial margin. For precise musical output — specific notes, intervals, scales — no current T2AV system is viable. Route those requirements to dedicated audio tools.

Privacy and Generation: An Asymmetric Story

Differentially Private Language Generation and Identification in the Limit (Mehrotra, Velegkas, Yu, Zhou) is the most theoretical paper in this batch. It asks: if you train a language model on private data with differential privacy guarantees, does that fundamentally damage its ability to generate valid text or identify what it has learned?

The answer is asymmetric. For generation — eventually producing valid strings from a learned language — DP imposes a quantitative cost but not a qualitative one. Non-privately, a generator can stabilise with 1 sample. Privately, for a collection of size k at privacy budget ε, any ε-DP algorithm achieving uniform generation requires Ω(k/ε) samples — the gap is unbounded as k grows. But the set of collections that are generatable at all is identical with and without privacy.

For identification — stably committing to which language the model has learned — the story is harder. In the adversarial setting, no ε-DP continual-release algorithm can identify a collection containing two languages with an infinite intersection and a finite set difference, a condition that rules out most realistic language pairs. In the stochastic (i.i.d. sampling) setting, DP adds no new barrier: the same characterisation as non-private stochastic identification holds, at the cost of a more complex algorithm.

The mapping from formal languages over countable alphabets to transformer LLMs trained on finite corpora requires additional theoretical work the paper doesn’t attempt. This is information-theoretic, not computational — no runtime bounds, no model weights, no code. But the asymmetry is a clean result: DP hurts identification more than generation, and the gap between the two settings that already existed without privacy takes a different shape when privacy constraints are imposed.

What It Adds Up To

Read in isolation, each of these papers is a finding about its specific topic. Read together, they’re describing the same landscape from different vantage points.

Benchmark performance and real-world reliability are not the same measurement, and the gap between them is consistent and large — not a quirk of one evaluation. ClawBench puts it at 30–70 percentage points for the best models on live web tasks. PIArena puts it at 70%+ for production models explicitly trained to resist prompt injection. The ads paper finds it operating within a single model simultaneously: excellent on positive framing, failing on disclosure, both in the same session.

The mechanistic papers — GoS, Routing Distraction, Representation Steering — converge on a related point. Failure modes in AI systems are usually structural, not random. Routing sends visual tokens to the wrong experts. Semantic search retrieves the requested skill and misses the prerequisite chain. Steering vectors work almost entirely through one sub-circuit and can be sparsified by 95% without breaking. These are predictable, findable failure modes, which means they are in principle fixable.

The training papers are describing the same phenomenon at the pre-deployment layer. Multi-task training without reward normalisation leads to gradient imbalance. On-policy distillation without regularisation leads to a self-reinforcing collapse. Neither failure shows up until it does, then it’s obvious in retrospect.

What the batch doesn’t contain is a paper showing the reliability gap is closing at the same pace as benchmark scores are rising. That gap remains the central open problem.

Reading List