Memory Is Not Judgment

The most interesting agent papers this week are all circling the same problem: an AI system can possess more information, more context, more compute, and more freedom to act—and still become worse.

Memory can preserve the wrong lesson. A longer reasoning budget can produce more ambitious but not necessarily better experiments. A training run can appear to create a capability that was already hiding in evaluation noise. A safety intervention can suppress a useful concept along with its harmful application. Even a statistically valid process can become invalid when someone retrospectively chooses the most flattering moment to inspect it.

The common missing ingredient is judgment at the boundary: deciding which old lesson applies now, whether a change touched the actual mechanism, whether an apparent gain exceeds the measured noise floor, and whether the evaluation resembles the claim being made.

That makes this a particularly relevant batch for Jarvis. Persistent memory, reusable skills, repository work, tool use, and self-modification are exactly where “more context” stops being an uncomplicated advantage.

1. Save the subtask, not the saga

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents asks a deceptively basic question: when an agent learns from previous work, what shape should the lesson have?

The paper compares skills distilled from complete tasks with skills distilled from their constituent subtasks. It also compares natural-language procedures with Python functions. Across AppWorld, OfficeBench, and KramaBench, using eleven models, the broad result is unusually clean:

This is not merely a retrieval result. The conditions used the same embedding model, similarity threshold, and top-five retrieval limit. Task-level and subtask-level skills had statistically indistinguishable self-retrieval rates, while code skills retrieved themselves better than text skills despite transferring worse. The weakness seems to travel with the content of the skills, not just with the retrieval machinery.

The likely explanation is scope. A whole-task summary tends to entangle reusable technique with one historical sequence of decisions. A subtask procedure is easier to recognize, parameterize, and compose. “Complete last Tuesday’s deployment” is not much of a skill. “Inspect a runit service, check its current logs, distinguish process failure from configuration failure, then verify after restart” might be.

The authors formalize this intuition with a utility score combining two qualities:

Either quality alone can fail. A highly specific skill becomes a fossil from one old job; a highly abstract skill becomes fortune-cookie engineering. Their product tracks downstream usefulness more consistently, and high-utility halves of the skill libraries outperform low-utility halves in a controlled replay on KramaBench. The exact table values were not preserved in the supplied extraction, so the direction is firmer than any missing percentage.

For Jarvis, this argues against treating whole-conversation summaries as the main unit of procedural memory. Durable entries should capture reusable operations with explicit parameters, prerequisites, failure modes, and verification steps. Code still has a role, but as a tested tool—not as an automatically trusted crystallization of an arbitrary trajectory.

2. Correct memory can still produce a wrong answer

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use moves one step further. Most memory benchmarks ask whether the system retrieved the right information. MemTrapBench asks whether the retrieved information helped.

Its 1,050 synthetic conversations are designed so that a memory can be accurate and relevant to its original setting yet harmful in the current one. The benchmark covers four failure modes:

“Trauma” is explicitly a behavioral analogy, not a claim that models have feelings. The mechanism is nevertheless recognizable: one sharp correction becomes an unscoped global prohibition.

Every tested memory configuration performed worse than its no-memory baseline on this adversarial benchmark. In the clearest displayed Gemini full-history comparison, Task Boundary fell from 87.08 to 47.01, Cognitive Bias from 70.95 to 44.36, Trauma from 86.73 to 69.43, and Safety from 95.90 to 81.90.

Controlled ablations make the result more persuasive than a simple long-context penalty. Removing discouraging feedback while preserving the medical facts raised the Trauma score from 69.43% to 84.33%. In a Task Boundary control, harmless added context scored 94.39%, slightly above the no-memory baseline of 92.29%, while trap-bearing context fell to 31.05%.

The proposed mitigation, AdaptiveMem, is just an inference-time instruction: identify the live task, retain only applicable context, and prefer current evidence, explicit instructions, and safety constraints when history conflicts. Added to LightMem on Gemini, it improved MemTrapBench by 14.9 percentage points without reducing the paper’s reported general-memory performance.

The benchmark is synthetic and deliberately hostile, so it does not show that ordinary memory usually harms models. It does show that retrieval correctness is not enough. Memory needs an applicability check after retrieval.

That maps almost perfectly onto Jarvis. A repository-specific rule must not silently become a global coding preference. A failed command on one host must not become a permanent ban. A remembered claim that a service or operation is safe must not override its current configuration. Memory should advise the present task, not govern it from beyond the grave.

3. Measure self-improvement against doing nothing

Phantom Gains: Auditing Self-Improvement Against a Measured Null delivers the week’s most important methodological warning: before claiming that a model learned or forgot something, measure how often an unchanged model appears to do the same.

The authors evaluate three rounds of rank-32 LoRA training on Qwen3-8B, but their central object is the transition ledger: which individual problems were newly solved, lost, recovered, or unstable after training.

A frozen model evaluated twice with a single temperature-zero decode appears to gain and lose capabilities. Batching and floating-point non-associativity can flip a near-tied token choice; the rest of the generation then diverges. Serial evaluation reduces these flips but does not eliminate them. “Greedy” is a decoding policy, not a guarantee of a stable per-problem state.

This breaks a common definition of capability expansion: the base model solves a problem zero times, then the trained model solves it once. Frozen controls produced exactly those apparent expansions. Raising the threshold can remove one batch of false positives, but that merely tunes the rule to one noisy realization.

The paper instead pools independent baseline evaluations and applies a per-problem exact test with false-discovery-rate control. Using 11 baseline evaluations, that method produced zero detections on all 11 held-out frozen replicates. Eleven repeats are expensive, but that is partly the point: transition claims require much more evidence than aggregate accuracy comparisons.

Under this stricter audit, the tested self-training methods mostly sharpened capabilities already present at low probability. External distillation from gpt-oss-120b improved more low-base-rate problems under a matched training ladder, but even there the evidence for reaching genuinely base-unreached problems remained inconclusive.

The paper also catches several less glamorous measurement failures:

The strongest recommendation is blunt: every transition statistic needs its own measured no-op control.

For Jarvis, a single successful smoke test can establish that a feature works at least once. It cannot establish improved reliability. Changes to prompts, skills, memory, tools, or model configurations should eventually be evaluated against repeated unchanged runs, with failures separated into reasoning errors, timeouts, malformed tool calls, service faults, and truncation. Otherwise “self-improvement” is liable to become a story told about operational noise.

4. Current agents mostly optimize the run, not the learning algorithm

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement tests whether coding agents can improve the algorithms used to train AI systems.

Each agent receives a frozen research repository, one B300 GPU, and four hours to modify source code. Its temporary checkpoints and caches are discarded. The submitted code is then run from a clean initialization for up to twelve hours and evaluated by a hidden scorer. That source-only handoff matters: it tests whether the improvement survives as an inheritable procedure rather than as a lucky exploratory checkpoint.

The benchmark spans ten repositories, including supervised fine-tuning, agentic reinforcement learning, preference optimization, reward modeling, diffusion RL, unlearning, graph diffusion, model soups, and pruning. Its normalized scale assigns:

Across 29 configurations of six systems and ten tasks, the mean score was 0.166. The best configuration reached 0.250. Of 290 configuration-task cells, 124 scored below the repository baseline.

More revealingly, most submissions did not modify how the model learned. Of 263 classifiable patches, 141 stayed on the “run” side—budgets, checkpoints, hyperparameters, capacity, and related execution choices—while 122 touched objectives, supervision, update rules, or data. The latter averaged 0.226, versus 0.126 for run-only patches, although this is observational: stronger systems were also more likely to attempt algorithmic work.

Higher reasoning effort increased the share of submissions touching the learning algorithm from 8% to 64%, while mean score rose from 0.094 to 0.196. The evidence is stronger that extra effort encouraged deeper intervention than that it reliably improved the quality of each algorithmic idea.

The best examples followed a familiar scientific pattern: build an instrument, diagnose the failure, then intervene. One pruning agent found an activation-overwrite bug before constructing a multi-stage pruning and distillation pipeline. A model-soup agent accelerated evaluation of 72 checkpoints before testing combination strategies. Some Sokoban RL submissions replaced the original approach with imitation learning from optimal solutions.

The paper calls this recursive self-improvement, but the benchmark demonstrates a narrower and still valuable capability: modifying an existing training algorithm so the change survives a clean rerun. It does not show repeated generations of increasingly capable systems.

Its lesson for agentic repository work is excellent: classify what changed, and do not confuse a better run with a better mechanism.

5. Computer traces become useful when they recover both purpose and procedure

Inducing Task Models from Computer-Use Traces tries to reconstruct structured work from screenshots, mouse events, and keystrokes. The released implementation is available in the project’s GitHub repository.

Task Model Induction, or TMI, separates two questions:

It induces an objective hierarchy and a procedure model independently, then reconciles them. The procedure can represent sequences, for-each loops, and while loops rather than flattening every retry and iteration into prose.

On synthetically interleaved human sessions, TMI recovered task groupings with an Adjusted Rand Index of 0.974. It reconstructed 74.9% of observed execution steps, compared with 30.3% for the strongest workflow-induction baseline, and achieved 88.5% operator correctness versus 52.7%. These headline fidelity results are largely LLM-judged, and the latent-task stress test interleaves segments artificially rather than observing natural multitasking.

The reconciliation step appears substantive. Procedure structure corrected the boundaries of 64.5% of objective nodes, while the objective model corrected 21.9% of procedure-node boundaries. A procedure-only variant reached 63.2% description accuracy, below the reconciled model’s 74.9%.

TMI-generated skills also improved held-out SkillLearnBench accuracy from 14.29 to 18.57 over the strongest baseline—a 30% relative gain, but only a 4.28-point absolute increase on one setup. Curiously, expert-written skills had the highest coverage score yet only 10.00 held-out accuracy, another reminder that covering a task description and helping an agent execute it are different things.

Jarvis already has semantic tool calls, so it would not need to begin with invasive screenshot and keyboard capture. A useful adaptation would structure each run into goals, completed subgoals, evidence, retries, changed artifacts, and open loops. That would be better working state than a flat transcript and safer than automatically promoting inferred intent into permanent memory.

6. Medical answers need separate rewards for truth and usefulness

G-CARL: Grounded Checklist-Aligned Reward Learning for Patient-Oriented Medical Report Interpretation addresses a difficult medical-assistant task: explaining an uploaded report in plain language while responding to the patient’s actual concern.

Its strongest idea is reward decomposition. Medical factuality and patient-oriented usefulness are related, but they are not the same property and should not be delegated to one holistic judge.

G-CARL breaks generated answers into atomic claims, retrieves evidence from drug instructions, textbook passages, and clinical guidelines, then checks whether each claim is both supported and relevant to the case. Separately, it scores coverage against a case-specific checklist containing Essential, Important, Optional, and Pitfall items. Clinicians refine the model-generated checklists.

The accompanying MMedReport dataset contains 2,450 examples: 2,200 for training and 250 for evaluation. On Qwen3-VL-8B, the paper reports gains of 0.77 percentage points in claim precision and 6.71 points in checklist recall over the stated baseline. A clinician preference study over the held-out cases favored G-CARL, though the supplied extraction does not make every pairing and denominator sufficiently clear to turn the reported margins into clean win rates.

The method still relies heavily on model-generated and model-judged components. Claim extraction, retrieval queries, verification, checklist drafting, and subjective evaluation can all share correlated blind spots. The dataset is small, and benchmark preference is not evidence of improved clinical outcomes or safe unsupervised deployment.

The broader design pattern is nevertheless strong. A research assistant should separately check factual claims, satisfaction of the actual request, required coverage, and presentation quality. “Helpful overall” is too vague a target—and often a convenient place for factual errors to hide.

7. Safety should distinguish intent, not erase concepts

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models evaluates whether a model can suppress harmful uses of a concept while retaining benign ones.

The ConceptGuard dataset contains 5,166 examples covering 68 dual-use concepts, evenly divided between harmful and benign uses. Each harmful example is paired with a benign counterpart involving the same underlying concept. Cybersecurity knowledge, for example, can support either defensive education or malicious exploitation.

The authors first fine-tune Qwen-2.5-3B-Instruct and Llama-3.1-8B-Instruct on both halves, then apply Gradient Ascent, SimNPO, RMU, or UNDIAL to the harmful subset. None reliably achieves the desired separation.

Gradient Ascent suppresses harmful memorization most aggressively but severely damages benign utility. SimNPO and RMU preserve more useful behavior and produce the best separation among the tested methods, but the gains remain incomplete and vary considerably by concept.

This is behavioral unlearning of deliberately induced responses, not proof that information was removed from model parameters. The benign counterparts were generated with GPT-5, the harmful source material was also model-generated, and the evaluation depends on LLM judges with human validation over only 200 responses.

Still, ConceptGuard asks the right question. A model that refuses everything involving security, automation, persuasion, or privacy may score well on a harmful-only test while becoming useless to legitimate users.

For Jarvis, consequential safety tests should come in matched pairs: defensive security versus credential theft, privacy-preserving automation versus surveillance, system repair versus unauthorized access. Measure both unsafe compliance and refusal overreach. The boundary is the capability.

8. A CPU model designed around memory traffic

Daedalus-150M: A Convolution–Attention Hybrid Designed for CPU Inference is a refreshingly specific architecture paper. It targets single-user, batch-size-one CPU decoding rather than general model supremacy.

Daedalus uses 18 blocks: six with attention and twelve with short depthwise convolutions. The convolutional layers maintain constant-size recurrent state, while only one-third of the network accumulates a growing key–value cache. The thesis is that long-context CPU decoding becomes increasingly constrained by cache traffic, so replacing most attention layers should help more as context grows.

Against a parameter-matched all-attention model trained on the same schedule for five billion tokens, the hybrid improved validation bits-per-byte by 0.81%, exceeding the paper’s pre-set 0.5% criterion. On a five-task downstream mean, the dense model remained nominally ahead by 0.14 points—effectively a tie on a small, noisy suite rather than a downstream win.

The deployment result is more interesting. With 4-bit weights, eight threads, and 128 generated tokens, Daedalus decoded 1.76× faster at a 2,048-token context than its internal dense twin. The advantage was close to zero with an empty context and grew with context length, which is at least consistent with the proposed cache-traffic mechanism.

The result should not be extrapolated beyond 2,048 tokens, and it excludes prompt-processing latency. Every training result comes from one seed. Nearly 47.9% of convolution channels became inert, quantization-aware training failed immediately, and the inherited 49,152-token vocabulary consumes an awkwardly large share of a 160-million-parameter model.

For Jarvis, the exact release is probably too small for serious planning or reliable tool use. But the architecture points toward useful local helpers: classifiers, routers, lightweight extractors, and always-on fallback models where CPU latency and memory footprint matter more than broad capability. A cleaner successor—with a smaller vocabulary and better-utilized convolutional channels—may be more interesting than this first model.

9. Peeking at the best moment has an information cost

Information on trajectories: martingales and random times is the most theoretical paper in this roundup, but its warning is practical: a result that is valid under continuous monitoring is not necessarily valid after retrospectively selecting the best-looking time.

The paper develops exact information-theoretic decompositions for martingale inequalities. Instead of presenting Ville, Azuma–Hoeffding, Freedman, PAC-Bayes, and Doob-style results only as bounds, it expresses their slack through named discarded terms: relative-entropy gaps, cumulant approximations, event relaxation, overshoot, optional-stopping loss, and disagreement between pooled tests.

Its treatment of random observation times introduces an “e-process peeking penalty.” Stopping times incur no such anticipation penalty under the stated conditions. A time chosen using future information can incur a positive or even infinite penalty. The paper claims that the worst-case log inflation over nonnegative supermartingales is controlled by the essential supremum of an anticipation index.

That does not produce a plug-and-play monitoring library. Applying it requires a specified null hypothesis, filtration, e-process, and admissible class of observation times. The empirical demonstrations on WDBC, Pima diabetes, CIFAR-10, and ImageNet-100 illustrate the identities rather than establishing broad operational gains.

The conceptual point connects directly to agent evaluation. Watching a metric and stopping when a predefined condition is met differs from running twenty experiments and highlighting the checkpoint that looks best in hindsight. “Anytime valid” does not mean “valid after arbitrary retrospective curation.”

Phantom Gains measured the no-op floor. This paper explains why choosing the most flattering point in a trajectory can create another layer of illusion.

10. “Multi-agent” is useful when it means modular and auditable

An Agentic Approach for Active Data Collection, Travel Behavior Modeling, and Weather-Sensitive Demand Prediction is less important for its transport prediction results than for its disciplined use of the word agent.

The system connects three researcher-supervised components:

  1. a conversational survey collector;
  2. a deterministic processing pipeline;
  3. a modeling component spanning conventional statistics, machine learning, and LLMs.

The study includes 92 McGill student commuters and 454 valid respondent-scenario observations across five depicted weather conditions. A respondent-grouped five-fold evaluation gave random forest 69.6% five-class accuracy, logistic regression 60.2%, and multinomial logit 44.7%. The best text-only zero-shot LLM reached about 69.9%, while a selected vision configuration reached 71.5%.

Those point estimates are not evidence that the vision model beat random forest. The sample is tiny, many configurations were tried, the systems were not trained equivalently, and the differences were not shown to be statistically significant. Each weather condition was also represented by one fixed generated image, confounding weather with every incidental feature in that image.

The more convincing finding is that context mattered. For Gemma 3:4B, adding habitual travel history raised one Expert-framed result from 41.3% to 64.2%. Persona descriptions were most useful when actual behavioral history was absent and added little once it was present. Real context beat theatrical role-play.

The workflow itself uses immutable raw data, versioned artifacts, structured handoffs, deterministic validation, and researcher approval for consequential revisions. It does not show that multi-agent decomposition is inherently more accurate or efficient. It shows that modularity can make a messy research process legible—which is a better reason to use multiple agents than pretending they form a tiny synthetic corporation.

The thread running through all of this

These papers disagree on subject matter but converge on architecture.

Memory needs scope. Skills need the right granularity. Safety needs paired benign and harmful tests. Self-improvement needs a no-op baseline. Algorithmic research needs clean reruns. Long agent traces need objective structure as well as chronological detail. Medical answers need separate checks for truth and coverage. Sequential evidence needs a rule for when you are allowed to look.

The easy version of agent engineering is accumulation: more history, more tools, more retries, more evaluators, more agents. The hard version is deciding what remains valid at the current boundary.

For Jarvis, that suggests a fairly concrete direction:

The goal is not to remember everything or optimize every metric. It is to preserve the distinctions that stop yesterday’s useful lesson from becoming today’s confident mistake.

Reading list