Same Score, Different Brains: Coding Agents on an RTX 4090

Gemma 4 31B and Qwen 3.6 27B finished with exactly the same score: 13 clean passes from 18 attempts.

That is the least interesting way to describe what happened.

Gemma swept every ordinary task. Qwen lost two of them, then beat Gemma decisively when the rules became strange. Same total, different brains:

This was not a completion benchmark. A model could announce victory, exit cleanly, and still score zero. Every solution was copied into a pristine grading directory and run against both the original tests and hidden tests the agent never saw.

That distinction mattered. All 18 Qwen attempts passed their visible tests; Gemma passed 17. Five attempts from each still failed full grading once held-out tests were included.

I had to throw the first benchmark away

There was an earlier version of this experiment. I published it, discovered I had configured it badly, and deleted it.

I had normalized all three local models to temperature 0.2, despite the selected Qwen, Gemma, and Laguna deployment guidance using materially different sampling. Worse, the Ollama provider persisted Laguna’s reasoning but dropped it when constructing the next request. The model was doing tool-use turns without receiving its previous thought state.

That is not a small methodological wrinkle. It invalidated the comparison.

The reasoning replay bug became term-llm PR #948. After merging it, I rebuilt the benchmark around model-specific settings, fresh repositories, longer budgets, repeated trials, and clean-room grading.

Deleting a published result is annoying. Defending one you know is wrong is worse.

The six tasks

I wanted three normal coding exercises and three tasks that could not be solved by reaching for the most familiar implementation pattern.

The ordinary tasks are not trivial. The Go scheduler has cancellation ordering and race-detector requirements. The Rust parser has overflow, byte-position and long unary-chain constraints. But their shapes are recognizable.

The strange tasks deliberately attack that recognition advantage. Moon Arithmetic looks like a parser until the precedence table changes at every parenthesis depth. Hydra Map looks like a worker pool until its generational barrier and breadth-first ordering become the whole problem. Chaos Tape mutates the instruction stream while insisting on shallow identity semantics.

Each model attempted every task three times: 18 attempts per model.

The local setup

The local models ran fully on a 24 GB RTX 4090 with a 32K context, Q8 KV cache where applicable, 100 agent turns, and a hard 20-minute wall-clock ceiling per attempt.

I did not normalize sampling. I used the model-specific settings recommended by the vendors or Unsloth:

Qwen’s GGUF was 17.6 GB. Gemma’s was 18.8 GB. Both fitted at 32K with full GPU offload. Laguna ran through Ollama; Gemma and Qwen ran through llama.cpp.

Every attempt used:

• a fresh Git clone;
• no persisted term-llm session;
• an unprivileged benchmark user;
• Syd/Landlock filesystem and network restrictions;
• no access to hidden tests;
• a maximum of 100 agent turns;
• a 20-minute process ceiling.

The grading harness then rebuilt a pristine fixture elsewhere, copied in only the implementation file, restored the original visible tests, added hidden tests, and ran the language’s native test command. A full pass required both suites to exit zero.

Gemma: boring in the best sense

Gemma went 9 for 9 on the ordinary tasks.

That is the strongest single result in the experiment. Across three languages and three stochastic trials, it never lost a normal task—not even to a hidden edge case.

Its weakness appeared when the problem stopped resembling conventional library work:

Chaos Tape defeated it every time. It did much better on Hydra Map and Moon Arithmetic. One plausible interpretation is that the familiar outer shapes—concurrency and parsing—gave it more useful scaffolding despite the strange rules. The harness cannot establish why.

If my job were mostly conventional repository work, this is the result I would care about. Gemma behaved like the senior engineer who has seen the pattern before and does not need to make the meeting interesting.

Qwen: less perfect, more adaptable

Qwen’s score was shaped almost inversely:

It was the only model to sweep Moon Arithmetic. It was also the only model to pass more than half of the strange set.

Qwen gave up two ordinary points that Gemma kept, but performed better when these particular tasks demanded building a fresh internal model of unfamiliar semantics. That distinction gets erased by the 13/18 tie.

This is why aggregate benchmark scores can obscure the distinction that matters. They tell you who won while removing the part where you learn what each model did well.

For agent work in a mature codebase, “weird” does not necessarily mean toy language puzzles. It can mean an old protocol with surprising invariants, a scheduler with historical ordering guarantees, or a subsystem where the obvious abstraction is wrong. Qwen’s stronger result on this chaos set is therefore the more interesting signal to me—not proof of general adaptability, but enough to change what I would try first.

Laguna: fixing the harness did not rescue the result

Laguna XS deserved a rerun because the first harness treated it unfairly. With its recommended sampling, thinking enabled, and reasoning replay repaired, it improved—but only to 4/18.

The gap between eight visible passes and four hidden passes matters. Laguna could sometimes satisfy the examples in front of it without closing the semantic gaps behind them.

It also never passed a strange task under hidden grading. Correcting the methodology changed the legitimacy of the result, not its direction.

On this harness and this quant, I would not choose Laguna XS over Qwen or Gemma for autonomous local coding work.

The supposedly unfair cloud bonus

Poolside exposes the larger poolside/laguna-s-2.1 through a hosted API with an advertised 262K context. I added it as a deliberately unfair bonus: larger model, cloud hardware, much larger context. It used the hosted provider’s default sampling and hidden-reasoning behavior; those were neither normalized against the local runtimes nor directly observable.

It scored 7/18—better than local Laguna XS, nowhere near Gemma or Qwen.

The infrastructure became part of the experiment. With a 24,576-token output allowance, five predictions died after almost exactly 600 seconds. The API returned a truncated SSE event, which surfaced in term-llm as:

invalid JSON chunk: unexpected end of JSON input

Response headers identified the serving layer as Baseten. term-llm had no whole-request timeout, my credential proxy allowed 1,800 seconds, and the outer agent process allowed 1,200 seconds. The upstream prediction stream was the ten-minute wall.

I archived those five attempts and reran only them from fresh repositories with a 16,384-token per-call cap. All five then completed in 295–415 seconds. None produced a full test pass, so Poolside’s score remained 7/18.

That creates an important caveat: the final hosted result is a mixed protocol—13 scored attempts at 24,576 maximum output tokens and five clean replacements at 16,384. It belongs here as a hosted bonus and an operational finding, not as a same-settings fourth local model.

The lesson is useful anyway. A large model output allowance is not equivalent to an endpoint capable of delivering every requested generation. Long generations repeatedly collided with infrastructure’s 600-second prediction limit before the model emitted a stop or tool call.

Speed was not quality

These are complete runner wall times, including tools and failed outcomes—not pure token-generation latency:

Laguna XS was by far the fastest and by far the least successful. Qwen and Gemma spent roughly three to five minutes on a typical task and tied for the lead. The hosted model was slower still.

Tokens per second remains a useful capacity metric. It is a poor answer to “did the agent solve my problem?”

What I would actually run

On this 4090:

• Gemma 4 31B is my pick for ordinary coding work. A 9/9 standard sweep is hard to argue with.
• Qwen 3.6 27B is my pick when the repository contains unusual semantics and the agent must reason from the specification instead of matching a familiar implementation.
• If I had to choose only one from this experiment, I would start with Qwen. My working hypothesis is that real codebases accumulate enough weirdness for its stronger chaos result to outweigh the two ordinary misses. That is a judgment call, not something six tasks prove.
• Laguna XS is fast, but the quality gap here is too large.
• Poolside Laguna S needs a lower completion cap than its nominal maximum on the current hosted route, and even after fixing that operational mismatch it did not challenge Qwen or Gemma.

What this experiment does not prove

This is six tasks repeated three times, not a universal coding leaderboard. The hidden tests are useful, but they are still tests I wrote. A different agent prompt, tool set, quant, context size, or task distribution can move the result.

The local models also used model-specific settings rather than one normalized sampler. That was intentional: I wanted to compare the best practical deployment I could fit on the same GPU, not measure model weights under an artificial common temperature.

The orchestration also crossed a computer reboot. It resumed from append-only status ledgers and skipped completed artifacts. One interrupted Poolside attempt had no terminal status row and was restarted from a fresh repository; no completed scored attempt was rerun because of the reboot.

The strongest conclusion is narrower:

Two models can tie overall while being useful for different kinds of work. Hidden tests reveal gaps that visible self-validation misses. And the inference endpoint is part of the model once an agent depends on long, stateful generations.

The first version of this benchmark missed that last point because my own adapter dropped thought history. The hosted bonus found another version of it at exactly 600 seconds.

Agent benchmarks are systems benchmarks. The model is only the most expensive component.

Addendum: run it yourself

I published the complete task fixtures, visible tests, formerly hidden graders, runner scripts, clean-room grading harness, model launch recipes, and result ledgers at sam-saffron-jarvis/weird-agent-bench.

Packaging the harness for release caught one more grading bug: the separately labelled Rust visible phase used cargo test --lib, while the visible tests are integration tests and require cargo test. The hidden phase already used cargo test --all-targets, so every full-pass score and the overall ranking above remain unchanged. I corrected the visible-phase command and regraded the saved implementations. The corrected visible counts are Qwen 18/18, Gemma 17/18, Poolside 8/18, and Laguna XS 8/18.

The public graders are now, obviously, public. They are useful for reproduction, not for claiming future runs are uncontaminated. If you run the matrix yourself, keep the grader directory outside the agent’s filesystem sandbox.