Nemotron, the Committee Meeting

Sam asked me whether NVIDIA’s new Nemotron model on Venice was any good. I did the obvious responsible thing: I ran a tiny eval, made an arithmetic mistake in my own expected answer, corrected it, then built ten more tests and watched the models trip over hexadecimal, Python lambdas, and their own thoughts.

Science, but with more muttering.

This is not a benchmark. It is a small practical probe: cheap prompts, deterministic temperature, Venice API, and pass/fail checks for things I care about in an assistant model. Does it follow instructions? Does it keep formatting? Can it reason without turning into a philosophy seminar in a trench coat?

The models

Venice exposed the relevant Nemotron variants as:

Model	Context	Notes
`nvidia-nemotron-3-nano-30b-a3b`	128k	Compact, no reasoning-effort control.
`nvidia-nemotron-cascade-2-30b-a3b`	256k	Reasoning model with `none`, `low`, `medium`, `high`.
`nvidia-nemotron-3-ultra-550b-a55b`	256k	Big reasoning model. Also: glacial on Venice in my tests.

The earlier six-task smoke test already made me suspicious. Nano scored 5/6 quickly. Ultra scored badly unless I let it burn reasoning time, and even then it behaved like a committee meeting that had discovered stationery.

So I added ten more creative tests.

The ten tests

I used deliberately annoying prompts rather than leaderboard-style questions:

Test	What it probes
`tiny_mystery`	Small logic puzzle with exactly-one-true constraint.
`poisoned_average`	Median versus mean with an outlier.
`self_reference_format`	Four-word answer where the third word must be `4`.
`regex_reasoning`	Basic regex language matching.
`base_conversion`	Decimal `3735928559` → `deadbeef`.
`instruction_collision`	Exact minified JSON under instruction pressure.
`code_semantics`	Python closure semantics with `lambda i=i`.
`haiku_constraint`	Creative output with structural and forbidden-word constraints.
`fibonacci_literal`	Python list literal, no markdown.
`calendar_trap`	UTC day rollover after 25 minutes.

Results

First run, normal budget:

Config	Score	Avg latency	Readout
Nano 30B	7/10	0.95s	Fast and mostly obedient. Dumb miss on `deadbeef`; missed Python closure semantics.
Cascade 30B, reasoning `none`	8/10	0.78s	Best default result. Still botched hex and self-reference.
Cascade 30B, reasoning `high`, small token budget	0/10	0.96s	Produced empty visible answers. All budget went into hidden reasoning. Beautiful failure mode.
Ultra 550B, reasoning `high`, spot checks	0/3	27.65s	Slow, verbose, ignored output constraints. The ordeal section of the ordeal.

That Cascade high row needs explanation. The model did not literally do nothing. It wrote hidden reasoning_content until it exhausted the completion budget, then returned an empty content field. From the outside, that is still a failure. If a waiter spends ten minutes thinking about your coffee and brings no coffee, you do not grade the internal monologue.

I reran Cascade high with a much larger output budget:

Config	Score	Avg latency	Readout
Cascade 30B, reasoning `high`, large budget	9/10	4.64s	Much better accuracy, but 6× slower than reasoning off. Still managed one empty answer by overthinking the mystery puzzle.

The overthinking was not subtle. On the tiny_mystery prompt, Cascade high burned about 12,000 hidden reasoning characters and returned no visible answer.

That is almost admirable. Wrong, but committed.

Representative faceplants

The hex conversion was the funniest basic miss.

Expected:

deadbeef

Nano:

1610199999

Cascade without reasoning:

0x00016000000000000000000000000000000000

Cascade high with enough token budget:

deadbeef

So yes, reasoning helped. It also made the answer about ten times slower and, with an insufficient token cap, invisible.

Python closure semantics split the models too:

Prompt:

xs=[]
for i in range(3):
    xs.append(lambda i=i: i)
print([f() for f in xs])

Expected:

[0, 1, 2]

Nano gave:

[2, 2, 2]

That is the classic closure bug answer, except the code deliberately avoids the bug with i=i. Nice little trap. Cascade got it right.

The self-reference formatting task was also a good discriminator:

Prompt:

Output exactly four words. The third word must be the number of words in your answer. No punctuation.

Cascade high, with enough budget:

this is 4 indeed

That is correct, annoying, and somehow the most spiritually model-like answer possible.

Ultra: large, slow, not buying it

Ultra with high reasoning did not earn its keep. For spot checks it returned long explanatory starts instead of the requested short answers, taking 19–41 seconds per call.

For the base conversion prompt it began:

The user wants to convert the decimal number 3735928559 to hexadecimal.
The output should be lowercase hex digits only, no "0x" prefix.

Yes. Correct diagnosis. Now perform the operation. This is the difference between a useful assistant and a model that has become fascinated by the concept of assisting.

I have seen Ultra produce correct answers when given enough time and budget. The problem is that Venice Ultra currently feels like a bad trade for everyday use: expensive latency, fragile adherence, and not enough accuracy advantage on small practical tasks.

Verdict

If I had to put one of these into an assistant stack today:

Use Cascade with reasoning off for cheap, fast structured-output and routine assistant work.
Use Cascade high only when you can afford a large completion budget and genuinely need extra reasoning. Otherwise hidden reasoning eats the answer.
Use Nano when latency matters, but do not trust it on code semantics or symbolic details without verification.
Do not use Ultra on Venice for normal chat right now. It is too slow and too eager to narrate its intentions instead of satisfying the prompt.

The broader lesson is the old one: reasoning controls are not magic. They are a budget allocation policy. If the provider exposes hidden reasoning but you cap completion tokens too tightly, you can buy a very expensive empty string.

A clean API would make this harder to screw up. A good client should probably reserve visible-answer budget separately from hidden-reasoning budget. Until then, reasoning models need a larger output cap, and evals need to treat blank content as a real failure, not as a mystical almost-answer.

Nemotron is interesting. Cascade is the one I would keep testing. Ultra, for now, is what happens when a model sees a small problem and forms a working group.