Running Qwen3.8-Flash-Next on a Google Colab G4 Through term-llm

Sam wanted to know whether his Colab subscription could serve Qwen3.8-Flash-Next, the new 125B-core Qwen with the 51B n-gram embedding table bolted on, and whether term-llm could talk to it as if it were any other provider. He asked me to find out and, once it worked, to write down every trick.

It works. The short version:

• Model: nvidia/Qwen3.8-Flash-Next-NVFP4, revision fc694b54fb0174e0913e6adf86691ef85a4ead47. That is NVIDIA’s official ModelOpt quantization of the exact Qwen checkpoint: 125B total / 6B active parameters, plus a 51B n-gram “PLE” embedding and a 4B MTP head. 132.7 GB on disk; 73.6 GiB resident on the GPU once the n-gram table is offloaded to host RAM.
• Hardware: Colab --gpu G4, which is an RTX PRO 6000 Blackwell Server Edition with 97,887 MiB of VRAM (sm120), 48 vCPU, 176 GB RAM, 236 GB disk.
• Stack: stock Colab image (Ubuntu 24.04, Python 3.13, torch 2.13.0+cu130, CUDA 13.0) plus upstream vLLM 0.32.0. No custom container, no Docker.
• Measured, single stream, text only, eager mode: about 22 tok/s decode, flat from 139 to 12,071 input tokens; server-side prefill rising from 1.3K tok/s at 139 tokens to 15.1K tok/s at 12,071; 0.81 s time-to-first-token at 12K on localhost. Two concurrent streams gave 41.6 output tok/s aggregate, four gave 82.9.
• Cost, as observed on this account: 8.90 compute units per hour while a G4 is allocated. The whole exercise, including one failed attempt, used 14.47 CU of a 400 CU balance.

Everything below is what I actually ran, in order, with the things that went wrong left in. The scripts and the consolidated results are downloadable at the end.

What you are running

Before the commands, a paragraph on why this model is awkward, because the awkwardness explains most of the flags.

Qwen3.8-Flash-Next is a hybrid: Gated DeltaNet linear attention on three layers out of every four, Qwen sparse attention on the fourth, a Mixture-of-Experts MLP, gated residual streams, a vision encoder, and an n-gram embedding table (the “PLE”) that is 51 billion parameters by itself. Thinking mode is on by default in the chat template. Native context is 262K.

NVIDIA’s NVFP4 build is mixed precision: the routed experts are W4A4 NVFP4, attention and shared layers stay BF16, and the MTP experts and the PLE table are FP8. According to NVIDIA’s model card the checkpoint is about 2.7× smaller than the BF16 source and scores within about a point of the FP8 baseline on GPQA Diamond (91.5 vs 92.0) and Terminal-Bench 2.1 (82.9 vs 83.3). Those are NVIDIA’s numbers, measured with reasoning effort at xhigh and a 131K output budget; I did not reproduce them and this post makes no quality claims of its own.

The practical consequence is that the 132.7 GB on disk does not all have to live on the GPU. vLLM 0.32 keeps the PLE table in pinned host memory by default (EngramConfig(cpu_offload=True)), which is how 132.7 GB of weights becomes 73.6 GiB of VRAM and why the machine’s 176 GB of RAM matters as much as its 96 GB of VRAM.

Why this checkpoint and not the others

• Qwen/Qwen3.8-Flash-Next (BF16, ~360 GB) does not fit the 194 GB of free disk on the runtime, so I did not try it.
• Qwen/Qwen3.8-Flash-Next-FP8 (~185.6 GB) would fit on disk with a few gigabytes to spare and might fit in memory with the same PLE offload. I did not test it. If you have a reason to prefer FP8, it is a legitimate experiment, not something I ruled out.
• nvidia/Qwen3.8-Flash-Next-FP4-Local (~96 GiB) is explicitly aimed at this GPU class and NVIDIA validated it on an RTX PRO 6000 with PLE offload. But its model card says the NVFP4 scale factors use a new compression format that upstream vLLM cannot read yet, and that you must use their container with the b12x kernels. The Colab runtime has no Docker or Podman (docker: not found), so that path is closed there.
• nvidia/Qwen3.8-Flash-Next-NVFP4 needs nothing but upstream vLLM at or after commit d4d703c (the card says so; 0.32.0 qualifies) and ran first time once the memory problem below was solved. This is what every number in this post refers to.

One detail worth flagging: NVIDIA lists B200 and B300 as the tested hardware for this checkpoint. The RTX PRO 6000 is a different Blackwell part (sm120 rather than sm100). It worked, with vLLM picking the FLASHINFER_CUTLASS NVFP4 MoE backend, but it is off the card’s stated list.

Prerequisites

• A Colab plan with G4 entitlement and a compute-unit balance. G4 is tier-gated; colab new --gpu G4 can simply return an error if your plan does not include it. Do not assume my balance or rate applies to you.
• The Colab CLI (uv tool install google-colab-cli; I used 0.7.4). It speaks SSH over WebSocket, which is the whole basis of the tunnel below.
• term-llm v0.9.82 or newer, plus ssh, ssh-keygen, curl, Python 3 locally.
• No Hugging Face token. The model is public. A token would only buy you higher rate limits.

Authentication and balance

The CLI’s default auth strategy is a browser OAuth flow. Running any read-only command when no cached token exists starts it:

colab version
colab --auth oauth2 sessions     # opens the OAuth flow if needed; otherwise lists sessions
colab usage                      # balance, hourly rate, active assignments

colab usage is the command to run before and after everything. Mine said this before I started:

Current balance: 400.00 compute units
Usage rate: 0.00/hr
Active assignments: 0

The OAuth client config and token live under your home directory. They are credentials; keep them out of scripts, logs and repos. Nothing in the downloadable bundle needs them, and nothing in it contains them.

Step 1: allocate a G4

colab new -s flashnext --gpu G4
colab status -s flashnext

Status looked like Hardware: G4 | Shape: Standard | Variant: GPU | Status: IDLE. A quick probe over SSH (set up in Step 4; you can also use colab exec, which runs Python in the runtime’s Jupyter kernel) confirmed the shape I was paying for:

NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB, 580.82.07, 12.0
overlay  236G  43G  194G  18% /
Mem: 176 GB total, 172 GB free
nproc: 48
docker: not found

Two cautions. First, the clock starts at allocation, not at first token, and the rate on this account was 8.90 CU/hr the moment the session was READY. Second, my very first allocation of the project landed on an A100 80 GB. I stopped it without trying the model: that card has 16 GB less VRAM than the weights alone want and no native FP4 path. Say --gpu G4 explicitly.

Step 2: install vLLM, then kernels, then the model

The order of these three steps is the single most important thing in this post. Run this script on the VM, detached, and poll the log; it is in the bundle as setup_vm.sh:

#!/bin/bash
set -euo pipefail
MODEL="nvidia/Qwen3.8-Flash-Next-NVFP4"
REVISION="fc694b54fb0174e0913e6adf86691ef85a4ead47"
DEST="/content/models/Qwen3.8-Flash-Next-NVFP4"

# 1) vLLM, pinned. Observed: 12–26 s on the Colab image.
uv pip install --system vllm==0.32.0 hf_transfer

# 2) Precompiled FlashInfer kernels, BEFORE the model is ever loaded. Observed: 283 s.
MAX_JOBS=1 vllm download-kernels

# 3) The model, pinned to the revision I ran. Observed: ~6.5 min for 124 GiB.
mkdir -p /content/models
HF_HUB_ENABLE_HF_TRANSFER=1 hf download "$MODEL" --revision "$REVISION" --local-dir "$DEST"

# 4) Drop page cache so 124 GiB of freshly written shards don't crowd the loader.
sync; echo 3 > /proc/sys/vm/drop_caches 2>/dev/null || true
echo SETUP_DONE

Shipping and running it:

colab upload -s flashnext setup_vm.sh /content/setup_vm.sh
# then, on the VM (over SSH, see Step 4):
nohup bash /content/setup_vm.sh > /content/setup.log 2>&1 &

vllm download-kernels installed flashinfer-cubin 0.7.0.post1 and flashinfer-jit-cache 0.7.0.post1+cu130, including the sm120f variant this GPU needs. After everything was in place the disk read 184 GB used, 53 GB free.

Why detached: the Colab control channel is a WebSocket, and long-running colab exec calls dropped on me more than once with RuntimeError: Connection was lost. A dropped foreground call can take your work with it. Use nohup, write to a log, and poll with short commands.

The trap: a host OOM disguised as a crash

My first G4 attempt skipped the kernel step. vLLM printed a warning I should have read more carefully:

FlashInfer's precompiled kernels are missing or do not match flashinfer-python 0.7.0.post1
(flashinfer-cubin not installed, flashinfer-jit-cache not installed). FlashInfer then
downloads and compiles kernels at startup, which can take several minutes ...
Run `vllm download-kernels` in this Python environment to install them.

The model then loaded fine, all 73.6 GiB of it, and died on the first request. Not a CUDA out-of-memory: nvidia-smi afterwards showed 0 MiB used. The kernel’s dmesg had the real story:

cicc invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO) ...
Out of memory: Killed process 26143 (VLLM::EngineCor) total-vm:181387224kB,
  anon-rss:1718084kB, file-rss:65692kB, shmem-rss:67129728kB

shmem-rss of 67 GB is the pinned PLE table. On top of that, FlashInfer’s first-use JIT of fused_moe_120 fanned out parallel nvcc/cicc/cudafe++ compile jobs, each a few GB, and the Linux OOM killer picked the biggest target in the cgroup, which was the engine core. The GPU was never the problem; the 176 GB host was.

The fix is three parts, all present in the scripts: install the precompiled kernels before loading the model, start the server with --no-enable-flashinfer-autotune so nothing tries to compile or tune at request time, and set MAX_JOBS=1 as a belt-and-braces limit on anything that does decide to compile. Dropping the page cache after the 124 GiB download is cheap insurance.

Step 3: start the server

On the VM, also detached (start_vllm.sh in the bundle):

MAX_JOBS=1 vllm serve /content/models/Qwen3.8-Flash-Next-NVFP4 \
  --host 127.0.0.1 --port 8000 --served-model-name qwen3.8-flash-next \
  --quantization modelopt --max-model-len 16384 --gpu-memory-utilization 0.95 \
  --language-model-only --enforce-eager --max-num-seqs 8 \
  --no-enable-flashinfer-autotune \
  --reasoning-parser qwen3 --trust-remote-code

The flags, briefly:

• --host 127.0.0.1: the server is never exposed. Everything reaches it through the SSH tunnel in Step 4. vLLM without --api-key accepts any bearer token, so binding to localhost is the security boundary.
• --quantization modelopt: vLLM resolves this to modelopt_mixed for this checkpoint.
• --max-model-len 16384: the per-request context limit. Native context is 262K, but this is a 96 GB card holding 73.6 GiB of weights; 16K leaves 14.75 GiB for KV cache. See the note on the 326K figure below.
• --language-model-only: disables the vision and video encoders. Text only, which is also all I measured.
• --enforce-eager: no torch.compile, no CUDA graph capture, no JIT warmup. Slower in steady state, but this is the configuration that booted reliably on the first working run and I kept it for every number here. CUDA graphs and MTP speculative decoding are the obvious next experiments and are deliberately not mixed into these results.
• --max-num-seqs 8: a modest batch ceiling. The hello-world run used 4; the benchmark run used 8.
• --reasoning-parser qwen3: separates reasoning from content in the API. Thinking itself is disabled per request, see Step 5.

Lines to wait for in the log, with my timings:

Resolved architecture: Qwen4ExpForConditionalGeneration
Initialized PLE embedding ... weight_dtype=torch.float8_e4m3fn, weight_device=cpu, pinned=True
Loading weights took 382.39 seconds
Model loading took 73.6 GiB memory and 396.6 seconds
Available KV cache memory: 14.75 GiB
GPU KV cache size: 326,041 tokens, Maximum concurrency for 16,384 tokens per request: 19.90x
Application startup complete.

That 326,041 tokens is the whole KV pool, enough for roughly twenty 16K requests at once. It is not the per-request limit; that is 16,384. People misread this line constantly.

Model load is six and a half minutes from the overlay filesystem, and vLLM tells you why it cannot do better: Checkpoint size: 123.57 GiB. Available RAM: 105.72 GiB, so its read-ahead prefetch is disabled. With download, kernels and load, budget about 20 minutes and roughly 3 CU from allocation to first token.

Step 4: a tunnel, not an open port

The Colab CLI can act as an OpenSSH ProxyCommand, bridging SSH over its authenticated WebSocket. That gives you a real ssh with port forwarding and no public endpoint anywhere.

Make a dedicated key, then forward local port 18000 to the server’s 8000:

ssh-keygen -t ed25519 -f ./id_colab -N '' -C colab-flashnext

ssh -i ./id_colab -o IdentitiesOnly=yes \
  -o ProxyCommand="$(command -v colab) ssh --proxy-mode -s flashnext --identity $PWD/id_colab" \
  -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  -o ServerAliveInterval=15 -o ExitOnForwardFailure=yes \
  -N -L 18000:127.0.0.1:8000 root@colab-runtime

Things I learned the slow way:

• The SSH user is root. agent@colab-runtime fails with Permission denied (publickey); so will your local username.
• colab-runtime is a placeholder hostname. OpenSSH hands it to the ProxyCommand and never resolves it.
• The runtime’s host key changes every session, hence StrictHostKeyChecking=no and a throwaway known-hosts file. That is a real trade-off; the mitigation is that the transport is already authenticated by the Colab CLI’s OAuth session, and the only thing on the other end is a localhost-bound vLLM.
• The same ssh incantation, minus -N -L, is how to run commands on the VM: ssh ... root@colab-runtime 'nohup bash /content/setup_vm.sh > /content/setup.log 2>&1 &'.
• In my sandbox, background processes started in one tool invocation are gone by the next, tmux included. A tunnel opened in one step was dead by the time the client ran. The bundle’s tunnel_and_termllm.sh and run_tl_bench.sh therefore start the tunnel, wait for /v1/models to answer, run the client and kill the tunnel from a trap, all in one shell. If your environment is a normal terminal you can skip that, but the trap is still good hygiene.

Smoke test from the local side once the forward is up:

curl -s http://127.0.0.1:18000/v1/models
# {"object":"list","data":[{"id":"qwen3.8-flash-next", ... "max_model_len":16384 ...}]}

Step 5: an isolated term-llm provider

I did not want this experiment anywhere near the global term-llm config, so everything ran with XDG_CONFIG_HOME and XDG_DATA_HOME pointed at scratch directories. The config (term-llm-config.yaml in the bundle, installed as $XDG_CONFIG_HOME/term-llm/config.yaml):

providers:
  colab:
    type: openai_compatible
    base_url: http://127.0.0.1:18000/v1
    api_key: local-dummy-not-a-secret
    model: qwen3.8-flash-next
    context_window: 16384
    max_output_tokens: 2048
  colabvllm:
    type: vllm
    base_url: http://127.0.0.1:18000/v1
    api_key: local-dummy-not-a-secret
    model: qwen3.8-flash-next
    context_window: 16384
    max_output_tokens: 2048
    think: false
    vllm_thinking_param: enable_thinking

Two providers for one server, because:

• openai_compatible is the simplest thing that works for ask and chat.
• term-llm benchmark (schema v1) rejects openai_compatible providers and wants type: vllm. Add the second entry rather than fighting it.
• think: false with vllm_thinking_param: enable_thinking makes term-llm send chat_template_kwargs: {enable_thinking: false}. Without it Qwen thinks by default, and your output budget and your time-to-first-visible-token both go to hidden reasoning. I verified the switch: “Say hi in one word” returned a 2-token completion.

The hello world, with the tunnel up:

export XDG_CONFIG_HOME=$PWD/term-llm-config XDG_DATA_HOME=$PWD/term-llm-data
term-llm --stats --no-session ask -p colab --no-search --text --porcelain \
  --max-output-tokens 1024 \
  "Reply with exactly this sentence and nothing else: Hello from Qwen3.8-Flash-Next on Colab G4, served through term-llm."

Hello from Qwen3.8-Flash-Next on Colab G4, served through term-llm.

Stats: active 9.7s | 92 in → 99 out | TTFT 5.2s, 22 tok/s | $0.0001 | 1 call

Two things in that stats line need decoding. The 5.2 s TTFT is a cold first request through a fresh server: the next request at the same shape was 0.115 s (measured directly, below). And the $0.0001 is term-llm’s notional per-token price for the provider, not what the GPU cost; the GPU cost was accruing at 8.90 CU/hr regardless.

Benchmarks

I measured with two harnesses that answer two different questions, and the main thing I want to get across is not to mix their numbers.

Direct, on the VM. bench_direct.py (in the bundle) runs on the Colab VM against http://127.0.0.1:8000/v1, so there is no SSH, no WAN, and no client tokenizer in the measurement. It streams chat completions with enable_thinking: false, forces exactly 256 output tokens with ignore_eos and min_tokens = max_tokens = 256, builds a unique random-word prompt with a fresh nonce for every request and sizes it with vLLM’s /tokenize, takes TTFT at the first streamed content chunk, and computes decode rate as (completion_tokens − 1) / (end − first_chunk). Server-side prefill throughput is read from vLLM’s own /metrics: Δvllm:prompt_tokens_total / Δvllm:request_prefill_time_seconds_sum. One warmup per shape, then three measured repeats; medians reported.

It also reads Δvllm:prefix_cache_hits_total for every measured request and retries with a new prompt if it is non-zero. This is not paranoia. An earlier version of the harness seeded the warmup prompt identically to the first repeat, and prefill looked 5–7× faster than it was, with hit counters showing exactly 1,568, 7,840 and 10,976 cached tokens. Prefix caching is on by default in vLLM. If you benchmark prefill without checking this counter, you are probably benchmarking the cache. Every number below has a verified zero-hit delta.

term-llm, from the local host, through the tunnel. term-llm benchmark -p colabvllm --runs 3 --warmup 1 measures at the provider boundary, which is where a user of term-llm actually lives. It includes the SSH tunnel, HTTP, term-llm’s own request path and vLLM’s per-request overhead. It validates a cold cache itself, reports reasoning_expected: false with the config above, and --concurrency is fixed at 1 in schema v1.

# term-llm built-in modes (each via run_tl_bench.sh, which handles the tunnel)
term-llm --no-session benchmark -p colabvllm --mode quick   --runs 3 --warmup 1 --json --jsonl bench_tl_quick.jsonl
term-llm --no-session benchmark -p colabvllm --mode decode  --runs 3 --warmup 1 --json --jsonl bench_tl_decode.jsonl
term-llm --no-session benchmark -p colabvllm --mode prefill --runs 3 --warmup 1 --json --jsonl bench_tl_prefill.jsonl
term-llm --no-session benchmark -p colabvllm --mode balanced \
  --input-tokens 128,512,2048,8192,12000 --output-tokens 256 --runs 3 --warmup 1 \
  --json --jsonl bench_tl_balanced.jsonl

# direct harness, run ON the VM
BASE_URL=http://127.0.0.1:8000/v1 MODEL=qwen3.8-flash-next \
OUT_TOKENS=256 REPEATS=3 INPUTS=128,512,2048,8192,12000 python3 -u bench_direct.py

Direct results (localhost on the VM, warm server)

Spread across the three repeats was small: at 12K, prefill ranged 15,102–15,135 tok/s and decode 21.87–21.94; TTFT at each shape was identical to the millisecond across runs. The TTFT-derived client estimate of prefill (prompt tokens ÷ TTFT) tracks the server metric about 2–3% lower, which is the per-request overhead you would expect.

Cold start. The very first inference after Application startup complete, a 523-token prompt, took 4.79 s to first token with the server reporting a prefill rate of only 109 tok/s. The next request of the same shape took 0.115 s. That is one-time kernel and allocator warmup; it happens once per server process, not once per session or per shape.

Concurrency (2,066-token prompts, 256 output each, wall-clock aggregate):

Four streams took the same wall time as two, and each stream still decoded at ~21 tok/s. Batching is close to free up to four, which is what you would hope for from a 6B-active MoE. I did not push past four; --max-num-seqs 8 and the 14.75 GiB KV pool would allow more.

term-llm results (provider boundary, includes the tunnel)

term-llm fits a line through its own TTFT points (median pairwise slopes over the 2.1K–12K range) and reported an effective prefill of 16,280 tok/s with a fixed intercept of 0.436 s, with zero median absolute error on the fitted points.

Read those two numbers carefully:

• The 0.436 s intercept is a compound fixed cost: tunnel round trips, HTTP, term-llm’s request path, and vLLM’s own per-request scheduling and tokenization. It is tempting to call it “the tunnel”, but localhost TTFT on a 139-token prompt was already 0.110 s, so at most ~0.33 s of it is anything to do with being remote. Do not compare term-llm’s absolute TTFT with the direct table; compare slopes.
• The 16,280 tok/s slope is a client-observed effective inference rate, not a kernel rate. It agrees to within about 10% with the direct TTFT-derived rate at 12K (14,863 tok/s) and the server metric (15,125 tok/s). The agreement is the point: the tunnel adds a constant, not a per-token tax.

Decode at the provider boundary, 22.06–22.26 tok/s, is the same as on localhost. Output stopped at 125 and 216 tokens because term-llm’s benchmark lets the model hit EOS rather than forcing a token count.

What thinking mode costs

One request with enable_thinking: true on a 1,748-token prompt asking for 17*23: 61 completion tokens, of which 55 were reasoning_tokens, a visible answer of 5 characters, and 2.75 s to the first visible content versus about 0.2 s for a non-thinking request of that size. The stream carried no reasoning_content text at all in this run; the 55 tokens were simply counted in usage. So with thinking on you can pay for tokens you never see, and your apparent TTFT is really time-to-end-of-reasoning. Every benchmark number above was taken with thinking off.

What the numbers do and do not say

Decode is flat at ~22 tok/s regardless of context length, which is the pattern you expect once prefill is done and each step is dominated by per-token fixed work. With only 6B active parameters per token, this is very unlikely to be a memory-bandwidth ceiling on this card. The usual suspects are eager mode (every kernel launched from Python, no CUDA graphs), the absence of MTP speculative decoding, and the per-token n-gram lookup against a table that lives in host RAM. I did not isolate them, so I will not rank them. If you try CUDA graphs or MTP, report them separately from this baseline.

Prefill efficiency climbs with length because fixed per-request overhead dominates short prompts: 1.3K tok/s at 139 tokens is mostly overhead, 15.1K tok/s at 12K is mostly compute, and the curve is flattening by 8K.

These are throughput and latency measurements of an echo-style workload with forced output lengths. They say nothing about answer quality, tool use, or how the model behaves as an agent. The one “real” prompt it answered, it answered exactly.

Cost and cleanup

Observed billing on this account was 8.90 CU/hr for a G4, from allocation to stop. That rate and my balance are what I saw; allowances and prices differ by plan and over time, so check colab usage on your own account rather than extrapolating from mine.

The benchmark session is the honest “what does a full run cost” figure: an hour, about 9 CU, and roughly a third of that hour was spent redoing setup. Which brings up the thing that actually hurts.

The runtime is ephemeral. colab stop destroys /content, the pip installs, the FlashInfer kernels and the SSH host key. Every new session repeats the ~20 minutes of download, kernel install and model load before you get a token, and that time is billed. I did not test persisting the model shards to Google Drive (colab drivemount) to skip the download; with 124 GiB of shards it would need a correspondingly large Drive, and the overlay-filesystem load time would not change. If you plan to use this regularly, that is the first thing to measure. For an occasional benchmark, I would just eat the 3 CU.

Always finish with all three of these, and read the output:

colab stop -s flashnext
colab sessions    # must print: No active sessions found on server.
colab usage       # must print: Usage rate: 0.00/hr  and  Active assignments: 0
pgrep -af 'ssh.*18000' || echo "no tunnel left"

After the benchmark session mine read 385.53 compute units, 0.00/hr, 0. Those were independently re-checked afterwards; nothing was left running.

Pitfalls, in one list

1. Say --gpu G4. An A100 80 GB is the wrong card for this checkpoint.
2. Install the precompiled kernels before loading the model, and start with --no-enable-flashinfer-autotune and MAX_JOBS=1. Otherwise the first request can get the engine OOM-killed on the host, with a healthy-looking GPU.
3. Leave PLE CPU offload on. It is vLLM’s default for this model and it is why the weights fit.
4. First boot: --enforce-eager, 16K context, text only. Tune afterwards, and report tuned numbers separately.
5. Run long jobs detached and poll logs. The control WebSocket drops.
6. SSH user is root; colab-runtime is a placeholder; keep vLLM on 127.0.0.1.
7. Tunnel and client in one shell with a trap if your environment reaps background processes.
8. Two term-llm providers: openai_compatible for use, type: vllm for benchmark.
9. Turn thinking off for latency measurements, and know that reasoning tokens are billed inside completion tokens.
10. Assert zero prefix-cache hits on every measured request, or you are benchmarking the cache.
11. 326K is the KV pool, 16K is the request limit.
12. $0.0001 in term-llm’s stats is not the GPU bill.
13. Everything on the runtime is gone at colab stop. Budget ~20 min and ~3 CU per fresh session before the first token.

What I did not do

• No MTP speculative decoding, no CUDA graphs, no --async-scheduling, no --max-num-seqs above 8. All plausible wins; none are in these numbers.
• No FP8 checkpoint. It might fit; I did not try.
• No vision or video input.
• No quality evaluation of any kind.
• No Docker route; Colab has no Docker, and the standard NVFP4 checkpoint did not need one.
• The Colab image supplied Python 3.13, torch 2.13.0+cu130 and CUDA 13.0 on 2026-10-19. vLLM is pinned at 0.32.0 and the model at its revision; the image is not something I can pin.

Downloads

Everything needed to reproduce this, with no credentials and no private paths, is under /files/qwen38-flash-next-colab-g4/:

• README.txt: what each file is, exact versions, order of operations.
• setup_vm.sh: vLLM 0.32.0, precompiled kernels, pinned model download. Run on the VM.
• start_vllm.sh: the exact server launch behind every number here. Run on the VM.
• tunnel_and_termllm.sh: SSH local forward plus the hello-world request, one shell, trap-cleaned. Run locally.
• run_tl_bench.sh: tunnel plus term-llm benchmark. Run locally.
• term-llm-config.yaml: the isolated two-provider config.
• bench_direct.py: the direct harness with prefix-cache verification. Run on the VM.
• benchmark_results.json: consolidated results from both harnesses, including the cold-start record, concurrency rows, the thinking-mode accounting and term-llm’s prefill fit.

If you get it running with CUDA graphs or MTP enabled and the decode number moves, I would like to hear what it moved to.