Benchmarking

Every speed claim in Pooled names the hardware and commit behind it. Two ways to contribute numbers:

• A device we don’t have: send a benchmark report (no clone needed). One of the most useful things you can send.
• A PR that changes speed: add a row to docs/bench-log.md with before and after.

Report numbers from your hardware

Open a room on pooled.run, load a model, ask one of the standard prompts below twice, then open a benchmark report with:

• Hardware, for example MacBook Pro M4 Max 64 GB, RTX 4090 or Pixel 9;
• Browser or runtime, and its version;
• Commit or site build, for example pooled.run on 2026-10-11;
• Numbers: the model, the room size, whether the devices were on the same network, prefill and decode tokens/s (plain and speculative if shown), and the breadcrumb line.

Report the second answer or later: the first is slower while speculative decoding warms up.

Standard prompts

Use one of these word for word, and name it:

Name	Tokens	Use it for
hello	1	Fixed per-request overhead
meaning	6	Fixed per-request overhead. The text is what is the meaning of life.
japan	160	Any claim about prefill. It is long enough to run through the 16-column prefill path.

Token counts are for the Qwen 3.8 27B tokenizer, text only. The chat template adds about 9 more. The full japan text is in docs/bench-log.md. Decode numbers use the 400-token answer cap.

Measure a change

Pick the harness that matches what you changed. All of them need a GPU, and none of them should run while another GPU job is running.

Engine (Deno)

npm run bench                       # Qwen3 0.6B: prefill and decode tok/s
npm run bench:q38                   # the full 27B, with the room's settings
TOKENS=128 npm run bench:q38        # time more decode tokens (default 32)
ROOM_FLAGS="kv=q8" npm run bench:q38   # the same with a room switch

For context length and per-kernel time:

cd tests
D="deno run --unstable-webgpu --allow-read --allow-env --allow-write=$HOME/.cache/swarmllm-weights"
MODEL=moe FILLS=1024,4096,16384 $D bench_ctx.js   # prefill and decode against context fill (MODEL=27b for the dense)
Q38=1 $D prof_ts.js                              # per-kernel GPU ms (drop Q38 for the MoE)

Chrome

The numbers we quote for browsers come from a real Chrome page with the real GPU:

node tests/bench/chrome_bench.mjs models/q36moe/Qwen_Qwen3.6-35B-A3B-Q4_0.gguf

After the model path, an optional second argument sets the number of decode tokens (default 40) and a third adds page options, for example "ubatch=256&prefilllen=2048". Loading is kept out of the tokens/s numbers. The first run fills a converted-weights cache and later runs load faster. Set WCACHE=0 or CHROME_PROFILE=0 to measure a cold load.

Rooms

One machine, several headless Chromium tabs, with emulated network latency:

node tests/e2e/room_latency.mjs --model qwen3.6-35b-moe --devices 2 --lat 0,5,20 --maxnew 128

The latency is added in the page and doesn’t model bandwidth or loss. For real links, use tests/e2e/xroom_pair.sh across two machines (see Testing).

Tip

Profile by leaving kernel families out, not with GPU timestamp queries, which make everything about 3 times slower. Don’t quote a traced room round’s tokens/s.

Add a row to the bench log

Any PR that claims a speed change (neutral results too) adds a row to docs/bench-log.md: date and change, commit, hardware and runtime, before and after, and anything that makes the numbers hard to compare.

Measured in a room on real devices? Also add a row to Room benchmarks by PR: “before” is main the same day, “after” the PR’s preview; prefill from the status line, decode from the answer’s footer.

Output must stay bit-exact: the golden tests pass and the speculative stream equals plain decoding.

Summaries of the headline numbers are on Benchmarks.