LocalMode Bench methodology

Metric definitions

• TTFT - wall-clock time from requesting the stream to the first non-empty chunk, measured with performance.now() on a cross-origin-isolated page (5 µs timer resolution on Chromium). Browser TTFT includes tokenization, prefill, the first decode step, and runtime scheduling.
• Decode throughput (tg128) - characters after the first chunk divided by the time between the first and last chunk; the first token is excluded (the MLPerf Client TPS definition), and rates derive from endpoint timestamps, never averaged per-token deltas (which sit at timer quantization). Chars/sec is the headline (tokenizer-independent); exact tokens/sec is computed post-hoc from the stored generated text.
• Stream coherence - TTFT, decode, and prefill rates are derived only when every timed iteration's chunk trace is genuinely incremental: at least two non-empty chunks whose visible span covers at least 20% of the request wall time. Some runtime surfaces compute the whole generation and flush every chunk in a terminal burst (observed on LiteRT-LM: 128 chunks inside 0.8 ms of a 30 s request); deriving decode timing from such a trace would report an artifact, so those lanes report the end-to-end rate instead, marked e2e.
• End-to-end rate - total generated characters over the full request wall time (prefill and decode conflated), reported for every generation lane. It is the only rate lanes with non-incremental streams can honestly claim.
• Prefill (pp128 / pp512) - approximate prompt tokens over TTFT, on fixed public prompts (llama-bench naming).
• Model load - the provider's preload path, cold (cache probe false) vs warm (probe true) reported separately. This is download-to-cache for wllama, and download plus a first engine/session initialization (then released) for WebLLM, LiteRT, and Transformers.js, so load alone is not comparable across runtimes. The untimed warmup that follows records first-inference readiness (engine init, shader/JIT compilation) as its own number; cold start = load + warmup is the cross-runtime comparable figure. The warm-reload cell repeats the preload path against a populated cache, so for wllama it times the cache probe; after that timed span the runner loads llama.cpp untimed, so the cell records the same offload report as the timed cells (before site build 2.16.15 its offloadedLayers read "unreported"). On the llama.cpp lanes the recorded backend is WebGPU only when llama.cpp offloaded at least one layer; a WebGPU-lane cell that offloaded none records WASM with gpu_fallback: true. Since site build 2.16.19 every llama.cpp cell also records wllamaBuild, the WASM build wllama started its worker on: memory64 (its default build) or compat (its compatibility build, 32-bit memory without JSPI). wllama picks the compatibility build only on a browser without JSPI or Memory64, such as Safari, and never on Firefox. On the LiteRT lane the recorded backend is the one the engine loaded on (since site build 2.16.16): LiteRT-LM 0.12.1's GPU backend refuses to stream-load some builds, the Qwen3 0.6B file among them, and the provider then creates the engine on the CPU backend, which the cell records as CPU with gpu_fallback: true; earlier LiteRT cells recorded GPU whenever the browser had a WebGPU adapter. Chrome Built-in AI is the exception: Chrome downloads Gemini Nano once, browser-wide, and only from a user activation, so the Run click starts that download while the other lanes run; the lane is recorded cold in that case and its load phase measures the remaining wait, not the whole download.
• Embeddings - single-query latency (median) and batch-32 throughput reported separately; WASM can win at batch 1 while losing at scale, so one number would mislead.
• Memory - performance.measureUserAgentSpecificMemory() deltas at protocol points (baseline → post-load → post-run, plus a failure-time sample on error cells), Chromium-only, never inside a timed region.

Run policy

• Per cell (runtime × model × workload): 1 untimed warmup, then 3 timed runs (5 in the thorough suite).
• Watchdog: a load that reports no progress for 3 minutes, a generation that streams nothing for 2 minutes, or a call that outlives its absolute budget (10 to 15 minutes per iteration, 30 to 45 per quality lane) is aborted as a timeout; the cell is retried once with the failed attempt kept on the cell, then recorded as an error, and the run continues. Nothing is retried silently. An abort a runtime raises on its own (a WebGPU buffer that can no longer be mapped, a failed fetch) is an error with its cause and the same single retry, never a cancellation: only the submitter's Cancel ends a run, and it ends it at once. The Transformers.js WASM lane runs in a dedicated worker so the page stays responsive during inference.
• Hidden tab: browsers throttle background tabs (timers, process priority, GPU work; iOS can suspend the tab), so a timed iteration the tab was hidden during measures the browser's scheduling, not the runtime. It is set aside on the cell, the runner waits up to 10 minutes for the tab to be visible again, and the iteration is repeated (twice at most); a tab that stays hidden leaves the cell invalid with the reason. Iterations never start while the tab is hidden.
• Performance runs use temperature 0, a 128-token generation budget, and fixed public prompts.
• Every runtime receives the fixed prompt as a single user turn through its own chat template; runtime-native templating is part of what is measured. Cross-request prompt/KV caching is disabled where a runtime enables it by default (wllama's cache_prompt), so each timed iteration pays prefill - with the cache on, a repeated prompt's TTFT fell from 1,018 ms to 24 ms from the second iteration.
• A timed iteration that generates fewer than 16 characters is gated as degenerate output (an instruct model answering with EOS has no decode phase to time) and the cell is marked invalid rather than scored.
• Deterministic runtime execution order: transformers-wasm, transformers-webgpu, chrome-ai, webllm, mediapipe, litert, wllama-webgpu, wllama. Fixing the order makes runs reproducible (runtime interleaving is not a confounder) and runs the WASM-arena runtimes before the multi-gigabyte-heap runtimes; it is a reproducibility measure, not a correctness fix. The Transformers.js WASM lane runs before its WebGPU lane because Transformers.js serializes every ONNX session creation on one promise chain that never catches a rejection: the first session that fails to create (a WebGPU execution provider the browser cannot initialize, an allocation failure under memory pressure) fails every later Transformers.js session in the page with the same error, which is how an iPhone on iOS 18 lost both Transformers.js lanes to one WebGPU error.
• 5–10 s cool-down after every model group that ran (5 s quick, 8 s standard, 10 s thorough; a group whose cells were all skipped pays none); on Chromium the next group also waits for CPU pressure to recover (15 s cap in quick, 30 s otherwise).
• A screen wake lock is requested and the trace records the outcome (wakelock-acquired or wakelock-denied); timed regions overlapping a hidden tab, a wake-lock release, or a GPU device loss are invalidated and recorded - never silently retried.
• Quality-fidelity lane (optional, untimed): tinyMMLU accuracy and STS-B Spearman at temperature 0 - these measure whether a runtime's build of the weights reproduces expected outputs, not model capability. MMLU items use a 48-token budget; reasoning blocks (<think>) are stripped before answer parsing; pairings that ship thinking-mode builds carry a fixed no-think prompt suffix applied identically to every runtime of that pairing; raw per-item outputs and the parse rate are stored so every score is auditable and recomputable. Unparsed items count as wrong, and the parse rate is shown beside the score: a low parse rate marks a format-limited result (a build that reasons out loud past the budget), not low fidelity.

Series, cold runs and link presets

None of this changes what a run measures: a series run or a cold run follows the same protocol as any other run and is scored and aggregated the same way. The runner only records how the run was started.

• Series. The runner's Runs control (1 to 30) repeats the run with the same suite, quality lane, lanes, publishing and cache setting. Every run of a series executes on a fresh page load: after each run the page keeps the result (published, or downloaded as JSON when publishing is off or the submission fails), reloads itself and starts the next run without a click. The reload is deliberate: the Transformers.js lanes share one ONNX Runtime WebAssembly heap per page that never shrinks, and the analysis counts one fresh page load as one independent sample. Each run file carries harness.series ({ id, index, count }). Stop series ends the series after the run in progress (never mid-run). A run that fails, is cancelled, or takes the page down pauses the series with the reason; it continues only on a click, so a run that crashes the page cannot loop.
• Cool-down between runs. An optional idle wait of 0 to 30 minutes (half-minute steps, default 0) between the runs of a series. After a run's result is kept (and the caches cleared, when that is on) the page stays loaded and idle, shows a countdown in the series box ("Cooling down: 2:45 until run 4 of 10"), keeps the wake lock and the tab title progress, and only then reloads for the next run; Stop series during the countdown ends the series at once. Laptops and phones need it: a 7-run back-to-back Quick series on a 15 W class laptop showed the llama.cpp CPU decode falling from about 135 to about 85 chars/s from run 3 on. The pause is idle time on the page, not a protocol change. Each run file of a series records the configured cool-down as harness.series.cooldownMs (run 1 included, because it describes the series) and the idle time measured before the run, from the moment the previous run was kept to this run's start across the reload, as harness.series.idleBeforeMs (absent on run 1 and on a run resumed with Continue series).
• Clear model caches. Deletes the model files the page's runtimes keep for the site: Transformers.js (Cache API transformers-cache), WebLLM (Cache API or IndexedDB webllm/*), LiteRT (Cache API litert-models) and wllama (the Origin Private File System directory cache, through wllama's own clearAllModelCache). It can run on demand or after every run. It cannot reach the browser's HTTP disk cache (no page can clear it, and MediaPipe keeps its files only there) or Gemini Nano, which Chrome installs browser-wide. A run that starts right after a clear that left those stores empty records harness.coldStart: "provider-caches-cleared"; a fresh browser profile cannot be detected from a page and is never claimed. The per-cell load.cached probe still says whether each provider found its model. Because the HTTP cache survives, a file can come from disk instead of the network, so an in-page cold load can be shorter than a first load in a new browser profile. Observed on Chrome 145 (macOS, Quick suite): after a clear, every Hugging Face-hosted weight file came over the network again, while MediaPipe's model and WASM files (which it never caches itself) came from the HTTP disk cache.
• Link presets. /bench/run?tier=quick|standard|thorough&quality=on|off&runs=N&cooldown=M&cold=on|off&publish=on|off prefills the suite, the quality lane, the series length, the cool-down between runs in minutes (0 to 30, rounded to the nearest half minute), clearing after each run, and publishing. A link never starts a run; only the reload inside a series does.
• Control runs. /bench/run?control=order-reversed runs the same suite with the runtimes in the exact reverse of the protocol's execution order, a control experiment for the methodology: under the protocol order a lane is tied to its position in the suite, and comparing the lane ratios of the two orders separates the two. /bench/run?control=fixed-order runs the suite in the protocol's own order, marked as a control run: a reference series recorded with the same browser build on the same day as a reversed series, so the two orders are compared within one build. /bench/run?control=wasm-default runs the suite in the protocol's order with the Transformers.js WASM lane at the runtime's own defaults: its models load at q8 (Transformers.js's WASM default) and ONNX Runtime runs one WASM thread per logical core instead of its default of at most four; every other lane is unchanged. /bench/run?control=wasm-q8 and /bench/run?control=wasm-threads each make one of those two changes, in the protocol's order: wasm-q8 loads the WASM lane at q8 and leaves ONNX Runtime's thread count at its default; wasm-threads keeps the protocol's files (q4 language model, fp32 embedding model) and runs one WASM thread per logical core. With fixed-order they separate the effect of the dtype from that of the thread count. /bench/run?control=wllama-compat runs the suite in the protocol's order with every lane at the protocol's configuration, except that both llama.cpp lanes load through wllama's compatibility build instead of the memory64 build, in any browser; beside a fixed-order series it shows what the build changes, for instance on Firefox, where wllama never picks the compatibility build by itself. /bench/run?control=current-releases runs the suite in the protocol's order and configuration on the current npm releases of Transformers.js, WebLLM, wllama and LiteRT-LM instead of the pinned ones (a second build of the provider packages, loaded only on this kind); every cell of those lanes records the version it ran, and a lane that fails on a current release is an error cell like any other. /bench/run?control=ort-profile runs the suite in the protocol's order and configuration, its timed iterations unprofiled, and after the Transformers.js Qwen3 cells runs an untimed pass per cell in a fresh session with ONNX Runtime Web's profiler on; the cell carries the operator times of that pass (per phase, operator type, kernel and provider), and the raw profile is kept beside the run. /bench/run?control=llamacpp-profile runs the suite in the protocol's order and configuration and time-stamps the llama.cpp lanes' token delivery path (the wllama worker's receipt and reply, the page's send and receipt, and the chunk handed to the runner); each llama.cpp generation cell carries the per-request gaps and stage times. Every kind runs on any tier. chrome-ai-tasks is not a suite run: on its own page it runs Chrome's built-in Summarizer, Translator and Language Detector against open fallback models run in the page, one item per call, and records each item's timings and output; it is export-only like every kind, the protocol stays localmode-bench/5, and its fields add to the schema without changing it. Cells, rules, traces and the environment capture are those of a protocol run; the run file carries harness.control (the kind and the order used, for wasm-default, wasm-q8 and wasm-threads the WASM lane's dtype and thread count, for wllama-compat the llama.cpp build, and for current-releases the runtime versions). A control run is never uploaded (Export JSON only, and the submission route refuses one), never enters the leaderboard and is never scored, so it needs no protocol version. Runs and the cool-down work as usual, and a series keeps its control mode across its reloads. A study link with the parameter is refused. Every run the page exports is first kept in the browser and listed under Saved exports on the run page, so a download the browser blocked is not lost.
• While a run or series is active the page holds a screen wake lock (requested again whenever the tab returns to the front) and shows the series progress in the tab title, for example 3/10 · LocalMode Bench.
• Safari and every iOS browser. WebKit grants the screen wake lock only on a user interaction. Run 1 of a series starts from the Run click and holds it, but every later run starts from the page's own reload and is refused, so without help the device sleeps during the series (Chromium and Firefox grant the lock after the reload). On WebKit the page asks again on the next tap or key press ("Tap anywhere to keep the screen awake"), and the series box explains how to keep the device awake for the whole series: on iPhone or iPad, Settings, Display & Brightness, Auto-Lock, Never; on a Mac, caffeinate -d in Terminal or a display that never turns off. The run file records the outcome in its trace: wakelock-denied with the error name when the request at the start of the run is refused, and wakelock-acquired with the detail user-activation when a later tap granted it (since bench 0.9.2). The page uses no silent video or audio to keep the screen on.

Statistics

Per metric: median headline; mean ± SD, IQR, and a Student-t 95% confidence interval in the payload. A coefficient of variation above 5% marks the cell high-variance. Geometric means are used only within one device's run; the leaderboard shows the median of per-submission medians and marks any (device, runtime, model, workload) group with fewer than 3 submissions provisional. Devices group by what the browser discloses, at two levels: the coarse class is the platform plus the WebGPU adapter's vendor and architecture (macos/apple-metal-3, windows/amd-rdna-2, or no-webgpu), which every browser can supply and which is the unit for cross-device rollups; the subclass splits a class by the GPU model where the browser names a specific part (macos/apple-m1-pro, android/adreno-650) and equals the class where it names nothing more specific (Safari's "Apple GPU", a generation-less "AMD Radeon(TM) Graphics", Firefox's masked "..., or similar" buckets). Leaderboard rows are subclass rows; the class is shown beneath. Nothing typed by a submitter enters either. Rows never mix protocol versions: every row carries the version its runs were measured under, and the leaderboard shows the current protocol and the previous one side by side (v5 beside v4, because v5 changed only the llama.cpp lanes and every other lane measures identically). Older archived runs stay in the dataset and never enter a leaderboard row.

Environment capture

Every run records what the browser discloses about the device, whether or not the leaderboard uses it yet, so later analyses never depend on a field that was not kept. Chromium reports UA Client Hints (platform, version, architecture, bitness, model, form factors); Firefox and Safari freeze their user-agent strings by design, so their OS versions are recorded as unknown rather than guessed, and the raw user-agent string is kept verbatim for future parsers. That now includes iPhones: Safari 27 on iOS 27 advertises "iPhone OS 18_7" (WebKit froze the token there, as macOS froze at 10_15_7), so a Safari or Firefox for iOS run whose token reads 18_7 under a Safari version of 26 or higher is recorded as unknown, while Chrome for iOS still writes the real version. The form factor (phone, tablet, desktop) is derived from the hints, the user agent, and touch points. WebGPU adapter identity comes from adapter.info together with the adapter's feature list and limits; the WebGL renderer string is kept as a second GPU identity channel and parsed into a GPU model (for example "Apple M4" or "NVIDIA GeForce RTX 4070"). Core counts and device memory are recorded but labeled clamped (browsers cap or randomize them: Chrome reports memory as a bucket capped at 8 GB on Android and older desktop builds and at 32 GB on current desktop builds, so the value is a floor, never the size; Safari clamps core counts); the JavaScript heap ceiling, storage quota, battery state, CPU pressure support, network type, display and locale tag are captured where the APIs exist. The WebAssembly proposal matrix (SIMD, relaxed SIMD, threads, exceptions, GC, memory64, tail calls, JSPI and the rest) is probed by validating canonical modules, and the availability of every API the runtimes depend on (WebGPU, WebNN, OPFS, Cache API, workers, Chrome Built-in AI) is recorded as a presence check. The harness stamps the exact versions of the runtime packages it bundled (and wllama's CDN pin), per run and per cell. The resolved execution backend (WebGPU vs WASM vs CPU) is probed, never assumed from the request.

What a public run file does not carry (schema 3): the time zone, UTC offset and calendar (they place a device in a city), the language list, the exact battery percentage and time-to-full (they track a device across runs; the level is kept to the quarter, and whether it is charging), display preferences such as color scheme, the submission nonce, and the submitter's network address, which the server uses only as a rate-limit key and never writes down. A file published before schema 3 that carried any of these was rewritten without them and carries scrubbedAt with a recomputed digest.

Submission integrity

• Verified-tier runs happen on this site and carry a server-issued, time-boxed session nonce that accepts exactly one submission (a second upload under the same nonce is refused; the page fetches a new one right before every upload, so a run that takes most of a day still uploads). Submissions are limited to 20 per hour per network address; a refused upload is retried by the page after the wait the server states, and nothing about the run is lost in between. A finished run whose upload never went through can be added by a maintainer from the file the participant exported: it passes the same checks as an upload except the session nonce, and its index entry carries importedAt. A run the page assembled after a series had finished, which carried that series' record numbered past its count, is imported with the record removed as a publication scrub (scrubbedAt stamped, digest recomputed after the participant's digest verified), and its entry carries importNote.
• Submissions contain the raw per-chunk timestamp trace and the full generated text; the server recomputes every statistic from the trace and rejects client summaries that disagree.
• Versioned plausibility rules: timestamp monotonicity, decode-rate and end-to-end-rate envelopes by model size, text/chunk-length agreement, stream-coherence gating of TTFT/decode claims, rejection of an ok cell whose timed iteration generated fewer than 16 characters (the degenerate-output gate, re-checked server-side), MMLU score recomputation from the stored raw outputs, timer-quantization-grid conformance, environment cross-field consistency, software/virtual-renderer detection (a GPU-lane result from a SwiftShader/WARP-class adapter is rejected - cloud VMs report CPU numbers as GPU numbers), and a mandatory deterministic matmul calibration (at least 600 ms of single-threaded f32 matmuls) whose recorded throughput must fall within silicon reality.
• Flagged runs are quarantined publicly (hidden from charts, never deleted). The entire dataset - verified and quarantined - is an open GitHub repository anyone can audit.
• Residual limits, stated honestly: we cannot detect background native load, virtual machines, or browser flags; the min-3-submissions rule and median-of-medians limit their influence. Memory is the other edge: a standard suite peaks near 9 GB of browser memory and a thorough suite above 8 GB, so on a 16 GB machine with other applications open the Transformers.js lanes (one shared ONNX Runtime WASM heap per page, which never shrinks) can fail session creation with std::bad_alloc, after which every later Transformers.js cell in that run fails too. Those cells are recorded as errors with their cause and a failure-time memory sample, never as data.
• Known device limitation: on the Galaxy Z Fold 7 (Adreno 830, Chrome 153 on Android) the llama.cpp WebGPU lane (wllama-webgpu) generates incoherent SmolLM2 text. All 18 timed chat iterations recorded on that phone across six runs and three protocol versions are strings of isolated letters and word fragments, and the lane's MMLU cell fails in every run with "Invalid typed array length". The WASM lane and every other runtime on the same phone produce coherent text, and no other device in the dataset shows this. The degenerate-output gate is length-based and 17 of the 18 iterations pass it, so those cells are ok and the adreno-830 wllama-webgpu SmolLM2 chat rows (about 60 chars/s under v4 and v5, plus a v2 adreno-830 llama.cpp row from before the lane split) are kept: they time a backend that is not producing a usable answer. The generated text of every iteration is stored in the run files, so anyone can check it. An output-coherence check would be a protocol change and is reserved for a future version.
• Known prompt limitation: the Llama 3.2 1B chat template writes the current local date into its default system header ("Today Date: 25 Sep 2026", through the template's strftime_now). The bench sends no system prompt, so on the three Llama lanes that render this template (Transformers.js WebGPU, llama.cpp CPU and llama.cpp WebGPU) the prompt changes with the device's local calendar date. Decoding is greedy, so on the Transformers.js lane each date yields a different fixed output, and the decode rate follows the text: with no change to the model files, the browser or the harness, the pp128 decode rate of one machine moved by about 9 to 13% from one date to another, stepping at local midnight on the two machines that ran across it, and one tinyMMLU item changed from wrong to right. The llama.cpp Llama lanes already produce a different text in every iteration, so there the date changes the prompt without leaving a mark in the output. The Qwen3, Gemma 4 and SmolLM2 templates carry no date on any lane. WebLLM's templates insert a fixed default system message instead (for Llama 3.2 1B, "You are a helpful, respectful and honest assistant."), so the prompts of one pairing already differ across runtimes; runtime-native templating is part of what is measured. In practice: same-day comparisons on a Llama lane use identical prompts; comparisons across days carry the date effect; and for crowd submissions the local date cannot be recovered (schema 3 stopped recording the time zone), so a Llama-lane row aggregates runs made with different prompts. The generated text of every iteration is stored in the run files, so anyone can partition the Transformers.js Llama cells by output. Pinning the date (or sending one fixed system prompt on every lane) and recording a digest of the rendered prompt per cell would be a protocol change and are reserved for a future version; no run is re-scored.
• Known model-file limitation: until site build 2.16.16 the bench downloaded the LiteRT and llama.cpp model files from each Hugging Face repository's main branch, and the providers cache a file by its URL without checking it again. On 2026-10-12 and 2026-10-13 the litert-community/Qwen3-0.6B repository replaced Qwen3-0.6B.litertlm with builds made for LiteRT-LM 0.18.0. On the LiteRT-LM 0.12.1 the bench runs, their chat template renders an empty user turn and the reply arrives on a channel the provider does not read, so a browser that downloaded the file after 2026-10-12 produced empty pp128 text and pp512 replies that ignore the prompt; browsers that had cached the earlier file kept running it. Eleven published v5 Thorough runs (one RTX 3080 Ti, 12 and 13 October) loaded the replaced file: their LiteRT Qwen3 pp128 and pp512 cells are invalid (degenerate output), and their LiteRT Qwen3 MMLU cell is recorded ok with a score of 0 at a parse rate of 0, which is not a fidelity reading. The dataset README lists the runs, and since site build 2.16.17 those MMLU cells and the one warm-reload cell that timed the replaced file are excluded from the leaderboard by erratum (see Errata); nothing is re-scored. Since 2.16.16 every model file URL is pinned to a repository revision (Qwen3 0.6B to the file every valid v5 LiteRT Qwen3 cell measured, unchanged from 2026-07-17 to 2026-10-12; the other files to the revisions they have had since before v5), and LiteRT cells record the revision and the bytes the engine read. This is not a protocol change: no prompt, budget, policy number or integrity rule changes, and the pin restores the file v5 measured. Moving the lane to the new file or to a newer LiteRT-LM would change the model and its template contract, and would bump the version.

Errata

A published cell that turns out not to measure what the protocol describes is excluded, never re-scored. The dataset keeps the list in index/errata.json: one entry per run, naming the run, the cells excluded, the reason, the date the erratum was recorded and where the evidence is documented. An entry names only cells recorded ok, because a cell with any other status already enters no aggregate. The run file is not changed: the cell keeps its status and its values. The leaderboard, the leaderboard API and the analysis tooling leave the listed cells out of every median and count them in an "excluded by erratum" tally, per row and in total, so an exclusion is never silent; a row whose every cell is excluded stays with zero runs. An entry that names a run or a cell the dataset does not hold is refused. Errata change no measurement rule, so they never bump the protocol version. The first entries exclude twelve cells of the eleven runs that loaded the replaced LiteRT Qwen3 file (see Residual limits): each run's MMLU cell, recorded ok with a score of 0 at a parse rate of 0, which is not a fidelity reading, and one run's warm-reload cell, recorded ok but timing the replaced file.

Protocol changelog

• localmode-bench/5 (2026-09-22) - the llama.cpp lanes request half the browser's logical thread count (at least two) instead of all of it, and load with a 2,048-token context. Under v4 the CPU lane asked for every logical thread, and on hybrid and SMT processors a pool that spans the efficiency cores or the second hardware threads runs at half speed with high variance: natively on an M1 Pro, tg128 was 178 ± 42 tokens/s at 10 threads against 395 ± 16 at 8, and in the browser the M4 Max's 16-thread pool decoded a third as fast as the M1 Pro's 10. The browser exposes no core topology, so half the logical count is the rule; it lands on the performance or physical cores on every lab device, and both the requested count and the pool the runtime built are recorded on the cell. The context change sizes the KV cache to what the workloads need (at most about 700 tokens) instead of the provider's 8,192 default, which matters most for the 3.46 GB Gemma 4 E2B GGUF on the CPU lane, under the 4 GiB link-time memory cap of wllama's memory64 build (the compatibility build for browsers without JSPI or Memory64, such as Safari, is 32-bit). It does not rescue that lane's Gemma quality cell: with the weights resident, the per-request state allocation still fails (std::bad_alloc) and the cell is recorded as an error, as it was under v4. Every other lane measures exactly as under v4, so the leaderboard shows v4 rows beside v5 rows; nothing is re-scored.
• localmode-bench/4 (2026-09-20) - the llama.cpp lanes load every language model as text only. The wllama provider attaches the vision projector its catalog lists for Gemma 4 E2B, so under v3 both llama.cpp lanes loaded a 557 MB projector the text-only workloads never use: its download and CLIP warmup ran inside the untimed warmup, the provider turned wllama's model cache off for the two-file source (so the warmup and the warm reload re-downloaded the 3.46 GB weights), and on the CPU lane the pair did not fit under the 4 GiB link-time memory cap of wllama's memory64 build, which failed the three Gemma 4 E2B cells on every Thorough run. v4 loads the language model alone (runtimeConfig.mmproj: false); the other pairings measure exactly as under v3, and the version is bumped so that no leaderboard row mixes the two configurations.
• localmode-bench/3 (2026-09-20) - the wllama lane is split in two. Under v2 the lane was documented and recorded as llama.cpp on the CPU (WASM), but wllama 3.5 offloads every layer to WebGPU by default whenever the browser exposes it, and llama.cpp's own load log on those runs reads "offloaded 31/31 layers to GPU": every v2 wllama cell measured on a WebGPU-capable browser is a llama.cpp-WebGPU number labelled "wasm", and its comparison against the Transformers.js WASM lane was GPU against CPU. v3 pins the wllama lane to the CPU (n_gpu_layers 0) and adds wllama-webgpu (every layer offloaded) over the same GGUF files; every cell now carries a runtimeConfig with the thread count, the GPU layers requested, and llama.cpp's offload report, and the recorded backend follows that report, not the request. Archived v2 runs remain published as v2; read their wllama cells as WebGPU wherever environment.gpu.available is true.
• localmode-bench/2 (2026-09-19) - from three real-Chrome thorough-suite pilots; the third ran all 51 cells green. (1) Stream-coherence gating - LiteRT-LM's surface flushed all chunks in a terminal burst, which v1 scored as a 693,902 chars/s decode rate (correctly quarantined by the envelope rule); v2 derives TTFT/decode only from incremental streams and reports the end-to-end rate otherwise. (2) Quality lane rebuilt - the 8-token budget scored thinking-mode builds (Qwen3) at 0 for format reasons, not fidelity; now a 48-token budget, reasoning-block stripping, uniform per-pairing no-think suffixes, stored raw outputs, server-side recomputation, and a surfaced parse rate. (3) Degenerate-output gate. (4) Uniform user-turn contract with cross-request prompt caching disabled (wllama's prompt-KV reuse had cut repeated-prompt TTFT 40x). (5) A deterministic runtime execution order for reproducibility (not a correctness fix: the Transformers.js failures that motivated it were primarily a session leak in the provider's preload path, fixed in @localmode/transformers 4.1.2, independent of order; the shared ONNX Runtime heap itself remains fragile under memory pressure, see Residual limits). Archived v1 runs remain published as v1 and are never re-scored.
• localmode-bench/1 (2026-09-18) - initial public protocol.

Reproduce it

The harness is MIT-licensed. Run the suite at /bench/run, export the raw JSON, or wire your own runtimes into @localmode/bench - the adapter interface takes any model exposing a streaming generation method. Aggregation and CSV tooling for analysis ship in the same package.