LocalMode Bench methodology

Protocol localmode-bench/1. Any change to prompts, budgets, policy numbers, or integrity rules bumps this version; archived runs are never re-scored silently. The reference implementation is the open-source @localmode/bench package - every definition below is executable code, and every published statistic is recomputed server-side from each submission's raw trace.

Metric definitions

  • TTFT - wall-clock time from requesting the stream to the first non-empty chunk, measured with performance.now()on a cross-origin-isolated page (5 µs timer resolution on Chromium). Browser TTFT includes tokenization, prefill, the first decode step, and runtime scheduling.
  • Decode throughput (tg128) - characters after the first chunk divided by the time between the first and last chunk; the first token is excluded (the MLPerf Client TPS definition), and rates derive from endpoint timestamps, never averaged per-token deltas (which sit at timer quantization). Chars/sec is the headline (tokenizer-independent); exact tokens/sec is computed post-hoc from the stored generated text.
  • Prefill (pp128 / pp512) - approximate prompt tokens over TTFT, on fixed public prompts (llama-bench naming).
  • Model load- the provider's download/cache phase, cold (cache probe false) vs warm (probe true) reported separately. The untimed warmup that follows absorbs and records first-inference readiness (engine init, shader/JIT compilation) as its own number - cold start = load + warmup.
  • Embeddings - single-query latency (median) and batch-32 throughput reported separately; WASM can win at batch 1 while losing at scale, so one number would mislead.
  • Memory - performance.measureUserAgentSpecificMemory() deltas at protocol points (baseline → post-load → post-run), Chromium-only, never inside a timed region.

Run policy

  • Per cell (runtime × model × workload): 1 untimed warmup, then 3 timed runs (5 in the thorough suite).
  • Performance runs use temperature 0, a 128-token generation budget, and fixed public prompts.
  • 8–10 s cool-down between model groups; on Chromium the next group also waits for CPU pressure to recover (30 s cap).
  • A screen wake lock is held; timed regions overlapping a hidden tab, a wake-lock release, or a GPU device loss are invalidated and recorded - never silently retried.
  • Quality-fidelity lane (optional, untimed): tinyMMLU accuracy and STS-B Spearman at temperature 0 - these measure whether a runtime's build of the weights reproduces expected outputs, not model capability.

Statistics

Per metric: median headline; mean ± SD, IQR, and a Student-t 95% confidence interval in the payload. A coefficient of variation above 5% marks the cell high-variance. Geometric means are used only within one device's run; the leaderboard shows the median of per-submission medians and marks any (device, runtime, model, workload) group with fewer than 3 submissions provisional.

Environment capture

Chromium reports UA Client Hints (platform, version, architecture, model); Firefox and Safari freeze their user-agent strings by design, so their OS versions are recorded as unknown rather than guessed. WebGPU adapter identity comes from adapter.info; core counts and device memory are recorded but labeled clamped (browsers cap or randomize them). Battery charging state and CPU pressure are captured where the APIs exist. The resolved execution backend (WebGPU vs WASM vs CPU) is probed, never assumed from the request.

Submission integrity

  • Verified-tier runs happen on this site and carry a server-issued, time-boxed session nonce.
  • Submissions contain the raw per-chunk timestamp trace and the full generated text; the server recomputes every statistic from the trace and rejects client summaries that disagree.
  • Versioned plausibility rules: timestamp monotonicity, decode-rate envelopes by model size, text/chunk-length agreement, timer-quantization-grid conformance, environment cross-field consistency, software/virtual-renderer detection (a GPU-lane result from a SwiftShader/WARP-class adapter is rejected - cloud VMs report CPU numbers as GPU numbers), and a mandatory ~1 s deterministic matmul calibration check whose throughput must be plausible for the claimed rates.
  • Flagged runs are quarantined publicly (hidden from charts, never deleted). The entire dataset - verified and quarantined - is an open GitHub repository anyone can audit.
  • Residual limits, stated honestly: we cannot detect background native load, virtual machines, or browser flags; the min-3-submissions rule and median-of-medians limit their influence.

Reproduce it

The harness is MIT-licensed. Run the suite at /bench/run, export the raw JSON, or wire your own runtimes into @localmode/bench - the adapter interface takes any model exposing a streaming generation method. Aggregation and CSV tooling for analysis ship in the same package.