LocalMode Bench
One protocol, five browser AI runtimes, real devices.
- Measures TTFT, decode throughput, and cold vs warm model loads for LLMs and embeddings.
- Runs the same weights family across WebLLM, wllama, Transformers.js (WebGPU and WASM), LiteRT, and Chrome Built-in AI.
- Every submission publishes its raw timing trace to the open dataset; every number here is recomputed from those traces.
Leaderboard
81 verified submissions under localmode-bench/5 and localmode-bench/4 (rows never mix versions; the previous protocol stays beside the current one because only the llama.cpp lanes changed) · medians of per-device medians · devices grouped by GPU model where the browser names one, otherwise by platform and WebGPU architecture · rows need 3+ submissions to leave provisional status
| Protocol | Device class | Model | Runtime | Workload | TTFT | Decode chars/s | Embed | Load cold/warm | Quality | Runs |
|---|---|---|---|---|---|---|---|---|---|---|
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | embed-batch32 | — | — | 16/s | — / — | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | embed-single | — | — | 202 ms | — / 2770 ms | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | quality-sts-100 | — | — | — | — / — | 0.940 | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | embed-batch32 | — | — | 12/s | — / — | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | embed-single | — | — | 91 ms | — / 3262 ms | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | quality-sts-100 | — | — | — | — / — | 0.610 | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | embed-batch32 | — | — | 5/s | — / — | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | embed-single | — | — | 233 ms | — / 667 ms | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | quality-sts-100 | — | — | — | — / — | 0.930 | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | SmolLM2 135M (GGUF Q4_K_M) | wllama (wasm) | chat-pp128-tg128 | 3636 ms | 101 | — | — / 427 ms | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | SmolLM2 135M (GGUF Q4_K_M) | wllama (wasm) | quality-mmlu-25 | — | — | — | — / — | 0.280 (96% parsed) | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | Universal Sentence Encoder (MediaPipe) | mediapipe (wasm) | embed-batch32 | — | — | 33/s | — / — | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | Universal Sentence Encoder (MediaPipe) | mediapipe (wasm) | embed-single | — | — | 30 ms | 267 ms / — | — | 1provisional |
| v5 | android/adreno-650android/qualcomm-adreno-6xx | Universal Sentence Encoder (MediaPipe) | mediapipe (wasm) | quality-sts-100 | — | — | — | — / — | 0.780 | 1provisional |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | embed-batch32 | — | — | 20/s | — / — | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | embed-single | — | — | 55 ms | — / 990 ms | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WASM) | transformers-wasm (wasm) | quality-sts-100 | — | — | — | — / — | 0.940 | 2provisional |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | embed-batch32 | — | — | 56/s | — / — | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | embed-single | — | — | 26 ms | — / 1593 ms | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (ONNX, WebGPU) | transformers-webgpu (webgpu) | quality-sts-100 | — | — | — | — / — | 0.940 | 2provisional |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | embed-batch32 | — | — | 11/s | — / — | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | embed-single | — | — | 88 ms | — / 544 ms | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (GGUF Q8_0) | wllama (wasm) | quality-sts-100 | — | — | — | — / — | 0.930 | 2provisional |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (GGUF WebGPU, Q8_0) | wllama-webgpu (webgpu) | embed-batch32 | — | — | 14/s | — / — | — | 3 |
| v5 | android/adreno-830android/qualcomm-adreno-8xx | BGE Small EN (GGUF WebGPU, Q8_0) | wllama-webgpu (webgpu) | embed-single | — | — | 70 ms | — / 375 ms | — | 3 |
Recent submissions
What each device disclosed to the browser. The full capture (WebGPU adapter, WebGL, WebAssembly features, API availability, display, network) is in each run's JSON.
| Run | Suite | Device class | Device | GPU | Browser | OS | Cores | Memory | Storage quota | Isolation |
|---|---|---|---|---|---|---|---|---|---|---|
| ba74812d2026-09-24 | thorough | macos/apple-m4-maxmacos/apple-metal-3 | desktop | Apple M4 Maxmetal-3 | Google Chrome 153.0.8010.52Blink | macOS 26.3.0arm | 16 | 32 GB+ | 28 GB | isolatedWebGPU |
| 750881cd2026-09-24 | quick | linux/intel-xe-lpg | desktop | Mesa Intel(R) Arc(tm) Graphics (MTL)xe-lpg | Google Chrome 144.0.7559.132Blink | Linuxx86 | 22 | 8 GB+ | 142 GB | isolatedWebGPU |
| 3697ae322026-09-24 | quick | linux/intel-xe-lpg | desktop | Mesa Intel(R) Arc(tm) Graphics (MTL)xe-lpg | Google Chrome 144.0.7559.132Blink | Linuxx86 | 22 | 8 GB+ | 142 GB | isolatedWebGPU |
| cdc18adc2026-09-23 | quick | macos/apple-m4macos/apple-metal-3 | desktop | Apple M4metal-3 | Google Chrome 145.0.7632.159Blink | macOS 26.3.0arm | 10 | 8 GB+ | 30 GB | isolatedWebGPU |
| 76c3b5302026-09-23 | quick | macos/gpu | desktop | Apple M1, or similar | Firefox 156.0Gecko | macOS | 10 | — | 10 GB | isolatedWebGPU |
| 06b717632026-09-23 | standard | macos/apple-m4-maxmacos/apple-metal-3 | desktop | Apple M4 Maxmetal-3 | Google Chrome 153.0.8010.52Blink | macOS 26.3.0arm | 16 | 32 GB+ | 28 GB | isolatedWebGPU |
| 731c22202026-09-23 | quick | macos/apple-apple | desktop | Apple GPUapple | Safari 26.5.2WebKit | macOS | 8 | — | 77 GB | isolatedWebGPU |
| d93b074f2026-09-23 | quick | linux/no-webgpu | desktop | Intel(R) HD Graphics, or similar | Firefox 150.0Gecko | Linux | 22 | — | 10 GB | isolatedno WebGPU |
| df30afa62026-09-23 | quick | macos/apple-m4macos/apple-metal-3 | desktop | Apple M4metal-3 | Google Chrome 145.0.7632.159Blink | macOS 26.3.0arm | 10 | 8 GB+ | 31 GB | isolatedWebGPU |
| d4d211752026-09-22 | standard | macos/apple-m1-promacos/apple-metal-3 | desktop | Apple M1 Prometal-3 | Google Chrome 153.0.8010.53Blink | macOS 26.5.2arm | 10 | 32 GB+ | 21 GB | isolatedWebGPU |
| 074066952026-09-22 | quick | linux/intel-xe-lpg | desktop | Mesa Intel(R) Arc(tm) Graphics (MTL)xe-lpg | Google Chrome 144.0.7559.132Blink | Linuxx86 | 22 | 8 GB+ | 142 GB | isolatedWebGPU |
| 0905c4732026-09-22 | quick | android/adreno-830android/qualcomm-adreno-8xx | tabletSM-F966U1 | Adreno (TM) 830adreno-8xx | Google Chrome 153.0.8010.52Blink | Android 16.0.0 | 8 | 8 GB+ | 11 GB | isolatedWebGPU |
| 4d8a0ec92026-09-22 | quick | linux/intel-xe-lpg | desktop | Mesa Intel(R) Arc(tm) Graphics (MTL)xe-lpg | Google Chrome 144.0.7559.132Blink | Linuxx86 | 22 | 8 GB+ | 142 GB | isolatedWebGPU |
| 21df50b52026-09-22 | quick | android/adreno-830android/qualcomm-adreno-8xx | tabletSM-F966U1 | Adreno (TM) 830adreno-8xx | Google Chrome 153.0.8010.52Blink | Android 16.0.0 | 8 | 8 GB+ | 11 GB | isolatedWebGPU |
| 9d37091d2026-09-22 | thorough | macos/apple-m1-promacos/apple-metal-3 | desktop | Apple M1 Prometal-3 | Google Chrome 153.0.8010.53Blink | macOS 26.5.2arm | 10 | 32 GB+ | 21 GB | isolatedWebGPU |
| c802047b2026-09-22 | quick | android/adreno-650android/qualcomm-adreno-6xx | phoneSM-G986U1 | Adreno (TM) 650adreno-6xx | Google Chrome 152.0.7977.82Blink | Android 13.0.0 | 8 | 8 GB+ | 10 GB | isolatedWebGPU |
| fe73ef382026-09-22 | quick | android/adreno-830android/qualcomm-adreno-8xx | tabletSM-F966U1 | Adreno (TM) 830adreno-8xx | Google Chrome 153.0.8010.52Blink | Android 16.0.0 | 8 | 8 GB+ | 11 GB | isolatedWebGPU |
| 24057b0e2026-09-22 | quick | linux/intel-xe-lpg | desktop | Mesa Intel(R) Arc(tm) Graphics (MTL)xe-lpg | Google Chrome 144.0.7559.132Blink | Linuxx86 | 22 | 8 GB+ | 142 GB | isolatedWebGPU |
| 4baac9c82026-09-22 | quick | windows/amd-rdna-2 | desktop | AMD Radeon(TM) Graphicsrdna-2 | Google Chrome 153.0.8010.53Blink | Windows 19.0.0x86 | 16 | 32 GB+ | 26 GB | isolatedWebGPU |
| db5f73862026-09-22 | thorough | macos/apple-m4macos/apple-metal-3 | desktop | Apple M4metal-3 | Google Chrome 145.0.7632.159Blink | macOS 26.3.0arm | 10 | 8 GB+ | 53 GB | isolatedWebGPU |
| a66fedcd2026-09-22 | thorough | macos/apple-m1-promacos/apple-metal-3 | desktop | Apple M1 Prometal-3 | Google Chrome 153.0.8010.53Blink | macOS 26.5.2arm | 10 | 32 GB+ | 21 GB | isolatedWebGPU |
| cad218a52026-09-22 | thorough | macos/apple-m4macos/apple-metal-3 | desktop | Apple M4metal-3 | Google Chrome 145.0.7632.159Blink | macOS 26.3.0arm | 10 | 8 GB+ | 53 GB | isolatedWebGPU |
| c13abcb62026-09-22 | thorough | windows/amd-rdna-2 | desktop | AMD Radeon(TM) Graphicsrdna-2 | Google Chrome 153.0.8010.53Blink | Windows 19.0.0x86 | 16 | 32 GB+ | 26 GB | isolatedWebGPU |
| eafe01a62026-09-22 | thorough | macos/apple-m1-promacos/apple-metal-3 | desktop | Apple M1 Prometal-3 | Google Chrome 153.0.8010.53Blink | macOS 26.5.2arm | 10 | 32 GB+ | 21 GB | isolatedWebGPU |
| 80cb10f82026-09-22 | thorough | macos/apple-m4macos/apple-metal-3 | desktop | Apple M4metal-3 | Google Chrome 145.0.7632.159Blink | macOS 26.3.0arm | 10 | 8 GB+ | 52 GB | isolatedWebGPU |