Running LLMs in the Browser: WebLLM…

LLMs No Longer Need Servers

GPT-4 requires cloud APIs? That's 2024 thinking. By 2026, 7B-parameter models run smoothly in the browser, and your data never leaves your device.

Privacy + Zero Cost + Offline = The three killer advantages of browser-side AI

Browser AI Evolution Timeline

2023 Q4    WebLLM launches, Llama 2 at 2 tok/s in browser
2024 Q2    Transformers.js released, Hugging Face ecosystem connected
2024 Q4    WebGPU full support, 5x inference speedup
2025 Q2    ONNX Runtime Web supports WebGPU, enterprise-grade
2025 Q4    Quantization breakthrough, 7B models compressed to 3GB
2026 Q2    Gemma 3 4B hits 25 tok/s in browser — quality inflection

Three Frameworks at a Glance

Framework	Focus	Core Tech	Best For
WebLLM	High-perf LLM inference	WebGPU + MLCEngine	Chat, code completion
Transformers.js	Full-stack ML inference	ONNX + WASM/WebGPU	NLP, vision, audio
ONNX Runtime Web	Enterprise inference engine	ONNX + WebGPU/WASM	Production deployment

WebLLM — The Performance King of Browser LLMs

Architecture

┌──────────────────────────────────────────────────┐
│                  Application Layer                │
│   Chat UI │ Code Completion │ Summarization │ ...  │
├──────────────────────────────────────────────────┤
│              WebLLM Engine                        │
│   ChatModule │ Pipeline │ Tokenizer │ Scheduler   │
├──────────────────────────────────────────────────┤
│              MLCEngine (Compilation Optimization) │
│   Model Compile │ Kernel Opt │ Quantization       │
├──────────────────────────────────────────────────┤
│              WebGPU Runtime                       │
│   Compute Shader │ GPU Buffer │ Pipeline State    │
└──────────────────────────────────────────────────┘

Quick Start

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("gemma-3-4b-it-q4f16_1-MLC", {
  initProgressCallback: (progress) => {
    console.log(`Loading: ${(progress.progress * 100).toFixed(1)}%`);
  },
});

const reply = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful AI assistant." },
    { role: "user", content: "Write a quicksort in TypeScript" },
  ],
  temperature: 0.7,
  max_tokens: 1024,
  stream: true,
});

for await (const chunk of reply) {
  process.stdout.write(chunk.choices[0]?.delta?.content || "");
}

Supported Models & Performance (June 2026)

Model	Params	Quantized Size	Speed (tok/s)	First Load
Gemma 3 4B IT	4B	2.3GB	25	8s
Phi-4 Mini	3.8B	2.1GB	28	7s
Qwen2.5 3B	3B	1.8GB	32	6s
SmolLM2 1.7B	1.7B	1.0GB	45	4s
Qwen2.5 0.5B	0.5B	0.4GB	85	2s

Test environment: M3 MacBook Pro / Chrome 126 / WebGPU

Transformers.js — Hugging Face Ecosystem in the Browser

Core Advantage: Richest Model Ecosystem

import { pipeline } from "@xenova/transformers";

const classifier = await pipeline("text-classification", "Xenova/distilbert-base-uncased-finetuned-sst-2-english");
const result = await classifier("This browser AI is amazing!");
// [{ label: "POSITIVE", score: 0.9998 }]

Supported Pipelines

Pipeline	Use Case	Example Model
text-classification	Sentiment analysis	distilbert-sst2
question-answering	Q&A system	distilbert-qa
summarization	Summarization	distilbart-cnn
translation	Translation	opus-mt-en-zh
image-classification	Image classification	vit-base-patch16
automatic-speech-recognition	Speech recognition	whisper-tiny

ONNX Runtime Web — Enterprise Inference Engine

Basic Usage

import ort from "onnxruntime-web";

async function runInference(modelPath: string, input: Float32Array) {
  const session = await ort.InferenceSession.create(modelPath, {
    executionProviders: ["webgpu", "wasm"],
    graphOptimizationLevel: "all",
  });

  const inputTensor = new ort.Tensor("float32", input, [1, input.length]);
  const results = await session.run({ input: inputTensor });
  return results.output.data;
}

Provider Selection Strategy

function getBestProvider(): string {
  if (navigator.gpu) return "webgpu";
  if (document.createElement("canvas").getContext("webgl2")) return "webgl";
  return "wasm";
}

Performance Benchmarks: Full Comparison

LLM Inference (Gemma 3 4B, 4-bit quantization)

Metric	WebLLM	Transformers.js	ONNX Runtime Web
Speed (tok/s)	25	18	22
First Load	8s	12s	10s
Memory	3.2GB	3.8GB	3.5GB
Streaming	✅	✅	❌
OpenAI API Compat	✅	❌	❌

Production Deployment Strategies

Graceful Degradation

async function createEngineWithFallback() {
  if (navigator.gpu) {
    try { return await CreateMLCEngine("gemma-3-4b-it-q4f16_1-MLC"); }
    catch (e) { console.warn("WebGPU failed, falling back to smaller model"); }
  }
  try { return await CreateMLCEngine("SmolLM2-1.7B-q4f16_1-MLC"); }
  catch (e) { console.warn("Small model failed, falling back to API"); }
  return new APIFallbackEngine({ endpoint: "/api/chat" });
}

Service Worker Offline Support

const MODEL_CACHE = "ai-models-v1";
self.addEventListener("install", (event) => {
  event.waitUntil(
    caches.open(MODEL_CACHE).then((cache) =>
      cache.addAll(["/models/smolm2-1.7b-q4.onnx", "/models/tokenizer.json"])
    )
  );
});

Decision Matrix

Your need?
├─ Chat / Conversational AI → ✅ WebLLM
├─ Multimodal AI → ✅ Transformers.js
├─ Enterprise Production → ✅ ONNX Runtime Web
├─ Offline-First App → ✅ WebLLM + Service Worker
└─ Quick Prototype → ✅ Transformers.js

2026 H2 Trends

Trend	Description
WebGPU Universal	Safari 18+ support, no more compat issues
MoE in Browser	Mixtral sparse models run in browser
On-Device Fine-tuning	LoRA weights downloaded, personalized models without server
W3C Web AI API	Browser-native AI capabilities standardized

Summary

Browser AI has gone from "toy" to "tool" — 4B model at 25 tok/s is usable
Three frameworks, three strengths — WebLLM fast, Transformers.js complete, ONNX stable
Privacy is the killer feature — Data never leaves the browser, zero server cost
Degradation strategy is key — WebGPU → small model → cloud API, guaranteed availability

If your AI product still sends every request to the cloud in 2026, you're not just wasting server costs — you're missing privacy protection, the biggest differentiator.