mlc-ai/web-llm

WebLLM is a high-performance in-browser LLM inference engine that runs language model inference directly in web browsers with WebGPU hardware acceleration.

WebLLM solves the problem of deploying language models without server infrastructure by executing inference entirely within the browser. It achieves this through WebGPU acceleration and is built as a companion to MLC LLM, which handles universal LLM deployment across hardware environments. The engine provides full compatibility with the OpenAI API, allowing developers to run open-source models locally with features including streaming, JSON-mode structured generation, and logit-level control.

Developers should choose WebLLM when building privacy-focused applications that benefit from client-side inference without server dependencies. It suits web applications requiring real-time interactions such as chatbots and virtual assistants, and supports a broad range of models including Llama 3, Phi 3, Gemma, Mistral, and Qwen. The tool integrates via npm, yarn, or CDN, and can offload computations to web workers or service workers to optimize UI performance. Custom models in MLC format can be integrated for specific use cases.

The project maintains a substantial base of adopters who report real-world issues, with almost all open issues raised by outside users rather than the core team. Responses to issues and pull requests typically arrive within one to two weeks. Bug reports dominate the issue tracker, reflecting active maintenance focused on stability and correctness.