LLM Inference & Local Models Repositories

Paddle-Lite

by PaddlePaddle

Total Stargazers:7275

Total Subscribers:330

PaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge (飞桨高性能深度学习端侧推理引擎）

llamacoder

by Nutlope

Total Stargazers:7128

Total Subscribers:95

Open source Claude Artifacts – built with Llama 3.1 405B

Dayflow

by JerryZLiu

Total Stargazers:7107

Total Subscribers:18

The automatic work journal/time tracker. Privately turns your screen into a timeline of what you actually accomplished. Open-source and local-first.

DeepSpec

by deepseek-ai

Total Stargazers:7107

Total Subscribers:33

DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms

vespa

by vespa-engine

Total Stargazers:7081

Total Subscribers:164

The AI search platform

GLM-4

by zai-org

Total Stargazers:7072

Total Subscribers:48

GLM-4 series: Open Multilingual Multimodal Chat LMs | 开源多语言多模态对话模型

gemma.cpp

by google

Total Stargazers:7036

Total Subscribers:48

lightweight, standalone C++ inference engine for Google's Gemma models.

turboquant_plus

by TheTom

Total Stargazers:7021

Total Subscribers:65

______________

opensquilla

by TokenRhythm

Total Stargazers:7013

Total Subscribers:137

OpenSquilla — Token-Efficient AI Agent with same budget, higher intelligence density

mlx-lm

by ml-explore

Total Stargazers:6988

Total Subscribers:51

Run LLMs with MLX

stable-diffusion.cpp

by leejet

Total Stargazers:6969

Total Subscribers:78

Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++

clearml

by clearml

Total Stargazers:6863

Total Subscribers:90

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

DeepSpeedExamples

by deepspeedai

Total Stargazers:6846

Total Subscribers:72

Example models using DeepSpeed

models.dev

by anomalyco

Total Stargazers:6831

Total Subscribers:24

An open-source database of AI models.

CapsWriter-Offline

by HaujetZhao

Total Stargazers:6790

Total Subscribers:35

PC 端语音输入工具，离线识别，高准确率、低延迟，支持热词、LLM润色。按住CapsLock或鼠标侧键X2说话，松开自动上屏。

vllm-omni

by vllm-project

Total Stargazers:6777

Total Subscribers:56

A framework for efficient model inference with omni-modality models

turbo-fieldfare

by drumih

Total Stargazers:6718

Total Subscribers:38

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

gpt-load

by tbphp

Total Stargazers:6685

Total Subscribers:18

Self-hosted AI gateway for multi-channel, multi-credential setups — API keys and subscription accounts, scheduling, failover, request logs and usage. 自托管 AI 网关：多渠道多凭据统一接入，含密钥与订阅账号、调度容错、日志与用量。

LLMLingua

by microsoft

Total Stargazers:6650

Total Subscribers:40

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

whichllm

by Andyyyy64

Total Stargazers:6624

Total Subscribers:25

Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.

Mooncake

by kvcache-ai

Total Stargazers:6563

Total Subscribers:48

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

RWKV-Runner

by josStorer

Total Stargazers:6475

Total Subscribers:48

A RWKV management and startup tool, full automation, only 8MB. And provides an interface compatible with the OpenAI API. RWKV is a large language model that is fully open source and available for commercial use.

genkit

by genkit-ai

Total Stargazers:6432

Total Subscribers:60

Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google

LiteRT-LM

by google-ai-edge

Total Stargazers:6431

Total Subscribers:75

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

ODS

by Osmantic

Total Stargazers:6409

Total Subscribers:39

Turn your PC, Mac, or Linux box into an AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.

flashinfer

by flashinfer-ai

Total Stargazers:6387

Total Subscribers:54

FlashInfer: Kernel Library for LLM Serving

apfel

by Arthur-Ficial

Total Stargazers:6374

Total Subscribers:18

The free AI already on your Mac. CLI tool, OpenAI-compatible server, and interactive chat — all on-device via Apple Intelligence. No API keys, no cloud, no downloads.

argmax-oss-swift

by argmaxinc

Total Stargazers:6364

Total Subscribers:46

On-device Speech AI for Apple Silicon

serving

by tensorflow

Total Stargazers:6362

Total Subscribers:217

A flexible, high-performance serving system for machine learning models

openplayground

by nat

Total Stargazers:6349

Total Subscribers:57

An LLM playground you can run on your laptop

Orpheus-TTS

by canopyai

Total Stargazers:6335

Total Subscribers:73

Towards Human-Sounding Speech

neutts

by neuphonic

Total Stargazers:6278

Total Subscribers:58

On-device TTS model by Neuphonic

chatgpt2api

by basketikun

Total Stargazers:6274

Total Subscribers:19

ChatGPT官网接口纯协议的逆向实现，支持GPT-Image-2模型、文本模型，兼容OpenAI接口协议，在线批量生图/编辑图，号池管理，支持可编辑PPT/PSD文件逆向，支持导入CPA、sub2api号池、支持接入Cherry Studio、New Api 等软件

Soup

by MakazhanAlpamys

Total Stargazers:6241

Total Subscribers:47

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

dflash

by z-lab

Total Stargazers:6076

Total Subscribers:45

DFlash: Block Diffusion for Flash Speculative Decoding

cactus

by cactus-compute

Total Stargazers:6008

Total Subscribers:47

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

enchanted

by gluonfield

Total Stargazers:5997

Total Subscribers:44

Enchanted is iOS and macOS app for chatting with private self hosted language models such as Llama2, Mistral or Vicuna using Ollama.

kserve

by kserve

Total Stargazers:5884

Total Subscribers:70

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

shimmy

by Michael-A-Kuykendall

Total Stargazers:5867

Total Subscribers:41

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

gemini-balance

by snailyp

Total Stargazers:5816

Total Subscribers:20

Gemini polling proxy service （gemini轮询代理服务）

AidLearning-FrameWork

by aidlearning

Total Stargazers:5800

Total Subscribers:217

🔥🔥🔥AidLearning is a powerful AIOT development platform, AidLearning builds a linux env supporting GUI, deep learning and visual IDE on Android...Now Aid supports CPU+GPU+NPU for inference with high performance acceleration...Linux on Android or HarmonyOS

semantic-router

by vllm-project

Total Stargazers:5760

Total Subscribers:62

A programmable Mixture-of-Models router for heterogeneous LLM inference

gemma_pytorch

by google

Total Stargazers:5718

Total Subscribers:42

The official PyTorch implementation of Google's Gemma models

lemonade

by lemonade-sdk

Total Stargazers:5700

Total Subscribers:37

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

gpustack

by gpustack

Total Stargazers:5677

Total Subscribers:45

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

llama-swap

by mostlygeek

Total Stargazers:5654

Total Subscribers:24

Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc

granian

by emmett-framework

Total Stargazers:5635

Total Subscribers:42

A Rust HTTP server for Python applications

rlm

by alexzhang13

Total Stargazers:5614

Total Subscribers:55

General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.

koharu

by koharu-rs

Total Stargazers:5549

Total Subscribers:16

AI-powered manga translator, written in Rust.

piper1-gpl

by OHF-Voice

Total Stargazers:5547

Total Subscribers:50

Fast and local neural text-to-speech engine

MineContext

by volcengine

Total Stargazers:5509

Total Subscribers:23

MineContext is your proactive context-aware AI partner（Context-Engineering+ChatGPT Pulse）

Awesome-LLM-Inference

by xlite-dev

Total Stargazers:5496

Total Subscribers:135

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

mlx-vlm

by Blaizzy

Total Stargazers:5491

Total Subscribers:33

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

Kokoro-FastAPI

by remsky

Total Stargazers:5434

Total Subscribers:36

Dockerized OpenAI-compatible wrapper for Kokoro-82M text-to-speech w/multiplatform CPU, AMD, NVIDIA GPU PyTorch; multi-speaker, clone-tuning, caption timestamps, SSML, optional readalong web UI

coremltools

by apple

Total Stargazers:5418

Total Subscribers:121

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.

genai-stack

by docker

Total Stargazers:5403

Total Subscribers:77

Langchain + Docker + Neo4j + Ollama

deepreasoning

by winfunc

Total Stargazers:5361

Total Subscribers:45

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.

openmed

by maziyarpanahi

Total Stargazers:5310

Total Subscribers:23

Local-first healthcare AI: clinical NER & HIPAA PII de-identification that runs 100% on-device. 2,200+ medical models, 21 languages, Apple MLX + Python, no cloud, no patient data leaving your network. Apache-2.0

lms

by lmstudio-ai

Total Stargazers:5287

Total Subscribers:39

LLM Inference & Local Models

Inference engines, model serving, quantization and running LLMs locally.

Paddle-Lite

by PaddlePaddle

PaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge (飞桨高性能深度学习端侧推理引擎）

llamacoder

by Nutlope

Open source Claude Artifacts – built with Llama 3.1 405B

Dayflow

by JerryZLiu

The automatic work journal/time tracker. Privately turns your screen into a timeline of what you actually accomplished. Open-source and local-first.

DeepSpec

by deepseek-ai

DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms

vespa

by vespa-engine

The AI search platform

GLM-4

by zai-org

GLM-4 series: Open Multilingual Multimodal Chat LMs | 开源多语言多模态对话模型

gemma.cpp

by google

lightweight, standalone C++ inference engine for Google's Gemma models.

turboquant_plus

by TheTom

______________

opensquilla

by TokenRhythm

OpenSquilla — Token-Efficient AI Agent with same budget, higher intelligence density

mlx-lm

by ml-explore

Run LLMs with MLX

stable-diffusion.cpp

by leejet

Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,...) inference in pure C/C++

clearml

by clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

DeepSpeedExamples

by deepspeedai

Example models using DeepSpeed

models.dev

by anomalyco

An open-source database of AI models.

CapsWriter-Offline

by HaujetZhao

PC 端语音输入工具，离线识别，高准确率、低延迟，支持热词、LLM润色。按住CapsLock或鼠标侧键X2说话，松开自动上屏。

vllm-omni

by vllm-project

A framework for efficient model inference with omni-modality models

turbo-fieldfare

by drumih

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

gpt-load

by tbphp

Self-hosted AI gateway for multi-channel, multi-credential setups — API keys and subscription accounts, scheduling, failover, request logs and usage. 自托管 AI 网关：多渠道多凭据统一接入，含密钥与订阅账号、调度容错、日志与用量。

LLMLingua

by microsoft

[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

whichllm

by Andyyyy64

Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.

Mooncake

by kvcache-ai

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

RWKV-Runner

by josStorer

A RWKV management and startup tool, full automation, only 8MB. And provides an interface compatible with the OpenAI API. RWKV is a large language model that is fully open source and available for commercial use.

genkit

by genkit-ai

Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google

LiteRT-LM

by google-ai-edge

LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.

ODS

by Osmantic

Turn your PC, Mac, or Linux box into an AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.

flashinfer

by flashinfer-ai

FlashInfer: Kernel Library for LLM Serving

ChatGPT官网接口纯协议的逆向实现，支持GPT-Image-2模型、文本模型，兼容OpenAI接口协议，在线批量生图/编辑图，号池管理，支持可编辑PPT/PSD文件逆向，支持导入CPA、sub2api号池、支持接入Cherry Studio、New Api 等软件