LLM Inference & Local Models Repositories

WhisperLiveKit

by QuentinFuxa

Total Stargazers:11028

Total Subscribers:61

Real-time, local speech-to-text with streaming ASR, speaker diarization, translation, and OpenAI/Deepgram-compatible APIs.

server

by triton-inference-server

Total Stargazers:10983

Total Subscribers:146

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

llama-gpt

by getumbrel

Total Stargazers:10938

Total Subscribers:77

A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!

needle

by cactus-compute

Total Stargazers:10915

Total Subscribers:63

14MB foundation model for tiny devices; phones, wearables, smart home, and robots.

MiniCPM

by OpenBMB

Total Stargazers:10885

Total Subscribers:86

MiniCPM5: SOTA on-device LLMs, small yet powerful.

openvino

by openvinotoolkit

Total Stargazers:10838

Total Subscribers:192

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

mistral-inference

by mistralai

Total Stargazers:10828

Total Subscribers:123

Official inference library for Mistral models

WasmEdge

by WasmEdge

Total Stargazers:10803

Total Subscribers:96

WasmEdge is a lightweight, high-performance, and extensible WebAssembly runtime for cloud native, edge, and decentralized applications. It powers serverless apps, embedded functions, microservices, smart contracts, and IoT devices.

go-openai

by sashabaranov

Total Stargazers:10763

Total Subscribers:70

OpenAI, GPT 5.6, GPT-Image-2, Whisper API clients for Go

automq

by AutoMQ

Total Stargazers:10712

Total Subscribers:64

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

Whisper

by Const-me

Total Stargazers:10659

Total Subscribers:111

High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model

llama-cpp-python

by abetlen

Total Stargazers:10615

Total Subscribers:85

Python bindings for llama.cpp

ollama-python

by ollama

Total Stargazers:10518

Total Subscribers:81

Ollama Python library

RTranslator

by niedev

Total Stargazers:10400

Total Subscribers:82

Open source real-time translation app for Android that runs locally

runanywhere-sdks

by RunanywhereAI

Total Stargazers:10281

Total Subscribers:29

Production ready toolkit to run AI locally

models

by onnx

Total Stargazers:9793

Total Subscribers:204

A collection of pre-trained, state-of-the-art models in the ONNX format

PowerInfer

by Tiiny-AI

Total Stargazers:9790

Total Subscribers:108

High-speed Large Language Model Serving for Local Deployment

OpenJarvis

by open-jarvis

Total Stargazers:9647

Total Subscribers:148

Personal AI, On Personal Devices

inference

by xorbitsai

Total Stargazers:9568

Total Subscribers:64

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

feynman

by advaitpaliwal

Total Stargazers:9511

Total Subscribers:35

The open source AI research agent.

ChatRWKV

by BlinkDL

Total Stargazers:9497

Total Subscribers:96

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

pocket-tts

by kyutai-labs

Total Stargazers:9483

Total Subscribers:70

A TTS that fits in your CPU (and pocket)

oumi

by oumi-ai

Total Stargazers:9382

Total Subscribers:67

Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!

higress

by higress-group

Total Stargazers:9368

Total Subscribers:75

🤖 AI Gateway | AI Native API Gateway

Horizon

by Thysrael

Total Stargazers:9331

Total Subscribers:30

📡 Your own AI-powered news radar. Generates daily briefings in English & Chinese. | 用 AI 构建你专属的新闻雷达

coai

by coaidev

Total Stargazers:9310

Total Subscribers:70

🚀 Next Gen Multi-tenant AI One-Stop Solution. Builtin Admin & Billing System. Enterprise-Grade Unified LLM Gateway Support for 200+ Models And 35+ Providers, Load Balacing w/ Priority-base Routing, Cost Management, Chat Share, Cloud Sync, Credit/Subscription Billing, All File Parsing, Web Search, Built-in Model Cache.

local-deep-research

by LearningCircuit

Total Stargazers:9079

Total Subscribers:43

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

astron-agent

by iflytek

Total Stargazers:8994

Total Subscribers:81

Enterprise-grade, commercial-friendly agentic workflow platform for building next-generation SuperAgents.

jetson-inference

by dusty-nv

Total Stargazers:8991

Total Subscribers:283

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.

nsfwjs

by infinitered

Total Stargazers:8985

Total Subscribers:92

NSFW detection on the client-side via TensorFlow.js

BentoML

by bentoml

Total Stargazers:8837

Total Subscribers:85

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

llrt

by awslabs

Total Stargazers:8803

Total Subscribers:48

LLRT (Low Latency Runtime) is an experimental, lightweight JavaScript runtime designed to address the growing demand for fast and efficient Serverless applications.

AIClient2API

by justlovemaki

Total Stargazers:8776

Total Subscribers:38

Self-hosted multi-protocol AI API proxy for Antigravity, Codex, Grok, Kiro, OpenAI, Claude, and custom providers. Supports OpenAI-compatible API, Claude API, Gemini protocol conversion, GPT, Grok Build, Claude Opus, Gemini Pro, Kimi, MiniMax, provider pools, smart routing, and automatic failover.

adk-go

by google

Total Stargazers:8771

Total Subscribers:103

An open-source, code-first Go toolkit for building, evaluating, and deploying sophisticated AI agents with flexibility and control.

LMFlow

by OptimalScale

Total Stargazers:8488

Total Subscribers:66

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

bitsandbytes

by bitsandbytes-foundation

Total Stargazers:8475

Total Subscribers:53

Accessible large language models via k-bit quantization for PyTorch.

ogx

by ogx-ai

Total Stargazers:8428

Total Subscribers:117

Open GenAI Stack

GenieX

by qualcomm

Total Stargazers:8365

Total Subscribers:531

Run frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of code

optimate

by nebuly-ai

Total Stargazers:8328

Total Subscribers:88

A collection of libraries to optimise AI model performances

pocketpal-ai

by a-ghorbani

Total Stargazers:8267

Total Subscribers:69

An app that brings language models directly to your phone.

imaginAIry

by brycedrennan

Total Stargazers:8185

Total Subscribers:49

Pythonic AI generation of images and videos

nullclaw

by nullclaw

Total Stargazers:8080

Total Subscribers:59

Fastest, smallest, and fully autonomous AI assistant infrastructure written in Zig

lmdeploy

by InternLM

Total Stargazers:8061

Total Subscribers:57

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

dynamo

by ai-dynamo

Total Stargazers:8044

Total Subscribers:77

A Datacenter Scale Distributed Inference Serving Framework

bifrost

by maximhq

Total Stargazers:8001

Total Subscribers:32

Fastest enterprise AI gateway (50x faster than LiteLLM) with adaptive load balancer, cluster mode, guardrails, 1000+ models support & <100 µs overhead at 5k RPS.

osaurus

by osaurus-ai

Total Stargazers:7904

Total Subscribers:35

Own your AI. The native macOS harness for AI agents -- any model, persistent memory, autonomous execution, cryptographic identity. Built in Swift. Fully offline. Open source.

tensorrtx

by wang-xinyu

Total Stargazers:7833

Total Subscribers:103

Implementation of popular deep learning networks with TensorRT network definition API

DeepGEMM

by deepseek-ai

Total Stargazers:7819

Total Subscribers:68

DeepGEMM: clean and efficient BLAS kernel library on GPU

RapidOCR

by RapidAI

Total Stargazers:7805

Total Subscribers:62

📄 Awesome OCR multiple programing languages toolkits based on ONNX Runtime, OpenVINO, MNN, PaddlePaddle, TensorRT and PyTorch.

kimi-k3-in-c

by FareedKhan-dev

Total Stargazers:7778

Total Subscribers:68

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

llama-models

by meta-llama

Total Stargazers:7690

Total Subscribers:111

Utilities intended for use with Llama models.

mistral.rs

by EricLBuehler

Total Stargazers:7677

Total Subscribers:45

Fast, flexible LLM inference

grok2api

by chenyme

Total Stargazers:7643

Total Subscribers:19

Multi-account API gateway for Grok Build, Grok Web, and Grok Console

awesome-free-llm-apis

by mnfst

Total Stargazers:7554

Total Subscribers:99

List of Permanent Free LLM API (API Keys)

llm-gateway

by mnfst

Total Stargazers:7521

Total Subscribers:29

Connect Your Agents And Harnesses With Any Provider 🦚

modly

by lightningpixel

Total Stargazers:7519

Total Subscribers:54

Desktop app to generate 3D models from images or prompt using local AI — runs entirely on your GPU

go-stock

by ArvinLovegood

Total Stargazers:7515

Total Subscribers:46

🦄🦄🦄AI赋能股票分析：AI加持的股票分析/选股工具。股票行情获取，AI热点资讯分析，AI资金/财务分析，涨跌报警推送。支持A股，港股，美股。支持市场整体/个股情绪分析，AI辅助选股等。数据全部保留在本地。支持DeepSeek，OpenAI， Ollama，LMStudio，AnythingLLM，硅基流动，火山方舟，阿里云百炼等平台或模型。

ml-fastvlm

by apple-aiml-research

Total Stargazers:7416

Total Subscribers:67

This repository contains the official implementation of "FastVLM: Efficient Vision Encoding for Vision Language Models" - CVPR 2025

sdnext

by vladmandic

Total Stargazers:7340

Total Subscribers:65

LLM Inference & Local Models

Inference engines, model serving, quantization and running LLMs locally.

WhisperLiveKit

by QuentinFuxa

Real-time, local speech-to-text with streaming ASR, speaker diarization, translation, and OpenAI/Deepgram-compatible APIs.

server

by triton-inference-server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

llama-gpt

by getumbrel

A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!

needle

by cactus-compute

14MB foundation model for tiny devices; phones, wearables, smart home, and robots.

MiniCPM

by OpenBMB

MiniCPM5: SOTA on-device LLMs, small yet powerful.

openvino

by openvinotoolkit

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

mistral-inference

by mistralai

Official inference library for Mistral models

WasmEdge

by WasmEdge

WasmEdge is a lightweight, high-performance, and extensible WebAssembly runtime for cloud native, edge, and decentralized applications. It powers serverless apps, embedded functions, microservices, smart contracts, and IoT devices.

go-openai

by sashabaranov

OpenAI, GPT 5.6, GPT-Image-2, Whisper API clients for Go

automq

by AutoMQ

Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.

Whisper

by Const-me

High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model

llama-cpp-python

by abetlen

Python bindings for llama.cpp

ollama-python

by ollama

Ollama Python library

RTranslator

by niedev

Open source real-time translation app for Android that runs locally

runanywhere-sdks

by RunanywhereAI

Production ready toolkit to run AI locally

models

by onnx

A collection of pre-trained, state-of-the-art models in the ONNX format

PowerInfer

by Tiiny-AI

High-speed Large Language Model Serving for Local Deployment

OpenJarvis

by open-jarvis

Personal AI, On Personal Devices

inference

by xorbitsai

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

feynman

by advaitpaliwal

The open source AI research agent.

ChatRWKV

by BlinkDL

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

pocket-tts

by kyutai-labs

A TTS that fits in your CPU (and pocket)

oumi

by oumi-ai

Easily fine-tune, evaluate and deploy Qwen, Gemma, or any open weight LLM!

higress

by higress-group

🤖 AI Gateway | AI Native API Gateway

Horizon

by Thysrael

📡 Your own AI-powered news radar. Generates daily briefings in English & Chinese. | 用 AI 构建你专属的新闻雷达

coai

by coaidev

local-deep-research