JARVIS is a Microsoft research project designed to connect large language models with the broader machine learning community by enabling LLMs to orchestrate and leverage numerous specialized AI models. The system treats natural language as an interface through which an LLM controller can coordinate expert models from sources like Hugging Face Hub to solve complex AI tasks that require multiple specialized capabilities.
The core architecture operates through four sequential stages. Task Planning uses ChatGPT to analyze user requests and decompose them into solvable subtasks. Model Selection leverages ChatGPT to identify appropriate expert models from Hugging Face based on their descriptions and capabilities. Task Execution invokes the selected models and collects their results. Response Generation integrates predictions from all executed models and synthesizes a final response using ChatGPT. This four-stage workflow enables the system to handle complicated AI problems by combining the reasoning capabilities of large language models with the specialized expertise of domain-specific models.
The repository supports multiple deployment configurations to accommodate different hardware constraints and use cases. The default configuration requires substantial resources including at least 24GB of VRAM, 16GB of RAM for standard deployment, and over 284GB of disk space to host various models locally. A lightweight configuration eliminates local model deployment requirements entirely, instead relying on Hugging Face Inference Endpoints for remote model access. A hybrid mode balances these approaches by supporting both local and remote inference endpoints. Users can configure the system for minimal, standard, or full local deployment scales depending on available memory.
JARVIS provides multiple interfaces for interaction including a server mode with Web API endpoints for task planning and model selection stages, a web interface built with Node.js and npm, a Gradio demo hosted on Hugging Face Spaces, and a command-line interface for lightweight usage. The system supports OpenAI's text-davinci-003 model and GPT-4, with Azure platform integration available. The repository includes experimental Docker support for NVIDIA Jetson embedded devices, specifically targeting Jetson AGX Orin family devices with 64GB of RAM.
The project has evolved significantly since its initial release. Recent additions include EasyTool released in January 2024 for simplified tool usage in LLM-based agents, and TaskBench released in November 2023 for evaluating task automation capabilities of large language models. A lightweight Langchain version was released in July 2023. The system now supports Azure OpenAI services and GPT-4 models as of April 2023.
The project is classified across multiple domains including chatbots, automation, question answering, knowledge representation, dialogue systems, semantic search, information retrieval, text generation, natural language processing, and multi-modal interactions, reflecting its broad applicability to conversational AI and task automation scenarios.
