ESP32 iPhone Agent is a tool that enables an AI model to control a real iPhone using a low-cost ESP32 microcontroller board as the interface hardware.
The tool solves the problem of giving AI agents direct control over an iPhone without requiring a Mac, developer account, or jailbreak. It works by having the ESP32 board act as both an AirPlay receiver to capture the phone's screen and a USB HID device to send touch and keyboard inputs. The board creates its own Wi-Fi hotspot that the iPhone connects to for screen mirroring, while the phone maintains its cellular connection for communicating with the vision model. An app running on the iPhone itself decodes the video stream, sends screenshots to any OpenAI-compatible vision model, and executes the model's decisions through taps, swipes, typing, and key presses. The system includes a feedback loop where the app tracks pointer position, corrects for drift before precise actions, and waits for the screen to settle after each action to determine what changed.
This approach suits developers and researchers who need to automate iPhone interactions for testing, demonstration, or agent development without the overhead of a separate computer or the complexity of jailbreaking. The tool is designed for minimal hardware requirements—just the phone and a seven-dollar board—making it accessible for prototyping and experimentation. The system preserves every run on the phone with step-by-step replay showing taps and swipes, and allows export of screenshots, actions, and annotations for analysis by other agents.
The project maintains active development with recent architectural changes that eliminated the need for an external computer entirely. The codebase is written primarily in C for the board firmware. The maintainers provide clear warnings against running installation commands from unofficial sources and maintain documentation of the legacy version for users who may need the earlier computer-based approach.
