ESP32 iPhone Agent

ESP32 iPhone Agent is an open-source loop that pairs AirPlay screen mirroring with a tiny ESP32-S3 board plugged into the phone as a USB mouse and keyboard. A planner and actor model (any OpenAI-compatible vision API, cloud or local) watch JPEG frames from UxPlay, then send move/click/type commands over Wi-Fi to the board. AssistiveTouch turns the HID pointer into taps and swipes. Nothing is installed on the iPhone; Linux or macOS can host the server.

Mobile agent demos still die on Apple's test stack: developer accounts, signed runners, and Mac-only toolchains. Founders who want agents to tap real apps, games, and Clock settings need the same path a person uses — the screen and a pointer — without installing anything on the phone.

Flash the ESP32-S3 with the HID bridge firmware, install UxPlay plus the Python server, and put your model base URL and key in server/.env. On the phone turn on AssistiveTouch, plug in the board, mirror to iPhone Agent, and run calibrate so the server finds the pointer on the live screen. Open the web UI on port 8890, give a goal like set a repeating alarm or draw on Whiteboard, and the agent plans steps, acts, checks that the screen changed, and re-plans on misses.

XCTest and WebDriverAgent need a Mac, Xcode, and a signed helper that expires on free accounts. This stack skips the accessibility tree and drives pixels plus HID like a person — any app including games, any OpenAI-compatible vision model, and hardware that costs about seven dollars. Complementary to desktop Chrome relays: those own authenticated web; this owns the physical phone.

• Public demo available

• Chrome Relay ↗
• chrome-devtools-mcp ↗