Hugging Face's logo
Jacob's picture

Jacob

ljunb
·

AI & ML interests

None yet

Recent Activity

repliedto NatalieY's post 2 days ago
Aiden: a physical AI agent that controls phones over USB HID Most GUI agent work assumes the agent lives inside the device or drives it through a debugging interface. We went the other way. Aiden is a small board that sits outside the host. It captures the screen over HDMI-to-CSI, runs the agent loop on-device, and sends actions back as a standard USB HID device — the host sees a keyboard and a mouse, nothing else. No app install, no root, no ADB, no cloud. Runtime is Go. Frame capture, full-duplex audio with VAD, the agent loop, and HID output all run as independent goroutines. There's no backend — nothing leaves the device, which is the only defensible design when the input is a live feed of someone's phone screen. Open questions we haven't solved: · Action verification — inferring success from a re-read of the screen breaks when loading states lie · Prompt injection — an agent that reads screens reads whatever an attacker puts on them · iOS pointer control requires AssistiveTouch Repo, including the HID gadget config and capture pipeline: github.com/AidenAI-IO/aiden-hardware-demo Wrote up how this differs from cloud-based computer use agents here: https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/ Note: current hardware is a dev board, not a finished product.
reacted to NatalieY's post with 👀 2 days ago
Aiden: a physical AI agent that controls phones over USB HID Most GUI agent work assumes the agent lives inside the device or drives it through a debugging interface. We went the other way. Aiden is a small board that sits outside the host. It captures the screen over HDMI-to-CSI, runs the agent loop on-device, and sends actions back as a standard USB HID device — the host sees a keyboard and a mouse, nothing else. No app install, no root, no ADB, no cloud. Runtime is Go. Frame capture, full-duplex audio with VAD, the agent loop, and HID output all run as independent goroutines. There's no backend — nothing leaves the device, which is the only defensible design when the input is a live feed of someone's phone screen. Open questions we haven't solved: · Action verification — inferring success from a re-read of the screen breaks when loading states lie · Prompt injection — an agent that reads screens reads whatever an attacker puts on them · iOS pointer control requires AssistiveTouch Repo, including the HID gadget config and capture pipeline: github.com/AidenAI-IO/aiden-hardware-demo Wrote up how this differs from cloud-based computer use agents here: https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/ Note: current hardware is a dev board, not a finished product.
reacted to NatalieY's post with 👍 2 days ago
Aiden: a physical AI agent that controls phones over USB HID Most GUI agent work assumes the agent lives inside the device or drives it through a debugging interface. We went the other way. Aiden is a small board that sits outside the host. It captures the screen over HDMI-to-CSI, runs the agent loop on-device, and sends actions back as a standard USB HID device — the host sees a keyboard and a mouse, nothing else. No app install, no root, no ADB, no cloud. Runtime is Go. Frame capture, full-duplex audio with VAD, the agent loop, and HID output all run as independent goroutines. There's no backend — nothing leaves the device, which is the only defensible design when the input is a live feed of someone's phone screen. Open questions we haven't solved: · Action verification — inferring success from a re-read of the screen breaks when loading states lie · Prompt injection — an agent that reads screens reads whatever an attacker puts on them · iOS pointer control requires AssistiveTouch Repo, including the HID gadget config and capture pipeline: github.com/AidenAI-IO/aiden-hardware-demo Wrote up how this differs from cloud-based computer use agents here: https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/ Note: current hardware is a dev board, not a finished product.
View all activity

Organizations

None yet