Artificial Intelligence 35 min read
Video-as-Experience: Building Audio-Visual LLM Agents for Computer Use
Multimodal LLMs can now ingest video streams to acquire GUI manipulation skills, merging visual cues with audio narration. This monograph dissects the systems pipeline—from frame-level perception to action synthesis—enabling staff engineers to build robust, auditable video-as-experience agents.


