Context-Aware Embodied AI Systems for Human-Centered Environments: From Assistive Guidance to Autonomous Robots

Abstract: Real-world, human-centered environments are highly unstructured, governed not just by geometric constraints, but by semantic cues, social norms, dynamic conditions, and cluttered layouts. While existing robotic systems have achieved geometric competence, they lack the contextual awareness necessary to operate in public spaces such as stores, offices, libraries, and transit stations. This contextual gap limits the generalizability of autonomous agents and also poses a significant barrier for people with visual impairments, for whom much of this environmental information is only present visually and therefore hard to access directly.

To bridge this gap, this dissertation develops context-aware embodied AI methods that interpret implicit environmental context for robust guidance and spatial grounding. Using assistive technology as a testbed, this work first establishes foundational methods for context-aware guidance by modeling social dynamics and learning from human hand movement for grasp guidance.

The research then extracts spatial knowledge from multimodal data to equip broader autonomous systems with semantic understanding. It introduces novel semantic localization methods for quasi-static environments, initially through distributional semantic particle filtering, and then through vision-language model (VLM) enhanced localization to resolve geometric aliasing. Finally, it presents a semantic-topology representation to support intent-aware search, zone classification, one-shot semantic localization, and the generation of natural language route instructions.

Across these contributions, this thesis demonstrates that social, semantic, and spatial cues can improve robotic systems in environments where geometry alone is insufficient. Together, these systems lay the groundwork for deploying robust, socially intelligent systems capable of long-term, practical autonomy in human-centered spaces.

Bio: Shivendra Agrawal is a Ph.D. candidate in Computer Science at the University of Colorado Boulder, advised by Prof. Bradley Hayes. His research focuses on context-aware embodied AI for human-centered environments, spanning assistive technology for people with visual impairments and semantic spatial understanding for broader autonomous systems. Following his defense, he will join Yale University as a postdoctoral researcher working with Prof. Marynel Vázquez.