<\/p>\n\n\n
Temporal features, relationships between spatial features across time, are integral to understanding many of the tasks that humans engage in daily. Unfortunately, learning temporal features directly from video is a complex problem and contemporary deep learning-based models are duration-specific making them vulnerable to variations in the duration over which activities are expressed and limiting the scale of the data they can model. I address an approach that can overcome these challenges when capturing temporal features from video of human-led demonstrations of sequential tasks. Inference using these temporal features is then leveraged for better performance on policy learning and activity recognition tasks.<\/p>\n
https:\/\/youtu.be\/seVCV3UGPUQ<\/a><\/p>\n Sponsor:<\/strong> National Science Foundation IIS 1664554<\/p>\n Investigator: <\/strong>\u00a0Madison Clark-Turner<\/span><\/p>\n Publications:<\/strong><\/p>\n We are interested in learning the high-level policy of multi-step sequential (MSS) tasks, such as activities of daily living, from video demonstrations. Videos of MSS tasks are typically long in duration and exhibit large feature variance, especially when captured in non-engineered settings. Learning task policy from such videos using state-of-the-art end-to-end approaches is sample inefficient due to a reliance on pixel-level information. Understanding the unique temporal structures of MSS tasks can make policy learning easier and sample efficient. However, understanding this temporal structure requires analyzing the entire content of the video or a task which is a complex and under-explored area in the current literature. We propose a hierarchical solution to this problem where i) an automated feature selection process extrapolates temporally grounded task-relevant features from the video and then ii) a stochastic policy learning model learns a feature-constrained task policy. The proposed model is sample efficient as a result of substituting selected temporally-grounded features for pixel-level information. We demonstrate the efficacy of our proposed framework by teaching a YuMi robot a tea making task from videos and then comparing our approach’s performance to a behavioral cloning baseline.<\/p>\n https:\/\/youtu.be\/DjUgI2-aMmo<\/p>\n Sponsor:<\/strong> National Science Foundation IIS 1830597<\/p>\n Investigator: <\/strong>\u00a0Mostafa Hussein<\/p>\n Publications:<\/strong><\/p>\n This project is focused on trajectory learning from demonstration (LfD) for assistive robotics. The primary objectives of trajectory LfD are to learn motion tasks from demonstrations and exhibit robustness to spatial and temporal perturbations. Depending on the operational environment, the specific objectives of LfD algorithms may very. For rehabilitation robotics, a robot therapist must demonstrate proper exercise movements and provide feedback on a human\u2019s exercise execution. In a workplace setting, a robot collaborator should fluently hand over objects to its human counterpart. In all applications, the LfD algorithm must provide a feasible trajectory for the robot to execute, ideally in the form of a controller. The key to success in multi-purpose trajectory LfD lies in 1) controller expressivity, which is the set of trajectories that can be reproduced, and 2) constraint enforcement, which eliminates infeasible trajectories.<\/p>\n Three different learning strategies\u00a0 have been developed, namely controller learning via regression, controller learning via optimization, and controller learning via inverse optimal control. These approaches have been published in\u00a0 ICRA 2019, IROS 2020, IROS 2021. Videos from each publication are shown below.<\/p>\n\n
Policy Learning for Sequential Tasks from Demonstrations<\/strong><\/h3>\n
\n\n
Trajectory Learning from Demonstrations<\/strong><\/h3>\n
\n