R255 Imitation Learning
Imitation learning was initially proposed in robotics as a way to better robots (Schaal, 1999). The connecting theme is to combine the reward function in the end of the action sequence with demonstrations of the task in hand by an expert. Since then it has been applied to a number of tasks which can be modelled as a sequence of actions taken by an agent. These include the video game agents, moving cameras to track players and structured prediction in various tasks in natural language processing. For a recent tutorial see here.
Over the years there has been a number of algorithms proposed, in the literature but without necessarily making the connections between the various approaches clear. The initial lecture will set the criteria to be used to examine the algorithms with.
Each student will present a paper from the list below (check with me if you want to present a different one) and will write a report on a mini-project related to them.
-
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Stephane Ross, Geoffrey J. Gordon, J. Andrew Bagnell Artificial Intelligence and Statistics Conference (AISTATS), 2011
-
Learning to search better than your teacher Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III and John Langford International Conference on Machine Learning (ICML), 2015
-
Sequence Level Training with Recurrent Neural Networks Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba International Conference on Machine Learning (ICLR), 2016
-
Generative Adversarial Imitation Learning Jonathan Ho and Stefano Ermon Neural Information Processing Systems (NeurIPS) 2016
-
One-Shot Imitation Learning Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, Wojciech Zaremba Neural Information Processing Systems (NeurIPS) 2017
-
SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards Siddharth Reddy, Anca D. Dragan, Sergey Levine Eighth International Conference on Learning Representations (ICLR), April 2020
-
Disagreement-Regularized Imitation Learning Kiante Brantley, Wen Sun, Mikael Henaff Eighth International Conference on Learning Representations (ICLR), April 2020
-
Optimal Transport for Offline Imitation Learning Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, Marc Peter Deisenroth International Conference on Machine Learning (ICLR), 2023