Invited Talks
Natasha Jaques
Bio: Natasha Jaques holds a joint position as a Senior Research Scientist at Google Brain and Visiting Postdoctoral Scholar at UC Berkeley. Her research focuses on Social Reinforcement Learning in multi-agent and human-AI interactions. Natasha completed her PhD at MIT, where her thesis received the Outstanding PhD Dissertation Award from the Association for the Advancement of Affective Computing. Her work has also received Best Demo at NeurIPS, an honourable mention for Best Paper at ICML, Best of Collection in the IEEE Transactions on Affective Computing, and Best Paper at the NeurIPS workshops on ML for Healthcare and Cooperative AI. She has interned at DeepMind, Google Brain, and was an OpenAI Scholars mentor. Her work has been featured in Science Magazine, Quartz, IEEE Spectrum, MIT Technology Review, Boston Magazine, and on CBC radio. Natasha earned her Masters degree from the University of British Columbia, and undergraduate degrees in Computer Science and Psychology from the University of Regina. More about Natasha, but more importantly her research, can be found on her website: https://natashajaques.ai
Title: Social Reinforcement Learning
Abstract: Social learning helps humans and animals rapidly adapt to new circumstances, coordinate with others, and drives the emergence of complex learned behaviors. What if it could do the same for AI? This talk describes how Social Reinforcement Learning in multi-agent and human-AI interactions can address fundamental issues in AI such as learning and generalization, while improving social abilities like coordination. I propose a unified method for improving coordination and communication based on causal social influence. I then demonstrate that multi-agent training can be a useful tool for improving learning and generalization. I present PAIRED, in which an adversary learns to construct training environments to maximize regret between a pair of learners, leading to the generation of a complex curriculum of environments. Agents trained with PAIRED generalize more than 20x better to unknown test environments. Finally, I demonstrate the difference between social learning and imitation learning, and present a method for selectively learning who and what to imitate by computing when following other agents’ policies would pay off under the learner’s own preferences. Together, this work argues that Social RL is a valuable approach for developing more general, sophisticated, and cooperative AI.
Bei Peng
Bio: Bei Peng is currently a Lecturer (Assistant Professor) in the Department of Computer Science at the University of Liverpool. Her research focuses mainly on deep reinforcement learning, multi-agent systems, interactive machine learning, and curriculum learning. Prior to Liverpool, Bei was a Postdoctoral Researcher in reinforcement learning at the Whiteson Research Lab at the University of Oxford, and a Non-Stipendiary Lecturer in Computer Science at St Catherine's College. Bei received a B.S. in Computer Science from the Huazhong University of Science and Technology in China in 2012 and a Ph.D. in Computer Science from the Washington State University in 2018.
Title: Cooperative Multi-Agent Reinforcement Learning
Abstract: Many real-world learning problems involve multiple agents acting and interacting in the same environment to achieve some common goal, which can be naturally modelled as cooperative multi-agent systems. In this talk, I first overview some of the key challenges we focus on in cooperative multi-agent reinforcement learning. I then focus on discussing two of our recent work on addressing some of these challenges. In one work, we show how overestimation in deep multi-agent Q-learning can be more severe than previously acknowledged and can lead to divergent learning behaviour in practice. We propose a method that uses a new regularisation-based update scheme and an approximate softmax operator, to reduce the potential overestimation bias. In another work, we propose a deep multi-agent actor-critic method that uses a centralised but factored critic and a new centralised policy gradient, to enable more efficient and scalable learning in both discrete and continuous cooperative tasks. Finally, I present Multi-Agent MuJoCo, a new comprehensive benchmark suite that we developed, based on the popular single-agent MuJoCo benchmark, to allow the study of decentralised continuous control
Patrick MacAlpine
Bio: Patrick MacAlpine is a research scientist at Sony AI, and his research spans the areas of autonomous multiagent systems, robotics, and machine learning with an emphasis on reinforcement learning. He completed a Ph.D. in the computer science department at the University of Texas at Austin where he was advised by Peter Stone. During his Ph.D. Patrick served as the team leader of the robot soccer UT Austin Villa RoboCup 3D Simulation League team, and much of his dissertation work contributed to the team winning the RoboCup 3D Simulation League world championship multiple years. Patrick received both bachelor’s and master’s degrees in electrical engineering from Rice University. Prior to joining Sony AI, Patrick was a postdoctoral researcher in the reinforcement learning group at Microsoft Research.
Title: Outracing Champion Gran Turismo Drivers with Deep Reinforcement Learning
Abstract: Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world’s best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing’s important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world’s best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.
Herke van Hoof
Affiliation: University of Amsterdam
Bio: Herke van Hoof is currently assistant professor at the University of Amsterdam in the Netherlands, where he is part of the Amlab. He is interested in reinforcement learning with structured data and prior knowledge. Examples of this line of work include reinforcement learning (RL) for combinatorial optimisation, RL with symbolic prior knowledge, and equivariant RL. Before joining the University of Amsterdam, Herke van Hoof was a postdoc at McGill University in Montreal, Canada, where he worked with Professors Joelle Pineau, Dave Meger, and Gregory Dudek. He obtained his PhD at TU Darmstadt, Germany, under the supervision of Professor Jan Peters, where he graduated in November 2016. Herke got his bachelor and master degrees in Artificial Intelligence at the University of Groningen in the Netherlands.
Title: Learning agent policies with RL and structure
Abstract: Reinforcement learning is a very general framework for learning agent policies. It can be applied in many different settings but this comes at the price of data-inefficiency. To learn better in practicable conditions, with my team I aim to impose more structure on models and/or architectures. In this talk, I will discuss two of our recent results in this space. In (multi-agent) MDP homomorphic networks, together with Elise van der Pol and others, we studied how assumptions on symmetries in the environment interaction can be used to improve the data-efficiency of deep reinforcement learning, while still allowing decentralised execution. In another work, with Niklas Höpner and Iliari Tiddi, we studied how prior knowledge of a concept ontology can improve data-efficiency and generalisation performance of an agent in text-based common sense games.
