The workshop will take place between 9 a.m. and 5 p.m. (SGT) on December 12 in the Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue).
Overview
Over the past years, the scientific community has grown more cognizant of the importance and challenges of transparent and reproducible research. This topic has become increasingly important given the rise of complex algorithms (e.g., machine learning models or optimization-based algorithms), which cannot be adequately documented in standard publications alone. Benchmarking and code sharing are two key instruments that researchers use to improve reproducibility. Benchmarks have played a critical role in advancing the state of the art in machine learning research. Analogously, well-established benchmarks in controls could enable researchers to compare the effectiveness of different control algorithms. There are currently only a few benchmarks available for comparing control algorithms (e.g., the Autonomie simulation model of a Toyota Prius or the shared experimental testbed Robotarium). Limited comparisons are also due to the modest number of open-source implementations of control algorithms. Over a six-year period (2016-2021), we found that the percentage of papers with code at CDC has more than doubled. However, we also found that at CDC 2021 only 2.6% of publications had code (compared to around 5% at the robotics conference ICRA and over 60% at the machine learning conference NeurIPS). These trends are encouraging, but there is still much work to be done to promote and increase efforts toward reproducible research that accelerates innovation. Benchmarking and releasing code alongside papers can serve as a critical first step in this direction. By organizing this workshop at the Conference on Decision and Control (CDC 2023), we aim to increase awareness of these challenges and inspire attendees to contribute to benchmarking efforts and share open-source code through publication in the future.
Discussion Themes | Call for Contributions | Program | Speakers | Organizers
Discussion Themes
Theme A: Reproducibility and Open-Source Code
Reproducibility of algorithms and experiments is a fundamental requirement of scientific progress. In recent years, we have seen the proliferation of tools to make algorithms, experiments, and results open-source and reproducible; however, their adoption in the controls community did not grow at the same pace. This has led to challenges in reproducing results within the controls community. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have made their research open-source and reproducible, what the challenges are, and the benefits they have experienced from doing so.
Theme B: Benchmarking
Benchmarks allow researchers to evaluate the effectiveness of their approaches against the state-of-the-art, and have played a critical role in advancing machine learning research. Similarly, well-established benchmarks in controls could enable quantitative comparisons between different control approaches and help direct the progress of controls research. However, there are only a few benchmarks currently available for comparing control algorithms. We additionally plan to coordinate with the organizers of the “Autonomous Driving Control Benchmark Challenge” to gain their perspective on benchmarking. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have benchmarked their research, and the benefits they have experienced from doing so.
Call for Contributions
We invite submissions of short abstracts (maximum 300 words) on efforts and challenges in improving accessibility and reproducibility of research in control through benchmarks, open source code, software/hardware platforms, and educational content. Submissions are due September 18, 23:59 PST. Accepted abstracts will be presented as lightning talks at the workshop. Topics of interest include but are not limited to:
– Any open-source implementation of control algorithms
– Any benchmarks and comparisons of control approaches, or competitions
– Tools and software/hardware platforms that enable accessible and reproducible research
– Tutorials/lectures on reproducible research best practices and standards
– Educational resources to make control theory accessible
By bringing together researchers in these topics, we aim to improve the accessibility, reproducibility, comparability, usability, and visibility of research in control theory. We look forward to your contributions!
Abstract submission link: http://tiny.cc/cdc23-ws-abstract
Program
The workshop location is Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue). For virtual attendance check our YouTube channel: http://tiny.cc/cdc-ws-23-youtube
Below is a tentative program for the workshop. Times are in SGT.
Theme A: Reproducibility and Open-Source Code
09:00 – 09:10: Opening Remarks
09:10 – 09:40: Theme A Introduction
09:40 – 10:20: Lightning Talks I
10:20 – 10:30: Coffee Break
10:30 – 11:00: Lightning Talks II
11:00 – 11:30: Theme A Invited Talk 1 by Magnus Egerstedt on “The Robotarium: A Remotely Accessible Robotics Lab”
11:30 – 12:00: Theme A Invited Talk 2 by Sawyer Fuller
12:00 – 12:30: Theme A Focused Panel Discussion with Sandra Hirche and Sawyer Fuller
12:30 – 13:30: Lunch Social
Theme B: Benchmarking
13:30 – 13:40: Theme B Introduction
13:40 – 14:20: Lightning Talks III
14:20 – 15:00: Lightning Talks IV
15:00 – 15:20: Coffee Break
15:20 – 15:50: Theme B Invited Talk 1 by Antonin Raffin on “Practical Tips for Reliable Reinforcement Learning”
15:50 – 16:20: Theme B Invited Talk 2 by Changliu Liu
16:20 – 16:50: Theme B Focused Panel Discussion with Antonin Raffin and Changliu Liu
16:50 – 17:00: Concluding Remarks
17:00 – 18:00: Social Gathering
Lightning Talks I
Majid Zamani (University of Colorado Boulder) on “Unlocking the Power of Symbolic Controller Synthesis via SCOTS”
Katrin Baumgärtner (University of Freiburg) on “acados – an open-source framework for fast embedded optimal control”
Jon Arrizabalaga Aguirregomezcorta (Technical University of Munich) on TBD
Armin Nurkanović (University of Freiburg) on “NOSNOC: A Software Package for Numerical Optimal Control of Nonsmooth Systems”
Lightning Talks II
Yifeng Zhu (University of Texas at Austin) on “Deoxys: A modular robot controller library that makes learning and research easy”
Jonas Eschmann (New York University, Technology Innovation Institute) on “RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control”
Jan Drgona (Pacific Northwest National Laboratory) on “Neuromancer: Differentiable Programming Library for Modeling and Control”
Lightning Talks III
Hanna Krasowski (Technical University of Munich) on “Benchmarking for motion planning – Showcasing CommonRoad and beyond”
Jinjie Li (University of Tokyo) on “The Open-Source Development of UTokyo Aerial Robot Team: From Hardware to Software”
Robert Dyro (Stanford University) on “Beyond Basics: X-Plane Offers Real-World Testing Environment for Control Systems”
Armin Mokhtarian (RWTH Aachen) on “Reproducibility and Accessibility in the Cyber-Physical Mobility Lab”
Lightning Talks IV
Enrica Soria (CERN) on “Open-source software for data-driven control of mechatronic and robotic systems at CERN”
Michael Bosello (Technology Innovation Institute) on “A Fully-annotated, Open-design Dataset of Autonomous and Piloted High-speed Flight”
Tim Salzman (Technical University of Munich) on “Real-Time Neural-MPC”
Spencer Teetaert (University of Toronto) on “Accessible zero-shot sim2real transfer for learning-control benchmarking and hands-on education”
