CDC 2023 Workshop on Benchmarking, Reproducibility, and Open-Source Code in Controls

The workshop will take place between 9 a.m. and 5 p.m. (SGT) on December 12 in the Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue).

Paper on the Impact of Releasing Code Recording of the Workshop


Over the past years, the scientific community has grown more cognizant of the importance and challenges of transparent and reproducible research. This topic has become increasingly important given the rise of complex algorithms (e.g., machine learning models or optimization-based algorithms), which cannot be adequately documented in standard publications alone. Benchmarking and code sharing are two key instruments that researchers use to improve reproducibility. Benchmarks have played a critical role in advancing the state of the art in machine learning research. Analogously, well-established benchmarks in controls could enable researchers to compare the effectiveness of different control algorithms. There are currently only a few benchmarks available for comparing control algorithms (e.g., the Autonomie simulation model of a Toyota Prius or the shared experimental testbed Robotarium). Limited comparisons are also due to the modest number of open-source implementations of control algorithms. Over a six-year period (2016-2021), we found that the percentage of papers with code at CDC has more than doubled. However, we also found that at CDC 2021 only 2.6% of publications had code (compared to around 5% at the robotics conference ICRA and over 60% at the machine learning conference NeurIPS). These trends are encouraging, but there is still much work to be done to promote and increase efforts toward reproducible research that accelerates innovation. Benchmarking and releasing code alongside papers can serve as a critical first step in this direction. By organizing this workshop at the Conference on Decision and Control (CDC 2023), we aim to increase awareness of these challenges and inspire attendees to contribute to benchmarking efforts and share open-source code through publication in the future.

Discussion Themes | Call for Contributions | Program | Speakers | Organizers

Discussion Themes

Theme A: Reproducibility and Open-Source Code

Reproducibility of algorithms and experiments is a fundamental requirement of scientific progress. In recent years, we have seen the proliferation of tools to make algorithms, experiments, and results open-source and reproducible; however, their adoption in the controls community did not grow at the same pace. This has led to challenges in reproducing results within the controls community. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have made their research open-source and reproducible, what the challenges are, and the benefits they have experienced from doing so.

Theme B: Benchmarking

Benchmarks allow researchers to evaluate the effectiveness of their approaches against the state-of-the-art, and have played a critical role in advancing machine learning research. Similarly, well-established benchmarks in controls could enable quantitative comparisons between different control approaches and help direct the progress of controls research. However, there are only a few benchmarks currently available for comparing control algorithms. We additionally plan to coordinate with the organizers of the “Autonomous Driving Control Benchmark Challenge” to gain their perspective on benchmarking. Through 30-minute lecture-style talks, the speakers will talk about their research, how they have benchmarked their research, and the benefits they have experienced from doing so.

Call for Contributions

We invite submissions of short abstracts (maximum 300 words) on efforts and challenges in improving accessibility and reproducibility of research in control through benchmarks, open source code, software/hardware platforms, and educational content. Submissions are due September 18, 23:59 PST. Accepted abstracts will be presented as lightning talks at the workshop. Topics of interest include but are not limited to:

– Any open-source implementation of control algorithms
– Any benchmarks and comparisons of control approaches, or competitions
– Tools and software/hardware platforms that enable accessible and reproducible research
– Tutorials/lectures on reproducible research best practices and standards
– Educational resources to make control theory accessible

By bringing together researchers in these topics, we aim to improve the accessibility, reproducibility, comparability, usability, and visibility of research in control theory. We look forward to your contributions!

Abstract submission link: 


The workshop location is Ballroom Melati Junior (Room No. 4111) of Marina Bay Sands, Singapore (CDC Conference Venue). For virtual attendance check our YouTube channel:

Below is a tentative program for the workshop. Times are in SGT.

Theme A: Reproducibility and Open-Source Code

09:00 – 09:10: Opening Remarks
09:10 – 09:40: Theme A Introduction
09:40 – 10:20: Lightning Talks I
10:20 – 10:30: Coffee Break
10:30 – 11:00: Lightning Talks II
11:00 – 11:30: Theme A Invited Talk 1 by Magnus Egerstedt on “The Robotarium: A Remotely Accessible Robotics Lab”
11:30 – 12:00: Theme A Invited Talk 2 by Sawyer Fuller on “Open Source Controls Software in Robot Flies and Education”
12:00 – 12:30: Theme A Focused Panel Discussion with Stefan Sosnowski and Sawyer Fuller
12:30 – 13:30: Lunch Social

Theme B: Benchmarking

13:30 – 13:40: Theme B Introduction
13:40 – 14:20: Lightning Talks III
14:20 – 15:00: Lightning Talks IV
15:00 – 15:20: Coffee Break
15:20 – 15:50: Theme B Invited Talk 1 by Antonin Raffin on “Practical Tips for Reliable Reinforcement Learning”
15:50 – 16:20: Theme B Invited Talk 2 by Changliu Liu
16:20 – 16:50: Theme B Focused Panel Discussion with Antonin Raffin and Changliu Liu
16:50 – 17:00: Concluding Remarks
17:00 – 18:00: Social Gathering

Lightning Talks I

Majid Zamani (University of Colorado Boulder) on “Unlocking the Power of Symbolic Controller Synthesis via SCOTS

Katrin Baumgärtner (University of Freiburg) on “acados – An Open-Source Framework for Fast Embedded Optimal Control

Jon Arrizabalaga Aguirregomezcorta (Technical University of Munich) on “Learning for CasADi: Data-driven models for Numerical Optimization

Armin Nurkanović (University of Freiburg) on “NOSNOC: A Software Package for Numerical Optimal Control of Nonsmooth Systems

Lightning Talks II

Yifeng Zhu (University of Texas at Austin) on “Deoxys: A Modular Robot Controller Library that Makes Learning and Research Easy

Jonas Eschmann (New York University, Technology Innovation Institute) on “RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control

Jan Drgona (Pacific Northwest National Laboratory) on “Neuromancer: Differentiable Programming Library for Modeling and Control

Lightning Talks III

Lightning Talks IV

Enrica Soria (CERN) on “Open-Source Software for Data-Driven Control of Mechatronic and Robotic Systems at CERN

Michael Bosello (Technology Innovation Institute) on “A Fully-Annotated, Open-Design Dataset of Autonomous and Piloted High-Speed Flight

Tim Salzmann (Technical University of Munich) on “Real-Time Neural-MPC

Spencer Teetaert (University of Toronto) on “Accessible Sim2Real for Benchmarking and Hands-on Education

Invited Speakers

Magnus Egerstedt
University of California, Irvine
Stefan Sosnowski
Technical University of Munich

Sawyer Fuller
University of Washington

Changliu Liu
Carnegie Mellon University

Antonin Raffin
German Aerospace Center


Angela Schoellig
Technical University of Munich

Jonathan How
Massachusetts Institute of Technology

Peter Corke
Queensland University of Technology

George Pappas
University of Pennsylvania

Sandra Hirche
Technical University of Munich

Federico Pizarro Bejarano
University of Toronto

Adam Hall
University of Toronto

SiQi Zhou
Technical University of Munich

Lukas Brunke
Technical University of Munich

Jacopo Panerati
Technology Innovation Institute
University of Toronto Institute for Aerospace Studies