202106-nuPlan：面向自动驾驶汽车的闭环机器学习规划基准

第001/5页（英文原文）

nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

Holger Caesar Juraj Kabzan Kok Seang Tan Whye Kit Fong Eric Wolff Alex Lang Luke Fletcher Oscar Beijbom Sammy Omari Motional

Abstract

In this work, we propose the world’s first closed-loop ML-based planning benchmark for autonomous driving. While there is a growing body of ML-based motion planners, the lack of established datasets and metrics has limited the progress in this area. Existing benchmarks for autonomous vehicle motion prediction have focused on short-term motion forecasting, rather than long-term planning. This has led previous works to use open-loop evaluation with L2-based metrics, which are not suitable for fairly evaluating long-term planning. Our benchmark overcomes these limitations by introducing a largescale driving dataset, lightweight closed-loop simulator, and motion-planning-specific metrics. We provide a highquality dataset with 1500h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Boston, Pittsburgh, Las Vegas and Singapore). We will provide a closed-loop simulation framework with reactive agents and provide a large set of both general and scenario-specific planning metrics. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.

1. Introduction

Large-scale human labeled datasets in combination with deep Convolutional Neural Networks have led to an impressive performance increase in autonomous vehicle (AV) perception over the last few years [9, 4]. In contrast, existing solutions for AV planning are still primarily based on carefully engineered expert systems, that require significant amounts of engineering to adapt to new geographies and do not scale with more training data. We believe that providing suitable data and metrics will enable ML-based planning and pave the way towards a full “Software 2.0” stack.

Existing real-world benchmarks are focused on shortterm motion forecasting, also known as prediction [6, 4, 11, 8], rather than planning. This is evident in the lack of high-level goals, the choice of metrics, and the openloop evaluation. Prediction focuses on the behavior of other agents, while planning relates to the ego vehicle behavior.

Figure 1. We show different driving scenarios to emphasize the limitations of existing benchmarks. The observed driving route of the ego vehicle in shown in white and the hypothetical planner route in red. (a) The absence of a goal leads to ambiguity at intersections. (b) Displacement metrics do not take into account the multi-modal nature of driving. © open-loop evaluation does not take into account agent interaction.

Prediction is typically multi-modal, which means that for each agent we predict theNNNmost likely trajectories. In contrast, planning is typically uni-modal (except for contingency planning) and we predict a single trajectory. As an example, in Fig. 1a, turning left or right at an intersection are equally likely options. Prediction datasets lack a baseline navigation route to indicate the high-level goals of the agents. In Fig. 1b, the options of merging immediately or later are both equally valid, but the commonly used L2 distance-based metrics (minADE, minFDE, and miss rate) penalize the option that was not observed in the data. Intuitively, the distance between the predicted trajectory and the observed trajectory is not a suitable indicator in a multimodal scenario. In Fig. 1c, the decision whether to continue to overtake or get back into the lane should be based on the consecutive actions of all agent vehicles, which is not possible in open-loop evaluation. Lack of closed-loop evaluation leads to systematic drift, making it difficult to evaluate beyond a short time horizon (3-8s).

We instead provide a planning benchmark to address these shortcomings. Our main contributions are:

• The largest existing public real-world dataset for autonomous driving with high quality autolabeled tracks from 4 cities.
• Planning metrics related to traffic rule violation, human driving similarity, vehicle dynamics, goal achievement, as well as scenario-based.
• The first public benchmark for real-world data with a closed-loop planner evaluation protocol.

第001/5页（中文翻译）

nuPlan：面向自动驾驶汽车的闭环机器学习规划基准

摘要

In this work, we propose the world’s first closed-loop ML-based planning benchmark for autonomous driving.

While there is a growing body of ML-based motion planners, the lack of established datasets and metrics has limited the progress in this area. Existing benchmarks for autonomous vehicle motion prediction have focused on short-term motion forecasting, rather than long-term planning. This has led previous works to use open-loop evaluation with L2-based metrics, which are not suitable for fairly evaluating long-term planning. Our benchmark overcomes these limitations by introducing a largescale driving dataset, lightweight closed-loop simulator, and motion-planning-specific metrics. We provide a highquality dataset with 1500h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Boston, Pittsburgh, Las Vegas and Singapore). We will provide a closed-loop simulation framework with reactive agents and provide a large set of both general and scenario-specific planning metrics. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.

在本工作中，我们提出了全球首个用于自动驾驶的闭环机器学习规划基准。

尽管基于机器学习的运动规划器日益增多，但由于缺乏公认的数据集和评估指标，该领域的进展受到了限制。

现有的自动驾驶汽车运动预测基准主要集中在短期运动预测，而非长期规划。

这导致以往的研究采用开环评估和基于 L2 的指标，而这些方法并不适合公平地评估长期规划。

我们的基准通过引入大规模驾驶数据集、轻量级闭环仿真器以及运动规划专用指标，克服了这些局限性。

我们提供了一个高质量的数据集，包含来自美国和亚洲四个城市（波士顿、匹兹堡、拉斯维加斯和新加坡）的1500 小时人类驾驶数据，这些城市的交通模式差异显著。

我们将提供一个具备反应式智能体的闭环仿真框架，并提供大量通用及场景特定的规划指标。

我们计划于 NeurIPS 2021 发布该数据集，并从 2022 年初开始组织基准挑战赛。

1. 简介

大规模人工标注数据集与深度卷积神经网络相结合，在过去几年中显著提升了自动驾驶汽车（AV）感知的性能 [9, 4]。相比之下，现有的自动驾驶规划解决方案仍主要基于精心设计的专家系统，这些系统需要大量的工程工作才能适应新的地理环境，且无法随着训练数据的增加而扩展。我们相信，提供合适的数据和评估指标将推动基于机器学习的规划，并为实现完整的“软件 2.0"栈铺平道路。

现有的现实世界基准测试专注于短期运动预测（也称为预测 [6, 4,11, 8], ），而非规划。这一点体现在缺乏高层目标、评估指标的选择以及开环评估上。预测关注其他智能体的行为，而规划则涉及自车的行为。

图 1. 我们展示了不同的驾驶场景，以强调现有基准的局限性。自车的实际行驶路线以白色显示，假设的规划器路线以红色显示。(a) 缺乏目标会导致在交叉路口产生歧义。(b) 位移评估指标未考虑驾驶的多模态特性。© 开环评估未考虑智能体交互。

预测通常是多模态的，这意味着对于每个智能体，我们预测其NNN最可能的轨迹。相比之下，规划通常是单模态的（应急规划除外），我们仅预测单一轨迹。例如，在图1a 中，在路口左转或右转是同等可能的选项。预测数据集缺乏基线导航路线来指示智能体的高层目标。在图1b 中，立即汇入或稍后汇入都是同样有效的选项，但常用的基于 L2 距离的评估指标（minADE、minFDE 和漏检率）会惩罚数据中未观察到的选项。直观地说，在多模态场景中，预测轨迹与观测轨迹之间的距离并不是合适的指标。在图1c 中，决定继续超车还是返回车道应基于所有车辆智能体的连续动作，而这在开环评估中是无法实现的。缺乏闭环评估会导致系统性漂移，使得难以评估超出短时间范围（3‑8 秒）的表现。

相反，我们提供了一个规划基准以解决这些不足。我们的主要贡献包括：

•现有最大的公开真实