Autoomous Vehicles (AVS) such as Zoox-Special-built Robotaxi take a new era in human mobility, but the implementation of AVS comes with many challenges. It is important to perform extensive safety test simulation, which requires the creation of synthetic driving scenarios on scale. Particularly important is to generate realistic safety -critical road scenarios, to test how AVs will respond to a wide range of driving sites included those who are relatively rare and potentially dangerous.
Traditional methods tend to produce scenarios with limited complexity and fight to replicate Mary real world situations. Recently, Machine Learning (ML) models have used deep learning to produce complex traffic scenarios based on specific regions, but they offered limited funds to shape the resulting scenarios in terms of vehicle positioning, speeds and lanes. This makes it difficult to create specific security critical scenarios on scale. Designing a large number of such scenarios by hand is meanwhile impite.
In a paper we presented at the 2023 conference on neural information processing systems (Neurips), we address these challenges with a method we call scenario -diffusion. Our system included in new ML architecture based on latent diffusion, an ML technique used in image generation where a model learns to convert random noise to detailed images.
Scenario -diffusion is able to emit very controllable and realistic traffic scenarios on scale. It is controllable because output from the scenario -diffusion model is based on the map of the desired area, but also on sets easily produced descriptors that can specific location and properties for some or all vehicles in a scene. These descriptors, which we call Agent -Tokens, take the form of function vectors. Similarly, we can specify specific global stage censuses that indicate how busy roads in a given scenario should be.
By combining a diffusion architecture with these token -based controls, we allow us to produce security -critical driving scenarios as desired, increasing our ability to validate the security of our purpose -built robotaxia. We are pleased to use generative AI, where it can have a major impact on the established practical challenge of AV security.
Inside the scenario diffusion model
AV-control software is typically divided into perception, prediction and motion planning modules. Along the way, an AV’s cameras and other sensors perceive the road situation, which can be represented for movement planning purposes as a simplified bird-eye view.
Each of the vehicles (“agents”) in this multi-channeled image, included by itself, is represented as a “bounding box” that reflects the width, length and position of the vehicle on the local map. The image also contains information about other properties of vehicles, such as the headline and lane. These properties and the map itself are the two key elements of a synthetic driving scenario required to validate movement planning in simulation.
The scenario’s diffusion model has two components. The first is an autoCoder, such as complex driving scenarios for a more manageable Represh room. The second component, the broadcast model, works in this room.
Like all diffusion models, bears are trained by adding noise to real world scenarios and asking the model to remove this noise. Once the model is trained, we can try random noise and the model to gradually convert this noise to a realistic driving scenario. For a detailed exploration of our training and inference processes and model architecture, dive into our paper.
We trained the model of both public and proprietary data sets in the real world of running logs containing millions of driving scenarios across a variety of geographical regions and sets.
Previous ML methods for generating driving scenarios typically place boundary boxes for agents on maps – essentially a static snapshot without motion information. They are opposite of recognition to identify these boxes before heuristics or learned methods to decide to follow the course for each agent. Such hybrid solutions can fight to catch the nuances of the real world driving.
A key contribution from our work is that it achieves the contemporary inference of agent location and behavior. When our trained model generates a traffic scenario for a given card, every agent it places in the stage has an associated function vector that describes its properties, such as dimensions, orientation and trajectory of the vehicle. The driving scenario appears fully.
Our function vector access not only provides more realistic scenarios, but also makes it very easy to add information to the model, making it very adaptable. In the paper we only deal with standard vehicle, but it would be straightforward to generate more complex scenarios that included bicycles, pedestrians, scooters, animal-something previously encouraged by a zoox robotaxia in the real world.
Creating security -critical “edge cases” we ask
If we just want to create many thousands of realistic driving scenarios, without any particular situation in mind, we let scenario -diffusion freely generate traffic on certain maps. This type of approach has been studied in previous research. But randomly generated scenarios are not an effective way to validate how AV software handles rare, security-critical events.
Imagine that we want to validate how an AV will behave in a security-critical situation-as a bus turning right in front of it is given short. Creating such scenarios is straightforward to scenario -diffusion thanks to its use of agent and global stage stage. Agent tokens can easily be calculated from data in real driving logs or created by humans. Then they can be used to get the model to place vehicles with desired properties in specific rents. The model will include these vehicles in its generated scenarios while creating additional agents to fill the rest of the stage in a realistic way.
With only one GPU, it takes about a second to generate a new scenario.
Successful generalization across regions
To evaluate our Model’s ability to generalize across geographical regions, we trained separate models about data from each region of the Zoox data set. A model that was trained exclusively to run logs from, for example, San Francisco made better generate realistic driving scenarios for San Francisco than a model trained on data from Seattle. However, models trained in the full zoox data set in four regions come very close to the performance of regional specialized models. These findings suggest that although there are unique aspects of each region, the fully trained model has sufficient capacity to capture this diversity.
The ability to generalize to other cities is good news for the future of AV validation when Zoox is expanded to new metropolitan areas. It will always be necessary to collect the real driving strains in new places using the AVS equipment with our full sensor architecture and monitored by a safety driver. However, the ability to generate soup -synthetic data will shorten the time it takes to validate the Ove AV control system in new areas.
We plan to build on this research by making the model’s output more and more rich and nuanced with a greater diversity of vehicle and objects to better match the complexity of real streets. For example, we could eventually design a model to generate very complex security scenarios, such as driving with a school space at the time of dismissal, with crowds of children and parents near or wasted on the road.
It is this powerful combination of flexibility, controllability and increasing realism that we believe will make our scenario diffusion method basic for the future of security validation for autoomous vehicles.
Recognitions: Meghana Reddy Ganesina, Noureldin Hendy, Zeyu Wang, Andres Morales, Nicholas Roy.