IT4Innovations national supercomputing center has been coordinating the EXA4MIND project since January last year. The consortium consists of ten partners from six European countries. Among the partners are the Czech Institute of Informatics, Robotics, and Cybernetics at the Czech Technical University (CIIRC CTU) and Valeo.ai company, which have now introduced a new method to search image data from autonomous vehicles using natural language.
Antonín Vobecký and Josef Šivic from CIIRC CTU, in collaboration with Valeo.ai, have achieved significant success in developing machine learning models for data analysis, particularly image data from autonomous vehicles. Their methods, which have the potential to enhance the safety and reliability of autonomous driving, were presented at the prestigious NeurIPS 2023 conference in New Orleans, which focuses on machine learning.
“In a paper POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images we presented a method that processes 360° camera images of a car’s surroundings as input and produces a 3D semantic map of the occupancy of the surrounding environment. In addition, the user can ask the system in natural language about the location of objects in this digital 3D space. The capability to make queries in natural language allows for semantic occupancy segmentation or text-driven search in 3D, both performed exclusively from 2D images taken by the vehicle’s cameras,” explains Antonín Vobecký, a PhD student in ELLIS Unit Prague at CIIRC CTU. Questions asked might be, for example, „Where is the plastic trash bin?“ or „Where is the black truck with the trailer?“.
The overall motivation is to develop methods to search for corner case situations, which are safety-critical but often rare in the driving recordings. The goal of the research is to enable the retrieval of these captured situations in huge files of petabytes of data by simply entering instructions in natural language. “Once such situations are found, they can be used as additional training data to improve existing machine-learning models and algorithms for autonomous driving so that systems can better deal with such situations and, as a result, improve the safety and reliability of autonomous vehicles,” adds Vobecký.
This research was carried out as part of the EXA4MIND project, an EU-funded Horizon Europe project that is creating an extreme data platform that combines supercomputing and big data infrastructure to automate data analysis, management, and storage. In the EXA4MIND project, the team collaborates with one of the European automotive leaders, Valeo.ai, to develop tools for large-scale management and analysis of driving data, which will lead to safer systems used in autonomous cars.
