|
Abstract
|
27th International Technical Conference on the Enhanced Safety of Vehicles (ESV): Enhanced and Equitable Vehicle Safety for All: Toward the Next 50 Years
https://www-esv.nhtsa.dot.gov/Proceedings/27/27ESV-000048.pdf
The verification and validation processes of machine learning applications in advanced driving assistance systems or automatic driving systems are presented, and the processes are implemented by using the forward collision warning of pedestrian automatic emergency braking. Supervised learning is one of the machine learning branches using image datasets to train the deep neural network for detecting or identifying the target object or scenario in a vision-based application. The verification process consists of specifying the requirements of a safety functionality, identifying the target objects in the Operation Design Domain (ODD) and pre-crash scenarios, and evaluating the quality and quantity of images based on safety requirements, also the coverage of ODD and pre-crash scenarios. The validation process consists of designing test procedures based on the specified ODD and pre-crash scenarios, conducting a sufficient number of tests, recording the test results, and evaluating the test results based on specified metrics. Eight published pedestrian datasets from 2010 to 2020 are reviewed. Three datasets contain the raining condition, but no dataset had images collected during snowing days. Fog or smoke images are not available in all datasets, and the headlight condition is not addressed in all datasets. The 3 datasets containing pedestrians in the nighttime did not label the vehicle's headlight status as low or high beam. All reviewed datasets had no annotations of pre-crash scenarios that the subject vehicle is maneuvering or not. The validation of pedestrian detection uses the activation of forward collision warning as the evaluation metric. Eleven vehicles were tested in 4 pre-crash scenarios with different pedestrian orientations and speeds: the test pedestrian crossing from the nearside, crossing from the offside, stationary facing away, and walking away in front of the vehicle. The vehicle speed under test is 40 kph and the test pedestrian's speed is 5 or 8 kph. The light conditions are daytime, nighttime with low beam, and nighttime with high beam without streetlighting in a test track. The statistical test results show that some vehicles under test behave inconsistently when the test pedestrian is crossing or not crossing. Test results in the nighttime with high beam are similar to that of the daytime; however, the test results in the nighttime show significant variations compared with that of daytime. No trend or similarity can be found among all vehicles under test, the same vehicle may behave inconsistently under different light conditions and pedestrian orientations. Also, the pedestrian detection time is longer when the test pedestrian is not crossing for some vehicles. The vision-based machine learning application for the vehicle safety functionality reveals the underlying uncertainty of a deep neural network, and it results in the inconsistent performance in differentiated ODD conditions and pre-scenarios.
Language: en |