Evaluation Methodology and Results
Objectives
The main objectives of WP6 were to define appropriate methodologies for the evaluation of AV interaction strategies, and to apply these methodologies to assess the impacts of the interACT communication solutions, in particular the eHMI and iHMI developed in WP4, and the safety layer developed in WP3.
Technical Approach
A multi-method approach was taken for this evaluation, as different methods have been shown to have different strengths and weaknesses. The demonstrator vehicles were evaluated using a mixture of test-track and real-world studies. These were complemented with pedestrian and driving simulator tests which allowed the evaluation of the eHMI and iHMI using a number of different scenarios. Threshold distribution models were used to test predictions about when the road user would start crossing the road in an accepted traffic gap, and what the impact of eHMI and deceleration would be on this timing. Finally, threat assessment models were used to evaluate the success of the interACT safety layer.
Figure 1: HIKER Pedestrian Simulator, University of Leeds
Figure 2: CRF Test Track
Figure 3: BMW Test Track
Main results
- Mathematical models show that the incorporation of the interACT safety layer means that the Automated Vehicle never causes an accident, no matter how other road users are moving. The fail-safe manoeuvres ensure the availability of safe actions, even in situations where vulnerable road users behave unexpectedly.
- Based on this, we expect that the safety layer will also increase the comfort and trust of humans in Automated Vehicles.
- The simulator and test-track studies show that when there is no eHMI present, participants are able to use kinematic information to make judgments about Automated Vehicle behaviour. There were no differences in crossing reaction times between no eHMI and eHMI conditions in the CRF test-track study, and mixed findings in the BMW real-world evaluation.
- However, results from the simulator studies also show that eHMIs could lead to changes in pedestrian crossing behaviour, including earlier crossings, shorter interaction times, and improved traffic flow.
- The visibility of eHMI solutions, and their ability to relay consistent and reliable messages, are important design considerations to ensure positive user evaluations and road user safety.
- The inclusion of failure trials in both pedestrian and driving simulator studies show that the consequences of eHMI failure or miscommunication can be severe, and therefore public guidance around eHMI capability will be required. More studies around the potential negative effects of eHMIs should be conducted.
- A comparison of the results emerging using different methodologies suggests that simulator studies seem to be capturing more effects of eHMI than test-track or real-world studies. This may be linked to the visibility of the eHMI, and is something which should be further investigated in future studies.
Read more
Lee, Y. M., Madigan, R., Markkula, G. Pekkanen, J., Merat, N. et. al. interACT D.6.1. Methodologies for the evaluation and impact assessment of the interACT solutions (2019)