PACE-ADS: Physiology and Cognition Enabled Automated Driving Systems

Abstract: Despite significant advancements over the past decades, autonomous vehicles (AVs) still face several limitations that impede their widespread deployment. Two key technical challenges stand out. First, AVs lack attentiveness to occupants. While some vehicles are equipped with sensors to monitor driver attention or fatigue and issue warnings, the integration of occupant physiological signals into AV decision-making remains unexplored. This gap limits AVs' ability to dynamically adjust their driving behavior in real time based on the occupants' state, reducing the potential for personalized riding experiences and hindering the broader adoption of automation technologies. For example, if occupants appear anxious due to an aggressive driving style, the vehicle should adapt by switching to a more conservative approach. Second, AVs struggle to handle corner cases. While they perform well in typical traffic conditions, they face significant challenges when confronted with rare or complex scenarios—situations where human drivers typically excel. AVs currently exhibit a crash rate of 9.1 incidents per million miles, significantly higher than the 4.1 incidents per million miles observed in manually driven vehicles, underscoring their difficulties in navigating unpredictable or unusual driving conditions.
This research is motivated to pioneer the development of physiology and cognition enabled automated driving systems (PACE-ADS). It enhances the responsiveness of AVs to both occupant needs and environmental factors, thereby providing a personalized riding experience and improving operational safety. A multi-agent collaborative decision-making framework is proposed to leverage the collective intelligence of several Multimodal Large Language Models (MLLMs). As the first of its kind, this innovative framework assigns specific roles to different MLLMs: one acts as a driver, analyzing external traffic conditions to suggest optimal maneuvers (such as car following, lane change, and emergency stop); another as a psychologist, interpreting occupants' physiological signals (e.g., ECG, facial expressions) and cognitive inputs (e.g., verbal commands) to assess their physiological states and intentions; and a third serves as a coordinator, selecting appropriate maneuver parameters to guide AV operations while addressing occupant needs monitoring occupants' physiological signals (ECG signal and facial expression) and cognitive commands (verbal command or gesture) to analyze occupants' physiological states and intention; and a third as a coordinator, selecting suitable control algorithms and generating appropriate value for the selected algorithm’s parameters, when it receives the decision results from the driver agent and the occupants' physiological states output by the psychologist.
A closed-loop simulation platform is established to evaluate the performance of PACE-ADS, using ChatGPT as the backbone model for vehicle control and CARLA for traffic scenario construction. The results show that PACE-ADS can detect subtle signs of stress or discomfort and promptly adjust driving behavior, tailoring the ride to individual preferences. It also exhibits significant safety improvements in handling corner cases by integrating advanced occupant cognitive responses. The success of PACE-ADS highlights the feasibility of delivering safer, more reliable, and personalized transportation solutions. This is crucial for fostering public trust and acceptance of AVs, potentially transforming urban mobility and enhancing quality of life.