Edge AI-Enhanced Traffic Monitoring and Anomaly Detection Using Multimodal Large Language Models

Abstract: This research presents an innovative approach to traffic monitoring and incident detection in remote areas, leveraging multimodal large language models (LLMs) deployed on edge AI devices. The study addresses the unique challenges posed by intermittent network connectivity in rural and wilderness routes, where traditional surveillance methods often fail due to limited bandwidth and power constraints.

Our novel solution utilizes edge computing to process video data locally, using multimodal LLMs to convert real-time events into concise, descriptive text summaries. This approach significantly reduces the amount of data that needs to be transmitted, enabling reliable detection and reporting of anomalies and incidents even in environments with poor connectivity. For example, instead of streaming hours of video footage, the system can send a brief text message like "Vehicle collision detected on Highway 7, mile marker 23, at 14:32. Two vehicles involved, no visible injuries."

The research methodology encompasses several key phases. We begin by curating a comprehensive dataset that captures a wide range of traffic scenarios typical of remote areas, including incidents and various traffic conditions. This is followed by a foundational model selection process, where we evaluate cutting-edge open-source multimodal LLMs, with LLaVA, based on the LLama series, chosen for further experimentation. We then focus on an iterative fine-tuning process to enhance the LLM's ability to interpret complex traffic images accurately through multiple rounds of training on our curated dataset. To ensure deployment feasibility on resource-constrained edge devices, we implement compression and quantization techniques to reduce model size while maintaining a balance between computational efficiency and detection accuracy.

The edge AI devices are designed to operate autonomously for extended periods, processing video feeds locally and generating text summaries. These summaries are then transmitted using minimal bandwidth, even over unreliable or intermittent connections such as low-bandwidth cellular networks or satellite communications. This approach allows for continuous monitoring and incident reporting without the need for constant high-bandwidth connectivity.

Our preliminary results from experiments using 100 traffic video sequences reveal promising outcomes in accident detection. The study identifies areas for improvement, including enhancing vehicle type distinction and optimizing performance across varied environmental conditions.

This research represents a significant advancement in applying AI to traffic monitoring and safety management in remote areas. By enabling more effective incident detection and response in challenging environments with limited connectivity, our approach has the potential to greatly enhance road safety and traffic management in underserved regions.