This work focuses on enhancing autonomous vehicle navigation systems by leveraging the semantic information contained in traffic signs, with a particular emphasis on text-based signs. We introduce an approach to interpret the text on signs for downstream autonomous vehicle applications, such as determining parking restrictions and identifying roadwork zones and directional commands. Specifically, we build upon our previously developed method [1] to create a more fine-grained representation of traffic sign parsing and utilize a pre-trained language model in a constrained setting. This approach helps alleviate the need for hand-crafted rules, making our method significantly more scalable. Our approach decomposes the parsing task into simpler tasks, each of which can be implemented using a single lightweight transformer-based language model, thereby improving efficiency and interpretability. We demonstrate the effectiveness of our method through experiments, showing substantial improvements in accuracy over a previously introduced baseline. To the best of our knowledge, this is the first application of a unified language model to this task, representing a significant step toward open-world sign interpretation.

Autonomous vehicles (AVs) are becoming increasingly prevalent. However, current AVs are unable to handle unexpected traffic signs (e.g., construction zones, road closures) encountered on the roads. To address this limitation, we propose MOSER, a Multimodal rOad Sign intERpretation system, to enable automated detection and interpretation of diverse road signs. Our system consists of a pipeline architecture with three main components, including perception, text processing, and planning. The perception component detects arbitrary road signs and extracts the sign text into proper groups and orders. The text processing component then identifies the high-level semantics of the text and determines whether any actions are required for the autonomous vehicle. Based on the interpretation of the signs, the planning component provides navigation guidance, such as instructing the vehicle to stop at a specific location or adding rules to its internal map. To the best of our knowledge, this is the first attempt to address the interpretation of arbitrary road signs using a multimodal processing strategy. Our work provides important insights and capabilities to support Level 4 autonomous vehicles, ensuring their safety and smoothness of operation.