Redundancy of visual tokens in multi-modal large language models (MLLMs) significantly reduces their computational efficiency.Recent approaches, such as resamplers and summarizers, have sought to reduce the number of visual tokens, but at the cost of visual reasoning ability.To address this, we propose LEO-MINI, a novel MLLM that significantly reduces the number of visual tokens and simultaneously boosts visual reasoning capabilities.For efficiency, LEO-MINI incorporates COTR, a novel token reduction module to consolidate a large number of visual tokens into a smaller set of tokens, using the similarity between visual tokens, text tokens, and a compact learnable query.For effectiveness, to scale up the model's ability with minimal computational overhead, LEO-MINI employs MMOE, a novel mixture of multi-modal experts module.MMOE employs a set of LoRA experts with a novel router to switch between them based on the input text and visual tokens instead of only using the input hidden state.MMOE also includes a general LoRA expert that is always activated to learn general knowledge for LLM reasoning.For extracting richer visual features, MMOE employs a set of vision experts trained on diverse domain-specific data.To demonstrate LEO-MINI's improved efficiency and performance, we evaluate it against existing efficient MLLMs on various benchmark visionlanguage tasks.

Improving the visual understanding ability of vision-language models (VLMs) is crucial for enhancing their performance across various tasks. While using multiple pretrained visual experts has shown great promise, it often incurs significant computational costs during training and inference. To address this challenge, we propose HAWAII, a novel framework that distills knowledge from multiple visual experts into a single vision encoder, enabling it to inherit the complementary strengths of several experts with minimal computational overhead. To mitigate conflicts among different teachers and switch between different teacher-specific knowledge, instead of using a fixed set of adapters for multiple teachers, we propose to use teacher-specific Low-Rank Adaptation (LoRA) adapters with a corresponding router. Each adapter is aligned with a specific teacher, avoiding noisy guidance during distillation. To enable efficient knowledge distillation, we propose fine-grained and coarse-grained distillation. At the fine-grained level, token importance scores are employed to emphasize the most informative tokens from each teacher adaptively. At the coarse-grained level, we summarize the knowledge from multiple teachers and transfer it to the student using a set of general-knowledge LoRA adapters with a router. Extensive experiments on various vision-language tasks demonstrate the superiority of HAWAII, compared to the popular open-source VLMs.

As the realms of Connected Automated Vehicles (CAVs) and the Internet of Vehicles develop, the ability to accurately forecast vehicle motions takes center stage in shaping the future of intelligent transportation systems, integrating vehicles harmoniously into a connected, data-driven ecosystem. Never-theless, vehicle motion forecasting for CAVs faces considerable challenges, including handling multimodal behavior and effectively considering the complex interactions between surrounding agents. To mitigate these challenges, we design an innovative Transformer-based model for Motion Forecasting (TMF) that takes into account the uncertainty in human driving behavior and the complex interactions between agents. More specifically, we explic-itly integrate map constraints by extracting agent-lane temporal and spatial interrelated features. Our transformer-based encoder benefits from an attention mechanism to enable social interactions, effectively acquiring meaningful representations of these scene elements to attain precise predictions. The evaluation results over the extensive Argoverse Motion Forecasting dataset demonstrate that TMF achieves higher performance when compared to several state-of-the-art models.

To reliably and safely navigate dynamic urban environments, connected automated vehicles should anticipate the future motion of surrounding traffic agents, which can have fundamental implications on road safety, traffic management, and network communications in vehicular networks. This requires considering the inherent uncertainty in agents' behavior, making motion prediction challenging. To tackle this issue, we propose conditioning the agents' future motions on both context information and potential multimodal goals. We design a novel Goal-based Motion Prediction approach (GMP) for multimodal motion prediction. By encoding both interactions between agents using temporal convolutions and dynamic and static context information using graph attention, our method estimates the distribution of target goals, efficiently takes the inherent uncertainty in the behavior of agents into account, and generates precise multimodal trajectories. Experimental results indicate that GMP outperforms several benchmarks on the Argoverse Motion Forecasting dataset.

Forecasting the future paths of surrounding vehicles of a Connected Autonomous Vehicle (CAV) can enhance connectivity and efficiency of vehicular networks, and accurate motion forecasting of nearby vulnerable road users can advance road safety and urban mobility. This task needs a high-level situational awareness for the CAV. Early methods rely solely on vehicle kinematics and overlook the uncertainty within agents behavior and the effects of surrounding context on the behavior of nearby agents, resulting in lower performance or infeasible predictions. In the current work, we introduce a novel context-aware forecasting approach for CAVs that benefits from inverse reinforcement learning (IRL) to condition the future motions of nearby agents on scene-based state sequences defined using a Markov Decision Process. More precisely, we map the images of the surrounding context and the behavior history of agents into rewards and learn optimal expert behaviors using IRL. We validate the path forecasting efficiency of our model using two large motion prediction benchmarks with different scenes and achieve state-of-the-art results in terms of FDE and ADE metrics.

Reliably anticipating the future behavior of surrounding vehicles is critical for the safe operation of the Connected Automated Vehicles (CAV) and improves traffic safety. This task requires processing the history and current behavior of a target vehicle and its surrounding vehicles. Nevertheless, this level of situational awareness is challenging due to the limited observability of the ego CAV’s mounted sensors, particularly in unsignalized intersections as an example of a complex scenario. In the current study, we propose an interaction-aware behavior prediction framework for CAVs which takes advantage of vehicular communication technologies to improve the prediction performance at the time of occlusion. With the help of Vehicle-to-Vehicle (V2V) communications, connected vehicles can gain an enriched understanding of the current behavior of the nearby vehicles, leading to an enhanced prediction. We benefit from graph convolutional networks to model the connection between the vehicles. We further analyze the proposed model over a large real-world dataset containing 14867 vehicle trajectories. The results indicate the higher performance of the introduced model against several benchmarks.

As a powerful non-intrusive method, driver identification based on driving data analysis has recently gained attention as it is beneficial for providing security, privacy, and personalization for driver assistance systems. Fortunately, the considerable variety of available in-vehicle sensors and net-working technologies has contributed to collecting high-quality data for driver identification purposes. Nevertheless, the main challenge in this task is extracting and capturing unique driving-related features and behavior of each individual. In this study, we analyze and compare the effectiveness of benchmark deep learning-based approaches in terms of driver identification accuracy. More specifically, we design an encoder-based framework to compare the performance of temporal convolutional and recur-rent neural networks in capturing the underlying features within the driving sequence data. We also provide insights on their strengths and limitations. Our qualitative and quantitative results demonstrate that a temporal convolution-based network can outperform recurrent architectures while reducing computational complexity by a factor of 5.6.

The advent of in-vehicle networking systems as well as state-of-the-art sensors and communication technologies have facilitated the collection of large volume and almost real-time data on vehicles and drivers, thus opening up future possibilities. Processing and analyzing this data provides unprecedented opportunities to offer remarkable insights and solutions for driving behavior analysis (DBA). Characterizing driving behavior plays a key role in a variety of research areas such as traffic safety, the development of automated vehicles, energy and fuel management, risk assessment, and driver identification and profiling. Advances in DBA-based driver inattention or drunk driver detection can help reduce fatal car crashes, and understanding the driving style (e.g. eco-friendly or aggressive) of drivers can contribute to fuel management and risk assessment of the drivers. These facts have led to a growing interest in addressing DBA challenges. This paper aims to present the state-of-the-art methodologies for DBA and provide a clear roadmap about the main current and future trends in DBA. To this end, we propose categorizing the current research on driving behavior based on the types of data employed for the analysis, the ultimate goals of the analysis, and the techniques based on which the driving data are modeled. We provide an overview of different data resources and available datasets for DBA. Moreover, we discuss the application of DBA along with the key research challenges in this field and potential future directions.

Driver identification has shown sustainable development in recent years in a wide variety of applications including but not limited to security, personalization, fleet management, insurance telematics, or ride-hailing. However, the current progress suffers from several challenges such as costly data collections and the need for a huge amount of data from each individual for both driver identification and impostor detection. Therefore, more novel and efficient solutions are required to mitigate the existing challenges. In this paper, we address driver identification and impostor detection tasks using driving behavior analysis of the drivers. We design a deep learning-based system architecture that analyzes windows of 30 seconds of driving data to capture the unique underlying characteristics of the individuals steering behavior based on which it further distinguishes the drivers. We also develop a novel strategy to tackle driver verification and impostor detection tasks based on the combination of the proposed system architecture and Siamese networks concepts. We map the steering behavior of the drivers into latent representations which can be later used to train a similarity function. The performance of the proposed systems is tested over a real-world dataset of 95 drivers. The evaluation results indicate that our system outperforms well-established benchmarks and baseline methodologies.

Accurate and reliable prediction of future motions of the nearby agents and effective environment understanding will contribute to high-quality and meticulous path planning for the automated vehicles under uncertainty and guarantee traffic safety for future real-world deployments. This task becomes more challenging in highly dynamic and complex scenarios such as unsignalized intersections where no lights exist to control vehicles behavior, or there are not multiple lines for the vehicles to anticipate drivers’ future intentions based on the lane in which they are driving. In this study, we introduce a novel deep learning-based methodology to anticipate vehicles path at unsignalized intersections. The method provides multimodal outputs to take into account the inherited uncertainty and multimodality nature of vehicles behavior. Our proposed model works based on dilated convolutional networks in combination with a mixture density layer. We then cluster various existing mixes into possible paths that are ranked based on probability. We assess the performance and generalization capability of our vehicle path prediction model using several metrics over a large naturalistic dataset containing more than 23800 vehicle trajectories. The obtained results reveal the higher performance of our path prediction approach compared with several baselines and benchmarks.

As a major type of driver assistance technologies, automated warning systems provide drivers and vulnerable road users with safety. These systems, such as forward collision warnings, can detect potential risks nearby and alert the drivers. One shortcoming of such warning systems is that their effectiveness and capability depend on the information collected from sensors existing in a single vehicle, which can be highly limited in the presence of occlusion, leading to irreversible consequences. To overcome this shortcoming, in this paper, we benefit from the vehicular sensing and communication technologies to propose a novel Internet-of-vehicles (IoV) powered framework for defensive driving warning, in which a vehicle can take advantage of other vehicles sensing data through V2V communications. We further evaluate the introduced framework in cyclist protection system scenarios. Simulation results demonstrate how the proposed IoV-based framework can improve warning systems by providing increased situational awareness.

High-level scene understanding and situational awareness are fundamental for autonomous vehicles before being widely used on public roads in a thoroughly efficient and safe manner. These tasks involve not only perceiving the current surrounding states but also predicting the future behavior of nearby human-driven vehicles. A large amount of vehicular sensing data can be collected using sensors and networking systems in vehicles. Moreover, with the advent of Dedicated Short Range Communication, vehicles can further transfer intention information to surrounding vehicles. However, real-time intention inference of nearby drivers is still challenging. Our primary focus is to present a novel deep learning-based approach to predict drivers driving behavior at unsignalized T-junctions. We use temporal convolutional networks to analyze sequences of trajectory data for a vehicle approaching the T-junction and classify the maneuver seconds before actual maneuver occurrence. The cross-validation results demonstrate the efficiency of the proposed methodology.

Driver identification plays a pivotal role in the design of advanced driver assistant systems. The continued development of in-vehicle networking systems, CAN-bus technology, and the ubiquitous presence of smartphones as well as the broad range of state-of-the-art sensors have paved the way to collect huge amount of data from both vehicles and drivers. This paper addresses the necessity of having a large volume of labeled data for driver identification and presents a novel methodology to identify drivers based on their driving behavior analysis. The proposed architecture benefits from triplet loss training for driving time series in an unsupervised approach. An encoder architecture based on exponentially dilated causal convolutions is employed to obtain the representations. An SVM classifier is then trained on top of the representations to predict the person behind the wheel. The experiment results demonstrated higher performance of the proposed methodology when compared to benchmark methods.

Modern cars can collect several hundreds of sensor data through the controller area network (CAN) bus technology that provides almost real-time information about the car, the surrounding environment, and the driver. These data can be later processed and analyzed to offer efficient solutions and insights for human behavior analysis and further applied in a variety of fields such as accident prevention, driver identification, driving models design, and vehicle energy consumption. By analyzing and identifying unique driving behavior, we can distinguish drivers, which can be helpful in driver profiling and security of the cars (anti-theft systems). In this paper, we evaluate the performance of data-driven end-to-end models designed for driving behavior identification. We present a critical analysis of the principles considered in designing the models. Moreover, various data-driven deep learning and machine learning models are implemented and the cross-validation results are presented employing the naturalistic driving dataset.

OBJECTIVE: Automatic text summarization offers an efficient solution to access the ever-growing amounts of both scientific and clinical literature in the biomedical domain by summarizing the source documents while maintaining their most informative contents. In this paper, we propose a novel graph-based summarization method that takes advantage of the domain-specific knowledge and a well-established data mining technique called frequent itemset mining. METHODS: Our summarizer exploits the Unified Medical Language System (UMLS) to construct a concept-based model of the source document and mapping the document to the concepts. Then, it discovers frequent itemsets to take the correlations among multiple concepts into account. The method uses these correlations to propose a similarity function based on which a represented graph is constructed. The summarizer then employs a minimum spanning tree based clustering algorithm to discover various subthemes of the document. Eventually, it generates the final summary by selecting the most informative and relative sentences from all subthemes within the text. RESULTS: We perform an automatic evaluation over a large number of summaries using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics. The results demonstrate that the proposed summarization system outperforms various baselines and benchmark approaches. CONCLUSION: The carried out research suggests that the incorporation of domain-specific knowledge and frequent itemset mining equips the summarization system in a better way to address the informativeness measurement of the sentences. Moreover, clustering the graph nodes (sentences) can enable the summarizer to target different main subthemes of a source document efficiently. The evaluation results show that the proposed approach can significantly improve the performance of the summarization systems in the biomedical domain.