Traffic Light (TL) state recognition is a complex yet crucial technology for many applications, including self-driving vehicles and Advanced Driver Assistance Systems (ADAS). Self-driving vehicle systems rely on detected traffic light states for operation planning, while ADAS uses them to provide timely alerts to drivers based on signal changes. Most existing TL detection methods rely on a single modality, such as images or videos, which limits their generalization and robustness. To address these challenges, we propose a multimodal framework, the Multimodal Detection Model (DetMM), for robust traffic light state detection. DetMM leverages Faster R-CNN’s efficiency and accuracy in TL localization, CLIP’s zero-shot object recognition capability, novel context-rich prompts, and decision functions designed for both simple and context-rich prompt-based recognition. Extensive experiments were conducted to evaluate DetMM using three traffic light datasets, without fine-tuning CLIP. Furthermore, ablation studies were performed to assess the effectiveness and robustness of various components within DetMM.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

In this paper, we present a machine learning approach that generates a system of driver-centered and roadway type-specific deep neural network models for accurate vehicle speed prediction (VSP) in short and long terms in a future horizon. This research focuses on addressing the following issues, proper attributes and window sizes with respect to the lengths of prediction horizons, impacts of roadway types to neural network models, importance of statistical traffic data, and effectiveness of three deep neural network frameworks applied to the short- and long-term VSP problem. Extensive experiments are conducted using the naturalistic driving trips collected from three different drivers on two different routes covering seven different roadway types. Our research results show that the deep learning models structured around roadway types, trained with driver-centered data using optimal window sizes of historical temporal features, including vehicle speed and traffic flow, are effective in both short- and long-term predictions.

Driving Maneuver Detection (DMD) is an important component in ADAS(Advanced Driver Assistance Systems). It provides information about driving maneuvers that can potentially lead to traffic accidents. This paper presents a DMD system that builds on deep learning models developed for extracting driver attention features and driver face shift features, and a Long Short-Term Memory (LSTM) based neural network designed to learn dependencies of maneuvers in a time period. We show through experiments that the proposed system is capable of learning the latent features of the five different classes of driving maneuvers, i.e. left turn, right turn, left lane change, right lane change, and driving straight, and the innovative use of the combined features, i.e. driver attention features, driver face shift and vehicle signals that makes the DMD system to perform significantly superior to a number of traditional methods on a naturalistic driving data set containing over 3100 maneuvers recorded from 20 different drivers.

Personalized session-based recommendation systems aim to predict the next items that a user would be interested in based on previously recorded user interactions. The task contains two connotations. Firstly, “personalized” indicates the recommendation generating process does not involve other users. A recommendation system is expected to capture the preference patterns of the target user only. Secondly, “session-based” signifies that the preference patterns are expressed by sequences of items or interactions. Personalized session-based recommendation technologies are of high practical values in many application areas as explosive volumes of information available for customers. In this research we model user-item sessions using graphs and propose a novel graph neural network based model, namely, Personalized Session-based Recommendation using Graph Attention Networks (PSR-GAT). The PSR-GAT makes preferred item predictions by exploiting not only instantaneous preference patterns in the current session, but also more generic preference patterns in users' historical records. The proposed PSR-GAT is built on an attention mechanism basis with a consideration of item transition linkage. PSR-GAT does not have assumptions of preference expressions in sessions. Extensive experiments are conducted to evaluate the PSR-GAT model using three datasets, one is a vehicle infotainment dataset provided by the Ford Motor Company, and the other two are publicly accessible datasets. The performances of the PSR-GAT are compared with seven state-of-art methods for session-based recommendations. The results attest to the effectiveness and advantages of the PSR-GAT in the personalized session-based next interested items prediction.

In a multi-camera system, matching the same pedestrian across different camera views is a challenging problem. Pedestrian detection and ReIDentification (ReID) plays an important role in preventing traffic accidents involving pedestrians, for both conventional and autonomous vehicles. To the best of our knowledge, there is no existing work which addresses the problem of pedestrian ReID in a typical vehicle setting, where the vehicle is equipped with a 360° surround-view with fisheye cameras. In this paper, we propose a deep learning system, Seeing Pedestrians in Surround-View Moving Cameras (SPinSVMC), that consists of a Single Camera Detection and Tracking (Single-Cam-D&T) module and Two Cameras ReID (2Cam-ReID) module applied to multi-camera views. The Single-Cam module uses a YOLOv3 model to detect the pedestrians in single camera view videos, and a model that combines OSnet with DeepSORT to track pedestrians and assign an ID for each detected pedestrian. Both models were adapted to the fisheye images through transfer learning processes. The 2Cam-ReID module consists of a camera constraint model and a pedestrian ReID model developed for tracking pedestrians in images captured by a pair of adjacent cameras with overlapping views and assigning unique IDs to the pedestrians captured by both cameras. We evaluated both models on a real-world traffic dataset captured by surround-view fisheye cameras mounted on top of a vehicle. Our experiments demonstrate that the proposed two modules in SPinSVMC achieve high accuracy in both pedestrian detection, tracking and ReID in fisheye image domains.

Energy optimization for plug-in hybrid electric vehicles (PHEVs) is a challenging problem due to the system complexity and many physical and operational constraints in PHEVs. In this paper, we present a Q-learning-based in-vehicle learning system that is free of physical models and can robustly converge to an optimal energy control solution. The proposed machine learning algorithms combine neuro-dynamic programming (NDP) with future trip information to effectively estimate the expected future energy cost (expected cost-to-go) for a given vehicle state and control actions. The convergences of these learning algorithms were demonstrated on both fixed and randomly selected drive cycles. Based on the characteristics of these learning algorithms, we propose a two-stage deployment solution for PHEV power management applications. Furthermore, we introduce a new initialization strategy, which combines the optimal learning with a properly selected penalty function. This initialization scheme can reduce the learning convergence time by 70%, which is a significant improvement for in-vehicle implementation efficiency. Finally, we develop a neural network (NN) for predicting battery state-of-charge (SoC), rendering the proposed power management controller completely free of physical models.

Driving Maneuver early Detection (DMD) is one of the most important tasks in Advanced Driver Assistance Systems (ADAS), it provides the early notification necessary for ADAS to predict dangerous circumstances and take appropriate actions. The end-to-end architectures such as Recurrent Neural Networks (RNNs) take advantage of deep networks to automatically learn non-linear discriminative features, which significantly boost the performances of DMD systems. However, due to the large number of parameters in the deep architectures, learning effective discriminative features requires millions of labeled images. Moreover the discriminative features are generally meaningless to human being, which makes the diagnose of end-to-end architectures extremely difficult. In this paper, we propose a novel DMD system, denoted as Attention-driven Driving Maneuver Detection system (ADMD), which uses drivers' attention as an intermediate concept (hint) to explore and understand the causal relation between driving surroundings and driving maneuvers. In the training phase, ADMD distills the knowledge from drivers' attention prediction model to provide initial search areas for learning effective features, which minimizes the labeled data requirements for training deep architectures and results in a fast optimization process. We compared the performances of ADMD with the state-of-the-art methods on 1953 miles (37 hours) of natural driving data collected from 7 drivers. The experimental results shows ADMD is capable of achieving better performances, which are mostly attributed to the proposed novel learning mechanism used to obtain meaningful interpretations of the driving surroundings that are closely related to the driver's intend maneuvers.

Detection of driver stress is an important component in many ADASs (Advanced Driver-Assistance Systems), and a challenging problem when it is applied to real-world driving environment. In this paper, we present a convolutional neural network (CNN) designed to detect driver’s stress levels with four physiological signals, i.e., heart rate, heart rate variability, breathing rate, and galvanic skin response. The proposed model is shift invariant and is capable of handling the imbalanced data set issue. The performances of the proposed models are evaluated using real-world driving data in three different types of driver stress detection tasks, i.e., the single-driver, the cohorts of drivers, and all-driver stress detection task respectively. The experimental results demonstrate that the proposed model is capable of reliably detecting the driver’s stress levels. More importantly we demonstrate that a model can be trained on the data collected from drivers with similar cognitive capabilities and then generalized to new drivers with similar cognitive capabilities for stress detection.

Human mental workload perception is an important problem with multiple applications in the areas of industry, transportation, military and medical contexts. In this paper, we present a deep learning model developed for detecting sequential driver mental workload using physiological signals. We modify a fully convolutional U-Net in order to learn from multiple temporal physiological signals as inputs. We also demonstrate that the element-wise cross-entropy loss function, which has been adopted by many other deep learning architectures for classification or segmentation purposes, is not sufficient for the sequential workload segmentation because it treats each data point as an independent classification object while ignoring the neighboring relation between the points. We introduce a new loss function that is sensitive to local continuity property in workload sequence. The proposed method is evaluated on two sets of data collected by wearable sensors to record multiple physiological signals. The effectiveness of an introduced scalar used to control the influence of the continuity-aware loss is investigated. Our experiments show that the proposed U-Net is effective in detecting sequential human mental workload. Furthermore, when the proposed continuity-aware loss is combined with the element-wise binary cross-entropy loss with the properly selected controlling scalar value, the quality of the predicted mental workload segments is significantly improved.

Objects in the periphery of fisheye images can become extremely distorted. This distortion can cause false positives and missed detections for automated object detection systems. This is problematic, not only for systems which have been trained on perspective images, but also for those that have been explicitly trained on fisheye data. In this paper we propose a new cost function for training object detectors on fisheye images. We model fisheye image distortion as an imbalanced domain problem and develop a domain association loss function to approach it with deep learning. We define separate domains based on the level of distortion within the image plane and propose a new objective function, inspired by the recently introduced focal loss for object detection, which we call a spatial focal loss. Our proposed loss incorporates a domain-modulating term which re-weights samples from different domains to encourage the learning of domain-invariant features. We implement spatial focal loss function in the YOLOv2 architecture and evaluate it on the task of pedestrian detection in a fisheye dataset captured by a 360 camera system mounted on a moving vehicle and labeled with over 11,000 pedestrian instances. Our experiments demonstrate that spatial focal loss can improve model performance in the highly distorted image periphery versus existing loss functions, including focal loss, without sacrificing performance in the less distorted image center, with no adaptations to network architectures required. By analyzing the locations of missed detections, we show further evidence that our loss function can improve the learning of domain-invariant features.

This paper presents a parametric low-dimensional (LD) representation learning method that allows to reconstruct high-dimensional (HD) input vectors in an unsupervised manner. Under the assumption that the HD data and its LD representation share the same or similar local sparse structure, the proposed method achieves reconstructible dimensionality reduction via jointly learning dictionaries in both the original HD data space and its LD representation space. By regarding the sparse representation as a smooth function with respect to a specific dictionary, we construct an encoding-decoding block for learning LD representations from sparse coefficients of HD data. It is expected that this learning process preserves the desirable structure of HD data in the LD representation space, and simultaneously allows a reliable reconstruction from the LD space back to the original HD space. In addition, the proposed single layer encoding-decoding block can be easily extended to deep learning structures. Numerical experiments on both synthetic data sets and real images show that the proposed method achieves strongly competitive and robust performance in data DR, reconstruction, and synthesis, even on heavily corrupted data. The proposed method can be used as an alternative approach to compressive sensing (CS); however, it can outperform the traditional CS methods in: 1) task-driven learning problems, such as 2-D/3-D data visualization, and 2) data reconstruction at a lower dimensional space.

Driving maneuver detection is one of the most challenging tasks in Advanced Driver Assistance Systems (ADAS). Research has shown that the early notification of improper driving maneuvers is helpful to avoid fatalities and serious accidents. In this paper, we introduce a driver maneuvering detection (DMD) system. The DMD system contains three major computational components, distance based representation of driving context, combined features of vehicle trajectory and VGG-19 network features extracted from the video images of vehicle front view, and a Long Short-Term Memory (LSTM)-based neural network model to learn sequence knowledge in driving maneuvering events. We show through experiments that the DMD system is capable of learning the latent features of five different classes of driving maneuvers and achieving significantly better performance than traditional classification methods on real-world driving trips.

Weather recognition is of great significance in traffic safety, environment and meteorology. However, the visual image features of the weather are highly abstractive, and the traditional method of weather recognition has a high computational complexity and low accuracy. In this paper, the edge deterioration phenomenon is introduced in convolution neural network (CNN) to solve the problem that common CNN cannot distinguish the specific weather. The proposed method used Mask R-CNN to extract the regions of interest including the foreground and foreground edges in the image, and superimposed them into the same-scale matrix and then input them into the network for classification. Outdoor traffic image experiments showed that this method can effectively improve the classification accuracy of the four weather conditions (sunny, foggy, rainy and snowy).

Recently, sparse representation (SR) over a redundant dictionary has become a popular way of representing the data. It has been verified as an efficient and useful tool to promote the discrimination between signals. This work develops a joint learning approach to find the low dimensional discriminative features for high dimensional data. To avoid the high computational cost of direct sparse coding on large scale input data, we first learn SR in an orthogonal projected space over a task-driven sparsifying dictionary. We then exploit the discriminative projection on SR. The whole learning process is treated as an optimization problem of trace quotient maximization, which involves an orthogonal projection on original data space, a dictionary and a discriminative projection on sparse codes. The related cost function is well defined on a product manifold of the Stiefel manifold, the Oblique manifold and the Grassmann manifold. Finally, we employ a stochastic gradient descent algorithm on the smooth product manifold to maximize the cost function. Our numerical experiments on visual recognition demonstrate the effectiveness of the proposed algorithm, in comparison with the state of the arts.

Time series data are ubiquitous and are of importance in many application problems in engineering, science, medicine, economics and entertainment. Many real world pattern classification problems involve the processing and analysis of multiple variables in the temporal domain. These types of problems are referred to as Multivariate Time Series (MTS) problems. In many real-world applications, an MTS problem can involve a large number of signals, and require algorithms to select signals and extract temporal and spatial features from them. In this paper, we present an innovative convolutional neural network, MTS-DeepNet that is specially designed for MTS pattern classification. The system integrates signal and feature selections with MTS pattern classification in one learning framework. MTS-DeepNet is applied to a real-world problem, namely predicting driver lane departure based on driver's physiological signals. Our experimental results showed that, in comparison to a multi-layer neural network trained with the backpropagation algorithm, MTS-DeepNet gave better prediction accuracy.

Modern automobiles are increasingly complicated machines with an ever-increasing number of features. Understanding how these features work, when to use them, and in general how to make the best use of your vehicle is not a simple task. This research presents an evolving fuzzy system that personalizes the fuzzy membership functions based on individual driving habits. The system was successfully applied to estimate the likelihood of a driver using cruise control based on past usage preferences, current context, and recent driving history. Experimental results show that the proposed fuzzy system can learn the membership functions adaptively according to the driving behavior, and predicts the cruise control usage with high confidence.

Driver yawning detection is one of the key technologies used in driver fatigue monitoring systems. Real-time driver yawning detection is a very challenging problem due to the dynamics in driver's movements and lighting conditions. In this paper, we present a yawning detection system that consists of a face detector, a nose detector, a nose tracker and a yawning detector. Deep learning algorithms are developed for detecting driver face area and nose location. A nose tracking algorithm that combines Kalman filter with a dedicated open-source TLD (Track-Learning-Detection) tracker is developed to generate robust tracking results under dynamic driving conditions. Finally a neural network is developed for yawning detection based on the features including nose tracking confidence value, gradient features around corners of mouth and face motion features. Experiments are conducted on real-world driving data, and results show that the deep convolutional networks can generate a satisfactory classification result for detecting driver's face and nose when compared with other pattern classification methods, and the proposed yawning detection system is effective in real-time detection of driver's yawning states.

This paper presents an automatic text document categorization system, HPHR. HPHR contains high precision, high recall and noise-filtered text categorization models. The text categorization models are generated through a suite of machine learning algorithms, a fast clustering algorithm that efficiently and effectively group documents into subcategories, and a text category generation algorithm that automatically generates text subcategories that represent high precision, high recall and noise-filtered text categorization models from a given set of training documents. The HPHR system was evaluated on documents drawn from two different applications, vehicle fault diagnostic documents, which are in a form of unstructured and verbatim text descriptions, and Reuters corpus. The performance of the proposed system, HPHR, on both document collections showed superiority over the systems commonly used in text document categorization.

This paper presents our research in building a virtual humidity sensor using recurrent Neural Networks. Recurrent Neural Networks are promising methods for the prediction of time series because they provide feedback connections from hidden layer to its inputs and, therefore, can store temporal information learned from previous time steps. This study applies Elman Recurrent Neural Network (ERNN) to forecast the specific humidity from three weather stations. In addition, this study examines the feasibility of applying ERNN in time series forecasting by comparing it with multilayer perceptron network. The experiment results indicate that ERNN is a promising alternative to specific humidity forecasting.

Real-time safety aware navigation of an intelligent vehicle is one of the major challenges in intelligent vehicle systems. Many studies have been focused on the obstacle avoidance to prevent an intelligent vehicle from approaching obstacles "too close" or "too far", but difficult to obtain an optimal trajectory. In this paper, a novel biologically inspired neural network methodology with safety consideration to realtime collision-free navigation of an intelligent vehicle with safety consideration in a non-stationary environment is proposed. The real-time vehicle trajectory is planned through the varying neural activity landscape, which represents the dynamic environment, in conjunction of a safety aware navigation algorithm. The proposed model for intelligent vehicle trajectory planning with safety consideration is capable of planning a real-time "comfortable" trajectory by overcoming the either "too close" or "too far" shortcoming. Simulation results are presented to demonstrate the effectiveness and efficiency of the proposed methodology that performs safer collision-free navigation of an intelligent vehicle.

Speed profile prediction on ramps is a challenging problem because speed changes on ramps involve complicated lane maneuvering and frequent acceleration or deceleration depending on geometry of the ramp and traffic volumes. Ramps can be categorized into three groups based on their interconnection of freeway: freeway entering ramps, freeway exit ramps, and inter freeway ramps. However, different geographical shapes of ramps within the same category cause different speed profile distributions. To predict speed profile on any ramp types, we proposed an Intelligent Trip Modeling on Ramp (ITMR) System that consists of a ramp classification method based on the decision tree and speed profile prediction neural networks. The proposed ITMR takes inputs from geographical data on the route and also the personal driving pattern extracted from the knowledge base built with the individual historical driving data. Experimental results show that the proposed system learned dynamic ramp speed changes very well to provide accurate prediction results on multiple freeway entering ramps, exit ramps and inter freeway ramps.

Accurate prediction of the traffic information in real time such as flow, density, speed, and travel time has important applications in many areas, including intelligent traffic control systems, optimizing vehicle operations, and the routing selection for individual drivers on the road. This is also a challenging problem due to dynamic changes of traffic states by many uncertain factors along a traveling route. In this paper, we present an Intelligent Trip Modeling System (ITMS) that was developed using machine learning to predict the traveling speed profile for a selected route based on the traffic information available at the trip starting time. The ITMS contains neural networks to predict short-term traffic speed based on the traveling day of the week, the traffic congestion levels at the sensor locations along the route, and the traveling time and distances to reach individual sensor locations. The ITMS was trained and evaluated by using ten months of traffic data provided by the California Freeway Performance Measurement System along a California Interstate I-405 route that is 26 mi long and contains 52 traffic sensors. The ITMS was also evaluated by the traffic data acquired from a 32-mi-long freeway section in the state of Michigan. Experimental results show that the proposed system, i.e., ITMS, has the capability of providing accurate predictions of dynamic traffic changes and traveling speed at the beginning of a trip and can generalize well to prediction of speed profiles on the freeway routes other than the routes the system was trained on.

This paper presents our research in text mining for discovering important engineering fault diagnostic knowledge from unstructured and verbatim text descriptions. In particular we focus on developing machine learning algorithms for detecting documents that contain descriptions of systematic failures and root causes to the faults. We developed a machine algorithm based on entropy analysis to extract an A-word list, a list of words that are important to characterize the documents of interests, a vector space model to represent features of important documents, and a constraint based k-means clustering algorithm to generate high purity clusters for use in detecting important documents. We applied the algorithms to automotive diagnostic text data, which are unstructured and verbatim descriptions by customers and technicians that contain many typos and self-invented terms. We were able to reduce a list of 2183 words to a list of 137 important words. The classification system generated by these machine learning algorithms showed high recall and accuracy in detecting important diagnostic descriptions.

Text description of engineering diagnoses recorded during and after vehicle repair process plays an important role in root cause analyzing and vehicle maintenance. The fact that such text is unstructured, lack of grammar, has a lot of spelling errors and a large amount of self-invented domain specific terminologies introduces challenges and difficulties for automatic information retrieving and categorization. This paper presents our research in text mining in vehicle diagnostic applications. Specifically, an automatic typo correction system is proposed and implemented. We build multiple knowledge bases to detect and correct typos, and a neural network classifier to select good candidates for correcting typos. Experiment results show that our system outperforms state-of-art spell checking systems.

Accurate prediction of traffic information such as flow, density, speed, and travel time is an important component for traffic control systems and optimizing vehicle operation. Prediction of an individual speed profile on an urban network is a challenging problem because traffic flow on urban routes is frequently interrupted and delayed by traffic lights, stop signs, and intersections. In this paper, we present an Intelligent Speed Profile Prediction on Urban Traffic Network (ISPP_UTN) that can predict a speed profile of a selected urban route with available traffic information at the trip starting time. ISPP_UTN consists of four speed prediction Neural Networks (NNs) that can predict speed in different traffic areas. ISPP_UTN takes inputs from three different categories of traffic information such as the historical individual driving data, geographical information, and traffic pattern data. Experimental results show that the proposed algorithm gave good prediction results on real traffic data and the predicted speed profiles are close to the real recorded speed profiles.

This paper presents our research in vehicle energy optimization for a low-cost HEV power system that only allows the control of engine on/off and driving at three different speed limits. We present algorithms for modeling vehicle energy flow and optimization and machine learning of optimal control settings generated by Dynamic Programming on real-world drive cycles, and an intelligent energy controller designed for online energy control. Experimental results show the intelligent controller has the capability of 11% fuel saving.

This paper presents our research in battery State of Charge (SOC) estimation for intelligent battery management. Our research focus is to investigate online dynamic SOC estimation using a combination of Kalman filtering and a neural network. First, we developed a method to model battery hysteresis effects using Extended Kalman Filter (EKF). Secondly, we designed a SOC estimation model, NN-EKF model, that incorporates the estimation made by the EKF into a neural network. The proposed methods have been evaluated using real data acquired from two different batteries, a lithium-ion battery U1-12XP and a NiMH battery with 1.2 V and 3.4 Ah. Our experiments show that our EKF method developed to model battery hysteresis based on separated charge and discharge Open Circuit Voltage (OCV) curves gave the top performances in estimating SOC when compared with other advanced methods. Secondly, the NN-EKF model for SOC estimation gave the best SOC estimation with and without temperature data.

Prediction of the traffic information such as flow, density, speed, and travel time is important for traffic control systems, optimizing vehicle operations, and the individual driver. Prediction of future traffic information is a challenging problem due to many dynamic contributing factors. In this paper, various methodologies for traffic information prediction are investigated. We present a speed prediction algorithm, NNTM-SP (Neural Network Traffic Modeling-Speed Prediction) that trained with the historical traffic data and is capable of predicting the vehicle speed profile with the current traffic information. Experimental results show that the proposed algorithm gave good prediction results on real traffic data and the predicted speed profile shows that NNTM-SP correctly predicts the dynamic traffic changes.

This paper presents the methodologies developed for solving IJCNN 2011's Ford Challenge II problem, where the driver's alertness is to be detected employing physiological, environmental and vehicular data acquired during driving. The solution is based on a thorough four-fold framework consisting of temporal processing, feature creation and extraction, and the training and ensemble of several learning systems, such as neural networks, random forest, support vector machine, trained from diverse features. The selection of input features to a learning machine has always been critique on signal classification. In our approach, the employment of Bayesian network filtered out a set of features and has been proved by the ensemble to be effective. The ensemble technique enhanced the performance of individual systems dramatically. The performance acquired on 30% of the test samples reached an accuracy of 78.34%. These results are significant for a real-world vehicular problem and we are quite confident this solution will become one of the top ones on the competition test data.

This paper presents a multi-agent system(MAS_VFD&HM) developed for complex vehicle fault diagnosis and health monitoring. The MAS_VFD&HM consists of signal diagnostic agents, special case agents, and a vehicle diagnostic/monitoring agent. A signal agent is responsible for the fault diagnosis or monitoring of one particular signal using either a single signal or multiple signals depending on the complexity of signal faults. Special case agents are those trained to detect specific component faults. All these agents are autonomous and report their results to the Vehicle System Agent. A computational framework is presented for agent learning and agent operation. The proposed MAS_VFD&HM is scalable, versatile, and has the capability of dealing complex problems such as multiple faults in a vehicle system. Although our focus was on the automotive diagnostics, the proposed MAS_VFD&HM is applicable to complex engineering diagnostic problems beyond vehicles.

Time series data classification is important in many applications. Learning temporal knowledge in time series data is challenging. In this paper we propose a novel machine learning algorithm, Feature Ensemble (FE), to learn effective subsequences of signal features distributed over time series data streams. Both the FE learning and the FE classification have been applied to an application problem. Our empirical results strongly suggest that FE learning is an effective technique for time series data classification.

Power management for the Hybrid Electric Vehicle (HEV) is a challenging problem because of the dual-power-source nature of HEV design and implementation. In this paper, we present an Intelligent Power Controller, UMD_IPC, trained with a machine learning approach to provide optimal power flow for in-vehicle operations. The UMD_IPC is implemented in a HEV model provided by PSAT simulation environment, and its performances on three drive cycles are close to the optimal results generated by Dynamic Programming.

This paper presents a novel computational framework, BNECC, Bayesian Network Enhanced Cascades Classification, for on-road vehicle detection. The objective of this research is to combine the texture features with geometric features of objects in an object recognition system. BNECC consists of two tiers of classifiers. The first tier is a Cascade of Boosted Ensembles (CoBE) classifiers trained on object texture features. The second tier is a Bayesian network trained using features of vehicle location, size and the confidence values generated by all the stage classifiers in CoBE. Experiment results on real world data show that proposed BNECC framework is effective in reducing false alarms significantly while keeping the detection rate high.

This paper presents a two-step ensemble approach for vehicle fault diagnostics, an ensemble selection algorithm, BFES, and an analog Bayesian ensemble decision function, A-Bayesian-Entropy. We show through experiments that a neural network ensemble designed and trained by the proposed methodology, and selected by BFES with A-Bayesian-Entropy as the ensemble decision function can generalize well to vehicle models that are different from the vehicles used to generate training data.

In this paper we present our research work in traffic sign detection and classification. Specifically we present a set of asymmetric Haar-like features that will be shown to be effective in reducing false alarm rates for traffic sign detection, and a robust multi-class traffic sign detection and classification system built based upon the stage-by-stage performance analysis of individual traffic sign detectors trained using Adaboost.

We present our research in optimal power management for a generic vehicle power system that has multiple power sources using machine learning and fuzzy logic. A machine learning algorithm, LOPPS, has been developed to learn about optimal power source combinations with respect to minimum power loss for all possible load requests and various system power states. The results generated by the LOPPS are used to build a fuzzy power controller (FPC). FPC is integrated into a simulation program implemented by using a generic simulation software as indicated in reference [22] and is used to dynamically allocate optimal power sources during online drive. The simulation results generated by FPC show that the proposed machine learning algorithm combined with fuzzy logic is a promising technology for vehicle power management.

This paper presents a pedestrian model built collectively on a group of strong local convex shape descriptors. The pedestrian model captures the most important features of a pedestrian: head, body contour, arms, legs and crotch, and is robust to variances in appearances and partial occlusions. For an image set of 2571 pedestrians and 4369 car and background images, the pedestrian recognition system, which was built upon the proposed pedestrian model, gave a recognition rate of 98.8% with a false positive rate of 1.56%. Furthermore, the pedestrian recognition requires a very small set of prototypes of pedestrians and non-pedestrians.

Vehicle power management has been an active research area in the past decade, and has intensified recently by the emergence of hybrid electric vehicle technologies. Research has shown that driving style and environment have strong influence over fuel consumption and emissions. In order to incorporate this type of knowledge into vehicle power management, an intelligent system has to be developed to predict the current traffic conditions. This paper presents our research in neural learning for predicting the driving environment such as road types and traffic congestions. We developed a prediction model, an effective set of features to characterize different types of roadways, and a neural network trained for online prediction of roadway types and traffic congestion levels. This prediction model was then used in conjunction with a power management strategy in a conventional (non-hybrid) vehicle. The benefits of having the predicted drive cycle available are demonstrated through simulation.

This paper presents a machine learning approach to the efficient vehicle power management and an intelligent power controller (IPC) that applies the learnt knowledge about the optimal power control parameters specific to specific road types and traffic congestion levels to online vehicle power control. The IPC uses a neural network for online prediction of roadway types and traffic congestion levels. The IPC and the prediction model have been implemented in a conventional (non-hybrid) vehicle model for online vehicle power control in a simulation program. The benefits of the IPC combined with the predicted drive cycle are demonstrated through simulation. Experiment results show that the IPC gives close to optimal performances.

This paper presents our research in incremental learning for text document classification. Incremental learning is important in text document classification since many applications have huge amount of training data, and training documents become available through time. We propose an incremental learning framework, ILTC(Incremental Learning of Text Classification) that involves the learning of features of text classes followed by an incremental Perceptron learning process. ILTC has the capabilities of incremental learning of new feature dimensions as well as new document classes. We applied the ILTC to a classification system of diagnostic text documents. The experiment results demonstrate that ILTC was able to incrementally learn new knowledge from newly available training data without either referring to the older training data or forgetting the already learnt knowledge.

This paper presents our research in multi-class pattern learning from imbalanced data. In many real world applications, the data among different pattern classes are imbalanced; some classes may have far more training data than the others. Typically a neural network classifier has troubles to learn from the imbalanced data distribution among different pattern classes. In this paper we propose a new pattern classification algorithm, One-Against-Higher-Order (OAHO), that effectively learn multi-class patterns from the imbalanced data, and a theoretical analysis of data imbalance problem related to other popular multi-class pattern classification approaches. We have conducted experiments on the two highly imbalanced data sets posted at the UCI site, and the results show that the neural network system trained with the proposed OAHO algorithm gives better performances on minority pattern classes over the neural network systems trained with the two other popular multi-class classification methods: OAO and OAA.

This paper presents our research in text document categorization using neural networks. In text document categorization typically the feature spaces have high dimensions, training data are large and the categories are many. A single neural network is often not sufficient to provide accurate classification or efficient training. We present a hierarchical neural network system and a categorical neural network system for document classification. We will show with an application in engineering diagnostic document categorization that the two proposed systems are more effective and efficient than a single neural network, and the hierarchical neural network system gives the highest accuracy in document categorization.

An incremental class learning system for support vector machine (SVM) is presented for learning new knowledge from newly available data without forgetting the existing knowledge. We present algorithms for knowledge domain description, new knowledge detection, and incremental learning of new class knowledge. We have applied the incremental learning system to a data set provided by the UCI machine learning Web site, and the results show that the proposed SVM incremental class learning system is quite effective.

We present a speaker identification system that uses synchronized speech signals and lip features. We developed an algorithm that automatically extracts lip areas from speaker images, and a neural network system that integrates the two different types of signals to give accurate identification of speakers. We show that the proposed system gives better performances than the systems that use only speech or lip features in both text dependant and text independent speaker identification applications.

This paper presents an intelligent system for protein secondary structure prediction. The system consists of three pairwisely trained neural networks and a Bayesian inference function applied to the neural network outputs for accurate prediction. We tested our system on two well-known protein data set drawn from PDB, our system showed top performances on both data sets.

This paper describes a neural network system that automatically detects whether a human head exists in a given image. We focus our research in the first two levels of head detections. At the first level, it extracts candidates of a head using range information, motion clue and 3D spherical shape. At the second level, the system uses multiple visual modalities including gray scale value distribution, shape, motion and range information obtained using a stereo vision system to represent head features. A neural network classifier is used to evaluate the effectiveness of various object features for generating and representing human head. The system is validated on a large collection of images taken from a stereo camera system mounted inside a vehicle. Our experiments show the presented system has an accurate rate over 96%.

Support vector machines have been gaining popularity in the research community of pattern classification. In this paper, we investigate efficient and effective algorithms for training SVMs on large data collections. We decompose SVM learning problem into two stages. At the first stage we developed an algorithm that uses a sequence of small subsets of training data to select the parameters /spl gamma/ and C. At the second stage, we developed an algorithm that generates a reliable set of support vectors using a small subset of the training data. Experiments are conducted on about 850K data samples for automotive engine misfire detection.

This paper presents our research in case-based reasoning (CBR) with application to vehicle fault diagnosis. We have developed a distributed diagnostic agent system, DDAS that detects faults of a device based on signal analysis and machine learning. The CBR techniques presented are used to rind root cause of vehicle faults based on the information provided by the signal agents in DDAS. Two CBR methods are presented, one used directly the diagnostic output from the signal agents and another uses the signal segment features. We present experiments conducted on real vehicle cases collected from auto dealers and the results show that both method are effective in finding root causes of vehicle faults.

In this paper, we describe a fuzzy logic system used in a signal diagnostic agent (SDA) for signal segment fault diagnosis. A SDA is trained to detection the fault of a signal. The SDA provides two levels of decisions, the signal segment level and signal level, using fuzzy logic. At the signal segment level, we developed a fuzzy learning algorithm that learns from good vehicle signals only. The fuzzy learning algorithm was implemented in the framework of a SDA, and the experiments using engine electronic control unit signals are presented and discussed in the paper.

This paper describes a hybrid feature selection algorithm that uses three different statistical measurements to evaluate features: between-class pairwise distance, linear separability, and overlapped feature histogram. The paper presents detailed steps of each feature measurement. The hybrid feature selection algorithm applies the Bayesian EM (expectation maximization) to the features ranked by the three measurements referred to above to select a sub-optimal feature set. The hybrid feature selection algorithm can be used as a preprocessing in a classification system and is independent of the classifier to be used in the subsequence stage. We applied the hybrid feature selection algorithm to select vehicle signal features for fault diagnosis. Our experiments show that the hybrid algorithm provides a sub-optimal feature set that can be used to train a classifier to have very good generalization capability.