Visualization methods have been used to reveal areas of images which influence the decision making of machine learning models, thereby helping to understand and diagnose the learned models and suggest ways to improve their performance. This concept is termed Explainable Artificial Intelligence. In this work, we focus on visualization methods for metric-learning based neural networks. We propose a gradient-weighted extended Excitation Back-Propagation (gweEBP) method that integrates the gradient information during its backpropagation for the accurate investigation of embedding networks. We perform an extensive evaluation of our gweEBP, and seven other visualization methods, on two neural networks, trained for heterogeneous face recognition. The evaluation is performed over two publicly available cross-modality datasets using two evaluation methods termed the “hiding game” and the “inpainting game”. Our experiments showed that the proposed method outperforms the competing methods in both games in most cases. Additionally, our comprehensive study also provides a benchmark for comparing visualization techniques, which may help other researchers develop new techniques and perform comparative studies on them. • We proposed gweEBP for analyzing embedding networks. • Two evaluation protocols were used: the hiding game and the inpainting game. • The inpainting game provided a more reliable metric than the hiding game.

Nondestructive detection methods, based on vibrational spectroscopy, are vitally important in a wide range of applications including industrial chemistry, pharmacy and national defense. Recently, deep learning has been introduced into vibrational spectroscopy showing great potential. Different from images, text, etc. that offer large labeled data sets, vibrational spectroscopic data is very limited, which requires novel concepts beyond transfer and meta learning. To tackle this, we propose a task-enhanced augmentation network (TeaNet). The key component of TeaNet is a reconstruction module that inputs randomly masked spectra and outputs reconstructed samples that are similar to the original ones, but include additional variations learned from the domain. These augmented samples are used to train the classification model. The reconstruction and prediction parts are trained simultaneously, end-to-end with back-propagation. Results on both synthetic and real-world datasets verified the superiority of the proposed method. In the most difficult synthetic scenarios TeaNet outperformed CNN by 17%. We visualized and analysed the neuron responses of TeaNet and CNN, and found that TeaNet's ability to identify discriminant wavenumbers was excellent compared to CNN. Our approach is general and can be easily adapted to other domains, offering a solution to more accurate and interpretable few-shot learning.

Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

In this paper, we study travel time and energy cost prediction at any future departure time for a targeted road segment and vehicle. These two prediction tasks play an important part in the design of advanced driver-assistance systems (ADAS) that can automatically manage battery charging, energy saving, and route planning for fully electric vehicles. Compared with the fundamental problem of travel time prediction, which usually learns from the historical and current data of travel time itself, energy cost prediction is a more complex problem that involves multiple context conditions and vehicle status measured by various time-invariant and time-variant data. We define a general learning problem based on multiple time-invariant and time-variant inputs to unify these two prediction tasks. To solve the defined learning problem, we propose heterogeneous delay embedding (HDE), which extracts an informative feature space for regression analysis and aims at achieving satisfactory prediction for any future departure time. The proposed HDE first categorizes the historical and current data of a time-variant measurement into different types, then incorporates different delay settings for embedding multiple types of time-series data, and finally removes redundant information and noise from the generated features using orthogonal locality preserving projection. Experimental results demonstrate the effectiveness of the proposed method for both short- and long-term predictions of travel time and energy cost.

Route planning for fully electric vehicles (FEVs) must take energy efficiency into account due to limited battery capacity and time-consuming recharging. In addition, the planning algorithm should allow for negative energy costs in the road network due to regenerative braking, which is a unique feature of FEVs. In this paper, we propose a framework for energy-driven and context-aware route planning for FEVs. It has two novel aspects: 1) It is context aware, i.e., the framework has access to real-time traffic data for routing cost estimation; and it is energy driven, i.e., both time and energy efficiency are accounted for; which implies a biobjective nature of the optimization. In addition, in the case of insufficient energy on board, an optimal detour via recharge points is computed. Our main contributions to address these issues can be highlighted as follows: A vehicle-to-vehicle (V2V) communication protocol is proposed to realize the context awareness, and we replace the original biobjective form of optimality with two single-objective forms and propose a constrained A* ( CA*) algorithm to find the solutions. The algorithm maintains a Pareto front while it confines its search by energy constraints. The best recharging detour can be also found using the algorithm. We first compared the performance of the CA* algorithm with other algorithms. We then evaluate the impact of the context awareness on road traffic by simulations using a realistic road network regarding different forms of optimality. Finally, we show that the CA* algorithm can effectively produce optimal recharging detours.

The objective of this paper is the design of an engine for the automatic generation of supervised manifold embedding models. It proposes a modular and adaptive data embedding framework for classification, referred to as DEFC, which realizes in different stages including initial data preprocessing, relation feature generation and embedding computation. For the computation of embeddings, the concepts of friend closeness and enemy dispersion are introduced, to better control at local level the relative positions of the intraclass and interclass data samples. These are shown to be general cases of the global information setup utilized in the Fisher criterion, and are employed for the construction of different optimization templates to drive the DEFC model generation. For model identification, we use a simple but effective bilevel evolutionary optimization, which searches for the optimal model and its best model parameters. The effectiveness of DEFC is demonstrated with experiments using noisy synthetic datasets possessing nonlinear distributions and real-world datasets from different application fields.

In this paper, we present a novel and robust road tracking system for vision-based personal navigation. Novelty of the work includes the use of multiple Condensation filters to track the road of arbitrary shape and automatic switching between trackers according to road conditions. The approach allows the road to be represented as a simple hyperbola. It also supports the representation of the road as a sequence of connected arcs/segments so that information from a digital map can be integrated into tracking. The parameters of the hyperbola road model are estimated using multiple vanishing points located in image strips. The road tracking method is robust in dealing with complex road shapes, background clutters, shadows, and road markings. Experiments using real videos demonstrate the robustness of our approach.

This paper presents a sensor-fusion framework for video-based navigation. Video-based navigation offers the advantages over existing approaches. With this type of navigation, road signs are directly superimposed onto the video of the road scene, as opposed to those superimposed onto a 2-D map, as is the case with conventional navigation systems. Drivers can then follow the virtual signs in the video to travel to the destination. The challenges of video-based navigation require the use of multiple sensors. The sensor-fusion framework that we propose has two major components: (1) a computer vision module for accurately detecting and tracking the road by using partition sampling and auxiliary variables and (2) a sensor-fusion module using multiple particle filters to integrate vision, Global Positioning Systems (GPSs), and Geographical Information Systems (GISs). GPS and GIS provide prior knowledge about the road for the vision module, and the vision module, in turn, corrects GPS errors.

In this paper, we present a robust road detection and tracking method based on a condensation particle filter for real-time video-based navigation applications. The image is divided into horizontal strips, and vanishing point (VP) detection is performed on each image strip. We propose a method for estimating the density of road boundary line segments in the image so that VP detection in an image strip takes into account the detection results in the neighboring image strips. This use of contextual information for VP detection leads to more accurate detection results. The estimated road parameters are then used to initialize the condensation tracker. Experiments using real road videos demonstrate the robustness of our method to difficult road conditions due to the presence of partial occlusion, shadows, and road signs.