We address the challenge of minimizing "true risk" in multi-node distributed learning.\footnote{We use the term node to refer to a client, FPGA, APU, CPU, GPU, or worker.} These systems are frequently exposed to both inter-node and intra-node "label shifts", which present a critical obstacle to effectively optimizing model performance while ensuring that data remains confined to each node. To tackle this, we propose the Versatile Robust Label Shift (VRLS) method, which enhances the maximum likelihood estimation of the test-to-train label importance ratio. VRLS incorporates Shannon entropy-based regularization and adjusts the importance ratio during training to better handle label shifts at the test time. In multi-node learning environments, VRLS further extends its capabilities by learning and adapting importance ratios across nodes, effectively mitigating label shifts and improving overall model performance. Experiments conducted on MNIST, Fashion MNIST, and CIFAR-10 demonstrate the effectiveness of VRLS, outperforming baselines by up to 20\% in imbalanced settings. These results highlight the significant improvements VRLS offers in addressing label shifts. Our theoretical analysis further supports this by establishing high-probability bounds on estimation errors.

We propose DocVXQA, a novel framework for visually self-explainable document question answering, where the goal is not only to produce accurate answers to questions but also to learn visual heatmaps that highlight critical regions, offering interpretable justifications for the model decision. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning criteria. Unlike conventional relevance map methods that solely emphasize regions relevant to the answer, our context-aware DocVXQA delivers explanations that are contextually sufficient yet representation-efficient. This fosters user trust while achieving a balance between predictive performance and interpretability in document visual question answering applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method.

Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

The recent development of self-explainable deep learning approaches has focused on integrating well-defined explainability principles into learning process, with the goal of achieving these principles through optimization. In this work, we propose DIB-X, a self-explainable deep learning approach for image data, which adheres to the principles of minimal, sufficient, and interactive explanations. The minimality and sufficiency principles are rooted from the trade-off relationship within the information bottleneck framework. Distinctly, DIB-X directly quantifies the minimality principle using the recently proposed matrix-based Rényi’s α-order entropy functional, circumventing the need for variational approximation and distributional assumption. The interactivity principle is realized by incorporating existing domain knowledge as prior explanations, fostering explanations that align with established domain understanding. Empirical results on MNIST and two marine environment monitoring datasets with different modalities reveal that our approach primarily provides improved explainability with the added advantage of enhanced classification performance.

In real-world applications, a single instance could have more than one label. To solve this task, multi-label learning methods emerged in recent years. It is a more challenging problem for many reasons, such as complex label correlation, long-tail label distribution, and data shortage. In general, overcoming these challenges and bettering learning performance could be achieved by utilizing more training samples and including label correlations. However, these solutions are expensive and inflexible. Large-scale, well-labeled datasets are difficult to obtain, and building label correlation maps requires task-specific semantic information as prior knowledge. To address these limitations, we propose a general and compact Multi-Label Correlation Learning (MUCO) framework. MUCO explicitly and effectively learns the latent label correlations by updating a label correlation tensor, which provides highly accurate and interpretable prediction results. In addition, a multi-label generative strategy is deployed to handle the long-tail label distribution challenge. It borrows the visual clues from limited samples and synthesizes more diverse samples. All networks in our model are optimized simultaneously. Extensive experiments illustrate the effectiveness and efficiency of MUCO. Ablation studies further prove the effectiveness of all the modules.

Deploying deep convolutional neural networks on ultra-low power systems is challenging due to the extremely limited resources. Especially, the memory becomes a bottleneck as the systems put a hard limit on the size of on-chip memory. Because peak memory explosion in the lower layers is critical even in tiny models, the size of an input image should be reduced with sacrifice in accuracy. To overcome this drawback, we propose a novel Raster-Scanning Network, named RaScaNet, inspired by raster-scanning in image sensors. RaScaNet reads only a few rows of pixels at a time using a convolutional neural network and then sequentially learns the representation of the whole image using a recurrent neural network. The proposed method operates on an ultra-low power system without input size reduction; it requires 15.9–24.3× smaller peak memory and 5.3–12.9× smaller weight memory than the state-of-the-art tiny models. Moreover, RaScaNet fully exploits on-chip SRAM and cache memory of the system as the sum of the peak memory and the weight memory does not exceed 60 KB, improving the power efficiency of the system. In our experiments, we demonstrate the binary classification performance of RaScaNet on Visual Wake Words and Pascal VOC datasets.

Post-training quantization is a representative technique for compressing neural networks, making them smaller and more efficient for deployment on edge devices. However, an inaccessible user dataset often makes it difficult to ensure the quality of the quantized neural network in practice. In addition, existing approaches may use a single uniform bit-width across the network, resulting in significant accuracy degradation at extremely low bit-widths. To utilize multiple bit-width, sensitivity metric plays a key role in balancing accuracy and compression. In this paper, we propose a novel sensitivity metric that considers the effect of quantization error on task loss and interaction with other layers. Moreover, we develop labeled data generation methods that are not dependent on a specific operation of the neural network. Our experiments show that the proposed metric better represents quantization sensitivity, and generated data are more feasible to apply to mixed-precision quantization.

Of late, deep neural networks have become ubiquitous in mobile applications. As mobile devices generally require immediate response while maintaining user privacy, the demand for on-device machine learning technology is on the increase. Nevertheless, mobile devices suffer from restricted hardware resources, whereas deep neural networks involve considerable computation and communication. Therefore, the implementation of a neural-network specialized hardware accelerator, generally called neural processing unit (NPU), has started to gain attention for the mobile application processor (AP). However, NPUs for commercial mobile AP face two challenges that are difficult to realize simultaneously: execution of a wide range of applications and efficient performance.In this paper, we propose a flexible but efficient NPU architecture for a Samsung flagship mobile system-on-chip (SoC). To implement an efficient NPU, we design an energy-efficient inner-product engine that utilizes the input feature map sparsity. We propose a re-configurable MAC array to enhance the flexibility of the proposed NPU, dynamic internal memory port assignment to maximize on-chip memory bandwidth utilization, and efficient architecture to support mixed-precision arithmetic. We implement the proposed NPU using the Samsung 5nm library. Our silicon measurement experiments demonstrate that the proposed NPU achieves 290.7 FPS and 13.6 TOPS/W, when executing an 8-bit quantized Inception-v3 model [1] with a single NPU core. In addition, we analyze the proposed zero-skipping architecture in detail. Finally, we present the findings and lessons learned when implementing the commercial mobile NPU and interesting avenues for future work.

Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.

Speech signal contains intrinsic and extrinsic variations such as accent, emotion, dialect, phoneme, speaking manner, noise, music, and reverberation. Some of these variations are unnecessary and are unspecified factors of variation. These factors lead to increased variability in speaker representation. In this paper, we assume that unspecified factors of variation exist in speaker representations, and we attempt to minimize variability in speaker representation. The key idea is that a primal speaker representation can be decomposed into orthogonal vectors and these vectors are recombined by using deep neural networks (DNN) to reduce speaker representation variability, yielding performance improvement for speaker verification (SV). The experimental results show that our proposed approach produces a relative equal error rate (EER) reduction of 47.1% compared to the use of the same convolutional neural network (CNN) architecture on the Vox-Celeb dataset. Furthermore, our proposed method provides significant improvement for short utterances.

The goal of Multi-label learning is to predict multiple labels of each single instance. This is a challenging problem since the training data is limited, long-tail label distribution, and complicated label correlations. Generally, more training samples and label correlation knowledge would benefit the learning performance. However, it is difficult to obtain large-scale well-labeled datasets, and building such a label correlation map requires sophisticated semantic knowledge. To this end, we propose an end-to-end Generative Correlation Discovery Network (GCDN) method for multi-label learning in this paper. GCDN captures the existing data distribution, and synthesizes diverse data to enlarge the diversity of the training features; meanwhile, it also learns the label correlations based on a specifically-designed, simple but effective correlation discovery network to automatically discover the label correlations and considerately improve the label prediction accuracy. Extensive experiments on several benchmarks are provided to demonstrate the effectiveness, efficiency, and high accuracy of our approach.

Metric learning has attracted significant attention in the past decades, because of its appealing advances in various real-world tasks, e.g., person re-identification and face recognition. Traditional supervised metric learning attempts to seek a discriminative metric, which could minimize the pairwise distance of within-class data samples, while maximizing the pairwise distance of data samples from various classes. However, it is still a challenge to build a robust and discriminative metric, especially for corrupted data in the real-world application. In this paper, we propose a Robust Discriminative Metric Learning algorithm through fast low-rank representation and denoising strategy. To be specific, the metric learning problem is guided by a discriminative regularization by incorporating the pair-wise or class-wise information. Moreover, the low-rank basis learning is jointly optimized with the metric to better uncover the global data structure and remove noise. Furthermore, the fast low-rank representation is implemented to mitigate the computational burden and ensure the scalability on large-scale datasets. Finally, we evaluate our learned metric on several challenging tasks, e.g., face recognition/verification, object recognition, image clustering, and person re-identification. The experimental results verify the effectiveness of our proposed algorithm in comparison to many metric learning algorithms, even deep learning ones.

Face Parsing assigns every pixel in a facial image with a semantic label, which could be applied in various applications including face recognition, facial beautification, affective computing and animation. While lots of progress have been made in this field, current state-of-the-art methods still fail to extract real effective feature and restore accurate score map, especially for those facial parts which have large variations of deformation and fairly similar appearance, e.g. mouth, eyes and thin eyebrows. In this paper, we propose a novel pixel-wise face parsing method called Residual Encoder Decoder Network (RED-Net), which combines a feature-rich encoder-decoder framework with adaptive prior mechanism. Our encoder-decoder framework extracts feature with ResNet and decodes the feature by elaborately fusing the residual architectures in to deconvolution. This framework learns more effective feature comparing to that learnt by decoding with interpolation or classic deconvolution operations. To overcome the appearance ambiguity between facial parts, an adaptive prior mechanism is proposed in term of the decoder prediction confidence, allowing refining the final result. The experimental results on two public datasets demonstrate that our method outperforms the state-of-the-arts significantly, achieving improvements of F-measure from 0.854 to 0.905 on Helen dataset, and pixel accuracy from 95.12% to 97.59% on the LFW dataset. In particular, convincing qualitative examples show that our method parses eye, eyebrow, and lip regins more accurately.

Accurate age estimation from a facial image is quite challenging, since physical age and apparent age can be quite different, and this difference is dependent on gender, ethnicity, and many other factors. Multitask deep learning is one of the approach to improve age estimation by employing auxiliary tasks, such as gender recognition, that are related to the primary task. However, in traditional multitask learning for age estimation, the relationship between the primary and auxiliary tasks is difficult to describe; how the auxiliary tasks enhance the model for the primary objective is ambiguous. In this letter, we propose a conditional multitask learning method that architecturally factorizes an age variable into gender-conditioned age probabilities in a deep neural network. The lack of accurate training labels with discrete age values is another critical limitation to training age estimation models. Therefore, we propose a label expansion method that increases the number of accurate labels from weakly supervised categorical labels. To verify the generality of the proposed method, we perform intensive experiments on the publicly available MORPH-II and FG-NET datasets. The proposed methods outperform state-of-the art methods in both age estimation and gender recognition accuracy. These performance gains are verified on well-known deep network architectures-VGG-16, CASIA-WebFace, and Alexnet-to confirm the proposed methods generality.

A new face detector with very high accuracy and real-time speed for mobile phone is introduced. The method achieves the fastest speed and the highest accuracy compared with other similar methods. A series of ideas are proposed in order to accelerate detection speed of the traditional Adaboost detector. First of all, the threshold for weak classifier is learned based on a new multiple instance pruning method regarding not only positive samples but also negative samples, by which, weak classifier is able to reject background more efficiently. Then, a coarse-to-fine scan is applied. Coarse scan is used to find possible face location, and the fine scan refines the face location and rejects false alarms. We further improve the speed of multi-scale face detection by introducing two different template sizes for detector training. By which, the smaller faces can be rapidly detected and the high performance is kept for larger faces. The proposed method is evaluated on public dataset FDDB, the result shows competitive performance against all Adaboost based methods. The method has been implemented on mobile phone and the speed is superior to all competitors.

Face recognition under viewpoint and illumination changes is a difficult problem, so many researchers have tried to solve this problem by producing the pose- and illumination- invariant feature. Zhu et al. [26] changed all arbitrary pose and illumination images to the frontal view image to use for the invariant feature. In this scheme, preserving identity while rotating pose image is a crucial issue. This paper proposes a new deep architecture based on a novel type of multitask learning, which can achieve superior performance in rotating to a target-pose face image from an arbitrary pose and illumination image while preserving identity. The target pose can be controlled by the user's intention. This novel type of multi-task model significantly improves identity preservation over the single task model. By using all the synthesized controlled pose images, called Controlled Pose Image (CPI), for the pose-illumination-invariant feature and voting among the multiple face recognition results, we clearly outperform the state-of-the-art algorithms by more than 4~6% on the MultiPIE dataset.

This paper presents a novel method which combines global shape parameters and scalable local descriptors for accurate body parts recognition from a single depth image in real-time. Human poses are of extremely large variation in aspects of visual shapes, because human can take poses from daily activities to gymnastic actions. In order to cover wide-range of the human poses, the proposed algorithm employs a unified structure which combines pose clustering and body parts classification. We name the proposed method Randomized Decision Bush (RDB). Specifically, global shape parameters which can discriminate coarse level shapes are utilized for pose clustering while scalable local shape descriptors are employed for accurate classification. RDB splits the various human poses into multiple clusters which contain similar shapes of the poses. As a result, it provides robust clustering which enables fine level classification within the cluster. The experimental results show improvements on recognizing body parts due to the pose clustering and classification with scalable local descriptors. Additionally, we significantly reduce the complexity of training a large number of human shapes.

In home environments, demands for a robot to serve a user are on the increase, such as cleaning rooms, bringing something to the user, and so on. To achieve these tasks, it is essential for developing a natural way of human-robot interaction (HRI). One of the most natural ways is that the robot approaches the user to do some tasks after recognizing the user's call and localizing its position. In this case, user localization becomes a key technology. In this paper, we propose a novel audio visual user localization system. It consists of a microphone array with eight sensors and a video camera. Estimating calling direction is achieved by the spectral subtraction of the spatial spectra. In particular, a novel beam forming method is proposed to suppress the nonstationary audio noises where they always occur in a real world. Furthermore, a robust method for face detection is proposed to double check the user based on an Adaboost classifier. It is improved to reduce the false alarms remarkably through a new postprocessing on face candidates. Successful results in a real home environment show its efficacy and feasibility. The implementation issues, limitations, and their possible solutions are also discussed.

Speaker segmentation is an important task in multi-party conversations. Overlapping speech poses a serious problem in segmenting audio into speaker turns. We propose an audio-visual speech separation system consisting of an array microphone with eight sensors and an omnidirectional color camera. Multiple concurrent speeches are segmented by fusing the two heterogeneous sensors. Each segmented speech is further enhanced by a linearly constrained minimum variance beamformer. Regardless of co-existing wide-band sound sources and pictures of human in a reverberant environment the proposed system effectively separates multiple target speeches.

Despite an abundance of research outcomes of blind source separation (BSS) in many types of simulated environments, their performances are still not satisfiable to apply to the real environments. The major obstacle may seem the finite filter length of the assumed mixing model and the nonlinear sensor noises. This paper presents a two-step speech enhancement method with stereo microphone inputs. The first step performs a frequency-domain BSS algorithm with no prior knowledge of the mixed source signals and generates stereo outputs. The second step further removes the remaining cross-channel interference by a spectral cancellation approach using a probabilistic source absence/presence detection technique. The desired primary source is detected every frame of the signal, and the secondary source is estimated in the powerspectral domain using the other BSS output as a reference interference source. Then the secondary source is subtracted to remove the cross-channel interference. Our experimental results show good separation enhancement performances on the real recordings of speech and music signals compared to the conventional BSS methods.

An improvement on the voice detection algorithm using global soft decision (GSD) is made in this paper. In GSD method, the speech and noise are modelled by the presumed probability density function, e.g. Gaussian pdf. We propose that the estimation and modelling of the signal is done in the domain of filterbank output which widely used in most speech processing applications. Since the output of filterbank is the weighted sum of outputs of several frequency bins, the signals can no longer be estimated using the Gaussian models but mixture of Gaussian models (GMM) in general. It is shown that the estimation of speech absence probability with GMM gives better performance.

Speech recognition using circular microphone array is addressed in this paper. Eight microphones are located around the service robot to form a 2D microphone array. To enhance the speech quality, a novel adaptive beamformer composed of a delay-and-sum beamformer, adaptive blocking filters (ABFs) and adaptive cancelling filters (ACFs) is proposed. While the adaptive generalized sidelobe canceller (AGSC) connects the ABF and the ACF in feedforward, the proposed adaptive beamformer has them in feedback. The advantages of the proposed structure are the robustness to the steering vector errors and cross-talks and the reduced number of filter taps that gives the same speech quality compared to the AGSC with a larger number of filter taps. The experimental results show that the proposed structure is superior to the AGSC in objective and subjective evaluations. Speech recognition result shows that the proposed robust adaptive beamformer guarantees the recognition performance even in a low SNR and highly reverberant environment.

Modeled from human neurons, various types of artificial neurons are developed and applied to control algorithm. In this paper, the weights and structure of feedforward neural network controller are updated using new skill learning paradigm which consists of supervisory controller, chaotic neuron filter and associative memory. The pattern of system nonlinearity along the desired path is extracted while supervisory controller guarantees stability in the sense of the boundedness of tracking error. Next the pattern is divided into small segments and encoded to bipolar codes depending on the existence of critical points. Comparing the encoded pattern with pre-stored neural parameters and pattern pairs through associative memory, the most similar one is obtained. Also, chaotic neuron filter is used to add perturbation to neural parameters when the training of feedforward neural network is not successful with the pre-stored parameters. Finally the memory is updated with new successful parameters and pattern pairs. Simulation is performed for simple two-link robot in case of the slight modification of desired trajectory.

This paper describes a dynamical local path-planning algorithm of an autonomous mobile robot available for stationary obstacle avoidance using nonlinear friction. Dynamical path-planning algorithm is considered to accommodate the mobile robot to the dynamic situation of the path-planning nature. Together with the previous virtual force field method, the path of the mobile robot is a solution of a path-planning equation. Local minima problems in stationary environments are solved by introducing nonlinear friction into the chaotic neuron. Because of the nonlinear friction, the proposed path-planner reveals chaotic dynamics in some parameter regions. This new path-planner is feasible to guide, in real-time, the mobile robot to avoid stationary obstacles and to reach the goal. Computer simulations are presented to show the effectiveness of the proposed algorithm.