Abstract Ensemble methods have been the norm for anomaly detection. However, existing ensemble methods for anomaly detection have three main issues: (1) Lack of diversity, the base classifiers are of the same algorithm with different initializations, for example, decision trees or neural networks, achieving only sub-optimal results. (2) The predictive uncertainty is not well calibrated for reliable and explainable anomaly detection, where overconfident predictions for both correct and erroneous classifications are made, making the results unreliable and hard to explain. (3) Traditional ensemble methods (e.g., bagging) cannot effectively capture the distinct distributions of predictions from diverse base models. In this paper, we propose a D iversified and H eterogeneous E nsemble learning framework with C alibrated predictive U ncertainty estimation (DHE-CU). We utilize a multi-layer perceptron (MLP) as the meta-classifier to combine the confidence of diverse base models, thereby achieving more explainable anomaly detection. We devise a global diversity loss that considers a global measure of diversity for the selection and pruning of the base models. The MLP meta-classifier can capture the diverse and distinct distributions of predictions from base classifiers. We use a simple yet effective method to quantify the predictive uncertainty of the meta-classifier. We propose a weighted accuracy-uncertainty calibration loss for class-imbalanced data to effectively calibrate predictive uncertainties. Various datasets are used to perform experimental evaluation extensively. The proposed DHE-CU framework demonstrates strong ensemble learning ability, achieving an average improvement of 8.8% in classification accuracy across 27 UCR time series datasets and other anomaly detection benchmark datasets.

Recognizing actions in RGB-D videos necessitates an in-depth understanding of spatial and temporal information from both RGB and depth modalities, as well as an efficient fusion of these data streams. Existing late fusion approaches often suffer from modality collapse due to high semantic similarity between modalities, where the fusion diminishes the distinct contributions of each modality and compromises overall effectiveness. In this context, we propose a novel End-to-end Cross-Modal Attention Transformer (E-CMAT) model. Our model processes RGB and depth inputs through two distinct expert encoders that utilize factorized spatio-temporal representations to capture dimension-independent features, enhancing motion understanding. The extracted RGB and depth tokens are subsequently fused via our innovative cross-modal attention, which is applied iteratively to ensure a robust integration that accentuates critical patterns and discrepancies between the modalities, thereby facilitating more precise action recognition. Our cross-modal attention utilizes one class token per expert for inter-expert information exchange, focusing attention on salient features and fostering synergistic predictions while reducing computational complexity from quadratic to linear. Furthermore, extensive experiments conducted on widely used benchmark datasets, such as NTU RGB-D 60, NTU RGB-D 120, and THU-READ, demonstrate the favorable performance of E-CMAT compared to state-of-the-art models.

Personalized speech synthesis has attracted increasing attention in the field of robotics. Compared to traditional speech synthesis, it faces two primary challenges: the limited availability of adaptation data and the necessity for highly efficient adaptation method using compact parameters to reduce both training time and memory consumption. To address both the challenges, this paper proposes a personalized speech synthesis approach that incorporates an Empirical Mode Decomposition (EMD)-based speaker modeling method alongside a novel decoder structure with masked inputs, which improves the model’s ability to extract speaker-specific features accurately. Furthermore, we introduce a parameter-efficient fine-tuning technique, Attention-based Speaker-Text Scaling and Shifting Feature (AST-SSF), to enhance adaptation efficiency. We validate our approach using the MAGICDATA Corpus. The results indicate that our proposed approach outperforms the baseline in both naturalness and similarity, demonstrating its effectiveness. Moreover, although the proposed adaptation method substantially reduces the number of parameters, it exhibits only minimal performance degradation compared to full and partial fine-tuning strategies.

Abstract Object detectors perform unseen class detection by fine-tuning frozen visual-language models with visual prompts and natural language supervision. However, the current visual prompt tuning methods struggle to learn categorywise shared knowledge when using only one single visual prompt, due to the inaccessibility of unseen classes referred to as novel classes during training. This leads to isolation between the unknown novel classes and the base classes. Inspired by the recently developed RPN-based open-vocabulary object detection (OVOD) methods, we propose a region-aware visual prompt selection (RVPS) module to adaptively combine region features with best-matched visual prompts based on decoupled proxy embeddings. Additionally, we introduce a category-aware patchwise maximal aggregation (CPMA) module to explore the relationships among visual patches with respect to the category-specific maximum activation patches contained within the target region. We evaluate the proposed approach on an open-vocabulary benchmarks: COCO and LVIS. Compared with other state-of-the-art approaches, our method achieves a 1.2% AP $$_{50}$$ improvement on COCO for novel classes and a 0.5% mask AP improvement on LVIS for rare categories.

Backdoor attacks involve the injection of a limited quantity of poisoned samples containing triggers into the training dataset. During the inference stage, backdoor attacks can uphold a high level of accuracy for normal examples, yet when presented with trigger-containing instances, the model may erroneously predict them as the targeted class designated by the attacker. This paper addresses the challenge of backdoor attacks by developing a novel method for filtering poisoned samples. We primarily leverage two key characteristics of backdoor attacks: 1) Multiple backdoors can exist simultaneously within a single model; 2) The discovery through Composite Backdoor Attack (CBA) that altering two triggers in a sample to new target labels does not compromise the original functionality of the triggers, yet enables the prediction of the data as a new target class when both triggers are present simultaneously. Therefore, a novel three-stage poisoning data filtering approach, known as Composite Backdoor Poisoning Filtering (CBPF), is proposed as an effective solution. Firstly, utilizing the identified distinctions in output between poisoned and clean samples, a subset of data is partitioned to include both poisoned and clean data. Subsequently, benign triggers are incorporated and labels are adjusted to create new target and benign target classes, thereby prompting the poisoned and clean data to be classified as distinct entities during the inference stage. The experimental results indicate that CBPF is successful in filtering out poisoned data produced by seven advanced attacks on CIFAR-10, GTSRB and ImageNet-12. On average, CBPF attains a notable filtering success rate of 99.88% for these attacks on CIFAR-10. Additionally, the model trained on the uncontaminated samples exhibits sustained high accuracy levels.

Recently, large-scale synthetic datasets have effectively alleviated the issue of insufficient person re-identification (Re-ID) datasets. However, synthetic datasets grapple with inherent challenges, including the subpar quality of synthetic pedestrians and single data collection. This paper presents InfinitePerson, a costless pipeline that fully utilizes the infinite generation capability of diffusion models to produce diverse UV texture images and effortlessly constructs high-quality synthetic datasets by simulating a real surveillance network. Specifically, we innovatively propose the utilization of diffusion models to generate high-quality, realistic, and diverse UV texture images to address the limitations of clothing textures. This ensures that our 3D character models have complete clothing texture information and look very similar to real-world pedestrians. Moreover, in response to the challenges in replicating synthetic data collection pipelines, we propose a sub-monitoring network data collection method, which can collect pedestrians data from different viewpoints, backgrounds, and lighting conditions through simple scene layout. Finally, a more scalable and realistic large synthetic dataset called InfinitePerson is created, containing 4,700 identities and 535,636 images. Experimental evidence demonstrates show that models trained on InfinitePerson exhibit superior generalization performance, surpassing those trained on both popular real-world and synthetic person Re-ID datasets. The InfinitePerson project is available athttps://github.com/zhguoqing/InfinitePerson.

Occluded person re-identification (Re-ID) is a challenging problem due to the absence of notable discriminative features resulting from incomplete body part images and interference from occluded regions. Recently, some transformer-based methods have demonstrated excellent capabilities in resolving this problem, however these methods are not able to precisely focus on the non-occluded body parts and cannot capture fine-grained local features. To achieve these we propose a Mask-Aware Hierarchical Aggregation TrAnsforMer (MAHATMA) method to enhance occluded person Re-ID. Specifically, we propose a Mask Information Embedding (MIE) module, which directs the model to focus on non-occluded body parts by incorporating the mask semantic information of a human body. Furthermore, to effectively capture fine-grained local features, we propose a Hierarchical Feature Aggregation (HFA) module that mines more exploitable high-quality detail information by aggregating hierarchical image patch representations. To further alleviate the feature loss problem, we propose a Diverse Feature Completion (DFC) module, which is able to complete global features through multi-path feature integration. Extensive experimental evaluations demonstrate that our method exhibits superior performance in dealing with occluded and holistic person datasets.

Huge amounts of data from various sources are substantial to dependable distributed machine learning, especially for trustworthy federated learning (FL). However, existing FL methods are difficult to collect enough data for training the global model more accurately, especially in cross-device scenarios. In this paper, we propose a new federated generative adversarial network empowered by differential privacy and knowledge transfer named Fed-GAN, which can be used to address the problem of data shortage and prevent generator leakage from resource-constrained devices, as well as generate high-quality synthetic data while ensuring strict DP guarantees. Different from other generative model methods, our Fed-GAN framework can achieve efficient and secure generative model training and limited permission for resource-constrained devices to prevent them from leaking or misusing the generator. In addition, we propose a pHash-KT method for our Fed-GAN framework, which selects potentially high-quality data through the knowledge of each client for improving the utility of synthetic data. Our FedGAN framework satisfies ($\frac{2kJ\lambda}{\sigma^2}+\frac{log 1/\delta}{\lambda-1},\delta}$)-DP, and also has high resistance when number of adversaries is 10%-70% of the total number of clients. Extensive experiments demonstrate that our Fed-GAN framework not only generates high-quality synthetic data, but also provides strict DP guarantees, compared with other generative model methods. Our code is publicly available at https://github.com/daxx1/fed-gan

Moving object tracking on cost-effective hardware is a crucial need in numerous research and industrial applications. However, current deep learning-based tracking algorithms usually prioritize exceptional performance at the expense of increased computational load. Due to the unavailability of expensive GPUs for many tracking tasks, these popular trackers often fall short in providing robust tracking capabilities with affordable computational resources. This study introduces RCNNshift, a kernel-based tracker that relies on feature extraction from a random-coupled neural network. This visual cortex inspired neural model can extract image features without requiring cumbersome pre-training or deep neural connections. By utilizing an enhanced one-dimensional feature representation, RCNNshift demonstrates superior performance compared to other kernel-based object tracking methods, even those employing higher-dimensional feature spaces. Its improvement in precision and success plots of OPE, compared to the Meanshift and Camshift in the HSV and RGB color spaces, exceeds over 160% and 190% respectively. Comparative experiments have validated the robustness of RCNNshift, showcasing its superior performance over various kernel-based and particle filter trackers. Its combination of robustness and computational efficiency makes RCNNshift an ideal choice for mid to low-end object tracking tasks such as surveillance and underwater tracking. The source code is available at https://github.com/HaoranLiu507/RCNNshift.

Self-supervised Learning (SSL) using extensive unlabeled speech data has significantly improved the performance of ASR models on datasets like LibriSpeech.However, few studies have addressed the issue of domain mismatch between the data used to pre-train and fine-tune ASR models.Moreover, the Empirical Risk Minimization (ERM) principle, commonly used to train deep learning models, often causes the trained models to exhibit undesirable behaviors such as memorizing training data and being sensitive to adversarial examples.Thus, in this paper, we propose an alternate fine-tuning strategy, called Mix-fine-tune, to address domain mismatch in ASR systems and the limitations of the ERM training principle.Mix-finetune use a data-driven weighted sum of two speech sequences as input and the corresponding text sequences are used to calculate a weighted audio-text alignment Connectionist Temporal Classification (CTC) loss for fine-tuning a pre-trained model.Additionally, Mix-fine-tune incorporates the masked Contrastive Predictive Coding (CPC) loss, previously used exclusively for pre-training, into the fine-tuning process.Our novel strategy alternates between minimizing the CTC loss and the CPC loss to address the domain mismatch between pre-training and fine-tuning.We validate our method by fine-tuning different sizes of the Wav2Vec model using the public Air Traffic Control (ATC) corpus.The experiments show that Mix-fine-tune efficiently adapts the models pre-trained on general speech corpora like LibriSpeech to a specific domain (e.g., the air traffic control domain) by fine-turning.

Effective online portfolio selection necessitates seamless integration of three key properties: diversity, sparsity , and risk control. However, existing algorithms often prioritize one property at the expense of the others due to inherent conflicts. To address this issue, we propose an asset subset-constrained minimax (ASCM) optimization framework, which generates optimal portfolios from diverse investment strategies represented as asset subsets. ASCM consists of: (i) a minimax optimization model that focuses on risk control by considering a set of loss functions constrained by different asset subsets; (ii) the construction of asset subsets via price-feature clipping, which effectively reduces redundant assets in the portfolio; (iii) a state-based estimation of price trends that guides all ASCM loss functions, facilitating the generation of sparse solutions. We solve the ASCM minimax model using an efficient iterative updating formula derived from the projected subgradient method. Furthermore, we achieve near O ( 1 ) time complexity through a novel initialization scheme. Experimental results demonstrate that ASCM outperforms eight representative algorithms, including the best constant rebalanced portfolio in hindsight (BCRP) on five out of the six real-world financial datasets. Notably, ASCM achieves a 67-fold improvement over BCRP in cumulative wealth on the TSE dataset.

Action Recognition aims to understand human behavior and predict a label for each action. Recently, Vision Transformer (ViT) has achieved remarkable performance on action recognition, which models the long sequences token over spatial and temporal index in a video. The fully-connected self-attention layer is the fundamental key in the vanilla Transformer. However, the redundant architecture of the vision Transformer model ignores the locality of video frame patches, which involves non-informative tokens and potentially leads to increased computational complexity. To solve this problem, we propose a k-NN attention-based Video Vision Transformer (k-ViViT) network for action recognition. We adopt k-NN attention to Video Vision Transformer (ViViT) instead of original self-attention, which can optimize the training process and neglect the irrelevant or noisy tokens in the input sequence. We conduct experiments on the UCF101 and HMDB51 datasets to verify the effectiveness of our model. The experimental results illustrate that the proposed k-ViViT achieves superior accuracy compared to several state-of-the-art models on these action recognition datasets.

Abstract Coreference resolution is the task of identifying and clustering mentions that refer to the same entity in a document. Based on state-of-the-art deep learning approaches, end-to-end coreference resolution considers all spans as candidate mentions and tackles mention detection and coreference resolution simultaneously. Recently, researchers have attempted to incorporate document-level context using higher-order inference (HOI) to improve end-to-end coreference resolution. However, HOI methods have been shown to have marginal or even negative impact on coreference resolution. In this paper, we reveal the reasons for the negative impact of HOI coreference resolution. Contextualized representations (e.g., those produced by BERT) for building span embeddings have been shown to be highly anisotropic. We show that HOI actually increases and thus worsens the anisotropy of span embeddings and makes it difficult to distinguish between related but distinct entities (e.g., pilots and flight attendants ). Instead of using HOI, we propose two methods, Less-Anisotropic Internal Representations (LAIR) and Data Augmentation with Document Synthesis and Mention Swap (DSMS), to learn less-anisotropic span embeddings for coreference resolution. LAIR uses a linear aggregation of the first layer and the topmost layer of contextualized embeddings. DSMS generates more diversified examples of related but distinct entities by synthesizing documents and by mention swapping. Our experiments show that less-anisotropic span embeddings improve the performance significantly (+2.8 F1 gain on the OntoNotes benchmark) reaching new state-of-the-art performance on the GAP dataset.

Weakly supervised salient object detection (WSOD) methods endeavor to boost sparse labels to get more salient cues in various ways. Among them, an effective approach is using pseudo labels from multiple unsupervised self-learning methods, but inaccurate and inconsistent pseudo labels could ultimately lead to detection performance degradation. To tackle this problem, we develop a new multi-source WSOD framework, WBNet, that can effectively utilize pseudo-background (non-salient region) labels combined with scribble labels to obtain more accurate salient features. We first design a comprehensive salient pseudo-mask generator from multiple self-learning features. Then, we pioneer the exploration of generating salient pseudo-labels via point-prompted and box-prompted Segment-Anything Models (SAM). Then, WBNet leverages a pixel-level Feature Aggregation Module (FAM), a mask-level Transformer-decoder (TFD), and an auxiliary Boundary Prediction Module (EPM) with a hybrid loss function to handle complex saliency detection tasks. Comprehensively evaluated with state-of-the-art methods on five widely used datasets, the proposed method significantly improves saliency detection performance. The code and results are publicly available at https://github.com/yiwangtz/WBNet.

Healthcare uses state-of-the-art technologies (such as wearable devices, blood glucose meters, electrocardiographs), which results in the generation of large amounts of data. Healthcare data is essential in patient management and plays a critical role in transforming healthcare services, medical scheme design, and scientific research. Missing data is a challenging problem in healthcare due to system failure and untimely filing, resulting in inaccurate diagnosis treatment anomalies. Therefore, there is a need to accurately predict and impute missing data as only complete data could provide a scientific and comprehensive basis for patients, doctors, and researchers. However, traditional approaches in this paradigm often neglect the effect of the time factor on forecasting results. This paper proposes a time-aware missing healthcare data prediction approach based on the autoregressive integrated moving average (ARIMA) model. We combine a truncated singular value decomposition (SVD) with the ARIMA model to improve the prediction efficiency of the ARIMA model and remove data redundancy and noise. Through the improved ARIMA model, our proposed approach (namedMHDP$_{SVD\_{A}RIMA}$) can capture underlying pattern of healthcare data changes with time and accurately predict missing data. The experiments conducted on the WISDM dataset show thatMHDP$_{SVD\_{A}RIMA}$approach is effective and efficient in predicting missing healthcare data.

Image dehazing is an emblematical low-level vision task that aims at restoring haze-free images from haze images. Recently, some methods adopts deep learning techniques to rebuild haze-free images. However, in real-world scenarios, complex degradation of captured images and non-uniform spatial distributions of haze will significantly weaken the generalization ability of these models. Accordingly, we propose a novel Spatial Dual-Branch Attention Dehazing network (SDBAD-Net) based on the Meta-Former paradigm for end-to-end dehazing. Specifically, we firstly design a robust Spatial Dual-Branch Attention (SDBA) module to filter the haze distribution features from different densities, which is suitable for both uniform and non-uniform situations. Secondly, we introduce a Structural Features Supplementary (SFS) module to dynamically fuse the contextual structural features in a nonlinear manner, so as to correct the image distortion caused by the lack of structural details. Finally, the quantitative and qualitative experiments are carried out on two challenging datasets, and the results show that our method outperforms most of state-of-the-art algorithms with fewer parameters and faster speed, especially surpassing FFA-Net with only 50% parameters and 7% computational costs. In addition, we ulteriorly explore its performance on object detection in foggy weather with our model on the challenging Real-world Task-driven Testing Set (RTTS), and the surprising results further prove the robustness and wide-applicability of our method.

Salient object detection (SOD) aims to mimic the human visual system (HVS) and cognition mechanisms to identify and segment salient objects. However, due to the complexity of these mechanisms, current methods are not perfect. Accuracy and robustness need to be further improved, particularly in complex scenes with multiple objects and background clutter. To address this issue, we propose a novel approach called Multiple Enhancement Network (MENet) that adopts the boundary sensibility, content integrity, iterative refinement, and frequency decomposition mechanisms of HVS. A multi-level hybrid loss is firstly designed to guide the network to learn pixel-level, region-level, and object-level features. A flexible multiscale feature enhancement module (ME-Module) is then designed to gradually aggregate and refine global or detailed features by changing the size order of the input feature sequence. An iterative training strategy is used to enhance boundary features and adaptive features in the dual-branch decoder of MENet. Comprehensive evaluations on six challenging benchmark datasets show that MENet achieves state-of-the-art results. Both the codes and results are publicly available at https://github.com/yiwangtz/MENet.

The self-media era provides us tremendous high quality videos. Unfortunately, frequent video copyright infringements are now seriously damaging the interests and enthusiasm of video creators. Identifying infringing videos is therefore a compelling task. Current state-of-the-art methods tend to simply feed high-dimensional mixed video features into deep neural networks and count on the networks to extract useful representations. Despite its simplicity, this paradigm heavily relies on the original entangled features and lacks constraints guaranteeing that useful task-relevant semantics are extracted from the features.

RGB-D action and gesture recognition remain an interesting topic in human-centered scene understanding, primarily due to the multiple granularities and large variation in human motion. Although many RGB-D based action and gesture recognition approaches have demonstrated remarkable results by utilizing highly integrated spatio-temporal representations across multiple modalities (i.e., RGB and depth data), they still encounter several challenges. Firstly, vanilla 3D convolution makes it hard to capture fine-grained motion differences between local clips under different modalities. Secondly, the intricate nature of highly integrated spatio-temporal modeling can lead to optimization difficulties. Thirdly, duplicate and unnecessary information can add complexity and complicate entangled spatio-temporal modeling. To address the above issues, we propose an innovative heuristic architecture called Multi-stage Factorized Spatio-Temporal (MFST) for RGB-D action and gesture recognition. The proposed MFST model comprises a 3D Central Difference Convolution Stem (CDC-Stem) module and multiple factorized spatio-temporal stages. The CDC-Stem enriches fine-grained temporal perception, and the multiple hierarchical spatio-temporal stages construct dimension-independent higher-order semantic primitives. Specifically, the CDC-Stem module captures bottom-level spatio-temporal features and passes them successively to the following spatio-temporal factored stages to capture the hierarchical spatial and temporal features through the Multi-Scale Convolution and Transformer (MSC-Trans) hybrid block and Weight-shared Multi-Scale Transformer (WMS-Trans) block. The seamless integration of these innovative designs results in a robust spatio-temporal representation that outperforms state-of-the-art approaches on RGB-D action and gesture recognition datasets.

Data augmentation is an effective way to enhance the performance of neural machine translation models, especially for low-resource languages. Existing data augmentation methods are either at a token level or a sentence level. The data augmented using token level methods lack syntactic diversity and may alter original meanings. Sentence level methods usually generate low-quality source sentences that are not semantically paired with the original target sentences. In this paper, we propose a novel data augmentation method to generate diverse, high-quality and meaning-preserved new instances. Our method leverages high-quality translation models trained with high-resource languages to rephrase an original sentence by translating it into an intermediate language and then back to the original language. Through this process, the high-performing translation models guarantee the quality of the rephrased sentences, and the syntactic knowledge from the intermediate language can bring syntactic diversity to the rephrased sentences. Experimental results show our method can enhance the performance in various low-resource machine translation tasks. Moreover, by combining our method with other techniques that facilitate NMT, we can yield even better results.

Recent status-of-the-art methods for multivariate time series forecasting can be categorized into graph-based approach and global-local approach. The former approach uses graphs to represent the dependencies among variables and apply graph neural networks to the forecasting problem. The latter approach decomposes the matrix of multivariate time series into global components and local components to capture the shared information across variables. However, both approaches cannot capture the propagation delay of the dependencies among individual variables of a multivariate time series, for example, the congestion at intersection A has a delayed effects on the neighbouring intersection B. In addition, graph-based forecasting methods cannot capture the shared global tendency across the variables of a multivariate time series; and global-local forecasting methods cannot reflect the nonlinear inter-dependencies among variables of a multivariate time series. In this paper, we propose to combine the advantages of both approaches by integrating Adaptive Global-Local Graph Structure Learning with Gated Recurrent Units (AGLG-GRU). We learn a global graph to represent the shared information across variables. And we learn dynamic local graphs to capture the local randomness and nonlinear dependencies among variables. We apply diffusion convolution and graph convolution operations to global and dynamic local graphs to integrate the information of graphs and update gated recurrent unit for multivariate time series forecasting. The experimental results on seven representative real-world datasets demonstrate that our approach outperform various existing methods.

Non-negative matrix factorization (NMF) has become a popular method for learning interpretable patterns from data. As one of the variants of standard NMF, convolutive NMF (CNMF) incorporates an extra time dimension to each basis, known as convolutive bases, which is well suited for representing sequential patterns. Previously proposed algorithms for solving CNMF use multiplicative updates which can be derived by either heuristic or majorization-minimization (MM) methods. However, these algorithms suffer from problems, such as low convergence rates, difficulty to reach exact zeroes during iterations and prone to poor local optima. Inspired by the success of alternating direction method of multipliers (ADMMs) on solving NMF, we explore variable splitting (i.e., the core idea of ADMM) for CNMF in this article. New closed-form algorithms of CNMF are derived with the commonly used β -divergences as optimization objectives. Experimental results have demonstrated the efficacy of the proposed algorithms on their faster convergence, better optima, and sparser results than state-of-the-art baselines.

Research on orthodontic treatment monitoring from oralscan video is a new direction in dental digitalization. We designed an approach to reconstruct, segment, and estimate the pose of individual teeth to measure orthodontic treatment. To handle the semantic gap in heterogeneous data on the condition that they are combined linearly, we present a multimedia interaction network (MIN) to combine heterogeneous information in point cloud segmentation by extending the graph attention mechanism. Moreover, a structure-aware quadruple loss is designed to explore the relation between multiple and diverse unmatched points in point cloud registration. The performance of our approach is evaluated on multiple tooth registration datasets, and extensive experiments show that our approach improves the accuracy by a margin of 1.4% in the inlier ratio on the Aoralscan3 dataset when it is compared with prevailing approaches.

Fine-grained entity typing (FGET) is an important natural language processing task. It is to assign fine-grained semantic types of a type taxonomy (e.g., Person/artist/actor) to entity mentions. Fine-grained entity semantic types have been successfully applied in many natural language processing (NLP) applications, such as relation extraction, entity linking and question answering. The key challenge for FGET is how to deal with label noises that disperse in the corpora since the corpora are normally automatically annotated. Various type taxonomies, typing methods and representation learning approaches for FGET have been proposed and developed in the past two decades. This paper systematically categorizes and reviews these various typing methods and representation learning approaches to provide a reference for future studies on FGET. We identify the current trends in FGET research: (i) Learning embedded feature representations to address the challenges posed by label noises, tail types and new entities; (ii) Tackling FGET jointly with other entity analysis sub-tasks (e.g., entity linking and coreference resolution) is also a promising direction. We also present a comprehensive review of type taxonomies, resources, applications for FGET and methods for automatically generating FGET training corpora.

Speech-based automatic smoker identification (also known as smoker/non-smoker classification) aims to identify speakers’ smoking status from their speech. In the COVID-19 pandemic, speech-based automatic smoker identification approaches have received more attention in smoking cessation research due to low cost and contactless sample collection. This study focuses on determining the best acoustic features for smoker identification. In this paper, we investigate the performance of four acoustic feature sets/representations extracted using three feature extraction/learning approaches: (i) hand-crafted feature sets including the extended Geneva Minimalistic Acoustic Parameter Set and the Computational Paralinguistics Challenge Set, (ii) the Bag-of-Audio-Words representations, (iii) the neural representations extracted from raw waveform signals by SincNet. Experimental results show that: (i) SincNet feature representations are the most effective for smoker identification and outperform the MFCC baseline features by 16% in absolute accuracy; (ii) the performance of hand-crafted feature sets and the Bag-of-Audio-Words representations rely on the scale of the dimensions of feature vectors.

We propose a new meta learning based framework for low resource speech recognition that improves the previous model agnostic meta learning (MAML) approach. The MAML is a simple yet powerful meta learning approach. However, the MAML presents some core deficiencies such as training instabilities and slower convergence speed. To address these issues, we adopt multi-step loss (MSL). The MSL aims to calculate losses at every step of the inner loop of MAML and then combines them with a weighted importance vector. The importance vector ensures that the loss at the last step has more importance than the previous steps. Our empirical evaluation shows that MSL significantly improves the stability of the training procedure and it thus also improves the accuracy of the overall system. Our proposed system outperforms MAML based low resource ASR system on various languages in terms of character error rates and stable training behavior.

Abstract Research on tooth surface design based on deep neural networks has recently achieved progress in terms of both accuracy and execution efficiency. However, unrealistic outputs are still a challenging issue, partially because of (1) the lack of semantic guidance, (2) the inability to discover and rectify false results, and (3) the lack of exploration of structural coherence in intermediate layers. In this paper, we present an approach to predict depth images for designed teeth based on a conditional generative adversarial network (CGAN) by incorporating semantic guidance. Moreover, the uncertainty of semantic inference is employed to improve the model outputs, and a structural coherence loss is proposed for adversarial learning to enhance the discrimination capability of the network in intermediate layers. We evaluate the performance of our approach with the Shining3D tooth dataset. The experimental results show that our method produces better results than the other available approaches in terms of accuracy.

The recent COVID-19 pandemic has highlighted the importance of food safety and supply chain governance. In other words, we need to ensure traceability along the supply chain and support high-frequency transactions, effective data collections, etc. Thus, we posit the potential of using a lightning network, which is a decentralized traceable paradigm for achieving high-frequency transactions in blockchain-based systems. In addition, we also utilize edge computing to help facilitate data collection. However, a key challenge in securing food supplies is determining the optimal global transaction path in the lightning network while achieving efficiency and meeting the dynamic nature of food supply management. Thus, we propose a blockchain-edge scheme that utilizes our proposed dynamic programming to produce optimal solutions for selecting global transaction paths. Specifically, our scheme optimizes routing fees under existing constraints (e.g., transmission cost, computing resource consumption, and lightning network balance). The findings from our evaluations demonstrate the utility of our proposed approach in facilitating food safety management.

This article proposes a deep learning solution to the online portfolio selection problem based on learning a latent structure directly from a price time series. It introduces a novel wealth flow matrix for representing a latent structure that has special regular conditions to encode the knowledge about the relative strengths of assets in portfolios. Therefore, a wealth flow model (WFM) is proposed to learn wealth flow matrices and maximize portfolio wealth simultaneously. Compared with existing approaches, our work has several distinctive benefits: (1) the learning of wealth flow matrices makes our model more generalizable than models that only predict wealth proportion vectors, and (2) the exploitation of wealth flow matrices and the exploration of wealth growth are integrated into our deep reinforcement algorithm for the WFM. These benefits, in combination, lead to a highly-effective approach for generating reasonable investment behavior, including short-term trend following, the following of a few losers, no self-investment, and sparse portfolios. Extensive experiments on five benchmark datasets from real-world stock markets confirm the theoretical advantage of the WFM, which achieves the Pareto improvements in terms of multiple performance indicators and the steady growth of wealth over the state-of-the-art algorithms.

We propose a novel pitch estimation technique called DeepF0, which leverages the available annotated data to directly learns from the raw audio in a data-driven manner. f0estimation is important in various speech processing and music information retrieval applications. Existing deep learning models for pitch estimations have relatively limited learning capabilities due to their shallow receptive field. The proposed model addresses this issue by extending the receptive field of a network by introducing the dilated convolutional blocks into the network. The dilation factor increases the network receptive field exponentially without increasing the parameters of the model exponentially. To make the training process more efficient and faster, DeepF0 is augmented with residual blocks with residual connections. Our empirical evaluation demonstrates that the proposed model outperforms the baselines in terms of raw pitch accuracy and raw chroma accuracy even using 77.4% fewer network parameters. We also show that our model can capture reasonably well pitch estimation even under the various levels of accompaniment noise.

With the continuous exploration of marine resources, underwater artificial intelligent robots play an increasingly important role in the fish industry. However, the detection of underwater objects is a very challenging problem due to the irregular movement of underwater objects, the occlusion of sand and rocks, the diversity of water illumination, and the poor visibility and low color contrast in the underwater environment. In this article, we first propose a real-world underwater object detection dataset (UODD), which covers more than 3K images of the most common aquatic products. Then we propose Channel Sharpening Attention Module (CSAM) as a plug-and-play module to further fuse high-level image information, providing the network with the privilege of selecting feature maps. Fusion of original images through CSAM can improve the accuracy of detecting small and medium objects, thereby improving the overall detection accuracy. We also use Water-Net as a preprocessing method to remove the haze and color cast in complex underwater scenes, which shows a satisfactory detection result on small-sized objects. In addition, we use the class weighted loss as the training loss, which can accurately describe the relationship between classification and precision of bounding boxes of targets, and the loss function converges faster during the training process. Experimental results show that the proposed method reaches a maximum AP of 50.1%, outperforming other traditional and state-of-the-art detectors. In addition, our model only needs an average inference time of 25.4 ms per image, which is quite fast and might suit the real-time scenario.

Road extraction in remote sensing images is of great importance for a wide range of applications. Because of the complex background, and high density, most of the existing methods fail to accurately extract a road network that appears correct and complete. Moreover, they suffer from either insufficient training data or high costs of manual annotation. To address these problems, we introduce a new model to apply structured domain adaption for synthetic image generation and road segmentation. We incorporate a feature pyramid (FP) network into generative adversarial networks to minimize the difference between the source and target domains. A generator is learned to produce quality synthetic images, and the discriminator attempts to distinguish them. We also propose a FP network that improves the performance of the proposed model by extracting effective features from all the layers of the network for describing different scales' objects. Indeed, a novel scale-wise architecture is introduced to learn from the multilevel feature maps and improve the semantics of the features. For optimization, the model is trained by a joint reconstruction loss function, which minimizes the difference between the fake images and the real ones. A wide range of experiments on three data sets prove the superior performance of the proposed approach in terms of accuracy and efficiency. In particular, our model achieves state-of-the-art 78.86 IOU on the Massachusetts data set with 14.89M parameters and 86.78B FLOPs, with 4× fewer FLOPs but higher accuracy (+3.47% IOU) than the top performer among state-of-the-art approaches used in the evaluation.

Video captioning is a challenging task in the field of multimedia processing, which aims to generate informative natural language descriptions/captions to describe video contents. Previous video captioning approaches mainly focused on capturing visual information in videos using an encoder-decoder structure to generate video captions. Recently, a new encoder-decoder-reconstructor structure was proposed for video captioning, which captured the information in both videos and captions. Based on this, this article proposes a novel multi-instance multi-label dual learning approach (MIMLDL) to generate video captions based on the encoder-decoder-reconstructor structure. Specifically, MIMLDL contains two modules: caption generation and video reconstruction modules. The caption generation module utilizes a lexical fully convolutional neural network (Lexical FCN) with a weakly supervised multi-instance multi-label learning mechanism to learn a translatable mapping between video regions and lexical labels to generate video captions. Then the video reconstruction module synthesizes visual sequences to reproduce raw videos using the outputs of the caption generation module. A dual learning mechanism fine-tunes the two modules according to the gap between the raw and the reproduced videos. Thus, our approach can minimize the semantic gap between raw videos and the generated captions by minimizing the differences between the reproduced and the raw visual sequences. Experimental results on a benchmark dataset demonstrate that MIMLDL can improve the accuracy of video captioning.

No abstract available.

Entity embeddings, which represent different aspects of each entity with a single vector like word embeddings, are a key component of neural entity linking models.Existing entity embeddings are learned from canonical Wikipedia articles and local contexts surrounding target entities.Such entity embeddings are effective, but too distinctive for linking models to learn contextual commonality.We propose a simple yet effective method, FGS2EE, to inject fine-grained semantic information into entity embeddings to reduce the distinctiveness and facilitate the learning of contextual commonality.FGS2EE first uses the embeddings of semantic type words to generate semantic embeddings, and then combines them with existing entity embeddings through linear aggregation.Extensive experiments show the effectiveness of such embeddings.Based on our entity embeddings, we achieved new state-ofthe-art performance on entity linking.

We propose KDSL, a new word sense disambiguation (WSD) framework that utilizes knowledge to automatically generate sense-labeled data for supervised learning. First, from WordNet, we automatically construct a semantic knowledge base called DisDict, which provides refined feature words that highlight the differences among word senses, i.e., synsets. Second, we automatically generate new sense-labeled data by DisDict from unlabeled corpora. Third, these generated data, together with manually labeled data and unlabeled data, are fed to a neural framework conducting supervised and unsupervised learning jointly to model the semantic relations among synsets, feature words and their contexts. The experimental results show that KDSL outperforms several representative state-of-the-art methods on various major benchmarks. Interestingly, it performs relatively well even when manually labeled data is unavailable, thus provides a potential solution for similar tasks in a lack of manual annotations.

Recent research on dense captioning based on the recurrent neural network and the convolutional neural network has made a great progress. However, mapping from an image feature space to a description space is a nonlinear and multimodel task, which makes it difficult for the current methods to get accurate results. In this paper, we put forward a novel approach for dense captioning based on hourglass-structured residual learning. Discriminant feature maps are obtained by incorporating dense connected networks and residual learning in our model. Finally, the performance of the approach on the Visual Genome V1.0 dataset and the region labelled MS-COCO (Microsoft Common Objects in Context) dataset are demonstrated. The experimental results have shown that our approach outperforms most current methods.

Local outlier detection is able to capture local behavior to improve detection performance compared to traditional global outlier detection techniques. Most existing local outlier detection methods have the fundamental assumption that attributes and attribute values are independent and identically distributed (IID). However, in many situations, since the attributes usually have an inner structure, they should not be handled equally. To address the issue above, we propose a novel automatic context-based similarity metric for local outlier detection tasks. This paper mainly includes three aspects: (i) to propose a novel approach to automatically detect the contextual attributes by capturing the attribute intra-coupling and inter-coupling; (ii) to introduce a Non-IID similarity metric to derive the kNN set and reachability distance of an object based on the attribute structure and incorporate it into local outlier detection tasks; (iii) to build a data set called EG-Permission, which is a real-world data set from an E-Government Information System for context-based local outlier detection. Results obtained from 10 data sets show the proposed approach can identify the attribute structure effectively and improve the performance in local outlier detection tasks.

As the most widely applied mobile operating system for smartphones, Android is challenged by fast growing security problems, which are caused by malicious applications. Behaviors of malicious applications have become more and more inconspicuous, which largely increase the difficulties of security detection. This paper provides a new security sandbox approach of Android based on hook mechanism, to further enrich Android malware detection technologies. This new sandbox monitors the behaviors of target application by using a process hook-based dynamic tracking method during its running period. Compared to existing techniques, (1) this approach can create a virtual space where apk can be installed, run, and uninstalled, and it is isolated from the outside and (2) a risk assessment approach based on behavior analysis is given so that users can obtain an explicit risk prognosis for an application to improve their safety. Tests on malware and normal application samples verify this new security sandbox.

nearest neighbor (kNN) method is a popular classification method in data mining and statistics because of its simple implementation and significant classification performance. However, it is impractical for traditional kNN methods to assign a fixed value (even though set by experts) to all test samples. Previous solutions assign different values to different test samples by the cross validation method but are usually time-consuming. This paper proposes a kTree method to learn different optimal values for different test/new samples, by involving a training stage in the kNN classification. Specifically, in the training stage, kTree method first learns optimal values for all training samples by a new sparse reconstruction model, and then constructs a decision tree (namely, kTree) using training samples and the learned optimal values. In the test stage, the kTree fast outputs the optimal value for each test sample, and then, the kNN classification can be conducted using the learned optimal value and all training samples. As a result, the proposed kTree method has a similar running cost but higher classification accuracy, compared with traditional kNN methods, which assign a fixed value to all test samples. Moreover, the proposed kTree method needs less running cost but achieves similar classification accuracy, compared with the newly kNN methods, which assign different values to different test samples. This paper further proposes an improvement version of kTree method (namely, k*Tree method) to speed its test stage by extra storing the information of the training samples in the leaf nodes of kTree, such as the training samples located in the leaf nodes, their kNNs, and the nearest neighbor of these kNNs. We call the resulting decision tree as k*Tree, which enables to conduct kNN classification using a subset of the training samples in the leaf nodes rather than all training samples used in the newly kNN methods. This actually reduces running cost of test stage. Finally, the experimental results on 20 real data sets showed that our proposed methods (i.e., kTree and k*Tree) are much more efficient than the compared methods in terms of classification tasks.

The alternating direction method of multipliers (ADMM) algorithm has been widely employed for distributed machine learning tasks. However, it suffers from several limitations, e.g., a relative low convergence speed, and an expensive time cost. To this end, in this paper, a novel method, namely the group-based ADMM (GADMM), is proposed for distributed linear classification. In particular, to accelerate the convergence speed and improve global consensus, a group layer is first utilized in GADMM to divide all the slave nodes into several groups. Then, all the local variables (from the slave nodes) are gathered in the group layer to generate different group variables. Finally, by using a weighted average method, the group variables are coordinated to update the global variable (from the master node) until the solution of the global problem is reached. According to the theoretical analysis, we found that: 1) GADMM can mathematically converge at the rate , where is the number of outer iterations and 2) by using the grouping methods, GADMM can improve the convergence speed compared with the distributed ADMM framework without grouping methods. Moreover, we systematically evaluate GADMM on four publicly available LIBSVM datasets. Compared with disADMM and stochastic dual coordinate ascent with alternating direction method of multipliers-ADMM, for distributed classification, GADMM is able to reduce the number of outer iterations, which leads to faster convergence speed and better global consensus. In particular, the statistical significance test has been experimentally conducted and the results validate that GADMM can significantly save up to 30% of the total time cost (with less than 0.6% accuracy loss) compared with disADMM on large-scale datasets, e.g., webspam and epsilon.

In this work, we present a new multiple channel feature called Deep Compact Channel Feature (DCCF), which generates a compact, discriminative feature representation by a pre-trained deep encoder-decoder. With the combination of DCCF and boosted decision trees, a new object detector is proposed which achieved outstanding performance on standard pedestrian dataset INRIA and Caltech. Furthermore, a large scale and challenging Chinese Traffic Sign Detection benchmark is constructed. DCCF and other related methods are evaluated on this dataset. The dataset and baselines are available online.

Dynamic vehicle detection and tracking is crucial for self-driving in urban environments. The main problem of the previous beam-model-based algorithms is that they cannot detect and track dynamic vehicles that are occluded by other objects. In this paper, we develop a novel dynamic vehicle detection and tracking algorithm to solve this problem for our autonomous land vehicle (ALV), which is equipped with a Velodyne LIDAR and a GPS-aid inertial navigation system. For detection, our improved two-dimensional virtual scan is presented to detect the potential dynamic vehicles with a scan differencing operation. Then, for each potential dynamic vehicle, a novel likelihood-field-based vehicle measurement model is proposed to weight its possible poses. Finally, our newly modified scaling series algorithm and the importance sampling technique are adopted to estimate the initial pose and the corresponding velocity for each vehicle, respectively. The scaling series algorithm coupled with a Bayesian filter (SSBF) was previously used to handle the tactile localization problem in static background scenes. For tracking dynamic vehicles, we improve the SSBF by adding the ego-motion compensation so that the improved algorithm is able to update the pose and velocity for each vehicle in dynamic background scenes. Both the quantitative and qualitative experimental results validate the performance of our dynamic vehicle detection and tracking algorithm on the KITTI datasets and the Velodyne data collected by our ALV in dynamic urban environments.

In this paper, we focus on the long time behavior of the mild solution to delayed reaction-diffusion Hopfield neural networks (DRDHNNs) driven by infinite dimensional Wiener processes. We analyze the existence, uniqueness, and stability of this system under the local Lipschitz function by constructing an appropriate Lyapunov-Krasovskii function and utilizing the semigroup theory. Some easy-to-test criteria affecting the well-posedness and stability of the networks, such as infinite dimensional noise and diffusion effect, are obtained. The criteria can be used as theoretic guidance to stabilize DRDHNNs in practical applications when infinite dimensional noise is taken into consideration. Meanwhile, considering the fact that the standard Brownian motion is a special case of infinite dimensional Wiener process, we undertake an analysis of the local Lipschitz condition, which has a wider range than the global Lipschitz condition. Two samples are given to examine the availability of the results in this paper. Simulations are also given using the MATLAB.

The key task in developing graph-based learning algorithms is constructing an informative graph to express the contextual information of a data manifold. Since traditional graph construction methods are sensitive to noise and less datum-adaptive to changes in density, a new method called$\ell^1$-graph was proposed recently. A graph construction needs to have two important properties: sparsity and locality. The$\ell^1$-graph has a strong sparsity property, but a weak locality property. Thus, we propose a new method of constructing an informative graph using auto-grouped sparse regularization based on the$\ell^1$-graph, which is called as Group Sparse graph (GS-graph). We also show how to efficiently construct a GS-graph in reproducing kernel Hilbert space with the kernel trick. The new methods, the GS-graph and its kernelized version (KGS-graph), have the same noise-insensitive property as that of$\ell^1$-graph and also can successively preserve the properties of sparsity and locality simultaneously. Furthermore, we integrate the proposed graph with several graph-based learning algorithms to demonstrate the effectiveness of our method. The empirical studies on benchmarks show that the proposed methods outperform the$\ell^1$-graph and other traditional graph construction methods in various learning tasks.

Canonical correlation analysis (CCA) has been widely applied to information fusion. It only considers the correlated information of the paired data, but ignores the correlated information between the samples in the same class. Furthermore, class information is useful for CCA, but there is little class information in the scenarios of real applications. Thus, it is difficult to utilize the correlated information between the samples in the same class. To utilize the correlated information between the samples, we propose a method named Unsupervised Discriminant Canonical Correlation Analysis (UDCCA). In UDCCA, the class membership and mapping are iteratively computed by using the normalized spectral clustering and generalized Eigen value methods alternatively. The experimental results on the MFD dataset and ORL dataset show that UDCCA outperforms traditional CCA and its variants in most situations.

In this paper, an online selective kernel-based temporal difference (OSKTD) learning algorithm is proposed to deal with large scale and/or continuous reinforcement learning problems. OSKTD includes two online procedures: online sparsification and parameter updating for the selective kernel-based value function. A new sparsification method (i.e., a kernel distance-based online sparsification method) is proposed based on selective ensemble learning, which is computationally less complex compared with other sparsification methods. With the proposed sparsification method, the sparsified dictionary of samples is constructed online by checking if a sample needs to be added to the sparsified dictionary. In addition, based on local validity, a selective kernel-based value function is proposed to select the best samples from the sample dictionary for the selective kernel-based value function approximator. The parameters of the selective kernel-based value function are iteratively updated by using the temporal difference (TD) learning algorithm combined with the gradient descent technique. The complexity of the online sparsification procedure in the OSKTD algorithm is O(n). In addition, two typical experiments (Maze and Mountain Car) are used to compare with both traditional and up-to-date O(n) algorithms (GTD, GTD2, and TDC using the kernel-based value function), and the results demonstrate the effectiveness of our proposed algorithm. In the Maze problem, OSKTD converges to an optimal policy and converges faster than both traditional and up-to-date algorithms. In the Mountain Car problem, OSKTD converges, requires less computation time compared with other sparsification methods, gets a better local optima than the traditional algorithms, and converges much faster than the up-to-date algorithms. In addition, OSKTD can reach a competitive ultimate optima compared with the up-to-date algorithms.

The key task in graph-oriented learning is constructing an informative graph to model the geometrical and discriminant structure of a data manifold. Since traditional graph construction methods are sensitive to noise and less datum-adaptive to changes in density, a new graph construction method so-called ℓ1-Graph has been proposed [1] recently. A graph construction method needs to have two important properties: sparsity and locality. However, the ℓ1-Graph is strong in sparsity property, but weak in locality. In order to overcome such limitation, we propose a new method of constructing an informative graph using automatic group sparse regularization based on the work of ℓ1-Graph, which is called as group sparse graph (GroupSp-Graph). The newly developed GroupSp-Graph has the same noise-insensitive property as ℓ1-Graph, and also can successively preserve the group and local information in the graph. In other words, the proposed group sparse graph has both properties of sparsity and locality simultaneously. Furthermore, we integrate the proposed graph with several graph-oriented learning algorithms: spectral embedding, spectral clustering, subspace learning and manifold regularized non-negative matrix factorization. The empirical studies on benchmark data sets show that the proposed algorithms achieve considerable improvement over classic graph constructing methods and the ℓ1-Graph method in various learning task.

In this paper, we proposed a novel real-time abnormal event detection framework that requires a short training period and has a fast processing speed. Our approach is based on phase correlation and our newly developed spatial-temporal co-occurrence Gaussian mixture models (STCOG)with the following steps: (i) a frame is divided into non-overlapping local regions; (ii) phase correlation is used to estimate the motion vectors between successive two frames for all corresponding local regions, and (iii) STCOG is used to model normal events and detect abnormal events if any deviation from the trained STCOG is found. Our proposed approach is also able to update the parameters incrementally and can be applied in complicated scenes. The proposed approach outperforms previous ones in terms of shorter training periods and lower computational complexity.

Unlike conventional automatic continuous speech segmentation models that deal with each boundary time-mark individually, in this paper, we propose an interval-data-based linear regression model for syllable nucleus durations estimation (LRM-DE), which treats syllable boundary time-marks in pairs. This characteristic of LRM-DE makes it more suitable for estimating syllable durations for English sentences, which can be used for sentence stress detection. LRM-DE combines the outcomes of multiple base automatic speech segmentation machines (ASMs) to generate final boundary time-marks that miminize the average distance of the predicted and reference boundary-pairs of syllable nuclei. Experimental results show that on TIMIT dataset, LRM-DE reduces the average difference between the predicted syllable nucleus durations and their reference ones from 13.64 ms (the best result of a single ASM) to 11.81 ms. Also, LRM-DE improves the syllable nucleus segmentation accuracy from 81.59% to 83.98% within a tolerance of 20 ms.