Existing drift detection methods focus on designing sensitive test statistics. They treat the detection threshold as a fixed hyperparameter, set once to balance false alarms and late detections, and applied uniformly across all datasets and over time. However, maintaining model performance is the key objective from the perspective of machine learning, and we observe that model performance is highly sensitive to this threshold. This observation inspires us to investigate whether a dynamic threshold could be provably better. In this paper, we prove that a threshold that adapts over time can outperform any single fixed threshold. The main idea of the proof is that a dynamic strategy, constructed by combining the best threshold from each individual data segment, is guaranteed to outperform any single threshold that apply to all segments. Based on the theorem, we propose a Dynamic Threshold Determination algorithm. It enhances existing drift detection frameworks with a novel comparison phase to inform how the threshold should be adjusted. Extensive experiments on a wide range of synthetic and real-world datasets, including both image and tabular data, validate that our approach substantially enhances the performance of state-of-the-art drift detectors.

Skill discovery has emerged as a popular route for unsupervised reinforcement learning (URL), offering agents a diverse, reusable set of behaviours learned before any task-specific reward is experienced. However, existing methodologies tend to favour either categorical codes or unimodal skill priors, which simplifies training at the cost of limiting the variety of behaviours they can represent. We introduce Discovery of Mixture Skills (DiMS), a URL algorithm that learns a latent Gaussian mixture by training a Gaussian Mixture Variational Autoencoder (GMVAE) in tandem with the unsupervised policy. In DiMS, a hierarchical GMVAE simultaneously discovers clusters of skills, while an auxiliary macro-latent dynamically positions mixture components to prevent mode collapse. A joint loss term combining log-likelihood and curiosity rewards enables simultaneous updates of representation and policy while improving exploration. Experiments on the Unsupervised Reinforcement Learning Benchmark (URLB) show that DiMS consistently outperforms a wide range of state-of-the-art baselines. Ablation studies confirm that the mixture prior is critical to these gains, and that DiMS is robust to alternative exploration bonuses. Overall, our results suggest that Gaussian mixture skill priors offer a compelling foundation for future unsupervised RL.

Recommender systems are widely required and deployed to address real-world problems. In this paper, we study a new yet challenging real-world setting for recommender systems, where only user browsing histories are available without any explicit feedback. No item acquisition information, e.g., purchasing or rating, is given. By assuming that user browsing sequences are likely to contain the items to acquire, we draw an analogy to the setting of partial label learning in weakly supervised learning. This enables us to train reliable recommender systems only using browsing histories. We term the proposed method as Partial Acquisition Recommender System (PARS). Empirical results on real-world benchmark datasets show the effectiveness of the proposed method. Surprisingly, we also show that the proposed method even surpasses some baselines using item acquisition information.

Learning from multiple data streams in real-world scenarios is fundamentally challenging due to intrinsic heterogeneity and unpredictable concept drifts. Existing methods typically assume homogeneous streams and employ static architectures with indiscriminate knowledge fusion, limiting generalizability in complex dynamic environments. To tackle this gap, we propose CAMEL, a dynamic Collaborative Assistance Mixture of Experts Learning framework. It addresses heterogeneity by assigning each stream an independent system with a dedicated feature extractor and task-specific head. Meanwhile, a dynamic pool of specialized private experts captures stream-specific idiosyncratic patterns. Crucially, collaboration across these heterogeneous streams is enabled by a dedicated assistance expert. This expert employs a multi-head attention mechanism to distill and integrate relevant context autonomously from all other concurrent streams. It facilitates targeted knowledge transfer while inherently mitigating negative transfer from irrelevant sources. Furthermore, we propose an Autonomous Expert Tuner (AET) strategy, which dynamically manages expert lifecycles in response to drift. It instantiates new experts for emerging concepts (freezing prior ones to prevent catastrophic forgetting) and prunes obsolete ones. This expert-level plasticity provides a robust and efficient mechanism for online model capacity adaptation. Extensive experiments demonstrate CAMEL’s superior generalizability across diverse multistreams and exceptional resilience against complex concept drifts.

Detecting hallucinations in Retrieval-Augmented Generation (RAG) remains a critical reliability challenge, as ungrounded responses can have severe consequences in high-stakes applications such as clinical decision support, legal research assistants, and autonomous agents that act on retrieved evidence.Prior approaches attribute hallucinations to a binary conflict between internal knowledge stored in FFNs and the retrieved context.However, this perspective is incomplete, failing to account for the impact of other components of the LLM, such as the user query, previously generated tokens, the self token, and the Final LayerNorm adjustment.To comprehensively capture the impact of these components on hallucination detection, we propose TPA which mathematically attributes each token's probability to seven distinct sources: Query, RAG Context, Past Token, Self Token, FFN, Final LayerNorm, and Initial Embedding.This attribution quantifies how each source contributes to the generation of the next token.Specifically, we aggregate these attribution scores by Part-of-Speech (POS) tags to quantify the contribution of each model component to the generation of specific linguistic categories within a response.By leveraging these patterns, such as detecting anomalies where Nouns rely heavily on LayerNorm, TPA effectively identifies hallucinated responses.Extensive experiments on five LLMs (Llama2-7B/13B, Llama3-8B, Mistral-7B, and Qwen3-8B) demonstrate that TPA achieves state-of-the-art performance across diverse architectures.

Aiming at identifying unexpected inputs from unknown classes, out-of-distribution (OOD) detection has emerged as a pivotal approach to enhancing the reliability of machine learning models. This paper focuses on the burgeoning paradigm of post-hoc OOD detection with pre-trained vision-language models (VLMs), where a popular pipeline is to detect OOD inputs by examining their affinities between ID labels and negative labels, i.e., those semantically different from ID labels. Due to the unavailability of target OOD labels, existing works predominantly rely on heuristic rules to mine negative labels from unlabeled wild corpus data. Despite the empirical success, we argue that the power of VLM-based OOD detection has yet to be fully unleashed since the notorious false negative problem is far from addressed in the literature. With this motivation, we are interested in addressing the challenge of mining true negative labels for OOD scoring. To this end, we develop a theoretical framework for correcting the sampling bias of negatives labels by indirectly approximating the distribution of negative labels. Perhaps surprisingly, we show that the debiased negative mining can be naturally converted into Monte-Carlo sampling based on ID labels and the unlabeled wild corpus data. Extensive experiments empirically manifest that our method establishes a new state-of-the-art in a variety of OOD detection setups.

Real-world data streams exhibit inherent non-stationarity characterized by concept drift, posing significant challenges for adaptive learning systems. While existing methods address isolated distribution shifts, they overlook the critical co-evolution of label spaces and distributions under limited supervision and persistent uncertainty. To address this, we formalize Generalized Incremental Learning under Concept Drift (GILCD), characterizing the joint evolution of distributions and label spaces in open-environment streaming contexts, and propose a novel framework called Calibrated Source-Free Adaptation (CSFA). First, CSFA introduces a training-free prototype calibration mechanism that dynamically fuses emerging prototypes with base representations, enabling stable new-class identification without optimization overhead. Second, we design a novel source-free adaptation algorithm, i.e., Reliable Surrogate Gap Sharpness-aware (RSGS) minimization. It integrates sharpness-aware perturbation loss optimization with surrogate gap minimization, while employing entropy-based uncertainty filtering to discard unreliable samples. This mechanism ensures robust distribution alignment and mitigates generalization degradation caused by uncertainties. Thus, CSFA establishes a unified framework for stable adaptation to evolving semantics and distributions in open-world streaming scenarios. Extensive experiments validate the superior performance and effectiveness of CSFA compared to SOTA approaches.

Video desnowing, aimed at removing snowflakes and enhancing the quality of videos, is a crucial yet intricate task essential for improving the effectiveness of outdoor vision systems. Compared to rain and haze, the inherent opacity and diverse morphology of snowflakes result in more pronounced background occlusions, thereby challenging the efficacy of current desnowing techniques, particularly those focusing solely on images or videos captured from a monocular perspective. To address these challenges, this paper proposes a Dual-Stream Temporal Transformer (DSTT) to advance snow removal and visual enhancement by leveraging comprehensive information from stereo views and spatial-temporal cues. More specifically, it incorporates a Dual-Stream Weight-shared Transformer (DSWT) module to exploit spatial information from different views. This module employs a hierarchical weight-sharing strategy to extract fused spatial features across different views from low-level to high-level layers. Subsequently, the Dual-Stream ConvLSTM (DS-CLSTM) module is introduced to capture temporal correlations across streaming frames. By combining temporal-spatial cues and complementary details from diverse views, videos can be effectively restored while preserving the original content’s details. In addition, two binocular snowy datasets – SnowKITTI2012 and SnowKITTI 2015 – are presented, providing a valuable resource for evaluating the binocular desnowing task. Comprehensive experiments evaluated on both synthetic and real-world snowy datasets demonstrate that our proposed method outperforms the state-of-the-art baselines.

Existing object re-identification (re-ID) and composed image retrieval (CIR) methods capture different aspects of real-world retrieval requirements; re-ID preserves identity but cannot specify desired appearance changes, whereas CIR supports attribute-guided retrieval but does not enforce identity consistency. To bridge this gap, we introduce composed object re-identification (CORI), a new task that requires the retrieved target to simultaneously satisfy identity preservation and text-guided attribute modification. This problem is fundamentally different from existing re-ID and CIR settings and has not been explicitly studied before. To make CORI feasible without costly manual annotation, we propose a weakly supervised framework that leverages large language models (LLMs) and visual question answering (VQA) models to automatically generate reference-to-target descriptions using ID labels alone. We further develop the first baseline model tailored for CORI, which jointly learns multimodal composition and identity-aware matching through shared-weight image encoders, a text encoder, and a compositor module optimized by contrastive, ID, and triplet losses. We also establish four CORI benchmark datasets covering person and vehicle retrieval. Experiments show that the proposed method consistently outperforms representative baselines adapted from existing CIR and re-ID methods for the newly introduced CORI setting, improving Rank@1 by 2.1% and 0.8% on RAP and Celeb-reID-light, and by 9.9% and 9.5% on VeRi-776 and VRIC, respectively.

Concept drift—characterized by the evolving and unpredictable nature of data distributions—is a persistent challenge for long-term modeling in non-stationary environments. In multi-stream scenarios, this problem is further complicated by the dynamic and often entangled dependencies across streams, which cannot be effectively handled by traditional single-stream adaptation methods. To address these challenges, we propose a Multi-scale Adaptive Convolutional Graph framework for multi stream concept drift, called MACG. Our framework represents the dynamic inter-dependencies between streams through a flexible and efficient graph structure, driving stable, long-term multi step prediction tasks across multiple streams. In MACG, conventional reliance on pre-defined graphs is replaced by a multi-scale adaptive convolutional graph learned directly from historical data. By considering multiple time scales, this adaptive structure is able to capture richer and more complex spatial–temporal dependencies, thereby overcoming the limitations of static graphs and achieving high generalizability to unknown or evolving data distributions. During online testing, the model adaptively updates its graph structure and proactively predicts drift by identifying structurally analogous patterns between current drift events and historical drift patterns. This joint design effectively mitigates catastrophic forgetting by leveraging both the recurrence and learnability of drift patterns. Rather than merely repeating past responses, the adaptive graph enables the model to generalize transferable patterns and dynamically update dependencies in drift-affected regions, ensuring that adaptations are both context aware and responsive to new changes. Extensive experiments on four large-scale real-world datasets show that MACG consistently outperforms state-of-the-art baselines in prediction accuracy.

Recommender systems often suffer from data sparsity, particularly when user interactions within a single domain are limited. Cross-domain recommender systems (CDRSs) address this challenge by transferring knowledge across related domains. However, existing approaches face two key limitations: 1) intra-domain noise, where skewed or unreliable interactions degrade representation quality and 2) negative transfer, where misaligned knowledge from the source domain harms target-domain performance. To tackle these issues, we propose GCLD-CDR, a novel cross-domain recommendation framework that integrates graph-based contrastive learning (CL) with diffusion-based knowledge transfer. To enhance intra-domain learning, GCLD-CDR incorporates two complementary augmentation modules: a feature perturbation generator that introduces controlled noise to improve representation diversity, and a denoising generator that prunes unreliable graph edges to refine structural signals. To mitigate negative transfer, we design a diffusion-based transfer mechanism that progressively perturbs source-domain user representations via a Gaussian diffusion process. A neural decoder then reverses this process, selectively recovering task-relevant information while filtering out noise and misaligned signals. Extensive experiments on real-world datasets demonstrate that GCLD-CDR consistently outperforms state-of-the-art baselines, underscoring its potential for advancing practical and trustworthy recommender systems.

Unsupervised domain adaptation (UDA) aims to improve a model’s performance on an unlabeled target domain by leveraging labeled data from a source domain. Traditional UDA methods often overlook the rich textual information inherent in class labels, limiting their effectiveness. Recent advances in visual language models (VLMs), especially the contrastive language–image pretraining (CLIP) model, provide a promising multimodal foundation by integrating both image and text representations. However, most CLIP-based UDA approaches rely on pseudo-labels generated in a zero-shot manner by the pretrained CLIP model, which may become suboptimal as training progresses. To address this limitation, we propose a pseudo-label refinement (PuRe) framework that begins training with zero-shot CLIP pseudo-labels and then transitions to self-training, where we iteratively refine the model’s own predictions as pseudo-labels. In this iterative process, we introduce label consistency learning to stabilize predictions under strong data augmentations, increasing robustness, and an information maximization (IM) loss to encourage high-confidence predictions while preserving prediction diversity. Together, these components progressively enhance pseudo-label reliability, leading to improved adaptation in the target domain. Furthermore, we propose a novel prompt learning (PL) method that fine-tunes only a minimal set of parameters, specifically a single context token and class embeddings. Extensive experiments demonstrate that PuRe surpasses existing UDA methods on multiple benchmarks.

Model-based reinforcement learning (MBRL) aims to promote sample efficiency and reduce the number of interactions with the true environment, via learning an environment dynamic model, compared with model-free reinforcement learning (MFRL). However, the success of MBRL heavily relies on two key aspects: model learning and planning. The former refers to learning an accurate model, and the latter aims to improve the behavior policy. In this article, we investigate these two aspects further with model ensemble learning. We design a deep residual attention U-Net (RauNet) with fewer neurons (or weights) than the widely used shallow neural network as our base models and further apply the Hilbert–Schmidt independence criterion (HSIC) as a regularization term to pursue model diversity explicitly for the model ensemble. Furthermore, we propose an adaptive trust region policy optimization (TRPO), in which the parametric Rényi alpha divergence substitutes for the Kullback–Leibler (KL) divergence for measuring the difference between two successive policies, and the alpha value can be adaptively adjusted during TRPO training iterations. This method is called diversity-driven model ensemble adaptive TRPO, or simply diversity-driven model ensemble adaptive trust region policy optimization. Our detailed experiments on six benchmark environments show that our proposed approach is optimal, compared with five state-of-the-art RL techniques.

Concept drift, characterized by unpredictable changes in data distribution over time, poses significant challenges to machine learning models in streaming data scenarios. Although error rate-based concept drift detectors are widely used, they often fail to identify drift in the early stages when the data distribution changes but error rates remain constant. This paper introduces the Prediction Uncertainty Index (PU-index), derived from the prediction uncertainty of the classifier, as a superior alternative to the error rate for drift detection. Our theoretical analysis demonstrates that: (1) The PU-index can detect drift even when error rates remain stable. (2) Any change in the error rate will lead to a corresponding change in the PU-index. These properties make the PU-index a more sensitive and robust indicator for drift detection compared to existing methods. We also propose a PU-index-based Drift Detector (PUDD) that employs a novel Adaptive PU-index Bucketing algorithm for detecting drift. Empirical evaluations on both synthetic and real-world datasets demonstrate PUDD’s efficacy in detecting drift in structured and image data.

Prompt Tuning adapts frozen models to new tasks by prepending a few learnable embeddings to the input. However, it struggles with tasks that suffer from data scarcity. To address this, we explore Cross-Modality Prompt Transfer, leveraging prompts pretrained on a data-rich modality to improve performance on data-scarce tasks in another modality. As a pioneering study, we first verify the feasibility of cross-modality prompt transfer by directly applying frozen source prompts (trained on the source modality) to the target modality task. To empirically study cross-modality prompt transferability, we train a linear layer to adapt source prompts to the target modality, thereby boosting performance and providing ground-truth transfer results. Regarding estimating prompt transferability, existing methods show ineffectiveness in cross-modality scenarios where the gap between source and target tasks is larger. We address this by decomposing the gap into the modality gap and the task gap, which we measure separately to autonomously select the best source prompt for a target task. Additionally, we propose Attention Transfer to further reduce the gaps by injecting target knowledge into the prompt and reorganizing a top-transferable source prompt using an attention block. We conduct extensive experiments involving prompt transfer from 13 source language tasks to 19 target vision tasks under three settings. Our findings demonstrate that: (i) cross-modality prompt transfer is feasible, supported by in-depth analysis; (ii) measuring both the modality and task gaps is crucial for accurate prompt transferability estimation, a factor overlooked by previous studies; (iii) cross-modality prompt transfer can significantly release the powers of prompt tuning on data-scarce tasks, as evidenced by comparisons with a newly released prompt-based benchmark.

Out-of-distribution (OOD) detection has emerged as a pivotal approach for enhancing the reliability of machine learning models, considering the potential for test data to be sampled from classes disparate from in-distribution (ID) data employed during model training. Detecting those OOD data is typically realized as a distance measurement problem, where those deviating far away from the training distribution in the learned feature space are considered OOD samples. Advanced works have shown great success in learning with prototypes for feature-based OOD detection methods, where each ID class is represented with single or multiple prototypes. However, modeling with a finite number of prototypes would fail to maximally capture intra-class variations. In view of this, this paper extends the existing prototype-based learning paradigm to an infinite setting. This motivates us to design two feasible formulations for the Distributional Prototype Learning (DPL) objective, where, to avoid intractable computation and exploding parameters caused by the infinity nature, our key idea is to model an infinite number of discrete prototypes of each ID class with a class-wise continuous distribution. We theoretically analyze both alternatives, identifying the more stable-converging version of the learning objective. We show that, by sampling prototypes from a mixture of class-conditioned Gaussian distributions, the objective can be efficiently computed in a closed form without resorting to the computationally expensive Monte-Carlo approximation of the involved expectation terms. Extensive evaluations across mainstream OOD detection benchmarks empirically manifest that our proposed DPL has established a new state-of-the-art in various OOD settings.

Recent advances in generative models have highlighted the need for robust detectors capable of distinguishing real images from AI-generated images. While existing methods perform well on known generators, their performance often declines when tested with newly emerging or unseen generative models due to overlapping feature embeddings that hinder accurate cross-generator classification. In this paper, we propose Multimodal Discriminative Representation Learning for Generalizable AI-generated Image Detection (MiraGe), a method designed to learn generator-invariant features. Motivated by theoretical insights on intra-class variation minimization and inter-class separation, MiraGe tightly aligns features within the same class while maximizing separation between classes, enhancing feature discriminability. Moreover, we apply multimodal prompt learning to further refine these principles into CLIP, leveraging text embeddings as semantic anchors for effective discriminative representation learning, thereby improving generalizability. Comprehensive experiments across multiple benchmarks show that MiraGe achieves state-of-the-art performance, maintaining robustness even against unseen generators like Sora.

Out-of-distribution (OOD) detection, recognized for its ability to identify samples of unknown classes, provides solid advantages in ensuring the reliability of machine learning models. Among existing OOD detection methods, pre-trained vision-language models have emerged as powerful post-hoc OOD detectors by leveraging textual and visual information. Despite the empirical success, there still remains a lack of research on a formal understanding of their effectiveness. This paper bridges the gap by theoretically demonstrating that existing CLIP-based post-hoc methods effectively perform a stochastic estimation of the point-wise mutual information (PMI) between the input image and each in-distribution label. This estimation is then utilized to construct energy functions for modeling in-distribution distributions. Different from prior methods that inherently consider PMI estimation as a whole task, we, motivated by the divide-and-conquer philosophy, decompose PMI estimation into multiple easier sub-tasks by applying the chain rule of PMI, which not only reduces the estimation complexity but also provably increases the estimation upper bound to reduce the underestimation bias. Extensive evaluations across mainstream benchmarks empirically manifest that our method establishes a new state-of-the-art in a variety of OOD detection setups.

Modern machine learning models struggle to maintain performance in dynamic environments where temporal distribution shifts, \textit{i.e., concept drift}, are prevalent. Temporal Domain Generalization (TDG) seeks to enable model generalization across evolving domains, yet existing approaches typically assume smooth incremental changes, struggling with complex real-world drifts involving both long-term structure (incremental evolution/periodicity) and local uncertainties. To overcome these limitations, we introduce FreKoo, which tackles these challenges through a novel frequency-domain analysis of parameter trajectories. It leverages the Fourier transform to disentangle parameter evolution into distinct spectral bands. Specifically, the low-frequency components with dominant dynamics are learned and extrapolated using the Koopman operator, robustly capturing diverse drift patterns including both incremental and periodic drifts. Simultaneously, potentially disruptive high-frequency variations are smoothed via targeted temporal regularization, preventing overfitting to transient noise and domain uncertainties. In addition, this dual-spectral strategy is rigorously grounded through theoretical analysis, providing stability guarantees for the Koopman prediction, a principled Bayesian justification for the high-frequency regularization, and culminating in a multiscale generalization bound connecting spectral dynamics to improved generalization. Extensive experiments demonstrate FreKoo's significant superiority over state-of-the-art TDG methods, particularly excelling in real-world streaming scenarios with complex drifts and uncertainties.

We study the machine unlearning problem which aims to remove specific training data from a pre-trained machine learning model to allow users to exercise their 'right to be forgotten' to protect user privacy. Conventional machine unlearning methods would degrade the model performance after the unlearning procedure. To mitigate the issue, they typically rely on the access to the remaining training data to fine-tune the unlearned model to mitigate the influence of unlearning. However, accessing the remaining training data may not always be practical for different reasons (e.g., data expiration policies, storage limitations, or additional privacy constraints). Machine unlearning without access to the remaining training data poses significant challenges to retaining model performance. In this paper, we study how to unlearn specific training data from a pre-trained model without accessing the remaining training data and protect model performance without dramatically changing the model's parameters. We propose a practical method called Targeted Label Noise Injection. Intuitively, our method assigns incorrect yet controllable labels to the examples that need to be forgotten and fine-tunes the pre-trained model to learn these new labels. This strategy effectively moves the to-be-forgotten examples across the decision boundary with a small impact on the model's overall performance. We theoretically prove the effectiveness of the proposed method and empirically show that it achieves state-of-the-art unlearning performance across various datasets.

Current genome-wide association studies provide valuable insights into the genetic basis of ischaemic stroke (IS) risk. However, polygenic risk scores, the most widely used method for genetic risk prediction, have notable limitations due to their linear nature and inability to capture complex, nonlinear interactions among genetic variants. While deep neural networks offer advantages in modeling these complex relationships, the multifactorial nature of IS and the influence of modifiable risk factors present additional challenges for genetic risk prediction. To address these challenges, we propose a Chromosome-wise Multi-task Genomic (MetaGeno) framework that utilizes genetic data from IS and five related diseases. The framework includes a chromosome-based embedding layer to model local and global interactions among adjacent variants, enabling a biologically informed approach. Incorporating multi-disease learning further enhances predictive accuracy by leveraging shared genetic information. Among various sequential models tested, the Transformer demonstrated superior performance, and outperformed other machine learning models and PRS baselines, achieving an AUROC of 0.809 on the UK Biobank dataset. Risk stratification identified a two-fold increased stroke risk (HR, 2.14; 95% CI: 1.81-2.46) in the top 1% risk group, with a nearly five-fold increase in those with modifiable risk factors such as atrial fibrillation and hypertension. Finally, the model was validated on the diverse All of Us dataset (AUROC = 0.764), highlighting ancestry and population differences while demonstrating effective generalization. This study introduces a predictive framework that identifies high-risk individuals and informs targeted prevention strategies, offering potential as a clinical decision-support tool.

Neural network models are widely adopted in real-world applications for processing streaming data. However, these applications often face challenges in terms of accuracy degradation, caused by changes in the data distribution of the stream data compared to the training data. Two underlying reasons contribute to these changes. The first, known as the concept drift problem, occurs when there is a change in the correlation between the input data and the prediction output, making the models trained on the training data no longer suitable for the new data. The second reason, known as the novelty problem, arises when real-world data contains unexpected data categories that were not present in the training data, resulting in incorrect predictions. The research community has divided into different groups and each developed various methods to detect either concept drift or novelty distribution changes. However, these methods only address one aspect of the problem and are unable to distinguish between them. This leads to an inappropriate allocation of model maintenance resources, including the high cost of model retraining and the acquisition of true label data. In this study, we aim to address this gap by proposing a novel concept drift detection method that is capable of distinguishing between known labeled concept drift and novelty. Our method is also more efficient than existing drift detection methods, making it suitable for applications on neural networks.

With the advance of powerful computational hardware in recent years, neural networks, once popular for machine learning on pre-collected datasets, are becoming applicable for streaming data processing. Stream data have the characteristics of high velocity, variety and volumes. In stream data mining a common challenge is concept drift, which refers to the phenomenon that the statistical properties of the target variable in predictive tasks change over time. For instance, in image prediction problems, the input facilities may be altered by the environment or device flaws. The materials generated from them could be distorted, such as blurring, discoloring or part-missing. Concept drift problem is considered a root cause of performance degradation in machine learning models on stream data. Traditional concept drift detection methods usually require large amount of history data, which leads to substantial memory footprint and high computational cost, and tend to be over sensitive to arbitrary distribution changes not related to prediction results. Such limitation is particularly manifest in settings using neural network models, where input data are usually high dimensional images or videos. Aiming to improve the accuracy and efficiency of concept drift detection in neural network models, we propose a new concept drift detection method that specifically addresses these limitations and is applicable to general neural network models. Our method represents the original data with a distance-based statistic, extracted from the layer outputs of the neural network models, and is able to adjust its sensitivity to input distribution changes based on their relevance to neural network features. We evaluated our method with popular neural network architectures on both synthetic and real-world data sets. The result showed our method not only outperforms existing concept drift methods in accuracy, but is also significantly faster and consumes less resources.

Legal case recommendation systems face significant challenges in the era of Large Language Models (LLMs). While LLMs offer unprecedented opportunities for understanding legal texts, they also introduce risks through AI-generated false legal content. Our survey reveals concerning gaps in public awareness: 41% of respondents incorrectly believe AI never generates false legal information, while only 6% understand potential legal liabilities. To address these issues, we propose RoSiLC-RS, a Robust Similar Legal Case Recommender System that guides LLMs to understand legal concepts at a higher abstraction level. Our system employs four key components: (1) abstraction processing to extract core legal elements, (2) semantic matching to identify similar case features, (3) LLM-powered explanation generation to provide detailed recommendation rationales, enhancing system explainability, and (4) a specialized detection module to identify and filter AI-generated false content. Comprehensive experiments on real-world legal datasets demonstrate that our method significantly outperforms traditional retrieval approaches in precision, relevance, explainability, and resistance to AI-generated content interference. This research provides both technological solutions and insights for the safe application of LLMs in legal domains.

Out-of-distribution (OOD) detection is crucial in modern deep learning applications, as it can identify OOD data drawn from distributions differing from those of the in-distribution (ID) data. Advanced OOD detection methods primarily rely on post-hoc strategies, which identify OOD data by analyzing the predictions of a model well-trained on ID data. However, deep models are known to be impacted by spurious features such as backgrounds, causing existing OOD detection methods to fail in identifying OOD data that share the same spurious features as ID data. Therefore, this paper studies how to mitigate spurious features to improve OOD detection. To address this challenge, we propose a novel method called N on- s emantic E xploration OOD D etection (NsED), which focuses on exploring and exploiting non-semantic features. In particular, NsED first explores non-semantic features in an OOD generalization manner. These non-semantic features are then used to train deep models to be more robust against spurious features. Through extensive experiments on representative benchmarks, we show that NsED significantly and consistently improves the detection performance of many representative post-hoc OOD detection methods. • NsED is proposed to address spurious correlation in OOD detection. • This paper first explores the link between OOD generalization and detection. • Experiments show NsED is robust and its components improve OOD detection.

Meta-reinforcement learning (meta-RL) algorithms extract task information from experienced context in order to reason about new tasks, and facilitate rapid adaptation. The quality of these contextual representations (or embeddings) is therefore crucial for a meta-RL agent to make effective decisions in unknown environments. Current methods predominantly assume the existence of a single underlying task, but using a single contextual embedding may not be expressive enough to fully capture the broader distribution of task variations that an agent might encounter. Decomposing that information into different representations can allow them to capture more relevant features in context space while applying additional structure that aids downstream exploitation. In this article, we develop global-local embeddings for contextual meta-RL (GLOBEX), an off-policy contextual meta-RL algorithm that decomposes the contextual representation into separate global and local embeddings. The learning process maximizes information retained by the embeddings and utilizes a mutual information constraint to encourage decoupling. Illustrative examples show that our method effectively adapts by identifying global task dynamics and exploiting temporally local signals. In addition, GLOBEX outperforms existing state-of-the-art meta-RL algorithms on standard MuJoCo benchmarks.

In a real-world setting, several correlational data streams are active at once. An essential question is how to use the correlations between data streams to enhance the effectiveness of machine learning models. The fact that data streams are nonstationary and the correlations across data streams might change over time presents another difficulty. We suggest an ensemble chain-structured model, Evolutionary regressor chains (RCs), to track the correlations between data streams to solve these issues. We develop a heuristic order searching approach to search for the chain's optimal order. With the ability to monitor the dynamicity of the correlations, the heuristic order searching technique can also update the chains over time. Furthermore, a way for reducing computing complexity while maintaining the ensemble's diversity is proposed. The method's theoretical foundation is established through a dynamic regret analysis proving optimal adaptation in the data streams. The outcomes of our experiments demonstrate the effectiveness of Evolutionary RCs.

A frequent problem in data stream mining is concept drift, meaning the data distribution changes over time. A common issue when dealing with concept drift is insufficient data. Real-world applications of data stream mining often involve multiple data streams. However, most concept drift methods handle these data streams separately. This study uses data from other data streams to handle the problem of insufficient data. We propose a novel Multistream Concept Drift Handling Framework via data sharing, containing a fuzzy membership-based drift detection (FMDD) component and a fuzzy membership-based drift adaptation (FMDA) component, to train the new learning model for drifting streams by sharing weighted data from other nondrifting streams. A stream fuzzy set is defined with membership functions that measure the degree to which samples belong to a data stream. Our Concept Drift Handling Framework can detect when and in which stream concept drift occurs, and therefore the insufficient data issue can be solved by adding the weighted data from nondrifting streams to train new learning models. Synthetic and real-world experimental results show that our method can help avoid the insufficient data issue and thereby significantly improve the prediction performance.

Unsupervised domain adaptation that relies on data matching can raise privacy concerns when leveraging transferable knowledge from the source domain(s). To address this issue, source-free domain adaptation has been proposed and subsequently developed. However, privacy risks persist due to potential data leaks from model inversion attacks even when using only source models without accessing the source data directly. Alongside this, another underexplored problem is feature heterogeneity across multiple source domains. In this article, we propose a fuzzy domain adaptation method, fuzzy heterogeneous domain adaptation (FuzHDA), that learns fuzzy rules from heterogeneous teacher models, even when these models are black boxes. The proposed method first trains heterogeneous source models privately on individual devices without sharing any information, including neither data nor model parameters. Then, using a memory bank that stores all target predictions from the pretrained source models, we apply a self-knowledge distillation approach to train a target model, which simulates the predictions from these heterogeneous source teachers. After completing the target model, a cross-modality hybrid model leveraging prompt learning and uncovering causal factors is fine-tuned via self-supervision at the last step to facilitate transfer performance. Experiments on real-world datasets demonstrate the superiority of the proposed FuzHDA.

Cross-domain recommender systems, which are designed to address issues with data sparsity, tend to suffer notable challenges with safeguarding user privacy. While existing cross-domain recommendation methods incorporate privacy mechanisms, they often fall short in practice, offering only one-sided benefits and limited privacy safeguards. In this study, we propose a novel privacy-preserving cross-domain recommender system that combines federated transfer learning with differential privacy to facilitate cross-domain knowledge transfer while ensuring strong privacy protection. First, we leverage federated transfer learning, treating each domain as an independent client to protect privacy for business partners by preventing the exchange of raw data. Second, we use a graph neural network (GNN) as the encoder to learn the user and item representations. We also design a consistency loss function that maintains the invariance between local and global user representations while preventing representation collapse. Third, we introduce a privacy mechanism that applies differential privacy to the output of each aggregation layer in the GNN—the aim being to protect transferred user representations while balancing privacy with accuracy. Finally, our transfer mechanism operates without user-identifying information, establishing connections between domains by detecting latent overlapping users and subsequently performing personalized preference aggregation. This allows for efficient knowledge transfer across domains. Experiments on real-world datasets show that our approach significantly enhances recommendation accuracy while offering robust privacy protection, outperforming state-of-the-art baselines.

Model-based reinforcement learning (RL) aims to learn the underlying dynamics of a given environment. The success of most existing works is built on the critical assumption that the dynamic is fixed, which is unrealistic in many open-world scenarios, such as drone delivery and online chatting, where agents may need to deal with environments with unpredictable changing dynamics (hereafter,real non-stationary environment). Therefore, learning changing dynamics in a real non-stationary environment offers both significant benefits and challenges. This paper proposes a new model-based reinforcement learning algorithm that proactively and dynamically detects possible changes and Learns these Latent and Changing Dynamics (LLCD) in a latent Markovian space for real non-stationary environments. To ensure the Markovian property of the RL model and improve computational efficiency, we employ a latent space model to learn the environment’s transition dynamics. Furthermore, we perform online change detection in the latent space to promptly identify change points in non-stationary environments. Then, we utilize the detected information to help the agent adapt to new conditions. Experiments indicate that the rewards of the proposed algorithm accumulate for the most rapid adaptions to environmental change, among other benefits. This work has a strong potential to enhance environmentally suitable model-based reinforcement learning capabilities.

The classification problem concerning crisp-valued data has been well resolved. However, interval-valued data, where all of the observations' features are described by intervals, are also a common data type in real-world scenarios. For example, the data extracted by many measuring devices are not exact numbers but intervals. In this article, we focus on a highly challenging problem called learning from interval-valued data (LIND), where we aim to learn a classifier with high performance on interval-valued observations. First, we obtain the estimation error bound of the LIND problem based on the Rademacher complexity. Then, we give the theoretical analysis to show the strengths of multiview learning on classification problems, which inspires us to construct a new algorithm called multiview interval information extraction (Mv-IIE) approach for improving classification accuracy on interval-valued data. The experiment comparisons with several baselines on both synthetic and real-world datasets illustrate the superiority of the proposed framework in handling interval-valued data. Moreover, we describe an application of Mv-IIE that we can prevent data privacy leakage by transforming crisp-valued (raw) data into interval-valued data.

Cancer risk prediction is a cornerstone of personalized medicine that offers opportunities for early detection and preventive interventions. However, the current models are designed to predict cancer risk face several challenges. First, most rely on traditional statistical methods, which struggle to capture the complexity of genetic, family medical history, and lifestyle factors. Hence, the accuracy of these models is limited. Additionally, the models neglect to integrate multidimensional data sources, particularly genetic information like single nucleotide polymorphisms (SNPs), which could enhance prediction accuracy. Third, while the system might effectively predict risk, it cannot translate those predictions into actionable healthcare recommendations to reduce cancer risk. In this study, we address all three of these limitations. With a focus on six prevalent cancers—we extracted SNP data from the UK Biobank and designed a novel risk prediction model for cancer and personalized healthcare recommendations based upon the mixture of experts (MoE) paradigm and large language models (LLMs), respectively. Named MoE-HRS, experts based two router networks for separate processing by the Transformer and the convolutional neural network (CNN). Experiments on UK Biobank data show that our model outperforms state-of-the-art cancer risk prediction models. To bridge the gap between risk prediction and practical healthcare applications, we devised a healthcare recommender system powered by LLMs. This approach holds promise for enhancing early detection rates and promoting preventive healthcare management (relevant coding and data are available at https://github.com/bjtu-lucas-nlp/MoE-HRS ).

Trust region policy optimization (TRPO) is an iterative reinforcement learning algorithm that both maximizes a surrogate objective based on a generalized advantage function and enforces a trust region constraint between two consecutive policies based on the Kullback–Leibler divergence in each iteration. On the one hand, its surrogate objective only approximates the positive theoretical improvement and thus relaxes the theoretical guarantee. One the other hand, currently, TRPO is approximately solved via a linear approximation for its objective and a quadratic approximation for its constraint, which results in an upper bound for the linear search of the step size. However, this solution strategy does not provide a lower bound for the step size search. Intuitively, a lower bound could be determined manually, but such a setting has no physical meaning. To this end, we present an alternative solution in this article, which is to exchange the objective and the constraint in TRPO according to reciprocal optimization technique, and then derive a reciprocal TRPO. Applying a similar approximation solution of TRPO to our reciprocal TRPO produces a lower bound for the step size search that has a physical interpretation for improving the surrogate objective or return. Further, we aggregate the original TRPO with this reciprocal TRPO to construct a twin TRPO, whose step size has a lower bound from our reciprocal TRPO and an upper bound from TRPO, to facilitate the policy optimization and achieve a least return improvement. Extensive experiments on twelve benchmark environments show that our twin TRPO is superior to six existing techniques: the original TRPO, proximal policy optimization, off-policy TRPO and two entropy regularized TRPOs, as well as our reciprocal TRPO. Our code is available at https://github.com/HTXu-UTS/twinTRPO.

Cross-domain recommendation (CDR) is a promising paradigm inspired by transfer learning to solve the cold-start problem in recommender systems. Existing state-of-the-art CDR methods train an explicit mapping function to transfer the cold-start users from a data-rich source domain to a target domain. However, a limitation of these methods is that the mapping function is trained on overlapping users across domains, while only a small number of overlapping users are available for training. By visualizing the loss landscape of the existing CDR model, we find that training on a small number of overlapping users causes the model to converge to sharp minima, leading to poor generalization. Based on this observation, we leverage loss-geometry-based machine learning approach and propose a novel CDR method called sharpness-aware CDR (SCDR). Our proposed method simultaneously optimizes recommendation loss and loss sharpness, leading to better generalization with theoretical guarantees. Empirical studies on real-world datasets show that SCDR significantly outperforms other CDR models on cold-start recommendation tasks. Additionally, our method enhances the model’s robustness to adversarial attacks.

Multistream classification poses significant challenges due to the necessity for rapid adaptation in dynamic streaming processes with concept drift. Despite the growing research outcomes in this area, there has been a notable oversight regarding the temporal dynamic relationships between these streams, leading to the issue of negative transfer arising from irrelevant data. In this paper, we propose a novel Online Boosting Adaptive Learning (OBAL) method that effectively addresses this limitation by adaptively learning the dynamic correlation among different streams. Specifically, OBAL operates in a dual-phase mechanism, in the first of which we design an Adaptive COvariate Shift Adaptation (AdaCOSA) algorithm to construct an initialized ensemble model using archived data from various source streams, thus mitigating the covariate shift while learning the dynamic correlations via an adaptive re-weighting strategy. During the online process, we employ a Gaussian Mixture Model-based weighting mechanism, which is seamlessly integrated with the acquired correlations via AdaCOSA to effectively handle asynchronous drift. This approach significantly improves the predictive performance and stability of the target stream. We conduct comprehensive experiments on several synthetic and real-world data streams, encompassing various drifting scenarios and types. The results clearly demonstrate that OBAL achieves remarkable advancements in addressing multistream classification problems by effectively leveraging positive knowledge derived from multiple sources.

Knowledge distillation (KD) provides an efficient framework for transferring knowledge from a teacher model to a student model by aligning their predictive distributions. The existing KD methods adopt the same strategy as the teacher to formulate the student’s predictive distribution. However, employing the same distribution-modeling strategy typically causes sub-optimal knowledge transfer due to the discrepancy in model capacity between teacher and student models. Designing student-friendly teachers contributes to alleviating the capacity discrepancy, while it requires either complicated or student-specific training schemes. To cast off this dilemma, we propose to introduce an auxiliary variable to promote the ability of the student to model predictive distribution. The auxiliary variable is defined to be related to target variables, which will boost the model prediction. Specifically, we reformulate the predictive distribution with the auxiliary variable, deriving a novel objective function of KD. Theoretically, we provide insights to explain why the proposed objective function can outperform the existing KD methods. Experimentally, we demonstrate that the proposed objective function can considerably and consistently outperform existing KD methods.

Column generation (CG) is a well-established method for solving large-scale linear programs. It involves iteratively optimizing a subproblem containing a subset of columns and using its dual solution to generate new columns with negative reduced costs. This process continues until the dual values converge to the optimal dual solution to the original problem. A natural phenomenon in CG is the heavy oscillation of the dual values during iterations, which can lead to a substantial slowdown in the convergence rate. *Stabilization* techniques are devised to accelerate the convergence of dual values by using information beyond the state of the current subproblem. However, there remains a significant gap in obtaining more accurate dual values at an earlier stage. To further narrow this gap, this paper introduces a novel approach consisting of 1) a *machine learning* approach for accurate prediction of optimal dual solutions and 2) an *adaptive stabilization* technique that effectively capitalizes on accurate predictions. On the graph coloring problem, we show that our method achieves a significantly improved convergence rate compared to traditional methods.

Unsupervised domain adaptation (UDA) employs labeled data from a source domain to train classifiers for an unlabeled target domain. We utilize Contrastive Language-Image Pre-training (CLIP) models to exploit textual information in labels, enabling simultaneous matching of textual and image features. However, adapting CLIP models for UDA tasks poses a significant challenge and necessitates further investigation. To this end, we introduce CLIP-Enhanced Unsupervised Domain Adaptation with Consistency Regularization, which employs consistency regularization for concurrent training of CLIP’s prompts and image adapters. Our approach, particularly under consistency regularization, incorporates data augmentation to enhance the model’s generalization capability. During training, we maintain consistent pseudo-labels for target domain data, regardless of whether weak or strong augmentation techniques are applied. This strategy improves our model’s robustness in adapting to various domains. Additionally, the integration of domain-specific prompts and image adapters in our model optimizes the learning of domain-related textual and image features. Experiments on real-world datasets substantiate the effectiveness of our proposed method. The outcomes illustrate its superior performance compared to existing techniques across multiple benchmarks.

Domain adaptation with inaccurate label is a challenging and interesting topic in transfer learning, dealing with source and target domains with shift label spaces. Most existing domain adaptation methods assume aware label distributions among source and target domains. However, this cannot always be guaranteed in reality. Furthermore, existing multi-domain adaptation methods rarely deal with label heterogeneity among source domains. Thus, in this paper, we propose a multi-source domain adaptation method handling Inaccurate label (IncLabDA) during transfer. The proposed method designs a module that can transfer knowledge from multi-source domains with both homogeneous and heterogeneous label spaces in universal scenario. Anchors are generated from pre-trained model to build data-matching via a contrastive method avoiding to referring original data. In addition, class center consistency combined with clustering strategy considering both global and local confidences is adopted to recognize out-of-distribution samples. By removing source private classes and target unknown samples, highly confident target samples are collected to self-supervise the adaptation. At the same time, constraints enlarging the distance among target known classes and between the known and unknown samples are applied to enhance the performance of the proposed model. Experiments on real-world datasets validate the superiority of the IncLabDA model.

Credit risk classification is a crucial area in machine learning-enhanced financial decision support systems, and numerous studies have achieved significant progress. However, data in the modern financial landscape is inherently complex, characterized by a lack of labeled data, cross-domain heterogeneity, and class imbalance. These three major problems hinder the achievement of credit risk classification. Therefore, we propose a semi-supervised heterogeneous domain adaptation method and an imbalanced data augmentation method to overcome these challenges with a single neural network-based model. Experimental results obtained from four representative benchmark datasets confirm the superior performance of the proposed method.

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

The aim of unsupervised domain adaptation (UDA) is to utilize knowledge from a source domain to enhance the performance of a given target domain. Due to the lack of accessibility to the target domain's labels, UDA's efficacy is highly reliant on the source domain's quality. However, it is often impractical and expensive to obtain an appropriate transferable source domain. To address this issue, we propose a novel UDA setting, source domain reconstruction (SDR), which seeks to construct a new transferable source domain utilizing labeled source samples and unlabeled target samples. SDR has a significant advantage over the conventional method as it is much less expensive to construct a suitable pseudo-source domain rather than collecting an actual transferable source domain in real-world scenarios. To test the practice of SDR, we investigate SDR theoretically. We propose an easily implementable algorithm, the domain MixUp (DMU), which is motivated by the MixUp strategy, to solve the SDR problem. The algorithm can be used to design a UDA framework to significantly enhance the performance of several existing UDA algorithms. Results from extensive experiments conducted on seven benchmarks (66 UDA tasks) indicate that the reconstructed source domain has stronger transferability than the original source domain.

Unsupervised multidomain adaptation attracts increasing attention as it delivers richer information when tackling a target task from an unlabeled target domain by leveraging the knowledge attained from labeled source domains. However, it is the quality of training samples, not just the quantity, that influences transfer performance. In this article, we propose a multidomain adaptation method with sample and source distillation (SSD), which develops a two-step selective strategy to distill source samples and define the importance of source domains. To distill samples, the pseudo-labeled target domain is constructed to learn a series of category classifiers to identify transfer and inefficient source samples. To rank domains, the agreements of accepting a target sample as the insider of source domains are estimated by constructing a domain discriminator based on selected transfer source samples. Using the selected samples and ranked domains, transfer from source domains to the target domain is achieved by adapting multilevel distributions in a latent feature space. Furthermore, to explore more usable target information which is expected to enhance the performance across domains of source predictors, an enhancement mechanism is built by matching selected pseudo-labeled and unlabeled target samples. The degrees of acceptance learned by the domain discriminator are finally employed as source merging weights to predict the target task. Superiority of the proposed SSD is validated on real-world visual classification tasks.

Deep reinforcement learning (DRL) is a powerful tool for learning from interactions within a stationary environment where state transition and reward distributions remain constant throughout the process. Addressing the practical but challenging nonstationary environments with time-varying state transition or reward function changes during the interactions, ingenious solutions are essential for the stability and robustness of DRL agents. A key assumption to cope with nonstationary environments is that the change points between the previous and the new environments are known beforehand. Unfortunately, this assumption is impractical in many cases, such as outdoor robots and online recommendations. To address this problem, this article presents a robust DRL algorithm for nonstationary environments with unknown change points. The algorithm actively detects change points by monitoring the joint distribution of states and actions. A detection boosted, gradient-constrained optimization method then adapts the training of the current policy with the supporting knowledge of formerly well-trained policies. The previous policies and experience help the current policy adapt rapidly to environmental changes. Experiments show that the proposed method accumulates the highest reward among several alternatives and is the fastest to adapt to new environments. This work has compelling potential for increasing the environmental suitability of intelligent agents, such as drones, autonomous vehicles, and underwater robots.

The Gaussian process (GP) algorithm is considered as a powerful nonparametric-learning approach, which can provide uncertainty measurements on the predictions. The standard GP requires clearly observed data, unexpected perturbations in the input may lead to learned regression model mismatching. Besides, GP also suffers from the lack of good generalization performance guarantees. To deal with data uncertainty and provide a numerical generalization performance guarantee on the unknown data distribution, this article proposes a novel robust noisy input GP (NIGP) algorithm based on the probably approximately correct (PAC) Bayes theory. Furthermore, to reduce the computational complexity, we develop a sparse NIGP algorithm, and then develop a sparse PAC-Bayes NIGP approach. Compared with NIGP algorithms, instead of maximizing the marginal log likelihood, one can optimize the PAC-Bayes bound to pursue a tighter generalization error upper bound. Experiments verify that the NIGP algorithms can attain greater accuracy. Besides, the PAC-NIGP algorithms proposed herein can achieve both robust performance and improved generalization error upper bound in the face of both uncertain input and output data.

The theoretical analysis of multiclass classification has proved that the existing multiclass classification methods can train a classifier with high classification accuracy on the test set, when the instances are precise in the training and test sets with same distribution and enough instances can be collected in the training set. However, one limitation with multiclass classification has not been solved: how to improve the classification accuracy of multiclass classification problems when only imprecise observations are available. Hence, in this article, we propose a novel framework to address a new realistic problem called multiclass classification with imprecise observations (MCIMO), where we need to train a classifier with fuzzy-feature observations. First, we give the theoretical analysis of the MCIMO problem based on fuzzy Rademacher complexity. Then, two practical algorithms based on support vector machine and neural networks are constructed to solve the proposed new problem. The experiments on both synthetic and real-world datasets verify the rationality of our theoretical analysis and the efficacy of the proposed algorithms.

Concept drift arises from the uncertainty of data distribution over time and is common in data stream. While numerous methods have been developed to assist machine learning models in adapting to such changeable data, the problem of improperly keeping or discarding data samples remains. This may results in the loss of valuable knowledge that could be utilized in subsequent time points, ultimately affecting the model's accuracy. To address this issue, a novel method called time segmentation-based data stream learning method (TS-DM) is developed to help segment and learn the streaming data for concept drift adaptation. First, a chunk-based segmentation strategy is given to segment normal and drift chunks. Building upon this, a chunk-based evolving segmentation (CES) strategy is proposed to mine and segment the data chunk when both old and new concepts coexist. Furthermore, a warning level data segmentation process (CES-W) and a high-low-drift tradeoff handling process are developed to enhance the generalization and robustness. To evaluate the performance and efficiency of our proposed method, we conduct experiments on both synthetic and real-world datasets. By comparing the results with several state-of-the-art data stream learning methods, the experimental findings demonstrate the efficiency of the proposed method.

Unsupervised domain adaptation leverages knowledge from source domain(s)/task(s) to facilitate learning in target task, particularly in unsatisfied and complex scenarios with data scarcity and distribution shifts. This approach helps reduce the high costs associated with collecting or labeling data for the target domain. However, it raises privacy concerns due to its matching techniques requiring access to source data, particularly in sensitive applications. In addition, most domain adaptation methods assume that source and target domains share the same label space, disregarding category shifts. In this article, we propose federated fuzzy transfer learning for category shifts (FdFTL) to address the before mentioned challenges-data privacy and category shifts. By combining a hybrid approach of fuzzy model and federated learning, a cloud model capable of performing across domains can be trained without the need for data sharing. This approach also results in a reduction of model parameters compared to traditional methods training individual models from multiple source domains. To eliminate domain and category shifts, we utilize a global clustering and a local semantic consensus clustering to effectively separate known target classes from out-of-distribution samples. Furthermore, we incorporate a confident score and the Silhouette analysis to elaborate the accuracy of categorizing target known classes. Experimental results on real-world visual tasks across universal, open-set, partial, and closed-set scenarios demonstrate the effectiveness of our proposed method.

Machine learning draws its power from various disciplines, including computer science, cognitive science, and statistics. Although machine learning has achieved great advancements in both theory and practice, its methods have some limitations when dealing with complex situations and highly uncertain environments. Insufficient data, imprecise observations, and ambiguous information/relationships can all confound traditional machine learning systems. To address these problems, researchers have integrate machine leaning from different aspects, and fuzzy techniques including fuzzy sets, fuzzy systems, fuzzy logic, fuzzy measures, fuzzy relations, and so on. This paper presents a systematic review of fuzzy machine learning, from theory, approach to application, with the overall objective of providing an overview of recent achievements in the field of fuzzy machine learning. To this end, the concepts and frameworks discussed are divided into five categories: (a) fuzzy classical machine learning; (b) fuzzy transfer learning; (c) fuzzy data stream learning; (d) fuzzy reinforcement learning; and (e) fuzzy recommender systems. The literature presented should provide researchers with a solid understanding of the current progress in fuzzy machine learning research and its applications.

Unsupervised Domain Adaptation (UDA) focuses on enhancing the model performance on an unlabeled target domain by leveraging knowledge from a source domain. The source and target domains usually share different distributions. Existing UDA research primarily concentrates on image data characterized by crisp-valued features. However, interval-valued data, where all the observations’ features are described by intervals, is also a common type of data in real-world scenarios. For instance, measurement instruments are unable to provide exact numerical outcomes, instead employing intervals to describe their results. Hence, this paper focuses on the highly challenging context known as domain adaptation with interval-valued observations. In this environment, the objective is to improve classification accuracy within an unlabeled target domain by capitalizing on knowledge gleaned from a labeled source domain, where both domains exclusively feature interval-valued observations. To address this, we first establish an upper bound on the risk in the interval-valued target domain, underpinning our analysis with rigorous theoretical insights. Subsequently, guided by our theoretical analysis, a new model based on Takagi-Sugeno Fuzzy rules and a Self-supervised Pseudo-labeling strategy (SP-TSF) is developed to address the proposed problem. Takagi-Sugeno fuzzy rules are harnessed to handle the inherent uncertainty intrinsic to interval-valued data, while a pseudo-labeling strategy is developed to augment distribution alignment between the source and target domains, each characterized by interval-valued observations. Extensive experiments on both synthetic and realworld datasets verify the rationality of our theoretical analysis and the efficacy of the proposed model.

Multi-source domain adaptation (MSDA) refers to the task of adapting a model from multiple source domains to a target domain that shares a different distribution with all source domains. However, most existing MSDA works focus on crispvalued data, while such data may not be available in some realworld scenarios. For example, data extracted by many measuring devices are not exact numbers but rather intervals. In this paper, a highly challenging problem called MSDA with interval-valued target data is presented. The objective is to learn a new model for interval-valued target data by leveraging knowledge from source models trained on multiple crisp-valued source data. First, a theoretical analysis is given to inform the appropriate combination of multi-source models. Then, we propose a new neural network model based on a fuzzy transformation function and fuzzy distances to address the proposed problem. The fuzzy transformation function is applied to extract valuable crisp-valued information from interval-valued target data, while fuzzy distances are designed to guide the fusion of multiple source models. Experiments on both synthetic and real-world datasets verify the superiority of our proposed MSDA method for classification task. Furthermore, the results of the ablation study and parameter sensitivity analysis illustrate the rationality of the proposed fuzzy distance-based model.

Unsupervised Domain Adaptation (UDA) addresses the challenge of distribution shift between a labeled source domain and an unlabeled target domain by utilizing knowledge from the source. Traditional UDA methods mainly focus on single-modal scenarios, either vision or language, thus not fully exploring the advantages of multi-modal representations. Vision-Language models utilize multi-modal information, applying prompt learning techniques for addressing target domain tasks. Motivated by the recent advancements in pre-trained Vision-Language models, this paper expands the UDA framework to incorporate multi-modal approaches using fuzzy techniques. The adoption of fuzzy techniques, preferred over conventional domain adaptation methods, is based on two key aspects: the nature of prompt learning is intrinsically linked to fuzzy logic, and the superior capability of fuzzy techniques in processing soft information and effectively utilizing inherent relationships both within and across domains. To this end, we propose UDA enhanced byFuzzy PromptLearning (FUZZLE), a simple and effective method for aligning the source and target domains via domain-specific prompt learning. Specifically, we introduce a novel technique to enhance prompt learning in the target domain. This method integrates fuzzy C-means clustering and a novel instance-level fuzzy vector into the prompt learning loss function, minimizing the distance between prompt cluster centers and instance prompts, thereby enhancing the prompt learning process. In addition, we propose a Kullback-Leibler (KL) divergence-based loss function with a fuzzification factor. This function is designed to minimize the distribution discrepancy in the classification of similar cross-domain data, aligning domain-specific prompts during the training process. We contribute an in-depth analysis to understand the effectiveness of Extensive experiments demonstrate that our method achieves superior performance on standard UDA benchmarks.

Multistream classification aims to predict the target stream by transferring knowledge from labeled source streams amid nonstationary processes with concept drifts. While existing methods address label scarcity, covariate shift, and asynchronous concept drift, they focus solely on the original feature space, neglecting the influence of redundant or low-quality features with uncertainties. Therefore, the advancement of this task is still challenged by how to: 1) ensure guaranteed joint representations of different streams, 2) grapple with uncertainty and interpretability during knowledge transfer, and 3) track and adapt the asynchronous drifts in each stream. To address these challenges, we propose an interpretable fuzzy shared representation learning (FSRL) method based on the Takagi–Sugeno–Kang (TSK) fuzzy system. Specifically, FSRL accomplishes the nonlinear transformation of individual streams by learning the fuzzy mapping with the antecedents of the TSK fuzzy system, thereby effectively preserving discriminative information for each original stream in an interpretable way. Then, a multistream joint distribution adaptation algorithm is proposed to optimize the consequent part of the TSK fuzzy system, which learns the final fuzzy shared representations for different streams. Hence, this method concurrently investigates both the commonalities across streams and the distinctive information within each stream. Following that, window-based and GMM-based online adaptation strategies are designed to address the asynchronous drifts over time. The former can directly demonstrate the effectiveness of FSRL in knowledge transfer across multiple streams, while the GMM-based method offers an informed way to overcome the asynchronous drift problem by integrating drift detection and adaptation. Finally, extensive experiments on several synthetic and real-world benchmarks with concept drift demonstrate the proposed method's effectiveness and efficiency.

Recommender systems are one of the most successful applications of using AI for providing personalized e-services to customers. However, data sparsity is presenting enormous challenges that are hindering the further development of advanced recommender systems. Although cross-domain recommendation partly overcomes data sparsity by transferring knowledge from a source domain with relatively dense data to augment data in the target domain, the current methods do not handle heterogeneous data very well. For example, using today’s cross-domain transfer learning schemes with data comprising clicks, ratings, user reviews, item metadata, and knowledge graphs will likely result in a poorly performing model. User preferences will not be comprehensively profiled, and accurate recommendations will not be generated. To solve these three challenges—handling heterogeneous data, avoiding negative transfer, and dealing with data sparsity—we designed a new end-to-end deep A dversarial M ulti-channel T ransfer network for C ross- D omain R ecommendation named AMT-CDR . Heterogeneous data is handled by constructing a cross-domain graph based on real-world knowledge graphs—we used Freebase and YAGO. Negative transfer is prevented through an adversarial learning strategy that maintains consistency across the different data channels. Data sparsity is addressed with an end-to-end neural network that considers data across multiple channels and generates accurate recommendations by leveraging knowledge from both the source and target domains. Extensive experiments on three dual-target cross-domain recommendation tasks demonstrate the superiority of AMT-CDR compared to eight state-of-the-art methods. All source code is available at https://github.com/bjtu-lucas-nlp/AMT-CDR .

Concept drift is a phenomenon that the distribution of data streams changes with time. When this happens, model predictions become less accurate. Hence, concept drift needs to be detected and adapted. Existing drift detection methods are good at determining when drift has occurred, but few retrieve information about how the drift came to be present in the stream, i.e., what type of drift has occurred. Hence, discussing the impact of the type of drift on adaptation is a difficult thing. To fill this gap, we propose a pre-trained framework for training a drift detector called a type-driven lite concept drift detector (Type-LDD) that retrieves information about both when and how a drift has occurred. In our proposed pre-trained framework, the Type-LDD including a drift-type identifier and a drift-point locator was based on a synthetic dataset containing a range of drift types. When repurposing the pre-trained model for detecting new data streams, a knowledge distillation module fine-tunes the proposed Type-LDD to speed up inference and keep detection accuracy. The proposed Type-LDD is validated on both synthetic data and real-world data, and demonstrated that accurately identifying the type of drift that has occurred can improve adaptation accuracy.

Concept drift is an inevitable problem in non-stationary data stream environments, due to changes in data distribution over time. In practical applications, multi-stream data is more common and complex than single-stream data, yet they have received little attention. Addressing the concept drift problem while mining correlations between data streams has become a significant challenge. Several research works focus on capturing the correlation between streams using graph neural networks (GNNs), which provides valuable insights. However, these methods fix the correlation graph structure after training and are unable to adapt to the new data distribution with dynamic correlations during testing. To bridge this gap, we propose a novel concept drift self-adaptation framework based on dynamic graph regularization for multi-stream, named Multi-stream Self-adaptation based on Graph Regularization (MSGR). A new graph neural network architecture is proposed to capture deep spatio-temporal correlations and learn a correlation graph structure without any pre-defined graphs. Each node on the graph represents a stream. The correlation graph structure is constructed through Gumbel sampling and an adaptive matrix from the perspective of stream pairs. Thus we attain a high-performance GNN as the base prediction model for the multi-stream multi-step prediction task in the testing stage. To adapt to the new data distribution, we design a self-adaptation mechanism performed by assigning dynamic learning weight for newly arriving samples. Intuitively, we should assign larger learning weights for relevant samples when drift occurs. The self-adaptation process is accomplished by the sub-graph updating and the proposed graph regularization. Error-based drift detection is integrated into the framework. When drift is detected, the weight for sub-graph updating is increased by adjusting the regularization coefficient. In this way, regardless of the type and degree of concept drift occurring on one or more streams, MSGR can achieve high self-adaptation performance and provide accurate prediction results consistently. The comprehensive testing results on both real-world and synthetic datasets show that MSGR can achieve state-of-the-art performance.

Unsupervised video prediction aims to predict future outcomes based on the observed video frames, thus removing the need for supervisory annotations. This research task has been argued as a key component of intelligent decision-making systems, as it presents the potential capacities of modeling the underlying patterns of videos. Essentially, the challenge of video prediction is to effectively model the complex spatiotemporal and often uncertain dynamics of high-dimensional video data. In this context, an appealing way of modeling spatiotemporal dynamics is to explore prior physical knowledge, such as partial differential equations (PDEs). In this article, considering real-world video data as a partly observed stochastic environment, we introduce a new stochastic PDE predictor (SPDE-predictor), which models the spatiotemporal dynamics by approximating a generalized form of PDEs while dealing with the stochasticity. A second contribution is that we disentangle the high-dimensional video prediction into low-level dimensional factors of variations: time-varying stochastic PDE dynamics and time-invariant content factors. Extensive experiments on four various video datasets show that SPDE video prediction model (SPDE-VP) outperforms both deterministic and stochastic state-of-the-art methods. Ablation studies highlight our superiority driven by both PDE dynamics modeling and disentangled representation learning and their relevance in long-term video prediction.

Out-of-distribution (OOD) detection is crucial to modern deep learning applications by identifying and alerting about the OOD samples that should not be tested or used for making predictions. Current OOD detection methods have made significant progress when in-distribution (ID) and OOD samples are drawn from static distributions. However, this can be unrealistic when applied to real-world systems which often undergo continuous variations and shifts in ID and OOD distributions over time. Therefore, for an effective application in real-world systems, the development of OOD detection methods that can adapt to these dynamic and evolving distributions is essential. In this paper, we propose a novel and more realistic setting called continuously adaptive out-of-distribution (CAOOD) detection which targets on developing an OOD detection model that enables dynamic and quick adaptation to a new arriving distribution, with insufficient ID samples during deployment time. To address CAOOD, we develop meta OOD learning (MOL) by designing a learning-to-adapt diagram such that a good initialized OOD detection model is learned during the training process. In the testing process, MOL ensures OOD detection performance over shifting distributions by quickly adapting to new distributions with a few adaptations. Extensive experiments on several OOD benchmarks endorse the effectiveness of our method in preserving both ID classification accuracy and OOD detection performance on continuously shifting distributions.

Unsupervised Domain adaptation (UDA) aims to train a target classifier by using massive accurate annotated source domain data and unlabeled target domain data. However, collecting massive accurate annotation can be labor-intensive, and even impractical especially when source domain training data shows label ambiguity. In this paper, we consider a novel domain adaptation setting where the model can be trained with partial-labeled source domain data, so that the cost of data labeling can be reduced. To alleviate the ambiguity induced by partial label, we propose a one-step domain adaptation approach trained from the partial-labeled source data and unlabeled target data. Our approach consists of two components: a feature extractor equipped with partial label loss that learns discriminative representation, and a domain classifier that learns domain-invariant representation across the source and target domains. Extensive experiments have shown that the proposed approach significantly outperforms a series of competitive baselines.

Despite the fact that Recommender Systems have been researched and developed for quite a while, they still hold some certain challenges, two of which are Cold Start and Data Sparsity. Being introduced as a savior, the Cross-domain Recommender system (CDR) owns the capacity to transfer knowledge across domains which makes it become the practical solution for the mentioned issues. In CDR approaches, the family of graph-based solutions is very effective, which builds graphs to illustrate the relationship between users, items, and other factors and learn their representations through graph representation learning. However, most graph-based approaches focus on extracting either domain-specific or domain-shared features and do not have the mechanism to prevent the transfer of private features, which can degrade the quality of vector representations. Moreover, the knowledge transfer process built on overlapping users makes the model biased toward these users, thus downgrading the performance on cold-start users. This paper proposes a meta-GRS framework that uses Graph Neural Network with Meta-Learning and Adversarial Learning for cross-domain recommendation. In meta-GRS, the representative user and item embeddings are improved thanks to the features extracted by the private and cross-domain graphs. Domain-shared features are learned under adversarial learning such that the domain discriminator is unable to determine whether the domain they came from can ensure the positive transfer. To optimize the model performance effectively, we use a Dynamic Weight Averaging algorithm to learn loss weights automatically. The model's parameters are optimized under the optimization-based meta-learning method provides our model the capability of generalization to the new users. Experiments on practical datasets illustrate that the meta-GRS leads the chart in the comparison of other state-of-the-art baselines in recommendation accuracy.

Domain adaptation is a practicable tool in real world application where there exists data scarcity. Multi-source domain adaptation attracts increasing attention due to its ability to enrich transfer knowledge by combining information from multiple domains. However, knowledge transfer can trigger privacy concerns by accessing source data. In addition, multiple domains can have label heterogeneity problem. In this paper, to solve the mentioned problems, we conduct an incomplete multi-source domain adaptation (IMSDA) method which can address transfer learning with and without the access to source data. As far as we are aware, this is the first work handling source-free incomplete domain adaptation. To take the benefits of multiple sources, multi-task learning is adopted to learn a general source model which can perform on multiple domains. A data matching strategy with and without source data forcing target sample to source latent feature space is developed to combine with self-supervision to adapt source model to the target domain. Experiments on real-world datasets indicate the superiority of the proposed method.

Data stream learning has received more and more attention in recent years, change tracking and real-time prediction of data streams under uncertainty have been highly focused. With the development of the information age, more and more different types of data streams have been generated, bringing challenges to the research in this field. Among them, text data streams, as one of the categories, also need to be mined and predicted in real-time. This paper addresses this problem by proposing a topic change recognition-based method (TCR-M) for data stream learning, thus helping support text data stream learning. We first propose a topic change recognition process that extracts the topics of the text data stream at each time point, tracks and determines the severity of the topic change, and locates the time points when significant changes occur. Next, an ensemble learning model is constructed and a separate base learner is simultaneously trained to correct the prediction results of the ensemble learning model, which is updated based on the topic change recognition results. To verify the effectiveness of the method, a number of text data streams are collected for evaluation, then outputting the topic change recognition results and prediction results. By comparing it with benchmark methods, the proposed method shows its efficiency. In future research, further improvements are needed for learning and application.

Abstract Predicting a researcher's knowledge trajectories beyond their current foci can leverage potential inter‐/cross‐/multi‐disciplinary interactions to achieve exploratory innovation. In this study, we present a method of diffusion‐based network analytics for knowledge trajectory recommendation. The method begins by constructing a heterogeneous bibliometric network consisting of a co‐topic layer and a co‐authorship layer. A novel link prediction approach with a diffusion strategy is then used to capture the interactions between social elements (e.g., collaboration) and knowledge elements (e.g., technological similarity) in the process of exploratory innovation. This diffusion strategy differentiates the interactions occurring among homogeneous and heterogeneous nodes in the heterogeneous bibliometric network and weights the strengths of these interactions. Two sets of experiments—one with a local dataset and the other with a global dataset—demonstrate that the proposed method is prior to 10 selected baselines in link prediction, recommender systems, and upstream graph representation learning. A case study recommending knowledge trajectories of information scientists with topical hierarchy and explainable mediators reveals the proposed method's reliability and potential practical uses in broad scenarios.

Semi-supervised heterogeneous domain adaptation (SsHeDA) aims to train a classifier for the target domain, in which only unlabeled and a small number of labeled data are available. This is done by leveraging knowledge acquired from a heterogeneous source domain. From algorithmic perspectives, several methods have been proposed to solve the SsHeDA problem; yet there is still no theoretical foundation to explain the nature of the SsHeDA problem or to guide new and better solutions. Motivated by compatibility condition in semi-supervised probably approximately correct (PAC) theory, we explain the SsHeDA problem by proving its generalization error - that is, why labeled heterogeneous source data and unlabeled target data help to reduce the target risk. Guided by our theory, we devise two algorithms as proof of concept. One, kernel heterogeneous domain alignment (KHDA), is a kernel-based algorithm; the other, joint mean embedding alignment (JMEA), is a neural network-based algorithm. When a dataset is small, KHDA's training time is less than JMEA's. When a dataset is large, JMEA is more accurate in the target domain. Comprehensive experiments with image/text classification tasks show KHDA to be the most accurate among all non-neural network baselines, and JMEA to be the most accurate among all baselines.

In nonstationary environments, data distributions can change over time. This phenomenon is known as concept drift, and the related models need to adapt if they are to remain accurate. With gradient boosting (GB) ensemble models, selecting which weak learners to keep/prune to maintain model accuracy under concept drift is nontrivial research. Unlike existing models such as AdaBoost, which can directly compare weak learners' performance by their accuracy (a metric between [0, 1]), in GB, weak learners' performance is measured with different scales. To address the performance measurement scaling issue, we propose a novel criterion to evaluate weak learners in GB models, called the loss improvement ratio (LIR). Based on LIR, we develop two pruning strategies: 1) naive pruning (NP), which simply deletes all learners with increasing loss and 2) statistical pruning (SP), which removes learners if their loss increase meets a significance threshold. We also devise a scheme to dynamically switch between NP and SP to achieve the best performance. We implement the scheme as a concept drift learning algorithm, called evolving gradient boost (LIR-eGB). On average, LIR-eGB delivered the best performance against state-of-the-art methods on both stationary and nonstationary data.

Unsupervised domain adaptation deals with a task from an unlabeled target domain by leveraging the knowledge gained from labeled source domain(s). The fuzzy system is adopted in domain adaptation to better tackle the uncertainty caused by information scarcity in the transfer. Most existing fuzzy and nonfuzzy domain adaptation methods depend on data-level distribution matching to eliminate the domain shift. However, data sharing can trigger privacy concerns. This situation results in the unavailability of source data, wherein most domain adaptation methods cannot be applied. Source-free domain adaptation is then proposed to handle this problem. But the existing source-free domain adaptation methods rarely deal with any soft information component due to data imprecision. Besides, fewer methods handle multiple source domains that provide richer transfer information. Thus, in this article, we propose source-free multidomain adaptation with fuzzy rule-based deep neural networks, which takes advantage of a fuzzy system to handle data uncertainty in domain adaptation without source data. To learn source private models with high generality, which is important to collect low-noise pseudotarget labels, auxiliary tasks are designed by jointly training source models from multiple domains, which share source parameters and fuzzy rules while protecting source data. To transfer fuzzy rules and fit source private parameters to the target domain, self-supervised learning and anchor-based alignment are built to force target data into source feature spaces. Experiments on real-world datasets under both homogeneous and heterogeneous label space scenarios are carried out to validate the proposed method. The results indicate the superiority of the proposed fuzzy rule-based source-free multidomain adaptation method.

Unsupervised domain adaptation leverages the previously gained knowledge from a labeled source domain to tackle the task from a different but similar unlabeled target domain. Most existing methods focus on transferring knowledge from a single source domain, but the information from a single domain may be inadequate to complete the target task. Some previous studies have turned to multi-view representations to enrich the transferable information. However, they simply concatenate multi-view features, which may result in information redundancy. In this paper, we propose a dynamic classifier alignment (DCA) method for multi-source domain adaptation, which aligns classifiers driven from multi-view features via a sample-wise automatic way. As proposed, both the importance of each view and the contribution of each source domain are investigated. To determine the important degrees of multiple views, an importance learning function is built by generating an auxiliary classifier. To learn the source combination parameters, a domain discriminator is developed to estimate the probability of a sample belonging to multiple source domains. Meanwhile, a self-training strategy is proposed to enhance the cross-domain ability of source classifiers with the assistance of pseudo target labels. Experiments on real-world visual datasets show the superiority of the proposed DCA.

Data streams may encounter data distribution changes, which can significantly impair the accuracy of models. Concept drift detection tracks data distribution changes and signals when to update models. Many drift detection methods apply thresholds to distinguish between drift or non-drift streams and to claim their method outperforms others with non-aligned drift thresholds. We consider that selecting a proper drift threshold could be more important than developing a new drift detection algorithm, and different drift detection algorithms may end up with very similar performance with aligned drift thresholds. To better understand this process, we propose a novel threshold selection algorithm to align the drift thresholds of a set of algorithms so that they are all at the same sensitivity level. Based on comprehensive experiment evaluations, we observed that several state-of-the-art drift detection algorithms could achieve similar results by aligning their thresholds, providing a novel insight to explain how drift detection algorithms contribute to data stream learning. We noticed that a higher detection sensitivity improves accuracy for data streams with frequent distribution change. The evaluation results are showing that drift thresholds should not be fixed during stream learning. Rather, they should adjust dynamically based on the prevailing conditions of the data stream.

Data sparsity is a common issue for most recommender systems and can severely degrade the usefulness of a system. One of the most successful solutions to this problem has been cross-domain recommender systems. These frameworks supplement the sparse data of the target domain with knowledge transferred from a source domain rich with data that is in some way related. However, there are three challenges that, if overcome, could significantly improve the quality and accuracy of cross-domain recommendation: 1) ensuring latent feature spaces of the users and items are both maximally matched; 2) taking consideration of user-item relationship and their interaction in modelling user preference; 3) enabling a two-way cross-domain recommendation that both the source and the target domains benefit from a knowledge exchange. Hence, in this paper, we propose a novel deep neural network called Dual Adversarial network for Cross-Domain Recommendation (DA-CDR). By training the shared encoders with a domain discriminator via dual adversarial learning, the latent feature spaces for both the users and items are maximally matched between the source and target domains. The domain-specific encoders are applied with an orthogonal constraint to ensure that any domain-specific features are properly extracted and work as supplement to the shared features. Allowing the two domains to collaboratively benefit from each other results in better recommendations for both domains. Extensive experiments with real-world datasets on six tasks demonstrate that DA-CDR significantly outperforms seven state-of-the-art baselines in terms of recommendation accuracy.

Concept drift is the phenomenon where the data distribution in a data stream changes over time. It is a ubiquitous problem in the real-world, for example, a traffic accident would cause a jam in a certain period, leading to a distribution change in traffic speed. Most research in the concept drift field focuses on single data stream, however, few of them consider multi-stream environments which are more in line with the application needs. To fill this gap, we propose a multi-stream prediction setting and a multi-stream concept drift self-adaptation framework using graph neural network, named SAGN. In SAGN, we reconsider the learning procedure of GNN-based predictors from an aspect of concept drift adaptation for multi-stream. By this design, the prediction task is converted into online streaming data tasks in sub-graphs. Each sub-graph corresponds to an adaptation target and will be updated over time. In this way, locally we can overcome drift in each sub-graph by a designed adaptation technique, and globally the correlation between different data streams is well-preserved as a graph structure. Therefore, whether drift occurs or not, in one or several streams, SAGN can provide consistently accurate prediction results. We comprehensively tested SAGN on both synthetic and real-world, drift and non-drift data in the multi-step prediction task. The experiment results show that SAGN is able to achieve state-of-the-art performance in most cases.

Causal inference from observational data lies at the heart of education, healthcare, optimal resource allocation and many other decision-making processes. Most of existing methods estimate the target treatment effect indirectly by inferring the underlying treatment response functions or the unobserved counterfactual outcome for every individual. These learning methods are subject to issues of model misspecification and high variability. As a complement of existing indirect learning methods, we propose a direct learning framework, called HTENet, for causal inference using deep multi-task learning. It is based on a novel empirical -risk for learning the causal effect model of direct interest in a supervised learning scheme. In our proposed framework, the target treatment effect model is parametrized as a neural network and learned jointly with other auxiliary models in an end-to-end manner. Moreover, we extend the nave HTENet into other two variants, HTENet-Simple and HTENet-Reg, by further incorporating shared representation learning layers and a propensity prediction regularizer. Experiments on simulated and real data demonstrate that the performances of the proposed methods match or are better than that of existing state-of-arts. Moreover, by learning the target treatment effect function directly, the proposed methods tend to obtain more stable estimates than existing methods.

To solve the user data sparsity problem, which is the main issue in generating user preference prediction, cross-domain recommender systems transfer knowledge from one source domain with dense data to assist recommendation tasks in the target domain with sparse data. However, data are usually sparsely scattered in multiple possible source domains, and in each domain (source/target) the data may be heterogeneous, thus it is difficult for existing cross-domain recommender systems to find one source domain with dense data from multiple domains. In this way, they fail to deal with data sparsity problems in the target domain and cannot provide an accurate recommendation. In this article, we propose a novel multidomain recommender system (called HMRec) to deal with two challenging issues: 1) how to exploit valuable information from multiple source domains when no single source domain is sufficient and 2) how to ensure positive transfer from heterogeneous data in source domains with different feature spaces. In HMRec, domain-shared and domain-specific features are extracted to enable the knowledge transfer between multiple heterogeneous source and target domains. To ensure positive transfer, the domain-shared subspaces from multiple domains are maximally matched by a multiclass domain discriminator in an adversarial learning process. The recommendation in the target domain is completed by a matrix factorization module with aligned latent features from both the user and the item side. Extensive experiments on four cross-domain recommendation tasks with real-world datasets demonstrate that HMRec can effectively transfer knowledge from multiple heterogeneous domains collaboratively to increase the rating prediction accuracy in the target domain and significantly outperforms six state-of-the-art non-transfer or cross-domain baselines.

In a data stream, concept drift refers to unpredictable distribution changes over time, which violates the identical-distribution assumption required by conventional machine learning methods. Current concept drift adaptation techniques mostly focus on a data stream with changing distributions. However, since each variable of a data stream is a time series, these variables normally have temporal dependency problems in the real world. How to solve concept drift and temporal dependency problems at the same time is rarely discussed in the concept-drift literature. To solve this situation, this article proves and validates that the testing error decreases faster if a predictor is trained on a temporally reconstructed space when drift occurs. Based on this theory, a novel drift adaptation regression (DAR) framework is designed to predict the label variable for data streams with concept drift and temporal dependency. A new statistic called local drift degree (LDD+) is proposed and used as a drift adaptation technique in the DAR framework to discard outdated instances in a timely way, thereby guaranteeing that the most relevant instances will be selected during the training process. The performance of DAR is demonstrated by a set of experimental evaluations on both synthetic data and real-world data streams.

In the unsupervised open set domain adaptation (UOSDA), the target domain contains unknown classes that are not observed in the source domain. Researchers in this area aim to train a classifier to accurately: 1) recognize unknown target data (data with unknown classes) and 2) classify other target data. To achieve this aim, a previous study has proven an upper bound of the target-domain risk, and the open set difference, as an important term in the upper bound, is used to measure the risk on unknown target data. By minimizing the upper bound, a shallow classifier can be trained to achieve the aim. However, if the classifier is very flexible [e.g., deep neural networks (DNNs)], the open set difference will converge to a negative value when minimizing the upper bound, which causes an issue where most target data are recognized as unknown data. To address this issue, we propose a new upper bound of target-domain risk for UOSDA, which includes four terms: source-domain risk,$\epsilon $-open set difference ($\Delta _\epsilon $), distributional discrepancy between domains, and a constant. Compared with the open set difference,$\Delta _\epsilon $is more robust against the issue when it is being minimized, and thus we are able to use very flexible classifiers (i.e., DNNs). Then, we propose a new principle-guided deep UOSDA method that trains DNNs via minimizing the new upper bound. Specifically, source-domain risk and$\Delta _\epsilon $are minimized by gradient descent, and the distributional discrepancy is minimized via a novel open set conditional adversarial training strategy. Finally, compared with the existing shallow and deep UOSDA methods, our method shows the state-of-the-art performance on several benchmark datasets, including digit recognition [modified National Institute of Standards and Technology database (MNIST), the Street View House Number (SVHN), U.S. Postal Service (USPS)], object recognition (Office-31, Office-Home), and face recognition [pose, illumination, and expression (PIE)].

Unsupervised domain adaptation transfers gained knowledge from labeled source domain(s) to a similar unlabeled target domain by eliminating the domain shifts. Most existing domain adaptation methods require the access to source data to match the source and target distributions. However, data privacy concerns make it difficult or impossible to share source data, leading to failures in existing domain adaptation methods. Admittedly, a few previous studies deal with domain adaptation without source data, but they rarely pay heed to data free domain adaptation with multiple source domains containing richer knowledge. In this paper, we propose a new multi-source data-free domain adaptation method- generally auxiliary model training (GAM)- which fits the source models to the target domain under the supervision of pseudo target labels rather than matching data distributions. To collect high-quality initial pseudo target labels, our approach learns both specific and general source models to improve the generality of source models based on auxiliary learning. Going further, we introduce a class balanced coefficient of each category based on the number of samples to reduce the misclassification often caused by data imbalance. Experiments on real-world classification datasets show that the propsosed generally auxiliary training has a superiority over the baselines.

Concept drift refers to changes in the underlying data distribution of data streams over time. A well-trained model will be outdated if concept drift occurs. Once concept drift is detected, it is necessary to understand where the drift occurs to support the drift adaptation strategy and effectively update the outdated models. This process, called drift understanding, has rarely been studied in this area. To fill this gap, this article develops a drift region-based data sample filtering method to update the obsolete model and track the new data pattern accurately. The proposed method can effectively identify the drift region and utilize information on the drift region to filter the data sample for training models. The theoretical proof guarantees the identified drift region converges uniformly to the real drift region as the sample size increases. Experimental evaluations based on four synthetic datasets and two real-world datasets demonstrate our method improves the learning accuracy when dealing with data streams involving concept drift.

Quadratic programming is the process of solving a special type of mathematical optimization problem. Recent advances in online solutions for quadratic programming problems (QPPs) have created opportunities to widen the scope of applications for support vector regression (SVR). In this vein, efforts to make SVR compatible with streaming data have been met with substantial success. However, streaming data with concept drift remain problematic because the trained prediction function in SVR tends to drift as the data distribution drifts. Aiming to contribute a solution to this aspect of SVR's advancement, we have developed continuous SVR (C-SVR) to solve regression problems with nonstationary streaming data, that is, data where the optimal input-output prediction function can drift over time. The basic idea of C-SVR is to continuously learn a series of input-output functions over a series of time windows to make predictions about different periods. However, strikingly, the learning process in different time windows is not independent. An additional similarity term in the QPP, which is solved incrementally, threads the various input-output functions together by conveying some learned knowledge through consecutive time windows. How much learned knowledge is transferred is determined by the extent of the concept drift. Experimental evaluations with both synthetic and real-world datasets indicate that C-SVR has better performance than most existing methods for nonstationary streaming data regression.

As a type of evolving-fuzzy system, the evolving-fuzzy-neuro (EFN) system uses the structure inspired by neural networks to determine its parameters (fuzzy sets and fuzzy rules), so EFN system can inherit the advantages of neural networks. However, for streaming data regression, EFN systems still have several drawbacks: determining fuzzy sets is not robust to data sequence; determining fuzzy rules is complex as subspaces that can approximate to a Takagi–Sugeno–Kang (TSK) rule need to be obtained, and many parameters need to be optimized; and it is difficult to detect and adapt to changes in the data distribution, i.e., concept drift, if the output is a continuous variable. Hence, in this article, a novel evolving-fuzzy-neuro system, called the topology learning-based fuzzy random neural network (TLFRNN), is proposed. In TLFRNN, an online topology learning algorithm is designed to self-organize each layer of TLFRNN. Different from current EFN systems, TLFRNN learns multiple fuzzy sets to reduce the impact of noises on each fuzzy set, and a randomness layer is designed, which assigning the probability of each fuzzy set. Also, TLFRNN does not utilize TSK rules; instead uses a simple inference that considers fuzzy and random information of data simultaneously. More importantly, in TLFRNN, concept drift can be detected and adapted easily and rapidly. The experiments demonstrate that TLFRNN achieves superior performance compared to other EFSs.

When a train leaves a platform, knowing the carriage load (the number of passengers in each carriage) of this train will support train managers to guide passengers at the next platform to choose carriages to avoid congestion. This capacity has become critical since the onset of the pandemic. However, with the dynamicity of passengers and the speed of trains improved (about 3 minutes travel between stations) as well as the station stop period reduced (60–90 second per station), the real-time prediction is more challenging. This paper presents an intelligent system, which is developed in collaboration with Sydney Trains, for real-time predicting carriage load across a city passenger train network. The system comprises three innovations. First, a fuzzy time-matching method significantly improves prediction accuracy in the uncertain situations and allows noisy historical data to be used for training. Second, the LightGBM model is extended with an incremental learning scheme to make forecasting in real-time possible. Third, a new multi-stream learning strategy that merges data streams with similar concept drift patterns is pioneered to increase the amount of suitable training data while reducing generalization errors. A comprehensive suite of practical tests on real-world datasets demonstrates the merit of these solutions.

Since support vector regression (SVR) is a flexible regression algorithm, its computational complexity does not depend on the dimensionality of the input space, and it has excellent generalization capability. However, a central assumption with SVRs is that all the required data is available at the time of construction, which means these algorithms cannot be used with data streams. Incremental SVR has been offered as a potential solution, but its accuracy suffers with noise and learning speeds are slow. To overcome these two limitations, we propose a novel incremental regression algorithm, called online robust support vector regression (ORSVR). ORSVR solves nonparallel bound functions simultaneously. Hence, the large quadratic programming problem (QPP) in classical v-SVR are decomposed into two smaller QPPs. An incremental learning algorithm then solves each QPP step-by-step. The results of a series of comparative experiments demonstrate that the ORSVR algorithm efficiently solves regression problems in data streams, with or without noise, and speeds up the learning process.

Transfer learning becomes an attractive technology to tackle a task from a target domain by leveraging previously acquired knowledge from a similar domain (source domain). Many existing transfer learning methods focus on learning one discriminator with single-source domain. Sometimes, knowledge from single-source domain might not be enough for predicting the target task. Thus, multiple source domains carrying richer transferable information are considered to complete the target task. Although there are some previous studies dealing with multi-source domain adaptation, these methods commonly combine source predictions by averaging source performances. Different source domains contain different transferable information; they may contribute differently to a target domain compared with each other. Hence, the source contribution should be taken into account when predicting a target task. In this article, we propose a novel multi-source contribution learning method for domain adaptation (MSCLDA). As proposed, the similarities and diversities of domains are learned simultaneously by extracting multi-view features. One view represents common features (similarities) among all domains. Other views represent different characteristics (diversities) in a target domain; each characteristic is expressed by features extracted in a source domain. Then multi-level distribution matching is employed to improve the transferability of latent features, aiming to reduce misclassification of boundary samples by maximizing discrepancy between different classes and minimizing discrepancy between the same classes. Concurrently, when completing a target task by combining source predictions, instead of averaging source predictions or weighting sources using normalized similarities, the original weights learned by normalizing similarities between source and target domains are adjusted using pseudo target labels to increase the disparities of weight values, which is desired to improve the performance of the final target predictor if the predictions of sources exist significant difference. Experiments on real-world visual data sets demonstrate the superiorities of our proposed method.

In concept drift adaptation, we aim to design a blind or an informed strategy to update our best predictor for future data at each time point. However, existing informed drift adaptation methods need to wait for an entire batch of data to detect drift and then update the predictor (if drift is detected), which causes adaptation delay. To overcome the adaptation delay, we propose a sequentially updated statistic, called drift-gradient to quantify the increase of distributional discrepancy when every new instance arrives. Based on drift-gradient, a segment-based drift adaptation (SEGA) method is developed to online update our best predictor. Drift-gradient is defined on a segment in the training set. It can precisely quantify the increase of distributional discrepancy between the old segment and the newest segment when only one new instance is available at each time point. A lower value of drift-gradient on the old segment represents that the distribution of the new instance is closer to the distribution of the old segment. Based on the drift-gradient, SEGA retrains our best predictors with the segments that have the minimum drift-gradient when every new instance arrives. SEGA has been validated by extensive experiments on both synthetic and real-world, classification and regression data streams. The experimental results show that SEGA outperforms competitive blind and informed drift adaptation methods.

In unsupervised domain adaptation (UDA), a classifier for the target domain is trained with massive true-label data from the source domain and unlabeled data from the target domain. However, collecting true-label data in the source domain can be expensive and sometimes impractical. Compared to the true label (TL), a complementary label (CL) specifies a class that a pattern does not belong to, and hence, collecting CLs would be less laborious than collecting TLs. In this article, we propose a novel setting where the source domain is composed of complementary-label data, and a theoretical bound of this setting is provided. We consider two cases of this setting: one is that the source domain only contains complementary-label data [completely complementary UDA (CC-UDA)] and the other is that the source domain has plenty of complementary-label data and a small amount of true-label data [partly complementary UDA (PC-UDA)]. To this end, a complementary label adversarial network (CLARINET) is proposed to solve CC-UDA and PC-UDA problems. CLARINET maintains two deep networks simultaneously, with one focusing on classifying the complementary-label source data and the other taking care of the source-to-target distributional adaptation. Experiments show that CLARINET significantly outperforms a series of competent baselines on handwritten digit-recognition and object-recognition tasks.

With the continuous generation of huge volumes of streaming data, streaming data regression has become more complicated. A regressor that predicts two or more outputs, i.e., multioutput regression, is commonly used in many applications. However, current multioutput regressors use a batch method to handle data, which presents compatibility issues for streaming data as they need to be analyzed online. To address this issue, we present a multioutput regression system, called MORStreaming, for streaming data. MORStreaming uses an instance-based model to make predictions because this model can quickly adapt to change by only storing new instances or by throwing away old instances. However, learning instances in our regression system are constrained by online demand and need to consider the relationship between outputs. Therefore, MORStreaming consists of two algorithms: 1) an online algorithm based on topology networks which is designed to learn the instances and 2) an online algorithm based on adaptive rules which is designed to learn the correlation between outputs automatically. Experiments involving both artificial and real-world datasets indicate MORStreaming can achieve superior performance compared with other multioutput methods.

Unsupervised domain adaptation (UDA) aims to train a target classifier with labeled samples from the source domain and unlabeled samples from the target domain. Classical UDA learning bounds show that target risk is upper bounded by three terms: source risk, distribution discrepancy, and combined risk. Based on the assumption that the combined risk is a small fixed value, methods based on this bound train a target classifier by only minimizing estimators of the source risk and the distribution discrepancy. However, the combined risk may increase when minimizing both estimators, which makes the target risk uncontrollable. Hence the target classifier cannot achieve ideal performance if we fail to control the combined risk. To control the combined risk, the key challenge takes root in the unavailability of the labeled samples in the target domain. To address this key challenge, we propose a method named E-MixNet. E-MixNet employs enhanced mixup, a generic vicinal distribution, on the labeled source samples and pseudo-labeled target samples to calculate a proxy of the combined risk. Experiments show that the proxy can effectively curb the increase of the combined risk when minimizing the source risk and distribution discrepancy. Furthermore, we show that if the proxy of the combined risk is added into loss functions of four representative UDA methods, their performance is also improved.

Unsupervised domain adaptation provides a variety of methods to leverage the previously gained knowledge from a labeled source domain to help complete a task from a similar unlabeled target domain. Many existing methods focus on transferring knowledge across single source and single target domains, while few studies deal with multi-source domain adaptation, which is more realistic and challengeable. Existing multi-source domain adaptation methods rarely consider the uncertainty of the transformed knowledge resulting from limited information in target domain. A fuzzy system allows imprecision and ambiguity within transfer, thus it can deal with problems with uncertainty. This work proposes a multi-source domain adaptation method with fuzzy-rule based deep neural networks (MDAFuz). The proposed method first extracts multi-view adapted features and pre-trains source classifiers. Using the learned features and classifiers, training samples are then split into multiple clusters, hence fuzzy rules can be built to learn new classifiers. At the same time, the cluster discriminator is trained to define the membership. Finally, by measuring the similarities among source and target domains using the pseudo target labels and a domain discriminator, the target task is completed by combining all source classifiers with regard to the learned weights. The experiment results on real-world visual datasets show the superiority of the proposed method.

In the problem of multi-class classification, researchers have proved that we can train a classifier that has good performance on the test set, as long as the training and test sets are precisely drawn from the same distribution and the size of the training set approaches infinity. However, in a realworld situation, such precise observations are often unavailable in some cases. For example, readings on analogue measurement equipment are not precise numbers but intervals since there is only a finite number of decimals available. Hence, in this paper, we propose a more realistic problem called learning from imprecise observations (LIMO), where we train a classifier with fuzzy observations (i.e., fuzzy vectors). We prove the estimation error bound of this novel problem based on the distribution of fuzzy random variables. This bound demonstrates that we can always learn the best classifier when we have infinite fuzzy observations. We also develop a practical algorithm to train a classifier using fuzzy observations. The experiment results verify the efficacy of our theory and algorithm.

Traditional supervised learning aims to train a classifier in the closed-set world, where training and test samples share the same label space. In this paper, we target a more challenging and re_x0002_alistic setting: open-set learning (OSL), where there exist test samples from the classes that are unseen during training. Although researchers have designed many methods from the algorith_x0002_mic perspectives, there are few methods that pro_x0002_vide generalization guarantees on their ability to achieve consistent performance on different train_x0002_ing samples drawn from the same distribution. Motivated by the transfer learning and probably approximate correct (PAC) theory, we make a bold attempt to study OSL by proving its general_x0002_ization error-given training samples with size n, the estimation error will get close to order Op(1/$\sqrt{}$n). This is the first study to provide a generalization bound for OSL, which we do by theoretically investigating the risk of the tar_x0002_get classifier on unknown classes. According to our theory, a novel algorithm, called auxiliary open-set risk (AOSR) is proposed to address the OSL problem. Experiments verify the efficacy of AOSR. The code is available at github.com/AnjinLiu/Openset_Learning_AOSR.

Spatial and temporal data such as multiple data streams often have concept drift problems, which refers to changes of the data distributions over time. Once concept drift occurs, a stationary machine learning predictor will probably be invalid because the testing data have a different data distribution from that of the training data. Recent studies in data streams aim to address this issue by concept drift adaptation techniques. Concept drift adaptation methods update the predictor by time and have been validated to provide accurate real-time prediction for a single data stream. However, handling multiple relevant data streams is still an unsolved challenge in this area, considering many real-world applications generate multiple data streams that are simultaneously evolving. To fill this gap, we here present a predicting network for multiple data streams, named MuNet. MuNet leverages the dependency between data streams to efficiently lower down the computational cost of real-time predictions for multiple data streams. In MuNet, an online-learned Bayesian neural network (BNN) is designed as a connector between streams. The BNN-connector can continuously use the real-time information from only one base stream to correct the stationary predictor for the other streams, which avoids the high cost caused by repeated adaptation.

The data stream poses additional challenges to statistical classification tasks because distributions of the training and target samples may differ as time passes. Such a distribution change in streaming data is called concept drift. Numerous histogram-based distribution change detection methods have been proposed to detect drift. Most histograms are developed on the grid-based or tree-based space partitioning algorithms which makes the space partitions arbitrary, unexplainable, and may cause drift blind spots. There is a need to improve the drift detection accuracy for the histogram-based methods with the unsupervised setting. To address this problem, we propose a cluster-based histogram, called equal intensity k -means space partitioning (EI-kMeans). In addition, a heuristic method to improve the sensitivity of drift detection is introduced. The fundamental idea of improving the sensitivity is to minimize the risk of creating partitions in distribution offset regions. Pearson's chi-square test is used as the statistical hypothesis test so that the test statistics remain independent of the sample distribution. The number of bins and their shapes, which strongly influence the ability to detect drift, are determined dynamically from the sample based on an asymptotic constraint in the chi-square test. Accordingly, three algorithms are developed to implement concept drift detection, including a greedy centroids initialization algorithm, a cluster amplify-shrink algorithm, and a drift detection algorithm. For drift adaptation, we recommend retraining the learner if a drift is detected. The results of experiments on the synthetic and real-world datasets demonstrate the advantages of EI-kMeans and show its efficacy in detecting concept drift.

Top-k words selection is a technique used to detect and return the k most similar words to a given word from a candidate set. This is a crucial and widely used tool in various tasks. The key issue in top-k words selection is how to measure the similarity between words. One popular and effective solution is to use a word embedding-based similarity measure, which represents words as low-dimensional vectors and measures the similarities between words according to the similarity of the vectors, using a metric. However, most word embedding methods only consider the local proximity properties of two words in a corpus. To mitigate this issue. In this article, we propose to use association rules for measuring word similarity at a global level, and a fuzzy similarity measure for top-k words selection that jointly encodes the local and the global similarities. Experiments on a real-world query task with three benchmark datasets, i.e., TREC-disk 4&5, WT10G, and RCV1, demonstrate the efficiency of the proposed method compared to several state-of-the-art baselines.

In data streams, the data distribution of arriving observations at different time points may change—a phenomenon called concept drift. While detecting concept drift is a relatively mature area of study, solutions to the uncertainty introduced by observations with missing values have only been studied in isolation. No one has yet explored whether or how these solutions might impact drift detection performance. We, however, believe that data imputation methods may actually increase uncertainty in the data rather than reducing it. We also conjecture that imputation can introduce bias into the process of estimating distribution changes during drift detection, which can make it more difficult to train a learning model. Our idea is to focus on estimating the distance between observations rather than estimating the missing values, and to define membership functions that allocate observations to histogram bins according to the estimation errors. Our solution comprises a novel masked distance learning (MDL) algorithm to reduce the cumulative errors caused by iteratively estimating each missing value in an observation and a fuzzy-weighted frequency (FWF) method for identifying discrepancies in the data distribution. The concept drift detection algorithm proposed in this article is a singular and unified algorithm that can handle missing values, but not an imputation algorithm combined with a concept drift detection algorithm. Experiments on both synthetic and real-world datasets demonstrate the advantages of this method and show its robustness in detecting drift in data with missing values. The results show that compared to the best-performing algorithm that handles imputation and drift detection separately, MDL-FWF reduced the average drift detection difference from 10.75% to 5.83%. This is a nearly 46% improvement. These findings reveal that missing values exert a profound impact on concept drift detection, but using fuzzy set theory to model observations can produce more reliable results than imputation.

In unsupervised domain adaptation (UDA), a classifier for a target domain is trained with labeled source data and unlabeled target data. Existing UDA methods assume that the source data come from the same source domain (i.e., single-source scenario) or from multiple source domains, whose feature spaces have the same dimension (homogeneous) but different distributions (i.e., multihomogeneous-source scenario). However, in the real world, for a specific target domain, we probably have multiple different-dimension (heterogeneous) source domains, which do not satisfy the assumption of existing UDA methods. To remove this assumption and move forward to a realistic UDA problem, this article presents a shared-fuzzy-equivalence-relation neural network (SFERNN) for addressing the multisource heterogeneous UDA problem. The SFERNN is a five-layer neural network containing$c$source branches and one target branch. The network structure of the SFERNN is first confirmed by a novel fuzzy relation called multisource shared fuzzy equivalence relation. Then, we optimize parameters of the SFERNN via minimizing cross-entropy loss on$c$source branches and the distributional discrepancy between each source branch and the target branch. Experiments distributed across eight real-world datasets are conducted to validate the SFERNN. This testing regime demonstrates that the SFERNN outperforms the existing single-source heterogeneous UDA methods, especially when the target domain contains few data.

Online data stream mining is of great significance in practice because of its ubiquity in many real-world scenarios, especially in the big data era. Traditional data mining algorithms cannot be directly applied to data streams due to (1) the possible change of underlying data distribution over time (i.e., concept drift ) and (2) delayed, short, or even no labels for streaming data in practice. A new research area, named unsupervised concept drift detection , has emerged to tackle this difficulty mainly based on two-sample hypothesis tests, such as the Kolmogorov–Smirnov test. However, it is surprising that none of the existing methods in this area exploit the Bayesian nonparametric hypothesis test, which has clear interpretability and straightforward prior knowledge encoding ability and no strict or unrealistic requirement of prefixing the form for the underlying data distribution. In this article, we present a Bayesian nonparametric unsupervised concept drift detection method based on the Polya tree hypothesis test. The basic idea is to decompose the underlying data distribution into a multi-resolution representation that transforms the whole distribution hypothesis test into recursive and simple binomial tests. Also, an incremental mechanism is especially designed to improve its efficiency in the stream setting. The method effectively detect drifts, and it also locates where a drift happens and the posteriors of hypotheses. The experiments on synthetic data verify the desired properties of the proposed method, and the experiments on real-world data show the better performance of the method for data stream mining compared with its frequentist counterpart in the literature.

The aim of unsupervised domain adaptation is to leverage the knowledge in a labeled (source) domain to improve a model's learning performance with an unlabeled (target) domain-the basic strategy being to mitigate the effects of discrepancies between the two distributions. Most existing algorithms can only handle unsupervised closed set domain adaptation (UCSDA), i.e., where the source and target domains are assumed to share the same label set. In this article, we target a more challenging but realistic setting: unsupervised open set domain adaptation (UOSDA), where the target domain has unknown classes that are not found in the source domain. This is the first study to provide learning bound for open set domain adaptation, which we do by theoretically investigating the risk of the target classifier on unknown classes. The proposed learning bound has a special term, namely, open set difference, which reflects the risk of the target classifier on unknown classes. Furthermore, we present a novel and theoretically guided unsupervised algorithm for open set domain adaptation, called distribution alignment with open difference (DAOD), which is based on regularizing this open set difference bound. The experiments on several benchmark data sets show the superior performance of the proposed UOSDA method compared with the state-of-the-art methods in the literature.

Concept drift refers to changes in the distribution of underlying data and is an inherent property of evolving data streams. Ensemble learning, with dynamic classifiers, has proved to be an efficient method of handling concept drift. However, the best way to create and maintain ensemble diversity with evolving streams is still a challenging problem. In contrast to estimating diversity via inputs, outputs, or classifier parameters, we propose a diversity measurement based on whether the ensemble members agree on the probability of a regional distribution change. In our method, estimations over regional distribution changes are used as instance weights. Constructing different region sets through different schemes will lead to different drift estimation results, thereby creating diversity. The classifiers that disagree the most are selected to maximize diversity. Accordingly, an instance-based ensemble learning algorithm, called the diverse instance-weighting ensemble (DiwE), is developed to address concept drift for data stream classification problems. Evaluations of various synthetic and real-world data stream benchmarks show the effectiveness and advantages of the proposed algorithm.

In kernel non-parametric two-sample test, we aim to determine whether two sets of precise observations (i.e., samples) are from the same distribution based on a selected kernel. However, in real world, precise observations may be unavailable. For example, readings on an analogue measurement equipment are not precise numbers but intervals since there is only a finite number of decimals available. Hence, we consider a new and more realistic problem setting-two-sample test on imprecise observations. We show that the test power of existing kernel two- sample tests will drop significantly if they do not take care of the vagueness of the imprecise observations, and to this end, we propose a fuzzy-based maximum mean discrepancy (F-MMD), a powerful two-sample test on imprecise observations. F-MMD is based on a novel fuzzy-based kernel function that can measure the discrepancy between two imprecise observations. This novel kernel function takes care of the vagueness of the imprecise observations and its parameters are optimized to maximize the approximate test power of F-MMD. Experiments demonstrate that F-MMD significantly outperforms competitive two-sample test methods when facing imprecise observations.

How to handle concept drift problem is a big challenge for algorithms designed for the data streams. Currently, techniques related to the concept drift problem focus on single data stream. However, it normally needs to handle multiple relevant data streams in the real-world application. Current concept drift methods can not be directly used in the multistream setting. They can only be limitedly applied on each stream separately, which omits the drift correlation between streams. In the multi-stream scenario, when drift occurs in a stream, other streams may face or have faced a similar drift problem as well. This pattern of simultaneous or delayed occurrence of drift is critical to analyze and predict multiple streams as a whole dynamic system. To fill the gap in the multi-stream scenario, this paper proposes a fuzzy drift variance (FDV) to measure the correlated drift patterns among streams. FDA is able to present how the pattern of drift occurrence for any two streams correlates and how delayed this correlation is. Seven synthetic streams are designed to validate FDA. The experimental results show a good presentation ability of FDA for drift-correlated multiple streams.

We propose a class of kernel-based two-sample tests, which aim to determine whether two sets of samples are drawn from the same distribution. Our tests are constructed from kernels parameterized by deep neural nets, trained to maximize test power. These tests adapt to variations in distribution smoothness and shape over space, and are especially suited to high dimensions and complex data. By contrast, the simpler kernels used in prior kernel testing work are spatially homogeneous, and adaptive only in lengthscale. We explain how this scheme includes popular classifier-based two-sample tests as a special case, but improves on them in general. We provide the first proof of consistency for the proposed adaptation method, which applies both to kernels on deep features and to simpler radial basis kernels or multiple kernel learning. In experiments, we establish the superior performance of our deep kernels in hypothesis testing on benchmark and real-world data. The code of our deep-kernel-based two-sample tests is available at github.com/fengliu90/DK-for-TST.

In unsupervised domain adaptation (UDA), classifiers for the target domain are trained with massive true-label data from the source domain and unlabeled data from the target domain. However, it may be difficult to collect fully-true-label data in a source domain given limited budget. To mitigate this problem, we consider a novel problem setting where the classifier for the target domain has to be trained with complementary-label data from the source domain and unlabeled data from the target domain named budget-friendly UDA (BFUDA). The key benefit is that it is much less costly to collect complementary-label source data (required by BFUDA) than collecting the true-label source data (required by ordinary UDA). To this end, complementary label adversarial network (CLARINET) is proposed to solve the BFUDA problem. CLARINET maintains two deep networks simultaneously, where one focuses on classifying complementary-label source data and the other takes care of the source-to-target distributional adaptation. Experiments show that CLARINET significantly outperforms a series of competent baselines.

Data sparsity remains a challenging and common problem in real-world recommender systems, which impairs the accuracy of recommendation thus damages user experience. Cross-domain recommender systems are developed to deal with data sparsity problem through transferring knowledge from a source domain with relatively abundant data to the target domain with insufficient data. However, two challenging issues exist in cross-domain recommender systems: 1) domain shift which makes the knowledge from source domain inconsistent with that in the target domain; 2) knowledge extracted from only one source domain is insufficient, while knowledge is potentially available in many other source domains. To handle the above issues, we develop a cross-domain recommendation method in this paper to extract group-level knowledge from multiple source domains to improve recommendation in a sparse target domain. Domain adaptation techniques are applied to eliminate the domain shift and align user and item groups to maintain knowledge consistency during the transfer learning process. Knowledge is extracted not from one but multiple source domains through an intermediate subspace and adapted through flexible constraints of matrix factorization in the target domain. Experiments conducted on five datasets in three categories show that the proposed method outperforms six benchmarks and increases the accuracy of recommendations in the target domain.

Transfer learning is gaining increasing attention due to its ability to leverage previously acquired knowledge to assist in completing a prediction task in a similar domain. While many existing transfer learning methods deal with single source and single target problem without considering the fact that a target domain maybe similar to multiple source domains, this work proposes a multi-source domain adaptation method based on a deep neural network. Our method contains common feature extraction, specific predictor learning and target predictor estimation. Common feature extraction explores the relationship between source domains and target domain by distribution fusion and guarantees the strength of similar source domains during training, something which has not been well considered in existing works. Specific predictor learning trains source tasks with cross-domain distribution constraint and cross-domain predictor constraint to enhance the performance of single source. Target predictor estimation employs relationship extraction and selective strategy to improve the performance of the target task and to avoid negative transfer. Experiments on real-world visual datasets show the performance of the proposed method is superior to other deep learning baselines.

In data stream mining, concept drift may cause the predictions given by machine learning models become less accurate as time passes. Existing concept drift detection and adaptation methods are built based on a framework that is buffering new samples if a drift-warming level is triggered and retraining a new model if a drift-alarm level is triggered. However, these methods neglected the problem that the performance of a learning model could be more sensitive to the amount of training data rather than the concept drift. In other words, a retrained model built on very few data instances could be even worse than the old model trained before the drift. To elaborate and address this problem, we propose a fast switch Naïve Bayes model (fsNB) for concept drift detection and adaptation. The intuition is to apply the idea of following the leader in online learning. We manipulate a sliding and an incremental Naïve Bayes classifier, if the sliding one overwhelms the incremental one, the model reports a drift. The experimental evaluation shows the advantages of fsNB and demonstrates that retraining may not be the best options for a marginal drift.

Domain adaptation uses the previously acquired knowledge (source domain) to support predicted tasks in the current domain without sufficient labeled data (target domain). Although many methods have been developed in domain adaptation, one issue hasn't been solved: how to implement knowledge transfer when more than one source domain is available. In this paper we present a neural network-based method which extracts domain knowledge in the form of rules to facilitate knowledge transfer, merge rules from all source domains and further select related rules for target domain and clip redundant rules. The method presented is validated on datasets that simulate the multi-source scenario and the experimental results verify the superiority of our method in handling multi-source domain adaptation problems.

Causal inference aims to estimate the treatment effect of an intervention on the target outcome variable and has received great attention across fields ranging from economics and statistics to machine learning . Observational causal inference is challenging because the pre-treatment variables may influence both the treatment and the outcome, resulting in confounding bias. The classic inverse propensity weighting (IPW) estimator is theoretically able to eliminate the confounding bias. However, in observational studies, the propensity scores used in the IPW estimator must be estimated from finite observational data and may be subject to extreme values, leading to the problem of highly variable importance weights, which consequently makes the estimated causal effect unstable or even misleading. In this paper, by reframing the IPW estimator in the importance sampling framework, we propose a Pareto-smoothing method to tackle this problem. The generalized Pareto distribution (GPD) from extreme value theory is used to fit the upper tail of the estimated importance weights and to replace them using the order statistics of the fitted GPD. To validate the performance of the new method, we conducted extensive experiments on simulated and semi-simulated datasets. Compared with two existing methods for importance weight stabilization, i.e., weight truncation and self-normalization, the proposed method generally achieves better performance in settings with a small sample size and high-dimensional covariates . Its application on a real-world heath dataset indicates its utility in estimating causal effects for program evaluation.

The performance of a classifier might greatly deteriorate due to missing data. Many different techniques to handle this problem have been developed. In this paper, we solve the problem of missing data using a novel transfer learning perspective and show that when an additive least squares support vector machine (LS-SVM) is adopted, model transfer learning can be used to enhance the classification performance on incomplete training datasets. A novel transfer-based additive LS-SVM classifier is accordingly proposed. This method also simultaneously determines the influence of classification errors caused by each incomplete sample using a fast leave-one-out cross validation strategy, as an alternative way to clean the training data to further improve the data quality. The proposed method has been applied to seven public datasets. The experimental results indicate that the proposed method achieves at least comparable, if not better, performance than case deletion, mean imputation, and k -nearest neighbor imputation methods, followed by the standard LS-SVM and support vector machine classifiers. Moreover, a case study on a community healthcare dataset using the proposed method is presented in detail, which particularly highlights the contributions and benefits of the proposed method to this real-world application.

Transfer learning is gaining increasing attention due to its ability to leverage previously acquired knowledge to assist in completing a prediction task in a related domain. Fuzzy transfer learning, which is based on fuzzy systems and particularly fuzzy rule-based models, was developed due to its capacity to deal with uncertainty. However, one issue with fuzzy transfer learning, even in the area of general transfer learning, has not been resolved: how to combine and then use knowledge when multiple-source domains are available. This study presents new methods for merging fuzzy rules from multiple domains for regression tasks. Two different settings are separately explored: homogeneous and heterogeneous space. In homogeneous situations, knowledge from the source domains is merged in the form of fuzzy rules. In heterogeneous situations, knowledge is merged in the form of both data and fuzzy rules. Experiments on both synthetic and real-world datasets provide insights into the scope of applications suitable for the proposed methods and validate their effectiveness through comparisons with other state-of-the-art transfer learning methods. An analysis of parameter sensitivity is also included.

Current models and algorithms have been increasingly required to learn in a nonstationary environment because the phenomenon of concept drift (or pattern shift) may occur, that is, the assumption that data are identically distributed may be invalid in data streams. Once the data pattern changes, a well-trained model built on the previous, now obsolete data cannot provide an accurate prediction for future data. To obtain reliable prediction, it is important to understand the existing patterns in the data stream and to know which pattern the current examples belong to during the modeling process. However, it is ambiguous to classify an example to a certain pattern in many real-world cases. In this paper, we propose a novel adaptive regression approach, called FUZZ-CARE, to dynamically recognize, train, and store patterns, and assign the membership degree of the upcoming examples belonging to these patterns. Membership degrees are presented by the membership matrix obtained from a kernel fuzzy c-means clustering, which is synchronously trained and adapted with regression parameters. Rather than designing a complicated procedure to continuously chase the newest pattern, which is a common approach in the literature, FUZZ-CARE abstracts useful past information to help predict newly arrived examples. It thus effectively avoids the risk of insufficient training due to the lack of new data and improves prediction accuracy. Experiments on six synthetic datasets and 21 real-world datasets validate the high accuracy and robustness of our approach.

Estimating causal effects by making causal inferences from observational data is common practice in scientific studies, business decision-making, and daily life. In today’s data-driven world, causal inference has become a key part of the evaluation process for many purposes, such as examining the effects of medicine or the impact of an economic policy on society. However, although the literature contains some excellent models, there is room to improve their representation power and their ability to capture complex relationships. For these reasons, we propose a novel prior called Causal DP and a model called CDP. The prior captures the complex relationships between covariates, treatments, and outcomes in observational data using a rational probabilistic dependency structure. The model is Bayesian, nonparametric, and generative and is not based on the assumption of any parametric distribution. CDP is designed to estimate various kinds of causal effects—average, conditional average, average treated, quantile, and so on. It performs well with missing covariates and does not suffer from overfitting. Comparative experiments on synthetic datasets against several state-of-the-art methods demonstrate that CDP has a superior ability to capture complex relationships. Further, a simple evaluation to infer the effect of a job training program on trainee earnings from real-world data shows that CDP is both effective and useful for causal inference.

Domain adaptation leverages the knowledge in one domain-the source domain-to improve learning efficiency in another domain-the target domain. Existing heterogeneous domain adaptation research is relatively well-progressed but only in situations where the target domain contains at least a few labeled instances. In contrast, heterogeneous domain adaptation with an unlabeled target domain has not been well-studied. To contribute to the research in this emerging field, this article presents: 1) an unsupervised knowledge transfer theorem that guarantees the correctness of transferring knowledge and 2) a principal angle-based metric to measure the distance between two pairs of domains: one pair comprises the original source and target domains and the other pair comprises two homogeneous representations of two domains. The theorem and the metric have been implemented in an innovative transfer model, called a Grassmann-linear monotonic maps-geodesic flow kernel (GLG), which is specifically designed for heterogeneous unsupervised domain adaptation (HeUDA). The linear monotonic maps (LMMs) meet the conditions of the theorem and are used to construct homogeneous representations of the heterogeneous domains. The metric shows the extent to which the homogeneous representations have preserved the information in the original source and target domains. By minimizing the proposed metric, the GLG model learns the homogeneous representations of heterogeneous domains and transfers knowledge through these learned representations via a geodesic flow kernel (GFK). To evaluate the model, five public data sets were reorganized into ten HeUDA tasks across three applications: cancer detection, the credit assessment, and text classification. The experiments demonstrate that the proposed model delivers superior performance over the existing baselines.

In order to extract useful information from data streams, incremental learning has been introduced in more and more data mining algorithms. For instance, a self-organizing incremental neural network (SOINN) has been proposed to extract a topological structure that consists of one or more neural networks to closely reflect the data distribution of data streams. However, SOINN has the tradeoff between deleting previously learned nodes and inserting new nodes, i.e., the stability-plasticity dilemma. Therefore, it is not guaranteed that the topological structure obtained by the SOINN will closely represent data distribution. For solving the stability-plasticity dilemma, we propose a Gaussian membership-based SOINN (Gm-SOINN). Unlike other SOINN-based methods that allow only one node to be identified as a "winner" (the nearest node), the Gm-SOINN uses a Gaussian membership to indicate to which degree the node is a winner. Hence, the Gm-SOINN avoids the topological structure that cannot represent the data distribution because previously learned nodes overly deleted or noisy nodes inserted. In addition, an evolving Gaussian mixture model is integrated into the Gm-SOINN to estimate the density distribution of nodes, thereby avoiding the wrong connection between two nodes. Experiments involving both artificial and real-world data sets indicate that our proposed Gm-SOINN achieves better performance than other topology learning methods.

How to leverage knowledge from labelled domain (source) to help classify unlabeled domain (target) is a key problem in the machine learning field. Unsupervised domain adaptation (UDA) provides a solution to this problem and has been well developed for two homogeneous domains. However, when the target domain is unlabeled and heterogeneous with the source domain, current UDA models cannot accurately transfer knowledge from a source domain to a target domain. Benefiting from development of neural networks, this paper presents a new neural network, shared fuzzy equivalence relations neural network (SFER-NN), to address the heterogeneous UDA (HeUDA) problem. SFER-NN transfers knowledge across two domains according to shared fuzzy equivalence relations that can simultaneously cluster features of two domains into several categories. Based on the clustered categories, SFER-NN is constructed to minimize the discrepancy between two domains. Compared to previous works, SFER-NN is more capable of minimizing discrepancy between two domains. As a result of this advantage, SFER-NN delivers a better performance than previous studies using two public datasets.

Concept drift referring to the changes of data distributions has been one critical challenge typically associated with mining data streams. Current drift detection and adaptation methods focus on how to immediately detect the distribution changes once the concept drift occurs and swiftly update the model to be applicable to the newly arrived data instances. Most of those methods assume the data does not have noise or the noise is too weak to affect the modeling procedure. However, realworld data are normally contaminated, and denoise techniques are highly preferred as a necessary preprocess. This issue is more complex for a data stream with concept drift because the noise is very likely to be confused with drift. Motivated by that, this paper proposes a Noise-tolerant Fuzzy c-means based drift Adaptation method (NFA) which can adapt to the changing distributions and is suitable for noisy data streams. The concept drift problem is solved by using a fuzzy c-means based regression model to continuously include the most relevant data instances to the latest pattern in the training set. In addition, a denoise technique is designed in NFA to remove noise, and the ability of incremental updating enables it to be embedded in the incremental drift adaptation process, and therefore NFA can solve concept drift and noise problems at the same time. Experimental evaluation results also show good performance of our method on handling data streams with concept drift and noise.

Transfer learning has emerged as a solution for the cases where little or no labeled data are available in the training process. It leverages the previously acquired knowledge (a source domain with a large amount of labeled data) to facilitate solving the current tasks (a target domain with little labeled data). Many transfer learning methods have been proposed, and especially fuzzy transfer learning method, which is based on fuzzy systems, has been developed because of its capability to deal with the uncertainty in transfer learning. However, there is one issue with fuzzy transfer learning that has not yet been resolved: the domain selection problem, which is heavily depended on the knowledge transfer method and the applied prediction model. In this work, we explore the domain selection problem in TakagiSugeno fuzzy model when multiple source domains are accessible, and define the similarity between the source and target domains to provide guidance for the domain selection. The experiments on synthetic datasets are designed to simulate the situations of multiple sources in transfer learning, and demonstrate the rationality of the proposed similarity in selecting the source domain for the target domain. Further, the real-world datasets are used to validate the proposed domain adaptation method, and verify its capability in solving practical situations.

Causal inference from observational data aims to estimate causal effects when controlled experimentation is not feasible, but it faces challenges when unobserved confounders exist. The instrumental variable method resolves this problem by introducing a variable that is correlated with the treatment and affects the outcome only through the treatment. However, existing instrumental variable methods require two stages to separately estimate the conditional treatment distribution and the outcome generating function, which is not sufficiently effective. This paper presents a one-stage approach to jointly estimate the treatment distribution and the outcome generating function through a cleverly designed deep neural network structure. This study is the first to merge the two stages to leverage the outcome to the treatment distribution estimation. Further, the new deep neural network architecture is designed with two strategies (i.e., shared and separate) of learning a confounder representation account for different observational data. Such network architecture can unveil complex relationships between confounders, treatments, and outcomes. Experimental results show that our proposed method outperforms the state-of-the-art methods. It has a wide range of applications, from medical treatment design to policy making, population regulation and beyond.

A knowledge graph (KG) represents a collection of interlinked descriptions of entities. It has become a key focus for organising and utilising this type of data for applications. Many graph embedding techniques have been proposed to simplify the manipulation while preserving the inherent structure of the KG. However, scant attention has been given to the investigation of the importance of the entities (the nodes of KGs). In this paper, we propose a novel entities importance learning framework that investigates how to weight the entities and use them as a prior knowledge for solving multi-stream regression problems. The framework consists of KG feature extraction, multi-stream correlation analysis, and entity importance learning. To evaluate the proposed method, we implemented the framework based on Wikidata and applied it to Australian retail fuel price forecasting. The experiment results indicate that the proposed method reduces prediction error, which supports the weighted knowledge graph information as a means for improving machine learning model accuracy.

Unsupervised domain adaptation aims to leverage the knowledge in training data (source domain) to improve the performance of tasks in the remaining unlabeled data (target domain) by mitigating the effect of the distribution discrepancy. Existing approaches resolve this problem mainly by 1) mapping data into a latent space where the distribution discrepancy between two domains is reduced; or 2) reducing the domain shift by weighting the source domain. However, most of these approaches share a common issue that they neglect inter-class margins while matching distributions, which has a significant impact on classification performance. In this paper, we analyze the issue from the theoretical aspect and propose a novel unsupervised domain adaptation approach: Sphere Retracting Transformation (SRT), which reduces the distribution discrepancy and increases inter-class margins. We implement SRT, according to our theoretical analysis by (1) assigning class-specific weights for data in the source domain, and (2) minimizing the intra-class variations. Experiments confirm that the SRT approach outperforms several competitive approaches for standard domain adaptation benchmarks.

Many recommender systems rely on the information of user-item interactions to generate recommendations. In real applications, the interaction matrix is usually very sparse, as a result, the model cannot be optimised stably with different initial parameters and the recommendation performance is unsatisfactory. Many works attempted to solve this problem, however, the parameters in their models may not be trained effectively due to the sparse nature of the dataset which results in a lower quality local optimum. In this paper, we propose a generative network for making user recommendations and a discriminative network to guide the training process. An adversarial training strategy is also applied to train the model. Under the guidance of a discriminative network, the generative network converges to an optimal solution and achieves better recommendation performance on a sparse dataset. We also show that the proposed method significantly improves the precision of the recommendation performance on several datasets.

The design of classical regression algorithms was based on the assumption that all the required data is obtained at one time. With the emergence of big data, however, data is increasingly displayed in sequence form, such as in data streams, and can be read only once in a specific order. Many incremental regression algorithms which process data in a sequential manner have been proposed, but the accuracy of these algorithms deteriorates when the value of the data is uncertain. This paper proposes a hybrid incremental regression neural network based on self-organizing incremental neural network and incremental fuzzy support vector regression. In our proposed network, the neurons of the regression neural network are obtained by an improved self-organized incremental neural network (SOINN). This enables the regression neural network structure to self-organize as the number of neurons increases. An incremental fuzzy support vector regression (IFSVR) algorithm is then used to modify the parameters of the regression neural network. By combining the improved SOINN and IFSVR algorithms, our proposed hybrid incremental regression neural network is able to learn an accurate regression model from large uncertain data. Experiments on both artificial and real-world datasets indicate that our proposed hybrid incremental regression neural network achieves superior performance compared to other incremental regression algorithms.

The tagging system provides users with a platform to express their preferences as they annotate terms or keywords to items. Tag information is a bridge between two domains for transferring knowledge and helping to alleviate the data sparsity problem, which is a crucial and challenging problem in most recommender systems. Existing methods incorporate correlations extracted from overlapping tags at a lexical level in cross-domain recommendation, but they neglect semantical relationships between different tags, which impairs prediction accuracy in the target domain. To solve this challenging problem, we propose a cross-domain recommendation method with semantic correlation in tagging systems. This method automatically captures the semantic relationships between non-identical tags and applies them to the recommendation. The word2vec technique is used to learn the latent representations of tags. Semantically equivalent tags are then grouped to form a joint embedding space comprised of tag clusters. This embedding space serves as the bridge between domains. By mapping users and items from both the source and target domains into the same embedding space, similar users or items across domains can be identified. Thus, the recommendation in a sparse target domain is improved by transferring knowledge through correlated users and items. Experimental results with three datasets on six cross-domain recommendation tasks demonstrate that the proposed method exploits the semantic links from tags in two domains and outperforms five benchmarks in prediction accuracy. The results indicate that transferring knowledge through tags semantics is feasible and effective.

Weather forecasting is usually solved through numerical weather prediction (NWP), which can sometimes lead to unsatisfactory performance due to inappropriate setting of the initial states. In this paper, we design a data-driven method augmented by an effective information fusion mechanism to learn from historical data that incorporates prior knowledge from NWP. We cast the weather forecasting problem as an end-to-end deep learning problem and solve it by proposing a novel negative log-likelihood error (NLE) loss function. A notable advantage of our proposed method is that it simultaneously implements single-value forecasting and uncertainty quantification, which we refer to as deep uncertainty quantification (DUQ). Efficient deep ensemble strategies are also explored to further improve performance. This new approach was evaluated on a public dataset collected from weather stations in Beijing, China. Experimental results demonstrate that the proposed NLE loss significantly improves generalization compared to mean squared error (MSE) loss and mean absolute error (MAE) loss. Compared with NWP, this approach significantly improves accuracy by 47.76%, which is a state-of-the-art result on this benchmark dataset.

With the increasing of online shopping services, fashion recommendation plays an important role in daily online shopping scenes. A lot of recommender systems have been developed with visual information. However, few works take into account compatibility relationship when they are generating recommendations. The challenge is that fashion concept is often subtle and subjective for different customers. In this paper, we propose a fashion compatibility knowledge learning method that incorporates visual compatibility relationships as well as style information. We also propose a fashion recommendation method with domain adaptation strategy to alleviate the distribution gap between the items in target domain and the items of external compatible outfits. Our results indicate that the proposed method is capable of learning visual compatibility knowledge and outperforms all the baselines.

Traditional recommender systems suffer from the data sparsity problem. However, user knowledge acquired in one domain can be transferred and exploited in several other relevant domains. In this context, cross-domain recommender systems have been proposed to create a new and effective recommendation paradigm in which to exploit rich data from auxiliary domains to assist recommendations in a target domain. Before knowledge transfer takes place, building reliable and concrete domain correlation is the key ensuring that only relevant knowledge will be transferred. Social tags are used to explicitly link different domains, especially when neither users nor items overlap. However, existing models only exploit a subset of tags that are shared by heterogeneous domains. In this paper, we propose a complete tag-induced cross-domain recommendation (CTagCDR) model, which infers interdomain and intradomain correlations from tagging history and applies the learned structural constraints to regularize joint matrix factorization. Compared to similar models, CTagCDR is able to fully explore knowledge encoded in both shared and domain-specific tags. We demonstrate the performance of our proposed model on three public datasets and compare it with five state-of-the-art single and cross-domain recommendation approaches. The results show that CTagCDR works well in both rating prediction and item recommendation tasks, and can effectively improve recommendation performance.

Transfer learning is gaining considerable attention due to its ability to leverage previously acquired knowledge to assist in completing a prediction task in a related domain. Fuzzy transfer learning, which is based on fuzzy system (especially fuzzy rule-based models), has been developed because of its capability to deal with the uncertainty in transfer learning. However, two issues with fuzzy transfer learning have not yet been resolved: choosing an appropriate source domain and efficiently selecting labeled data for the target domain. This paper proposes an innovative method based on fuzzy rules that combines an infinite Gaussian mixture model (IGMM) with active learning to enhance the performance and generalizability of the constructed model. An IGMM is used to identify the data structures in the source and target domains providing a promising solution to the domain selection dilemma. Further, we exploit the interactive query strategy in active learning to correct imbalances in the knowledge to improve the generalizability of fuzzy learning models. Through experiments on synthetic datasets, we demonstrate the rationality of employing an IGMM and the effectiveness of applying an active learning technique. Additional experiments on real-world datasets further support the capabilities of the proposed method in practical situations.

Domain adaptation aims to leverage knowledge acquired from a related domain (called a source domain) to improve the efficiency of completing a prediction task (classification or regression) in the current domain (called the target domain), which has a different probability distribution from the source domain. Although domain adaptation has been widely studied, most existing research has focused on homogeneous domain adaptation, where both domains have identical feature spaces. Recently, a new challenge proposed in this area is heterogeneous domain adaptation where both the probability distributions and the feature spaces are different. Moreover, in both homogeneous and heterogeneous domain adaptation, the greatest efforts and major achievements have been made with classification tasks, while successful solutions for tackling regression problems are limited. This paper proposes two innovative fuzzy rule-based methods to deal with regression problems. The first method, called fuzzy homogeneous domain adaptation, handles homogeneous spaces while the second method, called fuzzy heterogeneous domain adaptation, handles heterogeneous spaces. Fuzzy rules are first generated from the source domain through a learning process; these rules, also known as knowledge, are then transferred to the target domain by establishing a latent feature space to minimize the gap between the feature spaces of the two domains. Through experiments on synthetic datasets, we demonstrate the effectiveness of both methods and discuss the impact of some of the significant parameters that affect performance. Experiments on real-world datasets also show that the proposed methods improve the performance of the target model over an existing source model or a model built using a small amount of target data.

Concept drift describes unforeseeable changes in the underlying distribution of streaming data overtime. Concept drift research involves the development of methodologies and techniques for drift detection, understanding, and adaptation. Data analysis has revealed that machine learning in a concept drift environment will result in poor learning results if the drift is not addressed. To help researchers identify which research topics are significant and how to apply related techniques in data analysis tasks, it is necessary that a high quality, instructive review of current research developments and trends in the concept drift field is conducted. In addition, due to the rapid development of concept drift in recent years, the methodologies of learning under concept drift have become noticeably systematic, unveiling a framework which has not been mentioned in literature. This paper reviews over 130 high quality publications in concept drift related research areas, analyzes up-to-date developments in methodologies and techniques, and establishes a framework of learning under concept drift including three main components: concept drift detection, concept drift understanding, and concept drift adaptation. This paper lists and discusses 10 popular synthetic datasets and 14 publicly available benchmark datasets used for evaluating the performance of learning algorithms aiming at handling concept drift. Also, concept drift related research directions are covered and discussed. By providing state-of-the-art knowledge, this survey will directly support researchers in their understanding of research developments in the field of learning under concept drift.

The aim of recommender systems is to automatically identify user preferences within collected data, then use those preferences to make recommendations that help with decisions. However, recommender systems suffer from data sparsity problem, which is particularly prevalent in newly launched systems that have not yet had enough time to amass sufficient data. As a solution, cross-domain recommender systems transfer knowledge from a source domain with relatively rich data to assist recommendations in the target domain. These systems usually assume that the entities either fully overlap or do not overlap at all. In practice, it is more common for the entities in the two domains to partially overlap. Moreover, overlapping entities may have different expressions in each domain. Neglecting these two issues reduces prediction accuracy of cross-domain recommender systems in the target domain. To fully exploit partially overlapping entities and improve the accuracy of predictions, this paper presents a cross-domain recommender system based on kernel-induced knowledge transfer, called KerKT. Domain adaptation is used to adjust the feature spaces of overlapping entities, while diffusion kernel completion is used to correlate the non-overlapping entities between the two domains. With this approach, knowledge is effectively transferred through the overlapping entities, thus alleviating data sparsity issues. Experiments conducted on four data sets, each with three sparsity ratios, show that KerKT has 1.13%-20% better prediction accuracy compared with six benchmarks. In addition, the results indicate that transferring knowledge from the source domain to the target domain is both possible and beneficial with even small overlaps.

The additive kernel least squares support vector machine (AK-LS-SVM) has been well used in classification tasks due to its inherent advantages. For example, additive kernels work extremely well for some specific tasks, such as computer vision classification, medical research, and some specialized scenarios. Moreover, the analytical solution using AK-LS-SVM can formulate leave-one-out cross-validation error estimates in a closed form for parameter tuning, which drastically reduces the computational cost and guarantee the generalization performance especially on small and medium datasets. However, AK-LS-SVM still faces two main challenges: 1) improving the classification performance of AK-LS-SVM and 2) saving time when performing a grid search for model selection. Inspired by the stacked generalization principle and the transfer learning mechanism, a layer-by-layer combination of AK-LS-SVM classifiers embedded with transfer learning is proposed in this paper. This new classifier is called deep transfer additive kernel least square support vector machine (DTA-LS-SVM) which overcomes these two challenges. Also, considering that imbalanced datasets are involved in many real-world scenarios, especially for medical data analysis, the deep-transfer element is extended to compensate for this imbalance, thus leading to the development of another new classifier iDTA-LS-SVM. In the hierarchical structure of both DTA-LS-SVM and iDTA-LS-SVM, each layer has an AK-LS-SVM and the predictions from the previous layer act as an additional input feature for the current layer. Importantly, transfer learning is also embedded to guarantee generalization consistency between the adjacent layers. Moreover, both iDTA-LS-SVM and DTA-LS-SVM can ensure the minimal leaveone-out error by using the proposed fast leave-one-out cross validation strategy on the training set in each layer. We compared the proposed classifiers DTA-LS-SVM and iDTA-LS-SVM with the traditional LS-SVM and SVM using additive kernels on seven public UCI datasets and one real world dataset. The experimental results show that both DTA-LS-SVM and iDTALS-SVM exhibit better generalization performance and faster learning speed.

Most recommender systems (RSs), especially group RSs, focus on methods and accuracy but lack explanations, hence users find them difficult to trust. We present a hierarchy visualization method for group recommender (HVGR) systems to provide visual presentation and intuitive explanation. We first use a hierarchy graph to organize all the entities using nodes (e.g., neighbor nodes and recommendation nodes) and illustrate the overall recommender process using edges. Second, a pie chart is attached to every entity node in which each slice represents a single member, which makes it easy to track the influence of each member on a specific entity. HVGR can be extended to adapt different pseudouser modeling methods by resizing group member nodes and pseudouser nodes. It can also be easily extended to individual RSs through the use of a single member group. An implementation has been developed and feasibility is tested using a real data set.

Representing the semantics of words is a fundamental task in text processing. Several research studies have shown that text and knowledge bases (KBs) are complementary sources for word embedding learning. Most existing methods only consider relationships within word-pairs in the usage of KBs. We argue that the structural information of well-organized words within the KBs is able to convey more effective and stable knowledge in capturing semantics of words. In this paper, we propose a semantic structure-based word embedding method, and introduce concept convergence and word divergence to reveal semantic structures in the word embedding learning process. To assess the effectiveness of our method, we use WordNet for training and conduct extensive experiments on word similarity, word analogy, text classification and query expansion. The experimental results show that our method outperforms state-of-the-art methods, including the methods trained solely on the corpus, and others trained on the corpus and the KBs.

Domain adaptation can transfer knowledge from the source domain to improve pattern recognition accuracy in the target domain. However, it is rarely discussed when the target domain is unlabeled and heterogeneous with the source domain, which is a very challenging problem in the domain adaptation field. This paper presents a new feature reconstruction method: unconstrained fuzzy feature fusion. Through the reconstructed features of a source and a target domain, a geodesic flow kernel is applied to transfer knowledge between them. Furthermore, the original information of the target domain is also preserved when reconstructing the features of the two domains. Compared to the previous work, this work has two advantages: 1) the sum of the memberships of the original features to fuzzy features no longer must be one, and 2) the original information of the target domain is persevered. As a result of these advantages, this work delivers a better performance than previous studies using two public datasets.

Prediction in non-stationary environments, where data streams are ever-changing at very high speeds, has become more and more important in real-world applications. The uncertainty in data streams caused by changes in data distribution is described as concept drift. The appearance of concept drift in a data stream results in inconsistencies between the existing data and incoming data. Such inconsistencies pose a great challenge to conventional machine learning methods, given they are built on the assumption of independent and identically distributed data and cannot adapt to unpredictable changes in knowledge patterns. To solve such data stream uncertainty problem, this paper presents a window-based self-adaptive fuzzy network called adaptive fuzzy network (AFN), which can continuously modify the network through identifying new knowledge from the previous data samples. Three components are embedded in ANF: a drift detection module to identify whether the current window of data samples presents different pattern from the previous; a drift adaption module to retain useful knowledge in previous samples; and a fuzzy inference system, which integrates the detection and adaption modules for prediction. ANF has been evaluated through a set of experiments on non-stationary data streams. The experimental results show a good effectiveness of our method.

Uncertainty in dependent variables or independent variables is typically caused by randomness or fuzziness. But randomness and fuzziness are more and more often appearing simultaneously in independent variables or dependent variables, giving rise to the concept of a fuzzy random variable. Regression analysis is a statistical measure to model the relationship between a dependent variable and one or more independent variables. However, the standard regression algorithms cannot handle the fuzzy random variables, so we propose a dual neural network algorithm based on confidence intervals for fuzzy random regression problems in this paper. The algorithm relies on the expectations of, and variances in, fuzzy random variables to construct the confidence intervals for fuzzy random input-output data. A dual neural network then identifies the sides of the interval output data; one network identifies the upper side, another network identifies the lower side, while a dual v-support vector regression algorithm concurrently constructs the initial structure of the dual neural network. Lastly, a dynamic genetic backpropagation algorithm tunes the parameters of the dual neural network to improve performance. Experiment results demonstrate the validity and applicability of the proposed dual neural network algorithm based on confidence intervals.

The residual network has witnessed a great success in computer vision particularly on classification tasks, however, it has not been well studied in regression. In this work, we show its competence in a regression task - crowds flow prediction, which has strong implication to city safety and management. The problem of crowds flow prediction is challenging due to its fast dynamics. To address this issue, we explore residual learning with Gaussian regularization and propose a novel convolutional neural network called Gaussian noise residual networks (Noise-ResNet). Compared with the benchmark ST-ResNet on crowds flow prediction, the proposed architecture has three advantages: 1) Superior performance. Especially, it attains the state-of-the-art results on benchmark dataset BikeNYC. 2) Light architecture. Noise-ResNet only utilises one residual unit rather than STResNet with multiple ones, which greatly reduces the training time. 3) Interpretable input sequences. Noise-ResNet takes an input sequence that only considers the most important periodic data and closeness data, which makes the learning process more interpretable. Furthermore, experimental results substantiate that the Noise-ResNet can outperform ResNet with dropout on the same regression task.

The existing domain-specific methods for mining information networks in machine learning aims to represent the nodes of an information network into a vector format. However, the real-world large-scale information network cannot make well network representations by one network. When the information of the network structure transferred from one network to another network, the performance of network representation might decrease sharply. To achieve these ends, we propose a novel framework to transfer useful information across relational large-scale information networks (FTLSIN). The framework consists of a 2-layer random walks to measure the relations between two networks and predict links across them. Experiments on real-world datasets demonstrate the effectiveness of the proposed model.

Unsupervised domain adaptation (UDA) aims to recognize newly emerged patterns in target domains, which may be unlabeled, by leveraging knowledge from patterns learnt from source domains. However, existing UDA models and algorithms still suffer from heterogeneous domains, known as the heterogeneous unsupervised domain adaptation (HeUDA) issue. To address this issue, this paper presents a novel HeUDA model via n-dimensional fuzzy geometry and fuzzy equivalence relations, called F-HeUDA. The n-dimensional fuzzy geometry is used to propose a metric to measure the similarity between features on one domain. Then, based on this metric, shared fuzzy equivalence relations (SFER) are proposed. The SFER can allow two domains to use the same α to get the same number of clustering categories. Through these clustering categories, knowledge from the heterogeneous source domain can be transferred to the unlabeled target domain. Different to existing HeUDA models, the proposed F-HeUDA model does not need that two domains must have the same number of instances. As a result, the proposed model has a better ability to handle the issue of small datasets. Experiments distributed across four real datasets were conducted to validate the proposed model. This testing regime demonstrates that the proposed model outperforms the state-of-the-art models, especially when the target domain has very few instances.

Multiple kernel learning (MKL) is a principled approach to kernel combination and selection for a variety of learning tasks, such as classification, clustering, and dimensionality reduction. In this paper, we develop a novel fuzzy multiple kernel learning model based on the Hilbert-Schmidt independence criterion (HSIC) for classification, which we call HSIC-FMKL. In this model, we first propose an HSIC Lasso-based MKL formulation, which not only has a clear statistical interpretation that minimum redundant kernels with maximum dependence on output labels are found and combined, but also enables the global optimal solution to be computed efficiently by solving a Lasso optimization problem. Since the traditional support vector machine (SVM) is sensitive to outliers or noises in the dataset, fuzzy SVM (FSVM) is used to select the prediction hypothesis once the optimal kernel has been obtained. The main advantage of FSVM is that we can associate a fuzzy membership with each data point such that these data points can have different effects on the training of the learning machine. We propose a new fuzzy membership function using a heuristic strategy based on the HSIC. The proposed HSIC-FMKL is a two-stage kernel learning approach and the HSIC is applied in both stages. We perform extensive experiments on real-world datasets from the UCI benchmark repository and the application domain of computational biology which validate the superiority of the proposed model in terms of prediction accuracy.

In classical data-driven machine learning methods, massive amounts of labeled data are required to build a high-performance prediction model. However, the amount of labeled data in many real-world applications is insufficient, so establishing a prediction model is impossible. Transfer learning has recently emerged as a solution to this problem. It exploits the knowledge accumulated in auxiliary domains to help construct prediction models in a target domain with inadequate training data. Most existing transfer learning methods solve classification tasks; only a few are devoted to regression problems. In addition, the current methods ignore the inherent phenomenon of information granularity in transfer learning. In this study, granular computing techniques are applied to transfer learning. Three granular fuzzy regression domain adaptation methods to determine the estimated values for a regression target are proposed to address three challenging cases in domain adaptation. The proposed granular fuzzy regression domain adaptation methods change the input and/or output space of the source domain's model using space transformation, so that the fuzzy rules are more compatible with the target data. Experiments on synthetic and real-world datasets validate the effectiveness of the proposed methods.

Sparse nonnegative matrix factorization (SNMF) aims to factorize a data matrix into two optimized nonnegative sparse factor matrices, which could benefit many tasks, such as document-word co-clustering. However, the traditional SNMF typically assumes the number of latent factors (i.e., dimensionality of the factor matrices) to be fixed. This assumption makes it inflexible in practice. In this paper, we propose a doubly sparse nonparametric NMF framework to mitigate this issue by using dependent Indian buffet processes (dIBP). We apply a correlation function for the generation of two stick weights associated with each column pair of factor matrices while still maintaining their respective marginal distribution specified by IBP. As a consequence, the generation of two factor matrices will be columnwise correlated. Under this framework, two classes of correlation function are proposed: 1) using bivariate Beta distribution and 2) using Copula function. Compared with the single IBP-based NMF, this paper jointly makes two factor matrices nonparametric and sparse, which could be applied to broader scenarios, such as co-clustering. This paper is seen to be much more flexible than Gaussian process-based and hierarchial Beta process-based dIBPs in terms of allowing the two corresponding binary matrix columns to have greater variations in their nonzero entries. Our experiments on synthetic data show the merits of this paper compared with the state-of-the-art models in respect of factorization efficiency, sparsity, and flexibility. Experiments on real-world data sets demonstrate the efficiency of this paper in document-word co-clustering tasks.

With the accessibility to information, users often face the problem of selecting one item (a product or a service) from a huge search space. This problem is known as information overload. Recommender systems (RSs) personalize content to a user's interests to help them select the right item in information overload scenarios. Group RSs (GRSs) recommend items to a group of users. In GRSs, a recommendation is usually computed by a simple aggregation method for individual information. However, the aggregations are rigid and overlook certain group features, such as the relationships between the group members' preferences. In this paper, it is proposed a GRS based on opinion dynamics that considers these relationships using a smart weights matrix to drive the process. In some groups, opinions do not agree, hence the weights matrix is modified to reach a consensus value. The impact of ensuring agreed recommendations is evaluated through a set of experiments. Additionally, a sensitivity analysis studies its behavior. Compared to existing group recommendation models and frameworks, the proposal based on opinion dynamics would have the following advantages: 1) flexible aggregation method; 2) member relationships; and 3) agreed recommendations.

Query expansion has been widely used to select additional words that are related to the original query words in the field of information retrieval. In this paper, we present a novel query expansion method that jointly uses fuzzy rules and a word embedding similarity calculation. The expansion words are generated using a word embedding method and selected according to their semantic similarity to the original query. Fuzzy rules are used to enhance the word similarity calculations and reweight expansion words. When measuring and ranking the relevance of a retrieved document, the original query and the expansion words with their weights are considered. We conduct experiments on the query expansion in document ranking tasks. Experimental results from the document ranking task show that the proposed method is able to significantly outperform state-of-the-art baseline methods.

Domain adaptation is a transfer learning approach that has been widely studied in the last decade. However, existing works still have two limitations: 1) the feature spaces of the domains are homogeneous, and 2) the target domain has at least a few labeled instances. Both limitations significantly restrict the domain adaptation approach when knowledge is transferred across domains, especially in the current era of big data. To address both issues, this paper proposes a novel fuzzy-based heterogeneous unsupervised domain adaptation approach. This approach maps the feature spaces of the source and target domains onto the same latent space constructed by fuzzy features. In the new feature space, the label spaces of two domains are maintained to reduce the probability of negative transfer occurring. The proposed approach delivers superior performance over current benchmarks, and the heterogeneous unsupervised domain adaptation (HeUDA) method provides a promising means of giving a learning system the associative ability to judge unknown things using related knowledge.

The aim of machine learning is to find hidden insights into historical data, and then apply them to forecast the future data or trends. Machine learning algorithms optimize learning models for lowest error rate based on the assumption that the historical data and the data to be predicted conform to the same knowledge pattern (data distribution). However, if the historical data is not enough, or the knowledge pattern keeps changing (data uncertainty), this assumption will become invalid. In data stream mining, this phenomenon of knowledge pattern changing is called concept drift. To address this issue, we propose a novel fuzzy windowing concept drift adaptation (FW-DA) method. Compared to conventional windowing-based drift adaptation algorithms, FW-DA achieves higher accuracy by allowing the sliding windows to keep an overlapping period so that the data instances belonging to different concepts can be determined more precisely. In addition, FW-DA statistically guarantees that the upcoming data conforms to the inferred knowledge pattern with a certain confidence level. To evaluate FW-DA, four experiments were conducted using both synthetic and real-world data sets. The experiment results show that FW-DA outperforms the other windowing-based methods including state-of-the-art drift adaptation methods.

Concept drift, given the huge volume of high-speed data streams, requires traditional machine learning models to be self-adaptive. Techniques to handle drift are especially needed in regression cases for a wide range of applications in the real world. There is, however, a shortage of research on drift adaptation for regression cases in the literature. One of the main obstacles to further research is the resulting model complexity when regression methods and drift handling techniques are combined. This paper proposes a self-adaptive algorithm, based on a fuzzy kernel c-means clustering approach and a lazy learning algorithm, called FKLL, to handle drift in regression learning. Using FKLL, drift adaptation first updates the learning set using lazy learning, then fuzzy kernel c-means clustering is used to determine the most relevant learning set. Experiments show that the FKLL algorithm is better able to respond to drift as soon as the learning sets are updated, and is also suitable for dealing with reoccurring drift, when compared to the original lazy learning algorithm and other state-of-the-art regression methods.

Protein complexes play important roles in proteinprotein interaction networks. Recent studies reveal that many proteins have multiple functions and belong to more than one different complexes. To get better complex division, we need to consider time-dependent information of networks. However, only few studies can be found to concentrate on detecting overlapping clusters in time-dependent networks. To solve this problem, we propose integrated model of time-dependent network (IM-TDN) to describe time-dependent networks. On the base of this model, we propose similarity based dynamic fuzzy clustering (SDFC) algorithm to detect overlapping clusters. We apply the algorithm to synthetic data and real world protein-protein interaction network dataset. The results showed that our algorithm by using the model which we proposed achieved better results over the state-of-the-art baseline algorithms.

As the age of big data approaches, methods of massive scale data management are rapidly evolving. The traditional machine learning methods can no longer satisfy the exponential development of big data; there is a common assumption in these data-driving methods that the distribution of both the training data and testing data should be equivalent. A model built using today's data will not adequately address the classification tasks tomorrow if the distribution of the data item values has changed. Transfer learning is emerging as a solution to this issue, and many methods have been proposed. Few of the existing methods, however, explicitly indicate the solution to the case where the labels' distributions in two domains are different. This work proposes the fuzzy rule-based methods to deal with transfer learning problems where the discrepancy between the two domains shows in the label spaces. The presented methods are validated in both the synthetic and real-world datasets, and the experimental results verify the effectiveness of the introduced methods.

In data stream mining, the emergence of new patterns or a pattern ceasing to exist is called concept drift. Concept drift makes the learning process complicated because of the inconsistency between existing data and upcoming data. Since concept drift was first proposed, numerous articles have been published to address this issue in terms of distribution analysis. However, most distribution-based drift detection methods assume that a drift happens at an exact time point, and the data arrived before that time point is considered not important. Thus, if a drift only occurs in a small region of the entire feature space, the other non-drifted regions may also be suspended, thereby reducing the learning efficiency of models. To retrieve non-drifted information from suspended historical data, we propose a local drift degree (LDD) measurement that can continuously monitor regional density changes. Instead of suspending all historical data after a drift, we synchronize the regional density discrepancies according to LDD. Experimental evaluations on three public data sets show that our concept drift adaptation algorithm improves accuracy compared to other methods.

Many medical applications face a situation that the on-hand data cannot fully fit an existing predictive model or on-line tool, since these models or tools only use the most common predictors and the other valuable features collected in the current scenario are not considered altogether. On the other hand, the training data in the current scenario is not sufficient to learn a predictive model effectively yet. In order to overcome these problems and construct an efficient classifier, for these real situations in medical fields, in this work we present an approach based on the least squares support vector machine (LS-SVM), which utilizes a transfer learning framework to make maximum use of the data and guarantee its enhanced generalization capability. The proposed approach is capable of effectively learning a target domain with limited samples by relying on the probabilistic outputs from the other previously learned model using a heterogeneous method in the source domain. Moreover, it autonomously and quickly decides how much output knowledge to transfer from source domain to the target one using a fast leave-one-out cross validation strategy. This approach is applied on a real-world clinical dataset to predict 5-year mortality of bladder cancer patients after radical cystectomy, and the experimental results indicate that the proposed method can achieve better performances compared to traditional machine learning methods, consistently showing the potential of the proposed method under the circumstances with insufficient data.

Motivated by the real damped simple harmonic oscillator (SHO) system, in this paper, we propose a process interpretation of causality and the first-order causal process (FoCP) model for temporal causal modelling. Compared with existing causal models that are able to model feedbacks, such as the structural equation model (SEM) and the structure vector autoregressive (SVAR) model, the FoCP model entails a novel 2-stage evolution semantic for instantaneous and cross-temporal causal relations existing in many real world dynamic systems. Graphical representations are developed to illustrate the causal structure compactly. Useful properties of the new model are identified and used to develop a conditional independence based algorithm for learning the causal structure from a multivariate time series dataset. Experiments on both simulated and real data validate the feasibility of the method to discover simple while meaningful causal structures of dynamic systems.

Whereas traditional science maps emphasize citation statistics and static relationships, this paper presents a term‐based method to identify and visualize the evolutionary pathways of scientific topics in a series of time slices. First, we create a data preprocessing model for accurate term cleaning, consolidating, and clustering. Then we construct a simulated data streaming function and introduce a learning process to train a relationship identification function to adapt to changing environments in real time, where relationships of topic evolution, fusion, death, and novelty are identified. The main result of the method is a map of scientific evolutionary pathways. The visual routines provide a way to indicate the interactions among scientific subjects and a version in a series of time slices helps further illustrate such evolutionary pathways in detail. The detailed outline offers sufficient statistical information to delve into scientific topics and routines and then helps address meaningful insights with the assistance of expert knowledge. This empirical study focuses on scientific proposals granted by the United States National Science Foundation, and demonstrates the feasibility and reliability. Our method could be widely applied to a range of science, technology, and innovation policy research, and offer insight into the evolutionary pathways of scientific activities.

Recommender systems aim to identify relevant items for particular users in large-scale online applications. The historical rating data of users is a valuable input resource for many recommendation models such as collaborative filtering (CF), but these models are known to suffer from the rating sparsity problem when the users or items under consideration have insufficient rating records. With the continued growth of online social networks, the increased user-to-user relationships are reported to be helpful and can alleviate the CF rating sparsity problem. Although researchers have developed a range of social network-based recommender systems, there is no unified model to handle multirelational social networks. To address this challenge, this paper represents different user relationships in a multigraph and develops a multigraph ranking model to identify and recommend the nearest neighbors of particular users in high-order environments. We conduct empirical experiments on two real-world datasets: 1) Epinions and 2) Last.fm, and the comprehensive comparison with other approaches demonstrates that our model improves recommendation performance in terms of both recommendation coverage and accuracy, especially when the rating data are sparse.

Existing extreme learning algorithm have not taken into account four issues: 1) complexity; 2) uncertainty; 3) concept drift; and 4) high dimensionality. A novel incremental type-2 meta-cognitive extreme learning machine (ELM) called evolving type-2 ELM (eT2ELM) is proposed to cope with the four issues in this paper. The eT2ELM presents three main pillars of human meta-cognition: 1) what-to-learn; 2) how-to-learn; and 3) when-to-learn. The what-to-learn component selects important training samples for model updates by virtue of the online certainty-based active learning method, which renders eT2ELM as a semi-supervised classifier. The how-to-learn element develops a synergy between extreme learning theory and the evolving concept, whereby the hidden nodes can be generated and pruned automatically from data streams with no tuning of hidden nodes. The when-to-learn constituent makes use of the standard sample reserved strategy. A generalized interval type-2 fuzzy neural network is also put forward as a cognitive component, in which a hidden node is built upon the interval type-2 multivariate Gaussian function while exploiting a subset of Chebyshev series in the output node. The efficacy of the proposed eT2ELM is numerically validated in 12 data streams containing various concept drifts. The numerical results are confirmed by thorough statistical tests, where the eT2ELM demonstrates the most encouraging numerical results in delivering reliable prediction, while sustaining low complexity.

The papers in this special section address the use of fuzzy systems in data science. Fuzzy Systems in Data Science data science employs theories and techniques drawn from many fields to achieve knowledge extraction from large volumes of data. To facilitate and automate the achievement of useful insights, predictions, and decisions from collected data sets, scientists and engineers turned to Machine Learning solutions. Currently, with the birth of Big Data, the set of opportunities for inquiry has grown exponentially, thanks to these large and complex data sets. Being able to exploit this massive data effectively provides useful knowledge for decision-making or the exploration and comprehension of the phenomenon that produced the data. Data science includes several problems and tasks depending on the nature of the data and the type of knowledge that is to be extracted. From predictive and descriptive analysis to the application of recommendation systems, scientists and practitioners have considered the use of Soft Computing techniques based on fuzzy sets to overcome their problems. Fuzzy approaches have been applied with much success in several areas related to Data Science. Fuzzy sets can be used to effectively describe and incorporate uncertain data values, data measurement, and data relations; fuzzy systems can directly aid reasoning and inferencing in a learning machine and can be combined with machine learning methodologies to model human behaviors and complex data systems.

The age of online data stream and dynamic environments results in the increasing demand of advanced machine learning techniques to deal with concept drifts in large data streams. Evolving fuzzy systems (EFS) are one of recent initiatives from the fuzzy system community to resolve the issue. Existing EFSs are not robust against data uncertainty, temporal system dynamics, and the absence of system order, because a vast majority of EFSs are designed in the type-1 feedforward network architecture. This paper aims to solve the issue of data uncertainty, temporal behavior, and the absence of system order by developing a novel evolving recurrent fuzzy neural network, called evolving type-2 recurrent fuzzy neural network (eT2RFNN). eT2RFNN is constructed in a new recurrent network architecture, featuring double recurrent layers. The new recurrent network architecture evolves a generalized interval type-2 fuzzy rule, where the rule premise is built upon the interval type-2 multivariate Gaussian function, whereas the rule consequent is crafted by the nonlinear wavelet function. The eT2RFNN adopts a holistic concept of evolving systems, where the fuzzy rule can be automatically generated, pruned, merged, and recalled in the single-pass learning mode. eT2RFNN is capable of coping with the problem of high dimensionality because it is equipped with online feature selection technology. The efficacy of eT2RFNN was experimentally validated using artificial and real-world data streams and compared with prominent learning algorithms. eT2RFNN produced more reliable predictive accuracy, while retaining lower complexity than its counterparts.

Data science is a research field concerned with processes and systems that extract knowledge from massive amounts of data. In some situations, however, data shortage renders existing data-driven methods difficult or even impossible to apply. Transfer learning has recently emerged as a way of exploiting previously acquired knowledge to solve new yet similar problems much more quickly and effectively. In contrast to classical data-driven machine learning methods, transfer learning methods exploit the knowledge accumulated from data in auxiliary domains to facilitate predictive modeling in the current domain. A significant number of transfer learning methods that address classification tasks have been proposed, but studies on transfer learning in the case of regression problems are still scarce. This study focuses on using transfer learning techniques to handle regression problems in a domain that has insufficient training data. We propose an original fuzzy regression transfer learning method, based on fuzzy rules, to address the problem of estimating the value of the target for regression. A Takagi-Sugeno fuzzy regression model is developed to transfer knowledge from a source domain to a target domain. Experimental results using synthetic data and real-world datasets demonstrate that the proposed fuzzy regression transfer learning method significantly improves the performance of existing models when tackling regression problems in the target domain.

Traditional relational topic models provide a successful way to discover the hidden topics from a document network. Many theoretical and practical tasks, such as dimensional reduction, document clustering, and link prediction, could benefit from this revealed knowledge. However, existing relational topic models are based on an assumption that the number of hidden topics is known a priori, which is impractical in many real-world applications. Therefore, in order to relax this assumption, we propose a nonparametric relational topic model using stochastic processes instead of fixed-dimensional probability distributions in this paper. Specifically, each document is assigned a Gamma process, which represents the topic interest of this document. Although this method provides an elegant solution, it brings additional challenges when mathematically modeling the inherent network structure of typical document network, i.e., two spatially closer documents tend to have more similar topics. Furthermore, we require that the topics are shared by all the documents. In order to resolve these challenges, we use a subsampling strategy to assign each document a different Gamma process from the global Gamma process, and the subsampling probabilities of documents are assigned with a Markov Random Field constraint that inherits the document network structure. Through the designed posterior inference algorithm, we can discover the hidden topics and its number simultaneously. Experimental results on both synthetic and real-world network datasets demonstrate the capabilities of learning the hidden topics and, more importantly, the number of topics.

One challenge in recommender system is to deal with data sparsity. To handle this issue, social tags are utilized to bring disjoint domains together for knowledge transfer in cross-domain recommendation. The most intuitive way is to use common tags that present in both source and target domains. However, it is difficult to obtain a strong domain connection by exploiting a small amount of common tags, especially when the tagging data in target domain is too scarce to share enough common tags with source domain. In this paper we propose a novel framework, called Enhanced Tag-induced Cross Domain Collaborative Filtering (ETagiCDCF), to integrate the rich information contained in domain dependent tags into recommendation procedure. We perform experiments on two public datasets and compare with several single and cross domain recommendation approaches, the results demonstrate that ETagiCDCF can effectively address data sparseness and improve recommendation performance.

We propose an electroencephalography (EEG) prediction system based on a recurrent fuzzy neural network (RFNN) architecture to assess drivers' fatigue degrees during a virtual-reality (VR) dynamic driving environment. Prediction of fatigue degrees is a crucial and arduous biomedical issue for driving safety, which has attracted growing attention of the research community in the recent past. Meanwhile, combined with the benefits of measuring EEG signals facilitates, many EEG-based brain-computer interfaces (BCIs) have been developed for use in real-time mental assessment. In the literature, EEG signals are severely blended with stochastic noise; therefore, the performance of BCIs is constrained by low resolution in recognition tasks. For this rationale, independent component analysis (ICA) is usually used to find a source mapping from original data that has been blended with unrelated artificial noise. However, the mechanism of ICA cannot be used in real-time BCI design. To overcome this bottleneck, the proposed system in this paper utilizes a recurrent self-evolving fuzzy neural work (RSEFNN) to increase memory capability for adaptive noise cancellation when assessing drivers' mental states during a car driving task. The experimental results without the use of ICA procedure indicate that the proposed RSEFNN model remains superior performance compared with the state-of-the-arts models.

A brain-computer interface (BCI) system provides a convenient means of communication between the human brain and a computer, which is applied not only to healthy people but also for people that suffer from motor neuron diseases (MNDs). Motor imagery (MI) is one well-known basis for designing Electroencephalography (EEG)-based real-life BCI systems. However, EEG signals are often contaminated with severe noise and various uncertainties, imprecise and incomplete information streams. Therefore, this study proposes spectrum ensemble based on swam-optimized fuzzy integral for integrating decisions from sub-band classifiers that are established by a sub-band common spatial pattern (SBCSP) method. Firstly, the SBCSP effectively extracts features from EEG signals, and thereby the multiple linear discriminant analysis (MLDA) is employed during a MI classification task. Subsequently, particle swarm optimization (PSO) is used to regulate the subject-specific parameters for assigning optimal confidence levels for classifiers used in the fuzzy integral during the fuzzy fusion stage of the proposed system. Moreover, BCI systems usually tend to have complex architectures, be bulky in size, and require time-consuming processing. To overcome this drawback, a wireless and wearable EEG measurement system is investigated in this study. Finally, in our experimental result, the proposed system is found to produce significant improvement in terms of the receiver operating characteristic (ROC) curve. Furthermore, we demonstrate that a robotic arm can be reliably controlled using the proposed BCI system. This paper presents novel insights regarding the possibility of using the proposed MI-based BCI system in real-life applications.

This paper proposes a fast load pattern extraction approach to solve the time consuming problem in using a traditional k-means clustering method for large volumes of load curves. The approach, based on dimension reduction and sampling, segments and averages sampling characteristic points to reduce the load curve's dimensions, then reduces the overall size of the sample data set using representative random sampling. K-means clustering algorithm is used to extract load patterns from the representative data set, which will be used to classify the full data set. Reducing the size and dimension of the data set allows use of a less complex algorithm, and thus greatly improves the clustering speed. The validity of the approach is proven by experiments designed to evaluate the trade-off between complexity and consistency.

Recommender systems aim to provide personalized suggestions to users by modeling user-interests to deal with information overload problem, which is extremely severe in the era of big data. Since user-interests are drifting due to their taste variation on items, recommender systems without considering that will suffer degradation of prediction accuracy. There are two challenges about adapting to user-interest drift in recommender systems: 1) accurately modeling user-interests is not easy since the drift of user-interests may occur in different direction for each user; 2) item features and user-interests are often incomplete and vague, which makes it more difficult to model user-interests. To handle these two issues, this study proposes a fuzzy user-interest drift detection based recommender system that adapts to user-interest drift and improves prediction accuracy. A fuzzy user-interest consistency model is built based on fuzzy set theories, and a user-interest drift detection approach and algorithms are developed based on concept drift techniques to provide guidance to recommendation generation. Empirical experiments are conducted on synthetic and real-world MovieLens datasets. The results show that the proposed approach improves the performance of recommender systems in metric of MAE.

An important problem that remains in online data mining systems is how to accurately and efficiently detect changes in the underlying distribution of large data streams. The challenge for change detection methods is to maximise the accumulative effect of changing regions with unknown distribution, while at the same time providing sufficient information to describe the nature of the changes. In this paper, we propose a novel change detection method based on the estimation of equal density regions, with the aim of overcoming the issues of instability and inefficiency that underlie methods of predefined space partitioning schemes. Our method is general, nonparametric and requires no prior knowledge of the data distribution. A series of experiments demonstrate that our method effectively detects concept drift in single dimension as well as high dimension data, and is also able to explain the change by locating the data points that contribute most to the change. The detection result is guaranteed by statistical tests.

In order to better model complex real-world data and to develop robust features that capture relevant information, we usually employ unsupervised feature learning to learn a layer of features representations from unlabeled data. However, developing domain-specific features for each task is expensive, time-consuming and requires expertise of the data. In this paper, we introduce multi-instance clustering and graphical learning to unsupervised transfer learning. For a better clustering efficient, we proposed a set of algorithms on the application of traffic data learning, instance feature representation, distance calculation of multi-instance clustering, multi-instance graphical cluster initialisation, multi-instance multi-cluster update, and graphical multi-instance transfer clustering (GMITC). In the end of this paper, we examine the proposed algorithms on the Eastwest datasets by couples of baselines. The experiment results indicate that our proposed algorithms can get higher clustering accuracy and much higher programming speed.

Evolving fuzzy classifiers (EFCs) have achieved immense success in dealing with nonstationary data streams because of their flexible characteristics. Nonetheless, most real-world data streams feature highly uncertain characteristics, which cannot be handled by the type-1 EFC. A novel interval type-2 fuzzy classifier, namely evolving type-2 classifier (eT2Class), is proposed in this paper, which constructs an evolving working principle in the framework of interval type-2 fuzzy system. The eT2Class commences its learning process from scratch with an empty or initially trained rule base, and its fuzzy rules can be automatically grown, pruned, recalled, and merged on the fly referring to summarization power and generalization power of data streams. In addition, the eT2Class is driven by a generalized interval type-2 fuzzy rule, where the premise part is composed of the multivariate Gaussian function with an uncertain nondiagonal covariance matrix, while employing a subset of the nonlinear Chebyshev polynomial as the rule consequents. The efficacy of the eT2Class has been rigorously assessed by numerous real-world and artificial study cases, benchmarked against state-of-the-art classifiers, and validated through various statistical tests. Our numerical results demonstrate that the eT2Class produces more reliable classification rates, while retaining more compact and parsimonious rule base than state-of-the-art EFCs recently published in the literature.

The evolution potential estimation of news events can support the decision making of both corporations and governments. For example, a corporation could manage its public relations crisis in a timely manner if a negative news event about this corporation is known with large evolution potential in advance. However, existing state-of-the-art methods are mainly based on time series historical data, which are not suitable for the news events with limited historical data and bursty properties. In this article, we propose a purely content-based method to estimate the evolution potential of the news events. The proposed method considers a news event at a given time point as a system composed of different keywords, and the uncertainty of this system is defined and measured as the Semantic Uncertainty of this news event. At the same time, an uncertainty space is constructed with two extreme states: the most uncertain state and the most certain state. We believe that the Semantic Uncertainty has correlation with the content evolution of the news events, so it can be used to estimate the evolution potential of the news events. In order to verify the proposed method, we present detailed experimental setups and results measuring the correlation of the Semantic Uncertainty with the Content Change of news events using collected news events data. The results show that the correlation does exist and is stronger than the correlation of value from the time-series-based method with the Content Change. Therefore, we can use the Semantic Uncertainty to estimate the evolution potential of news events.

Webpage recommendations for hot Web events can assist people to easily follow the evolution of these Web events. At the same time, there are different levels of semantic uncertainty underlying the amount of Webpages for a Web event, such as recapitulative information and detailed information. Apparently, the grasp of the semantic uncertainty of Web events could improve the satisfactoriness of Webpage recommendations. However, traditional hit-rate-based or clustering-based Webpage recommendation methods have overlooked these different levels of semantic uncertainty. In this paper, we propose a framework to identify the different underlying levels of semantic uncertainty in terms of Web events, and then utilize these for Webpage recommendations. Our idea is to consider a Web event as a system composed of different keywords, and the uncertainty of this keyword system is related to the uncertainty of the particular Web event. Based on keyword association linked network Web event representation and Shannon entropy, we identify the different levels of semantic uncertainty, and construct a semantic pyramid (SP) to express the uncertainty hierarchy of a Web event. Finally, an SP-based Webpage recommendation system is developed. Experiments show that the proposed algorithm can significantly capture the different levels of the semantic uncertainties of Web events and it can be applied to Webpage recommendations.

Technology progress brings the very rapid growth of patent publications, which increases the difficulty of domain experts to measure the development of various topics, handle linguistic terms used in evaluation and understand massive technological content. To overcome the limitations of keyword-ranking type of text mining result in existing research, and at the same time deal with the vagueness of linguistic terms to assist thematic evaluation, this research proposes a fuzzy set-based topic development measurement (FTDM) approach to estimate and evaluate the topics hidden in a large volume of patent claims using Latent Dirichlet Allocation. In this study, latent semantic topics are first discovered from patent corpus and measured by a temporal-weight matrix to reveal the importance of all topics in different years. For each topic, we then calculate a temporal-weight coefficient based on the matrix, which is associated with a set of linguistic terms to describe its development state over time. After choosing a suitable linguistic term set, fuzzy membership functions are created for each term. The temporal-weight coefficients are then transformed to membership vectors related to the linguistic terms, which can be used to measure the development states of all topics directly and effectively. A case study using solar cell related patents is given to show the effectiveness of the proposed FTDM approach and its applicability for estimating hidden topics and measuring their corresponding development states efficiently.

E-Commerce products often come with rich and tree-structured content information describing the attributes. To well utilize the content information, this study proposed a fuzzy content matching-based recommendation approach to assist e-Commerce customers to choose their truly interested items. In this paper, users' ratings and preferences are represented using fuzzy numbers to remain uncertainties. Tree-structured content information is transformed to a set of descriptors, and users' preferences on these descriptors are derived from fuzzy ratings by using fuzzy number operations. A kind of preference dependence relations is established between descriptors to explore the relations of different content features, and as a base to sketch the complete profile of users. While the extended preference profile of a user is established, given a new item, the fuzzy match degree of the user preference and the item content information is carried out, and then a fuzzy Topsis ranking method is proposed to able to rank all candidate items according to the fuzzy match degrees, and the highest ranked items are recommended to the target user. We conduct empirical experiments on Yelp and MovieLens datasets. The results indicate that the proposed approach improve recommendation performance in terms of both coverage and accuracy.

Most real world classification problems involve a high degree of uncertainty, unsolved by a traditional type-1 fuzzy classifier. In this paper, a novel interval type-2 classifier, namely Evolving Type-2 Classifier (eT2Class), is proposed. The eT2Class features a flexible working principle built upon a fully sequential and local working principle. This learning notion allows eT2Class to automatically grow, adapt, prune, recall its knowledge from data streams in the single-pass learning fashion, while employing loosely coupled fuzzy sub-models. In addition, eT2Class introduces a generalized interval type-2 fuzzy neural network architecture, where a multivariate Gaussian function with uncertain non-diagonal covariance matrixes constructs the rule premise, while the rule consequent is crafted by a local non-linear Chebyshev polynomial. The efficacy of eT2Class is numerically validated by numerical studies with four data streams characterizing non-stationary behaviors, where eT2Class demonstrates the most encouraging learning performance in achieving a tradeoff between accuracy and complexity.

Incorporating the side information of text corpus, i.e., authors, time stamps, and emotional tags, into the traditional text mining models has gained significant interests in the area of information retrieval, statistical natural language processing, and machine learning. One branch of these works is the so-called Author Topic Model (ATM), which incorporates the authors's interests as side information into the classical topic model. However, the existing ATM needs to predefine the number of topics, which is difficult and inappropriate in many real-world settings. In this paper, we propose an Infinite Author Topic (IAT) model to resolve this issue. Instead of assigning a discrete probability on fixed number of topics, we use a stochastic process to determine the number of topics from the data itself. To be specific, we extend a gamma-negative binomial process to three levels in orderto capture the author-document-keyword hierarchical structure. Furthermore, each document is assigned a mixed gamma process that accounts for the multi-author's contribution towards this document. An efficient Gibbs sampling inference algorithm witheach conditional distribution being closed-form is developed for the IAT model. Experiments on several real-world datasets show the capabilities of our IAT model to learn the hidden topics, authors' interests on these topics and the number of topics simultaneously.

Extreme Learning Machine (ELM) is an answer to an increasing demand for a low-cost learning algorithm to handle big data applications. Nevertheless, existing ELMs leave four uncharted problems: complexity, uncertainty, concept drifts, curse of dimensionality. To correct these issues, a novel incremental meta-cognitive ELM, namely Evolving Type-2 Extreme Learning Machine (eT2ELM), is proposed. Et2Elm is built upon the three pillars of meta-cognitive learning, namely what-to-learn, how-to-learn, when-to-learn, where the notion of ELM is implemented in the how-to-learn component. On the other hand, eT2ELM is driven by a generalized interval type-2 Fuzzy Neural Network (FNN) as the cognitive constituent, where the interval type-2 multivariate Gaussian function is used in the hidden layer, whereas the nonlinear Chebyshev function is embedded in the output layer. The efficacy of eT2ELM is proven with four data streams possessing various concept drifts, comparisons with prominent classifiers, and statistical tests, where eT2ELM demonstrates the most encouraging learning performances in terms of accuracy and complexity.

Summary Bilevel decision addresses compromises between two interacting decision entities within a given hierarchical complex system under distributed environments. Bilevel programming typically solves bilevel decision problems. However, formulation of objectives and constraints in mathematical functions is required, which are difficult, and sometimes impossible, in real‐world situations because of various uncertainties. Our study develops a rule‐set based bilevel decision approach, which models a bilevel decision problem by creating, transforming and reducing related rule sets. This study develops a new rule‐sets based solution algorithm to obtain an optimal solution from the bilevel decision problem described by rule sets. A case study and a set of experiments illustrate both functions and the effectiveness of the developed algorithm in solving a bilevel decision problem. Copyright © 2012 John Wiley & Sons, Ltd.

Collaborative filtering (CF) is the most popular approach in personalized recommender systems. Although CF approaches have successfully been used and have the advantage in that it is unnecessary to analyze item content when generating recommendations, they nevertheless suffer from problems with accuracy. In this paper, we propose a new CF approach to improve recommendation performance. First, a new information entropy-driven user similarity measure model is proposed to measure the relative difference between ratings. A Manhattan distance-based model is then developed to address the fat tail problem by estimating the alternative active user average rating. The effectiveness of the proposed approach is analyzed on public and private data sets. As a result of the introduction of the new similarity measure and average rating estimation, we demonstrate that the proposed new CF recommendation approach provides better recommendations.

Graph mining has been a popular research area because of its numerous application scenarios. Many unstructured and structured data can be represented as graphs, such as, documents, chemical molecular structures, and images. However, an issue in relation to current research on graphs is that they cannot adequately discover the topics hidden in graph-structured data which can be beneficial for both the unsupervised learning and supervised learning of the graphs. Although topic models have proved to be very successful in discovering latent topics, the standard topic models cannot be directly applied to graph-structured data due to the "bag-of-word" assumption. In this paper, an innovative graph topic model (GTM) is proposed to address this issue, which uses Bernoulli distributions to model the edges between nodes in a graph. It can, therefore, make the edges in a graph contribute to latent topic discovery and further improve the accuracy of the supervised and unsupervised learning of graphs. The experimental results on two different types of graph datasets show that the proposed GTM outperforms the latent Dirichlet allocation on classification by using the unveiled topics of these two models to represent graphs.

Machine learning plays an important role in data classification and data-based prediction. In some real-world applications, however, the training data (coming from the source domain) and test data (from the target domain) come from different domains or time periods, and this may result in the different distributions of some features. Moreover, the values of the features and/or labels of the datasets might be nonnumeric and involve vague values. Traditional learning-based prediction and classification methods cannot handle these two issues. In this study, we propose a multistep fuzzy bridged refinement domain adaptation algorithm, which offers an effective way to deal with both issues. It utilizes a concept of similarity to modify the labels of the target instances that were initially predicted by a shift-unaware model. It then refines the labels using instances that are most similar to a given target instance. These instances are extracted from mixture domains composed of source and target domains. The proposed algorithm is built on a basis of some data and refines the labels, thus performing completely independently of the shift-unaware prediction model. The algorithm uses a fuzzy set-based approach to deal with the vague values of the features and labels. Four different datasets are used in the experiments to validate the proposed algorithm. The results, which are compared with those generated by the existing domain adaptation methods, demonstrate a significant improvement in prediction accuracy in both the above-mentioned datasets.

The rapid development of e-learning systems provides learners with great opportunities to access learning activities online, and this greatly supports and enhances the learning practices. However, an issue reduces the success of application of e-learning systems; too many learning activities (such as various leaning materials, subjects, and learning resources) are emerging in an e-learning system, making it difficult for individual learners to select proper activities for their particular situations/requirements because there is no personalized service function. Recommender systems, which aim to provide personalized recommendations for products or services, can be used to solve this issue. However, e-learning systems need to be able to handle certain special requirements: 1) leaning activities and learners' profiles often present tree structures; 2) learning activities contain vague and uncertain data, such as the uncertain categories that the learning activities belong to; 3) there are pedagogical issues, such as the precedence relations between learning activities. To deal with the three requirements, this study first proposes a fuzzy tree-structured learning activity model, and a learner profile model to comprehensively describe the complex learning activities and learner profiles. In the two models, fuzzy category trees and related similarity measures are presented to infer the semantic relations between learning activities or learner requirements. Since it is impossible to have two completely same trees, in practice, a fuzzy tree matching method is carefully discussed. A fuzzy tree matching-based hybrid learning activity recommendation approach is then developed. This approach takes advantage of both the knowledge-based and collaborative filtering-based recommendation approaches, and considers both the semantic and collaborative filtering similarities between learners. Finally, an e-learning recommender system prototype is well designed and developed based on the proposed models and recommendation approach. Experiments are done to evaluate the proposed recommendation approach, and the experimental results demonstrate the good accuracy performance of the proposed approach. A comprehensive case study about learning activity recommendation further demonstrates the effectiveness of the fuzzy tree matching-based personalized e-learning recommender system in practice.

The Web creates excellent opportunities for businesses to provide personalized online services to their customers. Recommender systems aim to automatically generate personalized suggestions of products/services to customers (businesses or individuals). Although recommender systems have been well studied, there are still two challenges in the development of a recommender system, particularly in real-world B2B e-services: (1) items or user profiles often present complicated tree structures in business applications, which cannot be handled by normal item similarity measures and (2) online users' preferences are often vague and fuzzy, and cannot be dealt with by existing recommendation methods. To handle both these challenges, this study first proposes a method for modeling fuzzy tree-structured user preferences, in which fuzzy set techniques are used to express user preferences. A recommendation approach to recommending tree-structured items is then developed. The key technique in this study is a comprehensive tree matching method, which can match two tree-structured data and identify their corresponding parts by considering all the information on tree structures, node attributes, and weights. Importantly, the proposed fuzzy preference tree-based recommendation approach is tested and validated using an Australian business dataset and the MovieLens dataset. Experimental results show that the proposed fuzzy tree-structured user preference profile reflects user preferences effectively and the recommendation approach demonstrates excellent performance for tree-structured items, especially in e-business applications. This study also applies the proposed recommendation approach to the development of a Web-based business partner recommender system.

The rapid development of e-learning systems provides learners great opportunities to access the learning activities online, which greatly supports and enhances learning practices. However, too many learning activities are emerging in the e-learning system, which makes it difficult for learners to select proper ones for their particular situations since there is no personalised service function. Recommender systems, which aim to provide personalised recommendations, can be used to solve this issue. However, e-learning systems have two features to handle: (1) data of learners and leaning activities often present tree structures; (2) data are often vague and uncertain in practice. In this study, a fuzzy tree-structured data model is proposed to comprehensively describe the complex learning activities and learner profiles. A tree matching method is then developed to match the similar learning activities or learners. To deal with the uncertain category issues, a fuzzy category tree and relevant similarity measure are developed. A hybrid recommendation approach, which considers precedence relations between learning activities and combines the semantic and collaborative filtering similarities between learners, is developed. The proposed approach can handle the special requirements in e-learning environment and make proper recommendations in e-learning systems.

To evaluate solutions of power distribution system planning (PDSP) is an essential task in smart grid and requires multi-criteria decision making (MCDM). However, the vagueness of attribute values and the fuzziness of weights of criteria need integrate fuzzy techniques with MCDM. In order to incorporate the issues with uncertainty in PDSP evaluation, this paper proposes a novel PDSP approach based on linear programming model and ELECTRE IIL The incomplete weight preference information of decision-maker is elicited and expressed by a group of weight constraint functions, combined these functions with the simple multi-attribute rating technique, a linear programming model is set up to obtain the weights for each solution. Then with the weights and a PDSP model based on ELECTRE III model, the outranking score of each solution compared with other solutions can be calculated, and a net present score for each solution will be obtained for ranking these solutions, DM can choose one desired. A case is demonstrated to show the evaluation process using this approach and the results indicate that this approach incorporating the issues with uncertainty is robust for PDSP evaluation. The results are acceptable to DM.

Similarity measures are the foundations of many research areas, e.g. information retrieval, recommender system and machine learning algorithms. Promoted by these application scenarios, a number of similarity measures have been proposed and proposing. In these state-of-the-art measures, vector-based representation is widely accepted based on Vector Space Model (VSM) in which an object is represented as a vector composed of its features. Then, the similarity between two objects is evaluated by the operations on two corresponding vectors, like cosine, extended jaccard, extended dice and so on. However, there is an assumption that the features are independent of each others. This assumption is apparently unrealistic, and normally, there are relations between features, i.e. the co-occurrence relations between keywords in text mining area. In this paper, a space geometry-based method is proposed to extend the VSM from the orthogonal coordinate system (OVSM) to affine coordinate system (AVSM) and OVSM is proved to be a special case of AVSM. Unit coordinate vectors of AVSM are inferred by the relations between features which are considered as angles between these unit coordinate vectors. At last, five different similarity measures are extended from OVSM to AVSM using unit coordinate vectors of AVSM. Within the numerous application fields of similarity measures, the task of text clustering is selected to be the evaluation criterion. Documents are represented as vectors in OVSM and AVSM, respectively. The clustering results show that AVSM outweighs the OVSM.

This paper develops a bi‐level decision model and a solution approach to optimizing service features for a company to reduce its customer churn rate. First, a bi‐level decision model, together with its modeling approach, are developed to describe the gaming relationship between decision makers in a company (service provider) and its customers. Then, a practical solution approach to reaching solutions for the bi‐level‐modeled customer churn problem is developed. Finally, experiments and case studies are conducted to illustrate the bi‐level decision model and the solution approach.

Fault tree analysis for nuclear power plant probabilistic safety assessment is an intricate process. Personal computer-based software systems have therefore been developed to conduct this analysis. However, all existing fault tree analysis software systems only accept quantitative data to characterized basic event reliabilities. In real-world applications, basic event reliabilities may not be represented by quantitative data but by qualitative justifications. The motivation of this work is to develop an intelligent system by fuzzy reliability algorithm in fault tree analysis, which can accept not only quantitative data but also qualitative information to characterized reliabilities of basic events. In this paper, a newly-developed system called InFaTAS-NuSA is presented and its main features and capabilities are discussed. To benchmark the applicability of the intelligent concept implemented in InFaTAS-NuSA, a case study is performed and the analysis results are compared to the results obtained from a well-known fault tree analysis software package. The results confirm that the intelligent concept implemented in InFaTAS-NuSA can be very useful to complement conventional fault tree analysis software systems.

Long-term bank failure prediction is a challenging real world problem in banking ecosystem and machine learning methods have been recently applied to improve the prediction accuracy. However, traditional machine learning methods assume that the training data and the test data are drawn from the same distribution, which is hard to be met in real world banking applications. This paper proposes a novel algorithm known as fuzzy refinement domain adaptation to solve this problem based on the ecosystem-oriented architecture. The algorithm utilizes the fuzzy system and similarity/dissimilarity concepts to modify the target instances' labels which were initially predicted by a shift-unaware prediction model. It employs a classifier to modify the label values of target instances based on their similarity/dissimilarity to the candidate positive and negative instances in mixture domains. Thirty six experiments are performed using three different shift-unaware prediction models. In these experiments bank failure financial data is used to evaluate the algorithm. The results demonstrate that the proposed algorithm significantly improves predictive accuracy and outperforms other refinement algorithms.

With the rapid growth of digital systems and associated information technologies, there is an emerging trend in the global economy to build digital customer relationship management (CRM) systems. This trend is more obvious in the telecommunications industry, where companies become increasingly digitalized. Customer churn prediction is a main feature of in modern telecomcommunication CRM systems. This research conducts a real-world study on customer churn prediction and proposes the use of boosting to enhance a customer churn prediction model. Unlike most research that uses boosting as a method to boost the accuracy of a given basis learner, this paper tries to separate customers into two clusters based on the weight assigned by the boosting algorithm. As a result, a higher risk customer cluster has been identified. Logistic regression is used in this research as a basis learner, and a churn prediction model is built on each cluster, respectively. The result is compared with a single logistic regression model. Experimental evaluation reveals that boosting also provides a good separation of churn data; thus, boosting is suggested for churn prediction analysis.

In the on-demand cloud environment, web application providers have the potential to scale virtual resources up or down to achieve cost-effective outcomes. True elasticity and cost-effectiveness in the pay-per-use cloud business model, however, have not yet been achieved. To address this challenge, we propose a novel cloud resource auto-scaling scheme at the virtual machine (VM) level for web application providers. The scheme automatically predicts the number of web requests and discovers an optimal cloud resource demand with cost-latency trade-off. Based on this demand, the scheme makes a resource scaling decision that is up or down or NOP (no operation) in each time-unit re-allocation. We have implemented the scheme on the Amazon cloud platform and evaluated it using three real-world web log datasets. Our experiment results demonstrate that the proposed scheme achieves resource auto-scaling with an optimal cost-latency trade-off, as well as low SLA violations.

Machine learning methods have attracted attention of researches in computational fields such as classification/categorization. However, these learning methods work under the assumption that the training and test data distributions are identical. In some real world applications, the training data (from the source domain) and test data (from the target domain) come from different domains and this may result in different data distributions. Moreover, the values of the features and/or labels of the data sets could be non-numeric and contain vague values. In this study, we propose a fuzzy domain adaptation method, which offers an effective way to deal with both issues. It utilizes the similarity concept to modify the target instances' labels, which were initially classified by a shift-unaware classifier. The proposed method is built on the given data and refines the labels. In this way it performs completely independently of the shift-unaware classifier. As an example of text categorization, 20Newsgroup data set is used in the experiments to validate the proposed method. The results, which are compared with those generated when using different baselines, demonstrate a significant improvement in the accuracy.

Situation awareness (SA), a state in the mind of a human, is essential to conduct decision-making activities. It is about the perception of the elements in the environment, the comprehension of their meaning, and the projection of their status in the near future. Two decades of investigation and analysis of accidents have showed that SA was behind of many serious large-scale technological systems' accidents. This emphasizes the importance of SA support systems development for complex and dynamic environments. This paper presents a fuzzy dynamic Bayesian network-based situation assessment approach to support the operators in decision making process in hazardous situations. The approach includes a dynamic Bayesian network-based situational network to model the hazardous situations where the existence of the situations can be inferred by sensor observations through the SCADA monitoring system using a fuzzy quantizer method. In addition to generate the assessment result, a fuzzy risk estimation method is proposed to show the risk level of situations. Ultimately a hazardous environment from U.S. Chemical Safety Board investigation reports has been used to illustrate the application of proposed approach.

Technology Intelligence (TI) indicates the concept and applications that transform data hidden in patents or scientific literature into technical insight for technology development planning and strategies formulation. Although much effort has been put into technology trend analysis in existing research, the majority of the results are still obtained from expert opinions on the basis of historical trends presented by content-based Technology Intelligence tools. To improve this situation, this paper proposes a time-series-based framework for TI that enables the system to be more effective when dealing with trend prediction requirements. Time-series analysis module is first applied in TI framework to process patent time series for technology trend predictions in a real sense, at the same time overcome the problem that prediction of future data points' values is insufficient to support TI construction. Based on explicit patent attributes and unknown patterns learned from the historical data, the framework combines the "trend" and "content" knowledge by analyzing both time-related property and semantic attributes of patent data, to support technology development planning more efficiently and satisfactorily. A case study is presented to demonstrate the validity of trend prediction functionality, which is the emphasis of the whole framework.

The recommender systems field has been well developed in the last few years to provide item recommendations to related users. Existing recommendation approaches, however, assume that an item is described by a single value or a vector. Unfortunately, some items in real world applications, such as telecom products, could have a tree structure. This paper aims to handle this issue by developing a comprehensive fuzzy tree similarity measure. The fuzzy tree similarity measure compares both the concepts and values in two trees of items. The focus of this study is primarily on the fuzzy value similarity between two trees. In the similarity measure, each attribute is associated with a set of linguistic terms to express the value granularly. The node values are first transformed to membership vectors related to the linguistic terms, and the values of the conceptual corresponding nodes are then compared. These local similarities are aggregated into the final fuzzy value similarity between the two trees. A telecom product recommendation case study shows the effectiveness of the proposed fuzzy tree similarity measure and its applicability for telecom product recommendations.

The web provides excellent opportunities to businesses in various aspects of development such as finding a business partner online. However, with the rapid growth of web information, business users struggle with information overload and increasingly find it difficult to locate the right information at the right time. Meanwhile, small and medium businesses (SMBs), in particular, are seeking “one‐to‐one” e‐services from government in current highly competitive markets. How can business users be provided with information and services specific to their needs, rather than an undifferentiated mass of information? An effective solution proposed in this study is the development of personalized e‐services. Recommender systems is an effective approach for the implementation of Personalized E‐Service which has gained wide exposure in e‐commerce in recent years. Accordingly, this paper first presents a hybrid fuzzy semantic recommendation (HFSR) approach which combines item‐based fuzzy semantic similarity and item‐based fuzzy collaborative filtering (CF) similarity techniques. This paper then presents the implementation of the proposed approach into an intelligent recommendation system prototype called Smart BizSeeker, which can recommend relevant business partners to individual business users, particularly for SMBs. Experimental results show that the HFSR approach can help overcome the semantic limitations of classical CF‐based recommendation approaches, namely sparsity and new “cold start” item problems.

Machine learning methods, such as neural network (NN) and support vector machine, assume that the training data and the test data are drawn from the same distribution. This assumption may not be satisfied in many real world applications, like long-term financial failure prediction, because the training and test data may each come from different time periods or domains. This paper proposes a novel algorithm known as fuzzy bridged refinement-based domain adaptation to solve the problem of long-term prediction. The algorithm utilizes the fuzzy system and similarity concepts to modify the target instances' labels which were initially predicted by a shift-unaware prediction model. The experiments are performed using three shift-unaware prediction models based on nine different settings including two main situations: (1) there is no labeled instance in the target domain; (2) there are a few labeled instances in the target domain. In these experiments bank failure financial data is used to validate the algorithm. The results demonstrate a significant improvement in the predictive accuracy, particularly in the second situation identified above.

Reliability data is essential for a nuclear power plant probabilistic safety assessment by fault tree analysis to assess the performance of the safety-related systems. The limitation of conventional reliability data arises from insufficient historical data for probabilistic calculation. This study describes a new approach to calculate nuclear event reliability data by utilizing the concept of failure possibilities, which are expressed in qualitative natural languages, mathematically represented by membership functions of fuzzy numbers, and subjectively justified by a group of experts based on their working experience and expertise. We also propose an area defuzzification technique to convert the membership function into nuclear event reliability data. The actual event reliability data, which are collected from the operational experiences of the reactor protection system in Babcock & Wilcox pressurized water reactor between 1984 and 1998, are then compared with the reliability data calculated from the new approach. The results show that fuzzy failure rates can be used as alternatives for probabilistic failure rates when nuclear event historical data are insufficient or unavailable for probabilistic calculation. This study also confirms that our proposed area defuzzification technique is a suitable technique to defuzzify failure possibilities into nuclear event reliability data.

Multiple periodic factor prediction (MPFP) problems exist widely in multisensor data fusion applications. Development of an effective prediction method should integrate information for multiple periodically changing factors. Because the uncertainty and periodicity coexist in the information used, the prediction method should be able to handle them simultaneously. In this study, complex fuzzy sets are used to represent the information with uncertainty and periodicity. A product-sum aggregation operator (PSAO) is developed for a set of complex fuzzy sets, which is used to integrate information with uncertainty and periodicity, and a PSAO-based prediction (PSAOP) method is then proposed to generate a solution of MPFP problems. This study illustrates the details of the PSAOP method through two real applications in annual sunspot number prediction and bushfire danger rating prediction. Experiments indicate that the proposed PSAOP method effectively handles the uncertainty and periodicity in the information of multiple periodic factors simultaneously and can generate accurate predictions for MPFP problems.

In a trilevel hierarchical decision problem, the objectives and variables of each decision entity at one level are controlled, in part, by the decision entities at other levels. The choice of values for the decision variables at each level may influence the decisions made at other levels, and may thereby improve/reduce the objective for each level. When multiple decision entities are involved at the middle and bottom levels of a trilevel decision problem, the top-level entity's decision will be affected, not only by these followers' individual reactions, but also by the relationships between them. We call this problem a multifollower trilevel (MFTL) decision. This paper firstly defines and analyzes various kinds of relationships between decision entities in an MFTL decision problem. We then propose an MFTL decision making framework, in which 64 standard MFTL decision situations and their possible combinations are identified. To model these MFTL decision situations, we developed an innovative decision entity-relationship diagram (DERD) approach. We also established a general model for MFTL decision making and a set of standard MFTL decision models using trilevel programming. A trilevel decision support system (TLDSS) software has also been developed to transfer a DERD into a programming model. Finally, a case study illustrates typical MFTL decision making models and their development, using both DERD and programming approaches.

Cloud computing has emerging as an extremely popular and cost-effective computational service model using pay-as-you-go executing environments that scale transparently to the user. However, cloud providers should tackle the challenge of configuring their systems to provide maximal performance while minimizing customer's cost of computing resources, which satisfy the customers' various workload requirements. To solve the above challenge, in this paper, we propose an innovative architecture of self-adaptive configuration optimization system which supports dynamic reconfiguration when workloads change. In addition, we develop an optimization algorithm by using genetic algorithm for this system. By using queuing theory and statistic techniques, we model and compute the SLAs metrics which are defined as the fitness function in the optimization algorithm. This optimization system can guide cloud customers to purchase appropriate resources and make decision of deployment configuration such as scale, scheduling and capacity.

Machine learning algorithms, which have been considered as robust methods in different computational fields, assume that the training and test data are drawn from the same distribution. This assumption may be violated in many real world applications like bank failure prediction because training and test data may come from different time periods or domains. An efficient novel algorithm known as Fuzzy Refinement (FR) is proposed in this paper to solve this problem and improve the performance. The algorithm utilizes the fuzzy system and similarity concept to modify the instances' labels in target domain which was initially predicted by shift-unaware Fuzzy Neural Network (FNN) proposed by [1]. The experiments are performed using bank failure financial data of United States to evaluate the algorithm performance. The results address a significant improvement in the predictive accuracy of FNN due to applying the proposed algorithm.

Collaborative filtering (CF) techniques have achieved widespread success in E-commerce nowadays. The tremendous growth of the number of customers and products in recent years poses some key challenges for recommender systems in which high quality recommendations are required and more recommendations per second for millions of customers and products need to be performed. Thus, the improvement of scalability and efficiency of collaborative filtering (CF) algorithms become increasingly important and difficult. In this paper, we developed and implemented a scaling-up item-based collaborative filtering algorithm on MapReduce, by splitting the three most costly computations in the proposed algorithm into four Map-Reduce phases, each of which can be independently executed on different nodes in parallel. We also proposed efficient partition strategies not only to enable the parallel computation in each Map-Reduce phase but also to maximize data locality to minimize the communication cost. Experimental results effectively showed the good performance in scalability and efficiency of the item-based CF algorithm on a Hadoop cluster.

Bilevel decision addresses the problem in which two levels of decision makers each tries to optimize their individual objectives under certain constraints, and to act and react in an uncooperative and sequential manner. Given the difficulty of formulating a bilevel decision problem by mathematical functions, a rule sets–based bilevel decision (RSBLD) model was proposed. This article presents an algorithm to solve a RSBLD problem. A case‐based example is given to illustrate the functions of the proposed algorithm. Finally, a set of experiments is analyzed to further show the functions and the effectiveness of the proposed algorithm.

The time series prediction of avian influenza epidemics is a complex issue, because avian influenza has latent seasonality which is difficult to identify. Although researchers have applied a neural network (NN) model and the Box-Jenkins model for the seasonal epidemic series research area, the results are limited. In this study, we develop a new prediction seasonal auto-regressive-based support vector regression (SAR-SVR) model which combines the seasonal auto-regressive (SAR) model with a support vector regression (SVR) model to address this prediction problem to overcome existing limitations. Fast Fourier transformation is also merged into this method to identify the latent seasonality inside the time series. The experiments demonstrate that the developed SAR-SVR method out-performs SVR, Box-Jenkins models and two layer feed forward NN model-both in accuracy and stability in the avian influenza epidemic disease time series prediction.

H5N1 avian influenza outbreak detection is a significant issue for early warning of epidemics. This paper proposes domain knowledge-based joint one class classification model for avian influenza outbreak. Instead of focusing on manipulations of the one class classification model, we delve into the one class avian influenza dataset, divide it into sub-classes by domain knowledge, train the sub-class classifiers and unify the result of each classifier. The proposed joint method solves the one class classification and features selection problems together. The experiment results demonstrate that the proposed joint model definitely outperforms the normal one class classification model on the animal avian influenza dataset.

The support vector machine (SVM) has provided higher performance than traditional learning machines and has been widely applied in real-world classification problems and nonlinear function estimation problems. Unfortunately, the training process of the SVM is sensitive to the outliers or noises in the training set. In this paper, a common misunderstanding of Gaussian-function-based kernel fuzzy clustering is corrected, and a kernel fuzzy c-means clustering-based fuzzy SVM algorithm (KFCM-FSVM) is developed to deal with the classification problems with outliers or noises. In the KFCM-FSVM algorithm, we first use the FCM clustering to cluster each of two classes from the training set in the high-dimensional feature space. The farthest pair of clusters, where one cluster comes from the positive class and the other from the negative class, is then searched and forms one new training set with membership degrees. Finally, we adopt FSVM to induce the final classification results on this new training set. The computational complexity of the KFCM-FSVM algorithm is analyzed. A set of experiments is conducted on six benchmarking datasets and four artificial datasets for testing the generalization performance of the KFCM-FSVM algorithm. The results indicate that the KFCM-FSVM algorithm is robust for classification problems with outliers or noises.

A complex fuzzy relation is defined as a fuzzy relation whose membership function takes values in the unit circle on a complex plane. This paper first investigates various operation properties of a complex fuzzy relation. It then defines the distance measure of two complex fuzzy relations that can measure the differences between the grades as well as the phases of two complex fuzzy relations. This distance measure is used to define δ-equalities of complex fuzzy relations that coincide with those of fuzzy relations already defined in the literature if complex fuzzy relations reduce to real-valued fuzzy relations. Two complex fuzzy relations are said to be δ-equal if the distance between them is less than 1-δ. This paper shows how various operations between complex fuzzy relations, including T-norms and S-norms, affect given δ-equalities of complex fuzzy relations. Finally, fuzzy inference is examined in the framework of delta- equalities of complex fuzzy relations.

Due to the huge product assortments and complex descriptions of mobile products/services, it is a great challenge for new customers to select appropriate products. To solve this issue, a fuzzy matching based recommendation approach for mobile products/services is proposed in this paper. In this approach, a new customer's requirements are obtained through asking a set of questions and represented by linguistic terms. When modeling the characteristics of mobile products/services, both their attributes and related users' usage records are considered. A fuzzy matching approach is applied to find the products that are most matched to the customer's requirements. A framework of the recommender system for mobile products/services applying the proposed approach is developed.

Association rule mining makes interesting associations and/or correlations among large sets of data. Those associations can be refined as decision rules to be used and stored in a knowledge base system. In this paper, an approach based on association rule and knowledge base is proposed and implemented in the fault diagnosis of a transformer system. According to the features of association rule, the Apriori algorithm is adopted and modified to generate decision rules from power transformer information for building knowledge base, then the rules can be refined to diagnose the fault of the transformer through reasoning, and a prototype system is developed. This approach based on association rule is described in detail and the application is illustrated by an example. A comparison with the IEC (International Electrotechnical Commission) three-ratio method shows the proposed method can provide better accuracy in performance.

Distribution network is the most important asset in electric utilities, to increase its efficiency, there is a need to effectively assess the efficiency of distribution network and provide solution for improvement. This paper presents a knowledge-based efficiency assessment system for distribution network using data envelopment analysis (DEA). From an input-output view, DEA method is used to calculate the efficiency of distribution lines and obtain gap information. Knowledge base is used to store facts and rules, facts include input-outputs data of DEA and other information about structure and operation of distribution lines. The rules can be empirical rules from domain expert or extracted from industry guidelines or government standards by knowledge worker. Considering the service requirements of power supply in rules, DEA assessment results can be effectively used or provided in the solution of improvement by reasoning. The knowledge base technology and DEA integration represents a step towards a real challenge of the near future. In this system, the final decision is based on DEA assessment results and reasoning. The suggested solution can assist decision maker in making planning to further strengthen distribution network in an efficient and effective manner.

Bilevel-programming techniques are developed to handle decentralized problems with two-level decision makers, which are leaders and followers, who may have more than one objective to achieve. This paper proposes a${\lambda}$-cut and goal-programming-based algorithm to solve fuzzy-linear multiple-objective bilevel (FLMOB) decision problems. First, based on the definition of a distance measure between two fuzzy vectors using${\lambda}$-cut, a fuzzy-linear bilevel goal (FLBG) model is formatted, and related theorems are proved. Then, using a${\lambda}$-cut for fuzzy coefficients and a goal-programming strategy for multiple objectives, a${\lambda}$-cut and goal-programming-based algorithm to solve FLMOB decision problems is presented. A case study for a newsboy problem is adopted to illustrate the application and executing procedure of this algorithm. Finally, experiments are carried out to discuss and analyze the performance of this algorithm.

An automated auction is an efficient market institution for real-world trading of commodities. This paper presents a novel fuzzy bidding strategy (FAA-Bid), which employs assessments of multiple attributes of items as well as agents' attitude on bidding item to procure an item in automated auction. The assessment of attributes adapts the fuzzy sets technique to handle uncertainty of the bidding process as well use heuristic rules to determine attitude of bidding agents in simulated auctions to procure goods. The overall assessment is used to determine a price range based on current bid, which finally selects the best one as the new bid.

Strategic bidding problems are becoming key issues in competitive electricity markets. This paper applies bilevel optimization theory to deal with this issue. We first analyze generating company strategic bidding behaviors and build a bilevel optimization model for a day-ahead electricity market. In this bilevel optimization model, each generating company will choose their bids in order to maximize their individual profits. A market operator will determine the output power for each unit and uniform marginal price based on the minimization purchase electricity fare. For solving this competitive strategic bidding problem described by the bilevel optimization model, a particle swarm optimization (PSO)-based algorithm is. Experiment results have demonstrated the validity of the PSO-based algorithm in solving the competitive strategic bidding problems for a day-ahead electricity market.

High pathogenic avian influenza remains rampant and the epidemic size has been growing in the world. The early warning system (EWS) for avian influenza becomes increasingly essential to militating against the risk of outbreak crisis. An EWS can generate timely early warnings to support decision makers in identifying underlying vulnerabilities and implementing relevant strategies. This paper addresses this crucial issue and focuses on how to make full use of previous events to perform comprehensive forecasting and generate reliable warning signals. It proposes a hybrid knowledge-based prediction (HKBP) method which combines case-based reasoning (CBR) with the fuzzy logic technique. The method can improve the prediction accuracy for avian influenza in a specific region at a specific time. An example is presented to illustrate the capabilities and procedures of the HKBP method.

Bilevel decision techniques are developed for decentralized planning problems with decision makers located in a two-level system. This study develops a particle swarm optimization based algorithm to solve fuzzy linear bilevel (FLBL) decision problems. A main advantage of this algorithm is that the optimization technique is adopted directly on FLBL problems by fully considering the original information carried by the fuzzy parameters, thus minimizing information loss. Experiments reveal that this algorithm can effectively solve the fuzzy linear bilevel decision problems.

Product prototype evaluation is an important phase in new product development (NPD). Such evaluation often requires multiple criteria that are within a hierarchy and a group of evaluators. The evaluation process and these evaluation criteria often involve uncertain and fuzzy data in the weights of these criteria and the judgments of these evaluators. To evaluate nonwoven cosmetic product prototypes, this study first develops a NPD evaluation model, which has evaluation criteria within three levels, based on the features of nonwoven products. It then proposes a fuzzy (multi-level) multi-criteria group decision-making (FMCGDM) method for supporting the evaluation task. A fuzzy multi-criteria group decision support system (FMCGDSS) is developed to implement the proposed method and applied in nonwoven cosmetic product development evaluation.

The emerging Semantic Web relies on the development of ontologies and the deployment of data annotated by ontologies. For a certain domain with a suitable ontology developed, its ontology annotated data (or simply ontology data) from different sources is often overlapping. Similar to a data warehousing process that transforms and merges data from different databases, an integration over the Semantic Web data sources needs to match relevant ontology data among them. This study develops a matching method to address the issue of ontology data matching. This method is different from other data matching or merging methods applied to database or data warehouse cleansing in that it employs more similarity measurements by exploring ontology features. Our experiments show that this proposed method increases matching accuracy.

We proposed a cognition-driven decision process model for business intelligence. In this model, a managerpsilas situation awareness (SA) and mental models are developed and enriched for naturalistic decision making based on traditional business intelligence systems. Mental models are also used to supervise the process of situation information retrieval and presentation. The final decision-making process is based on recognition-primed decision model.

Bilevel programming deals with hierarchical optimization problems in which the leader at the upper level attempts to optimize his or her objectives, but subject to a set of constraints and the follower's reactions. Typical bilevel programming considers one leader one follower situation and supposes each of them has only one objective. In real world situations, multiple followers may be involved and they may be with different relationships such as sharing decision variables or not, sharing objectives or not. Therefore, the leader's decision will be affected not only by those followers' reactions but also by their relationships. In addition, any of the leader and/or these followers may have multiple conflict objectives that should be optimized simultaneously. Furthermore, the parameters of a bilevel programming model may be described by uncertain values. This paper addresses all these three issues as a whole by particularly focusing on the situation of sharing decision variables among followers. It first proposes a set of fuzzy multi-objective multi-follower bilevel programming (FMMBP) models to describe the complex issue. It then presents an approximation branch-and-bound algorithm to solve the FMMBP problems. Finally, two examples illustrate the proposed models and algorithm.

This paper deals with multi-objective decision-making problem with fuzzy parameters under a group environment, called fuzzy multi-objective group decision-making (FMOGDM). It first presents an FMOGDM method, which integrates fuzzy multi-objective linear programming (FMOLP) with fuzzy group decision making techniques. Based on the method, a fuzzy multiple objective group decision support system (FMOGDSS) is developed. Finally, it gives a case-based example to demonstrate how an FMOLP problem is solved in a group supported by the FMOGDSS.

Basic bilevel programming deals with hierarchical optimization problems in which the leader at the upper level attempts to optimize his/her objective, subject to a set of constraints and his/her follower's solution, and the follower at the lower level tries to find an optimized strategy according to each of possible decisions made by the leader. Three issues may be involved in a basic bilevel decision problem. One is that bilevel decision making model may involve uncertain parameters which appear either in the objective functions or constraints of the leader or the follower or both. Second, the leader and the follower may have multiple conflict objectives that should be optimized simultaneously. Third, there may have multiple followers in a real decision situation. Following our previous work, this study proposes a set of fuzzy multi-objective multi-follower linear bilevel programming models to describe the three issues. It also develops an approximation branch-and-bound algorithm to solve such kinds of problems.

An exploratory study of Web-based cognitive business intelligence systems (CBIS) is presented in this paper. The underpinning concepts and theories are situation awareness, mental model, and naturalistic decision making (NDM). The CBIS is an extension of the traditional business intelligence system with cognitive orientation. It focuses on developing, enriching, and utilizing the executive's situation awareness, mental models, and other past experience during human-computer interaction, which drives the decision process to approach a naturalistic decision.

Electronic-service ~e-service! intelligence is a new research field that deals with fundamental roles, social impacts, and practical applications of various intelligent technologies and methodologies on Internet-based e-services.It has been recently identified as a novel direction and a next stage of e-services for current and future development.E-services involve various types, delivery systems, advanced information technologies, methodologies, and applications of online services that are provided by e-government, e-business, e-commerce, e-market, e-finance, and e-learning systems, to name a few.They thus offer great opportunities and challenges for many areas, such as government, business, commerce, marketing, finance, and education.E-service intelligence comes from real-world requirements, where many government and business organizations are nowadays developing their e-service systems with various intelligent functions.These organizations need to integrate various intelligent technologies and methodologies into their e-service systems for realizing intelligent online services for their users.Over the past decade, in most developed countries both government and business online services have mainly gone through three stages: ~1! online information presentation, ~2! online transaction, and ~3! online information integration.As an example, we take the case of the development of the Australian e-government.At stage one ~1997-2000!the Australian government made information available online and provided simple groupware functionalities such as e-mail queries and

Information integration with the aid of ontology can roughly be divided into two levels: schema level and data level. Most research has been focused on the schema level, i.e., mapping/matching concepts and properties in different ontologies with each other. However, the data level integration is equally important, especially in the decentralized semantic Web environment. Noticing that ontology data (in the form of instances of concepts) from different sources often have different perspectives and may overlap with each other, we develop a matching method that utilizes the features of ontology and employs the machine learning approach to integrate those instances. By exploring ontology features, this method performs better than other general methods, which is revealed in our experiments. Through the process that implements the matching method, ontology data can be integrated together to offer more sophisticated services

In a real world bilevel decision-making, the lower level of a bilevel decision usually involves multiple decision units. This paper proposes an extended Kuhn-Tucker approach for linear bilevel multifollower programming problems with partial shared variables among followers. Finally numeric examples are given to show how the Kuhn-Tucker approach works.

With the fast growth of e-commerce, product reviews on the Web have become an important information source for customers' decision making when they plan to buy products online. As the reviews are often too many for customers to go through, how to automatically classify them into different semantic orientations (i.e. recommend/not recommend) has become a research problem. Different from traditional approaches that treat a review as a whole, our approach performs semantic classifications at the sentence level by realizing reviews often contain mixed feelings or opinions. In this approach, a typical feature selection method based on sentence tagging is employed and a naive Bayes classifier is used to create a base classification model, which is then combined with certain heuristic rules for review sentence classification. Experiments show that this approach achieves better results than using general naive Bayes classifiers.