Evidential deep learning (EDL) models, based on Subjective Logic, introduce a principled and computationally efficient way to make deterministic neural networks uncertainty-aware. The resulting evidential models can quantify fine-grained uncertainty using learned evidence. However, the Subjective-Logic framework constrains evidence to be non-negative, requiring specific activation functions whose geometric properties can induce activation-dependent learning-freeze behavior-a regime where gradients become extremely small for samples mapped into low-evidence regions. We theoretically characterize this behavior and analyze how different evidential activations influence learning dynamics. Building on this analysis, we design a general family of activation functions and corresponding evidential regularizers that provide an alternative pathway for consistent evidence updates across activation regimes. Extensive experiments on four benchmark classification problems (MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet), two few-shot classification problems, and blind face restoration problem empirically validate the developed theory and demonstrate the effectiveness of the proposed generalized regularized evidential models.

Deep learning models with large-scale backbones have been increasingly adopted to tackle complex visual question answering (VQA) problems in real settings. While providing powerful learning capacities to handle the high-dimensional and multimodal VQA data, these models tend to suffer from the memorization effect leading to overconfident predictions. This can significantly limit their applicability in critical domains (e.g., medicine, cyber-security, and public safety), where confidently wrong predictions may lead to severe consequences. In this work, we propose to perform novel low-rank network factorization, resulting in much better-calibrated networks. These low-rank factorized networks are then aggregated into an ensemble guided by a generalized focal loss to further improve the overall performance and calibration. The overall framework, referred to as the Generalized focal Loss Ensemble of low-rank Networks (GLEN), is an important step toward developing well-calibrated VQA models. We theoretically demonstrate that the generalized focal loss provides a more balanced bias-variance trade-off, which guarantees to lower the confidence of the incorrect predictions, resulting in improved calibration. Extensive experimentation conducted on benchmark datasets and comparison on various VQA models shows that GLEN leads to much better calibration over both in-distribution and out-of-distribution data without sacrificing the VQA accuracy.

Video captioning automatically generates natural language phrases to explain the contents in video frames. When deploying captioning models in specialized domains, active learning can help reduce the high annotation cost. However, the generative nature of the captioning process is more complex than standard supervised learning tasks and introduces several challenges for active learning in video captioning. Entropy-based uncertainty estimation, which is widely used in active learning, may be inflated in captioning tasks and mislead active sampling. Another challenge arises from the rich content of videos, as each video could be described in multiple ways. A single uncertainty score obtained from one possible caption does not capture the diversity induced by the rich content. To fill out this gap, we propose identifying multiple sources of uncertainty and performing hierarchical aggregation to integrate uncertainty from distinct sources. This innovates a holistic uncertainty metric to quantify the overall informativeness of video content for active sampling. The overall uncertainty is built upon conditional vacuity, an extension of the second-order uncertainty introduced along with the evidential learning framework to the captioning setting, leading to more robust uncertainty estimation without inflation. Both theoretical analysis and experimental evaluation are conducted to demonstrate the effectiveness of the proposed framework for complex uncertainty estimation and interactive learning.

Reinforcement learning (RL) provides an effective means to capture users' evolving preferences, leading to improved recommendation performance over time. However, existing RL approaches primarily rely on standard exploration strategies, which are less effective for a large item space with sparse reward signals given the limited interactions for most users. Therefore, they may not be able to learn the optimal policy that effectively captures user's evolving preferences and achieves the maximum expected reward over the long term. In this paper, we propose a novel evidential conservative Q-learning framework (ECQL) that learns an effective and conservative recommendation policy by integrating evidence-based uncertainty and conservative learning. ECQL conducts evidence-aware explorations to discover items that are located beyond current observations but reflect users' long-term interests. It offers an uncertainty-aware conservative view on policy evaluation to discourage deviating too much from users' current interests. Two central components of ECQL include a uniquely designed sequential state encoder and a novel conservative evidential-actor-critic (CEAC) module. The former generates the current state of the environment by aggregating historical information and a sliding window that contains the current user interactions as well as newly recommended items from RL exploration that may represent short and long-term interests respectively. The latter performs an evidence-based rating prediction by maximizing the conservative evidential Q-value and leverages an uncertainty-aware ranking score to explore the item space for a more diverse and valuable recommendation. Experiments on multiple real-world dynamic datasets demonstrate the state-of-the-art performance of ECQL and its capability to capture users' long-term interests.

Out-of-distribution (OOD) detection methods have recently become more prominent, serving as a core element in safety-critical autonomous systems. One major purpose of OOD detection is to reject invalid inputs that could lead to unpredictable errors and compromise safety. Due to the cost of labeled data, recent works have investigated the feasibility of self-supervised learning (SSL) OOD detection, unlabled OOD detection, and zero shot OOD detection. In this work, we identify a set of conditions for a theoretical guarantee of failure in unlabeled OOD detection algorithms from an information-theoretic perspective. These conditions are present in all OOD tasks dealing with real world data: I) we provide theoretical proof of unlabeled OOD detection failure when there exists zero mutual information between the learning objective and the in-distribution labels, a.k.a. ‘label blindness’, II) we define a new OOD task – Adjacent OOD detection – that tests for label blindness and accounts for a previously ignored safety gap in all OOD detection benchmarks, and III) we perform experiments demonstrating that existing unlabeled OOD methods fail under conditions suggested by our label blindness theory and analyze the implications for future research in unlabeled OOD methods.

Fine-Grained Openset Detection (FGOD) poses a fundamental challenge due to the similarity between the openset classes and those closed-set ones. Since real-world objects/entities tend to form a hierarchical structure, the fine-grained relationship among the closed-set classes as captured by the hierarchy could potentially improve the FGOD performance. Intuitively, the hierarchical dependency among different classes allows the model to recognize their subtle differences, which in turn makes it better at differentiating similar open-set classes even they may share the same parent. However, simply performing openset detection in a top-down fashion by building a local detector for each node may result in a poor detection performance. Our theoretical analysis also reveals that maximizing the probability of the path leading to the ground-truth leaf node also results in a sub-optimal training process. To systematically address this issue, we propose to formulate a novel state-based node representation, which constructs a state space based upon the entire hierarchical structure. We prove that the state-based representation guarantees to maximize the probability on the path leading to the ground-truth leaf node. Extensive experiments on multiple real-world hierarchical datasets clearly demonstrate the superior performance of the proposed method.

Few-shot learning (FSL) is essential in many practical applications. However, the limited training examples make the models more vulnerable to label noise, which can lead to poor generalization capability. To address this critical challenge, we propose a curriculum meta-learning model that employs a novel dual-level class-example sampling strategy to create a robust curriculum for adaptive task distribution formulation and robust model training. The dual-level framework proposes a heuristic class sampling criterion that measures pairwise class boundary complexity to form a class curriculum; it uses effective example sampling through an under-trained proxy model to form an example curriculum. By utilizing both class-level and example-level information, our approach is more robust to handle limited training data and noisy labels that commonly occur in few-shot learning tasks. The model has efficient convergence behavior, which is verified through rigorous convergence analysis. Additionally, we establish a novel error bound through a hierarchical PAC-Bayesian analysis for curriculum meta-learning under noise. We conduct extensive experiments that demonstrate the effectiveness of our framework in outperforming existing noisy few-shot learning methods under various few-shot classification benchmarks. Our code is available at https://github.com/ritmininglab/DCML.

Subset or core-set selection offers a data-efficient way for training deep learning models. One-shot subset selection poses additional challenges as subset selection is only performed once and full set data become unavailable after the selection. However, most existing methods tend to choose either diverse or difficult data samples, which fail to faithfully represent the joint data distribution that is comprised of both feature and label information. The selection is also performed independently from the subset size, which plays an essential role in choosing what types of samples. To address this critical gap, we propose to conduct Feature similarity and Label variability Balanced One-shot Subset Selection (BOSS), aiming to construct an optimal size-aware subset for data-efficient deep learning. We show that a novel balanced core-set loss bound theoretically justifies the need to simultaneously consider both diversity and difficulty to form an optimal subset. It also reveals how the subset size influences the bound. We further connect the inaccessible bound to a practical surrogate target which is tailored to subset sizes and varying levels of overall difficulty. We design a novel Beta-scoring importance function to delicately control the optimal balance of diversity and difficulty. Comprehensive experiments conducted on both synthetic and real data justify the important theoretical properties and demonstrate the superior performance of BOSS as compared with the competitive baselines.

By leveraging a hierarchical structure of known classes, Hierarchical Novelty Detection (HND) offers fine-grained detection results that pair detected novel samples with their closest (known) parent class in the hierarchy. Prior knowledge on the parent class provides valuable insights to better understand these novel samples. However, traditional novelty detection methods try to separate novel samples from all known classes using uncertainty or distance based metrics so they are incapable of locating the closest known parent class. Since the novel class is also part of the hierarchy, the model can more easily get confused between samples from known classes and those from novel ones. To achieve effective HND, we propose to augment the known (leaf-level) classes with a set of novel classes, each of which is associated with one parent (i.e., non-leaf) class in the original hierarchy. Such a structure allows us to perform novel fine-grained evidence allocation to differentiate known and novel classes guided by a uniquely designed loss function. Our thorough theoretical analysis shows that fine-grained evidence allocation creates an evidence margin to more precisely separate known and novel classes. Extensive experiments conducted on real-world hierarchical datasets demonstrate the proposed model outperforms the strongest baselines and achieves the best HND performance.

Few-shot open-set recognition (FSOSR) aims to detect instances from unseen classes by utilizing a small set of labeled instances from closed-set classes. Accurately rejecting instances from open-set classes in the few-shot setting is fundamentally more challenging due to the weaker supervised signals resulting from fewer labels. Transformer-based few-shot methods exploit attention mapping to achieve a consistent representation. However, the softmax-generated attention map normalizes all the instances that assign unnecessary high attentive weights to those instances not close to the closed-set classes that negatively impact the detection performance. In addition, open-set samples that are similar to a certain closed-set class also pose a significant challenge to most existing FSOSR models. To address these challenges, we propose a novel Meta Evidential Transformer (MET) based FSOSR model that uses an evidential open-set loss to learn more compact closed-set class representations by effectively leveraging similar closed-set classes. MET further integrates an evidence-to-variance ratio to detect fundamentally challenging tasks and uses an evidence-guided cross-attention mechanism to better separate the difficult open-set samples. Experiments on real-world datasets demonstrate consistent improvement over existing competitive methods in unseen class recognition without deteriorating closed-set performance.

Prior research on neural architecture search (NAS) for adversarial robustness has revealed that a lightweight and adversarially robust sub-network could exist in a non-robust large teacher network. Such a sub-network is generally discovered based on heuristic rules to perform neural architecture search. However, heuristic rules are inadequate to handle diverse adversarial attacks and different "teacher" network capacity. To address this key challenge, we propose Reinforced Compressive Neural Architecture Search (RC-NAS), aiming to achieve Versatile Adversarial Robustness. Specifically, we define novel task settings that compose datasets, adversarial attacks, and teacher network configuration. Given diverse tasks, we develop an innovative dual-level training paradigm that consists of a meta-training and a fine-tuning phase to effectively expose the RL agent to diverse attack scenarios (in meta-training), and make it adapt quickly to locate an optimal sub-network (in fine-tuning) for previously unseen scenarios. Experiments show that our framework could achieve adaptive compression towards different initial teacher networks, datasets, and adversarial attacks, resulting in more lightweight and adversarially robust architectures. We also provide a theoretical analysis to explain why the reinforcement learning (RL)-guided adversarial architectural search helps adversarial robustness over standard adversarial training methods.

Sequential recommender systems are designed to capture users' evolving interests over time. Existing methods typically assume a uniform time interval among consecutive user interactions and may not capture users' continuously evolving behavior in the short and long term. In reality, the actual time intervals of user interactions vary dramatically. Consequently, as the time interval between interactions increases, so does the uncertainty in user behavior. Intuitively, it is beneficial to establish a correlation between the interaction time interval and the model uncertainty to provide effective recommendations. To this end, we formulate a novel Evidential Neural Stochastic Differential Equation (*E-NSDE*) to seamlessly integrate NSDE and evidential learning for effective time-aware sequential recommendations. The NSDE enables the model to learn users' fine-grained time-evolving behavior by capturing continuous user representation while evidential learning quantifies both aleatoric and epistemic uncertainties considering interaction time interval to provide model confidence during prediction. Furthermore, we derive a mathematical relationship between the interaction time interval and model uncertainty to guide the learning process. Experiments on real-world data demonstrate the effectiveness of the proposed method compared to the SOTA methods.

Large transformer-based foundation models have been commonly used as pre-trained models that can be adapted to different challenging datasets and settings with state-of-the-art generalization performance. Parameter efficient fine-tuning ($\texttt{PEFT}$) provides promising generalization performance in adaptation while incurring minimum computational overhead. However, adaptation of these foundation models through $\texttt{PEFT}$ leads to accurate but severely underconfident models, especially in few-shot learning settings. Moreover, the adapted models lack accurate fine-grained uncertainty quantification capabilities limiting their broader applicability in critical domains. To fill out this critical gap, we develop a novel lightweight {Bayesian Parameter Efficient Fine-Tuning} (referred to as $\texttt{Bayesian-PEFT}$) framework for large transformer-based foundation models. The framework integrates state-of-the-art $\texttt{PEFT}$ techniques with two Bayesian components to address the under-confidence issue while ensuring reliable prediction under challenging few-shot settings. The first component performs base rate adjustment to strengthen the prior belief corresponding to the knowledge gained through pre-training, making the model more confident in its predictions; the second component builds an evidential ensemble that leverages belief regularization to ensure diversity among different ensemble components. Our thorough theoretical analysis justifies that the Bayesian components can ensure reliable and accurate few-shot adaptations with well-calibrated uncertainty quantification. Extensive experiments across diverse datasets, few-shot learning scenarios, and multiple $\texttt{PEFT}$ techniques demonstrate the outstanding prediction and calibration performance by $\texttt{Bayesian-PEFT}$.

Existing state-of-the-art dense object detection techniques tend to produce a large number of false positive detections on difficult images with complex scenes because they focus on ensuring a high recall. To improve the detection accuracy, we propose an Adaptive Important Region Selection (AIRS) framework guided by Evidential Q-learning coupled with a uniquely designed reward function. Inspired by human visual attention, our detection model conducts object search in a top-down, hierarchical fashion. It starts from the top of the hierarchy with the coarsest granularity and then identifies the potential patches likely to contain objects of interest. It then discards non-informative patches and progressively moves downward on the selected ones for a fine-grained search. The proposed evidential Q-learning systematically encodes epistemic uncertainty in its evidential-Q value to encourage the exploration of unknown patches, especially in the early phase of model training. In this way, the proposed model dynamically balances exploration-exploitation to cover both highly valuable and informative patches. Theoretical analysis and extensive experiments on multiple datasets demonstrate that our proposed framework outperforms the SOTA models.

Multi-label active learning is a crucial yet challenging area in contemporary machine learning, often complicated by a large and sparse label space. This challenge is further exacerbated in active learning scenarios where labeling resources are constrained. Drawing inspiration from existing mixture of Bernoulli models, which efficiently compress the label space into a more manageable weight coefficient space by learning correlated Bernoulli components, we propose a novel model called Evidential Mixture Machines (EMM). Our model leverages mixture components derived from unsupervised learning in the label space and improves prediction accuracy by predicting weight coefficients following the evidential learning paradigm. These coefficients are aggregated as proxy pseudo counts to enhance component offset predictions. The evidential learning approach provides an uncertainty-aware connection between input features and the predicted coefficients and components. Additionally, our method combines evidential uncertainty with predicted label embedding covariances for active sample selection, creating a richer, multi-source uncertainty metric beyond traditional uncertainty scores. Experiments on synthetic datasets show the effectiveness of evidential uncertainty prediction and EMM's capability to capture label correlations through predicted components. Further testing on real-world datasets demonstrates improved performance compared to existing multi-label active learning methods.

Creating accessible software is imperative for making software inclusive for all users.Unfortunately, the topic of accessibility is frequently excluded from computing education, leading to scenarios where students are unaware of either how to develop accessible software or see the need to create it. To address this challenge, we have created a set of educational labs that are systematically designed to not only inform students about fundamental topics in producing accessible software but also demonstrate its importance. Over the previous year, these labs were included in several Computer Science 2 offerings at the Rochester Institute of Technology, comprising a total of 500 student participants. This article discusses instructional observations from these offerings, some of which include the following: (i) many of the research findings from previous efforts remain true with the larger, more diverse evaluation; (ii) our created material and format reduced students’ belief that creating accessible software was difficult in relation to the baseline,; (iii) we observed that our created material and format benefited student opinion that creating accessible software is important, and (iv) computing majors may not be uniformly impacted by experiential educational accessibility material. The educational labs are publicly available on the project website (https://all.rit.edu).

Recent years have seen a surge in research on dynamic graph representation learning, which aims to model temporal graphs that are dynamic and evolving constantly over time. However, current work typically models graph dynamics with recurrent neural networks (RNNs), making them suffer seriously from computation and memory overheads on large temporal graphs. So far, scalability of dynamic graph representation learning on large temporal graphs remains one of the major challenges. In this paper, we present a scalable framework, namely SpikeNet, to efficiently capture the temporal and structural patterns of temporal graphs. We explore a new direction in that we can capture the evolving dynamics of temporal graphs with spiking neural networks (SNNs) instead of RNNs. As a low-power alternative to RNNs, SNNs explicitly model graph dynamics as spike trains of neuron populations and enable spike-based propagation in an efficient way. Experiments on three large real-world temporal graph datasets demonstrate that SpikeNet outperforms strong baselines on the temporal node classification task with lower computational costs. Particularly, SpikeNet generalizes to a large temporal graph (2.7M nodes and 13.9M edges) with significantly fewer parameters and computation overheads.

The Conditional Neural Process (CNP) family of models offer a promising direction to tackle few-shot problems by achieving better scalability and competitive predictive performance. However, the current CNP models only capture the overall uncertainty for the prediction made on a target data point. They lack a systematic fine-grained quantification on the distinct sources of uncertainty that are essential for model training and decision-making under the few-shot setting. We propose Evidential Conditional Neural Processes (ECNP), which replace the standard Gaussian distribution used by CNP with a much richer hierarchical Bayesian structure through evidential learning to achieve epistemic-aleatoric uncertainty decomposition. The evidential hierarchical structure also leads to a theoretically justified robustness over noisy training tasks. Theoretical analysis on the proposed ECNP establishes the relationship with CNP while offering deeper insights on the roles of the evidential parameters. Extensive experiments conducted on both synthetic and real-world data demonstrate the effectiveness of our proposed model in various few-shot settings.

Active learning (AL) aims to sample the most informative data instances for labeling, which makes the model fitting data efficient while significantly reducing the annotation cost. However, most existing AL models make a strong assumption that the annotated data instances are always assigned correct labels, which may not hold true in many practical settings. In this paper, we develop a theoretical framework to formally analyze the impact of noisy annotations and show that systematically re-sampling guarantees to reduce the noise rate, which can lead to improved generalization capability. More importantly, the theoretical framework demonstrates the key benefit of conducting active re-sampling on label-efficient learning, which is critical for AL. The theoretical results also suggest essential properties of an active re-sampling function with a fast convergence speed and guaranteed error reduction. This inspires us to design a novel spatial-temporal active re-sampling function by leveraging the important spatial and temporal properties of maximum-margin classifiers. Extensive experiments conducted on both synthetic and real-world data clearly demonstrate the effectiveness of the proposed active re-sampling function.

We propose a multimodal data fusion framework to systematically analyze human behavioral data from specialized domains that are inherently dynamic, sparse, and heterogeneous. We develop a two-tier architecture of probabilistic mixtures, where the lower tier leverages parametric distributions from the exponential family to extract significant behavioral patterns from each data modality. These patterns are then organized into a dynamic latent state space at the higher tier to fuse patterns from different modalities. In addition, our framework jointly performs pattern discovery and maximum-margin learning for downstream classification tasks by using a group-wise sparse prior that regularizes the coefficients of the maximum-margin classifier. Therefore, the discovered patterns are highly interpretable and discriminative to support downstream classification tasks. Experiments on real-world behavioral data from medical and psychological domains demonstrate that our framework discovers meaningful multimodal behavioral patterns with improved interpretability and prediction performance.

Image captioning offers a computational process to understand the semantics of images and convey them using descriptive language. However, automated captioning models may not always generate satisfactory captions due to the complex nature of the images and the quality/size of the training data. We propose an interactive captioning framework to improve machine-generated captions by keeping humans in the loop and performing an online-offline knowledge acquisition (KA) process. In particular, online KA accepts a list of keywords specified by human users and fuses them with the image features to generate a readable sentence that captures the semantics of the image. It leverages a multimodal conditioned caption completion mechanism to ensure the appearance of all user-input keywords in the generated caption. Offline KA further learns from the user inputs to update the model and benefits caption generation for unseen images in the future. It is built upon a Bayesian transformer architecture that dynamically allocates neural resources and supports uncertainty-aware model updates to mitigate overfitting. Our theoretical analysis also proves that Offline KA automatically selects the best model capacity to accommodate the newly acquired knowledge. Experiments on real-world data demonstrate the effectiveness of the proposed framework.

Evidential deep learning, built upon belief theory and subjective logic, offers a principled and computationally efficient way to turn a deterministic neural network uncertainty-aware. The resultant evidential models can quantify fine-grained uncertainty using the learned evidence. To ensure theoretically sound evidential models, the evidence needs to be non-negative, which requires special activation functions for model training and inference. This constraint often leads to inferior predictive performance compared to standard softmax models, making it challenging to extend them to many large-scale datasets. To unveil the real cause of this undesired behavior, we theoretically investigate evidential models and identify a fundamental limitation that explains the inferior performance: existing evidential activation functions create *zero evidence regions*, which prevent the model to learn from training samples falling into such regions. A deeper analysis of evidential activation functions based on our theoretical underpinning inspires the design of a novel regularizer that effectively alleviates this fundamental limitation. Extensive experiments over many challenging real-world datasets and settings confirm our theoretical findings and demonstrate the effectiveness of our proposed approach.

Machine learning-driven human behavior analysis is gaining attention in behavioral/mental healthcare, due to its potential to identify behavioral patterns that cannot be recognized by traditional assessments. Real-life applications, such as digital behavioral biomarker identification, often require the discovery of complex spatiotemporal patterns in multimodal data, which is largely under-explored. To fill this gap, we propose a novel model that integrates uniquely designed Deep Temporal Sets (DTS) with Evidential Reinforced Attentions (ERA). DTS captures complex temporal relationships in the input and generates a set-based representation, while ERA captures the policy network’s uncertainty and conducts evidence-aware exploration to locate attentive regions in behavioral data. Using child-computer interaction data as a testing platform, we demonstrate the effectiveness of DTS-ERA in differentiating children with Autism Spectrum Disorder and typically developing children based on sequential multimodal visual and touch behaviors. Comparisons with baseline methods show that our model achieves superior performance and has the potential to provide objective, quantitative, and precise analysis of complex human behaviors.

We propose to approach active learning (AL) from a novel perspective of discovering and then ranking potential support vectors by leveraging the key properties of the dual space of a sparse kernel max-margin predictor. We theoretically analyze the change of a hinge loss in the dual form and provide both the upper and lower bounds that are deeply connected to the key geometric properties induced by the dual space, which then help us identify various types of important data samples for AL. These bounds inform the design of a novel sampling strategy that leverages class-wise evidence as a key vehicle, formed through an affine combination of dual variables and kernel evaluation. We construct two distinct types of sampling functions, including discovery and ranking. The former focuses on samples with low total evidence from all classes, which signifies their potential to support exploration; the latter exploits the current decision boundary to identify the most conflicting regions for sampling, aiming to further refine the decision boundary. These two functions, which are complementary to each other, are automatically arranged into a two-phase active sampling process that starts with the discovery and then transitions to the ranking of data points to most effectively balance exploration and exploitation. Experiments on various real-world data demonstrate the state-of-the-art AL performance achieved by our model.

Medical image captioning alleviates the burden of physicians and possibly reduces medical errors by automatically generating text descriptions to describe image contents and convey findings. It is more challenging than conventional image captioning due to the complexity of medical images and the difficulty of aligning image regions with medical terms. In this paper, we propose an evidential interactive learning framework that leverages evidence-based uncertainty estimation and interactive machine learning to improve image captioning with limited labeled data. The interactive learning process involves three stages: keyword prediction, caption generation, and model retraining. First, the model predicts a list of keywords with evidence-based uncertainty and selects the most informative keywords to seek user feedback. Second, user-approved keywords are used as model input to guide the model to generate satisfactory captions. Third, the model is updated based on user-approved keywords and captions, where evidence-based uncertainty is used to allocate different weights to different data instances. Experiments on two medical image datasets illustrate that the proposed framework can effectively learn from human feedback and improve the model's performance in the future.

The recently developed sparse network training methods, such as Lottery Ticket Hypothesis (LTH) and its variants, have shown impressive learning capacity by finding sparse sub-networks from a dense one. While these methods could largely sparsify deep networks, they generally focus more on realizing comparable accuracy to dense counterparts yet neglect network calibration. However, how to achieve calibrated network predictions lies at the core of improving model reliability, especially when it comes to addressing the overconfident issue and out-of-distribution cases. In this study, we propose a novel Distributionally Robust Optimization (DRO) framework to achieve an ensemble of lottery tickets towards calibrated network sparsification. Specifically, the proposed DRO ensemble aims to learn multiple diverse and complementary sparse sub-networks (tickets) with the guidance of uncertainty sets, which encourage tickets to gradually capture different data distributions from easy to hard and naturally complement each other. We theoretically justify the strong calibration performance by showing how the proposed robust training process guarantees to lower the confidence of incorrect predictions. Extensive experimental results on several benchmarks show that our proposed lottery ticket ensemble leads to a clear calibration improvement without sacrificing accuracy and burdening inference costs. Furthermore, experiments on OOD datasets demonstrate the robustness of our approach in the open-set environment.

In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, "testing'', which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for "testing'' (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing. ATL provides an unbiased sample-efficient estimation of the model risk during active learning. It leverages test samples annotated from different periods of a dynamic active learning process to achieve fair model evaluations based on a theoretically guaranteed optimal integration of different test samples. Periodic testing also enables effective early-stopping to further save the total annotation cost. ATL further integrates an "active feedback'' mechanism, which is inspired by human learning, where the teacher (active tester) provides immediate guidance given by the prior performance of the student (active learner). Our theoretical result reveals that active feedback maintains the label complexity of the integrated learning-testing objective, while improving the model's generalization capability. We study the realistic setting where we maximize the performance gain from choosing "testing'' samples for feedback without sacrificing the risk estimation accuracy. An agnostic-style analysis and empirical evaluations on real-world datasets demonstrate that the ATL framework can effectively improve the annotation efficiency of both active learning and evaluation tasks.

We present a novel dynamic recommendation model that focuses on users who have interactions in the past but turn relatively inactive recently. Making effective recommendations to these time-sensitive cold-start users is critical to maintain the user base of a recommender system. Due to the sparse recent interactions, it is challenging to capture these users' current preferences precisely. Solely relying on their historical interactions may also lead to outdated recommendations misaligned with their recent interests. The proposed model leverages historical and current user-item interactions and dynamically factorizes a user's (latent) preference into time-specific and time-evolving representations that jointly affect user behaviors. These latent factors further interact with an optimized item embedding to achieve accurate and timely recommendations. Experiments over real-world data help demonstrate the effectiveness of the proposed time-sensitive cold-start recommendation model.

Image segmentation can be performed interactively by accepting user annotations to refine the segmentation. It seeks frequent feedback from humans, and the model is updated with a smaller batch of data in each iteration of the feedback loop. Such a training paradigm requires effective information filtering to guide the model so that it can encode vital information and avoid overfitting due to limited data and inherent heterogeneity and noises thereof. We propose an adaptive interactive segmentation framework to support user interaction while introducing dual-level information filtering to train a robust model. The framework integrates an encoder-decoder architecture with a style-aware augmentation module that applies augmentation to feature maps and customizes the segmentation prediction for different latent styles. It also applies a systematic label softening strategy to generate uncertainty-aware soft labels for model updates. Experiments on both medical and natural image segmentation tasks demonstrate the effectiveness of the proposed framework.

Temporal Action Localization (TAL) has experienced remarkable success under the supervised learning paradigm. However, existing TAL methods are rooted in the closed set assumption, which cannot handle the inevitable unknown actions in open-world scenarios. In this paper, we, for the first time, step toward the Open Set TAL (OSTAL) problem and propose a general framework Open TAL based on Evidential Deep Learning (EDL). Specifically, the OpenTAL consists of uncertainty-aware action classification, actionness prediction, and temporal location regression. With the proposed importance-balanced EDL method, classification uncertainty is learned by collecting categorical evidence majorly from important samples. To distinguish the unknown actions from background video frames, the actionness is learned by the positive-unlabeled learning. The classification uncertainty is further calibrated by leveraging the guidance from the temporal localization quality. The OpenTAL is general to enable existing TAL models for open set scenarios, and experimental results on THUMOS14 and ActivityNet1.3 benchmarks show the effectiveness of our method. The code and pre-trained models are released at https://www.rit.edu/actionlab/opental.

Optimization-based meta-learning offers a promising direction for few-shot learning that is essential for many real-world computer vision applications. However, learning from few samples introduces uncertainty, and quantifying model confidence for few-shot predictions is essential for many critical domains. Furthermore, few-shot tasks used in meta training are usually sampled randomly from a task distribution for an iterative model update, leading to high labeling costs and computational overhead in meta-training. We propose a novel uncertainty-aware task selection model for label efficient meta-learning. The proposed model formulates a multidimensional belief measure, which can quantify the known uncertainty and lower bound the unknown uncertainty of any given task. Our theoretical result establishes an important relationship between the conflicting belief and the incorrect belief The theoretical result allows us to estimate the total uncertainty of a task, which provides a principled criterion for task selection. A novel multi-query task formulation is further developed to improve both the computational and labeling efficiency of meta-learning. Experiments conducted over multiple real-world few-shot image classification tasks demonstrate the effectiveness of the proposed model.

Multiple-instance learning (MIL) provides an effective way to tackle the video anomaly detection problem by modeling it as a weakly supervised problem as the labels are usually only available at the video level while missing for frames due to expensive labeling cost. We propose to conduct novel Bayesian non-parametric submodular video partition (BN-SVP) to significantly improve MIL model training that can offer a highly reliable solution for robust anomaly detection in practical settings that include outlier segments or multiple types of abnormal events. BN-SVP essentially performs dynamic non-parametric hierarchical clustering with an enhanced self-transition that groups segments in a video into temporally consistent and semantically coherent hidden states that can be naturally interpreted as scenes. Each segment is assumed to be generated through a non-parametric mixture process that allows variations of segments within the same scenes to accommodate the dynamic and noisy nature of many real-world surveillance videos. The scene and mixture component assignment of BN-SVP also induces a pairwise similarity among segments, resulting in non-parametric construction of a submodular set function. Integrating this function with an MIL loss effectively exposes the model to a diverse set of potentially positive instances to improve its training. A greedy algorithm is developed to optimize the submodular function and support efficient model training. Our theoretical analysis ensures a strong performance guarantee of the proposed algorithm. The effectiveness of the proposed approach is demonstrated over multiple real-world anomaly video datasets with robust detection performance.

Service composition algorithms support the construction of complex applications by combining various web services to fulfill diverse functional and Quality of Service (QoS) requirements. Moreover, composition algorithms must fulfill diverse user requirements while adhering to constraints such as limited computational resources. Recent research has demonstrated that using online learning to select different algorithms for specific tasks of a problem domain outperforms approaches that use a single algorithm for all tasks, in terms of computational resource usage and solution quality. Problematically, existing work in service composition does not leverage these advances, leading to multiple inefficient compositions. To address these challenges, we propose online composition algorithm selection using contextual multi-armed bandits to select an algorithm for each composition task at runtime. Our evaluations demonstrate the benefits of our approach by reducing time and memory usage by up to 54.2% and 15.5% while fulfilling QoS requirements, compared to using a single composition algorithm for all tasks.

Graph Convolutional Networks (GCNs) achieve an impressive performance due to the remarkable representation ability in learning the graph information. However, GCNs, when implemented on a deep network, require expensive computation power, making them difficult to be deployed on battery-powered devices. In contrast, Spiking Neural Networks (SNNs), which perform a bio-fidelity inference process, offer an energy-efficient neural architecture. In this work, we propose SpikingGCN, an end-to-end framework that aims to integrate the embedding of GCNs with the biofidelity characteristics of SNNs. The original graph data are encoded into spike trains based on the incorporation of graph convolution. We further model biological information processing by utilizing a fully connected layer combined with neuron nodes. In a wide range of scenarios (e.g., citation networks, image graph classification, and recommender systems), our experimental results show that the proposed method could gain competitive performance against state-of-the-art approaches. Furthermore, we show that SpikingGCN on a neuromorphic chip can bring a clear advantage of energy efficiency into graph data analysis, which demonstrates its great potential to construct environment-friendly machine learning models.

As a widely used weakly supervised learning scheme, modern multiple instance learning (MIL) models achieve competitive performance at the bag level. However, instance-level prediction, which is essential for many important applications, remains largely unsatisfactory. We propose to conduct novel active deep multiple instance learning that samples a small subset of informative instances for annotation, aiming to significantly boost the instance-level prediction. A variance regularized loss function is designed to properly balance the bias and variance of instance-level predictions, aiming to effectively accommodate the highly imbalanced instance distribution in MIL and other fundamental challenges. Instead of directly minimizing the variance regularized loss that is non-convex, we optimize a distributionally robust bag level likelihood as its convex surrogate. The robust bag likelihood provides a good approximation of the variance based MIL loss with a strong theoretical guarantee. It also automatically balances bias and variance, making it effective to identify the potentially positive instances to support active sampling. The robust bag likelihood can be naturally integrated with a deep architecture to support deep model training using mini-batches of positive-negative bag pairs. Finally, a novel P-F sampling function is developed that combines a probability vector and predicted instance scores, obtained by optimizing the robust bag likelihood. By leveraging the key MIL assumption, the sampling function can explore the most challenging bags and effectively detect their positive instances for annotation, which significantly improves the instance-level prediction. Experiments conducted over multiple real-world datasets clearly demonstrate the state-of-the-art instance-level prediction achieved by the proposed model.

App stores enable users to share their experiences directly with the developers in the form of app reviews. Recent studies have shown that the feedback received from users is a valuable source of information for requirements extraction, which encourages app developers to leverage the reviews for app update and maintenance purposes. Follow-up studies proposed automated techniques to help developers filter the large volume of daily and noisy reviews and/or summarize their content. However, all previous studies approached the app reviews classification and summarization as separate tasks, which complicated the process and introduced unnecessary overhead. Moreover, none of those approaches explored the potential of utilizing the hierarchical relationships that exist between the labels of app reviews for the purpose of building a more accurate model. In this work, we propose Hierarchical Multi-Kernel Relevance Vector Machines (HMK-RVM), a Bayesian multi-kernel technique that integrates app review classification and summarization using a unified model. Moreover, it can provide insights into the learned patterns and underlying data for easier model interpretation. We evaluated our proposed approach on two real-world datasets and showed that in addition to the gained insights, the model produces equal or better results than the state of the art.

Summary With the development of Internet technology and the cloud service industry, an increasing number of application programming interfaces (APIs) hosted in the cloud has been made publicly available. To facilitate cloud service APIs vendors and buyers, some large‐scale software cloud directory platforms have been established. Nevertheless, it is difficult for users to choose for renting from a massive number of cloud service APIs with similar functionalities in a software cloud directory platform. Recent efforts in building cloud service APIs recommender systems can help address this challenge. Relevant existing recommendation approaches are designed based on requirement election techniques to identify users' preferences to the quality of service (QoS) of the APIs. In particular, users' preferences are mainly obtained through their self‐description, in which users sometimes cannot accurately and completely express their preferences. In this article, we propose SVD‐APIR, a singular value decomposition (SVD)‐based behavior‐aware cloud service APIs recommendation approach for large‐scale software cloud directory platforms. In SVD‐APIR, users' historical behavior information is captured and APIs' association information is analyzed to identify the users' potential preferences to the APIs with specific QoS. A unified SVD model is utilized to prioritize the users preferred APIs. Experimental evaluation results conducted on WS‐Dream dataset demonstrate the effectiveness and efficiency of the proposed approach.

Deep learning models have achieved state-of-the-art performance in semantic image segmentation, but the results provided by fully automatic algorithms are not always guaranteed satisfactory to users. Interactive segmentation offers a solution by accepting user annotations on selective areas of the images to refine the segmentation results. However, most existing models only focus on correcting the current image's misclassified pixels, with no knowledge carried over to other images. In this work, we formulate interactive image segmentation as a continual learning problem and propose a framework to effectively learn from user annotations, aiming to improve the segmentation on both the current image and unseen images in future tasks while avoiding deteriorated performance on previously-seen images. It employs a probabilistic mask to control the neural network's kernel activation and extract the most suitable features for segmenting images in each task. We also apply a task-aware embedding to automatically infer the optimal kernel activation for initial segmentation and subsequent refinement. Interactions with users are guided through multi-source uncertainty estimation so that users can focus on the most important areas to minimize the overall manual annotation effort. Experiments are performed on both medical and natural image datasets to illustrate the proposed framework's effectiveness on basic segmentation performance, forward knowledge transfer, and backward knowledge transfer.

Multiple Instance Learning (MIL) provides a promising solution to many real-world problems, where labels are only available at the bag level but missing for instances due to a high labeling cost. As a powerful Bayesian non-parametric model, Gaussian Processes (GP) have been extended from classical supervised learning to MIL settings, aiming to identify the most likely positive (or least negative) instance from a positive (or negative) bag using only the bag-level labels. However, solely focusing on a single instance in a bag makes the model less robust to outliers or multi-modal scenarios, where a single bag contains a diverse set of positive instances. We propose a general GP mixture framework that simultaneously considers multiple instances through a latent mixture model. By adding a top-k constraint, the framework is equivalent to choosing the top-k most positive instances, making it more robust to outliers and multimodal scenarios. We further introduce a Distributionally Robust Optimization (DRO) constraint that removes the limitation of specifying a fix k value. To ensure the prediction power over high-dimensional data (e.g., videos and images) that are common in MIL, we augment the GP kernel with fixed basis functions by using a deep neural network to learn adaptive basis functions so that the covariance structure of high-dimensional data can be accurately captured. Experiments are conducted on highly challenging real-world video anomaly detection tasks to demonstrate the effectiveness of the proposed model.

We present a maximum-margin sparse Gaussian Process (MM-SGP) for active learning (AL) of classification models for multi-class problems. The proposed model makes novel extensions to a GP by integrating maximum-margin constraints into its learning process, aiming to further improve its predictive power while keeping its inherent capability for uncertainty quantification. The MM constraints ensure small "effective size" of the model, which allows MM-SGP to provide good predictive performance by using limited "active" data samples, a critical property for AL. Furthermore, as a Gaussian process model, MM-SGP will output both the predicted class distribution and the predictive variance, both of which are essential for defining a sampling function effective to improve the decision boundaries of a large number of classes simultaneously. Finally, the sparse nature of MM-SGP ensures that it can be efficiently trained by solving a low-rank convex dual problem. Experiment results on both synthetic and real-world datasets show the effectiveness and efficiency of the proposed AL model.

We propose a novel framework to classify large-scale time series data with long duration. Long time series classification (L-TSC) is a challenging problem because the data often contains a large amount of irrelevant information to the classification target. The irrelevant period degrades the classification performance while the relevance is unknown to the system. This paper proposes an uncertainty-aware multiple instance learning (MIL) framework to identify the most relevant period automatically. The predictive uncertainty enables designing an attention mechanism that forces the MIL model to learn from the possibly discriminant period. Moreover, the predicted uncertainty yields a principled estimator to identify whether a prediction is trustworthy or not. We further incorporate another modality to accommodate unreliable predictions by training a separate model based on its availability and conduct uncertainty aware fusion to produce the final prediction. Systematic evaluation is conducted on the Automatic Identification System (AIS) data, which is collected to identify and track real-world vessels. Empirical results demonstrate that the proposed method can effectively detect the types of vessels based on the trajectory and the uncertainty-aware fusion with other available data modality (Synthetic-Aperture Radar or SAR imagery is used in our experiments) can further improve the detection accuracy.

Traffic accident anticipation aims to accurately and promptly predict the occurrence of a future accident from dashcam videos, which is vital for a safety-guaranteed self-driving system. To encourage an early and accurate decision, existing approaches typically focus on capturing the cues of spatial and temporal context before a future accident occurs. However, their decision-making lacks visual explanation and ignores the dynamic interaction with the environment. In this paper, we propose Deep ReInforced accident anticipation with Visual Explanation, named DRIVE. The method simulates both the bottom-up and top-down visual attention mechanism in a dashcam observation environment so that the decision from the pro-posed stochastic multi-task agent can be visually explained by attentive regions. Moreover, the proposed dense anticipation reward and sparse fixation reward are effective in training the DRIVE model with our improved reinforcement learning algorithm. Experimental results show that the DRIVE model achieves state-of-the-art performance on multiple real-world traffic accident datasets. Code and pre-trained model are available at https://www.rit.edu/actionlab/drive.

In a real-world scenario, human actions are typically out of the distribution from training data, which requires a model to both recognize the known actions and reject the unknown. Different from image data, video actions are more challenging to be recognized in an open-set setting due to the uncertain temporal dynamics and static bias of human actions. In this paper, we propose a Deep Evidential Action Recognition (DEAR) method to recognize actions in an open testing set. Specifically, we formulate the action recognition problem from the evidential deep learning (EDL) perspective and propose a novel model calibration method to regularize the EDL training. Besides, to mitigate the static bias of video representation, we propose a plug-and-play module to debias the learned representation through contrastive learning. Experimental results show that our DEAR method achieves consistent performance gain on multiple mainstream action recognition models and benchmarks. Code and pre-trained models are available at https://www.rit.edu/actionlab/dear.

Recommender systems have been widely used to predict users’ interests and filter information from a large number of candidate items. However, accurately capturing the interests of users having limited interactions with a system remains a long-lasting challenge. Furthermore, existing recommender systems primarily focus on predicting user preferences without quantifying the prediction uncertainty. Uncertainty can help to quantify the model confidence when making a recommendation where low model confidence could serve as a more accurate indicator of a user’s cold-start level than simply using the number of interactions. We present a novel recommendation model that seamlessly integrates a meta-learning module with an evidential learning approach. The former module generalizes meta knowledge to tackle cold-start recommendations by exploiting fast adaptation. The latter quantifies both aleatoric and epistemic uncertainty without performing expensive posterior inference. Evidential learning achieves this by placing evidential priors and treating the output of the meta-learning module as evidence-based pseudo counts and learns a function to directly predict the evidence of a target interaction. Experiments on four benchmark datasets justify that our proposed model captures the uncertainty of users and demonstrates its superior performance over the state-of-the-art recommendation models.

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a specific image from a large gallery given a query sketch. Despite the widespread applicability of FG-SBIR in many critical domains (e.g., crime activity tracking), existing approaches still suffer from a low accuracy while being sensitive to external noises such as unnecessary strokes in the sketch. The retrieval performance will further deteriorate under a more practical on-the-fly setting, where only a partially complete sketch with only a few (noisy) strokes are available to retrieve corresponding images. We propose a novel framework that leverages a uniquely designed deep reinforcement learning model that performs a dual-level exploration to deal with partial sketch training and attention region selection. By enforcing the model’s attention on the important regions of the original sketches, it remains robust to unnecessary stroke noises and improve the retrieval accuracy by a large margin. To sufficiently explore partial sketches and locate the important regions to attend, the model performs bootstrapped policy gradient for global exploration while adjusting a standard deviation term that governs a locator network for local exploration. The training process is guided by a hybrid loss that integrates a reinforcement loss and a supervised loss. A dynamic ranking reward is developed to fit the on-the-fly image retrieval process using partial sketches. The extensive experimentation performed on three public datasets shows that our proposed approach achieves the state-of-the-art performance on partial sketch based image retrieval.

In service composition, complex applications are built by combining web services to fulfill user Quality of Service (QoS) and business requirements. To meet these requirements, applications are composed by evaluating all possible web service combinations using search algorithms. These algorithms need to be accurate and inexpensive to evaluate a large number of possible service combinations and services' fluctuating QoS attributes while meeting the constraints of limited computational resources. Recent research has shown that different search algorithms can outperform others on specific instances of a problem domain, in terms of solution quality and computational resource usage. Problematically, current service composition approaches ignore this property, leading to inefficient compositions. To address these limitations, we propose a composition algorithm selection framework which selects an algorithm per composition task at runtime, R-CASS. Our evaluations demonstrate that R-CASS leads to more efficient compositions, reducing composition time by 55.1% and memory by 37.5%.

Multi-label classification (MLC) allows complex dependencies among labels, making it more suitable to model many real-world problems. However, data annotation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and a potential large and sparse label space. We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesian Bernoulli Mixture model (GP-B$^2$M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B$^2$M encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds to a global pattern of label correlations. To tackle highly sparse labels under AL, the B$^2$M is further integrated with a predictive GP to connect data features as an effective inductive bias and achieve a feature-component-label mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference. The model also outputs a predictive distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly to naturally capture both the feature uncertainty (through GP) and label covariance (through B$^2$M) for effective data sampling. Experiments on real-world multi-label datasets demonstrate the state-of-the-art AL performance of the proposed GP-B$^2$M model.

Monocular 3D object detection aims to detect objects in a 3D physical world from a single camera. However, recent approaches either rely on expensive LiDAR devices, or resort to dense pixel-wise depth estimation that causes prohibitive computational cost. In this paper, we propose an end-to-end trainable monocular 3D object detector without learning the dense depth. Specifically, the grid coordinates of a 2D box are first projected back to 3D space with the pinhole model as 3D centroids proposals. Then, a novel object-aware voting approach is introduced, which considers both the region-wise appearance attention and the geometric projection distribution, to vote the 3D centroid proposals for 3D object localization. With the late fusion and the predicted 3D orientation and dimension, the 3D bounding boxes of objects can be detected from a single RGB image. The method is straightforward yet significantly superior to other monocular-based methods. Extensive experimental results on the challenging KITTI benchmark validate the effectiveness of the proposed method.

Traffic accident anticipation aims to predict accidents from dashcam videos as early as possible, which is critical to safety-guaranteed self-driving systems. With cluttered traffic scenes and limited visual cues, it is of great challenge to predict how long there will be an accident from early observed frames. Most existing approaches are developed to learn features of accident-relevant agents for accident anticipation, while ignoring the features of their spatial and temporal relations. Besides, current deterministic deep neural networks could be overconfident in false predictions, leading to high risk of traffic accidents caused by self-driving systems. In this paper, we propose an uncertainty-based accident anticipation model with spatio-temporal relational learning. It sequentially predicts the probability of traffic accident occurrence with dashcam videos. Specifically, we propose to take advantage of graph convolution and recurrent networks for relational feature learning, and leverage Bayesian neural networks to address the intrinsic variability of latent relational representations. The derived uncertainty-based ranking loss is found to significantly boost model performance by improving the quality of relational features. In addition, we collect a new Car Crash Dataset (CCD) for traffic accident anticipation which contains environmental attributes and accident reasons annotations. Experimental results on both public and the newly-compiled datasets show state-of-the-art performance of our model. Our code and CCD dataset are available at https://github.com/Cogito2012/UString.

We present a novel multi-source uncertainty prediction approach that enables deep learning (DL) models to be actively trained with much less labeled data. By leveraging the second-order uncertainty representation provided by subjective logic (SL), we conduct evidence-based theoretical analysis and formally decompose the predicted entropy over multiple classes into two distinct sources of uncertainty: vacuity and dissonance, caused by lack of evidence and conflict of strong evidence, respectively. The evidence based entropy decomposition provides deeper insights on the nature of uncertainty, which can help effectively explore a large and high-dimensional unlabeled data space. We develop a novel loss function that augments DL based evidence prediction with uncertainty anchor sample identification. The accurately estimated multiple sources of uncertainty are systematically integrated and dynamically balanced using a data sampling function for label-efficient active deep learning (ADL). Experiments conducted over both synthetic and real data and comparison with competitive AL methods demonstrate the effectiveness of the proposed ADL model.

We propose to jointly analyze experts' eye movements and verbal narrations to discover important and interpretable knowledge patterns to better understand their decision-making processes. The discovered patterns can further enhance data-driven statistical models by fusing experts' domain knowledge to support complex human-machine collaborative decision-making. Our key contribution is a novel dynamic Bayesian nonparametric model that assigns latent knowledge patterns into key phases involved in complex decision-making. Each phase is characterized by a unique distribution of word topics discovered from verbal narrations and their dynamic interactions with eye movement patterns, indicating experts' special perceptual behavior within a given decision-making stage. A new split-merge-switch sampler is developed to efficiently explore the posterior state space with an improved mixing rate. Case studies on diagnostic error prediction and disease morphology categorization help demonstrate the effectiveness of the proposed model and discovered knowledge patterns.

Background: As smart and automated applications pervade our lives, an increasing number of software developers are required to incorporate machine learning (ML) techniques into application development. However, acquiring the ML skill set can be nontrivial for software developers owing to both the breadth and depth of the ML domain. Aims: We seek to understand the challenges developers face in the process of ML application development and offer insights to simplify the process. Despite its importance, there has been little research on this topic. A few existing studies on development challenges with ML are outdated, small scale, or they do no involve a representative set of developers. Method: We conduct an empirical study of ML-related developer posts on Stack Overflow. We perform in-depth quantitative and qualitative analyses focusing on a series of research questions related to the challenges of developing ML applications and the directions to address them. Results: Our findings include: (1) ML questions suffer from a much higher percentage of unanswered questions on Stack Overflow than other domains; (2) there is a lack of ML experts in the Stack Overflow QA community; (3) the data preprocessing and model deployment phases are where most of the challenges lay; and (4) addressing most of these challenges require more ML implementation knowledge than ML conceptual knowledge. Conclusions: Our findings suggest that most challenges are under the data preparation and model deployment phases, i.e., early and late stages. Also, the implementation aspect of ML shows much higher difficulty level among developers than the conceptual aspect.

Active learning for multi-label classification poses fundamental challenges given the complex label correlations and a potentially large and sparse label space. We propose a novel CS-BPCA process that integrates compressed sensing and Bayesian principal component analysis to perform a two-level label transformation, resulting in an optimally compressed continuous target space. Besides leveraging correlation and sparsity of a large label space for effective compression, an optimal compressing rate and the relative importance of the resultant targets are automatically determined through Bayesian inference. Furthermore, the orthogonality of the transformed space completely decouples the correlations among targets, which significantly simplifies multi-label sampling in the target space. We define a novel sampling function that leverages a multi-output Gaussian Process (MOGP). Gradient-free optimization strategies are developed to achieve fast online hyper-parameter learning and model retraining for active learning. Experimental results over multiple real-world datasets and comparison with competitive multi-label active learning models demonstrate the effectiveness of the proposed framework.

When self-adaptive systems encounter changes withintheir surrounding environments, they enacttacticsto performnecessary adaptations. For example, a self-adaptive cloud-basedsystem may have a tactic that initiates additional computingresources when response time thresholds are surpassed, or theremay be a tactic to activate a specific security measure when anintrusion is detected. In real-world environments, these tacticsfrequently experiencetactic volatilitywhich is variable behaviorduring the execution of the tactic.Unfortunately, current self-adaptive approaches do not accountfor tactic volatility in their decision-making processes, and merelyassume that tactics do not experience volatility. This limitationcreates uncertainty in the decision-making process and mayadversely impact the system's ability to effectively and efficientlyadapt. Additionally, many processes do not properly account forvolatility that may effect the system's Service Level Agreement(SLA). This can limit the system's ability to act proactively, especially when utilizing tactics that contain latency.To address the challenge of sufficiently accounting for tacticvolatility, we propose aTactic Volatility Aware(TVA) solution.Using Multiple Regression Analysis (MRA), TVA enables self-adaptive systems to accurately estimate the cost and timerequired to execute tactics. TVA also utilizesAutoregressiveIntegrated Moving Average(ARIMA) for time series forecasting, allowing the system to proactively maintain specifications.

We propose a novel active learning (AL) model that integrates Bayesian and discriminative kernel machines for fast and accurate multi-class data sampling. By joining a sparse Bayesian model and a maximum margin machine under a unified kernel machine committee (KMC), the proposed model is able to identify a small number of data samples that best represent the overall data space while accurately capturing the decision boundaries. The integration is conducted using the maximum entropy discrimination framework, resulting in a joint objective function that contains generalized entropy as a regularizer. Such a property allows the proposed AL model to choose data samples that more effectively handle non-separable classification problems. Parameter learning is achieved through a principled optimization framework that leverages convex duality and sparse structure of KMC to efficiently optimize the joint objective function. Key model parameters are used to design a novel sampling function to choose data samples that can simultaneously improve multiple decision boundaries, making it an effective sampler for problems with a large number of classes. Experiments conducted over both synthetic and real data and comparison with competitive AL methods demonstrate the effectiveness of the proposed model.

We focus on using natural language unstructured textual Knowledge Bases (KBs) to answer questions from community-based Question-and-Answer (Q8A) websites. We propose a novel framework that integrates multi-level tag recommendation with external KBs to retrieve the most relevant KB articles to answer user posted questions. Different from many existing efforts that primarily rely on the Q8A sites’ own historical data (e.g., user answers), retrieving answers from authoritative external KBs (e.g., online programming documentation repositories) has the potential to provide rich information to help users better understand the problem, acquire the knowledge, and hence avoid asking similar questions in future. The proposed multi-level tag recommendation best leverages the rich tag information by first categorizing them into different semantic levels based on their usage frequencies. A post-tag co-clustering model, augmented by a two-step tag recommender, is used to predict tags at different levels for a given user posted question. A KB article retrieval component leverages the recommended multi-level tags to select the appropriate KBs and search/rank the matching articles thereof. We conduct extensive experiments using real-world data from a Q8A site and multiple external KBs to demonstrate the effectiveness of the proposed question-answering framework.

Service computing is an emerging technology in System of Systems Engineering (SoS Engineering or SoSE), which regards a System as a Service, and aims at constructing a robust and value-added complex system by outsourcing external component systems through service composition. The burgeoning Big Service computing just covers the significant challenges in constructing and maintaining a stable service-oriented SoS. A service-oriented SoS runs under a volatile and uncertain environment. As a step toward big service, service fault tolerance (FT) can guarantee the run-time quality of a service-oriented SoS. To successfully deploy FT in an SoS, online reliability time series prediction, which aims at predicting the reliability in near future for a service-oriented SoS arises as a grand challenge in SoS research. In particular, we need to tackle a number of big data related issues given the large and fast increasing size of the historical data that will be used for prediction purpose. The decision-making of prediction solution space be more complex. To provide highly accurate prediction results, we tackle the prediction challenges by identifying the evolution regularities of component systems' running states via different machine learning models. We present in this paper the motifs-based Dynamic Bayesian Networks (or m_DBNs) to perform one-step-ahead online reliability time series prediction. We also propose a multi-steps trajectory DBNs (or multi_DBNs) to further improve the accuracy of future reliability prediction. Finally, a Convolutional Neural Networks (CNN)-based prediction approach is developed to deal with the big data challenges. Extensive experiments conducted on real-world Web services demonstrate that our models outperform other well-known approaches consistently.

The high cost for labeling data instances is a key bottleneck for training effective supervised learning models. This is especially the case in domains such as medicine and bioinformatics, where expert knowledge is required for understanding and extracting the underlying semantics of data. Active learning provides a means to reduce human labeling efforts by identifying the most informative data instances. In this paper, we propose a cost-effective active learning framework to further lessen human efforts, especially in knowledge-rich domains where a large number of classes may be subject to scrutiny during decision making. In particular, this framework employs a novel many-class sampling model, MC-S, for data sample selection. MC-S is further augmented with convex hull-based sampling to achieve faster convergence of active learning. Evaluation studies conducted over multiple real-world datasets with many classes demonstrate that the proposed framework significantly reduces the overall labeling efforts through fast convergence and early stop of active learning.

As the number of Cloud services is growing at a tremendous speed, there is an increasing number of service providers offering similar functionalities. Selecting services with user desired non-functional properties (NFPs) becomes of significant importance but triggers a number of Big Data related research issues. First, the selection decision should deal with a large volume of service NFPs data. Second, service selection needs to reflect diverse user preferences, including both qualitative and quantitative ones. Third, the uncertainty of the network and service load leads to high variability in NFPs. Fourth, as the trust values of service NFPs are collected via historic user's feedbacks,it brings the veracity dimension to the NFPs of services. Fifth, multiple and sometimes conflicting decision objectives for optimal service selection should be balanced. An effective service selection mechanism is in demand that can tackle all the above Big Data challenges in an integrated way to handle the highly diverse QoS with significant variability along with the trust related issues giving rise to data veracity. Existing investigations focus on either users' QoS preferences or their trust concerns but fail to provide a systematic solution to integrate both criteria in the selection process. In this paper, we tackle heterogeneous preference- and trust-based service selection by developing a novel multi-objective optimization approach to make trade-off decision between service's trust value and user's QoS preference to rank candidate Cloud services based on their match degrees with users' requirements. We conduct extensive experiments to evaluate the effectiveness and efficiency of the proposed approach.

Many enterprises have a growing interest in service composition to construct their business applications. With the increase of alternative services, Quality of Service (QoS) becomes an important indicator of obtaining optimal composite services. Due to the dynamic nature of the service environment, a composite service may not guarantee to deliver an overall optimal QoS. Re-optimization approaches have been developed to handle a dynamic environment. However, these approaches do not consider the diversity of alternative solutions, which may lead to better solutions. In this work, we introduce an adaptive approach, called estimation of distribution algorithm based on Restricted Boltzmann Machine (rEDA). rEDA effectively maintains the diversity of alternative solutions, by leveraging the inference ability of Restricted Boltzmann Machine to capture the potential solutions. It also provides a predictive guidance for the exploration of solution space, by considering the degree of how well a service contributes to the global QoS. The experimental evaluation shows that rEDA has a significant improvement on effectiveness and efficiency over existing approaches.

Tag recommendation has gained significant popularity for annotating various web-based resources including web services. Compared with other approaches, tag recommendation based on supervised learning models usually lead to good accuracy. However, a high-quality training data set is needed, which demands manual tagging efforts from domain experts. While we could leverage the tags of existing web services assigned by their developers, the quality of these tags may not be good enough to build accurate classifiers for tag recommendation. In this paper, a novel multi-label active learning approach is proposed for web service tag recommendation. The proposed approach is able to identify a small number of most informative web services to be tagged by domain experts. We further minimize the domain expert efforts by learning and leveraging the correlations among tags to improve the active learning process. We conduct a comprehensive experimental study on a real-world data set and results demonstrate the effectiveness of our approach.

A service-oriented System of System (SoS) integrates component services into a value-added and more complex system to satisfy the complex requirements of users. Due to a dynamic running environment, online reliability prediction for the loosely coupled component systems that ensures the runtime quality poses a major challenge and attracts growing attention. To guarantee the stable and continuous operation of systems, we propose a online reliability time series prediction method basing on long short term memory (LSTM), which is a modified Recurrent Neural Networks trained with historical reliability time series to predict the reliability of component systems in the near future. We conduct a series of experiments on a dataset composed of real web services and compare with other competitive approaches. Experimental results have demonstrated the effectiveness of our approach.

Diagnostic error prevention is a long-established but specialized topic in clinical and psychological research. In this paper, we contribute to the field by exploring diagnostic decision-making via modeling physicians' utterances of medical concepts during image-based diagnoses. We conduct experiments to collect verbal narratives from dermatologists while they are examining and describing dermatology images towards diagnoses. We propose a hierarchical probabilistic framework to learn domain-specific patterns from the medical concepts in these narratives. The discovered patterns match the diagnostic units of thought identified by domain experts. These meaningful patterns uncover physicians' diagnostic decision-making processes while parsing the image content. Our evaluation shows that these patterns provide key information to classify narratives by diagnostic correctness levels.

Service-oriented architecture is a widely used software engineering paradigm to cope with complexity and dynamics in enterprise applications. Service composition, which provides a cost-effective way to implement software systems, has attracted significant attention from both industry and research communities. As online services may keep evolving over time and thus lead to a highly dynamic environment, service composition must be self-adaptive to tackle uninformed behavior during the evolution of services. In addition, service composition should also maintain high efficiency for large-scale services, which are common for enterprise applications. This article presents a new model for large-scale adaptive service composition based on multi-agent reinforcement learning. The model integrates reinforcement learning and game theory, where the former is to achieve adaptation in a highly dynamic environment and the latter is to enable agents to work for a common task (i.e., composition). In particular, we propose a multi-agent Q-learning algorithm for service composition, which is expected to achieve better performance when compared with the single-agent Q-learning method and multi-agent SARSA (State-Action-Reward-State-Action) method. Our experimental results demonstrate the effectiveness and efficiency of our approach.

With the fast increase of online services of all kinds, users start to care more about the Quality of Service (QoS) that a service provider can offer besides the functionalities of the services. As a result, QoS-based service selection and recommendation have received significant attention since the mid-2000s. However, existing approaches primarily consider a small number of standard QoS parameters, most of which relate to the response time, fee, availability of services, and so on. As online services start to diversify significantly over different domains, these small set of QoS parameters will not be able to capture the different quality aspects that users truly care about over different domains. Most existing approaches for QoS data collection depend on the information from service providers, which are sensitive to the trustworthiness of the providers. Some service monitoring mechanisms collect QoS data through actual service invocations but may be affected by actual hardware/software configurations. In either case, domain-specific QoS data that capture what users truly care about have not been successfully collected or analyzed by existing works in service computing. To address this demanding issue, we develop a statistical learning approach to extract domain-specific QoS features from user-provided service reviews. In particular, we aim to classify user reviews based on their sentiment orientations into either a positive or negative category. Meanwhile, statistical feature selection is performed to identify statistically nontrivial terms from review text, which can serve as candidate QoS features. We also develop a topic models-based approach that automatically groups relevant terms and returns the term groups to users, where each term group corresponds to one high-level quality aspect of services. We have conducted extensive experiments on three real-world datasets to demonstrates the effectiveness of our approach.

A service-oriented System of Systems (SoS) considers a system as a service and constructs a robust and value-added SoS by outsourcing external component systems through service composition techniques. Online reliability prediction for the component systems for the purpose of assuring the overall Quality of Service (QoS) is often a major challenge in coping with a loosely coupled SoS operating under dynamic and uncertain running environments. It is also a prerequisite for guaranteeing runtime QoS of a SoS through optimal service selection for reliable system construction. We propose a novel online reliability time series prediction approach for the component systems in a service-oriented SoS. We utilize Probabilistic Graphical Models (PGMs) to yield near-future, time series predictions. We assess the approach via invocation records collected from widely used real Web services. Experimental results have confirmed the effectiveness of the approach.

Classifying Web services and labeling them based on their functional features have played a major role in several fundamental service management tasks, such as service discovery, selection, ranking, and recommendation. Existing approaches leverage text mining techniques and follow a supervised learning process, which involves building a classifier from a training set of services and applying the classifier to other services. This process requires intensive human effort on labeling services in the training set. In this paper, we propose to leverage the idea of pool-based active learning to realize a scalable service classification approach. Instead of manually labeling a large number of services to construct a complete training set, the approach starts with a base classifier with a small set of training set and iteratively asks for the labels of the most informative services outside of the initial training set. By doing this, the classifier can achieve comparable accuracy compared to traditional classification method with much smaller size of training set. We use SVM as the base classifier due to its effectiveness in text classification. We also incorporate probabilistic topic models to address the issues caused by sparse term vectors generated from service descriptions and reduce the dimensions to improve the efficiency. We conducted a comprehensive experimental study on real-world service data to demonstrate the effectiveness of the proposed approach.

Developing efficient solutions to achieve automatic service composition has drawn significant attentions in service computing. As a composite service typically runs in a dynamic environment, adaptability of the composition solution arises as a central concern. Reinforcement Learning (RL) is one commonly used approach in service composition to achieve self-adaptability. However, traditional RL methods cannot guarantee good efficiency for large-scale composition problems. Hierarchical RL (HRL) has appeared to be a viable solution to address the efficiency issue. The applicability of existing HRL methods (e.g., MAXQ) requires a task graph, which can be generated by decomposing a composition plan into a task hierarchy. Current approaches that apply HRL to service composition generate the task graph manually, which will not scale to large service composition problems. In this paper, we address the above issue by systematically integrating automatic task decomposition and MAXQ HRL, resulting in an adaptive composition solution with good efficiency. Our experimental results demonstrate the effectiveness of the proposed service composition approach.

The papers in this paper are devoted to the topic of Big Data analytics via the Internet.

With the development of Service Computing and Big Data research, more and more heterogeneous data generated in the process of Service Computing attracts our attention. Combining correlated data sources may help improve the performance of a given task. For example, in service recommendation, one can combine (1) user profile data (e.g. Genders, age, etc.), (2) user log data (e.g., Click through data, service invocation records, etc.), (3) QoS data (e.g. Response time, cost, etc.), (4) service functional description (e.g., Service name, WSDL document, etc.) and (5) service tagging data (i.e., Tags annotated by users) to build a recommendation model. All these data sources provide informative but heterogeneous features. For instance, user profile and QoS data usually have nominal features reflecting users' background and services' qualities, log data provides term-based features about users' historical behaviors, and service functional description and tagging data have term-based features reflecting services' functionalities and users' collective opinions. Given multiple heterogeneous data sources, one important challenge is to find a unified feature subspace to capture the knowledge from all data sources. To handle this problem, in this paper, we propose a Heterogeneous Feature Selection framework, named as WS-HFS, in which the consensus and the weight of different sources are both considered. Moreover, we apply the proposed framework to Web service clustering as a case study, and compare it with the state of the art approaches. The comprehensive experiments based on real data demonstrate the effectiveness of WS-HFS.

Quality of Service (QoS) has become a standard way of evaluating web services and selecting the one that suites user interests the best. Traditional methods adopt a fixed set of QoS parameters and typical ones include response time, fee, and availability. There currently lacks an effective way of identifying quality features that users are actually interested in when choosing a service. Meanwhile, the traditional way of collecting QoS values relies on either public information released by service providers or test results from repeatedly invoking a service. Therefore, the values can be heavily affected by authenticity of the provider offered information or the quality/configuration of the test code/environment. As a result, existing QoS evaluation methods are not applicable to subject features, such as usability and affordability, where the values depend on user personal judgement. In this paper, we propose a novel approach to extracting domain-related QoS features, ranking those features based on their interestingness, evaluating the value of these features through sentiment analysis on user reviews. More specifically, we leverage natural language processing techniques and machine learning approaches to identify top QoS features that users are interested in and simultaneously learn their sentiment orientation towards those features. We model the problem as sentiment classification, where relevant terms in a review are modeled as features that determine whether a review is positive or negative. Logistic regression is used so that the impact of these terms are learned simultaneously when the classifier is learned through a supervised learning process. The nontrivial terms are selected as the candidate QoS featured. A comprehensive experiment has been conducted on a real-world dataset and the result demonstrates the effectiveness of our approach.

Application Programming Interfaces (APIs), which are emerging web services in general, are increasing with a rapid speed in recent years. With so many APIs, many management platforms have been developed and deployed, leading to the boom of API markets, that are similar to the mobile App markets. Meanwhile, it has become more and more difficult to select and manage APIs. In reality, most existing management platforms typically recommend currently popular APIs to developers. However, the fact that popularity of API varies over time is ignored in those platforms, leading to the difficulty of recommending APIs that are just released but may be popular in the near future. To tackle this challenge, an approach of predicting the popularity of APIs is proposed in this paper. Predicting the popularity of API can not only be used for API ranking, recommendation and selection, but also make it more convenient for API providers and consumers to manage or select API respectively. In this paper, we propose a time-aware linear model to predict the API popularity, using time series feature of APIs and API's self-features such as its' provider ranking and description features, which are called heterogeneous features in our paper. Comprehensive experiments have been conducted on a real-world Programmable Web dataset with 613 real APIs. The experimental results show that our model has a better performance, when compared with some other state-of-the-art prediction models.

As a result of recent trends in enhancing Service-Oriented Requirement Engineering (SORE) activities, a number of requirement specification methods have been proposed for fitting the reuse infrastructure in a Service-Oriented Architecture (SOA). The availability of different Requirement Engineering methods offers developers a range of options to choose from. However, most of existing research effort uses traditional Requirement Engineering methods in service-based application developments. During requirements specification, a reusable infrastructure of available web services is not considered at all. The risk is that atomic requirements do not always fit reusable services. As a result, the service composition is time-consuming and needs costly adaption. This paper therefore proposes a novel method by introducing service discovery in the early Requirement Engineering stages so as to guide the requirement decomposition process. Although several researchers have already recommended to involve service discovery in SORE, they do not focus on how to guide requirement decomposition. Our approach is implemented on top of the widely used goal-oriented approach. To this end, we leverage a semantic service discovery method as a means to act as a guide and sentinel in requirement elaboration. We demonstrate the requirement decomposition process by implementing a case study from the Business Traveling domain.

The past decade has witnessed a fast growth of web-based services, making discovery of user desired services from a large and diverse service space a fundamental challenge. Service clustering has been demonstrated as a promising solution by automatically detecting functionally similar services so that they can be searched and discovered together. In this way, both the efficiency and accuracy of service discovery can be improved. However, the autonomous nature of service providers leads to highly diverse usage of terms in their respective service descriptions. Furthermore, a typical service description is comprised of very limited terms due to the small number of (and focused) functionalities offered by the service. These unique characteristics make service descriptions different from regular text documents, which poses additional challenges when clustering large-scale services. Recent works show that service clustering can benefit from discovery and use of functionality-related latent factors to represent services as opposed to a large and diverse set of terms. Nonetheless, how to determine the total number of latent functional factors and sparsely assign them to each service description arises as a central challenge, especially for a large service space where there is no easy way to enumerate the types of different functionalities. In this paper, we propose a machine learning method that automatically learns the number of latent functional factors in a service space. It also enforces the sparsity constraint, which allows each service to be represented by a small number of latent functional factors. The sparsity constraint is in line with the fact that most real-world services only provide limited functionalities. We conduct extensive experiments on two sets of real-world service data to demonstrate the effectiveness of the proposed service clustering approach.

The articles in this special section aim at presenting the latest developments, trends, and solutions of Big Data analytics on the web.

Traditional content-based image retrieval techniques, which primarily rely on image content at the pixel level, are not effective in accessing images at the semantic level. Defining approaches to incorporate experts' perceptual and conceptual capabilities of image understanding in their domain of expertise into the retrieval processes promises to help bridge this semantic gap. Towards accomplishing this, we design and implement a novel multimodal interactive system for image retrieval. To incorporate human expertise, the system stores expert-derived information extracted from two human sensor modalities that intuitively relate to image search, eye movements and verbal descriptions, both generated by medical experts. Experimental evaluation of the system shows that by transferring experts' perceptual expertise and domain knowledge into image-based computational procedures, our system can take advantage of the different human-centered modalities' respective strengths and improve the retrieval performance over just using image-based features.

In the era of big data, data intensive applications have posed new challenges to the filed of service composition, i.e. composition efficiency and scalability. How to compose massive and evolving services in such dynamic scenarios is a vital problem demanding prompt solutions. As a consequence, we propose a new model for large-scale adaptive service composition in this paper. This model integrates the knowledge of reinforcement learning aiming at the problem of adaptability in a highly-dynamic environment and game theory used to coordinate agents' behavior for a common task. In particular, a multi-agent Q-learning algorithm for service composition based on this model is also proposed. The experimental results demonstrate the effectiveness and efficiency of our approach, and show a better performance compared with the single-agent Q-learning method.

Service composition is an emerging technology in System of Systems Engineering (SoS Engineering or SoSE), which aims to construct a robust and value-added complex system by outsourcing external component systems. A serviceoriented SoS runs under a dynamic and uncertain environment. To assure the overall Quality of Service (QoS), online reliability time series prediction, which aims to predict the reliability in near future for service-oriented SoS arises as a grand challenge in SoS research. In this paper, we tackle the prediction challenge by exploiting two Markov independence assumptions resulted from the special system dynamics of a SoS environment. A novel motifs-based Dynamic Bayesian Networks model is proposed that supports the independence assumptions. Experimental results conducted on real Web services demonstrate the effectiveness of our approach.

Web services are integrated software components for the support of interoperable machine-to-machine interaction over a network. Web services have been widely employed for building service-oriented applications in both industry and academia in recent years. The number of publicly available Web services is steadily increasing on the Internet. However, this proliferation makes it hard for a user to select a proper Web service among a large amount of service candidates. An inappropriate service selection may cause many problems (e.g., ill-suited performance) to the resulting applications. In this paper, we propose a novel collaborative filtering-based Web service recommender system to help users select services with optimal Quality-of-Service (QoS) performance. Our recommender system employs the location information and QoS values to cluster users and services, and makes personalized service recommendation for users based on the clustering results. Compared with existing service recommendation methods, our approach achieves considerable improvement on the recommendation accuracy. Comprehensive experiments are conducted involving more than 1.5 million QoS records of real-world Web services to demonstrate the effectiveness of our approach.

Viral marketing, which uses the "word of mouth" marketing technique over virtual networks, relies on the selection of a small subset of most influential users in the network for efficient marketing. Nonetheless, most existing viral marketing techniques ignore the dynamic nature of the virtual network. In this paper, we develop a novel framework that exploits the temporal dynamics of the network to select an optimal subset of users that maximize the marketing influence over the network.

We present in this paper a novel QoS prediction approach to tackle service recommendation, which is to recommend services with the best QoS to users. QoS prediction exploits available QoS information to estimate users' QoS experience from previously unknown services. In this regard, it can be modeled as a general matrix completion problem, which is to recover a large QoS matrix from a small subset of QoS entires. The infinite number of ways to complete an arbitrary QoS matrix makes the problem extremely ill posed. The highly sparse QoS data further complicates the challenges. Nonetheless, real-world QoS data exhibits two key features, which can be leveraged for accurate QoS predictions, leading to high-quality service recommendations. First, QoS delivery can be significantly affected by a small number dominant factors in the service environment (e.g., communication link and user-service distance). Hence, it is natural to assume that the QoS matrix has a low-rank or approximately low-rank structure. Second, users (or services) that share common environmental factors are expected to receive (or deliver) similar QoS and hence can be grouped together. The proposed approach seamlessly amalgamates these two features into a unified objective function and employs an effective iterative algorithm to approach the optimal completion of an arbitrary QoS matrix. We conduct a set of experiments on real-world QoS data to demonstrate the effectiveness of the proposed algorithm.

Clustering techniques offer a systematic approach to organize the diverse and fast increasing Web services by assigning relevant services into homogeneous service communities. However, the ever increasing number of Web services poses key challenges for building large-scale service communities. In this paper, we tackle the scalability issue in service clustering, aiming to accurately and efficiently discover service communities over very large-scale services. A key observation is that service descriptions are usually represented by long but very sparse term vectors as each service is only described by a limited number of terms. This inspires us to seek a new service representation that is economical to store, efficient to process, and intuitive to interpret. This new representation enables service clustering to scale to massive number of services. More specifically, a set of anchor services are identified that allows each service to represent as a linear combination of a small number of anchor services. In this way, the large number of services are encoded with a much more compact anchor service space. Despite service clustering can be performed much more efficiently in the compact anchor service space, discovery of anchor services from large-scale service descriptions may incur high computational cost. We develop principled optimization strategies for efficient anchor service discovery. Extensive experiments are conducted on real-world service data to assess both the effectiveness and efficiency of the proposed approach. Results on a dataset with over 3,700 Web services clearly demonstrate the good scalability of sparse functional representation and the efficiency of the optimization algorithms for anchor service discovery.

Service composition is emerging as an effective vehicle for integrating existing web services to create value-added and personalized composite services. As web services with similar functionality are expected to be provided by competing providers, a key challenge is to find the “best” web services to participate in the composition. When multiple quality aspects (e.g., response time, fee, etc.) are considered, a weighting mechanism is usually adopted by most existing approaches, which requires users to specify their preferences as numeric values. We propose to exploit the dominance relationship among service providers to find a set of “best” possible composite services, referred to as a composite service skyline. We develop efficient algorithms that allow us to find the composite service skyline from a significantly reduced searching space instead of considering all possible service compositions. We propose a novel bottom-up computation framework that enables the skyline algorithm to scale well with the number of services in a composition. We conduct a comprehensive analytical and experimental study to evaluate the effectiveness, efficiency, and scalability of the composite skyline computation approaches.

Collaborative Filtering (CF) has been increasingly employed as an effective vehicle for providing personalized service recommendations in service computing. CF exploits historical user-service interaction information to predict the preference of service users. A key challenge faced by CF is to handle new users with no previous interaction information. We present a novel strategy that integrates Matrix Factorization (MF) with decision tree learning to bootstrap service recommendation systems. The proposed strategy first employs MF to partition existing users into a set of user groups. In practice, only a small amount of user-service interaction information is observed. The MF based user partitioning scheme also provides a way to estimate the missing interaction information based on the group structure. The tree learning algorithm then leverages these estimated information and exploits user groups as class labels to learn a decision tree. Few highly discriminative services are identified as tree nodes to adaptively query a new user based on the interaction results with the prior services in the tree. Through a short and intuitive bootstrapping process, the new user is classified into one of the user groups, via which the user's preference is predicted. We conduct a set of experiments on real-world service data to demonstrate the effectiveness of the proposed bootstrapping strategy.

This paper presents a service-oriented digital government infrastructure focused on efficiently providing customized services to senior citizens. We designed and developed a Web Service Management System (WSMS), called WebSenior, which provides a service-centric framework to deliver government services to senior citizens. The proposed WSMS manages the entire life cycle of third-party web services. These act as proxies for real government services. Due to the specific requirements of our digital government application, we focus on the following key components of WebSenior: service composition, service optimization, and service privacy preservation. These components form the nucleus that achieves seamless cooperation among government agencies to provide prompt and customized services to senior citizens.

We propose a skyline computation approach that enables service users to optimally access sets of services as an integrated service package. We first present a one pass algorithm based on the observation that a multi-service skyline is completely determined by single service skylines. The skyline is returned after an enumeration on a significantly reduced candidate space. We then develop a dual progressive algorithm that is able to progressively report the skyline. We conduct an experimental study to assess the performance of the skyline computation approaches.

Efficient and accurate discovery of user desired Web services is a key component for achieving the full potential of service computing. However, service discovery is a non-trivial task considering the large and fast growing service space. Meanwhile, Web services are typically autonomous and a priori unknown. This further complicates the service discovery problem. We propose a service community learning algorithm that can generate homogeneous communities from the heterogeneous service space. This can greatly facilitate the service discovery process as the users only need to search within their desired service communities. A key ingredient of the community learning algorithm is a co-clustering scheme that leverages the duality relationship between services and operations. Experimental results on both synthetic and real Web services demonstrate the effectiveness of the proposed service community learning algorithm.

The performance of a service provider may fluctuate due to the dynamic service environment. Thus, the quality of service actually delivered by a service provider is inherently uncertain. Existing service optimization approaches usually assume that the quality of service does not change over time. Moreover, most of these approaches rely on computing a predefined objective function. When multiple quality criteria are considered, users are required to express their preference over different (and sometimes conflicting) quality attributes as numeric weights. This is rather a demanding task and an imprecise specification of the weights could miss user-desired services. We present a novel concept, called p-dominant service skyline. A provider S belongs to the p-dominant skyline if the chance that S is dominated by any other provider is less than p. Computing the p-dominant skyline provides an integrated solution to tackle the above two issues simultaneously. We present a p-R-tree indexing structure and a dual-pruning scheme to efficiently compute the p-dominant skyline. We assess the efficiency of the proposed algorithm with an analytical study and extensive experiments.

SERVICE-ORIENTED computing is gaining momentum as the next technological vehicle to leverage the huge investments in web application development. Web services are poised to take center stage as part of this adoption [4]. The ever-increasing number of web services will have the effect of transforming the web from a data-oriented repository to a service-oriented repository, also known as the Service Web [1]. In this new paradigm, existing business logic would be wrapped as web services to be accessible on the web via a web services middleware [2]. As the number of web services is expected to substantially increase, this would have the effect of introducing competition among web services that offer similar functionalities. Service users are enabled to select the “best” web services and/or their combinations with respect to their expected quality, such as price, response time, and reputation. There is a need to provide a sound framework to organize web services. This would form as a platform to query web services. Building this framework is especially important due to the ever-increasing large and heterogeneous web service deployment. A key ingredient of such a service framework is a formal service query model that can capture the key features of services to filter interactions and accelerate service searches. The query models must be congruent with the dynamic, active, autonomous, and highly heterogeneous nature of web services and their environment. Query languages and efficient selection techniques can then be developed once such a service model is in place. Existing service discovery technologies, such as service registries and service search engines, mainly support the simple keyword-based search on web services. However, keyword search cannot always precisely locate web services, partially because of the rich semantics embodied in these services. Due to the ambiguity of the keywords, which are typically described using natural language, either too many irrelevant services may be returned or some highly relevant services may be missed. As a key facilitator for application outsourcing, a common usage pattern of web services is to be programatically integrated into other applications (e.g., a travel package, navigation system, etc.). This further requires a service query mechanism that is more precise and reliable than keyword-based search. Query processing on web services is a novel concept that goes beyond the traditional data-centric view of query processing, which is mainly performance centered. It focuses on user quality parameters to select multiple services that are equivalent in functionality but exhibit a different quality of web service [3]. This special issue provides insights into the latest research on web service querying and efficient selection. Five articles were selected through a rigorous review process. They cover a set of key research topics including modeling techniques for web services, service query languages, algorithms for efficient service selection, as well as quality of web service modeling and quality-based service selection. The article by Skoutas et al., “Ranking and Clustering Web Services Using Multicriteria Dominance Relationships,” proposes a service selection framework that integrates the similarity matching scores of multiple parameters obtained from various matchmaking algorithms. The framework relies on the service dominance relationships to determine the relevance between services and users’ requests. Instead of using a weighting mechanism, the dominance relationship adopts a multi-objective strategy that simultaneously considers the matching scores of all the parameters for ranking the relevant services. A clustering algorithm is also proposed that captures the trade-offs among different parameters with respect to the considered matching criteria. The article by Grigori et al., “Ranking BPEL Processes for Service Discovery,” proposes a service discovery approach based on behavioral descriptions expressed in BPEL. Behavioral matchmaking goes beyond interface matchmaking as it considers the constraints on the invocation order of operations in service interfaces. Graph matching algorithms are applied, which enable the delivery of approximate behavioral matches. The article by Michlmayr et al., “End-to-End Support for QoSAware Service Selection, Binding, and Mediation in VRESCo,” describes a runtime environment for serviceoriented computing, called VRESCo. The proposed VRESCo framework provides a service metadata model. Service discovery and selection approaches are developed using this model. In addition, other important issues, such as QoS monitoring, dynamic binding, and service mediation, are addressed. The article by Barhamgi et al., “A Query Rewriting Approach for Web Service Composition,” proposes a service querying approach to compose dataproviding services. The data-providing services are modeled as RDF views over a mediated ontology specified in RDF to capture the consensual and shared knowledge in a IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 3, NO. 3, JULY-SEPTEMBER 2010 161

Web services are gaining momentum as a major vehicle to deliver business functionalities on the Web. More and more business organizations have begun to use Web services to facilitate user interactions and the collaboration among themselves. This essentially forms a large service space, which still keeps growing. Meanwhile, there may be functionality overlaps among different service providers. The concept of Quality of Web Service (QoWS) is emerging as a key feature in distinguishing between competing service providers. We present in this paper a systematic approach for efficient service selection by using QoWS as the major criterion. In particular, we adopt a relational approach that enables to store QoWS information in a relational DBMS and leverage standard relational operators for efficient service selection. We perform a preliminary set of experiments to evaluate the proposed service selection algorithms.

We present a query algebra that supports optimized access of Web services through service-oriented queries. The service query algebra is defined based on a formal service model that provides a high-level abstraction of Web services across an application domain. The algebra defines a set of algebraic operators. Algebraic service queries can be formulated using these operators. This allows users to query their desired services based on both functionality and quality. We provide the implementation of each algebraic operator. This enables the generation of Service Execution Plans (SEPs) that can be used by users to directly access services. We present an optimization algorithm by extending the Dynamic Programming (DP) approach to efficiently select the SEPs with the best user-desired quality. The experimental study validates the proposed algorithm by demonstrating significant performance improvement compared with the traditional DP approach.