Unsupervised industrial anomaly detection requires accurately identifying defects without labeled data. Traditional autoencoder-based methods often struggle with incomplete anomaly suppression and loss of fine details, as their single-pass decoding fails to effectively handle anomalies with varying severity and scale. We propose a recursive architecture for autoencoder (RcAE), which performs reconstruction iteratively to progressively suppress anomalies while refining normal structures. Unlike traditional single-pass models, this recursive design naturally produces a sequence of reconstructions, progressively exposing suppressed abnormal patterns. To leverage this reconstruction dynamics, we introduce a Cross Recursion Detection (CRD) module that tracks inconsistencies across recursion steps, enhancing detection of both subtle and large-scale anomalies. Additionally, we incorporate a Detail Preservation Network (DPN) to recover high-frequency textures typically lost during reconstruction. Extensive experiments demonstrate that our method significantly outperforms existing non-diffusion methods, and achieves performance on par with recent diffusion models with only 10% of their parameters and offering substantially faster inference. These results highlight the practicality and efficiency of our approach for real-world applications.

We examined the patterns of cues people utilised as they attempted to classify emails as legitimate or phishing via a novel approach – an information board study. In this paradigm, participants had to click on email elements to examine them, revealing which cues are utilised in the decision-making process as well as viewing duration. An online email classification task (87 Ps) was presented, including a between-subjects’ manipulation of cognitive load. This revealed consistent patterns of behaviour associated with identification of legitimate emails such as more time spent looking at signoffs aiding in identification, whereas more time spent looking at logo and greetings was associated with worse performance. We also found no-load condition participants with lower intuitive decision-making style scores identified more phishing emails correctly suggesting elaborative processing as an important determinant of good decision-making. However, none of; cyber security experience, impulsivity, and checking the sender’s address corresponded to higher rates of phishing email identification. Furthermore, we found no behavioural, cue-based patterns associated with correct decisions for phishing emails. We speculate that this decision-making process may be based on a global, holistic interpretation of the stimuli. This has important implications for awareness campaigns suggesting a more holistic approach to email evaluation.

Stock price forecasting is challenging due to market volatility and complex, dynamic inter-stock relationships. Existing graph-based approaches often rely primarily on static relational structures or simple time-aligned correlations, which capture only fixed or short-term relationships and fail to model transient cross-temporal dependencies and heterogeneous information sources. We propose a novel artificial intelligence framework for stock price forecasting that integrates heterogeneous data sources, including price co-movements, corporate linkages derived from Wikipedia, and industry affiliations, into a unified dynamic relational graph that combines both structural and behavioral dependencies. The proposed model, named Learning Unified Market Interdependencies (LUMI), adaptively models evolving inter-stock connections and uncovers latent dependencies beyond sectoral or time-aligned patterns. A dual-path temporal attention mechanism disentangles long-term trends from short-term fluctuations, capturing both periodic behaviors and abrupt market shifts. Extensive experiments on four market datasets demonstrate that the proposed deep learning framework outperforms strong baselines in predictive accuracy while providing interpretable insights into market interdependencies. These findings highlight the potential of artificial intelligence for modeling complex financial systems and improving algorithmic stock price forecasting.

The interpretability of deep neural networks is crucial for understanding model decisions in various applications, including computer vision. In this paper, we propose AttEXplore+, a unified adversarial attribution framework built upon AttEXplore that connects transferable adversarial exploration with gradient-path attribution. Rather than treating transferable attacks merely as additional attack choices, AttEXplore+ reformulates them as gradient acquisition operators for constructing smoother and more structured decision-boundary exploration paths. We instantiate AttEXplore++ with validated operators such as MIG and GRA, and conduct extensive experiments on five models, including CNNs (Inception-v3, ResNet-50, VGG16) and vision transformers (MaxViT-T, ViT-B/16), using the ImageNet dataset. Our method achieves an average performance improvement of 7.57% over AttEXplore and 32.62% compared to other state-of-the-art interpretability algorithms. Using insertion and deletion scores as evaluation metrics, we show that adversarial transferability plays a vital role in enhancing attribution results. Furthermore, we explore the impact of randomness, perturbation rate, noise amplitude, and diversity probability on attribution performance, demonstrating that AttEXplore++ provides more stable and reliable explanations across various models. We release our code at: https://github.com/KxPlaug/ATTEXPLOREP .

This paper explores the ability of Graph Neural Networks (GNNs) in learning various forms of information for link prediction, alongside a brief review of existing link prediction methods. Our analysis reveals that GNNs cannot effectively learn structural information related to the number of common neighbors between two nodes, primarily due to the nature of set-based pooling of the neighborhood aggregation scheme. Also, our extensive experiments indicate that trainable node embeddings can improve the performance of GNN-based link prediction models. Importantly, we observe that the denser the graph, the greater such the improvement. We attribute this to the characteristics of node embeddings, where the link state of each link sample could be encoded into the embeddings of nodes that are involved in the neighborhood aggregation of the two nodes in that link sample. In denser graphs, every node could have more opportunities to attend the neighborhood aggregation of other nodes and encode states of more link samples to its embedding, thus learning better node embeddings for link prediction. Lastly, we demonstrate that the insights gained from our research carry important implications in identifying the limitations of existing link prediction methods, which could guide the future development of more robust algorithms.

Machine learning algorithms often struggle to eliminate inherent data biases, particularly those arising from unreliable labels, which poses a significant challenge in ensuring fairness. Existing fairness techniques that address label bias typically involve modifying models and intervening in the training process, but these lack flexibility for large-scale datasets. To address this limitation, we introduce a data selection method designed to efficiently and flexibly mitigate label bias, tailored to more practical needs. Our approach utilizes a zero-shot predictor as a proxy model that simulates training on a clean holdout set. This strategy, supported by peer predictions, ensures the fairness of the proxy model and eliminates the need for an additional holdout set, which is a common requirement in previous methods. Without altering the classifier's architecture, our modality-agnostic method effectively selects appropriate training data and has proven efficient and effective in handling label bias and improving fairness across diverse datasets in experimental evaluations.

The task of identifying multimodal image-text representations has garnered increasing attention, particularly with models such as CLIP (Contrastive Language-Image Pretraining), which demonstrate exceptional performance in learning complex associations between images and text. Despite these advancements, ensuring the interpretability of such models is paramount for their safe deployment in real-world applications, such as healthcare. While numerous interpretability methods have been developed for unimodal tasks, these approaches often fail to transfer effectively to multimodal contexts due to inherent differences in the representation structures. Bottleneck methods, well-established in information theory, have been applied to enhance CLIP's interpretability. However, they are often hindered by strong assumptions or intrinsic randomness. To overcome these challenges, we propose the Narrowing Information Bottleneck Theory, a novel framework that fundamentally redefines the traditional bottleneck approach. This theory is specifically designed to satisfy contemporary attribution axioms, providing a more robust and reliable solution for improving the interpretability of multimodal models. In our experiments, compared to state-of-the-art methods, our approach enhances image interpretability by an average of 9\%, text interpretability by an average of 58.83\%, and accelerates processing speed by 63.95\%. Our code is publicly accessible at https://github.com/LMBTough/NIB.

Out-of-distribution (OOD) detection is essential for enhancing the robustness and security of deep learning models in unknown and dynamic data environments. Gradient-based OOD detection methods, such as GAIA, analyse the explanation pattern representations of in-distribution (ID) and OOD samples by examining the sensitivity of model outputs w.r.t. model inputs, resulting in superior performance compared to traditional OOD detection methods. However, we argue that the non-zero gradient behaviors of OOD samples do not exhibit significant distinguishability, especially when ID samples are perturbed by random perturbations in high-dimensional spaces, which negatively impacts the accuracy of OOD detection. In this paper, we propose a novel OOD detection method called S & I based on layer Splitting and gradient Integration via Adversarial Gradient Attribution. Specifically, our approach involves splitting the model’s intermediate layers and iteratively updating adversarial examples layer-by-layer. We then integrate the attribution gradients from each intermediate layer along the attribution path from adversarial examples to the actual input, yielding true explanation pattern representations for both ID and OOD samples. Experiments demonstrate that our S & I algorithm achieves state-of-the-art results, with the average FPR95 of 29.05% (ResNet34)/38.61% (WRN40) and 37.31% (BiT-S) on the CIFAR100 and ImageNet benchmarks, respectively. Our code is available at: https://github.com/LMBTough/S-Ihttps://github.com/LMBTough/S-I

Reproducibility is indispensable for transferring explainable-AI algorithms from academic prototypes to production systems. This companion paper documents the artefacts, procedures, and outcomes that reproduce the empirical claims of ''Enhancing Model Interpretability with Local Attribution over Global Exploration'' (ACM MM 2024). We release a containerised archive containing source code, data-serialisation scripts, one-click executables, and a detailed README, all conforming to the ACM Multimedia reproducibility guidelines. The regenerated Insertion and Deletion scores deviate by only 2.2% on average. In addition, an exhaustive 10, 20, 30 3 grid-search over key hyper-parameters reveals a new configuration, (30, 20, 30), that improves the Insertion score of three convolutional backbones by 7.51% without additional code changes. These artefacts provide a rigorous, extensible foundation for future research on local attribution methods. Our code is available at: https://github.com/LMBTough/LA/

Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data. The implementation code and data are available at https://github.com/JinLi-i/MLLM-CD.

Sequential recommender systems (SRSs) aim to recommend the next items to well match users' preferences. In addition to recommendation accuracy, diversity is another critical aspect in evaluating SRSs. Recently, the emerging diffusion models (DMs) have been widely adopted in SRSs. Their employed learning-to-generate paradigm allows them to cover a much broader range of users' preferences and thus generate more diversified items. However, existing DM-based SRSs still face two significant gaps that prevent them from further improving the recommendation diversity: (1) they often rely on non-diversified users' preferences as guidance to direct the training of diffusion networks, restricting networks' ability to generate diverse items; and (2) they are based on a homogeneous diffusion inference mechanism to generate the next items and thus can only accommodate users' major preferences. Such a practice neglects users' heterogeneous preferences towards various types of items, further limiting recommendation diversity. To bridge these two critical gaps and to further unleash the potential of DMs in enhancing the recommendation diversity of SRSs, we propose a novel diversity-guided diffusion model for sequential recommendations, called DiffDiv for short. To be specific, first, a new diversity-aware guidance learning mechanism is devised to direct the training of DMs to effectively capture users' diversified preferences from their historical interactions. Then, a novel heterogeneous diffusion inference mechanism is designed to generate diversified items to accommodate users' heterogeneous preferences, further boosting the recommendation diversity. Extensive experiments on real-world datasets validate the effectiveness of DiffDiv in terms of both recommendation accuracy and diversity.

Incomplete scenario is a prevalent, practical, yet challenging setting in Multimodal Recommendations (MMRec), where some item modalities are missing due to various factors. Recently, a few efforts have sought to improve the recommendation accuracy by exploring generic structures from incomplete data. However, two significant gaps persist: 1) the difficulty in accurately generating missing data due to the limited ability to capture modality distributions; and 2) the critical but overlooked visibility bias, where items with missing modalities are more likely to be disregarded due to the prioritization of items' multimodal data over user preference alignment. This bias raises serious concerns about the fair treatment of items. To bridge these two gaps, we propose a novel Modality-Diffused Counterfactual (MoDiCF) framework for incomplete multimodal recommendations. MoDiCF features two key modules: a novel modality-diffused data completion module and a new counterfactual multimodal recommendation module. The former, equipped with a particularly designed multimodal generative framework, accurately generates and iteratively refines missing data from learned modality-specific distribution spaces. The latter, grounded in the causal perspective, effectively mitigates the negative causal effects of visibility bias and thus assures fairness in recommendations. Both modules work collaboratively to address the two aforementioned significant gaps for generating more accurate and fair results. Extensive experiments on three real-world datasets demonstrate the superior performance of MoDiCF in terms of both recommendation accuracy and fairness. The code and processed datasets are released at https://github.com/JinLi-i/MoDiCF.

The classical temporal point process (TPP) constructs an intensity function by taking the occurrence times into account. Nevertheless, occurrence time may not be the only relevant factor, other contextual data, termed covariates, may also impact the event evolution. Incorporating such covariates into the model is beneficial, while distinguishing their relevance to the event dynamics is of great practical significance. In this work, we propose a Transformer-based covariate temporal point process (TransFeat-TPP) model to improve the interpretability of deep covariate-TPPs while maintaining powerful expressiveness. TransFeat-TPP can effectively model complex relationships between events and covariates, and provide enhanced interpretability by discerning the importance of various covariates. Experimental results on synthetic and real datasets demonstrate improved prediction accuracy and consistently interpretable feature importance when compared to existing deep covariate-TPPs. Our code is available at https://github.com/waystogetthere/TransFeat.git.

Imitation learning demonstrates remarkable performance in various domains. However, imitation learning is also constrained by many prerequisites. The research community has done intensive research to alleviate these constraints, such as introducing a stochastic policy to avoid unseen states, eliminating the need for action labels, and learning from the suboptimal demonstrations. Inspired by the natural reproduction process, we proposed a method called GenIL that integrates genetic operations with imitation learning. The involvement of the genetic operations improves the data efficiency by reproducing trajectories with various returns and assists the model in estimating more accurate and compact reward function parameters. We tested GenIL in both Atari and Mujoco domains, and the empirical result shows that it successfully outperforms the previous extrapolation methods in terms of extrapolation accuracy and overall policy performance when input data is limited. Code, data, and supplementary materials are publicly available at https://github.com/BryanZ666/GenIL.

In the era of information explosion, news recommender systems are crucial for users to effectively and efficiently discover their interested news. However, most of the existing news recommender systems face two major issues, hampering recommendation quality. Firstly, they often oversimplify users' reading interests, neglecting their hierarchical nature, spanning from high-level event (e.g., US Election) related interests to low-level news article-specifc interests. Secondly, existing work often assumes a simplistic context, disregarding the prevalence of fake news and political bias under the real-world context. This oversight leads to recommendations of biased or fake news, posing risks to individuals and society. To this end, this paper addresses these gaps by introducing a novel framework, the Hierarchical and Disentangling Interest learning framework (HDInt). HDInt incorporates a hierarchical interest learning module and a disentangling interest learning module. The former captures users' high- and low-level interests, enhancing next-news recommendation accuracy. The latter effectively separates polarity and veracity information from news contents and model them more specifcally, promoting fairness- and truth-aware reading interest learning for unbiased and true news recommendations. Extensive experiments on two real-world datasets demonstrate HDInt's superiority over state-of-the-art news recommender systems in delivering accurate, unbiased, and true news recommendations.

Traffic accidents are often inaccurately reported, with incorrect location and disruption duration due to various external factors. This can result in imprecise predictions and inaccurate decision-making in data-driven models. To address these challenges, our study presents a comprehensive framework for traffic disruption segmentation from traffic speed data (obtained from Caltrans Performance Measurements system) in the time-space proximity of reported accidents (from Countrywide Traffic Accident dataset). Furthermore, we evaluate multiple machine learning models on reported, estimated, and manually marked disruption intervals, and demonstrate that our enhanced modelling approach reduces the root mean squared error (RMSE) of traffic accident duration prediction while providing higher similarity with disruptions observed in traffic speed. Our algorithm yields higher disruption detection precision than reported accident timelines. Although using multiple segments offers a slight decrease in the quality of results, it highlights more disruptions. Future research could explore expanding the algorithm’s complexity and applying it to improve traffic incident impact predictions.

Imitation learning (IL) aims to extract knowledge from human experts' demonstrations or artificially created agents to replicate their behaviors. It promotes interdisciplinary communication and real-world automation applications. However, the process of replicating behaviors still exhibits various problems, such as the performance is highly dependent on the demonstration quality, and most trained agents are limited to perform well in task-specific environments. In this survey, we provide an insightful review on IL. We first introduce the background knowledge from development history and preliminaries, followed by presenting different taxonomies within IL and key milestones of the field. We then detail challenges in learning strategies and present research opportunities with learning policy from suboptimal demonstration, voice instructions, and other associated optimization schemes.

In learning with fairness, for every instance, its label can be randomly flipped to another class due to the practitioner’s prejudice, namely, label bias. The existing well-studied fair representation learning methods focus on removing the dependency between the sensitive factors and the input data, but do not address how the representations retain useful information when the labels are unreliable. In fact, we find that the learned representations become random or degenerated when the instance is contaminated by label bias. To alleviate this issue, we investigate the problem of learning fair representations that are independent of the sensitive factors while retaining the task-relevant information given only access to unreliable labels. Our model disentangles the dependency between fair representations and sensitive factors in the latent space. To remove the reliance between the labels and sensitive factors, we incorporate an additional penalty based on mutual information. The learned purged fair representations can then be used in any downstream processing. We demonstrate the superiority of our method over previous works through multiple experiments on both synthetic and real-world datasets.

Graph neural networks (GNNs) have proven their efficacy in a variety of real-world applications, but their underlying mechanisms remain a mystery. To address this challenge and enable reliable decision-making, many GNN explainers have been proposed in recent years. However, these methods often encounter limitations, including their dependence on specific instances, lack of generalizability to unseen graphs, producing potentially invalid explanations, and yielding inadequate fidelity. To overcome these limitations, we, in this paper, introduce the Auxiliary Classifier Generative Adversarial Network (ACGAN) into the field of GNN explanation and propose a new GNN explainer dubbed ACGAN-GNNExplainer. Our approach leverages a generator to produce explanations for the original input graphs while incorporating a discriminator to oversee the generation process, ensuring explanation fidelity and improving accuracy. Experimental evaluations conducted on both synthetic and real-world graph datasets demonstrate the superiority of our proposed method compared to other existing GNN explainers.

Contrastive-learning-based neural networks have recently been introduced to recommender systems, due to their unique advantage of injecting collaborative signals to model deep representations, and the self-supervision nature in the learning process. Existing contrastive learning methods for recommendations are mainly proposed through introducing augmentations to the user-item (U-I) bipartite graphs. Such a contrastive learning process, however, is susceptible to bias towards popular items and users, because higher-degree users/items are subject to more augmentations and their correlations are more captured. In this paper, we advocate a Siamese Graph Contrastive Consensus Learning (SGCCL) framework, to explore intrinsic correlations and alleviate the bias effects for personalized recommendation. Instead of augmenting original U-I networks, we introduce siamese graphs, which are homogeneous relations of user-user (U-U) similarity and item-item (I-I) correlations. A contrastive consensus optimization process is also adopted to learn effective features for user-item ratings, user-user similarity, and item-item correlation. Finally, we employ the self-supervised learning coupled with the siamese item-item/user-user graph relationships, which ensures unpopular users/items are well preserved in the embedding space. Different from existing studies, SGCCL performs well on both overall and debiasing recommendation tasks resulting in a balanced recommender. Experiments on four benchmark datasets demonstrate that SGCCL outperforms state-of-the-art methods with higher accuracy and greater long-tail item/user exposure.

Intelligent transportation system (ITS) is a crucial symbol of smart cities, which aims to provide sustainable and efficient services to residents. Playing a vital role in ITS, railway systems have been integrating with multiple Internet of Things (IoT) devices to monitor real-time inbound passenger flows to ensure pedestrian safety. But it remains difficult to consolidate real-time information from different IoT sources and accurately estimate the future flow due to the coarse-grained data, potential impacts of dynamic interchanged passengers, and real-time predictive capability, which have greatly hindered the progress of ITS transformation in smart cities. To tackle these challenges, we propose a two-stage self-adaptive model for accurately and timely predicting passenger flow in metropolitan railway systems. In the first stage, a self-attention-based prediction model is introduced to predict the next-day passenger flow based on the historical boarding records captured by IoT devices. The proposed decomposing components transferring the discrete boarding records into continuous patterns enable the module to deliver a robust minute-level prediction. In the second stage, a real-time fine-tuning model is developed to adjust the predicted passenger flow based on real-time emergencies and short-term changes in passenger flows from IoT devices. The combination of an offline deep learning mechanism and a real-time reallocation algorithm ensures the real-time response without loss of accuracy. Our end-to-end framework has been deployed to the railway system in Greater Sydney Area, Australia, which can offer accurate predictions to trip planners for timetable design and provide timely decision support for controllers when emergencies happen.

Instance image retrieval could greatly benefit from discovering objects in the image dataset. This not only helps produce more reliable feature representation but also better informs users by delineating query-matched object regions. However, object classes are usually not predefined in a retrieval dataset and class label information is generally unavailable in image retrieval. This situation makes object discovery a challenging task. To address this, we propose a novel dataset-driven unsupervised object discovery framework. By utilizing deep feature representation and weakly-supervised object detection, we explore supervisory information from within an image dataset, construct class-wise object detectors, and assign multiple detectors to each image for detection. To efficiently construct object detectors for large image datasets, we propose a novel "base-detector repository" and derive a fast way to generate the base detectors. In addition, the whole framework is designed to work in a self-boosting manner to iteratively refine object discovery. Compared with existing unsupervised object detection methods, our framework produces more accurate object discovery results. Different from supervised detection, we need neither manual annotation nor auxiliary datasets to train object detectors. Experimental study demonstrates the effectiveness of the proposed framework and the improved performance for region-based instance image retrieval.

Deep Neural Networks (DNNs) suffer from domain shift when the test dataset follows a distribution different from the training dataset. Domain generalization aims to tackle this issue by learning a model that can generalize to unseen domains. In this paper, we propose a new approach that aims to explicitly remove domain-specific features for domain generalization. Following this approach, we propose a novel framework called Learning and Removing Domain-specific features for Generalization (LRDG) that learns a domain-invariant model by tactically removing domain-specific features from the input images. Specifically, we design a classifier to effectively learn the domain-specific features for each source domain, respectively. We then develop an encoder-decoder network to map each input image into a new image space where the learned domain-specific features are removed. With the images output by the encoder-decoder network, another classifier is designed to learn the domain-invariant features to conduct image classification. Extensive experiments demonstrate that our framework achieves superior performance compared with state-of-the-art methods.

Public speaking is a critical skill in daily communication. While more practicing such as rehearsal is helpful to improve such a skill, lack of personalized feedback limits the effectiveness of practicing. Therefore, we formulate the task of personalized feedback as an affective audio annotation problem by learning knowledge from online public speech videos. Considering the great success of deep learning techniques such as convolutional neural networks in a wide range of applications including speech recognition and object recognition, we propose a novel convolutional clustering neural network (CCNN) to solve this multi-label classification problem. Instead of aggregating the features of different channels through pooling, we introduce a novel clustering layer to derive intermediate representation for improved annotation performance. In order to evaluate the performance of our proposed method, we purposely built an affective audio annotation dataset by collecting more than 2,000 video clips from the TED website. Experimental results on this dataset demonstrate that our proposed method outperforms traditional CNN-based approaches with a lower hamming loss for affective annotation.

Traffic control optimization is a challenging task for various traffic centers around the world and the majority of existing approaches focus only on developing adaptive methods for normal (recurrent) traffic conditions. Optimizing the control plans when severe incidents occur still remains an open problem, especially when a high number of lanes or entire intersections are affected. This paper presents a novel methodology for optimizing the traffic signal timings in signalized urban intersections, under non-recurrent traffic incidents. With the purpose of producing fast and reliable decisions, we combine the fast running Machine Learning (ML) algorithms and the reliable Genetic Algorithms (GA) into a single optimization framework. Firstly, we deploy a typical GA algorithm by considering the phase duration as the decision variable and the objective function as the total travel time in the network. We fine tune the GA for crossover, mutation, fitness calculation and obtain the optimal parameters. Secondly, we train several regression models to predict the total travel time in the studied traffic network, and select the best performing model which we further hyper-tune. Lastly, we propose a new algorithm BGA-ML combining the GA algorithm and the extreme-gradient decision-tree (XGBT), which is the best performing regression model, together in a single optimization framework. Comparison and results are generated by two experiments (one synthetic and one from real urban traffic network) and show that the new BGA-ML is much faster than the original GA algorithm and can reduce the total travel time by almost half when used under incident conditions.

For image captioning, recent works start to focus on exploring visual relationships for generating high-quality interactive words (i.e. verbs and prepositions). However, many existing works only focus on semantic level by analysing the feature similarity between objects in the visual domain but ignore the linguistic context included in the caption decoder. When captioning is being carried out, the entity words can be inferred based on visual information of objects. The interactive words representing the relationships between entity words can only be inferred based on high-level language meaning generated in the process of captioning decoding. Such high-level language meaning is called linguistic context, which refers to the relational context between words or phrases in the caption sentences. The linguistic context can be used as strong guidance to explore related visual relationships between different objects effectively. To achieve this, we propose a novel context-adaptive attention module that is strongly driven by the linguistic context from the caption decoder. In this module, a novel design of visual relationship attention is proposed based on a bilinear self-attention model to explore related visual relationships and encode more discriminative features under the linguistic context. To achieve the adaptive process of attending to related visual relationships for generating interactive words or related visual objects for entity words, an attention modulator is integrated as an attention channel controller responding to the changing linguistic context of the caption decoder dynamically. Experimented on MSCOCO dataset, our model achieves promising performances compared with all counterpart models that explore visual relationships.

The label bias and selection bias are acknowledged as two reasons in data that will hinder the fairness of machine-learning outcomes. The label bias occurs when the labeling decision is disturbed by sensitive features, while the selection bias occurs when subjective bias exists during the data sampling. Even worse, models trained on such data can inherit or even intensify the discrimination. Most algorithmic fairness approaches perform an empirical risk minimization with predefined fairness constraints, which tends to trade-off accuracy for fairness. However, such methods would achieve the desired fairness level with the sacrifice of the benefits (receive positive outcomes) for individuals affected by the bias. Therefore, we propose a \textbf{B}ias-Tolerant \textbf{FA}ir \textbf{R}egularized \textbf{L}oss (B-FARL), which tries to regain the benefits using data affected by label bias and selection bias. B-FARL takes the biased data as input, calls a model that approximates the one trained with fair but latent data, and thus prevents discrimination without constraints required. In addition, we show the effective components by decomposing B-FARL, and we utilize the meta-learning framework for the B-FARL optimization. The experimental results on real-world datasets show that our method is empirically effective in improving fairness towards the direction of true but latent labels.

Graphs/Networks are common in real-world applications where data have rich content and complex relationships. The increasing popularity also motivates many network learning algorithms, such as community detection, clustering, classification, and embedding learning, etc.. In reality, the large network volumes often hider a direct use of learning algorithms to the graphs. As a result, it is desirable to have the flexibility to condense a network to an arbitrary size, with well-preserved network topology and node content information. In this paper, we propose a graph compression network (GEN) to achieve network compression and embedding at the same time. Our theme is to leverage the network topology to find node mappings, such that densely connected nodes, including their node content, are compressed as a new node, with a latent vector (i.e. embedding) being learned to represent the compressed node. In addition to compression learning, we also develop a novel encoding-decoding framework, using feature diffusion process, to "decompress" the condensed network. Different from traditional graph convolution which uses direct-neighbor message passing, our decompression advocates high-order message passing within compressed nodes to learning feature representation for all nodes in the network. A unique strength of GEN is that it leverages the graph neural network principle to learn mapping automatically, so one can compress a network to an arbitrary size, and also decompress it to the original node space with minimum information loss. Experiments and comparisons confirm that GEN can automatically find clusters and communities, and compress them as new nodes. Results also show that GEN achieves improved performance for numerous tasks, including graph classification and node clustering.

Pipe failure prediction in the water industry aims to prioritize the pipes that are at high risk of failure for proactive maintenance. However, existing statistical or machine learning models that rely on historical failures and asset attributes can hardly leverage the structure information of pipe networks. In this work, we develop a failure prediction framework for pipe networks by jointly considering the pipes' features, the network structure, the geographical neighboring effect, and the temporal failure series. We apply a multi-hop Graph Neural Network (GNN) to failure prediction. We propose a method of constructing a geographical graph structure depending on not only the physical connections but also geographical distances between pipes. To differentiate the pipes with diverse properties, we employ an attention mechanism in the neighborhood aggregation process of each GNN layer. Also, residual connections and layer-wise aggregation are used to avoid the over-smoothing issue in deep GNNs. The historical failures exhibit a strong temporal pattern. Inspired by point process, we develop a module to learn the pipes' evolutionary effect and the time-decayed excitement of historical failures on the current state of the pipe. The proposed framework is evaluated on two real-world large-scale pipe networks. It outperforms the existing statistical, machine learning, and state-of-the-art GNN baselines. Our framework provides the water utility with core data-driven support for proactive maintenance including regular pipe inspection, pipe renewal planning, and sensor system deployment. It can be extended to other infrastructure networks in the future.

The recent Coronavirus Infectious Disease 2019 (COVID-19) pandemic has caused an unprecedented impact across the globe. We have also witnessed millions of people with increased mental health issues, such as depression, stress, worry, fear, disgust, sadness, and anxiety, which have become one of the major public health concerns during this severe health crisis. Depression can cause serious emotional, behavioral, and physical health problems with significant consequences, both personal and social costs included. This article studies community depression dynamics due to the COVID-19 pandemic through user-generated content on Twitter. A new approach based on multimodal features from tweets and term frequency-inverse document frequency (TF-IDF) is proposed to build depression classification models. Multimodal features capture depression cues from emotion, topic, and domain-specific perspectives. We study the problem using recently scraped tweets from Twitter users emanating from the state of New South Wales in Australia. Our novel classification model is capable of extracting depression polarities that may be affected by COVID-19 and related events during the COVID-19 period. The results found that people became more depressed after the outbreak of COVID-19. The measures implemented by the government, such as the state lockdown, also increased depression levels.

Motivated by the practical demands for simplification of data toward being consistent with human thinking and problem-solving, as well as tolerance of uncertainty, information granules are becoming important entities in data processing at different levels of data abstraction. This article proposes a method to construct classifiers from multiresolution hierarchical granular representations using hyperbox fuzzy sets. The proposed approach forms a series of granular inferences hierarchically through many levels of abstraction. An attractive characteristic of our classifier is that it can maintain a high accuracy in comparison to other fuzzy min-max models at a low degree of granularity based on reusing the knowledge learned from lower levels of abstraction. In addition, our approach can reduce the data size significantly as well as handle the uncertainty and incompleteness associated with data in real-world applications. The construction process of the classifier consists of two phases. The first phase is to formulate the model at the greatest level of granularity, while the later stage aims to reduce the complexity of the constructed model and deduce it from data at higher abstraction levels. Experimental analyses conducted comprehensively on both synthetic and real datasets indicated the efficiency of our method in terms of training time and predictive performance in comparison to other types of fuzzy min-max neural networks and common machine learning algorithms.

Multi-label image classification aims to predict the labels associated with a given image. While most existing methods utilize unified image representations, extracting label-specific features through input space learning would improve the discriminative power of the learned features. On the other hand, most feature learning studies often ignore the learning in the output label space, although taking advantage of label correlations can boost the classification performance. In this paper, we propose a deep learning framework that incorporates flexible modules which can learn from both input and output spaces for multi-label image classification. For the input space learning, we devise a label-specific feature pooling method to refine convolutional features for obtaining features specific to each label. For the output space learning, we design a Two-Stream Graph Convolutional Network (TSGCN) to learn multi-label classifiers by mapping spatial object relationships and semantic label correlations. More specifically, we build object spatial graphs to characterize the spatial relationships among objects in an image, which supplements the label semantic graphs modelling the semantic label correlations. Experimental results on two popular benchmark datasets (i.e., Pascal VOC and MS-COCO) show that our proposed method achieves superior performance over the state-of-the-arts.

Machine Learning (ML) plays a key role in various intelligent systems, and building an effective ML model for a data set is a difficult task involving various steps including data cleaning, feature definition and extraction, ML algorithms development, model training and evaluation as well as others. One of the most important steps in the process is to compare generated substantial amounts of ML models to find the optimal one for the deployment. It is challenging to compare such models with dynamic number of features. This paper proposes a novel visualisation approach based on a radial net to compare ML models trained with a different number of features of a given data set while revealing implicit dependent relations. In the proposed approach, ML models and features are represented by lines and arcs respectively. The dependence of ML models with dynamic number of features is encoded into the structure of visualisation, where ML models and their dependent features are directly revealed from related line connections. ML model performance information is encoded with colour and line width in the innovative visualisation. Together with the structure of visualization, feature importance can be directly discerned to help to understand ML models.

Preventing water pipe leaks and breaks has high priority for water utilities. It is a critical task for the utility to reduce water loss through leaks and breaks detection in water mains. The failure prediction and data analytics research have been conducted for an Australian water utility over the last few years to enhance the prediction of leaks and breaks detection in water mains. Intelligent sensing at sensitive locations with current research aids in prioritising investigation and prevention of potential breaks and leaks in water mains. The purpose of this work is to integrate the predictive analytics and intelligent sensing applications to identify high risk mains prior to failures. Predictive analytics and minimum night flow (MNF) analysis have been utilised to prioritise risky zones over the whole water network, and then risky pipes are identified to optimise sensors deployment. The sensing data is being collected for analysis and validation, and a machine learning model is being built based on the analysis results. This work is currently under progress and the planned outcomes will help the utility reduce water loss, improve leak detection, and enhance customer satisfaction by automating the process of leak detection using a data driven approach with smart sensors.

Multimodal sentiment analysis utilizes multiple heterogeneous modalities for sentiment classification. The recent multimodal fusion schemes customize LSTMs to discover intra-modal dynamics and design sophisticated attention mechanisms to discover the inter-modal dynamics from multimodal sequences. Although powerful, these schemes completely rely on attention mechanisms which is problematic due to two major drawbacks 1) deceptive attention masks, and 2) training dynamics. Nevertheless, strenuous efforts are required to optimize hyperparameters of these consolidate architectures, in particular their custom-designed LSTMs constrained by attention schemes. In this research, we first propose a common network to discover both intra-modal and inter-modal dynamics by utilizing basic LSTMs and tensor based convolution networks. We then propose unique networks to encapsulate temporal-granularity among the modalities which is essential while extracting information within asynchronous sequences. We then integrate these two kinds of information via a fusion layer and call our novel multimodal fusion scheme as Deep-HOSeq (Deep network with higher order Common and Unique Sequence information). The proposed Deep-HOSeq efficiently discovers all-important information from multimodal sequences and the effectiveness of utilizing both types of information is empirically demonstrated on CMU-MOSEI and CMU-MOSI benchmark datasets. The source code of proposed Deep-HOSeq is available at https://github.com/sverma88/Deep-HOSeq-ICDM-2020.

This paper proposes an improved version of the current online learning algorithm for a general fuzzy min-max (GFMM) neural network to tackle existing issues concerning expansion and contraction steps as well as the way of dealing with unseen data located on decision boundaries. These drawbacks lower its classification performance, so an improved algorithm is proposed in this study to address the above limitations. The proposed approach does not use the contraction process for overlapping hyperboxes, which is more likely to increase the error rate as shown in the literature. The empirical results indicated the improvement in the classification accuracy and stability of the proposed method compared to the original version and other fuzzy min-max classifiers. In order to reduce the sensitivity to the training samples presentation order of this new on-line learning algorithm, a simple ensemble method is also proposed.

Despite its success for anomaly detection in the scenario where only data representing normal behavior are available, one-class support vector machine (OCSVM) still has challenge in dealing with non-stationary data stream, where the underlying distributions of data are time-varying. Existing OCSVM-based online learning methods incrementally update the model to address the challenge, however, they solely rely on the location relationship between a test sample and error support vectors. To better accommodate normal behavior evolution, online anomaly detection in non-stationary data stream is formulated as a concept drift adaptation problem in this paper. It is proposed that OCSVM-based incremental learning is only performed in the case of a normal drift. For an incoming sample, its relative relationship with three sets of vectors in OCSVM, namely margin support vectors, error support vectors, and reserve vectors is fully utilized to estimate whether a normal drift is emerging. Extensive experiments in the field of structural health monitoring have been conducted and the results have shown that the proposed simple approach outperforms the existing OCSVM-based online learning algorithms for anomaly detection.

Monitoring drinking water quality in the entire delivery network, mainly indicated by total chlorine (TC), is a critical component of overall water supply management. However, it is extremely difficult to collect sufficient TC data from the network at customer sites, which makes it sparse for comprehensive modelling. This paper details an approach that provides TC prediction within the entire Woronora delivery network in Sydney in the next 24 hours. First, the hydraulic system is employed to capture the topology of the delivery network, so that the water travel time can be estimated using predicted water demand. The travel time links the upstream (reservoir) data to the downstream (resident) data. Then, a two-step strategy is proposed as a semi-parametric method to determine the crucial factors and build Bayesian model for TC decay to predict TC with the travel time. Lastly, the uncertainties of both data and the model are analysed to define the boundaries of prediction for better decision making. Several operational stages are involved when the approach is being deployed, including prediction interpretation, interactive tool development for water quality mapping and visualisation, and proactive optimisation. This has established a successful initiative to improve the overall water supply management for the entire Woronora delivery network.

In this paper, we propose a discriminative variational autoencoder (DVAE) to assist deep learning from data with imbalanced class distributions. DVAE is designed to alleviate the class imbalance by explicitly learning class boundaries between training samples, and uses learned class boundaries to guide the feature learning and sample generation. To learn class boundaries, DVAE learns a latent two-component mixture distributor, conditioned by the class labels, so the latent features can help differentiate minority class vs. majority class samples. In order to balance the training data for deep learning to emphasize on the minority class, we combine DVAE and generative adversarial networks (GAN) to form a unified model, DVAAN, which generates synthetic instances close to the class boundaries as training data to learn latent features and update the model. Experiments and comparisons confirm that DVAAN significantly alleviates the class imbalance and delivers accurate models for deep learning from imbalanced data.

Visual attention mechanisms have been broadly used by image captioning models to attend to related visual information dynamically, allowing fine-grained image understanding and reasoning. However, they are only designed to discover the region-level alignment between visual features and the language feature. The exploration of higher-level visual relationship information between image regions, which is rarely researched in recent works, is beyond their capabilities. To fill this gap, we propose a novel visual relationship attention model based on the parallel attention mechanism under the learnt spatial constraints. It can extract relationship information from visual regions and language and then achieve the relationship-level alignment between them. Using combined visual relationship attention and visual region attention to attend to related visual relationships and regions respectively, our image captioning model can achieve state-of-the-art performances on the MSCOCO dataset. Both quantitative analysis and qualitative analysis demonstrate that our novel visual relationship attention model can capture related visual relationship and further improve the caption quality.

In the human-machine collaboration context, understanding the reason behind each human decision is critical for interpreting the performance of the human-machine team. Via an experimental study of a system with varied levels of accuracy, we describe how human trust interplays with system performance, human perception and decisions. It is revealed that humans are able to perceive the performance of automatic systems and themselves, and adjust their trust levels according to the accuracy of systems. The 70% system accuracy suggests to be a threshold between increasing and decreasing human trust and system usage. We have also shown that trust can be derived from a series of users' decisions rather than from a single one, and relates to the perceptions of users. A general framework depicting how trust and perception affect human decision making is proposed, which can be used as future guidelines for human-machine collaboration design.

This special issue highlights 11 papers from the ACM Intelligent User Interface (IUI) 2017 international conference, which received 272 submissions and of which 63 of them were accepted by the program committee (acceptance rate: 23%).The 11 articles in this special issue of TiiS were selected by the program committee to be among the best papers accepted into the program of ACM IUI 2017.Each of these selected conference papers was extended, with at least 30% new materials in the new version, which allowed authors to elaborate on their excellent lines of research relevant to the ACM IUI community.The extended versions of the papers have gone through a new review process by at least three reviewers.This special issue starts with two articles on analyzing and learning from multimodal signals to inform human behavior.In "AttentiveVideo: A Multimodal Approach to Quantify Emotional Responses to Mobile Advertisements," the authors propose AttentiveVideo, a scalable intelligent mobile interface with corresponding inference algorithms to monitor and quantify the effects of mobile video advertising in real time.This interface employs a combination of implicit photoplethysmography sensing and facial expression analysis to detect attention, engagement, and sentiment of viewers as they watch video advertisements on unmodified smartphones.In "Wearables and Social Signal Processing for Better Public Presentations," the authors studied

The soft-attention mechanism is regarded as one of the representative methods for image captioning. Based on the end-to-end convolutional neural network (CNN)-long short term memory (LSTM) framework, the soft-attention mechanism attempts to link the semantic representation in text (i.e., captioning) with relevant visual information in the image for the first time. Motivated by this approach, several state-of-the-art attention methods are proposed. However, due to the constraints of CNN architecture, the given image is only segmented to the fixed-resolution grid at a coarse level. The visual feature extracted from each grid indiscriminately fuses all inside objects and/or their portions. There is no semantic link between grid cells. In addition, the large area “stuff” (e.g., the sky or a beach) cannot be represented using the current methods. To address these problems, this paper proposes a new model based on the fully convolutional network (FCN)-LSTM framework, which can generate an attention map at a fine-grained grid-wise resolution. Moreover, the visual feature of each grid cell is contributed only by the principal object. By adopting the grid-wise labels (i.e., semantic segmentation), the visual representations of different grid cells are correlated to each other. With the ability to attend to large area “stuff,” our method can further summarize an additional semantic context from semantic labels. This method can provide comprehensive context information to the language LSTM decoder. In this way, a mechanism of fine-grained and semantic-guided visual attention is created, which can accurately link the relevant visual information with each semantic meaning inside the text. Demonstrated by three experiments including both qualitative and quantitative analyses, our model can generate captions of high quality, specifically high levels of accuracy, completeness, and diversity. Moreover, our model significantly outperforms all other methods that use VGG-based CNN encoders without fine-tuning.

Failure event prediction is becoming increasingly important in wide applications, such as the planning of proactive maintenance, the active investment management, and disease surveillance. To address the issue, the hazard function in survival analysis has been employed to describe the pattern of failures. Different from traditional survival analysis, this paper discovers how to apply recurrent neural network (RNN) to the long-term hazard function prediction. The proposed Long-Term RNN (LT-RNN) is able to leverage the precedent information shared by other entities, leading to more reliable long-term predictions. Specifically, our method allows a black-box treatment for modelling the hazard function which is often a pre-defined parametric form in typical survival analysis. The key idea of our approach is to model the hazard function as a nonparameteric function of the history. The same precedent information from other entities is embedded to a stitched vector for LT-RNN to automatically learn a representation of the long-term hazard function. We apply our model to the proactive maintenance problem using a large dataset from a water utility in Australia.

Missing values in real world datasets are a common issue. Handling missing values is one of the most key aspects in data mining, as it can seriously impact the performance of predictive models. In this paper we proposed a unified Boosting framework that consolidates model construction and missing value handling. At each Boosting iteration, weights are assigned to both the samples and features. The sample weights make difficult samples become the learning focus, while the feature weights enable critical features to be compensated by less critical features when they are unavailable. A weak classifier that abstains (i.e, produce no prediction when required feature value is missing) is learned on a data subset determined by the feature weights. Experimental results demonstrate the efficacy and robustness of the proposed method over existing Boosting algorithms.

We present a new summary evaluation approach that does not require human model summaries. Our approach exploits the compositional capabilities of corpus-based and lexical resource-based word embeddings to develop the features reflecting coverage, diversity, informativeness, and coherence of summaries. The features are then used to train a learning model for predicting the summary content quality in the absence of gold models. We evaluate the proposed metric in replicating the human assigned scores for summarization systems and summaries on data from query-focused and update summarization tasks in TAC 2008 and 2009. The results show that our feature combination provides reliable estimates of summary content quality when model summaries are not available.

Locations of social media users are important to many applications such as rapid disaster response, targeted advertisement, and news recommendation.However, many users do not share their exact geographical coordinates due to reasons such as privacy concerns.The lack of explicit location information has motivated a growing body of research in recent years looking at different automatic ways of determining the user's primary location.In this paper, we propose a unified user geolocation method which relies on a fusion of neural networks.Our joint model incorporates different types of available information including tweet text, user network, and metadata to predict users' locations.Moreover, we utilize a bidirectional LSTM network augmented with an attention mechanism to identify the most location indicative words in textual content of tweets.The experiments demonstrate that our approach achieves state-of-the-art performance over two Twitter benchmark geolocation datasets.We also conduct an ablation study to evaluate the contribution of each type of information in user geolocation performance.

ROUGE is one of the first and most widely used evaluation metrics for text summarization.However, its assessment merely relies on surface similarities between peer and model summaries.Consequently, ROUGE is unable to fairly evaluate summaries including lexical variations and paraphrasing.We propose a graph-based approach adopted into ROUGE to evaluate summaries based on both lexical and semantic similarities.Experiment results over TAC AESOP datasets show that exploiting the lexico-semantic similarity of the words used in summaries would significantly help ROUGE correlate better with human judgments.

Urbanization is a global trend that we have all witnessed in the past decades. It brings us both opportunities and challenges. On the one hand, urban system is one of the most sophisticated social-economic systems that is responsible for efficiently providing supplies meeting the demand of residents in various of domains, e.g., dwelling, education, entertainment, healthcare, etc. On the other hand, significant diversity and inequality exist in the development patterns of urban systems, which makes urban data analysis difficult. Different urban regions often exhibit diverse urbanization patterns and provide distinct urban functions, e.g., commercial and residential areas offer significantly different urban functions. It is desired to develop the data analytic capabilities for discovering the underlying cross-domain urbanization patterns, clustering urban regions based on their function similarity and predicting region popularity in specified domains. Previous studies in the urban data analysis area often just focus on individual domains and rarely consider cross-domain urban development patterns hidden in different urban regions. In this paper, we propose the infinite urbanization process (IUP) model for simultaneous urban region function discovery and region popularity prediction. The IUP model is a generative Bayesian nonparametric process that is capable of describing a potentially infinite number of urbanization patterns. It is developed within the supervised topic modelling framework and is supported by a novel hierarchical spatial distance dependent Bayesian nonparametric prior over the spatial region partition space. The empirical study conducted on the real-world datasets shows promising outcome compared with the state-of-the-art techniques.

Identifying the discriminative characteristic of a query is important for image retrieval. For retrieval without human interaction, such characteristic is usually obtained by average query expansion (AQE) or its discriminative variant (DQE) learned from pseudo-examples online, among others. In this paper, we propose a new query expansion method to further improve the above ones. The key idea is to learn a "unique'' discriminative characteristic for each database image, in an offline manner. During retrieval, the characteristic of a query is obtained by aggregating the unique characteristics of the query-relevant images collected from an initial retrieval result. Compared with AQE which works in the original feature space, our method works in the space of the unique characteristics of database images, significantly enhancing the discriminative power of the characteristic identified for a query. Compared with DQE, our method needs neither pseudo-labeled negatives nor the online learning process, leading to more efficient retrieval and even better performance. The experimental study conducted on seven benchmark datasets verifies the considerable improvement achieved by the proposed method, and also demonstrates its application to the state-of-the-art diffusion-based image retrieval.

Soft-attention is regarded as one of the representative methods for image captioning. Based on the end-to-end CNN-LSTM framework, it tries to link the relevant visual information on the image with the semantic representation in the text (i.e. captioning) for the first time. In recent years, there are several state-of-the-art methods published, which are motivated by this approach and include more elegant fine-tune operation. However, due to the constraints of CNN architecture, the given image is only segmented to fixed-resolution grid at a coarse level. The overall visual feature created for each grid cell indiscriminately fuses all inside objects and/or their portions. There is no semantic link among grid cells, although an object may be segmented into different grid cells. In addition, the large-area stuff (e.g. sky and beach) cannot be represented in the current methods. To tackle the problems above, this paper proposes a new model based on the FCN-LSTM framework which can segment the input image into a fine-grained grid. Moreover, the visual feature representing each grid cell is contributed only by the principal object or its portion in the corresponding cell. By adopting the pixel-wise labels (i.e. semantic segmentation), the visual representations of different grid cells are correlated to each other. In this way, a mechanism of fine-grained and semantic-guided visual attention is created, which can better link the relevant visual information with each semantic meaning inside the text through LSTM. Without using the elegant fine-tune, the comprehensive experiments show promising performance consistently across different evaluation metrics.

Geolocated social media data provide a powerful source of information about places and regional human behavior. Because only a small amount of social media data have been geolocation‐annotated, inference techniques play a substantial role to increase the volume of annotated data. Conventional research in this area has been based on the text content of posts from a given user or the social network of the user, with some recent crossovers between the text‐ and network‐based approaches. This paper proposes a novel approach to categorize highly‐mentioned users (celebrities) into Local and Global types, and consequently use Local celebrities as location indicators. A label propagation algorithm is then used over the refined social network for geolocation inference. Finally, we propose a hybrid approach by merging a text‐based method as a back‐off strategy into our network‐based approach. Empirical experiments over three standard Twitter benchmark data sets demonstrate that our approach outperforms state‐of‐the‐art user geolocation methods.

This paper investigates End-User Development (EUD) for interactive data-analytic interfaces-building upon the ideas of making machine learning transparent. The research is carried out in a business operation environment (water pipe failure prediction in our case) motivated to integrate advanced analytics into decision-making processes of an urban Internet of Things (IoT) concept. We explore effects of revealing uncertainty and correlation on user confidence in a data-driven decision making scenario. It was found that user confidence varied significantly amongst various user groups when different machine learning models were displayed with/without supplementary information. Galvanic Skin Response (GSR) signals were analyzed and shown as reasonable indices for predicting user confidence levels. Supplementary data visualizations (of inherent uncertainty and correlation in data) contributed to explicability principles while GSR indexing added towards correctibility principles. We recommend transparent machine learning as the key to effective EUD for interactive data analytics.

As commercial pen-centric systems proliferate, they create a parallel need for analytic techniques based on dynamic writing. Within educational applications, recent empirical research has shown that signal-level features of students’ writing, such as stroke distance, pressure and duration, are adapted to conserve total energy expenditure as they consolidate expertise in a domain. The present research examined how accurately three different machine-learning algorithms could automatically classify users’ domain expertise based on signal features of their writing, without any content analysis. Compared with an unguided machine-learning classification accuracy of 71%, hybrid methods using empirical-statistical guidance correctly classified 79–92% of students by their domain expertise level. In addition to improved accuracy, the hybrid approach contributed a causal understanding of prediction success and generalization to new data. These novel findings open up opportunities to design new automated learning analytic systems and student-adaptive educational technologies for the rapidly expanding sector of commercial pen systems.

This paper is concerned with the one class classification(OCC) problem. By introducing the vector-valued function with regularizations in Y-valued Reproducing Hilbert Kernel Space(RHKS), we build an unsupervised classifier and discover the outliers and inliers simultaneously. Manifold regularization is employed to preserve the local similarity of data in input space. Experimental results of the proposed and comparing methods on OCC data sets demonstrate the performance of the proposed algorithm.

Customer behavior modeling is important for businesses in order to understand, attract and retain customers. It is critical that the models are able to track the dynamics of customer behavior over time. We propose FC-CSM, a Customer Segmentation Model based on a Fragmentation-Coagulation process, which can track the evolution of customer segmentation, including the splitting and merging of customer groups. We conduct a case study using transaction data from a major Australian supermarket chain, where we: 1) show that our model achieves high fitness of purchase rate, outperforming models using mixture of Poisson processes; 2) compare the impact of promotions on customers for different products; and 3) track how customer groups evolve over time and how individual customers shift across groups. Our model provides valuable information to stakeholders about the different types of customers, how they change purchase behavior, and which customers are more receptive to promotion campaigns.

Safety-oriented visualization is one of significant approaches to gain insights from time-spatial data while neural net currently serves as a decent way to perform machine learning in data mining industry. This paper proposes a visual analytics pipeline for trajectory data enabling better understanding movements pattern of people using Neural Network as back-end and other visualization techniques as front-end for gaining information of preferences of attractions, similarities of groups, popularities of attractions and pattern of movement flow. Such understandings help to address the management issue by extracting the outstanding features to detect abnormal pattern such as detection of crime and predicting overall movements, and so on. Successfully dealing with those issues would have significant improvements of entire management of public facility such as parks and transportation.

Trust is a key factor affecting the way people rely on automated systems. On the other hand, system performance has comprehensive implications on a user's trust variations. This paper examines systems of varied levels of accuracy, in order to reveal the relationship between system performance, a user's trust and reliance on the system. In particular, it is identified that system failures have a stronger effect on trust than system successes. We also describe how patterns of trust change according to a number of consecutive system failures or successes. Importantly, we show that increasing user familiarity with the system decreases the rate of trust change, which provides new insights on the development of user trust. Finally, our analysis established a correlation between a user's reliance on a system and their trust level. Combining all these findings can have important implications in general system design and implementation, by predicting how trust builds and when it stabilizes, as well as allowing for indirectly reading a user's trust in real time based on system reliance.

Traffic congestion is a condition of a segment in the road network where the traffic demand is greater than the available road capacity. The detection of unusual traffic patterns including congestions is a significant research problem in the data mining and knowledge discovery community. However, to the best of our knowledge, the discovery of propagations, or causal interactions among detected traffic congestions has not been appropriately investigated before. In this research, we introduce algorithms which construct causality trees from congestions and estimate their propagation probabilities based on temporal and spatial information of the congestions. Frequent sub-structures of these causality trees reveal not only recurring interactions among spatio-temporal congestions, but potential bottlenecks or flaws in the designs of existing traffic networks. Our algorithms have been validated by experiments on a travel time data set recorded from an urban road network.

Experiencing high cognitive load during complex and demanding tasks results in performance reduction, stress, and errors. However, these could be prevented by a system capable of constantly monitoring users’ cognitive load fluctuations and adjusting its interactions accordingly. Physiological data and behaviors have been found to be suitable measures of cognitive load and are now available in many consumer devices. An advantage of these measures over subjective and performance-based methods is that they are captured in real time and implicitly while the user interacts with the system, which makes them suitable for real-world applications. On the other hand, emotion interference can change physiological responses and make accurate cognitive load measurement more challenging. In this work, we have studied six galvanic skin response (GSR) features in detection of four cognitive load levels with the interference of emotions. The data was derived from two arithmetic experiments and emotions were induced by displaying pleasant and unpleasant pictures in the background. Two types of classifiers were applied to detect cognitive load levels. Results from both studies indicate that the features explored can detect four and two cognitive load levels with high accuracy even under emotional changes. More specifically, rise duration and accumulative GSR are the common best features in all situations, having the highest accuracy especially in the presence of emotions.

Stochastic partition processes for exchangeable graphs produce axis-aligned blocks on a product space. In relational modeling, the resulting blocks uncover the underlying interactions between two sets of entities of the relational data. Although some flexible axis-aligned partition processes, such as the Mondrian process, have been able to capture complex interacting patterns in a hierarchical fashion, they are still in short of capturing dependence between dimensions. To overcome this limitation, we propose the Ostomachion process (OP), which relaxes the cutting direction by allowing for oblique cuts. The partitions generated by an OP are convex polygons that can capture inter-dimensional dependence. The OP also exhibits interesting properties: 1) Along the time line the cutting times can be characterized by a homogeneous Poisson process, and 2) on the partition space the areas of the resulting components comply with a Dirichlet distribution. We can thus control the expected number of cuts and the expected areas of components through hyper-parameters. We adapt the reversible-jump MCMC algorithm for inferring OP partition structures. The experimental results on relational modeling and decision tree classification have validated the merit of the OP.

Many natural and social phenomena can be modeled by interaction point processes (IPPs) (Diggle et al. 1994), stochastic point processes considering the interaction between points. In this paper, we propose the infinite branching model (IBM), a Bayesian statistical model that can generalize and extend some popular IPPs, e.g., Hawkes process (Hawkes 1971; Hawkes and Oakes 1974). It treats IPP as a mixture of basis point processes with the aid of a distance dependent prior over branching structure that describes the relationship between points. The IBM can estimate point event intensity, interaction mechanism and branching structure simultaneously. A generic Metropolis-within-Gibbs sampling method is also developed for model parameter inference. The experiments on synthetic and real-world data demonstrate the superiority of the IBM.

Multi Sentence Compression (MSC) is of great value to many real world applications, such as guided microblog summarization, opinion summarization and newswire summarization. Recently, word graph-based approaches have been proposed and become popular in MSC. Their key assumption is that redundancy among a set of related sentences provides a reliable way to generate informative and grammatical sentences. In this paper, we propose an effective approach to enhance the word graph-based MSC and tackle the issue that most of the state-of-the-art MSC approaches are confronted with: i.e., improving both informativity and grammaticality at the same time. Our approach consists of three main components: (1) a merging method based on Multiword Expressions (MWE); (2) a mapping strategy based on synonymy between words; (3) a re-ranking step to identify the best compression candidates generated using a POS-based language model (POS-LM). We demonstrate the effectiveness of this novel approach using a dataset made of clusters of English newswire sentences. The observed improvements on informativity and grammaticality of the generated compressions show an up to 44% error reduction over state-of-the-art MSC systems.

Structural health monitoring is a condition-based technology to monitor infrastructure using sensing systems. Since we usually only have data associated with the healthy state of a structure, one-class approaches are more practical. However, tuning the parameters for one-class techniques (like one-class Support Vector Machines) still remains a relatively open and difficult problem. Moreover, in structural health monitoring, data are usually multi-way, highly redundant and correlated, which a matrix-based two-way approach cannot capture all these relationships and correlations together. Tensor analysis allows us to analyse the multi-way vibration data at the same time. In our approach, we propose the use of tensor learning and support vector machines with artificial negative data generated by density estimation techniques for damage detection, localization and estimation in a one-class manner. The artificial negative data can help tuning SVM parameters and calibrating probabilistic outputs, which is not possible to do with one-class SVM. The proposed method shows promising results using data from laboratory-based structures and also with data collected from the Sydney Harbour Bridge, one of the most iconic structures in Australia. The method works better than the one-class approach and the approach without using tensor analysis.

The supermarkets often use sales promotions to attract customers and create brand loyalty. They would often like to know if their promotions are effective for various customers, so that better timing and more suitable rate can be planned in the future. Given a transaction data set collected by an Australian national supermarket chain, in this paper we conduct a case study aimed at discovering customers' long-term purchase patterns, which may be induced by preference changes, as well as short-term purchase patterns, which may be induced by promotions. Since purchase events of individual customers may be too sparse to model, we propose to discover a number of latent purchase patterns from the data. The latent purchase patterns are modeled via a mixture of non-homogeneous Poisson processes where each Poisson intensity function is composed by long-term and short-term components. Through the case study, 1) we validate that our model can accurately estimate the occurrences of purchase events; 2) we discover easy-to-interpret long-term gradual changes and short-term periodic changes in different customer groups; 3) we identify the customers who are receptive to promotions through the correlation between behavior patterns and the promotions, which is particularly worthwhile for target marketing.

When making clinical decisions, practitioners need to rely on the most relevant evidence available. However, accessing a vast body of medical evidence and confronting with the issue of information overload can be challenging and time consuming. This paper proposes an effective summarizer for medical evidence by utilizing both UMLS and WordNet. Given a clinical query and a set of relevant abstracts, our aim is to generate a fluent, well-organized, and compact summary that answers the query. Analysis via ROUGE metrics shows that using WordNet as a general-purpose lexicon helps to capture the concepts not covered by the UMLS Metathesaurus, and hence significantly increases the performance. The effectiveness of our proposed approach is demonstrated by conducting a set of experiments over a specialized evidence-based medicine (EBM) corpus - which has been gathered and annotated for the purpose of biomedical text summarization.

The correlation between events is ubiquitous and important for temporal events modelling. In many cases, the correlation exists between not only events' emitted observations, but also their arrival times. State space models (e.g., hidden Markov model) and stochastic interaction point process models (e.g., Hawkes process) have been studied extensively yet separately for the two types of correlations in the past. In this paper, we propose a Bayesian nonparametric approach that considers both types of correlations via unifying and generalizing hidden semi-Markov model and interaction point process model. The proposed approach can simultaneously model both the observations and arrival times of temporal events, and determine the number of latent states from data. A Metropolis-within-particle-Gibbs sampler with ancestor resampling is developed for efficient posterior inference. The approach is tested on both synthetic and real-world data with promising outcomes.

Trust plays an important role in various user-facing systems and applications. It is particularly important in the context of decision support systems, where the system's output serves as one of the inputs for the users' decision making processes. In this work, we study the dynamics of explicit and implicit user trust in a simulated automated quality monitoring system, as a function of the system accuracy. We establish that users correctly perceive the accuracy of the system and adjust their trust accordingly.

BACKGROUND: Direct volume rendering is one of flexible and effective approaches to inspect large volumetric data such as medical and biological images. In conventional volume rendering, it is often time consuming to set up a meaningful illumination environment. Moreover, conventional illumination approaches usually assign same values of variables of an illumination model to different structures manually and thus neglect the important illumination variations due to structure differences. RESULTS: We introduce a novel illumination design paradigm for volume rendering on the basis of topology to automate illumination parameter definitions meaningfully. The topological features are extracted from the contour tree of an input volumetric data. The automation of illumination design is achieved based on four aspects of attenuation, distance, saliency, and contrast perception. To better distinguish structures and maximize illuminance perception differences of structures, a two-phase topology-aware illuminance perception contrast model is proposed based on the psychological concept of Just-Noticeable-Difference. CONCLUSIONS: The proposed approach allows meaningful and efficient automatic generations of illumination in volume rendering. Our results showed that our approach is more effective in depth and shape depiction, as well as providing higher perceptual differences between structures.

Water pipe failures can cause significant economic and social costs, hence have become the primary challenge to water utilities. In this paper, we propose a Bayesian nonparametric approach, namely the Dirichlet process mixture of hierarchical beta process model, for water pipe failure prediction. It can select high-risk pipes for physical condition assessment, thereby preventing disastrous failures proactively.

Prevailing social norms prohibit interrupting another person when they are speaking. In this research, simultaneous speech was investigated in groups of students as they jointly solved math problems and peer tutored one another. Analyses were based on the Math Data Corpus, which includes ground-truth performance coding and speech transcriptions. Simultaneous speech was elevated 120-143% during the most productive phase of problem solving, compared with matched intervals. It also was elevated 18-37% in students who were domain experts, compared with non-experts. Qualitative analyses revealed that experts differed from non-experts in the function of their interruptions. Analysis of these functional asymmetries produced nine key behaviors that were used to identify the dominant math expert in a group with 95-100% accuracy in three minutes. This research demonstrates that overlapped speech is a marker of group problem-solving progress and domain expertise. It provides valuable information for the emerging field of learning analytics.

Stochastic partition processes in a product space play an important role in modeling relational data. Recent studies on the Mondrian process have introduced more flexibility into the block structure in relational models. A side-effect of such high flexibility is that, in data sparsity scenarios, the model is prone to overfit. In reality, relational entities are always associated with meta information, such as user profiles in a social network. In this paper, we propose a metadata dependent Mondrian process (MDMP) to incorporate meta information into the stochastic partition process in the product space and the entity allocation process on the resulting block structure. MDMP can not only encourage homogeneous relational interactions within blocks but also discourage meta-label diversity within blocks. Regularized by meta information, MDMP becomes more robust in data sparsity scenarios and easier to converge in posterior inference. We apply MDMP to link prediction and rating prediction and demonstrate that MDMP is more effective than the baseline models in prediction accuracy with a more parsimonious model structure.

This article presents a framework of adaptive, measurable decision making for Multiple Attribute Decision Making (MADM) by varying decision factors in their types, numbers, and values. Under this framework, decision making is measured using physiological sensors such as Galvanic Skin Response (GSR) and eye-tracking while users are subjected to varying decision quality and difficulty levels. Following this quantifiable decision making, users are allowed to refine several decision factors in order to make decisions of high quality and with low difficulty levels. A case study of driving route selection is used to set up an experiment to test our hypotheses. In this study, GSR features exhibit the best performance in indexing decision quality. These results can be used to guide the design of intelligent user interfaces for decision-related applications in HCI that can adapt to user behavior and decision-making performance.

A recent study on collective attention in Twitter shows that an epidemic spreading of hashtags is predominantly driven by external factors. We extend a time-series form of susceptible-infectious-recovered (SIR) model to monitor microblog emerging outbreaks by considering both endogenous and exogenous drivers. In addition, we adopt partially labeled Dirichlet allocation (PLDA) model to generate both background latent topics and hashtag topics. It overcomes the problem of small available samples in hashtag analysis by including related but unlabeled tweets through inference. We standardize hashtag topic contagiousness measure as the estimated effective-reproduction-number R derived from epidemiology. It is obtained by Bayesian parameter estimation. Guided by R, one can profile and categorize emerging topics, and generate alerts on potential outbreaks. Experiment results confirm the effectiveness of this approach.

Many prevalent multi-class classification approaches can be unified and generalized by the output coding framework which usually consists of three phases: (1) coding, (2) learning binary classifiers, and (3) decoding. Most of these approaches focus on the first two phases and predefined distance function is used for decoding. In this paper, however, we propose to perform learning in coding space for more adaptive decoding, thereby improving overall performance. Ramp loss is exploited for measuring multi-class decoding error. The proposed algorithm has uniform stability. It is insensitive to data noises and scalable with large scale datasets. Generalization error bound and numerical results are given with promising outcomes.

An intelligent adaptable system, aware of a user’s experienced cognitive load, may help improve performance in complex, time-critical situations by dynamically deploying more appropriate output strategies to reduce cognitive load. However, measuring a user’s cognitive load robustly, in real-time is not a trivial task. Many research studies have attempted to assess users’ cognitive load using different measurements, but these are often unsuitable for deployment in real-life applications due to high intrusiveness. Relatively novel linguistic behavioral features as potential indices of user’s cognitive load is proposed. These features may be collected implicitly and nonintrusively supporting real-time assessment of users’ cognitive load and accordingly allowing adaptive usability evaluation and interaction. Results from a laboratory experiment show significantly different linguistic patterns under different task complexities and cognitive load levels. Implications of the research for adaptive interaction are also discussed, that is, how the cognitive load measurement-based approach could be used for user interface evaluation and interaction design improvement.

Cognitive load (CL) is experienced during critical tasks and also while engaged emotional states are induced either by the task itself or by extraneous experiences. Emotions irrelevant to the working memory representation may interfere with the processing of relevant tasks and can influence task performance and behavior, making the accurate detection of CL from nonverbal information challenging. This paper investigates automatic CL detection from facial features, physiology and task performance under affective interference. Data were collected from participants (n=20) solving mental arithmetic tasks with emotional stimuli in the background, and a combined classifier was used for detecting CL levels. Results indicate that the face modality for CL detection was more accurate under affective interference, whereas physiology and task performance were more accurate without the affective interference. Multimodal fusion improved detection accuracies, but it was less accurate under affective interferences. More specifically, the accuracy decreased with an increasing intensity of emotional arousal.

Supervised machine learning techniques have been applied to multilabel image classification problems with tremendous success. Despite disparate learning mechanisms, their performances heavily rely on the quality of training images. However, the acquisition of training images requires significant efforts from human annotators. This hinders the applications of supervised learning techniques to large scale problems. In this paper, we propose a high-order label correlation driven active learning (HoAL) approach that allows the iterative learning algorithm itself to select the informative example-label pairs from which it learns so as to learn an accurate classifier with less annotation efforts. Four crucial issues are considered by the proposed HoAL: 1) unlike binary cases, the selection granularity for multilabel active learning need to be fined from example to example-label pair; 2) different labels are seldom independent, and label correlations provide critical information for efficient learning; 3) in addition to pair-wise label correlations, high-order label correlations are also informative for multilabel active learning; and 4) since the number of label combinations increases exponentially with respect to the number of labels, an efficient mining method is required to discover informative label correlations. The proposed approach is tested on public data sets, and the empirical results demonstrate its effectiveness.

Mental workload is an important factor during writing, which may affect the writing efficiency and user experience. This paper aims at a method to classify the mental workload levels during writing process, via examination of online writing features in a two-stage algorithm structure. At the first stage, a curvature tracking method is applied to the handwriting script, to examine the curvature for individual writing points. Then a selection process allocates writing points into subsets, each corresponding to one curvature span. The second stage extracts velocity features, used to characterize mental workload, from points in each curvature span. A Parzen-window classifier is applied on velocity features from each curvature span. The classification decisions from individual classifiers are fused with a selective voting scheme for the overall mental workload classification decision. This paper finally discusses the classification accuracy for three mental workload levels and compares it with previous work.

A day in the life of a user can be segmented into a series of tasks: a user begins a task, becomes loaded perceptually and cognitively to some extent by the objects and mental challenge that comprise that task, then at some point switches or is distracted to a new task, and so on. Understanding the contextual task characteristics and user behavior in interaction can benefit the development of intelligent systems to aid user task management. Applications that aid the user in one way or another have proliferated as computing devices become more and more of a constant companion. However, direct and continuous observations of individual tasks in a naturalistic context and subsequent task analysis, for example the diary method, have traditionally been a manual process. We propose a method for automatic task analysis system, which monitors the user's current task and analyzes it in terms of the task transition, and perceptual and cognitive load imposed by the task. An experiment was conducted in which participants were required to work continuously on groups of three sequential tasks of different types. Three classes of eye activity, namely pupillary response, blink and eye movement, were analyzed to detect the task transition and non-transition states, and to estimate three levels of perceptual load and three levels of cognitive load every second to infer task characteristics. This paper reports statistically significant classification accuracies in all cases and demonstrates the feasibility of this approach for task monitoring and analysis.

Pupillary response is a popular physiological index of cognitive workload that can be used for design and evaluation of adaptive interface in various areas of human-computer interaction (HCI) research. However, in practice various confounding factors unrelated to workload, including changes of luminance condition and emotional arousal might degrade pupillary response based workload measures such as commonly used mean pupil diameter. This work investigates pupillary response as a cognitive workload measure under the influence of such confounding factors. Video-based eye tracker is used to record pupillary response during arithmetic tasks under luminance and emotional changes. Machine learning based feature selection and classification techniques are proposed to robustly index cognitive workload based on pupillary response even with the influence of noisy factors unrelated to workload.

Object detection is an important and challenging problem in the field of computer vision. Classical object detection approaches such as background subtraction and saliency detection do not require manual collection of training samples, but can be easily affected by noise factors, such as luminance changes and cluttered background. On the other hand, supervised learning based approaches such as Boosting and SVM usually have robust performance, but require substantial human effort to collect and label training samples. This study aims to combine the comparative advantages of both kinds of approaches, and its contributions are two-fold: (i) a weakly supervised approach for object detection, which does not require manual collection and labelling of training samples; (ii) an extension of Boosting algorithm denoted as Soft-Label Boosting, which is able to employ training samples with soft (probabilistic) labels instead of hard (binary) labels. Experimental results show that the proposed weakly supervised approach outperforms the state-of-the-art, and even achieves comparable performance to supervised approaches.

A novel bayesian non-parametric method for tracking is proposed in this paper. The foreground appearance distribution is modeled by unbounded mixtures controlled through a Bayesian non-parametric process. Two posterior inference strategies are provided: Gibbs sampling and sequential importance sampling. Both of these two sampling strategy benefits from the conjugate prior/posterior pairs by factorizing the joint posterior distributions. Once the mixture model is obtained/updated, the similarities/probablity of each observations assigned to this mixture model could be easily calculated. In model matching/verification, the Kullback-Leibler divergence and texture information is adopted for verification purpose. The robustness of our methods is demonstrated by the experiments.

High cognitive load arises from complex time and safety-critical tasks, for example, mapping out flight paths, monitoring traffic, or even managing nuclear reactors, causing stress, errors, and lowered performance. Over the last five years, our research has focused on using the multimodal interaction paradigm to detect fluctuations in cognitive load in user behavior during system interaction. Cognitive load variations have been found to impact interactive behavior: by monitoring variations in specific modal input features executed in tasks of varying complexity, we gain an understanding of the communicative changes that occur when cognitive load is high. So far, we have identified specific changes in: speech, namely acoustic, prosodic, and linguistic changes; interactive gesture; and digital pen input, both interactive and freeform. As ground-truth measurements, galvanic skin response, subjective, and performance ratings have been used to verify task complexity. The data suggest that it is feasible to use features extracted from behavioral changes in multiple modal inputs as indices of cognitive load. The speech-based indicators of load, based on data collected from user studies in a variety of domains, have shown considerable promise. Scenarios include single-user and team-based tasks; think-aloud and interactive speech; and single-word, reading, and conversational speech, among others. Pen-based cognitive load indices have also been tested with some success, specifically with pen-gesture, handwriting, and freeform pen input, including diagraming. After examining some of the properties of these measurements, we present a multimodal fusion model, which is illustrated with quantitative examples from a case study. The feasibility of employing user input and behavior patterns as indices of cognitive load is supported by experimental evidence. Moreover, symptomatic cues of cognitive load derived from user behavior such as acoustic speech signals, transcribed text, digital pen trajectories of handwriting, and shapes pen, can be supported by well-established theoretical frameworks, including O'Donnell and Eggemeier's workload measurement [1986] Sweller's Cognitive Load Theory [Chandler and Sweller 1991], and Baddeley's model of modal working memory [1992] as well as McKinstry et al.'s [2008] and Rosenbaum's [2005] action dynamics work. The benefit of using this approach to determine the user's cognitive load in real time is that the data can be collected implicitly that is, during day-to-day use of intelligent interactive systems, thus overcomes problems of intrusiveness and increases applicability in real-world environments, while adapting information selection and presentation in a dynamic computer interface with reference to load.

This paper presents a longitudinal study that explores the combined effect of cognitive load and expertise on the use of a scratchpad. Our results confirm that such cognitive support benefits users under high cognitive load through visual aid, perceptual motor use and helps to improve meaningful learning and successful problem solving. Indeed, we found significant changes in stroke frequency affected by cognitive load, which we believe are caused by the scratchpad essentially augmenting or extending working memory capacity. However, the discrepancy between stroke frequencies under low and high load is reduced with expertise. These results indicate that pen stroke frequency, which can be automated with electronic devices, could be used as an indicator of cognitive load, or conversely, of expertise level.

The measurement of a user's mental effort is a problem whose solutions may have important applications to adaptive interfaces and interface evaluation. Previous studies have empirically shown links between eye activity and mental effort; however these have usually investigated only one class of eye activity on tasks atypical of HCI. This paper reports on research into eight eye activity based features, spanning eye blink, pupillary response and eye movement information, for real time mental effort measurement. Results from an experiment conducted using a computer-based training system show that the three classes of eye features are capable of discriminating different cognitive load levels. Correlation analysis between various pairs of features suggests that significant improvements in discriminating different effort levels can be made by combining multiple features. This shows an initial step towards a real-time cognitive load measurement system in human-computer interaction.

This paper examines several writing features for the evaluation of cognitive load. Our analysis is focused on writing features within and between written strokes, including writing pressure, writing velocity, stroke length and inter-stroke movements. Based on a study of 20 subjects performing a sentence composition task, the reported findings reveal that writing pressure and writing velocity information are very good indicators of cognitive load. A stroke selection threshold was investigated for constraining the feature extraction to long strokes, which resulted in a small further improvement. Differing from most previous research investigating cognitive load during writing based on task performance criteria, this work proposes a new approach to cognitive load measurement using writing dynamics, with the potential to allow new or improve existing handwriting interfaces.

An adaptive interaction system, which is aware of the users' current cognitive load, can change its response, presentation and interaction flow to improve users' experience and their task performance. In this paper, we propose a novel speech content analysis approach for measuring users' cognitive load, based on their language and dialogue complexity. We have analysed the transcribed speech of operators working in computerized incident control rooms and involved in highly complex bushfire management tasks in Australia. The resulting patterns of language complexity show significant differences between the speech from cognitively low load and high load tasks. We also discuss the value of using this approach of cognitive load measurement for user interface evaluation and interaction design improvement.

The current automatic cognitive load measurement system based on MFCC and prosodic features does not take into account phase based speech information. This paper aims to improve the performance of the baseline system by introducing phase based features into the system. The additional features proposed are group delay features, all-pole model based FM features and zero crossing count based FM features. Decrease in performance is observed when phase based features are considered individually or when concatenated with baseline features. However, significant performance improvement is observed when group delay features are fused with baseline features using linear combination score level fusion.

Most existing duration modeling approaches facilitates phone recognizer and require manually annotated corpus to train the segmentation models, which is usually cost- and time-consuming. In this paper, a novel duration modeling approach is proposed, which does not require phone recognizer/annotated training data, and facilitates fast computation of language identification. In this approach, the segmentation is implemented by using articulatory features like voicing status. A pair of connected unvoiced and voiced segments is considered as the unit, and the duration of each segment is normalized for each utterance and then quantized into 20 discrete ranges. The ranges of units are later considered as symbol sequences and are modeled by n-gram models, to capture the temporal pattern, which is hypothesized to vary in different languages. The experiments based on the NIST LRE 2005 tasks show a relative 19.7% EER improvement by introducing the proposed duration modeling-based system into a fusion system containing two GMM-UBM based acoustic systems using MFCC and pitch+intensity features.

In a multimodal interface, a user can use multiple modalities, such as speech, gesture, and eye gaze etc., to communicate with a system. As a critical component in a multimodal interface, multimodal input fusion explores the ways to effectively interpret the combined semantic interpretation of user's multimodal inputs. Although multimodal inputs may contain spare information, few multimodal input fusion approaches have tackled how to deal with spare information in multimodal inputs. This paper proposes a novel multimodal input fusion approach to flexibly skip spare information in multimodal inputs and derive semantic interpretation of them. The evaluation about the proposed approach confirms that the approach makes human-computer interaction more natural and smooth.

Hierarchical language identification (HLID) is a novel framework for combining multiple features or primary systems in language identification. In this paper, several key components of HLID are investigated and developed. Crossing likelihood ratio and Kullback-Leibler distance measures are introduced for faster and more accurate clustering. A novel feature selection scheme based on fusion is proposed to incorporate multiple features at each classification level. Further, a phone recognizer followed by language model (PRLM) system is introduced in addition to the other three acoustic systems to provide phonetic information. These proposed techniques improve the performance of HLID system to an EER of 6.3% on the NIST LRE 2003 30s task.

Monitoring cognitive load is important for the prevention of faulty errors in task-critical operations, and the development of adaptive user interfaces, to maintain productivity and efficiency in work performance. Speech, as an objective and non-intrusive measure, is a suitable method for monitoring cognitive load. Existing approaches for cognitive load monitoring are limited in speaker-dependent recognition and need manually labeled data. We propose a novel automatic, speaker-independent classification approach to monitor, in real-time, the person's cognitive load level by using speech features. In this approach, a Gaussian mixture model (GMM) based classifier is created with unsupervised training. Channel and speaker normalization are deployed for improving robustness. Different delta techniques are investigated for capturing temporal information. And a background model is introduced to reduce the impact of insufficient training data. The final system achieves 71.1% and 77.5% accuracy on two different tasks, each of which has three discrete cognitive load levels. This performance shows a great potential in real-world applications.

Although relatively neglected in auditory analysis, phase information plays an important role in human auditory intelligibility. This paper investigates a Frequency Modulation (FM) based feature and its contribution to a Language Identification (LID) system, using a Hierarchical LID framework. FM components represent the phase information of a given signal in an AM-FM model. In this paper, we extract a FM-based feature using a technique which produces consistent and continuous FM components, and build a LID system on this feature with GMM based modeling. The performance is improved by combining this system with existing MFCC, Prosody based systems and a PRLM system. When compared to the baseline system without integrating a FM-based system, the proposed Hierarchical LID system shows improvements. Additionally, the proposed system outperforms the GMM fusion-based system integrating the same four primary systems, showing that the Hierarchical LID framework is more effective in integrating additional features. Index Terms: frequency modulation, language identification, hierarchical classification, fusion

The ability to measure cognitive load level in real time is extremely useful for improving the efficiency of interfaces and contents delivering, especially when interfaces and contents get complex in a multimedia environment. Speech is highly suitable for measuring cognitive load due to its non-intrusive nature and ease of collection. In this paper, we investigated the patterns of prosodic features and confirmed it is relevant to cognitive load. We also explored varied classification techniques to capture those relevant patterns of speech features. Gaussian Mixture Model (GMM), Support Vector Machine (SVM), and a hybrid SVM-GMM based classifiers were investigated with MFCC and pitch features. Individual systems and a fusion based system were evaluated on two different task scenarios - reading comprehension and Stroop test. The SVM-GMM based system achieved the highest performance on both tasks and improved the accuracy of three levels classification to 75.6% and 82.2%, respectively.

One of the key research issues in modern language identification (LID) research is how best to combine multiple approaches with different features. Existing statistical fusion techniques are popular but have serious limitations when development data is insufficient, since the data is used for training the statistical fuser. In this paper we compare existing fusion techniques for LID systems and propose an alternative to reduce this problem. By deriving the language-specific weighting directly from pair-wise LID performance, a novel weighting approach is introduced and implemented. Experiments on the NIST LRE 2003 task (CallFriend database) and OGI-TS databases demonstrate that the proposed weighting technique outperforms other recent fusion techniques when the available development data is limited.

Multimodal interfaces are known to be useful in map-based applications, and in complex, time-pressure based tasks. Cognitive load variations in such tasks have been found to impact multimodal behaviour. For example, users become more multimodal and tend towards semantic complementarity as cognitive load increases. The richness of multimodal data means that systems could monitor particular input features to detect experienced load variations. In this paper, we present our attempt to induce controlled levels of load and solicit natural speech and pen-gesture inputs. In particular, we analyse for these features in the pen gesture modality. Our experimental design relies on a map-based Wizard of Oz, using a tablet PC. This paper details analysis of pen-gesture interaction across subjects, and presents suggestive trends of increases in the degree of degeneration of pen-gestures in some subjects, and possible trends in gesture kinematics, when cognitive load increases.

Due to the limitation of single-level classification, existing fusion techniques experience difficulty in improving the performance of language identification when the number of languages and features are further increased. Given that the similarity of feature distribution between different languages may vary, we propose a novel hierarchical language identification framework with multi-level classification. In this approach, target languages are hierarchically clustered into groups according to the distance between them, models are trained both for individual languages and language groups, and classification is hierarchically done in multi-levels. This framework is implemented and evaluated in this paper, the results showing an relative 15.1% error-rate improvement in 30s case on OGI 10-language database compared to modern GMM fusion system.

This workshop addresses the issue of evaluating multimodal dialogue systems, and in particular the characteristics and interaction styles that are particularly effective for human-machine collaborative task performance.

Operators of traffic control rooms are often required to quickly respond to critical incidents using a complex array of multiple keyboards, mice, very large screen monitors and other peripheral equipment. To support the aim of finding more natural interfaces for this challenging application, this paper presents PEMMI (Perceptually Effective Multimodal Interface), a transport management system control prototype taking video-based manual gesture and speech recognition as inputs. A specific theme within this research is determining the optimum strategy for gesture input in terms of both single-point input selection and suitable multimodal feedback for selection. It has been found that users tend to prefer larger selection areas for targets in gesture interfaces, and tend to select within 44% of this selection radius. The minimum effective size for targets when using 'device-free' gesture interfaces was found to be 80 pixels (on a 1280x1024 screen). This paper also shows that feedback on gesture input via large screens is enhanced by the use of both audio and visual cues to guide the user's multimodal input. Audio feedback in particular was found to improve user response time by an average of 20% over existing gesture selection strategies for multimodal tasks.