With the increasing concerns about privacy and data regulations, federated learning (FL) has been emerging as a solution to train machine learning models collaboratively with non-exchangeable data from multiple clients. As a result of data locality, data is usually not identically or independently (non-IID) distributed across clients, and the non-IID property has long been the key challenge in FL. Furthermore, in real-world cross-silo scenarios, it is ubiquitous that clients are organizations owning private data from multiple domains internally, which exacerbates the non-IID issue. For example, in healthcare applications, each client (hospital) gathers data from patients with heterogeneous demographics. While previous works have made efforts to address the non-IID challenge across clients by assuming various relations among client-level data distributions and enabling personalized models at the client level, they ignore the internal data heterogeneity within each client or require explicit data domain indicators, which are hardly accessible in real-world data. Here, we propose Sample-Level Prototypical Federated Learning (SL-PFL) to bridge the gap. SL-PFL incorporates prototypical learning under the FL framework and provides a fine-grained personalized model for each data sample instead of learning one uniform model for all samples of each client. Meanwhile, it can be trained using data without ground-truth domain indicators. Experimental results demonstrate that our proposed method with sample-level personalized models outperforms existing FL methods with a global model or client-level personalized models on various real-world regression and classification tasks from weather, computer vision, and healthcare applications.

The rise of Large Visual-Language Models (LVLMs) has unlocked new possibilities for seamlessly integrating visual and textual information. However, their ability to interpret cartographic maps remains largely unexplored. In this paper, we introduce CartoMapQA, a benchmark specifically designed to evaluate LVLMs' understanding of cartographic maps through question-answering tasks. The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer. These tasks span key low-, mid- and high-level map interpretation skills, including symbol recognition, embedded information extraction, scale interpretation, and route-based reasoning. Our evaluation of both open-source and proprietary LVLMs reveals persistent challenges: models frequently struggle with map-specific semantics, exhibit limited geospatial reasoning, and are prone to Optical Character Recognition (OCR)-related errors. By isolating these weaknesses, CartoMapQA offers a valuable tool for guiding future improvements in LVLM architectures. Ultimately, it supports the development of models better equipped for real-world applications that depend on robust and reliable map understanding, such as navigation, geographic search, and urban planning. Our source code and data are openly available to the research community at: https://github.com/ungquanghuy-kddi/CartoMapQA.git

We hypothesize that optimal system responses emerge from adaptive strategies grounded in causal and counterfactual knowledge.Counterfactual inference allows us to create hypothetical scenarios to examine the effects of alternative system responses.We enhance this process through causal discovery, which identifies the strategies informed by the underlying causal structure that govern system behaviors.Moreover, we consider the psychological constructs and unobservable noises that might be influencing user-system interactions as latent factors.We show that these factors can be effectively estimated.We employ causal discovery to identify strategy-level causal relationships among user and system utterances, guiding the generation of personalized counterfactual dialogues.We model the user utterance strategies as causal factors, enabling system strategies to be treated as counterfactual actions.Furthermore, we optimize policies for selecting system responses based on counterfactual data.Our results using a real-world dataset on social good demonstrate significant improvements in persuasive system outcomes, with increased cumulative rewards validating the efficacy of causal discovery in guiding personalized counterfactual inference and optimizing dialogue policies for a persuasive dialogue system.

Most existing causal discovery methods rely on the assumption of no latent confounders, limiting their applicability in solving real-life problems. In this paper, we introduce a novel, versatile framework for causal discovery that accommodates the presence of causally-related hidden variables almost everywhere in the causal network (for instance, they can be effects of measured variables), based on rank information of covariance matrix over measured variables. We start by investigating the efficacy of rank in comparison to conditional independence and, theoretically, establish necessary and sufficient conditions for the identifiability of certain latent structural patterns. Furthermore, we develop a Rank-based Latent Causal Discovery algorithm, RLCD, that can efficiently locate hidden variables, determine their cardinalities, and discover the entire causal structure over both measured and hidden ones. We also show that, under certain graphical conditions, RLCD correctly identifies the Markov Equivalence Class of the whole latent causal graph asymptotically. Experimental results on both synthetic and real-world personality data sets demonstrate the efficacy of the proposed approach in finite-sample cases. Our code will be publicly available.

Multivariate long-term time series forecasting (MLTSF), applicable across various domains, has gained increasing research attention. Channel-independent (CI) models, including Linear and Transformer-based architectures, have recently achieved state-of-the-art (SOTA) performance for MLTSF. Notably, Linear models can deliver satisfactory forecasting performance even with just a single linear projection layer. However, we identify a limitation in this architecture: a single linear projection struggles to adequately capture the inter- and intra-variate heterogeneity in temporal patterns. Similarly, any complex models like Transformer-based models that use a single projection layer to generate final predictions, may face capacity bottlenecks. To overcome this, we propose the Mixture of Projection Experts (MoPE), which replaces the single linear projection with multiple projection branches, and employs a gate network to dynamically assign weights to each branch based on the input data. We applied MoPE to multiple SOTA models and evaluated it on nine real-world datasets. Results show that MoPE boosts forecasting accuracy by an average of 9.59%, demonstrating its effectiveness in mitigating the limitations of a single projection layer. Additionally, our experiments demonstrate that integrating our proposal into CI models enhances their generalization to unseen variates. Interpretability analysis also reveals MoPE's ability to disentangle different temporal patterns. Overall, our paper establishes MoPE as an effective solution for MLTSF tasks.

Linear causal models are important tools for modeling causal dependencies and yet in practice, only a subset of the variables can be observed. In this paper, we examine the parameter identifiability of these models by investigating whether the edge coefficients can be recovered given the causal structure and partially observed data. Our setting is more general than that of prior research—we allow all variables, including both observed and latent ones, to be flexibly related, and we consider the coefficients of all edges, whereas most existing works focus only on the edges between observed variables. Theoretically, we identify three types of indeterminacy for the parameters in partially observed linear causal models. We then provide graphical conditions that are sufficient for all parameters to be identifiable and show that some of them are provably necessary. Methodologically, we propose a novel likelihood-based parameter estimation method that addresses the variance indeterminacy of latent variables in a specific way and can asymptotically recover the underlying parameters up to trivial indeterminacy. Empirical studies on both synthetic and real-world datasets validate our identifiability theory and the effectiveness of the proposed method in the finite-sample regime.

The application of reinforcement learning (RL) involving interactions with individuals has grown significantly in recent years. These interactions, influenced by factors such as personal preferences and physiological differences, causally influence state transitions, ranging from health conditions in healthcare to learning progress in education. As a result, different individuals may exhibit different state-transition processes. Understanding individualized state-transition processes is essential for optimizing individualized policies. In practice, however, identifying these state-transition processes is challenging, as individual-specific factors often remain latent. In this paper, we establish the identifiability of these latent factors and introduce a practical method that effectively learns these processes from observed state-action trajectories. Experiments on various datasets show that the proposed method can effectively identify latent state-transition processes and facilitate the learning of individualized RL policies.

Sense of agency (SoA) is the perceived control over one’s actions and their consequences, and through this one feels responsible for the consequent outcomes in the world. We analyze the far-reaching implications of a two-pronged knowledge on SoA and its impact on human-AI interactions. We argue that although there are interesting research efforts for an AI to inherently possess SoA, they are still sparse, constrained in scope and present unclear immediate benefit to the design of AI-enabled systems. We also argue that the knowledge on how human SoA is affected by an AI that is perceived to possess a sense of control presents more immediate benefit to AI, in particular, to eliciting positive human attitudes toward AI. Third, and lastly, we argue that research efforts for an AI to adapt to the dynamic changes of human SoA are practically non-existent primarily due to the difficulty of modeling, inferring and adaptively responding to human SoA in complex natural settings. We proceed by first delving deep into the influential and recent theoretical underpinnings of SoA, and discuss its conceptual reach in different disciplines and how it is applied in real-world research. We organize a substantial part of our paper to put forward and elucidate our three argumentative points while supported by evidence in the literature.

We introduce a novel feature-based active learning approach that leverages leading feature selection methods from explainable AI (XAI) to elicit user corrections on both the classification label and the leading features contributing to the classification decision. Uniquely, our approach can handle such user corrections, which identify missing and superfluous features in a model-agnostic manner that can be applied to any supervised classification model. Our approach generates augmented samples in a fine-grained manner according to user corrections, and then adds them to the training data to teach relevant and irrelevant features. Our results show that our approach outperforms traditional active learning and a leading baseline in two domains, which includes improvement in classification performance from 0.572 to 0.637 given the same number of queries.

We posit that predicting sensor event sequence (SES) in a smart home can proactively support resident activities or recognize activities that have not been completed as intended and alert the resident. To realize this application, we propose a framework to support accurate SES prediction by leveraging online activity recognition. Our framework includes a novel method of applying a GPT2-based model, which is a sentence generation model, for SES prediction by taking advantage of the property that the relationship between ongoing activity and SES patterns is similar to the relationship between topic and word sequence patterns in NLP. We evaluated our method empirically using two real-world datasets where residents perform their usual daily activities. Our experimental results show the use of the GPT2-based model significantly improves the F1 value of SES prediction from 0.461 to 0.708 compared to the state-of-the-art method, and that using ongoing activity can further improve performance to 0.837. We found that the performance of the online activity recognition model required to achieve these SES predictions was about 80%, which could be achieved using simple feature engineering and modeling.

A major issue with real-time monitoring is to collect complete data. Hardware or software failures, network issues or, more frequently, time delays can disrupt such a collection. This results in having two versions of the same information: one in real-time but with potentially missing data, and the another, albeit complete, is delayed. Many works have studied how to handle missing data for classification and prediction. However, to the best of our knowledge, they do not consider how to leverage the delayed complete data to assist in learning the representation of real-time available data with missing values. This is despite the fact that the delayed complete data contain all the information (e.g., periodicities and trends). In this paper, we propose a framework to enhance the representation learning of the real-time available data by aligning the representation of past real-time but with missing data to that of past delayed but complete data. We test both a distance metric and contrastive learning to achieve this alignment. We implement our framework on a Transformer-based model and experiment it on three datasets. The efficiency of our solution is evaluated against seven baselines and considering four distinct patterns of missing data. Our experiments show that this proposal has a significant improvement in prediction accuracy (5.21% on average) over the baselines.

Previous studies have shown that the personality composition of a group significantly affects learners’ satisfaction during collaborative learning. However, while these studies investigated a group as a whole by focusing on group statistics, such as the mean and standard deviation of the members’ personalities, they paid little attention to the personality differences of individual pairs within the group, albeit the group contains many pairwise interactions. In this paper, we studied whether and how pairwise personality differences between a learner and groupmates affect the learner’s satisfaction. Examining data collected from an employee training program during which learners had reflective group discussions, we confirmed that pairwise personality differences significantly affect a learner’s level of satisfaction in the program. Specifically, satisfaction is affected by (1) the average of the personality differences between the learner and each individual groupmate, which reflects the degree to which the learner is different from the groupmates on average, and (2) the personality difference from the groupmate who has the most different/similar personality from/to the learner.

Understanding human mobility is mostly based on destination prediction. The reality that training and test data frequently differ makes it challenging to broaden the application of destination prediction, and when a prediction model that was trained on certain areas is then applied to the area of interest but with a different data distribution, the accuracy is suboptimal at best. The objective of the IEEE Big Data Cup 2022 is to solicit a robust and generalizable model that can effectively forecast a person’s daily destination based on facts and qualities obtained from four Japanese urban areas, as well as to predict the destination for a new metropolitan area using the models that were trained with the prior data. To address the issue and face the challenge of this Cup, our KDDI Research team has developed a prediction method based on Exploratory Data Analysis (EDA) that does not employ geographical zone information that varies from area to area. Instead, we employ the mobility characteristics of human groups, where each group is categorized according to demographics, which according to our EDA are relatively universal across areas. Our experimental findings show that our method achieves a relatively good prediction accuracy, which is attested by this Cup’s leaderboard. Albeit our method is simple and straightforward, we can argue that our approach can be used as good baseline for human mobility destination prediction.

Spatiotemporal data aggregated over regions or time windows at various resolutions demonstrate heterogeneous patterns and dynamics in each resolution. Meanwhile, the multi-resolution characteristic provides rich contextual information, which is critical for effective long-sequence forecasting. The importance of such inter-resolution information is more significant in practical cases, where fine-grained data is usually collected via approaches with lower costs but also lower qualities compared to those for coarse-grained data. However, existing works focus on uni-resolution data and cannot be directly applied to fully utilize the aforementioned extra information in multi-resolution data. In this work, we propose Spatiotemporal Koopman Multi-Resolution Network (ST-KMRN), a physics-informed learning framework for long-sequence forecasting from multi-resolution spatiotemporal data. Our method jointly models data aggregated in multiple resolutions and captures the inter-resolution dynamics with the self-attention mechanism. We also propose downsampling and upsampling modules among resolutions to further strengthen the connections among data of multiple resolutions. Moreover, we enhance the modeling of intra-resolution dynamics with physics-informed modules based on Koopman theory. Experimental results demonstrate that our proposed approach achieves the best performance on the long-sequence forecasting tasks compared to baselines without a specific design for multi-resolution data.

Sense of agency (SoA) is the subjective experience that one’s own volitional action caused an event to happen. This experience has cast light to understanding fundamental aspects of human behavior, which includes regulating actions during goal pursuit. Due to its many facets, investigating SoA has proved to be a strong challenge, compelling theorists and experimentalists to develop various paradigms to analyze it. While investigations on SoA have primarily focused on simple tasks that probe basic self-agency capacity awareness, and were carried out mostly under controlled laboratory settings over short experiment durations, we investigated this feeling of control in a complex, natural setting where participants performed daily their goal-directed tasks. More importantly, however, we investigated the SoA construct in a multidimensional way, i.e., simultaneously investigating its pre-reflective and reflective, local and general, and dynamic nature, as well as how individual differences moderated its influence on goal pursuit. We collected over 5,000 data points from 43 participants on their daily perceptions of self-agency and pursuance of healthy eating for more than a month outside the confines of a lab using a smartphone app that we designed. We present our analyses and insights that emerged from our empirical results on how the many facets of SoA impacted in various ways the pursuance of the goal. To our knowledge, we are the first to study the SoA construct in this manner, and we posit our method can be used for an intelligent system to enhance a human counterpart’s SoA for self-driven persuasion to follow through the goal.

The interpersonal compatibility of a company agent and a customer significantly affects the outcome of their communication. In existing research, however, compatibility has been studied only in terms of the similarity in personality and values. That is, agents and customers were compared only on the same dimensions that make up their personality and values, e.g., on the same trait of the Big Five (compared agent's Extraversion and customer's Extraversion) or same values as per Schwartz's Basic Values (agent's Conformity and customer's Conformity). In this paper, we studied compatibility from a broader perspective, i.e., in addition to the similarity, we investigated interactions across different dimensions (e.g., an agent's Extraversion and a customer's Conformity, or the former's Extraversion and the latter's Neuroticism). Examining 7,594 real call logs collected from telemarketing call centers, we have confirmed that such different dimensional interactions significantly affect a customer making a purchase (i.e., a customer's conversion) or not. A simulation where we matched agents and customers demonstrated that our compatibility model that incorporated the interactions across different dimensions yielded significant conversion lift, i.e., +46% on the average, compared from one that used only similarity in personality and values.

An upcoming trend of affective gaming is where a player's emotional state is used to manipulate game play. This is an interesting field to explore especially for the survival horror genre that is excellent at producing player's intense emotions. In this research, we analyzed different player affective states prior to (i.e., Neutral, Anxiety, Suspense) and after (i.e., Low-Fear, Mid-Fear, High-Fear) a scary event using an affect annotation tool to collect player self-reports of their affective states during the game. Brainwave signals, heart rate and keyboard-mouse activity were also collected for analyzing the potential of automatically detecting horror-related affect. Results indicated that players were more likely to experience fear from a scary event when they were in a suspense state compared to when they were in a neutral state. In this state, players only experienced fear after experiencing surprise. Heart rate data gave the best result in classifying player affect, which achieved up to 90% overall accuracy. This highlights the potential of using player affect in survival horror games to adapt a scary event to evoke more fear from players.

Self-reflection and self-evaluation are effective processes for identifying good learning behavior. These are essential in self-directed learning scenarios because students have to be responsible for their own learning. Although students benefit from doing fine-grained analysis of their own behavior, which we observed in our previous work, asking them to perform tasks such as analysis and making annotations are tedious and take significant amount of time and effort. In this paper, we present our work on the development of incremental affect models that can be used to minimize effort in analyzing and annotating behavior. Incremental models have an added benefit of adaptability to new information, which can be used by future systems to provide up-to-date affect-related feedback in real time.

Learning is commonly associated with knowledge transfer involving guidance from a teacher. However, as people grow older they are expected to know how to learn by themselves. In this research, we analyzed student learning in an unsupervised learning environment, i.e., performing academic research, wherein students have complete control over their learning thus requiring them to manage it. Transition likelihood metrics were used to analyze the interplay between emotion, learning and non-learning related activities while students did research. Several observations were seen from students learning in this environment such as students experiencing cognitive disequilibrium but experiencing disengagement faster. Non-learning related activities were also shown to have the potential of motivating students to resume learning. Lastly, user-specific traits and context seem to affect the interplay between learning and non-learning activities in an unsupervised learning environment. This highlights the need to not only create general models to predict student behavior but also user-specific models to allow future systems to provide appropriate feedback in this environment.

Commonly attributed to digital natives is the ability to quickly, yet effectively, shift from one task to another. However, several works have debunked this assumption by showing that multitasking even among digital natives led to poor learning performance and productivity. Our aim is to provide a tool to help digital natives be self-aware of desirable, while curbing undesirable, learning behaviors. Our tool is infused with self-annotation and feedback mechanisms that allow students to reflect upon their entire learning history. Our results indicate that the annotation process with the tool helped students understand their learning behaviors better and identify ways in which their behaviors can be improved.

Students have different ways of learning and have varied reactions to feedback. Thus, allowing a system to predict how students would appraise certain feedback gives it the capability to adapt to what would help a student learn better. This research focuses on the prediction of a student’s appraisal of feedback provided in an intelligent tutoring system (ITS). A regression model for frustration and excitement is created to perform prediction. The frustration model was able to achieve a 0.724 correlation with a 0.164 RMSE and the excitement model was able to achieve 0.6 a correlation with a 0.189 RMSE. These results indicate the potential of using these models for allowing systems to adjust feedback automatically based on student’s reactions while using an ITS.

Data-centric affect modeling may render itself restrictive in practical applications for three reasons, namely, it falls short of feature optimization, infers discrete affect classes, and deals with relatively small to average sized datasets. Though it seems practical to use the feature combinations already associated to commonly investigated sensors, there may be other potentially optimal features that can lead to new relations. Secondly, although it seems more realistic to view affect as continuous, it requires using continuous labels that will increase the difficulty of modeling. Lastly, although a large scale dataset reflects a more precise range of values for any given feature, it severely hinders computational efficiency. We address these problems when inferring physiology-affect relations from datasets that contain 2-3 million feature vectors, each with 49 features and labelled with continuous affect values. We employ automatic feature selection to acquire near optimal feature subsets and a fast approximate kNN algorithm to solve the regression problem and cope with the challenge of a large scale dataset. Our results show that high estimation accuracy may be achieved even when the selected feature subset is only about 7% of the original features. May the results here motivate the HCI community to pursue affect modeling without being deterred by large datasets and further the discussions on acquiring optimal features for accurate continuous affect approximation.

The consideration of human feelings in automated music generation by intelligent music systems, albeit a compelling theme, has received very little attention. This work aims to computationally specify a system's music compositional intelligence that tightly couples with the listener's affective perceptions. First, the system induces a model that describes the relationship between feelings and musical structures. The model is learned by applying the inductive logic programming paradigm of FOIL coupled with the Diverse Density weighting metric over a dataset that was constructed using musical score fragments that were hand-labeled by the listener according to a semantic differential scale that uses bipolar affective descriptor pairs. A genetic algorithm, whose fitness function is based on the acquired model and follows basic music theory, is then used to generate variants of the original musical structures. Lastly, the system creates chordal and non-chordal tones out of the GA-obtained variants. Empirical results show that the system is 80.6% accurate at the average in classifying the affective labels of the musical structures and that it is able to automatically generate musical pieces that stimulate four kinds of impressions, namely, favorable-unfavorable, bright-dark, happy-sad, and heartrending-not heartrending.

This paper describes a learning process for the tutor of an intelligent tutoring system (ITS) to automatically learn models of student categories and self-improve its instructional plans on the level of these categories. Using real-world teaching scenarios as experiment data, we empirically show that for every category the tutor is able to efficiently learn effective instructional plans. Our experiment results also show that the absence of category background knowledge decreases the tutor's learning performance as well the effectiveness of the learned instructional plans.

This paper discusses a machine learning framework that uses extraction, classification, and generalization techniques to classify students according to their cognitive and behavioral learning patterns and to categorize tutoring strategies of expert human tutors. A great deal of the discussion focuses on the use of reinforcement learning techniques, specifically the ?-greedy and temporal difference TD(0) methods in deriving tutoring policies over a class of students. Future works will deal on incremental learning and modification of learned policies while the tutor performs on-line in real-time, and extracting and learning the way expert tutors execute their tutoring activities.