Always-on sensing of AI applications on AR glasses makes traditional permission techniques inefficient for context-dependent private visual data within home environments. Home presents a challenging privacy context due to massive sensitive objects and the intimate nature of daily routines. We propose VisGuardian, a fine-grained content-based visual permission technique for AR glasses. VisGuardian features a group-based control mechanism that enables users to efficiently manage permissions for multiple private objects. VisGuardian detects objects using YOLO and adopts a pre-classified schema to group them. By selecting a single object, users can obscure groups of related objects based on criteria including privacy sensitivity, object category, or spatial proximity. A technical evaluation shows VisGuardian achieves mAP50 of 0.6704 with only 14.0 ms latency and a 1.7% increase in battery consumption per hour. Furthermore, a user study (N=24) comparing VisGuardian to slider-based and object-based baselines found it to be significantly faster for setting permissions and was preferred by users for its efficiency, effectiveness, and ease of use.

Recent HCI research has highlighted home automation’s potential in providing residents with technology-enhanced domestic autonomy. However, in the cohabitation context, the prevalent solutionist paradigm of automated systems introduces challenges to non-experts, paradoxically marginalizing specific members. This paper reports a co-creation initiative involving cohabitants, exploring a new understanding of empowerment in home automation. Participants collaborated to construct Trigger-Action Program (TAP) schemes using card-based tools during workshops. Our findings showcase how cohabitants engaged in collective ideations and embodied different negotiation patterns, which reveals the significance of more perceptible and participatory design. We frame home automation as "problematic co-design", arguing the universal overlook of collaborative resources. Furthermore, we examine how automation systems act as obstacles and sources of empowerment through the co-design lens. The paper concludes with pragmatic recommendations for designers and researchers, emphasizing the need to foster contestability for cohabitants in the evolving home automation landscape.

Emotion recognition in conversations (ERC) has emerged as an important research area in Natural Language Processing and Affective Computing, focusing on accurately identifying emotions within the conversational utterance. Conventional approaches typically rely on labeled training samples for fine-tuning pre-trained language models (PLMs) to enhance classification performance. However, the limited availability of labeled data in real-world scenarios poses a significant challenge, potentially resulting in diminished model performance. In response to this challenge, we present the Multi-modal Attentive Prompt (MAP) learning framework, tailored specifically for few-shot emotion recognition in conversations. The MAP framework consists of four integral modules: multi-modal feature extraction for the sequential embedding of text, visual, and acoustic inputs; a multi-modal prompt generation module that creates six manually-designed multi-modal prompts; an attention mechanism for prompt aggregation; and an emotion inference module for emotion prediction. To evaluate our proposed model’s efficacy, we conducted extensive experiments on two widely recognized benchmark datasets, MELD and IEMOCAP. Our results demonstrate that the MAP framework outperforms state-of-the-art ERC models, yielding notable improvements of 3.5% and 0.4% in micro F1 scores. These findings highlight the MAP learning framework’s ability to effectively address the challenge of limited labeled data in emotion recognition, offering a promising strategy for improving ERC model performance.

Towards sufficient music searching, it is vital to form a complete set of labels for each song. However, current solutions fail to resolve it as they cannot produce diverse enough mappings to make up for the information missed by the gold labels. Based on the observation that such missing information may already be presented in user comments, we propose to study the automated music labeling in an essential but under-explored setting, where the model is required to harvest more diverse and valid labels from the users' comments given limited gold labels. To this end, we design an iterative framework (DiVa) to harvest more Diverse and Valid labels from user comments for music. The framework makes a classifier able to form complete sets of labels for songs via pseudo-labels inferred from pre-trained classifiers and a novel joint score function. The experiment on a densely annotated testing set reveals the superiority of the DiVa over state-of-the-art solutions in producing more diverse labels missed by the gold labels. We hope our work can inspire future research on automated music labeling.

The state-of-the-art approaches to targeted aspect-based sentiment analysis (TABSA) are mostly built on deep neural networks with attention mechanisms. One problem is that embeddings of targets and aspects are either pre-trained from large external corpora or randomly initialized. We argue that affective commonsense knowledge and words indicative of sentiment could be used to learn better target and aspect embeddings. We therefore propose an embedding refinement framework calledRAEC(RefiningAffectiveEmbedding fromContext), in which sentiment concepts extracted from affective commonsense knowledge and word relative location information are incorporated to derive context-affective embeddings. Furthermore, a sparse coefficient vector is exploited in refining the embeddings of targets and aspects separately. In this way, embeddings of targets and aspects can capture the highly relevant affective words. Experimental results on two benchmark datasets show that our framework can be easily integrated with existing embedding-based TABSA models and achieves state-of-the-art results compared to models relying on pre-trained word embeddings or built on other embedding refinement methods.

Argument mining (AM) is a challenging task as it requires recognizing the complex argumentation structures involving multiple subtasks.To handle all subtasks of AM in an end-to-end fashion, previous works generally transform AM into a dependency parsing task.However, such methods largely require complex pre-and post-processing to realize the task transformation.In this paper, we investigate the endto-end AM task from a novel perspective by proposing a generative framework, in which the expected outputs of AM are framed as a simple target sequence.Then, we employ a pretrained sequence-to-sequence language model with a constrained pointer mechanism (CPM) to model the clues for all the subtasks of AM in the light of the target sequence.Furthermore, we devise a reconstructed positional encoding (RPE) to alleviate the order biases induced by the autoregressive generation paradigm.Experimental results show that our proposed framework achieves new state-of-the-art performance on two AM benchmarks. 1

This paper investigates the sentiment analysis task from a novel perspective by incorporating sentiment knowledge and eye movement into a graph architecture, aiming to draw the eye movement-based sentiment relationships for learning the sentiment expression of the context.To be specific, we first explore a linguistic probing eye movement paradigm to extract eye movement features based on the close relationship between linguistic features and the early and late processes of human reading behavior.Furthermore, to derive eye movement features with sentiment concepts, we devise a novel weighting strategy to integrate sentiment scores extracted from affective commonsense knowledge into eye movement features, called sentiment-eye movement weights.Then, the sentiment-eye movement weights are exploited to build the sentiment-eye movement guided graph (SEMGraph) model, so as to model the intricate sentiment relationships in the context.Experimental results on two sentiment analysis datasets with eye movement signals and three sentiment analysis datasets without eye movement signals show that the proposed SEM-Graph achieves state-of-the-art performance, and can also be directly generalized to those sentiment analysis datasets without eye movement signals.

Stance detection aims to identify the stance of the text towards a target. Different from conventional stance detection, Zero-Shot Stance Detection (ZSSD) needs to predict the stances of the unseen targets during the inference stage. For human beings, we generally tend to reason the stance of a new target by linking it with the related knowledge learned from the known ones. Therefore, in this paper, to better generalize the target-related stance features learned from the known targets to the unseen ones, we incorporate the targeted background knowledge from Wikipedia into the model. The background knowledge can be considered as a bridge for connecting the meanings between known targets and the unseen ones, which enables the generalization and reasoning ability of the model to be improved in dealing with ZSSD. Extensive experimental results demonstrate that our model outperforms the state-of-the-art methods on the ZSSD task.

Jianzhu Bao, Chuang Fan, Jipeng Wu, Yixue Dang, Jiachen Du, Ruifeng Xu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Showing empathy and reacting to users’ feeling are important social skills for current dialogue generation systems. In previous research, empathetic responses are generated by 1) only modeling the emotion of dialogue history or 2) indirectly leveraging the predicted emotion label of responses. In this paper, we propose a novel empathetic response generation method that incorporates the anticipated emotion into response generation by minimizing the divergence between distribution of responses’ anticipated emotion and ground-truth emotion. The anticipated emotion is predicted by an auxiliary emotion predictor whose input is the previous utterances. Additionally, we treat the generation as deliberation process and design a two-round training method to refine the response iteratively. Experimental results show that the proposed model outperforms the previous state-of-the-art for emphatic dialogue generation task.

Target plays an essential role in stance detection of an opinionated review/claim, since the stance expressed in the text often depends on the target. In practice, we need to deal with targets unseen in the annotated training data. As such, detecting stance for an unknown or unseen target is an important research problem. This paper presents a novel approach that automatically identifies and adapts the target-dependent and target-independent roles that a word plays with respect to a specific target in stance expressions, so as to achieve cross-target stance detection. More concretely, we explore a novel solution of constructing heterogeneous target-adaptive pragmatics dependency graphs (TPDG) for each sentence towards a given target. An in-target graph is constructed to produce inherent pragmatics dependencies of words for a distinct target. In addition, another cross-target graph is constructed to develop the versatility of words across all targets for boosting the learning of dominant word-level stance expressions available to an unknown target. A novel graph-aware model with interactive Graphical Convolutional Network (GCN) blocks is developed to derive the target-adaptive graph representation of the context for stance detection. The experimental results on a number of benchmark datasets show that our proposed model outperforms state-of-the-art methods in cross-target stance detection.

Emotion-cause pair extraction aims to extract all potential pairs of emotions and corresponding causes from unannotated emotion text.Most existing methods are pipelined framework, which identifies emotions and extracts causes separately, leading to a drawback of error propagation.Towards this issue, we propose a transition-based model to transform the task into a procedure of parsing-like directed graph construction.The proposed model incrementally generates the directed graph with labeled edges based on a sequence of actions, from which we can recognize emotions with the corresponding causes simultaneously, thereby optimizing separate subtasks jointly and maximizing mutual benefits of tasks interdependently.Experimental results show that our approach achieves the best performance, outperforming the state-of-the-art methods by 6.71% (p < 0.01) in F 1 measure.

In most previous studies, the aspect-related text is considered an important clue for the Aspect-based Sentiment Analysis (ABSA) task, and thus various attention mechanisms have been proposed to leverage the interactions between aspects and context. However, it is observed that some sentiment expressions carry the same polarity regardless of the aspects they are associated with. In such cases, it is not necessary to incorporate aspect information for ABSA. More observations on the experimental results show that blindly leveraging interactions between aspects and context as features may introduce noises when analyzing those aspect-invariant sentiment expressions, especially when the aspect-related annotated data is insufficient. Hence, in this paper, we propose an Adversarial Multi-task Learning framework to identify the aspect-invariant/dependent sentiment expressions without extra annotations. In addition, we adopt a gating mechanism to control the contribution of representations derived from aspect-invariant and aspect-dependent hidden states when generating the final contextual sentiment representations for the given aspect. This essentially allows the exploitation of aspect-invariant sentiment features for better ABSA results. Experimental results on two benchmark datasets show that extending existing neural models using our proposed framework achieves superior performance. In addition, the aspect-invariant data extracted by the proposed framework can be considered as pivot features for better transfer learning of the ABSA models on unseen aspects.

In this paper, we explore a novel solution of constructing a heterogeneous graph for each instance by leveraging aspect-focused and inter-aspect contextual dependencies for the specific aspect and propose an Interactive Graph Convolutional Networks (InterGCN) model for aspect sentiment analysis. Specifically, an ordinary dependency graph is first constructed for each sentence over the dependency tree. Then we refine the graph by considering the syntactical dependencies between contextual words and aspect-specific words to derive the aspect-focused graph. Subsequently, the aspect-focused graph and the corresponding embedding matrix are fed into the aspect-focused GCN to capture the key aspect and contextual words. Besides, to interactively extract the inter-aspect relations for the specific aspect, an inter-aspect GCN is adopted to model the representations learned by aspect-focused GCN based on the inter-aspect graph which is constructed by the relative dependencies between the aspect words and other aspects. Hence, the model can be aware of the significant contextual and aspect words when interactively learning the sentiment features for a specific aspect. Experimental results on four benchmark datasets illustrate that our proposed model outperforms state-of-the-art methods and substantially boosts the performance in comparison with BERT.

Attention-based neural models were employed to detect the different aspects and sentiment polarities of the same target in targeted aspectbased sentiment analysis (TABSA).However, existing methods do not specifically pre-train reasonable embeddings for targets and aspects in TABSA.This may result in targets or aspects having the same vector representations in different contexts and losing the contextdependent information.To address this problem, we propose a novel method to refine the embeddings of targets and aspects.Such pivotal embedding refinement utilizes a sparse coefficient vector to adjust the embeddings of target and aspect from the context.Hence the embeddings of targets and aspects can be refined from the highly correlative words instead of using context-independent or randomly initialized vectors.Experiment results on two benchmark datasets show that our approach yields the state-of-the-art performance in TABSA task.

Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Lidong Bing, Min Yang, Ruifeng Xu, Ruibin Mao. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Inferring the agreement/disagreement relation in debates, especially in online debates, is one of the fundamental tasks in argumentation mining. The expressions of agreement/disagreement usually rely on argumentative expressions in text as well as interactions between participants in debates. Previous works usually lack the capability of jointly modeling these two factors. To alleviate this problem, this paper proposes a hybrid neural attention model which combines self and cross attention mechanism to locate salient part from textual context and interaction between users. Experimental results on three (dis)agreement inference datasets show that our model outperforms the state-of-the-art models.

Combining the virtues of probability graphic models and neural networks, Conditional Variational Auto-encoder (CVAE) has shown promising performance in many applications such as response generation.However, existing CVAE-based models often generate responses from a single latent variable which may not be sufficient to model high variability in responses.To solve this problem, we propose a novel model that sequentially introduces a series of latent variables to condition the generation of each word in the response sequence.In addition, the approximate posteriors of these latent variables are augmented with a backward Recurrent Neural Network (RNN), which allows the latent variables to capture long-term dependencies of future tokens in generation.To facilitate training, we supplement our model with an auxiliary objective that predicts the subsequent bag of words.Empirical experiments conducted on the OpenSubtitle and Reddit datasets show that the proposed model leads to significant improvements on both relevance and diversity over state-of-the-art baselines.

Memory networks have shown expressive performance on aspect based sentiment analysis. However, ordinary memory networks only capture word-level information and lack the capacity for modeling complicated expressions which consist of multiple words. Targeting this problem, we propose a novel convolutional memory network which incorporates an attention mechanism. This model sequentially computes the weights of multiple memory units corresponding to multi-words. This model may capture both words and multi-words expressions in sentences for aspect-based sentiment analysis. Experimental results show that the proposed model outperforms the state-of-the-art baselines.

Emotion cause extraction aims to identify the reasons behind a certain emotion expressed in text.It is a much more difficult task compared to emotion classification.Inspired by recent advances in using deep memory networks for question answering (QA), we propose a new approach which considers emotion cause identification as a reading comprehension task in QA.Inspired by convolutional neural networks, we propose a new mechanism to store relevant context in different memory slots to model context information.Our proposed approach can extract both word level sequence features and lexical features.Performance evaluation shows that our method achieves the state-of-the-art performance on a recently released emotion cause dataset, outperforming a number of competitive baselines by at least 3.01% in F-measure.

Stance classification, which aims at detecting the stance expressed in text towards a specific target, is an emerging problem in sentiment analysis. A major difference between stance classification and traditional aspect-level sentiment classification is that the identification of stance is dependent on target which might not be explicitly mentioned in text. This indicates that apart from text content, the target information is important to stance detection. To this end, we propose a neural network-based model, which incorporates target-specific information into stance classification by following a novel attention mechanism. In specific, the attention mechanism is expected to locate the critical parts of text which are related to target. Our evaluations on both the English and Chinese Stance Detection datasets show that the proposed model achieves the state-of-the-art performance.