In this demo, we present NarrativePlay -- an innovative system enabling users to role-play a fictional character and interact with dynamically generated narrative environments. Unlike existing predefined sandbox approaches, NarrativePlay centres around the main storyline events extracted from the narrative, allowing users to experience the story from the perspective of a character they chose. To design versatile AI agents for diverse scenarios, we employ a framework built on a Large Language Models (LLMs) to extract detailed character traits from text. We also incorporate automatically generated visual displays of narrative settings, character portraits, and character speech, greatly enhancing the overall user experience.

The development of AI in mental health is a growing field with potential global impact. Machine agents need to perceive users' mental states and respond empathically. Since mental states are often latent and implicit, building such chatbots requires both knowledge learning and knowledge utilization. Our work contributes to this by developing a chatbot that aims to recognize and empathetically respond to users' mental states. We introduce a Conditional Variational Autoencoders (CVAE)-based model that utilizes relation-aware commonsense knowledge to generate responses. This model, while not a replacement for professional mental health support, demonstrates promise in offering informative and empathetic interactions in a controlled environment. On the dataset EmpatheticDialogues, we compare with several SOTA methods and empirically validate the effectiveness of our approach on response informativeness and empathy exhibition. Detailed analysis is also given to demonstrate the learning capability as well as model interpretability. Our code is accessible at http://github.com/ChangyuChen347/COMET-VAE.

Abstract. Each coil image in a parallel magnetic resonance imaging (pMRI) system is an imaging slice modulated by the corresponding coil sensitivity. These coil images, structurally similar to each other, are stacked together as 3-dimensional (3D) image data, and their sparsity property can be explored via 3D directional Haar tight framelets. The features of the 3D image data from the 3D framelet systems are utilized to regularize sensitivity encoding (SENSE) pMRI reconstruction. Accordingly, a so-called SENSE3d algorithm is proposed to reconstruct images of high quality from the sampled [Formula: see text]-space data with a high acceleration rate by decoupling effects of the desired image (slice) and sensitivity maps. Since both the imaging slice and sensitivity maps are unknown, this algorithm repeatedly performs a slice step followed by a sensitivity step by using updated estimations of the desired image and the sensitivity maps. In the slice step, for the given sensitivity maps, the estimation of the desired image is viewed as the solution to a convex optimization problem regularized by the sparsity of its 3D framelet coefficients of coil images. This optimization problem, involving data from the complex field, is solved by a primal-dual three-operator splitting (PD3O) method. In the sensitivity step, the estimation of sensitivity maps is modeled as the solution to a Tikhonov-type optimization problem that favors the smoothness of the sensitivity maps. This corresponding problem is nonconvex and could be solved by a forward-backward splitting method. Experiments on real phantoms and in vivo data show that the proposed SENSE3d algorithm can explore the sparsity property of the imaging slices and efficiently produce reconstructed images of high quality with reduced aliasing artifacts caused by high acceleration rate, additive noise, and the inaccurate estimation of each coil sensitivity. To provide a comprehensive picture of the overall performance of our SENSE3d model, we provide the quantitative index (HaarPSI) and comparisons to some deep learning methods such as VarNet and fastMRI-UNet.

The mainstream of the existing approaches for video prediction builds up their models based on a Single-In-Single-Out (SISO) architecture, which takes the current frame as input to predict the next frame in a recursive manner. This way often leads to severe performance degradation when they try to extrapolate a longer period of future, thus limiting the practical use of the prediction model. Alternatively, a Multi-In-Multi-Out (MIMO) architecture that outputs all the future frames at one shot naturally breaks the recursive manner and therefore prevents error accumulation. However, only a few MIMO models for video prediction are proposed and they only achieve inferior performance due to the date. The real strength of the MIMO model in this area is not well noticed and is largely under-explored. Motivated by that, we conduct a comprehensive investigation in this paper to thoroughly exploit how far a simple MIMO architecture can go. Surprisingly, our empirical studies reveal that a simple MIMO model can outperform the state-of-the-art work with a large margin much more than expected, especially in dealing with long-term error accumulation. After exploring a number of ways and designs, we propose a new MIMO architecture based on extending the pure Transformer with local spatio-temporal blocks and a new multi-output decoder, namely MIMO-VP, to establish a new standard in video prediction. We evaluate our model in four highly competitive benchmarks. Extensive experiments show that our model wins 1st place on all the benchmarks with remarkable performance gains and surpasses the best SISO model in all aspects including efficiency, quantity, and quality. A dramatic error reduction is achieved when predicting 10 frames on Moving MNIST and Weather datasets respectively. We believe our model can serve as a new baseline to facilitate the future research of video prediction tasks. The code will be released.

Previous works on emotion recognition in conversation (ERC) follow a two-step paradigm, which can be summarized as first producing context-independent features via fine-tuning pretrained language models (PLMs) and then analyzing contextual information and dialogue structure information among the extracted features. However, we discover that this paradigm has several limitations. Accordingly, we propose a novel paradigm, i.e., exploring contextual information and dialogue structure information in the fine-tuning step, and adapting the PLM to the ERC task in terms of input text, classification structure, and training strategy. Furthermore, we develop our model BERT-ERC according to the proposed paradigm, which improves ERC performance in three aspects, namely suggestive text, fine-grained classification module, and two-stage training. Compared to existing methods, BERT-ERC achieves substantial improvement on four datasets, indicating its effectiveness and generalization capability. Besides, we also set up the limited resources scenario and the online prediction scenario to approximate real-world scenarios. Extensive experiments demonstrate that the proposed paradigm significantly outperforms the previous one and can be adapted to various scenes.

In this work, we tackle the scenario of understanding characters in scripts, which aims to learn the characters' personalities and identities from their utterances.We begin by analyzing several challenges in this scenario, and then propose a multi-level contrastive learning framework to capture characters' global information in a fine-grained manner.To validate the proposed framework, we conduct extensive experiments on three character understanding sub-tasks by comparing with strong pretrained language models, including SpanBERT, Longformer, BigBird and ChatGPT-3.5.Experimental results demonstrate that our method improves the performances by a considerable margin.Through further in-depth analysis, we show the effectiveness of our method in addressing the challenges and provide more hints on the scenario of character understanding.We will open-source our work in this URL....

The automotive industry is shifting from hardware-centric to software-centric with the emergence of various intelligent features powered by software. This poses a new challenge for software testers to ensure software reliability by designing test plans that satisfy the test objectives while abiding by the constraints like scope, time, as well as various automotive safety standards. This paper proposed an automatic test plan generation framework built on the evolutionary algorithm. A novel encoding mechanism is proposed to represent the multi-dimensional test plan, while a belief model is proposed to reveal the underlying correlations between the relevant test attributes. Experiments conducted on an actual automotive software in production environment developed by our industry partner show that our method can achieve around 50% improvements in finding defects and covering high-priority test cases as compared to typical evolutionary algorithms while abiding by multiple constraints such as the total run time and custom objectives set by users.

Applying existing methods to emotional support conversation-which provides valuable assistance to people who are in need-has two major limitations: (a) they generally employ a conversation-level emotion label, which is too coarse-grained to capture user's instant mental state; (b) most of them focus on expressing empathy in the response(s) rather than gradually reducing user's distress.To address the problems, we propose a novel model MISC, which firstly infers the user's fine-grained emotional status, and then responds skillfully using a mixture of strategy.Experimental results on the benchmark dataset demonstrate the effectiveness of our method and reveal the benefits of fine-grained emotion understanding as well as mixed-up strategy modeling.Our code and data could be found in https: //github.com/morecry/MISC.

Excellent performance has been achieved on instance segmentation but the quality on the boundary area remains unsatisfactory, which leads to a rising attention on boundary refinement. For practical use, an ideal post-processing refinement scheme are required to be accurate, generic and efficient. However, most of existing approaches propose pixel-wise refinement, which either introduce a massive computation cost or design specifically for different backbone models. Contour-based models are efficient and generic to be incorporated with any existing segmentation methods, but they often generate over-smoothed contour and tend to fail on corner areas. In this paper, we propose an efficient contour-based boundary refinement approach, named SharpContour, to tackle the segmentation of boundary area. We design a novel contour evolution process together with an Instance-aware Point Classifier. Our method deforms the contour iteratively by updating offsets in a discrete manner. Differing from existing contour evolution methods, SharpContour estimates each offset more independently so that it predicts much sharper and accurate contours. Notably, our method is generic to seamlessly work with diverse existing models with a small computational cost. Experiments show that SharpContour achieves competitive gains whilst preserving high efficiency.

Skeleton-based human action recognition is a longstanding challenge due to its complex dynamics. Some fine-grain details of the dynamics play a vital role in classification. The existing work largely focuses on designing incremental neural networks with more complicated adjacent matrices to capture the details of joints relationships. However, they still have difficulties distinguishing actions that have broadly similar motion patterns but belong to different categories. Interestingly, we found that the subtle differences in motion patterns can be significantly amplified and become easy for audience to distinct through specified view directions, where this property haven't been fully explored before. Drastically different from previous work, we boost the performance by proposing a conceptually simple yet effective Multi-view strategy that recognizes actions from a collection of dynamic view features. Specifically, we design a novel Skeleton-Anchor Proposal (SAP) module which contains a Multi-head structure to learn a set of views. For feature learning of different views, we introduce a novel Angle Representation to transform the actions under different views and feed the transformations into the baseline model. Our module can work seamlessly with the existing action classification model. Incorporated with baseline models, our SAP module exhibits clear performance gains on many challenging benchmarks. Moreover, comprehensive experiments show that our model consistently beats down the state-of-the-art and remains effective and robust especially when dealing with corrupted data. Related code will be available on https://github.com/ideal-idea/SAP

Conversational recommendation system aims to recommend appropriate items to user by directly asking preference on attributes or recommending item list. However, most of existing methods only employ the flat item and attribute relationship, and ignore the hierarchical relationship connected by the similar user which can provide more comprehensive information. And these methods usually use the user accepted attributes to represent the conversational history and ignore the hierarchical information of sequential transition in the historical turns. In this paper, we propose Hierarchical Information-aware Conversational Recommender (HICR) to model the two types of hierarchical information to boost the performance of CRS. Experiments conducted on four benchmark datasets verify the effectiveness of our proposed model.

There are a variety of underlying factors influencing what and how people communicate in their daily life. The ability to capture and utilize these factors enables the conversational systems to generate favorable responses and set up amicable connections with users. In this work, we investigate two major factors in response generation, i.e., emotion and intention. To explore the dependency between them, we develop a hierarchical variational model that predicts in sequence the emotion and intention to be conveyed in a response. The response can then be generated word-by-word based on the predictions. We also apply a novel adversarial-augmented inference network to facilitate model training. The experimental results demonstrate the effectiveness of the proposed model as well as the novel adversarial objective. The hypothesis that emotion shapes human communication behavior is also validated.

A simile is a figure of speech that directly makes a comparison, showing similarities between two different things, e.g. ``Reading papers can be dull sometimes,like watching grass grow". Human writers often interpolate appropriate similes into proper locations of the plain text to vivify their writings. However, none of existing work has explored neural simile interpolation, including both locating and generation. In this paper, we propose a new task of Writing Polishment with Simile (WPS) to investigate whether machines are able to polish texts with similes as we human do. Accordingly, we design a two-staged Locate&Gen model based on transformer architecture. Our model firstly locates where the simile interpolation should happen, and then generates a location-specific simile. We also release a large-scale Chinese Simile (CS) dataset containing 5 million similes with context. The experimental results demonstrate the feasibility of WPS task and shed light on the future research directions towards better automatic text polishment.

Existing emotion-aware conversational models usually focus on controlling the response contents to align with a specific emotion class, whereas empathy is the ability to understand and concern the feelings and experience of others. Hence, it is critical to learn the causes that evoke the users' emotion for empathetic responding, a.k.a. emotion causes. To gather emotion causes in online environments, we leverage counseling strategies and develop an empathetic chatbot to utilize the causal emotion information. On a real-world online dataset, we verify the effectiveness of the proposed approach by comparing our chatbot with several SOTA methods using automatic metrics, expert-based human judgements as well as user-based online evaluation.

In this work, we establish a context graph from both conversation utterances and external knowledge, and develop a novel graph-based encoder to better understand the conversation context. Specifically, the encoder fuses the information in the context graph stage-by-stage and provides global context-graph-aware representations of each node in the graph to facilitate knowledge-grounded response generation. On a large-scale conversation corpus, we validate the effectiveness of the proposed approach and demonstrate the benefit of knowledge in conversation understanding.

Abstract In this paper, we make two contributions. The first is to propose a new keyframe extraction algorithm, which reduces the keyframe redundancy and reduces the motion sequence reconstruction error. Secondly, a new motion sequence reconstruction method is proposed, which further reduces the error of motion sequence reconstruction. Specifically, we treated the input motion sequence as curves, then the binomial fitting was extended to obtain the points where the slope changes dramatically in the vicinity. Then we took these points as inputs to obtain keyframes by density clustering. Finally, the motion curves were segmented by keyframes and the segmented curves were fitted by binomial formula again to obtain the binomial parameters for motion reconstruction. Experiments show that our methods outperform existing techniques, in terms of reconstruction error.

When predicting PM2.5 concentrations, it is necessary to consider complex information sources since the concentrations are influenced by various factors within a long period. In this paper, we identify a set of critical domain knowledge for PM2.5 forecasting and develop a novel graph based model, PM2.5-GNN, being capable of capturing long-term dependencies. On a real-world dataset, we validate the effectiveness of the proposed model and examine its abilities of capturing both fine-grained and long-term influences in PM2.5 process. The proposed PM2.5-GNN has also been deployed online to provide free forecasting service.

Abstract Human motion prediction is a fundamental problem in understanding human natural movements. This task is very challenging due to the complex human body constraints and diversity of action types. Due to the human body being a natural graph, graph convolutional network (GCN)‐based models perform better than the traditional recurrent neural network (RNN)‐based models on modeling the natural spatial and temporal dependencies lying in the motion data. In this paper, we develop the GCN‐based models further by adding densely connected links to increase their feature utilizations and address oversmoothing problem. More specifically, the GCN block is used to learn the spatial relationships between the nodes and each feature map of the GCN block propagates directly to every following block as input rather than residual linked. In this way, the spatial dependency of human motion data is exploited more sufficiently and the features of different level of scale are fused more efficiently. Extensive experiments demonstrate our model achieving the state‐of‐the‐art results on CMU dataset.

Abstract Sketch‐based modeling aims to recover three‐dimensional (3D) shape from two‐dimensional line drawings. However, due to the sparsity and ambiguity of the sketch, it is extremely challenging for computers to interpret line drawings of physical objects. Most conventional systems are restricted to specific scenarios such as recovering for specific shapes, which are not conducive to generalize. Recent progress of deep learning methods have sparked new ideas for solving computer vision and pattern recognition issues. In this work, we present an end‐to‐end learning framework to predict 3D shape from line drawings. Our approach is based on a two‐steps strategy, it converts the sketch image to its normal image, then recover the 3D shape subsequently. A differentiable renderer is proposed and incorporated into this framework, it allows the integration of the rendering pipeline with neural networks. Experimental results show our method outperforms the state‐of‐art, which demonstrates that our framework is able to cope with the challenges in single sketch‐based 3D shape modeling.

We propose a chatbot, namely MOCHA to make good use of relevant entities when generating responses. Augmented with meta-path information, MOCHA is able to mention proper entities following the conversation flow.

We present a unified learning framework for recovering both 3D mesh and camera pose of the object from a single image. Our approach learns to recover outer shape and surface geometric details of the mesh without relying on 3D supervision. We adopt multi-view normal maps as the 2D supervision so that the silhouette and geometric details information can be transferred to neural network. A normal mismatch based objective function is introduced to train the network, and the camera pose is parameterized into the objective, it integrates pose estimation with the mesh reconstruction in a same optimization procedure. We demonstrate the abilities of the proposed approach in generating 3D mesh and estimating camera pose with qualitative and quantitative experiments.

Human motion prediction is a challenging problem due to the complicated human body constraints and high-dimensional dynamics. Recent deep learning approaches adopt RNN, CNN or fully connected networks to learn the motion features which do not fully exploit the hierarchical structure of human anatomy. To address this problem, we propose a convolutional hierarchical autoencoder model for motion prediction with a novel encoder which incorporates 1D convolutional layers and hierarchical topology. The new network is more efficient compared to the existing deep learning models with respect to size and speed. We train the generic model on Human3.6M and CMU benchmark and conduct extensive experiments. The qualitative and quantitative results show that our model outperforms the state-of-the-art methods in both short-term prediction and long-term prediction.

Word embeddings have become widelyused in document analysis.While a large number of models for mapping words to vector spaces have been developed, it remains undetermined how much net gain can be achieved over traditional approaches based on bag-of-words.In this paper, we propose a new document clustering approach by combining any word embedding with a state-of-the-art algorithm for clustering empirical distributions.By using the Wasserstein distance between distributions, the word-to-word semantic relationship is taken into account in a principled way.The new clustering method is easy to use and consistently outperforms other methods on a variety of data sets.More importantly, the method provides an effective framework for determining when and how much word embeddings contribute to document analysis.Experimental results with multiple embedding models are reported.

We develop a high-quality multi-turn dialog dataset, DailyDialog, which is intriguing in several aspects. The language is human-written and less noisy. The dialogues in the dataset reflect our daily communication way and cover various topics about our daily life. We also manually label the developed dataset with communication intention and emotion information. Then, we evaluate existing approaches on DailyDialog dataset and hope it benefit the research field of dialog systems. The dataset is available on http://yanran.li/dailydialog

Query relevance ranking and sentence saliency ranking are the two main tasks in extractive query-focused summarization. Previous supervised summarization systems often perform the two tasks in isolation. However, since reference summaries are the trade-off between relevance and saliency, using them as supervision, neither of the two rankers could be trained well. This paper proposes a novel summarization system called AttSum, which tackles the two tasks jointly. It automatically learns distributed representations for sentences as well as the document cluster. Meanwhile, it applies the attention mechanism to simulate the attentive reading of human behavior when a query is given. Extensive experiments are conducted on DUC query-focused summarization benchmark datasets. Without using any hand-crafted features, AttSum achieves competitive performance. We also observe that the sentences recognized to focus on the query indeed meet the query need.

Parallel magnetic resonance imaging (pMRI) is a technique to accelerate the magnetic resonance imaging process. The problem of reconstructing an image from the collected pMRI data is ill-posed. Regularization is needed to make the problem well-posed. In this paper, we first construct a two-dimensional tight framelet system whose filters have the same support as the orthogonal Haar filters and are able to detect edges of an image in the horizontal, vertical, and $\pm 45^o$ directions. This system is referred to as directional Haar framelet (DHF). We then propose a pMRI reconstruction model whose regularization term is formed by the DHF. This model is solved by a fast proximal algorithm with low computational complexity. The regularization parameters are updated adaptively and determined automatically during the iteration of the algorithm. Numerical experiments for in-silico and in-vivo data sets are provided to demonstrate the superiority of the DHF-based model and the efficiency of our proposed algorithm for pMRI reconstruction.

Distributed word representations are very useful for capturing semantic information and have been successfully applied in a variety of NLP tasks, especially on En-glish. In this work, we innovatively de-velop two component-enhanced Chinese character embedding models and their bi-gram extensions. Distinguished from En-glish word embeddings, our models ex-plore the compositions of Chinese char-acters, which often serve as semantic in-dictors inherently. The evaluations on both word similarity and text classification demonstrate the effectiveness of our mod-els. 1