The empathetic dialogue systems aim to recognize user emotions and generate appropriate empathetic responses. However, existing approaches predominantly rely on dialogue history, contextual descriptions, and emotion category labels, failing to model the causal relationship between emotions and their underlying triggers. This limitation leads to generated responses that lack grounding, exhibit weak relevance, and suffer from poor interpretability in emotional expression. To address this, we propose MvP-ECR, a multi-perspective emotion cause reasoning framework that explicitly constructs emotion-cause structures to help models focus on the core emotional drivers. Additionally, we introduce an emotion-cause consistency evaluation metric to quantitatively assess a model’s ability to identify causal relationships. Experiments across multiple large language models (LLMs) demonstrate that the MvP-ECR framework can serve as a plug-and-play tool to help the model correctly infer emotions and causes in empathetic conversations, and provide more immersive responses for empathetic responses. All code and data will be publicly released to promote the development of empathy dialogue research.

Multimodal Sentiment Analysis (MSA) aims to infer human sentiment by integrating information from multiple modalities such as text, audio, and video. In real-world scenarios, however, the presence of missing modalities and noisy signals significantly hinders the robustness and accuracy of existing models. While prior works have made progress on these issues, they are typically addressed in isolation, limiting overall effectiveness in practical settings. To jointly mitigate the challenges posed by missing and noisy modalities, we propose a framework called Two-stage Modality Denoising and Complementation (TMDC). TMDC comprises two sequential training stages. In the Intra-Modality Denoising Stage, denoised modality-specific and modality-shared representations are extracted from complete data using dedicated denoising modules, reducing the impact of noise and enhancing representational robustness. In the Inter-Modality Complementation Stage, these representations are leveraged to compensate for missing modalities, thereby enriching the available information and further improving robustness. Extensive evaluations on MOSI, MOSEI, and IEMOCAP demonstrate that TMDC consistently achieves superior performance compared to existing methods, establishing new state-of-the-art results.

Intelligent dialogue systems are increasingly deployed in emotionally and ethically sensitive settings, where failures in either emotional attunement or ethical judgment can cause significant harm.Existing dialogue models typically address empathy and ethical safety in isolation, and often fail to adapt their behavior as ethical risk and user emotion evolve across multi-turn interactions.We formulate ethical-emotional alignment in dialogue as an explicit turn-level decision problem, and propose ETHICMIND, a risk-aware framework that implements this formulation in multi-turn dialogue at inference time.At each turn, ETHIC-MIND jointly analyzes ethical risk signals and user emotion, plans a high-level response strategy, and generates context-sensitive replies that balance ethical guidance with emotional engagement, without requiring additional model training.To evaluate alignment behavior under ethically complex interactions, we introduce a risk-stratified, multi-turn evaluation protocol with a context-aware user simulation procedure.Experimental results show that ETHICMIND achieves more consistent ethical guidance and emotional engagement than competitive baselines, particularly in high-risk and morally ambiguous scenarios.

Existing jamming attacks on Retrieval-Augmented Generation (RAG) systems typically induce explicit refusals or denial-ofservice behaviors, which are conspicuous and easy to detect.In this work, we formalize a subtler availability threat, termed soft failure, which degrades system utility by inducing fluent and coherent yet non-informative responses rather than overt failures.We propose Deceptive Evolutionary Jamming Attack (DEJA), an automated black-box attack framework that generates adversarial documents to trigger such soft failures by exploiting safety-aligned behaviors of large language models.DEJA employs an evolutionary optimization process guided by a fine-grained Answer Utility Score (AUS), computed via an LLM-based evaluator, to systematically degrade the certainty of answers while maintaining high retrieval success.Extensive experiments across multiple RAG configurations and benchmark datasets show that DEJA consistently drives responses toward low-utility soft failures, achieving SASR above 79% while keeping hard-failure rates below 15%, significantly outperforming prior attacks.The resulting adversarial documents exhibit high stealth, evading perplexity-based detection and resisting query paraphrasing, and transfer across model families to proprietary systems without retargeting.

Conversational AI is increasingly deployed in emotionally charged and ethically sensitive interactions. Previous research has primarily concentrated on emotional benchmarks or static safety checks, overlooking how alignment unfolds in evolving conversation. We explore the research question: what breakdowns arise when conversational agents confront emotionally and ethically sensitive behaviors, and how do these affect dialogue quality? To stress-test chatbot performance, we develop a persona-conditioned user simulator capable of engaging in multi-turn dialogue with psychological personas and staged emotional pacing. Our analysis reveals that mainstream models exhibit recurrent breakdowns that intensify as emotional trajectories escalate. We identify several common failure patterns, including affective misalignments, ethical guidance failures, and cross-dimensional trade-offs where empathy supersedes or undermines responsibility. We organize these patterns into a taxonomy and discuss the design implications, highlighting the necessity to maintain ethical coherence and affective sensitivity throughout dynamic interactions. The study offers the HCI community a new perspective on the diagnosis and improvement of conversational AI in value-sensitive and emotionally charged contexts.

Road rage poses great risks to road safety. Digital interventions show promising potential in regulating rage-related maladaptive behaviors to mitigate such risks. This requires a clear understanding of road rage dynamics. Unlike prior work using artificial scenarios, we build the first-of-its-kind Real Road Rage Footage (R3-Ftg) dataset and recreate some of its most risky and anger-inducing scenes in both audiovisual and simulated environments. Then we recruit 52 participants to experience those scenes and record their behavioral and physiological responses. We find: (1) Road rage has been successfully induced, and simulation provides more realistic experiences. (2) "Slow-rise, fast-decay" phenomena are observed in both environments, which can be interpreted by Parasympathetic rebound. (3) This process can be modeled using second-order damped oscillation distributions. To our knowledge, we are the first to model road rage dynamics in authentic scenarios recreated from real-world events, enabling a holistic understanding on road rage.

Multimodal fusion aims to integrate complementary information from different modalities, yet it is often hindered by data distribution imbalance and cross-modal conflicts. In this paper, we propose Beneficial Noise Learning for Robust Multimodal Fusion (BRMF), a novel framework that reframes these challenges as beneficial perturbations signals for robust training. Specifically, BRMF first learns a stable and consistent joint representation within a Gaussian latent space under distributional constraints. Based on this joint representation, it then constructs diverse distributional environments and further encourages invariance by extending invariant risk minimization (IRM) to multimodal settings. Extensive experiments across multiple benchmarks demonstrate that BRMF exhibits superior robustness to noise and establishes new state-of-the-art performance.

Multimodal sentiment analysis (MSA) with missing modalities involves understanding the person's sentiment using multimodal data where some modalities are missing. Most existing methods focus on reconstructing the missing modalities using the available modalities from each sample, relying on modality-common information. However, these methods overlook the modality-specific information that other samples can provide. Additionally, these approaches often require the guidance of full modality representations during the reconstruction process, which is impractical in resource-constrained real-world scenarios. To address these challenges, we propose theIntra-sample andIntra-modalEnhancement (IIE) framework. The IIE framework enhances both sample-level and modality-level representations to capture additional modality-common and modality-specific information from existing modalities, without requiring full modalities. Specifically, IIE first learns sample-level representations by distilling modality-common information from the available modalities into learnable latent units. Then, it enhances modality-level representations by leveraging modality-specific information from other samples with the same modality, which is crucial for improving robustness in the presence of missing modalities. Finally, IIE ensures consistency between the enhanced modality-level and sample-level representations, combining the enhanced and initial representations to make predictions. Extensive experiments on three datasets demonstrate that the IIE framework significantly outperforms existing methods in terms of both effectiveness and robustness in handling MSA with missing modalities. Code is available athttps://github.com/YetZzzzzz/IIE.

In recent years, knowledge graphs have been integrated into recommender systems as item-side auxiliary information, enhancing recommendation accuracy. However, constructing and integrating structural user-side knowledge remains a significant challenge due to the improper granularity and inherent scarcity of user-side features. Recent advancements in Large Language Models (LLMs) offer the potential to bridge this gap by leveraging their human behavior understanding and extensive real-world knowledge. Nevertheless, integrating LLM-generated information into recommender systems presents challenges, including the risk of noisy information and the need for additional knowledge transfer. In this paper, we propose an LLM-based user-side knowledge inference method alongside a carefully designed recommendation framework to address these challenges. Our approach employs LLMs to infer user interests based on historical behaviors, integrating this user-side information with item-side and collaborative data to construct a hybrid structure: the Collaborative Interest Knowledge Graph (CIKG). Furthermore, we propose a CIKG-based recommendation framework that includes a user interest reconstruction module and a cross-domain contrastive learning module to mitigate potential noise and facilitate knowledge transfer. We conduct extensive experiments on three real-world datasets to validate the effectiveness of our method. Our approach achieves state-of-the-art performance compared to competitive baselines, particularly for users with sparse interactions.

The empathy dialogue system requires understanding emotions and their underlying causes.However, existing datasets mainly focus on emotion labels, while cause annotations are added post hoc through costly and subjective manual processes.This leads to three limitations: subjective bias in cause labels, weak rationality due to ambiguous cause-emotion relationships, and high annotation costs that hinder scalability.To address these challenges, we propose ECC (Emotion-Cause Conversation Dataset), a scalable dataset with 2.4K dialogues, which is also the first dialogue dataset where conversations and their emotion-cause labels are automatically generated synergistically during creation.We create an automatic extension framework EC-DD for ECC that utilizes knowledge and large language models (LLMs) to automatically generate conversations, and train a causality-aware empathetic response model CAER on this dataset.Experimental results show that ECC can achieve comparable or even superior performance to artificially constructed empathy dialogue datasets.

As social media becomes a dominant platform for sharing content, predicting the popularity of user posts has become increasingly important for applications such as content recommendation, trend forecasting, and user engagement. However, this task is challenging due to the diverse and multimodal nature of social media posts, which often include unstructured text, images, and structured metadata. To address this challenge, we propose Fusion-Aware Multi-modal Ensemble (FAME), a framework effectively captures and integrates diverse information sources within social media content. Unlike prior approaches that rely on a single model to process all modalities, FAME leverages four specialized predictors. Three of them-CatBoost, LightGBM, and AutoGluon-are tree-based models that excel at handling structured metadata and its interactions with unstructured features. The fourth is a denoising autoencoder (DAE), which learns robust joint representations from unstructured text and image data. These models are combined through a weighted ensemble strategy, allowing FAME to leverage the complementary strengths of different architectures. Experiments on the Social Media Prediction Dataset demonstrate that FAME significantly outperforms existing baselines, achieving state-of-the-art results and validating its effectiveness in modeling the complex, multimodal nature of social media content.

Multimodal Sentiment Analysis (MSA) aims to infer human emotions by integrating complementary signals from diverse modalities. However, in real-world scenarios, missing modalities are common due to data corruption, sensor failure, or privacy concerns, which can significantly degrade model performance. To tackle this challenge, we propose Hyper-Modality Enhancement (HME), a novel framework that avoids explicit modality reconstruction by enriching each observed modality with semantically relevant cues retrieved from other samples. This cross-sample enhancement reduces reliance on fully observed data during training, making the method better suited to scenarios with inherently incomplete inputs. In addition, we introduce an uncertainty-aware fusion mechanism that adaptively balances original and enriched representations to improve robustness. Extensive experiments on three public benchmarks show that HME consistently outperforms state-of-the-art methods under various missing modality conditions, demonstrating its practicality in real-world MSA applications.

Large language models (LLMs) are expected to offer structured Markdown responses for the sake of readability in web chatbots (e.g., ChatGPT). Although there are a myriad of metrics to evaluate LLMs, they fail to evaluate the readability from the view of output content structure. To this end, we focus on an overlooked yet important metric --- Markdown Awareness, which directly impacts the readability and structure of the content generated by these language models. In this paper, we introduce MDEval, a comprehensive benchmark to assess Markdown Awareness for LLMs, by constructing a dataset with 20K instances covering 10 subjects in English and Chinese. Unlike traditional model-based evaluations, MDEval provides excellent interpretability by combining model-based generation tasks and statistical methods. Our results demonstrate that MDEval achieves a Spearman correlation of 0.791 and an accuracy of 84.1% with human, outperforming existing methods by a large margin. Extensive experimental results also show that through fine-tuning over our proposed dataset, less performant open-source models are able to achieve comparable performance to GPT-4o in terms of Markdown Awareness. To ensure reproducibility and transparency, MDEval is open sourced at https://github.com/SWUFE-DB-Group/MDEval-Benchmark.

Social bots are becoming increasingly common in social networks, and their activities affect the security and authenticity of social media platforms. Current state-of-the-art social bot detection methods leverage multimodal approaches that analyze various modalities, such as user metadata, text, and social network relationships. However, these methods may not always extract additional dimensions of semantic feature information that could offer a deeper understanding of users' social patterns. To address this issue, we propose ETS-MM, a multimodal detection framework designed to augment multidimensional information from text and extract the semantic feature representation of user text information. We first analyze the user's tweeting behavior based on topic preference and emotion tendency, integrating them into the textual data. Then, we try to extract enhanced semantic representations that reveal the latent relationship between tweeting behavior and tweet content while identifying potential contextual associations and emotional changes. Additionally, to capture the complex interaction between users, we integrate the user's multimodal information, including metadata, textual features, enhanced semantic features, and social network relationships to propagate and aggregate information across various modalities. Experimental results demonstrate that ETS-MM significantly outperforms existing methods across two widely used social bot detection benchmark datasets, validating its effectiveness and superiority.

The Sloan Digital Sky Survey (SDSS), a large-scale astronomical survey project, has released a vast volume of photometric images. These images play a pivotal role in deriving fundamental parameters of celestial objects and investigating the structure of the universe. Nevertheless, in dense star fields, the characteristics of celestial sources are intricate, rendering traditional methods incapable of conducting precise analyses. To address this challenge, this paper introduces a new algorithm named DSDNet for detecting celestial sources in dense star fields and counting them. During the feature extraction phase, DSDNet generates larger feature maps, thereby preserving more information about small targets. By incorporating a convolutional attention module, the model's capacity to learn the features of celestial sources in dense star fields is augmented. Furthermore, to more effectively manage the blending phenomenon among sources, DSDNet integrates CNN and Transformer architectures, enhancing the model's ability to comprehend global features. Experimental findings show that DSDNet exhibits excellent performance, attaining an F1-score of 97.23%. This makes it a valuable resource for aiding SDSS in processing dense star field images.

Emotion recognition in conversations (ERC) is a pivotal component of affective computing, involving a common two-stage paradigm where pre-trained language models first extract context-independent features, followed by the encoding of contextual information and the modeling of emotional dependencies. This paradigm faces two challenges: (1) Existing methods struggle to capture both the intra-dialogue emotional continuity and the inter-dialogue semantic similarity. (2) The complexity of emotional elicitation processes gives rise to entangled dependencies, termed “latent dependencies”, which are difficult for current methods to detect and analyze. To overcome these challenges, we propose a Hybrid-Context Encoder with an Automated Latent Dependency Mining model for ERC. Specifically, we examine the emotional continuity and the semantic similarity from the standpoint of context encoders. We experimentally find that context encoders with different architectures exhibit distinct benefits. Based on these findings, we design a hybrid contextual encoding module that effectively combines the strengths of various encoders. Additionally, we design a lightweight generative module for latent dependency mining that autonomously generates a context mask, enabling the effective discovery of latent dependencies. We conduct extensive experiments on three datasets in the text modality. Our model achieves the best performance, which validates the superiority of our approach.

Human emotion synthesis is a crucial aspect of affective computing. It involves using computational methods to mimic and convey human emotions through various modalities, with the goal of enabling more natural and effective human-computer interactions. Recent advancements in generative models, such as Autoencoders, Generative Adversarial Networks, Diffusion Models, Large Language Models, and Sequence-to-Sequence Models, have significantly contributed to the development of this field. However, there is a notable lack of comprehensive reviews in this field. To address this problem, this paper aims to address this gap by providing a thorough and systematic overview of recent advancements in human emotion synthesis based on generative models. Specifically, this review will first present the review methodology, the emotion models involved, the mathematical principles of generative models, and the datasets used. Then, the review covers the application of different generative models to emotion synthesis based on a variety of modalities, including facial images, speech, and text. It also examines mainstream evaluation metrics. Additionally, the review presents some major findings and suggests future research directions, providing a comprehensive understanding of the role of generative technology in the nuanced domain of emotion synthesis.

Multimodal Sentiment Analysis (MSA) is an attractive research that aims to integrate sentiment expressed in textual, visual, and acoustic signals. There are two main problems in the existing methods: 1) the dominant role of the text is underutilization in unaligned multimodal data, and 2) the modality under uncertain missing feature is not sufficiently explored. This paper proposes a Text-guided Reconstruction Network (TgRN) for MSA with uncertain missing modalities in non-aligned sequences. The TgRN network includes three primary modules: Text-guided Extraction Module (TEM), Reconstruction Module (RM) and Text-guided Fusion Module (TFM). First, the TEM consists of the text-guided cross attention units and self-attention units to capture inter-modal features and intra-modal features, respectively. Second, leveraging enhanced attention units and a three-way squeeze-and-excitation block, the RM is designed to learn semantic information from incomplete data and reconstruct missing modality features. Third, the TFM utilizes a progressive modality-mixing adaptation gate to explore the dynamic correlations between nonverbal and verbal modalities, effectively addressing the modality gap issue. Finally, under the supervision of sentiment prediction loss and reconstruction loss, the TgRN effectively processes both uncertain missing-modality conditions and ideal complete modality conditions. Extensive experiments on CMU-MOSI and CH-SIMS demonstrate that our proposed method outperforms state-of-the-art approaches.

Facial Expression Recognition (FER) has received considerable research attention owing to its poor robustness in real-world scenarios. This issue, defined as the uncertainty problem in FER, is often solved by recognizing the noise samples in FER datasets. Unlike noise samples with incorrect labels, ambiguous samples exhibit mixed emotions that align with multiple basic expressions. It makes them indistinguishable in training and harms model robustness. To address this issue, we propose an ambiguity-aware FER framework called Co-dance with Ambiguity (CoA). CoA combines an Emotion Extraction Module (EEM) and an Expression Description Module (EDM) to leverage ambiguity for better performance and robustness. Specifically, EEM employs a coupled-stream structure to extract both representative and detailed features through diverse-scale fusion and patch-attention sensing. EDM adjusts ground-truth labels of ambiguous samples by introducing label pairs derived from the top two highest predictions, describing the mixed-emotion nature. The pairs guide the model to align feature extraction with the inherent ambiguity of ambiguous samples during training. Extensive experiments on five in-the-wild FER datasets demonstrate the superiority of CoA over advanced methods. Moreover, introducing ambiguity-aware strategies enriches feature representations and significantly enhances robustness when faced with a high ratio of ambiguous samples in FER.

The Bandler-Kohout subproduct (BKS) method acts as one of the two representative fuzzy relational inference (FRI) strategies. Observing the BKS method using constraint modeling, two fuzzy implications, respectively, produce expression to the factors of inference mechanism and rule base. However, these two factors normally reflect different connotations from the perspectives of artificial intelligence applications and logical meaning. Enlightened by such idea, in this study, we propose and investigate the differently implicational BKS (DBKS) method. Initially, main properties of DBKS are validated. The reversibility and interpolativity of DBKS are proved under certain conditions. The equivalent relationship is verified between interpolativity and continuity for DBKS. The robustness of DBKS is confirmed from both the similarity and the extensional hull. Posteriorly, the computational performance of DBKS is analyzed. In DBKS, the preservation of the indistinguishability holds for input fuzzy sets, and it is proved that the first-aggregate-then-infer (FATI) reasoning strategy of DBKS is equivalent to the first-infer-then-aggregate (FITA) one. To improve the computational efficiency, the hierarchical DBKS method is presented. In addition, the fuzzy system is established on the strength of the DBKS method, the singleton fuzzifier and the centroid defuzzifier. Its response function is analyzed and a universal approximator is built by the fuzzy system via DBKS. At the end, we compare the results of DBKS with BKS by virtue of two examples in affective computing. It is discovered that DBKS can create superior forms of FRI in comparison to those produced by BKS.

Aspect-based Sentiment Analysis (ABSA) is a challenging and attractive fine-grained subtask in the Natural Language Processing (NLP) community, which aims to predict the sentiment polarity of each involved aspect term in the given sentence. With the numerous efforts contributed by related researchers, impressive success has been achieved in employing the external knowledge to enhance the sentiment representation. However, in previous works, the utilization of external knowledge is quite straightforward, which doesn't match the sentiment characteristics of human beings. Thus, this work proposes a novel humanized ABSA method, namely Fuzzy Humanized Sentiment-assisted Graph Attention Networks (FHSGAT). Preliminary, to acquire the essential semantic features, the sentence is re-constructed by the manually designed prompt template, which would assist the Pre-trained Language Model (PLM) in generating the target-guided semantic representation. Additionally, based on the normal dependency graph, triple-dimensional external knowledge is injected into the corresponding nodes. Specifically, this paper leverages the Gaussian membership function to fuzzify the sentiment knowledge-enhanced nodes, enabling the networks to obtain the humanized sentiment relationship between graph nodes, namely fuzzy humanized syntactic features. Besides, due to its exquisite performance in information fusion, Graph Attention Networks (GAT) is utilized to accomplish the integration of target-guided semantic representation and fuzzy humanized syntactic features synchronously. Eventually, to validate the effectiveness of the proposed method FHSGAT, extensive experiments are conducted on five publicly and widely used benchmarks, and the corresponding results show the performance of the proposed approach outperforms the state-of-the-art (SOTA) obviously.

Social recommendations leverage social networks to augment the performance of recommender systems. However, the critical task of denoising social information has not been thoroughly investigated in prior research. In this study, we introduce a hierarchical denoising robust social recommendation model to tackle noise at two levels: 1) intra-domain noise, resulting from user multi-faceted social trust relationships, and 2) inter-domain noise, stemming from the entanglement of the latent factors over heterogeneous relations (e.g., user-item interactions, user-user trust relationships). Specifically, our model advances a preference and social psychology-aware methodology for the fine-grained and multi-perspective estimation of tie strength within social networks. This serves as a precursor to an edge weight-guided edge pruning strategy that refines the model's diversity and robustness by dynamically filtering social ties. Additionally, we propose a user interest-aware cross-domain denoising gate, which not only filters noise during the knowledge transfer process but also captures the high-dimensional, nonlinear information prevalent in social domains. We conduct extensive experiments on three real-world datasets to validate the effectiveness of our proposed model against state-of-the-art baselines. We perform empirical studies on synthetic datasets to validate the strong robustness of our proposed model.

Multimodal sentiment analysis has garnered increasing attention. The bulk of existing work in multimodal sentiment analysis primarily focuses on designing various networks to align and subsequently fuse representations from individual modalities. Contrastive learning, recognized for its intrinsic alignment capabilities, has also been extensively applied in multimodal sentiment analysis. However, current contrastive learning methods are often limited to pairwise modalities and typically perform contrastive learning prior to modality fusion, neglecting the consistency of interactions across multiple modalities. Moreover, they overlook the overall consistency within samples. To address these issues, we introduce a novel Multi-Level Contrastive Learning (MLCL) framework for multimodal sentiment analysis, composed of Uni-Modal Contrastive Learning (UMCL), Bi-Modal Contrastive Learning (BMCL) and Tri-Modal Contrastive Learning (TMCL). UMCL enhances intra-modal representations by creating positive pairs using modality-specific random dropout, while BMCL leverages the asymmetry of attention mechanisms, using two directional attentions as positive samples. TMCL aligns non-overlapping uni-modal and bi-modal representations, underscoring the complementarity of tri-modal information. The effectiveness of MLCL is demonstrated through its performance on multiple datasets. Our comprehensive experiments across multiple datasets demonstrate the superiority of the MLCL framework, which achieves new state-of-the-art performance.

Unlike typical video action recognition, Dynamic Facial Expression Recognition (DFER) does not involve distinct moving targets but relies on localized changes in facial muscles. Addressing this distinctive attribute, we propose a MultiScale Spatio-temporal CNN-Transformer network (MSST-Net). Our approach takes spatial features of different scales extracted by CNN and feeds them into a Multi-scale Embedding Layer (MELayer). The MELayer extracts multi-scale spatial information and encodes these features before sending them into a Temporal Transformer (T-Former). The T-Former simultaneously extracts temporal information while continually integrating multi-scale spatial information. This process culminates in the generation of multi-scale spatio-temporal features that are utilized for the final classification. Our method achieves state-of-the-art results on two in-the-wild datasets. Furthermore, a series of ablation experiments and visualizations provide further validation of our approach's proficiency in leveraging spatio-temporal information within DFER.

Addressing the challenge in Facial Expression Recognition (FER) of majority expressions like happiness and neutral overshadowing minority ones, we introduce K-positive Face Net (K-Face Net), a two-stage framework enhancing diverse expression recognition. In its representation learning stage, K-Face Net employs a novel K-positive loss to select K same-class samples as positive groups, reducing the distance of features between the anchor and the groups and achieving compact clustering. The subsequent classification stage utilizes Spatial Pooling across spatial and global dimensions, increasing inter-class feature distances for balanced recognition. Extensive experiments demonstrate that K-Face Net makes better performance in minority categories and achieves state-of-the-art on three popular datasets in both average and overall accuracy. K-Face Net provides a new perspective in enhancing the recognition of underrepresented facial expressions in real-world scenarios.

Recent research on facial expression recognition (FER) has achieved significant performance on FER datasets. However, in real-world recognition scenarios, performance is often compromised by occlusion, particularly after the COVID-19. To address this issue, we propose a novel framework named OCLIPER, an visual-language model designed to enhance occluded facial expression recognition. Specifically, Our approach consists of two parts: 1) Visual Part: We initially generated realistic occlusions commonly encountered in daily life, such as masks, glasses, and hands, by incorporating them into FER datasets. Subsequently, we introduce a similarity loss between original facial images and occluded facial images. This guides the image encoder to learn a robust facial representation insensitive to occlusion. 2) Text Part: Learnable prompts and the Occluded Facial Expression Descriptor (OFED) are utilized as inputs to the text encoder. Learnable prompts assist the model in understanding relevant context information for each expression. OFED comprises a series of text descriptions of facial expressions behind occlusion, generated by ChatGPT. Ultimately, the text part guides the image part in learning occlusion-resistant features. Experimental results on various databases demonstrate the superiority of our proposed method over state-of-the-art approaches.

Facial Expression Recognition (FER) poses significant challenges due to various imaging conditions, including diverse head poses, lighting conditions, resolutions, and occlusions. Additionally, different personal attributes such as age, gender, and racial background further contribute to the complexity of FER. To accurately extract meaningful expression features amidst these interfering factors to enhance recognition accuracy and the model’s generalization, we propose a Self Decoupling-Reconstruction Network (SDRNet). Specifically, our approach involves two learning processes. In the first phase, the network is trained to decouple facial images with expressions into expression and neutral components. This process involves reconstructing neutral facial images and the original input, ensuring the preservation of meaningful expression components devoid of interference in the decoupling process. In the second learning phase, we employ simple convolutional neural networks (CNNs) to recognize the extracted expression components. Our method has achieved state-of-the-art results across multiple widely used datasets, providing substantial evidence of its effectiveness. Additionally, we demonstrate the robust generalization performance of our approach through cross-database evaluations.

Multimodal sentiment analysis (MSA) aims to integrate multiple modalities of information to better understand human sentiment. The current research mainly focuses on conducting multimodal fusion, which neglects the under-optimized modal representations generated by the imbalance of unimodal performances in joint learning. Moreover, the size of labeled datasets limits the generalization ability of existing supervised models. To address the above issues, this paper proposes a knowledge-enhanced self-supervised balanced representation approach (KEBR). First, a text-based cross-modal fusion method (TCMF) is constructed, which injects the non-verbal information from the videos into the semantic representation of text to enhance the multimodal representation of text. Then, a multimodal cosine constrained loss (MCC) is designed to constrain the fusion of non-verbal information in joint learning to balance the representation. Finally, with the help of sentiment knowledge and non-verbal information, KEBR conducts sentiment word masking and sentiment intensity prediction. Experimental results show that KEBR outperforms the baseline.

Multimodal Sentiment Analysis (MSA) has witnessed remarkable progress and gained increasing attention in recent decade. However, current MSA methodologies primarily rely on global representations extracted from different modalities, such as the mean of all token representations, to construct sophisticated fusion networks. These approaches often overlook the valuable details present in local representations, which consist of fused representations of consecutive several tokens. Additionally, the integration of multiple local representations, and the fusion of local and global information present significant challenges. To address these limitations, we propose the Global-Local Modal (GLoMo) Fusion framework. It comprises two essential components: (i) modality-specific mixture of experts layers that integrate diverse local representations within each modality, and (ii) a global-guided fusion module that effectively combines global and local representations. The former component leverages specialized expert networks to automatically select and integrate crucial local representations from each modality, while the latter ensures the preservation of global information during the fusion process. We evaluate GLoMo on various datasets, encompassing tasks in multimodal sentiment analysis, multimodal humor detection, and multimodal emotion recognition. Extensive experiments demonstrate that GLoMo outperforms existing state-of-the-art models, validating the effectiveness of our proposed framework. Our code is publicly available at https://github.com/YetZzzzzz/GLoMo.

Prompt engineering aims to adapt an AI foundation model on the token level without weight updating. Recently, with the development of visual models, many researchers have begun to study visual generation quality improvement using prompt engineering. However, while those studies mainly aim to improve visual quality, they overlook the safe factors in prompts. We find that adding specific camera descriptions not only prevents these issues but also enhances visual quality. Consequently, we propose a simple and safe prompt engineering method (SSP) to improve visual generation quality by providing optimal camera descriptions. Specifically, we create a dataset from multi-datasets as original prompts. To select the optimal camera, we design an optimal camera matching approach and implement a classifier for original prompts capable of automatically matching. Appending camera descriptions to original prompts generates optimized prompts for further visual generation. Experiments demonstrate that SSP improves semantic consistency by an average of 16 % compared to others and safety metrics by 35.8%.

Active Learning (AL) is a technique being widely employed to minimize the time and labor costs in the task of annotating data. By querying and extracting the specific instances to train the model, the relevant task’s performance is improved maximally within limited iterations. However, rare work was conducted to fully fuse features from different hierarchies to enhance the effectiveness of active learning. Inspired by the thought of information compensation in many famous deep learning models (such as ResNet, etc.), this work proposes a novel TextCNN-based Two ways Active Learning model (TCTWAL) to extract task-relevant texts. TextCNN takes the advantage of little hyper-parameter tuning and static vectors and achieves excellent results on various natural language processing (NLP) tasks, which are also beneficial to human-computer interaction (HCI) and the AL relevant tasks. In the process of the proposed AL model, the candidate texts are measured from both global and local features by the proposed AL framework TCTWAL depending on the modified TextCNN. Besides, the query strategy is strongly enhanced by maximum normalized log-probability (MNLP), which is sensitive to detecting the longer sentences. Additionally, the selected instances are characterized by general global information and abundant local features simultaneously. To validate the effectiveness of the proposed model, extensive experiments are conducted on three widely used text corpus, and the results are compared with with eight manual designed instance query strategies. The results show that our method outperforms the planned baselines in terms of accuracy, macro precision, macro recall, and macro F1 score. Especially, to the classification results on AG’s News corpus, the improvements of the four indicators after 39 iterations are 40.50%, 45.25%, 48.91%, and 45.25%, respectively.

Automatic medical image segmentation method is highly needed to help experts in lesion segmentation. The deep learning technology emerging has profoundly driven the development of medical image segmentation. While U-Net and attention mechanisms are widely utilized in this field, the application of attention, albeit successful in natural scene image segmentation, tends to inflate the number of model parameters and neglects the potential for feature fusion between different convolutional layers. In response to these challenges, we present the Multi-Attention and Depthwise Separable Convolution U-Net (MDSU-Net), designed to enhance feature extraction. The multi-attention aspect of our framework integrates dual attention and attention gates, adeptly capturing rich contextual details and seamlessly fusing features across diverse convolutional layers. Additionally, our encoder integrates a depthwise separable convolution layer, streamlining the model’s complexity without sacrificing its efficacy, ensuring versatility across various segmentation tasks. The results demonstrate that our method outperforms state-of-the-art across three diverse medical image datasets.

For text sentiment analysis, state-of-the-art neural language models have demonstrated promising performance. However, they lack interpretability, require vast volumes of annotated data, and are typically specialized for tasks. In this paper, we explore a connection between fine-tuned Transformer models and unsupervised LDA approach to cope with text sentiment analysis tasks, inspired by the concept of Neuro-symbolic AI. The Transformer and LDA models are combined as a feature extractor to extract the hidden representations of the input text sequences. Subsequently, we employ a feedforward network to forecast various sentiment analysis tasks, such as multi-label emotion prediction, dialogue quality prediction, and nugget detection. Our proposed method obtains the best results in the NTCIR-16 dialogue evaluation (DialEval-2) task, as well as cutting-edge results in emotional intensity prediction using the Ren_CECps corpus. Extensive experiments show that our proposed method is highly explainable, cost-effective in training, and superior in terms of accuracy and robustness.

The blossoming semi-automated vehicles allow drivers to engage in various non-driving-related tasks, which may stimulate diverse emotions, thus affecting takeover safety. Though the effects of emotion on takeover behavior have recently been examined, how to effectively obtain and utilize drivers' emotions for predicting takeover behavior remains largely unexplored. We propose EmoTake, a deep learning-empowered system that explores drivers' emotional and physical states to predict takeover readiness, reaction time, and quality. The key enabler is a deep neural framework that extracts drivers' fine-grained body movements from a camera and interprets them into drivers' multi-channel emotional and physical information (e.g., facial expression, and head pose) for prediction. Our study (N = 26) verifies the efficiency of EmoTake and shows that: 1) facial expression benefits prediction; 2) emotions have diverse impacts on takeovers. Our findings provide insights into takeover prediction and in-vehicle emotion regulation.

Textual emotion detection is playing an important role in the human-computer interaction domain. The mainstream methods of textual emotion detection are extracting semantic features and fine-tuning by language models. Due to the information redundancy in semantics, it is difficult for these methods to accurately detect all the emotions implied in the text. The prompting method has been shown to make the language models more purposeful in prediction by filling the cloze or prefix prompts defined. Therefore, we design a prompting method for multi-label classification. To stabilize the output, we design two consistency training strategies. We experiment on two multi-label emotion classification datasets: Ren-CECps and NLPCC2018. Our proposed prompting method with consistency training strategies for multi-label textual emotion detection (PC-MTED) model achieves state-of-the-art Macro F1 scores of 0.5432 and 0.5269, respectively. The experimental results indicate that our proposed method is effective in the multi-label textual emotion detection task.

Human speech can be characterized by different components, including semantic content, speaker identity and prosodic information. Significant progress has been made in disentangling representations for semantic content and speaker identity in Automatic Speech Recognition (ASR) and speaker verification tasks respectively. However, it is still an open challenging research question to extract prosodic information because of the intrinsic association of different attributes, such as timbre and rhythm, and because of the need for supervised training schemes to achieve robust large-scale and speaker-independent ASR. The aim of this paper is to address the disentanglement of emotional prosody from speech based on unsupervised reconstruction. Specifically, we identify, design, implement and integrate three crucial components in our proposed speech reconstruction model Prosody2Vec: (1) a unit encoder that transforms speech signals into discrete units for semantic content, (2) a pretrained speaker verification model to generate speaker identity embeddings, and (3) a trainable prosody encoder to learn prosody representations. We first pretrain the Prosody2Vec representations on unlabelled emotional speech corpora, then fine-tune the model on specific datasets to perform Speech Emotion Recognition (SER) and Emotional Voice Conversion (EVC) tasks. Both objective (weighted and unweighted accuracies) and subjective (mean opinion score) evaluations on the EVC task suggest that Prosody2Vec effectively captures general prosodic features that can be smoothly transferred to other emotional speech. In addition, our SER experiments on the IEMOCAP dataset reveal that the prosody features learned by Prosody2Vec are complementary and beneficial for the performance of widely used speech pretraining models and surpass the state-of-the-art methods when combining Prosody2Vec with HuBERT representations. Some audio samples can be found on our demo website

Behavior detection receives considerable attention in real-life human–computer interaction, where the complexity of background information and the variable durations of movements are two major factors affecting the accuracy of behavior detection. To overcome the inadequacy of these factors, this article proposes an enhancing region convolutional 3-D (ER-C3D) network with adaptive shrinkage and symmetrical multiscale for behavior detection. The improved ER-C3D network includes a feature subnet, a proposal subnet, and a classification subnet. First, a 3D-RSST unit is constructed by embedding an adaptive shrinkage structure and a soft thresholding operation. Meanwhile, a residual adaptive shrinkage mechanism, composed of multiple cascaded 3D-RSST units with different parameters, is designed to reduce redundant information of video streams in the feature subnet. Second, a spatiotemporal symmetrical multiscale structure is substituted for the single-layer convolution and embedded into the proposal subnet. Specially, contextual symmetrical multiscale motion characteristics with different levels and granularities are acquired by expanding the spatiotemporal receptive field of candidate temporal proposals. Finally, a soft-nonmaximal suppression strategy is introduced to filter high-quality temporal proposals in the classification subnet. The experimental results on the THUMOS’14 and ActivityNet1.2 datasets indicate that the [email protected] of the improved ER-C3D network reaches 39.4% and 42.2%, respectively, which is 10.5% and 15.4% higher than R-C3D. Compared with related methods, the proposed method shows improvement in both the positional precision of behavioral boundary and the accuracy of behavioral classification.

Emotion is the vital path to achieving strong artificial intelligence. Therefore, it is significant to study the emotional guiding and controlling theory to enhance system intelligence. Conversational datasets in social media contain useful information, including individuals exhibiting continuously changing emotion states in response to external stimuli, and are the foundation for research on artificial emotions. We define dialog-based emotional guidance in reinforcement learning to research the emotional guidance of the discrete emotion model. This article proposes three strategies to obtain the optimal policy: 1) given the current emotional transition matrix, use the emotional Markov decision process (E-MDP) algorithm to calculate the optimal stimulus policy for each target emotion; 2) given the emotional transition sequences, use the emotional Monte Carlo algorithm to calculate the optimal stimulus policy; and 3) given the emotional transition sequences, use the$Q$-learning algorithm to calculate the optimal stimulus policy. Besides, we improve the Markov chain Monte Carlo algorithm to sample emotional transition sequences and design a metric to evaluate the effectiveness of policies. Experimental results on three datasets show that these methods can more effectively guide emotion than traditional methods. Particularly, E-MDP achieves the best results while others can be more widely used in real-world scenarios.

In existing granular clustering algorithms, the design of coverage and specificity does not fully capture the inherent structural characteristics of granular data together with the optimization issue, and the current weight setting for the granular data is not sufficient. To address these problems, in this study, the trapezoidal information granule, which is rarely studied before, is concentrated, and we come up with a novel granular clustering algorithm called the weighted possibilistic fuzzy c-means algorithm for trapezoidal granularity (WPFCM-T). Firstly, under the acknowledged principle of justifiable granularity, novel functions of coverage and specificity are designed for trapezoidal information granules, considering the internal characteristics of such granules. The idea of particle swarm optimization (PSO) is exploited to upgrade the established granular data, and then the trapezoidal information granule construction (TIGC) method is proposed to realize granular modeling. Secondly, an exponential weight is constructed with regard to coverage and specificity, while a novel distance via α-cuts is given. The possibilistic fuzzy c-means (PFCM) structure is introduced into granular clustering, in which the new weight and distance are integrated, resulting in the proposed WPFCM-T algorithm. Thirdly, the reconstruction criterion is studied to evaluate granular clustering, and hence an overall framework including granular modeling, clustering and evaluation is constructed. Lastly, through experiments completed on artificial datasets, UCI datasets, large datasets, high-dimensional datasets, and noisy datasets, WPFCMT has superior granular data reconstruction ability by contrast with other granular clustering algorithms, indicating that the granular clustering performance of WPFCM-T is better than the others.

Human Multimodal Sentiment Analysis (MSA) is an attractive research that studies sentiment expressed from multiple heterogeneous modalities. While transformer-based methods have achieved great success, designing an effective “co-attention” model to associate text modality with nonverbal modalities remains challenging. There are two main problems: 1) the dominant role of the text in modalities is underutilization, and 2) the interaction between modalities is not sufficiently explored. This paper proposes a deep modular Co-Attention Shifting Network (CoASN) for MSA. A Cross-modal Modulation Module based on Co-attention (CMMC) and an Advanced Modality-mixing Adaptation Gate (AMAG) are constructed. The CMMC consists of the Text-guided Co-Attention (TCA) and Interior Transformer Encoder (ITE) units to capture inter-modal features and intra-modal features. With text modality as the core, the CMMC module aims to guide and promote the expression of emotion in nonverbal modalities, and the nonverbal modalities increase the richness of the text-based multimodal sentiment information. In addition, the AMAG module is introduced to explore the dynamical correlations among all modalities. Particularly, this efficient module first captures the nonverbal shifted representations and then combines them to calculate the shifted word embedding representations for the final MSA tasks. Extensive experiments on two commonly used datasets, CMU-MOSI and CMU-MOSEI, demonstrate that our proposed method is superior to the state-of-the-art performance.

Social networks exhibit a distinct "celebrity effect" whereby influential individuals have a more significant impact on others compared to ordinary individuals, unlike other network structures such as citation networks and knowledge graphs. Despite its common occurrence in social networks, the celebrity effect is frequently overlooked by existing social recommendation methods when modeling social relationships, thereby hindering the full exploitation of social networks to mine similarities between users. In this paper, we fill this gap and propose a Celebrity-aware Graph Contrastive Learning Framework for Social Recommendation (CGCL), which explicitly models the celebrity effect in the social domain. Technically, we measure the different influences of celebrity and ordinary nodes by mining social network structure features, such as closeness centrality. To model the celebrity effect in social networks, we design a novel user-user impact-aware aggregation method, which incorporates the celebrity-aware influence information into the message propagation process. Additionally, we design a graph neural network-based framework which incorporates social semantics into the user-item interaction modeling with contrastive learning-enhanced data augmentation. The experimental results on three real-world datasets show the effectiveness of the proposed framework. We conduct ablation experiments to prove that the key components of our model benefit the recommendation performance improvement.

As medical image analysis equipment has evolved and gained popularity, MRI has taken the forefront in radiological imaging. Since 3D medical images are more complex and contextual features are more difficult to capture, Transformer needs to be introduced to enhance the feature extraction capabilities of the network. We propose Dual Encoder U-Net (DEU-Net), which uses Transformer and CNN to extract medical image features in the encoder. Transformer is a pre-trained model in BTCV, which improves its ability to capture contextual features of medical images and increases the learning speed. To fuse the two kinds of features, we propose a Dual Feature Fusion Module (DFFM) to fuse the features extracted from the Transformer and CNN respectively, making full use of the feature extraction capabilities of the two extractors for 3D medical image. The results demonstrate that DEU-Net outperforms state-of-the-art for three segmentation tasks on the BraTS 2020 dataset.

Abstract We propose an Inner-Imaging three-dimensional (3D) attentional feature fusion module for a residual network, which is a simple yet effective approach for residual networks. In our attention module, we constructed a 3D soft attention feature map to refine the input feature. The map fuses the attentional features from different dimensions, including channel and spatial axes, to create a 3D attention map. Then, we implemented a feature fusion module to further fuse the attentional features. Lastly, the attention module outputs a 3D soft attention map that is applied to the residual branch. The attention module can also model the relationship between attentional features from different dimensions and achieve the interaction between attentional features. This function allows our attention module to acquire more attentional features. To demonstrate the effectiveness of our method, extensive experiments were conducted on several computer vision benchmark datasets, including ImageNet 2012 and Microsoft COCO (MS COCO) 2017 datasets. The experimental results show that our method performed better than the baseline methods in the tasks of image classification, object detection, and instance segmentation tasks.

Abstract The detection of river boundaries is a crucial branch of the intelligent perception of unmanned surface vehicles (USVs), it can be used to determine the driving areas of USVs, and also to ensure driving safety by limiting the effective drivable areas of USVs in the river areas. Aiming to detect the boundaries of incompletely structured river channels, this study proposes a real‐time detection method for river boundaries based on a Light Detection and Ranging (LiDAR) sensor. The point clouds that are disturbed by the water surface noise are filtered firstly, and then the spatial and geometric features are extracted separately from the point cloud detected above the water surface. To prevent the error detection and missing detection, the boundary point information is predicted and calibrated in real time by Extended Kalman Filter (EKF). A planning track generation algorithm for coastal autonomous cruise without relying on high‐precision maps, and a heading and distance adaptive control method by Proportional‐Integral‐Derivative (PID), and different driving line generation methods for driving along the narrow river and wide river are proposed respectively. The experimental data verification of river boundary detection shows that the algorithm is accurate, real‐time, and robust.

Textual language is the most natural carrier of human emotion. In natural language processing, textual emotion recognition (TER) has become an important topic due to its significant academic and commercial potential. With the advanced development of deep learning technologies, TER has attracted growing attention and has significantly been promoted in recent years. This article provides a systematic survey of the latest TER advances, focusing on approaches using deep neural networks. According to how deep learning works at each stage, TER approaches are reviewed on word embedding, architecture, and training levels, respectively. We discussed the remaining challenges and opportunities from four aspects: the shortage of large-scale and high-quality datasets, fuzzy emotional boundaries, incomplete extractable emotional information in texts, and TER in dialogue. This article creates a systematic and in-depth overview of deep TER technologies. It provides the necessary knowledge and new insights for relevant researchers to understand better the research state, remaining challenges, and future directions in this field.

Textual emotion detection is an attractive task while previous studies mainly focused on polarity or single-emotion classification. However, human expressions are complex, and multiple emotions often co-occur with non-negligible emotion correlations. In this paper, a Multi-label Emotion Detection Architecture (MEDA) is proposed to detect all associated emotions expressed in a given piece of text. MEDA is mainly composed of two modules: Multi-Channel Emotion-Specified Feature Extractor (MC-ESFE) and Emotion Correlation Learner (ECorL). MEDA captures underlying emotion-specified features through MC-ESFE module, which is composed of multiple channel-wise ESFE networks. Each channel in MC-ESFE is devoted to the feature extraction of a specified emotion from sentence-level to context-level through a hierarchical structure. With underlying features, emotion correlation learning is implemented through an emotion sequence predictor in ECorL. Furthermore, we define a new loss function: multi-label focal loss. With this loss function, the model can focus more on misclassified positive-negative emotion pairs and improve the overall performance by balancing the prediction of positive and negative emotions. The evaluation of proposed MEDA architecture is carried out on emotional corpus: RenCECps and NLPCC2018 datasets. The experimental results indicate that the proposed method can achieve better performance than state-of-the-art methods in this task.

Emotion plays a critical role in making the computer more human-like. As the first and most essential step, emotion recognition emerges recently as a hot but relatively nascent topic, i.e., current research mainly focuses on single modality (e.g., facial expression) while human emotion expressions are multi-modal in nature. To this end, we propose an unobtrusive emotion recognition system leveraging two emotion-rich and tightly-coupled modalities, i.e., gesture and facial expression. The system design faces two major challenges, namely, how to capture the emotional expression in both modalities without disturbing the subject and how to leverage the relationship between modalities for recognizing the emotion. For the former, we explore WiFi and vision for unobtrusive and contactless gesture and facial expression sensing, respectively. For the latter, we propose a novel deep learning framework named Multi-Source Learning (MSL) to efficiently exploit both self-correlation in the modality and cross-correlation between modalities for fine-grained emotion recognition. To evaluate the proposed method, we prototype the system on low-cost commodity WiFi and vision devices, build a first-of-its-kind WiFi-Vision emotion dataset, and conduct extensive experiments. Empirical results not only verify the effectiveness of WiFE in emotion recognition, but also confirm the superiority of multi-modality over single-modality.

High-quality corpora have been very scarce for the text emotion research. Existing corpora with multi-label emotion annotations have been either too small or too class-biased to properly support a supervised emotion learning. In this article, we propose a novel active learning method for efficiently instructing the human annotations for a less-biased and high-quality multi-label emotion corpus. Specifically, to compensate annotation for the minority-class examples, we propose a complementary sampling strategy based on unlabeled resources by measuring a probabilistic distance between the expected emotion label distribution in a temporary corpus and an uniform distribution. Qualitative evaluations are also given to the unlabeled examples, in which we evaluate the model uncertainties for multi-label emotion predictions, their syntactic representativeness for the other unlabeled examples, and their diverseness to the labeled examples, for a high-quality sampling. Through active learning, a supervised emotion classifier gets progressively improved by learning from these new examples. Experiment results suggest that by following these sampling strategies we can develop a corpus of high-quality examples with significantly relieved bias for emotion classes. Compared to the learning procedures based on traditional active learning algorithms, our learning procedure indicates the most efficient learning curve and estimates the best multi-label emotion predictions.

Neural machine translation has achieved remarkable progress over the past several years; however, little attention has been paid to machine translation (MT) between Japanese and Chinese, which share a large proportion of cognate words that can be utilized as additional linguistic knowledge to enhance translation performance. In this article, we seek to strengthen the semantic correlation between Japanese and Chinese by leveraging cognate words that share common Chinese characters. Specifically, we experiment with three strategies: (1) a shared vocabulary with cognate lexicon induction, which models the commonality between source and target cognates; (2) a shared private representation with a dynamic gating mechanism, which models the language-specific features on the source side; and (3) an embedding shortcut, which enables the decoder to access the shared private representation with shortest distance and aids the training process. The experiments and analysis presented in this article demonstrate that our proposed approaches can significantly improve the performance of both Japanese-to-Chinese and Chinese-to-Japanese translations and verify the effectiveness of exploiting Japanese–Chinese cognates for MT.

Word alignment is an important task of detecting translation equivalents between a sentence pair. Although word alignment is no longer necessarily needed for neural machine translation, it’s still useful in a wealth of applications, e.g., bilingual lexicon induction, constraint decoding, and so on. However, the most well-known word aligners are still Giza++ and fastAlign, both of which are implementations of traditional IBM models. To keep pace with the advance in NMT, there has been a surge of interest in replacing the IBM models with neural models. We follow this trend but aim to boost performance of word alignment between Japanese and Chinese, which share a large portion of Chinese characters. Our key idea is to leverage these common Chinese characters in both languages as an indicator for inferring alignment; i.e., the source and target words with the common Chinese characters should be most likely aligned. Following this idea, we propose three methods that leverage common Chinese characters to boost the mBERT-based word alignment, including reward factor, representation alignment, and contrastive training. Furthermore, we annotate and release a golden dataset for Japanese-Chinese word alignment. Experiments on the dataset show that our methods outperform several strong baselines in terms of AER score and verify the effectiveness of exploiting common Chinese characters.

Speech emotion recognition has always been one of the topics that have attracted a lot of attention from many researchers. In traditional feature fusion methods, the speech features used only come from the data set, and the weak robustness of features can easily lead to overfitting of the model. In addition, these methods often use simple concatenation to fuse features, which will cause the loss of speech information. In this paper, to solve the above problems and improve the recognition accuracy, we utilize self-supervised learning to enhance the robustness of speech features and propose a feature fusion model(Dual-TBNet) that consists of two 1D convolutional layers, two Transformer modules and two bidirectional long short-term memory (BiLSTM) modules. Our model uses 1D convolution to take features of different segment lengths and dimension sizes as input, uses the attention mechanism to capture the correspondence between the two features, and uses the bidirectional time series module to enhance the contextual information of the fused features. We designed a total of four fusion models to fuse five pre-trained features and acoustic features. In the comparison experiments, the Dual-TBNet model achieved a recognition accuracy and F1 score of 95.7% and 95.8% on the CASIA dataset, 66.7% and 65.6% on the eNTERFACE05 dataset, 64.8% and 64.9% on the IEMOCAP dataset, 84.1% and 84.3% on the EMO-DB dataset and 83.3% and 82.1% on the SAVEE dataset. The Dual-TBNet model effectively fuses acoustic features of different lengths and dimensions with pre-trained features, enhancing the robustness of the features, and achieved the best performance.

Facial Expression Recognition (FER) has recently emerged as a crucial area in Human-Computer Interaction (HCI) system for understanding the user’s inner state and intention. However, feature- and label-noise constitute the major challenge for FER in the wild due to the ambiguity of facial expressions worsened by low-quality images. To deal with this problem, in this paper, we propose a simple but effective Facial Expression Noise-tolerant Network (FENN) which explores the inter-class correlations for mitigating ambiguity that usually happens between morphologically similar classes. Specifically, FENN leverages a multivariate normal distribution to model such correlations at the final hidden layer of the neural network to suppress the heteroscedastic uncertainty caused by inter-class label noise. Furthermore, the discriminative ability of deep features is weakened by the subtle differences between expressions and the presence of feature noise. FENN utilizes a feature-noise mitigation module to extract compact intra-class feature representations under feature noise while preserving the intrinsic inter-class relationships. We conduct extensive experiments to evaluate the effectiveness of FENN on both original annotated images and synthetic noisy annotated images from RAF-DB, AffectNet, and FERPlus in-the-wild facial expression datasets. The results show that FENN significantly outperforms state-of-the-art FER methods.

The existing clustering validity indexes (CVIs) show some difficulties to produce the correct cluster number when some cluster centers are close to each other, and the separation processing mechanism appears simple. The results are imperfect in case of noisy data sets. For this reason, in this study, we come up with a novel CVI for fuzzy clustering, referred to as the triple center relation (TCR) index. The originality of this index is twofold. On the one hand, a new fuzzy cardinality is built on the strength of the maximum membership degree, and a novel compactness formula is constructed by combining it with the within-class weighted squared error sum. On the other hand, starting from the minimum distance between different cluster centers, the mean distance as well as the sample variance of cluster centers in the statistical sense are further integrated. These three factors are combined by means of product to form a triple characterization of the relationship between cluster centers, and hence a 3-D expression pattern of separability is formed. Subsequently, the TCR index is put forward by combining the compactness formula with the separability expression pattern. By virtue of the degenerate structure of hard clustering, we show an important property of the TCR index. Finally, based on the fuzzy C -means (FCMs) clustering algorithm, experimental studies were conducted on 36 data sets (incorporating artificial and UCI data sets, images, the Olivetti face database). For comparative purposes, 10 CVIs were also considered. It has been found that the proposed TCR index performs best in finding the correct cluster number, and has excellent stability.

Vital sign (breathing and heartbeat) monitoring is essential for patient care and sleep disease prevention. Most current solutions are based on wearable sensors or cameras; however, the former could affect sleep quality, while the latter often present privacy concerns. To address these shortcomings, we propose Wital, a contactless vital sign monitoring system based on low-cost and widespread commercial off-the-shelf (COTS) Wi-Fi devices. There are two challenges that need to be overcome. First, the torso deformations caused by breathing/heartbeats are weak. How can such deformations be effectively captured? Second, movements such as turning over affect the accuracy of vital sign monitoring. How can such detrimental effects be avoided? For the former, we propose a non-line-of-sight (NLOS) sensing model for modeling the relationship between the energy ratio of line-of-sight (LOS) to NLOS signals and the vital sign monitoring capability using Ricean K theory and use this model to guide the system construction to better capture the deformations caused by breathing/heartbeats. For the latter, we propose a motion segmentation method based on motion regularity detection that accurately distinguishes respiration from other motions, and we remove periods that include movements such as turning over to eliminate detrimental effects. We have implemented and validated Wital on low-cost COTS devices. The experimental results demonstrate the effectiveness of Wital in monitoring vital signs.

The recent work Unsupervised Linear Discriminant Analysis (Un-LDA) completes its clustering process during the alternating optimization by converting equivalently the objective and finally using the K-means algorithm. However, the K-means algorithm has its inherent drawbacks. It is hard for the K-means algorithm to deal well with some complex clustering cases where there are too many real clusters or non-convex clusters. In this paper, a novel clustering optimization method is presented to accomplish the clustering process in Un-LDA and the resulting method can be named Un-LDA(CD). Specifically, instead of the K-means algorithm, an elaborately designed coordinate descent algorithm is adopted to obtain the clusters after the objective function goes through a series of simple but deft equivalent conversions. Extensive experiments have demonstrated that the coordinate descent clustering solution for Un-LDA can outperform the original K-means based solution on the tested data sets especially those complex data sets with a pretty large number of real clusters.

Label-noise constitutes a major challenge for facial expression recognition in the wild due to the ambiguity of facial expressions worsened by low-quality images. To deal with this problem, we propose a simple but effective Label-noise Robust Network (LRN) which explores the inter-class correlations for mitigating ambiguity that usually happens between morphologically similar classes. Specifically, LRN leverages a multivariate normal distribution to model such correlations at the final hidden layer of the neural network to suppress the heteroscedastic uncertainty caused by inter-class label noise. Furthermore, LRN utilizes a confidence-based label-free loss to extract compact intra-class feature representations under label noise while preserving the intrinsic inter-class relationships. Experiments on three in-the-wild facial expression datasets demonstrates the superiority of our method.

Speech emotion recognition has been developed rapidly in recent decades because of the appearance of machine learning. Nevertheless, lack of corpus remains a significant issue. For actual speech emotion corpus construction, many professional actors are required to perform voices with various emotions in specific scenes. In the process of data labelling, since the number of samples of different emotion categories is extremely imbalanced, it is difficult to efficiently label the samples. Hence, we proposed an integrated active learning sampling strategy and designed an efficient framework for constructing speech emotion corpora in order to address the problems presented above. Comparing experiments with other active learning algorithms on 13 datasets, our method was shown to improve sampling efficiency. In addition, it is able to select small category samples to be labelled with preference in imbalanced datasets. During the actual corpus construction experiments, our method can prioritize selecting small class emotion samples. As even when the amount of labelled data is less than 50%, the accuracy rate still can reach 90%. This greatly enhances the efficiency of constructing the speech emotion corpus and fills in the gaps.

Continuous dimensional emotion recognition for facial video sequence is a crucial and challenging task in Affective Computing and Human-Computer Intelligent Interaction. The key of this task is to effectively extract and discriminate spatial-temporal features in a more fine-grained way. In this paper, a Two-Stage Spatiotemporal Attention Temporal Convolution Network (TS-SATCN) is designed for continuous dimensional emotion recognition of facial videos. The first stage generates an initial recognition result that is later fed into the second for correction. In each stage, the introduced spatiotemporal attention branch helps the network learn different attention levels and focuses on the informative spatial-temporal features adaptively. The network is trained by a proposed smooth loss function which can further improve the predictions' quality. Extensive experiments are performed on two datasets, RECOLA and AFEW-VA, which shows that the proposed method achieves significant improvement over state-of-the-art methods.

User authentication is the first and most critical step in protecting a man-machine system from a malicious spoofer. However, security and privacy are just like the two sides of one coin, hard to see both at the same time, especially by the current mainstream credential- and biometric-based approaches. To this end, we propose WiONE, a safe and privacy-preserving user authentication system leveraging the ubiquitous Wi-Fi infrastructure by exploring “how you behave” rather than “who you are”. The key idea is to apply deep learning to user physical behavior captured by Wi-Fi channel state information (CSI) to identify legitimate users while rejecting spoofers. The design of WiONE faces two challenges, namely, how to capture the subtle behavior, such as a keystroke on CSI, and how to mitigate the heavy environment-specific training required by deep learning. For the former, we design a behavior enhancement model based on the Rician fading to highlight the behavior-induced information by suppressing the behavior-unrelated information on channel response. For the latter, we develop a behavior characterization method tailored for the prototypical networks to facilitate the extraction of the domain-independent behavioral features and enable one-shot recognition of a new user in a new environment. Numerous experiments are conducted in several real-world environments, and the results show that WiONE outperforms its state-of-the-art rivals in authentication performance with much less training effort.

This study addresses the problem of non‐perspective pose estimation from line correspondences in the traffic scenarios. A coarse‐to‐fine 3D road registration method is proposed for this problem in two stages. Firstly, the iterative closest point algorithm is exploited to estimate the pose coarsely. An objective function is then established to incorporate the feature correspondences for refining the coarse pose. Besides, the framework including road registration is employed for traffic video augmentation. The framework begins with the inputs of traffic videos, road information from Geographic Information Systems and 3D models of traffic elements (e.g. vehicles, pedestrians). Subsequently, 3D road model generation and point‐to‐line correspondence establishment are achieved in the preprossessing stage. After road and viewpoint registration, the 3D graphic engine is employed to simulate the traffic scene with the road, viewpoints and traffic elements. The augmented videos are generated by fusing the original frames and newly projected traffic elements. The authors demonstrate the superiority of the proposed registration method by the comparison to state‐of‐the‐arts in both quantitative and qualitative experiments. In addition, the frames of the augmented videos validate the proposed method in the application.

Understanding search engine users' intents has been a popular study in information retrieval, which directly affects the quality of retrieved information. One of the fundamental problems in this field is to find a connection between the entity in a query and the potential intents of the users, the latter of which would further reveal important information for facilitating the users' future actions. In this article, we present a novel research method for mining the actionable intents for search users, by generating a ranked list of the potentially most informative actions based on a massive pool of action samples. We compare different search strategies and their combinations for retrieving the action pool and develop three criteria for measuring the informativeness of the selected action samples, that is, the significance of an action sample within the pool, the representativeness of an action sample for the other candidate samples, and the diverseness of an action sample with respect to the selected actions. Our experiment, based on the Action Mining (AM) query entity data set from the Actionable Knowledge Graph (AKG) task at NTCIR‐13, suggests that the proposed approach is effective in generating an informative and early‐satisfying ranking of potential actions for search users.

Sleep is a major event of our daily lives. Its quality constitutes a critical indicator of people's health conditions, both mentally and physically. Existing sensor-based or vision-based sleep monitoring systems either are obstructive to use or fail to provide adequate coverage. With the fast expansion of wireless infrastructures nowadays, channel data, which is pervasive and transparent, emerges as another alternative. To this end, we propose Sleepy, a wireless channel data driven sleep monitoring system leveraging commercial WiFi devices. The key idea of Sleepy is that the energy feature of the wireless channel follows a Gaussian Mixture Model (GMM) derived from the accumulated channel data over a long period. Therefore, a GMM based foreground extraction method has been designed to adaptively distinguish motions like rollovers (foreground) from background (stationary postures), leading to certain major merits, e.g., no calibrations or target-dependent training needed. We prototype Sleepy and evaluate it in two real environments. In the short-term controlled experiments, Sleepy achieves 95.65 percent detection accuracy (DA) and 2.16 percent false negative rate (FNR) on average. In the 60-minute real sleep studies, Sleepy demonstrates strong stability, i.e., 0 percent FNR and 98.22 percent DA. Considering that Sleepy is compatible with existing WiFi infrastructure, it constitutes a low-cost yet promising solution for sleep monitoring.

To improve the space-time similarity and motion smoothness of facial expression imitation (FEI), a real-time FEI method for a humanoid robot is proposed based on smooth-constraint reversed mechanical model (SRMM) by combining a sequence-to-sequence deep learning model and a motion-smoothing constraint. First, on the basis of facial data from a Kinect capture device, a facial feature vector is characterized based on 3 head postures, 17 facial animation units, and facial geometric deformation cascaded by Laplace coordinates. Second, a reversed mechanical model is constructed via a multilayer long short-term memory neural network to accomplish direct mapping from facial feature sequences to motor position sequences. Additionally, to overcome the motor chattering phenomenon during real-time FEI, a high-order polynomial is constructed to fit the position sequence of motors, and an SRMM is proposed and designed based on the deviation of position, velocity, and acceleration. Finally, aiming to imitate the real-time facial feature sequences of a performer captured from Kinect, the optimal position sequences generated based on the SRMM is sent to the hardware system to keep the space-time characteristics consistent with those of the performer. The experimental results demonstrate that the motor position deviation of the SRMM is less than 8%. The space-time similarity between the robot and the performer is greater than 85%, and the motion smoothness of the online FEI exceeded 90%. Compared with other related methods, the proposed method achieves a remarkable improvement in motor position deviation, space-time similarity, and motion smoothness.

Good quality sleep is essential for good health and sleep monitoring becomes a vital research topic. This paper provides a low cost, continuous and contactless WiFi-based vital signs (breathing and heart rate) monitoring method. In particular, we set up the antennas based on Fresnel diffraction model and signal propagation theory, which enhances the detection of weak breathing/heartbeat motion. We implement a prototype system using the off-shelf devices and a real-time processing system to monitor vital signs in real time. The experimental results indicate the accurate breathing rate and heart rate detection performance. To the best of our knowledge, this is the first work to use a pair of WiFi devices and omnidirectional antennas to achieve real-time individual breathing rate and heart rate monitoring in different sleeping postures.

Falls are the second leading cause of injury deaths worldwide, inducing over 0.6 million accidental deaths per year. Among various prevention strategies, fall-related research has been prioritized. However, conventional fall detection solutions rely on computer vision or wearable sensors embody several inherent limitations such as scalability, coverage, and privacy issues. To this end, we present FallSense, a transparent and real-time fall sensing system driven by wireless channel data. FallSense is built on a Dynamic Template Matching (DTM) algorithm, which can start with a light training set and keep updating on usage. FallSense has been realized on commodity WiFi devices and evaluated in real environments. Experimental results show that FallSense outperforms another state-of-the-art approach WiFall in terms of detection precision, false alarm rate and complexity.

This study proposes a vision‐based method for traffic sign attribute estimation, i.e. 3D position and pose, from image sequences by binocular or monocular cameras. The method starts with acquiring robust feature correspondences based on homography constraints from image pairs. Then the objective function is designed to integrate the feature correspondences to optimise the parameters of the traffic sign plane in the 3D coordinate. Finally, the sign plane is utilised for attribute estimation. In addition, the authors provide an extension for the raw KITTI dataset, which can be utilised for 3D tasks of traffic sign localisation and pose estimation. In the experiments, three popular methods are employed for comparisons based on the publicly available BelgiumTS and KITTI datasets. The results show that the authors’ method based on SIFT and SURF features can locate the traffic signs with a mean error of ∼0.44 and 0.51 m in the BelgiumTS and KITTI datasets, respectively, and estimate the pose with a mean error of ∼14.45° in the KITTI dataset.

With the rapid development of integrated circuit manufacturing processes, the degree of integration of system-on-chip(SoC) has increased dramatically. In this paper, a dictionary-based test data compression method with tri-state coding is proposed to reduce the increasing test data volume. Firstly the partial input reduction is used to preprocess the test set, and then the tri-state coding is presented to mark the index so that it can encode anywhere in the test set. Experimental results on ISCAS'89 benchmark circuits show that the average compression rate of the proposed scheme reaches 73.92% by increasing the hardware area overhead slightly.

In this paper, we present WoSense, a device-free and real-time behavior analysis system leveraging only WiFi infrastructures. WoSense aims to remotely recognize various human behaviors like surfing, gaming and working around computers, which are considered to be an essential part of our daily lives both at work and at home. The key of WoSense is to exploit the signal distortions on channel data caused by gestures like finger and hand movements, and then identify possible behaviors via the composite of gestures. Therefore, two critical challenges need to be tackled: how to enhance such insignificant distortions led by micro gestures, how to segment the continuous signals according to different gestures in a real-time manner? For the former, instead of relying on empirical studies like our rivals, WoSense offers a Fresnel zone based model with theoretic understandings between the gestures and signal distortions. For the latter, WoSense employs a light-weight automatic segmentation algorithm exploring the variance feature of channel data. We prototype WoSense on the commodity low-cost WiFi devices and evaluate its performance in extensive real- world experiments. WoSense achieves an average 96.77% accuracy for distinguishing the typing and mousing gestures, and 92.5% accuracy for recognizing four different behaviors, i.e., stationary, surfing, gaming and working.

Emotion is a unique feature of human beings. Recent research in emotion sensing has already revealed its potentials in enhancing our living experiences through applications like emotion companion and autism treatment. However, existing solutions exploring audiovisual clues or psychological sensors have several critical concerns such as the availability (specialized hardware), reliability (illumination and line-of-sight constraints) and privacy issues (being watched). To this end, we present EmoSense, a first-of-its-kind WiFi-based emotion sensing system leveraging the temporal and frequency fingerprints on the wireless channel data induced by the physical expression of emotion. EmoSense has been prototyped with off- the-shelf WiFi devices and evaluated by comparing with the main-stream sensor-based approach in real environments. Experimental results demonstrate its effectiveness and robustness. Considering that EmoSense is compatible with existing WiFi infrastructures, it constitutes a low-cost yet promising solution for emotion sensing.

In this paper, a new multi-modal traffic scene simulation framework with combined inputs of road image sequences and road information from Geographic Information Systems (GIS) is proposed. The proposed framework contains two major steps, with the first one being a preprocessing step, including 3D road model extraction, camera location and orientation estimation and lane extraction from both GIS and road image sequences. After such preprocessing, the traffic scene reconstruction is reformulated into a 6-degree of freedom (6DoF) pose estimation in the 3D road model. Subsequently, the iterative closest point (ICP) algorithm is exploited for coarse point registration by estimating the pose in the road model. In addition, an objective function is established to incorporate the image features (e.g., lanes) into the road model and to refine the pose estimation. In the experiments with the publicly available KITTI dataset, the proposed method achieves high average Intersection-over-Union (IoU) scores as compared to the ground truth image sequences.

Cognitive developmental robotics is one of the keys to creating intelligent robots based on human development. In our research, we focus on drawing development of a human infant for creating a developing drawing robot. As the basis of our development model, we adopt five stages development proposed by Luquet. Multiple Timescales Recurrent Neural Network (MTRNN) model, is utilized as the learning model. In the first stage of drawing development, "Scribbling", the robot generates random arm motions to draw meaningless objects to learn the relation between the arm motion and drawing result. The robot arm/pen dynamics is learned using MTRNN in this stage. In the second/third stages, the robot imitates drawing motions from a human to improve its drawing skills to draw shapes through incremental learning. In this paper, we introduce model retraining and motion trajectory modification into the learning model to improve the drawing performance. The method is inspired by the Adaptive VITEWRITE (AVITEWRITE) model proposed by Grossberg, which describes drawing skill improvement of children through perception/action cycle involving vision, attention, learning, and movement. Experiments were conducted using a humanoid robot Nao drawing on a pen tablet. The results of the experiment show the effectiveness of the method improving the drawing performance using the proposed method.

In this study, a novel multi-classifier ensemble method based on dynamic weights is proposed to reduce the interference of unreliable decision information and improve the accuracy of fusion decision. The algorithm defines decision credibility to describe the real-time importance of the classifier to the current target, combines this credibility with the reliability calculated by the classifier on the training data set and dynamically assigns the fusion weight to the classifier. Compared with other methods, the contribution of different classifiers to fusion decision in acquiring weights is fully evaluated in consideration of the capability of the classifier to not only identify different sample regions but also output decision information when identifying specific targets. Experimental results on public face databases show that the proposed method can obtain higher classification accuracy than that of single classifier and some popular fusion algorithms. The feasibility and effectiveness of the proposed method are verified.

This paper, we proposed an improved facial expression recognition (FER) method based on region of interesting (ROI) to guide the convolutional neutral networks (CNN) focus on the areas associated with the expression. This method can not only augment the training data, the relationship between the different ROI areas is helpful to intensify the reliability of the predicted targets. In test stage, we investigated two recognition methods: identify the test image directly; implemented decision fusion strategy on ROI areas. The model we used is fine-tuned from pre-trained deep CNN instead of training from scratch. In addition, we presented an innovative region-based image augmentation method named artificial face to increase the limited database. This method using expression retargeting as an expression-preserving data augmentation which is specific for FER. The performance of the proposed method has been validated on the public CK+ databases.

Human activity recognition based on wireless signals emerges as a research hotspot recently. Though tremendous efforts have been devoted and significant progresses have been achieved, one fundamental issue still remains open, i.e., theoretical modeling between signal dynamics and human activities. This paper fills in the blank by addressing several theoretical issues and providing insightful mathematical analysis including a signal-activity model. To validate such analysis, a prototype system has been built, where a series of real-world experiments has been conducted. Empirical results have justified our theoretical findings. Moreover, important hands-on experiences on the system implementation and parameter settings have been offered.

Motion is a critical indicator of human presence and activities. Recent developments in the field of indoor motion detection have the potential to enhance various aspects of our daily experiences like intrusion detection and sleep monitoring. Existing indoor motion detection solutions either are obstructive to use or fail to provide adequate coverage. To overcome these shortages, a noninvasive and cost-effective motion detection system (MoSense) is proposed for periodically detecting motions by exploring channel response in the commodity WiFi devices. The central idea is that signals in a ``silence'' environment serve well as a ``golden'' benchmark for recognizing motions that lead to signal fluctuations. A prototype of MoSense is realized and evaluated in real environments. By comparing MoSense with another state-of-the-art method, i.e., FIMD, we have shown that MoSense outperforms FIMD in terms of computational complexity, detection accuracy and false alarm rate. Considering that MoSense is compatible with existing WiFi infrastructure, it constitutes a low-cost yet promising solution for motion detection.

Motion is a critical indicator of human presence and activities. Recent developments in the field of indoor motion detection have revealed their potentials in enhancing our living experiences through applications like intrusion detection and sleep monitoring. However, existing solutions still face several critical downsides such as the availability (specialized hardware), reliability (illumination and line-of-sight constraints), and privacy issues (being watched). To overcome such shortages, a radio frequency (RF) based device-free motion detection system (MoSense) is designed via leveraging the attenuation of ubiquitous WiFi signals induced by motions to deliver a reliable and transparent detection service in realtime. The design and implementation of MoSense face two challenges: 1) characterizing stationary states and 2) the noisy subcarriers. For the first challenge, a silence analysis model is proposed to characterize stationary states for distinguishing motions. For the second challenge, we design a distance-based mechanism to select certain subcarriers that better capture the impact of motions from the noisy channel through measuring the similarity between subcarriers. A prototype of MoSense is realized and evaluated in real environments. By comparing MoSense with other two state-of-the-art systems, i.e., FIMD and FRID, we have shown that MoSense is superior in terms of precision, false negative rate and computational complexity. Considering that MoSense is compatible with existing WiFi infrastructure, it constitutes a low-cost yet promising solution for motion detection.

With the increasing impact of social networks, microblog becomes important carrier of information and social interaction for human beings, which contains emotional states that have important research significance. We try to analysis the microblog text with the methods of emotional vocabulary, combining domain knowledge of psychology and affective computing, continuous dimension of emotion psychology PAD model which is adopted as basis of sentiment analysis. Emotional state inherent in the text is analyzed to obtain a more accurate result and achieve purposes of emotional analysis. At the same time, to achieve emotional microblog text computability from the aspect of personal characteristics. Experimental results show that the method can improve the microblog text sentiment analysis accuracy and precision. The method is able to get a good application in the different themes and different emotional features.

Along with the development of social network, more and more people know the world by reading news. The problem about what kind of emotion is inspired when people read news is very worthy of discussion. This paper will mix Deep Belief Networks (DBN) model and Support Vector Machine (SVM) to a hybrid neural network model by using the Contrast Divergence (CD) algorithm to estimate the weights when training a generating model, ensure that each layer of the Restricted Boltzmann Machine (RBM) mapping the features of the inputs to the best. At the same time, we cascade the last layer of DBN and a SVM classifier to adjust judging performance. And a set of tags will be attached to the top (Associative Memory), through a process of parameter tuning, learn the identifying weights to obtain a network for the task of text classification. The experimental results show that the hybrid neural network model works better than the traditional text categorization method based on simple characteristics (such as CHI), and it is more suitable for extracting text semantic characteristics.

With the rapid development of B2C e-commerce and the popularity of online shopping, the Web storages huge number of product reviews comment by customers. Product reviews contain subjective feelings of customers who have used some products, more and more customers browse a large number of online reviews in order to know other customers word-of-mouth of product and service to make an informed choice. Manufacturers also need accurate user feedback from product reviews to improve their goods. However, a large number of reviews made it difficult for manufacturers or potential customers to track the comments and suggestions that customers made. This paper presents a method for extracting emotional elements containing emotional objects and emotional words and their tendencies from product reviews based on mixed model. First we constructed conditional random fields (CRFs) to extract emotional elements, lead-in semantic and word meaning as features to improve the robustness of feature template and used rules for hierarchical filtering errors. Then we constructed support vector machine (SVM) to classify the emotional tendency of the fine-grained elements to achieve key information from product reviews. Deep semantic information imported based on neural network (NN) to improve the traditional bag of word model. Experimental results show that the proposed model with deep features efficiently improved the F-Measure.

In this paper, the performance of Convolution Neural Network (CNN) in image recognition and emotion recognition in speech will be compared and presented. Feature extraction and selection in pattern recognition is an important issue and have been frequently discussed. Moreover, two-dimensional signals such as image and voice are hard to be modelled well by traditional models like SVM. The ability of CNN to characterize two-dimensional signals is prominent. And CNN can adaptively extract feature to eliminate the dependence on human subjectivity or experience. It mimics the effect of local filtering in visual cortex cells to dig local correlation in natural dimensional space. In this work, for the problems of the image recognition and emotion recognition in speech, CNN and SVM which is used as baseline for comparison of the recognition effect. Different kernel functions in SVM have been experimented for image recognition with, the best accuracy is 94.17%. However, the accuracy of using CNN is 95.5% (7291 pictures for train and 2007 pictures for test) with less time consuming. In the emotion recognition of speech, the accuracy of CNN is 97.6% corresponds to 55.5% by baseline model (4000 utterances for training, 1500 for validation, 500 for test). The experimental results showed that CNN can effectively extract features and its modeling capability for two-dimensional signals is prominent.

We study the problem of autonomous service migration in the cloud to satisfy an online sequence of mobile batch-request demands in a cost-effective way. As the origins of the mobile accesses frequently change over time, this problem is particularly important for time-bounded services to achieve enhanced QoS and cost effectiveness. Moving the service closer to its client locations not only reduces the service access latency but also minimizes the network costs for service providers. However, the migration comes at costs of bulk-data transfer and service disruption, as a result, increasing the overall service costs. To gain the benefits of service migration while minimizing the service costs, we propose an efficient search-based algorithm Dmig the service migration in an autonomous way. Compared with existing algorithms, the proposed algorithm is fully distributed, symmetric, and characterized by the effective use of historical access information to perform virtual migration that overcomes the limitation of traditional local search in cost reduction. To evaluate the algorithm, we compared it with some existing algorithms, and show that the proposed algorithm exhibits better performance by adapting to the changes of mobile access patterns in a cost effective way.

This paper presents a novel WLAN-based indoor localization algorithm (i.e., HED) to combat the environmental dynamics by tolerating the sequence disorders caused by AP (access point) changes, while harvesting from the bursting number of available wireless resources. Via extensive real-world experiments lasting for over 6 months, we show the superiority of our HED algorithm in terms of accuracy and complexity over two state-of-the-art solutions that are also designed to resist the dynamics, i.e., FreeLoc and LCS (Longest Common Subsequences). Moreover, experimental results not only confirm the benefits brought by environmental dynamics, but also provide valuable investigations and hand-on experiences on the real-world localization system.

Summary The flourishing social networks nowadays have greatly enriched our ways of communications and thus brought people in the world much closer than ever. However, critical contexts of the traditional face‐to‐face communications, for example, body gestures, could be missing during the online communication, hampering the user experiences. To fill in the blank, this paper presents a passive and devices‐free activity recognition system, by harvesting fingerprints of different activities from ubiquitous WiFi signals. It can be integrated into any existing WLAN networks without additional hardware supports. Also, it does not need the subjects to be cooperative during the recognition process. A prototype system is built and evaluated via extensive real‐world experiments. By comparing with three state‐of‐the art solutions, that is, K‐nearest neighbor, naive Bayes, and bagging, we show the superiority of the proposed method in terms of accuracy and complexity. Copyright © 2015 John Wiley & Sons, Ltd.

Indoor human activity recognition remains a hot topic and receives tremendous research efforts during the last few decades. However, previous solutions either rely on special hardware, or demand the cooperation of subjects. Therefore, the scalability issue remains a great challenge. To this end, we present an online activity recognition system, which explores WiFi ambient signals for received signal strength indicator (RSSI) fingerprint of different activities. It can be integrated into any existing WLAN networks without additional hardware support. Also, it does not need the subjects to be cooperative during the recognition process. More specifically, we first conduct an empirical study to gain in-depth understanding of WiFi characteristics, e.g., the impact of activities on the WiFi RSSI. Then, we present an online activity recognition architecture that is flexible and can adapt to different settings/conditions/scenarios. Lastly, a prototype system is built and evaluated via extensive real-world experiments. A novel fusion algorithm is specifically designed based on the classification tree to better classify activities with similar signatures. Experimental results show that the fusion algorithm outperforms three other well-known classifiers [i.e., NaiveBayes, Bagging, and k-nearest neighbor (k-NN)] in terms of accuracy and complexity. Important sights and hands-on experiences have been obtained to guide the system implementation and outline future research directions.

This paper proposes a method to semi-automatically construct a corpus that includes Japanese youth slang called Wakamono Kotoba. The process of semi-automatic corpus construction is composed of the first step is automatic collection of example sentence, the second step is tag annotation to collected sentences, and the final step is manually modifying tag and noise reduction. In this process, there are two problems. The first problem is quality of the automatic collected corpora. The second is the accuracy of tag annotation. If the automatically annotated tags are unreliable, after all, it takes long time to modify them manually. As a solution of the first problem, we proposed a filtering method to remove meaningless sentences (noise sentences) automatically. In order to solve a second problem, we proposed an emotion estimation method that can be applied to the sentences that included youth slang and were difficult to be analyzed automatically. The result of the accuracy evaluation showed improvement in F1-Score compared to the machine learning method and confirms the effectiveness of the proposed method.

The ability of a humanoid robot to display human-like facial expressions is crucial to the natural human-computer interaction. To fulfill this requirement for an imitative humanoid robot, XIN-REN, an automatic facial expression learning method is proposed. In this method, first, a forward kinematics model, which is designed to reflect nonlinear mapping relationships between servo displacement vectors and corresponding expression shape vectors, is converted into a linear relationships between the mechanical energy of servo displacements and the potential energy of feature points, based on the energy conservation principle. Second, an improved inverse kinematics model is established under the constraints of instantaneous similarity and movement smoothness. Finally, online expression learning is employed to determine the optimal servo displacements for transferring the facial expressions of a human performer to the robot. To illustrate the performance of the proposed method, we conduct evaluation experiments on the forward kinematics model and the inverse kinematics model, based on the data collected from the robot's random states as well as fixed procedures by animators. Further, we evaluate the facial imitation ability with different values of the weighting factor, according to three sequential indicators (space-similarity, time-similarity, and movement smoothness). Experimental results indicate that the deviations in mean shape and position do not exceed 6 pixels and 3 pixels, respectively, and the average servo displacement deviation does not exceed 0.8%. Compared with other related studies, the proposed method maintains better space-time similarity with the performer, besides ensuring smoother trajectory for multiframe sequential imitation.

Suicide has been a major cause of death throughout the world. Recent studies have proved a reliable connection between the emotional traits and suicide. However, detection and prevention of suicide are mostly carried out in the clinical centers, which limit the effective treatments to a restricted group of people. To assist detecting suicide risks among the public, we propose a novel method by exploring the accumulated emotional information from people's daily writings (i.e., Blogs), and examining these emotional traits that are predictive of suicidal behaviors. A complex emotion topic model is employed to detect the underlying emotions and emotion-related topics in the Blog streams, based on eight basic emotion categories and five levels of emotion intensities. Since suicide is caused through an accumulative process, we propose three accumulative emotional traits, i.e., accumulation, covariance, and transition of the consecutive Blog emotions, and employ a generalized linear regression algorithm to examine the relationship between emotional traits and suicide risk. Our experiment results suggest that the emotion transition trait turns to be more discriminative of the suicide risk, and that the combination of three traits in linear regression would generate even more discriminative predictions. A classification of the suicide and nonsuicide Blog articles in our additional experiment verifies this result. Finally, we conduct a case study of the most commonly mentioned emotion-related topics in the suicidal Blogs, to further understand the association between emotions and thoughts for these authors.

Recently, a portable digital device equipped with voice guidance has been widely used with increasing the demand for the usability-conscious dialogue system. One of the problems with the existing dialogue system is its immature application to non-task dialogue. Non-task-oriented dialogue requires some schemes that enable smooth and flexible conversations with a user. For example, it would be possible to go beyond the closed relationship between the system and the user by considering the user's relationship with others in real life. In this paper, we focused on the dialogue made by the two characters in a drama scenario, and tried to express their relationship with a scale of “intimacy degree.” There will be such various elements related to the intimacy degree as the frequency of response to the utterance and the attitude of a speaker during the dialogue. We focused on the emotional state of the speaker during the utterance and tried to realize intimacy estimation with higher accuracy. As the evaluation result, we achieved higher accuracy in intimacy estimation than the existing method based on speech role.

Related research for sentiment analysis on Chinese microblog is aiming at analyzing the emotion of posters. This paper presents a content extension method that combines post with its' comments into a microblog conversation for sentiment analysis. A new convolutional auto encoder which can extract contextual sentiment information from microblog conversation of the post is proposed. Furthermore, a DBN model, which is composed by several layers of RBM(Restricted Boltzmann Machine) stacked together, is implemented to extract some higher level feature for short text of a post. These RBM layers can encoder observed short text to learn hidden structures or semantics information for better feature representation. A ClassRBM (Classification RBM) layer, which is stacked on top of RBM layers, is adapted to achieve the final sentiment classification. The experiment results demonstrate that, with proper structure and parameter, the performance of the proposed deep learning method on sentiment classification is better than state-of-the-art surface learning models such as SVM or NB, which also proves that DBN is suitable for short-length document classification with the proposed feature dimensionality extension method.

Researches on emotion estimation from text mostly use machine learning method. Because machine learning requires a large amount of example corpora, how to acquire high quality training data has been discussed as one of its major problems. The existing language resources include emotion corpora; however, they are not available if the language is different. Constructing bilingual corpus manually is also financially difficult. We propose a method to convert a training data into different language using an existing Japanese-English parallel emotion corpus. With a bilingual dictionary, the translation candidates are extracted against every word of each sentence included in the corpus. Then the extracted translation candidates are narrowed down into a set of words that highly contribute to emotion estimation and we used the set of words as training data. As the result of the evaluation experiment using the training data created by our proposed method, the accuracy of emotion estimation increased up to 66.7% in Naive Bayes. 1 INTRODUCTION Recently, there have been many researches on emotion estimation from text in the field of sentiment analysis or opinion mining (Ren, 2009), (Ren and Quan, 2015), (Ren and Wu, 2013), (Quan and Ren, 2010), (Quan and Ren, 2014), (Ren and Matsumoto, 2015) and many of them adopted machine learning methods that used words as a feature. When the type of the target sentence for emotion estimation and the type of the sentence prepared as training data are different, as in the case of terminology in the problem of domain adaptation for document classification, the appearance tendency of the emotion words differs. This causes a problem in fluctuation of accuracy. On the other hand, when a word is used as a feature for emotion estimation, the sentence structure does not have to be considered. As a result, it is easy to apply the method to other languages. Only if we prepare a large number of corpora with annotation of emotion tags on each sentence, emotion would be easily estimated by using the machine learning method. In the machine learning method, because manual definition of a rule is not necessary, we can reduce costs to apply the method to other languages. However, just like the problem in the domain, depending on the

Result diversification is a topic of great value for enhancing user experience in many fields, such as web search and recommender systems. Many existing methods generate a diversified result in a sequential manner, but they work well only if the preceding choices are optimal or close to the optimal solution. Moreover, a manually tuned parameter (say,λ) is often required to trade off relevance and diversity. This makes it difficult to know whether the failures are caused by the optimization criterion or the setting of λ. In context of web search, we formulate the result diversification task as a 0-1 multiple subtopic knapsack problem (MSKP), where a subset of documents are optimally chosen like filling up multiple subtopic knapsacks. This formulation yields no trade-off parameters to be specified beforehand. Solving the 0-1 MSKP is NP-hard, we treat the optimization of 0-1 MSKP using a graphical model over latent binary variables as a maximum posterior inference problem, and tackle it with the max-sum belief propagation algorithm. To validate the effectiveness and efficiency of the proposed 0-1 MSKP model, we conduct a series of experiments on two TREC diversity collections. The experimental results show that the proposed model outperforms several state-of-the-art methods significantly, not only in terms of standard diversity metrics (α-nDCG, nERRIA and subtopic recall), but also in terms of efficiency.

Indoor localization remains a hot topic during the last few decades. Previous research mainly relies on wireless fingerprints and thus demands a large number of APs or labor-intensive site survey. To this end, by exploring Bluetooth features and leveraging user motions, we introduce a novel localization scheme that needs neither APs nor site survey. More specifically, through a systematic experimental study, we gain in-depth understandings of Bluetooth characteristics e.g., The impact of various factors such as distance, orientation, and obstacles on the Bluetooth RSSI (Received Signal Strength Indicator). With the empirical experiences, a novel motion-assisted localization model is built to describe the relationship between RSSI and device location. Based on the model, we design a localization scheme that iteratively adjusts the search directions according to RSSI changes to approach the target device. We prototype and evaluate our system in several real-world scenarios. Extensive experiments show that the proposed scheme is efficient in terms of localization accuracy, searching time and energy consumption.

In this paper, we study the network Lifetime Maximization problem in Mobile healthcare sensor systems (LMM). For the healthecare system, we consider a dynamic scenario where users are mobile at their own wills and periodically report their personal health information (PHI) to a static sink, e.g. a powerful server, for further processing and distributing. The objective is to optimize the network lifetime by flow scheduling. The major difficulty lies in the time-dependent network topologies. Therefore, we propose a novel temporal-spatial network modeling method by extending current model with time dimension. Based on this model, we show that if the movement of users are known in advance (i.e. offline case), the problem can be optimally solved in polynomial time by a linear programming. However, the online LMM problem is much more difficult to tackle, since we prove that there exists no online algorithm with a constant performance ratio to the offline optimal algorithm in terms of the network lifetime. We further design simulations to show the performance gap between online and offline LMM. Considering the user mobility within a given scenario follows some certain patterns, we show the potential improvements of using a prediction-based method. This investigation provides certain insights on designing efficient online algorithms for the LMM problem.

With popular microblogging services like Twitter, users are able to online share their real-time feelings in a more convenient way. The user generated data in Twitter is thus regarded as a resource providing individuals' spontaneous emotional information, and has attracted much attention of researchers. Prior work has measured the emotional expressions in users' tweets and then performed various analysis and learning. However, how to utilize those learned knowledge from the observed tweets and the context information to predict users' opinions toward specific topics they had not directly given yet, is a novel problem presenting both challenges and opportunities. In this paper, we mainly focus on solving this problem with a Social context and Topical context incorporated Matrix Factorization (ScTcMF) framework. The experimental results on a real-world Twitter data set show that this framework outperforms the state-of-the-art collaborative filtering methods, and demonstrate that both social context and topical context are effective in improving the user-topic opinion prediction performance.

Understanding the information need or intent encoded within a query has long been regarded as an essential factor of effective information retrieval. For better query representation and understanding, two intent roles (kernel-object and modifier) are introduced to structurally parse a class of role-explicit queries, which constitute a majority of common user queries. Furthermore, we focus on two research problems: RP-1: Given a role-explicit query, how to identify the kernel-object and modifier, namely intent role annotation; RP-2: How to determine whether an arbitrary query is role-explicit or not. To solve RP-1, we propose a simplified word n-gram role model (SWNR), which quantifies the generating probability of a role-explicit query and performs intent role annotation effectively. Using a set of discriminative features, we build classifiers to address RP-2 in a supervised manner. The experimental results show that: (1) SWNR can achieve a satisfactory performance, more than 73% in terms of different metrics; (2) The classifiers can achieve more than 90% precision in identifying role-explicit queries; (3) Compared with traditional techniques for query representation and understanding, e.g., name entity recognition in query and class-level query intent inference, intent role annotation provides a more flexible framework and a number of applications can benefit from annotating role-explicit queries, such as intent mining and diversified document ranking.

Some people who have tendency toward depression describe their inner thoughts vividly in their weblogs. Such descriptions are sometimes made consciously but at other times unconsciously by the authors. If such unconscious authors could become aware of their own symptom of the depression, that would contribute to early detection and treatment of the sickness. To achieve this, it is essential to find out the tendency of depression from the descriptions in weblog. Because the symptom of depression is not static, we attempted to detect the tendency of depression focusing on the temporal fluctuation of emotion of the authors. As many weblogs describe what happened or what the authors felt in daily life like a personal journal, we thought that weblogs were ideal for obtaining the time-varying emotion of the authors. In this paper, we proposed the method to detect the tendency of depression in weblog. Our method is based on the temporal fluctuation of emotion described by the authors.

This paper proposes an ontology learning system model based on the Web search engine and Protege-OWL API, which emphasizes iterative learning approach by the extracted instances. We discuss taxonomic and non-taxonomic relationship learning separately in ontology learning system, and investigate the importance of verb plus noun phrase learning for extraction of activity concepts in Chinese. We also propose an algorithm of relevance measurement for extracting relation instances by binary keywords based on co-occurrence statistics. Finally, we build a practical system of ontology learning through learning relation instances of the Chinese festival ontology, and test the effectiveness of our method.

Abstract Proper nouns may be considered the most important query words in information retrieval. If the two languages use the same alphabet, the same proper nouns can be found in either language. However, if the two languages use different alphabets, the names must be transliterated. Short vowels are not usually marked on Arabic words in almost all Arabic documents (except very important documents like the Muslim and Christian holy books). Moreover, most Arabic words have a syllable consisting of a consonant‐vowel combination (CV), which means that most Arabic words contain a short or long vowel between two successive consonant letters. That makes it difficult to create English‐Arabic transliteration pairs, since some English letters may not be matched with any romanized Arabic letter. In the present study, we present different approaches for extraction of transliteration proper‐noun pairs from parallel corpora based on different similarity measures between the English and romanized Arabic proper nouns under consideration. The strength of our new system is that it works well for low‐frequency proper noun pairs. We evaluate the new approaches presented using two different English‐Arabic parallel corpora. Most of our results outperform previously published results in terms of precision, recall, and F‐Measure.

In this paper, we describe a Japanese speech corpus collected for investigating the speech variability of a specific speaker over short and long time periods and then report the variability of speech recognition performance over short and long time periods. Although speakers use a speaker-dependent speech recognition system, it is known that speech recognition performance varies pending when the utterance was uttered. This is because speech quality varies by occasion even if the speaker and utterance remain constant. However, the relationships between intra-speaker speech variability and speech recognition performance are not clear. Hence, we have been collecting speech data to investigate these relationships since November 2002. In this paper, we introduce our speech corpus and report speech recognition experiments using our corpus. Experimental results show that the variability of recognition performance over different days is larger than variability of recognition performance within a day.

Parallel corpora have become an essential resource for work in multi lingual natural language processing. However, sentence aligned parallel corpora are more efficient than non-aligned parallel corpora for cross language information retrieval and machine translation applications. In this paper, we present a new approach to align sentences in bilingual parallel corpora based on feed forward neural network classifier. A feature parameter vector is extracted from the text pair under consideration. This vector contains text features such as length, punctuate score, and cognate score values. A set of manually prepared training data has been assigned to train the feed forward neural network. Another set of data was used for testing. Using this new approach, we could achieve an error reduction of 60% over length based approach when applied on English-Arabic parallel documents. Moreover this new approach is valid for any language pair and it is quite flexible approach since the feature parameter vector may contain more/less or different features than that we used in our system such as lexical match feature.

In this paper, we describe a novel acoustic model adaptation technique which generates “speaker-independent” HMM for the target environment. Recently, personal digital assistants like cellular phones are shifting to IP terminals. The encoding-decoding process utilized for transmitting over IP networks deteriorates the quality of speech data. This deterioration causes degradation in speech recognition performance. Acoustic model adaptations can improve recognition performance. However, the conventional adaptation methods usually require a large amount of adaptation data. The proposed method uses HMM-based speech synthesis to generate adaptation data from the acoustic model of HMM-based speech recognizer, and consequently does not require any speech data for adaptation. Experimental results on G.723.1 coded speech recognition show that the proposed method improves speech recognition performance. A relative word error rate reduction of approximately 12% was observed.

In this paper, we focus on distributed speaker recognition, a technique in which quantized feature parameters are sent to a server, as with distributed speech recognition. The Gaussian mixture model , the traditional method used for speaker recognition, is trained using the maximum likelihood approach. The GMM has output probability functions with continuous density functions. It is difficult to fit continuous density functions to quantized data. To overcome this problem, we propose a novel speaker recognition technique which does not need speaker model training. The proposed method directly calculates the distance between a set of quantized feature parameters of registered speech and a set of quantized feature parameters of test speech. To measure distance, we use Earth Mover’s Distance (EMD). The EMD has recently been successfully applied to image retrieval. We conduct text-independent speaker identification experiments using the proposed method. When compared to results using the traditional GMM, the proposed method yielded relative error reductions of 80% for quantized data.

The mobile telephone is the representation of mobile communication. In this paper we proposed a new machine translation method called Super-Function Based Machine Translation (SFBMT) to try to use for mobile terminal. SFBMT uses Super Function(SF) to translate the sentence without thorough syntactic and semantic analysis. And it can translate the sentence fast and high correct. Our experiment shows that this method can get 70% translation precision using the sentences that learned for extracting Super-Function, And 40% translation precision using the sentences that not learned for extracting Super-Function.

In the machine translation (MT) system, how to evaluate the sentence fluency of the translation results is an important research topic. Most of the current methods are based on the similarity of output words compared with the reference translations, which don't specially address the evaluation of the sentence fluency according to the syntactic structure. This paper proposes a statistical approach to the problem, which is based on the n-gram language model and reference-independent. Our approach is a beneficial compensation to the reference-dependent methods and has better robustness than those methods based on the syntactic analysis. The preliminary experimental results indicate that the approach basically reflects the reality of human's judgment.

Human cognition stems from his identifying the relationship or relative distance between himself and any other objects or persons, for which prerequisite is establishing his 'self'. In other words, it is presumable that the potential of intelligent machine's establishing its 'self' would require us to engineer its sense of 'ego' and 'intention', which involves such factors as 'emotions', 'instincts', and 'sensibilities'. This dissertation introduces the psycho-quantum theory which illustrates the human psychodynamics by the quantum mechanics approach, and its application for constructing the distribution model of emotional modalities and the autonomous system, the essential elements for creating an ego in a computer. We have attempted to create the artificial 'ego' and 'desire' based on our probability model and state transition analysis, then accordingly developed our methodology of the artificial thought process as well as the judging mechanism of such computer. Furthermore, this dissertation introduces the sensibility inference function of our psycho-quantum computer, by which we discuss our methods to itemize 126 emotional modalities of human, construct the three-dimensional dynamic model of emotions, and analyze the emotion distribution and its longitudinal modal transition so as to derive the artificial instinct and to determine/infer its relationship with external factors. Currently, our technology to construct the comprehensive psycho-quantum computer is yet in developmental stage; however, its potential is enormous so that it will constitute a key technology in new robotics industry.

In the near future, it will be necessary for the senior citizens to nurse the other senior citizens because of declining population of children and increasing a new type of family. We have been developing welfare robots which can support lives of the senior citizens and have sensibility to lighten the burden imposed on nursing. The measurement of emotions from the usual conversation is considered to be one of its basic researches. In this paper, we are going to propose the algorithm of the emotion measurement and the prototype system based on this algorithm, and we are also going to discuss its validity.

This paper reports an evaluation of European Telecommunications Standards Institute (ETSI) standard Distributed Speech Recognition (DSR) front-end through continuous word recognition on a Japanese speech corpus and proposes a method, the Bias Removal Method (BRM), that reduces the distortion between feature vector and VQ codebook. Experimental results show that using nonquantized features in acoustic model training procedure can improve the recognition performance of DSR fornt-end features and that the proposed method can improve recognition performances of DSR front-end feature.

We have designed a new general ontology based natural language processing system referred to as a Multi-lingual Multi-function Multi-media (MMM) intelligent system. In this paper, we describe a new function, machine-aided writing, in the MMM system. Our methodology can deal with any foreign language, but we only focus on how to construct a machine-aided English writing system for Japanese users. The basic idea of the method proposed in this paper is based on Super-Function (SF) means and the Corpora Intelligence Technique (CIT). SF is a new concept that we present to develop multi-lingual machine translation. A prototype machine-aided English writing system has been constructed based on the proposed method. Some experiments using the prototype system have been carried out and the results show that the proposed method is effective.

Parallel machine translation (PMT) is a new machine translation paradigm that takes advantage of high-speed and large-memory computers and existing machine translation systems with different characteristics to solve the difficult machine translation problem. PMT is based on technologies of parallel computing, machine translation, and artificial intelligence. A PMT system consists of many machine translation procedures running in parallel, coordinated by a controller to dissolve various ambiguities in machine translation. We have designed and implemented a PMT system based on the above approach at a coarse parallel level. The system consists of four independent machine translation subsystems. Each subsystem is implemented using an existing machine translation technique and has its own characteristics. We present the principles and practice of PMT. We also present some results of experiments with our experimental PMT system and point out some future research on PMT.

Being increasingly popular, the Internet greatly changes our live. We can conveniently receive and send information via the Internet. With the information explosion in Web, it is becoming crucial to develop means to automatically extract important sentences from the Web articles. In this paper, we propose a method which uses both statistical and structural information in sentence extraction. In addition, following the analysis of human's extractions, several heuristic rules are added to filter out non-important sentences and to prevent similar sentences from being extracted. Our experimental results proved the effectiveness of these means. In particular, once the heuristic rules being added, a significant improvement has been observed.

In this paper, we present a new machine translation (MT) approach using MT engines and sentence partitioning. A multiple engine MT system consists of several MT engines running in parallel, coordinated by a controller. Each engine is implemented using an existing MT technique and has its own characteristics. When translating a sentence, each engine translates it independently. If more than one engine translates the sentence successfully, the controller chooses the best translation according to a combining algorithm implemented using translation statistics. If no engine succeeds in translating the sentence, the, controller partitions the sentence, coordinates the engines to translates its constituent simple sentences, and combines the partial translation results into a translation result for the whole input sentence. A complex sentence is partitioned based on conjunctives and punctuation marks such as comma: and semicolon. We have developed a multiple engine MT system based on the above approach. The system consists of four independent MT engines. The experiments show that the proposed approach is effective for implementing practical MT systems.

Automatic Chinese text checking and error correction is an important and difficult problem. Compared with automatic checking and error correction of Western text automatic checking and error correction of Chinese text faces more challenges. The Chinese language has many characters and no delimiters separating words. It is impossible to detect. and correct errors by penetrating into the inner composition of a character. In this paper, we describe some special features of Chinese characters and text and some statistical information obtained from a real world Chinese text corpus, and we present a hybrid approach that combines a rule-based method and a probability-based method to automatic checking and error correction of Chinese text. We also present an experimental system, HSACCCT (Hybrid System of Automatic Checking and Correction for Chinese Text), that implements this hybrid approach and some experimental results on real world Chinese text.

As the traditional relational databases and the new XML document repositories are being widely used on the Web as information storage, there is a great need for easy access to the information sources, particularly through natural language interactions. In this paper, we present the design and implementation of an intelligent system that interacts with users using a natural language, English, and retrieves information from sources, for the users. The system consists of four major parts: part of speech tagging, query knowledge base, query formation, and answer synthesis. In implementation, the system first uses QTAG, a Hidden Markov Model based speech part tagger, to tag each word in the input sentence. Then, important words in the main phrase axe identified. A thesaurus is applied to reduce the important words to basic keywords, which are used to query the database. The query is formed based on the query knowledge stored in the query knowledge base. Finally, the query result is synthesized into an English sentence, which is presented to the user as the answer. With an efficient part of speech tagger, intelligent subsystems for query formation and synthesis of query result, and user-friendly interface, the intelligent system can answer questions effectively.

A large variety of information processing applications deal with natural language texts. Many of these applications require extracting and processing the meanings of natural language texts, in addition to processing their syntactic forms. In order to extract meanings from texts and manipulate them, a natural language processing system must have a significant amount of knowledge about the world and the domain of discourse. However, different kinds of knowledge are required for different languages, for different functions, and for different media. We design a new general ontology based natural language processing system referred to as the multi-lingual multi-function multimedia intelligent system (MMM-IS). MMM-IS is a complex system with multiple functions that can deal with multiple languages and multiple media. The general ontology consists of a static layer and a dynamic layer, and provides a real world knowledge model for the natural language processing system. The paper presents the concept of MMM-IS and how to build MMM-IS for real applications.

We have paid attention to machine translation (MT) users for a few years, and found that most users prefer that the system is fast, inexpensive, easy to control and easy to update. This means that sometimes we do not need to aim the MT system at highly fluent translations. In fact, usually we cannot build an MT system which can achieve highly fluent translations. We propose a new MT paradigm called super-function based machine translation (SFBMT) to try to address the MT users' requests. SFBMT uses a super function (SF) to translate without thorough syntactic and semantic analysis as most MT systems usually do. The SF is a function that shows the correspondence between the original language sentence patterns and target language sentence patterns. One key point for building an SFBMT system is how to acquire the SF from the practical natural language phenomenon. We present a method for aiding the acquisition of SF from the parallel corpus, and describe the tool called MKSF. MKSF has been constructed in Java. An experimental MT system based on the proposed method has been built using Java and some translation experiments have been carried out.