Multi-modal information extraction aims to identify structured information, such as entities or relations between entities, from text with the help of visual clues. Although existing studies have achieved great progress, they mainly focused on modality interactions in the global space while neglecting fine-grained modality alignment under the semantic subspace specific to each entity type or relation type. To solve this problem, we propose a multi-space modality alignment method (MSMA) in this letter. The core of our model is a typespecific modality interaction module (TMI), which constructs a unique semantic subspace for each entity/relation type and independently performs type-specific modality alignments under each subspace. To enable mutual promotion between different types, a global modality integration module (GMI) is designed to learn the associations between different subspaces. Furthermore, we execute these two modules iteratively for high-level semantic fusion. Extensive experiments on three benchmark datasets show that our model significantly outperforms advanced methods.

Knowledge graph (KG) is an important branch of artificial intelligence, which has attracted increasing research interest. However, in most enterprises, it is challenging to quickly construct KGs with multi-source and heterogeneous data and apply KGs to meet diverse business demands. To deal with these challenges, we propose an agile knowledge graph system following the novel principle of disentangling Demands, Algorithms, data reSources, and Humans (DASH). Specifically, our system is equipped with prior information-based knowledge extraction, self-supervised knowledge integration, and hierarchical knowledge base question answering algorithms that have outstanding generalizability and portability. Meanwhile, we propose a semi-automatic data accumulation framework to reduce labor costs of data annotations. Based on DASH, we develop a Web application with easy-to-use functionalities such as canvases and drag-and-drop, and illustrate its usage in a financial scenario.

Aspect sentiment triplet extraction (ASTE), which aims to identify aspects from review sentences along with their corresponding opinion expressions and sentiments, is an emerging task in fine-grained opinion mining. Since ASTE consists of multiple subtasks, including opinion entity extraction, relation detection, and sentiment classification, it is critical and challenging to appropriately capture and utilize the associations among them. In this paper, we transform ASTE task into a multi-turn machine reading comprehension (MTMRC) task and propose a bidirectional MRC (BMRC) framework to address this challenge. Specifically, we devise three types of queries, including non-restrictive extraction queries, restrictive extraction queries and sentiment classification queries, to build the associations among different subtasks. Furthermore, considering that an aspect sentiment triplet can derive from either an aspect or an opinion expression, we design a bidirectional MRC structure. One direction sequentially recognizes aspects, opinion expressions, and sentiments to obtain triplets, while the other direction identifies opinion expressions first, then aspects, and at last sentiments. By making the two directions complement each other, our framework can identify triplets more comprehensively. To verify the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets. The experimental results demonstrate that BMRC achieves state-of-the-art performances.

Industrial Internet of Things has significantly boosted predictive maintenance for complex industrial systems, where the accurate prediction of remaining useful life (RUL) with high-level confidence is challenging. By aggregating multiple informative sources of system degradation, information fusion can be applied to improve the prediction accuracy and reduce the uncertainty. It can be performed on the data-level, feature-level, and decision-level. To fully exploit the available degradation information, this article proposes a hybrid fusion method on both the data level and decision level to predict the RUL. On the data level, genetic programming (GP) is adopted to integrate physical sensor sources into a composite health indicator (HI), resulting in an explicit nonlinear data-level fusion model. Subsequently, the predictions of the RUL based on each physical sensor and the developed composite HI are synthesized in the framework of belief functions theory, as the decision-level fusion method. Moreover, the decision-level method is flexible for incorporating other statistical data-driven methods with explicit estimations of the RUL. The proposed method is verified via a case study on NASA's C-MAPSS data set. Compared to the single-level fusion methods, the results confirm the superiority of the proposed method for higher accuracy and certainty of predicting the RUL.

In order to prevent catastrophic failures in power electronic systems, multiple failure precursors have been identified to characterize the degradation of power devices. However, there are some practical challenges in determining the suitable failure precursor, which supports the high-accuracy prediction of remaining useful life (RUL). This article proposes a method to formulate a composite failure precursor (CFP) by taking full advantage of potential failure precursors (PFPs), where CFP is directly optimized in terms of the degradation model to improve the prediction performance. The RUL estimations of the degradation model are explicitly derived to facilitate the precursor quality calculation. For CFP formulation, a genetic programming method is applied to integrate the PFPs in a nonlinear way. As a result, a framework that can formulate a superior failure precursor for the given RUL prediction model is elaborated. The proposed method is validated with the power cycling testing results of SiC MOSFETs.

Opinion entity extraction is a fundamental task in fine-grained opinion mining.Related studies generally extract aspects and/or opinion expressions without recognizing the relations between them.However, the relations are crucial for downstream tasks, including sentiment classification, opinion summarization, etc.In this paper, we explore Aspect-Opinion Pair Extraction (AOPE) task, which aims at extracting aspects and opinion expressions in pairs.To deal with this task, we propose Synchronous Double-channel Recurrent Network (SDRN) mainly consisting of an opinion entity extraction unit, a relation detection unit, and a synchronization unit.The opinion entity extraction unit and the relation detection unit are developed as two channels to extract opinion entities and relations simultaneously.Furthermore, within the synchronization unit, we design Entity Synchronization Mechanism (ESM) and Relation Synchronization Mechanism (RSM) to enhance the mutual benefit on the above two channels.To verify the performance of SDRN, we manually build three datasets based on SemEval 2014 and 2015 benchmarks.Extensive experiments demonstrate that SDRN achieves state-of-the-art performances.

Joint entity and relation extraction is critical for many natural language processing (NLP) tasks, which has attracted increasing research interest. However, it is still faced with the challenges of identifying the overlapping relation triplets along with the entire entity boundary and detecting the multi-type relations. In this paper, we propose an attention-based joint model, which mainly contains an entity extraction module and a relation detection module, to address the challenges. The key of our model is devising a supervised multi-head self-attention mechanism as the relation detection module to learn the token-level correlation for each relation type separately. With the attention mechanism, our model can effectively identify overlapping relations and flexibly predict the relation type with its corresponding intensity. To verify the effectiveness of our model, we conduct comprehensive experiments on two benchmark datasets. The experimental results demonstrate that our model achieves state-of-the-art performances.

This paper presents a new general method for evaluating the reliability function and the mean residual life of degrading systems subject to condition monitoring and random failure. In the proposed method, the degradation process of the system is characterized by a continuous-time Markov chain, which is then incorporated into the proportional hazards model as a stochastic covariate process to describe the hazard rate of the time to system failure. Unlike the conventional method based on conditioning, which is applicable only for a small number of degradation states, the proposed method is capable of tackling the case with a general number of degradation states. Using the developed approximation techniques, closed-form formulas for related reliability characteristics are obtained in terms of the appropriate transition probability matrix. The proposed evaluation algorithm is computationally efficient and embeddable to support real-time reliability assessment of the system subject to condition monitoring for developing the optimal maintenance policy. The effectiveness and the accuracy of the method are validated by a numerical study and compared with the conventional method. A general case where the degradation path can be discretized up to ten states is also studied to illustrate the appealing general features.