Knowledge graphs (KGs) can provide rich factual knowledge for language models , enhancing reasoning ability and interpretability . However, existing knowledge injection methods usually ignore the structured information in KGs. Using structured knowledge to enhance pre-trained language models (PLMs) still has a set of challenging issues, including resource consumption of knowledge retraining, heterogeneous information, and knowledge noise. To address these issues, we explore how to flexibly inject structured knowledge into frozen PLMs. Inspired by prompt learning, we propose a novel method K nowledge G raph Prompt (KG-Prompt), which for the first time encodes the KG as structured prompts to enhance the knowledge expression ability of PLMs. KG-Prompt consists of a compressed subgraph construction module and a KG prompt generation module. In the compressed subgraph construction module, we construct compressed subgraphs based on a path-weighting strategy to reduce knowledge noise. In the KG prompt generation module, we propose a multi-hop consistency optimization strategy to learn the representation of compressed subgraphs, and then generate KG prompts based on a knowledge mapper to solve the heterogeneous information problem. The KG prompts can be inserted into the input of PLMs expediently, which decouples from PLMs and the downstream model without knowledge retraining and reduces computational resources . Extensive experiments on three knowledge-driven natural language understanding tasks demonstrate that our approach effectively improves the knowledge reasoning ability of PLMs. Furthermore, we provide a detailed analysis of different KG prompts and discuss the interpretability and generalizability of the proposed method.

Network Representation Learning (NRL) has achieved remarkable success in learning low-dimensional representations for network nodes. However, most NRL methods, including Graph Neural Networks (GNNs) and their variants, face critical challenges. First, labeled network data, which are required for training most GNNs, are expensive to obtain. Second, existing methods are sub-optimal in preserving comprehensive topological information, including structural and positional information. Finally, most GNN approaches ignore the rich node content information. To address these challenges, we propose a self-supervised Network-to-Network framework (Net2Net) to learn semantically meaningful node representations. Our framework employs a pretext task of node position prediction (PosPredict) to effectively fuse the topological and content knowledge into low-dimensional embeddings for every node in a semi-supervised manner. Specifically, we regard a network as node content and position networks, where Net2Net aims to learn the mapping between them. We utilize a multi-layer recursively composable encoder to integrate the content and topological knowledge into the egocentric network node embeddings. Furthermore, we design a cross-modal decoder to map the egocentric node embeddings into their node position identities (PosIDs) in the node position network. Extensive experiments on eight diverse networks demonstrate the superiority of Net2Net over comparable methods.

The ability of processing long contexts is crucial for large language models (LLMs), but training LLMs with a long-context window requires substantial computational resources. Many sought to mitigate this through the sparse attention mechanism. However, sparse attention faces a noticeable gap compared with full attention in capturing long-distance information, leading to limited long-context processing capabilities. To effectively address this issue, this article proposes a novel sparse transformer architecture called 2-D transformer (2D-former), aimed at extending the context windows of pretrained LLMs while reducing GPU memory requirements. The 2D-former incorporates a 2-D attention mechanism that consists of a long-distance information compressor (LDIC) and a blockwise attention (BA) mechanism. LDIC can self-adaptively extract blockwise representational features by convolution and compress long-distance information into a set of tokens based on the significance of each block. The BA mechanism integrates these features, enabling each token to directly communicate with any of its preceding tokens during the computation of sparse attention. In this way, sparse attention can fully utilize long-distance information to bridge the gap with full attention while greatly reducing computational requirements. The 2D-former only needs to add less than 0.14% of additional trainable parameters to extend the context length of LLaMA2 7B to 32k on 4 A100 GPUs with 40-GB memory. In addition, it is compatible with most current acceleration techniques and parameter-efficient fine-tuning (PEFT) methods. Furthermore, we conduct supervised fine-tuning with 2D-former using our self-collected long-instruction fine-tuning dataset, named LongTuning, which comprises over 11k long-context question-answer (QA) pairs. Experimental results demonstrate that 2D-former achieves efficient long-context extension with minimal GPU memory and computational time consumption, while maintaining superior performance across both downstream long-context and short-context tasks.

Multi-modal information extraction aims to identify structured information, such as entities or relations between entities, from text with the help of visual clues. Although existing studies have achieved great progress, they mainly focused on modality interactions in the global space while neglecting fine-grained modality alignment under the semantic subspace specific to each entity type or relation type. To solve this problem, we propose a multi-space modality alignment method (MSMA) in this letter. The core of our model is a typespecific modality interaction module (TMI), which constructs a unique semantic subspace for each entity/relation type and independently performs type-specific modality alignments under each subspace. To enable mutual promotion between different types, a global modality integration module (GMI) is designed to learn the associations between different subspaces. Furthermore, we execute these two modules iteratively for high-level semantic fusion. Extensive experiments on three benchmark datasets show that our model significantly outperforms advanced methods.

We study the problem of Network Embedding (NE) for content-rich networks. NE models aim to learn efficient low-dimensional dense vectors for network vertices which are crucial to many network analysis tasks. The core problem of content-rich network embedding is to learn and integrate the semantic information conveyed by network structure and node content. In this paper, we propose a general end-to-end model,DualGEnerativeNetworkEmbedding (DGENE), to leverage the complementary information of network structure and content. In this model, each vertex is regarded as an object with two modalities: node identity and textual content. Then we formulate two dual generation tasks, Node Identification (NI) which recognizes nodes’ identities given their contents, and Content Generation (CG) which generates textual contents given the nodes’ identities. We develop specific Content2Node and Node2Content models for the two tasks. Under the DGENE framework, the two dual models are learned by sharing and integrating intermediate layers. Extensive experimental results show that our model yields a significant performance gain compared to the state-of-the-art NE methods. Moreover, our model has an interesting and useful byproduct, that is, a component of our model can generate texts and nodes, which is potentially useful for many tasks.

Knowledge graph (KG) is an important branch of artificial intelligence, which has attracted increasing research interest. However, in most enterprises, it is challenging to quickly construct KGs with multi-source and heterogeneous data and apply KGs to meet diverse business demands. To deal with these challenges, we propose an agile knowledge graph system following the novel principle of disentangling Demands, Algorithms, data reSources, and Humans (DASH). Specifically, our system is equipped with prior information-based knowledge extraction, self-supervised knowledge integration, and hierarchical knowledge base question answering algorithms that have outstanding generalizability and portability. Meanwhile, we propose a semi-automatic data accumulation framework to reduce labor costs of data annotations. Based on DASH, we develop a Web application with easy-to-use functionalities such as canvases and drag-and-drop, and illustrate its usage in a financial scenario.

Aspect sentiment triplet extraction (ASTE), which aims to identify aspects from review sentences along with their corresponding opinion expressions and sentiments, is an emerging task in fine-grained opinion mining. Since ASTE consists of multiple subtasks, including opinion entity extraction, relation detection, and sentiment classification, it is critical and challenging to appropriately capture and utilize the associations among them. In this paper, we transform ASTE task into a multi-turn machine reading comprehension (MTMRC) task and propose a bidirectional MRC (BMRC) framework to address this challenge. Specifically, we devise three types of queries, including non-restrictive extraction queries, restrictive extraction queries and sentiment classification queries, to build the associations among different subtasks. Furthermore, considering that an aspect sentiment triplet can derive from either an aspect or an opinion expression, we design a bidirectional MRC structure. One direction sequentially recognizes aspects, opinion expressions, and sentiments to obtain triplets, while the other direction identifies opinion expressions first, then aspects, and at last sentiments. By making the two directions complement each other, our framework can identify triplets more comprehensively. To verify the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets. The experimental results demonstrate that BMRC achieves state-of-the-art performances.

Specifying an appropriate skill set for a job position is critical for talent recruitment. However, it is often more difficult than people think since it needs a great understanding of the role of the position, related technologies, and even the global situation of the job market. To this end, we propose to learn the mapping between the job position description and its required skills in a data-driven manner. This task is challenging due to the complex mapping relationships between job descriptions and skills, which is caused by complex influencing factors. In this paper, we propose a novel Multi-Graph Neural Network based Skill Prediction model (MGNSP) to make skill prediction by learning effective deep semantics matching of job positions and skills. Specifically, to capture the complex heterogeneous relations among the job positions, skills, and meta information, we devise a joint learning approach of graph neural networks for multiple information networks, which are J-Net, S-Net and JS-Net, respectively. After jointly learning complementary semantics of job positions and skills with three multi-layer graph neural networks from these information networks, the skills are predicted by learning to match their representations. Extensive experimental results on a real-world dataset validate the effectiveness of our model.

The interpretable charge prediction task is to predict the final charges according to the fact descriptions, at the same time generate the corresponding explanations. The task is often more difficult than people think, as the explanation and prediction are highly interdependent, which is ignored by existing methods. In this paper, we deal with the interpretable charge prediction task from the perspective of generating the prediction and explanation (court view) pair while considering their dependency. To this end, we propose a Joint Prediction and Generation Model, named JPGM, which includes a coarse-to-fine classifier and a keyword-aware generator. Specifically, firstly, the classifier predicts a group of charge labels given the fact description. Then, we select the most matchable keywords of every charge from the pre-defined charge-discriminative keywords by an attention mechanism. Furthermore, the generator generates explanation using both the fact description and the most matchable keywords. Finally, to refine the prediction, the generated explanation and fact description are fused for final charge prediction. The classifier and generator are trained under an alternating procedure, which alleviates the error propagation. The experimental results validate that our model can effectively address the dependency issue and predict the charge with more interpretability.

Grammatical error correction (GEC) is an important application aspect of natural language processing techniques, and GEC system is a kind of very important intelligent system that has long been explored both in academic and industrial communities. The past decade has witnessed significant progress achieved in GEC for the sake of increasing popularity of machine learning and deep learning. However, there is not a survey that untangles the large amount of research works and progress in this field. We present the first survey in GEC for a comprehensive retrospective of the literature in this area. We first give the definition of GEC task and introduce the public datasets and data annotation schema. After that, we discuss six kinds of basic approaches, six commonly applied performance boosting techniques for GEC systems, and three data augmentation methods. Since GEC is typically viewed as a sister task of Machine Translation (MT), we put more emphasis on the statistical machine translation (SMT)-based approaches and neural machine translation (NMT)-based approaches for the sake of their importance. Similarly, some performance-boosting techniques are adapted from MT and are successfully combined with GEC systems for enhancement on the final performance. More importantly, after the introduction of the evaluation in GEC, we make an in-depth analysis based on empirical results in aspects of GEC approaches and GEC systems for a clearer pattern of progress in GEC, where error type analysis and system recapitulation are clearly presented. Finally, we discuss five prospective directions for future GEC researches.

This paper concerns the problem of network embedding (NE), which aims to learn low-dimensional representations for network nodes. Such dense representations offer great promises for many network analysis problems. However, existing approaches are still faced with challenges posed by the characteristics of complex real-world networks. First, for networks associated with rich content information, previous methods often learn separated content and structure representations, which requires post-processing of combination. Empirical combination strategies often make the final vectors suboptimal. Second, existing methods preserve the structure information by considering short and fixed neighborhood scope, such as the first- and/or the second-order proximities. However, it is hard to decide the neighborhood scope in complex problems. To this end, we propose a novel sequence to sequence model based NE framework referred to as Self-Translation Network Embedding (STNE). With the sampled node sequences, STNE translates each sequence itself from the content sequence to the node sequence. On the one hand, the bi-directional LSTM encoder fuses the content and structure information seamlessly from the raw input. On the other hand, high-order proximity can be flexibly learned with the memories of LSTM to capture long-range structural information. Experimental results on three real-world datasets demonstrate the superiority of STNE.

Opinion entity extraction is a fundamental task in fine-grained opinion mining.Related studies generally extract aspects and/or opinion expressions without recognizing the relations between them.However, the relations are crucial for downstream tasks, including sentiment classification, opinion summarization, etc.In this paper, we explore Aspect-Opinion Pair Extraction (AOPE) task, which aims at extracting aspects and opinion expressions in pairs.To deal with this task, we propose Synchronous Double-channel Recurrent Network (SDRN) mainly consisting of an opinion entity extraction unit, a relation detection unit, and a synchronization unit.The opinion entity extraction unit and the relation detection unit are developed as two channels to extract opinion entities and relations simultaneously.Furthermore, within the synchronization unit, we design Entity Synchronization Mechanism (ESM) and Relation Synchronization Mechanism (RSM) to enhance the mutual benefit on the above two channels.To verify the performance of SDRN, we manually build three datasets based on SemEval 2014 and 2015 benchmarks.Extensive experiments demonstrate that SDRN achieves state-of-the-art performances.

Writing a good job posting is a critical step in the recruiting process, but the task is often more difficult than many people think.It is challenging to specify the level of education, experience, relevant skills per the company information and job description.To this end, we propose a novel task of Job Posting Generation (JPG) that is cast as a conditional text generation problem to generate job requirements according to the job descriptions.To deal with this task, we devise a data-driven global Skill-Aware Multi-Attention generation model, named SAMA.Specifically, to model the complex mapping relationships between input and output, we design a hierarchical decoder that we first label the job description with multiple skills, then we generate a complete text guided by the skill labels.At the same time, to exploit the prior knowledge about the skills, we further construct a skill knowledge graph to capture the global prior knowledge of skills and refine the generated results.The proposed approach is evaluated on real-world job posting data.Experimental results clearly demonstrate the effectiveness of the proposed method 1 .

Joint entity and relation extraction is critical for many natural language processing (NLP) tasks, which has attracted increasing research interest. However, it is still faced with the challenges of identifying the overlapping relation triplets along with the entire entity boundary and detecting the multi-type relations. In this paper, we propose an attention-based joint model, which mainly contains an entity extraction module and a relation detection module, to address the challenges. The key of our model is devising a supervised multi-head self-attention mechanism as the relation detection module to learn the token-level correlation for each relation type separately. With the attention mechanism, our model can effectively identify overlapping relations and flexibly predict the relation type with its corresponding intensity. To verify the effectiveness of our model, we conduct comprehensive experiments on two benchmark datasets. The experimental results demonstrate that our model achieves state-of-the-art performances.

We study the problem of Network Embedding (NE) for content-rich networks. NE models aim to learn efficient low-dimensional dense vectors for network vertices which are crucial to many network analysis tasks. The core problem of content-rich network embedding is to learn and integrate the semantic information conveyed by network structure and node content. In this paper, we propose a general end-to-end model, Dual GEnerative Network Embedding (DGENE), to leverage the complementary information of network structure and content. In this model, each vertex is regarded as an object with two modalities: node identity and textual content. Then we formulate two dual generation tasks. One is Node Identification (NI) which recognizes nodes’ identities given their contents. Inversely, the other one is Content Generation (CG) which generates textual contents given the nodes’ identities. We develop specific Content2Node and Node2Content models for the two tasks. Under the DGENE framework, the two dual models are learned by sharing and integrating intermediate layers, with which they mutually enhance each other. Extensive experimental results show that our model yields a significant performance gain compared to the state-of-the-art NE methods. Moreover, our model has an interesting and useful byproduct, that is, a component of our model can generate texts, which is potentially useful for many tasks.

Recently, network embedding (NE) has achieved great successes in learning low dimensional representations for network nodes and has been increasingly applied to various network analytic tasks. In this paper, we consider the representation learning problem for content-rich networks whose nodes are associated with rich content information. Content-rich network embedding is challenging in fusing the complex structural dependencies and the rich contents. To tackle the challenges, we propose a generative model, Network-to-Network Network Embedding (Net2Net-NE) model, which can effectively fuse the structure and content information into one continuous embedding vector for each node. Specifically, we regard the content-rich network as a pair of networks with different modalities, i.e., content network and node network. By exploiting the strong correlation between the focal node and the nodes to whom it is connected to, a multilayer recursively composable encoder is proposed to fuse the structure and content information of the entire ego network into the egocentric node embedding. Moreover, a cross-modal decoder is deployed to mapping the egocentric node embeddings into node identities in an interconnected network. By learning the identity of each node according to its content, the mapping from content network to node network is learned in a generative manner. Hence the latent encoding vectors learned by the Net2Net-NE can be used as effective node embeddings. Extensive experimental results on three real-world networks demonstrate the superiority of Net2Net-NE over state-of-the-art methods.

Hashtags have always been important elements in many social network platforms and micro-blog services. Semantic understanding of hashtags is a critical and fundamental task for many applications on social networks, such as event analysis, theme discovery, information retrieval, etc. However, this task is challenging due to the sparsity, polysemy, and synonymy of hashtags. In this paper, we investigate the problem of hashtag embedding by combining the short text content with the various heterogeneous relations in social networks. Specifically, we first establish a network with hashtags as its nodes. Hierarchically, each of the hashtag nodes is associated with a set of tweets and each tweet contains a set of words. Then we devise an embedding model, called Hashtag2Vec, which exploits multiple relations of hashtag-hashtag, hashtag-tweet, tweet-word, and word-word relations based on the hierarchical heterogeneous network. In addition to embedding the hashtags, our proposed framework is capable of embedding the short social texts as well. Extensive experiments are conducted on two real-world datasets, and the results demonstrate the effectiveness of the proposed method.

This paper concerns the problem of network embedding (NE), whose aim is to learn low-dimensional representations for nodes in networks. Such dense vector representations offer great promises for many network analysis problems. However, existing NE approaches are still faced with challenges posed by the characteristics of complex networks in real-world applications. First, for many real-world networks associated with rich content information, previous NE methods tend to learn separated content and structure representations for each node, which requires a post-processing of combination. The empirical and simple combination strategies often make the final vector suboptimal. Second, the existing NE methods preserve the structure information by considering short and fixed neighborhood scope, such as the first- and/or the second-order proximities. However, it is hard to decide the scope of the neighborhood when facing a complex problem. To this end, we propose a novel sequence-to-sequence model based NE framework which is referred to as Self-Translation Network Embedding (STNE) model. With the sequences generated by random walks on a network, STNE learns the mapping that translates each sequence itself from the content sequence to the node sequence. On the one hand, the bi-directional LSTM encoder of STNE fuses the content and structure information seamlessly from the raw input. On the other hand, high-order proximity can be flexibly learned with the memories of LSTM to capture long-range structural information. By such self-translation from content to node, the learned hidden representations can be adopted as node embeddings. Extensive experimental results based on three real-world datasets demonstrate that the proposed STNE outperforms the state-of-the-art NE approaches. To facilitate reproduction and further study, we provide Internet access to the code and datasets\footnotehttp://dm.nankai.edu.cn/code/STNE.rar.

Human mobility analysis is one of the most important research problems in the field of urban computing. Existing research mainly focuses on the intra-city ground travel behavior modeling, while the inter-city air travel behavior modeling has been largely ignored. Actually, the inter-city travel analysis can be of equivalent importance and complementary to the intra-city travel analysis. Understanding massive passenger-air-travel behavior delivers intelligence for airlines’ precision marketing and related socioeconomic activities, such as airport planning, emergency management, local transportation planning, and tourism-related businesses. Moreover, it provides opportunities to study the characteristics of cities and the mutual relationships between them. However, modeling and predicting air traveler behavior is challenging due to the complex factors of the market situation and individual characteristics of customers (e.g., airlines’ market share, customer membership, and travelers’ intrinsic interests on destinations). To this end, in this article, we present a systematic study on the personalized air travel prediction problem, namely where a customer will fly to and which airline carrier to fly with, by leveraging real-world anonymized Passenger Name Record (PNR) data. Specifically, we first propose a relational travel topic model, which combines the merits of latent factor model with a neighborhood-based method, to uncover the personal travel preferences of aviation customers and the latent travel topics of air routes and airline carriers simultaneously. Then we present a multi-factor travel prediction framework, which fuses complex factors of the market situation and individual characteristics of customers, to predict airline customers’ personalized travel demands. Experimental results on two real-world PNR datasets demonstrate the effectiveness of our approach on both travel topic discovery and customer travel prediction.

We consider incorporating topic information into a sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior human knowledge that guides them to form informative and interesting responses in conversation, and leverages topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention and synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, with these vectors jointly affecting the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical studies on both automatic evaluation metrics and human annotations show that TA-Seq2Seq can generate more informative and interesting responses, significantly outperforming state-of-the-art response generation models.

Sub-event discovery is an effective method for social event analysis in Twitter. It can discover sub-events from large amount of noisy event-related information in Twitter and semantically represent them. The task is challenging because tweets are short, informal and noisy. To solve this problem, we consider leveraging event-related hashtags that contain many locations, dates and concise sub-event related descriptions to enhance sub-event discovery. To this end, we propose a hashtag-based mutually generative Latent Dirichlet Allocation model(MGe-LDA). In MGe-LDA, hashtags and topics of a tweet are mutually generated by each other. The mutually generative process models the relationship between hashtags and topics of tweets, and highlights the role of hashtags as a semantic representation of the corresponding tweets. Experimental results show that MGe-LDA can significantly outperform state-of-the-art methods for sub-event discovery.

In this paper, we introduce a new topic model to understand the chaotic microblogging environment by using hashtag graphs. Inferring topics on Twitter becomes a vital but challenging task in many important applications. The shortness and informality of tweets leads to extreme sparse vector representations with a large vocabulary. This makes the conventional topic models (e.g., latent Dirichlet allocation [1] and latent semantic analysis [2]) fail to learn high quality topic structures. Tweets are always showing up with rich user-generated hashtags. The hashtags make tweets semi-structured inside and semantically related to each other. Since hashtags are utilized as keywords in tweets to mark messages or to form conversations, they provide an additional path to connect semantically related words. In this paper, treating tweets as semi-structured texts, we propose a novel topic model, denoted as Hashtag Graphbased Topic Model (HGTM) to discover topics of tweets. By utilizing hashtag relation information in hashtag graphs, HGTM is able to discover word semantic relations even if words are not co-occurred within a specific tweet. With this method, HGTM successfully alleviates the sparsity problem. Our investigation illustrates that the user-contributed hashtags could serve as weakly-supervised information for topic modeling, and the relation between hashtags could reveal latent semantic relation between words. We evaluate the effectiveness of HGTM on tweet (hashtag) clustering and hashtag classification problems. Experiments on two real-world tweet data sets show that HGTM has strong capability to handle sparseness and noise problem in tweets. Furthermore, HGTM can discover more distinct and coherent topics than the state-of-the-art baselines.

Mining topics in Twitter is increasingly attracting more attention. However, the shortness and informality of tweets leads to extreme sparse vector representation with a large vocabulary, which makes the conventional topic models (e.g., Latent Dirichlet Allocation) often fail to achieve high quality underlying topics. Luckily, tweets always show up with rich user-generated hash tags as keywords. In this paper, we propose a novel topic model to handle such semi-structured tweets, denoted as Hash tag Graph based Topic Model (HGTM). By utilizing relation information between hash tags in our hash tag graph, HGTM establishes word semantic relations, even if they haven't co-occurred within a specific tweet. In addition, we enhance the dependencies of both multiple words and hash tags via latent variables (topics) modeled by HGTM. We illustrate that the user-contributed hash tags could serve as weakly-supervised information for topic modeling, and hash tag relation could reveal the semantic relation between tweets. Experiments on a real-world twitter data set show that our model provides an effective solution to discover more distinct and coherent topics than the state-of-the-art baselines and has a strong ability to control sparseness and noise in tweets.

The ever increasing usage of acronyms in many kinds of documents, including web pages, is becoming an obstacle for average readers. This paper studies the task of finding expansions in documents for a given set of acronyms. We cast the expansion finding problem as a sequence labeling task and adapt Conditional Random Fields (CRF) to solve it. While adapting CRFs, we enhance the performance from two aspects. First, we introduce nonlinear hidden layers to learn better representations of the input data. Second, we design simple and effective features. We create a hand labeled evaluation data based on Wikipedia.org and web crawling. We evaluate the effectiveness of several algorithms in solving the expansion finding problem. The experimental results demonstrate that the new method achieves performs better than Support Vector Machine and standard Conditional Random Fields.

Recently, combining Conditional Random Fields (CRF) with Neural Network has shown the success of learning high-level features in sequence labeling tasks. However, such models are difficult to train because of the increase of the parameters to tune which needs enormous of labeled data to avoid over fitting. In this paper, we propose a transfer learning framework for the sequence labeling task of gesture recognition. Taking advantage of the frame correlation, we design an unsupervised sequence model as a pseudo auxiliary task to capture the underlying information from both the labeled and unlabeled data. The knowledge learnt by the auxiliary task can be transferred to the main task of CRF with a deep architecture by sharing the hidden layers, which is very helpful for learning meaningful representation and reducing the need of labeled data. We evaluate our model under 3 gesture recognition datasets. The experimental results of both supervised learning and semi-supervised learning show that the proposed model improves the performance of the CRF with Neural Network and other baseline models.