Word representations, usually derived from a large corpus and endowed with rich semantic information, have been widely applied to natural language tasks. Traditional deep language models, on the basis of dense word representations, requires large memory space and computing resource. The brain-inspired neuromorphic computing systems, with the advantages of better biological interpretability and less energy consumption, still have major difficulties in the representation of words in terms of neuronal activities, which has restricted their further application in more complicated downstream language tasks. Comprehensively exploring the diverse neuronal dynamics of both integration and resonance, we probe into three spiking neuron models to post-process the original dense word embeddings, and test the generated sparse temporal codes on several tasks concerning both word-level and sentence-level semantics. The experimental results show that our sparse binary word representations could perform on par with or even better than original word embeddings in capturing semantic information, while requiring less storage. Our methods provide a robust representation foundation of language in terms of neuronal activities, which could potentially be applied to future downstream natural language tasks under neuromorphic computing systems.

The brain-inspired spiking neural networks (SNNs) are receiving increasing attention due to their asynchronous event-driven characteristics and low power consumption. As attention mechanisms recently become an indispensable part of sequence dependence modeling, the combination of SNNs and attention mechanisms holds great potential for energy-efficient and high-performance computing paradigms. However, the existing works cannot benefit from both temporal-wise attention and the asynchronous characteristic of SNNs. To fully leverage the advantages of both SNNs and attention mechanisms, we propose an SNNs-based spatial-temporal self-attention (STSA) mechanism, which calculates the feature dependence across the time and space domains without destroying the asynchronous transmission properties of SNNs. To further improve the performance, we also propose a spatial-temporal relative position bias (STRPB) for STSA to consider the spatiotemporal position of spikes. Based on the STSA and STRPB, we construct a spatial-temporal spiking Transformer framework, named STS-Transformer, which is powerful and enables SNNs to work in an asynchronous event-driven manner. Extensive experiments are conducted on popular neuromorphic datasets and speech datasets, including DVS128 Gesture, CIFAR10-DVS, and Google Speech Commands, and our experimental results can outperform other state-of-the-art models.

In the natural language processing family, learning representations is a pioneering study, especially in sequence-to-sequence tasks where outputs are generated, totally relying on the learning representations of source sequence. Generally, classic methods infer that each word occurring in the source sequence, having more or less influence on the target sequence, should all be considered when generating outputs. As the summarization task requires the output sequence to only retain the essence, classic full consideration of the source sequence may not work well on it, which calls for more suitable methods with the ability to discard the misleading noise words. Motivated by this, with both relevance retaining and redundancy removal in mind, we propose a summarization learning model by implementing an encoder with copious contextual information represented and a decoder with a selecting mechanism integrated. Specifically, we equip the encoder with an asynchronous bi directional parallel structure, in order to obtain abundant semantic representation. The decoder, different from the classic attention-based works, employs a self-aware context selecting mechanism to generate summary in a more productive way. We evaluate the proposed methods on three benchmark summarization corpora. The experimental results demonstrate the effectiveness and applicability of the proposed framework in relation to several well-practiced and state-of-the-art summarization methods.

Graph convolutional network (GCN), with its capability to update the current node features according to the features of its first-order adjacent nodes and edges, has achieved impressive performance in dependency capturing. But some important nodes from which we should figure out the dependencies are not first-order reachable, which calls for multi-layer GCNs for indirect relevance capturing. In this paper, we propose a novel weighted graph convolutional network by constructing a logical adjacency matrix which effectively solves the feature fusion of multi-hop relation without additional layers and parameters for relation extraction task. And we apply an Entity-Attention mechanism to enrich the entity pairs with more focused semantic information. Experimental results on TACRED and SemEval 2010 task 8 show that our model can take better advantage of the structural information in the dependency tree and produce better results than previous models.

MLPQ/GIS [4,6] is a constraint database [5] system like CCUBE [1] and DEDALE [3] but with a special emphases on spatio-temporal data. Features include data entry tools (first four icons in Fig. 1), icon-based queries such as @@@@ Intersection, @@@@ Union, @@@@ Area, @@@@ Buffer, @@@@ Max and @@@@ Min, which optimize linear objective functions, and @@@@ for Datalog queries. For example, in Fig. 1 we loaded and displayed a constraint database that represents the midwest United States and loaded two contraint relations describing the movements of two persons. The query icon opened a dialog box into which we entered the query which finds (t, i) pairs such that the two people are in the same state i at the same time t.