Deep neural networks achieve high predictive accuracy by learning latent representations of complex data. However, the reasoning behind their decisions is difficult for humans to understand. On the other hand, rule-based approaches are able to justify the decisions by showing the decision rules leading to them, but they have relatively low accuracy. To improve the interpretability of neural networks, several techniques provide post-hoc explanations of decisions made by neural networks, but they cannot guarantee that the decisions are always explained in a simple form like decision rules because their explanations are generated after the decisions are made by neural networks.In this paper, to balance the accuracy of neural networks and the interpretability of decision rules, we propose a hybrid technique called rule-constrained networks, namely, neural networks that make decisions by selecting decision rules from a given ruleset. Because the networks are forced to make decisions based on decision rules, it is guaranteed that every decision is supported by a decision rule. Furthermore, we propose a technique to jointly optimize the neural network and the ruleset from which the network select rules. The log likelihood of correct classifications is maximized under a model with hyper parameters about the ruleset size and the prior probabilities of rules being selected. This feature makes it possible to limit the ruleset size or prioritize human-made rules over automatically acquired rules for promoting the interpretability of the output. Experiments on datasets of time-series and sentiment classification showed rule-constrained networks achieved accuracy as high as that achieved by original neural networks and significantly higher than that achieved by existing rule-based models, while presenting decision rules supporting the decisions.

Kosuke Akimoto, Takuya Hiraoka, Kunihiko Sadamasa, Mathias Niepert. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

The interpretability of machine learning models is a crucial factor if human experts have to check and explain why decisions are made. Because of the growing interest in in-terpretability, a simple rule-based model called a decision list has recently regained attention, and several techniques are proposed to improve the accuracy of decision lists by exploiting the power of modern high-speed processors. However, even with the help of modern processors, optimizing decision lists is still difficult and inefficient because of the discrete nature of decision lists. In this paper, we propose an algorithm that optimizes decision lists in a continuously relaxed form, instead of optimizing them in the original discrete form. First, from a given rule set, we construct a continuous decision list that is a continuously relaxed version of a decision list. It is optimized using a continuous optimization method and is finally converted into a decision list of the original discrete form. This approach can exploit the efficiency of continuous optimization methods supported by the optimization frameworks recently developed for deep learning. In experiments, our algorithm performed more efficiently than the state-of-the-art decision list learning algorithms, i.e., it achieved significantly higher accuracies in shorter times on large datasets.

Event relation knowledge is important for deep language understanding and inference.Previous work has established automatic acquisition methods of event relations that focus on common sense knowledge acquisition from largescale unlabeled corpus.However, in the case of domain-specific knowledge acquisition, such a method can not acquire much knowledge due to the limited amount of available knowledge sources.We propose an coverage-oriented acquisition method of event relations.The proposed method utilizes various patterns of dependency structures co-occurring with event relations than the existing method relying only on direct dependency relations between events.Experimental results show that the proposed method can acquire a larger amount of positive relation instances while keeping higher precision compared with the existing method and the proposed method also performs well for small sizes of corpora.

Daniel Andrade, Kunihiko Sadamasa, Akihiro Tamura, Masaaki Tsuchida. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

In this article we propose Supervised Semantic Indexing (SSI), an algorithm that is trained on (query, document) pairs of text documents to predict the quality of their match. Like Latent Semantic Indexing (LSI), our models take account of correlations between words (synonymy, polysemy). However, unlike LSI our models are trained with a supervised signal directly on the ranking task of interest, which we argue is the reason for our superior results. As the query and target texts are modeled separately, our approach is easily generalized to different retrieval tasks, such as online advertising placement. Dealing with models on all pairs of words features is computationally challenging. We propose several improvements to our basic model for addressing this issue, including low rank (but diagonal preserving) representations, and correlated feature hashing (CFH). We provide an empirical study of all these methods on retrieval tasks based on Wikipedia documents as well as an Internet advertisement task. We obtain state-of-the-art performance while providing realistically scalable methods.

Typical information extraction (IE) systems can be seen as tasks assigning labels to words in a natural language sequence. The performance is restricted by the availability of labeled words. To tackle this issue, we propose a semi-supervised approach to improve the sequence labeling procedure in IE through a class of algorithms with self-learned features (SLF). A supervised classifier can be trained with annotated text sequences and used to classify each word in a large set of unannotated sentences. By averaging predicted labels over all cases in the unlabeled corpus, SLF training builds class label distribution patterns for each word (or word attribute) in the dictionary and re-trains the current model iteratively adding these distributions as extra word features. Basic SLF models how likely a word could be assigned to target class types. Several extensions are proposed, such as learning words' class boundary distributions. SLF exhibits robust and scalable behaviour and is easy to tune. We applied this approach on four classical IE tasks: named entity recognition (German and English), part-of-speech tagging (English) and one gene name recognition corpus. Experimental results show effective improvements over the supervised baselines on all tasks. In addition, when compared with the closely related self-training idea, this approach shows favorable advantages.

We present a class of nonlinear (polynomial) models that are discriminatively trained to directly map from the word content in a query-document or document-document pair to a ranking score. Dealing with polynomial models on word features is computationally challenging. We propose a low rank (but diagonal preserving) representation of our polynomial models to induce feasible memory and computation requirements. We provide an empirical study on retrieval tasks based on Wikipedia documents, where we obtain state-of-the-art performance while providing realistically scalable methods.