Large language models (LLMs) have demonstrated superior performance to that of small language models (SLM) in information retrieval for various subtasks including dense retrieval, reranking, query expansion, and pseudo-document generation. However, the parameter sizes of LLMs are extremely large, making it expensive to operate LLMs stably for providing LLM-based retrieval services. Recently, retrieval-augmented language models have been widely employed to significantly reduce the parameter size by retrieving relevant knowledge from large-scale corpora and exploiting the resulting “in-context” knowledge as additional model input, thereby substantially reducing the burden of internalizing and retaining world knowledge in model parameters. Armed by the retrieval-augmented language models, we present a retrieval-augmented model specialization that distills the capability of LLMs to generate the chain-of-thoughts (CoT) for query expansion – that is, injects the LLM’s capability to generate CoT into a retrieval-augmented SLM – referred to as RADCoT. Experimental results on the MS-MARCO, TREC DL 19, 20 datasets show that RADCoT yields consistent improvements over distillation without retrieval, achieving comparable performance to that of the query expansion method using LLM-based CoTs. Our code is publicly available at https://github.com/ZIZUN/RADCoT.

We present a simple and versatile framework for evaluating ranked lists in terms of Group Fairness and Relevance, in which the groups (i.e., possible attribute values) can be either nominal or ordinal in nature. First, we demonstrate that when our framework is applied to a binary hard group membership setting, our Group Fairness and Relevance (GFR) measures can easily quantify the overall polarity of each ranked list. Second, by utilising an existing diversified search test collection and treating each intent as an attribute value, we demonstrate that our framework can also handle soft group membership and that the GFR measures are highly correlated with a diversified information retrieval (IR) measure in this context as well. Third, using real data from a Japanese local search service, we demonstrate how our framework enables researchers to study intersectional group fairness based on multiple attribute sets. We also show that the similarity function for comparing the achieved and target distributions over the attribute values should be chosen carefully when the attribute values are ordinal. For such situations, our recommendation is to use multiple similarity functions with our framework: for example, one based on Jensen-Shannon Divergence (which disregards the ordinal nature of the groups) and another based on Root Normalised Order-aware Divergence (which has been designed specifically for handling ordinal groups). In addition, we highlight the fundamental differences between our framework and Attention-Weighted Rank Fairness (AWRF), a group fairness measure used at the TREC Fair Ranking Track.

Most approaches used in open-domain question answering on hybrid data that comprises both tabular-and-textual contents are based on a Retrieval-Reader pipeline in which the retrieval module finds relevant “heterogenous” evidence for a given question and the reader module generates an answer from the retrieved evidence. In this paper, we present a Retriever-Reranker-Reader framework by newly proposing a Reader-INherited evidence reranKer (RINK) where a reranker module is designed by finetuning the reader’s neural architecture based on a simple prompting method. Our underlying assumption of reusing the reader’s module for the reranker is that the reader’s ability to generating an answer from evidence contains the knowledge required for the reranking, because the reranker needs to “read” in-depth a question and evidences more carefully and elaborately than a baseline retriever. Furthermore, we present a simple and effective pretraining method by extensively deploying the commonly used data augmentation methods of cell corruption and cell reordering based on the pretraining tasks - tabular-and-textual entailment and cross-modal masked language modeling. Experimental results on OTT-QA, a large-scale table-and-text open-domain question answering dataset, show that the proposed RINK armed with our pretraining procedure makes improvements over the baseline reranking method and leads to state-of-the-art performance.

This study proposes Semantic-Infused SElective Graph Reasoning (SISER) for fact verification, which newly presents semantic-level graph reasoning and injects its reasoning-enhanced representation into other types of graph-based and sequence-based reasoning methods. SISER combines three reasoning types: 1) semantic-level graph reasoning, which uses a semantic graph from evidence sentences, whose nodes are elements of a triple – <Subject, Verb, Object>, 2) “semantic-infused” sentence-level “selective” graph reasoning, which combine semantic-level and sentence-level representations and perform graph reasoning in a selective manner using the node selection mechanism, and 3) sequence reasoning, which concatenates all evidence sentences and performs attention-based reasoning. Experiment results on a large-scale dataset for Fact Extraction and VERification (FEVER) show that SISER outperforms the previous graph-based approaches and achieves state-of-the-art performance.

While pre-trained language models play a vital role in modern language processing tasks, but not every language can benefit from them. Most existing research on pre-trained language models focuses primarily on widely-used languages such as English, Chinese, and Indo-European languages. Additionally, such schemes usually require extensive computational resources alongside a large amount of data, which is infeasible for less-widely used languages. We aim to address this research niche by building a language model that understands the linguistic phenomena in the target language which can be trained with low-resources. In this paper, we discuss Korean language modeling, specifically methods for language representation and pre-training methods. With our Korean-specific language representation, we are able to build more powerful language models for Korean understanding, even with fewer resources. The paper proposes chunk-wise reconstruction of the Korean language based on a widely used transformer architecture and bidirectional language representation. We also introduce morphological features such as Part-of-Speech (PoS) into the language understanding by leveraging such information during the pre-training. Our experiment results prove that the proposed methods improve the model performance of the investigated Korean language understanding tasks.

Video Question Answering (VideoQA) requires fine-grained understanding of both video and language modalities to answer the given questions. In this paper, we propose novel training schemes for multiple-choice video question answering with a self-supervised pre-training stage and a supervised contrastive learning in the main stage as an auxiliary learning. In the self-supervised pre-training stage, we transform the original problem format of predicting the correct answer into the one that predicts the relevant question to provide a model with broader contextual inputs without any further dataset or annotation. For contrastive learning in the main stage, we add a masking noise to the input corresponding to the ground-truth answer, and consider the original input of the ground-truth answer as a positive sample, while treating the rest as negative samples. By mapping the positive sample closer to the masked input, we show that the model performance is improved. We further employ locally aligned attention to focus more effectively on the video frames that are particularly relevant to the given corresponding subtitle sentences. We evaluate our proposed model on highly competitive benchmark datasets related to multiple-choice video QA: TVQA, TVQA+, and DramaQA. Experimental results show that our model achieves state-of-the-art performance on all datasets. We also validate our approaches through further analyses.

Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, Nako Sung. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

NAVER, Korea's No. 1 internet company, and LINE, the Japan-based global messenger platform, have teamed up to use their extensive online content databases (documents, maps, encyclopedia, etc.) and user information to develop an AI platform called Clova (Cloud-based Virtual Assistant). Clova's deep-learning based methods have made tremendous progress in image classification and intelligent Q&A response. For example, in Naver, users can procure product or location information by showing pictures. Keyword-based matching is enhanced using semantic analysis, and retrieving top links is changed to answering questions by refining user queries through dialogue. In addition to PC and mobile, Clova is evolving to enable a user to access the relevant information using multiple devices and interfaces. Line released a smart speaker called Clova Friends that includes Line's free voice-call function and infrared home-appliance control. IPTV set-top box powered-by the Clova AI platform which can recommend movies based on Naver's user reviews. In this talk, I will cover some efforts and challenges in understanding and satisfying users on each device with sophisticated natural language processing technologies. I will introduce the technically challenging problems that we are currently tackling and future AI developments.

We consider the problem of ranking rich verticals, which we call "cards," for a given mobile search query. Examples of card types include "SHOP" (showing access and contact information of a shop), "WEATHER" (showing a weather forecast for a particular location), and "TV" (showing information about a TV programme). These cards can be highly visual and/or concise, and may often satisfy the user's information need without making her click on them. While this "good abandonment" of the search engine result page is ideal especially for mobile environments where the interaction between the user and the search engine should be minimal, it poses a challenge for search engine companies whose ranking algorithms rely heavily on click data. In order to provide the right card types to the user for a given query, we propose a graph-based approach which extends a click-based automatic relevance estimation algorithm of Agrawal et al., by incorporating an abandonment-based preference rule. Using a real mobile query log from a commercial search engine, we constructed a data set containing 2,472 pairwise card type preferences covering 992 distinct queries, by hiring three independent assessors. Our proposed method outperforms a click-only baseline by 53-68% in terms of card type preference accuracy. The improvement is also statistically highly significant, with p ≈ 0.0000 according to the paired randomisation test.

Naver has been the most popular search engine for over a decade in South Korea. As a search portal, Naver aims to match a user's search intentions to the information from the web pages and databases, and to connect users based on shared interests to provide the best way to find the information. Over the past decade, Naver has been trying to better understand Korean users, queries, and web pages for PC and mobile search. In 2002, Naver introduced Knowledge-IN, which was the forerunner of community Question Answering to find out the need of users and topic experts. Users can ask their specific inquiry to appropriate topic experts in their search results. In addition to PC and mobile, Naver is trying to enable a user to access the relevant information using any other device or interface. In detecting common interest groups and good creators, Naver adds device and interface factors. Not only the contents, but also the delivery media types are important in satisfying users on various devices. Deep learning (DL) based methods have tremendous progress in image and text classification. With DL based methods, not only queries, and text documents, but also images, videos, live-streams, locations, etc. are classified and linked to detect common interest groups, and select and rank good creators and good delivery types in each group. With DL, Naver seeks to provide search results that meet user needs more precisely while learning and improving on the fly. In this talk, I'll cover some efforts and challenges in understanding and satisfying users on various devices.

Feature selection is a data processing method that is used to select a few important features among many input features and to remove any irrelevant one. Although feature selection in classification problems has been the focus of much research, few feature selection methods are available for use in one-class classification problems (i.e., anomaly detection). In particular, existing feature selection methods cannot be applied for the feature selection of the one-class classification problem when there are no available observations for the anomaly (or the second class). In this study, we propose two support vector data description (SVDD)-based feature selection methods: SVDD-radius-recursive feature elimination (RFE) and SVDD dual-objective RFE. The SVDD-radius-RFE method can be used to minimize the size of the boundary of describing normal observations measured through the value of its radius squared and the SVDD-dual-objective-RFE method can be applied to obtain a compact description in the dual space of SVDD. Experimental results using both simulated and real-life datasets demonstrate that the proposed methods show the improved performance compared with existing support vector machine RFE methods even for the classification problems when available observations for the anomaly are few.