Query suggestion (QS) in web search must provide efficient and up-to-date suggestions that are relevant to the session context. Recent studies treat QS as generation, which captures session context well but suffers from high latency and costly retraining to maintain information freshness. In this study, we recast QS as dense retrieval: given a search session, the next query is retrieved from a large index of historical queries using efficient approximate nearest neighbor search. We propose Context-aware Asymmetric Dual Encoder for QS (CADE-QS), which uses an asymmetric dual encoder to model query -to- suggestion dependency and incorporates session history into the query encoder via context-aware contrastive learning. We evaluate our method on two real-world search-log datasets: a recent Japanese web search log and the public AOL log. CADE-QS rivals strong generative baselines in quality while reducing end-to-end latency to around 30 ms on CPU, representing an order-of-magnitude improvement. Detailed analyses confirm CADE-QS's robust context awareness, unidirectional modeling, and practicality for reflecting evolving information via index refreshes, as well as the effectiveness of an adaptive hybrid strategy for low-coverage scenarios. Our code is publicly available at https://github.com/lycorp-jp/cadeqs.

While feed-forward neurons in pre-trained language models (PLMs) can encode knowledge, past research targeted a small subset of neurons that heavily influence outputs.This leaves the broader role of neuron activations unclear, limiting progress in areas like knowledge editing.We uncover a global linear relationship between neuron activations and outputs using neuron interventions on a knowledge probing dataset.The gradient of this linear relationship, which we call the neuron empirical gradient (NEG), captures how changes in activations affect predictions.To compute NEG efficiently, we propose NeurGrad, enabling large-scale analysis of neuron behavior in PLMs.We also show that NEG effectively captures language skills across diverse prompts through skill neuron probing.Experiments on MCEval8k, a multi-genre multiple-choice knowledge benchmark, support NEG's ability to represent model knowledge.Further analysis highlights the key properties of NEG-based skill representation: efficiency, robustness, flexibility, and interdependency.The code and data are released.

Subtitles play a crucial role in improving the accessibility of the vast amount of audiovisual content available on the Internet, allowing audiences worldwide to comprehend and engage with this content in various languages.Automatic subtitling (AS) systems are essential for alleviating the substantial workload of human transcribers and translators.However, existing AS corpora and the primary metric SubER focus on European languages.This paper introduces A-TASC, an Asian TED-based automatic subtitling corpus derived from English TED Talks, comprising nearly 800 hours of audio segments, aligned English transcripts, and subtitles in Chinese, Japanese, Korean, and Vietnamese.We then present SacreSubER, a modification of SubER, to enable the reliable evaluation of subtitle quality for languages without explicit word boundaries.Experimental results, using both end-to-end systems and pipeline approaches built on strong ASR and LLM components, validate the quality of the proposed corpus and reveal differences in AS performance between European and Asian languages.The code to build our corpus is released.

Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs).In this study, we ask how ML-LMs acquire and represent factual knowledge.Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT).We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways in which ML-LMs acquire specific facts.We finally identified three patterns of acquiring and representing facts in ML-LMs: languageindependent, cross-lingual shared and transferred, and devised methods for differentiating them.Our findings highlight the challenge of maintaining consistent factual knowledge across languages, underscoring the need for better fact representation learning in ML-LMs. 1

We make decisions by reacting to changes in the real world, particularly the emergence and disappearance of impermanent entities such as restaurants, services, and events.Because we want to avoid missing out on opportunities or making fruitless actions after those entities have disappeared, it is important to know when * Part of this work was carried out during the period the author was at the University of Tokyo. 1 We release the datasets (tweet IDs) used in the experiments to promote reproducibility.URL I'm so sad to hear that Dave Laing has died.Dave was a very accomplished music industry journalist.Can't believe Google+ is being shut down.It's like when they just pulled Google Friends Connect… Here's your Demolition Day Planner for Martin Tower.A brief, stray shower can't be ruled out… Red Bull Air Race World Championship will not continue after 2019.

Although named entity recognition (NER) helps us to extract domain-specific entities from text (e.g., artists in the music domain), it is costly to create a large amount of training data or a structured knowledge base to perform accurate NER in the target domain.Here, we propose selfadaptive NER, which retrieves external knowledge from unstructured text to learn the usages of entities that have not been learned well.To retrieve useful knowledge for NER, we design an effective two-stage model that retrieves unstructured knowledge using uncertain entities as queries.Our model predicts the entities in the input and then finds those of which the prediction is not confident.Then, it retrieves knowledge by using these uncertain entities as queries and concatenates the retrieved text to the original input to revise the prediction.Experiments on CrossNER datasets demonstrated that our model outperforms strong baselines by 2.35 points in F 1 metric.

To prevent and curb viral outbreaks, such as COVID-19, it is important to increase vaccination coverage while resolving vaccine hesitancy and refusal. To understand why COVID-19 vaccination coverage had rapidly increased in Japan, we analyzed Twitter posts (tweets) to track the evolution of people's stance on vaccination and clarify the factors of why people decide to vaccinate. We collected all Japanese tweets related to vaccines over a five-month period and classified the vaccination stances of users who posted those tweets by using a deep neural network we designed. Examining diachronic changes in the users' stances on this large-scale vaccine dataset, we found that a certain number of neutral users changed to a pro-vaccine stance while very few changed to an anti-vaccine stance in Japan. Investigation of their information-sharing behaviors revealed what types of users and external sites were referred to when they changed their stances. These findings will help increase coverage of booster doses and future vaccinations.

Although screen readers enable visually impaired people to read written text via speech, the ambiguities in pronunciations of heteronyms cause wrong reading, which has a serious impact on the text understanding. Especially in Japanese, there are many common heteronyms expressed by logograms (Chinese characters or kanji) that have totally different pronunciations (and meanings). In this study, to improve the accuracy of pronunciation prediction, we construct two large-scale Japanese corpora that annotate kanji characters with their pronunciations. Using existing language resources on i) book titles compiled by the National Diet Library and ii) the books in a Japanese digital library called Aozora Bunko and their Braille translations, we develop two large-scale pronunciation-annotated corpora for training pronunciation prediction models. We first extract sentence-level alignments between the Aozora Bunko text and its pronunciation converted from the Braille data. We then perform dictionary-based pattern matching based on morphological dictionaries to find word-level pronunciation alignments. We have ultimately obtained the Book Title corpus with 336M characters (16.4M book titles) and the Aozora Bunko corpus with 52M characters (1.6M sentences). We analyzed pronunciation distributions for 203 common heteronyms, and trained a BERT-based pronunciation prediction model for 93 heteronyms, which achieved an average accuracy of 0.939.

Twitter is currently one of the most influential microblogging services on which users interact with messages. It is imperative to grasp the big picture of Twitter through analyzing its huge stream data. In this study, we develop a two-stage clustering method that automatically discovers coarse-grained topics from Twitter data. In the first stage, we use graph clustering to extract micro-clusters from the word co-occurrence graph. All the tweets in a micro-cluster share a fine-grained topic. We then obtain the time series of each micro-cluster by counting the number of tweets posted in a time window. In the second stage, we use time series clustering to identify the clusters corresponding to coarse-grained topics. We evaluate the computational efficacy of the proposed method and demonstrate its systematic improvement in scalability as the data volume increases. Next, we apply the proposed method to large-scale Twitter data (26 million tweets) about the COVID-19 Vaccination in Japan. The proposed method separately identifies the reactions to news and the reactions to tweets.

Although many end-to-end context-aware neural machine translation models have been proposed to incorporate inter-sentential contexts in translation, these models can be trained only in domains where parallel documents with sentential alignments exist.We therefore present a simple method to perform context-aware decoding with any pre-trained sentence-level translation model by using a document-level language model.Our context-aware decoder is built upon sentence-level parallel data and target-side document-level monolingual data.From a theoretical viewpoint, our core contribution is the novel representation of contextual information using point-wise mutual information between context and the current sentence.We demonstrate the effectiveness of our method on English to Russian translation, by evaluating with BLEU and contrastive tests for context-aware translation.

Long sentences have been one of the major challenges in neural machine translation (NMT).Although some approaches such as the attention mechanism have partially remedied the problem, we found that the current standard NMT model, Transformer, has difficulty in translating long sentences compared to the former standard, Recurrent Neural Network (RNN)-based model.One of the key differences of these NMT models is how the model handles position information which is essential to process sequential data.In this study, we focus on the position information type of NMT models, and hypothesize that relative position is better than absolute position.To examine the hypothesis, we propose RNN-Transformer which replaces positional encoding layer of Transformer by RNN, and then compare RNN-based model and four variants of Transformer.Experiments on ASPEC English-to-Japanese and WMT2014 Englishto-German translation tasks demonstrate that relative position helps translating sentences longer than those in the training data.Further experiments on length-controlled training data reveal that absolute position actually causes overfitting to the sentence length.

We present a method for applying a neural network trained on one (resource-rich) language for a given task to other (resource-poor) languages.We accomplish this by inducing a mapping from pre-trained cross-lingual word embeddings to the embedding layer of the neural network trained on the resource-rich language.To perform element-wise cross-task embedding projection, we invent locally linear mapping which assumes and preserves the local topology across the semantic spaces before and after the projection.Experimental results on topic classification task and sentiment analysis task showed that the fully task-specific multilingual model obtained using our method outperformed the existing multilingual models with embedding layers fixed to pre-trained cross-lingual word embeddings. 1

Keeping up to date on emerging entities that appear every day is indispensable for various applications, such as social-trend analysis and marketing research. Previous studies have attempted to detect unseen entities that are not registered in a particular knowledge base as emerging entities and consequently find non-emerging entities since the absence of entities in knowledge bases does not guarantee their emergence. We therefore introduce a novel task of discovering truly emerging entities when they have just been introduced to the public through microblogs and propose an effective method based on time-sensitive distant supervision, which exploits distinctive early-stage contexts of emerging entities. Experimental results with a large-scale Twitter archive show that the proposed method achieves 83.2% precision of the top 500 discovered emerging entities, which outperforms baselines based on unseen entity recognition with burst detection. Besides notable emerging entities, our method can discover massive long-tail and homographic emerging entities. An evaluation of relative recall shows that the method detects 80.4% emerging entities newly registered in Wikipedia; 92.8% of them are discovered earlier than their registration in Wikipedia, and the average lead-time is more than one year (578 days).

Shonosuke Ishiwatari, Jingtao Yao, Shujie Liu, Mu Li, Ming Zhou, Naoki Yoshinaga, Masaru Kitsuregawa, Weijia Jia. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

Various events, such as public gatherings, traffic accidents, and natural disasters, occur every day in mega-cities. Although understanding such ever-changing events over all these cities is important for urban planning, traffic management, and disaster response, this is quite a huge challenge. This paper proposes a method of visualizing spatio-temporal events with a multi-layered geo-locational word-cloud representation from a geo-parsed microblog stream. Real-time geo-parsing first geo-locates posts in the stream, using geo-tags and mentioned places and facilities as clues. Temporal local events are then identified and represented by a set of words specifically observed in a certain location and time grid, and then displayed above a map as word-clouds. We detect the locality of events to split them into multiple layers to avoid occlusions between local (e.g., music concerts) and global (e.g., earthquakes and marathon races) events. Users can thereby distinguish local from global events, and see their interactions over the layered maps. We demonstrate the effectiveness of our method by applying it to real events extracted from our archive accumulated from five years of Twitter posts.

We propose a method that learns a crosslingual projection of word representations from one language into another.Our method utilizes translatable context pairs as bonus terms of the objective function.In the experiments, our method outperformed existing methods in three language pairs, (English, Spanish), (Japanese, Chinese) and (English, Japanese), without using any additional supervisions.

Social media such as blogs and microblogs enable users to easily and rapidly publish information on their personal activities and interests. They are considered to provide valuable information from the viewpoints of sociology, linguistics, and marketing. This paper proposes a novel system for analyzing temporal changes in the activities and interests of bloggers through a 3D visualization of phrase dependency structures in sentences. We first extract events that represent bloggers' activities and interests through analyzing the phrase dependencies of sentences in a blog archive. We roughly categorize the retrieved events according to the thematic roles (such as the experiencer, agent, and location) of the noun within the events, and then store them in a dependency database so that we can retrieve events that involve a given topic. Second, we present a 3D visualization framework for exploring temporal changes in events related to a topic. Our framework enables users to find events about a topic that appear within a specific timing, and drill down details of the events. It also enables users to compare events with different timings and/or on multiple topics. Moreover, it allows them to observe an overview of temporal changes in sets of events, and long-term changes in the frequency of events to assist users in finding trends. We implement the proposed system on our own five-year blog archive that focused on Japanese, and we report the usefulness of our system by using various examples.

This paper presents a method for finding a specification page on the web for a given object (e.g."Titanic ö)and its class label (e.g."film ö). A specification page for an object is a web page which gives concise attribute-value information about the object (e.g."director ö-"James Cameron öfor "Titanic ö). A simple unsupervised method using layout and symbolic decoration cues was applied to a large number of web pages to acquire the class attributes. We used these acquired attributes to select a representative specification page for a given object from the web pages retrieved by a normal search engine. Experimental results revealed that our method greatly outperformed the normal search engine in terms of specification retrieval.