Today's large language models (LLMs) are capable of supporting multilingual scenarios, allowing users to interact with LLMs in their native languages. When LLMs respond to subjective questions posed by users, they are expected to align with the views of specific demographic groups or historical periods, shaped by the language in which the user interacts with the model. Existing studies mainly focus on researching the opinions represented by LLMs among demographic groups in the United States or a few countries, lacking worldwide country samples and studies on human opinions in different historical periods, as well as lacking discussion on using language to steer LLMs. Moreover, they also overlook the potential influence of prompt language on the alignment of LLMs' opinions. In this study, our goal is to fill these gaps. To this end, we create an evaluation framework based on the World Values Survey (WVS) to systematically assess the alignment of LLMs with human opinions across different countries, languages, and historical periods around the world. We find that LLMs appropriately or over-align the opinions with only a few countries while under-aligning the opinions with most countries. Furthermore, changing the language of the prompt to match the language used in the questionnaire can effectively steer LLMs to align with the opinions of the corresponding country more effectively than existing steering methods. At the same time, LLMs are more aligned with the opinions of the contemporary population. To our knowledge, our study is the first comprehensive investigation of the topic of opinion alignment in LLMs across global, language, and temporal dimensions.

Large Language Models (LLMs) can learn both useful knowledge and harmful stereotypes, making bias evaluation essential.Existing frameworks fall into two types: those considering reasoning steps (Thinking Process-Aware Evaluation, TPAE) and those focusing only on final outputs (Straight-to-the-Answer Evaluation, SAE).Prior TPAE studies showed effectiveness in assessing gender bias but relied on template-based, word-counting prompts, limiting generalization to other bias types, languages, and reasoning-based methods.In this study, we introduce MBTP 1 , a multilingual social bias benchmark that incorporates humangenerated pro-and anti-stereotype reasoning as part of the thinking process, and propose a few-shot meta-evaluation method that enables scalable bias assessment without model finetuning.From experiments evaluating 13 social bias categories across 8 languages, we find that human-generated thinking consistently yields higher-quality evaluations than LLM-generated or template-based approaches.Furthermore, TPAE demonstrates superior performance over SAE, highlighting the importance of considering reasoning processes in bias evaluation.Warning: This paper may contain offensive language or harmful content.

Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyan Su, Mervat Abassy, Saadeldine Eletter, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The output tendencies of PLMs vary markedly before and after FT due to the updates to the model parameters. These divergences in output tendencies result in a gap in the social biases of PLMs. For example, there exits a low correlation between intrinsic bias scores of a PLM and its extrinsic bias scores under FT-based debiasing methods. Additionally, applying FT-based debiasing methods to a PLM leads to a decline in performance in downstream tasks. On the other hand, PLMs trained on large datasets can learn without parameter updates via ICL using prompts. ICL induces smaller changes to PLMs compared to FT-based debiasing methods. Therefore, we hypothesize that the gap observed in pre-trained and FT models does not hold true for debiasing methods that use ICL. In this study, we demonstrate that ICL-based debiasing methods show a higher correlation between intrinsic and extrinsic bias scores compared to FT-based methods. Moreover, the performance degradation due to debiasing is also lower in the ICL case compared to that in the FT case.

Multilingual large language models (MLLMs) are able to leverage in-context learning (ICL) to achieve high performance by leveraging crosslingual knowledge transfer without parameter updates.However, their effectiveness is highly sensitive to example selection, particularly in multilingual settings.Based on the findings of existing work, three key factors influence multilingual ICL: (1) semantic similarity, (2) linguistic alignment, and (3) language-specific performance.However, existing approaches address these factors independently, without explicitly disentangling their combined impact, leaving optimal example selection underexplored.To address this gap, we propose balanced multi-factor ICL (BMF-ICL), a method that quantifies and optimally balances these factors for improved example selection.Experiments on mCSQA and TYDI across four MLLMs demonstrate that BMF-ICL outperforms existing methods.Further analysis highlights the importance of incorporating all three factors and the importance of selecting examples from multiple languages.

Large Language Models (LLMs) are trained on massive web-crawled corpora, often containing personal information, copyrighted text, and benchmark datasets.This inadvertent inclusion in the training dataset, known as data leakage, poses significant risks and could compromise the safety of LLM outputs.Despite its criticality, existing studies do not examine how leaked instances in the pre-training data influence LLMs' output and detection capabilities.In this paper, we conduct an experimental survey to elucidate the relationship between data leakage in training datasets and its effects on the generation and detection by LLMs.Our experiments reveal that LLMs often generate outputs containing leaked information, even when there is little such data in the training dataset.Moreover, the fewer the leaked instances, the more difficult it becomes to detect such leakage.Finally, we demonstrate that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance.

Recent studies have demonstrated that Large Language Models (LLMs) have ethical-related problems such as social biases, lack of moral reasoning, and generation of offensive content. The existing evaluation metrics and methods to address these ethical challenges use datasets intentionally created by instructing humans to create instances including ethical problems. Therefore, the data does not sufficiently include comprehensive prompts that users actually provide when using LLM services in everyday contexts and outputs that LLMs generate. There may be different tendencies between unethical instances intentionally created by humans and actual user interactions with LLM services, which could result in a lack of comprehensive evaluation. To investigate the difference, we create Eagle datasets extracted from actual interactions between ChatGPT and users that exhibit social biases, opinion biases, toxicity, and immoral problems. Our experiments show that Eagle captures complementary aspects, not covered by existing datasets proposed for evaluation and mitigation. We argue that using both existing and proposed datasets leads to a more comprehensive assessment of the ethics.

Adversarial attacks by malicious users that threaten the safety of large language models (LLMs) can be viewed as attempts to infer a target property $T$ that is unknown when an instruction is issued, and becomes knowable only after the model's reply is observed. Examples of target properties $T$ include the binary flag that triggers an LLM's harmful response or rejection, and the degree to which information deleted by unlearning can be restored, both elicited via adversarial instructions. The LLM reveals an \emph{observable signal} $Z$ that potentially leaks hints for attacking through a response containing answer tokens, thinking process tokens, or logits. Yet the scale of information leaked remains anecdotal, leaving auditors without principled guidance and defenders blind to the transparency--risk trade-off. We fill this gap with an information-theoretic framework that computes how much information can be safely disclosed, and enables auditors to gauge how close their methods come to the fundamental limit. Treating the mutual information $I(Z;T)$ between the observation $Z$ and the target property $T$ as the leaked bits per query, we show that achieving error $\varepsilon$ requires at least $\log(1/\varepsilon)/I(Z;T)$ queries, scaling linearly with the inverse leak rate and only logarithmically with the desired accuracy. Thus, even a modest increase in disclosure collapses the attack cost from quadratic to logarithmic in terms of the desired accuracy. Experiments on seven LLMs across system-prompt leakage, jailbreak, and relearning attacks corroborate the theory: exposing answer tokens alone requires about a thousand queries; adding logits cuts this to about a hundred; and revealing the full thinking process trims it to a few dozen. Our results provide the first principled yardstick for balancing transparency and security when deploying LLMs.

Large Language Models (LLMs) have achieved human-level fluency in text generation, making it difficult to distinguish between human-written and LLM-generated texts. This poses a growing risk of misuse of LLMs and demands the development of detectors to identify LLM-generated texts. However, existing detectors lack robustness against attacks: they degrade detection accuracy by simply paraphrasing LLM-generated texts. Furthermore, a malicious user might attempt to deliberately evade the detectors based on detection results, but this has not been assumed in previous studies. In this paper, we propose OUTFOX, a framework that improves the robustness of LLM-generated-text detectors by allowing both the detector and the attacker to consider each other's output. In this framework, the attacker uses the detector's prediction labels as examples for in-context learning and adversarially generates essays that are harder to detect, while the detector uses the adversarially generated essays as examples for in-context learning to learn to detect essays from a strong attacker. Experiments in the domain of student essays show that the proposed detector improves the detection performance on the attacker-generated texts by up to +41.3 points F1-score. Furthermore, the proposed detector shows a state-of-the-art detection performance: up to 96.9 points F1-score, beating existing detectors on non-attacked texts. Finally, the proposed attacker drastically degrades the performance of detectors by up to -57.0 points F1-score, massively outperforming the baseline paraphrasing method for evading detection.

Discriminatory gender biases have been found in Pre-trained Language Models (PLMs) for multiple languages. In Natural Language Inference (NLI), existing bias evaluation methods have focused on the prediction results of one specific label out of three labels, such as neutral. However, such evaluation methods can be inaccurate since unique biased inferences are associated with unique prediction labels. Addressing this limitation, we propose a bias evaluation method for PLMs, called NLI-CoAL, which considers all the three labels of NLI task. First, we create three evaluation data groups that represent different types of biases. Then, we define a bias measure based on the corresponding label output of each data group. In the experiments, we introduce a meta-evaluation technique for NLI bias measures and use it to confirm that our bias measure can distinguish biased, incorrect inferences from non-biased incorrect inferences better than the baseline, resulting in a more accurate bias evaluation. We create the datasets in English, Japanese, and Chinese, and successfully validate the compatibility of our bias measure across multiple languages. Lastly, we observe the bias tendencies in PLMs of different languages. To our knowledge, we are the first to construct evaluation datasets and measure PLMs’ bias from NLI in Japanese and Chinese.

In Grammatical Error Correction (GEC), it is crucial to ensure the user’s comprehension of a reason for correction. Existing studies present tokens, examples, and hints for corrections, but do not directly explain the reasons in natural language. Although methods that use Large Language Models (LLMs) to provide direct explanations in natural language have been proposed for various tasks, no such method exists for GEC. Generating explanations for GEC corrections involves aligning input and output tokens, identifying correction points, and presenting corresponding explanations consistently. However, it is not straightforward to specify a complex format to generate explanations, because explicit control of generation is difficult with prompts. This study introduces a method called controlled generation with Prompt Insertion (PI) so that LLMs can explain the reasons for corrections in natural language. In PI, LLMs first correct the input text, and then we automatically extract the correction points based on the rules. The extracted correction points are sequentially inserted into the LLM’s explanation output as prompts, guiding the LLMs to generate explanations for the correction points. We also create an Explainable GEC (XGEC) dataset of correction reasons by annotating NUCLE, CoNLL2013, and CoNLL2014. Although generations from GPT-3.5 and ChatGPT using original prompts miss some correction points, the generation control using PI can explicitly guide to describe explanations for all correction points, contributing to improved performance in generating correction reasons.

Large Language Models (LLMs) can justify or critique their predictions through discussions with other models or humans, thereby enriching their intrinsic understanding of instances. While proactive discussions in the inference phase have been shown to boost performance, such interactions have not been extensively explored during the training phase. We hypothesize that incorporating interactive discussions into the training process can enhance the models’ understanding and improve their reasoning and verbal expression abilities during inference. This work introduces the SAIE framework, facilitating supportive and adversarial discussions between learner and partner models. The learner model receives responses from the partner, and its parameters are then updated based on this discussion. This dynamic adjustment process continues throughout the training phase, responding to the evolving outputs of the learner model. Our empirical evaluation across various tasks, including math word problems, commonsense reasoning, and multi-domain knowledge, demonstrates that models fine-tuned with the SAIE framework outperform those trained with conventional fine-tuning approaches. Furthermore, our method enhances the models’ reasoning capabilities, improving both individual and multi-agent inference performance. Our code is available at https://github.com/loem-ms/saie.

Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases.Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures.However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators.To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples.Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using genderrelated word lists.Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed.Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.

Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention.LLMs are also used for local sequence transduction tasks, including grammatical error correction (GEC) and formality style transfer, where most tokens in a source text are kept unchanged.However, the models that generate all target tokens in such tasks have a tendency to simply copy the input text as is, without making needed changes, because the difference between input and output texts is minimal in the training data.This is also inefficient because the computational cost grows quadratically with the target sequence length with Transformer.This paper proposes predicting edit spans for the source text for local sequence transduction tasks.Representing an edit span with a position of the source text and corrected tokens, we can reduce the length of the target sequence and the computational cost for inference.We apply instruction tuning for LLMs on the supervision data of edit spans.Experiments show that the proposed method achieves comparable performance to the baseline in four tasks, paraphrasing, formality style transfer, GEC, and text simplification, despite reducing the length of the target text by as small as 21%.Furthermore, we report that the task-specific fine-tuning with the proposed method achieved state-of-the-art performance in the four tasks.

Masked Language Models (MLMs) have shown superior performances in numerous downstream Natural Language Processing (NLP) tasks. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show that the previously proposed evaluation metrics for quantifying the social biases in MLMs are problematic due to the following reasons: (1) prediction accuracy of the masked tokens itself tend to be low in some MLMs, which leads to unreliable evaluation metrics, and (2) in most downstream NLP tasks, masks are not used; therefore prediction of the mask is not directly related to them, and (3) high-frequency words in the training data are masked more often, introducing noise due to this selection bias in the test cases. Therefore, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input and AUL with Attention weights (AULA) to evaluate tokens based on their importance in a sentence. Our experimental results show that the proposed bias evaluation measures accurately detect different types of biases in MLMs, and unlike AUL and AULA, previously proposed measures for MLMs systematically overestimate the measured biases and are heavily influenced by the unmasked tokens in the context.

Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word.One sense of an ambiguous word might be socially biased while its other senses remain unbiased.In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied.We create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures.We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures.Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at senselevel, which are often ignored by the word-level bias evaluation measures.1 * Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar.This paper describes work performed at the University of Liverpool and is not associated with Amazon.1 The dataset and evaluation scripts are available at github.com/LivNLP/bias-sense.

Grammatical Error Correction (GEC) should focus not only on correction accuracy but also on the interpretability of the results for language learners.However, existing neuralbased GEC models mostly focus on improving accuracy, while their interpretability has not been explored.Example-based methods are promising for improving interpretability, which use similar retrieved examples to generate corrections.Furthermore, examples are beneficial in language learning, helping learners to understand the basis for grammatically incorrect/correct texts and improve their confidence in writing.Therefore, we hypothesized that incorporating an example-based method into GEC could improve interpretability and support language learners.In this study, we introduce an Example-Based GEC (EB-GEC) that presents examples to language learners as a basis for correction result.The examples consist of pairs of correct and incorrect sentences similar to a given input and its predicted correction.Experiments demonstrate that the examples presented by EB-GEC help language learners decide whether to accept or refuse suggestions from the GEC output.Furthermore, the experiments show that retrieved examples also improve the accuracy of corrections.

We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for MLMs, and find that there exists only a weak correlation between these two types of evaluation measures. Moreover, we find that MLMs debiased using different methods still re-learn social biases during fine-tuning on downstream tasks. We identify the social biases in both training instances as well as their assigned labels as reasons for the discrepancy between intrinsic and extrinsic bias evaluation measurements. Overall, our findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures.

Automatic evaluation of grammatical error correction (GEC) is essential in developing useful GEC systems. Existing methods for automatic evaluation require multiple reference sentences or manual scores. However, such resources are expensive, thereby hindering automatic evaluation for various domains and correction styles. This paper proposes an Impact-based Metric for GEC using PARAllel data, IMPARA, which utilizes correction impacts computed by parallel data comprising pairs of grammatical/ungrammatical sentences. As parallel data is cheaper than manually assessing evaluation scores, IMPARA can reduce the cost of data creation for automatic evaluation. Correlations between IMPARA and human scores indicate that IMPARA is comparable or better than existing evaluation methods. Furthermore, we find that IMPARA can perform evaluations that fit different domains and correction styles trained on various parallel data.

This study investigates how supervised quality estimation (QE) models of grammatical error correction (GEC) are affected by the learners’ proficiency with the data. QE models for GEC evaluations in prior work have obtained a high correlation with manual evaluations. However, when functioning in a real-world context, the data used for the reported results have limitations because prior works were biased toward data by learners with relatively high proficiency levels. To address this issue, we created a QE dataset that includes multiple proficiency levels and explored the necessity of performing proficiency-wise evaluation for QE of GEC. Our experiments demonstrated that differences in evaluation dataset proficiency affect the performance of QE models, and proficiency-wise evaluation helps create more robust models.

Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, Naoaki Okazaki. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Word embeddings trained on large corpora have shown to encode high levels of unfair discriminatory gender, racial, religious and ethnic biases.In contrast, human-written dictionaries describe the meanings of words in a concise, objective and an unbiased manner.We propose a method for debiasing pre-trained word embeddings using dictionaries, without requiring access to the original training resources or any knowledge regarding the word embedding algorithms used.Unlike prior work, our proposed method does not require the types of biases to be pre-defined in the form of word lists, and learns the constraints that must be satisfied by unbiased word embeddings automatically from dictionary definitions of the words.Specifically, we learn an encoder to generate a debiased version of an input word embedding such that it (a) retains the semantics of the pre-trained word embeddings, (b) agrees with the unbiased definition of the word according to the dictionary, and (c) remains orthogonal to the vector space spanned by any biased basis vectors in the pre-trained word embedding space.Experimental results on standard benchmark datasets show that the proposed method can accurately remove unfair biases encoded in pre-trained word embeddings, while preserving useful semantics.* Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar.This paper describes work performed at the University of Liverpool and is not associated with Amazon.

In comparison to the numerous debiasing methods proposed for the static noncontextualised word embeddings, the discriminative biases in contextualised embeddings have received relatively little attention.We propose a fine-tuning method that can be applied at token-or sentence-levels to debias pre-trained contextualised embeddings.Our proposed method can be applied to any pretrained contextualised embedding model, without requiring to retrain those models.Using gender bias as an illustrative example, we then conduct a systematic study using several state-of-the-art (SoTA) contextualised representations on multiple benchmark datasets to evaluate the level of biases encoded in different contextualised embeddings before and after debiasing using the proposed method.We find that applying token-level debiasing for all tokens and across all layers of a contextualised embedding model produces the best performance.Interestingly, we observe that there is a trade-off between creating an accurate vs. unbiased contextualised embedding model, and different contextualised embedding models respond differently to this trade-off.* Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar.This paper describes work performed at the University of Liverpool and is not associated with Amazon.

In a real-time simultaneous translation setting and neural machine translation (NMT) models start generating target language tokens from incomplete source language sentences and making them harder to translate and leading to poor translation quality. Previous research has shown that document-level NMT and comprising of sentence and context encoders and a decoder and leverages context from neighboring sentences and helps improve translation quality. In simultaneous translation settings and the context from previous sentences should be even more critical. To this end and in this paper and we propose wait-k simultaneous document-level NMT where we keep the context encoder as it is and replace the source sentence encoder and target language decoder with their wait-k equivalents. We experiment with low and high resource settings using the ALT and OpenSubtitles2018 corpora and where we observe minor improvements in translation quality. We then perform an analysis of the translations obtained using our models by focusing on sentences that should benefit from the context where we found out that the model does and in fact and benefit from context but is unable to effectively leverage it and especially in a low-resource setting. This shows that there is a need for further innovation in the way useful context is identified and leveraged.

This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoderdecoder (EncDec) model for grammatical error correction (GEC).The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC.For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods.Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the finetuned MLM as additional features in the GEC model, maximizes the benefit of the MLM.The best-performing model achieves state-ofthe-art performances on the BEA-2019 and CoNLL-2014 benchmarks.Our code is publicly available at: https://github.com/ kanekomasahiro/bert-gec.

In this study, we propose a beam search method to obtain diverse outputs in a local sequence transduction task where most of the tokens in the source and target sentences overlap, such as in grammatical error correction (GEC).In GEC, it is advisable to rewrite only the local sequences that must be rewritten while leaving the correct sequences unchanged.However, existing methods of acquiring various outputs focus on revising all tokens of a sentence.Therefore, existing methods may either generate ungrammatical sentences because they force the entire sentence to be changed or produce non-diversified sentences by weakening the constraints to avoid generating ungrammatical sentences.Considering these issues, we propose a method that does not rewrite all the tokens in a text, but only rewrites those parts that need to be diversely corrected.Our beam search method adjusts the search token in the beam according to the probability that the prediction is copied from the source sentence.The experimental results show that our proposed method generates more diverse corrections than existing methods without losing accuracy in the GEC task.

Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings.However, theoretically this post-processing step is equivalent to applying a linear autoencoder to minimise the squared ℓ 2 reconstruction error.This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings.We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labeled data.

In this study, we explore cross-lingual transfer learning in grammatical error correction (GEC) tasks.Many languages lack the resources required to train GEC models.Cross-lingual transfer learning from high-resource languages (the source models) is effective for training models of low-resource languages (the target models) for various tasks.However, in GEC tasks, the possibility of transferring grammatical knowledge (e.g., grammatical functions) across languages is not evident.Therefore, we investigate cross-lingual transfer learning methods for GEC.Our results demonstrate that transfer learning from other languages can improve the accuracy of GEC.We also demonstrate that proximity to source languages has a significant impact on the accuracy of correcting certain types of errors.

We propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction.Previous studies have shown that reference-less metrics are promising; however, existing metrics are not optimized for manual evaluation of the system output because there is no dataset of system output with manual evaluation.This study manually evaluates the output of grammatical error correction systems to optimize the metrics.Experimental results show that the proposed metric improves the correlation with manual evaluation in both systemand sentence-level meta-evaluation.Our dataset and metric will be made publicly available.

Word embeddings learnt from massive text collections have demonstrated significant levels of discriminative biases such as gender, racial or ethnic biases, which in turn bias the down-stream NLP applications that use those word embeddings. Taking gender-bias as a working example, we propose a debiasing method that preserves non-discriminative gender-related information, while removing stereotypical discriminative gender biases from pre-trained word embeddings. Specifically, we consider four types of information: feminine, masculine, gender-neutral and stereotypical, which represent the relationship between gender vs. bias, and propose a debiasing method that (a) preserves the gender-related information in feminine and masculine words, (b) preserves the neutrality in gender-neutral words, and (c) removes the biases from stereotypical words. Experimental results on several previously proposed benchmark datasets show that our proposed method can debias pre-trained word embeddings better than existing SoTA methods proposed for debiasing word embeddings while preserving gender-related but non-discriminative information.

In this study, we improve grammatical error detection by learning word embeddings that consider grammaticality and error patterns. Most existing algorithms for learning word embeddings usually model only the syntactic context of words so that classifiers treat erroneous and correct words as similar inputs. We address the problem of contextual information by considering learner errors. Specifically, we propose two models: one model that employs grammatical error patterns and another model that considers grammaticality of the target word. We determine grammaticality of n-gram sequence from the annotated error tags and extract grammatical error patterns for word embeddings from large-scale learner corpora. Experimental results show that a bidirectional long-short term memory model initialized by our word embeddings achieved the state-of-the-art accuracy by a large margin in an English grammatical error detection task on the First Certificate in English dataset.

We introduce three adaptive routing algorithms for the hierarchical interconnection network TESH (tori-connected meshes) and evaluate their hardware costs and delays. TESH, which consists of a hierarchical torus interconnection between meshes as a basic module, is one of the k-ary n-cube networks, for which many adaptive routing algorithms have already been proposed. Adaptive routings for TESH have also been proposed in previous work.

This paper aims at obtaining diagnosis and prognosis information by searching similar images into a three-dimensional (3-D) CT image database of pulmonary nodules for which diagnosis is known. For this purpose, we propose an automatic method to retrieve nodule candidates with similar characteristics from the database. Each pulmonary nodule image is represented by the distribution pattern of CT density and 3-D curvature index. The nodule representation is then applied to a similarity measure such as a correlation coefficient. Our database is composed of 248 pulmonary nodules with associated clinical information. For each new case, we sort all the nodules of the database from most to less similar ones. By applying the retrieval method to our database, we present its feasibility to search the similar 3-D nodule images.

We are developing computerized feature extraction and classification methods to analyze malignant and benign pulmonary nodules in three-dimensional (3-D) thoracic CT images. Surrounding structure features were designed to characterize the relationships between nodules and their surrounding structures such as vessel, bronchi, and pleura. Internal structure features were derived from CT density and 3-D curvatures to characterize the inhomogeneous of CT density distribution inside the nodule. The stepwise linear discriminant classifier was used to select the best feature subset from multidimensional feature spaces. The discriminant scores output from the classifier were analyzed by the receiver operating characteristic (ROC) method and the classification accuracy was quantified by the area, Az, under the ROC curve. We analyzed a data set of 248 pulmonary nodules in this study. The internal structure features (Az=0.88) were more effective than the surrounding structure features (Az=0.69) in distinguishing malignant and benign nodules. The highest classification accuracy (Az=0.94) was obtained in the combined internal and surrounding structure feature space. The improvement was statistically significant in comparison to classification in either the internal structure or the surrounding structure feature space alone. The results of this study indicate the potential of using combined internal and surrounding structure features for computer-aided classification of pulmonary nodules.

We have developed a computer assisted automatic detection system for lung cancer that detects tumor candidates at an early stage from helical CT images. In July 1997, we started the comparative field trial using our system prospectively. Chest CT images obtained by helical CT scanner have drawn great interest in the detection of suspicious regions. However, mass screening based on helical CT images leads to a considerable number of images to be diagnosed. We expect that our system can reduce the time complexity and increase diagnostic confidence. We describe the results of nodule detection for definite diagnosis. We show the prospective results and the retrospective results. These results show that the system can detect lung cancer candidates at an early stage successfully and can be applied to a mass screening.

The paper describes the extraction of pulmonary major and minor fissures from three-dimensional (3D) chest multidetector-row computed tomography (MDCT) images. These fissures are used for the diagnosis of lung cancer and the analysis of pulmonary conformation. We have proposed (see Kubo, M. et al, IEEE Trans. Nucl. Sci, vol.46, p.2128-33, 1999) an automatic fissures extraction method using thin-section CT images with much noise. The present study proposes a simpler algorithm to extract fissures using MDCT images with little noise. The new proposed algorithm consists of the highlight method using the VanderBrug operator and the extraction method using morphology filters. We applied the proposed algorithm to one patient. Our method could accurately extract fissures.

This paper presents a computerized classification scheme of pulmonary nodules in contrast enhanced dynamic CT images. Conventionally, we extracted 3-D nodule images by using a deformable surface model. However, there was a limit in segmenting the 3-D nodule images contacted with vessels and bronchi. In order to improve the segmentation accuracy of the 3-D nodule images, we developed a software tool to eliminate the leaked region of the 3-D nodule image due to vessels and bronchi interactively. Using our data set including 68 cases (28 benign and 40 malignant cases), we demonstrate how the segmentation accuracy affects the classification accuracy of our scheme.

We present the techniques of recognizing pulmonary structure for differential diagnosis of lung cancer based on multidetector-row computed tomography (MDCT) images. These techniques consist of two steps. In the first step, we extract the pulmonary organs, such as vessels and bronchus using anatomical information and a 3D image processing method from MDCT images. As second step, these extracted regions are classified. This classification enables the lung tumor position to be recognized.

This paper presents an approach for characterizing the internal structure which is one of important clues for differentiating between malignant and benign nodules in three-dimensional (3-D) thoracic images. In this approach, each voxel was described in terms of shape index derived from curvatures on the voxel. The voxels inside the nodule were aggregated via a shape histogram to quantify how much shape category was present in the nodule. Topological features were introduced to characterize the morphology of the cluster constructed from a set of voxels with the same shape category. In the classification step, a hybrid unsupervised/supervised structure was performed to improve the classifier performance. It combined the k-means clustering procedure and the linear discriminate classifier. Receiver operating characteristics analysis was used to evaluate the accuracy of the classifiers. Our results demonstrate the feasibility of the hybrid classifier based on the topological and histogram features to assist physicians in making diagnostic decisions.

Presents a scheme to analyze nodule surrounding using differential geometry based vector fields in three-dimensional (3-D) thoracic images. In this scheme the differential characteristics such as the principal curvatures and directions are computed from the differential values of the isointensity surfaces. Each voxel in the nodule surrounding is described in terms of shape index and curvedness derived from the principal curvatures. Two vector fields are formed from the directions of the maximum principal curvatures of nodule surrounding and gradient vectors of nodule surface, respectively. The gradient vector field is computed by diffusing the gradient vector on the nodule surface. The regions corresponding to the cylindrical or conic figures which are similar to vessel and plural images are segmented by the shape index and curvedness values. Then, the relationship between the segmented regions and the nodule is evaluated by the inner product of the direction of the maximum principal curvature and the gradient vector. The authors demonstrate the effectiveness of their scheme by using real pulmonary nodule images.

This paper present an automatic extraction algorithm of the pulmonary major and minor fissures from three-dimensional (3-D) chest thin-section computed tomography (CT) images of helical CT. These fissures are used for the diagnosis of lung cancer and the analysis of pulmonary conformation. The proposed algorithm improves on the previous extraction method using the surface-curvatures calculation for density profile and morphological filters. The proposed method can extract the major and minor fissures in contact with the nodule and the chest walls. We apply the proposed algorithm to 12 patients. The results of our method are more accuracy to extract fissures around pulmonary lesions than by the previous method. The warped fissures extracted by our method show that lesions near fissures are malignant. Extracted fissures will aid in the diagnosis of lung cancer and in the analysis of automatic pulmonary conformation by using a computer.

This paper presents a computerized approach to characterize pulmonary nodules as benign or malignant based on contrast enhancement patterns extracted from serial three-dimensional (3-D) thoracic CT images. In this approach the registration procedure of sequential 3-D pulmonary images consisted of the rigid transformation between two sequential region-of-interest (ROI) images including the pulmonary nodule. The normalized mutual information was used as a voxel-based similarity measure in the registration. After motion correction between successive ROI images, the enhancement rate within a core of the segmented 3-D nodule image was estimated from the difference between the preand post-contrast images. We analyzed a data set of twelve 3-D thoracic CT images with pulmonary nodules in this study. Based on the Wilcoxon rank sum test, the median enhancement of the malignant lesions was significantly higher than that of the benign lesions (p<0.01). The preliminary results of the approach are very promising in characterizing pulmonary nodules based on quantitative measures of the contrast enhancement.

This paper focuses on an approach for characterizing the internal structure which is one of important clues for differentiating between malignant and benign nodules in 3D thoracic images. In this approach, each voxel was described in terms of shape index derived from curvatures on the voxel. The voxels inside the nodule were aggregated via shape histogram to quantify how much shape category was present in the nodule. Topological features were introduced to characterize the morphology of the cluster constructed from a set of voxels with the same shape category. In the classification step, a hybrid unsupervised/supervised structure was performed to improve the classifier performance. It combined the k-means clustering procedure and the linear discriminate classifier. The receiver operating characteristics analysis was used to evaluate the accuracy of the classifiers. Our results demonstrate the feasibility of the hybrid classifier based on the topological and histogram features to assist physicians in making diagnostic decisions.

The objective of the present paper is to extract the pulmonary major and minor fissures from 3D chest thin-section computed tomography (CT) images obtained by helical scan. These fissures are used for the diagnosis of lung cancer and the analysis of pulmonary conformation. We have proposed fissures extraction method without reference to streak artifacts and motion artifacts on the CT images. The new proposed algorithm improves on the previous extraction method using the surface-curvatures analysis for density profile and the morphological filters. The proposed method can also extract pulmonary fissures in contact with the module and the chest walls. We applied the proposed algorithm to 12 patients. The results of our method were more accuracy to extract fissures around pulmonary lesion than by the previous method. The warped fissures extracted by our method show that lesion near fissures is malignancy. Extracted fissures will be aided to diagnose lung cancer and to analyze automatically pulmonary conformation by using computer.

Multi-scale curvature indexes are introduced to characterize the internal intensity structure of pulmonary nodules in thin-section CT images. This approach makes use of shape index, curvedness, and CT density to represent locally each voxel constructing the three-dimensional (3D) pulmonary nodule image. Using features extracted from the histogram of the multi-scale curvature indexes and CT density, the pulmonary nodules are discriminated between benign and malignant cases by the linear discriminant classifier. In this study a data set of 128 pulmonary nodules is analyzed to investigate which scale provides high classification accuracy between malignant and benign nodules. Additionally, the extracted features are evaluated for four different regions: (i) entire 3D pulmonary nodule; (ii) core region in the 3D pulmonary nodule; (iii) complement of the-core region in the 3D pulmonary nodule; (iv) neighborhood region surrounding the 3D pulmonary nodule. The effectiveness of the multi-scale curvature indexes in a computer-aided differential diagnosis is demonstrated by receiver operating characteristic (ROC) analysis.

In this paper we propose three strategies for analysis of solitary pulmonary nodules in contrast enhanced dynamic CT images. First we collect contrasted images with regale time interval, and we analyze their CT values with respect to the time variation. In the second, we compare the shape spectra characteristic of each contrasted image. The third strategy consists of subtracting the noncontrasted images from contrasted images. The results show that each starkly promise good differentiation between benign and malignant nodule. So we try differential diagnosis.

To diagnose the lung cancer as to determine if it has malignant or benign nature, it is important to understand the spatial relationship among the abnormal nodule and other pulmonary organs. But the lung field has very complicated structure, so it is difficult to understand the connectivity of the pulmonary organs using thin-section CT images. This method consists of two parts. The first is the classification of the pulmonary structure based on the anatomical information. The second is the quantitative analysis that is then applicable to differential diagnosis, such as differentiation of malignant or benign abnormal tissue.

Presents a method to characterize the internal structure of small pulmonary nodules through a curvature-based descriptor using thin-section CT images. The work is a first step toward the segmentation of the three-dimensional (3D) nodule images by using a 3D deformable surfaces approach. Secondly, a curvature-based representation of the pulmonary nodule is derived. Based on this representation, the pulmonary nodules are globally characterized through the shape spectra. This quantification emphasizes the difference between benign and malignant pulmonary nodules surroundings. Experiments on true 3D nodule images demonstrate good performance of our curvature based analysis technique.

Shape characterization of small pulmonary nodules plays a significant role in differential diagnosis that discriminates malignant and benign nodules at early stages of pulmonary lesion development. This paper presents a method to characterize small pulmonary nodules based on the morphology of the development of lung lesions in thin-section CT images. The feature extraction algorithms are designed to extract the shape characteristic parameters from three-dimensional (3-D) nodule images using surface curvatures and ridge line. Experiments which show the feasibility of our method to improve the diagnostic accuracy are also demonstrated by applying the method to nodule images.

Helical computed tomography (CT) is a promising tool for the early diagnosis of lung cancer. The three-dimensional information makes it possible to detect a subtle change in any field of the lung. However, the diagnostic procedure is time-consuming, since a considerable number of images have to be reviewed in one examination. In order to lessen the burden to the reviewing physician and to improve the accuracy of diagnosis, we are developing a computer system, by which shadows of diagnostic importance can be highlighted among a number of nuisance changes. In particular, the peripheral blood vessels are analyzed with a special focus on the changes caused by lung cancer. We developed a computer algorithm, by which pulmonary blood vessels are extracted after removing the background bias. The comparison between the computer algorithm and an expert physician's reading showed a good agreement. Furthermore, this system can provide temporal changes in blood vessels, which are extremely important in diagnosis.