The utility of synthetic data to enhance pretraining data quality and hence to improve downstream task accuracy has been widely explored in recent large language models (LLMs). Yet, these approaches fall inadequate in complex, multi-hop and mathematical reasoning tasks as the synthetic data typically fails to add complementary knowledge to the existing raw corpus. In this work, we propose a novel large-scale and diverse Math Informed syNthetic Dialogue (MIND) generation method that improves the mathematical reasoning ability of LLMs. Specifically, using MIND, we generate synthetic conversations based on OpenWebMath (OWM), resulting in a new math corpus, MIND-OWM. Our experiments with different conversational settings reveal that incorporating knowledge gaps between dialog participants is essential for generating high-quality math data. We further identify an effective way to format and integrate synthetic and raw data during pretraining to maximize the gain in mathematical reasoning, emphasizing the need to restructure raw data rather than use it as-is. Compared to pretraining just on raw data, a model pretrained on MIND-OWM shows significant boost in mathematical reasoning (GSM8K: +13.42%, MATH: +2.30%), including superior performance in specialized knowledge (MMLU: +4.55%, MMLU-STEM: +4.28%) and general purpose reasoning tasks (GENERAL REASONING: +2.51%).

The retrieval model is an indispensable component for real-world knowledge-intensive tasks, e.g., open-domain question answering (ODQA).As separate retrieval skills are annotated for different datasets, recent work focuses on customized methods, limiting the model transferability and scalability.In this work, we propose a modular retriever where individual modules correspond to key skills that can be reused across datasets.Our approach supports flexible skill configurations based on the target domain to boost performance.To mitigate task interference, we design a novel modularization parameterization inspired by sparse Transformer.We demonstrate that our model can benefit from self-supervised pretraining on Wikipedia and fine-tuning using multiple ODQA datasets, both in a multi-task fashion.Our approach outperforms recent self-supervised retrievers in zero-shot evaluations and achieves state-ofthe-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA.

The retriever-reader framework is popular for open-domain question answering (ODQA) due to its ability to use explicit knowledge.Although prior work has sought to increase the knowledge coverage by incorporating structured knowledge beyond text, accessing heterogeneous knowledge sources through a unified interface remains an open question.While data-to-text generation has the potential to serve as a universal interface for data and text, its feasibility for downstream tasks remains largely unknown.In this work, we bridge this gap and use the data-to-text method as a means for encoding structured knowledge for ODQA.Specifically, we propose a verbalizer-retrieverreader framework for ODQA over data and text where verbalized tables from Wikipedia and graphs from Wikidata are used as augmented knowledge sources.We show that our Unified Data and Text QA, UDT-QA, can effectively benefit from the expanded knowledge index, leading to large gains over textonly baselines.Notably, our approach sets the single-model state-of-the-art on Natural Questions.Furthermore, our analyses indicate that verbalized knowledge is preferred for answer reasoning for both adapted and hot-swap settings.

Procedural text understanding is a challenging language reasoning task that requires models to track entity states across the development of a narrative. We identify three core aspects required for modeling this task, namely the local and global view of the inputs, as well as the global view of outputs. Prior methods have considered a subset of these aspects, which leads to either low precision or low recall. In this paper, we propose a new model Coalescing Global and Local Information (CGLI), which builds entity- and timestep-aware input representations (local input) considering the whole context (global input), and we jointly model the entity states with a structured prediction objective (global output). Thus, CGLI simultaneously optimizes for both precision and recall. Moreover, we extend CGLI with additional output layers and integrate it into a story reasoning framework. Extensive experiments on a popular procedural text understanding dataset show that our model achieves state-of-the-art results, while experiments on a story reasoning benchmark show the positive impact of our model on downstream reasoning.

Code-switched (CS) data is ubiquitous in today's globalized world, but the dearth of annotated datasets in code-switching poses a significant challenge for learning diverse tasks across different language pairs.Parameter-efficient prompt-tuning approaches conditioned on frozen language models have shown promise for transfer learning in limited-resource setups.In this paper, we propose a novel instancebased prompt composition technique, PRO-CS, for CS tasks that combine language and task knowledge.We compare our approach with prompt-tuning and fine-tuning for codeswitched tasks on 10 datasets across 4 language pairs.Our model outperforms the prompttuning approach by significant margins across all datasets and outperforms or remains at par with fine-tuning by using just 0.18% of total parameters.We also achieve competitive results when compared with the fine-tuned model in the low-resource cross-lingual and crosstask setting, indicating the effectiveness of our approach to incorporate new code-switched tasks.

Recent developments in pre-trained neural language modeling have led to leaps in accuracy on common-sense question-answering benchmarks. However, there is increasing concern that models overfit to specific tasks, without learning to utilize external knowledge or perform general semantic reasoning. In contrast, zero-shot evaluations have shown promise as a more robust measure of a model’s general reasoning abilities. In this paper, we propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre-training models. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. Extending on prior work, we devise and compare four constrained distractor-sampling strategies. We provide empirical results across five commonsense question-answering tasks with data generated from five external knowledge resources. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks. In addition, both preserving the structure of the task as well as generating fair and informative questions help language models learn more effectively.

Commonsense reasoning benchmarks have been largely solved by fine-tuning language models.The downside is that fine-tuning may cause models to overfit to task-specific data and thereby forget their knowledge gained during pre-training.Recent works only propose lightweight model updates as models may already possess useful knowledge from past experience, but a challenge remains in understanding what parts and to what extent models should be refined for a given task.In this paper, we investigate what models learn from commonsense reasoning datasets.We measure the impact of three different adaptation methods on the generalization and accuracy of models.Our experiments with two models show that fine-tuning performs best, by learning both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers.We observe that alternative adaptation methods like prefixtuning have comparable accuracy, but generalize better to unseen answers and are more robust to adversarial splits.

Existing research on autonomous driving primarily focuses on urban driving, which is insufficient for characterising the complex driving behaviour underlying high-speed racing. At the same time, existing racing simulation frameworks struggle in capturing realism, with respect to visual rendering, vehicular dynamics, and task objectives, inhibiting the transfer of learning agents to real-world contexts. We introduce a new environment, where agents Learn-to-Race (L2R) in simulated competition-style racing, using multimodal information—from virtual cameras to a comprehensive array of inertial measurement sensors. Our environment, which includes a simulator and an interfacing training framework, accurately models vehicle dynamics and racing conditions. In this paper, we release the Arrival simulator for autonomous racing. Next, we propose the L2R task with challenging metrics, inspired by learning-to-drive challenges, Formula-style racing, and multimodal trajectory prediction for autonomous driving. Additionally, we provide the L2R framework suite, facilitating simulated racing on high-precision models of real-world tracks. Finally, we provide an official L2R task dataset of expert demonstrations, as well as a series of baseline experiments and reference implementations. We make all code available: https://github.com/learn-to-race/l2r.

While data science education has gained increased recognition in both academic institutions and industry, there has been a lack of research on automated coding assessment for novice students. Our work presents a first step in this direction, by leveraging the coding metrics from traditional software engineering (Halstead Volume and Cyclomatic Complexity) in combination with those that reflect a data science project’s learning objectives (number of library calls and number of common library calls with the solution code). Through these metrics, we examined the code submissions of 97 students across two semesters of an introductory data science course. Our results indicated that the metrics can identify cases where students had overly complicated codes and would benefit from scaffolding feedback. The number of library calls, in particular, was also a significant predictor of changes in submission score and submission runtime, which highlights the distinctive nature of data science programming. We conclude with suggestions for extending our analyses towards more actionable intervention strategies, for example by tracking the fine-grained submission grading outputs throughout a student’s submission history, to better model and support them in their data science learning process.

Information need of humans is essentially multimodal in nature, enabling maximum exploitation of situated context.We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain.The dataset consists of 16,441 cooking recipes with 160,479 photos associated with different steps.We setup a baseline motivated by the best performing model in terms of human evaluation for the Visual Story Telling (ViST) task.In addition, we introduce two models to incorporate high level structure learnt by a Finite State Machine (FSM) in neural sequential generation process by: (1) Scaffolding Structure in Decoder (SSiD) (2) Scaffolding Structure in Loss (SSiL).Our best performing model (SSiL) achieves a METEOR score of 0.31, which is an improvement of 0.6 over the baseline model.We also conducted human evaluation of the generated grounded recipes, which reveal that 61% found that our proposed (SSiL) model is better than the baseline model in terms of overall recipes.We also discuss analysis of the output highlighting key important NLP issues for prospective directions.

mHealth is becoming increasingly data-driven in healthcare professions. Clinicians use a variety of data collection methods to capture the language use and communication performance of clients with complex communication needs. One solution for collecting and analyzing augmentative and alternative communication (AAC) data is through automatic data logging from high-tech AAC devices, also known as speech generating devices (SGDs). However, there is no interoperable method to analyze data logs across various SGDs. To address this, our work presents an interoperable data log format and a parser as a prototype solution. The prototype was used successfully to analyze two common AAC data log formats, and can easily be extended to other formats. This approach has significant potential to improve AAC outcome measurement for individual users, as well as comparison of outcomes across multiple users and devices.

Portmanteaus are a word formation phenomenon where two words are combined to form a new word.We propose character-level neural sequence-tosequence (S2S) methods for the task of portmanteau generation that are end-toend-trainable, language independent, and do not explicitly use additional phonetic information.We propose a noisy-channelstyle model, which allows for the incorporation of unsupervised word lists, improving performance over a standard sourceto-target model.This model is made possible by an exhaustive candidate generation strategy specifically enabled by the features of the portmanteau task.Experiments find our approach superior to a state-of-the-art FST-based baseline with respect to ground truth accuracy and human evaluation.

Deep neural networks for machine comprehension typically utilizes only word or character embeddings without explicitly taking advantage of structured linguistic information such as constituency trees and dependency trees.In this paper, we propose structural embedding of syntactic trees (SEST), an algorithm framework to utilize structured information and encode them into vector representations that can boost the performance of algorithms for the machine comprehension.We evaluate our approach using a state-of-the-art neural attention model on the SQuAD dataset.Experimental results demonstrate that our model can accurately identify the syntactic boundaries of the sentences and extract answers that are syntactically coherent over the baseline methods.

We propose simple and flexible training and decoding methods for influencing output style and topic in neural encoderdecoder based language generation.This capability is desirable in a variety of applications, including conversational systems, where successful agents need to produce language in a specific style and generate responses steered by a human puppeteer or external knowledge.We decompose the neural generation process into empirically easier sub-problems: a faithfulness model and a decoding method based on selectivesampling.We also describe training and sampling algorithms that bias the generation process with a specific language style restriction, or a topic restriction.Human evaluation results show that our proposed methods are able to to restrict style and topic without degrading output quality in conversational tasks.chat context Where are you?Twitter bot i'm on my way to london now.Star Wars bot we're heading to the dark side of jabba's palace.Hillary bot i'm here in philadelphia, pennsylvania Trump bot i went to the white house.we were in brooklyn.Kennedy bot i am on the question of the united states.chat context What are you doing?Twitter bot i'm just chilling with my dad, i'm just sitting in the kitchen and listening to music Star Wars bot i'm doing my magic solo.Hillary bot i'm running for president, i'm going to be talking about some of these things Trump bot i'm building a wall, right now Kennedy bot today, i am asking the congress for a new program to make a new effort to increase the tax privileges and to stimulate

Building dialogue interfaces for realworld scenarios often entails training semantic parsers starting from zero examples.How can we build datasets that better capture the variety of ways users might phrase their queries, and what queries are actually realistic?Wang et al. (2015) proposed a method to build semantic parsing datasets by generating canonical utterances using a grammar and having crowdworkers paraphrase them into natural wording.A limitation of this approach is that it induces bias towards using similar language as the canonical utterances.In this work, we present a methodology that elicits meaningful and lexically diverse queries from users for semantic parsing tasks.Starting from a seed lexicon and a generative grammar, we pair logical forms with mixed text-image representations and ask crowdworkers to paraphrase and confirm the plausibility of the queries that they generated.We use this method to build a semantic parsing dataset from scratch for a dialog agent in a smart-home simulation.We find evidence that this dataset, which we have named SMARTHOME, is demonstrably more lexically diverse and difficult to parse than existing domain-specific semantic parsing datasets.

Retrieval pipelines commonly rely on a term-based search to obtain candidate records, which are subsequently re-ranked. Some candidates are missed by this approach, e.g., due to a vocabulary mismatch. We address this issue by replacing the term-based search with a generic k-NN retrieval algorithm, where a similarity function can take into account subtle term associations. While an exact brute-force k-NN search using this similarity function is slow, we demonstrate that an approximate algorithm can be nearly two orders of magnitude faster at the expense of only a small loss in accuracy. A retrieval pipeline using an approximate k-NN search can be more effective and efficient than the term-based pipeline. This opens up new possibilities for designing effective retrieval pipelines. Our software (including data-generating code) and derivative data based on the Stack Overflow collection is available online.

This paper presents first results from a proof of feasibility experiment in conceptual legal document retrieval in a particular domain (involving vaccine injury compensation). The conceptual markup of documents is done automatically using LUIMA, a law-specific semantic extraction toolbox based on the UIMA framework. The system consists of modules for automatic sub-sentence level annotation, machine learning based sentence annotation, basic retrieval using Apache Lucene and a machine learning based reranking of retrieved documents. In a leave-one-out experiment on a limited corpus, the resulting rankings scored higher for most tested queries than baseline rankings created using a commercial full-text legal information system.

Many search engine users attempt to satisfy an information need by issuing multiple queries, with the expectation that each result will contribute some portion of the required information. Previous research has shown that structured or semi-structured descriptive knowledge bases (such as Wikipedia) can be used to improve search quality and experience for general or entity-centric queries. However, such resources do not have sufficient coverage of procedural knowledge, i.e. what actions should be performed and what factors should be considered to achieve some goal; such procedural knowledge is crucial when responding to task-oriented search queries. This paper provides a first attempt to bridge the gap between two evolving research areas: development of procedural knowledge bases (such as wikiHow) and task-oriented search. We investigate whether task-oriented search can benefit from existing procedural knowledge (search task suggestion) and whether automatic procedural knowledge construction can benefit from users' search activities (automatic procedural knowledge base construction). We propose to create a three-way parallel corpus of queries, query contexts, and task descriptions, and reduce both problems to sequence labeling tasks. We propose a set of textual features and structural features to identify key search phrases from task descriptions, and then adapt similar features to extract wikiHow-style procedural knowledge descriptions from search queries and relevant text snippets. We compare our proposed solution with baseline algorithms, commercial search engines, and the (manually-curated) wikiHow procedural knowledge; experimental results show an improvement of +0.28 to +0.41 in terms of [email protected] and mean average precision (MAP).

We survey permutation-based methods for approximate k -nearest neighbor search. In these methods, every data point is represented by a ranked list of pivots sorted by the distance to this point. Such ranked lists are called permutations. The underpinning assumption is that, for both metric and non-metric spaces, the distance between permutations is a good proxy for the distance between original points. Thus, it should be possible to efficiently retrieve most true nearest neighbors by examining only a tiny subset of data points whose permutations are similar to the permutation of a query. We further test this assumption by carrying out an extensive experimental evaluation where permutation methods are pitted against state-of-the art benchmarks (the multi-probe LSH, the VP-tree, and proximity-graph based retrieval) on a variety of realistically large data set from the image and textual domain. The focus is on the high-accuracy retrieval methods for generic spaces. Additionally, we assume that both data and indices are stored in main memory. We find permutation methods to be reasonably efficient and describe a setup where these methods are most useful. To ease reproducibility, we make our software and data sets publicly available.

We show that it is possible to reliably dis-criminate whether a syntactic construction is meant literally or metaphorically using lexical semantic features of the words that participate in the construction. Our model is constructed using English resources, and we obtain state-of-the-art performance relative to previous work in this language. Using a model transfer approach by piv-oting through a bilingual dictionary, we show our model can identify metaphoric expressions in other languages. We pro-vide results on three new test sets in Span-ish, Farsi, and Russian. The results sup-port the hypothesis that metaphors are conceptual, rather than lexical, in nature. 1

As the scale of available on-line data grows ever larger, individuals and businesses must cope with increasing complexity in decision-making processes which utilize large volumes of unstructured, semi-structured and/or structured data to satisfy multiple, interrelated information needs which contribute to an overall decision. Traditional decision support systems (DSSs) have been developed to address this need, but such systems are typically expensive to build, and are purpose-built for a particular decision-making scenario, making them difficult to extend or adapt to new decision scenarios. In this paper, we propose a novel decision representation which allows decision makers to formulate and organize natural language questions or assertions into an analytic hierarchy, which can be evaluated as part of an ad hoc decision process or as a documented, repeatable analytic process. We then introduce a new decision support framework, QUADS, which takes advantage of automatic question answering (QA) technologies to automatically understand and process a decision representation, producing a final decision by gathering and weighting answers to individual questions using a Bayesian learning and inference process. An open source framework implementation is presented and applied to two real world applications: target validation, a fundamental decision-making task for the pharmaceutical industry, and product recommendation from review texts, an everyday decision-making situation faced by on-line consumers. In both applications, we implemented and compared a number of decision synthesis algorithms, and present experimental results which demonstrate the performance of the QUADS approach versus other baseline approaches.

We describe a general result ranking approach for multi-phase, multi strategy information systems, which has been applied to the task of question answering (QA). Many information systems incorporate multiple steps and each step or phase may incorporate multiple component algorithms to achieve acceptable robustness and overall performance. Such systems may produce and rank a large number of candidate results. Prior work includes many models that rank a particular type of information object (e.g. a retrieved document, a factoid answer) using features specific to that information type, without attempting to make use of other non-local features (e.g. features of the upstream information source). We propose an approach that allows each phase in a system to leverage information propagated from preceding phases to inform the ranking decision. This is accomplished by a system object graph which represents all of the objects created during system execution, object dependencies (e.g. provenance), and ranking feature values extracted for a specific object. We evaluate the effectiveness of the proposed ranking approach in a multi-phase question answering system built by recombining pre-existing software modules. Experimental results show that our proposed approach significantly outperforms comparable answer ranking models.

Software frameworks which support integration and scaling of text analysis algorithms make it possible to build complex, high performance information systems for information extraction, information retrieval, and question answering; IBM's Watson is a prominent example. As the complexity and scaling of information systems become ever greater, it is much more challenging to effectively and efficiently determine which toolkits, algorithms, knowledge bases or other resources should be integrated into an information system in order to achieve a desired or optimal level of performance on a given task. This paper presents a formal representation of the space of possible system configurations, given a set of information processing components and their parameters (configuration space) and discusses algorithmic approaches to determine the optimal configuration within a given configuration space (configuration space exploration or CSE). We introduce the CSE framework, an extension to the UIMA framework which provides a general distributed solution for building and exploring configuration spaces for information systems. The CSE framework was used to implement biomedical information systems in case studies involving over a trillion different configuration combinations of components and parameter values operating on question answering tasks from the TREC Genomics. The framework automatically and efficiently evaluated different system configurations, and identified configurations that achieved better results than prior published results.

Different parts of an instance may be strong or weak indicators of the instance's label. We propose a new annotation strategy, where in addition to an instance's label, the annotator indicates parts of the instance that are rationales for its label. For two text classification tasks, we show that rationales provide a significant improvement in performance. Each instance (with or without rationales) may provide different incremental value to the learning algorithm. Annotation cost may also vary across instances and annotation strategies. We propose a cost-sensitive active learning approach for joint selection of instance and strategy that automatically determines which instances to query and whether to ask for rationales. We show that the proposed approach outperforms instance selection with a fixed strategy. While the additional cost for rationales may vary across annotators, user interface design, task, etc. We show that the proposed approach is able to select the right annotation strategy for each scenario.

A source expansion algorithm automatically extends a given text corpus with related content from large external sources such as the Web. The expanded corpus is not intended for human consumption but can be used in question answering (QA) and other information retrieval or extraction tasks to find more relevant information and supporting evidence. We propose an algorithm that extends a corpus of seed documents with web content, using a statistical model to select text passages that are both relevant to the topics of the seeds and complement existing information.

This work presents a general rank-learning framework for passage ranking within Question Answering (QA) systems using linguistic and semantic features. The framework enables query-time checking of complex linguistic and semantic constraints over keywords. Constraints are composed of a mixture of keyword and named entity features, as well as features derived from semantic role labeling. The framework supports the checking of constraints of arbitrary length relating any number of keywords. We show that a trained ranking model using this rich feature set achieves greater than a 20% improvement in Mean Average Precision over baseline keyword retrieval models. We also show that constraints based on semantic role labeling features are particularly effective for passage retrieval; when they can be leveraged, an 40% improvement in MAP over the baseline can be realized.

This article presents two probabilistic models for answering ranking in the multilingual question-answering (QA) task, which finds exact answers to a natural language question written in different languages. Although some probabilistic methods have been utilized in traditional monolingual answer-ranking, limited prior research has been conducted for answer-ranking in multilingual question-answering with formal methods. This article first describes a probabilistic model that predicts the probabilities of correctness for individual answers in an independent way. It then proposes a novel probabilistic method to jointly predict the correctness of answers by considering both the correctness of individual answers as well as their correlations. As far as we know, this is the first probabilistic framework that proposes to model the correctness and correlation of answer candidates in multilingual question-answering and provide a novel approach to design a flexible and extensible system architecture for answer selection in multilingual QA. An extensive set of experiments were conducted to show the effectiveness of the proposed probabilistic methods in English-to-Chinese and English-to-Japanese cross-lingual QA, as well as English, Chinese, and Japanese monolingual QA using TREC and NTCIR questions.

Bag-of-words retrieval is popular among Question Answering (QA) system developers, but it does not support constraint checking and ranking on the linguistic and semantic information of interest to the QA system. We present anapproach to retrieval for QA, applying structured retrieval techniques to the types of text annotations that QA systems use. We demonstrate that the structured approach can retrieve more relevant results, more highly ranked, compared with bag-of-words, on a sentence retrieval task. We also characterize the extent to which structured retrieval effectiveness depends on the quality of the annotations.

Graphical models have been applied to various information retrieval and natural language processing tasks in the recent literature. In this paper, we apply a probabilistic graphical model for answer ranking in question answering. This model estimates the joint probability of correctness of all answer candidates, from which the probability of correctness of an individual candidate can be inferred. The joint prediction model can estimate both the correctness of individual answers as well as their correlations, which enables a list of accurate and comprehensive answers. This model was compared with a logistic regression model which directly estimates the probability of correctness of each individual answer candidate. An extensive set of empirical results based on TREC questions demonstrates the effectiveness of the joint model for answer ranking. Furthermore, we combine the joint model with the logistic regression model to improve the efficiency and accuracy of answer ranking.

This paper describes a prototype system which captures semantic knowledge from domain text using controlled language. The KANTOO system is used to analyze input sentences from college-level science textbooks, producing sentence-level meaning representations (interlingua). The interlingua expressions are mapped into F-logic statements, which are be stored in a separate knowledge base to support reasoning in the domain.

We describe new consumer services based on speech processing technologies to support a new digital/mobile era of ubiquitous communication. First, we propose, a compact and noise robust embedded speech recognition middleware implemented on microprocessors focused on sophisticated HMIs (human machine interfaces) for car information systems (i.e. car telematics). Second, we report on a novel and sophisticated dialog management/manager (DM) system, based on VoiceXML (voice extensible markup language), called CAMMIA (conversational agent for multimedia mobile information access). The proposed DM handles two important issues: an automatic generation scheme for lexicons and grammars, and an effective combination/merger between automatic speech recognition (ASR) and natural language processing (NLP). The new DM scheme has been evaluated for an application of the car telematics service task after integration with ASR and a VoiceXML interpreter (VXI).

Speech recognition errors are inevitable in a speech dialog system. It is important to provide a dialog flow that allows the user to correct system errors and quickly return to the original dialog. This paper describes explicit, final and implicit confirmation styles that are implemented in the CAMMIA speech dialog system, and compares them from the viewpoint of usability. Our results show that a final confirmation with fewer confirmation turns is preferred by the user when there is no error in the dialog. On the other hand, an explicit confirmation is preferred when an error occurs.

This paper presents a source language diagnostic system for controlled translation. Diagnostics were designed and implemented to address the most difficult rewrites for authors, based on an empirical analysis of log files containing over 180,000 sentences. The design and implementation of the diagnostic system are presented, along with experimental results from an empirical evaluation of the completed system. We found that the diagnostic system can correctly identify the problem in 90.2% of the cases. In addition, depending on the type of grammar problem, the diagnostic system may offer a rewritten sentence. We found that 89.4% of the rewritten sentences were correctly rewritten. The results suggest that these methods could be used as the basis for an automatic rewriting system in the future.

We describe the automatic resolution of pronominal anaphora using KANT Controlled English (KCE) and the KANTOO English-to-Spanish MT system. Our algorithm is based on a robust, syntax-based approach that applies a set of restrictions and preferences to select the correct antecedent. We report a success rate of 89.6% on a training corpus with 289 anaphors, and 87.5% on held-out data containing 145 anaphors. Resolution of anaphors is important in translation, due to gender mismatches among languages; our approach translates anaphors to Spanish with 97.2% accuracy.

Large-scale lexical knowledge acquisition is one of the most time critical steps in developing a knowledge-based machine translation system. In particular, developing the syntactic lexicon for the target language can be an unwieldy task, as on-line knowledge assets are likely to be more scarce than for the source language. This paper addresses this problem within the KANT machine translation system and describes how we structure the KA process to address this problem. This was done by first determining the nature of the desired process and then developing tools to implement that process. The tools themselves and the ways in which the helped us to realize our design goals are described. We conclude that, while the problem of lexical acquisition can be formidable, it can be overcome with proper foresight and tool design.>