The concept of surprise is central to human learning and development. However, compared to accuracy, surprise has received little attention in the IR community, yet it is an essential component of the information seeking process. This workshop brings together researchers and practitioners of IR to discuss the topic of computational surprise, to set a research agenda, and to examine how to build datasets for research into this fascinating topic. The themes in this workshop include discussion of what can be learned from some well-known surprise models in other fields, such as Bayesian surprise; how to evaluate surprise based on user experience; and how computational surprise is related to the newly emerging areas, such as fake news detection, computational contradiction, clickbait detection, etc.

We present a novel interactive framework for patent retrieval leveraging distributed representations of concepts and entities extracted from the patents text. We propose a simple and practical interactive relevance feedback mechanism where the user is asked to annotate relevant/irrelevant results from the top n hits. We then utilize this feedback for query reformulation and term weighting where weights are assigned based on how good each term is at discriminating the relevant vs. irrelevant candidates. First, we demonstrate the efficacy of the distributed representations on the CLEF-IP 2010 dataset where we achieve significant improvement of 4.6% in recall over the keyword search baseline. Second, we simulate interactivity to demonstrate the efficacy of our interactive term weighting mechanism. Simulation results show that we can achieve significant improvement in recall from one interaction iteration outperforming previous semantic and interactive patent retrieval methods.

Explicit concept space models have proven efficacy for text representation in many natural language and text mining applications. The idea is to embed textual structures into a semantic space of concepts which captures the main topics of these structures. Despite their wide applicability, existing models have many shortcomings such as sparsity and being restricted to Wikipedia as the main knowledge source from which concepts are extracted. In this paper we highlight some of these limitations. We also describe Mined Semantic Analysis (MSA); a novel concept space model which employs unsupervised learning in order to uncover implicit relations between concepts. MSA leverages the discovered concept-concept associations to enrich the semantic representations. We evaluate MSA’s performance on benchmark data sets for measuring lexical semantic relatedness. Empirical results show superior performance of MSA compared to prior state-of-the-art methods.

Online job boards are one of the central components of modern recruitment industry. With millions of candidates browsing through job postings everyday, the need for accurate, effective, meaningful, and transparent job recommendations is apparent more than ever. While recommendation systems are successfully advancing in variety of online domains by creating social and commercial value, the job recommendation domain is less explored. Existing systems are mostly focused on content analysis of resumes and job descriptions, relying heavily on the accuracy and coverage of the semantic analysis and modeling of the content in which case, they end up usually suffering from rigidity and the lack of implicit semantic relations that are uncovered from users' behavior and could be captured by Collaborative Filtering (CF) methods. Few works which utilize CF do not address the scalability challenges of real-world systems and the problem of cold-start. In this paper, we propose a scalable item-based recommendation system for online job recommendations. Our approach overcomes the major challenges of sparsity and scalability by leveraging a directed graph of jobs connected by multi-edges representing various behavioral and contextual similarity signals. The short lived nature of the items (jobs) in the system and the rapid rate in which new users and jobs enter the system make the cold-start a serious problem hindering CF methods. We address this problem by harnessing the power of deep learning in addition to user behavior to serve hybrid recommendations. Our technique has been leveraged by CareerBuilder.com which is one of the largest job boards in the world to generate high-quality recommendations for millions of users.

Mined Semantic Analysis (MSA) is a novel concept space model which employs unsupervised learning to generate semantic representations of text. MSA represents textual structures (terms, phrases, documents) as a Bag of Concepts (BoC) where concepts are derived from concept rich encyclopedic corpora. Traditional concept space models exploit only target corpus content to construct the concept space. MSA, alternatively, uncovers implicit relations between concepts by mining for their associations (e.g., mining Wikipedia's “See also” link graph). We evaluate MSA's performance on benchmark datasets for measuring semantic relatedness of words and sentences. Empirical results show competitive performance of MSA compared to prior state-of-the-art methods. Additionally, we introduce the first analytical study to examine statistical significance of results reported by different semantic relatedness methods. Our study shows that, the nuances of results across top performing methods could be statistically insignificant. The study positions MSA as one of state-of-the-art methods for measuring semantic relatedness, besides the inherent interpretability and simplicity of the generated semantic representation.

In this demo we present a semantic framework for innovation and patent analytics powered by Mined Semantic Analysis (MSA). Our framework provides cognitive assistance to its users through a Web-based visual and interactive interface. First, we describe building a conceptual knowledge graph by mining user-generated encyclopedic textual corpus for semantic associations. Then, we demonstrate applying the acquired knowledge to support many cognition and knowledge based use cases for innovation analysis including technology exploration and landscaping, competitive analysis, literature and prior art search and others.

In this experiment, we measure the potential contribution of internet search to question answering. The task is to correctly answer Jeopardy! questions, and for doing so we use our existing question answering system, "Watsonsim", the architecture of which follows the original IBM Watson. We compare the answering precision and recall with and without internet search data from Microsoft's Bing. To accommodate the new sources we included several additional procedures, in particular merging candidate answers with similar meaning, and removing any candidate answers which are forbidden. The experiments show the potential of the internet search to improve question answering in terms of both the accuracy and recall: the top rank accuracy increased by around 14%, reaching 20% precision, compared to only 6% precision when using local search, and the recall improved by 20%, reaching 52%, up from 32% using only local search engines. Tweaks to better handle web results later raised top-rank accuracy an additional 21%, reaching 41% precision, and raised recall by another 19%, to reach 71% recall. The overall conclusion of our experiments was to show that in question answering, a network enabled system can compensate for a smaller corpus and provide reasonably good answers.

Our paper is actually two contributions in one. First, we argue that IBM's Jeopardy! playing machine needs a formal semantics. We present several arguments as we discuss the system. We also situate the work in the broader context of contemporary AI. Our second point is that the work in this area might well be done as a broad collaborative project. Hence our "Blue Sky'' contribution is a proposal to organize a polymath-style effort aimed at developing formal tools for the study of state of the art question-answer systems, and other large scale NLP efforts whose architectures and algorithms lack a theoretical foundation.

IBM Watson exemplifies multiple innovations in natural language processing and question answering. In addition, Watson uses most of the known techniques in these two domains as well as many methods from related domains. Hence, there is pedagogical value in a rigorous understanding of its function. The paper provides the description of a text analytics course focused on building a simulator of IBM Watson, conducted in Spring 2014 at UNC Charlotte. We believe this is the first time a simulation containing all the major Watson components was created in a university classroom. The system achieved a respectable (close to) 20% accuracy on Jeopardy! questions, and there remain many known and new avenues of improving performance that can be explored in the future. The code and documentation are available on GitHub. The paper is a joint effort of the teacher and some of the students who were leading teams implementing component technologies, and therefore deeply involved in making the class successful.

In this paper we propose a novel technique for dimensionality reduction using freely available online knowledge bases. The complexity of our method is linearly proportional to the size of the full feature set, making it applicable efficiently to huge and complex datasets. We demonstrate this approach by investigating its effectiveness on patent data, the largest free technical text. We report empirical results on classification of the CLEF-IP 2010 dataset using bigram features supported by mentions in Wikipedia, Wiktionary, and GoogleBooks knowledge bases. We achieve a 13-fold reduction in number of bigrams features and a 1.7% increase in classification accuracy over the unigrams baseline. These results give concrete evidence that significant accuracy improvements and massive reduction in dimensionality could be achieved using our approach, hence help alleviating the tradeoff between task complexity and accuracy.

A source expansion algorithm automatically extends a given text corpus with related content from large external sources such as the Web. The expanded corpus is not intended for human consumption but can be used in question answering (QA) and other information retrieval or extraction tasks to find more relevant information and supporting evidence. We propose an algorithm that extends a corpus of seed documents with web content, using a statistical model to select text passages that are both relevant to the topics of the seeds and complement existing information.

This paper addresses the important problem of finding relevant information in the context of a business process. It presents an information access solution called EIL (enterprise information leverage) which combines information extraction and semantic search to support information needs of professionals selling IT services. EIL leverages structured and unstructured data using novel architecture and special purpose algorithms. Our approach is to organize information around business activities (e.g. a sale), and the system supports semantic concept based information retrieval by utilizing both database query and document search where the relevant business activities act as a contextual constraint. We experimentally show that this approach is promising for reducing noise in search results. EIL is currently under pilot deployment in one of the IBM services sales units.

Article Free Access Share on Evolution of the conversation machine: a case study of bringing advanced technology to the marketplace Authors: Catherine G. Wolf IBM T.J. Watson Research Center, 30 Saw Mill River Road, Hawthorne, NY IBM T.J. Watson Research Center, 30 Saw Mill River Road, Hawthorne, NYView Profile , Wlodek Zadrozny IBM T.J. Watson Research Center, 30 Saw Mill River Road, Hawthorne, NY IBM T.J. Watson Research Center, 30 Saw Mill River Road, Hawthorne, NYView Profile Authors Info & Claims CHI '98: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsJanuary 1998 Pages 488–495https://doi.org/10.1145/274644.274710Published:01 January 1998Publication History 7citation337DownloadsMetricsTotal Citations7Total Downloads337Last 12 Months14Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

We address the problem of reasoning in cases when knowledge bases containing background knowledge are understood not as sets of formulas (rules and facts) but as collections of partially ordered theories. In our system, the usual, two‐part logical structures, consisting of a metalevel and an object level, are augmented by a third level–a referential level. The referential level is a partially ordered collection of theories; it encodes background knowledge. As usual, current situations are described on the object level, and the metalevel is a place for rules that can eliminate some of the models permitted by the object level and the referential level. As a logic of reasoning the system generalizes the standard model of a rational agent: deducing actions and deriving new information about the world from a logical theory–its knowledge base. It is a natural logical system in which priorities on the possible readings of predicates, not special rules of inference, are the main source of nonmonotonicity. We introduce a theory forming operator PT(x) to exploit the priorities, and we investigate its basic logical properties. Then we show how such a system can be augmented by metarules. Although this paper concentrates on basic logical properties of the new theory, this formalism has already been applied to model a number of natural language phenomena such as the notion of text coherence, Gricean maxims, vagueness, and a few others. The paper also briefly compares it with the model of background knowledge of CYC, as proposed by Lenat and Guha.

We show that certain very different types of tasks can be modelled by one rule of abduction which we call coh(2), informally described as being able to go from the conclusions to the premises if the premises cohere, i.e. there is some semantic link between them. These tasks might come from the areas of natural language understanding, planning, perception, etc. Furthermore, there is a family of very closely related rules of coherence coh(n) that can do in one step what otherwise could require iteration of coh(2). More specifically, we formalize proximity-based reasoning as a kind of abduction; we show how it can be captured by such rules; and although we first present these rules informally, we later prove that their meaning can be made precise, and that it is possible to establish some formal properties of them. Of course, this requires a formalism for abduction, which is also sketched in the paper. All this leads to some mathematical results to the effect that certain natural strategies for using rules of coherence are sound and some others are not. Finally, we list some open problems that might be solvable by this approach.