The primary goal of the Midas project is to build a system that enables easy and scalable integration of unstructured and semi-structured information present across multiple data sources. As a first step in this direction, we have built a system that extracts and integrates information from regulatory filings submitted to the U.S. Securities and Exchange Commission (SEC) and the Federal Deposit Insurance Corporation (FDIC). Midas creates a repository of entities, events, and relationships by extracting, conceptualizing, integrating, and aggregating data from unstructured and semi-structured documents. This repository enables applications to use the extracted and integrated data in a variety of ways including mashups with other public data and complex risk analysis.

The ever increasing amount of content on the Internet has fostered many efforts seeking to leverage this potentially yottascale information source. Service systems using advanced data and text analytics techniques have been developed to perform knowledge gathering and information discovery over Web data. Information gathered from free and public sources on the Web is frequently integrated with enterprise and proprietary data to create sophisticated service systems able to provide insight in an increasing number of business critical areas. Unfortunately, for fixed and or limited resource projects, consistent and reliable ingestion and integration of content often dominates the effort, reducing the time available for developing core analytics and presentations that differentiate and define an information service. If this initial data extraction, translation and loading of information (known as ETL in the database world) can be abstracted for these web sources, it would provide an important core technology on which Web-based information services could be more rapidly and inexpensively developed and deployed. This paper presents such a system - MONGOOSE - an approach that seeks to reduce the time spent creating a reliable data ingest and integration system and thus reducing the time-to-impact of advanced analytics service solutions.

Many medium and large companies maintain internal employee directories. Unfortunately, most directories only allow the lookup of individual profiles, one profile at a time. Team Analytics is a novel application that integrates information from disparate enterprise tools for groups of people. Besides accelerating the lookup process, Team Analytics also displays information that is only available in the group context, such as an organizational chart and time zone awareness. We present the Team Analytics application, its integration with our corporate email client, and results from a user survey that evaluates various aspects of the application.

Data deduplication is a popular dictionary based compression method in storage archival and backup. The deduplication efficiency improves for smaller chunk sizes, however the files become highly fragmented requiring many disk accesses during reconstruction or chattiness in a client-server architecture. Within the sequence of chunks that an object (file) is decomposed into, sub-sequences of adjacent chunks tend to repeat. We exploit this insight to optimize the chunk sizes by joining repeated sub-sequences of small chunks into new super chunks with the constraint to achieve practically the same matching performance. We employ suffix arrays to find these repeating sub-sequences and to determine a new encoding that covers the original sequence. With super chunks we significantly reduce fragmentation, improving reconstruction time and the overall deduplication ratio by lowering the amount of metadata. As a result, fewer chunks are used to represent a file, reducing the number of disk accesses needed to reconstruct the file and requiring fewer entries in the chunk dictionary and fewer hashes to encode a file. To encode a sequence of chunks we proved two facts: (1) any subsequence that repeats is part of some super chunk (su- permaximal in Figure 1) - therefore the dictionary contains just supermaximals and non-repeats, and (2) maximals are the only repeats not covered (overlapped) by the containing supermaximals - so once we discover the maximals with the suffix array, and encode them, there is no need for maintaining auxiliary data structures like bit masks, to guarantee the coverage (encoding) of the entire object. Our experimental evaluation (Figure 2) yields a reduction in fragmentation between 89%-97% and a reduction of dictionary metadata (number of entries) between 80%-97%, without increasing the total amount of storage required for unique chunks.

There are a large number of Web sites serving valuable content that can be used by higher-level applications, Web services, Mashups, etc. Yet, due to various reasons (lack of computing resources, financial constraints etc.) they are unable to provide Web service APIs to access their data. In their desire to incorporate the latest and greatest technologies, as well as to adapt layouts that are more preferred by users, Web sites undergo changes over time. These changes can range from minor, e.g. function name changes, to major, e.g., shifting the Web platform to AJAX technologies. This paper addresses the problem of detecting layout changes for Web sites which are unable to provide any Web service to access their content, yet do not mind others harvesting said content.

Popularity based search engines have served to stagnate information retrieval from the web. Developed to deal with the very real problem of degrading quality within keyword based search they have had the unintended side effect of creating "icebergs" around topics, where only a small minority of the information is above the popularity water-line. This problem is especially pronounced with emerging information--new sites are often hidden until they become popular enough to be considered above the water-line. In domains new to a user this is often helpful--they can focus on popular sites first. Unfortunately it is not the best tool for a professional seeking to keep up-to-date with a topic as it emerges and evolves.We present a tool focused on this audience--a system that addresses the very large scale information gathering, filtering and routing, and presentation problems associated with creating a useful incremental stream of information from the web as a whole. Utilizing the WebFountain platform as the primary data engine and Really Simple Syndication (RSS) as the delivery mechanism, our "Daily Deltas" (Delta) application is able to provide an informative feed of relevant content directly to a user. Individuals receive a personalized, incremental feed of pages related to their topic allowing them to track their interests independent of the overall popularity of the topic.

XML provides a universal and portable format for document and data exchange. While the syntax and specification of XML makes documents both human readable and machine parsable, it is often at the expense of efficiency when representing simple data structures.We investigate the ``costs'' associated with XML serialization from several resource perspectives: storage, transport, processing and human readability. These experiments are done within the context of a large text-centric service oriented architecture -- IBM's WebFountain project.We find that for several applications, human readable formats outperform binary equivalents, especially in the area of data size, and that the costs of processing encoded binary data often exceeds that of processing terse human readable formats.

Speech is a tantalizing mode of human communication. On the one hand, humans understand speech with ease and use speech to express complex ideas, information, and knowledge. On the other hand, automatic speech recognition with computers is still very hard, and extracting knowledge from speech is even harder. In this paper we motivate the study of speech as a knowledge resource and briefly survey a family of related applications and systems being developed at IBM Research aimed towards the goal of exploiting speech as a knowledge resource.