Marko Sterbentz, Cameron Barrie, Shubham Shahi, Abhratanu Dutta, Donna Hooshmand, Harper Pack, Kristian J Hammond. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

The emergence of contract specific programming languages has struggled to translate into widespread adoption of computable contracts due largely to high conversion costs. In this work, we present the first system for converting natural language contracts into code through the extraction of key entities, relationships, and formulas into a graph representation called the Obligation Logic Graph (OLG). This approach allows the semantic meaning of contract obligations, including dependencies between obligations, to be captured through the OLG and mapped to code downstream. We also introduce OLG extraction as a new joint entity and relation prediction task for legal contracts, and present the Contract-OLG dataset, consisting of 1,876 contract provisions, 18,597 entities and 18,170 relationships. We perform detailed experiments to understand the capabilities of state-of-the-art Transformer and graph-based models at completing these tasks, and identify where there is currently a significant gap between human expert and machine performance, particularly for relation extraction.

The U.S. court system is the nation's arbiter of justice, tasked with the responsibility of ensuring equal protection under the law. But hurdles to information access obscure the inner workings of the system, preventing stakeholders - from legal scholars to journalists and members of the public - from understanding the state of justice in America at scale. There is an ongoing data access argument here: U.S. court records are public data and should be freely available. But open data arguments represent a half-measure; what we really need is open information. This distinction marks the difference between downloading a zip file containing a quarter-million case dockets and getting the real-time answer to a question like "Are pro se parties more or less likely to receive fee waivers?" To help bridge that gap, we introduce a novel platform and user experience that provides users with the tools necessary to explore data and drive analysis via natural language statements. Our approach leverages an ontology configuration that adds domain-relevant data semantics to database schemas to provide support for user guidance and for search and analysis without user-entered code or SQL. The system is embodied in a "natural-language notebook" user experience, and we apply this approach to the space of case docket data from the U.S. federal court system. Additionally, we provide detail on the collection, ingestion and processing of the dockets themselves, including early experiments in the use of language modeling for docket entry classification with an initial focus on motions.

The Web makes it possible for news readers to learn more about virtually any story that interests them. Media outlets and search engines typically augment their information with links to similar stories. It is up to the user to determine what new information is added by them, if any. In this paper we present Tell Me More, a system that performs this task automatically: given a seed news story, it mines the web for similar stories reported by different sources and selects snippets of text from those stories which offer new information beyond the seed story. New content may be classified as supplying: additional quotes, additional actors, additional figures and additional information depending on the criteria used to select it. In this paper we describe how the system identifies new and informative content with respect to a news story. We also how that providing an explicit categorization of new information is more useful than a binary classification (new/not-new). Lastly, we show encouraging results from a preliminary evaluation of the system that validates our approach and encourages further study.

A useful approach for enabling computers to automatically create new content is utilizing the text, media, and information already present on the World Wide Web. The newly created content is known as "machine-generated content". For example, a machine-generated content system may create a multimedia news show with two animated anchors presenting a news story; one anchor reads the news story with text taken from an existing news article, and the other anchor regularly interrupts with his or her own opinion about the story. In this paper, we present such a system, and describe in detail its strategy for autonomously extracting and selecting the opinions given by the second anchor.

In this paper we describe an automated system, and its attendant set of techniques and tools, that is able to generate novel multimedia experiences. Using existing online sources, external textual and multimedia repositories, and user preferences, the system builds a customized audio/visual experience for the user. We discuss one application in detail: News at Seven, an automatically generated, personalized news show. Beginning with a set of user preferences, the system is able to find relevant text, process that text, and supplement it with images, video, and blogger responses. The final output of the system is an online Flash presentation that uses animated avatars with generated speech and is modeled after traditional nightly news broadcast. We see this work as the beginning of an overall approach to machine-generated content.

We propose a machine learning system that learns to choose human gestures to accompany novel text. The system is trained on scripts comprised of speech and animations that were hand-coded by professional animators and shipped in video games. We treat this as a text-classification problem, classifying speech as corresponding with specific classes of gestures. We have built and tested two separate classifiers. The first is trained simply on the frequencies of different animations in the corpus. The second extracts text features from each script, and maps these features to the gestures that accompany the script. We have experimented with using a number of features of the text, including n-grams, emotional valence of the text, and parts-of-speech. Using a naïve Bayes classifier, the system learns to associate these features with appropriate classes of gestures. Once trained, the system can be given novel text for which it will attempt to assign appropriate gestures. We examine the performance of the two classifiers by using n-fold cross-validation over our training data, as well as two user studies of subjective evaluation of the results. Although there are many possible applications of automated gesture assignment, we hope to apply this technique to a system that produces an automated news show.

This paper describes a digital theater installation called Buzz. Buzz consists of virtual actors who express the collective voice generated by weblogs (blogs). These actors find compelling stories from blogs and perform them. In this paper, we explore what it means for a story to be compelling and describe a set of techniques for retrieving compelling stories. We also outline an architecture for high level direction of a performance using Adaptive Retrieval Charts (ARCs), allowing a director-level of interaction with the performance system. Our overall goal in this work is to build a model of human behavior on a new foundation of query formation, information retrieval and filtering.

In this paper, we describe MusicStory, a system that automatically creates videos to accompany music with lyrics. MusicStory uses common search engines, photo-sharing websites, and simple analysis of the dynamics and tempo of the music to create personalized photo-narratives. Video pacing and content is based on the content of the song and structure of the image repositories selected. The image associations MusicStory presents amplify the emotional experience by externalizing the imagery in song lyrics with the content found within a social network. The resulting work juxtaposes the meanings inherent in the social network with those in the song.

In this article, we describe the intrinsic constraints of mobility and discuss how we can work around and often exploit these constraints using information implied by the context of the mobile user. In particular, we outline some of the work we have been doing in providing information to users on the basis of "Low-Fi" location information retrieved from user's cellular phone. A user's location and other contextual information is used to retrieve task relevant information and avoid many of the problems posed by limits of mobile devices and their use. This is part of an ongoing effort to build "smart" navigational interfaces bound on information about a user's location.

In this article, we present the Association Engine, a multimedia installation that explores the space of language and exposes connections between words by performing a series of improvisational games. It stands out from previous efforts in the realm of computers and theater as the agency has been empowered with a cultural understanding, more specifically, it uses the Web as a source of cultural reality.

In this article, we explore the structure of the web as an indicator of popular culture and its use in multi-media exhibits. In a series of art and technology installations, the software agency needs to keep 'grounded' to what people can readily understand. We administered a survey to understand how people perceived word and phrase obscurity related with frequency information gathered from a popular Web search engine. We found the frequency data gathered from the engine closely matched judgments gathered from people. The results of this study point to the new applications of the WWW in art and multi-media exhibits as an indicator of popular culture.

The work presented in this paper takes a novel approach to the task of providing information to viewers of broadcast news. Instead of considering the broadcast news as the end product, this work uses it as a starting point to dynamically build an information space for the user to explore. This information space is designed to satisfy the users information needs, by containing more breadth, depth, and points of view than the original broadcast story. The architecture and current implementation are discussed, and preliminary results from the analysis of some its components are presented

This research discusses a method for delivering just-in-time information to television viewers to provide more depth and more breadth to television broadcasts. A novel aspect of this research is that it uses broadcast news as a starting point for gathering information regarding specific stories, as opposed to considering the broadcast version to be the end of the viewers exploration. This work is implemented in Cronkite, a system that provides viewers with expanded coverage of broadcast news stories.

This article describes the implementation of a system that 'imagines' while a movie is being played by finding associations in the movie's content and presenting them to the viewer. This related information, in the form of images and movie clips, helps enhance the viewer's experience in a new immerse environment.

Authors cite other work in many types of documents. Notable among these are research papers and web pages. Recently, several researchers have proposed using the text surrounding citations (references) as a means of automatically indexing documents for search engines, claiming that this technique is superior to indexing documents based on their content [1,2]. While we ourselves have made this claim, we acknowledge that little empirical data has been presented to support it. Therefore, in the limited space available we present a terse overview of a study comparing reference to content as bases for indexing documents. This study indicates that reference identifies the value of documents more accurately and with a greater diversity of language than content.

In this paper, we describe the Interactive Chef (IChef), an intelligent system that attempts to emulate a human cooking assistant. The user navigates through a recipe using voice, while IChef reads aloud each step. IChef is designed to be aware of the present context and exploit this knowledge by answering questions about ingredients or techniques posed by the user in natural spoken language.

Flytrap is a group music environment that knows its users' musical tastes and can automatically construct a soundtrack that tries to please everyone in the room. The system works by paying attention to what music people listen to on their computers. Users of the system have radio frequency ID badges that let the system know when they are nearby. Using the preference information it has gathered from watching its users, and knowledge of how genres of music interrelate, how artists have influenced each other, and what kinds of transitions between songs people tend to make, the 'virtual DJ' finds a compromise and chooses a song. The system tries to satisfy the tastes of people in the room, but it also makes a playlist that fits its own notion of what should come next. Once it has chosen a song, music is automatically broadcast over the network and played on the closest machine.

mpME! introduces users to new music and interweaves old music the listener has heard of before and enjoys. mpME! exposes listeners to new music by using existing music information on the Internet such as musical artist directories. mpME! also provides complementary information about the artist to educate the user throughout the listening experience. MpME! uses a feature intersection algorithm to make recommendations for artists using features of artists found on the web.

The ability to reason about the activity of a user is crucial to the implementation of any Intelligent User Interface. If it is able to recognize what a user is doing, a computer can act to cooperate. Most computer systems limit themselves to command-response interactions-their trivial understandings of their users cannot support a more complicated interaction. However, by looking at the tasks that their users are performing and reasoning about sequences of actions, a computer system can provide a more interesting level of interaction that is more efficient and does not demand as much of its users. Furthermore, the understanding of the user's activity provides a context within which to better understand future actions and to tune the sensing systems to look and listen for the actions that the user is most likely to take next. Finally, in many domains, such computer systems can recognize user tasks and act to cooperate without requiring a deep, goal-oriented understanding. In this paper, we look at the process-based interface used in the Intelligent Classroom, focusing on how a human lecturer can control it by simply going about her presentation. Also, we look at how the general ideas have been adapted to Jabberwocky, a speech-based interface to Microsoft PowerPoint that automatically switches slides, and how they are being applied to extend the functionality of Watson, an autonomous web research tool that uses the document a user is viewing as a search context.

Contents of documents can be characterized not just by subject matter or topic, but also in terms of more thematic or even stylistic attributes, such as level of detail, emotional vs. unemotional presentation, personal vs. impersonal voice, positive or negative attitude or "spin," theoretical vs. practical presentation, etc. As implied by these examples, such attributes often form continuous dimensions along which documents can be distinguished. In the best case, the appropriate place on these dimensions -- i.e., which documents will be of most value to the user at the current time -- can be determined by the user's current task, prior knowledge, etc. Does the user need more detail, or less, for example? In the current state of the art, it is difficult to compute this automatically. Instead, we propose that users should be enabled view these dimensions using themometers and to navigate them using themostats -- controls that offer users a choice of documents on the same topic, but with differing levels of detail, positive or negative spin, personal or impersonal voice, etc. The dimensions along which we can provide this kind of control must be easily computable. We have been able to characterize documents with respect to a number of these dimensions using simple statistical measures as well as specialized dictionaries, and describe these methods herein.

Java Settlers is an environment for doing research in the area of multi-agent negotiation. The vehicle for this research is a Web-deployed java program for playing the popular German board game, Settlers of Catan.

We describe ongoing work on I2I, a system aimed at fostering opportunistic communication among users viewing or manipulating content on the Web and in productivity applications. Unlike previous work in which the URLs of Web resources are used to group users visiting the same resource, we present a more general framework for clustering work contexts to group users together that accounts for dynamic content and distributional properties of Web accesses which can limit the utility URL based systems. In addition, we describe a method for scaffolding asynchronous communication in the context of an ongoing task that takes into account the ephemeral nature of the location of content on the Web. The techniques we describe also nicely cover local files in progress, in addition to publicly available Web content. We present the results of several evaluations that indicate systems that use the techniques we employ may be more useful than systems that are strictly URL based.

While recent work has focused on providing tools and infrastructure for users to access electronic information over the Internet, the relationship between the physical world and information available online has been relatively unexplored. Information about a user's location, and the objects she interacts with, can be sufficient to recognize enough of the user's task to drive retrieval of online information relevant to the task at hand. The XLibris system automatically retrieves, aggregates, and delivers information about books to users as they are checked out of the library, using information about the books themselves and the user's task. XLibris locates books in the Dewey Decimal subject hierarchy to automatically search for the most relevant information about the book for the user, tailoring both the sources queried and the information returned based on the book's position in the hierarchy.

Many case-based reasoning systems have been developed to aid designers by providing them with libraries of prior design experiences. Traditionally, these systems are implemented as stand-alone ``external memories'' for the designer to query manually. This paper presents a contrasting approach that integrates proactive case retrieval into the designer's normal task processes, automatically tailoring information selection and presentation emphasis to fit changing designer needs and attention flow. The paper presents a set of principles for this integrated intelligent design support and describes their application in the Stamping Advisor, a system to support design feasibility analysis for sheet metal automotive parts. The Stamping Advisor interface proactively provides designers with relevant information to support feasibility analysis, automatically prepares their information products, and unobtrusively gathers the information needed to generate new cases to improve the quality of future support.

We describe Rosetta, a digital library system for scientific literature. Rosetta makes it easy for people to find the information for which they are looking even when using short, imprecise queries. Rosetta indexes research articles based on the way they have been described when cited in other documents. The concise descriptions that occur in citations are similar to the short queries people typically form when searching; therefore, citations make a better basis for indexing than do the words used within a research article itself. Using this indexing technique we are able to provide a user interface that presents users with an automatically generated directory of the information space surrounding a query. Our objective with this interface is to present people with the information for which they have asked as well as the information for which they may have intended to ask.

Our central claim is that user interactions with everyday productivity applications (e.g., word processors, Web browsers, etc.) provide rich contextual information that can be leveraged to support just-in-time access to task-relevant information. We discuss the requirements for such systems, and develop a general architecture for systems of this type. As evidence for our claim, we present Watson, a system which gathers contextual information in the form of the text of the document the user is manipulating in order to proactively retrieve documents from distributed information repositories. We close by describing the results of several experiments with Watson, which show it consistently provides useful information to its users.

In this paper we discuss our use of multi-modal input to improve human computer interaction. Specifically we look at the methods used in the Intelligent Classroom to combine multiple input modes, and examine in particular the visual input modes. The Classroom provides context that improves the functioning of the visual input modes. It also determines which visual input modes are needed when. We examine a number of visual input modes to see how they fit into the general scheme, and look at how the Classroom controls their operation.

In designing Jabberwocky—a speech-based interface to Microsoft PowerPoint—we have tried to go beyond simple commands like “Next slide, please” and make a tool that aids speakers as they present and even learns as they rehearse their presentations. Jabberwocky looks at the contents of the slides, extracting key words and phrases and associating them with their places in the presentation. By listening for these phrases (and synonymous phrases derived using syntactic rules) Jabberwocky is able to follow along with the presentation, switching slides at the appropriate moments. In this paper, we discuss the implementation of this system—a component of our Intelligent Classroom project—and look at how we are using it.

Although a user's navigation history contains a lot of hidden information about the relationship between web pages and between users, this information is usually not exploited. The information hidden in the history can be an invaluable source of knowledge in assisting a user to better surf the Web. We presented a system which actively monitors and tracks a user's navigation. Once a user's navigation history is captured, we apply data mining techniques to discover the hidden knowledge contained in the history. The knowledge is then used to suggest potentially interesting web pages to users.

No abstract available.