Adam Yaari, Jan DeWitt, Henry Hu, Bennett Stankovits, Sue Felshin, Yevgeni Berzak, Helena Aparicio, Boris Katz, Ignacio Cases, Andrei Barbu. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

A robot’s ability to understand or ground natural language instructions is fundamentally tied to its knowledge about the surrounding world. We present an approach to grounding natural language utterances in the context of factual information gathered through natural-language interactions and past visual observations. A probabilistic model estimates, from a natural language utterance, the objects, relations, and actions that the utterance refers to, the objectives for future robotic actions it implies, and generates a plan to execute those actions while updating a state representation to include newly acquired knowledge from the visual-linguistic context. Grounding a command necessitates a representation for past observations and interactions; however, maintaining the full context consisting of all possible observed objects, attributes, spatial relations, actions, etc., over time is intractable. Instead, our model, Temporal Grounding Graphs, maintains a learned state representation for a belief over factual groundings, those derived from natural-language interactions, and lazily infers new groundings from visual observations using the context implied by the utterance. This work significantly expands the range of language that a robot can understand by incorporating factual knowledge and observations of its workspace into its inference about the meaning and grounding of natural-language utterances.

A natural language interface to answers on the Web can help us access information more efficiently.We start with an interesting source of information-infoboxes in Wikipedia that summarize factoid knowledge-and develop a comprehensive approach to answering questions with high precision.We first build a system to access data in infoboxes in a structured manner.We use our system to construct a crowdsourced dataset of over 15,000 highquality, diverse questions.With these questions, we train a convolutional neural network model that outperforms models that achieve top results in similar answer selection tasks.