We address the task of visual semantic role labeling (vSRL), the identification of the participants of a situation or event in a visual scene, and their labeling with their semantic relations to the event or situation.We render candidate participants as image regions of objects, and train a model which learns to ground roles in the regions which depict the corresponding participant.Experimental results demonstrate that we can train a vSRL model without reliance on prohibitive image-based role annotations, by utilizing noisy data which we extract automatically from image captions using a linguistic SRL system.Furthermore, our model induces frame-semantic visual representations, and their comparison to previous work on supervised visual verb sense disambiguation yields overall better results.

Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content. Prediction also affects perception and might be a key to robustness in human language processing. In this paper, we investigate the factors that affect human prediction by building a computational model that can predict upcoming discourse referents based on linguistic knowledge alone vs. linguistic knowledge jointly with common-sense knowledge in the form of scripts. We find that script knowledge significantly improves model estimates of human predictions. In a second study, we test the highly controversial hypothesis that predictability influences referring expression type but do not find evidence for such an effect.

This paper describes the first robust approach to automatically labeling clauses with their situation entity type (Smith, 2003), capturing aspectual phenomena at the clause level which are relevant for interpreting both semantics at the clause level and discourse structure.Previous work on this task used a small data set from a limited domain, and relied mainly on words as features, an approach which is impractical in larger settings.We provide a new corpus of texts from 13 genres (40,000 clauses) annotated with situation entity types.We show that our sequence labeling approach using distributional information in the form of Brown clusters, as well as syntactic-semantic features targeted to the task, is robust across genres, reaching accuracies of up to 76%.

Annemarie Friedrich, Manfred Pinkal. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

This paper provides the first fully automatic approach for classifying clauses with respect to their aspectual properties as habitual, episodic or static.We bring together two strands of previous work, which address only the related tasks of the episodic-habitual and stative-dynamic distinctions, respectively.Our method combines different sources of information found to be useful for these tasks.We are the first to exhaustively classify all clauses of a text, achieving up to 80% accuracy (baseline 58%) for the three-way classification task, and up to 85% accuracy for related subtasks (baselines 50% and 60%), outperforming previous work.In addition, we provide a new large corpus of Wikipedia texts labeled according to our linguistically motivated guidelines.

Scripts representing common sense knowledge about stereotyped sequences of events have been shown to be a valuable resource for NLP applications.We present a hierarchical Bayesian model for unsupervised learning of script knowledge from crowdsourced descriptions of human activities.Events and constraints on event ordering are induced jointly in one unified framework.We use a statistical model over permutations which captures event ordering constraints in a more flexible way than previous approaches.In order to alleviate the sparsity problem caused by using relatively small datasets, we incorporate in our hierarchical model an informed prior on word distributions.The resulting model substantially outperforms a state-of-the-art method on the event ordering task.

Humans use rich natural language to describe and communicate visual perceptions. In order to provide natural language descriptions for visual content, this paper combines two important ingredients. First, we generate a rich semantic representation of the visual content including e.g. object and activity labels. To predict the semantic representation we learn a CRF to model the relationships between different components of the visual input. And second, we propose to formulate the generation of natural language as a machine translation problem using the semantic representation as source language and the generated sentences as target language. For this we exploit the power of a parallel corpus of videos and textual descriptions and adapt statistical machine translation to translate between our two languages. We evaluate our video descriptions on the TACoS dataset, which contains video snippets aligned with sentence descriptions. Using automatic evaluation and human judgments we show significant improvements over several baseline approaches, motivated by prior work. Our translation approach also shows improvements over related work on an image description task.

Recent work has shown that the integration of visual information into text-based models can substantially improve model predictions, but so far only visual information extracted from static images has been used. In this paper, we consider the problem of grounding sentences describing actions in visual information extracted from videos. We present a general purpose corpus that aligns high quality videos with multiple natural language descriptions of the actions portrayed in the videos, together with an annotation of how similar the action descriptions are to each other. Experimental results demonstrate that a text-based model of similarity between actions improves substantially when combined with visual information from videos depicting the described actions.

Abstract In this article, we underpin the intuition that frame semantic information is a useful resource for modelling textual entailment. To this end, we provide a manual frame semantic annotation for the test set used in the second recognizing textual entailment (RTE) challenge – the FrameNet-annotated textual entailment (FATE) corpus – and discuss experiments we conducted on this basis. In particular, our experiments show that the frame semantic lexicon provided by the Berkeley FrameNet project provides surprisingly good coverage for the task at hand. We identify issues of automatic semantic analysis components, as well as insufficient modelling of the information provided by frame semantic analysis as reasons for ambivalent results of current systems based on frame semantics.

The potential of knowledge-based technological support for work in the legal domain has become widely recognized in recent time. This paper argues for an approach that is meant to complement existing work on top-level, statute-based ontologies for the law by extracting conceptual knowledge from verdict texts. Starting from a corpus-based survey on the definition and modification of concepts in German court verdicts we are currently working out a detailed description of the linguistic structures and semantic mechanisms involved. Based on this we want to develop tools that automatically recognize and process portions of verdicts that contain definitions. The resulting system will support the corpus-based extension of (hand-made) top-level legal ontologies and the update of legal terminological knowledge bases with regard to new jurisdiction. The research described in the paper is conducted at the Saarbrücken computational linguistics (CL) group within the project CORTE (computational linguistic methods for legal terminology).

We describe the ongoing construction of a large, semantically annotated corpus resource as reliable basis for the large-scale acquisition of word-semantic information, e.g. the construction of domain-independent lexica. The backbone of the annotation are semantic roles in the frame semantics paradigm. We report experiences and evaluate the annotated data from the first project stage. On this basis, we discuss the problems of vagueness and ambiguity in semantic annotation.

In this paper we revisit Pustejovsky's proposal to treat ontologically complex word meaning by so-called dotted pairs. We use a higher-order feature logic based on Ohori's record λ-calculus to model the semantics of words like book and library, in particular their behavior in the context of quantification and cardinality statements.

We propose a unified framework in which to treat semantic underspecification and parallelism phenomena in discourse. The framework employs a constraint language that can express equality and subtree relations between finite trees. In addition, our constraint language can express the equality up-to relation over trees which captures parallelism between them. The constraints are solved by context unification. We demonstrate the use of our framework at the examples of quantifier scope, ellipsis, and their interaction.