International audience

Diversity can be decomposed into three distinct concepts, namely: variety, balance and disparity. This paper borrows from the extensive formalization and measures of diversity developed in ecology in order to evaluate the variety and balance of multiword expression annotation produced by automatic annotation systems. The measures of richness, normalized richness, and two variations of Hill’s evenness are considered in this paper. We observe how these measures behave against increasingly smaller samples of gold annotations of multiword expressions and use their comportment to validate or invalidate their pertinence for multiword expressions in annotated texts. We apply the validated measures to annotations in 14 languages produced by systems during the PARSEME shared task on automatic identification of multiword expressions and on the gold versions of the corpora. We also explore the limits of such evaluation by studying the impact of lemmatization errors in the Turkish corpus used in the shared task.

This paper describes our efforts to extend the PARSEME framework to Modern Standard Arabic. Theapplicability of the PARSEME guidelines was tested by measuring the inter-annotator agreement in theearly annotation stage. A subset of 1,062 sentences from the Prague Arabic Dependency Treebank PADTwas selected and annotated by two Arabic native speakers independently. Following their annotations, anew Arabic corpus with over 1,250 annotated VMWEs has been built. This corpus already exceeds thesmallest corpora of the PARSEME suite, and enables first observations. We discuss our annotation guide-line schema that shows full MWE annotation is realizable in Arabic where we get good inter-annotator agreement.

Automatic identification of multiword expressions (MWEs), like to cut corners 'to do an incomplete job ', is a pre-requisite for semantically-oriented downstream applications.This task is challenging because MWEs, especially verbal ones (VMWEs), exhibit surface variability.This paper deals with a subproblem of VMWE identification: the identification of occurrences of previously seen VMWEs.A simple language-independent system based on a combination of filters competes with the best systems from a recent shared task: it obtains the best averaged F-score over 11 languages (0.6653) and even the best score for both seen and unseen VMWEs due to the high proportion of seen VMWEs in texts.This highlights the fact that focusing on the identification of seen VMWEs could be a strategy to improve VMWE identification in general.

This paper describes ODIL Syntax, a French treebank built on spontaneous speech transcripts. The syntactic structure of every speech turn is represented by constituent trees, through a procedure which combines an automatic annotation provided by a parser (here, the Stanford Parser) and a manual revision. ODIL Syntax respects the annotation scheme designed for the French TreeBank (FTB), with the addition of some annotation guidelines that aims at representing specific features of the spoken language such as speech disfluencies. The corpus will be freely distributed by January 2020 under a Creative Commons licence. It will ground a further semantic enrichment dedicated to the representation of temporal entities and temporal relations, as a second phase of the ODIL@Temporal project. The paper details the annotation scheme we followed with a emphasis on the representation of speech disfluencies. We then present the annotation procedure that was carried out on the Contemplata annotation platform. In the last section, we provide some distributional characteristics of the annotated corpus (POS distribution, multiword expressions).

This paper describes Contemplata, an annotation platform that offers a generic solution for treebank building as well as treebank enrichment with relations between syntactic nodes. Contemplata is dedicated to the annotation of constituency trees. The framework includes support for syntactic parsers, which provide automatic annotations to be manually revised. The balanced strategy of annotation between automatic parsing and manual revision allows to reduce the annotator workload, which favours data reliability. The paper presents the software architecture of Contemplata, describes its practical use and eventually gives two examples of annotation projects that were conducted on the platform.

Multiword expressions, especially verbal ones (VMWEs), show idiosyncratic variability, which is challenging for NLP applications, hence the need for VMWE identification. We focus on the task of variant identification, i.e. identifying variants of previously seen VMWEs, whatever their surface form. We model the problem as a classification task. Syntactic subtrees with previously seen combinations of lemmas are first extracted, and then classified on the basis of features relevant to morpho-syntactic variation of VMWEs. Feature values are both absolute, i.e. hold for a particular VMWE candidate, and relative, i.e. based on comparing a candidate with previously seen VMWEs. This approach outperforms a baseline by 4 percent points of F-measure on a French corpus.

The question of data reliability is of first importance to assess the quality of manually annotated corpora.Although Cohen ' s κ is the prevailing reliability measure used in NLP, alternative statistics have been proposed.This paper presents an experimental study with four measures (Cohen's κ, Scott's π, binary and weighted Krippendorff ' s α) on three tasks: emotion, opinion and coreference annotation.The reported studies investigate the factors of influence (annotator bias, category prevalence, number of coders, number of categories) that should affect reliability estimation.Results show that the use of a weighted measure restricts this influence on ordinal annotations.They suggest that weighted α is the most reliable metrics for such an annotation scheme.

In this paper, we describe the latest version of SIBYLLE, an AAC system that permits persons suffering from severe physical disabilities to enter text with any computer application and also to compose messages to be read out by a speech synthesis module. The system consists of a virtual keyboard comprising a set of keypads which allow entering characters or full words by a single-switch selection process. It also comprises a sophisticated word prediction component which dynamically calculates the most appropriate words for a given context. This component is auto-adaptive, i.e. it learns on every text the user has entered. It thus adapts its predictions to the user's language and the current topic of communication as well. So far the system works for French, German and English. Earlier versions of SIBYLLE have been used since 2001 in the Kerpape rehabilitation center (Brittany, France).

This paper discusses the issue of how a speech understanding system can be made robust against spontaneous speech phenomena (hesitations and repairs) as well as achieving a detailed analysis of spoken French. The Romus system is presented. It implements speech understanding in a two-stage process. The first stage achieves a finite-state shallow parsing that consists in segmenting the recognized sentence into basic units (spokenadapted chunks). The second one, a Link Grammar parser, looks for inter-chunks dependencies in order to build a rich representation of the semantic structure of the utterance. These dependencies are mainly investigated at a pragmatic level through the consideration of a task concept hierarchy. Discussion about the approach adopted, its benefits and limitations, is based on the results of the system's assessment carried out under different linguistic phenomena during an evaluation campaign held by the French CNRS.

This paper focuses on the modelling of the linguistic level of MICRO, a multi-agents speech understanding system largely inspired by cognitive models. It describes the cooperation between, on the one hand a syntactic parser using a Lexical Functional Grammar, and on the other hand an semantic analyser. The semantic analysis is achieved through a mechanism semantic priming carried out by a incremental associative network. We emphasize the adaptive abilities of such a cooperation, particularly in case of ungrammatical utterances, which are very common in spoken language. I.