Understanding natural language commands in situated Human-Robot Interaction (HRI) requires linking linguistic input to perceptual context.Traditional symbolic parsers lack the flexibility to operate in complex, dynamic environments.We introduce a novel Multimodal Grounded Semantic Role Labelling (G-SRL) framework that combines frame semantics with perceptual grounding, enabling robots to interpret commands via multimodal logical forms.Our approach leverages modern Vision Language Models (VLMs), which jointly process text and images, and is supported by an automated pipeline that generates high-quality training data.Structured command annotations are converted into photorealistic scenes via LLM-guided prompt engineering and diffusion models, then rigorously validated through object detection and visual question answering.The pipeline produces over 11,000 imagecommand pairs (3,500+ manually validated), while approaching the quality of manually curated datasets at significantly lower cost.

This paper explores Interactive Grounded Language Understanding (IGLU) challenges within Human-Robot Interaction (HRI). In this setting, a robot interprets user commands related to its environment, aiming to discern whether a specific command can be executed. If faced with ambiguities or incomplete data, the robot poses relevant clarification questions. Drawing from the NeurIPS 2022 IGLU competition, we enrich the dataset by introducing our multi-modal data and natural language descriptions in MM-IGLU: Multi-Modal Interactive Grounded Language Understanding. Utilizing a BART-based model that integrates the user’s statement with the environment’s description, and a cutting-edge Multi-Modal Large Language Model that merges both visual and textual data, we offer a valuable resource for ongoing research in the domain. Additionally, we discuss the evaluation methods for such tasks, highlighting potential limitations imposed by traditional string-match-based evaluations on this intricate multi-modal challenge. Moreover, we provide an evaluation benchmark based on human judgment to address the limits and capabilities of such baseline models. This resource is released on a dedicated GitHub repository at https://github.com/crux82/MM-IGLU.

In the rapidly evolving domain of medical technology, the utilization of sophisticated algorithms for deciphering transcriptional data has emerged as a critical aspect, especially in the oncology sector. these algorithms, drawing upon methodologies from fields such as natural language processing and advanced image analysis, can significantly enhance the accuracy in predicting cancer-related molecular states. notably, transformer models, renowned for their proficiency in handling extensive datasets, are now being adapted for breakthroughs in medical diagnostics or in stratifying patients according to prognostic levels. our study contributes to the field of precision medicine by integrating transformer-based learning, exemplified by the geneformer model, with explainable aI techniques. these techniques are employed to find out the input variables (genes resulting from genomic transcription) most correlated with the decisions of neural network systems. this insight, a key goal in genomic research, aims to select the most relevant gene subset for each specific task in which a neural network is employed. this selection approach has proven to be effective in two classification tasks: cell type classification and breast cancer type classification. such effectiveness has been demonstrated even across various cohorts of patients. when applying geneformer-like architecture analyses solely to the selected gene subsets, the outcomes either maintain their accuracy or significantly improve. this approach, aims not only to contribute to the identification of vital genetic markers in cancer genomics, but also to exemplify the adaptability of aI models to different datasets, marking a significant step towards the development of accurate and universally applicable diagnostic tools for precision medicine.

Danilo Croce, Simone Filice, Giuseppe Castellucci, Roberto Basili. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

While AI is going to produce a great impact on society, its alignment with human values and expectations is an essential step towards a correct harnessing of AI potentials for good. There is a corresponding growing need for mature and established technical standards to enable the assessment of an AI application as the evaluation of its graded adherence to formalized ethics. This is clearly dependent on methods to inject ethical awareness at all stages of an AI application development and use. For this reason we introduce the notion of Embedding Principles of ethics by Design (EPbD) as a comprehensive inductive framework. Although extending generic AI applications, it mainly aims at learning the ethical behaviour through numerical optimization, i.e. deep neural models. The core idea is to support ethics by integrating automated reasoning over formal knowledge and induction from ethically enriched training data. A deep neural network is proposed here to model both the functional as well as the ethical conditions characterizing a target decision. In this way, the discovery of latent ethical knowledge is enabled and made available to the learning process. The application of the above framework to a banking application, i.e. AI-driven Digital Lending, is used to show how accurate classification can be achieved without neglecting the ethical dimension. Results over existing datasets demonstrate that the ethical compliance of the sources can be used to output models able to optimally fine tune the balance between business and ethical accuracy.

Recent Transformer-based architectures, e.g., BERT, provide impressive results in many Natural Language Processing tasks.However, most of the adopted benchmarks are made of (sometimes hundreds of) thousands of examples.In many real scenarios, obtaining highquality annotated data is expensive and timeconsuming; in contrast, unlabeled examples characterizing the target task can be, in general, easily collected.One promising method to enable semi-supervised learning has been proposed in image processing, based on Semi-Supervised Generative Adversarial Networks.In this paper, we propose GAN-BERT that extends the fine-tuning of BERT-like architectures with unlabeled data in a generative adversarial setting.Experimental results show that the requirement for annotated examples can be drastically reduced (up to only 50-100 annotated examples), still obtaining good performances in several sentence classification tasks.

Danilo Croce, Daniele Rossini, Roberto Basili. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Abstract Sentence embeddings are the suitable input vectors for the neural learning of a number of inferences about content and meaning. Similarity estimation, classification, emotional characterization of sentences as well as pragmatic tasks, such as question answering or dialogue, have largely demonstrated the effectiveness of vector embeddings to model semantics. Unfortunately, most of the above decisions are epistemologically opaque as for the limited interpretability of the acquired neural models based on the involved embeddings. We think that any effective approach to meaning representation should be at least epistemologically coherent. In this paper, we concentrate on thereadabilityof neural models, as a core property of any embedding technique consistent and effective in representing sentence meaning. In this perspective, this paper discusses a novel embedding technique (the Nyström methodology) that corresponds to the reconstruction of a sentence in a kernel space, inspired by rich semantic similarity metrics (a semantic kernel) rather than by a language model. In addition to being based on a kernel that captures grammatical and lexical semantic information, the proposed embedding can be used as the input vector of an effective neural learning architecture, calledKernel-based deep architectures(KDA). Finally, it also characterizesby designthe KDA explanatory capability, as the proposed embedding is derived from examples that are both human readable and labeled. This property is obtained by the integration of KDAs with an explanation methodology, calledlayer-wise relevance propagation (LRP), already proposed in image processing. The Nyström embeddings support here the automatic compilation of argumentations in favor or against a KDA inference, in form of an explanation: each decision can in fact be linked through LRP back to the real examples, that is, the landmarks linguistically related to the input instance. The KDA network output is explained via the analogy with the activated landmarks. Quantitative evaluation of the explanations shows that richer explanations based on semantic and syntagmatic structures characterize convincing arguments, as they effectively help the user in assessing whether or not to trust the machine decisions in different tasks, for example, Question Classification or Semantic Role Labeling. This confirms the epistemological benefit that Nyström embeddings may bring, as linguistically rich and meaningful representations for a variety of inference tasks.

Kernel methods enable the direct usage of structured representations of textual data during language learning and inference tasks.Expressive kernels, such as Tree Kernels, achieve excellent performance in NLP.On the other side, deep neural networks have been demonstrated effective in automatically learning feature representations during training.However, their input is tensor data, i.e., they cannot manage rich structured information.In this paper, we show that expressive kernels and deep neural networks can be combined in a common framework in order to (i) explicitly model structured information and (ii) learn non-linear decision functions.We show that the input layer of a deep architecture can be pre-trained through the application of the Nyström low-rank approximation of kernel spaces.The resulting "kernelized" neural network achieves state-of-the-art accuracy in three different tasks.

In Kernel-based Learning the targeted phenomenon is summarized by a set of explanatory examples derived from the training set. When the model size grows with the complexity of the task, such approaches are so computationally demanding that the adoption of comprehensive models is not always viable.In this paper, a general framework aimed at minimizing this problem is proposed: multiple classifiers are stratified and dynamically invoked according to increasing levels of complexity corresponding to incrementally more expressive representation spaces.Computationally expensive inferences are thus adopted only when the classification at lower levels is too uncertain over an individual instance. The application of complex functions is thus avoided where possible, with a significant reduction of the overall costs. The proposed strategy has been integrated within two well-known algorithms: Support Vector Machines and Passive-Aggressive Online classifier.A significant cost reduction (up to 90%), with a negligible performance drop, is observed against two Natural Language Processing tasks, i.e. Question Classification and Sentiment Analysis in Twitter.

Modern robotic architectures are equipped with sensors enabling a deep analysis of the environment. In this work, we aim at demonstrating that such perceptual information (here modeled through semantic maps) can be effectively used to enhance the language understanding capabilities of the robot. A robust lexical mapping function based on the Distributional Semantics paradigm is here proposed as a basic model of grounding language towards the environment. We show that making such information available to the underlying language understanding algorithms improves the accuracy throughout the entire interpretation process.

Kernel-based learning has been largely applied to semantic textual inference tasks. In particular, Tree Kernels (TKs) are crucial in the modeling of syntactic similarity between linguistic instances in Question Answering or Information Extraction tasks. At the same time, lexical semantic information has been studied through the adoption of the so-called Distributional Semantics (DS) paradigm, where lexical vectors are acquired automatically from large corpora. Notice how methods to account for compositional linguistic structures (e.g. grammatically typed bi-grams or complex verb or noun phrases) have been proposed recently by defining algebras on lexical vectors. The result is an extended paradigm called Distributional Compositional Semantics (DCS). Although lexical extensions have been already proposed to generalize TKs towards semantic phenomena (e.g. the predicate argument structures as for role labeling), currently studied TKs do not account for compositionality, in general. In this paper, a novel kernel called Compositionally Smoothed Partial Tree Kernel is proposed to integrate DCS operators into the tree kernel evaluation, by acting both over lexical leaves and non-terminal, i.e. complex compositional, nodes. The empirical results obtained on a Question Classification and Paraphrase Identification tasks show that state-of-the-art performances can be achieved, without resorting to manual feature engineering, thus suggesting that a large set of Web and text mining tasks can be handled successfully by the kernel proposed here.

Robots are slowly becoming part of everyday life, as they are being marketed for commercial applications (viz. telepresence, cleaning or entertainment). Thus, the ability to interact with non-expert users is becoming a key requirement. Even if user utterances can be efficiently recognized and transcribed by Automatic Speech Recognition systems, several issues arise in translating them into suitable robotic actions. In this paper, we will discuss both approaches providing two existing Natural Language Understanding workflows for Human Robot Interaction. First, we discuss a grammar based approach: it is based on grammars thus recognizing a restricted set of commands. Then, a data driven approach, based on a free-from speech recognizer and a statistical semantic parser, is discussed. The main advantages of both approaches are discussed, also from an engineering perspective, i.e. considering the effort of realizing HRI systems, as well as their reusability and robustness. An empirical evaluation of the proposed approaches is carried out on several datasets, in order to understand performances and identify possible improvements towards the design of NLP components in HRI.

Online algorithms are an important class of learning machines as they are extremely simple and computationally efficient. Kernel methods versions can handle structured data, such as trees, and achieve state-of-the-art performance. However kernelized versions of Online Learning algorithms slow down when the number of support vectors becomes large. The traditional way to cope with this problem is introducing budgets that set the maximum number of support vectors. In this paper, we investigate Distributed Trees (DT) as an efficient way to use structured data in online learning. DTs effectively embed the huge feature space of the tree fragments into small vectors, so enabling the use of linear versions of kernel machines over tree structured data. We experiment with the Passive-Aggressive (PA) algorithm by comparing the linear and the kernelized version. A massive dataset made with tree structured data is employed: it is originated from a natural language processing task, the Boundary Detection in the context of Semantic Role Labeling over Frame Net. Results on a sample of the final data show that the DTs along with the Linear PA algorithm and the Tree Kernel along with the Bundgeted PA achieve comparable results in terms of f1-measure. Finally, the exploration of the full dataset allows the former to improve the performance on the classification task, with respect to the latter.

In recent years, natural language processing techniques have been used more and more in IR. Among other syntactic and semantic parsing are effective methods for the design of complex applications like for example question answering and sentiment analysis. Unfortunately, extracting feature representations suitable for machine learning algorithms from linguistic structures is typically difficult. In this paper, we describe one of the most advanced piece of technology for automatic engineering of syntactic and semantic patterns. This method merges together convolution dependency tree kernels with lexical similarities. It can efficiently and effectively measure the similarity between dependency structures, whose lexical nodes are in part or completely different. Its use in powerful algorithm such as Support Vector Machines (SVMs) allows for fast design of accurate automatic systems.

Purpose of this work is the study of cloud detection techniques. This work identifies the cloud cover of optical images acquired by the QuickBird satellite, comparing these with others of the same area, acquired by Landsat 7 in which there are no clouds. The images are combined using an early fusion technique [1]. The tool exploits the neighborhood model [2] for increasing the amount of information for the training set and the Singular Value Decomposition for carrying out the feature extraction [3]. In order to introduce these structures into thematic classification tasks by SVMs it was necessary develop a tree kernel function based on tree kernel function defined in SVM-LightTK. The aim of the tree kernel function is evaluate the similarity level between a generic couples of tree structures.In this paper we report the results obtained comparing the performance of different approaches in cloud classification problem. The final purpose is the production of cloud cover maps. Throughout such different experimental setups we measured the capabilities of each algorithm under different points of view. First of all, we considered the classification accuracy by computing traditional parameter such as overall accuracy. A second analysis regarded the efforts that are required in the design of optimal algorithms. Indeed, these techniques are characterized by different parameters that have to be appropriately tuned in order to obtain the best performance. Finally the robustness of the techniques has been also considered. In particular the classification accuracy has been evaluated also for images not considered in the training phase.

No abstract available.

The distributional hypothesis states thatwords with similar distributional properties have similar semantic properties(Harris 1968). This perspective on word semantics, was early discussed in linguistics (Firth 1957; Harris 1968), and then successfully applied to Information Retrieval (Salton, Wong and Yang 1975). In Information Retrieval, distributional notions (e.g. document frequency and word co-occurrence counts) have proved a key factor of success, as opposed to early logic-based approaches to relevance modeling (van Rijsbergen 1986; Chiaramella and Chevallet 1992; van Rijsbergen and Lalmas 1996).

Semantic role labeling systems are often designed as inductive processes over annotated resources. Supervised algorithms based on complex grammatical information achieve state-of-the-art accuracy. However, their generalization on the argument classification task is poorer, as large performance drops in out-of-domain tests showed. In this paper, a robust method based on a minimal set of grammatical features and a distributional model of lexical semantic information is proposed. The achievable generalization ability is studied in several training conditions where negligible performance drops are observed.

This paper studies the impact of several learning issues in an image classification task with SVMs, such as rich feature-based representations, optimization and sensitivity to novelty in the test data sets. The employed imagery refers to the city of Rome, Italy and is acquired in different years and seasons by the European Remote Sensing Satellites ERS-1 and ERS-1/2 tandem mission. A comprehensive evaluation according to varying training conditions is reported, showing that SVMs provide robust and largely applicable tools.

The availability of large scale data sets of manually annotated predicate-argument structures has recently favored the use of machine learning approaches to the design of automated semantic role labeling (SRL) systems. The main research in this area relates to the design choices for feature representation and for effective decompositions of the task in different learning models. Regarding the former choice, structural properties of full syntactic parses are largely employed as they represent ways to encode different principles suggested by the linking theory between syntax and semantics. The latter choice relates to several learning schemes over global views of the parses. For example, re-ranking stages operating over alternative predicate-argument sequences of the same sentence have shown to be very effective. In this article, we propose several kernel functions to model parse tree properties in kernel-based machines, for example, perceptrons or support vector machines. In particular, we define different kinds of tree kernels as general approaches to feature engineering in SRL. Moreover, we extensively experiment with such kernels to investigate their contribution to individual stages of an SRL architecture both in isolation and in combination with other traditional manually coded features. The results for boundary recognition, classification, and re-ranking stages provide systematic evidence about the significant impact of tree kernels on the overall accuracy, especially when the amount of training data is small. As a conclusive result, tree kernels allow for a general and easily portable feature engineering method which is applicable to a large family of natural language processing tasks.

In this paper we propose a new approach to the design of semantic smoothing kernels for text classification. These kernels implicitly encode a superconcept expansion in a semantic network using well-known measures of term similarity. The experimental evaluation on two different datasets indicates that our approach consistently improves performance in situations of little training data and data sparseness.

A tight integration between ontological and linguistic knowledge is critical within the information processes of the Semantic Web. In Information Extraction, ontologies should include knowledge components neglected in domain conceptualizations generally used for other tasks. In this paper, we analyze such critical information in the light of existing applications. Accordingly, a methodology for semi-automatic development of an IE ontology integrating pre-existing domain and lexical knowledge is presented. The proposed ontological framework supports the discovery of new relations among known concepts by means of text processing, but also induction of new conceptual information.

Text understanding makes strong assumptions about the conceptualisation of the underlying knowledge domain. This mediates between the accomplishment of the specific task at the one hand and the knowledge expressed in the target text fragments at the other. However, building domain conceptualisations from scratch is a very complex and time-consuming task. Traditionally, the reuse of available domain resources, although not constituting always the best, has been applied as an accurate and cost effective solution. Here, we investigate the possibility of exploiting sources of domain knowledge (e.g. a subject reference system) to build a linguistically motivated domain concept hierarchy. The limitation connected with the use of domain taxonomies as ontological resources will be firstly discussed in the specific light of IE, i.e. for supporting linguistic inference. We then define a method for integrating the taxonomical domain knowledge and a general-purpose lexical knowledge base, like WordNet. A case study, i.e. the integration of the MeSH, Medical Subject Headings, and WordNet, will be then presented as a proof of the effectiveness and accuracy of the overall approach.

Robustness has been traditionally stressed as a general desirable property of any computational model and system. The human NL interpretation device exhibits this property as the ability to deal with odd sentences. However, the difficulties in a theoretical explanation of robustness within the linguistic modelling suggested the adoption of an empirical notion. In this paper, we propose an empirical definition of robustness based on the notion of performance. Furthermore, a framework for controlling the parser robustness in the design phase is presented. The control is achieved via the adoption of two principles: the modularisation, typical of the software engineering practice, and the availability of domain adaptable components. The methodology has been adopted for the production of CHAOS, a pool of syntactic modules, which has been used in real applications. This pool of modules enables a large validation of the notion of empirical robustness, on the one side, and of the design methodology, on the other side, over different corpora and two different languages (English and Italian).

Methods for taking into account linguistic content into text retrieval are receiving growing attention. Text categorization is an interesting area for evaluating and quantifying the impact of linguistic information. Work on text retrieval through the Internet suggests that embedding linguistic information at a suitable level within traditional quantitative approaches is the crucial issue able to bring the experimental stage to operational results. This kind of representational problem is studied in this paper where traditional methods for statistical text categorization are augmented via a systematic use of linguistic information. The addition of NLP capabilities also suggested a different application of existing methods in revised forms. This paper presents an extension of the Rocchio formula as a feature weighting and selection model used as a basis for multilingual information extraction. It allows an effective exploitation of the available linguistic information that better emphasizes the latter with significant data compression and accuracy. The results is an original statistical classifier fed with linguistic features and characterized by the novel feature selection and weighting model. It outperforms existing systems while keeping most of their interesting properties. Extensive tests of the model suggest its application as a viable and robust tool for large scale text classification and filtering, as well as a basic module for more complex scenarios.