Utilizing richer information, such as structural and syntactic details, can enhance Natural Language Processing (NLP) tasks like Open Information Extraction (Open IE), particularly for languages with limited resources like Portuguese. Knowledge Graphs (KGs) offer a robust solution by unifying diverse annotations and enabling the application of Graph Machine Learning (Graph ML). This paper presents an advanced framework for Portuguese Open IE, integrating KGs and Graph ML with Large Language Model (LLM) augmentation. Our framework employs a three-stage process: (1) initial Knowledge Graph (KG) construction from text, followed by (2) Predicate Extraction and (3) Subject/Object Extraction, both leveraging GraphSAGE models. Large Language Models (LLMs) (DeepSeek) are used for augmentation when Graph ML predictions are absent or for refining/validating extractions. We present two versions of a system that was evaluated on a Portuguese dataset. Automatic evaluation (word-based) for the best version of the system yielded an F1-score of 64.9% for Predicate extraction and 89.7% for Subject/Object extraction. The final end-to-end performance of the system is an F1-score of 58.2%. A human evaluation was conducted on 51 Portuguese sentences (yielding 100 triples) by two annotators, achieving a substantial agreement (Cohen’s Kappa of 0.67). The system extracted an average of 1.84 triples per sentence, with 53.9% deemed correct. Notably, this version significantly reduced invalid/wrong extractions to 6.6% from 31.7% in the previous version, demonstrating improved Precision while maintaining the ability to extract multiple meaningful triples.

As smart technologies become increasingly embedded in daily life, the demand for natural and efficient human-machine interaction continues to grow. Multimodal systems, which allow interacting with multiple modalities, such as speech or gestures, potentially supporting a more natural and efficient way for users to communicate with machines. However, the complexity of these systems introduces significant challenges in development and evaluation. Nevertheless, even when interaction data are collected, these systems can generates large volumes of data, making it difficult to understand how users engage with the system, how the system responds, identify system limitations as well as uncover interaction patterns. In this sense, visualisation can play a crucial role in helping researchers and developers make sense of multimodal interaction data in order to reveal communication patterns, uncover system bottlenecks, and support deeper insight into system (and user) behavior. In this work we explore the potential of interactive visualisation, through a case study considering a multimodal system, and demonstrate how interactive visualisation can be useful to respond to a set of questions that we want to know about the system evaluation.

The aging population creates the need to promote and encourage healthy and active lifestyles to help prevent chronic diseases or injuries and consequently alleviate the pressure on healthcare systems. Technological advancements present an opportunity to monitor people more continuously and provide them with useful and relevant information that can indicate potential problems or simply help them to stay or be more active and healthy. Smart mirrors offer the promising capability of monitoring individuals daily, in their own homes, without interfering with their usual routine, a key factor for user acceptance and widespread adoption. To obtain an overview of the state-of-the-art on smart mirrors for health/well-being monitoring, we performed a review of the literature on the topic, focusing on mirrors incorporating sensing capabilities. Relevant information was then extracted from the articles found and a critical analysis was carried out. Based on the results of this analysis, as well as our own experience, we provide recommendations we consider important to take into account in the future development of smart mirrors that monitor people's health and well-being.

People with language impairments can have difficulties expressing themselves to others, leading to major limitations to their safety, independence, and quality of life in general. Aphasia is an example of an acquired language impairment that affects many people (around 2 million in the United States), being commonly caused by stroke, but also by other brain injuries. Several augmentative and alternative communication solutions are available to help people with communication difficulties, but they are generally not suitable for all contexts of use (e.g., lying in bed). In the scope of the “APH-ALARM” project, which aimed at developing solutions to support people with Aphasia, we envision a system for the bedroom that enables conveying messages to be sent to a caregiver or relative, for example. Focusing on gesture input, in this contribution, we investigated if smartwatch sensors and machine learning (ML) can be used to recognise arm gestures executed while lying. We explored different factors, namely the feature set, size of the sliding window used for feature extraction, and ML classifier. The results obtained with data gathered from ten subjects are promising, with the best factor combinations for the user-independent solution leading to a mean macro F1 score of 94% or 95%. They demonstrate the potential of using wearables to develop a gesture input modality for the in-bed scenario, which can also potentially be extended to other contexts (e.g., sitting in a bed, chair, or sofa, or standing). This research also provides useful insights that inform future work, including the development and deployment of communication support systems that can benefit not only people with communication difficulties (e.g., more independence), but also those caring for them (e.g., more peace of mind). • A new gesture-based interaction modality for communication support is proposed. • The proposal is aligned with the needs of users with Aphasia while lying in bed. • The use of wearable sensors for arm gesture recognition is investigated. • The best achieved mean F1 score was of 95%, for a user-independent solution.

The environments we live in are becoming increasingly smarter, creating the need of finding suitable ways to interact with them. While interaction modes such as speech, text, and touch remain prevalent, there are associated challenges related to background noise interference, accessibility issues, and reliance on devices (e.g., mobile, wearable). Gesture-based interaction emerges as a promising alternative or complement to those modes. In this context, radars represent a good option for enabling gesture recognition, since they rely on radio waves to detect moving targets, not presenting the disadvantages of other sensors such as cameras and wearables, which can be considered too intrusive by the users. This contribution intends to investigate the capabilities of radars for gesture-based interaction in smart environments, focusing on the possibility of using data provided by a radar and transfer learning to recognize a set of pre-defined arm gestures regardless of the distance between the user and radar. The results obtained based on data collected from a subject standing at four different distances (1, 2, 3, and 4 m) from the radar, showed a poor performance when considering a distance-independent solution. For a distance- dependent solution, relatively good results were achieved when considering all four distances together (mean accuracy of 96 %). These results are very useful for future developments of gesture- based interaction with smart environments in real scenarios.

People with aphasia (PwA) face a set of communication barriers in several contexts, e.g., in social contexts. Since augmentative and alternative communication (AAC) tools are often too general and provide limited support to this audience, the aim of this work is to explore novel ways of supporting communication for PwA that take into account their needs and characteristics and move beyond the scope covered by current solutions. This work describes the design and development of the first proof-of-concept of Aphluentia, a tool to support communication in everyday social situations for people with fluent aphasia, designed by working with speech and language therapists with experience working with PwA to inform and evaluate its characteristics. At its current stage, Aphluentia establishes the grounds for discussion and assessment of the proposal by PwA.

This study addresses effects of age and gender on acoustics of European Portuguese oral vowels, given to the fact of conflicting findings reported in prior research. Fundamental frequency (F0), formant frequencies (F1 and F2) and duration of vowels produced by a group of 113 adults, aged between 35 and 97 years old, were measured. Vowel space area (VSA) according to gender and age was also analysed. The results revealed that the most consistent age-related effect was an increase in vowel duration in both genders. F0 decreases above [50-64] for female and for male data suggests a slight drop over the age range [35- 64] and then an increase in an older age. That is, F0 tends to be closer between genders as age increases. In general, there is no evidence that F1 and F2 frequencies were lowering as age increased. Furthermore, there were no changes to VSA with ageing. These results provide a base of information to establish vowel acoustics normal patterns of ageing among Portuguese adults.

Gaze information has the potential to benefit Human-Computer Interaction (HCI) tasks, particularly when combined with speech. Gaze can improve our understanding of the user intention, as a secondary input modality, or it can be used as the main input modality by users with some level of permanent or temporary impairments. In this paper we describe a multimodal HCI system prototype which supports speech, gaze and the combination of both. The system has been developed for Active Assisted Living scenarios.

The elderly population is quickly increasing in the developed countries. However, in European Portuguese (EP) no studies have examined the impact of age-related structural changes in speech acoustics. The purpose of this paper is to analyse the effect of age ([60-70], [71-80] and [81-90]), gender and type of vowel in the acoustic characteristics (fundamental frequency (F0), first formant (F1), second formant (F2) and duration) of the EP vowels. A sample of 78 speakers was selected from the database of elderly speech collected by Microsoft Language Development Center (MLDC) within the Living Usability Lab (LUL) project. It was observed that duration is the only parameter that significantly changes with ageing, being the highest value found in the [81-90] group. Moreover, F0 decreases in females and increases in males with ageing. In general, F1 and F2 decreases with ageing, mainly in females. Comparing the data obtained with the results of previous studies with adult speakers, a trend towards the centralization of vowels with ageing is observed. This investigation is the starting point for a broader study which will allow to analyse the changes in vowels acoustics from childhood to old age in EP.

In research on Silent Speech Interfaces (SSI), different sources of information (modalities) have been combined, aiming at obtaining better performance than the individual modalities. However, when combining these modalities, the dimensionality of the feature space rapidly increases, yielding the well-known "curse of dimensionality". As a consequence, in order to extract useful information from this data, one has to resort to feature selection (FS) techniques to lower the dimensionality of the learning space. In this paper, we assess the impact of FS techniques for silent speech data, in a dataset with 4 non-invasive and promising modalities, namely: video, depth, ultrasonic Doppler sensing, and surface electromyography. We consider two supervised (mutual information and Fisher's ratio) and two unsupervised (meanmedian and arithmetic mean geometric mean) FS filters. The evaluation was made by assessing the classification accuracy (word recognition error) of three well-known classifiers (knearest neighbors, support vector machines, and dynamic time warping). The key results of this study show that both unsupervised and supervised FS techniques improve on the classification accuracy on both individual and combined modalities. For instance, on the video component, we attain relative performance gains of 36.2% in error rates. FS is also useful as pre-processing for feature fusion

This paper presents an evaluation of the dialogue manager (DM) used on Carl, a prototype of an intelligent service robot, designed and developed having in mind hosting tasks in a building or event. The developed DM, based on the “Information State” approach, is described. In an experimental evaluation, in which 10 participants attempted to complete several interaction tasks with the robot, 81% of tasks were performed successfully. The results of an usability evaluation are also presented and discussed.

Various information sources naturally contains new words that appear in a daily basis and which are not present in the vocabulary of the speech recognition system but are important for applications such as closed-captioning or information dissemination. To be recognized, those words need to be included in the vocabulary and the language model (LM) parameters updated. In this context, we propose a new method that allows including new words in the vocabulary even if no well suited training data is available, as is the case of archived documents, and without the need of LM retraining. It uses morpho-syntatic information about an in-domain corpus and part-of-speech word classes to define a new LM unigram distribution associated to the updated vocabulary. Experiments were carried out for a European Portuguese Broadcast News transcription system. Results showed a relative reduction of 4 % in word error rate, with 78 % of the occurrences of those newly included words being correctly recognized. Index Terms: morpho-syntactic analysis, POS tags, class-based language models, broadcast news, transcription systems

When transcribing Broadcast News data in highly inflected languages, the vocabulary growth leads to high out-of-vocabulary rates. To address this problem, we propose a daily and unsupervised adaptation approach which dynamically adapts the active vocabulary and LM to the topic of the current news segment during a multi-pass speech recognition process. Based on texts daily available on the Web, a story-based vocabulary is selected using a morpho-syntatic technique. Using an Information Retrieval engine, relevant documents are extracted from a large corpus to generate a story-based LM. Experiments were carried out for a European Portuguese BN transcription system. Preliminary results yield a relative reduction of 65.2% in OOV and 6.6% in WER.

In this work we present a recently acquired MRI database for European Portuguese. As a first example of possible studies, we present results on 2D and 3D analyses of European Portuguese nasals, particularly nasal vowels. This database will enable the extraction of 2D and/or 3D articulatory parameters as well as some dynamic information to include in articulatory synthesizers. It can also be useful to compare the production of European Portuguese with the production of other languages and have further insight on some of the European Portuguese characteristics, as the nasalization and coarticulation. The MRI database and related studies were made possible by the interdisciplinary nature of the research team, comprised of a radiologist, image processing specialists and a speech scientist.

Although the vocabularies of ASR systems are designed to achieve high coverage for the expected domain, out-ofvocabulary (OOV) words cannot be avoided. Particularly, for daily and real-time transcription of Broadcast News (BN) data in highly inflected languages, the rapid vocabulary growth leads to high OOV word rates. To overcome this problem, we present a new morpho-syntatic approach to dynamically select the target vocabulary for this particular domain by trading off between the OOV word rate and vocabulary size. We evaluate this approach against the common selection strategy based on word frequency. Experiments have been carried out for a European Portuguese BN transcription system. Results computed on seven news shows, yields a relative reduction of 37.8% in OOV word rate against the baseline system and 5.5% when compared with the word frequency common approach.

The paper focuses on an Information State (IS) based dialogue manager developed for Carl, an intelligent mobile robot. It uses a Knowledge Acquisition and Management (KAM) module that integrates information obtained from various interlocutors. This mixed-initiative dialogue manager (DM) handles pronoun resolution, is capable of performing different kinds of clarification/confirmation questions and generates observations based on the current knowledge acquired. An evaluation of the DM on knowledge acquisition tasks is shown.

The daily and real-time transcription of broadcast news (BN) is a challenging task both in acoustic and in language modeling. To achieve optimal performance, several problems have to be overcome. Particularly, when transcribing BN data in highly inflected languages, the vocabulary growth leads to high OOV word rates. To address this problem, we propose a daily vocabulary and LM adaptation framework which directly extracts new words based on contemporary written news available on the Internet and some linguistic knowledge about the words found on those news. Experiments have been carried out for a European Portuguese BN transcription system. Preliminary results computed on 7 shows, yields a relative reduction of 61% in OOV and 2.1% in WER.

The recent evolution of Carl, an intelligent mobile robot, is presented. The paper focuses on robust spoken language understanding (SLU) and on knowledge representation and reasoning (KRR). Robustness in SLU is achieved through the combination of deep and shallow parsing, tolerating non-grammatical utterances. The KRR module supports the integration of information coming from different interlocutors and is capable of handling contradictory facts. The knowledge representation language is based on semantic networks. Question answering is based on deductive as well as inductive inference. A preliminary evaluation of the efficiency of the SLU/KRR system, for the purpose of knowledge acquisition, is presented.

This paper presents three methods for dividing European Portuguese (EP) words into syllables, two of them handling graphemes as input, the other processing phone sequences. All three try to incorporate linguistic knowledge about EP syllable structure, but in different degrees. Experimental results showed, for the best method, percentage of correctly recognized syllable boundaries above 99.5 %, and comparable word accuracy. The much simpler finite state transducer based method also achieved a good performance, making it suitable for applications more interested in speed and memory footprint. Being syllabification an essential component of many speech and language processing systems, proposed methods can be useful to researchers working with the EP language.

This paper describes the architecture and current capabilities of Carl, a prototype of an intelligent service robot, designed having in mind such tasks as serving food in a reception or acting as a host in an organization. The approach that has been followed in the design of Carl is based on an explicit concern with the integration of the major dimensions of intelligence, namely communication, action, reasoning and learning. The paper focuses on the multi-modal human-robot communication capabilities of Carl, since these have been significantly improved during the last year.

The development of robots capable of accepting instructions in terms of familiar concepts to the user is still a challenge. For these robots to emerge it’s essential the development of natural language interfaces, since this is regarded as the only interface acceptable for a machine which expected to have a high level of interactivity with Man. Our group has been involved for several years in the development of a mobile intelligent robot, named Carl, designed having in mind such tasks as serving food in a reception or acting as a host in an organization. The approach that has been followed in the design of Carl is based on an ex-plicit concern with the integration of the major dimensions of intelligence, namely Communication, Action, Reasoning and Learning. This paper focuses on the multi-modal human-robot language communication capabilities of Carl, since these have been significantly improved during the last year. 1.

First attempts at incorporating models of frication into an articulatory synthesizer, with a modular and e xible design, are presented. Although the synthesizer allows the user to choose different combinations of source types, noise volume velocity sources have been used to generate turbulence. Preliminary results indicate that the model is capturing essential characteristics of the transfer functions and spectral characteristics of fricatives. Results also show the potential of performing synthesis based on broad articulatory congurations of fricatives.

In this paper new EMMA data regarding European Portuguese nasals is presented. Some details about corpus constitution, recording and annotation is given. First results from analysis are presented. Quantitative analysis of velum movement was done for nasal vowels between stops. For the other contexts representative examples are presented and qualitatively analysed. In all contexts nasal vowels are produced with an initial phase having an high velum position. This result supports our previous work conclusions, of nasal vowels viewed as dynamic sounds were beginning must have dominant lips radiation. Obtained knowledge has application in articulatory synthesis, our motivation for this study.

The development of robots that are able to accept instructions, via a friendly interface, in terms of concepts that are familiar to a human user remains a challenge. It is argued that designing and building such intelligent robots can be seen as the problem of integrating four main dimensions: human-robot communication, sensory motor skills and perception, decision-making capabilities, and learning. Although these dimensions have been thoroughly studied in the past, their integration has seldom been attempted in a systematic way. It is further argued that, for the common user, the only sufficiently practical interface is spoken language. The "body and soul" of CARL (Communication, Action, Reasoning and Learning), a robot currently under construction in our laboratory, are presented. The spoken-language interface is given particular attention.

In this paper we study the effect of source changes, caused by vocal tract load, in perception of nasality. For that we have developed an articulatory speech synthesizer, including a comprehensive nasal tract model and an interactive glottal source model. Our main objective was to investigate to what extent is necessary, in systems aimed to produce high quality synthetic sounds, to include the effect of source-tract interaction in the glottal source model when synthesizing nasal vowels. In our studies we used Portuguese nasal vowels. Portuguese uses nasalization of vowels in its phonological inventory. Changes in glottal wave, caused by the additional load of the nasal tract are more significant in vowels like [i] with low F1 and high F2. Effects are more dramatic in time rather frequency domain. Perception tests favor the idea that listener aren’t able to detect the perceptual effect of source-tract interaction changes caused by the additional coupling of the nasal tract. More tests are needed to support, or reject, this.