Large Language Models (LLMs) are increasingly costly to fine-tune due to their size, with embedding layers alone accounting for up to 20% of model parameters.While Parameter-Efficient Fine-Tuning (PEFT) methods exist, they largely overlook the embedding layer.In this paper, we introduce TinyTE, a novel PEFT approach that steers model behavior via minimal translational transformations in the embedding space.TinyTE modifies input embeddings without altering hidden layers, achieving competitive performance while requiring approximately 0.0001% of the parameters needed for full fine-tuning.Experiments across architectures provide a new lens for understanding the relationship between input representations and model behavior-revealing them to be more flexible at their foundation than previously thought. 1

Recent advances in vision-language models have enabled rich semantic understanding across modalities. However, these encoding methods lack the ability to interpret or reason about the moral dimensions of content---a crucial aspect of human cognition. In this paper, we address this gap by introducing MoralCLIP, a novel embedding representation method that extends multimodal learning with explicit moral grounding based on Moral Foundations Theory (MFT). Our approach integrates visual and textual moral cues into a unified embedding space, enabling cross-modal moral alignment. MoralCLIP is grounded on the multi-label dataset Social-Moral Image Database to identify co-occurring moral foundations in visual content. For MoralCLIP training, we design a moral data augmentation strategy to scale our annotated dataset to 15,000 image-text pairs labeled with MFT-aligned dimensions. Our results demonstrate that explicit moral supervision improves both unimodal and multimodal understanding of moral content, establishing a foundation for morally-aware AI systems capable of recognizing and aligning with human moral values.

Diogo Glória-Silva, Rafael Ferreira, Diogo Tavares, David Semedo, Joao Magalhaes. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

We present a method for performing zero-shot Dialogue State Tracking (DST) by casting the task as a learning-to-ask-questions framework. The framework learns to pair the best question generation (QG) strategy with in-domain question answering (QA) methods to extract slot values from a dialogue without any human intervention. A novel self-supervised QA pretraining step using in-domain data is essential to learn the structure without requiring any slot-filling annotations. Moreover, we show that QG methods need to be aligned with the same grammatical person used in the dialogue. Empirical evaluation on the MultiWOZ 2.1 dataset demonstrates that our approach, when used alongside robust QA models, outperforms existing zero-shot methods in the challenging task of zero-shot cross domain adaptation-given a comparable amount of domain knowledge during data creation. Finally, we analyze the impact of the types of questions used, and demonstrate that the algorithmic approach outperforms template-based question generation.

This paper introduces TWIZ, a multimodal conversational task wizard that supports an engaging experience, where users are guided through a multimodal conversation, towards the successful completion of recipes and DIY tasks. TWIZ leverages task guides from WikiHow and Recipe sources, as well as dialog AI-based methods to deliver a rich, compelling, and engaging experience when guiding users through complex manual tasks. TWIZ participated in the Amazon Alexa Prize Taskbot 2021.

The online shopping industry benefits from the usage of virtual assistants that are able to provide a 24/7 way of communication. This is normally done through the use of text based interactions. In order to improve the user experience and provide a more engaging user-to-customer interaction, we present a virtual shopping environment that emulates the physical store, providing the user with an experience similar to buying products in the actual store. This allows users to interact with the store items and have a full conversation with a virtual shopping assistant. This is all done through a virtual store where the user is able to walk, try-on and see the displayed items, while having a conversation with the store's assistant.

Multimodal conversational agents are an ever expanding field which benefits from the introduction of large language models. Production-ready robust conversational assistants trade breadth of scope for higher accuracy and general dialogue quality. These conversational assistants must be able to maintain the conversation focused, respond appropriately to user requests, maintain a certain level of natural response generation, be robust to out-of-scope and chitchat attempts, and, of course, be accurate in assisting the user in reaching their domain-specific goals. This work discusses data-centric observations, alongside providing research hypothesis for future, and some of my already developed work, to be expanded throughout my PhD.