Automated audio captioning (AAC) benefits from incorporating external context to interpret complex sounds, but doing so with retrieval-augmented generation (RAG) at inference is sometimes infeasible due to data availability or incurs significant latency and complexity. We propose DistillCaps, a novel training-time framework that leverages RAG to guide knowledge distillation for improved audio-language alignment, while lessening the reliance on retrieval during inference. In our framework, a RAG-equipped teacher model retrieves relevant textual information (e.g., similar captions) for each audio clip and uses it for training to generate context-enriched captions. Simultaneously, a student model is trained to imitate this teacher, learning to produce high-quality captions from audio alone. We further introduce a Fast Fourier Transform (FFT) adapter in the audio encoder to inject frequency-domain features, enhancing the quality of audio representations before feeding them into the language model. The result is an efficient captioning model that retains RAG's contextual benefits without its deployment overhead. On standard AAC benchmarks (AudioCaps and Clotho), DistillCaps achieves performance competitive with or exceeding prior RAG-based systems despite using no retrieval at test time. Notably, our distilled model matches state-of-the-art captioning results under real-time settings, and when optionally allowing retrieval, it even outperforms previous models by up to 4% on the Clotho benchmark on the in-distribution setting, demonstrating the effectiveness of RAG-guided distillation for audio-language alignment. Code and dataset are available here. https://github.com/pgthinh/DistillCaps.

Lifelogging is the passive collection, storage and analysis of daily data through wearable sensors. Question Answering (QA) for lifelog data enables natural language interactions with personal daily life records, providing insights into individual routines and behaviours. While this task has great potential for personal analytics and memory augmentation, progress has been limited due to the challenges of lifelog management, since they can comprise of enormous multi-modal data sets spanning a lifetime. We introduce a Retrieval-Augmented Generation (RAG) approach for addressing the lifelog QA task. A RAG approach first includes a retrieval model finding the correct lifelog events containing answers and then a large language model (LLM) generating answers from the questions. In addition, we construct an open-ended lifelog QA benchmark with 14,187 QA pairs to examine the RAG approach to lifelog QA. Using an embedding-based retrieval approach, our lifelog context retriever achieves a performance of 77.67% Recall@5 and 94.35% Recall@20 using an embedding-based retrieval approach with the Stella 1.5B model. Combined with the Mistral 7B model, the model achieves scores of 39.54% ROUGE-L and 3.475 Accuracy on a scale of 5 scored by GPT-4o. This approach potentially provides an effective approach to lifelog QA with high performance that does not require fine-tuning.

Lifelogging is the process of automatically logging data about an individual's daily life, which can then be used in various domains, such as behavior analysis and health monitoring. Various technological devices, including wearable cameras and smartwatches, can help record lifelog data, but getting access to lifelog cameras has proven difficult in recent years, due to a lack of such devices on the market. Creating a lifelog camera that is not only easy to use and cost-efficient but also provides comprehensive functions to log all images about life is challenging due to the lack of hardware and software. This paper introduces MemoriLens, a low-cost camera that efficiently collects, organizes, and stores lifelog data using a readily available custom-designed Raspberry Pi Zero board. The camera is designed to capture images automatically and send them to a private account in cloud services for storage. We open-source the implementing of the camera at: https://github.com/linh222/raspberry_lifelog_camera and we encourage lifelog researchers to use our designs and software as required.

Cross-data analytics and retrieval have gained significant improvement recently. People can now extract more data insights precisely and quickly towards having many excellent applications serving human lives. Since people create multimedia and other types of data that reflect the diverse perspectives of human lives, these data are just pieces of the puzzle of the world's pictures. Hence, it is necessary to assembly all these pieces towards having a better solution for human-centered problems. Hence, the workshop welcomes those who work with multimedia and others and come from diverse research domains and disciplines to work on intelligent cross-data analytics and retrieval to bring a smart, sustainable society to human beings. The research domain can vary from well-being, disaster prevention and mitigation, mobility to food computing, to name a few.

In many countries worldwide, effectively estimating AQI values and levels is essential for better monitoring the air pollution around the living area. This problem has become one of the interesting research subjects for many years, and there are many applications developed for personal usages. In this work, we aim to investigate a multi-source machine learning approach to approximate the local AQI scores at users' location in a big city. We conduct different experiments on three primary data sets: "SEPHLA-MediaEval 2019", "MNR-Air-HCM," and "MNR-HCM," collected in Ho Chi Minh City (Vietnam) and Fukuoka city (Japan). From the data sets provided, we extract different types of useful attributes for the problem: the timestamp information, the geographical data, sensor data (humidity and temperature), users' emotion tags (such as greenness, calmness, etc.), the semantic features from images captured by users as well as the public weather data (including temperature, dew point, humidity, wind speed, and pressure) of the related cities. After that, we compare five distinct machine learning models for estimating the local AQI score and level, including Support Vector Machine [1], Random Forest [2], Extreme Gradient Boosting [3], LightGBM [4] and CatBoost [5]. We use RMSE, MAE, and R2for measuring the performance of these approaches. The experimental results show that using random forest with sensor data, combined with public weather data, the results in AQI values regression and AQI ranks prediction can be the highest in many cases.

Nowadays, recommendation systems play an indispensable role in many fields, including e-commerce, finance, economy, and gaming. There is emerging research on publication venue recommendation systems to support researchers when submitting their scientific work. Several publishers such as IEEE, Springer, and Elsevier have implemented their submission recommendation systems only to help researchers choose appropriate conferences or journals for submission. In this work, we present a demo framework to construct an effective recommendation system for paper submission. With the input data (the title, the abstract, and the list of possible keywords) of a given manuscript, the system recommends the list of top relevant journals or conferences to authors. By using state-of-the-art techniques in natural language understanding, we combine the features extracted with other useful handcrafted features. We utilize deep learning models to build an efficient recommendation engine for the proposed system. Finally, we present the User Interface (UI) and the architecture of our paper submission recommendation system for later usage by researchers.

With the rapid growth of users' data in SaaS (Software-as-a-service) platforms using micro-services, it becomes essential to detect duplicated entities for ensuring the integrity and consistency of data in many companies and businesses (primarily multinational corporations). Due to the large volume of databases today, the expected duplicate detection algorithms need to be not only accurate but also practical, which means that it can release the detection results as fast as possible for a given request. Among existing algorithms for the deduplicate detection problem, using Siamese neural networks with the triplet loss has become one of the robust ways to measure the similarity of two entities (texts, paragraphs, or documents) for identifying all possible duplicated items. In this paper, we first propose a practical framework for building a duplicate detection system in a SaaS platform. Second, we present a new active learning schema for training and updating duplicate detection algorithms. In this schema, we not only allow the crowd to provide more annotated data for enhancing the chosen learning model but also use the Siamese neural networks as well as the triplet loss to construct an efficient model for the problem. Finally, we design a user interface of our proposed deduplicate detection system, which can easily apply for empirical applications in different companies.

Inferring abstract contexts and activities from heterogeneous data is vital to context-aware ubiquitous applications but still remains one of the most challenging problems. Recent advances in Bayesian nonparametric machine learning, in particular the theory of topic models based on Hierarchical Dirichlet Process (HDP), has provided an elegant solution towards these challenges. However, limited existing methods have addressed the problem of inferring latent multifaceted activities and contexts from heterogeneous data sources such as those collected from mobile devices. In this paper, we extend the original HDP to model heterogeneous data using a richer structure of the base measure being a product-space. The proposed model, called product-space HDP (PS-HDP), naturally handles the heterogeneous data from multiple sources and identify the unknown number of latent structures in a principle way. Although this framework is generic, our current work primarily focuses on inferring (latent) threefold activities of who-when-where simultaneously, which corresponds to inducing activities from data collected for identity, location and time. We demonstrate our model on synthetic data as well as on a real-world dataset – the StudentLife dataset. We report results and provide analysis on the discovered activities and patterns to demonstrate the merit of the model. We also quantitatively evaluate the performance of PS-HDP model using standard metrics including F1-score, NMI, RI, purity, and compare them with well-known existing baseline methods.

Low cost pervasive electrocardiogram (ECG) monitors is changing how sinus arrhythmia are diagnosed among patients with mild symptoms. With the large amount of data generated from long-term monitoring, come new data science and analytical challenges. Although traditional rule-based detection algorithms still work on relatively short clinical quality ECG, they are not optimal for pervasive signals collected from wearable devices—they don't adapt to individual difference and assume accurate identification of ECG fiducial points. To overcome these short-comings of the rule-based methods, this paper introduces an arrhythmia detection approach for low quality pervasive ECG signals. To achieve the robustness needed, two techniques were applied. First, a set of ECG features with minimal reliance on fiducial point identification were selected. Next, the features were normalized using robust statistics to factors out baseline individual differences and clinically irrelevant temporal drift that is common in pervasive ECG. The proposed method was evaluated using pervasive ECG signals we collected, in combination with clinician validated ECG signals from Physiobank. Empirical evaluation confirms accuracy improvements of the proposed approach over the traditional clinical rules.