In recent years, the increase in the elderly population has placed significant burdens on post-rehabilitation schemes, resulting in high logistical costs and considerable social impacts due to hospitalization or frequent visits. These challenges call for a transformation in the traditional approach to physical patient care, which can be achieved by leveraging the Internet of Medical Things (IoMT), particularly through the use of pervasive wearable sensors. When attached to patients during treatment or therapy, these sensors can provide valuable supplementary information to healthcare professionals. When it comes to adopting IoMT technologies, cost efficiency, portability, and generalization are key factors. Specifically, this study aims to enhance the cost-effectiveness and versatility of wearable eHealth monitoring architectures that utilize foot pressure-sensing hardware for the motor assessment of post-stroke and neurologically impaired patients. It leverages lower limb inertial measurement unit sensory information and machine learning to mitigate the reliance on foot pressure-sensing hardware. We demonstrate the potential of artificial intelligence (AI) in predicting fine-scale foot pressure using only inexpensive, off-the-shelf motion sensors. We propose a self-supervised, exercise-agnostic asynchronous foot pressure decoding model that does not require human annotation. The algorithm is thoroughly evaluated using appropriate performance metrics, and our experimental tests show promising results.

Human balance is essential for everyday activities, from basic functions like standing and walking to more complex movements required in sports and work tasks. Optimal balance reduces the risk of falls, a major cause of injuries among the elderly and individuals with certain medical conditions. However, traditional force platforms can be prohibitively expensive for smaller clinics and individual practitioners, and patients with mobility issues may find it difficult to access locations equipped with such platforms. In light of these challenges, this work explores the effectiveness and versatility of using simpler hardware for pressure sensing. As an alternative to conventional force platforms, low-cost wearable sensors, such as Inertial Measurement Units (IMUs), have been explored. This research focuses on developing a simple yet effective balance evaluation system using smart-glasses embedded with an IMU sensor to replace traditional feet pressure sensing machines. Furthermore, we developed a custom dataset for the estimation of sway parameters by simultaneously collecting the IMU data and labels from the force platform for a set of 20 participants. We experimented with various Deep Learning (DL) models, leveraging the latest advancements in Machine Learning (ML), to estimate sway parameters such as sway path, sway area, and their ratio, typically measured by the gold standard force platform. When evaluated with appropriate performance criteria, the experimental results indicate that our proposed methodology performs robustly using only accelerometer data.

The eHealth systems are in great demand, particularly during times of outbreak like COVID-19, when there is a shortage of caregivers. The technological advancements, such as wearable wireless devices, the Internet of Things, and improved machine learning methods have made these systems more reliable. In modern times, these systems can play a vital role in post-rehabilitation journeys, which have significant social impact and high costs in traditional settings. Cost efficiency, portability, and generalization are key factors in adopting new technology. In this study, we investigate the potential for optimizing and simplifying hardware in order to increase the cost-effectiveness and versatility of post-stroke eHealth rehabilitation systems. It leverages the rich information available from Inertial Measurement Unit (IMU) sensors to compensate the need for foot pressure sensing. We present the first attempt to demonstrate the potential of machine learning, aided by affordable off the shelf motion sensing devices, for foot pressure analysis. Our proposed foot pressure decoding model is trained in an exercise-agnostic, self-supervised manner that eliminates the need for human annotation. The algorithm is evaluated using appropriate performance metrics, and our experimental results show very promising performance.

We address the challenging Voice Activity Detection (VAD) problem, which determines "Who is Speaking and When?" in audiovisual recordings. The typical audio-based VAD systems can be ineffective in the presence of ambient noise or noise variations. Moreover, due to technical or privacy reasons, audio might not be always available. In such cases, the use of video modality to perform VAD is desirable. Almost all existing visual VAD methods rely on body part detection, e.g., face, lips, or hands. In contrast, we propose a novel visual VAD method operating directly on the entire video frame, without the explicit need of detecting a person or his/her body parts. Our method, named S-VVAD, learns body motion cues associated with speech activity within a weakly supervised segmentation framework. Therefore, it not only detects the speakers/not-speakers but simultaneously localizes the image positions of them. It is an end-to-end pipeline, person-independent and it does not require any prior knowledge nor pre-processing. S-VVAD performs well in various challenging conditions and demonstrates the state-of-the-art results on multiple datasets. Moreover, the better generalization capability of S-VVAD is confirmed for cross-dataset and person-independent scenarios.

This paper addresses nonverbal behavior analysis for the classification of perceived personality traits using novel deep visual activity (VA)-based features extracted only from key-dynamic images. Dynamic images represent short-term VA. Key-dynamic images carry more discriminative information i.e., nonverbal features (NFs) extracted from them contribute to the classification more than NFs extracted from other dynamic images. Dynamic image construction, learning long-term VA with CNN+LSTM, and detecting spatio-temporal saliency are applied to determine key-dynamic images. Once VA-based NFs are extracted, they are encoded using covariance, and resulting representation is used for classification. This method was evaluated on two datasets: small group meetings and vlogs. For the first dataset, proposed method outperforms not only the state-of-the-art VA-based methods but also multi-modal approaches for all personality traits. For extraversion classification, it performs better than i) the most popular key-frames selection algorithm, ii) random and uniform dynamic image selection, and iii) NFs extracted from all dynamic images. Furthermore, the ablation study proves the superiority of proposed method. For the further dataset, it performs as well as the state-of-the-art visual-NFs on average, while showing improved performance for agreeableness classification. Proposed method can be adapted to any application based on nonverbal behavior analysis, thanks to being data-driven.

We present an automatic voice activity detection (VAD) method that is solely based on visual cues. Unlike traditional approaches processing audio, we show that upper body motion analysis is desirable for the VAD task. The proposed method consists of components for body motion representation, feature extraction from a Convolutional Neural Network (CNN) architecture and unsupervised domain adaptation. The body motion representations as images are used by the feature extraction component, which is generic and person-invariant, thus, can be applied to a subject who has never been seen. The endmost component handles the domain-shift problem, which appears due to the fact that the way people move/ gesticulate while speaking might vary from subject to subject, which results in disparate body motion features and consequently poorer VAD performance. The experimental analyses applied on a publicly available real-world VAD dataset show that the proposed method performs better than the state-of-the-art video-only and multimodal VAD approaches. Moreover, the proposed method has a better generalization ability as VAD results are more consistent across different subjects. As another major contribution, we present a new multimodal dataset (called RealVAD), created from a real-world (no role-plays) panel discussion. This dataset contains many actual situations/ challenges that are missing in the previous VAD datasets. We benchmarked the RealVAD dataset by applying the proposed method as well as cross-dataset analyses. Particularly, the results of cross-dataset experiments highlight the remarkable positive contribution of the unsupervised domain adaptation applied.

We present the first publicly available annotations for the analysis of face-touching behavior. These annotations are for a dataset composed of audio-visual recordings of small group social interactions with a total number of 64 videos, each one lasting between 12 to 30 minutes and showing a single person while participating to four-people meetings. They were performed by in total 16 annotators with an almost perfect agreement (Cohen's Kappa=0.89) on average. In total, 74K and 2M video frames were labelled as face-touch and no-face-touch, respectively. Given the dataset and the collected annotations, we also present an extensive evaluation of several methods: rule-based, supervised learning with hand-crafted features and feature learning and inference with a Convolutional Neural Network (CNN) for Face-Touching detection. Our evaluation indicates that among all, CNN performed the best, reaching 83.76% F1-score and 0.84 Matthews Correlation Coefficient. To foster future research in this problem, code and dataset were made publicly available (github.com/IIT-PAVIS/Face-Touching-Behavior), providing all video frames, face-touch annotations, body pose estimations including face and hands key-points detection, face bounding boxes as well as the baseline methods implemented and the cross-validation splits used for training and evaluating our models.

Understanding small group face-to-face interactions is a prominent research problem for social psychology while the automatic realization of it recently became popular in social computing. This is mainly investigated in terms of nonverbal behaviors, as they are one of the main facet of communication. Among several multi-modal nonverbal cues, visual activity is an important one and its sufficiently good performance can be crucial for instance, when the audio sensors are missing. The existing visual activity-based nonverbal features, which are all hand-crafted, were able to perform well enough for some applications while did not perform well for some other problems. Given these observations, we claim that there is a need of more robust feature representations, which can be learned from data itself. To realize this, we propose a novel method, which is composed of optical flow computation, deep neural network based feature learning, feature encoding and classification. Additionally, a comprehensive analysis between different feature encoding techniques is also presented. The proposed method is tested on three research topics, which can be perceived during small group interactions i.e. meetings: i) emergent leader detection, ii) emergent leadership style prediction, and iii) high/low extraversion classification. The proposed method shows (significantly) better results not only as compared to the state of the art visual activity based-nonverbal features but also when the state of the art visual activity based-nonverbal features are combined with other audio-based and video-based nonverbal features.