Putting on hearing aids (HAs) is a regular and crucial task for every hearing aid wearer. A sub-optimal insertion can impact user adoption and audiological benefit. Ability to visually evaluate the insertion can be helpful to achieve a proper physical fit of hearing aids or similar devices, but this is currently a challenging task. In this work we present CheckMyFit, a smartphone-based, automated solution enabling users to quickly take a photo of their hearing aid placement, and compare it with a reference ideal insertion. To evaluate the tool's usability and potential benefit we conducted two user studies: a) a pilot lab study with 7 participants, and b) a field study with 17 participants. In the two-week field study, older participants with no prior hearing aid experiences were instructed on hearing aid insertion remotely and performed daily insertions independently at home. We found that CheckMyFit is easy and quick to use for almost all participants. Ear-photo-aided insertions tend to have higher quality than insertions without the tool. This correlation was significant and persisted throughout the 2 weeks of the study, and is retained after a short break. This suggests that CheckMyFit tool can provide real-world benefit to new users learning to insert their hearing aids. We also used CheckMyFit to remotely facilitate the field study, demonstrating its potential usefulness in tele-medicine.

Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we performed experiments with visual, physiological, and thermal information to explore potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual, physiological, and thermal groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the three modalities on identifying different types of driving distractions.

Considering the ever-growing presence of automobiles around the world, ensuring the safety of those on and near roadways is of great importance. From the causes of accidents, drowsiness and distractedness are among the most consequential. In this paper, we use a multimodal dataset consisting of 11 recorded channels over 45 subjects to model driver’s drowsiness and distraction. Our work puts forward the application of this dataset by using segmented windows as features, resulting in four main contributions. We explore the performance of each individual modality and specify which signals and features have a better capability of detecting drowsiness and different kinds of distractions. In addition, we analyze the effects of early fusion on the classification of the driver’s state using multiple physiological and thermal channels. Finally, we use cascaded late fusion and test three voting strategies to evaluate the performance of our proposed approach. Our results confirm the effectiveness of utilizing a multimodal approach in detecting both drowsiness and distraction as two separate factors influencing the driver and provide guidelines on which signals are appropriate for detecting different driver’s states.

Distracted driving is a leading cause of accidents worldwide. The tasks of distraction detection and recognition have been traditionally addressed as computer vision problems. However, distracted behaviors are not always expressed in a visually observable way. In this work, we introduce a novel multimodal dataset of distracted driver behaviors, consisting of data collected using twelve information channels coming from visual, acoustic, near-infrared, thermal, physiological and linguistic modalities. The data were collected from 45 subjects while being exposed to four different distractions (three cognitive and one physical). For the purposes of this paper, we experiment with visual and physiological information and explore the potential of multimodal modeling for distraction recognition. In addition, we analyze the value of different modalities by identifying specific visual and physiological groups of features that contribute the most to distraction characterization. Our results highlight the advantage of multimodal representations and reveal valuable insights for the role played by the two modalities on identifying different types of driving distractions.

Yiqun Yao, Michalis Papakostas, Mihai Burzo, Mohamed Abouelenien, Rada Mihalcea. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Smart wearable devices have lead to an increased need for processing and sharing large streams of physiological data in real-time. Modern Human-Machine Interaction (HMI) systems, especially applications designed for user training and assessment (e.g., educational or smart-rehabilitation systems), should be able to track and monitor those signals and adapt their parameters accordingly in order to optimally facilitate the special needs of each individual. Towards this end, we propose a passive Brain-Computer Interface (BCI), using a wireless non-intrusive EEG sensor under a robot assisted training task designed for cognitive assessment. As part of this ongoing work, we demonstrate our initial results on predicting user's task performance, from the EEG signals, before task completion. Our findings highlight the potentials of our hypotheses as we achieve a maximum accuracy rate equal to 74% when evaluated on 69 real subjects.

This paper proposes a novel system for assessing physical exercises specifically designed for cognitive behavior monitoring. The proposed system provides decision support to experts for helping with early childhood development. Our work is based on the well-established framework of Head-Toes-Knees-Shoulders (HTKS) that is known for its sufficient psychometric properties and its ability to assess cognitive dysfunctions. HTKS serves as a useful measure for behavioral self-regulation. Our system, CogniLearn, automates capturing and motion analysis of users performing the HTKS game and provides detailed evaluations using state-of-the-art computer vision and deep learning based techniques for activity recognition and evaluation. The proposed system is supported by an intuitive and specifically designed user interface that can help human experts to cross-validate and/or refine their diagnosis. To evaluate our system, we created a novel dataset, that we made open to the public to encourage further experimentation. The dataset consists of 15 subjects performing 4 different variations of the HTKS task and contains in total more than 60,000 RGB frames, of which 4,443 are fully annotated.

Training robots about the objects in their environment requires a multimodal correlation of features extracted from visual and linguistic sources. This work abstracts the task of collecting multimodal training data for object and feature learning by encapsulating it in an interactive game, I Spy, played between human players and robots. It introduces the concept of the game, briefly describes its methodology, and finally presents an evaluation of the game's performance and its appeal to human players.

Brand monitoring and reputation management are vital tasks in all modern business intelligence frameworks. However, recent related technologies rely mostly on the textual aspect of online content, in order to extract the underlying sentiment with respect to particular brands. In this work, we demonstrate the sentiment analysis method in the context of a brand monitoring framework, breaking the text-only barrier in the field. Towards this end, a wide range of visual features is extracted, some of which focus on the underlying semiotics and aesthetics of the images. In addition, we employ textual information embedded in the images under study, by adopting text mining techniques that focus on extracting sentiment. We evaluate the classification task for the particular binary task (negative vs positive sentiment) and propose a fusion approach that combines the two different modalities. Finally, the evaluation procedure has been carried out in the context of two different use cases, namely: (a) a general image sentiment classifier for brand and advertising images and (b) a brand-specific classification procedure, according to which the brand of the input images is known a-priori. Results have proven that the visual-based sentiment classification of brand and advertising information can outperform the respective text-based classification. In addition, fusing the two modalities leads to significant performance boosting.