Efficient video human activity recognition requires selecting relevant frames or segments of consecutive frames (clip) in a video, while also minimizing computational costs. Existing sampling methods are either deterministic and straight-forward, or complex and computationally expensive. In addition, some samplers often lack adaptability and fail to account for the uncertainty inherent in dynamic action sequences. To address these limitations, we present a probabilistic sampling strategy that balances adaptability and efficiency. Leveraging the Frobenius norm as a lightweight motion-change metric, our method assigns probabilistic importance scores to clips via softmax normalization and employs a stochastic sampling scheme based on the softmax scores to prioritize relevant segments. Unlike deterministic approaches, our method captures the dynamic and uncertainty of actions without the overhead of complex models. Experiments on UCF101, HMDB51 and Diving48 datasets validate that our method achieves competitive accuracy with significantly lower computational complexity.

In this work, we present an innovative approach utilizing ControlNet-based diffusion models along with upscaling capabilities for domain adaptation and quality refinement of 3D modelled synthetic datasets, focusing on autonomous vehicle applications. A significant domain gap often exists between synthetic and real-world data, hindering the applicability of deep learning models trained on synthetic data for real-world scenarios. Our methodology leverages the strengths of Controlled Augmentation by simultaneously utilizing multiple ControlNet signals, including edge detection, depth information, segmentation maps, and tile resampling. To improve how synthetic data aligns with the desired domain specifications, these signals guide the generative process, and we also incorporate text-guided prompts extracted via Large Language Models (LLMs), to improve control over the synthesis of desired features and attributes. We test the approach on diverse environmental conditions from the VKITTI dataset, a well-known 3D modelled synthetic dataset generated in Unity for autonomous driving research. The refined data is validated using quantitative metrics including FID, SSIM, and LPIPS, and is also evaluated on downstream machine learning tasks of object detection and classification, using YOLO-v8 to ensure its utility and effectiveness. Experimental analysis demonstrates the effectiveness of this method in improving the realism and usability of synthetic data. Our approach contributes to fields that require high-quality data synthesis and domain adaptation. The experimental work, along with ControlNet models used in this project is available online.1

As immersive 360° video experiences through head-mounted displays (HMDs) gain widespread adoption, the need for real-time, fine-grained assessment of Quality of Experience (QoE) becomes increasingly critical for optimising user engagement and system performance. This paper introduces RCQoEA-360VR, a novel multi-modal dataset designed for continuous QoE evaluation in virtual reality (VR) environments. In a controlled study (N=32), participants watched five selected 360° video sequences across eight different video quality configurations (from the VQEG database) using a Vive Pro Eye while providing continuous QoE annotations via a touchpad-based input method, enhanced by the DotMorph peripheral visualisation technique. The dataset also includes synchronised physiological signals (electrocardiogram and galvanic skin response), behavioural data (eye and head movements) and post-viewing QoE ratings gathered through a within-VR interface. RCQoEA-360VR addresses a critical gap in existing public datasets by providing a fine-grained, synchronised multimodal data for immersive QoE analysis. It offers a unique and valuable resource for the research community, supporting a wide range of research applications, including QoE prediction, behavioural modelling, adaptive streaming, and implicit perceptual analysis.

Considerable effort has been made in privacy-preserving video human activity recognition (HAR). Two primary approaches to ensure privacy preservation in Video HAR are differential privacy (DP) and visual privacy. Techniques enforcing DP during training provide strong theoretical privacy guarantees but offer limited capabilities for visual privacy assessment. Conversely, methods such as low-resolution transformations, data obfuscation and adversarial networks, emphasize visual privacy but lack clear theoretical privacy assurances. In this work, we focus on two main objectives: (1) leveraging DP properties to develop a model-free approach for visual privacy in videos and (2) evaluating our proposed technique using both differential privacy and visual privacy assessments on HAR tasks. To achieve goal (1), we introduce Video-DPRP: a Video-sample-wise Differentially Private Random Projection framework for privacy-preserved video reconstruction for HAR. By using random projections, noise matrices and right singular vectors derived from the singular value decomposition of videos, Video-DPRP reconstructs DP videos using privacy parameters ( $$\epsilon ,\delta $$ ) while enabling visual privacy assessment. For goal (2), using UCF101 and HMDB51 datasets, we compare Video-DPRP’s performance on activity recognition with traditional DP methods, and state-of-the-art (SOTA) visual privacy-preserving techniques. Additionally, we assess its effectiveness in preserving privacy-related attributes such as facial features, gender, and skin color, using the PA-HMDB and VISPR datasets. Video-DPRP combines privacy-preservation from both a DP and visual privacy perspective unlike SOTA methods that typically address only one of these aspects. The source code is publicly available on GitHub ( https://github.com/matzolla/Video-DPRP ).

The field of user quality of experience (QoE) in multimedia communications has become increasingly important due to the widespread use of digital technology in our everyday lives. The ability to accurately predict user QoE by processing physiological signals has significant applications. This study proposes a machine learning (ML) approach for predicting user QoE using physiological signals. It focuses on perceived overall quality and perceived audio quality by processing electrocardiogram (ECG) and respiration signals from the SoPMD Dataset 2. The study evaluated various ML models on individual and fused modalities while implementing dimensionality reduction, feature selection, and hyperparameter tuning. The SHapley Additive exPlanations (SHAP) method was used to interpret the ML models' outputs, which helped identify the features providing the most utility. The results show that the random forest model provided the best performance, with an F1-score of 87.91% for fusion data and 80.49% for ECG data in classifying perceived audio quality and overall quality, respectively. These results imply that physiological signals, such as ECG and respiration, hold great potential for predicting user QoE in multimedia experiences.

Taking inspiration from recent developments in visual generative tasks using diffusion models, we propose a method for end-to-end speech-driven video editing using a denoising diffusion model. Given a video of a talking person, and a separate auditory speech recording, the lip and jaw motions are re-synchronised without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. Proof of concept results are demonstrated on both single-speaker and multi-speaker video editing, providing a baseline model on the CREMA-D audiovisual data set. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing. All code, datasets, and models used as part of this work are made publicly available here: https://danbigioi.github.io/DiffusionVideoEditing/.

Event cameras also known as neuromorphic sensors are relatively a new technology with some privilege over the RGB cameras. The most important one is their difference in capturing the light changes in the environment, each pixel changes independently from the others when it captures a change in the environment light. To increase the user’s degree of freedom in controlling the output of these cameras, such as changing the sensitivity of the sensor to light changes, controlling the number of generated events and other similar operations, the camera manufacturers usually introduce some tools to make sensor level changes in camera settings. The contribution of this research is to examine and document the effects of changing the sensor settings on the sharpness as an indicator of quality of the generated stream of event data. To have a qualitative understanding this stream of event is converted to frames, then the average image gradient magnitude as an index of the number of edges and accordingly sharpness is calculated for these frames. Five different bias settings are explained and the effect of their change in the event output is surveyed and analyzed. In addition, the operation of the event camera sensing array is explained with an analogue circuit model and the functions of the bias foundations are linked with this model.

In this research work, we have proposed a thermal tiny-YOLO multi-class object detection (TTYMOD) system as a smart forward sensing system that should remain effective in all weather and harsh environmental conditions using an end-to-end YOLO deep learning framework. It provides enhanced safety and improved awareness features for driver assistance. The system is trained on large-scale thermal public datasets as well as newly gathered novel open-sourced dataset comprising of more than 35,000 distinct thermal frames. For optimal training and convergence of YOLO-v5 tiny network variant on thermal data, we have employed different optimizers which include stochastic decent gradient (SGD), Adam, and its variant AdamW which has an improved implementation of weight decay. The performance of thermally tuned tiny architecture is further evaluated on the public as well as locally gathered test data in diversified and challenging weather and environmental conditions. The efficacy of a thermally tuned nano network is quantified using various qualitative metrics which include mean average precision, frames per second rate, and average inference time. Experimental outcomes show that the network achieved the best mAP of 56.4% with an average inference time/ frame of 4 milliseconds. The study further incorporates the optimization of tiny network variant using the TensorFlow Lite quantization tool which is beneficial for the deployment of deep learning architectures on the edge and mobile devices. For this study, we have used a raspberry pi 4 computing board for evaluating the real-time feasibility performance of an optimized version of the thermal object detection network for the automotive sensor suite. The source code, trained and optimized models and complete validation/ testing results are publicly available at https://github.com/MAli-Farooq/Thermal-YOLO-And-Model-Optimization-Using-TensorFlowLite.

Driver monitoring systems (DMS) are a key component of vehicular safety and essential for the transition from semiautonomous to fully autonomous driving. A key task for DMS is to ascertain the cognitive state of a driver and to determine their level of tiredness. Neuromorphic vision systems, based on event camera technology, provide advanced sensing of facial characteristics, in particular the behavior of a driver’s eyes. This research explores the potential to extend neuromorphic sensing techniques to analyze the entire facial region, detecting yawning behaviors that give a complimentary indicator of tiredness. A neuromorphic dataset is constructed from 952 video clips (481 yawns, 471 not-yawns) captured with an RGB colour camera, with 37 subjects. A total of 95,200 neuromorphic image frames are generated from this video data using a video-to-event converter. From these data 21 subjects were selected to provide a training dataset, 8 subjects were used for validation data, and the remaining 8 subjects were reserved for an ‘unseen’ test dataset. An additional 12,300 frames were generated from event simulations of a public dataset to test against other methods. A convolutional neural network (CNN) with self-attention and a recurrent head was trained and tested with these data. Respective precision and recall scores of 95.9% and 94.7% were achieved on our test set, and 89.9% and 91% on the simulated public test set, demonstrating the feasibility to add yawn detection as a sensing component of a neuromorphic DMS.

Heart rate is one of the most vital health metrics which can be utilized to investigate and gain intuitions into various human physiological and psychological information.Estimating heart rate without the constraints of contact-based sensors thus presents itself as a very attractive field of research as it enables well-being monitoring in a wider variety of scenarios.Consequently, various techniques for camera-based heart rate estimation have been developed ranging from classical image processing to convoluted deep learning models and architectures.At the heart of such research efforts lies health and visual data acquisition, cleaning, transformation, and annotation.In this paper, we discuss how to prepare data for the task of developing or testing an algorithm or machine learning model for heart rate estimation from images of facial regions.The data prepared is to include camera frames as well as sensor readings from an electrocardiograph sensor.The proposed pipeline is divided into four main steps, namely removal of faulty data, frame and electrocardiograph timestamp dejittering, signal denoising and filtering, and frame annotation creation.Our main contributions are a novel technique of eliminating jitter from health sensor and camera timestamps and a method to accurately time align both visual frame and electrocardiogram sensor data which is also applicable to other sensor types.

Neuromorphic vision or event vision is an advanced vision technology, where in contrast to visible camera sensors that output pixels, the event vision generates neuromorphic events every time there’s a brightness change which exceeds a specific threshold in the field of view (FoV). This study focuses on leveraging neuromorphic event data for roadside object detection. This is a proof of concept towards building artificial intelligence (AI) based imaging pipelines which can be used for forward perception systems for advanced vehicular applications. The focus is on building efficient stateof- the-art object detection networks with better inference results for fast-moving forward perception using an event camera. In this article, the event simulated A2D2 dataset is manually annotated and trained on two different YOLOv5 networks (small and large variants). To further assess its robustness, single model testing and ensemble model testing are carried out.

Yoga can be seen as a set of fitness exercises involving various body postures. Most of the available pose and action recognition datasets are comprised of easy-to-moderate body pose orientations and do not offer much challenge to the learning algorithms in terms of the complexity of pose. In order to observe action recognition from a different perspective, we introduce YogaTube, a new large-scale video benchmark dataset for yoga action recognition. YogaTube aims at covering a wide range of complex yoga postures, which consist of 5484 videos belonging to a taxonomy of 82 classes of yoga asanas. Also, a three-stream architecture has been designed for yoga asanas pose recognition using two modules, feature extraction, and classification. Feature extraction comprises three parallel components. First, pose is estimated using the part affinity fields model to extract meaningful cues from the practitioner. Second, optical flow is used to extract temporal features. Third, raw RGB videos are used for extracting the spatiotemporal features. Finally in the classification module, pose, optical flow, and RGB streams are fused to get the final results of the yoga asanas. To the best of our knowledge, this is the first attempt to establish a video benchmark yoga recognition dataset. The code and dataset will be released soon.

This paper contains the research proposal of Sowmya Vijayakumar that was presented at the MMSys 2022 doctorial symposium. Multimedia applications can now be found across many application domains including but not limited to entertainment, communication, health, business, and education. It is becoming more and more important to understand the factors that influence user perceptual quality, and hence monitoring user quality of experience (QoE) for improving multimedia interaction and services is essential. In this PhD work, we propose advanced machine learning techniques to predict QoE from physiological signals for immersive multimedia experiences. The aim of this doctoral study is to investigate the utility of physiological responses for QoE assessment for different multimedia technologies. Here, the research questions and solutions proposed to address this challenge are presented. A multimodal QoE prediction model is being developed that integrates several physiological measurements to improve QoE prediction performance.

This paper presents an evaluation of a deep learning (DL) model to predict the user quality of experience (QoE) from physiological signals using a publicly available multimodal dataset, SoPMD. The subjective scores related to QoE factors, namely, perceptual video quality, immersion level, surrounding awareness, interest in video content and audio content are evaluated. A DL model, Bidirectional Long-short-term memory (BiLSTM), is trained on the fusion of electrocardiogram (ECG) and respiration features to predict subjective scores for the five QoE factors. This study achieved classification accuracies and Fl-scores ranging between 58% and 67% for different QoE factors. The results of the BiLSTM model were compared with machine learning techniques. The experimental results demonstrated that the proposed BiLSTM network has the potential to predict QoE from physiological signals.

Depth estimation from a single image frame is a fundamental challenge in computer vision, with many applications such as augmented reality, action recognition, image understanding, and autonomous driving. Large and diverse training sets are required for accurate depth estimation from a single image frame. Due to challenges in obtaining dense ground-truth depth, a new 3D pipeline of 100 synthetic virtual human models is presented to generate multiple 2D facial images and corresponding ground truth depth data, allowing complete control over image variations. To validate the synthetic facial depth data, we propose an evaluation of state-of-the-art depth estimation algorithms based on single image frames on the generated synthetic dataset. Furthermore, an improved encoder-decoder based neural network is presented. This network is computationally efficient and shows better performance than current state-of-the-art when tested and evaluated across 4 public datasets. Our training methodology relies on the use of synthetic data samples which provides a more reliable ground truth for depth estimation. Additionally, using a combination of appropriate loss functions leads to improved performance than the current state-of-the-art network performances. Our approach clearly outperforms competing methods across different test datasets, setting a new state-of-the-art for facial depth estimation from synthetic data.

Methods for generating synthetic data have become of increasing importance to build large datasets required for Convolution Neural Networks (CNN) based deep learning techniques for a wide range of computer vision applications. In this work, we extend existing methodologies to show how 2D thermal facial data can be mapped to provide 3D facial models. For the proposed research work we have used tufts datasets for generating 3D varying face poses by using a single frontal face pose. The system works by refining the existing image quality by performing fusion based image preprocessing operations. The refined outputs have better contrast adjustments, decreased noise level and higher exposedness of the dark regions. It makes the facial landmarks and temperature patterns on the human face more discernible and visible when compared to original raw data. Different image quality metrics are used to compare the refined version of images with original images. In the next phase of the proposed study, the refined version of images is used to create 3D facial geometry structures by using Convolution Neural Networks (CNN). The generated outputs are then imported in blender software to finally extract the 3D thermal facial outputs of both males and females. The same technique is also used on our thermal face data acquired using prototype thermal camera (developed under Heliaus EU project) in an indoor lab environment which is then used for generating synthetic 3D face data along with varying yaw face angles and lastly facial depth map is generated.

In recent years, large “in the wild” face datasets have been released in an attempt to facilitate progress in tasks such as face detection, face recognition, and other tasks. Most of these datasets are acquired from webpages with automatic procedures. As a consequence, noisy data are often found. Furthermore, in these large face datasets, the annotation of identities is important as they are used for training face recognition algorithms. But due to the automatic way of gathering these datasets and due to their large size, many identities folder contain mislabeled samples which deteriorates the quality of the datasets. In this work, it is presented a semi-automatic method for cleaning the noisy large face datasets with the use of face recognition. This methodology is applied to clean the CelebA dataset and show its effectiveness. Furthermore, the list with the mislabelled samples in the CelebA dataset is made available.

There is a need to standardize the performance of eye gaze estimation (EGE) methods in various platforms for human computer interaction (HCI). Because of lack of consistent schemes or protocols for summative evaluation of EGE systems, performance results in this field can neither be compared nor reproduced with any consistency. In contemporary literature, gaze tracking accuracy is measured under non-identical sets of conditions, with variable metrics and most results do not report the impact of system meta-parameters that significantly affect tracking performances. In this work, the diverse nature of these research outcomes and system parameters which affect gaze tracking in different platforms is investigated and their error contributions are estimated quantitatively. Then the concept and development of a performance evaluation framework is proposed- that can define design criteria and benchmark quality measures for the eye gaze research community.

The potential synergies between consumer handheld devices, particularly smartphones and biometric technologies is outlines. The practicalities and challenges for three such technologies - fingerprint, iris and palmprint - are presented. The use of biometrics for personal authentication is discussed, including the use of zero knowledge proof techniques to ensure that the biometric data does not leave the phone. The scope for data theft and breach through spoofing of the original biometric are discussed. Finally the potential impact of this technology synergy on personal privacy is considered.

In a world where smart phones are becoming ubiquitous and are more powerful than ever, security is becoming an issue. Equipped with good cameras, smart phones are prime candidates for software designed to recognize objects and people around them. By recognizing biometric patterns such as the face, palm-prints or hand knuckles, smart phones can heighten security standards. Depending on the intended action, specific security requirements should be met. For instance, bank services need a reliable authentication system, whereas unlocking a device can use a lower security feature. For this perspective we are working on a database of palm-prints that have only been acquired using smart phones to verify their users' identity and also designing an authentication system based on these results to be used on smartphones.

This article outlines various technical, social and ethical challenges in implementing and widely adopting iris recognition technology on consumer devices such as smartphone or tablets. Acquisition of sufficient quality iris images using today's consumer devices is noted to be the main challenge in implementing this technology. Current progress in this field is reviewed. A smartphone form factor camera is presented to be used as a front-facing camera. This device is modified to capture near infra-red iris images along with general purpose visible wavelength images. Analyses shows that such a device with improved optics and sensor could be used for implementing iris recognition in next generation hand held devices. The social impact of wider adoption of this technology is discussed. Iris pattern obfuscation is presented to address various security and privacy concerns which may arise when iris recognition will be a part of our daily life.

As the imaging systems in handheld devices continue to improve, both in terms of optical quality and the use of advanced computational imaging techniques, we are close to a point where high quality iris images can be obtained from personal images, which in turn can be used for spoofing attacks against iris recognition system. Thus an emerging challenge for next-generation personal imaging devices is to provide a means to obfuscate iris pattern in digital photographs and videos, but without destroying the photo-realistic qualities of the eye regions in a photograph or video. This will effectively reduce the chance of obtaining iris patterns easily, which can be later used for spoofing. In this paper we propose five different techniques for iris pattern obfuscation and perform some initial testing to evaluate which are more robust. In addition some representative samples are provided of the visible effects on the appearance of eye regions.

The extension of the standard grayscale active appearance model (AAM) techniques to color images is investigated. Prior work in this field has mainly focused on RGB color models which did not demonstrate noticeable benefits over grayscale models from the point of view of convergence accuracy. We improve on previous work by normalizing the color texture vector separately for intensity and chromaticity components. Where an appropriate color space is chosen, we demonstrate improvements in convergence accuracy as well as image synthesization quality for AAMs. Optimal results are achieved when a color space in which the image channels are strongly decorrelated is chosen. Our best results are achieved using the I1I2I3 color space, originally proposed by Ohta.