In this paper, we introduce a conditional transformer-based U-net architecture for speech emotion recognition (SER). The proposed architecture consists of convolutional transformer (CTR)-based encoder, feature decoder and emotion decoder. The CTR-based U-net encoder is designed to extract compressed bottleneck features for recognition. The emotion is then predicted by the CTR-based emotion decoder, where we use additional trainable emotion query to better capture the characteristics of different emotional states. The CTR-based feature decoder reconstructs the given input features from the bottleneck features. This auxiliary decoder allows us to use additional information while training the model, which further improves the recognition performance. Specifically, the proposed feature decoder is performed through sophisticated embedding of the predicted emotional states and the features of the encoder layers. In the proposed framework, we consider the channel attention for each time frame in the CTRs to better enable a real-time processing of emotion recognition. Experimental results showed that the proposed emotion classification method provided better recognition performance than the selected benchmark algorithms.

Robustness remains a fundamental challenge in real-world speech recognition, particularly under adverse acoustic conditions. In this paper, we introduce a robust speech emotion recognition (SER) approach using a conditional transformer-based architecture, specifically designed to maintain consistent performance across a wide range of noisy acoustic environments. The proposed convolutional transformer (CTr)-based architecture consists of three primary modules: a noise-classification module (NCM), a speech feature enhancement module (SFEM) and a speech emotion recognition module (SERM). The NCM first identifies the noise type presented in the given noisy speech features, producing embeddings that condition both the SFEM and SERM to better capture the characteristics of different noise types. The front-end SFEM enhances noisy speech features to improve the robustness of the SERM in adverse acoustic conditions and finally, the SERM predicts the emotional states. Experimental results show that that the proposed architecture performs robust under various noisy acoustic environment and provide better results than the selected benchmark methods.

We present a two-stage mechanism for generic image restoration in natural driving conditions, where multiple non-linear degradations simultaneously impact perception for humans and driving assistance systems. Our approach overcomes the limitations of utilizing a single neural network that incurs excessive computational overhead and yields sub-optimal recovery. The proposed first stage comprises computationally inexpensive image processing operations applied at a patch level using a lightweight convolutional neural network (CNN) that determines their intensity of operation. This patch size is guided by the receptive field of the CNN, allowing for dynamic restoration of non-linear and non-homogeneous degradation profiles. The second stage leverages a lightweight end-to-end neural network functioning as an inpainting network. It identifies inadequately restored regions and leverages global semantic and structural information to fill the affected areas. This approach enhances the restoration process by considering the entire image and addresses the remainder of localized deficiencies. In addition, we integrate dense perception tasks such as semantic and depth estimation during the optimization cycle to ensure restored images that are perceptually pleasing and conducive for downstream perception tasks. Since datasets covering diverse degradation scenarios for high- and low-level perception tasks are lacking, we utilize a synthetic data augmentation technique to generate non-homogeneous non-linear degradation profiles. Experiments on images captured in adverse weather conditions demonstrate the efficacy of our approach, yielding higher perceptual quality in restored images and improved performance in downstream perception tasks under adverse driving conditions. Importantly, our method offers computational efficiency compared to end-to-end image restoration algorithms, making it suitable for real-time applications.

In this work, we examine the potential application of thermal cameras in improving perception capabilities in adverse weather conditions like snow, night-time driving, and haze, focusing on retaining the performance of Advanced Driver Assistance Systems (ADAS), thus enhancing its functionality and safety characteristics. While thermal sensors offer the advantage of robust information capture in adverse weather conditions, their integration is plagued with issues surrounding poor feature capture in normal conditions, low imaging resolution, and high sensor costs. We address the former by formulating the problem definition as information switching wherein thermal images are selected when visible images are degraded. Furthermore, we consider a single object detector for RGB and thermal images to ensure low latency. We propose utilizing a learnable projection function that translates the thermal image into RGB color space, thus providing minimal modifications to the underlying object detector. We address the issues of low imaging resolution and cost by proposing a novel procedure that combines super-resolution and object detection, enabling the utilization of low-resolution and low-cost uncooled thermal imaging sensors. To ensure the complete pipeline meets the actual deployment requirements of real-time inference on resource-constrained devices, we introduce a lightweight super-resolution algorithm, implementing optimizations within the network structure followed by global pruning. In addition, to improve the feature representations extracted by lightweight encoders, we propose a bidirectional feature pyramid network to enhance the feature representation. We demonstrate the efficacy of the proposed mechanism through extensive simulated evaluations on automotive datasets such as FLIR, KAIST, DENSE, and Freiburg Thermal.

Visual information plays an important part in the perception of the world around us. Recently, head-mounted displays (HMD) came to the consumer market and became a part of everyday life of thousands of people. Like with the desktop screens or hand-held devices before, the public is concerned with the possible health consequences of the prolonged usage and question the adequacy of the default settings. It has been shown that the brightness and contrast of a display should be adjusted to match the external light to decrease eye strain and other symptoms. Currently, there is a noticeable mismatch in brightness between the screen and dark background of an HMD that might cause eye strain, insomnia, and other unpleasant symptoms. In this paper, we explore the possibility to significantly lower the screen brightness in the HMD and successfully compensate for the loss of the visual information on a dimmed screen. We designed a user study to explore the connection between the screen brightness in HMD and task performance, cybersickness, users' comfort, and preferences. We have tested three levels of brightness: the default Full Brightness, the optional Night Mode and a significantly lower brightness with original content and compensated content. Our results suggest that although users still prefer the brighter setting, the HMDs can be successfully used with significantly lower screen brightness, especially if the low screen brightness is compensated.