Omnidirectional videos (ODVs) are redefining viewer experiences in virtual reality (VR) by offering an unprecedented full field-of-view (FOV). This study extends the domain of saliency prediction to 360$^\circ$∘ environments, addressing the complexities of spherical distortion and the integration of spatial audio. Contextually, ODVs have transformed user experience by adding a spatial audio dimension that aligns sound direction with the viewer's perspective in spherical scenes. Motivated by the lack of comprehensive datasets for 360$^\circ$∘ audio-visual saliency prediction, our study curates YT360-EyeTracking, a new dataset of 81 ODVs, each observed under varying audio-visual conditions. Our goal is to explore how to utilize audio-visual cues to effectively predict visual saliency in 360$^\circ$∘ videos. Towards this aim, we propose two novel saliency prediction models: SalViT360, a vision-transformer-based framework for ODVs equipped with spherical geometry-aware spatio-temporal attention layers, and SalViT360-AV, which further incorporates transformer adapters conditioned on audio input. Our results on a number of benchmark datasets, including our YT360-EyeTracking, demonstrate that SalViT360 and SalViT360-AV significantly outperform existing methods in predicting viewer attention in 360$^\circ$∘ scenes. Interpreting these results, we suggest that integrating spatial audio cues in the model architecture is crucial for accurate saliency prediction in omnidirectional videos.

Abstract LiDAR odometry estimation and 3D semantic segmentation are crucial for autonomous driving, which has achieved remarkable advances recently. However, these tasks are challenging due to the imbalance of points in different semantic categories for 3D semantic segmentation and the influence of dynamic objects for LiDAR odometry estimation, which increases the importance of using representative/salient landmarks as reference points for robust feature learning. To address these challenges, we propose a saliency-guided approach that leverages attention information to improve the performance of LiDAR odometry estimation and semantic segmentation models. Unlike in the image domain, only a few studies have addressed point cloud saliency information due to the lack of annotated training data. To alleviate this, we first present a universal framework to transfer saliency distribution knowledge from color images to point clouds, and use this to construct a pseudo-saliency dataset (i.e. FordSaliency) for point clouds. Then, we adopt point cloud based backbones to learn saliency distribution from pseudo-saliency labels, which is followed by our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance. Finally, we introduce SalLONet, a self-supervised saliency-guided LiDAR odometry network that uses the semantic and saliency predictions of SalLiDAR to achieve better odometry estimation. Our extensive experiments on benchmark datasets demonstrate that the proposed SalLiDAR and SalLONet models achieve state-of-the-art performance against existing methods, highlighting the effectiveness of image-to-LiDAR saliency knowledge transfer. Source code will be available at https://github.com/nevrez/SalLONet

Omnidirectional images, aka 360° images, can deliver immersive and interactive visual experiences. As their popularity has increased dramatically in recent years, evaluating the quality of 360° images has become a problem of interest since it provides insights for capturing, transmitting, and consuming this new media. However, directly adapting quality assessment methods proposed for standard natural images for omnidirectional data poses certain challenges. These models need to deal with very high-resolution data and implicit distortions due to the spherical form of the images. In this study, we present a method for no-reference 360° image quality assessment. Our proposed ST360IQ model extracts tangent viewports from the salient parts of the input omnidirectional image and employs a vision-transformers based module processing saliency selective patches/tokens that estimates a quality score from each viewport. Then, it aggregates these scores to give a final quality score. Our experiments on two benchmark datasets, namely OIQA and CVIQ datasets, demonstrate that as compared to the state-of-the-art, our approach predicts the quality of an omnidirectional image correlated with the human-perceived image quality. The code has been available on https://github.com/Nafiseh-Tofighi/ST360IQ

The electrical property (EP) of human tissues is a quantitative biomarker that facilitates early diagnosis of cancerous tissues. Magnetic resonance electrical properties tomography (MREPT) is an imaging modality that reconstructs EPs by the radio-frequency field in an MRI system. MREPT reconstructs EPs by solving analytic models numerically based on Maxwell's equations. Most MREPT methods suffer from artifacts caused by inaccuracy of the hypotheses behind the models, and/or numerical errors. These artifacts can be mitigated by adding coefficients to stabilize the models, however, the selection of such coefficient has been empirical, which limit its medical application. Alternatively, end-to-end Neural networks-based MREPT (NN-MREPT) learns to reconstruct the EPs from training samples, circumventing Maxwell's equations. However, due to its pattern-matching nature, it is difficult for NN-MREPT to produce accurate reconstructions for new samples. In this work, we proposed a physics-coupled NN for MREPT (PCNN-MREPT), in which an analytic model, cr-MREPT, works with diffusion and convection coefficients, learned by NNs from the difference between the reconstructed and ground-truth EPs to reduce artifacts. With two simulated datasets, three generalization experiments in which test samples deviate gradually from the training samples, and one noise-robustness experiment were conducted. The results show that the proposed PCNN-MREPT achieves higher accuracy than two representative analytic methods. Moreover, compared with an end-to-end NN-MREPT, the proposed method attained higher accuracy in two critical generalization tests. This is an important step to practical MREPT medical diagnoses.

Since the huge earth quake in 2011 in Japan, the Japanese government has strongly encouraged growth of renewable energy use. As a result, even we focus on only mega-solar photovoltaic (PV) power plants (> 1,000 kW), the amount of the power generation significantly increased. Because such rapid increasing of the PV power plants changes land use widely, verification of the spreading of the PV power plants should be essential for assessing economic/environmental issues. In this study, to verify the increasing of the PV power plants based on observations, we applied a method that can detect the PV power plants efficiently with a convolutional neural network to Landsat-8 images. From the detection, we successfully identified the increasing of the area of the PV power plants quantitatively at the same time identifying the location of each PV power plant in many prefectures in the capital region of Japan.

Various saliency detection algorithms from color images have been proposed to mimic eye fixation or attentive object detection response of human observers for the same scenes. However, developments on hyperspectral imaging systems enable us to obtain redundant spectral information of the observed scenes from the reflected light source from objects. A few studies using low-level features on hyper-spectral images demonstrated that salient object detection can be achieved. In this work, we proposed a salient object detection model on hyperspectral images by applying manifold ranking (MR) on self-supervised Convolutional Neural Network (CNN) features (high-level features) from unsupervised image segmentation task. Self-supervision of CNN continues until clustering loss or saliency maps converges to a defined error between each iteration. Finally, saliency estimations is done as the saliency map at last iteration when the self-supervision procedure terminates with convergence. Experimental evaluations demonstrated that proposed saliency detection algorithm on hyperspectral images is outperforming state-of-the-arts hyperspectral saliency models including the original MR based saliency model.

Water/Ice/Land region classification is an important remote sensing tasks, which analyze the occurrence of water, ice on the earth surface. Common remote sensing practices such as thresholding, spectral analysis, and statistical approaches generally do not produce globally reliable classification results. Even the robust deep learning models do not perform enough due to the limitation of ground truth available for training and the medium resolution of the Open satellite images. Therefore, in this research, we used a relatively easy method to generate ground truth for randomly selected locations around the globe. Then, we utilized a simplified variant of well-known UNet deep convolutional neural network (CNN) structure with a dilated CNN layers, skip connections and without any max-pooling layers. The proposed model shows better performance in medium resolution satellite images (Landsat-8) compared to state-of-the-art models such as UNet and DeepWaterMap applied on the same task.

Many works have been done on salient object detection using supervised or unsupervised approaches on colour images. Recently, a few studies demonstrated that efficient salient object detection can also be implemented by using spectral features in visible spectrum of hyperspectral images from natural scenes. However, these models on hyperspectral salient object detection were tested with a very few number of data selected from various online public dataset, which are not specifically created for object detection purposes. Therefore, here, we aim to contribute to the field by releasing a hyperspectral salient object detection dataset with a collection of 60 hyperspectral images with their respective ground-truth binary images and representative rendered colour images (sRGB). We took several aspects in consideration during the data collection such as variation in object size, number of objects, foreground-background contrast, object position on the image, and etc. Then, we prepared ground truth binary images for each hyperspectral data, where salient objects are labelled on the images. Finally, we did performance evaluation using Area Under Curve (AUC) metric on some existing hyperspectral saliency detection models in literature.

As prior knowledge of objects or object features helps us make relations for similar objects on attentional tasks, pre-trained deep convolutional neural networks (CNNs) can be used to detect salient objects on images regardless of the object class is in the network knowledge or not. In this paper, we propose a top-down saliency model using CNN, a weakly supervised CNN model trained for 1000 object labelling task from RGB images. The model detects attentive regions based on their objectness scores predicted by selected features from CNNs. To estimate the salient objects effectively, we combine both forward and backward features, while demonstrating that partially-guided backpropagation will provide sufficient information for selecting the features from forward run of CNN model. Finally, these top-down cues are enhanced with a state-of-the-art bottom-up model as complementing the overall saliency. As the proposed model is an effective integration of forward and backward cues through objectness without any supervision or regression to ground truth data, it gives promising results compared to state-of-the-art models in two different datasets.

We demonstrate how to utilize a two-dimensional image sensor onboard a satellite for determining its attitude. The method is based on image matching between satellite images and Earth surfaces. To achieve this, image feature extraction and robust estimation techniques are employed. Experimental results showed that the accuracy of attitude determination is about 0.02° if the satellite position has been precisely determined. The contribution of this paper is not on technical novelty, as the method has already been described in [1], but on additional examples illustrating the utility of the method.

Studies on the Human Visual System (HVS) have demonstrated that human eyes are more attentive to spatial or spectral components with irregularities on the scene. This fact was modeled differently in many computational methods such as saliency residual (SR) approach, which tries to find the irregularity of frequency components by subtracting average filtered and original amplitude spectra. However, studies showed that the high frequency components have more effect on the HVS perception. In this paper, we propose a 2D mel-cepstrum based spectral residual saliency detection model (MCSR) to provide perceptually meaningful and more informative saliency map with less redundancy and without down-sampling as in saliency residual approach. Experimental results demonstrate that proposed MCSR model can yield promising results compared to the relevant state of the art models.

Researchers have been taking advantage of visual attention in various image processing applications such as image retargeting, video coding, etc. Recently, many saliency detection algorithms have been proposed by extracting features in spatial or transform domains. In this paper, a novel saliency detection model is introduced by utilizing low-level features obtained from the wavelet transform domain. Firstly, wavelet transform is employed to create the multi-scale feature maps which can represent different features from edge to texture. Then, we propose a computational model for the saliency map from these features. The proposed model aims to modulate local contrast at a location with its global saliency computed based on the likelihood of the features, and the proposed model considers local center-surround differences and global contrast in the final saliency map. Experimental evaluation depicts the promising results from the proposed model by outperforming the relevant state of the art saliency detection models.