Scanpath comparison metrics universally represent fixations as dimensionless (x, y) points, despite the human fovea subtending approximately 1° ∼ 2° of visual angle. This mismatch penalises spatially proximate fixations that process identical visual content. We propose FDISS (Foveal Disc IoU Scanpath Score), a metric that models each fixation as a foveal disc and computes similarity via bidirectional nearest-neighbour IoU matching, yielding interpretable Precision, Recall, and F1 scores. FDISS introduces a biologically grounded perceptual tolerance zone absent from all existing metrics and requires no free parameters at runtime.

Scanpath similarity metrics are central to eye-movement research, yet existing methods predominantly evaluate spatial and temporal alignment while neglecting semantic equivalence between attended image regions. We present a semantic scanpath similarity framework that integrates vision-language models (VLMs) into eye-tracking analysis. Each fixation is encoded under controlled visual context (patch-based and marker-based strategies) and transformed into concise textual descriptions, which are aggregated into scanpath-level representations. Semantic similarity is then computed using embedding-based and lexical NLP metrics and compared against established spatial measures, including MultiMatch and DTW. Experiments on free-viewing eye-tracking data demonstrate that semantic similarity captures partially independent variance from geometric alignment, revealing cases of high content agreement despite spatial divergence. We further analyze the impact of contextual encoding on description fidelity and metric stability. Our findings suggest that multimodal foundation models enable interpretable, content-aware extensions of classical scanpath analysis, providing a complementary dimension for gaze research within the ETRA community.

We introduce GazeVaLM, a public eye-tracking dataset for studying clinical perception during chest radiograph authenticity assessment. The dataset comprises 960 gaze recordings from 16 expert radiologists interpreting 30 real and 30 synthetic chest X-rays (generated by diffusion based generative AI) under two conditions: diagnostic assessment and real-fake classification (Visual Turing test). For each image–observer pair, we provide raw gaze samples, fixation maps, scanpaths, saliency density maps, structured diagnostic labels, and authenticity judgments. We extend the protocol to 6 state-of-the-art multimodal LLMs, releasing their predicted diagnoses, authenticity labels, and confidence scores under matched conditions — enabling direct human–AI comparison at both decision and uncertainty levels. We further provide analyses of gaze agreement, inter-observer consistency, and benchmarking of radiologists versus LLMs in diagnostic accuracy and authenticity detection. GazeVaLM supports research in gaze modeling, clinical decision-making, human–AI comparison, generative image realism assessment, and uncertainty quantification. By jointly releasing visual attention data, clinical labels, and model predictions, we aim to facilitate reproducible research on how experts and AI systems perceive, interpret, and evaluate medical images. The dataset is available at https://huggingface.co/datasets/davidcwong/GazeVaLM.

Knee osteoarthritis (KOA) is a prevalent musculoskeletal disorder, often diagnosed using X-rays due to its cost-effectiveness. While Magnetic Resonance Imaging (MRI) provides superior soft tissue visualization and serves as a valuable supplementary diagnostic tool, its high cost and limited accessibility significantly restrict its widespread use. To explore the feasibility of bridging this imaging gap, we conducted a feasibility study leveraging a diffusion-based model that uses an X-ray image as conditional input, alongside target depth and additional patient-specific feature information, to generate corresponding MRI sequences. Our findings demonstrate that the MRI volumes generated by our approach are not only visually closer to real MRI scans compared with other methods but also achieve the highest quantitative performance in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Furthermore, by increasing the number of inference steps to interpolate between slice depths, we enhance the continuity of the generated volume, achieving higher adjacent slice correlation coefficients. Through ablation studies, we further validate that integrating supplemental patient-specific information, beyond what X-rays alone can provide, enhances the accuracy and clinical relevance of the generated MRI, which underscores the potential of leveraging external patient-specific information to improve the performance of the MRI generation.

Enhancing video quality assessment (VQA) through semantic information integration is a critical research focus. Recent research has employed the Contrastive Language-Image Pre-training (CLIP) model as a foundation to improve semantic perception. However, the image-text alignment inherent in these pre-trained Vision-Language (VL) models frequently results in suboptimal VQA performance. While prompt engineering has recently targeted the language component to address this alignment issue, the unique insights resided in visual analysis is still overlooked for further advancing VQA tasks. Additionally, seeking a trade-off between quality separability and domain invariance in VQA remains largely unresolved within the VL paradigm. In this paper, we introduce a novel cross-modal prompt-based approach to tackle these challenges. Specifically, we propose learnable prompts within the vision branch to foster synergy between visual and language modalities through a language-to-vision coupling function. The multi-view backbone is then carefully crafted with content enhancement and distortion-aware temporal modulation to ensure quality separability. The language prompts, derived from visual representations, are further supported by adaptive weighting mechanisms to optimize the balance between quality separability and domain invariance. Experimental results demonstrate the effectiveness of our proposed method over leading VQA models, showing significant improvements in generalization across diverse datasets. The source code for this work is publicly available athttps://github.com/cpf0079/CM2PL.

In this paper, we introduce a benchmark dataset for the diagnosis and treatment planning of Class III malocclusion, a condition that requires precise evaluation to determine the necessity of surgical intervention. Our dataset comprises paired lateral cephalometric X-rays and profile photographs, each annotated with a treatment plan mainly indicating whether surgery is required or not. We assess state-of-the-art deep learning models on both imaging modalities to explore their potential for automating diagnosis and treatment decisions. Notably, the dataset facilitates research into non-radiographic diagnostic approaches, potentially enabling treatment planning based solely on profile photographs. We release this dataset as a resource to advance automated applications in orthodontics and maxillofacial surgery.

Abstract Quality assessment is a key element for the evaluation of hardware and software involved in image and video acquisition, processing, and visualization. In the medical field, user-based quality assessment is still considered more reliable than objective methods, which allow the implementation of automated and more efficient solutions. Regardless of increasing research on this topic in the last decade, defining quality standards for medical content remains a non-trivial task, as the focus should be on the diagnostic value assessed by expert viewers rather than the perceived quality from naïve viewers, and objective quality metrics should aim at estimating the first rather than the latter. In this paper, we present a survey of methodologies used for the objective quality assessment of medical images and videos, dividing them into visual quality-based and task-based approaches. Visual quality-based methods compute a quality index directly from visual attributes, while task-based methods, being increasingly explored, measure the impact of quality impairments on the performance of a specific task. A discussion on the limitations of state-of-the-art research on this topic is also provided, along with future challenges to be addressed.

Knee osteoarthritis (OA) is a prevalent and debilitating condition, significantly impacting quality of life and mobility. Traditional 3D image classification methods often falter in effectively diagnosing this complex condition, primarily due to their inability to capture and analyze the discrete, informative nuances inherent in individual MRI scan slices. To overcome this limitation, we present a pioneering deep learning framework leveraging Multi-Instance Learning (MIL) to enhance knee OA detection from 3D MRI scans. This novel approach treats each MRI slice as an independent instance, harnessing unique pathological details that contribute collectively to a more accurate and comprehensive diagnosis. Evaluated on an extensive dataset of$\mathbf{9 0 0}$patients from the public OAI database, our MIL-based method demonstrates superior diagnostic performance, marking a significant leap over conventional imaging techniques, achieving an AUC of$\mathbf{9 3. 4 1 \%}$.

Preserving cultural heritage is paramount for societal and historical identity. Paintings, spanning ancient to modern eras, are pivotal subjects under constant scrutiny. As art reflects human creativity, studying human visual behavior toward paintings becomes increasingly vital. Thus, we introduce the AVAtt dataset, providing eye movement data for a diverse collection of painting styles across various ages and geographical origins. This dataset aims to facilitate the development and evaluation of computational saliency and scanpath prediction methods in the unique domain of painting (Dataset available at Github).

The advancement of image restoration, especially in reconstructing missing or damaged image areas, has benefited significantly from self-supervised learning techniques, notably through the recent Masked Auto Encoder (MAE) strategy. In this project, we leverage eye-tracking data to enhance image reconstruction quality, and more specifically, with fixation-based saliency combined with the MAE strategy. By examining the emergent properties of representation learning and drawing parallels to human perceptual observation, we focus on how eye-tracking data informs the selection of image patches for reconstruction, aligning computational methods with human visual perception. Our findings reveal the potential of integrating eye-tracking insights to improve the accuracy and perceptual relevance of self-supervised learning models in computer vision. This study thus underscores the synergy between computational image restoration methods and human perception, facilitated by eye-tracking technology, opening new directions and insights for both fields. Our experiments are available for reproducibility in this GitHub Repository.

3D Point Clouds (PCs) have become a valuable tool for representing intricate 3D information. Assessing the quality of PCs remains a challenging task, especially when striving for optimal immersive experiences. This paper introduces a novel metric and training approach that leverages projection-based views to evaluate the quality of 3D content. Our approach addresses a critical issue related to the intrinsic bias of deep networks for image recognition towards building hierarchical representations including only the global semantic, at the expense of local details. This bias is a limiting factor in tasks like 3D point cloud quality assessment where instances of the same content with varying degrees and types of degradation can possess strikingly similar representations. We propose a novel point cloud quality metric using a dual supervised and unsupervised training strategy to balance semantic understanding and preservation of critical perceptual quality-relevant information. The results demonstrate the effectiveness and reliability of our solution compared to state-of-the-art metrics on two standard 3D PCs quality assessment benchmarks (3D PCQA).

Point clouds (PCs) gained popularity as a representation for 3D objects and scenes and are widely used in numerous applications in augmented and virtual reality domains. Concurrently, quality assessment of PCs became even more relevant to improve various aspects of these imaging pipelines. To stimulate further growth and interest in point cloud quality assessment (PCQA), we created a large-scale PCQA dataset (called “BASICS”) which provides the research community with a relevant and challenging dataset to develop reliable objective quality metrics, and we organized the PCVQA grand challenge at ICIP 2023. In this paper, we provide a track-based evaluation methodology for benchmarking visual quality metrics, mirroring the PCVQA grand challenge evaluation scenarios designed to mimic real-life applications. Furthermore, we provide a state-of-the-art benchmark for the point cloud quality metrics. The track-based benchmarking approach shows that there is room for improvement in certain research directions, drawing attention to open problems in the PCQA domain.

Image restoration aims at recovering a clean image from degraded observations. This paper presents a novel Bayesian framework for image restoration using a regularization-by-denoising (RED) prior. It introduces a probabilistic counterpart to the RED paradigm, and proposes a new Monte Carlo algorithm to efficiently sample from the resulting posterior distribution. The proposed method benefit from the recent developments of deep learning-based denoisers. Extensive numerical experiments illustrate the efficiency of the proposed method, showcasing its competitive performance against state-of-the-art methods.

In the new area of immersive multimedia environments, understanding and manipulating visual attention are crucial for enhancing user experience. This study introduces an innovative framework that extends traditional 2D saliency maps to the analysis of 3D point clouds, a step forward in adapting saliency prediction to more complex and immersive environments. Our framework centers on the orthographic projection of 3D point clouds onto 2D planes, enabling the application of established 2D saliency models to this novel context. We further delve into the evaluation of these models on a 3D point cloud eye-tracking dataset, exploring various projection settings and thresholding techniques to maintain the integrity of saliency information in the transition from 2D to 3D. This research not only bridges a gap in applying visual attention models to 3D data but also offers insights into the optimization of quality of experience in immersive multimedia systems.

Point clouds have become increasingly prevalent in representing 3D scenes within virtual environments, alongside 3D meshes. Their ease of capture has facilitated a wide array of applications on mobile devices, from smartphones to autonomous vehicles. Notably, point cloud compression has reached an advanced stage and has been standardized. However, the availability of quality assessment datasets, which are essential for developing improved objective quality metrics, remains limited. In this paper, we introduce BASICS, a large-scale quality assessment dataset tailored for static point clouds. The BASICS dataset comprises 75 unique point clouds, each compressed with four different algorithms including a learning-based method, resulting in the evaluation of nearly 1500 point clouds by 3500 unique participants. Furthermore, we conduct a comprehensive analysis of the gathered data, benchmark existing point cloud quality assessment metrics and identify their limitations. By publicly releasing the BASICS dataset, we lay the foundation for addressing these limitations and fostering the development of more precise quality metrics.

The evaluation of 3D point cloud quality is a critical component in the development of immersive multimedia systems for real-world applications. While perceptual quality evaluation technics for 2D images and videos have reached high performances, developing robust and efficient blind metrics for point cloud quality assessment is still challenging. In this paper, we propose a no-reference point cloud quality assessment method that evaluates the quality of degraded 3D objects using an end-to-end point-based multi-stream model. To capture the geometric degradation of the point cloud, we incorporate normals, curvatures and geometric coordinates. Then, we divide the distorted object into sub-objects, which are fed to a multi-stream network to extract significant features of the geometric degradation. Afterward, these features are used to predict the quality of each sub-object, and the perceptual quality score of the point cloud is obtained by averaging the quality scores of all sub-objects. Experimental results demonstrate that the proposed model achieves promising performance compared to state-of-the- art full and reduced methods.

Knee osteoarthritis (KOA) is a widespread condition that can cause chronic pain and stiffness in the knee joint. Early detection and diagnosis are crucial for successful clinical intervention and management to prevent severe complications, such as loss of mobility. In this paper, we propose an automated approach that employs the Swin Transformer to predict the severity of KOA. Our model uses publicly available radiographic datasets with Kellgren and Lawrence scores to enable early detection and severity assessment. To improve the accuracy of our model, we employ a multi-prediction head architecture that utilizes multi-layer perceptron classifiers. Additionally, we introduce a novel training approach that reduces the data drift between multiple datasets to ensure the generalization ability of the model. The results of our experiments demonstrate the effectiveness and feasibility of our approach in predicting KOA severity accurately.

Knee OsteoArthritis (KOA) is a common musculoskeletal disorder, which causes reduced mobility for seniors. Early detection of such a disorder is important to limit its impact on people. Computer-Aided Diagnosis (CAD) systems based on deep learning methods have shown success in KOA diagnosis. Due to the high cost of labelling, the lack of sufficient data in the medical field is a significant challenge for training machine learning models. To improve the generalization capability of deep neural network models and avoid overfitting, data augmentation is essential. However, existing data augmentation techniques such as rotation and gamma correction are not effective at increasing the diversity of the original data. In this paper, we propose a novel approach based on the Vision Transformer (ViT) model with a Selective Shuffled Position Embedding (SSPE) strategy that generates different input sequences fixing and shuffling the position embedding of key and non-key patches, respectively, as a novel method of data augmentation for early detection of KOA (KL-0 vs KL-2). Experimental results demonstrate that our approach is valid as it can significantly improve the model’s classification performance.

Webcam-based eye-tracking platforms have recently re-emerged due to improvements in machine learning-supported calibration processes and offer a scalable option for conducting eye movement studies. Although not yet comparable to the infrared-based ones regarding accuracy and frequency, some compelling performances have been observed, especially in those scenarios with medium-sized AOI (Areas of Interest) in images. In this study, we test the reliability of webcam-based eye-tracking on a specific task: Eye movement distribution analysis for CVD (Colour Vision Deficiency) detection. We introduce a new publicly available eye movement dataset based on a pilot study (n=12) on images with dominant red colour (previously shown to be difficult with dichromatic AOI to investigate CVD by comparing attention patterns obtained in webcam eye-tracking sessions). We hypothesized that webcam eye tracking without infrared support could detect differing attention patterns between CVD and non-CVD participants and observed statistically significant differences, allowing the retention of our hypothesis.

Following the advent of immersive technologies and the increasing interest in representing interactive geometrical format, 3D Point Clouds (PC) have emerged as a promising solution and effective means to display 3D visual information. In addition to other challenges in immersive applications, objective and subjective quality assessments of compressed 3D content remain open problems and an area of research interest. Yet most of the efforts in the research area ignore the local geometrical structures between points representation. In this paper, we overcome this limitation by introducing a novel and efficient objective metric for Point Clouds Quality Assessment, by learning local intrinsic dependencies using Graph Neural Network (GNN). To evaluate the performance of our method, two well-known datasets have been used. The results demonstrate the effectiveness and reliability of our solution compared to state-of-the-art metrics.

Over the past few years, the development of deep learning-based methods has revolutionised the field of image quality assessment. These methods have shown remarkable success in estimating the quality of 2D images. However, most of these approaches are trained using small patches of the image, with the subjective score of the entire image serving as the target. assuming that all patches have an equal perceptual impact on the image. This assumption is not entirely consistent with our Human Visual System (HVS), which processes different regions of the image to form an overall perception. In this study, we focus on the use of saliency information to estimate image quality. In this context, we explore the use of saliency information to estimate image quality by selecting only the most perceptually relevant patches. Specifically, we evaluate the accuracy of saliency- or scanpath-based patch selection methods for predicting 2D image quality. Our goal is to determine which approach provides the most accurate estimation of image quality and whether the use of saliency information can improve the performance of deep learning-based methods for image quality assessment.

The visual scanpath represents the fundamental concept upon which visual attention research is based. As a result, the ability to predict them has emerged as a crucial task in recent years. It is represented as a sequence of points through which the human gaze moves while exploring a scene. In this paper, we propose an inter-observer consistent adversarial training approach for scanpath prediction through a lightweight deep neural network. The proposed method employs a discriminative neural network as a dynamic loss that better models the natural stochastic phenomenon while maintaining consistency between the distributions related to the subjective nature of scanpaths traversed by different observers. The competitiveness of our approach against state-of-the-art methods is shown through a testing phase.

Cultural heritage understanding and preservation is an important issue for society as it represents a fundamental aspect of its identity. Paintings represent a significant part of cultural heritage, and are the subject of study continuously. However, the way viewers perceive paintings is strictly related to the so-called HVS (Human Vision System) behaviour. This paper focuses on the eye-movement analysis of viewers during the visual experience of a certain number of paintings. In further details, we introduce a new approach to predicting human visual attention, which impacts several cognitive functions for humans, including the fundamental understanding of a scene, and then extend it to painting images. The proposed new architecture ingests images and returns scanpaths, a sequence of points featuring a high likelihood of catching viewers’ attention. We use an FCNN (Fully Convolutional Neural Network), in which we exploit a differentiable channel-wise selection and Soft-Argmax modules. We also incorporate learnable Gaussian distributions onto the network bottleneck to simulate visual attention process bias in natural scene images. Furthermore, to reduce the effect of shifts between different domains (i.e. natural images, painting), we urge the model to learn unsupervised general features from other domains using a gradient reversal classifier. The results obtained by our model outperform existing state-of-the-art ones in terms of accuracy and efficiency.

Deep learning-based quality metrics have recently given significant improvement in Image Quality Assessment (IQA). In the field of stereoscopic vision, information is evenly distributed with slight disparity to the left and right eyes. However, due to asymmetric distortion, the objective quality ratings for the left and right images would differ, necessitating the learning of unique quality indicators for each view. Unlike existing stereoscopic IQA measures which focus mainly on estimating a global human score, we suggest incorporating left, right, and stereoscopic objective scores to extract the corresponding properties of each view, and so forth estimating stereoscopic image quality without reference. Therefore, we use a deep multi-score Convolutional Neural Network (CNN). Our model has been trained to perform four tasks: First, predict the left view’s quality. Second, predict the quality of the left view. Third and fourth, predict the quality of the stereo view and global quality, respectively, with the global score serving as the ultimate quality. Experiments are conducted on Waterloo IVC 3D Phase 1 and Phase 2 databases. The results obtained show the superiority of our method when comparing with those of the state-of-the-art. The implementation code can be found at: https://github.com/o-messai/multi-score-SIQA

Recent information and communication systems have employed 3D Point Cloud (PC) as an advanced geometrical representation modality for immersive applications. Like most multimedia data, PCs are often compressed for transmission and viewing purposes, which can impact the perceived quality. Developing robust and efficient objective quality metrics for PCs is still an open problem. In this paper, we propose an end-to-end deep approach for evaluating the perceptual effects of point cloud compression solutions without reference. Our approach focuses on leveraging the intrinsic point cloud characteristics to quantify the coding impairments from few distant randomly selected patches using supervised and unsupervised training strategies. To evaluate the performance of our method, two well-known datasets have been used. The results demonstrate the effectiveness and reliability of the proposed method compared to to state-of-the-art methods.

Predicting the quality of multimedia content is often needed in different fields. In some applications, quality metrics are crucial with a high impact, and can affect decision making such as diagnosis from medical multimedia. In this paper, we focus on such applications by proposing an efficient and shallow model for predicting the quality of medical images without reference from a small amount of annotated data. Our model is based on convolution self-attention that aims to model complex representation from relevant local characteristics of images, which itself slide over the image to interpolate the global quality score. We also apply domain adaptation learning in unsupervised and semi-supervised manner. The proposed model is evaluated through a dataset composed of several images and their corresponding subjective scores. The obtained results showed the efficiency of the proposed method, but also, the relevance of the applying domain adaptation to generalize over different multimedia domains regarding the downstream task of perceptual quality prediction.1

Abstract We propose a deep reinforcement learning‐based solution for the 3D reconstruction of objects of complex topologies from a single RGB image. We use a template‐based approach. However, unlike previous template‐based methods, which are limited to the reconstruction of 3D objects of fixed topology, our approach learns simultaneously the geometry and topology of the target 3D shape in the input image. To this end, we propose a neural network that learns to deform a template to fit the geometry of the target object. Our key contribution is a novel reinforcement learning framework that enables the network to also learn how to adjust, using pruning operations, the topology of the template to best fit the topology of the target object. We train the network in a supervised manner using a loss function that enforces smoothness and penalizes long edges in order to ensure high visual plausibility of the reconstructed 3D meshes. We evaluate the proposed approach on standard benchmarks such as ShapeNet, and in‐the‐wild using unseen real‐world images. We show that the proposed approach outperforms the state‐of‐the‐art in terms of the visual quality of the reconstructed 3D meshes, and also generalizes well to out‐of‐category images.

The use of 3D technologies is growing rapidly, and stereoscopic imaging is usually used to display the 3D contents. However, compression, transmission and other necessary treatments may reduce the quality of these images. Stereo Image Quality Assessment (SIQA) has attracted more attention to ensure good viewing experience for the users and thus several methods have been proposed in the literature with a clear improvement for deep learning-based methods. This paper introduces a new deep learning-based no-reference SIQA using cyclopean view hypothesis and human visual attention. First, the cyclopean image is constructed considering the presence of binocular rivalry that covers the asymmetric distortion case. Second, the saliency map is computed considering the depth information. The latter aims to extract patches on the most perceptual relevant regions. Finally, a modified version of the pre-trained Convolutional Neural Network (CNN) is fine-tuned and used to predict the quality score through the selected patches. Five distinct pre-trained models were analyzed and compared in term of results. The performance of the proposed metric has been evaluated on four commonly used datasets (3D LIVE phase I and phase II databases as well as Waterloo IVC 3D Phase 1 and Phase 2). Compared with the state-of-the-art metrics, the proposed method gives better outcomes. The implementation code will be made accessible to the public at: https://github.com/o-messai/3D-NR-SIQA

This paper addresses the problem of blind stereoscopic image quality assessment (NR-SIQA) using a new multi-task deep learning based-method. In the field of stereoscopic vision, the information is fairly distributed between the left and right views as well as the binocular phenomenon. In this work, we propose to integrate these characteristics to estimate the quality of stereoscopic images without reference through a convolutional neural network. Our method is based on two main tasks: the first task predicts naturalness analysis based features adapted to stereo images, while the second task predicts the quality of such images. The former, so-called auxiliary task, aims to find more robust and relevant features to improve the quality prediction. To do this, we compute naturalness-based features using a Natural Scene Statistics (NSS) model in the complex wavelet domain. It allows to capture the statistical dependency between pairs of the stereoscopic images. Experiments are conducted on the well known LIVE PHASE I and LIVE PHASE II databases. The results obtained show the relevance of our method when comparing with those of the state-of-the-art. Our code is available online on https://github.com/Bourbia-Salima/multitask-cnn-nrsiqa_2021

Human vision is naturally more attracted by some regions within their field of view than others. This intrinsic selectivity mechanism, so-called visual attention, is influenced by both high- and low-level factors; such as the global environment (illumination, background texture, etc.), stimulus characteristics (color, intensity, orientation, etc.), and some prior visual information. Visual attention is useful for many computer vision applications such as image compression, recognition, and captioning. In this paper, we propose an end-to-end deep-based method, so-called SALYPATH (SALiencY and scanPATH), that efficiently predicts the scanpath of an image through features of a saliency model. The idea is predict the scanpath by exploiting the capacity of a deep-based model to predict the saliency. The proposed method was evaluated through 2 well-known datasets. The results obtained showed the relevance of the proposed framework comparing to state-of-the-art models.

In this paper, we propose a new architecture to enhance dense layers with a Taylor Series Kernelized Layer (TSKL). The proposed layer expands the underlying linear kernel of dense layers to a higher-order Taylor series kernel. This kernel is able to learn more complex patterns than the linear one and thus be more discriminative. In other words, TKSL first maps input data to a higher-dimensional Reproducing Kernel Hilbert Space (RKHS). After that, it learns a linear classifier in that RKHS which corresponds to a powerful non-linear classifier in the original feature space. The mapping features to a higher-order RKHS is performed implicitly by leveraging the kernel trick and explicitly by combining multiple kernels. The experimental results demonstrate that the proposed layer outperforms the ordinary dense layer when uses in both Multilayer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs).

We propose in this work a graph-based deep learning method for mesh visual quality assessment. To carry out this proposal, we transform a given distorted mesh to a graph represented by its adjacency matrix, and extract a set of geometric and perceptual features to be stored into a feature matrix. The two matrices are then learned to a graph convolutional network (GCN). The network is composed by two convolutional layers followed by a max-pooling layer. The Softmax classifier is used to predict the quality relying on the node classification problem. Five classes are considered according to the ground truth scores: very bad, bad, medium, good or excellent quality. Experiments are conducted on two publicly available databases specifically constructed for the quality assessment task. Our method is compared to some influential and effective full and reduced reference methods. The good performance is proven by the excellent correlations with subjective decisions.

In recent years, the production of 3D content in the form of point clouds (PC) has increased considerably, especially in virtual reality applications. This enthusiasm is linked in particular to the development of acquisition technologies. In order to ensure a good quality of user experience, it is necessary to offer a high quality of visualization whatever the transmission medium used or the treatments applied. Thus, several metrics have been proposed which are essentially point-based metrics. In this article, we propose a deep learning-based method that efficiently predicts the quality of distorted PCs thanks to a set of features extracted from selected patches of the reference PC and its degraded version as well as the use of Convolutional Neural Networks (CNNs). The patches are selected randomly and the difference between corresponding patches is characterized by three attributes: geometry, curvature and color. The proposed method was evaluated and compared to state-of-the-art metrics using two datasets, including a large dataset more suited to deep learning models. We also compared different symmetrization functions and machine learning pooling as well as the ability of our method to predict the quality of unknown PCs through a cross-dataset evaluation. The results obtained show the relevance of the proposed framework with interesting perspectives.

We propose in this paper a novel objective method to evaluate the perceived visual quality of 3D meshes. The proposed method in no-reference, it relies only on the distorted mesh for the quality estimation. It is based on a pre-trained convolutional neural network (i.e VGG to extract features from the distorted mesh) and handcrafted features extracted directly from the 3D mesh (i.e curvature and dihedral angle). A General Regression Neural Network (GRNN) is used to learn the statistical parameters of the feature vectors and estimate the quality score. Experimental results from for subjective databases (LIRIS masking, LIRIS/EPFL generalpurpose, UWB compression and LEETA simplification) and comparisons with objective metrics cited in the state-of-the-art demonstrate the efficacy of the proposed metric in terms of the correlation to the mean opinion scores across these databases.

Visual masking is a well-studied phenomenon that has been exploited for signal compression, computer graphics and data hiding. Among the different types of visual masking, chromatic masking has received very little attention despite its importance and proven potential for the aforementioned applications. In this paper, we ask whether a deep neural network can learn to predict the detection thresholds in a chromatic masking paradigm. For that, a CNN model was trained and evaluated using a dataset made of 480 image patches for which chromatic thresholds were registered in terms of log-Gabor targets, as well as Root Mean Square (RMS) error. Experimental results show the superiority of the proposed approach.

Fully connected layer is an essential component of Convolutional Neural Networks (CNNs), which demonstrates its efficiency in computer vision tasks. The CNN process usually starts with convolution and pooling layers that first break down the input images into features, and then analyze them independently. The result of this process feeds into a fully connected neural network structure which drives the final classification decision. In this paper, we propose a Kernelized Dense Layer (KDL) which captures higher order feature interactions instead of conventional linear relations. We apply this method to Facial Expression Recognition (FER) and evaluate its performance on RAF, FER2013 and ExpW datasets. The experimental results demonstrate the benefits of such layer and show that our model achieves competitive results with respect to the state-of-the-art approaches.

We present the results of a subjective experiment where we measured detection thresholds for 2°wide noise targets placed in 23 different natural scenes. Unlike previous studies on visual masking, we focus particularly on dissociating cases of low-level and high-level masking. That is, cases where the target is not perceived predominantly due to limits of either early or late vision. To that end, we exploit the change blindness paradigm and analyse detection rates, times and primed subjective ratings of target visibility. Our results are of significance for developing advanced models of human vision for signal quality/fidelity assessment, particularly in the context of compression.

Images are often distorted by some necessary treatments (capture, compression, transmission, etc..) that can affect the perceptual quality. To evaluate the impact of these treatments, a plethora of metrics have been developed in the literature. In this paper, we propose an efficient blind method to estimate the quality of 2D-images based on the selection of relevant patches through the saliency information and a Convolutional Neural Network (CNN). Saliency information was here used to focus on regions that highly impact the subjective scores. Three CNN models were individually evaluated and compared (AlexNet, VGG19 and ResNet50). A pairwise combination of deep learning-based feature vectors was then applied to estimate the quality. Different pooling strategies were employed (Concatenation, Element-wise operation and Bilinear Pooling). Our method was compared to the state-of the-art using two common datasets (LIVE-P2 and CSIQ). The results obtained showed the efficiency of our approach and its generalization ability.

OsteoArthritis (OA) is the most common disorder of the musculoskeletal system and the major cause of reduced mobility among seniors. The visual evaluation of OA still suffers from subjectivity. Recently, Computer-Aided Diagnosis (CAD) systems based on learning methods showed potential for improving knee OA diagnostic accuracy. However, learning discriminative properties can be a challenging task, particularly when dealing with complex data such as X-ray images, typically used for knee OA diagnosis. In this paper, we introduce a Discriminative Regularized Auto Encoder (DRAE) that allows to learn both relevant and discriminative properties that improve the classification performance. More specifically, a penalty term, called discriminative loss is combined with the standard Auto-Encoder training criterion. This additional term aims to force the learned representation to contain discriminative information. Our experimental results on data from the public multicenter OsteoArthritis Initiative (OAI) show that the developed method presents potential results for early knee OA detection.

Image quality is one of the emerging domain, since it impacts the efficiency of computer vision applications. Another key information that can be exploited to analyze an image is the image utility which estimates the useful of an image. Both information have weak relationship and should be exploited as relevant indicators. In this paper, we focused on both concepts by proposing a Convolutional Neural Network-based method that predicts the subjective utility and quality scores of a given image without reference. For that, the well-known pre-trained VGG16 model was used and adapted to our context. A patch selection step based on the saliency information through a scanpath predictor was also introduced. The proposed method was evaluated using the CU-Nantes dataset and the results was compared to some quality and utility estimators. The results highlighted the efficiency of our method with high correlations obtained for both.

A plethora of image quality metrics have been proposed in the literature. These metrics aims to estimate the perceptual image quality automatically. One important key aspect that the perceived quality is dependent on is the viewing distance from the observer to the image. In this study, we propose to consider this information by estimating the quality of a given image without a reference image for different viewing distances. For that, a Convolutional Neural Network (CNN) model was used in this study. Relevant patches are first selected from the image and they are then used as inputs to the CNN. The selection is here based on saliency information. The used CNN is composed of two outputs that correspond to the predicted subjective scores for two viewing distances (50 cm and 100 cm). Our method was evaluated using the Colourlab Image Database: Image Quality (CID:IQ) that provides subjective scores at two different viewing distances. The obtained results show the efficiency of our method.

A plethora of Contrast Enhancement (CE) methods has been proposed in the literature. Each of these has its own strengths and limitations. Further, the quality of the resulting enhanced images depends upon the original image and its content. Hence, a given CE method can provide good quality for a certain image but a poorer quality for another. In this paper, we propose a novel workflow to provide an automatic ranking of enhanced images which may have been obtained using different techniques. The proposed technique is based on a Convolutional Neural Network (CNN) using saliency information. The idea is to start by comparing two enhanced versions of a given image in order to select the best one automatically based on perceived quality. Here, a saliency map is used to select relevant patches which are highly correlated with the human visual system sensitivity. The well-known Structural Similarity Image Metric (SSIM) map is also employed to compare the similarity between both enhanced images. Using such information, a CNN model is trained to predict the rank in terms of the image quality as perceived by humans. The algorithm is tested over three CE benchmarking databases with the experimental results validating the superiority of the proposed system as compared to state-of-the-art CE evaluation techniques.

In this study, the authors introduce new solutions and improvements to the multi‐channel blind image deconvolution problem. More precisely, authors’ contributions are threefold: (i) At first, a simplified version of the existing cross‐relation method for blind system identification is proposed; but most importantly, the authors incorporate into the channel estimation cost function a sparsity constraint to deal with the challenging issue of channel order overestimation errors; (ii) then, once the channel identification is achieved, a new image restoration method based on the stack decoding algorithm is introduced; and (iii) finally, a refining approach using an ‘all‐at‐once’ optimisation technique with an improved mixed norm regularisation is considered. The performance of the proposed approach was evaluated using several numerical simulations. Blind system identification and image restoration tasks were evaluated with respect to several criteria: numerical complexity, robustness to noise effects and channel order estimation. The results obtained are promising and highlight the effectiveness of the proposed approach.

In this work, we propose a convolutional neural network (CNN) framework to estimate the perceived visual quality of 3D meshes without having access to the reference. The proposed CNN architecture is fed by small patches selected carefully according to their level of saliency. To do so, the visual saliency of the 3D mesh is computed, then we render 2D projections from the 3D mesh and its corresponding 3D saliency map. Afterward, the obtained views are split to obtain 2D small patches that pass through a saliency filter to select the most relevant patches. Experiments are conducted on two MVQ assessment databases, and the results show that the trained CNN achieves good rates in terms of correlation with human judgment.

In this paper, we propose a degradation-based image quality metric without reference using a Convolutional Neural Network (CNN) model and saliency patch selection. The degradation of the image is first identified and the corresponding saliency map is computed. The saliency map is here used to select patches according to their perceptual relevance, while the degradation identification step allows having a specific CNN model, which permits to improve the performance. A CNN model is then used to estimate the quality of each patch and the overall quality is given by the average of the obtained scores. The proposed method was evaluated through three well-known datasets and was compared to some recent methods.

ARCADIA project aims to enhance the archaeological heritage of ceramic shards extracted in Saran (France). Dating from the High Middle Ages, these shards have been engraved by repeated patterns using a carved wooden wheel. The study of these shards allows the archeologists to better understand the diffusion of ceramic productions. In this paper, we propose to exploit Convolutional Neural Network (CNN) models to classify automatically these ceramic shards. The ultimate goal is to form clusters of shards to derive a map that represents the movements of potters. For that, several models have been tested and compared to some well-known handcrafted-methods. The Fully Connected part of the best model was modified to see its impact in terms of classification. A dataset composed of 888 binary images of ceramic shards was used. The results obtained outperform the state-of-the-art methods and show the relevance of the proposed approach.

A Full Reference Image Quality Metric (FR-IQM) for stereoscopic images based on two main steps is presented in this paper. The first step consists to select some features according to the considered degradation types and combine those features using an Artificial Neural Network (ANN). These features are here extracted from a Cyclopean Image (CI). At the end of this step, a FR-IQM per degradation is thus obtained. In order to provide a single objective score that is in accordance with the subjective judgment, outputs of all ANN models are combined in a second step. The efficiency of our method has been evaluated using the 3D LIVE Image Quality Database that is composed of 5 types of degradation with 365 degraded images. Its performance is discussed and compared to some recent metrics.

In this paper, we propose a new Reduced Reference Image Quality Metric for multiply degraded images based on a features extraction step and its combination. The selected features are extracted from the original image and its degraded version. Some of them aim to quantify the level of the considered degradation types, while the others quantify its sharpness. These features are then combined to obtain a single value, which corresponds to the predicted subjective score. Our method has been evaluated and compared in terms of correlation with subjective judgments to some recent methods by using the LIVE Multiply Distorted Image Quality Database.

In this work, we propose to study the universality of the Full-Reference Image Quality metrics (FR-IQMs) and show the no-relevance to use this kind of metrics without considering the degradation type contained in the image. Different experimental tests have been done in order to analyze its performance. Eight common FR-IQMs have been used and compared in terms of correlation with the subjective judgments. Obtained results show that the performance of a given FR-IQM differs totally from a degradation type to another. Therefore, we finally conclude by the pertinence of some recent works that propose alternative solutions to solve this limitation and then optimize the image quality estimation process.

In this paper, we present a New Stereo Full-Reference Image Quality Metric (SFR-IQM) based on Cyclopean Image (CI) computation and 2D IQM fusion. The Cyclopean images of the reference image and its degraded version are first computed from the left and the right views. 2D measures are then extracted from the obtained CIs and are combined using an Artificial Neural Networks (ANN) in order to derive a single index. The 3D LIVE Image Quality Database has been here used to evaluate our method and its capability to predict the subjective judgments. The obtained results have been compared to some recent methods considered as the state-of-the-art. The experimental results show the relevance of our method.

In this paper, we propose to use visual saliency to improve an indoor localization system based on image matching. A learning step permits to determinate the reference trajectory by selecting some key frames along the path. During the localization step, the current image is then compared to the obtained key frames in order to estimate the user's position. This comparison is realized by extracting primitive information through a saliency method, which aims to improve our localization system by focusing our attention on the more singular regions to match. Another advantage of the saliency-guided detection is to save computation time. The proposed framework has been developed and tested on a Smartphone. The obtained results show the interest of the use of saliency models by comparing the numbers of features and good matches in video sequence.

Indoor navigation assistance is a highly challenging task that is increasingly needed in various types of applications such as visually impaired guidance, emergency intervention, tourism, etc. Many alternative techniques to GPS have been explored to deal with this challenge like pre-installed sensor networks (Wifi, Ultra Wide Band, Bluetooth, Radio Frequency IDentification etc), inertial sensors or camera. This paper presents an indoor navigation system on Smartphone that was designed taking into consideration low cost, portability and the lightweight of the used algorithm in terms of computation power and storage space. The proposed solution relies on embedded vision. Robust and fast camera orientation (3 dof) is estimated by tracking three orthogonal vanishing points in a video stream acquired with the camera of a free-handled Smartphone. The developed algorithm enables indoor pedestrian localization in two steps: an off-line learning step defines a reference path by selecting key frames along the way using saliency extraction method and computing the camera orientation in these frames. Then, in localization step, an approximate but realistic position of the walker is estimated in real time by comparing the orientation of the camera in the current image and that of reference to assist the pedestrian with navigation guidance. Unlike SLAM, this approach does not require to build 3D mapping of the environment. Online walking direction is given by Smartphone camera which advantageously replaces the compass sensor since it performs very poorly indoors due to electromagnetic noise. Experiments, executed online on Smartphone, that show the feasibility and evaluate the accuracy of the proposed positioning approach for different indoor paths.

In this paper, we introduce a radon wigner ville based image dissimilarity measure. The proposed Image distortion measure aims to combine the useful properties of the wigner ville and the directionality of the finite Radon transform for image quality assessment. The results are compared with several HVS and transform based measures of image quality on the basis of complexity of these measures and their consistency with subjective assessment.

We present in this paper a new global Full-Reference (FR) image quality metric (IQM) based on the fusion of several conventional FR metrics using an ANN learning algorithm. The fusion is shown to result in improved performance compared to individual FR metrics. Indeed, existing FR metrics can provide excellent results for specific degradations but poor results for others. Here, we propose to overcome this limitation by first improving the performance of existing FR metrics across different degradations through a ranking process. Then, using an Artificial Neural Network, we fuse the best-performing measures into a single metric called Global Index Quality Metric (G-IQM). The experimental results using the TID 2008 image database demonstrate that this new G-IQM metric achieves consistent image quality evaluation results with subjective evaluation.

A plethora of Image Quality Metrics (IQM) has been proposed during the last two decades. However, at present time, there is no accepted IQM able to predict the perceptual level of image degradation across different types of visual distortions. Some measures are more adapted for a set of degradations but inefficient for others. Indeed, the efficiency of any IQM has been shown to depend upon the type of degradation. Thus, we propose here a new approach for predicting the type of degradation before using IQMs. The basic idea is first to identify the type of distortion using a Bayesian approach, then select the most appropriate IQM for estimating image quality for that specific type of distortion. The performance of the proposed method is evaluated in terms of classification accuracy across different types of degradations.

A new method for deblocking is proposed. It aims to reduce the blocking artifacts in the compressed image by analyzing their visibility. A perceptual map is obtained using some Human Visual System (HVS) characteristics. This perceptual map is used as input to a recursive filter to reduce the blocking effect. The obtained results have been compared with a very recent efficient method considered.