Neural networks are widely used for image–related tasks but typically demand considerable computing power. Once a network has been trained, however, its memory‑ and compute‑footprint can be reduced by compression. In this work, we focus on compression through tensorization and low‑rank representations. Whereas classical approaches search for a low‑rank approximation by minimizing an isotropic norm such as the Frobenius norm in weight‑space, we use data‑informed norms that measure the error in function space. Concretely, we minimize the change in the layer’s output distribution, which can be expressed as $\lVert (W - \widetilde{W}) \Sigma^{1/2}\rVert_F$ where $\Sigma^{1/2}$ is the square root of the covariance matrix of the layer’s input and $W$, $\widetilde{W}$ are the original and compressed weights. We propose new alternating least square algorithms for the two most common tensor decompositions (Tucker‑2 and CPD) that directly optimize the new norm. Unlike conventional compression pipelines, which almost always require post‑compression fine‑tuning, our data‑informed approach often achieves competitive accuracy without any fine‑tuning. We further show that the same covariance‑based norm can be transferred from one dataset to another with only a minor accuracy drop, enabling compression even when the original training dataset is unavailable. Experiments on several CNN architectures (ResNet‑18/50, and GoogLeNet) and datasets (ImageNet, FGVC‑Aircraft, Cifar10, and Cifar100) confirm the advantages of the proposed method.

Most of the recent literature on image Super-Resolution (SR) can be classified into two main approaches. The first one involves learning a corruption model tailored to a specific dataset, aiming to mimic the noise and corruption in low-resolution images, such as sensor noise. However, this approach is data-specific, tends to lack adaptability, and its accuracy diminishes when faced with unseen types of image corruptions. A second and more recent approach, referred to as Robust Super-Resolution (RSR), proposes to improve real-world SR by harnessing the generalization capabilities of a model by making it robust to adversarial attacks. To delve further into this second approach, our paper explores the universality of various methods for enhancing the robustness of deep learning SR models. In other words, we inquire: “Which robustness method exhibits the highest degree of adaptability when dealing with a wide range of adversarial attacks?”. Our extensive experimentation on both synthetic and real-world images empirically demonstrates that median randomized smoothing (MRS) is more general in terms of robustness compared to adversarial learning techniques, which tend to focus on specific types of attacks. Furthermore, as expected, we also illustrate that the proposed universal robust method enables the SR model to handle standard corruptions more effectively, such as blur and Gaussian noise, and notably, corruptions naturally present in real-world images. These results support the significance of shifting the paradigm in the development of real-world SR methods towards RSR, especially via MRS.

Neural networks (NN) often need to be lightweight and fast for practical deployment. Various NN compression techniques, including tensor decomposition, aim to achieve this goal. Tensor decomposition involves computing a decomposition of the weight tensor of a layer and replacing it with smaller layers using the decomposed weights. Subsequently, fine-tuning is performed to regain lost accuracy. However, while NNs are known to exhibit robustness issues, tensor decomposition for compression has primarily been evaluated on accuracy. In this study, we investigate the impact of tensor decomposition on the robustness of large CNN (Convolutional Neural Network) models. Through multiple experiments on different models trained on ImageNet, we demonstrate that tensor decomposition preserves model robustness. Furthermore, we observe that the choice of fine-tuning learning rate plays a crucial role in determining robustness. A high learning rate may enhance accuracy but significantly compromises robustness. Conversely, a low learning rate can effectively restore model robustness, albeit with a smaller accuracy improvement. These findings offer a practical approach to preserving model robustness without resorting to adversarial learning, thus eliminating the need for additional knowledge about the defense methods used in the original model.

We propose a new framework for tensor decomposition based on trace invariants, which are particular cases of tensor networks. In general, tensor networks are diagrams/graphs that specify a way to "multiply" a collection of tensors together to produce another tensor, matrix or scalar. The particularity of trace invariants is that the operation of multiplying copies of a certain input tensor that produces a scalar obeys specific symmetry constraints. In other words, the scalar resulting from this multiplication is invariant under some specific transformations of the involved tensor. We focus our study on the O(N)-invariant graphs, i.e. invariant under orthogonal transformations of the input tensor. The proposed approach is novel and versatile since it allows to address different theoretical and practical aspects of both CANDECOMP/PARAFAC (CP) and Tucker decomposition models. In particular we obtain several results: (i) we generalize the computational limit of Tensor PCA (a rank-one tensor decomposition) to the case of a tensor with axes of different dimensions (ii) we introduce new algorithms for both decomposition models (iii) we obtain theoretical guarantees for these algorithms and (iv) we show improvements with respect to state of the art on synthetic and real data which also highlights a promising potential for practical applications.

In this paper, based on an asymptotic analysis of the Softmax layer, we show that when training neural networks for classification tasks, the weight vectors corre sponding to each class of the Softmax layer tend to converge to the class-wise means computed at the representation layer (for specific choices of the representation activation). We further show some consequences of our findings to the context of transfer learning, essentially by proposing a simple yet effective initialization procedure that significantly accelerates the learning of the Softmax layer weights as the target domain gets closer to the source one. Experiments are notably performed on the datasets: MNIST, Fashion MNIST, Cifar10, and Cifar100 and using a standard CNN architecture.

This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs.

This paper presents a method for constructing a size compressed neural network with better or similar accuracy than a given dense neural network, therefore the compressed network requires less memory and computational resources. The presented method relies basically on learning successive mappings between the given dense neural network (teacher) hidden features and the size-compressed neural network (student) hidden features, where the latter is learned also to solve the initial task of the teacher network. The presented method is particularly compared to baselines where we specifically show that the additional learned mappings significantly improve the performance (accuracy and computation) of the student network.

This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices. The presented distillation methods are respectively based on Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The two methods basically rely on the successive dimension reduction of a given dense neural network (teacher) hidden features, and the learning of a smaller neural network (student) which solves the initial learning problem along with a mapping problem to the reduced successive features spaces. The presented methods are compared to baselines -learning the student networks from scratch-, and we show that the additional mapping problem significantly improves the performance (accuracy, memory and computing resources) of the student networks.

This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called \emph{concentrated} random vectors. Further exploiting the fact that Gram matrices, of the type $G = X^\intercal X$ with $X=[x_1,\ldots,x_n]\in \mathbb{R}^{p\times n}$ and $x_i$ independent concentrated random vectors from a mixture model, behave asymptotically (as $n,p\to \infty$) as if the $x_i$ were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.

We present a method to automatically learn to segment dynamic objects using SLAM outliers. It requires only one monocular sequence per dynamic object for training and consists in localizing dynamic objects using SLAM outliers, creating their masks, and using these masks to train a semantic segmentation network. We integrate the trained network in ORB-SLAM 2 and LDSO. At runtime we remove features on dynamic objects, making the SLAM unaffected by them. We also propose a new stereo dataset and new metrics to evaluate SLAM robustness. Our dataset includes consensus inversions, i.e., situations where the SLAM uses more features on dynamic objects that on the static background. Consensus inversions are challenging for SLAM as they may cause major SLAM failures. Our approach performs better than the State-of-the-Art on the TUM RGB-D dataset in monocular mode and on our dataset in both monocular and stereo modes.

A representation is supposed universal if it encodes any element of the visual world (e.g., objects, scenes) in any configuration (e.g., scale, context). While not expecting pure universal representations, the goal in the literature is to improve the universality level, starting from a representation with a certain level. To improve that universality level, one can diversify the source-task, but it requires many additive annotated data that is costly in terms of manual work and possible expertise. We formalize such a diversification process then propose two methods to improve the universality of CNN representations that limit the need for additive annotated data. The first relies on human categorization knowledge and the second on re-training using fine-tuning. We propose a new aggregating metric to evaluate the universality in a transfer-learning scheme, that addresses more aspects than previous works. Based on it, we show the interest of our methods on 10 target-problems, relating to classification on a variety of visual domains.

Based on recent random matrix advances in the analysis of kernel methods for classification and clustering, this paper proposes the study of large kernel methods for a wide class of random inputs, i.e., concentrated data, which are more generic than Gaussian mixtures. The concentration assumption is motivated by the fact that one can use generative models to design complex data structures, through Lipschitzally transformed concentrated vectors (e.g., Gaussian) which remain concentrated vectors. Applied to spectral clustering, we demonstrate that our theoretical findings closely match the behavior of large kernel matrices, when considering the fed-in data as CNN representations of GAN-generated images (i.e., concentrated vectors by design).

Here, the authors explore the potential of multispectral imaging applied to image completion. Snapshot multispectral cameras correspond to breakthrough technologies that are suitable for everyday use. Therefore, they correspond to an interesting alternative to digital cameras. In their experiments, multispectral images are acquired using an ultracompact snapshot camera‐recorder that senses 16 different spectral channels in the visible spectrum. Direct exploitation of completion algorithms by extension of the spectral channels exhibits only minimum enhancement. A dedicated method that consists in a prior segmentation of the scene has been developed to address this issue. The segmentation derives from an analysis of the spectral data and is employed to constrain research area of exemplar‐based completion algorithms. The full processing chain takes benefit from standard methods that were developed by both hyperspectral imaging and computer vision communities. Results indicate that image completion constrained by spectral presegmentation ensures better consideration of the surrounding materials and simultaneously improves rendering consistency, in particular for completion of flat regions that present no clear gradients and little structure variance. The authors validate their method with a perceptual evaluation based on 20 volunteers. This study shows for the first time the potential of multispectral imaging applied to image completion.

Specularities are often problematic in computer vision since they impact the dynamic range of the image intensity. A natural approach would be to predict and discard them using computer graphics models. However, these models depend on parameters which are difficult to estimate (light sources, objects' material properties and camera). We present a geometric model called JOLIMAS: JOint LIght-MAterial Specularity, which predicts the shape of specularities. JOLIMAS is reconstructed from images of specularities observed on a planar surface. It implicitly includes light and material properties, which are intrinsic to specularities. This model was motivated by the observation that specularities have a conic shape on planar surfaces. The conic shape is obtained by projecting a fixed quadric on the planar surface. JOLIMAS thus predicts the specularity using a simple geometric approach with static parameters (object material and light source shape). It is adapted to indoor light sources such as light bulbs and fluorescent lamps. The prediction has been tested on synthetic and real sequences. It works in a multi-light context by reconstructing a quadric for each light source with special cases such as lights being switched on or off. We also used specularity prediction for dynamic retexturing and obtained convincing rendering results. Further results are presented as supplementary video material, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TVCG.2017.2677445.

The aim of Diminished Reality (DR) is to remove a target object in a live video stream seamlessly. In our approach, the area of the target object is replaced with new texture that blends with the rest of the image. The result is then propagated to the next frames of the video. One of the important stages of this technique is to update the target region with respect to the illumination change. This is a complex and recurrent problem when the viewpoint changes. We show that the state-of-the-art in DR fails in solving this problem, even under simple scenarios. We then use local illumination models to address this problem. According to these models, the variation in illumination only affects the specular component of the image. In the context of DR, the problem is therefore solved by propagating the specularities in the target area. We list a set of structural properties of specularities which we incorporate in two new models for specularity propagation. Our first model includes the same property as the previous approaches, which is the smoothness of illumination variation, but has a different estimation method based on the Thin-Plate Spline. Our second model incorporates more properties of the specularity's shape on planar surfaces. Experimental results on synthetic and real data show that our strategy substantially improves the rendering quality compared to the state-of-the-art in DR.

The latest advancements in Augmented Reality (AR) and Diminished Reality (DR) have allowed the development of many consumer-oriented applications (such as sales and driving aid, or education). To increase the realism in rendering, estimating the illumination in the scene is a key element. A lot of works tackle this problem but rarely discuss the color of the reconstructed illumination. The Dichromatic Model indicates that the specular component is not affected in color by the texture underneath and holds the light source's color. Though theoretically sound, in practice consumer cameras are subject to nonlinear behaviors which change RGB ratios and create inconsistencies when estimating the illumination. In this paper, we study the conditioning and limits of inverting local illumination models while relying on the Dichromatic Model. We show that the reconstructed specular component has an inconsistent color because it changes depending on the surface's colors.

The specularity prediction task in images, given the camera pose and scene geometry, is challenging and ill-posed. A recent approach called JOint LIght-MAterial Specularity (JOLIMAS) addresses this problem using a geometric model under the assumption that specularities have an elliptical shape. We address the most recent version of the model, Dual JOLIMAS, which is limited to planar and convex surfaces where the local surface's curvature under the specularity is constant. We propose a canonical representation of the JOLIMAS model that is independent of the local surface curvature. To reconstruct our model represented by a 3D quadric, we use at least 3 ellipses fitted to specularities and transform their shape to fit a planar surface and simulate a planar mirror which does not distort the image of the reflected object. After reconstruction, we project the 3D quadric into an ellipse and transform it to fit the current local curvature of the surface on a new viewpoint. We assessed this method on both synthetic and real sequences, and compared it to the previous approach Dual JOLIMAS.

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non‐rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature‐based matching. The feature‐based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second‐order regularisation with external and self‐occlusion reasoning. Their framework achieves state‐of‐the‐art performance in several cases (standard optical flow benchmarks, wide‐baseline stereo and non‐rigid surface registration). Their framework has a modular design that customises to specific application needs.

Predicting specularities in images, given the camera pose and scene geometry from SLAM, forms a challenging and open problem. It is nonetheless essential in several applications such as retexturing. A recent geometric model called JOLIMAS partially answers this problem, under the assumptions that the specularities are elliptical and the scene is planar. JOLIMAS models a moving specularity as the image of a fixed 3D quadric. We propose dual JOLIMAS, a new model which raises the planarity assumption. It uses the fact that specularities remain elliptical on convex surfaces and that every surface can be divided in convex parts. The geometry of dual JOLIMAS then uses a 3D quadric per convex surface part and light source, and predicts the specularities by a means of virtual cameras, allowing it to cope with surface's unflatness. We assessed the efficiency and precision of dual JOLIMAS on multiple synthetic and real videos with various objects and lighting conditions. We give results of a retexturing application. Further results are presented as supplementary video material.

Specularities, which are often visible in images, may be problematic in computer vision since they depend on parameters which are difficult to estimate in practice. We present an empirical model called JOLIMAS: JOint LIght-MAterial Specularity, which allows specularity prediction. JOLIMAS is reconstructed from images of specular reflections observed on a planar surface and implicitly includes light and material properties which are intrinsic to specularities. This work was motivated by the observation that specularities have a conic shape on planar surfaces. A theoretical study on the well known illumination models of Phong and Blinn-Phong was conducted to support the accuracy of this hypothesis. A conic shape is obtained by projecting a quadric on a planar surface. We showed empirically the existence of a fixed quadric whose perspective projection fits the conic shaped specularity in the associated image. JOLIMAS predicts the complex phenomenon of specularity using a simple geometric approach with static parameters on the object material and on the light source shape. It is adapted to indoor light sources such as light bulbs or fluorescent lamps. The performance of the prediction was convincing on synthetic and real sequences. Additionally, we used the specularity prediction for dynamic retexturing and obtained convincing rendering results. Further results are presented as supplementary material.

In this paper, we present an innovative Augmented Reality prototype designed for industrial education and training applications. The system uses an Optical See-Through HMD integrating a calibrated camera and a laser pointer to interactively augment an industrial object with virtual sequences designed to train a user for specific maintenance tasks. The training leverages user interactions by simply pointing on a specific object component. The architecture of our prototype involves two main vision-based modules : camera localization and user-interaction handling. The first module includes markerless trackers for camera localization, which can deal with partial occlusions and specular reflections on the metallic object surfaces. In the second module, we developed fast image processing methods for red laser dot tracking. By combining these processing elements, the proposed system is able to interactively augment in real time an industrial object making the learning process more interesting and intuitive.

We address the challenging issue of camera localization in a partially known environment, i.e. for which a geometric 3D model that covers only a part of the observed scene is available. When this scene is static, both known and unknown parts of the environment provide constraints on the camera motion. This paper proposes a nonlinear refinement process of an initial SfM reconstruction that takes advantage of these two types of constraints. Compare to those that exploit only the model constraints i.e. the known part of the scene, including the unknown part of the environment in the optimization process yields a faster, more accurate and robust refinement. It also presents a much larger convergence basin. This paper will demonstrate these statements on varied synthetic and real sequences for both 3D object tracking and outdoor localization applications.