We propose Interpretable Neural Radiance Fields (InNeRF) for generalizable 3D scene representation and rendering. In contrast to previous image-based rendering, which used two independent working processes of pooling-based fusion and MLP-based rendering, our framework unifies source-view fusion and target-view rendering processes via an end-to-end interpretable Transformer-based network. InNeRF enables the investigation of deep relationships between the target-rendering view and source views that were previously neglected by pooling-based fusion and fragmented rendering procedures. As a result, InNeRF improves model interpretability by enhancing the shape and appearance consistency of a 3D scene in both the surrounding view space and the ray-cast space. For a query rendering 3D point, InNeRF integrates both its projected 2D pixels from the surrounding source views and its adjacent 3D points along the query ray and simultaneously decodes this information into the query 3D point representation. Experiments show that InNeRF outperforms state-of-the-art image-based neural rendering methods in both scene-agnostic and per-scene finetuning scenarios, especially when there is a considerable disparity between source views and rendering views. The interpretation experiment shows that InNeRF can explain a query rendering process.

Deep CNN-based methods have so far achieved the state of the art results in multi-view 3D object reconstruction. Despite the considerable progress, the two core modules of these methods - view feature extraction and multi-view fusion, are usually investigated separately, and the relations among multiple input views are rarely explored. Inspired by the recent great success in Transformer models, we reformulate the multi-view 3D reconstruction as a sequence-to-sequence prediction problem and propose a framework named 3D Volume Transformer. Unlike previous CNN-based methods using a separate design, we unify the feature extraction and view fusion in a single Transformer network. A natural advantage of our design lies in the exploration of view-to-view relationships using self-attention among multiple unordered inputs. On ShapeNet - a large-scale 3D reconstruction benchmark, our method achieves a new state-of-the-art accuracy in multi-view reconstruction with fewer parameters (70% less) than CNN-based methods. Experimental results also suggest the strong scaling capability of our method. Our code will be made publicly available.

With the great success of convolutional neural networks (CNNs), interpretation of their internal network mechanism has been increasingly critical, while the network decision-making logic is still an open issue. In the bottom-up hierarchical logic of neuroscience, the decision-making process can be deduced from a series of sub-decision-making processes from low to high levels. Inspired by this, we propose the Concept-harmonized HierArchical INference (CHAIN) interpretation scheme. In CHAIN, a network decision-making process from shallow to deep layers is interpreted by the hierarchical backward inference based on visual concepts from high to low semantic levels. Firstly, we learned a general hierarchical visual-concept representation in CNN layered feature space by concept harmonizing model on a large concept dataset. Secondly, for interpreting a specific network decision-making process, we conduct the concept-harmonized hierarchical inference backward from the highest to the lowest semantic level. Specifically, the network learning for a target concept at a deeper layer is disassembled into that for concepts at shallower layers. Finally, a specific network decision-making process is explained as a form of concept-harmonized hierarchical inference, which is intuitively comparable to the bottom-up hierarchical visual recognition way. Quantitative and qualitative experiments demonstrate the effectiveness of the proposed CHAIN at both instance and class levels.

Despite the great success of deep convolutional neural networks (DCNNs) in computer vision tasks, their black-box aspect remains a critical concern. The interpretability of DCNN models has been attracting increasing attention. In this work, we propose a novel model, Feature-fLOW INterpretation (FLOWIN) model, to interpret a DCNN by its feature-flow. The FLOWIN can express deep-layer features as a sparse representation of shallow-layer features. Based on that, it distills the optimal feature-flow for the prediction of a given instance, starting from deep layers to shallow layers. Therefore, the FLOWIN can provide an instance-specific interpretation, which presents its feature-flow units and their interpretable meanings for its network decision. The FLOWIN can also give the quantitative interpretation in which the contribution of each flow unit in different layers is used to interpret the net decision. From the class-level view, we can further understand networks by studying feature-flows within and between classes. The FLOWIN not only provides the visualization of the feature-flow but also studies feature-flow quantitatively by investigating its density and similarity metrics. In our experiments, the FLOWIN is evaluated on different datasets and networks by quantitative and qualitative ways to show its interpretability.

With the increasing popularity of deep convolutional neural networks (DCNNs), in addition to achieving high accuracy, it becomes increasingly important to explain how DCNNs make their decisions. In this article, we propose a CHannel-wise disentangled InterPretation (CHIP) model for visual interpretations of DCNN predictions. The proposed model distills the class-discriminative importance of channels in DCNN by utilizing sparse regularization. We first introduce network perturbation to learn the CHIP model. The proposed model is capable to not only distill the global perspective knowledge from networks but also present class-discriminative visual interpretations for the predictions of networks. It is noteworthy that the CHIP model is able to interpret different layers of networks without retraining. By combining the distilled interpretation knowledge at different layers, we further propose the Refined CHIP visual interpretation that is both high-resolution and class-discriminative. Based on qualitative and quantitative experiments on different data sets and networks, the proposed model provides promising visual interpretations for network predictions in an image classification task compared with the existing visual interpretation methods. The proposed model also outperforms the related approaches in the ILSVRC 2015 weakly supervised localization task.

With the rapid development of deep learning models, their performances in various tasks have improved; meanwhile, their increasingly intricate architectures make them difficult to interpret. To tackle this challenge, model interpretability is essential and has been investigated in a wide range of applications. For end users, model interpretability can be used to build trust in the deployed machine learning models. For practitioners, interpretability plays a critical role in model explanation, model validation, and model improvement to develop a faithful model. In this paper, we propose a novel Multi-scale Interpretation (MINT) model for convolutional neural networks using both the perturbation-based and the gradient-based interpretation approaches. It learns the class-discriminative interpretable knowledge from the multi-scale perturbation of feature information in different layers of deep networks. The proposed MINT model provides the coarse-scale and the fine-scale interpretations for the attention in the deep layer and specific features in the shallow layer, respectively. Experimental results show that the MINT model presents the class-discriminative interpretation of the network decision and explains the significance of the hierarchical network structure.

A linear sparse unmixing method based on spectral library has been widely used to tackle the hyperspectral unmixing problem, under the assumption that the spectrum of each pixel in the hyperspectral scene can be expressed as a linear combination of pure endmembers in the spectral library. However, because of the ion (atom) substitution in the geological process, there often exists spectral variability between the measured endmembers in the real environment and corresponding ones in the spectral library, which poses a significant challenge to linear sparse unmixing. Physically, the substitution leads to the variation of absorption peaks of endmembers, making the spectral variation of sparse property. To address the above problem, we introduce redundant spectrum to represent the spectral variation caused by ion (atom) substitution and develop a sparse redundant unmixing model by adding the redundant regularization into the classical sparse regression formulation. Based on the alternating direction method of multipliers, we develop a unified algorithm called sparse redundant unmixing to obtain the solution. Both simulation experiment and real data experiment demonstrate that the proposed method can effectively use the redundant spectrum to address the spectral variation problem caused by the ion (atom) substitution.