Real-world scenes likely involve repetitive objects indicating that the reconstruction of the target object can be supplemented by the views of other identical objects. However, traditional 3D reconstruction methods do not take this a priori knowledge into account and fail to make full use of the available information. In this paper, we propose an object-aware viewpoint augmentation scheme for indoor compositional reconstruction. Within this scheme, a viewpoint supplementation strategy based on signed distance function and neural radiance fields is proposed to fully leverage the information from repetitive objects such that the occlusion problem is suppressed. Moreover, this scheme introduces monocular uncertainty priors and regional smoothness constraints to enhance the reconstruction accuracy of slender and thin structures and the smoothness of occluded background, respectively. Experimental results considering both synthetic and real-world scenes demonstrate that our method effectively improves the reconstruction quality of repetitive objects and background.

Recently, neural implicit surfaces have become popular for multi-view reconstruction. To facilitate practical applications like scene editing and manipulation, some works extend the framework with semantic masks input for the object-compositional reconstruction rather than the holistic perspective. Though achieving plausible disentanglement, the performance drops significantly when processing the indoor scenes where objects are usually partially observed. We propose RICO to address this by regularizing the unobservable regions for indoor compositional reconstruction. Our key idea is to first regularize the smoothness of the occluded background, which then in turn guides the foreground object reconstruction in unobservable regions based on the object-background relationship. Particularly, we regularize the geometry smoothness of occluded background patches. With the improved background surface, the signed distance function and the reversedly rendered depth of objects can be optimized to bound them within the background range. Extensive experiments show our method outperforms other methods on synthetic and real-world indoor scenes and prove the effectiveness of proposed regularizations. The code is available at https://github.com/kyleleey/RICO

The COVID-19 crisis has led to an unusually large number of commercial aircraft being currently parked or stored. For airlines, airports, and civil aviation authorities around the world, monitoring, and protecting these parked aircraft to prevent them from causing human-made damage are becoming urgent problems that are receiving increasing attention. In this study, we use thermal infrared monitoring videos to establish a framework for individual surveillance around parked aircraft by proposing a human action recognition (HAR) algorithm. As the focus of this article, the proposed HAR algorithm seamlessly integrates a preprocessing module in which a novel data structure is constructed to introduce spatiotemporal information of the action; a convolutional neural network-based module for spatial feature extraction; a triple-layer convolutional long short-term memory network for temporal feature extraction; and two fully connected layers for classification. Moreover, because no infrared dataset is available for the HAR task on airport grounds at nighttime, we present a dataset called IIAR-30, which consists of eight action categories that frequently occur on airport grounds and 2000 video clips. The experimental results on the IIAR-30 dataset demonstrated that the recognition accuracy of the proposed method was higher than 96%. We also further evaluated the effectiveness of the proposed method by comparing it with five baselines and four other methods.

Nowadays more and more urban residents are aware of the importance of the air quality to their health, especially who are living in the large cities that are seriously threatened by air pollution. Meanwhile, being limited by the spare sense nodes, the air quality information is very coarse in resolution, which brings urgent demands for high-resolution air quality data acquisition. In this paper, we refer the real-time and fine-gained air quality data in city-scale by employing the crowdsource automobiles as well as their built-in sensors, which significantly improves the sensing system's feasibility and practicability. The main idea of this paper is motivated by that the air component concentration within a vehicle is very similar to that of its nearby environment when the vehicle's windows are open, given the fact that the air will exchange between the inside and outside of the vehicle though the opening window. Therefore, this paper first develops an intelligent algorithm to detect vehicular air exchange state, then extracts the concentration of pollutant in the condition that the concentration trend is convergent after opening the windows, finally, the sensed convergent value is denoted as the equivalent air quality level of the surrounding environment. Based on our Internet of Things cloud platform, real-time air quality data streams from all over the city are collected and analyzed in our data center, and then a fine-gained city level air quality map can be exhibited elaborately. In order to demonstrate the effective- ness of the proposed method, experiments crowdsourcing 500 floating vehicles are conducted in Beijing city for three months to ubiquitously sample the air quality data. Evaluations of the algorithm's performance in comparison with the ground truth indicate the proposed system is practical for collecting air quality data in urban environments.

Question generation, the task of automatically creating questions that can be answered by a certain span of text within a given passage, is important for question-answering and conversational systems in digital assistants such as Alexa, Cortana, Google Assistant and Siri.Recent sequence to sequence neural models have outperformed previous rule-based systems.Existing models mainly focused on using one or two sentences as the input.Long text has posed challenges for sequence to sequence neural models in question generation -worse performances were reported if using the whole paragraph (with multiple sentences) as the input.In reality, however, it often requires the whole paragraph as context in order to generate high quality questions.In this paper, we propose a maxout pointer mechanism with gated self-attention encoder to address the challenges of processing long text inputs for question generation.With sentence-level inputs, our model outperforms previous approaches with either sentence-level or paragraph-level inputs.Furthermore, our model can effectively utilize paragraphs as inputs, pushing the state-of-the-art result from 13.9 to 16.3 (BLEU 4).

The recent years witnessed great improvements in systems of region-based face detection. However, the variations in occlusion, scale, illumination, pose and facial expressions make face detection in the wild still a challenge to be solved. In this paper, a Region-based Fully Convolutional Networks (R-FCN) based deep face detection framework is proposed. Several new techniques are utilized in our framework, including Deformable Convolutional Networks (DCN), Feature Pyramid Networks (FPN) and Focal Loss. Experiment results on three common challenging face detection benchmarks, FDDB, AFW and WIDER FACE, show the proposed approach is robust and performance outperforms most of previous methods, especially for addressing heavy occlusion, part deformation and complex perspective.

In traditional image retrieval techniques, the query results are severely affected when the images of varying illumination and scale, as well as occlusion and corrosion. Seeking to solve this problem, this paper proposed a novel multi-feature fusion and sparse coding based framework for image retrieval. In the framework, firstly, inherent features of an image are extracted, and then dictionary learning method is utilized to construct them to be dictionary features. Finally, the proposed framework introduces sparse representation model to measure the similarity between two images. The merit is that a feature descriptor is coded as a sparse linear combination with respect to dictionary feature so as to achieve efficient feature representation and robust similarity measure. In order to check the validity of the framework, this paper conducted two groups of experiments on Corel-1000 image dataset and the Stirmark benchmark based database respectively. Experimental results show that the proposed framework is much more effective than the state-of-the-art methods not only in traditional image dataset but also in varying image dataset.

Conventional digital display devices, due to their hardware limitations, can't represent the whole range of luminance in High Dynamic Range (HDR) images. In order to solve this incompatible problem, many tone mapping techniques were introduced to reproduce HDR images. Unlike the traditional methods applied in standard scale space, this paper proposes a novel affine invariant features-based tone mapping algorithm in affine Gaussian scale space. The reason of using this scale space is due to the fact that it is able to extract anisotropic feature regions in addition to traditional isotropic feature regions. Firstly, the proposed method extracts the anisotropic features from HDR images and reforms them to be isotropic by Fitting & Affine transformation. Then, dodging-and-burning processing is utilized to obtain base layer of HDR images. Finally, two-scale edge-preserving decomposition is employed to generate detail layer of HDR images and combine two layers to produce output images. Experimental results show that the proposed algorithm outperforms previous methods for reproducing the real scene of HDR images, especially for large perspective and scale transformed images which contain considerable anisotropic feature regions.

Context has been playing an increasingly important role to improve the object detection performance. In this paper we propose an effective representation, Multi-Order Contextual co-Occurrence (MOCO), to implicitly model the high level context using solely detection responses from a baseline object detector. The so-called (1st-order) context feature is computed as a set of randomized binary comparisons on the response map of the baseline object detector. The statistics of the 1st-order binary context features are further calculated to construct a high order co-occurrence descriptor. Combining the MOCO feature with the original image feature, we can evolve the baseline object detector to a stronger context aware detector. With the updated detector, we can continue the evolution till the contextual improvements saturate. Using the successful deformable-part-model detector [13] as the baseline detector, we test the proposed MOCO evolution framework on the PASCAL VOC 2007 dataset [8] and Caltech pedestrian dataset [7]: The proposed MOCO detector outperforms all known state-of-the-art approaches, contextually boosting deformable part models (ver. 5) [13] by 3.3% in mean average precision on the PASCAL 2007 dataset. For the Caltech pedestrian dataset, our method further reduces the log-average miss rate from 48% to 46% and the miss rate at 1 FPPI from 25% to 23%, compared with the best prior art [6].

Pedestrian detection from images is an important and yet challenging task. The conventional methods usually identify human figures using image features inside the local regions. In this paper we present that, besides the local features, context cues in the neighborhood provide important constraints that are not yet well utilized. We propose a framework to incorporate the context constraints for detection. First, we combine the local window with neighborhood windows to construct a multi-scale image context descriptor, designed to represent the contextual cues in spatial, scaling, and color spaces. Second, we develop an iterative classification algorithm called contextual boost. At each iteration, the classifier responses from the previous iteration across the neighborhood and multiple image scales, called classification context, are incorporated as additional features to learn a new classifier. The number of iterations is determined in the training process when the error rate converges. Since the classification context incorporates contextual cues from the neighborhood, through iterations it implicitly propagates to greater areas and thus provides more global constraints. We evaluate our method on the Caltech benchmark dataset [11]. The results confirm the advantages of the proposed framework. Compared with state of the arts, our method reduces the miss rate from 29% by [30] to 25% at 1 false positive per image (FPPI).

We address the problem of motion deblurring using coded exposure. This approach allows for accurate estimation of a sharp latent image via well-posed deconvolution and avoids lost image content that cannot be recovered from images acquired with a traditional shutter. Previous work in this area has used either manual user input or alpha matting approaches to estimate the coded exposure Point Spread Function (PSF) from the captured image. In order to automate deblurring and to avoid the limitations of matting approaches, we propose a Fourier-domain statistical approach to coded exposure PSF estimation that allows us to estimate the latent image in cases of constant velocity, constant acceleration, and harmonic motion. We further demonstrate that previously used criteria to choose a coded exposure PSF do not produce one with optimal reconstruction error, and that an additional 30 percent reduction in Root Mean Squared Error (RMSE) of the latent image estimate can be achieved by incorporating natural image statistics.

Content-aware image retargeting has attracted a lot of interests recently. The key and most challenging issue for this task is how to balance the tradeoff between preserving the important contents and minimizing the visual distortions on the consistency of the image structure. In this paper we present a novel filtering-based technique to tackle this issue, called ”importance filtering”. Specifically, we first filter the image saliency guided by the image itself to achieve a structure-consistent importance map. We then use the pixel importance as the key constraint to compute the gradient map of pixel shifts from the original resolution to the target. Finally, we integrate the shift gradient across the image using a weighted filter to construct a smooth shift map and render the target image. The weight is again controlled by the pixel importance. The two filtering processes enforce to maintain the structural consistency and yet preserve the important contents in the target image. Furthermore, the simple nature of filter operations allows highly efficient implementation for real-time applications and easy extension to video retargeting, as the structural constraints from the original image naturally convey the temporal coherence between frames. The effectiveness and efficiency of our importance filtering algorithm are confirmed in extensive experiments.

We present a novel theory for characterizing defocus blurs in multi-perspective cameras such as catadioptric mirrors. Our approach studies how multi-perspective ray geometry transforms under the thin lens. We first use the General Linear Cameras (GLCs) to approximate the incident multi-perspective rays to the lens and then apply a Thin Lens Operator (TLO) to map an incident GLC to the exit GLC. To study defocus blurs caused by the GLC rays, we further introduce a new Ray Spread Function (RSF) model analogous the Point Spread Function (PSF). While PSF models defocus blurs caused by a 3D scene point, RSF models blurs spread by rays. We derive closed form RSFs for incident GLC rays, and we show that for catadioptric cameras with a circular aperture, the RSF can be effectively approximated as a single or mixtures of elliptic-shaped kernels. We apply our method for predicting defocus blurs on commonly used catadioptric cameras and for reducing de-focus blurs in catadioptric projections. Experiments on synthetic and real data demonstrate the accuracy and general applicability of our approach.

Acquiring dynamic 3D fluid surfaces is a challenging problem in computer vision. Single or stereo camera based solutions are sensitive to refraction distortions, fast fluid motions, and calibration errors. In this paper, we present a multi-view based solution for robustly capturing fast evolving fluid wavefronts. We first construct a portable, 3×3 camera array system as the main acquisition device. We elaborately design the system to allow high-resolution and high-speed capture. To recover fluid surfaces, we place a known pattern beneath the surface and position the camera array on top to observe the pattern. By tracking the distorted feature points over time and across cameras, we obtain spatial-temporal correspondence maps and we use them for specular carving to reconstruct the time-varying surface. In case one of the cameras loses track due to distortions or blurs, we use the rest of the cameras to construct the surface and then apply multi-perspective warping to locate the lost-track feature points so that we can continue using the camera in later frames. Our experiments on synthetic and real data demonstrate that our multi-view framework is robust and reliable.

Abstract We present a novel image‐based technique for modeling complex unfoliaged trees. Existing tree modeling tools either require capturing a large number of views for dense 3D reconstruction or rely on user inputs and botanic rules to synthesize natural‐looking tree geometry. In this paper, we focus on faithfully recovering real instead of realistically‐looking tree geometry from a sparse set of images. Our solution directly integrates 2D/3D tree topology as shape priors into the modeling process. For each input view, we first estimate a 2D skeleton graph from its matte image and then find a 2D skeleton tree from the graph by imposing tree topology. We develop a simple but effective technique for computing the optimal 3D skeleton tree most consistent with the 2D skeletons. For each edge in the 3D skeleton tree, we further apply volumetric reconstruction to recover its corresponding curved branch. Finally, we use piecewise cylinders to approximate each branch from the volumetric results. We demonstrate our framework on a variety of trees to illustrate the robustness and usefulness of our technique.

We present a catadioptric projector analogous to a catadioptric camera by combining a commodity digital projector with additional optical units. We show that, by using specially shaped reflectors/refractors, catadioptric projectors can offer an unprecedented level of flexibility in aspect ratio, size, and field of view. We also present efficient algorithms to reduce projection artifacts in catadioptric projectors, such as distortions, scattering, and defocusing. Instead of recovering the reflector/refractor geometry, our approach directly models the light transport between the projector and the viewpoint using the light transport matrix (LTM). We show how to efficiently approximate the pseudo inverse of the LTM and apply it to find the optimal input image that produces least projection distortions. Furthermore, we present a projection defocus analysis for reflector and thin refractor based catadioptric projectors. We show that defocus blur can be interpreted as spatially-varying Gaussian blurs on the input image. We then measure the kernels directly from the LTM and apply deconvolution to optimize the input image. We demonstrate the practical uses of catadioptric projectors in panoramic and omni-directional projections. Our new system achieves much wider field-of-view projection while maintaining sharpness and low geometric and photometric distortions.

We present a new shape-from-distortion framework for recovering specular (reflective/refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection/refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces.

Stereo matching and volumetric reconstruction are the most explored 3D scene recovery techniques in computer vision. Many existing approaches assume perspective input images and use the epipolar constraint to reduce the search space and improve the accuracy. In this paper we present a novel framework that uses multi-perspective cameras for stereo matching and volumetric reconstruction. Our approach first decomposes a multi-perspective camera into piecewise primitive General Linear Cameras or GLCs. A pair of GLCs in general do not satisfy the epipolar constraint. However, they still form a nearly stereo pair. We develop a new Graph-Cut-based algorithm to account for the slight vertical parallax using the GLC ray geometry. We show that the recovered pseudo disparity map conveys important depth cues analogous to perspective stereo matching. To more accurately reconstruct a 3D scene, we develop a new multi-perspective volumetric reconstruction method. We discretize the scene into voxels and apply the GLC back-projections to map the voxel onto each input multi-perspective camera. Finally, we apply the graph-cut algorithm to optimize the 3D embedded voxel graph. We demonstrate our algorithms on both synthetic and real multi-perspective cameras. Experimental results show that our methods are robust and reliable.

We consider the problem of capturing shape characteristics on specular (refractive and reflective) surfaces that are nearly flat. These surfaces are difficult to model using traditional methods based on reconstructing the surface positions and normals. These lower-order shape attributes provide little information to identify important surface characteristics related to distortions. In this paper, we present a framework for recovering the higher-order geometry attributes of specular surfaces. Our method models local reflections and refractions in terms of a special class of multiperspective cameras called the general linear cameras (GLCs). We then develop a new theory that correlates the higher-order differential geometry attributes with the local GLCs. Specifically, we show that Gaussian and mean curvature can be directly derived from the camera intrinsics of the local GLCs. We validate this theory on both synthetic and real-world specular surfaces. Our method places a known pattern in front of a reflective surface or beneath a refractive surface and captures a distorted image on the surface. We then compute the optimal GLC using a sparse set of correspondences and recover the curvatures from the GLC. Experiments demonstrate that our methods are robust and highly accurate.

Human stereo vision works by fusing a pair of perspective images with a purely horizontal parallax. Recent developments suggest that very few varieties of multiperspective stereo pairs exist. In this paper, we introduce a new stereo model, which we call epsilon stereo pairs, for fusing a broader class of multiperspective images. An epsilon stereo pair consists of two images with a slight vertical parallax. We show many multiperspective camera pairs that do not satisfy the stereo constraint can still form epsilon stereo pairs. We then introduce a new ray-space warping algorithm to minimize stereo inconsistencies in an epsilon pair using multiperspective collineations. This makes epsilon stereo model a promising tool for synthesizing close-to-stereo fusions from many non-stereo pairs.

BACKGROUND: Mean-based clustering algorithms such as bisecting k-means generally lack robustness. Although componentwise median is a more robust alternative, it can be a poor center representative for high dimensional data. We need a new algorithm that is robust and works well in high dimensional data sets e.g. gene expression data. RESULTS: Here we propose a new robust divisive clustering algorithm, the bisecting k-spatialMedian, based on the statistical spatial depth. A new subcluster selection rule, Relative Average Depth, is also introduced. We demonstrate that the proposed clustering algorithm outperforms the componentwise-median-based bisecting k-median algorithm for high dimension and low sample size (HDLSS) data via applications of the algorithms on two real HDLSS gene expression data sets. When further applied on noisy real data sets, the proposed algorithm compares favorably in terms of robustness with the componentwise-median-based bisecting k-median algorithm. CONCLUSION: Statistical data depths provide an alternative way to find the "center" of multivariate data sets and are useful and robust for clustering.

BACKGROUND: Recursive Feature Elimination is a common and well-studied method for reducing the number of attributes used for further analysis or development of prediction models. The effectiveness of the RFE algorithm is generally considered excellent, but the primary obstacle in using it is the amount of computational power required. RESULTS: Here we introduce a variant of RFE which employs ideas from simulated annealing. The goal of the algorithm is to improve the computational performance of recursive feature elimination by eliminating chunks of features at a time with as little effect on the quality of the reduced feature set as possible. The algorithm has been tested on several large gene expression data sets. The RFE algorithm is implemented using a Support Vector Machine to assist in identifying the least useful gene(s) to eliminate. CONCLUSION: The algorithm is simple and efficient and generates a set of attributes that is very similar to the set produced by RFE.