Small object detection is a very challenging yet practical vision task. With deep network-based methods, the contextual information of small objects may disappear when the network goes deeper. An intuitive solution to alleviate this issue is to increase the input resolution, however, it will aggravate the large variant of object scale and introduce unbearable computation cost. To leverage the benefits of high-resolution images without bringing up new problems, we propose a High-Resolution Detection Network (HRDNet) which takes multiple resolution inputs with multi-depth backbones. Meanwhile, we propose the Multi-Depth Image Pyramid Network (MD-IPN) and Multi-Scale Feature Pyramid Network (MS-FPN). The MD-IPN maintains multiple position information using multiple depth backbones. Specifically, high-resolution input will be fed into a shallow network to reserve more positional information and reduce computational costs, while low-resolution input will be fed into a deep network to extract more semantics. By extracting various features from high to low resolutions, the MD-IPN can improve the performance of small object detection and maintain the performance of middle and large objects. Additionally, MS-FPN is introduced to align and fuse multi-scale feature groups generated by MD-IPN to reduce the information imbalance. Extensive experiments are conducted on the COCO2017 and the typical small object dataset, VisDrone 2019. Notably, our HRDNet achieves the state-of-the-art on these two datasets with significant improvements on small objects.

Trajectory prediction is crucial for autonomous vehicles. The planning system not only needs to know the current state of the surrounding objects but also their possible states in the future. As for vehicles, their trajectories are significantly influenced by the lane geometry and how to effectively use the lane information is of active interest. Most of the existing works use rasterized maps to explore road information, which does not distinguish different lanes. In this paper, we propose a novel instance-aware representation for lane representation. By integrating the lane features and trajectory features, a goal-oriented lane attention module is proposed to predict the future locations of the vehicle. We show that the proposed lane representation together with the lane attention module can be integrated into the widely used encoder-decoder framework to generate diverse predictions. Most importantly, each generated trajectory is associated with a probability to handle the uncertainty. Our method does not suffer from collapsing to one behavior modal and can cover diverse possibilities. Extensive experiments and ablation studies on the benchmark datasets corroborate the effectiveness of our proposed method. Notably, our proposed method ranks third place in the Argoverse motion forecasting competition at NeurIPS 20191.

Recent voxel-based 3D object detectors for autonomous vehicles learn point cloud representations either from bird eye view (BEV) or range view (RV, a.k.a. the perspective view). However, each view has its own strengths and weaknesses. In this paper, we present a novel framework to unify and leverage the benefits from both BEV and RV. The widely-used cuboid-shaped voxels in Cartesian coordinate system only benefit learning BEV feature map. Therefore, to enable learning both BEV and RV feature maps, we introduce Hybrid-Cylindrical-Spherical voxelization. Our findings show that simply adding detection on another view as auxiliary supervision will lead to poor performance. We proposed a pair of cross-view transformers to transform the feature maps into the other view and introduce cross-view consistency loss on them. Comprehensive experiments on the challenging NuScenes Dataset validate the effectiveness of our proposed method by virtue of joint optimization and complementary information on both views. Remarkably, our approach achieved mAP of 55.8%, outperforming all published approaches by at least 3% in overall performance and up to 16.5% in safety-crucial categories like cyclist.

Learning robotic grasps from visual observations is a promising yet challenging task. Recent research shows its great potential by preparing and learning from large-scale synthetic datasets. For the popular, 6 degree-of-freedom (6-DOF) grasp setting of parallel-jaw gripper, most of existing methods take the strategy of heuristically sampling grasp candidates and then evaluating them using learned scoring functions. This strategy is limited in terms of the conflict between sampling efficiency and coverage of optimal grasps. To this end, we propose in this work a novel, end-to-end \emph{Grasp Proposal Network (GPNet)}, to predict a diverse set of 6-DOF grasps for an unseen object observed from a single and unknown camera view. GPNet builds on a key design of grasp proposal module that defines \emph{anchors of grasp centers} at discrete but regular 3D grid corners, which is flexible to support either more precise or more diverse grasp predictions. To test GPNet, we contribute a synthetic dataset of 6-DOF object grasps; evaluation is conducted using rule-based criteria, simulation test, and real test. Comparative results show the advantage of our methods over existing ones. Notably, GPNet gains better simulation results via the specified coverage, which helps achieve a ready translation in real test. Our code and dataset are available on \url{https://github.com/CZ-Wu/GPNet}.

TV-L1is a classical diffusion-reaction model for low-level vision tasks, which can be solved by a duality based iterative algorithm. Considering the recent success of end-to-end learned representations, we propose a TV-LSTM network to unfold the duality based iterations into long short-term memory (LSTM) cells. To provide a trainable network, we relax the difference operators in the gate and cell update of TV-LSTM to trainable parameters. Then, the proposed end-to-end trainable TV-LSTMs can be naturally connected with various task-specific networks, e.g., optical flow estimation and image decomposition. Extensive experiments on optical flow estimation and structure + texture decomposition have demonstrated the effectiveness and efficiency of the proposed method.

Urrently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer. This is usually actualized through feedforward multilayer neural networks, e.g. ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach in which the representation is formed in an iterative manner based on a feedback received from previous iterations output. We establish that a feedback based approach has several core advantages over feedforward: it enables making early predictions at the query time, its output naturally conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We observe that feedback develops a considerably different representation compared to feedforward counterparts, in line with the aforementioned advantages. We provide a general feedback based learning architecture, instantiated using existing RNNs, with the endpoint results on par or better than existing feedforward networks and the addition of the above advantages.

Human actions captured in video sequences are threedimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short- Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long.,,In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

In this work, we present a new multiple channel feature called Deep Compact Channel Feature (DCCF), which generates a compact, discriminative feature representation by a pre-trained deep encoder-decoder. With the combination of DCCF and boosted decision trees, a new object detector is proposed which achieved outstanding performance on standard pedestrian dataset INRIA and Caltech. Furthermore, a large scale and challenging Chinese Traffic Sign Detection benchmark is constructed. DCCF and other related methods are evaluated on this dataset. The dataset and baselines are available online.

Human actions in video sequences are three-dimensional (3D) spatio-temporal signals characterizing both the visual appearance and motion dynamics of the involved humans and objects. Inspired by the success of convolutional neural networks (CNN) for image classification, recent attempts have been made to learn 3D CNNs for recognizing human actions in videos. However, partly due to the high complexity of training 3D convolution kernels and the need for large quantities of training videos, only limited success has been reported. This has triggered us to investigate in this paper a new deep architecture which can handle 3D signals more effectively. Specifically, we propose factorized spatio-temporal convolutional networks (FstCN) that factorize the original 3D convolution kernel learning as a sequential process of learning 2D spatial kernels in the lower layers (called spatial convolutional layers), followed by learning 1D temporal kernels in the upper layers (called temporal convolutional layers). We introduce a novel transformation and permutation operator to make factorization in FstCN possible. Moreover, to address the issue of sequence alignment, we propose an effective training and inference strategy based on sampling multiple video clips from a given action video sequence. We have tested FstCN on two commonly used benchmark datasets (UCF-101 and HMDB-51). Without using auxiliary training videos to boost the performance, FstCN outperforms existing CNN based methods and achieves comparable performance with a recent method that benefits from using auxiliary training videos.

Most of the previous work on video action recognition use complex hand-designed local features, such as SIFT, HOG and SURF, but these approaches are implemented sophisticatedly and difficult to be extended to other sensor modalities. Recent studies discover that there are no universally best hand-engineered features for all datasets, and learning features directly from the data may be more advantageous. One such endeavor is Slow Feature Analysis (SFA) proposed by Wiskott and Sejnowski [33]. SFA can learn the invariant and slowly varying features from input signals and has been proved to be valuable in human action recognition [34]. It is also observed that the multi-layer feature representation has succeeded remarkably in widespread machine learning applications. In this paper, we propose to combine SFA with deep learning techniques to learn hierarchical representations from the video data itself. Specifically, we use a two-layered SFA learning structure with 3D convolution and max pooling operations to scale up the method to large inputs and capture abstract and structural features from the video. Thus, the proposed method is suitable for action recognition. At the same time, sharing the same merits of deep learning, the proposed method is generic and fully automated. Our classification results on Hollywood2, KTH and UCF Sports are competitive with previously published results. To highlight some, on the KTH dataset, our recognition rate shows approximately 1% improvement in comparison to state-of-the-art methods even without supervision or dense sampling.

This paper explores a novel rate control scheme for HEVC, based on Pre-analysis Sum of Absolute Transformed Differences (pSATD). The needs for improving the rate control method of HEVC have been observed since the latest method proposed in JCT-VC H213 is still using the classical quadratic model which is implemented in H.264/AVC. The same rate control scheme on different coding structures will definitely lead to a poor performance. The proposed method aims at selecting accurate quantization parameters (QP) for inter frames according to the target bit rate. After encoding video sequences exhaustively, a strong linear relationship between the pSATD and bpp (bit per pixel) is revealed. Thus, a rate control scheme for HEVC with parameters updating adaptively is proposed accordingly. Apart from that, a new rate and distortion model are established using pSATD and Quantization Step Size (QStep). By minimizing the cost function built based on the proposed rate-distortion (R-D) model, the optimization problem can be solved properly. According to the optimal solution, a bit allocation scheme with consideration of pSATD and buffer status is formed. The experimental results illustrate that the proposed method outperforms JCT-VC H213 implemented in HEVC reference software for all test sequences, up to 19% coding gain can be achieved.

In this paper, we present a novel framework for visual classification based on hierarchical image decomposition and hybrid midlevel feature extraction. Unlike most midlevel feature learning methods, which focus on the process of coding or pooling, we emphasize that the mechanism of image composition also strongly influences the feature extraction. To effectively explore the image content for the feature extraction, we model a multiplicity feature representation mechanism through meaningful hierarchical image decomposition followed by a fusion step. In particularly, we first propose a new hierarchical image decomposition approach in which each image is decomposed into a series of hierarchical semantical components, i.e, the structure and texture images. Then, different feature extraction schemes can be adopted to match the decomposed structure and texture processes in a dissociative manner. Here, two schemes are explored to produce property related feature representations. One is based on a single-stage network over hand-crafted features and the other is based on a multistage network, which can learn features from raw pixels automatically. Finally, those multiple midlevel features are incorporated by solving a multiple kernel learning task. Extensive experiments are conducted on several challenging data sets for visual classification, and experimental results demonstrate the effectiveness of the proposed method.

The deblocking filter in H.264/AVC is one of the most time consuming part of video decoder as its high content adaptation and data dependency lead to lots of computation. In this paper, we propose a novel parallel deblocking filter design based on the H.264/AVC video coding standard, taking the advantage that the data dependency of the deblocking filter are “periodic” in one dimension. Our proposed architecture successfully reduces the dependency between horizontal and vertical filters and utilizes the “periodic” property to achieve pixel-level parallelism. Algorithm analysis and experiment results on JM and GPU show that the proposed deblocking filter keeps as good a coding efficiency as that in H.264/AVC, and its high parallelism is suitable and promising in multi-core/multi-thread computing.

Motion estimation (ME) is one of the most time consuming parts in video coding standard. As fast integer-pixel ME algorithm becoming more and more powerful, it is important to develop fast sub-pixel ME algorithm since the computational complexity of sub-pixel ME compared to integer-pixel ME has become relatively significant. In this paper, a novel fast sub-pixel ME algorithm is proposed. This algorithm first approximates the error surface of the sub-pixel position by a second order function and predicts the minimum point by minimizing the function at half-pixel accuracy. Then another second order approximation within a smaller area which is determined by the previous step is modeled to predict the best sub-pixel position. Experimental results show that the proposed method can reduce the sub-pixel search points significantly with negligible quality degradation.

Motion estimation (ME) is one of the key elements in video coding standard which eliminates the temporal redundancies between successive frames. In recent international video coding standards, sub-pixel ME is proposed for its excellent coding performance. Compared with integer-pixel ME, sub-pixel ME needs interpolation to get the value in sub-pixel position. Also, Hadamard transform will be applied in order to achieve better performance. Therefore, it is becoming more and more critical to develop fast sub-pixel ME algorithms. In this paper, a novel fast sub-pixel ME algorithm is proposed which makes full use of 8 neighboring integer-pixel points. This algorithm models the error surface in sub-pixel position by a second order function with five parameters two times to predict the best sub-pixel position. Experimental results show that the proposed method can reduce the complexity significantly with negligible quality degradation.

In depth map coding for 3D video coding systems, coding errors in edges can severely affect the synthesis quality. Edge errors mainly compose of two parts: one is blurring and ringing artifact around sharp edge and the other is fake edge caused by blocking artifact. In this paper, we propose an adaptive depth map filter to remove blocking artifacts while preserving depth edges. The proposed filter is designed based on bilateral filter, in which the range kernel parameter is changed adaptively considering the strength of edges and blocking artifacts. Experimental results demonstrate that the proposed depth map filter can achieve up to 0.41 dB gain on the synthesis quality compared to the deblocking filter in MVC at low bit rate.

Medical imaging applications produce a huge amount of similar images. Instead of compressing each image individually, set redundancy compression (SRC) methods remove the inter image redundancy and reduce storage. However, in the previous SRC methods — -MMD, MMP and Centroid methods, the prediction templates for extracting set redundancy are not very efficient, especially when image sets are very large with several clusters. In this paper, inspired by face recognition techniques, a novel lossless SRC method is derived based onDCT pyramid multi-level low frequency template. The approximation subband is used as a prediction template for each image to calculate the residue. Intra prediction is also used to reduce the entropy of the residues. Experiments with 3 sets of MR brain images demonstrate the efficiency of our proposed algorithm in respect to bits/pixel (bpp).

Adaptive quantization matrix design for different block sizes is one of the possible methods to improve the RD performance in video coding and has recently attracted the focus of many researchers. In this paper, we first analyze the shortcomings of the evenly distributed distortion method which was proposed recently. In order to tackle these problems, we propose two modified methods, method I with relaxed distortion constraints and method II is iterative boundary distortion minimization problem considering variance adaptively. Both problems can be solved using convex optimization effectively and efficiently. Simulations have been conducted based on HM4.0, which is the reference software of the latest High Efficiency Video Coding (HEVC). Simulation results show the effect of our proposed methods. Both methods show their significance when evaluated by RD performance.

In video coding standard, motion estimation (ME) always plays an important role in reducing temporal redundancies at the expense of higher computational complexity. Many fast ME algorithms have been proposed to reduce the coding complexity. Some papers focus on applying specific search patterns to reduce the search points within a fixed search range (SR). But there are only a few of them trying to reduce the size of SR. In this paper, an adaptive SR algorithm is presented. Cauchy distribution is used to model the SR for one frame and the information of motion vector differences in the neighboring blocks is used to adjust the SR for a particular block. Experimental results show that the proposed algorithm can reduce the size of SR significantly with negligible quality degradation.

In this paper, we propose a simple but effective adaptive bilateral filter considering local characteristics. The presented method exploits the local gaussian gradient information of the processing image and applies bilateral filter with changing the range filter parameter $\sigma_{r}$ adaptively. The proposed adaptive bilateral filter could preserve more details of the image compared with original bilateral filter in edge or texture regions. What is more, it is more robust to the noise. Also compared with the original bilateral filter and the state-of-art wavelet de-noising method, the proposed method also can effectively remove signal noise while preserving the original structure.