Channel estimation is fundamental to leveraging the performance gains of reconfigurable intelligent surfaces (RISs), one of the key technologies for 6G. Due to the passive nature of RIS, most current research focuses on cascaded channel estimation. However, individual channel information is crucial for practical applications such as flexible precoding design. In this paper, we propose a hybrid RIS architecture integrated with dynamically controllable active elements, which reduces the cost of RIS deployment. Based on this novel architecture, we introduce an atomic norm minimization (ANM)-based individual channel estimation method, exploiting the sparse characteristics of high frequency channels. We theoretically prove that our proposed method retains its applicability even in the presence of random element damage in RIS. Furthermore, we extend the solution for individual channel estimation to passive RIS scenario under the mild condition that the locations of base station and RIS are known. Simulation experiments demonstrate that the proposed methods achieve super-resolution channel estimation, surpassing the performance of existing methods such as orthogonal matching pursuit.

Traffic scene segmentation is an important visual perception process to provide strong support for the decision-making of autonomous driving systems. The traffic scene is an open environment that is constantly changing, and the segmentation model needs to have the ability of domain incremental learning to maintain stable performance in the changing environment. The main challenges include the diversity of the traffic scene and the accumulation of forgetting about the previous traffic scenes. In this work, an adapter-based network model is proposed to solve the domain-incremental traffic scene segmentation task. Specifically, the domain-aware adapter (DAA) module is proposed, which divides the model parameters into domain-shared and domain-specific, so that the model can handle the information of all learned domains by dynamically expanding parameters. To further alleviate catastrophic forgetting, the inter-class correlation enhancement (ICE) module is proposed, which utilizes inter-class correlation to improve segmentation accuracy for single domain and knowledge transfer between domains. Extensive experimental results show that the proposed method can achieve promising results for retaining the competitive performance for both new and old domains.

Most existing near-field (NF) source localization methods are based on uniform/sparse symmetric linear arrays. But planar arrays will be more common in the future. In this paper, we propose an NF source localization method for rectangular array, which can be uniform rectangular array (URA) or centrally symmetric sparse rectangular array (SRA). We first use the forth-order cumulant to formulate a low-rank matrix reconstruction (LRMR) problem for angle estimation, and then, we use 1D-MUSIC to find the range estimates. We also consider the dual problem of the LRMR problem to reduce computations. Our method shows similar estimation accuracy to the maximum likelihood method while enjoys much less computations. Simulations results are provided to demonstrate the advantages of our method.

In remote sensing earth observation applications, there is often a trade-off between automated processing and improved observation accuracy. A key issue in this contradiction is the underutilization of rich knowledge embedded in historical remote sensing data. To address this, this paper proposes an object-level multi-source transfer learning method (OMSTL) that effectively leverages the knowledge in historical remote sensing data for unsupervised land object discrimination. This method effectively integrates multi-source heterogeneous features and represents land object knowledge, enabling reliable knowledge transfer. Multi-source transfer learning utilizes knowledge from remote sensing images at different times and locations, significantly enhancing transfer stability. The method's effectiveness and potential were demonstrated through experiments using five optical remote sensing images (GF-1B, GF-1D, GF-6, GF-7, ZY-3) and their corresponding SAR images (Sentinel-1).

This work presents a new registration method that especially designed for low-overlapping partial point clouds. Based on the assumption that the partial point clouds to be registered belong to the same target, the proposed Mutual a Priori based Completion (MPC) method uses the partial point clouds to be registered as the a priori for each other to expend each individual point clouds. The main challenge of this mutual reference approach is that the partial clouds without spatial alignment cannot provide reliable completion reference for each. Based on the mutual information maximization, a progressive completion structure is developed to make the alignment in the spatial, feature and completion space respectively. Experiments on public datasets show encouraging results. Especially for the ultra-low overlap cases, compared with the state-of-the-art (SOTA) models, the size of overlapping regions can be increased by about 14.0%, and the rotation and translation error can be reduced by 30.8% and 57.7% respectively.

This work presents a new completion method that specifically designed for low-overlapping partial point cloud registration. Based on the assumption that the candidate partial point clouds to be registered belong to the same target, the proposed mutual prior based completion (MPC) method uses these candidate partial point clouds as completion reference for each other to extend their overlapping regions. Without relying on shape prior knowledge, MPC can work for different types of point clouds, such as object, room scene, and street view. The main challenge of this mutual reference approach is that partial clouds without spatial alignment cannot provide a reliable completion reference. Based on the mutual information maximization, a progressive completion structure is developed to achieve pose, feature representation and completion alignment between input point clouds. Experiments on public datasets show encouraging results. Especially for the low-overlapping cases, compared with the state-of-the-art (SOTA) models, the size of overlapping regions can be increased by about 15.0%, and the rotation and translation error can be reduced by 30.8% and 57.7% respectively.

A simple method is presented for road scene anomaly detection, which is particularly important in autonomous driving. Without the prior knowledge about the appearance and size, anomaly detection is formulated as an OoD (Out-of-Distribution) detection task. Epistemic uncertainty indicates that when a model is trained with finite training data, it will fail to yield reliable prediction if the testing data is far from the training data distribution (OoD). Deviating from previous works, instead of modelling the uncertainty of a single model explicitly, this work casts the individual uncertainty into an ensemble uncertainty metric by measuring the discrepancy between the different models. The benefits are: 1) Following a meta-learning framework, outputs from the off-the-shelf models can be used directly, which results in a very small training cost; 2) With the ensemble approach, the model is especially robust to dataset shift. Experimental results show that the performance of the proposed method is competitive: 1) Its APs (Average precisions) are 3.5%, 3.1%, and 5.2% higher than the best SOTAs (State-of-the-arts) on LaF (Lost-and-Found), RA (Road Anomaly), and RO (Road Obstacles) respectively; 2) For all datasets with different road conditions, it obtains consistent performance. (Code is available at https://github.com/yazhouliu/EUG).

This paper addresses the problem of normalized scene flow (NSF): given a pair of RGB video frames, estimating the 3D motion, which consisted of optical flow and motion-in-depth estimation. NSF is a powerful tool for action prediction and autonomous robot navigation, presenting the advantage of only needing a monocular and uncalibrated camera. However, most existing methods directly regress motion-in-depth from two RGB frames or optical flow, resulting in sub-accurate and non-robust results. Our key insight is the scale matching scheme-establishing correlations between two frames containing objects in different scales, to estimate dense and continuous motion-in-depth. Based on the scale matching, we propose a unified framework: Scale-flow, which combines scale matching and optical flow estimation. This combination makes optical flow estimation can use dense and continuous scale information for the first time, so that the moving foreground objects can be estimated more accurately. On KITTI, our monocular approach achieves the lowest error in the foreground scene flow task, even compared with the multi-camera method. Moreover, on the motion-in-depth estimation task, Scale-flow reduces the error by 34% compared with the best-published method. Code will be available.

The dependency between global and local information can provide important contextual cues for semantic segmentation. Existing attention methods capture this dependency by calculating the pixel wise correlation between the learnt feature maps, which is of high space and time complexity. In this article, a new attention module, covariance attention, is presented, and which is interesting in the following aspects: 1) covariance matrix is used as a new attention module to model the global and local dependency for the feature maps and the local-global dependency is formulated as a simple matrix projection process; 2) since covariance matrix can encode the joint distribution information for the heterogeneous yet complementary statistics, the hand-engineered features are combined with the learnt features effectively using covariance matrix to boost the segmentation performance; 3) a covariance attention mechanism based semantic segmentation framework, CANet, is proposed and very competitive performance has been obtained. Comparisons with the state-of-the-art methods reveal the superiority of the proposed method.

This article presents a modular lightweight network model for road objects detection, such as car, pedestrian, and cyclist, especially when they are far away from the camera and their sizes are small. Great advances have been made for the deep networks, but small objects detection is still a challenging task. In order to solve this problem, a majority of existing methods utilize complicated network or bigger image size, which generally leads to higher computation cost. The proposed network model is referred to as modular feature fusion detector (MFFD), using a fast and efficient network architecture for detecting small objects. The contribution lies in the following aspects: 1) two base modules have been designed for efficient computation: a) Front module reduces the information loss from raw input images and b) Tinier module decreases the model size and computation cost, while ensuring the detection accuracy; 2) by stacking the base modules, we design a context features fusion framework for multiscale object detection; and 3) the proposed method is efficient in terms of model size and computation cost, which is applicable for resource-limited devices, such as embedded systems for advanced driver-assistance systems (ADASs). Comparisons with the state-of-the-art on the challenging KITTI dataset reveal the superiority of the proposed method. Especially, 100 ft/s can be achieved on the embedded GPUs such as Jetson TX2.

Real-time semantic segmentation is a challenging task as both segmentation accuracy and inference speed need to be considered at the same time. In this paper, we present a Dual Context Network (DCN et) to address this challenge. It contains two independent sub-networks: Region Context Network and Pixel Context Network. Region Context Network is main network with low-resolution input and feature re-weighting module to achieve sufficient receptive field. Meanwhile, utilizing Pixel Context Network with location attention module to capture the location dependencies of each pixel for assisting the main network to recover spatial detail. A contextual feature fusion is introduced to combine output features of these two subnetworks. The experiments show that DCNet can achieve high-quality segmentation while keeping a high speed. Specifically, for Cityscapes test dataset, we achieve 76.1 % Mean IOU with the speed of 82 FPS on a single GTX 2080Ti GPU when using ResNet50 as backbone, and 71.2% Mean IOU with the speed of 142 FPS when using ResNet18 as backbone.

Three‐dimensional (3D) driver pose estimation is a promising and challenging problem for computer–human interaction. Recently convolutional neural networks have been introduced into 3D pose estimation, but these methods have the problem of slow running speed and are not suitable for driving scenario. In this study, the proposed method is based on two types of inputs, infrared image and point cloud obtained from time‐of‐flight camera. The authors propose a joint 2D–3D network incorporating image‐based and point‐based feature to promote the performance of 3D human pose estimation and run on a high speed. For point cloud with invalid points, the authors first do preprocess and then design a denoising module to handle this problem. Experiments on private driver data set and public Invariant‐Top View data set show that the proposed method achieves efficient and competitive performance on 3D human pose estimation.

3D driver pose estimation is a promising and challenging problem for computer-human interaction. Recently convolutional neural networks (CNNs) have been introduced into 3D pose estimation, but these methods have the problem of slow running speed and are not suitable for driving scenario. In this paper, our method is based on two type of inputs, IR image and point cloud obtained from TOF camera. We propose a Joint 2D-3D network incorporating image-based and point-based feature to promote the performance of 3D human pose estimation and run in a high speed. For point cloud with invalid points, we firstly do preprocess and then design a denoising module to handle this problem. Experiments on private Driver dataset and public ITOP dataset show that our method achieves efficient and competitive performance on 3D human pose estimation.

Driver poses recognition contains three tasks such as body joint, head angle, and face landmark estimation, which is of paramount interest for the advanced driver assistance systems (ADAS). Recently proposed methods intend to use deeper and more complicated networks to achieve better performance, which leads to heavy models that are not feasible for the resource limited applications such as ADAS. To resolve this issue, we have worked on the following aspects: 1) a lightweight network model, which is referred to as recurrent multi-task thin net (RM-ThinNet), has been proposed which was especially designed for the computationally and memory limited devices; 2) a recurrent structure has been introduced to handle the scale difference and dependency between different tasks, and this recurrence ensures the different tasks are accomplished at different stages and their outputs can augment each other; and 3) a virtual data synthesization pipeline and a couple transfer learning method have been presented, by which network can be learnt effectively by relatively a small number of real data. Comparisons with the state-of-the-art methods reveal the superiority of the proposed method and competitive performance can be achieved with smaller model size and faster speed.

Great advances have been made for the deep networks, but relatively high memory and computation requirements limit their applications in the embedded device. In this paper, we introduce a class of efficient network architecture named ThinNet mainly for object detection applications on memory and computation limited platforms. The new architecture is based on two proposed modules: Front module and Tinier module. The Front module reduce the information loss from raw input images by utilizing more convolution layers with small size filters. The Tinier module use pointwise convolution layers before conventional convolution layer to decrease model size and computation, while ensuring the detection accuracy. Experimental evaluations on ImageNet classification and PASCAL VOC object detection datasets demonstrate the superior performance of ThinNet over other popular models. Our pretrained classification model(ThinNet_C) attains the same top-l and top-5 performance as the classic AlexNet but only with 1/50th the parameters. The detection model also obtains significant improvements over other detection methods, while requiring smaller model size to achieve high performance.

Recently, object classification of remote sensing images has attracted more and more research interests due to the development of satellite and aerial vehicle technologies. Hashing learning is an efficient method to handle the huge amount of the remote sensing data. In this paper, we proposed a novel hashing learning method named partial randomness supervised discrete hashing (PRSDH), which combines data-dependent methods and data-independent methods. It jointly learns a discrete binary codes generation and partial random constraint optimization model. By random projection, the computation complexity is reduced effectively. With the weight matrix derived from the training data, the semantic similarity between the data can be well preserved while generating the hashing codes. For the discrete constraint problem, this paper adopts the discrete cyclic coordinate descent (DCC) algorithm to optimize the codes bit by bit. The experimental results show that PRSDH outperforms other comparative methods and demonstrated that PRSDH has good adaptability to the characteristic of remote sensing object.

Studies on human motion have attracted a lot of attentions. Human motion capture data, which much more precisely records human motion than videos do, has been widely used in many areas. Motion segmentation is an indispensable step for many related applications, but current segmentation methods for motion capture data do not effectively model some important characteristics of motion capture data, such as Riemannian manifold structure and containing non-Gaussian noise. In this paper, we convert the segmentation of motion capture data into a temporal subspace clustering problem. Under the framework of sparse subspace clustering, we propose to use the geodesic exponential kernel to model the Riemannian manifold structure, use correntropy to measure the reconstruction error, use the triangle constraint to guarantee temporal continuity in each cluster and use multi-view reconstruction to extract the relations between different joints. Therefore, exploiting some special characteristics of motion capture data, we propose a new segmentation method, which is robust to non-Gaussian noise, since correntropy is a localized similarity measure. We also develop an efficient optimization algorithm based on block coordinate descent method to solve the proposed model. Our optimization algorithm has a linear complexity while sparse subspace clustering is originally a quadratic problem. Extensive experiment results both on simulated noisy data set and real noisy data set demonstrate the advantage of the proposed method.Studies on human motion have attracted a lot of attentions. Human motion capture data, which much more precisely records human motion than videos do, has been widely used in many areas. Motion segmentation is an indispensable step for many related applications, but current segmentation methods for motion capture data do not effectively model some important characteristics of motion capture data, such as Riemannian manifold structure and containing non-Gaussian noise. In this paper, we convert the segmentation of motion capture data into a temporal subspace clustering problem. Under the framework of sparse subspace clustering, we propose to use the geodesic exponential kernel to model the Riemannian manifold structure, use correntropy to measure the reconstruction error, use the triangle constraint to guarantee temporal continuity in each cluster and use multi-view reconstruction to extract the relations between different joints. Therefore, exploiting some special characteristics of motion capture data, we propose a new segmentation method, which is robust to non-Gaussian noise, since correntropy is a localized similarity measure. We also develop an efficient optimization algorithm based on block coordinate descent method to solve the proposed model. Our optimization algorithm has a linear complexity while sparse subspace clustering is originally a quadratic problem. Extensive experiment results both on simulated noisy data set and real noisy data set demonstrate the advantage of the proposed method.

In recent years, sparse representation-based classification (SRC) is one of the most successful methods and has been shown impressive performance in various classification tasks. However, when the training data have a different distribution than the testing data, the learned sparse representation may not be optimal, and the performance of SRC will be degraded significantly. To address this problem, in this paper, we propose an optimal couple projections for domain-adaptive SRC (OCPD-SRC) method, in which the discriminative features of data in the two domains are simultaneously learned with the dictionary that can succinctly represent the training and testing data in the projected space. OCPD-SRC is designed based on the decision rule of SRC, with the objective to learn coupled projection matrices and a common discriminative dictionary such that the between-class sparse reconstruction residuals of data from both domains are maximized, and the within-class sparse reconstruction residuals of data are minimized in the projected low-dimensional space. Thus, the resulting representations can well fit SRC and simultaneously have a better discriminant ability. In addition, our method can be easily extended to multiple domains and can be kernelized to deal with the nonlinear structure of data. The optimal solution for the proposed method can be efficiently obtained following the alternative optimization method. Extensive experimental results on a series of benchmark databases show that our method is better or comparable to many state-of-the-art methods.

This paper presents a content-adaptively sparse reconstruction method for abnormal events detection by exploiting the low-rank property of video sequences. In dictionary learning phase, the bases which describe more important characteristics of the normal behavior patterns are assigned with lower reconstruction costs. Based on the low-rank property of the bases captured by the low-rank approximation, a weighted sparse reconstruction method is proposed to measure the abnormality of testing samples. Multiscale 3-D gradient features, which encode the spatiotemporal information, are adopted as the low level descriptors. The benefits of the proposed method are threefold: first, the low-rank property is utilized to learn the underlying normal dictionaries, which can represent groups of similar normal features effectively; second, the sparsity-based algorithm can adaptively determine the number of dictionary bases, which makes it a preferable choice for representing the dynamic scene semantics; and third, based on the weighted sparse reconstruction method, the proposed method is more efficient for detecting the abnormal events. Experimental results on the public datasets have shown that the proposed method yields competitive performance comparing with the state-of-the-art methods.

This paper presents a new hyperspectral image (HSI) classification method which is capable of automatic feature learning while achieving high classification accuracy. The method contains two major modules: the spectral classification module and the spatial constraint module. Spectral classification module uses a deep network named stacked denoising autoencoders (SdA) to learn feature representation of the data. Through SdA, the data are projected nonlinearly from its original hyperspectral space to some higher dimensional space where more compact distribution is obtained. An interesting aspect of this method is that it does not need a feature design/extraction process guided by human prior. The suitable feature for the classification is learned by the deep network itself. Superpixel is utilized to generate the spatial constraints to refine the spectral classification results. By exploiting the spatial consistency of neighborhood pixels, the accuracy of classification is further improved by a big margin. Experiments on the public datasets reveal the superior performance of the proposed method.

In this paper, we propose an improved road detection method from high-resolution remote sensing images by means of a modified path opening and closing algorithm. To detect high curvature roads, we improved previously used adjacency graphs by enlarging the path angle of modified adjacency graphs. Furthermore, to remove noise efficiently, we combined the length of paths with road geometrical features to set up road extraction decision rules. A chief advantage of our approach is to be able to extract roads consisting of an unknown number of segments without manual parameter adjustment. We evaluated our proposed method systematically with a variety of images from IKONOS, EROS and aerial images. Experimental results demonstrated the accuracy and efficiency of our proposed method.

Different from the photometric images, depth images resolve the distance ambiguity of the scene, while the properties, such as weak texture, high noise, and low resolution, may limit the representation ability of the well-developed descriptors, which are elaborately designed for the photometric images. In this paper, a novel depth descriptor, geodesic invariant feature (GIF), is presented for representing the parts of the articulate objects in depth images. GIF is a multilevel feature representation framework, which is proposed based on the nature of depth images. Low-level, geodesic gradient is introduced to obtain the invariance to the articulate motion, such as scale and rotation variation. Midlevel, superpixel clustering is applied to reduce depth image redundancy, resulting in faster processing speed and better robustness to noise. High-level, deep network is used to exploit the nonlinearity of the data, which further improves the classification accuracy. The proposed descriptor is capable of encoding the local structures in the depth data effectively and efficiently. Comparisons with the state-of-the-art methods reveal the superiority of the proposed method.

This paper presents a new descriptor for human detection in still images. It is referred to as isotropic granularity-tunable gradients partition (IGGP), which is extended from granularity-tunable gradients partition (GGP) descriptors. The isotropic representation is achieved by aligning the features with different orientation channels according to their principal angles. The benefits of this extension are two folds: firstly, since the partitions’ sizes of all the orientation channels are equal, the noise introduce by the small partitions in the original GGP descriptors is eliminated and the performance can be essentially improved; secondly, the integral image based fast computation is applied and more than 20 times speedup has been achieved. In addition, we introduce a new human dataset HIMA. Unlike the previous available human datasets which are mainly captured on the street views for automobile safety or robotics, HIMA dataset is captured on the outdoor work fields for industry safety. The major challenges include: extreme light conditions, occlusion and strong noise. We benchmark several promising detection systems, providing an overview of state-of-the-art performance on the HIMA set. Experimental results show that the proposed method can yield very competitive results in both the detection speed and accuracy.

In this paper, a new scheme is presented to improve the coding efficiency of sequences captured by stationary cameras (or namely, static cameras) for video surveillance applications. We introduce two novel kinds of frames (namely background frame and difference frame) for input frames to represent the foreground/background without object detection, tracking or segmentation. The background frame is built using a background modeling procedure and periodically updated while encoding. The difference frame is calculated using the input frame and the background frame. A sequence structure is proposed to generate high quality background frames and efficiently code difference frames without delay, and then surveillance videos can be easily compressed by encoding the background frames and difference frames in a traditional manner. In practice, the H.264/AVC encoder JM 16.0 is employed as a build-in coding module to encode those frames. Experimental results on eight in-door and out-door surveillance videos show that the proposed scheme achieves 0.12 dB~1.53 dB gain in PSNR over the JM 16.0 anchor specially configured for surveillance videos.

This paper proposes a novel descriptor, granularity-tunable gradients partition (GGP), for human detection. The concept granularity is used to define the spatial and angular uncertainty of the line segments in the Hough space. Then this uncertainty is backprojected into the image space by orientation-space partitioning to achieve efficient implementation. By changing the granularity parameter, the level of uncertainty can be controlled quantitatively. Therefore a family of descriptors with versatile representation property can be generated. Specifically, the finely granular GGP descriptors can represent the specific geometry information of the object (the same as Edgelet); while the coarsely granular GGP descriptors can provide the statistical representation of the object (the same as histograms of oriented gradients, HOG). Moreover, the position, orientation, strength and distribution of the gradients are embedded into a unified descriptor to further improve the GGP's representation power. A cascade structured classifier is built by boosting the linear regression functions. Experimental results on INRIA dataset show that the proposed method achieves comparable results to those of the state-of-the-art methods.

One basic observation for pedestrian detection in video sequences is that both appearance and motion information are important to model the moving people. Based on this observation, we propose a new kind of features, 3D Haar-like (3DHaar) features. Motivated by the success of Haar-like features in image based face detection and differential-frame based pedestrian detection, we naturally extend this feature by defining seven types of volume filters in 3D space, instead of using rectangle filter in 2D space. The advantage is that it can not only represent pedestrian's appearance, but also capture the motion information. To validate the effectiveness of the proposed method, we combine the 3DHaar with support vector machine (SVM) for pedestrian detection. Our experiments demonstrate the 3DHaar are more effective for video based pedestrian detection.