Although text information has aided existing models to achieve promising results in open vocabulary object detection (OVD), the lack of semantic information has led to the difficulty in small objects detection (SOD). Moreover, such semantic gap also causes failure when matching texts and image features, resulting in false negative instances being detected. To address these issues, we propose a Cascade Open Vocabulary Detector (Cas-OVD), which builds upon existing multi-stage detection pipelines but specializes in text-vision alignment for small objects. In particular, we adapt a multi-refined region proposal network, guided by a non-sampled anchor strategy, to reduce the missing and false detections of small objects. Meanwhile, a deformable convolution network based feature conversion module is proposed to enhance the semantic information of small objects even the potential ones with low confidence. Unlike existing methods that rely on coarse-grained image-based features for image-text matching, Cas-OVD refines these features through a cascade alignment process, allowing each stage to build on the results of the previous one. This can progressively enhance the feature correlation between the image regions and the textual descriptions through successive error correction. On the joint BDD100K-SODA-D dataset, Cas-OVD achieved 17.95% AP$_{\mathrm{all}}$and 14.6% AP$_{\mathrm{s}}$, outperforming RegionCLIP by 3.5% AP$_{\mathrm{all}}$and 3.0% AP$_{\mathrm{s}}$, respectively. On the OV_COCO dataset, Cas-OVD has the 32.71% AP$_{\mathrm{all}}$and 17.26% AP$_{\mathrm{s}}$, surpassing the RegionCLIP by 6.6% AP$_{\mathrm{all}}$and 6.1% AP$_{\mathrm{s}}$, respectively.

ABSTRACT Multiple sclerosis (MS) lesion segmentation from MR imaging is a prerequisite step in clinical diagnosis and treatment of brain diseases. However, automated segmentation of MS lesions remains a challenging task, owing to the variant morphology and uncertain distribution of lesions across subjects. Despite the achieved success by existing methods, two problems still persist in automated segmentation of MS lesions, namely the lack of an effective feature enhancement approach for capturing locality context and the lack of global coherence in prediction for pixels. Hence, we propose a correlation learning network for both local and global context in this work. Specifically, we propose a sparse spatial correlation module to learn the spatial correlations within neighbours for local context. Besides, we propose a global coherence module to encode long‐range dependencies for global context. The proposed method is evaluated on a public ISBI2015 datatset and a private in‐house dataset collected from hospital. Experimental results show the competitive performance of our method against state‐of‐the‐art methods.

Neural network–based models have shown tremendous achievements in various fields. However, standard AI‐based systems suffer from catastrophic forgetting when undertaking sequential learning of multiple tasks in dynamic environments. Continual learning has emerged as a promising approach to address catastrophic forgetting. It enables AI systems to learn, transfer, augment, fine‐tune, and reuse knowledge for future tasks. The techniques used to achieve continual learning are inspired by the learning processes of the human brain. In this study, we present a comprehensive review of research and recent developments in continual learning, highlighting key contributions and challenges. We discuss essential functions of the biological brain that are pivotal for achieving continual learning and map these functions to the recent machine‐learning methods to aid understanding. Additionally, we offer a critical review of five recent types of continual learning methods inspired by the biological brain. We also provide empirical results, analysis, challenges, and future directions. We hope that this study will benefit both general readers and the research community by offering a complete picture of the latest developments in this field.

Oriented object detection in remote sensing is a critical task for accurately location and measurement of the interested targets. Despite of its success in object detection, deep learning-based detectors rely heavily on extensive data annotation. However, variations in object appearance significantly increase the difficulty and the cost of creating large-scale annotated datasets. Semi-supervised learning (SSL) aims to utilize unlabeled data to enhance object detectors. Among these, pseudo-label-based methods have shown promising results recently. Nonetheless, as training progresses, the accumulation of errors in pseudo labels leads to prediction bias without corrections. To tackle this particular challenge, we present a SSL pipeline, named “dual teacher,” for improving the reliability of pseudo labels in the semi-supervised oriented object detection. First, to mitigate the bias caused by limited annotated data, a global burn-in (GBI) strategy is introduced at the beginning of training, which guides the student detector to learn the feature extraction on a global scale. In addition, an online bounding box (bbox) correction module is proposed to decrease the occurrence of mislabeled instances and enhance the reliability of detection. These improvements are facilitated by an additional detector, instead of a single teacher model in the teacher-student architecture. Dual teacher reduces the dependency on the quality of pseudo labels related to the model complexity and combines the strengths of both the two-stage and one-stage detectors. With only 20% labeled data, dual teacher outperforms fully supervised rotated fully convolutional one-stage object detection (R-FCOS), you only look once X-small (YOLOX-s), and rotated region-based convolutional neural network (R-RCNN) by up to 2% on both a large-scale dataset for object detection in aerial images (DOTA) and SODA-A datasets. This reveals its potential in reducing labor-intensive tasks and enhancing robustness against environmental interference and noisy labels. The code is available at:https://github.com/ZYFFF-CV/DualTeacher-semisup.git.

As a global marine disaster, red tides pose serious threats to marine ecology and the blue economy, making their monitoring crucial for preventing harmful algal blooms (HABs) and protecting the marine environment. In this study, satellite remote sensing was utilized to provide timely, large-scale, and continuous observation capabilities, overcoming the high cost and spatial and temporal limitations of in situ monitoring. However, existing remote sensing-based methods often exhibit coarse segmentation granularity and suffer from high computational complexity. To overcome these challenges, we propose a novel bimodal multispectral dynamic offset binary quantization visual transformer (DoBi-SWiP-ViT) that utilizes the ViT for global feature aggregation and parameter quantization for efficient segmentation. With the bimodal Swin-ViT with unified perceptual parsing (UPP) architecture, our model integrates data from multiple spectral bands to achieve fine-grained segmentation of large-scale remote sensing images. Additionally, we introduce a dynamic magnitude offset binary quantization ViT block to reduce the parameter redundancy and improve the computational efficiency. In addition, we validated the performance of our model through extensive comparative experiments on high-resolution imagery datasets of sea surface red tides collected from different satellite platforms. The results show that our proposed DoBi-SWiP-ViT has significantly improved the mean accuracy (mAcc) of the segmentation results. For the two test areas acquired from different satellite platforms, the improvements are 8.78% and 10.18%, respectively. This has demonstrated the superior performance of our model in detecting the red tides from high-resolution visible images, highlighting its effectiveness in capturing complex patterns and subtle features in multispectral imagery.

With the popularization of intelligent agricultural facilities, the demand for electricity in modern agricultural systems has also increased. To meet the continuous demand for electricity in agricultural production, including crop growth, storage, and processing, fine-grained electricity load forecasting becomes crucial, which can provide crucial decision support for the power supply, allocation, and management of agricultural facilities. However, the electricity load data in agricultural facilities is a non-stationary time series, which presents significant challenges for achieving accurate and effective forecasting. Thus, we focus on investigating the electricity load data in agricultural facilities and incorporate covariates, such as temperature, humidity, wind speed, and rainfall, into our analysis. Specifically, we propose a deep learning model based on empirical mode decomposition called EMD-BiLSTM-DLSTM. This model initially decomposes the electricity load time series into a sequence of relatively stationary components using empirical mode decomposition. It then employs a bidirectional long short-term memory network to predict each component, obtaining preliminary prediction results. Finally, a deep long short-term memory network is applied to refine the prediction results by incorporating covariates, resulting in more accurate prediction results. Experimental results show that compared with other time series forecasting methods, the proposed model has significant advantages in prediction accuracy and correlation.

Underwater sound classification has been an area of interest in the research community because of its applications in military, commercial, and environmental domains. Underwater sound classification is a challenging task because of the high background noise and complex sound propagation patterns in the sea environment. For underwater sound classification, deterministic as well as stochastic techniques are being used. However, in recent years, stochastic techniques which are learning-based are getting a lot of attention. There exist few survey studies with a limited scope that cover the limited number of studies. In this study, we present the most comprehensive review of research and the latest developments in the field of underwater sound classification by highlighting the contributions and challenges from over 250 recent research papers. We discuss machine learning as well as deep learning-based methods for marine vessel sound classification and fish sound classification. The study also includes details of sources of underwater sound, features, classifiers, datasets, related techniques, challenges, and future trends. We hope that the study will benefit the general reader as well as the research community to have a complete picture of the latest research in the field.

Underwater Image Enhancement (UIE) encounters substantial challenges due to the intricate nature of physical degradation processes that diminish visibility in underwater images. Existing methods leverage learning-based models to delineate pixel mappings for either paired or unpaired images to ameliorate the quality of degraded visuals. Nonetheless, the paucity of paired image datasets and the erratic nature of unsupervised learning methods considerably hinder advancements in UIE. This study presents an innovative contrastive learning framework specifically designed for UIE, aimed at effectively addressing the aforementioned challenges. Our strategy reconceptualizes image enhancement as a multi-task joint learning problem, thus fortifying the enhancement process. We pinpoint three pivotal aspects for UIE: contrastive feature learning, semantic information coherence, and cross-domain feature transfer. These elements are imperative for augmenting contrast, preserving texture integrity, and ensuring color accuracy. The contrastive learning paradigm empowers the enhancement module to discern between unpaired positive (high-quality) and negative (degraded) underwater images, facilitating semantic learning in refining the enhancement network systematically. By leveraging the semantic feature domain extracted from unpaired high-quality images, our method demonstrates superior performance, validated by several quality metrics, outperforming recent advancements in unsupervised UIE techniques.

Detection of image splicing forgery become an increasingly difficult task due to the scale variations of the forged areas and the covered traces of manipulation from post-processing techniques. Most existing methods fail to jointly multi-scale local and global information and ignore the correlations between the tampered and real regions in inter-image, which affects the detection performance of multi-scale tampered regions. To tackle these challenges, in this paper, we propose a novel method based on feature aggregation and region-aware learning to detect the manipulated areas with varying scales. In specific, we first integrate multi-level adjacency features using a feature selection mechanism to improve feature representation. Second, a cross-domain correlation aggregation module is devised to perform correlation enhancement of local features from CNN and global representations from Transformer, allowing for a complementary fusion of dual-domain information. Third, a region-aware learning mechanism is designed to improve feature discrimination by comparing the similarities and differences of the features between different regions. Extensive evaluations on benchmark datasets indicate the effectiveness in detecting multi-scale spliced tampered regions.

Through truncating the weights and activations of a deep neural network, conventional binary quantization imposes limitations on the representation capability of the network parameters, which hence deteriorates the detection performance of the network. In this paper, a joint-guided distillation binary neural network via dynamic channel-wise diversity enhancement for object detection (JDBNet) is proposed to mitigate the gap of quantization errors. Our JDBNet includes a dynamic channel-wise diversity scheme and real-valued joint-guided teacher assistance to enhance the representation capability of the binary neural network in the object detection tasks. In the dynamic diversity scheme, the learning channel-wise bias (LCB) layer supports adjusting the magnitude of the parameters in which the sensitivity of the model parameters to the arbitrary quantization method is reduced, thereby improving the diversity expression ability of the feature parameters. In the joint-guided strategy, the single-precision implicit knowledge from the guiding teacher in the multilevel layer is utilized to supervise and penalize the quantitative model, enhancing the fitting performance of parameters in the binary quantized model. Extensive experiments on the PASCAL VOC, MS COCO, and VisDrone-DET datasets demonstrate that our JDBNet outperforms the state-of-the-art binary object detection networks in terms of mean Average Precision.

In the remote sensing field, detecting small objects is a pivotal task, yet achieving high performance in deep learning-based detectors heavily relies on extensive data annotation. The challenge intensifies as small objects in remote sensing imagery are typically densely distributed and numerous, leading to a substantial increase in the cost of creating large-scale annotated datasets. This elevated cost poses significant limitations on the application and advancement of small object detection. To address this issue, a point-based annotation (PBA) method is proposed, which generates bounding boxes (BBOXs) through graph-based segmentation. In this framework, user annotations categorize nodes into three distinct classes—positive, negative, and to-cut—facilitating a more intuitive and efficient annotation process. Utilizing the max-flow algorithm, our method seamlessly generates oriented BBOXs (OBBOXs) from these classified nodes. The efficacy of PBA is underscored by our empirical findings. Notably, annotation efficiency is enhanced by at least 40%, a significant leap forward. Moreover, the intersection over union (IoU) metric of our OBBOX outperforms existing methods like “segment anything model (SAM)” by 10%. Finally, when applied in training, models annotated with PBA exhibit a 3% increase in the mean average precision (mAP) compared with those using traditional annotation methods. These results not only affirm the technical superiority of PBA but also its practical impact on advancing small object detection in remote sensing.

We study the data offloading problem in space-air-ground integrated networks (SAGINs) by jointly optimizing task scheduling and power control to balance the total energy consumption and mean makespan. We consider a mixed integer nonlinear programming problem to minimize a normalized weighted combination of these two conflicting objectives. We first propose an approximation algorithm to find a high-quality solution, which is shown to be at most$\frac{1}{2}$from the optimum to this problem for given power allocation. We further show that optimal power allocation can be obtained in closed form under the assumption that satellite-ground links have low signal-to-noise ratio (SNR). Thus, the proposed approximation algorithm can be directly utilized to obtain a constant-factor solution to the studied problem in low-SNR scenarios. To extend our solution to more general scenarios, we further propose an efficient hybird algorithm based on a genetic framework. Our simulation results demonstrate the near-optimality and correctness of the proposed algorithms, and they unveil the interplay between total energy consumption and mean makespan in SAGINs as well.

This paper proposes a simple yet effective method to re-move shadows from text document images. It mainly includes several parts. Firstly, we propose a text elimination-based background extraction strategy to estimate shadow map. It indicates the shadow regions accurately and helps to predict global background. Secondly, a binarization-based text ex-traction algorithm is designed to obtain texts from document image. By fusing texts and global background, a preparatory shadow-free image can be obtained. Thirdly, we propose an adaptive text contrast enhancement strategy to generate shadow-free results with comfortable visual perception across shadow and non-shadow regions. Quantitative and visual results performed on open datasets indicate that the proposed method can generate clear shadow-free images from text document images. Our code will be publicly available soon.

Abstract In this work, a neural surface reconstruction framework is presented. In order to perform neural surface reconstruction using 2D supervision, a weighted random sampling based on saliency is introduced for training the deep neural network. In the proposed method, self‐attention is used to detect the saliency of input 2D images. The saliency map, that is, the weight matrix of the weighted random sampling, is used to sample the training samples. As a result, more samples in the reconstructed object area are collected. Moreover, an update strategy for weight based on sampling frequency is adopted to avoid the points that cannot be sampled all the time. The experiments are implemented in real‐world 2D images of objects with different material properties and lighting conditions based on the DTU dataset. The results show that the proposed method produces more detailed 3D surfaces, and the rendered results are closer to the raw images visually. In addition, the mean of peak signal‐to‐noise ratio (PNSR) is also improved.

Abstract Virtual globes render planetary‐scale terrain and have limited support for 3D applications development. Game engines provide development environment for interactive 3D applications development and have limited support for world‐scale terrain rendering. The game engine based terrain rendering methods lacks hardware based tessellation for high‐performance. This work presents a novel method for a high‐performance large‐scale terrain rendering for high‐fidelity display systems using game engine. The proposed method performs patch‐based hierarchical culling of a multi‐resolution terrain model to reduce rendering load. A view‐based algorithm simplifies the patches with error control on GPU. Simplified patches are efficiently submitted for drawing using indirect mesh instancing feature of game engine. The proposed method utilizes hardware tessellation feature for high‐performance model tessellation and accurate earth's surface construction using displacement mapping. The proposed method is evaluated by rendering scenes for high‐quality output on consumer‐level hardware. Flights are performed with various settings and results are compared with clipmap‐based and state‐of‐the‐art hardware tessellation based adaptive methods. The proposed method achieved 750, 575, and 540 frames‐per‐second for HD, full‐HD, and ultra‐HD display resolutions.

The exponential growth in data communication and increase in network size have led to various intrusions and attacks. An Intrusion Detection System (IDS) can be provided as a crucial component of a network or database to ensure the security of data communication over a network. The network size is large, a large dataset may comprise more irrelevant, redundant, and high-dimensional features that impact feature classification, thus affecting the intrusion detection rate. This study presents a new hybrid enhanced normalised Crow Search Algorithm (CSA) and Particle Swarm Optimisation (PSO) technique to address feature selection issues and to classify global best features using a random-forest classifier. In the proposed algorithm, the benefits of the CSA between the search strategy and rapid convergence phenomenon of the PSO algorithm are utilised to select the global best solution in a large search space. A random-forest classifier is used to classify the features after they are updated with weight values for significant features, assessing the asymptotic variance of features and points that are closest to the optimal solution. The asymptotic features are subjected to the weighted least mean square (WLS) method to eliminate large deviations among the features. The random-forest classifier distinguishes between normal records and abnormal intrusion records. The performance assessment of the proposed hybrid IDS model is performed by utilising two datasets, which reveals that the proposed model outperforms other existing models. The simulation outcomes show higher accuracy rate, precision value, recall factor, and F1-Score, revealing the efficacy of the IDS model.

With smart portable devices such as smartphones and tablets in usage and popularity, people are more willing to use these devices to scan and save digitized documents. However, when capturing document images, shadows are inevitable and influence clarity and readability. How to remove the shadows of document images is an important and meaningful task. In this paper, we propose a water-filling method using chroma adjustment for shadow removal. Firstly, a global and local jointly water-filling approach is designed to estimate the shading map. Then, we design an adaptive global brightness adjustment strategy to optimize the global luminance of the output image. Since only adjusting brightness can cause color distortion of output images, we propose an adaptive chroma adjustment strategy to ensure color consistency across all areas of output images. A series of experiments show that our method can remove shadows of digitized documents, outperforming some state-of the-art methods. Moreover, the proposed method can keep the brightness and color as consistent as possible with the non-shadow area.

The segmentation of multiple sclerosis (MS) lesions from MR imaging sequences remains a challenging task, due to the characteristics of variant shapes, scattered distributions and unknown numbers of lesions. However, the current automated MS segmentation methods with deep learning models face the challenges of (1) capturing the scattered lesions in multiple regions and (2) delineating the global contour of variant lesions. To address these challenges, in this paper, we propose a novel attention and graph-driven network (DAG-Net), which incorporates (1) the spatial correlations for embracing the lesions in distant regions and (2) the global context for better representing lesions of variant features in a unified architecture. Firstly, the novel local attention coherence mechanism is designed to construct dynamic and expansible graphs for the spatial correlations between pixels and their proximities. Secondly, the proposed spatial-channel attention module enhances features to optimize the global contour delineation, by aggregating relevant features. Moreover, with the dynamic graphs, the learning process of the DAG-Net is interpretable, which in turns support the reliability of segmentation results. Extensive experiments were conducted on a public ISBI2015 dataset and an in-house dataset in comparison to state-of-the-art methods, based on geometrical and clinical metrics. The experimental results validate the effectiveness of proposed DAG-Net on segmenting variant and scatted lesions in multiple regions.

The existing saliency detection models based on RGB colors only leverage appearance cues to detect salient objects. Depth information also plays a very important role in visual saliency detection and can supply complementary cues for saliency detection. Although many RGB-D saliency models have been proposed, they require to acquire depth data, which is expensive and not easy to get. In this paper, we propose to estimate depth information from monocular RGB images and leverage the intermediate depth features to enhance the saliency detection performance in a deep neural network framework. Specifically, we first use an encoder network to extract common features from each RGB image and then build two decoder networks for depth estimation and saliency detection, respectively. The depth decoder features can be fused with the RGB saliency features to enhance their capability. Furthermore, we also propose a novel dense multiscale fusion model to densely fuse multiscale depth and RGB features based on the dense ASPP model. A new global context branch is also added to boost the multiscale features. Experimental results demonstrate that the added depth cues and the proposed fusion model can both improve the saliency detection performance. Finally, our model not only outperforms state-of-the-art RGB saliency models, but also achieves comparable results compared with state-of-the-art RGB-D saliency models.

Abstract Low‐light image enhancement is rapidly gaining research attention due to the increasing demands of extreme visual tasks in various applications. Although numerous methods exist to enhance image qualities in low light, it is still undetermined how to trade‐off between the human observation and computer vision processing. In this work, an effective generative adversarial network structure is proposed comprising both the densely residual block (DRB) and the enhancing block (EB) for low‐light image enhancement. Specifically, the proposed end‐to‐end image enhancement method, consisting of a generator and a discriminator, is trained using the hyper loss function. The DRB adopts the residual and dense skip connections to connect and enhance the features extracted from different depths in the network while the EB receives unique multi‐scale features to ensure feature diversity. Additionally, increasing the feature sizes allows the discriminator to further distinguish between fake and real images from the patch levels. The merits of the loss function are also studied to recover both contextual and local details. Extensive experimental results show that our method is capable of dealing with extremely low‐light scenes and the realistic feature generator outperforms several state‐of‐the‐art methods in a number of qualitative and quantitative evaluation tests.

Abstract Human detection in crowded scenes is one of the research components of crowd safety problem analysis, such as emergency warning and security monitoring platforms. Although the existing anchor‐free methods have fast inference speed, they are not suitable for object detection in crowded scenes due to the model's inability to predict the well‐fined object detection bounding boxes. This work proposes an end‐to‐end anchor‐free network, Multi‐dimensional Weighted Cross‐Attention Network (MANet), which can perform real‐time human detection in crowded scenes. Specifically, the Double‐flow Weighted Feature Cascade Module (DW‐FCM) is used in the extractor to highlight the contribution of features at different levels. The Triplet Cross Attention Module (TCAM) is used in the detector head to enhance the association dependence of multi‐dimension features, further strengthening human boundary features' discrimination ability at a fine‐grained level. Moreover, the strategy of Adaptively Opposite Thrust Mapping (AOTM) ground‐truth annotation is proposed to achieve bias correction of erroneous mappings and reduce the iterations of useless learning of the network. These strategies effectively alleviate the defect that the existing anchor‐free network cannot correctly distinguish and locate the individual human in crowded scenes. Compared with the anchor‐based detection method, there is no need to set anchor parameters manually, and the detection speed can satisfy the real‐time application. Finally, through extensive comparative experiments on CrowdHuman and WIDER FACE datasets, the results demonstrate that the improved strategy achieves the state‐of‐the‐art result in the anchor‐free methods.

In this paper, we propose a novel approach to reconstruct 3D human body shapes based on a sparse set of RGBD frames using a single RGBD camera. We specifically focus on the realistic settings where human subjects move freely during the capture. The main challenge is how to robustly fuse these sparse frames into a canonical 3D model, under pose changes and surface occlusions. This is addressed by our new framework consisting of the following steps. First, based on a generative human template, for every two frames having sufficient overlap, an initial pairwise alignment is performed; It is followed by a global non-rigid registration procedure, in which partial results from RGBD frames are collected into a unified 3D shape, under the guidance of correspondences from the pairwise alignment; Finally, the texture map of the reconstructed human model is optimized to deliver a clear and spatially consistent texture. Empirical evaluations on synthetic and real datasets demonstrate both quantitatively and qualitatively the superior performance of our framework in reconstructing complete 3D human models with high fidelity. It is worth noting that our framework is flexible, with potential applications going beyond shape reconstruction. As an example, we showcase its use in reshaping and reposing to a new avatar.

Accurate segmentation of multiple sclerosis from MR images, faces the challenges imposed by the high variability in lesion appearance, and distant and disjoint lesion regions. Previous methods using multi-scale feature fusion or cascade networks, mostly rely on local feature representation learned from limited receptive field, which fail to leverage global context and model relations between multiple regions. To address these issues, we propose a hybrid feature network (HF-Net) driven by attention and graph convolution features, to improve the MS lesion segmentation from MR images. The attention features help to enhance discriminative feature representation. Specifically, the pyramid augmented attention module encodes spatial features into local features, while the channel augmented attention module models channel-wise interdependencies between features. Meanwhile, the graph feature module exploits the global relations between features over local receptive field. The proposed HF-Net was evaluated on the datasets from the MSSEG Challenge and the ongoing ISBI Challenge, which outperforms several state-of-the-art methods.

It is difficult to detect ports in polarimetric SAR images due to the complex components, the variety of contours and the complicated coastal terrains background. A novel port detection method is proposed by extracting the specific water area in port based on three-component decomposition. Firstly, the double-bounce and volume scattering components of terrains are obtained by a modified three-component decomposition. The scattering components of port water are compared with that of normal water. Then water and land are separated by thresholding the volume scattering power. Port water and normal water regions are separated by thresholding a proposed polarimetric parameter related to the double-bounce scattering power. Finally, ports are detected by placing the boundary and thresholding the average double-bounce power of the land in the detected port water region. Three RADARSAT-2 polarimetric SAR data covering the regions of Dalian, Zhanjiang and Fujian in China respectively are used to test the port detection method. Experimental results show that all ports are correctly and fast detected by the proposed method.

Scene flow estimation with RGB‐D frames is receiving increasing attention in digital video processing and computer vision due to the widespread use of depth sensors. Existing methods based on object segmentation have shown their effectiveness for object occlusion and large displacement. However, improper segmentation often causes incomplete segmented areas or incorrect edges, which will result in inaccurate occlusion inference and scene flow estimation. To this end, an automatic layered RGB‐D scene flow estimation method is proposed, which achieves more accurate layering of objects in depth image by exploring motion information. The authors exploit super‐pixel segmentation for initial layering, which is beneficial for preserving edges and integrity of the objects. Furthermore, an optical flow which is highly correlated with the scene is also used for automatic layering. Finally, the utilisation of super‐pixel segmentation and motion information would ensure the integrity of the object area and improve the accuracy of scene flow. They have validated their approach both qualitatively and quantitatively on several public datasets. Experimental results show that the proposed method is able to preserve the integrity of the object and achieve lower root mean square error and average angular error as compared with the current state‐of‐the‐art algorithms.

This article presents a novel approach for depth map enhancement from an RGB-D video sequence. The basic idea is to exploit the photometric information in the color sequence to resolve the inherent ambiguity of shape from shading problem. Instead of making any assumption about surface albedo or controlled object motion and lighting, we use the lighting variations introduced by casual object movement. We are effectively calculating photometric stereo from a moving object under natural illuminations. One of the key technical challenges is to establish correspondences over the entire image set. We, therefore, develop a lighting insensitive robust pixel matching technique that out-performs optical flow method in presence of lighting variations. An adaptive reference frame selection procedure is introduced to get more robust to imperfect lambertian reflections. In addition, we present an expectation-maximization framework to recover the surface normal and albedo simultaneously, without any regularization term. We have validated our method on both synthetic and real datasets to show its superior performance on both surface details recovery and intrinsic decomposition.

Tiny face detection aims to find faces with high degrees of variability in scale, resolution and occlusion in cluttered scenes. Due to the very little information available on tiny faces, it is not sufficient to detect them merely based on the information presented inside the tiny bounding boxes or their context. In this paper, we propose to exploit the semantic similarity among all predicted targets in each image to boost current face detectors. To this end, we present a novel framework to model semantic similarity as pairwise constraints within the metric learning scheme, and then refine our predictions with the semantic similarity by utilizing the graph cut techniques. Experiments conducted on three widely-used benchmark datasets have demonstrated the improvement over the-state-of-the-arts gained by applying this idea.

In this paper we present a novel approach for depth map enhancement from an RGB-D video sequence. The basic idea is to exploit the photometric information in the color sequence. Instead of making any assumption about surface albedo or controlled object motion and lighting, we use the lighting variations introduced by casual object movement. We are effectively calculating photometric stereo from a moving object under natural illuminations. The key technical challenge is to establish correspondences over the entire image set. We therefore develop a lighting insensitive robust pixel matching technique that out-performs optical flow method in presence of lighting variations. In addition we present an expectation-maximization framework to recover the surface normal and albedo simultaneously, without any regularization term. We have validated our method on both synthetic and real datasets to show its superior performance on both surface details recovery and intrinsic decomposition.

This letter proposes a rapid line-extraction (RLE) method for synthetic aperture radar (SAR) images. RLE first transforms an image in the space domain into an image in the frequency domain. Then, using the central-slice theorem, RLE skilfully maps the image in the frequency domain into a parameter space, which effectively accelerates the straight-line extraction process. Unlike the traditional Hough transform, RLE is performed directly on an edge-field image rather than on a binary edge map. Theoretical analysis proves the advantages of using the edge-field map. Notably, the computational complexity can be greatly reduced relative to the complexity of obtaining a binary edge map, and the method can efficiently avoid the negative influence of false edges in the binary edge map. More importantly, because speckle, clutter, and blurred edges in real-world images decrease the sharpness of peaks, edge-field images that include the strength and direction information of SAR images are adopted to reduce the diffusion of peaks and improve the detection accuracy. Experimental studies show that RLE works independently, is robust to noise, has low computational complexity, achieves high true-positive detection rates, and yields satisfactory detection precision.

High-speed video has been commonly adopted in consumer-grade cameras, augmenting these videos with a corresponding depth stream will enable new multimedia applications, such as 3D slow-motion video. In this paper, we present a hybrid camera system that combines a high-speed color camera with a depth sensor, e.g. Kinect depth sensor, to generate a depth stream that can produce both high-speed and high-resolution RGB+depth stream. Simply interpolating the low-speed depth frames is not satisfactory, where interpolation artifacts and lose in surface details are often visible. We have developed a novel framework that utilizes both shading constraints within each frame and optical flow constraints between neighboring frames. More specifically we present (a) an effective method to find the intrinsics images to allow more accurate normal estimation; and (b) an optimization-based framework to estimate the high-resolution/high-speed depth stream, taking into consideration temporal smoothness and shading/depth consistency. We evaluated our holistic framework with both synthetic and real sequences, it showed superior performance than previous state-of-the-art.

In this paper we present a method of using standard multi-view images for 3D surface reconstruction of non-Lambertian objects. We extend the original visual hull concept to incorporate 3D cues presented by internal occluding contours, i.e., occluding contours that are inside the object's silhouettes. We discovered that these internal contours, which are results of convex parts on an object's surface, can lead to a tighter fit than the original visual hull. We formulated a new visual hull refinement scheme -- Locally Convex Carving that can completely reconstruct concavity caused by two or more intersecting convex surfaces. In addition we develop a novel approach for contour tracking given labeled contours in sparse key frames. It is designed specifically for highly specular or transparent objects, for which assumptions made in traditional contour detection/tracking methods, such as highest gradient and stationary texture edges, are no longer valid. It is formulated as an energy minimization function where several novel terms are developed to increase robustness. Based on the two core algorithms, we have developed an interactive system for 3D modeling. We have validated our system, both quantitatively and qualitatively, with four datasets of different object materials. Results show that we are able to generate visually pleasing models for very challenging cases.

Depth images, in particular depth maps estimated from stereo vision, may have a substantial amount of outliers and result in inaccurate 3D modelling and reconstruction. To address this challenging issue, in this paper, a graph-cut based multiple depth maps integration approach is proposed to obtain smooth and watertight surfaces. First, confidence maps for the depth images are estimated to suppress noise, based on which reliable patches covering the object surface are determined. These patches are then exploited to estimate the path weight for 3D geodesic distance computation, where an adaptive regional term is introduced to deal with the "shorter-cuts" problem caused by the effect of the minimal surface bias. Finally, the adaptive regional term and the boundary term constructed using patches are combined in the graph-cut framework for more accurate and smoother 3D modelling. We demonstrate the superior performance of our algorithm on the well-known Middlebury multi-view database and additionally on real-world multiple depth images captured by Kinect. The experimental results have shown that our method is able to preserve the object protrusions and details while maintaining surface smoothness.

Feature extraction is of high importance for effective data classification in hyperspectral imaging (HSI). Considering the high correlation among band images, spectral-domain feature extraction is widely employed. For effective spatial information extraction, a 2-D extension to singular spectrum analysis (2D-SSA), which is a recent technique for generic data mining and temporal signal analysis, is proposed. With 2D-SSA applied to HSI, each band image is decomposed into varying trends, oscillations, and noise. Using the trend and the selected oscillations as features, the reconstructed signal, with noise highly suppressed, becomes more robust and effective for data classification. Three publicly available data sets for HSI remote sensing data classification are used in our experiments. Comprehensive results using a support vector machine classifier have quantitatively evaluated the efficacy of the proposed approach. Benchmarked with several state-of-the-art methods including 2-D empirical mode decomposition (2D-EMD), it is found that our proposed 2D-SSA approach generates the best results in most cases. Unlike 2D-EMD that requires sequential transforms to obtain detailed decomposition, 2D-SSA extracts all components simultaneously. As a result, the execution time in feature extraction can be also dramatically reduced. The superiority in terms of enhanced discrimination ability from 2D-SSA is further validated when a relatively weak classifier, i.e., the k-nearest neighbor, is used for data classification. In addition, the combination of 2D-SSA with 1-D principal component analysis (2D-SSA-PCA) has generated the best results among several other approaches, demonstrating the great potential in combining 2D-SSA with other approaches for effective spatial-spectral feature extraction and dimension reduction in HSI.

Vision-based registration techniques for augmented reality(AR) systems have been the subject of intensive research recently due to their potential to accurately align virtual objects with the real world. The downfall of these vision-based approaches, however, is their high computational cost and lack of robustness. To address these shortcomings, a robust pose estimation algorithm based on artificial planar markers is adopted. This algorithm solves the problem of camera pose ambiguities and is able to draw a unique and robust solution. Experiments show the robustness and effectiveness of this method in the context of real-time AR tracking.

In this article, the selective visual attention mechanism and curve detection by Connect The Dots model is introduced to small and dim target detection in deep space background. The greyscale and movement continual significance are fully taken into account to get focus of attention integration map. A curve detection method which based on Connect The Dots model is designed to detect the target trajectory. Qualitative and quantitative results prove that the proposed algorithm has strong anti-noise performance and improve calculate efficiency of detection system effectively.

Pedestrian detection plays an important role in automated surveillance system. However, it is challenging to detect pedestrian robustly and accurately in a cluttered environment. In this paper, we propose a new cooperative pedestrian detection method using both colour and thermal image sequences, which is compared with the method using only colour image sequence and that using multi-modal fusion. Experiment results show that our cooperative detection mechanism could get more accurate pedestrian areas, a lower false alarm rate and a higher detection precision. Therefore, it has broad application prospects in the field of industry and military.

This paper presents a new system for computer-aided orthognathic surgery which aims at correcting dento-facial defects and keeping patient away from much radiation and high cost of CT or MRI imaging. The system allows simulating the surgical correction on virtual 3-D model of the skull reconstructed with the acquisition of radiographs of three orthogonal views and facial mesh data generated by 3-D laser scanner and the use of build-in skull template mesh. Surgery simulation includes maxilla Le Fort I and mandible bilateral SSRO osteotomies by interactive cutting of jaw bone. With the help of a special registration procedure, we are able to align the reconstructed skull model and a facial skin model. Upon fulfilment of the registration and virtual osteotomies, the system provides further assistance with predicting postoperative appearance of patient. Finally, we apply the system on one patient in clinical practice and achieve satisfied result.

Trajectory matching is an important way to describe and classify behaviors of moving objects in a computer visual system. In this paper, we present two trajectory description methods, time-sampling sequence and space-sampling sequence, which can be used in different matching applications. We then propose two general trajectory matching schemes based on Levenshtein distance and relaxation matching respectively. Trajectory Levenshtein distance scheme is a good way to compare the topological shapes and directions of trajectories, and can be performed quickly. Trajectory relaxation matching scheme can gain the statistical optimal matching. Finally, we propose a top-to-bottom hierarchical clustering algorithm to classify trajectories, and several experiments demonstrate that our schemes are efficient in matching and classifying different shape and direction trajectories