Speech-Preserving Facial Expression Manipulation (SPFEM) is an innovative technique aimed at altering facial expressions in images and videos while retaining the original mouth movements. Despite advancements, SPFEM still struggles with accurate lip synchronization due to the complex interplay between facial expressions and mouth shapes. Capitalizing on the advanced capabilities of Audio-Driven Talking Head Generation (AD-THG) models in synthesizing precise lip movements, our research introduces a novel integration of these models with SPFEM. We present a new framework, Talking Head Facial Expression Manipulation (THFEM), which utilizes AD-THG models to generate frames with accurately synchronized lip movements from audio inputs and SPFEM-altered images. However, increasing the number of frames generated by AD-THG models tends to compromise the realism and expression fidelity of the images. To counter this, we develop an adjacent frame learning strategy that finetunes AD-THG models to predict sequences of consecutive frames. This strategy enables the models to incorporate information from neighboring frames, significantly improving image quality during testing. Our extensive experimental evaluations demonstrate that this framework effectively preserves mouth shapes during expression manipulations, highlighting the substantial benefits of integrating AD-THG with SPFEM.

Multi-label recognition with partial labels (MLR-PL), in which only some labels are known while others are unknown for each image, is a practical task in computer vision, since collecting large-scale and complete multi-label datasets is difficult in real application scenarios. Recently, vision language models (e.g., CLIP) have demonstrated impressive transferability to downstream tasks in data limited or label limited settings. However, current CLIP-based methods suffer from semantic confusion in MLR task due to the lack of fine-grained information in the single global visual and textual representation for all categories. In this work, we address this problem by introducing a semantic decoupling module and a category-specific prompt optimization method in CLIP-based framework. Specifically, the semantic decoupling module following the visual encoder learns category-specific feature maps by utilizing the semantic-guided spatial attention mechanism. Moreover, the category-specific prompt optimization method is introduced to learn text representations aligned with category semantics. Therefore, the prediction of each category is independent, which alleviate the semantic confusion problem. Extensive experiments on Microsoft COCO 2014 and Pascal VOC 2007 datasets demonstrate that the proposed framework significantly outperforms current state-of-art methods with a simpler model structure. Additionally, visual analysis shows that our method effectively separates information from different categories and achieves better performance compared to CLIP-based baseline method.

Speech-preserving Facial Expression Manipulation (SPFEM) aims to alter facial emotions in video content while preserving the facial movements associated with speech. Current works often fall short due to the inadequate representation of emotion as well as the absence of time-aligned paired data-two corresponding frames from the same speaker that showcase the same speech content but differ in emotional expression. In this work, we introduce a novel framework, Self-Supervised Emotion Representation Disentanglement (SSERD), to disentangle emotion representation for accurate emotion transfer while implementing a paired data construction module to facilitate automated, photorealistic facial animations. Specifically, We developed a module for learning emotion latent codes using StyleGAN's latent space, employing a cross-attention mechanism to extract and predict emotion editing codes, with contrastive learning to differentiate emotions. To overcome the lack of strictly paired data in the SPFEM task, we exploit pretrained StyleGAN to generate paired data, focusing on expression vectors unrelated to mouth shape. Additionally, we employed a hybrid training strategy using both synthetic paired and real unpaired data to enhance the realism of SPFEM model's generated images. Extensive experiments conducted on benchmark datasets, including MEAD and RAVDESS, have validated the effectiveness of our framework, demonstrating its superior capability in generating photorealistic and expressive facial animations.

Pile inventory is a common practice in terrestrial laser scanner applications. However, there are two main challenges when using the traditional terrestrial laser scanning (TLS) for pile inventory. One is the high cost and the other is the unreachability problem within a limited space. This letter develops a cost-effective system combining two single-pulse scanners and a rangefinder. The sensors are coupled together on a sliding rail to collect 3-D points for pile inventory. The proposed system is applied to an indoor scenario in a gold and copper open-pit mine. We investigate the effect of inclination angles on the point cloud. In addition, the generated point clouds are compared with those obtained by an advanced scanner in simultaneous localization and mapping configuration. The comparison shows higher completeness when using our proposed system. Cross-validation results show ~99.9% repeatability of the proposed system for pile inventory.

It is necessary to timely and accurately estimate the surface deformations in mining areas, especially the three-dimensional (3D) deformations during surface movement. At present, nearly all mining-induced 3D deformations retrieved by interferometric synthetic aperture radar (InSAR) pertain to the SAR imaging interval. Research on progressive 3D deformations during surface movement is limited, and the existing approaches are unsatisfactory in practical engineering. Aiming at these challenges, we proposed a novel method for deriving mining-induced 3D surface deformations at any moment by integrating single InSAR interferogram (SII), the Gompertz time function, and the probability integral model (PIM), named the SII-GPIM method. We established an inversion approach for GPIM parameters and derived the mining-induced 3D surface deformations at any moment. Subsequently, we conducted experiments considering two ALOS PALSAR images in the Huaibei mining area. The accuracy of the proposed method was evaluated in subsidence, tilt, curvature, horizontal displacement, and horizontal strain. Compared with existing methods, the SII-GPIM method is state of the art. Additionally, we assessed the building damage, performance of parameter inversion, and method generality. The results demonstrated that the proposed method can accurately determine the mining-induced 3D surface deformations and deformation level at any moment under different geological mining conditions. Moreover, accurate GPIM parameters can be acquired with only two SAR images and traditional measurement is nearly not required. Consequently, the SII-GPIM method owns great value for improving economic efficiency, assessing building damage, and restoring the ecological environment in the mining area.

The registration of terrestrial point clouds is a crucial technique to obtaining the entire scene geometry. However, urban scenes, characterized by repetitive components, symmetrical structures, and object occlusion, pose ambiguity challenge of matches in the accurate registration of terrestrial point clouds. This article proposes a coarse registration approach using 2-lines congruent sets (2LCS) for urban point clouds. 3-D line features are extracted in the point clouds and classified into two categories according to directions. With that, a pair of 3-D line features from different categories is selected in the target point cloud to construct the 2-lines base with significant geometric characteristics. Next, the congruent 2-lines bases are identified in the source point cloud using geometric constraints, leading to the 2LCS. The 2LCS are then used to estimate the rigid motion between pairwise point cloud, leading to a set of candidate transformation parameters. The largest common point (LCP) set and maximum line structure consistency (MLSC) are integrated to evaluate the transformations. Finally, the procedure is repeated based on the RANSAC scheme to achieve optimal transformation values. We applied the proposed method to nine urban scenes with varying conditions and compared its performance with other state-of-the-art methods. The results show that our proposed approach is more robust for registering terrestrial point clouds in urban scenes with repetitive components, symmetrical structures, and limited overlap, leading to optimal or near-optimal results after fine registration optimization.

Detection and tracking of oil spills using Moderate Resolution Imaging Spectroradiometer (MODIS) have received a considerable attention over the past few years, notably due to the wide area coverage, short revisit frequency and free of charge. Oil spills appear different signatures from the normal seawater depending on the spectral and spatial characteristics of remote sensing image. This paper uses a local difference box-counting (DBC) algorithm to construct the textural fractal map (TFM) from the MODIS imagery. Based on that, the oil spills can be highlighted, thus to be easily extracted. The DBC approach is verified over Mediterranean Sea on MODIS images in sun glint condition. Results show that the sun glint effects on the MODIS imagery have been removed from its fractal textural map. Following that, two oil slicks can be easily extracted from temporal MODIS TFMs for mapping their trajectories.

The 4-points congruent sets (4PCS) techniques have widely been used for global registration of point clouds in terrestrial laser scanning (TLS) applications. Nevertheless, due to many 4PCS methods adopt a downsampling strategy in the collection of correspondences; it is challenging to obtain a real congruent set of tuples in different point clouds with varying density. This letter embeds multiscale sparse features (MSSF) into 4PCS to enable efficient global registration of TLS clouds. Specifically, multiscale clusters are used to extract point features, among which a sparse coding is performed to obtain the representative MSSF. The obtained MSSF are then embedded into the 4PCS variants, taking both geometrical structure and representative feature similarity into account when performing a four-point congruent tuples matching. Moreover, a normal constraint is considered in the selection of noncoplanar four-point bases rather than coplanar ones in the source cloud. This configuration decreases the number of four-point bases and thus improves the processing efficiency and registration accuracy. The proposed method was applied to two experiments with TLS point clouds from building and hillslope scenarios, respectively. In addition, the proposed MSSF configuration was embedded in two 4PCS variants, and compared with the original methods for global registration. The comparison shows evident improvements in terms of the registration accuracy and efficiency if our embedment is used.

Due to errors in sensors and positioning, there exist mismatches between different phases of mobile laser scanning point clouds, which impede the application of point cloud, such as changing detection and deformation monitoring. To rectify such mismatches, we designed a 3-D deep feature construction method for point cloud registration. The proposed method combines two 3-D convolutional neural networks into a uniform deep learning model to extract 3-D deep features. First, the corresponding points and noncorresponding points are set to train the deep learning model to minimize the distance between corresponding points’ features and maximize the distance between features of noncorresponding points. Second, in the test phase, the 3-D deep feature for each keypoint was extracted by the trained deep learning model. This could be used to determine the corresponding points by the$k$-dimensional tree and random sample consensus (RANSAC) algorithm. Finally, a transformation matrix was calculated based on the corresponding points and was then applied to point cloud registration. The experimental results illustrated that the proposed method of using 3-D deep features is more efficient at a corresponding point search than representatives of three existing methods. It also improved registration accuracy.

Stochastic gradient descent and other adaptive optimization methods such as RMSprop, and Adam have been proved effective for training deep neural networks [1], [2]. Within each epoch of these methods, the whole training set is involved. In general, large training datasets have data redundancy. In this paper, we investigate an algorithm that reduce the training time of CNN by dropping certain samples out. Thus, it is called DropSample. This method can be viewed as a special type of truncated cross-entropy loss with a finite margin. We design experiments on several datasets to demonstrate the effects of acceleration. The results show that this method could decrease the training time of multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) significantly. Despite reduced number of training samples, the accuracies of networks are similar, or even better.

A detection procedure for a moving magnetic dipole target by using at least three scalar magnetometers is proposed in this letter. The magnetometers are deployed on the ground or seafloor, while the target is traveling along a straight line at a constant speed above the ground. First, the range from each magnetometer to the closest point of approach (CPA) on the flight line as well as the instant that the target passes by the CPA is estimated by adopting the orthonormal basis function expansion method. Then, the moving parameters, including the moving direction, speed, and elevation, are found through basic geometry relations. Finally, the magnetic moment is determined by solving a linear system by using the least-squares method. Both simulated and field experimental data are employed to demonstrate the proposed scheme.

Gully erosion removes large volumes of soil and water from complex areas. It's difficult to quantify the loss of soil because the footprint of individual gullies is too complex to be captured by most common measures, such as general available digital elevation models (DEMs). LiDAR (Light Detection and Ranging) is a recent innovation technique widely used in topographic surveying, smart city, historic preservation, revere engineering, etc. Ground-based LiDAR has the potential to provide the required data with non-contacting, convenient, fast and high accuracy at lower cost. In this study, two different Terrestrial Laser scanners (TLS) were used to monitor an individual gully. Terrain complexity and volumetric estimates of the eroded sediment were produced by comparing the LiDAR-derived DEMs under different point cloud resolution. The result showed that the surface roughness and erosion sediment were increased with higher resolution point cloud. However, time-consuming and computational complexity was suddenly raised as dense point cloud provided more details information. This paper provides a new method for how to choose a moderate resolution in dealing with gully erosion in regional area.

This paper presents a two-level Active Learning (AL) classification method for the interactive detection of earthquake-induced debris via the synergetic use of post-disaster Very High Resolution (VHR) satellite and local decimeter-resolution aerial images. The proposed method is performed by interactively guiding the human expert in the collection of labeled training samples from aerial images and optimally planning further aerial surveys. A label propagation mechanism is adopted to reliably propagate the labeled pixels annotated by the user on the aerial image by visual photointerpretation to the (lower resolution) satellite image. The experimental analysis is carried out using post-disaster images of the 2010 Yushu earthquake in China. The obtained results confirm that the proposed method can significantly reduce the user annotation effort and the cost for acquiring additional aerial images leading to more accurate and timely debris detection maps.

Many methods have been developed to detect damaged buildings due to earthquake. However, little attention has been paid to analyze slightly affected buildings. In this letter, an unsupervised method is presented to detect earthquake-triggered “roof-holes” on rural houses from unmanned aerial vehicle (UAV) images. First, both orthomosaic and gradient images are generated from a set of UAV images. Then, a modified Chinese restaurant franchise model is used to learn an unsupervised model of the geo-object classes in the area by fusing both oversegmented orthomosaic and gradient images. Finally, “roof-holes” on rural houses are detected using the learned model. The performance of the proposed method is evaluated in terms of both qualitative and quantitative indexes.

This study utilizes the unmanned aerial vehicle (UAV) to acquire high resolution images for feature matching, resulting in a point cloud. A progressive morphological filter is used to filter out nonground object points from point cloud. Multi-scale and different shape filter windows are adopted for the morphological filter to achieve good performance. The results show that multi-scale and multi-shape window can improve the performance of morphological filter compared with single direction filter window, as nonground objects cannot be completely removed with single direction filter window. With multi-shape or 2-D filter window, buildings can be effectively removed and ground points can be reserved.

There is no single sensor can acquire the complete information for disaster monitoring. This study investigates the applicability of registering multiple point clouds obtained from unmanned aerial vehicle (UAV) images and terrestrial laser scanning (TLS). Low attitude images with high overlaps were collected by an eight-rotor UAV platform and image-based 3D modeling techniques are used to generate 3D point cloud, covering most of roof information of the damaged buildings. TLS was used to collect the side information of the damaged buildings with multiple scans. Point clouds from the two platforms are iteratively registered using a method, from coarse to fine, to get complete geometry of the study area. Geometric features are subsequently extracted to help for the identification of damage degree of buildings. Experimental result shows that by analyzing the intersection lines of plane features, we can further detect the building's inclination.

Triangular Irregular Network (TIN), especially Constrained Delaunay Triangular Irregular Network (CD-TIN), has been proven to be more accurate for the modeling of complex urban terrain, compared with raster represented digital elevation model (DEM). DEM with high precision is of importance to simulate urban flood inundation. In the paper, the triangular-prism sets are introduced to calculate the inundation water surface within each basin. It is worth mentioning that the explored dichotomy numerical solution is efficient on the simulation of the inundation depth. The case of ramp-shape urban terrain validates the correctness of the algorithm. And the application of the “7.21” storm in main campus of Beijing Normal University (BNU) verifies the usability and practicality of the algorithm. The cases demonstrate the feasibility of the proposed algorithm for simulation of urban flood inundation associated with urban surface runoff and drainage system capacity information.

We address the problem of efficient image matching for large, highly redundant photo collections with highly complex topologies, such as photos acquired by unmanned aerial vehicles (UAV), focusing on disaster monitoring. Our approach conducts a skeleton graph which simplifies the image topology with the consideration of image importance and topological relationship. We define the image with the highest importance weight as candidate, adding the remaining images referring to the topology skeleton. To conduct the skeletal graph, the image topology is first computed depending on the overlapping relationships between images. Experimental results show that our technique drastically limits the searching range that is for feature similarity computation, resulting in dramatic speed up. A final bundler adjustment is implemented in the procedure of scene reconstruction, and the completeness and accuracy are far more comparable to the traditional method.

In this paper, a fast and robust video copy detection scheme is proposed, which is suitable for the DCT-coded video sequences. To address the efficiency and effectiveness issue, we extract the video signature directly from the compressed domain. The video sequence clusters are constructed with a fixed length. Each cluster consists of several fictional key-frames. For each key-frame, some low-middle frequency full DCT coefficients are obtained directly from block DCT coefficients, and their ordinal measure is computed and acts as video signature. A rotation compensational strategy is further employed to resist the rotation attacks. The experimental results show that the proposed scheme can be resilient to various types of video transformations, including scaling, rotation, speed change, text insertion, and subsequence insertion/deletion etc.. The most important thing is that the proposed approach not only handles geometric distortion perfectly, but also reduces the computation costs substantially.

Recently, many new encoding techniques have been suggested for magnetic resonance imaging and an important image reconstruction problem has been raised in order to fully exploit the advantages promised by these new encoding techniques. In terms of frames in an L2 (R) space, we treat these encoding techniques in a unified perspective and propose a solution for this image reconstruction problem. We first develop a matrix form of frame theory and apply it to numerically construct the duals of the encoding functions, which are the transverse excitation profiles generated by radio-frequency pulses along a linear magnetic field gradient. An analysis of the signal-to-noise ratio (SNR) is also presented. Simulations have been carried out and they show that our image reconstruction algorithm minimizes the mean square error between the original and reconstructed images. The SNRs evaluated from simulations agree with our theoretical predictions.