Natural language video localization plays a pivotal role in video understanding, and leveraging weakly-labeled data is considered a promising approach to circumvent the laborintensive process of manual annotations. However, this approach encounters two significant challenges: 1) limited input distribution, namely that the limited writing styles of the language query, annotated by human annotators, hinder the model’s generalization to real-world scenarios with diverse vocabularies and sentence structures; 2) the incomplete ground truth, whose supervision guidance is insufficient. To overcome these challenges, we propose an omnipotent distillation algorithm with large language models (LLM). The distribution of the input sample is enriched to obtain diverse multi-view versions while a consistency then comes to regularize the consistency of their results for distillation. Specifically, we first train our teacher model with the proposed intra-model agreement, where multiple sub-models are supervised by each other. Then, we leverage the LLM to paraphrase the language query and distill the teacher model to a lightweight student model by enforcing the consistency between the localization results of the paraphrased sentence and the original one. In addition, to assess the generalization of the model across different dimensions of language variation, we create extensive datasets by building upon existing datasets. Our experiments demonstrate substantial performance improvements adaptively to diverse kinds of language queries.

This paper for the first time leverages multi-modal videos for weakly-supervised temporal video grounding. As labeling the video moment is labor-intensive and subjective, the weakly-supervised approaches have gained increasing attention in recent years. However, these approaches could inherently compromise performance due to inadequate supervision. Therefore, to tackle this challenge, we for the first time pay attention to exploiting complementary information extracted from multi-modal videos (e.g., RGB frames, optical flows), where richer supervision is naturally introduced in the weaklysupervised context. Our motivation is that by integrating different modalities of the videos, the model is learned from synergic supervision and thereby can attain superior generalization capability. However, addressing multiple modalities† would also inevitably introduce additional computational overhead, and might become inapplicable if a particular modality is inaccessible. To solve this issue, we adopt a novel route: building a multi-modal distillation algorithm to capitalize on the multi-modal knowledge as supervision for model training, while still being able to work with only the single modal input during inference. As such, we can utilize the benefits brought by the supplementary nature of multiple modalities, without compromising the applicability in practical scenarios. Specifically, we first propose a cross-modal mutual learning framework and train a sophisticated teacher model to learn collaboratively from the multi-modal videos. Then we identify two sorts of knowledge from the teacher model, i.e., temporal boundaries and semantic activation map. And we devise a local-global distillation algorithm to transfer this knowledge to a student model of single-modal input at both local and global levels. Extensive experiments on large-scale datasets demonstrate that our method achieves state-of-the-art performance with/without multi-modal inputs.

One-shot skeleton action recognition, which aims to learn a skeleton action recognition model with a single training sample, has attracted increasing interest due to the challenge of collecting and annotating large-scale skeleton action data. However, most existing studies match skeleton sequences by comparing their feature vectors directly which neglects spatial structures and temporal orders of skeleton data. This paper presents a novel one-shot skeleton action recognition technique that handles skeleton action recognition via multi-scale spatial-temporal feature matching. We represent skeleton data at multiple spatial and temporal scales and achieve optimal feature matching from two perspectives. The first is multi-scale matching which captures the scale-wise semantic relevance of skeleton data at multiple spatial and temporal scales simultaneously. The second is cross-scale matching which handles different motion magnitudes and speeds by capturing sample-wise relevance across multiple scales. Extensive experiments over three large-scale datasets (NTU RGB+D, NTU RGB+D 120, and PKU-MMD) show that our method achieves superior one-shot skeleton action recognition, and outperforms SOTA consistently by large margins.

3D Skeleton-based human action recognition has attracted increasing attention in recent years. Most of the existing work focuses on supervised learning which requires a large number of labeled action sequences that are often expensive and time-consuming to annotate. In this paper, we address self-supervised 3D action representation learning for skeleton-based action recognition. We investigate self-supervised representation learning and design a novel skeleton cloud colorization technique that is capable of learning spatial and temporal skeleton representations from unlabeled skeleton sequence data. We represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. Specifically, we design a two-steam pretraining network that leverages fine-grained and coarse-grained colorization to learn multi-scale spatial-temporal features. In addition, we design a Masked Skeleton Cloud Repainting task that can pretrain the designed auto-encoder framework to learn informative representations. We evaluate our skeleton cloud colorization approach with linear classifiers trained under different configurations, including unsupervised, semi-supervised, fully-supervised, and transfer learning settings. Extensive experiments on NTU RGB+D, NTU RGB+D 120, PKU-MMD, NW-UCLA, and UWA3D datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins and achieves competitive performance in supervised 3D action recognition as well.

This paper for the first time explores audio-visual event localization in an unsupervised manner. Previous methods tackle this problem in a supervised setting and require segment-level or video-level event category ground-truth to train the model. However, building large-scale multi-modality datasets with category annotations is human-intensive and thus not scalable to real-world applications. To this end, we propose cross-modal label contrastive learning to exploit multi-modal information among unlabeled audio and visual streams as self-supervision signals. At the feature representation level, multi-modal representations are collaboratively learned from audio and visual components by using self-supervised representation learning. At the label level, we propose a novel self-supervised pretext task i.e. label contrasting to self-annotate videos with pseudo-labels for localization model training. Note that irrelevant background would hinder the acquisition of high-quality pseudo-labels and thus lead to an inferior localization model. To address this issue, we then propose an expectation-maximization algorithm that optimizes the pseudo-label acquisition and localization model in a coarse-to-fine manner. Extensive experiments demonstrate that our unsupervised approach performs reasonably well compared to the state-of-the-art supervised methods.

Differential space-time coding methods have been investigated for ultra-wideband communication systems to avoid channel estimation. However, their performance in the line-of-sight (LOS) environment is worse than the recently proposed orientational beamforming (OBF) system under low signal-to-noise ratios (SNRs). On the other hand, the OBF system cannot work well in the non-LOS (NLOS) environment. To address these issues, a hybrid differential detection (HDD) scheme is proposed in this paper, which combines the OBF system with a proposed differential OBF (DOBF) system. First, the DOBF system with a fully differential detection scheme is proposed, whose performance is almost the same as the differential space-time block coding method. However, its detection complexity only linearly increases with the number of transmitting antennas. Then, the HDD scheme is proposed, with its decision statistic being a combination of a modified OBF decision statistic and the DOBF decision statistic. In the NLOS environment, the HDD scheme is reduced to the DOBF system. In the LOS environment, a combination coefficient is obtained by mapping an SNR-related factor through a sigmoid function. The optimal parameters for the sigmoid function are determined through simulations, with which the proposed HDD scheme can achieve overall better performance than the OBF and DOBF systems.

In this paper, we investigate and extend the linearly constrained minimum variance (LCMV) algorithm for conventional wideband beamforming system to the recently proposed orientational beamforming (OBF) system. An orientational LCMV (O-LCMV) algorithm is proposed. It is constructed on the orientation dimension, and an orientational constraint instead of directional constraint is used to guarantee an orientational gain for the desired signal. This perfectly solves the problem of rejecting an interference arriving from the same direction as the desired signal, which current adaptive directional beamforming algorithms cannot handle. Numerous simulations show that the O-LCMV algorithm for the OBF system works effectively regardless of the number of narrowband interferences (NBIs) or their DOAs if the NBIs have the same center frequency. As the number of different center frequencies of the NBIs increases, the performance degrades slightly.

Skeleton-based human action recognition has attracted increasing attention in recent years. However, most of the existing works focus on supervised learning which requiring a large number of annotated action sequences that are often expensive to collect. We investigate unsupervised representation learning for skeleton action recognition, and design a novel skeleton cloud colorization technique that is capable of learning skeleton representations from unlabeled skeleton sequence data. Specifically, we represent a skeleton action sequence as a 3D skeleton cloud and colorize each point in the cloud according to its temporal and spatial orders in the original (unannotated) skeleton sequence. Leveraging the colorized skeleton point cloud, we design an auto-encoder framework that can learn spatial-temporal features from the artificial color labels of skeleton joints effectively. We evaluate our skeleton cloud colorization approach with action classifiers trained under different configurations, including unsupervised, semi-supervised and fully-supervised settings. Extensive experiments on NTU RGB+D and NW-UCLA datasets show that the proposed method outperforms existing unsupervised and semi-supervised 3D action recognition methods by large margins, and it achieves competitive performance in supervised 3D action recognition as well.

Imaging of ground multiple moving targets in a synthetic aperture radar (SAR) system is a challenging task due to the fact that targets are defocused owing to motions and contaminated by the strong background clutter. Motivated by recent advances in deep learning, a novel deep convolutional neural network (CNN)-based method, DeepImaging, is proposed for ground moving target imaging (GMTIm). Different from conventional imaging methods relying on the prior knowledge of imaging, the proposed DeepImaging is directly trained to learn an implicit imaging model of multiple moving targets. It is free of motion parameter estimation and iteration process. Then, the trained DeepImaging, as an imaging processor, can be applied to the SAR complex received data after clutter suppression to achieve the multiple moving target imaging and the residual clutter elimination simultaneously. Simulations and experiments on the Gotcha data show that the proposed method achieves significant improvements over existing state-of-the-art GMTIm methods in terms of imaging quality and efficiency.

In this paper, we consider designing the Filter-and-forward (FF) relay beamforming in frequency-selective channels using a new approach. The proposed approach aims to minimize the output power at the destination side while keeping the response to the desired signal at a constant level, and the problem is subject to both total and individual relay transmit power constraints. It is shown that the proposed beamforming design scheme is equivalent to the SINR maximization formulation, in terms of achieving the same output SINR. Despite the equivalence in performance, the proposed approach requires significantly lower computational load for solving the problem.

This paper addresses the Direction-of-Arrival (DOA) estimation problem for amplitude modulated signals whose number is more than that of the array sensors. The proposed method is based on an idea of virtual array. For source signals with amplitude modulation, such as binary phase shift keying (BPSK) and M-ary amplitude shift keying (M-ASK), we show that introducing in virtual array actually gives rise to processing the fourth-order moments of array output, which is related to higher-order statistics (HOS) techniques. While traditional HOS methods in array processing mainly exploit higher-order cumulants of the received data, we propose a DOA estimation method based on the fourth-order moments, which is of lower computational load than the fourth-order cumulants. Simulation results demonstrate the effectiveness of the proposed method for estimating DOAs of more source signals than array elements.

Many advanced adaptive beamformers are robust against arbitrary array steering vector (ASV) mismatches within a presumed uncertainty set. Adaptive array tolerating significant steering direction error usually requires a large size of ASV uncertainty set. In such case, however, the output signal-to-interference-plus-noise ratios (SINRs) of robust methods degrade quickly with the increasing size of the uncertainty set. In this paper, we propose a new compact ASV uncertainty set which is modelled explicitly by the uncertainty on steering direction and the other arbitrary ASV errors. A robust adaptive beamformer is derived based on this new ASV uncertainty set. To eliminate the non-convex constraint on array magnitude response, we force the real part of array response to exceed unity regarding the ASVs within the uncertainty set. Furthermore, using the worst-case optimization technique, the resultant beamformer is formulated as a quadratic optimization problem with semi-infinite second-order cone (SOC) constraints. Numerical studies show that a large robust response region is easy to achieve and the resultant beamformer achieves high performance on SINR enhancement.

A novel robust adaptive beamformer, with new robust constraints on array magnitude response was proposed by utilizing the autocorrelation sequence of array weight vector and the worst-case optimization technique. The proposed adaptive beamformer was formulated as a linear programming problem with second-order cone semi-infinite constraints, which can be eliminated by using the sampling technique. In this paper, we transform these semi-infinite second-order cone constraints into some norm constraints and linear matrix inequality (LMI) constraints. The advantage of this new formulation of the problem is that the sampling of the angles is avoided. The exact optimal result of the problem can be obtained instead of the approximated one provided by sampling technique. The resultant beamformer possesses superior robustness against arbitrary array imperfections and high performance on signal-to-interference-plus-noise ratio (SINR) enhancement even with a large controlled robust response region.

In this paper, a novel robust adaptive beamformer is proposed based on semidefinite programming (SDP) and worst- case optimization. With the SDP formulation, the array output power and magnitude response can be expressed as linear functions. New constraints on magnitude response are introduced in the adaptive array. The proposed method can flexibly control the robust response region with a specific beamwidth and response ripple. In practical applications, the array suffers from having not only steering direction error, but also many other array imperfections. To make the adaptive beamformer robust against all kinds of array imperfections, the worst-case optimization technique is proposed to reconstruct the beamformer. By minimizing the array output power with respect to the worst-case effect of array imperfections, the resultant beamformer possesses superior robustness against arbitrary array imperfections. Since the constraints on magnitude response are inequality constraints, most of them are inactive in the optimization process so that few degrees of freedom (DOFs) of the adaptive beamformer are consumed. Consequently, the resultant beamformer has high performance on signal-to-interference-plus-noise ratio (SINR) improvement. Simple implementation, flexible performance control as well as significant SINR enhancement support the practicability of the proposed method.

In this paper, a novel robust adaptive antenna array with new linear matrix inequality (LMI) constraints on the array magnitude response is proposed. Most of the advanced robust adaptive beamformers, which are robust against arbitrary array steering vector (ASV) errors within a presumed uncertainty set, have poor performance facing a large uncertainty on direction-of-arrival (DOA). Since the DOA uncertainty may result in a large error in the ASV, a big uncertainty set is required to make the adaptive array robust against a large DOA error. The adverse effect is the degraded output signal-to-interference- plus-noise ratio (SINR). In this paper, a compact model of ASV uncertainty set is described by the uncertainties on DOA and other arbitrary ASV errors explicitly. Based on this new ASV uncertainty set, a novel robust adaptive beamformer is derived. In order to eliminate the semi-infinite constraint in the proposed beamformer, new LMI constraints are derived. The proposed beamformer possesses superior robustness against arbitrary array imperfections as well as a large DOA error. A large robust response region is easy to achieve and the resultant beamformers still have high performance on SINR enhancement.

In this paper, a novel robust adaptive beamformers with linear matrix inequality (LMI) constraints on magnitude response is proposed. The recently proposed robust adaptive beamformer has a drawback that some of the constraints are semi-infinite. Although the sampling technique provides a good approximation of the optimization problem, the cost is the heavy computation load. In order to overcome this problem, new LMI constraints are derived in this paper to replace the semi- infinite constraints so that the sampling is avoid and an exact solution can be obtained.

A hybrid protocol for application level multicast (HPAM) for live video streaming without native IP multicast support is proposed. HPAM exploits the simplicity and optimality of a lightweight, centralized server with the robustness and scaleability of distributed clients. HPAM self-organizes clients on the fly to form efficient source-based overlay trees while the server facilitates peer discovery and also serves as a reliable backup should the distributed algorithms fails. Tree construction, refinement and recovery from partitions are carried out independently by the clients. Simulation results show that HPAM can build and maintain reasonably latency-efficient overlay trees with a lower overhead than a fully centralized system (host based multicast) and yet more responsive to group dynamics and network environment than a fully distributed system (host multicast).

A robust Capon beamformer (RCB) with a new constraint on the uncertainty of nominal array steering vector (ASV) is proposed in this paper. The new constraint is constructed by replacing the nominal ASV with a projected one onto the signal-plus-interference subspace. The proposed RCB achieves higher output signal-to-noise-plus-interference ratio (SINR) compared with the conventional RCBs. Theoretical analysis and simulation results show the effectiveness of the proposed method.

A steerable audio system can be realized using parametric array. However, the available steerable angle is often limited by the sampling interval used in the digital system. As such, the smallest steerable angle is large (/spl sim/26/spl deg/) for several hundred kilohertz of sampling frequency. Although there are some fractional delay or frequency domain algorithms can be used to improve the steering angle, most of the algorithms are either computational intensive or introduce error during the process. In this paper, an algorithm is proposed to rectify this problem by applying separate delays to the carrier and sideband frequencies. Different weighting functions also added to the carrier and sideband frequencies to control the difference frequency's beamwidth and sidelobe. Most importantly, the proposed system can steer the difference frequency to a small angle with minimal computation.

For a digital beamsteering system, the smallest time delay available is equal to the sampling period of the digital signal processing (DSP) board. As most of the time the sampling frequency is not high enough, the smallest steering angle available is large, which is undesirable. This limitation also occurs when performing beamsteering in a parametric array digitally. Although partial delay or frequency domain algorithms can be used to improve the steering angle, most of the algorithms are either computational intensive or introduce error during the process. In this paper, an algorithm is proposed to beamsteer the difference frequency in parametric array. The proposed system can be used to steer the difference frequency to a small angle, without the need to increase the sampling frequency or implement partial delay.

We propose a robust adaptive blind multichannel identification algorithm in the frequency domain. It utilizes the fast Fourier transform (FFT) to reduce computational complexity when the channel impulse response (IR) is long. Moreover, the Newton-LMS algorithm is obtained in the frequency domain with small computational load to improve the convergence speed. The advantage of the proposed method is its robustness to input noise, especially when the channel IR is long, e.g., the room acoustic IR with a length up to hundreds or thousands taps. The conventional methods cannot obtain an estimate with acceptable accuracy and low computational load. The situation becomes worse when the input signal-to-noise ratio (SNR) is low. Simulation results show that the proposed method is suitable for estimating long multichannel IRs in practical environments.

A multichannel wideband signal dereverberation and enhancement method is proposed in this paper. It uses the blindly estimated impulse responses (IR) relating the signal source and each sensor to form a multiple input inverse filter (MINT) for speech dereverberation and enhancement with extended generalized sidelobe canceller (GSC). With the replacement of MINT for fixed beamformer and modification of blocking matrix in the conventional GSC, the resulting extended GSC not only dereverberates the distorted target signal, but also suppresses the interference/noise. Computer simulation results show the effectiveness of the proposed method.

Sound reproduction in air by using a parametric acoustic array has been investigated for a few decades. Two inaudible ultrasonic frequencies are produced from the parametric array. Due to the nonlinearity of air, it is possible to produce an audible frequency with its frequency equal to the difference in the two ultrasonic frequencies. However, there is not much work done in controlling the beam pattern of the difference frequency generated by the primary waves. In this paper, an algorithm is proposed to control the sidelobe level of the difference frequency directivity. By making use of array signal processing techniques, the algorithm is also capable of producing a constant beamwidth for broadband difference frequency.

The sound reproduction in air by using a parametric acoustic array [P.J. Westervelt, 1963] has been reported for a few decades. Two inaudible ultrasonic frequencies are produced from the parametric array. Due to the nonlinearity of air, it is possible to produce an audible frequency with its frequency equal to the difference in the two ultrasonic frequencies. However, there is not much work done in controlling the beam pattern of the difference frequency generated by the primary waves. In this paper, an algorithm is proposed to control the sidelobe level of the difference frequency directivity. By making use of array signal processing techniques, the algorithm is also capable of producing a constant beamwidth for broadband difference frequency.

A new hierarchical image registration algorithm is presented which achieves subpixel accuracy for images with both translational and rotational movements. This wavenumber domain algorithm is efficient compared to the conventional approaches based on interpolation or correlation in spatial or frequency domain. In addition, the proposed approach is robust and can achieve subpixel accuracy registration even when the images contain aliasing errors due to undersampling. This hierarchical searching method which involved coarse and fine search further enhances the convergence time of registration. The accuracy of the proposed approach is demonstrated through computer simulations for different types of images.

A new quiescent pattern control strategy for the generalized sidelobe canceller (GSC) structure is presented. The synthesis technique is based on the constrained optimization algorithm with a penalty function used to design the quiescent array. The proposed quiescent array beampattern is highly directional with which an effective cancellation of multiple interference signals can be achieved. The design technique also permits the adaptive dimension of the GSC to be reduced significantly while preserving the quiescent beampattern. Numerical results showed that the new GSC structure is able to reject strong directional interference as compared to the full processor.

The authors describe an exploratory vergence method which has the ability to detect occlusion and specular highlights using information that is inherent in the vergence process. They propose an active exploratory vergence method based on concurrent cross-correlation of multi-scale stereo images. This method is capable of detecting occlusion and specular highlights concurrently during the vergence process. It not only improves computational efficiency but also leads to the exploration of a better viewing direction and position that overcome the problem created by occlusion and specular highlights. An efficient parallel implementation of the multi-scale cross-correlation algorithm is also described. The robustness of the proposed method against occlusion and specular highlights is demonstrated.>

A fast iterative algorithm for high-resolution harmonic retrieval is presented. By using a projection matrix, the prediction matrix equation is recast into a form where the iterative method can be applied. This form ensures that the iterative process converges to a unique minimum-norm least squares solution. The conjugate gradient method is used to speed-up the rate of convergence of the iterative process. No singular value decompositions of a data matrix or eigenvalue decompositions of a covariance matrix are needed. Furthermore, no prior information regarding the number of signals is required. It is shown that by incorporating known information about the signals into the iterative process, the algorithm will perform better than the MFBLP method of Tufts and Kumarsian (1982).>

An improved MUSIC algorithm for superresolution array processing is presented. The approach is formulated as a constrained least square minimization problem. The solution is to find a weight vector which is as close as possible to a predetermined weight vector maintaining a closed flat response over a spatial region of interest and at the same time lying in the noise subspace. The relationship of this approach to other subspace techniques is explored. Numerical results are presented to illustrate the performance achievable.>

An iterative regularized pseudoinverse (RPI) algorithm for restoring a linearly degraded image in the presence of noise is described. The array processing problem is recast into such a form that the iterative RPI algorithm can be used to compute the array weight vector recursively. This is done by absorbing the linear constraint vector into the array weight vector using a projection matrix. The convergence rate of the algorithm is very fast, and it is capable of resolving fully coherent sources. The performance of the algorithm is demonstrated through numerical examples.>

A new approach to the design of narrowband antenna array processors is proposed. The new approach allows a wide variety of possible errors to be incorporated in the problem formulation and leads to a robust optimum weight vector. The new approach can be used to make the optimum system robust against channel phase errors, array geometry errors and pointing errors, to name a few. Initially a general quadratic constraint on the weights is developed. However, it is then shown that the quadratic constraint can be replaced by linear constraints or at most linear constraints plus norm constraint. These latter set of constraints are no more complex than those required for designs which do not incorporate robustness features explicitly. The new optimum processor can be implemented adaptively.

This paper presents a new set of linear constraints for designing broadband time domain element space processor which can handle a variety of steering situations, namely, no pre-steering, coarse/ quantized pre-steering and exact pre-steering. The relationship that the new processor has to other broadband processors is established. Furthermore, the approach presented enables the processors to be made robust against directional errors. Analytical as well as simulation results on the new processors are presented.

This paper describes a new approach to the design of broadband element space antenna array processor which can handle a variety of steering situations. The approach is applicable to array processor without pre-steering, with coarse pre-steering and exact pre-steering. Furthermore, the approach presented enables mismatch between signal model and actual signal scenario to be incorporated in the problem formulation. Analytical as well as simulation results on the new class of antenna array processors are presented.