Zhisheng Zheng, Puyuan Peng, Anuj Diwan, Cong Phuoc Huynh, Xiaohang Sun, Zhu Liu, Vimal Bhat, David Harwath. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Target Speech Extraction (TSE) traditionally relies on explicit clues about the speaker’s identity like enrollment audio, face images, or videos, which may not always be available. In this paper, we propose a text-guided TSE model StyleTSE that uses natural language descriptions of speaking style in addition to the audio clue to extract the desired speech from a given mixture. Our model integrates a speech separation network adapted from SepFormer with a bi-modality clue network that flexibly processes both audio and text clues. To train and evaluate our model, we introduce a new dataset TextrolMix with speech mixtures and natural language descriptions. Experimental results demonstrate that our method effectively separates speech based not only on who is speaking, but also on how they are speaking, enhancing TSE when traditional audio clues are absent. Demos are at: https://mingyue66.github.io/TextrolMix/demo/

We present a simple yet effective self-supervised pre-training method for image harmonization which can leverage large-scale unannotated image datasets. To achieve this goal, we first generate pre-training data online with our Label-Efficient Masked Region Transform (LEMaRT) pipeline. Given an image, LEMaRT generates a foreground mask and then applies a set of transformations to perturb various visual attributes, e.g., defocus blur, contrast, saturation, of the region specified by the generated mask. We then pre-train image harmonization models by recovering the original image from the perturbed image. Secondly, we introduce an image harmonization model, namely SwinIH, by retrofitting the Swin Transformer [27] with a combination of local and global self-attention mechanisms. Pre-training SwinIH with LEMaRT results in a new state of the art for image harmonization, while being label-efficient, i.e., consuming less annotated data for fine-tuning than existing methods. Notably, on iHarmony4 dataset [8], SwinIH outperforms the state of the art, i.e., SCS-Co [16] by a margin of 0.4 dB when it is fine-tuned on only 50% of the training data, and by 1.0 dB when it is trained on the full training dataset.

Traditional Unsupervised Domain Adaptation (UDA) leverages the labeled source domain to tackle the learning tasks on the unlabeled target domain. It can be more challenging when a large domain gap exists between the source and the target domain. A more practical setting is to utilize a large-scale pre-trained model to fill the domain gap. For example, CLIP shows promising zero-shot generalizability to bridge the gap. However, after applying traditional fine-tuning to specifically adjust CLIP on a target domain, CLIP suffers from catastrophic forgetting issues where the new domain knowledge can quickly override CLIP’s pre-trained knowledge and decreases the accuracy by half. We propose Catastrophic Forgetting Measurement (CFM) to adjust the learning rate to avoid excessive training (thus mitigating the catastrophic forgetting issue). We then utilize CLIP’s zero-shot prediction to formulate a Pseudo-labeling setting with Adaptive Debiasing in CLIP (PADCLIP) by adjusting causal inference with our momentum and CFM. Our PADCLIP allows end-to-end training on source and target domains without extra overhead. We achieved the best results on four public datasets, with a significant improvement (+18.5% accuracy) on DomainNet.

In this paper, we address the problem of ground-based hyperspectral image segmentation by combining pixel-level and region-level classification with a region boundary refinement approach. To this end, we represent the spatio-spectral feature of image regions by a descriptor based on Vector of Locally Aggregated Descriptors (VLAD). Further, the region boundaries are refined by minimizing the total region perimeter. Experimental results on a ground-based hyperspectral image dataset clearly demonstrate the advantage of the proposed method over recent prior works, based on several metrics.

This paper presents an approach to pedestrian detection in thermal infrared (thermal) images with limited annotations. The key idea is to adapt the abundance of color images associated with bounding box annotations to the thermal domain for training the pedestrian detector. To this end, we couple a domain adaptation component that consists of a pair of image transformers with a pedestrian detector in the thermal domain and train the entire network end-to-end. The image transformers act as a data augmentation tool that progressively improves synthetic examples on the fly for training the pedestrian detector. To aid the training process, we introduce a detection loss defined on both real thermal images and synthetic thermal images transformed from the color domain. The proposed detector outperforms existing methods on the thermal images from the KAIST detection benchmark [1].

A fundamental problem in image deblurring is to recover reliably distinct spatial frequencies that have been suppressed by the blur kernel. To tackle this issue, existing image deblurring techniques often rely on generic image priors such as the sparsity of salient features including image gradients and edges. However, these priors only help recover part of the frequency spectrum, such as the frequencies near the high-end. To this end, we pose the following specific questions: (i) Does any image class information offer an advantage over existing generic priors for image quality restoration? (ii) If a class-specific prior exists, how should it be encoded into a deblurring framework to recover attenuated image frequencies? Throughout this work, we devise a class-specific prior based on the band-pass filter responses and incorporate it into a deblurring strategy. More specifically, we show that the subspace of band-pass filtered images and their intensity distributions serve as useful priors for recovering image frequencies that are difficult to recover by generic image priors. We demonstrate that our image deblurring framework, when equipped with the above priors, significantly outperforms many state-of-the-art methods using generic image priors or class-specific exemplars.

In this paper, we propose a spatial-spectral feature fusion model with a predictive feature weighting mechanism and demonstrate its applications to the problems of hyperspectral image classification and segmentation. To address these problems, we learn a set of 1-D convolutional local spectral filters and 2-D spatial-spectral filters that feed features into a fusion module, in an end-to-end fashion. We propose a lightweight predictive feature weighting component embedded in the fusion model and consider four design fusion options, i.e., by adding or concatenating features with equal or predicted weights. For the pixel classification task, the training input consists of image patches with labeled central pixels, whereas for the spatial segmentation task, it includes the label maps of image regions. The proposed networks have been evaluated on the Indian Pines, Pavia University, and Houston University for the classification problem and the SpaceNet data set for the spatial segmentation problem. The quantitative results favor the proposed approach over the state-of-the-art methods across all the four data sets.

We present a novel image denoising algorithm that uses external, category specific image database. In contrast to existing noisy image restoration algorithms that search patches either from a generic database or noisy image itself, our method first selects clean images similar to the noisy image from a database that consists of images of the same class. Then, within the spatial locality of each noisy patch, it assembles a set of "support patches" from the selected images. These noisy-free support samples resemble the noisy patch and correspond principally to the identical part of the depicted object. In addition, we employ a content adaptive distribution model for each patch, where we derive the parameters of the distribution from the support patches. We formulate noise removal task as an optimization problem in the transform domain. Our objective function composed of a Gaussian fidelity term that imposes category specific information, and a low-rank term that encourages the similarity between the noisy and the support patches in a robust manner. The denoising process is driven by an iterative selection of support patches and optimization of the objective function. Our extensive experiments on five different object categories confirm the benefit of incorporating category-specific information to noise removal and demonstrate the superior performance of our method over the state-of-the-art alternatives.

We propose a supervised approach to the classification and segmentation of material regions in hyperspectral imagery. Our algorithm is a two-stage process, combining a pixelwise classification step with a segmentation step aiming to minimise the total perimeters of the resulting regions. Our algorithm is distinctive in its ability to ensure label consistency within local homogeneous areas and to generate material segments with smooth boundaries. Furthermore, we establish a new hyperspectral benchmark dataset to demonstrate the advantages of the proposed approach over several state-of-the-art methods.

In image deblurring, a fundamental problem is that the blur kernel suppresses a number of spatial frequencies that are difficult to recover reliably. In this paper, we explore the potential of a class-specific image prior for recovering spatial frequencies attenuated by the blurring process. Specifically, we devise a prior based on the class-specific subspace of image intensity responses to band-pass filters. We learn that the aggregation of these subspaces across all frequency bands serves as a good class-specific prior for the restoration of frequencies that cannot be recovered with generic image priors. In an extensive validation, our method, equipped with the above prior, yields greater image quality than many state-of-the-art methods by up to 5 dB in terms of image PSNR, across various image categories including portraits, cars, cats, pedestrians and household objects.

This paper proposes a method to approximate the camera spectral sensitivity functions from a single colour image of a colour chart under an unknown illumination spectrum. Here we assume that the scene illumination has a smooth spectral variation. Although the original problem is rather ill-posed, we reformulate it as a well-posed optimisation one by introducing several constraints. The first constraint concerns with the smoothness of the illuminant power spectrum. The second one is based on the fact that the spectral sensitivity function of a digital camera should be constrained to a linear subspace according to Luther condition. The third constraint limits the influence of bands with low signal-to-noise ratios. By introducing these constraints, we can solve the problem in a coordinate-descent optimisation manner. We validate our method using data acquired from over 40 commercial camera models and compare with an alternative in the literature. We also demonstrate the utility of the estimated colour matching functions for colour simulation and colour transfer.

In this paper, we present an unsupervised method for segmenting the illuminant regions and estimating the illumination power spectrum from a single image of a scene lit by multiple light sources. Here, illuminant region segmentation is cast as a probabilistic clustering problem in the image spectral radiance space. We formulate the problem in an optimization setting, which aims to maximize the likelihood of the image radiance with respect to a mixture model while enforcing a spatial smoothness constraint on the illuminant spectrum. We initialize the sample pixel set under each illuminant via a projection of the image radiance spectra onto a low-dimensional subspace spanned by a randomly chosen subset of spectra. Subsequently, we optimize the objective function in a coordinate-ascent manner by updating the weights of the mixture components, sample pixel set under each illuminant, and illuminant posterior probabilities. We then estimate the illuminant power spectrum per pixel making use of these posterior probabilities. We compare our method with a number of alternatives for the tasks of illumination region segmentation, illumination color estimation, and color correction. Our experiments show the effectiveness of our method as applied to one hyperspectral and three trichromatic image data sets.

In this paper, we present a method which permits the creation of user colour preferences for object materials and lights in the scene making use of imaging spectroscopy data. To do this, we build upon the heterogeneous nature of the scene by imposing consistency over object materials so as to allow for small compositional variations across objects in the image. Once the consistency has been imposed, we aim at maximising the quality of the images under consideration based upon user input. This provides the flexibility necessary to utilise user profiles for the automatic processing of real world imagery while avoiding undesirable effects encountered when colour images are produced. We provide results on real-world imagery and illustrate how the method can be used to produce material-specific colours based upon user input.

In this paper, we propose an approach to the problem of simultaneous shape and refractive index recovery from multispectral polarisation imagery captured from a single viewpoint. The focus of this paper is on dielectric surfaces which diffusely polarise light transmitted from the dielectric body into the air. The diffuse polarisation of the reflection process is modelled using a Transmitted Radiance Sinusoid curve and the Fresnel transmission theory. We provide a method of estimating the azimuth angle of surface normals from the spectral variation of the phase of polarisation. Moreover, to render the problem of simultaneous estimation of surface orientation and index of refraction well-posed, we enforce a generative model on the material dispersion equations for the index of refraction. This generative model, together with the Fresnel transmission ratio, permit the recovery of the index of refraction and the zenith angle simultaneously. We show results on shape recovery and rendering for real world and synthetic imagery.

In this paper, we present a system for automatic spectral signature acquisition and recognition of skin from hyperspectral face imagery. In the acquisition step, hyperspectral cameras are used to capture multispectral or hyperspectral images of faces for skin recognition. The acquired signature may either be stored in a database for future testing or be used for purposes of identification. In the recognition step, the system accounts for variations in the illumination by recovering the light power spectrum in the scene and obtains the scene reflectance by normalising the input image radiance accordingly. Furthermore, incorporated into this system is a Non-Uniform Rational B-Spline (NURBS) compact descriptor of spectral reflectance for recognition purposes. We have employed this system as a profiling tool to classify a real-world multispectral face image database into separate ethnic groups.

In this paper, we present an approach to robust estimation of shape from single-view multi-spectral polarisation images. The developed technique tackles the problem of recovering the azimuth angle of surface normals robust to image noise and a low degree of polarisation. We note that the linear least-squares estimation results in a considerable phase shift from the ground truth in the presence of noise and weak polarisation in multispectral and hyper spectral imaging. This paper discusses the utility of robust statistics to discount the large error attributed to outliers and noise. Combining this approach with Shape from Shading, we fully recover the surface shape. We demonstrate the effectiveness of the robust estimator compared to the linear least-squares estimator through shape recovery experiments on both synthetic and real images.

In this paper, we identify the constraints under which the generally ill-posed problem of simultaneous recovery of surface shape and its photometric invariants can be rendered tractable. We examine the cases where a single or more images are acquired using different lighting directions with known illuminant power. Given these conditions, we state the constraints upon which the recovery of the surface geometry and its photometric parameters can be estimated. With these constraints, we then show how the recovery process may be formulated as an optimisation algorithm which aims to fit the reflectance models under study to the image reflectance. The approach presented here is general and can be applied to a family of reflectance models that are based on the Fresnel reflection theory. Thus, we provide a theoretical and computational background for recovering shape, material index of refraction and microscopic roughness from multi-spectral images.

In this paper, we present a surface reflectance descriptor based on the control points resulting from the interpolation of Non-Uniform Rational B-Spline (NURBS) curves to multispectral reflectance data. The interpolation is based upon a knot removal scheme in the parameter domain. Thus, we exploit the local support of NURBS so as to recover a compact descriptor robust to noise and local perturbation of the spectra. We demonstrate the utility of our NURBS-based descriptor for material identification. To this end, we perform skin spectra recognition making use of a Support Vector Machine classifier. We also provide results on hyperspectral imagery and elaborate on the preprocessing step for skin segmentation. We compare our results with those obtained using an alternative descriptor.