Atmospheric Turbulence (AT) degrades the clarity and accuracy of surveillance imagery, posing challenges not only for visualization quality but also for object classification and scene tracking. Deep learning-based methods have been proposed to improve visual quality, but spatio-temporal distortions remain a significant issue. Although deep learning-based object detection performs well under normal conditions, it struggles to operate effectively on sequences distorted by atmospheric turbulence. In this paper, we propose a novel framework that learns to compensate for distorted features while simultaneously improving visualization and object detection. This end-to-end training strategy leverages and exchanges knowledge of low-level distorted features in the AT mitigator with semantic features extracted in the object detector. Specifically, in the AT mitigator a 3D Mamba-based structure is used to handle the spatio-temporal displacements and blurring caused by turbulence. Optimization is achieved through back-propagation in both the AT mitigator and object detector. Our proposed DMAT outperforms state-of-the-art AT mitigation and object detection systems up to a 15% improvement on datasets corrupted by generated turbulence. The code is available at https://github.com/pui-nantheera/DMAT and datasets are available at https://zenodo.org/records/17673509.

Restoration and enhancement are essential for improving the quality of videos captured under atmospheric turbulence conditions, aiding visualization, object detection, classification, and tracking in surveillance systems. In this paper, we introduce a novel Mamba-Based method, the 3D Mamba-Based Atmospheric Turbulence Removal (MAMAT), which employs a dual-module strategy to mitigate these distortions. The first module utilizes deformable 3D convolutions for non-rigid registration to minimize spatial shifts, while the second module enhances contrast and detail. Leveraging the advanced capabilities of the 3D Mamba architecture, experimental results demonstrate that MAMAT outperforms state-of-the-art learning-based methods, achieving up to a 3% improvement in visual quality and a 15% boost in object detection. It not only enhances visualization but also significantly improves object detection accuracy, bridging the gap between visual restoration and the effectiveness of surveillance applications. The code is available at https://github.com/csprh/MAMAT.

A significant number of researchers have applied deep learning methods to image fusion. However, most works require a large amount of training data or depend on pre-trained models or frameworks to capture features from source images. This is inevitably hampered by a shortage of training data or a mismatch between the framework and the actual problem. Deep Image Prior (DIP) has been introduced to exploit convolutional neural networks’ ability to synthesize the ‘prior’ in the input image. However, the original design of DIP is hard to be generalized to multi-image processing problems, particularly for image fusion. Therefore, we propose a new image fusion technique that extends DIP to fusion tasks formulated as inverse problems. Additionally, we apply a multichannel approach to enhance DIP’s effect further. The evaluation is conducted with several commonly used image fusion assessment metrics. The results are compared with state-of-the-art image fusion methods. Our method outperforms these techniques for a range of metrics. In particular, it is shown to provide the best objective results for most metrics when applied to medical images.

A novel improvement to an existing wavelet representation is presented within the context of analysis and synthesis of harmonic audio signals. The approach replaces the discrete wavelet transform stage in the harmonic-band wavelet transform with the dual-tree complex wavelet transform. The harmonic-band wavelet transform lacks explicit phase information, is subject to considerable transform domain coefficient aliasing and its filters exhibit poor frequency selectivity and high sidelobe levels. Integration of the dual-tree complex wavelet transform mitigates all of these disadvantages and significantly improves the accuracy of signal synthesis where signals are synthesised through autoregressive analysis and linear prediction of transform domain coefficients.

This paper describes the application of belief propagation methods to image fusion within a complex wavelet decomposition (the Dual Tree Complex Wavelet Transform: DT-CWT). Belief propagation within each transform subband iterates through a lattice based Bayesian belief network. This leads to precisely controlled spatial coherence of subband coefficient fusion through the definition of belief graph probabilities. This results in a significant improvement in quantitatively measured fusion performance for a large database of over 160 fusion image pairs from a range of fusion applications including remote sensing, multi-focus and multi-modal sources. Improvements in qualitative image fusion performance is also demonstrated.

A perceptual image fusion method is proposed that employs explicit luminance and contrast masking models. These models are combined to give the perceptual importance of each coefficient produced by the dual-tree complex wavelet transform of each input image. This combined model of perceptual importance is used to select which coefficients are retained and furthermore to determine how to present the retained information in the most effective way. This paper is the first to give a principled approach to image fusion from a perceptual perspective. Furthermore, the proposed method is shown to give improved quantitative and qualitative results compared with previously developed methods.

A novel adaptation of the two dimensional Homomorphic filter is introduced using the Dual Tree Complex Wavelet Transform for improved illumination invariant processing. The Homomorphic filter is conventionally implemented within the log-Fourier domain using an isotropic high-pass filter based on the assumption that the illumination signal occupies low spatial frequencies. In this case however, low frequency structural reflectance content will be incorrectly attenuated. Our method implements the Homomorphic filter using the DT-CWT and exploits the property of cross scale persistence (for structural content) to generate a filter that retains cross scale content and therefore reduces incorrect attenuation of structural reflectance content.

Approximate Message Passing (AMP) is an iterative reconstruction algorithm that performs signal denoising within a compressive sensing framework. We propose the use of heavy tailed distribution based image denoising, specifically using a Cauchy prior based Maximum A-Posteriori (MAP) estimate within a wavelet based AMP compressive sensing structure. The use of this MAP denoising algorithm provides extremely fast convergence for image based compressive sensing. The proposed method converges approximately twice as fast as the compared AMP methods whilst providing superior final MSE results over a range of measurement rates.

Accurate estimation of the contrast sensitivity of the human visual system is crucial for perceptually based image processing in applications such as compression, fusion and denoising. Conventional contrast sensitivity functions (CSFs) have been obtained using fixed-sized Gabor functions. However, the basis functions of multiresolution decompositions such as wavelets often resemble Gabor functions but are of variable size and shape. Therefore to use the conventional CSFs in such cases is not appropriate. We have therefore conducted a set of psychophysical tests in order to obtain the CSF for a range of multiresolution transforms: the discrete wavelet transform, the steerable pyramid, the dual-tree complex wavelet transform, and the curvelet transform. These measures were obtained using contrast variation of each transforms' basis functions in a 2AFC experiment combined with an adapted version of the QUEST psychometric function method. The results enable future image processing applications that exploit these transforms such as signal fusion, superresolution processing, denoising and motion estimation, to be perceptually optimized in a principled fashion. The results are compared with an existing vision model (HDR-VDP2) and are used to show quantitative improvements within a denoising application compared with using conventional CSF values.

A novel system is introduced that is able to fuse two or more sets of multimodal videos in the transform domain. This is achieved without drift and produces an embedded bitstream that offers fine grain scalability. Previous attempts to fuse in the transform domain have not been possible for video compression systems due to the complications of predictive loops within conventional video encoding. The compression system is based on an optimised spatiotemporal codec using the 3D Discrete Dual-tree Wavelet Transform (DDWT) together with a bit plane encoding method (SPIHT) and a coefficient sparsification process (noise shaping). Together, these methods can efficiently encode a video sequence without the need for motion compensation due to the directional (in space and time) selectivity of the transform. This system offers extremely flexible video fusion in dynamic bandwidth environments where there are variable client receiving capabilities.

Wavelet shrinkage is a standard technique for denoising natural images. Originally proposed for univariate shrinkage in the Discrete Wavelet Transform (DWT) domain, it has since been optimised through the exploitation of translationally invariant wavelet decompositions such as the Dual-Tree Complex Wavelet Transform (DT-CWT) alongside bivariate analysis techniques that condition the shrinkage on spatially related coefficients across neighbouring scales. These more recent techniques have denoised the real and imaginary components of the DT-CWT coefficients separately. Processing real and imaginary components separately has been found to lead to an increase in the phase noise of the transform which in turn affects denoising performance. On this basis, the work presented in this paper offers improved denoising performance through modelling the bivariate distribution of the coefficient magnitudes. The results were compared to the current state of the art non-local means denoising technique BM3D, showing clear subjective improvements, through the retention of high frequency structural and textural information. The paper also compares objective measures, using both PSNR and the more perceptually valid structural similarity measure (SSIM). Whereas PSNR results were slightly below those for BM3D, those for SSIM showed closer correlation with subjective assessment, indicating improvements over BM3D for most noise levels on the images tested.

The Undecimated Dual Tree Complex Wavelet Transform (UDTCWT) is introduced together with its application to image denoising. The UDT-CWT extends the traditional DT-CWT using the methods of filter upsampling and the removal of downsampling developed for the Undecimated Discrete Wavelet Transform (UDWT). The UDTCWT results in a one-to-one relationship between co-located complex coefficients in all subbands and offers improved lower scale subband localisation together with improved directional selectivity (compared to the UDWT). These properties of the UDT-CWT have been exploited in the presented bivariate shrinkage denoising algorithm and gives quantitative improvements in the application to the denoising of images.

Modern video codecs such as MPEG2, MPEG4-ASP and H.264 depend on computationally complex sub-pixel motion estimation (ME) to optimise rate-distortion efficiency. Sub-pixel ME is implemented within these standards using interpolated values at 1/2 or 1/4 pixel accuracy. This paper presents a novel scheme for sub-pixel ME using a model free kernel method utilising the result of the whole-pixel SAD distribution applied to an H.264 encoder. While giving an equivalent rate distortion performance, this approach approximately halves the number of quarter-pixel search positions giving an overall speed up of approximately 10% compared to the EPZS quarter-pixel method (the state of the art H.264 optimised sub-pixel motion estimator).

An unsupervised image compression-by-synthesis system is proposed utilising wavelet based image segmentation and analysis combined with patch based texture synthesis. High perceptual quality is ensured using an artefact detection algorithm in the encoder loop. EBCOT is used to transform code texture samples and residual image data. Resulting bitrate savings of up to approximately 17% over JPEG2000 for little change in perceptual quality have been shown.

An image compression scheme is proposed, utilising wavelet- based image segmentation and texture analysis, and patch- based texture synthesis. This has been incorporated into a JPEG framework. Homogeneous textured regions are identified and removed prior to transform coding. These regions are then replaced at the decoder by synthesis from marked samples, and colour matched to ensure similarity to the original. Experimental results on natural images show bitrate savings of over 18% compared with JPEG for little change in measured visual quality.

Sub-pixel motion compensation plays an important role in compression efficiency within modern video codecs such as MPEG2, MPEG4 and H.264. Sub-pixel motion compensation is implemented within these standards using interpolated pixel values at 1/2 or 1/4 pixel accuracy. Such interpolation gives a good reduction in residual energy for each predicted macroblock and therefore improves compression. However, such interpolation is very computationally complex for the encoder. This is especially true for H.264 where the cost of an exhaustive set of macroblock segmentations need to be estimated in order to obtain an optimal mode for prediction. This paper presents a novel interpolation-free scheme for sub-pixel motion compensation using the result of the full pixel SAD distribution of each motion compensated block applied to an H.264 encoder. This system produces reduced complexity motion compensation with a controllable trade-off between compression performance and encoder speed. These methods facilitate the generation of a real time software H.264 encode.

Subpixel motion estimation plays an important role in compression efficiency within modern video codecs such as MPEG2, MPEG4, and H.264. Subpixel motion estimation is implemented within these standards using interpolated values at 1/2 or 1/4 pixel accuracy. Such interpolation gives a good reduction in residual energy for each predicted macroblock and, therefore, improves compression. However, this leads to a significant increase in computational complexity at the encoder. This is especially true for H.264 where the cost of an exhaustive set of macroblock segmentations need to be estimated in order to obtain an optimal mode for prediction. This paper presents a novel interpolation-free scheme for subpixel motion estimation using the result of the full pixel sum of absolute difference distribution of each motion compensated block applied to an H.264 encoder. This system produces reduced complexity motion estimation with a controllable tradeoff between compression performance and encoder speed. These methods facilitate the generation of a real time software H.264 encoder

Image fusion is the process of extracting meaningful visual information from two or more images and combining them to form one fused image. Image fusion is important within many different image processing fields from remote sensing to medical applications. Previously, real valued wavelet transforms have been used for image fusion. Although this technique has provided improvements over more naive methods, this transform suffers from the shift variance and lack of directionality associated with its wavelet bases. These problems have been overcome by the use of a reversible and discrete complex wavelet transform (the dual tree complex wavelet transform DT-CWT). However, the existing structure of this complex wavelet decomposition enforces a very strict choice of filters in order to achieve a necessary quarter shift in coefficient output. This paper therefore introduces an alternative structure to the DT-CWT that is more flexible in its potential choice of filters and can be implemented by the combination of four normally structured wavelet transforms. The use of these more common wavelet transforms enables this method to make use of existing optimised wavelet decomposition and recomposition methods, code and filter choice.

The segmentation of images into meaningful and homogenous regions is a key method for image analysis within applications such as content based retrieval. The watershed transform is a well established tool for the segmentation of images. However, watershed segmentation is often not effective for textured image regions that are perceptually homogeneous. In order to segment such regions properly, the concept of the "texture gradient" is introduced. Texture information and its gradient are extracted using a novel nondecimated form of a complex wavelet transform. A novel marker location algorithm is subsequently used to locate significant homogeneous textured or non textured regions. A marker driven watershed transform is then used to segment the identified regions properly. The combined algorithm produces effective texture and intensity based segmentation for application to content based image retrieval.

The fusion of images is the process of combining two or more images into a single image retaining important features from each. Fusion is an important technique within many disparate fields such as remote sensing, robotics and medical applications. Wavelet based fusion techniques have been reasonably effective in combining perceptually important image features. Shift invariance of the wavelet transform is important in ensuring robust subband fusion. Therefore, the novel application of the shift invariant and directionally selective Dual Tree Complex Wavelet Transform (DT-CWT) to image fusion is now introduced. This novel technique provides improved qualitative and quantitative results compared to previous wavelet fusion methods. 1

The watershed transform is a well established tool for the segmentation of images. However, watershed segmentation is often not effective for textured regions that are perceptually homogeneous. Such regions are usually inaccurately over-segmented with no reference to any texture changes. We now introduce a novel concept of “texture gradient” implemented using a non-decimated complex wavelet transform. A novel marker location algorithm is subsequently used to locate significant homogeneous textured or non textured regions. A marker driven watershed transform is then used to properly segment the identified regions. The combined algorithm produces effective texture and intensity based segmentation for the application to content based retrieval of images.

Simple wavelet and wavelet packet transforms have often been used for texture characterisation through the analysis of spatial-frequency content. However, most previous methods make no use of any statistical analysis of the transforms' subbands. A novel method is now presented for modelling the multivariate distributions of subband coefficients by considering spatially related coefficients. The Bhattacharya and divergence metrics are then used to produce an improved texture classification method for the application to content based image retrieval.

Content-based retrieval is ultimately dependent on the features used for the annotation of data and its efficiency is dependent on the invariance and robust properties of these features. For texture-based features an important form of invariance is rotational invariance. Novel rotationally invariant texture-based features are introduced that are extracted from a polar Fourier transform (PFT). The PFT is similar to the discrete Fourier transform in two dimensions but uses transform parameters radius and angle rather than the Cartesian co-ordinates. The PFT is discretised appropriately across the angular and radial frequency space with the transform magnitudes forming the rotationally invariant features. These features although rotationally invariant, capture the angular distribution together with the radial distribution of frequency within texture. Preliminary results show the method to give better results than rotationally variant and invariant Gabor filter schemes.

New rotationally invariant texture feature extraction methods are introduced that utilise the dual-tree complex wavelet transform (DT-CWT). The complex wavelet transform is a new technique that uses a dual tree of wavelet filters to obtain the real and imaginary parts of complex wavelet coefficients. When applied in two dimensions the DT-CWT produces shift invariant orientated subbands. Both isotropic and anisotropic rotationally invariant features can be extracted from the energies of these subbands. Using simple minimum distance classifiers, the classification performance of the proposed feature extraction methods were tested with rotated sample textures. The anisotropic features gave the best classification results for the rotated texture tests, outperforming a similar method using a real wavelet decomposition.