We identify two major limitations in the existing studies on retinal vessel segmentation: 1) Most existing works are restricted to one modality, i.e., the Color Fundus (CF). However, multi-modality retinal images are used every day in the study of the retina and diagnosis of retinal diseases, and the study of vessel segmentation on other modalities is scarce; 2) Even though a few works extended their experiments to new modalities such as the Multi-Color Scanning Laser Ophthalmoscopy (MC), these works still require fine-tuning a separate model for the new modality. The fine-tuning will require extra training data, which is difficult to acquire. In this work, we present a novel universal vessel segmentation model (URVSM) for multi-modality retinal images. In addition to performing the study on a much wider range of image modalities, we also propose a universal model to segment the vessels in all these commonly used modalities. While being much more versatile compared with existing methods, our universal model also demonstrates comparable performance to the state-of-the-art fine-tuned methods. To the best of our knowledge, this is the first work that achieves modality-agnostic retinal vessel segmentation and the first to study retinal vessel segmentation in several novel modalities (Code, model and 3 new retinal vessel segmentation datasets are available at https://github.com/JRC-VPLab/URVSM).

The Ultra-Wide-Field (UWF) retina images have attracted wide attentions in recent years in the study of retina. However, accurate registration between the UWF images and the other types of retina images could be challenging due to the distortion in the peripheral areas of an UWF image, which a 2D warping can not handle. In this paper, we propose a novel 3D distortion correction method which sets up a 3D projection model and optimizes a dense 3D retina mesh to correct the distortion in the UWF image. The corrected UWF image can then be accurately aligned to the target image using 2D alignment methods. The experimental results show that our proposed method outperforms the state-of-the-art method by 30%.

Optical Coherence Tomography (OCT) is a widely used non-invasive high resolution 3D imaging technique for biological tissues and plays an important role in ophthalmology. OCT retinal layer segmentation is a fundamental image processing step for OCT-Angiography projection, and disease analysis. A major problem in retinal imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose neural networks that jointly correct eye motion and retinal layer segmentation utilizing 3D OCT information, so that the segmentation among neighboring B-scans would be consistent. The experimental results show both visual and quantitative improvements by combining motion correction and 3D OCT layer segmentation comparing to conventional and deep-learning based 2D OCT layer segmentation.

The ability to accurately overlay one modality retinal image to another is critical in ophthalmology. Our previous framework achieved the state-of-the-art results for multimodal retinal image registration. However, it requires human-annotated labels due to the supervised approach of the previous work. In this paper, we propose a self-supervised multimodal retina registration method to alleviate the burdens of time and expense to prepare for training data, that is, aiming to automatically register multimodal retinal images without any human annotations. Specially, we focus on registering color fundus images with infrared reflectance and fluorescein angiography images, and compare registration results with several conventional and supervised and unsupervised deep learning methods. From the experimental results, the proposed self-supervised framework achieves a comparable accuracy comparing to the state-of-the-art supervised learning method in terms of registration accuracy and Dice coefficient.

Multi-modal retinal image registration plays an important role in the ophthalmological diagnosis process. The conventional methods lack robustness in aligning multi-modal images of various imaging qualities. Deep-learning methods have not been widely developed for this task, especially for the coarse-to-fine registration pipeline. To handle this task, we propose a two-step method based on deep convolutional networks, including a coarse alignment step and a fine alignment step. In the coarse alignment step, a global registration matrix is estimated by three sequentially connected networks for vessel segmentation, feature detection and description, and outlier rejection, respectively. In the fine alignment step, a deformable registration network is set up to find pixel-wise correspondence between a target image and a coarsely aligned image from the previous step to further improve the alignment accuracy. Particularly, an unsupervised learning framework is proposed to handle the difficulties of inconsistent modalities and lack of labeled training data for the fine alignment step. The proposed framework first changes multi-modal images into a same modality through modality transformers, and then adopts photometric consistency loss and smoothness loss to train the deformable registration network. The experimental results show that the proposed method achieves state-of-the-art results in Dice metrics and is more robust in challenging cases.

Foveal Avascular Zone (FAZ) is a crucial indicator for retinal disease detection and accurate automatic FAZ segmentation has a significant impact in clinical applications. Apart from the binary FAZ segmentation map, a vessel segmentation map can provide further information. To simultaneously implement vessel and accurate FAZ segmentation, an end-to-end trained network is proposed to achieve unsupervised vessel segmentation and supervised FAZ segmentation. Due to the lack of vessel labels, the style transfer with consistency loss is proposed to the vessel segmentation. Then FAZ segmentation is achieved with a U-Net structure based on vessel segmentation. Two superficial layer OCTA image datasets - OCTAGON3 [1] and sFAZDATA datasets [2] - are used to evaluate the proposed method. We achieve the Dice scores of 0.9263 and 0.9784, which are better than those from other approaches.

Optical Coherence Tomography (OCT) is a powerful technique for non-invasive 3D imaging of biological tissues at high resolution that has revolutionized retinal imaging. A major challenge in OCT imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose a convolutional neural network that learns to correct axial motion in OCT based on a single volumetric scan. The proposed method is able to correct large motion, while preserving the overall curvature of the retina. The experimental results show significant improvements in visual quality as well as overall error compared to the conventional methods in both normal and disease cases.

Multimodal retinal imaging plays an important role in ophthalmology. We propose a content-adaptive multimodal retinal image registration method in this paper that focuses on the globally coarse alignment and includes three weakly supervised neural networks for vessel segmentation, feature detection and description, and outlier rejection. We apply the proposed framework to register color fundus images with infrared reflectance and fluorescein angiography images, and compare it with several conventional and deep learning methods. Our proposed framework demonstrates a significant improvement in robustness and accuracy reflected by a higher success rate and Dice coefficient compared with other methods.

Multimodal image registration plays an important role in diagnosing and treating ophthalmologic diseases. In this paper, a deep learning framework for multimodal retinal image registration is proposed. The framework consists of a segmentation network, feature detection and description network, and an outlier rejection network, which focuses only on the globally coarse alignment step using the perspective transformation. We apply the proposed framework to register color fundus images with infrared reflectance images and compare it with the state-of-the-art conventional and learning-based approaches. The proposed framework demonstrates a significant improvement in robustness and accuracy reflected by a higher success rate and Dice coefficient compared to other coarse alignment methods.

In multi-modal retinal image registration task, there are two major challenges, i.e., poor performance in finding correspondence due to inconsistent features, and lack of labeled data for training learning-based models. In this paper, we propose a joint vessel segmentation and deformable registration model based on CNN for this task, built under the framework of weakly supervised style transfer learning and perceptual loss. In vessel segmentation, a style loss guides the model to generate segmentation maps that look authentic, and helps transform images of different modalities into consistent representations. In deformable registration, a content loss helps find dense correspondence for multi-modal images based on their consistent representations, and improves the segmentation results simultaneously. Experiment results show that our model has better performance than other deformable registration methods in both quantitative and visual evaluations, and the segmentation results also help the rigid transformation1.

Blocking and color bleeding are two well-known artifacts for 4:2:0 JPEG-compressed images. Blocking mainly results from the block-level quantization of the luma component, while color bleeding is mainly caused by the subsampling and quantization of chroma components. Restoring luma can reduce blocking distortion, but with little influence on color bleeding. On the contrary, color bleeding can be removed via chroma components restoration. This letter proposes a deep wide-activated residual network for reducing blocking and color bleeding artifacts simultaneously, in which the luma and chroma components are jointly restored. Chroma components usually suffer from more severe distortion than the luma component due to subsampling and coarse quantization. Thus, we use the luma component to guide the restoration of chroma components. Moreover, we reduce blocking and color bleeding artifacts in low-resolution space via pixel shuffle-based decimation and assembling, which allows to obtain high restoration speed. Experimental results show that the proposed approach achieves state-of-the-art performance on joint blocking and color bleeding artifacts reduction.

In this paper, we propose bit allocation between luma samples and chroma samples using chroma quantization parameter (QP) offsets for Cb and Cr. For this work, we propose an efficient adaptive Lagrange multiplier selection method using classification-maximization, and then apply the proposed adaptive Lagrange multiplier selection to decide chroma QP offsets for Cb and Cr. To our knowledge, this is the first proposal to adaptively decide chroma QP offsets. Because the default mapping function between a chroma QP and a luma QP in H.264 is unbalanced at especially low QPs, the proposed chroma QP offset decision achieves improvement up to 0.8 dB from the experimental results.

In this paper, we apply the primal-dual decomposition and subgradient projection methods to solve the rate-distortion optimization problem with the constant bit rate constraint. The primal decomposition method enables spatial or temporal prediction dependency within a group of picture (GOP) to be processed in the master primal problem. As a result, we can apply the dual decomposition to minimize independently the Lagrangian cost of all the MBs using the reference software model of H.264. Furthermore, the optimal Lagrange multiplier lambda* is iteratively derived from the solution of the dual problem. As an example, we derive the optimal bit allocation condition with the consideration of temporal prediction dependency among the pictures. Experimental results show that the proposed method achieves better performance than the reference software model of H.264 with rate control for given bit constraint.

In this paper, we improve the performance of intra prediction and simplify mode decision procedure at the same time. For these works, we apply a statistical learning method such as Support Vector Machines for Regression (SVR) to improve the performance of current H.264 intra prediction via batch learning. In addition, we only use single Macro Block type and one intra prediction mode with high prediction performance to simplify mode decision procedure. In our knowledge, this work is the first approach to apply a statistical learning method for prediction of video sequences. Therefore, we introduce theoretical backgrounds of SVR, and show the possibility of this challenge for video compression. From the experimental results, statistical learning based intra prediction improves significantly the average Peak Signal-to-Noise Ratio of intra prediction than the performance of current H.264.

In this paper, we apply the primal-dual decomposition and subgradient projection methods to solve the rate-distortion optimization problem with the constant bit rate constraint. The primal decomposition method enables spatial or temporal prediction dependency within a group of picture (GOP) to be processed in the master primal problem. As a result, we can apply the dual decomposition to minimize independently the Lagrangian cost of all the MBs using the reference software model of H.264. Furthermore, the optimal Lagrange multiplier lambda* is iteratively derived from the solution of the dual problem. As an example, we derive the optimal bit allocation condition with the consideration of temporal prediction dependency among the pictures. Experimental results show that the proposed method achieves better performance than the reference software model of H.264 with rate control.

The number of slices for error resilient video coding is jointly optimized with 802.11a-like media access control and the physical layers with automatic repeat request and rate compatible punctured convolutional code over additive white gaussian noise channel as well as channel times allocation for time division multiple access. For error resilient video coding, the relation between the number of slices and coding efficiency is analyzed and formulated as a mathematical model. It is applied for the joint optimization problem, and the problem is solved by a convex optimization method such as the primal-dual decomposition method. We compare the performance of a video communication system which uses the optimal number of slices with one that codes a picture as one slice. From numerical examples, end-to-end distortion of utility functions can be significantly reduced with the optimal slices of a picture especially at low signal-to-noise ratio.

Cross-layer optimization approach is applied to allocate channel times of time-division multiple access (TDMA) to utility functions which send different video streams with different rate-distortion characteristics. Given a channel time, each utility function solves an optimization problem to obtain the optimal source code rate, channel code rate and media access control (MAC) frame size. In this paper, we derive mathematical models to represent end-to-end distortion of video streams and 802.11a-like MAC and PHY with automatic repeat reQuest (ARQ) and rate compatible punctured convolutional code (RCPC) over additive white Gaussian noise (AWGN) channel. These models are formulated as a convex optimization problem, and it is solved by the primal-dual decomposition method. In addition, coexistence among the proposed utility functions and conventional utility functions is discussed. Finally, numerical examples show that significant reduction of overall distortion of utility functions can be achieved.

In this paper, cross-layer optimization approach is applied from video coding to the physical layer through media access control (MAC) layer. We mainly focus on resource allocation of source code rate and channel code rate with optimal MAC frame length. Therefore, elaborate mathematical models are derived to represent end-to-end distortion of video streams and 802.1 la-like MAC and PHY with automatic repeat request (ARQ) and rate compatible punctured convolutional code (RCPC) over additive white Gaussian noise (AWGN) channel. These models are formulated as a convex optimization problem, and it is solved by the dual decomposition method. From a numerical example, significant reduction of overall distortion can be achieved.

In this paper, we formulate the utility functions of distortion and peak signal-to-noise ratio (PSNR) which are generally used for the performance evaluation of video coding applications as the utility functions. The convexity property of these utility functions is analyzed in the original domain and transformed domain of an optimization variable. From this analysis, we derive joint optimization scheme with congestion control through the utility matching between TCP layer and video coding layer. Experimental results show that the overall PSNR increases and the variation of quality among the utility functions is reduced.