Diffusion models (DMs) are generative models that learn to synthesize images from Gaussian noise. DMs can be trained to do a variety of tasks such as image generation and image super-resolution. Researchers have made significant improvements in the capability of synthesizing photorealistic images in the past few years. These successes also hasten the need to address the potential misuse of synthesized images. In this paper, we highlighted the effectiveness of Bayer pattern and local statistics in distinguishing digital camera images from DM-generated images. We further hypothesized that local statistics should be used to address the spatial non-stationarity problems in images. We showed that our approach produced promising results for distinguishing real images from synthesized images. This approach is also robust to various perturbations such as image resizing and JPEG compression.

Due to the expensive segmentation annotation cost, cross-modality medical image segmentation aims to leverage annotations from a source modality (e.g. MRI) to learn a model for target modality (e.g. CT). In this paper, we present a novel method to tackle cross-modality medical image segmentation as semi-supervised multi-modal learning with image translation, which learns better feature representations and is more robust to source annotation scarcity. For semi-supervised multi-modal learning, we develop a deep co-training framework. We address the challenges of co-training on divergent labeled and unlabeled data distributions with a theoretical analysis on multi-view adaptation and propose decomposed multi-view adaptation, which shows better performance than a naive adaptation method on concatenated multi-view features. We further formulate inter-view regularization to alleviate overfitting in deep networks, which regularizes deep co-training networks to be compatible with the underlying data distribution. We perform extensive experiments to evaluate our framework. Our framework significantly outperforms state-of-the-art domain adaptation methods on three segmentation datasets, including two public datasets on cross-modality cardiac substructure segmentation and abdominal multi-organ segmentation and one large scale private dataset on cross-modality brain tissue segmentation. Our code is publicly available at https://github.com/zlheui/DCT.

Deep learning has achieved great success in a wide spectrum of multimedia applications such as image classification, natural language processing and multimodal data analysis. Recent years have seen the development of many deep learning frameworks that provide a high-level programming interface for users to design models, conduct training and deploy inference. However, it remains challenging to build an efficient end-to-end multimedia application with most existing frameworks. Specifically, in terms of usability, it is demanding for non-experts to implement deep learning models, obtain the right settings for the entire machine learning pipeline, manage models and datasets, and exploit external data sources all together. Further, in terms of adaptability, elastic computation solutions are much needed as the actual serving workload fluctuates constantly, and scaling the hardware resources to handle the fluctuating workload is typically infeasible. To address these challenges, we introduce SINGA-Easy, a new deep learning framework that provides distributed hyper-parameter tuning at the training stage, dynamic computational cost control at the inference stage, and intuitive user interactions with multimedia contents facilitated by model explanation. Our experiments on the training and deployment of multi-modality data analysis applications show that the framework is both usable and adaptable to dynamic inference loads. We implement SINGA-Easy on top of Apache SINGA and demonstrate our system with the entire machine learning life cycle.

Big data analytics is gaining massive momentum in the last few years. Applying machine learning models to big data has become an implicit requirement or an expectation for most analysis tasks, especially on high-stakes applications. Typical applications include sentiment analysis against reviews for analyzing on-line products, image classification in food logging applications for monitoring user's daily intake, and stock movement prediction. Extending traditional database systems to support the above analysis is intriguing but challenging. First, it is almost impossible to implement all machine learning models in the database engines. Second, expert knowledge is required to optimize the training and inference procedures in terms of efficiency and effectiveness, which imposes heavy burden on the system users. In this paper, we develop and present a system, called Rafiki, to provide the training and inference service of machine learning models. Rafiki provides distributed hyper-parameter tuning for the training service, and online ensemble modeling for the inference service which trades off between latency and accuracy. Experimental results confirm the efficiency, effectiveness, scalability and usability of Rafiki.

We investigated the problem of deducing the geographical coordinates of pixels in an oblique view video. Our goal is to register the oblique-view video of an urban scene with its cadastral map. The oblique-view videos were taken from a very low flying camera whereas the cadastral map contained only the top-down outline of buildings without any photographic content and without other objects such as trees, cars, people, or any street feature. Our registration comprises a two-step process that uses structure from motion and a matched-filter class of technique. The structure from motion step takes the video as input and outputs a 3D point cloud of the scene. As this point cloud contains objects that are not represented in the cadastral map, our algorithm was designed to emphasize automatically the scene points that are likely to be from building façade so that effects of mismatch in content between the cadastral map and the oblique video can be minimized. For the registration step, we used a coarse-to-fine iterative implementation of matched filter to get a globally optimum solution, with built in tolerance to some scale and rotation invariance. We had implemented the entire system and had tested with real world data. Good results were obtained.

Removal of rain streaks in video is a challenging problem due to the random spatial distribution and fast motion of rain. This paper presents a new rain removal algorithm that incorporates both temporal and chromatic properties of rain in video. The temporal property states that an image pixel is never always covered by rain throughout the entire video. The chromatic property states that the changes of R, G, and B values of rainaffected pixels are approximately the same. By using both properties, the algorithm can detect and remove rain streaks in both stationary and dynamic scenes taken by stationary cameras. To handle videos taken by moving cameras, the video can be stabilized for rain removal, and destabilized to restore camera motion after rain removal. It can handle both light rain and heavy rain conditions. Experimental results show that the algorithm performs better than existing algorithms.

In this paper, we present a robust face tracking system for video indexing and retrieval. Our face tracker is designed based on the condensation algorithm. The strength of our face tracking is in the incorporation of Lucas-Kanade feature tracker in the measurement stage of condensation. Skin color and facial feature points are used for tracking. The pros and cons of using color and facial feature points complement each other and ensure the effectiveness of our system. We also adopt a bi-directional tracking approach to enhance the robustness. We demonstrate the efficacy of our technique in the challenging task of tracking faces in various video sources.