Active learning methods recommend the most informative images from a large unlabeled dataset for manual labeling. These methods improve the performance of an image classifier while minimizing manual labeling efforts. We propose VisActive, a visual-concept-based active learning method for image classification under class imbalance. VisActive learns a visual concept, a generalized representation that holds the most important image characteristics for class prediction, and then recommends for each class four sets of unlabeled images with different visual concepts to increase the diversity and enlarge the training dataset. Experimental results on four datasets show that VisActive outperforms the state-of-the-art deep active learning methods.

A major problem of training deep learning image classifiers is the limited availability of domain-specific labeled datasets. This problem is particularly pressing in medicine due to the cost and expertise needed for annotation. This paper proposes a training strategy to reduce model misclassification by 1) using weakly supervised learning to learn class confusion, 2) creating a new class with synthetic training data highlighting the confusion, and 3) training with the expanded training data and using transfer learning. We tested our new training strategy using the open medical (Kvasir) and non-medical (CIFAR10) datasets. The proposed training strategy improves classification accuracy, precision, and recall when applied to a small subset of the training data.

During the past decades, many automated image analysis methods have been developed for colonoscopy. Real-time implementation of the most promising methods during colonoscopy has been tested in clinical trials, including several recent multi-center studies. All trials have shown results that may contribute to prevention of colorectal cancer. We summarize the past and present development of colonoscopy video analysis methods, focusing on two categories of artificial intelligence (AI) technologies used in clinical trials. These are (1) analysis and feedback for improving colonoscopy quality and (2) detection of abnormalities. Our survey includes methods that use traditional machine learning algorithms on carefully designed hand-crafted features as well as recent deep-learning methods. Lastly, we present the gap between current state-of-the-art technology and desirable clinical features and conclude with future directions of endoscopic AI technology development that will bridge the current gap.

Most state-of-the-art local interpretation methods explain the behavior of deep learning classification models by assigning importance scores to image pixels based on how influential each pixel was towards the final decision. These interpretations are unable to provide further details to aid understanding of a complex concept in a domain such as medicine. We propose a novel Hierarchical Visual Concept (HVC) interpretation framework for CNN-based image classification models. As an explanation of the classification decision of a given image, HVC presents a concept hierarchy of most relevant visual concepts at multiple semantic levels. These concepts are automatically learned during training such that the lower-level concepts in the hierarchy support the corresponding higher-level concepts. Our quantitative and qualitative evaluation of the interpretation of VGG16 and ResNet50 classifiers on public and private colonoscopy image datasets shows very promising results.

We report the technical challenges, solutions, and lessons learned from deploying real-time feedback systems in three hospitals as part of a multi-center controlled clinical trial to improve quality of colonoscopy. Previous clinical trials were conducted in one center. The technical challenges for our multicenter clinical trial include 1) reducing additional work by the endoscopists to utilize real-time feedback, 2) handling different colonoscopy practices at different hospitals, and 3) training an effective CNN-based classification model with a large variety of patterns of data in day-to-day colonoscopy practice. We report performance of our real-time systems over a period of 20 weeks at each hospital. We conclude that CNN-based classification can achieve very good performance in real-world deployment when trained with high quality data.

Many image classification tasks (e.g., medical image classification) have a severe class imbalance problem. Convolutional neural network (CNN) is currently a state-of-the-art method for image classification. CNN relies on a large training dataset to achieve high classification performance. However, manual labeling is costly and may not even be feasible for medical domain. In this paper, we propose a novel similarity-based active deep learning framework (SAL) that deals with class imbalance. SAL actively learns a similarity model to recommend unlabeled rare class samples for experts' manual labeling. Based on similarity ranking, SAL recommends high confidence unlabeled common class samples for automatic pseudo-labeling without experts' labeling effort. To the best of our knowledge, SAL is the first active deep learning framework that deals with a significant class imbalance. Our experiments show that SAL consistently outperforms two other recent active deep learning methods on two challenging datasets. What's more, SAL obtains nearly the upper bound classification performance (using all the images in the training dataset) while the domain experts labeled only 5.6% and 7.5% of all images in the Endoscopy dataset and the Caltech-256 dataset, respectively. SAL significantly reduces the experts' manual labeling efforts while achieving near optimal classification performance.

We describe a new and effective real-time solution for detecting video segments showing an instrument used during diagnostic or therapeutic operations in endoscopic procedures. In addition, we present a new method to collect a large training dataset: similarity-based data augmentation. This method automates most of the creation of a large training dataset and prevents extensive manual effort to collect and annotate training data by domain experts. Convolutional Neural Network (CNN) analysis using the training data collected with similarity-based data augmentation yields an average F1 score within 1% of that of the CNN analysis using a large manually collected training dataset.

This paper presents a novel technique for automated detection of protruding polyps in colonoscopy images using edge cross-section profiles (ECSP). We propose a part-based multiderivative ECSP that computes derivative functions of an edge cross-section profile and segments each of these profiles into parts. Therefore, we can model or extract features suitable for each part. Our features obtained from the parts can effectively describe complex properties of protruding polyps including the shape of the parts, texture, and protrusion and smoothness of the polyp surface. We evaluated our method against two existing polyp image detection techniques on 42 different polyps, including those with little protrusion. Each polyp has a large variation of appearance in viewing angles, light conditions, and scales in different images. The evaluation showed that our technique outperformed the existing techniques in both accuracy and analysis time. Our method has a higher area under the free-response receiver operating characteristic curve. For instance, when both techniques have a true positive rate for polyp image detection of 81.4%, the average number of false regions per image of our technique is 0.32 compared to 1.8 of the best existing technique under study. Additionally, our technique can precisely mark edges of candidate polyp regions as visual feedback. These results altogether indicate that our technique is promising to provide visual feedback of polyp regions in clinical practice.

Colonoscopy is the most popular screening tool for colorectal cancer. Recent studies reported that retroflexion during colonoscopy helped to detect more polyps. Retroflexion is an endoscope maneuver that enables visualization of internal mucosa along the shaft of the endoscope, enabling visualization of the mucosa area that is difficult to see with typical forward viewing. This paper describes our new method that detects the retroflexion during colonoscopy. We propose region shape and location (RSL) features and edgeless edge cross-section profile (ECSP) features that encapsulate important properties of endoscope appearance and edge information during retroflexion. Our experimental results on 50 colonoscopy test videos show that a simple ensemble classifier using both ECSP and RSL features can effectively identify retroflexion in terms of analysis time and detection rate.

This paper presents SAPPHIRE — a novel middleware and software development kit, developed to reduce implementation efforts in utilizing task parallelism to speed up execution time of analysis of a stream of data, such as medical video. As a case study, we implemented a real-time quality measurement of colonoscopy using SAPPHIRE. We increased the number of threads in our case study from 4 (prior to our middleware) to 28 with ease. As a result, through better load-balancing and taking better advantage of available processor cores, we increased the case study's maximum processing rate from 40 frames per second to 90 frames per second.

Colonoscopy is the most popular screening tool for colorectal cancer. Recent studies reported that retroflexion during colonoscopy improved polyp yields. Retroflexion is an endoscope maneuver that enables visualization of internal mucosa along the shaft of the endoscope, enabling visualization of the mucosa area that is difficult to see with typical forward viewing. This paper describes our new method that detects endoscopic images showing retroflexion. This problem has not been investigated in the literature. We propose new region features that encapsulate important properties of endoscope appearance during retroflexion. Our experimental results on 25 colonoscopy videos show that trained Decision Tree classifiers can effectively identify retroflexion in the rectum at 92.0% accuracy and 94.4% precision.

A new algorithm for reconstruction of a 3D virtual colon segment from an individual image captured from colonoscopy is presented. Colonoscopy is currently the gold standard method for detection and prevention of colorectal cancer. However, the protective effect depends on the amount of colon mucosa that is actually seen by the endoscopist. The proposed algorithm takes contours of colon folds in the image as input and calculates the depth and the slant angle of each fold. Finally, the colon mucosa is created using Cubic Bezier Curve interpolation between the folds. The proposed algorithm is an important step toward 3D reconstruction of the virtual colon from video of an entire colonoscopy procedure. The reconstruction is potentially useful to estimate percent of colon mucosa inspected by the endoscopist.

Early detection of polyps and cancers is one of the most important goals of colonoscopy. Computer-based analysis of video files using texture features, as has been proposed for polyps of the stomach and colon, has two major limitations: this method uses a fixed size analysis window and relies heavily on a training set of images for accuracy. To overcome these limitations, we propose a new technique focusing on shape instead of texture in this paper. The proposed polyp region detection method is based on the elliptical shape that is common for nearly all small colon polyps.

We describe a gradient vector flow driven active shape method for model-based image segmentation. Active shape algorithm retain the shape feature of the interested object, and its performance relies heavily on initialization. Because of a lack of global regulation, the control points tends to be trapped in a local optimum in searching. Our proposed method uses the gradient vector flow of an image to guide the optimization process. The control points of an active shape are steered by the direction and the magnitude of gradient vectors. Our experiments demonstrated great improvement in finding the global optimum and resulting correct segmentation.

We propose a new graph-based data structure, called Spatio Temporal Region Graph (STRG) which can represent the content of video sequence. Unlike existing ones which consider mainly spatial information in the frame level of video, the proposed STRG is able to formulate its temporal information in the video level additionally. After an STRG is constructed from a given video sequence, it is decomposed into its subgraphs called Object Graphs (OGs), which represent the temporal characteristics of video objects. For unsupervised learning, we cluster similar OGs into a group, in which we need to match two OGs. For this graph matching, we introduce a new distance measure, called Extended Graph Edit Distance (EGED), which can handle the temporal characteristics of OGs. For actual clustering, we exploit Expectation Maximization (EM) with EGED. The experiments have been conducted on real video streams, and their results show the effectiveness and robustness of the proposed schemes.

Colonoscopy is the accepted screening method for detection of colorectal cancer or its precursor lesions, colorectal polyps. Indeed, colonoscopy has contributed to a decline in the number of colorectal cancer related deaths. However, not all cancers or large polyps are detected at the time of colonoscopy, and methods to investigate why this occurs are needed. We present a new computer-based method that allows automated measurement of a number of metrics that likely reflect the quality of the colonoscopic procedure. The method is based on analysis of a digitized video file created during colonoscopy, and produces information regarding insertion time, withdrawal time, images at the time of maximal intubation, the time and ratio of clear versus blurred or non-informative images, and a first estimate of effort performed by the endoscopist. As these metrics can be obtained automatically, our method allows future quality control in the day-to-day medical practice setting on a large scale. In addition, our method can be adapted to other healthcare procedures. Last but not least, our method may be useful to assess progress during colonoscopy training, or as part of endoscopic skills assessment evaluations.

In this paper, we present scenario based dynamic video abstractions using graph matching. Our approach has two main components: multi-level scenario generations and dynamic video abstractions. Multi-level scenarios are generated by a graph-based video segmentation and a hierarchy of the segments. Dynamic video abstractions are accomplished by accessing the generated hierarchy level by level. The first step in the proposed approach is to segment a video into shots using Region Adjacency Graph (RAG). A RAG expresses spatial relationships among segmented regions of a frame. To measure the similarity between two consecutive RAGs, we propose a new similarity measure, called Graph Similarity Measure (GSM). Next, we construct a tree structure called scene tree based on the correlation between the detected shots. The correlation is computed by the GSM since it considers the relations between the detected shots properly. Multi-level scenarios which provide various levels of video abstractions are generated using the constructed scene tree. We provide two types of abstraction using multi-level scenarios: multi-level highlights and multi-length summarizations. Multi-level highlights are made by entire shots in each scenario level. To summarize a video in various lengths, we select key frames by considering temporal relationships among RAGs computed by the GSM. We have developed a system, called Automatic Video Analysis System (AVAS), by integrating the proposed techniques to show their effectiveness. The experimental results show that the proposed techniques are promising.

In this paper, we propose new graph-based data structure and indexing to organize and retrieve video data. Several researches have shown that a graph can be a better candidate for modeling semantically rich and complicated multimedia data. However, there are few methods that consider the temporal feature of video data, which is a distinguishable and representative characteristic when compared with other multimedia (i.e., images). In order to consider the temporal feature effectively and efficiently, we propose a new graph-based data structure called Spatio-Temporal Region Graph (STRG). Unlike existing graph-based data structures which provide only spatial features, the proposed STRG further provides temporal features, which represent temporal relationships among spatial objects. The STRG is decomposed into its subgraphs in which redundant subgraphs are eliminated to reduce the index size and search time, because the computational complexity of graph matching (subgraph isomorphism) is NP-complete. In addition, a new distance measure, called Extended Graph Edit Distance (EGED), is introduced in both non-metric and metric spaces for matching and indexing respectively. Based on STRG and EGED, we propose a new indexing method STRG-Index, which is faster and more accurate since it uses tree structure and clustering algorithm. We compare the STRG-Index with the M-tree, which is a popular tree-based indexing method for multimedia data. The STRG-Index outperforms the M-tree for various query loads in terms of cost and speed.

Colonoscopy is an important screening procedure for colorectal cancer. During this procedure, the endoscopist visually inspects the colon. Currently, there is no content-based analysis and retrieval system that automatically analyzes videos captured from colonoscopic procedures and provides a user-friendly and efficient access to important content. Such a system will be valuable for endoscopic research and education. The first necessary step for the analysis is parsing for semantic units. Since the characteristics of colonoscopy videos differ from those of videos studied in the literature, we introduce a new video parsing framework that includes: (i) a new scene definition and a new video parsing paradigm; (ii) a novel scene segmentation algorithm using audio analysis and finite state automata to recognize scenes and associated boundaries. Our experimental results show average precision and recall of 95% and 81%, respectively, for parsing scenes. The framework is extensible to videos captured from other endoscopic procedures such as upper gastrointestinal endoscopy, enteroscopy, cystoscopy, and laparoscopy.

Colonoscopy is an important screening tool for colorectal cancer. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. The video data are displayed on a monitor for real-time analysis by the endoscopist. We call videos captured from colonoscopic procedures colonoscopy videos. Because these videos possess unique characteristics, new types of semantic units and parsing techniques are required. In this paper, we define new semantic units called operation shots, each is a segment of visual and audio data that correspond to a therapeutic or biopsy operation. We introduce a new spatio-temporal analysis technique to detect operation shots. Our experiments on colonoscopy videos demonstrate that the technique does not miss any meaningful operation shots and incurs a small number of false operation shots. Our prototype parsing software implements the operation shot detection technique along with our other techniques previously developed for colonoscopy videos. Our browsing tool enables users to quickly locate operation shots of interest. The proposed technique and software are useful (1) for post-procedure reviews and analyses for causes of complications due to biopsy or therapeutic operations, (2) for developing an effective content-based retrieval system for colonoscopy videos to facilitate endoscopic research and education, and (3) for development of a systematic approach to assess endoscopists' procedural skills.

A typical query image contains not only relevant objects, but also irrelevant image areas. The latter, referred to as noise, has limited the effectiveness of existing image retrieval systems. In this paper, we propose a technique that allows users to define arbitrary-shaped queries out of example images. We present a new similarity model, and introduce an indexing technique for this new environment. Our query model is more expressive than the standard query-by-example. The user can draw a contour around a number of objects to specify spatial (relative distance) and scaling (relative size) constraints among them, or use separate contours to disassociate these objects. Our experimental results confirm that traditional approaches, such as Local Color Histogram and Correlogram, suffer from noisy queries. In contrast, our method can leverage arbitrary-shaped queries to offer significantly better performance. This is achieved using only a fraction of the storage overhead required by the other two techniques.

Investigates efficient ways of broadcasting data to mobile users over multiple physical channels, which cannot be coalesced into a lesser number of high-bandwidth channels. We propose the use of an MLMC (multi-level multi-channel) air cache, which can provide mobile users with data based on their popularity factor. We provide a wide range of design considerations for the server which broadcasts over the MLMC cache. We also investigate some novel techniques for a mobile user to access data from the MLMC cache and show the advantages of designing the broadcasting strategy in tandem with the access behavior of the mobile users. Finally, we provide experimental results to compare the techniques we introduce.

Shot boundary detection (SBD) is the first fundamental step to managing video databases. Existing automatic SBD techniques, however, are based on sequential search, and therefore too expensive for practical use. To address this problem, we investigate a non-linear approach in which most video frames do not need to be compared. It can improve their performance substantially. Our experiments show that this idea speeds up a conventional method based on color histograms up to 16 times (five times on average) while preserving the same accuracy.

We present in this paper a fully automatic content-based approach to organizing and indexing video data. Our methodology involves three steps:

The rapid growth of digital image data increases the need for efficient and effective image retrieval systems. Such systems should provide functionality that tailors to the user's need at the query time. In this paper, we propose a new image retrieval technique that allows users to control the relevantness of the results. For each image, the color contents of its regions are captured and used to compute similarity. Various factors, assigned automatically or by the user, allow high recall and precision to be obtained. We implemented the proposed technique for a large database of 16,000 images. Our experimental results show that this technique is not only space-time efficient but also more effective than recently proposed color histogram techniques.

Periodic broadcast and scheduled multicast have been shown to be very effective in reducing the demand on server bandwidth. While periodic broadcast is ideally suited for very popular videos, scheduled multicast is better for less demanded objects. Work has also been done to show that a hybrid of these techniques offers the best performance. Existing hybrid techniques, however, assume that the workload does not change with time. This assumption is not true for many applications, such as movie on demand, digital video libraries, or electronic commerce. In this paper we show evidence that existing scheduled multicast techniques are not suited for hybrid designs. To address this issue, we propose a new scheme, and use it to design an adaptive hybrid strategy which adjusts itself to cope with a changing workload. We provide simulation results to show that the proposed technique is significantly better than the best static approach in terms of service latency throughput, defection rate, and unfairness.