Emerging viral infections seriously threaten human health globally. Several challenges exist in identifying effective compounds against viral infections: (1) at the initial stage of a new virus outbreak, little information, except for its genome information, may be available; (2) although the identified compounds may be effective, they may be toxic in vivo and (3) cytokine release syndrome (CRS) triggered by viral infections is the primary cause of mortality. Currently, an integrative tool that takes all those aspects into consideration for identifying effective compounds to prevent viral infections is absent. In this study, we developed iDMer, as an integrative and mechanism-driven response system for addressing these challenges during the sudden virus outbreaks. iDMer comprises three mechanism-driven compound identification modules, that is, a virus-host interaction-oriented module, an autophagy-oriented module and a CRS-oriented module. As a one-stop integrative platform, iDMer incorporates compound toxicity evaluation and compound combination identification for virus treatment with clear mechanisms. iDMer was successfully tested on five viruses, including the current severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Our results indicated that, for all five tested viruses, compounds that were reported in the literature or experimentally validated for virus treatment were enriched at the top, demonstrating the generalized effectiveness of iDMer. Finally, we demonstrated that combinations of the individual modules successfully identified combinations of compounds effective for virus intervention with clear mechanisms.

For genome-wide CRISPR off-target cleavage sites (OTS) prediction, an important issue is data imbalance-the number of true OTS recognized by whole-genome off-target detection techniques is much smaller than that of all possible nucleotide mismatch loci, making the training of machine learning model very challenging. Therefore, computational models proposed for OTS prediction and scoring should be carefully designed and properly evaluated in order to avoid bias. In our study, two tools are taken as examples to further emphasize the data imbalance issue in CRISPR off-target prediction to achieve better sensitivity and specificity for optimized CRISPR gene editing. We would like to indicate that (1) the benchmark of CRISPR off-target prediction should be properly evaluated and not overestimated by considering data imbalance issue; (2) incorporation of efficient computational techniques (including ensemble learning and data synthesis techniques) can help to address the data imbalance issue and improve the performance of CRISPR off-target prediction. Taking together, we call for more efforts to address the data imbalance issue in CRISPR off-target prediction to facilitate clinical utility of CRISPR-based gene editing techniques.

To leverage large-scale weakly-tagged images for computer vision tasks (such as object detection and scene recognition), a novel cross-modal tag cleansing and junk image filtering algorithm is developed for cleansing the weakly-tagged images and their social tags (i.e., removing irrelevant images and finding the most relevant tags for each image) by integrating both the visual similarity contexts between the images and the semantic similarity contexts between their tags. Our algorithm can address the issues of spams, polysemes and synonyms more effectively and determine the relevance between the images and their social tags more precisely, thus it can allow us to create large amounts of training images with more reliable labels by harvesting from large-scale weakly-tagged images, which can further be used to achieve more effective classifier training for many computer vision tasks.

We present an intelligent photo slideshow system that automatically analyzes thematic information about the photo collection and utilizes such information to generate compositions and transitions in two modes: story-telling mode and person-highlighting mode. In the story-telling mode the system groups photos by a theme-based clustering algorithm and multiple photos in each theme cluster are seamlessly tiled on a slide. Multiple tiling layouts are generated for each theme cluster and the slideshow is animated by intra-cluster transitions. In the person-highlighting mode, the system first recognizes faces from photos and creates photo clusters for individuals. It then uses face areas as ROI (Regions of Interests) and creates various content-based transitions to highlight individuals in a cluster. With an emphasis on photo content, our system creates slideshows with more fluid, dynamic and meaningful structure compared to existing systems.

The amount of photo that is captured and stored with mobile devices is growing rapidly. We regularly see traditional desktop multimedia applications being ported to mobile devices. However, less often do we see novel interaction mechanism being developed to effectively deal with the physical limitations of the display and input of such devices. We built a mobile application called iPhotobook that enables users to create and edit photo books on iPhones. The application leveraged image analysis algorithms to automate user tasks of photo selection, grouping, editing and layout that would be difficult to accomplish without a large screen input. In this paper, we present these technologies and more importantly illustrate how they work together seamlessly with a gesture based user interface that creates a fun photo book authoring experience. Although the iPhone is our platform of choice and photo book creation is our target application, our innovations on UI design and automation algorithms may well be generalized to other small screen devices and applied to other mobile media applications.

Digital video now plays an important role in supporting more profitable online patient training and counseling, and integration of patient training videos from multiple competitive organizations in the health care network will result in better offerings for patients. However, privacy concerns often prevent multiple competitive organizations from sharing and integrating their patient training videos. In addition, patients with infectious or chronic diseases may not want the online patient training organizations to identify who they are or even which video clips they are interested in. Thus, there is an urgent need to develop more effective techniques to protect both video content privacy and access privacy . In this paper, we have developed a new approach to construct a distributed Hippocratic video database system for supporting more profitable online patient training and counseling. First, a new database modeling approach is developed to support concept-oriented video database organization and assign a degree of privacy of the video content for each database level automatically. Second, a new algorithm is developed to protect the video content privacy at the level of individual video clip by filtering out the privacy-sensitive human objects automatically. In order to integrate the patient training videos from multiple competitive organizations for constructing a centralized video database indexing structure, a privacy-preserving video sharing scheme is developed to support privacy-preserving distributed classifier training and prevent the statistical inferences from the videos that are shared for cross-validation of video classifiers. Our experiments on large-scale video databases have also provided very convincing results.

Many computer vision applications, such as image classification and video indexing, are usually multi-label classification problems in which an instance can be assigned to more than one category. In this paper, we present a novel multi-label classification approach with hypergraph regularization that addresses the correlations among different categories. First, a hypergraph is constructed to capture the correlations among different categories, in which each vertex represents one training instance and each hyperedge for one category contains all the instances belonging to the same category. Then, an improved SVM like learning system incorporating the hypergraph regularization, called Rank-HLapSVM, is proposed to handle the multi-label classification problems. We find that the corresponding optimization problem can be efficiently solved by the dual coordinate descent method. Many promising experimental results on the real datasets including ImageCLEF and MediaMill demonstrate the effectiveness and efficiency of the proposed algorithm.

Existing techniques to automatically create image thumbnail(s) for videos are mostly based on low-level feature analysis of the video frames, such as color and motion information. However, these approaches do not contain semantic models of the underlying theme of the video, and as a result, the selected frames may not be semantically representative. To address this problem, we propose a theme-based keyframe selection algorithm that explicitly models the visual characteristics of the underlying video theme. This thematic model is constructed by finding the common features of relevant visual samples, which are obtained by querying a visual database with keywords associated with the video. Our initial testing on a set of videos shows promising results of our video thumbnail image selection method.

As digital cameras and online photo sharing services become more popular, digital images are growing exponentially on the Internet. Thus supporting more effective retrieval from large-scale image collections has become a challenging issue for CBIR community. In this paper, we have developed a human-centered computing framework for assisting users to access large-scale online image collections: (a) filtering out junk images from Google images; and (b) exploring large-scale Flickr images.

Computer-assisted photo album creation continues to be a challenging application as it requires integrated technical solutions to many difficult problems. Effective solutions must leverage both design knowledge and image understanding algorithms to automate time-consuming tasks like image selection, grouping, cropping, layout and background selection. At the same time, they should allow the user to cater to personal tastes by fine-tuning aspects of album appearance. MagicPhotobook is a photobook authoring system that takes steps in these directions by providing advances over prior solutions in the following areas: automatic image selection and theme-based image grouping; dynamic page layout; automatic cropping; automatic background selection; design-preserving background artwork transformation; and a simple yet powerful user interface for personalization.

Due to the rapid growth in personal image collections, there is increasing interest on automatic detection of near duplicates. In this paper, we propose a novel fast near duplicate detection framework that takes advantages of heterogeneous features like EXIF data, global image histogram and local features. To improve the accuracy of local feature matching, we have developed a structure matching algorithm that takes into account of a local feature's neighborhood which can effectively reject mismatches. In addition, we developed a computation-sensitive cascade framework to combine stage classifiers trained on different feature spaces with different computational cost. This method can quickly accept easily identified duplicates using only cheap features without the need to extract more sophisticate but expensive ones. Compared with existing approaches, our experiments show very promising results using our new approach in terms of both efficiency and effectiveness.

In this paper, we have developed a novel framework calledJustClickto enable personalized image recommendation via exploratory search from large-scale collections of Flickr images. First, a topic network is automatically generated to summarize large-scale collections of Flickr images at a semantic level. Hyperbolic visualization is further used to enable interactive navigation and exploration of the topic network, so that users can gain insights of large-scale image collections at the first glance, build up their mental query models interactively and specify their queries (i.e., image needs) more precisely by selecting the image topics on the topic network directly. Thus, our personalized query recommendation framework can effectively address both the problem of query formulation and the problem of vocabulary discrepancy and null returns. Second, a small set of most representative images are recommended for the given image topic according to their representativeness scores. Kernel principal component analysis and hyperbolic visualization are seamlessly integrated to organize and layout the recommended images (i.e., most representative images) according to their nonlinear visual similarity contexts, so that users can assess the relevance between the recommended images and their real query intentions interactively. An interactive interface is implemented to allow users to express their time-varying query intentions precisely and to direct ourJustClicksystem to more relevant images according to their personal preferences. Our experiments on large-scale collections of Flickr images show very positive results.

The keyword-based Google images search engine is now becoming very popular for online image search. Unfortunately, only the text terms that are explicitly or implicitly linked with the images are used for image indexing but the associated text terms may not have exact correspondence with the underlying image semantics, thus the keyword-based Google images search engine may return large amounts of junk images which are irrelevant to the given keyword-based queries. Based on this observation, we have developed an interactive approach to filter out the junk images from keyword-based Google images search results and our approach consists of the following major components. a) A kernel-based image clustering technique is developed to partition the returned images into multiple clusters and outliers. b) Hyperbolic visualization is incorporated to display large amounts of returned images according to their nonlinear visual similarity contexts, so that users can assess the relevance between the returned images and their real query intentions interactively and select one or multiple images to express their query intentions and personal preferences precisely. c) An incremental kernel learning algorithm is developed to translate the users' query intentions and personal preferences for updating the mixture-of-kernels and generating better hypotheses to achieve more accurate clustering of the returned images and filter out the junk images more effectively. Experiments on diverse keyword-based queries from Google images search engine have obtained very positive results. Our junk image filtering system is released for public evaluation at: http://www.cs.uncc.edu/~jfan/google-demo/.

Creating an artifact that captures the story or memory from a large photo collection is a difficult task, because the tools available are either too difficult to learn, or oversimplified to the point that they lack flexibility. Individual techniques have been developed to automate parts of the selection-editing-composition cycle, but relatively little has been done to strike the right overall balance between the fully automatic and the fully manual. In this paper, we present miCollage, which attempts to piece together individual technologies to create a compelling collage authoring experience. The system consists of three main components. In the selection component, the system can make proactive suggestions about which photos to add to the collage as well as help the user to find similar or related photos. In the editing component, the system applies automatic cropping and enhancement to the images. In the layout component, the system suggests alternative layouts but is still able to accommodate manual changes, while satisfying various spatial constraints. The user interface connects the components seamlessly, allowing a best of both worlds between fully manual and fully automatic collage authoring.

In this paper, we have developed a new scheme for achieving multilevel annotations of large-scale images automatically. To achieve more sufficient representation of various visual properties of the images, both the global visual features and the local visual features are extracted for image content representation. To tackle the problem of huge intraconcept visual diversity, multiple types of kernels are integrated to characterize the diverse visual similarity relationships between the images more precisely, and a multiple kernel learning algorithm is developed for SVM image classifier training. To address the problem of huge interconcept visual similarity, a novel multitask learning algorithm is developed to learn the correlated classifiers for the sibling image concepts under the same parent concept and enhance their discrimination and adaptation power significantly. To tackle the problem of huge intraconcept visual diversity for the image concepts at the higher levels of the concept ontology, a novel hierarchical boosting algorithm is developed to learn their ensemble classifiers hierarchically. In order to assist users on selecting more effective hypotheses for image classifier training, we have developed a novel hyperbolic framework for large-scale image visualization and interactive hypotheses assessment. Our experiments on large-scale image collections have also obtained very positive results.

In this paper, we have proposed a novel framework for mining multilevel image semantics via hierarchical classification. To bridge the semantic gap more successfully, salient objects are used to characterize the intermediate image semantics effectively. The salient objects are defined as the connected image regions that capture the dominant visual properties linked to the corresponding physical objects in an image. To achieve a more reliable and tractable concept learning in high-dimensional feature space, a novel algorithm calledproduct of mixture-experts(PoM) is proposed to reduce the size of training images and speed up concept learning. A novel hierarchical concept learning algorithm is proposed by incorporating concept ontology and multitask learning to enhance the discrimination power of the concept models and reduce the computational complexity for learning the concept models for large amount of image concepts, which may have huge intra-concept variations and inter-concept similarities on their visual properties. A hyperbolic image visualization algorithm has been developed for allowing users to specify their queries easily and assess the query results interactively. Our experiments on large-scale image collections have also obtained very positive results.

For online medical education purposes, we have developed a novel scheme to incorporate the results of semantic video classification to select the most representative video shots for generating concept-oriented summarization and skimming ofsurgery education videos. First, salient objects are used as the video patterns for feature extraction to achieve a good representation of the intermediate video semantics. The salient objects are defined as the salient video compounds that can be used to characterize the most significant perceptual properties of the corresponding real world physical objects in a video, and thus the appearances of such salient objects can be used to predict the appearances of the relevant semantic video concepts in a specific video domain. Second, a novelmulti-modal boostingalgorithm is developed to achieve more reliable video classifier training by incorporating feature hierarchy and boosting to dramatically reduce both the training cost and the size of training samples, thus it can significantly speed up SVM (support vector machine) classifier training. In addition, the unlabeled samples are integrated to reduce the human efforts on labeling large amount of training samples. Finally, the results of semantic video classification are incorporated to enable concept-oriented video summarization and skimming. Experimental results in a specific domain ofsurgery education videosare provided.

In this paper, a hierarchical classification framework has been proposed for bridging the semantic gap effectively and achieving multi-level image annotation automatically. First, the semantic gap between the low-level computable visual features and users' real information needs is partitioned into four smaller gaps, and multiple approachesallare proposed to bridge these smaller gaps more effectively. To learn more reliable contextual relationships between the atomic image concepts and the co-appearances of salient objects, a multi-modal boosting algorithm is proposed. To enable hierarchical image classification and avoid inter-level error transmission, a hierarchical boosting algorithm is proposed by incorporating concept ontology and multi-task learning to achieve hierarchical image classifier training with automatic error recovery. To bridge the gap between the computable image concepts and the users' real information needs, a novel hyperbolic visualization framework is seamlessly incorporated to enable intuitive query specification and evaluation by acquainting the users with a good global view of large-scale image collections. Our experiments on large-scale image databases have also obtained very positive results.

Most existing content-based video retrieval (CBVR) systems are now amenable to support automatic low-level feature extraction, but they still have limited effectiveness from a user's perspective because of the semantic gap. Automatic video concept detection via semantic classification is one promising solution to bridge the semantic gap. To speed up SVM video classifier training in high-dimensional heterogeneous feature space, a novel multimodal boosting algorithm is proposed by incorporating feature hierarchy and boosting to reduce both the training cost and the size of training samples significantly. To avoid the inter-level error transmission problem, a novel hierarchical boosting scheme is proposed by incorporating concept ontology and multitask learning to boost hierarchical video classifier training through exploiting the strong correlations between the video concepts. To bridge the semantic gap between the available video concepts and the users' real needs, a novel hyperbolic visualization framework is seamlessly incorporated to enable intuitive query specification and evaluation by acquainting the users with a good global view of large-scale video collections. Our experiments in one specific domain of surgery education videos have also provided very convincing results.

Robust detection of a large dictionary of salient objects in natural image database is of fundamental importance to image retrieval systems. We review three popular frameworks for salient object detection, i.e., segmentation-based method, grid-based method and part-based method and discuss their advantages and limitations. We argue that using these frameworks individually is generally not enough to handle a large number of salient object classes accurately because of the intrinsic diversity of salient object features. Motivated by this observation, we have proposed a new system which combines the merits of these frameworks into one single hybrid system. The system automatically selects the appropriate modeling method for each individual object class using J measure and shape variance. We conduct comparison experiments on two popular image dataset – Corel and LabelMe. Empirical results have shown that the proposed hybrid method is more general and can handle much more salient object classes in a robust manner.

The performance of image classifiers largely depends on two inter-related issues:(1)suitable frameworks for image content representation and automatic feature extraction;(2) effective algorithms for image classifier training and feature subset selection. To address the first issue, a multiresolution grid-based framework is proposed for image content representation and feature extraction to bypass the time-consuming and erroneous process for image segmentation. To address the second issue, a hierarchical boosting algorithm is proposed by incorporating feature hierarchy and boosting to scale up SVM image classifier training in high-dimensional feature space. The high-dimensional multi-modal heterogeneous visual features are partitioned into multiple low-dimensional single-modal homogeneous feature subsets and each of them characterizes certain visual property of images. For each homogeneous feature subset, principal component analysis (PCA)is performed to exploit the feature correlations and a weak classifier is learned simultaneously. After the weak classifiers for different feature subsets and grid sizes are available, they are combined to boost an optimal classifier for the given object class or image concept, and the most representative feature subsets and grid sizes are selected. Our experiments on a specific domain of natural images have obtained very positive results.

Automatic image annotation is a promising solution to enable keyword-based semantic image retrieval. In this demo, we present our image search engine I-Search implemented using a multi-level semantic image annotation algorithm. By incorporating a domain-specific ontology into the autogenerated annotations, we are able to organize large scale natural image databases into hierarchical structures for browsing and keyword-based searching without referring to external text information.

As the content of everyday news reports is unpredictable, keyword based news search engine can't provide effective services to audiences because the audiences may not be able to figure out proper keywords to search. In this paper, a novel framework is proposed to help audiences browse and retrieve news video clips without the need of keywords. Interesting keyframes and keywords are automatically extracted from news video clips and visually represented according to their interestingness and informativeness measurement. A computational approach is also developed to quantify the interestingness measurement of video clips. The keyframes and keywords are carefully organized so that the audiences can find news stories of interest at first glance.

In this paper, we have proposed a novel framework to achieve more effective classifier training by using unlabeled samples. By integrating concept hierarchy for semantic image concept organization, a hierarchical mixture model is proposed to enable multi-level image concept modeling and hierarchical classifier training. To effectively learn the base-level classifiers for the atomic image concepts at the first level of the concept hierarchy, we have proposed a novel adaptive EM algorithm to achieve more effective classifier training with higher prediction accuracy. To effectively learn the classifiers for the higher-level semantic image concepts, we have also proposed a novel technique for classifier combining by using hierarchical mixture model. The experimental results on two large-scale image databases are also provided.

Automatic image annotation is a promising solution to enable semantic image retrieval via keywords. In this paper, we propose a multi-level approach to annotate the semantics of natural scenes by using both the dominant image components (salient objects) and the relevant semantic concepts. To achieve automatic image annotation at the content level, we use salient objects as the dominant image components for image content representation and feature extraction. To support automatic image annotation at the concept level, a novel image classification technique is developed to map the images into the most relevant semantic image concepts. In addition, Support Vector Machine (SVM) classifiers are used to learn the detection functions for the pre-defined salient objects and finite mixture models are used for semantic concept interpretation and modeling. An adaptive EM algorithm has been proposed to determine the optimal model structure and model parameters simultaneously. We have also demonstrated that our algorithms are very effective to enable multi-level annotation of natural scenes in a large-scale image dataset.

Multi-level annotation of images is a promising solution to enable more effective semantic image retrieval by using various keywords at different semantic levels. In this paper, we propose a multi-level approach to annotate the semantics of natural scenes by using both the dominant image components and the relevant semantic concepts. In contrast to the well-known image-based and region-based approaches, we use the salient objects as the dominant image components to achieve automatic image annotation at the content level. By using the salient objects for image content representation, a novel image classification technique is developed to achieve automatic image annotation at the concept level. To detect the salient objects automatically, a set of detection functions are learned from the labeled image regions by using Support Vector Machine (SVM) classifiers with an automatic scheme for searching the optimal model parameters. To generate the semantic concepts, finite mixture models are used to approximate the class distributions of the relevant salient objects. An adaptive EM algorithm has been proposed to determine the optimal model structure and model parameters simultaneously. We have also demonstrated that our algorithms are very effective to enable multi-level annotation of natural scenes in a large-scale dataset.