Fine-grained human action recognition is a core research topic in computer vision. Inspired by the recently proposed hierarchy representation of fine-grained actions in FineGym and SlowFast network for action recognition, we propose a novel multi-task network which exploits the FineGym hierarchy representation to achieve effective joint learning and prediction for fine-grained human action recognition. The multi-task network consists of three pathways of SlowOnly networks with gradually increased frame rates for events, sets and elements of fine-grained actions, followed by our proposed integration layers for joint learning and prediction. It is a two-stage approach, where it first learns deep feature representation at each hierarchical level, and is followed by feature encoding and fusion for multi-task learning. Our empirical results on the FineGym dataset achieve a new state-of-the-art performance, with 91.80% Top-1 accuracy and 88.46% mean accuracy for element actions, which are 3.40% and 7.26% higher than the previous best results.

The heart of every vehicle is its engine. Many factors, either internal or external, contribute to the aircraft engine's durability and lifespan. In this paper, we aim to assist the analyst with a qualitative analysis of the possible external cause of engine failures. We work closely with domain experts to study the domain knowledge, analyze challenging tasks, and abstract user requirements. We present EngineQV, a visualization system that integrates multiple geo-temporal engine-associated records. It provides intuitive exploration and understanding of the data from various aspects. The system features a dynamic query on the datasets and incorporates several customized interactive visualizations. A user may query a certain group of engines or compare multiple engine groups, identify an issue, and find its potential causes. The functionality and usability of EngineQV are evaluated by two case studies, through knowledge discovery from records of a single engine and visual comparison of multiple engines. The validity of the system is confirmed by expert feedback.

Coronary optical coherence tomography (OCT) is a new high-resolution intravascular imaging technology that clearly depicts coronary artery stenosis and plaque information. Study of coronary OCT images is of significance in the diagnosis of coronary atherosclerotic heart disease (CAD). We introduce a new method based on the convolutional neural network (CNN) and an improved random walk (RW) algorithm for the recognition and segmentation of calcified, lipid and fibrotic plaque in coronary OCT images. First, we design CNN with three different depths (2, 4 or 6 convolutional layers) to perform the automatic recognition and select the optimal CNN model. Then, we device an improved RW algorithm. According to the gray-level distribution characteristics of coronary OCT images, the weights of intensity and texture term in the weight function of RW algorithm are adjusted by an adaptive weight. Finally, we apply mathematical morphology in combination with two RWs to accurately segment the plaque area. Compared with the ground truth of clinical segmentation results, the Jaccard similarity coefficient (JSC) of calcified and lipid plaque segmentation results is 0.864, the average symmetric contour distance (ASCD) is 0.375[Formula: see text]mm, the JSC and ASCD reliabilities are 88.33% and 92.50% respectively. The JSC of fibrotic plaque is 0.876, the ASCD is 0.349[Formula: see text]mm, the JSC and ASCD reliabilities are 90.83% and 95.83% respectively. In addition, the average segmentation time (AST) does not exceed 5 s. Reliable and significantly improved results have been achieved in this study. Compared with the CNN, traditional RW algorithm and other methods. The proposed method has the advantages of fast segmentation, high accuracy and reliability, and holds promise as an aid to doctors in the diagnosis of CAD.

In the visual analysis of high-dimensional data points, star coordinate plot maps the data points into an observable space by conducting a linear combination of radial distributed axes vectors. However, when plotting additional data points which are quite common in an engine performance and simulation model evaluation, the star coordinate is not effective due to undetermined plotting boundary. In response, we propose a geometric algorithm to determine the axis-aligned bounding box, minimum bounding box, and bounding polygon of the star coordinate with an evenly distributed configuration and an arbitrary distributed configuration respectively. These boundaries enable a continuous point rendering that adapts to the screen space boundary. Moreover, the determination of the boundaries provides deeper understandings on the star coordinate plots. Firstly, we amend the expression of the star coordinate by considering the boundaries such that the plotting can be generated reasonably. Secondly, the boundaries provide an objective and quantitative transformation to the widely accepted RadViz measures, in order to exploit the advantages of both. Finally, property analysis, such as the clumping effect of the star coordinate plot, is conducted based on the calculated boundaries.

In this paper we consider the problem of graph-based transductive classification, and we are particularly interested in the directed graph scenario which is a natural form for many real world applications. Different from existing research efforts that either only deal with undirected graphs or circumvent directionality by means of symmetrization, we propose a novel random walk approach on directed graphs using absorbing Markov chains, which can be regarded as maximizing the accumulated expected number of visits from the unlabeled transient states. Our algorithm is simple, easy to implement, and works with large-scale graphs on binary, multiclass, and multi-label prediction problems. Moreover, it is capable of preserving the graph structure even when the input graph is sparse and changes over time, as well as retaining weak signals presented in the directed edges. We present its intimate connections to a number of existing methods, including graph kernels, graph Laplacian based methods, and spanning forest of graphs. Its computational complexity and the generalization error are also studied. Empirically, our algorithm is evaluated on a wide range of applications, where it has shown to perform competitively comparing to a suite of state-of-the-art methods. In particular, our algorithm is shown to work exceptionally well with large sparse directed graphs with e.g., millions of nodes and tens of millions of edges, where it significantly outperforms other state-of-the-art methods. In the dynamic graph setting involving insertion or deletion of nodes and edge-weight changes over time, it also allows efficient online updates that produce the same results as of the batch update counterparts.

In our turbine performance assessment, we need to provide an effective visual analytics tool in handling high-dimensional datasets. We have employed RadViz in 2D exploratory data analysis. However, with the increase of dataset size and dimensionality, the clumping of projected data points towards the origin in RadViz causes low space utilization, which largely degenerates the visibility of the feature characteristics. In this study, to better evaluate the hidden patterns in the center region, we propose histogram-based techniques to manipulate the radial distribution of data points in RadViz. We present RadViz in the polar coordinate system for convenient radial operations. Based on this, we define the radial equalization method to automatically spread out the frequency and the radial specification method to shape the distribution based on the user's requirement. Furthermore, we utilize the information in high-dimensional space as histogram and reference point to design and control the radial distribution of RadViz. Computational experiments have been conducted on turbine performance simulation data. Our proposed techniques are shown advantageous in query result display and outlier detection with a set of high-dimensional datasets.

BACKGROUND: Many human muscle wasting diseases are associated with abnormal nuclear localization. During metamorphosis in Drosophila melanogaster, multi-nucleated larval dorsal abdominal muscles either undergo cell death or are remodeled to temporary adult muscles. Muscle remodeling is associated with anti-polar nuclear migration and atrophy during early pupation followed by polar migration and muscle growth during late pupation. Muscle remodeling is a useful model to study genes involved in myonuclear migration. Previously, we showed that loss of Cathepsin-L inhibited anti-polar movements, while knockdown of autophagy-related genes affected nuclear positioning along the medial axis in late metamorphosis. RESULTS: To compare the phenotypic effects of gene perturbations on nuclear migration more objectively, we developed new descriptors of myonuclear distribution. To obtain nuclear pattern features, we designed an algorithm to detect and track nuclear regions inside live muscles. Nuclear tracks were used to distinguish between fast moving nuclei associated with fragments of dead muscles (sarcolytes) and slow-moving nuclei inside remodelled muscles. Nuclear spatial pattern features, such as longitudinal (lonNS) and lateral nuclear spread (latNS), allowed us to compare nuclear migration during muscle remodelling in different genetic backgrounds. Anti-polar migration leads to a lonNS decrease. As expected, lack of myonuclear migration caused by the loss of Cp1 was correlated with a significantly lower lonNS decrease. Unexpectedly, the decrease in lonNS was significantly enhanced by Atg9, Atg5 and Atg18 silencing, indicating that the loss of autophagy promotes the migration and clustering of nuclei. Loss of autophagy also caused a scattering of nuclei along the lateral axis, leading to a two-row as opposed to single row distribution in control muscles. Increased latNS resulting from knockdown of Atg9 and Atg18 was correlated with increased muscle diameter, suggesting that the wider muscle fibre promotes lateral displacement of nuclei from the medial axis during polar migration. CONCLUSIONS: We developed new nuclear features to characterize the dynamics of nuclear distribution in time-lapse images of Drosophila metamorphosis. Image quantification improved our understanding of phenotypic abnormalities in nuclear distribution resulting from gene perturbations. Therefore, in vivo imaging and quantitative image analysis of Drosophila metamorphosis promise to provide novel insights into the relationship between muscle wasting and myonuclear positioning.

Interpolation is the most executed operation and one of the main bottlenecks in embedded imaging, registration, and rendering systems. Existing methods either lack parallelization and scalability capabilities or are too computationally complex to execute efficiently. Acknowledging that improving execution time leads to degradation in image quality, we formulate a novel Negative Squared Distance (NSD) interpolation method that exhibits excellent performance by exploiting Look-Up Table (LUT) optimization for Field Programmable Gate Array (FPGA) speedup, with a balanced trade-off in quality in our embedded endomicroscopic imaging system. Quantitative analysis on performance and resource utilization of NSD against existing methods is reported through an implementation on a Xilinx ML605 platform. Functional validation using practical image resizing and rotation applications to compare qualitative performance against existing algorithms is performed and presented with visual and numerical results. Our method is shown to have a smaller design size and produces a maximum throughput of over twofold against trilinear interpolation with on-par image quality as the baseline method.

The aim of this study is about tracing filamentary structures in both neuronal and retinal images. It is often crucial to identify single neurons in neuronal networks, or separate vessel tree structures in retinal blood vessel networks, in applications such as drug screening for neurological disorders or computer-aided diagnosis of diabetic retinopathy. Both tasks are challenging as the same bottleneck issue of filament crossovers is commonly encountered, which essentially hinders the ability of existing systems to conduct large-scale drug screening or practical clinical usage. To address the filament crossovers' problem, a two-step graph-theoretical approach is proposed in this paper. The first step focuses on segmenting filamentary pixels out of the background. This produces a filament segmentation map used as input for the second step, where they are further separated into disjointed filaments. Key to our approach is the idea that the problem can be reformulated as label propagation over directed graphs, such that the graph is to be partitioned into disjoint sub-graphs, or equivalently, each of the neurons (vessel trees) is separated from the rest of the neuronal (vessel) network. This enables us to make the interesting connection between the tracing problem and the digraph matrix-forest theorem in algebraic graph theory for the first time. Empirical experiments on neuronal and retinal image datasets demonstrate the superior performance of our approach over existing methods.

Conventional protein docking methods use rigid docking algorithms but the results have been unsatisfactory, because protein structures are flexible in live environments. In this study, the side-chain flexibility is introduced into protein docking. First, the Morse theory is applied to curvature labelling and region-growing for segmenting the protein surface into smaller patches. The surface critical points are then extracted for each surface region which has been obtained in the segmentation, and a list of more precise and smaller surface patches can be obtained. Secondly, based on the segmented surface, we propose to describe the protein with an ensemble of conformations which incorporate the flexibility of interface side-chains by sampling their conformations using rotamers. We define a 3D rotation invariant shape descriptor based on the Spherical Harmonics Descriptor to deal with the flexible protein surface patches. The signature for each surface patch can be more accurately described and easily compared for the similarity between the signatures. The pairwise complementarity matching is needed only between the convex patches of the ligand with concave patches of the receptor and vice versa, thus the computational cost is greatly reduced. Finally, we are able to use the iterative closest point (ICP) algorithm, thanks to the advantage of our matching method that is able to pre-compute and store the transformation invariant, fast Fourier transform (FFT) docking correlation based representation of each flexible surface patch. Compared with the global geometric matching algorithm and other methods, our FlexDock system generates much less false positive docking candidates, it shows promising advantages in identification of the complementary candidates.

Physically based deformable models have been a hot topic in the computer graphics and visualization community. However, most of the implemented models work only for isotropic materials, leaving the more general anisotropic materials less studied. We look into the challenging issues in dynamics simulation and real time visualization for anisotropic materials, and present our theoretic and practical work along the way to provide an ultimate solution to deformation with such materials. First, our models are based on continuum mechanics and approximated by the Finite Element Method, and different approaches such as co rotated, invertible, and total Lagrangian explicit dynamics. Secondly, for controls of the deformation, we have focused on materials that have their own internal structures (fibers) that determine the dynamics behaviors. We propose a fibers incorporated deformable model that can approximate the anisotropic elastic material properties. The orientation information is combined into the existing deformable models by element transformations along the given directions, which provide a control of the desired deformation. Thirdly, constraints for the strain density in linear elastic models are analyzed, and a positive-definite elasticity tensor is derived for an anisotropic material. Fourthly, an orthotropic deformation controlling frame-field is conceptualized and the frame construction tool is developed for the user to define the desired material properties. A quaternion Laplacian smoothing algorithm is designed for propagating the rotation minimization frames into the entire object. And finally, the co rotational linear FEM model is coupled with the orthotropic frame-field to realize a dynamics system, which can deal with large anisotropic deformations.

To improve safety and ease operations in using the medical imaging devices and increase the influx of digital imagery, there is a need for Radiology Information System (RIS) and Picture Archiving and Communication System (PACS) to be used in conjunction to establish a secure network for transmission of patient information. We present a web-based RIS and PACS information retrieval and visualization system implemented for diagnostic radiology, addressing its benefits in workflow improvements, operational efficiencies and convenience for the users. The web application is based on the functional and non-functional requirements elicited on three major user groups, namely, patient, referring physician and radiologist. Novel methodology in the system design and visual analytics is also reported, including Use Case Design, Use Case Descriptions, Database Design, Web Navigation and Activity Diagrams. Experimental results are given to show its effective applications in a clinical setup.

Confocal laser endomicroscope (CLE) is emerging as an in vivo and in situ imaging tool for early diagnosis of mucosal cancers. While an optical slice of the lesion obtained by CLE provides partial information of the microstructures, a concurrent 3D reconstruction would be much more helpful in extracting meaningful inferences. The locality of different structures and their depth relations can be made easy to analyze and comprehend. Unfortunately, currently available CLE systems lack the function of real-time image registration and 3D reconstruction, which motivates us to develop a novel solution to meet the need in the diagnosis and prognosis of oral mucosal lesions. Technical challenge for concurrent 3D visualization of the acquired stack of confocal images is the real-time image registration and reconstruction algorithms. We propose to exploit the graphics hardware to speed up these algorithms. Especially, we present our novel algorithms for the nonrigid image registration and interpolation in cellular imaging, volumetric feature detection and enhancement in rendering, as well as ubiquitous volume rendering on various clinic terminals including nowadays tablets and smart phones. Experiments on animal models and clinical trials on human volunteers are carried out to verify the effectiveness of the proposed technologies.

Prediction of the 3D structure of protein-protein complexes is crucial for understanding the biological system of life and thus for designing drugs. However it is a challenging problem due to the flexibility of the backbone and side-chains of the receptor and ligand. In this paper, we present a new framework for fast and flexible protein-protein docking based on geometric and physicochemical complementarity. In our approach, we segment the protein surfaces into several local surface patches and describe the effect of side-chain flexibility on each patch by sampling the space of conformation of its side-chains. Then the geometric complementarity can be determined by matching these surface patches. During the matching, we propose a new 3D shape descriptor, Spherical Harmonics Descriptor. The main property of the Spherical Harmonics Descriptor is rotation invariance which avoids taking an exhaustive set of rotations for each pair of patches. Thus, the geometric complementarity matching problem is simplified to a histogram matching problem. This can greatly reduce the computational cost. Finally, the docking candidates are evaluated by a scoring function which can accommodate geometric complementarity and physicochemical matching. The experimental results demonstrate the high efficiency and accuracy of the proposed method.

BACKGROUND: During metamorphosis in Drosophila melanogaster, larval muscles undergo two different developmental fates; one population is removed by cell death, while the other persistent subset undergoes morphological remodeling and survives to adulthood. Thanks to the ability to perform live imaging of muscle development in transparent pupae and the power of genetics, metamorphosis in Drosophila can be used as a model to study the regulation of skeletal muscle mass. However, time-lapse microscopy generates sizeable image data that require new tools for high throughput image analysis. RESULTS: We performed targeted gene perturbation in muscles and acquired 3D time-series images of muscles in metamorphosis using laser scanning confocal microscopy. To quantify the phenotypic effects of gene perturbations, we designed the Fly Muscle Analysis tool (FMAj) which is based on the ImageJ and MySQL frameworks for image processing and data storage, respectively. The image analysis pipeline of FMAj contains three modules. The first module assists in adding annotations to time-lapse datasets, such as genotypes, experimental parameters and temporal reference points, which are used to compare different datasets. The second module performs segmentation and feature extraction of muscle cells and nuclei. Users can provide annotations to the detected objects, such as muscle identities and anatomical information. The third module performs comparative quantitative analysis of muscle phenotypes. We applied our tool to the phenotypic characterization of two atrophy related genes that were silenced by RNA interference. Reduction of Drosophila Tor (Target of Rapamycin) expression resulted in enhanced atrophy compared to control, while inhibition of the autophagy factor Atg9 caused suppression of atrophy and enlarged muscle fibers of abnormal morphology. FMAj enabled us to monitor the progression of atrophic and hypertrophic phenotypes of individual muscles throughout metamorphosis. CONCLUSIONS: We designed a new tool to visualize and quantify morphological changes of muscles in time-lapse images of Drosophila metamorphosis. Our in vivo imaging experiments revealed that evolutionarily conserved genes involved in Tor signalling and autophagy, perform similar functions in regulating muscle mass in mammals and Drosophila. Extending our approach to a genome-wide scale has the potential to identify new genes involved in muscle size regulation.

In this paper, we propose a feature-based registration algorithm for the confocal fluorescence images captured by an endomicroscopy system. We first extract a number of feature points from the endomicroscopy images by applying the binarization-thinning process and using the Rutovitz crossing number. These feature points are then post-processed to eliminate the spurious ones. After that we use the purified feature points to complete the image registration between every two consecutive slice images. The aligned image stack will be finally utilized to reconstruct and visualize the 3D structure of the living cell and tissue in real time, which provides the opportunity for the clinicians to diagnose various diseases including the early-stage cancers.

Plankton image classification plays an important role in ocean biological research. In this paper, we present an approach based on the bagging technique to classify the marine plankton images captured by the shadowed image particle profiling and evaluation recorder. The difficulty of such classification is multifold because the data set is much noisier, and the plankton images are deformable, projection-variant, and often in partial occlusion. In addition, the images in our experiments are binary, thus are lack of pixel-depth information. By random sampling with replacement on the original training set, a number of independent bootstrap replicates are generated. Using these replicates as new training sets, we construct multiple classifiers that are complementary of one another. While such individual classifiers are less effective than a single classifier trained on the whole training set, the fusion of them using majority voting produces an improved tenfold cross-validation accuracy by more than 93%.

Steroid hormones are necessary for vital functions of most of vertebrates. Hormone molecules act within cells via interaction with their receptor proteins which are transcription factors. Identification of hormone response elements (HREs) in DNA is essential for understanding the mechanism of hormone-mediated gene expression regulation. We present a systematic approach for recognition of HREs within promoters of vertebrate genes. The proposed approach is based on an experimentally validated dataset and a specifically reconstructed cascade Markov model for HRE recognition with reference to its complex composition. The approach provides a reliable accuracy for HRE prediction, and may be extended to other sequence motifs with repeated or multi-component structure. The developed FPGA implementation of the Markov model is tested using a Virtex-4 board. The model is trained for prediction of HREs in promoters of hormone-responsive genes, and for further study on direct targets for androgen, progesterone and glucocorticoid hormones.

Prediction of peptides that bind to major histocompatibility complex class II (MHC-II) molecules is vital for drug discovery and vaccine development. Prediction of peptides binding to MHC-II molecules is complicated because of the broad range of their lengths. Peptides bind to the molecules at an ungapped motif present at the binding site. Obtaining an alignment of binding sites of binding proteins facilitates determining of the binding motif. However, multiple sequence alignment often fails on peptides. In this paper, we propose a genetic annealing algorithm (GAA) to identify an alignment for binding peptides that can subsequently be used to predict binding peptides. Our approach is demonstrated with a dataset having difficulty in finding a consensus motif through experimental means and using existing motif detection methods. GAA based approach outperformed Gibbs motif sampler and RANKPEP approaches in predicting peptides binding to MHC II molecules.

BACKGROUND: Peptides binding to Major Histocompatibility Complex (MHC) class II molecules are crucial for initiation and regulation of immune responses. Predicting peptides that bind to a specific MHC molecule plays an important role in determining potential candidates for vaccines. The binding groove in class II MHC is open at both ends, allowing peptides longer than 9-mer to bind. Finding the consensus motif facilitating the binding of peptides to a MHC class II molecule is difficult because of different lengths of binding peptides and varying location of 9-mer binding core. The level of difficulty increases when the molecule is promiscuous and binds to a large number of low affinity peptides. In this paper, we propose two approaches using multi-objective evolutionary algorithms (MOEA) for predicting peptides binding to MHC class II molecules. One uses the information from both binders and non-binders for self-discovery of motifs. The other, in addition, uses information from experimentally determined motifs for guided-discovery of motifs. RESULTS: The proposed methods are intended for finding peptides binding to MHC class II I-Ag7 molecule - a promiscuous binder to a large number of low affinity peptides. Cross-validation results across experiments on two motifs derived for I-Ag7 datasets demonstrate better generalization abilities and accuracies of the present method over earlier approaches. Further, the proposed method was validated and compared on two publicly available benchmark datasets: (1) an ensemble of qualitative HLA-DRB1*0401 peptide data obtained from five different sources, and (2) quantitative peptide data obtained for sixteen different alleles comprising of three mouse alleles and thirteen HLA alleles. The proposed method outperformed earlier methods on most datasets, indicating that it is well suited for finding peptides binding to MHC class II molecules. CONCLUSION: We present two MOEA-based algorithms for finding motifs, one for self-discovery and the other for guided-discovery by experimentally determined motifs, and thereby predicting binding peptides to I-Ag7 molecule. Our experiments show that the proposed MOEA-based algorithms are better than earlier methods in predicting binding sites not only on I-Ag7 but also on most alleles of class II MHC benchmark datasets. This shows that our methods could be applicable to find binding motifs in a wide range of alleles.

BACKGROUND: An important step in understanding the conditions that specify gene expression is the recognition of gene regulatory elements. Due to high diversity of different types of transcription factors and their DNA binding preferences, it is a challenging problem to establish an accurate model for recognition of functional regulatory elements in promoters of eukaryotic genes. RESULTS: We present a method for precise prediction of a large group of transcription factor binding sites - steroid hormone response elements. We use a large training set of experimentally confirmed steroid hormone response elements, and adapt a sequence-based statistic method of position weight matrix, for identification of the binding sites in the query sequences. To estimate the accuracy level, a table of correspondence of sensitivity vs. specificity values is constructed from a number of independent tests. Furthermore, feed-forward neural network is used for cross-verification of the predicted response elements on genomic sequences. CONCLUSION: The proposed method demonstrates high accuracy level, and therefore can be used for prediction of hormone response elements de novo. Experimental results support our analysis by showing significant improvement of the proposed method over previous HRE recognition methods.

In this paper, we implement a random subspace based algorithm to classify the plankton images detected in real time by the shadowed image particle profiling and evaluation recorder. The difficulty of such classification is compounded because the data sets are not only much noisier but the plankton are deformable, projection-variant, and often in partial occlusion. In addition, the images in our experiments are binary thus are lack of texture information. Using random sampling, we construct a set of stable classifiers to take full advantage of nearly all the discriminative information in the feature space of plankton images. The combination of multiple stable classifiers is better than a single classifier. We achieve over 93% classification accuracy on a collection of more than 3000 images, making it comparable with what a trained biologist can achieve by using conventional manual techniques.

Video-based handwritten character recognition (VCR) system is a new type of character recognition system with many unique advantages over online character recognition system. Its main problem is to effectively extract stroke dynamic information from video data for character recognition. We propose a new stroke extraction algorithm through dynamic stroke information analysis for a VCR system. The experimental results on over 3000 video character sequences show that our system can extract the Chinese character stroke dynamic information similar to an online system.

In this paper, we propose a robust stroke-tracing algorithm for a video-based handwritten Chinese character recognition system. By using several error correction techniques, the algorithm works effectively against various shadow and noise problems for video-based stroke-tracing of complex Chinese characters. The stroke temporal information similar to an on-line OCR system is accurately extracted.