Abstract Learning representations and extracting meaningful patterns from electroencephalogram (EEG) recordings is critical for analyzing cognitive events (e.g., predicting cognitive load). The primary challenges include individual variability, technical noise from unreliable sensor-skin contacts, and rapid temporal changes in the EEG recordings. Given the multi-factorial nature of the problems, deep learning models are natural choices for learning representations from the data. However, the extensive time required for data collection limits the number of subjects (samples) available, which is essential for building robust deep learning models. We introduce an ensemble generative adversarial network (EGAN) to generate high-fidelity EEG data. The EGAN generates multichannel EEG recordings and their spatial-spectral representation. Key design constraints were preserving topological structure and maintaining proper bias-variance trade-offs for building robust models. To ensure the quality of the synthetic data, we visually inspected the data generated by EGAN. We conducted spectral analyses to confirm that the quality and spectral similarity were comparable to EEG recordings. To illustrate the efficacy of data generated by EGAN, we developed a convolutional neural network (CNN) model to predict four levels of cognitive load. We used spatial-spectral representations (Topomap) from three frequency bands (i.e., $$\theta $$ θ , $$\alpha $$ α , and $$\beta $$ β ). We trained our model on real data and a mixture of original and EGAN-generated data with varying proportions (e.g., 50%, 70%, 100%) of real data for training. We compared the performance of models trained solely on original data to those trained on mixed data. Our results indicate that CNN models trained on EGAN-supplemented data significantly outperformed those trained using only real data across all three frequency bands and all training set proportions. For example, the model trained on the mixed data with 50% real data achieved classification F1 and accuracy scores of approximately 71%, 69%, and 72% for $$\theta $$ θ , $$\alpha $$ α , and $$\beta $$ β bands, respectively. The same model trained on only real data achieved approximately 52%, 57%, and 59% accuracy and F1 scores for the respective bands. These results demonstrate the utility of EGAN in generating high-fidelity EEG data, significantly advancing cognitive load analysis and modeling other cognitive events.

Cognitive load refers to the amount of used working memory resources, which is limited in both capacity and duration. Predicting cognitive load from raw electroencephalogram (EEG) recordings remains a challenge because of the high degree of noise due to technical variations in the recording process and the multi-factorial nature of the mapping between the EEG data and cognitive load. We present parameter optimized deep convolutional neural network (CNN) models in predicting cognitive load and address issues related to noise, data representation, and a small number of samples. We reduce the noise by computing event-related potential (ERP) from raw data. We transform time-series signal into a spatial-spectral representation called Topomap, which maintains both spatial (electrode location) and spectral (frequency) information embedded in EEG recordings. We developed an eigenspace-based bootstrap sampling technique to generate enough samples to train deep CNN models.We use two different strategies to predict four different levels of cognitive load. First, we use power spectral densities of three individual frequency bands (Theta, Alpha, Beta) of the encoding period to create a spatial-spectral representation called topomap. Second, we combine all three bands to develop a composite topomap to compare the predictive power of individual and composite representation. We implement parameter-optimized CNN models to learn the mapping between spectral-spatial information and cognitive load for both individual and composite topomap. We performed Empirical evaluations to determine the role of different frequency bands in predicting four cognitive load levels. The prediction accuracy of CNN models built using Theta, Alpha, Beta bands, and composite representation are 86%, 85%, 88%, 90% respectively. The results suggest that the Beta band has the most predictive power and composite representation produces higher accuracy than the individual frequency bands.

Finding a free path without obstacles or situation that pose minimal risk is critical for safe navigation. People who are sighted and people who are blind or visually impaired require navigation safety while walking on a sidewalk. In this paper we develop assistive navigation on a sidewalk by integrating sensory inputs using reinforcement learning. We train the reinforcement model in a simulated robotic environment which is used to avoid sidewalk obstacles. A conversational agent is built by training with real conversation data. The reinforcement learning model along with a conversational agent improved the obstacle avoidance experience about 2.5% from the base case which is 78.75%.

Speech-in-noise (SIN) comprehension decreases with age, and these declines have been related to social isolation, depression, and dementia in the elderly. In this work, we build models to distinguish the normal hearing (NH) or mild hearing impairment (HI) using the different genres of machine learning. We compute band wise power spectral density (PSD) of source- derived EEGs as features in building models using support vector machine (SVM), k-nearest neighbors (KNN), and AdaBoost classifiers and compare their performance while listeners perceived clear or noise-degraded sounds. Combining all frequency bands features obtained from the whole-brain, the SVM registered the best performance. The group classification accuracy was found to be 94.90% [area under the curve (AUC) 94.75%; F1-score 95.00%] perceived the clear speech, and for noise- degraded speech perception, accuracy was found to be 92.52% (AUC 91.12%, and F1-score 93.00%). Remarkably, individual frequency band analysis on whole-brain data showed that γ frequency band segregated groups with a best accuracy of 96.78%, AUC 96.79% for clear speech data and noise-degraded speech data yielded slightly less accuracy of 93.62% with AUC 93.17% by using SVM. A separate analysis using the left hemisphere (LH) and right hemisphere (RH) data showed that the LH activity is a better predictor of groups compared to RH. These results are consistent with the dominance of LH in auditory-linguistic processing. Our results demonstrate that spectral features of the γ-band frequency could be used to differentiate NH and HI older adults in terms of their ability to process speech sounds. These findings would be useful to model attentional and listening assistive devices to amplify a more specific pitch than others.

Deep neural networks are ubiquitous due to the ease of developing models and their influence on other domains. At the heart of this progress is convolutional neural networks (CNNs) that are capable of learning representations or features given a set of data. Making sense of such complex models (i.e., millions of parameters and hundreds of layers) remains challenging for developers as well as the end-users. This is partially due to the lack of tools or interfaces capable of providing interpretability and transparency. A growing body of literature, for example, class activation map (CAM), focuses on making sense of what a model learns from the data or why it behaves poorly in a given task. This paper builds on previous ideas to cope with the increasing demand for interpretable, robust, and transparent models. Our approach provides a simpler and intuitive (or familiar) way of generating CAM. The proposed Eigen-CAM computes and visualizes the principle components of the learned features/representations from the convolutional layers. Empirical studies were performed to compare the Eigen-CAM with the state- of-the-art methods (such as Grad-CAM, Grad-CAM++, CNN- fixations) by evaluating on benchmark datasets such as weakly- supervised localization and localizing objects in the presence of adversarial noise. Eigen-CAM was found to be robust against classification errors made by fully connected layers in CNNs, does not rely on the backpropagation of gradients, class relevance score, maximum activation locations, or any other form of weighting features. In addition, it works with all CNN models without the need to modify layers or retrain models. Empirical results show up to 12% improvement over the best method among the methods compared on weakly supervised object localization.

The assessment of knee joint gap and severity of Osteoarthritis (OA) is subjective and often inaccurate. The main source of error is due to the judgement of human expert from low resolution images (i.e., X-ray images). To address the problem, we developed an ensemble of Deep Learning (DL) model to objectively score the severity of OA only from the radiometric images. The proposed method consists of two main modules. First, we developed a scale invariant and aspect ratio preserving automatic localization and characterization of the kneecap area. Second, we developed multiple instances of "hyper parameter optimized" DL models and fused them using ensemble classification to score the severity of OA. In this implementation, we used three convolutional neural networks to improve the bias-variance trade-off, and boost accuracy and generalization. We tested our modeling framework using a collection of 4,796 X-ray images from Osteoarthritis Initiative (OAI). Our results show a higher performance (~ 2-8%) when compared to the state-of-the-art methods. Finally, this machine learning-based methodology provides a pipeline in decision support system for assessing and quantifying the OA severity.

In this paper, we present a data-driven micro-states based complex network analysis on cortical surface data to understand the connectives in modeling elicited emotion. In particular, we focus on processing arousal in identifying the differences and similarities between males and females. Micro-states are transient, patterned, quasi-stable states of a time series (e.g., Electroencephalography recording) that allows visualizing dynamic coupling and possibly time-varying levels of correlated or mutually informed activity between brain regions. To obtain cortical surface data from EEG recording, we use source localization method from Brainstorm. We adopted t-distributed stochastic neighbor embedding (t-SNE), for better visualization and finding the optimal number of clusters using different algorithms (e.g., Gaussian Mixture Models(GMM), K-means, etc). Centroids of these clusters are considered as micro-states. We used Hidden Markov Model (HMM) to compute the transition probabilities among micro-states. Subsequently, p-values on graph theoretic measures (e.g., modularity, small-worldness, etc) computed from micro-sites were used to determine the significantly distinguishable, highly segregated and densely integrated network of brain connectivity. Empirical analysis using DEAP dataset reveals that males and females have mostly complimentary micro-states with some commonalities. Males are more likely to stay in specific stable state and females are more likely to stay in transient states. Both groups utilize highly segregated and densely integrated network structure among brain regions in processing arousal.

Recent advances in neuroscience made it possible to understand how the human brain processes emotions and affective states. However, the modeling of emotion remains elusive due to inherent ambiguity and complexity related to the perception of emotions, interpersonal variabilities, and context-specific interpretations. Here, we present a robust method of modeling 4-D continuous affective space (Valence, Arousal, Like, and Dominance). First, we determined the functional areas and frequency bands related to 4-D affective space. Second, we extracted and selected a set of stable features. For both steps, we used two different feature selection methods namely: Recursive Feature Elimination (RFE) and stability selection method. Moreover, compare their performances. For the RFE, we used Random Forest (RF), Support Vector Regression (SVR), Tree-based bagging, and for the stability selection, we used Randomized Lasso as an estimator. Empirical analysis on the DEAP data set shows that the stability selection method consistently provides relevant set of bands, electrode location and features over a range of model parameters. We also observed that only a small number of locations (40%–63%) and certain frequency bands specifically, gamma band frequency over Superior Temporal Gyrus, Supramarginal Gyrus, and Somatosensory Association Cortex were the highest ranked features across the affective dimensions. The selected features using the stability criteria were used to model 4-D affective space using SVR. Empirical analyses shows that the Root Mean Square Error (RMSE) for Valence, Arousal, Dominance, and Like are 2.13, 2.00, 2.07, and 2.11 respectively. In addition, we also compare the performance of this method with feature fusion and ensemble classification. It was observed that the SVR with selected features outperformed all other approaches. The predicted Valence-Arousal-Dominance were converted to categorical emotions for seamless interpretation.

Working memory (WM) plays a critical role in cognitive skills such as, learning, reasoning, and language comprehension. WM capacity is widely used as a predictor of academic performance and many other cognitive skills. Yet, how the human brain coordinates this complex skill through its functional networks and also whether individual's WM limitations are due to contrasts in regional activation or cross-regional interconnections, remain controversial. We hypothesized that, individual differences in behavioral WM capacity could be explained by temporal dynamics in functional coupling, and the information flow within the connectome of the WM network. We recorded electrical brain activity as participants performed a visual WM task with varying degrees of cognitive load. Functional connectivity analyses on the EEG recordings were performed to identify prominent hubs and the temporal segmentation of inter-regional activities as WM processing unfolds over time. We found that neural communication during memory processing occurs predominately through short pulses of alpha-band (8-13 Hz) phase coupling that propagate in a temporally sequential manner with feedforward (bottom-up) and feedback (top-down) communication within a set of frontoparietal and fronto-temporal regions of brain. We also observed that the degree of phase coupling within fronto-parietal/temporal pathways strongly predicts working memory capacity. Additionally, this functional connectivity could distinguish individual with low vs. high WM capacity. These results underscore the importance of long-range neural network communication (phase synchrony) in WM skills and additionally the significance of temporal dynamics in functional connectivity and in particular phase synchrony.

We optimize and compare the performance of different deep learning architectures for awareness on a sidewalk using small form factor devices such as Raspberry Pi 3. Our main objective is to find deep learning architecture that is complex enough to accurately classify a set obstacles on the sidewalk. Out selection criteria are: minimum number of parameters, lower power consumption, and robustness against the effect of the diurnal cycle. In particular, we compare the performance of GoogleNet, ResNet, and VGG-16 on a database constructed for AS applications. Empirical evaluation on AS database suggests that the performance of ResNet is superior compared to other architectures e.g., 99.46% and 97.69% on RGB and La ∗ b∗ respectively. To further our objective we optimize the hyperparameters of ResNet to find architecture with a lower number of parameters without losing accuracy. Furthermore, we investigate the efficacy of different color spaces to address problems related to the diurnal cycle and power usage without sacrificing accuracy and generalizability.

The inherent intangible nature, complexity, context-specific interpretations of emotions make it difficult to quantify and model affective space. Dimensional theory is one of the effective methods to describe and model emotions. Despite recent advances in affective computing, modeling continuous affective space remains a challenge. Here, we present a computational framework to study the role of functional areas of brain and band frequencies in modeling 4-D continuous affective space (Valence, Arousal, Like and Dominance). In particular, we used Electroencephalogram (EEG) recordings and adopted a recursive feature elimination (RFE) approach to select band frequencies and electrode locations (functional areas) that are most relevant for predicting affective space. Empirical analyses on DEAP dataset [1] reveals that only a small number of locations (7-12) and certain band frequencies carry most discriminative information. Using the selected features, we modeled 4-D affective space using Support Vector Regression (SVR). Regression analysis show that Root Mean Square Error (RMSE) for Valence, Arousal, Dominance, Like are 1.40, 1.23, 1.24 and, 1.24, respectively. Besides SVR, the performance of feature fusion and ensemble classifiers were also compared to determine the robust model against technical noise and individual variations. It was observed that the prediction accuracy of the final model is up to 37% better than human judgment evaluated on same data set. Spillover effect of our approach may include design of task-specific (i.e., emotion, memory capacity) EEG headset with a minimal number of electrodes.

This paper presents a unified framework for robust and real-time recognition of a full set of Action Units or a majority of them. The key idea is to systematically divide the AUs into two subsets: strongly connected AUs (SAUs) and weakly connected AUs (WAUs). A probabilistic scoring function is introduced to divide the AUs into SAUs and WAUs based on the strength of spatial AU relations. Then, a spatio-temporal model is created to predict WAUs in real-time without using any computer vision techniques. A number of empirical analyses were performed to validate the proposed framework. The systematic division of AUs is found to be consistent across various datasets. It is also observed that the spatio-temporal model is robust in predicting WAUs within and across datasets. For example, comparison of the proposed framework with the state-of-the-art technique, such as Computer Expression Recognition Tool Kit (CERT), is performed on posed, FERA 2011, and spontaneous expression databases. The average two-alternative forced choice (2AFC) scores of WAUs on these databases are found to be 0.931, 0.637, and 0.654 for the unified framework. The corresponding numbers for the CERT are 0.795, 0.434, and 0.414 for the same datasets. Hence, the 2AFCs are improved by 17.11, 46.77, and 57.97 percent, respectively. Most importantly, the proposed approach outperforms the CERT significantly with spontaneous facial expressions. In addition, the unified framework is found to be effective in minimizing errors introduced by simultaneous display of emotion and speaking. In particular, it is observed that it has an 75.87 percent improvement over the CERT in identifying WAUs on speaking part of the FERA 2011 dataset. Additionally, it was also observed that proposed framework has better 2AFC score compared to another state of the art AU detector: LAUD 2010. Finally, the proposed approach also contributes in a significant reduction of the run-time and improves robustness in predicting WAUs from various datasets.

Assistive technology solutions can help people with disability from social isolation, depression, facilitate social interaction and enhance the quality of life. Limited access to non-verbal cues hinders the social interaction of people who are blind or visually impaired. This paper presents 'Expression' - an integrated assistive solution using Google Glass. The key function of the system is to enable the user to perceive social signals during a natural face-to-face conversation. Subjective evaluation of Expression was performed using a five (5) point Likert Scale and was found to be excellent(4:383).

Background Obesity is a complex disease that causes health complications and mortality. Recently, the Centers for Disease Control (CDC) classified obesity as a disease and introduced guidelines for diagnosis. To understand complex relationships among all contributing factors, we have developed the ObNet (Network of Semantic Association for Obesity). The concepts were adopted from ARIANA [1,2]. The Local Latent Semantic Analysis tool [3], developed in the CVPIA lab, was used to extract network of semantic associations. Web services and visualization were developed to display the associations in a flexible network form that is easy to interact.

We explored the efficacy of smartphone App reviews to understand the user experience reports that may facilitate ranking and provide insight about accessibility gaps. The main goal was to analyze the contents of the reviews to infer the presence and polarity of accessibility information. In particular, we focused on applications that are used by the users who are blind or visually impaired. In this pilot study, the contents of 173 reviews from 25 applications were analyzed. The proposed system automatically detects accessibility information in the reviews and also tests their polarity. Such a system would be useful in application ranking based on accessibility features and improve the users' interaction experiences.

The inability to access non-verbal cues is a setback for people who are blind or visually impaired. A visual-to-auditory Sensory Substitution Device (SSD) may help improve the quality of their lives by transforming visual cues into auditory cues. In this paper, we describe the design and development of a robust and real-time SSD called iFEPS -- improved Facial Expression Perception through Sound. The implementation of the iFEPS evolved over time through a participatory design process. We conducted both subjective and objective experiments to quantify the usability of the system. Evaluation with 14 subjects (7 blind + 7 blind-folded) shows that the users were able to perceive the facial expressions in most of the time. In addition, the overall subjective usability of the system was found to be scoring 4.02 in a 5 point Likert scale.

This work demonstrates a visual-to-auditory Sensory Substitution System (SSD) called Facial Expression Perception through Sound (FEPS). It is designed to enable the visually impaired people to participate in a more effective social communication by perceiving their interlocutor's facial expressions. The earlier SSDs provided feedback on inferred emotions, where as, this system responds to the facial movements. This is a better method than emotion inference due to complexities in expression-to-emotion mapping, the problem of capturing multitude of possible emotions derived from a limited facial movements and the difficulty to correctly predict emotions due to lack in ground truth data. In this work, the user's ability to understand the facial expressions has been ensured by a usability study.

The lack of ability to perceive emotions and affective states is a setback for people who are blind or visually impaired in professional and social communications. Towards developing assistive technology solution in facilitating natural dyadic conversations for people with such disability, this paper describes the development of a smart phone based system called interactive mobile affect perception system (iMAPS) for prediction of affective dimensions (valence-arousal-dominance). The proposed solution utilizes an Android platform in conjunction with a wireless network to build a fully integrated iMAPS. Empirical analyses were conducted to measure the efficacy and utility of the proposed solution. It was found that the proposed framework can predict affect dimensions with good accuracy (Maximum Correlation Coefficient for valence: 0.68, arousal: 0.71, and dominance: 0.67) in natural dyadic conversation. The overall minimum and maximum response times are (181 milliseconds) and (500 milliseconds), respectively.

The availability of massive amounts of heterogeneous and distributed biological data has prompted the development of a wide range of data analysis and data mining tools in the area of bioinformatics.However, due to the nature of the biological data, performing a specific analysis by combining such tools can be complicated and cumbersome.Yet, integration of number of tools can provide complementary information, and improve the

This paper presents Fuzzy-Adaptive-Subspace-Iteration-based Two-way Clustering (FASIC) of microarray data for finding differentially expressed genes (DEGs) from two-sample microarray experiments. The concept of fuzzy membership is introduced to transform the hard adaptive subspace iteration (ASI) algorithm into a fuzzy-ASI algorithm to perform two-way clustering. The proposed approach follows a progressive framework to assign a relevance value to genes associated with each cluster. Subsequently, each gene cluster is scored and ranked based on its potential to provide a correct classification of the sample classes. These ranks are converted into P values using the R-test, and the significance of each gene is determined. A fivefold validation is performed on the DEGs selected using the proposed approach. Empirical analyses on a number of simulated microarray data sets are conducted to quantify the results obtained using the proposed approach. To exemplify the efficacy of the proposed approach, further analyses on different real microarray data sets are also performed.

Graph mining techniques for analyzing large collections of molecules to find regularity or patterns among molecules of a specific class, such as finding common properties in large numbers of drug candidates, finding molecular features that inhibit the desired reaction etc. is an important research issue in bioinformatics as well as molecular informatics. In this context, finding frequent graphs has received increasing attention over the past years. But, the computational complexity of the underlying problem and the large amount of data to be explored essentially render traditional sequential algorithms practically useless. To address such problems a distributed algorithm is adopted to find the frequent sub-graphs and to discover interesting patterns in molecular compounds. However, this problem is characterized by a highly irregular search tree, whereby reliable workload prediction is very hard. Therefore, a genetic algorithm (GA) is proposed to solve the dynamic load-balancing problem of highly irregular search tree.

This paper proposes a procedure for finding significance of gene ranking. The microarray data usually has a large number of genes that are not differentially expressed across multiple conditions. In microarray analysis, it is a common practice to first discard these genes as uninformative based on some filtering criterion. This filtering process results in the information loss as the uninformative genes may be used to construct an empirical distribution of genes under the null hypothesis. The distribution of the non-differentially expressed genes is complex and may be regarded as a mixture of distributions. The null hypothesis is that the gene is not differentially expressed. The significance of the differentially expressed genes therefore may be estimated by using the empirical distribution function of the large number of insignificant genes. The proposed method is efficient, less computation intensive and may be applied on microarray datasets of any sample size.

This paper presents two-way clustering of microarray data using fuzzy adaptive subspace iteration (ASI) based algorithm for knowledge discovery in microarrays. It is widely believed that each gene is involved in more than one cellular function or biological process. The proposed fuzzy ASI assigns a relevance value to each gene associated with each cluster. These functional categories are ranked based on their potential in providing maximal separation between the two tissues classes; which is an indication of differentially expressed genes (DEGs). Empirical analyses on simulated, 100 artificial microarray datasets are used to quantify the results obtained using the fuzzy-ASI algorithm. Further analyses on different microarray cancer datasets revealed several important genes that are relevant with various cancers.

This paper explores the relative importance of discourse features, prosodic features and their fusion in robust classification of speech acts. Five different feature selection algorithms were used to select set of features to improve the robustness of the classification. The results showed that the ensemble-based classifiers performed best in the classification of 12 speech acts using subsets of both prosodic and discourse features.

This paper presents a unified framework for finding differentially expressed genes (DEGs) from the two-sample microarray data. The proposed framework has three interrelated modules viz. (i) two-way clustering, ii) adaptive ranking and iii) visualization. The first module uses a progressive clustering technique to functionally classify the marker genes as well as finding the DEGs and the second module yields a list of DEGs ranked based on statistical significance. A weighted scheme is employed to fuse the two-way clustering and ranking modules to find DEGs. A visualization module is added to validate the results. Empirical analyses on 50 artificially generated microarray datasets and 2 cancer datasets show that the unified framework performs better in finding DEGs when compared to reported results on the same datasets.

This paper presents two algorithms for autonomously selecting the best projection among all possible configurations when projecting a high-dimensional (HD) data set on to a 3-dimensional (3D) space using 3D star coordinate projection (3D SCP). The proposed automated algorithms use two different objective functions that minimize the stress and preserve the pair wise distance among data points before and after the projection. The objective functions follow the principle of preserving topology similar to the multidimensional scaling (MDS). The concept of topology preserving mapping is found to be effective in autonomously selecting the best projection using the 3D SCP for visualization. Empirical analyses on artificial and real datasets are performed to show the utility of the proposed methods and their performances were also compared against linear and nonlinear projection-based visualization algorithms.

BACKGROUND: This paper presents a unified framework for finding differentially expressed genes (DEGs) from the microarray data. The proposed framework has three interrelated modules: (i) gene ranking, ii) significance analysis of genes and (iii) validation. The first module uses two gene selection algorithms, namely, a) two-way clustering and b) combined adaptive ranking to rank the genes. The second module converts the gene ranks into p-values using an R-test and fuses the two sets of p-values using the Fisher's omnibus criterion. The DEGs are selected using the FDR analysis. The third module performs three fold validations of the obtained DEGs. The robustness of the proposed unified framework in gene selection is first illustrated using false discovery rate analysis. In addition, the clustering-based validation of the DEGs is performed by employing an adaptive subspace-based clustering algorithm on the training and the test datasets. Finally, a projection-based visualization is performed to validate the DEGs obtained using the unified framework. RESULTS: The performance of the unified framework is compared with well-known ranking algorithms such as t-statistics, Significance Analysis of Microarrays (SAM), Adaptive Ranking, Combined Adaptive Ranking and Two-way Clustering. The performance curves obtained using 50 simulated microarray datasets each following two different distributions indicate the superiority of the unified framework over the other reported algorithms. Further analyses on 3 real cancer datasets and 3 Parkinson's datasets show the similar improvement in performance. First, a 3 fold validation process is provided for the two-sample cancer datasets. In addition, the analysis on 3 sets of Parkinson's data is performed to demonstrate the scalability of the proposed method to multi-sample microarray datasets. CONCLUSION: This paper presents a unified framework for the robust selection of genes from the two-sample as well as multi-sample microarray experiments. Two different ranking methods used in module 1 bring diversity in the selection of genes. The conversion of ranks to p-values, the fusion of p-values and FDR analysis aid in the identification of significant genes which cannot be judged based on gene ranking alone. The 3 fold validation, namely, robustness in selection of genes using FDR analysis, clustering, and visualization demonstrate the relevance of the DEGs. Empirical analyses on 50 artificial datasets and 6 real microarray datasets illustrate the efficacy of the proposed approach. The analyses on 3 cancer datasets demonstrate the utility of the proposed approach on microarray datasets with two classes of samples. The scalability of the proposed unified approach to multi-sample (more than two sample classes) microarray datasets is addressed using three sets of Parkinson's Data. Empirical analyses show that the unified framework outperformed other gene selection methods in selecting differentially expressed genes from microarray data.

This paper compares the performance of linear and non-linear projection techniques in functionally classifying genes. The performance of both linear and non-linear data projection techniques, namely, Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Locally Linear Embedding (LLE) and Supervised Locally Linear Embedding (SLLE) were evaluated to project the microarray data onto lower dimensional subspace to estimate the intrinsic dimensionality of the microarray gene expression. Artificially generated (simulated datasets) and real microarray dataset (for example, Sporulation of budding yeast and cancer dataset) have been used to validate the performance of proposed techniques. From the empirical analysis, it was found that the non-linear projection techniques perform better in classifying the gene functions and provide a better visualization of expression profile than the linear methods.

This paper presents a 3D star coordinate-based visualization technique for exploratory data analysis. To improve the data visualization and reveal the hidden patterns in complex high dimensional data sets, first the 2D star coordinate system is extended to the 3D star coordinate system. An autonomous procedure is defined to find the best configuration for the 3D star coordinate system based on cluster validation measures. To illustrate the efficacy of the proposed techniques, empirical analysis were conducted on a number of synthetic (Five dimensional Gaussian distribution with three classes) and real (Fisher's IRIS, Leukemia, Gastric cancer and Petroleum datasets) databases. Empirical analyses shows that automated 3D star coordinate system helps in better visualization of the complex high dimensional data when compared to 2D star coordinate system and also other projection-based visualization techniques. Also the automated configuration for 3D star coordinate system reveals the hidden patterns in the complex datasets without human intervention.

This paper presents the implementation and evaluation of subspace-based clustering algorithm for robust selection of differentially expressed genes as well as the classification of tissue types from microarray data. The performance of the proposed algorithm is compared against other well known clustering algorithms and the quality of clusters is evaluated using a number of cluster validation indices. Empirical analyses on a number of synthetic and real microarray data sets suggest that the proposed subspace-based algorithm is robust in selecting differentially expressed genes and performs significantly better compared to popular clustering algorithms in selecting differentially expressed genes and classifying different tissue types.

This paper presents an adaptive subspace based two-way clustering of microarray data. To analyze the data at various scales a "Progressive" framework is introduced. The goals are to functionally classify genes and also to find differentially expressed genes in microarray expression profiles. Empirical analysis on Colon Cancer dataset shows that ASI performs favorably in grouping genes with similar functions and finding genes that may have been involved in the formation of colon cancer. It was also observed that the proposed algorithm is robust against ordering of samples and yield results consistent with ground truth information.

This paper presents a spatio-temporal approach in recognizing six universal facial expressions from visual data and using them to compute levels of interest. The classification approach relies on a two-step strategy on the top of projected facial motion vectors obtained from video sequences of facial expressions. First a linear classification bank was applied on projected optical flow vectors and decisions made by the linear classifiers were coalesced to produce a characteristic signature for each universal facial expression. The signatures thus computed from the training data set were used to train discrete hidden Markov models (HMMs) to learn the underlying model for each facial expression. The performances of the proposed facial expressions recognition were computed using five fold cross-validation on Cohn-Kanade facial expressions database consisting of 488 video sequences that includes 97 subjects. The proposed approach achieved an average recognition rate of 90.9% on Cohn-Kanade facial expressions database. Recognized facial expressions were mapped to levels of interest using the affect space and the intensity of motion around apex frame. Computed level of interest was subjectively analyzed and was found to be consistent with "ground truth" information in most of the cases. To further illustrate the efficacy of the proposed approach, and also to better understand the effects of a number of factors that are detrimental to the facial expression recognition, a number of experiments were conducted. The first empirical analysis was conducted on a database consisting of 108 facial expressions collected from TV broadcasts and labeled by human coders for subsequent analysis. The second experiment (emotion elicitation) was conducted on facial expressions obtained from 21 subjects by showing the subjects six different movies clips chosen in a manner to arouse spontaneous emotional reactions that would produce natural facial expressions.

This paper compares the performances of linear and non-linear data projection techniques in classifying six universal facial expressions from only visual data. Three different data projection techniques, namely, principal component analysis (PCA), non-negative matrix factorization (NMF) and local linear embedding (LLE) were experimented to project the facial motion onto lower dimensional subspace to estimate the intrinsic dimension of the facial expression data. The classification approach relies on a two-step strategy on the top of projected facial motion information obtained from sequence of facial expression images. First, a bank of linear classifier was applied on projected data and decision made by the linear classifiers was coalesced to produce a characteristic signature for each universal facial expressions. The signatures thus computed from the training data set were used to train discrete hidden Markov models (HMMs) to learn the underlying model for each facial expression. The performances of each data representations in classifying the facial expressions were compared using five fold cross validation on a database of 488 video sequences that include 97 subjects. To further illustrate the efficacy of the proposed approach and to better understand the effects of a number of factors that are detrimental to the facial expression recognition using only visual data were conducted. The first empirical analysis was conducted on a database consisting of 108 facial expressions collected from TV broad cast and labeled by human coder for subsequent analysis. The second experiment was conducted on facial expressions obtained from 21 subjects by showing the subjects six different clips of movies chosen in a manner to arouse spontaneous emotional reactions that would produce natural facial expression.

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.

Automatic initialization and tracking of multiple people and their body parts is one of the first steps in designing interactive multimedia applications. The key problems in this context are robust detection and tracking of people and their body parts in an unconstrained environment. This paper presents an integrated framework to address detection and tracking of multiple objects in a computationally efficient manner. In particular, a neural network-based face detector was employed to detect faces and compute person specific statistical model for skin color from the face regions. A probabilistic model was proposed to fuse the color and motion information to localize the moving body parts (hands). Multiple hypothesis tracking (MHT) algorithm was adopted to track face and hands. In real world scenes extracted features (face and hands) usually contain spurious measurements that create unconvincing trajectories and needless computations. To deal with this problem a path coherence function was incorporated along with MHT to reduce the number of hypotheses, which in turn reduces the computational cost and improves the structure of trajectories. The performance of the framework was validated using experiments on synthetic and real sequence of images.

Despite recent advances in gesture recognition, reliance on the visual signal alone to classify unrestricted continuous gesticulation is inherently error-prone. Since spontaneous gesticulation is mostly coverbal in nature, there have been some attempts of using speech cues to improve gesture recognition. Some attempts have been made in using speech cues to improve gesture recognition, e.g., keyword-gesture co-analysis. Use of such scheme is burdened by the complexity of natural language understanding. This paper offers a "signal-level" perspective by exploring prosodic phenomena of spontaneous gesture and speech co-production. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (co-articulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions in multimodal production, are exploited in an audio-visual feature integration framework using hidden Markov models (HMMs). Co-articulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.

Computer vision systems for monitoring people and collecting valuable demographic information in a social environment is an important research problem. It is expected that such a system will play an increasingly important role in enhancing user's experience and can significantly improve the intelligibility of a human computer interaction (HCI) system. For example, a robust gender classification system can provide a basis for passive surveillance and access to a smart building using demographic information or can provide valuable consumer statistics in a public place. The option of an audio cue in addition to the visual cue promises a robust solution with high accuracy and ease-of-use in human computer interaction systems. This paper investigates gender classification using Support Vector Machines (SVMs). The visual (thumbnail frontal face) and the audio (features from speech data) cues were considered for designing the classifier. Three different representations of the data, namely, raw data, principle component analysis (PCA) and non-negative matrix factorization (NMF) were used for the experimentation with visual signal. For speech, mel-cepstral coefficient and pitch were used for the experimentation. It was found that the best overall classification rates obtained using the SVM for the visual and speech data were 95.31% and 100%, respectively, on data set collected in laboratory environment. The performance of the SVM was compared with two simple classifiers namely, the nearest prototype neighbor and the k-nearest neighbor on all feature sets. It was found that the SVM outperformed the other two classifiers on all datasets. To further understand the robustness issues, the proposed approach has been applied on a large balanced (roughly equal distribution of gender, ethnicity and age group) data-base consisting of 8000 faces collected in real world environment. While, the results are very promising it indicates more to be done to make a statistically meaningful conclusion.

Emergency response requires strategic assessment of risks, decisions, and communications that are time critical while requiring teams of individuals to have fast access to large volumes of complex information and technologies that enable tightly coordinated work. The access to this information by crisis management teams in emergency operations centers can be facilitated through various human-computer interfaces. Unfortunately, these interfaces are hard to use, require extensive training, and often impede rather than support teamwork. Dialogue-enabled devices, based on natural, multimodal interfaces, have the potential of making a variety of information technology tools accessible during crisis management. This paper establishes the importance of multimodal interfaces in various aspects of crisis management and explores many issues in realizing successful speech-gesture driven, dialogue-enabled interfaces for crisis management. This paper is organized in five parts. The first part discusses the needs of crisis management that can be potentially met by the development of appropriate interfaces. The second part discusses the issues related to the design and development of multimodal interfaces in the context of crisis management. The third part discusses the state of the art in both the theories and practices involving these human-computer interfaces. In particular, it describes the evolution and implementation details of two representative systems, Crisis Management (XISM) and Dialog Assisted Visual Environment for Geoinformation (DAVE/spl I.bar/G). The fourth part speculates on the short-term and long-term research directions that will help addressing the outstanding challenges in interfaces that support dialogue and collaboration. Finally, the fifth part concludes the paper.

Although recognition of natural speech and gestures have been studied extensively, previous attempts at combining them in a unified framework to boost classification were mostly semantically motivated, e.g., keyword-gesture co-occurrence. Such formulations inherit the complexity of natural language processing. This paper presents a Bayesian formulation that uses a phenomenon of gesture and speech articulation for improving accuracy of automatic recognition of continuous coverbal gestures. The prosodic features from the speech signal were co-analyzed with the visual signal to learn the prior probability of co-occurrence of the prominent spoken segments with the particular kinematical phases of gestures. It was found that the above co-analysis helps in detecting and disambiguating small hand movements, which subsequently improves the rate of continuous gesture recognition. The efficacy of the proposed approach was demonstrated on a large database collected front the weather channel broadcast. This formulation opens new avenues for bottom-up frameworks of multimodal integration.

This paper presents a framework for designing a natural multimodal human computer interaction (HCI) system. The core of the proposed framework is a principled method for combining information derived from audio and visual cues. To achieve natural interaction, both audio and visual modalities are fused along with feedback through a large screen display. Careful design along with due considerations of possible aspects of a systems interaction cycle and integration has resulted in a successful system. The performance of the proposed framework has been validated through the development of several prototype systems as well as commercial applications for the retail and entertainment industry. To assess the impact of these multimodal systems (MMS), informal studies have been conducted. It was found that the system performed according to its specifications in 95% of the cases and that users showed ad-hoc proficiency, indicating natural acceptance of such systems.

The ability to track many people and their body parts (i.e., face and hands) in a complex environment is crucial for designing collaborative natural human computer interaction (HCI). A challenging issue in tracking body parts is the data association uncertainty while assigning measurements to the proper tracks in the case of occlusion and close interaction of body parts of different people. This paper describes a framework for tracking body parts of people in 2D/3D using a multiple hypothesis tracking (MHT) algorithm. A path coherence function has been incorporated along with MHT to reduce the negative effects of closely spaced measurements that produce unconvincing tracks and unnecessary computations. The performance of the framework has been validated using experiments on a real sequence of images.

Foveating vision sensors are important in both machine and biological vision. The term space-variant or foveating vision refers to sensor architectures based on smooth variation of resolution across the visual field, like that of the human visual system. Traditional image processing techniques do not hold when applied directly to such an image representation since the translation symmetry and the neighborhood structure in the spatial domain is broken by the space-variant properties of the sensor. Unfortunately, there has been little systematic development of image processing tools that are explicitly designed for foveated vision. The author proposes a novel approach to compute the optical flow directly on log-mapped images. We propose the use of a generalized dynamic image model (GDIM) based method for computing the optical flow as opposed to the brightness constancy model (BCM) based method. We introduce a new notion of "variable window" and use the space-variant form of gradient operator while computing the spatio-temporal gradient in log-mapped images for a better accuracy and to ensure that the local neighborhood is preserved. We emphasize that the proposed method must be numerically accurate, provide a consistent interpretation, and be capable of computing the peripheral motion. Experimental results on both the synthetic and real images have been presented to show the efficacy of the proposed method.

Tracking of people and their body parts has found application in image processing and computer vision. This paper describes a framework for tracking body parts (eg, hands and face) of multiple people in image sequences. We use a probabilistic model to fuse color and motion information to localize the body parts and employ the multiple hypothesis tracking (MHT) algorithm to track these features simultaneously. We incorporate a path coherence function along with MHT to reduce the negative effects of spurious measurements that produce unconvincing tracks and needless computations. The performance of the framework is validated using experiments on synthetic and real sequences of images.

We have developed a system for detecting and tracking human face and eye in an unstructured environment. We adopt a biologically plausible retinally connected neural network architecture and integrate it with an active vision system. While the active vision system tracks the object moving in real time, the neural network detects the face and eye location from the video stream at a slower rate. The paper provides a systematic way of creating and selecting examples for training the network by exploring the link between theory and practice. Experimental results on a real sequence of images from a space-varying sensor depicts the performance of the system.

We propose a novel approach to program a robot by demonstrating the task multiple number of times in front of a binocular vision system. We track artificially-induced features appearing in the image plane due to nonimpedimental color stickers attached at different fingertips and wrist joint, in a simultaneous feature detection and tracking framework. A Kalman filter does the tracking by recursively predicting the tentative feature location and a higher order statistics (HOS)-based data clustering algorithm extracts the feature. A fast and efficient algorithm for the vision system thus developed processes a binocular video sequence to obtain the trajectories and the orientation information of the end effector from the images of a human hand. The concept of trajectory bundle is introduced to avoid singularities and to obtain an optimal path.

The analysis of a dynamic hand gesture requires processing a spatio-temporal image sequence. The actual length of the sequence varies with each instantiation of the gesture. We propose a novel, vision based system for automatic interpretation of a limited set of dynamic hand gestures. This involves extracting the temporal signature of the hand motion from the performed gesture and is subsequently analyzed by a finite-state machine to automatically interpret the performed gesture.

We propose a novel approach to program a robot by demonstrating the task multiple number of times in front of a vision system. Here we integrate human dexterity with sensory data using computer vision techniques in a single platform. A simultaneous feature detection and tracking framework is used to track various features (finger tips and the wrist joint). A Kalman filter does the tracking by predicting the tentative feature location and a HOS-based data clustering algorithm extracts the feature. Color information of the features are used for establishing correspondences. A fast, efficient and robust algorithm for the vision system thus developed process a binocular video sequence to obtain the trajectories and the orientation information of the end effector. The concept of a trajectory bundle is introduced to avoid singularities and to obtain an optimal path.